Bug report

Bug description:

When an extension module is imported the first time, we load the shared-object file and get the module's init function from it. We then run that init function and use the returned object to decide if the module is single-phase init or multi-phase init.

For isolated subinterpreters, wherePyInterpreterConfig.check_multi_interp_extensions (AKAPy_RTFLAGS_MULTI_INTERP_EXTENSIONS) is True, we immediately fail single-phase init modules. The problem is that at that point the init function has already run, which all sorts of potential side effects and process-global state (including registered callbacks) that we mostly can't clean up.

This has come up before, for example with the readline module. It's potentially a bigger problem than I thought at first, so I'd like to get it sorted out for 3.13.

FWIW, the simplest solution I can think of is to always call the module init func from the main interpreter (without necessarily doing all the other import steps there). That would look something like this:

1. start a normal import in an isolated subinterpreter
2. get the init function
3. switch to the main interpreter
4. call the init function
5. switch back
6. fail if it is single-phase init (and remember that fact)

For the main interpreter and non-isolated subinterpreters, nothing different would happen from now; there would be no switching. Also, if the first attempt was in an isolated interpreter (which would fail), a subsequent import of that module in the main interpreter (or a non-isolated one) would succeed.

The only tricky part is, when the init function raises an exception, how to an propagate it from the main interpreter to the subinterpreter. For multi-phase init (if known) we would just call the init func again after switching back. For single-phase init (or unknown) we'd have preserve the exception somehow. This is something I had to deal with for_interpreters.exec(), but I'm not sure the same thing will work here.

CPython versions tested on:

CPython main branch

Operating systems tested on:

No response

Linked PRs

• gh-117953: Always Run Extension Init Func in Main Interpreter First #117487
• gh-117953: Refactor Import Machinery Code for Extension Modules #118116
• gh-117953: Track All Extension Modules #118156
• gh-117953: Always Run Extension Init Func in Main Interpreter First #118157
• gh-117953: Small Cleanup of Extensions-Related Machinery Code #118167
• gh-117953: Cleanups For fix_up_extension() in import.c #118192
• gh-117953: Let update_global_state_for_extension() Caller Decide If Singlephase or Not #118193
• gh-117953: Add Internal struct _Py_ext_module_loader_info #118194
• gh-117953: Split Up _PyImport_LoadDynamicModuleWithSpec() #118203
• gh-117953: Share More Machinery Code Between Builtin and Dynamic Extensions #118204
• gh-117953: Work Relative to Specific Extension Kinds in the Import Machinery #118205
• gh-117953: Other Cleanups in the Extensions Machinery #118206
• gh-117953: Fix Refleaks Introduced by gh-118194 #118250
• gh-117953: Track Extra Details in Global Extensions Cache #118532
• gh-117953: Imply Single-phase Init if the Init Function Fails #118684
• gh-117953: Skiptest_interpreters properly without GIL #120689
• [3.13] gh-117953: Skiptest_interpreters properly without GIL (GH-120689) #120707