The interpreter switch

Python 3.12 added a way for modules to mark whether they may beloaded in a subinterpreter: thePy_mod_multiple_interpreters slot.Setting it to the “not supported” value signals that an extensioncan only be loaded in the main interpreter.

Unfortunately, Python can only get this information bycalling themodule export hook.For single-phase modules, that creates the module object and runs arbitraryinitialization code.For modules that setPy_mod_multiple_interpreters to “not supported”,this initialization needs to happen in the main interpreter.

To make this work, if a new module is loaded in a sub-interpreter, Pythontemporarily switches to the main interpreter, calls the export hookthere, and then either switches back and redoes the import, or fails.

This unnecessary and fragile extra work highlights the underlying design issue:Python has no way to get information about an extensionbefore the extension can, potentially, fully initialize itself.

Using slots without a wrapper struct

The existingPyModuleDef is a struct with some fixed fields anda “slots” array.Unlike slots, the fixed fields cannot be individually deprecated and replaced.This proposal does away with fixed fields and proposes using a slots arraydirectly, without a wrapper struct.

ThePyModuleDef_Slot struct does have some downsides compared to fixed fields.We believe these are fixable, but leave that out of scope of this PEP(see “Improving slots in general” in the Possible Future Directions section).

Tokens

A staticPyModuleDef has another purpose besides describinghow a module should be created.As a statically allocated singleton that remains attached to the module object,it allows extension authors to check whether a given Python module is “theirs”:if a module object has a knownPyModuleDef, its module state will havea known memory layout.

An analogous issue was solved for types by addingPy_tp_token.This proposal adds the same mechanism to modules.

Unlike types, the import mechanism often has a pointer that’s known to besuitable as a token value; in these cases it can provide a default token.Thus, module tokens do not need a variant of the inelegantPy_TP_USE_SPEC.

To help extensions that straddle Python versions,PyModuleDef addressesare used as default tokens, and where it’s reasonable, they are madeinterchangeable with tokens.

Soft-deprecating the existing export hook

The only reason for authors ofexisting extensions to switch to theAPI proposed here is that it allows a single module for both free-threadedand non-free-threaded builds.It is important that Pythonallows that, but for many existing modules,it is nowhere near worth losing compatibility with 3.14 and lower versions.

It is much too early to plan deprecation of the old API.

Instead, this PEP proposes to stop adding new features to thePyInit_*scheme.After all, the perfect time for extension authors to switch is when they wantto modify module initialization anyway.

The export hook

When importing an extension module, Python will now first look for an export hooklike this:

PyModuleDef_Slot*PyModExport_<NAME>(void);

where<NAME> is the name of the module.For non-ASCII names, it will instead look forPyModExportU_<NAME>,with<NAME> encoded as for existingPyInitU_* hooks(that is,punycode-encoded with hyphens replaced by underscores).

If not found, the import will continue as in previous Python versions (that is,by looking up aPyInit_* orPyInitU_* function).

If found, Python will call the hook with no arguments.

On failure, the export hook must return NULL with an exception set.This will cause the import to fail.(Python will not fall back toPyInit_* on error.)

On success, the hook must return a pointer to an array ofPyModuleDef_Slot structs.Python will then create a module based on the given slots by calling functionsproposed below:PyModule_FromSlotsAndSpec andPyModule_Exec.See their description for requirements on the slots array.

The returned array and all data it points to (recursively) must remain validand constant until runtime shutdown.(We expect functions to export a static constant, or one of several constantschosen depending on, for example,Py_Version. Dynamic behaviour shouldgenerally happen in thePy_mod_create andPy_mod_exec functions.)

Dynamic creation

A new function will be added to create a module from an array of slots:

PyObject*PyModule_FromSlotsAndSpec(constPyModuleDef_Slot*slots,PyObject*spec)

Theslots argument must point to an array ofPyModuleDef_Slot structures,terminated by a slot withslot=0 (typically written as{0} in C).There are no required slots, thoughslots must not beNULL.It follows that minimal input contains only the terminator slot.

Thespec argument is a duck-typed ModuleSpec-like object, meaning that anyattributes defined forimportlib.machinery.ModuleSpec have matchingsemantics.Thename attribute is required, but this limitation may be lifted in thefuture.Thename will be usedinstead of thePy_mod_name slot (just likePyModule_FromDefAndSpec ignoresPyModuleDef.m_name).

The slots arrays for bothPyModule_FromSlotsAndSpec and the new export hookwill only allow up to onePy_mod_exec slot.Arrays inPyModuleDef.m_slots may have more; this will not change.This limitation is easy to work around and multipleexec slots are rarelyused[1].

For modules created without aPyModuleDef, thePy_mod_create functionwill be called withNULL for the second argument (def).(In the future, if we find a use case for passing the input slots array, a newslot with an updated signature can be added.)

Unlike thePyModExport_* hook, theslots array may be changed ordestroyed after thePyModule_FromSlotsAndSpec call.(That is, Python must take a copy of all input data.)As an exception, anyPyMethodDef array given byPy_mod_methodsmust be statically allocated (or be otherwise guaranteed to outlive theobjects created from it). This limitation may be lifted in the future.

A new function,PyModule_Exec, will be added to run theexec slot(s) for a module.This acts likePyModule_ExecDef, but supports modules created using slots,and does not take an explicitdef:

intPyModule_Exec(PyObject*module)

Calling this is required to fully initialize a module.PyModule_FromSlotsAndSpec willnot run it (just likePyModule_FromDefAndSpec does not callPyModule_ExecDef).

For modules created from adef, calling this is equivalent tocallingPyModule_ExecDef(module,PyModule_GetDef(module)).

Tokens

Module objects will optionally store a “token”: avoid* pointersimilar toPy_tp_token for types.

If specified, using a newPy_mod_token slot, the module token must:

• outlive the module, so it’s not reused for something else while the moduleexists; and
• “belong” to the extension module where the module lives, so it will notclash with other extension modules.

(Typically, it should be the slots array orPyModuleDef that a module iscreated from, or another static constant for dynamically created modules.)

When the address of aPyModuleDef is used as a module’s token,the module should behave as if it was created from thatPyModuleDef.In particular, the module state must have matching layout and semantics.

Modules created using thePyModule_FromSlotsAndSpec or thePyModExport_<NAME> export hook can use a newPy_mod_token slotto set the token.

Modules created from aPyModuleDef will have the token set to thatdefinition. An explicitPy_mod_token slot will we rejected for these.(This allows implementations to share storage for the token and def.)

For modules created via the new export hook, the tokenwill be set to the address of the slots array by default.(This doesnot apply to modules created byPyModule_FromSlotsAndSpec,as that function’s input might not outlive the module.)

The token will not be set for non-PyModuleType instances.

APyModule_GetToken function will be added to get the token.Since the result may beNULL, it will be passed via a pointer; the functionwill return 0 on success and -1 on failure:

intPyModule_GetToken(PyObject*,void**token_p)

A newPyType_GetModuleByToken function will be added, with a signaturelike the existingPyType_GetModuleByDef but aconstvoid*token argument,and the same behaviour except matching tokens rather than only defs,and returning a strong reference.

For easier backwards compatibility, the existingPyType_GetModuleByDefwill be changed to also allow a token (cast to aPyModuleDef* pointer) asthedef argument.That is,PyType_GetModuleByToken andPyType_GetModuleByDef will differonly in the formal signature of the second argument and by returning aborrowed vs. strong reference.(ThePyModule_GetDef function will not get a similar change, as users mayaccess members of its result.)

New slots

For each field of thePyModuleDef struct, except ones fromPyModuleDef_HEAD_INIT, a new slot ID will be provided:Py_mod_name,Py_mod_doc,Py_mod_clear, etc.Slots related to the module state rather than the module object willuse aPy_mod_state_ prefix.SeeNew API summary for a full list.

All new slots – these andPy_tp_token discussed above – may not berepeated in the slots array, and may not be used in aPyModuleDef.m_slots array.They may not have aNULL value (instead, the slot can be omitted entirely).

Note that currently, for modules created from aspec (that is, usingPyModule_FromDefAndSpec), thePyModuleDef.m_name member is ignoredand the name from the spec is used instead.All API proposed in this document creates modules from aspec, and it willignorePy_mod_name in the same way.The slot will be optional, but extension authors are strongly encouraged toinclude it for the benefit of future APIs, external tooling, debugging,and introspection.

Bits & Pieces

APyMODEXPORT_FUNC macro will be added, similar to thePyMODINIT_FUNCmacro but withPyModuleDef_Slot* as the return type.

APyModule_GetStateSize function will be added to retrieve the size setbyPy_mod_state_size orPyModuleDef.m_size.Since the result may be -1 (for single-phase-init modules), it will be outputvia a pointer; the function will return 0 on success and -1 on failure:

intPyModule_GetStateSize(PyObject*,Py_ssize_t*result);

Soft-deprecating the existing export hook

ThePyInit_* export hook will besoft-deprecated.

New API summary

Python will load a new module export hook, with two variants:

PyModuleDef_Slot*PyModExport_<NAME>(void);PyModuleDef_Slot*PyModExportU_<ENCODED_NAME>(void);

The following functions will be added:

PyObject*PyModule_FromSlotsAndSpec(constPyModuleDef_Slot*,PyObject*spec)intPyModule_Exec(PyObject*)intPyModule_GetToken(PyObject*,void**)PyObject*PyType_GetModuleByToken(PyTypeObject*type,constvoid*token)intPyModule_GetStateSize(PyObject*,Py_ssize_t*result);

A new macro will be added:

PyMODEXPORT_FUNC

And new slot types (#defined names for small integers):

• Py_mod_name (equivalent toPyModuleDef.m_name)
• Py_mod_doc (equivalent toPyModuleDef.m_doc)
• Py_mod_state_size (equivalent toPyModuleDef.m_size)
• Py_mod_methods (equivalent toPyModuleDef.m_methods)
• Py_mod_state_traverse (equivalent toPyModuleDef.m_traverse)
• Py_mod_state_clear (equivalent toPyModuleDef.m_clear)
• Py_mod_state_free (equivalent toPyModuleDef.m_free)
• Py_mod_token (see above)

All this will be added to the Limited API.

Forward compatibility

If a module defines the new export hook, CPython versions that implement thisPEP will ignore the traditionalPyInit_* hook.

Extensions that straddle Python versions are expected to define both hooks;each build of CPython will “pick” the newest one that it supports.

Porting guide

Here is a guide to convert an existing module to the new API, includingsome tricky edge cases.It should be moved to a HOWTO in the documentation.

This guide is meant for hand-written modules. For code generators and languagewrappers, theBackwards compatibility shim below may be more useful.

1. Scan your code for uses ofPyModule_GetDef. This function willreturnNULL for modules that use the new mechanism. Instead:
   • For getting the contents of the module’sPyModuleDef, use the C structdirectly. Alternatively, get attributes from the module using, forexample,PyModule_GetNameObject, the__doc__ attribute, andPyModule_GetStateSize.(Note that Python code can mutate a module’s attributes.)
   • For testing if a module object is “yours”, usePyModule_GetTokeninstead.Later in this guide, you’ll set the token tobe the existingPyModuleDef structure.
2. Optionally, scan your code for uses ofPyType_GetModuleByDef,and replace them withPyType_GetModuleByToken.Later in this guide, you’ll set the token tobe the existingPyModuleDef structure.

   (You may skip this step if targetting Python versions that don’t exposePyType_GetModuleByToken, sincePyType_GetModuleByDef isbackwards-compatible.)

3. Look at the function identified byPy_mod_create, if any.Make sure that it does not use its second argument (PyModuleDef),as it will be called withNULL.Instead of the argument, use the existingPyModuleDef struct directly.
4. If using multiplePy_mod_exec slots, consolidate them: pick one ofthe functions, or write a new one, and call the others from it.Remove all but onePy_mod_exec slots.
5. Make a copy of the existingPyModuleDef_Slot array pointed to bythem_slots member of yourPyModuleDef. If you don’t have anexisting slots array, create one like this:

   staticPyModuleDef_Slotmodule_slots[]={{0}};

   Give this array a unique name.Further examples will assume that you’ve named itmodule_slots.

6. Add slots for all members of the existingPyModuleDef structure.SeeNew API summary for a list of the new slots.For example, to add a name and docstring:

   staticPyModuleDef_Slotmodule_slots[]={{Py_mod_name,"mymodule"},{Py_mod_doc,(char*)PyDoc_STR("my docstring")},// ... (keep existing slots here){0}};

7. If you switched fromPyModule_GetDef toPyModule_GetToken,and/or if you usePyType_GetModuleByDef orPyType_GetModuleByToken,add aPy_mod_token slot pointing to the existingPyModuleDef struct:

   staticPyModuleDef_Slotmodule_slots[]={// ... (keep existing slots here){Py_mod_token,&your_module_def},{0}};

8. Add a new export hook.

   PyMODEXPORT_FUNCPyModExport_examplemodule(PyObject);PyMODEXPORT_FUNCPyModExport_examplemodule(void){returnmodule_slots;}

The new export hook will be used on Python 3.15 and above.Once your module no longer supports lower versions:

1. Delete thePyInit_ function.
2. If the existingPyModuleDef struct is usedonly forPy_mod_tokenand/orPyType_GetModuleByToken, you may remove thePy_mod_tokenline and replace&your_module_def withmodule_slots everywhere else.
3. Delete any unused data.ThePyModuleDef struct and the original slots array are likely to beunused.

Backwards compatibility shim

It is possible to write a generic function that implements the “old” exporthook (PyInit_) in terms of the API proposed here.

The following implementation can be copied and pasted to a project; only thenamesPyInit_examplemodule (twice) andPyModExport_examplemodule shouldneed adjusting.

When added to theExample below and compiled with anon-free-threaded build of this PEP’s reference implementation, the resultingextension is compatible with non-free-threading 3.9+ builds, in addition to afree-threading build of the reference implementation.(The module must be named without a version tag, e.g.examplemodule.so,and be placed onsys.path.)

Full support for creating such modules will require backports of some newAPI, and support in build/install tools. This is out of scope of this PEP.(In particular, the demo “cheats” by using a subset of Limited API 3.15 thathappens to work on 3.9; a proper implementation would use Limited API 3.9with backport shims for new API likePy_mod_name.)

This implementation places a few additional requirements on the slots array:

• Slots that correspond toPyModuleDef members must come first.
• APy_mod_name slot is required.
• AnyPy_mod_token must be set to&module_def_and_token, defined here.

#include<string.h>     // memsetPyMODINIT_FUNCPyInit_examplemodule(void);staticPyModuleDefmodule_def_and_token;PyMODINIT_FUNCPyInit_examplemodule(void){PyModuleDef_Slot*slot=PyModExport_examplemodule();if(module_def_and_token.m_name){// Take care to only set up the static PyModuleDef once.// (PyModExport might theoretically return different data each time.)returnPyModuleDef_Init(&module_def_and_token);}intcopying_slots=1;for(/* slot set above */;slot->slot;slot++){switch(slot->slot){// Set PyModuleDef members from slots. These slots must come first.#       define COPYSLOT_CASE(SLOT, MEMBER, TYPE)                            \            case SLOT:                                                      \                if (!copying_slots) {                                       \                    PyErr_SetString(PyExc_SystemError,                      \                                    #SLOT " must be specified earlier");    \                    goto error;                                             \                }                                                           \                module_def_and_token.MEMBER = (TYPE)(slot->value);          \                break;                                                      \/////////////////////////////////////////////////////////////////COPYSLOT_CASE(Py_mod_name,m_name,char*)COPYSLOT_CASE(Py_mod_doc,m_doc,char*)COPYSLOT_CASE(Py_mod_state_size,m_size,Py_ssize_t)COPYSLOT_CASE(Py_mod_methods,m_methods,PyMethodDef*)COPYSLOT_CASE(Py_mod_state_traverse,m_traverse,traverseproc)COPYSLOT_CASE(Py_mod_state_clear,m_clear,inquiry)COPYSLOT_CASE(Py_mod_state_free,m_free,freefunc)casePy_mod_token:// With PyInit_, the PyModuleDef is used as the token.if(slot->value!=&module_def_and_token){PyErr_SetString(PyExc_SystemError,"Py_mod_token must be set to ""&module_def_and_token");gotoerror;}break;default:// The remaining slots become m_slots in the def.// (`slot` now points to the "rest" of the original//  zero-terminated array.)if(copying_slots){module_def_and_token.m_slots=slot;}copying_slots=0;break;}}if(!module_def_and_token.m_name){// This function needs m_name as the "is initialized" marker.PyErr_SetString(PyExc_SystemError,"Py_mod_name slot is required");gotoerror;}returnPyModuleDef_Init(&module_def_and_token);error:memset(&module_def_and_token,0,sizeof(module_def_and_token));returnNULL;}

Exporting a data pointer rather than a function

This proposes a new module exportfunction, which is expected toreturn static constant data.That data could be exported directly as a data pointer.

With a function, we avoid dealing with a new kind of exported symbol.

A function also allows the extension to introspect its environment in a limitedway – for example, to tailor the returned data to the current Python version.

ChangingPyModuleDef to not bePyObject

It is possible to changePyModuleDef to no longer include thePyObjectheader, and continue using the currentPyInit_* hook.There are several issues with this approach:

• The import machinery would need to examine bit-patterns in the objects todistinguish between different memory layouts:
  • the “old”PyObject-basedPyModuleDef, returned by currentabi3extensions,
  • the newPyModuleDef,
  • PyObject-based module objects, for single-phase initialization.

  This is fragile, and places constraints on future changes toPyObject:the memory layouts need to staydistinguishable until both single-phaseinitialization and the current Stable ABI are no longer supported.

• PyModuleDef_Init is documented to “Ensure a module definition is aproperly initialized Python object that correctly reports its type anda reference count.”This would need to change without warning, breaking any user code that treatsPyModuleDefs as Python objects.

Improving slots in general

Slots – and specifically the existingPyModuleDef_Slot – do have a fewshortcomings. The most important are:

• Type safety:void* is used for data pointers, function pointersand small integers, requiring casting that is technically undefinedbehaviour in C – but works in practice on all relevant architectures.(For example:Py_tp_doc marks a string;Py_mod_gil an integer.)
• Limited forward compatibility: if an extension provides a slot ID that’sunknown to the current interpreter, module creation will fail.This makes it cumbersome to use “optional” features – ones that should onlytake effect if the interpreter supports them.(The recently added slotsPy_mod_gil andPy_mod_multiple_interpretersare good examples.)

  One workaround is to checkPy_Version in the export function,and return a slot array suitable for the current interpreter.

Updating defaults

With a new API, we could update defaults for thePy_mod_multiple_interpreters andPy_mod_gil slots.

The inittab

We’ll need to allowPyModuleDef-less slots in the inittab –that is, add a new variant ofPyImport_ExtendInittab.Should that be part of this PEP?

The inittab is used for embedding, where a common/stable ABI is not thatimportant. So, it might be OK to leave this to a later change.