Abstract

This PEP proposes a redesign of the way in which built-in and extension modulesinteract with the import machinery. This was last revised for Python 3.0 in PEP3121, but did not solve all problems at the time. The goal is to solveimport-related problems by bringing extension modules closer to the way Pythonmodules behave; specifically to hook into the ModuleSpec-based loadingmechanism introduced inPEP 451.

This proposal draws inspiration fromPyType_Spec ofPEP 384 to allow extensionauthors to only define features they need, and to allow future additionsto extension module declarations.

Extensions modules are created in a two-step process, fitting better intothe ModuleSpec architecture, with parallels to__new__ and__init__ of classes.

Extension modules can safely store arbitrary C-level per-module state inthe module that is covered by normal garbage collection and supportsreloading and sub-interpreters.Extension authors are encouraged to take these issues into accountwhen using the new API.

The proposal also allows extension modules with non-ASCII names.

Not all problems tackled inPEP 3121 are solved in this proposal.In particular, problems with run-time module lookup (PyState_FindModule)are left to a future PEP.

Motivation

Python modules and extension modules are not being set up in the same way.For Python modules, the module object is created and set up first, then themodule code is being executed (PEP 302).A ModuleSpec object (PEP 451) is used to hold information about the module,and passed to the relevant hooks.

For extensions (i.e. shared libraries) and built-in modules, the moduleinit function is executed straight away and does both the creation andinitialization. The initialization function is not passed the ModuleSpec,or any information it contains, such as the__file__ or fully-qualifiedname. This hinders relative imports and resource loading.

In Py3, modules are also not being added to sys.modules, which means that a(potentially transitive) re-import of the module will really try to re-importit and thus run into an infinite loop when it executes the module init functionagain. Without access to the fully-qualified module name, it is not trivial tocorrectly add the module to sys.modules either.This is specifically a problem for Cython generated modules, for which it’snot uncommon that the module init code has the same level of complexity asthat of any ‘regular’ Python module. Also, the lack of__file__ and__name__information hinders the compilation of “__init__.py” modules, i.e. packages,especially when relative imports are being used at module init time.

Furthermore, the majority of currently existing extension modules hasproblems with sub-interpreter support and/or interpreter reloading, and, whileit is possible with the current infrastructure to support thesefeatures, it is neither easy nor efficient.Addressing these issues was the goal ofPEP 3121, but many extensions,including some in the standard library, took the least-effort approachto porting to Python 3, leaving these issues unresolved.This PEP keeps backwards compatibility, which should reduce pressure and giveextension authors adequate time to consider these issues when porting.

The current process

Currently, extension and built-in modules export an initialization functionnamed “PyInit_modulename”, named after the file name of the shared library.This function is executed by the import machinery and must return a fullyinitialized module object.The function receives no arguments, so it has no way of knowing about itsimport context.

During its execution, the module init function creates a module objectbased on aPyModuleDef object. It then continues to initialize it by addingattributes to the module dict, creating types, etc.

In the back, the shared library loader keeps a note of the fully qualifiedmodule name of the last module that it loaded, and when a module getscreated that has a matching name, this global variable is used to determinethe fully qualified name of the module object. This is not entirely safe as itrelies on the module init function creating its own module object first,but this assumption usually holds in practice.

The proposal

The initialization function (PyInit_modulename) will be allowed to returna pointer to aPyModuleDef object. The import machinery will be in chargeof constructing the module object, calling hooks provided in thePyModuleDefin the relevant phases of initialization (as described below).

This multi-phase initialization is an additional possibility. Single-phaseinitialization, the current practice of returning a fully initialized moduleobject, will still be accepted, so existing code will work unchanged,including binary compatibility.

ThePyModuleDef structure will be changed to contain a list of slots,similarly toPEP 384’sPyType_Spec for types.To keep binary compatibility, and avoid needing to introduce a new structure(which would introduce additional supporting functions and per-module storage),the currently unusedm_reload pointer ofPyModuleDef will be changed tohold the slots. The structures are defined as:

typedefstruct{intslot;void*value;}PyModuleDef_Slot;typedefstructPyModuleDef{PyModuleDef_Basem_base;constchar*m_name;constchar*m_doc;Py_ssize_tm_size;PyMethodDef*m_methods;PyModuleDef_Slot*m_slots;/* changed from `inquiry m_reload;` */traverseprocm_traverse;inquirym_clear;freefuncm_free;}PyModuleDef;

Them_slots member must be either NULL, or point to an array ofPyModuleDef_Slot structures, terminated by a slot with id set to 0(i.e.{0,NULL}).

To specify a slot, a unique slot ID must be provided.New Python versions may introduce new slot IDs, but slot IDs will never berecycled. Slots may get deprecated, but will continue to be supportedthroughout Python 3.x.

A slot’s value pointer may not be NULL, unless specified otherwise in theslot’s documentation.

The following slots are currently available, and described later:

• Py_mod_create
• Py_mod_exec

Unknown slot IDs will cause the import to fail with SystemError.

When using multi-phase initialization, them_name field ofPyModuleDef willnot be used during importing; the module name will be taken from the ModuleSpec.

Before it is returned from PyInit_*, thePyModuleDef object must be initializedusing the newly addedPyModuleDef_Init function. This sets the object type(which cannot be done statically on certain compilers), refcount, and internalbookkeeping data (m_index).For example, an extension module “example” would be exported as:

staticPyModuleDefexample_def={...}PyMODINIT_FUNCPyInit_example(void){returnPyModuleDef_Init(&example_def);}

ThePyModuleDef object must be available for the lifetime of the module createdfrom it – usually, it will be declared statically.

classBuiltinImporter:defcreate_module(self,spec):module=_imp.create_builtin(spec)defexec_module(self,module):_imp.exec_dynamic(module)defload_module(self,name):# use a backwards compatibility shim_load_module_shim(self,name)

classExtensionFileLoader:defcreate_module(self,spec):module=_imp.create_dynamic(spec)defexec_module(self,module):_imp.exec_dynamic(module)defload_module(self,name):# use a backwards compatibility shim_load_module_shim(self,name)

defcreate_dynamic(spec):name=spec.namepath=spec.origin# Find an already loaded module that used single-phase init.# For multi-phase initialization, mod is NULL, so a new module# is always created.mod=_PyImport_FindExtensionObject(name,name)ifmod:returnmodreturn_PyImport_LoadDynamicModuleWithSpec(spec)defexec_dynamic(module):ifnotisinstance(module,types.ModuleType):# non-modules are skipped -- PyModule_GetDef fails on themreturndef=PyModule_GetDef(module)state=PyModule_GetState(module)ifstateisNULL:PyModule_ExecDef(module,def)defcreate_builtin(spec):name=spec.name# Find an already loaded module that used single-phase init.# For multi-phase initialization, mod is NULL, so a new module# is always created.mod=_PyImport_FindExtensionObject(name,name)ifmod:returnmodforinitname,initfuncinPyImport_Inittab:ifname==initname:m=initfunc()ifisinstance(m,PyModuleDef):def=mreturnPyModule_FromDefAndSpec(def,spec)else:# fall back to single-phase initializationmodule=m_PyImport_FixupExtensionObject(module,name,name)returnmodule

def_PyImport_LoadDynamicModuleWithSpec(spec):path=spec.originpackage,dot,name=spec.name.rpartition('.')# see the "Non-ASCII module names" section for export_hook_namehook_name=export_hook_name(name)# call platform-specific function for loading exported function# from shared libraryexportfunc=_find_shared_funcptr(hook_name,path)m=exportfunc()ifisinstance(m,PyModuleDef):def=mreturnPyModule_FromDefAndSpec(def,spec)module=m# fall back to single-phase initialization....

defPyModule_FromDefAndSpec(def,spec):name=spec.namecreate=Noneforslot,valueindef.m_slots:ifslot==Py_mod_create:create=valueifcreate:m=create(spec,def)else:m=PyModule_New(name)ifisinstance(m,types.ModuleType):m.md_state=Nonem.md_def=defifdef.m_methods:PyModule_AddFunctions(m,def.m_methods)ifdef.m_doc:PyModule_SetDocString(m,def.m_doc)defPyModule_ExecDef(module,def):ifisinstance(module,types.module_type):ifmodule.md_stateisNULL:# allocate a block of zeroed-out memorymodule.md_state=_alloc(module.md_size)ifdef.m_slotsisNULL:returnforslot,valueindef.m_slots:ifslot==Py_mod_exec:value(module)

PyObject*(*PyModuleCreateFunction)(PyObject*spec,PyModuleDef*def)

int(*PyModuleExecFunction)(PyObject*module)

#include<Python.h>staticintspam_exec(PyObject*module){PyModule_AddStringConstant(module,"food","spam");return0;}#ifdef Py_mod_execstaticPyModuleDef_Slotspam_slots[]={{Py_mod_exec,spam_exec},{0,NULL}};#endifstaticPyModuleDefspam_def={PyModuleDef_HEAD_INIT,/* m_base */"spam",/* m_name */PyDoc_STR("Utilities for cooking spam"),/* m_doc */0,/* m_size */NULL,/* m_methods */#ifdef Py_mod_execspam_slots,/* m_slots */#elseNULL,#endifNULL,/* m_traverse */NULL,/* m_clear */NULL,/* m_free */};PyMODINIT_FUNCPyInit_spam(void){#ifdef Py_mod_execreturnPyModuleDef_Init(&spam_def);#elsePyObject*module;module=PyModule_Create(&spam_def);if(module==NULL)returnNULL;if(spam_exec(module)!=0){Py_DECREF(module);returnNULL;}returnmodule;#endif}

PyObject*PyModule_FromDefAndSpec(PyModuleDef*def,PyObject*spec)PyObject*PyModule_FromDefAndSpec2(PyModuleDef*def,PyObject*spec,intmodule_api_version)

PyAPI_FUNC(int)PyModule_ExecDef(PyObject*module,PyModuleDef*def)

PyObject*PyModuleDef_Init(PyModuleDef*);

intPyModule_SetDocString(PyObject*,constchar*)intPyModule_AddFunctions(PyObject*,PyMethodDef*)

defexport_hook_name(name):try:suffix=b'_'+name.encode('ascii')exceptUnicodeEncodeError:suffix=b'U_'+name.encode('punycode').replace(b'-',b'_')returnb'PyInit'+suffix

importimportlib.machineryimportimportlib.utilloader=importlib.machinery.ExtensionFileLoader(name,path)spec=importlib.util.spec_from_loader(name,loader)module=importlib.util.module_from_spec(spec)loader.exec_module(module)returnmodule

Summary of API Changes and Additions

New functions:

• PyModule_FromDefAndSpec (macro)
• PyModule_FromDefAndSpec2
• PyModule_ExecDef
• PyModule_SetDocString
• PyModule_AddFunctions
• PyModuleDef_Init

New macros:

• Py_mod_create
• Py_mod_exec

New types:

• PyModuleDef_Type will be exposed

New structures:

• PyModuleDef_Slot

Other changes:

PyModuleDef.m_reload changes toPyModuleDef.m_slots.

BuiltinImporter andExtensionFileLoader will now implementcreate_module andexec_module.

The internal_imp module will have backwards incompatible changes:create_builtin,create_dynamic, andexec_dynamic will be added;init_builtin,load_dynamic will be removed.

The undocumented functionsimp.load_dynamic andimp.init_builtin willbe replaced by backwards-compatible shims.

Possible Future Extensions

The slots mechanism, inspired byPyType_Slot fromPEP 384,allows later extensions.

Some extension modules exports many constants; for example_ssl hasa long list of calls in the form:

PyModule_AddIntConstant(m,"SSL_ERROR_ZERO_RETURN",PY_SSL_ERROR_ZERO_RETURN);

Converting this to a declarative list, similar toPyMethodDef,would reduce boilerplate, and provide free error-checking whichis often missing.

String constants and types can be handled similarly.(Note that non-default bases for types cannot be portably specifiedstatically; this case would need aPy_mod_exec function that runsbefore the slots are added. The free error-checking would still bebeneficial, though.)

Another possibility is providing a “main” function that would be runwhen the module is given to Python’s-m switch.For this to work, therunpy module will need to be modified to takeadvantage of ModuleSpec-based loading introduced inPEP 451.Also, it will be necessary to add a mechanism for setting up a moduleaccording to slots it wasn’t originally defined with.

Implementation

Work-in-progress implementation is available in a GitHub repository[3];a patchset is at[4].

Previous Approaches

Stefan Behnel’s initial proto-PEP[1]had a “PyInit_modulename” hook that would create a module class,whose__init__ would be then called to create the module.This proposal did not correspond to the (then nonexistent)PEP 451,where module creation and initialization is broken into distinct steps.It also did not support loading an extension into pre-existing module objects.

Alyssa (Nick) Coghlan proposed “Create” and “Exec” hooks, and wrote a prototypeimplementation[2].At this timePEP 451 was still not implemented, so the prototypedoes not use ModuleSpec.

The original version of this PEP usedCreate andExec hooks, and allowedloading into arbitrary pre-constructed objects withExec hook.The proposal made extension module initialization closer to how Python modulesare initialized, but it was later recognized that this isn’t an important goal.The current PEP describes a simpler solution.

A further iteration used a “PyModuleExport” hook as an alternative toPyInit,wherePyInit was used for existing scheme, andPyModuleExport for multi-phase.However, not being able to determine the hook name based on module namecomplicated automatic generation ofPyImport_Inittab by tools like freeze.Keeping only thePyInit hook name, even if it’s not entirely appropriate forexporting a definition, yielded a much simpler solution.

References

Copyright

This document has been placed in the public domain.