Cython – the most important use case for this PEP and the only explicitone – is not ready for multi-phase initialization yet.It keeps global state in C-level static variables.See discussion atCython issue 1923.
The PEP is deferred until the situation changes.
This PEP proposes implementation that allows built-in and extensionmodules to be executed in the__main__ namespace usingthePEP 489 multi-phase initialization.
With this, a multi-phase initialization enabled module can be runusing following command:
$ python3 -m _testmultiphaseThis is a test module named __main__.
Currently, extension modules do not support all functionality ofPython source modules.Specifically, it is not possible to run extension modules as scripts usingPython’s-m option.
The technical groundwork to make this possible has been done forPEP 489,and enabling the-m option is listed in that PEP’s“Possible Future Extensions” section.Technically, the additional changes proposed here are relatively small.
Extension modules’ lack of support for the-m option has traditionallybeen worked around by providing a Python wrapper.For example, the_pickle module’s command line interface is in thepure-Pythonpickle module (along with a pure-Python reimplementation).
This works well for standard library modules, as building command lineinterfaces using the C API is cumbersome.However, other users may want to create executable extension modules directly.
An important use case is Cython, a Python-like language that compiles toC extension modules.Cython is a (near) superset of Python, meaning that compiling a Python modulewith Cython will typically not change the module’s functionality, allowingCython-specific features to be added gradually.This PEP will allow Cython extension modules to behave the same as their Pythoncounterparts when run using the-m option.Cython developers consider the feature worth implementing (seeCython issue 1715).
Python’s-m option is handled by the functionrunpy._run_module_as_main.
The module specified by-m is not imported normally.Instead, it is executed in the namespace of the__main__ module,which is created quite early in interpreter initialization.
For Python source modules, running in another module’s namespace is nota problem: the code is executed withlocals andglobals set to theexisting module’s__dict__.This is not the case for extension modules, whosePyInit_* entry pointtraditionally both created a new module object (usingPyModule_Create),and initialized it.
Since Python 3.5, extension modules can usePEP 489 multi-phase initialization.In this scenario, thePyInit_* entry point returns aPyModuleDefstructure: a description of how the module should be created and initialized.The extension can choose to customize creation of the module object usingthePy_mod_create callback, or opt to use a normal module object by notspecifyingPy_mod_create.Another callback,Py_mod_exec, is then called to initialize the moduleobject, e.g. by populating it with methods and classes.
Multi-phase initialization makes it possible to execute an extension module inanother module’s namespace: if aPy_mod_create callback is not specified,the__main__ module can be passed to thePy_mod_exec callback to beinitialized, as if__main__ was a freshly constructed module object.
One complication in this scheme is C-level module state.Each module has amd_state pointer that points to a region of memoryallocated when an extension module is created.ThePyModuleDef specifies how much memory is to be allocated.
The implementation must take care thatmd_state memory is allocated at mostonce.Also, thePy_mod_exec callback should only be called once per module.The implications of multiply-initialized modules are too subtle to requireexpecting extension authors to reason about them.Themd_state pointer itself will serve as a guard: allocating the memoryand callingPy_mod_exec will always be done together, and initializing anextension module will fail ifmd_state is already non-NULL.
Since the__main__ module is not created as an extension module,itsmd_state is normallyNULL.Before initializing an extension module in__main__’s context, its modulestate will be allocated according to thePyModuleDef of that module.
WhilePEP 489 was designed to make these changes generally possible,it’s necessary to decouple module discovery, creation, and initializationsteps for extension modules, so that another module can be used instead ofa newly initialized one, and the functionality needs to be added torunpy andimportlib.
A new optional method for importlib loaders will be added.This method will be calledexec_in_module and will take twopositional arguments: module spec and an already existing module.Any import-related attributes, such as__spec__ or__name__,already set on the module will be ignored.
Therunpy._run_module_as_main function will look for this newloader method.If it is present,runpy will execute it instead of trying to load andrun the module’s Python code.Otherwise,runpy will act as before.
importlib’sExtensionFileLoader will get an implementation ofexec_in_module that will call a new function,_imp.exec_in_module.
_imp.exec_in_module will use existing machinery to find and call anextension module’sPyInit_* function.
ThePyInit_* function can return either a fully initialized module(single-phase initialization) or aPyModuleDef (forPEP 489 multi-phaseinitialization).
In the single-phase initialization case,_imp.exec_in_module will raiseImportError.
In the multi-phase initialization case, thePyModuleDef and the module tobe initialized will be passed to a new function,PyModule_ExecInModule.
This function raisesImportError if thePyModuleDef specifiesaPy_mod_create slot, or if the module has already been initialized(i.e. itsmd_state pointer is notNULL).Otherwise, the function will initialize the module according to thePyModuleDef.
This PEP maintains backwards compatibility.It only adds new functions, and a new loader method that is added fora loader that previously did not support running modules as__main__.
The reference implementation of this PEP is available atGitHub.
This document has been placed in the public domain.
Source:https://github.com/python/peps/blob/main/peps/pep-0547.rst
Last modified:2025-02-01 08:59:27 GMT