Movatterモバイル変換

[Python-Dev] Pre-PEP: Redesigning extension modules

• Previous message:[Python-Dev] PEP 446 (make FD non inheritable) ready for afinal review
• Next message:[Python-Dev] Pre-PEP: Redesigning extension modules
• Messages sorted by:[ date ][ thread ][ subject ][ author ]

Hi,this has been subject to a couple of threads on python-dev already, forexample:http://thread.gmane.org/gmane.comp.python.devel/135764/focus=140986http://thread.gmane.org/gmane.comp.python.devel/141037/focus=141046It originally came out of issues 13429 and 16392.http://bugs.python.org/issue13429http://bugs.python.org/issue16392Here's an initial attempt at a PEP for it. It is based on the (unfinished)ModuleSpec PEP, which is being discussed on the import-sig mailing list.http://mail.python.org/pipermail/import-sig/2013-August/000688.htmlStefanPEP: 4XXTitle: Redesigning extension modulesVersion: $Revision$Last-Modified: $Date$Author: Stefan Behnel <stefan_ml at behnel.de>BDFL-Delegate: ???Discussions-To: ???Status: DraftType: Standards TrackContent-Type: text/x-rstCreated: 11-Aug-2013Python-Version: 3.4Post-History: 23-Aug-2013Resolution:Abstract========This PEP proposes a redesign of the way in which extension modules interactwith the interpreter runtime. This was last revised for Python 3.0 in PEP3121, but did not solve all problems at the time. The goal is to solve themby bringing extension modules closer to the way Python modules behave.An implication of this PEP is that extension modules can use arbitrarytypes for their module implementation and are no longer restricted totypes.ModuleType. This makes it easy to support properties at the modulelevel and to safely store arbitrary global state in the module that iscovered by normal garbage collection and supports reloading andsub-interpreters.Motivation==========Python modules and extension modules are not being set up in the same way.For Python modules, the module is created and set up first, then the modulecode is being executed. For extensions, i.e. shared libraries, the moduleinit function is executed straight away and does both the creation andinitialisation. This means that it knows neither the __file__ it is beingloaded from nor its package (i.e. its fully qualified module name, FQMN).This hinders relative imports and resource loading. In Py3, it's also notbeing added to sys.modules, which means that a (potentially transitive)re-import of the module will really try to reimport it and thus run into aninfinite loop when it executes the module init function again. And withoutthe FQMN, it is not trivial to correctly add the module to sys.modules either.This is specifically a problem for Cython generated modules, for which it'snot uncommon that the module init code has the same level of complexity asthat of any 'regular' Python module. Also, the lack of a FQMN and correctfile path hinders the compilation of __init__.py modules, i.e. packages,especially when relative imports are being used at module init time.Furthermore, the majority of currently existing extension modules hasproblems with sub-interpreter support and/or reloading and it is neithereasy nor efficient with the current infrastructure to support thesefeatures. This PEP also addresses these issues.The current process===================Currently, extension modules export an initialisation function named"PyInit_modulename", named after the file name of the shared library. Thisfunction is executed by the import machinery and must return either NULL inthe case of an exception, or a fully initialised module object. Thefunction receives no arguments, so it has no way of knowing about itsimport context.During its execution, the module init function creates a module objectbased on a PyModuleDef struct. It then continues to initialise it by addingattributes to the module dict, creating types, etc.In the back, the shared library loader keeps a note of the fully qualifiedmodule name of the last module that it loaded, and when a module getscreated that has a matching name, this global variable is used to determinethe FQMN of the module object. This is not entirely safe as it relies onthe module init function creating its own module object first, but thisassumption usually holds in practice.The main problem in this process is the missing support for passing stateinto the module init function, and for safely passing state through to themodule creation code.The proposal============The current extension module initialisation will be deprecated in favour ofa new initialisation scheme. Since the current scheme will continue to beavailable, existing code will continue to work unchanged, including binarycompatibility.Extension modules that support the new initialisation scheme must export anew public symbol "PyModuleCreate_modulename", where "modulename" is thename of the shared library. This mimics the previous naming convention forthe "PyInit_modulename" function.This symbol must resolve to a C function with the following signature::    PyObject* (*PyModuleTypeCreateFunction)(PyObject* module_spec)The "module_spec" argument receives a "ModuleSpec" instance, as defined inPEP 4XX (FIXME). (All names are obviously up for debate and bike-sheddingat this point.)When called, this function must create and return a type object, either aPython class or an extension type that is allocated on the heap. This typewill be instantiated as module instance by the importer.There is no requirement for this type to be exactly or a subtype oftypes.ModuleType. Any type can be returned. This follows the currentsupport for allowing arbitrary objects in sys.modules and makes it easierfor extension modules to define a type that exactly matches their needs forholding module state.The constructor of this type must have the following signature::    def __init__(self, module_spec):The "module_spec" argument receives the same object as the one passed intothe module type creation function.Implementation==============XXX - not startedReloading and Sub-Interpreters==============================To "reload" an extension module, the module create function is executedagain and returns a new module type. This type is then instantiated as bythe original module loader and replaces the previous entry in sys.modules.Once the last references to the previous module and its type are gone, bothwill be subject to normal garbage collection.Sub-interpreter support is an inherent property of the design. Duringimport in the sub-interpreter, the module create function is executedand returns a new module type that is local to the sub-interpreter. Boththe type and its module instance are subject to garbage collection in thesub-interpreter.Open questions==============It is not immediately obvious how extensions should be handled that want toregister more than one module in their module init function, e.g. compiledpackages. One possibility would be to leave the setup to the user, whowould have to know all FQMNs anyway in this case (or could construct themfrom the module spec of the current module), although not the importfile path. A C-API could be provided to register new module types in thecurrent interpreter, given a user provided ModuleSpec.There is no inherent requirement for the module creation function toactually return a type. It could return a arbitrary callable that creates a'modulish' object when called. Should there be a type check in place thatmakes sure that what it returns is a type? I don't currently see a need forthis.Copyright=========This document has been placed in the public domain.

• Previous message:[Python-Dev] PEP 446 (make FD non inheritable) ready for afinal review
• Next message:[Python-Dev] Pre-PEP: Redesigning extension modules
• Messages sorted by:[ date ][ thread ][ subject ][ author ]