This PEP proposes the introduction of a new ‘ImportEngine’ class as part ofimportlib which would encapsulate all state related to importing modulesinto a single object. Creating new instances of this object would then providean alternative to completely replacing the built-in implementation of theimport statement, by overriding the__import__() function. To work withthe builtin import functionality and importing via import engine objects,this PEP proposes a context management based approach to temporarily replacingthe global import state.
The PEP also proposes inclusion of aGlobalImportEngine subclass and aglobally accessible instance of that class, which “writes through” to theprocess global state. This provides a backwards compatible bridge between theproposed encapsulated API and the legacy process global state, and allowsstraightforward support for related state updates (e.g. selectivelyinvalidating path cache entries whensys.path is modified).
The import system has seen substantial changes since this PEP was originallywritten, as part ofPEP 420 in Python 3.3 andPEP 451 in Python 3.4.
While providing an encapsulation of the import state is still highlydesirable, it is better tackled in a new PEP usingPEP 451 as a foundation,and permitting only the use ofPEP 451 compatible finders and loaders (asthose avoid many of the issues of direct manipulation of global stateassociated with the previous loader API).
Currently, most state related to the import system is stored as module levelattributes in thesys module. The one exception is the import lock, whichis not accessible directly, but only via the related functions in theimpmodule. The current process global import state comprises:
Isolating this state would allow multiple import states to beconveniently stored within a process. Placing the import functionalityin a self-contained object would also allow subclassing to add additionalfeatures (e.g. module import notifications or fine-grained controlover which modules can be imported). The engine would also besubclassed to make it possible to use the import engine API tointeract with the existing process-global state.
The namespace PEPs (especiallyPEP 402) raise a potential need foradditional process global state, in order to correctly update package pathsassys.path is modified.
Finally, providing a coherent object for all this state makes it feasible toalso provide context management features that allow the import state to betemporarily substituted.
We propose introducing an ImportEngine class to encapsulate importfunctionality. This includes an__import__() method which canbe used as an alternative to the built-in__import__() whendesired and also animport_module() method, equivalent toimportlib.import_module()[3].
Since there are global import state invariants that are assumed and should bemaintained, we introduce aGlobalImportState class with an interfaceidentical toImportEngine but directly accessing the current global importstate. This can be easily implemented using class properties.
The proposed extension consists of the following objects:
importlib.engine.ImportEngine
from_engine(self,other)Create a new import object from another ImportEngine instance. Thenew object is initialised with a copy of the state inother. Whencalled onimportlibengine.sysengine,from_engine()can beused to create anImportEngineobject with acopy of theglobal import state.
__import__(self,name,globals={},locals={},fromlist=[],level=0)Reimplementation of the builtin__import__()function. Theimport of a module will proceed using the state stored in theImportEngine instance rather than the global import state. For fulldocumentation of__import__functionality, see[2] .__import__()fromImportEngineand its subclasses can be usedto customise the behaviour of theimportstatement by replacing__builtin__.__import__withImportEngine().__import__.
import_module(name,package=None)A reimplementation ofimportlib.import_module()which uses theimport state stored in the ImportEngine instance. See[3] for a fullreference.
modules,path,path_hooks,meta_path,path_importer_cacheInstance-specific versions of their process globalsysequivalents
importlib.engine.GlobalImportEngine(ImportEngine)
Convenience class to provide engine-like access to the global state.Provides__import__(),import_module()andfrom_engine()methods likeImportEnginebut writes through to the global stateinsys.
To support various namespace package mechanisms, whensys.path is altered,tools likepkgutil.extend_path should be used to also modify other partsof the import state (in this case, package__path__ attributes). The pathimporter cache should also be invalidated when a variety of changes are made.
TheImportEngine API will provide convenience methods that automaticallymake related import state updates as part of a single operation.
importlib.engine.sysengine
A precreated instance ofGlobalImportEngine. Intended for use byimporters and loaders that have been updated to accept optionalengineparameters and withImportEngine.from_engine(sysengine)to start witha copy of the process global import state.
Rather than attempting to update thePEP 302 APIs to accept additional state,this PEP proposes thatImportEngine support the content managementprotocol (similar to the context substitution mechanisms in thedecimalmodule).
The context management mechanism forImportEngine would:
The precise API for this is TBD (but will probably use a distinct contextmanagement object, along the lines of that created bydecimal.localcontext).
The current proposal relies on thefrom_engine() API to fall back to theglobal import state. It may be desirable to offer a variant that instead fallsback to the global import state dynamically.
However, one big advantage of starting with an “as isolated as possible”design is that it becomes possible to experiment with subclasses that blurthe boundaries between the engine instance state and the process global statein various ways.
Due to platform limitations, only one copy of each builtin and extensionmodule can readily exist in each process. Accordingly, it is impossible foreachImportEngine instance to load such modules independently.
The simplest solution is forImportEngine to refuse to load such modules,raisingImportError.GlobalImportEngine would be able to load themnormally.
ImportEngine will still return such modules from a prepopulated modulecache - it’s only loading them directly which causes problems.
Related to the previous open issue is the question of what state to substitutewhen using the context management API. It is currently the case that replacingsys.modules can be unreliable due to cached references and there’s theunderlying fact that having independent copies of some modules is simplyimpossible due to platform limitations.
As part of this PEP, it will be necessary to document explicitly:
ImportEngine context management API, or otherwise scoped toImportEngine instances)A reference implementation[4] for an earlier draft of this PEP, based onBrett Cannon’s importlib has been developed by Greg Slodkowicz as part of the2011 Google Summer of Code. Note that the current implementation avoidsmodifying existing code, and hence duplicates a lot of things unnecessarily.An actual implementation would just modify any such affected code in place.
That earlier draft of the PEP proposed change thePEP 302 APIs to support passingin an optional engine instance. This had the (serious) downside of not correctlyaffecting further imports from the imported module, hence the change to thecontext management based proposal for substituting the global state.
This document has been placed in the public domain.
Source:https://github.com/python/peps/blob/main/peps/pep-0406.rst
Last modified:2025-02-01 08:59:27 GMT