This PEP proposes to add a new class to importlib.machinery called“ModuleSpec”. It will provide all the import-related information usedto load a module and will be available without needing to load themodule first. Finders will directly provide a module’s spec instead ofa loader (which they will continue to provide indirectly). The importmachinery will be adjusted to take advantage of module specs, includingusing them to load modules.
The changes in this proposal are an opportunity to make severalexisting terms and concepts more clear, whereas currently they are(unfortunately) ambiguous. New concepts are also introduced in thisproposal. Finally, it’s worth explaining a few other existing termswith which people may not be so familiar. For the sake of context, hereis a brief summary of all three groups of terms and concepts. A moredetailed explanation of the import system is found at[2].
In this proposal, a module’s “name” refers to its fully-qualified name,meaning the fully-qualified name of the module’s parent (if any) joinedto the simple name of the module by a period.
A “finder” is an object that identifies the loader that the importsystem should use to load a module. Currently this is accomplished bycalling the finder’s find_module() method, which returns the loader.
Finders are strictly responsible for providing the loader, which they dothrough their find_module() method. The import system then uses thatloader to load the module.
A “loader” is an object that is used to load a module during import.Currently this is done by calling the loader’s load_module() method. Aloader may also provide APIs for getting information about the modulesit can load, as well as about data from sources associated with such amodule.
Right now loaders (via load_module()) are responsible for certainboilerplate, import-related operations. These are:
This all takes place during the import system’s call toLoader.load_module().
This is a new term and concept. The idea of it exists subtly in theimport system already, but this proposal makes the concept explicit.
“origin” in an import context means the system (or resource within asystem) from which a module originates. For the purposes of thisproposal, “origin” is also a string which identifies such a resource orsystem. “origin” is applicable to all modules.
For example, the origin for built-in and frozen modules is theinterpreter itself. The import system already identifies this origin as“built-in” and “frozen”, respectively. This is demonstrated in thefollowing module repr: “<module ‘sys’ (built-in)>”.
In fact, the module repr is already a relatively reliable, thoughimplicit, indicator of a module’s origin. Other modules also indicatetheir origin through other means, as described in the entry for“location”.
It is up to the loader to decide on how to interpret and use a module’sorigin, if at all.
This is a new term. However the concept already exists clearly in theimport system, as associated with the__file__ and__path__attributes of modules, as well as the name/term “path” elsewhere.
A “location” is a resource or “place”, rather than a system at large,from which a module is loaded. It qualifies as an “origin”. Examplesof locations include filesystem paths and URLs. A location isidentified by the name of the resource, but may not necessarily identifythe system to which the resource pertains. In such cases the loaderwould have to identify the system itself.
In contrast to other kinds of module origin, a location cannot beinferred by the loader just by the module name. Instead, the loadermust be provided with a string to identify the location, usually by thefinder that generates the loader. The loader then uses this informationto locate the resource from which it will load the module. In theoryyou could load the module at a given location under various names.
The most common example of locations in the import system are thefiles from which source and extension modules are loaded. For thesemodules the location is identified by the string in the__file__attribute. Although__file__ isn’t particularly accurate for somemodules (e.g. zipped), it is currently the only way that the importsystem indicates that a module has a location.
A module that has a location may be called “locatable”.
The import system stores compiled modules in the __pycache__ directoryas an optimization. This module cache that we use today was provided byPEP 3147. For this proposal, the relevant API for module caching is the__cache__ attribute of modules and the cache_from_source() functionin importlib.util. Loaders are responsible for putting modules into thecache (and loading out of the cache). Currently the cache is only usedfor compiled source modules. However, loaders may take advantage ofthe module cache for other kinds of modules.
The concept does not change, nor does the term. However, thedistinction between modules and packages is mostly superficial.Packagesare modules. They simply have a__path__ attribute andimport may add attributes bound to submodules. The typically perceiveddifference is a source of confusion. This proposal explicitlyde-emphasizes the distinction between packages and modules where itmakes sense to do so.
The import system has evolved over the lifetime of Python. In late 2002PEP 302 introduced standardized import hooks via finders andloaders and sys.meta_path. The importlib module, introducedwith Python 3.1, now exposes a pure Python implementation of the APIsdescribed byPEP 302, as well as of the full import system. It is nowmuch easier to understand and extend the import system. While a benefitto the Python community, this greater accessibility also presents achallenge.
As more developers come to understand and customize the import system,any weaknesses in the finder and loader APIs will be more impactful. Sothe sooner we can address any such weaknesses the import system, thebetter…and there are a couple we hope to take care of with this proposal.
Firstly, any time the import system needs to save information about amodule we end up with more attributes on module objects that aregenerally only meaningful to the import system. It would be nice tohave a per-module namespace in which to put future import-relatedinformation and to pass around within the import system. Secondly,there’s an API void between finders and loaders that causes unduecomplexity when encountered. ThePEP 420 (namespace packages)implementation had to work around this. The complexity surfaced againduring recent efforts on a separate proposal.[1]
Thefinder andloader sections above detail current responsibilityof both. Notably, loaders are not required to provide any of thefunctionality of their load_module() method through other methods. Thus,though the import-related information about a module is likely availablewithout loading the module, it is not otherwise exposed.
Furthermore, the requirements associated with load_module() arecommon to all loaders and mostly are implemented in exactly the sameway. This means every loader has to duplicate the same boilerplatecode. importlib.util provides some tools that help with this, butit would be more helpful if the import system simply took charge ofthese responsibilities. The trouble is that this would limit the degreeof customization that load_module() could easily continue to facilitate.
More importantly, While a findercould provide the information thatthe loader’s load_module() would need, it currently has no consistentway to get it to the loader. This is a gap between finders and loaderswhich this proposal aims to fill.
Finally, when the import system calls a finder’s find_module(), thefinder makes use of a variety of information about the module that isuseful outside the context of the method. Currently the options arelimited for persisting that per-module information past the method call,since it only returns the loader. Popular options for this limitationare to store the information in a module-to-info mapping somewhere onthe finder itself, or store it on the loader.
Unfortunately, loaders are not required to be module-specific. On topof that, some of the useful information finders could provide iscommon to all finders, so ideally the import system could take care ofthose details. This is the same gap as before between finders andloaders.
As an example of complexity attributable to this flaw, theimplementation of namespace packages in Python 3.3 (seePEP 420) addedFileFinder.find_loader() because there was no good way forfind_module() to provide the namespace search locations.
The answer to this gap is a ModuleSpec object that contains theper-module information and takes care of the boilerplate functionalityinvolved with loading the module.
The goal is to address the gap between finders and loaders whilechanging as little of their semantics as possible. Though somefunctionality and information is moved to the new ModuleSpec type,their behavior should remain the same. However, for the sake of claritythe finder and loader semantics will be explicitly identified.
Here is a high-level summary of the changes described by this PEP. Moredetail is available in later sections.
An encapsulation of a module’s import-system-related state during import.See theModuleSpec section below for a more detailed description.
Attributes:
These are ModuleSpec factory functions, meant as a convenience forfinders. See theFactory Functions section below for more detail.
For finders:
For loaders:
For modules:
__spec__.These were introduced prior to Python 3.4’s release, so they can simplybe removed.
__spec__) will nolonger be used directly by the import system during that module’simport. However, this does not impact use of those attributes(e.g.__path__) when loading other modules (e.g. submodules).__repr__() will be a thin wrapper around a purePython implementation which will leverage ModuleSpec.__main__ module will reflect the appropriatename and origin.Here’s a quick breakdown of where responsibilities lie after this PEP.
finders:
loaders:
ModuleSpec:
Immediately? Nothing. The status quo will be deprecated, but willcontinue working. However, here are the things that the authors offinders and loaders should change relative to this PEP:
The ModuleSpec factory functions in importlib.util are intended to behelpful for converting existing finders. spec_from_loader() andspec_from_file_location() are both straightforward utilities in thisregard.
For existing loaders, exec_module() should be a relatively directconversion from the non-boilerplate portion of load_module(). In someuncommon cases the loader should also implement create_module().
ModuleSpec objects have 3 distinct target audiences: Python itself,import hooks, and normal Python users.
Python will use specs in the import machinery, in interpreter startup,and in various standard library modules. Some modules areimport-oriented, like pkgutil, and others are not, like pickle andpydoc. In all cases, the full ModuleSpec API will get used.
Import hooks (finders and loaders) will make use of the spec in specificways. First of all, finders may use the spec factory functions inimportlib.util to create spec objects. They may also directly adjustthe spec attributes after the spec is created. Secondly, the finder maybind additional information to the spec (in finder_extras) for theloader to consume during module creation/execution. Finally, loaderswill make use of the attributes on a spec when creating and/or executinga module.
Python users will be able to inspect a module’s__spec__ to getimport-related information about the object. Generally, Pythonapplications and interactive users will not be using theModuleSpecfactory functions nor any the instance methods.
Here is an outline of what the import machinery does during loading,adjusted to take advantage of the module’s spec and the new loader API:
module=Noneifspec.loaderisnotNoneandhasattr(spec.loader,'create_module'):module=spec.loader.create_module(spec)ifmoduleisNone:module=ModuleType(spec.name)# The import-related module attributes get set here:_init_module_attrs(spec,module)ifspec.loaderisNoneandspec.submodule_search_locationsisnotNone:# Namespace packagesys.modules[spec.name]=moduleelifnothasattr(spec.loader,'exec_module'):spec.loader.load_module(spec.name)# __loader__ and __package__ would be explicitly set here for# backwards-compatibility.else:sys.modules[spec.name]=moduletry:spec.loader.exec_module(module)exceptBaseException:try:delsys.modules[spec.name]exceptKeyError:passraisemodule_to_return=sys.modules[spec.name]
These steps are exactly what Loader.load_module() is alreadyexpected to do. Loaders will thus be simplified since they will onlyneed to implement exec_module().
Note that we must return the module from sys.modules. During loadingthe module may have replaced itself in sys.modules. Since we don’t havea post-import hook API to accommodate the use case, we have to deal withit. However, in the replacement case we do not worry about setting theimport-related module attributes on the object. The module writer is ontheir own if they are doing this.
Here is the corresponding outline for reload():
_RELOADING={}defreload(module):try:name=module.__spec__.nameexceptAttributeError:name=module.__name__spec=find_spec(name,target=module)ifsys.modules.get(name)isnotmodule:raiseImportErrorifspecin_RELOADING:return_RELOADING[name]_RELOADING[name]=moduletry:ifspec.loaderisNone:# Namespace loader_init_module_attrs(spec,module)returnmoduleifspec.parentandspec.parentnotinsys.modules:raiseImportError_init_module_attrs(spec,module)# Ignoring backwards-compatibility call to load_module()# for simplicity.spec.loader.exec_module(module)returnsys.modules[name]finally:del_RELOADING[name]
A key point here is the switch to Loader.exec_module() means thatloaders will no longer have an easy way to know at execution time if itis a reload or not. Before this proposal, they could simply check tosee if the module was already in sys.modules. Now, by the timeexec_module() is called during load (not reload) the import machinerywould already have placed the module in sys.modules. This is part ofthe reason why find_spec() hasthe “target” parameter.
The semantics of reload will remain essentially the same as they existalready[5]. The impact of this PEP on some kindsof lazy loading modules was a point of discussion.[4]
Each of the following names is an attribute on ModuleSpec objects. Avalue of None indicates “not set”. This contrasts with moduleobjects where the attribute simply doesn’t exist. Most of theattributes correspond to the import-related attributes of modules. Hereis the mapping. The reverse of this mapping describes how the importmachinery sets the module attributes right before calling exec_module().
| On ModuleSpec | On Modules |
|---|---|
| name | __name__ |
| loader | __loader__ |
| parent | __package__ |
| origin | __file__* |
| cached | __cached__*,** |
| submodule_search_locations | __path__** |
| loader_state | - |
| has_location | - |
While parent and has_location are read-only properties, the remainingattributes can be replaced after the module spec is created and evenafter import is complete. This allows for unusual cases where directlymodifying the spec is the best option. However, typical use should notinvolve changing the state of a module’s spec.
origin
“origin” is a string for the name of the place from which the moduleoriginates. Seeorigin above. Aside from the informational value,it is also used in the module’s repr. In the case of a spec where“has_location” is true,__file__ is set to the value of “origin”.For built-in modules “origin” would be set to “built-in”.
has_location
As explained in thelocation section above, many modules are“locatable”, meaning there is a corresponding resource from which themodule will be loaded and that resource can be described by a string.In contrast, non-locatable modules can’t be loaded in this fashion, e.g.builtin modules and modules dynamically created in code. For these, thename is the only way to access them, so they have an “origin” but not a“location”.
“has_location” is true if the module is locatable. In that case thespec’s origin is used as the location and__file__ is set tospec.origin. If additional location information is required (e.g.zipimport), that information may be stored in spec.loader_state.
“has_location” may be implied from the existence of a load_data() methodon the loader.
Incidentally, not all locatable modules will be cache-able, but mostwill.
submodule_search_locations
The list of location strings, typically directory paths, in which tosearch for submodules. If the module is a package this will be set toa list (even an empty one). Otherwise it is None.
The name of the corresponding module attribute,__path__, isrelatively ambiguous. Instead of mirroring it, we use a more explicitattribute name that makes the purpose clear.
loader_state
A finder may set loader_state to any value to provide additionaldata for the loader to use during loading. A value of None is thedefault and indicates that there is no additional data. Otherwise itcan be set to any object, such as a dict, list, ortypes.SimpleNamespace, containing the relevant extra information.
For example, zipimporter could use it to pass the zip archive nameto the loader directly, rather than needing to derive it from originor create a custom loader for each find operation.
loader_state is meant for use by the finder and corresponding loader.It is not guaranteed to be a stable resource for any other use.
spec_from_file_location(name, location, *, loader=None, submodule_search_locations=None)
Build a spec from file-oriented information and loader APIs.
spec_from_loader(name, loader, *, origin=None, is_package=None)
Build a spec with missing information filled in by using loader APIs.
ModuleSpec doesn’t have any. This would be a different story ifFinder.find_module() were to return a module spec instead of loader.In that case, specs would have to act like the loader that would havebeen returned instead. Doing so would be relatively simple, but is anunnecessary complication. It was part of earlier versions of this PEP.
Subclasses of ModuleSpec are allowed, but should not be necessary.Simply setting loader_state or adding functionality to a customfinder or loader will likely be a better fit and should be tried first.However, as long as a subclass still fulfills the requirements of theimport system, objects of that type are completely fine as the returnvalue of Finder.find_spec(). The same points apply to duck-typing.
Other than adding__spec__, none of the import-related moduleattributes will be changed or deprecated, though some of them could be;any such deprecation can wait until Python 4.
A module’s spec will not be kept in sync with the correspondingimport-related attributes. Though they may differ, in practice theywill typically be the same.
One notable exception is that case where a module is run as a script byusing the-m flag. In that casemodule.__spec__.name willreflect the actual module name whilemodule.__name__ will be__main__.
A module’s spec is not guaranteed to be identical between two moduleswith the same name. Likewise there is no guarantee that successivecalls to importlib.find_spec() will return the same object or even anequivalent object, though at least the latter is likely.
Finders are still responsible for identifying, and typically creating,the loader that should be used to load a module. That loader willnow be stored in the module spec returned by find_spec() ratherthan returned directly. As is currently the case without the PEP, if aloader would be costly to create, that loader can be designed to deferthe cost until later.
MetaPathFinder.find_spec(name, path=None, target=None)
PathEntryFinder.find_spec(name, target=None)
Finders must return ModuleSpec objects when find_spec() iscalled. This new method replaces find_module() andfind_loader() (in the PathEntryFinder case). If a loader doesnot have find_spec(), find_module() and find_loader() areused instead, for backward-compatibility.
Adding yet another similar method to loaders is a case of practicality.find_module() could be changed to return specs instead of loaders.This is tempting because the import APIs have suffered enough,especially considering PathEntryFinder.find_loader() was justadded in Python 3.3. However, the extra complexity and aless-than-explicit method name aren’t worth it.
A call to find_spec() may optionally include a “target” argument. Thisis the module object that will be used subsequently as the target ofloading. During normal import (and by default) “target” is None,meaning the target module has yet to be created. During reloading themodule passed in to reload() is passed through to find_spec() as thetarget. This argument allows the finder to build the module spec withmore information than is otherwise available. Doing so is particularlyrelevant in identifying the loader to use.
Through find_spec() the finder will always identify the loader itwill return in the spec (or return None). At the point the loader isidentified, the finder should also decide whether or not the loadersupports loading into the target module, in the case that “target” ispassed in. This decision may entail consulting with the loader.
If the finder determines that the loader does not support loading intothe target module, it should either find another loader or raiseImportError (completely stopping import of the module). Thisdetermination is especially important during reload since, as noted inHow Reloading Will Work, loaders will no longer be able to triviallyidentify a reload situation on their own.
Two alternatives were presented to the “target” parameter:Loader.supports_reload() and adding “target” to Loader.exec_module()instead of find_spec(). supports_reload() was the initial approach tothe reload situation.[6] However, there was someopposition to the loader-specific, reload-centric approach.[7]
As to “target” on exec_module(), the loader may need other informationfrom the target module (or spec) during reload, more than just “doesthis loader support reloading this module”, that is no longer availablewith the move away from load_module(). A proposal on the table was toadd something like “target” to exec_module().[8]However, putting “target” on find_spec() instead is more in line withthe goals of this PEP. Furthermore, it obviates the need forsupports_reload().
Currently a path entry finder may return (None, portions) fromfind_loader() to indicate it found part of a possible namespacepackage. To achieve the same effect, find_spec() must return a specwith “loader” set to None (a.k.a. not set) and withsubmodule_search_locations set to the same portions as would have beenprovided by find_loader(). It’s up to PathFinder how to handle suchspecs.
Loader.exec_module(module)
Loaders will have a new method, exec_module(). Its only jobis to “exec” the module and consequently populate the module’snamespace. It is not responsible for creating or preparing the moduleobject, nor for any cleanup afterward. It has no return value.exec_module() will be used during both loading and reloading.
exec_module() should properly handle the case where it is called morethan once. For some kinds of modules this may mean raising ImportErrorevery time after the first time the method is called. This isparticularly relevant for reloading, where some kinds of modules do notsupport in-place reloading.
Loader.create_module(spec)
Loaders may also implement create_module() that will return anew module to exec. It may return None to indicate that the defaultmodule creation code should be used. One use case, though atypical, forcreate_module() is to provide a module that is a subclass of the builtinmodule type. Most loaders will not need to implement create_module(),
create_module() should properly handle the case where it is called morethan once for the same spec/module. This may include returning None orraising ImportError.
Note
exec_module() and create_module() should not set any import-relatedmodule attributes. The fact that load_module() does is a design flawthat this proposal aims to correct.
Other changes:
PEP 420 introduced the optional module_repr() loader method to limitthe amount of special-casing in the module type’s__repr__(). Sincethis method is part of ModuleSpec, it will be deprecated on loaders.However, if it exists on a loader it will be used exclusively.
Loader.init_module_attr() method, added prior to Python 3.4’srelease, will be removed in favor of the same method on ModuleSpec.
However, InspectLoader.is_package() will not be deprecated eventhough the same information is found on ModuleSpec. ModuleSpeccan use it to populate its own is_package if that information isnot otherwise available. Still, it will be made optional.
In addition to executing a module during loading, loaders will still bedirectly responsible for providing APIs concerning module-related data.
__main__ module will reflect how the interpreterwas started. For instance, with-m the spec’s name will be thatof the module used, while__main__.__name__ will still be“__main__”.A reference implementation is available athttp://bugs.python.org/issue18864.
* The implementation of this PEP needs to be cognizant of its impact onpkgutil (and setuptools). pkgutil has some generic function-basedextensions toPEP 302 which may break if importlib starts wrappingloaders without the tools’ knowledge.
* Other modules to look at: runpy (and pythonrun.c), pickle, pydoc,inspect.
For instance, pickle should be updated in the__main__ case to lookatmodule.__spec__.name.
There were a few proposed additions to this proposal that did not fitwell enough into its scope.
There is no “PathModuleSpec” subclass of ModuleSpec that separates outhas_location, cached, and submodule_search_locations. While that mightmake the separation cleaner, module objects don’t have that distinction.ModuleSpec will support both cases equally well.
While “ModuleSpec.is_package” would be a simple additional attribute(aliasing self.submodule_search_locations is not None), it perpetuatesthe artificial (and mostly erroneous) distinction between modules andpackages.
The module specFactory Functions could be classmethods onModuleSpec. However that would expose them onall modules via__spec__, which has the potential to unnecessarily confusenon-advanced Python users. The factory functions have a specific usecase, to support finder authors. SeeModuleSpec Users.
Likewise, several other methods could be added to ModuleSpec that exposethe specific uses of module specs by the import machinery:
As with the factory functions, exposing these methods viamodule.__spec__ is less than desirable. They would end up being anattractive nuisance, even if only exposed as “private” attributes (asthey were in previous versions of this PEP). If someone finds a needfor these methods later, we can expose the via an appropriate API(separate from ModuleSpec) at that point, perhaps relative toPEP 406(import engine).
Conceivably, the load() method could optionally take a list ofmodules with which to interact instead of sys.modules. Also, load()could be leveraged to implement multi-version imports. Both areinteresting ideas, but definitely outside the scope of this proposal.
Others left out:
This document has been placed in the public domain.
Source:https://github.com/python/peps/blob/main/peps/pep-0451.rst
Last modified:2025-02-01 08:59:27 GMT