5.2.1.Regular packages¶

Python defines two types of packages,regular packages andnamespace packages. Regularpackages are traditional packages as they existed in Python 3.2 and earlier.A regular package is typically implemented as a directory containing an__init__.py file. When a regular package is imported, this__init__.py file is implicitly executed, and the objects it defines arebound to names in the package’s namespace. The__init__.py file cancontain the same Python code that any other module can contain, and Pythonwill add some additional attributes to the module when it is imported.

For example, the following file system layout defines a top levelparentpackage with three subpackages:

parent/__init__.pyone/__init__.pytwo/__init__.pythree/__init__.py

Importingparent.one will implicitly executeparent/__init__.py andparent/one/__init__.py. Subsequent imports ofparent.two orparent.three will executeparent/two/__init__.py andparent/three/__init__.py respectively.

5.2.2.Namespace packages¶

A namespace package is a composite of variousportions,where each portion contributes a subpackage to the parent package. Portionsmay reside in different locations on the file system. Portions may also befound in zip files, on the network, or anywhere else that Python searchesduring import. Namespace packages may or may not correspond directly toobjects on the file system; they may be virtual modules that have no concreterepresentation.

Namespace packages do not use an ordinary list for their__path__attribute. They instead use a custom iterable type which will automaticallyperform a new search for package portions on the next import attempt withinthat package if the path of their parent package (orsys.path for atop level package) changes.

With namespace packages, there is noparent/__init__.py file. In fact,there may be multipleparent directories found during import search, whereeach one is provided by a different portion. Thusparent/one may not bephysically located next toparent/two. In this case, Python will create anamespace package for the top-levelparent package whenever it or one ofits subpackages is imported.

See alsoPEP 420 for the namespace package specification.

5.3.1.The module cache¶

The first place checked during import search issys.modules. Thismapping serves as a cache of all modules that have been previously imported,including the intermediate paths. So iffoo.bar.baz was previouslyimported,sys.modules will contain entries forfoo,foo.bar,andfoo.bar.baz. Each key will have as its value the corresponding moduleobject.

During import, the module name is looked up insys.modules and ifpresent, the associated value is the module satisfying the import, and theprocess completes. However, if the value isNone, then aModuleNotFoundError is raised. If the module name is missing, Python willcontinue searching for the module.

sys.modules is writable. Deleting a key may not destroy theassociated module (as other modules may hold references to it),but it will invalidate the cache entry for the named module, causingPython to search anew for the named module upon its nextimport. The key can also be assigned toNone, forcing the next importof the module to result in aModuleNotFoundError.

Beware though, as if you keep a reference to the module object,invalidate its cache entry insys.modules, and then re-import thenamed module, the two module objects willnot be the same. By contrast,importlib.reload() will reuse thesame module object, and simplyreinitialise the module contents by rerunning the module’s code.

5.3.2.Finders and loaders¶

If the named module is not found insys.modules, then Python’s importprotocol is invoked to find and load the module. This protocol consists oftwo conceptual objects,finders andloaders.A finder’s job is to determine whether it can find the named module usingwhatever strategy it knows about. Objects that implement both of theseinterfaces are referred to asimporters - they returnthemselves when they find that they can load the requested module.

Python includes a number of default finders and importers. The first oneknows how to locate built-in modules, and the second knows how to locatefrozen modules. A third default finder searches animport pathfor modules. Theimport path is a list of locations that mayname file system paths or zip files. It can also be extended to searchfor any locatable resource, such as those identified by URLs.

The import machinery is extensible, so new finders can be added to extend therange and scope of module searching.

Finders do not actually load modules. If they can find the named module, theyreturn amodule spec, an encapsulation of the module’s import-relatedinformation, which the import machinery then uses when loading the module.

The following sections describe the protocol for finders and loaders in moredetail, including how you can create and register new ones to extend theimport machinery.

5.3.3.Import hooks¶

The import machinery is designed to be extensible; the primary mechanism forthis are theimport hooks. There are two types of import hooks:metahooks andimport path hooks.

Meta hooks are called at the start of import processing, before any otherimport processing has occurred, other thansys.modules cache look up.This allows meta hooks to overridesys.path processing, frozenmodules, or even built-in modules. Meta hooks are registered by adding newfinder objects tosys.meta_path, as described below.

Import path hooks are called as part ofsys.path (orpackage.__path__) processing, at the point where their associated pathitem is encountered. Import path hooks are registered by adding new callablestosys.path_hooks as described below.

5.3.4.The meta path¶

When the named module is not found insys.modules, Python nextsearchessys.meta_path, which contains a list of meta path finderobjects. These finders are queried in order to see if they know how to handlethe named module. Meta path finders must implement a method calledfind_spec() which takes three arguments:a name, an import path, and (optionally) a target module. The meta pathfinder can use any strategy it wants to determine whether it can handlethe named module or not.

If the meta path finder knows how to handle the named module, it returns aspec object. If it cannot handle the named module, it returnsNone. Ifsys.meta_path processing reaches the end of its list without returninga spec, then aModuleNotFoundError is raised. Any other exceptionsraised are simply propagated up, aborting the import process.

Thefind_spec() method of meta pathfinders is called with two or three arguments. The first is the fullyqualified name of the module being imported, for examplefoo.bar.baz.The second argument is the path entries to use for the module search. Fortop-level modules, the second argument isNone, but for submodules orsubpackages, the second argument is the value of the parent package’s__path__ attribute. If the appropriate__path__ attribute cannotbe accessed, aModuleNotFoundError is raised. The third argumentis an existing module object that will be the target of loading later.The import system passes in a target module only during reload.

The meta path may be traversed multiple times for a single import request.For example, assuming none of the modules involved has already been cached,importingfoo.bar.baz will first perform a top level import, callingmpf.find_spec("foo",None,None) on each meta path finder (mpf). Afterfoo has been imported,foo.bar will be imported by traversing themeta path a second time, callingmpf.find_spec("foo.bar",foo.__path__,None). Oncefoo.bar has beenimported, the final traversal will callmpf.find_spec("foo.bar.baz",foo.bar.__path__,None).

Some meta path finders only support top level imports. These importers willalways returnNone when anything other thanNone is passed as thesecond argument.

Python’s defaultsys.meta_path has three meta path finders, one thatknows how to import built-in modules, one that knows how to import frozenmodules, and one that knows how to import modules from animport path(i.e. thepath based finder).

5.4.1.Loaders¶

Module loaders provide the critical function of loading: module execution.The import machinery calls theimportlib.abc.Loader.exec_module()method with a single argument, the module object to execute. Any valuereturned fromexec_module() is ignored.

Loaders must satisfy the following requirements:

• If the module is a Python module (as opposed to a built-in module or adynamically loaded extension), the loader should execute the module’s codein the module’s global name space (module.__dict__).

• If the loader cannot execute the module, it should raise anImportError, although any other exception raised duringexec_module() will be propagated.

In many cases, the finder and loader can be the same object; in such cases thefind_spec() method would just return aspec with the loader set toself.

Module loaders may opt in to creating the module object during loadingby implementing acreate_module() method.It takes one argument, the module spec, and returns the new module objectto use during loading.create_module() does not need to set any attributeson the module object. If the method returnsNone, theimport machinery will create the new module itself.

5.4.2.Submodules¶

When a submodule is loaded using any mechanism (e.g.importlib APIs, theimport orimport-from statements, or built-in__import__()) abinding is placed in the parent module’s namespace to the submodule object.For example, if packagespam has a submodulefoo, after importingspam.foo,spam will have an attributefoo which is bound to thesubmodule. Let’s say you have the following directory structure:

spam/__init__.pyfoo.py

andspam/__init__.py has the following line in it:

from.fooimportFoo

then executing the following puts name bindings forfoo andFoo in thespam module:

>>>importspam>>>spam.foo<module 'spam.foo' from '/tmp/imports/spam/foo.py'>>>>spam.Foo<class 'spam.foo.Foo'>

Given Python’s familiar name binding rules this might seem surprising, butit’s actually a fundamental feature of the import system. The invariantholding is that if you havesys.modules['spam'] andsys.modules['spam.foo'] (as you would after the above import), the lattermust appear as thefoo attribute of the former.

5.4.3.Module specs¶

The import machinery uses a variety of information about each moduleduring import, especially before loading. Most of the information iscommon to all modules. The purpose of a module’s spec is to encapsulatethis import-related information on a per-module basis.

Using a spec during import allows state to be transferred between importsystem components, e.g. between the finder that creates the module specand the loader that executes it. Most importantly, it allows theimport machinery to perform the boilerplate operations of loading,whereas without a module spec the loader had that responsibility.

The module’s spec is exposed asmodule.__spec__. Setting__spec__ appropriately applies equally tomodules initialized during interpreter startup.The one exception is__main__, where__spec__ isset to None in some cases.

SeeModuleSpec for details on the contents ofthe module spec.

5.4.4.__path__ attributes on modules¶

The__path__ attribute should be a (possibly empty)sequence of strings enumerating the locations where the package’ssubmodules will be found. By definition, if a module has a__path__attribute, it is apackage.

A package’s__path__ attribute is used during imports of itssubpackages.Within the import machinery, it functions much the same assys.path,i.e. providing a list of locations to search for modules during import.However,__path__ is typically much more constrained thansys.path.

The same rules used forsys.path also apply to a package’s__path__.sys.path_hooks (described below) areconsulted when traversing a package’s__path__.

A package’s__init__.py file may set or alter the package’s__path__attribute, and this was typically the way namespace packages were implementedprior toPEP 420. With the adoption ofPEP 420, namespace packages nolonger need to supply__init__.py files containing only__path__manipulation code; the import machinery automatically sets__path__correctly for the namespace package.

5.4.5.Module reprs¶

By default, all modules have a usable repr, however depending on theattributes set above, and in the module’s spec, you can more explicitlycontrol the repr of module objects.

If the module has a spec (__spec__), the import machinery will tryto generate a repr from it. If that fails or there is no spec, the importsystem will craft a default repr using whatever information is availableon the module. It will try to use themodule.__name__,module.__file__, andmodule.__loader__ as input into the repr,with defaults for whatever information is missing.

Here are the exact rules used:

• If the module has a__spec__ attribute, the information in the specis used to generate the repr. The “name”, “loader”, “origin”, and“has_location” attributes are consulted.

• If the module has a__file__ attribute, this is used as part of themodule’s repr.

• If the module has no__file__ but does have a__loader__ that is notNone, then the loader’s repr is used as part of the module’s repr.

• Otherwise, just use the module’s__name__ in the repr.

5.4.6.Cached bytecode invalidation¶

Before Python loads cached bytecode from a.pyc file, it checks whether thecache is up-to-date with the source.py file. By default, Python does thisby storing the source’s last-modified timestamp and size in the cache file whenwriting it. At runtime, the import system then validates the cache file bychecking the stored metadata in the cache file against the source’smetadata.

Python also supports “hash-based” cache files, which store a hash of the sourcefile’s contents rather than its metadata. There are two variants of hash-based.pyc files: checked and unchecked. For checked hash-based.pyc files,Python validates the cache file by hashing the source file and comparing theresulting hash with the hash in the cache file. If a checked hash-based cachefile is found to be invalid, Python regenerates it and writes a new checkedhash-based cache file. For unchecked hash-based.pyc files, Python simplyassumes the cache file is valid if it exists. Hash-based.pyc filesvalidation behavior may be overridden with the--check-hash-based-pycsflag.

5.5.1.Path entry finders¶

Thepath based finder is responsible for finding and loadingPython modules and packages whose location is specified with a stringpath entry. Most path entries name locations in the file system,but they need not be limited to this.

As a meta path finder, thepath based finder implements thefind_spec() protocol previouslydescribed, however it exposes additional hooks that can be used tocustomize how modules are found and loaded from theimport path.

Three variables are used by thepath based finder,sys.path,sys.path_hooks andsys.path_importer_cache. The__path__attributes on package objects are also used. These provide additional waysthat the import machinery can be customized.

sys.path contains a list of strings providing search locations formodules and packages. It is initialized from thePYTHONPATHenvironment variable and various other installation- andimplementation-specific defaults. Entries insys.path can namedirectories on the file system, zip files, and potentially other “locations”(see thesite module) that should be searched for modules, such asURLs, or database queries. Only strings should be present onsys.path; all other data types are ignored.

Thepath based finder is ameta path finder, so the importmachinery begins theimport path search by calling the pathbased finder’sfind_spec() method asdescribed previously. When thepath argument tofind_spec() is given, it will be alist of string paths to traverse - typically a package’s__path__attribute for an import within that package. If thepath argument isNone, this indicates a top level import andsys.path is used.

The path based finder iterates over every entry in the search path, andfor each of these, looks for an appropriatepath entry finder(PathEntryFinder) for thepath entry. Because this can be an expensive operation (e.g. there may bestat() call overheads for this search), the path based finder maintainsa cache mapping path entries to path entry finders. This cache is maintainedinsys.path_importer_cache (despite the name, this cache actuallystores finder objects rather than being limited toimporter objects).In this way, the expensive search for a particularpath entrylocation’spath entry finder need only be done once. User code isfree to remove cache entries fromsys.path_importer_cache forcingthe path based finder to perform the path entry search again.

If the path entry is not present in the cache, the path based finder iteratesover every callable insys.path_hooks. Each of thepath entryhooks in this list is called with a single argument, thepath entry to be searched. This callable may either return apathentry finder that can handle the path entry, or it may raiseImportError. AnImportError is used by the path based finder tosignal that the hook cannot find apath entry finderfor thatpath entry. Theexception is ignored andimport path iteration continues. The hookshould expect either a string or bytes object; the encoding of bytes objectsis up to the hook (e.g. it may be a file system encoding, UTF-8, or somethingelse), and if the hook cannot decode the argument, it should raiseImportError.

Ifsys.path_hooks iteration ends with nopath entry finderbeing returned, then the path based finder’sfind_spec() method will storeNoneinsys.path_importer_cache (to indicate that there is no finder forthis path entry) and returnNone, indicating that thismeta path finder could not find the module.

If apath entry finderis returned by one of thepath entryhook callables onsys.path_hooks, then the following protocol is usedto ask the finder for a module spec, which is then used when loading themodule.

The current working directory – denoted by an empty string – is handledslightly differently from other entries onsys.path. First, if thecurrent working directory is found to not exist, no value is stored insys.path_importer_cache. Second, the value for the current workingdirectory is looked up fresh for each module lookup. Third, the path used forsys.path_importer_cache and returned byimportlib.machinery.PathFinder.find_spec() will be the actual currentworking directory and not the empty string.

5.5.2.Path entry finder protocol¶

In order to support imports of modules and initialized packages and also tocontribute portions to namespace packages, path entry finders must implementthefind_spec() method.

find_spec() takes two arguments: thefully qualified name of the module being imported, and the (optional) targetmodule.find_spec() returns a fully populated spec for the module.This spec will always have “loader” set (with one exception).

To indicate to the import machinery that the spec represents a namespaceportion, the path entry finder setssubmodule_search_locations toa list containing the portion.

5.8.1.__main__.__spec__¶

Depending on how__main__ is initialized,__main__.__spec__gets set appropriately or toNone.

When Python is started with the-m option,__spec__ is setto the module spec of the corresponding module or package.__spec__ isalso populated when the__main__ module is loaded as part of executing adirectory, zipfile or othersys.path entry.

Inthe remaining cases__main__.__spec__ is set toNone, as the code used to populate the__main__ does not correspond directly with an importable module:

• interactive prompt

• -c option

• running from stdin

• running directly from a source or bytecode file

Note that__main__.__spec__ is alwaysNone in the last case,even if the file could technically be imported directly as a moduleinstead. Use the-m switch if valid module metadata is desiredin__main__.

Note also that even when__main__ corresponds with an importable moduleand__main__.__spec__ is set accordingly, they’re still considereddistinct modules. This is due to the fact that blocks guarded byif__name__=="__main__": checks only execute when the module is usedto populate the__main__ namespace, and not during normal import.