CPython

When one of Python’s standard library modules (such as_sqlite3) cannot becompiled during a CPython build because of missing dependencies (e.g. SQLiteheader files), the module is simply skipped. If you then install this compiledPython and use it to try to import one of the missing modules, Python will failwith aModuleNotFoundError.

For example, after deliberately removingsqlite-devel from the localsystem:

$ ./python -c "import sqlite3"Traceback (most recent call last):  File "<string>", line 1, in <module>  File "/home/ncoghlan/devel/cpython/Lib/sqlite3/__init__.py", line 23, in <module>    from sqlite3.dbapi2 import *  File "/home/ncoghlan/devel/cpython/Lib/sqlite3/dbapi2.py", line 27, in <module>    from _sqlite3 import *ModuleNotFoundError: No module named '_sqlite3'

This can confuse users who may not understand why a cleanly built Python ismissing standard library modules.

Linux and other distributions

Many Linux and other distributions are already separating out parts of thestandard library to standalone packages. Among the most commonly excludedmodules are thetkinter module, since it draws in a dependency on thegraphical environment,idlelib, since it depends ontkinter (and mostLinux desktop environments provide their own default code editor), and thetest package, as it only serves to test Python internally and is about asbig as the rest of the standard library put together.

The methods of omission of these modules differ. For example, Debian patchesthe fileLib/tkinter/__init__.py to envelop the lineimport_tkinter inatry-except block and upon encountering anImportError it simply addsthe following to the error message:pleaseinstallthepython3-tkpackage[1]. Fedora and other distributions simply don’t include theomitted modules, potentially leaving users baffled as to where to find them.

An example from Fedora 29:

$ python3 -c "import tkinter"Traceback (most recent call last):  File "<string>", line 1, in <module>ModuleNotFoundError: No module named 'tkinter'

APIs to list expected standard library modules

To allow for easier identification of which module names areexpected to beresolved in the standard library, thesysconfig module will be extendedwith two additional functions:

• sysconfig.get_stdlib_modules(), which will provide a list of the names ofall top level Python standard library modules (including private modules)
• sysconfig.get_optional_modules(), which will list optional public top levelstandard library module names

The results ofsysconfig.get_optional_modules() and the existingsys.builtin_module_names will both be subsets of the full list provided bythe newsysconfig.get_stdlib_modules() function.

These added lists will be generated during the Python build process and saved inthe_sysconfigdata-*.py file along with othersysconfig values.

Possible reasons for modules being in the “optional” list will be:

• the module relies on an optional build dependency (e.g._sqlite3,tkinter,idlelib)
• the module is private for other reasons and hence may not be present on allimplementations (e.g._freeze_importlib,_collections_abc)
• the module is platform specific and hence may not be present in allinstallations (e.g.winreg)
• thetest package may also be freely omitted from Python runtimeinstallations, as it is intended for use in testing Python implementations,not as a runtime library for Python projects to use (the public API offeringtesting utilities isunittest)

(Note: theensurepip,venv, anddistutils modules are all consideredmandatory modules in this PEP, even though not all redistributors currentlyadhere to that practice)

Changes to the defaultsys.excepthook implementation

The default implementation of thesys.excepthook function will then bemodified to dispense an appropriate message when it detects a failure toimport a module identified by one of the two newsysconfig functions asbelonging to the Python standard library.

Revised error message for a module that relies on an optional build dependencyor is otherwise considered optional when Python is installed:

$ ./python -c "import sqlite3"Traceback (most recent call last):  File "<string>", line 1, in <module>  File "/home/ncoghlan/devel/cpython/Lib/sqlite3/__init__.py", line 23, in <module>    from sqlite3.dbapi2 import *  File "/home/ncoghlan/devel/cpython/Lib/sqlite3/dbapi2.py", line 27, in <module>    from _sqlite3 import *ModuleNotFoundError: Optional standard library module '_sqlite3' was not found

Revised error message for a submodule of an optional top level package when theentire top level package is missing:

$ ./python -c "import test.regrtest"Traceback (most recent call last):  File "<string>", line 1, in <module>ModuleNotFoundError: Optional standard library module 'test' was not found

Revised error message for a submodule of an optional top level package when thetop level package is present:

$ ./python -c "import test.regrtest"Traceback (most recent call last):  File "<string>", line 1, in <module>ModuleNotFoundError: No submodule named 'test.regrtest' in optional standard library module 'test'

Revised error message for a module that is always expected to be available:

$ ./python -c "import ensurepip"Traceback (most recent call last):  File "<string>", line 1, in <module>ModuleNotFoundError: Standard library module 'ensurepip' was not found

Revised error message for a missing submodule of a standard library package whenthe top level package is present:

$ ./python -c "import encodings.mbcs"Traceback (most recent call last):  File "<string>", line 1, in <module>ModuleNotFoundError: No submodule named 'encodings.mbcs' in standard library module 'encodings'

These revised error messages make it clear that the missing modules are expectedto be available from the standard library, but are not available for some reason,rather than being an indicator of a missing third party dependency in the currentenvironment.

Modifyingsys.excepthook

Thesys.excepthook function gets called when a raised exception is uncaughtand the program is about to exit or (in an interactive session) the control isbeing returned to the prompt. This makes it a perfect place for customizederror messages, as it will not influence caught errors and thus not slow downnormal execution of Python scripts.

Public API to query expected standard library module names

The inclusion of the functionssysconfig.get_stdlib_modules() andsysconfig.get_optional_modules() will provide a long sought-afterway of easily listing the names of Python standard library modules[2], which will (among other benefits) make it easier forcode analysis, profiling, and error reporting tools to offer runtime--ignore-stdlib flags.

Only including top level module names

This PEP proposes that only top level module and package names be reported bythe new query APIs. This is sufficient information to generate the proposederror messages, reduces the number of required entries by an order of magnitude,and simplifies the process of generating the related metadata during the buildprocess.

If this is eventually found to be overly limiting, a newinclude_submodulesflag could be added to the query APIs. However, this isnot part of the initialproposal, as the benefits of doing so aren’t currently seen as justifying theextra complexity.

There is one known consequence of this restriction, which is that the newdefaultexcepthook implementation will report incorrect submodules names thesame way that it reports genuinely missing standard library submodules:

$ ./python -c "import unittest.muck"Traceback (most recent call last):  File "<string>", line 1, in <module>ModuleNotFoundError: No submodule named 'unittest.muck' in standard library module 'unittest'

Listing private top level module names as optional standard library modules

Many of the modules that have an optional external build dependency are writtenas hybrid modules, where there is a shared Python wrapper around animplementation dependent interface to the underlying external library. In othercases, a private top level module may simply be a CPython implementation detail,and other implementations may not provide that module at all.

To report import errors involving these modules appropriately, the new defaultexcepthook implementation needs them to be reported by the new query APIs.

Deeming packaging related modules to be mandatory

Some redistributors aren’t entirely keen on installing the Python specificpackaging related modules (distutils,ensurepip,venv) by default,preferring that developers use their platform specific tooling instead.

This approach causes interoperability problems for developers working oncross-platform projects and educators attempting to write platform independentsetup instructions, so this PEP takes the view that these modules should beconsidered mandatory, and left out of the list of optional modules.

Platform dependent modules

Some standard library modules may be missing because they’re only provided onparticular platforms. For example, thewinreg module is only available onWindows:

$ python3 -c "import winreg"Traceback (most recent call last):  File "<string>", line 1, in <module>ModuleNotFoundError: No module named 'winreg'

In the current proposal, these platform dependent modules will simply beincluded with all the other optional modules rather than attempting to exposethe platform dependency information in a more structured way.

However, the platform dependence is at least tracked at the level of “Windows”,“Unix”, “Linux”, and “FreeBSD” for the benefit ofthe documentation, so itseems plausible that it could potentially be exposed programmatically as well.

Emitting a warning when__main__ shadows a standard library module

Given the new query APIs, the new defaultexcepthook implementation couldpotentially detect when__main__.__file__ or__main__.__spec__.namematch a standard library module, and emit a suitable warning.

However, actually doing anything along this lines should review more cases whereuses actually encounter this problem, and the various options for potentiallyoffering more information to assist in debugging the situation, rather thanneeding to be incorporated right now.