Abstract

PEP 3108 describes the reorganization of the Python standard libraryfor the Python 3.0 release. This PEP describes amechanism for transitioning from the Python 2.x standard library tothe Python 3.0 standard library. This transition will allow andencourage Python programmers to use the new Python 3.0 library namesstarting with Python 2.6, while maintaining the old names for backwardcompatibility. In this way, a Python programmer will be able to writeforward compatible code without sacrificing interoperability withexisting Python programs.

Rationale

PEP 3108 presents a rationale for Python standard library (stdlib)reorganization. The reader is encouraged to consult that PEP fordetails about why and how the library will be reorganized. ShouldPEP 3108 be accepted in part or in whole, then it is advantageous toallow Python programmers to begin the transition to the new stdlibmodule names in Python 2.x, so that they can write forward compatiblecode starting with Python 2.6.

Note thatPEP 3108 proposes to remove some “silly old stuff”,i.e. modules that are no longer useful or necessary. The PEP you arereading does not address this because there are no forwardcompatibility issues for modules that are to be removed, except tostop using such modules.

This PEP concerns only the mechanism by which mappings from old stdlibnames to new stdlib names are maintained. Please consultPEP 3108 forall specific module renaming proposals. Specifically see the sectiontitledModulestoRename for guidelines on the old name to newname mappings. The few examples in this PEP are given forillustrative purposes only and should not be used for specificrenaming recommendations.

Supported Renamings

There are at least 4 use cases explicitly supported by this PEP:

• Simple top-level package name renamings, such asStringIO tostringio;
• Sub-package renamings where the package name may or may not berenamed, such asemail.MIMEText toemail.mime.text;
• Extension module renaming, such ascStringIO tocstringio;
• Third party renaming of any of the above.

Two use cases supported by this PEP include renaming simple top-levelmodules, such asStringIO, as well as modules within packages,such asemail.MIMEText.

In the former case,PEP 3108 currently recommendsStringIO berenamed tostringio, followingPEP 8 recommendations.

In the latter case, the email 4.0 package distributed with Python 2.5already renamedemail.MIMEText toemail.mime.text, although itdid so in a one-off, uniquely hackish way inside the email package.The mechanism described in this PEP is general enough to handle allmodule renamings, obviating the need for the Python 2.5 hack (exceptfor backward compatibility with earlier Python versions).

An additional use case is to support the renaming of C extensionmodules. As long as the new name for the C module is importable, itcan be remapped to the new name. E.g.cStringIO renamed tocstringio.

Third party package renaming is also supported, via several publicinterfaces accessible by any Python module.

Remappings are not performed recursively.

.mv files

Remapping files are called.mv files; the suffix was chosen to beevocative of the Unix mv(1) command. An.mv file is a simpleline-oriented text file. All blank lines and lines that start with a# are ignored. All other lines must contain two whitespace separatedfields. The first field is the old module name, and the second fieldis the new module name. Both module names must be specified usingtheir full dotted-path names. Here is an example.mv file fromPython 2.6:

# Map the various string i/o libraries to their new namesStringIOstringiocStringIOcstringio

.mv files can appear anywhere in the file system, and there is aprogrammatic interface provided to parse them, and register theremappings inside them. By default, when Python starts up, all the.mv files in theoldlib package are read, and their remappingsare automatically registered. This is where all the module remappingsshould be specified for top-level Python 2.x standard library modules.

Implementation Specification

This section provides the full specification for how module renamingsin Python 2.x are implemented. The central mechanism relies onvarious import hooks as described inPEP 302. Specificallysys.path_importer_cache,sys.path, andsys.meta_path areall employed to provide the necessary functionality.

When Python’s import machinery is initialized, the oldlib package isimported. Inside oldlib there is a class calledOldStdlibLoader.This class implements thePEP 302 interface and is automaticallyinstantiated, with zero arguments. The constructor reads all the.mv files from the oldlib package directory, automaticallyregistering all the remappings found in those.mv files. This ishow the Python 2.x standard library is remapped.

The OldStdlibLoader class should not be instantiated by other Pythonmodules. Instead, you can access the global OldStdlibLoader instancevia thesys.stdlib_remapper instance. Use this instance if you wantprogrammatic access to the remapping machinery.

One important implementation detail: as needed by thePEP 302 API, amagic string is added to sys.path, and module __path__ attributes inorder to hook in our remapping loader. This magic string is currently<oldlib> and some changes were necessary to Python’s site.py filein order to treat all sys.path entries starting with< asspecial. Specifically, no attempt is made to make them absolute filenames (since they aren’t file names at all).

In order for the remapping import hooks to work, the module or packagemust be physically located under its new name. This is because theimport hooks catch only modules that are not already imported, andcannot be imported by Python’s built-in import rules. Thus, if amodule has been moved, say from Lib/StringIO.py to Lib/stringio.py,and the former’s.pyc file has been removed, then without theremapper, this would fail:

importStringIO

Instead, with the remapper, this failing import will be caught, theold name will be looked up in the registered remappings, and in thiscase, the new namestringio will be found. The remapper thenattempts to import the new name, and if that succeeds, it binds theresulting module into sys.modules, under both the old and new names.Thus, the above import will result in entries in sys.modules for‘StringIO’ and ‘stringio’, and both will point to the exact samemodule object.

Note that no way to disable the remapping machinery is proposed, shortof moving all the.mv files away or programmatically removing themin some custom start up code. In Python 3.0, the remappings will beeliminated, leaving only the “new” names.

Programmatic Interface

Several methods are added to thesys.stdlib_remapper object, whichthird party packages can use to register their own remappings. Notehowever that in all cases, there is one and only one mapping from anold name to a new name. If two.mv files contain differentmappings for an old name, or if a programmatic call is made with anold name that is already remapped, the previous mapping is lost. Thiswill not affect any already imported modules.

The following methods are available on thesys.stdlib_remapperobject:

• read_mv_file(filename) – Read the given file and register allremappings found in the file.
• read_directory_mv_files(dirname,suffix='.mv') – List the givendirectory, reading all files in that directory that have thematching suffix (.mv by default). For each parsed file,register all the remappings found in that file.
• set_mapping(oldname,newname) – Register a new mapping from anold module name to a new module name. Both must be the fulldotted-path name to the module. newname may beNone in whichcase any existing mapping for oldname will be removed (it is not anerror if there is no existing mapping).
• get_mapping(oldname,default=None) – Return any registerednewname for the given oldname. If there is no registered remapping,default is returned.

Open Issues

• Should there be a command line switch and/or environment variable todisable all remappings?
• Should remappings occur recursively?
• Should we automatically parse package directories for .mv files whenthe package’s __init__.py is loaded? This would allow packages toeasily include .mv files for their own remappings. Compare what theemail package currently has to do if we place its.mv file inthe email package instead of in the oldlib package:

  # Expose old namesimportos,syssys.stdlib_remapper.read_directory_mv_files(os.path.dirname(__file__))

  I think we should automatically read a package’s directory for any.mv files it might contain.

Reference Implementation

A reference implementation, in the form of a patch against the current(as of this writing) state of the Python 2.6 svn trunk, is availableas SourceForge patch #1675334[1]. Note that this patch includes arename ofcStringIO tocstringio, but this is primarily forillustrative and unit testing purposes. Should the patch be accepted,we might want to split this change off into otherPEP 3108 changes.

References

Copyright

This document has been placed in the public domain.