Abstract

PEP 3147 described an extension to Python’s import machinery thatimproved the sharing of Python source code, by allowing more than onebyte compilation file (.pyc) to be co-located with each source file.

This PEP defines an adjunct feature which allows the co-location ofextension module files (.so) in a similar manner. This optional,build-time feature will enable downstream distributions of Python tomore easily provide more than one Python major version at a time.

Background

PEP 3147 defined the file system layout for a pure-Python package,where multiple versions of Python are available on the system. Forexample, where thealpha package containing source modulesone.pyandtwo.py exist on a system with Python 3.2 and 3.3, the post-bytecompilation file system layout would be:

alpha/__pycache__/__init__.cpython-32.pyc__init__.cpython-33.pycone.cpython-32.pycone.cpython-33.pyctwo.cpython-32.pyctwo.cpython-33.pyc__init__.pyone.pytwo.py

For packages with extension modules, a similar differentiation isneeded for the module’s .so files. Extension modules compiled fordifferent Python major versions are incompatible with each other dueto changes in the ABI. Different configuration/compilation optionsfor the same Python version can result in different ABIs(e.g. –with-wide-unicode).

WhilePEP 384 defines a stable ABI, it will minimize, but noteliminate extension module incompatibilities between Python builds ormajor versions. Thus a mechanism for discriminating extension modulefile names is proposed.

Rationale

Linux distributions such as Ubuntu[3] and Debian[4] provide morethan one Python version at the same time to their users. For example,Ubuntu 9.10 Karmic Koala users can install Python 2.5, 2.6, and 3.1,with Python 2.6 being the default.

In order to share as much as possible between the available Pythonversions, these distributions install third party package modules(.pyc and.so files) into/usr/share/pyshared and symlink tothem from/usr/lib/pythonX.Y/dist-packages. The symlinks existbecause in a pre-PEP 3147 world (i.e < Python 3.2), the.pyc filesresulting from byte compilation by the various installed Pythons willname collide with each other. For Python versions >= 3.2, allpure-Python packages can be shared, because the.pyc files will nolonger cause file system naming conflicts. Eliminating these symlinksmakes for a simpler, more robust Python distribution.

A similar situation arises with shared library extensions. Becauseextension modules are typically namedfoo.so for afoo extensionmodule, these would also name collide iffoo was provided for morethan one Python version.

In addition, because different configuration/compilation options forthe same Python version can cause different ABIs to be presented toextension modules. On POSIX systems for example, the configureoptions--with-pydebug,--with-pymalloc, and--with-wide-unicode all change the ABI. This PEP proposes toencode build-time options in the file name of the.so extensionmodule files.

PyPy[5] can also benefit from this PEP, allowing it to avoid namecollisions in extension modules built for its API, but with adifferent.so tag.

Proposal

The configure/compilation options chosen at Python interpreterbuild-time will be encoded in the shared library file name forextension modules. This “tag” will appear between the module basename and the operation file system extension for shared libraries.

The following informationMUST be included in the shared libraryfile name:

• The Python implementation (e.g. cpython, pypy, jython, etc.)
• The interpreter’s major and minor version numbers

These two fields are separated by a hyphen and no dots are to appearbetween the major and minor version numbers. E.g.cpython-32.

Python implementationsMAY include additional flags in the file nametag as appropriate. For example, on POSIX systems these flags willalso contribute to the file name:

• --with-pydebug (flag:d)
• --with-pymalloc (flag:m)
• --with-wide-unicode (flag:u)

By default in Python 3.2,configure enables--with-pymalloc soshared library file names would appear asfoo.cpython-32m.so.When the other two flags are also enabled, the file names would befoo.cpython-32dmu.so.

The shared library file name tag is used unconditionally; it cannot bechanged. The tag and extension module suffix are available throughthesysconfig modules via the following variables:

>>>sysconfig.get_config_var('EXT_SUFFIX')'.cpython-32mu.so'>>>sysconfig.get_config_var('SOABI')'cpython-32mu'

Note that$SOABI contains just the tag, while$EXT_SUFFIX includes theplatform extension for shared library files, and is the exact suffixadded to the extension module name.

For an arbitrary packagefoo, you might see these files when thedistribution package was installed:

/usr/lib/python/foo.cpython-32m.so/usr/lib/python/foo.cpython-33m.so

(These paths are for example purposes only. Distributions are free touse whatever filesystem layout they choose, and nothing in this PEPchanges the locations where from-source builds of Python areinstalled.)

Python’s dynamic module loader will recognize and import sharedlibrary extension modules with a tag that matches its build-timeoptions. For backward compatibility, Python will also continue toimport untagged extension modules, e.g.foo.so.

This shared library tag would be used globally for all distutils-basedextension modules, regardless of where on the file system they arebuilt. Extension modules built by means other than distutils wouldeither have to calculate the tag manually, or fallback to thenon-tagged.so file name.

Proven approach

The approach described here is already proven, in a sense, on Debianand Ubuntu system where different extensions are used for debug buildsof Python and extension modules. Debug builds on Windows also alreadyuse a different file extension for dynamic libraries, and in factencoded (in a different way than proposed in this PEP) the Pythonmajor and minor version in the.dll file name.

Windows

This PEP only addresses build issues on POSIX systems that use theconfigure script. While Windows or other platform support is notexplicitly disallowed under this PEP, platform expertise is needed inorder to evaluate, describe, and implement support on such platforms.It is not currently clear that the facilities in this PEP are evenuseful for Windows.

PEP 384

PEP 384 defines a stable ABI for extension modules. In theory,universal adoption ofPEP 384 would eliminate the need for this PEPbecause all extension modules could be compatible with any Pythonversion. In practice of course, it will be impossible to achieveuniversal adoption, and as described above, different build-time flagsstill affect the ABI. Thus even with a stable ABI, this PEP may stillbe necessary. While a complete specification is reserved forPEP 384,here is a discussion of the relevant issues.

PEP 384 describes a change toPyModule_Create() where3 ispassed as the API version if the extension was compiled withPy_LIMITED_API. This should be formalized into an official macrocalledPYTHON_ABI_VERSION to mirrorPYTHON_API_VERSION. Ifand when the ABI changes in an incompatible way, this version numberwould be bumped. To facilitate sharing, Python would be extended tosearch for extension modules with thePYTHON_ABI_VERSION number inits name. The prefixabi is reserved for Python’s use.

Thus, an initial implementation ofPEP 384, when Python is configuredwith the default set of flags, would search for the following filenames when extension modulefoo is imported (in this order):

foo.cpython-XYm.sofoo.abi3.sofoo.so

The distutils[6]build_ext command would also have to beextended to compile to shared library files with theabi3 tag,when the module author indicates that their extension supports thatversion of the ABI. This could be done in a backward compatible wayby adding a keyword argument to theExtension class, such as:

Extension('foo',['foo.c'],abi=3)

Martin v. Löwis describes his thoughts[7] about the applicability of thisPEP toPEP 384. In summary:

• --with-pydebug would not be supported by the stable ABI becausethis changes the layout ofPyObject, which is an exposedstructure.
• --with-pymalloc has no bearing on the issue.
• --with-wide-unicode is trickier, though Martin’s inclination isto force the stable ABI to use aPy_UNICODE that matches theplatform’swchar_t.

Alternatives

In the initial python-dev thread[8] where this idea was firstintroduced, several alternatives were suggested. For completenessthey are listed here, along with the reasons for not adopting them.

Reference implementation

Work on this code is tracked in a Bazaar branch on Launchpad[9]until it’s ready for merge into Python 3.2. The work-in-progress diffcan also be viewed[10] and is updated automatically as new changesare uploaded.

References

Copyright

This document has been placed in the public domain.