This PEP proposes the creation of a new platform tag for Python package builtdistributions, such as wheels, calledmanylinux1_{x86_64,i686} withexternal dependencies limited to a standardized, restricted subset ofthe Linux kernel and core userspace ABI. It proposes that PyPI supportuploading and distributing wheels with this platform tag, and thatpipsupport downloading and installing these packages on compatible platforms.
Currently, distribution of binary Python extensions for Windows and OS X isstraightforward. Developers and packagers build wheels (PEP 427,PEP 491),which areassigned platform tags such aswin32 ormacosx_10_6_intel, and uploadthese wheels to PyPI. Users can download and install these wheels using toolssuch aspip.
For Linux, the situation is much more delicate. In general, compiled Pythonextension modules built on one Linux distribution will not work on other Linuxdistributions, or even on different machines running the same Linuxdistribution with different system libraries installed.
Build tools usingPEP 425 platform tags do not track information about theparticular Linux distribution or installed system libraries, and instead assignall wheels the too-vaguelinux_i686 orlinux_x86_64 tags. Because ofthis ambiguity, there is no expectation thatlinux-tagged builtdistributions compiled on one machine will work properly on another, and forthis reason, PyPI has not permitted the uploading of wheels for Linux.
It would be ideal if wheel packages could be compiled that would work onanylinux system. But, because of the incredible diversity of Linux systems – fromPCs to Android to embedded systems with custom libcs – this cannotbe guaranteed in general.
Instead, we define a standard subset of the kernel+core userspace ABI that,in practice, is compatible enough that packages conforming to this standardwill work onmany linux systems, including essentially all of the desktopand server distributions in common use. We know this because there arecompanies who have been distributing such widely-portable pre-compiled Pythonextension modules for Linux – e.g. Enthought with Canopy[4] and ContinuumAnalytics with Anaconda[5].
Building on the compatibility lessons learned from these companies, we thusdefine a baselinemanylinux1 platform tag for use by binary Pythonwheels, and introduce the implementation of preliminary tools to aid in theconstruction of thesemanylinux1 wheels.
To properly define a standard that will guarantee that wheel packages meetingthis specification will operate onmany linux platforms, it is necessary tounderstand the root causes which often prevent portability of pre-compiledbinaries on Linux. The two key causes are dependencies on shared librarieswhich are not present on users’ systems, and dependencies on particularversions of certain core libraries likeglibc.
Most desktop and server linux distributions come with a system package manager(examples includeAPT on Debian-based systems,yum onRPM-based systems, andpacman on Arch linux) that manages, among otherresponsibilities, the installation of shared libraries installed to systemdirectories such as/usr/lib. Most non-trivial Python extensions will dependon one or more of these shared libraries, and thus function properly only onsystems where the user has the proper libraries (and the properversions thereof), either installed using their package manager, or installedmanually by setting certain environment variables such asLD_LIBRARY_PATHto notify the runtime linker of the location of the depended-upon sharedlibraries.
Even if the developers a Python extension module wish to use noexternal shared libraries, the modules will generally have a dynamic runtimedependency on the GNU C library,glibc. While it is possible, staticallylinkingglibc is usually a bad idea because certain important C functionslikedlopen() cannot be called from code that statically linksglibc. Aruntime shared library dependency on a system-providedglibc is unavoidablein practice.
The maintainers of the GNU C library follow a strict symbol versioning schemefor backward compatibility. This ensures that binaries compiled against an olderversion ofglibc can run on systems that have a newerglibc. Theopposite is generally not true – binaries compiled on newer Linuxdistributions tend to rely upon versioned functions inglibc that are notavailable on older systems.
This generally prevents wheels compiled on the latest Linux distributionsfrom being portable.
manylinux1 policyFor these reasons, to achieve broad portability, Python wheels
To be eligible for themanylinux1 platform tag, a Python wheel musttherefore both (a) contain binary executables and compiled code that linksonly to libraries with SONAMEsincluded in the following list:
libpanelw.so.5libncursesw.so.5libgcc_s.so.1libstdc++.so.6libm.so.6libdl.so.2librt.so.1libc.so.6libnsl.so.1libutil.so.1libpthread.so.0libresolv.so.2libX11.so.6libXext.so.6libXrender.so.1libICE.so.6libSM.so.6libGL.so.1libgobject-2.0.so.0libgthread-2.0.so.0libglib-2.0.so.0
and, (b) work on a stock CentOS 5.11[6] system that contains the systempackage manager’s provided versions of these libraries.
libcrypt.so.1 was retrospectively removed from the whitelist afterFedora 30 was released withlibcrypt.so.2 instead.
Because CentOS 5 is only available for x86_64 and i686 architectures,these are the only architectures currently supported by themanylinux1policy.
On Debian-based systems, these libraries are provided by the packages
libncurses5libgcc1libstdc++6libc6libx11-6libxext6libxrender1libice6libsm6libgl1-mesa-glxlibglib2.0-0
On RPM-based systems, these libraries are provided by the packages
ncurseslibgcclibstdc++glibclibXextlibXrenderlibICElibSMmesa-libGLglib2
This list was compiled by checking the external shared library dependencies ofthe Canopy[4] and Anaconda[5] distributions, which both include a wide arrayof the most popular Python modules and have been confirmed in practice to workacross a wide swath of Linux systems in the wild.
Many of the permitted system libraries listed above use symbol versioningschemes for backward compatibility. The latest symbol versions provided withthe CentOS 5.11 versions of these libraries are:
GLIBC_2.5CXXABI_3.4.8GLIBCXX_3.4.9GCC_4.2.0
Therefore, as a consequence of requirement (b), any wheel that depends onversioned symbols from the above shared libraries may depend only on symbolswith the following versions:
GLIBC<=2.5CXXABI<=3.4.8GLIBCXX<=3.4.9GCC<=4.2.0
These recommendations are the outcome of the relevant discussions in January2016[7],[8].
Note that in our recommendations below, we do not suggest thatpipor PyPI should attempt to check for and enforce the details of thispolicy (just as they don’t check for and enforce the details ofexisting platform tags likewin32). The text above is provided (a)as advice to package builders, and (b) as a method for allocatingblame if a given wheel doesn’t work on some system: if it satisfiesthe policy above, then this is a bug in the spec or the installationtool; if it does not satisfy the policy above, then it’s a bug in thewheel. One useful consequence of this approach is that it leaves openthe possibility of further updates and tweaks as we gain moreexperience, e.g., we could have a “manylinux 1.1” policy which targetsthe same systems and uses the samemanylinux1 platform tag (andthus requires no further changes topip or PyPI), but that adjuststhe list above to remove libraries that have turned out to beproblematic or add libraries that have turned out to be safe.
Note thatlibpythonX.Y.so.1 isnot on the list of libraries thatamanylinux1 extension is allowed to link to. Explicitly linkingtolibpythonX.Y.so.1 is unnecessary in almost all cases: the wayELF linking works, extension modules that are loaded into theinterpreter automatically get access to all of the interpreter’ssymbols, regardless of whether or not the extension itself isexplicitly linked against libpython. Furthermore, explicit linking tolibpython creates problems in the common configuration where Python isnot built with--enable-shared. In particular, on Debian andUbuntu systems,aptinstallpythonX.Y does not even installlibpythonX.Y.so.1, meaning that any wheel thatdid depend onlibpythonX.Y.so.1 could fail to import.
There is one situation where extensions that are linked in this waycan fail to work: if a host program (e.g.,apache2) usesdlopen() to load a module (e.g.,mod_wsgi) that embeds theCPython interpreter, and the host program doesnot pass theRTLD_GLOBAL flag todlopen(), then the embedded CPython willbe unable to load any extension modules that do not themselves linkexplicitly tolibpythonX.Y.so.1. Fortunately,apache2doesset theRTLD_GLOBAL flag, as do all the other programs thatembed-CPython-via-a-dlopened-plugin that we could locate, so this doesnot seem to be a serious problem in practice. The incompatibility withDebian/Ubuntu is more of an issue than the theoretical incompatibilitywith a rather obscure corner case.
This is a rather complex and subtle issue that extends beyondthe scope ofmanylinux1; for more discussion see:[9],[10],[11].
All versions of CPython 2.x, plus CPython 3.0-3.2 inclusive, can bebuilt in two ABI-incompatible modes: builds using the--enable-unicode=ucs2 configure flag store Unicode data in UCS-2(or really UTF-16) format, while builds using the--enable-unicode=ucs4 configure flag store Unicode data inUCS-4. (CPython 3.3 and greater use a different storage method thatalways supports UCS-4.) If we want to make sureucs2 wheels don’tget installed intoucs4 CPythons and vice-versa, then somethingmust be done.
An earlier version of this PEP included a requirement thatmanylinux1 wheels targeting these older CPython versions shouldalways use theucs4 ABI. But then, in between the PEP’s initialacceptance and its implementation,pip andwheel gainedfirst-class support for tracking and checking this aspect of ABIcompatibility for the relevant CPython versions, which is a bettersolution. So we now allow themanylinux1 platform tags to be usedin combination with any ABI tag. However, to maintain compatibility itis crucial to ensure that allmanylinux1 wheels include anon-trivial abi tag. For example, a wheel built against aucs4CPython might have a name like:
PKG-VERSION-cp27-cp27mu-manylinux1_x86_64.whl ^^^^^^ Good!
While a wheel built against theucs2 ABI might have a name like:
PKG-VERSION-cp27-cp27m-manylinux1_x86_64.whl ^^^^^ Okay!
But you should never have a wheel with a name like:
PKG-VERSION-cp27-none-manylinux1_x86_64.whl ^^^^ BAD! Don't do this!
This wheel claims to be simultaneously compatible withboth ucs2 anducs4 builds, which is bad.
We note for information that theucs4 ABI appears to be much morewidespread among Linux CPython distributors.
All extant versions of CPython can be built either with or without the--with-fpectl flag toconfigure. It turns out that thischanges the CPython ABI: extensions that are built against ano-fpectl CPython are always compatible with yes-fpectlCPython, but the reverse is not necessarily true. (Symptom: errors atimport time complaining aboutundefinedsymbol:PyFPE_jbuf.) See:[16].
For maximum compatibility, therefore, the CPython used to buildmanylinux1 wheels must be compiledwithout the--with-fpectlflag, and manylinux1 extensions must not reference the symbolPyFPE_jbuf.
The way glibc, libgcc, and libstdc++ manage their symbol versioningmeans that in practice, the compiler toolchains that most developersuse to do their daily work are incapable of buildingmanylinux1-compliant wheels. Therefore, we do not attempt to changethe default behavior ofpipwheel /bdist_wheel: they willcontinue to generate regularlinux_* platform tags, and developerswho wish to use them to generatemanylinux1-tagged wheels willhave to change the tag as a second post-processing step.
To support the compilation of wheels meeting themanylinux1 standard, weprovide initial drafts of two tools.
The first tool is a Docker image based on CentOS 5.11, which is recommended asan easy to use self-contained build box for compilingmanylinux1 wheels[12]. Compiling on a more recently-released linux distribution will generallyintroduce dependencies on too-new versioned symbols. The image comes with afull compiler suite installed (gcc,g++, andgfortran 4.8.2) aswell as the latest releases of Python andpip.
The second tool is a command line executable calledauditwheel[13] thatmay aid in package maintainers in dealing with third-party externaldependencies.
There are at least three methods for building wheels that use third-partyexternal libraries in a way that meets the above policy.
All of these are valid option which may be effectively used by differentpackages and communities. Statically linking generally requirespackage-specific modifications to the build system, and distributingthird-party dependencies on PyPI may require some coordination of thecommunity of users of the package.
As an often-automatic alternative to these options, we introduceauditwheel.The tool inspects all of the ELF files inside a wheel to check fordependencies on versioned symbols or external shared libraries, and verifiesconformance with themanylinux1 policy. This includes the ability to addthe new platform tag to conforming wheels. More importantly,auditwheel hasthe ability to automatically modify wheels that depend on external sharedlibraries by copying those shared libraries from the system into the wheelitself, and modifying the appropriateRPATH entries such that theselibraries will be picked up at runtime. This accomplishes a similar result asif the libraries had been statically linked without requiring changes to thebuild system. Packagers are advised that bundling, like static linking, mayimplicate copyright concerns.
While we acknowledge many approaches for dealing with third-party librarydependencies withinmanylinux1 wheels, we recognize that themanylinux1policy encourages bundling external dependencies, a practicewhich runs counter to the package management policies of many linuxdistributions’ system package managers[14],[15]. The primary purpose ofthis is cross-distro compatibility. Furthermore,manylinux1 wheels on PyPIoccupy a different niche than the Python packages available through thesystem package manager.
The decision in this PEP to encourage departure from general Linux distributionunbundling policies is informed by the following concerns:
pip users remain free to use the"--no-binary" option if they wantto force local builds rather than using pre-built wheel files.The model described in this PEP is most ideally suited for cross-platformPython packages, because it means they can reuse much of thework that they’re already doing to make static Windows and OS X wheels. Werecognize that it is less optimal for Linux-specific packages that mightprefer to interact more closely with Linux’s unique package managementfunctionality and only care about targeting a small set of particular distos.
One of the advantages of dependencies on centralized libraries in Linux isthat bugfixes and security updates can be deployed system-wide, andapplications which depend on these libraries will automatically feel theeffects of these patches when the underlying libraries are updated. This canbe particularly important for security updates in packages engaged incommunication across the network or cryptography.
manylinux1 wheels distributed through PyPI that bundle security-criticallibraries like OpenSSL will thus assume responsibility for prompt updates inresponse disclosed vulnerabilities and patches. This closely parallels thesecurity implications of the distribution of binary wheels on Windows that,because the platform lacks a system package manager, generally bundle theirdependencies. In particular, because it lacks a stable ABI, OpenSSL cannot beincluded in themanylinux1 profile.
Above, we defined what it means for awheel to bemanylinux1-compatible. Here we discuss what it means for aPythoninstallation to bemanylinux1-compatible. In particular, this isimportant for tools likepip to know when deciding whether or notthey should considermanylinux1-tagged wheels for installation.
Because themanylinux1 profile is already known to work for themany thousands of users of popular commercial Python distributions, wesuggest that installation tools should error on the side of assumingthat a systemis compatible, unless there is specific reason tothink otherwise.
We know of four main sources of potential incompatibility that arelikely to arise in practice:
glibc (e.g. Alpine Linux, which isbased on musllibc, or Android)To address these we propose a two-prongedapproach. To handle potential future incompatibilities, we standardizea mechanism for a Python distributor to signal that a particularPython install definitely is or is not compatible withmanylinux1:this is done by installing a module named_manylinux, and settingitsmanylinux1_compatible attribute. We do not propose adding anysuch module to the standard library – this is merely a well-knownname by which distributors and installation tools canrendezvous. However, if a distributor does add this module,theyshould add it to the standard library rather than to asite-packages/ directory, because the standard library isinherited by virtualenvs (which we want), andsite-packages/ ingeneral is not.
Then, to handle the last two cases for existing Pythondistributions, we suggest a simple and reliable method to check forthe presence and version ofglibc (basically using it as a “clock”for the overall age of the distribution).
Specifically, the algorithm we propose is:
defis_manylinux1_compatible():# Only Linux, and only x86-64 / i686fromdistutils.utilimportget_platformifget_platform()notin["linux-x86_64","linux-i686"]:returnFalse# Check for presence of _manylinux moduletry:import_manylinuxreturnbool(_manylinux.manylinux1_compatible)except(ImportError,AttributeError):# Fall through to heuristic check belowpass# Check glibc version. CentOS 5 uses glibc 2.5.returnhave_compatible_glibc(2,5)defhave_compatible_glibc(major,minimum_minor):importctypesprocess_namespace=ctypes.CDLL(None)try:gnu_get_libc_version=process_namespace.gnu_get_libc_versionexceptAttributeError:# Symbol doesn't exist -> therefore, we are not linked to# glibc.returnFalse# Call gnu_get_libc_version, which returns a string like "2.5".gnu_get_libc_version.restype=ctypes.c_char_pversion_str=gnu_get_libc_version()# py2 / py3 compatibility:ifnotisinstance(version_str,str):version_str=version_str.decode("ascii")# Parse string and check against requested version.version=[int(piece)forpieceinversion_str.split(".")]assertlen(version)==2ifmajor!=version[0]:returnFalseifminimum_minor>version[1]:returnFalsereturnTrue
Rejected alternatives: We also considered using a configurationfile, e.g./etc/python/compatibility.cfg. The problem with this isthat a single filesystem might contain many different interpreterenvironments, each with their own ABI profile – themanylinux1compatibility of a system-installed x86_64 CPython might not tell usmuch about themanylinux1 compatibility of a user-installed i686PyPy. Locating this configuration information within the Pythonenvironment itself ensures that it remains attached to the correctbinary, and dramatically simplifies lookup code.
We also considered using a more elaborate structure, like a list ofall platform tags that should be considered compatible, together withtheir preference ordering, for example:_binary_compat.compatible=["manylinux1_x86_64","centos5_x86_64","linux_x86_64"]. However,this introduces several complications. For example, we want to be ableto distinguish between the state of “doesn’t supportmanylinux1”(or eventuallymanylinux2, etc.) versus “doesn’t specify eitherway whether it supportsmanylinux1”, which is not entirely obviousin the above representation; and, it’s not at all clear what featuresare really needed vis a vis preference ordering given that right nowthe only possible platform tags aremanylinux1 andlinux. Sowe’re deferring a more complete solution here for a separate PEP, when/ if Linux gets more platform tags.
For the library compatibility check, we also considered much moreelaborate checks (e.g. checking the kernel version, searching for andchecking the versions of all the individual libraries listed in themanylinux1 profile, etc.), but ultimately decided that this wouldbe more likely to introduce confusing bugs than actually help theuser. (For example: different distributions vary in where theyactually put these libraries, and if our checking code failed to usethe correct path search then it could easily return incorrectanswers.)
PyPI should permit wheels containing themanylinux1 platform tag to beuploaded. PyPI should not attempt to formally verify that wheels containingthemanylinux1 platform tag adhere to themanylinux1 policy describedin this document. This verification tasks should be left to other tools, likeauditwheel, that are developed separately.
One alternative would be to provide separate platform tags for each Linuxdistribution (and each version thereof), e.g.RHEL6,ubuntu14_10,debian_jessie, etc. Nothing in this proposal rules out the possibility ofadding such platform tags in the future, or of further extensions to wheelmetadata that would allow wheels to declare dependencies on externalsystem-installed packages. However, such extensions would require substantiallymore work than this proposal, and still might not be appreciated by packagedevelopers who would prefer not to have to maintain multiple build environmentsand build multiple wheels in order to cover all the common Linux distributions.Therefore, we consider such proposals to be out-of-scope for this PEP.
We anticipate that at some point in the future there will be amanylinux2 specifying a more modern baseline environment (perhapsbased on CentOS 6), and someday amanylinux3 and so forth, but wedefer specifying these until we have more experience with the initialmanylinux1 proposal.
This document has been placed into the public domain.
Source:https://github.com/python/peps/blob/main/peps/pep-0513.rst
Last modified:2025-02-01 08:59:27 GMT