This PEP proposes the creation of amanylinux2010 platform tag tosucceed themanylinux1 tag introduced byPEP 513. It alsoproposes that PyPI andpip both be updated to support uploading,downloading, and installingmanylinux2010 distributions on compatibleplatforms.
True to its name, themanylinux1 platform tag has made theinstallation of binary extension modules a reality on many Linuxsystems. Libraries likecryptography[2] andnumpy[3] aremore accessible to Python developers now that their installation oncommon architectures does not depend on fragile developmentenvironments and build toolchains.
manylinux1 wheels achieve their portability by allowing theextension modules they contain to link against only a small set ofsystem-level shared libraries that export versioned symbols old enoughto benefit from backwards-compatibility policies. Extension modulesin amanylinux1 wheel that rely onglibc, for example, must bebuilt against version 2.5 or earlier; they may then be run systemsthat provide more recentglibc version that still export therequired symbols at version 2.5.
PEP 513 drew its whitelisted shared libraries and their symbolversions from CentOS 5.11, which was the oldest supported CentOSrelease at the time of its writing. Unfortunately, CentOS 5.11reached its end-of-life on March 31st, 2017 with a clear warningagainst its continued use.[4] No further updates, such as securitypatches, will be made available. This means that its packages willremain at obsolete versions that hamper the efforts of Python softwarepackagers who use themanylinux1 Docker image.
CentOS 6 is now the oldest supported CentOS release, and will receivemaintenance updates through November 30th, 2020.[5] We propose thata newPEP 425-style platform tag calledmanylinux2010 be derivedfrom CentOS 6 and that themanylinux toolchain, PyPI, andpipbe updated to support it.
This was originally proposed asmanylinux2, but the versioning hasbeen changed to use calendar years (also known as CalVer[22]). Thismakes it easier to define futuremanylinux tags out of order: forexample, a hypotheticalmanylinux2017 standard may be defined viaa new PEP beforemanylinux2014, or amanylinux2007 standardmight be defined that targets systems older than this PEP but newerthanmanylinux1.
Calendar versioning also gives a rough idea of which Linuxdistribution versions support which tag:manylinux2010 will workon most distribution versions released since 2010. This is only anapproximation, however: the actual compatibility rules are definedbelow, and some newer distributions may not meet them.
manylinux2010 policyThe following criteria determine alinux wheel’s eligibility forthemanylinux2010 tag:
libgcc_s.so.1libstdc++.so.6libm.so.6libdl.so.2librt.so.1libc.so.6libnsl.so.1libutil.so.1libpthread.so.0libresolv.so.2libX11.so.6libXext.so.6libXrender.so.1libICE.so.6libSM.so.6libGL.so.1libgobject-2.0.so.0libgthread-2.0.so.0libglib-2.0.so.0
This list is identical to the externally-provided librarieswhitelisted formanylinux1, minuslibncursesw.so.5 andlibpanelw.so.5.[7]libpythonX.Y remains ineligible forinclusion for the same reasons outlined inPEP 513.
libcrypt.so.1 was retrospectively removed from the whitelist afterFedora 30 was released withlibcrypt.so.2 instead.
On Debian-based systems, these libraries are provided by the packages:
| Package | Libraries |
|---|---|
| libc6 | libdl.so.2, libresolv.so.2, librt.so.1, libc.so.6,libpthread.so.0, libm.so.6, libutil.so.1, libnsl.so.1 |
| libgcc1 | libgcc_s.so.1 |
| libgl1 | libGL.so.1 |
| libglib2.0-0 | libgobject-2.0.so.0, libgthread-2.0.so.0, libglib-2.0.so.0 |
| libice6 | libICE.so.6 |
| libsm6 | libSM.so.6 |
| libstdc++6 | libstdc++.so.6 |
| libx11-6 | libX11.so.6 |
| libxext6 | libXext.so.6 |
| libxrender1 | libXrender.so.1 |
On RPM-based systems, they are provided by these packages:
| Package | Libraries |
|---|---|
| glib2 | libglib-2.0.so.0, libgthread-2.0.so.0, libgobject-2.0.so.0 |
| glibc | libresolv.so.2, libutil.so.1, libnsl.so.1, librt.so.1,libpthread.so.0, libdl.so.2, libm.so.6, libc.so.6 |
| libICE | libICE.so.6 |
| libX11 | libX11.so.6 |
| libXext: | libXext.so.6 |
| libXrender | libXrender.so.1 |
| libgcc: | libgcc_s.so.1 |
| libstdc++ | libstdc++.so.6 |
| mesa | libGL.so.1 |
GLIBC_2.12CXXABI_1.3.3GLIBCXX_3.4.13GCC_4.5.0
As an example,manylinux2010 wheels may include binary artifactsthat requireglibc symbols at versionGLIBC_2.4, becausethis an earlier version than the maximum ofGLIBC_2.12.
manylinux2010 wheel builtagainst Python 2, then, must include either thecpy27mu tagindicating it was built against an interpreter with the UCS-4 ABIor thecpy27m tag indicating an interpreter with the UCS-2ABI. (PEP 3149,[9])PyFPE_jbuf symbol. This isachieved by building it against a Python compiledwithout the--with-fpectlconfigure flag.Likemanylinux1, theauditwheel tool addsmanylinux2010platform tags tolinux wheels built bypipwheel orbdist_wheel in amanylinux2010 Docker container.
Twomanylinux2010 Docker images based on CentOS 6 areprovided for building binarylinux wheels that can reliably beconverted tomanylinux2010 wheels.[10] The x86_64 and i686 images comes with anew compiler suite installed (gcc,g++, andgfortranfromdevtoolset-8) as well as the latest releases of Python andpip.
vsyscallA Docker container assumes that its userland is compatible with itshost’s kernel. Unfortunately, an increasingly common kernelconfiguration breaks this assumption for x86_64 CentOS 6 Dockerimages.
Versions 2.14 and earlier ofglibc require the kernel provide anarchaic system call optimization known asvsyscall on x86_64.[11]To effect the optimization, the kernel maps a read-only page offrequently-called system calls – most notablytime(2) – intoeach process at a fixed memory location.glibc then invokes thesesystem calls by dereferencing a function pointer to the appropriateoffset into thevsyscall page and calling it. This avoids theoverhead associated with invoking the kernel that affects normalsystem call invocation.vsyscall has long been deprecated infavor of an equivalent mechanism known as vDSO, or “virtual dynamicshared object”, in which the kernel instead maps a relocatable virtualshared object containing the optimized system calls into eachprocess.[12]
Thevsyscall page has serious security implications because itdoes not participate in address space layout randomization (ASLR).Its predictable location and contents make it a useful source ofgadgets used in return-oriented programming attacks.[13] At the sametime, its elimination breaks the x86_64 ABI, becauseglibcversions that depend onvsyscall suffer from segmentation faultswhen attempting to dereference a system call pointer into anon-existent page. As a compromise, Linux 3.1 implemented an“emulated”vsyscall that reduced the executable code, and thus thematerial for ROP gadgets, mapped into the process.[14]vsyscall=emulated has been the default configuration in mostdistribution’s kernels for many years.
Unfortunately,vsyscall emulation still exposes predictable codeat a reliable memory location, and continues to be useful forreturn-oriented programming.[15] Because most distributions have nowupgraded toglibc versions that do not depend onvsyscall,they are beginning to ship kernels that do not supportvsyscall atall.[16]
CentOS 5.11 and 6 both include versions ofglibc that depend onthevsyscall page (2.5 and 2.12.2 respectively), so containersbased on either cannot run under kernels provided with manydistribution’s upcoming releases.[17] If Travis CI, for example,begins running jobs undera kernel that does not provide thevsyscall interface, Pythonpackagers will not be able to use our Docker images there to buildmanylinux wheels.[18]
We have derived a patch from theglibc git repository thatbackports the removal of all dependencies onvsyscall to theversion ofglibc included with ourmanylinux2010 image.[19]Rebuildingglibc, and thus buildingmanylinux2010 image itself,still requires a host kernel that provides thevsyscall mechanism,but the resulting image can be both run on hosts that provide it andthose that do not. Because thevsyscall interface is anoptimization that is only applied to running processes, themanylinux2010 wheels built with this modified image should beidentical to those built on an unmodified CentOS 6 system. Also, thevsyscall problem applies only to x86_64; it is not part of thei686 ABI.
Theauditwheel tool has also been updated to producemanylinux2010 wheels.[20] Its behavior and purpose are otherwiseunchanged fromPEP 513.
Platforms may define amanylinux2010_compatible boolean attribute onthe_manylinux module described inPEP 513. A platform isconsidered incompatible withmanylinux2010 if the attribute isFalse.
If the_manylinux module is not found, or it does not have the attributemanylinux2010_compatible, tools may fall back to checking for glibc. If theplatform has glibc 2.12 or newer, it is assumed to be compatible unless the_manylinux module says otherwise.
Specifically, the algorithm we propose is:
defis_manylinux2010_compatible():# Only Linux, and only x86-64 / i686fromdistutils.utilimportget_platformifget_platform()notin["linux-x86_64","linux-i686"]:returnFalse# Check for presence of _manylinux moduletry:import_manylinuxreturnbool(_manylinux.manylinux2010_compatible)except(ImportError,AttributeError):# Fall through to heuristic check belowpass# Check glibc version. CentOS 6 uses glibc 2.12.# PEP 513 contains an implementation of this function.returnhave_compatible_glibc(2,12)
manylinux1 wheelsAs explained inPEP 513, the specified symbol versions formanylinux1 whitelisted libraries constitute anupper bound. Thesame is true for the symbol versions defined formanylinux2010 inthis PEP. As a result,manylinux1 wheels are consideredmanylinux2010 wheels. Apip that recognizes themanylinux2010platform tag will thus installmanylinux1 wheels formanylinux2010 platforms – even when explicitly set – when nomanylinux2010 wheels are available.[21]
PyPI should permit wheels containing themanylinux2010 platform tagto be uploaded in the same way that it permitsmanylinux1. Itshould not attempt to verify the compatibility ofmanylinux2010wheels.
The following changes were made to this PEP based on feedback received afterit was approved:
libgcc_s was updated fromGCC_4.3.0toGCC_4.5.0 to address 32-bit Cent OS 6. This doesn’t affect x86_64becauselibgcc_s for x86_64 has no additional symbol fromGCC_4.3.0 toGCC_4.5.0.This document has been placed into the public domain.
Source:https://github.com/python/peps/blob/main/peps/pep-0571.rst
Last modified:2025-02-01 08:55:40 GMT