Abstract

This PEP proposes a scheme for new ‘manylinux’ wheel tags to bedefined without requiring a PEP for every specific tag, similar to howWindows and macOS tags already work. This will allow packagemaintainers to take advantage of new tags more quickly, while makingbetter use of limited volunteer time.

Non-goals include: handling non-glibc-based platforms; integratingwith external package managers or handling external dependencies suchas CUDA; making manylinux tags more sophisticated than theirWindows/macOS equivalents; doing anything besides taking our existingtried-and-tested approach and streamlining it. These are importantissues and other PEPs may address them in the future, but for this PEPthey’re out of scope.

Rationale

Python users appreciate it when PyPI has pre-compiled packages fortheir platform, because it makes installation fast and simple. Butdistributing pre-compiled binaries on Linux is challenging because ofthe diversity of Linux-based platforms. For example, Debian, Android,and Alpine all use the Linux kernel, but with radically differentuserspace libraries, which makes it difficult or impossible to createa single wheel that works on all three. This complexity has causedmany previous discussions of Linux wheels to stall out.

The “manylinux” project succeeded by adopting a strategy of ruthlesspragmatism. We chose a large but tractable set of Linux platforms –specifically, mainstream glibc-based distributions like Debian,OpenSuSE, Ubuntu, RHEL, etc. – and then we did whatever it takes tomake wheels that work across all these platforms.

This approach requires many compromises. Manylinux wheels can onlyrely on external libraries that maintain a consistent ABI and areuniversally available across all these distributions, which inpractice restricts them to a small set of core libraries like glibcand a few others. Wheels have to be built on carefully-chosenplatforms of the oldest possible vintage, using a Python that isitself built in a carefully-chosen configuration. Other shared librarydependencies have to be bundled into the wheel, which requires acomplex process to avoid collisions between unrelated wheels. Andfinally, the details of these requirements change over time, as newdistro versions are released, and old ones fall out of use.

It turns out that these requirements are not too onerous: they’reessentially equivalent to what you have to do to ship Windows or macOSwheels, and the manylinux approach has achieved substantial uptakeamong both package maintainers and end-users. But any manylinux PEPneeds some way to address these complexities.

In previous manylinux PEPs (PEP 513,PEP 571,PEP 599), we’vedone this by attempting to write down in the PEP the exact set oflibraries, symbol versions, Python configuration, etc. that webelieved would lead to wheels that work on all mainstream glibc-basedLinux systems. But this created several problems:

First, PEPs are generally supposed to be normative references: ifsoftware doesn’t match the PEP, then we fix the software. But in thiscase, the PEPs are attempting to describe Linux distributions, whichare a moving target, and do not consider our PEPs to constrain theirbehavior. This means that we’ve been taking on an unbounded commitmentto keep updating every manylinux PEP whenever the Linux distrolandscape changes. This is a substantial commitment for unfundedvolunteers to take on, and it’s not clear that this work producesvalue for our users.

And second, every time we move manylinux forward to a newer range ofsupported platforms, or add support for a new architecture, we have togo through a fairly elaborate process: writing a new PEP, updating thePyPI and pip codebases to recognize the new tag, waiting for the newpip to percolate to users, etc. None of this happens on Windows/macOS;it’s only a tax on Linux maintainers. This slows deployment of newmanylinux versions, and consumes part of our community’s limited PEPreview bandwidth, thus slowing progress of the Python packagingecosystem as a whole. This is especially problematic for less-populararchitectures, who have less volunteer resources to overcome thesebarriers.

How can we fix it?

A manylinux PEP has to address three main audiences:

• Package installers, like pip, need to be able to determine whichwheel tags are compatible with the system they find themselvesrunning on. This requires some automated process to introspect thesystem and match it up with wheel tags.
• Package indexes, like PyPI, need to be able to validate whichwheel tags are valid. Generally, this just requires something like alist of valid tags, or regex they match, with no need to knowanything about the actual semantics for individual tags. (But seethe discussion of upload verification below.)
• Package maintainers need to be able to build wheels that meetthe requirements for a given wheel tag.

Here’s the key insight behind this new PEP: it’s crucial thatdifferentpackage installers andpackage indexes all agree onwhich manylinux tags are valid and which systems they install on, sowe need a PEP to specify these – but, these are straightforward, anddon’t really change between manylinux versions. The complicated partthat keeps changing is the process of actuallybuilding the wheels– but, if there are multiple competing build environments, itdoesn’tmatter whether they use exactly the same rules as each other, as longas they all produce wheels that work on end-user systems. Therefore,we don’t need an interoperability standard for building wheels, so wedon’t need to write the details into a PEP.

To further convince ourselves that this approach will work, let’s lookagain at how we handle wheels on Windows and macOS: the PEPs describewhich tags are valid, and which systems they’re supposed to work on,but not how to actually build wheels for those platforms. And inpractice, if you want to distribute Windows or macOS wheels, you mighthave to jump through some complicated and poorly documented hoops inorder to bundle dependencies, target the right range of OS versions,etc. But the system works, and the way to improve it is to writebetter docs and build better tooling; no-one thinks that the way tomake Windows wheels work better is to publish a PEP describingwhich symbols we think Microsoft should be including in theirlibraries and how their linker ought to work. This PEP extends thatphilosophy to manylinux as well.

Specification

manylinux_2_17_x86_64

manylinux_${GLIBCMAJOR}_${GLIBCMINOR}_${ARCH}

LEGACY_ALIASES={"manylinux1_x86_64":"manylinux_2_5_x86_64","manylinux1_i686":"manylinux_2_5_i686","manylinux2010_x86_64":"manylinux_2_12_x86_64","manylinux2010_i686":"manylinux_2_12_i686","manylinux2014_x86_64":"manylinux_2_17_x86_64","manylinux2014_i686":"manylinux_2_17_i686","manylinux2014_aarch64":"manylinux_2_17_aarch64","manylinux2014_armv7l":"manylinux_2_17_armv7l","manylinux2014_ppc64":"manylinux_2_17_ppc64","manylinux2014_ppc64le":"manylinux_2_17_ppc64le","manylinux2014_s390x":"manylinux_2_17_s390x",}defmanylinux_tag_is_compatible_with_this_system(tag):# Normalize and parse the tagtag=LEGACY_ALIASES.get(tag,tag)m=re.match("manylinux_([0-9]+)_([0-9]+)_(.*)",tag)ifnotm:returnFalsetag_major_str,tag_minor_str,tag_arch=m.groups()tag_major=int(tag_major_str)tag_minor=int(tag_minor_str)ifnotsystem_uses_glibc():returnFalsesys_major,sys_minor=get_system_glibc_version()if(sys_major,sys_minor)<(tag_major,tag_minor):returnFalsesys_arch=get_system_arch()ifsys_arch!=tag_arch:returnFalse# Check for manual overridetry:import_manylinuxexceptImportError:passelse:ifhasattr(_manylinux,"manylinux_compatible"):result=_manylinux.manylinux_compatible(tag_major,tag_minor,tag_arch,)ifresultisnotNone:returnbool(result)else:if(tag_major,tag_minor)==(2,5):ifhasattr(_manylinux,"manylinux1_compatible"):returnbool(_manylinux.manylinux1_compatible)if(tag_major,tag_minor)==(2,12):ifhasattr(_manylinux,"manylinux2010_compatible"):returnbool(_manylinux.manylinux2010_compatible)returnTrue

Rejected alternatives

Continuing the manylinux20XX series: As discussed above, thisleads to much more effort-intensive, slower, and more complex rolloutsof new versions. And while there are two places where it seems atfirst to have some compensating benefits, if you look more closelythis turns out not to be the case.

First, this forces us to produce human-readable descriptions of howLinux distros work, in the text of the PEP. But this is less valuablethan it might seem at first, and can actually be handled better by thenew “perennial” approach anyway.

If you’re trying to build wheels, the main thing you need is atutorial on how to use the build images and tooling around them. Ifyou’re trying to add support for a new build profile or create acompetitor to auditwheel, then your best resources will be theauditwheel source code and issue tracker, which are always going to bemore detailed, precise, and reliable than a summary spec written inEnglish and without tests. Documentation like the old manylinux20XXPEPs does add value! But in both cases, it’s primarily as a secondaryreference to provide overview and context.

And furthermore, the PEP process is poorly suited to maintaining thiskind of reference documentation – there’s a reason we don’t keep thepip user manual in the PEPs repository! The auditwheel maintainers arethe best situated to understand what kinds of documentation are usefulto their users, and to maintain that documentation over time. Forexample, there’s substantial overlap between the different manylinuxversions, and the PEP process currently forces us to handle this bycopy-pasting everything between a growing list of documents; instead,the auditwheel maintainers might choose to factor out the common partsinto a single piece of shared documentation.

A related concern was that with the perennial approach, it may becomeharder for package maintainers to decide which build profile totarget: instead of having to pick betweenmanylinux1,manylinux2010,manylinux2014, …, they now have a wider arrayof options likemanylinux_2_5,manylinux_2_6, …,manylinux_2_20, … But again, we don’t believe this will be aproblem in practice. In either system, most package maintainers won’tbe starting by reading PEPs and trying to implement them from scratch.If you’re a particularly expert and ambitious package maintainer whoneeds to target a new version or new architecture, the perennialapproach gives you additional flexibility. But for regular everydaymaintainers, we expect they’ll start from a tutorial likepackaging.python.org, and by choosing from existing build images. Atutorial can just as easily recommendmanylinux_2_17 as it canrecommendmanylinux2014, and we expect the actual set ofpre-provided build images to be identical in both cases. And again, bymaintaining this documentation in the right place, instead of tryingto do it PEPs repository, we expect that we’ll end up withdocumentation that’s higher-quality and more fitted to purpose.

Finally, some participants have pointed out that it’s very nice to beable to look at a wheel and tell definitively whether it meets therequirements of the spec. With the new “perennial” approach, we cannever say with 100% certainty that a wheel does meet the spec, becausethat depends on the Linux distros. As engineers we have awell-justified dislike for that kind of uncertainty.

However: as demonstrated by the examples above, we can still telldefinitively when a wheeldoesn’t meet the spec, which turns out tobe what’s important in practice. And, in practice, with themanylinux20XX approach, whenever distros change, we actually changethe spec; it takes a bit longer. So even if a wheel was complianttoday, it might be become non-compliant tomorrow. This is frustrating,but unfortunately this uncertainty is unavoidable if what you careabout is distributing working wheels to users.

So even on these points where the old approach initially seems to haveadvantages, we expect the new approach to actually do as well orbetter.

Switching to perennial tags, but continuing to write a PEP for eachversion: This was proposed as a kind of hybrid, to try to get someof the advantages of the perennial tagging system – like easierrollouts of new versions – while keeping the advantages of themanylinux20XX scheme, like forcing us to write documentation aboutLinux distros, simplifying options for package maintainers, and beingable to definitively tell when a wheel meets the spec. But asdiscussed above, on a closer look, it turns out that these advantagesare largely illusory. And this also inherits significantdisadvantages from the manylinux20XX scheme, like creatingindefinite obligations to update a growing list of copy-pasted PEPs.

Making auditwheel normative: Another possibility that wasconsidered was to make auditwheel the normative reference on thedefinition of manylinux, i.e., a wheel would be compliant if and onlyifauditwheelcheck completed without errors. This was rejectedbecause the point of packaging PEPs is to define interoperabilitybetween tools, not to bless specific tools.

Adding extra words to the tag string: Another proposal weconsidered was to add extra words to the wheel tag, e.g.manylinux_glibc_2_17 instead ofmanylinux_2_17. The motivationwould be to leave the door open to other kinds of versioningheuristics in the future – for example, we could havemanylinux_glibc_$VERSION andmanylinux_alpine_$VERSION.

But “manylinux” has always been a synonym for “broad compatibilitywith mainstream glibc-based distros”; reusing it for unrelated buildprofiles like alpine is more confusing than helpful. Also, some earlyreviewers who aren’t steeped in the details of packaging found thewordglibc actively misleading, jumping to the conclusion that itmeant they needed a system withexactly that glibc version. And tagslikemanylinux_$VERSION andalpine_$VERSION also have theadvantages of parsimony and directness. So we’ll go with that.