The currentwheel 1.0 specification was written over a decade ago,and has been extremely robust to changes in the Python packaging ecosystem.Previous efforts to improve the wheel specificationwere deferred to focus on other packagingspecifications. Meanwhile, the use of wheels has changed dramatically in thelast decade. There have been many requests for new wheel features over theyears; however, a fundamental obstacle to evolving the wheel specification hasbeen that there is no defined process for how to handle addingbackwards-incompatible features to wheels. Therefore, to enable other PEPs todescribe new enhancements to the wheel specification,this PEP prescribescompatibility requirements on future wheel revisions. This PEP doesnotspecify a new wheel revision. The specification of a new wheel format(“Wheel 2.0”) is left to a future PEP.
Currently, wheel specification changes that require new installer behavior are backwards incompatible and require a major version increase inthe wheel metadata format. An increase of the wheel major version has yet tohappen, partially because such a change has the potential to becatastrophically disruptive. Perthe wheel specification,any installer that does not support the new major version must abort at installtime. This means that if the major version were to be incremented withoutfurther planning, many users would see installation failures as older installers reject new wheelsuploaded to public package indices like the Python Package Index (PyPI). It iscritically important to carefully plan the interactions between build tools,package indices, and package installers to avoid incompatibility issues,especially considering the long tail of users who are slow to update theirinstallers.
The backward compatibility concerns have prevented valuable improvementsto the wheel file format, such asbetter compression,wheel data format improvements,better information about what is included in a wheel,andJSON formatted metadata in the “.dist-info” folder.
This PEP describes constraints and behavior for new wheel revisions to preservestability for existing tools that do not support a new major version of the wheel format.This ensures that backwards incompatible changes to the wheel specificationwill only affect users and tools that are properly set up to use the newerwheels. With a clear path for evolving the wheel specification, future PEPswill be able to improve the wheel format without needing to re-define acompletely new compatibility story.
Currently, thewheel 1.0 PEP, PEP 427, specifies that wheel filesmust contain aWHEEL metadata file that contains the version of the wheelspecification that the file conforms to. PEP 427 stipulates that installersMUST warn on installation of a wheel with a minor version greater than supported,and MUST abort on installation of wheels with a major version that is greater thanwhat the installer supports. This ensures that users do not get invalidinstallations from wheels that installers cannot properly install.
However, resolvers do not currently exclude wheels with an incompatible wheelversion. There is also currently no way for a resolver to check a wheel’sversion without downloading the wheel directly. To make wheel version filteringeasy for resolvers, the wheel versionMUST be included in the relevantmetadata file (currentlyMETADATA). This will allow resolvers toefficiently check the wheel version using thePEP 658 metadata API withoutneeding to download and inspect the.dist-info/WHEEL file.
To accomplish this, a new field,Wheel-Version, will be added to theCore Metadata Specification.This field is single use, and must contain the exact same version specified astheWheel-Version entry in theWHEEL file, or any future replacementfile defining metadata about the wheel file. IfWheel-Version is absentfrom the metadata file, then toolsMUST infer the wheel file majorversion as 1.
Wheel-VersionMUST NOT be included in source distribution metadata(PKG-INFO) files. If a tool encountersWheel-Version inside of a sourcedistribution metadata file, itSHOULD raise an error.
Wheel-VersionMAY be included in the metadata file for wheels ofversion 1, but for wheels of version 2 or higher, the metadata fileMUSTincludeWheel-Version. This enforces that future revisions of the wheelspecification can rely on resolvers skipping incompatible wheels by checkingtheWheel-Version field. Build backends are encouraged to includeWheel-Version in all wheels that they generate, regardless of version.
InstallersSHOULD copy the metadata file in a wheel unmodified duringinstallation. This prevents the need to update theRECORD file, which isan error prone process. Tools reading installed core metadataSHOULD NOTassume that the field is present, as other installation formats may omit it.
When installing a wheel, installersMUST take the following steps:
Wheel-Version in both the core metadata fileand wheel metadata file match. If they do not match, installersMUSTabort installation. Neither value takes precedence.Wheel-Version. IfWheel-Version is absent, assume the version is 1.0. Warn if minorversion is greater, abort if major version is greater. This procedure isidentital to that inPEP 427.Wheel-VersionResolvers, in the process of selecting a wheel to install,MUST check acandidate wheel’sWheel-Version, and ignore incompatible wheel files.Without ignoring these files, older installers might select a wheel that usesan unsupported wheel version for that installer, and force the installer toabort perPEP 427. By skipping incompatible wheel files, users will not seeinstallation errors when a project adopts a new wheel major version. As alreadyspecified in PEP 427, installersMUST abort if a user tries to directlyinstall a wheel that is incompatible. If, in the process of resolving packagesfound in multiple indices, a resolver comes across two wheels of the samedistribution and version, resolvers should prioritize the wheel of the highestcompatible version.
While the above protects users from unexpected breakages, users may miss a newrelease of a distribution if their installer does not support the wheel versionused in the release. Imagine in the future that a package publishes 3.0 wheelfiles. Downstream users won’t see that there is a new release available iftheir installers only support 2.x wheels. Therefore, installersSHOULD emita warning if, in the process of resolving packages, they come across an incompatible wheeland skip it.
Unfortunately, existing resolvers do not check the compatibility of wheelsbefore selecting them as installation candidates. Until a majority of usersupdate to installers that properly check for wheel compatibility, it is unsafeto allow publishing wheels of a new major version that existing resolvers mightselect. It could take upwards of four years before the majority of users are onupdated resolvers, based on current data about PyPI installer usage (See theAppendix: Analysis of Installer Usage on PyPI, fordetails). To allow for experimentation and faster adoption of 2.0 wheels,this PEP proposes a change to the file extension of thewheel file format, from.whl to.whlx for all future wheel versions.Note thatx inwhlx is the letter “x” and does not specify the wheelmajor version. The change to extension name resolves the initial transitionissue of 2.0 wheels breaking users on existing installers that do not implementWheel-Version checks. By using a different file extension, 2.0 wheels canimmediately be uploaded to PyPI, and users will be able to experiment with thenew features right away. Users on older installers will simply ignore these newfiles.
One rejected alternative would be to keep the.whl extension, but delay thepublishing of wheel 2.0 to PyPI. For more on that, please see Rejected Ideas.
Build backends are recommended to generate the most compatible wheel based onfeatures a project uses. For example, if a wheel does not use symbolic links,and such a feature was introduced in wheel 5.0, the build backend couldgenerate a wheel of version 4.0. On the other hand, some features will want tobe adopted by default. For example, if wheel 3.0 introduces better compression,the build backend may wish to enable this feature by default to improve thewheel size and download performance.
While it is difficult to know what future features may be planned for the wheelformat, it is important that certain compatibility promises are maintained.
Wheel files, when installed,MUST stay compatible with the Python standardlibrary’simportlib.metadata for all supported CPython versions. Forexample, replacing.dist-info/METADATA with a JSON formatted metadata fileMUST be a multi-major version migration with one version introducing the newJSON file alongside the existing email header format, and another futureversion removing the email header format metadata file. The version to remove.dist-info/METADATA alsoMUST be adopted only after the last CPythonrelease that lacked support for the new file reaches end of life. This ensuresthat code usingimportlib.metadata will not break with wheel major versionrevisions.
Wheel filesMUST remain ZIP format files as the outer container format.Additionally, the.dist-info metadata directoryMUST be placed at theroot of the archive without any compression, so that unpacking the wheel fileproduces a normal.dist-info directory holding any metadata for the wheel.Future wheel revisionsMAY modify the layout, compression, and otherattributes about non-metadata components of a wheel such as data and code. Thisassures that future wheel revisions remain compatible with tools operating onpackage metadata, while allowing for improvements to code storage in the wheel,such as adopting compression.
Package toolingMUST NOT assume that the contents and format of the wheelfile will remain the same for future wheel major versions beyond thelimitations above about metadata folder contents and outer container format.For example, newer wheel major versions may add or remove filename components,such as the build tag or the platform tag. Therefore it is incumbent upontooling to check the metadata for theWheel-Version before attempting toinstall a wheel.
Finally, future wheel revisionsMUST NOT use any compression formats not inthe CPython standard library of at least the latest release. Wheels generatedusing any new compression format should be tagged as requiring at least thefirst released version of CPython to support the new compression format,regardless of the Python API compatibility of the code within the wheel.
Backwards compatibility is an incredibly important issue for evolving the wheelformat. If adopting a new wheel revision is painful for downstream users,package creators will hesitate to adopt the new standards, and users will bestuck with failed CI pipelines and other installation woes.
Several choices in the above specification are made so that the adoption of anew feature is less painful. For example, today wheels of an incompatible majorversion are still selected by pip as installation candidates, which causesinstaller failures if a project starts publishing 2.0 wheels. To avoid thisissue, this PEP requires resolvers to filter out wheels with major versions orfeatures incompatible with the installer.
This PEP also defines constraints on future wheel revisions, with the goal ofmaintaining compatibility with CPython, but allowing evolution of wheelcontents. Wheel revisions shouldn’t cause package installations to break onolder CPython revisions, as not only would it be frustrating, it would beincredibly hard to debug for users.
This PEP relies on resolvers being able to efficiently acquire packagemetadata, usually throughPEP 658. This might present a problem for users ofpackage indices that do not servePEP 658 metadata. However, today mostinstallers fall back on using HTTP range requests to efficiently acquire onlythe part of a wheel needed to read the metadata, a feature most storageproviders and servers include. Furthermore, future improvements to wheelssuch as compression will make up performance losses due to inspecting filesin the wheel.
The main compatibility limitation of this PEP is for projects that startpublishing solely new wheels alongside a source distribution. If a user on anolder installer tries to install the package, it will fall back to the sourcedistribution, because the resolver will skip all newer wheels. Users are oftenpoorly set up to build projects from source, so this could lead to some failedbuilds users would not see otherwise. There are several approaches to resolvingthis issue, such as allowing dual-publishing for the initial migration, ormarking source distributions as not intended to be built.
The wheel format has been around for over 10 years, and in that time, Pythonpackages have changed a lot. It is much more common for packages to includeRust or C extension modules, increasing the size of packages. Bettercompression, such as lzma or zstd, could save a lot of time and bandwidth forPyPI and its users. Compatibility tags cannot express the wide variety ofhardware used to accelerate Python code today, nor encode shared librarycompatibility information. In order to address these issues, evolution of thewheel package format is necessary.
I do not believe that tying wheel revisions to CPythonreleases is beneficial. The main benefit of doing so is to make adoption of newwheels predictable - users with the latest CPython get the latest packageformat! This choice has several issues however. First, tying the new formatto the latest CPython makes adoption much slower. Users on LTS versions ofLinux with older Python installations are free to update their pip in a virtualenvironment, but cannot update the version of Python as easily. While somechanges to the wheel format must be tied to CPython changes necessarily, suchas adding new compression formats or changing the metadata format, many changesdo not need to be tied to the Python version, such as symlinks, enhancedcompatibility tags, and new formats that use existing compression formats inthe standard library. Additionally, wheels are used across multiple differentlanguage implementations, which lag behind the CPython version. It seems unfairto prevent their users from using a feature due to the Python version. Lastly,while this PEP does not suggest tying the wheel version to CPython releases, afuture PEP may still do so at any time, so this choice does not need to be madein this PEP.
.whl as the File ExtensionWhile keeping the extension.whl is appealing for many reasons, it presentsseveral problems that are difficult to surmount. First, current installerswould still pick a new wheel and fail to install the package. Furthermore,the file name of a wheel would not be able to change without breaking existinginstallers that expect a set wheel file name format. While the current filenamespecification for wheels is sufficient for current usage, the optionalbuild tag in the middle of the file name makes any extensions ambiguous (i.e.foo-0.3-py3-none-any-fancy_new_tag.whl would parse as the build tag beingpy3). This limits changes to information stored in the wheel file name.
.whl2)Storing the wheel major version in the file extension has several niceadvantages. For one, there is no need to introduce theWheel-Versionmetadata field, since installers could simply filter based on file extension.This would also allow future side-by-side packages. However, changing theextension for wheels each major version has some downsides. First, the versionstored in theWHEEL file must match the file extension, and this would needto be verified by installers. Additionally, many systems associate file type byfile extension (e.g. executable associations, various web caching software),and these would need to be updated every version that is released. Furthermore,part of the brittleness of the current wheel specification is that so muchmetadata is stored in the filename. Filenames are not well suited to storestructured data. Moving away from encoding information in the filename shouldbe a goal of future wheel revisions.
Another possibility is to use the file extension to encode the outer containerformat (i.e. a ZIP file containing.dist-info) separate from the innerwheel version. However, this could lead to confusion if the file extension andinnerWheel-Version diverge. If an installer raises an error due to anincompatible wheel 3.0 as obtained from the wheel metadata, some users willbe confused by the difference from the file extension.whl2.
Since wheel 2.0 will change the extension of wheel files, it is the bestopportunity to modify the outer container format. Compatibility does not needto be kept with a different file extension that tools will need to opt-in toreading. The main use-case for a different exterior compression format wouldbe better compression. For example, the outer container could be changed intoaZstandard tarfile,.tar.zst, whichwould decompress faster and produce smaller wheels. However, there are severalpractical issues with this. First, Zstandard is not part of the Python standardlibrary, so pure-Python packaging tools would need to ship an extension tounpack these wheels. This could cause some compatibility issues for severalplatforms where extension modules are not easy to install. Furthermore, afuture wheel revision could always introduce a new layout of non-metadata filesthat uses a.tar.zst inside the existing ZIP-based format.
Finally, it is not a good idea to change the wheel file format too much atonce. The goal of this PEP is to make evolving the specification easier, andpart of the rationale behind making wheel evolution easier is to avoid “allat once” changes. Changing the outer file format for wheels would requirere-writing how package metadata is not only discovered, but also installed.
There aremany features that could be included as part of wheel 2.0, but thisPEP does not cover them. The goal of this PEP is to define a compatibilitystory for the wheel file format. Changes that do not pertain to compatibilityfor wheel versions do not need to be in this PEP, and should be introductedin follow-up PEPs defining new wheel features.
Since.whl and.whlx will look different in file name, they could beuploaded side-by-side to package indices like PyPI. This has some nicebenefits, like dual-support for older and newer installers, so users who canget the latest features, while users who don’t upgrade still can install thelatest version of a package.
There are many complications however. Should we allow wheel 2 uploads toexisting wheel 1-only releases? Should we put any requirements on theside-by-side wheels, such as:
Constraints on dual-published wheels
A given index may contain identical-content wheels with different wheelversions, and installers should prefer the newest-available wheel format,with all other factors held constant.
Should we only allow uploading both withPEP 694 allowing “atomic”dual-publishing?
The author of this PEP is greatly indebted to the incredibly valuable review,advice, and feedback of Barry Warsaw and Michael Sarahan.
This document is placed in the public domain or under theCC0-1.0-Universal license, whichever is more permissive.
Source:https://github.com/python/peps/blob/main/peps/pep-0777.rst
Last modified:2025-02-01 07:28:42 GMT