This PEP proposes a mechanism by which projects hosted onpypi.org can safely host wheel artifacts on external sites otherthan PyPI. This PEP explicitly doesnot propose external hosting ofprojects, packages, or their metadata. That functionality is already availableby externally hosting independent package indexes. Because this PEP onlyprovides a mechanism for projects to customize the download URL for specificreleased wheel artifacts, dependency resolution as already implemented bycommon installer tools such aspip anduv does not need to change.
This PEP defines what it means to be “safe” in this context, along with a newpackage upload file format called a.rim file. It defines how.rimfiles affect the metadata returned for a package’sSimple Repository APIin both HTML and JSON formats, and how traditional wheels can easily be turnedinto.rim files.
This PEP was withdrawn by the authors on 2025-01-31. Our reading of the sentiment in the discussionthread is that, while the problems this PEP attempts to solve are valid, most folks would prefer adifferent approach. Specifically, our read is that most users would prefer more control over theability to specify multiple indexes, how those indexes interoperate, and the priority and trustassertions for those indexes. For example, solutions such asPEP 766 may provide a better wayforward. Existing stop gap measures (e.g. PyPI limit increase requests and the“wheel-stub” approach) are sufficient – if not ideal – in themeantime. The authors wish to thank everyone who contributed to the constructive discussion, andespecially those who showed their support for this PEP, both in public and private.
The Python Package Index, hosted athttps://pypi.org, imposesdefault limits on upload artifact file size (100 MiB) and total project size(10 GiB). Most projects can comfortably fit within these limits during the lifetime of theproject, through years of uploads. A few projects have encountered these limits, and havebeen granted both file size and project size exceptions, allowing them to continueuploading new releases without having to take more drastic measures, such as removingfiles which may potentially still be in use by consumers (e.g. through version pins).
A related workaround is the“wheel stub”approach, which provides an indirect link between PyPI and an external third party packageindex, where such limitations can be avoided. Wheel stubs aresource distributions (a.k.a. “sdists”) which utilize aPEP 517 buildbackend that, instead of turning source code into a binary wheel, performs some logic tocalculate the URL for an existing, externally hosted wheel to download and install. Thisapproach works, but it obscures the connection between PyPI, the sdist, and the externallyhosted wheel, since there is no way to present this information topip or other suchtools.
In 2013,PEP 438 proposed a “backward-compatible two-phase transitionprocess” to modify several aspects of release file hosting on PyPI. As thisPEP describes, PyPI originally supported only project and releaseregistration without also allowing for artifact file hosting. As such, mostprojects hosted release file artifacts elsewhere. Artifact hosting was lateradded, but the mix of externally and PyPI-hosted files led to a wide range ofusability and potential security-related problems. PEP 438 was an attempt toprovide several facilities to allow external hosting while promoting aPyPI-first hosting preference.
PEP 438 was complex, with three different “hosting modes”,rel metadata inthe simple HTML index pages to signify hosting locations, and a two-phasetransition plan affecting PyPI and installer tools. PEP 438 was ultimatelyretracted in 2015 byPEP 470, which acknowledges that PEP 438 did succeedin…
bringing about more people to utilize PyPI’s repository features, analtogether good thing given the global CDN powering PyPI providing speedups for a lot of people[…]
Instead of external hosting, PEP 470 promoted the use of explicit multiplerepositories, providing full package indexing and artifact hosting, andenabled through installer tool support, such aspipinstall--extra-index-url allowingpip to essentially treat multiplerepositories asone single global repositoryfor package installation resolution. Because this has been the blessed normfor so many years, all Python package installation tools support queryingmultiple indexes for dependency resolution.
Why then does this PEP propose to allow a more limited form of externalhosting, and how does this proposal avoid the problems documented in PEP 470?
One well-known problem that consolidating multiple indexes enables isdependency confusion attacks, towhich Pythoncan be particularly vulnerable, due to the algorithm thatpipinstall uses for resolving package dependencies and preferred versions. Theuv tool addresses this by supporting an additionalindex strategy option,whereby users can select between, e.g. apip-compatible strategy, and amore limited strategy that prevents such dependency confusion attacks.
PEP 708 provides additional background about dependency confusion attacks,and takes a different approach to preventing them. At its core, PEP 708 allowsrepository owners to indicate that projects track across differentrepositories, which allows installers to determine how to treat the globalpackage namespace when combined across multiple repositories. PEP 708 has beenprovisionally accepted, pending several required conditions as outlined in PEP708, some of which may have an indeterminate future. As PEP 708 itself says,this won’t by itself solve dependency confusion attacks, but is one way toprovide enough information to installers to help minimize these attacks.
While there can still be valid use cases for standing up a totally independentpackage index (such as providing richer platform support for GPUs until afully formedvariant proposalis accepted), this PEP takes a different, simpler approach and doesn’t replaceany of the existing, proposed, or approved package index cooperationspecifications.
This PEP also preserves the core purpose of PyPI, and allows it toremain the traditional, canonical, centralized index of all Pythonpackages.
This proposal also addresses the problem of size limits imposed by PyPI, where there is adefault artifact size limit of 100 MiB and adefault overallproject size limit of 10GiB. Most packages and artifacts can easily fit in these limits, even for packagescontaining binary extension modules for a variety of platforms. A small, but importantclass of packages routinely exceed these limits, requiring them to submit PyPIexceptionrequest support tickets. It’s not necessarily difficult to get resolution on suchexceptions, but it is a special process that can take some time to resolve, and thecriteria for granting such exceptions aren’t well documented.
Setting up and maintaining an entire package index can be a complexoperational solution, both time and resource intensive. This is especiallytrue if the main purpose of such an index is just to avoid file sizelimitations. The external index approach also imposes a tricky UX on consumersof projects on the external index, requiring them to understand how CLIoptions such as--external-index-url work, along with the securityimplications of such flags. It would be much easier for both producers andconsumers of large wheel packages to just set up and maintain a simple webserver, capable of serving individual files with no more complex API thanHTTPGET. Such an interface is also easily cacheable or placed behind aCDN. Simple HTTPservers are also much easier to audit for security purposes, easier to proxy,and usually take much less resources to run, support, and maintain. Evensomething likeAmazon S3 could be used tohost external wheels.
This PEP proposes an approach that favors such operational simplicity.
A new type of uploadable file is defined, called a “RIM” (i.e..rim), or “RemoteInstallable Metadata” file. The name evokes the image of a wheel with the tire removed,and emphasizes that.rim files are easily derived from.whl files. The process ofturning a.whl into a.rim isoutlined below. The file nameformat exactly matches thewheel file naming format specification, except that RIM files use the suffix.rim. This means that all the tags used to discriminate.whl files alsodistinguish between different.rim files, and thus can be used during dependencyresolution steps, exactly as.whl files are today. In this respect,.whl and.rim files are interchangeable.
The content of a.rim file isnearly identical to.whl files, however.rimfilesMUST contain only the.dist-info directory from a wheel. No other top-levelfile or directory is allowed in the.rim zip file. The.dist-info directoryMUST contain a single additional file in addition to thoseallowed in a.whlfile’s.dist-info directory: a file calledEXTERNAL-HOSTING.json.
This is a JSON file contains containing the following keys:
version1.0.owneruri.whl file hosted on anexternal site. This URLMUST use thehttps scheme.size.whl file onthe remote host.hashes.whl file, with the sameconstraints as proposed in that PEP. Since these hashes are immutable once uploadedto PyPI, they serve as a critical validation that the externally hosted wheel hasn’tbeen corrupted or compromised.The only effect of a.rim file is to change the download URL for the wheel artifact inboth the HTML and JSON interfaces in thesimple repository API. In the HTML page for apackage release, thehref attributeMUST be the value of theuri key,including a#<hashname>=<hashvalue> fragment. this hash fragmentMUST be inexactly the same format as described thePEP 376 originatedsigned wheel file formatin the.dist-info/RECORD file. The exact same rules for selection of hash algorithmand encoding is used here.
Similarly in theJSON response theurl key pointing to the download file must bethe value of theuri key, and thehashes dictionaryMUST beincluded with values populated from thehashes dictionary provided above.
In all other respects, a compliant package index should treat.rim files the same as.whl files, with some other minor exceptions as outlined below. For example,.rimfiles can bedeleted and yanked (PEP 592) justlike any.whl file, with the exact same semantics (i.e. deletions are permanent). Whena.rim is deleted, an indexMUST NOT allow a matching.whl or.rim file tobe (re-)uploaded.
Externally hosted wheelsMUST be available before the corresponding.rim file isuploaded to PyPI, otherwise a publishing race condition is introduced, although thisrequirementMAY be relaxed for.rim files uploaded to aPEP 694 staged release.
IndexesMUST reject.rim files if a matching.whl file already exists with theexact same file name tags. However, indexesMAY accept a.whl file if a matching.rim file exists, as long as that.rim file hasn’t been deleted or yanked. Thisallows uploaders to replace an externally hosted wheel file with an index hosted wheelfile, but the converse is prohibited. Since the default is to host wheels on the samepackage index that contains the package metadata, it is not allowed to “downgrade” anexisting wheel file once uploaded. When a.whl replaces a.rim, the indexMUSTprovide download URLs for the package using its own hosted file service. When uploadingthe overriding.whl file, the package indexMUST validate the hash from theexisting.rim file, and these hashes must match or the overriding uploadMUST berejected.
It’s likely that the changes are backward compatible enough that a bump in thePyPIrepository version is not necessary. Since.rim files are essentially changes onlyto the upload API, package resolvers and package installers can continue to function withthe APIs they’ve always supported.
One of the key concerns leading to PEP 438’s revocation in PEP 470 waspotential user confusion when an external index disappeared. From PEP 470:
This confusion comes down to end users of projects not realizing if aproject is hosted on PyPI or if it relies on an external service. Thisoften manifests itself when the external service is down but PyPI isnot. People will see that PyPI works, and other projects works, but thisone specific one does not. They oftentimes do not realize who they need tocontact in order to get this fixed or what their remediation steps are.
While the problem of external wheel hosting service going down is not directlysolved by this PEP, several safeguards are in place to greatly reduce thepotential burden on PyPI administrators.
This PEP thus proposes that:
mailto URI for a contact email address, or the URL tothe organization’s support tracker. Such a contact URI is optional for organizationswhich do not avail themselves of external wheel file hosting.Combined with theEXTERNAL-HOSTING.json file’sowner key, this allows forinstaller tools to unambiguously redirect any download errors away from the PyPI supportadmins and squarely to the organization’s support admins.
While the exact mechanics of storing and retrieving this organization supportURL will be defined separately, for the sake of example, let’s say a packagefoo externally hosts wheel files on`https://foo.example.com<https://foo.example.com>`__ and that host becomes unreachable. When aninstaller tool tries to download and install the packagefoo wheel, thedownload step will fail. The installer would then be able to query PyPI toprovide a useful error message to the end user:
.rim file and reads theowner key from theEXTERNAL-HOSTING.json file inside the.rim zip file.The externally hosted wheel filefoo-....whlcould not bedownloaded. Please contactsupport@foo.example.com for help. Do not reportthis to the PyPI administrators.
It is generally very easy to produce a.rim file from an existing.whlfile. This could be done efficiently by aPEP 518 build backend with anadditional command line option, or a separate tool which takes a.whl fileas input and creates the associated.rim file. To complete the analogy,the act of turning a.whl into a.rim is called “dismounting”. Thesteps such a tool would take are:
.whl file, the organization owner of thepackage, and URL at which the.whl will be hosted, and the support URIto report download problems from. These could in fact be captured in thepyproject.toml file, but that specification is out of scope for thisPEP..whl and create the.rim zip archive..rim file any path in the.whl thatisn’t rootedat the.dist-info directory..whl file.EXTERNAL-HOSTING.json file containing the JSON keys and values as describedabove, to the.rim archive.Theoretically, installer tools shouldn’t need any changes, since when theyhave identified the wheel to download and install, they simply consult thedownload URLs returned by PyPI’s Simple API. In practice though, tools such aspip anduv may have constrained lists of hosts they will allowdownloads from, such as PyPI’s ownpythonhosted.org domain.
In this case, such tools will need to relax those constraints, but the exact policy forthis is left to the installer tools themselves. Any number of approaches could beimplemented, such as downloading the.rim file and verifying theEXTERNAL-HOSTING.json metadata, or simply trusting the external downloads for anywheel with a matching checksum. They could also query PyPI for the project’s organizationowner and support URI before trusting the download. They could warn the user whenexternally hosted wheel files are encountered, and/or require the use of a command lineoption to enable additional download hosts. Any of these verification policies could bechosen in configuration files.
Installer tools should also probably provide better error messages whenexternally hosted wheels cannot be downloaded, e.g. because a host isunreachable. As described above, such tools could query enough metadata fromPyPI to provide clear and distinct error messages pointing users to thepackage’s external hosting support email or issue tracker.
The following constraints lead to reliable and compatible external wheel hosting services:
pip 24.2 on Python 3.10 or newer uses the systemcertificate store in addition to the Mozilla store provided by the third partycertifi Python package.uv uses the Mozilla storeprovided by thewebpki-roots crate, but notthe system store unless the--native-tls flag is given[1].The PyPIadministrators may modify this requirement in the future, but compatibility with popularinstallers will not be compromised..rim file is deleted from PyPI first, andMUST NOT remove externalwheels for yanked releases.Several factors as described in this proposal should mitigate securityconcerns with externally hosted wheels, such as:
.rim files, and once uploaded cannot bechanged. Since the checksum stored on PyPI is immutable and required, it is not possibleto spoof an external wheel file, even if the owning organization lost control of theirhosting domain.When users identify malware or vulnerabilities in PyPI-hosted projects, they can nowreport this using themalware reporting facilities onPyPI, as also described in thisblog post. The same process can be used to reportsecurity issues in externally hosted wheels, and the same remediation process should beused. In addition, since organizations with external hosting enabled MUST provide asupport contact URI, that URI can be used in some cases to report the security issue tothe hosting organization. Such organization reporting won’t make sense for malware, butcould indeed be a very useful way to report security vulnerabilities in externally hostedwheels.
Several ideas were considered and rejected.
.rim file uploads. PyPIcould verify that the hash in theuploaded.rim file matches the externally hosted wheel before it accepts the upload,but this requires downloading the external wheel and performing the checksum, which alsoimplies that the upload of the.rim file cannot be accepted until this external.whl file is downloaded and verified. This increases PyPI bandwidth and slows downthe upload query, althoughPEP 694 draft uploads could potentially mitigate theseconcerns. Still, the benefit is not likely worth the additional complexity..whl file itself is stillavailable, e.g. via anHTTP HEAD request. This is likely overkill and without alsoproviding the file’s checksum in the response[2], may not provide much additionalbenefit..rim file replacement. While it is allowed for.whl files to replaceexisting.rim files, as long as a) the.rim file hasn’t been deletedor yanked, b) the checksums match, we do not allow replacing.whl fileswith.rim files, nor do we allow a.rim file to overwrite anexisting.rim file. This latter could be a technique to change thehosting URL for an externally hosted.whl; however, we do not think thisis a good idea. There are other ways to “fix” an external host URL asdescribed above, and we do not want to encourage mass re-uploads of existing.rim files.uv--native-tls flagreplacesthewebpki-roots store.This document is placed in the public domain or under theCC0-1.0-Universal license, whichever is more permissive.
Source:https://github.com/python/peps/blob/main/peps/pep-0759.rst
Last modified:2025-01-31 18:51:56 GMT