This PEP has beenprovisionally accepted,with the following required conditions before the PEP is made Final:
Dependency confusion attacks, in which a malicious package is installed insteadof the one the user expected, are anincreasingly common supply chain threat.Most such attacks against Python dependencies, including therecent PyTorch incident,occur with multiple package repositories, where a dependency expected to comefrom one repository (e.g. a custom index) is installed from another (e.g. PyPI).
To help address this problem, this PEP proposes extending theSimple Repository APIto allow repository operators to indicate that a project found on theirrepository “tracks” a project on different repositories, and allows projects toextend their namespaces across multiple repositories.
These features will allow installers to determine when a project being madeavailable from a particular mix of repositories is expected and should beallowed, and when it is not and should halt the install with an error to protectthe user.
There is a long-standing class of attacks that are called “dependency confusion”attacks, which roughly boil down to an individual user expected to get packageA, but instead they gotB. In Python, this almost always happens due tothe configuration of multiple repositories (possibly including the default ofPyPI), where they expected packageA to come from repositoryX, butsomeone is able to publish packageB to repositoryY under the samename.
Dependency Confusion attacks have long been possible, but they’ve recentlygained press withpublic examples of cases where these attacks were successfully executed.
A specific example of this is the recent case where the PyTorch project had aninternal package namedtorchtriton which was only ever intended to beinstalled from their repositories located athttps://download.pytorch.org/,but that repository was designed to be used in conjunction with PyPI, andthe name oftorchtriton was not claimed on PyPI, which allowed the attackerto use that name and publish a malicious version.
There are a number of ways to mitigate against these attacks today, but they allrequire that the end user go out of their way to protect themselves, rather thanbeing protected by default. This means that for the vast bulk of users, they arelikely to remain vulnerable, even if they are ultimately aware of these types ofattacks.
Ultimately the underlying cause of these attacks come from the fact that thereis no globally unique namespace that all Python package names come from.Instead, each repository is its own distinct namespace, and when given an“abstract” name such asspam to install, an installer has to implicitly turnthat into a “concrete” name such aspypi.org:spam orexample.com:spam.Currently the standard behavior in Python installation tools is to implicitlyflatten these multiple namespaces into one that contains the files from allnamespaces.
This assumption that collapsing the namespaces is what was expected means thatwhen packages with the same name in different repositoriesare authored by different parties (such as in thetorchtriton case)dependency confusion attacks become possible.
This is made particularly tricky in that there is no “right” answer; there arevalid use cases both for wanting two repositories merged into one namespaceand for wanting two repositories to be treated as distinct namespaces. Thismeans that an installer needs some mechanism by which to determine when itshould merge the namespaces of multiple repositories and when it should not,rather than a blanket always merge or never merge rule.
This functionality could be pushed directly to the end user, since ultimatelythe end user is the person whose expectations of what gets installed from whatrepository actually matters. However, by extending the repository specificationto allow a repository to indicate when it is safe, we can enable individualprojects and repositories to “work by default”, even when theirproject naturally spans multiple distinct namespaces, while maintaining theability for an installer to be secure by default.
On its own, this PEP does not solve dependency confusion attacks, but what itdoes do is provide enough information so that installers can prevent themwithout causing too much collateral damage to otherwise valid and safe usecases.
There are two broad use cases for merging names across repositories that thisPEP seeks to enable.
The first use case is when one repository is not defining its own names, butrather is extending names defined in other repositories. This commonly happensin cases where a project is being mirrored from one repository to another (seeBandersnatch) or when a repositoryis providing supplementary artifacts for a specific platform (seePiwheels).
In this case neither the repositories nor the projects that are being extendedmay have any knowledge that they are being extended or by whom, so this cannotrely on any information that isn’t present in the “extending” repository itself.
The second use case is when the project wants to publish to one “main”repository, but then have additional repositories that provide binaries foradditional platforms, GPUs, CPUs, etc. Currently wheel tags are not sufficientlyable to express these types of binary compatibility, so projects that wish torely on them are forced to set up multiple repositories and have their usersmanually configure them to get the correct binaries for their platform, GPU,CPU, etc.
This use case is similar to the first, but the important difference that makesit a distinct use case on its own is who is providing the information and whattheir level of trust is.
When a user configures a specific repository (or relies on the default) thereis no ambiguity as to what repository they mean. A repository is identified byan URL, and through the domain system, URLs are globally unique identifiers.This lack of ambiguity means that an installer can assume that the repositoryoperator is trustworthy and can trust metadata that they provide without needingto validate it.
On the flip side, given an installer finds a name in multiple repositories it isambiguous which of them the installer should trust. This ambiguity means that aninstaller cannot assume that the project owner on either repository istrustworthy and needs to validate that they are indeed the same project and thatone isn’t a dependency confusion attack.
Without some way for the installer to validate the metadata between multiplerepositories, projects would be forced into becoming repository operators tosafely support this use case. That wouldn’t be a particularly wrong choice tomake; however, there is a danger that if we don’t provide a way for repositoriesto let project owners express this relationship safely, they will beincentivized to let them use the repository operator’s metadata instead whichwould reintroduce the original insecurity.
This specification defines the changes in version 1.2 of the simple repositoryAPI, adding new two new metadata items: Repository “Tracks” and “AlternateLocations”.
To enable one repository to host a project that is intended to “extend” aproject that is hosted at other repositories, this PEP allows the extendingrepository to declare that a particular project “tracks” a project at anotherrepository or repositories by adding the URLs of the project and repositoriesthat it is extending.
This is exposed in JSON as the keymeta.tracks and in HTML as a meta elementnamedpypi:tracks on the project specific URLs, ($root/$project/).
There are a few key properties thatMUST be preserved when using thismetadata:
It isNOT required that every name in a repository tracks the samerepository, or that they all track a repository at all. Mixed use repositorieswhere some names track a repository and some names do not are explicitlyallowed.
{"meta":{"api-version":"1.2","tracks":["https://pypi.org/simple/holygrail/","https://test.pypi.org/simple/holygrail/"]},"name":"holygrail","files":[{"filename":"holygrail-1.0.tar.gz","url":"https://example.com/files/holygrail-1.0.tar.gz","hashes":{"sha256":"...","blake2b":"..."},"requires-python":">=3.7","yanked":"Had a vulnerability"},{"filename":"holygrail-1.0-py3-none-any.whl","url":"https://example.com/files/holygrail-1.0-py3-none-any.whl","hashes":{"sha256":"...","blake2b":"..."},"requires-python":">=3.7","dist-info-metadata":true}]}
<!DOCTYPE html><html><head><metaname="pypi:repository-version"content="1.2"><metaname="pypi:tracks"content="https://pypi.org/simple/holygrail/"><metaname="pypi:tracks"content="https://test.pypi.org/simple/holygrail/"></head><body><ahref="https://example.com/files/holygrail-1.0.tar.gz#sha256=..."><ahref="https://example.com/files/holygrail-1.0-py3-none-any.whl#sha256=..."></body></html>
To enable a project to extend its namespace across multiple repositories, thisPEP allows a project owner to declare a list of “alternate locations” for theirproject. This is exposed in JSON as the keyalternate-locations and in HTMLas a meta element namedpypi-alternate-locations, which may be used multipletimes.
There are a few key properties thatMUST be observed when using thismetadata:
https://pypi.org/simple/foo/ and it has analternate-locations metadata that has the value["https://example.com/simple/foo/"], then youMUST treat it as if ithad the value["https://example.com/simple/foo/","https://pypi.org/simple/foo/"].When an installer encounters a project that is using the alternate locationsmetadata itSHOULD consider that all repositories named are extending thesame namespace across multiple repositories.
Note
This alternate locations metadata is project level metadata, not artifactlevel metadata, which means it doesn’t get included as part of the coremetadata spec, but rather it is something that each repository will have toprovide a configuration option for (if they choose to support it).
{"meta":{"api-version":"1.2"},"name":"holygrail","alternate-locations":["https://pypi.org/simple/holygrail/","https://test.pypi.org/simple/holygrail/"],"files":[{"filename":"holygrail-1.0.tar.gz","url":"https://example.com/files/holygrail-1.0.tar.gz","hashes":{"sha256":"...","blake2b":"..."},"requires-python":">=3.7","yanked":"Had a vulnerability"},{"filename":"holygrail-1.0-py3-none-any.whl","url":"https://example.com/files/holygrail-1.0-py3-none-any.whl","hashes":{"sha256":"...","blake2b":"..."},"requires-python":">=3.7","dist-info-metadata":true}]}
<!DOCTYPE html><html><head><metaname="pypi:repository-version"content="1.2"><metaname="pypi:alternate-locations"content="https://pypi.org/simple/holygrail/"><metaname="pypi:alternate-locations"content="https://test.pypi.org/simple/holygrail/"></head><body><ahref="https://example.com/files/holygrail-1.0.tar.gz#sha256=..."><ahref="https://example.com/files/holygrail-1.0-py3-none-any.whl#sha256=..."></body></html>
This section is non-normative; it provides recommendations to installers in howto interpret this metadata that this PEP feels provides the best tradeoffbetween protecting users by default and minimizing breakages to existingworkflows. These recommendations are not binding, and installers are free toignore them, or apply them selectively as they make sense in their specificsituations.
Note
This algorithm is written based on how pip currently discovers files;other installers may adapt this based on their own discovery procedures.
Currently the “standard” file discovery algorithm looks something like this:
It is recommended that installers change their file discovery algorithm to takeinto account the new metadata, and instead do:
This is somewhat subtle, but the key things in the recommendation are:
This algorithm ensures that an installer never assumes that two disparatenamespaces can be flattened into one, which for all practical purposeseliminates the possibility of any kind of dependency confusion attack, whilestill giving power throughout the stack in a safe way to allow people toexplicitly declare when those disparate namespaces are actually one logicalnamespace that can be safely merged.
The above algorithm is mostly a conceptual model. In reality the algorithm mayend up being slightly different in order to be more privacy preserving andfaster, or even just adapted to fit a specific installer better.
This PEP avoids dictating or recommending a specific mechanism by which aninstaller allows an end user to configure exactly what repositories they want aspecific package to be installed from. However, it does recommend thatinstallers do providesome mechanism for end users to provide thatconfiguration, as without it users can end up in a DoS situation in casesliketorchtriton where they’re just completely broken unless they resolvethe namespace collision externally (get the name taken down on one repository,stand up a personal repository that handles the merging, etc).
This configuration also allows end users to pre-emptively secure themselvesduring what is likely to be a long transition until the default behavior issafe.
Note
This example is pip specific and assumes specifics about how pip willchoose to implement this PEP; it’s included as an example of how we cancommunicate this change, and not intended to constrain pip or any otherinstaller in how they implement this. This may ultimately be the actual basisfor communication, and if so will need be edited for accuracy and clarity.
This section should be read as if it were an entire “post” to communicate thischange that could be used for a blog post, email, or discourse post.
There’s a long-standing class of attacks that are called “dependency confusion”attacks, which roughly boil down to an individual expected to get packageA,but instead they gotB. In Python, this almost always happens due to the enduser having configured multiple repositories, where they expect packageA tocome from repositoryX, but someone is able to publish packageB withthe same name as packageA in repositoryY.
There are a number of ways to mitigate against these attacks today, but they allrequire that the end user explicitly go out of their way to protect themselves,rather than it being inherently safe.
In an effort to secure pip’s users and protect them from these types of attacks,we will be changing how pip discovers packages to install.
When pip discovers that the same project is available from multiple remoterepositories, by default it will generate an error and refuse to proceed ratherthan make a guess about which repository was the correct one to install from.
Projects that natively publish to multiple repositories will be given theability to safely “link” their repositories together so that pip does not errorwhen those repositories are used together.
End users of pip will be given the ability to explicitly define one or morerepositories that are valid for a specific project, causing pip to only considerthose repositories for that project, and avoiding generating an erroraltogether.
See TBD for more information.
Users who are installing from multiple remote (e.g. not present on the localfilesystem) repositories may be affected by having pip error instead ofsuccessfully install if:
Users who are not using multiple remote repositories will not be affected atall, which includes users who are only using a single remote repository, plus alocal filesystem “wheel house”.
If you’re using only a single remote repository you do not have to do anything.
If you’re using multiple remote repositories, you can opt into the new behaviorby adding--use-feature=TBD to your pip invocation to see if any of yourdependencies are being served from multiple remote repositories. If they are,you should audit them to determine why they are, and what the best remediationstep will be for you.
Once this behavior becomes the default, you can opt out of it temporarily byadding--use-deprecated=TBD to your pip invocation.
If you’re using projects that are not hosted on a public repository, but youstill have the public repository as a fallback, consider configuring pip with arepository file to be explicit where that dependency is meant to come from toprevent registration of that name in a public repository to cause pip to errorfor you.
If you only publish your project to a single repository, then you do not have todo anything.
If you publish your project to multiple repositories that are intended to beused together at the same time, configure all repositories to serve thealternate repository metadata to prevent breakages for your end users.
If you publish your project to a single repository, but it is commonly used inconjunction with other repositories, consider preemptively registering yournames with those repositories to prevent a third party from being able to causeyour userspipinstall invocations to start failing. This may not beavailable if your project name is too generic or if the repositories havepolicies that prevent defensive name squatting.
You’ll need to decide how you intend for your repository to be used by your endusers and how you want them to use it.
For private repositories that host private projects, it is recommended that youmirror the public projects that your users depend on into your own repository,taking care not to let a public project merge with a private project, and tellyour users to use the--index-url option to use only your repository.
For public repositories that host public projects, you should implement thealternate repository mechanism and enable the owners of those projects toconfigure the list of repositories that their project is available from if theymake it available from more than one repository.
For public repositories that “track” another repository, but providesupplemental artifacts such as wheels built for a specific platform, you shouldimplement the “tracks” metadata for your repository. However, this informationMUST NOT be settable by end users who are publishing projects to yourrepository. See TBD for more information.
Note: Some of these are somewhat specific to pip, but any solution that doesn’twork for pip isn’t a particularly useful solution.
If every repository returns the exact same list of files, then it is safe toconsider those repositories to be the same namespace and implicitly merge them.This would possibly mean that mirrors would be automatically allowed without anywork on any user or repository operator’s part.
Unfortunately, this has two failings that make it undesirable:
Providing some mechanism to give the repositories an order, and then shortcircuiting the discovery algorithm when it finds the first repository thatprovides files for that project is another workable solution that is safe if theorder is specified correctly.
However, this has been rejected for a number of reasons:
There is a variant of this idea which effectively says that it’s really justPyPI’s nature of open registration that causes the real problems, so if we treatall repositories but the “default” one as equal priority, and then treat thedefault one as a lower priority then we’ll fix things.
That is true in that it does improve things, but it has many of the sameproblems as the general ordering idea (though not all of them).
It also assumes that PyPI, or whatever repository is configured as the“default”, is the only repository with open registration of names.However, projects likePiwheels existwhich users are expected to use in addition to PyPI,which also effectively have open registration of namessince it tracks whatever names are registered on PyPI.
One possible solution is to instead of having the installer have to solve this,to instead depend on repository proxies that can intelligently merge multiplerepositories safely. This could provide a better experience for people withcomplex needs because they can have configuration and features that arededicated to the problem space.
However, that has been rejected because:
Another possible solution is to rely on hash checking, since with hash checkingenabled users cannot get an artifact that they didn’t expect; it doesn’t matterif the namespaces are incorrectly merged or not.
This is certainly a solution; unfortunately it also suffers from problems thatmake it unworkable:
requirements.txt file as the source of their dependencies (this affectsbuild time dependencies, and dependencies provided at the command line).Another idea is that we can narrow the scope of--extra-index-url such thatits only supported use is to refer to supplemental repositories to the defaultrepository, effectively saying that the default repository defines thenamespace, and every additional repository just extends it with extra packages.
The implementation of this would roughly be to require that the projectMUSTbe registered with the default repository in order for any additionalrepositories to work.
This sort of works if you successfully narrow the scope in that way, butultimately it has been rejected because:
The main reason this problem exists is that we don’t have globally unique names,we have locally unique names that exist under multiple namespaces that we areattempting to merge into a single flat namespace. If we could instead come upwith a way to have globally unique names, we could sidestep the entire issue.
This idea has been rejected because:
One idea that has come up is to essentially just implement the explicitconfiguration and don’t make any other changes to anything else. The specificproposal for a mapping policy is what actually inspired the explicitconfiguration option, and created a file that looked something like:
{"repositories":{"PyTorch":["https://download.pytorch.org/whl/nightly"],"PyPI":["https://pypi.org/simple"]},"mapping":[{"paths":["torch*"],"repositories":["PyTorch"],"terminating":true},{"paths":["*"],"repositories":["PyPI"]}]}
The recommendation to have explicit configuration pushes the decision on how toimplement that onto each installer, allowing them to choose what works best fortheir users.
Ultimately only implementing some kind of explicit configuration was rejectedbecause by its nature it’s opt in, so it doesn’t protect average users who areleast capable to solve the problem with the existing tools; by adding additionalprotections alongside the explicit configuration, we are able to protect allusers by default.
Additionally, relying on only explicit configuration also means that every enduser has to resolve the same problem over and over again, even in cases likemirrors of PyPI, Piwheels, PyTorch, etc. In each and every case they have to sitthere and make decisions (or find some example to cargo cult) in order to besecure. Adding extra features into the mix allows us to centralize thoseprotections where we can, while still giving advanced end users the ability tocompletely control their own destiny.
There’s been some suggestion thatscopes similar to how npm has implemented themmay ultimately solve this. Ultimately scopes do not change anything about thisproblem. As far as I know scopes in npm are not globally unique, they’re tied toa specific registry just like unscoped names are. However what scopes do enableis an obvious mechanism for grouping related projects and the ability for a useror organization on npm.org to claim an entire scope, which makes explicitconfiguration significantly easier to handle because you can be assured thatthere’s a whole little slice of the namespace that wholly belongs to you, andyou can easily write a rule that assigns an entire scope to a specific nonpublic registry.
Unfortunately, it basically ends up being an easier version of the idea to onlyuse explicit configuration, which works ok in npm because its not particularlycommon for people to use their own registries, but in Python we encourage you todo just that.
This PEP recommends installers to have a mechanism for explicit configuration ofwhich repository a particular project comes from, but it does not define whatthat mechanism is. We are purposefully leave that undefined, as it is closelytied to the UX of each individual installer and we want to allow each individualinstaller the ability to expose that configuration in whatever way that they seefit for their particular use cases.
Further, when the idea of defining that mechanism came up, none of the otherinstallers seemed particularly interested in having that mechanism defined forthem, suggesting that they were happy to treat that as part of their UX.
Finally, that mechanism, if we did choose to define it, deserves it’s own PEPrather than baking it as part of the changes to the repository API in this PEPand it can be a future PEP if we ultimately decide we do want to go down thepath of standardization for it.
Thanks to Trishank Kuppusamy for kick starting the discussion that lead to thisPEP with hisproposal.
Thanks to Paul Moore, Pradyun Gedam, Steve Dower, and Trishank Kuppusamy forproviding early feedback and discussion on the ideas in this PEP.
Thanks to Jelle Zijlstra, C.A.M. Gerlach, Hugo van Kemenade, and Stefano Riverafor copy editing and improving the structure and quality of this PEP.
This document is placed in the public domain or under theCC0-1.0-Universal license, whichever is more permissive.
Source:https://github.com/python/peps/blob/main/peps/pep-0708.rst
Last modified:2025-02-01 08:55:40 GMT