Versioning

Versioning will adhere toPEP 629 format (Major.Minor), which has defined theexisting HTML responses to be1.0. Since this PEP does not introduce new featuresinto the API, rather it describes a different serialization format for the existingfeatures, this PEP does not change the existing1.0 version, and instead justdescribes how to serialize that into JSON.

Similar toPEP 629, the major version numberMUST be incremented if anychanges to the new format would result in no longer being able to expect existingclients to meaningfully understand the format.

Likewise, the minor versionMUST be incremented if features areadded or removed from the format, but existing clients would be expected to continueto meaningfully understand the format.

Changes that would not result in existing clients being unable to meaningfullyunderstand the format and which do not represent features being added or removedmay occur without changing the version number.

This is intentionally vague, as this PEP believes it is best left up to future PEPsthat make any changes to the API to investigate and decide whether or not thatchange should increment the major or minor version.

Future versions of the API may add things that can only be represented in a subsetof the available serializations of that version. All serializations version numbers,within a major version,SHOULD be kept in sync, but the specifics of how afeature serializes into each format may differ, including whether or not that featureis present at all.

It is the intent of this PEP that the API should be thought of as URL endpoints thatreturn data, whose interpretation is defined by the version of that data, and thenserialized into the target serialization format.

JSON Serialization

The URL structure fromPEP 503 still applies, as this PEP only adds an additionalserialization format for the already existing API.

The following constraints apply to all JSON serialized responses described in thisPEP:

• All JSON responses willalways be a JSON object rather than an array or othertype.
• While JSON doesn’t natively support an URL type, any value that represents anURL in this API may be either absolute or relative as long as they point tothe correct location. If relative, they are relative to the current URL as ifit were HTML.
• Additional keys may be added to any dictionary objects in the API responsesand clientsMUST ignore keys that they don’t understand.
• All JSON responses will have ameta key, which contains information related tothe response itself, rather than the content of the response.
• All JSON responses will have ameta.api-version key, which will be a string thatcontains thePEP 629Major.Minor version number, with the same fail/warnsemantics as defined inPEP 629.
• All requirements ofPEP 503 that are not HTML specific still apply.

{"meta":{"api-version":"1.0"},"projects":[{"name":"Frob"},{"name":"spamspamspam"}]}

{"meta":{"api-version":"1.0"},"name":"holygrail","files":[{"filename":"holygrail-1.0.tar.gz","url":"https://example.com/files/holygrail-1.0.tar.gz","hashes":{"sha256":"...","blake2b":"..."},"requires-python":">=3.7","yanked":"Had a vulnerability"},{"filename":"holygrail-1.0-py3-none-any.whl","url":"https://example.com/files/holygrail-1.0-py3-none-any.whl","hashes":{"sha256":"...","blake2b":"..."},"requires-python":">=3.7","dist-info-metadata":true}]}

Content-Types

This PEP proposes that all responses from the Simple API will have a standardcontent type that describes what the response is (a Simple API response), whatversion of the API it represents, and what serialization format has been used.

The structure of this content type will be:

application/vnd.pypi.simple.$version+format

Since only major versions should be disruptive to clients attempting tounderstand one of these API responses, only the major version will be includedin the content type, and will be prefixed with av to clarify that it is aversion number.

Which means that for the existing 1.0 API, the content types would be:

• JSON:application/vnd.pypi.simple.v1+json
• HTML:application/vnd.pypi.simple.v1+html

In addition to the above, a special “meta” version is supported namedlatest,whose purpose is to allow clients to request the absolute latest version, withouthaving to know ahead of time what that version is. It is recommended however,that clients be explicit about what versions they support.

To support existing clients which expect the existingPEP 503 API responses touse thetext/html content type, this PEP further definestext/html as an aliasfor theapplication/vnd.pypi.simple.v1+html content type.

Version + Format Selection

Now that there is multiple possible serializations, we need a mechanism to allowclients to indicate what serialization formats they’re able to understand. Inaddition, it would be beneficial if any possible new major version to the API canbe added without disrupting existing clients expecting the previous API version.

To enable this, this PEP standardizes on the use of HTTP’sServer-Driven Content Negotiation.

While this PEP won’t fully describe the entirety of server-driven contentnegotiation, the flow is roughly:

1. The client makes an HTTP request containing anAccept header listing allof the version+format content types that they are able to understand.
2. The server inspects that header, selects one of the listed content types,then returns a response using that content type (treating the absence ofanAccept header asAccept:*/*).
3. If the server does not support any of the content types in theAcceptheader then they are able to choose between 3 different options for how torespond:
   1. Select a default content type other than what the client has requestedand return a response with that.
   2. Return a HTTP406NotAcceptable response to indicate that none ofthe requested content types were available, and the server was unableor unwilling to select a default content type to respond with.
   3. Return a HTTP300MultipleChoices response that contains a list ofall of the possible responses that could have been chosen.
4. The client interprets the response, handling the different types of responsesthat the server may have responded with.

This PEP does not specify which choices the server makes in regards to handlinga content type that it isn’t able to return, and clientsSHOULD be preparedto handle all of the possible responses in whatever way makes the most sense forthat client.

However, as there is no standard format for how a300MultipleChoicesresponse can be interpreted, this PEP highly discourages servers from utilizingthat option, as clients will have no way to understand and select a differentcontent-type to request. In addition, it’s unlikely that the clientcouldunderstand a different content type anyways, so at best this response wouldlikely just be treated the same as a406NotAcceptable error.

This PEPdoes require that if the meta versionlatest is being used, theserverMUST respond with the content type for the actual version that iscontained in the response(i.e. AAccept:application/vnd.pypi.simple.latest+json request that returnsav1.x response should have aContent-Type ofapplication/vnd.pypi.simple.v1+json).

TheAccept header is a comma separated list of content types that the clientunderstands and is able to process. It supports three different formats for eachcontent type that is being requested:

• $type/$subtype
• $type/*
• */*

For the use of selecting a version+format, the most useful of these is$type/$subtype, as that is the only way to actually specify the versionand format you want.

The order of the content types listed in theAccept header does not have anyspecific meaning, and the serverSHOULD consider all of them to be equallyvalid to respond with. If a client wishes to specify that they prefer a specificcontent type over another, they may use theAccept header’squality valuesyntax.

This allows a client to specify a priority for a specific entry in theirAccept header, by appending a;q= followed by a value between0 and1 inclusive, with up to 3 decimal digits. When interpreting this value,an entry with a higher quality has priority over an entry with a lower quality,and any entry without a quality present will default to a quality of1.

However, clients should keep in mind that a server is free to selectany ofthe content types they’ve asked for, regardless of their requested priority, andit may even return a content type that they didnot ask for.

To aid clients in determining the content type of the response that they havereceived from an API request, this PEP requires that servers always include aContent-Type header indicating the content type of the response. This istechnically a backwards incompatible change, however in practicepip has been enforcing this requirementso the risks for actual breakages is low.

An example of how a client can operate would look like:

importemail.messageimportrequestsdefparse_content_type(header:str)->str:m=email.message.Message()m["content-type"]=headerreturnm.get_content_type()# Construct our list of acceptable content types, we want to prefer# that we get a v1 response serialized using JSON, however we also# can support a v1 response serialized using HTML. For compatibility# we also request text/html, but we prefer it least of all since we# don't know if it's actually a Simple API response, or just some# random HTML page that we've gotten due to a misconfiguration.CONTENT_TYPES=["application/vnd.pypi.simple.v1+json","application/vnd.pypi.simple.v1+html;q=0.2","text/html;q=0.01",# For legacy compatibility]ACCEPT=", ".join(CONTENT_TYPES)# Actually make our request to the API, requesting all of the content# types that we find acceptable, and letting the server select one of# them out of the list.resp=requests.get("https://pypi.org/simple/",headers={"Accept":ACCEPT})# If the server does not support any of the content types you requested,# AND it has chosen to return a HTTP 406 error instead of a default# response then this will raise an exception for the 406 error.resp.raise_for_status()# Determine what kind of response we've gotten to ensure that it is one# that we can support, and if it is, dispatch to a function that will# understand how to interpret that particular version+serialization. If# we don't understand the content type we've gotten, then we'll raise# an exception.content_type=parse_content_type(resp.headers.get("content-type",""))matchcontent_type:case"application/vnd.pypi.simple.v1+json":handle_v1_json(resp)case"application/vnd.pypi.simple.v1+html"|"text/html":handle_v1_html(resp)case_:raiseException(f"Unknown content type:{content_type}")

If a client wishes to only support HTML or only support JSON, then they wouldjust remove the content types that they do not want from theAccept header,and turn receiving them into an error.

TUF Support - PEP 458

PEP 458 requires that all API responses are hashable and that they can be uniquelyidentified by a path relative to the repository root. For a Simple API repository, thetarget path is the Root of our API (e.g./simple/ on PyPI). This createschallenges when accessing the API using a TUF client instead of directly using astandard HTTP client, as the TUF client cannot handle the fact that a target couldhave multiple different representations that all hash differently.

PEP 458 does not specify what the target path should be for the Simple API, butTUF requires that the target paths be “file-like”, in other words, a path likesimple/PROJECT/ is not acceptable, because it technically points to adirectory.

The saving grace is that the target path does nothave to actually match the URLbeing fetched from the Simple API, and it can just be a sigil that the fetching codeknows how to transform into the actual URL that needs to be fetched. This same thingcan hold true for other aspects of the actual HTTP request, such as theAcceptheader.

Ultimately figuring out how to map a directory to a filename is out of scope for thisPEP (but it would be in scope forPEP 458), and this PEP defers making a decisionabout how exactly to represent this inside ofPEP 458 metadata.

However, it appears that the current WIP branch against pip that attempts to implementPEP 458 is using a target path likesimple/PROJECT/index.html. This could bemodified to include the API version and serialization format using something likesimple/PROJECT/vnd.pypi.simple.vN.FORMAT. So the v1 HTML format would besimple/PROJECT/vnd.pypi.simple.v1.html and the v1 JSON format would besimple/PROJECT/vnd.pypi.simple.v1.json.

In this case, sincetext/html is an alias toapplication/vnd.pypi.simple.v1+htmlwhen interacting through TUF, it likely will make the most sense to normalize to themore explicit name.

Likewise thelatest metaversion should not be included in the targets, onlyexplicitly declared versions should be supported.

Does this mean PyPI is planning to drop support for HTML/PEP 503?

No, PyPI has no plans at this time to drop support forPEP 503 or HTMLresponses.

While this PEP does give repositories the flexibility to do that, that largelyexists to ensure that things like using the Endpoint Configuration mechanism isable to work, and to ensure that clients do not make any assumptions that wouldprevent, at some point in the future, gracefully dropping support for HTML.

The existing HTML responses incur almost no maintenance burden on PyPI andthere is no pressing need to remove them. The only real benefit to dropping themwould be to reduce the number of items cached in our CDN.

If in the future PyPIdoes wish to drop support for them, doing so wouldalmost certainly be the topic of a PEP, or at a minimum a public, open, discussionand would be informed by metrics showing any impact to end users.

Why JSON instead of X format?

JSON parsers are widely available in most, if not every, language. A JSONparser is also available in the Python standard library. It’s not the perfectformat, but it’s good enough.

Why not add X feature?

The general goal of this PEP is to change or add very little. We will instead focuslargely on translating the existing information contained within our HTML responsesinto a sensible JSON representation. This will includePEP 658 metadata requiredfor packaging tooling.

The only real new capability that is added in this PEP is the ability to havemultiple hashes for a single file. That was done because the current mechanism beinglimited to a single hash has made it painful in the past to migrate hashes(md5 to sha256) and the cost of making the hashes a dictionary and allowing multipleis pretty low.

The API was generally designed to allow further extension through adding new keys,so if there’s some new piece of data that an installer might need, future PEPs caneasily make that available.

Why include the filename when the URL has it already?

We could reduce the size of our responses by removing thefilename key and expectingclients to pull that information out of the URL.

Currently this PEP chooses not to do that, largely becausePEP 503 explicitly requiredthat the filename be available via the anchor tag of the links, though that was largelybecausesomething had to be there. It’s not clear if repositories in the wild alwayshave a filename as the last part of the URL or if they’re relying on the filename in theanchor tag.

It also makes the responses slightly nicer to read for a human, as you get a nice shortunique identifier.

If we got reasonable confidence that mandating the filename is in the URL, then we coulddrop this data and reduce the size of the JSON response.

Why not break out other pieces of information from the filename?

Currently clients are expected to parse a number of pieces of information from thefilename such as project name, version, ABI tags, etc. We could break these outand add them as keys to the file object.

This PEP has chosen not to do that because doing so would increase the size of theAPI responses, and most clients are going to require the ability to parse thatinformation out of file names anyways regardless of what the API does. Thus it makessense to keep that functionality inside of the clients.

Why Content Negotiation instead of multiple URLs?

Another reasonable way to implement this would be to duplicate the API routes andinclude some marker in the URL itself for JSON. Such as making the URLs be somethinglike/simple/foo.json,/simple/_index.json, etc.

This makes some things simpler like TUF integration and fully static serving of arepository (since.json files can just be written out).

However, this is two pretty major issues:

• Our current URL structure relies on the fact that there is an URL that representsthe “root”,/ to serve the list of projects. If we want to have separate URLsfor JSON and HTML, we would need to come up with some way to have two root URLs.

  Something like/ being HTML and/_index.json being JSON, since_indexisn’t a valid project name could work. But/ being HTML doesn’t work great ifa repository wants to remove support for HTML.

  Another option could be moving all of the existing HTML URLs under a namespace whilemaking a new namespace for JSON. Since/<project>/ was defined, we would have tomake these namespaces not valid project names, so something like/_html/ and/_json/ could work, then just redirect the non namespaced URLs to whatever the“default” for that repository is (likely HTML, unless they’ve disabled HTML then JSON).

• With separate URLs, there’s no good way to support zero configuration discoverythat a repository supports the JSON URLs without making additional HTTP requests todetermine if the JSON URL exists or not.

  The most naive implementation of this would be to request the JSON URL and fall backto the HTML URL forevery single request, but that would be horribly performantand violate the goal of minimal additional HTTP requests.

  The most likely implementation of this would be to make some sort of repository levelconfiguration file that somehow indicates what is supported. We would have the samenamespace problem as above, with the same solution, something like/_config.jsonor so could hold that data, and a client could first make an HTTP request to that,and if it exists pull it down and parse it to learn about the capabilities of thisparticular repository.

• The use ofAccept also allows us to add versioning into this field

All being said, it is the opinion of this PEP that those three issues combined makeusing separate API routes a less desirable solution than relying on contentnegotiation to select the most ideal representation of the data.

Does this mean that static servers are no longer supported?

In short, no, static servers are still (almost) fully supported by this PEP.

The specifics of how they are supported will depend on the static server inquestion. For example:

• S3: S3 fully supports custom content types, however it does not supportany form of content negotiation. In order to have a server hosted on S3, youwould have to use the “Endpoint configuration” style of negotiation, andusers would have to configure their clients explicitly.
• GitHub Pages: GitHub pages does not support custom content types, so theS3 solution is not currently workable, which means that onlytext/htmlrepositories would function.
• Apache: Apache fully supports server-driven content negotiation, and wouldjust need to be configured to map the custom content types to specific extension.

Why not add anapplication/json alias liketext/html?

This PEP believes that it is best for both clients and servers to be explicitabout the types of the API responses that are being used, and a content typelikeapplication/json is the exact opposite of explicit.

The existence of thetext/html alias exists as a compromise primarily toensure that existing consumers of the API continue to function as they alreadydo. There is no such expectation of existing clients using the Simple API withaapplication/json content type.

In addition,application/json has no versioning in it, which means thatif there is ever a2.x version of the Simple API, we will be forced to makea decision. Shouldapplication/json preserve backwards compatibility andcontinue to be an alias forapplication/vnd.pypi.simple.v1+json, or shouldit be updated to be an alias forapplication/vnd.pypi.simple.v2+json?

This problem doesn’t exist fortext/html, because the assumption is thatHTML will remain a legacy format, and will likely not gainany new features,much less features that require breaking compatibility. So having it be analias forapplication/vnd.pypi.simple.v1+html is effectively the same ashaving it be an alias forapplication/vnd.pypi.simple.latest+html, since1.x will likely be the only HTML version to exist.

The largest benefit to adding theapplication/json content type is thatthere do things that do not allow you to have custom content types, and requireyou to select one of their preset content types. The main example of this beingGitHub Pages, which the lack ofapplication/json support in this PEP meansthat static repositories will no longer be able to be hosted on GitHub Pagesunless GitHub adds theapplication/vnd.pypi.simple.v1+json content type.

This PEP believes that the benefits are not large enough to add that contenttype alias at this time, and that its inclusion would likely be a footgunwaiting for unsuspecting people to accidentally pick it up. Especially giventhat we can always add it in the future, but removing things is a lot harderto do.

Why add aapplication/vnd.pypi.simple.v1+html?

The PEP expects the HTML version of the API to become legacy, so one option itcould take is not add theapplication/vnd.pypi.simple.v1+html content type,and just usetext/html for that.

This PEP has decided that adding the new content type is better overall, since itmakes even the legacy format more self describing and makes them both more consistentwith each other. Overall I think it’s more confusing if the+html version doesn’texist.

Why v1.0 and not v1.1 or v2.0?

This PEP is still wholly backwards compatible with clients that could read theexisting v1.0 API, can still continue to read the API after these changes havebeen made. InPEP 629, the qualification for a major version bump is:

Incrementing the major version is used to signal a backwards incompatiblechange such that existing clients would no longer be expected to be able tomeaningfully use the API.

The changes in this PEP do not meet that bar, nothing has changed in a way thatexisting clients would no longer be expected to be able to meaningfully use theAPI.

That means we should still be within the v1.x version line.

The question of whether we should be v1.1 or v1.0 is a more interesting one, andthere are a few ways of looking at it:

• We’ve exposed new features to the API (the project name on the projectpage, multiple hashes), which is a sign that we should increment the minorversion.
• The new features exist wholly within the JSON serialization, which means thatno client that currently is requesting the HTML 1.0 page, would ever see anyof the new features anyways, so for them it is effectively still v1.0.
• No major client has implemented support for PEP 629 yet, which means that theminor version numbering is largely academic at this point anyways, since itexists to let clients provide feedback to end users.

The second and third points above end up making the first point kind ofmeaningless, and with that, it makes more sense to just call everything v1.0and be stricter about updating versions into the future.