PEP 694 -- PyPI upload API 2.0 (Round 2)

PackagingStandards
1

I’m very happy to share round 2 of PEP 694 – PyPI upload API 2.0. This latest round incorporates all the changes discussed in thefirst round of DPO discussion, plus changes from@EWDurbin based on our discussion at PyCon 2025. Please use this thread for any further feedback.

Given that@dstufft would normally be the PEP Delegate for PEPs affecting PyPI, and that he along with Ee are co-authors, we will be putting this PEP on the Steering Council’s agenda for pronouncement. Should they wish to delegate,@dustin has agreed to serve in that role.

Python Enhancement Proposals (PEPs)

PEP 694 – Upload 2.0 API for Python Package Indexes | peps.python.org

This PEP proposes an extensible API for uploading files to a Python package index such as PyPI. Along with standardization, the upload API provides additional useful features such as support for:

11 Likes
2

Thanks for sharing this@barry! I’m really excited to see the upload 2.0 designs move along.

Some scatterbrained thoughts from reading the current proposal:

  • Theexpires-at key is currently described as an ISO 8601 formatted timestamp. What do you think about constraining this a bit further, and requiring that it (1) be a UTC timestamp with the Zulu marker, and (2) forbidding fractional seconds (i.e. whole seconds only)? IME (in other contexts) these are unnecessary sources of malleability, especially since these are server-originated times that should only ever be UTC anyways.
  • This is pedantic, but perhaps thefilename andversion fields should specify that they correspond to their relevant PEPs/living standards? Forfilename I think that would be theSource distribution file name andBinary distribution file name convention and forversion I think it’sVersion specifiers. I think this is arguably obvious from context, but maybe being explicit will help future implementers.
  • Two thoughts about the nonce/session design:

    • The currentgentoken implementation lacks domain separation between the fields. I think this could result in some niche cases where two users end up with the same session token. Specifically, if two users both don’t include anonce and have a(name, version) pairs that compose into the same “domain,” then they’ll end up with the same session key.

      This is pretty easy to contrive: for example,(foo, 11) (foo version v11) and(foo1, 1) (foo1, version v1) both compose intofoo11 for the purpose of the session key computation. The “classic” way to solve this is to encode each field such that it’s implicitly domain-separated, e.g. length-prefix each component.

      Another option (which is probably sound but harder to formally assert) is adding delimiters between the fields that can’t occur in each field, e.g.\t or ASCII SOH.

    • More generally, I’m curious if anybody would object to making thenonce fully required – I think the above domain separation should still be addressed, but having thenonce would have effectively prevented it as well.

      I completely understand the value of being able to disclose the staging URL (I want that feature!), but I think maybe we can preserve that property while reducing the flexibility of the session creation itself: uploading clients are in full control of the session URL regardless of thenonce state, so the UX of a--publish-staging (or similar) flag isn’t contingent on the upload APIitself allowing thenonce to be unset.

      I think this has two virtuous effects: the first is that it’s a defense in depth against the scenario above, and the second is that it eliminates a source of user error/nonfamiliarity (i.e. users needing to understand what anonce is to get their intended behavior, versus the self-describing behavior of “you’ve disclosed the staging URL, regardless of how the machinery did that internally.”

2 Likes
3

Thanks for thorough review and comments@woodruffw ! I really apprecate your keen eye, especially on security.

William Woodruff:

Theexpires-at key is currently described as an ISO 8601 formatted timestamp. What do you think about constraining this a bit further, and requiring that it (1) be a UTC timestamp with the Zulu marker, and (2) forbidding fractional seconds (i.e. whole seconds only)?

IIRC@EWDurbin and I talk about the UTC/Zulu detail, and agree with it, but I wasn’t explicit in the last PR[1]. I’d put together a PR to update this.

William Woodruff:

This is pedantic, but perhaps thefilename andversion fields should specify that they correspond to their relevant PEPs/living standards?

Yes, I think that also makes sense. I should also add that the project name in the session creation request must conform to thestandard name format and will be normalized.

William Woodruff:

The currentgentoken implementation lacks domain separation between the fields.

Great observation, thanks for the callout!

William Woodruff:

The “classic” way to solve this is to encode each field such that it’s implicitly domain-separated, e.g. length-prefix each component.

So something like:

    def gentoken(name: bytes, version: bytes, nonce: bytes = b''):        h = sha256()        h.update(f'{len(name)}'.encode('ascii') + name)        h.update(f'{len(version)}'.encode('ascii') + version)        h.update(nonce)        return h.hexdigest()

?

William Woodruff:

More generally, I’m curious if anybody would object to making thenonce fully required

If we required it, would we have to disallow explicitly setting nonce tob''?

William Woodruff:

I think this has two virtuous effects: the first is that it’s a defense in depth against the scenario above, and the second is that it eliminates a source of user error/nonfamiliarity (i.e. users needing to understand what anonce is to get their intended behavior, versus the self-describing behavior of “you’ve disclosed the staging URL, regardless of how the machinery did that internally.”

We may still have to describe the nonce and its purpose though, although I could imagine upload tools doing the hefty lifting behind something like a--secret flag. If given, the upload tool could generate its own nonce internally without the user really having to understand the gory details.

  1. the given example is compliant↩︎

1 Like
4
woodruffw:

Theexpires-at key is currently described as an ISO 8601 formatted timestamp. What do you think about constraining this a bit further, and requiring that it (1) be a UTC timestamp with the Zulu marker, and (2) forbidding fractional seconds (i.e. whole seconds only)? IME (in other contexts) these are unnecessary sources of malleability, especially since these are server-originated times that should only ever be UTC anyways.

Maybe I’m missing something, but I don’t see much of a benefit in constraining this? Most of the time, thisexpires-at key will be read by some client software which (hopefully![1]) uses an ISO 8601-compliant datetime parser; so these additional constraints would actually be more work to implement. And in cases where this key will be read by a human directly (or if it’s just passed through by a simple client), then fractional seconds are a bit of noise (though fairly easy to ignore); but I could imagine scenarios where a non-UTC timestamp would be desirable. (E.g., for a company-internal package index, where the local time zone of the server and all users is the same.)

On the other hand, the additional complexity of specifying a custom subset of ISO 8601 seems like a noticeable downside to me. It makes it ever so slightly harder to read the PEP, a bit more work to implement, and I expect at least some client implementations will ignore these constraints anyway. (Which is probably fine, but I’d rather not rely onPostel’s law more than I absolutely need to.)

  1. I was taught that cryptography and datetimes are the two areas where you must always use a library written by domain experts and never write your own solution.↩︎

1 Like
5

I think it’s okay to be constraining here, since theexpires-at keys are send by the index servers, so it’s the “be conservative in what you send” side of the equation.

Jost Migenda:

E.g., for a company-internal package index, where the local time zone of the server and all users is the same.

Even in this case, it’s always better to use timezone-aware timestamps for clarity, and if you do that, you might as well just send UTC timestamps. They’re always going to be unambiguous, easy to parse in any language, and convertible to local timezones.

Aside: I’m now remembering the bit that@EWDurbin and I talked about: using a timestamp and anextend-until key in lieu of theextend-for key in request bodies. We decided against that, opting forextend-for and an integer number of seconds.

I was taught that cryptography and datetimes are the two areas where you must always use a library written by domain experts and never write your own solution.

Oh for sure. I recall some very entertaining talks by@pganssle on the topic at previous Pycons.

6 Likes
6
barry:

Even in this case, it’s always better to use timezone-aware timestamps for clarity,

I can see the practical value in that constraint, yes. (Sorry, the language in my earlier post was a bit too broad there.)

barry:

and if you do that, you might as well just send UTC timestamps. They’re always going to be unambiguous, easy to parse in any language, and convertible to local timezones.

… however, this I disagree with. Timestamps using non-UTC time zones are just as unambiguous, easy to parse and convertible; so for a software client, it makes absolutely no difference.

However, when this key is read by a human directly, having it in the local time zone can be much nicer in some scenarios. Using the example with the internal packaging server again: When that server sends2025-08-08T01:23:45+02:00I can just skip over the+02:00 at the end (because after the first 2–3 times I know the server uses my local time zone) and read the time off at a glance. If the server were to send2025-08-07T23:23:45ZI would have to do the math in my head every time and there’s a higher risk of making mistakes.[1]

I know this is a fairly niche scenario (and UTC certainly is a good default); but I don’t see a downside to allowing other time zones, where it makes sense to those users.

  1. Probably even more so for people whose time zones are offset by a non-integer number of hours. Or right after DST ends, when I need to retrain myself to add only one hour, instead of two.↩︎

7

Using fixed-precision, standardised-timezone ISO-8601 strings means you can compare and sort these date-times without needing to do any parsing.

Regarding the nonce and gentoken, what’s the rationale of making the token generation algorithm standard over simply returning the token in the get/create response and saying the token’s value is opaque?

2 Likes
8

ISO 8601 has cruft likeweek dates and fractional minutes, as in2024-W33-4T09:59,5+0200. You probably wantRFC 3339 instead of that.

But of course, integer seconds are even simpler.

5 Likes
9

Thanks@barry!

Barry Warsaw:

If we required it, would we have to disallow explicitly setting nonce tob''?

Yeah, that’s what I was thinking – IMO it’d be nice for the spec to say that thenonce MUST be present and nonempty, and perhaps stipulate a minimum length.

(Length being a very weak proxy for entropy, since there’s no easy way for the server to enforce entropy. But this is similar to TLS and other protocols, where the assumption is that the client is responsible for sufficiently strong nonce generation. The PEP could also guide that in the right direction by providing agennonce example like withgentoken:slightly_smiling_face:)

Barry Warsaw:

So something like:

    def gentoken(name: bytes, version: bytes, nonce: bytes = b''):        h = sha256()        h.update(f'{len(name)}'.encode('ascii') + name)        h.update(f'{len(version)}'.encode('ascii') + version)        h.update(nonce)        return h.hexdigest()

Yep! I need to convince myself that that’s sufficient, but that’s along the lines of what I was thinking.

(There might still be some collision potential there, since there’s no explicit field separation.)

Barry Warsaw:

We may still have to describe the nonce and its purpose though, although I could imagine upload tools doing the hefty lifting behind something like a--secret flag. If given, the upload tool could generate its own nonce internally without the user really having to understand the gory details.

Oh, perhaps this was my misunderstanding: I thinking that upload tools would handle the nonce entirely internally, with no user interaction with nonces whatsoever. My rationale for this was that gettingusers to properly enforce the “once” property of anonce is pretty hard (users love reusing nonces, especially if described generically as “secrets”), so the upload tool should be internally responsible for unique nonce generation.

To ground things, this is kind of the UX I was expecting:

$ twine upload --staged dist/*<... normal upload output ...>Staged index available at: hxxps://pypi.org/blahblah/staging/<token>/...

i.e. the tool entirely handles nonce generation, and it’s up to the user to decide whether or not to share that staging URL.

(This becomes interesting in public CI environments where logs are publicly viewable. But workflows likegh-action-pypi-publish already deal with this by registering secret masks:slightly_smiling_face:)

Jost Migenda:

Maybe I’m missing something, but I don’t see much of a benefit in constraining this?

I put a longer comment (sorry…) below about why I think transacting in UTC is generally good with services, but I think there’s a straightforward reason why this PEPspecifically should use it: the timestamps in question originate entirely from the server, and the client has noa priori knowledge of where the server is (much less has any reason to find that information useful).

Or in other words: IMO it’s very surprising for a server on the Internet to respond to client interactions with its own local times, especially in a world where “server local time” can change between HTTP requests in a session (e.g. if an upload session is rebalanced to another host in a different zone).

Jost Migenda:

On the other hand, the additional complexity of specifying a custom subset of ISO 8601 seems like a noticeable downside to me. It makes it ever so slightly harder to read the PEP, a bit more work to implement, and I expect at least some client implementations will ignore these constraints anyway. (Which is probably fine, but I’d rather not rely onPostel’s law more than I absolutely need to.)

I probably could have said this more explicitly: I was really just thinking of a subset like RFC 3339 (thanks@encukou!) with the stipulation that the server transacts in UTC:sweat_smile:. I agree it doesn’t make sense to really innovate on time formats; I think RFC 3339 with UTC time is on the happy path.

Jost Migenda:

However, when this key is read by a human directly, having it in the local time zone can be much nicer in some scenarios. Using the example with the internal packaging server again: When that server sends2025-08-08T01:23:45+02:00I can just skip over the+02:00 at the end (because after the first 2–3 times I know the server uses my local time zone) and read the time off at a glance. If the server were to send2025-08-07T23:23:45ZI would have to do the math in my head every time and there’s a higher risk of making mistakes.[1]

I think there’s really two things here: there’s what the server transacts in (which IMO should always be UTC), and what users are presented with (which should typically be a localized time). It’s possible I’ve misunderstood the PEP’s intended scope, but my interpretation of it was that it defines how an upload client and server interact, and leaves the client presentation layer open.

(With my maintainer hat on: insofar astwine would present these timestamps, I think it would make sense to localize them to the user.)

Petr Viktorin:

You probably wantRFC 3339 instead of that.

Yes, completely agreed! I think in effect what we really want is a RFC 3339 subset of ISO 8601:slightly_smiling_face:

1 Like
10

In the context of a public server like PyPI, I agree with everything you say. However, this PEP is more broadly applicable to Python package indices; and in the context of an internal package index, I think many of your points no longer apply.

Let me give a bit more context: I’m working in the e-Research team at a university. Among other things, we’re building Trusted Research Environments (VMs used for working with e.g. sensitive medical data), which have restrictions on network access and package installation. If this PEP is accepted, we will likely use it in a local package index and write various custom scripts to interact with the API.

In this context, I do know exactly which time zone (and which data centre and rack) the server is located in, I know this time zone is not going to change and I do actually want to see the server local time.

Most of those custom scripts are going to be fairly simple; in many cases, they’ll probably just echo the server response verbatim[1]. Enforcing UTC in the protocol and requiring all these scripts to explicitly parse the datetime and perform time zone conversion would add a bit of unnecessary friction.

(And yes, this isn’t too much work in the grand scheme of things; so if I haven’t convinced you after this post, I’m happy to drop the topic.)

  1. The proposed API is nicely designed and mostly human readable—at least if those humans are sysadmins:wink:↩︎

11
Jost Migenda:

In this context, I do know exactly which time zone (and which data centre and rack) the server is located in, I know this time zone is not going to change and I do actually want to see the server local time.

This is understandable, but I still think it’s principally a presentation question. To use your example: if local time is critical to present, then you’ll presumably want to present it correctly when the local timedoesn’t match the server’s local time as well. In other words you’ll still need to dosome kind of local time adjustment if (for example) you happen to be in the next time zone over on a given day.

(Overall, I think my opinion here is that this is a low-level implementation detail within the PEP, and that it’s up to client tools liketwine anduv to decide if and how to take machine-to-machine time information and present it sensibly to users. In other words, I think this should be UTC for roughly the same reason that the nonce would ideally be required – it reduces degrees of freedom/malleability at the protocol level while preserving them at the client layer.)

3 Likes
(fungi)12

Let me give a bit more context: I’m working in the e-Research team at a university. Among other things, we’re building Trusted Research Environments (VMs used for working with e.g. sensitive medical data), which have restrictions on network access and package installation. If this PEP is accepted, we will likely use it in a local package index and write various custom scripts to interact with the API.

In this context, I do know exactly which time zone (and which data centre and rack) the server is located in, I know this time zone is not going to change and I do actually want to see the server local time.

Does server local time change in this context? That is to say, does the server exist in a locale that observes twice-yearly time changes? Taking my time zone as a reference, if I upload a package in February then the local time is UTC-0500, but if I’m downloading it in July my local time is UTC-0400. Which one is the relevant time to display? Does the server or client need to perform some conversion to “correct” it to now-local vs then-local time? Do you simply swap the offset indicator or do you add/subtract an hour on the timestamp too? Do you care about uploads that occur during the overlap of “fall back” hours where a later upload can have an earlier “local” timestamp?

Using UTC for the storage and transmission protocols simplifies this into a purely front-end implementation detail that doesn’t require standardizing.

6 Likes
13
William Woodruff:
barry:

So something like:

    def gentoken(name: bytes, version: bytes, nonce: bytes = b''):        h = sha256()        h.update(f'{len(name)}'.encode('ascii') + name)        h.update(f'{len(version)}'.encode('ascii') + version)        h.update(nonce)        return h.hexdigest()

Yep! I need to convince myself that that’s sufficient, but that’s along the lines of what I was thinking.

(There might still be some collision potential there, since there’s no explicit field separation.)

I thought about this a bit more and I think it’s sound, but I find it a little scary (especially since Python package names can start with/be all numbers):sweat_smile:

Another option that occurred to me: what if the server was entirely responsible for generating the session token? That would allow it to be entirely random without requiring us to perform a safe hash construction here, and would still allow the same “disclose the staging URL” flow for users.

Is there any drawback/downside to having the server be responsible for this token?

(Barry Warsaw)14
William Woodruff:

Is there any drawback/downside to having the server be responsible for this token?

Following up a bit of of order, there’s use case I had in mind for the client to provide the nonce component of the session token: it (possibly?) makes out of band sharing of the stage URL easier. If the upload client and the stage-testing client were disconnected, they’d only have to agree on the nonce out of band, and then both could calculate the stage URL without having to communicate between them or with the server, assuming of course they also both know the package name and version.

If the server generates the token, then it can use any means necessary to do so; i.e. it doesn’t necessarily have to use the package name and version, or it could generate its own random nonce. But then of course the preview link used for--extra-index-url would have to be shared between the client creating the session and any client accessing the stage.

1 Like
(Barry Warsaw)15
William Woodruff:

especially since Python package names can start with/be all numbers

I thought about this too, but couldn’t come up with a way to exploit it.If we keep nonce generation in the client, then I’m not against using a field separator as you suggested earlier.

1 Like
(Barry Warsaw)16
Petr Viktorin:

You probably wantRFC 3339 instead of that

Thanks! RFC 3339 seems ideal.

fungi:

Using UTC for the storage and transmission protocols simplifies this into a purely front-end implementation detail that doesn’t require standardizing.

I agree, and would prefer to keep the UTC/Z specification. The other factor for me is that the protocol’sexpires-at field is not intended as an attestation of the server’s timezone. While that might be useful to know, I don’t think this PEP is the place to accommodate that information.

3 Likes
(Laurie O)17

An alternative is to encode the field lengths as 64-bit integers usingstruct.pack rather than including their textual representation.

18
Barry Warsaw:

it (possibly?) makes out of band sharing of the stage URL easier. If the upload client and the stage-testing client were disconnected, they’d only have to agree on the nonce out of band, and then both could calculate the stage URL without having to communicate between them or with the server, assuming of course they also both know the package name and version.

Makes sense, thanks for explaining! I figured it had something to do with being able to “auto discover” these staging URLs but wasn’t sure:slightly_smiling_face:

I think this would be a cool property to maintain, but I wonder about generality: as an end user I’d imagine wanting to testall staged dependency versions (i.e. in CI) to suss out upgrade problems. However to do that I would need to know each dependency’s (including transitive ones’) staged version(s), which isn’t easily discoverable at the moment.

To fix that we’d probably need some degree of server communication. For example, the server could advertise the set of staged versions, which the client could then use to compute the staging index URLs. Or the server could advertise the staging URLs directly, in which case clients don’t need to compute the token at all (per the idea above about having the server be fully responsible).

TL;DR: I can envision users wanting something like${TOOL} install --staging $MY_PROJECT, where--staging tells a compatible index to attempt to resolve all staged versions. But to do that I think we need some kind of server communication, since the installer/user won’t know AOT what the candidate staged versions are. I think this would be the fully general form of staging URLs being discoverable, but itdoes have the downside of making the discovery not be “out of band.”

Separate from the above: I realized thatgentoken also probably needs to stipulate thatname andversion are fully normalized, if the PEP ends up going that route. Otherwise installers/indices might disagree about which non-normal form to use and produce mismatching tokens, especially if the user is asked to provide the name/version pair:sweat_smile:

1 Like
(Michał Górny)19
barry:

So something like:

    def gentoken(name: bytes, version: bytes, nonce: bytes = b''):        h = sha256()        h.update(f'{len(name)}'.encode('ascii') + name)        h.update(f'{len(version)}'.encode('ascii') + version)        h.update(nonce)        return h.hexdigest()

?

To be honest, I wouldn’t go that way. I think it’s safe, or at least I can’t come up with a proper way to trigger a collision, but given that so many of us had to think if it’s safe, I think it’d be safer to use an explicit separator or some kind of safe encoding. It’s not like that we need to be compact here.

encukou:

You probably wantRFC 3339 instead of that.

+1000 to that. It’s one of my pet peeves.

2 Likes
20

Overall, it looked great, and the second iteration is even better. A few minor comments:

Content Types

[…]

ServersMUST NOT advertise support for API versions beyond those defined in approved PEPs. Any new versions or formats require standardization through a new PEP.

This sounds like belonging to (and already stated in) the previous section.

If an error occurs, the appropriate4xx code will be returned, as described in theErrors section.

Looks like the “Errors section” doesn’tdescribe error codes. (Also in a few other places.)

In either case, the server should include aLocation header pointing back to the Publishing Session status URL, and if the server returned a202 Accepted, the client may poll that URL to watch for the status to change.

I’m sorry for being picky but this sounds like the client “may” poll it only for a 202. Perhaps something like:

In either case, the server should include aLocation header pointing back to the Publishing Session status URL that can be used to query the current status. If the server returned a202 Accepted, polling it can be used to watch for the status to change.

The serverMAY allow parallel uploads of files, but is not required to.

Presumably also returning409 Conflict if it doesn’t and you attempt a second parallel upload? I wonder if it should advertise the support for them somewhere up front, so clients wouldn’t have to determine-by-trying.