Abstract

This PEP proposes introducing JSON encoded core metadata and wheel file formatmetadata files in Python packages. Python package metadata (“core metadata”)was first defined inPEP 241 to useRFC 822 email headers to encodeinformation about packages. This was reasonable in 2001; email messageswere the only widely used, standardized text format that had a parser inthe standard library. However, issues with handling different encodings,differing handling of line breaks, and other differences betweenimplementations have caused numerous packaging bugs. Using the JSON format forencoding metadata files would eliminate a wide range of these potential issues.

Motivation

The email message format has a number of complexities and limitations whichreduce its utility as a portable textual interchange format for packagingmetadata. Due to theemail parser requiring configuration changes toproperly generate valid core metadata, many projects do not use theemail module and instead generate core metadata in a custom manner.There are many pitfalls with generating email headers that can be encounteredby such custom generators. First, core metadata fields may contain newlines in thevalue of fields. These newlines must be handled properly to “unfolded” multiplelines perRFC 822. One particularly difficult to encode field is theDescription field, which may contain newlines and indentation. To encodethe field in email headers, CRLF line breaks must be followed by seven (7)spaces and a pipe (’|’) character. WhileDescription may now be encoded inthe message body, similar escaping issues occur for theAuthor andMaintainer fields. Improperly escaped newlines can lead to missing,partial, or invalid core metadata. Second, as discussed in thecore metadata specifications:

The standard file format for metadata (including in wheels and installedprojects) is based on the format of email headers. However, email formatshave been revised several times, and exactly which email RFC applies topackaging metadata is not specified. In the absence of a precisedefinition, the practical standard is set by what the standard libraryemail.parser module can parse using theemail.policy.compat32 policy.

Since no specific email RFC is selected, the current core metadataspecification is ambiguous whether a given core metadata document is valid.RFC 822 is the only email standard to be explicitly listed in a PEP.However, the core metadata specifications also requires that core metadata isencoded using UTF-8 when written to a file. This de-facto makes the coremetadata followRFC 6532, which specifies internationalization of emailheaders. This has practical interoperability concerns. Until a few years ago,it was unspecified how to properly encode non-ASCII emails in coremetadata, making parsing ambiguous. Third, the current format is difficult toproperly validate and parse. Many tools do not check for issues with the outputof theemail parser. If a document is malformed, it may still parsewithout error by theemail module as a valid email message. Furthermore,due to limitations in the email format, fields likeProject-Url must createcustom encodings of nested key-value items, further complicating parsing andvalidation. Finally, the lack of a schema makes it difficult to validate thecontents of email message encoded metadata. While introducing a specificationfor the current format has beendiscussed previously, no progress hadbeen made, and converting to JSON was a suggested resolution to the issuesraised.

TheWHEEL file format is currently encoded in a custom key-value format.While this format is easy to parse and write, it requires manual parsing andvalidation to ensure that the contents are valid. Moving to a JSON encodedformat will allow for easier parsing and validation of the contents, andsimplify packaging tools and services by using a consistent format fordistribution metadata.

Rationale

Introducing a new core metadata file with a well-specified format will greatlyease generating, parsing, and validating metadata. JSON is a natural choice forstoring package core metadata. It is easily machine readable and writable, isunderstandable to humans, and is well supported across many languages.Furthermore,PEP 566 already specifies a canonicalization of email formattedcore metadata to JSON. JSON is also a frequently used format for datainterchange on the web. For discussion of other formats considered, pleaserefer to the rejected ideas section.

To maintain backwards compatibility, the JSON metadata file MUST be generatedalongside the existing email formatted metadata file. This ensures that toolsthat do not support the new format can still read package metadata for newpackages.

The JSON formatted metadata file must be semantically equivalent to the emailencoded file. This ensures that the metadata is unambiguous between the twoformats, and tools may read either when both are present. To maintainperformance, this equivalence is not required to be verified by installers,though other tools may do so. Some tools may choose to make the check dependenton a configuration flag.

Package indexes SHOULD check that the metadata files are semanticallyequivalent when the package is added to the index. This is a low-cost, one-timecheck that ensures users of the index are served valid packages.

Specification

Backwards Compatibility

The specification forMETADATA.json andWHEEL.json is designed suchthat the new format is completely backwards compatible. Existing tools may readmetadata from the existing email formatted files, and new tools may takeadvantage of the new format.

A future major revision of the wheel specification may make theMETADATA,PKG-INFO, andWHEEL files optional and make theMETADATA.json andWHEEL.json files required.

Note that tools will need to maintain parsing of email metadata and thekey-value formattedWHEEL file indefinitely to support parsing metadatafor old packages which only have theMETADATA,PKG-INFO,orWHEEL files.

Security Implications

One attack vector with JSON encoded core metadata is if the JSON payload isdesigned to consume excessive memory or CPU resources in a denial of service(DoS) attack. While this attack is not likely to affect users whom can cancelresource-intensive interactive operations, it may be an issue for packageindexes.

There are several mitigations that can be made to prevent this:

1. The length of the JSON payload can be restricted to a reasonable size.
2. The reader may use aJSONDecoder to omit parsingintandfloat values to avoid quadratic number parsing time complexityattacks.
3. I plan to contribute a change toJSONDecoder in Python3.15+ that will allow it to be configured to restrict the nesting of JSONpayloads to a reasonable depth. Core metadata currently has a maximum depthof 2 to encode mapping and list fields.

With these mitigations in place, concerns about denial of service attacks withJSON encoded core metadata are minimal.

Reference Implementation

A reference implementation of the JSON schema for JSON core metadata isavailable inAppendix: JSON Schema for Core Metadata.

Furthermore, a reference implementation in thepackaging libraryisavailable.

A reference implementation generating bothMETADATA.json andWHEEL.jsonin theuv build backendis also available.

Rejected Ideas

Open Issues

Acknowledgements

Thanks to Konstantin Schütze for implementing the reference implementation ofthis PEP in theuv build backend and for providing valuable feedback on thespecification.

Copyright

This document is placed in the public domain or under theCC0-1.0-Universal license, whichever is more permissive.