Measurability and Phantom Dependencies

Python packages are particularly affected by the “phantom dependency”problem, where software components that aren’t written in Python are includedin Python packages for many reasons, such as ease of installation andcompatibility with standards:

• Python serves scientific, data, web, and machine-learning use-cases whichuse compiled or non-Python languages like Rust, C, C++, Fortran, JavaScript,and others.
• The Python wheel format is preferred by users due to the ease-of-installation.No code is executed during the installation step, only extracting the archive.
• The Python wheel format requires bundling shared compiled libraries withouta method to encode metadata about these libraries.
• Packages related to Python packaging sometimes need to solve the“bootstrapping” problem, so include pure Python projects inside theirsource code.

These software components can’t be described using Python package metadata andthus are likely to be missed by software composition analysis (SCA) softwarewhich can mean vulnerable software components aren’t reported accurately.

For example,the Python package Pillow includes 16 shared object libraries in the wheel thatwere bundled by auditwheel as a part of the build. None of those shared objectlibraries are detected when using common SCA tools like Syft and Grype.If an SBOM document is included annotating all the included shared librariesthen SCA tools can identify the included software reliably.

Build Tools, Environment, and Reproducibility

Going beyond the runtime dependencies of a package: SBOMs can also record thetools and environments used to build a package. Recording the exact toolsand versions used to build a package is often required to establishbuild reproducibility.Build reproducibility is a property of software that can be used to detectincorrectly or maliciously modified software components when compared to theirupstream sources. Without a recorded list of build tools and versions it canbecome difficult to impossible for a third-party to verify build reproducibility.

Regulations

SBOMs are required by recent software security regulations, like theSecure Software Development Framework (SSDF) and theCyber Resilience Act (CRA). Due to their inclusion in these regulations,the demand for SBOM documents of open source projects is expected to be high.One goal is to minimize the demands on open source project maintainers byenabling open source users that need SBOMs to self-serve using existingtooling.

Another goal is to enable contributions from users who need SBOMs to annotateprojects they depend on with SBOM information. Today there is no mechanism topropagate the results of those contributions for a Python package so there isno incentive for users to contribute this type of work.

Using SBOM standards instead of Core Metadata fields

Attempting to add every field offered by SBOM standards into Python packageCore Metadata would result in an explosion of new Core Metadata fields,including the need to keep up-to-date as SBOM standards continue to evolveto suit new needs in that space.

Instead, this proposal delegates SBOM-specific metadata to SBOM documents thatare included in Python packages into a named directory under dist-info.

This standard also doesn’t aim to replace Core Metadata with SBOMs,instead focusing on the SBOM information being supplemental to Core Metadata.Included SBOMs only contain information about dependencies included in thepackage archive or information about the top-level software in the package thatcan’t be encoded into Core Metadata but is relevant for the SBOM use-case(“software identifiers”, “purpose”, “support level”, etc).

Zero-or-more opaque SBOM documents

Rather than requiring at most one included SBOM document per Python package,this PEP proposes that one or more SBOM documents may be included in a Pythonpackage. This means that code attempting to annotate a Python package with SBOMdata may do so without being concerned about corrupting data already containedwithin other SBOM documents.

Additionally, this PEP treats SBOM document data opaquely instead relying onfinal end-users of the SBOM data to process the contained SBOM data.This choice acknowledges that SBOM standards are an active area of developmentwhere there is not yet (and may never be) a single definitive SBOM standardand that SBOM standards can continue to evolve independent of Python packagingstandards. Already tools that consume SBOM documents support a multitude ofSBOM standards to handle this reality.

These decisions mean this PEP is capable of supporting any SBOM standardand does not favor one over the other, instead deferring the decision toproducing projects and tools and consuming user tooling.

Adding data to Python packages without new metadata versions

The rollout of a new metadata version and field requires that many differentprojects and teams need to adopt the metadata version in sequence to avoidwidespread breakage. This effect usually means a substantial delay in howquickly users and tools can start using new packaging features.

For example, a single metadata version bump requiresupdates to PyPI, variouspyproject.toml parsing and schema projects,thepackaging library, wait for releases, thenpip and other installersneed to bundle the changes topackaging and release, then build backends canbegin emitting the new metadata version, again wait for releases, and only thencan projects begin using the new features. Even with this careful approach it’snot guaranteed that tools won’t break on new metadata versions and fields.

To avoid this delay, simplify overall how to include SBOMs, and to giveflexibility to build backends and tools, this PEP proposes using a subdirectoryunder.dist-info to safely add data to a Python package while avoiding theneed for new metadata fields and versions. This mechanism allows build backendsand tools to begin using the feature described in this PEP immediately afteracceptance without the head-of-line blocking on other projects adopting the PEP.

Storing files in the.dist-info or.data directory

There are two top-level directories in binary distributions where files beyondthe software itself can be stored:.dist-info and.data.This specification chose to use the.dist-info directory for storingsubdirectories and files.

Firstly, the.data directory has no corresponding location in the installedpackage, compared to.dist-info which does preserve the link between thebinary distribution to the installed package in an environment. The.datadirectory instead has all its contents merged between all installed packages inan environment which can lead to collisions between similarly named files.

Secondly, subdirectories under the.data directory require new definitionsto the Pythonsysconfigmodule. This means defining additional directories require waiting for a changeto Python andusing the directory requires waiting for adoption of the newPython version by users. Subdirectories under.dist-info don’t have theserequirements, they can be used by any user, build backend, and installerimmediately after a new subdirectory name is registered regardless of Pythonor metadata version.

What are the differences between PEP 770 and PEP 725?

PEP 725(“Specifying external dependencies in pyproject.toml”) is a differentPEP with some similarities to PEP 770, such as attempting to describe non-Pythonsoftware within Python packaging metadata. This section aims to show how thesetwo PEPs are tracking different information and serving different use-cases:

• PEP 725 describesabstract dependencies, such as requiring “a C compiler”as a build-time dependency (virtual:compiler/c) or needing to link “theOpenSSL library” at build time (pkg:generic/openssl). PEP 770 describesconcrete dependencies, more akin to dependencies in a “lock file”, such asan exact name, version, architecture, andhash of a software library distributed through AlmaLinux distribution(pkg:rpm/almalinux/libssl3@3.2.0). For cases like build dependencies thismight result in a dependency being requested via PEP 725 and then recordedconcretely in an SBOM post-build with PEP 770.
• PEP 725 is for describingexternal dependencies, provided by the systembeing used to either build or run the software. PEP 770 is for describingbundled software inside Python package archives, the SBOM documentsdon’t describe software on the system.
• PEP 725 is primarily about identification, using a list of softwareidentifiers. PEP 770 provides thecomplete functionality of SBOM standardsto describe various software attributes such as license, checksum, downloadlocation, etc.
• PEP 725 and PEP 770 have different users and use-cases. PEP 725 isprimarily for humans writing dependencies inpyproject.toml by hand.The users of the information are build backends and users who want to buildsoftware from source.PEP 770 is primarily for tools which are capable of generating SBOM documentsto be included in a Python package archive and SBOM/SCA tools which want toSBOM documents about installed software to do some other task such asvulnerability scanning or software analysis.

Reserving the.dist-info/sboms directory

This PEP introduces a new registry of reserved subdirectory names allowed inthe.dist-info directory for thedistribution archiveandinstalled project s project types. Future additions to this registrywill be made through the PEP process. The initial values in this registry are:

SeeBackwards Compatibility for a complete methodology foravoiding backwards incompatibilities with selecting this directory name.

SBOM files in project formats

A few additions will be made to the existing specifications.

Built distributions (wheels)

The wheel specification will be updated to add the new registry of reserveddirectory names and to reflect that if the.dist-info/sboms subdirectoryis specified that the directory contains SBOM files.

Installed projects

The Recording Installed Projects specification will be updated to reflectthat if the.dist-info/sboms subdirectory is specified that the directorycontains SBOM files and that any files in this directory MUST be copied fromwheels by install tools.

SBOM data interoperability

This PEP treats data contained within SBOM documents as opaque, recognizingthat SBOM standards are an active area of development. However, there are someconsiderations for SBOM data producers that when followed will improve theinteroperability and usability of SBOM data made available in Python packages:

• SBOM documents SHOULD use a widely-accepted SBOM standard, such asCycloneDX orSPDX.
• SBOM documents SHOULD use UTF-8-encoded JSON (RFC 8259) when availablefor the SBOM standard in use.
• SBOM documents SHOULD include all required fields for the SBOM standard inuse.
• SBOM documents SHOULD include a “time of creation” and “creating tool” fieldfor the SBOM standard in use. This information is important for usersattempting to reconstruct different stages for a Python package being built.
• The primary component described by the SBOM document SHOULD be the top-levelsoftware within the Python package (for example,“pkg:pypi/pillow” for the Pillow package).
• All non-primary components SHOULD have one or more paths in the relationshipgraph showing the relationship between components. If this information isn’tincluded, SCA tools might exclude components outside of the relationship graph.
• All software components SHOULD have a name, version, and one or more softwareidentifiers (PURL, CPE, download URL).

PyPI and other indices MAY validate the contents of SBOM documents specified bythis PEP, but MUST NOT validate or reject data for unknownSBOM standards, versions, or fields.

Reserved.dist-info/sboms subdirectory

The new reserved.dist-info/sboms subdirectory representsa new reservation that wasn’t previously documented, thus has the potential tobreak assumptions being made by already existing tools.

To check what.dist-info subdirectory names are in use todaya query acrossall files in package archives on PyPIwas executed:

SELECT(regexp_extract(archive_path,'.*\.dist-info/([^/]+)/',1)ASdirname,COUNT(DISTINCTproject_name)ASprojects)FROM'*.parquet'WHEREarchive_pathLIKE'%.dist-info/%/%'GROUPBYdirnameORDERBYprojectsDESC;

Note that this only includes records forfiles and thus won’t return results for empty directories. Empty directoriesbeing pervasively used and somehow load-bearing is unlikely, so is an acceptedrisk of using this method. This query yielded the following results:

Not shown above are around ~50 other subdirectory names that are used in asingle project. From these results we can see:

• Most subdirectories under.dist-info are to do with licensing,one of which (licenses) is specified byPEP 639 and others(license_files,LICENSES) are from draft implementationsofPEP 639.
• Thesboms subdirectory doesn’t collide with existing use.
• Other subdirectory names under.dist-info appear to be either notwidespread or accidental.

As a result of this query we can see there are already some projects placingdirectories under.dist-info, so we can’t require that build frontendsraise errors for unregistered subdirectories. Instead the recommendation isthat build frontends MAY warn the user or raise an error in this scenario.

What do Python package maintainers need to know?

Python package metadata can already describe the top-level software included ina package archive, but what if a package archive contains other softwarecomponents beyond the top-level software? For example, the Python wheel for“Pillow” contains a handful of other software libraries bundled inside, likelibjpeg,libpng,libwebp, and so on. This scenario is where this PEPis most useful, for adding metadata about bundled software to a Python package.

Some build tools may be able to automatically annotate bundled dependencies.Typically tools can automatically annotate bundled dependencies when thosedependencies come from a “packaging ecosystem” (such as PyPI, Linux distros,Crates.io, NPM, etc).

What do SBOM tool authors need to know?

Developers of SBOM generation tooling will need to know about the existenceof this PEP and that Python packages may begin publishing SBOM documentswithin package archives. This information needs to be included as a part ofgenerating an SBOM document for a particular Python package or Pythonenvironment.

A follow-up informational PEP will be authored to describe how to transformPython packaging metadata, including the mechanism described in this PEP,into an SBOM document describing Python packages. Once the informational PEP iscomplete, tracking issues will be opened specifically linking to theinformational PEP to spur the adoption of PEP 770 by SBOM tools.

Abenchmark is being createdto compare the outputs of different SBOM tools when run with various Pythonpackaging inputs (package archive, installed package, environment, containerimage) is being created to track the progress of different SBOM generationtools. This benchmark will inform where tools have gaps in supportof this PEP and Python packages.

What do users of SBOM documents need to know?

Many users of this PEP won’t know of its existence, instead their softwarecomposition analysis tools, SBOM tools, or vulnerability scanners will simplybegin giving more comprehensive information after an upgrade. For users that areinterested in the sources of this new information, the “tool” field of SBOMmetadata already provides linkages to the projects generating their SBOMs.

For users who need SBOM documents describing their open source dependencies thefirst step should always be “create them yourself”. Using the benchmarks abovea list of tools that are known to be accurate for Python packages can bedocumented and recommended to users. For projects which requireadditional manual SBOM annotation: tips for contributing this data and tools formaintaining the data can be recommended.

Note that SBOM documents can vary across different Python package archivesdue to variance in dependencies, Python version, platform, architecture, etc.For this reason users SHOULD only use the SBOM documents contained withinthe actual downloaded and installed Python package archive and not assume thatthe SBOM documents are the same for all archives in a given package release.

Requiring a single SBOM standard

There is no universally accepted SBOM standard and this area is stillrapidly evolving (for example, SPDX released a new major version of theirstandard in April 2024). Most discussion and development around SBOMs todayfocuses on two SBOM standards:CycloneDX andSPDX.

To avoid locking the Python ecosystem into a specificstandard ahead of when a clear winner emerges this PEP treats SBOM documentsas opaque and only makes recommendations to promote compatibility withdownstream consumers of SBOM document data.

None of the decisions in this PEP restrict a future PEP to selecta single SBOM standard. Tools that use SBOM data today already need to supportmultiple formats to handle this situation, so a future standard that updates torequire only one standard would have no effect on downstream SBOM tools.

Using metadata fields to specify SBOM files in archives

A previous iteration of this specification used anSbom-File metadatafield to specify an SBOM file within a source or binary distribution archive.This would make the implementation similar toPEP 639 which uses theLicense-File field to enumerate license files in archives.

The primary issue with this approach is that SBOM files can originate from bothstatic and dynamic sources: like versioned source code, the build backend,or from tools adding SBOM files after the build has completed (like auditwheel).

Metadata fields must either be static or dynamic, not both. This isin direct conflict with the best-case scenario for SBOM data: that SBOM filesare added automatically by tools during the build of a Python package withoutuser-involvement or knowledge. Compare this situation to license files whichare almost always static.

The 639-style approach was ultimately dropped in favor of defining SBOMs simplyby their presence in the.dist-info/sboms directory. This approach allowsbuild backends and tools to add their own SBOM data without the static/dynamicconflict.

A future PEP will define the process for statically defining SBOM files to beadded to the.dist-info/sboms directory.