File Format

We wanted the file format to be easy to read as a diff when auditinga change to the lock file. As such, and thanks toPEP 518 andpyproject.toml, we decided to go with theTOML file format.

Secure by Design

Viewing therequirements file format as the closest we have toa lock file standard, there are a few issues with the file format whenit comes to security. First is that the file format simply does notrequire you to specify the exact version of a package. This is whytools likepip-tools exist to help manage that users ofrequirements files.

Second, you must opt into specifying what files are acceptable to beinstalled by using the--hash argument for a specific dependency.This is also optional with pip-tools as it requires specifying the--generate-hashes CLI argument. This requires--require-hashesfor pip to make sure no dependencies lack a hash to check.

Third, even when you control what files may be installed, it does notprevent other packages from being installed. If a dependency is notlisted in the requirements file, pip will happily go searching for afile to meet that need. You must specify--no-deps as anargument to pip to prevent unintended dependency resolution outsideof the requirements file.

Fourth, the format allows for installing asource distribution file (aka “sdist”). By its very nature,installing an sdist requires executing arbitrary Python code, meaningthat there is no control over what files may be installed. Only byspecifying--only-binary:all: can you guarantee pip to only use awheel file for each package.

To recap, in order for a requirements file to be as secure as what isbeing proposed, a user should always do the following steps:

1. Use pip-tools and its commandpip-compile--generate-hashes
2. Install the requirements file usingpipinstall--require-hashes--no-deps--only-binary:all:

Critically, all of those flags, and both the specificity andexhaustion of what to install that pip-tools provides, are optionalfor requirements files.

As such, the proposal raised in this PEP is secure by design whichcombats some supply chain attacks. Hashes for files which would beused to install from arerequired. You canonly install fromwheels to unambiguously define what files will be placed in the filesystem. Installersmust lead to an deterministic installationfrom a lock file for a given platform. All of this leads to areproducible installation which you can deem trustworthy (when youhave audited the lock file and what it lists).

Cross-Platform

Various projects which already have a lock file, likePDM andPoetry, provide a lock file which iscross-platform. This allowsfor a single lock file to work on multiple platforms while stillleading to the exact same top-level requirements to be installedeverywhere with the installation being consistent/unambiguous oneach platform.

As to why this is useful, let’s use an example involvingPyWeek(a week-long game development competition). Assume you are developingon Linux, while someone you choose to partner with is using macOS.Now assume the judges are using Windows. How do you make sure everyoneis using the same top-level dependencies, while allowing for anyplatform-specific requirements (e.g. a package requires a helperpackage under Windows)?

With a cross-platform lock file, you can make sure that the keyrequirements are met consistently across all platforms. You can thenalso make sure that all users on the same platform get the samereproducible installation.

Simple Installer

The separation of concerns between a locker and an installer allowsfor an installer to have a much simpler operation to perform. Assuch, it not only allows for installers to be easier to write, butfacilitates in making sure installers create unambiguous, reproducibleinstallations correctly.

The installer can also expend less computation/energy in creating theinstallation. This is beneficial not only for faster installs, butalso from an energy consumption perspective, as installers areexpected to be run more often than lockers.

This has led to a design where the locker must do more work upfrontto the benefit installers. It also means the complexity of packagedependencies is simpler and easier to comprehend in a lock files toavoid ambiguity.

Details

Lock files MUST use theTOML file format. This not only prevents theneed to have another file format in the Python packaging ecosystemthanks to its adoption byPEP 518 forpyproject.toml, but alsoassists in making lock files more human-readable.

Lock files MUST end their file names with.pylock.toml. The.toml part unambiguously distinguishes the format of the file,and helps tools like code editors support the file appropriately. The.pylock part distinguishes the file from other TOML files the userhas, to make the logic easier for tools to create functionalityspecific to Python lock files, instead of TOML files in general.

The following sections are the top-level keys of the TOML file dataformat. Any field not listed asrequired is considered optional.

Example

version="1.0"created-at=2021-10-19T22:33:45.520739+00:00[tool]# Tool-specific table.[metadata]requires=["mousebender","coveragepy[toml]"]marker="sys_platform == 'linux'"# As an example for coverage.requires-python=">=3.7"[[package.attrs."21.2.0"]]filename="attrs-21.2.0-py2.py3-none-any.whl"hashes.sha256="149e90d6d8ac20db7a955ad60cf0e6881a3f20d37096140088356da6c716b0b1"url="https://files.pythonhosted.org/packages/20/a9/ba6f1cd1a1517ff022b35acd6a7e4246371dfab08b8e42b829b6d07913cc/attrs-21.2.0-py2.py3-none-any.whl"requires-python=">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*"[[package.attrs."21.2.0"]]# If attrs had another wheel file (e.g. that was platform-specific),# it could be listed here.[[package."coveragepy[toml]"."6.2.0"]]filename="coverage-6.2-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl"hashes.sha256="c7912d1526299cb04c88288e148c6c87c0df600eca76efd99d84396cfe00ef1d"url="https://files.pythonhosted.org/packages/da/64/468ca923e837285bd0b0a60bd9a287945d6b68e325705b66b368c07518b1/coverage-6.2-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl"requires-python=">=3.6"requires=["tomli"][[package."coveragepy[toml]"."6.2.0"]]filename="coverage-6.2-cp310-cp310-musllinux_1_1_x86_64.whl "hashes.sha256="276651978c94a8c5672ea60a2656e95a3cce2a3f31e9fb2d5ebd4c215d095840"url="https://files.pythonhosted.org/packages/17/d6/a29f2cccacf2315150c31d8685b4842a6e7609279939a478725219794355/coverage-6.2-cp310-cp310-musllinux_1_1_x86_64.whl"requires-python=">=3.6"requires=["tomli"]# More wheel files for `coverage` could be listed for more# extensive support (i.e. all Linux-based wheels).[[package.mousebender."2.0.0"]]filename="mousebender-2.0.0-py3-none-any.whl"hashes.sha256="a6f9adfbd17bfb0e6bb5de9a27083e01dfb86ed9c3861e04143d9fd6db373f7c"url="https://files.pythonhosted.org/packages/f4/b3/f6fdbff6395e9b77b5619160180489410fb2f42f41272994353e7ecf5bdf/mousebender-2.0.0-py3-none-any.whl"requires-python=">=3.6"requires=["attrs","packaging"][[package.packaging."20.9"]]filename="packaging-20.9-py2.py3-none-any.whl"hashes.blake-256="3e897ea760b4daa42653ece2380531c90f64788d979110a2ab51049d92f408af"hashes.sha256="67714da7f7bc052e064859c05c595155bd1ee9f69f76557e21f051443c20947a"url="https://files.pythonhosted.org/packages/3e/89/7ea760b4daa42653ece2380531c90f64788d979110a2ab51049d92f408af/packaging-20.9-py2.py3-none-any.whl"requires-python=">=3.6"requires=["pyparsing"][[package.pyparsing."2.4.7"]]filename="pyparsing-2.4.7-py2.py3-none-any.whl"hashes.sha256="ef9d7589ef3c200abe66653d3f1ab1033c3c419ae9b9bdb1240a85b024efc88b"url="https://files.pythonhosted.org/packages/8a/bb/488841f56197b13700afd5658fc279a2025a39e22449b7cf29864669b15d/pyparsing-2.4.7-py2.py3-none-any.whl"direct=true# For demonstration purposes.requires-python=">=2.6, !=3.0.*, !=3.1.*, !=3.2.*"[[package.tomli."2.0.0"]]filename="tomli-2.0.0-py3-none-any.whl"hashes.sha256="b5bde28da1fed24b9bd1d4d2b8cba62300bfb4ec9a6187a957e8ddb9434c5224"url="https://files.pythonhosted.org/packages/e2/9f/5e1557a57a7282f066351086e78f87289a3446c47b2cb5b8b2f614d8fe99/tomli-2.0.0-py3-none-any.whl"requires-python=">=3.7"

Expectations for Lockers

Lockers MUST create lock files for which a topological sort of thepackages which qualify for installation on the specified platformresults in a graph for which only a single version of any packagequalifies for installation and there is at least one compatible fileto install for each package. This leads to a lock file for anysupported platform where the only decision an installer can makeis what the “best-fitting” wheel is to install (which is discussedbelow).

Lockers are expected to utilizemetadata.marker,metadata.tag,andmetadata.requires-python as appropriate as well as environmentmarkers specified viarequires and Python version requirements viarequires-python to enforce this result for installers. Put anotherway, the information used in the lock file is not expected to bepristine/raw from the locker’s input and instead is to be changed asnecessary to the benefit of the locker’s goals.

Expectations for Installers

The expected algorithm for resolving what to install is:

1. Construct a dependency graph based on the data in the lock filewithmetadata.requires as the starting/root point.
2. Eliminate all files that are unsupported by the specified platform.
3. Eliminate all irrelevant edges between packages based on markerevaluation forrequires.
4. Raise an error if a package version is still reachable from theroot of the dependency graph but lacks any compatible file.
5. Verify that all packages left only have one version to install,raising an error otherwise.
6. Install the best-fitting wheel file for each package whichremains.

Installers MUST follow a deterministic algorithm determine what the“best-fitting wheel file” is. A simple solution for this is torely upon thepackaging projectand itspackaging.tags module to determine wheel file precedence.

Installers MUST support installing into an empty environment.Installers MAY support installing into an environment that alreadycontains installed packages (and whatever that would entail to besupported).

Usability

First, apipfreeze equivalent tool could be developed whichcreates a lock file. While installed packages do not by themselvesprovide enough information to statically create a lock file, a usercould provide local directories and index URLs to construct one. Thiswould then lead to lock files that are stricter than a requirementsfile by limiting the lock file to the current platform. This wouldalso allow people to see whether their environment would bereproducible.

Second, a stand-alone installer should be developed. As therequirements on an installer are much simpler than what pip provides,it should be reasonable to have an installer that is independentlydeveloped.

Third, a tool to convert a pinned requirements file as emitted bypip-tools could be developed. Much like thepipfreeze equivalentoutlined above, some input from the user may be needed. But this toolcould act as a transitioning step for anyone who has an appropriaterequirements file. This could also act as a test before potentiallyhaving pip-tools grow some--lockfile flag to use this PEP.

All of this could be required before the PEP transitions fromconditional acceptance to full acceptance (and give the community achance to test if this PEP is potentially useful).

Interoperability

At this point, the goal would be to increase interoperability betweentools.

First, pip would become an installer. By having the most widely usedinstaller support the format, people can innovate on the locker sidewhile knowing people will have the tools necessary to actually consumea lock file.

Second, pip becomes a locker. Once again, pip’s reach would make theformat accessible for the vast majority of Python users very quickly.

Third, a project with a pre-existing lock file format supports atleast exporting to the lock file format (e.g. PDM or Pyflow). Thiswould show that the format meets the needs of other projects.

Acceptance

With the tooling available throughout the community, acceptance wouldbe shown via those not exclusively tied to the Python communitysupporting the file format based on what they believe their userswant.

First, tools that operate on requirements files like code editorshaving equivalent support for lock files.

Second, consumers of requirements files like cloud providers wouldalso accept lock files.

At this point the PEP would have permeated out far enough to be onpar with requirements files in terms of general acceptance andpotentially more if projects had dropped their own lock files for thisPEP.

File Formats Other Than TOML

JSON was briefly considered, but due to:

1. TOML already being used forpyproject.toml
2. TOML being more human-readable
3. TOML leading to better diffs

the decision was made to go with TOML. There was some concern overPython’s standard library lacking a TOML parser, but most packagingtools already use a TOML parser thanks topyproject.toml so thisissue did not seem to be a showstopper. Some have also argued againstthis concern in the past by the fact that if packaging tools abhorinstalling dependencies and feel they can’t vendor a package then thepackaging ecosystem has much bigger issues to rectify than the need todepend on a third-party TOML parser.

Alternative Naming Schemes

Specifying a directory to install file to was considered, butultimately rejected due to people’s distaste for the idea.

It was also suggested to not have a special file name suffix, but itwas decided that hurt discoverability by tools too much.

Supporting a Single Lock File

At one point the idea of only supporting single lock file whichcontained all possible lock information was considered. But it quicklybecame apparent that trying to devise a data format which couldencompass both a lock file format which could support multipleenvironments as well as strict lock outcomes forreproducible builds would become quite complex and cumbersome.

The idea of supporting a directory of lock files as well as a singlelock file namedpyproject-lock.toml was also considered. But anypossible simplicity from skipping the directory in the case of asingle lock file seemed unnecessary. Trying to define appropriatelogic for what should be thepyproject-lock.toml file and whatshould go intopyproject-lock.d seemed unnecessarily complicated.

Using a Flat List Instead of a Dependency Graph

The first version of this PEP proposed that the lock file have noconcept of a dependency graph. Instead, the lock file would listexactly what should be installed for a specific platform such thatinstallers did not have to make any decisions aboutwhat to install,only validating that the lock file would work for the target platform.

This idea was eventually rejected due to the number of combinationsof potentialPEP 508 environment markers. The decision was made thattrying to have lockers generate all possible combinations asindividual lock files when a project wants to be cross-platform wouldbe too much.

Use Wheel Tags in the File Name

Instead of having themetadata.tag field there was a suggestionof encoding the tags into the file name. But due to the addition ofthemetadata.marker field and what to do when no tags were needed,the idea was dropped.

Alternative Names forrequires

Some other names for what becamerequires wereinstalls,needs, anddependencies. Initially this PEP choseneedsafter asking a Python beginner which term they preferred. But basedon feedback on an earlier draft of this PEP,requires was chosenas the term.

Accepting PEP 650

PEP 650 was an earlier attempt at trying to tackle this problem byspecifying an API for installers instead of standardizing on a lockfile format (alaPEP 517). Theinitial responsetoPEP 650 could be considered mild/lukewarm. People seemed to beconsistently confused over which tools should provide whatfunctionality to implement the PEP. It also potentially incurred moreoverhead as it would require executing Python APIs to perform anyactions involving packaging.

This PEP chooses to standardize around an artifact instead of an API(alaPEP 621). This would allow for more tool integrations as itremoves the need to specifically use Python to do things such ascreate a lock file, update it, or even install packages listed ina lock file. It also allows for easier introspection by forcingdependency graph details to be written in a human-readable format.It also allows for easier sharing of knowledge by standardizing whatpeople need to know more (e.g. tutorials become more portable betweentools when it comes to understanding the artifact they produce). It’salso simply the approach other language communities have taken andseem to be happy with.

Acceptance of this PEP would meanPEP 650 gets rejected.

Specifying Requirements per Package Instead of per File

An earlier draft of this PEP specified dependencies at the packagelevel instead of per file. While this has traditionally been howpackaging systems work, it actually did not reflect accurately howthings are specified. As such, this PEP was subsequently updated toreflect the granularity that dependencies can truly be specified at.

Specify Where Lockers Gather Input

This PEP does not specify how a locker gets its input. An initialsuggestion was to partially reusePEP 621, but due to disagreementson how flexible the potential input should be in terms of specifyingthings such as indexes, etc., it was decided this would best be leftto a separate PEP.

Allowing Source Distributions and Source Trees to be an Opt-In, Supported File Format

Afterextensive discussion,it was decided that this PEP would not support source distributions(aka sdists) or source trees as an acceptable format for code.Introducing sdists and source trees to this PEP would immediately undothe reproducibility and security goals due to needing to execute codeto build the sdist or source tree. It would also greatly increasethe complexity for (at least) installers as the dynamic build natureof sdists and source trees means the installer would need to handlefully resolving whatever requirements the sdists produced dynamically,both from a building and installation perspective.

Due to all of this, it was decided it was best to have a separatediscussion about what supporting sdists and source treesafterthis PEP is accepted/rejected. As the proposed file format isversioned, introducing sdists and source tree support in a later PEPis doable.

It should be noted, though, that this PEP isnot stop anout-of-band solution from being developed to be used in conjunctionwith this PEP. Building wheel files from sdists and shipping them withcode upon deployment so they can be included in the lock file is oneoption. Another is to use a requirements filejust for sdists andsource trees, then relying on a lock file for all wheels.