Important
This PEP is a historical document. The up-to-date, canonical spec,Recording the Direct URL Origin of installed distributions, is maintained on thePyPA specs page.
×
See thePyPA specification update process for how to propose changes.
FollowingPEP 440, a distribution can be identified by a name and either aversion, or a direct URL reference (seePEP440 Direct References).After installation, the name and version are captured in the project metadata,but currently there is no way to obtain details of the URL used when thedistribution was identified by a direct URL reference.
This proposal definesadditional metadata, to be added to the installed distribution by theinstallation front end, which records the Direct URL Origin for use byconsumers which introspect the database of installed packages (seePEP 376).
The original motivation of this PEP was to permit tools with a “freeze”operation allowing a Python environment to be recreated to work in a broaderrange of situations.
Specifically, the PEP originated from the desire to addresspip issue #609:i.e. improving the behavior ofpipfreeze in the presence of distributionsinstalled from direct URL references. It follows athread on discuss.python.org about the best course of action to implementit.
Python installers such as pip are capable of downloading and installingdistributions from package indexes. They are also capable of downloadingand installing source code from requirements specifying arbitrary URLs ofsource archives and Version Control Systems (VCS) repositories,as standardized inPEP440 Direct References.
In other words, two relevant installation modes exist.
In this case, the installer looks in a package index (or optionallyusing--find-linksin the case of pip) to find the distribution to install.
In this case, the installer downloads whatever is specified by the URL(typically a wheel, a source archive or a VCS repository) and installs it.In this mode, installers typically download the source code in atemporary directory, invoke thePEP 517 build backend to produce a wheelif needed, install the wheel, and delete the temporary directory.
After installation, no trace of the URL the user requested to download thepackage is left on the user system.
Pip also sports a command namedpipfreeze which examines the Database ofInstalled Python Distributions to generate a list of requirements. The maingoal of this command is to help users generating a list of requirements thatwill later allow the re-installation the same environment with the highestpossible fidelity.
As of pip version 19.3, thepipfreeze command outputs aname==versionline for each installeddistribution (except for editable installs). To achieve the goal ofreinstalling the same environment, this requires the (name, version)tuple to refer to an immutable version of thedistribution. The immutability is guaranteed by package indexessuch as Warehouse. The package index to use is typically known fromenvironmental or command line parameters of the installer.
This freeze mechanism therefore works fine for installation mode 1 (i.e.when the package to install was specified as a name plus version specifier).
For installation mode 2, i.e. when the package to install was specified as adirect URL reference, thename==version tuple is obviously not sufficientto reinstall the same distribution and users of the freeze command expect itto output the URL that was originally requested.
The reasoning above is equally applicable to tools, other thanpipfreeze,that would attempt to generate aPipfile.lock or any other similar formatfrom the Database of Installed Python Distributions. Unless specifiedotherwise, “freeze” is used in this document as a generic term for suchan operation.
For an application integrator, it is important to be able to reliably installand freeze unreleased version of python distributions.For instance when a developer needs to deploy an unreleased patched versionof a dependency, it is common to install the dependency directly from a VCSbranch that has the patch, while waiting for the maintainer to release anupdated version.
In such cases, it is important for “freeze” to pin the exact VCSreference (commit-hash if available) that was installed, in order to createreproducible builds with the highest possible fidelity.
For VCS URLs, there is additional origin information available only atinstall time useful for introspection and certain workflows. For example,when installing a revision from a VCS URL, a tool can determine if therevision corresponds to a branch, tag or (in the case of Git) a ref. Thisinformation can be used when introspecting the Database of Installed Distributionsto communicate to users more information about what version was installed(e.g. whether a branch or tag was installed and, if so, the name of thebranch or tag). This also permits one to know whether aPEP 440 directreference URL can be constructed using the tag form, as only tags have thesemantics of immutability.
In cases where the revision is mutable (e.g. branches and Git refs), knowingthis information enables workflows where users can e.g. update to the latestversion of a branch they are tracking, or update to the latest version of apull request they are reviewing locally. In contrast, when the revision is atag, tools can know in advance (e.g. without network calls) that no update isneeded.
As with the URL itself, if this information isn’t recorded at install timewhen the VCS repository is available, it would otherwise be lost.
The editable installation mode of pip roughly lets a user insert alocal directory in sys.path for development purpose. This mode is somewhatabused to work around the fact that a non-editable install from a VCS URLloses track of the origin after installation.Indeed, editable installs implicitly record the VCS origin in the checkoutdirectory, so the information can be recovered when running “freeze”.
The use of this workaround, although useful, is fragile, creates confusionabout the purpose of the editable mode, and works only when the distributioncan be installed with setuptools (i.e. it is not usable with otherPEP 517build backends).
When this PEP is implemented, it will not be necessary anymore to useeditable installs for the purpose of making pip freeze work correctly withVCS references.
This PEP specifies a newdirect_url.json metadata file in the.dist-info directory of an installed distribution.
The fields specified are sufficient to reproduce the source archive andVCSURLs supported by pip. They are also sufficient to reproducePEP440 Direct References,as well asPipfile and Pipfile.lock entries. Finally, theyare sufficient to record the branch, tag, and/or Git ref origin of theinstalled version that is already available for editable installs by virtueof a VCS checkout being present.
Since at least three different ways already exist to encode this type ofinformation, this PEP uses a dictionary format, so as not to make anyassumption on how a directURL reference must ultimately be encoded in a requirement or lockfile. See alsotheAlternatives section below for more discussion about this choice.
Information has been taken from Ruby’s bundler manual to verify it has similarcapabilities and inform the selection and naming of fields in thisspecifications.
The JSON format allows for the addition of additional fields in the future.
This PEP specifies adirect_url.json file in the.dist-info directoryof an installed distribution, to record the Direct URL Origin of the distribution.
The canonical source for the name and semantics of this metadata file istheRecording the Direct URL Origin of installed distributions document.
This file MUST be created by installers when installing a distributionfrom a requirement specifying a direct URL reference (including a VCS URL).
This file MUST NOT be created when installing a distribution from an othertype of requirement (i.e. name plus version specifier).
This JSON file MUST be a dictionary, compliant withRFC 8259 and UTF-8encoded.
If present, it MUST contain at least two fields. The first one isurl, withtypestring. Depending on whaturl refers to, the second field MUST beone ofvcs_info (ifurl is a VCS reference),archive_info (ifurl is a source archives or a wheel), ordir_info (ifurl is alocal directory). These info fields have a (possibly empty) subdictionary asvalue, with the possible keys defined below.
url MUST be stripped of any sensitive authentication information,for security reasons.
The user:password section of the URL MAY howeverbe composed of environment variables, matching the following regularexpression:
\$\{[A-Za-z0-9-_]+\}(:\$\{[A-Za-z0-9-_]+\})?Additionally, the user:password section of the URL MAY be awell-known, non security sensitive string. A typical example isgitin the case of an URL such asssh://git@gitlab.com.
Whenurl refers to a VCS repository, thevcs_info key MUST be presentas a dictionary with the following keys:
vcs key (typestring) MUST be present, containing the name of the VCS(i.e. one ofgit,hg,bzr,svn). Other VCS’s SHOULD be registered bywriting a PEP to amend this specification.Theurl value MUST be compatible with the corresponding VCS,so an installer can hand it off without transformation to acheckout/download command of the VCS.requested_revision key (typestring) MAY be present naming abranch/tag/ref/commit/revision/etc (in a format compatible with the VCS)to install.commit_id key (typestring) MUST be present, containing theexact commit/revision number that was installed.If the VCS supports commit-hashbased revision identifiers, such commit-hash MUST be used ascommit_id in order to reference the immutableversion of the source code that was installed.resolved_revision and/orresolved_revision_type field. If no revision was provided inthe requested URL,resolved_revision MAY contain the default branchthat was installed, andresolved_revision_type will bebranch.If the installer determines thatrequested_revision was a tag, it MAYaddresolved_revision_type with valuetag.Whenurl refers to a source archive or a wheel, thearchive_info keyMUST be present as a dictionary with the following key:
hash key (typestring) SHOULD be present, with value<hash-algorithm>=<expected-hash>.It is RECOMMENDED that only hashes which are unconditionally provided bythe latest version of the standard library’shashlib module be used forsource archive hashes. At time of writing, that list consists of ‘md5’,‘sha1’, ‘sha224’, ‘sha256’, ‘sha384’, and ‘sha512’.Whenurl refers to a local directory, thedir_info key MUST bepresent as a dictionary with the following key:
editable (type:boolean):true if the distribution was installedin editable mode,false otherwise. If absent, default tofalse.Whenurl refers to a local directory, it MUST have thefile schemeand be compliant withRFC 8089. In particular, the path component mustbe absolute. Symbolic links SHOULD be preserved when making relativepaths absolute.
Note
When the requested URL has thefile:// scheme and points to a local directory that happens to contain aVCS checkout, installers MUST NOT attempt to infer any VCS information andtherefore MUST NOT output any VCS related information (such asvcs_info)indirect_url.json.
A top-levelsubdirectory field MAY be present containing a directory path,relative to the root of the VCS repository, source archive or local directory,to specify wherepyproject.toml orsetup.py is located.
Note
As a general rule, installers should as much as possible preserve theinformation that was provided in the requested URL when generatingdirect_url.json. For example, user:password environment variablesshould be preserved andrequested_revision should reflect the revision that wasprovided in the requested URL as faithfully as possible. This information ishoweverenriched with more precise data, such ascommit_id.
This section lists the registered VCS’s; expanded, VCS-specific informationon how to use thevcs,requested_revision, and other fields ofvcs_info; and insome cases additional VCS-specific fields.Tools MAY support other VCS’s although it is RECOMMENDED to registerthem by writing a PEP to amend this specification. Thevcs field SHOULD be the command name(lowercased). Additional fields that would be necessary tosupport such VCS SHOULD be prefixed with the VCS command name.
Home page
vcs command
git
vcs field
git
requested_revision field
A tag name, branch name, Git ref, commit hash, shortened commit hash,or other commit-ish.
commit_id field
A commit hash (40 hexadecimal characters sha1).
Note
Installers can use thegitshow-ref andgitsymbolic-ref commandsto determine if therequested_revision corresponds to a Git ref.In turn, a ref beginning withrefs/tags/ corresponds to a tag, anda ref beginning withrefs/remotes/origin/ after cloning correspondsto a branch.
Home page
vcs command
hg
vcs field
hg
requested_revision field
A tag name, branch name, changeset ID, shortened changeset ID.
commit_id field
A changeset ID (40 hexadecimal characters).
Home page
vcs command
bzr
vcs field
bzr
requested_revision field
A tag name, branch name, revision id.
commit_id field
A revision id.
Home page
vcs command
svn
vcs field
svn
requested_revision field
requested_revisionmust be compatible withsvncheckout--revisionoption.In Subversion, branch or tag is part ofurl.
commit_id field
Since Subversion does not support globally unique identifiers,this field is the Subversion revision number in the correspondingrepository.
Source archive:
{"url":"https://github.com/pypa/pip/archive/1.3.1.zip","archive_info":{"hash":"sha256=2dc6b5a470a1bde68946f263f1af1515a2574a150a30d6ce02c6ff742fcc0db8"}}
Git URL with tag and commit-hash:
{"url":"https://github.com/pypa/pip.git","vcs_info":{"vcs":"git","requested_revision":"1.3.1","resolved_revision_type":"tag","commit_id":"7921be1537eac1e97bc40179a57f0349c2aee67d"}}
Local directory:
{"url":"file:///home/user/project","dir_info":{}}
Local directory installed in editable mode:
{"url":"file:///home/user/project","dir_info":{"editable":true}}
Commands that generate adirect_url.json:
url will be the local directory where the git repository has beencloned to, anddir_info will be present with"editable":true and novcs_info will be set)Commands thatdo not generate adirect_url.json
“Freezing” an environment
Tools, such aspipfreeze, which generate requirements from the Databaseof Installed Python Distributions SHOULD exploitdirect_url.jsonif it is present, and give it priority over the Version metadata in orderto generate a higher fidelity output. In the presence of avcsdirect URL reference,thecommit_idfield SHOULD be used in priority in order to providethe highest possible fidelity to the originally installed version. Ifsupported by their requirement format, tools are encouraged also to outputthetagvalue if present, as it has immutable semantics.Tools MAY choose another approach, depending on the needs of their users.Note the initial iteration of this PEP does not attempt to make environmentsthat include editable installs or installs from local directoriesreproducible, but it does attempt to make them readily identifiable. Bylocating the local project directory via the
urlanddir_infofieldsof this specification, tools can implement any strategy that fits their usecases.
Since this PEP specifies a new file in the.dist-info directory,there are no backwards compatibility implications.
The now withdrawnPEP 426 specifies asource_url metadata entry.It is also implemented indistlib.
It was intended for a slightly different purpose, for use in sdists.
This format lacks support for thesubdirectory option of pip requirementURLs. The same limitation is present inPEP440 Direct References.
It also lacks explicit support forenvironment variables in the user:passwordpart of URLs.
The introduction of a key/value extensibility mechanism and supportfor environment variables for user:password inPEP 440, would be necessaryfor use in this PEP.
Therequested_revision key was retained overrequested_ref as it is a more generic termacross various VCS andref has a specific meaning forgit.
Various people helped make this PEP a reality. Paul F. Moore provided theessence of the abstract. Alyssa Coghlan suggested the direct_url name.
This document is placed in the public domain or under theCC0-1.0-Universal license, whichever is more permissive.
Source:https://github.com/python/peps/blob/main/peps/pep-0610.rst
Last modified:2025-02-01 08:55:40 GMT