Important
This PEP is a historical document. The up-to-date, canonical spec,Core metadata specifications, is maintained on thePyPA specs page.
×
See thePyPA specification update process for how to propose changes.
Important
This PEP has been withdrawn.
×
The ground-up metadata redesign proposed in this PEP has been withdrawn infavour of the more modest proposal inPEP 566, which retains the basicKey:Value format of previous metadata versions, but also defines a standardisedmechanism for translating that format to nested JSON-compatible data structures.
Some of the ideas in this PEP (or the relatedPEP 459) may still be consideredas part of later proposals, but they will be handled in a more incrementalfashion, rather than as a single large proposed change with no feasiblemigration plan.
This PEP describes a mechanism for publishing and exchanging metadatarelated to Python distributions. It includes specifics of the field names,and their semantics and usage.
This document specifies the never released version 2.0 of the metadata format.
Version 1.0 is specified inPEP 241.Version 1.1 is specified inPEP 314.Version 1.2 is specified inPEP 345.
Version 2.0 of the metadata format proposed migrating from directly defining acustom key-value file format to instead defining a JSON-compatible in-memoryrepresentation that may be used to define metadata representation in othercontexts (such as API and archive format definitions).
This version also defines a formal extension mechanism, allowing newfields to be added for particular purposes without requiring updates tothe core metadata format.
This PEP was initially deferred for an extended period, from December 2013through to March 2017, as distutils-sig worked through a number of otherchanges. These changes included:
wheel) inPEP 427pyproject.toml) inPEP 518pip with CPython by default inPEP 453, and backporting thataddition to Python 2.7 inPEP 477The time spent pursuing these changes provided additional perspective on whichmetadata format changes were genuinely desirable, and which could be omittedfrom the revised specification as merely being “change for change’s sake”.
It also allowed a number of features that aren’t critical to the core activityof publishing and distributing software to be moved out toPEP 459, a separateproposal for a number of standard metadata extensions that provide additionaloptional information about a release.
As of September 2017, it was deferred again, on the grounds thatit doesn’t actually help solve any particularly pressing problems:
Finally, the PEP was withdrawn in February 2018 in favour ofPEP 566 (whichpursues that more incremental strategy).
The purpose of this PEP is to define a common metadata interchange formatfor communication between software publication tools and software integrationtools in the Python ecosystem. One key aim is to support full dependencyanalysis in that ecosystem without requiring the execution of arbitraryPython code by those doing the analysis. Another aim is to encourage goodsoftware distribution practices by default, while continuing to support thecurrent practices of almost all existing users of the Python Package Index(both publishers and integrators). Finally, the aim is to support an upgradepath from the currently in use metadata formats that is transparent to endusers.
The design draws on the Python community’s nearly 20 years of experience withdistutils based software distribution, and incorporates ideas and conceptsfrom other distribution systems, including Python’s setuptools, pip andother projects, Ruby’s gems, Perl’s CPAN, Node.js’s npm, PHP’s composerand Linux packaging systems such as RPM and APT.
While the specifics of this format are aimed at the Python ecosystem, someof the ideas may also be useful in the future evolution of other dependencymanagement ecosystems.
The metadata design in this PEP is based on a particular conceptual modelof the software development and distribution process. This model consists ofthe following phases:
The publication and integration phases are collectively referred to asthe distribution phase, and the individual software components distributedin that phase are formally referred to as “distribution packages”, but are morecolloquially known as just “packages” (relying on context to disambiguate themfrom the “module with submodules” kind of Python package).
The exact details of these phases will vary greatly for particular use cases.Deploying a web application to a public Platform-as-a-Service provider,publishing a new release of a web framework or scientific library,creating an integrated Linux distribution, or upgrading a custom applicationrunning in a secure enclave are all situations this metadata design shouldbe able to handle.
The complexity of the metadata described in this PEP thus arises directlyfrom the actual complexities associated with software development,distribution and deployment in a wide range of scenarios.
The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”,“SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “MAY”, and “OPTIONAL” in thisdocument are to be interpreted as described inRFC 2119.
“Projects” are software components that are made available for integration.Projects include Python libraries, frameworks, scripts, plugins,applications, collections of data or other resources, and variouscombinations thereof. Public Python projects are typically registered onthePython Package Index.
“Releases” are uniquely identified snapshots of a project.
“Distribution packages” are the packaged files which are used to publishand distribute a release.
Depending on context, “package” may refer to either a distribution, orto an importable Python module that has a__path__ attribute and hencemay also have importable submodules.
“Source archive” and “VCS checkout” both refer to the raw source code fora release, prior to creation of an sdist or binary archive.
An “sdist” is a publication format providing the distribution metadata andany source files that are essential to creating a binary archive forthe distribution. Creating a binary archive from an sdist requires thatthe appropriate build tools be available on the system.
“Binary archives” only require that prebuilt files be moved to the correctlocation on the target system. As Python is a dynamically boundcross-platform language, many so-called “binary” archives will contain onlypure Python source code.
“Contributors” are individuals and organizations that work together todevelop a software component.
“Publishers” are individuals and organizations that make software componentsavailable for integration (typically by uploading distributions to anindex server)
“Integrators” are individuals and organizations that incorporate publisheddistributions as components of an application or larger system.
“Build tools” are automated tools intended to run on development systems,producing source and binary distribution archives. Build tools may also beinvoked by integration tools in order to build software distributed assdists rather than prebuilt binary archives.
“Index servers” are active distribution registries which publish version anddependency metadata and place constraints on the permitted metadata.
“Public index servers” are index servers which allow distribution uploadsfrom untrusted third parties. ThePython Package Index is a public indexserver.
“Publication tools” are automated tools intended to run on developmentsystems and upload source and binary distribution archives to index servers.
“Integration tools” are automated tools that consume the metadata anddistribution archives published by an index server or other designatedsource, and make use of them in some fashion, such as installing them orconverting them to a platform specific packaging format.
“Installation tools” are integration tools specifically intended to run ondeployment targets, consuming source and binary distribution archives froman index server or other designated location and deploying them to the targetsystem.
“Automated tools” is a collective term covering build tools, index servers,publication tools, integration tools and any other software that producesor consumes distribution version and dependency metadata.
“Legacy metadata” refers to earlier versions of this metadata specification,along with the supporting metadata file formats defined by thesetuptools project.
“Distro” is used as the preferred term for Linux distributions, to helpavoid confusion with the Python-specific use of the term “distributionpackage”.
“Qualified name” is a dotted Python identifier. For imported modules andpackages, the qualified name is available as the__name__ attribute,while for functions and classes it is available as the__qualname__attribute.
A “fully qualified name” uniquely locates an object in the Python modulenamespace. For imported modules and packages, it is the same as thequalified name. For other Python objects, the fully qualified name consistsof the qualified name of the containing module or package, a colon (:)and the qualified name of the object relative to the containing module orpackage.
A “prefixed name” starts with a qualified name, but is not necessarily aqualified name - it may contain additional dot separated segments which arenot valid identifiers.
The primary purpose of the distribution metadata is to support integrationand deployment of distributions as part of larger applications and systems.
Integration and deployment can in turn be broken down into further substeps.
These three steps may all occur directly on the target system. Alternativelythe build step may be separated out by using binary archives provided by thepublisher of the distribution, or by creating the binary archives on aseparate system prior to deployment. The advantage of the latter approachis that it minimizes the dependencies that need to be installed ondeployment targets (as the build dependencies will be needed only on thebuild systems).
The published metadata for distribution packages SHOULD allow integrators, withthe aid of build and integration tools, to:
The secondary purpose of the distribution metadata is to support effectivecollaboration amongst software contributors and publishers during thedevelopment phase.
The published metadata for distributions SHOULD allow contributorsand publishers, with the aid of build and publication tools, to:
The format defined in this PEP is an in-memory representation of Pythondistribution metadata as a string-keyed dictionary. Permitted values forindividual entries are strings, lists of strings, and additionalnested string-keyed dictionaries.
Except where otherwise noted, dictionary keys in distribution metadata MUSTbe valid Python identifiers in order to support attribute based metadataaccess APIs.
The individual field descriptions show examples of the key name and valueas they would be serialised as part of a JSON mapping.
Unless otherwise indicated, the fields identified as core metadata are required.Automated tools MUST NOT accept distributions with missing core metadata asvalid Python distributions.
All other fields are optional. Automated tools MUST operate correctlyif a distribution does not provide them, except for those operationswhich specifically require the omitted fields.
Automated tools MUST NOT insert dummy data for missing fields. If a validvalue is not provided for a required field then the metadata and theassociated distribution MUST be rejected as invalid. If a valid valueis not provided for an optional field, that field MUST be omitted entirely.Automated tools MAY automatically derive valid values from otherinformation sources (such as a version control system).
Automated tools, especially public index servers, MAY impose additionallength restrictions on metadata beyond those enumerated in this PEP. Suchlimits SHOULD be imposed where necessary to protect the integrity of aservice, based on the available resources and the service provider’sjudgment of reasonable metadata capacity requirements.
The information defined in this PEP is serialised topysdist.jsonfiles for some use cases. These are files containing UTF-8 encoded JSONmetadata.
Each metadata file consists of a single serialised mapping, with fields asdescribed in this PEP. When serialising metadata, automated tools SHOULDlexically sort any keys and list elements in order to simplify reviewsof any changes.
There are expected to be three standard locations for these metadata files:
{distribution}-{version}.dist-info/pysdist.json file in ansdist source distribution archive{distribution}-{version}.dist-info/pysdist.json file in awheelbinary distribution archive{distribution}-{version}.dist-info/pysdist.json file in a localPython installation databaseThis file is expected to be identical in all three locations - it isgenerated when creating a source archive or binary archive from a sourcetree, and then preserved unchanged on installation, or when building abinary archive from a source archive.
Note
These locations are to be confirmed, since they depend on the definitionof sdist 2.0 and the revised installation database standard. There willalso be a wheel 1.1 format update after this PEP is approved thatmandates provision of 2.0+ metadata.
Note that these metadata files MAY be processed even if the version of thecontaining location is too low to indicate that they are valid. Specifically,unversionedsdist archives, unversioned installation database directoriesand version 1.0 of thewheel specification may still providepysdist.json files.
Note
Until this specification is formally marked as Active, it is recommendedthat tools following the draft format use an alternative filename likemetadata.json orpep426-20131213.json to avoid colliding withthe eventually standardised files.
Other tools involved in Python distribution MAY also use this format.
Note that these metadata files are generated by build tools based on otherinput formats (such assetup.py andpyproject.toml) rather than beingused directly as a data input format. Generating the metadata as part of thepublication process also helps to deal with version specific fields (includingthe source URL and the version field itself).
For backwards compatibility with older installation tools, metadata 2.0files MAY be distributed alongside legacy metadata.
Index servers MAY allow distributions to be uploaded and installation toolsMAY allow distributions to be installed with only legacy metadata.
Automated tools MAY attempt to automatically translate legacy metadata tothe format described in this PEP. Advice for doing so effectively is givenin Appendix A.
Ajsonschema description ofthe distribution metadata isavailable.
This schema does NOT currently handle validation of some of the more complexstring fields (instead treating them as opaque strings).
Except where otherwise noted, all URL fields in the metadata MUST complywithRFC 3986.
Note
The current version of the schema file covers the previous draft of thePEP, and has not yet been updated for the split into the essentialdependency resolution metadata and multiple standard extensions, and norhas it been updated for the various other differences between the currentdraft and the earlier drafts.
This section specifies the core metadata fields that are required for everyPython distribution.
Publication tools MUST ensure at least these fields are present whenpublishing a distribution.
Index servers MUST ensure at least these fields are present in the metadatawhen distributions are uploaded.
Installation tools MUST refuse to install distributions with one or moreof these fields missing by default, but MAY allow users to force such aninstallation to occur.
Version of the file format;"2.0" is the only legal value.
Automated tools consuming metadata SHOULD warn ifmetadata_version isgreater than the highest version they support, and MUST fail ifmetadata_version has a greater major version than the highestversion they support (as described inPEP 440, the major version is thevalue before the first dot).
For broader compatibility, build tools MAY choose to producedistribution metadata using the lowest metadata version that includesall of the needed fields.
Example:
"metadata_version":"2.0"
Name (and optional version) of the program that generated the file,if any. A manually produced file would omit this field.
Examples:
"generator":"flit""generator":"setuptools (34.3.1)"
The name of the distribution, as defined inPEP 508.
As distribution names are used as part of URLs, filenames, command lineparameters and must also interoperate with other packaging systems, thepermitted characters are constrained to:
[a-zA-Z])[0-9])_)-).)Distribution names MUST start and end with an ASCII letter or digit.
Automated tools MUST reject non-compliant names. A regular expression toenforce these constraints (when run withre.IGNORECASE) is:
^([A-Z0-9]|[A-Z0-9][A-Z0-9._-]*[A-Z0-9])$
All comparisons of distribution names MUST be case insensitive, and MUSTconsider hyphens and underscores to be equivalent.
Index servers MAY consider “confusable” characters (as defined by theUnicode Consortium inTR39: Unicode Security Mechanisms) to beequivalent.
Index servers that permit arbitrary distribution name registrations fromuntrusted sources SHOULD consider confusable characters to be equivalentwhen registering new distributions (and hence reject them as duplicates).
Integration tools MUST NOT silently accept a confusable alternatespelling as matching a requested distribution name.
At time of writing, the characters in the ASCII subset designated asconfusables by the Unicode Consortium are:
1 (DIGIT ONE),l (LATIN SMALL LETTER L), andI (LATIN CAPITALLETTER I)0 (DIGIT ZERO), andO (LATIN CAPITAL LETTER O)Example:
"name":"ComfyChair"
The distribution’s public or local version identifier, as defined inPEP 440.Version identifiers are designed for consumption by automated tools andsupport a variety of flexible version specification mechanisms (seePEP 440for details).
Version identifiers MUST comply with the format defined inPEP 440.
Version identifiers MUST be unique within each project.
Index servers MAY place restrictions on the use of local version identifiersas described inPEP 440.
Example:
"version":"1.0a2"
A short summary of what the distribution does.
This field SHOULD contain fewer than 512 characters and MUST contain fewerthan 2048.
This field SHOULD NOT contain any line breaks.
A more complete description SHOULD be included as a separate file in thesdist for the distribution. Refer to thepython-details extension inPEP 459 for more information.
Example:
"summary":"A module that is more fiendish than soft cushions."
This section specifies fields that provide identifying details for thesource code used to produce this distribution.
All of these fields are optional. Automated tools MUST operate correctly ifa distribution does not provide them, including failing cleanly when anoperation depending on one of these fields is requested.
Source labels are text strings with minimal defined semantics. They areintended to allow the original source code to be unambiguously identified,even if an integrator has applied additional local modifications to aparticular distribution.
To ensure source labels can be readily incorporated as part of file namesand URLs, and to avoid formatting inconsistencies in hexadecimal hashrepresentations they MUST be limited to the following set of permittedcharacters:
[a-z])[0-9])_)-).)+)Source labels MUST start and end with an ASCII letter or digit.
A regular expression to rnforce these constraints (when run withre.IGNORECASE) is:
^([A-Z0-9]|[A-Z0-9][A-Z0-9._-+]*[A-Z0-9])$
A source label for a project MUST NOT match any defined version for thatproject. This restriction ensures that there is no ambiguity between versionidentifiers and source labels.
Examples:
"source_label":"1.0.0-alpha.1""source_label":"1.3.7+build.11.e0f985a""source_label":"v1.8.1.301.ga0df26f""source_label":"2013.02.17.dev123"
A string containing a full URL where the source for this specific version ofthe distribution can be downloaded.
Source URLs MUST be unique within each project. This means that the URLcan’t be something like"https://github.com/pypa/pip/archive/main.zip",but instead must be"https://github.com/pypa/pip/archive/1.3.1.zip".
The source URL MUST reference either a source archive or a tag or specificcommit in an online version control system that permits creation of asuitable VCS checkout. It is intended primarily for integrators thatwish to recreate the distribution from the original source form.
All source URL references SHOULD specify a secure transport mechanism(such ashttps) AND include an expected hash value in the URL forverification purposes. If a source URL is specified without any hashinformation, with hash information that the tool doesn’t understand, orwith a selected hash algorithm that the tool considers too weak to trust,automated tools SHOULD at least emit a warning and MAY refuse to rely onthe URL. If such a source URL also uses an insecure transport, automatedtools SHOULD NOT rely on the URL.
For source archive references, an expected hash value may be specified byincluding a<hash-algorithm>=<expected-hash> entry as part of the URLfragment.
As of 2017, it is RECOMMENDED that'sha256' hashes be used for sourceURLs, as this hash is not yet known to be vulnerable to generation ofmalicious collisions, while also being widely available on client systems.
For version control references, theVCS+protocol scheme SHOULD beused to identify both the version control system and the secure transport,and a version control system with hash based commit identifiers SHOULD beused. Automated tools MAY omit warnings about missing hashes for versioncontrol systems that do not provide hash based commit identifiers.
To handle version control systems that do not support including commit ortag references directly in the URL, that information may be appended to theend of the URL using the@<commit-hash> or the@<tag>#<commit-hash>notation.
Note
This isn’tquite the same as the existing VCS reference notationsupported by pip. Firstly, the distribution name is a separate field ratherthan embedded as part of the URL. Secondly, the commit hash is includedeven when retrieving based on a tag, in order to meet the requirementabove thatevery link should include a hash to make things harder toforge (creating a malicious repo with a particular tag is easy, creatingone with a specifichash, less so).
Example:
"source_url":"https://github.com/pypa/pip/archive/1.3.1.zip#sha256=2dc6b5a470a1bde68946f263f1af1515a2574a150a30d6ce02c6ff742fcc0db8"source_url":"git+https://github.com/pypa/pip.git@1.3.1#7921be1537eac1e97bc40179a57f0349c2aee67d""source_url":"git+https://github.com/pypa/pip.git@7921be1537eac1e97bc40179a57f0349c2aee67d"
Dependency metadata allows published projects to make use of functionalityprovided by other published projects, without needing to bundle copies ofparticular releases of those projects.
Semantic dependencies allow publishers to indicate not only which otherprojects are needed, but alsowhy they’re needed. This additionalinformation allows integrators to install just the dependencies they needfor specific activities, making it easier to minimise installationfootprints in constrained environments (regardless of the reasons forthose constraints).
By default, dependency declarations are assumed to be for“runtime dependencies”: other releases that are needed to actually use thepublished release.
There are also four different kinds of optional dependency that releases maydeclare:
test dependencies: other releases that are needed to run theautomated test suite for this release, but are not needed just touse it (e.g.nose2 orpytest)build dependencies: other releases that are needed to build thisa deployable binary version of this release from source(e.g.flit orsetuptools)doc dependencies: other releases that are needed to build thedocumentation for this distribution (e.g. thesphinx build tool)dev dependencies: other releases that are needed when working on thisdistribution, but do not fit into exactly one of the other optionaldependency categories (e.g.pylint,flake8).dev dependenciesare also effectively considered as combinedtest,build, anddocdependencies, without needing to be listed three timesThese optional categories are known asExtras. In addition to the fourstandard categories, projects may also declare their own custom categoriesin theExtras field.
There are also two standard extra categories that imply dependencies onother extras:
alldev: implies thetest,build,doc,dev extrasall: if not otherwise defined, implies all declared extrasDependency management is heavily dependent on the version identificationand specification scheme defined inPEP 440 and the dependency specification,extra, and environment marker schemes defined inPEP 508.
All of these fields are optional. Automated tools MUST operate correctly ifa distribution does not provide them, by assuming that a missing fieldindicates “Not applicable for this distribution”.
The different categories of dependency are based on the various distributionand development activities identified above, and govern which dependenciesshould be installed for the specified activities:
build extradev extratest extratest extrabuild extradoc extradev extraThe notation described inExtras (optional dependencies) SHOULD be usedto determine exactly what gets installed for various operations.
Installation tools SHOULD report an error if dependencies cannot besatisfied, MUST at least emit a warning, and MAY allow the user to forcethe installation to proceed regardless.
See Appendix B for an overview of mapping these dependencies to an RPMspec file.
A list of optional sets of dependencies that may be used to defineconditional dependencies in dependency fields. SeeExtras (optional dependencies) for details.
The names of extras MUST abide by the same restrictions as those fordistribution names.
The following extra names are available by default and MUST NOT bedeclared explicitly in this field:
allalldevbuilddevdoctestExample:
"extras":["warmup","tea"]
A list of release requirements needed to actually run this release.
Public index servers MAY prohibit strict version matching clauses or directreferences in this field.
Example:
"dependencies":{"requires":["SciPy","PasteDeploy","zope.interface > 3.5.0"]},{"requires":["pywin32 > 1.0"],"environment":"sys_platform == 'win32'"},{"requires":["SoftCushions"],"extra":"warmup"}]
While many dependencies will be needed to use a project release at all, othersare needed only on particular platforms or only when particular optionalfeatures of the release are needed.
To handle this, release dependency specifiers are mappings with the followingsubfields:
requires: a list of requirements needed to satisfy the dependencyextra: the name of a set of optional dependencies that are requestedand installed together. SeeExtras (optional dependencies) for detailsenvironment: an environment marker defining the environment thatneeds these dependencies. The syntax and capabilities of environmentmarkers are defined inPEP 508Individual entries in therequires lists are strings using the dependencydeclaration format defined inPEP 508, with the exception that environmentmarkers MUST NOT be included in the individual dependency declarations, andare instead supplied in the separateenvironment field.
requires is the only required subfield. When it is the only subfield, thedependencies are said to beunconditional. Ifextra orenvironmentis specified, then the dependencies areconditional.
All three fields may be supplied, indicating that the dependencies areneeded only when the named extra is requested in a particular environment.
Automated tools MUST combine related dependency specifiers (those withcommon values forextra andenvironment) into a single specifierlisting multiple requirements when serialising metadata.
Despite this required normalisation, the same extra name or environmentmarker MAY appear in multiple conditional dependencies. This may happen,for example, if an extra itself only needs some of its dependencies inspecific environments. It is only the combination of extras and environmentmarkers that is required to be unique in a list of dependency specifiers.
Aside from the six standard extra categories, any extras referenced from adependency specifier MUST be named in theExtras field for this distribution.This helps avoid typographical errors and also makes it straightforward toidentify the available extras without scanning the full set of dependencies.
To reuse an extra definition as part of another extra, project releases MAYdeclare dependencies on themselves. To avoid infinite recursion in these cases,automated tools MUST special case dependencies from a project back onto itself.
Extensions to the metadata MAY be present in a mapping under theextensions key. The keys MUST be valid prefixed names, whilethe values MUST themselves be nested mappings.
Two key names are reserved and MUST NOT be used by extensions, except asdescribed below:
extension_versioninstaller_must_handleThe following example shows thepython.details andpython.commandsstandard extensions fromPEP 459:
"extensions":{"python.details":{"license":"GPL version 3, excluding DRM provisions","keywords":["comfy","chair","cushions","too silly","monty python"],"classifiers":["Development Status :: 4 - Beta","Environment :: Console (Text Based)","License :: OSI Approved :: GNU General Public License v3 (GPLv3)"],"document_names":{"description":"README.rst","license":"LICENSE.rst","changelog":"NEWS"}},"python.commands":{"wrap_console":[{"chair":"chair:run_cli"}],"wrap_gui":[{"chair-gui":"chair:run_gui"}],"prebuilt":["reduniforms"]},}
Extension names are defined by distributions that will then make use ofthe additional published metadata in some way.
To reduce the chance of name conflicts, extension names SHOULD use aprefix that corresponds to a module name in the distribution that definesthe meaning of the extension. This practice will also make it easier tofind authoritative documentation for metadata extensions.
Metadata extensions allow development tools to record information in themetadata that may be useful during later phases of distribution, but isnot essential for dependency resolution or building the software.
Extensions MUST be versioned, using theextension_version key.However, if this key is omitted, then the implied version is1.0.
Automated tools consuming extension metadata SHOULD warn ifextension_version is greater than the highest version they support,and MUST fail ifextension_version has a greater major version thanthe highest version they support (as described inPEP 440, the majorversion is the value before the first dot).
For broader compatibility, build tools MAY choose to produceextension metadata using the lowest metadata version that includesall of the needed fields.
A project may consider correct handling of some extensions to be essentialto correct installation of the software. This is indicated by setting theinstaller_must_handle field totrue. Setting it tofalse oromitting it altogether indicates that processing the extension wheninstalling the distribution is not considered mandatory by the developers.
Installation tools MUST fail ifinstaller_must_handle is set totruefor an extension and the tool does not have any ability to process thatparticular extension (whether directly or through a tool-specific pluginsystem).
If an installation tool encounters a required extension it doesn’tunderstand when attempting to install from a wheel archive, it MAY fallback on attempting to install from source rather than failing entirely.
As defined inPEP 508, extras are additional dependencies that enable anoptional aspect of a project release, often corresponding to atry:importoptional_dependency... block in the code. They are also used to indicatesemantic dependencies for activities other than normal runtime using (such astesting, building, or working on the component).
To support the use of the release with or without the optional dependencies,they are listed separately from the release’s core runtime dependenciesand must be requested explicitly, either in the dependency specifications ofanother project, or else when issuing a command to an installation tool.
Example of a distribution with optional dependencies:
"name":"ComfyChair","extras":["warmup"]"dependencies":[{"requires":["SoftCushions"],"extra":"warmup"},{"requires":["cython"],"extra":"build"}]
Other distributions require the additional dependencies by placing therelevant extra names inside square brackets after the distribution name whenspecifying the dependency. Multiple extras from a dependency can be requestedby placing to
If the standardall extra has no explicitly declared entries, thenintegration tools SHOULD implicitly define it as a dependency on all of theextras explicitly declared by the project.
If the standardalldev extra has no explicitly declared entries, thenintegration tools SHOULD implicitly define it as a dependency on the standardtest,build,doc, anddev extras.
The full set of dependency requirements is then based on the unconditionaldependencies, along with those of any requested extras.
Dependency examples (showing just therequires subfield):
"requires": ["ComfyChair"] -> requires ``ComfyChair`` only"requires": ["ComfyChair[warmup]"] -> requires ``ComfyChair`` and ``SoftCushions``"requires": ["ComfyChair[all]"] -> requires ``ComfyChair`` and ``SoftCushions``, but will also pick up any new extras defined in later versions
The metadata specification may be updated with clarifications withoutrequiring a new PEP or a change to the metadata version.
Changing the meaning of existing fields or adding new features (other thanthrough the extension mechanism) requires a new metadata version defined ina new PEP.
The reference implementations for converting from legacy metadata tometadata 2.0 are:
bdist_wheel command tosetuptoolsdistutils2 projectNote
These tools have yet to be updated for the switch to standard extensionsfor several fields.
While it is expected that there may be some edge cases where manualintervention is needed for clean conversion, the specification has beendesigned to allow fully automated conversion of almost all projects onPyPI.
Metadata conversion (especially on the part of the index server) is anecessary step to allow installation and analysis tools to startbenefiting from the new metadata format, without having to wait fordevelopers to upgrade to newer build systems.
As an example of mapping this PEP to Linux distro packages, assume anexample project without any extras defined is split into 2 RPMsin a SPEC file:example andexample-devel.
The unconditional dependencies would be mapped to the Requires dependenciesfor the “example” RPM (a mapping from environment markers relevant to Linuxto SPEC file conditions would also allow those to be handled correctly).
Thebuild anddev extra dependencies would be mapped to theBuildRequires dependencies for the “example” RPM. Depending on how the%check section in the RPM was defined, thetest extra may also bemapped to the BuildRequires declaration for the RPM.
All defined dependencies relevant to Linux in thedev,test,build,anddoc extras would become Requires dependencies for the “example-devel”RPM.
A documentation toolchain dependency like Sphinx would either go in thebuild extra (for example, if man pages were included in thebuilt distribution) or in thedoc extra (for example, if thedocumentation is published solely through Read the Docs or theproject website). This would be enough to allow an automated converterto map it to an appropriate dependency in the spec file.
If the project did define any extras, those could be mapped to additionalvirtual RPMs with appropriate BuildRequires and Requires entries based onthe details of the dependency specifications. Alternatively, they couldbe mapped to other system package manager features (such as weak dependencies).
The metadata extension format should also provide a way for distributionspecific hints to be included in the upstream project metadata without needingto manually duplicate any of the upstream metadata in a distribution specificformat.
The rationale for major changes is given in the following sections.
The semantics of major and minor version increments are now specified,and follow the same model as the format version semantics specified forthe wheel format inPEP 427: minor version increments must behavereasonably when processed by a tool that only understand earlier metadataversions with the same major version, while major version incrementsmay include changes that are not compatible with existing tools.
The major version number of the specification has been incrementedaccordingly, as interpretingPEP 426 metadata obviously cannot beinterpreted in accordance with earlier metadata specifications.
Whenever the major version number of the specification is incremented, itis expected that deployment will take some time, as either metadataconsuming tools must be updated before other tools can safely startproducing the new format, or else the sdist and wheel formats, along withthe installation database definition, will need to be updated to supportprovision of multiple versions of the metadata in parallel.
Existing tools won’t abide by this guideline until they’re updated tosupport the new metadata standard, so the new semantics will first takeeffect for a hypothetical 2.x -> 3.0 transition. For the 1.x -> 2.xtransition, we will use the approach where tools continue to produce theexisting supplementary files (such asentry_points.txt) in additionto any equivalents specified using the new features of the standardmetadata format (including the formal extension mechanism).
The old “Key:Value” format was becoming increasingly limiting, with variouscomplexities like parsers needing to know which fields were permitted tooccur more than once, which fields supported the environment markersyntax (with an optional";" to separate the value from the marker) andeventually even the option to embed arbitrary JSON inside particularsubfields.
The old serialisation format also wasn’t amenable to easy conversion tostandard Python data structures for use in any new install hook APIs, orin future extensions to the runtime importer APIs to allow them to provideinformation for inclusion in the installation database.
Accordingly, we’ve taken the step of switching to a JSON-compatible metadataformat. This works better for APIs and is much easier for tools to parse andgenerate correctly. Changing the name of the metadata file also makes iteasy to distribute 1.x and 2.x metadata in parallel, greatly simplifyingseveral aspects of the migration to the new metadata format.
The specific choice ofpydist.json as the preferred file name relatesto the fact that the metadata described in these files applies to thedistribution as a whole, rather than to any particular build. Additionalmetadata formats may be defined in the future to hold information that canonly be determined after building a binary distribution for a particulartarget environment.
SeePEP 440 for a detailed rationale for the various changes made to theversioning scheme.
The new source label support is intended to make it clearer that theconstraints on public version identifiers are there primarily to aid inthe creation of reliable automated dependency analysis tools. Projectsare free to use whatever versioning scheme they like internally, so longas they are able to translate it to something the dependency analysis toolswill understand.
Source labels also make it straightforward to record specific details of aversion, like a hash or tag name that allows the release to be reconstructedfrom the project version control system.
The new extras system allows distributions to declare optionalbehaviour, and to use the dependency fields to indicate whenparticular dependencies are needed only to support that behaviour. It isderived from the equivalent system that is already in widespread use aspart ofsetuptools and allows that aspect of the legacysetuptoolsmetadata to be accurately represented in the new metadata format.
The additions to the extras syntax relative to setuptools are defined tomake it easier to express the various possible combinations of dependencies,in particular those associated with build systems (with optional supportfor running the test suite) and development systems.
The separation of the five different kinds of dependency through the Extrassystem allows a project to optionally indicate whether a dependency is neededspecifically to develop, build, test or use the distribution.
The advantage of having these distinctions supported in the upstream Pythonspecific metadata is that even if a project doesn’t care about thesedistinction themselves, they may be more amenable to patches fromdownstream redistributors that separate the fields appropriately. Over time,this should allow much greater control over where and when particulardependencies end up being installed.
The new extension effectively allows sections of the metadatanamespace to be delegated to other projects, while preserving astandard overall format metadata format for easy of processing bydistribution tools that do not support a particular extension.
It also works well in combination with the newbuild extrato allow a distribution to depend on tools whichdo know how to handlethe chosen extension, and the new extras mechanism in general, allowingsupport for particular extensions to be provided as optional features.
Possible future uses for extensions include declaration of plugins forother projects and hints for automatic conversion to Linux systempackages.
The ability to declare an extension as required is included primarily toallow the definition of the metadata hooks extension to be deferred untilsome time after the initial adoption of the metadata 2.0 specification. Ifa release needs apostinstall hook to run in order to completethe installation successfully, then earlier versions of tools should fallback to installing from source rather than installing from a wheel file andthen failing to run the expected postinstall hook.
Several potentially useful features have been deliberately deferred inorder to better prioritise our efforts in migrating to the new metadatastandard. These all reflect information that may be nice to have in thenew metadata, but which can be readily added through metadata extensions orin metadata 2.1 without breaking any use cases already supported by metadata2.0.
Once thepypi,setuptools,pip,wheel anddistlibprojects support creation and consumption of metadata 2.0, then we mayrevisit the creation of metadata 2.1 with some or all of these additionalfeatures.
Some of the information provided by the legacy metadata system has beenmoved out to standard extensions defined inPEP 459.
This allows publication of the core dependency metadata in a more readilyconsumable format to proceed even before the full details of those extensionshave been resolved.
Earlier drafts of this PEP included newProvides andObsoleted-Byfields for more robust automated notifications and tracking of projectobsolescence, renames and mergers.
This isn’t an essential feature of a dependency management system, and hasbeen deferred indefinitely as a possible future metadata extension.
At some point after acceptance of the PEP, we may submit thefollowing MIME type registration request to IANA:
application/vnd.python.pydist+jsonIt’s even possible we may be able to just register thevnd.pythonnamespace under the banner of the PSF rather than having to registerthe individual subformats.
Supporting at least “.startswith” and “.endswith” string methods inenvironment markers would allow some conditions to be written morenaturally. For example,"sys.platform.startswith('win')" is asomewhat more intuitive way to mark Windows specific dependencies,since"'win'insys.platform" is incorrect thanks tocygwinand the fact that 64-bit Windows still shows up aswin32 is morethan a little strange.
The following features have been explicitly considered and rejected asintroducing too much additional complexity for too small a gain inexpressiveness.
Earlier versions of this PEP used separate lists for conditional andunconditional dependencies. This turned out to be annoying to handle inautomated tools and removing it also made the PEP and metadata schemasubstantially shorter, suggesting it was actually harder to explain as well.
Earlier versions of this PEP used separate fields rather than the extrassystem for test, build, documentation, and development dependencies. Thisturned out to be annoying to handle in automated tools and removing it alsomade the PEP and metadata schema substantially shorter, suggesting it wasactually harder to explain as well.
Earlier versions of this PEP attempted to introduce friction into theinappropriate use of overly strict dependency declarations in publishedreleases. Discussion on distutils-sig came to the conclusion that wasn’ta serious enough problem to tackle directly at the interoperabilityspecification layer, and if it does become a problem in the future,it would be better tackled at the point where projects are uploaded tothe public Python Package Index.
Debian doesn’t actually permit underscores in names, but that seemsunduly restrictive for this spec given the common practice of usingvalid Python identifiers as Python distribution names. A Debian sidepolicy of converting underscores to hyphens seems easy enough toimplement (and the requirement to consider hyphens and underscores asequivalent ensures that doing so won’t introduce any name conflicts).
This PEP deliberately avoids following Python 3 down the path of arbitraryUnicode identifiers, as the security implications of doing so aresubstantially worse in the software distribution use case (it opensup far more interesting attack vectors than mere code obfuscation).
In addition, the existing tools really only work properly if you restrictnames to ASCII and changing that would require alot of work for allthe automated tools in the chain.
It may be reasonable to revisit this question at some point in the (distant)future, but setting up a more reliable software distribution system ischallenging enough without adding more general Unicode identifier supportinto the mix.
There is no mechanism to express a dependency on a source label - theyare included in the metadata for internal project reference only. Instead,dependencies must be expressed in terms of either public versions or elsedirect URL references.
An earlier draft of this PEP considered allowing lists in place of theusual strings in dependency specifications to indicate that there aremultiple ways to satisfy a dependency.
If at least one of the individual dependencies was already available, thenthe entire dependency would be considered satisfied, otherwise the firstentry would be added to the dependency set.
Alternative dependency specification example:
["Pillow","PIL"]["mysql","psycopg2 >= 4","sqlite3"]
However, neither of the given examples is particularly compelling,since Pillow/PIL style forks aren’t common, and the database driver usecase would arguably be better served by an SQL Alchemy defined “supporteddatabase driver” metadata extension where a project depends on SQL Alchemy,and then declares in the extension which database drivers are checked forcompatibility by the upstream project.
PEP 440 defines a rich syntax for version comparisons that couldpotentially be useful withpython_version andpython_full_versionin environment markers. However, allowing the full syntax would meanenvironment markers are no longer a Python subset, while allowingonly some of the comparisons would introduce yet another special caseto handle.
Given that environment markers are only used in cases where a higher level“or” is implied by the metadata structure, it seems easier to require theuse of multiple comparisons against specific Python versions for the rarecases where this would be useful.
Under the revised metadata design, conditional “provides” based on runtimefeatures or the environment would go in a separate “may_provide” field.However, it isn’t clear there’s any use case for doing that, so the ideais rejected unless someone can present a compelling use case (and even thenthe idea won’t be reconsidered until metadata 2.1 at the earliest).
This document specifies version 2.0 of the metadata format.Version 1.0 is specified inPEP 241.Version 1.1 is specified inPEP 314.Version 1.2 is specified inPEP 345.
The initial attempt at a standardised version scheme, along with thejustifications for needing such a standard can be found inPEP 386.
This document has been placed in the public domain.
Source:https://github.com/python/peps/blob/main/peps/pep-0426.rst
Last modified:2024-10-06 16:27:00 GMT