Prior art

The R language has aSystem Requirements for R packages with a centralregistry that knows how to translate external dependency metadata to installcommands for package managers likeapt-get. This registry centralises themappings for a series of Linux distributions, and also Windows. macOS is notpresent. The“Rule Coverage” of its READMEused to show that this system improves the chance of success of building packagesfrom CRAN from source. Across all CRAN packages,Ubuntu 18 improved from 78.1% to 95.8%, CentOS 7 from 77.8% to 93.7% and openSUSE15.0 from 78.2% to 89.7%. The chance of success depends on how well the registryis maintained, but the gain is significant: ~4x fewer packages fail to build onUbuntu and CentOS in a Docker container.

RPM-based distributions, like Fedora, can use arule-based implementation(NameConvertor) inpyp2rpm. The main rule is that the RPM name for a PyPI package isf"python-{pypi_package_name}". This seems to work quite well; there are afew variants like Python version specific names, where the prefix contains thePython major and minor version numbers (e.g.python311- instead ofpython-).

Gentoo follows a similar approach to naming Python packages, using thedev-python/category and somewell-specified rules.

Conda-forge has a more explicit name mapping, because the base names are thesame in conda-forge as on PyPI (e.g.,numpy maps tonumpy), but thereare many exceptions because of both name collisions and renames (e.g., the PyPIname for PyTorch istorch while in conda-forge it’spytorch). There areseveral name mappings efforts maintained by different teams. Conda-forge’s infrastructuregenerates one inregro/cf-graph-countyfair.Grayskull maintainsits own curated mapping.Prefix.dev created theparselmouth mappingsto support conda and PyPI integrations in their tooling. A more complete overview oftheir approaches, strengths and weaknesses can be found inconda/grayskull#564.

TheOpenStack ecosystem also needs to deal withsome mapping efforts. All of them focus on Linux distributions, exclusively.pkg-mapaccompaniesdiskimage-builder and provides a file format where the user definesarbitrary variable names and their corresponding names in the target distro(Red Hat, Debian, OpenSUSE, etc). Seeexample for PyYAML.bindep defines a filebindep.txt(seeexample)where users can write down dependencies that are not installable from PyPI. The format isline-based, with each line containing a dependency as found in the Debian ecosystem.For other distributions, it offers a “filters” syntax between square brackets where userscan indicate other target platforms, optional dependencies and extras.

The need for mappings is also found in other ecosystems likeSageMath,but also by end-users themselves who want to install PyPI packages with their systempackage manager of choice (example StackOverflow question).

Governance and maintenance costs of name mappings

The maintenance cost of external dependency mappings to a large number of packagingecosystems is potentially high. We choose to define the registry in sucha way that:

• A central authority maintains the list of recognized DepURLs and theknown ecosystem mappings.
• The mappings themselves are maintained by the target packaging ecosystems.

Hence this system is opt-in for a given ecosystem, and the associatedmaintenance costs are distributed.

Generating package manager-specific install commands

Python package authors with external dependencies usually have installationinstructions for those external dependencies in their documentation. Theseinstructions are difficult to write and keep up-to-date, and are usually onlycovering one or at most a handful of platforms. As an example, here are SciPy’sinstructions for its external build dependencies (C/C++/Fortran compilers,OpenBLAS, pkg-config):

• Debian/Ubuntu:sudoaptinstall-ygccg++gfortranlibopenblas-devliblapack-devpkg-configpython3-pippython3-dev
• Fedora:sudodnfinstallgcc-gfortranpython3-developenblas-devellapack-develpkgconfig
• CentOS/RHEL:sudoyuminstallgcc-gfortranpython3-developenblas-devellapack-develpkgconfig
• Arch Linux:sudopacman-Sgcc-fortranopenblaspkgconf
• Homebrew on macOS:brewinstallgfortranopenblaspkg-config

The package names vary a lot, and there are differences like some distrossplitting off headers and other build-time dependencies in a separate-dev/-devel package while others do not. With the registry in this PEP,this could be made both more comprehensive and easier to maintain through a toolcommand with semantics of“show this ecosystem’s preferred package managerinstall command for all external dependencies”. This may be done as astandalone tool, or as a new subcommand in any Python development workflow tool(e.g., Pip, Poetry, Hatch, PDM, uv).

To this end, each ecosystem mapping can provide a list of package managersknown to be compatible, with templated instructions on how to install and querypackages. The provided install command templates are paired with query command templatesso those tools can check whether the needed packages are already present withouthaving to attempt an install operation (which might be expensive and have unintendedside effects like version upgrades).

Registry design

The mapping infrastructure has been designed to present the following components and properties:

• A central registry of PEP 725 identifiers (DepURLs), including at least thewell-known generic and virtual identifiers considered canonical.
• A list of known ecosystems, where ecosystem maintainers can register their name mapping(s).
• A standardized schema that defines how mappings should be structured. Each mapping canalso provide programmatic details about how their supported package manager(s) work.

The above documents are provided as JSON files validated by accompanying JSON schemas.A Python library and CLI is provided to query and utilize these resources. The user canconfigure which system package manager they prefer to use for the default package mappingsand command generation (e.g. a user on Ubuntu may preferconda,brew orspackinstead ofapt as their package manager of choice to provide external dependencies).

Central registry

The central registry defines which identifiers are recognized as canonical,plus known aliases. Each entry MUST provide a valid DepURL in thefieldid, with an optional free formdescription text. Additionallyan entry MAY refer to another entry via itsprovides field, which takesa string or a list of strings already defined asid in the registry. This is usefulfor both aliases (e.g.dep:generic/arrow anddep:github/apache/arrow) andconcrete implementations of adep:virtual/ entry (e.g.dep:generic/gccwould providedep:virtual/compiler/c). Entries withoutprovides contentor, if populated, only withdep:virtual/ identifiers, are consideredcanonical. Theprovides field MUST NOT be present indep:virtual/ definitions.

Having a central registry enables the validation of the[external] table.All involved tools MUST check that the provided identifiers are well formed.Additionally, some tools MAY check whether the identifiers in use are recognized ascanonical. More specifically:

• Build backends, build frontends, and installers SHOULD NOT do any validationof identifiers being canonical by default.
• Uploaders liketwine SHOULD validate if the identifiers are canonicaland warn or report an error to the user, with opt-out mechanisms. TheySHOULD suggest a canonical replacement, if available.
• Index servers like PyPI MAY perform the same validation as the uploaders andreject the artifact if necessary.

This registry SHOULD also centralize authoritative decisions about itscontents, such as which entry of a collection of aliases is preferred ascanonical, or which versioning scheme applies to virtual DepURLs (see AppendixB). The corresponding answers are not given in this PEP; instead we delegatethat responsibility to the central registry maintainers.

Mappings

The mappings specify which ecosystem-specific identifiers provide the canonicalentries available in the central registry. A mapping mainly consists of a listof dictionaries, in which each entry consists of:

• anid field with the canonical DepURL.
• an optional free formdescription text.
• aspecs field whose value MUST be one of:
  • a dictionary with three keys (build,host,run). The valuesMUST be a string or list of strings representing the ecosystem-specific packageidentifiers as needed as build-, host- and runtime dependencies (see PEP 725 fordetails on these definitions).
  • for convenience, a string or a list of strings are also accepted as ashorthand form. In this case, the identifier(s) will be used to populatethe three categories mentioned in the item above.
  • an empty list, which is understood as the ecosystem not having packages toprovide such dependency.
• aspecs_from field whose value is a DepURL from which thespecsfield will be imported. Eitherspecs orspecs_from MUST be present.
• an optionalurls field whose value MUST be a URL, a list of URLs, or adictionary that maps a string to a URL. This is useful to link to externalresources that provide more information about the mapped packages.

The mappings SHOULD also specify another sectionpackage_managers, reportingwhich package managers are available in the ecosystem and how to use them. This field MUSTtake a list of dictionaries, with each of them reporting the following fields:

• name (string), unique identifier for this package manager. Usually, the executable name.
• commands (list of dictionaries), the commands to run to install the mapped package(s) andcheck whether they are already installed.
• specifier_syntax: instructions on how to map a subset of PEP 440 specifiers tothe target package manager. Three levels of support are offered: name-only, exact-version-only,and version-range compatibility (with per-operator translations).

Each mapping MUST have a canonical URL for online retrieval. These mappingsMAY also be packaged for offline distribution in each platform. The authorsrecommend placing in the standard location for data artifacts in each operatingsystem; e.g.$XDG_DATA_DIRS on Linux and others,~/Library/ApplicationSupport onmacOS, and%LOCALAPPDATA% for Windows. The subdirectory identifier MUSTbeexternal-packaging-metadata-mappings. This data directory SHOULD onlycontain mapping documents named{ecosystem-identifier}.mapping.json. The centralregistry and known ecosystem documents MAY also be distributed in this directory,asregistry.json andknown-ecosystems.json, respectively.

Known ecosystems

The list of known ecosystems has two roles:

1. Reporting the canonical URL for its mapping.
2. Assigning a short identifier to each ecosystem. This is the identifierthat MUST be used in the mapping filenames mentioned above so they can befound in the local filesystem.

For ecosystems corresponding to Linux distributions, the identifier MUST be theone reported by theiros-releaseID parameter. For other ecosystems, it MUST be decided during the submission tothe list of known ecosystems document. It MUST only use the characters allowed inos-release’sID field, as per this regex[a-z0-9\-_.]+.

Schema details

Three JSON Schema documents are provided to fully standardize the registries and mappings.

Examples

{"$schema":"https://raw.githubusercontent.com/jaimergp/external-metadata-mappings/main/schemas/central-registry.schema.json","schema_version":1,"definitions":[{"id":"dep:generic/zlib","description":"A Massively Spiffy Yet Delicately Unobtrusive Compression Library"},{"id":"dep:generic/libwebp","description":"WebP codec is a library to encode and decode images in WebP format. This package contains the library that can be used in other programs to add WebP support"},{"id":"dep:generic/clang","description":"Language front-end and tooling infrastructure for languages in the C language family for the LLVM project."}]}

{"$schema":"https://raw.githubusercontent.com/jaimergp/external-metadata-mappings/main/schemas/known-ecosystems.schema.json","schema_version":1,"ecosystems":{"conda-forge":{"mapping":"https://raw.githubusercontent.com/jaimergp/external-metadata-mappings/refs/heads/main/data/conda-forge.mapping.json"}}

{"schema_version":1,"name":"conda-forge","description":"Mapping for the conda-forge ecosystem","mappings":[{"id":"dep:generic/zlib","description":"zlib data compression library for the next generation systems. From zlib-ng/zlib-ng.","specs":"zlib-ng",// Simplest form"urls":{"feedstock":"https://github.com/conda-forge/zlib-ng-feedstock"}},{"id":"dep:generic/libwebp","description":"WebP image library. libwebp-base ships libraries; libwebp ships the binaries.","specs":{// expanded form with single spec per category"build":"libwebp","host":"libwebp-base","run":"libwebp"},"urls":{"feedstock":"https://github.com/conda-forge/libwebp-feedstock"}},{"id":"dep:generic/clang","description":"Development headers and libraries for Clang","specs":{// expanded form with specs list"build":["clang","clangxx"],"host":["clangdev"],"run":["clang","clangxx","clang-format","clang-tools"]},"urls":{"feedstock":"https://github.com/conda-forge/clangdev-feedstock"}},],"package_managers":[{"name":"conda","commands":{"install":{"command":["conda","install","{}"],"multiple_specifiers":"always","requires_elevation":false,},"query":{"command":["conda","list","-f","{}"],"multiple_specifiers":"never","requires_elevation":false,}},"specifier_syntax":{"exact_version":["{name}=={version}"],"name_only":["{name}"],"version_ranges":{"and":",","equal":"={version}","greater_than":">{version}","greater_than_equal":">={version}","less_than":"<{version}","less_than_equal":"<={version}","not_equal":"!={version}","syntax":["{name}{ranges}"]}}}]}

[build-system]build-backend='mesonpy'requires=["meson-python>=0.13.1","pybind11>=2.10.4",][external]build-requires=["dep:virtual/compiler/cxx",]host-requires=["dep:generic/zlib",]

# show all external dependencies as DepURLs$python-mpyproject_externalshow.[external]build-requires=["dep:virtual/compiler/cxx",]host-requires=["dep:generic/zlib",]# show all external dependencies, but mapped to the autodetected ecosystem$python-mpyproject_externalshow--output=mapped.[external]build_requires=["g++","python3",]host_requires=["zlib1g","zlib1g-dev",]# show how to install external dependencies$python-mpyproject_externalshow--output=command.sudoaptinstall--yesg++zlib1gzlib1g-devpython3

$ pip install ....× Encountered error while generating package metadata.╰─> See above for output.note: This is an issue with the package mentioned above, not pip.This package has the following external dependencies, if those are missingon your system they are likely to be the cause of this build failure:  dep:virtual/compiler/cxx  dep:generic/zlib

Thefollowingexternaldependenciesareneededtoinstallthepackagementionedabove.Youmayneedtoinstallthemwith`apt`:g++zlib1gzlib1g-dev

preferred_package_manager="mamba"

$python-mpyproject_externalshow--outputcommand.mambainstall--yes--channel=conda-forge--channel-priority=strictcxx-compilerzlibpython

$python-mpyproject_externalshow--validategrpcio-1.71.0.tar.gzWARNINGDepURL'dep:virtual/compiler/cpp'isnotrecognizedinthecentralregistry.Didyoumeananyof['dep:virtual/compiler/c','dep:virtual/compiler/cxx','dep:virtual/compiler/cuda','dep:virtual/compiler/go','dep:virtual/compiler/c-sharp']?[external]build-requires=["dep:virtual/compiler/c","dep:virtual/compiler/cpp",]

>>>frompyproject_externalimportExternal>>>external=External.from_pyproject_data(      {        "external": {          "build-requires": [            "dep:virtual/compiler/c",            "dep:virtual/compiler/cpp",          ]        }      }    )>>>external.validate()Dep URL 'dep:virtual/compiler/cpp' is not recognized in the central registry. Did youmean any of ['dep:virtual/compiler/c', 'dep:virtual/compiler/cxx','dep:virtual/compiler/cuda', 'dep:virtual/compiler/go', 'dep:virtual/compiler/c-sharp']?>>>external=External.from_pyproject_data(      {        "external": {          "build-requires": [            "dep:virtual/compiler/c",            "dep:virtual/compiler/cxx",  # fixed          ]        }      }    )>>>external.validate()>>>external.to_dict(){'external': {'build_requires': ['dep:virtual/compiler/c', 'dep:virtual/compiler/cxx']}}>>>frompyproject_externalimportdetect_ecosystem_and_package_manager>>>ecosystem,package_manager=detect_ecosystem_and_package_manager()>>>ecosystem'conda-forge'>>>package_manager'pixi'>>>external.to_dict(mapped_for=ecosystem,package_manager=package_manager){'external': {'build_requires': ['c-compiler', 'cxx-compiler', 'python']}}>>>external.install_command(ecosystem,package_manager=package_manager)# {"command": ["pixi", "add", "{}"]}['pixi', 'add', 'c-compiler', 'cxx-compiler', 'python']>>>external.query_commands(ecosystem,package_manager=package_manager)# {"command": ["pixi", "list", "{}"]}[  ['pixi', 'list', 'c-compiler'],  ['pixi', 'list', 'cxx-compiler'],  ['pixi', 'list', 'python'],]

$ grayskull pypi my-cxx-pkg#### Initializing recipe for my-cxx-pkg (pypi) ####Recovering metadata from pypi...Starting the download of the sdist package my-cxx-pkgmy-cxx-pkg 100% Time:  0:00:10   5.3 MiB/s|###########|Checking for pyproject.toml...Build requirements:  - python                                 # [build_platform != target_platform]  - cross-python_{{ target_platform }}     # [build_platform != target_platform]  - meson-python >= 0.13.1                 # [build_platform != target_platform]  - pybind11 >= 2.10.4                     # [build_platform != target_platform]  - ninja                                  # [build_platform != target_platform]  - libboost-devel                         # [build_platform != target_platform]  - {{ compiler('cxx') }}Host requirements:  - python  - meson-python >=0.13.1  - pybind11 >=2.10.4  - ninja  - libboost-develRun requirements:  - python#### Recipe generated on /path/to/recipe/dir for my-cxx-pkg ####

Central DepURL registry maintainers

Central DepURL registry maintainers curate the collection of DepURLs and theknown ecosystems. These contributors need to be able to refer to clearlydefined rules for when a new DepURL can be defined. It is undesirable to beloose with canonical DepURL definitions, because each definition added increasesmaintenance effort in the mappings in the target ecosystems.

The central registry maintainers should agree on the ground rules and write themdown as part of the repository documentation, perhaps supported by additionalaffordances like issue and pull request templates, or linting tools.

Package ecosystem maintainers usage

Missing mapping entries will result in the absence tailored error messages andother UX affordances for end users of the impacted ecosystems. It is thusrecommended that each package ecosystem keeps their mappings up-to-date withthe central registry. The key to this will be automation, like linting scripts(see example atexternal-metadata-mappings),or periodic notifications via issues or draft submissions.

Establishing the initial mapping is likely to involve a lot of work, but ideally the maintenance on an ongoing basis effort should require smaller effort.

As best practices are discovered and agreed on, they should get documentedin the central registry repository as learning materials for the mappingmaintainers.

Maintainers of Python projects

A package maintainer’s responsibility is to decide the DepURL that bestrepresents the external dependency that their package needs. This is coveredinPEP 725; the interactive mappings browser demo located atexternal-metadata-mappings.streamlit.appmay come handy. The central registry documentation may include examples andfrequently asked questions to guide newcomers with their decisions.

If no suitable DepURL is available for a given dependency, maintainers mayconsider submitting a request in the central registry. Instructions on how to dothis should be provided as part of the central registry documentation.

End user package consumers

There will be no change in the user experience by default. This is particularlytrue if the user only relies on wheels, since the only impact will be driven byexternal runtime dependencies (expected to be rare), and even in those casesthey need to opt-in by installing a compatible tool.

Users that do opt-in may find missing entries in for their target ecosystems, forwhich they should obtain informative error messages that point to the relevantdocumentation sections. This will allow them to get acquainted with the natureof the issue and its potential solutions.

We hope that this results in a subset of them reporting the missing entries,submitting a fix to the affected mapping or, if totally absent, even decidingto maintain a new one on their own. To that end, they should get familiar withthe responsibilties of mapping maintainers (discussed above).

Centralized mappings governed by the same body

While a central authority for the registry is useful, the maintenance burdenof handling the mappings for multiple ecosystems is unfeasible at the scale of PyPI.Hence, we propose that the central authority only governs the central registry andthe list of known ecosystems, while the maintenance of the mappings themselves is handledby the target ecosystems.

Allowing ecosystem-specific variants of packages

Some ecosystems have their own variants of known packages; e.g. Debian’slibsymspg2-dev. While an identifier such asdep:debian/libsymspg2-devis syntactically valid, the central registry should not recognize it as awell-known identifier, preferring itsgeneric counterpart instead. Usersmay still choose to use it, but tools may warn about it and suggest using thegeneric one. This is meant to encourage ecosystem-agnostic metadata wheneverpossible to facilitate adoption across platforms and operating systems.

Adding more package metadata to the central registry

A central registry should only contain a list of DepURLs and aminimal set of metadata fields to facilitate its identification (a free-formtext description, and one or more URLs to relevant locations).

We have chosen to leave additional details out of the central registry, and insteadsuggest external contributors to maintain their own mappings where they canannotate the identifiers with extra metadata via the free-formextra_metadata field.

The reasons include:

• The existing fields should be sufficient to identify the project home,where that extra metadata can be obtained (e.g. the repository at the URL will likelyinclude details about authorship and licensing).
• These details can also be obtained from the actual target ecosystems. In somecases this might even be preferable; e.g., for licenses, where downstream packagingcan actually affect it by unvendoring dependencies or adjusting optional bits.
• Those details may change over the lifetime of the project, and keeping themup-to-date would increase the maintenance burden on the governance body.
• Centralizing additional metadata would hence introduce ambiguities anddiscrepancies across target ecosystems, where different versions may beavailable or required.

Mapping PyPI projects to repackaged counterparts in target ecosystems

It is common that other ecosystems redistribute Python projects with their ownpackaging system. While this is required for packages with compiled extensions, itis theoretically unnecessary for pure Python wheels; the only need for this seems tobe metadata translation. SeeWanting a singular packaging tool/vision #68,Wanting a singular packaging tool/vision #103,andspack/spack#28282for examples of discussions in this direction.

The proposals in this PEP do not consider PyPI ->ecosystem mappings, butthe same schemas can be repurposed to that end. After all, it is trivial to build a PURL orDepURL from a PyPI name (e.g.numpy becomespkg:pypi/numpy). A hypotheticalmapping maintainer could annotate their repackaging efforts with the source PURL identifier,and then use that metadata to generate compatible mappings, such as:

{"$schema":"https://raw.githubusercontent.com/jaimergp/external-metadata-mappings/main/schemas/external-mapping.schema.json","schema_version":1,"name":"PyPI packages in Ubuntu 24.04","description":"PyPI mapping for the Ubuntu 24.04 LTS (Noble) distro","mappings":[{"id":"dep:pypi/numpy","description":"The fundamental package for scientific computing with Python","specs":["python3-numpy"],"urls":{"home":"https://numpy.org/"}}]}

Such a mapping would allow downstream redistribution efforts to focus on thecompiled packages and instead delegate pure wheels to Python packagingsolutions directly.

Strict validation of identifiers

The central registry provides a list of canonical identifiers, which may temptimplementors into ensuring that all supplied identifiers are indeed canonical. Wehave decided to onlyrecommend this practice for some tool categories, but in nocaserequire such checks.

It is expected that as the[external] metadata tables are adopted by thepackaging community, thecanonical identifier list grows to accommodate therequirements found in different projects. For example, a new C++ library or anew language compiler are introduced.

If validation is made too strict and rejects unknown identifiers, this wouldintroduce unnecessary friction in the external metadata adoption, and requirehuman interaction to review and accept the newly requested identifiers ina time-critical manner, potentially blocking publication of the packagethat needs a new identifier added to the central registry.

We suggest simply checking that the provided identifiers are well-formed. Futurework may choose to also enforce that the identifiers are recognized as canonical,once the central registry has matured with significant adoption.