[run],[project.dependency-groups], …?--only-deps not sufficient?Important
This PEP is a historical document. The up-to-date, canonical spec,Dependency Groups, is maintained on thePyPA specs page.
×
See thePyPA specification update process for how to propose changes.
This PEP specifies a mechanism for storing package requirements inpyproject.toml files such that they are not included in any built distribution ofthe project.
This is suitable for creating named groups of dependencies, similar torequirements.txt files, which launchers, IDEs, and other tools can find andidentify by name.
The feature defined here is referred to as “Dependency Groups”.
There are two major use cases for which the Python community has nostandardized answer:
In support of these two needs, there are two common solutions which are similarto this proposal:
requirements.txt filesBothrequirements.txt files andextras have limitations which thisstandard seeks to overcome.
Note that the two use cases above describe two different types of projectswhich this PEP seeks to support:
Several motivating use cases are defined in detail in theUse Cases Appendix.
requirements.txt filesMany projects may define one or morerequirements.txt files,and may arrange them either at the project root (e.g.requirements.txt andtest-requirements.txt) or else in a directory (e.g.requirements/base.txt andrequirements/test.txt). However, there aremajor issues with the use of requirements files in this way:
requirements.txt files arenot standardized, but instead provideoptions topip.As a result, it is difficult to define tool behaviors based onrequirements.txt files. They are not trivial to discover or identify byname, and their contents may contain a mix of package specifiers and additionalpip options.
The lack of a standard forrequirements.txt contents also means they arenot portable to any alternative tools which wish to process them other thanpip.
Additionally,requirements.txt files require a file per dependency list.For some use-cases, this makes the marginal cost of dependency groupings high,relative to their benefit.A terser declaration is beneficial to projects with a number of small groups ofdependencies.
In contrast with this, Dependency Groups are defined at a well known locationinpyproject.toml with fully standardized contents. Not only will they haveimmediate utility, but they will also serve as a starting point for futurestandards.
extrasextras are additional package metadata declared in the[project.optional-dependencies] table. They provide names for lists ofpackage specifiers which are published as part of a package’s metadata, andwhich a user can request under that name, as inpipinstall'foo[bar]' toinstallfoo with thebar extra.
Becauseextras are package metadata, they are not guaranteed to bestatically defined and may require a build system to resolve.Furthermore, definition of a[project.optional-dependencies] indicates tomany tools that a project is a package, and may drive tool behaviors such asvalidation of the[project] table.
For projects which are packages,extras are a common solution for definingdevelopment dependencies, but even under these circumstances they havedownsides:
extra defines optionaladditional dependencies, it is notpossible to install anextra without installing the current package andits dependencies.extras are part of the public interfacefor packages. Becauseextras are published, package developers often areconcerned about ensuring that their development extras are not confused withuser-facing extras.This PEP defines the storage of requirements data in lists within a[dependency-groups] table.This name was chosen to match the canonical name of the feature(“Dependency Groups”).
This format should be as simple and learnable as possible, having a formatvery similar to existingrequirements.txt files for many cases. Each listin[dependency-groups] is defined as a list of package specifiers. Forexample:
[dependency-groups]test=["pytest>7","coverage"]
There are a number of use cases forrequirements.txt files which requiredata which cannot be expressed inPEP 508 dependency specifiers. Suchfields are not valid in Dependency Groups. Including many of the data andfields whichpip supports, such as index servers, hashes, and pathdependencies, requires new standards. This standard leaves room for newstandards and developments, but does not attempt to support all validrequirements.txt contents.
The only exception to this is the-r flag whichrequirements.txt filesuse to include one file in another. Dependency Groups support an “include”mechanism which is similar in meaning, allowing one dependency group to extendanother.
Dependency Groups have two additional features which are similar torequirements.txt files:
The following use cases are considered important targets for this PEP. They aredefined in greater detail in theUse Cases Appendix.
The existing Poetry and PDM tools already offer a feature which each calls“Dependency Groups”. However, absent any standard for specifying collectionsof dependencies, each tool defines these in a tool-specific way, in therelevant sections of the[tool] table.
(PDM also uses extras for some Dependency Groups, and overlaps the notionheavily with extras.)
This PEP does not support all of the features of Poetry and PDM, which, likerequirements.txt files forpip, support several non-standard extensionsto common dependency specifiers.
It should be possible for such tools to use standardized Dependency Groups asextensions of their own Dependency Group mechanisms.However, defining a new data format which replaces the existing Poetry and PDMsolutions is a non-goal. Doing so would require standardizing severaladditional features, such as path dependencies, which are supported by thesetools.
Dependency Groups are very similar to extras which go unpublished.However, there are two major features which distinguish them from extrasfurther:
Dependency Groups are intended to be extensible in future PEPs.However, Dependency Groups should also be usable by multiple tools in asingle Python project.With multiple tools using the same data, it is possible that one implementsa future PEP which extends Dependency Groups, while another does not.
To support users in this case, this PEP defines and recommends validationbehaviors in which tools only examine Dependency Groups which they are using.This allows multiple tools, using different versions of Dependency Groups data,to share a single table inpyproject.toml.
This PEP defines a new section (table) inpyproject.toml files nameddependency-groups. Thedependency-groups table contains an arbitrarynumber of user-defined keys, each of which has, as its value, a list ofrequirements (defined below). These keys must bevalid non-normalized names,and must benormalizedbefore comparisons.
Tools SHOULD prefer to present the original, non-normalized name to users bydefault. If duplicate names, after normalization, are encountered, tools SHOULDemit an error.
Requirement lists underdependency-groups may contain strings, tables(“dicts” in Python), or a mix of strings and tables.
Strings in requirement lists must be validDependency Specifiers,as defined inPEP 508.
Tables in requirement lists must be valid Dependency Object Specifiers.
Dependency Object Specifiers are tables which define zero or more dependencies.
This PEP standardizes only one type of Dependency Object Specifier, a“Dependency Group Include”. Other types may be added in future standards.
A Dependency Group Include includes the dependencies of another DependencyGroup in the current Dependency Group.
An include is defined as a table with exactly one key,"include-group",whose value is a string, the name of another Dependency Group.
For example,{include-group="test"} is an include which expands to thecontents of thetest Dependency Group.
Includes are defined to be exactly equivalent to the contents of the namedDependency Group, inserted into the current group at the location of the include.For example, iffoo=["a","b"] is one group, andbar=["c",{include-group="foo"},"d"] is another, thenbar shouldevaluate to["c","a","b","d"] when Dependency Group Includes are expanded.
Dependency Group Includes may specify the same package multiple times. ToolsSHOULD NOT deduplicate or otherwise alter the list contents produced by theinclude. For example, given the following table:
[dependency-groups]group-a=["foo"]group-b=["foo>1.0"]group-c=["foo<1.0"]all=["foo",{include-group="group-a"},{include-group="group-b"},{include-group="group-c"}]
The resolved value ofall SHOULD be["foo","foo","foo>1.0","foo<1.0"].Tools should handle such a list exactly as they would handle any other case inwhich they are asked to process the same requirement multiple times withdifferent version constraints.
Dependency Group Includes may include lists containing Dependency GroupIncludes, in which case those includes should be expanded as well. DependencyGroup Includes MUST NOT include cycles, and tools SHOULD report an error ifthey detect a cycle.
The following is an example of a partialpyproject.toml which uses this todefine four Dependency Groups:test,docs,typing, andtyping-test:
[dependency-groups]test=["pytest","coverage"]docs=["sphinx","sphinx-rtd-theme"]typing=["mypy","types-requests"]typing-test=[{include-group="typing"},{include-group="test"},"useful-types"]
Note that none of these Dependency Group declarations implicitly install thecurrent package, its dependencies, or any optional dependencies.Use of a Dependency Group liketest to test a package requires that theuser’s configuration or toolchain also installs the current package (.).For example,
$TOOLinstall-dependency-grouptestpipinstall-e.
could be used (supposing$TOOL is a tool which supports installingDependency Groups) to build a testing environment.
This also allows for thedocs dependency group to be used withoutinstalling the project as a package:
$TOOLinstall-dependency-groupdocsBuild backends MUST NOT include Dependency Group data in built distributions aspackage metadata. This means that PKG-INFO in sdists and METADATA in wheelsdo not include any referencable fields containing Dependency Groups.
It is valid to use Dependency Groups in the evaluation of dynamic metadata, andpyproject.toml files included in sdists will naturally still contain the[dependency-groups] table. However, the table contents are not part of apublished package’s interfaces.
Tools which support Dependency Groups are expected to provide new options andinterfaces to allow users to install from Dependency Groups.
No syntax is defined for expressing the Dependency Group of a package, for tworeasons:
For example, a possible pip interface for installing Dependency Groupswould be:
pipinstall--dependency-groups=test,typingNote that this is only an example. This PEP does not declare any requirementsfor how tools support the installation of Dependency Groups.
Tools MAY choose to provide the same interfaces for installing DependencyGroups as they do for installing extras.
Note that this specification does not forbid having an extra whose name matchesa Dependency Group.
Users are advised to avoid creating Dependency Groups whose names match extras.Tools MAY treat such matching as an error.
Tools supporting Dependency Groups may want to validate data before using it.However, tools implementing such validation behavior should be careful to allowfor future expansions to this spec, so that they do not unnecessarily emiterrors or warnings in the presence of new syntax.
Tools SHOULD error when evaluating or processing unrecognized data inDependency Groups.
Tools SHOULD NOT eagerly validate the list contents ofall DependencyGroups.
This means that in the presence of the following data, most tools will allowthefoo group to be used, and will only error when thebar group isused:
[dependency-groups]foo=["pyparsing"]bar=[{set-phasers-to="stun"}]
Eager validation is discouraged for tools which primarily install or resolveDependency Groups.Linters and validation tools may have good cause to ignore this recommendation.
The following Reference Implementation prints the contents of a DependencyGroup to stdout, newline delimited.The output is therefore validrequirements.txt data.
importreimportsysimporttomllibfromcollectionsimportdefaultdictfrompackaging.requirementsimportRequirementdef_normalize_name(name:str)->str:returnre.sub(r"[-_.]+","-",name).lower()def_normalize_group_names(dependency_groups:dict)->dict:original_names=defaultdict(list)normalized_groups={}forgroup_name,valueindependency_groups.items():normed_group_name=_normalize_name(group_name)original_names[normed_group_name].append(group_name)normalized_groups[normed_group_name]=valueerrors=[]fornormed_name,namesinoriginal_names.items():iflen(names)>1:errors.append(f"{normed_name} ({', '.join(names)})")iferrors:raiseValueError(f"Duplicate dependency group names:{', '.join(errors)}")returnnormalized_groupsdef_resolve_dependency_group(dependency_groups:dict,group:str,past_groups:tuple[str,...]=())->list[str]:ifgroupinpast_groups:raiseValueError(f"Cyclic dependency group include:{group} ->{past_groups}")ifgroupnotindependency_groups:raiseLookupError(f"Dependency group '{group}' not found")raw_group=dependency_groups[group]ifnotisinstance(raw_group,list):raiseValueError(f"Dependency group '{group}' is not a list")realized_group=[]foriteminraw_group:ifisinstance(item,str):# packaging.requirements.Requirement parsing ensures that this is a valid# PEP 508 Dependency Specifier# raises InvalidRequirement on failureRequirement(item)realized_group.append(item)elifisinstance(item,dict):iftuple(item.keys())!=("include-group",):raiseValueError(f"Invalid dependency group item:{item}")include_group=_normalize_name(next(iter(item.values())))realized_group.extend(_resolve_dependency_group(dependency_groups,include_group,past_groups+(group,)))else:raiseValueError(f"Invalid dependency group item:{item}")returnrealized_groupdefresolve(dependency_groups:dict,group:str)->list[str]:ifnotisinstance(dependency_groups,dict):raiseTypeError("Dependency Groups table is not a dict")ifnotisinstance(group,str):raiseTypeError("Dependency group name is not a str")return_resolve_dependency_group(dependency_groups,group)if__name__=="__main__":withopen("pyproject.toml","rb")asfp:pyproject=tomllib.load(fp)dependency_groups_raw=pyproject["dependency-groups"]dependency_groups=_normalize_group_names(dependency_groups_raw)print("\n".join(resolve(pyproject["dependency-groups"],sys.argv[1])))
At time of writing, thedependency-groups namespace within apyproject.toml file is unused. Since the top-level namespace isreserved for use only by standards specified at packaging.python.org,there are no direct backwards compatibility concerns.
However, the introduction of the feature has implications for anumber of ecosystem tools, especially those which attempt to supportexamination of data insetup.py andrequirements.txt.
A wide range of tools understand Python dependency data as expressed inrequirements.txt files. (e.g., Dependabot, Tidelift, etc)
Such tools inspect dependency data and, in some cases, offer tool-assisted orfully automated updates.It is our expectation that no such tools would support the new DependencyGroups at first, and broad ecosystem support could take many months or even somenumber of years to arrive.
As a result, users of Dependency Groups would experience a degradation in theirworkflows and tool support at the time that they start using Dependency Groups.This is true of any new standard for where and how dependency data are encoded.
This PEP introduces new syntaxes and data formats for specifying dependencyinformation in projects. However, it does not introduce newly specifiedmechanisms for handling or resolving dependencies.
It therefore does not carry security concerns other than those inherent in anytools which may already be used to install dependencies – i.e. maliciousdependencies may be specified here, just as they may be specified inrequirements.txt files.
This feature should be referred to by its canonical name, “Dependency Groups”.
The basic form of usage should be taught as a variant on typicalrequirements.txt data. Standard dependency specifiers (PEP 508) can beadded to a named list. Rather than asking pip to install from arequirements.txt file, either pip or a relevant workflow tool will installfrom a named Dependency Group.
For new Python users, they may be taught directly to create a section inpyproject.toml containing their Dependency Groups, similarly to how theyare currently taught to userequirements.txt files.This also allows new Python users to learn aboutpyproject.toml fileswithout needing to learn about package building.Apyproject.toml file with only[dependency-groups] and no other tablesis valid.
For both new and experienced users, the Dependency Group Includes will need tobe explained. For users with experience usingrequirements.txt, this can bedescribed as an analogue for-r. For new users, they should be taught thatan include allows one Dependency Group to extend another. Similar configurationinterfaces and the Pythonlist.extend method may be used to explain theidea by analogy.
Python users who have usedsetup.py packaging may be familiar with commonpractices which predatepyproject.toml, in which package metadata isdefined dynamically. Requirements loaded fromrequirements.txt files anddefinitions of static lists prior tosetup() invocation readily analogizewith Dependency Groups.
This specification provides no universal interface for interacting withDependency Groups, other than inclusion in a built package via theprojecttable. This has implications both for tool authors and for users.
Tool authors should determine how or if Dependency Groups are relevant to theiruser stories, and build their own interfaces to fit.For environment managers, resolvers, installers, and related non-build tools,they will be able to document that they support “PEP 735 Dependency Groups”,but they will be responsible for documenting their usage modes.For build backends, supporting Dependency Groups will require support forinclusion from theproject table, but set no other strict requirements.
For users, the primary consequence is that they must consult relevant tooldocumentation whenever they wish to use Dependency Groups outside of packagebuilds.Users should be advised by tools, either through documentation or runtimewarnings or errors, about usages which are disrecommended or not supported.For example, if a tool wishes to require that all Dependency Groups aremutually compatible, containing no contradictory package specifiers, itshould document that restriction and advise users on how to appropriatelyleverage Dependency Groups for its purposes.
If our goal is to allow for future expansion, then defining each DependencyGroup as a subtable, thus enabling us to attach future keys to each group,allows for the greatest future flexibility.
However, it also makes the structure nested more deeply, and therefore harderto teach and learn. One of the goals of this PEP is to be an easy replacementfor manyrequirements.txt use-cases.
Earlier drafts of this specification defined syntactic forms for DependencyGroup Includes and Path Dependencies.
However, there were three major issues with this approach:
Several use cases surfaced during discussion which need more expressivespecifiers than are possible withPEP 508.
“Path Dependencies”, referring to local paths, and references to[project.dependencies] were of particular interest.
However, there are no existing standards for these features (excepting thede-facto standard ofpip’s implementation details).
As a result, attempting to include these features in this PEP results in asignificant growth in scope, to attempt to standardize these various featuresandpip behaviors.
Special attention was devoted to attempting to standardize the expression ofeditable installations, as expressed bypipinstall-e andPEP 660.However, although the creation of editable installs is standardized for buildbackends, the behavior of editables is not standardized for installers.Inclusion of editables in this PEP requires that any supporting tool allows forthe installation of editables.
Therefore, although Poetry and PDM provide syntaxes for some of these features,they are considered insufficiently standardized at present for inclusion inDependency Groups.
[run],[project.dependency-groups], …?There are many possible names for this concept.It will have to live alongside the already existing[project.dependencies]and[project.optional-dependencies] tables, and possibly a new[external] dependency table as well (at time of writing,PEP 725, whichdefines the[external] table, is in progress).
[run] was a leading proposal in earlier discussions, but its proposed usagecentered around a single set of runtime dependencies. This PEP explicitlyoutlines multiple groups of dependencies, which makes[run] a lessappropriate fit – this is not just dependency data for a specific runtimecontext, but for multiple contexts.
[project.dependency-groups] would offer a nice parallel with[project.dependencies] and[project.optional-dependencies], but hasmajor downsides for non-package projects.[project] requires several keys to be defined, such asname andversion. Using this name would either require redefining the[project]table to allow for these keys to be absent, or else would impose a requirementon non-package projects to define and use these keys. By extension, it wouldeffectively require any non-package project allow itself to be treated as apackage.
--only-deps not sufficient?pip currently has a feature on the roadmap to add an–only-deps flag.This flag is intended to allow users to install package dependencies and extraswithout installing the current package.
It does not address the needs of non-package projects, nor does it allow forthe installation of an extra without the package dependencies.
Existing environment managers like tox, Nox, and Hatch already havethe ability to list inlined dependencies as part of their configuration data.This meets many development dependency needs, and clearly associates dependencygroups with relevant tasks which can be run.These mechanisms aregood but they are notsufficient.
First, they do not address the needs of non-package projects.
Second, there is no standard for other tools to use to access these data. Thishas impacts on high-level tools like IDEs and Dependabot, which cannot supportdeep integration with these Dependency Groups. (For example, at time of writingDependabot will not flag dependencies which are pinned intox.ini files.)
[project.dependencies] or[project.optional-dependencies]?Earlier drafts of this specification allowed Dependency Group Includes to beused in the[project] table.However, there were several issues raised during community feedback which ledto its removal.
Only a small number of additional use cases would be addressed by the inclusionof Dependency Groups, and it increased the scope of the specificationsignificantly. In particular, this inclusion would increase the number of partiesimpacted by the addition. Many readers of the[project] table, including buildbackends, SBOM generators, and dependency analyzers are implicated by a change to[project] but may continue to operate as-is in the presence of a new (butunconnected)[dependency-groups] table.
Separately from the above concern, allowing inclusion of dependency groups from the[project] table encourages package maintainers to move dependency metadata outof the current standard location.This complicates staticpyproject.toml metadata and conflicts with the goal ofPEP 621 to store dependency metadata in a single location.
Finally, exclusion of[project] support from this PEP is not final. Theuse of includes from that table, or an inclusion syntax from[dependency-groups] into[project], could be introduced by a futurePEP and considered on its own merits.
[project]Although deferred in this PEP, allowing includes from the[project]table would address several use cases.
In particular, there are cases in which package developers would like toinstall only the dependencies of a package, without the package itself.
For example:
For an example of the last case, consider the following samplepyproject.toml:
[project]dependencies=[{include="runtime"}][optional-dependencies]foo=[{include="foo"}][dependency-groups]runtime=["a","b"]foo=["c","d"]typing=["mypy",{include="runtime"},{include="foo"}]
In this case, atyping group can be defined, with all of the package’sruntime dependencies, but without the package itself. This allows uses of thetyping Dependency Group to skip installation of the package – not only isthis more efficient, but it may reduce the requirements for testing systems.
[build-system.requires]?Given that we will not allow for[project] usage of Dependency Groups,[build-system.requires] can be considered in comparison with[project.dependencies].
There are fewer theoretical usages for build requirements specified in a groupthan package requirements. Additionally, the impact of such a change implicatesPEP 517 frontend, which would need to support Dependency Groups in order toprepare a build environment.
Compared with changes to[project.dependencies] and[project.optional-dependencies], changing the behaviors of[build-system.requires] is higher impact and has fewer potential uses.Therefore, given that this PEP declines to make changes to the[project]table, changing[build-system] is also deferred.
Several usage scenarios for dependency groups revolve around installing adependency group alongside a package defined in the[project] table.For example, testing a package involves installing testing dependencies and thepackage itself. Additionally, the compatibility of a dependency group with themain package is a valuable input to lockfile generators.
In such cases, it is desirable for a Dependency Group to declare that itdepends upon the project itself. Example syntaxes from discussions included{include-project=true} and{include-group=":project:"}.
However, if a specification is established to extendPEP 508 with PathDependencies, this would result in Dependency Groups having two ways ofspecifying the main package. For example, if. becomes formally supported,and{include-project=true} is included in this PEP, then dependencygroups may specify any of the following groups
[dependency-groups]case1=[{include-project=true}]case2=["."]case3=[{include-project=true},"."]case4=[{include-project=false},"."]
In order to avoid a confusing future in which multiple different optionsspecify the package defined inpyproject.toml, any syntax for declaringthis relationship is omitted from this PEP.
This section is primarily informational and serves to document how otherlanguage ecosystems solve similar problems.
package.jsonIn the JavaScript community, packages contain a canonical configuration anddata file, similar in scope topyproject.toml, atpackage.json.
Two keys inpackage.json control dependency data:"dependencies" and"devDependencies". The role of"dependencies" is effectively the sameas that of[project.dependencies] inpyproject.toml, declaring thedirect dependencies of a package.
"dependencies" dataDependency data is declared inpackage.json as a mapping from package namesto version specifiers.
Version specifiers support a small grammar of possible versions, ranges, andother values, similar to Python’sPEP 440 version specifiers.
For example, here is a partialpackage.json file declaring a fewdependencies:
{"dependencies":{"@angular/compiler":"^17.0.2","camelcase":"8.0.0","diff":">=5.1.0 <6.0.0"}}
The use of the@ symbol is ascope which declares the packageowner, for organizationally owned packages."@angular/compiler" therefore declares a package namedcompiler groupedunderangular ownership.
Dependency specifiers support a syntax for URLs and Git repositories, similarto the provisions in Python packaging.
URLs may be used in lieu of version numbers.When used, they implicitly refer to tarballs of package source code.
Git repositories may be similarly used, including support for committishspecifiers.
UnlikePEP 440, NPM allows for the use of local paths to package source codedirectories for dependencies. When these data are added topackage.json viathe standardnpminstall--save command, the path is normalized to arelative path, from the directory containingpackage.json, and prefixedwithfile:. For example, the following partialpackage.json contains areference to a sibling of the current directory:
{"dependencies":{"my-package":"file:../foo"}}
Theofficial NPM documentationstates that local path dependencies “should not” be published to public packagerepositories, but makes no statement about the inherent validity or invalidityof such dependency data in published packages.
"devDependencies" datapackage.json is permitted to contain a second section named"devDependencies", in the same format as"dependencies".The dependencies declared in"devDependencies" are not installed by defaultwhen a package is installed from the package repository (e.g. as part of adependency being resolved) but are installed whennpminstall is run in thesource tree containingpackage.json.
Just as"dependencies" supports URLs and local paths, so does"devDependencies".
"peerDependencies" and"optionalDependencies"There are two additional, related sections inpackage.json which haverelevance.
"peerDependencies" declares a list of dependencies in the same format as"dependencies", but with the meaning that these are a compatibilitydeclaration.For example, the following data declares compatibility with packagefooversion 2:
{"peerDependencies":{"foo":"2.x"}}
"optionalDependencies" declares a list of dependencies which should beinstalled if possible, but which should not be treated as failures if they areunavailable. It also uses the same mapping format as"dependencies".
"peerDependenciesMeta""peerDependenciesMeta" is a section which allows for additional controlover how"peerDependencies" are treated.
Warnings about missing dependencies can be disabled by setting packages tooptional in this section, as in the following sample:
{"peerDependencies":{"foo":"2.x"},"peerDependenciesMeta":{"foo":{"optional":true}}}
--omit and--includeThenpminstall command supports two options,--omit and--include,which can control whether “prod”, “dev”, “optional”, or “peer” dependencies are installed.
The “prod” name refers to dependencies listed under"dependencies".
By default, all four groups are installed whennpminstall is executedagainst a source tree, but these options can be used to control installationbehavior more precisely.Furthermore, these values can be declared in.npmrc files, allowingper-user and per-project configurations to control installation behaviors.
Ruby projects may or may not be intended to produce packages (“gems”) in theRuby ecosystem. In fact, the expectation is that most users of the language donot want to produce gems and have no interest in producing their own packages.Many tutorials do not touch on how to produce packages, and the toolchain neverrequires user code to be packaged for supported use-cases.
Ruby splits requirement specification into two separate files.
Gemfile: a dedicated file which only supports requirement data in the formof dependency groups<package>.gemspec: a dedicated file for declaring package (gem) metadataThebundler tool, providing thebundle command, is the primary interfacefor usingGemfile data.
Thegem tool is responsible for building gems from.gemspec data, via thegembuild command.
AGemfile is a Ruby filecontaininggem directives enclosed in any number ofgroup declarations.gem directives may also be used outside of thegroup declaration, in whichcase they form an implicitly unnamed group of dependencies.
For example, the followingGemfile listsrails as a project dependency.All other dependencies are listed under groups:
source'https://rubygems.org'gem'rails'group:testdogem'rspec'endgroup:lintdogem'rubocop'endgroup:docsdogem'kramdown'gem'nokogiri'end
If a user executesbundleinstall with these data, all groups areinstalled. Users can deselect groups by creating or modifying a bundler configin.bundle/config, either manually or via the CLI. For example,bundleconfigset--localwithout'lint:docs'.
It is not possible, with the above data, to exclude the top-level use of the'rails' gem or to refer to that implicit grouping by name.
Agemspec file is aruby file containing aGem::Specificationinstance declaration.
Only two fields in aGem::Specification pertain to package dependency data.These areadd_development_dependency andadd_runtime_dependency.AGem::Specification object also provides methods for adding dependenciesdynamically, includingadd_dependency (which adds a runtime dependency).
Here is a variant of therails.gemspec file, with many fields removed orshortened to simplify:
version='7.1.2'Gem::Specification.newdo|s|s.platform=Gem::Platform::RUBYs.name="rails"s.version=versions.summary="Full-stack web application framework."s.license="MIT"s.author="David Heinemeier Hansson"s.files=["README.md","MIT-LICENSE"]# shortened from the real 'rails' projects.add_dependency"activesupport",versions.add_dependency"activerecord",versions.add_dependency"actionmailer",versions.add_dependency"activestorage",versions.add_dependency"railties",versionend
Note that there is no use ofadd_development_dependency.Some other mainstream, major packages (e.g.rubocop) do not use developmentdependencies in their gems.
Other projectsdo use this feature. For example,kramdown makes use ofdevelopment dependencies, containing the following specification in itsRakefile:
s.add_dependency"rexml"s.add_development_dependency'minitest','~> 5.0's.add_development_dependency'rouge','~> 3.0','>= 3.26.0's.add_development_dependency'stringex','~> 1.5.1'
The purpose of development dependencies is only to declare an implicit group,as part of the.gemspec, which can then be used bybundler.
For full details, see thegemspec directive inbundler'sdocumentation on Gemfiles.However, the integration between.gemspec development dependencies andGemfile/bundle usage is best understood via an example.
Consider the following simple project in the form of aGemfile and.gemspec.Thecool-gem.gemspec file:
Gem::Specification.newdo|s|s.author='Stephen Rosen's.name='cool-gem's.version='0.0.1's.summary='A very cool gem that does cool stuff's.license='MIT's.files=[]s.add_dependency'rails's.add_development_dependency'kramdown'end
and theGemfile:
source'https://rubygems.org'gemspec
Thegemspec directive inGemfile declares a dependency on the localpackage,cool-gem, defined in the locally availablecool-gem.gemspecfile. Italso implicitly adds all development dependencies to a dependencygroup nameddevelopment.
Therefore, in this case, thegemspec directive is equivalent to thefollowingGemfile content:
gem'cool-gem',:path=>'.'group:developmentdogem'kramdown'end
In the absence of any prior standard for Dependency Groups, two known workflowtools, PDM and Poetry, have defined their own solutions.
This section will primarily focus on these two tools as cases of prior artregarding the definition and use of Dependency Groups in Python.
Both PDM and Poetry treat the projects they support as packages.This allows them to use and interact with standardpyproject.toml metadatafor some of their needs, and allows them to support installation of the“current project” by doing a build and install using their build backends.
Effectively, this means that neither Poetry nor PDM supports non-package projects.
PDM and Poetry extendPEP 508 dependency specifiers with additional featureswhich are not part of any shared standard.The two tools use slightly different approaches to these problems, however.
PDM supports specifying local paths, and editable installs, via a syntax whichlooks like a set of arguments topipinstall. For example, the followingdependency group includes a local package in editable mode:
[tool.pdm.dev-dependencies]mygroup=["-e file:///${PROJECT_ROOT}/foo"]
This declares a dependency groupmygroup which includes a local editableinstall from thefoo directory.
Poetry describes dependency groups as tables, mapping package names tospecifiers. For example, the same configuration as the abovemygroupexample might appear as follows under Poetry:
[tool.poetry.group.mygroup]foo={path="foo",editable=true}
PDM restricts itself to a string syntax, and Poetry introduces tables whichdescribe dependencies.
Both PDM and Poetry have tool-specific support for installing dependencygroups. Because both projects support their own lockfile formats, they alsoboth have the capability to transparently use a dependency group name to referto thelocked dependency data for that group.
However, neither tool’s dependency groups can be referenced natively from othertools liketox,nox, orpip.Attempting to install a dependency group undertox, for example, requiresan explicit call to PDM or Poetry to parse their dependency data and do therelevant installation step.
A web application (e.g. a Django or Flask app) often does not need to build adistribution, but bundles and ships its source to a deployment toolchain.
For example, a source code repository may define Python packaging metadata aswell as containerization or other build pipeline metadata (Dockerfile,etc).The Python application is built by copying the entire repository into abuild context, installing dependencies, and bundling the result as a machineimage or container.
Such applications have dependency groups for the build, but also for linting,testing, etc. In practice, today, these applications often define themselves aspackages to be able to use packaging tools and mechanisms likeextras tomanage their dependency groups. However, they are not conceptually packages,meant for distribution in sdist or wheel format.
Dependency Groups allow these applications to define their various dependencieswithout relying on packaging metadata, and without trying to express theirneeds in packaging terms.
Libraries are Python packages which build distributions (sdist and wheel) andpublish them to PyPI.
For libraries, Dependency Groups represent an alternative toextras fordefining groups of development dependencies, with the important advantagesnoted above.
A library may define groups fortest andtyping which allow testing andtype-checking, and therefore rely on the library’s own dependencies (asspecified in[project.dependencies]).
Other development needs may not require installation of the package at all. Forexample, alint Dependency Group may be valid and faster to install withoutthe library, as it only installs tools likeblack,ruff, orflake8.
lint andtest environments may also be valuable locations to hook inIDE or editor support. See the case below for a fuller description of suchusage.
Here’s an example Dependency Groups table which might be suitable for alibrary:
[dependency-groups]test=["pytest<8","coverage"]typing=["mypy==1.7.1","types-requests"]lint=["black","flake8"]typing-test=[{include-group="typing"},"pytest<8"]
Note that none of these implicitly install the library itself.It is therefore the responsibility of any environment management toolchain toinstall the appropriate Dependency Groups along with the library when needed,as in the case oftest.
Data Science Projects typically take the form of a logical collection ofscripts and utilities for processing and analyzing data, using a commontoolchain. Components may be defined in the Jupyter Notebook format (ipynb),but rely on the same common core set of utilities.
In such a project, there is no package to build or install. Therefore,pyproject.toml currently does not offer any solution for dependencymanagement or declaration.
It is valuable for such a project to be able to define at least one majorgrouping of dependencies. For example:
[dependency-groups]main=["numpy","pandas","matplotlib"]
However, it may also be necessary for various scripts to have additionalsupporting tools. Projects may even have conflicting or incompatible tools ortool versions for different components, as they evolve over time.
Consider the following more elaborate configuration:
[dependency-groups]main=["numpy","pandas","matplotlib"]scikit=[{include-group="main"},"scikit-learn==1.3.2"]scikit-old=[{include-group="main"},"scikit-learn==0.24.2"]
This definesscikit andscikit-old as two similar variants of thecommon suite of dependencies, pulling in different versions ofscikit-learnto suit different scripts.
This PEP only defines these data. It does not formalize any mechanism for aData Science Project (or any other type of project) to install the dependenciesinto known environments or associate those environments with the variousscripts. Such combinations of data are left as a problem for tool authors tosolve, and perhaps eventually standardize.
There are a number of tools which generate lockfiles in the Python ecosystemtoday. PDM and Poetry each use their own lockfile formats, and pip-toolsgeneratesrequirements.txt files with version pins and hashes.
Dependency Groups are not an appropriate place to store lockfiles, as they lackmany of the necessary features. Most notably, they cannot store hashes, whichmost lockfile users consider essential.
However, Dependency Groups are a valid input to tools which generate lockfiles.Furthermore, PDM and Poetry both allow a Dependency Group name (under theirnotions of Dependency Groups) to be used to refer to its locked variant.
Therefore, consider a tool which produces lockfiles, here called$TOOL.It might be used as follows:
$TOOLlock--dependency-group=test$TOOLinstall--dependency-group=test--use-locked
All that such a tool needs to do is to ensure that its lockfile data recordsthe nametest in order to support such usage.
The mutual compatibility of Dependency Groups is not guaranteed. For example,the Data Science example above shows conflicting versions ofscikit-learn.Therefore, installing multiple locked dependency groups in tandem may requirethat tools apply additional constraints or generate additional lockfile data.These problems are considered out of scope for this PEP.
As two examples of how combinations might be locked:
A common usage in tox, Nox, and Hatch is to install a set of dependencies intoa testing environment.
For example, undertox.ini, type checking dependencies may be definedinline:
[testenv:typing]deps=pyrightuseful-typescommands=pyright src/
This combination provides a desirable developer experience within a limitedcontext. Under the relevant environment manager, the dependencies which areneeded for the test environment are declared alongside the commands which needthose dependencies. They are not published in package metadata, asextraswould be, and they are discoverable for the tool which needs them to build therelevant environment.
Dependency Groups apply to such usages by effectively “lifting” theserequirements data from a tool-specific location into a more broadly availableone. In the example above, onlytox has access to the declared list ofdependencies. Under an implementation supporting dependency groups, the samedata might be available in a Dependency Group:
[dependency-groups]typing=["pyright","useful-types"]
The data can then be used under multiple tools. For example,tox mightimplement support asdependency_groups=typing, replacing thedepsusage above.
In order for Dependency Groups to be a viable alternative for users ofenvironment managers, the environment managers will need to support processingDependency Groups similarly to how they support inline dependency declaration.
IDE and editor integrations may benefit from conventional or configurable namedefinitions for Dependency Groups which are used for integrations.
There are at least two known scenarios in which it is valuable for an editor orIDE to be capable of discovering the non-published dependencies of a project:
These cases could be handled by defining conventional group names liketest,lint, andfix, or by defining configuration mechanisms whichallow the selection of Dependency Groups.
For example, the followingpyproject.toml declares the three aforementionedgroups:
[dependency-groups]test=["pytest","pytest-timeout"]lint=["flake8","mypy"]fix=["black","isort","pyupgrade"]
This PEP makes no attempt to standardize such names or reserve them for suchuses. IDEs may standardize or may allow users to configure the group names usedfor various purposes.
This declaration allows the project author’s knowledge of the appropriate toolsfor the project to be shared with all editors of that project.
This document is placed in the public domain or under theCC0-1.0-Universal license, whichever is more permissive.
Source:https://github.com/python/peps/blob/main/peps/pep-0735.rst
Last modified:2025-01-18 20:45:37 GMT