Abstract

Whiledistutils /setuptools have taken us a long way, theysuffer from three serious problems: (a) they’re missing importantfeatures like usable build-time dependency declaration,autoconfiguration, and even basic ergonomic niceties likeDRY-compliantversion number management, and (b) extending them is difficult, sowhile there do exist various solutions to the above problems, they’reoften quirky, fragile, and expensive to maintain, and yet (c) it’svery difficult to use anything else, because distutils/setuptoolsprovide the standard interface for installing packages expected byboth users and installation tools likepip.

Previous efforts (e.g. distutils2 or setuptools itself) have attemptedto solve problems (a) and/or (b). This proposal aims to solve (c).

The goal of this PEP is get distutils-sig out of the business of beinga gatekeeper for Python build systems. If you want to use distutils,great; if you want to use something else, then that should be easy todo using standardized methods. The difficulty of interfacing withdistutils means that there aren’t many such systems right now, but togive a sense of what we’re thinking about seeflit orbento. Fortunately, wheels have nowsolved many of the hard problems here – e.g. it’s no longer necessarythat a build system also know about every possible installationconfiguration – so pretty much all we really need from a build systemis that it have some way to spit out standard-compliant wheels andsdists.

We therefore propose a new, relatively minimal interface forinstallation tools likepip to interact with package source treesand source distributions.

Terminology and goals

Asource tree is something like a VCS checkout. We need a standardinterface for installing from this format, to support usages likepipinstallsome-directory/.

Asource distribution is a static snapshot representing a particularrelease of some source code, likelxml-3.4.4.tar.gz. Sourcedistributions serve many purposes: they form an archival record ofreleases, they provide a stupid-simple de facto standard for toolsthat want to ingest and process large corpora of code, possiblywritten in many languages (e.g. code search), they act as the input todownstream packaging systems like Debian/Fedora/Conda/…, and soforth. In the Python ecosystem they additionally have a particularlyimportant role to play, because packaging tools likepip are ableto use source distributions to fulfill binary dependencies, e.g. ifthere is a distributionfoo.whl which declares a dependency onbar, then we need to support the case wherepipinstallbar orpipinstallfoo automatically locates the sdist forbar,downloads it, builds it, and installs the resulting package.

Source distributions are also known assdists for short.

Abuild frontend is a tool that users might run that takes arbitrarysource trees or source distributions and builds wheels from them. Theactual building is done by each source tree’sbuild backend. In acommand likepipwheelsome-directory/, pip is acting as a buildfrontend.

Anintegration frontend is a tool that users might run that takes aset of package requirements (e.g. a requirements.txt file) andattempts to update a working environment to satisfy thoserequirements. This may require locating, building, and installing acombination of wheels and sdists. In a command likepipinstalllxml==2.4.0, pip is acting as an integration frontend.

Source trees

There is an existing, legacy source tree format involvingsetup.py. We don’t try to specify it further; its de factospecification is encoded in the source code and documentation ofdistutils,setuptools,pip, and other tools. We’ll referto it as thesetup.py-style.

Here we define a new style of source tree based around thepyproject.toml file defined inPEP 518, extending the[build-system] table in that file with one additional key,build-backend. Here’s an example of how it would look:

[build-system]# Defined by PEP 518:requires=["flit"]# Defined by this PEP:build-backend="flit.api:main"

build-backend is a string naming a Python object that will beused to perform the build (see below for details). This is formattedfollowing the samemodule:object syntax as asetuptools entrypoint. For instance, if the string is"flit.api:main" as in theexample above, this object would be looked up by executing theequivalent of:

importflit.apibackend=flit.api.main

It’s also legal to leave out the:object part, e.g.

build-backend="flit.api"

which acts like:

importflit.apibackend=flit.api

Formally, the string should satisfy this grammar:

identifier = (letter | '_') (letter | '_' | digit)*module_path = identifier ('.' identifier)*object_path = identifier ('.' identifier)*entry_point = module_path (':' object_path)?

And we importmodule_path and then lookupmodule_path.object_path (or justmodule_path ifobject_path is missing).

When importing the module path, we donot look in the directory containing thesource tree, unless that would be onsys.path anyway (e.g. because it isspecified in PYTHONPATH). Although Python automatically adds the workingdirectory tosys.path in some situations, code to resolve the backend shouldnot be affected by this.

If thepyproject.toml file is absent, or thebuild-backendkey is missing, the source tree is not using this specification, andtools should revert to the legacy behaviour of runningsetup.py (eitherdirectly, or by implicitly invoking thesetuptools.build_meta:__legacy__backend).

Where thebuild-backend key exists, this takes precedence and the source tree follows the format andconventions of the specified backend (as such nosetup.py is needed unless the backend requires it).Projects may still wish to include asetup.py for compatibility with tools that do not use this spec.

This PEP also defines abackend-path key for use inpyproject.toml, seethe “In-Tree Build Backends” section below. This key would be used as follows:

[build-system]# Defined by PEP 518:requires=["flit"]# Defined by this PEP:build-backend="local_backend"backend-path=["backend"]

Build backend interface

The build backend object is expected to have attributes which providesome or all of the following hooks. The commonconfig_settingsargument is described after the individual hooks.

defbuild_wheel(wheel_directory,config_settings=None,metadata_directory=None):...

defbuild_sdist(sdist_directory,config_settings=None):...

defget_requires_for_build_wheel(config_settings=None):...

defget_requires_for_build_wheel(config_settings):return["wheel >= 0.25","setuptools"]

defprepare_metadata_for_build_wheel(metadata_directory,config_settings=None):...

defget_requires_for_build_sdist(config_settings=None):...

config_settings

pipinstall                                           \--package-configCC=gcc                             \--global-option="--some-global-option"              \--build-option="--build-option1"                    \--build-option="--build-option2"

{"CC":"gcc","--global-option":["--some-global-option"],"--build-option":["--build-option1","--build-option2"],}

Source distributions

We continue with the legacy sdist format, adding some new restrictions.This format is mostlyundefined, but basically comes down to: a file named{NAME}-{VERSION}.{EXT}, which unpacks into a buildable source treecalled{NAME}-{VERSION}/. Traditionally these have alwayscontainedsetup.py-style source trees; we now allow them to alsocontainpyproject.toml-style source trees.

Integration frontends require that an sdist named{NAME}-{VERSION}.{EXT} will generate a wheel named{NAME}-{VERSION}-{COMPAT-INFO}.whl.

The new restrictions for sdists built byPEP 517 backends are:

• They will be gzipped tar archives, with the.tar.gz extension. Ziparchives, or other compression formats for tarballs, are not allowed atpresent.
• Tar archives must be created in the modern POSIX.1-2001 pax tar format, whichuses UTF-8 for file names.
• The source tree contained in an sdist is expected to include thepyproject.toml file.

Evolutionary notes

A goal here is to make it as simple as possible to convert old-stylesdists to new-style sdists. (E.g., this is one motivation forsupporting dynamic build requirements.) The ideal would be that therewould be a single staticpyproject.toml that could be dropped into any“version 0” VCS checkout to convert it to the new shiny. This isprobably not 100% possible, but we can get close, and it’s importantto keep track of how close we are… hence this section.

A rough plan would be: Create a build system package(setuptools_pypackage or whatever) that knows how to speakwhatever hook language we come up with, and convert them into calls tosetup.py. This will probably require some sort of hooking ormonkeypatching to setuptools to provide a way to extract thesetup_requires= argument when needed, and to provide a new versionof the sdist command that generates the new-style format. This allseems doable and sufficient for a large proportion of packages (thoughobviously we’ll want to prototype such a system before we finalizeanything here). (Alternatively, these changes could be made tosetuptools itself rather than going into a separate package.)

But there remain two obstacles that mean we probably won’t be able toautomatically upgrade packages to the new format:

1. There currently exist packages which insist on particular packagesbeing available in their environment before setup.py isexecuted. This means that if we decide to execute build scripts inan isolated virtualenv-like environment, then projects will need tocheck whether they do this, and if so then when upgrading to thenew system they will have to start explicitly declaring thesedependencies (either viasetup_requires= or via staticdeclaration inpyproject.toml).
2. There currently exist packages which do not declare consistentmetadata (e.g.egg_info andbdist_wheel might get differentinstall_requires=). When upgrading to the new system, projectswill have to evaluate whether this applies to them, and if so theywill need to stop doing that.

Rejected options

• We discussed making the wheel and sdist hooks build unpacked directoriescontaining the same contents as their respective archives. In some cases thiscould avoid the need to pack and unpack an archive, but this seems likepremature optimisation. It’s advantageous for tools to work with archivesas the canonical interchange formats (especially for wheels, where the archiveformat is already standardised). Close control of archive creation isimportant for reproducible builds. And it’s not clear that tasks requiring anunpacked distribution will be more common than those requiring an archive.
• We considered an extra hook to copy files to a build directory before invokingbuild_wheel. Looking at existing build systems, we found that passinga build directory intobuild_wheel made more sense for many tools thanpre-emptively copying files into a build directory.
• The idea of passingbuild_wheel a build directory was then also deemed anunnecessary complication. Build tools can use a temporary directory or a cachedirectory to store intermediate files while building. If there is a need, afrontend-controlled cache directory could be added in the future.
• Forbuild_sdist to signal a failure for an expected reason, variousoptions were debated at great length, including raisingNotImplementedError and returning eitherNotImplemented orNone.Please do not attempt to reopen this discussion without anextremely goodreason, because we are quite tired of it.
• Allowing the backend to be imported from files in the source tree would bemore consistent with the way Python imports often work. However, not allowingthis prevents confusing errors from clashing module names. The initialversion of this PEP did not provide a means to allow backends to beimported from files within the source tree, but thebackend-path keywas added in the next revision to allow projects to opt into this behaviourif needed.

Summary of changes to PEP 517

The following changes were made to this PEP after the initial referenceimplementation was released in pip 19.0.

• Cycles in build requirements were explicitly prohibited.
• Support for in-tree backends and self-hosting of backends was added bythe introduction of thebackend-path key in the[build-system]table.
• Clarified that thesetuptools.build_meta:__legacy__PEP 517 backend isan acceptable alternative to directly invokingsetup.py for source treesthat don’t specifybuild-backend explicitly.

Appendix A: Comparison to PEP 516

PEP 516 is a competing proposal to specify a build system interface, whichhas now been rejected in favour of this PEP. The primary difference isthat our build backend is defined via a Python hook-based interfacerather than a command-line based interface.

This appendix documents the arguments advanced for this PEP overPEP 516.

We donot expect that specifying Python hooks rather than command lineinterfaces will, by itself, reduce thecomplexity of calling into the backend, because build frontends willin any case want to run hooks inside a child – this is important toisolate the build frontend itself from the backend code and to bettercontrol the build backends execution environment. So under bothproposals, there will need to be some code inpip to spawn asubprocess and talk to some kind of command-line/IPC interface, andthere will need to be some code in the subprocess that knows how toparse these command line arguments and call the actual build backendimplementation. So this diagram applies to all proposals equally:

+-----------++---------------++----------------+|frontend|-spawn->|childcmdline|-Python->|backend||(pip)||interface||implementation|+-----------++---------------++----------------+

The key difference between the two approaches is how these interfaceboundaries map onto project structure:

.-=ThisPEP=-.+-----------++---------------+|+----------------+|frontend|-spawn->|childcmdline|-Python->|backend||(pip)||interface|||implementation|+-----------++---------------+|+----------------+||______________________________________||Ownedbypip,updatedinlockstep|||PEP-definedinterfaceboundaryChangeshererequiredistutils-sig.-=Alternative=-.+-----------+|+---------------++----------------+|frontend|-spawn->|childcmdline|-Python->|backend||(pip)|||interface||implementation|+-----------+|+---------------++----------------+|||____________________________________________||Ownedbybuildbackend,updatedinlockstep|PEP-definedinterfaceboundaryChangeshererequiredistutils-sig

By moving the PEP-defined interface boundary into Python code, we gainthree key advantages.

First, because there will likely be only a small number of buildfrontends (pip, and… maybe a few others?), while there willlikely be a long tail of custom build backends (since these are chosenseparately by each package to match their particular buildrequirements), the actual diagrams probably look more like:

.-=ThisPEP=-.+-----------++---------------++----------------+|frontend|-spawn->|childcmdline|-Python+>|backend||(pip)||interface|||implementation|+-----------++---------------+|+----------------+||+----------------++>|backend|||implementation||+----------------+::.-=Alternative=-.+-----------++---------------++----------------+|frontend|-spawn+>|childcmdline|-Python->|backend||(pip)|||interface||implementation|+-----------+|+---------------++----------------+||+---------------++----------------++>|childcmdline|-Python->|backend|||interface||implementation||+---------------++----------------+::

That is, this PEP leads to less total code in the overallecosystem. And in particular, it reduces the barrier to entry ofmaking a new build system. For example, this is a complete, workingbuild backend:

# mypackage_custom_build_backend.pyimportos.pathimportpathlibimportshutilimporttarfileSDIST_NAME="mypackage-0.1"SDIST_FILENAME=SDIST_NAME+".tar.gz"WHEEL_FILENAME="mypackage-0.1-py2.py3-none-any.whl"################## sdist creation#################def_exclude_hidden_and_special_files(archive_entry):"""Tarfile filter to exclude hidden and special files from the archive"""ifarchive_entry.isfile()orarchive_entry.isdir():ifnotos.path.basename(archive_entry.name).startswith("."):returnarchive_entrydef_make_sdist(sdist_dir):"""Make an sdist and return both the Python object and its filename"""sdist_path=pathlib.Path(sdist_dir)/SDIST_FILENAMEsdist=tarfile.open(sdist_path,"w:gz",format=tarfile.PAX_FORMAT)# Tar up the whole directory, minus hidden and special filessdist.add(os.getcwd(),arcname=SDIST_NAME,filter=_exclude_hidden_and_special_files)returnsdist,SDIST_FILENAMEdefbuild_sdist(sdist_dir,config_settings):"""PEP 517 sdist creation hook"""sdist,sdist_filename=_make_sdist(sdist_dir)returnsdist_filename################## wheel creation#################defget_requires_for_build_wheel(config_settings):"""PEP 517 wheel building dependency definition hook"""# As a simple static requirement, this could also just be# listed in the project's build system dependencies insteadreturn["wheel"]defbuild_wheel(wheel_directory,metadata_directory=None,config_settings=None):"""PEP 517 wheel creation hook"""fromwheel.archiveimportarchive_wheelfilepath=os.path.join(wheel_directory,WHEEL_FILENAME)archive_wheelfile(path,"src/")returnWHEEL_FILENAME

Of course, this is aterrible build backend: it requires the user tohave manually set up the wheel metadata insrc/mypackage-0.1.dist-info/; when the version number changes itmust be manually updated in multiple places… but it works, and more featurescould be added incrementally. Much experience suggests that large successfulprojects often originate as quick hacks (e.g., Linux – “just a hobby,won’t be big and professional”;IPython/Jupyter –a gradstudent’s $PYTHONSTARTUP file),so if our goal is to encourage the growth of a vibrant ecosystem ofgood build tools, it’s important to minimize the barrier to entry.

Second, because Python provides a simpler yet richer structure fordescribing interfaces, we remove unnecessary complexity from thespecification – and specifications are the worst place forcomplexity, because changing specifications requires painfulconsensus-building across many stakeholders. In the command-lineinterface approach, we have to come up with ad hoc ways to mapmultiple different kinds of inputs into a single linear command line(e.g. how do we avoid collisions between user-specified configurationarguments and PEP-defined arguments? how do we specify optionalarguments? when working with a Python interface these questions havesimple, obvious answers). When spawning and managing subprocesses,there are many fiddly details that must be gotten right, subtlecross-platform differences, and some of the most obvious approaches –e.g., using stdout to return data for thebuild_requires operation– can create unexpected pitfalls (e.g., what happens when computingthe build requirements requires spawning some child processes, andthese children occasionally print an error message to stdout?obviously a careful build backend author can avoid this problem, butthe most obvious way of defining a Python interface removes thispossibility entirely, because the hook return value is clearlydemarcated).

In general, the need to isolate build backends into their own processmeans that we can’t remove IPC complexity entirely – but by placingboth sides of the IPC channel under the control of a single project,we make it much cheaper to fix bugs in the IPC interface than iffixing bugs requires coordinated agreement and coordinated changesacross the ecosystem.

Third, and most crucially, the Python hook approach gives us muchmore powerful options for evolving this specification in the future.

For concreteness, imagine that next year we add a newbuild_sdist_from_vcs hook, which provides an alternative to the currentbuild_sdist hook where the frontend is responsible for passingversion control tracking metadata to backends (including indicating when allon disk files are tracked), rather than individual backends having to query thatinformation themselves. In order to manage the transition, we’d want it to bepossible for build frontends to transparently usebuild_sdist_from_vcs whenavailable and fall back ontobuild_sdist otherwise; and we’d want it to bepossible for build backends to define both methods, for compatibilitywith both old and new build frontends.

Furthermore, our mechanism should also fulfill two more goals: (a) Ifnew versions of e.g.pip andflit are both updated to supportthe new interface, then this should be sufficient for it to be used;in particular, it shouldnot be necessary for every project thatusesflit to update its individualpyproject.toml file. (b)We do not want to have to spawn extra processes just to perform thisnegotiation, because process spawns can easily become a bottleneck whendeploying large multi-package stacks on some platforms (Windows).

In the interface described here, all of these goals are easy toachieve. Becausepip controls the code that runs inside the childprocess, it can easily write it to do something like:

command,backend,args=parse_command_line_args(...)ifcommand=="build_sdist":ifhasattr(backend,"build_sdist_from_vcs"):backend.build_sdist_from_vcs(...)elifhasattr(backend,"build_sdist"):backend.build_sdist(...)else:# error handling

In the alternative where the public interface boundary is placed atthe subprocess call, this is not possible – either we need to spawnan extra process just to query what interfaces are supported (as wasincluded in an earlier draft ofPEP 516, an alternative to this), orelse we give up on autonegotiation entirely (as in the current versionof that PEP), meaning that any changes in the interface will requireN individual packages to update theirpyproject.toml files beforeany change can go live, and that any changes will necessarily berestricted to new releases.

One specific consequence of this is that in this PEP, we’re able tomake theprepare_metadata_for_build_wheel command optional. In our design,this can be readily handled by build frontends, which can put code intheir subprocess runner like:

defdump_wheel_metadata(backend,working_dir):"""Dumps wheel metadata to working directory.       Returns absolute path to resulting metadata directory    """ifhasattr(backend,"prepare_metadata_for_build_wheel"):subdir=backend.prepare_metadata_for_build_wheel(working_dir)else:wheel_fname=backend.build_wheel(working_dir)already_built=os.path.join(working_dir,"ALREADY_BUILT_WHEEL")withopen(already_built,"w")asf:f.write(wheel_fname)subdir=unzip_metadata(os.path.join(working_dir,wheel_fname))returnos.path.join(working_dir,subdir)defensure_wheel_is_built(backend,output_dir,working_dir,metadata_dir):"""Ensures built wheel is available in output directory       Returns absolute path to resulting wheel file    """already_built=os.path.join(working_dir,"ALREADY_BUILT_WHEEL")ifos.path.exists(already_built):withopen(already_built,"r")asf:wheel_fname=f.read().strip()working_path=os.path.join(working_dir,wheel_fname)final_path=os.path.join(output_dir,wheel_fname)os.rename(working_path,final_path)os.remove(already_built)else:wheel_fname=backend.build_wheel(output_dir,metadata_dir=metadata_dir)returnos.path.join(output_dir,wheel_fname)

and thus expose a totally uniform interface to the rest of the frontend,with no extra subprocess calls, no duplicated builds, etc. Butobviously this is the kind of code that you only want to write as partof a private, within-project interface (e.g. the given example requires thatthe working directory be shared between the two calls, but not with anyother wheel builds, and that the return value from the metadata helper functionwill be passed back in to the wheel building one).

(And, of course, making themetadata command optional is one pieceof lowering the barrier to entry for developing new backends, as discussedabove.)

Copyright

This document has been placed in the public domain.