Important
This PEP is a historical document. The up-to-date, canonical spec,Externally Managed Environments, is maintained on thePyPA specs page.
×
See thePyPA specification update process for how to propose changes.
A long-standing practical problem for Python users has been conflictsbetween OS package managers and Python-specific package managementtools like pip. These conflicts include both Python-level APIincompatibilities and conflicts over file ownership.
Historically, Python-specific package management tools have defaultedto installing packages into an implicit global context. With thestandardization and popularity of virtual environments, a bettersolution for most (but not all) use cases is to use Python-specificpackage management tools only within a virtual environment.
This PEP proposes a mechanism for a Python installation to communicateto tools like pip that its global package installation context ismanaged by some means external to Python, such as an OS packagemanager. It specifies that Python-specific package management toolsshould neither install nor remove packages into the interpreter’sglobal context, by default, and should instead guide the end usertowards using a virtual environment.
It also standardizes an interpretation of thesysconfig schemes sothat, if a Python-specific package manager is about to install apackage in an interpreter-wide context, it can do so in a manner thatwill avoid conflicting with the external package manager and reducesthe risk of breaking software shipped by the external package manager.
A few terms used in this PEP have multiple meanings in the contextsthat it spans. For clarity, this PEP uses the following terms inspecific ways:
A distro can be an operating system (OS) of its own, such asDebian, Fedora, or FreeBSD. It can also be an overlay distributionthat installs on top of an existing OS, such as Homebrew orMacPorts.
This document uses the short term “distro,” because the term“distribution” has another meaning in Python packaging contexts: asource or binary distribution package of a single piece of Pythonlanguage software, that is, in the sense ofsetuptools.dist.Distribution or “sdist”. To avoid confusion,this document does not use the plain term “distribution” at all.In the Python packaging sense, it uses the full phrase“distribution package” or just “package” (see below).
The provider of a distro - the team or company that collects andpublishes the software and makes any needed modifications - is itsdistributor.
This document does not use “package” in the sense of an importablename that contains Python modules, though in many cases, adistribution package consists of a single importable package ofthe same name.
This document generally does not use the term “package” to referto units of installation by a distro’s package manager (such as.deb or.rpm files). When needed, it uses phrasing such as“a distro’s package.” (Again, in many cases, a Python package isshipped inside a distro’s package named something likepython-plus the Python package name.)
setup.py command.(Conda[3] is a bit of a special case, as thecondacommand can install much more than just Python packages, making itmore like a distro package manager in some senses. Since theconda command generally only operates on Conda-createdenvironments, most of the concerns in this document do not applytoconda when acting as a Python-specific package manager.)
apt,dpkg,dnf,rpm,pacman, andbrew. The salient feature is that ifa package was installed by a distro package manager, removing orupgrading it in a way that would satisfy a Python-specific packagemanager will generally leave a distro package manager in aninconsistent state.This document also uses phrases like “external package manager” or“system’s package manager” to refer to a distro package manager incertain contexts.
sys.path: if package A 2.0 installs modulea.py in onesys.path entry, and package A 1.0 installs modulea.py ina latersys.path entry, thenimporta returns the modulefrom the former, and we say that A 2.0 shadows A 1.0.Thanks to Python’s immense popularity, software distros (by which wemean Linux and other OS distros as well as overlay distros likeHomebrew and MacPorts) generally ship Python for two purposes: as asoftware package to be used in its own right by end users, and as alanguage dependency for other software in the distro.
For example, Fedora and Debian (and their downstream distros, as wellas many others) ship a/usr/bin/python3 binary which provides thepython3 command available to end users as well as the#!/usr/bin/python3 shebang for Python-language software includedin the distro. Because there are no official binary releases of Pythonfor Linux/UNIX, almost all Python end users on these OSes use thePython interpreter built and shipped with their distro.
Thepython3 executable available to the users of the distro andthepython3 executable available as a dependency for othersoftware in the distro are typically the same binary. This means thatif an end user installs a Python package using a tool likepipoutside the context of a virtual environment, that package is visibleto Python-language software shipped by the distro. If thenewly-installed package (or one of its dependencies) is a newer,backwards-incompatible version of a package that was installed throughthe distro, it may break software shipped by the distro.
This may pose a critical problem for the integrity of distros, whichoften have package-management tools that are themselves written inPython. For example, it’s possible to unintentionally break Fedora’sdnf command with apipinstall command, making it hard torecover.
This applies both to system-wide installs (sudopipinstall) aswell as user home directory installs (pipinstall--user), sincepackages in either location show up on thesys.path of/usr/bin/python3.
There is a worse problem with system-wide installs: if you attempt torecover from this situation withsudopipuninstall, you may endup removing packages that are shipped by the system’s package manager.In fact, this can even happen if you simply upgrade a package - pipwill try to remove the old version of the package, as shipped by theOS. At this point it may not be possible to recover the system to aconsistent state using just the software remaining on the system.
Over the past many years, a consensus has emerged that the best way toinstall Python libraries or applications (when not using a distro’spackage) is to use a virtual environment. This approach waspopularized by the PyPAvirtualenv project, and a simple version ofthat approach is now available in the Python standard library asvenv. Installing a Python package into a virtualenv prevents itfrom being visible to the unqualified/usr/bin/python3 interpreterand prevents breaking system software.
In some cases, however, it’s useful and intentional to install aPython package from outside of the distro that influences the behaviorof distro-shipped commands. This is common in the case of softwarelike Sphinx or Ansible which have a mechanism for writingPython-language extensions. A user may want to use their distro’sversion of the base software (for reasons of paid support or securityupdates) but install a small extension from PyPI, and they’d want thatextension to be importable by the software in their base system.
While this continues to carry the risk of installing a newer versionof a dependency than the operating system expects or otherwisenegatively affecting the behavior of an application, it does not needto carry the risk of removing files from the operating system. A toollike pip should be able to install packages in some directory on thedefaultsys.path, if specifically requested, without deletingfiles owned by the system’s package manager.
Therefore, this PEP proposes two things.
First, it proposesa way for distributors of a Python interpreter tomark that interpreter as having its packages managed by means externalto Python, such that Python-specific tools like pip should notchange the installed packages in the interpreter’s globalsys.pathin any way (add, upgrade/downgrade, or remove) unless specificallyoverridden. It also provides a means for the distributor to indicatehow to use a virtual environment as an alternative.
This is an opt-in mechanism: by default, the Python interpretercompiled from upstream sources will not be so marked, and so runningpipinstall with a self-compiled interpreter, or with a distrothat has not explicitly marked its interpreter, will work as it alwayshas worked.
Second, it sets the rule that when installing packages to aninterpreter’s global context (either to an unmarked interpreter, or ifoverriding the marking),Python-specific package managers shouldmodify or delete files only within the directories of the sysconfigscheme in which they would create files. This permits a distributorof a Python interpreter to set up two directories, one for its ownmanaged packages, and one for unmanaged packages installed by the enduser, and ensure that installing unmanaged packages will not delete(or overwrite) files owned by the external package manager.
As described in detail in the next section, the first behavior changeinvolves creating a marker file namedEXTERNALLY-MANAGED, whosepresence indicates that non-virtual-environment package installationsare managed by some means external to Python, such as a distro’spackage manager. This file is specified to live in thestdlibdirectory in the defaultsysconfig scheme, which marks theinterpreter / installation as a whole, not a particular location onsys.path. The reason for this is that, as identified above, thereare two related problems that risk breaking an externally-managedPython: you can install an incompatible new version of a packagesystem-wide (e.g., withsudopipinstall), and you can install onein your user account alone, but in a location that is on the standardPython command’ssys.path (e.g., withpipinstall--user). Ifthe marker file were in the system-widesite-packages directory,it would not clearly apply to the second case. TheAlternativessection has further discussion of possible locations.
The second behavior change takes advantage of the existingsysconfig setup in distros that have already encountered thisclass of problem, and specifically addresses the problem of aPython-specific package manager deleting or overwriting files that areowned by an external package manager.
The changed behavior in this PEP is intended to “do the right thing”for as many use cases as possible. In this section, we consider thechanges specified by this PEP for several representative use cases /contexts. Specifically, we ask about the two behaviors that could bechanged by this PEP:
pipinstall permitinstallations by default, after implementation of this PEP?(For simplicity, this section discusses pip as the Python-specificinstaller tool, though the analysis should apply equally to any otherPython-specific package management tool.)
This table summarizes the use cases discussed in detail below:
| Case | Description | pipinstall permitted | Deleting externally-installed packages permitted |
|---|---|---|---|
| 1 | Unpatched CPython | Currently yes; stays yes | Currently yes; stays yes |
| 2 | Distro/usr/bin/python3 | Currently yes; becomes no(assuming the distroadds a marker file) | Currently yes (except on Debian); becomes no |
| 3 | Distro Python in venv | Currently yes; stays yes | There are no externally-installed packages |
| 4 | Distro Python in venvwith--system-site-packages | Currently yes; stays yes | Currently no; stays no |
| 5 | Distro Python in Docker | Currently yes; becomes no(assuming the distroadds a marker file) | Currently yes; becomes no |
| 6 | Conda environment | Currently yes; stays yes | Currently yes; stays yes |
| 7 | Dev-facing distro | Currently yes; becomes no(assuming they add amarker file) | Currently often yes; becomes no(assuming they configuresysconfig as needed) |
| 8 | Distro building packages | Currently yes; can stay yes | Currently yes; becomes no |
| 9 | PYTHONHOME copied froma distro Python stdlib | Currently yes; becomes no | Currently yes; becomes no |
| 10 | PYTHONHOME copied fromupstream Python stdlib | Currently yes; stays yes | Currently yes; stays yes |
In more detail, the use cases above are:
sysconfig and without a marker file. This PEPdoes not change its behavior.Such a CPython should (regardless of this PEP) not be installed ina way that overlaps any distro-installed Python on the same system.For instance, on an OS that ships Python in/usr/bin, youshould not install a custom CPython built with./configure--prefix=/usr, or it will overwrite some files from the distroand the distro will eventually overwrite some files from yourinstallation. Instead, your installation should be in a separatedirectory (perhaps/usr/local,/opt, or your homedirectory).
Therefore, we can assume that such a CPython has its ownstdlibdirectory and its ownsysconfig schemes that do not overlap anydistro-installed Python. So any OS-installed packages are notvisible or relevant here.
If there is a concept of “externally-installed” packages in thiscase, it’s something outside the OS and generally managed bywhoever built and installed this CPython. Because the installerchose not to add a marker file or modifysysconfig schemes,they’re choosing the current behavior, andpipinstall canremove any packages available in this CPython.
/usr/bin/python3, either when runningpipinstall as root orpipinstall--user, following ourRecommendations for distros.These recommendations include shipping a marker file in thestdlib directory, to preventpipinstall by default, andplacing distro-shipped packages in a location other than thedefaultsysconfig scheme, so thatpip as root does notwrite to that location.
Many distros (including Debian, Fedora, and their derivatives) arealready doing the latter.
On Debian and derivatives,pipinstall does not currentlydelete distro-installed packages, because Debian carries apatchto pip to prevent this. So, for those distros, this PEP is not abehavior change; it simply standardizes that behavior in a way thatis no longer Debian-specific and can be included into upstream pip.
(We have seen user reports of externally-installed packages beingdeleted on Debian or a derivative. We suspect this is because theuser has previously runsudopipinstall--upgradepip andtherefore now has a version of/usr/bin/pip without the Debianpatch; standardizing this behavior in upstream package installerswould address this problem.)
venv orvirtualenv).Inside a virtual environment, all packages are owned by thatenvironment. Even whenpip,setuptools, etc. are installedinto the environment, they are and should be managed by toolsspecific to that environment; they are not system-managed.
--system-site-packages. This is like the previous case, butworth calling out explicitly, because anything on the globalsys.path is visible.Currently, the answer to “Willpip delete externally-installedpackages” is no, because pip has a special case for running in avirtual environment and attempting to delete packages outside it.After this PEP, the answer remains no, but the reasoning becomesmore general: system site packages will be outside any of thesysconfig schemes used for package management in theenvironment.
RUNpipinstall... statement,etc., and it would be good not to break those. So, builders of basecontainer images may want to ensure that the marker file is notpresent, even if the underlying OS ships one by default.There is a small behavior change: currently,pip run as rootwill delete externally-installed packages, but after this PEP itwill not. We don’t propose a way to override this. However, sincethe base image is generally minimal, there shouldn’t be much of ause case for simply uninstalling packages (especially without usingthe distro’s own tools). The common case is when pip wants toupgrade a package, which previously would have deleted the oldversion (except on Debian). After this change, the old version willstill be on disk, but pip will stillshadow externally-installedpackages, and we believe this to be sufficient for this not to be abreaking change in practice - a Pythonimport statement willstill get you the newly-installed package.
If it becomes necessary to have a way to do this, we suggest thatthe distro should document a way for the installer tool to accessthesysconfig scheme used by the distro itself. See theRecommendations for distros section for more discussion.
It is the view of the authors of this PEP that it’s still a goodidea to use virtual environments with distro-installed Pythoninterpreters, even in single-application container images. Eventhough they run a singleapplication, that application may runcommands from the OS that are implemented in Python, and if you’veinstalled or upgraded the distro-shipped Python packages usingPython-specific tools, those commands may break.
conda tools like pipto install software not available in the Conda repositories. Inthis context, Conda acts as the external package manager / distroand pip as the Python-specific one.In some sense, this is similar to the first case, since Condaprovides its own installation of the Python interpreter.
We don’t believe this PEP requires any changes to Conda, andversions of pip that have implemented the changes in this PEP willcontinue to behave as they currently do inside Conda environments.(That said, it may be worth considering whether to use separatesysconfig schemes for pip-installed and Conda-installedsoftware, for the same reasons it’s a good idea for other distros.)
In these cases, the distro may want to respond to an attemptedpipinstall with guidance encouraging use of the distro’s ownfacilities for adding new packages, along with a link todocumentation.
If the distro supports/encourages creating a virtual environmentfrom the distro’s Python interpreter, there may also be custominstructions for how to properly set up a virtual environment (asfor example Nixpkgs does).
pipinstall be usable as part of thedistro’s package build process. (Consider, for instance, building apython-xyz RPM by usingpipinstall. inside an sdist /source tarball forxyz.) The distro may also want to use a moretargeted but still Python-specific installation tool such asinstaller.For this case, the build process will need to find some way tosuppress the marker file to allowpipinstall to work, and willprobably need to point the Python-specific tool at the distro’ssysconfig scheme instead of the shipped default. See theRecommendations for distros section for more discussion on howto implement this.
As a result of this PEP, pip will no longer be able to removepackages already on the system. However, this behavior change isfine because a package build process should not (and generallycannot) include instructions to delete some other files on thesystem; it can only package up its own files.
PYTHONHOME to set up an alternativePython environment (as opposed to a virtual environment), wherePYTHONHOME is set to some directory copied directly from thedistro Python (e.g.,cp-a/usr/lib/python3.xpyhome/lib).Assuming there are no modifications, then the behavior is just likethe underlying distro Python (case 2). So there are behaviorchanges - you can no longerpipinstall by default, and if youoverride it, it will no longer delete externally-installed packages(i.e., Python packages that were copied from the OS and live in theOS-managedsys.path entry).
This behavior change seems to be defensible, in that if yourPYTHONHOME is a straight copy of the distro’s Python, it shouldbehave like the distro’s Python.
PYTHONHOME taken from a compatible unmodified upstream Python.Because the behavior changes in this PEP are keyed off of files inthe standard library (the marker file instdlib and thebehavior of thesysconfig module), the behavior is just likean unmodified upstream CPython (case 1).
Before a Python-specific package installer (that is, a tool such aspip - not an external tool such as apt) installs a package into acertain Python context, it should make the following checks bydefault:
sys.prefix==sys.base_prefix (but seeBackwards Compatibility).EXTERNALLY-MANAGED file in the directory identifiedbysysconfig.get_path("stdlib",sysconfig.get_default_scheme())?If both of these conditions are true, the installer should exit withan error message indicating that package installation into this Pythoninterpreter’s directory are disabled outside of a virtual environment.
The installer should have a way for the user to override these rules,such as a command-line flag--break-system-packages. This optionshould not be enabled by default and should carry some connotationthat its use is risky.
TheEXTERNALLY-MANAGED file is an INI-style metadata file intendedto be parsable by the standard libraryconfigparser module. If thefile can be parsed byconfigparser.ConfigParser(interpolation=None) using the UTF-8encoding, and it contains a section[externally-managed], then theinstaller should look for an error message specified in the file andoutput it as part of its error. If the first element of the tuplereturned bylocale.getlocale(locale.LC_MESSAGES), i.e., thelanguage code, is notNone, it should look for the error messageas the value of a key namedError- followed by the language code.If that key does not exist, and if the language code containsunderscore or hyphen, it should look for a key namedError-followed by the portion of the language code before the underscore orhyphen. If it cannot find either of those, or if the language code isNone, it should look for a key simply namedError.
If the installer cannot find an error message in the file (eitherbecause the file cannot be parsed or because no suitable error keyexists), then the installer should just use a pre-defined errormessage of its own, which should suggest that the user create avirtual environment to install packages.
Software distributors who have a non-Python-specific package managerthat manages libraries in thesys.path of their Python packageshould, in general, ship aEXTERNALLY-MANAGED file in theirstandard library directory. For instance, Debian may ship a file in/usr/lib/python3.9/EXTERNALLY-MANAGED consisting of something like
[externally-managed]Error=ToinstallPythonpackagessystem-wide,tryaptinstallpython3-xyz,wherexyzisthepackageyouaretryingtoinstall.Ifyouwishtoinstallanon-Debian-packagedPythonpackage,createavirtualenvironmentusingpython3-mvenvpath/to/venv.Thenusepath/to/venv/bin/pythonandpath/to/venv/bin/pip.Makesureyouhavepython3-fullinstalled.Ifyouwishtoinstallanon-DebianpackagedPythonapplication,itmaybeeasiesttousepipxinstallxyz,whichwillmanageavirtualenvironmentforyou.Makesureyouhavepipxinstalled.See/usr/share/doc/python3.9/README.venvformoreinformation.
which provides useful and distro-relevant informationto a user trying to install a package. Optionally,translations can be provided in the same file:
Error-de_DE=Wenn ist das Nunstück git und Slotermeyer? Ja! Beiherhund das Oder die Virtualenvironment gersput!
In certain contexts, such as single-application container images thataren’t updated after creation, a distributor may choose not to ship anEXTERNALLY-MANAGED file, so that users can install whatever theylike (as they can today) without having to manually override thisrule.
sysconfig schemeUsually, a Python package installer installs to directories in ascheme returned by thesysconfig standard library package.Ordinarily, this is the scheme returned bysysconfig.get_default_scheme(), but based on configuration (e.g.pipinstall--user), it may use a different scheme.
Whenever the installer is installing to asysconfig scheme, thisPEP specifies that the installer should never modify or delete filesoutside of that scheme. For instance, if it’s upgrading a package, andthe package is already installed in a directory outside that scheme(perhaps in a directory from another scheme), it should leave theexisting files alone.
If the installer does end up shadowing an existing installation duringan upgrade, we recommend that it produces a warning at the end of itsrun.
If the installer is installing to a location outside of asysconfig scheme (e.g.,pipinstall--target), then thissubsection does not apply.
This section is non-normative. It provides best practices we believedistros should follow unless they have a specific reason otherwise.
Distros should create anEXTERNALLY-MANAGED file in theirstdlib directory.
The file should contain a useful and distro-relevant error messageindicating both how to install system-wide packages via the distro’spackage manager and how to set up a virtual environment. If yourdistro is often used by users in a state where thepython3 commandis available (and especially wherepip orget-pip isavailable) butpython3-mvenv does not work, the message shouldindicate clearly how to makepython3-mvenv work properly.
Consider packagingpipx, a tool for installing Python-languageapplications, and suggesting it in the error. pipx automaticallycreates a virtual environment for that application alone, which is amuch better default for end users who want to install somePython-language software (which isn’t available in the distro) but arenot themselves Python users. Packaging pipx in the distro avoids theirony of instructing users topipinstall--user--break-system-packagespipx toavoid breaking system packages.Consider arranging things so your distro’s package / environment forPython for end users (e.g.,python3 on Fedora orpython3-fullon Debian) depends on pipx.
Distros that produce official images for single-application containers(e.g., Docker container images) should keep theEXTERNALLY-MANAGED file, preferably in a way that makes it notgo away if a user of that image installs package updates insidetheir image (thinkRUNapt-getdist-upgrade).
Distros should place two separate paths on the system interpreter’ssys.path, one for distro-installed packages and one for packagesinstalled by the local system administrator, and configuresysconfig.get_default_scheme() to point at the latter path. Thisensures that tools like pip will not modify distro-installed packages.The path for the local system administrator should come before thedistro path onsys.path so that local installs take preferenceover distro packages.
For example, Fedora and Debian (and their derivatives) both implementthis split by using/usr/local for locally-installed packages and/usr for distro-installed packages. Fedora uses/usr/local/lib/python3.x/site-packages vs./usr/lib/python3.x/site-packages. (Debian uses/usr/local/lib/python3/dist-packages vs./usr/lib/python3/dist-packages as an additional layer ofseparation from a locally-compiled Python interpreter: if you buildand install upstream CPython in/usr/local/bin, it will look at/usr/local/lib/python3/site-packages, and Debian wishes to makesure that packages installed via the locally-built interpreter don’tshow up onsys.path for the distro interpreter.)
Note that the/usr/local vs./usr split is analogous to howthePATH environment variable typically includes/usr/local/bin:/usr/bin and non-distro software installs to/usr/local by default. This split isrecommended by theFilesystem Hierarchy Standard.
There are two ways you could do this. One is, if you are building andpackaging Python libraries directly (e.g., your packaging helpersunpack aPEP 517-built wheel or callsetup.pyinstall), arrangefor those tools to use a directory that is not in asysconfigscheme but is still onsys.path.
The other is to arrange for the defaultsysconfig scheme to changewhen running inside a package build versus when running on aninstalled system. Thesysconfig customization hooks frombpo-43976 should make this easy (once accepted and implemented):make your packaging tool set anenvironment variable or some other detectable configuration, anddefine aget_preferred_schemes function to return a differentscheme when called from inside a package build. Then you can usepipinstall as part of your distro packaging.
We propose adding a--scheme=... option to instruct pip to runagainst a specific scheme. (SeeImplementation Notes below for howpip currently determines schemes.) Once that’s available, for localtesting and possibly for actual packaging, you would be able to runsomething likepipinstall--scheme=posix_distro to explicitlyinstall a package into your distro’s location (bypassingget_preferred_schemes). One could also, if absolutely needed, usepipuninstall--scheme=posix_distro to use pip to remove packagesfrom the system-managed directory, which addresses the (hopefullytheoretical) regression in use case 5 inRationale.
To install packages with pip, you would also need to either suppresstheEXTERNALLY-MANAGED marker file to allow pip to run or tooverride it on the command line. You may want to use the same meansfor suppressing the marker file in build chroots as you do incontainer images.
The advantage of setting these up to be automatic (suppressing themarker file in your build environment and havingget_preferred_schemes automatically return your distro’s scheme)is that an unadornedpipinstall will work inside a package build,which generally means that an unmodified upstream build script thathappens to internally callpipinstall will do the right thing.You can, of course, just ensure that your packaging process alwayscallspipinstall--scheme=posix_distro--break-system-packages,which would work too.
The best approach here depends a lot on your distro’s conventions andmechanisms for packaging.
Similarly, thesysconfig paths that are not for importable Pythoncode - that is,include,platinclude,scripts, anddata - should also have two variants, one for use bydistro-packaged software and one for use for locally-installedsoftware, and the distro should be set up such that both are usable.For instance, a typical FHS-compliant distro will use/usr/local/include for the default scheme’sinclude and/usr/include for distro-packaged headers and place both on thecompiler’s search path, and it will use/usr/local/bin for thedefault scheme’sscripts and/usr/bin for distro-packagedentry points and place both on$PATH.
All of these mechanisms are proposed for new distro releases and newversions of tools like pip only.
In particular, we strongly recommend that distros with a concept ofmajor versions only add the marker file or changesysconfigschemes in a new major version; otherwise there is a risk that, on anexisting system, software installed via a Python-specific packagemanager now becomes unmanageable (without an override option). For arolling-release distro, if possible, only add the marker file orchangesysconfig schemes in a new Python minor version.
One particular backwards-compatibility difficulty for packageinstallation tools is likely to be managing environments created byold versions ofvirtualenv which have the latest version of thetool installed. A “virtual environment” now has a fairly precisedefinition: it uses thepyvenv.cfg mechanism, which causessys.base_prefix!=sys.prefix. It is possible, however, that auser may have an old virtual environment created by an older versionofvirtualenv; as of this writing, pip supports Python 3.6onwards, which is in turn supported byvirtualenv 15.1.0 onwards,so this scenario is possible. In older versions ofvirtualenv, themechanism is instead to set a new attribute,sys.real_prefix, andit does not use the standard library support for virtual environments,sosys.base_prefix is the same assys.prefix. So the logic forrobustly detecting a virtual environment is something like:
defis_virtual_environment():returnsys.base_prefix!=sys.prefixorhasattr(sys,"real_prefix")
The purpose of this feature is not to implement a security boundary;it is to discourage well-intended changes from unexpectedly breaking auser’s environment. That is to say, the reason this PEP restrictspipinstall outside a virtual environment is not that it’s asecurity risk to be able to do so; it’s that “There should be one–and preferably only one –obvious way to do it,” and that way shouldbe using a virtual environment.pipinstall outside a virtualenvironment is rather too obvious for what is almost always the wrongway to do it.
If there is a case where a user should not be able tosudopipinstall orpipinstall--user and add files tosys.pathforsecurity reasons, that needs to be implemented either via accesscontrol rules on what files the user can write to or an explicitlysecuredsys.path for the program in question. Neither of themechanisms in this PEP should be interpreted as a way to address sucha scenario.
For those reasons, an attempted install with a marker file present isnot a security incident, and there is no need to raise an auditingevent for it. If the calling user legitimately has access tosudopipinstall orpipinstall--user, they can accomplish the sameinstallation entirely outside of Python; if they do not legitimatelyhave such access, that’s a problem outside the scope of this PEP.
The marker file itself is located in the standard library directory,which is a trusted location (i.e., anyone who can write to the markerfile used by a particular installer could, presumably, run arbitrarycode inside the installer). Therefore, there is generally no need tofilter out terminal escape sequences or other potentially-maliciouscontent in the error message.
There are a number of similar proposals we considered that this PEPrejects or defers, largely to preserve the behavior in thecase-by-case analysis inRationale.
Should the marker file be insys.path, marking a particulardirectory as not to be written to by a Python-specific packagemanager? This would help with the second problem addressed by this PEP(not overwriting deleting distro-owned files) but not the first(incompatible installs). A directory-specific marker in/usr/lib/python3.x/site-packages would not discourageinstallations into either/usr/local/lib/python3.x/site-packagesor~/.local/lib/python3.x/site-packages, both of which are onsys.path for/usr/bin/python3. In other words, the marker fileshould not be interpreted as marking a singledirectory asexternally managed (even though it happens to be in a directory onsys.path); it marks the entirePython installation as externallymanaged.
Another variant of the above: should the marker file be insys.path, where if it can be found in any directory insys.path, it marks the installation as externally managed? Anapparent advantage of this approach is that it automatically disablesitself in virtual environments. Unfortunately, This has the wrongbehavior with a--system-site-packages virtual environment, wherethe system-widesys.path is visible but package installations areallowed. (It could work if the rule of exempting virtual environmentsis preserved, but that seems to have no advantage over the currentscheme.)
Should the marker just be a new attribute of asysconfig scheme?There is some conceptual cleanliness to this, except that it’s hard tooverride. We want to make it easy for container images, package buildenvironments, etc. to suppress the marker file. A file that you canremove is easy; code insysconfig is much harder to modify.
Should the file be in/etc? No, because again, it refers to aspecific Python installation. A user who installs their own Python maywell want to install packages within the global context of thatinterpreter.
Should the configuration setting be inpip.conf ordistutils.cfg? Apart from the above objections about marking aninstallation, this mechanism isn’t specific to either of those tools.(It seems reasonable for pip toalso implement a configuration flagfor users to prevent themselves from performing accidentalnon-virtual-environment installs in any Python installation, but thatis outside the scope of this PEP.)
Should the file be TOML? TOML is gaining popularity for packaging (seee.g.PEP 517) but does not yet have an implementation in the standardlibrary. Strictly speaking, this isn’t a blocker - distros need onlywrite the file, not read it, so they don’t need a TOML library (thefile will probably be written by hand, regardless of format), andpackaging tools likely have a TOML reader already. However, the INIformat is currently used for various other forms of packaging metadata(e.g.,pydistutils.cfg andsetup.cfg), meets our needs, and isparsable by the standard library, and the pip maintainers expressed apreference to avoid using TOML for this yet.
Should the file beemail.message-style? While this format is alsoused for packaging metadata (e.g. sdist and wheel metadata) and isalso parsable by the standard library, it doesn’t handle multi-lineentries quite as clearly, and that is our primary use case.
Should the marker file be executable Python code that evaluateswhether installation should be allowed or not? Apart from the concernsabove about having the file insys.path, we have a concern thatmaking it executable is committing to too powerful of an API and risksmaking behavior harder to understand. (Note that theget_default_scheme hook ofbpo-43976 is in fact executable, butthat code needs to be supplied when the interpreter builds; it isn’tintended to be supplied post-build.)
When overriding the marker, should a Python-specific package managerbe disallowed from shadowing a package installed by the externalpackage manager (i.e., installing modules of the same name)? Thiswould minimize the risk of breaking system software, but it’s notclear it’s worth the additional user experience complexity. There arelegitimate use cases for shadowing system packages, and an additionalcommand-line option to permit it would be more confusing. Meanwhile,not passing that option wouldn’t eliminate the risk of breaking systemsoftware, which may be relying on atry:importxyz failing,finding a limited set of entry points, etc. Communicating thisdistinction seems difficult. We think it’s a good idea forPython-specific package managers to print a warning if they shadow apackage, but we think it’s not worth disabling it by default.
Why not use theINSTALLER file fromPEP 376 to determine whoinstalled a package and whether it can be removed? First, it’sspecific to a particular package (it’s in the package’sdist-infodirectory), so like some of the alternatives above, it doesn’t provideinformation on an entire environment and whether package installationsare permissible.PEP 627 also updatesPEP 376 to prevent programmaticuse ofINSTALLER, specifying that the file is “to be used forinformational purposes only. […] Our goal is supportinginteroperating tools, and basing any action on which tool happened toinstall a package runs counter to that goal.” Finally, asPEP 627envisions, there are legitimate use cases for one tool knowing how tohandle packages installed by another tool; for instance,conda cansafely remove a package installed bypip into a Conda environment.
Why does the specification give no means for disabling packageinstallations inside a virtual environment? We can’t see aparticularly strong use case for it (at least not one related to thepurposes of this PEP). If you need it, it’s simple enough topipuninstallpip inside that environment, which should discourage atleast unintentional changes to the environment (and this specificationmakes no provision to disableintentional changes, since after allthe marker file can be easily removed).
Shouldn’t distro software just run with the distrosite-packagesdirectory alone onsys.path and ignore the local systemadministrator’ssite-packages as well as the user-specific one?This is a worthwhile idea, and various versions of it have beencirculating for a while under the name of “system Python” or “platformPython” (with a separate “user Python” for end users writing Python orinstalling Python software separate from the system). However, it’smuch more involved of a change. First, it would be abackwards-incompatible change. As mentioned in theMotivationsection, there are valid use cases for running distro-installed Pythonapplications like Sphinx or Ansible with locally-installed Pythonlibraries available on theirsys.path. A wholesale switch toignoring local packages would break these use cases, and a distrowould have to make a case-by-case analysis of whether an applicationought to see locally-installed libraries or not.
Furthermore,Fedora attempted this change and reverted it, finding,ironically, that their implementation of the changebroke theirpackage manager. Given that experience, there are clearly details tobe worked out before distros can reliably implement that approach, anda PEP recommending it would be premature.
This PEP is intended to be a complete and self-contained change thatis independent of a distributor’s decision for or against “systemPython” or similar proposals. It is not incompatible with a distroimplementing “system Python” in the future, and even though bothproposals address the same class of problems, there are stillarguments in favor of implementing something like “system Python” evenafter implementing this PEP. At the same time, though, this PEPspecifically tries to make a more targeted and minimal change, suchthat it can be implemented by distributors who don’t expect to adopt“system Python” (or don’t expect to implement it immediately). Thechanges in this PEP stand on their own merits and are not anintermediate step for some future proposal. This PEP reduces (but doesnot eliminate) the risk of breaking system software while minimizing(but not completely avoiding) breaking changes, which should thereforebe much easier to implement than the full “system Python” idea, whichcomes with the downsides mentioned above.
We expect that the guidance in this PEP - that users should usevirtual environments whenever possible and that distros should haveseparatesys.path directories for distro-managed andlocally-managed modules - should make further experiments easier inthe future. These may include distributing wholly separate “system”and “user” Python interpreters, running system software out of adistro-owned virtual environment orPYTHONHOME (but shipping asingle interpreter), or modifying the entry points for certainsoftware (such as the distro’s package manager) to use asys.paththat only sees distro-managed directories. Those ideas themselves,however, remain outside the scope of this PEP.
This section is non-normative and contains notes relevant to both thespecification and potential implementations.
Currently, pip does not directly expose a way to choose a targetsysconfig scheme, but it has three ways of looking up schemes wheninstalling:
pipinstallsysconfig.get_default_scheme(), which is usually (inupstream CPython and most current distros) the same asget_preferred_scheme('prefix').pipinstall--prefix=/some/pathsysconfig.get_preferred_scheme('prefix').pipinstall--usersysconfig.get_preferred_scheme('user').Finally,pipinstall--target=/some/path writes directly to/some/path without looking up any schemes.
Debian currently carries apatch to change the default installlocation inside a virtual environment, using a few heuristics(including checking for theVIRTUAL_ENV environment variable),largely so that the directory used in a virtual environment remainssite-packages and notdist-packages. This does notparticularly affect this proposal, because the implementation of thatpatch does not actually change the defaultsysconfig scheme, andnotably does not change the result ofsysconfig.get_path("stdlib").
Fedora currently carries apatch to change the default installlocation when not running inside rpmbuild, which they use toimplement the two-system-wide-directories approach. This isconceptually the sort of hook envisioned bybpo-43976, exceptimplemented as a code patch todistutils instead of as a changedsysconfig scheme.
The implementation ofis_virtual_environment above, as well as thelogic to load theEXTERNALLY-MANAGED file and find the errormessage from it, may as well get added to the standard library(sys andsysconfig, respectively), to centralize theirimplementations, but they don’t need to be added yet.
For additional background on these problems and previous attempts tosolve them, seeDebian bug 771794 “pip silently removes/updatessystem provided python packages” from 2014, Fedora’s 2018 articleMaking sudo pip safe about pointingsudopip at /usr/local(which acknowledges that the changes still do not makesudopipcompletely safe), pip issues5605 (“Disable upgrades to existingpython modules which were not installed via pip”) and5722 (“pipshould respect /usr/local”) from 2018, and the post-PyCon US 2019discussion threadPlaying nice with external package managers.
easy_install command was removed insetuptools version 52, released 23 January 2021.)This document is placed in the public domain or under theCC0-1.0-Universal license, whichever is more permissive.
Source:https://github.com/python/peps/blob/main/peps/pep-0668.rst
Last modified:2025-02-01 08:55:40 GMT