This PEP proposes to add to Python a mechanism for lightweight“virtual environments” with their own site directories, optionallyisolated from system site directories. Each virtual environment hasits own Python binary (allowing creation of environments with variousPython versions) and can have its own independent set of installedPython packages in its site directories, but shares the standardlibrary with the base installed Python.
The utility of Python virtual environments has already been wellestablished by the popularity of existing third-partyvirtual-environment tools, primarily Ian Bicking’svirtualenv.Virtual environments are already widely used for dependency managementand isolation, ease of installing and using Python packages withoutsystem-administrator access, and automated testing of Python softwareacross multiple Python versions, among other uses.
Existing virtual environment tools suffer from lack of support fromthe behavior of Python itself. Tools such asrvirtualenv, which donot copy the Python binary into the virtual environment, cannotprovide reliable isolation from system site directories. Virtualenv,which does copy the Python binary, is forced to duplicate much ofPython’ssite module and manually symlink/copy an ever-changingset of standard-library modules into the virtual environment in orderto perform a delicate boot-strapping dance at every startup.(Virtualenv must copy the binary in order to provide isolation, asPython dereferences a symlinked executable before searching forsys.prefix.)
ThePYTHONHOME environment variable, Python’s only existingbuilt-in solution for virtual environments, requirescopying/symlinking the entire standard library into every environment.Copying the whole standard library is not a lightweight solution, andcross-platform support for symlinks remains inconsistent (even onWindows platforms that do support them, creating them often requiresadministrator privileges).
A virtual environment mechanism integrated with Python and drawing onyears of experience with existing third-party tools can lowermaintenance, raise reliability, and be more easily available to allPython users.
When the Python binary is executed, it attempts to determine itsprefix (which it stores insys.prefix), which is then used to findthe standard library and other key files, and by thesite moduleto determine the location of the site-package directories. Currentlythe prefix is found (assumingPYTHONHOME is not set) by firstwalking up the filesystem tree looking for a marker file (os.py)that signifies the presence of the standard library, and if none isfound, falling back to the build-time prefix hardcoded in the binary.
This PEP proposes to add a new first step to this search. If apyvenv.cfg file is found either adjacent to the Python executable orone directory above it (if the executable is a symlink, it is notdereferenced), this file is scanned for lines of the formkey=value. If ahome key is found, this signifies that the Pythonbinary belongs to a virtual environment, and the value of thehomekey is the directory containing the Python executable used to createthis virtual environment.
In this case, prefix-finding continues as normal using the value ofthehome key as the effective Python binary location, which findsthe prefix of the base installation.sys.base_prefix is set tothis value, whilesys.prefix is set to the directory containingpyvenv.cfg.
(Ifpyvenv.cfg is not found or does not contain thehome key,prefix-finding continues normally, andsys.prefix will be equal tosys.base_prefix.)
Also,sys.base_exec_prefix is added, and handled similarly withregard tosys.exec_prefix. (sys.exec_prefix is the equivalent ofsys.prefix, but for platform-specific files; by default it has thesame value assys.prefix.)
Thesite andsysconfig standard-library modules are modifiedsuch that the standard library and header files are found relativetosys.base_prefix /sys.base_exec_prefix, while site-packagedirectories (“purelib” and “platlib”, insysconfig terms) are stillfound relative tosys.prefix /sys.exec_prefix.
Thus, a Python virtual environment in its simplest form would consistof nothing more than a copy or symlink of the Python binaryaccompanied by apyvenv.cfg file and a site-packages directory.
By default, a virtual environment is entirely isolated from thesystem-level site-packages directories.
If thepyvenv.cfg file also contains a keyinclude-system-site-packages with a value oftrue (not casesensitive), thesite module will also add the system sitedirectories tosys.path after the virtual environment sitedirectories. Thus system-installed packages will still be importable,but a package of the same name installed in the virtual environmentwill take precedence.
PEP 370 user-level site-packages are considered part of the systemsite-packages for venv purposes: they are not available from anisolated venv, but are available from aninclude-system-site-packages=true venv.
This PEP also proposes adding a newvenv module to the standardlibrary which implements the creation of virtual environments. Thismodule can be executed using the-m flag:
python3-mvenv/path/to/new/virtual/environment
Apyvenv installed script is also provided to make this moreconvenient:
pyvenv/path/to/new/virtual/environment
Running this command creates the target directory (creating any parentdirectories that don’t exist already) and places apyvenv.cfg filein it with ahome key pointing to the Python installation thecommand was run from. It also creates abin/ (orScripts onWindows) subdirectory containing a copy (or symlink) of thepython3executable, and thepysetup3 script from thepackaging standardlibrary module (to facilitate easy installation of packages from PyPIinto the new venv). And it creates an (initially empty)lib/pythonX.Y/site-packages (orLib\site-packages on Windows)subdirectory.
If the target directory already exists an error will be raised, unlessthe--clear option was provided, in which case the targetdirectory will be deleted and virtual environment creation willproceed as usual.
The createdpyvenv.cfg file also includes theinclude-system-site-packages key, set totrue ifpyvenv isrun with the--system-site-packages option,false by default.
Multiple paths can be given topyvenv, in which case an identicalvenv will be created, according to the given options, at eachprovided path.
Thevenv module also places “shell activation scripts” for POSIX andWindows systems in thebin orScripts directory of thevenv. These scripts simply add the virtual environment’sbin (orScripts) directory to the front of the user’s shell PATH. This isnot strictly necessary for use of a virtual environment (as an explicitpath to the venv’s python binary or scripts can just as well be used),but it is convenient.
In order to allowpysetup and other Python package managers toinstall packages into the virtual environment the same way they wouldinstall into a normal Python installation, and avoid special-casingvirtual environments insysconfig beyond usingsys.base_prefixin place ofsys.prefix where appropriate, the internal virtualenvironment layout mimics the layout of the Python installation itselfon each platform. So a typical virtual environment layout on a POSIXsystem would be:
pyvenv.cfgbin/python3bin/pythonbin/pysetup3include/lib/python3.3/site-packages/
While on a Windows system:
pyvenv.cfgScripts/python.exeScripts/python3.dllScripts/pysetup3.exeScripts/pysetup3-script.py...otherDLLsandpyds...Include/Lib/site-packages/
Third-party packages installed into the virtual environment will havetheir Python modules placed in thesite-packages directory, andtheir executables placed inbin/ orScripts.
Note
On a normal Windows system-level installation, the Python binaryitself wouldn’t go inside the “Scripts/” subdirectory, as it doesin the default venv layout. This is useful in a virtualenvironment so that a user only has to add a single directory totheir shell PATH in order to effectively “activate” the virtualenvironment.
Note
On Windows, it is necessary to also copy or symlink DLLs and pydfiles from compiled stdlib modules into the env, because if thevenv is created from a non-system-wide Python installation,Windows won’t be able to find the Python installation’s copies ofthose files when Python is run from the venv.
This approach explicitly chooses not to introduce a new sysconfiginstall scheme for venvs. Rather, by modifyingsys.prefix weensure that existing install schemes which base locations onsys.prefix will simply work in a venv. Installation to otherinstall schemes (for instance, the user-site schemes) whose paths arenot relative tosys.prefix, will not be affected by a venv at all.
It may be feasible to create an alternative implementation of Pythonvirtual environments based on a virtual-specific sysconfig scheme, butit would be less robust, as it would require more code to be aware ofwhether it is operating within a virtual environment or not.
The technique in this PEP works equally well in general with a copiedor symlinked Python binary (and other needed DLLs on Windows).Symlinking is preferable where possible, because in the case of anupgrade to the underlying Python installation, a Python executablecopied in a venv might become out-of-sync with the installed standardlibrary and require manual upgrade.
There are some cross-platform difficulties with symlinks:
Thus, this PEP proposes to symlink the binary on all platforms exceptfor Windows, and OS X framework builds. A--symlink option isavailable to force the use of symlinks on Windows versions thatsupport them, if the appropriate permissions are available. (Thisoption has no effect on OS X framework builds, since symlinking cannever work there, and has no advantages).
On Windows, if--symlink is not used, this means that if theunderlying Python installation is upgraded, the Python binary and DLLsin the venv should be updated, or there could be issues of mismatchwith the upgraded standard library. The pyvenv script accepts a--upgrade option for easily performing this upgrade on an existingvenv.
Current virtualenv handles include files in this way:
On POSIX systems where the installed Python’s include files are found in${base_prefix}/include/pythonX.X, virtualenv creates${venv}/include/ and symlinks${base_prefix}/include/pythonX.Xto${venv}/include/pythonX.X. On Windows, where Python’s includefiles are found in{{sys.prefix}}/Include and symlinks are notreliably available, virtualenv copies{{sys.prefix}}/Include to${venv}/Include. This ensures that extension modules built andinstalled within the virtualenv will always find the Python header filesthey need in the expected location relative tosys.prefix.
This solution is not ideal when an extension module installs its ownheader files, as the default installation location for those headerfiles may be a symlink to a system directory that may not bewritable. One installer, pip, explicitly works around this byinstalling header files to a nonstandard location${venv}/include/site/pythonX.X/, as in Python there’s currently nostandard abstraction for a site-specific include directory.
This PEP proposes a slightly different approach, though one withessentially the same effect and the same set of advantages anddisadvantages. Rather than symlinking or copying include files into thevenv, we simply modify the sysconfig schemes so that header files arealways sought relative tobase_prefix rather thanprefix. (Wealso create aninclude/ directory within the venv, so installershave somewhere to put include files installed within the env).
Better handling of include files in distutils/packaging and, byextension, pyvenv, is an area that may deserve its own future PEP. Fornow, we propose that the behavior of virtualenv has thus far proveditself to be at least “good enough” in practice.
The high-level method described above makes use of a simple API whichprovides mechanisms for third-party virtual environment creators tocustomize environment creation according to their needs.
Thevenv module contains anEnvBuilder class which accepts thefollowing keyword arguments on instantiation:
system_site_packages - A Boolean value indicating that thesystem Python site-packages should be available to the environment.Defaults toFalse.clear - A Boolean value which, if true, will delete any existingtarget directory instead of raising an exception. Defaults toFalse.symlinks - A Boolean value indicating whether to attempt tosymlink the Python binary (and any necessary DLLs or other binaries,e.g.pythonw.exe), rather than copying. Defaults toFalse.The instantiated env-builder has acreate method, which takes asrequired argument the path (absolute or relative to the currentdirectory) of the target directory which is to contain the virtualenvironment. Thecreate method either creates the environment inthe specified directory, or raises an appropriate exception.
Thevenv module also provides a module-levelcreate functionas a convenience:
defcreate(env_dir,system_site_packages=False,clear=False,use_symlinks=False):builder=EnvBuilder(system_site_packages=system_site_packages,clear=clear,use_symlinks=use_symlinks)builder.create(env_dir)
Creators of third-party virtual environment tools are free to use theprovidedEnvBuilder class as a base class.
Thecreate method of theEnvBuilder class illustrates thehooks available for customization:
defcreate(self,env_dir):""" Create a virtualized Python environment in a directory. :param env_dir: The target directory to create an environment in. """env_dir=os.path.abspath(env_dir)context=self.create_directories(env_dir)self.create_configuration(context)self.setup_python(context)self.post_setup(context)
Each of the methodscreate_directories,create_configuration,setup_python, andpost_setup can be overridden. The functionsof these methods are:
create_directories - creates the environment directory and allnecessary directories, and returns a context object. This is just aholder for attributes (such as paths), for use by the other methods.create_configuration - creates thepyvenv.cfg configurationfile in the environment.setup_python - creates a copy of the Python executable (and,under Windows, DLLs) in the environment.post_setup - A (no-op by default) hook method which can beoverridden in third party subclasses to pre-install packages orinstall scripts in the virtual environment.In addition,EnvBuilder provides a utility method that can becalled frompost_setup in subclasses to assist in installingcustom scripts into the virtual environment. The methodinstall_scripts accepts as arguments thecontext object (seeabove) and a path to a directory. The directory should containsubdirectories “common”, “posix”, “nt”, each containing scriptsdestined for the bin directory in the environment. The contents of“common” and the directory corresponding toos.name are copiedafter doing some text replacement of placeholders:
__VENV_DIR__ is replaced with absolute path of the environmentdirectory.__VENV_NAME__ is replaced with the environment name (final pathsegment of environment directory).__VENV_BIN_NAME__ is replaced with the name of the bin directory(eitherbin orScripts).__VENV_PYTHON__ is replaced with the absolute path of theenvironment’s executable.TheDistributeEnvBuilder subclass in the reference implementationillustrates how the customization hook can be used in practice topre-install Distribute into the virtual environment. It’s notenvisaged thatDistributeEnvBuilder will be actually added toPython core, but it makes the reference implementation moreimmediately useful for testing and exploratory purposes.
sys.prefixAny virtual environment tool along these lines (which attempts toisolate site-packages, while still making use of the base Python’sstandard library with no need for it to be symlinked into the virtualenvironment) is proposing a split between two different meanings(among others) that are currently both wrapped up insys.prefix:the answers to the questions “Where is the standard library?” and“Where is the site-packages location where third-party modules shouldbe installed?”
This split could be handled by introducing a newsys attribute foreither the former prefix or the latter prefix. Either optionpotentially introduces some backwards-incompatibility with softwarewritten to assume the other meaning forsys.prefix. (Suchsoftware should preferably be using the APIs in thesite andsysconfig modules to answer these questions rather than usingsys.prefix directly, in which case there is nobackwards-compatibility issue, but in practicesys.prefix issometimes used.)
Thedocumentation forsys.prefix describes it as “A stringgiving the site-specific directory prefix where the platformindependent Python files are installed,” and specifically mentions thestandard library and header files as found undersys.prefix. Itdoes not mentionsite-packages.
Maintaining this documented definition would mean leavingsys.prefix pointing to the base system installation (which iswhere the standard library and header files are found), andintroducing a new value insys (something likesys.site_prefix) to point to the prefix forsite-packages.This would maintain the documented semantics ofsys.prefix, butrisk breaking isolation if third-party code usessys.prefix ratherthansys.site_prefix or the appropriatesite API to findsite-packages directories.
The most notable case is probablysetuptools and its forkdistribute, which mostly usedistutils andsysconfig APIs,but do usesys.prefix directly to build up a list of sitedirectories for pre-flight checking wherepth files can usefully beplaced.
Otherwise, a Google Code Search turns up what appears to be aroughly even mix of usage between packages usingsys.prefix tobuild up a site-packages path and packages using it to e.g. eliminatethe standard-library from code-execution tracing.
Although it requires modifying the documented definition ofsys.prefix, this PEP prefers to havesys.prefix point to thevirtual environment (wheresite-packages is found), and introducesys.base_prefix to point to the standard library and Python headerfiles. Rationale for this choice:
sys.prefix to point at the virtualenvironment, and in practice this has not been a problem.The majority of this PEP’s changes occur in the standard library, which isshared by other Python implementations and should not present anyproblem.
Other Python implementations will need to replicate the newsys.prefix-finding behavior of the interpreter bootstrap, includinglocating and parsing thepyvenv.cfg file, if it is present.
The reference implementation is found ina clone of the CPythonMercurial repository. To test it, build and runbin/pyvenv/path/to/new/venv to create a virtual environment.
This document has been placed in the public domain.
Source:https://github.com/python/peps/blob/main/peps/pep-0405.rst
Last modified:2025-02-01 08:59:27 GMT