This PEP proposes extending the existing mechanism for setting upsys.pathto include a new__pypackages__ directory, in addition to the existinglocations. The new directory will be added at the start ofsys.path, afterthe current working directory and just before the system site-packages, to givepackages installed there priority over other locations.
This is similar to the existing mechanism of adding the current directory (orthe directory the script is located in), but by using a subdirectory,additional libraries are kept separate from the user’s work.
New Python programmers can benefit from being taught the value of isolating anindividual project’s dependencies from their system environment. However, theexisting mechanism for doing this, virtual environments, is known to be complexand error-prone for beginners to understand. Explaining virtual environments isoften a distraction when trying to get a group of beginners set up - differencesin platform and shell environments require individual assistance, and the needfor activation in every new shell session makes it easy for students to makemistakes when coming back to work after a break. This proposal offers a lightweightsolution that gives isolation without the user needing to understand moreadvanced concepts.
Furthermore, standalone Python applications usually need 3rd party libraries tofunction. Typically, they are either designed to be run from a virtual environment,where the dependencies are installed into the environment alongside the application,or they bundle their dependencies in a subdirectory, and modifysys.path atapplication startup. Virtual environments, while a common and effective solution(used, for example, by thepipx tool), are somewhat awkward to set up and manage,and are not relocatable. On the other hand, manual manipulation ofsys.path isboilerplate that developers need to get right, and (being a runtime behaviour)it is not understood by tools like linters and type checkers. The__pypackages__proposal formalises the idea of a “bundled dependencies” location, avoiding theboilerplate and providing a standard location that development tools can be taughtto recognise.
It should be noted that in general, Python libraries cannot be simply copiedbetween machines, platforms, or even necessarily between Python versions. Thisproposal does nothing to change that fact, and while it is tempting to assumethat bundling a script and its__pypackages__ is a mechanism fordistributing applications, this is explicitlynot a goal of this proposal.Developers remain responsible for the portability of their code.
Whilesys.path can be manipulated at runtime, the default value is important, asit establishes a common baseline that users and tools can agree on. The current defaultdoes not include a location that could be viewed as “private to the current project”,and yet that is a useful concept.
This is similar to the npmnode_modules directory, which is popular in theJavascript community, and something that developers familiar with thatecosystem often ask for from Python.
This PEP proposes to add a new step in the process of calculatingsys.path atstartup.
When the interactive interpreter starts, if a__pypackages__ directory isfound in the current working directory, then it will be included insys.path after the entry for current working directory and just before thesystem site-packages.
When the interpreter runs a script, Python will try to find__pypackages__in the same directory as the script. If found (along with the current Pythonversion directory inside), then it will be used, otherwise Python will behaveas it does currently.
The behaviour should work exactly the same as the way the existing mechanismfor adding the current working directory or script directory tosys.pathworks. For example,__pypackages__ will be ignored if the-P option orthePYTHONSAFEPATH environment variable is set.
In order to be recognised, the__pypackages__ directory must be laid outaccording to a newlocalpackages scheme in the sysconfig module.Specifically, both of thepurelib andplatlib directories must bepresent, using the following code to determine the locations of thosedirectories:
scheme="localpackages"purelib=sysconfig.get_path("purelib",scheme,vars={"base":"__pypackages__","platbase":"__pypackages__"})platlib=sysconfig.get_path("platlib",scheme,vars={"base":"__pypackages__","platbase":"__pypackages__"})
These two locations will be added tosys.path, other directories orfiles in the__pypackages__ directory will be silently ignored. Thepaths will be based on Python versions.
Note
There is a possible option of having a separate new API, it is documented atissue #3013.
The following shows an example project directory structure, and different waysthe Python executable and any script will behave. The example is for Unix-likesystems - on Windows the subdirectories will be different.
foo__pypackages__libpython3.10site-packagesbottlemyscript.py/>pythonfoo/myscript.pysys.path[0]=='foo'sys.path[1]=='foo/__pypackages__/lib/python3.10/site-packages/'cdfoofoo>/usr/bin/ansible#! /usr/bin/env python3foo>python/usr/bin/ansiblefoo>pythonmyscript.pyfoo>pythonsys.path[0]=='.'sys.path[1]=='./__pypackages__/lib/python3.10/site-packages'foo>python-mbottle
We have a project directory calledfoo and it has a__pypackages__inside of it. We havebottle installed in that__pypackages__/lib/python3.10/site-packages/, and have amyscript.pyfile inside of the project directory. We have used whatever tool we generallyuse to installbottle in that location.
For invoking a script, Python will try to find a__pypackages__ inside ofthe directory that the script resides[1],/usr/bin. The same will happenin case of the last example, where we are executing/usr/bin/ansible frominside of thefoo directory. In both cases, it willnot use the__pypackages__ in the current working directory.
Similarly, if we invokemyscript.py from the first example, it will use the__pypackages__ directory that was in thefoo directory.
If we go inside of thefoo directory and start the Python executable (theinterpreter), it will find the__pypackages__ directory inside of thecurrent working directory and use it in thesys.path. The same happens if wetry to use the-m and use a module. In our example,bottle module willbe found inside of the__pypackages__ directory.
The above two examples are only cases where__pypackages__ from currentworking directory is used.
In another example scenario, a trainer of a Python class can say “Today we aregoing to learn how to use Twisted! To start, please checkout our exampleproject, go to that directory, and then run a given command to install Twisted.”
That will install Twisted into a directory separate frompython3. There’s noneed to discuss virtual environments, global versus user installs, etc. as theinstall will be local by default. The trainer can then just keep telling them tousepython3 without any activation step, etc.
At its heart, this proposal is simply to modify the calculation of the defaultvalue ofsys.path, and does not relate at all to the virtual environmentmechanism. However,__pypackages__ can be viewed as providing an isolationcapability, and in that sense, it “competes” with virtual environments.
However, there are significant differences:
- Virtual environments are isolated from the system environment, whereas
__pypackages__simply adds to the system environment.- Virtual environments include a full “installation scheme”, with directoriesfor binaries, C header files, etc., whereas
__pypackages__is solelyfor Python library code.- Virtual environments work most smoothly when “activated”. This proposalneeds no activation.
This proposal should be seen as independent of virtual environments, not competingwith them. At best, some use cases currently only served by virtual environmentscan also be served (possibly better) by__pypackages__.
It should be noted that libraries installed in__pypackages__ will be visiblein a virtual environment. This arguably breaks the isolation of virtual environments,but it is no different in principle to the presence of the current directory onsys.path (or mechanisms like thePYTHONPATH environment variable). The onlydifference is in degree, as the expectation is that people will more commonly installpackages in__pypackages__. The alternative would be to explicitly detect virtualenvironments and disable__pypackages__ in that case - however that would breakscripts with bundled dependencies. The PEP authors believe that developers usingvirtual environments should be experienced enough to understand the issue andanticipate and avoid any problems.
In theory, it is possible to add a library to the__pypackages__ directorythat overrides a stdlib module or an installed 3rd party library. For the__pypackages__ associated with a script, this is assumed not to be asignificant issue, as it is unlikely that anyone would be able to write to__pypackages__ unless they also had the ability to write to the script itself.
For a__pypackages__ directory in the current working directory, theinteractive interpreter could be affected. However, this is not significantlydifferent than the existing issue of someone having amath.py module in theircurrent directory, and while (just like that case) it can cause user confusion,it does not introduce any new security implications.
When running a script, any__pypackages__ directory in the current workingdirectory is ignored. This is the same approach Python uses for adding thecurrent working directory tosys.path and ensures that it is not possibleto change the behaviour of a script by modifying files in the currentdirectory.
Also, a__pypackages__ directory is only recognised in the current (orscript) directory. The interpreter willnot scan for__pypackages__ inparent directories. Doing so would open up the risk of security issues ifdirectory permissions on parents differ. In particular, scripts in thebindirectory or__pypackages__ (thescripts location insysconfigterms) have no special access to the libraries installed in__pypackages__.Putting executable scripts in abin directory is not supported by thisproposal.
The original motivation for this proposal was to make it easier to teach Pythonto beginners. To that end, it needs to be easy to explain, and simple to use.
At the most basic level, this is similar to the existing mechanism where thescript directory is added tosys.path and can be taught in a similar manner.However, for its intended use of “lightweight isolation”, it would likely be taughtin terms of “things you put in a__pypackages__ directory are private to yourscript”. The experience of the PEP authors suggests that this would be significantlyeasier to teach than the current alternative of introducing virtual environments.
As the intended use of the feature is to install 3rd party libraries in the newdirectory, it is important that tools, particularly installers, understand how tomanage__pypackages__.
It is hoped that tools will introduce a dedicated “pypackages” installationmode thatis guaranteed to match the expected layout in all cases. However,the question of how best to support the__pypackages__ layout is ultimatelyleft to individual tool maintainers to consider and decide on.
Tools that locate packages without actually running Python code (IDEs, linters,type checkers, etc.) would need updating to recognise__pypackages__. In theabsence of such updates, the__pypackages__ directory would work similarlyto directories currently added tosys.path at runtime (i.e., the tool wouldprobably ignore it).
The directory name__pypackages__ was chosen because it is unlikely to be incommon use. It is true that users who have chosen to use that name for their ownpurposes will be impacted, but at the time this PEP was written, this was viewedas a relatively low risk.
Unfortunately, in the time this PEP has been under discussion, a number of toolshave chosen to implement variations on what is being proposed here, which are notall compatible with the final form of the PEP. As a result, the risk of clashes isnow higher than originally anticipated.
It would be possible to mitigate this by choosing adifferent name, hopefully asuncommon as__pypackages__ originally was. But realistically, any compatibilityissues can be viewed as simply the consequences of people trying to implementdraft proposals, without making the effort to track changes in the proposal. As such,it seems reasonable to retain the__pypackages__ name, and put the burden ofaddressing the compatibility issue on the tools that implemented the draft version.
Other Python implementations will need to replicate the new behavior of theinterpreter bootstrap, including locating the__pypackages__ directory andadding it thesys.path just before site packages, if it is present. This isno different to any other Python change.
Here is a small script which willenable the implementation forCpython & inPyPy.
__pylocal__ andpython_modules. Ultimately, the name is arbitrary and the chosen name is good enough.__pypackages__. If we want to execute scripts inside of the~/bin/ directory, then the__pypackages__ directory must be inside of the~/bin/ directory. Doing any such scan for__pypackages__ (for the interpreter or a script) will have security implications and also increase startup time.__pypackages__. This is considered too strict, particularly as transitional approaches likepipinstall--prefix can create additional files in__pypackages__.sysconfig scheme, or a dedicatedpypackages scheme. While this is attractive in theory, it makes transition harder, as there will be no readily-available way of installing to__pypackages__ until tools implement explicit support. And while the PEP authors hope and assume that such support would be added, having the proposal dependent on such support in order to be usable seems like an unacceptable risk.This document has been placed in the public domain.
Source:https://github.com/python/peps/blob/main/peps/pep-0582.rst
Last modified:2025-02-01 08:59:27 GMT