Limitations ofpth files

If a library needs to perform any customization before an import or thatrelates to the general working of the interpreter, they often rely on thefact thatpth files, which are loaded at startup and implemented via thesite module[7], can include Python code that will be executed when thepth file is evaluated.

Note thatpth files were originally developed to just add additionaldirectories tosys.path, but they may also contain lines which startwith “import”, which will be passed toexec(). Users have exploited thisfeature to allow the customizations that they needed. See setuptools[4] or betterexceptions[5] as examples.

Usingpth files for this purpose is far from ideal for library developers,as they need to inject code into a single line preceded by an import, makingit rather unreadable. Library developers following that practice will usuallycreate a module that performs all actions on import, as done bybetterexceptions[5], but the approach is still not reallyuser friendly.

Additionally, it is also non-ideal for users of the interpreter if they wantto inspect what is being executed at Python startup as they need to reviewall thepth files for potential code execution which can be spread acrossall site paths. Most of thosepth files will be “legitimate”pthfiles that just modify the path, answering the question of “what is changingmy interpreter at startup” a rather complex one.

Lastly, there have been multiple suggestions for removing code execution frompth files, see[1] and[2].

Limitations ofsitecustomize.py

Whilst sitecustomize is an acceptable solution, it assumes a single person isin charge of the system and the interpreter. If both the system administratorand the responsibility of provisioning the interpreter want to addcustomizations at the interpreter startup they need to agree on the contentsof the file and combine all the changes. This is not a major limitationthough, and it is not the main driver of this change. Should the changehappen, it will also improve the situation for these users, as rather thanhaving asitecustomize.py which performs all those actions, they can havecustom isolated files named after the features they want to enhance. As anexample, Ubuntu could change their currentsitecustomize.py to just beubuntu_apport_python_hook. This not only better represents its intent butalso gives users of the interpreter a better understanding of themodifications happening on their interpreter.

Why__sitecustomize__

The name aims to follow the already existing concept ofsitecustomize.py.As the directory will be withinsys.path, given that it is located insite paths, we choose to use double underscore around its name, to preventcolliding with the already existingsitecustomize.py.

Discovering the new__sitecustomize__ directories

The Python interpreter will look at startup for directory named__sitecustomize__ within any of the standard site-packages path.

These are commonly the Python system location and the user location, but areultimately defined by the site module logic.

Users can usesite.sitepackages[8] andsite.usersitepackages[9] to know the paths wherethe interpreter can discover__sitecustomize__ directories.

Time of__sitecustomize__ discovery

The__sitecustomize__ directories will be discovered exactly afterpthfiles are discovered in a site-packages path as part ofsite.addsitedir[10].

These is repeated for each of the site-packages path in the exact same orderthat is being followed today forpth files.

Order of execution within__sitecustomize__

The implementation will execute the files within__sitecustomize__ bysorting them by name when discovering each of the__sitecustomize__directories. We discourage users to rely on the order of execution though.

We considered executing them in random order, but that could result indifferent results depending on how the interpreter chooses to pick up thosefiles. So even if it won’t be a good practice to rely on other files beingexecuted, we think that is better than having randomly different results oninterpreter startup. We chose to run the files after thepth files incase a user needs to add items to the path before running a files.

Interaction withpth files

pth files can be used to add paths intosys.path, but this should notaffect the__sitecustomize__ discovery process, as those directories arelooked up exclusively in site-packages paths.

Execution of files within__sitecustomize__

When a__sitecustomize__ directory is discovered, all of the files thathave a.py extension within it will be read withio.open_code andexecuted by usingexec[11].

An empty dictionary will be passed asglobals to theexec functionto prevent unexpected interactions between different files.

Failure handling

Any error on the execution of any of the files will not be logged unless theinterpreter is run in verbose mode and it should not stop the evaluation ofother files. The user will receive a message in stderr saying that the filefailed to be executed and that verbose mode can be used to get moreinformation. This behaviour mimics the one existing forsitecustomize.py.

Interaction with virtual environments

The customizations applied to an interpreter via the new__sitecustomize__ solutions will continue to work when a user creates avirtual environment the same way thatsitecustomize.pyinteract with virtual environments.

This is a difference when compared topth files, which are not propagatedinto virtual environments unlessinclude-system-site-packages is enabled.

If library maintainers have features installed via__sitecustomize__ thatthey do not want to propagate into virtual environments, they should detectif they are running within a virtual environment by checkingsys.prefix==sys.base_prefix. This behavior is similar to packages that modify the globalsitecustomize.py.

Interaction withsitecustomize.py andusercustomize.py

Until removed,sitecustomize andusercustomize will be executed after__sitecustomize__ similar to pth files. See the Backward compatibilitysection for information on removal plans forsitecustomize andusercustomize.

Identifying all installed files

To facilitate debugging of the Python startup, if the site module is invokedit will print the__sitecustomize__ directories that will be discoveredon startup.

Files naming convention

Packages will be encouraged to include the name of the package within thename of the file to avoid collisions between packages. But the onlyrequirement on the filename is that it ends in.py for the interpreter toexecute them.

Disabling start files

In some scenarios, like when the startup time is key, it might be desired todisable this option altogether. The already existing flag-S[3]will disable allsite-related manipulation, including this new feature.If the flag is passed in,__sitecustomize__ directories will not bediscovered.

Additionally, to allow for starting the interpreter disabling only this newfeature a new option will be added under-X:disablesitecustomize,which will disable the discovery of__sitecustomize__ exclusively.

Lastly, the user can disable the discovery of__sitecustomize__directories only in the user site by disabling the user site via any of themultiple options in thesite.py module.

Support in build backends

Whilst build backends can choose to provide an option to facilitate theinstallation of these files into a__sitecustomize__ directory, thisPEP does not address that directly. Similar topth files, build backendscan choose to not provide an easy-to-configure mechanism for__sitecustomize__ files and let users hook into the installationprocess to include such files. We do not think build backends enhancedsupport as a requirement for this PEP.

Impact on startup time

A concern in this implementation is how Python interpreter startup time canbe affected by this addition. We expect the performance impact to be highlycoupled to the logic in the files that a user or sysadmin installs in thePython environment being tested.

If the interpreter has any files in their__sitecustomize__ directory,the file execution time plus a call reading the code will be added to thestartup time. This is similar to how code execution is impacting startup timethroughsitecustomize.py,usercustomize.py and code inpth files.We will therefore focus here on comparing this solution against those three,as otherwise the actual time added to startup is highly dependent on the codethat is being executed in those files.

Results were gathered by running “./python.exe -c pass” with perf on 50iterations, repeating 50 times the command on each iteration and getting thegeometric mean of all the results. The file used to run those benchmarks ischecked in in the reference implementation[6].

The benchmark was run with 3.10 alpha 7 compiled with PGO and LTO with thefollowing parameters and system state:

• Perf event: Max sample rate set to 1 per second
• CPU Frequency: Minimum frequency of CPU 17,35 set to the maximum frequency
• Turbo Boost (MSR): Turbo Boost disabled on CPU 17: MSR 0x1a0 set to 0x4000850089
• IRQ affinity: Set default affinity to CPU 0-16,18-34
• IRQ affinity: Set affinity of IRQ 1,3-16,21,25-31,56-59,68-85,87,89-90,92-93,95-104 to CPU 0-16,18-34
• CPU: use 2 logical CPUs: 17,35
• Perf event: Maximum sample rate: 1 per second
• ASLR: Full randomization
• Linux scheduler: Isolated CPUs (2/36): 17,35
• Linux scheduler: RCU disabled on CPUs (2/36): 17,35
• CPU Frequency: 0-16,18-34=min=1200 MHz, max=3600 MHz; 17,35=min=max=3600 MHz
• Turbo Boost (MSR): CPU 17,35: disabled

The code placed to be executed inpth files,sitecustomize.py,usercustomize.py and files within__sitecustomize__ is the following:

import time; x = time.time() ** 5

The file is aimed at execution a simple operation but still expected to benegligible. This is to put the experiment in a situation where we makevisible any hit on performance due to the mechanism whilst still making itrelatively realistic. Additionally, it starts with an import and is a singleline to be able to be used inpth files.

Results can be reproduced withrun-benchmark.py script provided in thereference implementation[6].

We interpret the following from these results:

• Using two__sitecustomize__ scripts compared tositecustomize.pyandusercustomize.py slows down the interpreter by 0.3%. We expect thisslowdown untilsitecustomize.py andusercustomize.py are removed ina future release as even if the user does not create the files, theinterpreter will still attempt to import them.
• With the arbitrary 50 pth files with code tested, moving those to__sitecustomize__ produces a speedup of ~3.5% in startup. Which is likelyrelated to the simpler logic to evaluate__sitecustomize__ files comparedtopth file execution.
• In general all measurements show that there is a low impact on startup timewith this addition.

Audit Event

A new audit event will be added and triggered on__sitecustomize__execution to facilitate security inspection by callingsys.audit[12] with “sitecustimze.exec_file” as name and the filename asargument.

Security implications

This PEP aims to move all code execution frompth files to files within a__sitecustomize__ directory. We think this is an improvement to system adminsfor the following reasons:

• Allows to quickly identify the code being executed at startup time by theinterpreter by looking into a single directory rather than having to scanallpth files.
• Allows to track usage of this feature through the new proposed audit event.
• Gives finer grain control by allowing to tune permissions on the__sitecustomize__ directory, potentially allowing users to install onlypackages that does not change the interpreter startup.

In short, whilst this allows for a malicious users to drop a file that willbe executed at startup, it’s an improvement compared to the existingpthfiles.

Do nothing

Whilst the current status “works” it presents the issues listed in themotivation. After analyzing the impact of this change, we believe it is worthit, given the enhanced experience it brings.

Formalize usingpth files

Another option would be to just glorify and document the usage ofpth filesto inject code at startup code, but that is a suboptimal experience for usersas listed in the motivation.

Making__sitecustomize__ a namespace package

We considered making the directory a namespace package and just import allthe modules within it, which allowed searching across all paths insys.path at initialization time and provided a way to declaredependencies between files by importing each other. This was rejected formultiple reasons:

1. This was unnecessarily broadening the list of paths where arbitrary filesare executed.
2. The logic brought additional complexity, like what to do if a package wereto install an__init__.py file in one of the locations.
3. It’s cheaper to search for__sitecustomize__ as we are looking forpth files already in the site paths compared to performing an actualimport of a namespace package.

Support for shutdown customization

init.d users might be tempted to implement this feature in a way that userscould also add code at shutdown, but extra support for that is not needed, asPython users can already do that viaatexit.

Using entry_points

We considered extending the use of entry points to allow specifying filesthat should be executed at startup but we discarded that solution due to twomain reasons. The first one being impact on startup time. This approach willrequire scanning all packages distribution information to just execute ahandful of files. This has an impact on performance even if the user is notusing the feature and such impact growths linearly with the number of packagesinstalled in the environment. The second reason was that the proposedimplementation in this PEP offers a single solution for startup customizationfor packages and system administrators. Additionally, if the main objective ofentry points is to make it easy for libraries to install files at startup,that can still be added and make the build backends just install the fileswithin the__sitecustomize__ directory.