Examples

What would this look like in practice?

Let’s say we have a Python package namedalpha which contains asub-package namebeta. The source directory layout before bytecompilation might look like this:

alpha/__init__.pyone.pytwo.pybeta/__init__.pythree.pyfour.py

After byte compiling this package with Python 3.2, you would see thefollowing layout:

alpha/__pycache__/__init__.cpython-32.pycone.cpython-32.pyctwo.cpython-32.pyc__init__.pyone.pytwo.pybeta/__pycache__/__init__.cpython-32.pycthree.cpython-32.pycfour.cpython-32.pyc__init__.pythree.pyfour.py

Note: listing order may differ depending on the platform.

Let’s say that two new versions of Python are installed, one is Python3.3 and another is Unladen Swallow. After byte compilation, the filesystem would look like this:

alpha/__pycache__/__init__.cpython-32.pyc__init__.cpython-33.pyc__init__.unladen-10.pycone.cpython-32.pycone.cpython-33.pycone.unladen-10.pyctwo.cpython-32.pyctwo.cpython-33.pyctwo.unladen-10.pyc__init__.pyone.pytwo.pybeta/__pycache__/__init__.cpython-32.pyc__init__.cpython-33.pyc__init__.unladen-10.pycthree.cpython-32.pycthree.cpython-33.pycthree.unladen-10.pycfour.cpython-32.pycfour.cpython-33.pycfour.unladen-10.pyc__init__.pythree.pyfour.py

As you can see, as long as the Python version identifier string isunique, any number of pyc files can co-exist. These identifierstrings are described in more detail below.

A nice property of this layout is that the__pycache__ directoriescan generally be ignored, such that a normal directory listing wouldshow something like this:

alpha/__pycache__/__init__.pyone.pytwo.pybeta/__pycache__/__init__.pythree.pyfour.py

This is much less cluttered than even today’s Python.

Case 0: The steady state

When Python is asked to import modulefoo, it searches for afoo.py file (orfoo package, but that’s not important for thisdiscussion) along itssys.path. If found, Python looks to see ifthere is a matching__pycache__/foo.<magic>.pyc file, and if so,thatpyc file is loaded.

Case 1: The first import

When Python locates thefoo.py, if the__pycache__/foo.<magic>.pycfile is missing, Python will create it, also creating the__pycache__ directory if necessary. Python will parse and bytecompile thefoo.py file and save the byte code in__pycache__/foo.<magic>.pyc.

Case 2: The second import

When Python is asked to import modulefoo a second time (in adifferent process of course), it will again search for thefoo.pyfile along itssys.path. When Python locates thefoo.py file, itlooks for a matching__pycache__/foo.<magic>.pyc and finding this,it reads the byte code and continues as usual.

Case 3: __pycache__/foo.<magic>.pyc with no source

It’s possible that thefoo.py file somehow got removed, whileleaving the cached pyc file still on the file system. If the__pycache__/foo.<magic>.pyc file exists, but thefoo.py file usedto create it does not, Python will raise anImportError when askedto import foo. In other words, Python will not import a pyc file fromthe cache directory unless the source file exists.

Case 4: legacy pyc files and source-less imports

Python will ignore all legacy pyc files when a source file exists nextto it. In other words, if afoo.pyc file exists next to thefoo.py file, the pyc file will be ignored in all cases

In order to continue to support source-less distributions though, ifthe source file is missing, Python will import a lone pyc file if itlives where the source file would have been.

Case 5: read-only file systems

When the source lives on a read-only file system, or the__pycache__directory or pyc file cannot otherwise be written, all the same rulesapply. This is also the case when__pycache__ happens to be writtenwith permissions which do not allow for writing containing pyc files.

Detecting PEP 3147 availability

The easiest way to detect whether your version of Python provides PEP3147 functionality is to do the following check:

>>>importimp>>>has3147=hasattr(imp,'get_tag')

__file__

In Python 3, when you import a module, its__file__ attribute pointsto its sourcepy file (in Python 2, it points to thepyc file). Apackage’s__file__ points to thepy file for its__init__.py.E.g.:

>>>importfoo>>>foo.__file__'foo.py'# baz is a package>>>importbaz>>>baz.__file__'baz/__init__.py'

Nothing in this PEP would change the semantics of__file__.

This PEP proposes the addition of an__cached__ attribute tomodules, which will always point to the actualpyc file that wasread or written. When the environment variable$PYTHONDONTWRITEBYTECODE is set, or the-B option is given, or ifthe source lives on a read-only filesystem, then the__cached__attribute will point to the location that thepyc filewould havebeen written to if it didn’t exist. This location of course includesthe__pycache__ subdirectory in its path.

For alternative Python implementations which do not supportpycfiles, the__cached__ attribute may point to whatever informationmakes sense. E.g. on Jython, this might be the.class file for themodule:__pycache__/foo.jython-32.class. Some implementations mayuse multiple compiled files to create the module, in which case__cached__ may be a tuple. The exact contents of__cached__ arePython implementation specific.

It is recommended that when nothing sensible can be calculated,implementations should set the__cached__ attribute toNone.

py_compile and compileall

Python comes with two modules,py_compile[15] andcompileall[16] which support compiling Python modules external to the built-inimport machinery.py_compile in particular has intimate knowledgeof byte compilation, so these will be updated to understand the newlayout. The-b flag is added tocompileall for writing legacy.pyc byte-compiled file path names.

bdist_wininst and the Windows installer

These tools also compile modules explicitly on installation. If theydo not usepy_compile andcompileall, then they would also have tobe modified to understand the new layout.

File extension checks

There exists some code which checks for files ending in.pyc andsimply chops off the last character to find the matching.py file.This code will obviously fail once this PEP is implemented.

To support this use case, we’ll add two new methods to theimppackage[17]:

• imp.cache_from_source(py_path) ->pyc_path
• imp.source_from_cache(pyc_path) ->py_path

Alternative implementations are free to override these functions toreturn reasonable values based on their own support for this PEP.These methods are allowed to returnNone when the implementation (orPEP 302 loader in effect) for whatever reason cannot calculatethe appropriate file name. They should not raise exceptions.

Backports

For versions of Python earlier than 3.2 (and possibly 2.7), it ispossible to backport this PEP. However, in Python 3.2 (and possibly2.7), this behavior will be turned on by default, and in fact, it willreplace the old behavior. Backports will need to support the oldlayout by default. We suggest supportingPEP 3147 through the use ofan environment variable called$PYTHONENABLECACHEDIR or the commandline switch-Xenablecachedir to enable the feature.

Makefiles and other dependency tools

Makefiles and other tools which calculate dependencies on.pyc files(e.g. to byte-compile the source if the.pyc is missing) will haveto be updated to check the new paths.

Hexadecimal magic tags

pyc files inside of the__pycache__ directories contain a magic tagin their file names. These are mnemonic tags for the actual magicnumbers used by the importer. We could have used the hexadecimalrepresentation[10] of the binary magic number as a uniqueidentifier. For example, in Python 3.2:

>>>frombinasciiimporthexlify>>>fromimpimportget_magic>>>'foo.{}.pyc'.format(hexlify(get_magic()).decode('ascii'))'foo.580c0d0a.pyc'

This isn’t particularly human friendly though, thus the magic tagproposed in this PEP.

PEP 304

There is some overlap between the goals of this PEP andPEP 304,which has been withdrawn. HoweverPEP 304 would allow a user tocreate a shadow file system hierarchy in which to storepyc files.This concept of a shadow hierarchy forpyc files could be used tosatisfy the aims of this PEP. Although thePEP 304 does not indicatewhy it was withdrawn, shadow directories have a number of problems.The location of the shadowpyc files would not be easily discoveredand would depend on the proper and consistent use of the$PYTHONBYTECODE environment variable both by the system and by endusers. There are also global implications, meaning that while thesystem might want to shadowpyc files, users might not want to, butthe PEP defines only an all-or-nothing approach.

As an example of the problem, a common (though fragile) Python idiomfor locating data files is to do something like this:

fromosimportdirname,joinimportfoo.bardata_file=join(dirname(foo.bar.__file__),'my.dat')

This would be problematic sincefoo.bar.__file__ will give thelocation of thepyc file in the shadow directory, and it may not bepossible to find themy.dat file relative to the source directoryfrom there.

Fat byte compilation files

An earlier version of this PEP described “fat” Python byte code files.These files would contain the equivalent of multiplepyc files in asinglepyf file, with a lookup table keyed off the appropriate magicnumber. This was an extensible file format so that the first 5parallel Python implementations could be supported fairly efficiently,but with extension lookup tables available to scalepyf byte codeobjects as large as necessary.

The fat byte compilation files were fairly complex, and inherentlyintroduced difficult race conditions, so the current simplification ofusing directories was suggested. The same problem applies to usingzip files as the fat pyc file format.

Multiple file extensions

The PEP author also considered an approach where multiple thin bytecompiled files lived in the same place, but used different fileextensions to designate the Python version. E.g. foo.pyc25,foo.pyc26, foo.pyc31 etc. This was rejected because of the clutterinvolved in writing so many different files. The multiple extensionapproach makes it more difficult (and an ongoing task) to update anytools that are dependent on the file extension.

.pyc

A proposal was floated to call the__pycache__ directory.pyc orsome other dot-file name. This would have the effect on *nix systemsof hiding the directory. There are many reasons why this wasrejected by the BDFL[20] including the fact that dot-files are onlyspecial on some platforms, and we actually donot want to hide thesecompletely from users.