How distributions are installed

Right now, when a distribution is installed in Python, every element canbe installed in a different directory.

For instance,Distutils installs the pure Python code in thepurelibdirectory, which islib/python2.6/site-packages for unix-like systems andMac OS X, orLib\site-packages under Python’s installation directory forWindows.

Additionally, theinstall_egg_info subcommand of the Distutilsinstallcommand adds an.egg-info file for the project into thepurelibdirectory.

For example, for thedocutils distribution, which contains one package anextra module and executable scripts, three elements are installed insite-packages:

• docutils: Thedocutils package.
• roman.py: An extra module used bydocutils.
• docutils-0.5-py2.6.egg-info: A file containing the distribution metadataas described inPEP 314. This file corresponds to the filecalledPKG-INFO, built by thesdist command.

Some executable scripts, such asrst2html.py, are also added in thebin directory of the Python installation.

Another project calledsetuptools[1] has two other formatsto install distributions, calledEggFormats[4]:

• a self-contained.egg directory, that contains all the distribution filesand the distribution metadata in a file calledPKG-INFO in a subdirectorycalledEGG-INFO.setuptools creates other files in that directory that canbe considered as complementary metadata.
• an.egg-info directory installed insite-packages, that contains the samefilesEGG-INFO has in the.egg format.

The first format is automatically used when you install a distribution thatuses thesetuptools.setup function in its setup.py file, instead ofthedistutils.core.setup one.

setuptools also add a reference to the distribution into aneasy-install.pth file.

Last, thesetuptools project provides an executable script calledeasy_install[2] that installs all distributions, includingdistutils-based ones in self-contained.egg directories.

If you want to have standalone.egg-info directories for your distributions,e.g. the secondsetuptools format, you have to force it when you workwith a setuptools-based distribution or with theeasy_install script.You can force it by using the--single-version-externally-managed optionor the--root option. This will make thesetuptools project installthe project like distutils does.

This option is used by :

• thepip[3] installer
• the Fedora packagers[5].
• the Debian packagers[6].

Uninstall information

Distutils doesn’t provide anuninstall command. If you want to uninstalla distribution, you have to be a power user and remove the various elementsthat were installed, and then look over the.pth file to clean them ifnecessary.

And the process differs depending on the tools you have used to install thedistribution and if the distribution’ssetup.py uses Distutils orSetuptools.

Under some circumstances, you might not be able to know for sure that youhave removed everything, or that you didn’t break another distribution byremoving a file that is shared among several distributions.

But there’s a common behavior: when you install a distribution, files arecopied in your system. And it’s possible to keep track of these files forlater removal.

Moreover, the Pip project has gained anuninstall feature lately. Itrecords all installed files, using therecord option of theinstallcommand.

What this PEP proposes

To address those issues, this PEP proposes a few changes:

• A new.dist-info structure using a directory, inspired on one format oftheEggFormats standard fromsetuptools.
• New APIs inpkgutil to be able to query the information of installeddistributions.
• An uninstall function and an uninstall script in Distutils.

RECORD

ARECORD file is added inside the.dist-info directory at installationtime when installing a source distribution using theinstall command.Notice that when installing a binary distribution created withbdist commandor abdist-based command, theRECORD file will be installed as well sincethese commands use theinstall command to create binary distributions.

TheRECORD file holds the list of installed files. These correspondto the files listed by therecord option of theinstall command, and willbe generated by default. This allows the implementation of an uninstallationfeature, as explained later in this PEP. Theinstall command also providesan option to prevent theRECORD file from being written and this optionshould be used when creating system packages.

Third-party installation tools also should not overwrite or delete filesthat are not in a RECORD file without prompting or warning.

This RECORD file is inspired fromPEP 262 FILES.

TheRECORD file is a CSV file, composed of records, one line perinstalled file. Thecsv module is used to read the file, withthese options:

• field delimiter :,
• quoting char :".
• line terminator :os.linesep (so\r\n or\n)

When a distribution is installed, files can be installed under:

• thebase location: path defined by the--install-lib option,which defaults to the site-packages directory.
• theinstallation prefix: path defined by the--prefix option, whichdefaults tosys.prefix.
• any other path on the system.

Each record is composed of three elements:

• the file’spath
  • a ‘/’-separated path, relative to thebase location, if the file isunder thebase location.
  • a ‘/’-separated path, relative to thebase location, if the fileis under theinstallation prefix AND if thebase location is asubpath of theinstallation prefix.
  • an absolute path, using the local platform separator
• a hash of the file’s contents.Notice thatpyc andpyo generated files don’t have any hash becausethey are automatically produced frompy files. So checking the hashof the correspondingpy file is enough to decide if the file andits associatedpyc orpyo files have changed.

  The hash is either the empty string or the hash algorithm as named inhashlib.algorithms_guaranteed, followed by the equals character=, followed by the urlsafe-base64-nopad encoding of the digest(base64.urlsafe_b64encode(digest) with trailing= removed).

• the file’s size in bytes

Thecsv module is used to generate this file, so the field separator is“,”. Any “,” character found within a field is escaped automatically bycsv.

When the file is read, theU option is used so the universal newlinesupport (seePEP 278) is activated, avoiding any troublereading a file produced on a platform that uses a different new lineterminator.

Here’s an example of a RECORD file (extract):

lib/python2.6/site-packages/docutils/__init__.py,md5=nWt-Dge1eug4iAgqLS_uWg,9544lib/python2.6/site-packages/docutils/__init__.pyc,,lib/python2.6/site-packages/docutils/core.py,md5=X90C_JLIcC78PL74iuhPnA,66188lib/python2.6/site-packages/docutils/core.pyc,,lib/python2.6/site-packages/roman.py,md5=7YhfNczihNjOY0FXlupwBg,234lib/python2.6/site-packages/roman.pyc,,/usr/local/bin/rst2html.py,md5=g22D3amDLJP-FhBzCi7EvA,234/usr/local/bin/rst2html.pyc,,python2.6/site-packages/docutils-0.5.dist-info/METADATA,md5=ovJyUNzXdArGfmVyb0onyA,195lib/python2.6/site-packages/docutils-0.5.dist-info/RECORD,,

Notice that theRECORD file can’t contain a hash of itself and is just mentioned here

A project that installs aconfig.ini file in/etc/myapp will be added like this:

/etc/myapp/config.ini,md5=gLfd6IANquzGLhOkW4Mfgg,9544

For a windows platform, the drive letter is added for the absolute paths,so a file that is copied in c:MyAppwill be:

c:\etc\myapp\config.ini,md5=gLfd6IANquzGLhOkW4Mfgg,9544

INSTALLER

Theinstall command has a new option calledinstaller. This optionis the name of the tool used to invoke the installation. It’s a normalizedlower-case string matching[a-z0-9_\-\.].

$ python setup.py install –installer=pkg-system

It defaults todistutils if not provided.

When a distribution is installed, the INSTALLER file is generated in the.dist-info directory with this value, to keep track ofwho installed thedistribution. The file is a single-line text file.

REQUESTED

Some install tools automatically detect unfulfilled dependencies andinstall them. In these cases, it is useful to track whichdistributions were installed purely as a dependency, so if theirdependent distribution is later uninstalled, the user can be alertedof the orphaned dependency.

If a distribution is installed by direct user request (the usualcase), a file REQUESTED is added to the .dist-info directory of theinstalled distribution. The REQUESTED file may be empty, or maycontain a marker comment line beginning with the “#” character.

If an install tool installs a distribution automatically, as adependency of another distribution, the REQUESTED file should not becreated.

Theinstall command of distutils by default creates the REQUESTEDfile. It accepts--requested and--no-requested options to explicitlyspecify whether the file is created.

If a distribution that was already installed on the system as a dependencyis later installed by name, the distutilsinstall command willcreate the REQUESTED file in the .dist-info directory of the existinginstallation.

New functions and classes in pkgutil

To use the.dist-info directory content, we need to add in the standardlibrary a set of APIs. The best place to put these APIs ispkgutil.

>>>frompkgutilimportget_distribution,get_file_users,distinfo_dirname>>>dist=get_distribution('docutils')>>>dist.name'docutils'>>>dist.metadata.version'0.5'>>>distinfo_dirname('docutils','0.5')'docutils-0.5.dist-info'>>>distinfo_dirname('python-ldap','2.5')'python_ldap-2.5.dist-info'>>>distinfo_dirname('python-ldap','2.5 a---5')'python_ldap-2.5.a_5.dist-info'>>>forpath,hash,sizeindist.get_installed_files()::...print'%s%s%d'%(path,hash,size)...python2.6/site-packages/docutils/__init__.py,b690274f621402dda63bf11ba5373bf2,9544python2.6/site-packages/docutils/core.py,9c4b84aff68aa55f2e9bf70481b94333,66188python2.6/site-packages/roman.py,a4b84aff68aa55f2e9bf70481b943D3,234/usr/local/bin/rst2html.py,a4b84aff68aa55f2e9bf70481b943D3,234python2.6/site-packages/docutils-0.5.dist-info/METADATA,6fe57de576d749536082d8e205b77748,195python2.6/site-packages/docutils-0.5.dist-info/RECORD>>>dist.uses('docutils/core.py')True>>>dist.uses('/usr/local/bin/rst2html.py')True>>>dist.get_distinfo_file('METADATA')<open file at ...>>>>dist.requestedTrue

New functions in Distutils

Distutils already provides a very basic way to install a distribution, whichis running theinstall command over thesetup.py script of thedistribution.

Distutils2 will provide a very basicuninstall function, thatis added indistutils2.util and takes the name of the distribution touninstall as its argument.uninstall uses the APIs described earlier andremove all unique files, as long as their hash didn’t change. Then it removesempty directories left behind.

uninstall returns a list of uninstalled files:

>>>fromdistutils2.utilimportuninstall>>>uninstall('docutils')['/opt/local/lib/python2.6/site-packages/docutils/core.py', ... '/opt/local/lib/python2.6/site-packages/docutils/__init__.py']

If the distribution is not found, aDistutilsUninstallError is raised.

>>>def_remove_and_log(path):...logging.info('Removing%s'%path)...returnTrue...>>>uninstall('docutils',_remove_and_log)>>>def_dry_run(path):...logging.info('Removing%s (dry run)'%path)...returnFalse...>>>uninstall('docutils',_dry_run)

>>>uninstall('docutils')Traceback (most recent call last):...DistutilsUninstallError:docutils was installed by 'cool-pkg-manager'>>>uninstall('docutils',installer='cool-pkg-manager')

$ python -m distutils2.uninstall projectname