Distutils based source distributions¶

If you download a module source distribution, you can tell pretty quickly if itwas packaged and distributed in the standard way, i.e. using the Distutils.First, the distribution’s name and version number will be featured prominentlyin the name of the downloaded archive, e.g.foo-1.0.tar.gz orwidget-0.9.7.zip. Next, the archive will unpack into a similarly nameddirectory:foo-1.0 orwidget-0.9.7. Additionally, thedistribution will contain a setup scriptsetup.py, and a file namedREADME.txt or possibly justREADME, which should explain thatbuilding and installing the module distribution is a simple matter of runningone command from a terminal:

python setup.py install

For Windows, this command should be run from a command prompt window(Start ‣ Accessories):

setup.py install

If all these things are true, then you already know how to build and install themodules you’ve just downloaded: Run the command above. Unless you need toinstall things in a non-standard way or customize the build process, you don’treally need this manual. Or rather, the above command is everything you need toget out of this manual.

Platform variations¶

You should always run the setup command from the distribution root directory,i.e. the top-level subdirectory that the module source distribution unpacksinto. For example, if you’ve just downloaded a module source distributionfoo-1.0.tar.gz onto a Unix system, the normal thing to do is:

gunzip -c foo-1.0.tar.gz | tar xf -    # unpacks into directory foo-1.0cd foo-1.0python setup.py install

On Windows, you’d probably downloadfoo-1.0.zip. If you downloaded thearchive file toC:\Temp, then it would unpack intoC:\Temp\foo-1.0; you can use either an archive manipulator with agraphical user interface (such as WinZip) or a command-line tool (such asunzip orpkunzip) to unpack the archive. Then, open acommand prompt window and run:

cd c:\Temp\foo-1.0python setup.py install

Splitting the job up¶

Runningsetup.pyinstall builds and installs all modules in one run. If youprefer to work incrementally—especially useful if you want to customize thebuild process, or if things are going wrong—you can use the setup script to doone thing at a time. This is particularly helpful when the build and installwill be done by different users—for example, you might want to build a moduledistribution and hand it off to a system administrator for installation (or doit yourself, with super-user privileges).

For example, you can build everything in one step, and then install everythingin a second step, by invoking the setup script twice:

python setup.py buildpython setup.py install

If you do this, you will notice that running theinstall commandfirst runs thebuild command, which—in this case—quickly noticesthat it has nothing to do, since everything in thebuild directory isup-to-date.

You may not need this ability to break things down often if all you do isinstall modules downloaded off the ‘net, but it’s very handy for more advancedtasks. If you get into distributing your own Python modules and extensions,you’ll run lots of individual Distutils commands on their own.

How building works¶

As implied above, thebuild command is responsible for putting thefiles to install into abuild directory. By default, this isbuildunder the distribution root; if you’re excessively concerned with speed, or wantto keep the source tree pristine, you can change the build directory with the--build-base option. For example:

python setup.py build --build-base=/path/to/pybuild/foo-1.0

(Or you could do this permanently with a directive in your system or personalDistutils configuration file; see sectionDistutils Configuration Files.) Normally, thisisn’t necessary.

The default layout for the build tree is as follows:

--- build/ --- lib/or--- build/ --- lib.<plat>/               temp.<plat>/

where<plat> expands to a brief description of the current OS/hardwareplatform and Python version. The first form, with just alib directory,is used for “pure module distributions”—that is, module distributions thatinclude only pure Python modules. If a module distribution contains anyextensions (modules written in C/C++), then the second form, with two<plat>directories, is used. In that case, thetemp.plat directory holdstemporary files generated by the compile/link process that don’t actually getinstalled. In either case, thelib (orlib.plat) directorycontains all Python modules (pure Python and extensions) that will be installed.

In the future, more directories will be added to handle Python scripts,documentation, binary executables, and whatever else is needed to handle the jobof installing Python modules and applications.

How installation works¶

After thebuild command runs (whether you run it explicitly, or theinstall command does it for you), the work of theinstallcommand is relatively simple: all it has to do is copy everything underbuild/lib (orbuild/lib.plat) to your chosen installationdirectory.

If you don’t choose an installation directory—i.e., if you just runsetup.pyinstall—then theinstall command installs to the standardlocation for third-party Python modules. This location varies by platform andby how you built/installed Python itself. On Unix (and macOS, which is alsoUnix-based), it also depends on whether the module distribution being installedis pure Python or contains extensions (“non-pure”):

Notes:

1. Most Linux distributions include Python as a standard part of the system, soprefix andexec-prefix are usually both/usr onLinux. If you build Python yourself on Linux (or any Unix-like system), thedefaultprefix andexec-prefix are/usr/local.

2. The default installation directory on Windows wasC:\ProgramFiles\Python under Python 1.6a1, 1.5.2, and earlier.

prefix andexec-prefix stand for the directories that Pythonis installed to, and where it finds its libraries at run-time. They are alwaysthe same under Windows, and very often the same under Unix and macOS. Youcan find out what your Python installation uses forprefix andexec-prefix by running Python in interactive mode and typing a fewsimple commands. Under Unix, just typepython at the shell prompt. UnderWindows, chooseStart ‣ Programs ‣ Python X.Y ‣Python (command line). Once the interpreter is started, you type Python codeat the prompt. For example, on my Linux system, I type the three Pythonstatements shown below, and get the output as shown, to find out myprefix andexec-prefix:

Python 2.4 (#26, Aug  7 2004, 17:19:02)Type "help", "copyright", "credits" or "license" for more information.>>>importsys>>>sys.prefix'/usr'>>>sys.exec_prefix'/usr'

A few other placeholders are used in this document:X.Y stands for theversion of Python, for example3.2;abiflags will be replaced bythe value ofsys.abiflags or the empty string for platforms which don’tdefine ABI flags;distname will be replaced by the name of the moduledistribution being installed. Dots and capitalization are important in thepaths; for example, a value that usespython3.2 on UNIX will typically usePython32 on Windows.

If you don’t want to install modules to the standard location, or if you don’thave permission to write there, then you need to read about alternateinstallations in sectionAlternate Installation. If you want to customize yourinstallation directories more heavily, see sectionCustom Installation oncustom installations.

Alternate installation: the user scheme¶

This scheme is designed to be the most convenient solution for users that don’thave write permission to the global site-packages directory or don’t want toinstall into it. It is enabled with a simple option:

python setup.py install --user

Files will be installed into subdirectories ofsite.USER_BASE (writtenasuserbase hereafter). This scheme installs pure Python modules andextension modules in the same location (also known assite.USER_SITE).Here are the values for UNIX, including macOS:

And here are the values used on Windows:

The advantage of using this scheme compared to the other ones described below isthat the user site-packages directory is under normal conditions always includedinsys.path (seesite for more information), which means thatthere is no additional step to perform after running thesetup.py scriptto finalize the installation.

Thebuild_ext command also has a--user option to adduserbase/include to the compiler search path for header files anduserbase/lib to the compiler search path for libraries as well as tothe runtime search path for shared C libraries (rpath).

Alternate installation: the home scheme¶

The idea behind the “home scheme” is that you build and maintain a personalstash of Python modules. This scheme’s name is derived from the idea of a“home” directory on Unix, since it’s not unusual for a Unix user to make theirhome directory have a layout similar to/usr/ or/usr/local/.This scheme can be used by anyone, regardless of the operating system theyare installing for.

Installing a new module distribution is as simple as

python setup.py install --home=<dir>

where you can supply any directory you like for the--home option. OnUnix, lazy typists can just type a tilde (~); theinstall commandwill expand this to your home directory:

python setup.py install --home=~

To make Python find the distributions installed with this scheme, you may havetomodify Python’s search path or editsitecustomize (seesite) to callsite.addsitedir() or editsys.path.

The--home option defines the installation base directory. Files areinstalled to the following directories under the installation base as follows:

(Mentally replace slashes with backslashes if you’re on Windows.)

Alternate installation: Unix (the prefix scheme)¶

The “prefix scheme” is useful when you wish to use one Python installation toperform the build/install (i.e., to run the setup script), but install modulesinto the third-party module directory of a different Python installation (orsomething that looks like a different Python installation). If this sounds atrifle unusual, it is—that’s why the user and home schemes come before. However,there are at least two known cases where the prefix scheme will be useful.

First, consider that many Linux distributions put Python in/usr, ratherthan the more traditional/usr/local. This is entirely appropriate,since in those cases Python is part of “the system” rather than a local add-on.However, if you are installing Python modules from source, you probably wantthem to go in/usr/local/lib/python2.X rather than/usr/lib/python2.X. This can be done with

/usr/bin/python setup.py install --prefix=/usr/local

Another possibility is a network filesystem where the name used to write to aremote directory is different from the name used to read it: for example, thePython interpreter accessed as/usr/local/bin/python might search formodules in/usr/local/lib/python2.X, but those modules would have tobe installed to, say,/mnt/@server/export/lib/python2.X. This couldbe done with

/usr/local/bin/python setup.py install --prefix=/mnt/@server/export

In either case, the--prefix option defines the installation base, andthe--exec-prefix option defines the platform-specific installationbase, which is used for platform-specific files. (Currently, this just meansnon-pure module distributions, but could be expanded to C libraries, binaryexecutables, etc.) If--exec-prefix is not supplied, it defaults to--prefix. Files are installed as follows:

There is no requirement that--prefix or--exec-prefixactually point to an alternate Python installation; if the directories listedabove do not already exist, they are created at installation time.

Incidentally, the real reason the prefix scheme is important is simply that astandard Unix installation uses the prefix scheme, but with--prefixand--exec-prefix supplied by Python itself assys.prefix andsys.exec_prefix. Thus, you might think you’ll never use the prefix scheme,but every time you runpythonsetup.pyinstall without any other options,you’re using it.

Note that installing extensions to an alternate Python installation has noeffect on how those extensions are built: in particular, the Python header files(Python.h and friends) installed with the Python interpreter used to runthe setup script will be used in compiling extensions. It is yourresponsibility to ensure that the interpreter used to run extensions installedin this way is compatible with the interpreter used to build them. The best wayto do this is to ensure that the two interpreters are the same version of Python(possibly different builds, or possibly copies of the same build). (Of course,if your--prefix and--exec-prefix don’t even point to analternate Python installation, this is immaterial.)

Alternate installation: Windows (the prefix scheme)¶

Windows has no concept of a user’s home directory, and since the standard Pythoninstallation under Windows is simpler than under Unix, the--prefixoption has traditionally been used to install additional packages in separatelocations on Windows.

python setup.py install --prefix="\Temp\Python"

to install modules to the\Temp\Python directory on the current drive.

The installation base is defined by the--prefix option; the--exec-prefix option is not supported under Windows, which means thatpure Python modules and extension modules are installed into the same location.Files are installed as follows:

Modifying Python’s Search Path¶

When the Python interpreter executes animport statement, it searchesfor both Python code and extension modules along a search path. A default valuefor the path is configured into the Python binary when the interpreter is built.You can determine the path by importing thesys module and printing thevalue ofsys.path.

$ pythonPython 2.2 (#11, Oct  3 2002, 13:31:27)[GCC 2.96 20000731 (Red Hat Linux 7.3 2.96-112)] on linux2Type "help", "copyright", "credits" or "license" for more information.>>> import sys>>> sys.path['', '/usr/local/lib/python2.3', '/usr/local/lib/python2.3/plat-linux2', '/usr/local/lib/python2.3/lib-tk', '/usr/local/lib/python2.3/lib-dynload', '/usr/local/lib/python2.3/site-packages']>>>

The null string insys.path represents the current working directory.

The expected convention for locally installed packages is to put them in the.../site-packages/ directory, but you may want to install Pythonmodules into some arbitrary directory. For example, your site may have aconvention of keeping all software related to the web server under/www.Add-on Python modules might then belong in/www/python, and in order toimport them, this directory must be added tosys.path. There are severaldifferent ways to add the directory.

The most convenient way is to add a path configuration file to a directorythat’s already on Python’s path, usually to the.../site-packages/directory. Path configuration files have an extension of.pth, and eachline must contain a single path that will be appended tosys.path. (Becausethe new paths are appended tosys.path, modules in the added directorieswill not override standard modules. This means you can’t use this mechanism forinstalling fixed versions of standard modules.)

Paths can be absolute or relative, in which case they’re relative to thedirectory containing the.pth file. See the documentation ofthesite module for more information.

A slightly less convenient way is to edit thesite.py file in Python’sstandard library, and modifysys.path.site.py is automaticallyimported when the Python interpreter is executed, unless the-S switchis supplied to suppress this behaviour. So you could simply editsite.py and add two lines to it:

importsyssys.path.append('/www/python/')

However, if you reinstall the same minor version of Python (perhaps whenupgrading from 2.2 to 2.2.2, for example)site.py will be overwritten bythe stock version. You’d have to remember that it was modified and save a copybefore doing the installation.

There are two environment variables that can modifysys.path.PYTHONHOME sets an alternate value for the prefix of the Pythoninstallation. For example, ifPYTHONHOME is set to/www/python,the search path will be set to['','/www/python/lib/pythonX.Y/','/www/python/lib/pythonX.Y/plat-linux2',...].

ThePYTHONPATH variable can be set to a list of paths that will beadded to the beginning ofsys.path. For example, ifPYTHONPATH isset to/www/python:/opt/py, the search path will begin with['/www/python','/opt/py']. (Note that directories must exist in order tobe added tosys.path; thesite module removes paths that don’texist.)

Finally,sys.path is just a regular Python list, so any Python applicationcan modify it by adding or removing entries.

Location and names of config files¶

The names and locations of the configuration files vary slightly acrossplatforms. On Unix and macOS, the three configuration files (in the orderthey are processed) are:

And on Windows, the configuration files are:

On all platforms, the “personal” file can be temporarily disabled bypassing the--no-user-cfg option.

Notes:

1. Strictly speaking, the system-wide configuration file lives in the directorywhere the Distutils are installed; under Python 1.6 and later on Unix, this isas shown. For Python 1.5.2, the Distutils will normally be installed toprefix/lib/python1.5/site-packages/distutils, so the systemconfiguration file should be put there under Python 1.5.2.

2. On Unix, if theHOME environment variable is not defined, the user’shome directory will be determined with thegetpwuid() function from thestandardpwd module. This is done by theos.path.expanduser()function used by Distutils.

3. I.e., in the current directory (usually the location of the setup script).

4. (See also note (1).) Under Python 1.6 and later, Python’s default “installationprefix” isC:\Python, so the system configuration file is normallyC:\Python\Lib\distutils\distutils.cfg. Under Python 1.5.2, thedefault prefix wasC:\ProgramFiles\Python, and the Distutils were notpart of the standard library—so the system configuration file would beC:\ProgramFiles\Python\distutils\distutils.cfg in a standard Python1.5.2 installation under Windows.

5. On Windows, if theHOME environment variable is not defined,USERPROFILE thenHOMEDRIVE andHOMEPATH willbe tried. This is done by theos.path.expanduser() function usedby Distutils.

Syntax of config files¶

The Distutils configuration files all have the same syntax. The config filesare grouped into sections. There is one section for each Distutils command,plus aglobal section for global options that affect every command. Eachsection consists of one option per line, specified asoption=value.

For example, the following is a complete config file that just forces allcommands to run quietly by default:

[global]verbose=0

If this is installed as the system config file, it will affect all processing ofany Python module distribution by any user on the current system. If it isinstalled as your personal config file (on systems that support them), it willaffect only module distributions processed by you. And if it is used as thesetup.cfg for a particular module distribution, it affects only thatdistribution.

You could override the default “build base” directory and make thebuild* commands always forcibly rebuild all files with thefollowing:

[build]build-base=blibforce=1

which corresponds to the command-line arguments

python setup.py build --build-base=blib --force

except that including thebuild command on the command-line meansthat command will be run. Including a particular command in config files has nosuch implication; it only means that if the command is run, the options in theconfig file will apply. (Or if other commands that derive values from it arerun, they will use the values in the config file.)

You can find out the complete list of options for any command using the--help option, e.g.:

python setup.py build --help

and you can find out the complete list of global options by using--help without a command:

python setup.py --help

See also the “Reference” section of the “Distributing Python Modules” manual.

Tweaking compiler/linker flags¶

Compiling a Python extension written in C or C++ will sometimes requirespecifying custom flags for the compiler and linker in order to use a particularlibrary or produce a special kind of object code. This is especially true if theextension hasn’t been tested on your platform, or if you’re trying tocross-compile Python.

In the most general case, the extension author might have foreseen thatcompiling the extensions would be complicated, and provided aSetup filefor you to edit. This will likely only be done if the module distributioncontains many separate extension modules, or if they often require elaboratesets of compiler flags in order to work.

ASetup file, if present, is parsed in order to get a list of extensionsto build. Each line in aSetup describes a single module. Lines havethe following structure:

module ... [sourcefile ...] [cpparg ...] [library ...]

Let’s examine each of the fields in turn.

• module is the name of the extension module to be built, and should be avalid Python identifier. You can’t just change this in order to rename a module(edits to the source code would also be needed), so this should be left alone.

• sourcefile is anything that’s likely to be a source code file, at leastjudging by the filename. Filenames ending in.c are assumed to bewritten in C, filenames ending in.C,.cc, and.c++ areassumed to be C++, and filenames ending in.m or.mm are assumedto be in Objective C.

• cpparg is an argument for the C preprocessor, and is anything starting with-I,-D,-U or-C.

• library is anything ending in.a or beginning with-l or-L.

If a particular platform requires a special library on your platform, you canadd it by editing theSetup file and runningpythonsetup.pybuild.For example, if the module defined by the line

foo foomodule.c

must be linked with the math librarylibm.a on your platform, simply add-lm to the line:

foo foomodule.c -lm

Arbitrary switches intended for the compiler or the linker can be supplied withthe-Xcompilerarg and-Xlinkerarg options:

foo foomodule.c -Xcompiler -o32 -Xlinker -shared -lm

The next option after-Xcompiler and-Xlinker will beappended to the proper command line, so in the above example the compiler willbe passed the-o32 option, and the linker will be passed-shared. If a compiler option requires an argument, you’ll have tosupply multiple-Xcompiler options; for example, to pass-xc++theSetup file would have to contain-Xcompiler-x-Xcompilerc++.

Compiler flags can also be supplied through setting theCFLAGSenvironment variable. If set, the contents ofCFLAGS will be added tothe compiler flags specified in theSetup file.

Using non-Microsoft compilers on Windows¶

coff2omf python25.lib python25_bcpp.lib

python setup.py build --compiler=bcpp

python setup.py build --compiler=cygwin

python setup.py build --compiler=mingw32

pexports python25.dll >python25.def

/cygwin/bin/dlltool --dllname python25.dll --def python25.def --output-lib libpython25.a