Recipes

A part is created by a recipe. Recipes are always installed as Pythoneggs. They can be downloaded from a package server, such as thePython Package Index, or they can be developed as part of a projectusing a “develop” egg.

A develop egg is a special kind of egg that gets installed as an “egglink” that contains the name of a source directory. Develop eggsdon’t have to be packaged for distribution to be used and can bemodified in place, which is especially useful while they are beingdeveloped.

Let’s create a recipe as part of the sample project. We’ll create arecipe for creating directories. First, we’ll create a recipes sourcedirectory for our local recipes:

>>> mkdir(sample_buildout, 'recipes')

and then we’ll create a source file for our mkdir recipe:

>>> write(sample_buildout, 'recipes', 'mkdir.py',... """... import logging, os, zc.buildout...... class Mkdir:......     def __init__(self, buildout, name, options):...         self.name, self.options = name, options...         options['path'] = os.path.join(...                               buildout['buildout']['directory'],...                               options['path'],...                               )...         if not os.path.isdir(os.path.dirname(options['path'])):...             logging.getLogger(self.name).error(...                 'Cannot create %s. %s is not a directory.',...                 options['path'], os.path.dirname(options['path']))...             raise zc.buildout.UserError('Invalid Path').........     def install(self):...         path = self.options['path']...         logging.getLogger(self.name).info(...             'Creating directory %s', os.path.basename(path))...         os.mkdir(path)...         return path......     def update(self):...         pass... """)

Currently, recipes must define 3 methods[1]:

• a constructor,

• an install method, and

• an update method.

The constructor is responsible for updating a parts options to reflectdata read from other sections. The buildout system keeps track ofwhether a part specification has changed. A part specification haschanged if it’s options, after adjusting for data read from othersections, has changed, or if the recipe has changed. Only the optionsfor the part are considered. If data are read from other sections,then that information has to be reflected in the parts options. Inthe Mkdir example, the given path is interpreted relative to thebuildout directory, and data from the buildout directory is read. Thepath option is updated to reflect this. If the directory option waschanged in the buildout sections, we would know to update partscreated using the mkdir recipe using relative path names.

When buildout is run, it saves configuration data for installed partsin a file named “.installed.cfg”. In subsequent runs, it comparespart-configuration data stored in the .installed.cfg file and thepart-configuration data loaded from the configuration files asmodified by recipe constructors to decide if the configuration of apart has changed. If the configuration has changed, or if the recipehas changed, then the part is uninstalled and reinstalled. Thebuildout only looks at the part’s options, so any data used toconfigure the part needs to be reflected in the part’s options. It isthe job of a recipe constructor to make sure that the options includeall relevant data.

Of course, parts are also uninstalled if they are no-longer used.

The recipe defines a constructor that takes a buildout object, a partname, and an options dictionary. It saves them in instance attributes.If the path is relative, we’ll interpret it as relative to thebuildout directory. The buildout object passed in is a mapping fromsection name to a mapping of options for that section. The buildoutdirectory is available as the directory option of the buildoutsection. We normalize the path and save it back into the optionsdirectory.

The install method is responsible for creating the part. In thiscase, we need the path of the directory to create. We’ll use a pathoption from our options dictionary. The install method logs what it’sdoing using the Python logging call. We return the path that weinstalled. If the part is uninstalled or reinstalled, then the pathreturned will be removed by the buildout machinery. A recipe installmethod is expected to return a string, or an iterable of stringscontaining paths to be removed if a part is uninstalled. For mostrecipes, this is all of the uninstall support needed. For more complexuninstallation scenarios useUninstall recipes.

The update method is responsible for updating an already installedpart. An empty method is often provided, as in this example, if partscan’t be updated. An update method can return None, a string, or aniterable of strings. If a string or iterable of strings is returned,then the saved list of paths to be uninstalled is updated with the newinformation by adding any new files returned by the update method.

We need to provide packaging information so that our recipe can beinstalled as a develop egg. The minimum information we need to specify[2] is a name. For recipes, we also need to define thenames of the recipe classes as entry points. Packaging information isprovided via a setup.py script:

>>> write(sample_buildout, 'recipes', 'setup.py',... """... from setuptools import setup...... setup(...     name = "recipes",...     entry_points = {'zc.buildout': ['mkdir = mkdir:Mkdir']},...     )... """)

Our setup script defines an entry point. Entry points providea way for an egg to define the services it provides. Here we’ve saidthat we define a zc.buildout entry point named mkdir. Recipeclasses must be exposed as entry points in the zc.buildout group. wegive entry points names within the group.

We also need a README.txt for our recipes to avoid an annoying warningfrom distutils, on which setuptools and zc.buildout are based:

>>> write(sample_buildout, 'recipes', 'README.txt', " ")

Now let’s update our buildout.cfg:

>>> write(sample_buildout, 'buildout.cfg',... """... [buildout]... develop = recipes... parts = data-dir...... [data-dir]... recipe = recipes:mkdir... path = mystuff... """)

Let’s go through the changes one by one:

develop = recipes

This tells the buildout to install a development egg for our recipes.Any number of paths can be listed. The paths can be relative orabsolute. If relative, they are treated as relative to the buildoutdirectory. They can be directory or file paths. If a file path isgiven, it should point to a Python setup script. If a directory pathis given, it should point to a directory containing a setup.py file.Development eggs are installed before building any parts, as they mayprovide locally-defined recipes needed by the parts.

parts = data-dir

Here we’ve named a part to be “built”. We can use any name we wantexcept that different part names must be unique and recipes will oftenuse the part name to decide what to do.

[data-dir]recipe = recipes:mkdirpath = mystuff

When we name a part, we also create a section of the samename that contains part data. In this section, we’ll definethe recipe to be used to install the part. In this case, we alsospecify the path to be created.

Let’s run the buildout. We do so by running the build script in thebuildout:

>>> import os>>> os.chdir(sample_buildout)>>> buildout = os.path.join(sample_buildout, 'bin', 'buildout')>>> print system(buildout),Develop: '/sample-buildout/recipes'Installing data-dir.data-dir: Creating directory mystuff

We see that the recipe created the directory, as expected:

>>> ls(sample_buildout)-  .installed.cfgd  bin-  buildout.cfgd  develop-eggsd  eggsd  mystuffd  partsd  recipes

In addition, .installed.cfg has been created containing informationabout the part we installed:

>>> cat(sample_buildout, '.installed.cfg')[buildout]installed_develop_eggs = /sample-buildout/develop-eggs/recipes.egg-linkparts = data-dir<BLANKLINE>[data-dir]__buildout_installed__ = /sample-buildout/mystuff__buildout_signature__ = recipes-c7vHV6ekIDUPy/7fjAaYjg==path = /sample-buildout/mystuffrecipe = recipes:mkdir

Note that the directory we installed is included in .installed.cfg.In addition, the path option includes the actual destinationdirectory.

If we change the name of the directory in the configuration file,we’ll see that the directory gets removed and recreated:

>>> write(sample_buildout, 'buildout.cfg',... """... [buildout]... develop = recipes... parts = data-dir...... [data-dir]... recipe = recipes:mkdir... path = mydata... """)

>>> print system(buildout),Develop: '/sample-buildout/recipes'Uninstalling data-dir.Installing data-dir.data-dir: Creating directory mydata

>>> ls(sample_buildout)-  .installed.cfgd  bin-  buildout.cfgd  develop-eggsd  eggsd  mydatad  partsd  recipes

If any of the files or directories created by a recipe are removed,the part will be reinstalled:

>>> rmdir(sample_buildout, 'mydata')>>> print system(buildout),Develop: '/sample-buildout/recipes'Uninstalling data-dir.Installing data-dir.data-dir: Creating directory mydata

Error reporting

If a user makes an error, an error needs to be printed and work needsto stop. This is accomplished by logging a detailed error message andthen raising a (or an instance of a subclass of a)zc.buildout.UserError exception. Raising an error other than aUserError still displays the error, but labels it as a bug in thebuildout software or recipe. In the sample above, of someone gives anon-existent directory to create the directory in:

>>> write(sample_buildout, 'buildout.cfg',... """... [buildout]... develop = recipes... parts = data-dir...... [data-dir]... recipe = recipes:mkdir... path = /xxx/mydata... """)

We’ll get a user error, not a traceback.

>>> print system(buildout),Develop: '/sample-buildout/recipes'data-dir: Cannot create /xxx/mydata. /xxx is not a directory.While:  Installing.  Getting section data-dir.  Initializing part data-dir.Error: Invalid Path

Recipe Error Handling

If an error occurs during installation, it is up to the recipe toclean up any system side effects, such as files created. Let’s updatethe mkdir recipe to support multiple paths:

>>> write(sample_buildout, 'recipes', 'mkdir.py',... """... import logging, os, zc.buildout...... class Mkdir:......     def __init__(self, buildout, name, options):...         self.name, self.options = name, options......         # Normalize paths and check that their parent...         # directories exist:...         paths = []...         for path in options['path'].split():...             path = os.path.join(buildout['buildout']['directory'], path)...             if not os.path.isdir(os.path.dirname(path)):...                 logging.getLogger(self.name).error(...                     'Cannot create %s. %s is not a directory.',...                     options['path'], os.path.dirname(options['path']))...                 raise zc.buildout.UserError('Invalid Path')...             paths.append(path)...         options['path'] = ' '.join(paths)......     def install(self):...         paths = self.options['path'].split()...         for path in paths:...             logging.getLogger(self.name).info(...                 'Creating directory %s', os.path.basename(path))...             os.mkdir(path)...         return paths......     def update(self):...         pass... """)

If there is an error creating a path, the install method will exit andleave previously created paths in place:

>>> write(sample_buildout, 'buildout.cfg',... """... [buildout]... develop = recipes... parts = data-dir...... [data-dir]... recipe = recipes:mkdir... path = foo bin... """)

>>> print system(buildout), # doctest: +ELLIPSISDevelop: '/sample-buildout/recipes'Uninstalling data-dir.Installing data-dir.data-dir: Creating directory foodata-dir: Creating directory binWhile:  Installing data-dir.<BLANKLINE>An internal error occurred due to a bug in either zc.buildout or in arecipe being used:Traceback (most recent call last):  ...OSError: [Errno 17] File exists: '/sample-buildout/bin'

We meant to create a directory bins, but typed bin. Now foo wasleft behind.

>>> os.path.exists('foo')True

If we fix the typo:

>>> write(sample_buildout, 'buildout.cfg',... """... [buildout]... develop = recipes... parts = data-dir...... [data-dir]... recipe = recipes:mkdir... path = foo bins... """)

>>> print system(buildout), # doctest: +ELLIPSISDevelop: '/sample-buildout/recipes'Installing data-dir.data-dir: Creating directory fooWhile:  Installing data-dir.<BLANKLINE>An internal error occurred due to a bug in either zc.buildout or in arecipe being used:Traceback (most recent call last):...OSError: [Errno 17] File exists: '/sample-buildout/foo'

Now they fail because foo exists, because it was left behind.

>>> remove('foo')

Let’s fix the recipe:

>>> write(sample_buildout, 'recipes', 'mkdir.py',... """... import logging, os, zc.buildout...... class Mkdir:......     def __init__(self, buildout, name, options):...         self.name, self.options = name, options......         # Normalize paths and check that their parent...         # directories exist:...         paths = []...         for path in options['path'].split():...             path = os.path.join(buildout['buildout']['directory'], path)...             if not os.path.isdir(os.path.dirname(path)):...                 logging.getLogger(self.name).error(...                     'Cannot create %s. %s is not a directory.',...                     options['path'], os.path.dirname(options['path']))...                 raise zc.buildout.UserError('Invalid Path')...             paths.append(path)...         options['path'] = ' '.join(paths)......     def install(self):...         paths = self.options['path'].split()...         created = []...         try:...             for path in paths:...                 logging.getLogger(self.name).info(...                     'Creating directory %s', os.path.basename(path))...                 os.mkdir(path)...                 created.append(path)...         except:...             for d in created:...                 os.rmdir(d)...             raise......         return paths......     def update(self):...         pass... """)

And put back the typo:

>>> write(sample_buildout, 'buildout.cfg',... """... [buildout]... develop = recipes... parts = data-dir...... [data-dir]... recipe = recipes:mkdir... path = foo bin... """)

When we rerun the buildout:

>>> print system(buildout), # doctest: +ELLIPSISDevelop: '/sample-buildout/recipes'Installing data-dir.data-dir: Creating directory foodata-dir: Creating directory binWhile:  Installing data-dir.<BLANKLINE>An internal error occurred due to a bug in either zc.buildout or in arecipe being used:Traceback (most recent call last):...OSError: [Errno 17] File exists: '/sample-buildout/bin'

we get the same error, but we don’t get the directory left behind:

>>> os.path.exists('foo')False

It’s critical that recipes clean up partial effects when errorsoccur. Because recipes most commonly create files and directories,buildout provides a helper API for removing created files when anerror occurs. Option objects have a created method that can be calledto record files as they are created. If the install or update methodreturns with an error, then any registered paths are removedautomatically. The method returns the files registered and can beused to return the files created. Let’s use this API to simplify therecipe:

>>> write(sample_buildout, 'recipes', 'mkdir.py',... """... import logging, os, zc.buildout...... class Mkdir:......     def __init__(self, buildout, name, options):...         self.name, self.options = name, options......         # Normalize paths and check that their parent...         # directories exist:...         paths = []...         for path in options['path'].split():...             path = os.path.join(buildout['buildout']['directory'], path)...             if not os.path.isdir(os.path.dirname(path)):...                 logging.getLogger(self.name).error(...                     'Cannot create %s. %s is not a directory.',...                     options['path'], os.path.dirname(options['path']))...                 raise zc.buildout.UserError('Invalid Path')...             paths.append(path)...         options['path'] = ' '.join(paths)......     def install(self):...         paths = self.options['path'].split()...         for path in paths:...             logging.getLogger(self.name).info(...                 'Creating directory %s', os.path.basename(path))...             os.mkdir(path)...             self.options.created(path)......         return self.options.created()......     def update(self):...         pass... """)

>>> remove(sample_buildout, 'recipes', 'mkdir.pyc')

We returned by calling created, taking advantage of the fact that itreturns the registered paths. We did this for illustrative purposes.It would be simpler just to return the paths as before.

If we rerun the buildout, again, we’ll get the error and nodirectories will be created:

>>> print system(buildout), # doctest: +ELLIPSISDevelop: '/sample-buildout/recipes'Installing data-dir.data-dir: Creating directory foodata-dir: Creating directory binWhile:  Installing data-dir.<BLANKLINE>An internal error occurred due to a bug in either zc.buildout or in arecipe being used:Traceback (most recent call last):...OSError: [Errno 17] File exists: '/sample-buildout/bin'

>>> os.path.exists('foo')False

Now, we’ll fix the typo again and we’ll get the directories we expect:

>>> write(sample_buildout, 'buildout.cfg',... """... [buildout]... develop = recipes... parts = data-dir...... [data-dir]... recipe = recipes:mkdir... path = foo bins... """)

>>> print system(buildout),Develop: '/sample-buildout/recipes'Installing data-dir.data-dir: Creating directory foodata-dir: Creating directory bins

>>> os.path.exists('foo')True>>> os.path.exists('bins')True

Configuration file syntax

As mentioned earlier, buildout configuration files use the formatdefined by the Python ConfigParser module with extensions. Theextensions are:

• option names are case sensitive

• option values can use a substitution syntax, described below, torefer to option values in specific sections.

• option values can be appended or removed using the - and +operators.

The ConfigParser syntax is very flexible. Section names can containany characters other than newlines and right square braces (“]”).Option names can contain any characters other than newlines, colons,and equal signs, can not start with a space, and don’t includetrailing spaces.

It is likely that, in the future, some characters will be givenspecial buildout-defined meanings. This is already true of thecharacters “:”, “$”, “%”, “(”, and “)”. For now, it is a good idea tokeep section and option names simple, sticking to alphanumericcharacters, hyphens, and periods.

Annotated sections

When used with theannotate command, buildout displays annotated sections.All sections are displayed, sorted alphabetically. For each section,all key-value pairs are displayed, sorted alphabetically, along withthe origin of the value (file name or COMPUTED_VALUE, DEFAULT_VALUE,COMMAND_LINE_VALUE).

>>> print system(buildout+ ' annotate'),... # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE<BLANKLINE>Annotated sections==================<BLANKLINE>[buildout]accept-buildout-test-releases= false    DEFAULT_VALUEallow-hosts= *    DEFAULT_VALUEallow-picked-versions= true    DEFAULT_VALUEallowed-eggs-from-site-packages= *    DEFAULT_VALUEbin-directory= bin    DEFAULT_VALUEdevelop= recipes    /sample-buildout/buildout.cfgdevelop-eggs-directory= develop-eggs    DEFAULT_VALUEdirectory= /sample-buildout    COMPUTED_VALUEeggs-directory= eggs    DEFAULT_VALUEexec-sitecustomize= true    DEFAULT_VALUEexecutable= ...    DEFAULT_VALUEfind-links=    DEFAULT_VALUEinclude-site-packages= true    DEFAULT_VALUEinstall-from-cache= false    DEFAULT_VALUEinstalled= .installed.cfg    DEFAULT_VALUElog-format=    DEFAULT_VALUElog-level= INFO    DEFAULT_VALUEnewest= true    DEFAULT_VALUEoffline= false    DEFAULT_VALUEparts= data-dir    /sample-buildout/buildout.cfgparts-directory= parts    DEFAULT_VALUEprefer-final= false    DEFAULT_VALUEpython= buildout    DEFAULT_VALUErelative-paths= false    DEFAULT_VALUEsocket-timeout=    DEFAULT_VALUEunzip= false    DEFAULT_VALUEuse-dependency-links= true    DEFAULT_VALUE<BLANKLINE>[data-dir]path= foo bins    /sample-buildout/buildout.cfgrecipe= recipes:mkdir    /sample-buildout/buildout.cfg<BLANKLINE>

Variable substitutions

Buildout configuration files support variable substitution.To illustrate this, we’ll create an debug recipe toallow us to see interactions with the buildout:

>>> write(sample_buildout, 'recipes', 'debug.py',... """... class Debug:......     def __init__(self, buildout, name, options):...         self.buildout = buildout...         self.name = name...         self.options = options......     def install(self):...         items = self.options.items()...         items.sort()...         for option, value in items:...             print option, value...         return ()......     update = install... """)

This recipe doesn’t actually create anything. The install methoddoesn’t return anything, because it didn’t create any files ordirectories.

We also have to update our setup script:

>>> write(sample_buildout, 'recipes', 'setup.py',... """... from setuptools import setup... entry_points = (... '''... [zc.buildout]... mkdir = mkdir:Mkdir... debug = debug:Debug... ''')... setup(name="recipes", entry_points=entry_points)... """)

We’ve rearranged the script a bit to make the entry points easier toedit. In particular, entry points are now defined as a configurationstring, rather than a dictionary.

Let’s update our configuration to provide variable substitutionexamples:

>>> write(sample_buildout, 'buildout.cfg',... """... [buildout]... develop = recipes... parts = data-dir debug... log-level = INFO...... [debug]... recipe = recipes:debug... File 1 = ${data-dir:path}/file... File 2 = ${debug:File 1}/log...... [data-dir]... recipe = recipes:mkdir... path = mydata... """)

We used a string-template substitution for File 1 and File 2. Thistype of substitution uses the string.Template syntax. Namessubstituted are qualified option names, consisting of a section nameand option name joined by a colon.

Now, if we run the buildout, we’ll see the options with the valuessubstituted.

>>> print system(buildout),Develop: '/sample-buildout/recipes'Uninstalling data-dir.Installing data-dir.data-dir: Creating directory mydataInstalling debug.File 1 /sample-buildout/mydata/fileFile 2 /sample-buildout/mydata/file/logrecipe recipes:debug

Note that the substitution of the data-dir path option reflects theupdate to the option performed by the mkdir recipe.

It might seem surprising that mydata was created again. This isbecause we changed our recipes package by adding the debug module.The buildout system didn’t know if this module could effect the mkdirrecipe, so it assumed it could and reinstalled mydata. If we rerunthe buildout:

>>> print system(buildout),Develop: '/sample-buildout/recipes'Updating data-dir.Updating debug.File 1 /sample-buildout/mydata/fileFile 2 /sample-buildout/mydata/file/logrecipe recipes:debug

We can see that mydata was not recreated.

Note that, in this case, we didn’t specify a log level, sowe didn’t get output about what the buildout was doing.

Section and option names in variable substitutions are only allowed tocontain alphanumeric characters, hyphens, periods and spaces. Thisrestriction might be relaxed in future releases.

We can ommit the section name in a variable substitution to refer tothe current section. We can also use the special option,_buildout_section_name_ to get the current section name.

>>> write(sample_buildout, 'buildout.cfg',... """... [buildout]... develop = recipes... parts = data-dir debug... log-level = INFO...... [debug]... recipe = recipes:debug... File 1 = ${data-dir:path}/file... File 2 = ${:File 1}/log... my_name = ${:_buildout_section_name_}...... [data-dir]... recipe = recipes:mkdir... path = mydata... """)

>>> print system(buildout),Develop: '/sample-buildout/recipes'Uninstalling debug.Updating data-dir.Installing debug.File 1 /sample-buildout/mydata/fileFile 2 /sample-buildout/mydata/file/logmy_name debugrecipe recipes:debug

Automatic part selection and ordering

When a section with a recipe is referred to, either through variablesubstitution or by an initializing recipe, the section is treated as apart and added to the part list before the referencing part. Forexample, we can leave data-dir out of the parts list:

>>> write(sample_buildout, 'buildout.cfg',... """... [buildout]... develop = recipes... parts = debug... log-level = INFO...... [debug]... recipe = recipes:debug... File 1 = ${data-dir:path}/file... File 2 = ${debug:File 1}/log...... [data-dir]... recipe = recipes:mkdir... path = mydata... """)

It will still be treated as a part:

>>> print system(buildout),Develop: '/sample-buildout/recipes'Uninstalling debug.Updating data-dir.Installing debug.File 1 /sample-buildout/mydata/fileFile 2 /sample-buildout/mydata/file/logrecipe recipes:debug

>>> cat('.installed.cfg') # doctest: +ELLIPSIS[buildout]installed_develop_eggs = /sample-buildout/develop-eggs/recipes.egg-linkparts = data-dir debug...

Note that the data-dir part is includedbefore the debug part,because the debug part refers to the data-dir part. Even if we listthe data-dir part after the debug part, it will be included before:

>>> write(sample_buildout, 'buildout.cfg',... """... [buildout]... develop = recipes... parts = debug data-dir... log-level = INFO...... [debug]... recipe = recipes:debug... File 1 = ${data-dir:path}/file... File 2 = ${debug:File 1}/log...... [data-dir]... recipe = recipes:mkdir... path = mydata... """)

It will still be treated as a part:

>>> print system(buildout),Develop: '/sample-buildout/recipes'Updating data-dir.Updating debug.File 1 /sample-buildout/mydata/fileFile 2 /sample-buildout/mydata/file/logrecipe recipes:debug

>>> cat('.installed.cfg') # doctest: +ELLIPSIS[buildout]installed_develop_eggs = /sample-buildout/develop-eggs/recipes.egg-linkparts = data-dir debug...

Extending sections (macros)

A section (other than the buildout section) can extend one or moreother sections using the<= option. Options from the referencedsections are copied to the refering sectionbefore variablesubstitution. This, together with the ability to refer to variablesof the current section allows sections to be used as macros.

>>> write(sample_buildout, 'buildout.cfg',... """... [buildout]... develop = recipes... parts = myfiles... log-level = INFO...... [debug]... recipe = recipes:debug...... [with_file1]... <= debug... file1 = ${:path}/file1... color = red...... [with_file2]... <= debug... file2 = ${:path}/file2... color = blue...... [myfiles]... <= with_file1...    with_file2... path = mydata... """)

>>> print system(buildout),Develop: '/sample-buildout/recipes'Uninstalling debug.Uninstalling data-dir.Installing myfiles.color bluefile1 mydata/file1file2 mydata/file2path mydatarecipe recipes:debug

In this example, the debug, with_file1 and with_file2 sections act asmacros. In particular, the variable substitutions are performedrelative to the myfiles section.

Adding and removing options

We can append and remove values to an option by using the + and -operators.

This is illustrated below; first we define a base configuration.

>>> write(sample_buildout, 'base.cfg',... """... [buildout]... parts = part1 part2 part3...... [part1]... recipe =... option = a1 a2...... [part2]... recipe =... option = b1 b2 b3 b4...... [part3]... recipe =... option = c1 c2...... """)

Extending this configuration, we can “adjust” the values set in thebase configuration file.

>>> write(sample_buildout, 'extension1.cfg',... """... [buildout]... extends = base.cfg...... # appending values... [part1]... option += a3 a4...... # removing values... [part2]... option -= b1 b2...... # alt. spelling... [part3]... option+=c3 c4 c5...... # normal assignment... [part4]... option = h1 h2...... """)

An additional extension.

>>> write(sample_buildout, 'extension2.cfg',... """... [buildout]... extends = extension1.cfg...... # appending values... [part1]... option += a5...... # removing values... [part2]... option -= b1 b2 b3...... """)

To verify that the options are adjusted correctly, we’ll set up anextension that prints out the options.

>>> mkdir(sample_buildout, 'demo')>>> write(sample_buildout, 'demo', 'demo.py',... """... def ext(buildout):...     print [part['option'] for name, part in buildout.items() \...           if name.startswith('part')]... """)

>>> write(sample_buildout, 'demo', 'setup.py',... """... from setuptools import setup...... setup(...     name="demo",...     entry_points={'zc.buildout.extension': ['ext = demo:ext']},...     )... """)

Set up a buildout configuration for this extension.

>>> write(sample_buildout, 'buildout.cfg',... """... [buildout]... develop = demo... parts =... """)

>>> os.chdir(sample_buildout)>>> print system(os.path.join(sample_buildout, 'bin', 'buildout')),Develop: '/sample-buildout/demo'Uninstalling myfiles.Getting distribution for 'recipes'.zip_safe flag not set; analyzing archive contents...Got recipes 0.0.0.warning: install_lib: 'build/lib' does not exist -- no Python modules to install

Verify option values.

>>> write(sample_buildout, 'buildout.cfg',... """... [buildout]... develop = demo... extensions = demo... extends = extension2.cfg... """)

>>> print system(os.path.join('bin', 'buildout')),['a1 a2/na3 a4/na5', 'b1 b2 b3 b4', 'c1 c2/nc3 c4 c5', 'h1 h2']Develop: '/sample-buildout/demo'

Annotated sections output shows which files are responsible for whichoperations.

>>> print system(os.path.join('bin', 'buildout') + ' annotate'),... # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE<BLANKLINE>Annotated sections==================...<BLANKLINE>[part1]option= a1 a2a3 a4a5    /sample-buildout/base.cfg+=  /sample-buildout/extension1.cfg+=  /sample-buildout/extension2.cfgrecipe=    /sample-buildout/base.cfg<BLANKLINE>[part2]option= b1 b2 b3 b4    /sample-buildout/base.cfg-=  /sample-buildout/extension1.cfg-=  /sample-buildout/extension2.cfgrecipe=    /sample-buildout/base.cfg<BLANKLINE>[part3]option= c1 c2c3 c4 c5    /sample-buildout/base.cfg+=  /sample-buildout/extension1.cfgrecipe=    /sample-buildout/base.cfg<BLANKLINE>[part4]option= h1 h2    /sample-buildout/extension1.cfg

Cleanup.

>>> os.remove(os.path.join(sample_buildout, 'base.cfg'))>>> os.remove(os.path.join(sample_buildout, 'extension1.cfg'))>>> os.remove(os.path.join(sample_buildout, 'extension2.cfg'))

Multiple configuration files

A configuration file can “extend” another configuration file.Options are read from the other configuration file if they aren’talready defined by your configuration file.

The configuration files your file extends can extendother configuration files. The same file may beused more than once although, of course, cycles aren’t allowed.

To see how this works, we use an example:

>>> write(sample_buildout, 'buildout.cfg',... """... [buildout]... extends = base.cfg...... [debug]... op = buildout... """)

>>> write(sample_buildout, 'base.cfg',... """... [buildout]... develop = recipes... parts = debug...... [debug]... recipe = recipes:debug... op = base... """)

>>> print system(buildout),Develop: '/sample-buildout/recipes'Installing debug.op buildoutrecipe recipes:debug

The example is pretty trivial, but the pattern it illustrates ispretty common. In a more practical example, the base buildout mightrepresent a product and the extending buildout might be acustomization.

Here is a more elaborate example.

>>> other = tmpdir('other')

>>> write(sample_buildout, 'buildout.cfg',... """... [buildout]... extends = b1.cfg b2.cfg %(b3)s...... [debug]... op = buildout... """ % dict(b3=os.path.join(other, 'b3.cfg')))

>>> write(sample_buildout, 'b1.cfg',... """... [buildout]... extends = base.cfg...... [debug]... op1 = b1 1... op2 = b1 2... """)

>>> write(sample_buildout, 'b2.cfg',... """... [buildout]... extends = base.cfg...... [debug]... op2 = b2 2... op3 = b2 3... """)

>>> write(other, 'b3.cfg',... """... [buildout]... extends = b3base.cfg...... [debug]... op4 = b3 4... """)

>>> write(other, 'b3base.cfg',... """... [debug]... op5 = b3base 5... """)

>>> write(sample_buildout, 'base.cfg',... """... [buildout]... develop = recipes... parts = debug...... [debug]... recipe = recipes:debug... name = base... """)

>>> print system(buildout),Develop: '/sample-buildout/recipes'Uninstalling debug.Installing debug.name baseop buildoutop1 b1 1op2 b2 2op3 b2 3op4 b3 4op5 b3base 5recipe recipes:debug

There are several things to note about this example:

• We can name multiple files in an extends option.

• We can reference files recursively.

• Relative file names in extended options are interpreted relative tothe directory containing the referencing configuration file.

Loading Configuration from URLs

Configuration files can be loaded from URLs. To see how this works,we’ll set up a web server with some configuration files.

>>> server_data = tmpdir('server_data')

>>> write(server_data, "r1.cfg",... """... [debug]... op1 = r1 1... op2 = r1 2... """)

>>> write(server_data, "r2.cfg",... """... [buildout]... extends = r1.cfg...... [debug]... op2 = r2 2... op3 = r2 3... """)

>>> server_url = start_server(server_data)

>>> write('client.cfg',... """... [buildout]... develop = recipes... parts = debug... extends = %(url)s/r2.cfg...... [debug]... recipe = recipes:debug... name = base... """ % dict(url=server_url))

>>> print system(buildout+ ' -c client.cfg'),Develop: '/sample-buildout/recipes'Uninstalling debug.Installing debug.name baseop1 r1 1op2 r2 2op3 r2 3recipe recipes:debug

Here we specified a URL for the file we extended. The file wedownloaded, itself referred to a file on the server using a relativeURL reference. Relative references are interpreted relative to thebase URL when they appear in configuration files loaded via URL.

We can also specify a URL as the configuration file to be used by abuildout.

>>> os.remove('client.cfg')>>> write(server_data, 'remote.cfg',... """... [buildout]... develop = recipes... parts = debug... extends = r2.cfg...... [debug]... recipe = recipes:debug... name = remote... """)

>>> print system(buildout + ' -c ' + server_url + '/remote.cfg'),While:  Initializing.Error: Missing option: buildout:directory

Normally, the buildout directory defaults to directorycontaining a configuration file. This won’t work for configurationfiles loaded from URLs. In this case, the buildout directory wouldnormally be defined on the command line:

>>> print system(buildout...              + ' -c ' + server_url + '/remote.cfg'...              + ' buildout:directory=' + sample_buildout...              ),Develop: '/sample-buildout/recipes'Uninstalling debug.Installing debug.name remoteop1 r1 1op2 r2 2op3 r2 3recipe recipes:debug

User defaults

If the file $HOME/.buildout/default.cfg, exists, it is read beforereading the configuration file. ($HOME is the value of the HOMEenvironment variable. The ‘/’ is replaced by the operating system filedelimiter.)

>>> old_home = os.environ['HOME']>>> home = tmpdir('home')>>> mkdir(home, '.buildout')>>> write(home, '.buildout', 'default.cfg',... """... [debug]... op1 = 1... op7 = 7... """)

>>> os.environ['HOME'] = home>>> print system(buildout),Develop: '/sample-buildout/recipes'Uninstalling debug.Installing debug.name baseop buildoutop1 b1 1op2 b2 2op3 b2 3op4 b3 4op5 b3base 5op7 7recipe recipes:debug

A buildout command-line argument, -U, can be used to suppress readinguser defaults:

>>> print system(buildout + ' -U'),Develop: '/sample-buildout/recipes'Uninstalling debug.Installing debug.name baseop buildoutop1 b1 1op2 b2 2op3 b2 3op4 b3 4op5 b3base 5recipe recipes:debug

>>> os.environ['HOME'] = old_home

Log level

We can control the level of logging by specifying a log level in outconfiguration file. For example, so suppress info messages, we canset the logging level to WARNING

>>> write(sample_buildout, 'buildout.cfg',... """... [buildout]... log-level = WARNING... extends = b1.cfg b2.cfg... """)

>>> print system(buildout),name baseop1 b1 1op2 b2 2op3 b2 3recipe recipes:debug

Uninstall recipes

As we’ve seen, when parts are installed, buildout keeps track of filesand directories that they create. When the parts are uninstalled thesefiles and directories are deleted.

Sometimes more clean up is needed. For example, a recipe might add asystem service by calling chkconfig –add during installation. Laterduring uninstallation, chkconfig –del will need to be called toremove the system service.

In order to deal with these uninstallation issues, you can registeruninstall recipes. Uninstall recipes are registered using the‘zc.buildout.uninstall’ entry point. Parts specify uninstall recipesusing the ‘uninstall’ option.

In comparison to regular recipes, uninstall recipes are muchsimpler. They are simply callable objects that accept the name of thepart to be uninstalled and the part’s options dictionary. Uninstallrecipes don’t have access to the part itself since it maybe not beable to be instantiated at uninstallation time.

Here’s a recipe that simulates installation of a system service, alongwith an uninstall recipe that simulates removing the service.

>>> write(sample_buildout, 'recipes', 'service.py',... """... class Service:......     def __init__(self, buildout, name, options):...         self.buildout = buildout...         self.name = name...         self.options = options......     def install(self):...         print "chkconfig --add %s" % self.options['script']...         return ()......     def update(self):...         pass......... def uninstall_service(name, options):...     print "chkconfig --del %s" % options['script']... """)

To use these recipes we must register them using entry points. Makesure to use the same name for the recipe and uninstall recipe. This isrequired to let buildout know which uninstall recipe goes with whichrecipe.

>>> write(sample_buildout, 'recipes', 'setup.py',... """... from setuptools import setup... entry_points = (... '''... [zc.buildout]... mkdir = mkdir:Mkdir... debug = debug:Debug... service = service:Service...... [zc.buildout.uninstall]... service = service:uninstall_service... ''')... setup(name="recipes", entry_points=entry_points)... """)

Here’s how these recipes could be used in a buildout:

>>> write(sample_buildout, 'buildout.cfg',... """... [buildout]... develop = recipes... parts = service...... [service]... recipe = recipes:service... script = /path/to/script... """)

When the buildout is run the service will be installed

>>> print system(buildout)Develop: '/sample-buildout/recipes'Uninstalling debug.Installing service.chkconfig --add /path/to/script<BLANKLINE>

The service has been installed. If the buildout is run again with nochanges, the service shouldn’t be changed.

>>> print system(buildout)Develop: '/sample-buildout/recipes'Updating service.<BLANKLINE>

Now we change the service part to trigger uninstallation andre-installation.

>>> write(sample_buildout, 'buildout.cfg',... """... [buildout]... develop = recipes... parts = service...... [service]... recipe = recipes:service... script = /path/to/a/different/script... """)

>>> print system(buildout)Develop: '/sample-buildout/recipes'Uninstalling service.Running uninstall recipe.chkconfig --del /path/to/scriptInstalling service.chkconfig --add /path/to/a/different/script<BLANKLINE>

Now we remove the service part, and add another part.

>>> write(sample_buildout, 'buildout.cfg',... """... [buildout]... develop = recipes... parts = debug...... [debug]... recipe = recipes:debug... """)

>>> print system(buildout)Develop: '/sample-buildout/recipes'Uninstalling service.Running uninstall recipe.chkconfig --del /path/to/a/different/scriptInstalling debug.recipe recipes:debug<BLANKLINE>

Uninstall recipes don’t have to take care of removing all the filesand directories created by the part. This is still done automatically,following the execution of the uninstall recipe. An upshot is that anuninstallation recipe can access files and directories created by arecipe before they are deleted.

For example, here’s an uninstallation recipe that simulates backing upa directory before it is deleted. It is designed to work with themkdir recipe introduced earlier.

>>> write(sample_buildout, 'recipes', 'backup.py',... """... import os... def backup_directory(name, options):...     path = options['path']...     size = len(os.listdir(path))...     print "backing up directory %s of size %s" % (path, size)... """)

It must be registered with the zc.buildout.uninstall entrypoint. Notice how it is given the name ‘mkdir’ to associate it withthe mkdir recipe.

>>> write(sample_buildout, 'recipes', 'setup.py',... """... from setuptools import setup... entry_points = (... '''... [zc.buildout]... mkdir = mkdir:Mkdir... debug = debug:Debug... service = service:Service...... [zc.buildout.uninstall]... uninstall_service = service:uninstall_service... mkdir = backup:backup_directory... ''')... setup(name="recipes", entry_points=entry_points)... """)

Now we can use it with a mkdir part.

>>> write(sample_buildout, 'buildout.cfg',... """... [buildout]... develop = recipes... parts = dir debug...... [dir]... recipe = recipes:mkdir... path = my_directory...... [debug]... recipe = recipes:debug... """)

Run the buildout to install the part.

>>> print system(buildout)Develop: '/sample-buildout/recipes'Uninstalling debug.Installing dir.dir: Creating directory my_directoryInstalling debug.recipe recipes:debug<BLANKLINE>

Now we remove the part from the configuration file.

>>> write(sample_buildout, 'buildout.cfg',... """... [buildout]... develop = recipes... parts = debug...... [debug]... recipe = recipes:debug... """)

When the buildout is run the part is removed, and the uninstall recipeis run before the directory is deleted.

>>> print system(buildout)Develop: '/sample-buildout/recipes'Uninstalling dir.Running uninstall recipe.backing up directory /sample-buildout/my_directory of size 0Updating debug.recipe recipes:debug<BLANKLINE>

Now we will return the registration to normal for the benefit of therest of the examples.

>>> write(sample_buildout, 'recipes', 'setup.py',... """... from setuptools import setup... entry_points = (... '''... [zc.buildout]... mkdir = mkdir:Mkdir... debug = debug:Debug... ''')... setup(name="recipes", entry_points=entry_points)... """)

Command-line usage

A number of arguments can be given on the buildout command line. Thecommand usage is:

buildout [options and assignments] [command [command arguments]]

The following options are supported:

-h (or –help)

Print basic usage information. If this option is used, then allother options are ignored.

-cfilename

The -c option can be used to specify a configuration file, rather thanbuildout.cfg in the current directory.

-tsocket_timeout

Specify the socket timeout in seconds.

-v

Increment the verbosity by 10. The verbosity is used to adjustthe logging level. The verbosity is subtracted from the numericvalue of the log-level option specified in the configuration file.

-q

Decrement the verbosity by 10.

-U

Don’t read user-default configuration.

-o

Run in off-line mode. This is equivalent to the assignmentbuildout:offline=true.

-O

Run in non-off-line mode. This is equivalent to the assignmentbuildout:offline=false. This is the default buildout mode. The-O option would normally be used to override a true offlinesetting in a configuration file.

-n

Run in newest mode. This is equivalent to the assignmentbuildout:newest=true. With this setting, which is the default,buildout will try to find the newest versions of distributionsavailable that satisfy its requirements.

-N

Run in non-newest mode. This is equivalent to the assignmentbuildout:newest=false. With this setting, buildout will not seeknew distributions if installed distributions satisfy it’srequirements.

Assignments are of the form:

section_name:option_name=value

Options and assignments can be given in any order.

Here’s an example:

>>> write(sample_buildout, 'other.cfg',... """... [buildout]... develop = recipes... parts = debug... installed = .other.cfg... log-level = WARNING...... [debug]... name = other... recipe = recipes:debug... """)

Note that we used the installed buildout option to specify analternate file to store information about installed parts.

>>> print system(buildout+' -c other.cfg debug:op1=foo -v'),Develop: '/sample-buildout/recipes'Installing debug.name otherop1 foorecipe recipes:debug

Here we used the -c option to specify an alternate configuration file,and the -v option to increase the level of logging from the default,WARNING.

Options can also be combined in the usual Unix way, as in:

>>> print system(buildout+' -vcother.cfg debug:op1=foo'),Develop: '/sample-buildout/recipes'Updating debug.name otherop1 foorecipe recipes:debug

Here we combined the -v and -c options with the configuration filename. Note that the -c option has to be last, because it takes anargument.

>>> os.remove(os.path.join(sample_buildout, 'other.cfg'))>>> os.remove(os.path.join(sample_buildout, '.other.cfg'))

The most commonly used command is ‘install’ and it takes a list ofparts to install. if any parts are specified, only those parts areinstalled. To illustrate this, we’ll update our configuration and runthe buildout in the usual way:

>>> write(sample_buildout, 'buildout.cfg',... """... [buildout]... develop = recipes... parts = debug d1 d2 d3...... [d1]... recipe = recipes:mkdir... path = d1...... [d2]... recipe = recipes:mkdir... path = d2...... [d3]... recipe = recipes:mkdir... path = d3...... [debug]... recipe = recipes:debug... """)

>>> print system(buildout),Develop: '/sample-buildout/recipes'Uninstalling debug.Installing debug.recipe recipes:debugInstalling d1.d1: Creating directory d1Installing d2.d2: Creating directory d2Installing d3.d3: Creating directory d3

>>> ls(sample_buildout)-  .installed.cfg-  b1.cfg-  b2.cfg-  base.cfgd  bin-  buildout.cfgd  d1d  d2d  d3d  demod  develop-eggsd  eggsd  partsd  recipes

>>> cat(sample_buildout, '.installed.cfg')... # doctest: +NORMALIZE_WHITESPACE[buildout]installed_develop_eggs = /sample-buildout/develop-eggs/recipes.egg-linkparts = debug d1 d2 d3<BLANKLINE>[debug]__buildout_installed__ =__buildout_signature__ = recipes-PiIFiO8ny5yNZ1S3JfT0xg==recipe = recipes:debug<BLANKLINE>[d1]__buildout_installed__ = /sample-buildout/d1__buildout_signature__ = recipes-PiIFiO8ny5yNZ1S3JfT0xg==path = /sample-buildout/d1recipe = recipes:mkdir<BLANKLINE>[d2]__buildout_installed__ = /sample-buildout/d2__buildout_signature__ = recipes-PiIFiO8ny5yNZ1S3JfT0xg==path = /sample-buildout/d2recipe = recipes:mkdir<BLANKLINE>[d3]__buildout_installed__ = /sample-buildout/d3__buildout_signature__ = recipes-PiIFiO8ny5yNZ1S3JfT0xg==path = /sample-buildout/d3recipe = recipes:mkdir

Now we’ll update our configuration file:

>>> write(sample_buildout, 'buildout.cfg',... """... [buildout]... develop = recipes... parts = debug d2 d3 d4...... [d2]... recipe = recipes:mkdir... path = data2...... [d3]... recipe = recipes:mkdir... path = data3...... [d4]... recipe = recipes:mkdir... path = ${d2:path}-extra...... [debug]... recipe = recipes:debug... x = 1... """)

and run the buildout specifying just d3 and d4:

>>> print system(buildout+' install d3 d4'),Develop: '/sample-buildout/recipes'Uninstalling d3.Installing d3.d3: Creating directory data3Installing d4.d4: Creating directory data2-extra

>>> ls(sample_buildout)-  .installed.cfg-  b1.cfg-  b2.cfg-  base.cfgd  bin-  buildout.cfgd  d1d  d2d  data2-extrad  data3d  demod  develop-eggsd  eggsd  partsd  recipes

Only the d3 and d4 recipes ran. d3 was removed and data3 and data2-extrawere created.

The .installed.cfg is only updated for the recipes that ran:

>>> cat(sample_buildout, '.installed.cfg')... # doctest: +NORMALIZE_WHITESPACE[buildout]installed_develop_eggs = /sample-buildout/develop-eggs/recipes.egg-linkparts = debug d1 d2 d3 d4<BLANKLINE>[debug]__buildout_installed__ =__buildout_signature__ = recipes-PiIFiO8ny5yNZ1S3JfT0xg==recipe = recipes:debug<BLANKLINE>[d1]__buildout_installed__ = /sample-buildout/d1__buildout_signature__ = recipes-PiIFiO8ny5yNZ1S3JfT0xg==path = /sample-buildout/d1recipe = recipes:mkdir<BLANKLINE>[d2]__buildout_installed__ = /sample-buildout/d2__buildout_signature__ = recipes-PiIFiO8ny5yNZ1S3JfT0xg==path = /sample-buildout/d2recipe = recipes:mkdir<BLANKLINE>[d3]__buildout_installed__ = /sample-buildout/data3__buildout_signature__ = recipes-PiIFiO8ny5yNZ1S3JfT0xg==path = /sample-buildout/data3recipe = recipes:mkdir<BLANKLINE>[d4]__buildout_installed__ = /sample-buildout/data2-extra__buildout_signature__ = recipes-PiIFiO8ny5yNZ1S3JfT0xg==path = /sample-buildout/data2-extrarecipe = recipes:mkdir

Note that the installed data for debug, d1, and d2 haven’t changed,because we didn’t install those parts and that the d1 and d2directories are still there.

Now, if we run the buildout without the install command:

>>> print system(buildout),Develop: '/sample-buildout/recipes'Uninstalling d2.Uninstalling d1.Uninstalling debug.Installing debug.recipe recipes:debugx 1Installing d2.d2: Creating directory data2Updating d3.Updating d4.

We see the output of the debug recipe and that data2 was created. Wealso see that d1 and d2 have gone away:

>>> ls(sample_buildout)-  .installed.cfg-  b1.cfg-  b2.cfg-  base.cfgd  bin-  buildout.cfgd  data2d  data2-extrad  data3d  demod  develop-eggsd  eggsd  partsd  recipes

Alternate directory and file locations

The buildout normally puts the bin, eggs, and parts directories in thedirectory in the directory containing the configuration file. You canprovide alternate locations, and even names for these directories.

>>> alt = tmpdir('sample-alt')

>>> write(sample_buildout, 'buildout.cfg',... """... [buildout]... develop = recipes... parts =... develop-eggs-directory = %(developbasket)s... eggs-directory = %(basket)s... bin-directory = %(scripts)s... parts-directory = %(work)s... """ % dict(...    developbasket = os.path.join(alt, 'developbasket'),...    basket = os.path.join(alt, 'basket'),...    scripts = os.path.join(alt, 'scripts'),...    work = os.path.join(alt, 'work'),... ))

>>> print system(buildout),Creating directory '/sample-alt/scripts'.Creating directory '/sample-alt/work'.Creating directory '/sample-alt/basket'.Creating directory '/sample-alt/developbasket'.Develop: '/sample-buildout/recipes'Uninstalling d4.Uninstalling d3.Uninstalling d2.Uninstalling debug.

>>> ls(alt)d  basketd  developbasketd  scriptsd  work

>>> ls(alt, 'developbasket')-  recipes.egg-link

You can also specify an alternate buildout directory:

>>> rmdir(alt)>>> alt = tmpdir('sample-alt')

>>> write(sample_buildout, 'buildout.cfg',... """... [buildout]... directory = %(alt)s... develop = %(recipes)s... parts =... """ % dict(...    alt=alt,...    recipes=os.path.join(sample_buildout, 'recipes'),...    ))

>>> print system(buildout),Creating directory '/sample-alt/bin'.Creating directory '/sample-alt/parts'.Creating directory '/sample-alt/eggs'.Creating directory '/sample-alt/develop-eggs'.Develop: '/sample-buildout/recipes'

>>> ls(alt)-  .installed.cfgd  bind  develop-eggsd  eggsd  parts

>>> ls(alt, 'develop-eggs')-  recipes.egg-link

Logging control

Three buildout options are used to control logging:

log-level

specifies the log level

verbosity

adjusts the log level

log-format

allows an alternate logging for mat to be specified

We’ve already seen the log level and verbosity. Let’s look at an exampleof changing the format:

>>> write(sample_buildout, 'buildout.cfg',... """... [buildout]... develop = recipes... parts =... log-level = 25... verbosity = 5... log-format = %(levelname)s %(message)s... """)

Here, we’ve changed the format to include the log-level name, ratherthan the logger name.

We’ve also illustrated, with a contrived example, that the log levelcan be a numeric value and that the verbosity can be specified in theconfiguration file. Because the verbosity is subtracted from the loglevel, we get a final log level of 20, which is the INFO level.

>>> print system(buildout),INFO Develop: '/sample-buildout/recipes'

Predefined buildout options

Buildouts have a number of predefined options that recipes can useand that users can override in their configuration files. To seethese, we’ll run a minimal buildout configuration with a debug logginglevel. One of the features of debug logging is that the configurationdatabase is shown.

>>> write(sample_buildout, 'buildout.cfg',... """... [buildout]... parts =... """)

>>> print system(buildout+' -vv'), # doctest: +NORMALIZE_WHITESPACEInstalling 'zc.buildout', 'setuptools'.We have a develop egg: zc.buildout X.X.We have the best distribution that satisfies 'setuptools'.Picked: setuptools = V.V<BLANKLINE>Configuration data:[buildout]accept-buildout-test-releases = falseallow-hosts = *allow-picked-versions = trueallowed-eggs-from-site-packages = *bin-directory = /sample-buildout/bindevelop-eggs-directory = /sample-buildout/develop-eggsdirectory = /sample-buildouteggs-directory = /sample-buildout/eggsexec-sitecustomize = trueexecutable = pythonfind-links =include-site-packages = trueinstall-from-cache = falseinstalled = /sample-buildout/.installed.cfglog-format =log-level = INFOnewest = trueoffline = falseparts =parts-directory = /sample-buildout/partsprefer-final = falsepython = buildoutrelative-paths = falsesocket-timeout =unzip = falseuse-dependency-links = trueverbosity = 20<BLANKLINE>

All of these options can be overridden by configuration files or bycommand-line assignments. We’ve discussed most of these optionsalready, but let’s review them and touch on some we haven’t discussed:

allowed-eggs-from-site-packages

Sometimes you need or want to control what eggs from site-packages areused. The allowed-eggs-from-site-packages option allows you to specify awhitelist of project names that may be included from site-packages. Youcan use globs to specify the value. It defaults to a single value of ‘*’,indicating that any package may come from site-packages.

Here’s a usage example:

[buildout]...allowed-eggs-from-site-packages =    demo    bigdemo    zope.*

This option interacts with theinclude-site-packages option in thefollowing ways.

Ifinclude-site-packages is true, thenallowed-eggs-from-site-packages filters what eggs from site-packagesmay be chosen. Therefore, ifallowed-eggs-from-site-packages is anempty list, then no eggs from site-packages are chosen, but site-packageswill still be included at the end of path lists.

Ifinclude-site-packages is false, the value ofallowed-eggs-from-site-packages is irrelevant.

See theinclude-site-packages description for more information.

bin-directory

The directory path where scripts are written. This can be arelative path, which is interpreted relative to the directoryoption.

develop-eggs-directory

The directory path where development egg links are created for softwarebeing created in the local project. This can be a relative path,which is interpreted relative to the directory option.

directory

The buildout directory. This is the base for other buildout fileand directory locations, when relative locations are used.

eggs-directory

The directory path where downloaded eggs are put. It is common to sharethis directory across buildouts. Eggs in this directory shouldnever be modified. This can be a relative path, which isinterpreted relative to the directory option.

exec-sitecustomize

Normally the Python’s real sitecustomize module is processed.If you do not want it to be processed in order to increase therepeatability of your buildout, set this value to ‘false’. This willbe honored irrespective of the setting for include-site-packages.This option will be honored by some recipes and not others.z3c.recipe.scripts honors this and zc.recipe.egg does not, forinstance.

executable

The Python executable used to run the buildout. See the pythonoption below.

include-site-packages

You can choose not to have the site-packages of the underlying Pythonavailable to your script or interpreter, in addition to the packagesfrom your eggs. This can increase repeatability for your buildout.This option will be better used by some recipes than others.z3c.recipe.scripts honors this fully and zc.recipe.egg onlypartially, for instance.

installed

The file path where information about the results of the previousbuildout run is written. This can be a relative path, which isinterpreted relative to the directory option. This file providesan inventory of installed parts with information needed to decidewhich if any parts need to be uninstalled.

log-format

The format used for logging messages.

log-level

The log level before verbosity adjustment

parts

A white space separated list of parts to be installed.

parts-directory

A working directory that parts can used to store data.

python

The name of a section containing information about the defaultPython interpreter. Recipes that need a installationtypically have options to tell them which Python installation touse. By convention, if a section-specific option isn’t used, theoption is looked for in the buildout section. The option mustpoint to a section with an executable option giving the path to aPython executable. By default, the buildout section defines thedefault Python as the Python used to run the buildout.

relative-paths

The paths generated by zc.buildout are absolute by default, and thisoption isfalse. However, if you set this value to betrue,bin/buildout will be generated with code that makes the paths relative.Some recipes, such as zc.recipe.egg and z3c.recipe.scripts, honor thisvalue as well.

unzip

By default, zc.buildout doesn’t unzip zip-safe eggs (“unzip = false”).This follows the policy followed by setuptools itself. Experience showsthis policy to to be inconvenient. Zipped eggs make debugging moredifficult and often import more slowly. You can include an unzip option inthe buildout section to change the default unzipping policy (“unzip =true”).

use-dependency-links

By default buildout will obey the setuptools dependency_links metadatawhen it looks for dependencies. This behavior can be controlled withthe use-dependency-links buildout option:

[buildout]...use-dependency-links = false

The option defaults to true. If you set it to false, then dependencylinks are only looked for in the locations specified by find-links.

verbosity

A log-level adjustment. Typically, this is set via the -q and -vcommand-line options.

Creating new buildouts and bootstrapping

If zc.buildout is installed, you can use it to create a new buildoutwith it’s own local copies of zc.buildout and setuptools and withlocal buildout scripts.

>>> sample_bootstrapped = tmpdir('sample-bootstrapped')

>>> print system(buildout...              +' -c'+os.path.join(sample_bootstrapped, 'setup.cfg')...              +' init'),Creating '/sample-bootstrapped/setup.cfg'.Creating directory '/sample-bootstrapped/bin'.Creating directory '/sample-bootstrapped/parts'.Creating directory '/sample-bootstrapped/eggs'.Creating directory '/sample-bootstrapped/develop-eggs'.Generated script '/sample-bootstrapped/bin/buildout'.

Note that a basic setup.cfg was created for us.

>>> ls(sample_bootstrapped)d  bind  develop-eggsd  eggsd  parts-  setup.cfg

>>> ls(sample_bootstrapped, 'bin')-  buildout

>>> _ = (ls(sample_bootstrapped, 'eggs'),...      ls(sample_bootstrapped, 'develop-eggs'))-  setuptools-0.6-py2.3.egg-  zc.buildout-1.0-py2.3.egg

(We list both the eggs and develop-eggs directories because thebuildout or setuptools egg could be installed in the develop-eggsdirectory if the original buildout had develop eggs for eitherbuildout or setuptools.)

If relative-paths istrue, the buildout script uses relative paths.

>>> write(sample_bootstrapped, 'setup.cfg',... '''... [buildout]... relative-paths = true... parts =... ''')

>>> print system(buildout...              +' -c'+os.path.join(sample_bootstrapped, 'setup.cfg')...              +' bootstrap'),Generated script '/sample-bootstrapped/bin/buildout'.

>>> buildout_script = join(sample_bootstrapped, 'bin', 'buildout')>>> import sys>>> if sys.platform.startswith('win'):...     buildout_script += '-script.py'>>> print open(buildout_script).read() # doctest: +ELLIPSIS#!... -S<BLANKLINE>import os<BLANKLINE>join = os.path.joinbase = os.path.dirname(os.path.abspath(os.path.realpath(__file__)))base = os.path.dirname(base)<BLANKLINE>import syssys.path[0:0] = [    join(base, 'parts/buildout'),    ]<BLANKLINE><BLANKLINE>import ospath = sys.path[0]if os.environ.get('PYTHONPATH'):    path = os.pathsep.join([path, os.environ['PYTHONPATH']])os.environ['BUILDOUT_ORIGINAL_PYTHONPATH'] = os.environ.get('PYTHONPATH', '')os.environ['PYTHONPATH'] = pathimport site # imports custom buildout-generated site.py<BLANKLINE>import zc.buildout.buildout<BLANKLINE>if __name__ == '__main__':    zc.buildout.buildout.main()<BLANKLINE>

Note that, in the above two examples, the buildout script was installedbut not run. To run the buildout, we’d have to run the installedbuildout script.

If we have an existing buildout that already has a buildout.cfg, we’llnormally use the bootstrap command instead of init. It will complainif there isn’t a configuration file:

>>> sample_bootstrapped2 = tmpdir('sample-bootstrapped2')

>>> print system(buildout...              +' -c'+os.path.join(sample_bootstrapped2, 'setup.cfg')...              +' bootstrap'),While:  Initializing.Error: Couldn't open /sample-bootstrapped2/setup.cfg

>>> write(sample_bootstrapped2, 'setup.cfg',... """... [buildout]... parts =... """)

>>> print system(buildout...              +' -c'+os.path.join(sample_bootstrapped2, 'setup.cfg')...              +' bootstrap'),Creating directory '/sample-bootstrapped2/bin'.Creating directory '/sample-bootstrapped2/parts'.Creating directory '/sample-bootstrapped2/eggs'.Creating directory '/sample-bootstrapped2/develop-eggs'.Generated script '/sample-bootstrapped2/bin/buildout'.

Newest and Offline Modes

By default buildout and recipes will try to find the newest versionsof distributions needed to satisfy requirements. This can be verytime consuming, especially when incrementally working on setting up abuildout or working on a recipe. The buildout newest option can beused to to suppress this. If the newest option is set to false, thennew distributions won’t be sought if an installed distribution meetsrequirements. The newest option can be set to false using the -Ncommand-line option.

The offline option goes a bit further. If the buildout offline optionis given a value of “true”, the buildout and recipes that are aware ofthe option will avoid doing network access. This is handy whenrunning the buildout when not connected to the internet. It alsomakes buildouts run much faster. This option is typically set usingthe buildout -o option.

Preferring Final Releases

Currently, when searching for new releases of your project’sdependencies, the newest available release is used. This isn’t usuallyideal, as you may get a development release or alpha releases not readyto be widely used. You can request that final releases be preferredusing theprefer-final option in the buildout section:

[buildout]...prefer-final = true

When theprefer-final option is set to true, then when searching fornew releases, final releases are preferred. If there are finalreleases that satisfy distribution requirements, then those releasesare used even if newer non-final releases are available.

In buildout version 2, all final releases will be preferred bydefault–that isprefer-final will also default to ‘true’. You willthen need to use a ‘false’ value forprefer-final to get the newestreleases.

A separate option controls the behavior of the build system itself.When buildout looks for recipes, extensions, and for updates to itself,it does prefer final releases by default, as of the 1.5.0 release. Theaccept-buildout-test-releases option will let you override this behavior.However, it is typically changed by the –accept-buildout-test-releasesoption to the bootstrap script, since bootstrapping is the first step toselecting a buildout.

Finding distributions

By default, buildout searches the Python Package Index when lookingfor distributions. You can, instead, specify your own index to searchusing theindex option:

[buildout]...index = http://index.example.com/

This index, or the default ofhttp://pypi.python.org/simple/ if noindex is specified, will always be searched for distributions unlessrunning buildout with options that prevent searching fordistributions. The latest version of the distribution that meets therequirements of the buildout will always be used.

You can also specify more locations to search for distributions usingthefind-links option. All locations specified will be searched fordistributions along with the package index as described before.

Locations can be urls:

[buildout]...find-links = http://download.zope.org/distribution/

They can also be directories on disk:

[buildout]...find-links = /some/path

Finally, they can also be direct paths to distributions:

[buildout]...find-links = /some/path/someegg-1.0.0-py2.3.egg

Any number of locations can be specified in thefind-links option:

[buildout]...find-links =    http://download.zope.org/distribution/    /some/otherpath    /some/path/someegg-1.0.0-py2.3.egg

Dependency links

By default buildout will obey the setuptools dependency_links metadatawhen it looks for dependencies. This behavior can be controlled withthe use-dependency-links buildout option:

[buildout]...use-dependency-links = false

The option defaults to true. If you set it to false, then dependencylinks are only looked for in the locations specified by find-links.

Controlling the installation database

The buildout installed option is used to specify the file used to saveinformation on installed parts. This option is initialized to“.installed.cfg”, but it can be overridden in the configuration fileor on the command line:

>>> write('buildout.cfg',... """... [buildout]... develop = recipes... parts = debug...... [debug]... recipe = recipes:debug... """)

>>> print system(buildout+' buildout:installed=inst.cfg'),Develop: '/sample-buildout/recipes'Installing debug.recipe recipes:debug

>>> ls(sample_buildout)-  b1.cfg-  b2.cfg-  base.cfgd  bin-  buildout.cfgd  demod  develop-eggsd  eggs-  inst.cfgd  partsd  recipes

The installation database can be disabled by supplying an emptybuildout installed option:

>>> os.remove('inst.cfg')>>> print system(buildout+' buildout:installed='),Develop: '/sample-buildout/recipes'Installing debug.recipe recipes:debug

>>> ls(sample_buildout)-  b1.cfg-  b2.cfg-  base.cfgd  bin-  buildout.cfgd  demod  develop-eggsd  eggsd  partsd  recipes

Note that there will be no installation database if there are no parts:

>>> write('buildout.cfg',... """... [buildout]... parts =... """)

>>> print system(buildout+' buildout:installed=inst.cfg'),

>>> ls(sample_buildout)-  b1.cfg-  b2.cfg-  base.cfgd  bin-  buildout.cfgd  demod  develop-eggsd  eggsd  partsd  recipes

Extensions

A feature allows code to be loaded and run afterconfiguration files have been read but before the buildout has begunany processing. The intent is to allow special plugins such asurllib2 request handlers to be loaded.

To load an extension, we use the extensions option and list one ormore distribution requirements, on separate lines. The distributionsnamed will be loaded and anyzc.buildout.extension entry points foundwill be called with the buildout as an argument. When buildoutfinishes processing, anyzc.buildout.unloadextension entry pointsfound will be called with the buildout as an argument.

Let’s create a sample extension in our sample buildout created in theprevious section:

>>> mkdir(sample_bootstrapped, 'demo')

>>> write(sample_bootstrapped, 'demo', 'demo.py',... """... def ext(buildout):...     print 'ext', list(buildout)... def unload(buildout):...     print 'unload', list(buildout)... """)

>>> write(sample_bootstrapped, 'demo', 'setup.py',... """... from setuptools import setup...... setup(...     name = "demo",...     entry_points = {...        'zc.buildout.extension': ['ext = demo:ext'],...        'zc.buildout.unloadextension': ['ext = demo:unload'],...        },...     )... """)

Our extension just prints out the word ‘demo’, and lists the sectionsfound in the buildout passed to it.

We’ll update our buildout.cfg to list the demo directory as a developegg to be built:

>>> write(sample_bootstrapped, 'buildout.cfg',... """... [buildout]... develop = demo... parts =... """)

>>> os.chdir(sample_bootstrapped)>>> print system(os.path.join(sample_bootstrapped, 'bin', 'buildout')),Develop: '/sample-bootstrapped/demo'

Now we can add the extensions option. We were a bit tricky and ranthe buildout once with the demo develop egg defined but without theextension option. This is because extensions are loaded before thebuildout creates develop eggs. We needed to use a separate buildoutrun to create the develop egg. Normally, when eggs are loaded fromthe network, we wouldn’t need to do anything special.

>>> write(sample_bootstrapped, 'buildout.cfg',... """... [buildout]... develop = demo... extensions = demo... parts =... """)

We see that our extension is loaded and executed:

>>> print system(os.path.join(sample_bootstrapped, 'bin', 'buildout')),ext ['buildout']Develop: '/sample-bootstrapped/demo'unload ['buildout']

Allow hosts

On some environments the links visited byzc.buildout can be forbiddenby paranoiac firewalls. These URL might be on the chain of linksvisited byzc.buildout wheter they are defined in thefind-links option,wheter they are defined by various eggs in theirurl,download_url,dependency_links metadata.

It is even harder to track that package_index works like a spider andmight visit links and go to other location.

Theallow-hosts option provides a way to prevent this, andworks exactly like the one provided ineasy_install.

You can provide a list of allowed host, together with wildcards:

[buildout]...allow-hosts =    *.python.org    example.com

All urls that does not match these hosts will not be visited.

In the future, additional methods may beadded. Older recipes with fewer methods will still besupported.

If we wanted to create a distribution from thispackage, we would need specify much more information. See thesetuptools documentation.

Downloading without using the cache

If no download cache should be used, the download utility is instantiatedwithout any arguments:

>>> from zc.buildout.download import Download>>> download = Download()>>> print download.cache_dirNone

Downloading a file is achieved by calling the utility with the URL as anargument. A tuple is returned that consists of the path to the downloaded copyof the file and a boolean value indicating whether this is a temporary filemeant to be cleaned up during the same buildout run:

>>> path, is_temp = download(server_url+'foo.txt')>>> print path/.../buildout-...>>> cat(path)This is a foo text.

As we aren’t using the download cache and haven’t specified a target patheither, the download has ended up in a temporary file:

>>> is_tempTrue

>>> import tempfile>>> path.startswith(tempfile.gettempdir())True

We are responsible for cleaning up temporary files behind us:

>>> remove(path)

When trying to access a file that doesn’t exist, we’ll get an exception:

>>> try: download(server_url+'not-there') # doctest: +ELLIPSIS... except: print 'download error'... else: print 'woops'download error

Downloading a local file doesn’t produce a temporary file but simply returnsthe local file itself:

>>> download(join(server_data, 'foo.txt'))('/sample_files/foo.txt', False)

We can also have the downloaded file’s MD5 sum checked:

>>> try: from hashlib import md5... except ImportError: from md5 import new as md5

>>> path, is_temp = download(server_url+'foo.txt',...                          md5('This is a foo text.').hexdigest())>>> is_tempTrue>>> remove(path)

>>> download(server_url+'foo.txt',...          md5('The wrong text.').hexdigest())Traceback (most recent call last):ChecksumError: MD5 checksum mismatch downloading 'http://localhost/foo.txt'

The error message in the event of an MD5 checksum mismatch for a local filereads somewhat differently:

>>> download(join(server_data, 'foo.txt'),...               md5('This is a foo text.').hexdigest())('/sample_files/foo.txt', False)

>>> download(join(server_data, 'foo.txt'),...          md5('The wrong text.').hexdigest())Traceback (most recent call last):ChecksumError: MD5 checksum mismatch for local resource at '/sample_files/foo.txt'.

Finally, we can download the file to a specified place in the file system:

>>> target_dir = tmpdir('download-target')>>> path, is_temp = download(server_url+'foo.txt',...                          path=join(target_dir, 'downloaded.txt'))>>> print path/download-target/downloaded.txt>>> cat(path)This is a foo text.>>> is_tempFalse

Trying to download a file in offline mode will result in an error:

>>> download = Download(cache=None, offline=True)>>> download(server_url+'foo.txt')Traceback (most recent call last):UserError: Couldn't download 'http://localhost/foo.txt' in offline mode.

As an exception to this rule, file system paths and URLs in thefilescheme will still work:

>>> cat(download(join(server_data, 'foo.txt'))[0])This is a foo text.>>> cat(download('file:' + join(server_data, 'foo.txt'))[0])This is a foo text.

>>> remove(path)

Downloading using the download cache

In order to make use of the download cache, we need to configure the downloadutility differently. To do this, we pass a directory path as thecacheattribute upon instantiation:

>>> cache = tmpdir('download-cache')>>> download = Download(cache=cache)>>> print download.cache_dir/download-cache/

>>> ls(cache)>>> path, is_temp = download(server_url+'foo.txt')>>> print path/download-cache/foo.txt>>> cat(path)This is a foo text.>>> is_tempFalse

>>> write(server_data, 'foo.txt', 'The wrong text.')>>> path, is_temp = download(server_url+'foo.txt')>>> print path/download-cache/foo.txt>>> cat(path)This is a foo text.

>>> download(server_url+'foo.txt', md5('The wrong text.').hexdigest())Traceback (most recent call last):ChecksumError: MD5 checksum mismatch for cached download               from 'http://localhost/foo.txt' at '/download-cache/foo.txt'

>>> mkdir(server_data, 'other')>>> write(server_data, 'other', 'foo.txt', 'The wrong text.')>>> path, is_temp = download(server_url+'other/foo.txt')>>> print path/download-cache/foo.txt>>> cat(path)This is a foo text.

>>> remove(cache, 'foo.txt')>>> ls(cache)>>> write(server_data, 'foo.txt', 'This is a foo text.')

>>> path, is_temp = download(server_url+'foo.txt',...                          path=join(target_dir, 'downloaded.txt'))>>> print path/download-target/downloaded.txt>>> cat(path)This is a foo text.>>> is_tempFalse>>> ls(cache)- foo.txt

>>> remove(path)>>> write(server_data, 'foo.txt', 'The wrong text.')

>>> path, is_temp = download(server_url+'foo.txt',...                          path=join(target_dir, 'downloaded.txt'))>>> print path/download-target/downloaded.txt>>> cat(path)This is a foo text.>>> is_tempFalse

>>> download = Download(cache=cache, offline=True)>>> cat(download(server_url+'foo.txt')[0])This is a foo text.

>>> remove(cache, 'foo.txt')>>> ls(cache)

>>> write(server_data, 'foo.txt', 'This is a foo text.')>>> download = Download(cache=cache)

>>> cat(download('file:' + join(server_data, 'foo.txt'), path=path)[0])This is a foo text.>>> ls(cache)- foo.txt

>>> remove(cache, 'foo.txt')

>>> cat(download(join(server_data, 'foo.txt'), path=path)[0])This is a foo text.>>> ls(cache)- foo.txt

>>> remove(cache, 'foo.txt')

>>> download(server_url+'foo.txt', md5('The wrong text.').hexdigest())Traceback (most recent call last):ChecksumError: MD5 checksum mismatch downloading 'http://localhost/foo.txt'>>> ls(cache)

>>> remove(path)

>>> Download(cache=join(cache, 'non-existent'))(server_url+'foo.txt')Traceback (most recent call last):UserError: The directory:'/download-cache/non-existent'to be used as a download cache doesn't exist.

>>> download = Download(cache=cache, namespace='test')>>> print download.cache_dir/download-cache/test

>>> ls(cache)

>>> path, is_temp = download(server_url+'foo.txt')>>> print path/download-cache/test/foo.txt>>> ls(cache)d test>>> ls(cache, 'test')- foo.txt>>> cat(path)This is a foo text.>>> is_tempFalse

>>> write(server_data, 'foo.txt', 'The wrong text.')>>> write(cache, 'foo.txt', 'The wrong text.')

>>> path, is_temp = download(server_url+'foo.txt')>>> print path/download-cache/test/foo.txt>>> cat(path)This is a foo text.

>>> rmdir(cache, 'test')>>> remove(cache, 'foo.txt')>>> write(server_data, 'foo.txt', 'This is a foo text.')

>>> download = Download(cache=cache, hash_name=True)>>> path, is_temp = download(server_url+'foo.txt')>>> print path/download-cache/09f5793fcdc1716727f72d49519c688d>>> cat(path)This is a foo text.>>> ls(cache)- 09f5793fcdc1716727f72d49519c688d

>>> path.lower() == join(cache, md5(server_url+'foo.txt').hexdigest()).lower()True

>>> write(server_data, 'foo.txt', 'The wrong text.')>>> (path, is_temp) == download(server_url+'foo.txt')True>>> cat(path)This is a foo text.>>> ls(cache)- 09f5793fcdc1716727f72d49519c688d

>>> path2, is_temp = download(server_url+'other/foo.txt')>>> print path2/download-cache/537b6d73267f8f4447586989af8c470e>>> path == path2False>>> path2.lower() == join(cache, md5(server_url+'other/foo.txt').hexdigest()).lower()True>>> cat(path)This is a foo text.>>> cat(path2)The wrong text.>>> ls(cache)- 09f5793fcdc1716727f72d49519c688d- 537b6d73267f8f4447586989af8c470e

>>> remove(path)>>> remove(path2)>>> write(server_data, 'foo.txt', 'This is a foo text.')

Using the cache purely as a fall-back

Sometimes it is desirable to try downloading a file from the net if at allpossible, and use the cache purely as a fall-back option when a server isdown or if we are in offline mode. This mode is only in effect if a downloadcache is configured in the first place:

>>> download = Download(cache=cache, fallback=True)>>> print download.cache_dir/download-cache/

A downloaded file will be cached:

>>> ls(cache)>>> path, is_temp = download(server_url+'foo.txt')>>> ls(cache)- foo.txt>>> cat(cache, 'foo.txt')This is a foo text.>>> is_tempFalse

If the file cannot be served, the cached copy will be used:

>>> remove(server_data, 'foo.txt')>>> try: Download()(server_url+'foo.txt') # doctest: +ELLIPSIS... except: print 'download error'... else: print 'woops'download error>>> path, is_temp = download(server_url+'foo.txt')>>> cat(path)This is a foo text.>>> is_tempFalse

Similarly, if the file is served but we’re in offline mode, we’ll fall back tousing the cache:

>>> write(server_data, 'foo.txt', 'The wrong text.')>>> get(server_url+'foo.txt')'The wrong text.'

>>> offline_download = Download(cache=cache, offline=True, fallback=True)>>> path, is_temp = offline_download(server_url+'foo.txt')>>> print path/download-cache/foo.txt>>> cat(path)This is a foo text.>>> is_tempFalse

However, when downloading the file normally with the cache being used infall-back mode, the file will be downloaded from the net and the cached copywill be replaced with the new content:

>>> cat(download(server_url+'foo.txt')[0])The wrong text.>>> cat(cache, 'foo.txt')The wrong text.

When trying to download a resource whose checksum does not match, the cachedcopy will neither be used nor overwritten:

>>> write(server_data, 'foo.txt', 'This is a foo text.')>>> download(server_url+'foo.txt', md5('The wrong text.').hexdigest())Traceback (most recent call last):ChecksumError: MD5 checksum mismatch downloading 'http://localhost/foo.txt'>>> cat(cache, 'foo.txt')The wrong text.

Configuring the download utility from buildout options

The configuration options explained so far derive from the build logicimplemented by the calling code. Other options configure the download utilityfor use in a particular project or buildout run; they are read from thebuildout configuration section. The latter can be passed directly as thefirst argument to the download utility’s constructor.

The location of the download cache is specified by thedownload-cacheoption:

>>> download = Download({'download-cache': cache}, namespace='cmmi')>>> print download.cache_dir/download-cache/cmmi

If thedownload-cache option specifies a relative path, it is understoodrelative to the current working directory, or to the buildout directory ifthat is given:

>>> download = Download({'download-cache': 'relative-cache'})>>> print download.cache_dir/sample-buildout/relative-cache/

>>> download = Download({'directory': join(sample_buildout, 'root'),...                      'download-cache': 'relative-cache'})>>> print download.cache_dir/sample-buildout/root/relative-cache/

Keyword parameters take precedence over the corresponding options:

>>> download = Download({'download-cache': cache}, cache=None)>>> print download.cache_dirNone

Whether to assume offline mode can be inferred from either theoffline ortheinstall-from-cache option. As usual with zc.buildout, these optionsmust assume one of the values ‘true’ and ‘false’:

>>> download = Download({'offline': 'true'})>>> download.offlineTrue

>>> download = Download({'offline': 'false'})>>> download.offlineFalse

>>> download = Download({'install-from-cache': 'true'})>>> download.offlineTrue

>>> download = Download({'install-from-cache': 'false'})>>> download.offlineFalse

These two options are combined using logical ‘or’:

>>> download = Download({'offline': 'true', 'install-from-cache': 'false'})>>> download.offlineTrue

>>> download = Download({'offline': 'false', 'install-from-cache': 'true'})>>> download.offlineTrue

Theoffline keyword parameter takes precedence over both theofflineandinstall-from-cache options:

>>> download = Download({'offline': 'true'}, offline=False)>>> download.offlineFalse

>>> download = Download({'install-from-cache': 'false'}, offline=True)>>> download.offlineTrue

Regressions

MD5 checksum calculation needs to be reliable on all supported systems, whichrequires text files to be treated as binary to avoid implicit line-endingconversions:

>>> text = 'First line of text.\r\nSecond line.\r\n'>>> f = open(join(server_data, 'foo.txt'), 'wb')>>> f.write(text)>>> f.close()>>> path, is_temp = Download()(server_url+'foo.txt', md5(text).hexdigest())>>> remove(path)

Clean up

We should have cleaned up all temporary files created by downloading things:

>>> ls(tempfile.tempdir)

Reset the global temporary directory:

>>> tempfile.tempdir = old_tempdir

Installing solely from a download cache

A download cache can be used as the basis of application sourcereleases. In an application source release, we want to distribute anapplication that can be built without making any network accesses. Inthis case, we distribute a buildout with download cache and tell thebuildout to install from the download cache only, without makingnetwork accesses. The buildout install-from-cache option can be usedto signal that packages should be installed only from the downloadcache.

Let’s remove our installed eggs and run the buildout with theinstall-from-cache option set to true:

>>> for  f in os.listdir('eggs'):...     if f.startswith('demo'):...         remove('eggs', f)

>>> write('buildout.cfg',... '''... [buildout]... parts = eggs... download-cache = %(cache)s... install-from-cache = true... find-links = %(link_server)s...... [eggs]... recipe = zc.recipe.egg... eggs = demo... ''' % globals())

>>> print system(buildout),Uninstalling eggs.Installing eggs.Getting distribution for 'demo'.Got demo 0.2.Getting distribution for 'demoneeded'.Got demoneeded 1.2c1.Generated script '/sample-buildout/bin/demo'.

Basic use of the extends cache

We put some base configuration on a server and reference it from a samplebuildout:

>>> write(server_data, 'base.cfg', """\... [buildout]... parts =... foo = bar... """)

>>> write('buildout.cfg', """\... [buildout]... extends = %sbase.cfg... """ % server_url)

When trying to run this buildout offline, we’ll find that we cannot read allof the required configuration:

>>> print system(buildout + ' -o')While:  Initializing.Error: Couldn't download 'http://localhost/base.cfg' in offline mode.

Trying the same online, we can:

>>> print system(buildout)Unused options for buildout: 'foo'.

As long as we haven’t said anything about caching downloaded configuration,nothing gets cached. Offline mode will still cause the buildout to fail:

>>> print system(buildout + ' -o')While:  Initializing.Error: Couldn't download 'http://localhost/base.cfg' in offline mode.

Let’s now specify a cache for base configuration files. This cache isdifferent from the download cache used by recipes for caching distributionsand other files; one might, however, use a namespace subdirectory of thedownload cache for it. The configuration cache we specify will be created whenrunning buildout and the base.cfg file will be put in it (with the file namebeing a hash of the complete URL):

>>> mkdir('cache')>>> write('buildout.cfg', """\... [buildout]... extends = %sbase.cfg... extends-cache = cache... """ % server_url)

>>> print system(buildout)Unused options for buildout: 'foo'.

>>> cache = join(sample_buildout, 'cache')>>> ls(cache)-  5aedc98d7e769290a29d654a591a3a45

>>> import os>>> cat(cache, os.listdir(cache)[0])[buildout]parts =foo = bar

We can now run buildout offline as it will read base.cfg from the cache:

>>> print system(buildout + ' -o')Unused options for buildout: 'foo'.

The cache is being used purely as a fall-back in case we are offline or don’thave access to a configuration file to be downloaded. As long as we areonline, buildout attempts to download a fresh copy of each file even if acached copy of the file exists. To see this, we put different configuration inthe same place on the server and run buildout in offline mode so it takesbase.cfg from the cache:

>>> write(server_data, 'base.cfg', """\... [buildout]... parts =... bar = baz... """)

>>> print system(buildout + ' -o')Unused options for buildout: 'foo'.

In online mode, buildout will download and use the modified version:

>>> print system(buildout)Unused options for buildout: 'bar'.

Trying offline mode again, the new version will be used as it has been put inthe cache now:

>>> print system(buildout + ' -o')Unused options for buildout: 'bar'.

Clean up:

>>> rmdir(cache)

Specifying extends cache and offline mode

Normally, the values of buildout options such as the location of a downloadcache or whether to use offline mode are determined by first reading theuser’s default configuration, updating it with the project’s configuration andfinally applying command-line options. User and project configuration areassembled by reading a file such as~/.buildout/default.cfg,buildout.cfg or a URL given on the command line, recursively (depth-first)downloading any base configuration specified by thebuildout:extendsoption read from each of those config files, and finally evaluating eachconfig file to provide default values for options not yet read.

This works fine for all options that do not influence how configuration isdownloaded in the first place. Theextends-cache andoffline options,however, are treated differently from the procedure described in order to makeit simple and obvious to see where a particular configuration file came fromunder any particular circumstances.

• Offline and extends-cache settings are read from the two root config filesexclusively. Otherwise one could construct configuration files that, whenread, imply that they should have been read from a different source thanthey have. Also, specifying the extends cache within a file that might haveto be taken from the cache before being read wouldn’t make a lot of sense.

• Offline and extends-cache settings given by the user’s defaults apply to theprocess of assembling the project’s configuration. If no extends cache hasbeen specified by the user’s default configuration, the project’s rootconfig file must be available, be it from disk or from the net.

• Offline mode turned on by the-o command line option is honoured fromthe beginning even though command line options are applied to theconfiguration last. If offline mode is not requested by the command line, itmay be switched on by either the user’s or the project’s config root.

>>> mkdir('home')>>> mkdir('home', '.buildout')>>> mkdir('cache')>>> mkdir('user-cache')>>> os.environ['HOME'] = join(sample_buildout, 'home')>>> write('home', '.buildout', 'default.cfg', """\... [buildout]... extends = fancy_default.cfg... extends-cache = user-cache... """)>>> write('home', '.buildout', 'fancy_default.cfg', """\... [buildout]... extends = %sbase_default.cfg... """ % server_url)>>> write(server_data, 'base_default.cfg', """\... [buildout]... foo = bar... offline = false... """)

>>> write('buildout.cfg', """\... [buildout]... extends = fancy.cfg... extends-cache = cache... """)>>> write('fancy.cfg', """\... [buildout]... extends = %sbase.cfg... """ % server_url)>>> write(server_data, 'base.cfg', """\... [buildout]... parts =... offline = false... """)

>>> print system(buildout)Unused options for buildout: 'foo'.

>>> ls('user-cache')-  10e772cf422123ef6c64ae770f555740>>> cat('user-cache', os.listdir('user-cache')[0])[buildout]foo = baroffline = false

>>> ls('cache')-  c72213127e6eb2208a3e1fc1dba771a7>>> cat('cache', os.listdir('cache')[0])[buildout]parts =offline = false

>>> write('home', '.buildout', 'default.cfg', """\... [buildout]... extends = fancy_default.cfg... """)>>> write('home', '.buildout', 'fancy_default.cfg', """\... [buildout]... extends = %sbase_default.cfg... extends-cache = user-cache... """ % server_url)

>>> write('buildout.cfg', """\... [buildout]... extends = fancy.cfg... """)>>> write('fancy.cfg', """\... [buildout]... extends = %sbase.cfg... extends-cache = cache... """ % server_url)

>>> remove('user-cache', os.listdir('user-cache')[0])>>> remove('cache', os.listdir('cache')[0])

>>> print system(buildout)Unused options for buildout: 'foo'.

>>> ls('user-cache')-  0548bad6002359532de37385bb532e26>>> cat('user-cache', os.listdir('user-cache')[0])[buildout]parts =offline = false

>>> ls('cache')

>>> rmdir('user-cache')>>> rmdir('cache')

>>> print system(buildout + ' -o')While:  Initializing.Error: Couldn't download 'http://localhost/base_default.cfg' in offline mode.

>>> write('home', '.buildout', 'default.cfg', """\... [buildout]... extends = fancy_default.cfg... offline = true... """)>>> print system(buildout)While:  Initializing.Error: Couldn't download 'http://localhost/base_default.cfg' in offline mode.

>>> write('home', '.buildout', 'default.cfg', """\... [buildout]... extends = fancy_default.cfg... """)>>> write('home', '.buildout', 'fancy_default.cfg', """\... [buildout]... extends = %sbase_default.cfg... offline = true... """ % server_url)>>> print system(buildout)While:  Initializing.Error: Couldn't download 'http://localhost/base.cfg' in offline mode.

>>> write('home', '.buildout', 'fancy_default.cfg', """\... [buildout]... extends = %sbase_default.cfg... """ % server_url)>>> write('buildout.cfg', """\... [buildout]... extends = fancy.cfg... offline = true... """)>>> print system(buildout)While:  Initializing.Error: Couldn't download 'http://localhost/base.cfg' in offline mode.

>>> write('buildout.cfg', """\... [buildout]... extends = fancy.cfg... """)>>> write('fancy.cfg', """\... [buildout]... extends = %sbase.cfg... offline = true... """ % server_url)>>> print system(buildout)Unused options for buildout: 'foo'.

>>> write('home', '.buildout', 'default.cfg', """\... [buildout]... extends = fancy_default.cfg... install-from-cache = true... """)>>> print system(buildout)While:  Initializing.Error: Couldn't download 'http://localhost/base_default.cfg' in offline mode.

>>> write('home', '.buildout', 'default.cfg', """\... [buildout]... extends = fancy_default.cfg... """)>>> write('home', '.buildout', 'fancy_default.cfg', """\... [buildout]... extends = %sbase_default.cfg... install-from-cache = true... """ % server_url)>>> print system(buildout)While:  Initializing.Error: Couldn't download 'http://localhost/base.cfg' in offline mode.

>>> write('home', '.buildout', 'fancy_default.cfg', """\... [buildout]... extends = %sbase_default.cfg... """ % server_url)>>> write('buildout.cfg', """\... [buildout]... extends = fancy.cfg... install-from-cache = true... """)>>> print system(buildout)While:  Initializing.Error: Couldn't download 'http://localhost/base.cfg' in offline mode.

>>> write('buildout.cfg', """\... [buildout]... extends = fancy.cfg... """)>>> write('fancy.cfg', """\... [buildout]... extends = %sbase.cfg... install-from-cache = true... """ % server_url)>>> print system(buildout)While:  Installing.  Checking for upgrades.An internal error occurred ...ValueError: install_from_cache set to true with no download cache

Clean up

We should have cleaned up all temporary files created by downloading things:

>>> ls(tempfile.tempdir)

Reset the global temporary directory:

>>> tempfile.tempdir = old_tempdir

zc.buildout.testing.buildoutSetUp(test)

The buildoutSetup function can be used as a doctest setup function.It creates a sample buildout that can be used by tests, changing thecurrent working directory to the sample_buildout. It also adds anumber of names to the test namespace:

sample_buildout

This is the name of a buildout with a basic configuration.

buildout

This is the path of the buildout script in the sample buildout.

ls(*path)

List the contents of a directory. The directory path is provided as one ormore strings, to be joined with os.path.join.

cat(*path)

Display the contents of a file. The file path is provided as one ormore strings, to be joined with os.path.join.

On Windows, if the file doesn’t exist, the function will tryadding a ‘-script.py’ suffix. This helps to work around adifference in script generation on windows.

mkdir(*path)

Create a directory. The directory path is provided as one ormore strings, to be joined with os.path.join.

rmdir(*path)

Remove a directory. The directory path is provided as one ormore strings, to be joined with os.path.join.

remove(*path)

Remove a directory or file. The path is provided as one ormore strings, to be joined with os.path.join.

tmpdir(name)

Create a temporary directory with the given name. The directorywill be automatically removed at the end of the test. The path ofthe created directory is returned.

Further, if the the normalize_path normlaizing substitution (seebelow) is used, then any paths starting with this path will benormalized to:

/name/restofpath

No two temporary directories can be created with the same name. Adirectory created with tmpdir can be removed with rmdir and recreated.

Note that the sample_buildout directory is created by calling thisfunction.

write(*path_and_contents)

Create a file. The file path is provided as one or more strings,to be joined with os.path.join. The last argument is the file contents.

system(command,input='')

Execute a system command with the given input passed to thecommand’s standard input. The output (error and regular output)from the command is returned.

get(url)

Get a web page.

cd(*path)

Change to the given directory. The directory path is provided as one ormore strings, to be joined with os.path.join.

The directory will be reset at the end of the test.

join(*path)

A convenient reference to os.path.join.

register_teardown(func)

Register a tear-down function. The function will be called withno arguments at the end of the test.

start_server(path)

Start a web server on the given path. The server will be shutdown at the end of the test. The server URL is returned.

You can cause the server to start and stop logging it’s outputusing:

>>> get(server_url+'enable_server_logging')

and:

>>> get(server_url+'disable_server_logging')

This can be useful to see how buildout is interacting with aserver.

sdist(setup, dest)

Create a source distribution by running the given setup file andplacing the result in the given destination directory. If thesetup argument is a directory, the thge setup.py file in thatdirectory is used.

bdist_egg(setup, executable, dest)

Create an egg by running the given setup file with the givenPython executable and placing the result in the given destinationdirectory. If the setup argument is a directory, then thesetup.py file in that directory is used.

find_python(version)

Find a Python executable for the given version, where version is astring like “2.4”.

This function uses the following strategy to find a Python of thegiven version:

• Look for an environment variable of the form PYTHON%(version)s.

• On windows, look for Pythonm%(version)spython

• on Unix, try running python%(version)s or just python to get theexecutable

zc.buildout.testing.buildoutTearDown(test)

Tear down everything set up by zc.buildout.testing.buildoutSetUp. Anyfunctions passed to register_teardown are called as well.

install(project, destination)

Install eggs for a given project into a destination. If thedestination is a test object, then the eggs directory of thesample buildout (sample_buildout) defined by the test will be used.Tests will use this to install the distributions for the packagesbeing tested (and their dependencies) into a sample buildout. The eggto be used should already be loaded, by importing one of the modulesprovided, before calling this function.

install_develop(project, destination)

Like install, but a develop egg is installed even if the current eggif not a develop egg.

Output normalization

Recipe tests often generate output that is dependent on temporary filelocations, operating system conventions or Python versions. To dealwith these dependencies, we often usezope.testing.renormalizing.RENormalizing to normalize test output.zope.testing.renormalizing.RENormalizing takes pairs of regularexpressions and substitutions. The zc.buildout.testing module providesa few helpful variables that define regular-expression/substitutionpairs that you can pass to zope.testing.renormalizing.RENormalizing.

normalize_path

Converts tests paths, based on directories created with tmpdir(),to simple paths.

normalize_script

On Unix-like systems, scripts are implemented in single fileswithout suffixes. On windows, scripts are implemented with 2files, a -script.py file and a .exe file. This normalizationconverts directory listings of Windows scripts to the formgenerated on UNix-like systems.

normalize_egg_py

Normalize Python version and platform indicators, if specified, inegg names.

Distribution installation

The easy_install module provides a function, install, for installing oneor more packages and their dependencies. The install function takes 2positional arguments:

• An iterable of setuptools requirement strings for the distributionsto be installed, and

• A destination directory to install to and to satisfy requirementsfrom. The destination directory can be None, in which case, no newdistributions are downloaded and there will be an error if theneeded distributions can’t be found among those already installed.

It supports a number of optional keyword arguments:

links

A sequence of URLs, file names, or directories to look forlinks to distributions.

index

The URL of an index server, or almost any other valid URL. :)

If not specified, the Python Package Index,http://pypi.python.org/simple/, is used. You can specify analternate index with this option. If you use the links option andif the links point to the needed distributions, then the index canbe anything and will be largely ignored. In the examples, here,we’ll just point to an empty directory on our link server. Thiswill make our examples run a little bit faster.

executable

A path to a Python executable. Distributions will be installedusing this executable and will be for the matching Python version.

path

A list of additional directories to search for locally-installeddistributions.

always_unzip

A flag indicating that newly-downloaded distributions should bedirectories even if they could be installed as zip files.

working_set

An existing working set to be augmented with additionaldistributions, if necessary to satisfy requirements. This allowsyou to call install multiple times, if necessary, to gathermultiple sets of requirements.

newest

A boolean value indicating whether to search for new distributionswhen already-installed distributions meet the requirement. Whenthis is true, the default, and when the destination directory isnot None, then the install function will search for the newestdistributions that satisfy the requirements.

versions

A dictionary mapping project names to version numbers to be usedwhen selecting distributions. This can be used to specify a set ofdistribution versions independent of other requirements.

use_dependency_links

A flag indicating whether to search for dependencies using thesetup dependency_links metadata or not. If true, links are searchedfor using dependency_links in preference to otherlocations. Defaults to true.

include_site_packages

A flag indicating whether Python’s non-standard-library packages shouldbe available for finding dependencies. Defaults to true.

Paths outside of Python’s standard library–or more precisely, those thatare not included when Python is started with the -S argument–are looselyreferred to as “site-packages” here.

relative_paths

Adjust egg paths so they are relative to the script path. Thisallows scripts to work when scripts and eggs are moved, as long asthey are both moved in the same way.

The install method returns a working set containing the distributionsneeded to meet the given requirements.

We have a link server that has a number of eggs:

>>> print get(link_server),<html><body><a href="bigdemo-0.1-py2.4.egg">bigdemo-0.1-py2.4.egg</a><br><a href="demo-0.1-py2.4.egg">demo-0.1-py2.4.egg</a><br><a href="demo-0.2-py2.4.egg">demo-0.2-py2.4.egg</a><br><a href="demo-0.3-py2.4.egg">demo-0.3-py2.4.egg</a><br><a href="demo-0.4c1-py2.4.egg">demo-0.4c1-py2.4.egg</a><br><a href="demoneeded-1.0.zip">demoneeded-1.0.zip</a><br><a href="demoneeded-1.1.zip">demoneeded-1.1.zip</a><br><a href="demoneeded-1.2c1.zip">demoneeded-1.2c1.zip</a><br><a href="extdemo-1.4.zip">extdemo-1.4.zip</a><br><a href="index/">index/</a><br><a href="other-1.0-py2.4.egg">other-1.0-py2.4.egg</a><br></body></html>

Let’s make a directory and install the demo egg to it, using the demo:

>>> dest = tmpdir('sample-install')>>> import zc.buildout.easy_install>>> ws = zc.buildout.easy_install.install(...     ['demo==0.2'], dest,...     links=[link_server], index=link_server+'index/')

We requested version 0.2 of the demo distribution to be installed intothe destination server. We specified that we should search for linkson the link server and that we should use the (empty) link serverindex directory as a package index.

The working set contains the distributions we retrieved.

>>> for dist in ws:...     print distdemo 0.2demoneeded 1.1

We got demoneeded because it was a dependency of demo.

And the actual eggs were added to the eggs directory.

>>> ls(dest)-  demo-0.2-py2.4.egg-  demoneeded-1.1-py2.4.egg

If we remove the version restriction on demo, but specify a falsevalue for newest, no new distributions will be installed:

>>> ws = zc.buildout.easy_install.install(...     ['demo'], dest, links=[link_server], index=link_server+'index/',...     newest=False)>>> ls(dest)-  demo-0.2-py2.4.egg-  demoneeded-1.1-py2.4.egg

If we leave off the newest option, we’ll get an update for demo:

>>> ws = zc.buildout.easy_install.install(...     ['demo'], dest, links=[link_server], index=link_server+'index/')>>> ls(dest)-  demo-0.2-py2.4.egg-  demo-0.3-py2.4.egg-  demoneeded-1.1-py2.4.egg

Note that we didn’t get the newest versions available. There wererelease candidates for newer versions of both packages. By default,final releases are preferred. We can change this behavior using theprefer_final function:

>>> zc.buildout.easy_install.prefer_final(False)True

The old setting is returned.

>>> ws = zc.buildout.easy_install.install(...     ['demo'], dest, links=[link_server], index=link_server+'index/')>>> for dist in ws:...     print distdemo 0.4c1demoneeded 1.2c1

>>> ls(dest)-  demo-0.2-py2.4.egg-  demo-0.3-py2.4.egg-  demo-0.4c1-py2.4.egg-  demoneeded-1.1-py2.4.egg-  demoneeded-1.2c1-py2.4.egg

Let’s put the setting back to the default.

>>> zc.buildout.easy_install.prefer_final(True)False

We can supply additional distributions. We can also supplyspecifications for distributions that would normally be found viadependencies. We might do this to specify a specific version.

>>> ws = zc.buildout.easy_install.install(...     ['demo', 'other', 'demoneeded==1.0'], dest,...     links=[link_server], index=link_server+'index/')

>>> for dist in ws:...     print distdemo 0.3other 1.0demoneeded 1.0

>>> ls(dest)-  demo-0.2-py2.4.egg-  demo-0.3-py2.4.egg-  demo-0.4c1-py2.4.egg-  demoneeded-1.0-py2.4.egg-  demoneeded-1.1-py2.4.egg-  demoneeded-1.2c1-py2.4.eggd  other-1.0-py2.4.egg

We can request that eggs be unzipped even if they are zip safe. Thiscan be useful when debugging. (Note that Distribute will unzip eggs bydefault, so if you are using Distribute, most or all eggs will already beunzipped without this flag.)

>>> rmdir(dest)>>> dest = tmpdir('sample-install')>>> ws = zc.buildout.easy_install.install(...     ['demo'], dest, links=[link_server], index=link_server+'index/',...     always_unzip=True)

>>> ls(dest)d  demo-0.3-py2.4.eggd  demoneeded-1.1-py2.4.egg

>>> rmdir(dest)>>> dest = tmpdir('sample-install')>>> ws = zc.buildout.easy_install.install(...     ['demo'], dest, links=[link_server], index=link_server+'index/',...     always_unzip=False)

>>> ls(dest)-  demo-0.3-py2.4.egg-  demoneeded-1.1-py2.4.egg

We can also set a default by calling the always_unzip function:

>>> zc.buildout.easy_install.always_unzip(True)False

The old default is returned:

>>> rmdir(dest)>>> dest = tmpdir('sample-install')>>> ws = zc.buildout.easy_install.install(...     ['demo'], dest, links=[link_server], index=link_server+'index/')

>>> ls(dest)d  demo-0.3-py2.4.eggd  demoneeded-1.1-py2.4.egg

>>> zc.buildout.easy_install.always_unzip(False)True

>>> rmdir(dest)>>> dest = tmpdir('sample-install')>>> ws = zc.buildout.easy_install.install(...     ['demo'], dest, links=[link_server], index=link_server+'index/')

>>> ls(dest)-  demo-0.3-py2.4.egg-  demoneeded-1.1-py2.4.egg

>>> rmdir(dest)>>> dest = tmpdir('sample-install')>>> ws = zc.buildout.easy_install.install(...     ['demo'], dest, links=[link_server], index=link_server+'index/',...     always_unzip=True)

>>> ls(dest)d  demo-0.3-py2.4.eggd  demoneeded-1.1-py2.4.egg

Specifying version information independent of requirements

Sometimes it’s useful to specify version information independent ofnormal requirements specifications. For example, a buildout may needto lock down a set of versions, without having to put put versionnumbers in setup files or part definitions. If a dictionary is passedto the install function, mapping project names to version numbers,then the versions numbers will be used.

>>> ws = zc.buildout.easy_install.install(...     ['demo'], dest, links=[link_server], index=link_server+'index/',...     versions = dict(demo='0.2', demoneeded='1.0'))>>> [d.version for d in ws]['0.2', '1.0']

In this example, we specified a version for demoneeded, even though wedidn’t define a requirement for it. The versions specified apply todependencies as well as the specified requirements.

If we specify a version that’s incompatible with a requirement, thenwe’ll get an error:

>>> from zope.testing.loggingsupport import InstalledHandler>>> handler = InstalledHandler('zc.buildout.easy_install')>>> import logging>>> logging.getLogger('zc.buildout.easy_install').propagate = False

>>> ws = zc.buildout.easy_install.install(...     ['demo >0.2'], dest, links=[link_server],...     index=link_server+'index/',...     versions = dict(demo='0.2', demoneeded='1.0'))Traceback (most recent call last):...IncompatibleVersionError: Bad version 0.2

>>> print handlerzc.buildout.easy_install DEBUG  Installing 'demo >0.2'.zc.buildout.easy_install ERROR  The version, 0.2, is not consistent with the requirement, 'demo>0.2'.

>>> handler.clear()

If no versions are specified, a debugging message will be outputreporting that a version was picked automatically:

>>> ws = zc.buildout.easy_install.install(...     ['demo'], dest, links=[link_server], index=link_server+'index/',...     )

>>> print handlerzc.buildout.easy_install DEBUG  Installing 'demo'.zc.buildout.easy_install DEBUG  We have the best distribution that satisfies 'demo'.zc.buildout.easy_install DEBUG  Picked: demo = 0.3zc.buildout.easy_install DEBUG  Getting required 'demoneeded'zc.buildout.easy_install DEBUG    required by demo 0.3.zc.buildout.easy_install DEBUG  We have the best distribution that satisfies 'demoneeded'.zc.buildout.easy_install DEBUG  Picked: demoneeded = 1.1

>>> handler.uninstall()>>> logging.getLogger('zc.buildout.easy_install').propagate = True

We can request that we get an error if versions are picked:

>>> zc.buildout.easy_install.allow_picked_versions(False)True

(The old setting is returned.)

>>> ws = zc.buildout.easy_install.install(...     ['demo'], dest, links=[link_server], index=link_server+'index/',...     )Traceback (most recent call last):...UserError: Picked: demo = 0.3

>>> zc.buildout.easy_install.allow_picked_versions(True)False

The function default_versions can be used to get and set defaultversion information to be used when no version information is passes.If called with an argument, it sets the default versions:

>>> zc.buildout.easy_install.default_versions(dict(demoneeded='1')){}

It always returns the previous default versions. If called without anargument, it simply returns the default versions without changingthem:

>>> zc.buildout.easy_install.default_versions(){'demoneeded': '1'}

So with the default versions set, we’ll get the requested version evenif the versions option isn’t used:

>>> ws = zc.buildout.easy_install.install(...     ['demo'], dest, links=[link_server], index=link_server+'index/',...     )

>>> [d.version for d in ws]['0.3', '1.0']

Of course, we can unset the default versions by passing an emptydictionary:

>>> zc.buildout.easy_install.default_versions({}){'demoneeded': '1'}

>>> ws = zc.buildout.easy_install.install(...     ['demo'], dest, links=[link_server], index=link_server+'index/',...     )

>>> [d.version for d in ws]['0.3', '1.1']

Dependencies in Site Packages

Paths outside of Python’s standard library–or more precisely, those that arenot included when Python is started with the -S argument–are loosely referredto as “site-packages” here. These site-packages are searched by default fordistributions. This can be disabled, so that, for instance, a system Pythoncan be used with buildout, cleaned of any packages installed by a user orsystem package manager.

The default behavior can be controlled and introspected usingzc.buildout.easy_install.include_site_packages.

>>> zc.buildout.easy_install.include_site_packages()True

Here’s an example of using a Python executable that includes our dependencies.

Our “py_path” will have the “demoneeded,” and “demo” packages available.

We’ll simply be asking for “demoneeded” here, but without any externalindex or links.

>>> from zc.buildout.tests import create_sample_sys_install>>> py_path, site_packages_path = make_py()>>> create_sample_sys_install(site_packages_path)

>>> example_dest = tmpdir('site-packages-example-install')>>> workingset = zc.buildout.easy_install.install(...     ['demoneeded'], example_dest, links=[], executable=py_path,...     index=None)>>> [dist.project_name for dist in workingset]['demoneeded']

That worked fine. Let’s try again with site packages not allowed. We’llchange the policy by changing the default. Notice that the function forchanging the default value returns the previous value.

>>> zc.buildout.easy_install.include_site_packages(False)True

>>> zc.buildout.easy_install.include_site_packages()False

>>> zc.buildout.easy_install.clear_index_cache()>>> rmdir(example_dest)>>> example_dest = tmpdir('site-packages-example-install')>>> workingset = zc.buildout.easy_install.install(...     ['demoneeded'], example_dest, links=[], executable=py_path,...     index=None)Traceback (most recent call last):    ...MissingDistribution: Couldn't find a distribution for 'demoneeded'.>>> zc.buildout.easy_install.clear_index_cache()

Now we’ll reset the default.

>>> zc.buildout.easy_install.include_site_packages(True)False

>>> zc.buildout.easy_install.include_site_packages()True

Dependency links

Setuptools allows metadata that describes where to search for packagedependencies. This option is called dependency_links. Buildout has itsown notion of where to look for dependencies, but it also uses thesetup tools dependency_links information if it’s available.

Let’s demo this by creating an egg that specifies dependency_links.

To begin, let’s create a new egg repository. This repository hold anewer version of the ‘demoneeded’ egg than the sample repository does.

>>> repoloc = tmpdir('repo')>>> from zc.buildout.tests import create_egg>>> create_egg('demoneeded', '1.2', repoloc)>>> link_server2 = start_server(repoloc)

Turn on logging on this server so that we can see when eggs are pulledfrom it.

>>> get(link_server2 + 'enable_server_logging')GET 200 /enable_server_logging''

Now we can create an egg that specifies that its dependencies arefound on this server.

>>> repoloc = tmpdir('repo2')>>> create_egg('hasdeps', '1.0', repoloc,...            install_requires = "'demoneeded'",...            dependency_links = [link_server2])

Let’s add the egg to another repository.

>>> link_server3 = start_server(repoloc)

Now let’s install the egg.

>>> example_dest = tmpdir('example-install')>>> workingset = zc.buildout.easy_install.install(...     ['hasdeps'], example_dest,...     links=[link_server3], index=link_server3+'index/')GET 200 /GET 200 /demoneeded-1.2-pyN.N.egg

The server logs show that the dependency was retrieved from the serverspecified in the dependency_links.

Now let’s see what happens if we provide two different ways to retrievethe dependencies.

>>> rmdir(example_dest)>>> example_dest = tmpdir('example-install')>>> workingset = zc.buildout.easy_install.install(...     ['hasdeps'], example_dest, index=link_server+'index/',...     links=[link_server, link_server3])GET 200 /GET 200 /demoneeded-1.2-pyN.N.egg

Once again the dependency is fetched from the logging server eventhough it is also available from the non-logging server. This isbecause the version on the logging server is newer and buildoutnormally chooses the newest egg available.

If you wish to control where dependencies come from regardless ofdependency_links setup metadata use the ‘use_dependency_links’ optionto zc.buildout.easy_install.install().

>>> rmdir(example_dest)>>> example_dest = tmpdir('example-install')>>> workingset = zc.buildout.easy_install.install(...     ['hasdeps'], example_dest, index=link_server+'index/',...     links=[link_server, link_server3],...     use_dependency_links=False)

Notice that this time the dependency egg is not fetched from thelogging server. When you specify not to use dependency_links, eggswill only be searched for using the links you explicitly provide.

Another way to control this option is with thezc.buildout.easy_install.use_dependency_links() function. Thisfunction sets the default behavior for the zc.buildout.easy_install()function.

>>> zc.buildout.easy_install.use_dependency_links(False)True

The function returns its previous setting.

>>> rmdir(example_dest)>>> example_dest = tmpdir('example-install')>>> workingset = zc.buildout.easy_install.install(...     ['hasdeps'], example_dest, index=link_server+'index/',...     links=[link_server, link_server3])

It can be overridden by passing a keyword argument to the installfunction.

>>> rmdir(example_dest)>>> example_dest = tmpdir('example-install')>>> workingset = zc.buildout.easy_install.install(...     ['hasdeps'], example_dest, index=link_server+'index/',...     links=[link_server, link_server3],...     use_dependency_links=True)GET 200 /demoneeded-1.2-pyN.N.egg

To return the dependency_links behavior to normal call the function again.

>>> zc.buildout.easy_install.use_dependency_links(True)False>>> rmdir(example_dest)>>> example_dest = tmpdir('example-install')>>> workingset = zc.buildout.easy_install.install(...     ['hasdeps'], example_dest, index=link_server+'index/',...     links=[link_server, link_server3])GET 200 /demoneeded-1.2-pyN.N.egg

Script generation

The easy_install module provides support for creating scripts from eggs.It provides two competing functions. One,scripts, is awell-established approach to generating reliable scripts with a “clean”Python–e.g., one that does not have any packages in its site-packages.The other,sitepackage_safe_scripts, is newer, a bit trickier, and isdesigned to work with a Python that has code in its site-packages, suchas a system Python.

Both are similar to setuptools except that they provides facilities forbaking a script’s path into the script. This has two advantages:

• The eggs to be used by a script are not chosen at run time, makingstartup faster and, more importantly, deterministic.

• The script doesn’t have to import pkg_resources because the logic thatpkg_resources would execute at run time is executed at script-creationtime. (There is an exception insitepackage_safe_scripts if youwant to have your Python’s site packages available, as discussedbelow, but even in that case pkg_resources is only partiallyactivated, which can be a significant time savings.)

Handling custom build options for extensions provided in source distributions

Sometimes, we need to control how extension modules are built. Thebuild function provides this level of control. It takes a singlepackage specification, downloads a source distribution, and builds itwith specified custom build options.

The build function takes 3 positional arguments:

spec

A package specification for a source distribution

dest

A destination directory

build_ext

A dictionary of options to be passed to the distutils build_extcommand when building extensions.

It supports a number of optional keyword arguments:

links

a sequence of URLs, file names, or directories to look forlinks to distributions,

index

The URL of an index server, or almost any other valid URL. :)

If not specified, the Python Package Index,http://pypi.python.org/simple/, is used. You can specify analternate index with this option. If you use the links option andif the links point to the needed distributions, then the index canbe anything and will be largely ignored. In the examples, here,we’ll just point to an empty directory on our link server. Thiswill make our examples run a little bit faster.

executable

A path to a Python executable. Distributions will be installedusing this executable and will be for the matching Python version.

path

A list of additional directories to search for locally-installeddistributions.

newest

A boolean value indicating whether to search for new distributionswhen already-installed distributions meet the requirement. Whenthis is true, the default, and when the destination directory isnot None, then the install function will search for the newestdistributions that satisfy the requirements.

versions

A dictionary mapping project names to version numbers to be usedwhen selecting distributions. This can be used to specify a set ofdistribution versions independent of other requirements.

Our link server included a source distribution that includes a simpleextension, extdemo.c:

#include <Python.h>#include <extdemo.h>static PyMethodDef methods[] = {};PyMODINIT_FUNCinitextdemo(void){    PyObject *m;    m = Py_InitModule3("extdemo", methods, "");#ifdef TWO    PyModule_AddObject(m, "val", PyInt_FromLong(2));#else    PyModule_AddObject(m, "val", PyInt_FromLong(EXTDEMO));#endif}

The extension depends on a system-dependent include file, extdemo.h,that defines a constant, EXTDEMO, that is exposed by the extension.

We’ll add an include directory to our sample buildout and add theneeded include file to it:

>>> mkdir('include')>>> write('include', 'extdemo.h',... """... #define EXTDEMO 42... """)

Now, we can use the build function to create an egg from the sourcedistribution:

>>> zc.buildout.easy_install.build(...   'extdemo', dest,...   {'include-dirs': os.path.join(sample_buildout, 'include')},...   links=[link_server], index=link_server+'index/')['/sample-install/extdemo-1.4-py2.4-unix-i686.egg']

The function returns the list of eggs

Now if we look in our destination directory, we see we have an extdemo egg:

>>> ls(dest)-  demo-0.2-py2.4.eggd  demo-0.3-py2.4.egg-  demoneeded-1.0-py2.4.eggd  demoneeded-1.1-py2.4.eggd  extdemo-1.4-py2.4-unix-i686.egg

Let’s update our link server with a new version of extdemo:

>>> update_extdemo()>>> print get(link_server),<html><body><a href="bigdemo-0.1-py2.4.egg">bigdemo-0.1-py2.4.egg</a><br><a href="demo-0.1-py2.4.egg">demo-0.1-py2.4.egg</a><br><a href="demo-0.2-py2.4.egg">demo-0.2-py2.4.egg</a><br><a href="demo-0.3-py2.4.egg">demo-0.3-py2.4.egg</a><br><a href="demo-0.4c1-py2.4.egg">demo-0.4c1-py2.4.egg</a><br><a href="demoneeded-1.0.zip">demoneeded-1.0.zip</a><br><a href="demoneeded-1.1.zip">demoneeded-1.1.zip</a><br><a href="demoneeded-1.2c1.zip">demoneeded-1.2c1.zip</a><br><a href="extdemo-1.4.zip">extdemo-1.4.zip</a><br><a href="extdemo-1.5.zip">extdemo-1.5.zip</a><br><a href="index/">index/</a><br><a href="other-1.0-py2.4.egg">other-1.0-py2.4.egg</a><br></body></html>

The easy_install caches information about servers to reduce networkaccess. To see the update, we have to call the clear_index_cachefunction to clear the index cache:

>>> zc.buildout.easy_install.clear_index_cache()

If we run build with newest set to False, we won’t get an update:

>>> zc.buildout.easy_install.build(...   'extdemo', dest,...   {'include-dirs': os.path.join(sample_buildout, 'include')},...   links=[link_server], index=link_server+'index/',...   newest=False)['/sample-install/extdemo-1.4-py2.4-linux-i686.egg']

>>> ls(dest)-  demo-0.2-py2.4.eggd  demo-0.3-py2.4.egg-  demoneeded-1.0-py2.4.eggd  demoneeded-1.1-py2.4.eggd  extdemo-1.4-py2.4-unix-i686.egg

But if we run it with the default True setting for newest, then we’llget an updated egg:

>>> zc.buildout.easy_install.build(...   'extdemo', dest,...   {'include-dirs': os.path.join(sample_buildout, 'include')},...   links=[link_server], index=link_server+'index/')['/sample-install/extdemo-1.5-py2.4-unix-i686.egg']

>>> ls(dest)-  demo-0.2-py2.4.eggd  demo-0.3-py2.4.egg-  demoneeded-1.0-py2.4.eggd  demoneeded-1.1-py2.4.eggd  extdemo-1.4-py2.4-unix-i686.eggd  extdemo-1.5-py2.4-unix-i686.egg

The versions option also influences the versions used. For example,if we specify a version for extdemo, then that will be used, eventhough it isn’t the newest. Let’s clean out the destination directoryfirst:

>>> import os>>> for name in os.listdir(dest):...     remove(dest, name)

>>> zc.buildout.easy_install.build(...   'extdemo', dest,...   {'include-dirs': os.path.join(sample_buildout, 'include')},...   links=[link_server], index=link_server+'index/',...   versions=dict(extdemo='1.4'))['/sample-install/extdemo-1.4-py2.4-unix-i686.egg']

>>> ls(dest)d  extdemo-1.4-py2.4-unix-i686.egg

Handling custom build options for extensions in develop eggs

The develop function is similar to the build function, except that,rather than building an egg from a source directory containing asetup.py script.

The develop function takes 2 positional arguments:

setup

The path to a setup script, typically named “setup.py”, or adirectory containing a setup.py script.

dest

The directory to install the egg link to

It supports some optional keyword argument:

build_ext

A dictionary of options to be passed to the distutils build_extcommand when building extensions.

executable

A path to a Python executable. Distributions will be installedusing this executable and will be for the matching Python version.

We have a local directory containing the extdemo source:

>>> ls(extdemo)-  MANIFEST-  MANIFEST.in-  README-  extdemo.c-  setup.py

Now, we can use the develop function to create a develop egg from the sourcedistribution:

>>> zc.buildout.easy_install.develop(...   extdemo, dest,...   {'include-dirs': os.path.join(sample_buildout, 'include')})'/sample-install/extdemo.egg-link'

The name of the egg link created is returned.

Now if we look in our destination directory, we see we have an extdemoegg link:

>>> ls(dest)d  extdemo-1.4-py2.4-unix-i686.egg-  extdemo.egg-link

And that the source directory contains the compiled extension:

>>> ls(extdemo)-  MANIFEST-  MANIFEST.in-  READMEd  build-  extdemo.cd  extdemo.egg-info-  extdemo.so-  setup.py

Download cache

Normally, when distributions are installed, if any processing isneeded, they are downloaded from the internet to a temporary directoryand then installed from there. A download cache can be used to avoidthe download step. This can be useful to reduce network access and tocreate source distributions of an entire buildout.

A download cache is specified by calling the download_cachefunction. The function always returns the previous setting. If noargument is passed, then the setting is unchanged. If an argument ispassed, the download cache is set to the given path, which must pointto an existing directory. Passing None clears the cache setting.

To see this work, we’ll create a directory and set it as the cachedirectory:

>>> cache = tmpdir('cache')>>> zc.buildout.easy_install.download_cache(cache)

We’ll recreate our destination directory:

>>> remove(dest)>>> dest = tmpdir('sample-install')

We’d like to see what is being fetched from the server, so we’llenable server logging:

>>> get(link_server+'enable_server_logging')GET 200 /enable_server_logging''

Now, if we install demo, and extdemo:

>>> ws = zc.buildout.easy_install.install(...     ['demo==0.2'], dest,...     links=[link_server], index=link_server+'index/',...     always_unzip=True)GET 200 /GET 404 /index/demo/GET 200 /index/GET 200 /demo-0.2-py2.4.eggGET 404 /index/demoneeded/GET 200 /demoneeded-1.1.zip

>>> zc.buildout.easy_install.build(...   'extdemo', dest,...   {'include-dirs': os.path.join(sample_buildout, 'include')},...   links=[link_server], index=link_server+'index/')GET 404 /index/extdemo/GET 200 /extdemo-1.5.zip['/sample-install/extdemo-1.5-py2.4-linux-i686.egg']

Not only will we get eggs in our destination directory:

>>> ls(dest)d  demo-0.2-py2.4.eggd  demoneeded-1.1-py2.4.eggd  extdemo-1.5-py2.4-linux-i686.egg

But we’ll get distributions in the cache directory:

>>> ls(cache)-  demo-0.2-py2.4.egg-  demoneeded-1.1.zip-  extdemo-1.5.zip

The cache directory contains uninstalled distributions, such as zippedeggs or source distributions.

Let’s recreate our destination directory and clear the index cache:

>>> remove(dest)>>> dest = tmpdir('sample-install')>>> zc.buildout.easy_install.clear_index_cache()

Now when we install the distributions:

>>> ws = zc.buildout.easy_install.install(...     ['demo==0.2'], dest,...     links=[link_server], index=link_server+'index/',...     always_unzip=True)GET 200 /GET 404 /index/demo/GET 200 /index/GET 404 /index/demoneeded/

>>> zc.buildout.easy_install.build(...   'extdemo', dest,...   {'include-dirs': os.path.join(sample_buildout, 'include')},...   links=[link_server], index=link_server+'index/')GET 404 /index/extdemo/['/sample-install/extdemo-1.5-py2.4-linux-i686.egg']

>>> ls(dest)d  demo-0.2-py2.4.eggd  demoneeded-1.1-py2.4.eggd  extdemo-1.5-py2.4-linux-i686.egg

Note that we didn’t download the distributions from the link server.

If we remove the restriction on demo, we’ll download a newer versionfrom the link server:

>>> ws = zc.buildout.easy_install.install(...     ['demo'], dest,...     links=[link_server], index=link_server+'index/',...     always_unzip=True)GET 200 /demo-0.3-py2.4.egg

Normally, the download cache is the preferred source of downloads, butnot the only one.

Installing solely from a download cache

A download cache can be used as the basis of application sourcereleases. In an application source release, we want to distribute anapplication that can be built without making any network accesses. Inthis case, we distribute a download cache and tell the easy_installmodule to install from the download cache only, without making networkaccesses. The install_from_cache function can be used to signal thatpackages should be installed only from the download cache. Thefunction always returns the previous setting. Calling it with noarguments returns the current setting without changing it:

>>> zc.buildout.easy_install.install_from_cache()False

Calling it with a boolean value changes the setting and returns theprevious setting:

>>> zc.buildout.easy_install.install_from_cache(True)False

Let’s remove demo-0.3-py2.4.egg from the cache, clear the index cache,recreate the destination directory, and reinstall demo:

>>> for  f in os.listdir(cache):...     if f.startswith('demo-0.3-'):...         remove(cache, f)

>>> zc.buildout.easy_install.clear_index_cache()>>> remove(dest)>>> dest = tmpdir('sample-install')

>>> ws = zc.buildout.easy_install.install(...     ['demo'], dest,...     links=[link_server], index=link_server+'index/',...     always_unzip=True)

>>> ls(dest)d  demo-0.2-py2.4.eggd  demoneeded-1.1-py2.4.egg

This time, we didn’t download from or even query the link server.

Feature Changes

• Added a configuration option that allows buildouts to ignoredependency_links metadata specified in setup. By defaultdependency_links in setup are used in addition to buildout specifiedfind-links. This can make it hard to control where eggs comefrom. Here’s how to tell buildout to ignore URLs independency_links:

  [buildout]use-dependency-links = false

  By default use-dependency-links is true, which matches the behaviorof previous versions of buildout.

• Added a configuration option that causes buildout to error if aversion is picked. This is a nice safety belt when fixing allversions is intended, especially when creating releases.

Bugs Fixed

• 151820: Develop failed if the setup.py script imported modules inthe distribution directory.

• Verbose logging of the develop command was omitting detailedoutput.

• The setup command wasn’t documented.

• The setup command failed if run in a directory without specifying aconfiguration file.

• The setup command raised a stupid exception if run without arguments.

• When using a local find links or index, distributions weren’t copiedto the download cache.

• When installing from source releases, a version specification (via abuildout versions section) for setuptools was ignored when decidingwhich setuptools to use to build an egg from the source release.

Feature Changes

• Now, final distributions are prefered over non-final versions. Ifboth final and non-final versions satisfy a requirement, then thefinal version will be used even if it is older. The normal way tooverride this for specific packages is to specifically require anon-final version, either specifically or via a lower bound.

• There is a buildout prefer-final version that can be used with avalue of “false”:

  prefer-final = false

  To prefer newer versions, regardless of whether or not they arefinal, buildout-wide.

• The new simple Python index,http://cheeseshop.python.org/simple, isused as the default index. This will provide better performancethan the human package index interface,http://pypi.python.org/pypi. More importantly, it lists hiddendistributions, so buildouts with fixed distribution versions will beable to find old distributions even if the distributions have beenhidden in the human PyPI interface.

Bugs Fixed

• 126441: Look for default.cfg in the right place on Windows.

Feature Changes

• buildout now changes to the buildout directory before running recipeinstall and update methods.

• Added a new init command for creating a new buildout. This createsan empty configuration file and then bootstraps.

• Except when using the new init command, it is now an error to runbuildout without a configuration file.

• In verbose mode, when adding distributions to fulful requirements ofalready-added distributions, we now show why the new distributionsare being added.

• Changed the logging format to exclude the logger name for thezc.buildout logger. This reduces noise in the output.

• Clean up lots of messages, adding missing periods and adding quotes aroundrequirement strings and file paths.

Bugs Fixed

• 114614: Buildouts could take a very long time if there weredependency problems in large sets of pathologically interdependentpackages.

• 59270: Buggy recipes can cause failures in later recipes via chdir

• 61890:file:// urls don’t seem to work in find-links

  setuptools requires that file urls that point to directories mustend in a “/”. Added a workaround.

• 75607: buildout should not run if it creates an empty buildout.cfg

Feature Changes

• Improved error reporting by showing which packages require otherpackages that can’t be found or that cause version conflicts.

• Added an API for use by recipe writers to clean up created fileswhen recipe errors occur.

• Log installed scripts.

Bugs Fixed

• 92891: bootstrap crashes with recipe option in buildout section.

• 113085: Buildout exited with a zero exist status when internal errorsoccurred.

Feature Changes

• Added support for download caches. A buildout can specify a cachefor distribution downloads. The cache can be shared among buildoutsto reduce network access and to support creating sourcedistributions for applications allowing install without networkaccess.

• Log scripts created, as suggested in:https://bugs.launchpad.net/zc.buildout/+bug/71353

Bugs Fixed

• It wasn’t possible to give options on the command line for sectionsnot defined in a configuration file.

Feature Changes

• Added a buildout newest option, to control whether the newestdistributions should be sought to meet requirements. This mightalso provide a hint to recipes that don’t deal withdistributions. For example, a recipe that manages subversioncheckouts might not update a checkout if newest is set to “false”.

• Added anewest keyword parameter to thezc.buildout.easy_install.install and zc.buildout.easy_install.buildfunctions to control whether the newest distributions that meedgiven requirements should be sought. If a false value is providedfor this parameter and already installed eggs meet the givenrequirements, then no attempt will be made to search for newerdistributions.

• The recipe-testing support setUp function now adds the namebuildout to the test namespace with a value that is the path tothe buildout script in the sample buildout. This allows tests touse

  >>> print system(buildout),

  rather than:

  >>> print system(join('bin', 'buildout')),

Bugs Fixed

• Paths returned from update methods replaced lists of installed filesrather than augmenting them.

Feature Changes

• Added documentation for some previously undocumented features of theeasy_install APIs.

• By popular demand, added a -o command-line option that is a shorthand for the assignment buildout:offline=true.

Bugs Fixed

• When deciding whether recipe develop eggs had changed, buildoutincorrectly considered files in .svn and CVS directories.

Feature Changes

• Configuration files can now be loaded from URLs.

Bugs Fixed

• https://bugs.launchpad.net/products/zc.buildout/+bug/71246

  Buildout extensions installed as eggs couldn’t be loaded in offlinemode.

Feature Changes

• A new command-line argument, -U, suppresses reading user defaults.

• You can now suppress use of an installed-part database(e.g. .installed.cfg) by sprifying an empty value for the buildoutinstalled option.

Bugs Fixed

• When the install command is used with a list of parts, onlythose parts are supposed to be installed, but the buildout was alsobuilding parts that those parts depended on.

Feature Changes

• Added uninstall recipes for dealing with complex uninstallationscenarios.

Bugs Fixed

• Automatic upgrades weren’t performed on Windows due to a bug thatcaused buildout to incorrectly determine that it wasn’t runninglocally in a buildout.

• Fixed some spurious test failures on Windows.

Feature Changes

• Variable substitutions now reflect option data written by recipes.

• A part referenced by a part in a parts list is now added to the partslist before the referencing part. This means that you can omitparts from the parts list if they are referenced by other parts.

• Added a develop function to the easy_install module to aid increating develop eggs with custom build_ext options.

• The build and develop functions in the easy_install module nowreturn the path of the egg or egg link created.

• Removed the limitation that parts named in the install command canonly name configured parts.

• Removed support ConfigParser-style variable substitutions(e.g. %(foo)s). Only the string-template style of variable(e.g. ${section:option}) substitutions will be supported.Supporting both violates “there’s only one way to do it”.

• Deprecated the buildout-section extendedBy option.

Bugs Fixed

• We treat setuptools as a dependency of any distribution that(declares that it) uses namespace packages, whether it declaressetuptools as a dependency or not. This wasn’t working for eggsintalled by virtue of being dependencies.

Feature Changes

• Renamed the runsetup command to setup. (The old name still works.)

• Added a recipe update method. Now install is only called when a partis installed for the first time, or after an uninstall. Otherwise,update is called. For backward compatibility, recipes that don’tdefine update methiods are still supported.

• If a distribution defines namespace packages but fails to declaresetuptools as one of its dependencies, we now treat setuptools as animplicit dependency. We generate a warning if the distributionis a develop egg.

• You can now create develop eggs for setup scripts that don’t use setuptools.

Bugs Fixed

• Egg links weren’t removed when corresponding entries were removedfrom develop sections.

• Running a non-local buildout command (one not installed in thebuildout) ket to a hang if there were new versions of zc.buildout orsetuptools were available. Now we issue a warning and don’tupgrade.

• When installing zip-safe eggs from local directories, the eggs weremoved, rather than copied, removing them from the source directory.