Recipes

A part is created by a recipe. Recipes are always installed as Pythoneggs. They can be downloaded from a package server, such as thePython Package Index, or they can be developed as part of a projectusing a “develop” egg.

A develop egg is a special kind of egg that gets installed as an “egglink” that contains the name of a source directory. Develop eggsdon’t have to be packaged for distribution to be used and can bemodified in place, which is especially useful while they are beingdeveloped.

Let’s create a recipe as part of the sample project. We’ll create arecipe for creating directories. First, we’ll create a recipes sourcedirectory for our local recipes:

>>> mkdir(sample_buildout, 'recipes')

and then we’ll create a source file for our mkdir recipe:

>>> write(sample_buildout, 'recipes', 'mkdir.py',... """... import logging, os, zc.buildout...... class Mkdir:......     def __init__(self, buildout, name, options):...         self.name, self.options = name, options...         options['path'] = os.path.join(...                               buildout['buildout']['directory'],...                               options['path'],...                               )...         if not os.path.isdir(os.path.dirname(options['path'])):...             logging.getLogger(self.name).error(...                 'Cannot create %s. %s is not a directory.',...                 options['path'], os.path.dirname(options['path']))...             raise zc.buildout.UserError('Invalid Path').........     def install(self):...         path = self.options['path']...         logging.getLogger(self.name).info(...             'Creating directory %s', os.path.basename(path))...         os.mkdir(path)...         return path......     def update(self):...         pass... """)

Currently, recipes must define 3 methods:

• a constructor,

• an install method, and

• an update method.

The constructor is responsible for updating a parts options to reflectdata read from other sections. The buildout system keeps track ofwhether a part specification has changed. A part specification haschanged if it’s options, after adjusting for data read from othersections, has changed, or if the recipe has changed. Only the optionsfor the part are considered. If data are read from other sections,then that information has to be reflected in the parts options. Inthe Mkdir example, the given path is interpreted relative to thebuildout directory, and data from the buildout directory is read. Thepath option is updated to reflect this. If the directory option waschanged in the buildout sections, we would know to update partscreated using the mkdir recipe using relative path names.

When buildout is run, it saves configuration data for installed partsin a file named “.installed.cfg”. In subsequent runs, it comparespart-configuration data stored in the .installed.cfg file and thepart-configuration data loaded from the configuration files asmodified by recipe constructors to decide if the configuration of apart has changed. If the configuration has changed, or if the recipehas changed, then the part is uninstalled and reinstalled. Thebuildout only looks at the part’s options, so any data used toconfigure the part needs to be reflected in the part’s options. It isthe job of a recipe constructor to make sure that the options includeall relevant data.

Of course, parts are also uninstalled if they are no-longer used.

The recipe defines a constructor that takes a buildout object, a partname, and an options dictionary. It saves them in instance attributes.If the path is relative, we’ll interpret it as relative to thebuildout directory. The buildout object passed in is a mapping fromsection name to a mapping of options for that section. The buildoutdirectory is available as the directory option of the buildoutsection. We normalize the path and save it back into the optionsdirectory.

The install method is responsible for creating the part. In thiscase, we need the path of the directory to create. We’ll use a pathoption from our options dictionary. The install method logs what it’sdoing using the Python logging call. We return the path that weinstalled. If the part is uninstalled or reinstalled, then the pathreturned will be removed by the buildout machinery. A recipe installmethod is expected to return a string, or an iterable of stringscontaining paths to be removed if a part is uninstalled. For mostrecipes, this is all of the uninstall support needed. For more complexuninstallation scenarios useUninstall recipes.

The update method is responsible for updating an already installedpart. An empty method is often provided, as in this example, if partscan’t be updated. An update method can return None, a string, or aniterable of strings. If a string or iterable of strings is returned,then the saved list of paths to be uninstalled is updated with the newinformation by adding any new files returned by the update method.

We need to provide packaging information so that our recipe can beinstalled as a develop egg. The minimum information we need to specifyis a name. For recipes, we also need to define thenames of the recipe classes as entry points. Packaging information isprovided via a setup.py script:

>>> write(sample_buildout, 'recipes', 'setup.py',... """... from setuptools import setup...... setup(...     name = "recipes",...     entry_points = {'zc.buildout': ['mkdir = mkdir:Mkdir']},...     )... """)

Our setup script defines an entry point. Entry points providea way for an egg to define the services it provides. Here we’ve saidthat we define a zc.buildout entry point named mkdir. Recipeclasses must be exposed as entry points in the zc.buildout group. wegive entry points names within the group.

We also need a README.txt for our recipes to avoid an annoying warningfrom distutils, on which distribute and zc.buildout are based:

>>> write(sample_buildout, 'recipes', 'README.txt', " ")

Now let’s update our buildout.cfg:

>>> write(sample_buildout, 'buildout.cfg',... """... [buildout]... develop = recipes... parts = data-dir...... [data-dir]... recipe = recipes:mkdir... path = mystuff... """)

Let’s go through the changes one by one:

develop = recipes

This tells the buildout to install a development egg for our recipes.Any number of paths can be listed. The paths can be relative orabsolute. If relative, they are treated as relative to the buildoutdirectory. They can be directory or file paths. If a file path isgiven, it should point to a Python setup script. If a directory pathis given, it should point to a directory containing a setup.py file.Development eggs are installed before building any parts, as they mayprovide locally-defined recipes needed by the parts.

parts = data-dir

Here we’ve named a part to be “built”. We can use any name we wantexcept that different part names must be unique and recipes will oftenuse the part name to decide what to do.

[data-dir]recipe = recipes:mkdirpath = mystuff

When we name a part, we also create a section of the samename that contains part data. In this section, we’ll definethe recipe to be used to install the part. In this case, we alsospecify the path to be created.

Let’s run the buildout. We do so by running the build script in thebuildout:

>>> import os>>> os.chdir(sample_buildout)>>> buildout = os.path.join(sample_buildout, 'bin', 'buildout')>>> print_(system(buildout), end='')Develop: '/sample-buildout/recipes'Installing data-dir.data-dir: Creating directory mystuff

We see that the recipe created the directory, as expected:

>>> ls(sample_buildout)-  .installed.cfgd  bin-  buildout.cfgd  develop-eggsd  eggsd  mystuffd  partsd  recipes

In addition, .installed.cfg has been created containing informationabout the part we installed:

>>> cat(sample_buildout, '.installed.cfg')[buildout]installed_develop_eggs = /sample-buildout/develop-eggs/recipes.egg-linkparts = data-dir<BLANKLINE>[data-dir]__buildout_installed__ = /sample-buildout/mystuff__buildout_signature__ = recipes-c7vHV6ekIDUPy/7fjAaYjg==path = /sample-buildout/mystuffrecipe = recipes:mkdir

Note that the directory we installed is included in .installed.cfg.In addition, the path option includes the actual destinationdirectory.

If we change the name of the directory in the configuration file,we’ll see that the directory gets removed and recreated:

>>> write(sample_buildout, 'buildout.cfg',... """... [buildout]... develop = recipes... parts = data-dir...... [data-dir]... recipe = recipes:mkdir... path = mydata... """)

>>> print_(system(buildout), end='')Develop: '/sample-buildout/recipes'Uninstalling data-dir.Installing data-dir.data-dir: Creating directory mydata

>>> ls(sample_buildout)-  .installed.cfgd  bin-  buildout.cfgd  develop-eggsd  eggsd  mydatad  partsd  recipes

If any of the files or directories created by a recipe are removed,the part will be reinstalled:

>>> rmdir(sample_buildout, 'mydata')>>> print_(system(buildout), end='')Develop: '/sample-buildout/recipes'Uninstalling data-dir.Installing data-dir.data-dir: Creating directory mydata

Error reporting

If a user makes an error, an error needs to be printed and work needsto stop. This is accomplished by logging a detailed error message andthen raising a (or an instance of a subclass of a)zc.buildout.UserError exception. Raising an error other than aUserError still displays the error, but labels it as a bug in thebuildout software or recipe. In the sample above, of someone gives anon-existent directory to create the directory in:

>>> write(sample_buildout, 'buildout.cfg',... """... [buildout]... develop = recipes... parts = data-dir...... [data-dir]... recipe = recipes:mkdir... path = /xxx/mydata... """)

We’ll get a user error, not a traceback.

>>> print_(system(buildout), end='')Develop: '/sample-buildout/recipes'data-dir: Cannot create /xxx/mydata. /xxx is not a directory.While:  Installing.  Getting section data-dir.  Initializing part data-dir.Error: Invalid Path

Recipe Error Handling

If an error occurs during installation, it is up to the recipe toclean up any system side effects, such as files created. Let’s updatethe mkdir recipe to support multiple paths:

>>> write(sample_buildout, 'recipes', 'mkdir.py',... """... import logging, os, zc.buildout...... class Mkdir:......     def __init__(self, buildout, name, options):...         self.name, self.options = name, options......         # Normalize paths and check that their parent...         # directories exist:...         paths = []...         for path in options['path'].split():...             path = os.path.join(buildout['buildout']['directory'], path)...             if not os.path.isdir(os.path.dirname(path)):...                 logging.getLogger(self.name).error(...                     'Cannot create %s. %s is not a directory.',...                     options['path'], os.path.dirname(options['path']))...                 raise zc.buildout.UserError('Invalid Path')...             paths.append(path)...         options['path'] = ' '.join(paths)......     def install(self):...         paths = self.options['path'].split()...         for path in paths:...             logging.getLogger(self.name).info(...                 'Creating directory %s', os.path.basename(path))...             os.mkdir(path)...         return paths......     def update(self):...         pass... """)

>>> clean_up_pyc(sample_buildout, 'recipes', 'mkdir.py')

If there is an error creating a path, the install method will exit andleave previously created paths in place:

>>> write(sample_buildout, 'buildout.cfg',... """... [buildout]... develop = recipes... parts = data-dir...... [data-dir]... recipe = recipes:mkdir... path = foo bin... """)

>>> print_(system(buildout)) # doctest: +ELLIPSISDevelop: '/sample-buildout/recipes'Uninstalling data-dir.Installing data-dir.data-dir: Creating directory foodata-dir: Creating directory binWhile:  Installing data-dir.<BLANKLINE>An internal error occured due to a bug in either zc.buildout or in arecipe being used:Traceback (most recent call last):... exists...

We meant to create a directory bins, but typed bin. Now foo wasleft behind.

>>> os.path.exists('foo')True

If we fix the typo:

>>> write(sample_buildout, 'buildout.cfg',... """... [buildout]... develop = recipes... parts = data-dir...... [data-dir]... recipe = recipes:mkdir... path = foo bins... """)

>>> print_(system(buildout)) # doctest: +ELLIPSISDevelop: '/sample-buildout/recipes'Installing data-dir.data-dir: Creating directory fooWhile:  Installing data-dir.<BLANKLINE>An internal error occured due to a bug in either zc.buildout or in arecipe being used:Traceback (most recent call last):... exists...

Now they fail because foo exists, because it was left behind.

>>> remove('foo')

Let’s fix the recipe:

>>> write(sample_buildout, 'recipes', 'mkdir.py',... """... import logging, os, zc.buildout, sys...... class Mkdir:......     def __init__(self, buildout, name, options):...         self.name, self.options = name, options......         # Normalize paths and check that their parent...         # directories exist:...         paths = []...         for path in options['path'].split():...             path = os.path.join(buildout['buildout']['directory'], path)...             if not os.path.isdir(os.path.dirname(path)):...                 logging.getLogger(self.name).error(...                     'Cannot create %s. %s is not a directory.',...                     options['path'], os.path.dirname(options['path']))...                 raise zc.buildout.UserError('Invalid Path')...             paths.append(path)...         options['path'] = ' '.join(paths)......     def install(self):...         paths = self.options['path'].split()...         created = []...         try:...             for path in paths:...                 logging.getLogger(self.name).info(...                     'Creating directory %s', os.path.basename(path))...                 os.mkdir(path)...                 created.append(path)...         except Exception:...             for d in created:...                 os.rmdir(d)...                 assert not os.path.exists(d)...                 logging.getLogger(self.name).info(...                     'Removed %s due to error',...                      os.path.basename(d))...             sys.stderr.flush()...             sys.stdout.flush()...             raise......         return paths......     def update(self):...         pass... """)

>>> clean_up_pyc(sample_buildout, 'recipes', 'mkdir.py')

And put back the typo:

>>> write(sample_buildout, 'buildout.cfg',... """... [buildout]... develop = recipes... parts = data-dir...... [data-dir]... recipe = recipes:mkdir... path = foo bin... """)

When we rerun the buildout:

>>> print_(system(buildout)) # doctest: +ELLIPSISDevelop: '/sample-buildout/recipes'Installing data-dir.data-dir: Creating directory foodata-dir: Creating directory bindata-dir: Removed foo due to errorWhile:  Installing data-dir.<BLANKLINE>An internal error occured due to a bug in either zc.buildout or in arecipe being used:Traceback (most recent call last):... exists...

we get the same error, but we don’t get the directory left behind:

>>> os.path.exists('foo')False

It’s critical that recipes clean up partial effects when errorsoccur. Because recipes most commonly create files and directories,buildout provides a helper API for removing created files when anerror occurs. Option objects have a created method that can be calledto record files as they are created. If the install or update methodreturns with an error, then any registered paths are removedautomatically. The method returns the files registered and can beused to return the files created. Let’s use this API to simplify therecipe:

>>> write(sample_buildout, 'recipes', 'mkdir.py',... """... import logging, os, zc.buildout...... class Mkdir:......     def __init__(self, buildout, name, options):...         self.name, self.options = name, options......         # Normalize paths and check that their parent...         # directories exist:...         paths = []...         for path in options['path'].split():...             path = os.path.join(buildout['buildout']['directory'], path)...             if not os.path.isdir(os.path.dirname(path)):...                 logging.getLogger(self.name).error(...                     'Cannot create %s. %s is not a directory.',...                     options['path'], os.path.dirname(options['path']))...                 raise zc.buildout.UserError('Invalid Path')...             paths.append(path)...         options['path'] = ' '.join(paths)......     def install(self):...         paths = self.options['path'].split()...         for path in paths:...             logging.getLogger(self.name).info(...                 'Creating directory %s', os.path.basename(path))...             os.mkdir(path)...             self.options.created(path)......         return self.options.created()......     def update(self):...         pass... """)

>>> clean_up_pyc(sample_buildout, 'recipes', 'mkdir.py')

We returned by calling created, taking advantage of the fact that itreturns the registered paths. We did this for illustrative purposes.It would be simpler just to return the paths as before.

If we rerun the buildout, again, we’ll get the error and nodirectories will be created:

>>> print_(system(buildout)) # doctest: +ELLIPSISDevelop: '/sample-buildout/recipes'Installing data-dir.data-dir: Creating directory foodata-dir: Creating directory binWhile:  Installing data-dir.<BLANKLINE>An internal error occured due to a bug in either zc.buildout or in arecipe being used:Traceback (most recent call last):... exists...

>>> os.path.exists('foo')False

Now, we’ll fix the typo again and we’ll get the directories we expect:

>>> write(sample_buildout, 'buildout.cfg',... """... [buildout]... develop = recipes... parts = data-dir...... [data-dir]... recipe = recipes:mkdir... path = foo bins... """)

>>> print_(system(buildout), end='')Develop: '/sample-buildout/recipes'Installing data-dir.data-dir: Creating directory foodata-dir: Creating directory bins

>>> os.path.exists('foo')True>>> os.path.exists('bins')True

Configuration file syntax

A buildout configuration file consists of a sequence of sections. Asection has a section header followed by 0 or more section options.(Buildout configuration files may be viewed as a variation on INIfiles.)

A section header consists of a section name enclosed in square braces.A section name consists of one or more non-whitespace charactersother than square braces (‘[’, ‘]’), curly braces (‘{’, ‘}’), colons(‘:’) or equal signs (‘=’). Whitespace surrounding section names is ignored.

Options consist of option names, followed by optional space or tabcharacters, an optional plus or minus sign and an equal signs andvalues. An option value may be spread over multiple lines as long asthe lines after the first start with a whitespace character. Anoption name consists of one or more non-whitespace characters otherthan equal signs, square braces (“[”, “]”), curly braces (“{”, “}”),plus signs or colons (“:”). The option name ‘<’ is reserved. Anoption’s data consists of the characters following the equal sign onthe start line, plus the continuation lines.

Option values have extra whitespace stripped. How this is donedepends on whether the value has non-whitespace characterts on thefirst line. If an option value has non-whitespace characters on thefirst line, then each line is stripped and blank lines are removed.For exampe, in:

[foo]bar = 1baz = a      b      c

The value of ofbar is'1' and the value ofbaz is'a\nb\nc'.

If the first line of an option doesn’t contain whitespace, then thevalue is dedented (withtextwrap.dedent), trailing spaces in linesare removed, and leading and trailing blank lines are removed. Forexample, in:

[foo]bar =baz =  a    b  c

The value of bar is'', and the value of baz is'a\n b\n\nc'.

Lines starting with ‘#’ or ‘;’ characters are comments. Comments canalso be placed after the closing square bracket (‘]’) in a section header.

Buildout configuration data are Python strings, which are bytes inPython 2 and unicode in Python 3.

Sections and options within sections may be repeated. Multipleoccurrences of of a section are treated as if they were concantinated.The last option value for a given name in a section overrides previousvalues.

In addition top the syntactic details above:

• option names are case sensitive

• option values can use a substitution syntax, described below, torefer to option values in specific sections.

• option values can be appended or removed using the - and +operators.

Annotated sections

When used with theannotate command, buildout displays annotated sections.All sections are displayed, sorted alphabetically. For each section,all key-value pairs are displayed, sorted alphabetically, along withthe origin of the value (file name or COMPUTED_VALUE, DEFAULT_VALUE,COMMAND_LINE_VALUE).

>>> print_(system(buildout+ ' annotate'), end='')... # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE<BLANKLINE>Annotated sections==================<BLANKLINE>[buildout]allow-hosts= *    DEFAULT_VALUEallow-picked-versions= true    DEFAULT_VALUEbin-directory= bin    DEFAULT_VALUEdevelop= recipes    /sample-buildout/buildout.cfgdevelop-eggs-directory= develop-eggs    DEFAULT_VALUEdirectory= /sample-buildout    COMPUTED_VALUEeggs-directory= eggs    DEFAULT_VALUEexecutable= ...    DEFAULT_VALUEfind-links=    DEFAULT_VALUEinstall-from-cache= false    DEFAULT_VALUEinstalled= .installed.cfg    DEFAULT_VALUElog-format=    DEFAULT_VALUElog-level= INFO    DEFAULT_VALUEnewest= true    DEFAULT_VALUEoffline= false    DEFAULT_VALUEparts= data-dir    /sample-buildout/buildout.cfgparts-directory= parts    DEFAULT_VALUEprefer-final= true    DEFAULT_VALUEpython= buildout    DEFAULT_VALUEshow-picked-versions= false    DEFAULT_VALUEsocket-timeout=    DEFAULT_VALUEupdate-versions-file=    DEFAULT_VALUEuse-dependency-links= true    DEFAULT_VALUEversions= versions    DEFAULT_VALUE<BLANKLINE>[data-dir]path= foo bins    /sample-buildout/buildout.cfgrecipe= recipes:mkdir    /sample-buildout/buildout.cfg<BLANKLINE>[versions]zc.buildout= >=1.99    DEFAULT_VALUEzc.recipe.egg= >=1.99    DEFAULT_VALUE<BLANKLINE>

Variable substitutions

Buildout configuration files support variable substitution.To illustrate this, we’ll create an debug recipe toallow us to see interactions with the buildout:

>>> write(sample_buildout, 'recipes', 'debug.py',... """... import sys... class Debug:......     def __init__(self, buildout, name, options):...         self.buildout = buildout...         self.name = name...         self.options = options......     def install(self):...         for option, value in sorted(self.options.items()):...             sys.stdout.write('%s %s\\n' % (option, value))...         return ()......     update = install... """)

This recipe doesn’t actually create anything. The install methoddoesn’t return anything, because it didn’t create any files ordirectories.

We also have to update our setup script:

>>> write(sample_buildout, 'recipes', 'setup.py',... """... from setuptools import setup... entry_points = (... '''... [zc.buildout]... mkdir = mkdir:Mkdir... debug = debug:Debug... ''')... setup(name="recipes", entry_points=entry_points)... """)

We’ve rearranged the script a bit to make the entry points easier toedit. In particular, entry points are now defined as a configurationstring, rather than a dictionary.

Let’s update our configuration to provide variable substitutionexamples:

>>> write(sample_buildout, 'buildout.cfg',... """... [buildout]... develop = recipes... parts = data-dir debug... log-level = INFO...... [debug]... recipe = recipes:debug... File-1 = ${data-dir:path}/file... File-2 = ${debug:File-1}/log...... [data-dir]... recipe = recipes:mkdir... path = mydata... """)

We used a string-template substitution for File-1 and File-2. Thistype of substitution uses the string.Template syntax. Namessubstituted are qualified option names, consisting of a section nameand option name joined by a colon.

Now, if we run the buildout, we’ll see the options with the valuessubstituted.

>>> print_(system(buildout), end='')Develop: '/sample-buildout/recipes'Uninstalling data-dir.Installing data-dir.data-dir: Creating directory mydataInstalling debug.File-1 /sample-buildout/mydata/fileFile-2 /sample-buildout/mydata/file/logrecipe recipes:debug

Note that the substitution of the data-dir path option reflects theupdate to the option performed by the mkdir recipe.

It might seem surprising that mydata was created again. This isbecause we changed our recipes package by adding the debug module.The buildout system didn’t know if this module could effect the mkdirrecipe, so it assumed it could and reinstalled mydata. If we rerunthe buildout:

>>> print_(system(buildout), end='')Develop: '/sample-buildout/recipes'Updating data-dir.Updating debug.File-1 /sample-buildout/mydata/fileFile-2 /sample-buildout/mydata/file/logrecipe recipes:debug

We can see that mydata was not recreated.

Note that, in this case, we didn’t specify a log level, sowe didn’t get output about what the buildout was doing.

Section and option names in variable substitutions are only allowed tocontain alphanumeric characters, hyphens, periods and spaces. Thisrestriction might be relaxed in future releases.

We can ommit the section name in a variable substitution to refer tothe current section. We can also use the special option,_buildout_section_name_ to get the current section name.

>>> write(sample_buildout, 'buildout.cfg',... """... [buildout]... develop = recipes... parts = data-dir debug... log-level = INFO...... [debug]... recipe = recipes:debug... File-1 = ${data-dir:path}/file... File-2 = ${:File-1}/log... my_name = ${:_buildout_section_name_}...... [data-dir]... recipe = recipes:mkdir... path = mydata... """)

>>> print_(system(buildout), end='')Develop: '/sample-buildout/recipes'Uninstalling debug.Updating data-dir.Installing debug.File-1 /sample-buildout/mydata/fileFile-2 /sample-buildout/mydata/file/logmy_name debugrecipe recipes:debug

Automatic part selection and ordering

When a section with a recipe is referred to, either through variablesubstitution or by an initializing recipe, the section is treated as apart and added to the part list before the referencing part. Forexample, we can leave data-dir out of the parts list:

>>> write(sample_buildout, 'buildout.cfg',... """... [buildout]... develop = recipes... parts = debug... log-level = INFO...... [debug]... recipe = recipes:debug... File-1 = ${data-dir:path}/file... File-2 = ${debug:File-1}/log...... [data-dir]... recipe = recipes:mkdir... path = mydata... """)

It will still be treated as a part:

>>> print_(system(buildout), end='')Develop: '/sample-buildout/recipes'Uninstalling debug.Updating data-dir.Installing debug.File-1 /sample-buildout/mydata/fileFile-2 /sample-buildout/mydata/file/logrecipe recipes:debug

>>> cat('.installed.cfg') # doctest: +ELLIPSIS[buildout]installed_develop_eggs = /sample-buildout/develop-eggs/recipes.egg-linkparts = data-dir debug...

Note that the data-dir part is includedbefore the debug part,because the debug part refers to the data-dir part. Even if we listthe data-dir part after the debug part, it will be included before:

>>> write(sample_buildout, 'buildout.cfg',... """... [buildout]... develop = recipes... parts = debug data-dir... log-level = INFO...... [debug]... recipe = recipes:debug... File-1 = ${data-dir:path}/file... File-2 = ${debug:File-1}/log...... [data-dir]... recipe = recipes:mkdir... path = mydata... """)

It will still be treated as a part:

>>> print_(system(buildout), end='')Develop: '/sample-buildout/recipes'Updating data-dir.Updating debug.File-1 /sample-buildout/mydata/fileFile-2 /sample-buildout/mydata/file/logrecipe recipes:debug

>>> cat('.installed.cfg') # doctest: +ELLIPSIS[buildout]installed_develop_eggs = /sample-buildout/develop-eggs/recipes.egg-linkparts = data-dir debug...

Extending sections (macros)

A section (other than the buildout section) can extend one or moreother sections using the< option. Options from the referencedsections are copied to the referring sectionbefore variablesubstitution. This, together with the ability to refer to variablesof the current section allows sections to be used as macros.

>>> write(sample_buildout, 'buildout.cfg',... """... [buildout]... develop = recipes... parts = myfiles... log-level = INFO...... [debug]... recipe = recipes:debug...... [with_file1]... <= debug... file1 = ${:path}/file1... color = red...... [with_file2]... <= debug... file2 = ${:path}/file2... color = blue...... [myfiles]... <= with_file1...    with_file2... path = mydata... """)

>>> print_(system(buildout), end='')Develop: '/sample-buildout/recipes'Uninstalling debug.Uninstalling data-dir.Installing myfiles.color bluefile1 mydata/file1file2 mydata/file2path mydatarecipe recipes:debug

In this example, the debug, with_file1 and with_file2 sections act asmacros. In particular, the variable substitutions are performedrelative to the myfiles section.

Adding and removing options

We can append and remove values to an option by using the + and -operators.

This is illustrated below; first we define a base configuration.

>>> write(sample_buildout, 'base.cfg',... """... [buildout]... parts = part1 part2 part3...... [part1]... recipe =... option = a1 a2...... [part2]... recipe =... option = b1 b2 b3 b4...... [part3]... recipe =... option = c1 c2...... """)

Extending this configuration, we can “adjust” the values set in thebase configuration file.

>>> write(sample_buildout, 'extension1.cfg',... """... [buildout]... extends = base.cfg...... # appending values... [part1]... option += a3 a4...... # removing values... [part2]... option -= b1 b2...... # alt. spelling... [part3]... option+=c3 c4 c5...... # normal assignment... [part4]... option = h1 h2...... """)

An additional extension.

>>> write(sample_buildout, 'extension2.cfg',... """... [buildout]... extends = extension1.cfg...... # appending values... [part1]... option += a5...... # removing values... [part2]... option -= b1 b2 b3...... """)

To verify that the options are adjusted correctly, we’ll set up anextension that prints out the options.

>>> mkdir(sample_buildout, 'demo')>>> write(sample_buildout, 'demo', 'demo.py',... """... import sys... def ext(buildout):...     sys.stdout.write(str(...         [part['option'] for name, part in sorted(buildout.items())...          if name.startswith('part')])+'\\n')... """)

>>> write(sample_buildout, 'demo', 'setup.py',... """... from setuptools import setup...... setup(...     name="demo",...     entry_points={'zc.buildout.extension': ['ext = demo:ext']},...     )... """)

Set up a buildout configuration for this extension.

>>> write(sample_buildout, 'buildout.cfg',... """... [buildout]... develop = demo... parts =... """)

>>> os.chdir(sample_buildout)>>> print_(system(os.path.join(sample_buildout, 'bin', 'buildout')), end='')... # doctest: +ELLIPSISDevelop: '/sample-buildout/demo'...

Verify option values.

>>> write(sample_buildout, 'buildout.cfg',... """... [buildout]... develop = demo... extensions = demo... extends = extension2.cfg... """)

>>> print_(system(os.path.join('bin', 'buildout')), end='')['a1 a2/na3 a4/na5', 'b1 b2 b3 b4', 'c1 c2/nc3 c4 c5', 'h1 h2']Develop: '/sample-buildout/demo'

Annotated sections output shows which files are responsible for whichoperations.

>>> print_(system(os.path.join('bin', 'buildout') + ' annotate'), end='')... # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE<BLANKLINE>Annotated sections==================...<BLANKLINE>[part1]option= a1 a2a3 a4a5    /sample-buildout/base.cfg+=  /sample-buildout/extension1.cfg+=  /sample-buildout/extension2.cfgrecipe=    /sample-buildout/base.cfg<BLANKLINE>[part2]option= b1 b2 b3 b4    /sample-buildout/base.cfg-=  /sample-buildout/extension1.cfg-=  /sample-buildout/extension2.cfgrecipe=    /sample-buildout/base.cfg<BLANKLINE>[part3]option= c1 c2c3 c4 c5    /sample-buildout/base.cfg+=  /sample-buildout/extension1.cfgrecipe=    /sample-buildout/base.cfg<BLANKLINE>[part4]option= h1 h2    /sample-buildout/extension1.cfg[versions]zc.buildout= >=1.99    DEFAULT_VALUEzc.recipe.egg= >=1.99    DEFAULT_VALUE<BLANKLINE>

Cleanup.

>>> os.remove(os.path.join(sample_buildout, 'base.cfg'))>>> os.remove(os.path.join(sample_buildout, 'extension1.cfg'))>>> os.remove(os.path.join(sample_buildout, 'extension2.cfg'))

Multiple configuration files

A configuration file can “extend” another configuration file.Options are read from the other configuration file if they aren’talready defined by your configuration file.

The configuration files your file extends can extendother configuration files. The same file may beused more than once although, of course, cycles aren’t allowed.

To see how this works, we use an example:

>>> write(sample_buildout, 'buildout.cfg',... """... [buildout]... extends = base.cfg...... [debug]... op = buildout... """)

>>> write(sample_buildout, 'base.cfg',... """... [buildout]... develop = recipes... parts = debug...... [debug]... recipe = recipes:debug... op = base... """)

>>> print_(system(buildout), end='')Develop: '/sample-buildout/recipes'Installing debug.op buildoutrecipe recipes:debug

The example is pretty trivial, but the pattern it illustrates ispretty common. In a more practical example, the base buildout mightrepresent a product and the extending buildout might be acustomization.

Here is a more elaborate example.

>>> other = tmpdir('other')

>>> write(sample_buildout, 'buildout.cfg',... """... [buildout]... extends = b1.cfg b2.cfg %(b3)s...... [debug]... op = buildout... """ % dict(b3=os.path.join(other, 'b3.cfg')))

>>> write(sample_buildout, 'b1.cfg',... """... [buildout]... extends = base.cfg...... [debug]... op1 = b1 1... op2 = b1 2... """)

>>> write(sample_buildout, 'b2.cfg',... """... [buildout]... extends = base.cfg...... [debug]... op2 = b2 2... op3 = b2 3... """)

>>> write(other, 'b3.cfg',... """... [buildout]... extends = b3base.cfg...... [debug]... op4 = b3 4... """)

>>> write(other, 'b3base.cfg',... """... [debug]... op5 = b3base 5... """)

>>> write(sample_buildout, 'base.cfg',... """... [buildout]... develop = recipes... parts = debug...... [debug]... recipe = recipes:debug... name = base... """)

>>> print_(system(buildout), end='')Develop: '/sample-buildout/recipes'Uninstalling debug.Installing debug.name baseop buildoutop1 b1 1op2 b2 2op3 b2 3op4 b3 4op5 b3base 5recipe recipes:debug

There are several things to note about this example:

• We can name multiple files in an extends option.

• We can reference files recursively.

• Relative file names in extended options are interpreted relative tothe directory containing the referencing configuration file.

Loading Configuration from URLs

Configuration files can be loaded from URLs. To see how this works,we’ll set up a web server with some configuration files.

>>> server_data = tmpdir('server_data')

>>> write(server_data, "r1.cfg",... """... [debug]... op1 = r1 1... op2 = r1 2... """)

>>> write(server_data, "r2.cfg",... """... [buildout]... extends = r1.cfg...... [debug]... op2 = r2 2... op3 = r2 3... """)

>>> server_url = start_server(server_data)

>>> write('client.cfg',... """... [buildout]... develop = recipes... parts = debug... extends = %(url)s/r2.cfg...... [debug]... recipe = recipes:debug... name = base... """ % dict(url=server_url))

>>> print_(system(buildout+ ' -c client.cfg'), end='')Develop: '/sample-buildout/recipes'Uninstalling debug.Installing debug.name baseop1 r1 1op2 r2 2op3 r2 3recipe recipes:debug

Here we specified a URL for the file we extended. The file wedownloaded, itself referred to a file on the server using a relativeURL reference. Relative references are interpreted relative to thebase URL when they appear in configuration files loaded via URL.

We can also specify a URL as the configuration file to be used by abuildout.

>>> os.remove('client.cfg')>>> write(server_data, 'remote.cfg',... """... [buildout]... develop = recipes... parts = debug... extends = r2.cfg...... [debug]... recipe = recipes:debug... name = remote... """)

>>> print_(system(buildout + ' -c ' + server_url + '/remote.cfg'), end='')While:  Initializing.Error: Missing option: buildout:directory

Normally, the buildout directory defaults to directorycontaining a configuration file. This won’t work for configurationfiles loaded from URLs. In this case, the buildout directory wouldnormally be defined on the command line:

>>> print_(system(buildout...              + ' -c ' + server_url + '/remote.cfg'...              + ' buildout:directory=' + sample_buildout...              ), end='')Develop: '/sample-buildout/recipes'Uninstalling debug.Installing debug.name remoteop1 r1 1op2 r2 2op3 r2 3recipe recipes:debug

User defaults

If the file $HOME/.buildout/default.cfg, exists, it is read beforereading the configuration file. ($HOME is the value of the HOMEenvironment variable. The ‘/’ is replaced by the operating system filedelimiter.)

>>> old_home = os.environ['HOME']>>> home = tmpdir('home')>>> mkdir(home, '.buildout')>>> write(home, '.buildout', 'default.cfg',... """... [debug]... op1 = 1... op7 = 7... """)

>>> os.environ['HOME'] = home>>> print_(system(buildout), end='')Develop: '/sample-buildout/recipes'Uninstalling debug.Installing debug.name baseop buildoutop1 b1 1op2 b2 2op3 b2 3op4 b3 4op5 b3base 5op7 7recipe recipes:debug

A buildout command-line argument, -U, can be used to suppress readinguser defaults:

>>> print_(system(buildout + ' -U'), end='')Develop: '/sample-buildout/recipes'Uninstalling debug.Installing debug.name baseop buildoutop1 b1 1op2 b2 2op3 b2 3op4 b3 4op5 b3base 5recipe recipes:debug

>>> os.environ['HOME'] = old_home

Log level

We can control the level of logging by specifying a log level in outconfiguration file. For example, so suppress info messages, we canset the logging level to WARNING

>>> write(sample_buildout, 'buildout.cfg',... """... [buildout]... log-level = WARNING... extends = b1.cfg b2.cfg... """)

>>> print_(system(buildout), end='')name baseop1 b1 1op2 b2 2op3 b2 3recipe recipes:debug

Socket timeout

The timeout of the connections to egg and configuration servers can beconfigured in the buildout section. Its value is configured in seconds.

>>> write(sample_buildout, 'buildout.cfg',... """... [buildout]... socket-timeout = 5... develop = recipes... parts = debug...... [debug]... recipe = recipes:debug... op = timeout... """)

>>> print_(system(buildout), end='')Setting socket time out to 5 seconds.Develop: '/sample-buildout/recipes'Uninstalling debug.Installing debug.op timeoutrecipe recipes:debug

If the socket-timeout is not numeric, a warning is issued and the defaulttimeout of the Python socket module is used.

>>> write(sample_buildout, 'buildout.cfg',... """... [buildout]... socket-timeout = 5s... develop = recipes... parts = debug...... [debug]... recipe = recipes:debug... op = timeout... """)

>>> print_(system(buildout), end='')Default socket timeout is used !Value in configuration is not numeric: [5s].<BLANKLINE>Develop: '/sample-buildout/recipes'Updating debug.op timeoutrecipe recipes:debug

Uninstall recipes

As we’ve seen, when parts are installed, buildout keeps track of filesand directories that they create. When the parts are uninstalled thesefiles and directories are deleted.

Sometimes more clean up is needed. For example, a recipe might add asystem service by calling chkconfig –add during installation. Laterduring uninstallation, chkconfig –del will need to be called toremove the system service.

In order to deal with these uninstallation issues, you can registeruninstall recipes. Uninstall recipes are registered using the‘zc.buildout.uninstall’ entry point. Parts specify uninstall recipesusing the ‘uninstall’ option.

In comparison to regular recipes, uninstall recipes are muchsimpler. They are simply callable objects that accept the name of thepart to be uninstalled and the part’s options dictionary. Uninstallrecipes don’t have access to the part itself since it maybe not beable to be instantiated at uninstallation time.

Here’s a recipe that simulates installation of a system service, alongwith an uninstall recipe that simulates removing the service.

>>> write(sample_buildout, 'recipes', 'service.py',... """... import sys... class Service:......     def __init__(self, buildout, name, options):...         self.buildout = buildout...         self.name = name...         self.options = options......     def install(self):...         sys.stdout.write("chkconfig --add %s\\n"...                          % self.options['script'])...         return ()......     def update(self):...         pass......... def uninstall_service(name, options):...     sys.stdout.write("chkconfig --del %s\\n" % options['script'])... """)

To use these recipes we must register them using entry points. Makesure to use the same name for the recipe and uninstall recipe. This isrequired to let buildout know which uninstall recipe goes with whichrecipe.

>>> write(sample_buildout, 'recipes', 'setup.py',... """... from setuptools import setup... entry_points = (... '''... [zc.buildout]... mkdir = mkdir:Mkdir... debug = debug:Debug... service = service:Service...... [zc.buildout.uninstall]... service = service:uninstall_service... ''')... setup(name="recipes", entry_points=entry_points)... """)

Here’s how these recipes could be used in a buildout:

>>> write(sample_buildout, 'buildout.cfg',... """... [buildout]... develop = recipes... parts = service...... [service]... recipe = recipes:service... script = /path/to/script... """)

When the buildout is run the service will be installed

>>> print_(system(buildout), end='')Develop: '/sample-buildout/recipes'Uninstalling debug.Installing service.chkconfig --add /path/to/script

The service has been installed. If the buildout is run again with nochanges, the service shouldn’t be changed.

>>> print_(system(buildout), end='')Develop: '/sample-buildout/recipes'Updating service.

Now we change the service part to trigger uninstallation andre-installation.

>>> write(sample_buildout, 'buildout.cfg',... """... [buildout]... develop = recipes... parts = service...... [service]... recipe = recipes:service... script = /path/to/a/different/script... """)

>>> print_(system(buildout), end='')Develop: '/sample-buildout/recipes'Uninstalling service.Running uninstall recipe.chkconfig --del /path/to/scriptInstalling service.chkconfig --add /path/to/a/different/script

Now we remove the service part, and add another part.

>>> write(sample_buildout, 'buildout.cfg',... """... [buildout]... develop = recipes... parts = debug...... [debug]... recipe = recipes:debug... """)

>>> print_(system(buildout), end='')Develop: '/sample-buildout/recipes'Uninstalling service.Running uninstall recipe.chkconfig --del /path/to/a/different/scriptInstalling debug.recipe recipes:debug

Uninstall recipes don’t have to take care of removing all the filesand directories created by the part. This is still done automatically,following the execution of the uninstall recipe. An upshot is that anuninstallation recipe can access files and directories created by arecipe before they are deleted.

For example, here’s an uninstallation recipe that simulates backing upa directory before it is deleted. It is designed to work with themkdir recipe introduced earlier.

>>> write(sample_buildout, 'recipes', 'backup.py',... """... import os, sys... def backup_directory(name, options):...     path = options['path']...     size = len(os.listdir(path))...     sys.stdout.write("backing up directory %s of size %s\\n"...                      % (path, size))... """)

It must be registered with the zc.buildout.uninstall entrypoint. Notice how it is given the name ‘mkdir’ to associate it withthe mkdir recipe.

>>> write(sample_buildout, 'recipes', 'setup.py',... """... from setuptools import setup... entry_points = (... '''... [zc.buildout]... mkdir = mkdir:Mkdir... debug = debug:Debug... service = service:Service...... [zc.buildout.uninstall]... uninstall_service = service:uninstall_service... mkdir = backup:backup_directory... ''')... setup(name="recipes", entry_points=entry_points)... """)

Now we can use it with a mkdir part.

>>> write(sample_buildout, 'buildout.cfg',... """... [buildout]... develop = recipes... parts = dir debug...... [dir]... recipe = recipes:mkdir... path = my_directory...... [debug]... recipe = recipes:debug... """)

Run the buildout to install the part.

>>> print_(system(buildout), end='')Develop: '/sample-buildout/recipes'Uninstalling debug.Installing dir.dir: Creating directory my_directoryInstalling debug.recipe recipes:debug

Now we remove the part from the configuration file.

>>> write(sample_buildout, 'buildout.cfg',... """... [buildout]... develop = recipes... parts = debug...... [debug]... recipe = recipes:debug... """)

When the buildout is run the part is removed, and the uninstall recipeis run before the directory is deleted.

>>> print_(system(buildout), end='')Develop: '/sample-buildout/recipes'Uninstalling dir.Running uninstall recipe.backing up directory /sample-buildout/my_directory of size 0Updating debug.recipe recipes:debug

Now we will return the registration to normal for the benefit of therest of the examples.

>>> write(sample_buildout, 'recipes', 'setup.py',... """... from setuptools import setup... entry_points = (... '''... [zc.buildout]... mkdir = mkdir:Mkdir... debug = debug:Debug... ''')... setup(name="recipes", entry_points=entry_points)... """)

Command-line usage

A number of arguments can be given on the buildout command line. Thecommand usage is:

buildout [options and assignments] [command [command arguments]]

The following options are supported:

-h (or –help)

Print basic usage information. If this option is used, then allother options are ignored.

-cfilename

The -c option can be used to specify a configuration file, rather thanbuildout.cfg in the current directory.

-tsocket_timeout

Specify the socket timeout in seconds.

-v

Increment the verbosity by 10. The verbosity is used to adjustthe logging level. The verbosity is subtracted from the numericvalue of the log-level option specified in the configuration file.

-q

Decrement the verbosity by 10.

-U

Don’t read user-default configuration.

-o

Run in off-line mode. This is equivalent to the assignmentbuildout:offline=true.

-O

Run in non-off-line mode. This is equivalent to the assignmentbuildout:offline=false. This is the default buildout mode. The-O option would normally be used to override a true offlinesetting in a configuration file.

-n

Run in newest mode. This is equivalent to the assignmentbuildout:newest=true. With this setting, which is the default,buildout will try to find the newest versions of distributionsavailable that satisfy its requirements.

-N

Run in non-newest mode. This is equivalent to the assignmentbuildout:newest=false. With this setting, buildout will not seeknew distributions if installed distributions satisfy it’srequirements.

Assignments are of the form:

section_name:option_name=value

Or:

option_name=value

which is equivalent to:

buildout:option_name=value

Options and assignments can be given in any order.

Here’s an example:

>>> write(sample_buildout, 'other.cfg',... """... [buildout]... develop = recipes... parts = debug... installed = .other.cfg... log-level = WARNING...... [debug]... name = other... recipe = recipes:debug... """)

Note that we used the installed buildout option to specify analternate file to store information about installed parts.

>>> print_(system(buildout+' -c other.cfg debug:op1=foo -v'), end='')Develop: '/sample-buildout/recipes'Installing debug.name otherop1 foorecipe recipes:debug

Here we used the -c option to specify an alternate configuration file,and the -v option to increase the level of logging from the default,WARNING.

Options can also be combined in the usual Unix way, as in:

>>> print_(system(buildout+' -vcother.cfg debug:op1=foo'), end='')Develop: '/sample-buildout/recipes'Updating debug.name otherop1 foorecipe recipes:debug

Here we combined the -v and -c options with the configuration filename. Note that the -c option has to be last, because it takes anargument.

>>> os.remove(os.path.join(sample_buildout, 'other.cfg'))>>> os.remove(os.path.join(sample_buildout, '.other.cfg'))

The most commonly used command is ‘install’ and it takes a list ofparts to install. if any parts are specified, only those parts areinstalled. To illustrate this, we’ll update our configuration and runthe buildout in the usual way:

>>> write(sample_buildout, 'buildout.cfg',... """... [buildout]... develop = recipes... parts = debug d1 d2 d3...... [d1]... recipe = recipes:mkdir... path = d1...... [d2]... recipe = recipes:mkdir... path = d2...... [d3]... recipe = recipes:mkdir... path = d3...... [debug]... recipe = recipes:debug... """)

>>> print_(system(buildout), end='')Develop: '/sample-buildout/recipes'Uninstalling debug.Installing debug.recipe recipes:debugInstalling d1.d1: Creating directory d1Installing d2.d2: Creating directory d2Installing d3.d3: Creating directory d3

>>> ls(sample_buildout)-  .installed.cfg-  b1.cfg-  b2.cfg-  base.cfgd  bin-  buildout.cfgd  d1d  d2d  d3d  demod  develop-eggsd  eggsd  partsd  recipes

>>> cat(sample_buildout, '.installed.cfg')... # doctest: +NORMALIZE_WHITESPACE[buildout]installed_develop_eggs = /sample-buildout/develop-eggs/recipes.egg-linkparts = debug d1 d2 d3<BLANKLINE>[debug]__buildout_installed__ =__buildout_signature__ = recipes-PiIFiO8ny5yNZ1S3JfT0xg==recipe = recipes:debug<BLANKLINE>[d1]__buildout_installed__ = /sample-buildout/d1__buildout_signature__ = recipes-PiIFiO8ny5yNZ1S3JfT0xg==path = /sample-buildout/d1recipe = recipes:mkdir<BLANKLINE>[d2]__buildout_installed__ = /sample-buildout/d2__buildout_signature__ = recipes-PiIFiO8ny5yNZ1S3JfT0xg==path = /sample-buildout/d2recipe = recipes:mkdir<BLANKLINE>[d3]__buildout_installed__ = /sample-buildout/d3__buildout_signature__ = recipes-PiIFiO8ny5yNZ1S3JfT0xg==path = /sample-buildout/d3recipe = recipes:mkdir

Now we’ll update our configuration file:

>>> write(sample_buildout, 'buildout.cfg',... """... [buildout]... develop = recipes... parts = debug d2 d3 d4...... [d2]... recipe = recipes:mkdir... path = data2...... [d3]... recipe = recipes:mkdir... path = data3...... [d4]... recipe = recipes:mkdir... path = ${d2:path}-extra...... [debug]... recipe = recipes:debug... x = 1... """)

and run the buildout specifying just d3 and d4:

>>> print_(system(buildout+' install d3 d4'), end='')Develop: '/sample-buildout/recipes'Uninstalling d3.Installing d3.d3: Creating directory data3Installing d4.d4: Creating directory data2-extra

>>> ls(sample_buildout)-  .installed.cfg-  b1.cfg-  b2.cfg-  base.cfgd  bin-  buildout.cfgd  d1d  d2d  data2-extrad  data3d  demod  develop-eggsd  eggsd  partsd  recipes

Only the d3 and d4 recipes ran. d3 was removed and data3 and data2-extrawere created.

The .installed.cfg is only updated for the recipes that ran:

>>> cat(sample_buildout, '.installed.cfg')... # doctest: +NORMALIZE_WHITESPACE[buildout]installed_develop_eggs = /sample-buildout/develop-eggs/recipes.egg-linkparts = debug d1 d2 d3 d4<BLANKLINE>[debug]__buildout_installed__ =__buildout_signature__ = recipes-PiIFiO8ny5yNZ1S3JfT0xg==recipe = recipes:debug<BLANKLINE>[d1]__buildout_installed__ = /sample-buildout/d1__buildout_signature__ = recipes-PiIFiO8ny5yNZ1S3JfT0xg==path = /sample-buildout/d1recipe = recipes:mkdir<BLANKLINE>[d2]__buildout_installed__ = /sample-buildout/d2__buildout_signature__ = recipes-PiIFiO8ny5yNZ1S3JfT0xg==path = /sample-buildout/d2recipe = recipes:mkdir<BLANKLINE>[d3]__buildout_installed__ = /sample-buildout/data3__buildout_signature__ = recipes-PiIFiO8ny5yNZ1S3JfT0xg==path = /sample-buildout/data3recipe = recipes:mkdir<BLANKLINE>[d4]__buildout_installed__ = /sample-buildout/data2-extra__buildout_signature__ = recipes-PiIFiO8ny5yNZ1S3JfT0xg==path = /sample-buildout/data2-extrarecipe = recipes:mkdir

Note that the installed data for debug, d1, and d2 haven’t changed,because we didn’t install those parts and that the d1 and d2directories are still there.

Now, if we run the buildout without the install command:

>>> print_(system(buildout), end='')Develop: '/sample-buildout/recipes'Uninstalling d2.Uninstalling d1.Uninstalling debug.Installing debug.recipe recipes:debugx 1Installing d2.d2: Creating directory data2Updating d3.Updating d4.

We see the output of the debug recipe and that data2 was created. Wealso see that d1 and d2 have gone away:

>>> ls(sample_buildout)-  .installed.cfg-  b1.cfg-  b2.cfg-  base.cfgd  bin-  buildout.cfgd  data2d  data2-extrad  data3d  demod  develop-eggsd  eggsd  partsd  recipes

Alternate directory and file locations

The buildout normally puts the bin, eggs, and parts directories in thedirectory in the directory containing the configuration file. You canprovide alternate locations, and even names for these directories.

>>> alt = tmpdir('sample-alt')

>>> write(sample_buildout, 'buildout.cfg',... """... [buildout]... develop = recipes... parts =... develop-eggs-directory = %(developbasket)s... eggs-directory = %(basket)s... bin-directory = %(scripts)s... parts-directory = %(work)s... """ % dict(...    developbasket = os.path.join(alt, 'developbasket'),...    basket = os.path.join(alt, 'basket'),...    scripts = os.path.join(alt, 'scripts'),...    work = os.path.join(alt, 'work'),... ))

>>> print_(system(buildout), end='')Creating directory '/sample-alt/scripts'.Creating directory '/sample-alt/work'.Creating directory '/sample-alt/basket'.Creating directory '/sample-alt/developbasket'.Develop: '/sample-buildout/recipes'Uninstalling d4.Uninstalling d3.Uninstalling d2.Uninstalling debug.

>>> ls(alt)d  basketd  developbasketd  scriptsd  work

>>> ls(alt, 'developbasket')-  recipes.egg-link

You can also specify an alternate buildout directory:

>>> rmdir(alt)>>> alt = tmpdir('sample-alt')

>>> write(sample_buildout, 'buildout.cfg',... """... [buildout]... directory = %(alt)s... develop = %(recipes)s... parts =... """ % dict(...    alt=alt,...    recipes=os.path.join(sample_buildout, 'recipes'),...    ))

>>> print_(system(buildout), end='')Creating directory '/sample-alt/bin'.Creating directory '/sample-alt/parts'.Creating directory '/sample-alt/eggs'.Creating directory '/sample-alt/develop-eggs'.Develop: '/sample-buildout/recipes'

>>> ls(alt)-  .installed.cfgd  bind  develop-eggsd  eggsd  parts

>>> ls(alt, 'develop-eggs')-  recipes.egg-link

Logging control

Three buildout options are used to control logging:

log-level

specifies the log level

verbosity

adjusts the log level

log-format

allows an alternate logging for mat to be specified

We’ve already seen the log level and verbosity. Let’s look at an exampleof changing the format:

>>> write(sample_buildout, 'buildout.cfg',... """... [buildout]... develop = recipes... parts =... log-level = 25... verbosity = 5... log-format = %(levelname)s %(message)s... """)

Here, we’ve changed the format to include the log-level name, ratherthan the logger name.

We’ve also illustrated, with a contrived example, that the log levelcan be a numeric value and that the verbosity can be specified in theconfiguration file. Because the verbosity is subtracted from the loglevel, we get a final log level of 20, which is the INFO level.

>>> print_(system(buildout), end='')INFO Develop: '/sample-buildout/recipes'

Predefined buildout options

Buildouts have a number of predefined options that recipes can useand that users can override in their configuration files. To seethese, we’ll run a minimal buildout configuration with a debug logginglevel. One of the features of debug logging is that the configurationdatabase is shown.

>>> write(sample_buildout, 'buildout.cfg',... """... [buildout]... parts =... """)

>>> print_(system(buildout+' -vv'), end='') # doctest: +NORMALIZE_WHITESPACEInstalling 'zc.buildout', 'distribute'.We have a develop egg: zc.buildout 1.0.0.We have the best distribution that satisfies 'distribute'.Picked: distribute = 0.6<BLANKLINE>Configuration data:[buildout]allow-hosts = *allow-picked-versions = truebin-directory = /sample-buildout/bindevelop-eggs-directory = /sample-buildout/develop-eggsdirectory = /sample-buildouteggs-directory = /sample-buildout/eggsexecutable = pythonfind-links =install-from-cache = falseinstalled = /sample-buildout/.installed.cfglog-format =log-level = INFOnewest = trueoffline = falseparts =parts-directory = /sample-buildout/partsprefer-final = truepython = buildoutshow-picked-versions = falsesocket-timeout =update-versions-file =use-dependency-links = trueverbosity = 20versions = versions[versions]zc.buildout = >=1.99zc.recipe.egg = >=1.99<BLANKLINE>

All of these options can be overridden by configuration files or bycommand-line assignments. We’ve discussed most of these optionsalready, but let’s review them and touch on some we haven’t discussed:

allow-hosts

On some environments the links visited byzc.buildout can be forbidden byparanoid firewalls. These URLs might be in the chain of links visited byzc.buildout as defined by buildout’sfind-links option, or as definedby various eggs in theirurl,download_url,dependency_links metadata.

The fact that package_index works like a spider and might visit links andgo to other locations makes this even harder.

Theallow-hosts option provides a way to prevent this, andworks exactly like the one provided ineasy_install.

You can provide a list of allowed host, together with wildcards:

[buildout]...allow-hosts =    *.python.org    example.com

All URLs that does not match these hosts will not be visited.

allow-picked-versions

By default, the buildout will choose the best match for a given requirementif the requirement is not specified precisely (for instance, using the“versions” option. This behavior corresponds to the“allow-picked-versions” being set to its default value, “true”. If“allow-picked-versions” is “false,” instead of picking the best match,buildout will raise an error. This helps enforce repeatability.

bin-directory

The directory path where scripts are written. This can be arelative path, which is interpreted relative to the directoryoption.

develop-eggs-directory

The directory path where development egg links are created for softwarebeing created in the local project. This can be a relative path,which is interpreted relative to the directory option.

directory

The buildout directory. This is the base for other buildout fileand directory locations, when relative locations are used.

eggs-directory

The directory path where downloaded eggs are put. It is common to sharethis directory across buildouts. Eggs in this directory shouldnever be modified. This can be a relative path, which isinterpreted relative to the directory option.

executable

The Python executable used to run the buildout. See the pythonoption below.

find-links

You can specify more locations to search for distributions using thefind-links option. All locations specified will be searched fordistributions along with the package index as described before.

Locations can be urls:

[buildout]...find-links = http://download.zope.org/distribution/

They can also be directories on disk:

[buildout]...find-links = /some/path

Finally, they can also be direct paths to distributions:

[buildout]...find-links = /some/path/someegg-1.0.0-py2.3.egg

Any number of locations can be specified in thefind-links option:

[buildout]...find-links =    http://download.zope.org/distribution/    /some/otherpath    /some/path/someegg-1.0.0-py2.3.egg

install-from-cache

A download cache can be used as the basis of application source releases.In an application source release, we want to distribute an application thatcan be built without making any network accesses. In this case, wedistribute a buildout with download cache and tell the buildout to installfrom the download cache only, without making network accesses. Thebuildout install-from-cache option can be used to signal that packagesshould be installed only from the download cache.

installed

The file path where information about the results of the previousbuildout run is written. This can be a relative path, which isinterpreted relative to the directory option. This file providesan inventory of installed parts with information needed to decidewhich if any parts need to be uninstalled.

log-format

The format used for logging messages.

log-level

The log level before verbosity adjustment

newest

By default buildout and recipes will try to find the newest versions ofdistributions needed to satisfy requirements. This can be very timeconsuming, especially when incrementally working on setting up a buildoutor working on a recipe. The buildout “newest” option can be used to tosuppress this. If the “newest” option is set to false, then newdistributions won’t be sought if an installed distribution meetsrequirements. The “newest” option can also be set to false using the -Ncommand-line option. See also the “offline” option.

offline

The “offline” option goes a bit further than the “newest” option. If thebuildout “offline” option is given a value of “true”, the buildout andrecipes that are aware of the option will avoid doing network access. Thisis handy when running the buildout when not connected to the internet. Italso makes buildouts run much faster. This option is typically set usingthe buildout -o option.

parts

A white space separated list of parts to be installed.

parts-directory

A working directory that parts can used to store data.

prefer-final

Currently, when searching for new releases, the newest availablerelease is used. This isn’t usually ideal, as you may get adevelopment release or alpha releases not ready to be widely used.You can request that final releases be preferred using the preferfinal option in the buildout section:

[buildout]...prefer-final = true

When the prefer-final option is set to true, then when searching fornew releases, final releases are preferred. If there are finalreleases that satisfy distribution requirements, then those releasesare used even if newer non-final releases are available. The buildoutprefer-final option can be used to override this behavior.

In buildout version 2, final releases will be preferred by default.You will then need to use a false value for prefer-final to get thenewest releases.

python

The name of a section containing information about the defaultPython interpreter. Recipes that need a installationtypically have options to tell them which Python installation touse. By convention, if a section-specific option isn’t used, theoption is looked for in the buildout section. The option mustpoint to a section with an executable option giving the path to aPython executable. By default, the buildout section defines thedefault Python as the Python used to run the buildout.

use-dependency-links

By default buildout will obey the setuptools dependency_links metadatawhen it looks for dependencies. This behavior can be controlled withthe use-dependency-links buildout option:

[buildout]...use-dependency-links = false

The option defaults to true. If you set it to false, then dependencylinks are only looked for in the locations specified by find-links.

verbosity

A log-level adjustment. Typically, this is set via the -q and -vcommand-line options.

Creating new buildouts and bootstrapping

If zc.buildout is installed, you can use it to create a new buildoutwith it’s own local copies of zc.buildout and distribute and withlocal buildout scripts.

>>> sample_bootstrapped = tmpdir('sample-bootstrapped')

>>> print_(system(buildout...              +' -c'+os.path.join(sample_bootstrapped, 'setup.cfg')...              +' init'), end='')Creating '/sample-bootstrapped/setup.cfg'.Creating directory '/sample-bootstrapped/bin'.Creating directory '/sample-bootstrapped/parts'.Creating directory '/sample-bootstrapped/eggs'.Creating directory '/sample-bootstrapped/develop-eggs'.Generated script '/sample-bootstrapped/bin/buildout'.

Note that a basic setup.cfg was created for us. This is because weprovided an ‘init’ argument. By default, the generatedsetup.cfg is as minimal as it could be:

>>> cat(sample_bootstrapped, 'setup.cfg')[buildout]parts =

We also get other buildout artifacts:

>>> ls(sample_bootstrapped)d  bind  develop-eggsd  eggsd  parts-  setup.cfg

>>> ls(sample_bootstrapped, 'bin')-  buildout

>>> _ = (ls(sample_bootstrapped, 'eggs'),...      ls(sample_bootstrapped, 'develop-eggs'))-  distribute-0.6-py2.3.egg-  zc.buildout-1.0-py2.3.egg

(We list both the eggs and develop-eggs directories because thebuildout or distribute egg could be installed in the develop-eggsdirectory if the original buildout had develop eggs for eitherbuildout or distribute.)

Note that the buildout script was installed but not run. To runthe buildout, we’d have to run the installed buildout script.

If we have an existing buildout that already has a buildout.cfg, we’llnormally use the bootstrap command instead of init. It will complainif there isn’t a configuration file:

>>> sample_bootstrapped2 = tmpdir('sample-bootstrapped2')

>>> print_(system(buildout...              +' -c'+os.path.join(sample_bootstrapped2, 'setup.cfg')...              +' bootstrap'), end='')While:  Initializing.Error: Couldn't open /sample-bootstrapped2/setup.cfg

>>> write(sample_bootstrapped2, 'setup.cfg',... """... [buildout]... parts =... """)

>>> print_(system(buildout...              +' -c'+os.path.join(sample_bootstrapped2, 'setup.cfg')...              +' bootstrap'), end='')Creating directory '/sample-bootstrapped2/bin'.Creating directory '/sample-bootstrapped2/parts'.Creating directory '/sample-bootstrapped2/eggs'.Creating directory '/sample-bootstrapped2/develop-eggs'.Generated script '/sample-bootstrapped2/bin/buildout'.

Similarly, if there is a configuration file and we use the initcommand, we’ll get an error that the configuration file alreadyexists:

>>> print_(system(buildout...              +' -c'+os.path.join(sample_bootstrapped, 'setup.cfg')...              +' init'), end='')While:  Initializing.Error: '/sample-bootstrapped/setup.cfg' already exists.

Initial eggs

When using theinit command, you can specify distribution requirementsor paths to use:

>>> cd(sample_bootstrapped)>>> remove('setup.cfg')>>> print_(system(buildout + ' -csetup.cfg init demo other ./src'), end='')Creating '/sample-bootstrapped/setup.cfg'.Getting distribution for 'zc.recipe.egg>=2.0.0a3'.Got zc.recipe.eggInstalling py.Getting distribution for 'demo'.Got demo 0.3.Getting distribution for 'other'.Got other 1.0.Getting distribution for 'demoneeded'.Got demoneeded 1.1.Generated script '/sample-bootstrapped/bin/demo'.Generated script '/sample-bootstrapped/bin/distutilsscript'.Generated interpreter '/sample-bootstrapped/bin/py'.

This causes apy part to be included that sets up a custom pythoninterpreter with the given requirements or paths:

>>> cat('setup.cfg')[buildout]parts = py<BLANKLINE>[py]recipe = zc.recipe.egginterpreter = pyeggs =  demo  otherextra-paths =  ./src

Passing requirements or paths causes the the builout to be run as partof initialization. In the example above, we got a number ofdistributions installed and 2 scripts generated. The first,demo,was defined by thedemo project. The second,py was defined bythe generated configuration. It’s a “custom interpreter” that behaveslike a standard Python interpeter, except that includes the specifiedeggs and extra paths in it’s Python path.

We specified a source directory that didn’t exist. Buildout created itfor us:

>>> ls('.')-  .installed.cfgd  bind  develop-eggsd  eggsd  parts-  setup.cfgd  src

>>> uncd()

Finding distributions

By default, buildout searches the Python Package Index when lookingfor distributions. You can, instead, specify your own index to searchusing theindex option:

[buildout]...index = http://index.example.com/

This index, or the default ofhttp://pypi.python.org/simple/ if noindex is specified, will always be searched for distributions unlessrunning buildout with options that prevent searching fordistributions. The latest version of the distribution that meets therequirements of the buildout will always be used.

You can also specify more locations to search for distributions usingthefind-links option. See its description above.

Controlling the installation database

The buildout installed option is used to specify the file used to saveinformation on installed parts. This option is initialized to“.installed.cfg”, but it can be overridden in the configuration fileor on the command line:

>>> write('buildout.cfg',... """... [buildout]... develop = recipes... parts = debug...... [debug]... recipe = recipes:debug... """)

>>> print_(system(buildout+' buildout:installed=inst.cfg'), end='')Develop: '/sample-buildout/recipes'Installing debug.recipe recipes:debug

>>> ls(sample_buildout)-  b1.cfg-  b2.cfg-  base.cfgd  bin-  buildout.cfgd  demod  develop-eggsd  eggs-  inst.cfgd  partsd  recipes

The installation database can be disabled by supplying an emptybuildout installed option:

>>> os.remove('inst.cfg')>>> print_(system(buildout+' buildout:installed='), end='')Develop: '/sample-buildout/recipes'Installing debug.recipe recipes:debug

>>> ls(sample_buildout)-  b1.cfg-  b2.cfg-  base.cfgd  bin-  buildout.cfgd  demod  develop-eggsd  eggsd  partsd  recipes

Note that there will be no installation database if there are no parts:

>>> write('buildout.cfg',... """... [buildout]... parts =... """)

>>> print_(system(buildout+' buildout:installed=inst.cfg'), end='')

>>> ls(sample_buildout)-  b1.cfg-  b2.cfg-  base.cfgd  bin-  buildout.cfgd  demod  develop-eggsd  eggsd  partsd  recipes

Extensions

A feature allows code to be loaded and run afterconfiguration files have been read but before the buildout has begunany processing. The intent is to allow special plugins such asurllib2 request handlers to be loaded.

To load an extension, we use the extensions option and list one ormore distribution requirements, on separate lines. The distributionsnamed will be loaded and anyzc.buildout.extension entry points foundwill be called with the buildout as an argument. When buildoutfinishes processing, anyzc.buildout.unloadextension entry pointsfound will be called with the buildout as an argument.

Let’s create a sample extension in our sample buildout created in theprevious section:

>>> mkdir(sample_bootstrapped, 'demo')

>>> write(sample_bootstrapped, 'demo', 'demo.py',... """... import sys... def ext(buildout):...     sys.stdout.write('%s %s\\n' % ('ext', sorted(buildout)))... def unload(buildout):...     sys.stdout.write('%s %s\\n' % ('unload', sorted(buildout)))... """)

>>> write(sample_bootstrapped, 'demo', 'setup.py',... """... from setuptools import setup...... setup(...     name = "demo",...     entry_points = {...        'zc.buildout.extension': ['ext = demo:ext'],...        'zc.buildout.unloadextension': ['ext = demo:unload'],...        },...     )... """)

Our extension just prints out the word ‘demo’, and lists the sectionsfound in the buildout passed to it.

We’ll update our buildout.cfg to list the demo directory as a developegg to be built:

>>> write(sample_bootstrapped, 'buildout.cfg',... """... [buildout]... develop = demo... parts =... """)

>>> os.chdir(sample_bootstrapped)>>> print_(system(os.path.join(sample_bootstrapped, 'bin', 'buildout')),...        end='')Develop: '/sample-bootstrapped/demo'

Now we can add the extensions option. We were a bit tricky and ranthe buildout once with the demo develop egg defined but without theextension option. This is because extensions are loaded before thebuildout creates develop eggs. We needed to use a separate buildoutrun to create the develop egg. Normally, when eggs are loaded fromthe network, we wouldn’t need to do anything special.

>>> write(sample_bootstrapped, 'buildout.cfg',... """... [buildout]... develop = demo... extensions = demo... parts =... """)

We see that our extension is loaded and executed:

>>> print_(system(os.path.join(sample_bootstrapped, 'bin', 'buildout')),...        end='')ext ['buildout', 'versions']Develop: '/sample-bootstrapped/demo'unload ['buildout', 'versions']

Easier reporting and managing of versions (new in buildout 2.0)

Since buildout 2.0, the functionality of thebuildout-versions extension is part ofbuildout itself. This makes reporting and managing versions easier.

Buildout picks a version for distribute and for the tests, we need to grab theversion number:

>>> import pkg_resources>>> req = pkg_resources.Requirement.parse('distribute')>>> distribute_version = pkg_resources.working_set.find(req).version

If you set theshow-picked-versions option, buildout will printversions it picked at the end of its run:

>>> write('buildout.cfg',... '''... [buildout]... parts = foo... find-links = %s... show-picked-versions = true...... [versions]...... [foo]... recipe = spam... ''' % join('recipe', 'dist'))>>> print_(system(buildout), end='') # doctest: +ELLIPSISUpdating foo.recipe v2Versions had to be automatically picked.The following part definition lists the versions picked:[versions]distribute = 0.6.99spam = 2

When everything is pinned, no output is generated:

>>> write('buildout.cfg',... '''... [buildout]... parts = foo... find-links = %s... show-picked-versions = true...... [versions]... distribute = %s... spam = 2...... [foo]... recipe = spam... ''' % (join('recipe', 'dist'), distribute_version))>>> print_(system(buildout), end='') # doctest: +ELLIPSISUpdating foo.recipe v2

The Python package index is case-insensitive. Bothhttp://pypi.python.org/simple/Django/ andhttp://pypi.python.org/simple/dJaNgO/ work. And distributions aren’t alwaysnaming themselves consistently case-wise. So all version names are normalizedand case differences won’t impact the pinning:

>>> write('buildout.cfg',... '''... [buildout]... parts = foo... find-links = %s... show-picked-versions = true...... [versions]... distriBUTE = %s... Spam = 2...... [foo]... recipe = spam... ''' % (join('recipe', 'dist'), distribute_version))>>> print_(system(buildout), end='') # doctest: +ELLIPSISUpdating foo.recipe v2

Sometimes it is handy to have a separate file with versions. This is a regularbuildout file with a single[versions] section. You include it byextending from that versions file:

>>> write('my_versions.cfg',... '''... [versions]... distribute = %s... spam = 2... ''' % distribute_version)>>> write('buildout.cfg',... '''... [buildout]... parts = foo... extends = my_versions.cfg... find-links = %s... show-picked-versions = true...... [foo]... recipe = spam... ''' % join('recipe', 'dist'))>>> print_(system(buildout), end='') # doctest: +ELLIPSISUpdating foo.recipe v2

If not everything is pinned and buildout has to pick versions, you can tellbuildout to append the versions to your versions file. It simply appends themat the end.

>>> write('my_versions.cfg',... '''... [versions]... distribute = %s... ''' % distribute_version)>>> write('buildout.cfg',... '''... [buildout]... parts = foo... extends = my_versions.cfg... update-versions-file = my_versions.cfg... find-links = %s... show-picked-versions = true...... [foo]... recipe = spam... ''' % join('recipe', 'dist'))>>> print_(system(buildout), end='') # doctest: +ELLIPSISUpdating foo.recipe v2Versions had to be automatically picked.The following part definition lists the versions picked:[versions]spam = 2Picked versions have been written to my_versions.cfg

The versions file now contains the extra pin:

>>> 'spam = 2' in open('my_versions.cfg').read()True

And re-running buildout doesn’t report any picked versions anymore:

>>> print_(system(buildout), end='') # doctest: +ELLIPSISUpdating foo.recipe v2

If you’ve enabledupdate-versions-file but notshow-picked-versions,buildout will append the versions to your versions file anyway (withoutprinting them to the console):

>>> write('my_versions.cfg',... '''... [versions]... distribute = %s... ''' % distribute_version)>>> write('buildout.cfg',... '''... [buildout]... parts = foo... extends = my_versions.cfg... update-versions-file = my_versions.cfg... find-links = %s... show-picked-versions = false...... [foo]... recipe = spam... ''' % join('recipe', 'dist'))>>> print_(system(buildout), end='') # doctest: +ELLIPSISUpdating foo.recipe v2Picked versions have been written to my_versions.cfg

The versions file contains the extra pin:

>>> 'spam = 2' in open('my_versions.cfg').read()True

Because buildout now includes buildout-versions’ (and the olderbuildout.dumppickedversions’) functionality, it warns if these extensions areconfigured.

>>> write(sample_buildout, 'buildout.cfg',... """... [buildout]... parts = foo... extensions = buildout-versions...... [foo]... recipe = spam... """)>>> print_(system(buildout), end='') # doctest: +ELLIPSISWhile:  Installing.  Loading extensions.Error: The extension buildout-versions is now included in buildout itself.Remove the extension from your configuration and look at the `show-picked-versions`option in buildout's documentation.

Downloading without using the cache

If no download cache should be used, the download utility is instantiatedwithout any arguments:

>>> from zc.buildout.download import Download>>> download = Download()>>> print_(download.cache_dir)None

Downloading a file is achieved by calling the utility with the URL as anargument. A tuple is returned that consists of the path to the downloaded copyof the file and a boolean value indicating whether this is a temporary filemeant to be cleaned up during the same buildout run:

>>> path, is_temp = download(server_url+'foo.txt')>>> print_(path)/.../buildout-...>>> cat(path)This is a foo text.

As we aren’t using the download cache and haven’t specified a target patheither, the download has ended up in a temporary file:

>>> is_tempTrue

>>> import tempfile>>> path.startswith(tempfile.gettempdir())True

We are responsible for cleaning up temporary files behind us:

>>> remove(path)

When trying to access a file that doesn’t exist, we’ll get an exception:

>>> try: download(server_url+'not-there') # doctest: +ELLIPSIS... except: print_('download error')... else: print_('woops')download error

Downloading a local file doesn’t produce a temporary file but simply returnsthe local file itself:

>>> download(join(server_data, 'foo.txt'))('/sample_files/foo.txt', False)

We can also have the downloaded file’s MD5 sum checked:

>>> try: from hashlib import md5... except ImportError: from md5 import new as md5

>>> path, is_temp = download(server_url+'foo.txt',...                          md5('This is a foo text.'.encode()).hexdigest())>>> is_tempTrue>>> remove(path)

>>> download(server_url+'foo.txt',...          md5('The wrong text.'.encode()).hexdigest())Traceback (most recent call last):ChecksumError: MD5 checksum mismatch downloading 'http://localhost/foo.txt'

The error message in the event of an MD5 checksum mismatch for a local filereads somewhat differently:

>>> download(join(server_data, 'foo.txt'),...               md5('This is a foo text.'.encode()).hexdigest())('/sample_files/foo.txt', False)

>>> download(join(server_data, 'foo.txt'),...          md5('The wrong text.'.encode()).hexdigest())Traceback (most recent call last):ChecksumError: MD5 checksum mismatch for local resource at '/sample_files/foo.txt'.

Finally, we can download the file to a specified place in the file system:

>>> target_dir = tmpdir('download-target')>>> path, is_temp = download(server_url+'foo.txt',...                          path=join(target_dir, 'downloaded.txt'))>>> print_(path)/download-target/downloaded.txt>>> cat(path)This is a foo text.>>> is_tempFalse

Trying to download a file in offline mode will result in an error:

>>> download = Download(cache=None, offline=True)>>> download(server_url+'foo.txt')Traceback (most recent call last):UserError: Couldn't download 'http://localhost/foo.txt' in offline mode.

As an exception to this rule, file system paths and URLs in thefilescheme will still work:

>>> cat(download(join(server_data, 'foo.txt'))[0])This is a foo text.>>> cat(download('file:' + join(server_data, 'foo.txt'))[0])This is a foo text.

>>> remove(path)

Downloading using the download cache

In order to make use of the download cache, we need to configure the downloadutility differently. To do this, we pass a directory path as thecacheattribute upon instantiation:

>>> cache = tmpdir('download-cache')>>> download = Download(cache=cache)>>> print_(download.cache_dir)/download-cache/

>>> ls(cache)>>> path, is_temp = download(server_url+'foo.txt')>>> print_(path)/download-cache/foo.txt>>> cat(path)This is a foo text.>>> is_tempFalse

>>> write(server_data, 'foo.txt', 'The wrong text.')>>> path, is_temp = download(server_url+'foo.txt')>>> print_(path)/download-cache/foo.txt>>> cat(path)This is a foo text.

>>> download(server_url+'foo.txt', md5('The wrong text.'.encode()).hexdigest())Traceback (most recent call last):ChecksumError: MD5 checksum mismatch for cached download               from 'http://localhost/foo.txt' at '/download-cache/foo.txt'

>>> mkdir(server_data, 'other')>>> write(server_data, 'other', 'foo.txt', 'The wrong text.')>>> path, is_temp = download(server_url+'other/foo.txt')>>> print_(path)/download-cache/foo.txt>>> cat(path)This is a foo text.

>>> remove(cache, 'foo.txt')>>> ls(cache)>>> write(server_data, 'foo.txt', 'This is a foo text.')

>>> path, is_temp = download(server_url+'foo.txt',...                          path=join(target_dir, 'downloaded.txt'))>>> print_(path)/download-target/downloaded.txt>>> cat(path)This is a foo text.>>> is_tempFalse>>> ls(cache)- foo.txt

>>> remove(path)>>> write(server_data, 'foo.txt', 'The wrong text.')

>>> path, is_temp = download(server_url+'foo.txt',...                          path=join(target_dir, 'downloaded.txt'))>>> print_(path)/download-target/downloaded.txt>>> cat(path)This is a foo text.>>> is_tempFalse

>>> download = Download(cache=cache, offline=True)>>> cat(download(server_url+'foo.txt')[0])This is a foo text.

>>> remove(cache, 'foo.txt')>>> ls(cache)

>>> write(server_data, 'foo.txt', 'This is a foo text.')>>> download = Download(cache=cache)

>>> cat(download('file:' + join(server_data, 'foo.txt'), path=path)[0])This is a foo text.>>> ls(cache)- foo.txt

>>> remove(cache, 'foo.txt')

>>> cat(download(join(server_data, 'foo.txt'), path=path)[0])This is a foo text.>>> ls(cache)- foo.txt

>>> remove(cache, 'foo.txt')

>>> download(server_url+'foo.txt', md5('The wrong text.'.encode()).hexdigest())Traceback (most recent call last):ChecksumError: MD5 checksum mismatch downloading 'http://localhost/foo.txt'>>> ls(cache)

>>> remove(path)

>>> download(server_url+'bar.txt') # doctest: +NORMALIZE_WHITESPACE +ELLIPSISTraceback (most recent call last):...UserError: Error downloading extends for URL http://localhost/bar.txt:...404...>>> ls(cache)

>>> Download(cache=join(cache, 'non-existent'))(server_url+'foo.txt')Traceback (most recent call last):UserError: The directory:'/download-cache/non-existent'to be used as a download cache doesn't exist.

>>> download = Download(cache=cache, namespace='test')>>> print_(download.cache_dir)/download-cache/test

>>> ls(cache)

>>> path, is_temp = download(server_url+'foo.txt')>>> print_(path)/download-cache/test/foo.txt>>> ls(cache)d test>>> ls(cache, 'test')- foo.txt>>> cat(path)This is a foo text.>>> is_tempFalse

>>> write(server_data, 'foo.txt', 'The wrong text.')>>> write(cache, 'foo.txt', 'The wrong text.')

>>> path, is_temp = download(server_url+'foo.txt')>>> print_(path)/download-cache/test/foo.txt>>> cat(path)This is a foo text.

>>> rmdir(cache, 'test')>>> remove(cache, 'foo.txt')>>> write(server_data, 'foo.txt', 'This is a foo text.')

>>> download = Download(cache=cache, hash_name=True)>>> path, is_temp = download(server_url+'foo.txt')>>> print_(path)/download-cache/09f5793fcdc1716727f72d49519c688d>>> cat(path)This is a foo text.>>> ls(cache)- 09f5793fcdc1716727f72d49519c688d

>>> (path.lower() ==...  join(cache, md5((server_url+'foo.txt').encode()).hexdigest()).lower())True

>>> write(server_data, 'foo.txt', 'The wrong text.')>>> (path, is_temp) == download(server_url+'foo.txt')True>>> cat(path)This is a foo text.>>> ls(cache)- 09f5793fcdc1716727f72d49519c688d

>>> path2, is_temp = download(server_url+'other/foo.txt')>>> print_(path2)/download-cache/537b6d73267f8f4447586989af8c470e>>> path == path2False>>> (path2.lower() ==...  join(cache, md5((server_url+'other/foo.txt').encode()).hexdigest()...       ).lower())True>>> cat(path)This is a foo text.>>> cat(path2)The wrong text.>>> ls(cache)- 09f5793fcdc1716727f72d49519c688d- 537b6d73267f8f4447586989af8c470e

>>> remove(path)>>> remove(path2)>>> write(server_data, 'foo.txt', 'This is a foo text.')

Using the cache purely as a fall-back

Sometimes it is desirable to try downloading a file from the net if at allpossible, and use the cache purely as a fall-back option when a server isdown or if we are in offline mode. This mode is only in effect if a downloadcache is configured in the first place:

>>> download = Download(cache=cache, fallback=True)>>> print_(download.cache_dir)/download-cache/

A downloaded file will be cached:

>>> ls(cache)>>> path, is_temp = download(server_url+'foo.txt')>>> ls(cache)- foo.txt>>> cat(cache, 'foo.txt')This is a foo text.>>> is_tempFalse

If the file cannot be served, the cached copy will be used:

>>> remove(server_data, 'foo.txt')>>> try: Download()(server_url+'foo.txt') # doctest: +ELLIPSIS... except: print_('download error')... else: print_('woops')download error>>> path, is_temp = download(server_url+'foo.txt')>>> cat(path)This is a foo text.>>> is_tempFalse

Similarly, if the file is served but we’re in offline mode, we’ll fall back tousing the cache:

>>> write(server_data, 'foo.txt', 'The wrong text.')>>> get(server_url+'foo.txt')'The wrong text.'

>>> offline_download = Download(cache=cache, offline=True, fallback=True)>>> path, is_temp = offline_download(server_url+'foo.txt')>>> print_(path)/download-cache/foo.txt>>> cat(path)This is a foo text.>>> is_tempFalse

However, when downloading the file normally with the cache being used infall-back mode, the file will be downloaded from the net and the cached copywill be replaced with the new content:

>>> cat(download(server_url+'foo.txt')[0])The wrong text.>>> cat(cache, 'foo.txt')The wrong text.

When trying to download a resource whose checksum does not match, the cachedcopy will neither be used nor overwritten:

>>> write(server_data, 'foo.txt', 'This is a foo text.')>>> download(server_url+'foo.txt', md5('The wrong text.'.encode()).hexdigest())Traceback (most recent call last):ChecksumError: MD5 checksum mismatch downloading 'http://localhost/foo.txt'>>> cat(cache, 'foo.txt')The wrong text.

Configuring the download utility from buildout options

The configuration options explained so far derive from the build logicimplemented by the calling code. Other options configure the download utilityfor use in a particular project or buildout run; they are read from thebuildout configuration section. The latter can be passed directly as thefirst argument to the download utility’s constructor.

The location of the download cache is specified by thedownload-cacheoption:

>>> download = Download({'download-cache': cache}, namespace='cmmi')>>> print_(download.cache_dir)/download-cache/cmmi

If thedownload-cache option specifies a relative path, it is understoodrelative to the current working directory, or to the buildout directory ifthat is given:

>>> download = Download({'download-cache': 'relative-cache'})>>> print_(download.cache_dir)/sample-buildout/relative-cache/

>>> download = Download({'directory': join(sample_buildout, 'root'),...                      'download-cache': 'relative-cache'})>>> print_(download.cache_dir)/sample-buildout/root/relative-cache/

Keyword parameters take precedence over the corresponding options:

>>> download = Download({'download-cache': cache}, cache=None)>>> print_(download.cache_dir)None

Whether to assume offline mode can be inferred from either theoffline ortheinstall-from-cache option. As usual with zc.buildout, these optionsmust assume one of the values ‘true’ and ‘false’:

>>> download = Download({'offline': 'true'})>>> download.offlineTrue

>>> download = Download({'offline': 'false'})>>> download.offlineFalse

>>> download = Download({'install-from-cache': 'true'})>>> download.offlineTrue

>>> download = Download({'install-from-cache': 'false'})>>> download.offlineFalse

These two options are combined using logical ‘or’:

>>> download = Download({'offline': 'true', 'install-from-cache': 'false'})>>> download.offlineTrue

>>> download = Download({'offline': 'false', 'install-from-cache': 'true'})>>> download.offlineTrue

Theoffline keyword parameter takes precedence over both theofflineandinstall-from-cache options:

>>> download = Download({'offline': 'true'}, offline=False)>>> download.offlineFalse

>>> download = Download({'install-from-cache': 'false'}, offline=True)>>> download.offlineTrue

Regressions

MD5 checksum calculation needs to be reliable on all supported systems, whichrequires text files to be treated as binary to avoid implicit line-endingconversions:

>>> text = 'First line of text.\r\nSecond line.\r\n'>>> f = open(join(server_data, 'foo.txt'), 'wb')>>> _ = f.write(text.encode())>>> f.close()>>> path, is_temp = Download()(server_url+'foo.txt',...                            md5(text.encode()).hexdigest())>>> remove(path)

When “downloading” a directory given by file-system path orfile: URL andusing a download cache at the same time, the cached directory wasn’t handledcorrectly. Consequently, the cache was defeated and an attempt to cache thedirectory a second time broke. This is how it should work:

>>> download = Download(cache=cache)>>> dirpath = join(server_data, 'some_directory')>>> mkdir(dirpath)>>> dest, _ = download(dirpath)

If we now modify the source tree, the second download will produce theoriginal one from the cache:

>>> mkdir(join(dirpath, 'foo'))>>> ls(dirpath)d foo>>> dest, _ = download(dirpath)>>> ls(dest)

Clean up

We should have cleaned up all temporary files created by downloading things:

>>> ls(tempfile.tempdir)

Reset the global temporary directory:

>>> tempfile.tempdir = old_tempdir

Installing solely from a download cache

A download cache can be used as the basis of application sourcereleases. In an application source release, we want to distribute anapplication that can be built without making any network accesses. Inthis case, we distribute a buildout with download cache and tell thebuildout to install from the download cache only, without makingnetwork accesses. The buildout install-from-cache option can be usedto signal that packages should be installed only from the downloadcache.

Let’s remove our installed eggs and run the buildout with theinstall-from-cache option set to true:

>>> for  f in os.listdir('eggs'):...     if f.startswith('demo'):...         remove('eggs', f)

>>> write('buildout.cfg',... '''... [buildout]... parts = eggs... download-cache = %(cache)s... install-from-cache = true... find-links = %(link_server)s...... [eggs]... recipe = zc.recipe.egg... eggs = demo... ''' % globals())

>>> print_(system(buildout), end='')Uninstalling eggs.Installing eggs.Getting distribution for 'demo'.Got demo 0.2.Getting distribution for 'demoneeded'.Got demoneeded 1.1.Generated script '/sample-buildout/bin/demo'.

Basic use of the extends cache

We put some base configuration on a server and reference it from a samplebuildout:

>>> write(server_data, 'base.cfg', """\... [buildout]... parts =... foo = bar... """)

>>> write('buildout.cfg', """\... [buildout]... extends = %sbase.cfg... """ % server_url)

When trying to run this buildout offline, we’ll find that we cannot read allof the required configuration:

>>> print_(system(buildout + ' -o'))While:  Initializing.Error: Couldn't download 'http://localhost/base.cfg' in offline mode.

Trying the same online, we can:

>>> print_(system(buildout))Unused options for buildout: 'foo'.

As long as we haven’t said anything about caching downloaded configuration,nothing gets cached. Offline mode will still cause the buildout to fail:

>>> print_(system(buildout + ' -o'))While:  Initializing.Error: Couldn't download 'http://localhost/base.cfg' in offline mode.

Let’s now specify a cache for base configuration files. This cache isdifferent from the download cache used by recipes for caching distributionsand other files; one might, however, use a namespace subdirectory of thedownload cache for it. The configuration cache we specify will be created whenrunning buildout and the base.cfg file will be put in it (with the file namebeing a hash of the complete URL):

>>> mkdir('cache')>>> write('buildout.cfg', """\... [buildout]... extends = %sbase.cfg... extends-cache = cache... """ % server_url)

>>> print_(system(buildout))Unused options for buildout: 'foo'.

>>> cache = join(sample_buildout, 'cache')>>> ls(cache)-  5aedc98d7e769290a29d654a591a3a45

>>> import os>>> cat(cache, os.listdir(cache)[0])[buildout]parts =foo = bar

We can now run buildout offline as it will read base.cfg from the cache:

>>> print_(system(buildout + ' -o'))Unused options for buildout: 'foo'.

The cache is being used purely as a fall-back in case we are offline or don’thave access to a configuration file to be downloaded. As long as we areonline, buildout attempts to download a fresh copy of each file even if acached copy of the file exists. To see this, we put different configuration inthe same place on the server and run buildout in offline mode so it takesbase.cfg from the cache:

>>> write(server_data, 'base.cfg', """\... [buildout]... parts =... bar = baz... """)

>>> print_(system(buildout + ' -o'))Unused options for buildout: 'foo'.

In online mode, buildout will download and use the modified version:

>>> print_(system(buildout))Unused options for buildout: 'bar'.

Trying offline mode again, the new version will be used as it has been put inthe cache now:

>>> print_(system(buildout + ' -o'))Unused options for buildout: 'bar'.

Clean up:

>>> rmdir(cache)

Specifying extends cache and offline mode

Normally, the values of buildout options such as the location of a downloadcache or whether to use offline mode are determined by first reading theuser’s default configuration, updating it with the project’s configuration andfinally applying command-line options. User and project configuration areassembled by reading a file such as~/.buildout/default.cfg,buildout.cfg or a URL given on the command line, recursively (depth-first)downloading any base configuration specified by thebuildout:extendsoption read from each of those config files, and finally evaluating eachconfig file to provide default values for options not yet read.

This works fine for all options that do not influence how configuration isdownloaded in the first place. Theextends-cache andoffline options,however, are treated differently from the procedure described in order to makeit simple and obvious to see where a particular configuration file came fromunder any particular circumstances.

• Offline and extends-cache settings are read from the two root config filesexclusively. Otherwise one could construct configuration files that, whenread, imply that they should have been read from a different source thanthey have. Also, specifying the extends cache within a file that might haveto be taken from the cache before being read wouldn’t make a lot of sense.

• Offline and extends-cache settings given by the user’s defaults apply to theprocess of assembling the project’s configuration. If no extends cache hasbeen specified by the user’s default configuration, the project’s rootconfig file must be available, be it from disk or from the net.

• Offline mode turned on by the-o command line option is honoured fromthe beginning even though command line options are applied to theconfiguration last. If offline mode is not requested by the command line, itmay be switched on by either the user’s or the project’s config root.

>>> mkdir('home')>>> mkdir('home', '.buildout')>>> mkdir('cache')>>> mkdir('user-cache')>>> os.environ['HOME'] = join(sample_buildout, 'home')>>> write('home', '.buildout', 'default.cfg', """\... [buildout]... extends = fancy_default.cfg... extends-cache = user-cache... """)>>> write('home', '.buildout', 'fancy_default.cfg', """\... [buildout]... extends = %sbase_default.cfg... """ % server_url)>>> write(server_data, 'base_default.cfg', """\... [buildout]... foo = bar... offline = false... """)

>>> write('buildout.cfg', """\... [buildout]... extends = fancy.cfg... extends-cache = cache... """)>>> write('fancy.cfg', """\... [buildout]... extends = %sbase.cfg... """ % server_url)>>> write(server_data, 'base.cfg', """\... [buildout]... parts =... offline = false... """)

>>> print_(system(buildout))Unused options for buildout: 'foo'.

>>> ls('user-cache')-  10e772cf422123ef6c64ae770f555740>>> cat('user-cache', os.listdir('user-cache')[0])[buildout]foo = baroffline = false

>>> ls('cache')-  c72213127e6eb2208a3e1fc1dba771a7>>> cat('cache', os.listdir('cache')[0])[buildout]parts =offline = false

>>> write('home', '.buildout', 'default.cfg', """\... [buildout]... extends = fancy_default.cfg... """)>>> write('home', '.buildout', 'fancy_default.cfg', """\... [buildout]... extends = %sbase_default.cfg... extends-cache = user-cache... """ % server_url)

>>> write('buildout.cfg', """\... [buildout]... extends = fancy.cfg... """)>>> write('fancy.cfg', """\... [buildout]... extends = %sbase.cfg... extends-cache = cache... """ % server_url)

>>> remove('user-cache', os.listdir('user-cache')[0])>>> remove('cache', os.listdir('cache')[0])

>>> print_(system(buildout))Unused options for buildout: 'foo'.

>>> ls('user-cache')-  0548bad6002359532de37385bb532e26>>> cat('user-cache', os.listdir('user-cache')[0])[buildout]parts =offline = false

>>> ls('cache')

>>> rmdir('user-cache')>>> rmdir('cache')

>>> print_(system(buildout + ' -o'))While:  Initializing.Error: Couldn't download 'http://localhost/base_default.cfg' in offline mode.

>>> write('home', '.buildout', 'default.cfg', """\... [buildout]... extends = fancy_default.cfg... offline = true... """)>>> print_(system(buildout))While:  Initializing.Error: Couldn't download 'http://localhost/base_default.cfg' in offline mode.

>>> write('home', '.buildout', 'default.cfg', """\... [buildout]... extends = fancy_default.cfg... """)>>> write('home', '.buildout', 'fancy_default.cfg', """\... [buildout]... extends = %sbase_default.cfg... offline = true... """ % server_url)>>> print_(system(buildout))While:  Initializing.Error: Couldn't download 'http://localhost/base.cfg' in offline mode.

>>> write('home', '.buildout', 'fancy_default.cfg', """\... [buildout]... extends = %sbase_default.cfg... """ % server_url)>>> write('buildout.cfg', """\... [buildout]... extends = fancy.cfg... offline = true... """)>>> print_(system(buildout))While:  Initializing.Error: Couldn't download 'http://localhost/base.cfg' in offline mode.

>>> write('buildout.cfg', """\... [buildout]... extends = fancy.cfg... """)>>> write('fancy.cfg', """\... [buildout]... extends = %sbase.cfg... offline = true... """ % server_url)>>> print_(system(buildout))Unused options for buildout: 'foo'.

>>> write('home', '.buildout', 'default.cfg', """\... [buildout]... extends = fancy_default.cfg... install-from-cache = true... """)>>> print_(system(buildout))While:  Initializing.Error: Couldn't download 'http://localhost/base_default.cfg' in offline mode.

>>> write('home', '.buildout', 'default.cfg', """\... [buildout]... extends = fancy_default.cfg... """)>>> write('home', '.buildout', 'fancy_default.cfg', """\... [buildout]... extends = %sbase_default.cfg... install-from-cache = true... """ % server_url)>>> print_(system(buildout))While:  Initializing.Error: Couldn't download 'http://localhost/base.cfg' in offline mode.

>>> write('home', '.buildout', 'fancy_default.cfg', """\... [buildout]... extends = %sbase_default.cfg... """ % server_url)>>> write('buildout.cfg', """\... [buildout]... extends = fancy.cfg... install-from-cache = true... """)>>> print_(system(buildout))While:  Initializing.Error: Couldn't download 'http://localhost/base.cfg' in offline mode.

>>> write('buildout.cfg', """\... [buildout]... extends = fancy.cfg... """)>>> write('fancy.cfg', """\... [buildout]... extends = %sbase.cfg... install-from-cache = true... """ % server_url)>>> print_(system(buildout))While:  Installing.  Checking for upgrades.An internal error occured ...ValueError: install_from_cache set to true with no download cache

>>> rmdir('home', '.buildout')

Newest and non-newest behaviour for extends cache

While offline mode forbids network access completely, ‘newest’ mode determineswhether to look for updated versions of a resource even if some version of itis already present locally. If we run buildout in newest mode(newest = true), the configuration files are updated with each run:

>>> mkdir("cache")>>> write(server_data, 'base.cfg', """\... [buildout]... parts =... """)>>> write('buildout.cfg', """\... [buildout]... extends-cache = cache... extends = %sbase.cfg... """ % server_url)>>> print_(system(buildout))>>> ls('cache')-  5aedc98d7e769290a29d654a591a3a45>>> cat('cache', os.listdir(cache)[0])[buildout]parts =

A change tobase.cfg is picked up on the next buildout run:

>>> write(server_data, 'base.cfg', """\... [buildout]... parts =... foo = bar... """)>>> print_(system(buildout + " -n"))Unused options for buildout: 'foo'.>>> cat('cache', os.listdir(cache)[0])[buildout]parts =foo = bar

In contrast, when not usingnewest mode (newest = false), the filesalready present in the extends cache will not be updated:

>>> write(server_data, 'base.cfg', """\... [buildout]... parts =... """)>>> print_(system(buildout + " -N"))Unused options for buildout: 'foo'.>>> cat('cache', os.listdir(cache)[0])[buildout]parts =foo = bar

Even when updating base configuration files with a buildout run, any givenconfiguration file will be downloaded only once during that particular run. Ifsome base configuration file is extended more than once, its cached copy isused:

>>> write(server_data, 'baseA.cfg', """\... [buildout]... extends = %sbase.cfg... foo = bar... """ % server_url)>>> write(server_data, 'baseB.cfg', """\... [buildout]... extends-cache = cache... extends = %sbase.cfg... bar = foo... """ % server_url)>>> write('buildout.cfg', """\... [buildout]... extends-cache = cache... newest = true... extends = %sbaseA.cfg %sbaseB.cfg... """ % (server_url, server_url))>>> print_(system(buildout + " -n"))Unused options for buildout: 'bar' 'foo'.

(XXX We patch download utility’s API to produce readable output for the test;a better solution would utilise the logging already done by the utility.)

>>> import zc.buildout>>> old_download = zc.buildout.download.Download.download>>> def wrapper_download(self, url, md5sum=None, path=None):...   print_("The URL %s was downloaded." % url)...   return old_download(url, md5sum, path)>>> zc.buildout.download.Download.download = wrapper_download

>>> zc.buildout.buildout.main([])The URL http://localhost/baseA.cfg was downloaded.The URL http://localhost/base.cfg was downloaded.The URL http://localhost/baseB.cfg was downloaded.Not upgrading because not running a local buildout command.Unused options for buildout: 'bar' 'foo'.

>>> zc.buildout.download.Download.download = old_download

The deprecatedextended-by option

Thebuildout section used to recognise an option namedextended-bythat was deprecated at some point and removed in the 1.5 line. Since ignoringthis option silently was considered harmful as a matter of principle, aUserError is raised if that option is encountered now:

>>> write(server_data, 'base.cfg', """\... [buildout]... parts =... extended-by = foo.cfg... """)>>> print_(system(buildout))While:  Initializing.Error: No-longer supported "extended-by" option found in http://localhost/base.cfg.

Clean up

We should have cleaned up all temporary files created by downloading things:

>>> ls(tempfile.tempdir)

Reset the global temporary directory:

>>> tempfile.tempdir = old_tempdir

zc.buildout.testing.buildoutSetUp(test)

The buildoutSetup function can be used as a doctest setup function.It creates a sample buildout that can be used by tests, changing thecurrent working directory to the sample_buildout. It also adds anumber of names to the test namespace:

sample_buildout

This is the name of a buildout with a basic configuration.

buildout

This is the path of the buildout script in the sample buildout.

ls(*path)

List the contents of a directory. The directory path is provided as one ormore strings, to be joined with os.path.join.

cat(*path)

Display the contents of a file. The file path is provided as one ormore strings, to be joined with os.path.join.

On Windows, if the file doesn’t exist, the function will tryadding a ‘-script.py’ suffix. This helps to work around adifference in script generation on windows.

mkdir(*path)

Create a directory. The directory path is provided as one ormore strings, to be joined with os.path.join.

rmdir(*path)

Remove a directory. The directory path is provided as one ormore strings, to be joined with os.path.join.

remove(*path)

Remove a directory or file. The path is provided as one ormore strings, to be joined with os.path.join.

tmpdir(name)

Create a temporary directory with the given name. The directorywill be automatically removed at the end of the test. The path ofthe created directory is returned.

Further, if the the normalize_path normlaizing substitution (seebelow) is used, then any paths starting with this path will benormalized to:

/name/restofpath

No two temporary directories can be created with the same name. Adirectory created with tmpdir can be removed with rmdir and recreated.

Note that the sample_buildout directory is created by calling thisfunction.

write(*path_and_contents)

Create a file. The file path is provided as one or more strings,to be joined with os.path.join. The last argument is the file contents.

system(command,input='')

Execute a system command with the given input passed to thecommand’s standard input. The output (error and regular output)from the command is returned.

get(url)

Get a web page.

cd(*path)

Change to the given directory. The directory path is provided as one ormore strings, to be joined with os.path.join.

The directory will be reset at the end of the test.

uncd()

Change to the directory that was current prior to the previouscall tocd. You can callcd multiple times and thenuncd the same number of times to return to the same location.

join(*path)

A convenient reference to os.path.join.

register_teardown(func)

Register a tear-down function. The function will be called withno arguments at the end of the test.

start_server(path)

Start a web server on the given path. The server will be shutdown at the end of the test. The server URL is returned.

You can cause the server to start and stop logging it’s outputusing:

>>> get(server_url+'enable_server_logging')

and:

>>> get(server_url+'disable_server_logging')

This can be useful to see how buildout is interacting with aserver.

sdist(setup, dest)

Create a source distribution by running the given setup file andplacing the result in the given destination directory. If thesetup argument is a directory, the thge setup.py file in thatdirectory is used.

bdist_egg(setup, dest)

Create an egg by running the given setup fileand placing the result in the given destinationdirectory. If the setup argument is a directory, then thesetup.py file in that directory is used.

zc.buildout.testing.buildoutTearDown(test)

Tear down everything set up by zc.buildout.testing.buildoutSetUp. Anyfunctions passed to register_teardown are called as well.

install(project, destination)

Install eggs for a given project into a destination. If thedestination is a test object, then the eggs directory of thesample buildout (sample_buildout) defined by the test will be used.Tests will use this to install the distributions for the packagesbeing tested (and their dependencies) into a sample buildout. The eggto be used should already be loaded, by importing one of the modulesprovided, before calling this function.

install_develop(project, destination)

Like install, but a develop egg is installed even if the current eggif not a develop egg.

Output normalization

Recipe tests often generate output that is dependent on temporary filelocations, operating system conventions or Python versions. To dealwith these dependencies, we often usezope.testing.renormalizing.RENormalizing to normalize test output.zope.testing.renormalizing.RENormalizing takes pairs of regularexpressions and substitutions. The zc.buildout.testing module providesa few helpful variables that define regular-expression/substitutionpairs that you can pass to zope.testing.renormalizing.RENormalizing.

normalize_path

Converts tests paths, based on directories created with tmpdir(),to simple paths.

normalize_script

On Unix-like systems, scripts are implemented in single fileswithout suffixes. On windows, scripts are implemented with 2files, a -script.py file and a .exe file. This normalizationconverts directory listings of Windows scripts to the formgenerated on UNix-like systems.

normalize_egg_py

Normalize Python version and platform indicators, if specified, inegg names.

Distribution installation

The easy_install module provides a function, install, for installing oneor more packages and their dependencies. The install function takes 2positional arguments:

• An iterable of setuptools requirement strings for the distributionsto be installed, and

• A destination directory to install to and to satisfy requirementsfrom. The destination directory can be None, in which case, no newdistributions are downloaded and there will be an error if theneeded distributions can’t be found among those already installed.

It supports a number of optional keyword arguments:

links

A sequence of URLs, file names, or directories to look forlinks to distributions.

index

The URL of an index server, or almost any other valid URL. :)

If not specified, the Python Package Index,http://pypi.python.org/simple/, is used. You can specify analternate index with this option. If you use the links option andif the links point to the needed distributions, then the index canbe anything and will be largely ignored. In the examples, here,we’ll just point to an empty directory on our link server. Thiswill make our examples run a little bit faster.

path

A list of additional directories to search for locally-installeddistributions.

working_set

An existing working set to be augmented with additionaldistributions, if necessary to satisfy requirements. This allowsyou to call install multiple times, if necessary, to gathermultiple sets of requirements.

newest

A boolean value indicating whether to search for new distributionswhen already-installed distributions meet the requirement. Whenthis is true, the default, and when the destination directory isnot None, then the install function will search for the newestdistributions that satisfy the requirements.

versions

A dictionary mapping project names to version numbers to be usedwhen selecting distributions. This can be used to specify a set ofdistribution versions independent of other requirements.

use_dependency_links

A flag indicating whether to search for dependencies using thesetup dependency_links metadata or not. If true, links are searchedfor using dependency_links in preference to otherlocations. Defaults to true.

relative_paths

Adjust egg paths so they are relative to the script path. Thisallows scripts to work when scripts and eggs are moved, as long asthey are both moved in the same way.

The install method returns a working set containing the distributionsneeded to meet the given requirements.

We have a link server that has a number of eggs:

>>> print_(get(link_server), end='')<html><body><a href="bigdemo-0.1-py2.4.egg">bigdemo-0.1-py2.4.egg</a><br><a href="demo-0.1-py2.4.egg">demo-0.1-py2.4.egg</a><br><a href="demo-0.2-py2.4.egg">demo-0.2-py2.4.egg</a><br><a href="demo-0.3-py2.4.egg">demo-0.3-py2.4.egg</a><br><a href="demo-0.4c1-py2.4.egg">demo-0.4c1-py2.4.egg</a><br><a href="demoneeded-1.0.zip">demoneeded-1.0.zip</a><br><a href="demoneeded-1.1.zip">demoneeded-1.1.zip</a><br><a href="demoneeded-1.2c1.zip">demoneeded-1.2c1.zip</a><br><a href="du_zipped-1.0-pyN.N.egg">du_zipped-1.0-pyN.N.egg</a><br><a href="extdemo-1.4.zip">extdemo-1.4.zip</a><br><a href="index/">index/</a><br><a href="other-1.0-py2.4.egg">other-1.0-py2.4.egg</a><br></body></html>

Let’s make a directory and install the demo egg to it, using the demo:

>>> dest = tmpdir('sample-install')>>> import zc.buildout.easy_install>>> ws = zc.buildout.easy_install.install(...     ['demo==0.2'], dest,...     links=[link_server], index=link_server+'index/')

We requested version 0.2 of the demo distribution to be installed intothe destination server. We specified that we should search for linkson the link server and that we should use the (empty) link serverindex directory as a package index.

The working set contains the distributions we retrieved.

>>> for dist in ws:...     print_(dist)demo 0.2demoneeded 1.1

We got demoneeded because it was a dependency of demo.

And the actual eggs were added to the eggs directory.

>>> ls(dest)d  demo-0.2-py2.4.eggd  demoneeded-1.1-py2.4.egg

If we remove the version restriction on demo, but specify a falsevalue for newest, no new distributions will be installed:

>>> ws = zc.buildout.easy_install.install(...     ['demo'], dest, links=[link_server], index=link_server+'index/',...     newest=False)>>> ls(dest)d  demo-0.2-py2.4.eggd  demoneeded-1.1-py2.4.egg

If we leave off the newest option, we’ll get an update for demo:

>>> ws = zc.buildout.easy_install.install(...     ['demo'], dest, links=[link_server], index=link_server+'index/')>>> ls(dest)d  demo-0.2-py2.4.eggd  demo-0.3-py2.4.eggd  demoneeded-1.1-py2.4.egg

Note that we didn’t get the newest versions available. There wererelease candidates for newer versions of both packages. By default,final releases are preferred. We can change this behavior using theprefer_final function:

>>> zc.buildout.easy_install.prefer_final(False)True

The old setting is returned.

>>> ws = zc.buildout.easy_install.install(...     ['demo'], dest, links=[link_server], index=link_server+'index/')>>> for dist in ws:...     print_(dist)demo 0.4c1demoneeded 1.2c1

>>> ls(dest)d  demo-0.2-py2.4.eggd  demo-0.3-py2.4.eggd  demo-0.4c1-py2.4.eggd  demoneeded-1.1-py2.4.eggd  demoneeded-1.2c1-py2.4.egg

Let’s put the setting back to the default.

>>> zc.buildout.easy_install.prefer_final(True)False

We can supply additional distributions. We can also supplyspecifications for distributions that would normally be found viadependencies. We might do this to specify a specific version.

>>> ws = zc.buildout.easy_install.install(...     ['demo', 'other', 'demoneeded==1.0'], dest,...     links=[link_server], index=link_server+'index/')

>>> for dist in ws:...     print_(dist)demo 0.3other 1.0demoneeded 1.0

>>> ls(dest)d  demo-0.2-py2.4.eggd  demo-0.3-py2.4.eggd  demo-0.4c1-py2.4.eggd  demoneeded-1.0-py2.4.eggd  demoneeded-1.1-py2.4.eggd  demoneeded-1.2c1-py2.4.eggd  other-1.0-py2.4.egg

>>> rmdir(dest)

Specifying version information independent of requirements

Sometimes it’s useful to specify version information independent ofnormal requirements specifications. For example, a buildout may needto lock down a set of versions, without having to put put versionnumbers in setup files or part definitions. If a dictionary is passedto the install function, mapping project names to version numbers,then the versions numbers will be used.

>>> ws = zc.buildout.easy_install.install(...     ['demo'], dest, links=[link_server], index=link_server+'index/',...     versions = dict(demo='0.2', demoneeded='1.0'))>>> [d.version for d in ws]['0.2', '1.0']

In this example, we specified a version for demoneeded, even though wedidn’t define a requirement for it. The versions specified apply todependencies as well as the specified requirements.

If we specify a version that’s incompatible with a requirement, thenwe’ll get an error:

>>> from zope.testing.loggingsupport import InstalledHandler>>> handler = InstalledHandler('zc.buildout.easy_install')>>> import logging>>> logging.getLogger('zc.buildout.easy_install').propagate = False

>>> ws = zc.buildout.easy_install.install(...     ['demo >0.2'], dest, links=[link_server],...     index=link_server+'index/',...     versions = dict(demo='0.2', demoneeded='1.0'))Traceback (most recent call last):...IncompatibleConstraintError: Bad constraint 0.2 demo>0.2

>>> print_(handler)zc.buildout.easy_install DEBUG  Installing 'demo >0.2'.zc.buildout.easy_install ERROR  The constraint, 0.2, is not consistent with the requirement, 'demo>0.2'.

>>> handler.clear()

If no versions are specified, a debugging message will be outputreporting that a version was picked automatically:

>>> ws = zc.buildout.easy_install.install(...     ['demo'], dest, links=[link_server], index=link_server+'index/',...     )

>>> print_(handler) # doctest: +ELLIPSISzc.buildout.easy_install DEBUG  Installing 'demo'.zc.buildout.easy_install INFO  Getting distribution for 'demo'.zc.buildout.easy_install INFO  Got demo 0.3.zc.buildout.easy_install DEBUG  Picked: demo = 0.3zc.buildout.easy_install DEBUG  Getting required 'demoneeded'zc.buildout.easy_install DEBUG    required by demo 0.3.zc.buildout.easy_install INFO  Getting distribution for 'demoneeded'.zc.buildout.easy_install DEBUG  Running easy_install:...zc.buildout.easy_install INFO  Got demoneeded 1.1.zc.buildout.easy_install DEBUG  Picked: demoneeded = 1.1

zc.buildout.easy_install DEBUG

Installing ‘demo’.

zc.buildout.easy_install DEBUG

We have the best distribution that satisfies ‘demo’.

zc.buildout.easy_install DEBUG

Picked: demo = 0.3

zc.buildout.easy_install DEBUG

Getting required ‘demoneeded’

zc.buildout.easy_install DEBUG

required by demo 0.3.

zc.buildout.easy_install DEBUG

We have the best distribution that satisfies ‘demoneeded’.

zc.buildout.easy_install DEBUG

Picked: demoneeded = 1.1

>>> handler.uninstall()>>> logging.getLogger('zc.buildout.easy_install').propagate = True

We can request that we get an error if versions are picked:

>>> zc.buildout.easy_install.allow_picked_versions(False)True

(The old setting is returned.)

>>> ws = zc.buildout.easy_install.install(...     ['demo'], dest, links=[link_server], index=link_server+'index/',...     )Traceback (most recent call last):...UserError: Picked: demo = 0.3

>>> zc.buildout.easy_install.allow_picked_versions(True)False

The function default_versions can be used to get and set defaultversion information to be used when no version information is passes.If called with an argument, it sets the default versions:

>>> zc.buildout.easy_install.default_versions(dict(demoneeded='1'))... # doctest: +ELLIPSIS{...}

It always returns the previous default versions. If called without anargument, it simply returns the default versions without changingthem:

>>> zc.buildout.easy_install.default_versions(){'demoneeded': '1'}

So with the default versions set, we’ll get the requested version evenif the versions option isn’t used:

>>> ws = zc.buildout.easy_install.install(...     ['demo'], dest, links=[link_server], index=link_server+'index/',...     )

>>> [d.version for d in ws]['0.3', '1.0']

Of course, we can unset the default versions by passing an emptydictionary:

>>> zc.buildout.easy_install.default_versions({}){'demoneeded': '1'}

>>> ws = zc.buildout.easy_install.install(...     ['demo'], dest, links=[link_server], index=link_server+'index/',...     )

>>> [d.version for d in ws]['0.3', '1.1']

Dependency links

Setuptools allows metadata that describes where to search for packagedependencies. This option is called dependency_links. Buildout has itsown notion of where to look for dependencies, but it also uses thesetup tools dependency_links information if it’s available.

Let’s demo this by creating an egg that specifies dependency_links.

To begin, let’s create a new egg repository. This repository hold anewer version of the ‘demoneeded’ egg than the sample repository does.

>>> repoloc = tmpdir('repo')>>> from zc.buildout.tests import create_egg>>> create_egg('demoneeded', '1.2', repoloc)>>> link_server2 = start_server(repoloc)

Turn on logging on this server so that we can see when eggs are pulledfrom it.

>>> _ = get(link_server2 + 'enable_server_logging')GET 200 /enable_server_logging

Now we can create an egg that specifies that its dependencies arefound on this server.

>>> repoloc = tmpdir('repo2')>>> create_egg('hasdeps', '1.0', repoloc,...            install_requires = "'demoneeded'",...            dependency_links = [link_server2])

Let’s add the egg to another repository.

>>> link_server3 = start_server(repoloc)

Now let’s install the egg.

>>> example_dest = tmpdir('example-install')>>> workingset = zc.buildout.easy_install.install(...     ['hasdeps'], example_dest,...     links=[link_server3], index=link_server3+'index/')GET 200 /GET 200 /demoneeded-1.2-pyN.N.egg

The server logs show that the dependency was retrieved from the serverspecified in the dependency_links.

Now let’s see what happens if we provide two different ways to retrievethe dependencies.

>>> rmdir(example_dest)>>> example_dest = tmpdir('example-install')>>> workingset = zc.buildout.easy_install.install(...     ['hasdeps'], example_dest, index=link_server+'index/',...     links=[link_server, link_server3])GET 200 /GET 200 /demoneeded-1.2-pyN.N.egg

Once again the dependency is fetched from the logging server eventhough it is also available from the non-logging server. This isbecause the version on the logging server is newer and buildoutnormally chooses the newest egg available.

If you wish to control where dependencies come from regardless ofdependency_links setup metadata use the ‘use_dependency_links’ optionto zc.buildout.easy_install.install().

>>> rmdir(example_dest)>>> example_dest = tmpdir('example-install')>>> workingset = zc.buildout.easy_install.install(...     ['hasdeps'], example_dest, index=link_server+'index/',...     links=[link_server, link_server3],...     use_dependency_links=False)

Notice that this time the dependency egg is not fetched from thelogging server. When you specify not to use dependency_links, eggswill only be searched for using the links you explicitly provide.

Another way to control this option is with thezc.buildout.easy_install.use_dependency_links() function. Thisfunction sets the default behavior for the zc.buildout.easy_install()function.

>>> zc.buildout.easy_install.use_dependency_links(False)True

The function returns its previous setting.

>>> rmdir(example_dest)>>> example_dest = tmpdir('example-install')>>> workingset = zc.buildout.easy_install.install(...     ['hasdeps'], example_dest, index=link_server+'index/',...     links=[link_server, link_server3])

It can be overridden by passing a keyword argument to the installfunction.

>>> rmdir(example_dest)>>> example_dest = tmpdir('example-install')>>> workingset = zc.buildout.easy_install.install(...     ['hasdeps'], example_dest, index=link_server+'index/',...     links=[link_server, link_server3],...     use_dependency_links=True)GET 200 /demoneeded-1.2-pyN.N.egg

To return the dependency_links behavior to normal call the function again.

>>> zc.buildout.easy_install.use_dependency_links(True)False>>> rmdir(example_dest)>>> example_dest = tmpdir('example-install')>>> workingset = zc.buildout.easy_install.install(...     ['hasdeps'], example_dest, index=link_server+'index/',...     links=[link_server, link_server3])GET 200 /demoneeded-1.2-pyN.N.egg

Script generation

The easy_install module provides support for creating scripts fromeggs. It provides a function similar to setuptools except that itprovides facilities for baking a script’s path into the script. Thishas two advantages:

• The eggs to be used by a script are not chosen at run time, makingstartup faster and, more importantly, deterministic.

• The script doesn’t have to import pkg_resources because the logicthat pkg_resources would execute at run time is executed atscript-creation time.

The scripts method can be used to generate scripts. Let’s create adestination directory for it to place them in:

>>> import tempfile>>> bin = tmpdir('bin')

Now, we’ll use the scripts method to generate scripts in this directoryfrom the demo egg:

>>> import sys>>> scripts = zc.buildout.easy_install.scripts(...     ['demo'], ws, sys.executable, bin)

the three arguments we passed were:

1. A sequence of distribution requirements. These are of the sameform as setuptools requirements. Here we passed a singlerequirement, for the version 0.1 demo distribution.

2. A working set,

3. The destination directory.

The bin directory now contains a generated script:

>>> ls(bin)-  demo

The return value is a list of the scripts generated:

>>> import os, sys>>> if sys.platform == 'win32':...     scripts == [os.path.join(bin, 'demo.exe'),...                 os.path.join(bin, 'demo-script.py')]... else:...     scripts == [os.path.join(bin, 'demo')]True

Note that in Windows, 2 files are generated for each script. A scriptfile, ending in ‘-script.py’, and an exe file that allows the scriptto be invoked directly without having to specify the Pythoninterpreter and without having to provide a ‘.py’ suffix.

The demo script run the entry point defined in the demo egg:

>>> cat(bin, 'demo') # doctest: +NORMALIZE_WHITESPACE#!/usr/local/bin/python2.7<BLANKLINE>import syssys.path[0:0] = [  '/sample-install/demo-0.3-py2.4.egg',  '/sample-install/demoneeded-1.1-py2.4.egg',  ]<BLANKLINE>import eggrecipedemo<BLANKLINE>if __name__ == '__main__':    sys.exit(eggrecipedemo.main())

Some things to note:

• The demo and demoneeded eggs are added to the beginning of sys.path.

• The module for the script entry point is imported and the entrypoint, in this case, ‘main’, is run.

Rather than requirement strings, you can pass tuples containing 3strings:

• A script name,

• A module,

• An attribute expression for an entry point within the module.

For example, we could have passed entry point information directlyrather than passing a requirement:

>>> scripts = zc.buildout.easy_install.scripts(...     [('demo', 'eggrecipedemo', 'main')], ws,...     sys.executable, bin)

>>> cat(bin, 'demo') # doctest: +NORMALIZE_WHITESPACE#!/usr/local/bin/python2.7<BLANKLINE>import syssys.path[0:0] = [  '/sample-install/demo-0.3-py2.4.egg',  '/sample-install/demoneeded-1.1-py2.4.egg',  ]<BLANKLINE>import eggrecipedemo<BLANKLINE>if __name__ == '__main__':    sys.exit(eggrecipedemo.main())

Passing entry-point information directly is handy when using eggs (ordistributions) that don’t declare their entry points, such asdistributions that aren’t based on setuptools.

The interpreter keyword argument can be used to generate a script that canbe used to invoke the Python interactive interpreter with the path setbased on the working set. This generated script can also be used torun other scripts with the path set on the working set:

>>> scripts = zc.buildout.easy_install.scripts(...     ['demo'], ws, sys.executable, bin, interpreter='py')

>>> ls(bin)-  demo-  py

>>> if sys.platform == 'win32':...     scripts == [os.path.join(bin, 'demo.exe'),...                 os.path.join(bin, 'demo-script.py'),...                 os.path.join(bin, 'py.exe'),...                 os.path.join(bin, 'py-script.py')]... else:...     scripts == [os.path.join(bin, 'demo'),...                 os.path.join(bin, 'py')]True

The py script simply runs the Python interactive interpreter withthe path set:

>>> cat(bin, 'py') # doctest: +NORMALIZE_WHITESPACE#!/usr/local/bin/python2.7<BLANKLINE>import sys<BLANKLINE>sys.path[0:0] = [  '/sample-install/demo-0.3-pyN.N.egg',  '/sample-install/demoneeded-1.1-pyN.N.egg',  ]<BLANKLINE>_interactive = Trueif len(sys.argv) > 1:    _options, _args = __import__("getopt").getopt(sys.argv[1:], 'ic:m:')    _interactive = False    for (_opt, _val) in _options:        if _opt == '-i':            _interactive = True        elif _opt == '-c':            exec(_val)        elif _opt == '-m':            sys.argv[1:] = _args            _args = []            __import__("runpy").run_module(                 _val, {}, "__main__", alter_sys=True)<BLANKLINE>    if _args:        sys.argv[:] = _args        __file__ = _args[0]        del _options, _args        __file__f = open(__file__)        exec(compile(__file__f.read(), __file__, "exec"))        __file__f.close(); del __file__f<BLANKLINE>if _interactive:    del _interactive    __import__("code").interact(banner="", local=globals())

If invoked with a script name and arguments, it will run that script, instead.

>>> write('ascript', r'''... "demo doc"... import sys... print_ = lambda *a: sys.stdout.write(' '.join(map(str, a))+'\n')... print_(sys.argv)... print_((__name__, __file__, __doc__))... ''')>>> print_(system(join(bin, 'py')+' ascript a b c'), end='')['ascript', 'a', 'b', 'c']('__main__', 'ascript', 'demo doc')

For Python 2.5 and higher, you can also use the -m option to run amodule:

>>> if sys.version_info < (2, 5):...    print ('usage: pdb.py blah blah blah')... else:...    print_(system(join(bin, 'py')+' -m pdb'), end='')... # doctest: +ELLIPSISusage: pdb.py ...

>>> print_(system(join(bin, 'py')+' -m pdb what'), end='')Error: what does not exist

An additional argument can be passed to define which scripts to installand to provide script names. The argument is a dictionary mappingoriginal script names to new script names.

>>> bin = tmpdir('bin2')>>> scripts = zc.buildout.easy_install.scripts(...     ['demo'], ws, sys.executable, bin, dict(demo='run'))

>>> if sys.platform == 'win32':...     scripts == [os.path.join(bin, 'run.exe'),...                 os.path.join(bin, 'run-script.py')]... else:...     scripts == [os.path.join(bin, 'run')]True>>> ls(bin)-  run

>>> print_(system(os.path.join(bin, 'run')), end='')3 1

The scripts that are generated are made executable:

>>> if sys.platform == 'win32':...     os.access(os.path.join(bin, 'run.exe'), os.X_OK)... else:...     os.access(os.path.join(bin, 'run'), os.X_OK)True

Including extra paths in scripts

We can pass a keyword argument, extra paths, to cause additional pathsto be included in the a generated script:

>>> foo = tmpdir('foo')>>> scripts = zc.buildout.easy_install.scripts(...    ['demo'], ws, sys.executable, bin, dict(demo='run'),...    extra_paths=[foo])

>>> cat(bin, 'run') # doctest: +NORMALIZE_WHITESPACE#!/usr/local/bin/python2.7<BLANKLINE>import syssys.path[0:0] = [  '/sample-install/demo-0.3-py2.4.egg',  '/sample-install/demoneeded-1.1-py2.4.egg',  '/foo',  ]<BLANKLINE>import eggrecipedemo<BLANKLINE>if __name__ == '__main__':    sys.exit(eggrecipedemo.main())

Providing script arguments

An “argument” keyword argument can be used to pass arguments to anentry point. The value passed is a source string to be placed between theparentheses in the call:

>>> scripts = zc.buildout.easy_install.scripts(...    ['demo'], ws, sys.executable, bin, dict(demo='run'),...    arguments='1, 2')

>>> cat(bin, 'run') # doctest: +NORMALIZE_WHITESPACE#!/usr/local/bin/python2.7import syssys.path[0:0] = [  '/sample-install/demo-0.3-py2.4.egg',  '/sample-install/demoneeded-1.1-py2.4.egg',  ]<BLANKLINE>import eggrecipedemo<BLANKLINE>if __name__ == '__main__':    sys.exit(eggrecipedemo.main(1, 2))

Passing initialization code

You can also pass script initialization code:

>>> scripts = zc.buildout.easy_install.scripts(...    ['demo'], ws, sys.executable, bin, dict(demo='run'),...    arguments='1, 2',...    initialization='import os\nos.chdir("foo")',...    interpreter='py')

>>> cat(bin, 'run') # doctest: +NORMALIZE_WHITESPACE#!/usr/local/bin/python2.7import syssys.path[0:0] = [  '/sample-install/demo-0.3-py2.4.egg',  '/sample-install/demoneeded-1.1-py2.4.egg',  ]<BLANKLINE>import osos.chdir("foo")<BLANKLINE>import eggrecipedemo<BLANKLINE>if __name__ == '__main__':    sys.exit(eggrecipedemo.main(1, 2))

It will be included in interpreters too:

>>> cat(bin, 'py') # doctest: +NORMALIZE_WHITESPACE +ELLIPSIS#!/usr/local/bin/python2.7<BLANKLINE>import sys<BLANKLINE>sys.path[0:0] = [  '/sample-install/demo-0.3-py3.3.egg',  '/sample-install/demoneeded-1.1-py3.3.egg',  ]<BLANKLINE>import osos.chdir("foo")<BLANKLINE><BLANKLINE>_interactive = True...

Relative paths

Sometimes, you want to be able to move a buildout directory around andhave scripts still work without having to rebuild them. We cancontrol this using the relative_paths option to install. You needto pass a common base directory of the scripts and eggs:

>>> bo = tmpdir('bo')>>> ba = tmpdir('ba')>>> mkdir(bo, 'eggs')>>> mkdir(bo, 'bin')>>> mkdir(bo, 'other')

>>> ws = zc.buildout.easy_install.install(...     ['demo'], join(bo, 'eggs'), links=[link_server],...     index=link_server+'index/')

>>> scripts = zc.buildout.easy_install.scripts(...    ['demo'], ws, sys.executable, join(bo, 'bin'), dict(demo='run'),...    extra_paths=[ba, join(bo, 'bar')],...    interpreter='py',...    relative_paths=bo)

>>> cat(bo, 'bin', 'run')#!/usr/local/bin/python2.7<BLANKLINE>import os<BLANKLINE>join = os.path.joinbase = os.path.dirname(os.path.abspath(os.path.realpath(__file__)))base = os.path.dirname(base)<BLANKLINE>import syssys.path[0:0] = [  join(base, 'eggs/demo-0.3-pyN.N.egg'),  join(base, 'eggs/demoneeded-1.1-pyN.N.egg'),  '/ba',  join(base, 'bar'),  ]<BLANKLINE>import eggrecipedemo<BLANKLINE>if __name__ == '__main__':    sys.exit(eggrecipedemo.main())

Note that the extra path we specified that was outside the directorypassed as relative_paths wasn’t converted to a relative path.

Of course, running the script works:

>>> print_(system(join(bo, 'bin', 'run')), end='')3 1

We specified an interpreter and its paths are adjusted too:

>>> cat(bo, 'bin', 'py') # doctest: +NORMALIZE_WHITESPACE#!/usr/local/bin/python2.7<BLANKLINE>import os<BLANKLINE>join = os.path.joinbase = os.path.dirname(os.path.abspath(os.path.realpath(__file__)))base = os.path.dirname(base)<BLANKLINE>import sys<BLANKLINE>sys.path[0:0] = [  join(base, 'eggs/demo-0.3-pyN.N.egg'),  join(base, 'eggs/demoneeded-1.1-pyN.N.egg'),  '/ba',  join(base, 'bar'),  ]<BLANKLINE>_interactive = Trueif len(sys.argv) > 1:    _options, _args = __import__("getopt").getopt(sys.argv[1:], 'ic:m:')    _interactive = False    for (_opt, _val) in _options:        if _opt == '-i':            _interactive = True        elif _opt == '-c':            exec(_val)        elif _opt == '-m':            sys.argv[1:] = _args            _args = []            __import__("runpy").run_module(                 _val, {}, "__main__", alter_sys=True)<BLANKLINE>    if _args:        sys.argv[:] = _args        __file__ = _args[0]        del _options, _args        __file__f = open(__file__)        exec(compile(__file__f.read(), __file__, "exec"))        __file__f.close(); del __file__f<BLANKLINE>if _interactive:    del _interactive    __import__("code").interact(banner="", local=globals())

Installing distutils-style scripts

Most python libraries use the console_scripts entry point nowadays. Butseveral still have ascripts=['bin/something'] in their setup() call.Buildout also installs those:

>>> distdir = tmpdir('distutilsscriptdir')>>> distbin = tmpdir('distutilsscriptbin')>>> ws = zc.buildout.easy_install.install(...     ['other'], distdir,...     links=[link_server], index=link_server+'index/')>>> scripts = zc.buildout.easy_install.scripts(...     ['other'], ws, sys.executable, distbin)>>> ls(distbin)-  distutilsscript

Like for console_scripts, the output is a list of the scriptsgenerated. Likewise, on windows two files, an.exe and a script with-script.py appended, are generated:

>>> import os, sys>>> if sys.platform == 'win32':...     scripts == [os.path.join(distbin, 'distutilsscript.exe'),...                 os.path.join(distbin, 'distutilsscript-script.py')]... else:...     scripts == [os.path.join(distbin, 'distutilsscript')]True

It also works for zipped eggs:

>>> distdir2 = tmpdir('distutilsscriptdir2')>>> distbin2 = tmpdir('distutilsscriptbin2')>>> ws = zc.buildout.easy_install.install(...     ['du_zipped'], distdir2,...     links=[link_server], index=link_server+'index/')>>> scripts = zc.buildout.easy_install.scripts(...     ['du_zipped'], ws, sys.executable, distbin2)>>> ls(distbin2)-  distutilsscript

Distutils copies the script files verbatim, apart from a line at the top thatlooks like#!/usr/bin/python, which gets replaced by the actual pythoninterpreter. Buildout does the same, but additionally also adds the sys.pathlike for the console_scripts. Note that the second line in the scripts cancontain a source encoding hint; buildout retains it.

>>> cat(distbin, 'distutilsscript')#!/usr/local/bin/python2.7# -*- coding: utf-8 -*-<BLANKLINE>import syssys.path[0:0] = [  '/distutilsscriptdir/other-1.0-pyN.N.egg',  ]<BLANKLINE><BLANKLINE>import sys; sys.stdout.write("distutils!\n")

Due to the nature of distutils scripts, buildout cannot pass arguments asthere’s no specific method to pass them to.

In some cases, a python 3__pycache__ directory can end up in an internalEGG-INFO metadata directory, next to the script information we’re lookingfor. Buildout doesn’t crash on that:

>>> eggname = [name for name in os.listdir(distdir2)...            if name.endswith('egg')][0]>>> scripts_metadata_dir = os.path.join(...     distdir2, eggname, 'EGG-INFO', 'scripts')>>> os.mkdir(os.path.join(scripts_metadata_dir, '__dummy__'))>>> scripts = zc.buildout.easy_install.scripts(...     ['du_zipped'], ws, sys.executable, distbin2)>>> ls(distbin2)-  distutilsscript

Handling custom build options for extensions provided in source distributions

Sometimes, we need to control how extension modules are built. Thebuild function provides this level of control. It takes a singlepackage specification, downloads a source distribution, and builds itwith specified custom build options.

The build function takes 3 positional arguments:

spec

A package specification for a source distribution

dest

A destination directory

build_ext

A dictionary of options to be passed to the distutils build_extcommand when building extensions.

It supports a number of optional keyword arguments:

links

a sequence of URLs, file names, or directories to look forlinks to distributions,

index

The URL of an index server, or almost any other valid URL. :)

If not specified, the Python Package Index,http://pypi.python.org/simple/, is used. You can specify analternate index with this option. If you use the links option andif the links point to the needed distributions, then the index canbe anything and will be largely ignored. In the examples, here,we’ll just point to an empty directory on our link server. Thiswill make our examples run a little bit faster.

path

A list of additional directories to search for locally-installeddistributions.

newest

A boolean value indicating whether to search for new distributionswhen already-installed distributions meet the requirement. Whenthis is true, the default, and when the destination directory isnot None, then the install function will search for the newestdistributions that satisfy the requirements.

versions

A dictionary mapping project names to version numbers to be usedwhen selecting distributions. This can be used to specify a set ofdistribution versions independent of other requirements.

Our link server included a source distribution that includes a simpleextension, extdemo.c:

#include <Python.h>#include <extdemo.h>static PyMethodDef methods[] = {};PyMODINIT_FUNCinitextdemo(void){    PyObject *m;    m = Py_InitModule3("extdemo", methods, "");#ifdef TWO    PyModule_AddObject(m, "val", PyInt_FromLong(2));#else    PyModule_AddObject(m, "val", PyInt_FromLong(EXTDEMO));#endif}

The extension depends on a system-dependent include file, extdemo.h,that defines a constant, EXTDEMO, that is exposed by the extension.

We’ll add an include directory to our sample buildout and add theneeded include file to it:

>>> mkdir('include')>>> write('include', 'extdemo.h',... """... #define EXTDEMO 42... """)

Now, we can use the build function to create an egg from the sourcedistribution:

>>> zc.buildout.easy_install.build(...   'extdemo', dest,...   {'include-dirs': os.path.join(sample_buildout, 'include')},...   links=[link_server], index=link_server+'index/')['/sample-install/extdemo-1.4-py2.4-unix-i686.egg']

The function returns the list of eggs

Now if we look in our destination directory, we see we have an extdemo egg:

>>> ls(dest)d  demo-0.2-py2.4.eggd  demo-0.3-py2.4.eggd  demoneeded-1.0-py2.4.eggd  demoneeded-1.1-py2.4.eggd  extdemo-1.4-py2.4-unix-i686.egg

Let’s update our link server with a new version of extdemo:

>>> update_extdemo()>>> print_(get(link_server), end='')<html><body><a href="bigdemo-0.1-py2.4.egg">bigdemo-0.1-py2.4.egg</a><br><a href="demo-0.1-py2.4.egg">demo-0.1-py2.4.egg</a><br><a href="demo-0.2-py2.4.egg">demo-0.2-py2.4.egg</a><br><a href="demo-0.3-py2.4.egg">demo-0.3-py2.4.egg</a><br><a href="demo-0.4c1-py2.4.egg">demo-0.4c1-py2.4.egg</a><br><a href="demoneeded-1.0.zip">demoneeded-1.0.zip</a><br><a href="demoneeded-1.1.zip">demoneeded-1.1.zip</a><br><a href="demoneeded-1.2c1.zip">demoneeded-1.2c1.zip</a><br><a href="du_zipped-1.0-pyN.N.egg">du_zipped-1.0-pyN.N.egg</a><br><a href="extdemo-1.4.zip">extdemo-1.4.zip</a><br><a href="extdemo-1.5.zip">extdemo-1.5.zip</a><br><a href="index/">index/</a><br><a href="other-1.0-py2.4.egg">other-1.0-py2.4.egg</a><br></body></html>

The easy_install caches information about servers to reduce networkaccess. To see the update, we have to call the clear_index_cachefunction to clear the index cache:

>>> zc.buildout.easy_install.clear_index_cache()

If we run build with newest set to False, we won’t get an update:

>>> zc.buildout.easy_install.build(...   'extdemo', dest,...   {'include-dirs': os.path.join(sample_buildout, 'include')},...   links=[link_server], index=link_server+'index/',...   newest=False)['/sample-install/extdemo-1.4-py2.4-linux-i686.egg']

>>> ls(dest)d  demo-0.2-py2.4.eggd  demo-0.3-py2.4.eggd  demoneeded-1.0-py2.4.eggd  demoneeded-1.1-py2.4.eggd  extdemo-1.4-py2.4-unix-i686.egg

But if we run it with the default True setting for newest, then we’llget an updated egg:

>>> zc.buildout.easy_install.build(...   'extdemo', dest,...   {'include-dirs': os.path.join(sample_buildout, 'include')},...   links=[link_server], index=link_server+'index/')['/sample-install/extdemo-1.5-py2.4-unix-i686.egg']

>>> ls(dest)d  demo-0.2-py2.4.eggd  demo-0.3-py2.4.eggd  demoneeded-1.0-py2.4.eggd  demoneeded-1.1-py2.4.eggd  extdemo-1.4-py2.4-unix-i686.eggd  extdemo-1.5-py2.4-unix-i686.egg

The versions option also influences the versions used. For example,if we specify a version for extdemo, then that will be used, eventhough it isn’t the newest. Let’s clean out the destination directoryfirst:

>>> import os>>> for name in os.listdir(dest):...     remove(dest, name)

>>> zc.buildout.easy_install.build(...   'extdemo', dest,...   {'include-dirs': os.path.join(sample_buildout, 'include')},...   links=[link_server], index=link_server+'index/',...   versions=dict(extdemo='1.4'))['/sample-install/extdemo-1.4-py2.4-unix-i686.egg']

>>> ls(dest)d  extdemo-1.4-py2.4-unix-i686.egg

Handling custom build options for extensions in develop eggs

The develop function is similar to the build function, except that,rather than building an egg from a source directory containing asetup.py script.

The develop function takes 2 positional arguments:

setup

The path to a setup script, typically named “setup.py”, or adirectory containing a setup.py script.

dest

The directory to install the egg link to

It supports some optional keyword argument:

build_ext

A dictionary of options to be passed to the distutils build_extcommand when building extensions.

We have a local directory containing the extdemo source:

>>> ls(extdemo)-  MANIFEST-  MANIFEST.in-  README-  extdemo.c-  setup.py

Now, we can use the develop function to create a develop egg from the sourcedistribution:

>>> zc.buildout.easy_install.develop(...   extdemo, dest,...   {'include-dirs': os.path.join(sample_buildout, 'include')})'/sample-install/extdemo.egg-link'

The name of the egg link created is returned.

Now if we look in our destination directory, we see we have an extdemoegg link:

>>> ls(dest)d  extdemo-1.4-py2.4-unix-i686.egg-  extdemo.egg-link

And that the source directory contains the compiled extension:

>>> contents = os.listdir(extdemo)>>> bool([f for f in contents if f.endswith('.so') or f.endswith('.pyd')])True

Download cache

Normally, when distributions are installed, if any processing isneeded, they are downloaded from the internet to a temporary directoryand then installed from there. A download cache can be used to avoidthe download step. This can be useful to reduce network access and tocreate source distributions of an entire buildout.

A download cache is specified by calling the download_cachefunction. The function always returns the previous setting. If noargument is passed, then the setting is unchanged. If an argument ispassed, the download cache is set to the given path, which must pointto an existing directory. Passing None clears the cache setting.

To see this work, we’ll create a directory and set it as the cachedirectory:

>>> cache = tmpdir('cache')>>> zc.buildout.easy_install.download_cache(cache)

We’ll recreate our destination directory:

>>> remove(dest)>>> dest = tmpdir('sample-install')

We’d like to see what is being fetched from the server, so we’llenable server logging:

>>> _ = get(link_server+'enable_server_logging')GET 200 /enable_server_logging

Now, if we install demo, and extdemo:

>>> ws = zc.buildout.easy_install.install(...     ['demo==0.2'], dest,...     links=[link_server], index=link_server+'index/')GET 200 /GET 404 /index/demo/GET 200 /index/GET 200 /demo-0.2-py2.4.eggGET 404 /index/demoneeded/GET 200 /demoneeded-1.1.zip

>>> zc.buildout.easy_install.build(...   'extdemo', dest,...   {'include-dirs': os.path.join(sample_buildout, 'include')},...   links=[link_server], index=link_server+'index/')GET 404 /index/extdemo/GET 200 /extdemo-1.5.zip['/sample-install/extdemo-1.5-py2.4-linux-i686.egg']

Not only will we get eggs in our destination directory:

>>> ls(dest)d  demo-0.2-py2.4.eggd  demoneeded-1.1-py2.4.eggd  extdemo-1.5-py2.4-linux-i686.egg

But we’ll get distributions in the cache directory:

>>> ls(cache)-  demo-0.2-py2.4.egg-  demoneeded-1.1.zip-  extdemo-1.5.zip

The cache directory contains uninstalled distributions, such as zippedeggs or source distributions.

Let’s recreate our destination directory and clear the index cache:

>>> remove(dest)>>> dest = tmpdir('sample-install')>>> zc.buildout.easy_install.clear_index_cache()

Now when we install the distributions:

>>> ws = zc.buildout.easy_install.install(...     ['demo==0.2'], dest,...     links=[link_server], index=link_server+'index/')GET 200 /GET 404 /index/demo/GET 200 /index/GET 404 /index/demoneeded/

>>> zc.buildout.easy_install.build(...   'extdemo', dest,...   {'include-dirs': os.path.join(sample_buildout, 'include')},...   links=[link_server], index=link_server+'index/')GET 404 /index/extdemo/['/sample-install/extdemo-1.5-py2.4-linux-i686.egg']

>>> ls(dest)d  demo-0.2-py2.4.eggd  demoneeded-1.1-py2.4.eggd  extdemo-1.5-py2.4-linux-i686.egg

Note that we didn’t download the distributions from the link server.

If we remove the restriction on demo, we’ll download a newer versionfrom the link server:

>>> ws = zc.buildout.easy_install.install(...     ['demo'], dest,...     links=[link_server], index=link_server+'index/')GET 200 /demo-0.3-py2.4.egg

Normally, the download cache is the preferred source of downloads, butnot the only one.

Installing solely from a download cache

A download cache can be used as the basis of application sourcereleases. In an application source release, we want to distribute anapplication that can be built without making any network accesses. Inthis case, we distribute a download cache and tell the easy_installmodule to install from the download cache only, without making networkaccesses. The install_from_cache function can be used to signal thatpackages should be installed only from the download cache. Thefunction always returns the previous setting. Calling it with noarguments returns the current setting without changing it:

>>> zc.buildout.easy_install.install_from_cache()False

Calling it with a boolean value changes the setting and returns theprevious setting:

>>> zc.buildout.easy_install.install_from_cache(True)False

Let’s remove demo-0.3-py2.4.egg from the cache, clear the index cache,recreate the destination directory, and reinstall demo:

>>> for  f in os.listdir(cache):...     if f.startswith('demo-0.3-'):...         remove(cache, f)

>>> zc.buildout.easy_install.clear_index_cache()>>> remove(dest)>>> dest = tmpdir('sample-install')

>>> ws = zc.buildout.easy_install.install(...     ['demo'], dest,...     links=[link_server], index=link_server+'index/')

>>> ls(dest)d  demo-0.2-py2.4.eggd  demoneeded-1.1-py2.4.egg

This time, we didn’t download from or even query the link server.

Feature Changes

• Added a configuration option that allows buildouts to ignoredependency_links metadata specified in setup. By defaultdependency_links in setup are used in addition to buildout specifiedfind-links. This can make it hard to control where eggs comefrom. Here’s how to tell buildout to ignore URLs independency_links:

  [buildout]use-dependency-links = false

  By default use-dependency-links is true, which matches the behaviorof previous versions of buildout.

• Added a configuration option that causes buildout to error if aversion is picked. This is a nice safety belt when fixing allversions is intended, especially when creating releases.

Bugs Fixed

• 151820: Develop failed if the setup.py script imported modules inthe distribution directory.

• Verbose logging of the develop command was omitting detailedoutput.

• The setup command wasn’t documented.

• The setup command failed if run in a directory without specifying aconfiguration file.

• The setup command raised a stupid exception if run without arguments.

• When using a local find links or index, distributions weren’t copiedto the download cache.

• When installing from source releases, a version specification (via abuildout versions section) for setuptools was ignored when decidingwhich setuptools to use to build an egg from the source release.

Feature Changes

• Now, final distributions are prefered over non-final versions. Ifboth final and non-final versions satisfy a requirement, then thefinal version will be used even if it is older. The normal way tooverride this for specific packages is to specifically require anon-final version, either specifically or via a lower bound.

• There is a buildout prefer-final version that can be used with avalue of “false”:

  prefer-final = false

  To prefer newer versions, regardless of whether or not they arefinal, buildout-wide.

• The new simple Python index,http://cheeseshop.python.org/simple, isused as the default index. This will provide better performancethan the human package index interface,http://pypi.python.org/pypi. More importantly, it lists hiddendistributions, so buildouts with fixed distribution versions will beable to find old distributions even if the distributions have beenhidden in the human PyPI interface.

Bugs Fixed

• 126441: Look for default.cfg in the right place on Windows.

Feature Changes

• buildout now changes to the buildout directory before running recipeinstall and update methods.

• Added a new init command for creating a new buildout. This createsan empty configuration file and then bootstraps.

• Except when using the new init command, it is now an error to runbuildout without a configuration file.

• In verbose mode, when adding distributions to fulful requirements ofalready-added distributions, we now show why the new distributionsare being added.

• Changed the logging format to exclude the logger name for thebuildout logger. This reduces noise in the output.

• Clean up lots of messages, adding missing periods and adding quotes aroundrequirement strings and file paths.

Bugs Fixed

• 114614: Buildouts could take a very long time if there weredependency problems in large sets of pathologically interdependentpackages.

• 59270: Buggy recipes can cause failures in later recipes via chdir

• 61890:file:// urls don’t seem to work in find-links

  setuptools requires that file urls that point to directories mustend in a “/”. Added a workaround.

• 75607: buildout should not run if it creates an empty buildout.cfg

Feature Changes

• Improved error reporting by showing which packages require otherpackages that can’t be found or that cause version conflicts.

• Added an API for use by recipe writers to clean up created fileswhen recipe errors occur.

• Log installed scripts.

Bugs Fixed

• 92891: bootstrap crashes with recipe option in buildout section.

• 113085: Buildout exited with a zero exist status when internal errorsoccured.

Feature Changes

• Added support for download caches. A buildout can specify a cachefor distribution downloads. The cache can be shared among buildoutsto reduce network access and to support creating sourcedistributions for applications allowing install without networkaccess.

• Log scripts created, as suggested in:https://bugs.launchpad.net/zc.buildout/+bug/71353

Bugs Fixed

• It wasn’t possible to give options on the command line for sectionsnot defined in a configuration file.

Feature Changes

• Added a buildout newest option, to control whether the newestdistributions should be sought to meet requirements. This mightalso provide a hint to recipes that don’t deal withdistributions. For example, a recipe that manages subversioncheckouts might not update a checkout if newest is set to “false”.

• Added anewest keyword parameter to thezc.buildout.easy_install.install and zc.buildout.easy_install.buildfunctions to control whether the newest distributions that meedgiven requirements should be sought. If a false value is providedfor this parameter and already installed eggs meet the givenrequirements, then no attempt will be made to search for newerdistributions.

• The recipe-testing support setUp function now adds the namebuildout to the test namespace with a value that is the path tothe buildout script in the sample buildout. This allows tests touse

  >>> print system(buildout),

  rather than:

  >>> print system(join('bin', 'buildout')),

Bugs Fixed

• Paths returned from update methods replaced lists of installed filesrather than augmenting them.

Feature Changes

• Added documentation for some previously undocumented features of theeasy_install APIs.

• By popular demand, added a -o command-line option that is a shorthand for the assignment buildout:offline=true.

Bugs Fixed

• When deciding whether recipe develop eggs had changed, buildoutincorrectly considered files in .svn and CVS directories.

Feature Changes

• Configuration files can now be loaded from URLs.

Bugs Fixed

• https://bugs.launchpad.net/products/zc.buildout/+bug/71246

  Buildout extensions installed as eggs couldn’t be loaded in offlinemode.

Feature Changes

• A new command-line argument, -U, suppresses reading user defaults.

• You can now suppress use of an installed-part database(e.g. .installed.cfg) by sprifying an empty value for the buildoutinstalled option.

Bugs Fixed

• When the install command is used with a list of parts, onlythose parts are supposed to be installed, but the buildout was alsobuilding parts that those parts depended on.

Feature Changes

• Added uninstall recipes for dealing with complex uninstallationscenarios.

Bugs Fixed

• Automatic upgrades weren’t performed on Windows due to a bug thatcaused buildout to incorrectly determine that it wasn’t runninglocally in a buildout.

• Fixed some spurious test failures on Windows.

Feature Changes

• Variable substitutions now reflect option data written by recipes.

• A part referenced by a part in a parts list is now added to the partslist before the referencing part. This means that you can omitparts from the parts list if they are referenced by other parts.

• Added a develop function to the easy_install module to aid increating develop eggs with custom build_ext options.

• The build and develop functions in the easy_install module nowreturn the path of the egg or egg link created.

• Removed the limitation that parts named in the install command canonly name configured parts.

• Removed support ConfigParser-style variable substitutions(e.g. %(foo)s). Only the string-template style of variable(e.g. ${section:option}) substitutions will be supported.Supporting both violates “there’s only one way to do it”.

• Deprecated the buildout-section extendedBy option.

Bugs Fixed

• We treat setuptools as a dependency of any distribution that(declares that it) uses namespace packages, whether it declaressetuptools as a dependency or not. This wasn’t working for eggsintalled by virtue of being dependencies.

Feature Changes

• Renamed the runsetup command to setup. (The old name still works.)

• Added a recipe update method. Now install is only called when a partis installed for the first time, or after an uninstall. Otherwise,update is called. For backward compatibility, recipes that don’tdefine update methiods are still supported.

• If a distribution defines namespace packages but fails to declaresetuptools as one of its dependencies, we now treat setuptools as animplicit dependency. We generate a warning if the distributionis a develop egg.

• You can now create develop eggs for setup scripts that don’t use setuptools.

Bugs Fixed

• Egg links weren’t removed when corresponding entries were removedfrom develop sections.

• Running a non-local buildout command (one not installed in thebuildout) ket to a hang if there were new versions of buildout orsetuptools were available. Now we issue a warning and don’tupgrade.

• When installing zip-safe eggs from local directories, the eggs weremoved, rather than copied, removing them from the source directory.