Examples#

List the available images:

archerydockerimages

Execute a build:

archerydockerrunconda-python

Archery calls the following docker compose commands:

dockercomposepull--ignore-pull-failuresconda-cppdockercomposepull--ignore-pull-failuresconda-pythondockercomposebuildconda-cppdockercomposebuildconda-pythondockercomposerun--rmconda-python

Show the Docker Compose commands instead of executing them:

archerydocker--dry-runrunconda-python

To disable the image pulling:

archerydockerrun--no-cacheconda-python

Which translates to:

dockercomposebuild--no-cacheconda-cppdockercomposebuild--no-cacheconda-pythondockercomposerun--rmconda-python

To disable the cache only for the leaf image:

Useful to force building the development version of a dependency.In case of the example below the command builds theconda-cpp>conda-python>conda-python-pandas branch of the image treewhere the leaf image isconda-python-pandas.

PANDAS=upstream_develarcherydockerrun--no-leaf-cacheconda-python-pandas

Which translates to:

exportPANDAS=upstream_develdockercomposepull--ignore-pull-failuresconda-cppdockercomposepull--ignore-pull-failuresconda-pythondockercomposebuildconda-cppdockercomposebuildconda-pythondockercomposebuild--no-cacheconda-python-pandasdockercomposerun--rmconda-python-pandas

Note that it doesn’t pull the conda-python-pandas image and disable the cachewhen building it.

PANDAS is abuild parameter, see thedefaults in the.env file.

To entirely skip building the image:

The layer-caching mechanism of docker-compose can be less reliable thandocker’s, depending on the version, thecache_from build entry, and thebackend used (docker-py, docker-cli, docker-cli and buildkit). This can lead todifferent layer hashes - even when executing the same build commandrepeatedly - eventually causing cache misses full image rebuilds.

If the image has been already built but the cache doesn’t work properly, itcan be useful to skip the build phases:

# first run ensures that the image is builtarcherydockerrunconda-python# if the second run tries the build the image again and none of the files# referenced in the relevant dockerfile have changed, then it indicates a# cache miss caused by the issue described abovearcherydockerrunconda-python# since the image is properly built with the first command, there is no# need to rebuild it, so manually disable the pull and build phases to# spare the some timearcherydockerrun--no-pull--no-buildconda-python

Pass environment variables to the container:

Most of the build scripts used within the containers can be configured throughenvironment variables. Pass them using--env or-e CLI options -similar to thedockerrun anddockercomposerun interface.

archerydockerrun--envCMAKE_BUILD_TYPE=releaseubuntu-cpp

For the available environment variables in the C++ builds see theci/scripts/cpp_build.sh script.

Run the image with custom command:

Custom docker commands may be passed as the second argument toarcherydockerrun.

The following example starts an interactivebash session in the container- useful for debugging the build interactively:

archerydockerrunubuntu-cppbash

Build the image with increased debugging output:

To enable additional logging output for debugging, pass the--debug flagtoarchery.

archery--debugdockerrunubuntu-cpp

In addition to enablingDEBUG-level logging, this also translates topassing--progress=plain to docker(-compose) build command.

Docker Volume Caches#

Most of the compose container have specific directories mounted from the hostto reuseccache andmaven artifacts. These docker volumes are placedin the.docker directory.

In order to clean up the cache simply delete one or more directories (or thewhole.docker directory).

Docker Build Parameters#

The build time parameters are pushed down to the dockerfiles to make theimage building more flexible. These parameters are usually called as dockerbuild args, but we pass these values as environment variables todocker-compose.yml. The build parameters are extensively used for:

• defining the docker registry used for caching

• platform architectures

• operation systems and versions

• defining various versions if dependencies

The default parameter values are stored in the top level .env file.For detailed examples see the docker-compose.yml.

Build Scripts#

The scripts maintained under ci/scripts directory should be keptparameterizable but reasonably minimal to clearly encapsulate the tasks it isresponsible for. Like:

• cpp_build.sh: build the C++ implementation without running the tests.

• cpp_test.sh: execute the C++ tests.

• python_build.sh: build the Python bindings without running the tests.

• python_test.sh: execute the Python tests.

• docs_build.sh: build the Sphinx documentation.

• integration_dask.sh: execute the dask integration tests.

• integration_pandas.sh: execute the pandas integration tests.

• install_minio.sh: install minio server for multiple platforms.

• install_conda.sh: install miniconda for multiple platforms.

• install_gcs_testbench.sh: install the GCS testbench for multiple platforms.

The parametrization (like the C++ CMake options) is achieved via environmentvariables with useful defaults to keep the build configurations declarative.

A good example iscpp_build.sh build script which forwards environmentvariables as CMake options - so the same scripts can be invoked in variousconfigurations without the necessity of changing it. For examples see how theenvironment variables are passed in the docker-compose.yml’s C++ images.

Adding New Images#

See the inline comments available in the docker-compose.yml file.