R tests

Making changes to CI containers

Many of the CI pipelines use Docker containers to ensure consistent testing environmentwith a variety of software packages. We have a separate repo,dmlc/xgboost-devops, to host the logic forbuilding and publishing CI containers.

To make changes to the CI container, carry out the following steps:

1. Identify which container needs updating. Example:492475357299.dkr.ecr.us-west-2.amazonaws.com/xgb-ci.gpu:main

2. Clonedmlc/xgboost-devops and make changes to thecorresponding Dockerfile. Example:containers/dockerfile/Dockerfile.gpu.

3. Locally build the container, to ensure that the container successfully builds.ConsultReproducing CI testing environments locally for this step.

4. Submit a pull request todmlc/xgboost-devops withthe proposed changes to the Dockerfile. Make note of the pull request number. Example:#204

5. Clonedmlc/xgboost. Locate the fileops/pipeline/get-image-tag.sh, which should have a single line

   IMAGE_TAG=main

   To use the new container, revise the file as follows:

   IMAGE_TAG=PR-XX

   whereXX is the pull request number. E.g.PR-204.

6. Now submit a pull request todmlc/xgboost. The CI willrun tests using the new container. Verify that all tests pass.

7. Merge the pull request indmlc/xgboost-devops. Wait until the CI completes on themain branch.

8. Go back to the the pull request fordmlc/xgboost and changeops/pipeline/get-image-tag.shback toIMAGE_TAG=main.

9. Merge the pull request indmlc/xgboost.

Reproducing CI testing environments locally

You can reproduce the same testing environment as the CI pipelines by building and running Dockercontainers locally.

Prerequisites

1. Install Docker:https://docs.docker.com/engine/install/ubuntu/

2. Install NVIDIA Docker runtime:https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html.The runtime lets you access NVIDIA GPUs inside a Docker container.

# The following env vars are only relevant for CI# For local testing, set them to "main"exportGITHUB_SHA="main"exportBRANCH_NAME="main"bashcontainers/docker_build.shIMAGE_REPO

xgb-ci.gpu:container_def:gpubuild_args:CUDA_VERSION_ARG:"12.4.1"NCCL_VERSION_ARG:"2.23.4-1"RAPIDS_VERSION_ARG:"24.10"

--build-argCUDA_VERSION_ARG=12.4.1--build-argNCCL_VERSION_ARG=2.23.4-1 \--build-argRAPIDS_VERSION_ARG=24.10

python3ops/docker_run.py\--image-uri492475357299.dkr.ecr.us-west-2.amazonaws.com/[image_repo]:[image_tag]\[--use-gpus]\--"command to run inside the container"

# Run without GPUpython3ops/docker_run.py\--image-uri492475357299.dkr.ecr.us-west-2.amazonaws.com/xgb-ci.cpu:main\--bashops/pipeline/build-cpu-impl.shcpu# Run with NVIDIA GPUpython3ops/docker_run.py\--image-uri492475357299.dkr.ecr.us-west-2.amazonaws.com/xgb-ci.gpu:main\--use-gpus\--bashops/pipeline/test-python-wheel-impl.shgpu

# Allocate extra space in /dev/shm to enable NCCL# Also run the container with elevated privilegespython3ops/docker_run.py\--image-uri492475357299.dkr.ecr.us-west-2.amazonaws.com/xgb-ci.gpu:main\--use-gpus\--run-args='--shm-size=4g --privileged'\--bashops/pipeline/test-python-wheel-impl.shgpu

GitHub Actions

We make the extensive use ofGitHub Actions to host ourCI pipelines. Most of the tests listed in the configuration files run automatically for everyincoming pull requests and every update to branches.

Self-Hosted Runners with RunsOn

RunsOn is a SaaS (Software as a Service) app that lets us to easily createself-hosted runners to use with GitHub Actions pipelines. RunsOn usesAmazon Web Services (AWS) under the hood to provision runners withaccess to various amount of CPUs, memory, and NVIDIA GPUs. Thanks to this app, we are able to testGPU-accelerated and distributed algorithms of XGBoost while using the familar interface ofGitHub Actions.

In GitHub Actions, jobs run on Microsoft-hosted runners by default.To opt into self-hosted runners (enabled by RunsOn), we use the following special syntax:

runs-on:-runs-on-runner=runner-name-run-id=${{ github.run_id }}-tag=[unique tag that uniquely identifies the job in the GH Action workflow]

where the runner is defined in.github/runs-on.yml.

The Lay of the Land: how CI pipelines are organized in the codebase

The XGBoost project stores the configuration for its CI pipelines as part of the codebase.The git repository therefore stores not only the change history for its source code but alsothe change history for the CI pipelines.

The CI pipelines are organized into the following directories and files:

• .github/workflows/: Definition of CI pipelines, using the GitHub Actions syntax

• .github/runs-on.yml: Configuration for the RunsOn service. Specifies the spec forthe self-hosted CI runners.

• ops/conda_env/: Definitions for Conda environments

• ops/patch/: Patch files

• ops/pipeline/: Shell scripts defining CI/CD pipelines. Most of these scripts can be runlocally (to assist with development and debugging); a few must run in the CI.

• ops/script/: Various utility scripts useful for testing

• ops/docker_run.py: Wrapper script to run commands inside a container

To inspect a given CI pipeline, inspect files in the following order:

Many of the CI pipelines use Docker containers to ensure consistent testing environmentwith a variety of software packages. We have a separate repo,dmlc/xgboost-devops, thathosts the code for building the CI containers. The repository is organized as follows:

• actions/: Custom actions to be used with GitHub Actions. SeeCustom actions for GitHub Actionsfor more details.

• containers/dockerfile/: Dockerfiles to define containers

• containers/ci_container.yml: Defines the mapping between Dockerfiles and containers.Also specifies the build arguments to be used with each container.

• containers/docker_build.{py,sh}: Wrapper scripts to build and test CI containers.

• vm_images/: Defines bootstrap scripts to build VM images for Amazon EC2. SeeNotes on VM images to learn about how VM images relate to container images.

SeeReproducing CI testing environments locally to learn about the utility scripts for building andusing containers.

Artifact sharing between jobs via Amazon S3

We make artifacts from one workflow job available to another job, by uploading theartifacts toAmazon S3. In the CI, we utilize thescriptops/pipeline/manage-artifacts.py to coordinate artifact sharing.

To upload files to S3: In the workflow YAML, add the following lines:

-name:Upload files to S3run:|REMOTE_PREFIX="remote directory to place the artifact(s)"python3 ops/pipeline/manage-artifacts.py upload \--s3-bucket ${{ env.RUNS_ON_S3_BUCKET_CACHE }} \--prefix cache/${{ github.run_id }}/${REMOTE_PREFIX} \path/to/file

The--prefix argument specifies the remote directory in which the artifact(s)should be placed. The artifact(s) will be placed ins3://{RUNS_ON_S3_BUCKET_CACHE}/cache/{GITHUB_RUN_ID}/{REMOTE_PREFIX}/whereRUNS_ON_S3_BUCKET_CACHE andGITHUB_RUN_ID are set by the CI.

You can upload multiple files, possibly with wildcard globbing:

-name:Upload files to S3run:|python3 ops/pipeline/manage-artifacts.py upload \--s3-bucket ${{ env.RUNS_ON_S3_BUCKET_CACHE }} \--prefix cache/${{ github.run_id }}/build-cuda \build/testxgboost python-package/dist/*.whl

To download files from S3: In the workflow YAML, add the following lines:

-name:Download files from S3run:|REMOTE_PREFIX="remote directory where the artifact(s) were placed"python3 ops/pipeline/manage-artifacts.py download \--s3-bucket ${{ env.RUNS_ON_S3_BUCKET_CACHE }} \--prefix cache/${{ github.run_id }}/${REMOTE_PREFIX} \--dest-dir path/to/destination_directory \artifacts

You can also use the wildcard globbing. The script will locate all artifactsunder the given prefix that matches the wildcard pattern.

-name:Download files from S3run:|# Locate all artifacts with name *.whl under prefix# cache/${GITHUB_RUN_ID}/${REMOTE_PREFIX} and# download them to wheelhouse/.python3 ops/pipeline/manage-artifacts.py download \--s3-bucket ${{ env.RUNS_ON_S3_BUCKET_CACHE }} \--prefix cache/${{ github.run_id }}/${REMOTE_PREFIX} \--dest-dir wheelhouse/ \*.whl

Custom actions for GitHub Actions

XGBoost implements a few customcomposite actionsto reduce duplicated code within workflow YAML files. The custom actions are hosted in a separate repository,dmlc/xgboost-devops, to make it easy to test changes to the custom actions ina pull request or a fork.

In a workflow file, we’d refer todmlc/xgboost-devops/actions/{custom-action}@main. For example:

-uses:dmlc/xgboost-devops/actions/miniforge-setup@mainwith:environment-name:cpp_testenvironment-file:ops/conda_env/cpp_test.yml

Each custom action consists of two components:

• Main script (dmlc/xgboost-devops/actions/{custom-action}/action.yml): dispatches to a specific versionof the implementation script (see the next item). The main script clonesxgboost-devops froma specified fork at a particular ref, allowing us to easily test changes to the custom action.

• Implementation script (dmlc/xgboost-devops/actions/impls/{custom-action}/action.yml): Implements thecustom script.

This design was inspired by Mike Sarahan’s work inrapidsai/shared-actions.

Infra for building and publishing CI containers and VM images

# docker_build.sh calls docker_build.py...python3containers/docker_build.py--container-defgpu\--image-uri492475357299.dkr.ecr.us-west-2.amazonaws.com/xgb-ci.gpu:main\--build-argCUDA_VERSION_ARG=12.4.1--build-argNCCL_VERSION_ARG=2.23.4-1\--build-argRAPIDS_VERSION_ARG=24.10...# .. and docker_build.py in turn calls "docker build"...dockerbuild--build-argCUDA_VERSION_ARG=12.4.1\--build-argNCCL_VERSION_ARG=2.23.4-1\--build-argRAPIDS_VERSION_ARG=24.10\--load--progress=plain\--ulimitnofile=1024000:1024000\-t492475357299.dkr.ecr.us-west-2.amazonaws.com/xgb-ci.gpu:main\-fcontainers/dockerfile/Dockerfile.gpu\containers/

# Run without GPUpython3ops/docker_run.py\--image-uri492475357299.dkr.ecr.us-west-2.amazonaws.com/xgb-ci.cpu:main\--bashops/pipeline/build-cpu-impl.shcpu# Run with NVIDIA GPU# Allocate extra space in /dev/shm to enable NCCL# Also run the container with elevated privilegespython3ops/docker_run.py\--image-uri492475357299.dkr.ecr.us-west-2.amazonaws.com/xgb-ci.gpu:main\--use-gpus\--run-args='--shm-size=4g --privileged'\--bashops/pipeline/test-python-wheel-impl.shgpu

dockerrun--rm--pid=host\-w/workspace-v/path/to/xgboost:/workspace\-eCI_BUILD_UID=<uid>-eCI_BUILD_USER=<user_name>\-eCI_BUILD_GID=<gid>-eCI_BUILD_GROUP=<group_name>\492475357299.dkr.ecr.us-west-2.amazonaws.com/xgb-ci.cpu:main\bashops/pipeline/build-cpu-impl.shcpudockerrun--rm--pid=host--gpusall\-w/workspace-v/path/to/xgboost:/workspace\-eCI_BUILD_UID=<uid>-eCI_BUILD_USER=<user_name>\-eCI_BUILD_GID=<gid>-eCI_BUILD_GROUP=<group_name>\--shm-size=4g--privileged\492475357299.dkr.ecr.us-west-2.amazonaws.com/xgb-ci.gpu:main\bashops/pipeline/test-python-wheel-impl.shgpu