Repository files navigation

Therust-vmm-ci repository containsintegration testsandBuildkite pipeline definitions that are used forrunning the CI for all rust-vmm crates.

Having a centralized place for the tests is one of the enablers for keeping thesame quality standard for all crates in rust-vmm.

To run the integration tests defined in the pipeline as part of the CI:

1. Add rust-vmm-ci as a git submodule to your repository

# Add rust-vmm-ci as a submodule. This will point to the latest rust-vmm-ci# commit from the main branch. The following command will also add a# `.gitmodules` file and the `rust-vmm-ci` to the index.git submodule add https://github.com/rust-vmm/rust-vmm-ci.git# Commit the changes to your repository so that the CI can run using the# rust-vmm-ci pipeline and tests.git commit -s -m"Added rust-vmm-ci as submodule"

2. Create the coverage test configuration file namedcoverage_config_ARCH.json in the root of the repository, whereARCH is thearchitecture of the machine.There are two coverage test configuration files, one per each platform.The example of the configuration file for thex86_64 architecture can befound incoverage_config_x86_64.json.sample,and the example of the configuration file for theaarch64 architecture can befound incoverage_config_aarch64.json.sample.

The json must have the following fields:

• coverage_score: The coverage of the repository.
• exclude_path: This field is used for excluding files from the report. Itshould be used to exclude autogenerated files. Files inexclude_path areseparated by one comma. If the repository does not have any autogeneratedfiles,exclude_path should be an empty string.
• crate_features:cargo kcov does not build crate features by default. Toget the coverage report including optional features, these need to bespecified incrate_features separated by comma. If the crate does not haveany features, this field should be empty.

This file is required for the coverage integration so it needs to be addedto the repository as well.

3. Create a new pipeline definition in Buildkite. For this step ask one of therust-vmm Buildkiteadmins to create one for you. The process is explainedhere.

4. There is a script that autogenerates a dynamic Buildkite pipeline. Each stepin the pipeline has a default timeout of 5 minutes. To run the CI using this dynamic pipeline,you need to add a step that is uploading the rust-vmm-ci pipeline:

./rust-vmm-ci/.buildkite/autogenerate_pipeline.py| buildkite-agent pipeline upload

This allows overriding some values and extending others through environmentvariables.

• X86_LINUX_AGENT_TAGS: overrides the tags by which the x86_64 linux agent isselected; the default values are

  {"os": "linux", "platform": "x86.metal"}

• AARCH64_LINUX_AGENT_TAGS: overrides the tags by which the aarch64 linuxagent is selected. The default values are

  {"os": "linux", "platform": "arm.metal"}

• DOCKER_PLUGIN_CONFIG: specifies additional configuration for the dockerplugin. For available configuration, please check thehttps://github.com/buildkite-plugins/docker-buildkite-plugin.

• TESTS_TO_SKIP: specifies a list of tests to be skipped.

• TIMEOUTS_MIN: overrides the timeout value for specific tests.

• DEFAULT_AGENT_TAG_HYPERVISOR: sets the hypervisor on which all the tests inthe pipeline run. By default, the selected hypervisor is KVM because thehosts running KVM at the time of this change showed better performance andexperienced timeouts less often. NOTE: This will not override the hypervisordefined at the test step level. If a test already defines a hypervisor tagthat will remain intact.

The variableTESTS_TO_SKIP is specified as a JSON list with the namesof the tests to be skipped. The variableTIMEOUTS_MIN is a dictionary whereeach key is the name of a test and each value is the number of minutes for thetimeout. The other variables are specified as dictionaries, where the first keyistests and its value is a list of test names where the configuration shouldbe applied; the second key iscfg and its value is a dictionary with theactual configuration.

For example, we can skip the testcommit-format, have a timeout of 30 minutesfor the teststyle and extend the docker plugin specification as follows:

TESTS_TO_SKIP='["commit-format"]' TIMEOUTS_MIN='{"style": 30}' DOCKER_PLUGIN_CONFIG='{    "tests": ["coverage"],    "cfg": {        "devices": [ "/dev/vhost-vdpa-0" ],        "privileged": true    }}' ./rust-vmm-ci/.buildkite/autogenerate_pipeline.py| buildkite-agent pipeline upload

For most use cases, overriding or extending the configuration is not necessary. We maywant to do so if, for example, the platform needs a custom device that is not availableon the existing test instances or if we need a specialized hypervisor.

5. The code owners of the repository will have to setup a WebHook fortriggering the CI onpull requestandpushevents.

TheBuildkite pipeline is the definition of tests tobe run as part of the CI. It includes steps for running unit tests and linters(including coding style checks), and computing the coverage.

Currently the tests can run on Linuxx86_64 andaarch64 hosts.

Example of step that checks the build:

steps:-label:build-gnu-x86_64command:cargo build --releaseretry:automatic:falseagents:os:linuxplatform:x86_64.metalplugins:  -docker#v3.8.0:image:rustvmm/dev:v16always-pull:truetimeout_in_minutes:5

To see all steps in the pipeline check the output of the.buildkite/autogenerate_pipeline.py script.

Some crates might need to test functionality that is specific to thatparticular component and thus cannot be added to the common pipeline.

In this situation, the repositories need to create a JSON file with a customtest configuration. The preferred path is.buildkite/custom-tests.json.

For example to test the build with one non-defaultfeatureenabled, the following configuration can be added:

{"tests": [    {"test_name":"build-bzimage","command":"cargo build --release --features bzimage","platform": ["x86_64"      ]    }  ]}

To run this custom pipeline, you need to add a step that is uploading it in Buildkite. The samescript that autogenerates the main pipeline can be used with the option-t PATH_TO_CUSTOM_CONFIGURATION:

./rust-vmm-ci/.buildkite/autogenerate_pipeline.py -t .buildkite/custom-tests.json| buildkite-agent pipeline upload

In addition to the one-liner tests defined in theBuildkite Pipeline, the rust-vmm-ci also has morecomplex tests defined inintegration_tests.

The integration tests support two test profiles:

• devel: this is the recommended profile for running the integration testson a local development machine.
• ci (default option): this is the profile used when running theintegration tests as part of the the Continuous Integration (CI).

The test profiles are applicable topytest, the integration test frameworkused with rust-vmm-ci. Currently only thecoverage test follows this model as all the otherintegration tests are run using the Buildkite pipeline.

The difference between is declaring tests as passed or failed:

• with thedevel profile the coverage test passes if the current coverageis equal or higher than the upstream coverage value. In case the currentcoverage is higher, the coverage file is updated to the new coverage value.
• with theci profile the coverage test passes only if the current coverageis equal to the upstream coverage value.

Further details about the coverage test can be found in theAdaptive Coverage section.

The line coverage is saved intests/coverage. To update thecoverage before submitting a PR, run the coverage test:

CRATE="kvm-ioctls"# NOTE: This might not be the latest container version, you can check which one we're using# by looking into the .buildkite/autogenerate_pipeline.py file.LATEST=16docker run --device=/dev/kvm \           -it \           --security-opt seccomp=unconfined \           --volume$(pwd)/${CRATE}:/${CRATE} \           rustvmm/dev:v${LATEST}cd${crate}pytest --profile=devel rust-vmm-ci/integration_tests/test_coverage.py

If the PR coverage is higher than the upstream coverage, the coverage fileneeds to be manually added to the commit before submitting the PR:

git add tests/coverage

Failing to do so will generate a fail on the CI pipeline when publishing thePR.

NOTE: The coverage file is only updated in thedevel test profile. Intheci profile the coverage test will fail if the current coverage is higherthan the coverage reported intests/coverage.

rust-vmm-ci includes an integration test that can run a battery ofbenchmarks at every pull request, comparing the results with the tip of theupstreammain branch. The test is not included in the default Buildkitepipeline. Each crate that requires the test to be run as part of the CI mustadd acustom pipeline.

An example of a pipeline that runs the test for ARM platforms and prints theresults:

steps:-label:bench-aarch64command:pytest rust-vmm-ci/integration_tests/test_benchmark.py -sretry:automatic:falseagents:os:linuxplatform:arm.metalplugins:  -docker#v3.8.0:image:rustvmm/dev:v16always-pull:true

The test requirescriterionbenchmarks to be exported by the crate. The test expects the entry pointinto the performance benchmarks to be namedmain. In other words, thefollowing configuration is expected inCargo.toml:

[[bench]]name ="main"

All benchmarks need to be collected in a main.rs file placed inbenches/.

criterion collects performance results by running a function for auser-configured number of iterations, timing the runs, and applying statistics.The individual benchmark tests must be added in the crate. They can be runoutside the CI with:

cargo bench [--all-features] OR [--features<features>]

rust-vmm-ci usescritcmp tocompare the results yielded bycargo bench --all-features on the PR beingtested with those from the tip of the upstreammain branch. The testrunscargo bench twice, once on the currentHEAD, then again aftergit checkout origin/main.critcmp takes care of the comparison, makinguse ofcriterion's stable format foroutput files.The results are printed tostdout and can be visually inspected in thepipeline output. In its present form, the test cannot fail.

To run the test locally:

docker run --device=/dev/kvm \           -it \           --security-opt seccomp=unconfined \           --volume$(pwd)/${CRATE}:/${CRATE} \           rustvmm/dev:v${LATEST}cd${CRATE}pytest rust-vmm-ci/integration_tests/test_benchmark.py -s

Note that performance is highly dependent on the underlying platform that thetests are running on. The raw numbers obtained are likely to differ from theircounterparts on a CI instance.

To run the integration tests locally, you can run the following from the crate you need to test.You can find the latest container version in thescriptthat autogenerates the pipeline. For example:

cd~/vm-superioCRATE="vm-superio"# NOTE: This might not be the latest container version, you can check which one we're using# by looking into the .buildkite/autogenerate_pipeline.py file.LATEST=16docker run -it \           --security-opt seccomp=unconfined \           --volume$(pwd):/${CRATE} \           --volume~/.ssh:/root/.ssh \           rustvmm/dev:v${LATEST}cd vm-superio./rust-vmm-ci/test_run.py

Known issues:

• When running thecargo-audit test, the following error may occur:

test_cargo-audit (__main__.TestsContainer) ... error: couldn’t fetch advisory database: git operation failed: reference ‘refs/heads/main’ not found; class=Reference (4); code=NotFound (-3)

A fix for this is to remove~/.cargo/advisory-db in the container, and then reruntest_run.py:

rm -rf ~/.cargo/advisory-db./rust-vmm-ci/test_run.py