Code Coverage

The “Code coverage” value of a codebase indicates how much of the production/development code is covered by the running unit tests. Maintainers try their best to keep this percentage high, and this process is often automated using tools such asGitHub Actions andCodecov. Hence, code coverage becomes a good metric (not always) to check if a particular codebase is well tested and reliable.

Tools and libraries used to calculate, read, and visualize coverage reports:

• coverage (AKAcoverage.py): a tool to calculate and visualize Python coverage
• pytest-cov: an integration ofcoverage withpytest
• Codecov: integrates with remote repositories, allowing developers to see and compare coverage value with each CI run
• GitHub Actions: allows users to automatically upload coverage reports toCodecov

Are there any alternatives?

Coveralls is an alternative coverage platform, but we recommend using Codecov because of its ease of use and integration with GitHub Actions.

Should increasing the coverage value be my top priority?

A low coverage percentage will definitely motivate you to add more tests, but adding weak tests just for coverage’s sake is not a good idea. The tests should test your codebase thoroughly and should not be unreliable.

Running your tests with coverage

There are two common ways to calculate coverage: usingcoverage or usingpytest-cov. Whilepytest-cov is simpler on the command line, and it promises nice integration with features likepytest-xdist, in practice it tends to be more rigid and have more issues than runningcoverage directly. If you are using a task runner likenox, the extra complexity of running coverage directly is hidden away from normal use, so we recommend that, but will show both methods below. If you are not runningpytest, but instead are running an example or a script, you have to usecoverage directly.

coverage run-m pytest

coverage report

python-m pytest-ra--cov=vector--cov-config=pyproject.toml

Configuring coverage

There is a configuration section inpyproject.toml for coverage. Here are some common options(see the docs for more):

[tool.coverage]run.core="sysmon"run.disable_warnings=["no-sysmon"]run.relative_files=truerun.source_pkgs=["package"]report.show_missing=true

Settingrun.core tosysmon will make coverage much faster on 3.12+ without branch coverage, or 3.14+ if you are using branch coverage (see below). To avoid a warning on older Pythons withoutsysmon, you need to add theno-sysmon code torun.disable_warnings. If you want relative paths reported,relative_files=true does that. Andsource_pkgs is one way to tell coverage which packages are to be monitored for coverage. You can also setrun.source_dirs. You can usereport.show_missing to show the missing line number ranges.

If you want to include branches in your coverage, setrun.branch = true. If you are using subprocess calls, you can add the subprocess patching mechanism withrun.patch = [ "subprocess" ]; though keep in mind, this behaves like (and enables)parallel=True; you’ll need to combine coverage files after running.pytest-cov sort of does this combine for you, sometimes.

There are also useful reporting options.report.exclude_lines = [...] allows you to exclude lines from coverage.report.fail_under can trigger a failure if coverage is below a percent (like 100).

Calculating code coverage in your workflows

Your workflows should produce a.coverage file as outlined above. This file can be uploaded toCodecov using thecodecov/codecov-action action.

If you would rather do it yourself, you should collect coverage files from all your jobs and combine them into one.coverage file before runningcoverage report, so that you get a combined score.

Manually combining coverage

If you are running in parallel, either withpytest-xdist, you can setrun.parallel totrue, which will add a unique suffix to the coverage file(s) produced. If you are using the multiprocessing patch, that also will generate a unique suffix for each process launch.

You can control the prefix with the environment variableCOVERAGE_FILE, which defaults to.coverage. This is mostly commonly used to add your own custom suffixes, likeCOVERAGE_FILE=.coverage.win32.py312. This is how you can manually produce multiple files from task runner jobs.

Here’s an example nox job:

@nox.session(python=ALL_PYTHONS)deftests(session:nox.Session)->None:coverage_file=f".coverage.{sys.platform}.{session.python}"session.install("-e.","--group=cov")session.run("coverage","run","-m","pytest",*session.posargs,env={"COVERAGE_FILE":coverage_file},)

@nox.session(python=ALL_PYTHONS)deftests(session:nox.Session)->None:coverage_file=f".coverage.{sys.platform}.{session.python}"session.install("-e.","--group=cov")session.run("pytest","--cov=<package_name>","--cov-config=pyproject.toml",*session.posargs,env={"COVERAGE_FILE":coverage_file},)

Merging and reporting

If you are running in multiple jobs, you should use upload/download artifacts so they are all available in a single combine job at the end. Each one should have a unique suffix. Then you just needcoverage combine, which will combine all the files into a single.coverage file which you can use withcoverage report. You can even report in markdown format and write it to the GitHub Actions summary if you want.

Here’s an example in nox:

@nox.session(default=False,requires=["tests"])defcoverage(session:nox.Session)->None:session.install("coverage[toml]")session.run("coverage","combine")session.run("coverage","report")session.run("coverage","erase")

Configuring Codecov and uploading coverage reports

Interestingly,Codecov does not require any initial configurations for your project, given that you have already signed up for the same using your GitHub account.Codecov requires you to push or upload your coverage report, after which it automatically generates aCodecov project for you.

Codecov maintains thecodecov/codecov-action GitHub Action to make uploading coverage reports easy for users. A minimal working example for uploading coverage reports through your workflow, which should be more than enough for a simple testing suite, can be written as follows:

-name:Upload coverage reportuses:codecov/codecov-action@v5with:token:${{ secrets.CODECOV_TOKEN }}

The lines above should be added after the step that runs your tests with the--cov option. See thedocs for all the optional options. You’ll need to specify aCODECOV_TOKEN secret, as well.

Using codecov.yml

One can also configureCodecov and coverage reports passed toCodecov usingcodecov.yml.codecov.yml should be placed inside the.github folder, along with yourworkflows folder. Additionally,Codecov allows you to create and edit thisYAML file directly through yourCodecov project’s settings!

A recommended configuration for.github/codecov.yml:

codecov:notify:after_n_builds:xcoverage:status:project:default:target:autothreshold:5%patch:default:informational:true

wherex is the number of uploaded reportsCodecov should wait to receive before sending statuses. This would ensure that theCodecov checks don’t fail before all the coverage reports are uploaded. You can control the levels which are considered failing; the config above sets a loss of up to 5% as okay, and avoids patch coverage reporting a failure (otherwise, just changing a single uncovered line could cause a “failure” report on the PR). If you have 100% coverage, then you can remove the coverage failure settings, as you want any loss of coverage to fail. See thedocs for all the options.