Choosing a parent commit

Codecov must choose a parent commit when comparing coverage reports. Having accurate coverage data is crucial. Therefore, by default,Codecov only chooses commits that have successful CI builds.

To illustrate this point, we can review the following commits:

Order of events

• CommitA was uploaded. The CI passed for this commit. Coverage did not change.
• CommitB was uploaded. The CI failed.
• CommitC was uploaded. The CI failed.
• CommitD was uploaded. The CI passed and coverage increased.

Codecov will compareA againstD because commitsB andC failed in CI, and likely have invalid or incomplete coverage data. This results in a+2% change in coverage.

Commits with Failed CI

When coverage runs on your build, it is essential that Codecov recognizes that CI has passed or failed. If CI fails, one or more of the following assumptions can be made:

1. Not all tests were executed; therefore coverage is incomplete.
2. Exceptions may call new execution paths, resulting in different coverage metrics.
3. A failed test could produce different coverage than the same test ran successfully.

Rebase vs Merge Target

There are two techniques to updating pull requests: rebasing and merging target.

Both techniques are respected by Codecov. The pull request base will be updated froma tob, and Codecov will use reports fromb when comparing against pull request head (c).

Rebasing

# git diagram - before```master a . . bpull    \ . . c```# rebase actiongit rebase master# git diagram - after```master a . . bpull          \ . . c```

Merging Target

# git diagram - before```master a . . bpull    \ . . c```# merging target actiongit merge master# git diagram - after```master a . . b               \pull    \ . . . mc c```

Comparing pull request base reports

Pseudo-Comparison

When Codecov generates a comparison for a pull-request, but the pull-request’s base in git does not have coverage information, Codecov will try to find an appropriate substitute (a “pseudo-base”). This is referred to as pseudo-comparison. Pseudo-comparison can be turned off via the Codecov YAML option:

codecov:  allow_pseudo_compare: False # the default is true

When pseudo-comparison is disallowed and the base commit of a PR has no coverage data, Codecov will render an error in the UI.

Pseudo-Comparison Example

The base commit of the pull request did not upload coverage resulting in Codecov's inability to compare reports.

master . pseudo . . basepull                  \ . . head

• pseudo did upload coverage
• basedid not upload coverage
• head did upload coverage

Examples of the use cases:

• Base commit skipped CI via[ci skip], use the parent commit as the pseudo commit below.
• Base commit was a merge commitMerge abc into xyz, usexyz as the pseudo commit below.

Codecov will seek a parent commit and offset the report to get an approximate base report.

# first get the git diff of pseudo...basepseudo_diff_base = diff(pseudo...base)# adjust the report with changes in pseudo_diff_baseapprox_base_report = pseudo.report.adjust_forward(pseudo_diff_base)# get the pull diffpull_diff = diff(base...head)use compare(approx_base_report...head.report) in pull comment/statuses/etc.

Updated about 1 year ago