Introduction¶

This document explains how to use clang’s source-based code coverage feature.It’s called “source-based” because it operates on AST and preprocessorinformation directly. This allows it to generate very precise coverage data.

Clang ships two other code coverage implementations:

• SanitizerCoverage - A low-overhead tool meant for use alongside thevarious sanitizers. It can provide up to edge-level coverage.

• gcov - A GCC-compatible coverage implementation which operates on DebugInfo.This is enabled by-ftest-coverage or--coverage.

From this point onwards “code coverage” will refer to the source-based kind.

The code coverage workflow¶

The code coverage workflow consists of three main steps:

• Compiling with coverage enabled.

• Running the instrumented program.

• Creating coverage reports.

The next few sections work through a complete, copy-‘n-paste friendly examplebased on this program:

%cat<<EOF>foo.cc#define BAR(x) ((x) || (x))template<typenameT>voidfoo(Tx){for(unsignedI=0;I<10;++I){BAR(I);}}intmain(){foo<int>(0);foo<float>(0);return0;}EOF

Compiling with coverage enabled¶

To compile code with coverage enabled, pass-fprofile-instr-generate-fcoverage-mapping to the compiler:

#Step1:Compilewithcoverageenabled.%clang++-fprofile-instr-generate-fcoverage-mappingfoo.cc-ofoo

Note that linking together code with and without coverage instrumentation issupported. Uninstrumented code simply won’t be accounted for in reports.

To compile code with Modified Condition/Decision Coverage (MC/DC) enabled,pass-fcoverage-mcdc in addition to the clang options specified above.MC/DC is an advanced form of code coverage most applicable in the embeddedspace.

Running the instrumented program¶

The next step is to run the instrumented program. When the program exits itwill write araw profile to the path specified by theLLVM_PROFILE_FILEenvironment variable. If that variable does not exist, the profile is writtentodefault.profraw in the current directory of the program. IfLLVM_PROFILE_FILE contains a path to a non-existent directory, the missingdirectory structure will be created. Additionally, the following specialpattern strings are rewritten:

• “%p” expands out to the process ID.

• “%h” expands out to the hostname of the machine running the program.

• “%t” expands out to the value of theTMPDIR environment variable. OnDarwin, this is typically set to a temporary scratch directory.

• “%Nm” expands out to the instrumented binary’s signature. When this patternis specified, the runtime creates a pool of N raw profiles which are used foron-line profile merging. The runtime takes care of selecting a raw profilefrom the pool, locking it, and updating it before the program exits. If N isnot specified (i.e the pattern is “%m”), it’s assumed thatN=1. Themerge pool specifier can only occur once per filename pattern.

• “%b” expands out to the binary ID (build ID). It can be used with “%Nm” toavoid binary signature collisions. To use it, the program should be compiledwith the build ID linker option (--build-id for GNU ld or LLD,/build-id for lld-link on Windows). Linux, Windows and AIX are supported.

• “%c” expands out to nothing, but enables a mode in which profile counterupdates are continuously synced to a file. This means that if theinstrumented program crashes, or is killed by a signal, perfect coverageinformation can still be recovered. Continuous mode does not support valueprofiling for PGO, and is only supported on Darwin at the moment. Support forLinux may be mostly complete but requires testing, and support for Windowsmay require more extensive changes: please get involved if you are interestedin porting this feature.

#Step2:Runtheprogram.%LLVM_PROFILE_FILE="foo.profraw"./foo

Note that continuous mode is also used on Fuchsia where it’s the only supportedmode, but the implementation is different. The Darwin and Linux implementationrelies on padding and the ability to map a file over the existing memorymapping which is generally only available on POSIX systems and isn’t suitablefor other platforms.

On Fuchsia, we rely on the ability to relocate counters at runtime using alevel of indirection. On every counter access, we add a bias to the counteraddress. This bias is stored in__llvm_profile_counter_bias symbol that’sprovided by the profile runtime and is initially set to zero, meaning norelocation. The runtime can map the profile into memory at arbitrary locations,and set bias to the offset between the original and the new counter location,at which point every subsequent counter access will be to the new location,which allows updating profile directly akin to the continuous mode.

The advantage of this approach is that doesn’t require any special OS support.The disadvantage is the extra overhead due to additional instructions requiredfor each counter access (overhead both in terms of binary size and performance)plus duplication of counters (i.e. one copy in the binary itself and anothercopy that’s mapped into memory). This implementation can be also enabled forother platforms by passing the-runtime-counter-relocation option to thebackend during compilation.

For a program such as theLittesting tool which invokes other programs, it may be necessary to setLLVM_PROFILE_FILE for each invocation. The pattern strings “%p” or “%Nm”may help to avoid corruption due to concurrency. Note that “%p” is also a Littoken and needs to be escaped as “%%p”.

%clang++-fprofile-instr-generate-fcoverage-mapping-mllvm-runtime-counter-relocationfoo.cc-ofoo

Creating coverage reports¶

Raw profiles have to beindexed before they can be used to generatecoverage reports. This is done using the “merge” tool inllvm-profdata(which can combine multiple raw profiles and index them at the same time):

#Step3(a):Indextherawprofile.%llvm-profdatamerge-sparsefoo.profraw-ofoo.profdata

For an example of merging multiple profiles created by testing,see the LLVMcoverage build script.

There are multiple different ways to render coverage reports. The simplestoption is to generate a line-oriented report:

#Step3(b):Createaline-orientedcoveragereport.%llvm-covshow./foo-instr-profile=foo.profdata

This report includes a summary view as well as dedicated sub-views fortemplated functions and their instantiations. For our example program, we getdistinct views forfoo<int>(...) andfoo<float>(...). If-show-line-counts-or-regions is enabled,llvm-cov displays sub-lineregion counts (even in macro expansions):

    1|   20|#define BAR(x) ((x) || (x))                           ^20     ^2    2|    2|template <typename T> void foo(T x) {    3|   22|  for (unsigned I = 0; I < 10; ++I) { BAR(I); }                                   ^22     ^20  ^20^20    4|    2|}------------------| void foo<int>(int):|      2|    1|template <typename T> void foo(T x) {|      3|   11|  for (unsigned I = 0; I < 10; ++I) { BAR(I); }|                                     ^11     ^10  ^10^10|      4|    1|}------------------| void foo<float>(int):|      2|    1|template <typename T> void foo(T x) {|      3|   11|  for (unsigned I = 0; I < 10; ++I) { BAR(I); }|                                     ^11     ^10  ^10^10|      4|    1|}------------------

If--show-branches=count and--show-expansions are also enabled, thesub-views will show detailed branch coverage information in addition to theregion counts:

------------------| void foo<float>(int):|      2|    1|template <typename T> void foo(T x) {|      3|   11|  for (unsigned I = 0; I < 10; ++I) { BAR(I); }|                                     ^11     ^10  ^10^10|  ------------------|  |  |    1|     10|#define BAR(x) ((x) || (x))|  |  |                             ^10     ^1|  |  |  ------------------|  |  |  |  Branch (1:17): [True: 9, False: 1]|  |  |  |  Branch (1:24): [True: 0, False: 1]|  |  |  ------------------|  ------------------|  |  Branch (3:23): [True: 10, False: 1]|  ------------------|      4|    1|}------------------

If the application was instrumented for Modified Condition/Decision Coverage(MC/DC) using the clang option-fcoverage-mcdc, an MC/DC subview can beenabled using--show-mcdc that will show detailed MC/DC information foreach complex condition boolean expression containing at most six conditions.

To generate a file-level summary of coverage statistics instead of aline-oriented report, try:

#Step3(c):Createacoveragesummary.%llvm-covreport./foo-instr-profile=foo.profdataFilename           Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover     Branches    Missed Branches     Cover--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------/tmp/foo.cc             13                 0   100.00%           3                 0   100.00%          13                 0   100.00%           12                  2    83.33%--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------TOTAL                   13                 0   100.00%           3                 0   100.00%          13                 0   100.00%           12                  2    83.33%

Thellvm-cov tool supports specifying a custom demangler, writing outreports in a directory structure, and generating html reports. For the fulllist of options, please refer to thecommand guide.

A few final notes:

• The-sparse flag is optional but can result in dramatically smallerindexed profiles. This option should not be used if the indexed profile willbe reused for PGO.

• Raw profiles can be discarded after they are indexed. Advanced use of theprofile runtime library allows an instrumented program to merge profilinginformation directly into an existing raw profile on disk. The details areout of scope.

• Thellvm-profdata tool can be used to merge together multiple raw orindexed profiles. To combine profiling data from multiple runs of a program,try e.g:

  %llvm-profdatamerge-sparsefoo1.profrawfoo2.profdata-ofoo3.profdata

Exporting coverage data¶

Coverage data can be exported into JSON using thellvm-covexportsub-command. There is a comprehensive reference which defines the structure ofthe exported data at a high level in the llvm-cov source code.

Interpreting reports¶

There are six statistics tracked in a coverage summary:

• Function coverage is the percentage of functions which have been executed atleast once. A function is considered to be executed if any of itsinstantiations are executed.

• Instantiation coverage is the percentage of function instantiations whichhave been executed at least once. Template functions and static inlinefunctions from headers are two kinds of functions which may have multipleinstantiations. This statistic is hidden by default in reports, but can beenabled via the-show-instantiation-summary option.

• Line coverage is the percentage of code lines which have been executed atleast once. Only executable lines within function bodies are considered to becode lines.

• Region coverage is the percentage of code regions which have been executed atleast once. A code region may span multiple lines (e.g in a large functionbody with no control flow). However, it’s also possible for a single line tocontain multiple code regions (e.g in “return x || y && z”).

• Branch coverage is the percentage of “true” and “false” branches that havebeen taken at least once. Each branch is tied to individual conditions in thesource code that may each evaluate to either “true” or “false”. Theseconditions may comprise larger boolean expressions linked by boolean logicaloperators. For example, “x = (y == 2) || (z < 10)” is a boolean expressionthat is comprised of two individual conditions, each of which evaluates toeither true or false, producing four total branch outcomes.

• Modified Condition/Decision Coverage (MC/DC) is the percentage of individualbranch conditions that have been shown to independently affect the decisionoutcome of the boolean expression they comprise. This is accomplished usingthe analysis of executed control flow through the expression (i.e. testvectors) to show that as a condition’s outcome is varied between “true” andfalse”, the decision’s outcome also varies between “true” and false”, whilethe outcome of all other conditions is held fixed (or they are masked out asunevaluatable, as happens in languages whose logical operators haveshort-circuit semantics). MC/DC builds on top of branch coverage andrequires that all code blocks and all execution paths have been tested. Thisstatistic is hidden by default in reports, but it can be enabled via the-show-mcdc-summary option as long as code was also compiled using theclang option-fcoverage-mcdc.

  • Boolean expressions that are only comprised of one condition (and thereforehave no logical operators) are not included in MC/DC analysis and aretrivially deducible using branch coverage.

Of these six statistics, function coverage is usually the least granular whilebranch coverage (with MC/DC) is the most granular. 100% branch coverage for afunction implies 100% region coverage for a function. The project-wide totalsfor each statistic are listed in the summary.

Format compatibility guarantees¶

• There are no backwards or forwards compatibility guarantees for the rawprofile format. Raw profiles may be dependent on the specific compilerrevision used to generate them. It’s inadvisable to store raw profiles forlong periods of time.

• Tools must retainbackwards compatibility with indexed profile formats.These formats are not forwards-compatible: i.e, a tool which uses formatversion X will not be able to understand format version (X+k).

• Tools must also retainbackwards compatibility with the format of thecoverage mappings emitted into instrumented binaries. These formats are notforwards-compatible.

• The JSON coverage export format has a (major, minor, patch) version triple.Only a major version increment indicates a backwards-incompatible change. Aminor version increment is for added functionality, and patch versionincrements are for bugfixes.

Impact of llvm optimizations on coverage reports¶

llvm optimizations (such as inlining or CFG simplification) should have noimpact on coverage report quality. This is due to the fact that the mappingfrom source regions to profile counters is immutable, and is generated beforethe llvm optimizer kicks in. The optimizer can’t prove that profile counterinstrumentation is safe to delete (because it’s not: it affects the profile theprogram emits), and so leaves it alone.

Note that this coverage feature does not rely on information that can degradeduring the course of optimization, such as debug info line tables.

Using the profiling runtime without static initializers¶

By default the compiler runtime uses a static initializer to determine theprofile output path and to register a writer function. To collect profileswithout using static initializers, do this manually:

• Export aint__llvm_profile_runtime symbol from each instrumented sharedlibrary and executable. When the linker finds a definition of this symbol, itknows to skip loading the object which contains the profiling runtime’sstatic initializer.

• Forward-declarevoid__llvm_profile_initialize_file(void) and call itonce from each instrumented executable. This function parsesLLVM_PROFILE_FILE, sets the output path, and truncates any existing filesat that path. To get the same behavior without truncating existing files,pass a filename pattern string tovoid__llvm_profile_set_filename(char*). These calls can be placed anywhere so long as they precede all callsto__llvm_profile_write_file.

• Forward-declareint__llvm_profile_write_file(void) and call it to writeout a profile. This function returns 0 when it succeeds, and a non-zero valueotherwise. Calling this function multiple times appends profile data to anexisting on-disk raw profile.

In C++ files, declare these asextern"C".

Collecting coverage reports for the llvm project¶

To prepare a coverage report for llvm (and any of its sub-projects), add-DLLVM_BUILD_INSTRUMENTED_COVERAGE=On to the cmake configuration. Rawprofiles will be written to$BUILD_DIR/profiles/. To prepare an htmlreport, runllvm/utils/prepare-code-coverage-artifact.py.

To specify an alternate directory for raw profiles, use-DLLVM_PROFILE_DATA_DIR. To change the size of the profile merge pool, use-DLLVM_PROFILE_MERGE_POOL_SIZE.

Drawbacks and limitations¶

• Prior to version 2.26, the GNU binutils BFD linker is not able link programscompiled with-fcoverage-mapping in its--gc-sections mode. Possibleworkarounds include disabling--gc-sections, upgrading to a newer versionof BFD, or using the Gold linker.

• Code coverage does not handle unpredictable changes in control flow or stackunwinding in the presence of exceptions precisely. Consider the followingfunction:

  intf(){may_throw();return0;}

  If the call tomay_throw() propagates an exception intof, the codecoverage tool may mark thereturn statement as executed even though it isnot. A call tolongjmp() can have similar effects.

Clang implementation details¶

This section may be of interest to those wishing to understand or improvethe clang code coverage implementation.