| Yuke Liao | 8bc0085 | 2019-01-29 19:22:39 | [diff] [blame] | 1 | # Code Coverage in Gerrit |
| 2 | |
| Yuke Liao | 0e5a20e | 2019-06-17 19:52:57 | [diff] [blame] | 3 | Tests are critical because they find bugsand regressions, enforce better |
| 4 | designsand make code easier to maintain.**Code coverage helps youensure your |
| 5 | tests are thorough**. |
| Yuke Liao | 8bc0085 | 2019-01-29 19:22:39 | [diff] [blame] | 6 | |
| Yuke Liao | 0e5a20e | 2019-06-17 19:52:57 | [diff] [blame] | 7 | ChromiumCLs can show a line-by-line breakdown of test coverage.**You canuse |
| Yuke Liao | 78aa43b | 2019-08-29 16:58:17 | [diff] [blame] | 8 | it toensure you only submit well-tested code**. |
| Yuke Liao | 0e5a20e | 2019-06-17 19:52:57 | [diff] [blame] | 9 | |
| Yuke Liao | 78aa43b | 2019-08-29 16:58:17 | [diff] [blame] | 10 | To see code coveragefor aChromium CL,**trigger a CQ dry run**,and once the |
| Yuke Liao | a085b98 | 2019-10-24 21:37:11 | [diff] [blame] | 11 | builds finishand code coverage datais processed successfully,**look at the |
| 12 | change view to see absoluteand incremental code coverage percentages**: |
| 13 | |
| 14 | ![code_coverage_percentages] |
| 15 | |
| 16 | Absolute coverage percentageis the percentage of lines coveredby tests |
| 17 | out of**all the lines**in the file,while incremental coverage percentage only |
| Prakhar | c7ae92e | 2023-09-20 09:34:52 | [diff] [blame] | 18 | accountsfor**newly addedor modified lines**.Both these coverage metrics, are further |
| 19 | classifiedintoUnitTests coverage(coveragefrom just unit tests)andAllTests coverage(covererdby all tests runningin CQ, including unit tests).AllTests coverageis a superset ofUnitTests coverage. |
| Yuke Liao | a085b98 | 2019-10-24 21:37:11 | [diff] [blame] | 20 | |
| 21 | To further diginto specific lines that arenot coveredby tests,**look at the |
| dpapad | 3eef948 | 2023-11-08 19:13:41 | [diff] [blame] | 22 | right column of the sideby side diff view,and specifically notice the |
| 23 | background color of each line number**,where a light orange color indicates |
| 24 | missing coverageand a light blue color indicates existing coverage.Moreover |
| 25 | hovering over the line number shows an informative tooltipwith |
| 26 | "Not covered by tests"or"Covered by tests" text respectively.It only shows |
| 27 | AllTestsCoverage right now |
| Yuke Liao | 8bc0085 | 2019-01-29 19:22:39 | [diff] [blame] | 28 | |
| Yuke Liao | 0e5a20e | 2019-06-17 19:52:57 | [diff] [blame] | 29 | ![code_coverage_annotations] |
| 30 | |
| Yuke Liao | 78aa43b | 2019-08-29 16:58:17 | [diff] [blame] | 31 | **Code coverage datais shared between patchsets that are commit-message-editor |
| 32 | trivial-rebase away**, however,if a newly uploaded patchset has |
| 33 | non-trivial code change, anew CQ dry run must be triggered before coverage data |
| 34 | shows up again. |
| 35 | |
| Prakhar | c7ae92e | 2023-09-20 09:34:52 | [diff] [blame] | 36 | The code coverage tool supports coveragefor C/C++, JAVAandJavascript on all major platforms(Linux,MacOS,Windows,Android, iOSandChromeOS) |
| Yuke Liao | 6e57a8a | 2019-11-12 22:23:49 | [diff] [blame] | 37 | |
| Prakhar | c7ae92e | 2023-09-20 09:34:52 | [diff] [blame] | 38 | ## CLs Blocked Due to Low Coverage |
| 39 | For some teamsinChrome, we have turned on a coverage check, which blocks a CLfrom submissionif the incremental coverageis below a preset threshold(default=70%).CLswith insufficient test coverage have a`CodeCoverage-1` label added to them, which prevents themfrom being submitted.Also, a descriptive messageis added to the CL, notifying developer of why the CL was blocked,and how to resolve it. |
| 40 | ![low_coverage_message] |
| 41 | |
| 42 | Once the tests are added, another run of coverage builders(through CQ+1or CQ+2) changes the label to`CodeCoverage+1`, allowingCLs to proceedwith submission. |
| 43 | |
| Ashley Newson | a0b1f81d | 2024-06-12 10:22:54 | [diff] [blame] | 44 | Tests themselves,as wellas test-only files, are generally excludedfrom coverage checks based on their pathor filename.If you are getting coverage warningsfor test-related files themselves, check whether the filesendin"test"or"tests"(for example,"SomethingTest.java"or"something_unittests.cc")or that their path contains a directory named exactly"test","tests",or"testing".Thereisno manual list to which files can be addedforlong-term exclusion. |
| 45 | |
| Prakhar | d4ac7f6 | 2024-01-16 21:19:12 | [diff] [blame] | 46 | Devs can also choose to bypassthis block,incase they think they are being unfairly punished.They cando soby adding a*Low-Coverage-Reason: reason* footer to the change description.This should follow certain formatting constraints which are mentioned below |
| 47 | |
| 48 | ### Mention the Bypass Category |
| 49 | |
| 50 | The`reason`string should mention the category the bypass reason belongs to.For e.g.*Low-Coverage-Reason: TRIVIAL_CHANGEThis change contains only minor cosmetic changes.*(TRIVIAL_CHANGEis the category) |
| Prakhar | c7ae92e | 2023-09-20 09:34:52 | [diff] [blame] | 51 | |
| 52 | Available category choices are: |
| 53 | ***TRIVIAL_CHANGE**: CL contains mostly minor changes e.g. renaming, file moves, logging statements, simpleinterface definitions etc. |
| 54 | ***TESTS_ARE_DISABLED**:Corresponding tests exist, but they are currently disabled. |
| 55 | ***TESTS_IN_SEPARATE_CL**:Developer plan to write testsin a separate CL(Shouldnot be exercised oftenas per best practices) |
| 56 | ***HARD_TO_TEST**:The code under considerationis hard to test.For e.g.Interfaceswith system, real hardware etc. |
| 57 | ***COVERAGE_UNDERREPORTED**:To be usedwhen the developer thinks that tests exist, but corresponding coverageis missing. |
| 58 | ***LARGE_SCALE_REFACTOR**:The current changeis part of a large scale refactor.Should explain why the refactor shouldn't have tests. |
| 59 | * **EXPERIMENTAL_CODE**: The current code is experimental and unlikely to be released to users. |
| 60 | * **OTHER**: None of the above categories are the right fit |
| 61 | |
| 62 | In case the developer doesn't specify the coverage categoryas prescribed, a warning will be shownin the UI,with details on how to fix |
| 63 | ![impropery_formatted_coverage_footer] |
| Yuke Liao | 0e5a20e | 2019-06-17 19:52:57 | [diff] [blame] | 64 | |
| Prakhar | d4ac7f6 | 2024-01-16 21:19:12 | [diff] [blame] | 65 | ### No empty line after the footer |
| 66 | In orderfor*Low-Coverage-Reason: reason* to work properly, it should occur after thelast empty linein CL description, otherwise gerrit recognizes itas part of the commit message, rather than the footer i.e.Following wouldnot work |
| 67 | ![empty_line_after_footer] |
| 68 | |
| 69 | Removing the empty line should fix it |
| 70 | ![no_empty_line_after_footer] |
| 71 | |
| 72 | ### Be careful with long footer strings |
| 73 | Either keep the footer messagein one line i.e.donot add line breaks;orif youdo, add whitespace onnew footer lines, otherwise[gerrit doesn’t parse them right]. e.g. along footer message can be writtenas |
| 74 | ![long_footer] |
| 75 | or |
| 76 | ![line_break_footer] |
| 77 | |
| Yuke Liao | 8bc0085 | 2019-01-29 19:22:39 | [diff] [blame] | 78 | ## Contacts |
| 79 | |
| 80 | ### Reporting problems |
| 81 | For any breakage reportand feature requests, please[file a bug]. |
| 82 | |
| 83 | ### Mailing list |
| 84 | For questionsand general discussions, please join[code-coveragegroup]. |
| 85 | |
| 86 | ## FAQ |
| Yuke Liao | 8bc0085 | 2019-01-29 19:22:39 | [diff] [blame] | 87 | ### Why is coverage not shown even though the try job finished successfully? |
| 88 | |
| 89 | There are several possible reasons: |
| Yuke Liao | 8bc0085 | 2019-01-29 19:22:39 | [diff] [blame] | 90 | * A particular source file/test maynot be available on a particular projector |
| Prakhar | 28ffd9b1 | 2024-02-01 18:49:12 | [diff] [blame] | 91 | platform. |
| Yuke Liao | 8bc0085 | 2019-01-29 19:22:39 | [diff] [blame] | 92 | *Thereis a bugin the pipeline.Please[file a bug] to report the breakage. |
| 93 | |
| 94 | ### How does it work? |
| 95 | |
| 96 | Please refer to[code_coverage.md]for how code coverage worksinChromiumin |
| 97 | general,and specifically,for per-CL coverageinGerrit, the |
| 98 | [clang_code_coverage_wrapper]is used to compileand instrument ONLY the source |
| Yuke Liao | 0e5a20e | 2019-06-17 19:52:57 | [diff] [blame] | 99 | files that are affectedby the CLfor the sake of performanceand a |
| 100 | [chromium-coverageGerrit plugin]is used to display code coverage information |
| 101 | inGerrit. |
| Yuke Liao | 8bc0085 | 2019-01-29 19:22:39 | [diff] [blame] | 102 | |
| 103 | |
| 104 | [choose_tryjobs]: images/code_coverage_choose_tryjobs.png |
| Yuke Liao | 0e5a20e | 2019-06-17 19:52:57 | [diff] [blame] | 105 | [code_coverage_annotations]: images/code_coverage_annotations.png |
| Yuke Liao | a085b98 | 2019-10-24 21:37:11 | [diff] [blame] | 106 | [code_coverage_percentages]: images/code_coverage_percentages.png |
| Prakhar | c7ae92e | 2023-09-20 09:34:52 | [diff] [blame] | 107 | [low_coverage_message]: images/low_coverage_message.png |
| Prakhar | d4ac7f6 | 2024-01-16 21:19:12 | [diff] [blame] | 108 | [empty_line_after_footer]: images/empty_line_after_footer.png |
| 109 | [no_empty_line_after_footer]: images/no_empty_line_after_footer.png |
| 110 | [long_footer]: images/long_footer.png |
| 111 | [line_break_footer]: images/line_break_footer.png |
| Prakhar | c7ae92e | 2023-09-20 09:34:52 | [diff] [blame] | 112 | [impropery_formatted_coverage_footer]: images/improperly_formatted_coverage_footer.png |
| Yuke Liao | 03c64407 | 2019-07-30 18:33:40 | [diff] [blame] | 113 | [file a bug]: https://bugs.chromium.org/p/chromium/issues/entry?components=Infra%3ETest%3ECodeCoverage |
| Yuke Liao | 8bc0085 | 2019-01-29 19:22:39 | [diff] [blame] | 114 | [code-coveragegroup]: https://groups.google.com/a/chromium.org/forum/#!forum/code-coverage |
| 115 | [code_coverage.md]: code_coverage.md |
| John Palmer | 046f987 | 2021-05-24 01:24:56 | [diff] [blame] | 116 | [clang_code_coverage_wrapper]: https://chromium.googlesource.com/chromium/src/+/main/docs/clang_code_coverage_wrapper.md |
| Yuke Liao | 8bc0085 | 2019-01-29 19:22:39 | [diff] [blame] | 117 | [chromium-coverageGerrit plugin]: https://chromium.googlesource.com/infra/gerrit-plugins/code-coverage/ |
| Prakhar | d4ac7f6 | 2024-01-16 21:19:12 | [diff] [blame] | 118 | [gerrit doesn’t parse them right]: https://bugs.chromium.org/p/chromium/issues/detail?id=1459714#c9 |
