| erikchen | b83e010a | 2018-10-08 21:34:01 | [diff] [blame] | 1 | # Parsing Test Results |
| 2 | |
| 3 | Chromium runs over500,000 testsfor each CL.There are many layers of UIfor |
| 4 | parsingand interpreting these test results.This doc provides a brief guide |
| 5 | for navigating these UI layers. |
| 6 | |
| 7 | ## Polygerrit UI |
| 8 | |
| 9 | Tests are segmentedby buildand test configurations.The segments are usually |
| 10 | referred toas*builds*.In the example below, each greenand red rectangle |
| 11 | refers to a*build*. |
| 12 | |
| 13 |  |
| 14 | |
| 15 | The name of each build usually contains enough information toget a rough idea |
| 16 | of the configuration.Some examples: |
| 17 | |
| 18 | **android_compile_dbg*is a compile-only[no tests] build ofChromiumfor |
| 19 | Android,using the*debug* configuration. |
| Haiyang Pan | b2c3d97a | 2020-04-24 20:43:43 | [diff] [blame] | 20 | **android-lollipop-arm-rel* buildsand runs testsforChromiumforAndroid, |
| 21 | using the*release* configuration on a lollipop device. |
| Stephen Martinis | 089f5f0 | 2019-02-12 02:42:24 | [diff] [blame] | 22 | **win7-rel* buildsand runs testsforChromium onWindows,using |
| erikchen | b83e010a | 2018-10-08 21:34:01 | [diff] [blame] | 23 | the release configuration on aWindows7 device.*ng* standsfornext |
| 24 | generation, butthis hasno meaningas the previous generation was already |
| 25 | phasedout. |
| 26 | |
| 27 | Green boxes refer to builds that passed.Red boxes refer to builds that failed. |
| 28 | Some failed buildsget automatically retriedby the CQ.Inthis example, |
| Stephen Martinis | 089f5f0 | 2019-02-12 02:42:24 | [diff] [blame] | 29 | *linux-rel*and*mac-rel* were automatically retried[hence the two**X**s], but |
| 30 | *win7-rel* wasnot.The**X** on the leftis the first build,and the**X** on the right |
| 31 | is the second build. |
| erikchen | b83e010a | 2018-10-08 21:34:01 | [diff] [blame] | 32 | |
| 33 | Each of these boxesis a link that provides more information about the build |
| 34 | failure. |
| 35 | |
| 36 | ## Build Results UI |
| 37 | |
| 38 | Selecting any of the build resultsfrom the previous section will navigate to |
| 39 | the build results UI.Each buildis implementedby a[recipe]-- |
| 40 | effectively aPython script.Each recipeis dividedinto*steps*.Each*step* |
| 41 | represents a well-defined action, suchas updating the repository to point to |
| 42 | tip of tree,or compiling the necessary build artifacts. |
| 43 | |
| John Palmer | 046f987 | 2021-05-24 01:24:56 | [diff] [blame] | 44 | [recipe]: https://chromium.googlesource.com/external/github.com/luci/recipes-py/+/main/doc/user_guide.md |
| erikchen | b83e010a | 2018-10-08 21:34:01 | [diff] [blame] | 45 | |
| 46 |  |
| 47 | |
| 48 | Under the**StepsandLogfiles** headingis a list of numbered*steps*.Each |
| 49 | *step* has a color(red, greenor purple) which indicates whether the*step* |
| 50 | failed, succeeded,or encountered an unexpected condition.Failing steps are |
| 51 | also groupedinto the**Results** section at the very topfor convenience. |
| 52 | |
| 53 | ## Build Results UI -- Overview |
| 54 | |
| 55 | Most builds follow a similar pattern.The key*steps* are listed here. |
| 56 | |
| 57 | ***bot_update**Update the repository to point to tip of tree.Apply the CL |
| 58 | as a patch. |
| 59 | ***analyze**Analyze dependencies of test suites to determine which test |
| 60 | suites are affectedby the patch. |
| 61 | ***compile(with patch)**Builds test suitesand associated artifacts. |
| 62 | ***isolate tests**Archives test suite binariesand artifacts. |
| John Chen | aab16fc0 | 2020-03-03 06:35:45 | [diff] [blame] | 63 | ***test_pre_run.[trigger] blink_web_tests(with patch)**Triggers a test |
| erikchen | b83e010a | 2018-10-08 21:34:01 | [diff] [blame] | 64 | suite on swarming[remote execution framework]--inthiscase, |
| John Chen | aab16fc0 | 2020-03-03 06:35:45 | [diff] [blame] | 65 | blink_web_tests. |
| 66 | ***blink_web_tests(with patch)**Collects the resultsfrom swarmingfor a |
| erikchen | b83e010a | 2018-10-08 21:34:01 | [diff] [blame] | 67 | test suite. |
| 68 | |
| 69 | If all test suitespass,then the*build*is markedas a successandno further |
| 70 | steps are run.If at least one test suite has failures,then the failing tests |
| 71 | are rerunwith the patch deapplied.This allows the recipe to determineif the |
| 72 | test failureis due to the CLor due to a problemwith tip of tree. |
| 73 | |
| 74 | ***bot_update[without patch]**Deapplies the CL patch. |
| 75 | ***compile[without patch]**Compiles test suites. |
| 76 | ***isolate tests(2)**Archives test suite binariesand artifacts. |
| John Chen | aab16fc0 | 2020-03-03 06:35:45 | [diff] [blame] | 77 | ***test_pre_run.[trigger] blink_web_tests(without patch)**Triggers test |
| erikchen | b83e010a | 2018-10-08 21:34:01 | [diff] [blame] | 78 | suite on swarming.Only failing tests are rerun. |
| John Chen | aab16fc0 | 2020-03-03 06:35:45 | [diff] [blame] | 79 | ***blink_web_tests(without patch)**Collects resultsfrom swarming. |
| erikchen | b83e010a | 2018-10-08 21:34:01 | [diff] [blame] | 80 | |
| 81 | **Important safety notice**.When test suites are runwith the patch applied, |
| 82 | each testis run up to N times-- any success will mark the testas a success. |
| 83 | When test suites are run without the patch, each failing testis run exactly N |
| 84 | times.Any failure will mark the testas a failure. |
| 85 | |
| 86 | If there are tests that failedwith the patch applied, butnotwith the patch |
| 87 | deapplied,then that implies that it's likely that the CL broke a test. Just to |
| 88 | confirm, the first suite of steps is run again, this time with the suffix |
| 89 | **(retry with patch)**. |
