| Ben Pastene | b30c66e9 | 2019-08-30 23:25:10 | [diff] [blame] | 1 | # Chrome OS Debugging Instructions |
| 2 | Chrome onChrome OSis testedusing a handful of frameworks, each of which |
| 3 | you'll find running on Chrome's CQand waterfalls.If you're investigating |
| 4 | failures in these tests, below are some tips for debugging and identifying the |
| 5 | cause. |
| 6 | |
| 7 | *** note |
| 8 | |
| 9 | This doc outlines tests running in true Chrome OS environments (ie: on virtual |
| 10 | machines or real devices). [linux-chromeos] tests, on the other hand, can be |
| 11 | debugged like any other linux test. |
| 12 | *** |
| 13 | |
| 14 | ## Tast |
| 15 | |
| 16 | [Tast] is Chrome OS's integration testing framework.SinceChrome itselfis |
| 17 | instrumental to theChrome OS system, it's equally important that we run some |
| 18 | of these integration tests on Chrome's waterfalls.If you find one of these |
| 19 | tests failing(likelyin the`chrome_all_tast_tests` step), you can: |
| 20 | |
| Ben Pastene | f8d6812 | 2021-09-20 22:49:10 | [diff] [blame] | 21 | -**Inspect the failed test's log snippet**: There should be a log snippet for |
| 22 | each failed test in the `Test Results` tab in the build UI. eg: For this |
| 23 | [failed build], clicking on the `policy.IncognitoModeAvailability` expands to |
| 24 | include stack traces and error messages. |
| Ben Pastene | b30c66e9 | 2019-08-30 23:25:10 | [diff] [blame] | 25 | |
| 26 | - **View browser & system logs**: A common cause of failure on Chrome's builders |
| 27 | are browser crashes.Whenthis happens, each test's log snippets will simply |
| 28 | contain warnings like "[Chrome probably crashed]". To debug these crashes, |
| Ben Pastene | f8d6812 | 2021-09-20 22:49:10 | [diff] [blame] | 29 | expand the list of attached artifacts for the test by clicking the `Artifacts` |
| 30 | link under the failed test in the `Test Results` tab. There you'll find an |
| 31 | extended logfor the test under`log.txt`.Additionally, you can find system |
| 32 | logs includedin that list.To find a system logfor a particular test, match |
| Ben Pastene | 0fed3a7 | 2020-11-06 19:25:15 | [diff] [blame] | 33 | the timestamps printedin the test's log with the timestamps present in the |
| Ben Pastene | f8d6812 | 2021-09-20 22:49:10 | [diff] [blame] | 34 | system log filename. For instance, the previous `example.ChromeFixture` failure |
| 35 | matches the [chrome/chrome_20210920-051805] browser log, which contains the |
| 36 | culprit Chrome crash and backtrace. |
| Ben Pastene | 0fed3a7 | 2020-11-06 19:25:15 | [diff] [blame] | 37 | |
| Ben Pastene | 44862a2 | 2020-12-12 00:53:07 | [diff] [blame] | 38 | - **Symbolizing a browser crash dump**: See [below](#symbolizing-a-crash-dump). |
| 39 | |
| Ben Pastene | 0fed3a7 | 2020-11-06 19:25:15 | [diff] [blame] | 40 | ### Disabling a test |
| Ben Pastene | b30c66e9 | 2019-08-30 23:25:10 | [diff] [blame] | 41 | |
| Timothy Loh | eb761b0 | 2022-06-06 01:46:12 | [diff] [blame] | 42 | If you are a Chrome Sheriff, please read the sheriff documentation |
| 43 | [here](http://go/chrome-sheriff-tast) before disabling any tests. |
| 44 | |
| 45 | Tast tests are run under both Chrome's buildersandCrOS's builders. They can be |
| 46 | disabled either completely (in all builders), or in Chrome's builders alone.The |
| 47 | latter should be used onlyfor changes which arenot expected to occur onCrOS's |
| 48 | builders. |
| 49 | |
| 50 | - **Disabling in all builders**: If you have a full CrOS checkout, you can add |
| Ben Pastene | 4327118 | 2021-04-30 19:57:07 | [diff] [blame] | 51 | the `informational` [attribute] to the test's definition.(You may be able to |
| 52 | bypass the needfor a fullCrOS checkoutbyusing the`Edit code` buttonin |
| 53 | codesearch UI, butthis flowis unverified.)This can take time(ie: many hours) |
| 54 | to landand propagate ontoChrome's builders. So if you need the test disabled |
| 55 | ASAP, consult the next option. |
| Timothy Loh | eb761b0 | 2022-06-06 01:46:12 | [diff] [blame] | 56 | - **Disabling in only Chrome's builders**:You can add the test to the list of |
| Ben Pastene | b30c66e9 | 2019-08-30 23:25:10 | [diff] [blame] | 57 | disabled testsfor the step's GN target. For example, to disable a test in the |
| Ben Pastene | 4327118 | 2021-04-30 19:57:07 | [diff] [blame] | 58 | `chrome_all_tast_tests` step, add it to [this list]. **Note**: If the test is |
| 59 | failing consistently, and you only disable it here, it will likely start to fail |
| 60 | in the next [Chrome uprev] on CrOS's builders, which can lead to further |
| 61 | problems down the road.So please make sure you pursue the first optionas well |
| 62 | in thatcase. |
| Ben Pastene | b30c66e9 | 2019-08-30 23:25:10 | [diff] [blame] | 63 | |
| Ben Pastene | 48021b4 | 2020-04-21 19:18:46 | [diff] [blame] | 64 | In both cases, please make sure a bugis filedfor the test,and route it to |
| 65 | the appropriate owners. |
| 66 | |
| Ben Pastene | 44862a2 | 2020-12-12 00:53:07 | [diff] [blame] | 67 | ### Symbolizing a crash dump |
| 68 | |
| 69 | If a test fails due to a browser crash, there should be aMinidump crash report |
| 70 | presentin the test's isolated out under the prefix `crashes/chrome...`. These |
| 71 | reports aren't very usefulby themselves, butwith a few commands you can |
| 72 | symbolize the report locally toget insightinto what conditions causedChrome |
| 73 | to crash. |
| 74 | |
| Joel Hockey | f5f032d | 2024-02-16 00:03:39 | [diff] [blame] | 75 | If you are running a locally compiled[SimpleChrome] binary on a deviceor VM, |
| 76 | you can can build`minidump_stackwalk`and download the |
| 77 | `/home/chronos/crash/chrome*.dmp` file. |
| 78 | ``` |
| 79 | autoninja -C out/Release minidump_stackwalk dump_syms |
| 80 | |
| 81 | rsync -r -e "ssh -p 9222" root@localhost:/home/chronos/crash /tmp |
| 82 | ``` |
| 83 | |
| 84 | For a crash on a bot, download both the task's input files (this provides the |
| Roman Sorokin | faf5ec7 | 2022-08-18 09:23:23 | [diff] [blame] | 85 | symbols and the symbolizing tools) as well as the task's output results(this |
| Ben Pastene | 44862a2 | 2020-12-12 00:53:07 | [diff] [blame] | 86 | provides the crash reports).See the commands listed under the*Reproducing the |
| 87 | task locally* section on the task page.For example, to download themfor |
| Roman Sorokin | faf5ec7 | 2022-08-18 09:23:23 | [diff] [blame] | 88 | [this task](https://chrome-swarming.appspot.com/task?id=5cc272e0a839b311), `cd` |
| Ben Pastene | 44862a2 | 2020-12-12 00:53:07 | [diff] [blame] | 89 | into a tmp directoryand run: |
| 90 | ``` |
| Roman Sorokin | faf5ec7 | 2022-08-18 09:23:23 | [diff] [blame] | 91 | cipd install "infra/tools/luci/cas/\${platform}" -root bar |
| 92 | ./bar/cas login |
| 93 | ./bar/cas download -cas-instance projects/chrome-swarming/instances/default_instance -digest 1ad29e201e4ae7e3056a8b17935edbcd62fb54befdfeba221f2e82e54f150c86/812 -dir foo |
| 94 | |
| 95 | cipd install "infra/tools/luci/swarming/\${platform}" -root bar |
| 96 | ./bar/swarming login |
| 97 | ./bar/swarming collect -S chrome-swarming.appspot.com -output-dir=foo 5cc272e0a839b311 |
| Ben Pastene | 44862a2 | 2020-12-12 00:53:07 | [diff] [blame] | 98 | ``` |
| 99 | |
| Joel Hockey | f5f032d | 2024-02-16 00:03:39 | [diff] [blame] | 100 | Generate the breakpad symbolsby pointing the`generate_breakpad_symbols.py` script to |
| 101 | yourlocal binary,or the downloaded input build dir: |
| Ben Pastene | 44862a2 | 2020-12-12 00:53:07 | [diff] [blame] | 102 | ``` |
| Roman Sorokin | faf5ec7 | 2022-08-18 09:23:23 | [diff] [blame] | 103 | cd foo |
| Joel Hockey | f5f032d | 2024-02-16 00:03:39 | [diff] [blame] | 104 | vpython3 components/crash/content/tools/generate_breakpad_symbols.py --symbols-dir symbols --build-dir out/Release/ --binary out/Release/chrome --platform chromeos |
| Ben Pastene | 44862a2 | 2020-12-12 00:53:07 | [diff] [blame] | 105 | ``` |
| 106 | |
| 107 | That will generate the symbolsin the`symbols/` dir.Then to symbolize aChrome |
| Joel Hockey | f5f032d | 2024-02-16 00:03:39 | [diff] [blame] | 108 | crash report(eitherin the tasks's output, or the `/tmp/crash` dir): |
| Ben Pastene | 44862a2 | 2020-12-12 00:53:07 | [diff] [blame] | 109 | ``` |
| Roman Sorokin | faf5ec7 | 2022-08-18 09:23:23 | [diff] [blame] | 110 | ./out/Release/minidump_stackwalk 5cc272e0a839b311/crashes/chrome.20220816.214251.44917.24579.dmp symbols/ |
| Ben Pastene | 44862a2 | 2020-12-12 00:53:07 | [diff] [blame] | 111 | ``` |
| 112 | |
| 113 | |
| Ben Pastene | 0fed3a7 | 2020-11-06 19:25:15 | [diff] [blame] | 114 | ### Running a test locally |
| 115 | |
| 116 | To run a Tast test the same way it's ran onChrome's builders: |
| 117 | |
| 118 | - Decide which Chrome OS device type or VM to test on. |
| 119 | |
| 120 | - Build Chrome via the [Simple Chrome] workflow for that board. |
| 121 | |
| 122 | - Deploy your Chrome to the device via the [deploy_chrome.py] tool. |
| 123 | |
| 124 | - Finally, run the Tast test on the device via the `cros_run_test` tool under |
| 125 | `//third_party/chromite/bin/`. eg: |
| Roman Sorokin | 2339741 | 2021-07-12 10:29:09 | [diff] [blame] | 126 | `cros_run_test --device $DEVICE_IP --tast login.Chrome`. See [here] for more |
| Ben Pastene | 0fed3a7 | 2020-11-06 19:25:15 | [diff] [blame] | 127 | info on cros_run_test. |
| 128 | |
| Ben Pastene | b30c66e9 | 2019-08-30 23:25:10 | [diff] [blame] | 129 | ## Telemetry |
| 130 | |
| 131 | >TODO: Add instructions for debugging telemetry failures. |
| 132 | |
| 133 | ## GTest |
| 134 | |
| 135 | >TODO: Add instructions for debugging GTest failures. |
| 136 | |
| 137 | ## Rerunning these tests locally |
| 138 | |
| 139 | >TODO: Add instructions for rerunning these tests locally. |
| 140 | |
| 141 | |
| Ben Pastene | 4327118 | 2021-04-30 19:57:07 | [diff] [blame] | 142 | [linux-chromeos]: https://chromium.googlesource.com/chromium/src/+/HEAD/docs/chromeos_build_instructions.md |
| Ben Pastene | b30c66e9 | 2019-08-30 23:25:10 | [diff] [blame] | 143 | [Tast]: https://chromium.googlesource.com/chromiumos/platform/tast/+/HEAD/README.md |
| Ben Pastene | f8d6812 | 2021-09-20 22:49:10 | [diff] [blame] | 144 | [failed build]: https://ci.chromium.org/ui/p/chromium/builders/ci/chromeos-kevin-rel/37300/test-results |
| 145 | [Chrome probably crashed]: https://luci-milo.appspot.com/ui/inv/build-8835572137562508161/test-results?q=example.ChromeFixture |
| 146 | [chrome/chrome_20210920-051805]: https://luci-milo.appspot.com/ui/artifact/raw/invocations/task-chromium-swarm.appspot.com-561bed66572a9411/artifacts/chrome%2Fchrome_20210920-051805 |
| Ben Pastene | 4327118 | 2021-04-30 19:57:07 | [diff] [blame] | 147 | [attribute]: https://chromium.googlesource.com/chromiumos/platform/tast/+/HEAD/docs/test_attributes.md |
| Nicholas Verne | f4b76305 | 2021-01-27 04:08:05 | [diff] [blame] | 148 | [this list]: https://codesearch.chromium.org/chromium/src/chromeos/tast_control.gni |
| Ben Pastene | 4327118 | 2021-04-30 19:57:07 | [diff] [blame] | 149 | [Chrome uprev]: https://chromium.googlesource.com/chromiumos/docs/+/HEAD/chrome_commit_pipeline.md#the-chrome-os-commit-pipeline-for-chrome-changes |
| 150 | [Simple Chrome]: https://chromium.googlesource.com/chromiumos/docs/+/HEAD/simple_chrome_workflow.md |
| 151 | [deploy_chrome.py]: https://chromium.googlesource.com/chromiumos/docs/+/HEAD/simple_chrome_workflow.md#Deploying-Chrome-to-the-device |
| 152 | [here]: https://chromium.googlesource.com/chromiumos/docs/+/HEAD/cros_vm.md#in-simple-chrome |
