Automated testing for Chrome for iOS

See theinstructions for how to check out and build Chromium for iOS.

Automated testing is a crucial part of ensuring the quality of Chromium.

Unit testing

Unit testing is done via gtests. The easiest way to run a unit test is to use the wrapper scripts generated at build time:

<output directory>/bin/run_<target_name>[options]

The wrapper scripts take care of invoking//ios/build/bots/scripts/run.py with options like--iossim set appropriately. In general, you need to pass at least--out-dir (a directory where test results will be stored),--platform (a device available to the simulator),--xcode-build (obtained via e.g. About XCode) and--version (iOS version to run). The--gtest_filter option is also supported.

A more complete example looks like this:

out/Debug-iphonesimulator/run_base_unittests \--gtest_filter=Base64Test.Basic \--platform"iPhone 16" \--version18.2 \--xcode-build16c5032a

Integration testing

EarlGrey (EG2) is the integration testing framework used by Chromium for iOS.

Writing EarlGrey tests

Before you start

• Just write a unit test if the purpose of your test does not involve UI.
• Learn about EarlGrey test framework principles and APIs inEarlGrey.
• Learn aboutDefining Test Cases and Test Methods from Apple.

Creating test files and writing EG2 tests

1. EG2 test files are ended with _egtest.mm, and usually located within the same directory of the UI code you wish to test.

2. Basic imports of a EG2 test file:

   • You’ll have to include:

   #import "ios/chrome/test/earl_grey/chrome_test_case.h"

   • You’ll most likely find util functions in these files helpful.

   #import "ios/chrome/test/earl_grey/chrome_earl_grey.h"#import "ios/chrome/test/earl_grey/chrome_earl_grey_ui.h"#import "ios/chrome/test/earl_grey/chrome_matchers.h"

   • Beside these, directly import an EG2 header for an EG2 API you are using.

3. TestCase/testMethods definitions. CreateSomeGreatTestCase as a subclass ofChromeTestCase. Create test methods, eg-(void)testMyGreatUIFeature {...}, and put UI actions within the test methods.

   • Put your setup and tear down code for theTestCase in+(void)setUpForTestCase and+tearDown. These will run once before and after all tests for the test class.
   • Put your setup and tear down code for eachtest method in-(void)setUp and-(void)tearDown. These will run before and after every-(void)testMethod in the file.

4. Writing test contents. See the chrome helpers (imports in 2.) as well asEarlGrey APIs to write a UI action/assertion in your testMethod.

Interacting with the app in a test

Relaunch app with different flags

In EG2 tests, the test process launches the host app process at the beginning, then runs UI actions/assertions in the app. To pass args or feature flags to the app at initial launching, or relaunch the app in the middle of your test, seethis AppLaunchManager API.

Accessing app internals

EG2 test targets are built with test-related code but without app code.

To access anything from the app side, use an “app interface”. App interface is implemented as a class that lives in the app process, but can be accessed in the test process througheDO. You can include the header in your test side code and call class methods of the interface class. The methods will execute code in the app process and can return basic Objective-C types. See thisExample of App Interface.

Seeeg_test_support+eg2 (test side utilities) andeg_app_support+eg2 (app side utilities) targets inBUILD.gn files to learn how test utilities are organized in targets. If you added an app side helper (app interface), you’ll also need to include your neweg_app_support+eg2 target in//ios/chrome/test/earl_grey/BUILD.gn’seg_app_support+eg2 target. (Example CL adding App Interface).

Note that if you create an App interface, you can’t build the app interface class in your eg2_tests target, but you need to include and refer to it. To satisfy the linker, you'll need to create amy_test_app_interface_stub.mm file with the following content in it and build it as a dependency of your tests that use the app interface.

#import "ios_internal/chrome/test/earl_grey2/my_test_app_interface.h"#import "ios/testing/earl_grey/earl_grey_test.h"GREY_STUB_CLASS_IN_APP_MAIN_QUEUE(MyTestAppInterface)

If you don‘t you’ll get linker errors that read like “Undefined symbols for architecture… MyTestAppInterface”

Creating test targets and adding the target to test suites

1. Create a test target. Add a target(source_set) named “eg2_tests” into the closestBUILD.gn file. Put the test file into thesources array and put the targets containing headers used in your test file intodeps array. This is to organize test source files and dependencies so that the GN build system can correctly build the test module. The skeleton of the target:

source_set("eg2_tests") {  configs += [    "//build/config/ios:xctest_config",  ]  testonly = true  sources = [    "some_egtest.mm"  ]  deps = [    "//ios/chrome/test/earl_grey:eg_test_support+eg2",    "//ios/testing/earl_grey:eg_test_support+eg2",  ]  frameworks = [ "UIKit.framework" ]}

2. Include your test target in thedeps array of a suitable suite in//src/ios/chrome/test/earl_grey2/BUILD.gn.
3. Optional: If you feel like your new test should be in a new suite, or you want to delete an existing suite to make tests better organized, you’ll need to change the suites in//src/ios/chrome/test/earl_grey2/BUILD.gn in the format of existing ones. (Do not forget toconfig the bots so the new suite can run in infra.)
4. Ensure your dependencies are correct.

$ gn gen --check out/Debug-iphonesimulator

Running EarlGrey tests

EarlGrey tests are based on Apple'sXCUITest.

Running tests from Xcode

1. If you added a new test file / suite, rungclient runhooks to sync for the list of tests in Xcode.
2. Run a test suite(module), TestCase or testMethod in test navigator. Xcode will build the targets and run the test(s) you choose. Alternatively, use ⌘+U to run all the tests. See Apple'sRunning Tests and Viewing Results.
3. You can pass extra arguments to the app process with--extra-app-args, e.g.--extra-app-args='--enable-features=Foo'.
   • This might not work consistently as tests can re-launch the app with arbitrary command-line arguments.

Running from the command-line

EG2 tests can run in the command line with test runner scripts. You’ll need to build the targets before running tests in cmd. This is used by continuous integration infra and thus not user friendly. Running UI tests directly in Xcode is recommended.

Important notes:

• The test runner can invoke mac_toolchain to install a new Xcode of the version specified to the path specified. You may want to choose a different path from your daily use Xcode.
• If test_cases is empty in --args-json, all tests will run. Specifying a testMethod to run is currently not supported in the test runner.

Example:

src/ios/build/bots/scripts/run.py    --app    src/out/Debug-iphonesimulator/ios_chrome_ui_eg2tests_module-Runner.app    --host-app    src/out/Debug-iphonesimulator/ios_chrome_eg2tests.app    --args-json    {"test_args": [], "xctest": false, "test_cases": ["ReadingListTestCase"],    "restart": false, "xcode_parallelization": true, "xcodebuild_device_runner":    false}    --out-dir   path/to/output/dir    --retries    3    --shards    1    --xcode-build-version    11c29    --mac-toolchain-cmd    path/to/mac_toolchain    --xcode-path    path/to/Xcode.app    --wpr-tools-path    NO_PATH    --replay-path    NO_PATH    --iossim    src/out/Debug-iphonesimulator/iossim    --platform    iPad (6th generation)    --version    13.3

The invocation args are logged. You can find the latest arg format at the beginning of stdout from an infra test shard if the above doesn't work.