This guide describes how to run an instrumentation, Robo, or Game Loop testusing thegcloud CLI.

For a complete list ofgcloud commands you can usewith your Android app inTest Lab, visit thereference documentation forgcloud firebase test android.

Before you begin

If you haven't already,add Firebase to your Android project.

Step 1. Set up the gcloud CLI

1. Download theGoogle Cloud SDK

This includes the gcloud CLI tool.

2. Make sure your installation is up-to-date:

   gcloud components update

3. Sign in to the gcloud CLI using your Google Account:

   gcloud auth login

4. Set your Firebase project in gcloud, wherePROJECT_ID is the ID of your Firebase project:

   gcloud config set projectPROJECT_ID

Step 2. Check available test devices

Use the following gcloud commands to view test devices and locales that areavailable for your test.

As an option, you can also download the sampleNotepad app to startrunning the commands right away. Use the binary fileapp-debug-unaligned.apkand instrumentation tests fileapp-debug-test-unaligned.apk, which are locatedinNotePad/app/build/outputs/apk/.

• models list: Get a current list of Android devices available for you to testagainst.

  gcloudfirebasetestandroidmodelslist

  In the command output:

  • ColumnMODEL_ID contains the identifier you can later use to run testson the device model.
  • ColumnOS_VERSION_ID contains the operating system versions supportedby the device.

  Example output

  

• models describe: Get more information about a specific AndroidMODEL_ID.

  gcloudfirebasetestandroidmodelsdescribeMODEL_ID

  The output contains the device model's brand, manufacturer, OS versions,supported API levels, supported Application Binary Interfaces (ABI),release dates, and whether the device is physical or virtual.

• versions list: Get a list of currently available OS versions to testagainst.

  gcloudfirebasetestandroidversionslist

  You can use an identifier from either of the first two columns of commandoutput (OS_VERSION_ID andVERSION), to later run tests against an AndroidOS version. If you don't specify the Android OS versions to test against, thedefault noted under theTAGS column is used.

  Example output

  

• locales list: Get the current list of locales available to test against.

  gcloudfirebasetestandroidlocaleslist

  The first column of the command output,LOCALE, contains the identifierthat you can use later to run tests against a locale. If you don't specifythe locales to test against, English is used as the default locale.

Step 3. Run your test

Now that you know the range of device models, locales, and OS versions availablefor testing your app, you can specify devices using thegcloud firebase test android run command and the--device flag to run Robo or instrumentation tests.

Run a Robo test

Even if you don't have any instrumentation tests, you can still lookfor bugs in your app. Use the Robo test to perform an automated review of yourapp's user interface. Robo test exercises the app by performing a staticanalysis of the various paths through the app's user interface, and thencrawls through the app to find crashes and other potential issues.

To run a Robo test, run the following example command:

gcloud firebase test android run \  --type robo \  --app app-debug-unaligned.apk \  --device model=Nexus6,version=21,locale=en,orientation=portrait  \  --device model=Nexus7,version=19,locale=fr,orientation=landscape \  --timeout 90s \  --client-details matrixLabel="Example matrix label"

• The--type robo parameter is implicit if no--type value is specified.
• To help you identify and locate your test matrices in the Firebase console,you can use the optional--client-details matrixLabel="Example matrix label"flag to label your test matrix.
• You can see the complete set of command line options forrunning tests by typing:gcloud help firebase test android run.

As an alternative to specifying these arguments on the command line, you canoptionally specify your arguments in a YAML-formatted argument file. Rungcloud topic arg-files to learn how to use this feature.

To learn how to investigate the test results from the Robo test, SeeAnalyze your test results.

Run an instrumentation test

Now use thegcloud command line tool to run the Notepad app'sEspresso tests on your specified Android device configurations. Use theinstrumentation test type to run the tests inapp-debug-test-unaligned.apkas follows:

gcloud firebase test android run \  --type instrumentation \  --app app-debug-unaligned.apk \  --test app-debug-test-unaligned.apk \  --device model=Nexus6,version=21,locale=en,orientation=portrait  \  --device model=Nexus7,version=19,locale=fr,orientation=landscape  --client-details matrixLabel="Example matrix label"

• The--type instrumentation parameter is implicit if a test APK is specifiedwith--test.
• To help you identify and locate your test matrices in theFirebase console,you can use the optional--client-details matrixLabel="Example matrix label"flag to label your test matrix.
• You can see the complete set of command line options forrunning tests by typinggcloud help firebase test android run.

As an alternative to specifying these arguments on the command line, you canoptionally specify your arguments in a YAML-formatted argument file. Rungcloud topic arg-files to learn how to use this feature.

The gcloud CLI supportsAndroid Test Orchestrator.Orchestrator requires AndroidJUnitRunner v1.1 or higher. To enable it, usegcloud firebase test android run with the
--use-orchestratorflag. To disable it use the--no-use-orchestrator flag.

You can also control howTest Lab runs your instrumentation testsusing additional flags that are not shown above. For example, you can use the--test-targets flag to test a single class or a class method used by your testAPK. You can also find out whether your test that failed was actually flaky ornot by using--num-flaky-test-attempts flag, which specifies the number oftimes a test execution should be re-attempted if one or more of its test casesfail for any reason. To learn more, seegcloud firebase test android run.

Code coverage reports for instrumentation tests

Test Lab supports code coverage reporting toolsEMMA andJaCoCo. If you have either toolintegrated into the build for your app, you can get a code coverage report forTest Lab tests by runninggcloud firebase test android run with someadditional arguments. If Android Test Orchestrator is not enabled, use thefollowing:

gcloud firebase test android run \  --type instrumentation \  --appyour-app.apk \  --testyour-app-test.apk \  --device model=TestDevice,version=AndroidVersion  \  --environment-variables coverage=true,coverageFile="/sdcard/Download/coverage.ec" \  --directories-to-pull /sdcard/Download

If you are generating code coverage reports while also using Android TestOrchestrator, modify your environment variables as follows:

gcloud firebase test android run \  --type instrumentation \  --appyour-app.apk \  --testyour-app-test.apk \  --device model=TestDevice,version=AndroidVersion  \  --environment-variables clearPackageData=true,coverage=true,coverageFilePath="/sdcard/Download/" \  --directories-to-pull /sdcard/Download

WhenTest Lab finishes running your tests, find your code coverage reportsinGoogle Cloud Storage:

1. Open theFirebase console link that thegcloud tool printed above thetest result table in your terminal.
2. Click a test execution from the list at that link to open that execution'sdetails page.
3. ClickTest results to go to theCloud Storage bucket with thatexecution's test results.
4. Openartifacts/coverage.ec to see your code coverage report.

The output of your command line test run also includes a link to view testresults. To learn more about how to interpret these results, seeAnalyzingFirebase Test Lab for Android Results.

Custom login and text input with Robo test

Robo test automatically completes sign-in screens that use a Google accountfor authentication, unless you use the--no-auto-google-loginparameter. It can also complete custom login screens using test accountcredentials that you provide. You can also use this parameter to provide custominput text for other text fields used by your app.

To complete text fields in your app, use the--robo-directivesparameter and provide a comma-separated list ofkey-value pairs, where thekey is the Android resource name of the target UI element, and thevalue isthe text string. You can also use this flag to tell Robo to ignore specificUI elements (e.g., "logout" button).EditTextfields are supported but not text fields inWebView UI elements.

For example, you could use the following parameter for customlogin:

--robo-directives username_resource=username,password_resource=password

• Android Test Orchestrator flag:A flag to enableOrchestrator, a tool that allows you to run each of yourapp's tests in its own invocation ofInstrumentation.Test Lab alwaysruns the latest version of Orchestrator.

• Game Loop test flags:A set of config flags that enable and control a "demomode" to simulate player actions in game apps. Learn more about runningGame Loop tests withTest Lab.

• Uniform Sharding flag (in beta):A flag that specifies the number of shards into which you want to evenlydistribute test cases. The shards are run in parallel on separate devices.Note: The limits for physical devices are 50, 200 forArm virtual, and 500 for x86 virtual. This also applies to theManual Sharding flag.

• Manual Sharding flag (in beta):A flag that specifies a group of packages, classes, and/or test cases to runin a shard (a group of test cases). The shards are run in parallel onseparate devices.

• Network traffic profiles flag (in beta):A flag that specifies which network profile your tests use with physicaldevices. Network profiles emulate a variety of networks conditions, allowingyou to test your app's performance on unreliable or unpredictable networks.

Scripting gcloud commands withTest Lab

You can use shell scripts or batch files to automate mobile app testing commandsthat you would otherwise run using the gcloud command line. The followingexample bash script runs an instrumentation test with a two-minute timeout, andreports if the test run completed successfully:

if gcloud firebase test android run --app my-app.apk --test my-test.apk --timeout 2mthen    echo "Test matrix successfully finished"else    echo "Test matrix exited abnormally with non-zero exit code: " $?fi

Script exit codes

Test Lab provides several exit codes that you can use to better understand theresults of tests that you run using scripts or batch files.

Scripting exit codes forTest Lab