gcloud firebase test android run - invoke a test in Firebase Test Lab for Android and view test results

gcloud firebase test android run[ARGSPEC][--app=APP][--device=DIMENSION=VALUE,[…]][--test=TEST][--timeout=TIMEOUT][--type=TYPE][--additional-apks=APK,[APK,…]][--app-package=APP_PACKAGE][--async][--auto-google-login][--client-details=[KEY=VALUE,…]][--directories-to-pull=[DIR_TO_PULL,…]][--environment-variables=[KEY=VALUE,…]][--network-profile=PROFILE_ID][--num-flaky-test-attempts=int][--obb-files=OBB_FILE,[OBB_FILE]][--other-files=DEVICE_PATH=FILE_PATH,[…]][--performance-metrics][--record-video][--results-bucket=RESULTS_BUCKET][--results-dir=RESULTS_DIR][--results-history-name=RESULTS_HISTORY_NAME][--scenario-labels=LABEL,[LABEL,…]][--scenario-numbers=int,[int,…]][--test-package=TEST_PACKAGE][--test-runner-class=TEST_RUNNER_CLASS][--test-targets=TEST_TARGET,[TEST_TARGET,…]][--use-orchestrator][--resign][--robo-directives=[TYPE:RESOURCE_NAME=INPUT,…]][--robo-script=ROBO_SCRIPT][--device-ids=MODEL_ID,[MODEL_ID,…],-dMODEL_ID,[MODEL_ID,…]][--locales=LOCALE,[LOCALE,…],-lLOCALE,[LOCALE,…]][--orientations=ORIENTATION,[ORIENTATION],-oORIENTATION,[ORIENTATION]][--os-version-ids=OS_VERSION_ID,[…],-vOS_VERSION_ID,[…]][--filter=EXPRESSION][--limit=LIMIT][--page-size=PAGE_SIZE][--sort-by=[FIELD,…]][GCLOUD_WIDE_FLAG …]

Three main types of Android tests are currently supported:

• robo: runs a smart, automated exploration of the activities in yourAndroid app which records any installation failures or crashes and builds anactivity map with associated screenshots and video.
• instrumentation: runs automated unit or integration tests writtenusing a testing framework. Firebase Test Lab for Android currently supports theEspresso and UI Automator 2.0 testing frameworks.
• game-loop: executes a special intent built into the game app (a"demo mode") that simulates the actions of a real player. This test type caninclude multiple game loops (also called "scenarios"), which can be logicallyorganized using scenario labels so that you can run related loops together.Refer tohttps://firebase.google.com/docs/test-lab/android/game-loopfor more information about how to build and run Game Loop tests.

The type of test to run can be specified with the--type flag,although the type can often be inferred from other flags. Specifically, if the--test flag is present, the test--type defaults toinstrumentation. If--test is not present, then--type defaults torobo.

All arguments forgcloud firebase test android run may be specifiedon the command line and/or within an argument file. Run$gcloud topic arg-filesfor more information about argument files.

gcloudfirebasetestandroidrun--app=APP_APK--timeout=100s

When specifying devices to test against, the preferred method is to use the--device flag. For example, to invoke a robo test against a virtual, genericMDPI Nexus device in landscape orientation, run:

gcloudfirebasetestandroidrun--app=APP_APK--device=model=NexusLowRes,orientation=landscape

To invoke an instrumentation test against a physical Nexus 6 device (MODEL_ID:shamu) which is running Android API level 21 in French, run:

gcloudfirebasetestandroidrun--app=APP_APK--test=TEST_APK--device=model=shamu,version=21,locale=fr

To test against multiple devices, specify --device more than once:

gcloudfirebasetestandroidrun--app=APP_APK--test=TEST_APK--device=model=Nexus4,version=19--device=model=Nexus4,version=21--device=model=NexusLowRes,version=25

To invoke a robo test on an Android App Bundle, pass the .aab file using the--app flag.

gcloudfirebasetestandroidrun--app=bundle.aab

You may also use the legacy dimension flags (deprecated) to specify whichdevices to use. Firebase Test Lab will run tests against every possiblecombination of the listed device dimensions. Note that some combinations ofdevice models and OS versions may not be valid or available in Test Lab. Anyunsupported combinations of dimensions in the test matrix will be skipped.

For example, to execute a series of 5-minute robo tests against a verycomprehensive matrix of virtual and physical devices, OS versions, locales andorientations, run:

gcloudfirebasetestandroidrun--app=APP_APK--timeout=5m--device-ids=shamu,NexusLowRes,Nexus5,g3,zeroflte--os-version-ids=19,21,22,23,24,25--locales=en_GB,es,fr,ru,zh--orientations=portrait,landscape

The above command will generate a test matrix with a total of 300 testexecutions, but only the subset of executions with valid dimension combinationswill actually run your tests.

To help you identify and locate your test matrix in the Firebase console, run:

gcloudfirebasetestandroidrun--app=APP_APK--client-details=matrixLabel="Example matrix label"

Controlling Results Storage

By default, Firebase Test Lab stores detailed test results for a limited time ina Google Cloud Storage bucket provided for you at no charge. Note: This requiresthe principal executing the test to have "roles/editor" role for the project. Ifeither you do not have that role, you wish to use a storage bucket that youcontrol, or you need to retain detailed test results for a longer period, usethe--results-bucket option. Seehttps://firebase.google.com/docs/test-lab/analyzing-results#detailedfor more information.

Detailed test result files are prefixed by default with a timestamp and a randomcharacter string. If you require a predictable path where detailed test resultsare stored within the results bucket (say, if you have a Continuous Integrationsystem which does custom post-processing of test result artifacts), use the--results-dir option.Note that each test invocationmust have a unique storage location, so never reuse the same valuefor--results-dir between different test runs.Possible strategies could include using a UUID or sequence number for--results-dir.

For example, to run a robo test using a specific Google Cloud Storage locationto hold the raw test results, run:

gcloudfirebasetestandroidrun--app=APP_APK--results-bucket=gs://my-bucket--results-dir=my/test/results/<unique-value>

To run an instrumentation test and specify a custom name under which the historyof your tests will be collected and displayed in the Firebase console, run:

gcloudfirebasetestandroidrun--app=APP_APK--test=TEST_APK--results-history-name='Excelsior App Test History'

Argument Files

All test arguments for a given test may alternatively be stored in an argumentgroup within a YAML-formatted argument file. TheARG_FILE may contain one or more named argument groups,and argument groups may be combined using theinclude: attribute(Run$gcloud topicarg-files for more information). The ARG_FILE can easily be shared withcolleagues or placed under source control to ensure consistent test executions.

To run a test using arguments loaded from an ARG_FILE namedexcelsior_args, which contains an argument group namedrobo-args:, use the following syntax:

gcloudfirebasetestandroidrunpath/to/excelsior_args:robo-args

[ARGSPEC]

An ARG_FILE:ARG_GROUP_NAME pair, where ARG_FILE is the path to a file containinggroups of test arguments in yaml format, and ARG_GROUP_NAME is the particularyaml object holding a group of arg:value pairs to use. Run$gcloud topic arg-filesfor more information and examples.

--app=APP

The path to the application binary file. The path may be in the local filesystemor in Google Cloud Storage using gs:// notation. Android App Bundles arespecified as .aab, all other files are assumed to be APKs.

--device=DIMENSION=VALUE,[…]

A list ofDIMENSION=VALUE pairs whichspecify a target device to test against. This flag may be repeated to specifymultiple devices. The four device dimensions are:model,version,locale, andorientation. If anydimensions are omitted, they will use a default value. The default value, andall possible values, for each dimension can be found with thelist command for that dimension, such as$gcloudfirebase test android models list.--device is now thepreferred way to specify test devices and may not be used in conjunction with--devices-ids,--os-version-ids,--locales, or--orientations. Omitting all of thepreceding dimension-related flags will run tests against a single device usingdefaults for all four device dimensions.

Examples:

--devicemodel=Nexus6--deviceversion=23,orientation=portrait--devicemodel=shamu,version=22,locale=zh_CN,orientation=default

--test=TEST

The path to the binary file containing instrumentation tests. The given path maybe in the local filesystem or in Google Cloud Storage using a URL beginning withgs://.

--timeout=TIMEOUT

The max time this test execution can run before it is cancelled (default: 15m).It does not include any time necessary to prepare and clean up the targetdevice. The maximum possible testing time is 45m on physical devices and 60m onvirtual devices. TheTIMEOUT units can be h, m, or s. Ifno unit is given, seconds are assumed. Examples:

• --timeout 1h is 1 hour
• --timeout 5m is 5 minutes
• --timeout 200s is 200 seconds
• --timeout 100 is 100 seconds

--type=TYPE

The type of test to run.TYPE must be one of:instrumentation,robo,game-loop.

--additional-apks=APK,[APK,…]

A list of up to 100 additional APKs to install, in addition to those beingdirectly tested. The path may be in the local filesystem or in Google CloudStorage using gs:// notation.

--app-package=APP_PACKAGE

(REMOVED) The Java package of the application under test. By default, theapplication package name is parsed from the APK manifest.

Flag --app-package has been removed.

--async

Invoke a test asynchronously without waiting for test results.

--auto-google-login

Automatically log into the test device using a preconfigured Google accountbefore beginning the test. Enabled by default, use --no-auto-google-login todisable.

--client-details=[KEY=VALUE,…]

Comma-separated, KEY=VALUE map of additional details to attach to the testmatrix. Arbitrary KEY=VALUE pairs may be attached to a test matrix to provideadditional context about the tests being run. When consuming the test results,such as in Cloud Functions or a CI system, these details can add additionalcontext such as a link to the corresponding pull request.

Example:

--client-details=buildNumber=1234,pullRequest=https://example.com/link/to/pull-request

To help you identify and locate your test matrix in the Firebase console, usethe matrixLabel key.

Example:

--client-details=matrixLabel="Example matrix label"

--directories-to-pull=[DIR_TO_PULL,…]

A list of paths that will be copied from the device's storage to the designatedresults bucket after the test is complete. These must be absolute paths under/sdcard,/storage, or/data/local/tmp(for example,--directories-to-pull/sdcard/tempDir1,/data/local/tmp/tempDir2). Path names are restricted tothe charactersa-zA-Z0-9_-./+. The paths/sdcard and/data will be made available and treated as implicit pathsubstitutions. E.g. if/sdcard on a particular device does not mapto external storage, the system will replace it with the external storage pathprefix for that device. Note that access to some directories on API levels 29and later may also be limited by scoped storage rules.

--environment-variables=[KEY=VALUE,…]

A comma-separated, key=value map of environment variables and their desiredvalues. The environment variables are mirrored as extra options to theaminstrument -e KEY1 VALUE1 … command and passed to your test runner(typically AndroidJUnitRunner). Examples:

Enable code coverage and provide a directory to store the coverage results whenusing Android Test Orchestrator (--use-orchestrator):

--environment-variablesclearPackageData=true,coverage=true,coverageFilePath=/sdcard/Download/

Enable code coverage and provide a file path to store the coverage results whennot using Android Test Orchestrator(--no-use-orchestrator):

--environment-variablescoverage=true,coverageFile=/sdcard/Download/coverage.ec

Note: If you need to embed a comma into aVALUE string, pleaserefer togcloud topicescaping for ways to change the default list delimiter.

--network-profile=PROFILE_ID

The name of the network traffic profile, for example--network-profile=LTE, which consists of a set of parameters toemulate network conditions when running the test (default: no network shaping;see available profiles listed by the $gcloud firebasetest network-profiles list command). This feature only works on physicaldevices.

--num-flaky-test-attempts=int

Specifies the number of times a test execution should be reattempted if one ormore of its test cases fail for any reason. An execution that initially failsbut succeeds on any reattempt is reported as FLAKY.

The maximum number of reruns allowed is 10. (Default: 0, which implies noreruns.) All additional attempts are executed in parallel.

--obb-files=OBB_FILE,[OBB_FILE]

A list of one or two Android OBB file names which will be copied to each testdevice before the tests will run (default: None). Each OBB file name mustconform to the format as specified by Android (e.g.[main|patch].0300110.com.example.android.obb) and will be installed into<shared-storage>/Android/obb/<package-name>/ on the test device.

--other-files=DEVICE_PATH=FILE_PATH,[…]

A list of device-path=file-path pairs that indicate the device paths to pushfiles to the device before starting tests, and the paths of files to push.

Device paths must be under absolute, approved paths (${EXTERNAL_STORAGE}, or${ANDROID_DATA}/local/tmp). Source file paths may be in the local filesystem orin Google Cloud Storage (gs://…).

Examples:

--other-files/sdcard/dir1/file1.txt=local/file.txt,/storage/dir2/file2.jpg=gs://bucket/file.jpg

This flag only copies files to the device. To install files, like OBB or APKfiles, see --obb-files and --additional-apks.

--performance-metrics

Monitor and record performance metrics: CPU, memory, network usage, and FPS(game-loop only). Enabled by default, use --no-performance-metrics to disable.

--record-video

Enable video recording during the test. Enabled by default, use--no-record-video to disable.

--results-bucket=RESULTS_BUCKET

The name of a Google Cloud Storage bucket where raw test results will be stored(default: "test-lab-<random-UUID>"). Note that the bucket must be owned bya billing-enabled project, and that using a non-default bucket will result inbilling charges for the storage used.

--results-dir=RESULTS_DIR

The name of aunique Google Cloud Storage object within the resultsbucket where raw test results will be stored (default: a timestamp with a randomsuffix). Caution: if specified, this argumentmust be unique foreach test matrix you create, otherwise results from multiple test matrices willbe overwritten or intermingled.

--results-history-name=RESULTS_HISTORY_NAME

The history name for your test results (an arbitrary string label; default: theapplication's label from the APK manifest). All tests which use the same historyname will have their results grouped together in the Firebase console in atime-ordered test history list.

--scenario-labels=LABEL,[LABEL,…]

A list of game-loop scenario labels (default: None). Each game-loop scenario maybe labeled in the APK manifest file with one or more arbitrary strings, creatinglogical groupings (e.g. GPU_COMPATIBILITY_TESTS). If--scenario-numbers and--scenario-labels are specifiedtogether, Firebase Test Lab will first execute each scenario from--scenario-numbers. It will then expand each given scenario labelinto a list of scenario numbers marked with that label, and execute thosescenarios.

--scenario-numbers=int,[int,…]

A list of game-loop scenario numbers which will be run as part of the test(default: all scenarios). A maximum of 1024 scenarios may be specified in onetest matrix, but the maximum number may also be limited by the overall test--timeout setting.

--test-package=TEST_PACKAGE

(REMOVED) The Java package name of the instrumentation test. By default, thetest package name is parsed from the APK manifest.

Flag --test-package has been removed.

--test-runner-class=TEST_RUNNER_CLASS

The fully-qualified Java class name of the instrumentation test runner (default:the last name extracted from the APK manifest).

--test-targets=TEST_TARGET,[TEST_TARGET,…]

A list of one or more test target filters to apply (default: run all testtargets). Each target filter must be fully qualified with the package name,class name, or test annotation desired. Any test filter supported byaminstrument -e … is supported. Seehttps://developer.android.com/reference/androidx/test/runner/AndroidJUnitRunnerfor more information. Examples:

• --test-targets "package com.my.package.name"
• --test-targets "notPackage com.package.to.skip"
• --test-targets "class com.foo.ClassName"
• --test-targets "notClass com.foo.ClassName#testMethodToSkip"
• --test-targets "annotation com.foo.AnnotationToRun"
• --test-targets "size large notAnnotation com.foo.AnnotationToSkip"

--use-orchestrator

Whether each test runs in its own Instrumentation instance with the Android TestOrchestrator (default: Orchestrator is not used, same as specifying--no-use-orchestrator). Orchestrator is only compatible with AndroidJUnitRunnerv1.1 or higher. Seehttps://developer.android.com/training/testing/junit-runner.html#using-android-test-orchestratorfor more information about Android Test Orchestrator.

--resign

Make Robo re-sign the app-under-test APK for a higher quality crawl. If your appcannot properly function when re-signed, disable this feature. When anapp-under-test APK is not re-signed, Robo crawl is slower and Robo has access toless information about the state of the crawled app, which reduces crawlquality. Consequently, if your Roboscript has actions on elements ofRecyclerView or AdapterView, and you disable APK re-signing, those actions mightrequire manual tweaking because Robo does not identify RecyclerView andAdapterView in this mode. Enabled by default, use--no-resign todisable.

--robo-directives=[TYPE:RESOURCE_NAME=INPUT,…]

A comma-separated (<type>:<key>=<value>) map ofrobo_directives that you can use to customize the behavior of Robotest. Thetype specifies the action type of the directive, whichmay take on valuesclick,text orignore.If notype is provided,text will be used by default.Each key should be the Android resource name of a target UI element and eachvalue should be the text input for that element. Values are only permitted fortext type elements, so no value should be specified forclick andignore type elements. No more than oneclick element is allowed.

To provide custom login credentials for your app, use

--robo-directivestext:username_resource=username,text:password_resource=password

To instruct Robo to click on the sign-in button, use

--robo-directivesclick:sign_in_button=

To instruct Robo to ignore any UI elements with resource names which equal orstart with the user-defined value, use

--robo-directivesignore:ignored_ui_element_resource_name=

To learn more about Robo test and robo_directives, seehttps://firebase.google.com/docs/test-lab/android/command-line#custom_login_and_text_input_with_robo_test.

Caution: You should only use credentials for test accounts that are notassociated with real users.

--robo-script=ROBO_SCRIPT

The path to a Robo Script JSON file. The path may be in the local filesystem orin Google Cloud Storage using gs:// notation. You can guide the Robo test toperform specific actions by recording a Robo Script in Android Studio and thenspecifying this argument. Learn more athttps://firebase.google.com/docs/test-lab/robo-ux-test#scripting.

--device-ids=MODEL_ID,[MODEL_ID,…],-dMODEL_ID,[MODEL_ID,…]

The list of MODEL_IDs to test against (default: one device model determined bythe Firebase Test Lab device catalog; see TAGS listed by the$gcloud firebasetest android models list command).

--locales=LOCALE,[LOCALE,…],-lLOCALE,[LOCALE,…]

The list of LOCALEs to test against (default: a single locale determined by theFirebase Test Lab device catalog).

--orientations=ORIENTATION,[ORIENTATION],-oORIENTATION,[ORIENTATION]

The device orientation(s) to test against (default: portrait). Specifying'default' will pick the preferred orientation for the app.ORIENTATION must be one of:portrait,landscape,default.

--os-version-ids=OS_VERSION_ID,[…],-vOS_VERSION_ID,[…]

The list of OS_VERSION_IDs to test against (default: a version ID determined bythe Firebase Test Lab device catalog).

--filter=EXPRESSION

Apply a Boolean filterEXPRESSION to each resource itemto be listed. If the expression evaluatesTrue, then that item islisted. For more details and examples of filter expressions, run $gcloud topic filters. This flaginteracts with other flags that are applied in this order:--flatten,--sort-by,--filter,--limit.

--limit=LIMIT

Maximum number of resources to list. The default isunlimited. Thisflag interacts with other flags that are applied in this order:--flatten,--sort-by,--filter,--limit.

--page-size=PAGE_SIZE

Some services group resource list output into pages. This flag specifies themaximum number of resources per page. The default is determined by the serviceif it supports paging, otherwise it isunlimited (no paging).Paging may be applied before or after--filter and--limit depending on the service.

--sort-by=[FIELD,…]

Comma-separated list of resource field key names to sort by. The default orderis ascending. Prefix a field with ``~´´ for descending order on thatfield. This flag interacts with other flags that are applied in this order:--flatten,--sort-by,--filter,--limit.

Run$gcloud help for details.