This page describes an old version ofFirebase ML. See thelatest version of this page in theFirebase ML section.

After you train a new custom model or AutoML Vision Edge model, you can useA/B Testing to see how well the new model performs in real-world conditions,compared to the model you already use. After you confirm that your new model isan improvement, you can easily roll out the new model to all of your users,without requiring an app update.

This page shows how you might conduct an A/B test that evaluates two versionsof a model that powers a hypothetical visual plant search feature. This featureuses a custom image labeling model to help users identify plant species fromimages of them.

Suppose you just published a new plant labeling model,plant_labeler_v2 and you want to run an experiment that compares itwith your current model, namedplant_labeler_v1. The steps belowshow how to set up the experiment, run it, and take action on the results.

1. Make your model remotely configurable

The first step to A/B testing your models is to modify your app to use aRemote Config parameter to determine which model it uses. Initially, youwill set the default value of this parameter to be the model that your appalready uses, but because the model name is controlled by a remotelyconfigurable parameter, you can change and experiment with different modelswithout having to push app updates to your users every time.

So, if you published your current model under the nameplant_labeler_v1, you would, in your app initialization code, setplant_labeler_v1 as the default value of theplant_labeler_model parameter, as in the following example:

valremoteConfig=FirebaseRemoteConfig.getInstance()valremoteConfigDefaults=HashMap<String,Any>()remoteConfigDefaults["plant_labeler_model"]="plant_labeler_v1"Tasks.await(remoteConfig.setDefaultsAsync(remoteConfigDefaults))remoteConfig.fetchAndActivate().addOnSuccessListener{success->if(success){// Okay to get remote values.// ...}}

finalFirebaseRemoteConfigremoteConfig=FirebaseRemoteConfig.getInstance();Map<String,Object>remoteConfigDefaults=newHashMap<>();remoteConfigDefaults.put("plant_labeler_model","plant_labeler_v1");Tasks.await(remoteConfig.setDefaultsAsync(remoteConfigDefaults));remoteConfig.fetchAndActivate().addOnSuccessListener(newOnSuccessListener<Boolean>(){@OverridepublicvoidonSuccess(Booleansuccess){if(success){// Okay to get remote values.// ...}}});

Then, change your model setup code to load the model specified by theplant_labeler_model parameter:

valrcValue=remoteConfig.getValue("plant_labeler_model")valremoteModelName=rcValue.asString()// ...valremoteModel=FirebaseRemoteModel.Builder(remoteModelName).enableModelUpdates(true).setInitialDownloadConditions(initialConditions).setUpdatesDownloadConditions(updateConditions).build()FirebaseModelManager.getInstance().registerRemoteModel(remoteModel)// Optionally configure a local model:// https://firebase.google.com/docs/ml-kit/android/label-images-with-automl#configure-a-local-model-source// https://firebase.google.com/docs/ml-kit/android/use-custom-models#configure_a_local_model

FirebaseRemoteConfigValuercValue=remoteConfig.getValue("plant_labeler_model");StringremoteModelName=rcValue.asString();// ...FirebaseRemoteModelremoteModel=newFirebaseRemoteModel.Builder(remoteModelName).enableModelUpdates(true).setInitialDownloadConditions(initialConditions).setUpdatesDownloadConditions(updateConditions).build();FirebaseModelManager.getInstance().registerRemoteModel(remoteModel);// Optionally configure a local model:// https://firebase.google.com/docs/ml-kit/android/label-images-with-automl#configure-a-local-model-source// https://firebase.google.com/docs/ml-kit/android/use-custom-models#configure_a_local_model

Now that your app uses aRemote Config parameter to determine which model toload, you can change the model just by publishing a new model and assigning itsname to theRemote Config parameter. This capability letsA/B Testing assigndifferent models to different users for the purpose of comparing them.

Before you continue, also make the following addition to your model downloadcode:

FirebaseModelManager.getInstance().downloadRemoteModelIfNeeded(remoteModel).addOnSuccessListener{// If the model downloaded was specified by a remote parameter, log an// event, which will be our experiment's activation event.if(rcValue.source==FirebaseRemoteConfig.VALUE_SOURCE_REMOTE){FirebaseAnalytics.getInstance(this).logEvent("nondefault_model_downloaded",null)}}

FirebaseModelManager.getInstance().downloadRemoteModelIfNeeded(remoteModel).addOnSuccessListener(newOnSuccessListener<Void>(){@OverridepublicvoidonSuccess(VoidaVoid){// If the model downloaded was specified by a remote parameter, log an// event, which will be our experiment's activation event.if(rcValue.getSource()==FirebaseRemoteConfig.VALUE_SOURCE_REMOTE){FirebaseAnalytics.getInstance(YourActivity.this).logEvent("nondefault_model_downloaded",null);}}});

The above code logs a custom Analytics event that you will use later as yourexperiment'sactivation event. An activation event is an event theuser must trigger before they are considered to be part of the experiment. Thisensures that users won't be recorded in your A/B test until their device hasfinished downloading their custom ML Kit model.

2. Determine a goal metric

The next step is to decide how you will measure the success of your model,and to make sure your app is collecting the data necessary to test how welldifferent versions of the model perform according to that metric.

A/B Testing has several built-in metrics, including revenue, dailyengagement, and user retention. These metrics are often useful for testingdifferent UX flows or fine-tuning parameters, but might not make sense forevaluating your model and use case. In this situation, you can instead try tooptimize for a custom Analytics event.

Using the hypothetical visual plant search feature as an example, suppose youpresented search results to your user in the order of the model's confidence ineach result. One way you could get an idea of your model's accuracy would be bylooking at how often users opened the first search result.

To test which model best achieved the goal of maximizing top result clicks,you would log a custom event whenever a user tapped the first item in the resultlist.

FirebaseAnalytics.getInstance(this).logEvent("first_result_opened",null)

FirebaseAnalytics.getInstance(YourActivity.this).logEvent("first_result_opened",null);

The metric you test for ultimately depends on how your app uses yourmodel.

At this point, you can deploy your app to the Play Store. Your app will continue to use your original model,but theRemote Config and Analytics code you added will let you experimentwith different models using only theFirebase console.

3. Run anA/B Testing experiment

Now that your app is in your users' hands and is collecting analytics data,create anA/B Testing experiment that tests the effect of using your newmodel instead of the current model.

To create the experiment:

1. On theEvents page of theFirebase console, verify you are logging the relevant Analytics events: the activation event and goal metric.

   Your app needs to log each event at least once before it appears in theFirebase console.

2. In theFirebase console, open theA/B Testing section.

3. Create a new experiment:

   1. ClickCreate experiment >Remote Config.

   2. In theTargeting section:

      • Choose your app from the list
      • Specify how many of your users you want to include in the experiment
      • Select the activation event you started logging (in this example,nondefault_model_downloaded)

   3. In theGoals section, choose the goal metric you determined in the previous section (in this example,first_result_opened) from the list of goal metrics, and select any additional metrics you want to track, such as purchase revenue or crash-free users.

   4. In theVariants section, define two variants:

      • Control group (created automatically)
      • Experimental plant labeler

      For theControl group, create aplant_labeler_model parameter and set it toplant_labeler_v1. Users assigned to the control group will use the old model. (Don't set the parameter to(no change), since in your app, you're testing that you're using a remote value.)

      For theExperimental plant labeler variant, set theplant_labeler_model parameter toplant_labeler_v2 (assuming you published your new model under that name). Users assigned to this variant will use the new model.

   

Start the experiment and let it run for several days or more, untilA/B Testing declares a leader. If the experiment cannot determine a leader,you might need toexpand the experiment to more users.

4. Roll out the winning variant to all users

AfterA/B Testing has collected enough information to declare aleader—in this case, the variant that maximized top search resultclicks—you can decide whether to roll out the winning variant (or anothervariant) to all of your users.

In theA/B Testing section of theFirebase console, open the detailsview of the completed experiment. From this view, you can see how each variantperformed according to your goal metric and any secondary metrics you selected.With this information, you can decide whether to roll out the leading variant oranother variant.

To roll out a variant to all users, clickmore_vert > Roll out variant on theexperiment's details page. Once you do so, the value of theplant_labeler_model parameter will beplant_labeler_v2for all users.

In a future app update, you should change the default value of theplant_labeler_model parameter toplant_labeler_v2 and update the bundledmodel if you use one. Your users are already using the latest model, though, soyou can push this update as part of the published app whenever it's convenient,such as when you next make a feature update.