This page is about an old version of the Custom Model API, which was part of ML Kit for Firebase. For the latest docs, seethe latest version in theFirebase ML section.

You can use ML Kit to perform on-device inference with aTensorFlow Lite model.

This API requires Android SDK level 16 (Jelly Bean) or newer.

Before you begin

1. If you haven't already,add Firebase to your Android project.
2. Add the dependencies for the ML Kit Android libraries to your module (app-level) Gradle file (usuallyapp/build.gradle):

   applyplugin:'com.android.application'applyplugin:'com.google.gms.google-services'dependencies{// ...implementation'com.google.firebase:firebase-ml-model-interpreter:22.0.3'}

3. Convert the TensorFlow model you want to use to TensorFlow Lite format. See TOCO: TensorFlow Lite Optimizing Converter.

Host or bundle your model

Before you can use a TensorFlow Lite model for inference in your app, you must make the model available to ML Kit. ML Kit can use TensorFlow Lite models hosted remotely using Firebase, bundled with the app binary, or both.

By hosting a model on Firebase, you can update the model without releasing a new app version, and you can useRemote Config andA/B Testing to dynamically serve different models to different sets of users.

If you choose to only provide the model by hosting it with Firebase, and not bundle it with your app, you can reduce the initial download size of your app. Keep in mind, though, that if the model is not bundled with your app, any model-related functionality will not be available until your app downloads the model for the first time.

By bundling your model with your app, you can ensure your app's ML features still work when the Firebase-hosted model isn't available.

Host models on Firebase

To host your TensorFlow Lite model on Firebase:

1. In theML Kit section of theFirebase console, click theCustom tab.
2. ClickAdd custom model (orAdd another model).
3. Specify a name that will be used to identify your model in your Firebase project, then upload the TensorFlow Lite model file (usually ending in.tflite or.lite).
4. In your app's manifest, declare that INTERNET permission is required:

   <uses-permission android:name="android.permission.INTERNET" />

After you add a custom model to your Firebase project, you can reference the model in your apps using the name you specified. At any time, you can upload a new TensorFlow Lite model, and your app will download the new model and start using it when the app next restarts. You can define the device conditions required for your app to attempt to update the model (see below).

Bundle models with an app

To bundle your TensorFlow Lite model with your app, copy the model file (usuallyending in.tflite or.lite) to your app'sassets/ folder. (You might needto create the folder first by right-clicking theapp/ folder, then clickingNew > Folder > Assets Folder.)

Then, add the following to your app'sbuild.gradle file to ensure Gradledoesn’t compress the models when building the app:

android {    // ...    aaptOptions {        noCompress "tflite"  // Your model's file extension: "tflite", "lite", etc.    }}

The model file will be included in the app package and available to ML Kitas a raw asset.

Load the model

Configure a Firebase-hosted model

If you hosted your model with Firebase, create aFirebaseCustomRemoteModelobject, specifying the name you assigned the model when you uploaded it:

FirebaseCustomRemoteModelremoteModel=newFirebaseCustomRemoteModel.Builder("your_model").build();

valremoteModel=FirebaseCustomRemoteModel.Builder("your_model").build()

Then, start the model download task, specifying the conditions under which youwant to allow downloading. If the model isn't on the device, or if a newerversion of the model is available, the task will asynchronously download themodel from Firebase:

FirebaseModelDownloadConditionsconditions=newFirebaseModelDownloadConditions.Builder().requireWifi().build();FirebaseModelManager.getInstance().download(remoteModel,conditions).addOnCompleteListener(newOnCompleteListener<Void>(){@OverridepublicvoidonComplete(@NonNullTask<Void>task){// Success.}});

valconditions=FirebaseModelDownloadConditions.Builder().requireWifi().build()FirebaseModelManager.getInstance().download(remoteModel,conditions).addOnCompleteListener{// Success.}

Many apps start the download task in their initialization code, but you can doso at any point before you need to use the model.

Configure a local model

If you bundled the model with your app, create aFirebaseCustomLocalModelobject, specifying the filename of the TensorFlow Lite model:

FirebaseCustomLocalModellocalModel=newFirebaseCustomLocalModel.Builder().setAssetFilePath("your_model.tflite").build();

vallocalModel=FirebaseCustomLocalModel.Builder().setAssetFilePath("your_model.tflite").build()

Create an interpreter from your model

After you configure your model sources, create aFirebaseModelInterpreterobject from one of them.

If you only have a locally-bundled model, just create an interpreter from yourFirebaseCustomLocalModel object:

FirebaseModelInterpreterinterpreter;try{FirebaseModelInterpreterOptionsoptions=newFirebaseModelInterpreterOptions.Builder(localModel).build();interpreter=FirebaseModelInterpreter.getInstance(options);}catch(FirebaseMLExceptione){// ...}

valoptions=FirebaseModelInterpreterOptions.Builder(localModel).build()valinterpreter=FirebaseModelInterpreter.getInstance(options)

If you have a remotely-hosted model, you will have to check that it has beendownloaded before you run it. You can check the status of the model downloadtask using the model manager'sisModelDownloaded() method.

Although you only have to confirm this before running the interpreter, if youhave both a remotely-hosted model and a locally-bundled model, it might makesense to perform this check when instantiating the model interpreter: create aninterpreter from the remote model if it's been downloaded, and from the localmodel otherwise.

FirebaseModelManager.getInstance().isModelDownloaded(remoteModel).addOnSuccessListener(newOnSuccessListener<Boolean>(){@OverridepublicvoidonSuccess(BooleanisDownloaded){FirebaseModelInterpreterOptionsoptions;if(isDownloaded){options=newFirebaseModelInterpreterOptions.Builder(remoteModel).build();}else{options=newFirebaseModelInterpreterOptions.Builder(localModel).build();}FirebaseModelInterpreterinterpreter=FirebaseModelInterpreter.getInstance(options);// ...}});

FirebaseModelManager.getInstance().isModelDownloaded(remoteModel).addOnSuccessListener{isDownloaded->valoptions=if(isDownloaded){FirebaseModelInterpreterOptions.Builder(remoteModel).build()}else{FirebaseModelInterpreterOptions.Builder(localModel).build()}valinterpreter=FirebaseModelInterpreter.getInstance(options)}

If you only have a remotely-hosted model, you should disable model-relatedfunctionality—for example, grey-out or hide part of your UI—untilyou confirm the model has been downloaded. You can do so by attaching a listenerto the model manager'sdownload() method:

FirebaseModelManager.getInstance().download(remoteModel,conditions).addOnSuccessListener(newOnSuccessListener<Void>(){@OverridepublicvoidonSuccess(Voidv){// Download complete. Depending on your app, you could enable// the ML feature, or switch from the local model to the remote// model, etc.}});

FirebaseModelManager.getInstance().download(remoteModel,conditions).addOnCompleteListener{// Download complete. Depending on your app, you could enable the ML// feature, or switch from the local model to the remote model, etc.}

Specify the model's input and output

Next, configure the model interpreter's input and output formats.

A TensorFlow Lite model takes as input and produces as output one or moremultidimensional arrays. These arrays contain eitherbyte,int,long, orfloat values. You mustconfigure ML Kit with the number and dimensions ("shape") of the arrays yourmodel uses.

If you don't know the shape and data type of your model's input and output,you can use the TensorFlow Lite Python interpreter to inspect your model. Forexample:

importtensorflowastfinterpreter=tf.lite.Interpreter(model_path="my_model.tflite")interpreter.allocate_tensors()# Print input shape and typeprint(interpreter.get_input_details()[0]['shape'])# Example: [1 224 224 3]print(interpreter.get_input_details()[0]['dtype'])# Example: <class 'numpy.float32'># Print output shape and typeprint(interpreter.get_output_details()[0]['shape'])# Example: [1 1000]print(interpreter.get_output_details()[0]['dtype'])# Example: <class 'numpy.float32'>

After you have determined the format of your model's input and output, you canconfigure your app's model interpreter by creating aFirebaseModelInputOutputOptions object.

For example, a floating-point image classification model might take as input anNx224x224x3 array offloat values, representing a batch ofN 224x224 three-channel (RGB) images, and produce as output a list of1000float values, each representing the probability the image is a member ofone of the 1000 categories the model predicts.

For such a model, you would configure the model interpreter's input and outputas shown below:

FirebaseModelInputOutputOptionsinputOutputOptions=newFirebaseModelInputOutputOptions.Builder().setInputFormat(0,FirebaseModelDataType.FLOAT32,newint[]{1,224,224,3}).setOutputFormat(0,FirebaseModelDataType.FLOAT32,newint[]{1,5}).build();

valinputOutputOptions=FirebaseModelInputOutputOptions.Builder().setInputFormat(0,FirebaseModelDataType.FLOAT32,intArrayOf(1,224,224,3)).setOutputFormat(0,FirebaseModelDataType.FLOAT32,intArrayOf(1,5)).build()

Perform inference on input data

For example, if you have an image classification model with an input shape of[1 224 224 3] floating-point values, you could generate an input array from aBitmap object as shown in the following example:

Bitmapbitmap=getYourInputImage();bitmap=Bitmap.createScaledBitmap(bitmap,224,224,true);intbatchNum=0;float[][][][]input=newfloat[1][224][224][3];for(intx=0;x <224;x++){for(inty=0;y <224;y++){intpixel=bitmap.getPixel(x,y);// Normalize channel values to [-1.0, 1.0]. This requirement varies by// model. For example, some models might require values to be normalized// to the range [0.0, 1.0] instead.input[batchNum][x][y][0]=(Color.red(pixel)-127)/128.0f;input[batchNum][x][y][1]=(Color.green(pixel)-127)/128.0f;input[batchNum][x][y][2]=(Color.blue(pixel)-127)/128.0f;}}

valbitmap=Bitmap.createScaledBitmap(yourInputImage,224,224,true)valbatchNum=0valinput=Array(1){Array(224){Array(224){FloatArray(3)}}}for(xin0..223){for(yin0..223){valpixel=bitmap.getPixel(x,y)// Normalize channel values to [-1.0, 1.0]. This requirement varies by// model. For example, some models might require values to be normalized// to the range [0.0, 1.0] instead.input[batchNum][x][y][0]=(Color.red(pixel)-127)/255.0finput[batchNum][x][y][1]=(Color.green(pixel)-127)/255.0finput[batchNum][x][y][2]=(Color.blue(pixel)-127)/255.0f}}

Then, create aFirebaseModelInputs object with yourinput data, and pass it and the model's input and output specification to themodel interpreter'srun method:

FirebaseModelInputsinputs=newFirebaseModelInputs.Builder().add(input)// add() as many input arrays as your model requires.build();firebaseInterpreter.run(inputs,inputOutputOptions).addOnSuccessListener(newOnSuccessListener<FirebaseModelOutputs>(){@OverridepublicvoidonSuccess(FirebaseModelOutputsresult){// ...}}).addOnFailureListener(newOnFailureListener(){@OverridepublicvoidonFailure(@NonNullExceptione){// Task failed with an exception// ...}});

valinputs=FirebaseModelInputs.Builder().add(input)// add() as many input arrays as your model requires.build()firebaseInterpreter.run(inputs,inputOutputOptions).addOnSuccessListener{result->// ...}.addOnFailureListener{e->// Task failed with an exception// ...}

If the call succeeds, you can get the output by calling thegetOutput() methodof the object that is passed to the success listener. For example:

float[][]output=result.getOutput(0);float[]probabilities=output[0];

valoutput=result.getOutput<Array<FloatArray>>(0)valprobabilities=output[0]

How you use the output depends on the model you are using.

For example, if you are performing classification, as a next step, you mightmap the indexes of the result to the labels they represent:

BufferedReaderreader=newBufferedReader(newInputStreamReader(getAssets().open("retrained_labels.txt")));for(inti=0;i <probabilities.length;i++){Stringlabel=reader.readLine();Log.i("MLKit",String.format("%s: %1.4f",label,probabilities[i]));}

valreader=BufferedReader(InputStreamReader(assets.open("retrained_labels.txt")))for(iinprobabilities.indices){vallabel=reader.readLine()Log.i("MLKit",String.format("%s: %1.4f",label,probabilities[i]))}

Appendix: Model security

Regardless of how you make your TensorFlow Lite models available toML Kit, ML Kit stores them in the standard serialized protobuf format inlocal storage.

In theory, this means that anybody can copy your model. However,in practice, most models are so application-specific and obfuscated byoptimizations that the risk is similar to that of competitors disassembling andreusing your code. Nevertheless, you should be aware of this risk before you usea custom model in your app.

On Android API level 21 (Lollipop) and newer, the model is downloaded to a directory that is excluded from automatic backup.

On Android API level 20 and older, the model is downloaded to a directory namedcom.google.firebase.ml.custom.models in app-private internal storage. If you enabled file backup usingBackupAgent, you might choose to exclude this directory.