An object detection model is similar to an image labeling model, but rather thanassign labels to entire images, it assigns labels to regions of images. You canuse object detection models to recognize and locate objects in an image or totrack an object's movements across a series of images.

To train an object detection model, you provide AutoML Vision Edge a set ofimages with corresponding object labels and object boundaries. AutoML VisionEdge uses this dataset to train a new model in the cloud, which you can use foron-device object detection.

Before you begin

• If you don't already have a Firebase project, create one in theFirebase console.

• Familiarize yourself with the guidelines presented inInclusive ML guide - AutoML.

• If you just want to try AutoML Vision Edge, and don't have your own trainingdata, download a sample dataset such as one of the following:

  • One of TensorFlow'srecommended datasets.
  • A dataset hosted onKaggle.

1. Assemble your training data

First, you need to put together a training dataset of labeled images. Keep thefollowing guidelines in mind:

• The images must be in one of the following formats: JPEG, PNG, GIF, BMP, ICO.

• Each image must be 30MB or smaller. Note that AutoML Vision Edge downscalesmost images during preprocessing, so there's generally no accuracy benefit toproviding very high resolution images.

• Include at least 10, and preferably 100 or more, examples of each label.

• Include multiple angles, resolutions, and backgrounds for each label.

• The training data should be as close as possible to the data on whichpredictions are to be made. For example, if your use case involves blurry andlow-resolution images (such as from a security camera), your training datashould be composed of blurry, low-resolution images.

• The models generated by AutoML Vision Edge are optimized for photographs ofobjects in the real world. They might not work well for X-rays, hand drawings,scanned documents, receipts, and so on.

  Also, the models can't generally predict labels that humans can't assign. So,if a human can't assign labels by looking at the image for 1-2 seconds, themodel likely can't be trained to do it either.

If you need help collecting training data, take a look at the open-sourceCustom Image Classifier app on GitHub.

When you have your training images ready, prepare them to import intoGoogle Cloud. You have two options:

Option 1:Cloud Storage with CSV index

Upload your training images toGoogle Cloud Storage and prepare a CSV file listing the URL of each image, and, optionally, thecorrect object labels and bounding regions for each image. This option ishelpful when using large datasets.

For example, upload your images toCloud Storage, and prepare a CSV file likethe following:

gs://your-training-data-bucket/001.jpg,accordion,0.2,0.4,,,0.3,0.5,,gs://your-training-data-bucket/001.jpg,tuba,0.2,0.5,,,0.4,0.8,,gs://your-training-data-bucket/002.jpg,accordion,0.2,0.2,,,0.9,0.8,,

Object bounding boxes are specified as relative coordinates in the image. SeeFormatting a training data CSV.

The images must be stored in a bucket that's in theus-central1 region andpart of your Firebase project's correspondingGoogle Cloud project.

Option 2: Unlabeled images

Label your training images and draw object boundaries in theGoogle Cloud console after you upload them. This is only recommended for smalldatasets. See the next step.

2. Train your model

Next, train a model using your images:

1. Open theVision Datasetspage in theGoogle Cloud console. Select your project when prompted.

2. ClickNew dataset, provide a name for the dataset, select the type ofmodel you want to train, and clickCreate dataset.

3. On your dataset'sImport tab, upload your training images, a zip archiveof your training images or a CSV file containing theCloud Storagelocations you uploaded them to. SeeAssemble your training data.

4. After the import task completes, use theImages tab to verify thetraining data.

5. If you didn't upload a CSV, for each image, draw bounding boxes around theobjects you want to recognize and label each object.

6. On theTrain tab, clickStart training.

   1. Name the model and select theEdge model type.

   2. Configure the following training settings, which govern the performanceof the generated model:

      Optimize model for...

      The model configuration to use. You can train faster, smaller, models when low latency or small package size are important, or slower, larger, models when accuracy is most important.

      Node hour budget

      The maximum time, in compute hours, to spend training the model. More training time generally results in a more accurate model. Note that training can be completed in less than the specified time if the system determines that the model is optimized and additional training would not improve accuracy. You are billed only for the hours actually used. Typical training times Very small sets 1 hour 500 images 2 hours 1,000 images 3 hours 5,000 images 6 hours 10,000 images 7 hours 50,000 images 11 hours 100,000 images 13 hours 1,000,000 images 18 hours

3. Evaluate your model

When training completes, you can click theEvaluate tab tosee performance metrics for the model.

One important use of this page is to determine the confidence threshold that worksbest for your model. The confidence threshold is the minimum confidence the modelmust have for it to assign a label to an image. By moving theConfidence thresholdslider, you can see how different thresholds affect the model’s performance.Model performance is measured using two metrics:precision andrecall.

In the context of image classification,precision is the ratio of the numberof images that were correctly labeled to the number of images the model labeledgiven the selected threshold. When a model has high precision, it assignslabels incorrectly less often (fewer false positives).

Recall is the ratio of the number of images that were correctly labeled to thenumber of images that had content the model should have been able to label. Whena model has high recall, it fails to assign any label less often (fewer falsenegatives).

Whether you optimize for precision or recall will depend on your use case. SeetheAutoML Vision beginners' guide and theInclusive ML guide - AutoML for more information.

When you find a confidence threshold that produces metrics you're comfortable with,make note of it; you will use the confidence threshold to configure the model in yourapp. (You can use this tool any time to get an appropriate threshold value.)

4. Publish or download your model

If you are satisfied with the model's performance and want to use it in an app,you have three options, from which you can choose any combination: deploy themodel for online prediction, publish the model to Firebase, or download themodel and bundle it with your app.

Deploy the model

On your dataset'sTest & use tab, you can deploy your model for onlineprediction, which runs your model in the cloud. This option is covered in theCloud AutoML docs. Thedocs on this site deal with the remaining two options.

Publish the model

By publishing the model to Firebase, you can update the model without releasinga new app version, and you can useRemote Config andA/B Testing todynamically serve different models to different sets of users.

If you choose to only provide the model by hosting it with Firebase, and notbundle it with your app, you can reduce the initial download size of your app.Keep in mind, though, that if the model is not bundled with your app, anymodel-related functionality will not be available until your app downloads themodel for the first time.

To publish your model, you can use either of two methods:

• Download the TF Lite model from your dataset'sTest & use page in theGoogle Cloud console, and then upload the model on theCustom model page of theFirebase console. This is usuallythe easiest way to publish a single model.
• Publish the model directly from yourGoogle Cloud project to Firebase usingthe Admin SDK. You can use this method to batch publish several models or tohelp create automated publishing pipelines.

To publish the model with the Admin SDKmodel management API:

1. Install and initialize the SDK.

2. Publish the model.

   You will need to specify the model's resource identifier, which is a stringthat looks like the following example:

   projects/PROJECT_NUMBER/locations/us-central1/models/MODEL_ID

   PROJECT_NUMBER	The project number of theCloud Storage bucket that contains the model. This might be your Firebase project or anotherGoogle Cloud project. You can find this value on the Settings page of theFirebase console or theGoogle Cloud console dashboard.
   MODEL_ID	The model's ID, which you got from the AutoML Cloud API.

   Python

   # First, import and initialize the SDK.# Get a reference to the AutoML modelsource=ml.TFLiteAutoMlSource('projects/{}/locations/us-central1/models/{}'.format(# See above for information on these values.project_number,model_id))# Create the model objecttflite_format=ml.TFLiteFormat(model_source=source)model=ml.Model(display_name="example_model",# This is the name you will use from your app to load the model.tags=["examples"],# Optional tags for easier management.model_format=tflite_format)# Add the model to your Firebase project and publish itnew_model=ml.create_model(model)new_model.wait_for_unlocked()ml.publish_model(new_model.model_id)

   Node.js

   // First, import and initialize the SDK.(async()=>{// Get a reference to the AutoML model. See above for information on these// values.constautomlModel=`projects/${projectNumber}/locations/us-central1/models/${modelId}`;// Create the model object and add the model to your Firebase project.constmodel=awaitml.createModel({displayName:'example_model',// This is the name you use from your app to load the model.tags:['examples'],// Optional tags for easier management.tfliteModel:{automlModel:automlModel},});// Wait for the model to be ready.awaitmodel.waitForUnlocked();// Publish the model.awaitml.publishModel(model.modelId);process.exit();})().catch(console.error);

Download & bundle the model with your app

By bundling your model with your app, you can ensure your app's ML featuresstill work when the Firebase-hosted model isn't available.

If you both publish the model and bundle it with your app, the app will use thelatest version available.

To download your model, clickTF Lite on your dataset'sTest & use page.

Next steps

Now that you have published or downloaded the model, learn how to use the modelin youriOS+ andAndroid apps.