In this tutorial, you learn how to build an object detection model using ML.NET Model Builder and Azure Machine Learning to detect and locate stop signs in images.

Prerequisites

For a list of prerequisites and installation instructions, visit theModel Builder installation guide.

Model Builder object detection overview

Object detection is a computer vision problem. While closely related to image classification, object detection performs image classification at a more granular scale. Object detection both locatesand categorizes entities within images. Object detection models are commonly trained using deep learning and neural networks. SeeDeep learning vs machine learning for more information.

Use object detection when images contain multiple objects of different types.

Some use cases for object detection include:

• Self-Driving Cars
• Robotics
• Face Detection
• Workplace Safety
• Object Counting
• Activity Recognition

This sample creates a C# console application that detects stop signs in images using a machine learning model built with Model Builder. You can find the source code for this tutorial at thesamples browser.

Prepare and understand the data

The Stop Sign dataset consists of 50 images downloaded fromUnsplash, each of which contain at least one stop sign.

Create a new VoTT project

1. Download the dataset of 50 stop sign images and unzip.

2. Download VoTT (Visual Object Tagging Tool).

3. Open VoTT and selectNew Project.

   

4. InProject Settings, change theDisplay Name to "StopSignObjDetection".

5. Change theSecurity Token toGenerate New Security Token.

6. Next toSource Connection, selectAdd Connection.

7. InConnection Settings, change theDisplay Name for the source connection to "StopSignImages", and selectLocal File System as theProvider. For theFolder Path, select theStop-Signs folder which contains the 50 training images, and then selectSave Connection.

   

8. InProject Settings, change theSource Connection toStopSignImages (the connection you just created).

9. Change theTarget Connection toStopSignImages as well. YourProject Settings should now look similar to this screenshot:

   

10. SelectSave Project.

Add tag and label images

You should now see a window with preview images of all the training images on the left, a preview of the selected image in the middle, and aTags column on the right. This screen is theTags editor.

1. Select the first (plus-shaped) icon in theTags toolbar to add a new tag.

   

2. Name the tag "Stop-Sign" and hitEnter on your keyboard.

   

3. Click and drag to draw a rectangle around each stop sign in the image. If the cursor does not let you draw a rectangle, try selecting theDraw Rectangle tool from the toolbar on the top, or use the keyboard shortcutR.

4. After drawing your rectangle, select theStop-Sign tag that you created in the previous steps to add the tag to the bounding box.

5. Click on the preview image for the next image in the dataset and repeat this process.

6. Continue steps 3-4 for every stop sign in every image.

   

Export your VoTT JSON

Once you have labeled all of your training images, you can export the file that will be used by Model Builder for training.

1. Select the fourth icon in the left toolbar (the one with the diagonal arrow in a box) to go to theExport Settings.

2. Leave theProvider asVoTT JSON.

3. Change theAsset State toOnly tagged Assets.

4. UncheckInclude Images. If you include the images, then the training images will be copied to the export folder that is generated, which is not necessary.

5. SelectSave Export Settings.

   

6. Go back to theTags editor (the second icon in the left toolbar shaped like a ribbon). In the top toolbar, select theExport Project icon (the last icon shaped like an arrow in a box), or use the keyboard shortcutCtrl+E.

   

This export will create a new folder calledvott-json-export in yourStop-Sign-Images folder and will generate a JSON file namedStopSignObjDetection-export in that new folder. You will use this JSON file in the next steps for training an object detection model in Model Builder.

Create a console application

In Visual Studio, create aC# console application calledStopSignDetection.

Create anmbconfig file

1. InSolution Explorer, right-click theStopSignDetection project, and selectAdd >Machine Learning Model to open the Model Builder UI.
2. In the dialog, name the Model Builder projectStopSignDetection, and clickAdd.

Choose a scenario

For this sample, the scenario is object detection. In theScenario step of Model Builder, select theObject Detection scenario.

If you don't seeObject Detection in the list of scenarios, you may need toupdate your version of Model Builder.

Choose the training environment

Currently, Model Builder supports training object detection models with Azure Machine Learning only, so the Azure training environment is selected by default.

To train a model using Azure ML, you must create an Azure ML experiment from Model Builder.

AnAzure ML experiment is a resource that encapsulates the configuration and results for one or more machine learning training runs.

To create an Azure ML experiment, you first need to configure your environment in Azure. An experiment needs the following to run:

• An Azure subscription
• Aworkspace: an Azure ML resource that provides a central place for all Azure ML resources and artifacts created as part of a training run.
• Acompute: an Azure Machine Learning compute is a cloud-based Linux VM used for training. Learn more aboutcompute types supported by Model Builder.

Set up an Azure ML workspace

To configure your environment:

1. Select theSet up workspace button.

2. In theCreate new experiment dialog, select your Azure subscription.

3. Select an existing workspace or create a new Azure ML workspace.

   When you create a new workspace, the following resources are provisioned:

   • Azure Machine Learning workspace
   • Azure Storage
   • Azure Application Insights
   • Azure Container Registry
   • Azure Key Vault

   As a result, this process may take a few minutes.

4. Select an existing compute or create a new Azure ML compute. This process may take a few minutes.

5. Leave the default experiment name and selectCreate.

   

The first experiment is created, and the experiment name is registered in the workspace. Any subsequent runs (if the same experiment name is used ) are logged as part of the same experiment. Otherwise, a new experiment is created.

If you’re satisfied with your configuration, select theNext step button in Model Builder to move to theData step.

Load the data

In theData step of Model Builder, you will select your training dataset.

1. Select the button insideInput section and use the File Explorer to find theStopSignObjDetection-export.json which should be located in theStop-Signs/vott-json-export directory.

   

2. If your data looks correct in theData Preview, selectNext step to move on to theTrain step.

Train the model

The next step is to train your model.

In the Model BuilderTrain screen, select theStart training button.

At this point, your data is uploaded to Azure Storage and the training process begins in Azure ML.

The training process takes some time, and the amount of time may vary depending on the size of compute selected as well as the amount of data. The first time a model is trained in Azure, you can expect a slightly longer training time because resources have to be provisioned. For this sample of 50 images, training took about 16 minutes.

You can track the progress of your runs in the Azure Machine Learning portal by selecting theMonitor current run in Azure portal link in Visual Studio.

Once training is complete, select theNext step button to move on to theEvaluate step.

Evaluate the model

In the Evaluate screen, you get an overview of the results from the training process, including the model accuracy.

In this case, the accuracy says 100%, which means that the model is more than likelyoverfit due to too few images in the dataset.

You can use theTry your model experience to quickly check whether your model is performing as expected.

SelectBrowse an image and provide a test image, preferably one that the model did not use as part of training.

The score shown on each detected bounding box indicates theconfidence of the detected object. For instance, in the screenshot above, the score on the bounding box around the stop sign indicates that the model is 99% sure that the detected object is a stop sign.

TheScore threshold, which can be increased or decreased with the threshold slider, will add and remove detected objects based on their scores. For instance, if the threshold is .51, then the model will only show objects that have a confidence score of .51 or above. As you increase the threshold, you will see less detected objects, and as you decrease the threshold, you will see more detected objects.

If you're not satisfied with your accuracy metrics, one easy way to try to improve model accuracy is to use more data. Otherwise, select theNext step link to move on to theConsume step in Model Builder.

(Optional) Consume the model

This step will have project templates that you can use to consume the model. This step is optional and you can choose the method that best suits your needs on how to serve the model.

• Console App
• Web API

Console App

When adding a console app to your solution, you will be prompted to name the project.

1. Name the console projectStopSignDetection_Console.

2. ClickAdd to solution to add the project to your current solution.

3. Run the application.

   The output generated by the program should look similar to the snippet below:

   Predicted Boxes:Top: 73.225296, Left: 256.89764, Right: 533.8884, Bottom: 484.24243, Label: stop-sign, Score: 0.9970765

Web API

When you add a web API to your solution, you're prompted to name the project.

1. Name the Web API projectStopSignDetection_API.

2. SelectAdd to solution to add the project to your current solution.

3. Run the application.

4. Open PowerShell and enter the following code where PORT is the port your application is listening on.

   $body = @{     ImageSource = <Image location on your local machine> } Invoke-RestMethod "https://localhost:<PORT>/predict" -Method Post -Body ($body | ConvertTo-Json) -ContentType "application/json"

5. If successful, the output should look similar to the following text.

   boxes                                      labels scores       boundingBoxes-----                                      ------ ------       -------------{339.97797, 154.43184, 472.6338, 245.0796} {1}    {0.99273646} {}

   • Theboxes column gives the bounding box coordinates of the object that was detected. The values here belong to the left, top, right, and bottom coordinates respectively.
   • Thelabels are the index of the predicted labels. In this case, the value 1 is a stop sign.
   • Thescores defines how confident the model is that the bounding box belongs to that label.

Congratulations! You've successfully built a machine learning model to detect stop signs in images using Model Builder. You can find the source code for this tutorial at thesamples browser.

Additional resources

To learn more about topics mentioned in this tutorial, visit the following resources:

• Model Builder Scenarios
• Object Detection using ONNX in ML.NET