This page describes how to evaluate models using Vertex AI. Foran overview, seemodel evaluation in Vertex AI.

Prerequisites

1. Follow the steps atSet up a project and a development environment.In addition, enable the following services:

   • Compute Engine API
   • Dataflow API

2. Vertex AI can evaluate models that are trained eitherthrough AutoML or custom training. For the Google Cloud consoleguide, you should have a trained modelimported toVertex AI Model Registry.

3. Upload your test dataset toBigQuery orCloud Storage. The test dataset should contain the groundtruth, which is the actual result expected for an inference. Obtain the linkto the file or the dataset ID.

4. Have abatch inference output in the form of aBigQuery table or Cloud Storage URI.

5. Make sure yourdefault Compute Engine service account has thefollowingIAM permissions:

   • Vertex AI Administrator (aiplatform.admin)
   • Vertex AI Service Agent (aiplatform.serviceAgent)
   • Storage Object Admin (storage.objectAdmin)
   • Dataflow Worker (dataflow.worker)
   • BigQuery Data Editor (bigquery.dataEditor) (only requiredif you are providing data in the form of BigQuery tables)

Create an evaluation

Vertex AI automatically sends an email notification whena model evaluation job is complete.

View evaluation metrics

Note: ForBigQuery ML models that are registered to Model Registry, Vertex AI only shows evaluation metrics for regression and binary classification models.

Compare evaluation metrics

You can compare evaluation results across different models, model versions, andevaluation jobs. For more information about model versioning, seeVersioning inModel Registry.

You can only compare models of the same type, such as classification,regression, or forecasting. When comparing different models, all themodel versions must be the same type.

You can only compare 5 or fewer evaluations at a time.

1. Go to the Vertex AI Model Registry in the Google Cloud console:

   Go to the Models page

2. Navigate to your model or model version:

   • To compare across different models on theModels page, select thecheckboxes next to the names of the models you want to compare.

   • To compare across different model versions:

     1. Click the name of your model on theModels page to open the listof model versions.

     2. Select the checkboxes next to the versions you want to compare.

   • To compare across evaluation jobs for the same model version:

     1. Click the name of your model on theModels page to open the listof model versions.

     2. Click the version number.

     3. Select the checkboxes next to the evaluation jobs you want to compare.

3. ClickCompare.

What's next

• Learn how toiterate on your model.