Evaluate and iterate AutoML image object detection models

This page shows you how to evaluate your AutoML image object detectionmodels so that you can iterate on your model.

Vertex AI provides model evaluation metrics to help you determine theperformance of your models, such as precision and recall metrics.Vertex AI calculates evaluation metrics by using thetestset.

How you use model evaluation metrics

Model evaluation metrics provide quantitative measurements of how your modelperformed on the test set. How you interpret and use those metrics depends onyour business need and the problem your model is trained to solve. For example,you might have a lower tolerance for false positives than for false negatives orthe other way around. These kinds of questions affect which metrics you wouldfocus on.

For more information about iterating on your model to improve its performance,seeIterating on your model.

Evaluation metrics returned by Vertex AI

Vertex AI returns several different evaluation metrics such asprecision, recall, and confidence thresholds. The metrics thatVertex AI returns depend on your model's objective. For example,Vertex AI provides different evaluation metrics for an imageobject detection model compared to an image object classification model.

A schema file, downloadable from a Cloud Storage location, determines whichevaluation metrics Vertex AI provides for each objective. Thefollowing tabs provide links to the schema files and describes the evaluationmetrics for each model objective.

You can view and download schema files from the following Cloud Storagelocation:
gs://google-cloud-aiplatform/schema/modelevaluation/

  • IoU threshold: Anintersectionover union threshold value that determines which inferences to return. Amodel returns inferences that are at this value or higher. The higher thethreshold, the closer the predicted bounding box values must be to the actualbounding box values.
  • Mean average precision: also known as theaverageprecision. This value ranges from zero to one, where a higher valueindicates a higher-quality model.
  • Confidence threshold: A confidence score that determines whichinferences to return. A model returns inferences that are at this value orhigher. A higher confidence threshold increases precision but lowers recall.Vertex AI returns confidence metrics at different threshold valuesto show how the threshold affectsprecisionandrecall.
  • Recall: The fraction of inferences with this class that the modelcorrectly predicted. Also calledtrue positive rate.
  • Precision: The fraction of classification inferences produced by themodel that were correct.
  • F1 score: The harmonic mean of precision and recall. F1 is a usefulmetric if you're looking for a balance between precision and recall and there'san uneven class distribution.
  • Bounding box mean average precision: The single metric for bounding boxevaluations: themeanAveragePrecision averaged over allboundingBoxMetrics.

Getting evaluation metrics

You can get an aggregate set of evaluation metrics for your model and, for someobjectives, evaluation metrics for a particular class or label. Evaluationmetrics for a particular class or label is also known as anevaluation slice.The following content describes how to get aggregate evaluation metrics andevaluation slices by using the Google Cloud console or API.

Google Cloud console

  1. In the Google Cloud console, in the Vertex AI section, go totheModels page.

    Go to the Models page

  2. In theRegion drop-down, select the region where your model is located.

  3. From the list of models, click your model, which opens the model'sEvaluate tab.

    In theEvaluate tab, you can view your model's aggregate evaluationmetrics, such as theAverage precision andRecall.

    If the model objective has evaluation slices, the console shows a list oflabels. You can click a label to view evaluation metrics for that label,as shown in the following example:

    label selection in console

API

API requests for getting evaluation metrics is the same for each data type andobjective, but the outputs are different. The following samples show the samerequest but different responses.

Getting aggregate model evaluation metrics

The aggregate model evaluation metrics provide information about the model as awhole. To see information about a specific slice, list themodelevaluation slices.

To view aggregate model evaluation metrics, use theprojects.locations.models.evaluations.getmethod.

For the bounding box metric, Vertex AI returns an array of metricvalues at different IoU threshold values (between 0 and 1) and confidencethreshold values (between 0 and 1). For example, you can narrow in on evaluationmetrics at an IoU threshold of 0.85 and a confidence threshold of 0.8228. Byviewing these different threshold values, you can see how they affect othermetrics such as precision and recall.

Select a tab that corresponds to your language or environment:

REST

Before using any of the request data, make the following replacements:

HTTP method and URL:

GET https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations

To send your request, choose one of these options:

curl

Note: The following command assumes that you have logged in to thegcloud CLI with your user account by runninggcloud init orgcloud auth login , or by usingCloud Shell, which automatically logs you into thegcloud CLI . You can check the currently active account by runninggcloud auth list.

Execute the following command:

curl -X GET \
     -H "Authorization: Bearer $(gcloud auth print-access-token)" \
     "https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations"

PowerShell

Note: The following command assumes that you have logged in to thegcloud CLI with your user account by runninggcloud init orgcloud auth login . You can check the currently active account by runninggcloud auth list.

Execute the following command:

$cred = gcloud auth print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
    -Method GET `
    -Headers $headers `
    -Uri "https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations" | Select-Object -Expand Content

You should receive a JSON response similar to the following:

Response

{  "modelEvaluations": [    {      "name": "projects/PROJECT_NUMBER/locations/LOCATION/models/MODEL_ID/evaluations/EVALUATION_ID",      "metricsSchemaUri": "gs://google-cloud-aiplatform/schema/modelevaluation/image_object_detection_metrics_1.0.0.yaml",      "metrics": {        "evaluatedBoundingBoxCount": 190,        "boundingBoxMetrics": [          {            "iouThreshold": 0.85,            "meanAveragePrecision": 0.0818413,            "confidenceMetrics": [              {                "confidenceThreshold": 7.7341465e-06,                "recall": 0.22105263,                "precision": 0.045652173,                "f1Score": 0.075675674              },              {                "confidenceThreshold": 0.017749911,                "recall": 0.21578947,                "precision": 0.078998074,                "f1Score": 0.11565586              },              {                "confidenceThreshold": 0.022688486,                "recall": 0.21052632,                "precision": 0.08583691,                "f1Score": 0.121951215              },              ...        ],        "boundingBoxMeanAveragePrecision": 0.3128878      },      "createTime": "2020-10-09T23:35:43.325822Z",      "sliceDimensions": [        "annotationSpec"      ]    }  ]}

Java

Before trying this sample, follow theJava setup instructions in theVertex AI quickstart using client libraries. For more information, see theVertex AIJava API reference documentation.

To authenticate to Vertex AI, set up Application Default Credentials. For more information, seeSet up authentication for a local development environment.

importcom.google.cloud.aiplatform.v1.ModelEvaluation;importcom.google.cloud.aiplatform.v1.ModelEvaluationName;importcom.google.cloud.aiplatform.v1.ModelServiceClient;importcom.google.cloud.aiplatform.v1.ModelServiceSettings;importjava.io.IOException;publicclassGetModelEvaluationImageObjectDetectionSample{publicstaticvoidmain(String[]args)throwsIOException{// TODO(developer): Replace these variables before running the sample.// To obtain evaluationId run the code block below after setting modelServiceSettings.//// try (ModelServiceClient modelServiceClient = ModelServiceClient.create(modelServiceSettings))// {//   String location = "us-central1";//   ModelName modelFullId = ModelName.of(project, location, modelId);//   ListModelEvaluationsRequest modelEvaluationsrequest =//   ListModelEvaluationsRequest.newBuilder().setParent(modelFullId.toString()).build();//   for (ModelEvaluation modelEvaluation ://     modelServiceClient.listModelEvaluations(modelEvaluationsrequest).iterateAll()) {//       System.out.format("Model Evaluation Name: %s%n", modelEvaluation.getName());//   }// }Stringproject="YOUR_PROJECT_ID";StringmodelId="YOUR_MODEL_ID";StringevaluationId="YOUR_EVALUATION_ID";getModelEvaluationImageObjectDetectionSample(project,modelId,evaluationId);}staticvoidgetModelEvaluationImageObjectDetectionSample(Stringproject,StringmodelId,StringevaluationId)throwsIOException{ModelServiceSettingsmodelServiceSettings=ModelServiceSettings.newBuilder().setEndpoint("us-central1-aiplatform.googleapis.com:443").build();// Initialize client that will be used to send requests. This client only needs to be created// once, and can be reused for multiple requests. After completing all of your requests, call// the "close" method on the client to safely clean up any remaining background resources.try(ModelServiceClientmodelServiceClient=ModelServiceClient.create(modelServiceSettings)){Stringlocation="us-central1";ModelEvaluationNamemodelEvaluationName=ModelEvaluationName.of(project,location,modelId,evaluationId);ModelEvaluationmodelEvaluation=modelServiceClient.getModelEvaluation(modelEvaluationName);System.out.println("Get Model Evaluation Image Object Detection Response");System.out.format("\tName: %s\n",modelEvaluation.getName());System.out.format("\tMetrics Schema Uri: %s\n",modelEvaluation.getMetricsSchemaUri());System.out.format("\tMetrics: %s\n",modelEvaluation.getMetrics());System.out.format("\tCreate Time: %s\n",modelEvaluation.getCreateTime());System.out.format("\tSlice Dimensions: %s\n",modelEvaluation.getSliceDimensionsList());}}}

Node.js

Before trying this sample, follow theNode.js setup instructions in theVertex AI quickstart using client libraries. For more information, see theVertex AINode.js API reference documentation.

To authenticate to Vertex AI, set up Application Default Credentials. For more information, seeSet up authentication for a local development environment.

/** * TODO(developer): Uncomment these variables before running the sample * (not necessary if passing values as arguments). To obtain evaluationId, * instantiate the client and run the following the commands. */// const parentName = `projects/${project}/locations/${location}/models/${modelId}`;// const evalRequest = {//   parent: parentName// };// const [evalResponse] = await modelServiceClient.listModelEvaluations(evalRequest);// console.log(evalResponse);// const modelId = 'YOUR_MODEL_ID';// const evaluationId = 'YOUR_EVALUATION_ID';// const project = 'YOUR_PROJECT_ID';// const location = 'YOUR_PROJECT_LOCATION';// Imports the Google Cloud Model Service Client libraryconst{ModelServiceClient}=require('@google-cloud/aiplatform');// Specifies the location of the api endpointconstclientOptions={apiEndpoint:'us-central1-aiplatform.googleapis.com',};// Instantiates a clientconstmodelServiceClient=newModelServiceClient(clientOptions);asyncfunctiongetModelEvaluationImageObjectDetection(){// Configure the name resourcesconstname=`projects/${project}/locations/${location}/models/${modelId}/evaluations/${evaluationId}`;constrequest={name,};// Create get model evaluation requestconst[response]=awaitmodelServiceClient.getModelEvaluation(request);console.log('Get model evaluation image object detection response');console.log(`\tName :${response.name}`);console.log(`\tMetrics schema uri :${response.metricsSchemaUri}`);console.log(`\tCreate time :${JSON.stringify(response.createTime)}`);console.log(`\tSlice dimensions :${response.sliceDimensions}`);constmodelExplanation=response.modelExplanation;console.log('\tModel explanation');if(modelExplanation===null){console.log('\t\t{}');}else{constmeanAttributions=modelExplanation.meanAttributions;if(meanAttributions===null){console.log('\t\t\t []');}else{for(constmeanAttributionofmeanAttributions){console.log('\t\tMean attribution');console.log(`\t\t\tBaseline output value : \${meanAttribution.baselineOutputValue}`);console.log(`\t\t\tInstance output value : \${meanAttribution.instanceOutputValue}`);console.log(`\t\t\tFeature attributions : \${meanAttribution.featureAttributions}`);console.log(`\t\t\tOutput index :${meanAttribution.outputIndex}`);console.log(`\t\t\tOutput display name : \${meanAttribution.outputDisplayName}`);console.log(`\t\t\tApproximation error : \${meanAttribution.approximationError}`);}}}}getModelEvaluationImageObjectDetection();

Python

To learn how to install or update the Vertex AI SDK for Python, seeInstall the Vertex AI SDK for Python. For more information, see thePython API reference documentation.

fromgoogle.cloudimportaiplatformdefget_model_evaluation_image_object_detection_sample(project:str,model_id:str,evaluation_id:str,location:str="us-central1",api_endpoint:str="us-central1-aiplatform.googleapis.com",):"""    To obtain evaluation_id run the following commands where LOCATION    is the region where the model is stored, PROJECT is the project ID,    and MODEL_ID is the ID of your model.    model_client = aiplatform.gapic.ModelServiceClient(        client_options={            'api_endpoint':'LOCATION-aiplatform.googleapis.com'            }        )    evaluations = model_client.list_model_evaluations(parent='projects/PROJECT/locations/LOCATION/models/MODEL_ID')    print("evaluations:", evaluations)    """# The AI Platform services require regional API endpoints.client_options={"api_endpoint":api_endpoint}# Initialize client that will be used to create and send requests.# This client only needs to be created once, and can be reused for multiple requests.client=aiplatform.gapic.ModelServiceClient(client_options=client_options)name=client.model_evaluation_path(project=project,location=location,model=model_id,evaluation=evaluation_id)response=client.get_model_evaluation(name=name)print("response:",response)

Listing all evaluation slices

Theprojects.locations.models.evaluations.slices.listmethod lists all evaluation slices for your model. You must have the model'sevaluation ID, which you can get when youview the aggregated evaluationmetrics.

You can use model evaluation slices to determine how the model performed on aspecific label. Thevalue field tells you which label the metrics are for.

For the bounding box metric, Vertex AI returns an array of metricvalues at different IoU threshold values (between 0 and 1) and confidencethreshold values (between 0 and 1). For example, you can narrow in on evaluationmetrics at an IoU threshold of 0.85 and a confidence threshold of 0.8228. Byviewing these different threshold values, you can see how they affect othermetrics such as precision and recall.

REST

Before using any of the request data, make the following replacements:

  • LOCATION: Region where Model is located. For example,us-central1.
  • PROJECT: .
  • MODEL_ID: The ID of your model.
  • EVALUATION_ID: ID of the model evaluation that contains the evaluation slices to list.

HTTP method and URL:

GET https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations/EVALUATION_ID/slices

To send your request, choose one of these options:

curl

Note: The following command assumes that you have logged in to thegcloud CLI with your user account by runninggcloud init orgcloud auth login , or by usingCloud Shell, which automatically logs you into thegcloud CLI . You can check the currently active account by runninggcloud auth list.

Execute the following command:

curl -X GET \
     -H "Authorization: Bearer $(gcloud auth print-access-token)" \
     "https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations/EVALUATION_ID/slices"

PowerShell

Note: The following command assumes that you have logged in to thegcloud CLI with your user account by runninggcloud init orgcloud auth login . You can check the currently active account by runninggcloud auth list.

Execute the following command:

$cred = gcloud auth print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
    -Method GET `
    -Headers $headers `
    -Uri "https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations/EVALUATION_ID/slices" | Select-Object -Expand Content

You should receive a JSON response similar to the following:

Response

{  "modelEvaluationSlices": [    {      "name": "projects/693884908213/locations/us-central1/models/9199094820113481728/evaluations/3223746102406676480/slices/466270444805616887",      "slice": {        "dimension": "annotationSpec",        "value": "Tomato"      },      "metricsSchemaUri": "gs://google-cloud-aiplatform/schema/modelevaluation/image_object_detection_metrics_1.0.0.yaml",      "metrics": {        "evaluatedBoundingBoxCount": 98,        "boundingBoxMetrics": [          {            "iouThreshold": 0.75,            "meanAveragePrecision": 0.22579378,            "confidenceMetrics": [              {                "confidenceThreshold": 8.980461e-06,                "recall": 0.3469388,                "precision": 0.15525115,                "f1Score": 0.21451104              },              {                "confidenceThreshold": 0.10109353,                "recall": 0.33673468,                "precision": 0.35106382,                "f1Score": 0.34374997              },              {                "confidenceThreshold": 0.112416565,                "recall": 0.3265306,                "precision": 0.35555556,                "f1Score": 0.34042552              },              ...            ]          },          {            "iouThreshold": 0.8,            "meanAveragePrecision": 0.14461963,            "confidenceMetrics": [              {                "confidenceThreshold": 8.980461e-06,                "recall": 0.25510204,                "precision": 0.11415525,                "f1Score": 0.15772872              },              {                "confidenceThreshold": 0.11577999,                "recall": 0.24489796,                "precision": 0.27272728,                "f1Score": 0.25806454              },              {                "confidenceThreshold": 0.1570652,                "recall": 0.23469388,                "precision": 0.28395063,                "f1Score": 0.25698322              },              ...            ]          }        ],        "boundingBoxMeanAveragePrecision": 0.35238627      },      "createTime": "2020-10-09T23:35:43.654861Z"    },    {      "name": "projects/693884908213/locations/us-central1/models/9199094820113481728/evaluations/3223746102406676480/slices/2285744162687356072",      "slice": {        "dimension": "annotationSpec",        "value": "Cheese"      },      "metricsSchemaUri": "gs://google-cloud-aiplatform/schema/modelevaluation/image_object_detection_metrics_1.0.0.yaml",      "metrics": {        "evaluatedBoundingBoxCount": 46,        "boundingBoxMetrics": [          {            "iouThreshold": 0.85,            "meanAveragePrecision": 0.0110351965,            "confidenceMetrics": [              {                "confidenceThreshold": 1.1757337e-05,                "recall": 0.08695652,                "precision": 0.019607844,                "f1Score": 0.032              },              {                "confidenceThreshold": 0.32720456,                "recall": 0.06521739,                "precision": 0.125,                "f1Score": 0.08571429              },              {                "confidenceThreshold": 0.4676356,                "recall": 0.04347826,                "precision": 0.1,                "f1Score": 0.060606066              },              ...            ]          }        ],        "boundingBoxMeanAveragePrecision": 0.3591522      },      "createTime": "2020-10-09T23:35:43.661512Z"    }  ]}

Java

Before trying this sample, follow theJava setup instructions in theVertex AI quickstart using client libraries. For more information, see theVertex AIJava API reference documentation.

To authenticate to Vertex AI, set up Application Default Credentials. For more information, seeSet up authentication for a local development environment.

importcom.google.cloud.aiplatform.v1.ModelEvaluationName;importcom.google.cloud.aiplatform.v1.ModelEvaluationSlice;importcom.google.cloud.aiplatform.v1.ModelEvaluationSlice.Slice;importcom.google.cloud.aiplatform.v1.ModelServiceClient;importcom.google.cloud.aiplatform.v1.ModelServiceSettings;importjava.io.IOException;publicclassListModelEvaluationSliceSample{publicstaticvoidmain(String[]args)throwsIOException{// TODO(developer): Replace these variables before running the sample.// To obtain evaluationId run the code block below after setting modelServiceSettings.//// try (ModelServiceClient modelServiceClient = ModelServiceClient.create(modelServiceSettings))// {//   String location = "us-central1";//   ModelName modelFullId = ModelName.of(project, location, modelId);//   ListModelEvaluationsRequest modelEvaluationsrequest =//   ListModelEvaluationsRequest.newBuilder().setParent(modelFullId.toString()).build();//   for (ModelEvaluation modelEvaluation ://     modelServiceClient.listModelEvaluations(modelEvaluationsrequest).iterateAll()) {//       System.out.format("Model Evaluation Name: %s%n", modelEvaluation.getName());//   }// }Stringproject="YOUR_PROJECT_ID";StringmodelId="YOUR_MODEL_ID";StringevaluationId="YOUR_EVALUATION_ID";listModelEvaluationSliceSample(project,modelId,evaluationId);}staticvoidlistModelEvaluationSliceSample(Stringproject,StringmodelId,StringevaluationId)throwsIOException{ModelServiceSettingsmodelServiceSettings=ModelServiceSettings.newBuilder().setEndpoint("us-central1-aiplatform.googleapis.com:443").build();// Initialize client that will be used to send requests. This client only needs to be created// once, and can be reused for multiple requests. After completing all of your requests, call// the "close" method on the client to safely clean up any remaining background resources.try(ModelServiceClientmodelServiceClient=ModelServiceClient.create(modelServiceSettings)){Stringlocation="us-central1";ModelEvaluationNamemodelEvaluationName=ModelEvaluationName.of(project,location,modelId,evaluationId);for(ModelEvaluationSlicemodelEvaluationSlice:modelServiceClient.listModelEvaluationSlices(modelEvaluationName).iterateAll()){System.out.format("Model Evaluation Slice Name: %s\n",modelEvaluationSlice.getName());System.out.format("Metrics Schema Uri: %s\n",modelEvaluationSlice.getMetricsSchemaUri());System.out.format("Metrics: %s\n",modelEvaluationSlice.getMetrics());System.out.format("Create Time: %s\n",modelEvaluationSlice.getCreateTime());Sliceslice=modelEvaluationSlice.getSlice();System.out.format("Slice Dimensions: %s\n",slice.getDimension());System.out.format("Slice Value: %s\n\n",slice.getValue());}}}}

Node.js

Before trying this sample, follow theNode.js setup instructions in theVertex AI quickstart using client libraries. For more information, see theVertex AINode.js API reference documentation.

To authenticate to Vertex AI, set up Application Default Credentials. For more information, seeSet up authentication for a local development environment.

/** * TODO(developer): Uncomment these variables before running the sample * (not necessary if passing values as arguments). To obtain evaluationId, * instantiate the client and run the following the commands. */// const parentName = `projects/${project}/locations/${location}/models/${modelId}`;// const evalRequest = {//   parent: parentName// };// const [evalResponse] = await modelServiceClient.listModelEvaluations(evalRequest);// console.log(evalResponse);// const modelId = 'YOUR_MODEL_ID';// const evaluationId = 'YOUR_EVALUATION_ID';// const project = 'YOUR_PROJECT_ID';// const location = 'YOUR_PROJECT_LOCATION';// Imports the Google Cloud Model Service Client libraryconst{ModelServiceClient}=require('@google-cloud/aiplatform');// Specifies the location of the api endpointconstclientOptions={apiEndpoint:'us-central1-aiplatform.googleapis.com',};// Instantiates a clientconstmodelServiceClient=newModelServiceClient(clientOptions);asyncfunctionlistModelEvaluationSlices(){// Configure the parent resourcesconstparent=`projects/${project}/locations/${location}/models/${modelId}/evaluations/${evaluationId}`;constrequest={parent,};// Get and print out a list of all the evaluation slices for this resourceconst[response]=awaitmodelServiceClient.listModelEvaluationSlices(request);console.log('List model evaluation response',response);console.log(response);}listModelEvaluationSlices();

Python

To learn how to install or update the Vertex AI SDK for Python, seeInstall the Vertex AI SDK for Python. For more information, see thePython API reference documentation.

fromgoogle.cloudimportaiplatformdeflist_model_evaluation_slices_sample(project:str,model_id:str,evaluation_id:str,location:str="us-central1",api_endpoint:str="us-central1-aiplatform.googleapis.com",):"""    To obtain evaluation_id run the following commands where LOCATION    is the region where the model is stored, PROJECT is the project ID,    and MODEL_ID is the ID of your model.    model_client = aiplatform.gapic.ModelServiceClient(        client_options={            'api_endpoint':'LOCATION-aiplatform.googleapis.com'            }        )    evaluations = model_client.list_model_evaluations(parent='projects/PROJECT/locations/LOCATION/models/MODEL_ID')    print("evaluations:", evaluations)    """# The AI Platform services require regional API endpoints.client_options={"api_endpoint":api_endpoint}# Initialize client that will be used to create and send requests.# This client only needs to be created once, and can be reused for multiple requests.client=aiplatform.gapic.ModelServiceClient(client_options=client_options)parent=client.model_evaluation_path(project=project,location=location,model=model_id,evaluation=evaluation_id)response=client.list_model_evaluation_slices(parent=parent)formodel_evaluation_sliceinresponse:print("model_evaluation_slice:",model_evaluation_slice)

Getting metrics for a single slice

To view evaluation metrics for a single slice, use theprojects.locations.models.evaluations.slices.getmethod. You must have the slice ID, which is provided when youlist allslices. The following sample applies to all data types andobjectives.

REST

Before using any of the request data, make the following replacements:

  • LOCATION: Region where Model is located. For example, us-central1.
  • PROJECT: .
  • MODEL_ID: The ID of your model.
  • EVALUATION_ID: ID of the model evaluation that contains the evaluation slice to retrieve.
  • SLICE_ID: ID of an evaluation slice to get.
  • PROJECT_NUMBER: Your project's automatically generatedproject number.
  • EVALUATION_METRIC_SCHEMA_FILE_NAME: The name of a schema file that defines the evaluation metrics to return such asclassification_metrics_1.0.0.

HTTP method and URL:

GET https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations/EVALUATION_ID/slices/SLICE_ID

To send your request, choose one of these options:

curl

Note: The following command assumes that you have logged in to thegcloud CLI with your user account by runninggcloud init orgcloud auth login , or by usingCloud Shell, which automatically logs you into thegcloud CLI . You can check the currently active account by runninggcloud auth list.

Execute the following command:

curl -X GET \
     -H "Authorization: Bearer $(gcloud auth print-access-token)" \
     "https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations/EVALUATION_ID/slices/SLICE_ID"

PowerShell

Note: The following command assumes that you have logged in to thegcloud CLI with your user account by runninggcloud init orgcloud auth login . You can check the currently active account by runninggcloud auth list.

Execute the following command:

$cred = gcloud auth print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
    -Method GET `
    -Headers $headers `
    -Uri "https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations/EVALUATION_ID/slices/SLICE_ID" | Select-Object -Expand Content

You should receive a JSON response similar to the following:

Response

{  "name": "projects/PROJECT_NUMBER/locations/LOCATION/models/MODEL_ID/evaluations/EVALUATION_ID/slices/SLICE_ID",  "slice": {    "dimension": "annotationSpec",    "value": "a particular class or label"  },  "metricsSchemaUri": "gs://google-cloud-aiplatform/schema/modelevaluation/EVALUATION_METRIC_SCHEMA_FILE_NAME.yaml",  "metrics": {evaluation metrics for the slice  },  "createTime": "2020-10-08T23:35:54.770876Z"}

Java

Before trying this sample, follow theJava setup instructions in theVertex AI quickstart using client libraries. For more information, see theVertex AIJava API reference documentation.

To authenticate to Vertex AI, set up Application Default Credentials. For more information, seeSet up authentication for a local development environment.

importcom.google.cloud.aiplatform.v1.ModelEvaluationSlice;importcom.google.cloud.aiplatform.v1.ModelEvaluationSlice.Slice;importcom.google.cloud.aiplatform.v1.ModelEvaluationSliceName;importcom.google.cloud.aiplatform.v1.ModelServiceClient;importcom.google.cloud.aiplatform.v1.ModelServiceSettings;importjava.io.IOException;publicclassGetModelEvaluationSliceSample{publicstaticvoidmain(String[]args)throwsIOException{// TODO(developer): Replace these variables before running the sample.// To obtain evaluationId run the code block below after setting modelServiceSettings.//// try (ModelServiceClient modelServiceClient = ModelServiceClient.create(modelServiceSettings))// {//   String location = "us-central1";//   ModelName modelFullId = ModelName.of(project, location, modelId);//   ListModelEvaluationsRequest modelEvaluationsrequest =//   ListModelEvaluationsRequest.newBuilder().setParent(modelFullId.toString()).build();//   for (ModelEvaluation modelEvaluation ://     modelServiceClient.listModelEvaluations(modelEvaluationsrequest).iterateAll()) {//       System.out.format("Model Evaluation Name: %s%n", modelEvaluation.getName());//   }// }Stringproject="YOUR_PROJECT_ID";StringmodelId="YOUR_MODEL_ID";StringevaluationId="YOUR_EVALUATION_ID";StringsliceId="YOUR_SLICE_ID";getModelEvaluationSliceSample(project,modelId,evaluationId,sliceId);}staticvoidgetModelEvaluationSliceSample(Stringproject,StringmodelId,StringevaluationId,StringsliceId)throwsIOException{ModelServiceSettingsmodelServiceSettings=ModelServiceSettings.newBuilder().setEndpoint("us-central1-aiplatform.googleapis.com:443").build();// Initialize client that will be used to send requests. This client only needs to be created// once, and can be reused for multiple requests. After completing all of your requests, call// the "close" method on the client to safely clean up any remaining background resources.try(ModelServiceClientmodelServiceClient=ModelServiceClient.create(modelServiceSettings)){Stringlocation="us-central1";ModelEvaluationSliceNamemodelEvaluationSliceName=ModelEvaluationSliceName.of(project,location,modelId,evaluationId,sliceId);ModelEvaluationSlicemodelEvaluationSlice=modelServiceClient.getModelEvaluationSlice(modelEvaluationSliceName);System.out.println("Get Model Evaluation Slice Response");System.out.format("Model Evaluation Slice Name: %s\n",modelEvaluationSlice.getName());System.out.format("Metrics Schema Uri: %s\n",modelEvaluationSlice.getMetricsSchemaUri());System.out.format("Metrics: %s\n",modelEvaluationSlice.getMetrics());System.out.format("Create Time: %s\n",modelEvaluationSlice.getCreateTime());Sliceslice=modelEvaluationSlice.getSlice();System.out.format("Slice Dimensions: %s\n",slice.getDimension());System.out.format("Slice Value: %s\n",slice.getValue());}}}

Node.js

Before trying this sample, follow theNode.js setup instructions in theVertex AI quickstart using client libraries. For more information, see theVertex AINode.js API reference documentation.

To authenticate to Vertex AI, set up Application Default Credentials. For more information, seeSet up authentication for a local development environment.

/** * TODO(developer): Uncomment these variables before running the sample * (not necessary if passing values as arguments). To obtain evaluationId, * instantiate the client and run the following the commands. */// const parentName = `projects/${project}/locations/${location}/models/${modelId}`;// const evalRequest = {//   parent: parentName// };// const [evalResponse] = await modelServiceClient.listModelEvaluations(evalRequest);// console.log(evalResponse);// const modelId = 'YOUR_MODEL_ID';// const evaluationId = 'YOUR_EVALUATION_ID';// const sliceId = 'YOUR_SLICE_ID';// const project = 'YOUR_PROJECT_ID';// const location = 'YOUR_PROJECT_LOCATION';// Imports the Google Cloud Model Service client libraryconst{ModelServiceClient}=require('@google-cloud/aiplatform');// Specifies the location of the api endpointconstclientOptions={apiEndpoint:'us-central1-aiplatform.googleapis.com',};// Specifies the location of the api endpointconstmodelServiceClient=newModelServiceClient(clientOptions);asyncfunctiongetModelEvaluationSlice(){// Configure the parent resourceconstname=`projects/${project}/locations/${location}/models/${modelId}/evaluations/${evaluationId}/slices/${sliceId}`;constrequest={name,};// Get and print out a list of all the endpoints for this resourceconst[response]=awaitmodelServiceClient.getModelEvaluationSlice(request);console.log('Get model evaluation slice');console.log(`\tName :${response.name}`);console.log(`\tMetrics_Schema_Uri :${response.metricsSchemaUri}`);console.log(`\tMetrics :${JSON.stringify(response.metrics)}`);console.log(`\tCreate time :${JSON.stringify(response.createTime)}`);console.log('Slice');constslice=response.slice;console.log(`\tDimension :${slice.dimension}`);console.log(`\tValue :${slice.value}`);}getModelEvaluationSlice();

Python

To learn how to install or update the Vertex AI SDK for Python, seeInstall the Vertex AI SDK for Python. For more information, see thePython API reference documentation.

fromgoogle.cloudimportaiplatformdefget_model_evaluation_slice_sample(project:str,model_id:str,evaluation_id:str,slice_id:str,location:str="us-central1",api_endpoint:str="us-central1-aiplatform.googleapis.com",):"""    To obtain evaluation_id run the following commands where LOCATION    is the region where the model is stored, PROJECT is the project ID,    and MODEL_ID is the ID of your model.    model_client = aiplatform.gapic.ModelServiceClient(        client_options={            'api_endpoint':'LOCATION-aiplatform.googleapis.com'            }        )    evaluations = model_client.list_model_evaluations(parent='projects/PROJECT/locations/LOCATION/models/MODEL_ID')    print("evaluations:", evaluations)    """# The AI Platform services require regional API endpoints.client_options={"api_endpoint":api_endpoint}# Initialize client that will be used to create and send requests.# This client only needs to be created once, and can be reused for multiple requests.client=aiplatform.gapic.ModelServiceClient(client_options=client_options)name=client.model_evaluation_slice_path(project=project,location=location,model=model_id,evaluation=evaluation_id,slice=slice_id,)response=client.get_model_evaluation_slice(name=name)print("response:",response)

Iterate on your model

Model evaluation metrics provide a starting point for debugging your model whenthe model isn't meeting your expectations. For example, low precision and recallscores can indicate that your model needs additional training data or hasinconsistent labels. Perfect precision and recall can indicate that thetest data is too easy to predict and might not generalize well.

You can iterate on your training data and create a new model. After you create anew model, you can compare the evaluation metrics between the existing model andthe new model.

The following suggestions can help you improve models that label items, suchas object detection or detection models:

  • Consider adding more examples or a wider range of examples in your trainingdata. For example, for an image object detection model, you might include widerangle images, higher or lower resolution images, or different points of view.For more guidance, seePreparing data.
  • Consider removing classes or labels that don't have a lot of examples.Insufficient examples prevent the model from consistently and confidentlymaking predictions about those classes or labels.
  • Machines can't interpret the name of your classes or labels and don'tunderstand the nuances between them, such as "door" and "door_with_knob." Youmust provide data to help machines recognize such nuances.
  • Augment your data with more examples of true positives and true negatives,especially examples that are close to a decision boundary to mitigate modelconfusion.
  • Specify your own data split (training, validation, and test).Vertex AI randomly assigns items to each set. Therefore,near-duplicates can beallocated in the training and validation sets, which could lead to overfittingand then poor performance on the test set. For more information about settingyour own data split, seeAbout data splits for AutoMLmodels.
  • If your model's evaluation metrics include a confusion matrix, you can see ifthe model is confusing two labels, where the model is predicting a particularlabel significantly more than the true label. Review your data and make surethe examples are correctly labeled.
  • If you had a short training time (low maximum number of node hours), you mightget a higher-quality model by allowing it to train for a longer period of time(higher maximum number of node hours).

Except as otherwise noted, the content of this page is licensed under theCreative Commons Attribution 4.0 License, and code samples are licensed under theApache 2.0 License. For details, see theGoogle Developers Site Policies. Java is a registered trademark of Oracle and/or its affiliates.

Last updated 2025-11-24 UTC.