Vertex AI allocatesnodes to handle online and batch inferences.When youdeploy a custom-trained model or AutoML model to anEndpointresource to serve online inferences or whenyourequest batch inferences, you cancustomize the type of virtual machine that the inference service uses forthese nodes. You can optionally configure inference nodes to use GPUs.

Machine types differ in a few ways:

• Number of virtual CPUs (vCPUs) per node
• Amount of memory per node
• Pricing

By selecting a machine type with more computing resources, you can serveinferences with lower latency or handle more inference requests at the sametime.

Manage cost and availability

To help manage costs or ensure availability of VM resources,Vertex AI provides the following:

• To help ensure that you pay only for the computing resources that you need,you can use Vertex AI Inference autoscaling. For more information, seeScale inference nodes for Vertex AI Inference.

• To make sure that VM resources are available when your inference jobs needthem, you can use Compute Engine reservations. Reservations provide ahigh level of assurance in obtaining capacity for Compute Engineresources. For more information, seeUse reservations with inference.

• To reduce the cost of running your inference jobs, you can useSpot VMs. Spot VMs are virtual machine (VM)instances that are excess Compute Engine capacity.Spot VMs have significant discounts, butCompute Engine might preemptively stop or deleteSpot VMs to reclaim the capacity at any time.For more information, seeUse Spot VMs with inference.

Where to specify compute resources

Online inference

If you want to use a custom-trained model or an AutoML tabular model to serveonline inferences, you must specify a machine type when you deploy theModelresource as aDeployedModel to anEndpoint. For other types of AutoMLmodels, Vertex AI configures the machine types automatically.

Specify the machine type (and, optionally, GPU configuration) in thededicatedResources.machineSpec field of yourDeployedModel.

Learn how to deploy each model type:

• Deploy an AutoML tabular model in Google Cloud console
• Deploy a custom-trained model in Google Cloud console
• Deploy a custom-trained model using client libraries

Batch inference

If you want to get batch inferences from a custom-trained model or an AutoMLtabular model, you must specify a machine type when youcreate aBatchPredictionJob resource. Specify themachine type (and, optionally, GPU configuration) in thededicatedResources.machineSpec field of yourBatchPredictionJob.

Machine types

The following tables compare the available machine types for serving inferencesfrom custom-trained models and AutoML tabular models.

For information about TPU accelerator types, seeDeploy a model to Cloud TPU VMs.

Machine types: CPU

Machine types: GPU

Learn aboutpricing for each machinetype. Read more about the detailed specifications ofthese machine types in theCompute Engine documentation about machinetypes.

Find the ideal machine type

Online inference

To find the ideal machine type for your use case, we recommend loading your modelon multiple machine types and measuring characteristics such as the latency,cost, concurrency, and throughput.

One way to do this is to runthis notebookon multiple machine types and compare the results to find the one that worksbest for you.

Vertex AI reserves approximately 1 vCPU on each replicafor running system processes. This means that running the notebook on a singlecore machine type would be comparable to using a 2-core machine type for servinginferences.

When considering inference costs, remember that although larger machines costmore, they can lower overall cost because fewer replicas are required to servethe same workload. This is particularly evident for GPUs, which tend to costmore per hour, but can both provide lower latency and cost less overall.

Batch inference

For more information, seeChoose machine type and replica count.

Optional GPU accelerators

Some configurations, such as theA2 seriesandG2 series, have afixed number of GPUs built-in.

TheA4X (a4x-highgpu-4g)series requires a minimum replica count of 18. This machine is purchased perrack, and has a minimum of 18 VMs.

Other configurations, such as the N1 series, let you optionally add GPUs to accelerate eachinference node.

• You can only use GPUs when yourModel resource is based on aTensorFlowSavedModel, or when youuse a custom container that has beendesigned to take advantage of GPUs. You can't use GPUs for scikit-learn orXGBoost models.
• The availability of each type of GPU varies depending on which region you usefor your model. Learnwhich types of GPUs are available in whichregions.
• You can only use one type of GPU for yourDeployedModel resource orBatchPredictionJob, and there arelimitations on the number of GPUs you can add depending on which machine typeyou are using. The following table describes these limitations.

The following table shows the optional GPUs that are available for onlineinference and how many of each type of GPU you can use with eachCompute Engine machine type:

Optional GPUs incuradditional costs.

Coschedule multiple replicas on a single VM

This product or feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of theService Specific Terms. Pre-GA products and features are available "as is" and might have limited support. For more information, see thelaunch stage descriptions.

To optimize the cost of your deployment, you can deploy multiple replicas of thesame model onto a single VM equipped with multiple GPU hardware accelerators,such as thea3-highgpu-8g VM, which has eight NVIDIA H100 GPUs. Each modelreplica can be assigned to one or more GPUs.

For smaller workloads, you can also partition a single GPU into multiplesmaller instances usingNVIDIA multi-instance GPUs (MIG).This lets you assign resources at a sub-GPU level, maximizing theutilization of each accelerator. For more information on multi-instance GPUs,see theNVIDIA multi-instance GPU user guide.

Both of these capabilities are designed to provide more efficient resourceutilization and greater cost-effectiveness for your serving workloads.

Limitations

This feature is subject to the following limitations:

• All of the coscheduled model replicas must be the same model version.
• Usingdeployment resource poolsto share resources across deployments isn't supported.

Supported machine types

The following machine types are supported. Note that, for machine types thatonly have one GPU, no coscheduling is needed.

Prerequisites

Before using this feature, readDeploy a model by using the gcloud CLI or Vertex AI API.

Deploying the model replicas

The following samples demonstrate how to deploy coscheduled model replicas.

gcloudaiendpointsdeploy-modelENDPOINT_ID\--region=LOCATION_ID\--model=MODEL_ID\--display-name=DEPLOYED_MODEL_NAME\--min-replica-count=MIN_REPLICA_COUNT\--max-replica-count=MAX_REPLICA_COUNT\--machine-type=MACHINE_TYPE\--accelerator=type=ACC_TYPE,count=ACC_COUNT\--traffic-split=0=100

POST https://LOCATION_ID-aiplatform.googleapis.com/v1beta1/projects/PROJECT_ID/locations/LOCATION_ID/endpoints/ENDPOINT_ID:deployModel

{  "deployedModel": {    "model": "projects/PROJECT_NUMBER/locations/LOCATION_ID/models/MODEL_ID",    "displayName": "DEPLOYED_MODEL_NAME",    "dedicatedResources": {      "machineSpec": {        "machineType": "MACHINE_TYPE",        "acceleratorType": "ACC_TYPE",        "gpuPartitionSize": "GPU_PARTITION_SIZE",        "acceleratorCount": "ACC_COUNT""      },      "minReplicaCount":MIN_REPLICA_COUNT,      "maxReplicaCount":MAX_REPLICA_COUNT,      "autoscalingMetricSpecs": [        {          "metricName": "aiplatform.googleapis.com/prediction/online/accelerator/duty_cycle",          "target": 70        }      ]    }  }}

curl -X POST \
     -H "Content-Type: application/json; charset=utf-8" \
     -d @request.json \
     "https://LOCATION_ID-aiplatform.googleapis.com/v1beta1/projects/PROJECT_ID/locations/LOCATION_ID/endpoints/ENDPOINT_ID:deployModel"

$headers = @{  }

Invoke-WebRequest `
    -Method POST `
    -Headers $headers `
    -ContentType: "application/json; charset=utf-8" `
    -InFile request.json `
    -Uri "https://LOCATION_ID-aiplatform.googleapis.com/v1beta1/projects/PROJECT_ID/locations/LOCATION_ID/endpoints/ENDPOINT_ID:deployModel" | Select-Object -Expand Content

endpoint.deploy(model=<var>MODEL</var>,machine_type=MACHINE_TYPE,min_replica_count=MIN_REPLICA_COUNT,max_replica_count=MAX_REPLICA_COUNT,accelerator_type=ACC_TYPE,gpu_partition_size=GPU_PARTITION_SIZE,accelerator_count=ACC_COUNT)

Monitor VM usage

Use the following instructions to monitor the actual machine count for yourdeployed replicas in the Metrics Explorer.

1. In the Google Cloud console, go to theMetrics Explorer page.

   Goto Metrics Explorer

2. Select the project you want to view metrics for.

3. From theMetric drop-down menu, clickSelect a metric.

4. In theFilter by resource or metric name search bar, enterVertex AI Endpoint.

5. Select theVertex AI Endpoint > Prediction metric category. UnderActive metrics, selectMachine count.

6. ClickApply.

Billing

Billing is based on the number of VMs that are used, not the number of GPUs.You canmonitor your VM usage by using Metrics Explorer.

High availability

Because more than one replica is being coscheduled on the same VM,Vertex AI Inference cannot spread your deployment across multiple VMs andtherefore multiple zones until your replica count exceeds the single VM node.For high availability purposes, Google recommends deploying on at least twonodes (VMs).

What's next

• Deploy an AutoML tabular model in Google Cloud console
• Deploy a custom-trained model in Google Cloud console
• Deploy a custom-trained model using client libraries
• Get batch inferences