This document explains how to create a virtual machine (VM) instance that usesa machine type from theA2 orA3 Highaccelerator-optimized machine series.

For A3 High machine types, this document only covers machine types that havefewer than 8 GPUs attached. These A3 High machine types with fewer than 8 GPUscan only be created as Spot VMs or Flex-start VMs. To createan A3 instance that has 8 GPUs attached, seeCreate an A3 Mega, A3 High, or A3 Edge instance with GPUDirect enabled.

To create multiple A3 or A2 VMs, you can also use one of the following options:

• Managed instance groups (MIGs): for workloads that require highavailability, scalability, and automated repairs, you cancreate a MIG that uses a GPU instance template.
• Bulk instance creation: to create a large number of independent instances,you cancreate A3 or A2 VMs in bulk.

Before you begin

• To review limitations and additional prerequisite steps for creatinginstances with attached GPUs, such as selecting an OS image and checkingGPU quota, seeOverview of creating an instance with attached GPUs.
• If you haven't already, set upauthentication. Authentication verifies your identity for access to Google Cloud services and APIs. To run code or samples from a local development environment, you can authenticate to Compute Engine by selecting one of the following options:

  Select the tab for how you plan to use the samples on this page:

  Console

  When you use the Google Cloud console to access Google Cloud services and APIs, you don't need to set up authentication.

  gcloud

  1. Install the Google Cloud CLI. After installation,initialize the Google Cloud CLI by running the following command:

     gcloudinit

     If you're using an external identity provider (IdP), you must first sign in to the gcloud CLI with your federated identity.

     Note: If you installed the gcloud CLI previously, make sure you have the latest version by runninggcloud components update.
  2. Set a default region and zone.

  REST

  To use the REST API samples on this page in a local development environment, you use the credentials you provide to the gcloud CLI.

  Install the Google Cloud CLI. After installation,initialize the Google Cloud CLI by running the following command:

  gcloudinit

  If you're using an external identity provider (IdP), you must first sign in to the gcloud CLI with your federated identity.

  Note: If you installed the gcloud CLI previously, make sure you have the latest version by runninggcloud components update.

  For more information, seeAuthenticate for using REST in the Google Cloud authentication documentation.

Required roles

To get the permissions that you need to create VMs, ask your administrator to grant you theCompute Instance Admin (v1) (roles/compute.instanceAdmin.v1) IAM role on the project. For more information about granting roles, seeManage access to projects, folders, and organizations.

This predefined role contains the permissions required to create VMs. To see the exact permissions that are required, expand theRequired permissions section:

You might also be able to get these permissions withcustom roles or otherpredefined roles.

Create a VM that has attached GPUs

You can create an A2 or A3 accelerator-optimized VM by using theGoogle Cloud console, Google Cloud CLI, or REST.

  gcloud compute instances createVM_NAME \      --machine-type=MACHINE_TYPE \      --zone=ZONE \      --boot-disk-size=DISK_SIZE \      --image=IMAGE \      --image-project=IMAGE_PROJECT \      --maintenance-policy=TERMINATE \      --provisioning-model=PROVISIONING_MODEL

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/zones/ZONE/instances{"machineType": "projects/PROJECT_ID/zones/ZONE/machineTypes/MACHINE_TYPE","disks":[  {    "type": "PERSISTENT",    "initializeParams":    {      "diskSizeGb": "DISK_SIZE",      "sourceImage": "SOURCE_IMAGE_URI"    },    "boot": true  }],"name": "VM_NAME","networkInterfaces":[  {    "network": "projects/PROJECT_ID/global/networks/NETWORK"  }],"scheduling":{  "onHostMaintenance": "terminate",  "automaticRestart": true}}

Install drivers

For the VM to use the GPU, you need toInstall the GPU driver on your VM.

Examples

In these examples, most of the VMs are created by using the Google Cloud CLI.However, you can also use either theGoogle Cloud console orREST to create these VMs.

The following examples show how to createan A3 Spot VM by using a standard OS image, and an A2 VM by usingaDeep Learning VM Images image.

Create an A3 Spot VM by using the Debian 13 OS image family

This example creates an A3 (a3-highgpu-1g) Spot VMby using the Debian 13 OS image family.

gcloud compute instances createVM_NAME \    --project=PROJECT_ID \    --zone=ZONE \    --machine-type=a3-highgpu-1g \    --provisioning-model=SPOT \    --maintenance-policy=TERMINATE \    --image-family=debian-13 \    --image-project=debian-cloud \    --boot-disk-size=200GB \    --scopes=https://www.googleapis.com/auth/cloud-platform

Replace the following:

• VM_NAME: the name of your VM instance
• PROJECT_ID : your project ID
• ZONE: the zone for the VM instance

Create an A2 VM with a Vertex AI Workbench user-managed notebooks instance on the VM

This example creates an A2 Standard (a2-highgpu-1g) VM by using thetf2-ent-2-3-cu110Deep Learning VM Images image. In thisexample, optional flags such as boot disk size and scope are specified.

Using DLVM images is the easiest way to get started because theseimages already have the NVIDIA drivers and CUDA libraries pre-installed.

These images also provide performance optimizations.

The following DLVM images are supported for NVIDIA A100:

• common-cu110: NVIDIA driver and CUDA pre-installed
• tf-ent-1-15-cu110: NVIDIA driver, CUDA, TensorFlow Enterprise 1.15.3pre-installed
• tf2-ent-2-1-cu110: NVIDIA driver, CUDA, TensorFlow Enterprise 2.1.1pre-installed
• tf2-ent-2-3-cu110: NVIDIA driver, CUDA, TensorFlow Enterprise 2.3.1pre-installed
• pytorch-1-6-cu110: NVIDIA driver, CUDA, Pytorch 1.6

For more information about the DLVM images that are available,and the packages installed on the images, see theDeep Learning VM documentation.

gcloud compute instances createVM_NAME \    --project=PROJECT_ID \    --zone=ZONE \    --machine-type=a2-highgpu-1g \    --maintenance-policy=TERMINATE \    --image-family=tf2-ent-2-3-cu110 \    --image-project=deeplearning-platform-release \    --boot-disk-size=200GB \    --metadata="install-nvidia-driver=True,proxy-mode=project_editors" \    --scopes=https://www.googleapis.com/auth/cloud-platform

Replace the following:

• VM_NAME: the name of your VM instance
• PROJECT_ID : your project ID
• ZONE: the zone for the VM instance

The preceding example command also generates aVertex AI Workbench user-managed notebooksinstancefor the VM. To access the notebook, in the Google Cloud console, go to theVertex AI Workbench >User-managed notebookspage.

Go to the User-managed notebooks page

Multi-Instance GPU

AMulti-Instance GPUpartitions a single NVIDIA A100 or NVIDIA H100 GPU within the same VM into as many asseven independent GPU instances. They run simultaneously, each with its own memory,cache and streaming multiprocessors. This setup enables the NVIDIA A100 and H100GPU to deliver consistent quality-of-service (QoS) at up to 7x higherutilization compared to earlier GPU models.

You can create up to seven Multi-instance GPUs. For A100 40GB GPUs, eachMulti-instance GPU is allocated 5 GB of memory. With the A100 80GBGPUs the allocated memory doubles to 10 GB each. With H100 80GB GPUs, eachMulti-instance GPU is also allocated 10 GB of memory.

For more information about using Multi-Instance GPUs, seeNVIDIA Multi-Instance GPU User Guide.

To create Multi-Instance GPUs, complete the following steps:

1. Create an A2 (A100) or A3 (H100) accelerator-optimized VM instance.

2. Connect to the VM instance. For more information, seeConnect to Linux VMs orConnect to Windows VMs.

3. EnableNVIDIA GPU drivers.

   Pro Tip: You can skip thisstep by creating VMs withDeep Learning VM Images. EachDeep Learning VM Images has an NVIDIA GPU driver pre-installed.

4. Enable Multi-Instance GPUs.

   sudo nvidia-smi -mig 1

5. Review the Multi-Instance GPU shapes that are available.

   sudo nvidia-smi mig --list-gpu-instance-profiles

   The output is similar to the following:

   +-----------------------------------------------------------------------------+| GPU instance profiles:                                                      || GPU   Name             ID    Instances   Memory     P2P    SM    DEC   ENC  ||                              Free/Total   GiB              CE    JPEG  OFA  ||=============================================================================||   0  MIG 1g.10gb       19     7/7        9.62       No     16     1     0   ||                                                             1     1     0   |+-----------------------------------------------------------------------------+|   0  MIG 1g.10gb+me    20     1/1        9.62       No     16     1     0   ||                                                             1     1     1   |+-----------------------------------------------------------------------------+|   0  MIG 1g.20gb       15     4/4        19.50      No     26     1     0   ||                                                             1     1     0   |+-----------------------------------------------------------------------------+|   0  MIG 2g.20gb       14     3/3        19.50      No     32     2     0   ||                                                             2     2     0   |+-----------------------------------------------------------------------------+|   0  MIG 3g.40gb        9     2/2        39.25      No     60     3     0   ||                                                             3     3     0   |+-----------------------------------------------------------------------------+.......

6. Create the Multi-Instance GPU (GI) and associated compute instances (CI)that you want. You can create these instances by specifying either the fullor shortened profile name, profile ID, or a combination of both. For moreinformation, seeCreating GPU Instances.

   The following example creates twoMIG 3g.20gb GPU instances by using theprofile ID (9).

   The-C flag is also specified which creates the associated computeinstances for the required profile.

   sudo nvidia-smi mig -cgi 9,9 -C

7. Check that the two Multi-Instance GPUs are created:

   sudo nvidia-smi mig -lgi

8. Check that both the GIs and corresponding CIs are created.

   sudo nvidia-smi

   The output is similar to the following:

   +-----------------------------------------------------------------------------+| NVIDIA-SMI 525.125.06   Driver Version: 525.125.06   CUDA Version: 12.0     ||-------------------------------+----------------------+----------------------+| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC || Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. ||                               |                      |               MIG M. ||===============================+======================+======================||   0  NVIDIA H100 80G...  Off  | 00000000:04:00.0 Off |                   On || N/A   33C    P0    70W / 700W |     39MiB / 81559MiB |     N/A      Default ||                               |                      |              Enabled |+-------------------------------+----------------------+----------------------+|   1  NVIDIA H100 80G...  Off  | 00000000:05:00.0 Off |                   On || N/A   32C    P0    69W / 700W |     39MiB / 81559MiB |     N/A      Default ||                               |                      |              Enabled |+-------------------------------+----------------------+----------------------+......+-----------------------------------------------------------------------------+| MIG devices:                                                                |+------------------+----------------------+-----------+-----------------------+| GPU  GI  CI  MIG |         Memory-Usage |        Vol|         Shared        ||      ID  ID  Dev |           BAR1-Usage | SM     Unc| CE  ENC  DEC  OFA  JPG||                  |                      |        ECC|                       ||==================+======================+===========+=======================||  0    1   0   0  |     19MiB / 40192MiB | 60      0 |  3   0    3    0    3 ||                  |      0MiB / 65535MiB |           |                       |+------------------+----------------------+-----------+-----------------------+|  0    2   0   1  |     19MiB / 40192MiB | 60      0 |  3   0    3    0    3 ||                  |      0MiB / 65535MiB |           |                       |+------------------+----------------------+-----------+-----------------------+......+-----------------------------------------------------------------------------+| Processes:                                                                  ||  GPU   GI   CI        PID   Type   Process name                  GPU Memory ||        ID   ID                                                   Usage      ||=============================================================================||  No running processes found                                                 |+-----------------------------------------------------------------------------+

What's next?

• Learn more aboutGPU platforms.
• For A2 VMs,add a Local SSD to your VM.Local SSD devices pair well with GPUs when your apps requirehigh-performance storage. A3 VMs have Local SSD attached by default.
• Install the GPU drivers.
• To handle GPU host maintenance, seeHandling GPU host maintenance events.