Create an instance that uses Cloud RDMA

This page discusses how to create HPC instances with Cloud RDMA-enablednetwork interfaces. For instructions for creating A4 or A3 Ultraaccelerator-optimized instances that use RDMA, seeCreate an AI-optimized instance with A4 or A3 Ultra.

To create a compute instance that uses remote direct memory access (RDMA), youmust configure at least two network interfaces (NICs) when creating theinstance. One NIC must be an IRDMA NIC that attaches to a VPCnetwork with aFalcon RDMA network profile,and the other NIC must useGVNIC.

Before you begin

Required roles

To get the permissions that you need to create an instance that supports Cloud RDMA, ask your administrator to grant you theCompute Instance Admin (v1) (roles/compute.instanceAdmin.v1) IAM role on the project. For more information about granting roles, seeManage access to projects, folders, and organizations.

This predefined role contains the permissions required to create an instance that supports Cloud RDMA. To see the exact permissions that are required, expand theRequired permissions section:

Required permissions

The following permissions are required to create an instance that supports Cloud RDMA:

  • compute.instances.create on the project
  • To use a custom image to create the VM: compute.images.useReadOnly on the image
  • To use a snapshot to create the VM: compute.snapshots.useReadOnly on the snapshot
  • To use an instance template to create the VM: compute.instanceTemplates.useReadOnly on the instance template
  • To specify a subnet for your VM: compute.subnetworks.use on the project or on the chosen subnet
  • To specify a static IP address for the VM: compute.addresses.use on the project
  • To assign an external IP address to the VM when using a VPC network: compute.subnetworks.useExternalIp on the project or on the chosen subnet
  • To set VM instance metadata for the VM: compute.instances.setMetadata on the project
  • To set tags for the VM: compute.instances.setTags on the VM
  • To set labels for the VM: compute.instances.setLabels on the VM
  • To set a service account for the VM to use: compute.instances.setServiceAccount on the VM
  • To create a new disk for the VM: compute.disks.create on the project
  • To attach an existing disk in read-only or read-write mode: compute.disks.use on the disk
  • To attach an existing disk in read-only mode: compute.disks.useReadOnly on the disk

You might also be able to get these permissions withcustom roles or otherpredefined roles.

Requirements

Review the following rules for configuring a network interface for an instance:

  • Each network interface must connect to a different Virtual Private Cloudnetwork.
  • To configure an IRDMA network interface, you must first create aVPC network with the Falcon RDMA network profile in the sameregion and zone as the instance.
  • If you don't specify a network or subnet, Compute Engine uses thedefault VPC network and the auto subnet that's in the sameregion as the instance.
  • If you specify a subnet but don't specify a network, Compute Engineinfers the network from the subnet specified.
  • If you specify a network, then you must specify a subnet and it must belongto the same network. Otherwise, instance creation fails.

Limitations

  • You can't use live migration with VMs that use Cloud RDMA. You mustconfigure the instance to terminate during maintenance events.
  • You can't use Cloud RDMA with managed instance groups (MIGs)(managed or unmanaged).
  • You can use onlyIPv4_ONLY network stack types with aCloud RDMA-enabled instance.
  • You can use only theH4D machine seriesto create an instance that uses Cloud RDMA.

Create a Cloud RDMA-enabled instance

Cloud RDMA-enabled instances require a minimum of two networkinterfaces (NICs):

  • NIC typeGVNIC: uses thegve driver for TCP/IP and Internet traffic fornormal VM-VM and VM-Internet communication
  • NIC typeIRDMA: uses IDPF/iRDMA drivers for RDMA based communicationbetween instances

An instance can have only oneIRDMA interface. Each instance can havea from one to 10 network interfaces.

To create an instance that uses IRDMA, complete the steps in the followingsections:

  1. Create a placement policy, if a suitable onedoesn't exist.

  2. Create a new VPC network, or use an existingFalcon VPC network.

  3. Create a Cloud RDMA-enabled instance, thathas anIRDMA network interface, at least oneGVNIC network interface, anduses a compact placement policy.

Create a placement policy for Cloud RDMA instances

Instances that communicate using Cloud RDMA must be located in a singlezone, and more stringently within a single cluster fabric. You can create acompact placement policyand specify a max distance value to set the minimum compactness within azone. There is a limit to the number of instances you can assign the compactplacement policy to when you specify a max distance value, and the limitchanges depending on what value you choose.

  • To create a compact placement policy to use with your Cloud RDMAinstances, follow the steps inCreate a compact placement policyand specify a maximum distance value of 3 or lower.

Create Virtual Private Cloud networks

To set up the networks, you can either follow the documented instructions oruse the provided script.

Instruction guides

To create the networks, you can use the following instructions:

Script

To create the networks, you can use the following script.

  1. Optional: Before running the script, list the Falcon RDMAnetwork profiles to verify they are available.

    gcloud compute network-profiles list --filter=falcon

  2. Copy the following code and run it in a Linux shell window.

#!/bin/bash# Create standard VPC (network and subnet) for the GVNIC interfacegcloudcomputenetworkscreateGVNIC_NAME_PREFIX-net-0\--subnet-mode=customgcloudcomputenetworkssubnetscreateGVNIC_NAME_PREFIX-sub-0\--network=GVNIC_NAME_PREFIX-net-0\--region=REGION\--range=10.0.0.0/16gcloudcomputefirewall-rulescreateGVNIC_NAME_PREFIX-internal-0\--network=GVNIC_NAME_PREFIX-net-0\--action=ALLOW\--rules=tcp:0-65535,udp:0-65535,icmp\--source-ranges=10.0.0.0/8# Create SSH firewall rulesgcloudcomputefirewall-rulescreateGVNIC_NAME_PREFIX-ssh\--network=GVNIC_NAME_PREFIX-net-0\--action=ALLOW\--rules=tcp:22\--source-ranges=IP_RANGE# Optional: Create an external IP for only the GVNIC interfacegcloudcomputefirewall-rulescreateGVNIC_NAME_PREFIX-allow-ping-net-0\--network=GVNIC_NAME_PREFIX-net-0\--action=ALLOW\--rules=icmp\--source-ranges=IP_RANGE# Create network for Cloud RDMA over Falcon transportgcloudcomputenetworkscreateRDMA_NAME_PREFIX-irdma\--network-profile=ZONE-vpc-falcon\--subnet-modecustom# Create subnet for Cloud RDMAgcloudcomputenetworkssubnetscreateRDMA_NAME_PREFIX-irdma-sub\--network=RDMA_NAME_PREFIX-irdma\--region=REGION\--range=10.1.0.0/16# offset to avoid overlap with GVNIC network

Replace the following:

  • GVNIC_NAME_PREFIX: the name prefix to use for theregular Virtual Private Cloud network and subnet that uses a GVNIC NIC type.
  • RDMA_NAME_PREFIX: the name prefix to use for theVirtual Private Cloud network and subnet that uses the IRDMA NIC type.
  • ZONE: thezonewhere you want to create the networks and compute instances. Use eitherus-central1-a oreurope-west4-b.
  • REGION: the region where you want to create thenetworks. This must correspond to the zone specified. For example, ifyour zone iseurope-west4-b, then your region iseurope-west4.
  • IP_RANGE: the range of IP addresses outside of theVPC network to use for theSSH firewall rules.As a best practice, specify the specific IP address ranges that youneed to allow access from, rather than all IPv4 or IPv6 sources.Don't use0.0.0.0/0 or::/0 as a source range because this allowstraffic from all IPv4 or IPv6 sources, including sources outside ofGoogle Cloud.

Create a Cloud RDMA-enabled instance

The following steps show how to create an instancewith the first network interface configured as a gVNIC interface, andsecond network interface configured as an IRDMA network interface:

Console

  1. In the Google Cloud console, go to theCreate an instance page.

    Go to Create an instance

    If prompted, select your project and clickContinue.

    TheCreate an instance page appears and displays theMachine configuration pane.

  2. In theMachine configuration pane, do the following:

    1. In theName field, specify a name for your instance. For moreinformation, seeResource naming convention.

    2. Optional: In theZone field, select a zone for this instance.

      Choose the zone that you used to set up the FalconVPC network.

    3. Choose theCompute-optimized machine family.

    4. In theSeries column, select theH4D machine series.

    5. In theMachine type section, select the machine type for yourinstance.

  3. In the left-side navigation menu, clickOS and Storage. In theOperating system and storage pane that appears, do the following:

    1. At the bottom of theOperating system & Storage section, clickChange.
    2. In theOperating system andVersion lists, select an OStype and version that offers Cloud RDMA support.
    3. Make sure theBoot disk type is set to Hyperdisk Balanced.
    4. ClickSelect at the bottom of the pane to save the changes.

  4. In the left-side navigation menu, clickNetworking. In theNetworking pane that appears, do the following:

    1. Go to theNetwork interfaces section. There should already beone network interface, labeleddefault.
    2. ClickAdd a network interface. In theNew network interfacesection that appears, do the following:
      1. In theNetwork interface card list, selectIRDMA.
      2. In theNetwork field, select the Falcon VPCnetwork.
      3. Optional: In theSubnetwork list, select the subnet forthe instance to use.
      4. In theIP stack type field, make sure it is set toIPV4-only.
      5. In theExternal IPv4 address list, selectNone.
      6. To confirm the network interface details, clickDone.

  5. Optional: Specify other configuration options for the instance. Formore information, seeConfiguration options during instance creation.

  6. To create and start the instance, clickCreate.

gcloud

  1. In the Google Cloud console, activate Cloud Shell.

    Activate Cloud Shell

    At the bottom of the Google Cloud console, aCloud Shell session starts and displays a command-line prompt. Cloud Shell is a shell environment with the Google Cloud CLI already installed and with values already set for your current project. It can take a few seconds for the session to initialize.

  2. Use thegcloud compute instances create commandwith at least two--network-interface flags, one for the GVNICinterface and one for the IRDMA interface. You can customize the restof the command options, as needed.

    Note: The gVNIC network interface can have only one IPv6 address, eitherinternal or external, but not both. The IRDMA network interface canonly use IPv4 addresses.
    gcloud compute instances createINSTANCE_NAME \    --zone=ZONE \    --machine-type=MACHINE_TYPE \    --create-disk=boot=yes, \        image=projects/IMAGE_PROJECT/global/images/IMAGE, \        size=SIZE \    --maintenance-policy=TERMINATE \    --network-interface \        nic-type=GVNIC, \        network=NETWORK_NAME,subnet=SUBNET_NAME, \        stack-type=STACK_TYPE, \        private-network-ip=INTERNAL_IPV4_ADDRESS, \        address=EXTERNAL_IPV4_ADDRESS \    --network-interface \        nic-type=IRDMA, \        network=RDMA_NETWORK_NAME,subnet=RDMA_SUBNET_NAME, \        stack-type=IPV4_ONLY, \        no-address \

    Replace the following:

    • INSTANCE_NAME: thenameof the compute instance
    • ZONE: the zone where the instance is created,such aseurope-west1-b. The instance's region is inferred from thezone.
    • MACHINE_TYPE: Optional: the machine type touse for the instance.
    • IMAGE_PROJECT: Optional: theimage projectthat contains the image.
    • IMAGE: Optional: specify one of thefollowing:
      • A specific version of the OS image—for example,hpc-rocky-linux-8-v20250721.
      • Animage family,which must be formatted asfamily/IMAGE_FAMILY. This createsthe instance from the most recent, non-deprecated OS image. Forexample, if you specifyfamily/hpc-rocky-linux-8, thenCompute Engine creates an instance using the latestversion of the OS image in the HPC Rocky Linux 8 image family. Formore information about using image families, seeImage families best practices.
    • SIZE: Optional: the size of the newdisk. The value must be a whole number. The default unit ofmeasurement is GiB.
    • NETWORK_NAME: Optional: name of the network

    • SUBNET_NAME: name of the subnet to usefor the network interface. To view a list of subnets in the network,use thegcloud compute networks subnets list command.

      For the GVNIC network interface, you can omit thenetwork andsubnet flags and use thedefault network instead.

    • STACK_TYPE: Optional: the stack type forthe GVNIC network interface.STACK_TYPE must be one of:IPV4_ONLY,IPV4_IPV6, orIPV6_ONLY.The default value isIPV4_ONLY.

    • INTERNAL_IPV4_ADDRESS: Optional: theinternal IPv4 address that you want the compute instance to use in thetarget subnet. Omit this flag if you don't need a specific IP address.

      To specify an internal IPv6 address, use the flag--internal-ipv6-address instead.

    • EXTERNAL_IPV4_ADDRESS: Optional: the staticexternal IPv4 address to use with the network interface. You musthave previouslyreserved an external IPv4 address.Do one of the following:

      • Specify a valid IPv4 address from the subnet.
      • Use the flagno-address instead if you don'twant the network interface to have an external IP address.
      • Specifyaddress='' if you want the interface to receivean ephemeral external IP address.

      To specify an external IPv6 address, use the flag--external-ipv6-address instead.

    • RDMA_NETWORK_NAME: The name of theVPC network that you created with a Falcon RDMAnetwork profile.

    • RDMA_SUBNET_NAME: The name of a subnet in theFalcon VPC network.

REST

To create an instance configured to use Cloud RDMA, make aPOSTrequest to theinstances.insert method.Include thenetworkInterfaces object with at least two networkconfigurations, one for the gVNIC interface and one for the IRDMA interface.You can customize the rest of the instance properties, as needed.

Note: The gVNIC network interface can have only one IPv6 address, eitherinternal or external, but not both. The IRDMA network interface can only useIPv4 addresses.

Before using any of the request data, make the following replacements:

HTTP method and URL:

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/zones/ZONE/instances

Request JSON body:

{    "machineType": "zones/ZONE/machineTypes/MACHINE_TYPE",    "name": "INSTANCE_NAME",    "disks": [      {        "initializeParams": {          "sourceImage": "projects/IMAGE_PROJECT/global/images/IMAGE"        },        "boot": true      }    ],    "networkInterfaces": [      {        "network": "NETWORK_NAME",        "subnetwork": "SUBNET_NAME",        "networkIP": "INTERNAL_IPV4_ADDRESS",        "accessConfigs": [          {            "type": "ONE_TO_ONE_NAT",            "name": "External IP",            "natIP": "EXTERNAL_IPV4_ADDRESS"          }        ],        "stackType": "IPV4_ONLY",        "nicType": "GVNIC",      },      {        "network": "RDMA_NETWORK_NAME",        "subnetwork": "RDMA_SUBNET_NAME",        "stackType": "IPV4_ONLY",        "nicType": "IRDMA",      }    ]}

To send your request, expand one of these options:

curl (Linux, macOS, or Cloud Shell)

Note: The following command assumes that you have logged in to thegcloud CLI with your user account by runninggcloud init orgcloud auth login , or by usingCloud Shell, which automatically logs you into thegcloud CLI . You can check the currently active account by runninggcloud auth list.

Save the request body in a file namedrequest.json, and execute the following command:

curl -X POST \
     -H "Authorization: Bearer $(gcloud auth print-access-token)" \
     -H "Content-Type: application/json; charset=utf-8" \
     -d @request.json \
     "https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/zones/ZONE/instances"

PowerShell (Windows)

Note: The following command assumes that you have logged in to thegcloud CLI with your user account by runninggcloud init orgcloud auth login . You can check the currently active account by runninggcloud auth list.

Save the request body in a file namedrequest.json, and execute the following command:

$cred = gcloud auth print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
    -Method POST `
    -Headers $headers `
    -ContentType: "application/json; charset=utf-8" `
    -InFile request.json `
    -Uri "https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/zones/ZONE/instances" | Select-Object -Expand Content

You should receive a JSON response similar to the following:

{  "kind": "compute#operation",  "id": "9216044482154695709",  "name": "operation-1739207409646-62dccc7d7bc58-d1350b42-64fdb7f7",  "zone": "https://www.googleapis.com/compute/v1/projects/`PROJECT_ID`/zones/`ZONE`",  "operationType": "insert",  "targetLink": "https://www.googleapis.com/compute/v1/projects/`PROJECT_ID`/zones/`ZONE`/instances/`INSTANCE_NAME`",  "targetId": "2679381553616227357",  "status": "RUNNING",  "user": "USER_ID",  "progress": 0,  "insertTime": "2025-02-10T09:10:10.551-08:00",  "startTime": "2025-02-10T09:10:10.551-08:00",  "selfLink": "https://www.googleapis.com/compute/v1/projects/`PROJECT_ID`/zones/`ZONE`/operations/operation-1565289606387-58f9f62f5989c-e582f586-6d22f38" "kind": "compute#operation"}

What's next?

Except as otherwise noted, the content of this page is licensed under theCreative Commons Attribution 4.0 License, and code samples are licensed under theApache 2.0 License. For details, see theGoogle Developers Site Policies. Java is a registered trademark of Oracle and/or its affiliates.

Last updated 2025-12-15 UTC.