Tightly coupled high performance computing (HPC) workloads often use the MessagePassing Interface (MPI) to communicate between processes and virtual machine(VM) instances. But building your own VM image that is tuned for optimal MPIperformance requires systems expertise, Google Cloud knowledge, and extra timefor maintenance. To quickly set up an optimized environment for your HPCworkloads, use theHPC VM image.

The HPC VM image is a Rocky Linux 8 based VM image that isoptimized for tightly coupled HPC workloads. It includes pre-configured kerneland network tuning parameters required to create VM instances that achieveoptimal MPI performance on Google Cloud.

For best results, deploy the HPC VM image on a machine type from thecompute-optimized machine series, such as the H4D, H3, C2, or C2D machineseries. These machine types are designed for tightly-coupled HPC applications.To use Cloud RDMA with H4D, you must use version 20250917 or later ofthe HPC VM image.

You can create an HPC-ready VM instance by using the following options:

• Google Cloud CLI
• Google Cloud console. In the console, the image is available throughCloud Marketplace.
• SchedMD's Slurm workload manager,which uses the HPC VM image by default.

Before you begin

• If you haven't already, set upauthentication. Authentication verifies your identity for access to Google Cloud services and APIs. To run code or samples from a local development environment, you can authenticate to Compute Engine by selecting one of the following options:

  Select the tab for how you plan to use the samples on this page:

  Console

  When you use the Google Cloud console to access Google Cloud services and APIs, you don't need to set up authentication.

  gcloud

  1. Install the Google Cloud CLI. After installation,initialize the Google Cloud CLI by running the following command:

     gcloudinit

     If you're using an external identity provider (IdP), you must first sign in to the gcloud CLI with your federated identity.

     Note: If you installed the gcloud CLI previously, make sure you have the latest version by runninggcloud components update.
  2. Set a default region and zone.

Benefits

The HPC VM image provides the following benefits:

1. VM instances ready for HPC workloads out-of-the-box. There is no needto manually tune performance, manage VM instance reboots, or stay up to datewith the latest Google Cloud updates for tightly coupled HPCworkloads.
2. Networking optimizations for tightly-coupled workloads. Optimizationsthat reduce latency for small messages are included, which benefitsapplications that are heavily dependent on point-to-point and collectivecommunications. If using the H4D machine series, the HPC VM image containsthe necessary Cloud RDMA drivers.
3. Compute optimizations for HPC workloads. Optimizations that reduce systemjitter are included, which makes single-node high performance morepredictable.
4. Consistent, reproducible performance. VM image standardization gives youconsistent, reproducible application-level performance.
5. Improved application compatibility. Alignment with the node-levelrequirements of theIntel HPC platform specificationenables a high degree of interoperability between systems.

HPC VM image features

The HPC VM image offers several features that are designed to optimizeperformance for high performance computing (HPC) workloads:

• Automatic updates disabled
• Intel MPI collective tunings
• Pre-installed RPMs

Automatic updates disabled

Automatic updates can have a negative impact on the performance of HPCworkloads. Automatic updates can be disabled when using the HPC VM images bysetting thegoogle_disable_automatic_updates metadata entry toTRUE whencreating a VM instance. How this metadata entry is set during instance creationdepends on the tool that you use to create the instance.

For example, when using thegcloud compute instances createcommand to create a VM instance, provide the--metadataargument. For more information, seeAbout VM metadata.

If you disable automatic updates on your H4D instances, then you shouldregularly run thednf update command on the instance to keep theCloud RDMA driver up to date.

Intel MPI Library

Google recommends that you use the Intel MPI 2021 library for running MPI jobson Google Cloud. For more information, see the following sections:

• Use Intel MPI 2021
• Intel MPI Library 2018 and MPI collective tunings

Pre-installed RPMs

The HPC VM image comes with the following RPM packages pre-installed:

• daos-client
• gcc-gfortran
• gcc-toolset-12
• Lmod
• dkms
• htop
• hwloc
• hwloc-devel
• infiniband-diags
• kernel-devel
• kmod-idpf-irdma
• libfabric
• librdmacm-utils
• libibverbs-utils
• libXt
• ltrace
• nfs-utils
• numactl
• numactl-devel
• papi
• pciutils
• pdsh
• perf
• perftest
• rdma-core
• redhat-lsb-core
• redhat-lsb-cxx
• rsh
• screen
• strace
• wget
• zsh
• "Development Tools" package group

Quickstarts

These tutorials describe the steps to set up an HPC-optimized VM instance. Theprocedures describe how to:

• Create an HPC VM instance (without configuring Cloud RDMA)
• Specify a compact placement policy when creating HPC VM instances
• Create an HPC VM instance that uses Cloud RDMA

You can't add Cloud RDMA to an existing HPC VM instance, so follow theappropriate steps when creating your instance.

Before you begin

1. To use the Google Cloud CLI for this quickstart, you must first install andinitialize theGoogle Cloud CLI:
   • Download and install the Google Cloud CLI.
   • Initialize the Google Cloud CLI.
2. In the Google Cloud console, on the project selector page, select or create aGoogle Cloud project.

Create an HPC VM instance

We strongly recommend choosing a machine type from the following machineseries:

• H4D
• H3
• C2D
• C2

These VMs have fixed virtual-to-physical core mapping and expose NUMA cellarchitecture to guest OS, both of which are critical for performance of tightlycoupled HPC applications.

gcloud compute instances createINSTANCE_NAME \        --zone=ZONE \        --image-family=IMAGE_FAMILY \        --image-project=cloud-hpc-image-public \        --maintenance-policy=TERMINATE \        --machine-type=MACHINE_TYPE

gcloud compute instances describeINSTANCE_NAME

Create HPC VM instances with compact placement policies

You can reduce the latency between VM instances by creating acompact placement policy.A compact placement policy ensures that instances in the same availabilityzone are located close to each other.

If you need more VM instances than can fit in a single compact placementpolicy, then divide your instances into multiple placement policies. Use theminimum number of placement policies that can fit all your instances.

To create HPC VM instances that specify a compact placement policy, followthese steps:

1. Create a compact placement policy.

2. Do one of the following:

   • Apply the compact placement policy to an existing HPC VM instance.

   • Create an instance that specifies the compact placement policy.

     To create an instance that specifies a compact placement policy, usethe--maintenance-policy and--resource-policies flags withthegcloud compute instances create command.

Create a HPC VM instance that uses Cloud RDMA

To create a HPC VM instance that uses Cloud RDMA, you must firstcreate at least one regular VPC network and oneFalcon VPC network. TheFalcon VPC network uses an RDMA network profile that enablesRDMA traffic between compute instances. This network is separate from theregular VPC network that carries non-RDMA traffic to otherGoogle Cloud services or the internet.

To create a HPC VM instance that uses Cloud RDMA, the tasks tocomplete are as follows:

1. Identify or create at least two VPC networks:

   • A regular VPC network for the traffic that goes through thegVNIC network interface
   • A Falcon VPC network for the RDMA traffic

2. Create a HPC VM instance.

   1. For the source image, use the HPC VM image. This image includes thedrivers needed for Cloud RDMA.

   2. During the instance creation, configure at least two networkinterfaces—one that uses the gVNIC driver and one that uses theIRDMA driver.

      For a more complete explanation, seeCreate an instance that uses Cloud RDMA.

3. If you plan to run MPI applications on the HPC VM instances that useCloud RDMA, then following the MPI configuration steps inSet up and scale MPI applications on H4D VMs with Cloud RDMA.

Access the HPC VM instance

After you create the HPC VM instance, it starts automatically. To accessthe instance, do one of the following:

gcloud compute sshINSTANCE_NAME

Clean up

To avoid incurring charges to your Google Cloud account for the resourcesused in these quickstarts, delete any HPC VM instances that you created.

gcloud compute instances deleteINSTANCE_NAME

Configure your HPC VM instance according to best practices

To get better and more predictable performance for your HPC VM instance, werecommend that you use the following best practices.

Disable simultaneous multithreading

The HPC VM image enables simultaneous multithreading (SMT), also known asHyper-Threading on Intel processors,by default. Disabling SMT can make your performance more predictable and candecrease job times.

• To disable SMT while creating a new HPC VM, follow the steps tocreate an HPC VM and include the flag--threads-per-core=1.

• To disable SMT on an existing HPC VM,connect to the VMand run the following command from the VM:

  sudo google_mpi_tuning --nosmt

For more information, seeSet the number of threads per core.

Configure gVNIC as the network interface type for C2 and C2D instances

The HPC VM image supports both Virtio-net and Google Virtual NIC (gVNIC) asvirtual network interfaces. Using gVNIC instead of Virtio-net can improve thescalability of MPI applications by providing better communicationperformance and higher throughput. Additionally, gVNIC is a prerequisite forper VM Tier_1 networking performance,which provides higher bandwidth and allows for higher throughput.

If you create a new C2 or C2D instance, then, by default, Virtio-net is used forthe virtual network interface. To use gVNIC, follow the steps tocreate an HPC VM and do one of the following:

The HPC VM image includes the gVNIC driver as aDynamic Kernel Module Support (DKMS).For more information, seeUsing Google Virtual NIC.

Turn off Meltdown and Spectre mitigations

The HPC VM image enables the Meltdown and Spectre mitigations by default.In some cases, these mitigations might result in workload-specificperformance degradation. To disable these mitigations and incurthe associated security risks, do the following:

1. Run the following command on your HPC instance:

   sudo google_mpi_tuning --nomitigation

2. Reboot the instance.

Improve network performance

To improve the network performance of your instance, set up one or more of thefollowing configurations:

• Configure a higher bandwidth. To configure Tier_1 networking for C2or C2D VMs, use thegcloud compute instances create commandto create the VM and specify the--network-performance-configs flag.For more information, seeCreating a VM with high-bandwidth configuration.

  Note: Don't configure Tier_1 networking for H4D or H3 instances.

• Use jumbo frames. To help minimize the processing overhead for networkpackets, we recommend using a larger packet size. You need to validate largerpacket sizes for the specifics of your application. For information about theuse of jumbo frames and packet sizes, seeMaximum transmission unit guide.

• Increase the TCP memory limits. Higher bandwidth requires larger TCPmemory. Follow the steps toincreasetcp_*mem settings.

• Use the network-latency profile. Evaluate your application's latencyand enable busy polling that reduces latency in the network receivepath. Adjust thenet.core.busy_poll andnet.core.busy_read settingsin/etc/sysctl.conf, orusetuned-adm.

Use Intel MPI 2021

Google recommends to use the Intel MPI 2021 library for running MPI jobs onGoogle Cloud.

MPI implementations have many internal configuration parameters that can affectcommunication performance. These parameters are especially relevant for MPICollective communication, which lets you specify algorithms and configurationparameters that can perform very differently in the Google Cloud environment.

The HPC VM image includes a utility,google-hpc-compute, thatinstalls the recommended MPI libraries and uses Google Cloud tailored libfabricproviders over the TCP transport.

Limitations

The benefits of tuning vary from application to application. In some cases, aparticular tuning might have a negative effect on performance. Considerbenchmarking your applications to find the most efficient or cost-effectiveconfiguration.

Usegoogle-hpc-compute utility for Intel MPI 2021 support

Thegoogle_install_intelmpi script is the MPI related toolin theGoogle-hpc-compute utility. It helps to install and configure IntelMPI.

Thegoogle-hpc-compute utility is included in the HPC VM image.

Install Intel MPI 2021

To install the Intel MPI library while creating a new HPC VM, follow the stepstocreate an HPC VM and include the following when creating the VMinstance:

--metadata=google_install_intelmpi="--impi_2021"

To install the library on an existing HPC VM, run the following command on thatVM:

sudo google_install_intelmpi --impi_2021 --install_dir=PATH_INSTALL_MPI

The default location forinstall_dir is set to/opt/intel.

Intel MPI Library 2018 and MPI collective tunings

The HPC VM image includesIntel MPI collective tunings performed onc2-standard-60 andc2d-standard-112 instances usingcompact placement policies.These tunings are available in the directory/usr/share/google-hpc-compute/mpitune-configs/intelmpi-2018.

To apply these tuning configurations on an installed MPI library, sourcethempivars shell script to set up the proper environment. You can theninstall the tunings directly with the following command (use the--sudooption if you need root access to the directory):google_install_mpitune

You can usempitune to manually specify the algorithms and configurationparameters for MPI Collective communication and generate configuration files.

For example, to tune for 22 VM instances and 30 vCPUs per instance, source thempivars shell script to set up the proper environment, and then run thefollowing command. You must havewrite access to the directory or run thecommand as the root user.

mpitune-hfhostfile-fl'shm:tcp'-pr30:30-hr22:22

This command generates a configuration file in the Intel MPI directory whichcan be used later to run applications. To make use of the tuning configurationfor an application, add the-tune option tompirun command line, for example:

mpirun-tune-hostfileHOSTFILE-genvI_MPI_FABRICS'shm:tcp'-np660-ppn30./APPLICATION_NAME

Replace the following:

• HOSTFILE: The name of a file in the guest OS that lists thenetwork names of the VM instances
• APPLICATION_NAME: The name of the application file to run

Create a custom image using the HPC VM image

To implement the best practices at scale, create a custom OS image to use withyour HPC VM instances. Complete the following tasks to create a custom image touse when creating HPC VM instances:

1. Create a VM that uses the HPC VM image.

2. Customize the VM with MPI tunings.

3. Create a custom image using the boot disk of your HPC VM image as thesource disk. You can do so using theGoogle Cloud consoleor theGoogle Cloud CLI.

gcloud compute images createIMAGE_NAME \         --source-disk=INSTANCE_NAME \         --source-disk-zone=VM_ZONE \         --family=IMAGE_FAMILY \         --storage-location=LOCATION

Pricing

The HPC VM image is available at no additional cost. Because the HPC VM imageruns on Compute Engine, you might incur charges forCompute Engine resources such as vCPUs, disks, and memory.To learn more, seeCompute Engine pricing.

What's next

• ReviewBest practices for running HPC workloads.
• Learn how toSet up and scale MPI applications on H4D VMs with Cloud RDMA onGoogle Cloud.
• Learn more about theCompute-optimized machine family.
• Learn how tocreate H4D HPC clusters with enhanced cluster management capabilities.
• If you have feedback or require support, emailhpc-image-feedback@google.com.