This page describes how to create a High Performance Computing (HPC)Slurm cluster that usesremote direct memory access (RDMA) with H4D VMs with enhanced cluster managementcapabilities. You use the gcloud CLI andCluster Toolkitto configure the cluster.

The H4D machine series is specifically designed to meet the needs of demandingHPC workloads. H4D offers instances with improved workload scalability throughCloud RDMA networking with 200 Gbps throughput. For more information on H4Dcompute-optimized machine types on Google Cloud, seeH4D machine series.

Before you begin

Before creating a Slurm cluster, if you haven't already done so, complete the followingsteps:

1. Choose a consumption option: the option that you pick determines how you want to obtain and use vCPU resources.
2. Obtain capacity: obtain capacity for the selected consumption option.

To learn more, seeChoose a consumption option and obtain capacity.

3. Ensure that you have enough Filestore quota: you need a minimum of 10,240 GiB of zonal (also known as high scale SSD) capacity.
   • To check quota, seeView API-specific quota.
   • If you don't have enough quota,request a quota increase.
4. Install Cluster Toolkit: to provision Slurm clusters, you must useCluster Toolkit versionv1.62.0 or later.

   To install Cluster Toolkit, seeSet up Cluster Toolkit.

In the Google Cloud console, activate Cloud Shell.

Activate Cloud Shell

At the bottom of the Google Cloud console, aCloud Shell session starts and displays a command-line prompt. Cloud Shell is a shell environment with the Google Cloud CLI already installed and with values already set for your current project. It can take a few seconds for the session to initialize.

Set up a storage bucket

Cluster blueprints use Terraform modules to provision Cloud infrastructure. Abest practice when working with Terraform is tostore the state remotelyin a version enabled file. On Google Cloud, you can create a Cloud Storagebucket that has versioning enabled.

To create this bucket and enable versioning from the CLI, run the following commands:

gcloud storage buckets create gs://BUCKET_NAME \    --project=PROJECT_ID \    --default-storage-class=STANDARD --location=BUCKET_REGION \    --uniform-bucket-level-accessgcloud storage buckets update gs://BUCKET_NAME --versioning

Replace the following:

• BUCKET_NAME: a name for your Cloud Storage bucketthat meets thebucket naming requirements.
• PROJECT_ID: your project ID.
• BUCKET_REGION: anyavailable location.

Open the Cluster Toolkit directory

Ensure that you are in the Cluster Toolkit directory by running thefollowing command:

cd cluster-toolkit

This cluster deployment requires Cluster Toolkitv1.70.0 orlater. To check your version, you can run the following command:

./gcluster --version

Create a deployment file

Create a deployment file to specify the Cloud Storage bucket, set names foryour network and subnetwork, and set deployment variables such as project ID,region, and zone.

To create a deployment file, follow the steps for the H4D machine type:

The parameters that you need to add to your deployment file depend on theconsumption option that you're using for your deployment. Select the tab that corresponds to your consumption option you want to use.

terraform_backend_defaults:  type: gcs  configuration:    bucket:BUCKET_NAMEvars:  deployment_name:DEPLOYMENT_NAME  project_id:PROJECT_ID  region:REGION  zone:ZONE  h4d_cluster_size:NUMBER_OF_VMS  h4d_reservation_name:RESERVATION_NAME

terraform_backend_defaults:  type: gcs  configuration:    bucket:BUCKET_NAMEvars:  deployment_name:DEPLOYMENT_NAME  project_id:PROJECT_ID  region:REGION  zone:ZONE  h4d_cluster_size:NUMBER_OF_VMS  h4d_dws_flex_enabled: true

      node_count_static: 0      node_count_dynamic_max: $(vars.h4d_cluster_size)

terraform_backend_defaults:  type: gcs  configuration:    bucket:BUCKET_NAMEvars:  deployment_name:DEPLOYMENT_NAME  project_id:PROJECT_ID  region:REGION  zone:ZONE  h4d_cluster_size:NUMBER_OF_VMS  h4d_enable_spot_vm: true

Provision an H4D Slurm cluster

Cluster Toolkit provisions the cluster based on the deployment file youcreated in the previous step and the default cluster blueprint. For moreinformation about the software that is installed by the blueprint, seelearn more about Slurm custom images.

Using Cloud Shell, from the directory where you installed Cluster Toolkitand created the deployment file, you can provision the cluster with the following command,which uses theH4D Slurm blueprint file.This step takes approximately 20-30 minutes.

./gcluster deploy -d h4d-slurm-deployment.yaml examples/hpc-slurm-h4d/hpc-slurm-h4d.yaml --auto-approve

Redeploy the Slurm cluster

If you need to increase the number of compute nodes or add new partitions toyour cluster, you might need to update configurations for your Slurm cluster byredeploying.

To redeploy the cluster using an existing image do the following:

1. Connect to the cluster

2. Run the following command:

   ./gcluster deploy -d h4d-slurm-deployment.yaml examples/h4d/h4d-slurm-deployment.yaml --only cluster-env,cluster --auto-approve -w

   This command is only for redeployments where an image already exists; it onlyredeploys the cluster and its infrastructure.

Destroy the Slurm cluster

To remove the Slurm cluster and the instances within it, use complete thefollowing steps:

1. Disconnect from the cluster if you haven't already.

2. Before running the destroy command, navigate to the root of theCluster Toolkit directory. By default,DEPLOYMENT_FOLDERis located at the root of the Cluster Toolkit directory.

3. To destroy the cluster, run:

   ./gcluster destroyDEPLOYMENT_FOLDER --auto-approve

   Replace the following:

   • DEPLOYMENT_FOLDER: the name of the deploymentfolder. It's typically the same asDEPLOYMENT_NAME.

When the cluster removal is complete you should see a message similar to thefollowing:

  Destroy complete! Resources: xx destroyed.

To learn how to cleanly destroy infrastructure and for advanced manualdeployment instructions, see the deployment folder located at the root ofthe Cluster Toolkit directory:DEPLOYMENT_FOLDER/instructions.txt

What's next

• Verify reservation consumption
• View VMs topology
• Manage host events across VMs
• Manage host events across reservations
• Monitor VMs in your Slurm cluster
• Report faulty host