Filestore

Key uses of Filestore with training clusters

Beyond its role as the mandatory home directory, Filestore provides a flexibleway to share data with your cluster.
Additional shared storage: You can attach one or more additional Filestore instances to any node pool. This is useful for providing shared datasets, application binaries, or other common files to your training jobs. When specified in the node pool configuration, training clusters automatically mounts these instances to the/mnt/filestore directory on each node.

Filestore Requirements

A successful Filestore integration with training clusters requires the following configuration:

• Enable the API: The Filestore API must be enabled in your Google Cloud project before you can create the cluster.
• Mandatory/home Directory: Every training cluster requires a dedicated Filestore instance to serve as the shared/home directory. This instance has specific configuration requirements:
  • Network: It must reside in the same VPC network as the cluster's compute and login nodes.
  • Location: It must be located in the same region or zone as the cluster.
  • Configuration: You must specify the full resource name of this instance in theorchestrator_spec.slurm_spec.home_directory_storage field when creating the cluster via the API.

Configure Filestore storage

Create a zonal or regional Filestore instance in the zone where you want tocreate the cluster. Vertex AI API requires aFilestore to be attached to the cluster to serve as the/home directory.This Filestore has to be in the same zone or region and in the same networkas all the compute nodes and login nodes. In the example below, 172.16.10.0/24 is used for theFilestore deployment.

SERVICE_TIER=ZONAL# Can use BASIC_SSD# Create reserved IP address rangegcloudcomputeaddressescreateCLUSTER_IDfs-ip-range \--project=PROJECT_ID \--global \--purpose=VPC_PEERING \--addresses=172.16.10.0 \--prefix-length=24 \--description="Filestore instance reserved IP range" \--network=NETWORK# Get the CIDR rangeFS_IP_RANGE=$(gcloudcomputeaddressesdescribeCLUSTER_IDfs-ip-range \--global  \--format="value[separator=/](address, prefixLength)")# Create the Filestore instancegcloudfilestoreinstancescreateFS_INSTANCE_ID \--project=PROJECT_ID \--location=ZONE \--tier=ZONAL \--file-share=name="nfsshare",capacity=1024 \--network=name=NETWORK,connect-mode=DIRECT_PEERING,reserved-ip-range="${FS_IP_RANGE}"

Lustre

Google Cloud Managed Lustre delivers a high-performance, fully managed parallel file systemoptimized for AI and HPC applications. With multi-petabyte-scale capacity and up to 1 TBpsthroughput, Managed Lustre facilitates the migration of demanding workloads tothe cloud.

Managed Lustre instances live inzones within regions. A region is aspecific geographical location where you can run your resources. Each region is subdivided intoseveral zones. For example, theus-central1 region in the central United States has zonesus-central1-a,us-central1-b,us-central1-c, andus-central1-f. For more information, seeGeography and regions.

To decrease network latency, we recommend creating a Managed Lustre instance in a region andzone that's close to where you plan to use it.

When creating a Managed Lustre instance, you must define the following properties:

• The name of the instance used by Google Cloud.
• The file system name used by client-side tools, for examplelfs.
• The storage capacity in gibibytes (GiB). Capacity can range from 9,000 GiB to ~8 PiB (7,632,000 GiB). The maximum size of an instance depends on its performance tier.
• Managed Lustre offersperformance tiers ranging from 125 MBps per TiB to 1000 MBps per TiB.
• For best performance, create your instance in the same zone as your training cluster.
• The VPC network for this instance must be the same one your training cluster uses.

Managed Lustre offers4 performance tiers,each with a different maximum throughput speed per TiB. Performance tiers also affect the minimumand maximum instance size, and the step size between acceptable capacity values. You cannot changean instance's performance tier after it's been created.

Deploying Managed Lustre requires Private Service Access, which establishes VPC peering betweenthe training cluster's VPC and the VPC hostingManaged Lustre, using a dedicated /20 subnet.

Configure Managed Lustre instance (optional)

Use Google Cloud Managed Lustre only if you want to use the Managed Lustre in Model Development Service.

Google Cloud Managed Lustre is a fully managed, high-performance parallel file system service on Google Cloud. It's specifically designed to accelerate demanding workloads in AI/Machine Learning and High-Performance Computing (HPC).

For optimal performance when using training clusters, Google Cloud Managed Lustre should be deployed from the same VPC and zone as your training cluster using VPC peering toservices networking.

Create Lustre instance

    gcloud lustre instances createLUSTRE_INSTANCE_ID \    --project=PROJECT_ID \    --location=ZONE \    --filesystem=lustrefs \    --per-unit-storage-throughput=500 \    --capacity-gib=36000 \    --network=NETWORK_NAME