This overview introduces theCloud Storage FUSE CSIdriver for mountingCloud Storage buckets as localfile systems in Google Kubernetes Engine (GKE).This feature is particularly useful for machine learning workloads that need tostore training data, models, and checkpoints in Cloud Storage.

This overview is for Developers and Data scientists who wantto access or store training data, inference model weights, and checkpointsstored in Cloud Storage from their Kubernetes applications.

Before reading this page, ensure you're familiar with Kubernetes,GKE, and Cloud Storage.

Tip: To get started, follow theCloud Storage FUSE CSI driver quickstart.

How it works

The driver uses theContainer Storage Interface(CSI) standard to allow your applications runningin Pods to seamlessly access Cloud Storage bucketsas if they were mounted filesystems. This means you can treat yourCloud Storage buckets as a persistent and scalable data source for yourKubernetes applications without complex configuration or code changes.

The Cloud Storage FUSE CSI driver provides a fully-managed experience powered bythe open sourceGoogle Cloud Storage FUSE CSI plugin. The CSI driver lets you use the Kubernetes API to consume pre-existingCloud Storage buckets as volumes. Your applications can upload and downloadobjects usingCloud Storage FUSE file system semantics.

Filesystem in Userspace (FUSE)is an interface used to export a file system to the Linux kernel. Cloud Storage FUSElets you mount Cloud Storage buckets as a file system so that applicationscan access the objects in a bucket using common File I/O operations (for example,open, read, write, and close) rather than using cloud-specific APIs.

The driver natively supports the following ways for you to configure your Cloud Storage-backed volumes:

• CSI ephemeral volumes: You specify the Cloud Storage bucket in-linewith the Pod specification. Use ephemeral CSI volumes if you want a streamlined Pod-based interface that requires no previous experience with Kubernetes persistent volumes.To use this option, seeMount Cloud Storage buckets as CSI ephemeral volumes.
• PersistentVolumes: You create a PersistentVolume resource that refers tothe Cloud Storage bucket, using static provisioning. Your Pod can thenreference a PersistentVolumeClaim that is bound to this PersistentVolume. Usethis option if you are already familiar with PersistentVolumes and wantconsistency with your existing deployments that rely on this resource type. Touse this option, seeMount Cloud Storage buckets as persistent volumes.

Use cases

The Cloud Storage FUSE CSI driver is suitable for the scenarios like the following:

AI and machine learning

• Training: You can use the Cloud Storage FUSE CSI driver to read trainingdata and checkpoint saved models using Cloud Storage as the source of truth.For example, when training a model on GKE using PyTorch, JAX, orTensorFlow, the driver can provide access to training datasets storedin Cloud Storage buckets.
• Inference: You can serve ML inference models that infer results fromfiles stored in Cloud Storage. You can use Cloud Storage FUSE CSI to preloadmodel weights stored in Cloud Storage. Additionally, you can use theparallel download withfile cache to accelerate reading large files from Cloud Storagefor multi-threaded downloads. You can use this feature to improve model loadtimes, especially for reads of over 1 GB in size

Data analytics pipelines

You can use the Cloud Storage FUSE CSI driver to streamline data processing tasksby allowing applications to directly access and analyze large datasets storedin Cloud Storage. For example, a Spark job running on GKEcould use the CSI driver to process data stored in Cloud Storage withoutneeding to download it first.

Benefits

Using the CSI driver gives you these benefits:

• Easy to set up: The Cloud Storage FUSE CSI driver automatically deploysand manages the driver on both Standard and Autopilot clusters.Using CSI ephemeral volumes simplifies volume configuration and management.This is because there is no need for PersistentVolumeClaim and PersistentVolume objects.
• Security: The Cloud Storage FUSE CSI driver does not need privileged access.This minimizes the risks associated with privileged access and leads to a bettersecurity posture. You can useWorkload Identity Federation for GKE to manage authentication, giving you granular control overhow your Pods access Cloud Storage objects.
• Performance: The Cloud Storage FUSE CSI driver enhances performance throughfeatures like a sidecar for optimized interactions, parallel downloads forfaster data access, and metadata and file caching to improve read performanceand reduce latency. To learn more about these features, seePerformance tuning options and features.
• Portability and flexibility: The Cloud Storage FUSE CSI driver lets youuse standard file system semantics to mount and access Cloud Storage buckets.This provides a familiar interface that improves portability for ML workloads,and avoids the need for you to make extensive code or application changes.The driver is supported on all accelerators available on GKEincluding GPUs and TPUs. The Cloud Storage FUSE CSI driver supports theReadWriteMany,ReadOnlyMany, andReadWriteOnceaccess modes. You can consume Cloud Storage FUSE volumes ininit containers.
• Manageability: The driver lets you run Cloud Storage FUSE under thecovers without needing to install or manage it. You can also viewmetricsinsights for Cloud Storage FUSE, includingfile system, Cloud Storage, and file cache usage.

Performance tuning options and features

The Cloud Storage FUSE CSI driver comes with several performance tuning optionsand features for optimizing how your Pods access data stored in Cloud Storage buckets.

For example, by enabling file caching and adjusting the request concurrency,you could significantly reduce the time it takes to load the training data,leading to faster training times.

• Native sidecar: The Cloud Storage FUSE CSI driverattaches a sidecar containerin your Podsto manage interactions with Cloud Storage. The sidecar handlesthe mounting and interaction with Cloud Storage, allowing your applicationsto seamlessly access data. You can fine-tune performance by configuring resourceslike CPU and memory for the sidecar container, or by adjusting settings relatedto caching and buffering. The Cloud Storage FUSE CSI driver sidecar container andIstio can coexist and run concurrently in your Pod.

• Parallel download: Starting fromGKE version 1.30.3-gke.1571000 andCloud Storage FUSE v.2.4.0 with file cache enabled,you can use theparallel downloadfeature to accelerate reading large files from Cloud Storage for multi-threadeddownloads. You can use this feature to improve model load times, especially forreads over 1 GB in size (for example, up to twice as fast when loading Llama 2 70B).

• Metadata caching support: The Cloud Storage FUSE CSI driver enhancesperformance by caching file metadata, like size and modification time. The CSIdriver enables thisstat cacheby default and reduces latency by storing information locally instead of repeatedlyrequesting it from Cloud Storage. You can configure its maximum size and theduration for which the data stays in the cache. By fine-tuning the metadata cache, you can reduceAPI calls to Google Cloud Storage, improving application performance andefficiency by minimizing network traffic and latency.

• File caching support: You can use the Cloud Storage FUSE CSI driver withfile caching to improve the read performanceof applications handling small files from Cloud Storage buckets. TheCloud Storage FUSE file cache feature is a client-based read cache that allowsrepeated file reads to be served more quickly from cache storage of your choice.You can choose from a range of storage options for the read cache, includingLocal SSDs, Persistent Disk-based storage, and RAM disk, based on your price-performanceneeds.

• Node restart support: Starting from GKE version 1.33.1-gke.1959000, Pods using GCSFuse CSI volumes are able to automatically recover after a GKE node restart on nodes withoutGraceful Node Shutdown enabled. For clusters running earlier versions or nodess with Graceful Node Shutdown enabled, Pods are unable to recover and you need to redeploy. Graceful Node Shutdown is enabled by default only for preemptible VMs, Spot VMs, Confidential VMs, GPU VMs, TPU VMs, and the Z3, C4, C3 Metal machine families.

For performance tuning best practices, refer toOptimize Cloud Storage FUSE CSI driver for GKE performance.

Limitations

The CSI driver has these limitations:

• The Cloud Storage FUSE file system hasdifferences in performance, availability, access authorization, and semantics compared to aPOSIX file system.
• The Cloud Storage FUSE CSI driver is not supported onGKE Sandbox.
• The Cloud Storage FUSE CSI driver does not support volume snapshots,volume cloning, or volume expansions.
• Uniform bucket-level access isrequired for read-write workloads when using Workload Identity Federation for GKE.
• See theknown issues in the Cloud Storage FUSE CSI driver GitHub project.
• See theopen issuesin the Cloud Storage FUSE CSI driver GitHub project.
• For GKE versions earlier than1.33.3-gke.1226000, the Cloud Storage FUSE CSI driver does not support Pods running on thehost network (hostNetwork: true) due torestrictions of Workload Identity Federation for GKE. However, on Standard GKE clusters with version1.33.3-gke.1226000 or later, the Pods on the host network that use the Cloud Storage FUSE CSI driver can use their own Kubernetes Service Account to authenticate. For more information, seeConfigure access for Pods with host network.
• GKE cluster versions earlier than 1.32.3-gke.1099000 have a restrictive limit on the size of JSON encryptedmountOption strings. The combination of bucket name, user provided mount options, and the driver providedmountOptions must be fewer than 500 characters. We increased the threshold of JSON message to 5,000 bytes in GKE cluster versions 1.32.3-gke.1099000 or later.

Requirements

To use the Cloud Storage FUSE CSI driver, your clusters must meet the followingGKE version requirements:

• Use Linux clusters running GKE version 1.24 or later.
• HaveWorkload Identity Federation for GKE andGKE metadata server enabled.
• Use the latest version of theGoogle Cloud CLI.

To use specific features for the Cloud Storage FUSE CSI driver, you also need to meet the requirements specified in theCloud Storage FUSE CSI driver repository.

What's next

• Learn how to prepare to use the Cloud Storage FUSE CSI for GKE.
• Troubleshoot Cloud Storage FUSE
• Optimize Cloud Storage FUSE for performance