This document provides an overview of Cloud Storage FUSE file caching andinstructions on how to configure and use file caching.

Cloud Storage FUSE file caching is a client-side read cache that enhances theperformance of read operations by serving repeat file reads from a faster cachestorage of your choice. When file caching is enabled, Cloud Storage FUSE storescopies of frequently accessed files locally, allowing subsequent reads to beserved directly from the cache, which reduces latency and improves throughput.

Benefits of file caching

File caching provides the following benefits:

• Improved performance for small and random I/Os: file caching improveslatency and throughput by serving reads directly from the cache media. Smalland random I/O operations can be significantly faster when served from thecache.

• Parallel downloads enabled automatically: parallel downloads are enabledautomatically on Cloud Storage FUSE versions 2.12 and later when the file cacheis enabled. Parallel downloads utilize multiple workers to download a filein parallel using the file cache directory as a prefetch buffer, which canresult in up to nine times faster model load time. We recommend that youuse parallel downloads for single-threaded read scenarios that load largefiles such as model serving and checkpoint restores.

• Use of existing capacity: file caching can use existing provisioned machinecapacity for your cache directory without incurring charges for additionalstorage. This includes Local SSDs that come bundled with Cloud GPUs machinetypes such asa2-ultragpu,a3-highgpu, Persistent Disk (which isthe boot disk used by each VM), or in-memory/tmpfs.

• Reduced charges: cache hits are served locally and don't incurCloud Storage operation or network charges.

• Improved total cost of ownership for AI and ML training: file cachingincreases Cloud GPUs and Cloud TPU utilization by loading datafaster, which reduces time to training and provides a greaterprice-performance ratio for artificial intelligence and machine learning(AI/ML) training workloads.

Parallel downloads

Parallel downloads can improve read performance by usingmultiple workers to download multiple parts of a file in parallel using thefile cache directory as a prefetch buffer. We recommend using paralleldownloads for read scenarios that load large filessuch as model serving, checkpoint restores, and training on large objects.

Use cases for enabling file caching with parallel downloads include thefollowing:

Considerations

• File cache time to live (TTL): if a file cache entry hasn't yet expiredbased on its TTL and the file is in the cache, read operations to that fileare served from the local client cache without any request being issued toCloud Storage.

• File cache entry expiration: if a file cache entry has expired, aGETfile attributes call is first made to Cloud Storage. If the file ismissing or its attributes or contents have changed, the new content isretrieved. If the attributes were only invalidated but the content remainsvalid, meaning object generation hasn't changed, the content is served fromthe cache only after the attribute call confirms its validity. Bothoperations incur network latencies.

• File cache invalidation: when a Cloud Storage FUSE client modifies a cachedfile or its attributes, that client's cache entry is immediately invalidatedfor consistency. However, other clients accessing the same file continue toread their cached versions until their individual TTL settings cause aninvalidation.

• File size and available capacity: the file being read must fit within theavailable capacity in the file cache directory available capacity which canbe controlled using either the--file-cache-max-size-mb option or thefile-cache:max-size-mb field.

• Cache eviction: the eviction of cached metadata and data is based on aleast recently used (LRU) algorithm that begins once the space thresholdconfigured per--file-cache-max-size-mb limit is reached. If theentry expires based on its TTL, aGET metadata call is first made toCloud Storage and is subject to network latencies. Since the data andmetadata are managed separately, you might experience one entity beingevicted or invalidated and not the other.

• Cache persistence: Cloud Storage FUSE caches aren't persisted on unmounts andrestarts. For file caching, while the metadata entries needed to serve filesfrom the cache are evicted on unmounts and restarts, data in the file cachemight still be present in the file directory. We recommend that you deletedata in the file cache directory after unmounts or restarts.

• Random and partial read management: when the first file read operationstarts from the beginning of the file, at offset0, the Cloud Storage FUSEfile cache ingests and loads the entire file into the cache, even if you'reonly reading from a small range subset. This lets subsequent random orpartial reads from the same object get served directly from the cache.

  By default, reading from any other offset doesn't trigger an asynchronousfull file fetch. To change this behavior so that Cloud Storage FUSE ingests a fileto the cache upon an initial random read, set either the--file-cache-cache-file-for-range-read option orfile-cache:cache-file-for-range-read field totrue.

  We recommend that you enable this property if many different random orpartial read operations are performed on the same object.

• Data security: when you enable caching, Cloud Storage FUSE uses the cachedirectory you specified using either the--cache-dir option orcache-dir field as the underlying directory for the cache to persistfiles from your Cloud Storage bucket in an encrypted format. Any user orprocess that has access to this cache directory can access thesefiles. We recommend restricting access to this directory.

• Direct or multiple access to the file cache:using a process other than Cloud Storage FUSE to access or modify a file in thecache directory can lead to data corruption. Cloud Storage FUSE caches are specificto each Cloud Storage FUSE running process with no awareness across differentCloud Storage FUSE processes running on the same or different machines. Therefore,we don't recommend using the same cache directory for different Cloud Storage FUSEprocesses.

• Running multiple Cloud Storage FUSE processes on the same machine:if multiple Cloud Storage FUSE processes need to run on the same machine, eachCloud Storage FUSE process should get its own specific cache directory, or useone of following methods to ensure your data doesn't get corrupted:

  • Mount all buckets with a shared cache: use dynamic mounting to mount allbuckets you have access to in a single process with a shared cache. To learnmore, seeCloud Storage FUSE dynamic mounting.

  • Enable caching on a specific bucket: enable caching on only a specifiedbucket using static mounting. To learn more, seeCloud Storage FUSE static mounting.

  • Cache only a specific folder or directory: mount and cache only aspecific bucket-level folder instead of mounting an entire bucket. To learnmore, seeMount a directory within a bucket.

Before you begin

The file cache requires a directory path to be used to cache files. You cancreate a new directory on an existing file system or create a new file systemon provisioned storage. If you are provisioning new storage to be used,use the following instructions to create a new file system:

1. For Google Cloud Hyperdisk, seeCreate a new Google Cloud Hyperdisk volume.

2. For Persistent Disk, seeCreate a new Persistent Disk volume.

3. For Local SSDs, seeAdd a Local SSD to your VM.

4. For in-memory RAM disks, seeCreating in-memory RAM disks.

Enable and configure file caching behavior

1. Select the method through which you want to enable and configure filecaching using one of the following methods:

   • Supply it as the value for agcsfuse option

   • Specify it in aCloud Storage FUSE configuration file

   Note: You can also use sample configurations to enable and configurefile caching. For more information, seeSample configuration for enabling file caching and parallel downloads.

2. Specify the cache directory you want to use with one of the followingmethods. This lets you enable the file cache for non-Google Kubernetes Enginedeployments:

   • gcsfuse option:--cache-dir
   • Configuration file field:cache-dir

   If you're using a Google Kubernetes Engine deployment using the Cloud Storage FUSE CSI driverfor Google Kubernetes Engine, specify one of the following methods:

   • gcsfuse option:--file-cache-max-size-mb
   • Configuration file field:file-cache:max-size-mb
   Note: For more information on how to enable filecaching on Google Kubernetes Engine, seeEnable and use file caching.

3. Optional: enable parallel downloads by setting one of the following methodstotrue if parallel downloads weren't enabled automatically:

   • gcsfuse option:--file-cache-enable-parallel-downloads
   • Configuration file field:file-cache:enable-parallel-downloads

4. Limit the total capacity the Cloud Storage FUSE cache can use within itsmounted directory by adjusting one of the following options, which isautomatically set to a value of-1 when you specify a cache directory:

   • gcsfuse option:--file-cache-max-size-mb
   • Configuration file field:file-cache:max-size-mb

   You can also specify a value in MiB or GiB to limit the cache size.

   Note: If you're using Compute Engine virtual machines (VMs) such asstandalone Cloud Storage FUSE or non-Google Kubernetes Engine based deployments, the--file-cache-max-size-mb option orfile-cache:max-size-mbfield is enabled automatically and set to-1 when you enablecache-dir.

5. Optional: bypass the TTL expiration of cached entries and serve file metadatafrom the cache if it's available using one of the following methods andsetting a value of-1:

   • gcsfuse option:--metadata-cache-ttl-secs
   • Configuration file field:metadata-cache:ttl-secs

   The default is 60 seconds, and a value of-1 sets it tounlimited. You can also specify a high value based on your requirements. Werecommend that you set thettl-secs value to as high as your workload letsyou. For more information about the TTL for cached entries, seeConsiderations.

6. Optional: enable the file cache's ability to asynchronously load the entirefile into the cache if the file's first read operation starts from anywhereother thanoffset 0 so that subsequent reads of different offsets from thesame file can also be served from the cache. Use one of the following methodsand set the option totrue:

   • gcsfuse option:--file-cache-cache-file-for-range-read
   • Configuration file field:file-cache:cache-file-for-range-read

7. Optional: configure stat caching and type caching. To learn more about statand type caches, seeOverview of type caching orOverview of stat caching.

8. Manually run thels -R command on your mounted bucket before you run yourworkload to pre-populate metadata to ensure the type cache populates aheadof the first read in a faster, batched method. For more information abouthow to improve first time read performance, seeImprove first-time reads.

Once you enable file caching, parallel downloads are enabled automaticallyon Cloud Storage FUSE versions 2.12 and later. If you're using an older version ofCloud Storage FUSE, set theenable-parallel-downloads option totrue toenable parallel downloads.

Configure supporting properties for parallel downloads

You can optionally configure the following supporting properties for paralleldownloads using the Cloud Storage FUSE CLI or a Cloud Storage FUSE configuration file:

Disable parallel downloads

To disable parallel downloads, set one of the following tofalse:

• gcsfuse option:--file-cache-enable-parallel-downloads
• Configuration file field:file-cache:enable-parallel-downloads

Control cacheability at file granularity using regular expressions

By default, all files that are read are cached. You can control which files arecached at file granularity by using the optionalinclude-regex andexclude-regex fields in thefile-cache configuration. These fields let you specify regular expressions tomatch against file paths in the formatbucket_name/object_key.

Include specific files for caching

To cache only files that match a specific pattern, use theinclude-regexoption. Any file that doesn't match this regular expression isn't cached.For example, to cache only.tfrecord files from your training dataset,specify the following in your configuration file:

file-cache:include-regex:".*\\.tfrecord$"

Or, use the--file-cache-include-regex flag:

gcsfuse--file-cache-include-regex".*\\.(tfrecord)$"..

Exclude specific files from caching

To prevent specific files from being cached, use theexclude-regex option.Any file that matches this regular expression isn't cached. For example, toexclude all files in a directory namedlogs/, specify the following in yourconfiguration file:

file-cache:exclude-regex:".*/logs/.*"

Or, use the--file-cache-exclude-regex flag:

gcsfuse--file-cache-exclude-regex".*/logs/.*"..

Use include and exclude rules together

You can use bothinclude-regex andexclude-regex at the same time. Theexclude-regex rule is always evaluated first. If a file matchesexclude-regex, it won't be cached, even if it also matchesinclude-regex.For example, to cache all.tfrecord files except for those in theoutput/directory, specify the following in your configuration file:

file-cache:include-regex:".*\\.tfrecord$"exclude-regex:".*/output/.*"

Or, use both flags:

gcsfuse--file-cache-include-regex".*\\.tfrecord$"--file-cache-exclude-regex".*/output/.*"..

What's next

• Reviewconsiderations for caching in Cloud Storage FUSE.

• Learn how to improveCloud Storage FUSE performance.