Cloud Storage FUSE
This document provides an overview of Cloud Storage FUSE, aFUSEadapter that lets you mount and access Cloud Storage buckets aslocal file systems, so applications can read and write objects in your bucketusing standard file system semantics.
This documentation always reflects the latest version of Cloud Storage FUSE. For details on the latestversion, seeCloud Storage FUSE releases on GitHub.
Cloud Storage FUSE is an open source product supported by Google.Cloud Storage FUSE uses FUSE and Cloud Storage APIs to transparentlyexpose buckets as locally mounted folders on your file system.
To use Cloud Storage FUSE, you install the Cloud Storage FUSE package on a machine with acompatible operating system such as Linux or Windows Subsystem forLinux, ensure proper Google Cloud authentication and permissions, and thenexecute thegcsfuse command to mount a specific Cloud Storage bucket to alocal directory.
Cloud Storage FUSE is integrated with other Google Cloud services. For example, theCloud Storage FUSE CSI driver lets you use the Google Kubernetes Engine (GKE) API toconsume buckets as volumes, so you can read from and write toCloud Storage from within your Kubernetes pods. For more information onother integrations, seeIntegrations.
How Cloud Storage FUSE works
Cloud Storage FUSE works by translating object storage names into a directory-likestructure, interpreting the slash character (/) in object names as a directoryseparator. Objects with the same common prefix are treated as files in thesame directory, allowing applications to interact with the mounted bucket like afile system. Objects can also be organized into a logical file systemstructure usinghierarchical namespace, which lets you organize objectsinto folders.
Cloud Storage FUSE can be run from anywhere with connectivityto Cloud Storage, including Google Kubernetes Engine, Compute Engine VMs,or on-premises systems.
Cloud Storage FUSE for machine learning
Cloud Storage FUSE is ideal for use cases where Cloud Storage hasthe right performance and scalability characteristics for an application thatrequires file system semantics. For example, Cloud Storage FUSE isuseful for machine learning (ML) projects because it provides a way tostore data, models, checkpoints, and logs directly in Cloud Storage.For more information, seeCloud Storage FUSE for ML workloads.
Cloud Storage FUSE is a common choice for developers looking to store and access MLtraining and model data as objects in Cloud Storage. Cloud Storage FUSEprovides several benefits for developing ML projects:
Cloud Storage FUSE lets you mount Cloud Storage buckets as a local filesystem so your applications can access training and model data using standardfile system semantics. This means that you can avoid the cost of rewriting orrefactoring your application's code when using Cloud Storage tostore ML data.
From training to inference, Cloud Storage FUSE lets you use the built-in highscalability, performance, andcost effectiveness ofCloud Storage, so you can run your ML workloads at scale.
Cloud Storage FUSE lets you start training jobs quickly by providingcompute resources with direct access to data in Cloud Storage, so youdon't need to download training data to the compute resource.
For more information, seeFrameworks, operating systems, and architectures supported by Cloud Storage FUSE.
Frameworks, operating systems, and architectures
Cloud Storage FUSE has been validated with the following frameworks:
TensorFlow V2.x
TensorFlow V1.x
PyTorch V2.x
PyTorch V1.x
JAX 0.4.x
Cloud Storage FUSE supports the following operating systems and architectures:
Rocky Linux 8.9 or later
Ubuntu 18.04 or later
Debian 10 or later
CentOS 7.9 or later
RHEL 7.9 or later
SLES 15 or later
x86_64
ARM64
Integrations with other Google Cloud products
Cloud Storage FUSE integrates with the following Google Cloud products:
| Product | How Cloud Storage FUSE is integrated |
|---|---|
| AI Hypercomputer | Cloud Storage FUSE is recommended for AI and ML use cases because it lets you mount buckets as local file systems and scale your data storage with more cost efficiency than file system services. For more information, seeStorage services. |
| Batch | Cloud Storage FUSE lets you mount Cloud Storage buckets as storage volumes when you create and run Batch jobs. You canspecify a bucket in a job's definition, and the bucket gets automatically mounted to the VMs for the job when the job runs. |
| Cloud Composer | When you create an environment, Cloud Composerstores the source code for your workflows and their dependencies in specific folders in a Cloud Storage bucket. Cloud Composer uses Cloud Storage FUSE to map the folders in the bucket to the Airflow components in the Cloud Composer environment. |
| Cloud Run | Cloud Run lets you mount a Cloud Storage bucket as a volume and presents the bucket content as files in the container file system. To set up volume mounting, seeMount a Cloud Storage volume. |
| Cluster Toolkit | Cluster Toolkit lets you create or mount a Cloud Storage bucket as a file system. You can specify the bucket in a blueprint YAML file using theappropriate module. The bucket is then automatically created or mounted when the deployment runs. |
| Dataflow | When you use Cloud Storage FUSE tomount Cloud Storage buckets directly onto the worker file system, theApache Beam pipeline code underlying Dataflow can access files in Cloud Storage directly using standard file system semantics. This is particularly helpful when using Dataflow for AI/ML tasks that involve large datasets and software that requires file access. |
| Deep Learning Containers | To mount Cloud Storage buckets for Deep Learning Containers, you can either use theCloud Storage FUSE CSI driver (recommended) orinstall Cloud Storage FUSE. |
| Deep Learning VM Images | Cloud Storage FUSE comes pre-installed with Deep Learning VM Images. |
| Google Kubernetes Engine (GKE) | TheCloud Storage FUSE CSI driver manages the integration of Cloud Storage FUSE with the Kubernetes API to consume Cloud Storage buckets as volumes. You can use the Cloud Storage FUSE CSI driver to mount buckets as file systems on Google Kubernetes Engine nodes. |
| Vertex AI training | You can access data from a Cloud Storage bucket as amounted file system when you perform custom training on Vertex AI. For more information, seeRead and write Cloud Storage files with Cloud Storage FUSE. |
| Vertex AI Workbench | Vertex AI Workbench instances include a Cloud Storage integration that lets you browse buckets and work with compatible files located in Cloud Storage from within the JupyterLab interface. The Cloud Storage integration lets you access all of the Cloud Storage buckets and files that your instance has access to within the same project as your Vertex AI Workbench instance. To set up the integration, seeVertex AI Workbench instructions for how to access Cloud Storage buckets and files in JupyterLab. |
For a list of Google Cloud products that are integrated with Cloud Storagegenerally, seeIntegration with Google Cloud services and tools.
Caching
Cloud Storage FUSE offers four types of caching to help increase performanceand reduce cost: file caching, stat caching, type caching, and list caching.For more information about these caches, seeOverview of caching.
Directory semantics
Cloud Storage offers buckets with aflat namespace and bucketswithhierarchical namespace enabled. By default, Cloud Storage FUSE caninfer explicitly-defined directories, also known asfolders, in bucketswith hierarchical namespace enabled but it can't infer implicitly-defineddirectories in buckets with a flat namespace, includingsimulated folders andmanaged folders.
Explicitly defined directories are folders that arerepresented by their own objects in Cloud Storage buckets.Implicitly defined directories are directories that don't have their owncorresponding objects in Cloud Storage buckets.
For example, say you mount a bucket namedmy-bucket, which contains an objectnamedmy-directory/my-object.txt, wheremy-directory/ is a simulated folder.When you runls on the bucket mount point, by default, Cloud Storage FUSE cannotaccess the simulated directorymy-bucket/my-directory/ nor the objectmy-object.txt within it. To enable Cloud Storage FUSE to infer the simulated folderand the object within it, include the--implicit-dirsgcsfuse option or theimplicit-dirs configuration file field as part of yourgcsfuse mountcommand when mounting a flat namespace bucket.
If you need to store and access your data using a file system, use buckets withhierarchical namespace enabled. To learn how to create such buckets, seeCreate buckets with hierarchical namespace enabled.
For more information about directory semantics, including how to mount bucketswith implicitly-defined directories, seeFiles and directories in the Cloud Storage FUSE GitHub documentation.
Cloud Storage FUSE retry strategies
By default, failed requests from Cloud Storage FUSE to Cloud Storage areretried with exponential backoff up to a specified maximum backoffduration, which has a value of30s (30 seconds) by default. Once the backoffduration exceeds the specified maximum duration, the retry continues with thespecified maximum duration. You can use the--max-retry-sleep option or thegcs-retries:max-retry-sleep field as part of agcsfuse mount call tospecify the backoff duration.
Retry strategy for stalledGET orREAD requests
When you perform aGET orREAD request with Cloud Storage FUSE, a timeout periodis applied. If the request exceeds the timeout period, Cloud Storage FUSE cancelsthe request and retries using anexponential backoff algorithm.
The timeout is dynamic and is based on the 99th percentile latency of pastsuccessful or canceledGET orREAD requests, with a 1.5-second minimum. Thisensures that only the slowest 1% of requests, those exceeding the 99thpercentile historical latency, are retried.
Retry strategy for stalled uploads
Large file writes are uploaded in chunks. To help reduce tail end writelatencies, if a chunk-level write operation stallsor fails, Cloud Storage FUSE attempts a retry after 10 seconds. A maximum of fourretry operations are performed for each stalled chunk.
Cloud Storage FUSE operations associated with Cloud Storage operations
When you perform an operation using Cloud Storage FUSE, you also perform theCloud Storage operations associated with the Cloud Storage FUSEoperation. The following table describes common Cloud Storage FUSE commands andtheir associated Cloud Storage JSON API operations. You can displayinformation about the Cloud Storage FUSE operations by setting the--log-severity option or thelogging:severity field toTRACE in yourgcsfuse command.
| Command | JSON API Operations |
|---|---|
gcsfuse --log-severity=TRACE example-bucket mp | Objects.list (to check credentials) |
cd mp | n/a |
ls mp | Objects.list("") |
mkdir subdir | Objects.get("subdir") Objects.get("subdir/") Objects.insert("subdir/") |
cp ~/local.txt subdir/ | Objects.get("subdir/local.txt") Objects.get("subdir/local.txt/") Objects.insert("subdir/local.txt"), to create an empty object Objects.insert("subdir/local.txt"), when closing after done writing |
rm -rf subdir | Objects.list("subdir") Objects.list("subdir/") Objects.delete("subdir/local.txt") Objects.list("subdir/") Objects.delete("subdir/") |
Metrics
Cloud Storage offers in-depth metrics which can help you optimizeCloud Storage FUSE performance and costs. To learn more about metrics forCloud Storage FUSE, seeCloud Storage FUSE Metrics.
Security
Cloud Storage FUSE applies Google Cloud's standard authentication throughApplication Default Credentials to identify users or service accounts.Access to the contents within buckets is governed byIdentity and Access Managementpermissions. Local system permissions secure the mount point itself and anylocally cached data.
Pricing for Cloud Storage FUSE
Cloud Storage FUSE is available free of charge, but the storage, metadata,and network I/O it generates to and from Cloud Storage are chargedlike any other Cloud Storage interface. In other words, all datatransfer and operations performed by Cloud Storage FUSE map toCloud Storage transfers and operations, and are charged accordingly.For more information on common Cloud Storage FUSE operations and how they map toCloud Storage operations, seeoperations mapping.
To avoid surprises, you should estimate how your use of Cloud Storage FUSE translatesto Cloud Storage charges. For example, if you are using Cloud Storage FUSEto store log files, you can incur charges quickly if logs are aggressivelyflushed on hundreds or thousands of machines at the same time.
SeeCloud Storage pricing for information on charges such asstorage, network usage, and operations.
Limitations
While Cloud Storage FUSE has a file system interface, it is not like an NFSor CIFS file system on the backend. Additionally, Cloud Storage FUSE is notPOSIX compliant. For a POSIX file system product in Google Cloud, seeFilestore.
When using Cloud Storage FUSE, be aware of its limitations andsemantics,which are different than that of POSIX file systems. Cloud Storage FUSE should onlybe used within its capabilities.
Limitations and differences from POSIX file systems
The following list describes the limitations of Cloud Storage FUSE:
- Metadata: Cloud Storage FUSE does not transfer objectmetadata when uploading files to Cloud Storage, with the exception ofmtime and symlink targets. This means that youcannot setobject metadata whenyou upload files using Cloud Storage FUSE. If you need to preserve objectmetadata, consider uploading files using theGoogle Cloud CLI,theJSON API, or theGoogle Cloud console.
- Concurrency: when you configure multiple mount points toaccess the same Cloud Storage bucket, and two or more mounts attempt towrite to the same Cloud Storage object simultaneously, the first mount tocomplete the write operation and successfully flush its changes to the objectis saved. Other mounts that haven't updated their local file descriptors afterthe object is modified encounter a
syscall.ESTALEerror whenattempting to save their edits due to precondition checks. To ensure that yourdata is consistently written, we strongly recommend against multiple sourcesmodifying the same object.Note that multiple readers can access the same ordifferent objects within a bucket, and multiple writers can modifydifferent objects in the same bucket simultaneously. Concurrent writes to thesame Cloud Storage object are supported from the same mount and behavesimilarly to built-in file systems.
- Linking: Cloud Storage FUSE does not support hard links.
- File locking and file patching: Cloud Storage FUSE does notsupport file locking or file patching. As such, you shouldn't store versioncontrol system repositories in Cloud Storage FUSE mount points, as versioncontrol systems rely on file locking and patching. Additionally, you shouldn'tuse Cloud Storage FUSE as a file replacement.
- Semantics: semantics in Cloud Storage FUSE are differentfrom semantics in a conventional file system. For example, metadata like lastaccess time are not supported, and some metadata operations like directoryrenaming are not atomic unless you use buckets withhierarchical namespace enabled.For a list of differences betweenCloud Storage FUSE semantics and conventional file system semantics, seeCloud Storage FUSE semanticsin the Cloud Storage FUSE GitHub documentation. To learn about how Cloud Storage FUSEinfers directories in Cloud Storage, seedirectory semantics.
- Workloads that do file patching (or overwrites in place):Cloud Storage FUSE can only write whole objects at a time toCloud Storage and does not provide a mechanism for patching. If you tryto patch a file, Cloud Storage FUSE will reupload the entire file. The onlyexception to this behavior is that you can append content to the end of a filethat's 2 MB and larger, where Cloud Storage FUSE will only reupload the appendedcontent.
- Access: authorization for files is governed byCloud Storage permissions. POSIX-style access control does not work.
- Performance: Cloud Storage FUSE has much higher latency thana local file system, and as such, shouldn't be used as the backend for storinga database. Throughput may be reduced when reading or writingone small file at a time. Using larger files or transferring multiple filesat a time will help increase throughput.
- Availability: transient errors can sometimes occur whenyou use Cloud Storage FUSE to access Cloud Storage. It'srecommended that you retry failed operations usingretry strategies.
- Object versioning: Cloud Storage FUSE does notformally support usage with buckets that have object versioning enabled.Attempting to use Cloud Storage FUSE with buckets that have object versioningenabled can produce unpredictable behavior.
- File transcoding:
objects with
content-encoding: gzipin metadata: Any such object in a Cloud Storage FUSE-mounted directory does not undergodecompressive transcoding. Instead, the object remains compressed in the same manner that it's stored in the bucket.For example, a file of 1000 bytes, uploaded to a bucket using the
gcloud storage cpcommand with the--gzip-localflag, might become 60 bytes (the actual compressed size depends on the content and the gzip implementation used by the gcloud CLI) as a Cloud Storage object. If the bucket is mounted using `gcsfuse`, and the corresponding file is listed or read from the mount directory, its size is returned as 60 bytes, and its contents are a compressed version of the original 1000 bytes content.This is in contrast to a download using
Note: Attempting to use Cloud Storage FUSE to edit or modify objects withgcloud storage cp gs://bucket/path /local/pathwhich undergoes decompressive transcoding: in thegcloudcommand, the content is auto-decompressed during the download, and the original, uncompressed content is served.content-encoding: gzipcan produce unpredictable behavior. This is because Cloud Storage FUSE uploads the object content as it is (without compressing it) while retainingcontent-encoding: gzip, and if this content is not properly gzip-compressed, it might fail in being read from the server by other clients, such as the gcloud CLI. This is because other clients employ decompressive transcoding while reading, and it fails for improper gzip content. - Retention policies: Cloud Storage FUSE does notsupport writing to buckets with aretention policy.If you attempt to write to a bucket with a retention policy, your writes willfail.
Cloud Storage FUSE supports reading objects from buckets with a retentionpolicy, but the bucket must be mounted as
Read-Onlyby passing the-o ROflag during bucket mounting. - Local storage: objects that are new or modified arestored in their entirety in a local temporary file until they are closed orsynced. When working with large files, make sure you have enough local storagecapacity for temporary copies of the files, particularly if you are working withCompute Engine instances. For moreinformation, see theCloud Storage FUSE README overview file in theCloud Storage FUSE GitHub documentation.
- File handle limits: the Linux kernel has a default limit of1,024 open file handles. When using Cloud Storage FUSE as a server to handle multipleconcurrent connections, you might exceed this limit. To avoid issues, ensure thenumber of concurrent connections to a single host remains under the limit, andconsider increasing the limit. Scenarios where this is important include using aCloud Storage FUSE mount to serve web content, host network-attached storage (NAS),or host a file transfer protocol (FTP) server. When serving web content onCloud Run from a Cloud Storage FUSE mount, themaximum concurrent requests per instanceis restricted to less than 1,000.
rsynclimitations: Cloud Storage FUSE's file systemlatency affectsrsync, which reads and writes onlyone file at a time. To transfer multiple files to or from your bucket inparallel, use theGoogle Cloud CLIby runninggcloud storage rsync.For more information, see thersyncdocumentation.- List operations limitations: when you list all theobjects in a mounted bucket, for example, by running
ls,Cloud Storage FUSE calls theObjects: list APIon Cloud Storage. The API paginates results, which means thatCloud Storage FUSE might need to issue multiple calls, depending on how manyobjects are in your bucket, which can make a list operation expensive and slow.
Known issues
For a list of known issues in Cloud Storage FUSE, see theopen Cloud Storage FUSE issues in GitHub.
Get support
You can get support, submit general questions, and request new features byusing one ofGoogle Cloud's official support channels. You can alsoget support byfiling issues in GitHub.
For solutions to commonly-encountered issues, seeTroubleshooting for production issues in the Cloud Storage FUSE GitHubdocumentation.
What's next
Learn how toinstall the Cloud Storage FUSE CLI.
Discover Cloud Storage FUSE by completing aquickstart.
Learn how tomount buckets.
Learn how to configure the behavior of Cloud Storage FUSE, using the
gcsfuseCLI or aCloud Storage FUSE configuration file.
Except as otherwise noted, the content of this page is licensed under theCreative Commons Attribution 4.0 License, and code samples are licensed under theApache 2.0 License. For details, see theGoogle Developers Site Policies. Java is a registered trademark of Oracle and/or its affiliates.
Last updated 2025-12-17 UTC.