Hierarchical namespace is a capability offered by Cloud Storage thatlets you organize objects into folders and store your data in a logical filesystem structure, making it easier to streamline data management tasks.Hierarchical namespace provides the high-performance directory semantics andatomic folder operations that are required to accelerate data-intensive AI/MLand analytics workloads.

This page provides an overview of hierarchical namespace, including its keyfeatures, common use cases, benefits, and limitations.

How it works

To use folders in a bucket, you must enable hierarchical namespace when youcreate the bucket. Your bucket's hierarchical namespace setting can't bechanged after the bucket is created. For information about enablinghierarchical namespace for your bucket, seeCreate and manage buckets with hierarchical namespace enabled.

The following diagram shows an exampleof a bucket with hierarchical namespace enabled where objects are organized ina hierarchical structure of folders. A folder in a bucket withhierarchical namespace enabled can contain both objects and child folders.

Key features

Hierarchical namespace provides the following features:

• Higher initial queries per second (QPS): Buckets withhierarchical namespace enabled offer up to 8 times higher initial QPSlimits for reading and writing objects compared to buckets withouthierarchical namespace enabled. The higher initial QPS makes it easier toscale data-intensive workloads and provides enhanced throughput. Forinformation about performance optimization methods while using folders inbuckets with hierarchical namespace enabled, seeFolder management.

• Folders: Folders act as a container for objects and other folders,with support for operations such as creating, deleting and getting folders.Storing your data in a folder structure enhances performance, ensuresconsistency, and simplifies the management of data-intensive andfile-oriented workloads.

• Folder operations: Folder operations provide reliability and managementcapabilities, including creating, deleting, listing, and renaming folders.

  • Renaming folders: The rename folders operation helps you toatomically rename the path of a folder and its underlying folderswithout deleting any objects. This technique is efficient andtime-saving, especially for large folders with multiple objects.

  • Listing folders: The list folders operation lists all folders in thebucket or underneath a specific folder, helping you manage andunderstand the structure of your stored data.

When should you use hierarchical namespace?

You should consider enabling hierarchical namespace when working withapplications that expect a file system hierarchy and semantics.Hierarchical namespace is beneficial for data-intensive tasks like analyticsand AI/ML workloads. Here are some common scenarios where you should considerusing hierarchical namespace:

• Hadoop based processing: Hadoop and Spark workloads traditionally expecta file system storage structure and time-based naming for files and folders.Hierarchical namespace integrates with theCloud Storage connectorto provide enhanced throughput and atomic folder renames, improving dataintegrity and consistency for many data processing pipelines.

• File-oriented workloads processing: Workloads such as batch analyticsprocessing, financial services, or high performance computing are structuredinto partitions based on a hierarchy of folders and files.Hierarchical namespace helps to manage these environments with a dedicatedAPI for folder management. Additionally, hierarchical namespace simplifiesmanaging folders that contain other folders and objects. With a single APIcommand, you can swiftly rename a folder along with all its contents, savingvaluable time and resources.

• AI/ML processing: AI/ML tools such as TensorFlow,Pandas, and PyTorch expect file system access control and semantics.Hierarchical namespace, especially when combined with Cloud Storage FUSE,delivers increased throughput and efficient data access. As a result,hierarchical namespace enhances the performance and reliability of the MLmodel iteration.

Before enabling hierarchical namespace for your bucket, you should consider thelimitations of hierarchical namespace. For information abouthierarchical namespace limitations, seeLimitations.

Benefits of hierarchical namespace

When you enable Hierarchical namespace for your buckets, you can do thefollowing:

• Optimize organization: You can organize your data into a hierarchicalfolder structure, that helps you to manage and locate files or datasets.

• Establish a file system ecosystem: Hierarchical namespaceintroduces file system features such as folders, folder renaming, andfolder listing, which are beneficial for file-oriented applications,including the Hadoop ecosystem and AI/ML workloads.

• Performance improvement: By scaling data-intensive workloads to handlehigher throughput, you can enhance the overall performance of yourapplication.

Platform support

Buckets with hierarchical namespace support the followingCloud Storage platform capabilities:

• All Cloud Storage object APIs and widely-usedCloud Storage features. For details about unsupported features,seeLimitations.

• Data transfer from a standard bucket to a bucket withhierarchical namespace using Storage Transfer Service.

• Integration with the following products:

  • Cloud Storage Connector, maintained by Dataproc forHadoop workloads. For more information, seeUse hierarchical namespaceenabled buckets for Hadoop workloads.

  • Cloud Storage FUSE for file system-like bucket access using clients.

Compatibility with Cloud Storage operations and features

Buckets with hierarchical namespace enabled have the following interactionswith other Cloud Storage operations:

How object operations are handled

Buckets with hierarchical namespace enabled handleobject operations inthe following ways:

• Operations likeUpload,Rewrite, andComposeautomatically create any missing parent folders, as long as you have thenecessary permissions. As a result, you don't need to pre-create foldersbefore uploading objects.
• While folders can be created automatically during object operations, youneed to delete them explicitly using theDeleteFolder operation.
• When using theListObjects operation with thedelimiter parameter,buckets return each child folder as aprefix. However, empty folders areexcluded by default. To include empty folders, similar to a typical filesystem listing, you must set theincludeFoldersAsPrefixes parameter. Forinformation about performance optimization methods while listing objects inbuckets with hierarchical namespace enabled, seeListing objects.

How managed folders are handled

Buckets with hierarchical namespace enabled offer granular access controlwhen used withmanaged folders. In buckets with hierarchical namespaceenabled, thefolder is the actual directory that holds your data, whilethemanaged folder is a specialized resource used to apply granularIAM permissions to that directory. To manage access to folders,you must create a managed folder with the same name as the folder and then applyIAM policies to it.

Managed folders are handled in the following ways in buckets withhierarchical namespace enabled.

• Creating a managed folder automatically creates any missing parent folders,including the folder with the same name. For example, say you have an emptybucket with hierarchical namespace enabled calledexample-ai-datasets.If you create a managed folder withinexample-ai-datasets called2025/training-data/, Cloud Storage automatically creates the parentfolder2025/ and the target foldertraining-data/.
• A managed folder cannot exist without the associated folder.
• Deleting a folder automatically deletes the associated managed folder.
• Renaming a folder automatically renames the associated managed folder.
• Buckets with hierarchical namespace must followmanaged folder namingrules andfolder naming rules. Although folder names can benested up to 50 levels deep, managed folder names can only be nested up to15 levels deep. The maximum managed folder name size is limited by themaximum folder name size, which is 512 bytes when UTF-8 encoded.

How bucket operations are handled

The following sections describe how bucket operations are handled whenhierarchical namespace is enabled.

• You can list all buckets with hierarchical namespace enabled, regardless oftheir storage layout. A bucket'sstorage layout describes how objects are arranged within a bucket,either in a flat namespace or a hierarchical namespace. For instructions onviewing a bucket's storage layout, seeGet a bucket's storage layout.To list all buckets, follow the instructions detailed inList buckets.

• You candelete buckets with hierarchical namespace enabled as long asthe bucket is empty. Empty buckets contain no objects ormanaged folders. Empty buckets can contain empty folders(with no objects within).

How Object Lifecycle Management rules are handled

Object Lifecycle Management lets you automate actions on objectsbased on conditions, such as age or prefix. However, Object Lifecycle Managementrules can behave differently in buckets with hierarchical namespace and inbuckets with a flat namespace due to theRenameFolder operation:

• Object Lifecycle Management rules for buckets with a flat namespace:The renaming operation involves renaming every object using tools by copyingevery object to a destination location and deleting the original object fromthe source location. As a result, new objects are created with new creationtimes at the destination location. Ifage-based Object Lifecycle Managementrules are applied for the destination location, they won't apply to thenew objects immediately as their creation times are reset.

• Object Lifecycle Management rules for buckets with hierarchical namespaceenabled: Renaming a folder operates at the folder level, without having torename every single object. As a result, the creation time of the objects ispreserved, meaning theage-based Object Lifecycle Management rules areapplied to renamed objects immediately if they meet the age criteria.

Pricing

For pricing information, refer toCloud Storage pricing.

Limitations

The following are the limitations of hierarchical namespace:

• You must choose whether or not to use hierarchical namespace when youcreate the bucket; your bucket's hierarchical namespace setting can't bechanged after the bucket is created.

• In order to enable hierarchical namespace, a bucket must also enableuniform bucket-level access.

• The following Cloud Storage capabilities are not supported forbuckets that use hierarchical namespace:

  • Bucket Lock
  • Bucket relocation
  • Cross-bucket replication
  • Object holds
  • Object Retention Lock
  • Object Versioning

What's next

• Create buckets with hierarchical namespace enabled.
• Create and manage folders.
• Rename folders.
• Use hierarchical namespace for Hadoop workloads.
• Optimize performance.