Note: This feature is available with the Spanner Enterprise edition and Enterprise Plus edition. For more information, see theSpanner editions overview.

This page describes and explains how tiered storage works in Spanner.This feature is supported in both GoogleSQL-dialect and PostgreSQL-dialect databases.

Spanner tiered storage is a fully-managed storage feature thatlets you choose whether to store your data on solid-state drives (SSD) or harddisk drives (HDD). By default, when you're not using tiered storage, your datais stored on SSD storage. Depending on how often you use or access the data, youmight consider using tiered storage and storing data on both SSD and HDDstorage.

• SSD storage is the most performant (higher queries per second) andcost-effective choice for most use cases. You should use it to store activedata with high write and read throughput and data that requires low-latencydata access.
• HDD storage is sometimes appropriate for large datasets that aren'tlatency-sensitive, are infrequently accessed, or if the cost of storage isan important consideration.

Using tiered storage lets you take advantage of both SSD storage, which supportsthe high performance of active data, and HDD storage, which supports infrequentdata access at a lower cost.

Choose between SSD and HDD storage

The following table lists the differences and similarities between SSD and HDDstorage. The throughput figures in the table are per node and scale linearlywith the number of nodes in your Spanner instance. When in doubt,we recommend that you choose SSD storage.

Usethroughput optimized writes toincrease write throughput beyond the numbers in the table. For more information,seePerformance overview.

Benefits

Tiered storage offers the following benefits by letting you use both SSD and HDDstorage:

• Significant total cost of ownership reduction: HDD storage provides alower cost option for large datasets that aren't latency-sensitive or areinfrequently accessed.
• Ease of management: Provides a fully-managed tiering service without thecomplexity of additional pipelines and split logic.
• Unified and consistent experience: Provides unified data access and asingle set of metrics across hot and (mutable) cold data
• Enhanced performance: Improves query performance by organizing your datain different locality group, which provides data locality and isolation acrosscolumns. Data in the same locality group is stored physically close together.

How tiered storage works

By default, when you create a new instance, data is only stored on SSD storage.Similarly, data in existing instances is also only stored on SSD storage.

If you choose to use tiered storage to store some data in HDD storage, you mustcreate alocality group,which is used to define the tiered storage policy for data in your schema. Whenyou create a locality group, you can define the storage type, eitherssd orhdd. Optionally, you can also define the amount of time that data is stored onSSD storage before it's moved to HDD storage. This time is relative to thedata's commit timestamp. After the specified time passes, Spannermigrates the data to HDD storage during its normal compaction cycle, whichtypically occurs over the course of seven days from the specified time. This isknown as an age-based tiered storage policy. When using an age-based tieredstorage policy, the minimum amount of time that data must be stored on SSDbefore it's moved to HDD storage is one hour.

With your locality groups defined, when you create your tables, you can set thetiered storage policy at the database, table, column, or secondary index-level.The tiered storage policy determines how and where data is stored. Forinstructions, seeCreate and manage locality groups.

Key behavior details

The following behaviors apply when using tiered storage:

• Storage tiers are applied at the cell level: Age-based tiered storagepolicies apply to individual cells within a row, based on the cell's committimestamp. Data is moved to HDD storage if its commit timestamp is olderthan the specified duration. A single row can therefore have some data onSSD and some on HDD, depending on when each cell was last updated.
• Interleaved tables are independent: Locality group settings and tieredstorage policies aren't inherited by interleaved child tables. You mustapply these settings to each table individually.
• Read operations don't affect data aging: Reading data from a tabledoesn't reset its age or prevent it from being moved to HDD. Only a writeoperation that updates the data resets its commit timestamp and keeps it inSSD storage.
• Moving data from HDD back to SSD: When you update data stored on HDD,Spanner writes the new version of the data directly to SSDstorage. The previous version of the data on HDD is marked as obsolete andis cleaned up later by a background compaction process. This ensures thatupdates to cold data immediately benefit from SSD performance for subsequentreads of the new data.

Back up and restore

You can back up and restore your data usingSpanner backups.The backup contains all storage schema information, includingINFORMATION_SCHEMA.LOCALITY_GROUP_OPTIONS, whichspecifies the storage type of each locality group. To restore a backup thatcontains locality groups to a new instance, the destination instance must be inthe Spanner Enterprise edition or Spanner Enterprise Plus edition.

Data Boost

You can useSpanner Data Boostto access data on SSD or HDD storage. Querying data on HDD uses the instance'sHDD disk load capacity,which is part of your compute capacity.

Search indexes

Full-text search andvector indexes inherit the locality groupthat is set on thedatabase object.

Observability

The following observability features are available for tiered storage.

Cloud Monitoring metrics

Spanner provides the following metrics to help you monitor yourtiered storage usage and data using Cloud Monitoring:

• spanner.googleapis.com/instance/storage/used_bytes (Total storage): Showsthe total bytes of data stored on SSD and HDD storage.
• spanner.googleapis.com/instance/storage/combined/limit_bytes: Shows thecombined SSD and HDD storage limit.
• spanner.googleapis.com/instance/storage/combined/limit_bytes_per_processing_unit:Shows the combined SSD and HDD storage limit for each processing unit.
• spanner.googleapis.com/instance/storage/combined/utilization: Shows thecombined SSD and HDD storage utilization, compared against the combinedstorage limit.
• spanner.googleapis.com/instance/disk_load: Shows the HDD usage inpercentage. If your instance reaches 100% HDD disk load, you experiencesignificant increased latency.

If you have existing queries that filter existing metrics bystorage_class:ssd, you must remove the filter to see your HDD usage.

To learn more about monitoring your Spanner resources, seeMonitor instances with system insights andMonitor instances with Cloud Monitoring.

Information schema

INFORMATION_SCHEMA.LOCALITY_GROUP_OPTIONS contains the list of locality groupsand options in your Spanner database. It includes informationfor thedefault locality group. For more information, seelocality_group_options for GoogleSQL-dialect databasesandlocality_group_options for PostgreSQL-dialect databases.

Built-in statistics tables

The following built-in statistics tables are available for databases usingtiered storage:

• SPANNER_SYS.TABLE_SIZES_STATS_1HOUR: Shows HDD and SSD storage usage foreach table in your database.
• SPANNER_SYS.TABLE_SIZES_STATS_PER_LOCALITY_GROUP_1HOUR: Shows HDD and SSDstorage usage for each locality group in your database.

For more information, seeTable sizes statistics.

The query statistics and read statistics tables have the following column thatis related to tiered storage:

• AVG_DISK_IO_COST: The average cost of this query in terms ofSpannerHDD disk load.Use this value to make relative HDD I/O cost comparisons between reads thatyou run in the database. A higher value indicates that you are using more HDDdisk load and your query might be slower than if it was running on SSD.Furthermore, if your HDD disk load is at capacity, the performance of yourqueries might be further impacted.

For more information, seeQuery statisticsandRead statistics.

Pricing

There is no additional charge for using tiered storage. You are charged thestandard Spanner pricing for the amount of compute capacity thatyour instance uses and the amount of storage that your database uses. Data thatis stored on SSD and HDD is billed at their respective storage rates. You aren'tcharged for moving data between SSD and HDD storage. Querying data on HDD usesthe instance'sHDD disk load capacity,which is part of your compute capacity pricing. For more information, seeSpanner pricing.

What's next

• Learn more aboutlocality groups.
• Learn how tocreate and manage locality groups.
• Learn more aboutoptimizing queries with timestamp predicate pushdown.