This document describes the features and limits ofExapools. Hyperdisk Exapools areHyperdisk pools that are designed forworkloads where Hyperdisk Storage Pools don't provide sufficient Hyperdiskstorage in a single zone. With Exapools, you purchasestorage and performance in bulk, then create disks in the pool to consume thestorage.

Note: Hyperdisk Exapools are generally available with an allowlist. To getaccess to this feature,contact your account team.

When to use Hyperdisk Exapools

Exapools are for large scale workloads with tens ofthousands of disks in a single zone that require between 500 TiB and2.5 EiB of durable block storage and more than 100 GiB/s of concurrentperformance. If your workload requires a lower scale of capacity or performance,use aHyperdisk Storage Pool or create disks outsideof a pool.

Exapools offerstorage and performance at the largest scale available in Compute Engine,while also offering cost savings by letting you provision moreperformance and capacity than you need, but only charging you for what you use.

Hyperdisk Exapools are ideal for the following workloads:

• Artificial intelligence (AI)
• Machine learning (ML)
• Scale out file systems, like HDFS and Lustre

How Exapools work

You create an Exapool with the aggregate capacity andperformance that all of your workload's disks within a zone will need, thenyou create disks in the pool as needed. You can use disks in anExapool as boot disks or data disks for your Compute Engineinstances and containers.

When you create a disk in an Exapool, you allocatesome of the pool's resources (size and performance) to the disk. When youdelete a disk in a pool, the resources allocated to the disk are returned to thepool for use by other disks.

For example, suppose you create a Hyperdisk Balanced Exapool with 50 PiB of capacity.If you create 100 10 TiB disks in the pool, the remaining availablecapacity of the pool decreases by 1,000 TiB.

Exapool types

When you create an Exapool, you must choose aHyperdisk type for the disks that will be in the pool.The following types are available:

• Hyperdisk Balanced Exapools: all the disks in the ExapoolareHyperdisk Balanced volumes.
• Hyperdisk Throughput Exapool: all the disks in the Exapool areHyperdisk Throughput volumes.

For more information about choosing a Hyperdisk type, seeChoose a Hyperdisk type for your workload.

Machine series support

Hyperdisk Throughput Exapools and Hyperdisk Balanced Exapools are supported with the same machineseries that support Hyperdisk Throughput and Hyperdisk Balanced, respectively.For a list of the supported machine series, seeMachine series support for Hyperdisk ThroughputandMachine series support for Hyperdisk Balanced.

Performance and capacity provisioning for Exapools

The provisioning type of a pool determines how disks in thepool consume resources.

Exapools use advanced provisioning for both capacity andperformance, which offers the most cost and time savings.

Advanced capacity provisioning

All Exapools use advanced capacity provisioning, whichoffers the following benefits:

• Thin provisioning: Compute Engine allocates data to disks in thepool as needed, not when the disk is provisioned.
• Overprovisioning: You can provision the disks in the poolwith up to 50x more capacity than you purchased for theExapool. This simplifies capacity planning and avoids downtimefor manual disk resizing.

Note: Exapools don't offer autogrow for capacity orperformance, and doesn't offer data compression.

For a detailed explanation of advanced capacity provisioning, seeAdvanced capacity provisioning.

Advanced performance provisioning

Exapools use advanced performance provisioning, whichoffers the following benefits:

• Thin provisioning: Compute Engine allocates performance resourcesto the disks in the pool as needed. Only the amount ofIOPS and throughput used by a disk in an Exapool consumesperformance.
• Overprovisioning: the total performance for all the disks in apool can be up to 50 times thepool's provisioned write performance.
• Shared performance: Disks in an Exapool share thepool's provisioned performance up to each disk's limit,saving costs for disks with different peak usage times.

For a detailed explanation of advanced performance provisioning, seeAdvanced performance provisioning.

Purchasing capacity and performance

An Exapool's provisioned performance, or how much IOPS andthroughput you can allocate to disks within the pool,depends on the following factors:

• The number of capacity units in the pool
• The disk type of the Exapool

How capacity units work

A capacity unit includes 1 GiB of capacity and a fixed amount ofperformance (IOPS and throughput) per GiB. For example, to add 1 PiB ofcapacity to a pool, you purchase 1,048,576 capacity units(1,048,576 GiB = 1 PiB). Exapools offerthree capacity unit types to choose from. The amount of performance added to thepool depends on the type of capacity units:

• Capacity-optimized units: offer a balanced rate of IOPS and throughputfor both reads and writes.
• Read-optimized units: offer the highest rate of read operations andthroughput per GiB.
• Write-optimized units: offer the highest rate of write operations andthroughput per GiB.

If your workload has more read operations than writes, or if it requires moreread throughput than write throughput, you'll achieve the best value bypurchasing read-optimized units.

You can combine different types of capacity units when you provision or expandan Exapool. The type and number of capacity units you purchasefor a pool is referred to as theblend of capacity units.

An Exapool's blend of units doesn't affect the performance of thedisks in the Exapool. The blend of units affects onlythe following factors:

• How you're billed for the Exapool.
• The maximum provisioned performance for the Exapool.

Choose a blend of capacity units for an Exapool

To meet your workload's needs most cost-effectively, work with youraccount team to choose a blend of capacity units that best fits yourworkload. For example, you can create an Exapool with a highnumber of read-optimized units for all the read-intensive disks in a project.

Capacity unit ratios for Hyperdisk Throughput Exapools

Hyperdisk Throughput Exapools have the following performance ratios based on the amountof capacity purchased:

Capacity unit type	Capacity units purchased	Exapool capacity (TiB)	Read throughput (MiB/s)	Write throughput (MiB/s)
Capacity-optimized	1,024	1	0.97	0.10
Read-optimized	1,024	1	3.38	0.10
Write-optimized	1,024	1	0.97	1.84

Capacity unit ratios for Hyperdisk Balanced Exapool

Hyperdisk Balanced Exapools have the following performance ratios based on the amount ofcapacity purchased:

Capacity unit type	Capacity units purchased	Exapool Capacity (TiB)	Read throughput (MiB/s)	Write throughput (MiB/s)	Read IOPS	Write IOPS
Capacity-optimized	1,024	1	5.12	1.02	275	102
Read-optimized	1,024	1	378.88	30.72	21,311	3,072
Write-optimized	1,024	1	220.16	71.68	12,288	6,656

How capacity units affect disk performance

An Exapool's blend of capacity units doesn't affect how disks inthe Exapool consume performance. You aren't billed differentlyfor the individual read and write operations of the disks in the pool.Purchasing units of a specific type doesn't restrict the disks to a fixedamount of read or write IOPS. Also, when you create a disk in anExapool, you don't specify a read IOPS or write IOPS limit, youonly specify a provisioned IOPS limit.

For example, consider two Hyperdisk Balanced Exapools,Pool-1 andPool-2.Both have 100,000,000 capacity-optimized units.Pool-1 also has 5,000,000write-optimized units, whilePool-2 has 5,000,000 read-optimized units.

Since both pools have a total of 105,000,000 units,they'll have the same total capacity of 100.1 PiB (105,000,000 GiB).However, because they have different types of capacity units, thepools have different maximum performance limits andcosts. There's no performance difference between disks in both pools.

Example

Suppose a Hyperdisk Balanced Exapool has 12,400,000 capacity units, consisting of:

• 5,000,000 capacity-optimized units
• 2,400,000 read-optimized units
• 5,000,000 write-optimized units

The Exapool has 12,109.4 TiB of capacity(1 TiB for every 1,024 units).

The Exapool has 151,488,476 IOPS, calculated as follows:

• IOPS from 5,000,000 capacity-optimized units: 1,840,820(1,342,773 read IOPS + 498,047 write IOPS)

• IOPS from 2,400,000 read-optimized units: 57,147,656(49,947,656 read IOPS + 7,200,000 write IOPS)

• IOPS from 5,000,000 write-optimized units: 92,500,000(60,000,000 read IOPS + 32,500,000 write IOPS)

Review performance and capacity utilization

Compute Engine offers metrics that you can use to monitor yourExapools. These metrics answer questions like:

• How much read and write IOPS is left in the Exapool?
• How much capacity has been allocated to disks in the Exapool?
• How many disks are in the Exapool?

You can view these metrics in Cloud Monitoring. To learn more, seeMonitor Hyperdisk pools.

Size and performance limits for Hyperdisk Exapools

This section lists the limits for each Exapool type.

Property	Hyperdisk Balanced Exapools	Hyperdisk Throughput Exapools
Minimum capacity	500 TiB	500 PiB
Maximum capacity	1 EiB	5 EiB
Minimum throughput	1 TiB/s	250 GiB/s
Maximum throughput	10 TiB/s	10 TiB/s
Maximum number of disks per pool	500,000	500,000
Capacity increments	1 TiB	1 PiB
Maximum Exapools per project per zone	1	1
Performance ratios	32KB I/O size for read and write throughput; 4KB I/O size for read and write IOPS	1MB I/O size for reads, 256K I/O size for writes

Size and performance limits for disks within an Exapool

Disks in an Exapool have the same size and performance limits asdisks that aren't in a pool, as follows:

Hyperdisk type	Max provisionable performance per disk	Customizable throughput	Customizable IOPS
Hyperdisk Balanced	160,000 IOPS; 2,400 MiB/s throughput	Yes	Yes
Hyperdisk Throughput	2,400 MiB/s throughput	Yes	No; 4 IOPS per MiB/s of throughput, up to 9,600 IOPS

You can specify a performance limit for the disk at creation time and modifythe performance limit while the disk is in use.

For details of the size and performance limits, seeSize and performance limits for Hyperdisk ThroughputandSize and performance limits for Hyperdisk Balanced.

Regional availability

Hyperdisk Balanced Exapools and Hyperdisk Throughput Exapools are available in all zones.

Encryption

Encrypting disks in a Hyperdisk pool works the same way as encryptingdisks outside of a pool.

For more information, seeAbout disk encryption.

Pricing

You're billed for Exapools based on the number and type ofcapacity and performance units you purchased for the pool.You aren't billed for the provisioned IOPS, throughput, or capacity for thedisks created in the pool.

Hyperdisk Exapools are eligible for resource-based committed usediscounts (CUDs) with a minimum 1-year or up to3-year commitment. When you purchase a 1- or 3-year Exapool, you automaticallyqualify for resource-based CUDs in 1- or 3-year commitments.

For more information, seeDisk pricing.

Limitations

The following limitations apply to Exapools.

• You mustcontact your account team to create, modify, or delete aHyperdisk Exapool.
• The amount of performance you can provision for anExapool depends on the type and number of capacity units forthe Exapool. However, performance for disks created in anExapool is independent of the pool's blend of capacity units.
• You can't use Confidential mode for Hyperdisk Balanced volumes in an Exapool.
• Exapools only use advanced capacity and advancedperformance provisioning. They don't support standard capacity or standardperformance provisioning.
• Exapools don't use data compression.
• Exapools don't support autogrow for performance orcapacity. You must monitor your Exapool's utilization ofperformance and capacity yourself. If you need to increase the pool's performanceor capacity, you mustcontact your account team.
• You can't move existing disks into or out of an Exapool.You must create a standard snapshot of the disk, then use the snapshotto create a new disk.

What's next

• Contact your account team to create an Exapool
• View the properties of a Hyperdisk pool
• Add disks in a pool to a VMs