Autoscaling

This page explains how autoscaling works. Before you read this page, you shouldbe familiar with theOverview of Bigtable andInstances, clusters, and nodes.

In Bigtable, instances are containers for clusters, which arelocation-specific resources that handle requests. Each cluster has one or morenodes, which are compute resources used to manage your data. When you create acluster in an instance, you choose eithermanual node allocation orautoscaling.

With manual node allocation, the number of nodes in the cluster remains constantuntil you change it. When autoscaling is enabled, Bigtablecontinuously monitors the cluster and automatically adjusts the number of nodesin the cluster when necessary. Autoscaling works on both HDD and SSD clusters,in all Bigtable regions.

You can configure autoscaling in the Google Cloud console, usinggcloud, or usingthe Cloud Bigtable client library for Java.

When to use autoscaling

We recommend that you enable autoscaling in most cases. The benefits ofautoscaling include the following:

• Costs - Autoscaling can help you optimize costs becauseBigtable reduces the number of nodes in your cluster wheneverpossible. This can help you avoid over-provisioning.
• Performance - Autoscaling lets Bigtable automatically addnodes to a cluster when a workload changes or there is an increase in datastorage requirements. This helps maintain workload performance objectives byensuring that the cluster has enough nodes to meet the target CPUutilization and storage requirements.
• Automation - Autoscaling reduces management complexity. You don't needto monitor and scale the cluster size manually or write an application todo these tasks, because the Bigtable service handles them foryou.

Autoscaling alone might not work well for the following workload types, becauseeven though Bigtable quickly adds nodes when traffic increases, itcan take time to balance the additional nodes.

• Bursty traffic
• Sudden batch workloads

If your spikes in usage are predictable or regularly scheduled, you can useautoscaling and adjust the settings before the planned bursts. SeeDelay whilenodes rebalance for details.

How autoscaling works

Autoscaling is the process of automaticallyscaling, or changing the sizeof, a cluster by adding or removing nodes. When you enable autoscaling,Bigtable automatically adjusts the size of your cluster for you.When your cluster's workload or storage needs fluctuate, Bigtableeitherscales up, adding nodes to the cluster, or itscales down, removingnodes from the cluster.

Bigtable autoscaling determines the number of nodes required,based on the following dimensions:

• CPU utilization target
• Storage utilization target
• Minimum number of nodes
• Maximum number of nodes

Each scaling dimension generates a recommended node count, andBigtable automatically uses the highest one. This means, forexample, that if your cluster needs 10 nodes to meet your storage utilizationtarget but 12 to meet your CPU utilization target, Bigtable scalesthe cluster to 12 nodes.

As the number of nodes changes, Bigtable continuously optimizesthe storage, rebalancing data across the nodes, to ensure that traffic is spreadevenly and no node is overloaded.

After a cluster is scaled up, Bigtable automatically rebalancesthe nodes in your cluster for optimal performance. All requests continue toreach the cluster while scaling and rebalancing are in progress. SeeScalinglimitations for more information.

If a cluster has scaled up to its maximum number of nodes and the CPUutilization target is exceeded, requests might have high latency or fail. If acluster has scaled up to its maximum number of nodes and the storageutilizationlimit is exceeded, write requests will fail. SeeStorage pernode for more details on storage limits.

When a node is added to a small cluster, such as a one-node cluster, you mightobserve a temporary increase in latency as the cluster rebalances. This isbecause the additional node proportionally doubles the size of the cluster.Similarly, if a cluster decreases in size from two nodes to one node, somelatency might occur.

When a cluster is scaled down, nodes are removed at a slower rate than whenscaling up, to prevent any impact on latency. Seescaling limitations for more details.

Autoscaling parameters

When you create or edit a cluster and choose autoscaling, you define the valuesfor CPU utilization target, min nodes, and max nodes. You can either configurethe storage utilization target or leave it at the default, which is 50%(2.5 TB for SSD and8 TB for HDD).

Configure autoscaling

This section describes how to choose your autoscaling parameters. After you setyour initial values,monitor your cluster and adjust thenumbers if necessary.

Determine the CPU utilization target

Base the CPU utilization target on your unique workload. Theoptimal target for your cluster depends on the latency and throughputrequirements of your workload. For more information, seePlan yourBigtable capacity.

In general, if you observe unacceptably high latency, you should lower the CPUutilization target.

Determine the storage utilization target

If your application is latency-sensitive, keep storage utilization below 60%.If your application is not latency-sensitive, you can choose a storageutilization target of 70% or more. For more information, seePlan your Bigtable capacity.

For autoscaling, storage utilization is expressed as the number of bytes ofstorage per node rather than as a percentage. The storage utilization target isspecified per node but is applied to the entire cluster. The capacity limits fornodes are 5 TB per node for SSD storage and16 TB per node for HDD storage.

The following table shows target amounts for typical storage utilizationtarget percentages. The Google Cloud console accepts the value in TB per node,and the gcloud CLI, API, and Cloud Bigtable client librariesaccept an integer value in GiB per node.

Tiered storage and autoscaling

Tiered storage (Preview) doesn't impact SSDautoscaling described in theDetermine the storage utilizationtarget section of this document. When youenable infrequent access as part of tiered storage, autoscaling additionallymakes sure that the combined SSD and the infrequent access storage doesn'texceed the limit of 32 TB per node. Whenthe limit is reached, Bigtable scales up automatically.

For example, on an SSD cluster, if you set a storage utilization target of2.5 TB (50%) per node, and your infrequent accessusage is high enough to push the storage usage with tiered storage over thelimit, Bigtable adds nodes. This happens even if your SSD usageremains within the 50% target.

The following table helps you understand how autoscaling recommends a node countbased on both the SSD usage and the infrequent access usage:

For more information about tiered storage, seeTiered storage overview.

Determine the maximum number of nodes

The value that you choose as the maximum number of nodes should be the number ofnodes that the cluster needs to handle your workload's heaviest traffic, even ifyou don't expect to reach that volume most of the time. Bigtablenever scales up to more nodes than it needs. You can also think of this numberas the highest number of nodes that you are willing to pay for. For details onaccepted values, seeAutoscaling parameters.

The maximum number needs to allow for both the CPU utilization target set by youand the storage utilization target set by Bigtable.

If you're changing a cluster from manual allocation to autoscaling, find thehighest number of nodes that the cluster has had over the last month or so. Yourautoscaling maximum should be at least that number.

If you are enabling autoscaling for a new cluster in an existinginstance, use metrics from other clusters in the instance as your guidepost.

If you have a new workload and you're not sure how it's going to grow, youcan estimate the number of nodes that you need to meet the built-in storageutilization target and then adjust the number later.

If you enabled tiered storage, you must account for the storage limit of32 TB per node. Even if your SSD usageis low, a large volume of data in the infrequent access tier can trigger nodescaling.

To arrive at this number, estimate the amount of data that you plan to store onthe cluster, and then divide that number by the storage utilization target forthe storage type that you use.

For example, if you store 10 TB on an SSD cluster, you can divide 10 TB by2.5 TB, which is the storage utilization target set by default for SSD clustersthat use autoscaling. The result is 4, which means that 4 is the number of nodesthat can handle that amount of data, andthe maximum should be some numberhigher than that.

Using the same formula, the following examples show the number of nodes youmight need for some sample storage amounts:

After your cluster is up and running with autoscaling enabled,monitor thecluster and make sure that the value you choose for maximum numberof nodes is at least as high as therecommended number of nodes for CPU targetand therecommended number of nodes for storage target.

Determine the minimum number of nodes

You can set the minimum as low as 1 to ensure that your Bigtablecan scale down to the smallest, most cost-efficient sizeif possible. The cluster never becomes too small because Bigtableautomatically prevents the node count from dropping below the minimum needed tomaintain the CPU and storage utilization targets. SeeAutoscalingparameters for details on accepted values.

However, in many cases you will want to set this value to more than one. Choosea higher number or raise the minimum number of nodes for the followingsituations:

• You have an upcoming peak scale event, when you expect yourtraffic to temporarily increase, and you want to make sure you have enoughcapacity.
• Your application sends spiky traffic. When new nodes are added,Bigtable automatically rebalances onto the new nodes. Becausethis process can take several minutes, you are often better off taking aconservative approach and choosing a higher minimum so that your cluster canseamlessly accommodate the spikes.
• You increase the maximum number of nodes. The minimum must always be tenpercent or less of the maximum number of nodes. For example, if you set themaximum to 30, you must set the minimum to at least 3.

If you increase the value for a cluster's minimum number of nodes,Bigtable immediately attempts to scale the cluster to the newminimum. Thestandard constraints apply,however; if a zone is out of nodes, additional nodes are not provisioned to meetthe configured minimum. Bigtable continues to attempt to add nodesand creates an audit log entry for each failed attempt until it successfullyscales the cluster to the new minimum number of nodes. Bigtabledoes not change the configured value in this situation. As a result, you mightobserve that the cluster's node count is less than the minimum untilscaling is complete.

Fine-tune your settings

After you enable autoscaling, it's important to monitor its behavior and adjustyour settings as needed. Use themonitoring metrics to observethe following:

• Recommended node counts: Pay close attention to therecommended_node_count_for_cpu andrecommended_node_count_for_storagemetrics. These metrics indicate the ideal number of nodes based on your CPUand storage utilization targets, respectively.
• Provisioned node count: Compare the recommended node counts with theactual number of nodes that are provisioned. If your cluster is consistentlyusing more nodes than recommended, you can consider reducing theminimum number of nodes to lower your costs.
• Maximum node count: Ensure that yourmaximum number ofnodes is high enough to allow your cluster to scale up tomeet demand. If you observe that your cluster is frequently hitting themaximum node count, increase it to provide more capacity.

You can alsoadjust CPU utilization target for performance. If you observelatency issues, you might need to lower theCPU utilization target to ensure that your cluster scales upmore aggressively when demand increases. Conversely, if your cluster isconsistently over-provisioned and your latency is acceptable, you can considerincreasing the CPU target.

Account for replication

In an instance that uses replication, each cluster's autoscaling settings andactivity are completely independent of those for the other clusters in theinstance. You must configure the scaling mode for each cluster in an instance.

In general, for replicated instances, you should enable autoscaling for everycluster in the instance. Your autoscaling configuration will often be the samefor every cluster in the instance, but might be different depending on the usecase, workload, and performance requirements for each cluster.

Because clusters in a replicated instance do some additional work tomanage replication, you should choose a higher maximum number of nodes for themthan you would for a single-cluster instance. To learn more, seeReplicationand performance.

Access control

To configure autoscaling, you need to be a principal in a role that hascreate andupdate permissions for the cluster and instance that you are configuring.

Monitoring

Bigtable provides several metrics to help you understand howBigtable autoscaling is working as it scales up and down to meetworkload requirements. The metrics also can help you gauge whether yoursettings are optimal to meet your business's workload and cost requirements. Forexample, if you observe that the node count for a cluster is often close to themaximum number of nodes, you might consider raising the maximum. To learn moreabout monitoring your Bigtable resources, seeMonitoring aninstance.

The following metrics aredisplayed in graphs on thecluster overview page in the Google Cloud console. You can also view thesemetrics usingCloud Monitoring.

• bigtable.googleapis.com/cluster/autoscaling/min_node_count
• bigtable.googleapis.com/cluster/autoscaling/max_node_count
• bigtable.googleapis.com/cluster/autoscaling/recommended_node_count_for_cpu
• bigtable.googleapis.com/cluster/autoscaling/recommended_node_count_for_storage

Logging

Bigtable emits a system event audit log each time it scales acluster. The log entry is similar to the following:

Grew from 9 to 10 nodes to maintain CPU utilization at 60%.

You can view autoscaling system event logs on the Bigtable clusteroverview page in the Google Cloud console. You can also view them usingLogs Explorer:

1. Navigate to the Logs Explorer:

   Go to the Logs Explorer

   Select the appropriate Google Cloud project.

2. In theQuery field, enter the following:

   resource.type="audited_resource" resource.labels.service="bigtableadmin.googleapis.com"resource.labels.method="AutoscaleCluster"

3. ClickRun query.

   TheQuery results pane displays the logs for the last hour.

To learn more about viewing logs, seeCloud Logging.

What's next

• Enable autoscaling on an existing cluster.
• Understand Bigtable performance.