This page describes the CPU utilization metrics that Spanner provides.You can view these metricsin the Google Cloud console andinthe Cloud Monitoring console.

CPU utilization and task priority

Spanner measures CPU utilization based on thesource and thepriority of the task.

• Source: A task can either be initiated by theuser or thesystem.

• Priority: Thepriority helpsSpanner determine which tasks should execute first. Thepriority ofsystem tasks is predetermined and cannot be configured.User tasks run at high priority unless otherwise specified. Many datarequests, such asreadandexecuteSql, let you specify a lower priority for the request. This can be useful,for example, when you are running batch, maintenance, or analytical queriesthat don't have strict performance SLOs.

  Higher-priority tasks are in general going to be executed ahead oflower-priority tasks. Spanner allows high-priority tasks toutilize up to 100% of the available CPU resources even if there arecompeting lower-priority tasks. While lower-priority system tasks can bedelayed in the short term, they must run eventually. Therefore, you mustprovision your instance with enough compute capacityto handle all tasks.

  If there are no high-priority tasks, Spanner will utilize upto 100% of the available CPU resources to complete lower-priority tasks morequickly. Spikes in background usage are not a sign of a problem.Lower-priority tasks can yield to higher-priority tasks, including usertasks, almost instantly.

The following table shows examples for each task:

	User tasks	System tasks
High priority	Includes data requests, such asread orexecuteSql, where either no priority orPRIORITY_HIGH is specified.	Includes data splitting.
Medium priority	Includes: Data requests wherePRIORITY_MEDIUM is specified Reads and writes issued from Dataflow jobs, includingImport andExport.	Includes: Database compaction Schema change validation The optimization phase ofdatabase restore
Low priority	Includes data requests wherePRIORITY_LOW is specified.	Includes: Backfilling an index. Backfilling a generated column

Note: Backups are not listed in this table because Spanner createsdedicated backup jobs to take backups instead of using instance CPU. For moreinformation, seeBackup time andperformance.

Available metrics

Spanner provides the following metrics for CPU utilization:

• Smoothed CPU utilization: A rolling average of total CPU utilization, asa percentage of the instance's CPU resources, for each database. Each datapoint is an average for the previous 24 hours. Use this metric to createalerts and analyze CPU usage over long period of time, for example, 24hours. You can view a chart for this metricin the Google Cloud console orin the Cloud Monitoring console asRolling average 24 hour.

• CPU Utilization by priority: The CPU utilization, as a percentage of theinstance's CPU resources, grouped by priority, user-initiated tasks andsystem-initiated tasks. Use this metric to create alerts and analyze CPUusage at a high level. You can view a chart for this metricin the Google Cloud console orin the Cloud Monitoring console.

• CPU Utilization by operation type: The CPU utilization, as a percentageof the instance's CPU resources, grouped by user-initiated operations such asreads, writes, and commits. Use this metric to get a detailed breakdown ofCPU usage and to troubleshoot further, as explained inInvestigating high CPU utilization.You can create a chart for this metricin the Cloud Monitoring console.

  You can also use the Cloud Monitoring console tocreate alerts for CPU utilization, as described later.

Alerts for high CPU utilization

The following table specifies our recommendations for maximum CPU usage forregional, dual-region, and multi-region instances.These numbers are to ensurethat your instance has enoughcompute capacityto continue to serve your traffic in the event of the loss of an entire zone(for regional instances) or an entire region (for dual-region and multi-regioninstances).

Metric	Maximum for regional instances	Maximum per region for dual-region and multi-region instances
High priority total	65%	45%
24-hour smoothed aggregate	90%	90%

To help you stay below the recommended maximums,create alerts in Cloud Monitoring that track high-priorityCPU utilization and the average CPU utilization over 24 hours.

CPU utilization can have an impact on request latencies. Overloading of anindividual backend server will trigger higher request latencies. Applicationsshould run benchmarks and active monitoring to verify thatSpanner meets their performance requirements.

Thus, for performance-sensitive applications, you may need to further reduce CPUutilization using techniques described in the following section.

CPU utilization over 100%

In certain cases the CPU utilization of a Spanner instance may reachabove 100%. This means that the instance is using more CPU resources than theamount configured for the instance.

CPU resources above 100% might be used to provide better and more predictableperformance during CPU utilization spikes, for example, caused by suddenincrease in request traffic.

Any CPU capacity above 100% is NOT guaranteed and shouldn't be relied on fornormal database operations.

Running a Spanner instance near or over 100% CPU utilization for anextended period of time has a risk of degrading normal operation performance andlatency. Additional CPU resources are not a safe mechanism to rely on forconsistent performance.

Customers are not billed for this additional CPU utilization.

Reducing CPU utilization

This section explains how to reduce an instance's CPU utilization.

In general, we recommend that you increase thecompute capacity of yourinstance as a starting point. After you increase the compute capacity, you caninvestigate and address the root causes of high CPU utilization.

Increasing compute capacity

If you exceed the recommended maximums for CPU utilization, we stronglyrecommend increasing thecompute capacity of your instance so it cancontinue to operate effectively. If you want to automate this process, you cancreate an application that monitors CPU utilization, then increases or decreasescompute capacity as needed, using theUpdateInstancemethod.

To determine how much compute capacity you need, consider the peak high-priorityCPU utilization as well as the 24-hour smoothed average. Always allocate enoughcompute capacity to keep the CPU utilization below the recommended maximums. Aspreviously described, you may need to allocate extra compute capacity forperformance-sensitive applications (for example, to accommodate workloadspikes).

If you don't have enough compute capacity, Spanner postponestasks by priority level. Low-priority system tasks, like database compaction andschema change validation, can be deferred in favor of user tasks. However, thesetasks are critical to the health of your instance, and Spannercannot defer them indefinitely. If Spanner cannot complete itslow-priority system tasks within a certain time window—on the order of severalhours to a day—due to insufficient compute resources, Spannermight increase the priority of the system tasks.This change affects theperformance of user tasks.

Investigating further with introspection tools

If theCPU Utilization by operation type metric indicates that a particulartype of operation is contributing to high CPU utilization, use theSpanner introspection tools to troubleshoot further. For moreinformation, seeInvestigating high CPU utilization.

What's next

• Monitor your instance with theGoogle Cloud consoleor theCloud Monitoring console.
• Create alerts for Spanner CPU utilization.
• Find out how tochange the compute capacity of aSpanner instance.

• Learn how tofind correlations between high latency and othermetrics.

• To learn how to troubleshoot high CPU usage caused by a particular operationtype, seeInvestigating high CPU utilization.