This document describes how to use the Google Cloud console to create ametric-based alertingpolicy that sends notificationswhen values of a metric are more than, or less than, the thresholdfor a specificretest window. For example, the conditionof an alerting policymight be met when the CPU utilization is higher than 80% for at least fiveminutes.

This content does not apply to log-based alerting policies. For information about log-based alerting policies, which notify you when aparticular message appears in your logs, seeMonitoring your logs.

This document doesn't describe the following:

• How to be notified when data stops arriving. For more information,seeCreate metric-absence alerting policies.

• How to be notified based on the predicted value of a metric. For moreinformation, seeCreate forecasted metric-value alerting policies.

• How to create an alerting policy by using the Cloud Monitoring API. For moreinformation, seeCreate alerting policies by using the API.

• How to create an alerting policy whose condition includes a Monitoring Query Language (MQL) query. These policies can use a static or dynamicthreshold. For more information, see the following documents:

  • Alerting policies with MQL.
  • Create dynamic severity levels using MQL.

This feature is supported only for Google Cloud projects.ForApp Hubconfigurations, select the App Hub host project or management project.

Before you begin

1. To get the permissions that you need to create and modify alerting policies by using the Google Cloud console, ask your administrator to grant you theMonitoring Editor (roles/monitoring.editor) IAM role on your project. For more information about granting roles, seeManage access to projects, folders, and organizations.

   You might also be able to get the required permissions throughcustom roles or otherpredefined roles.

   For more information about Cloud Monitoring roles,seeControl access with Identity and Access Management.

2. Ensure that you're familiar with the general concepts of alerting policies.For information about these topics, seeAlerting overview.

3. Configure the notification channels that you want to use to receive anynotifications. For redundancy purposes, we recommend that you createmultiple types of notification channels. For more information, seeCreate and manage notification channels.

Create alerting policy

To create an alerting policy that compares the value of that metric to a staticthreshold, do the following:

1. In the Google Cloud console, go to thenotifications Alerting page:

   Go toAlerting

   If you use the search bar to find this page, then select the result whose subheading isMonitoring.

2. In the toolbar of the Google Cloud console,select your Google Cloud project. ForApp Hubconfigurations, select the App Hub host project or management project.
3. SelectCreate policy.

4. Select the time series to be monitored:

   1. ClickSelect a metric, navigate through the menus to select aresource type and metric type, and then clickApply.

      TheSelect a metric menu contains features that help you findthe metric types available:

      • To find a specific metric type, use thefilter_listFilter bar.For example, if you by enterutil, then you restrict the menu toshow entries that includeutil. Entries are shown when they passa case-insensitive "contains" test.
      • To show all metric types, even those without data, clickdone Active. By default, the menusonly show metric types with data. For more information, seeMetric not listed in menu.

      You can monitor anybuilt-in metric or any user-definedmetric.

   2. Optional: To monitor a subset of the time series that match themetric and resource types you selected in the previous step,clickAdd filter. In the filter dialog, select the label bywhich to filter, a comparator, and then the filter value.For example, the filterzone =~ ^us.*.a$ uses a regular expression tomatch all time-series data whose zone name starts withus and endswitha. For more information, seeFilter the selected time series.

   3. Optional: To change how the points in a time series are aligned,in theTransform data section, set theRolling window andRolling window function fields.

      If you are monitoring a log-based metric, then we recommend that theRolling window menu is set to at least 10 minutes.

      These fields specify how the points that are recorded in a window arecombined. For example, assume that the window is 15 minutes and thewindow function ismax. The aligned point is the maximum value of allpoints in the most recent 15 minutes.For more information, seeAlignment: within-series regularization.

      You can also monitor the rate at which a metric value changesby using theRolling window function field topercent change.For more information, seeMonitor a rate of change.

   4. Optional: Combine time series when you want to reduce the number oftime series monitored by a policy, or when you want to monitor onlya collection of time series. For example, instead of monitoring theCPU utilization of each VM instance, you might want to compute theaverage of the CPU utilization for all VMs in a zone, and thenmonitor that average. By default,time series aren't combined. For general information, seeReduction: combining time series.

      To combine all time series, do the following:

      1. In theAcross time series section,clickexpand_moreExpand.
      2. Set theTime series aggregation field to a value other thannone. For example, to display the average value of thetime series, selectmean.
      3. Ensure that theTime series group by field is empty.

      To combine, or group, time series by label values, do the following:

      1. In theAcross time series section,clickexpand_moreExpand.
      2. Set theTime series aggregation field to a value other thannone.
      3. In theTime series group by field, select the labelsby which to group.

      For example, if you group by thezone label and then set theaggregation field to a value ofmean, then the chart displaysone time series for each zone for which there is data. Thetime series shown for a specific zone is the average of all time serieswith that zone.

      Note: To delete a grouping, clear theTime series group by fieldand set theTime series aggregation field tonone.

   5. ClickNext.

5. Configure the condition trigger:

   1. Leave theCondition type field at the default value ofThreshold.

   2. Optional: Update theAlert trigger menu, whichhas the following values:

      • Any time series violates: Default setting. Any time seriesviolating the threshold for the entire retest windowcauses the condition to be met.

      • Percent of time series violates: A percentage of time seriesmust violate the threshold for the entire retest windowbefore the condition is met. For example,you could be notified when 50% of the monitored time series violatethe threshold for the entire retest window.

      • Number of time series violates: A specific number of time seriesmust violate the threshold for the entire retest windowbefore the condition is met. Forexample, you could be notified when 32 of the monitoredtime series violate the threshold for the entire retest window.

      • All time series violate: All time series must violate thethreshold for the entire retest window before the condition is met.

      For information about the intervals that Monitoringuses to align and measure time series data, seeAlignment periods and retest windows.

   3. Enter when the value of a metric violates the threshold by using theThreshold position andThreshold value fields. For example,if you set these values toAbove threshold and0.3, then anymeasurement higher than0.3 violates the threshold.

   4. Optional: To select how long measurements must violate the thresholdbefore Monitoring sends a notification, expandAdvanced options and then use theRetest window menu.

      The default value isNo retest. With this setting, a singlemeasurement can result in a notification. For more information andan example, seeAlignment period and duration settings.

   5. Optional: To specify how Monitoring evaluates thecondition when data stops arriving, expandAdvanced options, and then use theEvaluation missing datamenu.

      TheEvaluation missing data menu is disabled when the value of theRetest window isNo retest.

      Google Cloud console "Evaluation of missing data" field	Summary	Details
      Missing data empty	Open incidents stay open. New incidents aren't opened.	For conditions that are met, the condition continues to be met when data stops arriving. If an incident is open for this condition, then the incident stays open. When an incident is open and no data arrives, the auto-close timer starts after a delay of at least 15 minutes. If the timer expires, then the incident is closed. For conditions that aren't met, the condition continues to not be met when data stops arriving.
      Missing data points treated as values that violate the policy condition	Open incidents stay open. New incidents can be opened.	For conditions that are met, the condition continues to be met when data stops arriving. If an incident is open for this condition, then the incident stays open. When an incident is open and no data arrives for the auto-close duration plus 24 hours, the incident is closed. For conditions that aren't met, this setting causes the metric-threshold condition to behave like ametric-absence condition. If data doesn't arrive in the time specified by the retest window, then the condition is evaluated as met. For an alerting policy with one condition, the condition being met results in an incident being opened.
      Missing data points treated as values that don't violate the policy condition	Open incidents are closed. New incidents aren't opened.	For conditions that are met, the condition stops being met when data stops arriving. If an incident is open for this condition, then the incident is closed. For conditions that aren't met, the condition continues to not be met when data stops arriving.

   6. ClickNext.

6. Optional: Create an alerting policy with multiple conditions.

   Most policies monitor a single metric type, for example, a policy mightmonitor the number of bytes written to a VM instance. When you wantto monitor multiple metric types, create a policy with multiple conditions.Each condition monitors one metric type. After you create the conditions,you specify how the conditions are combined. For information, seePolicies with multiple conditions.

   To create an alerting policy with multiple conditions, do the following:

   1. For each additional condition, clickAdd alert condition and thenconfigure that condition.
   2. ClickNext and configure how conditions are combined.
   3. ClickNext to advance to the notifications and documentationset up.

7. Configure the notification and add user labels:

   1. Expand theNotifications and name menu and select your notificationchannels. For redundancy purposes, we recommend that you add to analerting policy multiple types of notification channels.For more information, seeManage notification channels.

   2. Optional: To use a custom subject line in your notification instead ofthe default, update theNotification subject line field.

   3. Optional: To be notified when an incident is closed, selectNotify on incident closure. By default, when you create an alerting policy with the Google Cloud console, a notification is sent only when an incident is created.

   4. Optional: To change how long Monitoring waits beforeclosing an incident after data stops arriving, select an optionfrom theIncident autoclose duration menu. By default, when datastops arriving, Monitoring waitsseven days before closing an openincident.

   5. Optional: To associate your alerting policy with anApp Hub application, in theApplication labelssection, select an application and either a service or workload.Incidents and notifications display these labels.

   6. Optional: Select an option from thePolicy severity level menu.Incidents and notifications display the severity level.

   7. Optional: To add custom labels to the alerting policy, in thePolicy user labels section, do the following:

      1. ClickAdd label, and in theKey field enter a name for thelabel. Label names must start with a lowercase letter, and they cancontain lowercase letters, numerals, underscores, and dashes.For example, enterseverity.
      2. ClickValue and enter a value for your label. Label values cancontain lowercase letters, numerals, underscores, and dashes.For example, entercritical.

      For information about how you can use policy labels to help you manageyour notifications, seeAnnotate incidents with labels.

8. Optional: In theDocumentation section, enter any content that you wantincluded with the notification.

   To format your documentation, you can use plain text,Markdown, andvariables. You can alsoinclude links to help users debug the incident, suchas links to internal playbooks, Google Cloud dashboards, and externalpages. For example, the following documentation template describes a CPUutilization incident for agce_instance resource and includesseveral variables to reference the alerting policy and condition RESTresources. The documentation templatethen directs readers to external pages to help with debugging.

   When notifications are created, Monitoring replacesthe documentation variables with their values.The values replace the variables only in notifications. Thepreview pane and other places in the Google Cloud consoleshow only the Markdown formatting.

   Preview

   ## CPU utilization exceeded### SummaryThe ${metric.display_name} of the ${resource.type}${resource.label.instance_id} in the project ${resource.project} hasexceeded 90% for over 15 minutes.### Additional resource informationCondition resource name: ${condition.name}  Alerting policy resource name: ${policy.name}  ### Troubleshooting and Debug ReferencesRepository with debug scripts: example.com  Internal troubleshooting guide: example.com  ${resource.type} dashboard: example.com

   Format in notification

   

   For more information, seeAnnotate notifications with user-defined documentationandUsing channel controls.

9. ClickAlert name and enter a name for the alerting policy.

10. ClickCreate policy.

Filter the selected time series

Filters ensure that only time series that meet some set of criteria aremonitored.When you apply filters, you might reduce the number of lines on the chart,which can improve the performance of the chart. You can also reduce theamount of data being monitored by applying aggregation.Filters ensure that only time series that meet some set of criteria are used.When you apply filters, there are fewer time series to evaluate, which canimprove the performance of the alert.

A filter is composed of a label, a comparator, and a value. For example,to match all time series whosezone label starts with"us-central1", youcould use the filterzone=~"us-central1.*", which uses a regular expressionto perform the comparison.

When you filter by the project ID or the resource container,you must use the equals operator,(=). When you filter byother labels, you can use any supported comparator.Typically, you can filter metric and resource labels, and byresource group.

When you supply multiple filtering criteria,only the time series that meetall criteria are monitored.

To add a filter, clickAdd filter, complete the dialog, and then clickDone. In the dialog, you use theFilter field to select thecriterion by which to filter, select the comparison operator, and thenselect or input the value. The drop-down menu only lists values that appear overthe last week, but you can input any value. Each row in the following tablelists a comparison operator, its meaning, and an example:

Troubleshoot

This section contains troubleshooting tips.

Metric not listed in menu of available metrics

To monitor a metric that isn't listed in theSelect a metric menu, doone of the following:

• To create an alerting policy that monitors a Google Cloud metric,expand theSelect a metric menu and thenclickdone Active.When disabled, the menu lists all metrics forGoogle Cloud services, and all metrics with data.

• To configure a condition for a custom metric type before that metrictype generates data, you must specify the metric type by using aMonitoring filter:

  1. Select? on theSelect metric section header and then selectDirect filter mode in the tooltip.

  2. Enter a monitoring filter or a time series selector.For information about syntax, see the following documents:

     • Monitoring filters
     • Retrieving SLO data
     • Process-health filters

Monitor a rate of change

To monitor the rate of change of a metric value, set theRolling window function field topercent change.When the condition is evaluated, Monitoring calculates therate of change of the metric as a percentage, and then compares that percentageto the condition's threshold. This comparison process follows two steps:

1. If the time series has aDELTA orCUMULATIVE metric kind, thenthe time series is converted to one that has aGAUGE metric kind.For information about theconversion, seeKinds, types, and conversions.
2. Monitoring computes the percentage changed by comparingthe average value in the most recent 10-minute sliding window to theaverage value from the 10-minute sliding window before the beginning ofthe alignment period.

You can't change the 10-minute sliding window used for comparisons ina rate-of-change alerting policy.However, you do specify the alignment period when you create the condition.

What's next

• To create a policy that compares the value of a time series to a dynamicthreshold, you must use MQL. For more information,seeCreate dynamic severity levels using MQL.

• The instructions on this page apply to any alerting policy. The followingdocuments provide guidance for specific configurations:

  • Monitor a ratio of metric values
  • Monitor a dynamic resource group
  • Monitor the count of processes on a VM
  • Monitor a microservice
  • Monitor your logs