This page describes how to create a PromQL-based alerting policyby using the Cloud Monitoring API. You can use PromQLqueries in your alerting policies to create complex conditions withfeatures such as ratios, dynamic thresholds, and metric evaluation.

For general information, seePromQL-based alerting overview.

If you work in a Prometheus environment outside Cloud Monitoring and havePrometheus alerting rules, then you can use the Google Cloud CLI tomigrate them to PromQL-based alerting policies in Monitoring.For more information, seeMigrate alerting rules and receivers from Prometheus.

Create alerting policies with PromQL queries

You use thealertPolicies.create method toprogrammatically create alerting policies.

The only difference between creating PromQL-based alerting policiesand other alerting policies is that yourConditiontype must bePrometheusQueryLanguageCondition.This condition type allows alerting policies to be defined with PromQL.

The following shows a PromQL query for an alerting policycondition that uses a metric from thekube-state exporter to find the numberof times that a container has been restarted in the last 30 minutes:

rate(kube_pod_container_status_restarts[30m]) * 1800 > 1

Constructing the alerting policy

To build a PromQL-based alerting policy, use theAlertPolicycondition typePrometheusQueryLanguageCondition.ThePrometheusQueryLanguageCondition has the following structure:

{  "query": string,  "duration": string,  "evaluationInterval": string,  "labels": {string: string},  "ruleGroup": string,  "alertRule": string}

• query: The PromQL expression to evaluate. Equivalent to theexpr field from a standard Prometheus alerting rule.
• duration: Specifies the length of time duringwhich each evaluation of the query must generate atrue value before thecondition of the alerting policy is met. The value must be a number ofminutes, expressed in seconds; for example,600s for a 10-minute duration. For more information, seeBehavior of metric-based alerting policies.

• evaluationInterval: The interval of time, in seconds, between PromQLevaluations of the query.The default value is 30 seconds. If thePrometheusQueryLanguageCondition was created by migrating a Prometheusalerting rule, then this value comes from the Prometheus rule group thatcontained the Prometheus alerting rule.

  Note: Incidents created by a PromQL-based alerting policydon't follow the alerting policy autoclose duration. Instead,the incident is automatically closed when it reaches a set time afterthe last time the condition was met. This time is the largerof 270 seconds and twice the condition's evaluation interval.

• labels: An optional way to add or overwrite labels in the PromQLexpression result.

• ruleGroup: If the alerting policy was migrated from a Prometheusconfiguration file, then this field contains the value of thename fieldfrom therule group in the Prometheusconfiguration file. This field isn't required when you make aPromQL alerting policy in Cloud Monitoring API.

• alertRule: If the alerting policy was migrated from a Prometheusconfiguration file, then this field contains the value of thealert fieldfrom thealerting rule in the Prometheusconfiguration file. This field isn't required when you make aPromQL alerting policy in Cloud Monitoring API.

For example, the following condition uses a PromQL query tofind the number of times that a container has been restarted in the last30 minutes:

"conditionPrometheusQueryLanguage": {  "query": "rate(kube_pod_container_status_restarts[30m]) * 1800 > 1",  "duration": "600s",  evaluationInterval: "60s",  "alertRule": "ContainerRestartCount",  "labels": {    "action_required":"true",    "severity":"critical/warning/info"}}

Use this structure as the value of aconditionPrometheusQueryLanguage field ina condition, which is in turn embedded in an alerting-policy structure.For more information about these structures, seeAlertPolicy.

The following shows a complete policy with aPrometheusQueryLanguageConditioncondition in JSON:

{  "displayName": "Container Restarts",  "documentation": {    "content": "Pod ${resource.label.namespace_name}/${resource.label.pod_name} has restarted more than once during the last 30 minutes.",    "mimeType": "text/markdown",    "subject": "Container ${resource.label.container_name} in Pod ${resource.label.namespace_name}/${resource.label.pod_name} has restarted more than once during the last 30 minutes."  },  "userLabels": {},  "conditions": [    {      "displayName": "Container has restarted",      "conditionPrometheusQueryLanguage": {        "query": "rate(kubernetes_io:container_restart_count[30m]) * 1800",        "duration": "600s",        evaluationInterval: "60s",        "alertRule": "ContainerRestart",        "labels": {          "action_required":"true",          "severity":"critical/warning/info"}      }    }  ],  "combiner": "OR",  "enabled": true}

Create an alerting policy

To create the alerting policy, put the alerting policy JSON into a filecalledPOLICY_NAME.json, and then run the following command:

curl -d @POLICY_NAME.json -H "Authorization: Bearer $TOKEN"-H 'Content-Type: application/json'-X POST https://monitoring.googleapis.com/v3/projects/${PROJECT}/alertPolicies

For more information about the Monitoring API for alerting policies,seeManaging alerting policies by API.

For more information about usingcurl, seeInvokingcurl.

Disable check for metric existence

When you create a PromQL-based alerting policy, Google Cloudruns a validation to ensure that the metrics referenced in thecondition already exist in Monitoring. However, you canoverride this validation if you need to create an alerting policy before themetrics exist. For example, you might want to do so whenusing automation to create new projects that come with astandard set of predefined alerting policies.If you don't disable the validation, then your alerting policy creationfails until the underlying metrics are created.

To disable the check for metric existence, add the field"disableMetricValidation": true to yourPrometheusQueryLanguageCondition:

{  "query": string,  "duration": string,  "evaluationInterval": string,  "labels": {string: string},  "ruleGroup": string,"disableMetricValidation": true,  "alertRule": string}

If the condition of an alerting policy references a metric that doesn't exist,then the condition still runs according to its evaluation interval. However,the query result is always empty.After the underlying metric exists, the query returns data.

For instructions on configuring PromQL-based alerting policies usingTerraform, see thecondition_prometheus_query_language section of thegoogle_monitoring_alert_policy Terraformregistry.

For general information about using Google Cloud with Terraform, seeTerraform with Google Cloud.

Invokingcurl

Eachcurl invocation includes a set of arguments,followed by the URL of an API resource. The common arguments includea Google Cloud project ID and an authentication token. These valuesare represented here by thePROJECT_ID andTOKEN environment variables.

You might also have to specify other arguments, for example, to specify the typeof the HTTP request (for example,-X DELETE). The default request isGET,so the examples don't specify it.

Eachcurl invocation has this general structure:

curl --http1.1 --header "Authorization: Bearer ${TOKEN}" <other_args> https://monitoring.googleapis.com/v3/projects/${PROJECT_ID}/<request>

To usecurl, you must specify your project ID and an accesstoken. To reduce typing and errors, you can put these into environment variablesas pass them tocurl that way.

To set these variables, do the following:

1. Create an environment variable to hold the ID of yourscoping project of a metrics scope. These steps call the variablePROJECT_ID:

   PROJECT_ID=a-sample-project

2. Authenticate to the Google Cloud CLI:

   gcloud auth login

3. Optional. To avoid having to specify your project ID with eachgcloudcommand, set your project ID as the default by using gcloud CLI:

   gcloudconfigsetproject${PROJECT_ID}

4. Create an authorization token and capture it in an environment variable.These steps call the variableTOKEN:

   TOKEN=`gcloud auth print-access-token`

   You have to periodically refresh the access token. If commands that workedsuddenly report that you are unauthenticated, reissue this command.

5. To verify that you got an access token, echo theTOKEN variable:

   echo${TOKEN}ya29.GluiBj8o....