This document describes how you can use Cloud Monitoring to observe trendsin your log data and to notify you when conditions that you specify occur.To provide Cloud Monitoring with data from your log data,Logging supports the following features:

• You can generate custom metrics from your log entries. These metrics arecalledlog-based-metrics. You can also createmetric-based alertingpolicies to notify you when a log-based metric meets a condition.For more information, seeVisualize log entry data with log-based metrics.

• You can use alerting policies to monitor, in near real time, when a messageappears in your log entries. These alerting policies are calledlog-based alerting policies. For more information, seeMonitor individual log entries for messages.

• You can write a SQL query in Log Analytics and create an alerting policy thatmonitors the results of the query. These alerting policies are calledSQL-based alerting policies. For more information, seeMonitor SQL query results.

  SQL-based alerting policies is in Public Preview.

The rest of this document describes the differences between these threealerting policies, and provides information about authorization, costs, andlimits.

Visualize log entry data with log-based metrics

When you want to monitor recurring events in your log data over time, uselog-based metrics. Log-based metrics generate numeric data from your log dataand they are suitable when you want to do any of the following:

• Count the occurrences of a message, like a warning or error, in yourlog data and receive a notification when the number of occurrences crossesa threshold.
• Observe trends in your data, like latency values in yourlog data, and receive a notification when the values change in an unacceptableway.
• Create charts to display the extracted numeric data.

Because log-based metrics generate numeric data from your log data, you can usethese metrics in alerting policies and display them in charts.For information about creating alerting policies and charts for log-basedmetrics, seeConfigure notifications for log-based metrics.

Cloud Monitoring provides a set of predefined log-based metrics, and youcan define your own. To see a list of the system-defined log-based metrics,click the followingadd_circle button:

User-defined log-based metrics

You can create log-based metrics to extract numeric data from yourlog data. User-defined log-based metrics calculate values from both includedand excluded log entries.

By default, user-defined log-based metrics collect data from all log entriesreceived by the Log Router in your Google Cloud project, but you candefine log-based metrics that collect data from log entries routed to a specificlog bucket.

• For information about defining and using project-level log-based metrics, seeUsing log-based metrics.
• For information about defining and using bucket-level log-based metrics, seeLog-based metrics on log buckets.

If you define your own log-based metrics,you might incur charges. For more information about costs associated withmetric ingestion, seeGoogle Cloud Observability pricing.

Monitor individual log entries for messages

When you want to be notified anytime a specific message occurs in a log entry,use log-based alerting policies. Log-based alerting policiesare useful for catching security-related events in log entries,like the following:

• You want to be notified when an event appears in an audit log; for example,a human user accesses the security key of a service account.
• Your application writes deployment messages to logs, and you want to benotified when a deployment change is logged.

Log-based alerting policies are useful for events that you expect to beboth rare and important. You don't want to know about a trend or pattern;you want to know that something occurred.

Log-based alerting policies are defined in a project, and they scan a log entrywhen the following conditions are true:

• Billing is enabled.
• The log entry originates in a project.

• Sinks in the project where the log entry originates, or in a projectto which the log entry is routed, route the log entry to a log bucket.

  For example, assume a log entry originates in projectA. If a log sinkin projectA routes the log entry to a log bucket, thenlog-based alerting policies defined in projectA scan the log entry.

  As a second example, assume a log entry originates in projectX and thata log sink in projectX routes the log entry to projectY. If a sink inprojectY routes the log entry to a log bucket, thenlog-based alerting policies defined in projectX and in projectYscan the log entry.

You can't use log-based alerting policies to monitor log entries that originatein folders, billing accounts, or organizations, or to monitor log entries thataren't stored in log buckets. If you create aggregated sinks, then these sinksmight intercept a log entry and prevent it from being routed by thesinks in the project where the log entry originates.

For information about creating log-based alerting policies, seeConfigure log-based alerting policies.

Monitor SQL query results

You can configure alerting policies that use Log Analytics to run SQLqueries on your log entry data. These types of alerting policiesare effective when you want to get notifiedbased on patterns that can't be evaluated by log-based alerting policies,such as complex patterns in log entriesor aggregations of log data. For more information, seeMonitor your SQL query results with an alerting policy.

Comparison of alerting options

This section compares alerting policies built on log-based metrics,log-based alerting policies, and SQL-based alerting policies.

Summary table

The following table summarizes the alerting techniques and provideslinks to additional information:

Available log entries

User-defined log-based metrics are calculated from all log entriesreceivedby the Logging API for the Google Cloud project, regardless of anyinclusion filters orexclusion filters that may apply tothe Google Cloud project. If you create an alerting policy based on auser-defined log-based metric, then the policy monitors data from alllog entries.

System-defined log-based metrics are calculated only fromlog entries that have been stored in log buckets in theGoogle Cloud project. If a log has been explicitlyexcluded, then it isn't included in these metrics.If you createan alerting policy based on a system-defined log-based metric, then the policymonitors data only from included log entries.

Log-based alerting policies are defined in a project, and they scan a log entrywhen the following conditions are true:

• Billing is enabled.
• The log entry originates in a project.

• Sinks in the project where the log entry originates, or in a projectto which the log entry is routed, route the log entry to a log bucket.

  For example, assume a log entry originates in projectA. If a log sinkin projectA routes the log entry to a log bucket, thenlog-based alerting policies defined in projectA scan the log entry.

  As a second example, assume a log entry originates in projectX and thata log sink in projectX routes the log entry to projectY. If a sink inprojectY routes the log entry to a log bucket, thenlog-based alerting policies defined in projectX and in projectYscan the log entry.

You can't use log-based alerting policies to monitor log entries that originatein folders, billing accounts, or organizations, or to monitor log entries thataren't stored in log buckets. If you create aggregated sinks, then these sinksmight intercept a log entry and prevent it from being routed by thesinks in the project where the log entry originates.

SQL-based alerting policies query log views on log buckets.These log buckets must beupgraded to use Log Analyticsand thenlinked to a BigQuery dataset.For more information about SQL-based alerting policies, seeMonitor your SQL query results with an alerting policy.

Monitor metrics across multiple projects

You can monitor metrics from several projects by configuring ametrics scope. Ametrics scope lists all the projects andaccounts that it monitors. Ascoping project hosts the metrics scope.The scoping project stores the alerting policies and other configurationsthat you create for the metrics scope. The scoping project for ametrics scope is the project selected by the Google Cloud console project picker.

Alerting policies based on log-based metrics, like alerting policies basedon other metrics, work across all projectsin the metrics scope of the scoping project.

The metrics scope isn't relevant to alerting policies based on log entries,like log-based and SQL-based policies.

For more information about metrics scopes, including multi-projectmetrics scopes, and about scoping projects, see the following:

• Metrics scopes
• View metrics for multiple projects

Incidents and notifications

When the condition of an alerting policy is met, Monitoringopens an incident and sends notifications to the notification channels ofthe alerting policy. To see the details of the incident, clickView incident in the notification message, or navigate directly totheIncidents page in Monitoring.

Incidents for metric-based alerting policies

Alerting policies based on log-based metrics create incident andnotifications like all other metric-based alerting policies inMonitoring, as described inAlerting behavior. For more informationabout managing incidents for metric-based alerting policies,seeIncidents for metric-based alerting policies.

Incidents for log-based alerting policies

Log-based alerting policies aren't metric-based alerting policies.When a log entry meets the condition of a log-based alerting policy,Monitoring creates incidents and notifications as follows:

• The first time Cloud Logging writes a log entry that matches your alertingpolicy's query on a log bucket, an incident is created and anotification is sent. If another matching log entry is then written, then anew incident is created only if the previous incident has beenclosed. However, it might take up to three minutes for a closedincident to be purged. When a matching log entry is received in thethree minutes after you closed an incident, the systemmight reopen that incident rather than creating a new incident.

• When you create a log-based alerting policy, you can specify the minimum timebetween notifications. For example, youselect 10 minutes as the time between notifications. If the condition of yourlog-based alerting policy is met twice within that period,then you receive only one notification.

  The maximum rate of incidents for log-based alerting policies is1 incident every 5 minutes.If the query of your log-based alerting policy extracts label values,then each combination of extracted valuesrepresents its own incident timeline. For example, assume a log-basedalerting policy extracts the values of a label and that the label can have twovalues. With this configuration, two incidents can be opened, one foreach label value, in the same 5 minutes.

• There is a limit of 20 incidents a day for each log-based alerting policy. If you reach thislimit, then your notification includes a message that you have reached thelimit for the day.

• Incidents automatically close when the auto-close duration expires.By default, the auto-close duration is 7 days.

  The auto-close duration specifies the time that must elapse, without a repeatof the cause of the incident, before the incident closes. For thisreason, when an incident is open and its cause reoccurs, theincident can stay open longer than the auto-close duration.

For more information about managing incidents for log-based alertingpolicies, seeManage incidents for log-based alerting policies.

Incidents for SQL-based alerting policies

For SQL-based alerting policies, Cloud Monitoringcreates an incident the first time the result ofthe SQL query meets the condition specified in the policy. Eachalerting policy has only one open incident. While the incident isopen, if the condition is met again, Monitoring doesn't createfurther incidents or send additional notifications.Monitoring closes SQL-based incidents after seven days,unless you configure a shorter incident closure period or close theincident on your own.

For more information about managing incidents for SQL-based alertingpolicies, seeManage incidents for SQL-based alerting policies.

Creating and managing alerting policies

You create, modify, and delete alerting policies based on log-based metricsin Cloud Monitoring, like any other metric-based alerting policy.For more information, seeManaging policies.

You can create log-based alerting policies by using the Logs Explorer or theCloud Monitoring API. You modify and delete log-based alerting policies inMonitoring or with the Cloud Monitoring API.For more information, seeManaging log-based alerting policies.

You can create SQL-based alerting policies by usingLog Analytics or the Cloud Monitoring API. You modify and delete SQL-based alertingpolicies in Monitoring or byusing the Cloud Monitoring API. For more information, seeMonitor your SQL query results with an alerting policy.

Viewing alerting policies

ThePolicies page in Monitoring lists all thealerting policies in your Google Cloud project. This list includes policies thatuse log-based metrics and log-based alerting policies.

1. In the Google Cloud console, go to thenotifications Alerting page:

   Go toAlerting

   If you use the search bar to find this page, then select the result whose subheading isMonitoring.

2. SelectSee all policies.

Log-based alerting policies appear in the list with the valueLogs in theType column. Alerting policies based on metrics, including log-basedmetrics, appear in the list with the valueMetrics in theType column.SQL-based alerting policies appear in thelist with the valueSQL in theTypes column.The following screenshot shows an excerpt of a policy list:

Notification channels

You can send notifications from any type of alerting policy toany of the notification channels supported by Monitoring. Youmust configure these channels before you can use them in alerting policies.

For more information, seeManaging notificationchannels.

Authorization requirements

Using log-based metrics or log-based alerting policies requires authorizationfor both Cloud Logging and Cloud Monitoring.

• For user-defined log-based metrics, seePermissions for log-based metrics.

• For log-based alerting policies, seePermissions for log-based alerting policies.

• For SQL-based alerting policies, seePermissions for SQL-based alerting policies.

Costs and limits

If you define your own log-based metrics, then the following apply:

• There are limits on the number and structure of user-defined log-basedmetrics. For more information about these limits, seelimits forlog-based metrics.
• You might incur charges for user-defined log-based metrics.For more information about costs associated with metric ingestion, seeGoogle Cloud Observability pricing.
• SQL-based alerting policies run in a BigQueryreservation in your Google Cloud project. You might incur charges for having aBigQuery reservation. For more information about costsassociated with BigQuery reservations, seeBigQuery pricing.

There are no charges associated with using alerting policies based onlog-based metrics.

The following Monitoring limits related to alerting policiesapply: