This document describes aggregated sinks, which let you collate and routelog entries that originate in resources within a folder or organization toa supported destination. We recommend that you use aggregated sinksto route your log data to a central storage location.

Note: If your data is managed through anAssured Workloads environment,then this feature might be impacted or restricted. For information, seeRestrictions and limitations in Assured Workloads.

About aggregated sinks

An aggregated sink is similar to a project-levelsinkin that an aggregated sink contains filters and a destination. However,the Log Router sends to an aggregated sink the following log entries:

• All log entries that originate in a folder or organization.
• All log entries that originate in the child resources of the folder ororganization.

For example, if you create a folder-level aggregated sink,then the Log Router sends to that sink all log entries thatoriginate in the folder or in child resources of the folder.

When aggregated sinks exist in theresource hierarchy ofa log entry, the Log Router initially sends the log entry to those sinks.Because aggregated sinks can be intercepting or non-intercepting, theLog Router might not send a log entry that is routed by an aggregated sinksent to the project-level sinks.

Intercepting aggregated sink

An intercepting aggregated sink prevents log entries from being routedto sinks in child resources, except for the_Required sinks in theresources where the log entries originate. An intercepting aggregated sinkcan be useful in preventing duplicate copies of log entries from being storedin multiple places.

For example, suppose that you need to enableData Access audit logs for auditing purposes. To simplifyyour analysis, you want to store these logs in a central location. However,for security and cost reasons, you also want to prevent these logs from beingstored at the project level. For this scenario, you can create anintercepting aggregated sink.

Non-intercepting aggregated sink

A non-intercepting aggregated sink doesn't affect how log entries are routedto other sinks. That is, even when a log entry matches the filter of anon-intercepting aggregated sink, that log entry is then forwarded toother sinks in the log entry's resource hierarchy.A non-intercepting aggregated sink lets you maintain visibility oflog entries in the resources in which they were generated.

For example, you might create a non-intercepting aggregated sink that routesall log entries generated from the folders contained by an organizationto a central log bucket. The log entries are stored in the centrallog bucket. However, because the sink is non-intercepting, the Log Routeralso sends log entries to the log sinks in the resource inwhich they were generated.

Routing examples

This section illustrates how a log entry that originates in a projectmight flow through the sinks in its resource hierarchy.

Example: No aggregated sinks exist

When no aggregated sinks exist in the resource hierarchy of the log entry,the log entry is sent to the log sinks in the project where the log entryoriginates. A project-level sink routes the log entry to the sink'sdestination when the log entry matches the sink's inclusion filter but doesn'tmatch any of the sink's exclusion filters.

Example: A non-intercepting aggregated sink exists

Assume that a non-intercepting aggregated sink exists in the resourcehierarchy for a log entry. After the Log Router sends the log entry tothe non-intercepting aggregated sink, the following occurs:

1. The non-intercepting aggregated sink routes the log entry to the sink'sdestination when the log entry matches the inclusion filter but doesn'tmatch any exclusion filter.

2. The Log Router sends the log entry to the log sinks in the projectwhere the log entry originated.

   A project-level sink routes the log entry to the sink'sdestination when the log entry matches the sink's inclusion filter but doesn'tmatch any of the sink's exclusion filters.

Example: An intercepting aggregated sink exists

Assume that an intercepting aggregated sink exists in the resourcehierarchy for a log entry. After the Log Router sends the log entry tothe intercepting aggregated sink, one of the following occurs:

• The log entry matches the inclusion filter but doesn't matchany exclusion filter:

  1. The log entry is routed to the destination of theintercepting aggregated sink.
  2. The log entry is sent to the_Required sink in the project where thelog entry originated.

• The log entry doesn't match the inclusion filter or it matchesat least one exclusion filter:

  1. The log entry isn't routed by the intercepting aggregated sink.

  2. The Log Router sends the log entry to the log sinks in the projectwhere the log entry originated.

     A project-level sink routes the log entry to the sink'sdestination when the log entry matches the sink's inclusion filter but doesn'tmatch any of the sink's exclusion filters.

Supported destinations for aggregated sinks

This section lists which destinations are supported for aggregated sinks.

Intercepting sinks

The destination of an intercepting aggregated sink must be aGoogle Cloud project.

The log sinks in the destination project reroute the log entries to theirdestinations. All destinations except projectsare supported. For example, the log sinksin the destination project might reroute log entries to a log bucket.

Non-intercepting sinks

The destination of an non-intercepting aggregated sink can be any of thefollowing:

Note: To use the visualization and analysis tools of Cloud Logging or to useError Reporting, you must store your log entries in log buckets.These log buckets don't have to be in the same resource where thelog entries originate. For example, you might configure an aggregated sink toroute log entries to a Google Cloud project, and then configure the sinksin that project to reroute the log entries to local log buckets.

The destination of a sink can be in a different resource than the sink.For example, you can use a log sink to route log entries from one project to alog bucket stored in a different project.

The following destinations are supported:

Google Cloud project

Select this destination when you want the log sinks in thedestination project to reroute your log entries, or when you have createdan intercepting aggregated sink. The log sinks in the project that is thesink destination can reroute the log entries to any supported destinationexcept a project.

Note: This is the only type of destination where log entries are rerouted.For example, if you route log entries from one project to a log bucket inanother project, then those log entries aren't rerouted by the log sinksin the project that stores the log bucket.

Log bucket

Select this destination when you want to store your log data inresources managed by Cloud Logging. Log data stored in log bucketscan be viewed and analyzed using services like the Logs Explorerand Log Analytics.

If you want to join your log data with other business data, then youcan store your log data in a log bucket and create a linkedBigQuery dataset. A linked dataset is a read-only datasetthat can be queried like any other BigQuery dataset.

BigQuery dataset

Select this destination when you want to join your log data withother business data. The dataset you specify must be write-enabled.Don't set the destination of a sink to be a linkedBigQuery dataset. Linked datasets are read-only.

Cloud Storage bucket

Select this destination when you want long-term storage of your log data.The Cloud Storage bucket can be in the same project in which log entriesoriginate, or in a different project. Log entries are stored as JSON files.

Pub/Sub topic

Select this destination when you want to export your log data fromGoogle Cloud and then use third-party integrations like Splunk or Datadog.Log entries are formatted into JSON and then routed toa Pub/Sub topic.

Best practices

We recommend that the destination of an aggregated sink be a Google Cloud project.With this destination, the log sinks in the destination Google Cloud projectreroute the log entries.The_Required sink only routes log entries that match its filter and thatoriginate in the resource where the sink is defined. Therefore, if you wantto store additional copies of log entries that match the filter of the_Required sink, then you must create a custom log sink or modify the filterof the_Default log sink.

When you create an intercepting sink, we recommend that you do the following:

• Consider whether child resources need independent control of routing theirlog entries. If a child resource needs independent control of certainlog entries, the verify that your intercepting sink doesn't route thoselog entries.

• Add contact information to the description of an intercepting sink. Thismight be helpful if those who manage the intercepting sink are differentfrom those who manage the projects whose log entries are being intercepted.

• Test your sink configuration by first creating a non-intercepting aggregatedsink to verify that the correct log entries are being routed.

Intercepting aggregated sinks and log-based metrics

Log-based metrics areCloud Monitoring metricsthat are derived from the content of log entries. The way a log entry is routeddetermines which log-based metrics that log entry can count toward.Because an intercepting aggregated sink affects how log entries are routed,creating this type of sink can result in changes to the values of existinglog-based metrics.

For more information, seeHow routing log entries affects log-based metrics.

Aggregated sinks and VPC Service Controls

The following limitations apply when you use aggregated sinks andVPC Service Controls:

• Aggregated sinks can access data from projects inside a serviceperimeter. To restrict aggregated sinks from accessing data inside aperimeter, we recommend using IAM to manageLogging permissions.

• VPC Service Controls doesn't support adding folder ororganization resources to service perimeters. Therefore, you can't useVPC Service Controls to protect folder- and organization-level logs,including aggregate logs. To manage Loggingpermissions at the folder or organizational level, we recommend usingIAM.

• If you route logs by using a folder- or organization-level sink to aresource that a service perimeter protects, then you must add aningress rule to the service perimeter. The ingress rule must allow accessto the resource from the service account that the aggregated sink uses.For more information, refer to the following pages:

  • Ingress and egress rules
  • Managing service perimeters

• When you specify an ingress or egress policy for a service perimeter,you can't useANY_SERVICE_ACCOUNT andANY_USER_ACCOUNT as an identity type when you use a log sinkto route logs toCloud Storage resources.However, you can useANY_IDENTITY as the identity type.

What's next

• To learn how to create an aggregated sink, seeCollate and route organization- and folder-level logs to supported destinations

• For a tutorial, seeAggregate and store your organization's logs.

• For information about managing existing sinks, seeRoute logs to supported destinations: Manage sinks.

• To learn how to view your logs in their destinations, as well as howthe logs are formatted and organized, seeView logs in sink destinations