This document describes how to manage the log entries generated by the resourcescontained in your Google Cloud organization by using a non-interceptingaggregated sink.

You can configure an aggregated sink to be intercepting or non-intercepting,depending on whether you want control over which log entries can be queried in,or routed through the sinks in child resources. In this tutorial, you createan aggregated sink that routes your organization's audit logs to aGoogle Cloud project, which then routes theaggregated audit logs to a log bucket. For more information, seeAggregated sinks overview.

In this tutorial, you perform the following steps:

1. You start by creating a log bucket and a log sink in theGoogle Cloud project where you want to store your aggregated log entries.

2. Next, you create a non-intercepting aggregated sink at the organization levelto route log entries to the Google Cloud project that contains thelog bucket.

3. You then configure read access to log views on the new log bucket.

4. Lastly, you query and view your log entries from the Logs Explorer page.

Before you begin

Ensure the following:

• To get the permissions that you need to configure an aggregated sink, ask your administrator to grant you the following IAM roles on your organization:

  • To create log buckets and sinks in a project: Logs Configuration Writer (roles/logging.configWriter) - your project
  • To create an aggregated sink: Logs Configuration Writer (roles/logging.configWriter) - your organization
  • To grant roles to principals: Owner (roles/owner) - your project

  For more information about granting roles, seeManage access to projects, folders, and organizations.

  You might also be able to get the required permissions throughcustom roles or otherpredefined roles.

• If you use VPC Service Controls, then you must add an ingress rule to the service perimeter. For more information about VPC Service Controls limitations, seeLimitations.

• In the Google Cloud console, activate Cloud Shell.

  Activate Cloud Shell

  At the bottom of the Google Cloud console, aCloud Shell session starts and displays a command-line prompt. Cloud Shell is a shell environment with the Google Cloud CLI already installed and with values already set for your current project. It can take a few seconds for the session to initialize.

Create a log bucket

Log buckets store the log entries that are routed from otherGoogle Cloud projects, folders, or organizations. For more information, seeConfigure log buckets.

To create the log bucket in the Google Cloud project that you want toaggregate log entries into, complete the following steps:

1. Open a shell.

   For example, to use the Cloud Shell, do the following:

   1. Go to the Google Cloud console:

      Go to Google Cloud console

   2. In the toolbar, clickterminal Activate Cloud Shell.

2. In the shell, run thegcloud logging buckets create command.

   Before running the following command, make the followingreplacements:

   • BUCKET_NAME: The name of the log bucket.
   • LOCATION: The location of the log bucket. After you create your log bucket,you can't change its location.
   • PROJECT_ID: The identifier of the project in which to create thelog bucket.

   Execute thegcloud logging buckets createcommand:

   gcloudloggingbucketscreateBUCKET_NAME\--location=LOCATION--project=PROJECT_ID

3. Verify that the log bucket was created:

   gcloudloggingbucketslist--project=PROJECT_ID

   The response of the command is a list of the log buckets in yourproject.

4. Log buckets have configurable retention periods. If you want to setthe retention period of the log entries in your log bucket, then usethegcloud logging buckets update command. For example, the followingcommand extends the retention of log entries stored in thelog bucket to 365 days:

   gcloud logging buckets updateBUCKET_NAME \   --location=LOCATION --project=PROJECT_ID \   --retention-days=365

   For more information about options, seegcloud logging buckets update.

Create the project-level log sink

You route log entries to a log bucket by creating a sink. A sink includes aninclusion filter, exclusion filters, and a destination. In thistutorial, you configure an inclusion filter and the destination toyour new log bucket. Your sink doesn't contain any exclusion filters.For more information about sinks, seeRoute logs to supported destinations.

To create a sink that routes log entries to the log bucket you just created,run thegcloud logging sinks create command.

Before running the following command, make the followingreplacements:

• PROJECT_LEVEL_SINK_NAME: The name of the project-levellog sink.

• SINK_DESTINATION: The log bucket where your log entries are routed. Thedestination path format for a log bucket is the following:

  logging.googleapis.com/projects/PROJECT_ID/locations/LOCATION/buckets/BUCKET_NAME

• PROJECT_ID: The identifier of the project in which to create thelog sink. Set this option to the same project where you created the logbucket.

• Include the following options:

  • --log-filter : Use this option to set afilter that matches the logentries you want to include in your sink. In this tutorial, the filter isset to select all audit log entries. If you don't set a filter, then alllog entries from your Google Cloud project are routed to the destination.
  • --description: Use this option to describe the purpose or use case forthe sink.

Execute thegcloud logging sinks createcommand:

gcloud logging sinks createPROJECT_LEVEL_SINK_NAMESINK_DESTINATION--project=PROJECT_ID--log-filter='logName:cloudaudit.googleapis.com' \--description="Audit logs from my organization" \

Create the aggregated sink

Aggregated sinks combine and route log entries from the resources contained byan organization or folder to a destination.

In this tutorial, you create an aggregated sink that is non-intercepting. Thismeans every log entry that is routed by the aggregated sink is also routed bythe sinks in the resource in which the log entry originates. For example, anaudit log that originates in a project is routed by the aggregated sink and thesinks in that project. Therefore, it is possible for you to store multiplecopies of a log entry.

You can create intercepting sinks. For more information, seeAggregated sinks overview.

Set up the sink at the organization level

To create an aggregated sink that is non-intercepting and that routes logentries to a project, complete the following steps:

1. Run thegcloud logging sinks createcommand.

   Before running the following command, make the followingreplacements:

   • SINK_NAME: The name of the log sink. You can't change the name of a sink after you create it.
   • PROJECT_ID: The identifier of the project which stores the log bucket.
   • ORGANIZATION_ID: The identifier of the organization.

   Execute thegcloud logging sinks create command:

   gcloud logging sinks createSINK_NAME \logging.googleapis.com/projects/PROJECT_ID  \  --log-filter='logName:cloudaudit.googleapis.com' \  --description="Audit logs from my organization" \  --organization=ORGANIZATION_ID \  --include-children

   The--include-children option is important. This option ensures thatlog entries from all theGoogle Cloud projects and folders within your organization are routed.For more information, seeCollate and route organization-level logs to supported destinations.

2. Verify that the sink was created:

   gcloud logging sinks list --organization=ORGANIZATION_ID

3. Get the name of the service account:

   gcloud logging sinks describeSINK_NAME --organization=ORGANIZATION_ID

   The output looks similar to the following:

   writerIdentity: serviceAccount:o1234567890-ORGANIZATION_ID@gcp-sa-logging.iam.gserviceaccount.com

4. Copy the value of theserviceAccount field into your clipboard.

   Note: To route log entries to a resource protected by aservice perimeter,you must add the service account for that sink to an access level and thenassign it to the destination service perimeter. This isn't necessary fornon-aggregated sinks. For details, seeVPC Service Controls: Cloud Logging.

Grant access to the sink

After creating the aggregated sink, you must grant permission for the sink towrite log entries to the project that you set as the destination. You can grantpermission by using the Google Cloud consoleor by editing the Identity and Access Management (IAM) policy, as described inSet destination permissions.

To grant your sink permission to write log entries, do the following:

1. In the Google Cloud console, go to theIAM page:

   Go toIAM

   If you use the search bar to find this page, then select the result whose subheading isIAM & Admin.

2. Select the Google Cloud project that contains your log bucket.

3. Clickperson_addGrant access and add theservice account as a new principal. Don't include theserviceAccount: prefix.

4. In theSelect a role menu, selectLogs Writer.

5. ClickSave.

Generate log entries to assist in sink verification

To verify that your aggregated sink is properly configured, try thefollowing:

1. Generate audit log entries that should be routed to your log bucket.

   • If you have many Google Cloud projects in your organization, thenyou might have enough audit-log traffic that you don't need tocreate any for validation purposes. Go to the next step.

   • Otherwise, go to a different project, create a Compute Engine VMinstance, and then delete the instance you created.Audit logs are written when a VM is created, started, and deleted.

2. Follow the procedure in the section titledView logs in the Logs Explorer page to view youraudit logs. Be sure that you select the_AllLogs view.

Configure read access to a log view on a log bucket

When you create a log bucket, Cloud Logging automatically creates alog view named_AllLogs.This view includes every log entry stored in the log bucket.

To restrict a principal to having access only to specific log entries, createa log view, and then do one of the following:

• Grant them the role ofroles/logging.viewAccessor along with an IAM condition that restricts the grant to the log view.

• On the IAM policy associated with the log view, grant a principal access. We recommend this approach when you create a large number of log views.

For more information about these two approaches, seeControl access to a log view.

In the following steps, you grant a principal the role ofroles/logging.viewAccessor along with an IAM conditionthat restricts the grant to the view named_AllLogs:

1. In the Google Cloud console, go to theIAM page:

   Go toIAM

   If you use the search bar to find this page, then select the result whose subheading isIAM & Admin.

2. Select the Google Cloud project that contains the log bucket.

3. Clickperson_addAdd.

4. In theNew principal field, add a principal.

5. In theSelect a role menu, selectLogs Views Accessor.

   If you don't add a condition to this role, then the principal hasaccess to all log views on all user-defined log bucketsin the Google Cloud project.

6. Add an IAM condition to the binding:

   1. ClickAdd condition, enter a title and description.
   2. In theCondition type menu, scroll toResource, and thenselectName.
   3. In theOperator menu, selectEnds with.

   4. In theValue field, enter the full name of the log view:

      locations/LOCATION/buckets/BUCKET_NAME/views/_AllLogs

   5. ClickSave to save the condition.

7. ClickSave to save the binding.

View log entries in the Logs Explorer page

To view the log entries in your log bucket, do the following:

1. In the Google Cloud console, go to theLogs Explorer page:

   Go toLogs Explorer

   If you use the search bar to find this page, then select the result whose subheading isLogging.

2. SelectRefine Scope.

3. On theRefine scope panel, selectLog view.

4. Select the log view, or log views, whose log entries you want to see.For example, to view all log entries, select the view named_AllLogs.

5. ClickApply.

   TheLogs Explorer refreshes to show log entries from your log bucket.For information about using the Logs Explorer, seeUsing the Logs Explorer.