Ingest Google Cloud data to Google Security Operations

This page describes how to enable and disable Google Cloud data ingestion into Google SecOps. This lets you store, search, and examine aggregated security information for your enterprise, going back for months or longer, in accordance with your data retention period.

Overview

There are two options to send Google Cloud data to Google SecOps. Choosing the right option depends on log type.

Option 1: Direct ingestion

A special Cloud Logging filter can be configured in Google Cloud to send specific log types to Google SecOps in real-time. These logs are generated by Google Cloud services.Logs are collected starting from the time the filter is configured. Logs generated prior to configuration are not included.This real-time forwarding applies to Cloud Logging, Cloud Asset Metadata, and Security Command Center Premium findings.

Google Security Operations only ingests supported log types. Available log types include:

• Cloud Audit Logs
• Cloud NAT
• Cloud DNS
• Cloud Next Generation Firewall
• Cloud Intrusion Detection System
• Cloud Load Balancing
• Cloud SQL
• Windows Event logs
• Linux syslog
• Linux Sysmon
• Zeek
• Google Kubernetes Engine
• Audit Daemon (auditd)
• Apigee
• reCAPTCHA Enterprise
• Cloud Run logs (GCP_RUN)
• Google Cloud Abuse Events
• Google Cloud DNS Advanced Threat Detection (GCP_DNS_ATD)
• Model Armor logs

For details about the specific log filters and more ingestion details, seeExport Google Cloud logs to Google SecOps.

You can also send Google Cloud asset metadata used for context enrichment.For details, seeExport Google Cloud asset metadata to Google SecOps.

Cloud Logging canroute logs to Cloud Storage by Google SecOps on a scheduled basis.

For details about how to configure Cloud Storage for Google SecOps, seeFeed Management: Cloud Storage.

Before you begin

Before you can ingest Google Cloud data into a Google SecOps instance, you must complete the following steps:

1. Grant the followingIdentity and Access Management (IAM) roles at an organizational level to access the Google SecOps section:

   • Chronicle Service Admin (roles/chroniclesm.admin): IAM role for performing all activities.
   • Chronicle Service Viewer (roles/chroniclesm.viewer): IAM role to only view the state of ingestion.
   • Security Center Admin Editor (roles/securitycenter.adminEditor): Required to enable the ingestion ofCloud Asset Metadata.

2. If you plan to enableCloud Asset Metadata, you must onboard theorganization to the Security Command Center.SeeOverview of organization-level activation for more information.

Granting IAM roles

You can grant the required IAM roles using either the Google Cloud console or using the gcloud CLI.

To grant IAM roles using Google Cloud console, complete the following steps:

1. Sign in to the Google Cloud organization you want to connect to and go to the IAM screen usingProducts > IAM & Admin > IAM.

2. From theIAM screen, select the user and clickEdit Member.

   Note: If you're not in the Organization view of IAM, theEdit Member button is disabled. To enable it, go to the organization's IAM screen.

3. In the Edit Permissions screen, clickAdd Another Role and search for Google SecOps to find the IAM roles.

4. Once you have assigned the roles, clickSave.

To grant IAM roles using the Google Cloud CLI, complete the following steps:

1. Rungcloud init to verify that you're logged into the correct organization and project.

2. To grant the Chronicle Service Admin IAM role usinggcloud, run the following command:

   gcloudorganizationsadd-iam-policy-bindingORGANIZATION_ID\--member"user:USER_EMAIL"\--roleroles/chroniclesm.admin

   Replace the following:

   • ORGANIZATION_ID: the numeric organization ID.
   • USER_EMAIL: the user's email address.

3. To grant the Chronicle Service Viewer IAM role usinggcloud, run the following command:

   gcloudorganizationsadd-iam-policy-bindingORGANIZATION_ID\--member"user:USER_EMAIL"\--roleroles/chroniclesm.viewer

4. To grant the Security Center Admin Editor IAM role usinggcloud, run the following command:

   gcloudorganizationsadd-iam-policy-bindingORGANIZATION_ID\--member"user:USER_EMAIL"\--roleroles/securitycenter.adminEditor`

5. To grant the Organization Role Viewer IAM role usinggcloud, run the following command:

   gcloudorganizationsadd-iam-policy-bindingORGANIZATION_ID\--member"user:USER_EMAIL"\--roleroles/iam.organizationRoleViewer

Enable direct ingestion from Google Cloud

The steps to enable direct ingestion from Google Cloud are different dependingon the ownership of the project that your Google SecOps instance is bound to.

• If it is bound to a project that you own and manage, follow the steps inConfigure ingestion when project owned by customer.This approach lets you configure the data ingestion from more than one Google Cloud organization.

• If it is bound to a project that Google Cloud owns and manages,follow the steps inConfigure ingestion for a Google managed project.

After you configure direct ingestion, your Google Cloud data is sent to Google SecOps.You can use Google SecOps's analysis features to investigate security related issues.

Configure ingestion when project owned by customer

Do the following steps if you own the Google Cloud project.

You can configure direct ingestion from multiple organizations using the same project-levelconfiguration page. Do the following steps to create a new configuration and edit anexisting configuration.

When you migrate an existing Google SecOps instance so that it binds to a project that you own,and if direct ingestion was configured before the migration, the direct ingestion configuration ismigrated as well.

1. Go to theGoogle SecOps >Ingestion Settings page in the Google Cloud console.
   Go to theGoogle SecOps page
2. Select the project that is bound to your Google SecOps instance.

3. In theOrganization menu, select the organization from which logs willbe exported.The menu displays organizations you have permission to access. The list can include organizations that aren't linked to the Google SecOps instance. You cannot configure an organization that sends data to a different Google SecOps instance.

   

4. Under theGoogle Cloud Ingestion setting section, click theSending data to Google Security Operations toggle to enable logs to be sent to Google SecOps.

5. Select one or more of the following options to define the type of data sent to Google SecOps:

   • Google Cloud Logging: For more information about this option, seeExport Google Cloud logs.
   • Cloud Asset Metadata: For more information about this option, seeExport Google Cloud asset metadata.
   • Security Center Premium findings: For more information about this option, seeExport Security Center Premium findings.

6. Under theCustomer export filter settings section, configureexport filters to customize the Cloud Loggingdata sent to Google SecOps. SeeGoogle Cloud log typessupported for export.

7. To ingest logs from an additional organization to the same Google SecOps instance,select the organization from theOrganization menu, and then repeat thesteps to define the type of data to export and export filters.You will see multiple organizations listed in theOrganization menu.

8. To export Sensitive Data Protection data (previously called Google Cloud Data Loss Prevention data) to Google SecOps, seeExport Sensitive Data Protection data.

Configure ingestion for a Google managed project

If Google Cloud owns the project, do the following to configure direct ingestionfrom your Google Cloud organization into your Google SecOps instance:

1. Go to theGoogle SecOps >Overview >Ingestion tab in the Google Cloud console.Go to the Google SecOps Ingestion tab
2. Click theManage organization ingestion settings button.
3. If aPage not viewable for projects message appears, select anorganization, and clickSelect.
4. Enter your one-time access code in the1-time Google SecOps access code field.
5. Check the box labeledI consent to the terms and conditions ofGoogle SecOps's usage of my Google Cloud data.
6. ClickConnect Google SecOps.
7. Go to theGlobal Ingestion Settings tab for the organization.

8. Select the type of data that will be sent by enabling one or more of the following options:

   • Google Cloud Logging: For more information about this option, seeExport Google Cloud Logs.
   • Cloud Asset Metadata For more information about this option, seeExport Google Cloud asset metadata.
   • Security Center Premium findings: For more information about this option, seeExport Security Center Premium findings.

9. Go to theExport Filter Settings tab.

10. Under theCustomer export filter settings section, configureexport filters to customize the Cloud Loggingdata sent to Google SecOps. SeeGoogle Cloud log typessupported for export.

11. To export Sensitive Data Protection data (previously called Google Cloud Data Loss Prevention data) to Google SecOps, seeExport Sensitive Data Protection data.

Export Google Cloud logs

Afterenabling Cloud Logging, you can export log data for the supportedGoogle Cloud log types to your Google SecOps instance.

To export Google Cloud logs to Google SecOps, set theEnable Cloud logs toggle toEnabled.

You cancustomize the export filter of the logs to export to Google SecOps. Include or exclude log types by adding or removing supported export filters, which are listed in this section.

• Cloud Audit Logs (GCP_CLOUDAUDIT):

  This includes Admin Activity, Data Access, System Event, Access Transparency, and Policy Denied logs.

  Caution: Data Access logs written by Cloud Audit Logs can produce a large volume of data that has low value for threat detection. The volume might result in data-throttling due to the enforcement ofburst limits. To learn how to filter out logs that are generated by routine activities, seeTune Cloud Audit Logs filters.
  • log_id("cloudaudit.googleapis.com/activity") (exported by the default filter)
  • log_id("cloudaudit.googleapis.com/system_event") (exported by the default filter)
  • log_id("cloudaudit.googleapis.com/policy")
  • log_id("cloudaudit.googleapis.com/access_transparency")

• Cloud NAT logs (GCP_CLOUD_NAT):

  • log_id("compute.googleapis.com/nat_flows")

• Cloud DNS logs (GCP_DNS):

  • log_id("dns.googleapis.com/dns_queries") (exported by the default filter)

• Cloud Next Generation Firewall logs (GCP_FIREWALL):

  • log_id("compute.googleapis.com/firewall")

• GCP_IDS:

  • log_id("ids.googleapis.com/threat")
  • log_id("ids.googleapis.com/traffic")

• GCP_LOADBALANCING:

  This includes logs from Google Cloud Armor and Cloud Load Balancing (both External and Internal).

  • log_id("requests")
  • log_id("loadbalancing.googleapis.com/requests")

• GCP_CLOUDSQL:

  • log_id("cloudsql.googleapis.com/mysql-general.log")
  • log_id("cloudsql.googleapis.com/mysql.err")
  • log_id("cloudsql.googleapis.com/postgres.log")
  • log_id("cloudsql.googleapis.com/sqlagent.out")
  • log_id("cloudsql.googleapis.com/sqlserver.err")

• GCP_VPC_FLOW:

  • log_id("compute.googleapis.com/vpc_flows") (for US and EU regions only)

• NIX_SYSTEM:

  • log_id("syslog")
  • log_id("authlog")
  • log_id("securelog")
  • log_id("osconfig.googleapis.com/patch_job")

• LINUX_SYSMON:

  • log_id("sysmon.raw")

• WINEVTLOG:

  • log_id("winevt.raw")
  • log_id("windows_event_log")

• BRO_JSON:

  • log_id("zeek_json_streaming_conn")
  • log_id("zeek_json_streaming_dhcp")
  • log_id("zeek_json_streaming_dns")
  • log_id("zeek_json_streaming_http")
  • log_id("zeek_json_streaming_ssh")
  • log_id("zeek_json_streaming_ssl")

• KUBERNETES_NODE:

  • log_id("events")
  • log_id("stdout")
  • log_id("stderr")

• AUDITD:

  • log_id("audit_log")

• GCP_APIGEE_X:

  • log_id("apigee.googleapis.com/ingress_instance")
  • log_id("apigee.googleapis.com")
  • log_id("apigee-logs")
  • log_id("apigee")
  • logName =~ "^projects/[\w\-]+/logs/apigee[\w\-\.]*$"

• GCP_RECAPTCHA_ENTERPRISE:

  • log_id("recaptchaenterprise.googleapis.com/assessment")
  • log_id("recaptchaenterprise.googleapis.com/annotation")

• GCP_RUN:

  • log_id("run.googleapis.com/stderr")
  • log_id("run.googleapis.com/stdout")
  • log_id("run.googleapis.com/requests")
  • log_id("run.googleapis.com/varlog/system")

• GCP_NGFW_ENTERPRISE:

  • log_id("networksecurity.googleapis.com/firewall_threat")

• GCP_ABUSE_EVENTS:

  • log_id("abuseevent.googleapis.com/abuse_events")

• GCP_DNS_ATD

  • log_id("networksecurity.googleapis.com/dns_threat_events")

• Model Armor logs (GCP_MODEL_ARMOR):

  This includes logs for sanitization operations (screening prompts and responses) and template operations (creation, updates).

  • log_id("modelarmor.googleapis.com/sanitize_operations")
  • log_id("modelarmor.googleapis.com/templates")

Customize export filter settings

By default, your Cloud Audit Logs (Admin Activity and System Event) and Cloud DNSlogs are sent to your Google SecOps instance. However, you can customizethe export filter to include or exclude specific types of logs.

To define a custom filter for your logs, do the following:

1. Identify the logs for your custom filter using thelog scoping tool.

2. In theAuto-generated log filter section that follows the log scoping tool, copy the generated custom log filter code.

3. Go to theGoogle SecOps page in the Google Cloud consoleand select a project.
   Go to theGoogle SecOps page
   

4. Launch the Logs Explorer using the link on theExport Filter Settings tab.

5. Copy your new query into theQuery field and clickRun Query to test it.

6. Copy your new query into theLogs Explorer >Query field, and then clickRun Query to test it.

7. Verify that the matched logs displayed in theLogs Explorer are exactlywhat you intend to export to Google SecOps. When the filter is ready,copy it to theCustom export filter settings section for Google SecOps.

8. Go back to theCustom export filter settings section on theGoogle SecOps page.

9. Click theEdit icon in theExport filter field, and then paste the copied filter into the field.

10. ClickSave.

    • If the following error message appears: "The provided filter canpotentially allow unsupported log types", there may be an unsupported logtype included in the export filter. Remove the unsupported log type fromthe export filter. Only include log types listed in:Google Cloud log types supported forexport.

    • If the save is successful, your new custom filter works against all new logs exported to yourGoogle SecOps instance.

    • Optional: To reset the export filter to the default version, save a copy of your custom filter, and then clickResetto Default.

Tune Cloud Audit Logs filters

Data Access logs written by Cloud Audit Logs can produce a large volume of data without much value for threat detection. If you choose to send these logs to Google SecOps, you should filter out logs that are generated by routine activities.

The following export filter captures Data Access logs and excludes high-volume events such as Read and List operations of Cloud Storage and Cloud SQL:

( log_id("cloudaudit.googleapis.com/data_access")  AND NOT protoPayload.methodName =~ "^storage\.(buckets|objects)\.(get|list)$"  AND NOT protoPayload.request.cmd = "select"    AND NOT protoPayload.methodName =~ "^google\.spanner\.v1\.Spanner\.(ExecuteStreamingSql|BeginTransaction|Commit)$" )

For more information about tuning Data Access logs generated by Cloud Audit Logs, seeManage the volume of Data Access audit logs.

Export Filter examples

The following export filter examples illustrate how you can include or exclude certain types of logs from the export to your Google SecOps instance.

Export Filter Example: Include additional log types

The following export filter exports access transparency logs in addition to the default logs:

log_id("dns.googleapis.com/dns_queries") ORlog_id("cloudaudit.googleapis.com/activity") ORlog_id("cloudaudit.googleapis.com/system_event") ORlog_id("cloudaudit.googleapis.com/access_transparency")

Export Filter Example: Include additional logs from a specific project

The following export filter exports access transparency logs from a specific project, in addition to the default logs:

log_id("dns.googleapis.com/dns_queries") ORlog_id("cloudaudit.googleapis.com/activity") ORlog_id("cloudaudit.googleapis.com/system_event") ORlogName = "projects/my-project-id/logs/cloudaudit.googleapis.com%2Faccess_transparency"

Export Filter Example: Include additional logs from a specific folder

The following export filter exports access transparency logs from a specific folder, in addition to the default logs:

log_id("dns.googleapis.com/dns_queries") ORlog_id("cloudaudit.googleapis.com/activity") ORlog_id("cloudaudit.googleapis.com/system_event") ORlogName = "folders/my-folder-id/logs/cloudaudit.googleapis.com%2Faccess_transparency"

Export Filter Example: Exclude logs from a specific project

The following export filter exports the default logs from the entire Google Cloud organization with the exception of a specific project:

(log_id("dns.googleapis.com/dns_queries") ORlog_id("cloudaudit.googleapis.com/activity") ORlog_id("cloudaudit.googleapis.com/system_event")) AND(NOT logName =~ "^projects/my-project-id/logs/.*$")

Export Google Cloud asset metadata

You can export your Google Cloud asset metadata from Cloud Asset Inventory to Google SecOps.This asset metadata is drawn from your Cloud Asset Inventory and consists of informationabout your assets, resources, and identities including the following:

• Environment
• Location
• Zone
• Hardware models
• Access control relationships between resources and identities

The following types of Google Cloud asset metadata will be exported to your Google SecOps instance:

• GCP_BIGQUERY_CONTEXT
• GCP_COMPUTE_CONTEXT
• GCP_IAM_CONTEXT
• GCP_IAM_ANALYSIS
• GCP_STORAGE_CONTEXT
• GCP_CLOUD_FUNCTIONS_CONTEXT
• GCP_SQL_CONTEXT
• GCP_NETWORK_CONNECTIVITY_CONTEXT
• GCP_RESOURCE_MANAGER_CONTEXT

The following are examples of Google Cloud asset metadata:

• Application name—Google-iamSample/0.1
• Project name—projects/my-project

For more information about exporting specific context logs and ingesting them into Google SecOps, seeDefault parser configuration and ingestion and search for 'context' or 'analysis'.

Export Security Command Center findings

You can export Security Command Center Premium Event Threat Detection findings and all other findings to Google SecOps.

Google SecOps supports the followingSecurity Command Center finding classes:

• ERROR
• MISCONFIGURATION
• OBSERVATION
• POSITIVE_VALIDATION
• POSTURE_VIOLATION
• THREAT
• TOXIC_COMBINATION
• UNSPECIFIED
• VULNERABILITY

For more information about ETD findings, seeOverview of Event Threat Detection.

1. Enable Google Cloud data ingestion by completing the previous section in this document.
2. Configure Sensitive Data Protection toprofile data.
3. Set the scan configuration topublish data profilesto Google SecOps.

SeeSensitive Data Protection documentation for detailed informationabout creating data profiles for BigQuery data.

Disable Google Cloud data ingestion

The steps to disable direct ingestion of data from Google Cloud are different depending on how Google SecOps is configured. Choose one of the following:

• If your Google SecOps instance is bound to a project that you own andmanage, perform the following steps:

  1. Select the project that is bound to your Google SecOps instance.
  2. In the Google Cloud console, go to theIngestion tab underGoogle SecOps.
     Go to theGoogle SecOps page
  3. In theOrganization menu, select the organization from which logs are exported.
  4. Set theSending data to Google Security Operations toggle toDisabled.
  5. If you configured data export from multiple Organizations, and you want todisable these as well, do these steps for each organization.

• If your Google SecOps instance is bound to a project that Google Cloud ownsand manages, perform the following steps:

  Note: After completing these steps, you will need to obtain a new one-time accesscode from your Google SecOps representative and complete the procedure toenable direct ingestion from Google Cloud to restart Google Cloud log ingestion.
  1. Go to theGoogle SecOps >Ingestion page in the Google Cloud console.
     Go to theGoogle SecOps page
  2. In the resource menu, select the organization that is bound to your Google SecOpsinstance and that you are ingesting data from.
  3. Check the box labeledI want to disconnect Google SecOps and stop sending Google Cloud Logs into Google SecOps.
  4. ClickDisconnect Google SecOps.

Control the rate of ingestion

When the data ingestion rate for a tenant reaches a certain threshold,Google Security Operations restricts the rate of ingestion for new data feeds to preventa source with a high ingestion rate from affecting the ingestion rate of anotherdata source. In this case, there is a delay but no data is lost. The ingestionvolume and tenant's usage history determine the threshold.

You can request a rate limit increase by contactingCloud Customer Care.

Troubleshooting

• If the relationships between resources and identities are missing from yourGoogle SecOps instance, disable and then re-enable direct ingestion of log data to Google SecOps.
• Google Cloud asset metadata is periodically ingested into Google SecOps. Allow several hours for changes to appearin the Google SecOps UI and APIs.

• When you add a log type to the export filter, you may see this message: "The provided filter can potentially allow unsupported log types".

  Workaround: Only include log types to the export filter, that appear in the following list:Google Cloud log typessupported for export.

What's next

• Open your Google SecOps instance using the customer specific URL provided by your Google SecOps representative.
• Learn more aboutGoogle SecOps.

Need more help?Get answers from Community members and Google SecOps professionals.