This document describes how to create a linked dataset for a log bucket,and how to query and view logs in that dataset. For example, you mightcreate a linked dataset so that you can run queries on Log Analytics byusing yourreserved BigQuery slots.Similarly, you might want alinked dataset so that you can write queries in BigQuery orin Looker Studiothat join your log data with other business data.

You can't create a linked BigQuery dataset forananalytics view.

If you aren't familiar with Log Analytics, then seeLog Analytics overview.

When you need a linked dataset

You don't need alinked BigQuery datasetto issue queries to a log view on theLog Analytics pagewhen you use the default Cloud Logging service. You can saveand share your queries, and you can save the results to a custom dashboard.

You do need a linked BigQuery dataset when you want to doany of the following:

• Join log entry data with otherBigQuery datasets.
• Query log data from another service like theBigQuery Studio page or Looker Studio.
• Improve the performance of the queries that you run from theLog Analyticsby running them on yourBigQuery reserved slots.
• Create analerting policy that monitors the resultof a SQL query. For more information, seeMonitor your SQL query results with an alerting policy.

A linked BigQuery dataset for a log bucket letsBigQuery read the data in the log bucket.

If you run your queries on reserved BigQuery slots,then your queries are subject tocapacity compute pricing. Also, if youquery your data by using a service other than Log Analytics,then your queries might be subject to othercharges based on that service. See the pricing page for the service that youare using.

How log buckets and log views map to BigQuery objects

When you create a linked BigQuery dataset for a log bucket,a new dataset becomes available to BigQuery. You specify thename of the linked dataset during the linking process. Any log views withinthe log bucket are automaticallymapped tovirtual views in the dataset.The name of a virtual view is the same as the corresponding log view.

For example, suppose the projectmy_project has a log bucket namedexample, and that this log bucket has a log view named_AllLogs. If youcreate a linked dataset for that log bucket with the namemy_dataset, thentheExplorer page of BigQuery displays amy_project entry.A child of themy_project entry is a dataset with the namemy_dataset,and that dataset lists a virtual view named_AllLogs.

Access control implications when using linked datasets

When a principal queries a view on a linked BigQuery dataset,that query runs through the BigQuery permission layer, notthe Cloud Logging permission layer. Therefore,the BigQuery roles and permissions granted to the principal onthe linked dataset determine whether they can query the dataset.

You can't restrict a principal to a specific virtual view within a linkeddataset. When a principal has been granted access to the linked dataset,they can query every view in that dataset.

Before you begin

This section describes steps that you must complete before youcan use Log Analytics.

Configure log buckets

Ensure that your log buckets have been upgraded to use Log Analytics:

1. In the Google Cloud console, go to theLogs Storage page:

   Go toLogs Storage

   If you use the search bar to find this page, then select the result whose subheading isLogging.

2. For each log bucket that has a log view that you want to query, ensure that theLog Analytics available column displaysOpen. IfUpgrade is shown, then clickUpgrade and complete the dialog.

Configure IAM roles and permissions

This section describes the IAM roles or permissions thatare required to use Log Analytics:

• To get the permissions that you need to use Log Analytics and query log views, ask your administrator to grant you the following IAM roles on your project:

  • To query the_Required and_Default log buckets:Logs Viewer (roles/logging.viewer)
  • To query all log views in a project:Logs View Accessor (roles/logging.viewAccessor)

  You can restrict a principal to a specific log view either by adding an IAM condition to the Logs View Accessor role grant made at the project level, or by adding an IAM binding to the policy file of the log view. For more information, seeControl access to a log view.

  These are the same permissions that you need to view log entries on theLogs Explorer page. For information about additional roles that you need to query views on user-defined buckets or to query the_AllLogs view of the_Default log bucket, seeCloud Logging roles.

• To get the permissions that you need to create and query linked datasets, ask your administrator to grant you the following IAM roles on the project that stores the log bucket:

  • To create and view linked datasets:Logs Configuration Writer (roles/logging.configWriter)
  • To run queries on linked datasets using reserved BigQuery slots:
    • BigQuery User (roles/bigquery.user)
    • BigQuery Job User (roles/bigquery.jobUser)
  • To view linked datasets in BigQuery Studio, grant all roles mentioned in this step, and the following role. You can grant this role to a dataset or to your project:BigQuery Data Viewer (roles/bigquery.dataViewer)

Verify your BigQuery edition

If you want to query your log data from theLog Analytics pageby using reserved BigQuery slots and if you are usingprojects with Virtual Private Cloud (VPC) Service Controls,then ensure you are using the Enterprise Edition of BigQuery.For more information, seeUnderstand BigQuery editions.

Create a linked BigQuery dataset

When you want to use the capabilities of BigQuery to analyze yourlog data, upgrade a log bucket to use Log Analytics, and then create alinked dataset.With this configuration,Logging stores your log data butBigQuery can read the log data.

gcloud logging links createLINK_ID --bucket=BUCKET_ID --location=LOCATION

gcloud logging links create mylink --bucket=my-bucket --location=global

Run queries from BigQuery

When you have a log bucket that uses Log Analytics and linked datasets,you can view and query your linked datasets by using theBigQuery Studiopage. With this configuration, you can analyze yourdatasets using commands, workflows, and datasetsavailable only inBigQuery Studio.

To open BigQuery from the Log Analytics and then query linked dataset,do the following:

1. Ensure that a linked dataset exists for the log bucket which hosts thelog view that you want to query:

   1. In the Google Cloud console, go to theLogs Storage page:

      Go toLogs Storage

      If you use the search bar to find this page, then select the result whose subheading isLogging.

   2. Locate the log bucket and verify that theBigQuery linked datasetcolumn displays a URL. If the entry is empty, then you must createa linked dataset. For information on these steps, seeCreate a linked BigQuery dataset

2. In the Google Cloud console, go to theLog Analytics page:

   Go toLog Analytics

   If you use the search bar to find this page, then select the result whose subheading isLogging.

3. In theLog views list, find and select the log view.

4. In theSchema pane, selectQuery.

   TheQuery pane is populated with a default query.You can also enter a query in theQuery pane, or edit a displayed query.

5. In the toolbar, expand the button labeledRun query orRun on BigQuery, and selectOpen in BigQuery.

   TheBigQuery Studio page opens. TheFROMstatement of the query is modified to specify the path to the log view on thelinked dataset by using the BigQueryTable path syntax.

   You can also edit the displayed query.

6. On theBigQuery Studio page, clickRun query.

Run Log Analytics queries on your reserved BigQuery slots

This product or feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of theService Specific Terms. Pre-GA products and features are available "as is" and might have limited support. For more information, see thelaunch stage descriptions.

When you want to minimize the execution time of your Log Analytics queries,run them on your reserved BigQuery slots. TheLog Analytics pageis preconfigured to use the default Log Analytics service, which means that yourqueries must compete for limited slot space with other queries. When therearen't available slots, your the execution of your query is delayed. You caneliminate these delays by running your queries onreserved BigQuery slots.

To issue a SQL query to a log view, do the following:

1. Ensure that a linked dataset exists for the log bucket which hosts thelog view that you want to query:

   1. In the Google Cloud console, go to theLogs Storage page:

      Go toLogs Storage

      If you use the search bar to find this page, then select the result whose subheading isLogging.

   2. Locate the log bucket and verify that theBigQuery linked datasetcolumn displays a URL. If the entry is empty, then you must createa linked dataset. For information on these steps, seeCreate a linked BigQuery dataset

2. Ensure that you have configured your reserved BigQuery slots:

   1. Create a reservation with dedicated slots
   2. Create reservation assignments

3. ConfigureLog Analytics to run your queries on yourreserved BigQuery slots:

   1. In the Google Cloud console, go to theLog Analytics page:

      Go toLog Analytics

      If you use the search bar to find this page, then select the result whose subheading isLogging.

   2. In theLog views list, find the view, and then selectQuery orenter a query.

      If the query pane displays an error message that references theFROM statement, then the table can't be resolved to a specific log view.For information about how to resolve this failure, seeErrorFROM clause must contain exactly one log view.

   3. Go to the toolbar and ensure that a button labeledRun on BigQueryis displayed.

      If the toolbar displaysRun Query, thenclicksettingsSettings and selectBigQuery.

      If theRun on BigQuery button is disabled, then youneed to create a linked dataset.

   4. Run your query.

      You can use the toolbar options to format your query, clear the query,and open the BigQuery SQL reference documentation.

What's next

• Save and share a SQL query
• Sample SQL queries
• Chart SQL query results