Monitor and view logs for BigQuery Data Transfer Service

BigQuery Data Transfer Servicemonitoring andlogging provideinformation about the service's workload performance and status.BigQuery Data Transfer Service exports monitoring data toCloud Monitoring.

Monitor BigQuery Data Transfer Service

You can use monitoring metrics for the following purposes:

• Evaluate the usage and performance a data transfer configuration.
• Troubleshoot problems.
• Monitor transfer run statuses.

To create custom dashboards, set up alerts, and query metrics withMonitoring, you can use the Google Cloud console or theMonitoring API.

View transfer data in Metrics Explorer

1. In the Google Cloud console, go to theMonitoring page.

   Go to Monitoring

2. In the navigation pane, clickMetrics Explorer.

3. Select your project.

4. In theFind resource type and metric box, enter the following:

   • ForResource type, enterBigQuery DTS Config.

   • ForMetric, select one of the metrics listed inMonitoring metrics for transfer configurations, for example,Completed run count.

     

5. Optional: Select aligner, reducer, and other parameters.

6. The metrics are displayed in theMetrics explorer window.

   

Define Cloud Monitoring alerts

You can defineMonitoring alerts forBigQuery Data Transfer Service metrics:

1. In the Google Cloud console, go to theMonitoring page.

   Go to Monitoring

2. In the navigation pane, selectAlerting> Create policy.

   For more information about alerting policies and concepts behind them, seeTypes of alerting policies.

3. ClickAdd Condition and select a condition type.

4. Select metrics and filters. For metrics, the resource type isBigQuery DTSConfig.

5. ClickSave Condition.

6. Enter policy name, and then clickSave Policy.

For more information about alerting policies and concepts, seeIntroduction to alerting.

Define Cloud Monitoring custom dashboards

You can create custom dashboards over BigQuery Data Transfer Service metrics:

1. In the Google Cloud console, go to theMonitoring page.

   Go to Monitoring

2. In the navigation pane, selectDashboards> Create Dashboard.

3. ClickAdd Chart.

4. Give the chart a title.

5. Select metrics and filters. For metrics, the resource type isBigQuery DTSConfig.

6. ClickSave.

For more information, seeManage custom dashboards.

Metric reporting frequency and retention

Metrics for BigQuery Data Transfer Service runs are exported to Monitoring inbatches, at 1-minute intervals. Monitoring data is retained for 6 weeks.

The dashboard provides data analysis in default intervals of1h (1 hour),6H(6 hours),1D (1 day),1W (1 week), and6W (6 weeks). You can manuallyrequest analysis in any interval between1M (1 minute) to6W (6 weeks).

Monitor metrics for transfer configurations

The following metrics for BigQuery Data Transfer Service configs are exported toMonitoring:

Filter dimensions for metrics

Metrics are aggregated for each BigQuery Data Transfer Service configuration. Youcan filter aggregated metrics by the following dimensions:

BigQuery Data Transfer Service logs

Each BigQuery Data Transfer Service run is logged usingCloud Logging.Logging is automatically enabled for all data transfers.

Required roles

The Logs Viewer role (roles/logging.viewer) gives you read-only access to allfeatures of Logging. For more information about theIdentity and Access Management (IAM) permissions and roles that apply toLogging data, see theLogging access controlguide.

View logs

To view logs, go to theLogs Explorer page.

Go to Logs Explorer

BigQuery Data Transfer Service logs are indexed first by the transfer configuration andthen by the individual transfer run.

View transfer run logs

To show only the log entries from a given transferrun_id, in theQuerybuilder, add the following filters:

resource.type="bigquery_dts_config"labels.run_id="transfer_run_id"

View transfer configuration logs

To show log entries from a given transferconfig_id, in theQuery builder,add the following filters:

resource.type="bigquery_dts_config"labels.run_id="transfer_config_id"

View all logs

To see all BigQuery Data Transfer Service logs, do one of the following:

• In theLog fields section, forResource type, selectBigQuery DTSConfig.

  

• In theQuery builder, add the following filter:

  resource.type="bigquery_dts_config"

For more information about how to use the Log Explorer, seeUsing the Log Explorer.

Log format

BigQuery Data Transfer Service logs messages in the following format:

{"insertId":"0000000000","jsonPayload":{"message":"DTS transfer run message."},"resource":{"type":"bigquery_dts_config","labels":{"project_id":"my_project_id","config_id":"transfer_config_id","location":"us"}},"timestamp":"2020-11-25T04:45:48.545732221Z","severity":"INFO","labels":{"run_id":"transfer_run_id"},"logName":"projects/your_project_id/logs/bigquerydatatransfer.googleapis.com%2Ftransfer_config","receiveTimestamp":"2020-11-25T04:45:48.960214929Z"}

What is logged

BigQuery Data Transfer Service log entries contain information that is useful formonitoring and debugging your transfer runs. Log entries contain the followingtypes of information:

• timestamp: used to compute the log entry's age and to enforce the log'sretention period
• severity: can beINFO,WARNING orERROR
• message_text: holds a string that explains the current status of thetransfer run

What's next

• Learn more aboutMonitoring.
• Read an overview ofCloud Audit Logs andCloud Logging.