BigQuery Data Transfer Service run notifications

This page provides an overview of run notifications for theBigQuery Data Transfer Service.

There are two types of run notifications you can configure for theBigQuery Data Transfer Service:

Pub/Sub notifications: machine-readable notifications sent whena transfer run succeeds or fails
Email notifications: human-readable notifications sent when a transferrun fails

You can configure each type individually, or you can use both Pub/Suband email run notifications.

Pub/Sub notifications

Pub/Sub notifications send information about transfer runs toaPub/Sub topic. Pub/Sub notifications are triggered bycompleted transfer runs in the followingstates:

SUCCEEDED
FAILED
CANCELLED

You can send notifications to any Pub/Sub topic in any project forwhich you have sufficient permissions. Once received by thePub/Sub topic, the resulting message can be sent to any number ofsubscribers to the topic.

Before you begin

Before configuring Pub/Sub transfer run notifications, you should:

Enable the Pub/Sub API for the project that will receivenotifications.
Enablethe API
Have sufficient permissions on the project that will receive notifications:
- If you own the project that will receive notifications, you most likelyhave the necessary permission.
- If you plan to create topics for receiving notifications, you shouldhavepubsub.topics.create permissions.
- Whether you plan to use new or existing topics, you should havepubsub.topics.getIamPolicy andpubsub.topics.setIamPolicy permissions. If you create a topic,you typically have permission for it already. The following predefinedIAM role has bothpubsub.topics.getIamPolicy andpubsub.topics.setIamPolicy permissions:pubsub.admin. SeePub/Sub access control for more information.
Have an existing Pub/Sub topic that you want to sendnotifications to.

Caution: Don't remove the BigQuery Data Transfer Service Agentfrom thepubsub.publisher predefined IAM role. The removal cancause publishing notification failures to the Pub/Sub topic.Caution: Don't specify any custom schema when creating thePub/Sub topic. Specifying a custom schema can cause thenotification publication to fail.

Notification format

Notifications sent to the Pub/Sub topic consist of two parts:

Attributes: A set of key:value pairs describing the event.
Payload: A string that contains the metadata of the changed object.

Attributes

Attributes are key:value pairs contained in all notifications sent byBigQuery Data Transfer Service to your Pub/Sub topic. Notifications alwayscontain the following set of key:value pairs, regardless of the notification'spayload:

Attribute name	Example	Description
eventType	`TRANSFER_RUN_FINISHED`	The type of event that has just occurred.`TRANSFER_RUN_FINISHED` is the only possible value.
payloadFormat	`JSON_API_V1`	The format of the object payload.`JSON_API_V1` is the only possible value.

Payload

The payload is a string that contains the metadata of the transfer run. The typeof payload is not configurable at this time and is provided to accommodatefuture API version changes.

Payload type	Description
JSON_API_V1	The payload will be a UTF-8 JSON-serialized string containing theresource representation of a`TransferRun`.

Email notifications

Email notifications send human-readable email messages when a transfer runfails. These messages are sent to the email of thetransfer administrator -the account that set up the transfer. You cannot configure the content of themessage, and you cannot configure the recipient of the message.

If you used a service account to authenticate a transfer configuration, then youmight not have access to the email to receive transfer run notification emails.In such cases, we recommend that you set up Pub/Sub notificationsto receive transfer run notifications.

To send transfer run email notifications to more users, set up emailforwarding rules to distribute the messages. If you are using Gmail, you canAutomatically forward Gmail messages to another account.

The email notification is sent by the BigQuery Data Transfer Service and contains detailson the transfer configuration, the transfer run, and a link to the run historyfor the failed run. For example:

From: bigquery-data-transfer-service-noreply@google.comTo:TRANSFER_ADMINTitle: BigQuery Data Transfer Service — Transfer Run Failure —DISPLAY_NAMETransfer ConfigurationDisplay Name:DISPLAY_NAMESource:DATA_SOURCEDestination:PROJECT_IDRun SummaryRun:RUN_NAMESchedule Time:SCHEDULE_TIMERun Time:RUN_TIMEView Run HistoryGoogle LLC 1600 Amphitheatre Parkway, Mountain View, CA 94043This email was sent because you indicated you are willing to receive RunNotifications from the BigQuery Data Transfer Service. If you do not wish toreceive such emails in the future, click View Transfer Configuration andun-check the "Send E-mail Notifications" option.

Turn on or edit notifications

To turn on notifications, or edit an existing one, choose one of the following:

Console

Go to the BigQuery page in the Google Cloud console.
Go to the BigQuery page
In the navigation menu, clickData transfers.
To turn on notifications for a new transfer, clickCreate transfer. To adjustnotifications for an existing transfer, click the name of the transferand then clickEdit.
In theNotification options section, click the toggles next to thenotification types to enable.
- Email notifications: When you enable this option, the transferadministrator receives an email notification when a transfer runfails.
- Pub/Sub notifications: When you enable this option, chooseyourtopic name or clickCreate a topic. This option configures Pub/Sub runnotifications for yourtransfer.

Java

Before trying this sample, follow theJava setup instructions in theBigQuery quickstart using client libraries. For more information, see theBigQueryJava API reference documentation.

To authenticate to BigQuery, set up Application Default Credentials. For more information, seeSet up authentication for client libraries.

importcom.google.api.gax.rpc.ApiException;importcom.google.cloud.bigquery.datatransfer.v1.CreateTransferConfigRequest;importcom.google.cloud.bigquery.datatransfer.v1.DataTransferServiceClient;importcom.google.cloud.bigquery.datatransfer.v1.ProjectName;importcom.google.cloud.bigquery.datatransfer.v1.TransferConfig;importcom.google.protobuf.Struct;importcom.google.protobuf.Value;importjava.io.IOException;importjava.util.HashMap;importjava.util.Map;// Sample to get run notificationpublicclassRunNotification{publicstaticvoidmain(String[]args)throwsIOException{// TODO(developer): Replace these variables before running the sample.finalStringprojectId="MY_PROJECT_ID";finalStringdatasetId="MY_DATASET_ID";finalStringpubsubTopicName="MY_TOPIC_NAME";finalStringquery="SELECT CURRENT_TIMESTAMP() as current_time, @run_time as intended_run_time, "+"@run_date as intended_run_date, 17 as some_integer";Map<String,Value>params=newHashMap<>();params.put("query",Value.newBuilder().setStringValue(query).build());params.put("destination_table_name_template",Value.newBuilder().setStringValue("my_destination_table_{run_date}").build());params.put("write_disposition",Value.newBuilder().setStringValue("WRITE_TRUNCATE").build());params.put("partitioning_field",Value.newBuilder().build());TransferConfigtransferConfig=TransferConfig.newBuilder().setDestinationDatasetId(datasetId).setDisplayName("Your Scheduled Query Name").setDataSourceId("scheduled_query").setParams(Struct.newBuilder().putAllFields(params).build()).setSchedule("every 24 hours").setNotificationPubsubTopic(pubsubTopicName).build();runNotification(projectId,transferConfig);}publicstaticvoidrunNotification(StringprojectId,TransferConfigtransferConfig)throwsIOException{try(DataTransferServiceClientdataTransferServiceClient=DataTransferServiceClient.create()){ProjectNameparent=ProjectName.of(projectId);CreateTransferConfigRequestrequest=CreateTransferConfigRequest.newBuilder().setParent(parent.toString()).setTransferConfig(transferConfig).build();TransferConfigconfig=dataTransferServiceClient.createTransferConfig(request);System.out.println("\nScheduled query with run notification created successfully :"+config.getName());}catch(ApiExceptionex){System.out.print("\nScheduled query with run notification was not created."+ex.toString());}}}

Python

Before trying this sample, follow thePython setup instructions in theBigQuery quickstart using client libraries. For more information, see theBigQueryPython API reference documentation.