Use service accounts with BigQuery Data Transfer Service

Some data sources support data transfer authentication by using aservice accountthrough the Google Cloud console, API, or thebq command line. A serviceaccount is a Google Account associated with your Google Cloud project. A serviceaccount can run jobs, such as scheduled queries or batch processing pipelines byauthenticating with the service account credentials rather than a user'scredentials.

You can update an existing data transfer with the credentials of a serviceaccount. For more information, seeUpdate data transfer credentials.

The following situations require updating credentials:

Your transfer failed to authorize the user's access to the data source:
Error code 401 : Request is missing required authentication credential. UNAUTHENTICATED
You receive anINVALID_USER error when you attempt to run the transfer:
Error code 5 : Authentication failure: User Id not found. Error code: INVALID_USERID

To learn more about authenticating with service accounts, seeIntroduction to authentication.

Data sources with service account support

BigQuery Data Transfer Service can use service account credentials for transfers with thefollowing:

*Cloud Storage*Amazon Redshift*Amazon S3*Campaign Manager*Carbon Footprint*Dataset Copy*Display & Video 360*Google Ad Manager*Google Ads*Google Merchant Center*Google Play*Scheduled Queries*Search Ads 360*Teradata*YouTube Content Owner

Before you begin

Verify that you have completed all actions required inEnabling BigQuery Data Transfer Service.
Grant Identity and Access Management (IAM) roles that give users the necessarypermissions to perform each task in this document.

Required permissions

To update a data transfer to use a service account, you must have the followingpermissions:

Thebigquery.transfers.update permission to modify the transfer.
The predefinedroles/bigquery.admin IAM role includes thispermission.
Access to the service account. For more information about granting users theservice account role, seeService Account User role.

Ensure that the service account you choose to run the transfer has thefollowing permissions:

Thebigquery.datasets.get andbigquery.datasets.update permissions onthe target dataset. If the table usescolumn-level access control,the service account must also have thebigquery.tables.setCategorypermission.
Thebigquery.admin predefined IAM roleincludes all of these permissions. For more information aboutIAM roles in BigQuery Data Transfer Service, seeIntroduction to IAM.
Access to the configured transfer data source. For more information about therequired permissions for different data sources, seeData sources with service account support.
For Google Ads transfers, the service account must be granteddomain-wide authority. For more information, seeGoogle Ads API Service Account guide.

Update data transfer credentials

Console

The following procedure updates a data transfer configuration toauthenticate as a service account instead of your individual user account.

In the Google Cloud console, go to the Data transfers page.
Go to Data transfers
Click the transfer in the data transfers list.
ClickEDIT to update the transfer configuration.
In theService Account field, enter the service account name.
ClickSave.

bq

To update the credentials of adata transfer, you can use the bq command-line tool toupdate the transfer configuration.

Use thebq update command with the--transfer_config,--update_credentials, and--service_account_name flags.

For example, the following command updates a data transfer configuration toauthenticate as a service account instead of your individual user account:

bqupdate\--transfer_config \--update_credentials \--service_account_name=abcdef-test-sa@abcdef-test.iam.gserviceaccount.com projects/862514376110/locations/us/transferConfigs/5dd12f26-0000-262f-bc38-089e0820fe38 \

Note: If you are using the bq command-line tool, use the--service_account_name flag instead of authenticating as a serviceaccount.

Java

Before trying this sample, follow theJava setup instructions in the BigQuery quickstart using client libraries. For more information, see theBigQueryJava API reference documentation.

To authenticate to BigQuery, set up Application Default Credentials. For more information, seeSet up authentication for client libraries.

importcom.google.api.gax.rpc.ApiException;importcom.google.cloud.bigquery.datatransfer.v1.DataTransferServiceClient;importcom.google.cloud.bigquery.datatransfer.v1.TransferConfig;importcom.google.cloud.bigquery.datatransfer.v1.UpdateTransferConfigRequest;importcom.google.protobuf.FieldMask;importcom.google.protobuf.util.FieldMaskUtil;importjava.io.IOException;// Sample to update credentials in transfer config.publicclassUpdateCredentials{publicstaticvoidmain(String[]args)throwsIOException{// TODO(developer): Replace these variables before running the sample.StringconfigId="MY_CONFIG_ID";StringserviceAccount="MY_SERVICE_ACCOUNT";TransferConfigtransferConfig=TransferConfig.newBuilder().setName(configId).build();FieldMaskupdateMask=FieldMaskUtil.fromString("service_account_name");updateCredentials(transferConfig,serviceAccount,updateMask);}publicstaticvoidupdateCredentials(TransferConfigtransferConfig,StringserviceAccount,FieldMaskupdateMask)throwsIOException{try(DataTransferServiceClientdataTransferServiceClient=DataTransferServiceClient.create()){UpdateTransferConfigRequestrequest=UpdateTransferConfigRequest.newBuilder().setTransferConfig(transferConfig).setUpdateMask(updateMask).setServiceAccountName(serviceAccount).build();dataTransferServiceClient.updateTransferConfig(request);System.out.println("Credentials updated successfully");}catch(ApiExceptionex){System.out.print("Credentials was not updated."+ex.toString());}}}

Python

Before trying this sample, follow thePython setup instructions in theBigQuery quickstart using client libraries. For more information, see theBigQueryPython API reference documentation.