Authorize accounts for data transfer

This document provides an overview of how BigQuery Data Transfer Service interacts withdifferent account types, the types of account authorization that you need toperform general transfer tasks, and troubleshooting steps for common permissionerrors.

To start using the BigQuery Data Transfer Service, ensure that the accountsassociated with your project—both user accounts and service accounts—areauthenticated and authorized with the correct permissions to perform yourtransfer needs. For information about data source-specific permissions, see thetransfer guide for each data source.

Key concepts

The BigQuery Data Transfer Service automates data transfers from various datasources into BigQuery. The authentication and authorization modeloperates at two different stages, the control plane and the data plane, and fortwo types of users, a transfer creator or transfer owner.

Control plane

The control plane represents the stage in the authorization process where anauthenticated user is able to control and manage transfer configurations andruns. A user in the control plane must have the appropriateIdentity and Access Management (IAM) permissions to control and manage their transferconfigurations and runs:

• Thebigquery.transfers.update permission, which lets users do the following:
  • Set up data transfer configurations.
  • Administer the existing transfers, such as updating, disabling or deletinga transfer.
• Thebigquery.transfers.get permission, which lets users monitor transferruns, such as checking transfer run status or viewing transfer run historyand logs.

If you are using the Google Cloud console or the bq command-line tool to create atransfer, you must also have thebigquery.transfers.get permission.

Thebigquery.transfers.update permission is not required to set up ascheduled query. For more information, seetherequired permissionsfor scheduled queries.

Data plane

The data plane represents the stage outside of a user's direct control. In thedata plane, the BigQuery Data Transfer Service is able to operate data transfers inan offline mode and can trigger transfer runs automatically based on auser-specified schedule. In the data plane, the transfer owner's credential isused to access the source data, and (depending on the data source) either thetransfer owner's credentials or the BigQuery Data Transfer Serviceservice agentis used for starting BigQuery jobs and writing data into thedestination dataset.

For more details on required permissions, refer to the following sections inthis guide:

• Read-access authorization for external data sources
• Authorization to start BigQuery jobs
• Authorization to execute BigQuery jobs and write data to the destination dataset

Transfer creator versus transfer owner

A transfer creator refers to the user identity who created and set up thetransfer configuration. A BigQuery Data Transfer Service user and transfer creatorcan be a user account or aservice account.

A transfer owner refers to the user identity that the BigQuery Data Transfer Serviceuses to authorize the data transfer, specifically, for extracting the sourcedata. Forthe data sources that support service accounts,the transfer owner can be a user account or a service account. For other datasources, the transfer owner must be a user account.

The transfer owner and transfer creator can have the same user identity, but itis not a requirement. There are multiple ways of setting the transfer owner tobe a different user than the transfer creator:

• When creating a transfer, you can set the owner to a service account if thedata source supports services accounts.
• Once a transfer has been created, you can transfer ownership to a new useraccount (or to a service account if the data source supports service accounts)that has thebigquery.transfers.update andbigquery.transfers.getpermissions. You must be logged in to the new account when youupdate the credentials.

Read-access authorization for external data sources

The permissions required to read source data might vary from one data source toanother. For example, accessingGoogle Adsrequires read-access permissions to the Google Ads Customer ID.Similarly,Google Playrequires report access in theGoogle Play console. For more information about permissions that are specificto a data source, see the transfer guides for each data source.

Depending on the transfer owner's identity type, a different authorizationmethod is required to retrieve the access token to access the source data.

Transfer owner as a service account

When a service account is used as the transfer owner, the necessary permissionsare automatically granted when the BigQuery Data Transfer Service API is enabled foryour project. The BigQuery Data Transfer Service uses aserviceagent to get the access tokenfor the user-provided service account (transfer owner).

When you enable the BigQuery Data Transfer Service API, aservice agentis created for your project. The system also grants the service agent theBigQuery Data Transfer Service Agent role (roles/bigquerydatatransfer.serviceAgent),which includes the permissioniam.serviceAccounts.getAccessToken. Thatpermission allows the BigQuery Data Transfer Service service agent to impersonatethe transfer owner service account to retrieve the access token.

For more information about the BigQuery Data Transfer Service service agent,seeService agent.For more information about using service accounts and the up-to-date list ofdata sources that have service account support, seeUse service accounts.

Warning: Don't remove the predefined role from the service agent. TheBigQuery Data Transfer Service Agent roleis required for the BigQuery Data Transfer Service to work.

Transfer owner as a user account

If the transfer owner creating the transfer configuration is a user account (nota service account), you must manually grant permission for theBigQuery Data Transfer Service to get the access token for the user account andaccess the source data on the transfer owner's behalf. You can grant manualapproval with the OAuth dialog interface.

You only need to give permission to the BigQuery Data Transfer Service for thefirst time when creating a transfer for a given data source. You must give thepermission again when you create the first transfer for a newly used region,even if you are using the same data source. Data transfers from Youtube Channelsare the exception - you must manually grant permissions approval every time youcreate a Youtube Channel data transfer.

Changing the transfer owner by updating credentials also requires manualapproval if the new owner has never created a transfer for the data source inthat region before.

The following screenshot shows the OAuth dialog interface when you are creating aGoogle Ads transfer. The dialog displays data source-specificpermissions to be given:

Note: The BigQuery Data Transfer Service no longer supports theauthorization_codeparameter for Youtube Channel data transfers. You can use theversion_infoparameter to provide your authorization result to the transfer to allow it toget credentials. Theversion_info parameter is only required in thebq CLIor API calls.

To revoke the permissions that were given, follow these steps:

1. Go to theGoogle Account page.
2. ClickBigQuery Data Transfer Service.
3. To revoke the permissions, clickRemove access.

Warning: Revoking access permissions prevents any future transfer runs for thetransfer configurations that this account owns across all regions.

Authorization to start BigQuery jobs

When you migrate from most data sources, except when migrating usingscheduled queriesordataset copies, theBigQuery Data Transfer Service relies onservice agentsto start BigQuery jobs foryour project. The required permissionbigquery.job.create is automaticallygiven to theservice agentwhen you enable the BigQuery Data Transfer Service API for your project. For moreinformation, seeEnable the BigQuery Data Transfer Service.

When you migrate usingscheduled queries ordataset copies,BigQuery Data Transfer Service uses the transfer owner's credentials to start theBigQuery jobs.

Warning: Don't remove the predefined role from the service agent. The serviceagent role is required for BigQuery Data Transfer Service to work.

Authorization to execute BigQuery jobs and write data to the destination dataset

When you migrate from most data sources, except when migrating usingscheduled queries ordataset copies,the BigQuery Data Transfer Service relies on the service agent to write data intothe BigQuery destination dataset. The required permission,roles/bigquery.dataEditor, is granted to the service agent by theBigQuery Data Transfer Service when you create the transfer. You must havebigquery.datasets.update permission on the destination dataset to successfullygrant the permission.

When you migrate usingscheduled queries ordataset copies,BigQuery Data Transfer Service uses the transfer owner's credentials to execute theBigQuery jobs and write the data into BigQuerydestination dataset.

Note: Theroles/bigquery.dataEditor role granted to theBigQuery Data Transfer Service agent is only limited to the destination dataset thatis used in a transfer configuration. Other BigQuery datasetsunder the same project are not affected.Warning: Don't remove the service agent'sroles/bigquery.dataEditor role from thedestination dataset. Theroles/bigquery.dataEditor role is required for BigQuery Data Transfer Serviceto work.

Troubleshoot permission errors

If you are encountering authorization or permissions related issues for yourtransfer, seeAuthorization and permission issues.