Use third party transfers

Third party transfers for BigQuery Data Transfer Service allow you to automaticallyschedule and manage recurring load jobs for external data sources such asSalesforce CRM, Adobe Analytics, and Facebook Ads.

Before you begin

Before you create a third party data transfer:

• Verify that you have completed all actions required toenable the BigQuery Data Transfer Service.
• Create a BigQuery datasetto store the data.
• Consult the documentation for your third party data source to ensure you haveconfigured any permissions necessary to enable the transfer.
• If you intend to setup transfer run notifications for Pub/Sub, youmust havepubsub.topics.setIamPolicy permissions. Pub/Subpermissions are not required if you just set up email notifications. For moreinformation, seeBigQuery Data Transfer Service run notifications.

Required BigQuery roles

To get the permissions that you need to create a BigQuery Data Transfer Service data transfer, ask your administrator to grant you theBigQuery Admin (roles/bigquery.admin) IAM role on your project. For more information about granting roles, seeManage access to projects, folders, and organizations.

This predefined role contains the permissions required to create a BigQuery Data Transfer Service data transfer. To see the exact permissions that are required, expand theRequired permissions section:

You might also be able to get these permissions withcustom roles or otherpredefined roles.

For more information, seeGrantbigquery.admin access.

Limitations

Third party transfers are subject to the following limitations:

• You must create or update a third party transfer using theGoogle Cloud console.
• You cannot configure or update a third party transfers using thebq command-line tool.

Set up a third party data transfer

To create a third party data transfer by using the Google Cloud console:

1. Go to the Google Cloud Marketplace.

   Go to the Google Cloud Marketplace

2. Click the appropriate third party provider.

3. On the documentation page for the third party provider, clickEnroll. The enrollment process may take a moment.

4. After the enrollment is complete, clickConfigure Transfer.

5. On theCreate Transfer page:

   • ForSource, choose the appropriate third party data source. You canclickExplore Data Sources to see the list of third party providersin the Google Cloud Marketplace.

     

   • ForDisplay name, enter a name for the transfer such asMy Transfer.The transfer name can be any value that allows you to easily identify thetransfer if you need to modify it later.

     

   • ForSchedule, leave the default value (Start now) or clickStart at a set time.

     • ForRepeats, choose an option for how often to run the transfer.Options include:

       • Daily (default)
       • Weekly
       • Monthly
       • Custom
       • On-demand

       If you choose an option other than Daily, additional options areavailable. For example, if you choose Weekly, an option appears foryou to select the day of the week.

     • ForStart date and run time, enter the date and time to start thetransfer. If you chooseStart now, this option is disabled.

       

   • ForDestination dataset, choose the dataset you created to store yourdata.

     

   • (Optional) In theNotification options section:

     • Click the toggle to enable email notifications. When you enable thisoption, the transfer administrator receives an email notification when atransfer run fails.
     • ForSelect a Pub/Sub topic, choose yourtopicname or clickCreate a topic to create one. This option configuresPub/Sub runnotificationsfor your transfer.

6. ClickConnect Source.

   Note: If you did not select a destination dataset, clickingConnectSource produces the following error:A selected destination dataset isrequired before connecting to the source.

7. When prompted, clickAccept to give the BigQuery Data Transfer Service permissionto connect to the data source and to manage your data inBigQuery.

8. Follow the instructions in the subsequent pages to configure the connectionto your external data source.

9. After you complete the configuration steps, clickSave.

Troubleshoot third party transfer setup

If you are having issues setting up your transfer, consult the appropriate thirdparty vendor. Contact information is available on the transfer's documentationpage in the Google Cloud Marketplace.

Query your data

When your data is transferred to BigQuery, the data iswritten to ingestion-time partitioned tables. For more information, seeIntroduction to partitioned tables.

If you query your tables directly instead of using the auto-generated views, youmust use the_PARTITIONTIME pseudocolumn in your query. For more information,seeQuerying partitioned tables.

What's next

• For an overview of BigQuery Data Transfer Service, seeIntroduction to BigQuery Data Transfer Service.
• For information on using transfers including getting information about atransfer configuration, listing transfer configurations, and viewing atransfer's run history, seeWorking with transfers.