Load YouTube Content Owner data into BigQuery

You can load data from YouTube Content Owner to BigQuery using theBigQuery Data Transfer Service for YouTube Content Owner connector. With theBigQuery Data Transfer Service, you can schedule recurring transfer jobs thatadd your latest data from YouTube Content Owner toBigQuery.

Connector overview

The BigQuery Data Transfer Service for the YouTube Content Owner connector supports the following options for your data transfer.

Data ingestion from YouTube Content Owner transfers

Refresh windows

Arefresh window is the number of days that a data transfer retrieves datawhen a data transfer occurs. For example, if the refresh window is three daysand a daily transfer occurs, the BigQuery Data Transfer Service retrieves all data fromyour source table from the past three days. In thisexample, when a daily transfer occurs, the BigQuery Data Transfer Service creates a newBigQuery destination table partition with a copy of your source table datafrom the current day, then automatically triggers backfill runs to update theBigQuery destination table partitions with your source table data from thepast two days. The automatically triggered backfill runs will either overwriteor incrementally update your BigQuery destination table,depending on whether or not incremental updates are supported in theBigQuery Data Transfer Service connector.

When you run a data transfer for the first time, the data transfer retrieves allsource data available within the refresh window. For example, if the refreshwindow is three days and you run the data transfer for the first time, theBigQuery Data Transfer Service retrieves all source data within three days.

To retrieve data outside the refresh window, such as historical data, or torecover data from any transfer outages or gaps, you can initiate or schedule abackfill run.

Limitations

• The minimum frequency that you can schedule a data transfer for is once every24 hours. By default, a data transfer starts at the time that you create thedata transfer. However, you can configure the transfer start time when youset up your transfer.
• The BigQuery Data Transfer Service does not support incremental data transfers during aYouTube Content Owner transfer. When you specify a date for a data transfer,all of the data that is available for that date is transferred.

Before you begin

Before you create a YouTube Content Owner data transfer:

• Verify that you have completed all actions required toenable the BigQuery Data Transfer Service.
• Create a BigQuery datasetto store YouTube data.
• Verify that you have aYouTube Content Owneraccount. A YouTube Content Owner is not the same as a YouTube channel.Typically, you only have a YouTube Content Owner account if you manage manydifferent channels.
• If you intend to setup transfer run notifications for Pub/Sub, youmust havepubsub.topics.setIamPolicy permissions. Pub/Subpermissions are not required if you just set up email notifications. For moreinformation, seeBigQuery Data Transfer Service run notifications.

Required permissions

Ensure that you have granted the following permissions.

Required BigQuery roles

To get the permissions that you need to create a BigQuery Data Transfer Service data transfer, ask your administrator to grant you theBigQuery Admin (roles/bigquery.admin) IAM role on your project. For more information about granting roles, seeManage access to projects, folders, and organizations.

This predefined role contains the permissions required to create a BigQuery Data Transfer Service data transfer. To see the exact permissions that are required, expand theRequired permissions section:

You might also be able to get these permissions withcustom roles or otherpredefined roles.

For more information, seeGrantbigquery.admin access.

Required YouTube roles

• YouTube Content Manager or YouTube Content Owner.

  A Content Manager is granted rights to administer YouTube content for aContent Owner. A Content Owner is an umbrella account that owns one or moreYouTube channels and the videos on those channels.

• Hide revenue data is unchecked in YouTube Content Owner report settings.

  For revenue-related reports to transfer, the YouTube reports permissionsettingHide revenue data should be unchecked for the user creating thetransfer.

  

Set up a YouTube Content Owner transfer

Setting up a YouTube Content Owner data transfer requires a:

• Content Owner ID: Provided by YouTube. When you login to YouTube as aContent Owner or Manager, your ID appears in the URL aftero=. For example,if the URL ishttps://studio.youtube.com/owner/AbCDE_8FghIjK?o=AbCDE_8FghIjK,the Content Owner ID isAbCDE_8FghIjK. To select a different Content Manageraccount, seeSign in to a Content Manager accountorYouTube Channel Switcher. Formore information on creating and managing your Content Manager account, seeConfigure Content Manager account settings.
• Table Suffix: A user-friendly name for the channel provided by you whenyou set up the transfer. The suffix is appended to the job ID to create thetable name, for examplereportTypeId_suffix. The suffix is used toprevent separate data transfers from writing to the same tables. The table suffixmust be unique across all transfers that load data into the same dataset, andthe suffix should be short to minimize the length of the resulting table name.

If you use theYouTube Reporting APIand have existing reporting jobs, the BigQuery Data Transfer Service loads your reportdata. If you don't have existing reporting jobs, setting up the data transferautomatically enables YouTube reporting jobs.

To set up a YouTube Content Owner data transfer:

bqmk\--transfer_config\--project_id=project_id\--target_dataset=dataset\--display_name=name\--params='parameters'\--data_source=data_source\--service_account_name=service_account_name

bq mk \--transfer_config \--target_dataset=mydataset \--display_name='My Transfer' \--params='{"content_owner_id":"abCDE_8FghIjK","table_suffix":"MT","configure_jobs":"true"}' \--data_source=youtube_content_owner

importcom.google.api.gax.rpc.ApiException;importcom.google.cloud.bigquery.datatransfer.v1.CreateTransferConfigRequest;importcom.google.cloud.bigquery.datatransfer.v1.DataTransferServiceClient;importcom.google.cloud.bigquery.datatransfer.v1.ProjectName;importcom.google.cloud.bigquery.datatransfer.v1.TransferConfig;importcom.google.protobuf.Struct;importcom.google.protobuf.Value;importjava.io.IOException;importjava.util.HashMap;importjava.util.Map;// Sample to create youtube content owner channel transfer configpublicclassCreateYoutubeContentOwnerTransfer{publicstaticvoidmain(String[]args)throwsIOException{// TODO(developer): Replace these variables before running the sample.finalStringprojectId="MY_PROJECT_ID";StringdatasetId="MY_DATASET_ID";StringcontentOwnerId="MY_CONTENT_OWNER_ID";StringtableSuffix="_test";Map<String,Value>params=newHashMap<>();params.put("content_owner_id",Value.newBuilder().setStringValue(contentOwnerId).build());params.put("table_suffix",Value.newBuilder().setStringValue(tableSuffix).build());TransferConfigtransferConfig=TransferConfig.newBuilder().setDestinationDatasetId(datasetId).setDisplayName("Your Youtube Owner Channel Config Name").setDataSourceId("youtube_content_owner").setParams(Struct.newBuilder().putAllFields(params).build()).build();createYoutubeContentOwnerTransfer(projectId,transferConfig);}publicstaticvoidcreateYoutubeContentOwnerTransfer(StringprojectId,TransferConfigtransferConfig)throwsIOException{try(DataTransferServiceClientclient=DataTransferServiceClient.create()){ProjectNameparent=ProjectName.of(projectId);CreateTransferConfigRequestrequest=CreateTransferConfigRequest.newBuilder().setParent(parent.toString()).setTransferConfig(transferConfig).build();TransferConfigconfig=client.createTransferConfig(request);System.out.println("Youtube content owner channel transfer created successfully :"+config.getName());}catch(ApiExceptionex){System.out.print("Youtube content owner channel transfer was not created."+ex.toString());}}}

Query your data

When your data is transferred to BigQuery, the data iswritten to ingestion-time partitioned tables. For more information, seePartitioned tables.

If you query your tables directly instead of using the auto-generated views, youmust use the_PARTITIONTIME pseudocolumn in your query. For more information,seeQuerying partitioned tables.

Troubleshoot YouTube Content Owner transfer setup

If you are having issues setting up your data transfer, seeYouTube transfer issuesinTroubleshooting transfer configurations.