Schedule a backfill run Stay organized with collections Save and categorize content based on your preferences.
Initiate a data backfill to load historical data into BigQuery. For information about how much data is available for backfill, see the documentation for your data source.
Explore further
For detailed documentation that includes this code sample, see the following:
Code sample
Java
Before trying this sample, follow theJava setup instructions in theBigQuery quickstart using client libraries. For more information, see theBigQueryJava API reference documentation.
To authenticate to BigQuery, set up Application Default Credentials. For more information, seeSet up authentication for client libraries.
importcom.google.api.gax.rpc.ApiException;importcom.google.cloud.bigquery.datatransfer.v1.DataTransferServiceClient;importcom.google.cloud.bigquery.datatransfer.v1.ScheduleTransferRunsRequest;importcom.google.cloud.bigquery.datatransfer.v1.ScheduleTransferRunsResponse;importcom.google.protobuf.Timestamp;importjava.io.IOException;importorg.threeten.bp.Clock;importorg.threeten.bp.Instant;importorg.threeten.bp.temporal.ChronoUnit;// Sample to run schedule back fill for transfer configpublicclassScheduleBackFill{publicstaticvoidmain(String[]args)throwsIOException{// TODO(developer): Replace these variables before running the sample.StringconfigId="MY_CONFIG_ID";Clockclock=Clock.systemDefaultZone();Instantinstant=clock.instant();TimestampstartTime=Timestamp.newBuilder().setSeconds(instant.minus(5,ChronoUnit.DAYS).getEpochSecond()).setNanos(instant.minus(5,ChronoUnit.DAYS).getNano()).build();TimestampendTime=Timestamp.newBuilder().setSeconds(instant.minus(2,ChronoUnit.DAYS).getEpochSecond()).setNanos(instant.minus(2,ChronoUnit.DAYS).getNano()).build();scheduleBackFill(configId,startTime,endTime);}publicstaticvoidscheduleBackFill(StringconfigId,TimestampstartTime,TimestampendTime)throwsIOException{try(DataTransferServiceClientclient=DataTransferServiceClient.create()){ScheduleTransferRunsRequestrequest=ScheduleTransferRunsRequest.newBuilder().setParent(configId).setStartTime(startTime).setEndTime(endTime).build();ScheduleTransferRunsResponseresponse=client.scheduleTransferRuns(request);System.out.println("Schedule backfill run successfully :"+response.getRunsCount());}catch(ApiExceptionex){System.out.print("Schedule backfill was not run."+ex.toString());}}}Python
Before trying this sample, follow thePython setup instructions in theBigQuery quickstart using client libraries. For more information, see theBigQueryPython API reference documentation.
To authenticate to BigQuery, set up Application Default Credentials. For more information, seeSet up authentication for client libraries.
importdatetimefromgoogle.cloud.bigquery_datatransfer_v1import(DataTransferServiceClient,StartManualTransferRunsRequest,)# Create a client objectclient=DataTransferServiceClient()# Replace with your transfer configuration nametransfer_config_name="projects/1234/locations/us/transferConfigs/abcd"now=datetime.datetime.now(datetime.timezone.utc)start_time=now-datetime.timedelta(days=5)end_time=now-datetime.timedelta(days=2)# Some data sources, such as scheduled_query only support daily run.# Truncate start_time and end_time to midnight time (00:00AM UTC).start_time=datetime.datetime(start_time.year,start_time.month,start_time.day,tzinfo=datetime.timezone.utc)end_time=datetime.datetime(end_time.year,end_time.month,end_time.day,tzinfo=datetime.timezone.utc)requested_time_range=StartManualTransferRunsRequest.TimeRange(start_time=start_time,end_time=end_time,)# Initialize request argument(s)request=StartManualTransferRunsRequest(parent=transfer_config_name,requested_time_range=requested_time_range,)# Make the requestresponse=client.start_manual_transfer_runs(request=request)# Handle the responseprint("Started manual transfer runs:")forruninresponse.runs:print(f"backfill:{run.run_time} run:{run.name}")What's next
To search and filter code samples for other Google Cloud products, see theGoogle Cloud sample browser.
Except as otherwise noted, the content of this page is licensed under theCreative Commons Attribution 4.0 License, and code samples are licensed under theApache 2.0 License. For details, see theGoogle Developers Site Policies. Java is a registered trademark of Oracle and/or its affiliates.