Schedule pipelines

This document describes how to scheduleBigQuery pipelines,including how to schedule pipelines and inspect scheduled pipeline runs.

Pipelines are powered byDataform.Each pipeline schedule is run using your Google Account user credentialsor aDataform service accountthat you select when you configure the schedule.

Changes you make to a pipeline are automatically saved,but are available only to you and to users granted the Dataform Admin role onthe project. To update the schedule with a new versionof the pipeline, you need todeploy the pipeline.Deploying updates the schedule to use your current version of the pipeline.Schedules always run the latest deployed version.

Schedules of pipelines that contain notebooks use adefault runtime specification.During a scheduled run of apipeline containing notebooks, BigQuery writes notebook output to theCloud Storage bucket selected duringschedule creation.

Before you begin

Before you begin,create a pipeline.

Enable pipeline scheduling

To schedule pipelines, you must grant the following roles to theservice account which you plan to use forpipeline schedules:

Service Account User (roles/iam.serviceAccountUser)
FollowGrant a single role on a service accountto add your service account as a principal toitself. In other words, add the service accountas a principal to the same service account.Then, grant the Service Account User role to this principal.

If your pipeline contains SQL queries, you must grant thefollowing roles to the service account which you plan to use forpipeline schedules:

BigQuery Job User (roles/bigquery.jobUser)
FollowGrant a single role on a projectto grant the BigQuery Job User role to yourservice account on projects from which your pipelines read data.
BigQuery Data Viewer (roles/bigquery.dataViewer)
FollowGrant a single role on a projectto grant the BigQuery Data Viewer role to yourservice account on projects from which your pipelines read data.
BigQuery Data Editor (roles/bigquery.dataEditor)
FollowGrant a single role on a projectto grant the BigQuery Data Editor role to yourservice account on projects to which your pipelines write data.

If your pipeline contains notebooks, you must grant thefollowing roles to the service account which you plan to use forpipeline schedules:

Notebook Executor User (roles/aiplatform.notebookExecutorUser)
FollowGrant a single role on a projectto grant the Notebook Executor User role to yourservice account on the selected project.
Storage Admin (roles/storage.admin)
FollowAdd a principal to a bucket-level policyto add your service account as a principal to theCloud Storage bucket which you plan to use for storing output ofnotebooks executed in scheduled pipeline runs,and grant the Storage Admin role to this principal.

Additionally, you must grant the following role to thedefault Dataform service agent:

Service Account Token Creator (roles/iam.serviceAccountTokenCreator)
FollowGrant token creation access to a service accountto add the default Dataform service agent as a principal to yourservice account, and grant the Service Account Token Creator roleto this principal.

To learn more about service accounts in Dataform, seeAbout service accounts in Dataform.

Required roles

To get the permissions that you need to manage pipelines, ask your administrator to grant you the following IAM roles:

  • Delete pipelines:Dataform Admin (roles/dataform.Admin) on the pipeline
  • Create, edit, run, and delete pipeline schedules:Dataform Admin (roles/dataform.Admin) on the pipeline
  • View and run pipelines:Dataform Viewer (roles/dataform.Viewer) on the project
  • View pipeline schedules:Dataform Editor (roles/dataform.Editor) on the project

For more information about granting roles, seeManage access to projects, folders, and organizations.

You might also be able to get the required permissions throughcustom roles or otherpredefined roles.

To enhance security for scheduling, seeImplement enhanced scheduling permissions.

For more information about Dataform IAM, seeControl access with IAM.

To use Colab notebook runtime templates when scheduling pipelines,you need theNotebook Runtime User role(roles/aiplatform.notebookRuntimeUser).

Create a pipeline schedule

To create a pipeline schedule, follow these steps:

Explorer pane

  1. In the Google Cloud console, go to theBigQuery page.

    Go to BigQuery

  2. In the left pane, clickExplorer:

    Highlighted button for the Explorer pane.

    If you don't see the left pane, clickExpand left pane to open the pane.

  3. In theExplorer pane, expand your project, clickPipelines,and then select a pipeline.

  4. ClickSchedule.

  5. In theSchedule pipeline pane, in theSchedule name field,enter a name for the schedule.

  6. In theAuthentication section, authorize thepipeline with your Google Account user credentials or a serviceaccount.

    • To use your Google Account user credentials(Preview), selectExecute with my user credentials.
    • To use a service account, selectExecute with selected service account, and then select a service account.

  7. If your pipeline contains a notebook, in theNotebook options section,in theRuntime template field, select a Colab notebook runtime templateor the default runtime specifications. For details on creating a Colabnotebook runtime template, seeCreate a runtime template.

    Note: A notebook runtime template must be in the same region as thepipeline.Note: If you don't have therequired rolefor using Colab notebook runtime templates, you can still run and schedulepipelines with the default runtime specifications.

  8. If your pipeline contains a notebook, in theNotebook options section, in theCloud Storage bucket field,clickBrowse and select or create a Cloud Storage bucket forstoring the output of notebooks in your pipeline.

    Your selected service account must be granted theStorage Admin IAM role on the selected bucket.For more information, seeEnable pipeline scheduling.

  9. In theSchedule frequency section, do the following:

    1. In theRepeats menu, select the frequency of scheduledpipeline runs.
    2. In theAt time field, enter the time for scheduled pipeline runs.
    3. In theTimezone menu, select the timezone for the schedule.

  10. Set the BigQuery query job priority with theExecute as interactive job with high priority (default) option.By default, BigQuery runs queries asinteractive query jobs,which are intended to start running as quickly as possible.Clearing this option runs the queries asbatch query jobs,which have lower priority.

  11. ClickCreate schedule. If you selectedExecute with my user credentialsfor your authentication method, you mustauthorize your Google Account(Preview).

When you create the schedule, the current version of the pipeline isautomatically deployed. To update the schedule with a new version of thepipeline,deploy the pipeline.

The latest deployed version of the pipeline runs at the selected time andfrequency.

Scheduling page

  1. In the Google Cloud console, go to theScheduling page.

    Go to Scheduling

  2. ClickCreate, and then selectPipeline schedule from the menu.

  3. In theSchedule pipeline pane, select a pipeline to schedule.

  4. In theSchedule name field, enter a name for the schedule.

  5. In theAuthentication section, authorize thepipeline with your Google Account user credentials or a serviceaccount.

    • To use your Google Account user credentials(Preview), selectExecute with my user credentials.
    • To use a service account, selectExecute with selected service account, and then select a service account.

  6. If your pipeline contains a notebook, in theNotebook options section,in theRuntime template field, select a Colab notebook runtime templateor the default runtime specifications. For details on creating a Colabnotebook runtime template, seeCreate a runtime template.

    Note: A notebook runtime template must be in the same region as thepipeline.Note: If you don't have therequired rolefor using Colab notebook runtime templates, you can still run and schedulepipelines with the default runtime specifications.

  7. If your pipeline contains a notebook, in theCloud Storage bucket field,clickBrowse and select or create a Cloud Storage bucket forstoring the output of notebooks in your pipeline.

    Your selected service account must be granted theStorage Admin IAM role on the selected bucket.For more information, seeEnable pipeline scheduling.

  8. In theSchedule frequency section, do the following:

    1. In theRepeats menu, select the frequency of scheduled pipeline runs.
    2. In theAt time field, enter the time for scheduled pipeline runs.
    3. In theTimezone menu, select the timezone for the schedule.

  9. Set the BigQuery query job priority with theExecute as interactive job with high priority (default) option.By default, BigQuery runs queries asinteractive query jobs,which are intended to start running as quickly as possible.Clearing this option runs the queries asbatch query jobs,which have lower priority.

  10. ClickCreate schedule. If you selectedExecute with my user credentialsfor your authentication method, you mustauthorize your Google Account(Preview).

Authorize your Google Account

Preview

This product or feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of theService Specific Terms. Pre-GA products and features are available "as is" and might have limited support. For more information, see thelaunch stage descriptions.

Note: To request support or provide feedback for this feature, contactdataform-preview-support@google.com.

To authenticate the resource with yourGoogle Accountuser credentials, you must manually grant permission for BigQuerypipelines to get the access token for your Google Account and access the sourcedata on your behalf. You can grant manual approval with the OAuth dialoginterface.

You only need to give permission to BigQuery pipelines once.

To revoke the permission that you granted, follow these steps:

  1. Go to yourGoogle Account page.
  2. ClickBigQuery Pipelines.
  3. ClickRemove access.
Warning: Revoking access permissions prevents any future pipeline runsthat this Google Account owns across all regions.

Changing the pipeline schedule owner by updating credentials alsorequires manual approval if the new Google Account owner has never created aschedule before.

If your pipeline contains a notebook, you must also manually grantpermission for Colab Enterprise to get the access token for yourGoogle Account and access the source data on your behalf. You only needto give permission once. You can revoke this permission on theGoogle Account page.

Deploy a pipeline

Deploying a pipeline updates its schedule with the current version of thepipeline. Schedules run the latest deployed version of the pipeline.

To deploy a pipeline, follow these steps:

  1. In the Google Cloud console, go to theBigQuery page.

    Go to BigQuery

  2. In the left pane, clickExplorer:

    Highlighted button for the Explorer pane.

  3. In theExplorer pane, expand your project, clickPipelines,and then select a pipeline.

  4. ClickDeploy.

The corresponding schedule is updated with the current version of the pipeline.The latest deployed version of the pipeline runs at the scheduled time.

Disable a schedule

To pause the scheduled runs of a selected pipeline without deleting the schedule,you can disable the schedule.

To disable a schedule for a selected pipeline, follow these steps:

Explorer pane

  1. In the Google Cloud console, go to theBigQuery page.

    Go to BigQuery

  2. In the left pane, clickExplorer:

    Highlighted button for the Explorer pane.

  3. In theExplorer pane, expand your project, clickPipelines,and then select a pipeline.

  4. ClickView schedule.

  5. In theSchedule details table, in theSchedule state row,click theSchedule is enabled toggle.

Scheduling page

  1. In the Google Cloud console, go to theScheduling page.

    Go to Scheduling

  2. Click the name of the selected pipeline.

  3. On theSchedule details page, clickDisable.

Enable a schedule

To resume scheduled runs of a disabled pipeline schedule, follow these steps:

Explorer pane

  1. In the Google Cloud console, go to theBigQuery page.

    Go to BigQuery

  2. In the left pane, clickExplorer:

    Highlighted button for the Explorer pane.

  3. In theExplorer pane, expand your project, clickPipelines,and then select a pipeline.

  4. ClickView schedule.

  5. In theSchedule details table, in theSchedule state row,click theSchedule is disabled toggle.

Scheduling page

  1. In the Google Cloud console, go to theScheduling page.

    Go to Scheduling

  2. Click the name of the selected pipeline.

  3. On theSchedule details page, clickEnable.

Manually run a deployed pipeline

When you manually run a pipeline deployed in a selected schedule,BigQuery executes the deployed pipeline once,independently from the schedule.

To manually run a deployed pipeline, follow these steps:

  1. In the Google Cloud console, go to theScheduling page.

    Go to Scheduling

  2. Click the name of the selected pipeline schedule.

  3. On theSchedule details page, clickRun.

View all pipeline schedules

To view all pipeline schedules in your Google Cloud project, follow these steps:

  1. In the Google Cloud console, go to theScheduling page.

    Go to Scheduling

  2. Optional: To display additional columns with pipeline schedule details,clickColumn display options,and then select columns and clickOK.

View pipeline schedule details

To view details for a selected pipeline schedule, follow these steps:

Explorer pane

  1. In the Google Cloud console, go to theBigQuery page.

    Go to BigQuery

  2. In the left pane, clickExplorer:

    Highlighted button for the Explorer pane.

  3. In theExplorer pane, expand your project, clickPipelines,and then select a pipeline.

  4. ClickView schedule.

Scheduling page

  1. In the Google Cloud console, go to theScheduling page.

    Go to Scheduling

  2. Click the name of the selected pipeline schedule.

View past scheduled runs

To view past runs of a selected pipeline schedule, follow these steps:

Explorer pane

  1. In the Google Cloud console, go to theBigQuery page.

    Go to BigQuery

  2. In the left pane, clickExplorer:

    Highlighted button for the Explorer pane.

  3. In theExplorer pane, expand your project, clickPipelines,and then select a pipeline.

  4. ClickExecutions.

  5. Optional: To refresh the list of past runs, clickRefresh.

Scheduling page

  1. In the Google Cloud console, go to theScheduling page.

    Go to Scheduling

  2. Click the name of the selected pipeline.

  3. On theSchedule details page, in thePast executions section,inspect past runs.

  4. Optional: To refresh the list of past runs, clickRefresh.

Edit a pipeline schedule

To edit a pipeline schedule, follow these steps:

Explorer pane

  1. In the Google Cloud console, go to theBigQuery page.

    Go to BigQuery

  2. In the left pane, clickExplorer:

    Highlighted button for the Explorer pane.

  3. In theExplorer pane, expand your project, clickPipelines,and then select a pipeline.

  4. ClickView schedule, and then clickEdit.

  5. In theSchedule pipeline dialog, edit the schedule,and then clickUpdate schedule.

Scheduling page

  1. In the Google Cloud console, go to theScheduling page.

    Go to Scheduling

  2. Click the name of the selected pipeline.

  3. On theSchedule details page, clickEdit.

  4. ClickView schedule, and then clickEdit.

  5. In theSchedule pipeline dialog, edit the schedule,and then clickUpdate schedule.

Delete a pipeline schedule

To permanently delete a pipeline schedule, follow these steps:

  1. In the Google Cloud console, go to theScheduling page.

    Go to Scheduling

  2. Do either of the following:

    • Click the name of the selected pipeline schedule, and thenon theSchedule details page, clickDelete.

    • In the row that contains the selected pipeline schedule, clickView actions in theActions column, and then clickDelete.

  3. In the dialog that appears, clickDelete.

What's next

Except as otherwise noted, the content of this page is licensed under theCreative Commons Attribution 4.0 License, and code samples are licensed under theApache 2.0 License. For details, see theGoogle Developers Site Policies. Java is a registered trademark of Oracle and/or its affiliates.

Last updated 2025-12-15 UTC.