Schedule data preparations

This document describes how to schedule data preparation pipelinesand perform manual runs.

Data preparations are powered byDataform.Each data preparation schedule is run using your Google Account user credentialsor aDataform service accountthat you select when you configure the schedule or test run.

Changes you make to the data preparation steps aren't automatically saved. Youmust save and deploy the changes before they can be executed with a schedule.Schedules always run the latest deployed version of your data preparation andexclude any undeployed changes you might be developing.

Before you begin

Before you begin,create a data preparation.

Required roles

To authorize a data preparation with a service account whenmanually running the data preparation in developmentorscheduling the data preparation,you must grant the roles to the service account that you plan to use forexecuting the data preparation runs. For more information, seeGive access to the Dataform service account.

To enhance security for scheduling, seeImplement enhanced scheduling permissions.

Develop a data preparation

As you develop a data preparation, you can manually run the steps and inspectthe output before you deploy the changes to production. You can test the currentversion you're developing on your data, whileBigQuery continues to run the latest deployed version, according to aschedule. Before you can perform the run, you mustconfigure the destination,and fix any validation errors.

Manually run a data preparation in development

To test your data preparation steps and validate the results in your destinationtable, run the data preparation manually from the data preparation editor:

1. In the Google Cloud console, go to theBigQuery page.

   Go to BigQuery

2. In the left pane, clickexploreExplorer:

   

   If you don't see the left pane, clicklast_pageExpand left pane to open the pane.

3. In theExplorer pane, expand your project and clickData preparations.

4. Click the name of the data preparation that you want to run.

5. In the data preparation editor toolbar, clickMore> Configurerun now experience.

6. In theAuthentication section, authorize the data preparation withyour Google Account user credentials or a service account.

   • To use your Google Account user credentials(Preview), selectExecute with my user credentials. This is the default option.
   • To use a service account, selectExecute with selected service account, and then select a serviceaccount. If the service account needs additional permissions, grant itthe required roles by clickingGrant all.

7. ClickSave.

8. Fix any validation errors that appear.

9. From the data preparation editor toolbar, clickRun.

10. In theRun now dialog, clickConfirm to acknowledge that thismanual run writes data to a destination table, which you might also beusing for scheduled runs.

    If you selectedExecute with my user credentialsfor your authentication method, you mustauthorize your Google Account(Preview).

    The run then executes your steps and loads the output to thedestination.

11. Optional: After the run is complete, you can view the details about theexecution in theExecutions pane.

Deploy a data preparation

To schedule runs for a version of your data preparation, you must first deployit. Schedules run the most recently deployed version.

To deploy a data preparation, follow these steps:

1. In the Google Cloud console, go to theBigQuery page.

   Go to BigQuery

2. In the left pane, clickexploreExplorer:

   

3. In theExplorer pane, expand your project and clickData preparations.

4. Click the name of the chosen data preparation.

   The data preparation editor opens.

5. In the data preparation editor toolbar, clickDeploy.

Create a schedule

To create a schedule that executes the deployed data preparation steps andloads the prepared data into the destination table, you must first schedule adata preparation run. To schedule the run, you mustconfigure the destination,and fix any validation errors.

To create a data preparation schedule, follow these steps:

Authorize your Google Account

Preview

This product or feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of theService Specific Terms. Pre-GA products and features are available "as is" and might have limited support. For more information, see thelaunch stage descriptions.

Note: To request support or provide feedback for this feature, contactdataform-preview-support@google.com.

To authenticate the resource with yourGoogle Accountuser credentials, you must manually grant permission for BigQuerypipelines to get the access token for your Google Account and access the sourcedata on your behalf. You can grant manual approval with the OAuth dialoginterface.

You only need to give permission to BigQuery pipelines once.

To revoke the permission that you granted, follow these steps:

1. Go to yourGoogle Account page.
2. ClickBigQuery Pipelines.
3. ClickRemove access.

Warning: Revoking access permissions prevents any future pipeline runsthat this Google Account owns across all regions.

Changing the data preparation schedule owner by updating credentialsalso requires manual approval if the new Google Account owner has nevercreated a schedule before.

Manually run a scheduled data preparation

When you manually run a data preparation in a selected schedule,BigQuery executes the data preparation once,independently from the schedule.

To manually run a scheduled data preparation, follow these steps:

1. In the Google Cloud console, go to theScheduling page.

   Go to Scheduling

2. Click the name of the selected data preparation schedule.

3. On theSchedule details page, clickRun.

View schedules

You can view data preparation schedules from the data preparation editor or theScheduling page.

Edit a schedule

You can edit a schedule from the data preparation editor or theScheduling page.

Delete a schedule

To permanently delete a schedule for a selected data preparation, follow thesesteps:

1. In the Google Cloud console, go to theScheduling page.

   Go to Scheduling

2. In the row that contains the schedule, clickmore_vertActions> Delete.

What's next

• Learn how tocreate data preparations.
• Learn more aboutmanaging data preparations.