Manage pipelines

This document describes how to manageBigQuery pipelines,including how to schedule and delete pipelines.

This document also describes how to view and manage pipeline metadata inDataplex Universal Catalog.

Pipelines are powered byDataform.

Before you begin

1. Create a BigQuery pipeline.
2. To manage pipeline metadata in Dataplex Universal Catalog,ensure that theDataplex API is enabledin your Google Cloud project.

Required roles

To get the permissions that you need to manage pipelines, ask your administrator to grant you the following IAM roles:

• To delete pipelines:Dataform Admin (roles/dataform.Admin) on the pipeline
• To view and run pipelines:Dataform Viewer (roles/dataform.Viewer) on the project

For more information about granting roles, seeManage access to projects, folders, and organizations.

You might also be able to get the required permissions throughcustom roles or otherpredefined roles.

To manage pipeline metadata in Dataplex Universal Catalog,ensure that you have the requiredDataplex Universal Catalog roles

For more information about Dataform IAM, seeControl access with IAM.

Note: When you create a pipeline, BigQuery grants you theDataform Admin role(roles/dataform.admin) on that pipeline. All users with theDataform Admin role granted on the Google Cloud project have owner access toall pipelines created in the project.

View all pipelines

To view a list of all pipelines in your project, do the following:

1. In the Google Cloud console, go to theBigQuery page.

   Go to BigQuery

2. In the left pane, clickexploreExplorer:

   

   If you don't see the left pane, clicklast_pageExpand left pane to open the pane.

3. In theExplorer pane, expand your project and clickPipelines.

View past manual runs

To view past manual runs of a selected pipeline, follow these steps:

1. In the Google Cloud console, go to theBigQuery page.

   Go to BigQuery

2. In the left pane, clickexploreExplorer:

   

3. In theExplorer pane, expand your project, clickPipelines,and then select a pipeline.

4. ClickExecutions.

5. Optional: To refresh the list of past runs, clickRefresh.

Configure alerts for failed pipeline runs

Each pipeline has a corresponding Dataform repository ID.Each BigQuery pipeline run is logged inCloud Logging using the correspondingDataform repository ID. You can use Cloud Monitoring to observetrends in Cloud Logging logs forBigQuery pipeline runs and to notifyyou when conditions you describe occur.

To receive alerts when a BigQuery pipeline run fails,you can create a log-based alerting policy for the correspondingDataform repository ID. For instructions, seeConfigure alerts for failed workflow invocations.

To find the Dataform repository ID of your pipeline, do the following:

1. In the Google Cloud console, go to theBigQuery page.

   Go to BigQuery

2. In the left pane, clickexploreExplorer:

   

3. In theExplorer pane, expand your project, clickPipelines,and then select a pipeline.

4. ClickSettings.

   The Dataform repository ID of your pipeline is displayed atthe bottom of theSettings tab.

Delete a pipeline

To permanently delete a pipeline, follow these steps:

1. In the Google Cloud console, go to theBigQuery page.

   Go to BigQuery

2. In the left pane, clickexploreExplorer:

   

3. In theExplorer pane, expand your project and clickPipelines.

4. Find the pipeline that you want to delete.

5. Clickmore_vertView actions next to the pipeline, and then clickDelete.

6. ClickDelete.

Manage metadata in Dataplex Universal Catalog

Dataplex Universal Catalog lets you store and manage metadata forpipelines. Pipelines are available in Dataplex Universal Catalogby default, without additional configuration.

You can use Dataplex Universal Catalog to manage pipelinesin allpipeline locations.Managing pipelines in Dataplex Universal Catalogis subject toDataplex Universal Catalog quotas and limitsandDataplex Universal Catalog pricing.

Dataplex Universal Catalog automatically retrievesthe following metadata from pipelines:

• Data asset name
• Data asset parent
• Data asset location
• Data asset type
• Corresponding Google Cloud project

Dataplex Universal Catalog logs pipelines asentries with the followingentry values:

System entry group

Thesystem entry groupfor pipelines is@dataform. To view details of pipeline entriesin Dataplex Universal Catalog, you need to view thedataform system entry group.For instructions about how to view a list of all entries in an entry group, seeView details of an entry groupin the Dataplex Universal Catalog documentation.

System entry type

Thesystem entry typefor pipelines isdataform-code-asset. To view details ofpipelines,you need to view thedataform-code-asset system entry type,filter the results with an aspect-based filter,andset thetype field insidedataform-code-asset aspect toWORKFLOW.Then, select an entry of the selected pipeline.For instructions about how to view details of a selected entry type, seeView details of an entry typein the Dataplex Universal Catalog documentation.For instructions about how to view details of a selected entry, seeView details of an entryin the Dataplex Universal Catalog documentation.

System aspect type

Thesystem aspect typefor pipelines isdataform-code-asset. Toprovide additional context to pipelines in Dataplex Universal Catalogby annotating data pipeline entries withaspects,view thedataform-code-asset aspect type,filter the results with an aspect-based filter,andset thetype field insidedataform-code-asset aspect toWORKFLOW.For instructions about how to annotate entries with aspects, seeManage aspects and enrich metadatain the Dataplex Universal Catalog documentation.

Type

The type for data canvases isWORKFLOW.This type lets you filter pipelines in thedataform-code-assetsystem entry type and thedataform-code-asset aspect type by using theaspect:dataplex-types.global.dataform-code-asset.type=WORKFLOWquery in anaspect-based filter.

For instructions about how to search for assets in Dataplex Universal Catalog, seeSearch for data assets in Dataplex Universal Catalogin the Dataplex Universal Catalog documentation.

What's next

• Learn more aboutBigQuery pipelines.
• Learn how tocreate pipelines.
• Learn how toschedule pipelines.