Introduction to BigQuery pipelines

You can use BigQuery pipelines to automate and streamline yourBigQuery data processes. With pipelines, you can schedule andexecute code assets in sequence to improve efficiency and reduce manual effort.

Overview

Pipelines are powered byDataform.

A pipeline consists of one or more of the following code assets:

• Notebooks
• SQL queries
• Data preparations

You can use pipelines to schedule the execution of code assets. For example,you can schedule a SQL query to run daily and update a table with the mostrecent source data, which can then power a dashboard.

In a pipeline with multiple code assets, you define the execution sequence.For example, to train a machine learning model, you can create a workflow inwhich a SQL query prepares data, and then a subsequent notebook trains themodel using that data.

Capabilities

You can do the following in a pipeline:

• Create new or import existingSQL queries or notebooks into a pipeline.
• Schedule a pipeline to automaticallyrun at a specified time and frequency.
• Share a pipeline with users or groups you specify.
• Share a link to a pipeline.

Limitations

Pipelines are subject to the following limitations:

• Pipelines are available only in the Google Cloud console.
• You can't change the region for storing a pipeline after it is created.
• You can grant users or groups access to a selected pipeline, but you can'tgrant them access to individual tasks within the pipeline.

Set the default region for code assets

If this is the first time you are creating a code asset, you should set thedefault region for code assets. You can't change the region for a code assetafter it is created.

1. Go to theBigQuery page.

   Go to BigQuery

2. In theExplorer pane, find the project in which you have enabled codeassets.

3. Clickmore_vertView actions next to the project, and then clickChange my default code region.

4. ForRegion, select the region that you want to use for code assets.

5. ClickSelect.

For a list of supported regions, seeBigQuery Studio locations.

Supported regions

All code assets are stored in yourdefault region for code assets.Updating the default region changes the region for all code assets createdafter that point.

The following table lists the regions where pipelines are available:

	Region description	Region name	Details
Africa
	Johannesburg	africa-south1
Americas
	Columbus	us-east5
	Dallas	us-south1	Low CO2
	Iowa	us-central1	Low CO2
	Los Angeles	us-west2
	Las Vegas	us-west4
	Montréal	northamerica-northeast1	Low CO2
	N. Virginia	us-east4
	Oregon	us-west1	Low CO2
	São Paulo	southamerica-east1	Low CO2
	South Carolina	us-east1
Asia Pacific
	Hong Kong	asia-east2
	Jakarta	asia-southeast2
	Mumbai	asia-south1
	Seoul	asia-northeast3
	Singapore	asia-southeast1
	Sydney	australia-southeast1
	Taiwan	asia-east1
	Tokyo	asia-northeast1
Europe
	Belgium	europe-west1	Low CO2
	Frankfurt	europe-west3
	London	europe-west2	Low CO2
	Madrid	europe-southwest1	Low CO2
	Netherlands	europe-west4	Low CO2
	Turin	europe-west12
	Zürich	europe-west6	Low CO2
Middle East
	Doha	me-central1
	Dammam	me-central2

Quotas and limits

BigQuery pipelines are subject toDataform quotas and limits.

Pricing

The execution of BigQuery pipeline tasks incurs compute and storagecharges in BigQuery. For more information, seeBigQuery pricing.

Pipelines containing notebooks incur Colab Enterprise runtime chargesbased on thedefault machine type.For pricing details, seeColab Enterprise pricing.

Each BigQuery pipeline run is logged usingCloud Logging. Logging is automaticallyenabled for BigQuery pipeline runs, which can incurCloud Logging billing charges. For more information, seeCloud Logging pricing.

What's next

• Learn how tocreate pipelines.
• Learn how tomanage pipelines.
• Learn how toschedule pipelines.