Work with Data Catalog

Caution: Data Catalog isdeprecated in favor ofDataplex Universal Catalog, which offers intelligent governance for data and AI assets across Google Cloud. Key Dataplex Universal Catalog capabilities are integrated with BigQuery and are also available in the BigQuery experience. SeeManage aspects and enrich metadata for details on enriching your data with aspects, which are the equivalent of Data Catalog tags.

Data Catalog integrates withBigQuery by automatically cataloging metadata aboutBigQuery resources like tables, datasets, views, and models. Thisdocument describes how to search these resources, view data lineage, and addtags by using Data Catalog.

Search for BigQuery resources

To use Data Catalog to search for BigQuery datasets, tables, and starred projects, follow these steps:

1. In the Google Cloud console, go to the Data CatalogSearch page.

   Go to Search

2. In theSearch field, enter a query, and then clickSearch.

   

   To refine your search parameters, use theFilters panel. For example, in theSystems section, select theBigQuery checkbox. The results are filtered to BigQuery systems.

You can perform basic searches in Data Catalog through theGoogle Cloud console. For more information about searching in the Google Cloud console,seeOpen a publicdataset.

Data lineage

Data lineage is aDataplex Universal Catalog feature thatlets you track how data moves through your systems: where it comes from, whereit is passed to, and what transformations are applied to it. You can access thedata lineage feature directly from BigQuery.

Enabling data lineage in your BigQuery project causesDataplex Universal Catalog to automatically record lineage information for tables createdby the following operations:

• Copy jobs.

• Query jobs that use the following data definition language (DDL) or data manipulation language (DML) statements in GoogleSQL:

  • CREATE TABLE (including theCREATE TABLE AS SELECT statement)
  • INSERT
  • UPDATE
  • DELETE
  • MERGE

Before you begin

In this section, you enable the Data Lineage API and grantIdentity and Access Management (IAM) roles that give users thenecessary permissions to perform each task in this document.

Enable data lineage

1. In the Google Cloud console, on the project selector page, select the project that contains the resources for which you want to track lineage.

   Go to project selector

2. Enable the Data Lineage API and Dataplex API.

   Enable the APIs

Note: Enabling the Data Lineage API may incur additional costs. For more information, seeData lineage considerations.

Required IAM roles

Lineage information is tracked automatically when you enable the Data Lineage API.

To get the permissions that you need to view lineage graphs, ask your administrator to grant you the following IAM roles:

• Data Catalog Viewer (roles/datacatalog.viewer) on a Data Catalog resource project.
• Data lineage viewer (roles/datalineage.viewer) on the project where you use systems supported by data lineage.
• BigQuery Metadata (roles/bigquery.metadataViewer)

For more information about granting roles, seeManage access to projects, folders, and organizations.

You might also be able to get the required permissions throughcustom roles or otherpredefined roles.

For more information, seeData lineage roles.

View lineage graphs in BigQuery

To view the data lineage graph from BigQuery follow these steps:

1. In the Google Cloud console, go to theBigQuery page.

   Go to BigQuery

2. In the left pane, clickexploreExplorer:

   

   If you don't see the left pane, clicklast_pageExpand left pane to open the pane.

3. In theExplorer pane, expand your project and then clickDatasets.

4. ClickOverview> Tables, and then select a table.

5. Click theLineage tab.

   

   Your data lineage graph is displayed.

   

6. Optional: Select a node to view additional details about the entities or processes involved in constructing lineage information.

For more information about data lineage, seeAbout data lineage.

Tags and tag templates

Tags let organizations create, search, andmanage metadata for all their data entries in a unified service.

This section explains two key Data Catalog concepts:

• Tags let you provide context for a data entry by attaching custom metadata fields.

• Tag templates are reusable structures that you can use to rapidly create new tags.

Tags

Data Catalog provides two types of tags: private tags and public tags.

Private tags

Private tags provide strict access controls. You can searchor view the tags and the data entries associated with the tags only if you aregranted therequired view permissions on both the private tag template and the dataentries.

Searching for private tags in the Data Catalog page requires thatyou use thetag: search syntax or the search filters.

Private tags are suitable for scenarios where you need to store some sensitiveinformation in the tag and you want to apply additional access restrictionsbeyond checking whether the user has the permissions to view the tagged entry.

Public tags

Public tags provide less strict access control for searching and viewing the tagas compared to private tags. Any user who has the required view permissions fora data entry can view all the public tags associated with it. View permissionsfor public tags are only required when you perform a search in Data Catalogusing thetag: syntax or when you view an unattached tag template.

Public tags support both simple search and search with predicates in the Data Catalog search page. When you create a tag template, the option to create apublic tag template is the default and recommended option in the Google Cloud console.

For example, let's assume you have a public tag template calledemployee datathat you used to create tags for three data entries calledName,Location,andSalary. Among the three data entries, only members of a specific groupcalledHR can view theSalary data entry. The other two data entrieshave view permissions for all employees of the company.

If any employee who is not a member of theHR group uses the Data Catalogsearch page and searches with the wordemployee, the search result displaysonlyName andLocation data entries with the associated public tags.

Public tags are useful for a broad set of scenarios. Public tags support simple search and search with predicates, whileprivate tags support only search with predicates.

Tag templates

To start tagging metadata, you first need to create one or more tag templates. A tagtemplate can be a public or private tag template. When you create a tagtemplate, the option to create a public tag template is the default andrecommended option in the Google Cloud console. A tag template is a group of metadatakey-value pairs calledfields. Having a set of templates is similar tohaving a database schema for your metadata.

You can structure your tags by topic. For example:

• Adata governance tag with fields for data governor, retention date,deletion date, PII (yes or no), data classification (public, confidential,sensitive, regulatory)
• Adata quality tag with fields for quality issues, update frequency, SLOinformation
• Adata usage tag with fields for top users, top queries, average daily users

You can then mix and match tags, using only the tags relevant for each dataasset and your business needs.

View the tag template gallery

To help you get started, Data Catalog includes a gallery ofsample tag templates to illustrate common tagging use cases. Use these examplesto learn about the power of tagging, for inspiration, or as a starting point forcreating your own tagging infrastructure.

To use a tag template gallery, perform the following steps:

1. In the Google Cloud console, go to the Dataplex Universal CatalogTag templates page.

   Go to Tag templates

2. ClickCreate tag template.

   The template gallery is displayed as part of theCreate template page.

After you select a template from the gallery, you can use it just like anyother tag template. You can add or delete attributes and change anything inthe template to suit your business needs. You can then search for the templatefields and values using Data Catalog.

For more information about tags and tag templates, seeTags and tag templates.

Regional resources

Every tag template and tag is stored in a particularGoogle Cloudregion.You can use a tag template to create a tag in any region, so you don't need tocreate copies of your template if you have metadata entries spread across multipleregions.