This document describes how to configure Dataplex Universal Catalog metadata changefeeds to receive notifications through Pub/Sub when metadatawithin Dataplex Universal Catalog is created, updated, or deleted.

For more information about metadata change feeds, seeMetadata change feeds overview.

Before you begin

Familiarize yourself withPub/Sub andtheDataplex Universal Catalog API.

1. Enable the Dataplex Universal Catalog and Pub/Sub APIs.

   Roles required to enable APIs

   To enable APIs, you need the Service Usage Admin IAM role (roles/serviceusage.serviceUsageAdmin), which contains theserviceusage.services.enable permission.Learn how to grant roles.

   Enable the APIs

2. Create a Pub/Sub topic to receive notifications. For moreinformation, seeCreate a topic.

3. Installgcloud.Make sure the short alias ofgcloud is set.

4. Set an alias ofgcurl. This creates a shortcut that includes yourauthentication token and sets the JSON content-type for API requests:

   aliasgcurl='curl -H "Authorization: Bearer $(gcloud auth print-access-token)" -H "Content-Type: application/json"'

5. Set theDATAPLEX_API variable:

   DATAPLEX_API="dataplex.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION"

   Replace the following:

   • PROJECT_ID: your project ID where the Dataplex API is enabled
   • LOCATION: the location where the job runs—for example,us-central1,europe-west3, orasia-south1

Required roles and permissions

To configure metadata change feeds, ensure that you and theDataplex Universal Catalogservice account have therequired IAM roles and permissions.

User roles and permissions

To get the permissions that you need to export metadata and access metadata change feeds messages, ask your administrator to grant you the following IAM roles on the project or organization:

• Export metadata:Dataplex Entry Group Exporter (roles/dataplex.entryGroupExporter)
• Access metadata change feeds messages:Pub/Sub Subscriber (roles/pubsub.subscriber)

For more information about granting roles, seeManage access to projects, folders, and organizations.

These predefined roles contain the permissions required to export metadata and access metadata change feeds messages. To see the exact permissions that are required, expand theRequired permissions section:

You might also be able to get these permissions withcustom roles or otherpredefined roles.

Dataplex Universal Catalog service account roles and permissions

To ensure that the Dataplex Universal Catalog service account has the necessary permission to publish metadata change feeds messages, ask your administrator to grant the Dataplex Universal Catalog service account thePub/Sub Publisher (roles/pubsub.publisher) IAM role on the Pub/Sub topic.Important: You must grant this role to the Dataplex Universal Catalog service account,not to your user account. Failure to grant the role to the correct principal might result in permission errors. For more information about granting roles, seeManage access to projects, folders, and organizations.

This predefined role contains the pubsub.topics.publish permission, which is required to publish metadata change feeds messages.

Your administrator might also be able to give the Dataplex Universal Catalog service account this permission withcustom roles or otherpredefined roles.

Grant permissions to the Dataplex Universal Catalog service account

The Dataplex Universal Catalog service agent is created when youenable the Dataplex API. You can identify the service agent by its email:

service-PROJECT_NUMBER@gcp-sa-dataplex.iam.gserviceaccount.com

Here,PROJECT_NUMBER is the projectnumber of the project where you enabled the Dataplex API.

The Dataplex Universal Catalog service accountrequires the ability to publish messages to your Pub/Sub topic.To grant this permission, give the service account thePub/Sub Publisher role(roles/pubsub.publisher) on the Pub/Sub topic:

gcloudpubsubtopicsadd-iam-policy-bindingTOPIC_ID\--member="serviceAccount:service-PROJECT_NUMBER@gcp-sa-dataplex.iam.gserviceaccount.com"\--role="roles/pubsub.publisher"

Create a metadata change feed

To control which changes generate notifications, you can configure a metadatachange feed to monitor specific resources. You do this by specifying a scope,such as your entire organization, specific projects, or specific entry groups.While scope lets you definewhich resources to monitor, you can use filters tofurther refinewhen Dataplex Universal Catalog sends notifications.

For more information, seeMetadata change feed.

gcurl-XPOST-d"$(cat<<EOF{"scope":{"organizationLevel":true},"pubsubTopic":"projects/PROJECT_ID_PUBSUB/topics/TOPIC_ID"}EOF)""https://${DATAPLEX_API}/metadataFeeds?metadataFeedId=FEED_ID"

gcurl-XPOST-d"$(cat<<EOF{"scope":{"projects":["projects/PROJECT_ID_1","projects/PROJECT_ID_2"]},"pubsubTopic":"projects/PROJECT_ID_PUBSUB/topics/TOPIC_ID"}EOF)""https://${DATAPLEX_API}/metadataFeeds?metadataFeedId=FEED_ID"

gcurl-XPOST-d"$(cat<<EOF{"scope":{"entryGroups":["projects/PROJECT_ID/locations/LOCATION/entryGroups/ENTRY_GROUP_ID_1","projects/PROJECT_ID/locations/LOCATION/entryGroups/ENTRY_GROUP_ID_2"]},"pubsubTopic":"projects/PROJECT_ID_PUBSUB/topics/TOPIC_ID"}EOF)""https://${DATAPLEX_API}/metadataFeeds?metadataFeedId=FEED_ID"

gcurl-XPOST-d"$(cat<<EOF{"scope":{"projects":["projects/PROJECT_ID_1","projects/PROJECT_ID_2"]},"filter":{"entryTypes":["projects/PROJECT_ID/locations/global/entryTypes/bigquery-table"]},"pubsubTopic":"projects/PROJECT_ID_PUBSUB/topics/TOPIC_ID"}EOF)""https://${DATAPLEX_API}/metadataFeeds?metadataFeedId=FEED_ID"

{"name":"projects/PROJECT_ID/locations/LOCATION/operations/OPERATION_ID","metadata":{"@type":"type.googleapis.com/google.cloud.dataplex.v1.OperationMetadata","createTime":"2023-10-02T15:01:23Z","target":"projects/PROJECT_ID/locations/LOCATION/metadataFeeds/FEED_ID","verb":"create","apiVersion":"v1"},"done":false}

View metadata change feeds

You can view the details of a metadata change feed.

gcurl"https://${DATAPLEX_API}/metadataFeeds/FEED_ID"

List metadata change feeds

You can list the metadata change feeds in a project and location.

gcurl"https://${DATAPLEX_API}/metadataFeeds"

Update a metadata change feed

You can update the scope or filters of an existing metadata change feed.

gcurl-XPATCH-d"$(cat<<EOF{"filter":{"entryTypes":[]}}EOF)""https://${DATAPLEX_API}/metadataFeeds/FEED_ID?updateMask=filter"

Delete a metadata change feed

You can delete a metadata change feed if you no longer need to receivenotifications.

Deleting a metadata change feed stops the feed from publishing new metadatachanges to your Pub/Sub topic. However, it doesn't delete thetopic or the subscription associated with the feed. You must delete themmanually if they're no longer needed.

Before you delete the feed or the subscription, ensure that your subscriberapplications have processed all outstanding messages in thePub/Sub subscription.

gcurl-XDELETE\"https://${DATAPLEX_API}/metadataFeeds/FEED_ID"

Consume notification messages

After you configure a metadata change feed, Dataplex Universal Catalog publishesmessages to the specified Pub/Sub topic when metadata changesoccur. To consume these messages, create a Pub/Sub subscriptionto the topic.

For example, you can create a pull subscription and use the Google Cloud CLIto view messages:

1. Create a subscription:

   gcloudpubsubsubscriptionscreateSUBSCRIPTION_ID--topic=TOPIC_ID

   Replace the following:

   • SUBSCRIPTION_ID: the subscription ID that you want to create
   • TOPIC_ID: the Pub/Sub topic ID to which the metadata change feed messages are published.

2. Pull messages from the subscription:

   gcloudpubsubsubscriptionspullSUBSCRIPTION_ID--auto-ack--limit=10

   Replace the following:

   • SUBSCRIPTION_ID: the subscription ID from which you want to pull messages

For more information about processing Pub/Sub messages, seeReceive messages from a pull subscription. For information aboutthe message format, seeData payload.

What's next

• Read theMetadata change feeds overview.
• Learn aboutDataplex Universal Catalog IAM and access control.
• Troubleshoot metadata change feeds.