Stream sharing with Pub/Sub

You can share Pub/Sub topics through BigQuery sharing to curateand distribute a library of streaming data across multiple internal and externalorganizational boundaries. Real-time streaming data is shared usingBigQuery sharing exchanges and listings, allowing you to logicallycategorize and group large sets of Pub/Sub topics and provision accessat scale.

You can share streaming data to do the following:

• Financial services:
  • Share rapidly changing instrument prices, quotes, and orders in real-time
  • Detect money laundering and payment fraud
  • Support trading risk computations
• Retail and consumer packaged goods (CPG):
  • Help stores manage inventory in real-time
  • Personalize marketing and customer support
  • Adjust prices dynamically
  • Monitor social media channels
  • Optimize physical store layouts
• Healthcare:
  • Power predictive algorithms to monitor patients and analyze risk in real time
  • Monitor vitals with wearable medical devices
  • Automatic acquisition, structuring, storage, and processing of data from patients, healthcare professionals, and facility administration
• Telecommunications:
  • Review network health and predict failures
  • Discover user behavior patterns to better locate devices and antennas

Required roles

The roles for sharing Pub/Sub topics are analogous to sharing BigQuery datasets:

• Administrator: a user that manages permissions, visibility, and membership ofexchanges and listings. This role is analogous to theAnalytics Hub Admin(roles/analyticshub.admin)IAM role.
• Shared topic publisher: a user that creates, manages, and grantsaccess to listings of the shared Pub/Sub topics. This role is analogousto theAnalytics Hub Publisher or Analytics Hub Listing AdminIAM role. The publisher referred to here is a user and not thepublisher applicationthat creates and sends the messages to the topic.
• Shared topic subscriber: a user that subscribes to the shared topiclisting. Shared topic subscribers can configure how topic messages will bedelivered. This role is analogous to theAnalytics Hub SubscriberorAnalytics Hub Subscription OwnerIAM role. Depending on how the message data is delivered,additional permissions may be required, such asPub/Sub service account permissionsfor writing messages to BigQuery.

Architecture

The following diagram describes how publishers and subscribers ofPub/Sub resources interact with BigQuery sharing:

Limitations

Stream sharing with Pub/Sub has the following limitations:

• A shared topic can have a maximum of 10,000 Pub/Sub subscriptions. This limit includes linked Pub/Sub subscriptions and Pub/Sub subscriptions created outside of Sharing, for example, subscriptions created directly from Pub/Sub.

• Shared topics are indexed in Data Catalog (deprecated) and Dataplex Universal Catalog, but you cannot filter specifically for its resource type.

• Usage metrics are captured and displayed in the Metrics dashboard forshared Pub/Sub topics and subscriptions, but are notavailable inINFORMATION_SCHEMA.SHARED_DATASET_USAGE.

  To monitor metrics, seeMonitor Pub/Sub inCloud Monitoring.

  These metrics can include the following:

  • Publish message count
  • Publish requests
  • Publish throughput in bytes
  • Top 5 subscriptions
  • Ingested bytes
  • Others

Enable the Analytics Hub API

To enable the Analytics Hub API, do the following:

gcloud services enable analyticshub.googleapis.com

Enable the Pub/Sub API

Go to the API library and enable the Pub/Sub API for your Google Cloud project.

Enable the Pub/Sub API

Shared topic publisher workflows

As a shared topic publisher, you can do the following:

• Add shared topics to an exchange by creating a listing
• Update a listing
• Delete a listing
• Share a listing
• Manage subscriptions to your listing
• Remove subscribers from your listing

Additional publisher permissions

To perform shared topic publisher tasks, you must have theAnalytics Hub Publisher role(roles/analyticshub.publisher) on an exchange or listing. To view all thedata exchangesacross projects in an organization that you have access to, you must have theresourcemanager.organizations.get permission.

In addition, you need thepubsub.topics.setIamPolicy permission on thePub/Sub topic you want to list. To share topics with a schema attached,you must grant BigQuery sharing subscribers thepubsub.schemas.getpermission on the schema attached to the topic. This permission allowsBigQuery sharing subscribers to properly parse messages from thePub/Sub subscription.

Create a listing (to add a shared topic)

To add a shared topic to a listing, select one of the following options:

POST https://analyticshub.googleapis.com/v1/projects/PROJECT_ID/location/LOCATION/dataExchanges/DATAEXCHANGE_ID/listings?listingId=LISTING_ID

Update a listing

To update a listing, select one of the following options:

PATCH https://analyticshub.googleapis.com/v1/projects/PROJECT_ID/location/LOCATION/dataExchanges/DATAEXCHANGE_ID/listings/LISTING_ID?updateMask=UPDATEMASK

Delete a listing

Listings with shared Pub/Sub topics cannot be deleted with active subscriptions.Revoke all active subscriptions before attempting to delete a shared topic listing. Once a listing is deleted, you cannot undo it. To delete a listing, select one of the following options:

DELETE https://analyticshub.googleapis.com/v1/projects/PROJECT_ID/location/LOCATION/dataExchanges/DATAEXCHANGE_ID/listings/LISTING_ID

Share a listing

If you want to give users access to a private listing, you must set IAM policy for an individual or a group for that listing. For a commercial listing, yourdata exchange must be public. Listings in a public data exchange appears in Sharing for allGoogle Cloud users (allAuthenticatedUsers). To enable users to request access for commercial listings, you must grant users theAnalytics Hub Viewer role(roles/analyticshub.viewer).

To give users access to view or subscribe to your listing, do the following steps:

1. Go to theSharing (Analytics Hub) page.

   Go to Sharing (Analytics Hub)

2. Click the data exchange name that contains the listing.

3. Click the listing for which you want to add shared topic subscribers.

4. ClickpersonSet permissions.

5. To add principals, clickperson_addAdd principal.

6. In theNew principals field, add the following details based on the type of listing:

   • For a private listing, enter emails of the identity to whom you want to grant access.
   • For a public listing, addallAuthenticatedUsers.

7. In theSelect a role menu, selectAnalytics Hub, and then based on the type of listing, select one of the following roles:

   • For a commercial listing, select theAnalytics Hub Viewer role. Thisrole lets usersview the listing and request access.
   • For a private or non-commercial public listing, select theAnalytics Hub Subscriber role. This role lets userssubscribe to your listing.

8. ClickSave.

For more information, seeAnalytics Hub Subscriber and Viewer roles.

Manage subscriptions

To allow a user to subscribe to a listing with a shared topic, grant them theAnalytics Hub Subscriber(roles/analyticshub.subscriber) andAnalytics Hub Subscription Owner(roles/analyticshub.subscriptionOwner) roles on a specific listing:

1. Go to theSharing (Analytics Hub) page.

   Go to Sharing (Analytics Hub)

2. Click the data exchange name that contains the listing for which you want to manage the subscriptions.

3. Click the listing for which you want to list all shared topic subscribers.

4. ClickpersonSet permissions.

5. To add principals, clickperson_addAdd principal.

6. In theNew principals field, enter the usernames or emails of the subscribers that you want to add.

7. ForSelect a role, selectAnalytics Hub> Analytics Hub Subscriber.

8. Clickadd_boxAdd another role.

9. ForSelect a role, selectAnalytics Hub> Analytics Hub Subscription Owner.

10. ClickSave.

You can delete and update subscribers at any time by clickingSet Permissions.

Revoke a subscription

To remove a subscription to a shared topic listing from BigQuery sharing, select one of the following options:

POST https://analyticshub.googleapis.com/v1/projects/PROJECT_ID/location/LOCATION/subscriptions/SUBSCRIPTION_ID:revoke

By revoking a subscription from BigQuery sharing, a shared topic subscriberno longer receives message data from the shared topic. The Pub/Subsubscription is detached from the shared topic. If a subscription is deleteddirectly from Pub/Sub, the BigQuery sharing subscription willpersist and need to be cleaned up.

Subscriber workflows

As a BigQuery sharing subscriber, you can view and subscribe to a listing.Subscribing to a listing of a shared topic creates one linked Pub/Subsubscription in the subscriber's project. The Pub/Sub subscription will bevisible in the shared topic publisher's project.

Additional subscriber permissions

You must have theAnalytics Hub Subscriber(roles/analyticshub.subscriber) role granted at the project, exchange, or listing-level to perform subscriber tasks.

Subscribe to a shared topic listing

To subscribe to a listing with a shared topic, select one of the following options:

POST https://analyticshub.googleapis.com/v1/projects/PROJECT_ID/location/LOCATION/dataExchanges/DATAEXCHANGE_ID/listings/LISTING_ID:subscribe

For more information about Pub/Sub subscriptions, seesubscription overview.

Pricing

There is no additional charge for Pub/Sub topic publishers to list andshare topics through BigQuery sharing. Shared topic publishers are chargedfor the total number of bytes written (publish throughput) to the shared topicand network egress (if applicable). Shared topic subscribers are charged for thetotal number of bytes read (subscribe throughput) from the linked subscription andnetwork egress (if applicable). SeePub/Sub pricingfor additional details.

VPC Service Controls

For shared topics from projects that have VPC Service Controls perimeters, you can set the appropriate ingress and egress rules required to grant access to shared topic publishers and subscribers.