Introduction to BigQuery sharing

BigQuery sharing (formerly Analytics Hub) is a data exchange platform thatlets you share data and insights at scale across organizational boundaries witha robust security and privacy framework. Sharing lets youdiscover and access a data library curated by various data providers. This datalibrary also includes Google-provided datasets.

For example, you can use sharing to augment youranalytics and ML initiatives with third-party and Google datasets.

Analytics Hub Identity and Access Management (IAM) roles let you perform thefollowing sharing tasks:

• As an Analytics Hub Publisher, you can share data with your partner networkor within your own organization in real time.Listings let youshare data without replicating the shared data, and they can be monetized ontheGoogle Cloud Marketplaceor through your own channels. You can build a catalog of analytics-ready datasources with granular permissions that let you deliver data to the rightaudiences. You can also manage subscriptions and view the usage metrics foryour listings.

• As an Analytics Hub Subscriber, you can discover the data that you are lookingfor, combine shared data with your existing data, and use thebuilt-in features of BigQuery.When you subscribe to a listing, alinked dataset orlinked Pub/Sub subscription is created in your project. You can manageyour subscriptions by using theSubscription resource,which stores relevant information about the subscriber and represents theconnection between publisher and subscriber.

• As an Analytics Hub Viewer, you can browse through the shared resources thatyou have access to in BigQuery sharing and make a request tothe publisher to access the shared data. You can discoverCloud Marketplace-integrated commercial listings on both BigQuery sharing and Cloud Marketplace.

• As an Analytics Hub Admin, you can createdata exchangesthat enable data sharing, and then give permissions to data publishers andsubscribers to access these data exchanges.

For more information, seeConfigure Analytics Hub roles.

Architecture

Sharing is built on a publish and subscribe model ofGoogle Cloud data resources, allowing for zero-copy sharing in place.Sharing supports the following Google Cloud resources:

• BigQuery datasets
• Pub/Sub topics

Publisher workflow

The following diagram describes how a publisher shares assets:

The following sections describe the features in this workflow.

Shared resources

Shared resources are the unit of sharing by a publisher inBigQuery sharing.

Shared datasets

A shared dataset is a BigQuery dataset that is the unit ofdata sharing in BigQuery sharing. The separation of compute and storagein the BigQuery architecture enables data publishers to share datasetswith as many subscribers as they want, without having to make multiple copies ofthe data. As a publisher, you create or use an existing BigQuerydataset in your project with the following supported objects that you want todeliver to your subscribers:

• Authorized views
• Authorized datasets
• BigQuery ML models
• External tables
• Materialized views
• Routines
  • User-defined functions (UDFs)
  • Table functions
  • SQL stored procedures
• Tables
• Table snapshots
• Views

Shared datasets supportcolumn-level securityandrow-level security.

Be aware of the following limitations regarding VPC Service Controls and sharing:

• Publishers are not recommended to publish shared data in projects insideVPC Service Controls perimeters. If shared datasets in a project are withina VPC Service Controls perimeters, appropriateingress and egress rulesare required for both the exchange project (hosted listings) and all of thesubscriber's projects to successfully subscribe to the publisher's listings.

• Exchange admins are not recommended to put their exchange project in aVPC Service Controls perimeter, as it might interrupt the publishingworkflows, requiringingress and egress rulesfor the publisher project and all of the subscribers' projects to successfullysubscribe to their listings.

Shared topics

A shared topic is aPub/Sub topicthat is the unit ofstreaming data sharing in BigQuery.As a publisher, you create or use an existing Pub/Sub topic in yourproject and distribute that with your subscribers.

Data exchanges

A data exchange is a container that enables self-service data sharing. Itcontains listings that reference shared resources. Publishers and administratorscan grant access to subscribers at the exchange and the listing level. Thismethod helps to avoid granting access on the underlying shared resourcesexplicitly. A subscriber can browse through data exchanges, discover data thatthey can access, and subscribe to shared resources. When youcreate a dataexchange, youcan assign a primary contact email to it. The primary contact email provides away for users to contact the owner of a data exchange with questions or concernsabout the data exchange. A data exchange can be of the following types:

• Private data exchange. By default, a data exchange is private and onlyusers or groups that have access to that exchange can view or subscribe to itslistings.
• Public data exchange. By default, a data exchange is private and onlyusers or groups that have access to that exchange can view or subscribe to itslistings. However, you can choose to make a data exchange public. Listings inpublic data exchanges can bediscoveredandsubscribed tobyGoogle Cloud users (allAuthenticatedUsers).For more information about public data exchanges, seeMake a data exchange public.

The Analytics Hub Admin role lets you create multiple data exchanges and manageother users performing sharing tasks.

Listings

A listing is a reference to a shared resource that a publisher lists ina data exchange. As a publisher, you can create a listing and specify theresource description, sample queries to run or sample message data, links to anyrelevant documentation, and any additional information that can help subscribersto use your shared resource. When you create a listing, you can assign a primarycontact email, a provider name and contact, and a publisher name and contact.The primary contact email provides a way for users to contact the owner of alisting with questions or concerns about the data exchange. The provider nameand contact is the information of the agency that originally provided the datafor the listing. This information is optional. The publisher name and contact isthe agency that published the data for use in BigQuery sharing. Thisinformation is optional. For more information, seeManage listings.

A listing can be of the following two types based on theIdentity and Access Management (IAM) policy that is set for the listing and the type of dataexchange that contains the listing:

• Public listing. It is shared with allGoogle Cloud users (allAuthenticatedUsers).Listings in a public data exchange are public listings. These listings can bereferences of afree public resource or acommercial resource.If the listing is of a commercial resource, subscribers can either requestaccess to the listing directly from the data provider, or they canbrowse and purchaseGoogle Cloud Marketplace-integrated commercial listings.
• Private listing. It is shared directly with individuals orgroups. For example, a private listing can reference marketingmetrics dataset that you share with other internal teams within yourorganization.

Subscriber workflow

The following diagram describes how Analytics Hub subscribers interact withshared resources:

The following sections describe the features in the subscriber workflow.

Linked resources

Linked resources are created when subscribing to a BigQuery sharinglisting, connecting a subscriber to the underlying shared resource.

Linked datasets

A linked dataset is aread-only BigQuery dataset that serves asa pointer or reference to a shared dataset. Subscribing to a listing creates a linkeddataset in your project and not a copy of the dataset, so subscribers can readthe data but cannot add or update objects within it. When you query objectssuch as tables and views through a linked dataset, the data from the shareddataset is returned. For more information about linked datasets, seeView and subscribe to listings and data exchanges.

Linked datasets are authorized to access tables and views of a shared dataset.Subscribers with linked datasets access tables and views of a shared datasetwithout any additional Identity and Access Management authorization.

Linked datasets supports the following objects:

• Authorized views
• Authorized datasets
• Authorized routines

Linked Pub/Sub subscriptions

Subscribing to a listing with a shared topic creates a linked Pub/Subsubscription in the subscriber project. No copies of the shared topic or messagedata are created. Subscribers of the linkedPub/Sub subscriptioncan access the messages published to the shared topic. Subscribers accessthe message data of a shared topic without any additional Identity and Access Managementauthorization. Publishers can manage subscriptions both in Pub/Subdirectly or through BigQuery sharing subscription management. For moreinformation about linked Pub/Sub subscriptions, seeStream sharing with Pub/Sub.

Data egress options (BigQuery shared datasets only)

Data egress options letpublishers restrict the export by subscribers of data out of BigQuerylinked datasets.

Publishers can enable data egress restriction on a listing, the results of a query, orboth. When data egress is restricted, the following restrictions are applied:

• Copy, clone, export, and snapshot APIs are disabled.

• Copy, clone, export, and snapshot options in the Google Cloud console aredisabled.

• Connecting the restricted dataset to the table explorer is disabled.

• BigQuery Data Transfer Service is disabled on the restricted dataset.

• CREATE TABLE AS SELECT statementsandwriting to a destination table aredisabled.

• CREATE VIEW AS SELECT statementsand writing to a destination view are disabled.

When youcreate alisting, you canset the appropriate data egress options.

Limitations

Sharing has the following limitations:

• A shared dataset can have a maximum of 1,000 linked datasets.

• A shared topic can have amaximum of 10,000 Pub/Sub subscriptions. This limit includes linked Pub/Sub subscriptions and Pub/Sub subscriptions created outside of BigQuery sharing (for example, directly from Pub/Sub).

• A dataset with unsupported resources cannot be selected as a shared datasetwhen youcreate a listing.For more information about the BigQuery objects thatBigQuery sharing supports, seeShared datasetsin this document.

• You can't setIAM roles orIAMpolicies onindividual tables within a linked dataset. Apply them at the linked datasetlevel instead.

• You can't attachIAM tagson tables within a linked dataset. Apply them at the linked dataset levelinstead.

• Linked datasets created before July 25, 2023, aren't backfilled by thesubscription resource. Onlysubscriptions created after July 25, 2023 work with the API methods.

• If you are a publisher, the following BigQuery interoperabilitylimitations apply:

  • Subscribers must be given explicit permissions to read the sourcedataset to be able to query views within linked datasets. Togrant access to views, as a best practice publishers shouldcreate authorized views. Authorizedviews can grant subscribers access to the view data without giving themaccess to the underlying source data.

  • Thequery plan reveals the sharedview query and the routine query, including project IDs, and otherdatasets involved in authorized views. Never include anything such asencryption keys that you consider sensitive in the shared view or routinequery.

  • Shared datasets are indexed inData Catalog(deprecated) andDataplex Universal Catalog.Updates on a shared dataset, such as adding tables or views, are madeavailable to subscribers without any delay. However, in certainscenarios, for example, when there are more than one hundred subscribers ortables in a shared dataset, the updates might take up to 18 hours toget indexed in these services. Due to the delay inindexing, subscribers cannot search for these updatedresources in the Google Cloud console immediately.

  • Shared topics are indexed in Data Catalog (deprecated) andDataplex Universal Catalog, butyou cannot filter specifically for its resource type.

  • If you have set uprow-level securityordata masking policies onthe tables that are listed, then subscribers must be anEnterprise or Enterprise Plus customer to run the queryjob on the linked dataset. For information about editions, seeIntroduction to BigQuery editions.

• If you are a subscriber, the following BigQueryinteroperability limitations apply:

  • Materialized views that refer to tables in the linked dataset aren'tsupported.

  • Takingsnapshots of linked datasettables isn't supported.

  • Queries with linked datasets andJOIN statements that are larger than1 TB (physical storage) might fail. You cancontact support to resolve this issue.

  • You cannot useregion qualifierswithINFORMATION_SCHEMA views toview metadata for your linkeddataset.

  • The following limitations apply to listings for multiple regions:

  • Listings for multiple regions are supported only for shared datasetsand linked dataset replicas. Listings for multiple regions aren't supportedfor shared Pub/Sub topics and subscriptions.

  • Listings for multiple regions aren't supported in data clean rooms.

  • Listings for multiple regions aren't supported inBigQuery Omniregions.

• The following limitations apply for the usage metrics:

  • You can't get the usage metrics for listings that were subscribed beforeJuly 20, 2023.

  • External table usage metrics for thenum_rows_processed andtotal_bytes_processed fieldsmight contain inaccurate data.

  • Usage metrics for consumption are supported only for usage usingBigQuery jobs. Consumptionby using the following resources is not supported:

    • BigQuery Storage Read API
    • tabledata.list
    • BigQuery BI Engine queries

  • Usage metrics forviews are only populatedfor queries after April 22, 2024.

  • Usage metrics aren't captured for linked Pub/Sub subscriptionsin BigQuery (you can continue to see usage directly inPub/Sub).

  • SQL stored procedures aren't available in thesharing usage metrics dashboard. You can view detailsin theINFORMATION_SCHEMA.ROUTINES view, but not in theINFORMATION_SCHEMA.SHARED_DATASET_USAGE view. For more information, seeUseINFORMATION_SCHEMA view.

• The following limitations apply when subscribing to Salesforce Data Cloud data:

  • Data Cloud data is shared as views. As a subscriber, youcan't access the underlying tables that the views reference.

Supported regions

BigQuery sharing is supported in the following regions andmulti-regions.

Regions

The following table lists the regions in the Americas where sharing is available.

Region description	Region name	Details
Columbus, Ohio	us-east5
Dallas	us-south1	Low CO2
Iowa	us-central1	Low CO2
Las Vegas	us-west4
Los Angeles	us-west2
Mexico	northamerica-south1
Montréal	northamerica-northeast1	Low CO2
Northern Virginia	us-east4
Oklahoma	us-central2	Low CO2
Oregon	us-west1	Low CO2
Salt Lake City	us-west3
São Paulo	southamerica-east1	Low CO2
Santiago	southamerica-west1
South Carolina	us-east1
Toronto	northamerica-northeast2

The following table lists the regions in Asia Pacific where sharing is available.

Region description	Region name	Details
Delhi	asia-south2
Hong Kong	asia-east2
Jakarta	asia-southeast2
Melbourne	australia-southeast2
Mumbai	asia-south1
Osaka	asia-northeast2
Seoul	asia-northeast3
Singapore	asia-southeast1
Sydney	australia-southeast1
Taiwan	asia-east1
Tokyo	asia-northeast1

The following table lists the regions in Europe where sharing is available.

Region description	Region name	Details
Belgium	europe-west1	Low CO2
Berlin	europe-west10
Finland	europe-north1	Low CO2
Frankfurt	europe-west3
London	europe-west2	Low CO2
Madrid	europe-southwest1	Low CO2
Milan	europe-west8
Netherlands	europe-west4	Low CO2
Paris	europe-west9	Low CO2
Turin	europe-west12
Warsaw	europe-central2
Zürich	europe-west6	Low CO2

The following table lists the regions in the Middle East where sharing is available.

Region description	Region name	Details
Dammam	me-central2
Doha	me-central1
Tel Aviv	me-west1

The following table lists the regions in Africa where sharing is available.

Region description	Region name	Details
Johannesburg	africa-south1

Multi-regions

The following table lists the multi-regions where sharing is available.

Multi-region description	Multi-region name
Data centers withinmember states of the European Union1	EU
Data centers in the United States	US

¹ Data located in theEU multi-region is notstored in theeurope-west2 (London) oreurope-west6 (Zürich) datacenters.

Omni regions

The following table lists the Omni where sharing is available.

	Omni region description	Omni region name
AWS
	AWS - US East (N. Virginia)	aws-us-east-1
	AWS - US West (Oregon)	aws-us-west-2
	AWS - Asia Pacific (Seoul)	aws-ap-northeast-2
	AWS - Asia Pacific (Sydney)	aws-ap-southeast-2
	AWS - Europe (Ireland)	aws-eu-west-1
	AWS - Europe (Frankfurt)	aws-eu-central-1
Azure
	Azure - East US 2	azure-eastus2

Example use case

This section shows an example of how you can use sharing inBigQuery.

Suppose you are a retailer and your organization has real-timedemand forecasting data in a Google Cloud project namedForecasting.You want to share this demand forecasting data with hundreds of vendors inyour supply-chain system. Here's how you can share your data with vendorsthrough BigQuery sharing:

Administrators

As the owner of theForecasting project, you must first enable theAPI and then assign theAnalytics Hub Admin roleto a user who administers the data exchange in the project. Users with theAnalytics Hub Admin role are called theadministrators.

This administrator can perform the following tasks:

• Create, update, delete, and share the data exchange in your organization'sForecasting project.

• Manage otheradministrators with the Analytics Hub Admin role.

• Managepublishers by granting the Analytics Hub Publisher role to yourorganization's employees. If you want some employees toonly be able to update, delete, and share listings but not create them, thenyou can grant them the Analytics Hub Listing Admin role.

• Managesubscribers by granting the Analytics HubSubscriber role to a Google group consisting of all vendors. If you want somevendors to only have view access to the available exchanges and listings thenyou can grant them the Analytics Hub Viewer role. These vendorsaren't able to subscribe to listings.

For more information, seeManage data exchanges.

Publishers

Publishers create the following listings for their datasets in theForecasting project or in a different project:

• Listing A: Demand Forecast Dataset 1
• Listing B: Demand Forecast Dataset 2
• Listing C: Demand Forecast Dataset 3

As a data provider, you cantrack the usage metricsfor your shared dataset. The usage metrics include the following details:

• Jobs that run against your shared dataset.
• The consumption details of your shared dataset by subscribers' projects andorganization.
• The number of rows and bytes processed by the job.

For more information, seeManage listings.

Subscribers

Subscribers can browse through listings that they have access to in dataexchanges. They can also subscribe to these listings and add these datasets totheir projects by creating a linked dataset. Vendors can then run queries onthese linked datasets and retrieve results in real time.

For more information, seeView and subscribe to listings and data exchanges.

Pricing

There is no additional cost for managing data exchanges or listings.

For BigQuery datasets, publishers are charged for data storage,whereas subscribers pay for queries that run against the shared data based oneither on-demand or capacity-based pricing models. For information about pricing,seeBigQuery pricing.

For Pub/Sub, topic publishers are charged for the total number of bytes written (publish throughput) to the shared topic and network egress (if applicable). Subscribers are charged for the total number of bytes read (subscribe throughput) from the linked subscription and network egress (if applicable). SeePub/Sub pricing for additional details.

Quotas

For information about BigQuery sharing quotas, seeQuotas andlimits.

Compliance

Sharing, as part of BigQuery, is compliantwith the following compliance programs:

• ISO 27001
• ISO 27017
• ISO 27018
• SOC 1
• SOC 2
• SOC 3
• PCI
• Penetration Testing
• HIPAA
• HITRUST

VPC Service Controls

You can set the ingress and egress rules needed to let publishers andsubscribers access data from projects that haveVPC Service Controls perimeters. For more information, seeSharing VPC Service Controls rules.

What's next

• Learn how toview and subscribe to listings and data exchanges.
• Learn how to grantAnalytics Hub roles.