Introduction to BigQuery sharing

BigQuery sharing (formerly Analytics Hub) is a data exchange platform thatlets you share data and insights at scale across organizational boundaries witha robust security and privacy framework. BigQuery sharing lets youdiscover and access a data library curated by various data providers. This datalibrary also includes Google-provided datasets.

For example, you can use sharing to augment youranalytics and ML initiatives with third-party and Google datasets.

Analytics Hub Identity and Access Management (IAM) roles let you perform thefollowing sharing tasks:

• As an Analytics Hub Publisher, you can share data with your partner networkor within your own organization in real time.Listings let youshare data without replicating the shared data, and they can be monetized ontheGoogle Cloud Marketplaceor through your own channels. You can build a catalog of analytics-ready datasources with granular permissions that let you deliver data to the rightaudiences. You can also manage subscriptions and view the usage metrics foryour listings.

• As an Analytics Hub Subscriber, you can discover the data that you are lookingfor, combine shared data with your existing data, and use thebuilt-in features of BigQuery.When you subscribe to a listing, alinked dataset orlinked Pub/Sub subscription is created in your Google Cloud project.You can manage your subscriptions by using theSubscription resource,which stores relevant information about the subscriber and represents theconnection between publisher and subscriber.

• As an Analytics Hub Viewer, you can browse through the shared resources thatyou have access to in BigQuery sharing and make a request to thepublisher to access the shared data. You can discoverCloud Marketplace-integrated commercial listings on bothBigQuery sharing and Cloud Marketplace.

• As an Analytics Hub Admin, you can createdata exchangesthat enable data sharing, and then give permissions to data publishers andsubscribers to access these data exchanges.

For more information, seeConfigure Analytics Hub roles.

Architecture

BigQuery sharing is built on a publish and subscribe model ofGoogle Cloud data resources, allowing for zero-copy sharing in place.BigQuery sharing supports the following Google Cloud resources:

• BigQuery datasets
• Pub/Sub topics

Publisher workflow

The following diagram describes how a publisher shares assets:

The following sections describe the features in this workflow.

Shared resources

Shared resources are the unit of sharing by a publisher inBigQuery sharing.

Shared datasets

A shared dataset is a BigQuery dataset that is the unit ofdata sharing in BigQuery sharing. The separation of compute and storagein the BigQuery architecture lets data publishers share datasetswith as many subscribers as they want, without having to make multiple copies ofthe data. As a publisher, you create or use an existing BigQuerydataset in your project with the following supported objects that you want todeliver to your subscribers:

• Authorized views
• Authorized datasets
• BigQuery ML models
• External tables
• Materialized views
• Routines
  • User-defined functions (UDFs)
  • Table functions
  • SQL stored procedures
• Tables
• Table snapshots
• Views

Shared datasets supportcolumn-level securityandrow-level security.

Consider the following limitations for VPC Service Controls and sharing:

• Don't publish shared data in projects inside VPC Service Controls perimeters. Ifshared datasets in a project are within a VPC Service Controls perimeter, you needappropriateingress and egress rulesfor both the exchange project (hosted listings) and all subscriber projects tosuccessfully subscribe to the publisher's listings.

• Don't put your exchange project in a VPC Service Controls perimeter, as it mightinterrupt publishing workflows, requiringingress and egress rulesfor the publisher project and all subscriber projects to successfullysubscribe to their listings.

Shared topics

A shared topic is aPub/Sub topic,which is the unit ofstreaming data sharing in BigQuery.As a publisher, you create or use an existing Pub/Sub topic inyour project and distribute it to your subscribers.

Data exchanges

A data exchange is a container that lets you share data through self-service. Itcontains listings that reference shared resources. Publishers and administratorscan grant access to subscribers at the exchange and listing level. This helps youavoid explicitly granting access on the underlying shared resources. You canbrowse through data exchanges, discover data that you can access, and subscribe toshared resources. When youcreate a data exchange,you can assign a primary contact email address. This email address lets userscontact the data exchange owner with questions or concerns.

A data exchange can be one of the following types:

• Private data exchange. By default, a data exchange is private and onlyusers or groups that have access to that exchange can view or subscribe to itslistings.
• Public data exchange. By default, a data exchange is private and onlyusers or groups that have access to that exchange can view or subscribe to itslistings. However, you can choose to make a data exchange public. Listings inpublic data exchanges can bediscoveredandsubscribed tobyGoogle Cloud users (allAuthenticatedUsers).For more information about public data exchanges, seeMake a data exchange public.

The Analytics Hub Admin role lets you create multiple data exchanges and manageother users performing sharing tasks.

Listings

A listing is a reference to a shared resource that a publisher lists in a dataexchange. As a publisher, you can create a listing and specify the resourcedescription, sample queries to run or sample message data, links to anyrelevant documentation, and any additional information that helps subscribersuse your shared resource. When you create a listing, you can assign a primarycontact email address, a provider name and contact, and a publisher name andcontact.

The primary contact email address lets users contact the listing owner withquestions or concerns about the data exchange. The provider name and contact isthe agency that originally provided the data for the listing. This informationis optional. The publisher name and contact is the agency that publishes thedata for use in BigQuery sharing. This information is optional. For moreinformation, seeManage listings.

A listing can be one of two types, based on the IAM policy set for the listing and the type of data exchange that contains the listing:

• Public listing. A public listing is shared with allGoogle Cloud users (allAuthenticatedUsers).Listings in a public data exchange are public listings. These listings can bereferences of afree public resource or acommercial resource. If thelisting is of a commercial resource, subscribers can either request access to thelisting directly from the data provider, or they can browse and purchaseGoogle Cloud Marketplace-integrated commercial listings.
• Private listing. A private listing is shared directly with individuals orgroups. For example, a private listing can reference a marketing metrics datasetthat you share with other internal teams within your organization.

Subscriber workflow

The following diagram describes how Analytics Hub subscribers interact withshared resources:

The following sections describe the features in the subscriber workflow.

Linked resources

Linked resources are created when subscribing to a BigQuery sharinglisting, connecting a subscriber to the underlying shared resource.

Linked datasets

A linked dataset is aread-only BigQuery dataset thatserves as a pointer or reference to a shared dataset. Subscribing to a listingcreates a linked dataset in your project and not a copy of the dataset, sosubscribers can read the data but cannot add or update objects within it. When youquery objects such as tables and views through a linked dataset, the data from theshared dataset is returned. For more information about linked datasets, seeView and subscribe to listings and data exchanges.

Linked datasets are authorized to access tables and views of a shared dataset.Subscribers with linked datasets access tables and views of a shared datasetwithout any additional Identity and Access Management authorization.

Linked datasets support the following objects:

• Authorized views
• Authorized datasets
• Authorized routines

Linked Pub/Sub subscriptions

Subscribing to a listing with a shared topic creates a linked Pub/Subsubscription in the subscriber project. No copies of the shared topic or messagedata are created. Subscribers of the linkedPub/Sub subscriptioncan access the messages published to the shared topic. Subscribers accessthe message data of a shared topic without any additional IAMauthorization. Publishers can manage subscriptions both in Pub/Subdirectly or through BigQuery sharing subscription management. For moreinformation about linked Pub/Sub subscriptions, seeStream sharing with Pub/Sub.

Data egress options (BigQuery shared datasets only)

Data egress options let publishers restrict subscribers from exporting data out ofBigQuery linked datasets.

Publishers can enable data egress restriction on a listing, the results of aquery, or both. When data egress is restricted, the following restrictions apply:

• Copy, clone, export, and snapshot APIs are unavailable.

• Copy, clone, export, and snapshot options are unavailable in the Google Cloud console.

• Connecting the restricted dataset to the table explorer is unavailable.

• BigQuery Data Transfer Service is unavailable on the restricted dataset.

• CREATE TABLE AS SELECT statementsandwriting to a destination tableare unavailable.

• CREATE VIEW AS SELECT statementsand writing to a destination view are unavailable.

When youcreate a listing,you can set the appropriate data egress options.

Limitations

BigQuery sharing has the following limitations:

• A shared dataset can have a maximum of 1,000 linked datasets.

• A shared topic can have amaximum of 10,000Pub/Sub subscriptions. This limit includes linkedPub/Sub subscriptions and Pub/Sub subscriptionscreated outside of BigQuery sharing (for example, directly fromPub/Sub).

• A dataset with unsupported resources cannot be selected as a shared datasetwhen youcreate a listing.For more information about the BigQuery objects thatBigQuery sharing supports, seeShared datasets.

• You can't setIAM rolesorIAM policieson individual tables within a linked dataset. Apply them at the linked datasetlevel instead.

• You can't attachIAM tagson tables within a linked dataset. Apply them at the linked dataset levelinstead.

• Linked datasets created before July 25, 2023, aren't backfilled by thesubscription resource.Only subscriptions created after July 25, 2023 work with the API methods.

• If you are a publisher, the following BigQuery interoperabilitylimitations apply:

  • You must grant subscribers explicit permissions to read the sourcedataset to query views within linked datasets. To grant access to views,as a best practice, createauthorized views.Authorized views can grant subscribers access to the view data withoutgiving them access to the underlying source data.

  • Thequery planreveals the shared view query and the routine query, including project IDs,and other datasets involved in authorized views. Never include anythingsuch as encryption keys that you consider sensitive in the shared view orroutine query.

  • Shared datasets are indexed inData Catalog(deprecated) andDataplex Universal Catalog.Updates on a shared dataset, such as adding tables or views, becomeavailable to subscribers without delay. However, in certain scenarios, forexample, when there are more than 100 subscribers or tables in a shareddataset, the updates might take up to 18 hours to get indexed inthese services. Due to the indexing delay, subscribers cannot search forthese updated resources in the Google Cloud console immediately.

  • Shared topics are indexed in Data Catalog (deprecated) andDataplex Universal Catalog, but you cannot filter specifically for itsresource type.

  • If you have set uprow-level securityordata maskingpolicies on the tables that are listed, then subscribers must be anEnterprise or Enterprise Plus customer to run the queryjob on the linked dataset. For information about editions, seeIntroduction to BigQuery editions.

• If you are a subscriber, the following BigQueryinteroperability limitations apply:

  • Materialized views that refer to tables in the linked dataset aren'tsupported.

  • Takingsnapshotsof linked dataset tables isn't supported.

  • Queries with linked datasets andJOIN statements that are larger than1 TB (physical storage) might fail. You cancontact supportto resolve this issue.

  • You cannot useregion qualifierswithINFORMATION_SCHEMA views toview metadata for your linked dataset.

  • The following limitations apply to listings for multiple regions:

  • Listings for multiple regions are supported only for shared datasetsand linked dataset replicas. Listings for multiple regions aren't supportedfor shared Pub/Sub topics and subscriptions.

  • Listings for multiple regions aren't supported in data clean rooms.

  • Listings for multiple regions aren't supported inBigQuery Omni regions.

• The following limitations apply for the usage metrics:

  • You can't get the usage metrics for listings that were subscribed beforeJuly 20, 2023.

  • External tableusage metrics for thenum_rows_processed andtotal_bytes_processed fieldsmight contain inaccurate data.

  • Usage metrics for consumption are supported only for usage withBigQuery jobs.The following resources don't support consumption:

    • BigQuery Storage Read API
    • tabledata.list
    • BigQuery BI Engine queries

  • Usage metrics forviews are populated only for queries after April 22, 2024.

  • Usage metrics aren't captured for linked Pub/Sub subscriptionsin BigQuery. You can continue to see usage directly inPub/Sub.

  • SQL stored procedures aren't available in theBigQuery sharing usage metrics dashboard. You can view detailsin theINFORMATION_SCHEMA.ROUTINES view, but not in theINFORMATION_SCHEMA.SHARED_DATASET_USAGE view. For more information, seeUseINFORMATION_SCHEMA view.

• The following limitations apply when subscribing to Salesforce Data Cloud data:

  • Data Cloud data is shared as views. As a subscriber, youcan't access the underlying tables that the views reference.

Supported regions

BigQuery sharing is supported in the following regions andmulti-regions.

Regions

The following table lists the regions in the Americas where sharing is available.

Region description	Region name	Details
Columbus, Ohio	us-east5
Dallas	us-south1	Low CO2
Iowa	us-central1	Low CO2
Las Vegas	us-west4
Los Angeles	us-west2
Mexico	northamerica-south1
Montréal	northamerica-northeast1	Low CO2
Northern Virginia	us-east4
Oklahoma	us-central2	Low CO2
Oregon	us-west1	Low CO2
Salt Lake City	us-west3
São Paulo	southamerica-east1	Low CO2
Santiago	southamerica-west1
South Carolina	us-east1
Toronto	northamerica-northeast2

The following table lists the regions in Asia Pacific where sharing is available.

Region description	Region name	Details
Delhi	asia-south2
Hong Kong	asia-east2
Jakarta	asia-southeast2
Melbourne	australia-southeast2
Mumbai	asia-south1
Osaka	asia-northeast2
Seoul	asia-northeast3
Singapore	asia-southeast1
Sydney	australia-southeast1
Taiwan	asia-east1
Tokyo	asia-northeast1

The following table lists the regions in Europe where sharing is available.

Region description	Region name	Details
Belgium	europe-west1	Low CO2
Berlin	europe-west10
Finland	europe-north1	Low CO2
Frankfurt	europe-west3
London	europe-west2	Low CO2
Madrid	europe-southwest1	Low CO2
Milan	europe-west8
Netherlands	europe-west4	Low CO2
Paris	europe-west9	Low CO2
Turin	europe-west12
Warsaw	europe-central2
Zürich	europe-west6	Low CO2

The following table lists the regions in the Middle East where sharing is available.

Region description	Region name	Details
Dammam	me-central2
Doha	me-central1
Tel Aviv	me-west1

The following table lists the regions in Africa where sharing is available.

Region description	Region name	Details
Johannesburg	africa-south1

Multi-regions

The following table lists the multi-regions where sharing is available.

Multi-region description	Multi-region name
Data centers withinmember states of the European Union1	EU
Data centers in the United States	US

¹ Data located in theEU multi-region is notstored in theeurope-west2 (London) oreurope-west6 (Zürich) datacenters.

Omni regions

The following table lists the Omni where sharing is available.

	Omni region description	Omni region name
AWS
	AWS - US East (N. Virginia)	aws-us-east-1
	AWS - US West (Oregon)	aws-us-west-2
	AWS - Asia Pacific (Seoul)	aws-ap-northeast-2
	AWS - Asia Pacific (Sydney)	aws-ap-southeast-2
	AWS - Europe (Ireland)	aws-eu-west-1
	AWS - Europe (Frankfurt)	aws-eu-central-1
Azure
	Azure - East US 2	azure-eastus2

Example use case

This section provides an example of how to use sharing inBigQuery.

Suppose you are a retailer and your organization has real-time demand forecastingdata in a Google Cloud project named Forecasting. You want to sharethis demand forecasting data with hundreds of vendors in your supply-chainsystem. The following sections describe how you can share your data with vendorsthrough BigQuery sharing.

Administrators

As the owner of the Forecasting project, you must first enable the API and thenassign theAnalytics Hub Admin role(roles/analyticshub.admin) to a user who administers the data exchange in theproject. Users with the Analytics Hub Admin role are referred to asBigQuery sharing administrators.

A BigQuery sharing administrator can perform the following tasks:

• Create, update, delete, and share the data exchange in your organization'sForecasting project.

• Manage otherBigQuery sharing administrators with theAnalytics Hub Admin role.

• ManageBigQuery sharing publishers by granting theAnalytics Hub Publisher role(roles/analyticshub.publisher) to your organization's employees. If you wantemployees to only update, delete, and share listings, but not create them,grant them theAnalytics Hub Listing Admin role(roles/analyticshub.listingAdmin).

• ManageBigQuery sharing subscribers by granting theAnalytics Hub Subscriber role(roles/analyticshub.subscriber) to a Google group consisting of all vendors.If you want vendors to only view available exchanges and listings, grant themtheAnalytics Hub Viewer role(roles/analyticshub.viewer). These vendors aren't able to subscribe tolistings.

For more information, seeBigQuery sharing IAM rolesandManage data exchanges.

Publishers

Publishers create the following listings for their datasets in theForecasting project or in a different project:

• Listing A: Demand Forecast Dataset 1
• Listing B: Demand Forecast Dataset 2
• Listing C: Demand Forecast Dataset 3

As a data provider, you cantrack the usage metricsfor your shared dataset. The usage metrics include the following details:

• Jobs that run against your shared dataset.
• Consumption details of your shared dataset by subscriber projects and organizations.
• The number of rows and bytes the job processes.

For more information, seeManage listings.

Subscribers

Subscribers can browse through listings that they have access to in dataexchanges. They can also subscribe to these listings and add these datasets totheir projects by creating a linked dataset. Vendors can then run queries onthese linked datasets and retrieve results in real time.

For more information, seeView and subscribe to listings and data exchanges.

Pricing

There is no additional cost for managing data exchanges or listings.

For BigQuery datasets, publishers pay for data storage, whereassubscribers pay for queries that run against the shared data based oneither on-demand or capacity-based pricing models. For information aboutpricing, seeBigQuery pricing.

For Pub/Sub, topic publishers pay for the total number of byteswritten (publish throughput) to the shared topic and network egress (ifapplicable). Subscribers pay for the total number of bytes read (subscribethroughput) from the linked subscription and network egress (if applicable).For more information, seePub/Sub pricing.

Quotas

For information about BigQuery sharing quotas, seeQuotas and limits.

Compliance

BigQuery sharing, as part of BigQuery, is compliantwith the following compliance programs:

• ISO 27001
• ISO 27017
• ISO 27018
• SOC 1
• SOC 2
• SOC 3
• PCI
• Penetration Testing
• HIPAA
• HITRUST

VPC Service Controls

You can set the ingress and egress rules needed to let publishers andsubscribers access data from projects that have VPC Service Controlsperimeters. For more information, seeSharing VPC Service Controls rules.

What's next

• Learn how toview and subscribe to listings and data exchanges.
• Learn how to grantAnalytics Hub roles.