Cloud Billing export toBigQuery lets you export detailed Google Cloud billing data (such as usage,cost estimates, and pricing data) automatically throughout the day to aBigQuery dataset that you specify. Then you can access yourCloud Billing data from BigQuery for detailed analysis, oruse a tool likeLooker Studio to visualize your data. You can also use this export method to export data toa JSON file.

Timing is important. To access a more comprehensive set ofGoogle Cloud billing data for your analysis needs, we recommend thatyou enable Cloud Billing data export to BigQuery at the sametime that youcreate a Cloud Billing account.

See thelimitations that might impact exporting your billing data to BigQuery.

Next steps

Managing and reporting costs effectively is a critical part of financialstewardship, whether you're running a multi-billion-dollar enterprisebusiness or small household budget. Making data-driven decisions about yourGoogle Cloud costs and usage starts with collecting the data you'll needto inform those decisions.

Refer to the guides in this section to learn about the following tasks:

• Set up Cloud Billing data export to BigQuery
• Understand the Cloud Billing data tables
• Find example queries for Cloud Billing data export

Set up Cloud Billing data export to BigQuery

To start collecting your Cloud Billing data, you mustenable Cloud Billing data export to BigQuery.

Thesetup guide provides best practice recommendations and detailed instructionsfor enabling Cloud Billing data export to BigQuery.These are the following types of Cloud Billing data you can enable forexport:

• Standard usage cost data -Contains standard Cloud Billing account cost usage information,such as account ID, invoice date, services, SKUs, projects, labels,locations, cost, usage, credits, adjustments, and currency.

  Use the Standard usage export to analyze broad trends in your cost data.

• Detailed usage cost data -Contains detailed Cloud Billing account cost usage information.Includes everything in thestandard usage cost data plus resource-levelcost data, like a virtual machine or SSD that generates service usage.

  Use the Detailed usage export to analyze costs at the resource level, andidentify specific resources that might be driving up costs. The detailedexport includes resource-level information for the following products:

  • Compute Engine
  • Google Kubernetes Engine (GKE)
  • Cloud Run functions
  • Cloud Run

  To view information about GKE,enable cost allocation in detailed exports.

  Review theschema of the detailed usage cost data for further recommendations and limitations.

• (Resellers only)Rebilling data export -Contains detailed Cloud Billing account cost usage informationacross all your Reseller Billing Accounts annotated with Partner specificattributes.

  Use the Rebilling data export to manage billing operations for yourGoogle Cloud customers. Learn more aboutRepricing configurationswhich let you generate end-customer costs.

• Pricing data -Contains Cloud Billing account pricing information, such asaccount ID, services, SKUs, products, geographic metadata, pricing units,currency, aggregation, and tiers.

  You can also get your Cloud Billing account pricing data in theseways:

  • Use the Pricing page in the Google Cloud console
  • Use the Google Cloud Pricing API

• Committed use discounts (CUD) metadata(Preview) -For customers who purchase CUDs, this daily export provides CUD metadata toa BigQuery table, which you can join with other billingdata exports for better CUD management and reporting. CUD metadata includesinformation such as billing account ID, product ID,consumption model ID,commitment amount units and values, and more.

Using BigQuery to store and query Cloud Billingdata will incur minimal fees. For more information, seeCost of use.

See thelimitations that might impact exporting your billing data to BigQuery.

Understand the Cloud Billing data tables

After youenable Cloud Billing export to BigQuery,Cloud Billingdata tables are automatically created in theBigQuery dataset.

To understand the data schema of your exported content, see the referenceinformation for the contents of the Cloud Billing data that'sexported to each table in the BigQuery dataset.

• Standard usage cost table
• Detailed usage cost table
• Pricing table
• CUD metadata export

Find example queries for Cloud Billing data

For tips and guidance for using SQL to run queries on yourCloud Billing data, view theexample queries.

On the example queries page, you'll find various SQL examples, including thefollowing:

• Return the total costs on an invoice
• Query your data using labels
• Return data about your committed use discounts
• Query costs and credits by project for a specified invoice month
• Join pricing data with detailed usage cost data

Cost of use

Using BigQuery to store and analyze billing usage and cost datatypically incurs minimal fees.

• Loading data into the designated dataset is free; this action takes advantageof BigQuery's pool of shared resources to load data in batches.
• When exporting and analyzing Cloud Billing data with BigQuery,the associated cost depends on the amount of data you stream, store, andquery.
• Many partitioned table operations are free, including loading data intopartitions, copying partitions, and exporting data from partitions. Thoughfree, these operations are subject to BigQuery's quotas andlimits.

Generally, querying the Detailed cost export might cost more than querying theStandard export. To optimize your costs, we recommend using the Standard exportto analyze trends in your costs, and using the Detailed export to track costs atthe resource level, and identify specific resources that might be driving yourcosts.

To get an idea of what charges to expect, seeEstimating storage and query costs.

For more information on best practices for optimizing costs inBigQuery, seeControl costs in BigQuery.

For detailed prices, reviewBigQuery pricing.

Limitations

Exporting Cloud Billing data to BigQuery is subject to thefollowing limitations.

• The table schema for any of the Cloud Billing data exports can change

  When the table schema changes, such as when new fields are added to aBigQuery table schema for a Cloud Billing dataexport, any queries thatdirectly reference the exported columns mightfail. To resolve this, we recommend creatingBigQuery views that query the exported tables and present the information in yourpreferred structure.

  You can then adjust the queries that feed your reports and dashboards topull from the views, instead of the exported tables. By using views, youcan standardize the structure of the data used in your queries anddashboards.

  The views you create should normalize the data so that all of the relevanttables present the same schema to your queries. This protects you fromfuture schema changes, allowing you to modify the view's underlying queryin those instances when the data schema changes.

• BigQuery dataset locations supported for use with Cloud Billing data

  BigQuery datasets are configured to use alocation – either amulti-region location (EU or US), or aregion location. Thedataset location is set at creation time. After a dataset is created, itslocation can't be changed.

  Cloud Billing data export supports allmulti-region locations(EU or US), but only a subset ofregion locations. When you'reconfiguring your Cloud Billing export settings, if you create or selecta dataset that's configured to use an unsupportedregion location,when you attempt to save your export settings, you'll see anInvalid dataset region error.

  The following table lists themulti-region locations and theregion locations that are supported for use with BigQuery datasets thatcontain Cloud Billing data.

  Americas	Asia Pacific	Europe
  Multi-region: US Regions: northamerica-northeast1 (Montréal) southamerica-east1 (São Paulo) us-central1 (Iowa) us-east1 (South Carolina) us-east4 (Northern Virginia) us-west1 (Oregon) us-west2 (Los Angeles) us-west3 (Salt Lake City) us-west4 (Las Vegas)	Regions: asia-east1 (Taiwan) asia-east2 (Hong Kong) asia-northeast1 (Tokyo) asia-northeast2 (Osaka) asia-northeast3 (Seoul) asia-south1 (Mumbai) asia-southeast1 (Singapore) asia-southeast2 (Jakarta) australia-southeast1 (Sydney)	Multi-region: EU Regions: europe-central2 (Warsaw) europe-north1 (Finland) europe-west1 (Belgium) europe-west2 (London) europe-west3 (Frankfurt) europe-west4 (Netherlands) europe-west6 (Zurich)

• If you edit your export settings to update the project or dataset whereyour exported billing data is stored, previously exported billing data isn'tbackfilled to your new dataset. To include the billing information that wasexported prior to the switch, you must manually join the new dataset withthe previous dataset. For more information, seeJoin operation.

• For yourBigQuery datasets containingstandard usage cost data ordetailed usage cost data, thetype of location you configure on the dataset impacts thetimingof when your Google Cloud billing data is exported to the dataset:

  • If you configure the dataset to use amulti-region location (EU or US), the dataset includes Google Cloud billing data incurredfrom the start of the previous month from when youfirst enabled theexport. That is,Google Cloud billing data is added retroactivelyfor the current and previous month. For theinitial backfill ofexported data, it might take up to five days for your retroactiveCloud Billing data to finish exporting before you start seeing yourcurrent usage data. This retroactive backfill of data only occurs thefirst time you enable export. If youedit the export settings to changethe project or dataset where your exported billing data is stored, or youarere-enabling the export, then billing data isnot addedretroactively.

  • If your dataset is configured to use a supportedregion location,yourstandard usage cost data and yourdetailed usage cost data onlyreflect Google Cloud billing data incurred starting from the dateyou enabled Cloud Billing export, and after. That is,Google Cloud billing data isnot added retroactively fornon-multi-region dataset locations, so you won't see Cloud Billingdata from before you enable export.

    Note: Cloud Billing data export supports allmulti-region locations (EU or US), and asubset ofregion locations.When you're configuring your Cloud Billing export settings, if youcreate or select a dataset that's configured to use an unsupportedregion location, when you attempt to save your export settings, you'llsee anInvalid dataset region error.

  • For more details, seeData availability.

• YourBigQuery datasets containingpricing data only collect Google Cloud billing data incurred fromthe date you set up Cloud Billing export, and after. That is,_Google Cloud pricing data isn't added retroactively, so you won'tsee Cloud Billing pricing data from before you enable export. For more details, seeData availability.

• When exportingdetailed usage cost data, the detailed exportautomatically includes resource-level information about Compute Engine.To view a breakdown of Google Kubernetes Engine (GKE) cluster costs in adetailed data export, you must alsoenable cost allocation for GKE.

• Dataset encryption:Customer-managed encryption keys (CMEK)aren't supported when exporting billing data toBigQuery. If you enable CMEK encryption for your billingdata dataset, this type of encryption prevents Cloud Billing fromwriting billing data to the appropriate tables within that dataset. Instead,you need to enable thedataset to use a Google-owned and Google-managed encryption key.

• If you want to useBigQuery row-level security on the table that contains your exported data, you must give theCloud Billing exportservice accountbilling-export-bigquery@system.gserviceaccount.comfull access to the tableusing the BigQueryTRUE filter.The following command grants access to the Cloud Billing service account:

  CREATE ROW ACCESS POLICY cloud_billing_export_policyON `__project_id__.__dataset_id__.__table_id__`GRANT TO ('serviceAccount:billing-export-bigquery@system.gserviceaccount.com')FILTER USING (TRUE);

• Resource-level Tags might take up to an hour to propagate toBigQuery exports. If a tag was added or removedwithin an hour, or if a resource has existed for less than an hour, it mightnot appear in the export.

  Resource-level tags are available for the following resources:

  • Compute Engine instances
  • Spanner instances
  • Cloud Run services
  • Artifact Registry repositories