Storage Insights datasets tables and schemas
This document describes the tables and their schemas included inStorage Insights datasets, which you can use to analyze operational patterns,monitor resource usage, and understand activity across your storage environment.
Dataset schema of metadata
The following sections describe the metadata fields included in datasets. For more informationabout BigQuery column modes, seeModes. The column modes determinehow BigQuery stores and queries the data.
Bucket metadata
The bucket metadata schema includes thebucket_attributes_view andbucket_attributes_latest_snapshot_view tables. The bucket metadata tables have the followingfields:
| Metadata field | Mode | Type | Description |
|---|---|---|---|
snapshotTime | NULLABLE | TIMESTAMP | ThesnapshotTime field stores the time of the bucket metadatasnapshot refresh inRFC 3339 format. |
name | NULLABLE | STRING | The name of the bucket. |
location | NULLABLE | STRING | The location of the bucket. Object data for objects in the bucket resides in physical storage within this location. |
project | NULLABLE | INTEGER | The project number of the project the bucket belongs to. |
storageClass | NULLABLE | STRING | The bucket's defaultstorage class. |
public | NULLABLE | RECORD | Deprecated. This field indicates whether a bucket is publicly accessible. UseiamConfiguration instead. |
public.bucketPolicyOnly | NULLABLE | BOOLEAN | Deprecated. This field, part of thepublic record, indicates whether uniform bucket-level access is enabled, which prevents access through object-level ACLs. |
public.publicAccessPrevention | NULLABLE | STRING | Deprecated. This field, part of thepublic record, indicates whether public access to the bucket is prevented. |
autoclass | NULLABLE | RECORD | The bucket'sAutoclass configuration, which, when enabled, controls the storage class of objects based on how and when the objects are accessed. |
autoclass.enabled | NULLABLE | BOOLEAN | Whether or not Autoclass is enabled. |
autoclass.toggleTime | NULLABLE | TIMESTAMP | The time at which Autoclass was last enabled or disabled for this bucket, inRFC 3339 format. |
versioning | NULLABLE | BOOLEAN | Whether or not the bucket has versioning enabled. For more information, seeObject Versioning. |
lifecycle | NULLABLE | BOOLEAN | Whether or not the bucket has a lifecycle configuration. Seelifecycle management for more information. |
metageneration | NULLABLE | INTEGER | The metadata generation of this bucket. |
timeCreated | NULLABLE | TIMESTAMP | The creation time of the bucket inRFC 3339 format. |
tags | NULLABLE | RECORD | Deprecated. This field contains user-defined key-value pairs associated with the bucket. UseresourceTags instead. |
tags.lastUpdatedTime | NULLABLE | TIMESTAMP | Deprecated. This field indicates the last time thetags were updated. |
tags.tagMap | REPEATED | RECORD | Deprecated. This field, part of thetags record, contains the map of tag keys and values. |
tags.tagMap.key | NULLABLE | STRING | Deprecated. This field, part of thetags.tagMap record, represents the key of a tag. |
tags.tagMap.value | NULLABLE | STRING | Deprecated. This field, part of thetags.tagMap record, represents the value of a tag. |
labels | REPEATED | RECORD | User-providedbucket labels, in key-value pairs. |
labels.key | NULLABLE | STRING | An individual label entry. |
labels.value | NULLABLE | STRING | The label's value. |
softDeletePolicy | NULLABLE | RECORD | The bucket'ssoft delete policy, which defines the period of time during which objects in the bucket are retained in a soft-deleted state after being deleted. Objects in a soft-deleted state cannot be permanently deleted, and are restorable until theirhardDeleteTime. |
softDeletePolicy.effectiveTime | NULLABLE | DATETIME | The datetime at which the soft delete policy becomes effective, inRFC 3339 format.
|
softDeletePolicy.retentionDurationSeconds | NULLABLE | LONG | The period of time during which a soft-deleted object is retained and cannot be permanently deleted, in seconds. The value must be greater than or equal to604800 (7 days) and less than7776000 (90 days). The value can also be set to0, which disables the soft delete policy. |
iamConfiguration | NULLABLE | RECORD | The IAM configuration for a bucket. |
iamConfiguration.uniformBucketLevelAccess | NULLABLE | RECORD | The bucket'suniform bucket-level access configuration. |
iamConfiguration.uniformBucketLevelAccess.enabled | NULLABLE | BOOLEAN | Whether or not the bucket uses uniform bucket-level access. |
iamConfiguration.publicAccessPrevention | NULLABLE | STRING | The bucket'spublic access prevention status, which is either"inherited" or"enforced". |
resourceTags | REPEATED | RECORD | The bucket'stags. For more information, seeCloud Resource Manager API. |
resourceTags.key | NULLABLE | STRING | TheresourceTags key. |
resourceTags.value | NULLABLE | STRING | TheresourceTags value. |
Object metadata
The object metadata schema includes theobject_attributes_view andobject_attributes_latest_snapshot_view tables. The object metadata tables havethe following fields:
| Metadata field | Mode | Type | Description |
|---|---|---|---|
snapshotTime | NULLABLE | TIMESTAMP | ThesnapshotTime field stores the time of the object metadata snapshot refresh inRFC 3339 format. |
bucket | NULLABLE | STRING | The name of the bucket containing this object. |
location | NULLABLE | STRING | The location of the bucket. Object data for objects in the bucket resides in physical storage within this location. |
componentCount | NULLABLE | INTEGER | Returned forcomposite objects only. Number of non-composite objects in the composite object.componentCount includes non-composite objects that were part of any composite objects used tocompose the current object. |
contentDisposition | NULLABLE | STRING | Content-Disposition of the object data. |
contentEncoding | NULLABLE | STRING | Content-Encoding of the object data. |
contentLanguage | NULLABLE | STRING | Content-Language of the object data. |
contentType | NULLABLE | STRING | Content-Type of the object data. |
crc32c | NULLABLE | INTEGER | CRC32c checksum, as described inRFC 4960, Appendix B; encoded usingbase64 in big-endian byte order. |
customTime | NULLABLE | TIMESTAMP | A user-specified timestamp for the object inRFC 3339 format. |
etag | NULLABLE | STRING | HTTP 1.1Entity tag for the object. |
eventBasedHold | NULLABLE | BOOLEAN | Whether or not the object is subject to anevent-based hold. |
generation | NULLABLE | INTEGER | The content generation of this object. |
md5Hash | NULLABLE | STRING | MD5 hash of the data, encoded usingbase64. This field is not present forcomposite objects. |
mediaLink | NULLABLE | STRING | A URL for downloading the object's data. |
metadata | REPEATED | RECORD | User-provided metadata, in key-value pairs. |
metadata.key | NULLABLE | STRING | An individual metadata entry. |
metadata.value | NULLABLE | STRING | The metadata value. |
metageneration | NULLABLE | INTEGER | The version of the metadata for this object at this generation. |
name | NULLABLE | STRING | The name of the object. |
selfLink | NULLABLE | STRING | A URL for this object. |
size | NULLABLE | INTEGER | Content-Length of the data in bytes. |
storageClass | NULLABLE | STRING | Storage class of the object. |
temporaryHold | NULLABLE | BOOLEAN | Whether or not the object is subject to atemporary hold. |
timeCreated | NULLABLE | TIMESTAMP | The creation time of the object inRFC 3339 format. |
timeDeleted | NULLABLE | TIMESTAMP | The deletion time of the object inRFC 3339 format. |
updated | NULLABLE | TIMESTAMP | Themodification time of the object metadata inRFC 3339 format. |
timeStorageClassUpdated | NULLABLE | TIMESTAMP | The time at which the object's storage class was last changed. |
retentionExpirationTime | NULLABLE | TIMESTAMP | The earliest time that the object can be deleted, which depends on anyretention configuration set for the object and anyretention policy set for the bucket that contains the object. The value forretentionExpirationTime is given inRFC 3339 format. |
softDeleteTime | NULLABLE | DATETIME | The time at which the object wassoft deleted. Only available for objects in buckets with a soft delete policy. |
hardDeleteTime | NULLABLE | DATETIME | The time at which a soft-deleted object is permanently deleted and can no longer be restored. The value is the sum of thesoftDeleteTime value and thesoftDeletePolicy.retentionDurationSeconds value of the bucket. Only available for objects in buckets with a soft delete policy. |
project | NULLABLE | INTEGER | The project number of the project the bucket belongs to. |
Latest bucket and object metadata snapshot
The linked dataset exposes the latest snapshot of the bucket and object metadatathrough the following dedicated views:
The
bucket_attributes_latest_snapshot_viewtable provides the latest metadatafor your Cloud Storage buckets. Its structure matches theBucketmetadata schema.The
object_attributes_latest_snapshot_viewtable provides the latest metadatafor your Cloud Storage objects. Its structure matches theObjectmetadata schema.
Project metadata
The linked dataset exposes project metadata as a view namedproject_attributes_view:
| Metadata field | Mode | Type | Description |
|---|---|---|---|
snapshotTime | NULLABLE | TIMESTAMP | ThesnapshotTime field stores the time of the project metadatasnapshot refresh inRFC 3339 format. |
name | NULLABLE | STRING | The name of the project. |
id | NULLABLE | STRING | The unique identifier for the project. |
number | NULLABLE | INTEGER | A numeric value associated with the project. |
Dataset schema for activity data tables
Storage Insights datasets provide activity data, including operational metrics,error logs, and aggregated views with customizable scopes to support diverseanalysis needs. The following sections describe the activity data schemas andthe data available in them.
Object activity data
Theobject_events_view table in your Storage Insights dataset providesqueryable records of create, update, and delete object operations, metadata, anddetails about the errors encountered during these operations. You can use thedata to track and troubleshoot changes to your storage assets by querying theobject_events_view view. The data in theobject_events_view view is scopedto your Storage Intelligence subscription and lists all the activelyenabled objects within it.
Theobject_events_view table includes the following fields:
| Field | Mode | Type | Description |
|---|---|---|---|
requestCompletionTimestamp | NULLABLE | TIMESTAMP | The timestamp when the request was completed, inRFC 3339 format. |
project | NULLABLE | INTEGER | The project number that owns the bucket resource. |
bucketName | NULLABLE | STRING | The name of the bucket specified in the request. |
objectName | NULLABLE | STRING | The name of the object specified in the request. This field can be null. |
requestOperation | NULLABLE | STRING | The Cloud Storage operation performed. This field is limited to write, update, and delete operations and can be null. |
requestBytes | NULLABLE | INTEGER | The number of bytes sent in the request. |
responseBytes | NULLABLE | INTEGER | The number of bytes sent in the response. |
requestProcessingTimeMicros | NULLABLE | INTEGER | The time in microseconds the server took to process the request. This time is measured from when the first byte is received until the response is sent. Forresumable uploads, the response to the final upload request determines the endpoint. |
requestId | NULLABLE | STRING | The identifier for the request. |
requestHost | NULLABLE | STRING | The host specified in the original request. |
requestHttpMethod | NULLABLE | STRING | The HTTP method used for this request. |
responseStatus | NULLABLE | INTEGER | The HTTP status code sent in the server's response. |
requestReferrer | NULLABLE | STRING | The HTTP referrer for the request. |
size | NULLABLE | INTEGER | The size of the object in bytes. |
generation | NULLABLE | INTEGER | The content generation of the object, used for object versioning. |
timeCreated | NULLABLE | TIMESTAMP | The object's creation time, inRFC 3339 format. |
storageClass | NULLABLE | STRING | The storage class of the object. |
errorReason | NULLABLE | STRING | The reason for the error (limited to 429 errors). |
location | NULLABLE | STRING | The location of the object. |
Project activity view
Theproject_activity_view table offers aggregated operational insights acrossyour projects. The table summarizes total activity by counting operation types,bytes transferred, and response codes and errors. You can use the projectactivity insights for monitoring, tracking, and internal cost attribution. Thedata in the view is determined by your Storage Intelligencesubscription scope. The view displays aggregated data only for the buckets andobjects that are actively enabled under your subscription. For example, if youenable Storage Intelligence only for ten specific buckets, the viewdisplays data exclusively for those ten buckets. Conversely, if you enableStorage Intelligence at a higher level (organization, folder, orproject), all buckets and objects within that scope are automatically included,and the view displays the aggregated data for them.
Theproject_activity_view table includes the following fields:
| Field | Mode | Type | Description |
|---|---|---|---|
snapshotStartTime | NULLABLE | TIMESTAMP | The snapshot's start time, inRFC 3339 format. |
snapshotEndTime | NULLABLE | TIMESTAMP | The snapshot's end time, inRFC 3339 format. |
project | NULLABLE | INTEGER | The project number. |
totalRequests | NULLABLE | INTEGER | The total number of requests made across the project. |
totalReads | NULLABLE | INTEGER | The total number of read operations. |
totalWrites | NULLABLE | INTEGER | The total number of write operations. |
totalUpdates | NULLABLE | INTEGER | The total number of update operations. |
totalDeletes | NULLABLE | INTEGER | The total number of delete operations. |
totalLists | NULLABLE | INTEGER | The total number of list operations. |
totalRequestBytes | NULLABLE | INTEGER | The total number of request bytes transferred. |
totalResponseBytes | NULLABLE | INTEGER | The total number of response bytes transferred. |
count4xx | NULLABLE | INTEGER | The total number of 4xx errors. |
count5xx | NULLABLE | INTEGER | The total number of 5xx errors. |
count2xx | NULLABLE | INTEGER | The total number of 2xx success responses. |
count400 | NULLABLE | INTEGER | The total number of 400 errors for bad requests. |
count401 | NULLABLE | INTEGER | The total number of 401 errors for unauthorized requests. |
count403 | NULLABLE | INTEGER | The total number of 403 errors for forbidden requests. |
count429 | NULLABLE | INTEGER | The total number of 429 errors for usage limit exceeded requests. |
Bucket activity view
Thebucket_activity_view table offers aggregated operational insights acrossyour buckets. The table summarizes total activity by counting operation types,bytes transferred, response codes, errors, and includes details about topprefixes. You can use the bucket activity insights for bucket-level monitoring,troubleshooting, metering, and analyzing your most and least active buckets. Thedata inbucket_activity_view table is scoped to yourStorage Intelligence subscription and includes records for all activelyenabled buckets.
Thebucket_activity_view table includes the following fields:
| Field | Mode | Type | Description |
|---|---|---|---|
snapshotStartTime | NULLABLE | TIMESTAMP | The snapshot's start time, inRFC 3339 format. |
snapshotEndTime | NULLABLE | TIMESTAMP | The snapshot's end time, inRFC 3339 format. |
name | NULLABLE | STRING | The name of the bucket. |
generation | NULLABLE | INTEGER | The bucket's generation number. |
project | NULLABLE | INTEGER | The project number. |
location | NULLABLE | STRING | The location of the bucket. |
totalRequests | NULLABLE | INTEGER | The total number of requests made on the bucket. |
totalReads | NULLABLE | INTEGER | The total number of read operations. |
totalWrites | NULLABLE | INTEGER | The total number of write operations. |
totalUpdates | NULLABLE | INTEGER | The total number of update operations. |
totalDeletes | NULLABLE | INTEGER | The total number of delete operations. |
totalLists | NULLABLE | INTEGER | The total number of list operations. |
totalRequestBytes | NULLABLE | INTEGER | The total number of request bytes transferred. |
totalResponseBytes | NULLABLE | INTEGER | The total number of response bytes transferred. |
count4xx | NULLABLE | INTEGER | The total number of 4xx errors. |
count5xx | NULLABLE | INTEGER | The total number of 5xx errors. |
count2xx | NULLABLE | INTEGER | The total number of 2xx success responses. |
count400 | NULLABLE | INTEGER | The total number of 400 errors for bad requests. |
count401 | NULLABLE | INTEGER | The total number of 401 errors for unauthorized requests. |
count403 | NULLABLE | INTEGER | The total number of 403 errors for forbidden requests. |
count429 | NULLABLE | INTEGER | The total number of 429 errors for usage limit exceeded requests. |
topPrefixes429Errors | REPEATED | RECORD | The top 5 bucket prefixes with the most 429 errors, returned as aRECORD containing the prefix as aSTRING and the error count as anINTEGER. |
topPrefixesRequestBytes | REPEATED | RECORD | The top 5 bucket prefixes with the highest number of requested bytes, returned as aRECORD containing the prefix as aSTRING and the count as anINTEGER. |
topPrefixesResponseBytes | REPEATED | RECORD | The top 5 bucket prefixes with the highest number of response bytes, returned as aRECORD containing the prefix as aSTRING and the count as anINTEGER. |
topPrefixesHighestOperations | REPEATED | RECORD | The top 5 bucket prefixes with the highest number of operations, returned as aRECORD containing the prefix as aSTRING and the count as anINTEGER. |
Bucket region activity view
Thebucket_region_activity_view table summarizes the total bytes sent andreceived for a bucket, aggregated by destination region. Use the bucket activityregional data to identify the most active regions, helping you determine theoptimal bucket location, analyze regional egress, and considerbucket relocation as needed tooptimize performance, cost, or resilience. Thedata inbucket_region_activity_view table is scoped to yourStorage Intelligence subscription and includes records for all activelyenabled buckets.
| Field | Mode | Type | Description |
|---|---|---|---|
snapshotStartTime | NULLABLE | TIMESTAMP | The snapshot's start time, inRFC 3339 format. |
snapshotEndTime | NULLABLE | TIMESTAMP | The snapshot's end time, inRFC 3339 format. |
project | NULLABLE | INTEGER | The project number for the bucket. |
name | NULLABLE | STRING | The name of the bucket. |
generation | NULLABLE | INTEGER | The bucket's generation number. |
requestLocation | NULLABLE | STRING | The Google Cloud location where the request originated. |
bucketLocation | NULLABLE | STRING | The location of the bucket. |
requestBytes | NULLABLE | INTEGER | The total number of bytes transferred for requests between the specified bucket and location. |
responseBytes | NULLABLE | INTEGER | The total number of bytes transferred for responses between the specified bucket and location. |
Dataset schema for events and errors
In the linked dataset, you can also view the snapshot processing events anderrors in theevents_view anderror_attributes_view views. Totroubleshoot the snapshot processing errors, seeTroubleshoot dataseterrors.
Events log
You can view event logs in theevents_view view in the linked dataset:
| Metadata field | Mode | Type | Description |
|---|---|---|---|
manifest.snapshotTime | NULLABLE | TIMESTAMP | The time, inRFC 3339 format, when the snapshot of the events is refreshed. |
manifest.viewName | NULLABLE | STRING | The name of the view that refreshes. |
manifest.location | NULLABLE | STRING | The source location of the data that refreshes. |
globalManifest.snapshotTime | NULLABLE | TIMESTAMP | The ingestion completion time, inRFC 3339 format, for the bucket and object attributes tables across all source locations. |
eventTime | NULLABLE | TIMESTAMP | The time the event occurred. |
eventCode | NULLABLE | STRING | The event code associated with the corresponding entry. TheeventCode values are as follows:
|
Error codes
You can view error codes in theerror_attributes_view table in the linkeddataset:
| Metadata field | Mode | Type | Description |
|---|---|---|---|
errorCode | NULLABLE | INTEGER | The error code associated with this entry. For a list of valid values and how to resolve these errors, seeTroubleshoot dataset errors. |
errorSource | NULLABLE | STRING | The source of the error. Valid value:CONFIGURATION_PREPROCESSING. |
errorTime | NULLABLE | TIMESTAMP | The time the error occurs. |
sourceGcsLocation | NULLABLE | STRING | The source Cloud Storage location of the error. For projects, this field is null because projects are not associated with a location. |
bucketErrorRecord.bucketName | NULLABLE | STRING | The name of the bucket involved in the error. You can use this information to debug a bucket error. |
bucketErrorRecord.serviceAccount | NULLABLE | STRING | The service account that needs permission to ingest objects from the bucket. You can use this information to debug a bucket error. |
projectErrorRecord.projectNumber | NULLABLE | INTEGER | The number of the project involved in the error. You can use this information to debug a project error. |
projectErrorRecord.organizationName | NULLABLE | STRING | The name of the organization the project must belong to for processing. A value of0 indicates that the project is not in the organization. You can use this information to debug a project error. |
Troubleshoot dataset errors
To troubleshoot the snapshot processing errors that are logged in theerror_attributes_view view in the linked dataset, see the following table:
| Error code | Error case | Error message | Troubleshooting |
|---|---|---|---|
| 1 | Source project doesn't belong to the organization | Source projectprojectErrorRecord.projectNumber doesn't belong to the organizationprojectErrorRecord.organizationName. | Add source projectprojectErrorRecord.projectNumber to organizationprojectErrorRecord.organizationName. For instructions to migrate a project between organizations, seeMigrate projects between organizations. |
| 2 | Bucket authorization error | Permission denied for ingesting objects for bucketbucketErrorRecord.bucketName. | Give service accountbucketErrorRecord.serviceAccount Identity and Access Management (IAM) permissions to allow ingestion of objects for bucketbucketErrorRecord.bucketName. For more information, seeGrant required permissions to the service agent. |
| 3 | Destination project doesn't belong to the organization | Destination projectprojectErrorRecord.projectNumber not in organizationprojectErrorRecord.organizationName. | Add destination projectprojectErrorRecord.projectNumber to organizationprojectErrorRecord.organizationName. For instructions to migrate a project between organizations, seeMigrate projects between organizations. |
| 4 | Source project doesn't have Storage Intelligence configured. | Source projectprojectErrorRecord.projectNumber doesn't have Storage Intelligence configured. | Configure Storage Intelligence for the source projectprojectErrorRecord.projectNumber. For more information, seeConfigure and manage Storage Intelligence. |
| 5 | Bucket doesn't have Storage Intelligence configured. | BucketbucketErrorRecord.bucketName doesn't have Storage Intelligence configured. | Configure Storage Intelligence for the bucketbucketErrorRecord.bucketName. For more information, seeConfigure and manage Storage Intelligence. |
| 6 | Bucket access authentication error | ACTIVITY_BUCKET_ACCESS_AUTHORIZATION_ERROR for bucketbucketErrorRecord.bucketName. | Grant service accountbucketErrorRecord.serviceAccount Identity and Access Management (IAM) permissions to allow ingestion of activity for bucketbucketErrorRecord.bucketName. For more information, seeGrant required permissions to the service agent. |
What's next
- Configure Storage Insights datasets.
- Manage Storage Insights datasets.
- Run SQL queries on the datasets in BigQuery.
- Learn about BigQuery analytics.
Except as otherwise noted, the content of this page is licensed under theCreative Commons Attribution 4.0 License, and code samples are licensed under theApache 2.0 License. For details, see theGoogle Developers Site Policies. Java is a registered trademark of Oracle and/or its affiliates.
Last updated 2025-12-17 UTC.