Mask column data
This document shows you how to implement data masking in order toselectively obscure sensitive data. By implementing data masking, you canprovide different levels of visibility to different groups of users.For general information, seeIntroduction to data masking.
You implement data masking by adding a data policy to a column. To add a datamasking policy to a column, you must complete the following steps :
- Create a taxonomy with at least one policy tag.
- Optional: Grant the Data Catalog Fine-Grained Reader roleto one or more principals on one or more of the policy tags you created.
- Create up to three data policies for the policy tag, to map masking rulesand principals (which represent users or groups) to that tag.
- Set the policy tag on a column. That maps the data policies associated withthe policy tag to the selected column.
- Assign users who should have access to masked data to theBigQuery Masked Reader role. As a best practice, assign theBigQuery Masked Reader role at the data policy level.Assigning the role at the project level or higher grants users permissions toall data policies under the project, which can lead to issues caused byexcess permissions.
You can use the Google Cloud console or the BigQuery Data Policy API to work withdata policies.
When you have completed these steps, users running queries against the columnget unmasked data, masked data, or an access denied error, depending on thegroups that they belong to and the roles that they have been granted. For moreinformation, seeHow Masked Reader and Fine-Grained Reader roles interact.
Alternatively, you can apply data policies directly on a column(Preview). For more information, seeMaskdata with data policies directly on a column.
Mask data with policy tags
Use policy tags to selectively obscure sensitive data.
Before you begin
- Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.
In the Google Cloud console, on the project selector page, select or create a Google Cloud project.
Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.Roles required to select or create a project
- Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
- Create a project: To create a project, you need the Project Creator role (
roles/resourcemanager.projectCreator), which contains theresourcemanager.projects.createpermission.Learn how to grant roles.
Verify that billing is enabled for your Google Cloud project.
Enable the Data Catalog and BigQuery Data Policy APIs.
Roles required to enable APIs
To enable APIs, you need the Service Usage Admin IAM role (
roles/serviceusage.serviceUsageAdmin), which contains theserviceusage.services.enablepermission.Learn how to grant roles.In the Google Cloud console, on the project selector page, select or create a Google Cloud project.
Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.Roles required to select or create a project
- Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
- Create a project: To create a project, you need the Project Creator role (
roles/resourcemanager.projectCreator), which contains theresourcemanager.projects.createpermission.Learn how to grant roles.
Verify that billing is enabled for your Google Cloud project.
Enable the Data Catalog and BigQuery Data Policy APIs.
Roles required to enable APIs
To enable APIs, you need the Service Usage Admin IAM role (
roles/serviceusage.serviceUsageAdmin), which contains theserviceusage.services.enablepermission.Learn how to grant roles.- BigQuery is automatically enabled in new projects, but you might need to activate it in a pre-existing project.
Enable the BigQuery API.
Roles required to enable APIs
To enable APIs, you need the Service Usage Admin IAM role (
roles/serviceusage.serviceUsageAdmin), which contains theserviceusage.services.enablepermission.Learn how to grant roles. - If you are creating a data policy that references acustom masking routine, create the associated masking UDF so that it is available in the following steps.
Create taxonomies
The user or service account that creates a taxonomy must be granted theData Catalog Policy Tag Admin role.
Console
- Open thePolicy tag taxonomies page in theGoogle Cloud console.
- ClickCreate taxonomy.
On theNew taxonomy page:
- ForTaxonomy name, enter the name of the taxonomy that you wantto create.
- ForDescription, enter a description.
- If needed, change the project listed underProject.
- If needed, change the location listed underLocation.
- UnderPolicy Tags, enter a policy tag name and description.
- To add a child policy tag for a policy tag, clickAdd subtag.
- To add a new policy tag at the same level as another policy tag,click+ Add policy tag.
- Continue adding policy tags and child policy tags as needed for yourtaxonomy.
- When you are done creating policy tags for your hierarchy, clickCreate.
API
To use existing taxonomies, calltaxonomies.importin place of the first two steps of the following procedure.
- Call
taxonomies.createto create a taxonomy. - Call
taxonomies.policytag.createto create a policy tag.
Work with policy tags
For more information about how to work with policy tags, such as how to view orupdate them, seeWork with policy tags.For best practices, seeBest practices for using policy tags in BigQuery.
Create data policies
The user or service account that creates a data policy must have thebigquery.dataPolicies.create,bigquery.dataPolicies.setIamPolicy, anddatacatalog.taxonomies.get permissions.
Thebigquery.dataPolicies.create andbigquery.dataPolicies.setIamPolicypermissions are included in theBigQuery Data Policy Admin, BigQuery Admin and BigQuery Data Owner roles.Thedatacatalog.taxonomies.get permission is included in theData Catalog Admin and Data Catalog Viewer roles.
If you are creating a data policy that references acustom masking routine,you also needroutine permissions.
In case of custom masking, grant users the BigQuery Admin or BigQuery Data Ownerroles to ensure they have the necessary permissions for both routines and data policies.
You can create up to nine data policies for a policy tag. One of these policiesis reserved forcolumn-level access control settings.
Console
- Open thePolicy tag taxonomies page in theGoogle Cloud console.
- Click the name of the taxonomy to open.
- Select a policy tag.
- ClickManage Data Policies.
- ForData Policy Name, type a name for the data policy. The datapolicy name must be unique within the project that data policy residesin.
- ForMasking Rule, choose a predefined masking rule or a custommasking routine. If you are selecting a custom masking routine, ensurethat you have both the
bigquery.routines.getand thebigquery.routines.listpermissions at the project level. - ForPrincipal, type the name of one or more users or groups to whomyou want to grant masked access to the column. Note that all users andgroups you enter here are granted the BigQuery MaskedReader role.
- ClickSubmit.
API
Call the
createmethod. Pass in aDataPolicyresource That meets the following requirements:- The
dataPolicyTypefield is set toDATA_MASKING_POLICY. - The
dataMaskingPolicyfield identifies the data masking rule orroutine to use. - The
dataPolicyIdfield provides a name for the data policy that is unique within the project that data policy resides in.
- The
Call the
setIamPolicymethod and pass in aPolicy. ThePolicymustidentify the principals who are granted access to masked data,and specifyroles/bigquerydatapolicy.maskedReaderfor therolefield.
Node.js
Before trying this sample, follow theNode.js setup instructions in theBigQuery quickstart using client libraries. For more information, see theBigQueryNode.js API reference documentation.
To authenticate to BigQuery, set up Application Default Credentials. For more information, seeSet up authentication for client libraries.
constdatapolicy=require('@google-cloud/bigquery-datapolicies');const{DataPolicyServiceClient}=datapolicy.v2;constprotos=datapolicy.protos.google.cloud.bigquery.datapolicies.v2;const{status}=require('@grpc/grpc-js');constdataPolicyServiceClient=newDataPolicyServiceClient();/** * Creates a data policy to apply a data masking rule to a specific BigQuery table column. * This is a primary mechanism for implementing column-level security in BigQuery. * * @param {string} projectId The Google Cloud project ID (for example, 'example-project-id') * @param {string} location The Google Cloud location. Example: 'us' * @param {string} dataPolicyId The user-assigned ID of the data policy. Example: 'example-data-policy-id' */asyncfunctioncreateDataPolicy(projectId,location,dataPolicyId){constparent=`projects/${projectId}/locations/${location}`;constdataPolicy={dataPolicyType:protos.DataPolicy.DataPolicyType.DATA_MASKING_POLICY,dataMaskingPolicy:{predefinedExpression:protos.DataMaskingPolicy.PredefinedExpression.SHA256,},};constrequest={parent,dataPolicyId,dataPolicy,};try{const[response]=awaitdataPolicyServiceClient.createDataPolicy(request);console.log(`Successfully created data policy:${response.name}`);console.log(`Data policy ID:${response.dataPolicyId}`);console.log(`Data policy type:${response.dataPolicyType}`);if(response.dataMaskingPolicy){console.log(`Data masking expression:${response.dataMaskingPolicy.predefinedExpression}`,);}}catch(err){if(err.code===status.ALREADY_EXISTS){console.log(`Data policy '${dataPolicyId}' already exists in location '${location}' of project '${projectId}'.`,);console.log('Consider updating the existing data policy or using a different dataPolicyId.',);}else{console.error('Error creating data policy:',err.message);}}}Python
Before trying this sample, follow thePython setup instructions in theBigQuery quickstart using client libraries. For more information, see theBigQueryPython API reference documentation.
To authenticate to BigQuery, set up Application Default Credentials. For more information, seeSet up authentication for client libraries.
fromgoogle.api_coreimportexceptionsfromgoogle.cloudimportbigquery_datapolicies_v2client=bigquery_datapolicies_v2.DataPolicyServiceClient()defcreate_data_policy(project_id:str,location:str,data_policy_id:str)->None:"""Creates a data policy to apply a data masking rule to a specific BigQuery table column. This is a primary mechanism for implementing column-level security in BigQuery. Args: project_id (str): The Google Cloud project ID. location (str): The geographic location of the data policy (for example, "us-central1"). data_policy_id (str): The ID for the new data policy. """parent=f"projects/{project_id}/locations/{location}"# Define the data masking policy.# Here, we specify a SHA-256 predefined expression for data masking.data_masking_policy=bigquery_datapolicies_v2.DataMaskingPolicy(predefined_expression=bigquery_datapolicies_v2.DataMaskingPolicy.PredefinedExpression.SHA256)# Create the DataPolicy object.# We set the type to DATA_MASKING_POLICY and assign the defined masking policy.data_policy=bigquery_datapolicies_v2.DataPolicy(data_policy_type=bigquery_datapolicies_v2.DataPolicy.DataPolicyType.DATA_MASKING_POLICY,data_masking_policy=data_masking_policy,)request=bigquery_datapolicies_v2.CreateDataPolicyRequest(parent=parent,data_policy_id=data_policy_id,data_policy=data_policy,)try:response=client.create_data_policy(request=request)print(f"Successfully created data policy:{response.name}")print(f"Data Policy ID:{response.data_policy_id}")print(f"Data Policy Type:{response.data_policy_type.name}")print("Data Masking Predefined Expression:"f"{response.data_masking_policy.predefined_expression.name}")exceptexceptions.AlreadyExistsase:print(f"Error: Data policy '{data_policy_id}' already exists in project"f" '{project_id}' in location '{location}'. Use a unique ID or"" update the existing policy if needed.")exceptexceptions.NotFoundase:print(f"Error: The specified project '{project_id}' or location '{location}'"" was not found or is inaccessible. Make sure the project ID and"" location are correct and you have the necessary permissions.")exceptExceptionase:print(f"An unexpected error occurred:{e}")Set policy tags on columns
Set a data policy on a column by attaching the policy tag associated withthe data policy to the column.
The user or service account that sets a policy tag needs thedatacatalog.taxonomies.get andbigquery.tables.setCategory permissions.datacatalog.taxonomies.get is included in theData Catalog Policy Tags Admin and Project Viewer roles.bigquery.tables.setCategory is included in theBigQuery Admin (roles/bigquery.admin) andBigQuery Data Owner (roles/bigquery.dataOwner) roles.
To view taxonomies and policy tags across all projects in an organization inGoogle Cloud console, users need theresourcemanager.organizations.getpermission, which is included in the Organization Viewer role.
Console
Set the policy tag bymodifying a schema using theGoogle Cloud console.
Open the BigQuery page in the Google Cloud console.
In the BigQuery Explorer, locate and select the table thatyou want to update. The table schema for that table opens.
ClickEdit Schema.
In theCurrent schema screen, select the target column and clickAddpolicy tag.
In theAdd a policy tag screen, locate and select the policy tag that you wantto apply to the column.
ClickSelect. Your screen should look similar to the following:

ClickSave.
bq
Write the schema to a local file.
bqshow--schema--format=prettyjson\project-id:dataset.table>schema.json
where:
- project-id is your project ID.
- dataset is the name of the dataset that contains the tableyou're updating.
- table is the name of the table you're updating.
Modify schema.json to set a policy tag on a column. For the value of the
namesfield ofpolicyTags, use thepolicy tag resource name.[...{"name":"ssn","type":"STRING","mode":"REQUIRED","policyTags":{"names":["projects/project-id/locations/location/taxonomies/taxonomy-id/policyTags/policytag-id"]}},...]
Update the schema.
bqupdate\project-id:dataset.tableschema.json
API
For existing tables, calltables.patch, or for new tables calltables.insert. Use theschema property of theTable object that you pass into set a policy tag in your schema definition. See the command-line exampleschema to see how to set a policy tag.
When working with an existing table, thetables.patch method is preferred,because thetables.update method replaces the entire table resource.
Enforce access control
When you create a data policy for a policy tag, access control is automaticallyenforced. All columns that have that policy tag applied return masked data inresponse to queries from users who have the Masked Reader role.
To stop enforcement of access control, you mustfirst delete all data policies associated with the policy tags in thetaxonomy. For more information, seeEnforce access control.
Get a data policy
Follow these steps to get information about a data policy:
Node.js
Before trying this sample, follow theNode.js setup instructions in theBigQuery quickstart using client libraries. For more information, see theBigQueryNode.js API reference documentation.
To authenticate to BigQuery, set up Application Default Credentials. For more information, seeSet up authentication for client libraries.
const{DataPolicyServiceClient}=require('@google-cloud/bigquery-datapolicies').v2;const{status}=require('@grpc/grpc-js');constclient=newDataPolicyServiceClient();/** * Gets a specific data policy from the BigQuery Data Policy API by its name. * * This sample demonstrates how to fetch the details of an existing data policy. * Data policies are used to define rules for data masking or row-level security * on BigQuery tables. * * @param {string} projectId The Google Cloud project ID (for example, 'example-project-id') * @param {string} [location='us'] The Google Cloud location of the data policy (For example, 'us', 'europe-west2'). * @param {string} [dataPolicyId='example-data-policy'] The ID of the data policy to retrieve. */asyncfunctiongetDataPolicy(projectId,location='us',dataPolicyId='example-data-policy',){constname=client.dataPolicyPath(projectId,location,dataPolicyId);constrequest={name,};try{const[dataPolicy]=awaitclient.getDataPolicy(request);console.log('Successfully retrieved data policy:');console.log(` Name:${dataPolicy.name}`);console.log(` Type:${dataPolicy.dataPolicyType}`);if(dataPolicy.dataMaskingPolicy){console.log(` Data Masking Policy:${dataPolicy.dataMaskingPolicy.predefinedExpression||dataPolicy.dataMaskingPolicy.routine}`,);}if(dataPolicy.grantees &&dataPolicy.grantees.length >0){console.log(` Grantees:${dataPolicy.grantees.join(', ')}`);}}catch(err){if(err.code===status.NOT_FOUND){console.error(`Error: Data policy '${dataPolicyId}' not found in location '${location}' for project '${projectId}'.`,);console.error('Make sure the data policy ID, project ID, and location are correct.',);}else{console.error('Error retrieving data policy:',err.message);}}}Python
Before trying this sample, follow thePython setup instructions in theBigQuery quickstart using client libraries. For more information, see theBigQueryPython API reference documentation.
To authenticate to BigQuery, set up Application Default Credentials. For more information, seeSet up authentication for client libraries.
fromgoogle.api_coreimportexceptionsfromgoogle.cloudimportbigquery_datapolicies_v2client=bigquery_datapolicies_v2.DataPolicyServiceClient()defget_data_policy(project_id:str,location:str,data_policy_id:str,)->None:"""Gets a specific data policy from the BigQuery Data Policy API by its name. Args: project_id: The Google Cloud project ID. location: The geographic location of the data policy (for example, "us", "eu"). data_policy_id: The user-assigned ID of the data policy. """client=bigquery_datapolicies_v2.DataPolicyServiceClient()data_policy_name=client.data_policy_path(project=project_id,location=location,data_policy=data_policy_id,)try:response=client.get_data_policy(name=data_policy_name)print(f"Successfully retrieved data policy:{response.name}")print(f" Data Policy ID:{response.data_policy_id}")print(f" Data Policy Type:{response.data_policy_type.name}")ifresponse.policy_tag:print(f" Policy Tag:{response.policy_tag}")ifresponse.grantees:print(f" Grantees:{', '.join(response.grantees)}")ifresponse.data_masking_policy:masking_policy=response.data_masking_policyifmasking_policy.predefined_expression:print(f" Data Masking Predefined Expression:{masking_policy.predefined_expression.name}")elifmasking_policy.routine:print(f" Data Masking Routine:{masking_policy.routine}")exceptexceptions.NotFound:print(f"Error: Data policy '{data_policy_name}' not found.")print("Make sure the data policy ID, project ID, and location are correct.")exceptExceptionase:print(f"An unexpected error occurred:{e}")Check IAM permissions on a data policy
Follow these steps to get the IAM policy for a data policy:
API
Call thetestIamPermissions method.
Node.js
Before trying this sample, follow theNode.js setup instructions in theBigQuery quickstart using client libraries. For more information, see theBigQueryNode.js API reference documentation.
To authenticate to BigQuery, set up Application Default Credentials. For more information, seeSet up authentication for client libraries.
const{DataPolicyServiceClient}=require('@google-cloud/bigquery-datapolicies').v2;const{status}=require('@grpc/grpc-js');constclient=newDataPolicyServiceClient();/** * Get the IAM policy for a specified data policy resource from the BigQuery Data Policy API. * This is useful for auditing which members have which roles on the policy. * * * @param {string} projectId Google Cloud Project ID (For example, 'example-project-id') * @param {string} location Google Cloud Location (For example, 'us-central1') * @param {string} dataPolicyId The ID of the data policy (For example, 'example-data-policy-id') */asyncfunctiongetIamPolicy(projectId,location,dataPolicyId){constresourceName=client.dataPolicyPath(projectId,location,dataPolicyId);constrequest={resource:resourceName,};try{const[policy]=awaitclient.getIamPolicy(request);console.log('Successfully retrieved IAM policy for data policy %s:',resourceName,);console.log(JSON.stringify(policy,null,2));}catch(err){if(err.code===status.NOT_FOUND){console.error(`Error: Data Policy '${dataPolicyId}' not found in location '${location}' of project '${projectId}'. `+'Make sure the data policy exists and the resource name is correct.',);}else{console.error(`Error getting IAM policy for data policy '${dataPolicyId}':`,err,);}}}Python
Before trying this sample, follow thePython setup instructions in theBigQuery quickstart using client libraries. For more information, see theBigQueryPython API reference documentation.
To authenticate to BigQuery, set up Application Default Credentials. For more information, seeSet up authentication for client libraries.
fromgoogle.api_coreimportexceptionsfromgoogle.cloudimportbigquery_datapolicies_v2fromgoogle.iam.v1importiam_policy_pb2client=bigquery_datapolicies_v2.DataPolicyServiceClient()defget_data_policy_iam_policy(project_id:str,location:str,data_policy_id:str,)->None:"""Get the IAM policy for a specified data policy resource from the BigQuery Data Policy API. This is useful for auditing which members have which roles on the policy. Args: project_id: The Google Cloud project ID. location: The geographic location of the data policy (for example, "us"). data_policy_id: The ID of the data policy. """resource_name=client.data_policy_path(project=project_id,location=location,data_policy=data_policy_id,)request=iam_policy_pb2.GetIamPolicyRequest(resource=resource_name)try:policy=client.get_iam_policy(request=request)print(f"Successfully retrieved IAM policy for data policy:{resource_name}")print("Policy Version:",policy.version)ifpolicy.bindings:print("Policy Bindings:")forbindinginpolicy.bindings:print(f" Role:{binding.role}")print(f" Members:{', '.join(binding.members)}")ifbinding.condition.expression:print(f" Condition:{binding.condition.expression}")else:print("No bindings found in the policy.")exceptexceptions.NotFound:print(f"Error: Data policy '{resource_name}' not found.")print("Make sure the project ID, location, and data policy ID are correct.")exceptexceptions.GoogleAPIErrorase:print(f"An API error occurred:{e}")exceptExceptionase:print(f"An unexpected error occurred:{e}")List data policies
Follow these steps to list data policies:
Node.js
Before trying this sample, follow theNode.js setup instructions in theBigQuery quickstart using client libraries. For more information, see theBigQueryNode.js API reference documentation.
To authenticate to BigQuery, set up Application Default Credentials. For more information, seeSet up authentication for client libraries.
const{DataPolicyServiceClient}=require('@google-cloud/bigquery-datapolicies').v2;const{status}=require('@grpc/grpc-js');constclient=newDataPolicyServiceClient();/** * Lists all data policies in a given project and location. * * Data policies define rules for data masking, row-level security, or column-level security. * * @param {string} projectId The Google Cloud project ID. (for example, 'example-project-id') * @param {string} location The Google Cloud location of the data policies. (For example, 'us') */asyncfunctionlistDataPolicies(projectId,location){constparent=`projects/${projectId}/locations/${location}`;constrequest={parent,};try{console.log(`Listing data policies for project:${projectId} in location:${location}`,);const[dataPolicies]=awaitclient.listDataPolicies(request);if(dataPolicies.length===0){console.log(`No data policies found in location${location} for project${projectId}.`,);return;}console.log('Data Policies:');for(constdataPolicyofdataPolicies){console.log(` Data Policy Name:${dataPolicy.name}`);console.log(` ID:${dataPolicy.dataPolicyId}`);console.log(` Type:${dataPolicy.dataPolicyType}`);if(dataPolicy.policyTag){console.log(` Policy Tag:${dataPolicy.policyTag}`);}if(dataPolicy.grantees &&dataPolicy.grantees.length >0){console.log(` Grantees:${dataPolicy.grantees.join(', ')}`);}if(dataPolicy.dataMaskingPolicy){if(dataPolicy.dataMaskingPolicy.predefinedExpression){console.log(` Data Masking Predefined Expression:${dataPolicy.dataMaskingPolicy.predefinedExpression}`,);}elseif(dataPolicy.dataMaskingPolicy.routine){console.log(` Data Masking Routine:${dataPolicy.dataMaskingPolicy.routine}`,);}}}console.log(`Successfully listed${dataPolicies.length} data policies.`);}catch(err){if(err.code===status.NOT_FOUND){console.error(`Error: The project or location '${location}' for project '${projectId}' was not found. `+'Make sure the project ID and location are correct and that the BigQuery Data Policy API is enabled.',);}elseif(err.code===status.PERMISSION_DENIED){console.error(`Error: Permission denied when listing data policies for project '${projectId}' in location '${location}'. `+'Make sure the authenticated account has the necessary permissions (For example, bigquery.datapolicies.list).',);}else{console.error(`Error listing data policies:${err.message}`);}}}Python
Before trying this sample, follow thePython setup instructions in theBigQuery quickstart using client libraries. For more information, see theBigQueryPython API reference documentation.
To authenticate to BigQuery, set up Application Default Credentials. For more information, seeSet up authentication for client libraries.
importgoogle.api_core.exceptionsfromgoogle.cloudimportbigquery_datapolicies_v2client=bigquery_datapolicies_v2.DataPolicyServiceClient()deflist_data_policies(project_id:str,location:str)->None:"""Lists all data policies in a specified project. Args: project_id: The Google Cloud project ID. location: The geographic location of the data policies (for example, "us", "us-central1"). """parent=f"projects/{project_id}/locations/{location}"try:request=bigquery_datapolicies_v2.ListDataPoliciesRequest(parent=parent)print(f"Listing data policies for project '{project_id}' in location '{location}':")page_result=client.list_data_policies(request=request)found_policies=Falsefordata_policyinpage_result:found_policies=Trueprint(f" Data Policy Name:{data_policy.name}")print(f" Data Policy ID:{data_policy.data_policy_id}")print(f" Data Policy Type:{data_policy.data_policy_type.name}")ifdata_policy.policy_tag:print(f" Policy Tag:{data_policy.policy_tag}")ifdata_policy.grantees:print(f" Grantees:{', '.join(data_policy.grantees)}")print("-"*20)ifnotfound_policies:print("No data policies found.")exceptgoogle.api_core.exceptions.NotFoundase:print(f"Error: The specified project or location was not found or accessible.")print(f"Details:{e}")print("Make sure the project ID and location are correct and you have the necessary permissions.")exceptExceptionase:print(f"An unexpected error occurred:{e}")Update data policies
The user or service account that updates a data policy must have thebigquery.dataPolicies.update permission.If you are updating the policy tag the data policy is associated with, you alsorequire thedatacatalog.taxonomies.get permission.
If you are updating the principals associated with the data policy, yourequire thebigquery.dataPolicies.setIamPolicy permission.
Thebigquery.dataPolicies.update andbigquery.dataPolicies.setIamPolicypermissions are included in theBigQuery Data Policy Admin, BigQuery Admin and BigQuery Data Owner roles.Thedatacatalog.taxonomies.get permission is included in theData Catalog Admin and Data Catalog Viewer roles.
Console
- Open thePolicy tag taxonomies page in theGoogle Cloud console.
- Click the name of the taxonomy to open.
- Select a policy tag.
- ClickManage Data Policies.
- Optionally, change the masking rule.
- Optional: Add or remove principals.
- ClickSubmit.
API
To change the data masking rule, call thepatchmethod and pass in aDataPolicyresource with an updateddataMaskingPolicy field.
To change the principals associated with a data policy, call thesetIamPolicymethod and pass in aPolicy that updatesthe principals that are granted access to masked data.
Node.js
Before trying this sample, follow theNode.js setup instructions in theBigQuery quickstart using client libraries. For more information, see theBigQueryNode.js API reference documentation.
To authenticate to BigQuery, set up Application Default Credentials. For more information, seeSet up authentication for client libraries.
constdatapolicy=require('@google-cloud/bigquery-datapolicies');const{DataPolicyServiceClient}=datapolicy.v2;constprotos=datapolicy.protos.google.cloud.bigquery.datapolicies.v2;const{status}=require('@grpc/grpc-js');constclient=newDataPolicyServiceClient();/** * Updates the data masking configuration of an existing data policy. * This example demonstrates how to use a FieldMask to selectively update the * `data_masking_policy` (for example, changing the masking expression from * ALWAYS_NULL to SHA256) without affecting other fields or recreating the policy. * * @param {string} projectId The Google Cloud project ID (For example, 'example-project-id'). * @param {string} location The location of the data policy (For example, 'us'). * @param {string} dataPolicyId The ID of the data policy to update (For example, 'example-data-policy-id'). */asyncfunctionupdateDataPolicy(projectId,location,dataPolicyId){constresourceName=client.dataPolicyPath(projectId,location,dataPolicyId);constgetRequest={name:resourceName,};try{// To prevent race conditions, use the policy's etag in the update.const[currentDataPolicy]=awaitclient.getDataPolicy(getRequest);constcurrentETag=currentDataPolicy.etag;// This example transitions a masking rule from ALWAYS_NULL to SHA256.constdataPolicy={name:resourceName,etag:currentETag,dataMaskingPolicy:{predefinedExpression:protos.DataMaskingPolicy.PredefinedExpression.SHA256,},};// Use a field mask to selectively update only the data masking policy.constupdateMask={paths:['data_masking_policy'],};constrequest={dataPolicy,updateMask,};const[response]=awaitclient.updateDataPolicy(request);console.log(`Successfully updated data policy:${response.name}`);console.log(`New masking expression:${response.dataMaskingPolicy.predefinedExpression}`);}catch(err){if(err.code===status.NOT_FOUND){console.error(`Error: Data policy '${resourceName}' not found. `+'Make sure the data policy exists and the project, location, and data policy ID are correct.');}else{console.error('Error updating data policy:',err.message,err);}}}Python
Before trying this sample, follow thePython setup instructions in theBigQuery quickstart using client libraries. For more information, see theBigQueryPython API reference documentation.
To authenticate to BigQuery, set up Application Default Credentials. For more information, seeSet up authentication for client libraries.
fromgoogle.api_coreimportexceptionsfromgoogle.cloudimportbigquery_datapolicies_v2fromgoogle.protobufimportfield_mask_pb2client=bigquery_datapolicies_v2.DataPolicyServiceClient()defupdate_data_policy(project_id:str,location:str,data_policy_id:str)->None:"""Updates the data masking configuration of an existing data policy. This example demonstrates how to use a FieldMask to selectively update the `data_masking_policy` (for example, changing the masking expression from ALWAYS_NULL to SHA256) without affecting other fields or recreating the policy. Args: project_id: The Google Cloud project ID. location: The geographic location (for example, "us") of the data policy. data_policy_id: The ID of the data policy to update. """data_policy_name=client.data_policy_path(project=project_id,location=location,data_policy=data_policy_id,)# To prevent race conditions, use the policy's etag in the update.existing_policy=client.get_data_policy(name=data_policy_name)# This example transitions a masking rule from ALWAYS_NULL to SHA256.updated_data_policy=bigquery_datapolicies_v2.DataPolicy(name=data_policy_name,data_masking_policy=bigquery_datapolicies_v2.DataMaskingPolicy(predefined_expression=bigquery_datapolicies_v2.DataMaskingPolicy.PredefinedExpression.SHA256),etag=existing_policy.etag,)# Use a field mask to selectively update only the data masking policy.update_mask=field_mask_pb2.FieldMask(paths=["data_masking_policy"])request=bigquery_datapolicies_v2.UpdateDataPolicyRequest(data_policy=updated_data_policy,update_mask=update_mask,)try:response=client.update_data_policy(request=request)print(f"Successfully updated data policy:{response.name}")print(f"New data policy type:{response.data_policy_type.name}")ifresponse.data_masking_policy:print(f"New masking expression:{response.data_masking_policy.predefined_expression.name}")exceptexceptions.NotFound:print(f"Error: Data policy '{data_policy_name}' not found.")print("Make sure the data policy ID and location are correct.")exceptExceptionase:print(f"An unexpected error occurred:{e}")Delete data policies
The user or service account that creates a data policy must have thebigquery.dataPolicies.delete permission. This permission is included in theBigQuery Data Policy Admin, BigQuery Admin and BigQuery Data Owner roles.
Console
- Open thePolicy tag taxonomies page in theGoogle Cloud console.
- Click the name of the taxonomy to open.
- Select a policy tag.
- ClickManage Data Policies.
- Click next to the data policyto delete.
- ClickSubmit.
- ClickConfirm.
API
To delete a data policy, call thedeletemethod.
Node.js
Before trying this sample, follow theNode.js setup instructions in theBigQuery quickstart using client libraries. For more information, see theBigQueryNode.js API reference documentation.
To authenticate to BigQuery, set up Application Default Credentials. For more information, seeSet up authentication for client libraries.
const{DataPolicyServiceClient}=require('@google-cloud/bigquery-datapolicies').v2;const{status}=require('@grpc/grpc-js');constclient=newDataPolicyServiceClient();/** * Deletes a data policy from the BigQuery Data Policy API, which is identified by its project ID, location, and data policy ID. * * @param {string} projectId The Google Cloud project ID. * @param {string} location The Google Cloud location (For example, 'us'). * @param {string} dataPolicyId The ID of the data policy to delete (For example, 'example-data-policy'). */asyncfunctiondeleteDataPolicy(projectId,location,dataPolicyId){constname=client.dataPolicyPath(projectId,location,dataPolicyId);constrequest={name,};try{awaitclient.deleteDataPolicy(request);console.log(`Successfully deleted data policy:${name}`);}catch(err){if(err.code===status.NOT_FOUND){console.error(`Data policy${name} not found. Make sure the data policy ID and location are correct.`,);}else{console.error(`Error deleting data policy${name}:`,err.message);}}}Python
Before trying this sample, follow thePython setup instructions in theBigQuery quickstart using client libraries. For more information, see theBigQueryPython API reference documentation.
To authenticate to BigQuery, set up Application Default Credentials. For more information, seeSet up authentication for client libraries.
fromgoogle.api_coreimportexceptionsascore_exceptionsfromgoogle.cloudimportbigquery_datapolicies_v2client=bigquery_datapolicies_v2.DataPolicyServiceClient()defdelete_data_policy(project_id:str,location:str,data_policy_id:str)->None:"""Deletes a data policy from the BigQuery Data Policy APIs. Args: project_id: The Google Cloud project ID. location: The location of the data policy (for example, "us"). data_policy_id: The ID of the data policy to delete. """name=client.data_policy_path(project=project_id,location=location,data_policy=data_policy_id)try:client.delete_data_policy(name=name)print(f"Successfully deleted data policy:{name}")exceptcore_exceptions.NotFound:print(f"Data policy '{name}' not found. It may have already been deleted.")exceptExceptionase:print(f"Error deleting data policy '{name}':{e}")Mask data by applying data policies to a column
Preview
This product or feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of theService Specific Terms. Pre-GA products and features are available "as is" and might have limited support. For more information, see thelaunch stage descriptions.
Note: To provide feedback or request support for this feature, send an email tobigquery-security@google.com.As an alternative to creating policy tags, you can create data policies andapply them directly on a column.
Work with data policies
You can create, update, and delete data policies using theBigQuery Data Policy API. Toapply a data policy directly on a column, you can't use thePolicy tagtaxonomies page in the Google Cloud console.
To work with data policies, use thev2.projects.locations.datapoliciesresource.
Create data policies
The user or service account that creates a data policy must have thebigquery.dataPolicies.create permission.
Thebigquery.dataPolicies.create permission is included in theBigQuery Data Policy Admin, BigQuery Admin and BigQuery Data Owner roles.Thedatacatalog.taxonomies.get permission is included in theData Catalog Admin and Data Catalog Viewer roles.
If you are creating a data policy that references acustom masking routine,you also needroutine permissions.
If you use custom masking, grant users the BigQuery Data Ownerrole to ensure they have the necessary permissions for both routines and data policies.
API
To create a data policy, call thecreatemethod. Pass in aDataPolicyresource that meets the following requirements:
- The
dataPolicyTypefield is set toDATA_MASKING_POLICYorRAW_DATA_ACCESS_POLICY. - The
dataMaskingPolicyfield identifies the data masking rule orroutine to use. - The
dataPolicyIdfield provides a name for the data policy thatis unique within the project that the data policy resides in.
SQL
To create a data policy with masked access, use theCREATE DATA_POLICYstatement and set the value ofdata_policy_type toDATA_MASKING_POLICY:
CREATE[ORREPLACE]DATA_POLICY[IFNOTEXISTS]`myproject.region-us.data_policy_name`OPTIONS(data_policy_type="DATA_MASKING_POLICY",masking_expression="ALWAYS_NULL");
To create a data policy with raw access, use theCREATE DATA_POLICY statementand set the value ofdata_policy_type toRAW_DATA_ACCESS_POLICY:
CREATE[ORREPLACE]DATA_POLICY[IFNOTEXISTS]`myproject.region-us.data_policy_name`OPTIONS(data_policy_type="RAW_DATA_ACCESS_POLICY");
If the value ofdata_policy_type isn't specified, the default value isRAW_DATA_ACCESS_POLICY.
CREATE[ORREPLACE]DATA_POLICY[IFNOTEXISTS]myproject.region-us.data_policy_name;- The
data_policy_typefield is set toDATA_MASKING_POLICYorRAW_DATA_ACCESS_POLICY. You can't update this field oncethe data policy has been created. - The
masking_expressionfield identifies the data masking rule orroutine to use.
Update data policies
The user or service account that updates a data policy must have thebigquery.dataPolicies.update permission.
Thebigquery.dataPolicies.update permission is included in theBigQuery Data Policy Admin, BigQuery Admin and BigQuery Data Owner roles.
API
To change the data masking rule, call thepatchmethod and pass in aDataPolicyresource with an updateddataMaskingPolicy field.
SQL
Use theALTER DATA_POLICY statement to update the data masking rules. For example:
ALTERDATA_POLICY`myproject.region-us.data_policy_name`SETOPTIONS(data_policy_type="DATA_MASKING_POLICY",masking_expression="SHA256");
You can also grant fine-grained access control access to data policies.
The permissions to grant fine-grained access control access to data policies and manage datapolicies are different. To control fine-grained access control permissions, you must updatethegrantees field of the data policy. To control access to the data policies,set the IAM roles using thesetIamPolicymethod.
To set grantees on a data policy, use thev2patchmethod. To manage data policy permissions, use thev1setIamPolicymethod.
API
To grant fine-grained access control access to data policies, call thepatchmethod and pass in aDataPolicyresource with an updatedgrantees field.
SQL
To grant fine-grained access control access to data policies, use theGRANT FINE_GRAINED_READ statement to addgrantees.For example:
GRANTFINE_GRAINED_READONDATA_POLICY`myproject.region-us.data_policy_name`TO"principal://goog/subject/user1@example.com","principal://goog/subject/user2@example.com"
To revoke fine-grained access control access from data policies, use theREVOKE FINE_GRAINED_READ statement to removegrantees.For example:
REVOKEFINE_GRAINED_READONDATA_POLICY`myproject.region-us.data_policy_name`FROM"principal://goog/subject/user1@example.com","principal://goog/subject/user2@example.com"
Delete data policies
The user or service account that creates a data policy must have thebigquery.dataPolicies.delete permission. This permission is included in theBigQuery Data Policy Admin, BigQuery Admin and BigQuery Data Owner roles.
API
To delete a data policy, call thedeletemethod.
SQL
UseDROP DATA_POLICY statement to delete a data policy:
DROPDATA_POLICY`myproject.region-us.data_policy_name`;
Assign a data policy directly on a column
You can assign a data policy directly on a column without using policy tags.
Before you begin
To get the permissions that you need to assign a data policy directly on a column, ask your administrator to grant you theBigQuery Data Policy Admin (roles/bigquerydatapolicy.admin) IAM role on your table. For more information about granting roles, seeManage access to projects, folders, and organizations.
This predefined role contains the permissions required to assign a data policy directly on a column. To see the exact permissions that are required, expand theRequired permissions section:
Required permissions
The following permissions are required to assign a data policy directly on a column:
bigquery.tables.updatebigquery.tables.setColumnDataPolicybigquery.dataPolicies.attach
You might also be able to get these permissions withcustom roles or otherpredefined roles.
Assign a data policy
To assign a data policy directly on a column, do one of the following:
SQL
To attach a data policy to a column, use theCREATETABLE,ALTER TABLE ADDCOLUMN,orALTER COLUMN SETOPTIONSDDL statements.
The following example uses theCREATE TABLE statement and sets data policieson a column:
CREATETABLEmyproject.table1(nameINT64OPTIONS(data_policies=["{'name':'myproject.region-us.data_policy_name1'}","{'name':'myproject.region-us.data_policy_name2'}"]));
The following example uses theALTER COLUMN SET OPTIONS to add a data policyto an existing column on a table:
ALTERTABLEmyproject.table1ALTERCOLUMNcolumn_nameSETOPTIONS(data_policies+=["{'name':'myproject.region-us.data_policy_name1'}","{'name':'myproject.region-us.data_policy_name2'}"]);
API
To assign a data policy to a column, call thepatch method on the tableand update the table schema with the applicable data policies.
Unassign a data policy
To unassign a data policy directly on a column, do one of the following:
SQL
To detach a data policy to a column, use theALTER COLUMN SETOPTIONSDDL statement.
The following example uses theALTER COLUMN SET OPTIONS to remove all data policiesfrom an existing column on a table:
ALTERTABLEmyproject.table1ALTERCOLUMNcolumn_nameSETOPTIONS(data_policies=[]);
The following example uses theALTER COLUMN SET OPTIONS to replace data policiesfrom an existing column on a table:
ALTERTABLEmyproject.table1ALTERCOLUMNcolumn_nameSETOPTIONS(data_policies=["{'name':'myproject.region-us.new_data_policy_name'}"]);
API
To unassign a data policy to a column, call thepatch method on the tableand update the table schema empty or updated data policies.
Limitations
Assigning a data policy directly on a column is subject to the following limitations:
- You must use the
v2.projects.locations.datapoliciesresource. - You can't apply both policy tags and data policies to the same column.
- You can attach a maximum of eight data policies to a column.
- A table can reference a maximum of 1,000 unique data policies through itscolumns.
- A query can reference a maximum of 2,000 data policies.
- You can delete a data policy only if no table column references it.
- If a user only has the
maskedAccessrole, thetabledata.listAPI call fails. - Table copy operations fail on tables protected by column data policies ifthe user lacks raw data access.
- Cross-region table copy operations don't support tables protected by columndata policies.
- Column data policies are unavailable in BigQuery Omni regions.
- Legacy SQL fails if the target table has column data policies.
- Load jobs don't support user-specified schemas with column data policies.
- If you overwrite a destination table, the system removes any existing policytags from the table, unless you use the
--destination_schemaflag tospecify a schema with column data policies. - By default, data masking does not support partitioned or clustered columns.This is a general limitation of data masking, not specific to column datapolicies. Data masking on partitioned or clustered columns can significantlyincrease query costs.
- You can't assign a data policy directly on a column inBigLake tables for Apache Iceberg in BigQuery,object tables,non-BigLake external tables,Apache Iceberg external tables andDelta Lake.
- Fine Grained Access can be granted only at the data policy level. For more information, seeUpdate data policies.
Except as otherwise noted, the content of this page is licensed under theCreative Commons Attribution 4.0 License, and code samples are licensed under theApache 2.0 License. For details, see theGoogle Developers Site Policies. Java is a registered trademark of Oracle and/or its affiliates.
Last updated 2025-12-15 UTC.