This page describes the sensitive data discovery service. This servicecontinuously monitors your data, classifies it, and shows you where sensitiveand high-risk data reside. Discovering and classifying sensitive data is animportant element of a strongcloud data security and riskmanagement strategy.

The Sensitive Data Protection discovery service (sometimes calleddataprofiler) continuously monitors the data resources in your organization,folder, or project. It classifies the data intoinfoTypes, and assesses the datasensitivity and risk levels. This service generates data profiles, which provideinsights and metrics about your data. You can send data profiles to otherGoogle Cloud services, like Security Command Center and Dataplex Universal Catalog, to takeadvantage of the insights that the profiles provide.

1. You enable sensitive data discovery by creating adiscovery scanconfiguration (alsocalled adata profile configuration) that is scoped to an organization,folder, or project. In the scan configuration, you can set filters tospecify subsets of data that you want to profile or skip. You can also setthe profiling schedule.

   In the scan configuration, you also set theinspectiontemplate to use. The inspection template is where youspecify the types of sensitive data (also calledinfoTypes) thatSensitive Data Protection must scan for.

   You can also enableactions that you wantSensitive Data Protection to take after each scan. For example, you canconfigure an action that sends a Pub/Sub notification whenever thereis a change in the sensitivity level of a data profile.

2. Within the scope of the configuration, Sensitive Data Protection scansall supported data resources for sensitive information and context that arespecified as infoTypes in your configuration. Sensitive Data Protectionanalyzes your data based on your scan configuration and inspection template.

3. Sensitive Data Protection generates data profiles, which provide metricsand insights about your data. In Sensitive Data Protection, this processis calleddata profiling. The types of profiles generated depend onthe type of data that Sensitive Data Protection scanned.

   • If you profile data stored in BigQuery or Cloud SQL,Sensitive Data Protection generates the following:

     • Onetable dataprofile foreach table.
     • Onecolumn dataprofile foreach column in the table.
     • Oneproject dataprofile foreach project scanned. This profile aggregates insights and metricsfrom all data profiles in the project.

   • If you profile data stored in Cloud Storage or a storage servicein other clouds, Sensitive Data Protection generates the following:

     • Onefile store dataprofilefor each bucket.
     • One project data profile for each project scanned. This profileaggregates insights and metrics from all data profiles in the project.

4. Sensitive Data Protection performs any actions that you enabled in yourdiscovery scan configuration.

5. As long as the discovery scan configuration is active,Sensitive Data Protection automatically profiles data that you add ormodify.

   Sensitive Data Protection reprofiles data as described inFrequency of data profilegeneration. You can customize the profiling frequency in your scan configuration bycreating a schedule.To force the discovery service to reprofile your data, seeForce a reprofileoperation.

Data profiles

Eachdata profile is a set of insights and metadata that thediscovery service gathers from scanning asupportedresource. Insights include the predictedinfoTypes and thecalculated data risk and sensitivitylevels of yourdata. Use these insights to make informed decisions about how you protect,share, and use your data.

Data profiles are generated at various levels of detail. For example, when youprofile BigQuery data, profiles are generated at the project,table, and column levels.

The following image shows a list of column-level data profiles. Click the imageto enlarge it.

For a list of insights and metadata included in each data profile, seeMetrics reference.

For more information about the Google Cloud resource hierarchy, seeResource hierarchy.

Types of sensitive data discovery

This section describes the types of discovery operations that you can performand the supported data resources.

Discovery for BigQuery and BigLake

When you profile BigQuery data,data profiles are generated at the project, table, and column levels. Afterprofiling a BigQuery table, you can further investigate thefindings byperforming a deepinspection.

Sensitive Data Protection profiles tables that are supported by theBigQuery Storage Read API, including thefollowing:

• Standard BigQuery tables
• Table snapshots
• BigLake tables stored in Cloud Storage

The following aren't supported:

• BigQuery Omni tables.
• Tables where the serialized data size of individual rows exceed themaximum serialized data sizethat the BigQuery Storage Read API supports—128 MB.
• Non-BigLake external tables, like Google Sheets.

For information about how to profile BigQuery data, see thefollowing:

• Profile BigQuery data in a singleproject
• Profile BigQuery data in an organization orfolder

For more information about BigQuery, see theBigQuery documentation.

Discovery for Cloud SQL

When you profile Cloud SQL data, data profiles are generated atthe project, table, and column levels. Before discovery can begin,you need to provide the connection details for each Cloud SQL instanceto be profiled.

For information about how to profile Cloud SQL data, see thefollowing:

• Profile Cloud SQL data in a singleproject
• Profile Cloud SQL data in an organization orfolder

For more information about Cloud SQL, see theCloud SQLdocumentation.

Discovery for Cloud Storage

When you profile Cloud Storage data, data profiles are generated at thebucket level. Sensitive Data Protection groups the detected files intofileclustersand provides a summary for each cluster.

For information about how to profile Cloud Storage data, see thefollowing:

• Profile Cloud Storage data in a singleproject
• Profile Cloud Storage data in an organization orfolder

For more information about Cloud Storage, see theCloud Storage documentation.

Discovery for Vertex AI

When you profile a Vertex AI dataset,Sensitive Data Protection generates a file store data profile or a tabledata profile, depending on where your training data is stored:Cloud Storage or BigQuery.

For more information, see the following:

• Sensitive data discovery forVertex AI
• Profile Vertex AI data in a singleproject
• Profile Vertex AI data in an organization orfolder

For more information about Vertex AI, see theVertex AI documentation.

Discovery for other cloud providers

When you profile S3 data, data profiles are generated atthe bucket level. When you profile Azure Blob Storage data, dataprofiles are generated at the container level.

In both cases, Sensitive Data Protection groups the detected files intofileclustersand provides a summary for each cluster.

For more information, see the following:

• Sensitive data discovery for Amazon S3data
• Profile Azure Blob Storagedata

Cloud Run environment variables

The discovery service can detect the presence ofsecrets inCloud Run functions and Cloud Run service revision environment variables,and send any findings to Security Command Center. No data profiles are generated.

For more information, seeReport secrets in environment variables toSecurity Command Center.

Roles required to configure and view data profiles

The following sections list the required user roles, categorized according totheir purpose. Depending on how your organization is set up, you might decide tohave different people perform different tasks. For example, the person whoconfigures data profiles might be different from the person who regularly monitorsthem.

Roles required to work with data profiles at the organization or folder level

These roles let you configure and view data profiles at the organization orfolder level.

Make sure these roles are granted to the proper people at the organizationlevel. Alternatively, your Google Cloud administrator cancreate custom roles that only have therelevant permissions.

¹ If you don't have the ProjectCreator (roles/resourcemanager.projectCreator) role, you can still create a scanconfiguration, but theservice agentcontainer that you use must be an existing project.

² If you don't have the OrganizationAdministrator (roles/resourcemanager.organizationAdmin) or Security Admin(roles/iam.securityAdmin) role, you can still create a scan configuration. After youcreate the scan configuration, someone in your organization who has one of these roles mustgrant discovery access to theservice agent.

Roles required to work with data profiles at the project level

These roles let you configure and view data profiles at the project level.

Make sure these roles are granted to the proper people at the projectlevel. Alternatively, your Google Cloud administrator cancreate custom roles that only have therelevant permissions.

Sensitive data discovery scan configuration

Adiscovery scan configuration (sometimes calleddiscovery configuration orscan configuration) specifies how Sensitive Data Protection should profileyour data. It includes the following settings:

• Scope(organization, folder, or project) of the discovery operation
• Type ofresourceto profile
• Inspectiontemplatesto use
• Scanfrequency
• Specific subsets of data that should be included in or excluded from discovery
• Actionsthat you want Sensitive Data Protection to take after discovery—forexample, which Google Cloud services to publish the profiles to
• Serviceagentto use for discovery operations

For information about how to create an organization-level or project-leveldiscovery scan configuration, see the following pages:

¹ Not suitable for customers who have an organization-level discoverysubscription, such as one provided through Security Command Center

Scan configuration scopes

You can create a scan configuration at the following levels:

• Organization
• Folder
• Project
• Single data resource

At the organization and folder levels, if two or more active scan configurationshave the same project in their scope, Sensitive Data Protection determines whichscan configuration can generate profiles for that project. For more information,seeOverriding scanconfigurations on this page.

A project-level scan configuration can always profile the target project anddoes not compete with other configurations at the level of the parent folder ororganization.

Asingle-resource scanconfiguration is intendedto help you explore and test profiling on a single data resource.

Scan configuration location

The first time you create a scan configuration, you specify where you wantSensitive Data Protection to store it. All subsequent scanconfigurations that you create are stored in that same region.

For example, if you create a scan configuration for Folder A and store it in theus-west1 region, then any scan configuration that you later create for anyother resource is also stored in that region.

Metadata about the data to be profiled is copied to the same regionas your scan configurations, but the data itself isn'tmoved or copied. For more information, seeData residencyconsiderations.

Inspection template

Aninspection template specifies what informationtypes (orinfoTypes) Sensitive Data Protection looks for while scanningyour data. Here, you provide a combination ofbuilt-in infoTypesand optionalcustom infoTypes.

You can also provide alikelihood level tonarrow down what Sensitive Data Protection considers to be a match. You canadd rule sets to exclude unwanted findings or include additional findings.

By default, if you change an inspection template that your scan configurationuses, the changes are applied only to future scans. Your action doesn'tcause a reprofile operation on your data.

If you want inspection template changes to trigger reprofile operations on theaffected data, add or update a schedule in your scan configuration, and turn onthe option to reprofile the data when the inspection template changes. For moreinformation, seeFrequency of data profile generation.

Note:We regularly improve our detection algorithm. If we find that your organizationor project would benefit from a new improvement that we implement, we mightautomatically regenerate your data profiles and redo theactions in your scanconfiguration. You won't incur Sensitive Data Protection charges for thisoperation. However, because we will redo the actions, you might incur chargesfor your use of other Google Cloud services. For example, if you configuredSensitive Data Protection to save the data profiles to BigQuery, youmight incur BigQuery charges.

You must have an inspection template in each region where you have data to beprofiled. If you want to use a single template for multiple regions, you can usea template that is stored in theglobal region. If organizationalpolicies prevent you from creating an inspection template in theglobal region, thenyou must set a dedicated inspection template for each region. For moreinformation, seeData residency considerations.

Inspection templates are a core component of the Sensitive Data Protection platform.Data profiles use the same inspection templates that you can use across allSensitive Data Protection services. For more information on inspection templates,seeTemplates.

Service agent container and service agent

When you create a scan configuration for your organization or for a folder,Sensitive Data Protectionrequires you to provide a service agent container. Aservice agent containeris a Google Cloud project that Sensitive Data Protection uses to trackbilled charges related to organization- and folder-level profiling operations.

The service agent container contains aservice agent, whichSensitive Data Protection uses to profile data on your behalf. You need a serviceagent to authenticate to Sensitive Data Protection and other APIs. Your serviceagent must have all the required permissions to access and profile your data.The service agent's ID is in the following format:

service-PROJECT_NUMBER@dlp-api.iam.gserviceaccount.com

Here, thePROJECT_NUMBER is the numerical identifier of the serviceagent container.

When setting the service agent container, you can choose an existing project.If the project you select contains a service agent, Sensitive Data Protectiongrants the required IAM permissionsto that service agent. If the project doesn't have a service agent,Sensitive Data Protection creates one and automatically grants data profilingpermissions to it.

Alternatively, you can choose to have Sensitive Data Protectionautomatically create the service agent container and service agent.Sensitive Data Protection automatically grants data profiling permissions to theservice agent.

In both cases, if Sensitive Data Protection fails to grant data profilingaccess to your service agent, it shows anerrorwhen youview the scan configuration details.

For project-level scan configurations, you don't need a service agentcontainer. The project you're profiling serves the service agentcontainer's purpose. To run profiling operations, Sensitive Data Protection usesthat project's own service agent.

Data profiling access at the organization or folder level

When youconfigure sensitive datadiscovery at theorganization or folder level, Sensitive Data Protection attempts to automaticallygrant data profiling access to your service agent. However, if you don't havethepermissions to grant discovery access,Sensitive Data Protection can't do this action on your behalf. Someone with thosepermissions in your organization, such as a Google Cloud administrator, mustgrant data profiling access to your serviceagent.

Frequency of data profile generation

After you create a discovery scan configuration for a particular resource,Sensitive Data Protection performs an initial scan, profiling the data inthe scope of your scan configuration.

After the initial scan, Sensitive Data Protection continuously monitors theprofiled resource. Data added in the resource isautomatically profiled shortly after it is added.

Default reprofiling frequency

The default reprofiling frequency differs depending on thediscoverytype of your scan configuration:

• BigQuery profiling: for each table, wait 30 days and thenreprofile the table if it has changes in the schema, table rows, or inspectiontemplate.
• Cloud SQL profiling: for each table, wait 30 days and then reprofilethe table if it has changes in the schema or inspectiontemplate.
• Vertex AI profiling: for each dataset, wait 30 days and thenreprofile the dataset if the inspection template has changes.

• File store profiling: for each file store in Google Cloud or in other clouds,wait 30 days and then reprofile the file store if the inspection template haschanges.

  Sensitive Data Protection uses the termfile store torefer to a file storage bucket or container.

Customizing the reprofiling frequency

In your scan configuration, you can customize the reprofiling frequency bycreating one or more schedules for different subsets of your data.

The following reprofiling frequencies are available:

• Do not reprofile: Never reprofile after the initial profiles aregenerated.
• Reprofile daily: Wait 24 hours before reprofiling.
• Reprofile weekly: Wait 7 days before reprofiling.
• Reprofile monthly: Wait 30 days before reprofiling.

Reprofiling on a schedule

In your scan configuration, you can specify whether a subset of data should bereprofiled regularly regardless of whether the data underwent changes.The frequency you set specifies how much time must pass between profilingoperations. For example, if you set the frequency to weekly,Sensitive Data Protection profiles a data resource seven days after it waslast profiled.

Reprofiling on update

In your scan configuration, you can specify events that can triggerreprofiling operations. Examples of such events are inspection template updates.

When you select these events, the schedule you set specifies the longest timeSensitive Data Protection waits for updates to accumulate before itreprofiles your data. If no applicable changes—like schema changes orinspection template changes—occur within your specified period, no data isreprofiled. When the next applicable change occurs, the affected data isreprofiled at the next opportunity, which is determined by various factors (suchas the available machine capacity or thesubscription unitspurchased).Sensitive Data Protection then starts waiting for updates to accumulate againaccording to your set schedule.

For example, suppose your scan configuration is set to reprofile monthly onschema change. The data profiles were first created on day 0. No schema changesoccur by day 30, so no data is reprofiled. On day 35, the first schema changeoccurs. Sensitive Data Protection reprofiles the updated data at the nextopportunity. The system then waits another 30 days for schema updates toaccumulate before it reprofiles any updated data.

From the time reprofiling begins, it can take up to 24 hours for the operationto complete. If the delay lasts longer than 24 hours and you're in subscriptionpricing mode, confirm whether you haveremaining capacity for themonth.

For example scenarios, seeData profiling pricing examples.

To force the discovery service to reprofile your data, seeForce a reprofileoperation.

Note:We regularly improve our detection algorithm. If we find that your organizationor project would benefit from a new improvement that we implement, we mightautomatically regenerate your data profiles and redo theactions in your scanconfiguration. You won't incur Sensitive Data Protection charges for thisoperation. However, because we will redo the actions, you might incur chargesfor your use of other Google Cloud services. For example, if you configuredSensitive Data Protection to save the data profiles to BigQuery, youmight incur BigQuery charges.

Discovery performance

The time it takes to profile your data varies depending on several factors,including, but not limited to, the following:

• Number of data resources being profiled
• Sizes of the data resources
• For tables, the number of columns
• For tables, the data types in the columns

Therefore,Sensitive Data Protection's performance in a past inspection or profiling taskisn't indicative of how it will perform in future profiling tasks.

Retention of data profiles

Sensitive Data Protection retains the latest version of a data profileindefinitely. When Sensitive Data Protection reprofiles a data resource, the systemreplaces that data resource's existing profiles with new ones.

Deleting a data profileforces the system toreprofilethe source data, unless you exclude it from your discovery configuration.

For information on how Sensitive Data Protection charges for profiling data, seeDiscovery pricing.

If you want to explore data profiles, join them with other data sources, or keep arecord of the changes they undergo, consider saving the data profiles toBigQuery when youconfigure sensitive datadiscovery. You choosewhichBigQuery dataset to save the profiles to, and you control thetable expiration policy for that dataset.

Override scan configurations

You can create only one scan configuration for each combination ofscope anddiscovery type. For example, you cancreate only one organization-level scan configuration forBigQuery data profiling and one organization-level scanconfiguration for secrets discovery. Similarly, you can create only oneproject-level scan configuration for BigQuery data profiling andone project-level scan configuration for secrets discovery.

If two or more active scan configurations have the same project and discoverytype in their scope, the following rules apply:

• Among organization-level and folder-level scan configurations, the one that isclosest to the project will be able to run discovery for that project.This rule applies even if a project-level scan configuration with the samediscovery type also exists.
• Sensitive Data Protection treats project-level scan configurations independentlyof organization-level and folder-level configurations. A scan configuration thatyou create at the project level can't override one that you create for aparent folder or organization.

Consider the following example, where there are three active scan configurations.Assume that all of these scan configurations are for BigQuerydata profiling.

Here,Scan configuration 1 applies to the entire organization,Scan configuration 2 applies to theTeam B folder, andScanconfiguration 3 applies to theProduction project. In this example:

• Sensitive Data Protection profiles all tables in projects that aren't intheTeam B folder according toScan configuration 1.
• Sensitive Data Protection profiles all tables in projects in theTeam Bfolder—including tables in theProduction project—accordingtoScan configuration 2.
• Sensitive Data Protection profiles all tables in theProduction projectaccording toScan configuration 3.

In this example, Sensitive Data Protection generates two sets ofprofiles for theProduction project—one set for each of the following scanconfigurations:

• Scan configuration 2
• Scan configuration 3

However, even though there are two sets of profiles for the same project, youdon't see them all together in your dashboard. You only see the profiles thatwere generated in the resource—organization, folder, or project—andregion that you're viewing.

For more information on Google Cloud's resource hierarchy, seeResource hierarchy.

Data profile snapshots

Each data profile includes a snapshot of the scan configuration and theinspection template that were used to generate it. You can use this snapshot tocheck the settings that you used to generate a particular data profile.

Data residency considerations

Data residency considerations differ depending on whether you are scanningGoogle Cloud data or data fromother cloudproviders.

Data residency considerations for Google Cloud data

This section applies only to sensitive data discovery for Google Cloudresources. For data residency considerations related to resources from othercloud providers, seeData residencyconsiderations for data from other cloud providers on this page.

Sensitive Data Protection is designed to supportdata residency. If you must complywith data residency requirements, consider the following points:

Regional inspection templates

This section applies only to sensitive data discovery for Google Cloudresources. For data residency considerations related to resources from othercloud providers, seeData residencyconsiderations for data from other cloud providers on this page.

Sensitive Data Protection processes your datain the same region where that data is stored. That is, your data doesn't leaveits current region.

Furthermore, an inspection template can only be used to profile data thatresides in the same region as that template. For example, if you configurediscovery to use an inspection template that is stored in theus-west1region, Sensitive Data Protection can only profile data in that region.

You can set a dedicated inspection template for each region where you have data.If you provide an inspection template that's stored in theglobal region,Sensitive Data Protection uses that template for data in regions with nodedicated inspection template.

This section applies only to sensitive data discovery for Google Cloudresources. For data residency considerations related to resources from othercloud providers, seeData residencyconsiderations for data from other cloud providers on this page.

When Sensitive Data Protection creates data profiles, it takes a snapshot of yourscan configuration and inspection template and stores them in eachtable dataprofileorfile store dataprofile.If you configure discovery to use an inspection template from theglobalregion, then Sensitive Data Protection copies that template to any region that hasdata to be profiled. Similarly, it copies the scan configuration to thoseregions.

Consider this example: Project A contains Table 1. Table 1 isin theus-west1 region; the scan configuration is in theus-west2 region;and the inspection template is in theglobal region.

When Sensitive Data Protection scans Project A, it creates data profiles forTable 1 and stores them in theus-west1 region. Table 1's table data profilecontains copies of the scan configuration and the inspection template used inthe profiling operation.

If you don't want your inspection template to be copied to other regions,don't configure Sensitive Data Protection to scan data in those regions.

Regional storage of data profiles

This section applies only to sensitive data discovery for Google Cloudresources. For data residency considerations related to resources from othercloud providers, seeData residencyconsiderations for data from other cloud providers on this page.

Sensitive Data Protection processes your data in the regionor multi-region where they reside and stores the generated data profiles in thesame region or multi-region.

Toview dataprofilesin the Google Cloud console, you must first select the region where they reside. Ifyou have data in multiple regions, then you must switch regions to view each setof profiles.

Unsupported regions

This section applies only to sensitive data discovery for Google Cloudresources. For data residency considerations related to resources from othercloud providers, seeData residencyconsiderations for data from other cloud providers on this page.

If you have data in aregion thatSensitive Data Protection doesn't support, then the discovery service skips thosedata resources and shows an error when youview the dataprofiles.

Multi-regions

Sensitive Data Protection treats amulti-region as oneregion, and not a collection of regions. For example, theus multi-regionand theus-west1 region are treated as two separate regions as far as dataresidency is concerned.

Zonal resources

Sensitive Data Protection is a regional and multi-regional service; itdoesn't distinguish between zones. For a supported zonal resource like aCloud SQL instance, the data is processed in its current region, butnot necessarily its current zone. For example, if a Cloud SQL instanceis stored in theus-central1-a zone, Sensitive Data Protectionprocesses and stores the data profiles in theus-central1 region.

For general information about Google Cloud locations, seeGeography andregions.

Data residency considerations for data from other cloud providers

Consider the following when you plan to profile data fromother cloud providers:

• The data profiles are stored alongside the discovery scan configuration. In contrast, when you profile Google Cloud data, the profiles are stored in the same region as the data to be profiled.
• If you store your inspection template in theglobal region, an in-memory copy of that template is read in the region where you store the discovery scan configuration.
• Your data is not modified. An in-memory copy of your data is read in the region where you store the discovery scan configuration. However, Sensitive Data Protection makes no guarantees about where the data passes through after it reaches the public internet. The data is encrypted with SSL.

Compliance

For information on how Sensitive Data Protection handles your data and helps youmeet compliance requirements, seeData security.

What's next

• Read the Identity & Security blog postAutomatic data risk management forBigQuery usingSensitive Data Protection.

• Learn how toestimate data profiling cost.

• Learn how toconfigure sensitive datadiscovery.

• Learn about how Sensitive Data Protectioncalculates the data risk andsensitivity levels when profiling your data.

• Learn how toremediate discovery findings.

• Learn how totroubleshoot issues with the data profiler.