Console

1. In the Google Cloud console, go to the Dataplex Universal CatalogData profiling & quality page.

   Go to Data profiling & quality

2. ClickCreate data quality scan.

3. In theDefine scan window, fill in the following fields:

   1. Optional: Enter aDisplay name.

   2. Enter anID. See theresource naming conventions.

   3. Optional: Enter aDescription.

   4. In theTable field, clickBrowse. Choose the table to scan, andthen clickSelect.Only standard BigQuery tables aresupported.

      For tables in multi-region datasets, choose a region where to createthe data scan.

      To browse the tables organized within Dataplex Universal Catalog lakes,clickBrowse within Dataplex Lakes.

   5. In theScope field, chooseIncremental orEntire data.

      • If you chooseIncremental: In theTimestamp column field,select a column of typeDATE orTIMESTAMP from yourBigQuery table that increases as new records are added,and that can be used to identify new records. It can be a column thatpartitions the table.

   6. To filter your data, select theFilter rows checkbox. Provide arow filter consisting of a valid SQL expression that can be used as a part of aWHERE clause in GoogleSQL syntax.For example,col1 >= 0.The filter can be a combination of multiple column conditions. Forexample,col1 >= 0 AND col2 < 10.

   7. To sample your data, in theSampling size list, select asampling percentage. Choose a percentage value that ranges between0.0% and 100.0% with up to 3 decimal digits. For largerdatasets, choose a lower sampling percentage. For example, for a1 PB table, if you enter a value between 0.1% and 1.0%,the data quality scan samples between 1-10 TB of data. Forincremental data scans, the data quality scan applies sampling to thelatest increment.

   8. To publish the data quality scan results as Dataplex Universal Catalogmetadata, select thePublish results to BigQuery and Dataplex Catalog checkbox.

      You can view the latest scan results on theData quality tab in theBigQuery and Dataplex Universal Catalog pages for the sourcetable. To enable users to access the published scan results, see theGrant access to data profile scan results sectionof this document.

   9. In theSchedule section, choose one of the following options:

      • Repeat: Run the data quality scan on a schedule: hourly, daily,weekly, monthly, or custom. Specify how often the scan runs andat what time. If you choose custom, usecronformat to specify the schedule.

      • On-demand: Run the data quality scan on demand.

   10. ClickContinue.

4. In theData quality rules window, define the rules toconfigure for this data quality scan.

   1. ClickAdd rules, and then choose from the following options.

      • Profile based recommendations: Build rules from therecommendations based on an existing data profiling scan.

        1. Choose columns: Select the columns to get recommended rules for.

        2. Choose scan project: If the data profiling scan is in adifferent project than the project where you are creatingthe data quality scan, then select the project to pull profilescans from.

        3. Choose profile results: Select one or more profile results andthen clickOK. This populates a list of suggested rules thatyou can use as a starting point.

        4. Select the checkbox for the rules that you want to add, and thenclickSelect. Once selected, the rules are added to yourcurrent rule list. Then, you can edit the rules.

      • Built-in rule types: Build rules from predefined rules.See the list ofpredefined rules.

        1. Choose columns: Select the columns to select rules for.

        2. Choose rule types: Select the rule types that you want tochoose from, and then clickOK. The rule types that appeardepend on the columns that you selected.

        3. Select the checkbox for the rules that you want to add, and thenclickSelect. Once selected, the rules are added to yourcurrent rules list. Then, you can edit the rules.

      • SQL row check rule: Create a custom SQL rule to apply to each row.

        1. InDimension, choose one dimension.

        2. InPassing threshold, choose a percentage of records that mustpass the check.

        3. InColumn name, choose a column.

        4. In theProvide a SQL expression field, enter a SQL expressionthat evaluates to a booleantrue (pass) orfalse (fail). Formore information, seeSupported custom SQL rule typesand the examples inDefine data quality rules.

        5. ClickAdd.

      • SQL aggregate check rule: Create a custom SQLtable condition rule.

        1. InDimension, choose one dimension.

        2. InColumn name, choose a column.

        3. In theProvide a SQL expression field, enter a SQL expressionthat evaluates to a booleantrue (pass) orfalse (fail). Formore information, seeSupported custom SQL rule typesand the examples inDefine data quality rules.

        4. ClickAdd.

      • SQL assertion rule: Create a custom SQL assertion rule to checkfor an invalid state of the data.

        1. InDimension, choose one dimension.

        2. Optional: InColumn name, choose a column.

        3. In theProvide a SQL statement field, enter a SQL statementthat returns rows that match the invalid state. If any rows arereturned, this rule fails. Omit the trailing semicolon from the SQLstatement. For more information, seeSupported custom SQL rule typesand the examples inDefine data quality rules.

        4. ClickAdd.

   2. Optional: For any data quality rule, you can assign a custom rule nameto use for monitoring and alerting, and a description. To do this,edit a rule and specify the following details:

      • Rule name: Enter a custom rule name with up to 63 characters.The rule name can include letters (a-z, A-Z), digits (0-9), andhyphens (-) and must start with a letter and end with a numberor a letter.
      • Description: Enter a rule description with a maximumlength of 1,024 characters.

   3. Repeat the previous steps to add additional rules to the data qualityscan. When finished, clickContinue.

5. Optional: Export the scan results to a BigQuery standardtable. In theExport scan results to BigQuery table section, do thefollowing:

   1. In theSelect BigQuery dataset field, clickBrowse. Select aBigQuery dataset to store the data quality scan results.

   2. In theBigQuery table field, specify the table to store the dataquality scan results. If you're using an existing table, make surethat it is compatible with theexport table schema.If the specified table doesn't exist, Dataplex Universal Catalog createsit for you.

      Note: You can use the same results table for multiple data qualityscans.

6. Optional: Add labels. Labels are key-value pairs that let you grouprelated objects together or with other Google Cloud resources.

7. Optional: Set up email notification reports to alert people about thestatus and results of a data quality scan job. In theNotification reportsection, clickaddAdd email ID andenter up to five email addresses. Then, select the scenarios that you wantto send reports for:

   • Quality score (<=): sends a report when a job succeeds with a dataquality score that is lower than the specified target score. Enter atarget quality score between 0 and 100.
   • Job failures: sends a report when the job itself fails, regardlessof the data quality results.
   • Job completion (success or failure): sends a report when the jobends, regardless of the data quality results.

8. ClickCreate.

   After the scan is created, you can run it at any time by clickingRun now.

gcloud

To create a data quality scan, use thegcloud dataplex datascans create data-quality command.

If the source data is organized in a Dataplex Universal Catalog lake, include the--data-source-entity flag:

gclouddataplexdatascanscreatedata-qualityDATASCAN\--location=LOCATION\--data-quality-spec-file=DATA_QUALITY_SPEC_FILE\--data-source-entity=DATA_SOURCE_ENTITY

If the source data isn't organized in a Dataplex Universal Catalog lake, includethe--data-source-resource flag:

gclouddataplexdatascanscreatedata-qualityDATASCAN\--location=LOCATION\--data-quality-spec-file=DATA_QUALITY_SPEC_FILE\--data-source-resource=DATA_SOURCE_RESOURCE

Replace the following variables:

• DATASCAN: The name of the data quality scan.
• LOCATION: The Google Cloud region in which tocreate the data quality scan.
• DATA_QUALITY_SPEC_FILE: The path to the JSON orYAML file containing the specifications for the data quality scan. The filecan be a local file or a Cloud Storage path with the prefixgs://.Use this file to specify the data quality rules for the scan. You can alsospecify additional details in this file, such as filters, sampling percent,and post-scan actions like exporting to BigQuery or sendingemail notification reports. See thedocumentation for JSON representationand theexample YAML representation.
• DATA_SOURCE_ENTITY: The Dataplex Universal Catalogentity that contains the data for the data quality scan. For example,projects/test-project/locations/test-location/lakes/test-lake/zones/test-zone/entities/test-entity.
• DATA_SOURCE_RESOURCE: The name of the resourcethat contains the data for the data quality scan. For example,//bigquery.googleapis.com/projects/test-project/datasets/test-dataset/tables/test-table.

REST

To create a data quality scan, use thedataScans.create method.

If you want to build rules for the data quality scan by using rulerecommendations that are based on the results of a data profiling scan, getthe recommendations by calling thedataScans.jobs.generateDataQualityRules methodon the data profiling scan.

Console

1. In the Google Cloud console, go to the Dataplex Universal CatalogData profiling & quality page.

   Go to Data profiling & quality

2. Click the data quality scan to run.

3. ClickRun now.

gcloud

To run a data quality scan, use thegcloud dataplex datascans run command:

gcloud dataplex datascans runDATASCAN \--location=LOCATION \

Replace the following variables:

• LOCATION: The Google Cloud region in which thedata quality scan was created.
• DATASCAN: The name of the data quality scan.

REST

To run a data quality scan, use thedataScans.run method.

Console

1. In the Google Cloud console, go to the Dataplex Universal CatalogData profiling & quality page.

   Go to Data profiling & quality

2. Click the name of a data quality scan.

   • TheOverview section displays information about the most recentjobs, including when the scan was run, the number of recordsscanned in each job, whether all the data quality checks passed, andif there were failures, the number of data quality checks that failed.

   • TheData quality scan configuration section displays details about thescan.

3. To see detailed information about a job, such as data quality scores thatindicate the percentage of rules that passed, which rules failed, and thejob logs, click theJobs history tab. Then, click a job ID.

gcloud

To view the results of a data quality scan job, use thegcloud dataplex datascans jobs describe command:

gcloud dataplex datascans jobs describeJOB \--location=LOCATION \--datascan=DATASCAN \--view=FULL

Replace the following variables:

• JOB: The job ID of the data quality scan job.
• LOCATION: The Google Cloud region in which the dataquality scan was created.
• DATASCAN: The name of the data quality scan the jobbelongs to.
• --view=FULL: To see the scan job result, specifyFULL.

REST

To view the results of a data quality scan, use thedataScans.get method.

Console

1. In the Google Cloud console, go to the Dataplex Universal CatalogData profiling & quality page.

   Go to Data profiling & quality

2. Click the name of a data quality scan.

3. Click theJobs history tab.

   TheJobs history tab provides information about past jobs, such asthe number of records scanned in each job, the job status, the timethe job was run, and whether each rule passed or failed.

4. To view detailed information about a job, click any of the jobs in theJob ID column.

gcloud

To view historical data quality scan jobs, use thegcloud dataplex datascans jobs list command:

gcloud dataplex datascans jobs list \--location=LOCATION \--datascan=DATASCAN \

Replace the following variables:

• LOCATION: The Google Cloud region in which the dataquality scan was created.
• DATASCAN: The name of the data quality scan to viewhistorical jobs for.

REST

To view historical data quality scan jobs, use thedataScans.jobs.list method.

Console

1. In the Google Cloud console, go to the Cloud LoggingLogs Explorer.

   Go to Logs explorer

2. In theQuery window, enter your query. Seesample queries.

3. ClickRun Query.

4. ClickCreate alert. This opens a side panel.

5. Enter your alert policy name and clickNext.

6. Review the query.

   1. Click thePreview Logs button to test your query. This shows logswith matching conditions.

   2. ClickNext.

7. Set the time between notifications and clickNext.

8. Define who should be notified for the alert and clickSave to createthe alert policy.

Alternatively, you can configure and edit your alerts by navigating in theGoogle Cloud console toMonitoring>Alerting.

gcloud

Not supported.

REST

For more information about how to set alerts in Cloud Logging, seeCreate a log-based alerting policy by using the Monitoring API.

1. In the Google Cloud console, go to the Dataplex Universal CatalogData profiling & quality page.

   Go to Data profiling & quality

2. Click the name of the data quality scan whose records you want to troubleshoot.

3. Click theJobs history tab.

4. Click the job ID of the job that identified data quality failures.

5. In the job results window that opens, in theRules section, find the columnQuery to get failed records. ClickCopy query to clipboard for thefailed rule.

6. Run the query in BigQueryto see the records that caused the job to fail.

gcloud

Not supported.

REST

1. To get the job that identified data quality failures, use thedataScans.get method.

   In the response object, thefailingRowsQuery field shows the query.

2. Run the query in BigQueryto see the records that caused the job to fail.

1. In the Google Cloud console, go to the Dataplex Universal CatalogData profiling & quality page.

   Go to Data profiling & quality

2. Click the name of a data quality scan.

3. To edit settings including the display name, filters, and schedule, clickEdit. Edit the values and then clickSave.

4. To edit the data quality rules, on the scan details page, click theCurrent rules tab. ClickModify rules. Update the rules andthen clickSave.

gcloud

To update the description of a data quality scan, use thegcloud dataplex datascans update data-quality command:

gcloud dataplex datascans update data-qualityDATASCAN \--location=LOCATION \--description=DESCRIPTION

Replace the following:

• DATASCAN: The name of the data quality scan toupdate.
• LOCATION: The Google Cloud region in which the dataquality scan was created.
• DESCRIPTION: The new description for the dataquality scan.

REST

To edit a data quality scan, use thedataScans.patch method.

Console

1. In the Google Cloud console, go to the Dataplex Universal CatalogData profiling & quality page.

   Go to Data profiling & quality

2. Click the scan you want to delete.

3. ClickDelete, and then confirm when prompted.

gcloud

To delete a data quality scan, use thegcloud dataplex datascans delete command:

gcloud dataplex datascans deleteDATASCAN \--location=LOCATION \--async

Replace the following variables:

• DATASCAN: The name of the data quality scan todelete.
• LOCATION: The Google Cloud region in which the dataquality scan was created.

REST

To delete a data quality scan, use thedataScans.delete method.