Repository files navigation

Data quality testing for SQL-, Spark-, and Pandas-accessible data.

✔ An open-source, CLI tool and Python library for data quality testing
✔ Compatible with theSoda Checks Language (SodaCL)
✔ Enables data quality testing both in and out of your data pipelines and development workflows
✔ Integrated to allow a Soda scan in a data pipeline, or programmatic scans on a time-based schedule

Soda Core is a free, open-source, command-line tool and Python library that enables you to use the Soda Checks Language to turn user-defined input into aggregated SQL queries.

When it runs a scan on a dataset, Soda Core executes the checks to find invalid, missing, or unexpected data. When your Soda Checks fail, they surface the data that you defined as bad-quality.

Consider migrating toSoda Library, an extension of Soda Core that offers more features and functionality, and enables you to connect to aSoda Cloud account to collaborate with your team on data quality.

• UseGroup by andGroup Evolution configurations to intelligently group check results
• LeverageReconciliation checks to compare data between data sources for data migration projects.
• UseSchema Evolution checks to automatically validate schemas.
• Set upAnomaly Detection checks to automatically learn patterns and discover anomalies in your data.

Install Soda Library and get started with a 45-day free trial.

Soda Core currently supports connections to several data sources. SeeCompatibility for a complete list.

Requirements

• Python 3.8 or greater
• Pip 21.0 or greater

Install and run

1. To get started, use the install command, replacingsoda-core-postgres with the package that matches your data source. SeeInstall Soda Core for a complete list.
   

   pip install soda-core-postgres

2. Prepare aconfiguration.yml file to connect to your data source. Then, write data quality checks in achecks.yml file. SeeConfigure Soda Core.

3. Run a scan to review checks that passed, failed, or warned during a scan. SeeRun a Soda Core scan.

   soda scan -d your_datasource -c configuration.yml checks.yml

# Checks for basic validationschecks for dim_customer:  -row_count between 10 and 1000  -missing_count(birth_date) = 0  -invalid_percent(phone) < 1 %:valid format:phone number  -invalid_count(number_cars_owned) = 0:valid min:1valid max:6  -duplicate_count(phone) = 0# Checks for schema changeschecks for dim_product:  -schema:name:Find forbidden, missing, or wrong typewarn:when required column missing:[dealer_price, list_price]when forbidden column present:[credit_card]when wrong column type:standard_cost:moneyfail:when forbidden column present:[pii*]when wrong column index:model_name:22# Check for freshness  -freshness(start_date) < 1d# Check for referential integritychecks for dim_department_group:  -values in (department_group_name) must exist in dim_employee (department_name)

• Soda Core
• Soda Checks Language (SodaCL)