Repository files navigation

TensorFlow Data Validation (TFDV) is a library for exploring and validatingmachine learning data. It is designed to be highly scalableand to work well with TensorFlow andTensorFlow Extended (TFX).

TF Data Validation includes:

• Scalable calculation of summary statistics of training and test data.
• Integration with a viewer for data distributions and statistics, as wellas faceted comparison of pairs of features (Facets)
• Automateddata-schemageneration to describe expectations about datalike required values, ranges, and vocabularies
• A schema viewer to help you inspect the schema.
• Anomaly detection to identifyanomalies,such as missing features,out-of-range values, or wrong feature types, to name a few.
• An anomalies viewer so that you can see what features have anomalies andlearn more in order to correct them.

For instructions on using TFDV, see theget started guideand try out theexample notebook.Some of the techniques implemented in TFDV are described in atechnical paper published in SysML'19.

The recommended way to install TFDV is using thePyPI package:

pip install tensorflow-data-validation

TFDV also hosts nightly packages athttps://pypi-nightly.tensorflow.org onGoogle Cloud. To install the latest nightly package, please use the followingcommand:

pip install -i https://pypi-nightly.tensorflow.org/simple tensorflow-data-validation

This will install the nightly packages for the major dependencies of TFDV suchas TFX Basic Shared Libraries (TFX-BSL) and TensorFlow Metadata (TFMD).

This is the recommended way to build TFDV under Linux, and is continuouslytested at Google.

Please first installdocker anddocker-compose by following the directions:docker;docker-compose.

git clone https://github.com/tensorflow/data-validationcd data-validation

Note that these instructions will install the latest master branch of TensorFlowData Validation. If you want to install a specific branch (such as a releasebranch), pass-b <branchname> to thegit clone command.

Then, run the following at the project root:

sudo docker-compose build manylinux2010sudo docker-compose run -e PYTHON_VERSION=${PYTHON_VERSION} manylinux2010

wherePYTHON_VERSION is one of{37, 38}.

A wheel will be produced underdist/.

pip install dist/*.whl

To compile and use TFDV, you need to set up some prerequisites.

If NumPy is not installed on your system, install it now by followingthesedirections.

If Bazel is not installed on your system, install it now by followingthesedirections.

git clone https://github.com/tensorflow/data-validationcd data-validation

Note that these instructions will install the latest master branch of TensorFlowData Validation. If you want to install a specific branch (such as a releasebranch), pass-b <branchname> to thegit clone command.

TFDV wheel is Python version dependent -- to build the pip package thatworks for a specific Python version, use that Python binary to run:

python setup.py bdist_wheel

You can find the generated.whl file in thedist subdirectory.

pip install dist/*.whl

TFDV is tested on the following 64-bit operating systems:

• macOS 10.14.6 (Mojave) or later.
• Ubuntu 16.04 or later.
• Windows 7 or later.

TensorFlow is required.

Apache Beam is required; it's the way that efficientdistributed computation is supported. By default, Apache Beam runs in localmode but can also run in distributed mode usingGoogle Cloud Dataflow and other ApacheBeamrunners.

Apache Arrow is also required. TFDV uses Arrow torepresent data internally in order to make use of vectorized numpy functions.

The following table shows the package versions that arecompatible with each other. This is determined by our testing framework, butotheruntested combinations may also work.

Please direct any questions about working with TF Data Validation toStack Overflow using thetensorflow-data-validationtag.

• TensorFlow Data Validation Getting Started Guide
• TensorFlow Data Validation Notebook
• TensorFlow Data Validation API Documentation
• TensorFlow Data Validation Blog Post
• TensorFlow Data Validation PyPI
• TensorFlow Data Validation Paper
• TensorFlow Data Validation Slides