Repository files navigation

TensorFlow Data Validation (TFDV) is a library for exploring and validatingmachine learning data. It is designed to be highly scalableand to work well with TensorFlow andTensorFlow Extended (TFX).

TF Data Validation includes:

• Scalable calculation of summary statistics of training and test data.
• Integration with a viewer for data distributions and statistics, as wellas faceted comparison of pairs of features (Facets)
• Automateddata-schemageneration to describe expectations about datalike required values, ranges, and vocabularies
• A schema viewer to help you inspect the schema.
• Anomaly detection to identify anomalies, such as missing features,out-of-range values, or wrong feature types, to name a few.
• An anomalies viewer so that you can see what features have anomalies andlearn more in order to correct them.

For instructions on using TFDV, see theget started guideand try out theexample notebook.

Caution: TFDV may be backwards incompatible before version 1.0.

The recommended way to install TFDV is using thePyPI package:

pip install tensorflow-data-validation

To compile and use TFDV, you need to set up some prerequisites.

If NumPy is not installed on your system, install it now by followingthesedirections.

If Bazel is not installed on your system, install it now by followingthesedirections.

git clone https://github.com/tensorflow/data-validationcd data-validation

Note that these instructions will install the latest master branch of TensorFlowData Validation. If you want to install a specific branch (such as a release branch),pass-b <branchname> to thegit clone command.

TFDV uses Bazel to build the pip package from source:

bazel run -c opt tensorflow_data_validation:build_pip_package

You can find the generated.whl file in thedist subdirectory.

pip install dist/*.whl

TFDV is built and tested on the following 64-bit operating systems:

• macOS 10.12.6 (Sierra) or later.
• Ubuntu 14.04 or later.

TFDV requires TensorFlow but does not depend on thetensorflowPyPI package. See theTensorFlow install guidesfor instructions on how to get started with TensorFlow.

Apache Beam is required; it's the way that efficientdistributed computation is supported. By default, Apache Beam runs in localmode but can also run in distributed mode usingGoogle Cloud Dataflow.TFDV is designed to be extensible for other Apache Beam runners.

The following table shows the package versions that arecompatible with each other. This is determined by our testing framework, butotheruntested combinations may also work.

Please direct any questions about working with TF Data Validation toStack Overflow using thetensorflow-data-validationtag.