Repository files navigation

TensorFlow Transform is a library for preprocessing data with TensorFlow.tf.Transform is useful for data that requires a full-pass, such as:

• Normalize an input value by mean and standard deviation.
• Convert strings to integers by generating a vocabulary over all input values.
• Convert floats to integers by assigning them to buckets based on the observeddata distribution.

TensorFlow has built-in support for manipulations on a single example or a batchof examples.tf.Transform extends these capabilities to support full-passesover the example data.

The output oftf.Transform is exported as aTensorFlow graph to use for training andserving. Using the same graph for both training and serving can prevent skewsince the same transformations are applied in both stages.

For an introduction totf.Transform, see thetf.Transform section of theTFX Dev Summit talk on TFX(link).

Thetensorflow-transformPyPI package is therecommended way to installtf.Transform:

pip install tensorflow-transform

To build from source follow the following steps:Create a virtual environment by running the commands

python -m venv<virtualenv_name>source<virtualenv_name>/bin/activategit clone https://github.com/tensorflow/transform.gitcd transformpip install.

If you are doing development on the TFT repo, replace

pip install.

with

pip install -e.

The-e flag causes TFT to be installed indevelopment mode.

TFT also hosts nightly packages athttps://pypi-nightly.tensorflow.org onGoogle Cloud. To install the latest nightly package, please use the followingcommand:

pip install --extra-index-url https://pypi-nightly.tensorflow.org/simple tensorflow-transform

This will install the nightly packages for the major dependencies of TFT suchas TensorFlow Metadata (TFMD), TFX Basic Shared Libraries (TFX-BSL).

To run TFT tests, run the following command from the root of the repository:

pytest

TensorFlow is required.

Apache Beam is required; it's the way that efficientdistributed computation is supported. By default, Apache Beam runs in localmode but can also run in distributed mode usingGoogle Cloud Dataflow and other ApacheBeamrunners.

Apache Arrow is also required. TFT uses Arrow torepresent data internally in order to make use of vectorized numpy functions.

The following table is thetf.Transform package versions that arecompatible with each other. This is determined by our testing framework, butotheruntested combinations may also work.

Please direct any questions about working withtf.Transform toStack Overflow using thetensorflow-transformtag.