Repository files navigation

TensorFlow I/O is a collection of file systems and file formats that are notavailable in TensorFlow's built-in support. A full list of supported file systemsand file formats by TensorFlow I/O can be foundhere.

The use of tensorflow-io is straightforward with keras. Below is an exampletoGet Started with TensorFlow withthe data processing aspect replaced by tensorflow-io:

importtensorflowastfimporttensorflow_ioastfio# Read the MNIST data into the IODataset.dataset_url="https://storage.googleapis.com/cvdf-datasets/mnist/"d_train=tfio.IODataset.from_mnist(dataset_url+"train-images-idx3-ubyte.gz",dataset_url+"train-labels-idx1-ubyte.gz",)# Shuffle the elements of the dataset.d_train=d_train.shuffle(buffer_size=1024)# By default image data is uint8, so convert to float32 using map().d_train=d_train.map(lambdax,y: (tf.image.convert_image_dtype(x,tf.float32),y))# prepare batches the data just like any other tf.data.Datasetd_train=d_train.batch(32)# Build the model.model=tf.keras.models.Sequential(    [tf.keras.layers.Flatten(input_shape=(28,28)),tf.keras.layers.Dense(512,activation=tf.nn.relu),tf.keras.layers.Dropout(0.2),tf.keras.layers.Dense(10,activation=tf.nn.softmax),    ])# Compile the model.model.compile(optimizer="adam",loss="sparse_categorical_crossentropy",metrics=["accuracy"])# Fit the model.model.fit(d_train,epochs=5,steps_per_epoch=200)

In the aboveMNIST example, the URL'sto access the dataset files are passed directly to thetfio.IODataset.from_mnist API call.This is due to the inherent support thattensorflow-io provides forHTTP/HTTPS file system,thus eliminating the need for downloading and saving datasets on a local directory.

NOTE: Sincetensorflow-io is able to detect and uncompress the MNIST dataset automatically if needed,we can pass the URL's for the compressed files (gzip) to the API call as is.

Please check the officialdocumentation for moredetailed and interesting usages of the package.

Thetensorflow-io Python package can be installed with pip directly using:

$ pip install tensorflow-io

People who are a little more adventurous can also try our nightly binaries:

$ pip install tensorflow-io-nightly

To ensure you have a version of TensorFlow that is compatible with TensorFlow-IO,you can specify thetensorflow extra requirement during install:

pip install tensorflow-io[tensorflow]

Similar extras exist for thetensorflow-gpu,tensorflow-cpu andtensorflow-rocmpackages.

In addition to the pip packages, the docker images can be used to quickly get started.

For stable builds:

$ docker pull tfsigio/tfio:latest$ docker run -it --rm --name tfio-latest tfsigio/tfio:latest

For nightly builds:

$ docker pull tfsigio/tfio:nightly$ docker run -it --rm --name tfio-nightly tfsigio/tfio:nightly

Once thetensorflow-io Python package has been successfully installed, youcan install the development version of the R package from GitHub via the following:

if (!require("remotes")) install.packages("remotes")remotes::install_github("tensorflow/io",subdir="R-package")

To ensure compatibility with TensorFlow, it is recommended to install a matchingversion of TensorFlow I/O according to the table below. You can find the listof releaseshere.

We usegithub-pages to document the results of API performance benchmarks. The benchmark job is triggered on every commit tomaster branch andfacilitates tracking performance w.r.t commits.

Tensorflow I/O is a community led open source project. As such, the projectdepends on public contributions, bug-fixes, and documentation. Please see:

• contribution guidelines for a guide on how to contribute.
• development doc for instructions on the development environment setup.
• tutorials for a list of tutorial notebooks and instructions on how to write one.

Build	Status
Linux CPU Python 2
Linux CPU Python 3
Linux GPU Python 2
Linux GPU Python 3

Because of manylinux2010 requirement, TensorFlow I/O is built withUbuntu:16.04 + Developer Toolset 7 (GCC 7.3) on Linux. Configurationwith Ubuntu 16.04 with Developer Toolset 7 is not exactly straightforward.If the system have docker installed, then the following commandwill automatically build manylinux2010 compatible whl package:

#!/usr/bin/env bashls dist/*forfin dist/*.whl;do  docker run -i --rm -v$PWD:/v -w /v --net=host quay.io/pypa/manylinux2010_x86_64 bash -x -e /v/tools/build/auditwheel repair --plat manylinux2010_x86_64$fdonesudo chown -R$(id -nu):$(id -ng).ls wheelhouse/*

It takes some time to build, but once complete, there will be python3.5,3.6,3.7 compatible whl packages available inwheelhousedirectory.

On macOS, the same command could be used. However, the script expectspython in shelland will only generate a whl package that matches the version ofpython in shell. Ifyou want to build a whl package for a specific python then you have to alias this versionof python topython in shell. See.github/workflows/build.ymlAuditwheel step for instructions how to do that.

Note the above command is also the command we use when releasing packages for Linux and macOS.

TensorFlow I/O uses both GitHub Workflows and Google CI (Kokoro) for continuous integration.GitHub Workflows is used for macOS build and test. Kokoro is used for Linux build and test.Again, because of the manylinux2010 requirement, on Linux whl packages are alwaysbuilt with Ubuntu 16.04 + Developer Toolset 7. Tests are done on a variatiy of systemswith different python3 versions to ensure a good coverage:

TensorFlow I/O has integrations with many systems and cloud vendors such asPrometheus, Apache Kafka, Apache Ignite, Google Cloud PubSub, AWS Kinesis,Microsoft Azure Storage, Alibaba Cloud OSS etc.

We tried our best to test against those systems in our continuous integrationwhenever possible. Some tests such as Prometheus, Kafka, and Igniteare done with live systems, meaning we install Prometheus/Kafka/Ignite on CI machine beforethe test is run. Some tests such as Kinesis, PubSub, and Azure Storage are donethrough official or non-official emulators. Offline tests are also performed wheneverpossible, though systems covered through offine tests may not have the samelevel of coverage as live systems or emulators.

References for emulators:

• OfficialPubSub Emulator by Google Cloud for Cloud PubSub.
• OfficialAzurite Emulator by Azure for Azure Storage.
• None-officialLocalStack emulator by LocalStack for AWS Kinesis.

• SIG IOGoogle Group and mailing list:io@tensorflow.org
• SIG IOMonthly Meeting Notes
• Gitter room:tensorflow/sig-io

• Streaming Machine Learning with Tiered Storage and Without a Data Lake -Kai Waehner
• TensorFlow with Apache Arrow Datasets -Bryan Cutler
• How to build a custom Dataset for Tensorflow -Ivelin Ivanov
• TensorFlow on Apache Ignite -Anton Dmitriev

Apache License 2.0