Bigtable Beam connector

The Bigtable Beam connector (BigtableIO) is an open sourceApacheBeam I/O connector that can help you perform batch and streamingoperations on Bigtable data in apipeline usingDataflow.

If you are migrating from HBase to Bigtable or you are running anapplication uses the HBase API instead of the BigtableAPIs, use theBigtable HBase Beam connector(CloudBigtableIO) instead of the connector described on this page.

Connector details

The Bigtable Beam connector is a component of theApache Beam GitHubrepository. The Javadoc is availableatClassBigtableIO.

Before you create a Dataflow pipeline, checkApache Beamruntime support to make sure youare using a version of Java that is supported for Dataflow. Usethe most recent supported release of Apache Beam.

The Bigtable Beam connector is used in conjunction with theBigtable client for Java, a client library that calls theBigtable APIs. You write code to deploy a pipeline that uses theconnector to Dataflow, which handles the provisioning andmanagement of resources and assists with the scalability and reliability of dataprocessing.

For more information on the Apache Beam programming model, see theBeamdocumentation.

Batch write flow control

When you send batch writes (including delete requests) to a table using theBigtable Beam connector, you can enablebatch write flow control. Whenthis feature is enabled, Bigtable automatically does thefollowing:

• Rate-limits traffic to avoid overloading your Bigtable cluster
• Ensures the cluster is under enough load to trigger Bigtableautoscaling (if enabled), so that more nodes are automatically added to thecluster when needed

For more information, seeBatch write flowcontrol. For a code sample, seeEnablebatch write flow control.

What's next

• Read an overview of Bigtable write requests.
• Review a list of Dataflow templates that work withBigtable.
• Bigtable Kafka Connect sink connector