Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

A fully asynchronous, futures-based Kafka client library for Rust based on librdkafka

License

NotificationsYou must be signed in to change notification settings

fede1024/rust-rdkafka

Repository files navigation

crates.iodocs.rsBuild StatuscoverateJoin the chat at https://gitter.im/rust-rdkafka/Lobby

A fully asynchronous,futures-enabledApache Kafka clientlibrary for Rust based onlibrdkafka.

The library

rust-rdkafka provides a safe Rust interface to librdkafka. This versionis compatible with librdkafka v1.9.2+.

Documentation

Features

The main features provided at the moment are:

  • Support for all Kafka versions since 0.8.x. For more information aboutbroker compatibility options, check thelibrdkafkadocumentation.
  • Consume from single or multiple topics.
  • Automatic consumer rebalancing.
  • Customizable rebalance, with pre and post rebalance callbacks.
  • Synchronous or asynchronous message production.
  • Customizable offset commit.
  • Create and delete topics and add and edit partitions.
  • Alter broker and topic configurations.
  • Access to cluster metadata (list of topic-partitions, replicas, activebrokers etc).
  • Access to group metadata (list groups, list members of groups, hostnames,etc.).
  • Access to producer and consumer metrics, errors and callbacks.
  • Exactly-once semantics (EOS) via idempotent and transactional producersand read-committed consumers.

One million messages per second

rust-rdkafka is designed to be easy and safe to use thanks to theabstraction layer written in Rust, while at the same time being extremelyfast thanks to the librdkafka C library.

Here are some benchmark results using theBaseProducer,sending data to a single Kafka 0.11 process running in localhost (defaultconfiguration, 3 partitions). Hardware: Dell laptop, with Intel Corei7-4712HQ @ 2.30GHz.

  • Scenario: produce 5 million messages, 10 bytes each, wait for all of them to be acked

    • 1045413 messages/s, 9.970 MB/s (average over 5 runs)
  • Scenario: produce 100000 messages, 10 KB each, wait for all of them to be acked

    • 24623 messages/s, 234.826 MB/s (average over 5 runs)

For more numbers, check out thekafka-benchmark project.

Client types

rust-rdkafka provides low level and high level consumers and producers.

Low level:

  • BaseConsumer: a simple wrapper around the librdkafka consumer. Itmust be periodicallypoll()ed in order to execute callbacks, rebalancesand to receive messages.
  • BaseProducer: a simple wrapper around the librdkafka producer. As inthe consumer case, the user must callpoll() periodically to executedelivery callbacks.
  • ThreadedProducer: aBaseProducer with a separate thread dedicated topolling the producer.

High level:

For more information about consumers and producers, refer to theirmodule-level documentation.

Warning: the library is under active development and the APIs are likelyto change.

Asynchronous data processing with Tokio

Tokio is a platform for fast processing of asynchronous events in Rust.The interfaces exposed by theStreamConsumer and theFutureProducerallow rust-rdkafka users to easily integrate Kafka consumers and producerswithin the Tokio platform, and write asynchronous message processing code.Note that rust-rdkafka can be used without Tokio.

To see rust-rdkafka in action with Tokio, check out theasynchronous processing example in the examples folder.

At-least-once delivery

At-least-once delivery semantics are common in many streaming applications:every message is guaranteed to be processed at least once; in case oftemporary failure, the message can be re-processed and/or re-delivered,but no message will be lost.

In order to implement at-least-once delivery the stream processingapplication has to carefully commit the offset only once the message hasbeen processed. Committing the offset too early, instead, might causemessage loss, since upon recovery the consumer will start from the nextmessage, skipping the one where the failure occurred.

To see how to implement at-least-once delivery withrdkafka, check out theat-least-once delivery example in the examples folder. To know more aboutdelivery semantics, check themessage delivery semantics chapter in theKafka documentation.

Exactly-once semantics

Exactly-once semantics (EOS) can be achieved using transactional producers,which allow produced records and consumer offsets to be committed or abortedatomically. Consumers that set theirisolation.level toread_committedwill only observe committed messages.

EOS is useful in read-process-write scenarios that require messages to beprocessed exactly once.

To learn more about using transactions in rust-rdkafka, see theTransactions section of the producer documentation.

Users

Here are some of the projects using rust-rdkafka:

  • kafka-view: a web interface for Kafka clusters.
  • kafka-benchmark: a high performance benchmarking tool for Kafka.
  • callysto: Stream processing framework in Rust.
  • bytewax: Python stream processing framework using Timely Dataflow.

If you are using rust-rdkafka, please let us know!

Installation

Add this to yourCargo.toml:

[dependencies]rdkafka = {version ="0.25",features = ["cmake-build"] }

This crate will compile librdkafka from sources and link it statically toyour executable. To compile librdkafka you'll need:

  • the GNU toolchain
  • GNUmake
  • pthreads
  • zlib: optional, but included by default (feature:libz)
  • cmake: optional,not included by default (feature:cmake-build)
  • libssl-dev: optional,not included by default (feature:ssl)
  • libsasl2-dev: optional,not included by default (feature:gssapi)
  • libzstd-dev: optional,not included by default (feature:zstd-pkg-config)

Note that using the CMake build system, via thecmake-build feature, isencouraged if you can take the dependency on CMake.

By default a submodule with the librdkafka sources pinned to a specificcommit will be used to compile and statically link the library. Thedynamic-linking feature can be used to instead dynamically link rdkafka tothe system's version of librdkafka. Example:

[dependencies]rdkafka = {version ="0.25",features = ["dynamic-linking"] }

If you'd like to compile librdkafka statically yourself, then usethat, you can usestatic-linking while supplyingDEP_LIBRDKAFKA_STATIC_ROOTwith path to where librdkafka was built.

For a full listing of features, consult therdkafka-sys crate'sdocumentation. All of rdkafka-sys features arere-exported as rdkafka features.

Minimum supported Rust version (MSRV)

The current minimum supported Rust version (MSRV) is 1.70.0. Note thatbumping the MSRV is not considered a breaking change. Any release ofrust-rdkafka may bump the MSRV.

Asynchronous runtimes

Some features of theStreamConsumer andFutureProducer depend onTokio, which can be a heavyweight dependency for users who only intend touse the low-level consumers and producers. The Tokio integration isenabled by default, but can be disabled by turning off default features:

[dependencies]rdkafka = {version ="0.25",default-features =false }

If you would like to use an asynchronous runtime besides Tokio, you canintegrate it with rust-rdkafka by providing a shim that implements theAsyncRuntime trait. See the following examples for details:

Examples

You can find examples in theexamples folder. To run them:

cargo run --example<example_name> --<example_args>

Debugging

rust-rdkafka uses thelog crate to handle logging.Optionally, enable thetracing feature to emittracingevents as opposed tolog records.

In test and examples, rust-rdkafka uses theenv_logger crateto format logs. In those contexts, logging can be enabledusing theRUST_LOG environment variable, for example:

RUST_LOG="librdkafka=trace,rdkafka::client=debug" cargotest

This will configure the logging level of librdkafka to trace, and the levelof the client module of the Rust client to debug. To actually receive logsfrom librdkafka, you also have to set thedebug option in the producer orconsumer configuration (see librdkafkaconfiguration).

To enable debugging in your project, make sure you initialize the loggerwithenv_logger::init(), or the equivalent for anylog-compatiblelogging framework.

rdkafka-sys

Seerdkafka-sys.

Contributors

Thanks to:

Alternatives

  • kafka-rust: a pure Rust implementation of the Kafka client.

About

A fully asynchronous, futures-based Kafka client library for Rust based on librdkafka

Topics

Resources

License

Stars

Watchers

Forks

Packages

No packages published

Languages


[8]ページ先頭

©2009-2025 Movatter.jp