Repository files navigation

Copyright (c) 2012-2022,Magnus Edenhill.2023Confluent Inc..

https://github.com/confluentinc/librdkafka

librdkafka is a C library implementation of theApache Kafka protocol, providing Producer, Consumerand Admin clients. It was designed with message delivery reliabilityand high performance in mind, current figures exceed 1 million msgs/second forthe producer and 3 million msgs/second for the consumer.

librdkafka is licensed under the 2-clause BSD license.

KAFKA is a registered trademark of The Apache Software Foundation andhas been licensed for use by librdkafka. librdkafka has noaffiliation with and is not endorsed by The Apache Software Foundation.

• Full Exactly-Once-Semantics (EOS) support
• High-level producer, including Idempotent and Transactional producers
• High-level balanced KafkaConsumer (requires broker >= 0.9)
• Simple (legacy) consumer
• Admin client
• Compression: snappy, gzip, lz4, zstd
• SSL support
• SASL (GSSAPI/Kerberos/SSPI, PLAIN, SCRAM, OAUTHBEARER) support
• Full list ofsupported KIPs
• Broker version support: >=0.8 (seeBroker version compatibility)
• Guaranteed API stability for C & C++ APIs (ABI safety guaranteed for C)
• Statistics metrics
• Debian package: librdkafka1 and librdkafka-dev in Debian and Ubuntu
• RPM package: librdkafka and librdkafka-devel
• Gentoo package: dev-libs/librdkafka
• Portable: runs on Linux, MacOS X, Windows, Solaris, FreeBSD, AIX, ...

• Public API inC header andC++ header.
• Introduction and manual inINTRODUCTION.md.
• Configuration properties inCONFIGURATION.md.
• Statistics metrics inSTATISTICS.md.
• Frequently asked questions.
• Step-by-step tutorialGetting Started with Apache Kafka and C/C++.

NOTE: Themaster branch is actively developed, use latestrelease for production use.

On Mac OSX, install librdkafka with homebrew:

$ brew install librdkafka

On Debian and Ubuntu, install librdkafka from the Confluent APT repositories,see instructionshere and then install librdkafka:

$ apt install librdkafka-dev

On RedHat, CentOS, Fedora, install librdkafka from the Confluent YUM repositories,instructionshere and then install librdkafka:

$ yum install librdkafka-devel

On Windows, referencelibrdkafka.redist NuGet package in your Visual Studio project.

For other platforms, follow the source building instructions below.

You can download and install librdkafka using thevcpkg dependency manager:

# Install vcpkg if not already installed$ git clone https://github.com/Microsoft/vcpkg.git$cd vcpkg$ ./bootstrap-vcpkg.sh$ ./vcpkg integrate install# Install librdkafka$ vcpkg install librdkafka

The librdkafka package in vcpkg is kept up to date by Microsoft team members and community contributors.If the version is out of date, pleasecreate an issue or pull request on the vcpkg repository.

The GNU toolchainGNU makepthreadszlib-dev (optional, for gzip compression support)libssl-dev (optional, for SSL and SASL SCRAM support)libsasl2-dev (optional, for SASL GSSAPI support)libzstd-dev (optional, for ZStd compression support)libcurl-dev (optional, for SASL OAUTHBEARER OIDC support)

NOTE: Static linking of ZStd (requires zstd >= 1.2.1) in the producerenables encoding the original size in the compression frame header,which will speed up the consumer.UseSTATIC_LIB_libzstd=/path/to/libzstd.a ./configure --enable-staticto enable static ZStd linking.MacOSX example:STATIC_LIB_libzstd=$(brew ls -v zstd | grep libzstd.a$) ./configure --enable-static

  ./configure  # Or, to automatically install dependencies using the system's package manager:  # ./configure --install-deps  # Or, build dependencies from source:  # ./configure --install-deps --source-deps-only  make  sudo make install

NOTE: SeeREADME.win32 for instructions how to buildon Windows with Microsoft Visual Studio.

NOTE: SeeCMake instructions for experimentalCMake build (unsupported).

Seegetting Started with Apache Kafka and C/C++ for a basic tutorial.

1. Refer to theexamples directory for code using:

   • Producers: basic producers, idempotent producers, transactional producers.
   • Consumers: basic consumers, reading batches of messages.
   • Performance and latency testing tools.

2. Refer to theexamples GitHub repo for code connecting to a cloud streaming data service based on Apache Kafka

3. Link your program with-lrdkafka (C) or-lrdkafka++ (C++).

Commercial support is available fromConfluent Inc

Only thelatest official release is supported for community members.

File bug reports and feature requests usingGitHub Issues.

Questions and discussions are welcome on theDiscussions forum, and on theConfluent Community slack #clients channel.

• C#/.NET:confluent-kafka-dotnet (based onrdkafka-dotnet)
• C++:cppkafka
• C++:modern-cpp-kafka
• Common Lisp:cl-rdkafka
• D (C-like):librdkafka
• D (C++-like):librdkafkad
• Erlang:erlkaf
• Go:confluent-kafka-go
• Haskell (kafka, conduit, avro, schema registry):hw-kafka
• Kotlin Native:Kafka-Kotlin-Native
• Lua:luardkafka
• Node.js:node-rdkafka
• OCaml:ocaml-kafka
• Perl:Net::Kafka
• PHP:php-rdkafka
• PHP:php-simple-kafka-client
• Python:confluent-kafka-python
• Python:PyKafka
• Ruby:Hermann
• Ruby:rdkafka-ruby
• Rust:rust-rdkafka
• Tcl:KafkaTcl
• Shell:kafkacat - Apache Kafka command line tool
• Swift:Perfect-Kafka

SeePowered by librdkafka for an incomplete list of librdkafka users.