Repository files navigation

Karafka is a Ruby and Rails multi-threaded efficient Kafka processing framework that:

• Has a built-inWeb UI providing a convenient way to monitor and manage Karafka-based applications.
• Supports parallel processing inmultiple threads (also for asingle topic partition work) andprocesses.
• Automatically integrates with Ruby on Rails
• HasActiveJob backend support (includingordered jobs)
• Has a seamlessDead Letter Queue functionality built-in
• Supports in-developmentcode reloading
• Is powered bylibrdkafka (the Apache Kafka C/C++ client library)
• Has an out-of the boxAppSignal andStatsD/DataDog monitoring with dashboard templates.

# Define what topics you want to consume with which consumers in karafka.rbKarafka::App.routes.drawdotopic'system_events'doconsumerEventsConsumerendend# And create your consumers, within which your messages will be processedclassEventsConsumer <ApplicationConsumer# Example that utilizes ActiveRecord#insert_all and Karafka batch processingdefconsume# Store all of the incoming Kafka events locally in an efficient wayEvent.insert_allmessages.payloadsendend

Karafkauses threads to handle many messages simultaneously in the same process. It does not require Rails but will integrate tightly with any Ruby on Rails applications to make event processing dead simple.

If you're entirely new to the subject, you can start with our "Kafka on Rails" articles series, which will get you up and running with the terminology and basic ideas behind using Kafka:

• Kafka on Rails: Using Kafka with Ruby on Rails – Part 1 – Kafka basics and its advantages
• Kafka on Rails: Using Kafka with Ruby on Rails – Part 2 – Getting started with Rails and Kafka

If you want to get started with Kafka and Karafka as fast as possible, then the best idea is to visit ourGetting started guides and theexample apps repository.

We also maintain manyintegration specs illustrating various use-cases and features of the framework.

Prerequisites: Kafka running. You can start it by following instructions fromhere.

1. Add and install Karafka:

# Make sure to install Karafka 2.4bundle add karafka --version">= 2.4.0"bundleexec karafka install

2. Dispatch a message to the example topic using the Rails or Ruby console:

Karafka.producer.produce_sync(topic:'example',payload:{'ping'=>'pong'}.to_json)

3. Run Karafka server and see the consumption magic happen:

bundleexec karafka server[86d47f0b92f7] Polled 1 messagein 1000ms[3732873c8a74] Consume jobfor ExampleConsumer on example started{"ping"=>"pong"}[3732873c8a74] Consume jobforExampleConsumer on example finishedin 0ms

I also sell Karafka Pro subscriptions. It includes a commercial-friendly license, priority support, architecture consultations, enhanced Web UI and high throughput data processing-related features (virtual partitions, long-running jobs, and more).

Part of the income isdistributed back to other OSS projects that Karafka uses under the hood.

Help me provide high-quality open-source software. Please see the Karafkahomepage for more details.

Karafka hasWiki pages for almost everything and a pretty decentFAQ. It covers the installation, setup, and deployment, along with other useful details on how to run Karafka.

If you have questions about using Karafka, feel free to join ourSlack channel.

Karafka haspriority support for technical and architectural questions that is part of the Karafka Pro subscription.