Repository files navigation

// suggested Node.js version: v12.16.1npm install --save kafka-streams

const{KafkaStreams}=require("kafka-streams");constconfig=require("./config.json");constfactory=newKafkaStreams(config);constkstream=factory.getKStream("input-topic");constktable=factory.getKTable(/* .. */);kstream.merge(ktable).filter(/* .. */).map(/* .. */).reduce(/* .. */).to("output-topic");

CHANGES: The latest version brings a lot of changes, please checkhere before updating.

• Quick Start
• Message 'to' and 'from' Apache Kafka
• API Info
• Documentation
• Operator descriptions
• Examples
• Native Client |SSL, SASL, Kerberos

• node-kafka-connect
• node-schema-registry
• node-kafka-rest-ui

• node-kafka-streams
  • API Overview
  • You might also like
  • README Overview
  • Prerequisites
    • A note on native mode
  • Aim of this Library
  • Description
  • Port Progress Overview
  • Operator Implementations
  • Additional Operators
  • Stream Action Implementations
  • Join Operations
    • KStream Status
    • KTable Status
    • KTable <-> KStream Status
  • Window Operations
    • KStream
• More
  • Can I use this library yet?
  • Are we ready for production yet?
  • Even More

• Kafka broker should be version>= 0.11.x
• Node.js should be version>= 8.x.x

If you are using the native mode (config: { noptions: {} }).You will have to manually installnode-rdkafka alongside kafka-streams.(This requires a Node.js version between 9 and 12 and will not work with Node.js >= 13, last tested with 12.16.1)

On Mac OS High Sierra / Mojave:CPPFLAGS=-I/usr/local/opt/openssl/include LDFLAGS=-L/usr/local/opt/openssl/lib yarn add --frozen-lockfile node-rdkafka@2.7.4

Otherwise:yarn add --frozen-lockfile node-rdkafka@2.7.4

(Please also note: Doing this with npm does not work, it will remove your deps,npm i -g yarn)

• this is not a 1:1 port of the official JAVA kafka-streams
• the goal of this project is to give at least the same options toa nodejs developer that kafka-streams provides for JVM developers
• stream-state processing, table representation, joins, aggregate etc.I am aiming for the easiest api access possible checkout theword count example

kafka-streams 🐙 equivalent for nodejs ✨🐢🚀✨build on super fast 🔥 observables usingmost.js 🤘

ships withsinek 🙏 for backpressure

comes with js and native Kafka client, for more performance and SSL, SASL and Kerberos features

the lib also comes with a fewwindow operations that are more similar toApache Flink,yet they still feel natural in this api :squirrel:

overwriteable local-storage solution allows for any kind of datastore e.g. RocksDB, Redis, Postgres..

async (Promises) and sync stream operators e.g.stream$.map() orstream$.asyncMap()

super easy API

the lib is based onsinek, which is based on kafka-node'sConsumerGroups

• core structure
• KStream base - stream as a changelog
• KTable base - stream as a database
• KStream & KTable cloning
• complex stream join structure
• advanced joinssee
• windows (for joins)see
• flink like window operations
• word-count example
• more examples
• local-storage for etl actions
• local-storage factory (one per action)
• KStorage example for any DB that supports atomic actions
• backing-up local-storage via kafka
• kafka client implementation
• KTable replay to Kafka (produce)
• stream for topic message production only
• sinek implementation
• backpressure mode for KafkaClient
• auto-json payloads (read-map/write-map)
• auto producer partition and keyed-message handling
• documentation
• API description
• higher join & combine examples
• embed native clientlibrdkafka for more performance
• SSL
• SASL
• Kerberos

• map
• asyncMap
• constant
• scan
• timestamp
• tap
• filter
• skipRepeats
• skipRepeatsWith
• slice
• take
• skip
• takeWhile
• skipWhile
• until
• since
• reduce
• chainReduce
• forEach (observe)
• chainForEach
• drain
• _zip
• _merge
• _join
• _combine
• _sample
• throttle
• debounce
• delay
• multicast
• A description of the operators can be foundhere
• Missing an operator? Feel free to open an issue 👮

• mapStringToArray
• mapArrayToKV
• mapStringToKV
• mapParse
• mapStringify
• atThroughput
• mapWrapKafkaPayload
• mapToFormat
• mapFromFormat
• Want more? Feel free to open an issue 👮

• countByKey
• sumByKey
• min
• max
• Want more? Feel free to open an issue 👮

Operation description

• merge
• outerJoin
• innerJoin
• leftJoin
• branch

• merge
• outerJoin
• innerJoin
• leftJoin

• merge
• outerJoin
• innerJoin
• leftJoin

• window
• advanced window
• rolling window

Yes.

Probably, yes. 😄

Forks or Stars give motivation