Repository files navigation

KStream is a light-weight implementation ofkafka streams.It is heavily inspired by Kafka-Streams(Java) library. KStreamimplements features like Internal Stores, Remote Stores, Local Tables, Global Tables and it guarantees Exactly OnceProcessing with its de-duplicator. It can process message as micro batches or one by one depending on the throughputrequired. KStream can handle good amount of throughput (50,000 messages pe second with micro batch enabled) in a faulttolerable manner with a very minimal amount of latency (2 ~ 5 ms)

Project uses two external librariessarama for consumers and producers

Stream Topology is a set of processor nodes typically starts with a source node and ends with a sink node

Stream builder is responsible for building stream topologies into kafka streams with their dependencies like changelogtopics, re-partition topics, etc...

KStream is a regular kafka stream which takes an input topic as an upstream and process the record stream to anotherkafka topic(downstream topic)It supports several functions like transforming, merging to another stream, joins with other streams etc.

@TODO

Global KTable also a KTable except for is each instance of the application has its owncopy of all the partitions and it will be running on a separate thread so applications dose'nt have to worryabout handling failures.

Store is a pluggable local key-val store which is used by KTable, Global KTable and other oparations like joins, mergesand removing duplicates.

Store is a pluggable local key-val store which is used by KTable, Global KTable and other oparations like joins, mergesand removing duplicates.

@TODO

@TODO