streaming-data

Star

Here are 446 public repositories matching this topic...

Language:All

Filter by language

All446 Python118 Java50 JavaScript43 Jupyter Notebook36 Go25 TypeScript21 C#18 C++16 Scala12 HTML11

Sort:Most stars

Sort options

Most stars Fewest stars Most forks Fewest forks Recently updated Least recently updated

oxnr /awesome-bigdata

Star13.5k

A curated list of awesome big data frameworks, ressources and other awesomeness.

data-science data awesome database data-stream bigdata series-database data-visualization data-warehouse stream-processing data-analytics awesome-list distributed-database visualize-data streaming-data

UpdatedFeb 14, 2025

provectus /kafka-ui

Star10.5k

Open-Source Web UI for Apache Kafka Management

opensource kafka big-data web-ui streams kafka-connect apache-kafka kafka-producer kafka-client kafka-streams hacktoberfest streaming-data kafka-manager kafka-cluster event-streaming cluster-management kafka-ui kafka-brokers

UpdatedJul 26, 2024
Java

johnkerl /miller

Star9.2k

Miller is like awk, sed, cut, join, and sort for name-indexed data such as CSV, TSV, and tabular JSON

tsv devops json statistics csv command-line json-data tabular-data data-reduction unix-toolkit statistical-analysis csv-format devops-tools data-regression data-processing command-line-tools data-cleaning streaming-algorithms streaming-data miller

UpdatedMar 20, 2025
Go

redpanda-data /connect

Star8.3k

Fancy stream processing made operationally mundane

go golang kafka cqrs etl rabbitmq amqp logs message-bus event-sourcing nats stream-processing message-queue data-engineering streaming-data stream-processor data-ops

UpdatedMar 21, 2025
Go

MaterializeInc /materialize

Star5.9k

Real-time Data Integration and Transformation: use SQL to transform, deliver, and act on fast-changing data.

rust distributed-systems streaming sql database kafka postgresql data-store stream-processing streaming-data materialized-view postgresql-dialect operational-data-store

UpdatedMar 21, 2025
Rust

online-ml /river

Star5.2k

🌊 Online machine learning in Python

python data-science machine-learning streaming stream-processing online-learning streaming-data concept-drift real-time-processing incremental-learning online-machine-learning online-statistics

UpdatedMar 6, 2025
Python

Readyset is a MySQL and Postgres wire-compatible caching layer that sits in front of existing databases to speed up queries and horizontally scale read throughput. Under the hood, ReadySet caches the results of cached select statements and incrementally updates these results over time as the underlying data changes.

mysql rust caching postgres sql backend cache postgresql databases rust-lang mysql-database postgresql-database streaming-data caching-proxy

UpdatedMar 21, 2025
Rust

infinyon /fluvio

Star4.4k

Lean and mean distributed stream processing system written in rust and web assembly. Alternative to Kafka + Flink in one.

rust distributed-systems streaming real-time serverless webassembly data-flow stream-processing data-analytics data-integration cloud-native data-pipelines stateful streaming-data stream-processing-engine event-driven-architecture streaming-analytics streaming-data-processing streaming-data-pipelines

UpdatedMar 20, 2025
Rust

piskvorky /smart_open

Sponsor

Star3.3k

Utils for streaming large files (S3, HDFS, gzip, bz2...)

python streaming s3 file hdfs hacktoberfest webhdfs boto gzip-stream bz2 streaming-data

UpdatedMar 6, 2025
Python

memgraph /memgraph

Star2.7k

Open-source graph database, tuned for dynamic analytics environments. Easy to adopt, scale and own.

kafka graph graph-algorithms nosql stream-processing graph-database kafka-streams cypher graph-analysis streaming-data opencypher

UpdatedMar 21, 2025
C++

reugn /go-streams

Star2k

A lightweight stream processing library for Go

redis workflow kafka pipeline etl websocket data-stream streams stream-processing pulsar windowing aerospike streaming-api kafka-streams throttling data-pipeline streaming-data nats-streaming stream-processor low-code

UpdatedMar 21, 2025
Go

pravega /pravega

Star2k

Pravega - Streaming as a new software defined storage primitive

streaming distributed-storage real-time-data streaming-data data-ingestion

UpdatedMar 2, 2025
Java

bytewax /bytewax

Star1.7k

Python Stream Processing

python rust data-science machine-learning stream-processing data-engineering dataflow data-processing streaming-data

UpdatedMar 18, 2025
Python

quixio /quix-streams

Star1.3k

Python Streaming DataFrames for Kafka

python data-science machine-learning kafka stream-processing data-engineering streaming-data stream-processor event-driven-architecture streaming-data-processing time-series-data data-intensive-applications streaming-data-pipelines real-time-data-processing

UpdatedMar 21, 2025
Python

python-streamz /streamz

Star1.3k

Real-time stream processing for python

python real-time async streaming-data

UpdatedNov 22, 2024
Python

microsoft /Trill

Star1.3k

Trill is a single-node query processor for temporal or streaming data.

streaming-data temporal-data

UpdatedJan 8, 2024
C#

zpl-c /zpl

Star1k

📐 Pushing the boundaries of simplicity

c hashing cli helper time cross-platform math cpp timer coroutines tar header-only threading thread-pool memory-management zpl memory-allocation csv-parser streaming-data json5-parser

UpdatedAug 18, 2024
C

DoneDeal0 /superdiff

Sponsor

Star910

Superdiff provides a complete and readable diff for both arrays and objects. Plus, it supports stream and file inputs for handling large datasets efficiently, is battle-tested, has zero dependencies, and is super fast.

react nodejs diff streaming typescript comparison json-diff streaming-data comparison-tool object-comparison deep-diff array-comparison object-diff objectdifference objectdiff