data-pipeline

Star

Here are 1,375 public repositories matching this topic...

Language:All

Filter by language

All1,375 Python711 Jupyter Notebook263 Go39 Java30 JavaScript30 HTML27 TypeScript26 Scala21 Rust13 Shell13

Sort:Most stars

Sort options

Most stars Fewest stars Most forks Fewest forks Recently updated Least recently updated

apache /shardingsphere

Star20.6k

Empowering Data Intelligence with Distributed SQL for Sharding, Scalability, and Security Across All Databases.

mysql sql database bigdata postgresql shard distributed-database encrypt data-pipeline data-encryption database-cluster distributed-transaction read-write-splitting database-middleware distributed-sql-database database-gateway

UpdatedDec 17, 2025
Java

airbytehq /airbyte

Star20.3k

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.

mysql python java bigquery data pipeline etl postgresql s3 snowflake self-hosted data-engineering data-analysis mssql data-integration data-collection redshift elt data-pipeline change-data-capture

UpdatedDec 18, 2025
Python

debezium /debezium

Star12.2k

Change data capture for a variety of databases. Please log issues athttps://github.com/debezium/dbz/issues.

database kafka kafka-connect apache-kafka kafka-producer cdc hacktoberfest data-pipeline change-data-capture debezium event-streaming

UpdatedDec 17, 2025
Java

snowplow /snowplow

Star7k

The leader in Customer Data Infrastructure

data analytics snowplow data-collection data-pipeline product-analytics marketing-analytics snowplow-pipeline snowplow-events

UpdatedJun 4, 2025
Scala

apache /flink-cdc

Star6.3k

Flink CDC is a streaming data integration tool

mysql real-time kafka etl postgresql distributed batch data-integration schema-evolution elt flink cdc data-pipeline change-data-capture paimon

UpdatedDec 16, 2025
Java

datajuicer /data-juicer

Star5.6k

Data processing for and with foundation models! 🍎 🍋 🌽 ➡️ ➡️🍸 🍹 🍷

data-science data data-visualization data-analysis data-processing multi-modal data-pipeline synthetic-data pre-training foundation-models large-language-models llm llms instruction-tuning

UpdatedDec 17, 2025
Python

rudderlabs /rudder-server

Star4.3k

Privacy and Security focused Segment-alternative, in Golang and React

bigquery privacy etl snowflake data-warehouse data-engineering data-integration redshift elt cdp warehouse-management data-synchronization data-pipeline customer-data event-streaming customer-data-platform segment-alternative customer-data-pipeline customer-data-lake warehouse-native

UpdatedDec 17, 2025
Go

adilkhash /Data-Engineering-HowTo

Star3.9k

A list of useful resources to learn Data Engineering from scratch

distributed-systems scala cloud-providers data-engineering data-pipeline

UpdatedJun 19, 2024

superstreamlabs /memphis

Star3.4k

Memphis.dev is a highly scalable and effortless data streaming platform

kubernetes golang data enrichment microservices schema-registry message-bus message-queue data-engineering data-pipeline message-broker data-streaming data-stream-processing messaging-queue

UpdatedMay 27, 2024
Go

bruin-data /ingestr

Star3.4k

ingestr is a CLI tool to copy data between any databases with a single command seamlessly.

bigquery postgresql snowflake mssql data-integration data-pipeline data-ingestion copy-database ingestion-pipeline duckdb

UpdatedDec 17, 2025
Python

dagu-org /dagu

Sponsor

Star2.9k

Lightweight, local-first workflow engine for enterprise and small teams. 100% open source. No vendor lock-in. Offline or air-gapped environment ready.

devops cron continuous-delivery workflow-engine job-scheduler task-scheduler workflow-management task-automation dag data-pipeline workflow-scheduler directed-acyclic-graph workflow-orchestration durable-execution ai-workflow durable-workflows agent-workflow

UpdatedDec 17, 2025
Go

An open-source data logging library for machine learning models and data pipelines. 📚 Provides visibility into data quality & model performance over time. 🛡️ Supports privacy-preserving data collection, ensuring safety & robustness. 📈

python data-science machine-learning analytics logging constraints dataset dataops data-pipeline data-quality calculate-statistics data-constraints mlops model-performance ml-pipelines ai-pipelines approximate-statistics statistical-properties

UpdatedJan 10, 2025
Jupyter Notebook

elementary-data /elementary

Star2.2k

The dbt-native data observability solution for data & analytics engineers. Monitor your data pipelines in minutes. Available as self-hosted or cloud service with premium features.

bigquery snowflake data-warehouse dataops data-analysis redshift dbt data-pipelines data-pipeline lineage data-governance data-lineage analytics-engineer dbt-packages data-observability data-reliability dbt-artifacts

UpdatedDec 16, 2025
HTML

reugn /go-streams

Star2.1k

A lightweight stream processing library for Go

redis workflow kafka pipeline etl websocket data-stream streams stream-processing pulsar windowing aerospike streaming-api kafka-streams throttling data-pipeline streaming-data nats-streaming stream-processor low-code

UpdatedNov 20, 2025
Go

pydoit /doit

Star2k

CLI task management & automation tool

python cli workflow data-science build-automation task-runner build-tool build-system workflow-management hacktoberfest data-pipeline workflow-automation

UpdatedJul 4, 2024
Python

bytedance /bitsail

Star1.7k

BitSail is a distributed high-performance data integration engine which supports batch, streaming and incremental scenarios. BitSail is widely used to synchronize hundreds of trillions of data every day.

real-time big-data high-performance data-lake data-integration flink data-synchronization data-pipeline

UpdatedJan 1, 2024
Java

Multiwoven /multiwoven

Star1.6k

🔥🔥🔥 Open source Reverse ETL - alternative to hightouch and census.

react ruby open-source bigquery typescript etl snowflake self-hosted data-warehouse data-engineering redshift dbt databricks hacktoberfest cdp data-pipeline postresql customer-data-platform reverse-etl data-activation

UpdatedDec 17, 2025
Ruby

superlinked /superlinked

Star1.5k

Superlinked is a Python framework for AI Engineers building high-performance search & recommendation applications that combine structured and unstructured data.

python nlp natural-language-processing information-retrieval deep-learning etl retrieval ml embeddings vectorization semantic-search data-pipeline mlops vector-search vector-database llm retrieval-augmented-generation