data-integration

Star

Here are 538 public repositories matching this topic...

Language:All

Filter by language

All538 Python150 Jupyter Notebook77 Java59 R37 JavaScript28 HTML18 Scala15 C#12 Go9 TypeScript9

Sort:Most stars

Sort options

Most stars Fewest stars Most forks Fewest forks Recently updated Least recently updated

apache /airflow

Star42.7k

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

python workflow data-science machine-learning airflow automation etl workflow-engine scheduler apache orchestration data-engineering data-integration elt data-pipelines dag apache-airflow mlops workflow-orchestration data-orchestrator

UpdatedOct 7, 2025
Python

airbytehq /airbyte

Star19.7k

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.

mysql python java bigquery data pipeline etl postgresql s3 snowflake self-hosted data-engineering data-analysis mssql data-integration data-collection redshift elt data-pipeline change-data-capture

UpdatedOct 8, 2025
Python

Avaiga /taipy

Star18.8k

Turns Data and AI algorithms into production-ready web applications in no time.

python workflow automation pipeline job-scheduler pipelines data-visualization orchestration datascience data-engineering developer-tools data-integration scenario hacktoberfest mlops data-ops scenario-analysis hacktoberfest2023 taipy-core taipy-gui

UpdatedOct 7, 2025
Python

dagster-io /dagster

Star14.2k

An orchestration platform for the development, production, and observation of data assets.

python metadata workflow data-science etl analytics scheduler orchestration data-engineering data-integration data-pipelines hacktoberfest workflow-automation mlops dagster data-orchestrator

UpdatedOct 7, 2025
Python

apache /seatunnel

Star8.8k

SeaTunnel is a multimodal, high-performance, distributed, massive data integration tool.

streaming real-time offline high-performance apache embeddings batch data-integration elt cdc change-data-capture data-ingestion multimodal llm

UpdatedOct 6, 2025
Java

mage-ai /mage-ai

Star8.5k

🧙 Build, run, and manage data pipelines for integrating and transforming data.

python data-science data machine-learning sql spark pipeline etl pipelines orchestration artificial-intelligence data-engineering data-integration dbt elt transformation data-pipelines reverse-etl

UpdatedOct 7, 2025
Python

apache /flink-cdc

Star6.2k

Flink CDC is a streaming data integration tool

mysql real-time kafka etl postgresql distributed batch data-integration schema-evolution elt flink cdc data-pipeline change-data-capture paimon

UpdatedOct 5, 2025
Java

cloudquery /cloudquery

Star6.2k

The open source ELT framework powered by Apache Arrow

go kubernetes github-api bigquery aws data google sql etl azure gcp data-engineering data-analysis data-integration data-collection elt etl-framework cspm airbyte attack-surface-management

UpdatedOct 7, 2025
Go

apache /hudi

Star6k

Upserts, Deletes And Incremental Processing on Big Data.

bigdata stream-processing data-integration datalake apachespark hudi apachehudi incremental-processing apacheflink

UpdatedOct 7, 2025
Java

infinyon /fluvio

Star5k

🦀 event stream processing for developers to collect and transform data in motion to power responsive data intensive applications.

rust distributed-systems streaming real-time serverless webassembly data-flow stream-processing data-analytics data-integration cloud-native data-pipelines stateful streaming-data stream-processing-engine event-driven-architecture streaming-analytics streaming-data-processing streaming-data-pipelines

UpdatedOct 1, 2025
Rust

jitsucom /jitsu

Star4.5k

Jitsu is an open-source Segment alternative. Fully-scriptable data ingestion engine for modern data teams. Set-up a real-time data pipeline in minutes, not days

golang bigquery postgres clickhouse snowflake data-integration data-collection redshift data-connectors

UpdatedOct 2, 2025
TypeScript

rudderlabs /rudder-server

Star4.3k

Privacy and Security focused Segment-alternative, in Golang and React

bigquery privacy etl snowflake data-warehouse data-engineering data-integration redshift elt cdp warehouse-management data-synchronization data-pipeline customer-data event-streaming customer-data-platform segment-alternative customer-data-pipeline customer-data-lake warehouse-native

UpdatedOct 7, 2025
Go

DTStack /chunjun

Star4.1k

A data integration framework

java framework bigdata data-integration flink

UpdatedMar 5, 2025
Java

seandavi /awesome-single-cell

Star3.5k

Community-curated list of software packages and data resources for single-cell, including RNA-seq, ATAC-seq, etc.

python bioinformatics analysis clustering gene-expression data-visualization dimensionality-reduction awesome-list data-integration atac-seq single-cell rna-seq-data scrna-seq-data cell-cycle cell-differentiation gene-expression-profiles analysis-pipeline cell-populations rna-seq-experiments cell-clusters

UpdatedAug 26, 2025

bruin-data /ingestr

Star3.2k

ingestr is a CLI tool to copy data between any databases with a single command seamlessly.

bigquery postgresql snowflake mssql data-integration data-pipeline data-ingestion copy-database ingestion-pipeline duckdb

UpdatedOct 7, 2025
Python

Apache DevLake is an open-source dev data platform to ingest, analyze, and visualize the fragmented data from DevOps tools, extracting insights for engineering excellence, developer experience, and community growth.

golang open-source devops data jira integration etl data-engineering data-analysis data-integration user-friendly hacktoberfest dora domain-layer data-transfers dashboard-friendly

UpdatedSep 30, 2025
Go

mara /mara-pipelines

Star2.1k

A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow

python data pipeline etl postgresql data-integration

UpdatedDec 15, 2023
Python

bytedance /bitsail

Star1.7k

BitSail is a distributed high-performance data integration engine which supports batch, streaming and incremental scenarios. BitSail is widely used to synchronize hundreds of trillions of data every day.

real-time big-data high-performance data-lake data-integration flink data-synchronization data-pipeline