data-integration

Star

Here are 588 public repositories matching this topic...

Language:All

Filter by language

All588 Python157 Jupyter Notebook80 Java59 R37 JavaScript30 HTML20 Scala15 C#12 TypeScript10 Go9

Sort:Most stars

Sort options

Most stars Fewest stars Most forks Fewest forks Recently updated Least recently updated

apache /airflow

Star43.4k

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

python workflow data-science machine-learning airflow automation etl workflow-engine scheduler apache orchestration data-engineering data-integration elt data-pipelines dag apache-airflow mlops workflow-orchestration data-orchestrator

UpdatedNov 29, 2025
Python

airbytehq /airbyte

Star20.1k

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.

mysql python java bigquery data pipeline etl postgresql s3 snowflake self-hosted data-engineering data-analysis mssql data-integration data-collection redshift elt data-pipeline change-data-capture

UpdatedNov 28, 2025
Python

Avaiga /taipy

Star18.9k

Turns Data and AI algorithms into production-ready web applications in no time.

python workflow automation pipeline job-scheduler pipelines data-visualization orchestration datascience data-engineering developer-tools data-integration scenario hacktoberfest mlops data-ops scenario-analysis hacktoberfest2023 taipy-core taipy-gui

UpdatedNov 24, 2025
Python

dagster-io /dagster

Star14.5k

An orchestration platform for the development, production, and observation of data assets.

python metadata workflow data-science etl analytics scheduler orchestration data-engineering data-integration data-pipelines workflow-automation mlops dagster data-orchestrator

UpdatedNov 26, 2025
Python

apache /seatunnel

Star8.9k

SeaTunnel is a multimodal, high-performance, distributed, massive data integration tool.

streaming real-time offline high-performance apache embeddings batch data-integration elt cdc change-data-capture data-ingestion multimodal llm

UpdatedNov 29, 2025
Java

mage-ai /mage-ai

Star8.6k

🧙 Build, run, and manage data pipelines for integrating and transforming data.

python data-science data machine-learning sql spark pipeline etl pipelines orchestration artificial-intelligence data-engineering data-integration dbt elt transformation data-pipelines reverse-etl

UpdatedNov 27, 2025
Python

apache /flink-cdc

Star6.3k

Flink CDC is a streaming data integration tool

mysql real-time kafka etl postgresql distributed batch data-integration schema-evolution elt flink cdc data-pipeline change-data-capture paimon

UpdatedNov 26, 2025
Java

cloudquery /cloudquery

Star6.3k

Data pipelines for cloud config and security data. Build cloud asset inventory, CSPM, FinOps, and vulnerability management solutions. Extract from AWS, Azure, GCP, and 70+ cloud and SaaS sources.

go kubernetes github-api bigquery aws data google sql etl azure gcp data-engineering data-analysis data-integration data-collection elt etl-framework cspm airbyte attack-surface-management

UpdatedNov 29, 2025
Go

apache /hudi

Star6k

Upserts, Deletes And Incremental Processing on Big Data.

bigdata stream-processing data-integration datalake apachespark hudi apachehudi incremental-processing apacheflink

UpdatedNov 29, 2025
Java

infinyon /fluvio

Star5.1k

🦀 event stream processing for developers to collect and transform data in motion to power responsive data intensive applications.

rust distributed-systems streaming real-time serverless webassembly data-flow stream-processing data-analytics data-integration cloud-native data-pipelines stateful streaming-data stream-processing-engine event-driven-architecture streaming-analytics streaming-data-processing streaming-data-pipelines

UpdatedNov 20, 2025
Rust

jitsucom /jitsu

Star4.6k

Jitsu is an open-source Segment alternative. Fully-scriptable data ingestion engine for modern data teams. Set-up a real-time data pipeline in minutes, not days

golang bigquery postgres clickhouse snowflake data-integration data-collection redshift data-connectors

UpdatedNov 28, 2025
TypeScript

rudderlabs /rudder-server

Star4.3k

Privacy and Security focused Segment-alternative, in Golang and React

bigquery privacy etl snowflake data-warehouse data-engineering data-integration redshift elt cdp warehouse-management data-synchronization data-pipeline customer-data event-streaming customer-data-platform segment-alternative customer-data-pipeline customer-data-lake warehouse-native

UpdatedNov 28, 2025
Go

DTStack /chunjun

Star4.1k

A data integration framework

java framework bigdata data-integration flink

UpdatedNov 24, 2025
Java

seandavi /awesome-single-cell

Star3.6k

Community-curated list of software packages and data resources for single-cell, including RNA-seq, ATAC-seq, etc.

python bioinformatics analysis clustering gene-expression data-visualization dimensionality-reduction awesome-list data-integration atac-seq single-cell rna-seq-data scrna-seq-data cell-cycle cell-differentiation gene-expression-profiles analysis-pipeline cell-populations rna-seq-experiments cell-clusters

UpdatedOct 23, 2025

bruin-data /ingestr

Star3.3k

ingestr is a CLI tool to copy data between any databases with a single command seamlessly.

bigquery postgresql snowflake mssql data-integration data-pipeline data-ingestion copy-database ingestion-pipeline duckdb

UpdatedNov 26, 2025
Python

Apache DevLake is an open-source dev data platform to ingest, analyze, and visualize the fragmented data from DevOps tools, extracting insights for engineering excellence, developer experience, and community growth.

golang open-source devops data jira integration etl data-engineering data-analysis data-integration user-friendly hacktoberfest dora domain-layer data-transfers dashboard-friendly

UpdatedNov 25, 2025
Go

mara /mara-pipelines

Star2.1k

A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow

python data pipeline etl postgresql data-integration

UpdatedDec 15, 2023
Python

bytedance /bitsail

Star1.7k

BitSail is a distributed high-performance data integration engine which supports batch, streaming and incremental scenarios. BitSail is widely used to synchronize hundreds of trillions of data every day.

real-time big-data high-performance data-lake data-integration flink data-synchronization data-pipeline