data-integration

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.

mysql python java bigquery data pipeline etl postgresql s3 snowflake self-hosted data-engineering data-analysis mssql data-integration data-collection redshift elt data-pipeline change-data-capture

UpdatedApr 4, 2025
Python

dagster-io /dagster

Star12.8k

An orchestration platform for the development, production, and observation of data assets.

python metadata workflow data-science etl analytics scheduler orchestration data-engineering data-integration data-pipelines workflow-automation mlops dagster data-orchestrator

UpdatedApr 4, 2025
Python

apache /seatunnel

Star8.4k

SeaTunnel is a next-generation super high-performance, distributed, massive data integration tool.

streaming real-time offline high-performance apache batch data-integration elt cdc change-data-capture data-ingestion

UpdatedApr 3, 2025
Java

mage-ai /mage-ai

Star8.2k

🧙 Build, run, and manage data pipelines for integrating and transforming data.

python data-science data machine-learning sql spark pipeline etl pipelines orchestration artificial-intelligence data-engineering data-integration dbt elt transformation data-pipelines reverse-etl

UpdatedApr 1, 2025
Python

cloudquery /cloudquery

Star6.1k

The developer first cloud governance platform

go kubernetes github-api bigquery aws data google sql etl azure gcp data-engineering data-analysis data-integration data-collection elt etl-framework cspm airbyte attack-surface-management

UpdatedApr 3, 2025
Go

apache /flink-cdc

Star6k

Flink CDC is a streaming data integration tool

mysql real-time kafka etl postgresql distributed batch data-integration schema-evolution elt flink cdc data-pipeline change-data-capture paimon

UpdatedApr 3, 2025
Java

apache /hudi

Star5.7k

Upserts, Deletes And Incremental Processing on Big Data.

bigdata stream-processing data-integration datalake apachespark hudi apachehudi incremental-processing apacheflink

UpdatedApr 4, 2025
Java

infinyon /fluvio

Star4.4k

Lean and mean distributed stream processing system written in rust and web assembly. Alternative to Kafka + Flink in one.

rust distributed-systems streaming real-time serverless webassembly data-flow stream-processing data-analytics data-integration cloud-native data-pipelines stateful streaming-data stream-processing-engine event-driven-architecture streaming-analytics streaming-data-processing streaming-data-pipelines

UpdatedApr 3, 2025
Rust

jitsucom /jitsu

Star4.3k

Jitsu is an open-source Segment alternative. Fully-scriptable data ingestion engine for modern data teams. Set-up a real-time data pipeline in minutes, not days

golang bigquery postgres clickhouse snowflake data-integration data-collection redshift data-connectors

UpdatedApr 3, 2025
TypeScript

rudderlabs /rudder-server

Star4.2k

Privacy and Security focused Segment-alternative, in Golang and React

bigquery privacy etl snowflake data-warehouse data-engineering data-integration redshift elt cdp warehouse-management data-synchronization data-pipeline customer-data event-streaming customer-data-platform segment-alternative customer-data-pipeline customer-data-lake warehouse-native

UpdatedApr 4, 2025
Go

DTStack /chunjun

Star4k

A data integration framework

java framework bigdata data-integration flink

UpdatedMar 5, 2025
Java

seandavi /awesome-single-cell

Star3.3k

Community-curated list of software packages and data resources for single-cell, including RNA-seq, ATAC-seq, etc.

python bioinformatics analysis clustering gene-expression data-visualization dimensionality-reduction awesome-list data-integration atac-seq single-cell rna-seq-data scrna-seq-data cell-cycle cell-differentiation gene-expression-profiles analysis-pipeline cell-populations rna-seq-experiments cell-clusters

UpdatedMar 27, 2025

bruin-data /ingestr

Star2.9k

ingestr is a CLI tool to copy data between any databases with a single command seamlessly.

bigquery postgresql snowflake mssql data-integration data-pipeline data-ingestion copy-database ingestion-pipeline duckdb

UpdatedApr 1, 2025
Python

Apache DevLake is an open-source dev data platform to ingest, analyze, and visualize the fragmented data from DevOps tools, extracting insights for engineering excellence, developer experience, and community growth.

golang open-source devops data jira integration etl data-engineering data-analysis data-integration user-friendly hacktoberfest dora domain-layer data-transfers dashboard-friendly

UpdatedMar 31, 2025
Go

mara /mara-pipelines

Star2.1k

A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow

python data pipeline etl postgresql data-integration

UpdatedDec 15, 2023
Python

bytedance /bitsail

Star1.7k

BitSail is a distributed high-performance data integration engine which supports batch, streaming and incremental scenarios. BitSail is widely used to synchronize hundreds of trillions of data every day.

real-time big-data high-performance data-lake data-integration flink data-synchronization data-pipeline

UpdatedJan 1, 2024
Java

apache /hop

Star1.1k

Hop Orchestration Platform

java workflow streaming pipeline etl apache orchestration data-integration hop

UpdatedApr 3, 2025
Java

kuwala-io /kuwala

Star792

Kuwala is the no-code data platform for BI analysts and engineers enabling you to build powerful analytics workflows. We are set out to bring state-of-the-art data engineering tools you love, such as Airbyte, dbt, or Great Expectations together in one intuitive interface built with React Flow. In addition we provide third-party data into data sc…

react python open-source postgres data-science data jupyter scraping open-data pyspark population data-integration spatial-analysis dbt elt google-trends no-code react-flow admin-boundaries kuwala

UpdatedAug 10, 2022
JavaScript

Improve this page

Add a description, image, and links to thedata-integration topic page so that developers can more easily learn about it.

Curate this topic

Add this topic to your repo

To associate your repository with thedata-integration topic, visit your repo's landing page and select "manage topics."

Learn more

Movatterモバイル変換

Navigation Menu

Search code, repositories, users, issues, pull requests...

Provide feedback

Saved searches

Use saved searches to filter your results more quickly