etl

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.

mysql python java bigquery data pipeline etl postgresql s3 snowflake self-hosted data-engineering data-analysis mssql data-integration data-collection redshift elt data-pipeline change-data-capture

UpdatedJul 18, 2025
Python

dagster-io /dagster

Star13.6k

An orchestration platform for the development, production, and observation of data assets.

python metadata workflow data-science etl analytics scheduler orchestration data-engineering data-integration data-pipelines workflow-automation mlops dagster data-orchestrator

UpdatedJul 18, 2025
Python

mage-ai /mage-ai

Star8.4k

🧙 Build, run, and manage data pipelines for integrating and transforming data.

python data-science data machine-learning sql spark pipeline etl pipelines orchestration artificial-intelligence data-engineering data-integration dbt elt transformation data-pipelines reverse-etl

UpdatedJul 18, 2025
Python

redpanda-data /connect

Star8.4k

Fancy stream processing made operationally mundane

go golang kafka cqrs etl rabbitmq amqp logs message-bus event-sourcing nats stream-processing message-queue data-engineering streaming-data stream-processor data-ops

UpdatedJul 18, 2025
Go

turbot /steampipe

Star7.4k

Zero-ETL, infinite possibilities. Live query APIs, code & more with SQL. No DB required.

kubernetes golang aws security devops cis cloud etl azure sqlite terraform postgresql gcp hacktoberfest devsecops cspm steampipe postgresql-fdw cnapp zero-etl

UpdatedJul 17, 2025
Go

apache /flink-cdc

Star6.1k

Flink CDC is a streaming data integration tool

mysql real-time kafka etl postgresql distributed batch data-integration schema-evolution elt flink cdc data-pipeline change-data-capture paimon

UpdatedJul 10, 2025
Java

cloudquery /cloudquery

Star6.1k

The developer first cloud governance platform

go kubernetes github-api bigquery aws data google sql etl azure gcp data-engineering data-analysis data-integration data-collection elt etl-framework cspm airbyte attack-surface-management

UpdatedJul 18, 2025
Go

rudderlabs /rudder-server

Star4.2k

Privacy and Security focused Segment-alternative, in Golang and React

bigquery privacy etl snowflake data-warehouse data-engineering data-integration redshift elt cdp warehouse-management data-synchronization data-pipeline customer-data event-streaming customer-data-platform segment-alternative customer-data-pipeline customer-data-lake warehouse-native

UpdatedJul 18, 2025
Go

orchest /orchest

Star4.1k

Build data pipelines, the easy way 🛠️

python docker kubernetes data-science machine-learning airflow cloud deployment jupyter etl ide pipelines self-hosted jupyterlab notebooks data-pipelines dag etl-pipeline orchest

UpdatedJun 6, 2023
TypeScript

pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).

mysql python emr aws data-science lambda aws-lambda athena etl pandas data-engineering redshift ray apache-parquet amazon-athena apache-arrow aws-glue modin glue-catalog amazon-sagemaker-notebook

UpdatedJul 14, 2025
Python

nucleuscloud /neosync

Star3.9k

Open Source Data Security Platform for Developers to Monitor and Detect PII, Anonymize Production Data and Sync it across environments.

mysql testing docker kubernetes golang open-source typescript etl reactjs nextjs postgresql faker self-hosted orchestration test-data-generator synthetic-data fine-tuning benthos synthetic-data-generation

UpdatedJul 18, 2025
Go

quadratichq /quadratic

Star3.8k

Spreadsheet with AI, Code, Connections

python webgl data-science data sql ai etl spreadsheet wasm data-engineering data-analysis quadratic

UpdatedJul 18, 2025
Rust

Netflix /maestro

Star3.5k

Maestro: Netflix’s Workflow Orchestrator

java workflow data-science machine-learning automation etl analytics workflow-engine scheduler orchestration data-engineering maestro elt data-pipelines batch-processing dag mlops data-ops workflow-orchestration data-orchestrator

UpdatedJul 11, 2025
Java

xyflow /awesome-node-based-uis

Sponsor

Star3.1k

A curated list with resources about node-based UIs

etl awesome-list visual-programming workflow-editor node-based-ui

UpdatedJun 29, 2025

blockchain-etl /ethereum-etl

Star3k

Python scripts for ETL (extract, transform and load) jobs for Ethereum blocks, transactions, ERC20 / ERC721 tokens, transfers, receipts, logs, contracts, internal transactions. Data is available in Google BigQueryhttps://goo.gl/oY5BCQ

export bigquery aws csv sql etl ethereum transaction gcp google-cloud erc20 erc20-tokens blockchain-analytics erc721

UpdatedApr 30, 2025
Python

apache /incubator-devlake

Star2.8k

Apache DevLake is an open-source dev data platform to ingest, analyze, and visualize the fragmented data from DevOps tools, extracting insights for engineering excellence, developer experience, and community growth.

golang open-source devops data jira integration etl data-engineering data-analysis data-integration user-friendly hacktoberfest dora domain-layer data-transfers dashboard-friendly