data-engineering

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.

mysql python java bigquery data pipeline etl postgresql s3 snowflake self-hosted data-engineering data-analysis mssql data-integration data-collection redshift elt data-pipeline change-data-capture

UpdatedJul 18, 2025
Python

Avaiga /taipy

Star18.2k

Turns Data and AI algorithms into production-ready web applications in no time.

python workflow automation pipeline job-scheduler pipelines data-visualization orchestration datascience data-engineering developer-tools data-integration scenario hacktoberfest mlops data-ops scenario-analysis hacktoberfest2023 taipy-core taipy-gui

UpdatedJun 29, 2025
Python

argoproj /argo-workflows

Star15.8k

Workflow Engine for Kubernetes

kubernetes workflow machine-learning airflow workflow-engine cncf pipelines data-engineering argo k8s cloud-native batch-processing hacktoberfest dag gitops mlops knative argo-workflows

UpdatedJul 18, 2025
Go

andkret /Cookbook

Star14.4k

The Data Engineering Cookbook

big-data best-practices cookbook data-engineering data-engineer

UpdatedJun 11, 2025
Python

dagster-io /dagster

Star13.6k

An orchestration platform for the development, production, and observation of data assets.

python metadata workflow data-science etl analytics scheduler orchestration data-engineering data-integration data-pipelines workflow-automation mlops dagster data-orchestrator

UpdatedJul 18, 2025
Python

datastacktv /data-engineer-roadmap

Star12.7k

Roadmap to becoming a data engineer in 2021

roadmap cloud data-engineering data-engineer-roadmap

UpdatedJan 25, 2022

great-expectations /great_expectations

Star10.6k

Always know what to expect from your data.

data-science pipeline exploratory-data-analysis eda data-engineering data-quality data-profiling datacleaner exploratory-analysis cleandata dataquality datacleaning mlops pipeline-tests pipeline-testing dataunittest data-unit-tests exploratorydataanalysis pipeline-debt data-profilers

UpdatedJul 18, 2025
Python

xonsh /xonsh

Sponsor

Star8.9k

🐚 Python-powered shell. Full-featured and cross-platform.

python shell bash zsh cli console fish raspberry-pi devops data-science command-line iterm2 artificial-intelligence data-engineering xonsh security-automation

UpdatedJul 15, 2025
Python

mage-ai /mage-ai

Star8.4k

🧙 Build, run, and manage data pipelines for integrating and transforming data.

python data-science data machine-learning sql spark pipeline etl pipelines orchestration artificial-intelligence data-engineering data-integration dbt elt transformation data-pipelines reverse-etl

UpdatedJul 18, 2025
Python

redpanda-data /connect

Star8.4k

Fancy stream processing made operationally mundane

go golang kafka cqrs etl rabbitmq amqp logs message-bus event-sourcing nats stream-processing message-queue data-engineering streaming-data stream-processor data-ops

UpdatedJul 18, 2025
Go

risingwavelabs /risingwave

Star8.1k

Stream processing and management platform.

rust database kafka postgresql stream-processing data-engineering materialized-view apache-iceberg

UpdatedJul 18, 2025
Rust

growthbook /growthbook

Star6.7k

Open Source Feature Flagging and A/B Testing Platform

bigquery data-science statistics analytics continuous-delivery clickhouse mixpanel snowflake feature-flags data-engineering experimentation data-analysis ab-testing redshift abtesting abtest split-testing remote-config feature-flagging

UpdatedJul 18, 2025
TypeScript

feast-dev /feast

Star6.2k

The Open Source Feature Store for AI/ML

python data-science machine-learning big-data ml data-engineering features data-quality mlops feature-store

UpdatedJul 18, 2025
Python

cloudquery /cloudquery

Star6.1k

The developer first cloud governance platform

go kubernetes github-api bigquery aws data google sql etl azure gcp data-engineering data-analysis data-integration data-collection elt etl-framework cspm airbyte attack-surface-management

UpdatedJul 18, 2025
Go

Improve this page

Add a description, image, and links to thedata-engineering topic page so that developers can more easily learn about it.

Curate this topic

Add this topic to your repo

To associate your repository with thedata-engineering topic, visit your repo's landing page and select "manage topics."

Learn more

Movatterモバイル変換

Navigation Menu

Search code, repositories, users, issues, pull requests...

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

data-engineering

Here are 4,696 public repositories matching this topic...

apache /superset

apache /airflow

GokuMohandas /Made-With-ML

DataTalksClub /data-engineering-zoomcamp

eugeneyan /applied-ml

PrefectHQ /prefect

airbytehq /airbyte