data-pipelines

Convert documents to structured data effortlessly. Unstructured is open-source ETL solution for transforming complex documents into clean, structured formats for language models. Visit our website to learn more about our enterprise grade Platform product for production grade workflows, partitioning, enrichments, chunking and embedding.

nlp pdf machine-learning natural-language-processing information-retrieval ocr deep-learning ml docx preprocessing pdf-to-text data-pipelines donut document-image-processing document-parser pdf-to-json document-image-analysis llm document-parsing langchain

UpdatedSep 26, 2025
HTML

mage-ai /mage-ai

Star8.5k

🧙 Build, run, and manage data pipelines for integrating and transforming data.

python data-science data machine-learning sql spark pipeline etl pipelines orchestration artificial-intelligence data-engineering data-integration dbt elt transformation data-pipelines reverse-etl

UpdatedOct 7, 2025
Python

infinyon /fluvio

Star5k

🦀 event stream processing for developers to collect and transform data in motion to power responsive data intensive applications.

rust distributed-systems streaming real-time serverless webassembly data-flow stream-processing data-analytics data-integration cloud-native data-pipelines stateful streaming-data stream-processing-engine event-driven-architecture streaming-analytics streaming-data-processing streaming-data-pipelines

UpdatedOct 1, 2025
Rust

StructuredLabs /preswald

Star4.3k

Preswald is a WASM packager for Python-based interactive data apps: bundle full complex data workflows, particularly visualizations, into single files, runnable completely in-browser, using Pyodide, DuckDB, Pandas, and Plotly, Matplotlib, etc. Build dashboards, reports, and notebooks that run offline, load fast, and share like a document.

python open-source data ai analytics vscode data-visualization gpt copilot data-pipelines schema-management data-infrastructure analytics-engineering data-sdk llm data-applications

UpdatedJul 17, 2025
Python

orchest /orchest

Star4.1k

Build data pipelines, the easy way 🛠️

python docker kubernetes data-science machine-learning airflow cloud deployment jupyter etl ide pipelines self-hosted jupyterlab notebooks data-pipelines dag etl-pipeline orchest

UpdatedJun 6, 2023
TypeScript

Netflix /maestro

Star3.6k

Maestro: Netflix’s Workflow Orchestrator

java workflow data-science machine-learning automation etl analytics workflow-engine scheduler orchestration data-engineering maestro elt data-pipelines batch-processing dag mlops data-ops workflow-orchestration data-orchestrator

UpdatedOct 4, 2025
Java

ucbepic /docetl

Star3k

A system for agentic LLM-powered data processing and ETL

python workflow data etl semantic-data elt data-pipelines agents document-analysis document-processing unstructured-data unstructured-data-analysis llm

UpdatedOct 7, 2025
Python

meltano /meltano

Star2.2k

Meltano: the declarative code-first data integration engine that powers your wildest data and ML-powered product ideas. Say goodbye to writing, maintaining, and scaling your own API integrations.

open-source tap data opensource integration pipelines data-engineering target dataops loaders elt extract-data data-pipelines singer connectors targets taps meltano dataops-platform meltano-sdk

UpdatedOct 8, 2025
Python

elementary-data /elementary

Star2.2k

The dbt-native data observability solution for data & analytics engineers. Monitor your data pipelines in minutes. Available as self-hosted or cloud service with premium features.

bigquery snowflake data-warehouse dataops data-analysis redshift dbt data-pipelines data-pipeline lineage data-governance data-lineage analytics-engineer dbt-packages data-observability data-reliability dbt-artifacts