airflow-dag

This project demonstrates how to build and automate an ETL pipeline written in Python and schedule it using open source Apache Airflow orchestration tool on AWS EC2 instance.

aws data airflow automation database virtual-machine ssh-server visual-studio-code python3 artificial-intelligence data-engineering aws-ec2 elt transformation etl-pipeline dags airflow-dag reverse-etl

UpdatedMar 14, 2025
Python

nathadriele /airflow-kpi-insertion-pipeline

Star7

The script automates the collection and insertion of KPIs related to transaction time and storage usage in a Data Warehouse, using Apache Airflow. It calculates the time elapsed since the last transaction and the percentage of storage usage, recording this data periodically in specific tables.

python-script data-warehouse data-engineering data-ingestion etl-pipeline airflow-dag transaction-metrics storage-usage data-kpi-scheduler kpi-data-pipeline

UpdatedFeb 3, 2025
Python

angadsingh /airflow-ditto

Star5

An airflow DAG transformation framework

emr aws airflow framework spark yarn graph-algorithms azure networkx extensible dataflow isomorphism livy hdinsight graph-manipulation airflow-dag

UpdatedJul 10, 2020
Python

vicmar57 /COVID-19-Live-Tweet-Analyzer-Kafka-Spark-Cassandra-and-Airflow

Star2

Analysing live tweets from twitter by generating a big data pipeline and scheduling it with Airflow (Using also Kafka for tweet ingestion, Cassandra for storing parsed tweets, and Spark for Analysis)

airflow kafka spark python3 python-3 cassandra-cql cassandra-database airflow-dag

UpdatedAug 11, 2020
Python

minhky2185 /incident_data_warehouse

Star2

Build a data warehouse from scratch, including full load, daily incremental load, design schema, SCD Type 1 and 2.

python docker airflow s3 orchestration data-warehouse data-engineering redshift powerbi airflow-plugin staging-area star-schema etl-pipeline etl-automation scd-type-2 airflow-dag bridge-table incident-data-warehouse scd-type-1

UpdatedFeb 1, 2023
Python

judeleonard /e-commerce_activity_tracking

Star2

This is an ELT data pipeline setup to track the activities of an e-commerce website based on orders, reviews, deliveries and shipment date. This project utilized technologies like Airflow, AWS RDS-Postgres, Python etc.

postgres airflow e-commerce datapipeline airflow-docker dataengineering airflow-dag elt-pipeline

UpdatedJul 5, 2024
Python

k0rsakov /dag_factory

Star2

Фабрика DAG

data airflow data-engineering apache-airflow airflow-dags airflow-dag dag-factory data-engineering-project

UpdatedAug 15, 2024
Python

IgnacioNMiranda /trigger-dag-run-params-demo

Star1

Apache Airflow demo project that setup 3 DAGs to explain how to pass parameters from a DAG to a triggered DAG.

python python3 apache-airflow airflow-dags airflow-operator airflow-operators airflow-dag

UpdatedJan 7, 2023
Python

djamier /docker-airflow

Star1

Orchestrate data pipeline using airflow

python airflow docker-airflow airflow-docker airflow-dag

UpdatedJul 12, 2024
Python

pratiksha2712 /ETL-Pipeline

Star0

An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.

python airflow spark apache-spark scheduler s3 data-engineering data-lake warehouse redshift data-migration livy etl-framework etl-pipeline airflow-dag

UpdatedJan 8, 2024
Python

torshin5ergey /1tdata-do_2024-08-21_6.8

Star0

1T Data "Data architect (DevOps)". Задание 2024-08-21 6.8

python apache-airflow airflow-dag

UpdatedAug 27, 2024
Python

wlopezm-unal /reddit_project_airflow_aws

Star0

This project focuses on implementing an ETL pipeline using Apache Airflow to efficiently extract data from Reddit, transform it as needed, and load it into an AWS S3 bucket. The use of Airflow allows for robust orchestration of the data workflow, ensuring that each step of the ETL process is executed in a reliable and repeatable manner.

python git aws airflow docker-compose terraform airflow-docker airflow-dag