Repository files navigation

This is a practical example of a data engineering project with real-estates. The connected blog post aboutBuilding a Data Engineering Project in 20 Minutes you can find on mywebsite. Topics are:

• Getting the Data – Scraping withBeautifulSoup
• Storing on S3-MinIO
• Custom Change Data Capture (CDC)
• Adding Database features to S3 –Delta Lake &Spark
• Machine Learning part –Jupyter Notebook
• Ingesting Data Warehouse for low latency –Apache Druid
• The UI with Dashboards and more –Apache Superset
• Orchestrating everything together –Dagster
• DevOps engine –Kubernetes

The Status of the project you findhere.

To get MinIO, Spark, Kubernetes, etc. ready, check the representive folder inhere.

1. MinIO started
2. Kubernetes ready
3. Spark image and role and namespaces ready
4. cdsrc/pipelines/real-estate and start dagit withdagit