deltalake

Star

Here are 63 public repositories matching this topic...

Language:All

Filter by language

All63 Python23 Jupyter Notebook13 Rust6 Scala3 Dockerfile2 Java2 TSQL2 CSS1 Go1 HCL1

Sort:Most stars

Sort options

Most stars Fewest stars Most forks Fewest forks Recently updated Least recently updated

paradedb /pg_analytics

Sponsor

Star523

DuckDB-powered data lake analytics from Postgres

postgres sql database big-data analytics arrow postgresql olap parquet iceberg object-storage datalake realtime-analytics columnar deltalake datafusion lakehouse duckdb lakehouse-platform paradedb

UpdatedMar 19, 2025
Rust

Generate relevant synthetic data quickly for your projects. The Databricks Labs synthetic data generator (aka `dbldatagen`) may be used to generate large simulated / synthetic data sets for test, POCs, and other uses in Databricks environments including in Delta Live Tables pipelines

python spark faker pyspark spark-streaming data-generation databricks synthetic-data datagen datagenerator deltalake datageneration delta-live-tables

UpdatedMar 7, 2025
Python

delta-io /kafka-delta-ingest

Star397

A highly efficient daemon for streaming data from Kafka into Delta Lake

rust kafka delta deltalake

UpdatedApr 20, 2025
Rust

MrPowers /mack

Star322

Delta Lake helper methods in PySpark

pyspark deltalake

UpdatedSep 5, 2024
Python

japila-books /delta-lake-internals

Star183

The Internals of Delta Lake

books book internals datalake delta-lake deltalake

UpdatedJan 12, 2025

smart-data-lake /smart-data-lake

Star121

Smart Automation Tool for building modern Data Lakes and Data Pipelines

scala spark hive hadoop transform-data data-lake data-pipelines deltalake smart-data-lake

UpdatedApr 25, 2025
Scala

uname-n /deltabase

Star121

a lightweight, comprehensive solution for managing delta tables built on polars and deltalake

sql database deltalake delta-tables polars

UpdatedJan 1, 2025
Python

izhangzhihao /Real-time-Data-Warehouse

Star113

Real-time Data Warehouse with Apache Flink & Apache Kafka & Apache Hudi

elasticsearch sql kafka spark hoodie data-warehouse delta flink cdc change-data-capture iceberg datalake debezium spark-sql data-warehousing hudi delta-lake deltalake flink-sql real-time-data-warehouse

UpdatedDec 15, 2023
Dockerfile

WeBankFinTech /Streamis

Star107

Streaming application development and management system, based on Linkis and DSS, planning to provide the workflow-like graphical drag-and-drop development capability.

streaming kafka warehouse flink iceberg datalake hudi deltalake linkis dataspherestudio wedatasphere streamis

UpdatedApr 25, 2025
Java

martandsingh /ApacheSpark

Star98

This repository will help you to learn about databricks concept with the help of examples. It will include all the important topics which we need in our real life experience as a data engineer. We will be using pyspark & sparksql for the development. At the end of the course we also cover few case studies.

sql database spark hive hadoop etl pyspark data-engineering spark-streaming data-analysis databricks datalake spark-sql timetravel apachespark etl-pipeline deltalake

UpdatedJul 28, 2024
Python

anneglienke /101_upsert-delta

Star89

This repository exemplifies a simple ELT process using delta to perform upsert and remove data files that aren't in the latest state of the transaction log for the table.

delta delta-lake deltalake