datalake

Star

Here are 263 public repositories matching this topic...

Language:All

Filter by language

All263 Python77 Jupyter Notebook52 Java20 HCL10 JavaScript8 C#6 TSQL6 Rust5 Dockerfile4 Scala4

Sort:Most stars

Sort options

Most stars Fewest stars Most forks Fewest forks Recently updated Least recently updated

sinaptik-ai /pandas-ai

Star18.1k

Chat with your database or your datalake (SQL, CSV, parquet). PandasAI makes data analysis conversational using LLMs and RAG.

data-science data csv sql database ai pandas data-visualization data-analysis datalake text-to-sql gpt-4 llm

UpdatedMar 16, 2025
Python

trinodb /trino

Star11k

Official repository of Trino, the distributed SQL query engine for big data, formerly known as PrestoSQL (https://trino.io)

java distributed-systems data-science sql database big-data presto hive hadoop analytics jdbc databases distributed-database query-engine iceberg datalake prestodb trino delta-lake

UpdatedMar 17, 2025
Java

The world's fastest open query engine for sub-second analytics both on and off the data lakehouse. With the flexibility to support nearly any scenario, StarRocks provides best-in-class performance for multi-dimensional analytics, real-time analytics, and ad-hoc queries. A Linux Foundation project.

sql database big-data analytics olap join distributed-database realtime-database mpp cloudnative iceberg real-time-analytics datalake vectorized real-time-updates star-schema hudi delta-lake lakehouse lakehouse-platform

UpdatedMar 17, 2025
Java

activeloopai /deeplake

Star8.5k

Database for AI. Store Vectors, Images, Texts, Videos, etc. Use with LLMs/LangChain. Store, query, version, & visualize any AI data. Stream data in real-time to PyTorch/TensorFlow.https://activeloop.ai

python data-science machine-learning ai computer-vision deep-learning tensorflow cv image-processing ml pytorch datasets multi-modal datalake mlops vector-search vector-database large-language-models llm langchain

UpdatedMar 11, 2025
Python

paradedb /paradedb

Sponsor

Star6.8k

Postgres for Search and Analytics

elasticsearch sql database big-data analytics postgresql olap full-text-search mpp iceberg bm25 object-storage real-time-analytics datalake aggregations similarity-search htap sparse-vector hybrid-search lakehouse-platform

UpdatedMar 16, 2025
Rust

apache /hudi

Star5.7k

Upserts, Deletes And Incremental Processing on Big Data.

bigdata stream-processing data-integration datalake apachespark hudi apachehudi incremental-processing apacheflink

UpdatedMar 16, 2025
Java

treeverse /lakeFS

Star4.6k

lakeFS - Data version control for your data lake | Git for data

go golang apache-spark aws-s3 google-cloud-storage data-engineering data-lake azure-storage data-version-control object-storage datalake hadoop-filesystem data-quality data-versioning azure-blob-storage apache-sparksql git-for-data lakefs datalakes

UpdatedMar 16, 2025
Go

DataLinkDC /dinky

Star3.3k

Dinky is a real-time data development platform based on Apache Flink, enabling agile data development, deployment and operation.

sql olap flink datawarehouse datalake flinksql flinkcdc real-time-computing-platform

UpdatedMar 14, 2025
Java

lakesoul-io /LakeSoul

Star2.6k

LakeSoul is an end-to-end, realtime and cloud native Lakehouse framework with fast data ingestion, concurrent update and incremental data analytics on cloud storages for both BI and AI applications.

python rust streaming sql big-data spark arrow postgresql pytorch flink datalake vectorized velox huggingface datafusion lakehouse lakesoul

UpdatedMar 13, 2025
Java

leo-project /leofs

Star1.6k

The LeoFS Storage System

erlang s3 nfs s3-storage distributed-storage distributed-file-system leofs nfs-server datalake

UpdatedJun 2, 2020
Erlang

apache /gravitino

Star1.4k

World's most powerful open data catalog for building a high-performance, geo-distributed and federated metadata lake.

metadata data-catalog datalake stratosphere federated-query lakehouse model-catalog metalake skycomputing ai-catalog opendatacatalog

UpdatedMar 14, 2025
Java

zinggAI /zingg

Star986

Scalable identity resolution, entity resolution, data mastering and deduplication using ML

identity data-science identity-resolution spark etl analytics dedupe entity-resolution data-transformation ml fuzzy-matching deduplication datalake masterdata dataengineering fuzzymatch dataquality analytics-engineering data-transformations modern-data-stack

UpdatedMar 16, 2025
Java

apache /amoro

Star936

Apache Amoro (incubating) is a Lakehouse management system built on open data lake formats.

bigdata datalake lakehouse

UpdatedMar 17, 2025
Java

leesf /hudi-resources

Star549

汇总Apache Hudi相关资料

bigdata apache stream-processing data-integration datalake hudi apachehudi incremental-processing hudi-resources

UpdatedMar 9, 2025

Datavault-UK /automate-dv

Star530

A free to use dbt package for creating and loading Data Vault 2.0 compliant Data Warehouses (powered by dbt, an open source data engineering tool, registered trademark of dbt Labs)

metadata sql etl snowflake datawarehousing dbt elt datawarehouse datalake dataengineering datavault datavault20 data-vault