hdfs

Star

Here are 1,051 public repositories matching this topic...

Language:All

Filter by language

All1,051 Java253 Python202 Jupyter Notebook113 Shell99 Scala77 Go26 JavaScript24 HTML16 C++15 Dockerfile13

Sort:Most stars

Sort options

Most stars Fewest stars Most forks Fewest forks Recently updated Least recently updated

seaweedfs /seaweedfs

Star28.9k

SeaweedFS is a fast distributed storage system for blobs, objects, files, and data lake, for billions of files! Blob store has O(1) disk seek, cloud tiering. Filer supports Cloud Drive, xDC replication, Kubernetes, POSIX FUSE mount, S3 API, S3 Gateway, Hadoop, WebDAV, encryption, Erasure Coding. Enterprise version is at seaweedfs.com.

kubernetes distributed-systems fuse replication cloud-drive s3 posix s3-storage hdfs distributed-storage distributed-file-system erasure-coding object-storage blob-storage seaweedfs hadoop-hdfs tiered-file-system

UpdatedDec 18, 2025
Go

heibaiying /BigData-Notes

Star16.8k

大数据入门指南 ⭐

phoenix scala kafka big-data spark yarn hive hadoop storm bigdata hbase zookeeper hdfs mapreduce flume azkaban sqoop

UpdatedJan 5, 2024
Java

ceph /ceph

Star16k

Ceph is a distributed object, block, and file storage platform

kubernetes fuse replication storage high-performance smb s3 cloud-storage nfs posix object-store hdfs block-storage distributed-storage iscsi software-defined-storage distributed-file-system erasure-coding highly-available nvme-over-fabrics

UpdatedDec 17, 2025
C++

juicedata /juicefs

Star12.5k

JuiceFS is a distributed POSIX file system built on top of Redis and S3.

go redis golang distributed-systems storage filesystem bigdata s3 posix hdfs cloud-native object-storage

UpdatedDec 17, 2025
Go

wangzhiwubigdata /God-Of-BigData

Star10.4k

专注大数据学习面试，大数据成神之路开启。Flink/Spark/Hadoop/Hbase/Hive...

kafka spark hive hadoop bigdata hbase zookeeper hdfs flume flink azkaban

UpdatedAug 7, 2023

piskvorky /smart_open

Sponsor

Star3.4k

Utils for streaming large files (S3, HDFS, gzip, bz2...)

python streaming s3 file hdfs hacktoberfest webhdfs boto gzip-stream bz2 streaming-data

UpdatedDec 1, 2025
Python

TileDB-Inc /TileDB

Star2k

The Universal Storage Engine

data-science storage-engine s3 sparse-data scientific-computing s3-storage arrays hdfs data-analysis dataframes tiledb dense-data sparse-arrays

UpdatedDec 17, 2025
C++

collabH /bigdata-growth

Star1.7k

大数据知识仓库涉及到数据仓库建模、实时计算、大数据、数据中台、系统设计、Java、算法等。

kafka spark hive hadoop bigdata kudu hbase olap hdfs mapreduce flink debezium bigdatalearning hudi

UpdatedAug 20, 2025
Shell

water8394 /BigData-Interview

Star1.6k

🎯 🌟[大数据面试题]分享自己在网络上收集的大数据相关的面试题以及自己的答案总结.目前包含Hadoop/Hive/Spark/Flink/Hbase/Kafka/Zookeeper框架的面试题知识总结

kafka spark yarn hadoop bigdata interview hbase hdfs mapreduce interview-questions flink

UpdatedAug 30, 2021

colinmarc /hdfs

Star1.4k

A native go client for HDFS

go commandline hdfs

UpdatedJan 22, 2025
Go

wgzhao /Addax

Star1.4k

A fast and versatile ETL tool that can transfer data between RDBMS and NoSQL seamlessly

mysql database influxdb hive hadoop etl clickhouse excel kudu impala postgresql oracle hdfs sqlserver trino

UpdatedDec 12, 2025
Java

spotify /snakebite

Star859

A pure python HDFS client

python hdfs python-hdfs-client

UpdatedApr 19, 2022
Python

HariSekhon /DevOps-Python-tools

Star808

80+ DevOps & Data CLI Tools - AWS, GCP, GCF Python Cloud Functions, Log Anonymizer, Spark, Hadoop, HBase, Hive, Impala, Linux, Docker, Spark Data Converters & Validators (Avro/Parquet/JSON/CSV/INI/XML/YAML), Travis CI, AWS CloudFormation, Elasticsearch, Solr etc.

python linux docker aws elasticsearch devops json cloudformation spark hadoop avro travis-ci solr gcp hbase pyspark hdfs parquet dockerhub gcf

UpdatedNov 6, 2025
Python

sunnyandgood /BigData

Star682

💎🔥大数据学习笔记

mysql linux shell hive hadoop hbase zookeeper hdfs mapreduce flume sqoop

UpdatedMay 13, 2019
Java

Stratio /sparta

Star529

Real Time Analytics and Data Pipelines based on Spark Streaming

workflow lambda streaming real-time scala kafka spark analytics spark-streaming olap hdfs sparksql triggers streaming-data stratio sparta stratio-sparta acctbl-compae

UpdatedOct 24, 2019
Scala

lensesio /kafka-connect-ui

Star516

Deprecated - See Lenses.io Community Edition

redis mqtt elasticsearch twitter kafka cassandra influxdb jms s3 documentdb hdfs kafka-connect

UpdatedMay 7, 2025
JavaScript

dromara /CloudEon

Sponsor

Star488

CloudEon uses Kubernetes to install and deploy open-source big data components, enabling the containerized operation of an open-source big data platform. This allows you to reduce your focus on underlying resource management and maintenance.

kubernetes yarn hadoop bigdata hdfs cloudnative doris