Apache Spark

Apache Spark is an open source distributed general-purpose cluster-computing framework. It provides an interface for programming entire clusters with implicit data parallelism and fault tolerance.

Here are 9,439 public repositories matching this topic...

Language:All

Filter by language

All9,439 Python2,066 Jupyter Notebook1,967 Scala1,945 Java1,193 Shell345 HTML260 JavaScript184 Dockerfile126 TypeScript64 R57

Sort:Most stars

Sort options

Most stars Fewest stars Most forks Fewest forks Recently updated Least recently updated

apache /spark

Star42.5k

Apache Spark - A unified analytics engine for large-scale data processing

python java r scala sql big-data spark jdbc

UpdatedDec 18, 2025
Scala

DataTalksClub /data-engineering-zoomcamp

Star33.9k

Data Engineering Zoomcamp is a free 9-week course on building production-ready data pipelines. The next cohort starts in January 2026. Join the course here 👇🏼

docker course kafka spark data-engineering free dbt kestra

UpdatedDec 16, 2025
Jupyter Notebook

donnemartin /data-science-ipython-notebooks

Star28.8k

Data science Python notebooks: Deep learning (TensorFlow, Theano, Caffe, Keras), scikit-learn, Kaggle, big data (Spark, Hadoop MapReduce, HDFS), matplotlib, pandas, NumPy, SciPy, Python essentials, AWS, and various command lines.

python aws data-science machine-learning caffe theano big-data spark deep-learning hadoop tensorflow numpy scikit-learn keras pandas kaggle scipy matplotlib mapreduce

UpdatedMar 20, 2024
Python

getredash /redash

Star28.1k

Make Your Company Data Driven. Connect to any data source, easily visualize, dashboard and share your data.

visualization javascript mysql python bigquery bi spark dashboard athena analytics postgresql business-intelligence redash redshift databricks hacktoberfest spark-sql

UpdatedDec 18, 2025
Python

yeasy /docker_practice

Star25.7k

Learn and understand Docker&Container technologies, with real DevOps practice!

linux docker kubernetes devops spark book container mesos swarm cloud-computing

UpdatedDec 17, 2025
Go

heibaiying /BigData-Notes

Star16.8k

大数据入门指南 ⭐

phoenix scala kafka big-data spark yarn hive hadoop storm bigdata hbase zookeeper hdfs mapreduce flume azkaban sqoop

UpdatedJan 5, 2024
Java

FavioVazquez /ds-cheatsheets

Star15.9k

List of Data Science Cheatsheets to rule the world

python r programming spark jupyter datascience cheatsheet

UpdatedJul 18, 2024

GaiZhenbiao /ChuanhuChatGPT

Star15.4k

GUI for ChatGPT API and many LLMs. Supports agents, file-based QA, GPT finetuning and query with web search. All with a neat UI.

spark chatbot gemini llama minimax moss gemma claude ernie midjourney chatgpt-api chatglm stablelm ollama qwen dalle3 inspurai

UpdatedAug 15, 2025
Python

zhisheng17 /flink-learning

Star15k

flink learning blog.http://www.54tianzhisheng.cn/ 含 Flink 入门、概念、原理、实战、性能调优、源码解析等内容。涉及 Flink Connector、Metrics、Library、DataStream API、Table API & SQL 等内容的学习案例，还有 Flink 落地应用的大型项目案例（PVUV、日志存储、百亿数据实时去重、监控告警）分享。欢迎大家支持我的专栏《大数据实时计算引擎 Flink 实战与性能优化》

mysql redis elasticsearch streaming kafka spark influxdb rabbitmq clickhouse hbase stream-processing opentsdb loki flink rocketmq

UpdatedMar 12, 2025
Java

apache /doris

Star14.7k

Apache Doris is an easy-to-use, high performance and unified analytics database.

agent bigquery real-time sql database ai spark snowflake olap query-engine redshift dbt elt iceberg hudi delta-lake lakehouse paimon

UpdatedDec 18, 2025
Java

aalansehaiyang /technology-talk

Star14.7k

【大厂面试专栏】一份Java程序员需要的技术指南，这里有面试题、系统架构、职场锦囊、主流中间件等，让你成为更牛的自己！

git java kafka spark spring es6 hbase springboot dubbo mycat

UpdatedJul 21, 2025

horovod /horovod

Star14.6k

Distributed training framework for TensorFlow, Keras, PyTorch, and Apache MXNet.

machine-learning spark deep-learning uber mxnet tensorflow mpi keras pytorch machinelearning baidu deeplearning ray

UpdatedDec 1, 2025
Python

deeplearning4j /deeplearning4j

Star14.2k

Suite of tools for deploying and training deep learning models using the JVM. Highlights include model import for keras, tensorflow, and onnx/pytorch, a modular and tiny c++ library for running math code and a java based math library on top of the core c++ library. Also includes samediff: a pytorch/tensorflow like library for running deep learn...

python java clojure scala spark hadoop gpu intellij linear-algebra artificial-intelligence deeplearning neural-nets dl4j matrix-library deeplearning4j

UpdatedDec 17, 2025
Java

wangzhiwubigdata /God-Of-BigData

Star10.4k

专注大数据学习面试，大数据成神之路开启。Flink/Spark/Hadoop/Hbase/Hive...

kafka spark hive hadoop bigdata hbase zookeeper hdfs flume flink azkaban

UpdatedAug 7, 2023

tobymao /sqlglot

Star8.7k

Python SQL Parser and Transpiler

mysql python bigquery parser postgres sql spark presto hive clickhouse sqlite snowflake optimizer transpiler redshift databricks tsql trino sqlparser duckdb

UpdatedDec 18, 2025
Python

mage-ai /mage-ai

Star8.6k

🧙 Build, run, and manage data pipelines for integrating and transforming data.

python data-science data machine-learning sql spark pipeline etl pipelines orchestration artificial-intelligence data-engineering data-integration dbt elt transformation data-pipelines reverse-etl

UpdatedDec 17, 2025
Python

delta-io /delta

Star8.5k

An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs

big-data spark analytics acid delta-lake

UpdatedDec 18, 2025
Scala

h2oai /h2o-3

Star7.4k

H2O is an Open Source, Distributed, Fast & Scalable Machine Learning Platform: Deep Learning, Gradient Boosting (GBM) & XGBoost, Random Forest, Generalized Linear Modeling (GLM with Elastic Net), K-Means, PCA, Generalized Additive Models (GAM), RuleFit, Support Vector Machine (SVM), Stacked Ensembles, Automatic Machine Learning (AutoML), etc.