Apache Spark

Apache Spark is an open source distributed general-purpose cluster-computing framework. It provides an interface for programming entire clusters with implicit data parallelism and fault tolerance.

Here are 9,120 public repositories matching this topic...

Language:All

Filter by language

All9,120 Scala1,930 Python1,907 Jupyter Notebook1,880 Java1,186 Shell342 HTML253 JavaScript182 Dockerfile126 R56 TypeScript54

Sort:Most stars

Sort options

Most stars Fewest stars Most forks Fewest forks Recently updated Least recently updated

apache /spark

Star41.5k

Apache Spark - A unified analytics engine for large-scale data processing

python java r scala sql big-data spark jdbc

UpdatedJul 18, 2025
Scala

DataTalksClub /data-engineering-zoomcamp

Star31.9k

Data Engineering Zoomcamp is a free nine-week course that covers the fundamentals of data engineering.

docker kafka spark data-engineering dbt kestra

UpdatedJul 17, 2025
Jupyter Notebook

donnemartin /data-science-ipython-notebooks

Star28.4k

Data science Python notebooks: Deep learning (TensorFlow, Theano, Caffe, Keras), scikit-learn, Kaggle, big data (Spark, Hadoop MapReduce, HDFS), matplotlib, pandas, NumPy, SciPy, Python essentials, AWS, and various command lines.

python aws data-science machine-learning caffe theano big-data spark deep-learning hadoop tensorflow numpy scikit-learn keras pandas kaggle scipy matplotlib mapreduce

UpdatedMar 20, 2024
Python

getredash /redash

Star27.5k

Make Your Company Data Driven. Connect to any data source, easily visualize, dashboard and share your data.

visualization javascript mysql python bigquery bi spark dashboard athena analytics postgresql business-intelligence redash redshift databricks hacktoberfest spark-sql

UpdatedJul 17, 2025
Python

yeasy /docker_practice

Star25.5k

Learn and understand Docker&Container technologies, with real DevOps practice!

linux docker kubernetes devops spark book container mesos swarm cloud-computing

UpdatedDec 26, 2024
Go

heibaiying /BigData-Notes

Star16.5k

大数据入门指南 ⭐

phoenix scala kafka big-data spark yarn hive hadoop storm bigdata hbase zookeeper hdfs mapreduce flume azkaban sqoop

UpdatedJan 5, 2024
Java

FavioVazquez /ds-cheatsheets

Star15.5k

List of Data Science Cheatsheets to rule the world

python r programming spark jupyter datascience cheatsheet

UpdatedJul 18, 2024

GaiZhenbiao /ChuanhuChatGPT

Star15.4k

GUI for ChatGPT API and many LLMs. Supports agents, file-based QA, GPT finetuning and query with web search. All with a neat UI.

spark chatbot gemini llama minimax moss gemma claude ernie midjourney chatgpt-api chatglm stablelm ollama qwen dalle3 inspurai

UpdatedMar 13, 2025
Python

zhisheng17 /flink-learning

Star14.9k

flink learning blog.http://www.54tianzhisheng.cn/ 含 Flink 入门、概念、原理、实战、性能调优、源码解析等内容。涉及 Flink Connector、Metrics、Library、DataStream API、Table API & SQL 等内容的学习案例，还有 Flink 落地应用的大型项目案例（PVUV、日志存储、百亿数据实时去重、监控告警）分享。欢迎大家支持我的专栏《大数据实时计算引擎 Flink 实战与性能优化》

mysql redis elasticsearch streaming kafka spark influxdb rabbitmq clickhouse hbase stream-processing opentsdb loki flink rocketmq

UpdatedMar 12, 2025
Java

horovod /horovod

Star14.5k

Distributed training framework for TensorFlow, Keras, PyTorch, and Apache MXNet.

machine-learning spark deep-learning uber mxnet tensorflow mpi keras pytorch machinelearning baidu deeplearning ray

UpdatedJul 1, 2025
Python

aalansehaiyang /technology-talk

Star14.5k

【大厂面试专栏】一份Java程序员需要的技术指南，这里有面试题、系统架构、职场锦囊、主流中间件等，让你成为更牛的自己！

git java kafka spark spring es6 hbase springboot dubbo mycat

UpdatedOct 28, 2023

deeplearning4j /deeplearning4j

Star14k

Suite of tools for deploying and training deep learning models using the JVM. Highlights include model import for keras, tensorflow, and onnx/pytorch, a modular and tiny c++ library for running math code and a java based math library on top of the core c++ library. Also includes samediff: a pytorch/tensorflow like library for running deep learn...

python java clojure scala spark hadoop gpu intellij linear-algebra artificial-intelligence deeplearning neural-nets dl4j matrix-library deeplearning4j

UpdatedJul 18, 2025
Java

apache /doris

Star14k

Apache Doris is an easy-to-use, high performance and unified analytics database.

agent bigquery real-time sql database ai spark snowflake olap query-engine redshift dbt elt iceberg hudi delta-lake lakehouse paimon

UpdatedJul 18, 2025
Groovy

wangzhiwubigdata /God-Of-BigData

Star10.2k

专注大数据学习面试，大数据成神之路开启。Flink/Spark/Hadoop/Hbase/Hive...

kafka spark hive hadoop bigdata hbase zookeeper hdfs flume flink azkaban

UpdatedAug 7, 2023

mage-ai /mage-ai

Star8.4k

🧙 Build, run, and manage data pipelines for integrating and transforming data.

python data-science data machine-learning sql spark pipeline etl pipelines orchestration artificial-intelligence data-engineering data-integration dbt elt transformation data-pipelines reverse-etl

UpdatedJul 18, 2025
Python

delta-io /delta

Star8.2k

An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs

big-data spark analytics acid delta-lake

UpdatedJul 18, 2025
Scala

tobymao /sqlglot

Star8k

Python SQL Parser and Transpiler

mysql python bigquery parser postgres sql spark presto hive clickhouse sqlite snowflake optimizer transpiler redshift databricks tsql trino sqlparser duckdb

UpdatedJul 18, 2025
Python

h2oai /h2o-3

Star7.2k

H2O is an Open Source, Distributed, Fast & Scalable Machine Learning Platform: Deep Learning, Gradient Boosting (GBM) & XGBoost, Random Forest, Generalized Linear Modeling (GLM with Elastic Net), K-Means, PCA, Generalized Additive Models (GAM), RuleFit, Support Vector Machine (SVM), Stacked Ensembles, Automatic Machine Learning (AutoML), etc.