Apache Spark is an open source distributed general-purpose cluster-computing framework. It provides an interface for programming entire clusters with implicit data parallelism and fault tolerance.

Open source platform for the machine learning lifecycle

• UpdatedMar 18, 2025
• Python

Simple and Distributed Machine Learning

• UpdatedMar 12, 2025
• Scala

lakeFS - Data version control for your data lake | Git for data

• UpdatedMar 18, 2025
• Go

酷玩 Spark: Spark 源代码解析、Spark 类库等

• UpdatedMay 18, 2022
• Scala

Interactive and Reactive Data Science using Scala and Spark.

• UpdatedMay 16, 2023
• JavaScript

Kubernetes operator for managing the lifecycle of Apache Spark applications on Kubernetes.

• UpdatedMar 18, 2025
• Go

BigDL: Distributed TensorFlow, Keras and PyTorch on Apache Spark/Flink & Ray

• UpdatedJan 21, 2025
• Jupyter Notebook

Apache Spark docker image

• UpdatedApr 21, 2023
• Shell

.NET for Apache® Spark™ makes Apache Spark™ easily accessible to .NET developers.

• UpdatedFeb 19, 2025
• C#

Feathr – A scalable, unified data and AI engineering platform for enterprise

• UpdatedApr 4, 2024
• Scala

Oryx 2: Lambda architecture on Apache Spark, Apache Kafka for real-time large scale machine learning

• UpdatedAug 16, 2021
• Java

A curated list of awesome Apache Spark packages and resources.

• UpdatedOct 24, 2024
• Shell

The Internals of Apache Spark

• UpdatedSep 14, 2024

SQL data analysis & visualization projects using MySQL, PostgreSQL, SQLite, Tableau, Apache Spark and pySpark.

• UpdatedJul 18, 2022
• Jupyter Notebook

An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.

• UpdatedMar 9, 2020
• Python

This is the github repo for Learning Spark: Lightning-Fast Data Analytics [2nd Edition]

• UpdatedJan 28, 2025
• Scala

PySpark + Scikit-learn = Sparkit-learn

• UpdatedDec 31, 2020
• Python

(Deprecated) Scikit-learn integration package for Apache Spark

• UpdatedDec 3, 2019
• Python

MapReduce, Spark, Java, and Scala for Data Algorithms Book

• UpdatedOct 14, 2024
• Java

GraphFrames is a package for Apache Spark which provides DataFrame-based Graphs

• UpdatedMar 17, 2025
• Scala