Repository files navigation

The Kubernetes Operator for Apache Spark aims to make specifying and runningSpark applications as easy and idiomatic as running other workloads on Kubernetes. It usesKubernetes custom resources for specifying, running, and surfacing status of Spark applications.

For a more detailed guide, please refer to theGetting Started guide.

# Add the Helm repositoryhelm repo add spark-operator https://kubeflow.github.io/spark-operatorhelm repo update# Install the operator into the spark-operator namespace and wait for deployments to be readyhelm install spark-operator spark-operator/spark-operator \    --namespace spark-operator --create-namespace --wait# Create an example application in the default namespacekubectl apply -f https://raw.githubusercontent.com/kubeflow/spark-operator/refs/heads/master/examples/spark-pi.yaml# Get the status of the applicationkubectl get sparkapp spark-pi

For a complete reference of the custom resource definitions, please refer to theAPI Definition. For details on its design, please refer to theArchitecture. It requires Spark 2.3 and above that supports Kubernetes as a native scheduler backend.

The Kubernetes Operator for Apache Spark currently supports the following list of features:

• Supports Spark 2.3 and up.
• Enables declarative application specification and management of applications through custom resources.
• Automatically runsspark-submit on behalf of users for eachSparkApplication eligible for submission.
• Provides nativecron support for running scheduled applications.
• Supports customization of Spark pods beyond what Spark natively is able to do through the mutating admission webhook, e.g., mounting ConfigMaps and volumes, and setting pod affinity/anti-affinity.
• Supports automatic application re-submission for updatedSparkApplication objects with updated specification.
• Supports automatic application restart with a configurable restart policy.
• Supports automatic retries of failed submissions with optional linear back-off.
• Supports mounting local Hadoop configuration as a Kubernetes ConfigMap automatically viasparkctl.
• Supports automatically staging local application dependencies to Google Cloud Storage (GCS) viasparkctl.
• Supports collecting and exporting application-level metrics and driver/executor metrics to Prometheus.

Project status:beta

Current API version:v1beta2

If you are currently using thev1beta1 version of the APIs in your manifests, please update them to use thev1beta2 version by changingapiVersion: "sparkoperator.k8s.io/<version>" toapiVersion: "sparkoperator.k8s.io/v1beta2". You will also need to delete theprevious version of the CustomResourceDefinitions namedsparkapplications.sparkoperator.k8s.io andscheduledsparkapplications.sparkoperator.k8s.io, and replace them with thev1beta2 version either by installing the latest version of the operator or by runningkubectl create -f config/crd/bases.

• Version >= 1.13 of Kubernetes to use thesubresource support for CustomResourceDefinitions, which became beta in 1.13 and is enabled by default in 1.13 and higher.

• Version >= 1.16 of Kubernetes to use theMutatingWebhook andValidatingWebhook ofapiVersion: admissionregistration.k8s.io/v1.

For getting started with Spark operator, please refer toGetting Started.

For detailed user guide and API documentation, please refer toUser Guide andAPI Specification.

If you are running Spark operator on Google Kubernetes Engine (GKE) and want to use Google Cloud Storage (GCS) and/or BigQuery for reading/writing data, also refer to theGCP guide.

The following table lists the most recent few versions of the operator.

For developing with Spark Operator, please refer toDeveloper Guide.

For contributing to Spark Operator, please refer toContributor Guide.

• Join theCNCF Slack Channel and then join#kubeflow-spark-operator Channel.
• Check out our blog postAnnouncing the Kubeflow Spark Operator: Building a Stronger Spark on Kubernetes Community.
• Join our monthly community meetingKubeflow Spark Operator Meeting Notes.

Check outadopters of Spark Operator.