This is not an officially supported Google product.
- Join our Slack channel.
- Check out who is using the Kubernetes Operator for Apache Spark.
Project status: beta
The Kubernetes Operator for Apache Spark is under active development, but backward compatibility of the APIs is guaranteed for beta releases.
If you are currently using the
v1alpha1 version of the APIs in your manifests, please update them to use the
v1beta1 version by changing
apiVersion: "sparkoperator.k8s.io/v1alpha1" to
apiVersion: "sparkoperator.k8s.io/v1beta1". You will also need to delete the
v1alpha1 version of the CustomResourceDefinitions named
scheduledsparkapplications.sparkoperator.k8s.io, and replace them with the
v1beta1 version either by installing the latest version of the operator or by running
kubectl create -f manifest/spark-operator-crds.yaml.
Customization of Spark pods, e.g., mounting arbitrary volumes and setting pod affinity, is currently experimental and implemented using a Kubernetes Mutating Admission Webhook, which became beta in Kubernetes 1.9. The mutating admission webhook is disabled by default but can be enabled if there are needs for pod customization. Check out the Quick Start Guide on how to enable the webhook.
- Version >= 1.8 of Kubernetes.
- Version >= 1.9 of Kubernetes if using the mutating admission webhook for Spark pod customization.
The Kubernetes Operator for Apache Spark relies on garbage collection support for custom resources that is available in Kubernetes 1.8+ and optionally the Mutating Admission Webhook which is available in Kubernetes 1.9+.
Due to this bug in Kubernetes 1.9 and earlier, CRD objects with escaped quotes (e.g.,
spark.ui.port\") in map keys can cause serialization problems in the API server. So please pay extra attention to make sure no offending escaping is in your
SparkApplication CRD objects, particularly if you use Kubernetes prior to 1.10.
The easiest way to install the Kubernetes Operator for Apache Spark is to use the Helm chart.
$ helm repo add incubator http://storage.googleapis.com/kubernetes-charts-incubator $ helm install incubator/sparkoperator --namespace spark-operator
Get started quickly with the Kubernetes Operator for Apache Spark using the Quick Start Guide.
If you are running the Kubernetes Operator for Apache Spark on Google Kubernetes Engine and want to use Google Cloud Storage (GCS) and/or BigQuery for reading/writing data, also refer to the GCP guide.
The Kubernetes Operator for Apache Spark aims to make specifying and running Spark applications as easy and idiomatic as running other workloads on Kubernetes. It uses Kubernetes custom resources for specifying, running, and surfacing status of Spark applications. For a complete reference of the custom resource definitions, please refer to the API Definition. For details on its design, please refer to the design doc. It requires Spark 2.3 and above that supports Kubernetes as a native scheduler backend.
The Kubernetes Operator for Apache Spark currently supports the following list of features:
- Supports Spark 2.3 and up.
- Enables declarative application specification and management of applications through custom resources.
- Automatically runs
spark-submiton behalf of users for each
SparkApplicationeligible for submission.
- Provides native cron support for running scheduled applications.
- Supports customization of Spark pods beyond what Spark natively is able to do through the mutating admission webhook, e.g., mounting ConfigMaps and volumes, and setting pod affinity/anti-affinity.
- Supports automatic application re-submission for updated
SparkAppliationobjects with updated specification.
- Supports automatic application restart with a configurable restart policy.
- Supports automatic retries of failed submissions with optional linear back-off.
- Supports mounting local Hadoop configuration as a Kubernetes ConfigMap automatically via
- Supports automatically staging local application dependencies to Google Cloud Storage (GCS) via
- Supports collecting and exporting application-level metrics and driver/executor metrics to Prometheus.