# spark-on-kubernetes-helm
**Repository Path**: wei_shujie/spark-on-kubernetes-helm
## Basic Information
- **Project Name**: spark-on-kubernetes-helm
- **Description**: Spark on Kubernetes infrastructure Helm charts repo
- **Primary Language**: Unknown
- **License**: Apache-2.0
- **Default Branch**: master
- **Homepage**: None
- **GVP Project**: No
## Statistics
- **Stars**: 0
- **Forks**: 0
- **Created**: 2023-05-10
- **Last Updated**: 2023-05-10
## Categories & Tags
**Categories**: Uncategorized
**Tags**: None
## README
[![CircleCI][circle-ci-badge]][circle-ci-repo]
# Spark on Kubernetes Cluster Helm Chart
This repo contains the Helm chart for the fully functional and production ready [Spark on Kubernetes][spark-on-kubernetes-docs] cluster setup integrated with the [Spark History Server][spark-history-server-helm], [JupyterHub][jupyterhub-k8s] and [Prometheus][prometheus] stack.
Refer the [design concept][design-concept] for the implementation details.
## Getting Started
##### Initialize Helm (for Helm 2.x)
In order to use [Helm charts][helm] for the Spark on Kubernetes cluster deployment first we need to initialize Helm client.
```bash
kubectl create serviceaccount tiller --namespace kube-system
kubectl create clusterrolebinding tiller --clusterrole cluster-admin --serviceaccount=kube-system:tiller
helm init --upgrade --service-account tiller --tiller-namespace kube-system
kubectl get pods --namespace kube-system -w
# Wait until Pod `tiller-deploy-*` moves to Running state
```
##### Install Livy
The basic Spark on Kubernetes setup consists of the only [Apache Livy][livy-docs] server deployment, which can be installed with the [Livy Helm chart][livy-helm].
```bash
helm repo add joshuazsu https://joshuazsu.github.io/helm-charts
helm repo update
kubectl create namespace livy
helm upgrade --install livy --namespace livy jahstreet/livy \
--set rbac.create=true # If you are running RBAC-enabled Kubernetes cluster
kubectl get pods --namespace livy -w
# Wait until Pod `livy-0` moves to Running state
```
For more advanced Spark cluster setups refer the [Documentation page][documentation-page].
##### Run Spark Job
Now when Livy is up and running we can submit Spark job via [Livy REST API][livy-rest-api].
```bash
kubectl exec --namespace livy livy-0 -- \
curl -s -k -H 'Content-Type: application/json' -X POST \
-d '{
"name": "SparkPi-01",
"className": "org.apache.spark.examples.SparkPi",
"numExecutors": 2,
"file": "local:///opt/spark/examples/jars/spark-examples_2.11-2.4.5.jar",
"args": ["10000"],
"conf": {
"spark.kubernetes.namespace": "livy"
}
}' "http://localhost:8998/batches" | jq
# Record BATCH_ID from the response
```
##### Track running job
To track the running Spark job we can use all the available [Kubernetes tools][spark-on-kubernetes-debugging] and the [Livy REST API][livy-rest-api].
```bash
# Watch running Spark Pods
kubectl get pods --namespace livy -w --show-labels
# Check Livy batch status
kubectl exec --namespace livy livy-0 -- curl -s http://localhost:8998/batches/$BATCH_ID | jq
```
To configure [Ingress][kubernetes-ingress] for direct access to Livy UI and Spark UI refer the [Documentation page][documentation-page].
## Spark on Kubernetes Cluster Design Concept
### Motivation
Running [Spark on Kubernetes][spark-on-kubernetes-docs] is available since Spark `v2.3.0` release on February 28, 2018. Now it is `v2.4.5` and still lacks much comparing to the well known Yarn setups on Hadoop-like clusters.
Corresponding to the official [documentation][spark-on-kubernetes-docs] user is able to run Spark on Kubernetes via [`spark-submit` CLI script][spark-on-kubernetes-cluster-mode]. And actually it is the only in-built into Apache Spark Kubernetes related capability along with some [config options][spark-on-kubernetes-config]. [Debugging proposal][spark-on-kubernetes-debugging] from Apache docs is too poor to use it easily and available only for console based tools. Schedulers integration is not available either, which makes it too tricky to setup convenient pipelines with Spark on Kubernetes out of the box. Yarn based Hadoop clusters in turn has all the UIs, Proxies, Schedulers and APIs to make your life easier.
On the other hand the usage of Kubernetes clusters in opposite to Yarn ones has definite benefits (July 2019 comparison):
- Pricing. Comparing the similar cluster setups on Azure Cloud shows that [AKS][azure-aks] is about 35% cheaper than [HDInsight Spark][azure-hdinsight-spark].
- Scaling. Kubernetes cluster in Cloud support elastic autoscaling with many cool related features alongside, eg: Nodepools. Scaling of Hadoop clusters is far not as fast though, can be done either [manually][azure-hdinsight-manual-scaling] or [automatically][azure-hdinsight-auto-scaling] (on July 2019 was in preview).
- Integrations. You can run any workloads in Kubernetes cluster wrapped into the Docker container. But do you know anyone who has ever written Yarn App in the modern world?
- Support. You don't have a full control over the cluster setup provided by Cloud and usually there are no latest versions of software available for months after the release. With Kubernetes you can build image on your own.
- Other Kuebernetes pros. CI/CD with Helm, Monitoring stacks ready for use in-one-button-click, huge popularity and community support, good tooling and of course HYPE.
All that makes much sense to try to improve Spark on Kubernetes usability to take the whole advantage of modern Kubernetes setups in use.
### Design concept
The heart of all the problems solution is [Apache Livy][livy-docs]. Apache Livy is a service that enables easy interaction with a Spark cluster over a REST interface. It is supported by [Apache Incubator][apache-incubator] community and [Azure HDInsight][azure-hdinsight-livy] team, which uses it as a first class citizen in their Yarn cluster setup and does many integrations with it. Watch [Spark Summit 2016, Cloudera and Microsoft, Livy concepts and motivation][youtube-azure-livy] for the details.
The cons is that Livy is written for Yarn. But Yarn is just Yet Another resource manager with containers abstraction adaptable to the Kubernetes concepts. Livy is fully open-sourced as well, its codebase is RM aware enough to make Yet Another One implementation of it's interfaces to add Kubernetes support. So why not!? Check the [WIP PR][livy-588] with Kubernetes support proposal for Livy.
The high-level architecture of Livy on Kubernetes is the same as for Yarn.
![Livy schema][image-livy-schema]
Livy server just wraps all the logic concerning interaction with Spark cluster and provides simple [REST interface][livy-rest-api].
[EXPAND] For example, to submit Spark Job to the cluster you just need to send `POST /batches` with JSON body containing Spark config options, mapped to `spark-submit` script analogous arguments.
```bash
$SPARK_HOME/bin/spark-submit \
--master k8s://https://: \
--deploy-mode cluster \
--name SparkPi \
--class org.apache.spark.examples.SparkPi \
--conf spark.executor.instances=5 \
--conf spark.kubernetes.container.image= \
local:///path/to/examples.jar
# Has the similar effect as calling Livy via REST API
curl -H 'Content-Type: application/json' -X POST \
-d '{
"name": "SparkPi",
"className": "org.apache.spark.examples.SparkPi",
"numExecutors": 5,
"conf": {
"spark.kubernetes.container.image": ""
},
"file": "local:///path/to/examples.jar"
}' "http://livy.endpoint.com/batches"
```