The Cognitive Services on Spark:![]() |
LIME on Spark:![]() |
Spark Serving:![]() |
 |
 |
 |
| Leverage the Microsoft Cognitive Services at Unprecedented Scales in your existing SparkML pipelines |
Distributed, Model Agnostic, Interpretations for Image Classifiers |
Serve any Spark Computation as a Web Service with Sub-Millisecond Latency |
LightGBM on Spark:![]() |
CNTK on Spark:![]() |
HTTP on Spark:![]() |
 |
 |
 |
| Train Gradient Boosted Machines with LightGBM |
Distributed Deep Learning with the Microsoft Cognitive Toolkit |
An Integration Between Spark and the HTTP Protocol, enabling Distributed Microservice Orchestration |
## Examples
* Create a deep image classifier with transfer learning ([example 9])
* Fit a LightGBM classification or regression model on a biochemical dataset
([example 3]), to learn more check out the [LightGBM documentation
page](docs/lightgbm.md).
* Deploy a deep network as a distributed web service with [MMLSpark
Serving](docs/mmlspark-serving.md)
* Use web services in Spark with [HTTP on Apache Spark](docs/http.md)
* Use Bi-directional LSTMs from Keras for medical entity extraction
([example 8])
* Create a text analytics system on Amazon book reviews ([example 4])
* Perform distributed hyperparameter tuning to identify Breast Cancer
([example 5])
* Easily ingest images from HDFS into Spark `DataFrame` ([example 6])
* Use OpenCV on Spark to manipulate images ([example 7])
* Train classification and regression models easily via implicit featurization
of data ([example 1])
* Train and evaluate a flight delay prediction system ([example 2])
See our [notebooks](notebooks/samples/) for all examples.
[example 1]: notebooks/samples/Classification%20-%20Adult%20Census.ipynb
"Adult Census Income Training"
[example 2]: notebooks/samples/Regression%20-%20Flight%20Delays.ipynb
"Regression Example with Flight Delay Dataset"
[example 3]: notebooks/samples/LightGBM%20-%20Quantile%20Regression%20for%20Drug%20Discovery.ipynb
"Quantile Regression with LightGBM"
[example 4]: notebooks/samples/TextAnalytics%20-%20Amazon%20Book%20Reviews.ipynb
"Amazon Book Reviews - TextFeaturizer"
[example 5]: notebooks/samples/HyperParameterTuning%20-%20Fighting%20Breast%20Cancer.ipynb
"Hyperparameter Tuning with MMLSpark"
[example 6]: notebooks/samples/DeepLearning%20-%20CIFAR10%20Convolutional%20Network.ipynb
"CIFAR10 CNTK CNN Evaluation"
[example 7]: notebooks/samples/OpenCV%20-%20Pipeline%20Image%20Transformations.ipynb
"Pipeline Image Transformations"
[example 8]: notebooks/samples/DeepLearning%20-%20BiLSTM%20Medical%20Entity%20Extraction.ipynb
"Medical Entity Extraction"
[example 9]: notebooks/samples/DeepLearning%20-%20Flower%20Image%20Classification.ipynb
"Deep Flower Classification"
[example 10]: notebooks/gpu/DeepLearning%20-%20Distributed%20CNTK%20training.ipynb
"CIFAR10 CNTK CNN Training"
## A short example
Below is an excerpt from a simple example of using a pre-trained CNN to
classify images in the CIFAR-10 dataset. View the whole source code in notebook [example 9].
```python
...
import mmlspark
# Initialize CNTKModel and define input and output columns
cntkModel = mmlspark.cntk.CNTKModel() \
.setInputCol("images").setOutputCol("output") \
.setModelLocation(modelFile)
# Train on dataset with internal spark pipeline
scoredImages = cntkModel.transform(imagesWithLabels)
...
```
See [other sample notebooks](notebooks/samples/) as well as the MMLSpark
documentation for [Scala](http://mmlspark.azureedge.net/docs/scala/) and
[PySpark](http://mmlspark.azureedge.net/docs/pyspark/).
## Setup and installation
### Spark package
MMLSpark can be conveniently installed on existing Spark clusters via the
`--packages` option, examples:
```bash
spark-shell --packages Azure:mmlspark:0.17
pyspark --packages Azure:mmlspark:0.17
spark-submit --packages Azure:mmlspark:0.17 MyApp.jar
```
This can be used in other Spark contexts too. For example, you can use MMLSpark
in [AZTK](https://github.com/Azure/aztk/) by [adding it to the
`.aztk/spark-defaults.conf`
file](https://github.com/Azure/aztk/wiki/PySpark-on-Azure-with-AZTK#optional-set-up-mmlspark).
### Databricks
To install MMLSpark on the [Databricks
cloud](http://community.cloud.databricks.com), create a new [library from Maven
coordinates](https://docs.databricks.com/user-guide/libraries.html#libraries-from-maven-pypi-or-spark-packages)
in your workspace.
For the coordinates use: `Azure:mmlspark:0.17`. Ensure this library is
attached to all clusters you create.
Finally, ensure that your Spark cluster has at least Spark 2.1 and Scala 2.11.
You can use MMLSpark in both your Scala and PySpark notebooks. To get started with our example notebooks import the following databricks archive:
```https://mmlspark.blob.core.windows.net/dbcs/MMLSpark%20Examples%20v0.17.dbc```
### Docker
The easiest way to evaluate MMLSpark is via our pre-built Docker container. To
do so, run the following command:
```bash
docker run -it -p 8888:8888 -e ACCEPT_EULA=yes mcr.microsoft.com/mmlspark/release
```
Navigate to 
](https://github.com/Azure/mmlspark/releases)
### HDInsight
To install MMLSpark on an existing [HDInsight Spark
Cluster](https://docs.microsoft.com/en-us/azure/hdinsight/), you can execute a
script action on the cluster head and worker nodes. For instructions on
running script actions, see [this
guide](https://docs.microsoft.com/en-us/azure/hdinsight/hdinsight-hadoop-customize-cluster-linux#use-a-script-action-during-cluster-creation).
The script action url is: