Repository files navigation

SynapseML (previously known as MMLSpark), is an open-source library that simplifies the creation of massively scalable machine learning (ML) pipelines. SynapseML provides simple, composable, and distributed APIs for a wide variety of different machine learning tasks such as text analytics, vision, anomaly detection, and many others. SynapseML is built on theApache Spark distributed computing framework and shares the same API as theSparkML/MLLib library, allowing you to seamlessly embed SynapseML models into existing Apache Spark workflows.

With SynapseML, you can build scalable and intelligent systems to solve challenges in domains such as anomaly detection, computer vision, deep learning, text analytics, and others. SynapseML can train and evaluate models on single-node, multi-node, and elastically resizable clusters of computers. This lets you scale your work without wasting resources. SynapseML is usable across Python, R, Scala, Java, and .NET. Furthermore, its API abstracts over a wide variety of databases, file systems, and cloud data stores to simplify experiments no matter where data is located.

SynapseML requires Scala 2.12, Spark 3.4+, and Python 3.8+.

Topics	Links
Build
Version
Docs
Support
Binder
Usage

Vowpal Wabbit on Spark	The Cognitive Services for Big Data	LightGBM on Spark	Spark Serving
Fast, Sparse, and Effective Text Analytics	Leverage the Microsoft Cognitive Services at Unprecedented Scales in your existing SparkML pipelines	Train Gradient Boosted Machines with LightGBM	Serve any Spark Computation as a Web Service with Sub-Millisecond Latency

HTTP on Spark	ONNX on Spark	Responsible AI	Spark Binding Autogeneration
An Integration Between Spark and the HTTP Protocol, enabling Distributed Microservice Orchestration	Distributed and Hardware Accelerated Model Inference on Spark	Understand Opaque-box Models and Measure Dataset Biases	Automatically Generate Spark bindings for PySpark and SparklyR

Isolation Forest on Spark	CyberML	Conditional KNN
Distributed Nonlinear Outlier Detection	Machine Learning Tools for Cyber Security	Scalable KNN Models with Conditional Queries

For quickstarts, documentation, demos, and examples please see ourwebsite.

First select the correct platform that you are installing SynapseML into:

• Synapse Machine Learning
  • Features
  • Documentation and Examples
  • Setup and installation
    • Microsoft Fabric
    • Synapse Analytics
    • Databricks
    • Python Standalone
    • Spark Submit
    • SBT
    • Apache Livy and HDInsight
    • Docker
    • R
    • Building from source
  • Papers
  • Learn More
  • Contributing & feedback
  • Other relevant projects

In Microsoft Fabric notebooks SynapseML is already installed. To change the version please place the following in the first cell of your notebook.

%%configure -f{"name":"synapseml","conf": {"spark.jars.packages":"com.microsoft.azure:synapseml_2.12:<THE_SYNAPSEML_VERSION_YOU_WANT>","spark.jars.repositories":"https://mmlspark.azureedge.net/maven","spark.jars.excludes":"org.scala-lang:scala-reflect,org.apache.spark:spark-tags_2.12,org.scalactic:scalactic_2.12,org.scalatest:scalatest_2.12,com.fasterxml.jackson.core:jackson-databind","spark.yarn.user.classpath.first":"true","spark.sql.parquet.enableVectorizedReader":"false"  }}

In Azure Synapse notebooks please place the following in the first cell of your notebook.

• For Spark 3.5 Pools:

%%configure -f{"name":"synapseml","conf": {"spark.jars.packages":"com.microsoft.azure:synapseml_2.12:1.1.0","spark.jars.repositories":"https://mmlspark.azureedge.net/maven","spark.jars.excludes":"org.scala-lang:scala-reflect,org.apache.spark:spark-tags_2.12,org.scalactic:scalactic_2.12,org.scalatest:scalatest_2.12,com.fasterxml.jackson.core:jackson-databind","spark.yarn.user.classpath.first":"true","spark.sql.parquet.enableVectorizedReader":"false"  }}

• For Spark 3.4 Pools:

%%configure -f{"name":"synapseml","conf": {"spark.jars.packages":"com.microsoft.azure:synapseml_2.12:1.0.15","spark.jars.repositories":"https://mmlspark.azureedge.net/maven","spark.jars.excludes":"org.scala-lang:scala-reflect,org.apache.spark:spark-tags_2.12,org.scalactic:scalactic_2.12,org.scalatest:scalatest_2.12,com.fasterxml.jackson.core:jackson-databind","spark.yarn.user.classpath.first":"true","spark.sql.parquet.enableVectorizedReader":"false"  }}

• For Spark 3.3 Pools:

%%configure -f{"name":"synapseml","conf": {"spark.jars.packages":"com.microsoft.azure:synapseml_2.12:0.11.4-spark3.3","spark.jars.repositories":"https://mmlspark.azureedge.net/maven","spark.jars.excludes":"org.scala-lang:scala-reflect,org.apache.spark:spark-tags_2.12,org.scalactic:scalactic_2.12,org.scalatest:scalatest_2.12,com.fasterxml.jackson.core:jackson-databind","spark.yarn.user.classpath.first":"true","spark.sql.parquet.enableVectorizedReader":"false"  }}

To install at the pool level instead of the notebook leveladd the spark properties listed above to the pool configuration.

To install SynapseML on theDatabrickscloud, create a newlibrary from Mavencoordinatesin your workspace.

For the coordinates use:com.microsoft.azure:synapseml_2.12:1.1.0with the resolver:https://mmlspark.azureedge.net/maven. Ensure this library isattached to your target cluster(s).

Finally, ensure that your Spark cluster has at least Spark 3.2 and Scala 2.12. If you encounter Netty dependency issues please use DBR 10.1.

You can use SynapseML in both your Scala and PySpark notebooks. To get started with our example notebooks import the following databricks archive:

https://mmlspark.blob.core.windows.net/dbcs/SynapseMLExamplesv1.1.0.dbc

To try out SynapseML on a Python (or Conda) installation you can get Sparkinstalled via pip withpip install pyspark. You can then usepyspark as inthe above example, or from python:

importpysparkspark=pyspark.sql.SparkSession.builder.appName("MyApp") \            .config("spark.jars.packages","com.microsoft.azure:synapseml_2.12:1.1.0") \            .getOrCreate()importsynapse.ml

SynapseML can be conveniently installed on existing Spark clusters via the--packages option, examples:

spark-shell --packages com.microsoft.azure:synapseml_2.12:1.1.0pyspark --packages com.microsoft.azure:synapseml_2.12:1.1.0spark-submit --packages com.microsoft.azure:synapseml_2.12:1.1.0 MyApp.jar

If you are building a Spark application in Scala, add the following lines toyourbuild.sbt:

libraryDependencies+="com.microsoft.azure"%"synapseml_2.12"%"1.1.0"

To install SynapseML from within a Jupyter notebook served by Apache Livy the following configure magic can be used. You will need to start a new session after this configure cell is executed.

Excluding certain packages from the library may be necessary due to current issues with Livy 0.5.

%%configure -f{"name":"synapseml","conf": {"spark.jars.packages":"com.microsoft.azure:synapseml_2.12:1.1.0","spark.jars.excludes":"org.scala-lang:scala-reflect,org.apache.spark:spark-tags_2.12,org.scalactic:scalactic_2.12,org.scalatest:scalatest_2.12,com.fasterxml.jackson.core:jackson-databind"    }}

The easiest way to evaluate SynapseML is via our pre-built Docker container. Todo so, run the following command:

docker run -it -p 8888:8888 -e ACCEPT_EULA=yes mcr.microsoft.com/mmlspark/release jupyter notebook

Navigate tohttp://localhost:8888/ in your web browser to run the samplenotebooks. See thedocumentation for more on Docker use.

To read the EULA for using the docker image, rundocker run -it -p 8888:8888 mcr.microsoft.com/mmlspark/release eula

To try out SynapseML using the R autogenerated wrapperssee ourinstructions. Note: This feature is still under developmentand some necessary custom wrappers may be missing.

SynapseML has recently transitioned to a new build infrastructure.For detailed developer docs please see theDeveloper Readme

If you are an existing synapsemldeveloper, you will need to reconfigure yourdevelopment setup. We now support platform independent development andbetter integrate with intellij and SBT.If you encounter issues please reach out to our support email!

• Large Scale Intelligent Microservices

• Conditional Image Retrieval

• MMLSpark: Unifying Machine Learning Ecosystems at Massive Scales

• Flexible and Scalable Deep Learning with SynapseML

• Large-Scale Automatic Audiobook Creation

• Visit ourwebsite.

• Watch our keynote demos atthe Spark+AI Summit 2019,the Spark+AI European Summit 2018,the Spark+AI Summit 2018 andSynapseML at the Spark Summit.

• See how SynapseML is used tohelp endangered species.

• Explore generative adversarial artwork inour collaboration with The MET and MIT.

• Exploreour collaboration with Apache Spark on image analysis.

This project has adopted theMicrosoft Open Source Code of Conduct. For moreinformation see theCode of Conduct FAQ or contactopencode@microsoft.com with any additionalquestions or comments.

SeeCONTRIBUTING.md for contribution guidelines.

To give feedback and/or report an issue, open aGitHubIssue.

• Vowpal Wabbit

• LightGBM

• DMTK: Microsoft Distributed Machine Learning Toolkit

• Recommenders

• JPMML-SparkML plugin for converting SynapseML LightGBM models to PMML

• Microsoft Cognitive Toolkit

Apache®, Apache Spark, and Spark® are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.