Apache SystemDS
Developer(s)	Apache Software Foundation,IBM
Initial release	November 2, 2015; 9 years ago (2015-11-02)
Stable release	3.0.0 / July 5, 2022; 2 years ago (2022-07-05)
Repository	SystemDS Repository
Written in	Java,Python, DML,C
Operating system	Linux,macOS,Windows
Type	Machine Learning,Deep Learning,Data Science
License	Apache License 2.0
Website	systemds.apache.org

Apache SystemDS (Previously, Apache SystemML) is an open source ML system for the end-to-end data science lifecycle.

SystemDS's distinguishing characteristics are:

1. Algorithm customizability via R-like and Python-like languages.
2. Multiple execution modes, including Standalone,Spark Batch,Spark MLContext,Hadoop Batch, and JMLC.
3. Automatic optimization based on data and cluster characteristics to ensure both efficiency and scalability.

SystemML was created in 2010 by researchers at theIBM Almaden Research Center led by IBM Fellow Shivakumar Vaithyanathan. It was observed that data scientists would write machine learning algorithms in languages such asR andPython for small data. When it came time to scale to big data, a systems programmer would be needed to scale the algorithm in a language such asScala. This process typically involved days or weeks per iteration, and errors would occur translating the algorithms to operate on big data. SystemML seeks to simplify this process. A primary goal of SystemML is to automatically scale an algorithm written in an R-like or Python-like language to operate on big data, generating the same answer without the error-prone, multi-iterative translation approach.

On June 15, 2015, at the Spark Summit in San Francisco, Beth Smith, General Manager of IBM Analytics, announced that IBM was open-sourcing SystemML as part of IBM's major commitment toApache Spark and Spark-related projects. SystemML became publicly available onGitHub on August 27, 2015 and became an Apache Incubator project on November 2, 2015. On May 17, 2017, the Apache Software Foundation Board approved the graduation of Apache SystemML as an Apache Top Level Project.

The following are some of the technologies built into the SystemDS engine.

• Compressed Linear Algebra for Large Scale Machine Learning
• Declarative Machine Learning Language

The following code snippet^[1] does thePrincipal component analysis of input matrix${\displaystyle A}$ , which returns the${\displaystyle eigenvectors}$ and the$eigenvalues$.

# PCA.dml# Refer: https://github.com/apache/systemds/blob/master/scripts/algorithms/PCA.dml#L61N=nrow(A);D=ncol(A);# perform z-scoring (centering and scaling)A=scale(A,center==1,scale==1);# co-variance matrixmu=colSums(A)/N;C=(t(A)%*%A)/(N-1)-(N/(N-1))*t(mu)%*%mu;# compute eigen vectors and values[evalues,evectors]=eigen(C);

spark-submit SystemDS.jar -f PCA.dml -nvargs INPUT=INPUT_DIR/pca-1000x1000 \  OUTPUT=OUTPUT_DIR/pca-1000x1000-model PROJDATA=1 CENTER=1 SCALE=1

DBSCAN clustering algorithm withEuclidean distance.

X=rand(rows=1780,cols=180,min=1,max=20)[indices,model]=dbscan(X=X,eps=2.5,minPts=360)

SystemDS 2.0.0 is the first major release under the new name. This release contains a major refactoring, a few major features, a large number of improvements and fixes, and some experimental features to better support the end-to-end data science lifecycle. In addition to that, this release also removes several features that are not up date and outdated.

• New mechanism for DML-bodied (script-level)builtin functions, and a wealth of new built-in functions for data preprocessing including data cleaning, augmentation and feature engineering techniques, new ML algorithms, and model debugging.
• Several methods for data cleaning have been implemented including multiple imputations with multivariate imputation by chained equations (MICE) and other techniques, SMOTE, an oversampling technique for class imbalance, forward and backward NA filling, cleaning using schema and length information, support for outlier detection using standard deviation and inter-quartile range, and functional dependency discovery.
• A complete framework for lineage tracing and reuse including support for loop deduplication, full and partial reuse, compiler assisted reuse, several new rewrites to facilitate reuse.
• New federated runtime backend including support for federated matrices and frames, federatedbuiltins (transform-encode,decode etc.).
• Refactor compression package and add functionalities including quantization for lossy compression, binary cell operations, left matrix multiplication. [experimental]
• New python bindings with supports for severalbuiltins, matrix operations, federated tensors and lineage traces.
• Cuda implementation of cumulative aggregate operators (cumsum,cumprod etc.)
• New model debugging technique with slice finder.
• New tensor data model (basic tensors of different value types, data tensors with schema) [experimental]
• Cloud deployment scripts for AWS and scripts to set up and start federated operations.
• Performance improvements withparallel sort,gpu cum agg,append cbind etc.
• Various compiler and runtime improvements including new and improved rewrites, reduced Spark context creation, neweval framework, list operations, updated native kernel libraries to name a few.
• New data reader/writer forjson frames and support forsql as a data source.
• Miscellaneous improvements: improved documentation, better testing, run/release scripts, improved packaging, Docker container for systemds, support for lambda expressions, bug fixes.
• Removed MapReduce compiler and runtime backend,pydml parser, Java-UDF framework, script-level debugger.
• Deprecated./scripts/algorithms, as those algorithms gradually will be part of SystemDSbuiltins.

^[2]

Apache SystemDS welcomes contributions in code, question and answer, community building, or spreading the word. The contributor guide is available athttps://github.com/apache/systemds/blob/main/CONTRIBUTING.md

• Comparison of deep learning software

• Apache SystemML website
• IBM Research - SystemML
• Q & A with Shiv Vaithyanathan, Creator of SystemML and IBM Fellow
• A Universal Translator for Big Data and Machine Learning
• SystemML: Declarative Machine Learning at Scale presentation by Fred Reiss
• SystemML: Declarative Machine Learning on MapReduceArchived 2016-03-10 at theWayback Machine
• Hybrid Parallelization Strategies for Large-Scale Machine Learning in SystemML
• SystemML's Optimizer: Plan Generation for Large-Scale Machine Learning Programs
• IBM's SystemML machine learning system becomes Apache Incubator project
• IBM donates machine learning tech to Apache Spark open source community
• IBM's SystemML Moves Forward as Apache Incubator Project

• Cluster computing
• Data mining and machine learning software
• Hadoop
• Apache Software Foundation projects
• Software using the Apache license
• Java platform
• Big data products
• 2015 software

• Articles with short description
• Short description is different from Wikidata
• Webarchive template wayback links