Transformer()

Abstract class for transformers that transform one dataset into another.

UnaryTransformer()

Abstract class for transformers that take one input column, apply transformation, and output the result as a new column.

Estimator()

Abstract class for estimators that fit models to data.

Model()

Abstract class for models that are fitted by estimators.

Predictor()

Estimator for prediction tasks (regression and classification).

PredictionModel()

Model for prediction tasks (regression and classification).

Pipeline(*[, stages])

A simple pipeline, which acts as an estimator.

PipelineModel(stages)

Represents a compiled pipeline with transformers and fitted models.

Param(parent, name, doc[, typeConverter])

A param with self-contained documentation.

Params()

Components that take parameters.

TypeConverters()

Factory methods for common type conversion functions forParam.typeConverter.

Binarizer(*[, threshold, inputCol, ...])

Binarize a column of continuous features given a threshold.

BucketedRandomProjectionLSH(*[, inputCol, ...])

LSH class for Euclidean distance metrics.

BucketedRandomProjectionLSHModel([java_model])

Model fitted byBucketedRandomProjectionLSH, where multiple random vectors are stored.

Bucketizer(*[, splits, inputCol, outputCol, ...])

Maps a column of continuous features to a column of feature buckets.

ChiSqSelector(*[, numTopFeatures, ...])

Chi-Squared feature selection, which selects categorical features to use for predicting a categorical label.

ChiSqSelectorModel([java_model])

Model fitted byChiSqSelector.

CountVectorizer(*[, minTF, minDF, maxDF, ...])

Extracts a vocabulary from document collections and generates aCountVectorizerModel.

CountVectorizerModel([java_model])

Model fitted byCountVectorizer.

DCT(*[, inverse, inputCol, outputCol])

A feature transformer that takes the 1D discrete cosine transform of a real vector.

ElementwiseProduct(*[, scalingVec, ...])

Outputs the Hadamard product (i.e., the element-wise product) of each input vector with a provided "weight" vector.

FeatureHasher(*[, numFeatures, inputCols, ...])

Feature hashing projects a set of categorical or numerical features into a feature vector of specified dimension (typically substantially smaller than that of the original feature space).

HashingTF(*[, numFeatures, binary, ...])

Maps a sequence of terms to their term frequencies using the hashing trick.

IDF(*[, minDocFreq, inputCol, outputCol])

Compute the Inverse Document Frequency (IDF) given a collection of documents.

IDFModel([java_model])

Model fitted byIDF.

Imputer(*[, strategy, missingValue, ...])

Imputation estimator for completing missing values, using the mean, median or mode of the columns in which the missing values are located.

ImputerModel([java_model])

Model fitted byImputer.

IndexToString(*[, inputCol, outputCol, labels])

Apyspark.ml.base.Transformer that maps a column of indices back to a new column of corresponding string values.

Interaction(*[, inputCols, outputCol])

Implements the feature interaction transform.

MaxAbsScaler(*[, inputCol, outputCol])

Rescale each feature individually to range [-1, 1] by dividing through the largest maximum absolute value in each feature.

MaxAbsScalerModel([java_model])

Model fitted byMaxAbsScaler.

MinHashLSH(*[, inputCol, outputCol, seed, ...])

LSH class for Jaccard distance.

MinHashLSHModel([java_model])

Model produced byMinHashLSH, where where multiple hash functions are stored.

MinMaxScaler(*[, min, max, inputCol, outputCol])

Rescale each feature individually to a common range [min, max] linearly using column summary statistics, which is also known as min-max normalization or Rescaling.

MinMaxScalerModel([java_model])

Model fitted byMinMaxScaler.

NGram(*[, n, inputCol, outputCol])

A feature transformer that converts the input array of strings into an array of n-grams.

Normalizer(*[, p, inputCol, outputCol])

Normalize a vector to have unit norm using the given p-norm.

OneHotEncoder(*[, inputCols, outputCols, ...])

A one-hot encoder that maps a column of category indices to a column of binary vectors, with at most a single one-value per row that indicates the input category index.

OneHotEncoderModel([java_model])

Model fitted byOneHotEncoder.

PCA(*[, k, inputCol, outputCol])

PCA trains a model to project vectors to a lower dimensional space of the topk principal components.

PCAModel([java_model])

Model fitted byPCA.

PolynomialExpansion(*[, degree, inputCol, ...])

Perform feature expansion in a polynomial space.

QuantileDiscretizer(*[, numBuckets, ...])

QuantileDiscretizer takes a column with continuous features and outputs a column with binned categorical features.

RobustScaler(*[, lower, upper, ...])

RobustScaler removes the median and scales the data according to the quantile range.

RobustScalerModel([java_model])

Model fitted byRobustScaler.

RegexTokenizer(*[, minTokenLength, gaps, ...])

A regex based tokenizer that extracts tokens either by using the provided regex pattern (in Java dialect) to split the text (default) or repeatedly matching the regex (if gaps is false).

RFormula(*[, formula, featuresCol, ...])

Implements the transforms required for fitting a dataset against an R model formula.

RFormulaModel([java_model])

Model fitted byRFormula.

SQLTransformer(*[, statement])

Implements the transforms which are defined by SQL statement.

StandardScaler(*[, withMean, withStd, ...])

Standardizes features by removing the mean and scaling to unit variance using column summary statistics on the samples in the training set.

StandardScalerModel([java_model])

Model fitted byStandardScaler.

StopWordsRemover(*[, inputCol, outputCol, ...])

A feature transformer that filters out stop words from input.

StringIndexer(*[, inputCol, outputCol, ...])

A label indexer that maps a string column of labels to an ML column of label indices.

StringIndexerModel([java_model])

Model fitted byStringIndexer.

TargetEncoder(*[, inputCols, outputCols, ...])

Target Encoding maps a column of categorical indices into a numerical feature derived from the target.

TargetEncoderModel([java_model])

Model fitted byTargetEncoder.

Tokenizer(*[, inputCol, outputCol])

A tokenizer that converts the input string to lowercase and then splits it by white spaces.

UnivariateFeatureSelector(*[, featuresCol, ...])

Feature selector based on univariate statistical tests against labels.

UnivariateFeatureSelectorModel([java_model])

Model fitted byUnivariateFeatureSelector.

VarianceThresholdSelector(*[, featuresCol, ...])

Feature selector that removes all low-variance features.

VarianceThresholdSelectorModel([java_model])

Model fitted byVarianceThresholdSelector.

VectorAssembler(*[, inputCols, outputCol, ...])

A feature transformer that merges multiple columns into a vector column.

VectorIndexer(*[, maxCategories, inputCol, ...])

Class for indexing categorical feature columns in a dataset ofVector.

VectorIndexerModel([java_model])

Model fitted byVectorIndexer.

VectorSizeHint(*[, inputCol, size, ...])

A feature transformer that adds size information to the metadata of a vector column.

VectorSlicer(*[, inputCol, outputCol, ...])

This class takes a feature vector and outputs a new feature vector with a subarray of the original features.

Word2Vec(*[, vectorSize, minCount, ...])

Word2Vec trains a model ofMap(String, Vector), i.e. transforms a word into a code for further natural language processing or machine learning process.

Word2VecModel([java_model])

Model fitted byWord2Vec.

LinearSVC(*[, featuresCol, labelCol, ...])

This binary classifier optimizes the Hinge Loss using the OWLQN optimizer.

LinearSVCModel([java_model])

Model fitted by LinearSVC.

LinearSVCSummary([java_obj])

Abstraction for LinearSVC Results for a given model.

LinearSVCTrainingSummary([java_obj])

Abstraction for LinearSVC Training results.

LogisticRegression(*[, featuresCol, ...])

Logistic regression.

LogisticRegressionModel([java_model])

Model fitted by LogisticRegression.

LogisticRegressionSummary([java_obj])

Abstraction for Logistic Regression Results for a given model.

LogisticRegressionTrainingSummary([java_obj])

Abstraction for multinomial Logistic Regression Training results.

BinaryLogisticRegressionSummary([java_obj])

Binary Logistic regression results for a given model.

BinaryLogisticRegressionTrainingSummary([...])

Binary Logistic regression training results for a given model.

DecisionTreeClassifier(*[, featuresCol, ...])

Decision tree learning algorithm for classification.

DecisionTreeClassificationModel([java_model])

Model fitted by DecisionTreeClassifier.

GBTClassifier(*[, featuresCol, labelCol, ...])

Gradient-Boosted Trees (GBTs) learning algorithm for classification.

GBTClassificationModel([java_model])

Model fitted by GBTClassifier.

RandomForestClassifier(*[, featuresCol, ...])

Random Forest learning algorithm for classification.

RandomForestClassificationModel([java_model])

Model fitted by RandomForestClassifier.

RandomForestClassificationSummary([java_obj])

Abstraction for RandomForestClassification Results for a given model.

RandomForestClassificationTrainingSummary([...])

Abstraction for RandomForestClassificationTraining Training results.

BinaryRandomForestClassificationSummary([...])

BinaryRandomForestClassification results for a given model.

BinaryRandomForestClassificationTrainingSummary([...])

BinaryRandomForestClassification training results for a given model.

NaiveBayes(*[, featuresCol, labelCol, ...])

Naive Bayes Classifiers.

NaiveBayesModel([java_model])

Model fitted by NaiveBayes.

MultilayerPerceptronClassifier(*[, ...])

Classifier trainer based on the Multilayer Perceptron.

MultilayerPerceptronClassificationModel([...])

Model fitted by MultilayerPerceptronClassifier.

MultilayerPerceptronClassificationSummary([...])

Abstraction for MultilayerPerceptronClassifier Results for a given model.

MultilayerPerceptronClassificationTrainingSummary([...])

Abstraction for MultilayerPerceptronClassifier Training results.

OneVsRest(*[, featuresCol, labelCol, ...])

Reduction of Multiclass Classification to Binary Classification.

OneVsRestModel(models)

Model fitted by OneVsRest.

FMClassifier(*[, featuresCol, labelCol, ...])

Factorization Machines learning algorithm for classification.

FMClassificationModel([java_model])

Model fitted byFMClassifier.

FMClassificationSummary([java_obj])

Abstraction for FMClassifier Results for a given model.

FMClassificationTrainingSummary([java_obj])

Abstraction for FMClassifier Training results.

BisectingKMeans(*[, featuresCol, ...])

A bisecting k-means algorithm based on the paper "A comparison of document clustering techniques" by Steinbach, Karypis, and Kumar, with modification to fit Spark.

BisectingKMeansModel([java_model])

Model fitted by BisectingKMeans.

BisectingKMeansSummary([java_obj])

Bisecting KMeans clustering results for a given model.

KMeans(*[, featuresCol, predictionCol, k, ...])

K-means clustering with a k-means++ like initialization mode (the k-means|| algorithm by Bahmani et al).

KMeansModel([java_model])

Model fitted by KMeans.

KMeansSummary([java_obj])

Summary of KMeans.

GaussianMixture(*[, featuresCol, ...])

GaussianMixture clustering.

GaussianMixtureModel([java_model])

Model fitted by GaussianMixture.

GaussianMixtureSummary([java_obj])

Gaussian mixture clustering results for a given model.

LDA(*[, featuresCol, maxIter, seed, ...])

Latent Dirichlet Allocation (LDA), a topic model designed for text documents.

LDAModel([java_model])

Latent Dirichlet Allocation (LDA) model.

LocalLDAModel([java_model])

Local (non-distributed) model fitted byLDA.

DistributedLDAModel([java_model])

Distributed model fitted byLDA.

PowerIterationClustering(*[, k, maxIter, ...])

Power Iteration Clustering (PIC), a scalable graph clustering algorithm developed byLin and Cohen.

array_to_vector(col)

Converts a column of array of numeric type into a column of pyspark.ml.linalg.DenseVector instances

vector_to_array(col[, dtype])

Converts a column of MLlib sparse/dense vectors into a column of dense arrays.

predict_batch_udf(make_predict_fn, *, ...[, ...])

Given a function which loads a model and returns apredict function for inference over a batch of numpy inputs, returns a Pandas UDF wrapper for inference over a Spark DataFrame.

Vector()

DenseVector(ar)

A dense vector represented by a value array.

SparseVector(size, *args)

A simple sparse vector class for passing data to MLlib.

Vectors()

Factory methods for working with vectors.

Matrix(numRows, numCols[, isTransposed])

DenseMatrix(numRows, numCols, values[, ...])

Column-major dense matrix.

SparseMatrix(numRows, numCols, colPtrs, ...)

Sparse Matrix stored in CSC format.

Matrices()

ALS(*[, rank, maxIter, regParam, ...])

Alternating Least Squares (ALS) matrix factorization.

ALSModel([java_model])

Model fitted by ALS.

AFTSurvivalRegression(*[, featuresCol, ...])

Accelerated Failure Time (AFT) Model Survival Regression

AFTSurvivalRegressionModel([java_model])

Model fitted byAFTSurvivalRegression.

DecisionTreeRegressor(*[, featuresCol, ...])

Decision tree learning algorithm for regression.

DecisionTreeRegressionModel([java_model])

Model fitted byDecisionTreeRegressor.

GBTRegressor(*[, featuresCol, labelCol, ...])

Gradient-Boosted Trees (GBTs) learning algorithm for regression.

GBTRegressionModel([java_model])

Model fitted byGBTRegressor.

GeneralizedLinearRegression(*[, labelCol, ...])

Generalized Linear Regression.

GeneralizedLinearRegressionModel([java_model])

Model fitted byGeneralizedLinearRegression.

GeneralizedLinearRegressionSummary([java_obj])

Generalized linear regression results evaluated on a dataset.

GeneralizedLinearRegressionTrainingSummary([...])

Generalized linear regression training results.

IsotonicRegression(*[, featuresCol, ...])

Currently implemented using parallelized pool adjacent violators algorithm.

IsotonicRegressionModel([java_model])

Model fitted byIsotonicRegression.

LinearRegression(*[, featuresCol, labelCol, ...])

Linear regression.

LinearRegressionModel([java_model])

Model fitted byLinearRegression.

LinearRegressionSummary([java_obj])

Linear regression results evaluated on a dataset.

LinearRegressionTrainingSummary([java_obj])

Linear regression training results.

RandomForestRegressor(*[, featuresCol, ...])

Random Forest learning algorithm for regression.

RandomForestRegressionModel([java_model])

Model fitted byRandomForestRegressor.

FMRegressor(*[, featuresCol, labelCol, ...])

Factorization Machines learning algorithm for regression.

FMRegressionModel([java_model])

Model fitted byFMRegressor.

ChiSquareTest()

Conduct Pearson's independence test for every feature against the label.

Correlation()

Compute the correlation matrix for the input dataset of Vectors using the specified method.

KolmogorovSmirnovTest()

Conduct the two-sided Kolmogorov Smirnov (KS) test for data sampled from a continuous distribution.

MultivariateGaussian(mean, cov)

Represents a (mean, cov) tuple

Summarizer()

Tools for vectorized statistics on MLlib Vectors.

SummaryBuilder(jSummaryBuilder)

A builder object that provides summary statistics about a given column.

ParamGridBuilder()

Builder for a param grid used in grid search-based model selection.

CrossValidator(*[, estimator, ...])

K-fold cross validation performs model selection by splitting the dataset into a set of non-overlapping randomly partitioned folds which are used as separate training and test datasets e.g., with k=3 folds, K-fold cross validation will generate 3 (training, test) dataset pairs, each of which uses 2/3 of the data for training and 1/3 for testing.

CrossValidatorModel(bestModel[, avgMetrics, ...])

CrossValidatorModel contains the model with the highest average cross-validation metric across folds and uses this model to transform input data.

TrainValidationSplit(*[, estimator, ...])

Validation for hyper-parameter tuning.

TrainValidationSplitModel(bestModel[, ...])

Model from train validation split.

Evaluator()

Base class for evaluators that compute metrics from predictions.

BinaryClassificationEvaluator(*[, ...])

Evaluator for binary classification, which expects input columns rawPrediction, label and an optional weight column.

RegressionEvaluator(*[, predictionCol, ...])

Evaluator for Regression, which expects input columns prediction, label and an optional weight column.

MulticlassClassificationEvaluator(*[, ...])

Evaluator for Multiclass Classification, which expects input columns: prediction, label, weight (optional) and probabilityCol (only for logLoss).

MultilabelClassificationEvaluator(*[, ...])

Evaluator for Multilabel Classification, which expects two input columns: prediction and label.

ClusteringEvaluator(*[, predictionCol, ...])

Evaluator for Clustering results, which expects two input columns: prediction and features.

RankingEvaluator(*[, predictionCol, ...])

Evaluator for Ranking, which expects two input columns: prediction and label.

FPGrowth(*[, minSupport, minConfidence, ...])

A parallel FP-growth algorithm to mine frequent itemsets.

FPGrowthModel([java_model])

Model fitted by FPGrowth.

PrefixSpan(*[, minSupport, ...])

A parallel PrefixSpan algorithm to mine frequent sequential patterns.

ImageSchema

Internal class forpyspark.ml.image.ImageSchema attribute.

_ImageSchema()

Internal class forpyspark.ml.image.ImageSchema attribute.

TorchDistributor([num_processes, ...])

A class to support distributed training on PyTorch and PyTorch Lightning using PySpark.

DeepspeedTorchDistributor([numGpus, nnodes, ...])

BaseReadWrite()

Base class for MLWriter and MLReader.

DefaultParamsReadable()

Helper trait for making simpleParams types readable.

DefaultParamsReader(cls)

Specialization ofMLReader forParams types

DefaultParamsWritable()

Helper trait for making simpleParams types writable.

DefaultParamsWriter(instance)

Specialization ofMLWriter forParams types

GeneralMLWriter()

Utility class that can save ML instances in different formats.

HasTrainingSummary()

Base class for models that provides Training summary.

Identifiable()

Object with a unique ID.

MLReadable()

Mixin for instances that provideMLReader.

MLReader()

Utility class that can load ML instances.

MLWritable()

Mixin for ML instances that provideMLWriter.

MLWriter()

Utility class that can save ML instances.