Inmachine learning, alinear classifier makes aclassification decision for each object based on alinear combination of itsfeatures. Such classifiers work well for practical problems such asdocument classification, and more generally for problems with many variables (features), reaching accuracy levels comparable to non-linear classifiers while taking less time to train and use.^[1]

If the input feature vector to the classifier is areal vector${\displaystyle \overrightarrow{x}}$, then the output score is

${\displaystyle y=f(\overrightarrow{w}\cdot \overrightarrow{x})=f\left(\sum _j{w}_j{x}_j\right),}$

where${\displaystyle \overrightarrow{w}}$ is a real vector of weights andf is a function that converts thedot product of the two vectors into the desired output. (In other words,${\displaystyle \overrightarrow{w}}$ is aone-form orlinear functional mapping${\displaystyle \overrightarrow{x}}$ ontoR.) The weight vector${\displaystyle \overrightarrow{w}}$ is learned from a set of labeled training samples. Oftenf is athreshold function, which maps all values of${\displaystyle \overrightarrow{w}\cdot \overrightarrow{x}}$ above a certain threshold to the first class and all other values to the second class; e.g.,

The superscript T indicates the transpose and${\displaystyle \theta }$ is a scalar threshold. A more complexf might give the probability that an item belongs to a certain class.

For a two-class classification problem, one can visualize the operation of a linear classifier as splitting ahigh-dimensional input space with ahyperplane: all points on one side of the hyperplane are classified as "yes", while the others are classified as "no".

A linear classifier is often used in situations where the speed of classification is an issue, since it is often the fastest classifier, especially when${\displaystyle \overrightarrow{x}}$ is sparse. Also, linear classifiers often work very well when the number of dimensions in${\displaystyle \overrightarrow{x}}$ is large, as indocument classification, where each element in${\displaystyle \overrightarrow{x}}$ is typically the number of occurrences of a word in a document (seedocument-term matrix). In such cases, the classifier should be well-regularized.

There are two broad classes of methods for determining the parameters of a linear classifier${\displaystyle \overrightarrow{w}}$. They can begenerative anddiscriminative models.^[2][3] Methods of the former modeljoint probability distribution, whereas methods of the latter modelconditional density functions${\displaystyle P(\mathrm{c}\mathrm{l}\mathrm{a}\mathrm{s}\mathrm{s}|\overrightarrow{x})}$. Examples of such algorithms include:

• Linear Discriminant Analysis (LDA)—assumesGaussian conditional density models
• Naive Bayes classifier with multinomial or multivariate Bernoulli event models.

The second set of methods includesdiscriminative models, which attempt to maximize the quality of the output on atraining set. Additional terms in the training cost function can easily performregularization of the final model. Examples of discriminative training of linear classifiers include:

• Logistic regression—maximum likelihood estimation of${\displaystyle \overrightarrow{w}}$ assuming that the observed training set was generated by a binomial model that depends on the output of the classifier.
• Perceptron—an algorithm that attempts to fix all errors encountered in the training set
• Fisher's Linear Discriminant Analysis—an algorithm (different than "LDA") that maximizes the ratio of between-class scatter to within-class scatter, without any other assumptions. It is in essence a method of dimensionality reduction for binary classification.^[4]
• Support vector machine—an algorithm that maximizes themargin between the decision hyperplane and the examples in the training set.

Note: Despite its name, LDA does not belong to the class of discriminative models in this taxonomy. However, its name makes sense when we compare LDA to the other main lineardimensionality reduction algorithm:principal components analysis (PCA). LDA is asupervised learning algorithm that utilizes the labels of the data, while PCA is anunsupervised learning algorithm that ignores the labels. To summarize, the name is a historical artifact.^[5]

Discriminative training often yields higher accuracy than modeling the conditional density functions^{[citation needed]}. However, handling missing data is often easier with conditional density models^{[citation needed]}.

All of the linear classifier algorithms listed above can be converted into non-linear algorithms operating on a different input space${\displaystyle \phi (\overrightarrow{x})}$, using thekernel trick.

Discriminative training of linear classifiers usually proceeds in asupervised way, by means of anoptimization algorithm that is given a training set with desired outputs and aloss function that measures the discrepancy between the classifier's outputs and the desired outputs. Thus, the learning algorithm solves an optimization problem of the form^[1]

${\displaystyle \underset{\mathbf{w}}{\arg min}R(\mathbf{w})+C\sum _{i=1}^{N}L(y_i,{\mathbf{w}}^{\mathsf{T}}{\mathbf{x}}_i)}$

where

• w is a vector of classifier parameters,
• L(y_i,w^Tx_i) is a loss function that measures the discrepancy between the classifier's prediction and the true outputy_i for thei'th training example,
• R(w) is aregularization function that prevents the parameters from getting too large (causingoverfitting), and
• C is a scalar constant (set by the user of the learning algorithm) that controls the balance between the regularization and the loss function.

Popular loss functions include thehinge loss (for linear SVMs) and thelog loss (for linear logistic regression). If the regularization functionR isconvex, then the above is aconvex problem.^[1] Many algorithms exist for solving such problems; popular ones for linear classification include (stochastic)gradient descent,L-BFGS,coordinate descent andNewton methods.

• Backpropagation
• Linear regression
• Perceptron
• Quadratic classifier
• Support vector machines
• Winnow (algorithm)

1. Y. Yang, X. Liu, "A re-examination of text categorization", Proc. ACM SIGIR Conference, pp. 42–49, (1999).paper @ citeseer
2. R. Herbrich, "Learning Kernel Classifiers: Theory and Algorithms," MIT Press, (2001).ISBN 0-262-08306-X

• Classification algorithms
• Statistical classification

• Articles with short description
• Short description matches Wikidata
• All articles with unsourced statements
• Articles with unsourced statements from October 2017