Discriminative models, also referred to asconditional models, are a class of models frequently used forclassification. Inmachine learning, it typically models the conditional distribution P(Y∣X), or it learns a direct decision rule that maps inputs X to outputs Y. Discriminative models are commonly used forclassification andregression, where the main goal is accurate prediction on new data. They are typically used to solvebinary classification problems, i.e. assign labels, such as pass/fail, win/lose, alive/dead or healthy/sick, to existing datapoints. Discriminative models are usually trained to separate classes or to minimizeprediction error under a chosenloss function. They are often contrasted with generative models, which aim to model how the data are generated and can be used to sample new data.

Types of discriminative models includelogistic regression (LR),conditional random fields (CRFs),decision trees among many others.

Unlike generative modelling, which studies thejoint probability${\displaystyle P(x,y)}$, discriminative modeling studies the${\displaystyle P(y|x)}$ or maps the given unobserved variable (target)${\displaystyle x}$ to a class label${\displaystyle y}$ dependent on the observed variables (training samples). For example, inobject recognition,${\displaystyle x}$ is likely to be a vector of raw pixels (or features extracted from the raw pixels of the image). Within a probabilistic framework, this is done by modeling theconditional probability distribution${\displaystyle P(y|x)}$, which can be used for predicting${\displaystyle y}$ from${\displaystyle x}$. Note that there is still distinction between the conditional model and the discriminative model, though more often they are simply categorised as discriminative model.

Aconditional model models the conditionalprobability distribution, while the traditional discriminative model aims to optimize on mapping the input around the most similar trained samples.^[1]

Instatistical classification, two main approaches are called thegenerative approach and thediscriminative approach. These computeclassifiers by different approaches, differing in the degree ofstatistical modelling. Terminology is inconsistent,^[a] but three major types can be distinguished:

1. A generative model is astatistical model of thejoint probability distribution${\displaystyle P(X,Y)}$ on a givenobservable variableX andtarget variableY; A generative model can be used to "generate" random instances (outcomes) of an observationx.
2. Adiscriminative model is a model of theconditional probability${\displaystyle P(Y\mid X=x)}$ of the targetY, given an observationx. It can be used to "discriminate" the value of the target variableY, given an observationx.
3. Classifiers computed without using a probability model are also referred to loosely as "discriminative".

The distinction between these last two classes is not consistently made.

An alternative division defines these symmetrically as:

• agenerative model is a model of the conditional probability of the observableX, given a targety, symbolically,${\displaystyle P(X\mid Y=y)}$
• adiscriminative model is a model of the conditional probability of the targetY, given an observationx, symbolically,${\displaystyle P(Y\mid X=x)}$

Regardless of precise definition, the terminology is constitutional because a generative model can be used to "generate" random instances (outcomes), either of an observation and target${\displaystyle (x,y)}$, or of an observationx given a target valuey, while a discriminative model or discriminative classifier (without a model) can be used to "discriminate" the value of the target variableY, given an observationx.

Let's say we are given the${\displaystyle m}$ class labels (classification) and${\displaystyle n}$ feature variables,${\displaystyle Y:\{y_1,y_2,\dots ,y_m\},X:\{x_1,x_2,\dots ,x_n\}}$, as the training samples.

A generative model takes the joint probability${\displaystyle P(x,y)}$, where${\displaystyle x}$ is the input and${\displaystyle y}$ is the label, and predicts the most possible known label${\displaystyle \tilde{y}\in Y}$ for the unknown variable${\displaystyle \tilde{x}}$ usingBayes' theorem.

Discriminative models, as opposed togenerative models, do not allow one to generate samples from thejoint distribution of observed and target variables. However, for tasks such asclassification andregression that do not require the joint distribution, discriminative models can yield superior performance (in part because they have fewer variables to compute). On the other hand, generative models are typically more flexible than discriminative models in expressing dependencies in complex learning tasks. In addition, most discriminative models are inherentlysupervised and cannot easily supportunsupervised learning. Application-specific details ultimately dictate the suitability of selecting a discriminative versus generative model.

Discriminative models and generative models also differ in introducing theposterior possibility. To maintain the least expected loss, the minimization of result's misclassification should be acquired. In the discriminative model, the posterior probabilities,${\displaystyle P(y|x)}$, is inferred from a parametric model, where the parameters come from the training data. Points of estimation of the parameters are obtained from the maximization of likelihood or distribution computation over the parameters. On the other hand, considering that the generative models focus on the joint probability, the class posterior possibility${\displaystyle P(k)}$ is considered inBayes' theorem, which is

${\displaystyle P(y|x)=\frac{p(x|y)p(y)}{\sum _{i}p(x|i)p(i){\displaystyle }}=\frac{p(x|y)p(y)}{p(x)}}$.

In the repeated experiments, logistic regression and naive Bayes are applied here for different models on binary classification task, discriminative learning results in lower asymptotic errors, while generative one results in higher asymptotic errors faster. However, in Ulusoy and Bishop's joint work,Comparison of Generative and Discriminative Techniques for Object Detection and Classification, they state that the above statement is true only when the model is the appropriate one for data (i.e.the data distribution is correctly modeled by the generative model).

Significant advantages of using discriminative modeling are:

• Higher accuracy, which mostly leads to better learning result.
• Allows simplification of the input and provides a direct approach to${\displaystyle P(y|x)}$
• Saves calculation resource
• Generates lower asymptotic errors

Compared with the advantages of using generative modeling:

• Takes all data into consideration, which could result in slower processing as a disadvantage
• Requires fewer training samples
• A flexible framework that could easily cooperate with other needs of the application

• Training method usually requires multiple numerical optimization techniques
• Similarly by the definition, the discriminative model will need the combination of multiple subtasks for solving a complex real-world problem

The following approach is based on the assumption that it is given the training data-set${\displaystyle D=\{(x_i;y_i)|i\le N\in \mathbb{Z}\}}$, where${\displaystyle y_i}$ is the corresponding output for the input${\displaystyle x_i}$.^[2]

We intend to use the function${\displaystyle f(x)}$ to simulate the behavior of what we observed from the training data-set by thelinear classifier method. Using the joint feature vector${\displaystyle \varphi (x,y)}$, the decision function is defined as:

${\displaystyle f(x;w)=\arg \underset{y}{max}{w}^T\varphi (x,y)}$

According to Memisevic's interpretation,^[2]${\displaystyle w^T\varphi (x,y)}$, which is also${\displaystyle c(x,y;w)}$, computes a score which measures the compatibility of the input${\displaystyle x}$ with the potential output${\displaystyle y}$. Then the${\displaystyle \arg max}$ determines the class with the highest score.

Since the0-1 loss function is a commonly used one in the decision theory, the conditionalprobability distribution${\displaystyle P(y|x;w)}$, where${\displaystyle w}$ is a parameter vector for optimizing the training data, could be reconsidered as following for the logistics regression model:

${\displaystyle P(y|x;w)=\frac{1}{Z(x;w)}\exp (w^T\varphi (x,y))}$, with

${\displaystyle Z(x;w)=\sum _y{\displaystyle \exp (w^T\varphi (x,y))}}$

The equation above representslogistic regression. Notice that a major distinction between models is their way of introducing posterior probability. Posterior probability is inferred from the parametric model. We then can maximize the parameter by following equation:

${\displaystyle L(w)=\sum _i{\displaystyle \log p(y^i|x^i;w)}}$

It could also be replaced by thelog-loss equation below:

${\displaystyle l^{\log }(x^i,y^i,c(x^i;w))=-\log p(y^i|x^i;w)=\log Z(x^i;w)-w^T\varphi (x^i,y^i)}$

Since thelog-loss is differentiable, a gradient-based method can be used to optimize the model. A global optimum is guaranteed because the objective function is convex. The gradient of log likelihood is represented by:

${\displaystyle \frac{\mathrm{\partial }L(w)}{\mathrm{\partial }w}=\sum _i{\displaystyle \varphi (x^i,y^i)-E_{p(y|x^i;w)}\varphi (x^i,y)}}$

where${\displaystyle E_{p(y|x^i;w)}}$ is the expectation of${\displaystyle p(y|x^i;w)}$.

The above method will provide efficient computation for the relative small number of classification.

Since both advantages and disadvantages present on the two way of modeling, combining both approaches will be a good modeling in practice. For example, in Marras' articleA Joint Discriminative Generative Model for Deformable Model Construction and Classification,^[3] he and his coauthors apply the combination of two modelings on face classification of the models, and receive a higher accuracy than the traditional approach.

Similarly, Kelm^[4] also proposed the combination of two modelings for pixel classification in his articleCombining Generative and Discriminative Methods for Pixel Classification with Multi-Conditional Learning.

During the process of extracting the discriminative features prior to the clustering,Principal component analysis (PCA), though commonly used, is not a necessarily discriminative approach. In contrast, LDA is a discriminative one.^[5]Linear discriminant analysis (LDA), provides an efficient way of eliminating the disadvantage we list above. As we know, the discriminative model needs a combination of multiple subtasks before classification, and LDA provides appropriate solution towards this problem by reducing dimension.

• Empirical risk minimization
• Common loss functions (log loss, hinge loss, squared loss)
• Regularization (L1/L2)
• Optimization methods (gradient descent family)

Examples of discriminative models include:

• Logistic regression, a type ofgeneralized linear regression used for predictingbinary orcategorical outputs (also known asmaximum entropy classifiers)
• Boosting (meta-algorithm)
• Conditional random fields
• Linear regression
• Computer vision
• Random forests
• k-nearest neighbors algorithm
• Support Vector Machines
• Decision Tree Learning
• Maximum-entropy Markov models

• Generative model

• Jebara, Tony (2004).Machine Learning: Discriminative and Generative. The Springer International Series in Engineering and Computer Science. Kluwer Academic (Springer).ISBN 978-1-4020-7647-3.
• Mitchell, Tom M. (2015)."3. Generative and Discriminative Classifiers: Naive Bayes and Logistic Regression"(PDF).Machine Learning.
• Ng, Andrew Y.;Jordan, Michael I. (2002)."On discriminative vs. generative classifiers: A comparison of logistic regression and naive bayes"(PDF).Advances in Neural Information Processing Systems.

• Regression models

• All articles with dead external links
• Articles with dead external links from November 2019
• Articles with permanently dead external links
• Articles with short description
• Short description matches Wikidata