Information theory
Entropy Differential entropy Conditional entropy Joint entropy Mutual information Directed information Conditional mutual information Relative entropy Entropy rate Limiting density of discrete points
Asymptotic equipartition property Rate–distortion theory
Shannon's source coding theorem Channel capacity Noisy-channel coding theorem Shannon–Hartley theorem
v t e

Ininformation theory, thecross-entropy between twoprobability distributions${\displaystyle p}$ and${\displaystyle q}$, over the same underlying set of events, measures the average number ofbits needed to identify an event drawn from the set when the coding scheme used for the set is optimized for an estimated probability distribution${\displaystyle q}$, rather than the true distribution${\displaystyle p}$.

The cross-entropy of the distribution${\displaystyle q}$ relative to a distribution${\displaystyle p}$ over a given set is defined as follows:

$${\displaystyle H(p,q)=-{\mathrm{E}}_p[\log q],}$$

where${\displaystyle E_p[\cdot ]}$ is theexpected value operator with respect to the distribution${\displaystyle p}$.

The definition may be formulated using theKullback–Leibler divergence${\displaystyle D_{\mathrm{K}\mathrm{L}}(p\parallel q)}$, divergence of${\displaystyle p}$ from${\displaystyle q}$ (also known as therelative entropy of${\displaystyle p}$ with respect to${\displaystyle q}$).

$${\displaystyle H(p,q)=H(p)+D_{\mathrm{K}\mathrm{L}}(p\parallel q),}$$

where${\displaystyle H(p)}$ is theentropy of${\displaystyle p}$.

Fordiscrete probability distributions${\displaystyle p}$ and${\displaystyle q}$ with the samesupport${\displaystyle \mathcal{X}}$, this means

The situation forcontinuous distributions is analogous. We have to assume that${\displaystyle p}$ and${\displaystyle q}$ areabsolutely continuous with respect to some referencemeasure${\displaystyle r}$ (usually${\displaystyle r}$ is aLebesgue measure on aBorelσ-algebra). Let${\displaystyle P}$ and${\displaystyle Q}$ be probability density functions of${\displaystyle p}$ and${\displaystyle q}$ with respect to${\displaystyle r}$. Then

$${\displaystyle -{\int }_{\mathcal{X}}P(x)\log Q(x)\mathrm{d}x={\mathrm{E}}_p[-\log Q],}$$

and therefore

NB: The notation${\displaystyle H(p,q)}$ is also used for a different concept, thejoint entropy of${\displaystyle p}$ and${\displaystyle q}$.

Ininformation theory, theKraft–McMillan theorem establishes that any directly decodable coding scheme for coding a message to identify one value${\displaystyle x_i}$ out of a set of possibilities${\displaystyle \{x_1,\dots ,x_n\}}$ can be seen as representing an implicit probability distribution${\displaystyle q(x_i)={\left(\frac{1}{2}\right)}^{{\ell }_i}}$ over${\displaystyle \{x_1,\dots ,x_n\}}$, where${\displaystyle {\ell }_i}$ is the length of the code for${\displaystyle x_i}$ in bits. Therefore, cross-entropy can be interpreted as the expected message-length per datum when a wrong distribution${\displaystyle q}$ is assumed while the data actually follows a distribution${\displaystyle p}$. That is why the expectation is taken over the true probability distribution${\displaystyle p}$ and not${\displaystyle q.}$ Indeed the expected message-length under the true distribution${\displaystyle p}$ is

$${\displaystyle {\mathrm{E}}_p[\ell ]=-{\mathrm{E}}_p\left[\frac{\ln q(x)}{\ln (2)}\right]}$$$${\displaystyle =-{\mathrm{E}}_p\left[{\log }_{2}q(x)\right]=-\sum _{x_i}p(x_i){\log }_{2}q(x_i)}$$$${\displaystyle =-\sum _{x}p(x){\log }_{2}q(x)=H(p,q).}$$

There are many situations where cross-entropy needs to be measured but the distribution of${\displaystyle p}$ is unknown. An example islanguage modeling, where a model is created based on a training set${\displaystyle T}$, and then its cross-entropy is measured on a test set to assess how accurate the model is in predicting the test data. In this example,${\displaystyle p}$ is the true distribution of words in any corpus, and${\displaystyle q}$ is the distribution of words as predicted by the model. Since the true distribution is unknown, cross-entropy cannot be directly calculated. In these cases, an estimate of cross-entropy is calculated using the following formula:

$${\displaystyle H(T,q)=-\sum _{i=1}^N\frac{1}{N}{\log }_{2}q(x_i)}$$

where${\displaystyle N}$ is the size of the test set, and${\displaystyle q(x)}$ is the probability of event${\displaystyle x}$ estimated from the training set. In other words,${\displaystyle q(x_i)}$ is the probability estimate of the model that the i-th word of the text is${\displaystyle x_i}$. The sum is averaged over the${\displaystyle N}$ words of the test. This is aMonte Carlo estimate of the true cross-entropy, where the test set is treated as samples from${\displaystyle p(x)}$.^{[citation needed]}

The cross entropy arises in classification problems when introducing a logarithm in the guise of thelog-likelihood function.

The section is concerned with the subject of estimation of the probability of different possible discrete outcomes. To this end, denote a parametrized family of distributions by${\displaystyle q_{\theta }}$, with${\displaystyle \theta }$ subject to the optimization effort. Consider a given finite sequence of${\displaystyle N}$ values${\displaystyle x_i}$ from a training set, obtained fromconditionally independent sampling. The likelihood assigned to any considered parameter${\displaystyle \theta }$ of the model is then given by the product over all probabilities${\displaystyle q_{\theta }(X=x_i)}$.Repeated occurrences are possible, leading to equal factors in the product. If the count of occurrences of the value equal to${\displaystyle x_i}$ (for some index${\displaystyle i}$) is denoted by${\displaystyle \mathrm{\#}{x}_i}$, then the frequency of that value equals${\displaystyle \mathrm{\#}{x}_i/N}$. Denote the latter by${\displaystyle p(X=x_i)}$, as it may be understood as empirical approximation to the probability distribution underlying the scenario. Further denote by${\displaystyle PP:={\mathrm{e}}^{H(p,q_{\theta })}}$ theperplexity, which can be seen to equal$\prod _{x_i}{q}_{\theta }(X=x_i{)}^{-p(X=x_i)}$ by thecalculation rules for the logarithm, and where the product is over the values without double counting. So$${\displaystyle \mathcal{L}(\theta ;\mathbf{x})=\prod _i{q}_{\theta }(X=x_i)=\prod _{x_i}{q}_{\theta }(X=x_i{)}^{\mathrm{\#}{x}_i}=P{P}^{-N}={\mathrm{e}}^{-N\cdot H(p,q_{\theta })}}$$or$${\displaystyle \log \mathcal{L}(\theta ;\mathbf{x})=-N\cdot H(p,q_{\theta }).}$$Since the logarithm is amonotonically increasing function, it does not affect extremization. So observe that thelikelihood maximization amounts to minimization of the cross-entropy.

Cross-entropy minimization is frequently used in optimization and rare-event probability estimation. When comparing a distribution${\displaystyle q}$ against a fixed reference distribution${\displaystyle p}$, cross-entropy andKL divergence are identical up to an additive constant (since${\displaystyle p}$ is fixed): According to theGibbs' inequality, both take on their minimal values when${\displaystyle p=q}$, which is${\displaystyle 0}$ for KL divergence, and${\displaystyle \mathrm{H}(p)}$ for cross-entropy. In the engineering literature, the principle of minimizing KL divergence (Kullback's "Principle of Minimum Discrimination Information") is often called thePrinciple of Minimum Cross-Entropy (MCE), orMinxent.

However, as discussed in the articleKullback–Leibler divergence, sometimes the distribution${\displaystyle q}$ is the fixed prior reference distribution, and the distribution${\displaystyle p}$ is optimized to be as close to${\displaystyle q}$ as possible, subject to some constraint. In this case the two minimizations arenot equivalent. This has led to some ambiguity in the literature, with some authors attempting to resolve the inconsistency by restating cross-entropy to be${\displaystyle D_{\mathrm{K}\mathrm{L}}(p\parallel q)}$, rather than${\displaystyle H(p,q)}$. In fact, cross-entropy is another name forrelative entropy; see Cover and Thomas^[1] and Good.^[2] On the other hand,${\displaystyle H(p,q)}$ does not agree with the literature and can be misleading.

Cross-entropy can be used to define a loss function inmachine learning andoptimization. Mao, Mohri, and Zhong (2023) give an extensive analysis of the properties of the family of cross-entropy loss functions in machine learning, including theoretical learning guarantees and extensions toadversarial learning.^[3] The true probability${\displaystyle p_i}$ is the true label, and the given distribution${\displaystyle q_i}$ is the predicted value of the current model. This is also known as thelog loss (orlogarithmic loss^[4] orlogistic loss);^[5] the terms "log loss" and "cross-entropy loss" are used interchangeably.^[6]

More specifically, consider abinary regression model which can be used to classify observations into two possible classes (often simply labelled${\displaystyle 0}$ and${\displaystyle 1}$). The output of the model for a given observation, given a vector of input features${\displaystyle x}$, can be interpreted as a probability, which serves as the basis for classifying the observation. Inlogistic regression, the probability is modeled using thelogistic function${\displaystyle g(z)=1/(1+e^{-z})}$ where${\displaystyle z}$ is some function of the input vector${\displaystyle x}$, commonly just a linear function. The probability of the output${\displaystyle y=1}$ is given by$${\displaystyle q_{y=1}=\hat{y}\equiv g(\mathbf{w}\cdot \mathbf{x})=\frac{1}{1+e^{-\mathbf{w}\cdot \mathbf{x}}},}$$where the vector of weights${\displaystyle \mathbf{w}}$ is optimized through some appropriate algorithm such asgradient descent. Similarly, the complementary probability of finding the output${\displaystyle y=0}$ is simply given by$${\displaystyle q_{y=0}=1-\hat{y}.}$$

Having set up our notation,${\displaystyle p\in \{y,1-y\}}$ and${\displaystyle q\in \{\hat{y},1-\hat{y}\}}$, we can use cross-entropy to get a measure of dissimilarity between${\displaystyle p}$ and${\displaystyle q}$:$${\displaystyle H(p,q)=-\sum _i{p}_i\log q_i=-y\log \hat{y}-(1-y)\log (1-\hat{y}).}$$

Logistic regression typically optimizes the log loss for all the observations on which it is trained, which is the same as optimizing the average cross-entropy in the sample. Other loss functions that penalize errors differently can be also used for training, resulting in models with different final test accuracy.^[7] For example, suppose we have${\displaystyle N}$ samples with each sample indexed by${\displaystyle n=1,\dots ,N}$. Theaverage of the loss function is then given by:

$${\displaystyle J(\mathbf{w})=\frac{1}{N}\sum _{n=1}^{N}H(p_n,q_n)=-\frac{1}{N}\sum _{n=1}^N[y_n\log {\hat{y}}_n+(1-y_n)\log (1-{\hat{y}}_n)],}$$

where${\displaystyle {\hat{y}}_n\equiv g(\mathbf{w}\cdot {\mathbf{x}}_n)=1/(1+e^{-\mathbf{w}\cdot {\mathbf{x}}_n})}$, with${\displaystyle g(z)}$ the logistic function as before.

The logistic loss is sometimes called cross-entropy loss. It is also known as log loss.^{[duplication?]} (In this case, the binary label is often denoted by {−1,+1}.^[8])

Remark: The gradient of the cross-entropy loss for logistic regression is the same as the gradient of the squared-error loss forlinear regression. That is, define

$${\displaystyle \widehat{y_i}=\hat{f}(x_{i1},\dots ,x_{ip})=\frac{1}{1+\exp (-{\beta }_0-{\beta }_1{x}_{i1}-\cdots -{\beta }_p{x}_{ip})},}$$$${\displaystyle L(\mathit{\beta })=-\sum _{i=1}^N\left[y_i\log {\hat{y}}_i+(1-y_i)\log (1-{\hat{y}}_i)\right].}$$

Then we have the result

$${\displaystyle \frac{\mathrm{\partial }}{\mathrm{\partial }\mathit{\beta }}L(\mathit{\beta })=X^T(\hat{Y}-Y).}$$

The proof is as follows. For any${\displaystyle {\hat{y}}_i}$, we have

$${\displaystyle \frac{\mathrm{\partial }}{\mathrm{\partial }{\beta }_0}\ln \frac{1}{1+e^{-{\beta }_0+k_0}}=\frac{e^{-{\beta }_0+k_0}}{1+e^{-{\beta }_0+k_0}},}$$$${\displaystyle \frac{\mathrm{\partial }}{\mathrm{\partial }{\beta }_0}\ln \left(1-\frac{1}{1+e^{-{\beta }_0+k_0}}\right)=\frac{-1}{1+e^{-{\beta }_0+k_0}},}$$$${\displaystyle \frac{\mathrm{\partial }}{\mathrm{\partial }{\beta }_1}\ln \frac{1}{1+e^{-{\beta }_1{x}_{i1}+k_1}}=\frac{x_{i1}{e}^{k_1}}{e^{{\beta }_1{x}_{i1}}+e^{k_1}},}$$$${\displaystyle \frac{\mathrm{\partial }}{\mathrm{\partial }{\beta }_1}\ln \left[1-\frac{1}{1+e^{-{\beta }_1{x}_{i1}+k_1}}\right]=\frac{-x_{i1}{e}^{{\beta }_1{x}_{i1}}}{e^{{\beta }_1{x}_{i1}}+e^{k_1}},}$$$${\displaystyle \frac{\mathrm{\partial }}{\mathrm{\partial }{\beta }_1}L(\mathit{\beta })=-\sum _{i=1}^N{x}_{i1}(y_i-{\hat{y}}_i)=\sum _{i=1}^N{x}_{i1}({\hat{y}}_i-y_i).}$$

In a similar way, we eventually obtain the desired result.

It may be beneficial to train an ensemble of models that have diversity, such that when they are combined, their predictive accuracy is augmented.^[9][10]Assuming a simple ensemble of${\displaystyle K}$ classifiers is assembled via averaging the outputs, then the amended cross-entropy is given by$${\displaystyle e^k=H(p,q^k)-\frac{\lambda }{K}\sum _{j\ne k}H(q^j,q^k)}$$where${\displaystyle e^k}$ is the cost function of the${\displaystyle k^{th}}$ classifier,${\displaystyle q^k}$ is the output probability of the${\displaystyle k^{th}}$ classifier,${\displaystyle p}$ is the true probability to be estimated, and${\displaystyle \lambda }$ is a parameter between 0 and 1 that defines the 'diversity' that we would like to establish among the ensemble. When${\displaystyle \lambda =0}$ we want each classifier to do its best regardless of the ensemble and when${\displaystyle \lambda =1}$ we would like the classifier to be as diverse as possible.

• Cross-entropy method
• Logistic regression
• Conditional entropy
• Kullback–Leibler distance
• Maximum-likelihood estimation
• Mutual information
• Perplexity

• de Boer, Kroese, D.P., Mannor, S. and Rubinstein, R.Y. (2005).A tutorial on the cross-entropy method.Annals of Operations Research 134 (1), 19–67.

• Entropy and information
• Loss functions

• Articles with short description
• Short description is different from Wikidata
• All articles with unsourced statements
• Articles with unsourced statements from May 2019
• All articles with minor POV problems
• Articles with minor POV problems from November 2023