Inmathematical statistics, theKullback–Leibler (KL)divergence (also calledrelative entropy andI-divergence^[1]), denoted${\displaystyle D_{\text{KL}}(P\parallel Q)}$, is a type ofstatistical distance: a measure of how much an approximatingprobability distributionQ is different from a true probability distributionP.^[2][3] Mathematically, it is defined as

$${\displaystyle D_{\text{KL}}(P\parallel Q)=\sum _{x\in \mathcal{X}}P(x)\log \frac{P(x)}{Q(x)}\text{.}}$$

A simpleinterpretation of the KL divergence ofP fromQ is theexpected excesssurprisal from using the approximationQ instead ofP when the actual isP. While it is a measure of how different two distributions are and is thus a distance in some sense, it is not actually ametric, which is the most familiar and formal type of distance. In particular, it is not symmetric in the two distributions (in contrast tovariation of information), and does not satisfy thetriangle inequality. Instead, in terms ofinformation geometry, it is a type ofdivergence,^[4] a generalization ofsquared distance, and for certain classes of distributions (notably anexponential family), it satisfies a generalizedPythagorean theorem (which applies to squared distances).^[5]

Relative entropy is always a non-negativereal number, with value 0 if and only if the two distributions in question are identical. It has diverse applications, both theoretical, such as characterizing the relative(Shannon) entropy in information systems, randomness in continuoustime-series, and information gain when comparing statistical models ofinference; and practical, such as applied statistics,fluid mechanics,neuroscience,bioinformatics, andmachine learning.

Consider two probability distributions, a trueP and an approximatingQ. Often,P represents the data, the observations, or a measured probability distribution and distributionQ represents instead a theory, a model, a description, or another approximation ofP. However, sometimes the true distributionP represents a model and the approximating distributionQ represents (simulated) data that are intended to match the true distribution. The Kullback–Leibler divergence${\displaystyle D_{\text{KL}}(P\parallel Q)}$ is then interpreted as the average difference of the number of bits required for encoding samples ofP using acode optimized forQ rather than one optimized forP.

Note that the roles ofP andQ can be reversed in some situations where that is easier to compute and the goal is to minimize${\displaystyle D_{\text{KL}}(P\parallel Q)}$, such as with theexpectation–maximization algorithm (EM) andevidence lower bound (ELBO) computations. This role-reversal approach exploits that${\displaystyle D_{\text{KL}}(P\parallel Q)=0}$ if and only if${\displaystyle D_{\text{KL}}(Q\parallel P)=0}$ and that, in many cases, reducing one has the effect of reducing the other.

The relative entropy was introduced bySolomon Kullback andRichard Leibler inKullback & Leibler (1951) as "the mean information for discrimination between${\displaystyle H_1}$ and${\displaystyle H_2}$ per observation from${\displaystyle {\mu }_1}$",^[6] where one is comparing two probability measures${\displaystyle {\mu }_1,{\mu }_2}$, and${\displaystyle H_1,H_2}$ are the hypotheses that one is selecting from measure${\displaystyle {\mu }_1,{\mu }_2}$ (respectively). They denoted this by${\displaystyle I(1:2)}$, and defined the "'divergence' between${\displaystyle {\mu }_1}$ and${\displaystyle {\mu }_2}$" as the symmetrized quantity${\displaystyle J(1,2)=I(1:2)+I(2:1)}$, which had already been defined and used byHarold Jeffreys in 1948.^[7] InKullback (1959), the symmetrized form is again referred to as the "divergence", and the relative entropies in each direction are referred to as a "directed divergences" between two distributions;^[8] Kullback preferred the termdiscrimination information.^[9] The term "divergence" is in contrast to a distance (metric), since the symmetrized divergence does not satisfy the triangle inequality.^[10] Numerous references to earlier uses of the symmetrized divergence and to otherstatistical distances are given inKullback (1959, pp. 6–7, §1.3 Divergence). The asymmetric "directed divergence" has come to be known as the Kullback–Leibler divergence, while the symmetrized "divergence" is now referred to as theJeffreys divergence.

Fordiscrete probability distributionsP andQ defined on the samesample space,${\displaystyle \mathcal{X}}$, the relative entropy fromQ toP is defined^[11] to be

$${\displaystyle D_{\text{KL}}(P\parallel Q)=\sum _{x\in \mathcal{X}}P(x)\log \frac{P(x)}{Q(x)}\text{,}}$$

which is equivalent to

$${\displaystyle D_{\text{KL}}(P\parallel Q)=\left(-\sum _{x\in \mathcal{X}}P(x)\log Q(x)\right)-\left(-\sum _{x\in \mathcal{X}}P(x)\log P(x)\right)\text{.}}$$

In other words, it is theexpectation of the logarithmic difference between the probabilitiesP andQ, where the expectation is taken using the probabilitiesP.

Relative entropy is only defined in this way if, for allx,${\displaystyle Q(x)=0}$ implies${\displaystyle P(x)=0}$ (absolute continuity). Otherwise, it is often defined as${\displaystyle +\mathrm{\infty }}$,^[1] but the value${\displaystyle +\mathrm{\infty }}$ is possible even if${\displaystyle Q(x)\ne 0}$ everywhere,^[12][13] provided that${\displaystyle \mathcal{X}}$ is infinite in extent. Analogous comments apply to the continuous and general measure cases defined below.

Whenever${\displaystyle P(x)}$ is zero the contribution of the corresponding term is interpreted as zero because

$${\displaystyle \underset{x\to 0^{+}}{lim}x\log (x)=0\text{.}}$$

For distributionsP andQ of acontinuous random variable, relative entropy is defined to be the integral^[14]

$${\displaystyle D_{\text{KL}}(P\parallel Q)={\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}p(x)\log \frac{p(x)}{q(x)}dx\text{.}}$$

wherep andq denote theprobability densities ofP andQ.

More generally, ifP andQ are probabilitymeasures on ameasurable space${\displaystyle \mathcal{X},}$ andP isabsolutely continuous with respect toQ, then the relative entropy fromQ toP is defined as

$${\displaystyle D_{\text{KL}}(P\parallel Q)={\int }_{x\in \mathcal{X}}\log \frac{P(dx)}{Q(dx)}P(dx)\text{,}}$$

where${\displaystyle \frac{P(dx)}{Q(dx)}}$ is theRadon–Nikodym derivative ofP with respect toQ, i.e. the uniqueQ almost everywhere defined functionr on${\displaystyle \mathcal{X}}$ such that${\displaystyle P(dx)=r(x)Q(dx)}$ which exists becauseP is absolutely continuous with respect toQ. Also we assume the expression on the right-hand side exists. Equivalently (by thechain rule), this can be written as

$${\displaystyle D_{\text{KL}}(P\parallel Q)={\int }_{x\in \mathcal{X}}\frac{P(dx)}{Q(dx)}\log \frac{P(dx)}{Q(dx)}Q(dx)\text{,}}$$

which is theentropy ofP relative toQ. Continuing in this case, if${\displaystyle \mu }$ is any measure on${\displaystyle \mathcal{X}}$ for which densitiesp andq with${\displaystyle P(dx)=p(x)\mu (dx)}$ and${\displaystyle Q(dx)=q(x)\mu (dx)}$ exist (meaning thatP andQ are both absolutely continuous with respect to${\displaystyle \mu }$), then the relative entropy fromQ toP is given as

$${\displaystyle D_{\text{KL}}(P\parallel Q)={\int }_{x\in \mathcal{X}}p(x)\log \frac{p(x)}{q(x)}\mu (dx)\text{.}}$$

Note that such a measure${\displaystyle \mu }$ for which densities can be defined always exists, since one can take$\mu =\frac{1}{2}\left(P+Q\right)$ although in practice it will usually be one that applies in the context such ascounting measure for discrete distributions, orLebesgue measure or a convenient variant thereof such asGaussian measure or the uniform measure on thesphere,Haar measure on aLie group etc. for continuous distributions.The logarithms in these formulae are usually taken tobase 2 if information is measured in units ofbits, or to basee if information is measured innats. Most formulas involving relative entropy hold regardless of the base of the logarithm.

Various conventions exist for referring to${\displaystyle D_{\text{KL}}(P\parallel Q)}$ in words. Often it is referred to as the divergencebetweenP andQ, but this fails to convey the fundamental asymmetry in the relation. Sometimes, as in this article, it may be described as the divergence ofPfromQ or as the divergencefromQtoP. This reflects theasymmetry inBayesian inference, which startsfrom apriorQ and updatesto theposteriorP. Another common way to refer to${\displaystyle D_{\text{KL}}(P\parallel Q)}$ is as the relative entropy ofPwith respect toQ or the information gain fromP overQ.

Kullback^[3] gives the following example (Table 2.1, Example 2.1). LetP andQ be the distributions shown in the table and figure.P is the distribution on the left side of the figure, abinomial distribution with${\displaystyle N=2}$ and${\displaystyle p=0.4}$.Q is the distribution on the right side of the figure, adiscrete uniform distribution with the three possible outcomesx =0,1,2 (i.e.${\displaystyle \mathcal{X}=\{0,1,2\}}$), each with probability${\displaystyle p=1/3}$.

x Distribution	0	1	2
${\displaystyle P(x)}$	⁠9/25⁠	⁠12/25⁠	⁠4/25⁠
${\displaystyle Q(x)}$	⁠1/3⁠	⁠1/3⁠	⁠1/3⁠

Relative entropies${\displaystyle D_{\text{KL}}(P\parallel Q)}$ and${\displaystyle D_{\text{KL}}(Q\parallel P)}$ are calculated as follows. This example uses thenatural log with basee, designatedln to get results innats (seeunits of information):

In the field of statistics, theNeyman–Pearson lemma states that the most powerful way to distinguish between the two distributionsP andQ based on an observationY (drawn from one of them) is through the log of the ratio of their likelihoods:${\displaystyle \log P(Y)-\log Q(Y)}$. The KL divergence is the expected value of this statistic ifY is actually drawn fromP. Kullback motivated the statistic as an expected log likelihood ratio.^[15]

In the context ofcoding theory,${\displaystyle D_{\text{KL}}(P\parallel Q)}$ can be constructed by measuring the expected number of extrabits required tocode samples fromP using a code optimized forQ rather than the code optimized forP.

In the context ofmachine learning,${\displaystyle D_{\text{KL}}(P\parallel Q)}$ is often called theinformation gain achieved ifP would be used instead ofQ which is currently used. By analogy with information theory, it is called therelative entropy ofP with respect toQ.

Expressed in the language ofBayesian inference,${\displaystyle D_{\text{KL}}(P\parallel Q)}$ is a measure of the information gained by revising one's beliefs from theprior probability distributionQ to theposterior probability distributionP. In other words, it is the amount of information lost whenQ is used to approximateP.^[16]

In applications,P typically represents the "true" distribution of data, observations, or a precisely calculated theoretical distribution, whileQ typically represents a theory, model, description, orapproximation ofP. In order to find a distributionQ that is closest toP, we can minimize the KL divergence and compute aninformation projection.

While it is astatistical distance, it is not ametric, the most familiar type of distance, but instead it is adivergence.^[4] While metrics are symmetric and generalizelinear distance, satisfying thetriangle inequality, divergences are asymmetric and generalizesquared distance, in some cases satisfying a generalizedPythagorean theorem. In general${\displaystyle D_{\text{KL}}(P\parallel Q)}$ does not equal${\displaystyle D_{\text{KL}}(Q\parallel P)}$, and the asymmetry is an important part of the geometry.^[4] Theinfinitesimal form of relative entropy, specifically itsHessian, gives ametric tensor that equals theFisher information metric; see§ Fisher information metric. Fisher information metric on the certain probability distribution let determine the natural gradient for information-geometric optimization algorithms.^[17] Its quantum version is Fubini-study metric.^[18] Relative entropy satisfies a generalized Pythagorean theorem forexponential families (geometrically interpreted asdually flat manifolds), and this allows one to minimize relative entropy by geometric means, for example byinformation projection and inmaximum likelihood estimation.^[5]

The relative entropy is theBregman divergence generated by the negative entropy, but it is also of the form of anf-divergence. For probabilities over a finitealphabet, it is unique in being a member of both of these classes ofstatistical divergences. The application of Bregman divergence can be found in mirror descent.^[19]

Consider a growth-optimizing investor in a fair game with mutually exclusive outcomes(e.g. a "horse race" in which the official odds add up to one).The rate of return expected by such an investor is equal to the relative entropybetween the investor's believed probabilities and the official odds.^[20]This is a special case of a much more general connection between financial returns and divergence measures.^[21]

Financial risks are connected to${\displaystyle D_{\text{KL}}}$ via information geometry.^[22] Investors' views, the prevailing market view, and risky scenarios form triangles on the relevant manifold of probability distributions. The shape of the triangles determines key financial risks (both qualitatively and quantitatively). For instance, obtuse triangles in which investors' views and risk scenarios appear on "opposite sides" relative to the market describe negative risks, acute triangles describe positive exposure, and the right-angled situation in the middle corresponds to zero risk. Extending this concept, relative entropy can be hypothetically utilised to identify the behaviour of informed investors, if one takes this to be represented by the magnitude and deviations away from the prior expectations of fund flows, for example.^[23]

In information theory, theKraft–McMillan theorem establishes that any directly decodable coding scheme for coding a message to identify one value${\displaystyle x_i}$ out of a set of possibilitiesX can be seen as representing an implicit probability distribution${\displaystyle q(x_i)=2^{-{\ell }_i}}$ overX, where${\displaystyle {\ell }_i}$ is the length of the code for${\displaystyle x_i}$ in bits. Therefore, relative entropy can be interpreted as the expected extra message-length per datum that must be communicated if a code that is optimal for a given (wrong) distributionQ is used, compared to using a code based on the true distributionP: it is theexcess entropy.

where${\displaystyle \mathrm{H}(P,Q)}$ is thecross entropy ofQ relative toP and${\displaystyle \mathrm{H}(P)}$ is theentropy ofP (which is the same as the cross-entropy of P with itself).

The relative entropy${\displaystyle D_{\text{KL}}(P\parallel Q)}$ can be thought of geometrically as astatistical distance, a measure of how far the distributionQ is from the distributionP. Geometrically it is adivergence: an asymmetric, generalized form of squared distance. The cross-entropy${\displaystyle H(P,Q)}$ is itself such a measurement (formally aloss function), but it cannot be thought of as a distance, since${\displaystyle H(P,P)=:H(P)}$ is not zero. This can be fixed by subtracting${\displaystyle H(P)}$ to make${\displaystyle D_{\text{KL}}(P\parallel Q)}$ agree more closely with our notion of distance, as theexcess loss. The resulting function is asymmetric, and while this can be symmetrized (see§ Symmetrised divergence), the asymmetric form is more useful. See§ Interpretations for more on the geometric interpretation.

Relative entropy relates to "rate function" in the theory oflarge deviations.^[24][25]

Arthur Hobson proved that relative entropy is the only measure of difference between probability distributions that satisfies some desired properties, which are the canonical extension to those appearing in a commonly usedcharacterization of entropy.^[26] Consequently,mutual information is the only measure of mutual dependence that obeys certain related conditions, since it can be definedin terms of Kullback–Leibler divergence.

• Relative entropy is alwaysnon-negative,$${\displaystyle D_{\text{KL}}(P\parallel Q)\ge 0,}$$ a result known asGibbs' inequality, with${\displaystyle D_{\text{KL}}(P\parallel Q)}$ equals zeroif and only if${\displaystyle P=Q}$ as measures.

In particular, if${\displaystyle P(dx)=p(x)\mu (dx)}$ and${\displaystyle Q(dx)=q(x)\mu (dx)}$, then${\displaystyle p(x)=q(x)}$${\displaystyle \mu }$-almost everywhere. The entropy${\displaystyle \mathrm{H}(P)}$ thus sets a minimum value for the cross-entropy${\displaystyle \mathrm{H}(P,Q)}$, theexpected number ofbits required when using a code based onQ rather thanP; and the Kullback–Leibler divergence therefore represents the expected number of extra bits that must be transmitted to identify a valuex drawn fromX, if a code is used corresponding to the probability distributionQ, rather than the "true" distributionP.

• No upper-bound exists for the general case. However, it is shown that ifP andQ are two discrete probability distributions built by distributing the same discrete quantity, then the maximum value of${\displaystyle D_{\text{KL}}(P\parallel Q)}$ can be calculated.^[27]
• Relative entropy remains well-defined for continuous distributions, and furthermore is invariant underparameter transformations. For example, if a transformation is made from variablex to variable${\displaystyle y(x)}$, then, since${\displaystyle P(dx)=p(x)dx=\tilde{p}(y)dy=\tilde{p}(y(x))\left|\frac{dy}{dx}(x)\right|dx}$ and${\displaystyle Q(dx)=q(x)dx=\tilde{q}(y)dy=\tilde{q}(y)\left|\frac{dy}{dx}(x)\right|dx}$ where${\displaystyle \left|\frac{dy}{dx}(x)\right|}$ is the absolute value of the derivative or more generally of theJacobian, the relative entropy may be rewritten: where${\displaystyle y_a=y(x_a)}$ and${\displaystyle y_b=y(x_b)}$. Although it was assumed that the transformation was continuous, this need not be the case. This also shows that the relative entropy produces adimensionally consistent quantity, since ifx is a dimensioned variable,${\displaystyle p(x)}$ and${\displaystyle q(x)}$ are also dimensioned, since e.g.${\displaystyle P(dx)=p(x)dx}$ is dimensionless. The argument of the logarithmic term is and remains dimensionless, as it must. It can therefore be seen as in some ways a more fundamental quantity than some other properties in information theory^[28] (such asself-information orShannon entropy), which can become undefined or negative for non-discrete probabilities.
• Relative entropy isadditive forindependent distributions in much the same way as Shannon entropy. If${\displaystyle P_1,P_2}$ are independent distributions, and${\displaystyle P(dx,dy)=P_1(dx)P_2(dy)}$, and likewise${\displaystyle Q(dx,dy)=Q_1(dx)Q_2(dy)}$ for independent distributions${\displaystyle Q_1,Q_2}$ then$${\displaystyle D_{\text{KL}}(P\parallel Q)=D_{\text{KL}}(P_1\parallel Q_1)+D_{\text{KL}}(P_2\parallel Q_2).}$$
• Relative entropy${\displaystyle D_{\text{KL}}(P\parallel Q)}$ isconvex in the pair ofprobability measures${\displaystyle (P,Q)}$, i.e. if${\displaystyle (P_1,Q_1)}$ and${\displaystyle (P_2,Q_2)}$ are two pairs of probability measures then$${\displaystyle D_{\text{KL}}(\lambda P_1+(1-\lambda )P_2\parallel \lambda Q_1+(1-\lambda )Q_2)\le \lambda D_{\text{KL}}(P_1\parallel Q_1)+(1-\lambda )D_{\text{KL}}(P_2\parallel Q_2)\text{ for }0\le \lambda \le 1.}$$
• ${\displaystyle D_{\text{KL}}(P\parallel Q)}$ may be Taylor expanded about its minimum (i.e.${\displaystyle P=Q}$) as$${\displaystyle D_{\text{KL}}(P\parallel Q)=\sum _{n=2}^{\mathrm{\infty }}\frac{1}{n(n-1)}\sum _{x\in \mathcal{X}}\frac{(Q(x)-P(x){)}^n}{Q(x{)}^{n-1}}}$$ which converges if and only if${\displaystyle P\le 2Q}$almost surely w.r.t${\displaystyle Q}$.

The following result, due to Donsker and Varadhan,^[29] is known asDonsker and Varadhan's variational formula.

Suppose that we have twomultivariate normal distributions, with means${\displaystyle {\mu }_0,{\mu }_1}$ and with (non-singular)covariance matrices${\displaystyle {\mathrm{\Sigma }}_0,{\mathrm{\Sigma }}_1.}$ If the two distributions have the same dimension,k, then the relative entropy between the distributions is as follows:^[30]

$${\displaystyle D_{\text{KL}}\left({\mathcal{N}}_0\parallel {\mathcal{N}}_1\right)=\frac{1}{2}\left[\mathrm{tr}\left({\mathrm{\Sigma }}_1^{-1}{\mathrm{\Sigma }}_0\right)-k+{\left({\mu }_1-{\mu }_0\right)}^{\mathsf{T}}{\mathrm{\Sigma }}_1^{-1}\left({\mu }_1-{\mu }_0\right)+\ln \frac{det{\mathrm{\Sigma }}_1}{det{\mathrm{\Sigma }}_0}\right]\text{.}}$$

Thelogarithm in the last term must be taken to basee since all terms apart from the last are base-e logarithms of expressions that are either factors of the density function or otherwise arise naturally. The equation therefore gives a result measured innats. Dividing the entire expression above by${\displaystyle \ln (2)}$ yields the divergence inbits.

In a numerical implementation, it is helpful to express the result in terms of the Cholesky decompositions${\displaystyle L_0,L_1}$ such that${\displaystyle {\mathrm{\Sigma }}_0=L_0{L}_0^T}$ and${\displaystyle {\mathrm{\Sigma }}_1=L_1{L}_1^T}$. Then withM andy solutions to the triangular linear systems${\displaystyle L_{1}M=L_0}$, and${\displaystyle L_{1}y={\mu }_1-{\mu }_0}$,

$${\displaystyle D_{\text{KL}}\left({\mathcal{N}}_0\parallel {\mathcal{N}}_1\right)=\frac{1}{2}\left(\sum _{i,j=1}^k{\left(M_{ij}\right)}^2-k+|y{|}^2+2\sum _{i=1}^k\ln \frac{(L_1{)}_{ii}}{(L_0{)}_{ii}}\right)\text{.}}$$

A special case, and a common quantity invariational inference, is the relative entropy between a diagonal multivariate normal, and a standard normal distribution (with zero mean and unit variance):

$${\displaystyle D_{\text{KL}}\left(\mathcal{N}\left({\left({\mu }_1,\dots ,{\mu }_k\right)}^{\mathsf{T}},\mathrm{diag}\left({\sigma }_1^2,\dots ,{\sigma }_k^2\right)\right)\parallel \mathcal{N}\left(\mathbf{0},\mathbf{I}\right)\right)=\frac{1}{2}\sum _{i=1}^k\left[{\sigma }_i^2+{\mu }_i^2-1-\ln \left({\sigma }_i^2\right)\right]\text{.}}$$

For two univariate normal distributionsp andq the above simplifies to^[31]$${\displaystyle D_{\text{KL}}\left(\mathcal{p}\parallel \mathcal{q}\right)=\log \frac{{\sigma }_1}{{\sigma }_0}+\frac{{\sigma }_0^2+{\left({\mu }_0-{\mu }_1\right)}^2}{2{\sigma }_1^2}-\frac{1}{2}}$$

In the case of co-centered normal distributions with${\displaystyle k={\sigma }_1/{\sigma }_0}$, this simplifies^[32] to:

$${\displaystyle D_{\text{KL}}\left(\mathcal{p}\parallel \mathcal{q}\right)={\log }_{2}k+(k^{-2}-1)/2/\ln (2)\mathrm{b}\mathrm{i}\mathrm{t}\mathrm{s}}$$

Consider two uniform distributions, with the support of${\displaystyle p=[A,B]}$ enclosed within${\displaystyle q=[C,D]}$ (). Then the information gain is:

$${\displaystyle D_{\text{KL}}\left(\mathcal{p}\parallel \mathcal{q}\right)=\log \frac{D-C}{B-A}}$$

Intuitively,^[32] the information gain to ak times narrower uniform distribution contains${\displaystyle {\log }_{2}k}$ bits. This connects with the use of bits in computing, where${\displaystyle {\log }_{2}k}$ bits would be needed to identify one element of ak long stream.

Theexponential family of distribution is given by

$${\displaystyle p_X(x|\theta )=h(x)\exp \left({\theta }^{\mathsf{T}}T(x)-A(\theta )\right)}$$

where${\displaystyle h(x)}$ is reference measure,${\displaystyle T(x)}$ issufficient statistics,${\displaystyle \theta }$ is canonical natural parameters, and${\displaystyle A(\theta )}$ is the log-partition function.

The KL divergence between two distributions${\displaystyle p(x|{\theta }_1)}$ and${\displaystyle p(x|{\theta }_2)}$ is given by^[33]

$${\displaystyle D_{\text{KL}}({\theta }_1\parallel {\theta }_2)={\left({\theta }_1-{\theta }_2\right)}^{\mathsf{T}}{\mu }_1-A({\theta }_1)+A({\theta }_2)}$$

where${\displaystyle {\mu }_1=E_{{\theta }_1}[T(X)]=\mathrm{\nabla }A({\theta }_1)}$ is the mean parameter of${\displaystyle p(x|{\theta }_1)}$.

For example, for thePoisson distribution with mean${\displaystyle \lambda }$, the sufficient statistics${\displaystyle T(x)=x}$, the natural parameter${\displaystyle \theta =\log \lambda }$, and log partition function${\displaystyle A(\theta )=e^{\theta }}$. As such, the divergence between two Poisson distributions with means${\displaystyle {\lambda }_1}$ and${\displaystyle {\lambda }_2}$ is

$${\displaystyle D_{\text{KL}}({\lambda }_1\parallel {\lambda }_2)={\lambda }_1\log \frac{{\lambda }_1}{{\lambda }_2}-{\lambda }_1+{\lambda }_2\text{.}}$$

As another example, for a normal distribution with unit variance${\displaystyle N(\mu ,1)}$, the sufficient statistics${\displaystyle T(x)=x}$, the natural parameter${\displaystyle \theta =\mu }$, and log partition function${\displaystyle A(\theta )={\mu }^2/2}$. Thus, the divergence between two normal distributions${\displaystyle N({\mu }_1,1)}$ and${\displaystyle N({\mu }_2,1)}$ is

$${\displaystyle D_{\text{KL}}({\mu }_1\parallel {\mu }_2)=\left({\mu }_1-{\mu }_2\right){\mu }_1-\frac{{\mu }_1^2}{2}+\frac{{\mu }_2^2}{2}=\frac{{\left({\mu }_2-{\mu }_1\right)}^2}{2}\text{.}}$$

As final example, the divergence between a normal distribution with unit variance${\displaystyle N(\mu ,1)}$ and a Poisson distribution with mean${\displaystyle \lambda }$ is

$${\displaystyle D_{\text{KL}}(\mu \parallel \lambda )=(\mu -\log \lambda )\mu -\frac{{\mu }^2}{2}+\lambda \text{.}}$$

While relative entropy is astatistical distance, it is not ametric on the space of probability distributions, but instead it is adivergence.^[4] While metrics are symmetric and generalizelinear distance, satisfying thetriangle inequality, divergences are asymmetric in general and generalizesquared distance, in some cases satisfying a generalizedPythagorean theorem. In general${\displaystyle D_{\text{KL}}(P\parallel Q)}$ does not equal${\displaystyle D_{\text{KL}}(Q\parallel P)}$, and while this can be symmetrized (see§ Symmetrised divergence), the asymmetry is an important part of the geometry.^[4]

It generates atopology on the space ofprobability distributions. More concretely, if${\displaystyle \{P_1,P_2,\dots \}}$ is a sequence of distributions such that

$${\displaystyle \underset{n\to \mathrm{\infty }}{lim}{D}_{\text{KL}}(P_n\parallel Q)=0\text{,}}$$

then it is said that

$${\displaystyle P_n\stackrel{D}{\to }Q\text{.}}$$

Pinsker's inequality entails that

$${\displaystyle P_n\stackrel{D}{\to }P\Rightarrow P_n\stackrel{TV}{\to }P\text{,}}$$

where the latter stands for the usual convergence intotal variation.

Relative entropy is directly related to theFisher information metric. This can be made explicit as follows. Assume that the probability distributionsP andQ are both parameterized by some (possibly multi-dimensional) parameter${\displaystyle \theta }$. Consider then two close by values of${\displaystyle P=P(\theta )}$ and${\displaystyle Q=P({\theta }_0)}$ so that the parameter${\displaystyle \theta }$ differs by only a small amount from the parameter value${\displaystyle {\theta }_0}$. Specifically, up to first order one has (using theEinstein summation convention)$${\displaystyle P(\theta )=P({\theta }_0)+\mathrm{\Delta }{\theta }_j{P}_j({\theta }_0)+\cdots }$$

with${\displaystyle \mathrm{\Delta }{\theta }_j=(\theta -{\theta }_0{)}_j}$ a small change of${\displaystyle \theta }$ in thej direction, and${\displaystyle P_j\left({\theta }_0\right)=\frac{\mathrm{\partial }P}{\mathrm{\partial }{\theta }_j}({\theta }_0)}$ the corresponding rate of change in the probability distribution. Since relative entropy has an absolute minimum 0 for${\displaystyle P=Q}$, i.e.${\displaystyle \theta ={\theta }_0}$, it changes only tosecond order in the small parameters${\displaystyle \mathrm{\Delta }{\theta }_j}$. More formally, as for any minimum, the first derivatives of the divergence vanish

$${\displaystyle {\frac{\mathrm{\partial }}{\mathrm{\partial }{\theta }_j}|}_{\theta ={\theta }_0}{D}_{\text{KL}}(P(\theta )\parallel P({\theta }_0))=0,}$$

and by theTaylor expansion one has up to second order

$${\displaystyle D_{\text{KL}}(P(\theta )\parallel P({\theta }_0))=\frac{1}{2}\mathrm{\Delta }{\theta }_j\mathrm{\Delta }{\theta }_k{g}_{jk}({\theta }_0)+\cdots }$$

where theHessian matrix of the divergence

$${\displaystyle g_{jk}({\theta }_0)={\frac{{\mathrm{\partial }}^2}{\mathrm{\partial }{\theta }_j\mathrm{\partial }{\theta }_k}|}_{\theta ={\theta }_0}{D}_{\text{KL}}(P(\theta )\parallel P({\theta }_0))}$$

must bepositive semidefinite. Letting${\displaystyle {\theta }_0}$ vary (and dropping the subindex 0) the Hessian${\displaystyle g_{jk}(\theta )}$ defines a (possibly degenerate)Riemannian metric on theθparameter space, called the Fisher information metric.

There is an associated theorem.^[3]When${\displaystyle p_{(x,\rho )}}$ satisfies the following regularity conditions:

$${\displaystyle \frac{\mathrm{\partial }\log (p)}{\mathrm{\partial }\rho },\frac{{\mathrm{\partial }}^2\log (p)}{\mathrm{\partial }{\rho }^2},\frac{{\mathrm{\partial }}^3\log (p)}{\mathrm{\partial }{\rho }^3}}$$ exist,

whereξ is independent ofρ$${\displaystyle {{\int }_{x=0}^{\mathrm{\infty }}\frac{\mathrm{\partial }p(x,\rho )}{\mathrm{\partial }\rho }|}_{\rho =0}dx={{\int }_{x=0}^{\mathrm{\infty }}\frac{{\mathrm{\partial }}^{2}p(x,\rho )}{\mathrm{\partial }{\rho }^2}|}_{\rho =0}dx=0}$$

then:$${\displaystyle \mathcal{D}(p(x,0)\parallel p(x,\rho ))=\frac{c{\rho }^2}{2}+\mathcal{O}\left({\rho }^3\right)\text{ as }\rho \to 0\text{.}}$$

Another information-theoretic metric isvariation of information, which is roughly a symmetrization ofconditional entropy. It is a metric on the set ofpartitions of a discreteprobability space.

MAUVE is a measure of the statistical gap between two text distributions, such as the difference between text generated by a model and human-written text. This measure is computed using Kullback–Leibler divergences between the two distributions in a quantized embedding space of a foundation model.

Many of the other quantities of information theory can be interpreted as applications of relative entropy to specific cases.

Theself-information, also known as theinformation content of a signal, random variable, orevent is defined as the negative logarithm of theprobability of the given outcome occurring.

When applied to adiscrete random variable, the self-information can be represented as^{[citation needed]}

$${\displaystyle \mathrm{I}(m)=D_{\text{KL}}\left({\delta }_{\text{im}}\parallel \{p_i\}\right),}$$

is the relative entropy of the probability distribution${\displaystyle P(i)}$ from aKronecker delta representing certainty that${\displaystyle i=m}$ — i.e. the number of extra bits that must be transmitted to identifyi if only the probability distribution${\displaystyle P(i)}$ is available to the receiver, not the fact that${\displaystyle i=m}$.

Themutual information,

is the relative entropy of thejoint probability distribution${\displaystyle P_{X,Y}(x,y)}$ from the product${\displaystyle (P_X\cdot P_Y)(x,y)=P_X(x)P_Y(y)}$ of the twomarginal probability distributions — i.e. the expected number of extra bits that must be transmitted to identifyX andY if they are coded using only their marginal distributions instead of the joint distribution.

TheShannon entropy,

is the number of bits which would have to be transmitted to identifyX fromN equally likely possibilities,less the relative entropy of the uniform distribution on therandom variates ofX,${\displaystyle P_U(X)}$, from the true distribution${\displaystyle P(X)}$ — i.e.less the expected number of bits saved, which would have had to be sent if the value ofX were coded according to the uniform distribution${\displaystyle P_U(X)}$ rather than the true distribution${\displaystyle P(X)}$. This definition of Shannon entropy forms the basis ofE.T. Jaynes's alternative generalization to continuous distributions, thelimiting density of discrete points (as opposed to the usualdifferential entropy), which defines the continuous entropy as$${\displaystyle \underset{N\to \mathrm{\infty }}{lim}{H}_N(X)=\log N-\int p(x)\log \frac{p(x)}{m(x)}dx\text{,}}$$which is equivalent to:$${\displaystyle \log (N)-D_{\text{KL}}(p(x)||m(x))}$$