Ininformation geometry, adivergence is a kind ofstatistical distance: abinary function which establishes the separation from oneprobability distribution to another on astatistical manifold.

The simplest divergence issquared Euclidean distance (SED), and divergences can be viewed as generalizations of SED. The other most important divergence isrelative entropy (also calledKullback–Leibler divergence), which is central toinformation theory. There are numerous other specific divergences and classes of divergences, notablyf-divergences andBregman divergences (see§ Examples).

Given adifferentiable manifold^[a]${\displaystyle M}$ of dimension${\displaystyle n}$, adivergence on${\displaystyle M}$ is a${\displaystyle C^2}$-function${\displaystyle D:M\times M\to [0,\mathrm{\infty })}$ satisfying:^[1][2]

1. ${\displaystyle D(p,q)\ge 0}$ for all${\displaystyle p,q\in M}$ (non-negativity),
2. ${\displaystyle D(p,q)=0}$ if and only if${\displaystyle p=q}$ (positivity),
3. At every point${\displaystyle p\in M}$,${\displaystyle D(p,p+dp)}$ is a positive-definitequadratic form for infinitesimal displacements${\displaystyle dp}$ from${\displaystyle p}$.

In applications to statistics, the manifold${\displaystyle M}$ is typically the space of parameters of aparametric family of probability distributions.

Condition 3 means that${\displaystyle D}$ defines an inner product on the tangent space${\displaystyle T_{p}M}$ for every${\displaystyle p\in M}$. Since${\displaystyle D}$ is${\displaystyle C^2}$ on${\displaystyle M}$, this defines a Riemannian metric${\displaystyle g}$ on${\displaystyle M}$.

Locally at${\displaystyle p\in M}$, we may construct a localcoordinate chart with coordinates${\displaystyle x}$, then the divergence is$${\displaystyle D(x(p),x(p)+dx)=\frac{1}{2}d{x}^T{g}_p(x)dx+O(|dx{|}^3)}$$where${\displaystyle g_p(x)}$ is a matrix of size${\displaystyle n\times n}$. It is the Riemannian metric at point${\displaystyle p}$ expressed in coordinates${\displaystyle x}$.

Dimensional analysis of condition 3 shows that divergence has the dimension of squared distance.^[3]

Thedual divergence${\displaystyle D^{\ast }}$ is defined as

${\displaystyle D^{\ast }(p,q)=D(q,p).}$

When we wish to contrast${\displaystyle D}$ against${\displaystyle D^{\ast }}$, we refer to${\displaystyle D}$ asprimal divergence.

Given any divergence${\displaystyle D}$, its symmetrized version is obtained by averaging it with its dual divergence:^[3]

${\displaystyle D_S(p,q)=\frac{1}{2}(D(p,q)+D(q,p)).}$

Unlikemetrics, divergences are not required to be symmetric, and the asymmetry is important in applications.^[3] Accordingly, one often refers asymmetrically to the divergence "ofq fromp" or "fromp toq", rather than "betweenp andq". Secondly, divergences generalizesquared distance, not linear distance, and thus do not satisfy thetriangle inequality, but some divergences (such as theBregman divergence) do satisfy generalizations of thePythagorean theorem.

In general statistics and probability, "divergence" generally refers to any kind of function${\displaystyle D(p,q)}$, where${\displaystyle p,q}$ are probability distributions or other objects under consideration, such that conditions 1, 2 are satisfied. Condition 3 is required for "divergence" as used in information geometry.

As an example, thetotal variation distance, a commonly used statistical divergence, does not satisfy condition 3.

Notation for divergences varies significantly between fields, though there are some conventions.

Divergences are generally notated with an uppercase 'D', as in${\displaystyle D(x,y)}$, to distinguish them from metric distances, which are notated with a lowercase 'd'. When multiple divergences are in use, they are commonly distinguished with subscripts, as in${\displaystyle D_{\text{KL}}}$ forKullback–Leibler divergence (KL divergence).

Often a different separator between parameters is used, particularly to emphasize the asymmetry. Ininformation theory, a double bar is commonly used:${\displaystyle D(p\parallel q)}$; this is similar to, but distinct from, the notation forconditional probability,${\displaystyle P(A|B)}$, and emphasizes interpreting the divergence as a relative measurement, as inrelative entropy; this notation is common for the KL divergence. A colon may be used instead,^[b] as${\displaystyle D(p:q)}$; this emphasizes the relative information supporting the two distributions.

The notation for parameters varies as well. Uppercase${\displaystyle P,Q}$ interprets the parameters as probability distributions, while lowercase${\displaystyle p,q}$ or${\displaystyle x,y}$ interprets them geometrically as points in a space, and${\displaystyle {\mu }_1,{\mu }_2}$ or${\displaystyle m_1,m_2}$ interprets them as measures.

Many properties of divergences can be derived if we restrictS to be a statistical manifold, meaning that it can be parametrized with a finite-dimensional coordinate systemθ, so that for a distributionp ∈S we can writep =p(θ).

For a pair of pointsp,q ∈S with coordinatesθ_p andθ_q, denote the partial derivatives ofD(p,q) as

Now we restrict these functions to a diagonalp =q, and denote^[4]

By definition, the functionD(p,q) is minimized atp =q, and therefore

where matrixg^(D) ispositive semi-definite and defines a uniqueRiemannian metric on the manifoldS.

DivergenceD(·, ·) also defines a uniquetorsion-freeaffine connection ∇^(D) with coefficients

${\displaystyle {\mathrm{\Gamma }}_{ij,k}^{(D)}=-D[{\mathrm{\partial }}_i{\mathrm{\partial }}_j,{\mathrm{\partial }}_k],}$

and thedual to this connection ∇* is generated by the dual divergenceD*.

Thus, a divergenceD(·, ·) generates on a statistical manifold a unique dualistic structure (g^(D), ∇^(D), ∇^(D*)). The converse is also true: every torsion-free dualistic structure on a statistical manifold is induced from some globally defined divergence function (which however need not be unique).^[5]

For example, whenD is anf-divergence^[6] for some function ƒ(·), then it generates themetricg^(D_f) =c·g and the connection∇^(D_f) = ∇^(α), whereg is the canonicalFisher information metric, ∇^(α) is the α-connection,c = ƒ′′(1), andα = 3 + 2ƒ′′′(1)/ƒ′′(1).

The two most important divergences are therelative entropy (Kullback–Leibler divergence, KL divergence), which is central toinformation theory and statistics, and thesquared Euclidean distance (SED). Minimizing these two divergences is the main way thatlinear inverse problems are solved, via theprinciple of maximum entropy andleast squares, notably inlogistic regression andlinear regression.^[7]

The two most important classes of divergences are thef-divergences andBregman divergences; however, other types of divergence functions are also encountered in the literature. The only divergence for probabilities over a finitealphabet that is both anf-divergence and a Bregman divergence is the Kullback–Leibler divergence.^[8] The squared Euclidean divergence is a Bregman divergence (corresponding to the function⁠${\displaystyle x^2}$⁠) but not anf-divergence.

Given a convex function${\displaystyle f:[0,+\mathrm{\infty })\to (-\mathrm{\infty },+\mathrm{\infty }]}$ such that${\displaystyle f(0)=\underset{t\to 0^{+}}{lim}f(t),f(1)=0}$, thef-divergence generated by${\displaystyle f}$ is defined as

${\displaystyle D_f(p,q)=\int p(x)f(\frac{q(x)}{p(x)})dx}$.

Kullback–Leibler divergence:	${\displaystyle D_{\mathrm{K}\mathrm{L}}(p,q)=\int p(x)\ln \left(\frac{p(x)}{q(x)}\right)dx}$
squaredHellinger distance:	${\displaystyle H^2(p,q)=2\int (\sqrt{p(x)}-\sqrt{q(x)}{)}^{2}dx}$
Jensen–Shannon divergence:	${\displaystyle D_{JS}(p,q)=\frac{1}{2}\int p(x)\ln \left(p(x)\right)+q(x)\ln \left(q(x)\right)-(p(x)+q(x))\ln \left(\frac{p(x)+q(x)}{2}\right)dx}$
α-divergence	${\displaystyle D^{(\alpha )}(p,q)=\frac{4}{1-{\alpha }^2}(1-\int p(x{)}^{\frac{1-\alpha }{2}}q(x{)}^{\frac{1+\alpha }{2}}dx)}$
chi-squared divergence:	${\displaystyle D_{{\chi }^2}(p,q)=\int \frac{(p(x)-q(x){)}^2}{p(x)}dx}$
(α,β)-product divergence[citation needed]:	${\displaystyle D_{\alpha ,\beta }(p,q)=\frac{2}{(1-\alpha )(1-\beta )}\int (1-(\frac{q(x)}{p(x)}{)}^{\frac{1-\alpha }{2}})(1-(\frac{q(x)}{p(x)}{)}^{\frac{1-\beta }{2}})p(x)dx}$

Bregman divergences correspond to convex functions on convex sets. Given astrictly convex, continuously differentiable functionF on aconvex set, known as theBregman generator, theBregman divergence measures the convexity of: the error of the linear approximation ofF fromq as an approximation of the value atp:

${\displaystyle D_F(p,q)=F(p)-F(q)-⟨\mathrm{\nabla }F(q),p-q⟩.}$

The dual divergence to a Bregman divergence is the divergence generated by theconvex conjugateF^* of the Bregman generator of the original divergence. For example, for the squared Euclidean distance, the generator is⁠${\displaystyle x^2}$⁠, while for the relative entropy the generator is thenegative entropy⁠${\displaystyle x\log x}$⁠.

The use of the term "divergence" – both what functions it refers to, and what various statistical distances are called – has varied significantly over time, but by c. 2000 had settled on the current usage within information geometry, notably in the textbookAmari & Nagaoka (2000).^[1]

The term "divergence" for a statistical distance was used informally in various contexts from c. 1910 to c. 1940. Its formal use dates at least toBhattacharyya (1943), entitled "On a measure of divergence between two statistical populations defined by their probability distributions", which defined theBhattacharyya distance, andBhattacharyya (1946), entitled "On a Measure of Divergence between Two Multinomial Populations", which defined theBhattacharyya angle. The term was popularized by its use for theKullback–Leibler divergence inKullback & Leibler (1951) and its use in the textbookKullback (1959). The term "divergence" was used generally byAli & Silvey (1966) for statistically distances. Numerous references to earlier uses ofstatistical distances are given inAdhikari & Joshi (1956) andKullback (1959, pp. 6–7, §1.3 Divergence).

Kullback & Leibler (1951) actually used "divergence" to refer to thesymmetrized divergence (this function had already been defined and used byHarold Jeffreys in 1948^[9]), referring to the asymmetric function as "the mean information for discrimination ... per observation",^[10] whileKullback (1959) referred to the asymmetric function as the "directed divergence".^[11]Ali & Silvey (1966) referred generally to such a function as a "coefficient of divergence", and showed that many existing functions could be expressed asf-divergences, referring to Jeffreys' function as "Jeffreys' measure of divergence" (today "Jeffreys divergence"), and Kullback–Leibler's asymmetric function (in each direction) as "Kullback's and Leibler's measures of discriminatory information" (today "Kullback–Leibler divergence").^[12]

The information geometry definition of divergence (the subject of this article) was initially referred to by alternative terms, including "quasi-distance"Amari (1982, p. 369) and "contrast function"Eguchi (1985), though "divergence" was used inAmari (1985) for theα-divergence, and has become standard for the general class.^[1][2]

The term "divergence" is in contrast to a distance (metric), since the symmetrized divergence does not satisfy the triangle inequality.^[13] For example, the term "Bregman distance" is still found, but "Bregman divergence" is now preferred.

Notationally,Kullback & Leibler (1951) denoted their asymmetric function as${\displaystyle I(1:2)}$, whileAli & Silvey (1966) denote their functions with a lowercase 'd' as${\displaystyle d\left(P_1,P_2\right)}$.

• Statistical distance

• Statistical distance
• F-divergences

• Articles with short description
• Short description matches Wikidata
• All articles with unsourced statements
• Articles with unsourced statements from May 2022