Inprobability theory andinformation theory, themutual information (MI) of tworandom variables is a measure of the mutualdependence between the two variables. More specifically, it quantifies the "amount of information" (inunits such asshannons (bits),nats orhartleys) obtained about one random variable by observing the other random variable. The concept of mutual information is intimately linked to that ofentropy of a random variable, a fundamental notion in information theory that quantifies the expected "amount of information" held in a random variable.

Not limited to real-valued random variables and linear dependence like thecorrelation coefficient, MI is more general and determines how different thejoint distribution of the pair${\displaystyle (X,Y)}$ is from the product of the marginal distributions of${\displaystyle X}$ and${\displaystyle Y}$. MI is theexpected value of thepointwise mutual information (PMI).

The quantity was defined and analyzed byClaude Shannon in his landmark paper "A Mathematical Theory of Communication", although he did not call it "mutual information". This term was coined later byRobert Fano.^[2] Mutual Information is also known asinformation gain.

Let${\displaystyle (X,Y)}$ be a pair ofrandom variables with values over the space${\displaystyle \mathcal{X}\times \mathcal{Y}}$. If their joint distribution is${\displaystyle P_{(X,Y)}}$ and the marginal distributions are${\displaystyle P_X}$ and${\displaystyle P_Y}$, the mutual information is defined as

${\displaystyle I(X;Y)=D_{\mathrm{K}\mathrm{L}}(P_{(X,Y)}\parallel P_X\otimes P_Y)}$

where${\displaystyle D_{\mathrm{K}\mathrm{L}}}$ is theKullback–Leibler divergence, and${\displaystyle P_X\otimes P_Y}$ is theouter product distribution which assigns probability${\displaystyle P_X(x)\cdot P_Y(y)}$ to each${\displaystyle (x,y)}$.

Expressed in terms of theentropy${\displaystyle H(\cdot )}$ and theconditional entropy${\displaystyle H(\cdot |\cdot )}$ of the random variables${\displaystyle X}$ and${\displaystyle Y}$, one also has (Seerelation to conditional and joint entropy):

${\displaystyle I(X;Y)=H(X)-H(X|Y)=H(Y)-H(Y|X)}$

Notice, as per property of theKullback–Leibler divergence, that${\displaystyle I(X;Y)}$ is equal to zero precisely when the joint distribution coincides with the product of the marginals, i.e. when${\displaystyle X}$ and${\displaystyle Y}$ are independent (and hence observing${\displaystyle Y}$ tells you nothing about${\displaystyle X}$).${\displaystyle I(X;Y)}$ is non-negative. It is a measure of the price for encoding${\displaystyle (X,Y)}$ as a pair of independent random variables when in reality they are not.

If thenatural logarithm is used, the unit of mutual information is thenat. If thelog base 2 is used, the unit of mutual information is theshannon, also known as the bit. If thelog base 10 is used, the unit of mutual information is thehartley, also known as the ban or the dit.

The mutual information of two jointly discrete random variables${\displaystyle X}$ and${\displaystyle Y}$ is calculated as a double sum:^[3]: 20

${\displaystyle \mathrm{I}(X;Y)=\sum _{y\in \mathcal{Y}}\sum _{x\in \mathcal{X}}{P}_{(X,Y)}(x,y)\log \left(\frac{P_{(X,Y)}(x,y)}{P_X(x)P_Y(y)}\right)}$,

where${\displaystyle P_{(X,Y)}}$ is thejoint probabilitymass function of${\displaystyle X}$ and${\displaystyle Y}$, and${\displaystyle P_X}$ and${\displaystyle P_Y}$ are themarginal probability mass functions of${\displaystyle X}$ and${\displaystyle Y}$ respectively.

In the case of jointly continuous random variables, the double sum is replaced by adouble integral:^[3]: 251

${\displaystyle \mathrm{I}(X;Y)={\int }_{\mathcal{Y}}{\int }_{\mathcal{X}}{P}_{(X,Y)}(x,y)\log \left(\frac{P_{(X,Y)}(x,y)}{P_X(x)P_Y(y)}\right)dxdy}$,

where${\displaystyle P_{(X,Y)}}$ is now the joint probabilitydensity function of${\displaystyle X}$ and${\displaystyle Y}$, and${\displaystyle P_X}$ and${\displaystyle P_Y}$ are the marginal probability density functions of${\displaystyle X}$ and${\displaystyle Y}$ respectively.

Intuitively, mutual information measures the information that${\displaystyle X}$ and${\displaystyle Y}$ share: It measures how much knowing one of these variables reduces uncertainty about the other. For example, if${\displaystyle X}$ and${\displaystyle Y}$ are independent, then knowing${\displaystyle X}$ does not give any information about${\displaystyle Y}$ and vice versa, so their mutual information is zero. At the other extreme, if${\displaystyle X}$ is a deterministic function of${\displaystyle Y}$ and${\displaystyle Y}$ is a deterministic function of${\displaystyle X}$ then all information conveyed by${\displaystyle X}$ is shared with${\displaystyle Y}$: knowing${\displaystyle X}$ determines the value of${\displaystyle Y}$ and vice versa. As a result, the mutual information is the same as the uncertainty contained in${\displaystyle Y}$ (or${\displaystyle X}$) alone, namely theentropy of${\displaystyle Y}$ (or${\displaystyle X}$). A very special case of this is when${\displaystyle X}$ and${\displaystyle Y}$ are the same random variable.

Mutual information is a measure of the inherent dependence expressed in thejoint distribution of${\displaystyle X}$ and${\displaystyle Y}$ relative to the marginal distribution of${\displaystyle X}$ and${\displaystyle Y}$ under the assumption of independence. Mutual information therefore measures dependence in the following sense:${\displaystyle \mathrm{I}(X;Y)=0}$if and only if${\displaystyle X}$ and${\displaystyle Y}$ are independent random variables. This is easy to see in one direction: if${\displaystyle X}$ and${\displaystyle Y}$ are independent, then${\displaystyle p_{(X,Y)}(x,y)=p_X(x)\cdot p_Y(y)}$, and therefore:

${\displaystyle \log \left(\frac{p_{(X,Y)}(x,y)}{p_X(x)p_Y(y)}\right)=\log 1=0}$.

Moreover, mutual information is nonnegative (i.e.${\displaystyle \mathrm{I}(X;Y)\ge 0}$ see below) andsymmetric (i.e.${\displaystyle \mathrm{I}(X;Y)=\mathrm{I}(Y;X)}$ see below).

UsingJensen's inequality on the definition of mutual information we can show that${\displaystyle \mathrm{I}(X;Y)}$ is non-negative, i.e.^[3]: 28

${\displaystyle \mathrm{I}(X;Y)\ge 0}$

${\displaystyle \mathrm{I}(X;Y)=\mathrm{I}(Y;X)}$

The proof is given considering the relationship with entropy, as shown below.

If${\displaystyle C}$ is independent of${\displaystyle (A,B)}$, then

${\displaystyle \mathrm{I}(Y;A,B,C)-\mathrm{I}(Y;A,B)\ge \mathrm{I}(Y;A,C)-\mathrm{I}(Y;A)}$.^[4]

Mutual information can be equivalently expressed as:

where${\displaystyle \mathrm{H}(X)}$ and${\displaystyle \mathrm{H}(Y)}$ are the marginalentropies,${\displaystyle \mathrm{H}(X\mid Y)}$ and${\displaystyle \mathrm{H}(Y\mid X)}$ are theconditional entropies, and${\displaystyle \mathrm{H}(X,Y)}$ is thejoint entropy of${\displaystyle X}$ and${\displaystyle Y}$.

Notice the analogy to the union, difference, and intersection of two sets: in this respect, all the formulas given above are apparent from the Venn diagram reported at the beginning of the article.

In terms of a communication channel in which the output${\displaystyle Y}$ is a noisy version of the input${\displaystyle X}$, these relations are summarised in the figure:

Because${\displaystyle \mathrm{I}(X;Y)}$ is non-negative, consequently,${\displaystyle \mathrm{H}(X)\ge \mathrm{H}(X\mid Y)}$. Here we give the detailed deduction of${\displaystyle \mathrm{I}(X;Y)=\mathrm{H}(Y)-\mathrm{H}(Y\mid X)}$ for the case of jointly discrete random variables:

The proofs of the other identities above are similar. The proof of the general case (not just discrete) is similar, with integrals replacing sums.

Intuitively, if entropy${\displaystyle \mathrm{H}(Y)}$ is regarded as a measure of uncertainty about a random variable, then${\displaystyle \mathrm{H}(Y\mid X)}$ is a measure of what${\displaystyle X}$ doesnot say about${\displaystyle Y}$. This is "the amount of uncertainty remaining about${\displaystyle Y}$ after${\displaystyle X}$ is known", and thus the right side of the second of these equalities can be read as "the amount of uncertainty in${\displaystyle Y}$, minus the amount of uncertainty in${\displaystyle Y}$ which remains after${\displaystyle X}$ is known", which is equivalent to "the amount of uncertainty in${\displaystyle Y}$ which is removed by knowing${\displaystyle X}$". This corroborates the intuitive meaning of mutual information as the amount of information (that is, reduction in uncertainty) that knowing either variable provides about the other.

Note that in the discrete case${\displaystyle \mathrm{H}(Y\mid Y)=0}$ and therefore${\displaystyle \mathrm{H}(Y)=\mathrm{I}(Y;Y)}$. Thus${\displaystyle \mathrm{I}(Y;Y)\ge \mathrm{I}(X;Y)}$, and one can formulate the basic principle that a variable contains at least as much information about itself as any other variable can provide.

For jointly discrete or jointly continuous pairs${\displaystyle (X,Y)}$, mutual information is theKullback–Leibler divergence from the product of themarginal distributions,${\displaystyle p_X\cdot p_Y}$, of thejoint distribution${\displaystyle p_{(X,Y)}}$, that is,

${\displaystyle \mathrm{I}(X;Y)=D_{\text{KL}}\left(p_{(X,Y)}\parallel p_X{p}_Y\right)}$

Furthermore, let${\displaystyle p_{(X,Y)}(x,y)=p_{X\mid Y=y}(x)\ast p_Y(y)}$ be the conditional mass or density function. Then, we have the identity

${\displaystyle \mathrm{I}(X;Y)={\mathbb{E}}_Y\left[D_{\text{KL}}\left(p_{X\mid Y}\parallel p_X\right)\right]}$

The proof for jointly discrete random variables is as follows:

Similarly this identity can be established for jointly continuous random variables.

Note that here the Kullback–Leibler divergence involves integration over the values of the random variable${\displaystyle X}$ only, and the expression${\displaystyle D_{\text{KL}}(p_{X\mid Y}\parallel p_X)}$ still denotes a random variable because${\displaystyle Y}$ is random. Thus mutual information can also be understood as theexpectation over${\displaystyle Y}$ of the Kullback–Leibler divergence of theconditional distribution${\displaystyle p_{X\mid Y}}$ of${\displaystyle X}$ given${\displaystyle Y}$ from theunivariate distribution${\displaystyle p_X}$ of${\displaystyle X}$: the more different the distributions${\displaystyle p_{X\mid Y}}$ and${\displaystyle p_X}$ are on average, the greater theinformation gain.

If samples from a joint distribution are available, a Bayesian approach can be used to estimate the mutual information of that distribution. The first work to do this, which also showed how to do Bayesian estimation of many other information-theoretic properties besides mutual information, was.^[5] Subsequent researchers have rederived^[6] and extended^[7]this analysis. See^[8] for a recent paper based on a prior specifically tailored to estimation of mutual information per se. Besides, recently an estimation method accounting for continuous and multivariate outputs,${\displaystyle Y}$, was proposed in .^[9]

The Kullback-Leibler divergence formulation of the mutual information is predicated on that one is interested in comparing${\displaystyle p(x,y)}$ to the fully factorizedouter product${\displaystyle p(x)\cdot p(y)}$. In many problems, such asnon-negative matrix factorization, one is interested in less extreme factorizations; specifically, one wishes to compare${\displaystyle p(x,y)}$ to a low-rank matrix approximation in some unknown variable${\displaystyle w}$; that is, to what degree one might have

${\displaystyle p(x,y)\approx \sum _w{p}^{\mathrm{\prime }}(x,w)p^{\mathrm{\prime }\mathrm{\prime }}(w,y)}$

Alternately, one might be interested in knowing how much more information${\displaystyle p(x,y)}$ carries over its factorization. In such a case, the excess information that the full distribution${\displaystyle p(x,y)}$ carries over the matrix factorization is given by the Kullback-Leibler divergence

${\displaystyle {\mathrm{I}}_{LRMA}=\sum _{y\in \mathcal{Y}}\sum _{x\in \mathcal{X}}p(x,y)\log \left(\frac{p(x,y)}{\sum _w{p}^{\mathrm{\prime }}(x,w)p^{\mathrm{\prime }\mathrm{\prime }}(w,y)}\right),}$

The conventional definition of the mutual information is recovered in the extreme case that the process${\displaystyle W}$ has only one value for${\displaystyle w}$.

Several variations on mutual information have been proposed to suit various needs. Among these are normalized variants and generalizations to more than two variables.

Many applications require ametric, that is, a distance measure between pairs of points. The quantity

satisfies the properties of a metric (triangle inequality,non-negativity,indiscernability and symmetry), where equality${\displaystyle X=Y}$ is understood to mean that${\displaystyle X}$ can be completely determined from${\displaystyle Y}$.^[10]

This distance metric is also known as thevariation of information.

If${\displaystyle X,Y}$ are discrete random variables then all the entropy terms are non-negative, so${\displaystyle 0\le d(X,Y)\le \mathrm{H}(X,Y)}$ and one can define a normalized distance

${\displaystyle D(X,Y)=\frac{d(X,Y)}{\mathrm{H}(X,Y)}\le 1.}$

Plugging in the definitions shows that

${\displaystyle D(X,Y)=1-\frac{\mathrm{I}(X;Y)}{\mathrm{H}(X,Y)}.}$

This is known as the Rajski Distance.^[11] In a set-theoretic interpretation of information (see the figure forConditional entropy), this is effectively theJaccard distance between${\displaystyle X}$ and${\displaystyle Y}$.

Finally,

${\displaystyle D^{\mathrm{\prime }}(X,Y)=1-\frac{\mathrm{I}(X;Y)}{max\left\{\mathrm{H}(X),\mathrm{H}(Y)\right\}}}$

is also a metric.

Sometimes it is useful to express the mutual information of two random variables conditioned on a third.

${\displaystyle \mathrm{I}(X;Y|Z)={\mathbb{E}}_Z[D_{\mathrm{K}\mathrm{L}}(P_{(X,Y)|Z}\Vert P_{X|Z}\otimes P_{Y|Z})]}$

For jointlydiscrete random variables this takes the form

${\displaystyle \mathrm{I}(X;Y|Z)=\sum _{z\in \mathcal{Z}}\sum _{y\in \mathcal{Y}}\sum _{x\in \mathcal{X}}{p}_Z(z)p_{X,Y|Z}(x,y|z)\log \left[\frac{p_{X,Y|Z}(x,y|z)}{p_{X|Z}(x|z)p_{Y|Z}(y|z)}\right],}$

which can be simplified as

${\displaystyle \mathrm{I}(X;Y|Z)=\sum _{z\in \mathcal{Z}}\sum _{y\in \mathcal{Y}}\sum _{x\in \mathcal{X}}{p}_{X,Y,Z}(x,y,z)\log \frac{p_{X,Y,Z}(x,y,z)p_Z(z)}{p_{X,Z}(x,z)p_{Y,Z}(y,z)}.}$

For jointlycontinuous random variables this takes the form

${\displaystyle \mathrm{I}(X;Y|Z)={\int }_{\mathcal{Z}}{\int }_{\mathcal{Y}}{\int }_{\mathcal{X}}{p}_Z(z)p_{X,Y|Z}(x,y|z)\log \left[\frac{p_{X,Y|Z}(x,y|z)}{p_{X|Z}(x|z)p_{Y|Z}(y|z)}\right]dxdydz,}$

which can be simplified as

${\displaystyle \mathrm{I}(X;Y|Z)={\int }_{\mathcal{Z}}{\int }_{\mathcal{Y}}{\int }_{\mathcal{X}}{p}_{X,Y,Z}(x,y,z)\log \frac{p_{X,Y,Z}(x,y,z)p_Z(z)}{p_{X,Z}(x,z)p_{Y,Z}(y,z)}dxdydz.}$

Conditioning on a third random variable may either increase or decrease the mutual information, but it is always true that

${\displaystyle \mathrm{I}(X;Y|Z)\ge 0}$

for discrete, jointly distributed random variables${\displaystyle X,Y,Z}$. This result has been used as a basic building block for proving otherinequalities in information theory.

Several generalizations of mutual information to more than two random variables have been proposed, such astotal correlation (or multi-information) anddual total correlation. The expression and study of multivariate higher-degree mutual information was achieved in two seemingly independent works: McGill (1954)^[12] who called these functions "interaction information", and Hu Kuo Ting (1962).^[13] Interaction information is defined for one variable as follows:

${\displaystyle \mathrm{I}(X_1)=\mathrm{H}(X_1)}$

and for${\displaystyle n>1,}$

${\displaystyle \mathrm{I}(X_1;...;X_n)=\mathrm{I}(X_1;...;X_{n-1})-\mathrm{I}(X_1;...;X_{n-1}\mid X_n).}$

Some authors reverse the order of the terms on the right-hand side of the preceding equation, which changes the sign when the number of random variables is odd. (And in this case, the single-variable expression becomes the negative of the entropy.) Note that

${\displaystyle I(X_1;\dots ;X_{n-1}\mid X_n)={\mathbb{E}}_{X_n}[D_{\mathrm{K}\mathrm{L}}(P_{(X_1,\dots ,X_{n-1})\mid X_n}\Vert P_{X_1\mid X_n}\otimes \cdots \otimes P_{X_{n-1}\mid X_n})].}$

The multivariate mutual information functions generalize thepairwise independence case that states that${\displaystyle X_1,X_2}$ if and only if${\displaystyle I(X_1;X_2)=0}$, to arbitrary numerous variable. n variables are mutually independent if and only if the${\displaystyle 2^n-n-1}$ mutual information functions vanish${\displaystyle I(X_1;\dots ;X_k)=0}$ with${\displaystyle n\ge k\ge 2}$ (theorem 2^[14]). In this sense, the${\displaystyle I(X_1;\dots ;X_k)=0}$ can be used as a refined statistical independence criterion.

For 3 variables, Brenner et al. applied multivariate mutual information toneural coding and called its negativity "synergy"^[15] and Watkinson et al. applied it to genetic expression.^[16] For arbitrary k variables, Tapia et al. applied multivariate mutual information togene expression.^[17][14] It can be zero, positive, or negative.^[13] The positivity corresponds to relations generalizing the pairwise correlations, nullity corresponds to a refined notion of independence, and negativity detects high dimensional "emergent" relations and clustered datapoints^[17]).

One high-dimensional generalization scheme which maximizes the mutual information between the joint distribution and other target variables is found to be useful infeature selection.^[18]

Mutual information is also used in the area of signal processing as ameasure of similarity between two signals. For example, FMI metric^[19] is an image fusion performance measure that makes use of mutual information in order to measure the amount of information that the fused image contains about the source images. TheMatlab code for this metric can be found at.^[20] A python package for computing all multivariate mutual informations,conditional mutual information, joint entropies, total correlations, information distance in a dataset of n variables is available.^[21]

Directed information,${\displaystyle \mathrm{I}\left(X^n\to Y^n\right)}$, measures the amount of information that flows from the process${\displaystyle X^n}$ to${\displaystyle Y^n}$, where${\displaystyle X^n}$ denotes the vector${\displaystyle X_1,X_2,...,X_n}$ and${\displaystyle Y^n}$ denotes${\displaystyle Y_1,Y_2,...,Y_n}$. The termdirected information was coined byJames Massey and is defined as

${\displaystyle \mathrm{I}\left(X^n\to Y^n\right)=\sum _{i=1}^n\mathrm{I}\left(X^i;Y_i\mid Y^{i-1}\right)}$.

Note that if${\displaystyle n=1}$, the directed information becomes the mutual information. Directed information has many applications in problems wherecausality plays an important role, such ascapacity of channel with feedback.^[22][23]

Normalized variants of the mutual information are provided by thecoefficients of constraint,^[24]uncertainty coefficient^[25] or proficiency:^[26]

${\displaystyle C_{XY}=\frac{\mathrm{I}(X;Y)}{\mathrm{H}(Y)}\text{and}{C}_{YX}=\frac{\mathrm{I}(X;Y)}{\mathrm{H}(X)}.}$

The two coefficients have a value ranging in [0, 1], but are not necessarily equal. This measure is not symmetric. If one desires a symmetric measure, one may consider the followingredundancy measure:

${\displaystyle R=\frac{\mathrm{I}(X;Y)}{\mathrm{H}(X)+\mathrm{H}(Y)}}$

which attains a minimum of zero when the variables are independent and a maximum value of

${\displaystyle R_{max}=\frac{min\left\{\mathrm{H}(X),\mathrm{H}(Y)\right\}}{\mathrm{H}(X)+\mathrm{H}(Y)}}$

when one variable becomes completely redundant with the knowledge of the other. See alsoRedundancy (information theory).

Another symmetrical measure is thesymmetric uncertainty (Witten & Frank 2005), given by

${\displaystyle U(X,Y)=2R=2\frac{\mathrm{I}(X;Y)}{\mathrm{H}(X)+\mathrm{H}(Y)}}$

which represents theharmonic mean of the two uncertainty coefficients${\displaystyle C_{XY},C_{YX}}$.^[25]

If we consider mutual information as a special case of thetotal correlation ordual total correlation, the normalized versions are respectively,

${\displaystyle \frac{\mathrm{I}(X;Y)}{min\left[\mathrm{H}(X),\mathrm{H}(Y)\right]}}$ and${\displaystyle \frac{\mathrm{I}(X;Y)}{\mathrm{H}(X,Y)}.}$

This normalized version is also known asInformation Quality Ratio (IQR) and quantifies the amount of information of a variable based on another variable against total uncertainty:^[27]

${\displaystyle IQR(X,Y)=\mathrm{E}[\mathrm{I}(X;Y)]=\frac{\mathrm{I}(X;Y)}{\mathrm{H}(X,Y)}=\frac{\sum _{x\in X}\sum _{y\in Y}p(x,y)\log p(x)p(y)}{\sum _{x\in X}\sum _{y\in Y}p(x,y)\log p(x,y)}-1}$

There exists a normalization^[28] which derives from first thinking of mutual information as an analogue tocovariance (thusShannon entropy is analogous tovariance). Then the normalized mutual information is calculated akin to thePearson correlation coefficient,

${\displaystyle \frac{\mathrm{I}(X;Y)}{\sqrt{\mathrm{H}(X)\mathrm{H}(Y)}}.}$

A naive normalization may lead to biased interpretation and introduce spurious dependences.^[29]

In the traditional formulation of the mutual information,

${\displaystyle \mathrm{I}(X;Y)=\sum _{y\in Y}\sum _{x\in X}p(x,y)\log \frac{p(x,y)}{p(x)p(y)},}$

eachevent orobject specified by${\displaystyle (x,y)}$ is weighted by the corresponding probability${\displaystyle p(x,y)}$. This assumes that all objects or events are equivalentapart from their probability of occurrence. However, in some applications it may be the case that certain objects or events are moresignificant than others, or that certain patterns of association are more semantically important than others.

For example, the deterministic mapping${\displaystyle \{(1,1),(2,2),(3,3)\}}$ may be viewed as stronger than the deterministic mapping${\displaystyle \{(1,3),(2,1),(3,2)\}}$, although these relationships would yield the same mutual information. This is because the mutual information is not sensitive at all to any inherent ordering in the variable values (Cronbach 1954,Coombs, Dawes & Tversky 1970,Lockhead 1970), and is therefore not sensitive at all to theform of the relational mapping between the associated variables. If it is desired that the former relation—showing agreement on all variable values—be judged stronger than the later relation, then it is possible to use the followingweighted mutual information (Guiasu 1977).

${\displaystyle \mathrm{I}(X;Y)=\sum _{y\in Y}\sum _{x\in X}w(x,y)p(x,y)\log \frac{p(x,y)}{p(x)p(y)},}$

which places a weight${\displaystyle w(x,y)}$ on the probability of each variable value co-occurrence,${\displaystyle p(x,y)}$. This allows that certain probabilities may carry more or less significance than others, thereby allowing the quantification of relevantholistic orPrägnanz factors. In the above example, using larger relative weights for${\displaystyle w(1,1)}$,${\displaystyle w(2,2)}$, and${\displaystyle w(3,3)}$ would have the effect of assessing greaterinformativeness for the relation${\displaystyle \{(1,1),(2,2),(3,3)\}}$ than for the relation${\displaystyle \{(1,3),(2,1),(3,2)\}}$, which may be desirable in some cases of pattern recognition, and the like. This weighted mutual information is a form of weighted KL-Divergence, which is known to take negative values for some inputs,^[30] and there are examples where the weighted mutual information also takes negative values.^[31]

A probability distribution can be viewed as apartition of a set. One may then ask: if a set were partitioned randomly, what would the distribution of probabilities be? What would the expectation value of the mutual information be? Theadjusted mutual information or AMI subtracts the expectation value of the MI, so that the AMI is zero when two different distributions are random, and one when two distributions are identical. The AMI is defined in analogy to theadjusted Rand index of two different partitions of a set.

Using the ideas ofKolmogorov complexity, one can consider the mutual information of two sequences independent of any probability distribution:

${\displaystyle {\mathrm{I}}_K(X;Y)=K(X)-K(X\mid Y).}$

To establish that this quantity is symmetric up to a logarithmic factor (${\displaystyle {\mathrm{I}}_K(X;Y)\approx {\mathrm{I}}_K(Y;X)}$) one requires thechain rule for Kolmogorov complexity (Li & Vitányi 1997). Approximations of this quantity viacompression can be used to define adistance measure to perform ahierarchical clustering of sequences without having anydomain knowledge of the sequences (Cilibrasi & Vitányi 2005).

Unlike correlation coefficients, such as theproduct moment correlation coefficient, mutual information contains information about all dependence—linear and nonlinear—and not just linear dependence as the correlation coefficient measures. However, in the narrow case that the joint distribution for${\displaystyle X}$ and${\displaystyle Y}$ is abivariate normal distribution (implying in particular that both marginal distributions are normally distributed), there is an exact relationship between${\displaystyle \mathrm{I}}$ and the correlation coefficient${\displaystyle \rho }$ (Gel'fand & Yaglom 1957).

${\displaystyle \mathrm{I}=-\frac{1}{2}\log \left(1-{\rho }^2\right)}$

The equation above can be derived as follows for a bivariate Gaussian:

Therefore,

${\displaystyle \mathrm{I}\left(X_1;X_2\right)=\mathrm{H}\left(X_1\right)+\mathrm{H}\left(X_2\right)-\mathrm{H}\left(X_1,X_2\right)=-\frac{1}{2}\log \left(1-{\rho }^2\right)}$

When${\displaystyle X}$ and${\displaystyle Y}$ are limited to be in a discrete number of states, observation data is summarized in acontingency table, with row variable${\displaystyle X}$ (or${\displaystyle i}$) and column variable${\displaystyle Y}$ (or${\displaystyle j}$). Mutual information is one of the measures ofassociation orcorrelation between the row and column variables.

Other measures of association includePearson's chi-squared test statistics,G-test statistics, etc. In fact, with the same log base, mutual information will be equal to theG-test log-likelihood statistic divided by${\displaystyle 2N}$, where${\displaystyle N}$ is the sample size.

In many applications, one wants to maximize mutual information (thus increasing dependencies), which is often equivalent to minimizingconditional entropy. Examples include:

• Insearch engine technology, mutual information between phrases and contexts is used as a feature fork-means clustering to discover semantic clusters (concepts).^[32] For example, the mutual information of a bigram might be calculated as:

${\displaystyle MI(x,y)=\log \frac{P_{X,Y}(x,y)}{P_X(x)P_Y(y)}\approx \log \frac{\frac{f_{XY}}{B}}{\frac{f_X}{U}\frac{f_Y}{U}}}$

where${\displaystyle f_{XY}}$ is the number of times the bigram xy appears in the corpus,${\displaystyle f_X}$ is the number of times the unigram x appears in the corpus, B is the total number of bigrams, and U is the total number of unigrams.^[32]

• Intelecommunications, thechannel capacity is equal to the mutual information, maximized over all input distributions.
• Discriminative training procedures forhidden Markov models have been proposed based on themaximum mutual information (MMI) criterion.
• RNA secondary structure prediction from amultiple sequence alignment.
• Phylogenetic profiling prediction from pairwise present and disappearance of functionally linkgenes.
• Mutual information has been used as a criterion forfeature selection and feature transformations inmachine learning. It can be used to characterize both the relevance and redundancy of variables, such as theminimum redundancy feature selection.
• Mutual information is used in determining the similarity of two differentclusterings of a dataset. As such, it provides some advantages over the traditionalRand index.
• Mutual information of words is often used as a significance function for the computation ofcollocations incorpus linguistics. This has the added complexity that no word-instance is an instance to two different words; rather, one counts instances where 2 words occur adjacent or in close proximity; this slightly complicates the calculation, since the expected probability of one word occurring within${\displaystyle N}$ words of another, goes up with${\displaystyle N}$
• Mutual information is used inmedical imaging forimage registration. Given a reference image (for example, a brain scan), and a second image which needs to be put into the samecoordinate system as the reference image, this image is deformed until the mutual information between it and the reference image is maximized.
• Detection ofphase synchronization intime series analysis.
• In theinfomax method for neural-net and other machine learning, including the infomax-basedIndependent component analysis algorithm
• Average mutual information indelay embedding theorem is used for determining theembedding delay parameter.
• Mutual information betweengenes inexpression microarray data is used by the ARACNE algorithm for reconstruction ofgene networks.
• Instatistical mechanics,Loschmidt's paradox may be expressed in terms of mutual information.^[33][34] Loschmidt noted that it must be impossible to determine a physical law which lackstime reversal symmetry (e.g. thesecond law of thermodynamics) only from physical laws which have this symmetry. He pointed out that theH-theorem ofBoltzmann made the assumption that the velocities of particles in a gas were permanently uncorrelated, which removed the time symmetry inherent in the H-theorem. It can be shown that if a system is described by a probability density inphase space, thenLiouville's theorem implies that the joint information (negative of the joint entropy) of the distribution remains constant in time. The joint information is equal to the mutual information plus the sum of all the marginal information (negative of the marginal entropies) for each particle coordinate. Boltzmann's assumption amounts to ignoring the mutual information in the calculation of entropy, which yields the thermodynamic entropy (divided by the Boltzmann constant).
• Instochastic processes coupled to changing environments, mutual information can be used to disentangle internal and effective environmental dependencies.^[35][36] This is particularly useful when a physical system undergoes changes in the parameters describing its dynamics, e.g., changes in temperature.
• The mutual information is used to learn the structure ofBayesian networks/dynamic Bayesian networks, which is thought to explain the causal relationship between random variables, as exemplified by the GlobalMIT toolkit:^[37] learning the globally optimal dynamic Bayesian network with the Mutual Information Test criterion.
• The mutual information is used to quantify information transmitted during the updating procedure in theGibbs sampling algorithm.^[38]
• Popular cost function indecision tree learning.
• The mutual information is used incosmology to test the influence of large-scale environments on galaxy properties in theGalaxy Zoo.
• The mutual information was used inSolar Physics to derive the solardifferential rotation profile, a travel-time deviation map for sunspots, and a time–distance diagram from quiet-Sun measurements^[39]
• Used in Invariant Information Clustering to automatically train neural network classifiers and image segmenters given no labelled data.^[40]
• Instochastic dynamical systems with multiple timescales, mutual information has been shown to capture the functional couplings between different temporal scales.^[41] Importantly, it was shown that physical interactions may or may not give rise to mutual information, depending on the typical timescale of their dynamics.

• Data differencing
• Pointwise mutual information
• Quantum mutual information
• Specific-information

• Baudot, P.; Tapia, M.; Bennequin, D.; Goaillard, J.M. (2019)."Topological Information Data Analysis".Entropy.21 (9). 869.arXiv:1907.04242.Bibcode:2019Entrp..21..869B.doi:10.3390/e21090869.PMC 7515398.S2CID 195848308.
• Cilibrasi, R.; Vitányi, Paul (2005)."Clustering by compression"(PDF).IEEE Transactions on Information Theory.51 (4):1523–1545.arXiv:cs/0312044.doi:10.1109/TIT.2005.844059.S2CID 911.
• Cronbach, L. J. (1954). "On the non-rational application of information measures in psychology". InQuastler, Henry (ed.).Information Theory in Psychology: Problems and Methods. Glencoe, Illinois: Free Press. pp. 14–30.
• Coombs, C. H.; Dawes, R. M.; Tversky, A. (1970).Mathematical Psychology: An Elementary Introduction. Englewood Cliffs, New Jersey: Prentice-Hall.
• Church, Kenneth Ward; Hanks, Patrick (1989)."Word association norms, mutual information, and lexicography".Proceedings of the 27th Annual Meeting of the Association for Computational Linguistics.16 (1):76–83.doi:10.3115/981623.981633.
• Gel'fand, I.M.; Yaglom, A.M. (1957). "Calculation of amount of information about a random function contained in another such function".American Mathematical Society Translations. Series 2.12:199–246.doi:10.1090/trans2/012/09.ISBN 978-0-8218-1712-4.{{cite journal}}:ISBN / Date incompatibility (help) English translation of original inUspekhi Matematicheskikh Nauk12 (1): 3-52.
• Guiasu, Silviu (1977).Information Theory with Applications. McGraw-Hill, New York.ISBN 978-0-07-025109-0.
• Li, Ming; Vitányi, Paul (February 1997).An introduction to Kolmogorov complexity and its applications. New York:Springer-Verlag.ISBN 978-0-387-94868-3.
• Lockhead, G. R. (1970). "Identification and the form of multidimensional discrimination space".Journal of Experimental Psychology.85 (1):1–10.doi:10.1037/h0029508.PMID 5458322.
• David J. C. MacKay.Information Theory, Inference, and Learning Algorithms Cambridge: Cambridge University Press, 2003.ISBN 0-521-64298-1 (available free online)
• Haghighat, M. B. A.; Aghagolzadeh, A.; Seyedarabi, H. (2011). "A non-reference image fusion metric based on mutual information of image features".Computers & Electrical Engineering.37 (5):744–756.doi:10.1016/j.compeleceng.2011.07.012.S2CID 7738541.
• Athanasios Papoulis.Probability, Random Variables, and Stochastic Processes, second edition. New York: McGraw-Hill, 1984.(See Chapter 15.)
• Witten, Ian H. & Frank, Eibe (2005).Data Mining: Practical Machine Learning Tools and Techniques. Morgan Kaufmann, Amsterdam.ISBN 978-0-12-374856-0.
• Peng, H.C.; Long, F. & Ding, C. (2005)."Feature selection based on mutual information: criteria of max-dependency, max-relevance, and min-redundancy".IEEE Transactions on Pattern Analysis and Machine Intelligence.27 (8):1226–1238.CiteSeerX 10.1.1.63.5765.doi:10.1109/tpami.2005.159.PMID 16119262.S2CID 206764015.
• Andre S. Ribeiro; Stuart A. Kauffman; Jason Lloyd-Price; Bjorn Samuelsson & Joshua Socolar (2008). "Mutual Information in Random Boolean models of regulatory networks".Physical Review E.77 (1) 011901.arXiv:0707.3642.Bibcode:2008PhRvE..77a1901R.doi:10.1103/physreve.77.011901.PMID 18351870.S2CID 15232112.
• Wells, W.M. III; Viola, P.; Atsumi, H.; Nakajima, S.; Kikinis, R. (1996)."Multi-modal volume registration by maximization of mutual information"(PDF).Medical Image Analysis.1 (1):35–51.doi:10.1016/S1361-8415(01)80004-9.PMID 9873920. Archived fromthe original(PDF) on 2008-09-06. Retrieved2010-08-05.
• Pandey, Biswajit; Sarkar, Suman (2017)."How much a galaxy knows about its large-scale environment?: An information theoretic perspective".Monthly Notices of the Royal Astronomical Society Letters.467 (1): L6.arXiv:1611.00283.Bibcode:2017MNRAS.467L...6P.doi:10.1093/mnrasl/slw250.S2CID 119095496.

• Janssen, Joseph; Guan, Vincent; Robeva, Elina (2023)."Ultra-marginal Feature Importance: Learning from Data with Causal Guarantees".International Conference on Artificial Intelligence and Statistics:10782–10814.arXiv:2204.09938.

• Information theory
• Entropy and information

• CS1: unfit URL
• Articles with short description
• Short description matches Wikidata
• CS1 errors: ISBN date