This article has multiple issues. Please helpimprove it or discuss these issues on thetalk page.(Learn how and when to remove these messages) This articleneeds additional citations forverification. Please helpimprove this article byadding citations to reliable sources. Unsourced material may be challenged and removed. Find sources: "Maximum entropy probability distribution" – news ·newspapers ·books ·scholar ·JSTOR(August 2009) (Learn how and when to remove this message) This article includes a list ofgeneral references, butit lacks sufficient correspondinginline citations. Please help toimprove this article byintroducing more precise citations.(December 2013) (Learn how and when to remove this message) (Learn how and when to remove this message)

Instatistics andinformation theory, amaximum entropy probability distribution hasentropy that is at least as great as that of all other members of a specified class ofprobability distributions. According to theprinciple of maximum entropy, if nothing is known about a distribution except that it belongs to a certain class (usually defined in terms of specified properties or measures), then the distribution with the largest entropy should be chosen as the least-informative default. The motivation is twofold: first, maximizing entropy minimizes the amount ofprior information built into the distribution; second, many physical systems tend to move towards maximal entropy configurations over time.

If${\displaystyle X}$ is acontinuous random variable withprobability density${\displaystyle p(x)}$, then thedifferential entropy of${\displaystyle X}$ is defined as^[1][2][3]

$${\displaystyle H(X)=-{\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}p(x)\log p(x)dx.}$$

If${\displaystyle X}$ is adiscrete random variable with distribution given by$${\displaystyle Pr(X=x_k)=p_k\text{ for }k=1,2,\dots }$$then the entropy of${\displaystyle X}$ is defined as$${\displaystyle H(X)=-\sum _{k\ge 1}{p}_k\log p_k.}$$

The seemingly divergent term${\displaystyle p(x)\log p(x)}$ is replaced by zero, whenever${\displaystyle p(x)=0.}$

This is a special case of more general forms described in the articlesEntropy (information theory),Principle of maximum entropy, and differential entropy. In connection with maximum entropy distributions, this is the only one needed, because maximizing${\displaystyle H(X)}$ will also maximize the more general forms.

The base of thelogarithm is not important, as long as the same one is used consistently: Change of base merely results in a rescaling of the entropy. Information theorists may prefer to use base 2 in order to express the entropy inbits; mathematicians and physicists often prefer thenatural logarithm, resulting in a unit of"nat"s for the entropy.

However, the chosenmeasure${\displaystyle dx}$ is crucial, even though the typical use of theLebesgue measure is often defended as a "natural" choice: Which measure is chosen determines the entropy and the consequent maximum entropy distribution.

Many statistical distributions of applicable interest are those for which themoments or other measurable quantities are constrained to be constants. The following theorem byLudwig Boltzmann gives the form of the probability density under these constraints.

Suppose${\displaystyle S}$ is a continuous,closed subset of thereal numbers${\displaystyle \mathbb{R}}$ and we choose to specify${\displaystyle n}$measurable functions${\displaystyle f_1,\dots ,f_n}$ and${\displaystyle n}$ numbers${\displaystyle a_1,\dots ,a_n.}$ We consider the class${\displaystyle C}$ of all real-valued random variables which are supported on${\displaystyle S}$ (i.e. whose density function is zero outside of${\displaystyle S}$) and which satisfy the${\displaystyle n}$ moment conditions:

$${\displaystyle \mathrm{E}[f_j(X)]\ge a_j\text{for }j=1,\dots ,n}$$

If there is a member in${\displaystyle C}$ whosedensity function is positive everywhere in${\displaystyle S,}$ and if there exists a maximal entropy distribution for${\displaystyle C,}$ then its probability density${\displaystyle p(x)}$ has the following form:

$${\displaystyle p(x)=\exp \left(\sum _{j=0}^n{\lambda }_j{f}_j(x)\right)\text{ for all }x\in S}$$

where we assume that${\displaystyle f_0(x)=1.}$ The constant${\displaystyle {\lambda }_0}$ and the${\displaystyle n}$Lagrange multipliers${\displaystyle \mathit{\lambda }=({\lambda }_1,\dots ,{\lambda }_n)}$ solve the constrained optimization problem with${\displaystyle a_0=1}$ (which ensures that${\displaystyle p}$ integrates to unity):^[4]

$${\displaystyle \underset{{\lambda }_0;\mathit{\lambda }}{max}\left\{\sum _{j=0}^n{\lambda }_j{a}_j-\int \exp \left(\sum _{j=0}^n{\lambda }_j{f}_j(x)\right)dx\right\}\text{ subject to }\mathit{\lambda }\ge \mathbf{0}}$$

Using theKarush–Kuhn–Tucker conditions, it can be shown that the optimization problem has a unique solution because the objective function in the optimization is concave in${\displaystyle \mathit{\lambda }.}$

Note that when the moment constraints are equalities (instead of inequalities), that is,

$${\displaystyle \mathrm{E}[f_j(X)]=a_j\text{ for }j=1,\dots ,n,}$$

then the constraint condition${\displaystyle \mathit{\lambda }\ge \mathbf{0}}$ can be dropped, which makes optimization over the Lagrange multipliers unconstrained.

Suppose${\displaystyle S=\{x_1,x_2,\dots \}}$ is a (finite or infinite) discrete subset of the reals, and that we choose to specify${\displaystyle n}$ functions${\displaystyle f_1,\dots ,f_n}$ and${\displaystyle n}$ numbers${\displaystyle a_1,\dots ,a_n.}$ We consider the class${\displaystyle C}$ of all discrete random variables${\displaystyle X}$ which are supported on${\displaystyle S}$ and which satisfy the${\displaystyle n}$ moment conditions

$${\displaystyle \mathrm{E}[f_j(X)]\ge a_j\text{ for }j=1,\dots ,n}$$

If there exists a member of class${\displaystyle C}$ which assigns positive probability to all members of${\displaystyle S}$ and if there exists a maximum entropy distribution for${\displaystyle C,}$ then this distribution has the following shape:

$${\displaystyle Pr(X=x_k)=\exp \left(\sum _{j=0}^n{\lambda }_j{f}_j(x_k)\right)\text{ for }k=1,2,\dots }$$

where we assume that${\displaystyle f_0=1}$ and the constants${\displaystyle {\lambda }_0,\mathit{\lambda }\equiv ({\lambda }_1,\dots ,{\lambda }_n)}$ solve the constrained optimization problem with${\displaystyle a_0=1}$:^[5]

$${\displaystyle \underset{{\lambda }_0;\mathit{\lambda }}{max}\left\{\sum _{j=0}^n{\lambda }_j{a}_j-\sum _{k\ge 1}\exp \left(\sum _{j=0}^n{\lambda }_j{f}_j(x_k)\right)\right\}\text{ for which }\mathit{\lambda }\ge \mathbf{0}}$$

Again as above, if the moment conditions are equalities (instead of inequalities), then the constraint condition${\displaystyle \mathit{\lambda }\ge \mathbf{0}}$ is not present in the optimization.

In the case of equality constraints, this theorem is proved with thecalculus of variations andLagrange multipliers. The constraints can be written as

$${\displaystyle {\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}{f}_j(x)p(x)dx=a_j}$$

We consider thefunctional

$${\displaystyle J(p)={\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}p(x)\ln p(x)dx-{\eta }_0\left({\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}p(x)dx-1\right)-\sum _{j=1}^n{\lambda }_j\left({\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}{f}_j(x)p(x)dx-a_j\right)}$$

where${\displaystyle {\eta }_0}$ and${\displaystyle {\lambda }_j,j\ge 1}$ are the Lagrange multipliers. The zeroth constraint ensures thesecond axiom of probability. The other constraints are that the measurements of the function are given constants up to order${\displaystyle n}$. The entropy attains an extremum when thefunctional derivative is equal to zero:

$${\displaystyle \frac{\delta J(p)}{\delta p}=\ln p(x)+1-{\eta }_0-\sum _{j=1}^n{\lambda }_j{f}_j(x)=0}$$

Therefore, the extremal entropy probability distribution in this case must be of the form (${\displaystyle {\lambda }_0:={\eta }_0-1}$),

$${\displaystyle p(x)=e^{-1+{\eta }_0}{e}^{\sum _{j=1}^n{\lambda }_j{f}_j(x)}=\exp \left(\sum _{j=0}^n{\lambda }_j{f}_j(x)\right),}$$

remembering that${\displaystyle f_0(x)=1}$. It can be verified that this is the maximal solution by checking that the variation around this solution is always negative.

Suppose${\displaystyle p}$ and${\displaystyle p^{\prime }}$ are distributions satisfying the expectation-constraints. Letting${\displaystyle \alpha \in (0,1)}$ and considering the distribution${\displaystyle q=\alpha p+(1-\alpha )p^{\prime }}$ it is clear that this distribution satisfies the expectation-constraints and furthermore has as support${\displaystyle \mathrm{supp}(q)=\mathrm{supp}(p)\cup \mathrm{supp}(p^{\prime }).}$ From basic facts about entropy, it holds that${\displaystyle \mathcal{H}(q)\ge \alpha \mathcal{H}(p)+(1-\alpha )\mathcal{H}(p^{\prime }).}$ Taking limits${\displaystyle \alpha \to 1}$ and${\displaystyle \alpha \to 0,}$ respectively, yields${\displaystyle \mathcal{H}(q)\ge \mathcal{H}(p),\mathcal{H}(p^{\prime }).}$

It follows that a distribution satisfying the expectation-constraints and maximising entropy must necessarily have full support —i. e. the distribution is almost everywhere strictly positive. It follows that the maximising distribution must be an internal point in the space of distributions satisfying the expectation-constraints, that is, it must be a local extreme. Thus it suffices to show that the local extreme is unique, in order to show both that the entropy-maximising distribution is unique (and this also shows that the local extreme is the global maximum).

Suppose${\displaystyle p}$ and${\displaystyle p^{\prime }}$ are local extremes. Reformulating the above computations these are characterised by parameters${\displaystyle \mathit{\lambda },{\mathit{\lambda }}^{\prime }\in {\mathbb{R}}^n}$ via${\displaystyle p(x)=\exp ⟨\mathit{\lambda },\mathbf{f}(x)⟩/C(\mathit{\lambda })}$ and similarly for${\displaystyle p^{\prime },}$ where$C(\mathit{\lambda })={\int }_{\mathbb{R}}\exp ⟨\mathit{\lambda },\mathbf{f}(x)⟩dx.$ We now note a series of identities: Via the satisfaction of the expectation-constraints and utilising gradients / directional derivatives, one has

${\displaystyle {D\log C(\cdot )|}_{\mathit{\lambda }}={\frac{DC(\cdot )}{C(\cdot )}|}_{\mathit{\lambda }}={\mathrm{E}}_p\left[\mathbf{f}(X)\right]=\mathbf{a}}$ and similarly for${\displaystyle {\mathit{\lambda }}^{\prime }.}$ Letting${\displaystyle u={\mathit{\lambda }}^{\prime }-\mathit{\lambda }\in {\mathbb{R}}^n}$ one obtains:

$${\displaystyle 0=⟨u,\mathbf{a}-\mathbf{a}⟩={D_u\log C(\cdot )|}_{{\mathit{\lambda }}^{\prime }}-{D_u\log C(\cdot )|}_{\mathit{\lambda }}={D_u^2\log C(\cdot )|}_{\mathit{\gamma }}}$$

where${\displaystyle \mathit{\gamma }=\theta \mathit{\lambda }+(1-\theta ){\mathit{\lambda }}^{\prime }}$ for some${\displaystyle \theta \in (0,1).}$ Computing further, one has

where${\displaystyle q}$ is similar to the distribution above, only parameterised by${\displaystyle \mathit{\gamma },}$Assuming that no non-trivial linear combination of the observables isalmost everywhere (a.e.) constant, (whiche.g. holds if the observables are independent and not a.e. constant), it holds that${\displaystyle ⟨u,\mathbf{f}(X)⟩}$ has non-zero variance, unless${\displaystyle u=0.}$ By the above equation it is thus clear, that the latter must be the case. Hence${\displaystyle {\mathit{\lambda }}^{\prime }-\mathit{\lambda }=u=0,}$ so the parameters characterising the local extrema${\displaystyle p,p^{\prime }}$ are identical, which means that the distributions themselves are identical. Thus, the local extreme is unique and by the above discussion, the maximum is unique – provided a local extreme actually exists.

Note that not all classes of distributions contain a maximum entropy distribution. It is possible that a class contains distributions of arbitrarily large entropy (e.g. the class of all continuous distributions onR with mean 0 but arbitrary standard deviation), or that the entropies are bounded above but there is no distribution which attains the maximal entropy.^[a] It is also possible that the expected value restrictions for the classC force the probability distribution to be zero in certain subsets ofS. In that case our theorem doesn't apply, but one can work around this by shrinking the setS.

Every probability distribution is trivially a maximum entropy probability distribution under the constraint that the distribution has its own entropy. To see this, rewrite the density as${\displaystyle p(x)=\exp (\ln p(x))}$ and compare to the expression of the theorem above. By choosing${\displaystyle \ln p(x)\to f(x)}$ to be the measurable function and

$${\displaystyle \int \exp (f(x))f(x)dx=-H}$$

to be the constant,${\displaystyle p(x)}$ is the maximum entropy probability distribution under the constraint

$${\displaystyle \int p(x)f(x)dx=-H.}$$

Nontrivial examples are distributions that are subject to multiple constraints that are different from the assignment of the entropy. These are often found by starting with the same procedure${\displaystyle \ln p(x)\to f(x)}$ and finding that${\displaystyle f(x)}$ can be separated into parts.

A table of examples of maximum entropy distributions is given in Lisman (1972)^[6] and Park & Bera (2009).^[7]

Theuniform distribution on the interval [a,b] is the maximum entropy distribution among all continuous distributions which are supported in the interval [a,b], and thus the probability density is 0 outside of the interval. This uniform density can be related to Laplace'sprinciple of indifference, sometimes called the principle of insufficient reason. More generally, if we are given a subdivisiona=a₀ <a₁ < ... <a_k =b of the interval [a,b] and probabilitiesp₁,...,p_k that add up to one, then we can consider the class of all continuous distributions such thatThe density of the maximum entropy distribution for this class is constant on each of the intervals [a_j−1,a_j). The uniform distribution on the finite set {x₁,...,x_n} (which assigns a probability of 1/n to each of these values) is the maximum entropy distribution among all discrete distributions supported on this set.

Theexponential distribution, for which the density function is

is the maximum entropy distribution among all continuous distributions supported in [0,∞) that have a specified mean of 1/λ.

In the case of distributions supported on [0,∞), the maximum entropy distribution depends on relationships between the first and second moments. In specific cases, it may be the exponential distribution, or may be another distribution, or may be undefinable.^[8]

Thenormal distributionN(μ,σ²), for which the density function is

$${\displaystyle p(x|\mu ,\sigma )=\frac{1}{\sigma \sqrt{2\pi }}{e}^{-\frac{(x-\mu {)}^2}{2{\sigma }^2}},}$$

has maximum entropy among allreal-valued distributions supported on (−∞,∞) with a specifiedvarianceσ² (a particularmoment). The same is true when themeanμ and thevarianceσ² is specified (the first two moments), since entropy is translation invariant on (−∞,∞). Therefore, the assumption of normality imposes the minimal prior structural constraint beyond these moments. (See thedifferential entropy article for a derivation.)

Among all the discrete distributions supported on the set {x₁,...,x_n} with a specified mean μ, the maximum entropy distribution has the following shape:$${\displaystyle Pr(X=x_k)=C{r}^{x_k}\text{ for }k=1,\dots ,n}$$where the positive constantsC andr can be determined by the requirements that the sum of all the probabilities must be 1 and the expected value must be μ.

For example, if a large numberN of dice are thrown, and you are told that the sum of all the shown numbers isS. Based on this information alone, what would be a reasonable assumption for the number of dice showing 1, 2, ..., 6? This is an instance of the situation considered above, with {x₁,...,x₆} = {1,...,6} andμ =S/N.

Finally, among all the discrete distributions supported on the infinite set${\displaystyle \{x_1,x_2,...\}}$ with meanμ, the maximum entropy distribution has the shape:$${\displaystyle Pr(X=x_k)=C{r}^{x_k}\text{ for }k=1,2,\dots ,}$$where again the constantsC andr were determined by the requirements that the sum of all the probabilities must be 1 and the expected value must be μ. For example, in the case thatx_k = k, this gives$${\displaystyle C=\frac{1}{\mu -1},r=\frac{\mu -1}{\mu },}$$

such that respective maximum entropy distribution is thegeometric distribution.

For a continuous random variable${\displaystyle {\theta }_i}$ distributed about the unit circle, theVon Mises distribution maximizes the entropy when the real and imaginary parts of the firstcircular moment are specified^[9] or, equivalently, thecircular mean andcircular variance are specified.

When the mean and variance of the angles${\displaystyle {\theta }_i}$ modulo${\displaystyle 2\pi }$ are specified, thewrapped normal distribution maximizes the entropy.^[9]

There exists an upper bound on the entropy of continuous random variables on${\displaystyle \mathbb{R}}$ with a specified mean, variance, and skew. However, there isno distribution which achieves this upper bound, because${\displaystyle p(x)=c\exp ({\lambda }_{1}x+{\lambda }_2{x}^2+{\lambda }_3{x}^3)}$ is unbounded when${\displaystyle {\lambda }_3\ne 0}$ (see Cover & Thomas (2006: chapter 12)).

However, the maximum entropy isε-achievable: a distribution's entropy can be arbitrarily close to the upper bound. Start with a normal distribution of the specified mean and variance. To introduce a positive skew, perturb the normal distribution upward by a small amount at a value manyσ larger than the mean. The skewness, being proportional to the third moment, will be affected more than the lower order moments.

This is a special case of the general case in which the exponential of any odd-order polynomial inx will be unbounded on${\displaystyle \mathbb{R}}$. For example,${\displaystyle c{e}^{\lambda x}}$ will likewise be unbounded on${\displaystyle \mathbb{R}}$, but when the support is limited to a bounded or semi-bounded interval the upper entropy bound may be achieved (e.g. ifx lies in the interval [0,∞] andλ< 0, theexponential distribution will result).

Every distribution withlog-concave density is a maximal entropy distribution with specified meanμ anddeviation risk measureD .^[10]

In particular, the maximal entropy distribution with specified mean${\displaystyle E(X)\equiv \mu }$ and deviation${\displaystyle D(X)\equiv d}$ is:

• Thenormal distribution${\displaystyle \mathcal{N}(m,d^2),}$ if$D(X)=\sqrt{\mathrm{E}\left[{\left(X-\mu \right)}^2\right]}$ is thestandard deviation;
• TheLaplace distribution, if${\displaystyle D(X)=\mathrm{E}\left\{\left|X-\mu \right|\right\}}$ is theaverage absolute deviation;^[6]
• The distribution with density of the form${\displaystyle f(x)=c\exp \left(ax+b{\downharpoonright x-\mu \downharpoonleft }_{-}^2\right)}$ if$D(X)=\sqrt{\mathrm{E}\left\{{\downharpoonright X-\mu \downharpoonleft }_{-}^2\right\}}$ is the standard lower semi-deviation, where${\displaystyle a,b,c}$ are constants and the function${\displaystyle \downharpoonright y{\downharpoonleft }_{-}\equiv min\left\{0,y\right\}\text{ for any }y\in \mathbb{R},}$ returns only the negative values of its argument, otherwise zero.^[10]

In the table below, each listed distribution maximizes the entropy for a particular set of functional constraints listed in the third column, and the constraint that${\displaystyle x}$ be included in the support of the probability density, which is listed in the fourth column.^[6][7]

Several listed examples (Bernoulli,geometric,exponential,Laplace,Pareto) are trivially true, because their associated constraints are equivalent to the assignment of their entropy. They are included anyway because their constraint is related to a common or easily measured quantity.

For reference,${\displaystyle \mathrm{\Gamma }(x)={\int }_0^{\mathrm{\infty }}{e}^{-t}{t}^{x-1}dt}$ is thegamma function,${\displaystyle \psi (x)=\frac{d}{dx}\ln \mathrm{\Gamma }(x)=\frac{{\mathrm{\Gamma }}^{\prime }(x)}{\mathrm{\Gamma }(x)}}$ is thedigamma function,${\displaystyle B(p,q)=\frac{\mathrm{\Gamma }(p)\mathrm{\Gamma }(q)}{\mathrm{\Gamma }(p+q)}}$ is thebeta function, and${\displaystyle {\gamma }_{\mathsf{E}}}$ is theEuler-Mascheroni constant.

Table of probability distributions and corresponding maximum entropy constraints

Distribution name	Probability density / mass function	Maximum entropy constraint	Support
Uniform (discrete)	${\displaystyle f(k)=\frac{1}{b-a+1}}$	None	${\displaystyle \{a,a+1,...,b-1,b\}}$
Uniform (continuous)	${\displaystyle f(x)=\frac{1}{b-a}}$	None	${\displaystyle [a,b]}$
Bernoulli	${\displaystyle f(k)=p^k{\left(1-p\right)}^{1-k}}$	${\displaystyle \mathrm{E}[K]=p}$	${\displaystyle \{0,1\}}$
Geometric	${\displaystyle f(k)={\left(1-p\right)}^{k-1}p}$	${\displaystyle \mathrm{E}[K]=\frac{1}{p}}$	${\displaystyle \mathbb{N}\setminus \left\{0\right\}=\{1,2,3,...\}}$
Exponential	${\displaystyle f(x)=\lambda \exp \left(-\lambda x\right)}$	${\displaystyle \mathrm{E}[X]=\frac{1}{\lambda }}$	${\displaystyle [0,\mathrm{\infty })}$
Laplace	${\displaystyle f(x)=\frac{1}{2b}\exp \left(-\frac{|x-\mu |}{b}\right)}$	${\displaystyle \mathrm{E}[|X-\mu |]=b}$	${\displaystyle \mathbb{R}}$
Asymmetric Laplace	${\displaystyle f(x)=\frac{\lambda \exp \left[-(x-m)\lambda s{\kappa }^s\right]}{\kappa +\frac{1}{\kappa }}}$ where${\displaystyle s\equiv \mathrm{sgn}(x-m)}$	${\displaystyle \mathrm{E}\left[(X-m)s{\kappa }^s\right]=\frac{1}{\lambda }}$	${\displaystyle \mathbb{R}}$
Pareto	${\displaystyle f(x)=\frac{\alpha x_m^{\alpha }}{x^{\alpha +1}}}$	${\displaystyle \mathrm{E}[\ln X]=\frac{1}{\alpha }+\ln (x_m)}$	${\displaystyle [x_m,\mathrm{\infty })}$
Normal	${\displaystyle f(x)=\frac{1}{\sqrt{2\pi {\sigma }^2}}\exp \left(-\frac{(x-\mu {)}^2}{2{\sigma }^2}\right)}$	${\displaystyle \mathrm{E}[X]=\mu ,}$ ${\displaystyle \mathrm{E}\left[X^2\right]={\sigma }^2+{\mu }^2}$	${\displaystyle \mathbb{R}}$
Truncated normal	(see article)	${\displaystyle \mathrm{E}\left[X\right]={\mu }_{\mathsf{T}},}$ ${\displaystyle \mathrm{E}\left[X^2\right]={\sigma }_{\mathsf{T}}^2+{\mu }_{\mathsf{T}}^2}$	${\displaystyle [a,b]}$
von Mises	${\displaystyle f(\theta )=\frac{1}{2\pi I_0(\kappa )}\exp \left(\kappa \cos (\theta -\mu )\right)}$	${\displaystyle \mathrm{E}[\cos \mathrm{\Theta }]=\frac{I_1(\kappa )}{I_0(\kappa )}\cos \mu ,}$ ${\displaystyle \mathrm{E}[\sin \mathrm{\Theta }]=\frac{I_1(\kappa )}{I_0(\kappa )}\sin \mu }$	${\displaystyle [0,2\pi )}$
Rayleigh	${\displaystyle f(x)=\frac{x}{{\sigma }^2}\exp \left(-\frac{x^2}{2{\sigma }^2}\right)}$	${\displaystyle \mathrm{E}\left[X^2\right]=2{\sigma }^2,}$ ${\displaystyle \mathrm{E}[\ln X]=\frac{\ln (2{\sigma }^2)-{\gamma }_{\mathrm{E}}}{2}}$	${\displaystyle [0,\mathrm{\infty })}$
Beta	${\displaystyle f(x)=\frac{x^{\alpha -1}(1-x{)}^{\beta -1}}{B(\alpha ,\beta )}}$ for${\displaystyle 0\le x\le 1}$	${\displaystyle \mathrm{E}[\ln X]=\psi (\alpha )-\psi (\alpha +\beta ),}$ ${\displaystyle \mathrm{E}[\ln (1-X)]=\psi (\beta )-\psi (\alpha +\beta )}$	${\displaystyle [0,1]}$
Cauchy	${\displaystyle f(x)=\frac{1}{\pi \left(1+x^2\right)}}$	${\displaystyle \mathrm{E}\left[\ln \left(1+X^2\right)\right]=2\ln 2}$	${\displaystyle \mathbb{R}}$
Chi	${\displaystyle f(x)=\frac{2}{2^{k/2}\mathrm{\Gamma }(k/2)}{x}^{k-1}\exp \left(-\frac{x^2}{2}\right)}$	${\displaystyle \mathrm{E}\left[X^2\right]=k,}$ ${\displaystyle \mathrm{E}[\ln X]=\frac{1}{2}\left[\psi \left(\frac{k}{2}\right)+\ln (2)\right]}$	${\displaystyle [0,\mathrm{\infty })}$
Chi-squared	${\displaystyle f(x)=\frac{1}{2^{k/2}\mathrm{\Gamma }(k/2)}{x}^{\frac{k}{2}-1}\exp \left(-\frac{x}{2}\right)}$	${\displaystyle \mathrm{E}[X]=k,}$ ${\displaystyle \mathrm{E}[\ln X]=\psi \left(\frac{k}{2}\right)+\ln (2)}$	${\displaystyle [0,\mathrm{\infty })}$
Erlang	${\displaystyle f(x)=\frac{{\lambda }^k}{(k-1)!}{x}^{k-1}\exp (-\lambda x)}$	${\displaystyle \mathrm{E}[X]=k/\lambda ,}$ ${\displaystyle \mathrm{E}[\ln X]=\psi (k)-\ln (\lambda )}$	${\displaystyle [0,\mathrm{\infty })}$
Gamma	${\displaystyle f(x)=\frac{x^{k-1}{e}^{-x/\theta }}{{\theta }^k\mathrm{\Gamma }(k)}}$	${\displaystyle \mathrm{E}[X]=k\theta ,}$ ${\displaystyle \mathrm{E}[\ln X]=\psi (k)+\ln \theta }$	${\displaystyle [0,\mathrm{\infty })}$
Lognormal	${\displaystyle f(x)=\frac{1}{\sigma x\sqrt{2\pi }}\exp \left(-\frac{{\left(\ln x-\mu \right)}^2}{2{\sigma }^2}\right)}$	${\displaystyle \mathrm{E}\left[\ln X\right]=\mu ,}$ ${\displaystyle \mathrm{E}\left[\ln (X{)}^2\right]={\sigma }^2+{\mu }^2}$	${\displaystyle (0,\mathrm{\infty })}$
Maxwell–Boltzmann	${\displaystyle f(x)=\frac{1}{a^3}\sqrt{\frac{2}{\pi }}{x}^2\exp \left(-\frac{x^2}{2a^2}\right)}$	${\displaystyle \mathrm{E}\left[X^2\right]=3a^2,}$ ${\displaystyle \mathrm{E}\left[\ln X\right]=1+\ln \left(\frac{a}{\sqrt{2}}\right)-\frac{{\gamma }_{\mathrm{E}}}{2}}$	${\displaystyle [0,\mathrm{\infty })}$
Weibull	${\displaystyle f(x)=\frac{k}{{\lambda }^k}{x}^{k-1}\exp \left(-\frac{x^k}{{\lambda }^k}\right)}$	${\displaystyle \mathrm{E}\left[X^k\right]={\lambda }^k,}$ ${\displaystyle \mathrm{E}\left[\ln X\right]=\ln (\lambda )-\frac{{\gamma }_{\mathrm{E}}}{k}}$	${\displaystyle [0,\mathrm{\infty })}$
Multivariate normal	${\displaystyle f_X(\mathbf{x})=\frac{\exp \left[-\frac{1}{2}{\left(\mathbf{x}-\mathit{\mu }\right)}^{\mathsf{T}}{\mathrm{\Sigma }}^{-1}\left(\mathbf{x}-\mathit{\mu }\right)\right]}{\sqrt{{\left(2\pi \right)}^N\left|\mathrm{\Sigma }\right|}}}$	${\displaystyle \mathrm{E}\left[\mathbf{x}\right]=\mathit{\mu },}$ ${\displaystyle \mathrm{E}\left[(\mathbf{x}-\mathit{\mu })(\mathbf{x}-\mathit{\mu }{)}^{\mathsf{T}}\right]=\mathrm{\Sigma }}$	${\displaystyle {\mathbb{R}}^n}$
Binomial	${\displaystyle f(k)=(\genfrac{}{}{0ex}{}{n}{k})p^k{\left(1-p\right)}^{n-k}}$	${\displaystyle \mathrm{E}[X]=\mu ,}$ ${\displaystyle f\in }$ n-generalized binomial distribution[11]	${\displaystyle \left\{0,\dots ,n\right\}}$
Poisson	${\displaystyle f(k)=\frac{{\lambda }^k{e}^{-\lambda }}{k!}}$	${\displaystyle \mathrm{E}[X]=\lambda ,}$ ${\displaystyle f\in \mathrm{\infty }}$-generalized binomial distribution[11]	${\displaystyle \mathbb{N}=\left\{0,1,\dots \right\}}$
Logistic	${\displaystyle f(x)=\frac{e^{-x}}{{\left(1+e^{-x}\right)}^2}=\frac{e^{+x}}{{\left(e^{+x}+1\right)}^2}}$	${\displaystyle \mathrm{E}[X]=0,}$ ${\displaystyle \mathrm{E}\left[\ln \left(1+e^{-X}\right)\right]=1}$	${\displaystyle \left\{-\mathrm{\infty },\mathrm{\infty }\right\}}$
Random Group Formation distribution[12]	${\displaystyle f(k)=\frac{A{e}^{-bk}}{k^{\gamma }}}$	${\displaystyle \sum _{k}f(k)\ln (kN(k))}$, where${\displaystyle N(k)}$ are group sizes.	${\displaystyle [0,\mathrm{\infty })}$

The maximum entropy principle can be used to upper bound the entropy of statistical mixtures.^[13]

• Exponential family
• Gibbs measure
• Partition function (mathematics)
• Maximal entropy random walk - maximizing entropy rate for a graph

• Entropy and information
• Continuous distributions
• Discrete distributions
• Particle statistics
• Types of probability distributions

• CS1 maint: DOI inactive as of July 2025
• Articles with short description
• Short description matches Wikidata
• Articles needing additional references from August 2009
• All articles needing additional references
• Articles lacking in-text citations from December 2013
• All articles lacking in-text citations
• Articles with multiple maintenance issues