Ininformation theory, theRényi entropy is a quantity that generalizes various notions ofentropy, includingHartley entropy,Shannon entropy,collision entropy, andmin-entropy. The Rényi entropy is named afterAlfréd Rényi, who looked for the most general way to quantify information while preserving additivity for independent events.^[1][2] In the context offractal dimension estimation, the Rényi entropy forms the basis of the concept ofgeneralized dimensions.^[3]

The Rényi entropy is important in ecology and statistics asindex of diversity. The Rényi entropy is also important inquantum information, where it can be used as a measure ofentanglement. In the Heisenberg XY spin chain model, the Rényi entropy as a function ofα can be calculated explicitly because it is anautomorphic function with respect to a particular subgroup of themodular group.^[4][5] Intheoretical computer science, the min-entropy is used in the context ofrandomness extractors.

The Rényi entropy of order⁠${\displaystyle \alpha }$⁠, where and⁠${\displaystyle \alpha \ne 1}$⁠, is defined as^[1]$${\displaystyle {\mathrm{H}}_{\alpha }(X)=\frac{1}{1-\alpha }\log \left(\sum _{i=1}^n{p}_i^{\alpha }\right).}$$It is further defined at${\displaystyle \alpha =0,1,\mathrm{\infty }}$ as$${\displaystyle {\mathrm{H}}_{\alpha }(X)=\underset{\gamma \to \alpha }{lim}{\mathrm{H}}_{\gamma }(X).}$$

Here,${\displaystyle X}$ is adiscrete random variable with possible outcomes in the set${\displaystyle \mathcal{A}=\{x_1,x_2,...,x_n\}}$ and corresponding probabilities${\displaystyle p_i\doteq Pr(X=x_i)}$ for⁠${\displaystyle i=1,\dots ,n}$⁠. The resultingunit of information is determined by the base of thelogarithm, e.g.shannon for base 2, ornat for basee.If the probabilities are${\displaystyle p_i=1/n}$ for all⁠${\displaystyle i=1,\dots ,n}$⁠, then all the Rényi entropies of the distribution are equal:⁠${\displaystyle {\mathrm{H}}_{\alpha }(X)=\log n}$⁠.In general, for all discrete random variables⁠${\displaystyle X}$⁠,${\displaystyle {\mathrm{H}}_{\alpha }(X)}$ is a non-increasing function in⁠${\displaystyle \alpha }$⁠.

Applications often exploit the following relation between the Rényi entropy and theα-norm of the vector of probabilities:$${\displaystyle {\mathrm{H}}_{\alpha }(X)=\frac{\alpha }{1-\alpha }\log \left({\Vert P\Vert }_{\alpha }\right).}$$Here, the discrete probability distribution${\displaystyle P=(p_1,\dots ,p_n)}$ is interpreted as a vector in${\displaystyle {\mathbb{R}}^n}$ with${\displaystyle p_i\ge 0}$ and$\sum _{i=1}^n{p}_i=1$.

The Rényi entropy for any${\displaystyle \alpha \ge 0}$ isSchur concave, a property that can be proven by theSchur–Ostrowski criterion.

As${\displaystyle \alpha }$ approaches zero, the Rényi entropy increasingly weighs all events with nonzero probability more equally, regardless of their probabilities. In the limit for⁠${\displaystyle \alpha \to 0}$⁠, the Rényi entropy is just the logarithm of the size of thesupport ofX. The limit for${\displaystyle \alpha \to 1}$ is theShannon entropy. As${\displaystyle \alpha }$ approaches infinity, the Rényi entropy is increasingly determined by the events of highest probability.

${\displaystyle {\mathrm{H}}_0(X)}$ is${\displaystyle \log n}$ where${\displaystyle n}$ is the number of non-zero probabilities.^[6] If the probabilities are all nonzero, it is simply the logarithm of thecardinality of the alphabet (⁠${\displaystyle \mathcal{A}}$⁠) of⁠${\displaystyle X}$⁠, sometimes called theHartley entropy of⁠${\displaystyle X}$⁠,$${\displaystyle {\mathrm{H}}_0(X)=\log n=\log |\mathcal{A}|}$$

The limiting value of${\displaystyle {\mathrm{H}}_{\alpha }}$ as${\displaystyle \alpha \to 1}$ is theShannon entropy:^[7]$${\displaystyle {\mathrm{H}}_1(X)\equiv \underset{\alpha \to 1}{lim}{\mathrm{H}}_{\alpha }(X)=-\sum _{i=1}^n{p}_i\log p_i}$$

Collision entropy, sometimes just called "Rényi entropy", refers to the case⁠${\displaystyle \alpha =2}$⁠,$${\displaystyle {\mathrm{H}}_2(X)=-\log \sum _{i=1}^n{p}_i^2=-\log P(X=Y),}$$where${\displaystyle X}$ and${\displaystyle Y}$ areindependent and identically distributed. The collision entropy is related to theindex of coincidence. It is the negative logarithm of theSimpson diversity index.

In the limit as⁠${\displaystyle \alpha \to \mathrm{\infty }}$⁠, the Rényi entropy${\displaystyle {\mathrm{H}}_{\alpha }}$ converges to themin-entropy⁠${\displaystyle {\mathrm{H}}_{\mathrm{\infty }}}$⁠:$${\displaystyle {\mathrm{H}}_{\mathrm{\infty }}(X)\doteq \underset{i}{min}(-\log p_i)=-(\underset{i}{max}\log p_i)=-\log \underset{i}{max}{p}_i.}$$

Equivalently, the min-entropy${\displaystyle {\mathrm{H}}_{\mathrm{\infty }}(X)}$ is the largest real numberb such that all events occur with probability at most⁠${\displaystyle 2^{-b}}$⁠.

The namemin-entropy stems from the fact that it is the smallest entropy measure in the family of Rényi entropies.In this sense, it is the strongest way to measure the information content of a discrete random variable.In particular, the min-entropy is never larger than theShannon entropy.

The min-entropy has important applications forrandomness extractors intheoretical computer science:Extractors are able to extract randomness from random sources that have a large min-entropy; merely having a largeShannon entropy does not suffice for this task.

That${\displaystyle {\mathrm{H}}_{\alpha }}$ is non-increasing in${\displaystyle \alpha }$ for any given distribution of probabilities⁠${\displaystyle p_i}$⁠,which can be proven by differentiation,^[8] as$${\displaystyle -\frac{d{\mathrm{H}}_{\alpha }}{d\alpha }=\frac{1}{(1-\alpha {)}^2}\sum _{i=1}^n{z}_i\log (z_i/p_i)=\frac{1}{(1-\alpha {)}^2}{D}_{\text{KL}}(z\Vert p)}$$which is proportional toKullback–Leibler divergence (which is always non-negative), where$z_i=p_i^{\alpha }/\sum _{j=1}^n{p}_j^{\alpha }$. In particular, it is strictly positive except when the distribution is uniform.

At the${\displaystyle \alpha \to 1}$ limit, we have$-\frac{d{\mathrm{H}}_{\alpha }}{d\alpha }\to \frac{1}{2}\sum _i{p}_i{\left(\ln p_i+H(p)\right)}^2$.

In particular cases inequalities can be proven also byJensen's inequality:^[9][10]$${\displaystyle \log n={\mathrm{H}}_0\ge {\mathrm{H}}_1\ge {\mathrm{H}}_2\ge {\mathrm{H}}_{\mathrm{\infty }}.}$$

For values of⁠${\displaystyle \alpha >1}$⁠, inequalities in the other direction also hold. In particular, we have^[11][12]$${\displaystyle {\mathrm{H}}_2\le 2{\mathrm{H}}_{\mathrm{\infty }}.}$$

On the other hand, the Shannon entropy${\displaystyle {\mathrm{H}}_1}$ can be arbitrarily high for a random variable${\displaystyle X}$ that has a given min-entropy. An example of this is given by the sequence of random variables${\displaystyle X_n\sim \{0,\dots ,n\}}$ for${\displaystyle n\ge 1}$ such that${\displaystyle P(X_n=0)=1/2}$ and${\displaystyle P(X_n=x)=1/(2n)}$ since${\displaystyle {\mathrm{H}}_{\mathrm{\infty }}(X_n)=\log 2}$ but⁠${\displaystyle {\mathrm{H}}_1(X_n)=(\log 2+\log 2n)/2}$⁠.

As well as the absolute Rényi entropies, Rényi also defined a spectrum of divergence measures generalising theKullback–Leibler divergence.^[13]

TheRényi divergence of order⁠${\displaystyle \alpha }$⁠ oralpha-divergence of a distributionP from a distributionQ is defined to bewhen⁠⁠ and⁠${\displaystyle \alpha \ne 1}$⁠. We can define the Rényi divergence for the special valuesα = 0, 1, ∞ by taking a limit, and in particular the limitα → 1 gives the Kullback–Leibler divergence.

Some special cases:

• ⁠${\displaystyle D_0(P\Vert Q)=-\log Q(\{i:p_i>0\})}$⁠: minus thelog probability underQ thatp_i > 0;
• ⁠${\displaystyle D_{1/2}(P\Vert Q)=-2\log \sum _{i=1}^n\sqrt{p_i{q}_i}}$⁠: minus twice the logarithm of theBhattacharyya coefficient; (Nielsen & Boltz (2010))
• ⁠${\displaystyle D_1(P\Vert Q)=\sum _{i=1}^n{p}_i\log \frac{p_i}{q_i}}$⁠: theKullback–Leibler divergence;
• ⁠${\displaystyle D_2(P\Vert Q)=\log ⟨\frac{p_i}{q_i}⟩}$⁠: the log of the expected ratio of the probabilities;
• ⁠${\displaystyle D_{\mathrm{\infty }}(P\Vert Q)=\log \underset{i}{sup}\frac{p_i}{q_i}}$⁠: the log of the maximum ratio of the probabilities.

The Rényi divergence is indeed adivergence, meaning simply that${\displaystyle D_{\alpha }(P\Vert Q)}$ is greater than or equal to zero, and zero only whenP =Q. For any fixed distributionsP andQ, the Rényi divergence is nondecreasing as a function of its orderα, and it is continuous on the set ofα for which it is finite,^[13] or for the sake of brevity, the information of orderα obtained if the distributionP is replaced by the distributionQ.^[1]

A pair of probability distributions can be viewed as a game of chance in which one of the distributions defines official odds and the other contains the actual probabilities. Knowledge of the actual probabilities allows a player to profit from the game. The expected profit rate is connected to the Rényi divergence as follows^[14]$${\displaystyle \mathrm{E}\mathrm{x}\mathrm{p}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{e}\mathrm{d}\mathrm{R}\mathrm{a}\mathrm{t}\mathrm{e}=\frac{1}{R}{D}_1(b\Vert m)+\frac{R-1}{R}{D}_{1/R}(b\Vert m),}$$where${\displaystyle m}$ is the distribution defining the official odds (i.e. the "market") for the game,${\displaystyle b}$ is the investor-believed distribution and${\displaystyle R}$ is the investor's risk aversion (theArrow–Pratt relative risk aversion).

If the true distribution is${\displaystyle p}$ (not necessarily coinciding with the investor's belief⁠${\displaystyle b}$⁠), the long-term realized rate converges to the true expectation which has a similar mathematical structure^[14]$${\displaystyle \mathrm{R}\mathrm{e}\mathrm{a}\mathrm{l}\mathrm{i}\mathrm{z}\mathrm{e}\mathrm{d}\mathrm{R}\mathrm{a}\mathrm{t}\mathrm{e}=\frac{1}{R}(D_1(p\Vert m)-D_1(p\Vert b))+\frac{R-1}{R}{D}_{1/R}(b\Vert m).}$$

The value⁠${\displaystyle \alpha =1}$⁠, which gives theShannon entropy and theKullback–Leibler divergence, is the only value at which thechain rule of conditional probability holds exactly:$${\displaystyle \mathrm{H}(A,X)=\mathrm{H}(A)+{\mathbb{E}}_{a\sim A}[\mathrm{H}(X|A=a)]}$$for the absolute entropies, and$${\displaystyle D_{\mathrm{K}\mathrm{L}}(p(x|a)p(a)\Vert m(x,a))=D_{\mathrm{K}\mathrm{L}}(p(a)\Vert m(a))+{\mathbb{E}}_{p(a)}\{D_{\mathrm{K}\mathrm{L}}(p(x|a)\Vert m(x|a))\},}$$for the relative entropies.

The latter in particular means that if we seek a distributionp(x,a) which minimizes the divergence from some underlying prior measurem(x,a), and we acquire new information which only affects the distribution ofa, then the distribution ofp(x|a) remainsm(x|a), unchanged.

The other Rényi divergences satisfy the criteria of being positive and continuous, being invariant under 1-to-1 co-ordinate transformations, and of combining additively whenA andX are independent, so that ifp(A,X) =p(A)p(X), then$${\displaystyle {\mathrm{H}}_{\alpha }(A,X)={\mathrm{H}}_{\alpha }(A)+{\mathrm{H}}_{\alpha }(X)}$$and$${\displaystyle D_{\alpha }(P(A)P(X)\Vert Q(A)Q(X))=D_{\alpha }(P(A)\Vert Q(A))+D_{\alpha }(P(X)\Vert Q(X)).}$$

The stronger properties of the${\displaystyle \alpha =1}$ quantities allow the definition ofconditional information andmutual information from communication theory.

The Rényi entropies and divergences for anexponential family admit simple expressions^[15]$${\displaystyle {\mathrm{H}}_{\alpha }(p_F(x;\theta ))=\frac{1}{1-\alpha }\left(F(\alpha \theta )-\alpha F(\theta )+\log E_p\left[e^{(\alpha -1)k(x)}\right]\right)}$$and$${\displaystyle D_{\alpha }(p:q)=\frac{J_{F,\alpha }(\theta :{\theta }^{\prime })}{1-\alpha }}$$where$${\displaystyle J_{F,\alpha }(\theta :{\theta }^{\prime })=\alpha F(\theta )+(1-\alpha )F({\theta }^{\prime })-F(\alpha \theta +(1-\alpha ){\theta }^{\prime })}$$is a Jensen difference divergence.

The Rényi entropy can be generalized to the quantum case as

${\displaystyle H_{\alpha }(\rho ):=\frac{1}{1-\alpha }\log \mathrm{tr}({\rho }^{\alpha })}$

where${\displaystyle \rho }$ is normalizeddensity matrix. Its limit as${\displaystyle \alpha \to 1}$ is thevon Neumann entropy.^[16]

The Rényi relative entropy (or divergence) can be generalized in two possible ways: the straightforward quantum Rényi relative entropy

${\displaystyle D_{\alpha }(\rho \Vert \sigma )=\frac{1}{\alpha -1}\log \mathrm{tr}({\rho }^{\alpha }{\sigma }^{1-\alpha })}$

and the sandwiched Rényi relative entropy

${\displaystyle {\tilde{D}}_{\alpha }(\rho \Vert \sigma )=\frac{1}{\alpha -1}\log \mathrm{tr}\left[{\left({\sigma }^{\frac{1-\alpha }{2\alpha }}\rho {\sigma }^{\frac{1-\alpha }{2\alpha }}\right)}^{\alpha }\right]}$

Both converge to thequantum relative entropy in the limit${\displaystyle \alpha \to 1}$. However, the sandwiched Rényi relative entropy has more convenient properties when used to defined conditional entropies,^[16] and has found applications inquantum key distribution.^[17][18]

• Diversity indices
• Tsallis entropy
• Generalized entropy index

• Information theory
• Entropy and information

• Articles with short description
• Short description is different from Wikidata