Adiversity index is a method of measuring how many different types (e.g.species) there are in a dataset (e.g. a community). Diversity indices are statistical representations of different aspects of biodiversity (e.g.richness,evenness, anddominance), which are useful simplifications for comparing different communities or sites.

When diversity indices are used inecology, the types of interest are usually species, but they can also be other categories, such asgenera,families,functional types, orhaplotypes. The entities of interest are usually individual organisms (e.g. plants or animals), and the measure of abundance can be, for example, number of individuals, biomass or coverage. Indemography, the entities of interest can be people, and the types of interest various demographic groups. Ininformation science, the entities can be characters and the types of the different letters of the alphabet. The most commonly used diversity indices are simple transformations of the effective number of types (also known as 'true diversity'), but each diversity index can also be interpreted in its own right as a measure corresponding to some real phenomenon (but a different one for each diversity index).^[1][2][3][4]

Many indices only account for categorical diversity between subjects or entities. Such indices, however do not account for the total variation (diversity) that can be held between subjects or entities which occurs only when both categorical and qualitative diversity are calculated.

Diversity indices described in this article include:

• Richness, simply a count of the number of types in a dataset.
• Shannon index, which also takes into account the proportional abundance of each class under a weighted geometric mean.
  • The Rényi entropy, which adds the ability to freely vary the kind of weighted mean used.
• Simpson index, which too takes into account the proportional abundance of each class under a weighted arithmetic mean
• Berger–Parker index, which gives the proportional abundance of the most abundant type.
• Effective number of species (true diversity), which allows for freely varying the kind of weighted mean used, and has a intuitive meaning.^[4]

Some more sophisticated indices also account for thephylogenetic relatedness among the types. These are called phylo-divergence indices, and are not yet described in this article.^[5]

True diversity, or the effective number of types, refers to the number of equally abundant types needed for the average proportional abundance of the types to equal that observed in the dataset of interest (where all types may not be equally abundant). The true diversity in a dataset is calculated by first taking the weightedgeneralized meanM_q−1 of the proportional abundances of the types in the dataset, and then taking thereciprocal of this. The equation is:^[3][4]

$${\displaystyle {}^{q}D=\frac{1}{M_{q-1}}=\frac{1}{\sqrt[q-1]{\sum _{i=1}^R{p}_i{p}_i^{q-1}}}={\left(\sum _{i=1}^R{p}_i^q\right)}^{1/(1-q)}}$$

ThedenominatorM_q−1 equals the average proportional abundance of the types in the dataset as calculated with the weightedgeneralized mean with exponentq − 1. In the equation,R is richness (the total number of types in the dataset), and the proportional abundance of theith type isp_i. The proportional abundances themselves are used as the nominal weights. The numbers${\displaystyle {}^{q}D}$ are calledHill numbers of orderq oreffective number of species.^[6]

Whenq = 1, the above equation is undefined. However, themathematical limit asq approaches 1 is well defined and the corresponding diversity is calculated with the following equation:

$${\displaystyle {}^{1}D=\frac{1}{\prod _{i=1}^R{p}_i^{p_i}}=\exp \left(-\sum _{i=1}^R{p}_i\ln (p_i)\right)}$$

which is the exponential of theShannon entropy calculated with natural logarithms (see above). In other domains, this statistic is also known as theperplexity.

The general equation of diversity is often written in the form^[1][2]

$${\displaystyle {}^{q}D={\left(\sum _{i=1}^R{p}_i^q\right)}^{1/(1-q)}}$$

and the term inside the parentheses is called the basic sum. Some popular diversity indices correspond to the basic sum as calculated with different values ofq.^[2]

The value ofq is often referred to as the order of the diversity. It defines the sensitivity of the true diversity to rare vs. abundant species by modifying how the weighted mean of the species' proportional abundances is calculated. With some values of the parameterq, the value of the generalized meanM_q−1 assumes familiar kinds of weighted means as special cases. In particular,

• q = 0 corresponds to the weightedarithmetic mean,
• q = 1 to the weightedgeometric mean, and
• q = 2 to the weightedharmonic mean.^[7]
• Asq approachesinfinity, the weighted generalized mean with exponentq − 1 approaches the maximump_i value, which is the proportional abundance of the most abundant species in the dataset.

Generally, increasing the value ofq increases the effective weight given to the most abundant species. This leads to obtaining a largerM_q−1 value and a smaller true diversity (^qD) value with increasingq.

Whenq = 1, the weighted geometric mean of thep_i values is used, and each species is exactly weighted by its proportional abundance (in the weighted geometric mean, the weights are the exponents). Whenq > 1, the weight given to abundant species is exaggerated, and whenq < 1, the weight given to rare species is. Atq = 0, the species weights exactly cancel out the species proportional abundances, such that the weighted mean of thep_i values equals1 /R even when all species are not equally abundant. Atq = 0, the effective number of species,⁰D, hence equals the actual number of speciesR. In the context of diversity,q is generally limited to non-negative values. This is because negative values ofq would give rare species so much more weight than abundant ones that^qD would exceedR.^[3][4]

RichnessR simply quantifies how many different types the dataset of interest contains. For example, species richness (usually notedS) is simply the number of species, e.g. at a particular site. Richness is a simple measure, so it has been a popular diversity index in ecology, where abundance data are often not available.^[8] If true diversity is calculated withq = 0, the effective number of types (⁰D) equals the actual number of types, which is identical to Richness (R).^[2][4]

TheShannon index has been a popular diversity index in the ecological literature, where it is also known asShannon's diversity index,Shannon–Wiener index, and (erroneously)Shannon–Weaver index.^[9] The measure was originally proposed byClaude Shannon in 1948 to quantify theentropy (henceShannon entropy, related toShannon information content) in strings of text.^[10] The idea is that the more letters there are, and the closer their proportional abundances in the string of interest, the more difficult it is to correctly predict which letter will be the next one in the string. The Shannon entropy quantifies the uncertainty (entropy or degree of surprise) associated with this prediction. It is most often calculated as follows:

$${\displaystyle H^{\prime }=-\sum _{i=1}^R{p}_i\ln (p_i)}$$

wherep_i is the proportion of characters belonging to theith type of letter in the string of interest. In ecology,p_i is often the proportion of individuals belonging to theith species in the dataset of interest. Then the Shannon entropy quantifies the uncertainty in predicting the species identity of an individual that is taken at random from the dataset.

Although the equation is here written with natural logarithms, the base of the logarithm used when calculating the Shannon entropy can be chosen freely. Shannon himself discussed logarithm bases 2, 10 ande, and these have since become the most popular bases in applications that use the Shannon entropy. Each log base corresponds to a different measurement unit, which has been called binary digits (bits), decimal digits (decits), and natural digits (nats) for the bases 2, 10 ande, respectively. Comparing Shannon entropy values that were originally calculated with different log bases requires converting them to the same log base: change from the basea to baseb is obtained with multiplication bylog_b(a).^[10]

The Shannon index (H') is related to theweighted geometric mean of the proportional abundances of the types. Specifically, it equals the logarithm of true diversity as calculated withq = 1:^[3]

$${\displaystyle H^{\prime }=-\sum _{i=1}^R{p}_i\ln (p_i)=-\sum _{i=1}^R\ln \left(p_i^{p_i}\right)}$$

This can also be written

$${\displaystyle H^{\prime }=-\left[\ln \left(p_1^{p_1}\right)+\ln \left(p_2^{p_2}\right)+\ln \left(p_3^{p_3}\right)+\cdots +\ln \left(p_R^{p_R}\right)\right]}$$

which equals

$${\displaystyle H^{\prime }=-\ln \left(p_1^{p_1}{p}_2^{p_2}{p}_3^{p_3}\cdots p_R^{p_R}\right)=\ln \left(\frac{1}{p_1^{p_1}{p}_2^{p_2}{p}_3^{p_3}\cdots p_R^{p_R}}\right)=\ln \left(\frac{1}{\prod _{i=1}^R{p}_i^{p_i}}\right)}$$

Since the sum of thep_i values equals 1 by definition, thedenominator equals the weighted geometric mean of thep_i values, with thep_i values themselves being used as the weights (exponents in the equation). The term within the parentheses hence equals true diversity¹D, andH' equalsln(¹D).^[1][3][4]

When all types in the dataset of interest are equally common, allp_i values equal1 /R, and the Shannon index hence takes the valueln(R). The more unequal the abundances of the types, the larger the weighted geometric mean of thep_i values, and the smaller the corresponding Shannon entropy. If practically all abundance is concentrated to one type, and the other types are very rare (even if there are many of them), Shannon entropy approaches zero. When there is only one type in the dataset, Shannon entropy exactly equals zero (there is no uncertainty in predicting the type of the next randomly chosen entity).

In machine learning the Shannon index is also called asInformation gain.

TheRényi entropy is a generalization of the Shannon entropy to other values ofq than 1. It can be expressed:

$${\displaystyle {}^{q}H=\frac{1}{1-q}\ln \left(\sum _{i=1}^R{p}_i^q\right)}$$

which equals

$${\displaystyle {}^{q}H=\ln \left(\frac{1}{\sqrt[q-1]{\sum _{i=1}^R{p}_i{p}_i^{q-1}}}\right)=\ln ({}^{q}D)}$$

This means that taking the logarithm of true diversity based on any value ofq gives the Rényi entropy corresponding to the same value ofq.

The Simpson index was introduced in 1949 byEdward H. Simpson to measure the degree of concentration when individuals are classified into types.^[11] The same index was rediscovered byOrris C. Herfindahl in 1950.^[12] The square root of the index had already been introduced in 1945 by the economistAlbert O. Hirschman.^[13] As a result, the same measure is usually known as the Simpson index in ecology, and as theHerfindahl index or the Herfindahl–Hirschman index (HHI) in economics.

The measure equals the probability that two entities taken at random from the dataset of interest represent the same type.^[11] It equals:

$${\displaystyle \lambda =\sum _{i=1}^R{p}_i^2,}$$

whereR is richness (the total number of types in the dataset). This equation is also equal to the weighted arithmetic mean of the proportional abundancesp_i of the types of interest, with the proportional abundances themselves being used as the weights.^[1] Proportional abundances are by definition constrained to values between zero and one, but it is a weighted arithmetic mean, henceλ ≥ 1/R, which is reached when all types are equally abundant.

By comparing the equation used to calculate λ with the equations used to calculate true diversity, it can be seen that1/λ equals²D, i.e., true diversity as calculated withq = 2. The original Simpson's index hence equals the corresponding basic sum.^[2]

The interpretation of λ as the probability that two entities taken at random from the dataset of interest represent the same type assumes that the entities are sampled with replacement. If the dataset is very large, sampling without replacement gives approximately the same result, but in small datasets, the difference can be substantial. If the dataset is small, and sampling without replacement is assumed, the probability of obtaining the same type with both random draws is:

$${\displaystyle \ell =\frac{\sum _{i=1}^R{n}_i(n_i-1)}{N(N-1)}}$$

wheren_i is the number of entities belonging to theith type andN is the total number of entities in the dataset.^[11] This form of the Simpson index is also known as the Hunter–Gaston index in microbiology.^[14]

Since the mean proportional abundance of the types increases with decreasing number of types and increasing abundance of the most abundant type, λ obtains small values in datasets of high diversity and large values in datasets of low diversity. This is counterintuitive behavior for a diversity index, so often, such transformations of λ that increase with increasing diversity have been used instead. The most popular of such indices have been the inverse Simpson index (1/λ) and theGini–Simpson index (1 − λ).^[1][2] Both of these have also been called the Simpson index in the ecological literature, so care is needed to avoid accidentally comparing the different indices as if they were the same.

The inverse Simpson index equals:

$${\displaystyle \frac{1}{\lambda }=\frac{1}{\sum _{i=1}^R{p}_i^2}={}^{2}D}$$

This simply equals true diversity of order 2, i.e. the effective number of types that is obtained when the weighted arithmetic mean is used to quantify average proportional abundance of types in the dataset of interest.

The index is also used as a measure of theeffective number of parties.

The Gini-Simpson Index is also calledGini impurity, orGini's diversity index^[15] in the field ofMachine Learning. The original Simpson index λ equals the probability that two entities taken at random from the dataset of interest (with replacement) represent the same type. Its transformation 1 − λ, therefore, equals the probability that the two entities represent different types. This measure is also known in ecology as the probability of interspecific encounter (PIE)^[16] and the Gini–Simpson index.^[2] It can be expressed as a transformation of the true diversity of order 2:

$${\displaystyle 1-\lambda =1-\sum _{i=1}^R{p}_i^2=1-\frac{1}{{}^{2}D}}$$

The Gibbs–Martin index of sociology, psychology, and management studies,^[17] which is also known as the Blau index, is the same measure as the Gini–Simpson index.

The quantity is also known as theexpected heterozygosity in population genetics.

The Berger–Parker index, named afterWolfgang H. Berger andFrances Lawrence Parker,^[18] equals the maximump_i value in the dataset, i.e., the proportional abundance of the most abundant type. This corresponds to the weightedgeneralized mean of thep_i values whenq approaches infinity, and hence equals the inverse of the true diversity of order infinity (1/^∞D).

• Colinvaux, Paul A. (1973).Introduction to Ecology. Wiley.ISBN 0-471-16498-4.
• Cover, Thomas M.; Thomas, Joy A. (1991).Elements of Information Theory. Wiley.ISBN 0-471-06259-6. See chapter 5 for an elaboration of coding procedures described informally above.
• Chao, A.; Shen, T-J. (2003)."Nonparametric estimation of Shannon's index of diversity when there are unseen species in sample"(PDF).Environmental and Ecological Statistics.10 (4):429–443.Bibcode:2003EnvES..10..429C.doi:10.1023/A:1026096204727.S2CID 20389926.

• Simpson's Diversity index
• Diversity indicesArchived 2005-12-19 at theWayback Machine gives some examples of estimates of Simpson's index for real ecosystems.

• Measurement of biodiversity
• Index numbers
• Summary statistics for categorical data

• Articles with short description
• Short description is different from Wikidata
• Webarchive template wayback links