TheTheil index is a statistic primarily used to measureeconomic inequality^[1] and other economic phenomena, though it has also been used to measure racial segregation.^[2][3] The Theil indexT_T is the same asredundancy ininformation theory which is the maximum possibleentropy of the data minus the observed entropy. It is a special case of thegeneralized entropy index. It can be viewed as a measure of redundancy, lack of diversity, isolation, segregation, inequality, non-randomness, and compressibility. It was proposed by a DutcheconometricianHenri Theil (1924–2000) at theErasmus University Rotterdam.^[3]

Henri Theil himself said (1967): "The (Theil) index can be interpreted as the expected information content of the indirect message which transforms the population shares as prior probabilities into the income shares as posterior probabilities."^[4]Amartya Sen noted, "But the fact remains that the Theil index is an arbitrary formula, and the average of the logarithms of the reciprocals of income shares weighted by income is not a measure that is exactly overflowing with intuitive sense."^[4]

For a population ofN "agents" each with characteristicx, the situation may be represented by the listx_i (i = 1,...,N) wherex_i is the characteristic of agenti. For example, if the characteristic is income, thenx_i is the income of agenti.

The TheilT index is defined as^[5]

${\displaystyle T_T=T_{\alpha =1}=\frac{1}{N}\sum _{i=1}^N\frac{x_i}{\mu }\ln \left(\frac{x_i}{\mu }\right)}$

and the TheilL index is defined as^[5]

${\displaystyle T_L=T_{\alpha =0}=\frac{1}{N}\sum _{i=1}^N\ln \left(\frac{\mu }{x_i}\right)}$

where${\displaystyle \mu }$ is the mean income:

${\displaystyle \mu =\frac{1}{N}\sum _{i=1}^N{x}_i}$

Theil-L is an income-distribution's dis-entropy per person, measured with respect to maximum entropy (...which is achieved with complete equality).

(In an alternative interpretation of it, Theil-L is the natural-logarithm of the geometric-mean of the ratio: (mean income)/(income i), over all the incomes. The related Atkinson(1) is just 1 minus the geometric-mean of (income i)/(mean income), over the income distribution.)

Because a transfer between a larger income & a smaller one will change the smaller income's ratio more than it changes the larger income's ratio, the transfer-principle is satisfied by this index.

Equivalently, if the situation is characterized by a discrete distribution functionf_k (k = 0,...,W) wheref_k is the fraction of the population with incomek andW =Nμ is the total income, then${\displaystyle \sum _{k=0}^W{f}_k=1}$ and the Theil index is:

${\displaystyle T_T=\sum _{k=0}^W{f}_k\frac{k}{\mu }\ln \left(\frac{k}{\mu }\right)}$

where${\displaystyle \mu }$ is again the mean income:

${\displaystyle \mu =\sum _{k=0}^{W}k{f}_k}$

Note that in this case incomek is aninteger andk=1 represents the smallest increment of income possible (e.g., cents).

if the situation is characterized by a continuous distribution functionf(k) (supported from 0 to infinity) wheref(k) dk is the fraction of the population with incomek tok + dk, then the Theil index is:

${\displaystyle T_T={\int }_0^{\mathrm{\infty }}f(k)\frac{k}{\mu }\ln \left(\frac{k}{\mu }\right)dk}$

where the mean is:

${\displaystyle \mu ={\int }_0^{\mathrm{\infty }}kf(k)dk}$

Theil indices for some common continuous probability distributions are given in the table below:

Income distribution function	PDF(x) (x ≥ 0)	Theil coefficient (nats)
Dirac delta function	${\displaystyle \delta (x-x_0),x_0>0}$	0
Uniform distribution		${\displaystyle \ln \left(\frac{2a}{(a+b)\sqrt{e}}\right)+\frac{b^2}{b^2-a^2}\ln (b/a)}$
Exponential distribution	${\displaystyle \lambda e^{-x\lambda },x>0}$	${\displaystyle 1-}$${\displaystyle \gamma }$
Log-normal distribution	${\displaystyle \frac{1}{\sigma \sqrt{2\pi }}{e}^{(-(\ln (x)-\mu {)}^2)/{\sigma }^2}}$	${\displaystyle \frac{{\sigma }^2}{2}}$
Pareto distribution		${\displaystyle \ln (1-1/\alpha )+\frac{1}{\alpha -1}}$    (α>1)
Chi-squared distribution	${\displaystyle \frac{2^{-k/2}{e}^{-x/2}{x}^{k/2-1}}{\mathrm{\Gamma }(k/2)}}$	${\displaystyle \ln (2/k)+}$${\displaystyle {\psi }^{(0)}}$${\displaystyle (1+k/2)}$
Gamma distribution[6]	${\displaystyle \frac{e^{-x/\theta }{x}^{k-1}{\theta }^{-k}}{\mathrm{\Gamma }(k)}}$	${\displaystyle {\psi }^{(0)}}$${\displaystyle (1+k)-\ln (k)}$
Weibull distribution	${\displaystyle \frac{k}{\lambda }{\left(\frac{x}{\lambda }\right)}^{k-1}{e}^{-(x/\lambda {)}^k}}$	${\displaystyle \frac{1}{k}}$${\displaystyle {\psi }^{(0)}}$${\displaystyle (1+1/k)-\ln \left(\mathrm{\Gamma }(1+1/k)\right)}$

If everyone has the same income, thenT_T equals 0. If one person has all the income, thenT_T gives the result${\displaystyle \ln N}$, which is maximum inequality. DividingT_T by${\displaystyle \ln N}$ can normalize the equation to range from 0 to 1, but then theindependence axiom is violated:${\displaystyle T[x\cup x]\ne T[x]}$ and does not qualify as a measure of inequality.

The Theil index measures an entropic "distance" the population is away from the egalitarian state of everyone having the same income. The numerical result is in terms of negative entropy so that a higher number indicates more order that is further away from the complete equality. Formulating the index to represent negative entropy instead of entropy allows it to be a measure of inequality rather than equality.

The Theil index can be transformed into anAtkinson index, which has a range between 0 and 1 (0% and 100%), where 0 indicates perfect equality and 1 (100%) indicates maximum inequality. (SeeGeneralized entropy index for the transformation.)

The Theil index is derived fromShannon's measure ofinformation entropy${\displaystyle S}$, where entropy is a measure of randomness in a given set of information. In information theory, physics, and the Theil index, the general form of entropy is

${\displaystyle S=k\sum _{i=1}^N\left(p_i{\log }_a\left(\frac{1}{p_i}\right)\right)=-k\sum _{i=1}^N\left(p_i{\log }_a\left(p_i\right)\right)}$

where

• ${\displaystyle i}$ is an individual item from the set (such as an individual member from a population, or an individual byte from acomputer file).
• ${\displaystyle p_i}$ is the probability of finding${\displaystyle i}$ from a random sample from the set.
• ${\displaystyle k}$ is a constant.^{[note 1]}
• ${\displaystyle {\log }_a\left(x\right)}$ is alogarithm with a base equal to${\displaystyle a}$.^{[note 2]}

When looking at the distribution of income in a population,${\displaystyle p_i}$ is equal to the ratio of a particular individual's income to the total income of the entire population. This gives the observed entropy${\displaystyle S_{\text{Theil}}}$ of a population to be:

${\displaystyle S_{\text{Theil}}=\sum _{i=1}^N\left(\frac{x_i}{N\overline{x}}\ln \left(\frac{N\overline{x}}{x_i}\right)\right)}$

where

• ${\displaystyle x_i}$ is the income of a particular individual.
• ${\displaystyle \left(N\overline{x}\right)}$ is the total income of the entire population, with

• ${\displaystyle N}$ being the number of individuals in the population.
• ${\displaystyle \overline{x}}$ ("x bar") being the average income of the population.

• ${\displaystyle \ln \left(x\right)}$ is thenatural logarithm of${\displaystyle x}$:${\displaystyle \left({\log }_e\left(x\right)\right)}$.

The Theil index${\displaystyle T_T}$ measures how far the observed entropy (${\displaystyle S_{\text{Theil}}}$, which represents how randomly income is distributed) is from the highest possible entropy (${\displaystyle S_{\text{max}}=\ln \left(N\right)}$,^{[note 3]} which represents income being maximally distributed amongst individuals in the population– a distribution analogous to the [most likely] outcome of an infinite number of random coin tosses: an equal distribution of heads and tails). Therefore, the Theil index is the difference between the theoretical maximum entropy (which would be reached if the incomes of every individual were equal) minus the observed entropy:

${\displaystyle T_T=S_{\text{max}}-S_{\text{Theil}}=\ln \left(N\right)-S_{\text{Theil}}}$

When${\displaystyle x}$ is in units of population/species,${\displaystyle S_{\text{Theil}}}$ is a measure of biodiversity and is called theShannon index. If the Theil index is used with x=population/species, it is a measure of inequality of population among a set of species, or "bio-isolation" as opposed to "wealth isolation".

The Theil index measures what is calledredundancy in information theory.^[5] It is the left over "information space" that was not utilized to convey information, which reduces the effectiveness of theprice signal.^{[original research?]} The Theil index is a measure of the redundancy of income (or other measure of wealth) in some individuals. Redundancy in some individuals implies scarcity in others. A high Theil index indicates the total income is not distributed evenly among individuals in the same way an uncompressedtext file does not have a similar number of byte locations assigned to the available unique byte characters.

Notation	Information theory	Theil index TT
${\displaystyle N}$	number of unique characters	number of individuals
${\displaystyle i}$	a particular character	a particular individual
${\displaystyle x_i}$	count ofith character	income ofith individual
${\displaystyle N\overline{x}}$	total characters in document	total income in population
${\displaystyle T_T}$	unused information space	unused potential in price mechanism[original research?]
	data compression	progressive tax[original research?]

According to theWorld Bank,

"The best-known entropy measures are Theil’s T (${\displaystyle T_T}$) and Theil’s L (${\displaystyle T_L}$), both of which allow one to decompose inequality into the part that is due to inequality within areas (e.g. urban, rural) and the part that is due to differences between areas (e.g. the rural-urban income gap). Typically at least three-quarters of inequality in a country is due to within-group inequality, and the remaining quarter to between-group differences."^[7]

If the population is divided into${\displaystyle m}$ subgroups and

• ${\displaystyle s_i}$ is the income share of group${\displaystyle i}$,
• ${\displaystyle N}$ is the total population and${\displaystyle N_i}$ is the population of group${\displaystyle i}$,
• ${\displaystyle T_i}$ is the Theil index for that subgroup,
• ${\displaystyle {\overline{x}}_i}$ is the average income in group${\displaystyle i}$, and
• ${\displaystyle \mu }$ is the average income of the population,

then Theil's T index is

${\displaystyle T_T=\sum _{i=1}^m{s}_i{T}_i+\sum _{i=1}^m{s}_i\ln \frac{{\overline{x}}_i}{\mu }}$ for${\displaystyle s_i=\frac{N_i}{N}\frac{{\overline{x}}_i}{\mu }}$

For example, inequality within the United States is the average inequality within each state, weighted by state income, plus the inequality between states.

Note: This image is not the Theil Index in each area of the United States, but of contributions to the Theil Index for the U.S. by each area. The Theil Index is always positive, although individual contributions to the Theil Index may be negative or positive.

The decomposition of the Theil index which identifies the share attributable to the between-region component becomes a helpful tool for the positive analysis of regional inequality as it suggests the relative importance of spatial dimension of inequality.^[8]

Both Theil'sT and Theil'sL are decomposable. The difference between them is based on the part of the outcomes distribution that each is used for. Indexes of inequality in the generalized entropy (GE) family are more sensitive to differences in income shares among the poor or among the rich depending on a parameter that defines the GE index. The smaller the parameter value for GE, the more sensitive it is to differences at the bottom of the distribution.^[9]

GE(0) = Theil'sL and is more sensitive to differences at the lower end of the distribution. It is also referred to as themean log deviation measure.

GE(1) = Theil'sT and is more sensitive to differences at the top of the distribution.

The decomposability is a property of the Theil index which the more popularGini coefficient does not offer. The Gini coefficient is more intuitive to many people since it is based on theLorenz curve. However, it is not easily decomposable like the Theil.

In addition to multitude of economic applications, the Theil index has been applied to assess performance ofirrigation systems^[10] and distribution ofsoftware metrics.^[11]

• Software:
  • Free Online Calculator computes the Gini Coefficient, plots the Lorenz curve, and computes many other measures of concentration for any dataset
  • Free Calculator:Online anddownloadable scripts (Python andLua) for Atkinson, Gini, and Hoover inequalities
  • Users of theR data analysis software can install the "ineq" package which allows for computation of a variety of inequality indices including Gini, Atkinson, Theil.
  • AMATLAB Inequality PackageArchived 2008-10-04 at theWayback Machine, including code for computing Gini, Atkinson, Theil indexes and for plotting the Lorenz Curve. Many examples are available.

• Eponyms in economics
• Information theory
• Income inequality metrics
• Welfare economics

• Articles with short description
• Short description matches Wikidata
• Use dmy dates from June 2024
• All articles that may contain original research
• Articles that may contain original research from June 2020
• Webarchive template wayback links