Empirical Bayes methods are procedures forstatistical inference in which theprior probability distribution is estimated from the data. This approach stands in contrast to standardBayesian methods, for which the prior distribution is fixed before any data are observed. Despite this difference in perspective, empirical Bayes may be viewed as an approximation to a fully Bayesian treatment of ahierarchical model wherein the parameters at the highest level of the hierarchy are set to their most likely values, instead of being integrated out.^[1]

Empirical Bayes methods can be seen as an approximation to a fully Bayesian treatment of ahierarchical Bayes model.

In, for example, a two-stage hierarchical Bayes model, observed data${\displaystyle y=\{y_1,y_2,\dots ,y_n\}}$ are assumed to be generated from an unobserved set of parameters${\displaystyle \theta =\{{\theta }_1,{\theta }_2,\dots ,{\theta }_n\}}$ according to a probability distribution${\displaystyle p(y\mid \theta )}$. In turn, the parameters${\displaystyle \theta }$ can be considered samples drawn from a population characterised byhyperparameters${\displaystyle \eta }$ according to a probability distribution${\displaystyle p(\theta \mid \eta )}$. In the hierarchical Bayes model, though not in the empirical Bayes approximation, the hyperparameters${\displaystyle \eta }$ are considered to be drawn from an unparameterized distribution${\displaystyle p(\eta )}$.

Information about a particular quantity of interest${\displaystyle {\theta }_i}$ therefore comes not only from the properties of those data${\displaystyle y}$ that directly depend on it, but also from the properties of the population of parameters${\displaystyle \theta }$ as a whole, inferred from the data as a whole, summarised by the hyperparameters${\displaystyle \eta }$.

UsingBayes' theorem,

${\displaystyle p(\theta \mid y)=\frac{p(y\mid \theta )p(\theta )}{p(y)}=\frac{p(y\mid \theta )}{p(y)}\int p(\theta \mid \eta )p(\eta )d\eta .}$

In general, this integral will not be tractableanalytically orsymbolically and must be evaluated bynumerical methods. Stochastic (random) or deterministic approximations may be used. Example stochastic methods areMarkov Chain Monte Carlo andMonte Carlo sampling. Deterministic approximations are discussed inquadrature.

Alternatively, the expression can be written as

${\displaystyle p(\theta \mid y)=\int p(\theta \mid \eta ,y)p(\eta \mid y)d\eta =\int \frac{p(y\mid \theta )p(\theta \mid \eta )}{p(y\mid \eta )}p(\eta \mid y)d\eta ,}$

and the final factor in the integral can in turn be expressed as

${\displaystyle p(\eta \mid y)=\int p(\eta \mid \theta )p(\theta \mid y)d\theta .}$

These suggest an iterative scheme, qualitatively similar in structure to aGibbs sampler, to evolve successively improved approximations to${\displaystyle p(\theta \mid y)}$ and${\displaystyle p(\eta \mid y)}$. First, calculate an initial approximation to${\displaystyle p(\theta \mid y)}$ ignoring the${\displaystyle \eta }$ dependence completely; then calculate an approximation to${\displaystyle p(\eta \mid y)}$ based upon the initial approximate distribution of${\displaystyle p(\theta \mid y)}$; then use this${\displaystyle p(\eta \mid y)}$ to update the approximation for${\displaystyle p(\theta \mid y)}$; then update${\displaystyle p(\eta \mid y)}$; and so on.

When the true distribution${\displaystyle p(\eta \mid y)}$ is sharply peaked, the integral determining${\displaystyle p(\theta \mid y)}$ may be not much changed by replacing the probability distribution over${\displaystyle \eta }$ with a point estimate${\displaystyle {\eta }^{\ast }}$ representing the distribution's peak (or, alternatively, its mean),

${\displaystyle p(\theta \mid y)\simeq \frac{p(y\mid \theta )p(\theta \mid {\eta }^{\ast })}{p(y\mid {\eta }^{\ast })}.}$

With this approximation, the above iterative scheme becomes theEM algorithm.

The term "Empirical Bayes" can cover a wide variety of methods, but most can be regarded as an early truncation of either the above scheme or something quite like it. Point estimates, rather than the whole distribution, are typically used for the parameter(s)${\displaystyle \eta }$. The estimates for${\displaystyle {\eta }^{\ast }}$ are typically made from the first approximation to${\displaystyle p(\theta \mid y)}$ without subsequent refinement. These estimates for${\displaystyle {\eta }^{\ast }}$ are usually made without considering an appropriate prior distribution for${\displaystyle \eta }$.

Robbins^[2] considered a case of sampling from amixed distribution, where probability for each${\displaystyle y_i}$ (conditional on${\displaystyle {\theta }_i}$) is specified by aPoisson distribution,

${\displaystyle p(y_i\mid {\theta }_i)=\frac{{{\theta }_i}^{y_i}{e}^{-{\theta }_i}}{y_i!}}$

while the prior onθ is unspecified except that it is alsoi.i.d. from an unknown distribution, withcumulative distribution function${\displaystyle G(\theta )}$. Compound sampling arises in a variety of statistical estimation problems, such as accident rates and clinical trials.^{[citation needed]} We simply seek a point prediction of${\displaystyle {\theta }_i}$ given all the observed data. Because the prior is unspecified, we seek to do this without knowledge ofG.^[3]

Undersquared error loss (SEL), theconditional expectation E(θ_i | Y_i = y_i) is a reasonable quantity to use for prediction. For the Poisson compound sampling model, this quantity is

${\displaystyle \mathrm{E}({\theta }_i\mid y_i)=\frac{\int ({\theta }^{y_i+1}{e}^{-\theta }/y_i!)dG(\theta )}{\int ({\theta }^{y_i}{e}^{-\theta }/y_i!)dG(\theta )}.}$

This can be simplified by multiplying both the numerator and denominator by${\displaystyle (y_i+1)}$, yielding

${\displaystyle \mathrm{E}({\theta }_i\mid y_i)=\frac{(y_i+1)p_G(y_i+1)}{p_G(y_i)},}$

wherep_G is the marginal probability mass function obtained by integrating outθ overG.

To take advantage of this, Robbins^[2] suggested estimating the marginals with their empirical frequencies (${\displaystyle \mathrm{\#}\{Y_j\}}$), yielding the fully non-parametric estimate as:

${\displaystyle \mathrm{E}({\theta }_i\mid y_i)\approx (y_i+1)\frac{\mathrm{\#}\{Y_j=y_i+1\}}{\mathrm{\#}\{Y_j=y_i\}},}$

where${\displaystyle \mathrm{\#}}$ denotes "number of". (See alsoGood–Turing frequency estimation.)

Example – Accident rates

Suppose each customer of an insurance company has an "accident rate" Θ and is insured against accidents; the probability distribution of Θ is the underlying distribution, and is unknown. The number of accidents suffered by each customer in a specified time period has aPoisson distribution with expected value equal to the particular customer's accident rate. The actual number of accidents experienced by a customer is the observable quantity. A crude way to estimate the underlying probability distribution of the accident rate Θ is to estimate the proportion of members of the whole population suffering 0, 1, 2, 3, ... accidents during the specified time period as the corresponding proportion in the observed random sample. Having done so, it is then desired to predict the accident rate of each customer in the sample. As above, one may use theconditionalexpected value of the accident rate Θ given the observed number of accidents during the baseline period. Thus, if a customer suffers six accidents during the baseline period, that customer's estimated accident rate is 7 × [the proportion of the sample who suffered 7 accidents] / [the proportion of the sample who suffered 6 accidents]. Note that if the proportion of people sufferingk accidents is a decreasing function ofk, the customer's predicted accident rate will often be lower than their observed number of accidents.

Thisshrinkage effect is typical of empirical Bayes analyses.

Suppose${\displaystyle X,Y}$ are random variables, such that${\displaystyle Y}$ is observed, but${\displaystyle X}$ is hidden. The problem is to find the expectation of${\displaystyle X}$, conditional on${\displaystyle Y}$. Suppose further that${\displaystyle Y|X\sim \mathcal{N}(X,\mathrm{\Sigma })}$, that is,${\displaystyle Y=X+Z}$, where${\displaystyle Z}$ is amultivariate gaussian with variance${\displaystyle \mathrm{\Sigma }}$.

Then, we have the formula$${\displaystyle \mathrm{\Sigma }{\mathrm{\nabla }}_y\rho (y|x)=\rho (y|x)(x-y)}$$by direct calculation with the probability density function of multivariate gaussians. Integrating over${\displaystyle \rho (x)dx}$, we obtain$${\displaystyle \mathrm{\Sigma }{\mathrm{\nabla }}_y\rho (y)=(\mathbb{E}[x|y]-y)\rho (y)⟹\mathbb{E}[x|y]=y+\mathrm{\Sigma }{\mathrm{\nabla }}_y\ln \rho (y)}$$ In particular, this means that one can perform Bayesian estimation of${\displaystyle X}$ without access to either the prior density of${\displaystyle X}$ or the posterior density of${\displaystyle Y}$. The only requirement is to have access to thescore function of${\displaystyle Y}$. This has applications inscore-based generative modeling.^[4]

If the likelihood and its prior take on simple parametric forms (such as 1- or 2-dimensional likelihood functions with simpleconjugate priors), then the empirical Bayes problem is only to estimate the marginal${\displaystyle m(y\mid \eta )}$ and the hyperparameters${\displaystyle \eta }$ using the complete set of empirical measurements. For example, one common approach, called parametric empirical Bayes point estimation, is to approximate the marginal using themaximum likelihood estimate (MLE), or amoments expansion, which allows one to express the hyperparameters${\displaystyle \eta }$ in terms of the empirical mean and variance. This simplified marginal allows one to plug in the empirical averages into a point estimate for the prior${\displaystyle \theta }$. The resulting equation for the prior${\displaystyle \theta }$ is greatly simplified, as shown below.

There are several common parametric empirical Bayes models, including thePoisson–gamma model (below), theBeta-binomial model, theGaussian–Gaussian model, theDirichlet-multinomial model, as well specific models forBayesian linear regression (see below) andBayesian multivariate linear regression. More advanced approaches includehierarchical Bayes models andBayesian mixture models.

For an example of empirical Bayes estimation using a Gaussian-Gaussian model, seeEmpirical Bayes estimators.

For example, in the example above, let the likelihood be aPoisson distribution, and let the prior now be specified by theconjugate prior, which is agamma distribution (${\displaystyle G(\alpha ,\beta )}$) (where${\displaystyle \eta =(\alpha ,\beta )}$):

${\displaystyle \rho (\theta \mid \alpha ,\beta )d\theta =\frac{(\theta /\beta {)}^{\alpha -1}{e}^{-\theta /\beta }}{\mathrm{\Gamma }(\alpha )}(d\theta /\beta )\text{ for }\theta >0,\alpha >0,\beta >0.}$

It is straightforward to show theposterior is also a gamma distribution. Write

${\displaystyle \rho (\theta \mid y)\propto \rho (y\mid \theta )\rho (\theta \mid \alpha ,\beta ),}$

where the marginal distribution has been omitted since it does not depend explicitly on${\displaystyle \theta }$.Expanding terms which do depend on${\displaystyle \theta }$ gives the posterior as:

${\displaystyle \rho (\theta \mid y)\propto ({\theta }^y{e}^{-\theta })({\theta }^{\alpha -1}{e}^{-\theta /\beta })={\theta }^{y+\alpha -1}{e}^{-\theta (1+1/\beta )}.}$

So the posterior density is also agamma distribution${\displaystyle G({\alpha }^{\prime },{\beta }^{\prime })}$, where${\displaystyle {\alpha }^{\prime }=y+\alpha }$, and${\displaystyle {\beta }^{\prime }=(1+1/\beta {)}^{-1}}$. Also notice that the marginal is simply the integral of the posterior over all${\displaystyle \mathrm{\Theta }}$, which turns out to be anegative binomial distribution.

To apply empirical Bayes, we will approximate the marginal using themaximum likelihood estimate (MLE). But since the posterior is a gamma distribution, the MLE of the marginal turns out to be just the mean of the posterior, which is the point estimate${\displaystyle \mathrm{E}(\theta \mid y)}$ we need. Recalling that the mean${\displaystyle \mu }$ of a gamma distribution${\displaystyle G({\alpha }^{\prime },{\beta }^{\prime })}$ is simply${\displaystyle {\alpha }^{\prime }{\beta }^{\prime }}$, we have

${\displaystyle \mathrm{E}(\theta \mid y)={\alpha }^{\prime }{\beta }^{\prime }=\frac{\overline{y}+\alpha }{1+1/\beta }=\frac{\beta }{1+\beta }\overline{y}+\frac{1}{1+\beta }(\alpha \beta ).}$

To obtain the values of${\displaystyle \alpha }$ and${\displaystyle \beta }$, empirical Bayes prescribes estimating mean${\displaystyle \alpha \beta }$ and variance${\displaystyle \alpha {\beta }^2}$ using the complete set of empirical data.

The resulting point estimate${\displaystyle \mathrm{E}(\theta \mid y)}$ is therefore like a weighted average of the sample mean${\displaystyle \overline{y}}$ and the prior mean${\displaystyle \mu =\alpha \beta }$. This turns out to be a general feature of empirical Bayes; the point estimates for the prior (i.e. mean) will look like a weighted averages of the sample estimate and the prior estimate (likewise for estimates of the variance).

• Bayes estimator
• Bayesian network
• Hyperparameter
• Hyperprior
• Best linear unbiased prediction
• Robbins lemma
• Spike-and-slab variable selection

This article includes a list ofgeneral references, butit lacks sufficient correspondinginline citations. Please help toimprove this article byintroducing more precise citations.(February 2012) (Learn how and when to remove this message)

• Peter E. Rossi; Greg M. Allenby; Rob McCulloch (14 May 2012).Bayesian Statistics and Marketing. John Wiley & Sons.ISBN 978-0-470-86368-8.
• Casella, George (May 1985)."An Introduction to Empirical Bayes Data Analysis"(PDF).American Statistician.39 (2):83–87.doi:10.2307/2682801.hdl:1813/32886.JSTOR 2682801.MR 0789118.
• Nikulin, Mikhail (1987)."Bernstein's regularity conditions in a problem of empirical Bayesian approach".Journal of Soviet Mathematics.36 (5):596–600.doi:10.1007/BF01093293.S2CID 122405908.

• Use of empirical Bayes Method in estimating road safety (North America)
• Empirical Bayes methods for missing data analysis
• Using the Beta-Binomial distribution to assess performance of a biometric identification device
• A Hierarchical Naive Bayes Classifiers (for continuous anddiscrete variables).

• Nonparametric Bayesian statistics

• CS1 errors: ISBN date
• Articles with short description
• Short description is different from Wikidata
• All articles with unsourced statements
• Articles with unsourced statements from February 2012
• Articles lacking in-text citations from February 2012
• All articles lacking in-text citations