This article includes a list ofgeneral references, butit lacks sufficient correspondinginline citations. Please help toimprove this article byintroducing more precise citations.(November 2009) (Learn how and when to remove this message)

Part of a series on
Bayesian statistics
Posterior =Likelihood ×Prior ÷Evidence
Background
Bayesian inference Bayesian probability Bayes' theorem Bernstein–von Mises theorem Coherence Cox's theorem Cromwell's rule Likelihood principle Principle of indifference Principle of maximum entropy
Model building
Conjugate prior Linear regression Empirical Bayes Hierarchical model
Posterior approximation
Markov chain Monte Carlo Laplace's approximation Integrated nested Laplace approximations Variational inference Approximate Bayesian computation
Estimators
Bayesian estimator Credible interval Maximum a posteriori estimation
Evidence approximation
Evidence lower bound Nested sampling
Model evaluation
Bayes factor (Schwarz criterion) Model averaging Posterior predictive
Mathematics portal
v t e

Inestimation theory anddecision theory, aBayes estimator or aBayes action is anestimator ordecision rule that minimizes theposteriorexpected value of aloss function (i.e., theposterior expected loss). Equivalently, it maximizes the posterior expectation of autility function. An alternative way of formulating an estimator withinBayesian statistics ismaximum a posteriori estimation.

Suppose an unknown parameter${\displaystyle \theta }$ is known to have aprior distribution${\displaystyle \pi }$. Let${\displaystyle \hat{\theta }=\hat{\theta }(x)}$ be an estimator of${\displaystyle \theta }$ (based on some measurementsx), and let${\displaystyle L(\theta ,\hat{\theta })}$ be aloss function, such as squared error. TheBayes risk of${\displaystyle \hat{\theta }}$ is defined as${\displaystyle E_{\pi }(L(\theta ,\hat{\theta }))}$, where theexpectation is taken over the probability distribution of${\displaystyle \theta }$: this defines the risk function as a function of${\displaystyle \hat{\theta }}$. An estimator${\displaystyle \hat{\theta }}$ is said to be aBayes estimator if it minimizes the Bayes risk among all estimators. Equivalently, the estimator which minimizes the posterior expected loss${\displaystyle E(L(\theta ,\hat{\theta })|x)}$for each${\displaystyle x}$ also minimizes the Bayes risk and therefore is a Bayes estimator.^[1]

If the prior isimproper then an estimator which minimizes the posterior expected lossfor each${\displaystyle x}$ is called ageneralized Bayes estimator.^[2]

The most common risk function used for Bayesian estimation is themean square error (MSE), also calledsquared error risk. The MSE is defined by

${\displaystyle \mathrm{M}\mathrm{S}\mathrm{E}=E\left[(\hat{\theta }(x)-\theta {)}^2\right],}$

where the expectation is taken over the joint distribution of${\displaystyle \theta }$ and${\displaystyle x}$.

Using the MSE as risk, the Bayes estimate of the unknown parameter is simply the mean of theposterior distribution,^[3]

${\displaystyle \hat{\theta }(x)=E[\theta |x]=\int \theta p(\theta |x)d\theta .}$

This is known as theminimum mean square error (MMSE) estimator.

If there is no inherent reason to prefer one prior probability distribution over another, aconjugate prior is sometimes chosen for simplicity. A conjugate prior is defined as a prior distribution belonging to someparametric family, for which the resulting posterior distribution also belongs to the same family. This is an important property, since the Bayes estimator, as well as its statistical properties (variance, confidence interval, etc.), can all be derived from the posterior distribution.

Conjugate priors are especially useful for sequential estimation, where the posterior of the current measurement is used as the prior in the next measurement. In sequential estimation, unless a conjugate prior is used, the posterior distribution typically becomes more complex with each added measurement, and the Bayes estimator cannot usually be calculated without resorting to numerical methods.

Following are some examples of conjugate priors.

• If${\displaystyle x|\theta }$ isNormal,${\displaystyle x|\theta \sim N(\theta ,{\sigma }^2)}$, and the prior is normal,${\displaystyle \theta \sim N(\mu ,{\tau }^2)}$, then the posterior is also Normal and the Bayes estimator under MSE is given by

${\displaystyle \hat{\theta }(x)=\frac{{\sigma }^2}{{\sigma }^2+{\tau }^2}\mu +\frac{{\tau }^2}{{\sigma }^2+{\tau }^2}x.}$

• If${\displaystyle x_1,...,x_n}$ areiidPoisson random variables${\displaystyle x_i|\theta \sim P(\theta )}$, and if the prior isGamma distributed${\displaystyle \theta \sim G(a,b)}$, then the posterior is also Gamma distributed, and the Bayes estimator under MSE is given by

${\displaystyle \hat{\theta }(X)=\frac{n\overline{X}+a}{n+b}.}$

• If${\displaystyle x_1,...,x_n}$ are iiduniformly distributed${\displaystyle x_i|\theta \sim U(0,\theta )}$, and if the prior isPareto distributed${\displaystyle \theta \sim Pa({\theta }_0,a)}$, then the posterior is also Pareto distributed, and the Bayes estimator under MSE is given by

${\displaystyle \hat{\theta }(X)=\frac{(a+n)max({\theta }_0,x_1,...,x_n)}{a+n-1}.}$

Risk functions are chosen depending on how one measures the distance between the estimate and the unknown parameter. The MSE is the most common risk function in use, primarily due to its simplicity. However, alternative risk functions are also occasionally used. The following are several examples of such alternatives. We denote the posterior generalized distribution function by${\displaystyle F}$.

A "linear" loss function, with${\displaystyle a>0}$, which yields the posterior median as the Bayes' estimate:

${\displaystyle L(\theta ,\hat{\theta })=a|\theta -\hat{\theta }|}$

${\displaystyle F(\hat{\theta }(x)|X)=\frac{1}{2}.}$

Another "linear" loss function, which assigns different "weights"${\displaystyle a,b>0}$ to over or sub estimation. It yields aquantile from the posterior distribution, and is a generalization of the previous loss function:

${\displaystyle F(\hat{\theta }(x)|X)=\frac{a}{a+b}.}$

The following loss function is trickier: it yields either theposterior mode, or a point close to it depending on the curvature and properties of the posterior distribution. Small values of the parameter${\displaystyle K>0}$ are recommended, in order to use the mode as an approximation (${\displaystyle L>0}$):

One can also consider${\displaystyle L^P}$ risk for which the loss is given by

${\displaystyle L(\theta ,\hat{\theta })=|\theta -\hat{\theta }{|}^p,p>0.}$

While optimal${\displaystyle L^p}$ estimators can be difficult to characterize in closed-form, they do share many similar properties to those in${\displaystyle L^2}$ case.^[4]

Other loss functions can be conceived, although themean squared error is the most widely used and validated. Other loss functions are used in statistics, particularly inrobust statistics.

The prior distribution${\displaystyle p}$ has thus far been assumed to be a true probability distribution, in that

${\displaystyle \int p(\theta )d\theta =1.}$

However, occasionally this can be a restrictive requirement. For example, there is no distribution (covering the set,R, of all real numbers) for which every real number is equally likely. Yet, in some sense, such a "distribution" seems like a natural choice for anon-informative prior, i.e., a prior distribution which does not imply a preference for any particular value of the unknown parameter. One can still define a function${\displaystyle p(\theta )=1}$, but this would not be a proper probability distribution since it has infinite mass,

${\displaystyle \int p(\theta )d\theta =\mathrm{\infty }.}$

Suchmeasures${\displaystyle p(\theta )}$, which are not probability distributions, are referred to asimproper priors.

The use of an improper prior means that the Bayes risk is undefined (since the prior is not a probability distribution and we cannot take an expectation under it). As a consequence, it is no longer meaningful to speak of a Bayes estimator that minimizes the Bayes risk. Nevertheless, in many cases, one can define the posterior distribution

${\displaystyle p(\theta |x)=\frac{p(x|\theta )p(\theta )}{\int p(x|\theta )p(\theta )d\theta }.}$

This is a definition, and not an application ofBayes' theorem, since Bayes' theorem can only be applied when all distributions are proper. However, it is not uncommon for the resulting "posterior" to be a valid probability distribution. In this case, the posterior expected loss

${\displaystyle \int L(\theta ,a)p(\theta |x)d\theta }$

is typically well-defined and finite. Recall that, for a proper prior, the Bayes estimator minimizes the posterior expected loss. When the prior is improper, an estimator which minimizes the posterior expected loss is referred to as ageneralized Bayes estimator.^[2]

A typical example is estimation of alocation parameter with a loss function of the type${\displaystyle L(a-\theta )}$. Here${\displaystyle \theta }$ is a location parameter, i.e.,${\displaystyle p(x|\theta )=f(x-\theta )}$.

It is common to use the improper prior${\displaystyle p(\theta )=1}$ in this case, especially when no other more subjective information is available. This yields

${\displaystyle p(\theta |x)=\frac{p(x|\theta )p(\theta )}{p(x)}=\frac{f(x-\theta )}{p(x)}}$

so the posterior expected loss

${\displaystyle E[L(a-\theta )|x]=\int L(a-\theta )p(\theta |x)d\theta =\frac{1}{p(x)}\int L(a-\theta )f(x-\theta )d\theta .}$

The generalized Bayes estimator is the value${\displaystyle a(x)}$ that minimizes this expression for a given${\displaystyle x}$. This is equivalent to minimizing

${\displaystyle \int L(a-\theta )f(x-\theta )d\theta }$ for a given${\displaystyle x.}$        (1)

In this case it can be shown that the generalized Bayes estimator has the form${\displaystyle x+a_0}$, for some constant${\displaystyle a_0}$. To see this, let${\displaystyle a_0}$ be the value minimizing (1) when${\displaystyle x=0}$. Then, given a different value${\displaystyle x_1}$, we must minimize

${\displaystyle \int L(a-\theta )f(x_1-\theta )d\theta =\int L(a-x_1-{\theta }^{\prime })f(-{\theta }^{\prime })d{\theta }^{\prime }.}$        (2)

This is identical to (1), except that${\displaystyle a}$ has been replaced by${\displaystyle a-x_1}$. Thus, the expression minimizing is given by${\displaystyle a-x_1=a_0}$, so that the optimal estimator has the form

${\displaystyle a(x)=a_0+x.}$

A Bayes estimator derived through theempirical Bayes method is called anempirical Bayes estimator. Empirical Bayes methods enable the use of auxiliary empirical data, from observations of related parameters, in the development of a Bayes estimator. This is done under the assumption that the estimated parameters are obtained from a common prior. For example, if independent observations of different parameters are performed, then the estimation performance of a particular parameter can sometimes be improved by using data from other observations.

There are bothparametric andnon-parametric approaches to empirical Bayes estimation.^[5]

The following is a simple example of parametric empirical Bayes estimation. Given past observations${\displaystyle x_1,\dots ,x_n}$ having conditional distribution${\displaystyle f(x_i|{\theta }_i)}$, one is interested in estimating${\displaystyle {\theta }_{n+1}}$ based on${\displaystyle x_{n+1}}$. Assume that the${\displaystyle {\theta }_i}$'s have a common prior${\displaystyle \pi }$ which depends on unknown parameters. For example, suppose that${\displaystyle \pi }$ is normal with unknown mean${\displaystyle {\mu }_{\pi }}$ and variance${\displaystyle {\sigma }_{\pi }.}$ We can then use the past observations to determine the mean and variance of${\displaystyle \pi }$ in the following way.

First, we estimate the mean${\displaystyle {\mu }_m}$ and variance${\displaystyle {\sigma }_m}$ of the marginal distribution of${\displaystyle x_1,\dots ,x_n}$ using themaximum likelihood approach:

${\displaystyle {\hat{\mu }}_m=\frac{1}{n}\sum x_i,}$

${\displaystyle {\hat{\sigma }}_m^2=\frac{1}{n}\sum (x_i-{\hat{\mu }}_m{)}^2.}$

Next, we use thelaw of total expectation to compute${\displaystyle {\mu }_m}$ and thelaw of total variance to compute${\displaystyle {\sigma }_m^2}$ such that

${\displaystyle {\mu }_m=E_{\pi }[{\mu }_f(\theta )],}$

${\displaystyle {\sigma }_m^2=E_{\pi }[{\sigma }_f^2(\theta )]+E_{\pi }[({\mu }_f(\theta )-{\mu }_m{)}^2],}$

where${\displaystyle {\mu }_f(\theta )}$ and${\displaystyle {\sigma }_f(\theta )}$ are the moments of the conditional distribution${\displaystyle f(x_i|{\theta }_i)}$, which are assumed to be known. In particular, suppose that${\displaystyle {\mu }_f(\theta )=\theta }$ and that${\displaystyle {\sigma }_f^2(\theta )=K}$; we then have

${\displaystyle {\mu }_{\pi }={\mu }_m,}$

${\displaystyle {\sigma }_{\pi }^2={\sigma }_m^2-{\sigma }_f^2={\sigma }_m^2-K.}$

Finally, we obtain the estimated moments of the prior,

${\displaystyle {\hat{\mu }}_{\pi }={\hat{\mu }}_m,}$

${\displaystyle {\hat{\sigma }}_{\pi }^2={\hat{\sigma }}_m^2-K.}$

For example, if${\displaystyle x_i|{\theta }_i\sim N({\theta }_i,1)}$, and if we assume a normal prior (which is a conjugate prior in this case), we conclude that${\displaystyle {\theta }_{n+1}\sim N({\hat{\mu }}_{\pi },{\hat{\sigma }}_{\pi }^2)}$, from which the Bayes estimator of${\displaystyle {\theta }_{n+1}}$ based on${\displaystyle x_{n+1}}$ can be calculated.

Bayes rules having finite Bayes risk are typicallyadmissible. The following are some specific examples of admissibility theorems.

• If a Bayes rule is unique then it is admissible.^[6] For example, as stated above, under mean squared error (MSE) the Bayes rule is unique and therefore admissible.
• If θ belongs to adiscrete set, then all Bayes rules are admissible.
• If θ belongs to a continuous (non-discrete) set, and if the risk function R(θ,δ) is continuous in θ for every δ, then all Bayes rules are admissible.

By contrast, generalized Bayes rules often have undefined Bayes risk in the case of improper priors. These rules are often inadmissible and the verification of their admissibility can be difficult. For example, the generalized Bayes estimator of a location parameter θ based on Gaussian samples (described in the "Generalized Bayes estimator" section above) is inadmissible for${\displaystyle p>2}$; this is known asStein's phenomenon.

Let θ be an unknown random variable, and suppose that${\displaystyle x_1,x_2,\dots }$ areiid samples with density${\displaystyle f(x_i|\theta )}$. Let${\displaystyle {\delta }_n={\delta }_n(x_1,\dots ,x_n)}$ be a sequence of Bayes estimators of θ based on an increasing number of measurements. We are interested in analyzing the asymptotic performance of this sequence of estimators, i.e., the performance of${\displaystyle {\delta }_n}$ for largen.

To this end, it is customary to regard θ as a deterministic parameter whose true value is${\displaystyle {\theta }_0}$. Under specific conditions,^[7] for large samples (large values ofn), the posterior density of θ is approximately normal. In other words, for largen, the effect of the prior probability on the posterior is negligible. Moreover, if δ is the Bayes estimator under MSE risk, then it isasymptotically unbiased and itconverges in distribution to thenormal distribution:

${\displaystyle \sqrt{n}({\delta }_n-{\theta }_0)\to N\left(0,\frac{1}{I({\theta }_0)}\right),}$

whereI(θ₀) is theFisher information of θ₀.It follows that the Bayes estimator δ_n under MSE isasymptotically efficient.

Another estimator which is asymptotically normal and efficient is themaximum likelihood estimator (MLE). The relations between the maximum likelihood and Bayes estimators can be shown in the following simple example.

Consider the estimator of θ based on binomial samplex~b(θ,n) where θ denotes the probability for success. Assuming θ is distributed according to the conjugate prior, which in this case is theBeta distribution B(a,b), the posterior distribution is known to be B(a+x,b+n-x). Thus, the Bayes estimator under MSE is

${\displaystyle {\delta }_n(x)=E[\theta |x]=\frac{a+x}{a+b+n}.}$

The MLE in this case is x/n and so we get,

${\displaystyle {\delta }_n(x)=\frac{a+b}{a+b+n}E[\theta ]+\frac{n}{a+b+n}{\delta }_{MLE}.}$

The last equation implies that, forn → ∞, the Bayes estimator (in the described problem) is close to the MLE.

On the other hand, whenn is small, the prior information is still relevant to the decision problem and affects the estimate. To see the relative weight of the prior information, assume thata=b; in this case each measurement brings in 1 new bit of information; the formula above shows that the prior information has the same weight asa+b bits of the new information. In applications, one often knows very little about fine details of the prior distribution; in particular, there is no reason to assume that it coincides with B(a,b) exactly. In such a case, one possible interpretation of this calculation is: "there is a non-pathological prior distribution with the mean value 0.5 and the standard deviationd which gives the weight of prior information equal to 1/(4d²)-1 bits of new information."

Another example of the same phenomena is the case when the prior estimate and a measurement are normally distributed. If the prior is centered atB with deviation Σ, and the measurement is centered atb with deviation σ,then the posterior is centered at${\displaystyle \frac{\alpha }{\alpha +\beta }B+\frac{\beta }{\alpha +\beta }b}$, with weights in this weighted average being α=σ², β=Σ². Moreover, the squared posterior deviation is Σ²+σ². In other words, the prior is combined with the measurement inexactly the same way as if it were an extra measurement to take into account.

For example, if Σ=σ/2, then the deviation of 4 measurements combined matches the deviation of the prior (assuming that errors of measurements are independent). And the weights α,β in the formula for posterior match this: the weight of the prior is 4 times the weight of the measurement. Combining this prior withn measurements with averagev results in the posterior centered at${\displaystyle \frac{4}{4+n}V+\frac{n}{4+n}v}$; in particular, the prior plays the same role as 4 measurements made in advance. In general, the prior has the weight of (σ/Σ)² measurements.

Compare to the example of binomial distribution: there the prior has the weight of (σ/Σ)²−1 measurements. One can see that the exact weight does depend on the details of the distribution, but when σ≫Σ, the difference becomes small.

TheInternet Movie Database uses a formula for calculating and comparing the ratings of films by its users, including theirTop Rated 250 Titles which is claimed to give "a true Bayesian estimate".^[8] The following Bayesian formula was initially used to calculate a weighted average score for the Top 250, though the formula has since changed:

${\displaystyle W=\frac{Rv+Cm}{v+m}}$

where:

${\displaystyle W}$ = weighted rating

${\displaystyle R}$ = average rating for the movie as a number from 1 to 10 (mean) = (Rating)

${\displaystyle v}$ = number of votes/ratings for the movie = (votes)

${\displaystyle m}$ = weight given to the prior estimate (in this case, the number of votes IMDB deemed necessary for average rating to approach statistical validity)

${\displaystyle C}$ = the mean vote across the whole pool (currently 7.0)

Note thatW is just theweighted arithmetic mean ofR andC with weight vector(v, m). As the number of ratings surpassesm, the confidence of the average rating surpasses the confidence of the mean vote for all films (C), and the weighted bayesian rating (W) approaches a straight average (R). The closerv (the number of ratings for the film) is to zero, the closerW is toC, where W is the weighted rating and C is the average rating of all films. So, in simpler terms, the fewer ratings/votes cast for a film, the more that film's Weighted Rating will skew towards the average across all films, while films with many ratings/votes will have a rating approaching its pure arithmetic average rating.

IMDb's approach ensures that a film with only a few ratings, all at 10, would not rank above "the Godfather", for example, with a 9.2 average from over 500,000 ratings.

• Recursive Bayesian estimation
• Generalized expected utility

• Berger, James O. (1985).Statistical decision theory and Bayesian Analysis (2nd ed.). New York: Springer-Verlag.ISBN 0-387-96098-8.MR 0804611.
• Lehmann, E. L.; Casella, G. (1998).Theory of Point Estimation (2nd ed.). Springer.ISBN 0-387-98502-6.
• Pilz, Jürgen (1991). "Bayesian estimation".Bayesian Estimation and Experimental Design in Linear Regression Models. Chichester: John Wiley & Sons. pp. 38–117.ISBN 0-471-91732-X.

• "Bayesian estimator",Encyclopedia of Mathematics,EMS Press, 2001 [1994]

• Estimator
• Bayesian estimation

• Articles with short description
• Short description is different from Wikidata
• Articles lacking in-text citations from November 2009
• All articles lacking in-text citations