Movatterモバイル変換

[0]ホーム
URL:

Jump to content
WikipediaThe Free Encyclopedia
Search

Maximum likelihood estimation

From Wikipedia, the free encyclopedia
(Redirected fromMaximum likelihood)
Method of estimating the parameters of a statistical model, given observations
This article is about the statistical techniques. For computer data storage, seepartial-response maximum-likelihood.

Instatistics,maximum likelihood estimation (MLE) is a method ofestimating theparameters of an assumedprobability distribution, given some observed data. This is achieved bymaximizing alikelihood function so that, under the assumedstatistical model, theobserved data is most probable. Thepoint in theparameter space that maximizes the likelihood function is called the maximum likelihood estimate.[1] The logic of maximum likelihood is both intuitive and flexible, and as such the method has become a dominant means ofstatistical inference.[2][3][4]

If the likelihood function isdifferentiable, thederivative test for finding maxima can be applied. In some cases, the first-order conditions of the likelihood function can be solved analytically; for instance, theordinary least squares estimator for alinear regression model maximizes the likelihood when the random errors are assumed to havenormal distributions with the same variance.[5]

From the perspective ofBayesian inference, MLE is generally equivalent tomaximum a posteriori (MAP) estimation with aprior distribution that isuniform in the region of interest. Infrequentist inference, MLE is a special case of anextremum estimator, with the objective function being the likelihood.

Principles

[edit]

We model a set of observations as a randomsample from an unknownjoint probability distribution which is expressed in terms of a set ofparameters. The goal of maximum likelihood estimation is to determine the parameters for which the observed data have the highest joint probability. We write the parameters governing the joint distribution as a vectorθ=[θ1,θ2,…,θk]T{\displaystyle \;\theta =\left[\theta _{1},\,\theta _{2},\,\ldots ,\,\theta _{k}\right]^{\mathsf {T}}\;} so that this distribution falls within aparametric family{f(⋅;θ)∣θ∈Θ},{\displaystyle \;\{f(\cdot \,;\theta )\mid \theta \in \Theta \}\;,} whereΘ{\displaystyle \,\Theta \,} is called theparameter space, a finite-dimensional subset ofEuclidean space. Evaluating the joint density at the observed data sampley=(y1,y2,…,yn){\displaystyle \;\mathbf {y} =(y_{1},y_{2},\ldots ,y_{n})\;} gives a real-valued function,Ln(θ)=Ln(θ;y)=fn(y;θ),{\displaystyle {\mathcal {L}}_{n}(\theta )={\mathcal {L}}_{n}(\theta ;\mathbf {y} )=f_{n}(\mathbf {y} ;\theta )\;,}which is called thelikelihood function. Forindependent and identically distributed random variables,fn(y;θ){\displaystyle f_{n}(\mathbf {y} ;\theta )} will be the product of univariatedensity functions:fn(y;θ)=∏k=1nfkunivar(yk;θ) .{\displaystyle f_{n}(\mathbf {y} ;\theta )=\prod _{k=1}^{n}\,f_{k}^{\mathsf {univar}}(y_{k};\theta )~.}

The goal of maximum likelihood estimation is to find the values of the model parameters that maximize the likelihood function over the parameter space,[6] that is:θ^=argmaxθ∈ΘLn(θ;y) .{\displaystyle {\hat {\theta }}={\underset {\theta \in \Theta }{\operatorname {arg\;max} }}\,{\mathcal {L}}_{n}(\theta \,;\mathbf {y} )~.}

Intuitively, this selects the parameter values that make the observed data most probable. The specific value θ^=θ^n(y)∈Θ {\displaystyle ~{\hat {\theta }}={\hat {\theta }}_{n}(\mathbf {y} )\in \Theta ~} that maximizes the likelihood functionLn{\displaystyle \,{\mathcal {L}}_{n}\,} is called the maximum likelihood estimate. Further, if the functionθ^n:Rn→Θ{\displaystyle \;{\hat {\theta }}_{n}:\mathbb {R} ^{n}\to \Theta \;} so defined ismeasurable, then it is called the maximum likelihoodestimator. It is generally a function defined over thesample space, i.e. taking a given sample as its argument. Asufficient but not necessary condition for its existence is for the likelihood function to becontinuous over a parameter spaceΘ{\displaystyle \,\Theta \,} that iscompact.[7] For anopenΘ{\displaystyle \,\Theta \,} the likelihood function may increase without ever reaching a supremum value.

In practice, it is often convenient to work with thenatural logarithm of the likelihood function, called thelog-likelihood:ℓ(θ;y)=ln⁡Ln(θ;y) .{\displaystyle \ell (\theta \,;\mathbf {y} )=\ln {\mathcal {L}}_{n}(\theta \,;\mathbf {y} )~.}Since the logarithm is amonotonic function, the maximum ofℓ(θ;y){\displaystyle \;\ell (\theta \,;\mathbf {y} )\;} occurs at the same value ofθ{\displaystyle \theta } as does the maximum ofLn .{\displaystyle \,{\mathcal {L}}_{n}~.}[8] Ifℓ(θ;y){\displaystyle \ell (\theta \,;\mathbf {y} )} isdifferentiable inΘ,{\displaystyle \,\Theta \,,}sufficient conditions for the occurrence of a maximum (or a minimum) are∂ℓ∂θ1=0,∂ℓ∂θ2=0,…,∂ℓ∂θk=0 ,{\displaystyle {\frac {\partial \ell }{\partial \theta _{1}}}=0,\quad {\frac {\partial \ell }{\partial \theta _{2}}}=0,\quad \ldots ,\quad {\frac {\partial \ell }{\partial \theta _{k}}}=0~,}known as the likelihood equations. For some models, these equations can be explicitly solved forθ^,{\displaystyle \,{\widehat {\theta \,}}\,,} but in general no closed-form solution to the maximization problem is known or available, and an MLE can only be found vianumerical optimization. Another problem is that in finite samples, there may exist multipleroots for the likelihood equations.[9] Whether the identified rootθ^{\displaystyle \,{\widehat {\theta \,}}\,} of the likelihood equations is indeed a (local) maximum depends on whether the matrix of second-order partial and cross-partial derivatives, the so-calledHessian matrix

H(θ^)=[∂2ℓ∂θ12|θ=θ^∂2ℓ∂θ1∂θ2|θ=θ^…∂2ℓ∂θ1∂θk|θ=θ^∂2ℓ∂θ2∂θ1|θ=θ^∂2ℓ∂θ22|θ=θ^…∂2ℓ∂θ2∂θk|θ=θ^⋮⋮⋱⋮∂2ℓ∂θk∂θ1|θ=θ^∂2ℓ∂θk∂θ2|θ=θ^…∂2ℓ∂θk2|θ=θ^] ,{\displaystyle \mathbf {H} \left({\widehat {\theta \,}}\right)={\begin{bmatrix}\left.{\frac {\partial ^{2}\ell }{\partial \theta _{1}^{2}}}\right|_{\theta ={\widehat {\theta \,}}}&\left.{\frac {\partial ^{2}\ell }{\partial \theta _{1}\,\partial \theta _{2}}}\right|_{\theta ={\widehat {\theta \,}}}&\dots &\left.{\frac {\partial ^{2}\ell }{\partial \theta _{1}\,\partial \theta _{k}}}\right|_{\theta ={\widehat {\theta \,}}}\\\left.{\frac {\partial ^{2}\ell }{\partial \theta _{2}\,\partial \theta _{1}}}\right|_{\theta ={\widehat {\theta \,}}}&\left.{\frac {\partial ^{2}\ell }{\partial \theta _{2}^{2}}}\right|_{\theta ={\widehat {\theta \,}}}&\dots &\left.{\frac {\partial ^{2}\ell }{\partial \theta _{2}\,\partial \theta _{k}}}\right|_{\theta ={\widehat {\theta \,}}}\\\vdots &\vdots &\ddots &\vdots \\\left.{\frac {\partial ^{2}\ell }{\partial \theta _{k}\,\partial \theta _{1}}}\right|_{\theta ={\widehat {\theta \,}}}&\left.{\frac {\partial ^{2}\ell }{\partial \theta _{k}\,\partial \theta _{2}}}\right|_{\theta ={\widehat {\theta \,}}}&\dots &\left.{\frac {\partial ^{2}\ell }{\partial \theta _{k}^{2}}}\right|_{\theta ={\widehat {\theta \,}}}\end{bmatrix}}~,}

isnegative semi-definite atθ^{\displaystyle {\widehat {\theta \,}}}, as this indicates localconcavity. Conveniently, most commonprobability distributions – in particular theexponential family – arelogarithmically concave.[10][11]

Restricted parameter space

[edit]
Not to be confused withrestricted maximum likelihood.

While the domain of the likelihood function—theparameter space—is generally a finite-dimensional subset ofEuclidean space, additionalrestrictions sometimes need to be incorporated into the estimation process. The parameter space can be expressed asΘ={θ:θ∈Rk,h(θ)=0} ,{\displaystyle \Theta =\left\{\theta :\theta \in \mathbb {R} ^{k},\;h(\theta )=0\right\}~,}

whereh(θ)=[h1(θ),h2(θ),…,hr(θ)]{\displaystyle \;h(\theta )=\left[h_{1}(\theta ),h_{2}(\theta ),\ldots ,h_{r}(\theta )\right]\;} is avector-valued function mappingRk{\displaystyle \,\mathbb {R} ^{k}\,} intoRr .{\displaystyle \;\mathbb {R} ^{r}~.} Estimating the true parameterθ{\displaystyle \theta } belonging toΘ{\displaystyle \Theta } then, as a practical matter, means to find the maximum of the likelihood function subject to theconstraint h(θ)=0 .{\displaystyle ~h(\theta )=0~.}

Theoretically, the most natural approach to thisconstrained optimization problem is the method of substitution, that is "filling out" the restrictionsh1,h2,…,hr{\displaystyle \;h_{1},h_{2},\ldots ,h_{r}\;} to a seth1,h2,…,hr,hr+1,…,hk{\displaystyle \;h_{1},h_{2},\ldots ,h_{r},h_{r+1},\ldots ,h_{k}\;} in such a way thath∗=[h1,h2,…,hk]{\displaystyle \;h^{\ast }=\left[h_{1},h_{2},\ldots ,h_{k}\right]\;} is aone-to-one function fromRk{\displaystyle \mathbb {R} ^{k}} to itself, and reparameterize the likelihood function by settingϕi=hi(θ1,θ2,…,θk) .{\displaystyle \;\phi _{i}=h_{i}(\theta _{1},\theta _{2},\ldots ,\theta _{k})~.}[12] Because of the equivariance of the maximum likelihood estimator, the properties of the MLE apply to the restricted estimates also.[13] For instance, in amultivariate normal distribution thecovariance matrixΣ{\displaystyle \,\Sigma \,} must bepositive-definite; this restriction can be imposed by replacingΣ=ΓTΓ,{\displaystyle \;\Sigma =\Gamma ^{\mathsf {T}}\Gamma \;,} whereΓ{\displaystyle \Gamma } is a realupper triangular matrix andΓT{\displaystyle \Gamma ^{\mathsf {T}}} is itstranspose.[14]

In practice, restrictions are usually imposed using the method of Lagrange which, given the constraints as defined above, leads to therestricted likelihood equations∂ℓ∂θ−∂h(θ)T∂θλ=0{\displaystyle {\frac {\partial \ell }{\partial \theta }}-{\frac {\partial h(\theta )^{\mathsf {T}}}{\partial \theta }}\lambda =0} andh(θ)=0,{\displaystyle h(\theta )=0\;,}

where λ=[λ1,λ2,…,λr]T {\displaystyle ~\lambda =\left[\lambda _{1},\lambda _{2},\ldots ,\lambda _{r}\right]^{\mathsf {T}}~} is a column-vector ofLagrange multipliers and∂h(θ)T∂θ{\displaystyle \;{\frac {\partial h(\theta )^{\mathsf {T}}}{\partial \theta }}\;} is thek × rJacobian matrix of partial derivatives.[12] Naturally, if the constraints are not binding at the maximum, the Lagrange multipliers should be zero.[15] This in turn allows for a statistical test of the "validity" of the constraint, known as theLagrange multiplier test.

Nonparametric maximum likelihood estimation

[edit]

Nonparametric maximum likelihood estimation can be performed using theempirical likelihood.

Properties

[edit]

A maximum likelihood estimator is anextremum estimator obtained by maximizing, as a function ofθ, theobjective functionℓ^(θ;x){\displaystyle {\widehat {\ell \,}}(\theta \,;x)}. If the data areindependent and identically distributed, then we haveℓ^(θ;x)=∑i=1nln⁡f(xi∣θ),{\displaystyle {\widehat {\ell \,}}(\theta \,;x)=\sum _{i=1}^{n}\ln f(x_{i}\mid \theta ),}this being the sample analogue of the expected log-likelihoodℓ(θ)=E⁡[ln⁡f(xi∣θ)]{\displaystyle \ell (\theta )=\operatorname {\mathbb {E} } [\,\ln f(x_{i}\mid \theta )\,]}, where this expectation is taken with respect to the true density.

Maximum-likelihood estimators have no optimum properties for finite samples, in the sense that (when evaluated on finite samples) other estimators may have greater concentration around the true parameter-value.[16] However, like other estimation methods, maximum likelihood estimation possesses a number of attractivelimiting properties: As the sample size increases to infinity, sequences of maximum likelihood estimators have these properties:

Consistency

[edit]

Under the conditions outlined below, the maximum likelihood estimator isconsistent. The consistency means that if the data were generated byf(⋅;θ0){\displaystyle f(\cdot \,;\theta _{0})} and we have a sufficiently large number of observationsn, then it is possible to find the value ofθ0 with arbitrary precision. In mathematical terms this means that asn goes to infinity the estimatorθ^{\displaystyle {\widehat {\theta \,}}}converges in probability to its true value:θ^mle →p θ0.{\displaystyle {\widehat {\theta \,}}_{\mathrm {mle} }\ {\xrightarrow {\text{p}}}\ \theta _{0}.}

Under slightly stronger conditions, the estimator convergesalmost surely (orstrongly):θ^mle →a.s. θ0.{\displaystyle {\widehat {\theta \,}}_{\mathrm {mle} }\ {\xrightarrow {\text{a.s.}}}\ \theta _{0}.}

In practical applications, data is never generated byf(⋅;θ0){\displaystyle f(\cdot \,;\theta _{0})}. Rather,f(⋅;θ0){\displaystyle f(\cdot \,;\theta _{0})} is a model, often in idealized form, of the process generated by the data. It is a common aphorism in statistics thatall models are wrong. Thus, true consistency does not occur in practical applications. Nevertheless, consistency is often considered to be a desirable property for an estimator to have.

To establish consistency, the following conditions are sufficient.[17]

  1. Identification of the model:

    θ≠θ0⇔f(⋅∣θ)≠f(⋅∣θ0).{\displaystyle \theta \neq \theta _{0}\quad \Leftrightarrow \quad f(\cdot \mid \theta )\neq f(\cdot \mid \theta _{0}).}In other words, different parameter valuesθ correspond to different distributions within the model. If this condition did not hold, there would be some valueθ1 such thatθ0 andθ1 generate an identical distribution of the observable data. Then we would not be able to distinguish between these two parameters even with an infinite amount of data—these parameters would have beenobservationally equivalent.

    The identification condition is absolutely necessary for the ML estimator to be consistent. When this condition holds, the limiting likelihood functionℓ(θ|·) has unique global maximum atθ0.
  2. Compactness: the parameter space Θ of the model iscompact.

    The identification condition establishes that the log-likelihood has a unique global maximum. Compactness implies that the likelihood cannot approach the maximum value arbitrarily close at some other point (as demonstrated for example in the picture on the right).

    Compactness is only a sufficient condition and not a necessary condition. Compactness can be replaced by some other conditions, such as:

    • bothconcavity of the log-likelihood function and compactness of some (nonempty) upperlevel sets of the log-likelihood function, or
    • existence of a compactneighborhoodN ofθ0 such that outside ofN the log-likelihood function is less than the maximum by at least someε > 0.
  3. Continuity: the functionlnf(x |θ) is continuous inθ for almost all values ofx:

    P⁡[ln⁡f(x∣θ)∈C0(Θ)]=1.{\displaystyle \operatorname {\mathbb {P} } {\Bigl [}\;\ln f(x\mid \theta )\;\in \;C^{0}(\Theta )\;{\Bigr ]}=1.}

    The continuity here can be replaced with a slightly weaker condition ofupper semi-continuity.
  4. Dominance: there existsD(x) integrable with respect to the distributionf(x | θ0) such that|ln⁡f(x∣θ)|<D(x) for all θ∈Θ.{\displaystyle {\Bigl |}\ln f(x\mid \theta ){\Bigr |}<D(x)\quad {\text{ for all }}\theta \in \Theta .}By theuniform law of large numbers, the dominance condition together with continuity establish the uniform convergence in probability of the log-likelihood:supθ∈Θ|ℓ^(θ∣x)−ℓ(θ)| →p 0.{\displaystyle \sup _{\theta \in \Theta }\left|{\widehat {\ell \,}}(\theta \mid x)-\ell (\theta )\,\right|\ \xrightarrow {\text{p}} \ 0.}

The dominance condition can be employed in the case ofi.i.d. observations. In the non-i.i.d. case, the uniform convergence in probability can be checked by showing that the sequenceℓ^(θ∣x){\displaystyle {\widehat {\ell \,}}(\theta \mid x)} isstochastically equicontinuous.

If one wants to demonstrate that the ML estimatorθ^{\displaystyle {\widehat {\theta \,}}} converges toθ0almost surely, then a stronger condition of uniform convergence almost surely has to be imposed:supθ∈Θ‖ℓ^(θ∣x)−ℓ(θ)‖ →a.s. 0.{\displaystyle \sup _{\theta \in \Theta }\left\|\;{\widehat {\ell \,}}(\theta \mid x)-\ell (\theta )\;\right\|\ \xrightarrow {\text{a.s.}} \ 0.}

Additionally, if (as assumed above) the data were generated byf(⋅;θ0){\displaystyle f(\cdot \,;\theta _{0})}, then under certain conditions, it can also be shown that the maximum likelihood estimatorconverges in distribution to a normal distribution. Specifically,[18]n(θ^mle−θ0) →d N(0,I−1){\displaystyle {\sqrt {n}}\left({\widehat {\theta \,}}_{\mathrm {mle} }-\theta _{0}\right)\ \xrightarrow {d} \ {\mathcal {N}}\left(0,\,I^{-1}\right)}whereI is theFisher information matrix.

Functional invariance

[edit]

The maximum likelihood estimator selects the parameter value which gives the observed data the largest possible probability (or probability density, in the continuous case). If the parameter consists of a number of components, then we define their separate maximum likelihood estimators, as the corresponding component of the MLE of the complete parameter. Consistent with this, ifθ^{\displaystyle {\widehat {\theta \,}}} is the MLE forθ{\displaystyle \theta }, and ifg(θ){\displaystyle g(\theta )} is any transformation ofθ{\displaystyle \theta }, then the MLE forα=g(θ){\displaystyle \alpha =g(\theta )} is by definition[19]

α^=g(θ^).{\displaystyle {\widehat {\alpha }}=g(\,{\widehat {\theta \,}}\,).\,}

It maximizes the so-calledprofile likelihood:

L¯(α)=supθ:α=g(θ)L(θ).{\displaystyle {\bar {L}}(\alpha )=\sup _{\theta :\alpha =g(\theta )}L(\theta ).\,}

The MLE is also equivariant with respect to certain transformations of the data. Ify=g(x){\displaystyle y=g(x)} whereg{\displaystyle g} is one to one and does not depend on the parameters to be estimated, then the density functions satisfy

fY(y)=fX(g−1(y))|(g−1(y))′|{\displaystyle f_{Y}(y)=f_{X}(g^{-1}(y))\,|(g^{-1}(y))^{\prime }|}

and hence the likelihood functions forX{\displaystyle X} andY{\displaystyle Y} differ only by a factor that does not depend on the model parameters.

For example, the MLE parameters of the log-normal distribution are the same as those of the normal distribution fitted to the logarithm of the data. In fact, in the log-normal case ifX∼N(0,1){\displaystyle X\sim {\mathcal {N}}(0,1)}, thenY=g(X)=eX{\displaystyle Y=g(X)=e^{X}} follows alog-normal distribution. The density of Y follows withfX{\displaystyle f_{X}} standardNormal andg−1(y)=log⁡(y){\displaystyle g^{-1}(y)=\log(y)},|(g−1(y))′|=1y{\displaystyle |(g^{-1}(y))^{\prime }|={\frac {1}{y}}} fory>0{\displaystyle y>0}.

Efficiency

[edit]

As assumed above, if the data were generated by f(⋅;θ0) ,{\displaystyle ~f(\cdot \,;\theta _{0})~,} then under certain conditions, it can also be shown that the maximum likelihood estimatorconverges in distribution to a normal distribution. It is√n -consistent and asymptotically efficient, meaning that it reaches theCramér–Rao bound. Specifically,[18]

n(θ^mle−θ0)  →d  N(0, I−1) ,{\displaystyle {\sqrt {n\,}}\,\left({\widehat {\theta \,}}_{\text{mle}}-\theta _{0}\right)\ \ \xrightarrow {d} \ \ {\mathcal {N}}\left(0,\ {\mathcal {I}}^{-1}\right)~,}where I {\displaystyle ~{\mathcal {I}}~} is theFisher information matrix:Ijk=E[−∂2ln⁡fθ0(Xt)∂θj∂θk] .{\displaystyle {\mathcal {I}}_{jk}=\operatorname {\mathbb {E} } \,{\biggl [}\;-{\frac {\partial ^{2}\ln f_{\theta _{0}}(X_{t})}{\partial \theta _{j}\,\partial \theta _{k}}}\;{\biggr ]}~.}

In particular, it means that thebias of the maximum likelihood estimator is equal to zero up to the order⁠1/√n ⁠.

Second-order efficiency after correction for bias

[edit]

However, when we consider the higher-order terms in theexpansion of the distribution of this estimator, it turns out thatθmle has bias of order1⁄n. This bias is equal to (componentwise)[20]

bh≡E⁡[(θ^mle−θ0)h]=1n∑i,j,k=1mIhiIjk(12Kijk+Jj,ik){\displaystyle b_{h}\;\equiv \;\operatorname {\mathbb {E} } {\biggl [}\;\left({\widehat {\theta }}_{\mathrm {mle} }-\theta _{0}\right)_{h}\;{\biggr ]}\;=\;{\frac {1}{\,n\,}}\,\sum _{i,j,k=1}^{m}\;{\mathcal {I}}^{hi}\;{\mathcal {I}}^{jk}\left({\frac {1}{\,2\,}}\,K_{ijk}\;+\;J_{j,ik}\right)}

whereIjk{\displaystyle {\mathcal {I}}^{jk}} (with superscripts) denotes the (j,k)-th component of theinverse Fisher information matrixI−1{\displaystyle {\mathcal {I}}^{-1}}, and

12Kijk+Jj,ik=E[12∂3ln⁡fθ0(Xt)∂θi∂θj∂θk+∂ln⁡fθ0(Xt)∂θj∂2ln⁡fθ0(Xt)∂θi∂θk] .{\displaystyle {\frac {1}{\,2\,}}\,K_{ijk}\;+\;J_{j,ik}\;=\;\operatorname {\mathbb {E} } \,{\biggl [}\;{\frac {1}{2}}{\frac {\partial ^{3}\ln f_{\theta _{0}}(X_{t})}{\partial \theta _{i}\;\partial \theta _{j}\;\partial \theta _{k}}}+{\frac {\;\partial \ln f_{\theta _{0}}(X_{t})\;}{\partial \theta _{j}}}\,{\frac {\;\partial ^{2}\ln f_{\theta _{0}}(X_{t})\;}{\partial \theta _{i}\,\partial \theta _{k}}}\;{\biggr ]}~.}

Using these formulae it is possible to estimate the second-order bias of the maximum likelihood estimator, andcorrect for that bias by subtracting it:θ^mle∗=θ^mle−b^ .{\displaystyle {\widehat {\theta \,}}_{\text{mle}}^{*}={\widehat {\theta \,}}_{\text{mle}}-{\widehat {b\,}}~.}This estimator is unbiased up to the terms of order⁠1/ n ⁠, and is called thebias-corrected maximum likelihood estimator.

This bias-corrected estimator issecond-order efficient (at least within the curved exponential family), meaning that it has minimal mean squared error among all second-order bias-corrected estimators, up to the terms of the order⁠1/ n2 ⁠ . It is possible to continue this process, that is to derive the third-order bias-correction term, and so on. However, the maximum likelihood estimator isnot third-order efficient.[21]

Relation to Bayesian inference

[edit]

A maximum likelihood estimator coincides with themost probableBayesian estimator given auniformprior distribution on theparameters. Indeed, themaximum a posteriori estimate is the parameterθ that maximizes the probability ofθ given the data, given by Bayes' theorem:

P⁡(θ∣x1,x2,…,xn)=f(x1,x2,…,xn∣θ)P⁡(θ)P⁡(x1,x2,…,xn){\displaystyle \operatorname {\mathbb {P} } (\theta \mid x_{1},x_{2},\ldots ,x_{n})={\frac {f(x_{1},x_{2},\ldots ,x_{n}\mid \theta )\operatorname {\mathbb {P} } (\theta )}{\operatorname {\mathbb {P} } (x_{1},x_{2},\ldots ,x_{n})}}}

whereP⁡(θ){\displaystyle \operatorname {\mathbb {P} } (\theta )} is the prior distribution for the parameterθ and whereP⁡(x1,x2,…,xn){\displaystyle \operatorname {\mathbb {P} } (x_{1},x_{2},\ldots ,x_{n})} is the probability of the data averaged over all parameters. Since the denominator is independent ofθ, the Bayesian estimator is obtained by maximizingf(x1,x2,…,xn∣θ)P⁡(θ){\displaystyle f(x_{1},x_{2},\ldots ,x_{n}\mid \theta )\operatorname {\mathbb {P} } (\theta )} with respect toθ. If we further assume that the priorP⁡(θ){\displaystyle \operatorname {\mathbb {P} } (\theta )} is a uniform distribution, the Bayesian estimator is obtained by maximizing the likelihood functionf(x1,x2,…,xn∣θ){\displaystyle f(x_{1},x_{2},\ldots ,x_{n}\mid \theta )}. Thus the Bayesian estimator coincides with the maximum likelihood estimator for a uniform prior distributionP⁡(θ){\displaystyle \operatorname {\mathbb {P} } (\theta )}.

Application of maximum-likelihood estimation in Bayes decision theory

[edit]

In many practical applications inmachine learning, maximum-likelihood estimation is used as the model for parameter estimation.

The Bayesian Decision theory is about designing a classifier that minimizes total expected risk, especially, when the costs (the loss function) associated with different decisions are equal, the classifier is minimizing the error over the whole distribution.[22]

Thus, the Bayes Decision Rule is stated as

"decidew1{\displaystyle \;w_{1}\;} if P⁡(w1|x)>P⁡(w2|x) ; {\displaystyle ~\operatorname {\mathbb {P} } (w_{1}|x)\;>\;\operatorname {\mathbb {P} } (w_{2}|x)~;~} otherwise decidew2{\displaystyle \;w_{2}\;}"

wherew1,w2{\displaystyle \;w_{1}\,,w_{2}\;} are predictions of different classes. From a perspective of minimizing error, it can also be stated asw=argmaxw∫−∞∞P⁡( error∣x)P⁡(x)d⁡x {\displaystyle w={\underset {w}{\operatorname {arg\;max} }}\;\int _{-\infty }^{\infty }\operatorname {\mathbb {P} } ({\text{ error}}\mid x)\operatorname {\mathbb {P} } (x)\,\operatorname {d} x~}whereP⁡( error∣x)=P⁡(w1∣x) {\displaystyle \operatorname {\mathbb {P} } ({\text{ error}}\mid x)=\operatorname {\mathbb {P} } (w_{1}\mid x)~}if we decidew2{\displaystyle \;w_{2}\;} andP⁡( error∣x)=P⁡(w2∣x){\displaystyle \;\operatorname {\mathbb {P} } ({\text{ error}}\mid x)=\operatorname {\mathbb {P} } (w_{2}\mid x)\;} if we decidew1.{\displaystyle \;w_{1}\;.}

By applyingBayes' theoremP⁡(wi∣x)=P⁡(x∣wi)P⁡(wi)P⁡(x){\displaystyle \operatorname {\mathbb {P} } (w_{i}\mid x)={\frac {\operatorname {\mathbb {P} } (x\mid w_{i})\operatorname {\mathbb {P} } (w_{i})}{\operatorname {\mathbb {P} } (x)}}},and if we further assume the zero-or-one loss function, which is a same loss for all errors, the Bayes Decision rule can be reformulated as:hBayes=argmaxw[P⁡(x∣w)P⁡(w)],{\displaystyle h_{\text{Bayes}}={\underset {w}{\operatorname {arg\;max} }}\,{\bigl [}\,\operatorname {\mathbb {P} } (x\mid w)\,\operatorname {\mathbb {P} } (w)\,{\bigr ]}\;,}wherehBayes{\displaystyle h_{\text{Bayes}}} is the prediction andP⁡(w){\displaystyle \;\operatorname {\mathbb {P} } (w)\;} is theprior probability.

Relation to minimizing Kullback–Leibler divergence and cross entropy

[edit]

Findingθ^{\displaystyle {\hat {\theta }}} that maximizes the likelihood is asymptotically equivalent to finding theθ^{\displaystyle {\hat {\theta }}} that defines a probability distribution (Qθ^{\displaystyle Q_{\hat {\theta }}}) that has a minimal distance, in terms ofKullback–Leibler divergence, to the real probability distribution from which our data were generated (i.e., generated byPθ0{\displaystyle P_{\theta _{0}}}).[23] In an ideal world, P and Q are the same (and the only thing unknown isθ{\displaystyle \theta } that defines P), but even if they are not and the model we use is misspecified, still the MLE will give us the "closest" distribution (within the restriction of a model Q that depends onθ^{\displaystyle {\hat {\theta }}}) to the real distributionPθ0{\displaystyle P_{\theta _{0}}}.[24]

Proof.

For simplicity of notation, let's assume that P=Q. Let there beni.i.d data samplesy=(y1,y2,…,yn){\displaystyle \mathbf {y} =(y_{1},y_{2},\ldots ,y_{n})} from some probabilityy∼Pθ0{\displaystyle y\sim P_{\theta _{0}}}, that we try to estimate by findingθ^{\displaystyle {\hat {\theta }}} that will maximize the likelihood usingPθ{\displaystyle P_{\theta }}, then:θ^=argmaxθLPθ(y)=argmaxθPθ(y)=argmaxθP(y∣θ)=argmaxθ∏i=1nP(yi∣θ)=argmaxθ∑i=1nlog⁡P(yi∣θ)=argmaxθ(∑i=1nlog⁡P(yi∣θ)−∑i=1nlog⁡P(yi∣θ0))=argmaxθ∑i=1n(log⁡P(yi∣θ)−log⁡P(yi∣θ0))=argmaxθ∑i=1nlog⁡P(yi∣θ)P(yi∣θ0)=argminθ∑i=1nlog⁡P(yi∣θ0)P(yi∣θ)=argminθ1n∑i=1nlog⁡P(yi∣θ0)P(yi∣θ)=argminθ1n∑i=1nhθ(yi)⟶n→∞argminθE[hθ(y)]=argminθ∫Pθ0(y)hθ(y)dy=argminθ∫Pθ0(y)log⁡P(y∣θ0)P(y∣θ)dy=argminθDKL(Pθ0∥Pθ){\displaystyle {\begin{aligned}{\hat {\theta }}&={\underset {\theta }{\operatorname {arg\,max} }}\,L_{P_{\theta }}(\mathbf {y} )={\underset {\theta }{\operatorname {arg\,max} }}\,P_{\theta }(\mathbf {y} )={\underset {\theta }{\operatorname {arg\,max} }}\,P(\mathbf {y} \mid \theta )\\&={\underset {\theta }{\operatorname {arg\,max} }}\,\prod _{i=1}^{n}P(y_{i}\mid \theta )={\underset {\theta }{\operatorname {arg\,max} }}\,\sum _{i=1}^{n}\log P(y_{i}\mid \theta )\\&={\underset {\theta }{\operatorname {arg\,max} }}\,\left(\sum _{i=1}^{n}\log P(y_{i}\mid \theta )-\sum _{i=1}^{n}\log P(y_{i}\mid \theta _{0})\right)={\underset {\theta }{\operatorname {arg\,max} }}\,\sum _{i=1}^{n}\left(\log P(y_{i}\mid \theta )-\log P(y_{i}\mid \theta _{0})\right)\\&={\underset {\theta }{\operatorname {arg\,max} }}\,\sum _{i=1}^{n}\log {\frac {P(y_{i}\mid \theta )}{P(y_{i}\mid \theta _{0})}}={\underset {\theta }{\operatorname {arg\,min} }}\,\sum _{i=1}^{n}\log {\frac {P(y_{i}\mid \theta _{0})}{P(y_{i}\mid \theta )}}={\underset {\theta }{\operatorname {arg\,min} }}\,{\frac {1}{n}}\sum _{i=1}^{n}\log {\frac {P(y_{i}\mid \theta _{0})}{P(y_{i}\mid \theta )}}\\&={\underset {\theta }{\operatorname {arg\,min} }}\,{\frac {1}{n}}\sum _{i=1}^{n}h_{\theta }(y_{i})\quad {\underset {n\to \infty }{\longrightarrow }}\quad {\underset {\theta }{\operatorname {arg\,min} }}\,E[h_{\theta }(y)]\\&={\underset {\theta }{\operatorname {arg\,min} }}\,\int P_{\theta _{0}}(y)h_{\theta }(y)dy={\underset {\theta }{\operatorname {arg\,min} }}\,\int P_{\theta _{0}}(y)\log {\frac {P(y\mid \theta _{0})}{P(y\mid \theta )}}dy\\&={\underset {\theta }{\operatorname {arg\,min} }}\,D_{\text{KL}}(P_{\theta _{0}}\parallel P_{\theta })\end{aligned}}}

Wherehθ(x)=log⁡P(x∣θ0)P(x∣θ){\displaystyle h_{\theta }(x)=\log {\frac {P(x\mid \theta _{0})}{P(x\mid \theta )}}}. Usingh helps see how we are using thelaw of large numbers to move from the average ofh(x) to theexpectancy of it using thelaw of the unconscious statistician. The first several transitions have to do with laws oflogarithm and that findingθ^{\displaystyle {\hat {\theta }}} that maximizes some function will also be the one that maximizes some monotonic transformation of that function (i.e.: adding/multiplying by a constant).

Sincecross entropy is justShannon's entropy plus KL divergence, and since the entropy ofPθ0{\displaystyle P_{\theta _{0}}} is constant, then the MLE is also asymptotically minimizing cross entropy.[25]

Examples

[edit]

Discrete uniform distribution

[edit]
Main article:German tank problem

Consider a case wheren tickets numbered from 1 ton are placed in a box and one is selected at random (seeuniform distribution); thus, the sample size is 1. Ifn is unknown, then the maximum likelihood estimatorn^{\displaystyle {\widehat {n}}} ofn is the numberm on the drawn ticket. (The likelihood is 0 forn < m,1⁄n forn ≥ m, and this is greatest whenn = m. Note that the maximum likelihood estimate ofn occurs at the lower extreme of possible values {m, m + 1, ...}, rather than somewhere in the "middle" of the range of possible values, which would result in less bias.) Theexpected value of the numberm on the drawn ticket, and therefore the expected value ofn^{\displaystyle {\widehat {n}}}, is (n + 1)/2. As a result, with a sample size of 1, the maximum likelihood estimator forn will systematically underestimaten by (n − 1)/2.

Discrete distribution, finite parameter space

[edit]

Suppose one wishes to determine just how biased anunfair coin is. Call the probability of tossing a 'head'p. The goal then becomes to determinep.

Suppose the coin is tossed 80 times: i.e. the sample might be something likex1 = H,x2 = T, ...,x80 = T, and the count of the number ofheads "H" is observed.

The probability of tossingtails is 1 − p (so herep isθ above). Suppose the outcome is 49 heads and 31 tails, and suppose the coin was taken from a box containing three coins: one which gives heads with probabilityp = 1⁄3, one which gives heads with probabilityp = 1⁄2 and another which gives heads with probabilityp = 2⁄3. The coins have lost their labels, so which one it was is unknown. Using maximum likelihood estimation, the coin that has the largest likelihood can be found, given the data that were observed. By using theprobability mass function of thebinomial distribution with sample size equal to 80, number successes equal to 49 but for different values ofp (the "probability of success"), the likelihood function (defined below) takes one of three values:

P⁡[H=49∣p=13]=(8049)(13)49(1−13)31≈0.000,P⁡[H=49∣p=12]=(8049)(12)49(1−12)31≈0.012,P⁡[H=49∣p=23]=(8049)(23)49(1−23)31≈0.054 .{\displaystyle {\begin{aligned}\operatorname {\mathbb {P} } {\bigl [}\;\mathrm {H} =49\mid p={\tfrac {1}{3}}\;{\bigr ]}&={\binom {80}{49}}({\tfrac {1}{3}})^{49}(1-{\tfrac {1}{3}})^{31}\approx 0.000,\\[6pt]\operatorname {\mathbb {P} } {\bigl [}\;\mathrm {H} =49\mid p={\tfrac {1}{2}}\;{\bigr ]}&={\binom {80}{49}}({\tfrac {1}{2}})^{49}(1-{\tfrac {1}{2}})^{31}\approx 0.012,\\[6pt]\operatorname {\mathbb {P} } {\bigl [}\;\mathrm {H} =49\mid p={\tfrac {2}{3}}\;{\bigr ]}&={\binom {80}{49}}({\tfrac {2}{3}})^{49}(1-{\tfrac {2}{3}})^{31}\approx 0.054~.\end{aligned}}}

The likelihood is maximized whenp = 2⁄3, and so this is themaximum likelihood estimate for p.

Discrete distribution, continuous parameter space

[edit]

Now suppose that there was only one coin but itsp could have been any value 0 ≤p ≤ 1 . The likelihood function to be maximised isL(p)=fD(H=49∣p)=(8049)p49(1−p)31 ,{\displaystyle L(p)=f_{D}(\mathrm {H} =49\mid p)={\binom {80}{49}}p^{49}(1-p)^{31}~,}

and the maximisation is over all possible values0 ≤p ≤ 1 .

Likelihood function for proportion value of a binomial process (n = 10)

One way to maximize this function is bydifferentiating with respect top and setting to zero:

0=∂∂p((8049)p49(1−p)31) ,0=49p48(1−p)31−31p49(1−p)30=p48(1−p)30[49(1−p)−31p]=p48(1−p)30[49−80p] .{\displaystyle {\begin{aligned}0&={\frac {\partial }{\partial p}}\left({\binom {80}{49}}p^{49}(1-p)^{31}\right)~,\\[8pt]0&=49p^{48}(1-p)^{31}-31p^{49}(1-p)^{30}\\[8pt]&=p^{48}(1-p)^{30}\left[49(1-p)-31p\right]\\[8pt]&=p^{48}(1-p)^{30}\left[49-80p\right]~.\end{aligned}}}

This is a product of three terms. The first term is 0 whenp = 0. The second is 0 whenp = 1. The third is zero whenp = 49⁄80. The solution that maximizes the likelihood is clearlyp = 49⁄80 (sincep = 0 andp = 1 result in a likelihood of 0). Thus themaximum likelihood estimator forp is49⁄80.

This result is easily generalized by substituting a letter such ass in the place of 49 to represent the observed number of 'successes' of ourBernoulli trials, and a letter such asn in the place of 80 to represent the number of Bernoulli trials. Exactly the same calculation yieldss⁄n which is the maximum likelihood estimator for any sequence ofn Bernoulli trials resulting ins 'successes'.

Continuous distribution, continuous parameter space

[edit]

For thenormal distributionN(μ,σ2){\displaystyle {\mathcal {N}}(\mu ,\sigma ^{2})} which hasprobability density function

f(x∣μ,σ2)=12πσ2 exp⁡(−(x−μ)22σ2),{\displaystyle f(x\mid \mu ,\sigma ^{2})={\frac {1}{{\sqrt {2\pi \sigma ^{2}}}\ }}\exp \left(-{\frac {(x-\mu )^{2}}{2\sigma ^{2}}}\right),}

the correspondingprobability density function for a sample ofnindependent identically distributed normal random variables (the likelihood) is

f(x1,…,xn∣μ,σ2)=∏i=1nf(xi∣μ,σ2)=(12πσ2)n/2exp⁡(−∑i=1n(xi−μ)22σ2).{\displaystyle f(x_{1},\ldots ,x_{n}\mid \mu ,\sigma ^{2})=\prod _{i=1}^{n}f(x_{i}\mid \mu ,\sigma ^{2})=\left({\frac {1}{2\pi \sigma ^{2}}}\right)^{n/2}\exp \left(-{\frac {\sum _{i=1}^{n}(x_{i}-\mu )^{2}}{2\sigma ^{2}}}\right).}

This family of distributions has two parameters:θ = (μ, σ); so we maximize the likelihood,L(μ,σ2)=f(x1,…,xn∣μ,σ2){\displaystyle {\mathcal {L}}(\mu ,\sigma ^{2})=f(x_{1},\ldots ,x_{n}\mid \mu ,\sigma ^{2})}, over both parameters simultaneously, or if possible, individually.

Since thelogarithm function itself is acontinuousstrictly increasing function over therange of the likelihood, the values which maximize the likelihood will also maximize its logarithm (the log-likelihood itself is not necessarily strictly increasing). The log-likelihood can be written as follows:

log⁡(L(μ,σ2))=−n2log⁡(2πσ2)−12σ2∑i=1n(xi−μ)2{\displaystyle \log {\Bigl (}{\mathcal {L}}(\mu ,\sigma ^{2}){\Bigr )}=-{\frac {\,n\,}{2}}\log(2\pi \sigma ^{2})-{\frac {1}{2\sigma ^{2}}}\sum _{i=1}^{n}(\,x_{i}-\mu \,)^{2}}

(Note: the log-likelihood is closely related toinformation entropy andFisher information.)

We now compute the derivatives of this log-likelihood as follows.

0=∂∂μlog⁡(L(μ,σ2))=0−−2n(x¯−μ)2σ2.{\displaystyle {\begin{aligned}0&={\frac {\partial }{\partial \mu }}\log {\Bigl (}{\mathcal {L}}(\mu ,\sigma ^{2}){\Bigr )}=0-{\frac {\;-2n({\bar {x}}-\mu )\;}{2\sigma ^{2}}}.\end{aligned}}}wherex¯{\displaystyle {\bar {x}}} is thesample mean. This is solved by

μ^=x¯=∑i=1nxin.{\displaystyle {\widehat {\mu }}={\bar {x}}=\sum _{i=1}^{n}{\frac {\,x_{i}\,}{n}}.}

This is indeed the maximum of the function, since it is the only turning point inμ and the second derivative is strictly less than zero. Itsexpected value is equal to the parameterμ of the given distribution,

E⁡[μ^]=μ,{\displaystyle \operatorname {\mathbb {E} } {\bigl [}\;{\widehat {\mu }}\;{\bigr ]}=\mu ,\,}

which means that the maximum likelihood estimatorμ^{\displaystyle {\widehat {\mu }}} is unbiased.

Similarly we differentiate the log-likelihood with respect toσ and equate to zero:

0=∂∂σlog⁡(L(μ,σ2))=−nσ+1σ3∑i=1n(xi−μ)2.{\displaystyle {\begin{aligned}0&={\frac {\partial }{\partial \sigma }}\log {\Bigl (}{\mathcal {L}}(\mu ,\sigma ^{2}){\Bigr )}=-{\frac {\,n\,}{\sigma }}+{\frac {1}{\sigma ^{3}}}\sum _{i=1}^{n}(\,x_{i}-\mu \,)^{2}.\end{aligned}}}

which is solved by

σ^2=1n∑i=1n(xi−μ)2.{\displaystyle {\widehat {\sigma }}^{2}={\frac {1}{n}}\sum _{i=1}^{n}(x_{i}-\mu )^{2}.}

Inserting the estimateμ=μ^{\displaystyle \mu ={\widehat {\mu }}} we obtain

σ^2=1n∑i=1n(xi−x¯)2=1n∑i=1nxi2−1n2∑i=1n∑j=1nxixj.{\displaystyle {\widehat {\sigma }}^{2}={\frac {1}{n}}\sum _{i=1}^{n}(x_{i}-{\bar {x}})^{2}={\frac {1}{n}}\sum _{i=1}^{n}x_{i}^{2}-{\frac {1}{n^{2}}}\sum _{i=1}^{n}\sum _{j=1}^{n}x_{i}x_{j}.}

To calculate its expected value, it is convenient to rewrite the expression in terms of zero-mean random variables (statistical error)δi≡μ−xi{\displaystyle \delta _{i}\equiv \mu -x_{i}}. Expressing the estimate in these variables yields

σ^2=1n∑i=1n(μ−δi)2−1n2∑i=1n∑j=1n(μ−δi)(μ−δj).{\displaystyle {\widehat {\sigma }}^{2}={\frac {1}{n}}\sum _{i=1}^{n}(\mu -\delta _{i})^{2}-{\frac {1}{n^{2}}}\sum _{i=1}^{n}\sum _{j=1}^{n}(\mu -\delta _{i})(\mu -\delta _{j}).}

Simplifying the expression above, utilizing the facts thatE⁡[δi]=0{\displaystyle \operatorname {\mathbb {E} } {\bigl [}\;\delta _{i}\;{\bigr ]}=0} andE⁡[δi2]=σ2{\displaystyle \operatorname {E} {\bigl [}\;\delta _{i}^{2}\;{\bigr ]}=\sigma ^{2}}, allows us to obtain

E⁡[σ^2]=n−1nσ2.{\displaystyle \operatorname {\mathbb {E} } {\bigl [}\;{\widehat {\sigma }}^{2}\;{\bigr ]}={\frac {\,n-1\,}{n}}\sigma ^{2}.}

This means that the estimatorσ^2{\displaystyle {\widehat {\sigma }}^{2}} is biased forσ2{\displaystyle \sigma ^{2}}. It can also be shown thatσ^{\displaystyle {\widehat {\sigma }}} is biased forσ{\displaystyle \sigma }, but that bothσ^2{\displaystyle {\widehat {\sigma }}^{2}} andσ^{\displaystyle {\widehat {\sigma }}} are consistent.

Formally we say that themaximum likelihood estimator forθ=(μ,σ2){\displaystyle \theta =(\mu ,\sigma ^{2})} is

θ^=(μ^,σ^2).{\displaystyle {\widehat {\theta \,}}=\left({\widehat {\mu }},{\widehat {\sigma }}^{2}\right).}

In this case the MLEs could be obtained individually. In general this may not be the case, and the MLEs would have to be obtained simultaneously.

The normal log-likelihood at its maximum takes a particularly simple form:

log⁡(L(μ^,σ^))=−n2(log⁡(2πσ^2)+1){\displaystyle \log {\Bigl (}{\mathcal {L}}({\widehat {\mu }},{\widehat {\sigma }}){\Bigr )}={\frac {\,-n\;\;}{2}}{\bigl (}\,\log(2\pi {\widehat {\sigma }}^{2})+1\,{\bigr )}}

This maximum log-likelihood can be shown to be the same for more generalleast squares, even fornon-linear least squares. This is often used in determining likelihood-based approximateconfidence intervals andconfidence regions, which are generally more accurate than those using the asymptotic normality discussed above.

Non-independent variables

[edit]

It may be the case that variables are correlated, or more generally, not independent. Two random variablesy1{\displaystyle y_{1}} andy2{\displaystyle y_{2}} are independent only if their joint probability density function is the product of the individual probability density functions, i.e.

f(y1,y2)=f(y1)f(y2){\displaystyle f(y_{1},y_{2})=f(y_{1})f(y_{2})\,}

Suppose one constructs an order-n Gaussian vector out of random variables(y1,…,yn){\displaystyle (y_{1},\ldots ,y_{n})}, where each variable has means given by(μ1,…,μn){\displaystyle (\mu _{1},\ldots ,\mu _{n})}. Furthermore, let thecovariance matrix be denoted byΣ{\displaystyle {\mathit {\Sigma }}}. The joint probability density function of thesen random variables then follows amultivariate normal distribution given by:

f(y1,…,yn)=1(2π)n/2det(Σ)exp⁡(−12[y1−μ1,…,yn−μn]Σ−1[y1−μ1,…,yn−μn]T){\displaystyle f(y_{1},\ldots ,y_{n})={\frac {1}{(2\pi )^{n/2}{\sqrt {\det({\mathit {\Sigma }})}}}}\exp \left(-{\frac {1}{2}}\left[y_{1}-\mu _{1},\ldots ,y_{n}-\mu _{n}\right]{\mathit {\Sigma }}^{-1}\left[y_{1}-\mu _{1},\ldots ,y_{n}-\mu _{n}\right]^{\mathrm {T} }\right)}

In thebivariate case, the joint probability density function is given by:

f(y1,y2)=12πσ1σ21−ρ2exp⁡[−12(1−ρ2)((y1−μ1)2σ12−2ρ(y1−μ1)(y2−μ2)σ1σ2+(y2−μ2)2σ22)]{\displaystyle f(y_{1},y_{2})={\frac {1}{2\pi \sigma _{1}\sigma _{2}{\sqrt {1-\rho ^{2}}}}}\exp \left[-{\frac {1}{2(1-\rho ^{2})}}\left({\frac {(y_{1}-\mu _{1})^{2}}{\sigma _{1}^{2}}}-{\frac {2\rho (y_{1}-\mu _{1})(y_{2}-\mu _{2})}{\sigma _{1}\sigma _{2}}}+{\frac {(y_{2}-\mu _{2})^{2}}{\sigma _{2}^{2}}}\right)\right]}

In this and other cases where a joint density function exists, the likelihood function is defined as above, in the section "principles," using this density.

Example

[edit]

X1, X2,…, Xm{\displaystyle X_{1},\ X_{2},\ldots ,\ X_{m}} are counts in cells / boxes 1 up to m; each box has a different probability (think of the boxes being bigger or smaller) and we fix the number of balls that fall to ben{\displaystyle n}:x1+x2+⋯+xm=n{\displaystyle x_{1}+x_{2}+\cdots +x_{m}=n}. The probability of each box ispi{\displaystyle p_{i}}, with a constraint:p1+p2+⋯+pm=1{\displaystyle p_{1}+p_{2}+\cdots +p_{m}=1}. This is a case in which theXi{\displaystyle X_{i}}s are not independent, the joint probability of a vectorx1, x2,…,xm{\displaystyle x_{1},\ x_{2},\ldots ,x_{m}} is called the multinomial and has the form:

f(x1,x2,…,xm∣p1,p2,…,pm)=n!∏xi!∏pixi=(nx1,x2,…,xm)p1x1p2x2⋯pmxm{\displaystyle f(x_{1},x_{2},\ldots ,x_{m}\mid p_{1},p_{2},\ldots ,p_{m})={\frac {n!}{\prod x_{i}!}}\prod p_{i}^{x_{i}}={\binom {n}{x_{1},x_{2},\ldots ,x_{m}}}p_{1}^{x_{1}}p_{2}^{x_{2}}\cdots p_{m}^{x_{m}}}

Each box taken separately against all the other boxes is a binomial and this is an extension thereof.

The log-likelihood of this is:

ℓ(p1,p2,…,pm)=log⁡n!−∑i=1mlog⁡xi!+∑i=1mxilog⁡pi{\displaystyle \ell (p_{1},p_{2},\ldots ,p_{m})=\log n!-\sum _{i=1}^{m}\log x_{i}!+\sum _{i=1}^{m}x_{i}\log p_{i}}

The constraint has to be taken into account and use the Lagrange multipliers:

L(p1,p2,…,pm,λ)=ℓ(p1,p2,…,pm)+λ(1−∑i=1mpi){\displaystyle L(p_{1},p_{2},\ldots ,p_{m},\lambda )=\ell (p_{1},p_{2},\ldots ,p_{m})+\lambda \left(1-\sum _{i=1}^{m}p_{i}\right)}

By posing all the derivatives to be 0, the most natural estimate is derived

p^i=xin{\displaystyle {\hat {p}}_{i}={\frac {x_{i}}{n}}}

Maximizing log likelihood, with and without constraints, can be an unsolvable problem in closed form, then we have to use iterative procedures.

Iterative procedures

[edit]

Except for special cases, the likelihood equations∂ℓ(θ;y)∂θ=0{\displaystyle {\frac {\partial \ell (\theta ;\mathbf {y} )}{\partial \theta }}=0}

cannot be solved explicitly for an estimatorθ^=θ^(y){\displaystyle {\widehat {\theta }}={\widehat {\theta }}(\mathbf {y} )}. Instead, they need to be solvediteratively: starting from an initial guess ofθ{\displaystyle \theta } (sayθ^1{\displaystyle {\widehat {\theta }}_{1}}), one seeks to obtain a convergent sequence{θ^r}{\displaystyle \left\{{\widehat {\theta }}_{r}\right\}}. Many methods for this kind ofoptimization problem are available,[26][27] but the most commonly used ones are algorithms based on an updating formula of the formθ^r+1=θ^r+ηrdr(θ^){\displaystyle {\widehat {\theta }}_{r+1}={\widehat {\theta }}_{r}+\eta _{r}\mathbf {d} _{r}\left({\widehat {\theta }}\right)}

where the vectordr(θ^){\displaystyle \mathbf {d} _{r}\left({\widehat {\theta }}\right)} indicates thedescent direction of therth "step," and the scalarηr{\displaystyle \eta _{r}} captures the "step length,"[28][29] also known as thelearning rate.[30]

Gradient descent method

[edit]

(Note: here it is a maximization problem, so the sign before gradient is flipped)

ηr∈R+{\displaystyle \eta _{r}\in \mathbb {R} ^{+}} that is small enough for convergence anddr(θ^)=∇ℓ(θ^r;y){\displaystyle \mathbf {d} _{r}\left({\widehat {\theta }}\right)=\nabla \ell \left({\widehat {\theta }}_{r};\mathbf {y} \right)}

Gradient descent method requires to calculate the gradient at the rth iteration, but no need to calculate the inverse of second-order derivative, i.e., the Hessian matrix. Therefore, it is computationally faster than Newton-Raphson method.

Newton–Raphson method

[edit]

ηr=1{\displaystyle \eta _{r}=1} anddr(θ^)=−Hr−1(θ^)sr(θ^){\displaystyle \mathbf {d} _{r}\left({\widehat {\theta }}\right)=-\mathbf {H} _{r}^{-1}\left({\widehat {\theta }}\right)\mathbf {s} _{r}\left({\widehat {\theta }}\right)}

wheresr(θ^){\displaystyle \mathbf {s} _{r}({\widehat {\theta }})} is thescore andHr−1(θ^){\displaystyle \mathbf {H} _{r}^{-1}\left({\widehat {\theta }}\right)} is theinverse of theHessian matrix of the log-likelihood function, both evaluated therth iteration.[31][32] But because the calculation of the Hessian matrix iscomputationally costly, numerous alternatives have been proposed. The popularBerndt–Hall–Hall–Hausman algorithm approximates the Hessian with theouter product of the expected gradient, such that

dr(θ^)=−[1n∑t=1n∂ℓ(θ;y)∂θ(∂ℓ(θ;y)∂θ)T]−1sr(θ^){\displaystyle \mathbf {d} _{r}\left({\widehat {\theta }}\right)=-\left[{\frac {1}{n}}\sum _{t=1}^{n}{\frac {\partial \ell (\theta ;\mathbf {y} )}{\partial \theta }}\left({\frac {\partial \ell (\theta ;\mathbf {y} )}{\partial \theta }}\right)^{\mathsf {T}}\right]^{-1}\mathbf {s} _{r}\left({\widehat {\theta }}\right)}

Quasi-Newton methods

[edit]

Other quasi-Newton methods use more elaborate secant updates to give approximation of Hessian matrix.

Davidon–Fletcher–Powell formula

[edit]

DFP formula finds a solution that is symmetric, positive-definite and closest to the current approximate value of second-order derivative:Hk+1=(I−γkykskT)Hk(I−γkskykT)+γkykykT,{\displaystyle \mathbf {H} _{k+1}=\left(I-\gamma _{k}y_{k}s_{k}^{\mathsf {T}}\right)\mathbf {H} _{k}\left(I-\gamma _{k}s_{k}y_{k}^{\mathsf {T}}\right)+\gamma _{k}y_{k}y_{k}^{\mathsf {T}},}

where

yk=∇ℓ(xk+sk)−∇ℓ(xk),{\displaystyle y_{k}=\nabla \ell (x_{k}+s_{k})-\nabla \ell (x_{k}),}γk=1ykTsk,{\displaystyle \gamma _{k}={\frac {1}{y_{k}^{T}s_{k}}},}sk=xk+1−xk.{\displaystyle s_{k}=x_{k+1}-x_{k}.}

Broyden–Fletcher–Goldfarb–Shanno algorithm

[edit]

BFGS also gives a solution that is symmetric and positive-definite:

Bk+1=Bk+ykykTykTsk−BkskskTBkTskTBksk ,{\displaystyle B_{k+1}=B_{k}+{\frac {y_{k}y_{k}^{\mathsf {T}}}{y_{k}^{\mathsf {T}}s_{k}}}-{\frac {B_{k}s_{k}s_{k}^{\mathsf {T}}B_{k}^{\mathsf {T}}}{s_{k}^{\mathsf {T}}B_{k}s_{k}}}\ ,}

where

yk=∇ℓ(xk+sk)−∇ℓ(xk),{\displaystyle y_{k}=\nabla \ell (x_{k}+s_{k})-\nabla \ell (x_{k}),}sk=xk+1−xk.{\displaystyle s_{k}=x_{k+1}-x_{k}.}

BFGS method is not guaranteed to converge unless the function has a quadraticTaylor expansion near an optimum. However, BFGS can have acceptable performance even for non-smooth optimization instances

Fisher's scoring

[edit]

Another popular method is to replace the Hessian with theFisher information matrix,I(θ)=E⁡[Hr(θ^)]{\displaystyle {\mathcal {I}}(\theta )=\operatorname {\mathbb {E} } \left[\mathbf {H} _{r}\left({\widehat {\theta }}\right)\right]}, giving us the Fisher scoring algorithm. This procedure is standard in the estimation of many methods, such asgeneralized linear models.

Although popular, quasi-Newton methods may converge to astationary point that is not necessarily a local or global maximum,[33] but rather a local minimum or asaddle point. Therefore, it is important to assess the validity of the obtained solution to the likelihood equations, by verifying that the Hessian, evaluated at the solution, is bothnegative definite andwell-conditioned.[34]

History

[edit]
Ronald Fisher in 1913

Early users of maximum likelihood includeCarl Friedrich Gauss,Pierre-Simon Laplace,Thorvald N. Thiele, andFrancis Ysidro Edgeworth.[35][36] It wasRonald Fisher however, between 1912 and 1922, who singlehandedly created the modern version of the method.[37][38]

Maximum-likelihood estimation finally transcendedheuristic justification in a proof published bySamuel S. Wilks in 1938, now calledWilks' theorem.[39] The theorem shows that the error in the logarithm of likelihood values for estimates from multiple independent observations is asymptoticallyχ 2-distributed, which enables convenient determination of aconfidence region around any estimate of the parameters. The only difficult part of Wilks' proof depends on the expected value of theFisher information matrix, which is provided by a theorem proven by Fisher.[40] Wilks continued to improve on the generality of the theorem throughout his life, with his most general proof published in 1962.[41]

Reviews of the development of maximum likelihood estimation have been provided by a number of authors.[42][43][44][45][46][47][48][49]

See also

[edit]

Related concepts

[edit]
  • Akaike information criterion: a criterion to compare statistical models, based on MLE
  • Extremum estimator: a more general class of estimators to which MLE belongs
  • Fisher information: information matrix, its relationship to covariance matrix of ML estimates
  • Mean squared error: a measure of how 'good' an estimator of a distributional parameter is (be it the maximum likelihood estimator or some other estimator)
  • RANSAC: a method to estimate parameters of a mathematical model given data that containsoutliers
  • Rao–Blackwell theorem: yields a process for finding the best possible unbiased estimator (in the sense of having minimalmean squared error); the MLE is often a good starting place for the process
  • Wilks' theorem: provides a means of estimating the size and shape of the region of roughly equally-probable estimates for the population's parameter values, using the information from a single sample, using achi-squared distribution

Other estimation methods

[edit]

References

[edit]
  1. ^Rossi, Richard J. (2018).Mathematical Statistics: An Introduction to Likelihood Based Inference. New York: John Wiley & Sons. p. 227.ISBN 978-1-118-77104-4.
  2. ^Hendry, David F.; Nielsen, Bent (2007).Econometric Modeling: A Likelihood Approach. Princeton: Princeton University Press.ISBN 978-0-691-13128-3.
  3. ^Chambers, Raymond L.; Steel, David G.; Wang, Suojin; Welsh, Alan (2012).Maximum Likelihood Estimation for Sample Surveys. Boca Raton: CRC Press.ISBN 978-1-58488-632-7.
  4. ^Ward, Michael Don; Ahlquist, John S. (2018).Maximum Likelihood for Social Science: Strategies for Analysis. New York: Cambridge University Press.ISBN 978-1-107-18582-1.
  5. ^Press, W.H.; Flannery, B.P.; Teukolsky, S.A.; Vetterling, W.T. (1992)."Least Squares as a Maximum Likelihood Estimator".Numerical Recipes in FORTRAN: The Art of Scientific Computing (2nd ed.). Cambridge: Cambridge University Press. pp. 651–655.ISBN 0-521-43064-X.
  6. ^Myung, I.J. (2003). "Tutorial on maximum likelihood Estimation".Journal of Mathematical Psychology.47 (1):90–100.doi:10.1016/S0022-2496(02)00028-7.
  7. ^Gourieroux, Christian; Monfort, Alain (1995).Statistics and Econometrics Models. Cambridge University Press. p. 161.ISBN 0-521-40551-3.
  8. ^Kane, Edward J. (1968).Economic Statistics and Econometrics. New York, NY: Harper & Row. p. 179.
  9. ^Small, Christoper G.; Wang, Jinfang (2003)."Working with roots".Numerical Methods for Nonlinear Estimating Equations. Oxford University Press. pp. 74–124.ISBN 0-19-850688-0.
  10. ^Kass, Robert E.; Vos, Paul W. (1997).Geometrical Foundations of Asymptotic Inference. New York, NY: John Wiley & Sons. p. 14.ISBN 0-471-82668-5.
  11. ^Papadopoulos, Alecos (25 September 2013)."Why we always put log() before the joint pdf when we use MLE (Maximum likelihood Estimation)?".Stack Exchange.
  12. ^abSilvey, S. D. (1975).Statistical Inference. London, UK: Chapman and Hall. p. 79.ISBN 0-412-13820-4.
  13. ^Olive, David (2004)."Does the MLE maximize the likelihood?"(PDF).Southern Illinois University.
  14. ^Schwallie, Daniel P. (1985). "Positive definite maximum likelihood covariance estimators".Economics Letters.17 (1–2):115–117.doi:10.1016/0165-1765(85)90139-9.
  15. ^Magnus, Jan R. (2017).Introduction to the Theory of Econometrics. Amsterdam: VU University Press. pp. 64–65.ISBN 978-90-8659-766-6.
  16. ^Pfanzagl (1994, p. 206)
  17. ^By Theorem 2.5 inNewey, Whitney K.;McFadden, Daniel (1994). "Chapter 36: Large sample estimation and hypothesis testing". In Engle, Robert; McFadden, Dan (eds.).Handbook of Econometrics, Vol.4. Elsevier Science. pp. 2111–2245.ISBN 978-0-444-88766-5.
  18. ^abBy Theorem 3.3 inNewey, Whitney K.;McFadden, Daniel (1994). "Chapter 36: Large sample estimation and hypothesis testing". In Engle, Robert; McFadden, Dan (eds.).Handbook of Econometrics, Vol.4. Elsevier Science. pp. 2111–2245.ISBN 978-0-444-88766-5.
  19. ^Zacks, Shelemyahu (1971).The Theory of Statistical Inference. New York: John Wiley & Sons. p. 223.ISBN 0-471-98103-6.
  20. ^See formula 20 inCox, David R.;Snell, E. Joyce (1968). "A general definition of residuals".Journal of the Royal Statistical Society, Series B.30 (2):248–275.JSTOR 2984505.
  21. ^Kano, Yutaka (1996)."Third-order efficiency implies fourth-order efficiency".Journal of the Japan Statistical Society.26:101–117.doi:10.14490/jjss1995.26.101.
  22. ^Christensen, Henrikt I."Pattern Recognition"(PDF) (lecture). Bayesian Decision Theory - CS 7616. Georgia Tech.
  23. ^cmplx96 (https://stats.stackexchange.com/users/177679/cmplx96), Kullback–Leibler divergence, URL (version: 2017-11-18):https://stats.stackexchange.com/q/314472 (at the youtube video, look at minutes 13 to 25)
  24. ^Introduction to Statistical Inference | Stanford (Lecture 16 — MLE under model misspecification)
  25. ^Sycorax says Reinstate Monica (https://stats.stackexchange.com/users/22311/sycorax-says-reinstate-monica), the relationship between maximizing the likelihood and minimizing the cross-entropy, URL (version: 2019-11-06):https://stats.stackexchange.com/q/364237
  26. ^Fletcher, R. (1987).Practical Methods of Optimization (Second ed.). New York, NY: John Wiley & Sons.ISBN 0-471-91547-5.
  27. ^Nocedal, Jorge; Wright, Stephen J. (2006).Numerical Optimization (Second ed.). New York, NY: Springer.ISBN 0-387-30303-0.
  28. ^Daganzo, Carlos (1979).Multinomial Probit: The Theory and its Application to Demand Forecasting. New York: Academic Press. pp. 61–78.ISBN 0-12-201150-3.
  29. ^Gould, William; Pitblado, Jeffrey; Poi, Brian (2010).Maximum Likelihood Estimation with Stata (Fourth ed.). College Station: Stata Press. pp. 13–20.ISBN 978-1-59718-078-8.
  30. ^Murphy, Kevin P. (2012).Machine Learning: A Probabilistic Perspective. Cambridge: MIT Press. p. 247.ISBN 978-0-262-01802-9.
  31. ^Amemiya, Takeshi (1985).Advanced Econometrics. Cambridge: Harvard University Press. pp. 137–138.ISBN 0-674-00560-0.
  32. ^Sargan, Denis (1988). "Methods of Numerical Optimization".Lecture Notes on Advanced Econometric Theory. Oxford: Basil Blackwell. pp. 161–169.ISBN 0-631-14956-2.
  33. ^See theorem 10.1 inAvriel, Mordecai (1976).Nonlinear Programming: Analysis and Methods. Englewood Cliffs, NJ: Prentice-Hall. pp. 293–294.ISBN 978-0-486-43227-4.
  34. ^Gill, Philip E.; Murray, Walter;Wright, Margaret H. (1981).Practical Optimization. London, UK: Academic Press. pp. 312–313.ISBN 0-12-283950-1.
  35. ^Edgeworth, Francis Y. (Sep 1908)."On the probable errors of frequency-constants".Journal of the Royal Statistical Society.71 (3):499–512.doi:10.2307/2339293.JSTOR 2339293.
  36. ^Edgeworth, Francis Y. (Dec 1908)."On the probable errors of frequency-constants".Journal of the Royal Statistical Society.71 (4):651–678.doi:10.2307/2339378.JSTOR 2339378.
  37. ^Pfanzagl, Johann (1994).Parametric Statistical Theory.Walter de Gruyter. pp. 207–208.doi:10.1515/9783110889765.ISBN 978-3-11-013863-4.MR 1291393.
  38. ^Hald, Anders (1999)."On the History of Maximum Likelihood in Relation to Inverse Probability and Least Squares".Statistical Science.14 (2):214–222.ISSN 0883-4237.
  39. ^Wilks, S.S. (1938)."The large-sample distribution of the likelihood ratio for testing composite hypotheses".Annals of Mathematical Statistics.9:60–62.doi:10.1214/aoms/1177732360.
  40. ^Owen, Art B. (2001).Empirical Likelihood. London, UK; Boca Raton, FL: Chapman & Hall; CRC Press.ISBN 978-1-58488-071-4.
  41. ^Wilks, Samuel S. (1962).Mathematical Statistics. New York, NY: John Wiley & Sons.ISBN 978-0-471-94650-2.
  42. ^Savage, Leonard J. (1976)."On rereading R.A. Fisher".The Annals of Statistics.4 (3):441–500.doi:10.1214/aos/1176343456.JSTOR 2958221.
  43. ^Pratt, John W. (1976)."F. Y. Edgeworth and R. A. Fisher on the efficiency of maximum likelihood estimation".The Annals of Statistics.4 (3):501–514.doi:10.1214/aos/1176343457.JSTOR 2958222.
  44. ^Stigler, Stephen M. (1978). "Francis Ysidro Edgeworth, statistician".Journal of the Royal Statistical Society, Series A.141 (3):287–322.doi:10.2307/2344804.JSTOR 2344804.
  45. ^Stigler, Stephen M. (1986).The history of statistics: the measurement of uncertainty before 1900. Harvard University Press.ISBN 978-0-674-40340-6.
  46. ^Stigler, Stephen M. (1999).Statistics on the table: the history of statistical concepts and methods. Harvard University Press.ISBN 978-0-674-83601-3.
  47. ^Hald, Anders (1998).A history of mathematical statistics from 1750 to 1930. New York, NY: Wiley.ISBN 978-0-471-17912-2.
  48. ^Hald, Anders (1999)."On the history of maximum likelihood in relation to inverse probability and least squares".Statistical Science.14 (2):214–222.doi:10.1214/ss/1009212248.JSTOR 2676741.
  49. ^Aldrich, John (1997)."R.A. Fisher and the making of maximum likelihood 1912–1922".Statistical Science.12 (3):162–176.doi:10.1214/ss/1030037906.MR 1617519.

Further reading

[edit]

External links

[edit]
Continuous data
Center
Dispersion
Shape
Count data
Summary tables
Dependence
Graphics
Study design
Survey methodology
Controlled experiments
Adaptive designs
Observational studies
Statistical theory
Frequentist inference
Point estimation
Interval estimation
Testing hypotheses
Parametric tests
Specific tests
Goodness of fit
Rank statistics
Bayesian inference
Correlation
Regression analysis
Linear regression
Non-standard predictors
Generalized linear model
Partition of variance
Categorical
Multivariate
Time-series
General
Specific tests
Time domain
Frequency domain
Survival
Survival function
Hazard function
Test
Biostatistics
Engineering statistics
Social statistics
Spatial statistics
Retrieved from "https://en.wikipedia.org/w/index.php?title=Maximum_likelihood_estimation&oldid=1281914241"
Categories:
Hidden categories:
[8]ページ先頭
©2009-2025 Movatter.jp