| Part of a series on |
| Regression analysis |
|---|
| Models |
| Estimation |
| Background |
Inregression analysis,least squares is a method to determine the best-fitmodel by minimizing the sum of the squaredresiduals—the differences between observed values and the values predicted by the model.
Least squares problems fall into two categories: linear orordinary least squares andnonlinear least squares, depending on whether or not the model functions are linear in all unknowns. The linear least-squares problem occurs in statisticalregression analysis; it has aclosed-form solution. The nonlinear problem is usually solved byiterative refinement; at each iteration the system is approximated by a linear one, and thus the core calculation is similar in both cases.
Polynomial least squares describes thevariance in a prediction of the dependent variable as a function of the independent variable and thedeviations from the fitted curve.
When the observations come from anexponential family with identity as its natural sufficient statistics and mild-conditions are satisfied (e.g. fornormal,exponential,Poisson andbinomial distributions), standardized least-squares estimates andmaximum-likelihood estimates are identical.[1] The method of least squares can also be derived as amethod of moments estimator.
The method was the culmination of several advances that took place during the course of the eighteenth century:[2]
The first clear and concise exposition of the method of least squares was published byLegendre in 1805.[6] The technique is described as an algebraic procedure for fitting linear equations to data and Legendre demonstrates the new method by analyzing the same data as Laplace for the shape of the Earth. Within ten years after Legendre's publication, the method of least squares had been adopted as a standard tool in astronomy andgeodesy inFrance,Italy, andPrussia, which constitutes an extraordinarily rapid acceptance of a scientific technique.[2]
In 1809Carl Friedrich Gauss published his method of calculating the orbits of celestial bodies. In that work he claimed to have been in possession of the method of least squares since 1795.[7] This naturally led to a priority dispute with Legendre. However, to Gauss's credit, he went beyond Legendre and succeeded in connecting the method of least squares with the principles of probability and to thenormal distribution. He had managed to complete Laplace's program of specifying a mathematical form of the probability density for the observations, depending on a finite number of unknown parameters, and define a method of estimation that minimizes the error of estimation. Gauss showed that thearithmetic mean is indeed the best estimate of the location parameter by changing both theprobability density and the method of estimation. He then turned the problem around by asking what form the density should have and what method of estimation should be used to get the arithmetic mean as estimate of the location parameter. In this attempt, he invented the normal distribution.
An early demonstration of the strength ofGauss's method came when it was used to predict the future location of the newly discovered asteroidCeres. On 1 January 1801, the Italian astronomerGiuseppe Piazzi discovered Ceres and was able to track its path for 40 days before it was lost in the glare of the Sun. Based on these data, astronomers desired to determine the location of Ceres after it emerged from behind the Sun without solvingKepler's complicated nonlinear equations of planetary motion. The only predictions that successfully allowed Hungarian astronomerFranz Xaver von Zach to relocate Ceres were those performed by the 24-year-old Gauss using least-squares analysis.
In 1810, after reading Gauss's work, Laplace, after proving thecentral limit theorem, used it to give a large sample justification for the method of least squares and the normal distribution. In 1822, Gauss was able to state that the least-squares approach to regression analysis is optimal in the sense that in a linear model where the errors have a mean of zero, are uncorrelated, normally distributed, and have equal variances, the best linear unbiased estimator of the coefficients is the least-squares estimator. An extended version of this result is known as theGauss–Markov theorem.
The idea of least-squares analysis was also independently formulated by the AmericanRobert Adrain in 1808. In the next two centuries workers in the theory of errors and in statistics found many different ways of implementing least squares.[8]
The objective consists of adjusting the parameters of a model function to best fit a data set. A simple data set consists ofn points (data pairs),i = 1, …,n, where is anindependent variable and is adependent variable whose value is found by observation. The model function has the form, wherem adjustable parameters are held in thevector. The goal is to find the parameter values for the model that "best" fits the data. The fit of a model to a data point is measured by itsresidual, defined as the difference between the observed value of the dependent variable and the value predicted by the model:
The least-squares method finds the optimal parameter values by minimizing thesum of squared residuals,:[9]
In the simplest case, and the result of the least-squares method is thearithmetic mean of the input data.
An example of a model in two dimensions is that of the straight line. Denoting they-intercept as and the slope as, the model function is given by. Seelinear least squares for a fully worked out example of this model.
A data point may consist of more than one independent variable. For example, when fitting a plane to a set of height measurements, the plane is a function of two independent variables,x andz, say. In the most general case there may be one or more independent variables and one or more dependent variables at each data point.
To the right is a residual plot illustrating random fluctuations about, indicating that a linear model is appropriate. is an independent, random variable.[9]
If the residual points had some sort of a shape and were not randomly fluctuating, a linear model would not be appropriate. For example, if the residual plot had a parabolic shape as seen to the right, a parabolic model would be appropriate for the data. The residuals for a parabolic model can be calculated via.[9]
This regression formulation considers only observational errors in the dependent variable (but the alternativetotal least squares regression can account for errors in both variables). There are two rather different contexts with different implications:
Theminimum of the sum of squares is found by setting thegradient to zero. Since the model containsm parameters, there arem gradient equations:and since, the gradient equations become
The gradient equations apply to all least squares problems. Each particular problem requires particular expressions for the model and itspartial derivatives.[11]
A regression model is a linear one when the model comprises alinear combination of the parameters, i.e.,where the function is a function of.[11]
Letting and putting the independent and dependent variables in matrices and respectively, we can compute the least squares in the following way. Note that is the set of all data.[11][12]
The gradient of the loss is:
Setting the gradient of the loss to zero and solving for, we get:[12][11]
There is, in some cases, aclosed-form solution to a non-linear least squares problem – but in general there is not. In the case of no closed-form solution, numerical algorithms are used to find the value of the parameters that minimizes the objective. Most algorithms involve choosing initial values for the parameters. Then, the parameters are refined iteratively, that is, the values are obtained by successive approximation:where a superscriptk is an iteration number, and the vector of increments is called the shift vector. In some commonly used algorithms, at each iteration the model may be linearized by approximation to a first-orderTaylor series expansion about:
TheJacobianJ is a function of constants, the independent variableand the parameters, so it changes from one iteration to the next. The residuals are given by
To minimize the sum of squares of, the gradient equation is set to zero and solved for:which, on rearrangement, becomem simultaneous linear equations, thenormal equations:
The normal equations are written in matrix notation as
These are the defining equations of theGauss–Newton algorithm.
These differences must be considered whenever the solution to a nonlinear least squares problem is being sought.[11]
Consider a simple example drawn from physics. A spring should obeyHooke's law which states that the extension of a springy is proportional to the force,F, applied to it.constitutes the model, whereF is the independent variable. In order to estimate theforce constant,k, we conduct a series ofn measurements with different forces to produce a set of data,, whereyi is a measured spring extension.[13] Each experimental observation will contain some error,, and so we may specify an empirical model for our observations,
There are many methods we might use to estimate the unknown parameterk. Since then equations in them variables in our data comprise anoverdetermined system with one unknown andn equations, we estimatek using least squares. The sum of squares to be minimized is[11]
The least squares estimate of the force constant,k, is given by
We assume that applying forcecauses the spring to expand. After having derived the force constant by least squares fitting, we predict the extension from Hooke's law.
In a least squares calculation with unit weights, or in linear regression, the variance on thejth parameter, denoted, is usually estimated withwhere the true error varianceσ2 is replaced by an estimate, thereduced chi-squared statistic, based on the minimized value of theresidual sum of squares (objective function),S. The denominator,n − m, is thestatistical degrees of freedom; seeeffective degrees of freedom for generalizations.[11]C is thecovariance matrix.
If theprobability distribution of the parameters is known or an asymptotic approximation is made,confidence limits can be found. Similarly, statistical tests on the residuals can be conducted if the probability distribution of the residuals is known or assumed. We can derive the probability distribution of any linear combination of the dependent variables if the probability distribution of experimental errors is known or assumed. Inferring is easy when assuming that the errors follow a normal distribution, consequently implying that the parameter estimates and residuals will also be normally distributed conditional on the values of the independent variables.[11]
It is necessary to make assumptions about the nature of the experimental errors to test the results statistically. A common assumption is that the errors belong to a normal distribution. Thecentral limit theorem supports the idea that this is a good approximation in many cases.
However, suppose the errors are not normally distributed. In that case, acentral limit theorem often nonetheless implies that the parameter estimates will be approximately normally distributed so long as the sample is reasonably large. For this reason, given the important property that the error mean is independent of the independent variables, the distribution of the error term is not an important issue in regression analysis. Specifically, it is not typically important whether the error term follows a normal distribution.
A special case ofgeneralized least squares calledweighted least squares occurs when all the off-diagonal entries of Ω (the correlation matrix of the residuals) are null; thevariances of the observations (along the covariance matrix diagonal) may still be unequal (heteroscedasticity). In simpler terms,heteroscedasticity is when the variance of depends on the value of which causes the residual plot to create a "fanning out" effect towards larger or smaller values as seen in the residual plot to the right. On the other hand,homoscedasticity is assuming that the variance of and variance of are equal.[9]
The firstprincipal component about the mean of a set of points can be represented by that line which most closely approaches the data points (as measured by squared distance of closest approach, i.e. perpendicular to the line). In contrast, linear least squares tries to minimize the distance in the direction only. Thus, although the two use a similar error metric, linear least squares is a method that treats one dimension of the data preferentially, while PCA treats all dimensions equally.
Notable statisticianSara van de Geer usedempirical process theory and theVapnik–Chervonenkis dimension to prove a least-squares estimator can be interpreted as ameasure on the space ofsquare-integrable functions.[15]
In some contexts, aregularized version of the least squares solution may be preferable.Tikhonov regularization (orridge regression) adds a constraint that, the squared-norm of the parameter vector, is not greater than a given value to the least squares formulation, leading to a constrained minimization problem. This is equivalent to the unconstrained minimization problem where the objective function is the residual sum of squares plus a penalty term and is a tuning parameter (this is theLagrangian form of the constrained minimization problem).[16]
In aBayesian context, this is equivalent to placing a zero-mean normally distributedprior on the parameter vector.
An alternativeregularized version of least squares isLasso (least absolute shrinkage and selection operator), which uses the constraint that, theL1-norm of the parameter vector, is no greater than a given value.[17][18][19] (One can show like above using Lagrange multipliers that this is equivalent to an unconstrained minimization of the least-squares penalty with added.) In aBayesian context, this is equivalent to placing a zero-meanLaplaceprior distribution on the parameter vector.[20] The optimization problem may be solved usingquadratic programming or more generalconvex optimization methods, as well as by specific algorithms such as theleast angle regression algorithm.
One of the prime differences between Lasso and ridge regression is that in ridge regression, as the penalty is increased, all parameters are reduced while still remaining non-zero, while in Lasso, increasing the penalty will cause more and more of the parameters to be driven to zero. This is an advantage of Lasso over ridge regression, as driving parameters to zero deselects the features from the regression. Thus, Lasso automatically selects more relevant features and discards the others, whereas Ridge regression never fully discards any features. Somefeature selection techniques are developed based on the LASSO including Bolasso which bootstraps samples,[21] and FeaLect which analyzes the regression coefficients corresponding to different values of to score all the features.[22]
TheL1-regularized formulation is useful in some contexts due to its tendency to prefer solutions where more parameters are zero, which gives solutions that depend on fewer variables.[17] For this reason, the Lasso and its variants are fundamental to the field ofcompressed sensing. An extension of this approach iselastic net regularization.
{{cite book}}: CS1 maint: location (link) CS1 maint: location missing publisher (link) CS1 maint: multiple names: authors list (link) CS1 maint: numeric names: authors list (link)
Media related toLeast squares at Wikimedia Commons| Computational statistics | |||||||||
|---|---|---|---|---|---|---|---|---|---|
| Correlation and dependence | |||||||||
| Regression analysis | |||||||||
| Regression as a statistical model |
| ||||||||
| Decomposition of variance | |||||||||
| Model exploration | |||||||||
| Background | |||||||||
| Design of experiments | |||||||||
| Numericalapproximation | |||||||||
| Applications | |||||||||
