Instatistics,ordinary least squares (OLS) is a type oflinear least squares method for choosing the unknownparameters in alinear regression model (with fixed level-one^{[clarification needed]} effects of alinear function of a set ofexplanatory variables) by the principle ofleast squares: minimizing the sum of the squares of the differences between the observeddependent variable (values of the variable being observed) in the inputdataset and the output of the (linear) function of theindependent variable. Some sources consider OLS to be linear regression.^[1]

Geometrically, this is seen as the sum of the squared distances, parallel to the axis of the dependent variable, between each data point in the set and the corresponding point on the regression surface—the smaller the differences, the better the model fits the data. The resultingestimator can be expressed by a simple formula, especially in the case of asimple linear regression, in which there is a singleregressor on the right side of the regression equation.

The OLS estimator isconsistent for the level-one fixed effects when the regressors areexogenous and forms perfectcolinearity (rank condition), consistent for the variance estimate of the residuals when regressors have finite fourth moments^[2] and—by theGauss–Markov theorem—optimal in the class of linear unbiased estimators when theerrors arehomoscedastic andserially uncorrelated. Under these conditions, the method of OLS providesminimum-variance mean-unbiased estimation when the errors have finitevariances. Under the additional assumption that the errors arenormally distributed with zero mean, OLS is themaximum likelihood estimator that outperforms any non-linear unbiased estimator.

Suppose the data consists of${\displaystyle n}$observations${\displaystyle {\left\{{\mathbf{x}}_i,y_i\right\}}_{i=1}^n}$. Each observation${\displaystyle i}$ includes a scalar response${\displaystyle y_i}$ and a column vector${\displaystyle {\mathbf{x}}_i}$ of${\displaystyle p}$ parameters (regressors), i.e.,${\displaystyle {\mathbf{x}}_i={\left[x_{i1},x_{i2},\dots ,x_{ip}\right]}^{\mathrm{T}}}$. In alinear regression model, the response variable,${\displaystyle y_i}$, is a linear function of the regressors:

${\displaystyle y_i={\beta }_1{x}_{i1}+{\beta }_2{x}_{i2}+\cdots +{\beta }_p{x}_{ip}+{\epsilon }_i,}$

or invector form,

${\displaystyle y_i={\mathbf{x}}_i^{\mathrm{T}}\mathit{\beta }+{\epsilon }_i,}$

where${\displaystyle {\mathbf{x}}_i}$, as introduced previously, is a column vector of the${\displaystyle i}$-th observation of all the explanatory variables;${\displaystyle \mathit{\beta }}$ is a${\displaystyle p\times 1}$ vector of unknown parameters; and the scalar${\displaystyle {\epsilon }_i}$ represents unobserved random variables (errors) of the${\displaystyle i}$-th observation.${\displaystyle {\epsilon }_i}$ accounts for the influences upon the responses${\displaystyle y_i}$ from sources other than the explanatory variables${\displaystyle {\mathbf{x}}_i}$. This model can also be written in matrix notation as

${\displaystyle \mathbf{y}=\mathbf{X}\mathit{\beta }+\mathit{\epsilon },}$

where${\displaystyle \mathbf{y}}$ and${\displaystyle \mathit{\epsilon }}$ are${\displaystyle n\times 1}$ vectors of the response variables and the errors of the${\displaystyle n}$ observations, and${\displaystyle \mathbf{X}}$ is an${\displaystyle n\times p}$ matrix of regressors, also sometimes called thedesign matrix, whose row${\displaystyle i}$ is${\displaystyle {\mathbf{x}}_i^{\mathrm{T}}}$ and contains the${\displaystyle i}$-th observations on all the explanatory variables.

Typically, a constant term is included in the set of regressors${\displaystyle \mathbf{X}}$, say, by taking${\displaystyle x_{i1}=1}$ for all${\displaystyle i=1,\dots ,n}$. The coefficient${\displaystyle {\beta }_1}$ corresponding to this regressor is called theintercept. Without the intercept, the fitted line is forced to cross the origin when${\displaystyle x_i=\overrightarrow{0}}$.

Regressors do not have to be independent for estimation to be consistent e.g. they may be non-linearly dependent. Short of perfect multicollinearity, parameter estimates may still be consistent; however, as multicollinearity rises the standard error around such estimates increases and reduces the precision of such estimates. When there is perfect multicollinearity, it is no longer possible to obtain unique estimates for the coefficients to the related regressors; estimation for these parameters cannot converge (thus, it cannot be consistent).

As a concrete example where regressors are non-linearly dependent yet estimation may still be consistent, we might suspect the response depends linearly both on a value and its square; in which case we would include one regressor whose value is just the square of another regressor. In that case, the model would bequadratic in the second regressor, but none-the-less is still considered alinear model because the modelis still linear in the parameters (${\displaystyle \mathit{\beta }}$).

Consider anoverdetermined system

${\displaystyle \sum _{j=1}^p{x}_{ij}{\beta }_j=y_i,(i=1,2,\dots ,n),}$

of${\displaystyle n}$linear equations in${\displaystyle p}$ unknowncoefficients,${\displaystyle {\beta }_1,{\beta }_2,\dots ,{\beta }_p}$, with${\displaystyle n>p}$. This can be written inmatrix form as

${\displaystyle \mathbf{X}\mathit{\beta }=\mathbf{y},}$

where

(Note: for a linear model as above, not all elements in${\displaystyle \mathbf{X}}$ contains information on the data points. The first column is populated with ones,${\displaystyle X_{i1}=1}$. Only the other columns contain actual data. So here${\displaystyle p}$ is equal to the number of regressors plus one).

Such a system usually has no exact solution, so the goal is instead to find the coefficients${\displaystyle \mathit{\beta }}$ which fit the equations "best", in the sense of solving thequadraticminimization problem

${\displaystyle \hat{\mathit{\beta }}=\underset{\mathit{\beta }}{\mathrm{a}\mathrm{r}\mathrm{g}\mathrm{m}\mathrm{i}\mathrm{n}}S(\mathit{\beta }),}$

where the objective function${\displaystyle S}$ is given by

${\displaystyle S(\mathit{\beta })=\sum _{i=1}^n{\left|y_i-\sum _{j=1}^p{X}_{ij}{\beta }_j\right|}^2={\Vert \mathbf{y}-\mathbf{X}\mathit{\beta }\Vert }^2.}$

A justification for choosing this criterion is given inProperties below. This minimization problem has a unique solution, provided that the${\displaystyle p}$ columns of the matrix${\displaystyle \mathbf{X}}$ arelinearly independent, given by solving the so-callednormal equations:

${\displaystyle \left({\mathbf{X}}^{\mathrm{T}}\mathbf{X}\right)\hat{\mathit{\beta }}={\mathbf{X}}^{\mathrm{T}}\mathbf{y}.}$

The matrix${\displaystyle {\mathbf{X}}^{\mathrm{T}}\mathbf{X}}$ is known as thenormal matrix orGram matrix and the matrix${\displaystyle {\mathbf{X}}^{\mathrm{T}}\mathbf{y}}$ is known as themoment matrix of regressand by regressors.^[3] Finally,${\displaystyle \hat{\mathit{\beta }}}$ is the coefficient vector of the least-squareshyperplane, expressed as

${\displaystyle \hat{\mathit{\beta }}={\left({\mathbf{X}}^{\mathrm{\top }}\mathbf{X}\right)}^{-1}{\mathbf{X}}^{\mathrm{\top }}\mathbf{y}.}$

or

${\displaystyle \hat{\mathit{\beta }}=\mathit{\beta }+{\left({\mathbf{X}}^{\mathrm{\top }}\mathbf{X}\right)}^{-1}{\mathbf{X}}^{\mathrm{\top }}\mathit{\epsilon }.}$

Supposeb is a "candidate" value for the parameter vectorβ. The quantityy_i −x_i^Tb, called theresidual for thei-th observation, measures the vertical distance between the data point(x_i,y_i) and the hyperplaney =x^Tb, and thus assesses the degree of fit between the actual data and the model. Thesum of squared residuals (SSR) (also called theerror sum of squares (ESS) orresidual sum of squares (RSS))^[4] is a measure of the overall model fit:

${\displaystyle S(b)=\sum _{i=1}^n(y_i-x_i^{\mathrm{T}}b{)}^2=(y-Xb{)}^{\mathrm{T}}(y-Xb),}$

whereT denotes the matrixtranspose, and the rows ofX, denoting the values of all the independent variables associated with a particular value of the dependent variable, areX_i = x_i^T. The value ofb which minimizes this sum is called theOLS estimator forβ. The functionS(b) is quadratic inb with positive-definiteHessian, and therefore this function possesses a unique global minimum at${\displaystyle b=\hat{\beta }}$, which can be given by the explicit formula^[5][proof]

${\displaystyle \hat{\beta }={\mathrm{argmin}}_{b\in {\mathbb{R}}^p}S(b)=(X^{\mathrm{T}}X{)}^{-1}{X}^{\mathrm{T}}y.}$

The productN =X^TX is aGram matrix, and its inverse,Q =N⁻¹, is thecofactor matrix ofβ,^[6][7][8] closely related to itscovariance matrix,C_β.The matrix (X^T X)⁻¹ X^T =Q X^T is called theMoore–Penrose pseudoinverse matrix ofX. This formulation highlights the point that estimation can be carried out if, and only if, there is no perfectmulticollinearity between the explanatory variables (which would cause the Gram matrix to have no inverse).

After we have estimatedβ, thefitted values (orpredicted values) from the regression will be

${\displaystyle \hat{y}=X\hat{\beta }=Py,}$

whereP =X(X^TX)⁻¹X^T is theprojection matrix onto the spaceV spanned by the columns ofX. This matrixP is also sometimes called thehat matrix because it "puts a hat" onto the variabley. Another matrix, closely related toP is theannihilator matrixM =I_n −P; this is a projection matrix onto the space orthogonal toV. Both matricesP andM aresymmetric andidempotent (meaning thatP² =P andM² =M), and relate to the data matrixX via identitiesPX =X andMX = 0.^[9] MatrixM creates theresiduals from the regression:

${\displaystyle \hat{\epsilon }=y-\hat{y}=y-X\hat{\beta }=My=M(X\beta +\epsilon )=(MX)\beta +M\epsilon =M\epsilon .}$

The variances of the predicted values${\displaystyle s_{{\hat{y}}_i}^2}$ are found in the main diagonal of thevariance-covariance matrix of predicted values:

${\displaystyle C_{\hat{y}}=s^{2}P,}$

whereP is the projection matrix ands² is the sample variance.^[10]The full matrix is very large; its diagonal elements can be calculated individually as:

${\displaystyle s_{{\hat{y}}_i}^2=s^2{X}_i(X^{T}X{)}^{-1}{X}_i^T,}$

whereX_i is thei-th row of matrixX.

Using these residuals we can estimate the sample variances² using thereduced chi-squared statistic:

${\displaystyle s^2=\frac{{\hat{\epsilon }}^{\mathrm{T}}\hat{\epsilon }}{n-p}=\frac{(My{)}^{\mathrm{T}}My}{n-p}=\frac{y^{\mathrm{T}}{M}^{\mathrm{T}}My}{n-p}=\frac{y^{\mathrm{T}}My}{n-p}=\frac{S(\hat{\beta })}{n-p},{\hat{\sigma }}^2=\frac{n-p}{n}{s}^2}$

The denominator,n−p, is thestatistical degrees of freedom. The first quantity,s², is the OLS estimate forσ², whereas the second,${\displaystyle {\hat{\sigma }}^2}$, is the MLE estimate forσ². The two estimators are quite similar in large samples; the first estimator is alwaysunbiased, while the second estimator is biased but has a smallermean squared error. In practices² is used more often, since it is more convenient for the hypothesis testing. The square root ofs² is called theregression standard error,^[11]standard error of the regression,^[12][13] orstandard error of the equation.^[9]

It is common to assess the goodness-of-fit of the OLS regression by comparing how much the initial variation in the sample can be reduced by regressing ontoX. Thecoefficient of determinationR² is defined as a ratio of "explained" variance to the "total" variance of the dependent variabley, in the cases where the regression sum of squares equals the sum of squares of residuals:^[14]

${\displaystyle R^2=\frac{\sum ({\hat{y}}_i-\overline{y}{)}^2}{\sum (y_i-\overline{y}{)}^2}=\frac{y^{\mathrm{T}}{P}^{\mathrm{T}}LPy}{y^{\mathrm{T}}Ly}=1-\frac{y^{\mathrm{T}}My}{y^{\mathrm{T}}Ly}=1-\frac{\mathrm{R}\mathrm{S}\mathrm{S}}{\mathrm{T}\mathrm{S}\mathrm{S}}}$

where TSS is thetotal sum of squares for the dependent variable,$L=I_n-\frac{1}{n}{J}_n$, and$J_n$ is ann×n matrix of ones. (${\displaystyle L}$ is acentering matrix which is equivalent to regression on a constant; it simply subtracts the mean from a variable.) In order forR² to be meaningful, the matrixX of data on regressors must contain a column vector of ones to represent the constant whose coefficient is the regression intercept. In that case,R² will always be a number between 0 and 1, with values close to 1 indicating a good degree of fit.

If the data matrixX contains only two variables, a constant and a scalar regressorx_i, then this is called the "simple regression model". This case is often considered in the beginner statistics classes, as it provides much simpler formulas even suitable for manual calculation. The parameters are commonly denoted as(α,β):

${\displaystyle y_i=\alpha +\beta x_i+{\epsilon }_i.}$

The least squares estimates in this case are given by simple formulas

In the previous section the least squares estimator${\displaystyle \hat{\beta }}$ was obtained as a value that minimizes the sum of squared residuals of the model. However it is also possible to derive the same estimator from other approaches. In all cases the formula for OLS estimator remains the same:^{^}β = (X^TX)⁻¹X^Ty; the only difference is in how we interpret this result.

This sectionmay need to be cleaned up. It has beenmerged fromLinear least squares.

For mathematicians, OLS is an approximate solution to an overdetermined system of linear equationsXβ ≈y, whereβ is the unknown. Assuming the system cannot be solved exactly (the number of equationsn is much larger than the number of unknownsp), we are looking for a solution that could provide the smallest discrepancy between the right- and left- hand sides. In other words, we are looking for the solution that satisfies

${\displaystyle \hat{\beta }=\mathrm{a}\mathrm{r}\mathrm{g}\underset{\beta }{min}\Vert \mathbf{y}-\mathbf{X}\mathit{\beta }{\Vert }^2,}$

where‖·‖ is the standardL² norm in then-dimensionalEuclidean spaceRⁿ. The predicted quantityXβ is just a certain linear combination of the vectors of regressors. Thus, the residual vectory −Xβ will have the smallest length wheny isprojected orthogonally onto thelinear subspacespanned by the columns ofX. The OLS estimator${\displaystyle \hat{\beta }}$ in this case can be interpreted as the coefficients ofvector decomposition of^{^}y =Py along the basis ofX.

In other words, the gradient equations at the minimum can be written as:

${\displaystyle (\mathbf{y}-\mathbf{X}\hat{\mathit{\beta }}{)}^{\mathrm{\top }}\mathbf{X}=0.}$

A geometrical interpretation of these equations is that the vector of residuals,${\displaystyle \mathbf{y}-X\hat{\mathit{\beta }}}$ is orthogonal to thecolumn space ofX, since thedot product${\displaystyle (\mathbf{y}-\mathbf{X}\hat{\mathit{\beta }})\cdot \mathbf{X}\mathbf{v}}$ is equal to zero forany conformal vector,v. This means that${\displaystyle \mathbf{y}-\mathbf{X}\hat{\mathit{\beta }}}$ is the shortest of all possible vectors${\displaystyle \mathbf{y}-\mathbf{X}\mathit{\beta }}$, that is, the variance of the residuals is the minimum possible. This is illustrated at the right.

Introducing${\displaystyle \hat{\mathit{\gamma }}}$ and a matrixK with the assumption that a matrix${\displaystyle [\mathbf{X}\mathbf{K}]}$ is non-singular andK^TX = 0 (cf.Orthogonal projections), the residual vector should satisfy the following equation:

${\displaystyle \hat{\mathbf{r}}:=\mathbf{y}-\mathbf{X}\hat{\mathit{\beta }}=\mathbf{K}\hat{\mathit{\gamma }}.}$

The equation and solution of linear least squares are thus described as follows:

Another way of looking at it is to consider the regression line to be a weighted average of the lines passing through the combination of any two points in the dataset.^[15] Although this way of calculation is more computationally expensive, it provides a better intuition on OLS.

The OLS estimator is identical to themaximum likelihood estimator (MLE) under the normality assumption for the error terms.^[16][proof] This normality assumption has historical importance, as it provided the basis for the early work in linear regression analysis byYule andPearson.^{[citation needed]} From the properties of MLE, we can infer that the OLS estimator is asymptotically efficient (in the sense of attaining theCramér–Rao bound for variance) if the normality assumption is satisfied.^[17]

Iniid case the OLS estimator can also be viewed as aGMM estimator arising from the moment conditions

${\displaystyle \mathrm{E}[x_i\left(y_i-x_i^{\mathrm{T}}\beta \right)]=0.}$

These moment conditions state that the regressors should be uncorrelated with the errors. Sincex_i is ap-vector, the number of moment conditions is equal to the dimension of the parameter vectorβ, and thus the system is exactly identified. This is the so-called classical GMM case, when the estimator does not depend on the choice of the weighting matrix.

Note that the original strict exogeneity assumptionE[ε_i | x_i] = 0 implies a far richer set of moment conditions than stated above. In particular, this assumption implies that for any vector-functionƒ, the moment conditionE[ƒ(x_i)·ε_i] = 0 will hold. However it can be shown using theGauss–Markov theorem that the optimal choice of functionƒ is to takeƒ(x) =x, which results in the moment equation posted above.

There are several different frameworks in which thelinear regression model can be cast in order to make the OLS technique applicable. Each of these settings produces the same formulas and same results. The only difference is the interpretation and the assumptions which have to be imposed in order for the method to give meaningful results. The choice of the applicable framework depends mostly on the nature of data in hand, and on the inference task which has to be performed.

One of the lines of difference in interpretation is whether to treat the regressors as random variables, or as predefined constants. In the first case (random design) the regressorsx_i are random and sampled together with they_i's from somepopulation, as in anobservational study. This approach allows for more natural study of theasymptotic properties of the estimators. In the other interpretation (fixed design), the regressorsX are treated as known constants set by adesign, andy is sampled conditionally on the values ofX as in anexperiment. For practical purposes, this distinction is often unimportant, since estimation and inference is carried out while conditioning onX. All results stated in this article are within the random design framework.

The classical model focuses on the "finite sample" estimation and inference, meaning that the number of observationsn is fixed. This contrasts with the other approaches, which study theasymptotic behavior of OLS, and in which the behavior at a large number of samples is studied.To prove finite sample unbiasedness of the OLS estimator, we require the following assumptions.

• Exogeneity. The regressors do notcovary with the error term:$${\displaystyle \mathbb{E}[{\epsilon }_i{x}_i]=0.}$$ This requires, for example, that there are noomitted variables that covary with observed variables and affect the response variable. An alternative (but stronger) statement that is often required when explaining linear regression inmathematical statistics is that the predictor variablesx can be treated as fixed values, rather thanrandom variables. This stronger form means, for example, that the predictor variables are assumed to be error-free, that is, not contaminated with measurement error. Although this assumption is not realistic in many settings, dropping it leads to more complexerrors-in-variables models,instrumental variable models and the like.
• Linearity, orcorrect specification. This means that the mean of the response variable is alinear combination of the parameters (regression coefficients) and the predictor variables. Note that this assumption is much less restrictive than it may at first seem. Because the predictor variables are treated as fixed values (see above), linearity is really only a restriction on the parameters. The predictor variables themselves can be arbitrarily transformed, and in fact multiple copies of the same underlying predictor variable can be added, each one transformed differently. This technique is used, for example, inpolynomial regression, which uses linear regression to fit the response variable as an arbitrarypolynomial function (up to a given degree) of a predictor variable. With this much flexibility, models such as polynomial regression often have "too much power", in that they tend tooverfit the data. As a result, some kind ofregularization must typically be used to prevent unreasonable solutions coming out of the estimation process. Common examples areridge regression andlasso regression.Bayesian linear regression can also be used, which by its nature is more or less immune to the problem of overfitting. (In fact,ridge regression andlasso regression can both be viewed as special cases of Bayesian linear regression, with particular types ofprior distributions placed on the regression coefficients.)

• 

  Constant variance orhomoscedasticity. This means that the variance of the errors does not depend on the values of the predictor variables:$${\displaystyle \mathbb{E}[{\epsilon }_i^2|x_i]={\sigma }^2.}$$ Thus the variability of the responses for given fixed values of the predictors is the same regardless of how large or small the responses are. This is often not the case, as a variable whose mean is large will typically have a greater variance than one whose mean is small. For example, a person whose income is predicted to be $100,000 may easily have an actual income of $80,000 or $120,000—i.e., astandard deviation of around $20,000—while another person with a predicted income of $10,000 is unlikely to have the same $20,000 standard deviation, since that would imply their actual income could vary anywhere between −$10,000 and $30,000. (In fact, as this shows, in many cases—often the same cases where the assumption of normally distributed errors fails—the variance or standard deviation should be predicted to be proportional to the mean, rather than constant.) The absence of homoscedasticity is calledheteroscedasticity. In order to check this assumption, a plot of residuals versus predicted values (or the values of each individual predictor) can be examined for a "fanning effect" (i.e., increasing or decreasing vertical spread as one moves left to right on the plot). A plot of the absolute or squared residuals versus the predicted values (or each predictor) can also be examined for a trend or curvature. Formal tests can also be used; seeHeteroscedasticity. The presence of heteroscedasticity will result in an overall "average" estimate of variance being used instead of one that takes into account the true variance structure. This leads to less precise (but in the case ofordinary least squares, not biased) parameter estimates and biased standard errors, resulting in misleading tests and interval estimates. Themean squared error for the model will also be wrong. Various estimation techniques includingweighted least squares and the use ofheteroscedasticity-consistent standard errors can handle heteroscedasticity in a quite general way.Bayesian linear regression techniques can also be used when the variance is assumed to be a function of the mean. It is also possible in some cases to fix the problem by applying a transformation to the response variable (e.g., fitting thelogarithm of the response variable using a linear regression model, which implies that the response variable itself has alog-normal distribution rather than anormal distribution).

• Uncorrelatedness of errors. This assumes that the errors of the response variables are uncorrelated with each other:$${\displaystyle \mathbb{E}[{\epsilon }_i{\epsilon }_j|x_i,x_j]=0.}$$ Some methods such asgeneralized least squares are capable of handling correlated errors, although they typically require significantly more data unless some sort ofregularization is used to bias the model towards assuming uncorrelated errors.Bayesian linear regression is a general way of handling this issue. Fullstatistical independence is a stronger condition than mere lack of correlation and is often not needed, although it implies mean-independence.
• Lack of perfect multicollinearity in the predictors. For standardleast squares estimation methods, the design matrixX must have fullcolumn rankp:^[18]$${\displaystyle Pr[\mathrm{rank}(X)=p]=1.}$$ If this assumption is violated, perfectmulticollinearity exists in the predictor variables, meaning a linear relationship exists between two or more predictor variables. Multicollinearity can be caused by accidentally duplicating a variable in the data, using a linear transformation of a variable along with the original (e.g., the same temperature measurements expressed in Fahrenheit and Celsius), or including a linear combination of multiple variables in the model, such as their mean. It can also happen if there is too little data available compared to the number of parameters to be estimated (e.g., fewer data points than regression coefficients). Near violations of this assumption, where predictors are highly but not perfectly correlated, can reduce the precision of parameter estimates (seeVariance inflation factor). In the case of perfect multicollinearity, the parameter vectorβ will benon-identifiable—it has no unique solution. In such a case, only some of the parameters can be identified (i.e., their values can only be estimated within some linear subspace of the full parameter spaceR^p). Seepartial least squares regression. Methods for fitting linear models with multicollinearity have been developed,^{[19][20][21][22]} some of which require additional assumptions such as "effect sparsity"—that a large fraction of the effects are exactly zero. Note that the more computationally expensive iterated algorithms for parameter estimation, such as those used ingeneralized linear models, do not suffer from this problem.

Violations of these assumptions can result in biased estimations ofβ, biased standard errors, untrustworthy confidence intervals and significance tests. Beyond these assumptions, several other statistical properties of the data strongly influence the performance of different estimation methods:

• The statistical relationship between the error terms and the regressors plays an important role in determining whether an estimation procedure has desirable sampling properties such as being unbiased and consistent.
• The arrangement, orprobability distribution of the predictor variablesx has a major influence on the precision of estimates ofβ.Sampling anddesign of experiments are highly developed subfields of statistics that provide guidance for collecting data in such a way to achieve a precise estimate ofβ.

First of all, under thestrict exogeneity assumption the OLS estimators${\displaystyle \hat{\beta }}$ ands² areunbiased, meaning that their expected values coincide with the true values of the parameters:^[23][proof]

${\displaystyle \mathrm{E}[\hat{\beta }\mid X]=\beta ,\mathrm{E}[s^2\mid X]={\sigma }^2.}$

If the strict exogeneity does not hold (as is the case with manytime series models, where exogeneity is assumed only with respect to the past shocks but not the future ones), then these estimators will be biased in finite samples.

Thevariance-covariance matrix (or simplycovariance matrix) of${\displaystyle \hat{\beta }}$ is equal to^[24]

${\displaystyle \mathrm{Var}[\hat{\beta }\mid X]={\sigma }^2{\left(X^{\mathrm{T}}X\right)}^{-1}={\sigma }^{2}Q.}$

In particular, the standard error of each coefficient${\displaystyle {\hat{\beta }}_j}$ is equal to square root of thej-th diagonal element of this matrix. The estimate of this standard error is obtained by replacing the unknown quantityσ² with its estimates². Thus,

${\displaystyle \widehat{\mathrm{s}.\mathrm{e}.}({\hat{\beta }}_j)=\sqrt{s^2{\left(X^{\mathrm{T}}X\right)}_{jj}^{-1}}}$

It can also be easily shown that the estimator${\displaystyle \hat{\beta }}$ is uncorrelated with the residuals from the model:^[24]

${\displaystyle \mathrm{Cov}[\hat{\beta },\hat{\epsilon }\mid X]=0.}$

TheGauss–Markov theorem states that under thespherical errors assumption (that is, the errors should beuncorrelated andhomoscedastic) the estimator${\displaystyle \hat{\beta }}$ is efficient in the class of linear unbiased estimators. This is called thebest linear unbiased estimator (BLUE). Efficiency should be understood as if we were to find some other estimator${\displaystyle \tilde{\beta }}$ which would be linear iny and unbiased, then^[24]

${\displaystyle \mathrm{Var}[\tilde{\beta }\mid X]-\mathrm{Var}[\hat{\beta }\mid X]\ge 0}$

in the sense that this is anonnegative-definite matrix. This theorem establishes optimality only in the class of linear unbiased estimators, which is quite restrictive. Depending on the distribution of the error termsε, other, non-linear estimators may provide better results than OLS.

The properties listed so far are all valid regardless of the underlying distribution of the error terms. However, if you are willing to assume that thenormality assumption holds (that is, thatε ~N(0,σ²I_n)), then additional properties of the OLS estimators can be stated.

The estimator${\displaystyle \hat{\beta }}$ is normally distributed, with mean and variance as given before:^[25]

${\displaystyle \hat{\beta }\sim \mathcal{N}(\beta ,{\sigma }^2(X^{\mathrm{T}}X{)}^{-1}).}$

This estimator reaches theCramér–Rao bound for the model, and thus is optimal in the class of all unbiased estimators.^[17] Note that unlike theGauss–Markov theorem, this result establishes optimality among both linear and non-linear estimators, but only in the case of normally distributed error terms.

The estimators² will be proportional to thechi-squared distribution:^[26]

${\displaystyle s^2\sim \frac{{\sigma }^2}{n-p}\cdot {\chi }_{n-p}^2}$

The variance of this estimator is equal to2σ⁴/(n − p), which does not attain theCramér–Rao bound of2σ⁴/n. However it was shown that there are no unbiased estimators ofσ² with variance smaller than that of the estimators².^[27] If we are willing to allow biased estimators, and consider the class of estimators that are proportional to the sum of squared residuals (SSR) of the model, then the best (in the sense of themean squared error) estimator in this class will be^~σ² = SSR / (n − p + 2), which even beats the Cramér–Rao bound in case when there is only one regressor (p = 1).^[28]

Moreover, the estimators${\displaystyle \hat{\beta }}$ ands² areindependent,^[29] the fact which comes in useful when constructing the t- and F-tests for the regression.

As was mentioned before, the estimator${\displaystyle \hat{\beta }}$ is linear iny, meaning that it represents a linear combination of the dependent variablesy_i. The weights in this linear combination are functions of the regressorsX, and generally are unequal. The observations with high weights are calledinfluential because they have a more pronounced effect on the value of the estimator.

To analyze which observations are influential we remove a specificj-th observation and consider how much the estimated quantities are going to change (similarly to thejackknife method). It can be shown that the change in the OLS estimator forβ will be equal to^[30]

${\displaystyle {\hat{\beta }}^{(j)}-\hat{\beta }=-\frac{1}{1-h_j}(X^{\mathrm{T}}X{)}^{-1}{x}_j^{\mathrm{T}}{\hat{\epsilon }}_j,}$

whereh_j =x_j^T (X^TX)⁻¹x_j is thej-th diagonal element of the hat matrixP, andx_j is the vector of regressors corresponding to thej-th observation. Similarly, the change in the predicted value forj-th observation resulting from omitting that observation from the dataset will be equal to^[30]

${\displaystyle {\hat{y}}_j^{(j)}-{\hat{y}}_j=x_j^{\mathrm{T}}{\hat{\beta }}^{(j)}-x_j^{\mathrm{T}}\hat{\beta }=-\frac{h_j}{1-h_j}{\hat{\epsilon }}_j}$

From the properties of the hat matrix,0 ≤h_j ≤ 1, and they sum up top, so that on averageh_j ≈p/n. These quantitiesh_j are called theleverages, and observations with highh_j are calledleverage points.^[31] Usually the observations with high leverage ought to be scrutinized more carefully, in case they are erroneous, or outliers, or in some other way atypical of the rest of the dataset.

Sometimes the variables and corresponding parameters in the regression can be logically split into two groups, so that the regression takes form

${\displaystyle y=X_1{\beta }_1+X_2{\beta }_2+\epsilon ,}$

whereX₁ andX₂ have dimensionsn×p₁,n×p₂, andβ₁,β₂ arep₁×1 andp₂×1 vectors, withp₁ +p₂ =p.

TheFrisch–Waugh–Lovell theorem states that in this regression the residuals${\displaystyle \hat{\epsilon }}$ and the OLS estimate${\displaystyle {\hat{\beta }}_2}$ will be numerically identical to the residuals and the OLS estimate forβ₂ in the following regression:^[32]

${\displaystyle M_{1}y=M_1{X}_2{\beta }_2+\eta ,}$

whereM₁ is theannihilator matrix for regressorsX₁.

The theorem can be used to establish a number of theoretical results. For example, having a regression with a constant and another regressor is equivalent to subtracting the means from the dependent variable and the regressor and then running the regression for the de-meaned variables but without the constant term.

The least squares estimators arepoint estimates of the linear regression model parametersβ. However, generally we also want to know how close those estimates might be to the true values of parameters. In other words, we want to construct theinterval estimates.

Since we have not made any assumption about the distribution of error termε_i, it is impossible to infer the distribution of the estimators${\displaystyle \hat{\beta }}$ and${\displaystyle {\hat{\sigma }}^2}$. Nevertheless, we can apply thecentral limit theorem to derive theirasymptotic properties as sample sizen goes to infinity. While the sample size is necessarily finite, it is customary to assume thatn is "large enough" so that the true distribution of the OLS estimator is close to its asymptotic limit.

We can show that under the model assumptions, the least squares estimator forβ isconsistent (that is${\displaystyle \hat{\beta }}$converges in probability toβ) and asymptotically normal:^[proof]

${\displaystyle (\hat{\beta }-\beta )\stackrel{d}{\to }\mathcal{N}(0,{\sigma }^2{Q}_{xx}^{-1}),}$

where${\displaystyle Q_{xx}=X^{\mathrm{T}}X.}$

Using this asymptotic distribution, approximate two-sided confidence intervals for thej-th component of the vector${\displaystyle \hat{\beta }}$ can be constructed as

${\displaystyle {\beta }_j\in [{\hat{\beta }}_j\pm q_{1-\frac{\alpha }{2}}^{\mathcal{N}(0,1)}\sqrt{{\hat{\sigma }}^2{\left[Q_{xx}^{-1}\right]}_{jj}}]}$   at the1 − α confidence level,

whereq denotes thequantile function of standard normal distribution, and [·]_jj is thej-th diagonal element of a matrix.

Similarly, the least squares estimator forσ² is also consistent and asymptotically normal (provided that the fourth moment ofε_i exists) with limiting distribution

${\displaystyle ({\hat{\sigma }}^2-{\sigma }^2)\stackrel{d}{\to }\mathcal{N}\left(0,\mathrm{E}\left[{\epsilon }_i^4\right]-{\sigma }^4\right).}$

These asymptotic distributions can be used for prediction, testing hypotheses, constructing other estimators, etc.. As an example consider the problem of prediction. Suppose${\displaystyle x_0}$ is some point within the domain of distribution of the regressors, and one wants to know what the response variable would have been at that point. Themean response is the quantity${\displaystyle y_0=x_0^{\mathrm{T}}\beta }$, whereas thepredicted response is${\displaystyle {\hat{y}}_0=x_0^{\mathrm{T}}\hat{\beta }}$. Clearly the predicted response is a random variable, its distribution can be derived from that of${\displaystyle \hat{\beta }}$:

${\displaystyle \left({\hat{y}}_0-y_0\right)\stackrel{d}{\to }\mathcal{N}\left(0,{\sigma }^2{x}_0^{\mathrm{T}}{Q}_{xx}^{-1}{x}_0\right),}$

which allows construct confidence intervals for mean response${\displaystyle y_0}$ to be constructed:

${\displaystyle y_0\in \left[x_0^{\mathrm{T}}\hat{\beta }\pm q_{1-\frac{\alpha }{2}}^{\mathcal{N}(0,1)}\sqrt{{\hat{\sigma }}^2{x}_0^{\mathrm{T}}{Q}_{xx}^{-1}{x}_0}\right]}$   at the1 − α confidence level.

This sectionneeds expansion. You can help byadding missing information.(February 2017)

Two hypothesis tests are particularly widely used. First, one wants to know if the estimated regression equation is any better than simply predicting that all values of the response variable equal its sample mean (if not, it is said to have no explanatory power). Thenull hypothesis of no explanatory value of the estimated regression is tested using anF-test. If the calculated F-value is found to be large enough to exceed its critical value for the pre-chosen level of significance, the null hypothesis is rejected and thealternative hypothesis, that the regression has explanatory power, is accepted. Otherwise, the null hypothesis of no explanatory power is accepted.

Second, for each explanatory variable of interest, one wants to know whether its estimated coefficient differs significantly from zero—that is, whether this particular explanatory variable in fact has explanatory power in predicting the response variable. Here the null hypothesis is that the true coefficient is zero. This hypothesis is tested by computing the coefficient'st-statistic, as the ratio of the coefficient estimate to itsstandard error. If the t-statistic is larger than a predetermined value, the null hypothesis is rejected and the variable is found to have explanatory power, with its coefficient significantly different from zero. Otherwise, the null hypothesis of a zero value of the true coefficient is accepted.

In addition, theChow test is used to test whether two subsamples both have the same underlying true coefficient values. The sum of squared residuals of regressions on each of the subsets and on the combined data set are compared by computing an F-statistic; if this exceeds a critical value, the null hypothesis of no difference between the two subsets is rejected; otherwise, it is accepted.

In atime series model, we require thestochastic process {x_i,y_i} to bestationary andergodic; if {x_i,y_i} is nonstationary, OLS results are often biased unless {x_i,y_i} isco-integrating.^[33]

We still require the regressors to bestrictly exogenous: E[x_iε_i] = 0 for alli = 1, ...,n. If they are onlypredetermined, OLS is biased in finite sample;

Finally, the assumptions on the variance take the form of requiring that {x_iε_i} is amartingale difference sequence, with a finite matrix of second momentsQ_xxε² = E[ ε_i²x_i x_i^T ].

Suppose it is known that the coefficients in the regression satisfy a system of linear equations

${\displaystyle A:Q^{\mathrm{T}}\beta =c,}$

whereQ is ap×q matrix of full rank, andc is aq×1 vector of known constants, whereq < p. In this case least squares estimation is equivalent to minimizing the sum of squared residuals of the model subject to the constraintA. Theconstrained least squares (CLS) estimator can be given by an explicit formula:^[34]

${\displaystyle {\hat{\beta }}^c=\hat{\beta }-(X^{\mathrm{T}}X{)}^{-1}Q(Q^{\mathrm{T}}(X^{\mathrm{T}}X{)}^{-1}Q{)}^{-1}(Q^{\mathrm{T}}\hat{\beta }-c).}$

This expression for the constrained estimator is valid as long as the matrixX^TX is invertible. It was assumed from the beginning of this article that this matrix is of full rank, and it was noted that when the rank condition fails,β will not be identifiable. However it may happen that adding the restrictionA makesβ identifiable, in which case one would like to find the formula for the estimator. The estimator is equal to^[35]

${\displaystyle {\hat{\beta }}^c=R(R^{\mathrm{T}}{X}^{\mathrm{T}}XR{)}^{-1}{R}^{\mathrm{T}}{X}^{\mathrm{T}}y+(I_p-R(R^{\mathrm{T}}{X}^{\mathrm{T}}XR{)}^{-1}{R}^{\mathrm{T}}{X}^{\mathrm{T}}X)Q(Q^{\mathrm{T}}Q{)}^{-1}c,}$

whereR is ap×(p − q) matrix such that the matrix[Q R] is non-singular, andR^TQ = 0. Such a matrix can always be found, although generally it is not unique. The second formula coincides with the first in case whenX^TX is invertible.^[35]

The following data set gives average heights and weights for American women aged 30–39 (source:The World Almanac and Book of Facts, 1975).

Height (m)	1.47	1.50	1.52	1.55	1.57
Weight (kg)	52.21	53.12	54.48	55.84	57.20
Height (m)	1.60	1.63	1.65	1.68	1.70
Weight (kg)	58.57	59.93	61.29	63.11	64.47
Height (m)	1.73	1.75	1.78	1.80	1.83
Weight (kg)	66.28	68.10	69.92	72.19	74.46

When only one dependent variable is being modeled, ascatterplot will suggest the form and strength of the relationship between the dependent variable and regressors. It might also reveal outliers, heteroscedasticity, and other aspects of the data that may complicate the interpretation of a fitted regression model. The scatterplot suggests that the relationship is strong and can be approximated as a quadratic function. OLS can handle non-linear relationships by introducing the regressorHEIGHT². The regression model then becomes a multiple linear model:

${\displaystyle w_i={\beta }_1+{\beta }_2{h}_i+{\beta }_3{h}_i^2+{\epsilon }_i.}$

The output from most popularstatistical packages will look similar to this:

Method	Least squares
Dependent variable	WEIGHT
Observations	15
Parameter	Value	Std error	t-statistic	p-value
${\displaystyle {\beta }_1}$	128.8128	16.3083	7.8986	0.0000
${\displaystyle {\beta }_2}$	−143.1620	19.8332	−7.2183	0.0000
${\displaystyle {\beta }_3}$	61.9603	6.0084	10.3122	0.0000
R2	0.9989	S.E. of regression		0.2516
Adjusted R2	0.9987	Model sum-of-sq.		692.61
Log-likelihood	1.0890	Residual sum-of-sq.		0.7595
Durbin–Watson stat.	2.1013	Total sum-of-sq.		693.37
Akaike criterion	0.2548	F-statistic		5471.2
Schwarz criterion	0.3964	p-value (F-stat)		0.0000

In this table:

• TheValue column gives the least squares estimates of parametersβ_j
• TheStd error column showsstandard errors of each coefficient estimate:${\displaystyle {\hat{\sigma }}_j={\left({\hat{\sigma }}^2{\left[Q_{xx}^{-1}\right]}_{jj}\right)}^{\frac{1}{2}}}$
• Thet-statistic andp-value columns are testing whether any of the coefficients might be equal to zero. Thet-statistic is calculated simply as${\displaystyle t={\hat{\beta }}_j/{\hat{\sigma }}_j}$. If the errors ε follow a normal distribution,t follows a Student-t distribution. Under weaker conditions,t is asymptotically normal. Large values oft indicate that the null hypothesis can be rejected and that the corresponding coefficient is not zero. The second column,p-value, expresses the results of the hypothesis test as asignificance level. Conventionally,p-values smaller than 0.05 are taken as evidence that the population coefficient is nonzero.
• R-squared is thecoefficient of determination indicating goodness-of-fit of the regression. This statistic will be equal to one if fit is perfect, and to zero when regressorsX have no explanatory power whatsoever. This is a biased estimate of the populationR-squared, and will never decrease if additional regressors are added, even if they are irrelevant.
• Adjusted R-squared is a slightly modified version of${\displaystyle R^2}$, designed to penalize for the excess number of regressors which do not add to the explanatory power of the regression. This statistic is always smaller than${\displaystyle R^2}$, can decrease as new regressors are added, and even be negative for poorly fitting models:

${\displaystyle {\overline{R}}^2=1-\frac{n-1}{n-p}(1-R^2)}$

• Log-likelihood is calculated under the assumption that errors follow normal distribution. Even though the assumption is not very reasonable, this statistic may still find its use in conducting LR tests.
• Durbin–Watson statistic tests whether there is any evidence of serial correlation between the residuals. As a rule of thumb, the value smaller than 2 will be an evidence of positive correlation.
• Akaike information criterion andSchwarz criterion are both used for model selection. Generally when comparing two alternative models, smaller values of one of these criteria will indicate a better model.^[36]
• Standard error of regression is an estimate ofσ, standard error of the error term.
• Total sum of squares,model sum of squared, andresidual sum of squares tell us how much of the initial variation in the sample were explained by the regression.
• F-statistic tries to test the hypothesis that all coefficients (except the intercept) are equal to zero. This statistic hasF(p–1,n–p) distribution under the null hypothesis and normality assumption, and itsp-value indicates probability that the hypothesis is indeed true. Note that when errors are not normal this statistic becomes invalid, and other tests such asWald test orLR test should be used.

Ordinary least squares analysis often includes the use of diagnostic plots designed to detect departures of the data from the assumed form of the model. These are some of the common diagnostic plots:

• Residuals against the explanatory variables in the model. A non-linear relation between these variables suggests that the linearity of the conditional mean function may not hold. Different levels of variability in the residuals for different levels of the explanatory variables suggests possible heteroscedasticity.
• Residuals against explanatory variables not in the model. Any relation of the residuals to these variables would suggest considering these variables for inclusion in the model.
• Residuals against the fitted values,${\displaystyle \hat{y}}$.
• Residuals against the preceding residual. This plot may identify serial correlations in the residuals.

An important consideration when carrying out statistical inference using regression models is how the data were sampled. In this example, the data are averages rather than measurements on individual women. The fit of the model is very good, but this does not imply that the weight of an individual woman can be predicted with high accuracy based only on her height.

This example also demonstrates that coefficients determined by these calculations are sensitive to how the data is prepared. The heights were originally given rounded to the nearest inch and have been converted and rounded to the nearest centimetre. Since the conversion factor is one inch to 2.54 cm this isnot an exact conversion. The original inches can be recovered by Round(x/0.0254) and then re-converted to metric without rounding. If this is done the results become:

Using either of these equations to predict the weight of a 5' 6" (1.6764 m) woman gives similar values: 62.94 kg with rounding vs. 62.98 kg without rounding. Thus a seemingly small variation in the data has a real effect on the coefficients but a small effect on the results of the equation.

While this may look innocuous in the middle of the data range it could become significant at the extremes or in the case where the fitted model is used to project outside the data range (extrapolation).

This highlights a common error: this example is an abuse of OLS which inherently requires that the errors in the independent variable (in this case height) are zero or at least negligible. The initial rounding to nearest inch plus any actual measurement errors constitute a finite and non-negligible error. As a result, the fitted parameters are not the best estimates they are presumed to be. Though not totally spurious the error in the estimation will depend upon relative size of thex andy errors.

We can use the least square mechanism to figure out the equation of a two body orbit in polar base co-ordinates. The equation typically used is${\displaystyle r(\theta )=\frac{p}{1-e\cos (\theta )}}$ where${\displaystyle r(\theta )}$ is the radius of how far the object is from one of the bodies. In the equation the parameters${\displaystyle p}$ and${\displaystyle e}$ are used to determine the path of the orbit. We have measured the following data.

${\displaystyle \theta }$ (in degrees)	43	45	52	93	108	116
${\displaystyle r(\theta )}$	4.7126	4.5542	4.0419	2.2187	1.8910	1.7599

We need to find the least-squares approximation of${\displaystyle e}$ and${\displaystyle p}$ for the given data.

First we need to represent e and p in a linear form. So we are going to rewrite the equation${\displaystyle r(\theta )}$ as${\displaystyle \frac{1}{r(\theta )}=\frac{1}{p}-\frac{e}{p}\cos (\theta )}$.

Furthermore, one could fit forapsides by expanding${\displaystyle \cos (\theta )}$ with an extra parameter as${\displaystyle \cos (\theta -{\theta }_0)=\cos (\theta )\cos ({\theta }_0)+\sin (\theta )\sin ({\theta }_0)}$, which is linear in both${\displaystyle \cos (\theta )}$ and in the extra basis function${\displaystyle \sin (\theta )}$.

We use the original two-parameter form to represent our observational data as:

${\displaystyle A^{T}A(\genfrac{}{}{0ex}{}{x}{y})=A^{T}b,}$

where:

${\displaystyle x=1/p}$;${\displaystyle y=e/p}$;${\displaystyle A}$ contains the coefficients of${\displaystyle 1/p}$ in the first column, which are all 1, and the coefficients of${\displaystyle e/p}$ in the second column, given by${\displaystyle \cos (\theta )}$; and${\displaystyle b=1/r(\theta )}$, such that:

On solving we get${\displaystyle (\genfrac{}{}{0ex}{}{x}{y})=(\genfrac{}{}{0ex}{}{0.43478}{0.30435})}$,

so${\displaystyle p=\frac{1}{x}=2.3000}$ and${\displaystyle e=p\cdot y=0.70001}$

• Bayesian least squares
• Fama–MacBeth regression
• Nonlinear least squares
• Numerical methods for linear least squares
• Nonlinear system identification

• Dougherty, Christopher (2002).Introduction to Econometrics (2nd ed.). New York: Oxford University Press. pp. 48–113.ISBN 0-19-877643-8.
• Gujarati, Damodar N.;Porter, Dawn C. (2009).Basic Econometics (Fifth ed.). Boston: McGraw-Hill Irwin. pp. 55–96.ISBN 978-0-07-337577-9.
• Heij, Christiaan; Boer, Paul;Franses, Philip H.;Kloek, Teun;van Dijk, Herman K. (2004).Econometric Methods with Applications in Business and Economics (1st ed.). Oxford: Oxford University Press. pp. 76–115.ISBN 978-0-19-926801-6.
• Hill, R. Carter; Griffiths, William E.; Lim, Guay C. (2008).Principles of Econometrics (3rd ed.). Hoboken, NJ: John Wiley & Sons. pp. 8–47.ISBN 978-0-471-72360-8.
• Wooldridge, Jeffrey (2008)."The Simple Regression Model".Introductory Econometrics: A Modern Approach (4th ed.). Mason, OH: Cengage Learning. pp. 22–67.ISBN 978-0-324-58162-1.

• Parametric statistics
• Least squares

• CS1 errors: ISBN date
• Articles with short description
• Short description matches Wikidata
• Wikipedia articles needing clarification from December 2023
• All articles with unsourced statements
• Articles with unsourced statements from February 2010
• Articles to be expanded from February 2017
• All articles to be expanded