Movatterモバイル変換

[0]ホーム

Jump to content

Partition of sums of squares

Italiano

Edit links

From Wikipedia, the free encyclopedia

(Redirected fromSum of squares (statistics))

Concept that permeates much of inferential statistics and descriptive statistics

This article is about the partition of sums of squares in statistics. For other uses, seeSum of squares.

For broader coverage of this topic, seeAnalysis of variance.

"Variance partitioning" redirects here; not to be confused withVariance decomposition.

Thepartition of sums of squares is a concept that permeates much ofinferential statistics anddescriptive statistics. More properly, it is thepartitioning of sums ofsquared deviations or errors. Mathematically, the sum of squared deviations is an unscaled, or unadjusted measure ofdispersion (also calledvariability). When scaled for the number ofdegrees of freedom, it estimates thevariance, or spread of the observations about their mean value. Partitioning of the sum of squared deviations into various components allows the overall variability in a dataset to be ascribed to different types or sources of variability, with the relative importance of each being quantified by the size of each component of the overall sum of squares.

Background

[edit]

The distance from any point in a collection of data, to the mean of the data, is the deviation. This can be written as $y_{i}-{\overline {y}}$ , where $y_{i}$ is the ith data point, and ${\overline {y}}$ is the estimate of the mean. If all such deviations are squared, then summed, as in $\sum _{i=1}^{n}\left(y_{i}-{\overline {y}}\,\right)^{2}$ , this gives the "sum of squares" for these data.

When more data are added to the collection the sum of squares will increase, except in unlikely cases such as the new data being equal to the mean. So usually, the sum of squares will grow with the size of the data collection. That is a manifestation of the fact that it is unscaled.

In many cases, the number ofdegrees of freedom is simply the number of data points in the collection, minus one. We write this asn − 1, wheren is the number of data points.

Scaling (also known as normalizing) means adjusting the sum of squares so that it does not grow as the size of the data collection grows. This is important when we want to compare samples of different sizes, such as a sample of 100 people compared to a sample of 20 people. If the sum of squares were not normalized, its value would always be larger for the sample of 100 people than for the sample of 20 people. To scale the sum of squares, we divide it by the degrees of freedom, i.e., calculate the sum of squares per degree of freedom, or variance.Standard deviation, in turn, is thesquare root of the variance.

The above describes how the sum of squares is used in descriptive statistics; see the article ontotal sum of squares for an application of this broad principle toinferential statistics.

Partitioning the sum of squares in linear regression

[edit]

Theorem. Given alinear regression model $y_{i}=\beta _{0}+\beta _{1}x_{i1}+\cdots +\beta _{p}x_{ip}+\varepsilon _{i}$ including a constant $\beta _{0}$ , based on a sample $(y_{i},x_{i1},\ldots ,x_{ip}),\,i=1,\ldots ,n$ containingn observations, the total sum of squares $\mathrm {TSS} =\sum _{i=1}^{n}(y_{i}-{\bar {y}})^{2}$ can be partitioned as follows into theexplained sum of squares (ESS) and theresidual sum of squares (RSS):

\mathrm {TSS} =\mathrm {ESS} +\mathrm {RSS} ,

where this equation is equivalent to each of the following forms:

{\begin{aligned}\left\|y-{\bar {y}}\mathbf {1} \right\|^{2}&=\left\|{\hat {y}}-{\bar {y}}\mathbf {1} \right\|^{2}+\left\|{\hat {\varepsilon }}\right\|^{2},\quad \mathbf {1} =(1,1,\ldots ,1)^{T},\\\sum _{i=1}^{n}(y_{i}-{\bar {y}})^{2}&=\sum _{i=1}^{n}({\hat {y}}_{i}-{\bar {y}})^{2}+\sum _{i=1}^{n}(y_{i}-{\hat {y}}_{i})^{2},\\\sum _{i=1}^{n}(y_{i}-{\bar {y}})^{2}&=\sum _{i=1}^{n}({\hat {y}}_{i}-{\bar {y}})^{2}+\sum _{i=1}^{n}{\hat {\varepsilon }}_{i}^{2},\\\end{aligned}}

where

{\hat {y}}_{i}

is the value estimated by the regression line having

{\hat {b}}_{0}

{\hat {b}}_{1}

, ...,

{\hat {b}}_{p}

as the estimatedcoefficients.^[1]

Proof

[edit]

{\begin{aligned}\sum _{i=1}^{n}(y_{i}-{\overline {y}})^{2}&=\sum _{i=1}^{n}(y_{i}-{\overline {y}}+{\hat {y}}_{i}-{\hat {y}}_{i})^{2}=\sum _{i=1}^{n}(({\hat {y}}_{i}-{\bar {y}})+\underbrace {(y_{i}-{\hat {y}}_{i})} _{{\hat {\varepsilon }}_{i}})^{2}\\&=\sum _{i=1}^{n}(({\hat {y}}_{i}-{\bar {y}})^{2}+2{\hat {\varepsilon }}_{i}({\hat {y}}_{i}-{\bar {y}})+{\hat {\varepsilon }}_{i}^{2})\\&=\sum _{i=1}^{n}({\hat {y}}_{i}-{\bar {y}})^{2}+\sum _{i=1}^{n}{\hat {\varepsilon }}_{i}^{2}+2\sum _{i=1}^{n}{\hat {\varepsilon }}_{i}({\hat {y}}_{i}-{\bar {y}})\\&=\sum _{i=1}^{n}({\hat {y}}_{i}-{\bar {y}})^{2}+\sum _{i=1}^{n}{\hat {\varepsilon }}_{i}^{2}+2\sum _{i=1}^{n}{\hat {\varepsilon }}_{i}({\hat {\beta }}_{0}+{\hat {\beta }}_{1}x_{i1}+\cdots +{\hat {\beta }}_{p}x_{ip}-{\overline {y}})\\&=\sum _{i=1}^{n}({\hat {y}}_{i}-{\bar {y}})^{2}+\sum _{i=1}^{n}{\hat {\varepsilon }}_{i}^{2}+2({\hat {\beta }}_{0}-{\overline {y}})\underbrace {\sum _{i=1}^{n}{\hat {\varepsilon }}_{i}} _{0}+2{\hat {\beta }}_{1}\underbrace {\sum _{i=1}^{n}{\hat {\varepsilon }}_{i}x_{i1}} _{0}+\cdots +2{\hat {\beta }}_{p}\underbrace {\sum _{i=1}^{n}{\hat {\varepsilon }}_{i}x_{ip}} _{0}\\&=\sum _{i=1}^{n}({\hat {y}}_{i}-{\bar {y}})^{2}+\sum _{i=1}^{n}{\hat {\varepsilon }}_{i}^{2}=\mathrm {ESS} +\mathrm {RSS} \\\end{aligned}}

The requirement that the model include a constant or equivalently that thedesign matrix contain a column of ones ensures that $\sum _{i=1}^{n}{\hat {\varepsilon }}_{i}=0$ , i.e. ${\hat {\varepsilon }}^{T}\mathbf {1} =0$ .

The proof can also be expressed in vector form, as follows:

{\begin{aligned}SS_{\text{total}}=\Vert \mathbf {y} -{\bar {y}}\mathbf {1} \Vert ^{2}&=\Vert \mathbf {y} -{\bar {y}}\mathbf {1} +\mathbf {\hat {y}} -\mathbf {\hat {y}} \Vert ^{2},\\&=\Vert \left(\mathbf {\hat {y}} -{\bar {y}}\mathbf {1} \right)+\left(\mathbf {y} -\mathbf {\hat {y}} \right)\Vert ^{2},\\&=\Vert {\mathbf {\hat {y}} -{\bar {y}}\mathbf {1} }\Vert ^{2}+\Vert {\hat {\varepsilon }}\Vert ^{2}+2{\hat {\varepsilon }}^{T}\left(\mathbf {\hat {y}} -{\bar {y}}\mathbf {1} \right),\\&=SS_{\text{regression}}+SS_{\text{error}}+2{\hat {\varepsilon }}^{T}\left(X{\hat {\beta }}-{\bar {y}}\mathbf {1} \right),\\&=SS_{\text{regression}}+SS_{\text{error}}+2\left({\hat {\varepsilon }}^{T}X\right){\hat {\beta }}-2{\bar {y}}\underbrace {{\hat {\varepsilon }}^{T}\mathbf {1} } _{0},\\&=SS_{\text{regression}}+SS_{\text{error}}.\end{aligned}}

The elimination of terms in the last line, used the fact that

{\hat {\varepsilon }}^{T}X=\left(\mathbf {y} -\mathbf {\hat {y}} \right)^{T}X=\mathbf {y} ^{T}(I-X(X^{T}X)^{-1}X^{T})^{T}X={\mathbf {y} }^{T}(X^{T}-X^{T})^{T}={\mathbf {0} }.

Further partitioning

[edit]

Note that the residual sum of squares can be further partitioned as thelack-of-fit sum of squares plus the sum of squares due to pure error.

References

[edit]

^"Sum of Squares - Definition, Formulas, Regression Analysis".Corporate Finance Institute. Retrieved2020-10-16.

Bailey, R. A. (2008).Design of Comparative Experiments. Cambridge University Press.ISBN 978-0-521-68357-9. Pre-publication chapters are available on-line.
Christensen, Ronald (2002).Plane Answers to Complex Questions: The Theory of Linear Models (Third ed.). New York: Springer.ISBN 0-387-95361-2.
Whittle, Peter (1963).Prediction and Regulation. English Universities Press.ISBN 0-8166-1147-5.{{cite book}}:ISBN / Date incompatibility (help)
Republished as:Whittle, P. (1983).Prediction and Regulation by Linear Least-Square Methods. University of Minnesota Press.ISBN 0-8166-1148-3.
Whittle, P. (20 April 2000).Probability Via Expectation (4th ed.). Springer.ISBN 0-387-98955-2.

Retrieved from "https://en.wikipedia.org/w/index.php?title=Partition_of_sums_of_squares&oldid=1305882814"

Categories:

Hidden categories:

[8]ページ先頭

Movatterモバイル変換

Background

Partitioning the sum of squares in linear regression

Proof

Further partitioning

See also

References