Instatistics,principal component regression (PCR) is aregression analysis technique that is based onprincipal component analysis (PCA). PCR is a form ofreduced rank regression.^[1] More specifically, PCR is used forestimating the unknownregression coefficients in astandard linear regression model.

In PCR, instead of regressing the dependent variable on the explanatory variables directly, theprincipal components of the explanatory variables are used asregressors. One typically uses only a subset of all the principal components for regression, making PCR a kind ofregularized procedure and also a type ofshrinkage estimator.

Often the principal components with highervariances (the ones based oneigenvectors corresponding to the highereigenvalues of thesamplevariance-covariance matrix of the explanatory variables) are selected as regressors. However, for the purpose ofpredicting the outcome, the principal components with low variances may also be important, in some cases even more important.^[2]

One major use of PCR lies in overcoming themulticollinearity problem which arises when two or more of the explanatory variables are close to beingcollinear.^[3] PCR can aptly deal with such situations by excluding some of the low-variance principal components in the regression step. In addition, by usually regressing on only a subset of all the principal components, PCR can result indimension reduction through substantially lowering the effective number of parameters characterizing the underlying model. This can be particularly useful in settings withhigh-dimensional covariates. Also, through appropriate selection of the principal components to be used for regression, PCR can lead to efficientprediction of the outcome based on the assumed model.

The PCR method may be broadly divided into three major steps:

1.${\displaystyle }$ PerformPCA on the observeddata matrix for the explanatory variables to obtain the principal components, and then (usually) select a subset, based on some appropriate criteria, of the principal components so obtained for further use.

2.${\displaystyle }$ Now regress the observed vector of outcomes on the selected principal components as covariates, usingordinary least squares regression (linear regression) to get a vector of estimated regression coefficients (withdimension equal to the number of selected principal components).

3.${\displaystyle }$ Nowtransform this vector back to the scale of the actual covariates, using the selectedPCA loadings (the eigenvectors corresponding to the selected principal components) to get thefinal PCR estimator (with dimension equal to the total number of covariates) for estimating the regression coefficients characterizing the original model.

Data representation: Let${\displaystyle {\mathbf{Y}}_{n\times 1}={\left(y_1,\dots ,y_n\right)}^T}$ denote the vector of observed outcomes and${\displaystyle {\mathbf{X}}_{n\times p}={\left({\mathbf{x}}_1,\dots ,{\mathbf{x}}_n\right)}^T}$ denote the correspondingdata matrix of observed covariates where,${\displaystyle n}$ and${\displaystyle p}$ denote the size of the observedsample and the number of covariates respectively, with${\displaystyle n\ge p}$. Each of the${\displaystyle n}$ rows of${\displaystyle \mathbf{X}}$ denotes one set of observations for the${\displaystyle p}$dimensional covariate and the respective entry of${\displaystyle \mathbf{Y}}$ denotes the corresponding observed outcome.

Data pre-processing: Assume that${\displaystyle \mathbf{Y}}$ and each of the${\displaystyle p}$ columns of${\displaystyle \mathbf{X}}$ have already beencentered so that all of them have zeroempirical means. This centering step is crucial (at least for the columns of${\displaystyle \mathbf{X}}$) since PCR involves the use of PCA on${\displaystyle \mathbf{X}}$ andPCA is sensitive tocentering of the data.

Underlying model: Following centering, the standardGauss–Markovlinear regression model for${\displaystyle \mathbf{Y}}$ on${\displaystyle \mathbf{X}}$ can be represented as:${\displaystyle \mathbf{Y}=\mathbf{X}\mathit{\beta }+\mathit{\epsilon },}$ where${\displaystyle \mathit{\beta }\in {\mathbb{R}}^p}$ denotes the unknown parameter vector of regression coefficients and${\displaystyle \mathit{\epsilon }}$ denotes the vector of random errors with${\displaystyle \mathrm{E}\left(\mathit{\epsilon }\right)=\mathbf{0}}$ and${\displaystyle \mathrm{Var}\left(\mathit{\epsilon }\right)={\sigma }^2{I}_{n\times n}}$ for some unknownvariance parameter${\displaystyle {\sigma }^2>0}$

Objective: The primary goal is to obtain an efficientestimator${\displaystyle \hat{\mathit{\beta }}}$ for the parameter${\displaystyle \mathit{\beta }}$, based on the data. One frequently used approach for this isordinary least squares regression which, assuming${\displaystyle \mathbf{X}}$ isfull column rank, gives theunbiased estimator:${\displaystyle {\hat{\mathit{\beta }}}_{\mathrm{o}\mathrm{l}\mathrm{s}}=({\mathbf{X}}^T\mathbf{X}{)}^{-1}{\mathbf{X}}^T\mathbf{Y}}$ of${\displaystyle \mathit{\beta }}$. PCR is another technique that may be used for the same purpose of estimating${\displaystyle \mathit{\beta }}$.

PCA step: PCR starts by performing a PCA on the centered data matrix${\displaystyle \mathbf{X}}$. For this, let${\displaystyle \mathbf{X}=U\mathrm{\Delta }{V}^T}$ denote thesingular value decomposition of${\displaystyle \mathbf{X}}$ where,${\displaystyle {\mathrm{\Delta }}_{p\times p}=\mathrm{diag}\left[{\delta }_1,\dots ,{\delta }_p\right]}$ with${\displaystyle {\delta }_1\ge \cdots \ge {\delta }_p\ge 0}$ denoting the non-negativesingular values of${\displaystyle \mathbf{X}}$, while thecolumns of${\displaystyle U_{n\times p}=[{\mathbf{u}}_1,\dots ,{\mathbf{u}}_p]}$ and${\displaystyle V_{p\times p}=[{\mathbf{v}}_1,\dots ,{\mathbf{v}}_p]}$ are bothorthonormal sets of vectors denoting theleft and right singular vectors of${\displaystyle \mathbf{X}}$ respectively.

The principal components:${\displaystyle V\mathrm{\Lambda }{V}^T}$ gives aspectral decomposition of${\displaystyle {\mathbf{X}}^T\mathbf{X}}$ where${\displaystyle {\mathrm{\Lambda }}_{p\times p}=\mathrm{diag}\left[{\lambda }_1,\dots ,{\lambda }_p\right]=\mathrm{diag}\left[{\delta }_1^2,\dots ,{\delta }_p^2\right]={\mathrm{\Delta }}^2}$ with${\displaystyle {\lambda }_1\ge \cdots \ge {\lambda }_p\ge 0}$ denoting the non-negative eigenvalues (also known as theprincipal values) of${\displaystyle {\mathbf{X}}^T\mathbf{X}}$, while the columns of${\displaystyle V}$ denote the corresponding orthonormal set of eigenvectors. Then,${\displaystyle \mathbf{X}{\mathbf{v}}_j}$ and${\displaystyle {\mathbf{v}}_j}$ respectively denote the${\displaystyle j^{th}}$principal component and the${\displaystyle j^{th}}$principal component direction (orPCA loading) corresponding to the${\displaystyle j^{\text{th}}}$ largestprincipal value${\displaystyle {\lambda }_j}$ for each${\displaystyle j\in \{1,\dots ,p\}}$.

Derived covariates: For any${\displaystyle k\in \{1,\dots ,p\}}$, let${\displaystyle V_k}$ denote the${\displaystyle p\times k}$ matrix with orthonormal columns consisting of the first${\displaystyle k}$ columns of${\displaystyle V}$. Let${\displaystyle W_k=\mathbf{X}{V}_k}$${\displaystyle =[\mathbf{X}{\mathbf{v}}_1,\dots ,\mathbf{X}{\mathbf{v}}_k]}$ denote the${\displaystyle n\times k}$ matrix having the first${\displaystyle k}$ principal components as its columns.${\displaystyle W}$ may be viewed as the data matrix obtained by using thetransformed covariates${\displaystyle {\mathbf{x}}_i^k=V_k^T{\mathbf{x}}_i\in {\mathbb{R}}^k}$ instead of using the original covariates${\displaystyle {\mathbf{x}}_i\in {\mathbb{R}}^p\mathrm{\forall }1\le i\le n}$.

The PCR estimator: Let${\displaystyle {\hat{\gamma }}_k=(W_k^T{W}_k{)}^{-1}{W}_k^T\mathbf{Y}\in {\mathbb{R}}^k}$ denote the vector of estimated regression coefficients obtained byordinary least squares regression of the response vector${\displaystyle \mathbf{Y}}$ on the data matrix${\displaystyle W_k}$. Then, for any${\displaystyle k\in \{1,\dots ,p\}}$, the final PCR estimator of${\displaystyle \mathit{\beta }}$ based on using the first${\displaystyle k}$ principal components is given by:${\displaystyle {\hat{\mathit{\beta }}}_k=V_k{\hat{\gamma }}_k\in {\mathbb{R}}^p}$.

The fitting process for obtaining the PCR estimator involves regressing the response vector on the derived data matrix${\displaystyle W_k}$ which hasorthogonal columns for any${\displaystyle k\in \{1,\dots ,p\}}$ since the principal components aremutually orthogonal to each other. Thus in the regression step, performing amultiple linear regression jointly on the${\displaystyle k}$ selected principal components as covariates is equivalent to carrying out${\displaystyle k}$ independentsimple linear regressions (or univariate regressions) separately on each of the${\displaystyle k}$ selected principal components as a covariate.

When all the principal components are selected for regression so that${\displaystyle k=p}$, then the PCR estimator is equivalent to theordinary least squares estimator. Thus,${\displaystyle {\hat{\mathit{\beta }}}_p={\hat{\mathit{\beta }}}_{\mathrm{o}\mathrm{l}\mathrm{s}}}$. This is easily seen from the fact that${\displaystyle W_p=\mathbf{X}{V}_p=\mathbf{X}V}$ and also observing that${\displaystyle V}$ is anorthogonal matrix.

For any${\displaystyle k\in \{1,\dots ,p\}}$, the variance of${\displaystyle {\hat{\mathit{\beta }}}_k}$ is given by

${\displaystyle \mathrm{Var}({\hat{\mathit{\beta }}}_k)={\sigma }^2{V}_k(W_k^T{W}_k{)}^{-1}{V}_k^T={\sigma }^2{V}_k\mathrm{diag}\left({\lambda }_1^{-1},\dots ,{\lambda }_k^{-1}\right)V_k^T={\sigma }^2\underset{j=1}{\overset{k}{\phantom{\sum }\sum }}\frac{{\mathbf{v}}_j{\mathbf{v}}_j^T}{{\lambda }_j}.}$

In particular:

${\displaystyle \mathrm{Var}({\hat{\mathit{\beta }}}_p)=\mathrm{Var}({\hat{\mathit{\beta }}}_{\mathrm{o}\mathrm{l}\mathrm{s}})={\sigma }^2\underset{j=1}{\overset{p}{\phantom{\sum }\sum }}\frac{{\mathbf{v}}_j{\mathbf{v}}_j^T}{{\lambda }_j}.}$

Hence for all${\displaystyle k\in \{1,\dots ,p-1\}}$ we have:

${\displaystyle \mathrm{Var}({\hat{\mathit{\beta }}}_{\mathrm{o}\mathrm{l}\mathrm{s}})-\mathrm{Var}({\hat{\mathit{\beta }}}_k)={\sigma }^2\underset{j=k+1}{\overset{p}{\phantom{\sum }\sum }}\frac{{\mathbf{v}}_j{\mathbf{v}}_j^T}{{\lambda }_j}.}$

Thus, for all${\displaystyle k\in \{1,\dots ,p\}}$ we have:

${\displaystyle \mathrm{Var}({\hat{\mathit{\beta }}}_{\mathrm{o}\mathrm{l}\mathrm{s}})-\mathrm{Var}({\hat{\mathit{\beta }}}_k)⪰0}$

where${\displaystyle A⪰0}$ indicates that a square symmetric matrix${\displaystyle A}$ isnon-negative definite. Consequently, any givenlinear form of the PCR estimator has a lower variance compared to that of the samelinear form of the ordinary least squares estimator.

Undermulticollinearity, two or more of the covariates are highlycorrelated, so that one can be linearly predicted from the others with a non-trivial degree of accuracy. Consequently, the columns of the data matrix${\displaystyle \mathbf{X}}$ that correspond to the observations for these covariates tend to becomelinearly dependent and therefore,${\displaystyle \mathbf{X}}$ tends to becomerank deficient losing its full column rank structure. More quantitatively, one or more of the smaller eigenvalues of${\displaystyle {\mathbf{X}}^T\mathbf{X}}$ get(s) very close or become(s) exactly equal to${\displaystyle 0}$ under such situations. The variance expressions above indicate that these small eigenvalues have the maximuminflation effect on the variance of the least squares estimator, therebydestabilizing the estimator significantly when they are close to${\displaystyle 0}$. This issue can be effectively addressed through using a PCR estimator obtained by excluding the principal components corresponding to these small eigenvalues.

PCR may also be used for performingdimension reduction. To see this, let${\displaystyle L_k}$ denote any${\displaystyle p\times k}$ matrix having orthonormal columns, for any${\displaystyle k\in \{1,\dots ,p\}.}$ Suppose now that we want toapproximate each of the covariate observations${\displaystyle {\mathbf{x}}_i}$ through therank${\displaystyle k}$linear transformation${\displaystyle L_k{\mathbf{z}}_i}$ for some${\displaystyle {\mathbf{z}}_i\in {\mathbb{R}}^k(1\le i\le n)}$.

Then, it can be shown that

${\displaystyle \sum _{i=1}^n{\Vert {\mathbf{x}}_i-L_k{\mathbf{z}}_i\Vert }^2}$

is minimized at${\displaystyle L_k=V_k,}$ the matrix with the first${\displaystyle k}$ principal component directions as columns, and${\displaystyle {\mathbf{z}}_i={\mathbf{x}}_i^k=V_k^T{\mathbf{x}}_i,}$ the corresponding${\displaystyle k}$ dimensional derived covariates. Thus the${\displaystyle k}$ dimensional principal components provide the bestlinear approximation of rank${\displaystyle k}$ to the observed data matrix${\displaystyle \mathbf{X}}$.

The correspondingreconstruction error is given by:

Thus any potentialdimension reduction may be achieved by choosing${\displaystyle k}$, the number of principal components to be used, through appropriate thresholding on the cumulative sum of theeigenvalues of${\displaystyle {\mathbf{X}}^T\mathbf{X}}$. Since the smaller eigenvalues do not contribute significantly to the cumulative sum, the corresponding principal components may be continued to be dropped as long as the desired threshold limit is not exceeded. The same criteria may also be used for addressing themulticollinearity issue whereby the principal components corresponding to the smaller eigenvalues may be ignored as long as the threshold limit is maintained.

Since the PCR estimator typically uses only a subset of all the principal components for regression, it can be viewed as some sort of aregularized procedure. More specifically, for any, the PCR estimator${\displaystyle {\hat{\mathit{\beta }}}_k}$ denotes the regularized solution to the followingconstrained minimization problem:

${\displaystyle \underset{{\mathit{\beta }}_{\ast }\in {\mathbb{R}}^p}{min}{\Vert \mathbf{Y}-\mathbf{X}{\mathit{\beta }}_{\ast }\Vert }^2\text{ subject to }{\mathit{\beta }}_{\ast }\perp \{{\mathbf{v}}_{k+1},\dots ,{\mathbf{v}}_p\}.}$

The constraint may be equivalently written as:

${\displaystyle V_{(p-k)}^T{\mathit{\beta }}_{\ast }=\mathbf{0},}$

where:

${\displaystyle V_{(p-k)}={\left[{\mathbf{v}}_{k+1},\dots ,{\mathbf{v}}_p\right]}_{p\times (p-k)}.}$

Thus, when only a proper subset of all the principal components are selected for regression, the PCR estimator so obtained is based on a hard form ofregularization that constrains the resulting solution to thecolumn space of the selected principal component directions, and consequently restricts it to beorthogonal to the excluded directions.

Given the constrained minimization problem as defined above, consider the following generalized version of it:

${\displaystyle \underset{{\mathit{\beta }}_{\ast }\in {\mathbb{R}}^p}{min}\Vert \mathbf{Y}-\mathbf{X}{\mathit{\beta }}_{\ast }{\Vert }^2\text{ subject to }{L}_{(p-k)}^T{\mathit{\beta }}_{\ast }=\mathbf{0}}$

where,${\displaystyle L_{(p-k)}}$ denotes any full column rank matrix of order${\displaystyle p\times (p-k)}$ with.

Let${\displaystyle {\hat{\mathit{\beta }}}_L}$ denote the corresponding solution. Thus

${\displaystyle {\hat{\mathit{\beta }}}_L=\arg \underset{{\mathit{\beta }}_{\ast }\in {\mathbb{R}}^p}{min}\Vert \mathbf{Y}-\mathbf{X}{\mathit{\beta }}_{\ast }{\Vert }^2\text{ subject to }{L}_{(p-k)}^T{\mathit{\beta }}_{\ast }=\mathbf{0}.}$

Then the optimal choice of the restriction matrix${\displaystyle L_{(p-k)}}$ for which the corresponding estimator${\displaystyle {\hat{\mathit{\beta }}}_L}$ achieves the minimum prediction error is given by:^[4]

${\displaystyle L_{(p-k)}^{\ast }=V_{(p-k)}{\mathrm{\Lambda }}_{(p-k)}^{1/2},}$

where

${\displaystyle {\mathrm{\Lambda }}_{(p-k)}^{1/2}=\mathrm{diag}\left({\lambda }_{k+1}^{1/2},\dots ,{\lambda }_p^{1/2}\right).}$

Quite clearly, the resulting optimal estimator${\displaystyle {\hat{\mathit{\beta }}}_{L^{\ast }}}$ is then simply given by the PCR estimator${\displaystyle {\hat{\mathit{\beta }}}_k}$ based on the first${\displaystyle k}$ principal components.

Since the ordinary least squares estimator isunbiased for${\displaystyle \mathit{\beta }}$, we have

${\displaystyle \mathrm{Var}({\hat{\mathit{\beta }}}_{\mathrm{o}\mathrm{l}\mathrm{s}})=\mathrm{MSE}({\hat{\mathit{\beta }}}_{\mathrm{o}\mathrm{l}\mathrm{s}}),}$

where, MSE denotes themean squared error. Now, if for some${\displaystyle k\in \{1,\dots ,p\}}$, we additionally have:${\displaystyle V_{(p-k)}^T\mathit{\beta }=\mathbf{0}}$, then the corresponding${\displaystyle {\hat{\mathit{\beta }}}_k}$ is alsounbiased for${\displaystyle \mathit{\beta }}$ and therefore

${\displaystyle \mathrm{Var}({\hat{\mathit{\beta }}}_k)=\mathrm{MSE}({\hat{\mathit{\beta }}}_k).}$

We have already seen that

${\displaystyle \mathrm{\forall }j\in \{1,\dots ,p\}:\mathrm{Var}({\hat{\mathit{\beta }}}_{\mathrm{o}\mathrm{l}\mathrm{s}})-\mathrm{Var}({\hat{\mathit{\beta }}}_j)⪰0,}$

which then implies:

${\displaystyle \mathrm{MSE}({\hat{\mathit{\beta }}}_{\mathrm{o}\mathrm{l}\mathrm{s}})-\mathrm{MSE}({\hat{\mathit{\beta }}}_k)⪰0}$

for that particular${\displaystyle k}$. Thus in that case, the corresponding${\displaystyle {\hat{\mathit{\beta }}}_k}$ would be a moreefficient estimator of${\displaystyle \mathit{\beta }}$ compared to${\displaystyle {\hat{\mathit{\beta }}}_{\mathrm{o}\mathrm{l}\mathrm{s}}}$, based on using the mean squared error as the performance criteria. In addition, any givenlinear form of the corresponding${\displaystyle {\hat{\mathit{\beta }}}_k}$ would also have a lowermean squared error compared to that of the samelinear form of${\displaystyle {\hat{\mathit{\beta }}}_{\mathrm{o}\mathrm{l}\mathrm{s}}}$.

Now suppose that for a given${\displaystyle k\in \{1,\dots ,p\},V_{(p-k)}^T\mathit{\beta }\ne \mathbf{0}}$. Then the corresponding${\displaystyle {\hat{\mathit{\beta }}}_k}$ isbiased for${\displaystyle \mathit{\beta }}$. However, since

${\displaystyle \mathrm{\forall }k\in \{1,\dots ,p\}:\mathrm{Var}({\hat{\mathit{\beta }}}_{\mathrm{o}\mathrm{l}\mathrm{s}})-\mathrm{Var}({\hat{\mathit{\beta }}}_k)⪰0,}$

it is still possible that${\displaystyle \mathrm{MSE}({\hat{\mathit{\beta }}}_{\mathrm{o}\mathrm{l}\mathrm{s}})-\mathrm{MSE}({\hat{\mathit{\beta }}}_k)⪰0}$, especially if${\displaystyle k}$ is such that the excluded principal components correspond to the smaller eigenvalues, thereby resulting in lowerbias.

In order to ensure efficient estimation and prediction performance of PCR as an estimator of${\displaystyle \mathit{\beta }}$, Park (1981)^[4] proposes the following guideline for selecting the principal components to be used for regression: Drop the${\displaystyle j^{th}}$ principal component if and only if Practical implementation of this guideline of course requires estimates for the unknown model parameters${\displaystyle {\sigma }^2}$ and${\displaystyle \mathit{\beta }}$. In general, they may be estimated using the unrestricted least squares estimates obtained from the original full model. Park (1981) however provides a slightly modified set of estimates that may be better suited for this purpose.^[4]

Unlike the criteria based on the cumulative sum of the eigenvalues of${\displaystyle {\mathbf{X}}^T\mathbf{X}}$, which is probably more suited for addressing the multicollinearity problem and for performing dimension reduction, the above criteria actually attempts to improve the prediction and estimation efficiency of the PCR estimator by involving both the outcome as well as the covariates in the process of selecting the principal components to be used in the regression step. Alternative approaches with similar goals include selection of the principal components based oncross-validation or theMallow's C_p criteria. Often, the principal components are also selected based on their degree ofassociation with the outcome.

In general, PCR is essentially ashrinkage estimator that usually retains the high variance principal components (corresponding to the higher eigenvalues of${\displaystyle {\mathbf{X}}^T\mathbf{X}}$) as covariates in the model and discards the remaining low variance components (corresponding to the lower eigenvalues of${\displaystyle {\mathbf{X}}^T\mathbf{X}}$). Thus it exerts a discreteshrinkage effect on the low variance components nullifying their contribution completely in the original model. In contrast, theridge regression estimator exerts a smooth shrinkage effect through theregularization parameter (or the tuning parameter) inherently involved in its construction. While it does not completely discard any of the components, it exerts a shrinkage effect over all of them in a continuous manner so that the extent of shrinkage is higher for the low variance components and lower for the high variance components. Frank and Friedman (1993)^[5] conclude that for the purpose of prediction itself, the ridge estimator, owing to its smooth shrinkage effect, is perhaps a better choice compared to the PCR estimator having a discrete shrinkage effect.

In addition, the principal components are obtained from theeigen-decomposition of${\displaystyle \mathbf{X}}$ that involves the observations for the explanatory variables only. Therefore, the resulting PCR estimator obtained from using these principal components as covariates need not necessarily have satisfactory predictive performance for the outcome. A somewhat similar estimator that tries to address this issue through its very construction is thepartial least squares (PLS) estimator. Similar to PCR, PLS also uses derived covariates of lower dimensions. However unlike PCR, the derived covariates for PLS are obtained based on using both the outcome as well as the covariates. While PCR seeks the high variance directions in the space of the covariates, PLS seeks the directions in the covariate space that are most useful for the prediction of the outcome.

2006 a variant of the classical PCR known as thesupervised PCR was proposed.^[6] In a spirit similar to that of PLS, it attempts at obtaining derived covariates of lower dimensions based on a criterion that involves both the outcome as well as the covariates. The method starts by performing a set of${\displaystyle p}$simple linear regressions (or univariate regressions) wherein the outcome vector is regressed separately on each of the${\displaystyle p}$ covariates taken one at a time. Then, for some${\displaystyle m\in \{1,\dots ,p\}}$, the first${\displaystyle m}$ covariates that turn out to be the most correlated with the outcome (based on the degree of significance of the corresponding estimated regression coefficients) are selected for further use. A conventional PCR, as described earlier, is then performed, but now it is based on only the${\displaystyle n\times m}$ data matrix corresponding to the observations for the selected covariates. The number of covariates used:${\displaystyle m\in \{1,\dots ,p\}}$ and the subsequent number of principal components used:${\displaystyle k\in \{1,\dots ,m\}}$ are usually selected bycross-validation.

The classical PCR method as described above is based onclassical PCA and considers alinear regression model for predicting the outcome based on the covariates. However, it can be easily generalized to akernel machine setting whereby theregression function need not necessarily belinear in the covariates, but instead it can belong to theReproducing Kernel Hilbert Space associated with any arbitrary (possiblynon-linear),symmetricpositive-definite kernel. Thelinear regression model turns out to be a special case of this setting when thekernel function is chosen to be thelinear kernel.

In general, under thekernel machine setting, the vector of covariates is firstmapped into ahigh-dimensional (potentiallyinfinite-dimensional)feature space characterized by thekernel function chosen. Themapping so obtained is known as thefeature map and each of itscoordinates, also known as thefeature elements, corresponds to one feature (may belinear ornon-linear) of the covariates. Theregression function is then assumed to be alinear combination of thesefeature elements. Thus, theunderlying regression model in thekernel machine setting is essentially alinear regression model with the understanding that instead of the original set of covariates, the predictors are now given by the vector (potentiallyinfinite-dimensional) offeature elements obtained bytransforming the actual covariates using thefeature map.

However, thekernel trick actually enables us to operate in thefeature space without ever explicitly computing thefeature map. It turns out that it is only sufficient to compute the pairwiseinner products among the feature maps for the observed covariate vectors and theseinner products are simply given by the values of thekernel function evaluated at the corresponding pairs of covariate vectors. The pairwise inner products so obtained may therefore be represented in the form of a${\displaystyle n\times n}$ symmetric non-negative definite matrix also known as thekernel matrix.

PCR in thekernel machine setting can now be implemented by firstappropriately centering thiskernel matrix (K, say) with respect to thefeature space and then performing akernel PCA on thecentered kernel matrix (K', say) whereby aneigendecomposition of K' is obtained. Kernel PCR then proceeds by (usually) selecting a subset of all theeigenvectors so obtained and then performing astandard linear regression of the outcome vector on these selectedeigenvectors. Theeigenvectors to be used for regression are usually selected usingcross-validation. The estimated regression coefficients (having the same dimension as the number of selected eigenvectors) along with the corresponding selected eigenvectors are then used for predicting the outcome for a future observation. Inmachine learning, this technique is also known asspectral regression.

Clearly, kernel PCR has a discrete shrinkage effect on the eigenvectors of K', quite similar to the discrete shrinkage effect of classical PCR on the principal components, as discussed earlier. However, the feature map associated with the chosen kernel could potentially be infinite-dimensional, and hence the corresponding principal components and principal component directions could be infinite-dimensional as well. Therefore, these quantities are often practically intractable under the kernel machine setting. Kernel PCR essentially works around this problem by considering an equivalent dual formulation based on using thespectral decomposition of the associated kernel matrix. Under the linear regression model (which corresponds to choosing the kernel function as the linear kernel), this amounts to considering a spectral decomposition of the corresponding${\displaystyle n\times n}$ kernel matrix${\displaystyle \mathbf{X}{\mathbf{X}}^T}$ and then regressing the outcome vector on a selected subset of the eigenvectors of${\displaystyle \mathbf{X}{\mathbf{X}}^T}$ so obtained. It can be easily shown that this is the same as regressing the outcome vector on the corresponding principal components (which are finite-dimensional in this case), as defined in the context of the classical PCR. Thus, for the linear kernel, the kernel PCR based on a dual formulation is exactly equivalent to the classical PCR based on a primal formulation. However, for arbitrary (and possibly non-linear) kernels, this primal formulation may become intractable owing to the infinite dimensionality of the associated feature map. Thus classical PCR becomes practically infeasible in that case, but kernel PCR based on the dual formulation still remains valid and computationally scalable.

• Principal component analysis
• Partial least squares regression
• Ridge regression
• Canonical correlation
• Deming regression
• Total sum of squares

• Amemiya, Takeshi (1985).Advanced Econometrics. Harvard University Press. pp. 57–60.ISBN 978-0-674-00560-0.
• Theil, Henri (1971).Principles of Econometrics. Wiley. pp. 46–55.ISBN 978-0-471-85845-4.

• Regression analysis
• Factor analysis

• Articles with short description
• Short description matches Wikidata