Movatterモバイル変換

Best Orthogonalized Subset Selection (BOSS).

Description

• Compute the solution path of BOSS and forward stepwise selection (FS).

• Compute various information criteria based on a heuristic degrees of freedom (hdf)that can serve as the selection rule to choose the subset given by BOSS.

Usage

boss(  x,  y,  maxstep = min(nrow(x) - intercept - 1, ncol(x)),  intercept = TRUE,  hdf.ic.boss = TRUE,  mu = NULL,  sigma = NULL,  ...)

Arguments

Details

This function computes the full solution path given by BOSS and FS on a givendataset (x,y) with n observations and p predictors. It also calculatesthe heuristic degrees of freedom for BOSS, and various information criteria, which can furtherbe used to select the subset from the candidates. Please refer to the Vignettefor implementation details and Tian et al. (2021) for methodology details (links are given below).

Value

• beta_fs: A matrix of regression coefficients for all the subsets given by FS,from a null model until stop, withnrow=p andncol=min(n,p)+1, wheremin(n,p) isthe maximum number of steps performed.

• beta_boss: A matrix of regression coefficients for all the subsets given byBOSS, withnrow=p andncol=min(n,p)+1. Note that unlike beta_fs and due to the nature of BOSS,the number of non-zero components in columns of beta_boss may not be unique, i.e.there maybe multiple columns corresponding to the same size of subset.

• steps_x: A vector of numbers representing which predictor joins at each step,withlength(steps)=min(n,p). The ordering is determined by the partial correlation between a predictorx_jand the responsey.

• steps_q: A vector of numbers representing which predictor joins at each step in the orthogonal basis,withlength(steps)=min(n,p). BOSS takes the ordered predictors (ordering given insteps_x) and performs bestsubset regression upon their orthogonal basis, which is essentially ordering the orthogonalized predictors by theirmarginal correlations with the responsey. For example,steps_q=c(2,1) indicates that the orthogonal basis ofx_2 joins first.

• hdf_boss: A vector of heuristic degrees of freedom (hdf) for BOSS, withlength(hdf_boss)=p+1. Note thathdf_boss=NULL if n<=p orhdf.ic.boss=FALSE.

• IC_boss: A list of information criteria (IC) for BOSS, where each elementin the list is a vector representing values of a given IC for each candidate subsetof BOSS (or each column in beta_boss). The output IC includes AIC, BIC, AICc, BICc,GCV and Mallows' Cp. Note that each IC is calculated by plugging in hdf_boss.

• sigma: estimated error standard deviation. It is only returned when hdf is calculated, i.e.hdf.ic.boss=TRUE.

Author(s)

Sen Tian

References

• Tian, S., Hurvich, C. and Simonoff, J. (2021), On the Use of Information Criteriafor Subset Selection in Least Squares Regression. https://arxiv.org/abs/1911.10191

• Reid, S., Tibshirani, R. and Friedman, J. (2016), A Study of Error Variance Estimation in Lasso Regression. Statistica Sinica,P35-67, JSTOR.

• BOSSreg Vignette https://github.com/sentian/BOSSreg/blob/master/r-package/vignettes/BOSSreg.pdf

See Also

predict andcoef methods for "boss" object, and thecv.boss function

Examples

## Generate a trivial dataset, X has mean 0 and norm 1, y has mean 0set.seed(11)n = 20p = 5x = matrix(rnorm(n*p), nrow=n, ncol=p)x = scale(x, center = colMeans(x))x = scale(x, scale = sqrt(colSums(x^2)))beta = c(1, 1, 0, 0, 0)y = x%*%beta + scale(rnorm(n, sd=0.01), center = TRUE, scale = FALSE)## Fit the modelboss_result = boss(x, y)## Get the coefficient vector selected by AICc-hdf (S3 method for boss)beta_boss_aicc = coef(boss_result)# the above is equivalent to the followingbeta_boss_aicc = boss_result$beta_boss[, which.min(boss_result$IC_boss$aicc), drop=FALSE]## Get the fitted values of BOSS-AICc-hdf (S3 method for boss)mu_boss_aicc = predict(boss_result, newx=x)# the above is equivalent to the followingmu_boss_aicc = cbind(1,x) %*% beta_boss_aicc## Repeat the above process, but using Cp-hdf instead of AICc-hdf## coefficient vectorbeta_boss_cp = coef(boss_result, method.boss='cp')beta_boss_cp = boss_result$beta_boss[, which.min(boss_result$IC_boss$cp), drop=FALSE]## fitted values of BOSS-Cp-hdfmu_boss_cp = predict(boss_result, newx=x, method.boss='cp')mu_boss_cp = cbind(1,x) %*% beta_boss_cp

Calculate an information criterion.

Description

Calculate a specified information criterion (IC) for an estimate or a group of estimates.The choices of IC include AIC, BIC, AICc, BICc, GCV and Mallows' Cp.

Usage

calc.ic(  y_hat,  y,  ic = c("aicc", "bicc", "aic", "bic", "gcv", "cp"),  df,  sigma = NULL)

Arguments

Details

This function enables the computation of various common IC for model fits, which canfurther be used to choose the optimal fit. This allows user comparing the effect of different IC.In order to calculate an IC, degrees of freedoms (df) needs to be specified. To be more specific,here are the formulas used to calculate each IC:

AIC = \log(\frac{RSS}{n}) + 2\frac{df}{n}

BIC = \log(\frac{RSS}{n}) + \log(n)\frac{df}{n}

AICc = \log(\frac{RSS}{n}) + 2\frac{df+1}{n-df-2}

BICc = \log(\frac{RSS}{n}) + \log(n)\frac{df+1}{n-df-2}

GCV = \frac{RSS}{(n-df)^2}

Mallows' Cp = RSS + 2\times \sigma^2 \times df

Value

The value(s) of the specified IC for each fit.

Author(s)

Sen Tian

Examples

## Generate a trivial dataset, X has mean 0 and norm 1, y has mean 0set.seed(11)n = 20p = 5x = matrix(rnorm(n*p), nrow=n, ncol=p)x = scale(x, center = colMeans(x))x = scale(x, scale = sqrt(colSums(x^2)))beta = c(1, 1, 0, 0, 0)y = x%*%beta + scale(rnorm(20, sd=0.01), center = TRUE, scale = FALSE)## Fit the modelboss_result = boss(x, y)## Print the values of AICc-hdf for all subsets given by BOSSprint(boss_result$IC_boss$aicc)## calculate them manually using the calc.ic functiony_hat = cbind(rep(1,n),x)%*%boss_result$beta_bossprint(calc.ic(y_hat, y, df=boss_result$hdf_boss))

Select coefficient vector(s) for BOSS.

Description

This function returns the optimal coefficient vector of BOSS selected by AICc(by default) or other types of information criterion.

Usage

## S3 method for class 'boss'coef(  object,  ic = c("aicc", "bicc", "aic", "bic", "gcv", "cp"),  select.boss = NULL,  ...)

Arguments

Details

Ifselect.boss is specified, the function returnscorresponding column(s) in the coefficient matrix.

Ifselect.boss is unspecified, the function returns the optimal coefficientvector selected by AICc-hdf (other choice of IC can be specified in the argumentic).

Value

The chosen coefficient vector(s) for BOSS.

Examples

## Generate a trivial dataset, X has mean 0 and norm 1, y has mean 0set.seed(11)n = 20p = 5x = matrix(rnorm(n*p), nrow=n, ncol=p)x = scale(x, center = colMeans(x))x = scale(x, scale = sqrt(colSums(x^2)))beta = c(1, 1, 0, 0, 0)y = x%*%beta + scale(rnorm(n, sd=0.01), center = TRUE, scale = FALSE)## Fit the modelboss_result = boss(x, y)## Get the coefficient vector selected by AICc-hdf (S3 method for boss)beta_boss_aicc = coef(boss_result)# the above is equivalent to the followingbeta_boss_aicc = boss_result$beta_boss[, which.min(boss_result$IC_boss$aicc), drop=FALSE]## Get the fitted values of BOSS-AICc-hdf (S3 method for boss)mu_boss_aicc = predict(boss_result, newx=x)# the above is equivalent to the followingmu_boss_aicc = cbind(1,x) %*% beta_boss_aicc## Repeat the above process, but using Cp-hdf instead of AICc-hdf## coefficient vectorbeta_boss_cp = coef(boss_result, method.boss='cp')beta_boss_cp = boss_result$beta_boss[, which.min(boss_result$IC_boss$cp), drop=FALSE]## fitted values of BOSS-Cp-hdfmu_boss_cp = predict(boss_result, newx=x, method.boss='cp')mu_boss_cp = cbind(1,x) %*% beta_boss_cp

Select coefficient vector based on cross-validation for BOSS or FS.

Description

This function returns coefficient vector that minimizes out-of-sample (OOS) crossvalidation score.

Usage

## S3 method for class 'cv.boss'coef(object, method = c("boss", "fs"), ...)

Arguments

Value

The chosen coefficient vector for BOSS or FS.

Examples

## Generate a trivial dataset, X has mean 0 and norm 1, y has mean 0set.seed(11)n = 20p = 5x = matrix(rnorm(n*p), nrow=n, ncol=p)x = scale(x, center = colMeans(x))x = scale(x, scale = sqrt(colSums(x^2)))beta = c(1, 1, 0, 0, 0)y = x%*%beta + scale(rnorm(20, sd=0.01), center = TRUE, scale = FALSE)## Perform 10-fold CV without replicationboss_cv_result = cv.boss(x, y)## Get the coefficient vector of BOSS that gives minimum CV OSS score (S3 method for cv.boss)beta_boss_cv = coef(boss_cv_result)# the above is equivalent toboss_result = boss_cv_result$bossbeta_boss_cv = boss_result$beta_boss[, boss_cv_result$i.min.boss, drop=FALSE]## Get the fitted values of BOSS-CV (S3 method for cv.boss)mu_boss_cv = predict(boss_cv_result, newx=x)# the above is equivalent tomu_boss_cv = cbind(1,x) %*% beta_boss_cv## Get the coefficient vector of FS that gives minimum CV OSS score (S3 method for cv.boss)beta_fs_cv = coef(boss_cv_result, method='fs')## Get the fitted values of FS-CV (S3 method for cv.boss)mu_fs_cv = predict(boss_cv_result, newx=x, method='fs')

Cross-validation for Best Orthogonalized Subset Selection (BOSS) and Forward Stepwise Selection (FS).

Description

Cross-validation for Best Orthogonalized Subset Selection (BOSS) and Forward Stepwise Selection (FS).

Usage

cv.boss(  x,  y,  maxstep = min(nrow(x) - intercept - 1, ncol(x)),  intercept = TRUE,  n.folds = 10,  n.rep = 1,  show.warning = TRUE,  ...)

Arguments

Details

This function fits BOSS and FS (boss) on the full dataset, and performsn.foldscross-validation. The cross-validation process can be repeatedn.rep times to evaluate theout-of-sample (OOS) performance for the candidate subsets given by both methods.

Value

• boss: An objectboss that fits on the full dataset.

• n.folds: The number of cross validation folds.

• cvm.fs: Mean OOS deviance for each candidate given by FS.

• cvm.boss: Mean OSS deviance for each candidate given by BOSS.

• i.min.fs: The index of minimum cvm.fs.

• i.min.boss: The index of minimum cvm.boss.

Author(s)

Sen Tian

References

• Tian, S., Hurvich, C. and Simonoff, J. (2021), On the Use of Information Criteriafor Subset Selection in Least Squares Regression. https://arxiv.org/abs/1911.10191

• BOSSreg Vignette https://github.com/sentian/BOSSreg/blob/master/r-package/vignettes/BOSSreg.pdf

See Also

predict andcoef methods forcv.boss object, and theboss function

Examples

## Generate a trivial dataset, X has mean 0 and norm 1, y has mean 0set.seed(11)n = 20p = 5x = matrix(rnorm(n*p), nrow=n, ncol=p)x = scale(x, center = colMeans(x))x = scale(x, scale = sqrt(colSums(x^2)))beta = c(1, 1, 0, 0, 0)y = x%*%beta + scale(rnorm(20, sd=0.01), center = TRUE, scale = FALSE)## Perform 10-fold CV without replicationboss_cv_result = cv.boss(x, y)## Get the coefficient vector of BOSS that gives minimum CV OSS score (S3 method for cv.boss)beta_boss_cv = coef(boss_cv_result)# the above is equivalent toboss_result = boss_cv_result$bossbeta_boss_cv = boss_result$beta_boss[, boss_cv_result$i.min.boss, drop=FALSE]## Get the fitted values of BOSS-CV (S3 method for cv.boss)mu_boss_cv = predict(boss_cv_result, newx=x)# the above is equivalent tomu_boss_cv = cbind(1,x) %*% beta_boss_cv## Get the coefficient vector of FS that gives minimum CV OSS score (S3 method for cv.boss)beta_fs_cv = coef(boss_cv_result, method='fs')## Get the fitted values of FS-CV (S3 method for cv.boss)mu_fs_cv = predict(boss_cv_result, newx=x, method='fs')

Prediction given new data entries.

Description

This function returns the prediction(s) given new observation(s), for BOSS,where the optimal coefficient vector is chosen via certain selection rule.

Usage

## S3 method for class 'boss'predict(object, newx, ...)

Arguments

Details

The function basically calculatesx * coef, wherecoefis a coefficient vector chosen by a selection rule. See more details about the defaultand available choices of the selection rule in the description ofcoef.boss.

Value

The prediction(s) for BOSS.

Examples

## Generate a trivial dataset, X has mean 0 and norm 1, y has mean 0set.seed(11)n = 20p = 5x = matrix(rnorm(n*p), nrow=n, ncol=p)x = scale(x, center = colMeans(x))x = scale(x, scale = sqrt(colSums(x^2)))beta = c(1, 1, 0, 0, 0)y = x%*%beta + scale(rnorm(n, sd=0.01), center = TRUE, scale = FALSE)## Fit the modelboss_result = boss(x, y)## Get the coefficient vector selected by AICc-hdf (S3 method for boss)beta_boss_aicc = coef(boss_result)# the above is equivalent to the followingbeta_boss_aicc = boss_result$beta_boss[, which.min(boss_result$IC_boss$aicc), drop=FALSE]## Get the fitted values of BOSS-AICc-hdf (S3 method for boss)mu_boss_aicc = predict(boss_result, newx=x)# the above is equivalent to the followingmu_boss_aicc = cbind(1,x) %*% beta_boss_aicc## Repeat the above process, but using Cp-hdf instead of AICc-hdf## coefficient vectorbeta_boss_cp = coef(boss_result, method.boss='cp')beta_boss_cp = boss_result$beta_boss[, which.min(boss_result$IC_boss$cp), drop=FALSE]## fitted values of BOSS-Cp-hdfmu_boss_cp = predict(boss_result, newx=x, method.boss='cp')mu_boss_cp = cbind(1,x) %*% beta_boss_cp

Prediction given new data entries.

Description

This function returns the prediction(s) given new observation(s) for BOSS or FS,where the optimal coefficient vector is chosen via cross-validation.

Usage

## S3 method for class 'cv.boss'predict(object, newx, ...)

Arguments

Value

The prediction for BOSS or FS.

Examples

## Generate a trivial dataset, X has mean 0 and norm 1, y has mean 0set.seed(11)n = 20p = 5x = matrix(rnorm(n*p), nrow=n, ncol=p)x = scale(x, center = colMeans(x))x = scale(x, scale = sqrt(colSums(x^2)))beta = c(1, 1, 0, 0, 0)y = x%*%beta + scale(rnorm(20, sd=0.01), center = TRUE, scale = FALSE)## Perform 10-fold CV without replicationboss_cv_result = cv.boss(x, y)## Get the coefficient vector of BOSS that gives minimum CV OSS score (S3 method for cv.boss)beta_boss_cv = coef(boss_cv_result)# the above is equivalent toboss_result = boss_cv_result$bossbeta_boss_cv = boss_result$beta_boss[, boss_cv_result$i.min.boss, drop=FALSE]## Get the fitted values of BOSS-CV (S3 method for cv.boss)mu_boss_cv = predict(boss_cv_result, newx=x)# the above is equivalent tomu_boss_cv = cbind(1,x) %*% beta_boss_cv## Get the coefficient vector of FS that gives minimum CV OSS score (S3 method for cv.boss)beta_fs_cv = coef(boss_cv_result, method='fs')## Get the fitted values of FS-CV (S3 method for cv.boss)mu_fs_cv = predict(boss_cv_result, newx=x, method='fs')