Description

MachineShop is a meta-package for statistical andmachine learning with a unified interface for model fitting, prediction,performance assessment, and presentation of results. Support is providedfor predictive modeling of numerical, categorical, and censoredtime-to-event outcomes and for resample (bootstrap, cross-validation,and split training-test sets) estimation of model performance. Thisvignette introduces the package interface with a survival data analysisexample, followed by supported methods of variable specification;applications to other response variable types; available performancemetrics, resampling techniques, and graphical and tabular summaries; andmodeling strategies.

Features

• Unified and concise interface for model fitting, prediction, andperformance assessment.
• Support for 53+ models from 28R packages,including model specifications from theparsnippackage.
• Dynamic model parameters.
• Ensemble modeling with stacked regression and super learners.
• Modeling of response variables types: binary factors, multi-classnominal and ordinal factors, numeric vectors and matrices, and censoredtime-to-event survival.
• Model specification with traditional formulas, design matrices, andflexible pre-processingrecipes.
• Resample estimation of predictive performance, includingcross-validation, bootstrap resampling, and split training-test setvalidation.
• Parallel execution of resampling algorithms.
• Choices of performance metrics: accuracy, areas under ROC andprecision recall curves, Brier score, coefficient of determination(R²), concordance index, cross entropy, F score, Ginicoefficient, unweighted and weighted Cohen’s kappa, mean absolute error,mean squared error, mean squared log error, positive and negativepredictive values, precision and recall, and sensitivity andspecificity.
• Graphical and tabular performance summaries: calibration curves,confusion matrices, partial dependence plots, performance curves, liftcurves, and model-specific and permutation-based variableimportance.
• Model tuning over automatically generated grids and with exhaustiveand random grid searches, Bayesian optimization, particle swarmoptimization, quasi-Newton BFGS optimization, simulated annealing, andsupport for user-defined optimization functions.
• Model selection and comparisons for any combination of models andmodel parameter values.
• Recursive feature elimination.
• User-definable models and performance metrics.

Getting Started

Installation

# Current release from CRANinstall.packages("MachineShop")# Development version from GitHub# install.packages("devtools")devtools::install_github("brian-j-smith/MachineShop")# Development version with vignettesdevtools::install_github("brian-j-smith/MachineShop",build_vignettes =TRUE)

Documentation

Once installed, the followingR commands will loadthe package and display its help system documentation. Onlinedocumentation and examples are available at theMachineShopwebsite.

library(MachineShop)# Package help summary?MachineShop# VignetteRShowDoc("UserGuide",package ="MachineShop")