Movatterモバイル変換

Type:	Package
Title:	Display and Analyze ROC Curves
Version:	1.19.0.1
Date:	2025-07-31
Encoding:	UTF-8
Depends:	R (≥ 2.14)
Imports:	methods, Rcpp (≥ 0.11.1)
Suggests:	tcltk, MASS, logcondens, testthat, vdiffr, ggplot2, rlang
LinkingTo:	Rcpp
Description:	Tools for visualizing, smoothing and comparing receiver operating characteristic (ROC curves). (Partial) area under the curve (AUC) can be compared with statistical tests based on U-statistics or bootstrap. Confidence intervals can be computed for (p)AUC or ROC curves.
License:	GPL (≥ 3)
URL:	https://xrobin.github.io/pROC/
BugReports:	https://github.com/xrobin/pROC/issues
LazyData:	yes
NeedsCompilation:	yes
Packaged:	2025-07-31 07:57:35 UTC; xavier
Author:	Xavier Robin [cre, aut], Natacha Turck [aut], Alexandre Hainard [aut], Natalia Tiberti [aut], Frédérique Lisacek [aut], Jean-Charles Sanchez [aut], Markus Müller [aut], Stefan Siegert [ctb] (Fast DeLong code), Matthias Doering [ctb] (Hand & Till Multiclass), Zane Billings [ctb] (DeLong paired test CI)
Maintainer:	Xavier Robin <pROC-cran@xavier.robin.name>
Repository:	CRAN
Date/Publication:	2025-07-31 10:00:14 UTC

pROC

Description

Tools for visualizing, smoothing and comparing receiver operatingcharacteristic (ROC curves). (Partial) area under the curve (AUC) canbe compared with statistical tests based on U-statistics orbootstrap. Confidence intervals can be computed for (p)AUC or ROCcurves. Sample size / power computation for one or two ROC curves are available.

Details

The basic unit of the pROC package is theroc function. Itwill build a ROC curve, smooth it if requested (ifsmooth=TRUE),compute the AUC (ifauc=TRUE), the confidence interval (CI) if requested (ifci=TRUE) and plot the curve if requested (ifplot=TRUE).

Theroc function will callsmooth,auc,ci andplot as necessary. See theseindividual functions for the arguments that can be passed to themthroughroc. These function can be called separately.

Two paired (that isroc objects with the sameresponse) or unpaired (with differentresponse) ROCcurves can be compared with theroc.test function.

Citation

If you use pROC in published research, please cite the following paper:

Xavier Robin, Natacha Turck, Alexandre Hainard, Natalia Tiberti,Frédérique Lisacek, Jean-Charles Sanchez and Markus Müller (2011).“pROC: an open-source package for R and S+ to analyze and compare ROCcurves”.BMC Bioinformatics,12, p. 77. DOI:doi:10.1186/1471-2105-12-77

Typecitation("pROC") for a BibTeX entry.

The authors would be glad to hear how pROC is employed. You are kindlyencouraged to notify Xavier Robin <pROC-cran@xavier.robin.name>about any work you publish.

Abbreviations

The following abbreviations are employed extensively in this package:

• ROC: receiver operating characteristic

• AUC: area under the ROC curve

• pAUC: partial area under the ROC curve

• CI: confidence interval

• SP: specificity

• SE: sensitivity

Functions

Dataset

This package comes with a dataset of 141 patients with aneurysmalsubarachnoid hemorrhage:aSAH.

Installing and using

To install this package, make sure you are connected to the internet and issue the following command in the R prompt:

    install.packages("pROC")

To load the package in R:

    library(pROC)

Experimental: pipelines

Since version 1.15.0, theroc function can be used in pipelines, for instance withdplyr ormagrittr. This is still a highly experimental feature and will change significantly in future versions (seeissue 54).Theroc.data.frame method supports both standard and non-standard evaluation (NSE), and theroc_function supports standard evaluation only.

library(dplyr)aSAH %>%     filter(gender == "Female") %>%    roc(outcome, s100b)

By default it returns theroc object, which can then be piped tothecoords function to extract coordinates that can be usedin further pipelines.

aSAH %>%     filter(gender == "Female") %>%    roc(outcome, s100b) %>%    coords(transpose=FALSE) %>%    filter(sensitivity > 0.6,            specificity > 0.6)

More details and use cases are available in theroc help page.

Bootstrap

All the bootstrap operations forsignificance testing,confidence interval,variance andcovariance computation are performed with non-parametric stratified or non-stratified resampling (according to thestratified argument) and with the percentile method, as described in Carpenter and Bithell (2000) sections 2.1 and 3.3.

Stratification of bootstrap can be controlledwithboot.stratified. In stratified bootstrap (the default), each replicatecontains the same number of cases and controls than the originalsample. Stratification is especially useful if one group has onlylittle observations, or if groups are not balanced.

The number of bootstrap replicates is controlled byboot.n. Higher numbers will give a more precise estimate of the significance tests and confidence intervalsbut take more time to compute. 2000 is recommanded by Carpenter and Bithell (2000) for confidence intervals. In our experience this is sufficient for a good estimation of the first significant digit only, so we recommend the use of 10000 bootstrap replicates to obtain a good estimate of the second significant digit whenever possible.

Progress bars

Progress bars are no longer supported in pROC 1.19. The use of theprogress argument is deprecated, and any value other thanNULLwill produce a warning. The argument will be removed in a future version.

Handling large datasets

Algorithms

Over the years, a significant amount of time has been invested in making pROC run faster and faster.From the naive algorithm iterating over all thresholds implemented in the first version, we went to aC++ implementation (withRcpp), and a different algorithm using cummulative sum of responses sorted by the predictor, which scales only with the number of data points, independently on the number of thresholds (algorithm = 2).The curves themselves are identical, but computation time has been decreased massively.

Earlier versions ofpROC allowed choosing the optimal algorithm. Since pROC 1.19, only a singlealgorithm (previouslyalgorithm = 2) is available. This allowed us to remove legacy codeand dependencies, and makepROC more maintainable and future proof.

Boostrap

Bootstrap is typically slow because it involves repeatedly computing the ROC curve (or a part of it).

Some bootstrap functions are faster than others. Typically,ci.thresholds is the fastest, andci.coords the slowest. Useci.coords only if the CI you need cannot be computed by the specialized CI functionsci.thresholds,ci.se andci.sp. Note thatci.auc cannot be replaced anyway.

A naive way to speed-up the boostrap is by removing the progress bar:

rocobj <- roc(response, round(predictor))system.time(ci(rocobj))system.time(ci(rocobj, progress = "none"))

It is of course possible to reduce the number of boostrap iterations. See theboot.n argument toci. This will reduce the precision of the bootstrap estimate.

Parallel processing

Parallel processing is no longer supported in pROC 1.19. The use of theparallel argument is deprecated, and any value other thanFALSE will produce a warning. The argument will be removed in afuture version.

Using DeLong instead of boostrap

DeLong is an asymptotically exact method to evaluate the uncertainty of an AUC (DeLonget al. (1988)). Since version 1.9, pROC uses the algorithm proposed by Sun and Xu (2014) which has an O(N log N) complexity and is always faster than bootstrapping. By default, pROC will choose the DeLong method whenever possible.

rocobj <- roc(response, round(predictor), algorithm=3)system.time(ci(rocobj, method="delong"))

Author(s)

Xavier Robin, Natacha Turck, Jean-Charles Sanchez and Markus Müller

Maintainer: Xavier Robin <pROC-cran@xavier.robin.name>

References

James Carpenter and John Bithell (2000) “Bootstrap condence intervals:when, which, what? A practical guide for medical statisticians”.Statistics in Medicine19, 1141–1164.DOI:doi:10.1002/(SICI)1097-0258(20000515)19:9<1141::AID-SIM479>3.0.CO;2-F.

Elisabeth R. DeLong, David M. DeLong and Daniel L. Clarke-Pearson(1988) “Comparing the areas under two or more correlated receiveroperating characteristic curves: a nonparametricapproach”.Biometrics44, 837–845.

Tom Fawcett (2006) “An introduction to ROC analysis”.PatternRecognition Letters27, 861–874. DOI:doi:10.1016/j.patrec.2005.10.010.

Xavier Robin, Natacha Turck, Alexandre Hainard,et al.(2011) “pROC: an open-source package for R and S+ to analyze andcompare ROC curves”.BMC Bioinformatics,7, 77.DOI:doi:10.1186/1471-2105-12-77.

Xu Sun and Weichao Xu (2014) “Fast Implementation of DeLongs Algorithm for Comparingthe Areas Under Correlated Receiver Operating Characteristic Curves”.IEEE SignalProcessing Letters,21, 1389–1393. DOI:doi:10.1109/LSP.2014.2337313.

Hadley Wickham (2011) “The Split-Apply-Combine Strategy for Data Analysis”.Journal of Statistical Software,40, 1–29.URL:doi:10.18637/jss.v040.i01.

See Also

CRAN packagesROCR,verification or Bioconductor'srocfor ROC curves.

CRAN packagesMASS andlogcondens employed in this package.

Examples

data(aSAH)## Build a ROC object and compute the AUC ##roc1 <- roc(aSAH$outcome, aSAH$s100b)print(roc1)# With a formularoc(outcome ~ s100b, aSAH)# With pipes, dplyr-style:## Not run: library(dplyr)aSAH %>% roc(outcome, s100b)## End(Not run)# Create a few more curves for the next examplesroc2 <- roc(aSAH$outcome, aSAH$wfns)roc3 <- roc(aSAH$outcome, aSAH$ndka)## AUC ##auc(roc1, partial.auc = c(1, .9))## Smooth ROC curve ##smooth(roc1)## Summary statisticsvar(roc1)cov(roc1, roc3)## Plot the curve ##plot(roc1)#  More plotting options, CI and plotting# with all-in-one syntax:roc4 <- roc(aSAH$outcome,            aSAH$s100b, percent=TRUE,            # arguments for auc            partial.auc=c(100, 90), partial.auc.correct=TRUE,            partial.auc.focus="sens",            # arguments for ci            ci=TRUE, boot.n=100, ci.alpha=0.9, stratified=FALSE,            # arguments for plot            plot=TRUE, auc.polygon=TRUE, max.auc.polygon=TRUE, grid=TRUE,            print.auc=TRUE, show.thres=TRUE)# Add to an existing plot. Beware of 'percent' specification!roc5 <- roc(aSAH$outcome, aSAH$wfns,            plot=TRUE, add=TRUE, percent=roc4$percent)## With ggplot2 ##if (require(ggplot2)) {# Create multiple curves to plotrocs <- roc(outcome ~ wfns + s100b + ndka, data = aSAH)ggroc(rocs)}## Coordinates of the curve ##coords(roc1, "best", ret=c("threshold", "specificity", "1-npv"))coords(roc2, "local maximas", ret=c("threshold", "sens", "spec", "ppv", "npv"))## Confidence intervals ### CI of the AUCci(roc2)## Not run: # CI of the curvesens.ci <- ci.se(roc1, specificities=seq(0, 100, 5))plot(sens.ci, type="shape", col="lightblue")plot(sens.ci, type="bars")## End(Not run)# need to re-add roc2 over the shapeplot(roc2, add=TRUE)## Not run: # CI of thresholdsplot(ci.thresholds(roc2))## End(Not run)## Comparisons ### Test on the whole AUCroc.test(roc1, roc2, reuse.auc=FALSE)## Not run: # Test on a portion of the whole AUCroc.test(roc1, roc2, reuse.auc=FALSE, partial.auc=c(100, 90),         partial.auc.focus="se", partial.auc.correct=TRUE)# With modified bootstrap parametersroc.test(roc1, roc2, reuse.auc=FALSE, partial.auc=c(100, 90),         partial.auc.correct=TRUE, boot.n=1000, boot.stratified=FALSE)## End(Not run)## Power & sample size ### Power# 1 curvepower.roc.test(roc1)# 2 curvespower.roc.test(roc3, roc2)# Sample size # 1 curvepower.roc.test(roc3, power = 0.9)# 2 curvespower.roc.test(roc1, roc2, power = 0.9)# Also without ROC objects.# For instance what AUC would be significantly different from 0.5?power.roc.test(ncases=41, ncontrols=72, sig.level=0.05, power=0.95)

Subarachnoid hemorrhage data

Description

This dataset summarizes several clinical and one laboratory variableof 113 patients with an aneurysmal subarachnoid hemorrhage.

Usage

aSAH

Format

A data.frame containing 113 observations of 7 variables.

Source

Natacha Turck, Laszlo Vutskits, Paola Sanchez-Pena, XavierRobin, Alexandre Hainard, Marianne Gex-Fabry, Catherine Fouda, HadijiBassem, Markus Mueller, Frédérique Lisacek, Louis Puybasset andJean-Charles Sanchez (2010) “A multiparameter panel method for outcomeprediction following aneurysmal subarachnoid hemorrhage”.Intensive Care Medicine36(1), 107–115. DOI:doi:10.1007/s00134-009-1641-y.

See Also

Other examples can be found in all the documentation pages of thispackage:roc,auc,ci,ci.auc,ci.se,ci.sp,ci.thresholds,coords,plot.ci,plot.roc,print.roc,roc.test andsmooth.

An example analysis with pROC is shown in:

Xavier Robin, Natacha Turck, Alexandre Hainard,et al.(2011) “pROC: an open-source package for R and S+ to analyze andcompare ROC curves”.BMC Bioinformatics,7, 77.DOI:doi:10.1186/1471-2105-12-77

Examples

# load the datasetdata(aSAH)# Gender, outcome and setwith(aSAH, table(gender, outcome))# Agewith(aSAH, by(age, outcome, mean))with(aSAH, by(age, outcome,     function(x) sprintf("mean: %.1f (+/- %.1f), median: %.1f (%i-%i)",                         mean(x), sd(x), median(x), min(x), max(x))))# WFNS scorewith(aSAH, table(wfns=ifelse(wfns<=2, "1-2", "3-4-5"), outcome))

Are two ROC curves paired?

Description

This function determines if two ROC curves can be paired.

Usage

are.paired(...)## S3 method for class 'auc'are.paired(roc1, roc2, ...)## S3 method for class 'smooth.roc'are.paired(roc1, roc2, ...)## S3 method for class 'roc'are.paired(roc1, roc2, return.paired.rocs=FALSE,  reuse.auc = TRUE, reuse.ci = FALSE, reuse.smooth=TRUE, ...)

Arguments

Details

Two ROC curves are paired if they are built on two variables observedon the same sample.

In practice, the paired status is granted if theresponse andlevels vectorof both ROC curves areidentical. If theresponses are different, this can bedue to missing values differing between the curves. In this case, thefunction will strip allNAs in both curves and check foridentity again.

It can raise false positives if the responses are identical but correspondto different patients.

Value

TRUE ifroc1 androc2 are paired,FALSEotherwise.

In addition, ifTRUE andreturn.paired.rocs=TRUE, thefollowing atributes are defined:

See Also

roc,roc.test

Examples

data(aSAH)aSAH.copy <- aSAH# artificially insert NAs for demonstration purposesaSAH.copy$outcome[42] <- NAaSAH.copy$s100b[24] <- NAaSAH.copy$ndka[1:10] <- NA# Call roc() on the whole dataroc1 <- roc(aSAH.copy$outcome, aSAH.copy$s100b)roc2 <- roc(aSAH.copy$outcome, aSAH.copy$ndka)# are.paired can still find that the curves were pairedare.paired(roc1, roc2) # TRUE# Removing the NAs manually before passing to roc() un-pairs the ROC curvesnas <- is.na(aSAH.copy$outcome) | is.na(aSAH.copy$ndka)roc2b <- roc(aSAH.copy$outcome[!nas], aSAH.copy$ndka[!nas])are.paired(roc1, roc2b) # FALSE# Getting the two paired ROC curves with additional smoothing and ci optionsroc2$ci <- ci(roc2)paired <- are.paired(smooth(roc1), roc2, return.paired.rocs=TRUE, reuse.ci=TRUE)paired.roc1 <- attr(paired, "roc1")paired.roc2 <- attr(paired, "roc2")

Compute the area under the ROC curve

Description

This function computes the numeric value of area under the ROC curve(AUC) with the trapezoidal rule. Two syntaxes are possible: one object of class “roc”, or eithertwo vectors (response, predictor) or a formula (response~predictor) asin theroc function.By default, the total AUC is computed, but a portion of the ROC curvecan be specified withpartial.auc.

Usage

auc(...)## S3 method for class 'roc'auc(roc, partial.auc=FALSE, partial.auc.focus=c("specificity","sensitivity"), partial.auc.correct=FALSE, allow.invalid.partial.auc.correct = FALSE, ...)## S3 method for class 'smooth.roc'auc(smooth.roc, ...)## S3 method for class 'multiclass.roc'auc(multiclass.roc, ...)## S3 method for class 'formula'auc(formula, data, ...)## Default S3 method:auc(response, predictor, ...)

Arguments

Details

This function is typically called fromroc whenauc=TRUE(default). It is also used byci. When it is called withtwo vectors (response, predictor) or a formula (response~predictor)arguments, theroc function is called and only the AUC isreturned.

By default the total area under the curve is computed, but a partial AUC (pAUC)can bespecified with thepartial.auc argument. It specifies the bounds ofspecificity or sensitivity (depending onpartial.auc.focus) betweenwhich the AUC will be computed. As it specifies specificities orsensitivities, you must adapt it in relation to the 'percent'specification (see details inroc).

partial.auc.focus is ignored ifpartial.auc=FALSE (default). If a partial AUC is computed,partial.auc.focus specifies if the bounds specified inpartial.auc must be interpreted as sensitivity orspecificity. Any other value will produce an error. It is recommended toplot the ROC curve withauc.polygon=TRUE in order to make sure the specification is correct.

If a pAUC is defined, it can be standardized (corrected). This correction is controled by thepartial.auc.correct argument. Ifpartial.auc.correct=TRUE,the correction by McClish will be applied:

\frac{1+\frac{auc-min}{max-min}}{2}

where auc is the uncorrected pAUC computed in the region defined bypartial.auc,min is the value of the non-discriminant AUC (with an AUC of 0.5 or 50in the region and max is the maximum possible AUC in the region. With this correction, the AUCwill be 0.5 if non discriminant and 1.0 if maximal, whatever the regiondefined. This correction is fully compatible withpercent.

Note that this correction is undefined for curves below the diagonal (auc < min). Attemptingto correct such an AUC will returnNA with a warning.

Value

The numeric AUC value, of classc("auc", "numeric") (orc("multiclass.auc", "numeric") orc("mv.multiclass.auc", "numeric")if a “multiclass.roc” was supplied), infraction of the area or in percent ifpercent=TRUE, with thefollowing attributes:

Smoothed ROC curves

There is no difference in the computation of the area under a smoothedROC curve, except for curves smoothed withmethod="binomial". In this caseand only if a full AUC is requested, the classical binormal AUC formula is applied:

auc=\phi\frac{a}{\sqrt{1 + b^2}}.

If the ROC curve is smoothed with any othermethod or if a partial AUCis requested, the empirical AUC described in the previous section is applied.

Multi-class AUCs

With an object of class “multiclass.roc”, a multi-class AUC iscomputed as an average AUC as defined by Hand and Till (equation 7).

auc=\frac{2}{c(c-1)}\sum{aucs}

with aucs all the pairwise roc curves.

References

Tom Fawcett (2006) “An introduction to ROC analysis”.PatternRecognition Letters27, 861–874. DOI:doi:10.1016/j.patrec.2005.10.010.

David J. Hand and Robert J. Till (2001). A Simple Generalisation ofthe Area Under the ROC Curve for Multiple Class ClassificationProblems.Machine Learning45(2), p. 171–186. DOI:doi:10.1023/A:1010920819831.

Donna Katzman McClish (1989) “Analyzing a Portion of the ROCCurve”.Medical Decision Making9(3), 190–195. DOI:doi:10.1177/0272989X8900900307.

Xavier Robin, Natacha Turck, Alexandre Hainard,et al.(2011) “pROC: an open-source package for R and S+ to analyze andcompare ROC curves”.BMC Bioinformatics,7, 77.DOI:doi:10.1186/1471-2105-12-77.

See Also

roc,ci.auc

Examples

# Create a ROC curve:data(aSAH)roc.s100b <- roc(aSAH$outcome, aSAH$s100b)# Get the full AUCauc(roc.s100b)# Get the partial AUC:auc(roc.s100b, partial.auc=c(1, .8), partial.auc.focus="se", partial.auc.correct=TRUE)

Compute the confidence interval of a ROC curve

Description

This function computes the confidence interval (CI) of a ROC curve. Theof argument controls the type of CI that will be computed.

Usage

ci(...)## S3 method for class 'roc'ci(roc, of = c("auc", "thresholds", "sp", "se", "coords"), ...)## S3 method for class 'smooth.roc'ci(smooth.roc, of = c("auc", "sp", "se", "coords"), ...)## S3 method for class 'multiclass.roc'ci(multiclass.roc, of = "auc", ...)## S3 method for class 'multiclass.auc'ci(multiclass.auc, of = "auc", ...)## S3 method for class 'formula'ci(formula, data, ...)## Default S3 method:ci(response, predictor, ...)

Arguments

Details

ci.formula andci.default are convenience methodsthat build the ROC curve (with theroc function) beforecallingci.roc. You can pass them arguments for bothroc andci.roc. Simply usecithat will dispatch to the correct method.

This function is typically called fromroc whenci=TRUE (not bydefault). Depending on theof argument, the specificci functionsci.auc,ci.thresholds,ci.sp,ci.se orci.coords are called.

When the ROC curve has anauc of 1 (or 100%), the confidence interval will always be null(there is no interval). This is true for both “delong” and “bootstrap” methods that cannot properly assess the variance in this case. This result is misleading, as the variance is of course not null.Awarning will be displayed to inform of this condition, and of the misleading output.

CI of multiclass ROC curves and AUC is not implemented yet. Attempting to call thesemethods returns an error.

Value

The return value of the specificci functionsci.auc,ci.thresholds,ci.sp,ci.se orci.coords, depending on theof argument.

References

Xavier Robin, Natacha Turck, Alexandre Hainard,et al.(2011) “pROC: an open-source package for R and S+ to analyze andcompare ROC curves”.BMC Bioinformatics,7, 77.DOI:doi:10.1186/1471-2105-12-77.

See Also

roc,auc,ci.auc,ci.thresholds,ci.sp,ci.se,ci.coords

Examples

# Create a ROC curve:data(aSAH)roc1 <- roc(aSAH$outcome, aSAH$s100b)## AUC ## ci(roc1)# this is equivalent to:ci(roc1, of = "auc")# or:ci.auc(roc1)## Coordinates #### Not run: # Thresholdsci(roc1, of = "thresholds")ci(roc1, of = "thresholds", thresholds = "all")ci(roc1, of = "thresholds", thresholds = 0.51)# equivalent to:ci.thresholds(roc1, thresholds = 0.51)# SE/SPci(roc1, of = "sp", sensitivities = c(.95, .9, .85))ci.sp(roc1)ci(roc1, of = "se")ci.se(roc1)# Arbitrary coordinatesci(roc1, of = "coords", "best")ci.coords(roc1, 0.51, "threshold")## End(Not run)

Compute the confidence interval of the AUC

Description

This function computes the confidence interval (CI) of an area underthe curve (AUC).

Usage

# ci.auc(...)## S3 method for class 'roc'ci.auc(roc, conf.level=0.95, method=c("delong","bootstrap"), boot.n = 2000, boot.stratified = TRUE, reuse.auc=TRUE,progress = NULL, parallel=FALSE, ...)## S3 method for class 'smooth.roc'ci.auc(smooth.roc, conf.level=0.95, boot.n=2000,boot.stratified=TRUE, reuse.auc=TRUE,progress = NULL, parallel=FALSE, ...)## S3 method for class 'auc'ci.auc(auc, ...)## S3 method for class 'multiclass.roc'ci.auc(multiclass.roc, ...)## S3 method for class 'multiclass.auc'ci.auc(multiclass.auc, ...)## S3 method for class 'auc'ci.auc(auc, ...)## S3 method for class 'formula'ci.auc(formula, data, ...)## Default S3 method:ci.auc(response, predictor, ...)

Arguments

Details

This function computes the CI of an AUC. Two methods are available:“delong” and “bootstrap” with the parameters defined in “roc$auc” tocompute a CI. When it is called with two vectors (response, predictor)or a formula (response~predictor) arguments, therocfunction is called to build the ROC curve first.

The default is to use the “delong” method, except for comparison ofpartial AUC and smoothed curves, wherebootstrap is used. Using“delong” for partial AUC and smoothed ROCs is not supported.

Withmethod="bootstrap", the function callsaucboot.n times. For more details about the bootstrap, see the Bootstrap section inthis package's documentation.

Forsmoothed ROC curves, smoothing is performed again at eachbootstrap replicate with the parameters originally provided.If a density smoothing was performed with user-provideddensity.cases ordensity.controls the bootstrap cannotbe performed and an error is issued.

Withmethod="delong", the variance of the AUC is computed asdefined by DeLonget al. (1988) using the algorithm by Sun and Xu (2014) and the CI is deduced withqnorm.

CI of multiclass ROC curves and AUC is not implemented yet. Attempting to call thesemethods returns an error.

Value

A numeric vector of length 3 and class “ci.auc”, “ci” and “numeric” (in this order), with the lowerbound, the median and the upper bound of the CI, and the following attributes:

Theaucs item is not included in this list since version 1.2 forconsistency reasons.

AUC specification

The comparison of the CI needs a specification of the AUC. This allowsto compute the CI for full or partial AUCs. The specification is defined by:

1. the “auc” field in the “roc” object ifreuse.auc is set toTRUE (default). It is naturallyinherited from any call toroc and fits most cases.

2. passing the specification toauc with ...(argumentspartial.auc,partial.auc.correct andpartial.auc.focus). In this case, you must ensure either thattheroc object do not contain anauc field (ifyou calledroc withauc=FALSE), or setreuse.auc=FALSE.

Ifreuse.auc=FALSE theauc function will alwaysbe called with... to determine the specification, even ifthe “roc” object do contain anauc field.

As well if the “roc” object do not contain anaucfield, theauc function will always be called with... to determine the specification.

Warning: if the roc object passed to ci contains anaucfield andreuse.auc=TRUE,auc is not called andarguments such aspartial.auc are silently ignored.

Warnings

Ifmethod="delong" and the AUC specification specifies apartial AUC, the warning “Using DeLong's test for partial AUC isnot supported. Using bootstrap test instead.” is issued. Themethod argument is ignored and “bootstrap” is usedinstead.

Ifboot.stratified=FALSE and the sample has a large imbalance betweencases and controls, it could happen that one or more of the replicatescontains no case or control observation, or that there are not enoughpoints for smoothing, producing aNA area.The warning “NA value(s) produced during bootstrap were ignored.”will be issued and the observation will be ignored. If you have a largeimbalance in your sample, it could be safer to keepboot.stratified=TRUE.

Errors

Ifdensity.cases anddensity.controls were providedfor smoothing, the error “Cannot compute the statistic on ROCcurves smoothed with density.controls and density.cases.” is issued.

References

James Carpenter and John Bithell (2000) “Bootstrap condence intervals:when, which, what? A practical guide for medical statisticians”.Statistics in Medicine19, 1141–1164.DOI:doi:10.1002/(SICI)1097-0258(20000515)19:9<1141::AID-SIM479>3.0.CO;2-F.

Elisabeth R. DeLong, David M. DeLong and Daniel L. Clarke-Pearson(1988) “Comparing the areas under two or more correlated receiveroperating characteristic curves: a nonparametricapproach”.Biometrics44, 837–845.

Xu Sun and Weichao Xu (2014) “Fast Implementation of DeLongs Algorithm for Comparingthe Areas Under Correlated Receiver Operating Characteristic Curves”.IEEE SignalProcessing Letters,21, 1389–1393. DOI:doi:10.1109/LSP.2014.2337313.

Xavier Robin, Natacha Turck, Alexandre Hainard,et al.(2011) “pROC: an open-source package for R and S+ to analyze andcompare ROC curves”.BMC Bioinformatics,7, 77.DOI:doi:10.1186/1471-2105-12-77.

Hadley Wickham (2011) “The Split-Apply-Combine Strategy for Data Analysis”.Journal of Statistical Software,40, 1–29.URL:doi:10.18637/jss.v040.i01.

See Also

roc,auc,ci

Examples

# Create a ROC curve:data(aSAH)roc1 <- roc(aSAH$outcome, aSAH$s100b)## Basic example ##ci.auc(roc1)# You can also write:ci(roc1)ci(auc(roc1))## More options ### Partial AUC and customized bootstrap:## Not run: ci.auc(roc1,       conf.level=0.9,   partial.auc=c(1, .8),  partial.auc.focus="se", partial.auc.correct=TRUE,       boot.n=10000, stratified=FALSE)## End(Not run)# Note that the following will NOT give a CI of the partial AUC:## Not run: ci.auc(roc1,       partial.auc=c(1, .8), partial.auc.focus="se", partial.auc.correct=FALSE)## End(Not run)# This is because rocobj$auc is not a partial AUC and reuse.auc = TRUE by default.# You can overcome this problem by passing an AUC instead:auc1 <- auc(roc1, partial.auc=c(1, .8), partial.auc.focus="se",             partial.auc.correct=FALSE)## Not run: ci.auc(auc1)## End(Not run)## On smoothed ROC curves with bootstrap #### Not run: ci.auc(smooth(roc1, method="density"))## End(Not run)

Compute the confidence interval of arbitrary coordinates

Description

This function computes the confidence interval (CI) of the coordinatesof a ROC curves with thecoords function.

Usage

# ci.coords(...)## S3 method for class 'roc'ci.coords(roc, x,input="threshold",ret=c("threshold", "specificity", "sensitivity"),best.method=c("youden", "closest.topleft"), best.weights=c(1, 0.5),best.policy = c("stop", "omit", "random"),conf.level=0.95, boot.n=2000,boot.stratified=TRUE,progress = NULL, ...) ## S3 method for class 'formula'ci.coords(formula, data, ...)## S3 method for class 'smooth.roc'ci.coords(smooth.roc, x,input=c("specificity", "sensitivity"), ret=c("specificity", "sensitivity"),best.method=c("youden", "closest.topleft"), best.weights=c(1, 0.5),best.policy = c("stop", "omit", "random"),conf.level=0.95, boot.n=2000,boot.stratified=TRUE,progress = NULL, ...)## Default S3 method:ci.coords(response, predictor, ...)

Arguments

Details

ci.coords.formula andci.coords.default are convenience methodsthat build the ROC curve (with theroc function) beforecallingci.coords.roc. You can pass them arguments for bothroc andci.coords.roc. Simply useci.coordsthat will dispatch to the correct method.

This function createsboot.n bootstrap replicate of the ROCcurve, and evaluates the coordinates specified by thex,input,ret,best.method andbest.weights arguments. Then it computes theconfidence interval as the percentiles given byconf.level.

Whenx="best", the best threshold is determined at each bootstrap iteration, effectively assessing the confidence interval of choice of the "best"threshold itself. This differs from the behavior ofci.thresholds,where the "best" threshold is assessed on the given ROC curve beforeresampling.

For more details about the bootstrap, see the Bootstrap section inthis package's documentation.

Value

Note: changed in version 1.16.

A list of the same length asret and named asret, and of class “ci.thresholds”, “ci” and “list” (in this order).

Each element of the list is a matrix of the confidence intervals withrows given byx and with 3 columns, the lower bound of the CI, the median, and the upper bound of the CI.

Additionally, the list has the following attributes:

Warnings

Ifboot.stratified=FALSE and the sample has a large imbalance betweencases and controls, it could happen that one or more of the replicatescontains no case or control observation, producing aNA area.The warning “NA value(s) produced during bootstrap were ignored.”will be issued and the observation will be ignored. If you have a largeimbalance in your sample, it could be safer to keepboot.stratified=TRUE.

This warning will also be displayed if you chosebest.policy = "omit"and a ROC curve with multiple “best” threshold was generatedduring at least one of the replicates.

References

James Carpenter and John Bithell (2000) “Bootstrap condence intervals:when, which, what? A practical guide for medical statisticians”.Statistics in Medicine19, 1141–1164.DOI:doi:10.1002/(SICI)1097-0258(20000515)19:9<1141::AID-SIM479>3.0.CO;2-F.

Tom Fawcett (2006) “An introduction to ROC analysis”.PatternRecognition Letters27, 861–874. DOI:doi:10.1016/j.patrec.2005.10.010.

Hadley Wickham (2011) “The Split-Apply-Combine Strategy for Data Analysis”.Journal of Statistical Software,40, 1–29.URL:doi:10.18637/jss.v040.i01.

See Also

roc,coords,ci

Examples

# Create a ROC curve:data(aSAH)roc1 <- roc(aSAH$outcome, aSAH$s100b)## Basic example #### Not run: ci.coords(roc1, x="best", input = "threshold",           ret=c("specificity", "ppv", "tp"))## More options ##ci.coords(roc1, x=0.9, input = "sensitivity", ret="specificity")ci.coords(roc1, x=0.9, input = "sensitivity", ret=c("specificity", "ppv", "tp"))ci.coords(roc1, x=c(0.1, 0.5, 0.9), input = "sensitivity", ret="specificity")ci.coords(roc1, x=c(0.1, 0.5, 0.9), input = "sensitivity", ret=c("specificity", "ppv", "tp"))# Return everything we can:rets <- c("threshold", "specificity", "sensitivity", "accuracy", "tn", "tp", "fn", "fp", "npv",           "ppv", "1-specificity", "1-sensitivity", "1-accuracy", "1-npv", "1-ppv")ci.coords(roc1, x="best", input = "threshold", ret=rets)## End(Not run)## On smoothed ROC curves with bootstrap #### Not run: ci.coords(smooth(roc1), x=0.9, input = "sensitivity", ret=c("specificity", "ppv", "tp"))## End(Not run)

Compute the confidence interval of sensitivities at given specificities

Description

This function computes the confidence interval (CI) of the sensitivityat the given specificity points.

Usage

# ci.se(...)## S3 method for class 'roc'ci.se(roc, specificities = seq(0, 1, .1) * ifelse(roc$percent,100, 1), conf.level=0.95, boot.n=2000, boot.stratified=TRUE,progress = NULL, parallel=FALSE, ...) ## S3 method for class 'smooth.roc'ci.se(smooth.roc, specificities = seq(0, 1, .1) *ifelse(smooth.roc$percent, 100, 1), conf.level=0.95, boot.n=2000,boot.stratified=TRUE, progress = NULL,parallel=FALSE, ...)## S3 method for class 'formula'ci.se(formula, data, ...)## Default S3 method:ci.se(response, predictor, ...)

Arguments

Details

ci.se.formula andci.se.default are convenience methodsthat build the ROC curve (with theroc function) beforecallingci.se.roc. You can pass them arguments for bothroc andci.se.roc. Simply useci.sethat will dispatch to the correct method.

Theci.se.roc function createsboot.n bootstrap replicate of the ROCcurve, and evaluates the sensitivity at specificitiesgiven by thespecificities argument. Then it computes theconfidence interval as the percentiles given byconf.level.

For more details about the bootstrap, see the Bootstrap section inthis package's documentation.

Forsmoothed ROC curves, smoothing is performed again at eachbootstrap replicate with the parameters originally provided.If a density smoothing was performed with user-provideddensity.cases ordensity.controls the bootstrap cannotbe performed and an error is issued.

Value

A matrix of class “ci.se”, “ci” and “matrix” (in this order)containing the given sensitivities. Row (names) are thespecificities, the first column the lower bound, the 2nd column themedian and the 3rd column the upper bound.

Additionally, the list has the following attributes:

Warnings

Ifboot.stratified=FALSE and the sample has a large imbalance betweencases and controls, it could happen that one or more of the replicatescontains no case or control observation, or that there are not enoughpoints for smoothing, producing aNA area.The warning “NA value(s) produced during bootstrap were ignored.”will be issued and the observation will be ignored. If you have a largeimbalance in your sample, it could be safer to keepboot.stratified=TRUE.

Errors

Ifdensity.cases anddensity.controls were providedfor smoothing, the error “Cannot compute the statistic on ROCcurves smoothed with density.controls and density.cases.” is issued.

References

James Carpenter and John Bithell (2000) “Bootstrap condence intervals:when, which, what? A practical guide for medical statisticians”.Statistics in Medicine19, 1141–1164.DOI:doi:10.1002/(SICI)1097-0258(20000515)19:9<1141::AID-SIM479>3.0.CO;2-F.

Tom Fawcett (2006) “An introduction to ROC analysis”.PatternRecognition Letters27, 861–874. DOI:doi:10.1016/j.patrec.2005.10.010.

Xavier Robin, Natacha Turck, Alexandre Hainard,et al.(2011) “pROC: an open-source package for R and S+ to analyze andcompare ROC curves”.BMC Bioinformatics,7, 77.DOI:doi:10.1186/1471-2105-12-77.

Hadley Wickham (2011) “The Split-Apply-Combine Strategy for Data Analysis”.Journal of Statistical Software,40, 1–29.URL:doi:10.18637/jss.v040.i01.

See Also

roc,ci,ci.sp,plot.ci

Examples

# Create a ROC curve:data(aSAH)roc1 <- roc(aSAH$outcome, aSAH$s100b)## Basic example #### Not run: ci.se(roc1)## End(Not run)## More options ### Customized bootstrap and specificities:## Not run: ci.se(roc1, c(.95, .9, .85), boot.n=10000, conf.level=0.9, stratified=FALSE)## End(Not run)## Plotting the CI ##ci1 <- ci.se(roc1, boot.n = 10)plot(roc1)plot(ci1)## On smoothed ROC curves with bootstrap #### Not run: ci.se(smooth(roc1, method="density"))## End(Not run)

Compute the confidence interval of specificities at given sensitivities

Description

This function computes the confidence interval (CI) of the specificityat the given sensitivity points.

Usage

# ci.sp(...)## S3 method for class 'roc'ci.sp(roc, sensitivities = seq(0, 1, .1) * ifelse(roc$percent,100, 1), conf.level=0.95, boot.n=2000, boot.stratified=TRUE,progress = NULL, parallel=FALSE, ...) ## S3 method for class 'smooth.roc'ci.sp(smooth.roc, sensitivities = seq(0, 1, .1) *ifelse(smooth.roc$percent, 100, 1), conf.level=0.95, boot.n=2000,boot.stratified=TRUE, progress = NULL, parallel=FALSE, ...)## S3 method for class 'formula'ci.sp(formula, data, ...)## Default S3 method:ci.sp(response, predictor, ...)

Arguments

Details

ci.sp.formula andci.sp.default are convenience methodsthat build the ROC curve (with theroc function) beforecallingci.sp.roc. You can pass them arguments for bothroc andci.sp.roc. Simply useci.spthat will dispatch to the correct method.

Theci.sp.roc function createsboot.n bootstrap replicate of the ROCcurve, and evaluates the specificity at sensitivitiesgiven by thesensitivities argument. Then it computes theconfidence interval as the percentiles given byconf.level.

For more details about the bootstrap, see the Bootstrap section inthis package's documentation.

Forsmoothed ROC curves, smoothing is performed again at eachbootstrap replicate with the parameters originally provided.If a density smoothing was performed with user-provideddensity.cases ordensity.controls the bootstrap cannotbe performed and an error is issued.

Value

A matrix of class “ci.sp”, “ci” and “matrix” (in this order)containing the given specificities. Row (names) are thesensitivities, the first column the lower bound, the 2nd column themedian and the 3rd column the upper bound.

Additionally, the list has the following attributes:

Warnings

Ifboot.stratified=FALSE and the sample has a large imbalance betweencases and controls, it could happen that one or more of the replicatescontains no case or control observation, or that there are not enoughpoints for smoothing, producing aNA area.The warning “NA value(s) produced during bootstrap were ignored.”will be issued and the observation will be ignored. If you have a largeimbalance in your sample, it could be safer to keepboot.stratified=TRUE.

Errors

Ifdensity.cases anddensity.controls were providedfor smoothing, the error “Cannot compute the statistic on ROCcurves smoothed with density.controls and density.cases.” is issued.

References

James Carpenter and John Bithell (2000) “Bootstrap condence intervals:when, which, what? A practical guide for medical statisticians”.Statistics in Medicine19, 1141–1164.DOI:doi:10.1002/(SICI)1097-0258(20000515)19:9<1141::AID-SIM479>3.0.CO;2-F.

Tom Fawcett (2006) “An introduction to ROC analysis”.PatternRecognition Letters27, 861–874. DOI:doi:10.1016/j.patrec.2005.10.010.

Xavier Robin, Natacha Turck, Alexandre Hainard,et al.(2011) “pROC: an open-source package for R and S+ to analyze andcompare ROC curves”.BMC Bioinformatics,7, 77.DOI:doi:10.1186/1471-2105-12-77.

Hadley Wickham (2011) “The Split-Apply-Combine Strategy for Data Analysis”.Journal of Statistical Software,40, 1–29.URL:doi:10.18637/jss.v040.i01.

See Also

roc,ci,ci.se,plot.ci

Examples

# Create a ROC curve:data(aSAH)roc1 <- roc(aSAH$outcome, aSAH$s100b)## Basic example #### Not run: ci.sp(roc1)## End(Not run)## More options ### Customized bootstrap and sensitivities:## Not run: ci.sp(roc1, c(.95, .9, .85), boot.n=10000, conf.level=0.9, stratified=FALSE)## End(Not run)## Plotting the CI ##ci1 <- ci.sp(roc1, boot.n = 10)plot(roc1)plot(ci1)## On smoothed ROC curves with bootstrap #### Not run: ci.sp(smooth(roc1, method="density"))## End(Not run)

Compute the confidence interval of thresholds

Description

This function computes the confidence interval (CI) of the sensitivityand specificity of the thresholds given in argument.

Usage

# ci.thresholds(...)## S3 method for class 'roc'ci.thresholds(roc, conf.level=0.95, boot.n=2000,boot.stratified=TRUE, thresholds = "local maximas",progress = NULL, parallel=FALSE, ...) ## S3 method for class 'formula'ci.thresholds(formula, data, ...)## S3 method for class 'smooth.roc'ci.thresholds(smooth.roc, ...)## Default S3 method:ci.thresholds(response, predictor, ...)

Arguments

Details

ci.thresholds.formula andci.thresholds.default are convenience methodsthat build the ROC curve (with theroc function) beforecallingci.thresholds.roc. You can pass them arguments for bothroc andci.thresholds.roc. Simply useci.thresholdsthat will dispatch to the correct method.

This function createsboot.n bootstrap replicate of the ROCcurve, and evaluates the sensitivity and specificity at thresholdsgiven by thethresholds argument. Then it computes theconfidence interval as the percentiles given byconf.level.

A threshold given as alogical vector orcharacter is converted to the corresponding numeric vector onceusing the supplied ROC curve, and not at each bootstrap iteration. Seeci.coords for the latter behaviour.

For more details about the bootstrap, see the Bootstrap section inthis package's documentation.

Value

A list of length 2 and class “ci.thresholds”, “ci” and “list” (in this order), with the confidenceintervals of the CI and the following items:

Additionally, the list has the following attributes:

Warnings

Ifboot.stratified=FALSE and the sample has a large imbalance betweencases and controls, it could happen that one or more of the replicatescontains no case or control observation, producing aNA area.The warning “NA value(s) produced during bootstrap were ignored.”will be issued and the observation will be ignored. If you have a largeimbalance in your sample, it could be safer to keepboot.stratified=TRUE.

References

James Carpenter and John Bithell (2000) “Bootstrap condence intervals:when, which, what? A practical guide for medical statisticians”.Statistics in Medicine19, 1141–1164.DOI:doi:10.1002/(SICI)1097-0258(20000515)19:9<1141::AID-SIM479>3.0.CO;2-F.

Tom Fawcett (2006) “An introduction to ROC analysis”.PatternRecognition Letters27, 861–874. DOI:doi:10.1016/j.patrec.2005.10.010.

Xavier Robin, Natacha Turck, Alexandre Hainard,et al.(2011) “pROC: an open-source package for R and S+ to analyze andcompare ROC curves”.BMC Bioinformatics,7, 77.DOI:doi:10.1186/1471-2105-12-77.

Hadley Wickham (2011) “The Split-Apply-Combine Strategy for Data Analysis”.Journal of Statistical Software,40, 1–29.URL:doi:10.18637/jss.v040.i01.

See Also

roc,ci

Examples

data(aSAH)# Create a ROC curve:data(aSAH)roc1 <- roc(aSAH$outcome, aSAH$s100b)## Basic example ### Compute CI of all local maxima thresholds## Not run: ci.thresholds(roc1)## End(Not run)## More options ### Customized bootstrap and thresholds:## Not run: ci.thresholds(roc1,  thresholds=c(0.5, 1, 2),              boot.n=10000, conf.level=0.9, stratified=FALSE)## End(Not run)## Plotting the CI #### Not run: ci1 <- ci.thresholds(roc1)## End(Not run)plot(roc1)plot(ci1)

Coordinates of a ROC curve

Description

This function returns the coordinates of the ROC curve at oneor several specified point(s).

Usage

coords(...)## S3 method for class 'auc'coords(auc, ...)## S3 method for class 'roc'coords(roc, x, input="threshold", ret=c("threshold","specificity", "sensitivity"),  ignore.partial.auc=FALSE,as.list=FALSE, drop=TRUE, best.method=c("youden", "closest.topleft"),best.weights=c(1, 0.5), transpose = FALSE, as.matrix=FALSE, ...)## S3 method for class 'smooth.roc'coords(smooth.roc, x, input, ret=c("specificity","sensitivity"), ignore.partial.auc=FALSE, as.list=FALSE, drop=TRUE, best.method=c("youden", "closest.topleft"), best.weights=c(1, 0.5), transpose = FALSE, as.matrix=FALSE, ...)

Arguments

Details

This function takes a “roc” or “smooth.roc” object asfirst argument, on which the coordinates will be determined. Thecoordinates are defined by thex andinputarguments. “threshold” coordinates cannot be determined in asmoothed ROC.

Ifinput="threshold", the coordinates for the thresholdare reported, even if the exact threshold do not define the ROCcurve. The following convenience characters are allowed: “all”,“local maximas” and “best”. They will return all thethresholds, only the thresholds defining local maximas (upper angles of theROC curve), or only the threshold(s) corresponding to the best sum ofsensitivity + specificity respectively. Note that “best” canreturn more than one threshold. Ifx is a character, andignore.partial.auc=TRUE, the coordinates are limited to thethresholds within the partial AUC if it has been defined, and notnecessarily to the whole curve.

Forinput="specificity" andinput="sensitivity",the function checks if the specificity or sensitivity is one of thepoints of the ROC curve (inroc$sensitivities orroc$specificities). More than one point may match (instep curves), then only the upper-left-most point coordinatesare returned. Otherwise,the specificity and specificity of the point is interpolated andNA is returned as threshold.

The coords function in this package is a generic, but it might besuperseded by functions in other packages such ascolorspace orspatstat if they are loaded afterpROC. In this case, call thepROC::coords explicitly.

Best thresholds

Ifx="best", thebest.method argument controls how theoptimal threshold is determined.

“youden”

Youden's J statistic (Youden, 1950) is employed (default). The optimal cut-off is the threshold that maximizes the distance to the identity(diagonal) line. Can be shortened to “y”.

The optimality criterion is:

max(sensitivities + specificities)

“closest.topleft”

The optimal threshold is the point closest to the top-left part ofthe plot with perfect sensitivity or specificity. Can be shortenedto “c” or “t”.

The optimality criterion is:

min((1 - sensitivities)^2 + (1- specificities)^2)

In addition, weights can be supplied if false positive and falsenegative predictions are not equivalent: a numeric vector of length 2to thebest.weights argument. The elements define

1. the relative cost of of a false negative classification (as compared with a false positive classification)

2. the prevalence, or the proportion of cases in the population (\frac{n_{cases}}{n_{controls}+n_{cases}}).

The optimality criteria are modified as proposed by Perkins and Schisterman:

“youden”

max(sensitivities + r * specificities)

“closest.topleft”

min((1 - sensitivities)^2 + r * (1- specificities)^2)

with

r = \frac{1 - prevalence}{cost * prevalence}

By default, prevalence is 0.5 and cost is 1 so that no weight isapplied in effect.

Note that several thresholds might be equally optimal.

Available coordinates

The following table lists the coordinates that are available in theretandinput arguments.

The value “threshold” is not allowed incoords.smooth.roc.

Values can be shortenend (for example to “thr”, “sens” and “spec”, or even to“se”, “sp” or “1-np”). In addition, some values can be prefixed with1- to get their complement:1-specificity,1-sensitivity,1-accuracy,1-npv,1-ppv.

The valuesnpe andppe are automatically replaced with1-npv and1-ppv, respectively (and will therefore not appearas is in the output, but as1-npv and1-ppv instead).These must be used verbatim in ROC curves withpercent=TRUE (ie. “100-ppv” is never accepted).

The “youden” and “closest.topleft” are weighted withr, according to the value of thebest.weights argument. See the “Best thresholds” section above for more details.

Forret, the single value “all” can be used to returnevery coordinate available.

Value

Adata.frame withret as columns and as many rows asgiven byx.

In all cases whereinput="specificity" orinput="sensitivity"and interpolation was required, threshold is returned asNA.

Note that if giving a character asx (“all”,“local maximas” or “best”), you cannot predict the dimension ofthe return value. Even “best” may return more than one value (forexample if the ROC curve is below the identity line, both extreme points).

References

Neil J. Perkins, Enrique F. Schisterman (2006) “The Inconsistency of "Optimal" CutpointsObtained using Two Criteria based on the Receiver OperatingCharacteristic Curve”.American Journal of Epidemiology163(7), 670–675. DOI:doi:10.1093/aje/kwj063.

Xavier Robin, Natacha Turck, Alexandre Hainard,et al.(2011) “pROC: an open-source package for R and S+ to analyze andcompare ROC curves”.BMC Bioinformatics,7, 77.DOI:doi:10.1186/1471-2105-12-77.

W. J. Youden (1950) “Index for rating diagnostic tests”.Cancer,3, 32–35. DOI:doi:10.1002/1097-0142(1950)3:1<32::AID-CNCR2820030106>3.0.CO;2-3.

See Also

roc,ci.coords

Examples

# Create a ROC curve:data(aSAH)roc.s100b <- roc(aSAH$outcome, aSAH$s100b, percent = TRUE)# Get the coordinates of S100B threshold 0.55coords(roc.s100b, 0.55)# Get the coordinates at 50% sensitivitycoords(roc=roc.s100b, x=50, input="sensitivity")# Can be abbreviated:coords(roc.s100b, 50, "se")# Works with smoothed ROC curvescoords(smooth(roc.s100b), 90, "specificity")# Get the sensitivities for all thresholdscc <- coords(roc.s100b, "all", ret="sensitivity")print(cc$sensitivity)# Get the best thresholdcoords(roc.s100b, "best", ret="threshold")# Get the best threshold according to different methodsroc.ndka <- roc(aSAH$outcome, aSAH$ndka, percent=TRUE)coords(roc.ndka, "best", ret="threshold",        best.method="youden") # defaultcoords(roc.ndka, "best", ret="threshold",        best.method="closest.topleft")# and with different weightscoords(roc.ndka, "best", ret="threshold",        best.method="youden", best.weights=c(50, 0.2))coords(roc.ndka, "best", ret="threshold",        best.method="closest.topleft", best.weights=c(5, 0.2))       # This is available with the plot.roc function too:plot(roc.ndka, print.thres="best", print.thres.best.method="youden",                                 print.thres.best.weights=c(50, 0.2)) # Return more values:coords(roc.s100b, "best", ret=c("threshold", "specificity", "sensitivity", "accuracy",                           "precision", "recall"))# Return all valuescoords(roc.s100b, "best", ret = "all")                           # You can use coords to plot for instance a sensitivity + specificity vs. cut-off diagramplot(specificity + sensitivity ~ threshold,      coords(roc.ndka, "all"),      type = "l", log="x",      subset = is.finite(threshold))# Plot the Precision-Recall curveplot(precision ~ recall,      coords(roc.ndka, "all", ret = c("recall", "precision")),     type="l", ylim = c(0, 100))# Alternatively plot the curve with TPR and FPR instead of SE/SP # (identical curve, only the axis change)plot(tpr ~ fpr,      coords(roc.ndka, "all", ret = c("tpr", "fpr")),     type="l")

Transposing the output ofcoords

Description

This help page desribes recent and upcoming changes in the return values of thecoords function.

Background information

Until the release of pROC 1.16, thecoords function was returning a matrix with thresholds in columns, and the coordinate variables in rows.

data(aSAH)rocobj <- roc(aSAH$outcome, aSAH$s100b)coords(rocobj, c(0.05, 0.2, 0.5))#                   0.05       0.2       0.5# threshold   0.05000000 0.2000000 0.5000000# specificity 0.06944444 0.8055556 0.9722222# sensitivity 0.97560976 0.6341463 0.2926829

This format didn't conform to the grammar of thetidyverse which has become prevalent in modernR language.

In addition, the dropping of dimensions by default makes it difficult to guesswhat type of datacoords is going to return.

coords(rocobj, "best")#   threshold specificity sensitivity #   0.2050000   0.8055556   0.6341463 # A numeric vector

Although it is possible to passdrop = FALSE, the fact that it is not thedefault makes the behaviour unintuitive.

In pROC version 1.16, this was changed andcoords now returnsadata.frame with the thresholds in rows and measurement in colums by default.

 coords(rocobj, c(0.05, 0.2, 0.5), transpose = FALSE)#      threshold specificity sensitivity# 0.05      0.05  0.06944444   0.9756098# 0.2       0.20  0.80555556   0.6341463# 0.5       0.50  0.97222222   0.2926829

Changes in 1.15

1. Addition of thetranspose argument.

2. Display a warning iftranspose is missing. Passtranspose explicitly to silence the warning.

3. Deprecation ofas.list.

Changes in 1.16

• Change of the defaulttranspose toTRUE.

THIS CHANGE IS BACKWARDS INCOMPATIBLE AND IS EXPECTED TO BREAK LEGACY CODE.

Changes in 1.17

• Dropped the warning iftranspose is missing.

Changes in 1.19

1. Settingtranspose=TRUE is deprecated and triggers a warning.

2. Theas.list,as.matrix anddrop are deprecated.Setting them to any value triggers a waring.

3. transpose=FALSE continues to work normally.

Changes in future versions

1. Settingtranspose toTRUE will stop working and resultin an error.

2. Thedrop,as.list andas.matrix arguments will be removed.

3. transpose=FALSE will keep working indefinitely.

Related changes in ci.coords

In version 1.16, the format of theci.coords return value was changed from a matrix-like object with mixedx andret in rows and 3 columns, into a list-like object which should be easier to use programatically.

See also

The GitHub issue tracking the changes described in this manual page.

Covariance of two paired ROC curves

Description

This function computes the covariance between the AUC of two correlated (or paired) ROCcurves.

Usage

cov(...)## Default S3 method:cov(...)## S3 method for class 'auc'cov(roc1, roc2, ...)## S3 method for class 'smooth.roc'cov(roc1, roc2, ...)## S3 method for class 'roc'cov(roc1, roc2, method=c("delong", "bootstrap", "obuchowski"),  reuse.auc=TRUE, boot.n=2000, boot.stratified=TRUE, boot.return=FALSE,  progress=NULL, parallel=FALSE, ...)

Arguments

Details

This function computes the covariance between the AUC of twocorrelated (or paired, according to the detection ofare.paired) ROCcurves. It is typically called with the tworoc objects ofinterest. Two methods are available: “delong” and“bootstrap” (see “Computationaldetails” section below).

The default is to use “delong” method except withpartial AUC and smoothed curves where “bootstrap” is employed.Using “delong” for partial AUC and smoothed ROCs is notsupported.

Forsmoothed ROC curves, smoothing is performed again at eachbootstrap replicate with the parameters originally provided.If a density smoothing was performed with user-provideddensity.cases ordensity.controls the bootstrap cannotbe performed and an error is issued.

cov.default forces the usage of thecov function in thestats package, sothat other code relying oncov should continue to functionnormally.

Value

The numeric value of the covariance.

Ifboot.return=TRUE andmethod="bootstrap", an attributeresampled.values is set with the resampled (bootstrapped)values. It contains a matrix with the columns representing the two ROCcurves, and the rows theboot.n bootstrap replicates.

AUC specification

To compute the covariance of the AUC of the ROC curves,cov needs a specification of theAUC. The specification is defined by:

1. the “auc” field in the “roc” objects ifreuse.auc is set toTRUE (default)

2. passing the specification toauc with ...(argumentspartial.auc,partial.auc.correct andpartial.auc.focus). In this case, you must ensure either thattheroc object do not contain anauc field (ifyou calledroc withauc=FALSE), or setreuse.auc=FALSE.

Ifreuse.auc=FALSE theauc function will alwaysbe called with... to determine the specification, even ifthe “roc” objects do contain anauc field.

As well if the “roc” objects do not contain anaucfield, theauc function will always be called with... to determine the specification.

Warning: if the roc object passed to roc.test contains anaucfield andreuse.auc=TRUE,auc is not called andarguments such aspartial.auc are silently ignored.

Computation details

Withmethod="bootstrap", the processing is done as follow:

1. boot.n bootstrap replicates are drawn from thedata. Ifboot.stratified isTRUE, each replicate containsexactly the same number of controls and cases than the originalsample, otherwise ifFALSE the numbers can vary.

2. for each bootstrap replicate, the AUC of the two ROC curvesare computed and stored.

3. the variance (as pervar.roc) of the resampledAUCs and their covariance are assessed in a single bootstrap pass.

4. The following formula is used to compute the final covariance:Var[AUC1] + Var[AUC2] - 2cov[AUC1,AUC2]

Withmethod="delong", the processing is done as described inHanley and Hajian-Tilaki (1997) using the algorithm by Sun and Xu (2014).

Withmethod="obuchowski", the processing is done as describedin Obuchowski and McClish (1997), Table 1 and Equation 5, p. 1531. Thecomputation ofg for partial area under the ROC curve ismodified as:

expr1 * (2 * pi * expr2) ^ {(-1)} * (-expr4) - A * B * expr1 * (2 * pi * expr2^3) ^ {(-1/2)} * expr3

.

Binormality assumption

The “obuchowski” method makes the assumption that the data is binormal.If the data shows a deviation from this assumption, it might help tonormalize the data first (that is, before callingroc),for example with quantile normalization:

    norm.x <- qnorm(rank(x)/(length(x)+1))    cov(roc(response, norm.x, ...), ...)

“delong” and “bootstrap” methods make no such assumption.

Errors

Ifdensity.cases anddensity.controls were providedfor smoothing, the error “Cannot compute the covariance on ROCcurves smoothed with density.controls and density.cases.” isissued.

Warnings

If “auc” specifications are different in both roc objects, thewarning “Different AUC specifications in the ROCcurves. Enforcing the inconsistency, but unexpected results may beproduced.” is issued. Unexpected results may be produced.

If one or both ROC curves are “smooth.roc” objects withdifferent smoothing specifications, the warning “Different smoothing parameters in the ROC curves. Enforcingthe inconsistency, but unexpected results may be produced.” is issued.This warning can be benign, especially if ROC curves were generatedwithroc(..., smooth=TRUE) with different arguments to otherfunctions (such as plot), or if you really want to compare two ROCcurves smoothed differently.

Ifmethod="delong" and the AUC specification specifies apartial AUC, the warning “Using DeLong for partial AUC isnot supported. Using bootstrap test instead.” is issued. Themethod argument is ignored and “bootstrap” is used instead.

Ifmethod="delong" and the ROCcurve is smoothed, the warning “Using DeLong forsmoothed ROCs is not supported. Using bootstrap instead.” isissued. Themethod argument is ignored and “bootstrap”is used instead.

DeLong ignores the direction of the ROC curve so that if twoROC curves have a differentdirection, the warning“"DeLong should not be applied to ROC curves with a differentdirection."” is printed. However, the spurious computation is enforced.

Ifboot.stratified=FALSE and the sample has a large imbalance betweencases and controls, it could happen that one or more of the replicatescontains no case or control observation, or that there are not enoughpoints for smoothing, producing aNA area.The warning “NA value(s) produced during bootstrap were ignored.”will be issued and the observation will be ignored. If you have a largeimbalance in your sample, it could be safer to keepboot.stratified=TRUE.

When both ROC curves have anauc of 1 (or 100%), their covariance will always be null.This is true for both “delong” and “bootstrap” and methods. This result is misleading,as the covariance is of course not null.Awarning will be displayed to inform of this condition, and of the misleading output.

Messages

The covariance can only be computed on paired data. Thisassumption is enforced byare.paired. If the ROC curvesare not paired, the covariance is0 and the message “ROCcurves are unpaired.” is printed. If your ROC curves are paired, makesure they fitare.paired criteria.

References

Elisabeth R. DeLong, David M. DeLong and Daniel L. Clarke-Pearson(1988) “Comparing the areas under two or more correlated receiveroperating characteristic curves: a nonparametricapproach”.Biometrics44, 837–845.

James A. Hanley and Karim O. Hajian-Tilaki (1997) “Samplingvariability of nonparametric estimates of the areas under receiveroperating characteristic curves: An update”.AcademicRadiology4, 49–58. DOI:doi:10.1016/S1076-6332(97)80161-4.

Nancy A. Obuchowski, Donna K. McClish (1997). “Sample sizedetermination for diagnostic accurary studies involving binormal ROCcurve indices”.Statistics in Medicine,16(13),1529–1542. DOI:doi:10.1002/(SICI)1097-0258(19970715)16:13<1529::AID-SIM565>3.0.CO;2-H.

Xu Sun and Weichao Xu (2014) “Fast Implementation of DeLongs Algorithm for Comparingthe Areas Under Correlated Receiver Operating Characteristic Curves”.IEEE SignalProcessing Letters,21, 1389–1393. DOI:doi:10.1109/LSP.2014.2337313.

Hadley Wickham (2011) “The Split-Apply-Combine Strategy for Data Analysis”.Journal of Statistical Software,40, 1–29.URL:doi:10.18637/jss.v040.i01.

See Also

roc,var.roc

Examples

data(aSAH)# Basic example with 2 roc objectsroc1 <- roc(aSAH$outcome, aSAH$s100b)roc2 <- roc(aSAH$outcome, aSAH$wfns)cov(roc1, roc2)## Not run: # The latter used Delong. To use bootstrap:cov(roc1, roc2, method="bootstrap")# Decrease boot.n for a faster execution:cov(roc1, roc2, method="bootstrap", boot.n=1000)## End(Not run)# To use Obuchowski:cov(roc1, roc2, method="obuchowski")## Not run: # Comparison can be done on smoothed ROCs# Smoothing is re-done at each iteration, and execution is slowcov(smooth(roc1), smooth(roc2))## End(Not run)# or from an AUC (no smoothing)cov(auc(roc1), roc2)## Not run: # With bootstrap and return.values, one can compute the variances of the# ROC curves in one single bootstrap run:cov.rocs <- cov(roc1, roc2, method="bootstrap", boot.return=TRUE)# var(roc1):var(attr(cov.rocs, "resampled.values")[,1])# var(roc2):var(attr(cov.rocs, "resampled.values")[,2])## End(Not run)## Not run: # Covariance of partial AUC:roc3 <- roc(aSAH$outcome, aSAH$s100b, partial.auc=c(1, 0.8), partial.auc.focus="se")roc4 <- roc(aSAH$outcome, aSAH$wfns, partial.auc=c(1, 0.8), partial.auc.focus="se")cov(roc3, roc4)# This is strictly equivalent to:cov(roc3, roc4, method="bootstrap")# Alternatively, we could re-use roc1 and roc2 to get the same result:cov(roc1, roc2, reuse.auc=FALSE, partial.auc=c(1, 0.8), partial.auc.focus="se")## End(Not run)# Spurious use of DeLong's test with different direction:roc5 <- roc(aSAH$outcome, aSAH$s100b, direction="<")roc6 <- roc(aSAH$outcome, aSAH$s100b, direction=">")cov(roc5, roc6, method="delong")## Test data from Hanley and Hajian-Tilaki, 1997disease.present <- c("Yes", "No", "Yes", "No", "No", "Yes", "Yes", "No",                     "No", "Yes", "No", "No", "Yes", "No", "No")field.strength.1 <- c(1, 2, 5, 1, 1, 1, 2, 1, 2, 2, 1, 1, 5, 1, 1)field.strength.2 <- c(1, 1, 5, 1, 1, 1, 4, 1, 2, 2, 1, 1, 5, 1, 1)roc7 <- roc(disease.present, field.strength.1)roc8 <- roc(disease.present, field.strength.2)# Assess the covariance:cov(roc7, roc8)## Not run: # With bootstrap:cov(roc7, roc8, method="bootstrap")## End(Not run)

Add an AUC polygon to a ggroc plot

Description

EXPERIMENTAL - Add an AUC polygon to a ggroc plot.

Usage

## S3 method for class 'auc'geom_polygon_auc(data, legacy.axes = FALSE, ...)## S3 method for class 'roc'geom_polygon_auc(data, ...)## S3 method for class 'smooth.roc'geom_polygon_auc(data, ...)

Arguments

Details

This is highly experimental and may change in the future.

See Also

ggroc

Examples

# Create a ROC curve:data(aSAH)roc.s100b <- roc(aSAH$outcome, aSAH$s100b)roc.s100b.percent <- roc(aSAH$outcome, aSAH$s100b, percent = TRUE)ggroc(roc.s100b) + geom_polygon_auc(roc.s100b$auc)# legacy.axes must be repeatedggroc(roc.s100b.percent, legacy.axes=TRUE) + geom_polygon_auc(roc.s100b.percent, legacy.axes=TRUE)# Partial AUCsauc.s100b.partial.sp <- auc(roc.s100b, partial.auc = c(0.9, 1))auc.s100b.partial.se <- auc(roc.s100b, partial.auc = c(0.8, 0.9), partial.auc.focus="se")ggroc(roc.s100b) + geom_polygon_auc(auc.s100b.partial.sp)ggroc(roc.s100b) + geom_polygon_auc(auc.s100b.partial.se)

Plot a ROC curve with ggplot2

Description

This function plots a ROC curve with ggplot2.

Usage

## S3 method for class 'roc'ggroc(data, legacy.axes = FALSE, ...)## S3 method for class 'smooth.roc'ggroc(data, legacy.axes = FALSE, ...)## S3 method for class 'list'ggroc(data, aes = c("colour", "alpha", "linetype", "linewidth", "size", "group"),      legacy.axes = FALSE, ...)

Arguments

Details

This function initializes a ggplot object from a ROC curve (or multiple if a list is passed). It returns the ggplot with a line layer on it. You can print it directly or add your own layers and theme elements.

See Also

roc,plot.roc,ggplot2

Examples

# Create a basic roc objectdata(aSAH)rocobj <- roc(aSAH$outcome, aSAH$s100b)rocobj2 <- roc(aSAH$outcome, aSAH$wfns)if (require(ggplot2)) {g <- ggroc(rocobj)g# with additional aesthetics:ggroc(rocobj, alpha = 0.5, colour = "red", linetype = 2, size = 2)# You can then your own theme, etc.g + theme_minimal() + ggtitle("My ROC curve") +     geom_segment(aes(x = 1, xend = 0, y = 0, yend = 1), color="grey", linetype="dashed")# And change axis labels to FPR/FPRgl <- ggroc(rocobj, legacy.axes = TRUE)glgl + xlab("FPR") + ylab("TPR") +     geom_segment(aes(x = 0, xend = 1, y = 0, yend = 1), color="darkgrey", linetype="dashed")# Multiple curves:g2 <- ggroc(list(s100b=rocobj, wfns=rocobj2, ndka=roc(aSAH$outcome, aSAH$ndka)))g2# This is equivalent to using roc.formula:roc.list <- roc(outcome ~ s100b + ndka + wfns, data = aSAH)g.list <- ggroc(roc.list)g.list# You can change the aesthetics as you normally would with ggplot2:g.list + scale_colour_brewer(palette="RdGy")g.list + scale_colour_manual(values = c("red", "blue", "black"))# with additional aesthetics:g3 <- ggroc(roc.list, linetype=2)g3g4 <- ggroc(roc.list, aes="linetype", color="red")g4# changing multiple aesthetics:g5 <- ggroc(roc.list, aes=c("linetype", "color"))g5# OR facetingg.list + facet_grid(.~name) + theme(legend.position="none")}

pROC Group Generic Functions

Description

Redefinebase groupGeneric functions to handleauc andci objects properly on operations and mathematical operations.Attributes are dropped so that the AUC/CI behaves as a numeric value/matrix, respectively.In the case of AUC, all attributes are dropped, while in CI only the CI-specific attributes are, keeping those necessary for the matrices.

Usage

Math(x, ...)Ops(e1, e2)

Arguments

See Also

groupGeneric,auc

Examples

data(aSAH)# Create a roc object:aucobj1 <- auc(roc(aSAH$outcome, aSAH$s100b))aucobj2 <- auc(roc(aSAH$outcome, aSAH$wfns))# Mathsqrt(aucobj1)round(aucobj2, digits=1)# Opsaucobj1 * 22 * aucobj2aucobj1 + aucobj2# With CIciaucobj <- ci(aucobj1)ciaucobj * 2sqrt(ciaucobj)

Does the ROC curve have a partial AUC?

Description

This function determines if the ROC curve has a partial AUC.

Usage

has.partial.auc(roc)## S3 method for class 'auc'has.partial.auc(roc)## S3 method for class 'smooth.roc'has.partial.auc(roc)## S3 method for class 'roc'has.partial.auc(roc)

Arguments

Value

TRUE if the AUC is a partial AUC,FALSE otherwise.

If the AUC is not defined (i. e. if roc was called withAUC=FALSE), returnsNULL.

See Also

auc

Examples

data(aSAH)# Full AUCroc1 <- roc(aSAH$outcome, aSAH$s100b)has.partial.auc(roc1)has.partial.auc(auc(roc1))has.partial.auc(smooth(roc1))# Partial AUCroc2 <- roc(aSAH$outcome, aSAH$s100b, partial.auc = c(1, 0.9))has.partial.auc(roc2)has.partial.auc(smooth(roc2))# No AUCroc3 <- roc(aSAH$outcome, aSAH$s100b, auc = FALSE)has.partial.auc(roc3)

Add a ROC line to a ROC plot

Description

This convenience function adds a ROC line to a ROC curve.

Usage

## S3 method for class 'roc'lines(x, ...)## S3 method for class 'smooth.roc'lines(x, ...)## S3 method for class 'roc'lines.roc(x, lwd=2, ...)## S3 method for class 'formula'lines.roc(x, data, subset, na.action, ...)## Default S3 method:lines.roc(x, predictor, ...)## S3 method for class 'smooth.roc'lines.roc(x, ...)

Arguments

Value

This function returns a list of class “roc” invisibly. Seeroc for more details.

See Also

roc,plot.roc

Examples

# Create a few ROC curves:data(aSAH)roc.s100b <- roc(aSAH$outcome, aSAH$s100b)roc.wfns <- roc(aSAH$outcome, aSAH$wfns)# We need a plot to be readyplot(roc.s100b, type = "n") # but don't actually plot the curve# Add the linelines(roc.s100b, type="b", pch=21, col="blue", bg="grey")# Add the line of an other ROC curvelines(roc.wfns, type="o", pch=19, col="red")# Without using 'lines':rocobj <- plot.roc(aSAH$outcome, aSAH$s100b, type="b", pch=21, col="blue", bg="grey")

Multi-class AUC

Description

This function builds builds multiple ROC curve to compute themulti-class AUC as defined by Hand and Till.

Usage

multiclass.roc(...)## S3 method for class 'formula'multiclass.roc(formula, data, ...)## Default S3 method:multiclass.roc(response, predictor,levels=base::levels(as.factor(response)), percent=FALSE, direction = c("auto", "<", ">"), ...)

Arguments

Details

This function performs multiclass AUC as defined by Hand and Till(2001). A multiclass AUC is a mean of severalauc andcannot be plotted. Only AUCs can be computed for such curves.Confidence intervals, standard deviation, smoothing andcomparison tests are not implemented.

Themulticlass.roc function can handle two types of datasets: uni- and multi-variate.In the univariate case, a singlepredictor vector is passedand all the combinations of responses are assessed.I the multivariate case, amatrix ordata.frameis passed aspredictor. The columns must be named according to thelevels of theresponse.

This function has been much less tested than the rest of the package andis more subject to bugs. Please report them if you find one.

Value

Ifpredictor is a vector, a list of class “multiclass.roc” (univariate) or “mv.multiclass.roc” (multivariate), with the following fields:

Warnings

Ifresponse is an ordered factor and one of the levelsspecified inlevels is missing, a warning is issued and thelevel is ignored.

References

David J. Hand and Robert J. Till (2001). A Simple Generalisation ofthe Area Under the ROC Curve for Multiple Class ClassificationProblems.Machine Learning45(2), p. 171–186. DOI:doi:10.1023/A:1010920819831.

See Also

auc

Examples

##### Examples for a univariate decision value####data(aSAH)# Basic examplemulticlass.roc(aSAH$gos6, aSAH$s100b)# Produces an innocuous warning because one level has no observation# Select only 3 of the aSAH$gos6 levels:multiclass.roc(aSAH$gos6, aSAH$s100b, levels=c(3, 4, 5))# Give the result in percentmulticlass.roc(aSAH$gos6, aSAH$s100b, percent=TRUE)##### Examples for multivariate decision values (e.g. class probabilities)###### Not run: # Example with a multinomial log-linear model from nnet# We use the iris dataset and split into a training and test setrequireNamespace("nnet")data(iris)iris.sample <- sample(1:150)iris.train <- iris[iris.sample[1:75],]iris.test <- iris[iris.sample[76:150],]mn.net <- nnet::multinom(Species ~ ., iris.train)# Use predict with type="prob" to get class probabilitiesiris.predictions <- predict(mn.net, newdata=iris.test, type="prob")head(iris.predictions)# This can be used directly in multiclass.roc:multiclass.roc(iris.test$Species, iris.predictions)## End(Not run)# Let's see an other example with an artificial datasetn <- c(100, 80, 150)responses <- factor(c(rep("X1", n[1]), rep("X2", n[2]), rep("X3", n[3])))# construct prediction matrix: one column per classpreds <- lapply(n, function(x) runif(x, 0.4, 0.6))predictor <- as.matrix(data.frame(                "X1" = c(preds[[1]], runif(n[2] + n[3], 0, 0.7)),                "X2" = c(runif(n[1], 0.1, 0.4), preds[[2]], runif(n[3], 0.2, 0.8)),                "X3" = c(runif(n[1] + n[2], 0.3, 0.7), preds[[3]])             ))multiclass.roc(responses, predictor)# One can change direction , partial.auc, percent, etc:multiclass.roc(responses, predictor, direction = ">")multiclass.roc(responses, predictor, percent = TRUE, partial.auc = c(100, 90), partial.auc.focus = "se")# Limit set of levelsmulticlass.roc(responses, predictor, levels = c("X1", "X2"))# Use with formula. Here we need a data.frame to store the responses as charactersdata <- cbind(as.data.frame(predictor), "response" = responses)multiclass.roc(response ~ X1+X3, data)

Plot confidence intervals

Description

This function adds confidence intervals to a ROC curve plot, either asbars or as a confidence shape.

Usage

## S3 method for class 'ci.thresholds'plot(x, length=.01*ifelse(attr(x,  "roc")$percent, 100, 1), col=par("fg"), ...)## S3 method for class 'ci.sp'plot(x, type=c("bars", "shape"), length=.01*ifelse(attr(x,"roc")$percent, 100, 1), col=ifelse(type=="bars", par("fg"),"gainsboro"), no.roc=FALSE, ...)## S3 method for class 'ci.se'plot(x, type=c("bars", "shape"), length=.01*ifelse(attr(x,"roc")$percent, 100, 1), col=ifelse(type=="bars", par("fg"),"gainsboro"), no.roc=FALSE, ...)## S3 method for class 'ci.coords'plot(x, type=c("bars", "shape"), length=NULL,col=ifelse(type=="bars", par("fg"), "gainsboro"), ...)

Arguments

Details

This function adds confidence intervals to a ROC curve plot, either asbars or as a confidence shape, depending on the state of thetype argument. The shape is plotted over the ROC curve, so thatthe curve is re-plotted unlessno.roc=TRUE.

Graphical functions are called withsuppressWarnings.

Value

This function returns the confidence interval object invisibly.

Warnings

Withtype="shape", the warning “Low definition shape” isissued when the shape is defined by less than 15 confidenceintervals. In such a case, the shape is not well defined and the ROCcurve could pass outside the shape. To get a better shape, increasethe number of intervals, for example with:

plot(ci.sp(rocobj, sensitivities=seq(0, 1, .01)), type="shape")

References

Xavier Robin, Natacha Turck, Alexandre Hainard,et al.(2011) “pROC: an open-source package for R and S+ to analyze andcompare ROC curves”.BMC Bioinformatics,7, 77.DOI:doi:10.1186/1471-2105-12-77.

See Also

plot.roc,ci.thresholds,ci.sp,ci.se

Examples

data(aSAH)## Not run: # Start a ROC plotrocobj <- plot.roc(aSAH$outcome, aSAH$s100b)plot(rocobj)# Thresholdsci.thresolds.obj <- ci.thresholds(rocobj)plot(ci.thresolds.obj)# Specificitiesplot(rocobj) # restart a new plotci.sp.obj <- ci.sp(rocobj, boot.n=500)plot(ci.sp.obj)# Sensitivitiesplot(rocobj) # restart a new plotci.se.obj <- ci(rocobj, of="se", boot.n=500)plot(ci.se.obj)# Plotting a shape. We need moreci.sp.obj <- ci.sp(rocobj, sensitivities=seq(0, 1, .01), boot.n=100)plot(rocobj) # restart a new plotplot(ci.sp.obj, type="shape", col="blue")# Direct syntax (response, predictor):plot.roc(aSAH$outcome, aSAH$s100b,         ci=TRUE, of="thresholds")# CI of a PR curveco <- coords(rocobj, x = "all", input="recall", ret=c("recall", "precision"))ci <- ci.coords(rocobj, x = seq(0, 1, .1), input="recall", ret="precision")plot(co, type="l", ylim = c(0, 1))plot(ci, type="shape")plot(ci, type="bars")lines(co)## End(Not run)

Plot a ROC curve

Description

This function plots a ROC curve. It can accept many arguments totweak the appearance of the plot. Two syntaxes are possible: oneobject of class “roc”, or either two vectors (response, predictor) or aformula (response~predictor) as in theroc function.

Usage

## S3 method for class 'roc'plot(x, ...)## S3 method for class 'smooth.roc'plot(x, ...)## S3 method for class 'roc'plot.roc(x, add=FALSE, reuse.auc=TRUE,axes=TRUE, legacy.axes=FALSE,# Generic arguments for par:xlim=if(x$percent){c(100, 0)} else{c(1, 0)},ylim=if(x$percent){c(0, 100)} else{c(0, 1)},xlab=ifelse(x$percent, ifelse(legacy.axes, "100 - Specificity (%)", "Specificity (%)"),            ifelse(legacy.axes, "1 - Specificity", "Specificity")),ylab=ifelse(x$percent, "Sensitivity (%)", "Sensitivity"),asp=1,mar=c(4, 4, 2, 2)+.1,mgp=c(2.5, 1, 0),# col, lty and lwd for the ROC line onlycol=par("col"),lty=par("lty"),lwd=2,type="l",# Identity lineidentity=!add,identity.col="darkgrey",identity.lty=1,identity.lwd=1,# Print the thresholds on the plotprint.thres=FALSE,print.thres.pch=20,print.thres.adj=c(-.05,1.25),print.thres.col="black",print.thres.pattern=ifelse(x$percent, "%.1f (%.1f%%, %.1f%%)", "%.3f (%.3f, %.3f)"),print.thres.cex=par("cex"),print.thres.pattern.cex=print.thres.cex,print.thres.best.method=NULL,print.thres.best.weights=c(1, 0.5),# Print the AUC on the plotprint.auc=FALSE,print.auc.pattern=NULL,print.auc.x=ifelse(x$percent, 50, .5),print.auc.y=ifelse(x$percent, 50, .5),print.auc.adj=c(0,1),print.auc.col=col,print.auc.cex=par("cex"),# Gridgrid=FALSE,grid.v={if(is.logical(grid) && grid[1]==TRUE)          {seq(0, 1, 0.1) * ifelse(x$percent, 100, 1)}        else if(is.numeric(grid))           {seq(0, ifelse(x$percent, 100, 1), grid[1])} else {NULL}},grid.h={if (length(grid) == 1) {grid.v}         else if (is.logical(grid) && grid[2]==TRUE)          {seq(0, 1, 0.1) * ifelse(x$percent, 100, 1)}         else if(is.numeric(grid))          {seq(0, ifelse(x$percent, 100, 1), grid[2])} else {NULL}},grid.lty=3,grid.lwd=1,grid.col="#DDDDDD",# Polygon for the AUCauc.polygon=FALSE,auc.polygon.col="gainsboro",auc.polygon.lty=par("lty"),auc.polygon.density=NULL,auc.polygon.angle=45,auc.polygon.border=NULL,# Polygon for the maximal AUC possible                           max.auc.polygon=FALSE,max.auc.polygon.col="#EEEEEE", max.auc.polygon.lty=par("lty"),max.auc.polygon.density=NULL,max.auc.polygon.angle=45,max.auc.polygon.border=NULL,# Confidence intervalci=!is.null(x$ci),ci.type=c("bars", "shape", "no"),ci.col=ifelse(ci.type=="bars", par("fg"), "gainsboro"),...)## S3 method for class 'formula'plot.roc(x, data, subset, na.action, ...)## Default S3 method:plot.roc(x, predictor, ...)## S3 method for class 'smooth.roc'plot.roc(x, ...)

Arguments

Details

This function is typically called fromroc when plot=TRUE (not bydefault).plot.roc.formula andplot.roc.default are convenience methodsthat build the ROC curve (with theroc function) beforecallingplot.roc.roc. You can pass them arguments for bothroc andplot.roc.roc. Simply useplot.rocthat will dispatch to the correct method.

The plotting is done in the following order:

1. A new plot is created ifadd=FALSE.

2. The grid is added ifgrid.v andgrid.h are not NULL.

3. The maximal AUC polygon is added ifmax.auc.polygon=TRUE.

4. The CI shape is added ifci=TRUE,ci.type="shape" andx$ci isn't a “ci.auc”.

5. The AUC polygon is added ifauc.polygon=TRUE.

6. The identity line ifidentity=TRUE.

7. The actual ROC line is added.

8. The CI bars are added ifci=TRUE,ci.type="bars" andx$ci isn't a “ci.auc”.

9. The selected thresholds are printed ifprint.thres isTRUE or numeric.

10. The AUC is printed ifprint.auc=TRUE.

Graphical functions are called withsuppressWarnings.

Value

This function returns a list of class “roc” invisibly. Seeroc for more details.

AUC specification

Forprint.auc,auc.polygon andmax.auc.polygonarguments, an AUC specification isrequired. By default, the total AUC is plotted, but you may want apartial AUCs. The specification is defined by:

1. the “auc” field in the “roc” object ifreuse.auc is set toTRUE (default). It is naturallyinherited from any call toroc and fits most cases.

2. passing the specification toauc with ...(argumentspartial.auc,partial.auc.correct andpartial.auc.focus). In this case, you must ensure either thattheroc object do not contain anauc field (ifyou calledroc withauc=FALSE), or setreuse.auc=FALSE.

Ifreuse.auc=FALSE theauc function will alwaysbe called with... to determine the specification, even ifthe “roc” object do contain anauc field.

As well if the “roc” object do not contain anaucfield, theauc function will always be called with... to determine the specification.

Warning: if the roc object passed to plot.roc contains anaucfield andreuse.auc=TRUE,auc is not called andarguments such aspartial.auc are silently ignored.

References

Xavier Robin, Natacha Turck, Alexandre Hainard,et al.(2011) “pROC: an open-source package for R and S+ to analyze andcompare ROC curves”.BMC Bioinformatics,7, 77.DOI:doi:10.1186/1471-2105-12-77.

See Also

roc,auc,ci

Examples

# Create a few ROC curves:data(aSAH)roc.s100b <- roc(aSAH$outcome, aSAH$s100b)roc.wfns <- roc(aSAH$outcome, aSAH$wfns)roc.ndka <- roc(aSAH$outcome, aSAH$wfns)# Simple example:plot(roc.s100b)# Add a smoothed ROC:plot(smooth(roc.s100b), add=TRUE, col="blue")legend("bottomright", legend=c("Empirical", "Smoothed"),       col=c(par("fg"), "blue"), lwd=2)# With more options:plot(roc.s100b, print.auc=TRUE, auc.polygon=TRUE, grid=c(0.1, 0.2),     grid.col=c("green", "red"), max.auc.polygon=TRUE,     auc.polygon.col="lightblue", print.thres=TRUE)# To plot a different partial AUC, we need to ignore the existing value# with reuse.auc=FALSE:plot(roc.s100b, print.auc=TRUE, auc.polygon=TRUE, partial.auc=c(1, 0.8),     partial.auc.focus="se", grid=c(0.1, 0.2), grid.col=c("green", "red"),     max.auc.polygon=TRUE, auc.polygon.col="lightblue",      print.thres=TRUE, print.thres.adj = c(1, -1),     reuse.auc=FALSE)# Add a second ROC curve to the previous plot:plot(roc.wfns, add=TRUE)# Plot some thresholds, add them to the same plotplot(roc.ndka, print.thres="best", print.thres.best.method="youden")plot(roc.ndka, print.thres="best", print.thres.best.method="closest.topleft",                                   add = TRUE)plot(roc.ndka, print.thres="best", print.thres.best.method="youden",                                   print.thres.best.weights=c(50, 0.2),                                   print.thres.adj = c(1.1, 1.25),                                   add = TRUE)

Sample size and power computation for ROC curves

Description

Computes sample size, power, significance level or minimum AUC for ROC curves.

Usage

power.roc.test(...)# One or Two ROC curves test with roc objects:## S3 method for class 'roc'power.roc.test(roc1, roc2, sig.level = 0.05, power = NULL, kappa = NULL, alternative = c("two.sided", "one.sided"),reuse.auc=TRUE, method = c("delong", "bootstrap", "obuchowski"), ...)# One ROC curve with a given AUC:## S3 method for class 'numeric'power.roc.test(auc = NULL, ncontrols = NULL, ncases = NULL, sig.level = 0.05, power = NULL, kappa = 1, alternative = c("two.sided", "one.sided"), ...)# Two ROC curves with the given parameters:## S3 method for class 'list'power.roc.test(parslist, ncontrols = NULL, ncases = NULL, sig.level = 0.05, power = NULL,  kappa = 1, alternative = c("two.sided", "one.sided"), ...)

Arguments

Value

An object of classpower.htest (such as that given bypower.t.test) with the supplied and computed values.

One ROC curve power calculation

If one or no ROC curves are passed topower.roc.test, a one ROCcurve power calculation is performed. The function expects eitherpower,sig.level orauc, or bothncontrolsandncases to be missing, so that the parameter is determinedfrom the others with the formula by Obuchowskiet al., 2004 (formulas2 and 3, p. 1123).

For the sample size,ncases is computed directly from formulas2 and 3 and ncontrols is deduced withkappa (defaults to theratio of controls to cases).AUC is optimized byuniroot whilesig.levelandpower are solved as quadratic equations.

power.roc.test can also be passed aroc object from therocfunction, but the empirical ROC will not be used, only the number ofpatients and the AUC.

Two paired ROC curves power calculation

If two ROC curves are passed topower.roc.test, the functionwill compute either the required sample size (ifpower is supplied),the significance level (ifsig.level=NULL andpower issupplied) or the power of a test of a difference between to AUCsaccording to the formula by Obuchowski and McClish, 1997(formulas 2 and 3, p. 1530–1531). The null hypothesis is that the AUCofroc1 is the same than the AUC ofroc2, withroc1 taken as the reference ROC curve.

For the sample size,ncases is computed directly from formula 2and ncontrols is deduced withkappa (defaults to theratio of controls to cases inroc1).sig.level andpower are solved as quadratic equations.

The variance and covariance of the ROC curve are computed with thevar andcov functions. By default, DeLongmethod using the algorithm by Sun and Xu (2014) is used for fullAUCs and the bootstrap for partial AUCs. It ispossible to force the use of Obuchowski's variance by specifyingmethod="obuchowski".

Alternatively when no empirical ROC curve is known, or if only one isavailable, a list can be passed topower.roc.test, with thecontents defined in the “Arguments” section. The variance andcovariance are computed from Table 1 and Equation 4 and 5 ofObuchowski and McClish (1997), p. 1530–1531.

Power calculation for unpaired ROC curves is not implemented.

AUC specification

The comparison of the AUC of the ROC curves needs a specification of theAUC. The specification is defined by:

1. the “auc” field in the “roc” objects ifreuse.auc is set toTRUE (default)

2. passing the specification toauc with ...(argumentspartial.auc,partial.auc.correct andpartial.auc.focus). In this case, you must ensure either thattheroc object do not contain anauc field (ifyou calledroc withauc=FALSE), or setreuse.auc=FALSE.

Ifreuse.auc=FALSE theauc function will alwaysbe called with... to determine the specification, even ifthe “roc” objects do contain anauc field.

As well if the “roc” objects do not contain anaucfield, theauc function will always be called with... to determine the specification.

Warning: if the roc object passed to roc.test contains anaucfield andreuse.auc=TRUE,auc is not called andarguments such aspartial.auc are silently ignored.

Acknowledgements

The authors would like to thank Christophe Combescure and Anne-SophieJannot for their help with the implementation of this section of the package.

References

Elisabeth R. DeLong, David M. DeLong and Daniel L. Clarke-Pearson(1988) “Comparing the areas under two or more correlated receiveroperating characteristic curves: a nonparametricapproach”.Biometrics44, 837–845.

Nancy A. Obuchowski, Donna K. McClish (1997). “Sample sizedetermination for diagnostic accurary studies involving binormal ROCcurve indices”.Statistics in Medicine,16,1529–1542. DOI:doi:10.1002/(SICI)1097-0258(19970715)16:13<1529::AID-SIM565>3.0.CO;2-H.

Nancy A. Obuchowski, Micharl L. Lieber, Frank H. WiansJr. (2004). “ROC Curves in Clinical Chemistry: Uses, Misuses, andPossible Solutions”.Clinical Chemistry,50, 1118–1125. DOI:doi:10.1373/clinchem.2004.031823.

Xu Sun and Weichao Xu (2014) “Fast Implementation of DeLongs Algorithm for Comparingthe Areas Under Correlated Receiver Operating Characteristic Curves”.IEEE SignalProcessing Letters,21, 1389–1393. DOI:doi:10.1109/LSP.2014.2337313.

See Also

roc,roc.test

Examples

data(aSAH)#### One ROC curve ##### Build a roc object:rocobj <- roc(aSAH$outcome, aSAH$s100b)# Determine power of one ROC curve:power.roc.test(rocobj)# Same as:power.roc.test(ncases=41, ncontrols=72, auc=0.73, sig.level=0.05)# sig.level=0.05 is implicit and can be omitted:power.roc.test(ncases=41, ncontrols=72, auc=0.73)# Determine ncases & ncontrols:power.roc.test(auc=rocobj$auc, sig.level=0.05, power=0.95, kappa=1.7)power.roc.test(auc=0.73, sig.level=0.05, power=0.95, kappa=1.7)# Determine sig.level:power.roc.test(ncases=41, ncontrols=72, auc=0.73, power=0.95, sig.level=NULL)# Derermine detectable AUC:power.roc.test(ncases=41, ncontrols=72, sig.level=0.05, power=0.95)#### Two ROC curves #######  Full AUCroc1 <- roc(aSAH$outcome, aSAH$ndka)roc2 <- roc(aSAH$outcome, aSAH$wfns)## Sample size# With DeLong variance (default)power.roc.test(roc1, roc2, power=0.9)# With Obuchowski variancepower.roc.test(roc1, roc2, power=0.9, method="obuchowski")## Power test# With DeLong variance (default)power.roc.test(roc1, roc2)# With Obuchowski variancepower.roc.test(roc1, roc2, method="obuchowski")## Significance level# With DeLong variance (default)power.roc.test(roc1, roc2, power=0.9, sig.level=NULL)# With Obuchowski variancepower.roc.test(roc1, roc2, power=0.9, sig.level=NULL, method="obuchowski")### Partial AUCroc3 <- roc(aSAH$outcome, aSAH$ndka, partial.auc=c(1, 0.9))roc4 <- roc(aSAH$outcome, aSAH$wfns, partial.auc=c(1, 0.9))## Sample size# With bootstrap variance (default)## Not run: power.roc.test(roc3, roc4, power=0.9)## End(Not run)# With Obuchowski variancepower.roc.test(roc3, roc4, power=0.9, method="obuchowski")## Power test# With bootstrap variance (default)## Not run: power.roc.test(roc3, roc4)# This is exactly equivalent:power.roc.test(roc1, roc2, reuse.auc=FALSE, partial.auc=c(1, 0.9))## End(Not run)# With Obuchowski variancepower.roc.test(roc3, roc4, method="obuchowski")## Significance level# With bootstrap variance (default)## Not run: power.roc.test(roc3, roc4, power=0.9, sig.level=NULL)## End(Not run)# With Obuchowski variancepower.roc.test(roc3, roc4, power=0.9, sig.level=NULL, method="obuchowski")## With only binormal parameters given# From example 2 of Obuchowski and McClish, 1997.ob.params <- list(A1=2.6, B1=1, A2=1.9, B2=1, rn=0.6, ra=0.6, FPR11=0,FPR12=0.2, FPR21=0, FPR22=0.2, delta=0.037) power.roc.test(ob.params, power=0.8, sig.level=0.05)power.roc.test(ob.params, power=0.8, sig.level=NULL, ncases=107)power.roc.test(ob.params, power=NULL, sig.level=0.05, ncases=107)

Print a ROC curve object

Description

This function prints a ROC curve, AUC or CI object and return it invisibly.

Usage

## S3 method for class 'roc'print(x, digits=max(3, getOption("digits") - 3), call=TRUE, ...)## S3 method for class 'multiclass.roc'print(x, digits=max(3, getOption("digits") -  3), call=TRUE, ...) ## S3 method for class 'mv.multiclass.roc'print(x, digits=max(3, getOption("digits") -  3), call=TRUE, ...) ## S3 method for class 'smooth.roc'print(x, digits=max(3, getOption("digits") - 3),call=TRUE, ...)## S3 method for class 'auc'print(x, digits=max(3, getOption("digits") - 3), ...)## S3 method for class 'multiclass.auc'print(x, digits=max(3, getOption("digits") - 3), ...)## S3 method for class 'ci.auc'print(x, digits=max(3, getOption("digits") - 3), ...)## S3 method for class 'ci.thresholds'print(x, digits=max(3, getOption("digits") - 3), ...)## S3 method for class 'ci.se'print(x, digits=max(3, getOption("digits") - 3), ...)## S3 method for class 'ci.sp'print(x, digits=max(3, getOption("digits") - 3), ...)## S3 method for class 'ci.coords'print(x, digits=max(3, getOption("digits") - 3), ...)

Arguments

Value

These functions return the object they were passed invisibly.

See Also

roc,auc,ci,coords

Examples

data(aSAH)# Print a roc object:rocobj <- roc(aSAH$outcome, aSAH$s100b)print(rocobj)# Print a smoothed roc objectprint(smooth(rocobj))# implicit printing roc(aSAH$outcome, aSAH$s100b)# Print an auc and a ci object, from the ROC object or calling# the dedicated function:print(rocobj$auc)print(ci(rocobj))

Build a ROC curve

Description

This is the main function of the pROC package. It builds a ROCcurve and returns a “roc” object, a list of class“roc”. This object can beprinted,plotted, orpassed to the functionsauc,ci,smooth.roc andcoords. Additionally, tworoc objects can be compared withroc.test.

Usage

roc(...)## S3 method for class 'formula'roc(formula, data, ...)## S3 method for class 'data.frame'roc(data, response, predictor,ret = c("roc", "coords", "all_coords"), ...)## Default S3 method:roc(response, predictor, controls, cases,density.controls, density.cases,levels=base::levels(as.factor(response)), percent=FALSE, na.rm=TRUE,direction=c("auto", "<", ">"), algorithm = 2, quiet = FALSE, smooth=FALSE, auc=TRUE, ci=FALSE, plot=FALSE, smooth.method="binormal",smooth.n=512, ci.method=NULL, density=NULL, ...)roc_(data, response, predictor, ret = c("roc", "coords", "all_coords"), ...)

Arguments

Details

This function's main job is to build a ROC object. See the“Value” section to this page for more details. Beforereturning, it will call (in this order) thesmooth,auc,ci andplot.rocfunctions ifsmoothauc,ci andplot.roc(respectively) arguments are set to TRUE. By default, onlyaucis called.

Data can be provided asresponse, predictor, where thepredictor is the numeric (or ordered) level of the evaluated signal, andthe response encodes the observation class (control or case). Thelevel argument specifies which response level must be taken ascontrols (first value oflevel) or cases (second). It cansafely be ignored when the response is encoded as0 and1, but it will frequently fail otherwise. By default, the firsttwo values oflevels(as.factor(response)) are taken, and theremaining levels are ignored. This means that if your response iscoded “control” and “case”, the levels will beinverted.

In some cases, it is more convenient to pass the data ascontrols, cases, but both arguments are ignored ifresponse, predictor was specified to non-NULL values.It is also possible to pass density data withdensity.controls, density.cases, which will result in a smoothed ROC curve even ifsmooth=FALSE, but are ignored ifresponse, predictor orcontrols, cases are provided.

Thresholds are selected as the means between any two consecutive values observed in the data. This choice is aimed to facilitate their interpretation,as any data point will be unambiguously positive or negativeregardless of whether the comparison operator includes equalityor not. As a corollary, thresholds do not correspond to actual valuesin the data.

Specifications forauc,ci andplot.roc are not kept ifauc,ci orplot are set toFALSE. Especially, in the following case:

    myRoc <- roc(..., auc.polygon=TRUE, grid=TRUE, plot=FALSE)    plot(myRoc)

the plot will not have the AUC polygon nor the grid. Similarly, whencomparing “roc” objects, the following is not possible:

    roc1 <- roc(..., partial.auc=c(1, 0.8), auc=FALSE)    roc2 <- roc(..., partial.auc=c(1, 0.8), auc=FALSE)    roc.test(roc1, roc2)

This will produce a test on the full AUC, not the partial AUC. To makea comparison on the partial AUC, you must repeat the specificationswhen callingroc.test:

    roc.test(roc1, roc2, partial.auc=c(1, 0.8))

Note that ifroc was called withauc=TRUE, the latter syntax will notallow redefining the AUC specifications. You must usereuse.auc=FALSE for that.

Value

If the data contained anyNA value andna.rm=FALSE,NA isreturned. Otherwise, ifsmooth=FALSE, a list of class“roc” with the following fields:

Ifsmooth=TRUE a list of class “smooth.roc” as returnedbysmooth, with or without additional elementsauc andci (according to the call).

Pipelines

Theroc function can be used in pipelines, for instance withdplyr ormagrittr.Theroc.data.frame method supports both standard and non-standard evaluation (NSE):

library(dplyr)# Standard evaluation:aSAH %>%    filter(gender == "Female") %>%    roc("outcome", "s100b")# Non-Standard Evaluation:aSAH %>%    filter(gender == "Female") %>%    roc(outcome, s100b)

For tasks involving programming and variable column names, theroc_ function providesstandard evaluation:

# Standard evaluation:aSAH %>%    filter(gender == "Female") %>%    roc_("outcome", "s100b")

By default it returns theroc object, which can then be piped tothecoords function to extract coordinates that can be usedin further pipelines.

# Returns thresholds, sensitivities and specificities:aSAH  %>%    roc(outcome, s100b) %>%    coords(transpose = FALSE) %>%    filter(sensitivity > 0.6,            specificity > 0.6)# Returns all existing coordinates, then select precision and recall:aSAH  %>%    roc(outcome, s100b) %>%    coords(ret = "all", transpose = FALSE) %>%    select(precision, recall)

Errors

If no control or case observation exist for the given levels ofresponse, no ROC curve can be built and an error is triggered withmessage “No control observation” or “No caseobservation”.

If the predictor is not a numeric or ordered, as defined byas.numeric oras.ordered, the message“Predictor must be numeric or ordered” is returned.

The message “No valid data provided” is issued when the datawasn't properly passed. Remember you need bothresponse andpredictor of the same (not null) length, or bothcontrolsandcases. Combinations such aspredictor andcases are not valid and will trigger this error.

Infinite values of the predictor cannot always be thresholded byinfinity and can cause ROC curves to not reach 0 or 100% specificity or sensitivity. Since version 1.13.0, pROC returnsNaNwith a warning message “Infinite value(s) in predictor” ifpredictor contains anyinfinite values.

References

Tom Fawcett (2006) “An introduction to ROC analysis”.PatternRecognition Letters27, 861–874. DOI:doi:10.1016/j.patrec.2005.10.010.

Xavier Robin, Natacha Turck, Alexandre Hainard,et al.(2011) “pROC: an open-source package for R and S+ to analyze andcompare ROC curves”.BMC Bioinformatics,7, 77.DOI:doi:10.1186/1471-2105-12-77.

See Also

auc,ci,plot.roc,print.roc,roc.test

Examples

data(aSAH)# Basic exampleroc(aSAH$outcome, aSAH$s100b,    levels=c("Good", "Poor"))# As levels aSAH$outcome == c("Good", "Poor"),# this is equivalent to:roc(aSAH$outcome, aSAH$s100b)# In some cases, ignoring levels could lead to unexpected results# Equivalent syntaxes:roc(outcome ~ s100b, aSAH)roc(aSAH$outcome ~ aSAH$s100b)with(aSAH, roc(outcome, s100b))with(aSAH, roc(outcome ~ s100b))# With a formula:roc(outcome ~ s100b, data=aSAH)## Not run: library(dplyr)aSAH %>%    filter(gender == "Female") %>%    roc(outcome, s100b)## End(Not run)# Using subset (only with formula)roc(outcome ~ s100b, data=aSAH, subset=(gender == "Male"))roc(outcome ~ s100b, data=aSAH, subset=(gender == "Female"))# With numeric controls/casesroc(controls=aSAH$s100b[aSAH$outcome=="Good"], cases=aSAH$s100b[aSAH$outcome=="Poor"])# With ordered controls/casesroc(controls=aSAH$wfns[aSAH$outcome=="Good"], cases=aSAH$wfns[aSAH$outcome=="Poor"])# Inverted the levels: "Poor" are now controls and "Good" cases:roc(aSAH$outcome, aSAH$s100b,    levels=c("Poor", "Good"))# The result was exactly the same because of direction="auto".# The following will give an AUC < 0.5:roc(aSAH$outcome, aSAH$s100b,    levels=c("Poor", "Good"), direction="<")# If we are sure about levels and direction auto-detection,# we can turn off the messages:roc(aSAH$outcome, aSAH$s100b, quiet = TRUE)# If we prefer counting in percent:roc(aSAH$outcome, aSAH$s100b, percent=TRUE)# Plot and CI (see plot.roc and ci for more options):roc(aSAH$outcome, aSAH$s100b,    percent=TRUE, plot=TRUE, ci=TRUE)# Smoothed ROC curveroc(aSAH$outcome, aSAH$s100b, smooth=TRUE)# this is not identical tosmooth(roc(aSAH$outcome, aSAH$s100b))# because in the latter case, the returned object contains no AUC

Compare two ROC curves

Description

This function compares two correlated (or paired) or uncorrelated (unpaired)ROC curves. Delong and bootstrap methods test for a difference in the(partial) AUC of the ROC curves. The Venkatraman method tests if the twocurves are perfectly superposed. The sensitivity and specificity methodstest if the sensitivity (respectively specificity) of the ROC curves aredifferent at the given level of specificity (respectively sensitivity).Several syntaxes are available: two object of class roc (which can be AUCor smoothed ROC), or either three vectors (response, predictor1, predictor2)or a response vector and a matrix or data.frame with two columns(predictors).

Usage

# roc.test(...)## S3 method for class 'roc'roc.test(roc1, roc2, method=c("delong", "bootstrap","venkatraman", "sensitivity", "specificity"), sensitivity = NULL,specificity = NULL, alternative = c("two.sided", "less", "greater"),paired=NULL, reuse.auc=TRUE, boot.n=2000, boot.stratified=TRUE,ties.method="first", progress = NULL,parallel=FALSE, conf.level=0.95, ...)## S3 method for class 'auc'roc.test(roc1, roc2, ...)## S3 method for class 'smooth.roc'roc.test(roc1, roc2, ...)## S3 method for class 'formula'roc.test(formula, data, ...)## Default S3 method:roc.test(response, predictor1, predictor2=NULL,na.rm=TRUE, method=NULL, ...)

Arguments

Details

This function compares two ROC curves. It is typically called with the tworoc objects tocompare.roc.test.default is provided as a conveniencemethod and creates tworoc objects before callingroc.test.roc.

Three methods are available: “delong”, “bootstrap” and “venkatraman” (see“Computational details” section below). “delong” and“bootstrap” are tests over the AUC whereas “venkatraman”compares the the ROC curves themselves.

Default is to use “delong” method except for comparison of partial AUC, smoothedcurves and curves with differentdirection, wherebootstrapis used. Using “delong” for partial AUC and smoothed ROCs is notsupported in pROC and result in an error.It is spurious to use “delong” forroc with differentdirection (a warning is issued but the spurious comparison isenforced). “venkatraman”'s test cannot be employed to compare smoothedROC curves, or curves with partial AUC specifications. In addition,and comparison of ROC curves with differentdirection should be used with care (a warning is produced as well).

Ifalternative="two.sided", a two-sided test for difference in AUC is performed. Ifalternative="less", the alternative is that the AUC of roc1 issmaller than the AUC of roc2. Formethod="venkatraman", only“two.sided” test is available.

If thepaired argument is not provided, theare.paired function isemployed to detect the paired status of the ROC curves. It will test if the originalresponse isidentical between the two ROC curves (this is always the case if the call is made withroc.test.default). This detection is unlikely to raise false positives, butthis possibility cannot be excluded entierly. It would require equal sample sizesandresponse values and order in both ROC curves. If it happens to you, usepaired=FALSE.If you know the ROC curves are paired you can passpaired=TRUE. However this isuseless as it will be tested anyway.

Forsmoothed ROC curves, smoothing is performed again at eachbootstrap replicate with the parameters originally provided.If a density smoothing was performed with user-provideddensity.cases ordensity.controls the bootstrap cannotbe performed and an error is issued.

Value

A list of class "htest" with following content:

AUC specification

The comparison of the AUC of the ROC curves needs a specification of theAUC. The specification is defined by:

1. the “auc” field in the “roc” objects ifreuse.auc is set toTRUE (default)

2. passing the specification toauc with ...(argumentspartial.auc,partial.auc.correct andpartial.auc.focus). In this case, you must ensure either thattheroc object do not contain anauc field (ifyou calledroc withauc=FALSE), or setreuse.auc=FALSE.

Ifreuse.auc=FALSE theauc function will alwaysbe called with... to determine the specification, even ifthe “roc” objects do contain anauc field.

As well if the “roc” objects do not contain anaucfield, theauc function will always be called with... to determine the specification.

The AUC specification is ignored in the Venkatraman test.

Warning: if the roc object passed to roc.test contains anaucfield andreuse.auc=TRUE,auc is not called andarguments such aspartial.auc are silently ignored.

Computation details

Withmethod="bootstrap", the processing is done as follow:

1. boot.n bootstrap replicates are drawn from thedata. Ifboot.stratified isTRUE, each replicate containsexactly the same number of controls and cases than the originalsample, otherwise ifFALSE the numbers can vary.

2. for each bootstrap replicate, the AUC of the two ROC curvesare computed and the difference is stored.

3. The following formula is used:

   D=\frac{AUC1-AUC2}{s}

   where s is the standard deviation ofthe bootstrap differences and AUC1 and AUC2 the AUC of the two(original) ROC curves.

4. D is then compared to the normal distribution,according to the value ofalternative.

See also the Bootstrap section inthis package's documentation.

Withmethod="delong", the processing is done as described inDeLonget al. (1988) for paired ROC curves, using the algorithmof Sun and Xu (2014). Only comparison oftwo ROC curves is implemented. The method has been extended forunpaired ROC curves where the p-value is computed with an unpairedt-test with unequal sample size and unequal variance, with

D=\frac{V^r(\theta^r) - V^s(\theta^s) }{ \sqrt{S^r + S^s}}

Withmethod="venkatraman", the processing is done as describedin Venkatraman and Begg (1996) (for paired ROC curves) and Venkatraman(2000) (for unpaired ROC curves) withboot.n permutation ofsample ranks (with ties breaking). For consistency reasons, the same argumentboot.n asin bootstrap defines the number of permutations to execute,even though no bootstrap is performed.

Formethod="specificity", the test assesses if the sensitivity ofthe ROC curves are different at the level of specificity given by thespecificity argument, which must be a numeric of length 1. Bootstrap is employed as withmethod="bootstrap"andboot.n andboot.stratified are available. This isidentical to the test proposed by Pepeet al. (2009).Themethod="sensitivity" is very similar, but assesses if the specificity ofthe ROC curves are different at the level of sensitivity given by thesensitivity argument.

Warnings

If “auc” specifications are different in both roc objects, thewarning “Different AUC specifications in the ROCcurves. Enforcing the inconsistency, but unexpected results may beproduced.” is issued. Unexpected results may be produced.

If one or both ROC curves are “smooth.roc” objects withdifferent smoothing specifications, the warning “Different smoothing parameters in the ROC curves. Enforcingthe inconsistency, but unexpected results may be produced.” is issued.This warning can be benign, especially if ROC curves were generatedwithroc(..., smooth=TRUE) with different arguments to otherfunctions (such as plot), or if you really want to compare two ROCcurves smoothed differently.

Ifmethod="venkatraman", andalternative is“less” or “greater”, the warning “Only two-sidedtests are available for Venkatraman. Performing two-sided test instead.”is produced and a two tailed test is performed.

Both DeLong and Venkatraman's test ignores the direction of the ROC curve so that if twoROC curves have a different differ in the value ofdirection, the warning “(DeLong|Venkatraman)'s test should not beapplied to ROC curves with different directions.” isprinted. However, the spurious test is enforced.

Ifboot.stratified=FALSE and the sample has a large imbalance betweencases and controls, it could happen that one or more of the replicatescontains no case or control observation, or that there are not enoughpoints for smoothing, producing aNA area.The warning “NA value(s) produced during bootstrap were ignored.”will be issued and the observation will be ignored. If you have a largeimbalance in your sample, it could be safer to keepboot.stratified=TRUE.

When both ROC curves have anauc of 1 (or 100%), their variances and covariance will always be null,and therefore the p-value will always be 1. This is true for both “delong”, “bootstrap” and “venkatraman” methods. This result is misleading, as the variances and covariance are of course not null.Awarning will be displayed to inform of this condition, and of the misleading output.

Errors

An error will also occur if you give apredictor2 whenpredictor1 is amatrix or adata.frame, ifpredictor1 has more than twocolumns, or if you do not give apredictor2 whenpredictor1 is a vector.

Ifdensity.cases anddensity.controls were providedfor smoothing, the error “Cannot compute the statistic on ROCcurves smoothed with density.controls and density.cases.” isissued.

Ifmethod="venkatraman" and one of the ROC curves is smoothed,the error “Using Venkatraman's test for smoothed ROCs is notsupported.” is produced.

Withmethod="specificity", the error “Argument'specificity' must be numeric of length 1 for a specificity test.”is given unless the specificity argument is specified as a numeric oflength 1. The “Argument 'sensitivity' must be numeric of length1 for a sensitivity test.” message is given formethod="sensitivity" under similar conditions.

Acknowledgements

We would like to thank E. S. Venkatraman and Colin B. Begg for theirsupport in the implementation of their test.

References

Elisabeth R. DeLong, David M. DeLong and Daniel L. Clarke-Pearson(1988) “Comparing the areas under two or more correlated receiveroperating characteristic curves: a nonparametricapproach”.Biometrics44, 837–845.

James A. Hanley and Barbara J. McNeil (1982) “The meaning and use ofthe area under a receiver operating characteristic (ROC)curve”.Radiology143, 29–36.

Margaret Pepe, Gary Longton and Holly Janes (2009) “Estimation andComparison of Receiver Operating Characteristic Curves”.TheStata journal9, 1.

Xavier Robin, Natacha Turck, Jean-Charles Sanchez and Markus Müller(2009) “Combination of protein biomarkers”.useR! 2009, Rennes.https://www.r-project.org/nosvn/conferences/useR-2009/abstracts/user_author.html

Xavier Robin, Natacha Turck, Alexandre Hainard,et al.(2011) “pROC: an open-source package for R and S+ to analyze andcompare ROC curves”.BMC Bioinformatics,7, 77.DOI:doi:10.1186/1471-2105-12-77.

Xu Sun and Weichao Xu (2014) “Fast Implementation of DeLongs Algorithm for Comparingthe Areas Under Correlated Receiver Operating Characteristic Curves”.IEEE SignalProcessing Letters,21, 1389–1393. DOI:doi:10.1109/LSP.2014.2337313.

E. S. Venkatraman and Colin B. Begg (1996) “A distribution-freeprocedure for comparing receiver operating characteristic curves froma paired experiment”.Biometrika83, 835–848.DOI:doi:10.1093/biomet/83.4.835.

E. S. Venkatraman (2000) “A Permutation Test to Compare ReceiverOperating Characteristic Curves”.Biometrics56,1134–1138. DOI:doi:10.1111/j.0006-341X.2000.01134.x.

Hadley Wickham (2011) “The Split-Apply-Combine Strategy for Data Analysis”.Journal of Statistical Software,40, 1–29.URL:doi:10.18637/jss.v040.i01.

See Also

roc,power.roc.test

Examples

data(aSAH)# Basic example with 2 roc objectsroc1 <- roc(aSAH$outcome, aSAH$s100b)roc2 <- roc(aSAH$outcome, aSAH$wfns)roc.test(roc1, roc2)## Not run: # The latter used Delong's test. To use bootstrap test:roc.test(roc1, roc2, method="bootstrap")# Increase boot.n for a more precise p-value:roc.test(roc1, roc2, method="bootstrap", boot.n=10000)## End(Not run)# Alternative syntaxesroc.test(aSAH$outcome, aSAH$s100b, aSAH$wfns)roc.test(aSAH$outcome, data.frame(aSAH$s100b, aSAH$wfns))# If we had a good a priori reason to think that wfns gives a# better classification than s100b (in other words, AUC of roc1# should be lower than AUC of roc2):roc.test(roc1, roc2, alternative="less")## Not run: # Comparison can be done on smoothed ROCs# Smoothing is re-done at each iteration, and execution is slowroc.test(smooth(roc1), smooth(roc2))# or:roc.test(aSAH$outcome, aSAH$s100b, aSAH$wfns, smooth=TRUE, boot.n=100)## End(Not run)# or from an AUC (no smoothing)roc.test(auc(roc1), roc2)## Not run: # Comparison of partial AUC:roc3 <- roc(aSAH$outcome, aSAH$s100b, partial.auc=c(1, 0.8), partial.auc.focus="se")roc4 <- roc(aSAH$outcome, aSAH$wfns, partial.auc=c(1, 0.8), partial.auc.focus="se")roc.test(roc3, roc4)# This is strictly equivalent to:roc.test(roc3, roc4, method="bootstrap")# Alternatively, we could re-use roc1 and roc2 to get the same result:roc.test(roc1, roc2, reuse.auc=FALSE, partial.auc=c(1, 0.8), partial.auc.focus="se")# Comparison on specificity and sensitivityroc.test(roc1, roc2, method="specificity", specificity=0.9)roc.test(roc1, roc2, method="sensitivity", sensitivity=0.9)## End(Not run)# Spurious use of DeLong's test with different direction:roc5 <- roc(aSAH$outcome, aSAH$s100b, direction="<")roc6 <- roc(aSAH$outcome, aSAH$s100b, direction=">")roc.test(roc5, roc6, method="delong")## Not run: # Comparisons of the ROC curvesroc.test(roc1, roc2, method="venkatraman")## End(Not run)# Unpaired testsroc7 <- roc(aSAH$outcome, aSAH$s100b)# artificially create an roc8 unpaired with roc7roc8 <- roc(aSAH$outcome[1:100], aSAH$s100b[1:100])## Not run: roc.test(roc7, roc8, paired=FALSE, method="delong")roc.test(roc7, roc8, paired=FALSE, method="bootstrap")roc.test(roc7, roc8, paired=FALSE, method="venkatraman")roc.test(roc7, roc8, paired=FALSE, method="specificity", specificity=0.9)## End(Not run)

Smooth a ROC curve

Description

This function smoothes a ROC curve of numeric predictor. By default, abinormal smoothing is performed, but density or custom smoothings aresupported.

Usage

smooth(...)## Default S3 method:smooth(...)## S3 method for class 'roc'smooth(roc,method=c("binormal", "density", "fitdistr", "logcondens","logcondens.smooth"), n=512, bw = "nrd0", density=NULL,density.controls=density, density.cases=density,start=NULL, start.controls=start, start.cases=start, reuse.auc=TRUE, reuse.ci=FALSE, ...)## S3 method for class 'smooth.roc'smooth(smooth.roc, ...)

Arguments

Details

Ifmethod="binormal", a linear model is fitted to the quantiles ofthe sensitivities and specificities. Smoothed sensitivities andspecificities are then generated from this model onn points.This simple approach was found to work well for most ROC curves, butit may produce hooked smooths in some situations (see in Hanley (1988)).

Withmethod="density", thedensityfunction is employed to generate a smooth kerneldensity of the control and case observations as described by Zhouet al. (1997), unlessdensity.controls ordensity.cases are provideddirectly.bw can be given tospecify a bandwidth to use withdensity. It can be anumeric value or a character string (“nrd0”, “nrd”,“ucv”, “bcv” or “SJ”, but any namematching a function prefixed with “bw.” issupported). In the case of a characterstring, the whole predictor data is employed to determine the numericvalue to use on both controls and cases.Depending on your data, it might be a good idea to specify thekernel argument fordensity. By default,“gaussian” is used, but “epanechnikov”,“rectangular”, “triangular”, “biweight”,“cosine” and “optcosine” are supported. As all thekernels are symetrical, it might help to normalize the data first(that is, before callingroc), for example with quantilenormalization:

    norm.x <- qnorm(rank(x)/(length(x)+1))    smooth(roc(response, norm.x, ...), ...)

Additionally,density can be a function which must returneither a numeric vector of densities over the y axis or alistwith a “y” item like thedensity function. Itmust accept the following input:

    density.fun(x, n, from, to, bw, kernel, ...)

It is important to honourn,from andto in orderto have the densities evaluated on the same points for controls andcases. Failing to do so and returning densities of different lengthwill produce an error. It is also a good idea to use a constantsmoothing parameter (such asbw) especially when controls andcases have a different number of observations, to avoid producingsmoother or rougher densities.

Ifmethod="fitdistr", thefitdistrfunction from theMASS package is employed to fit parameters forthe density functiondensity with optionnal start parametersstart. The density function are fittedseparately in control (density.controls,start.controls)and case observations (density.cases,start.cases).density can be one of the character valuesallowed byfitdistr or a density function (suchasdnorm,dweibull, ...).

Themethod="logcondens" andmethod="logcondens.smooth" use thelogcondens package to generate a non smoothed or smoothed(respectively) log-concave density estimate of of the control and caseobservation with thelogConROC function.

smooth.default forces the usage of thesmooth function in thestats package, sothat other code relying onsmooth should continue to functionnormally.

Smoothed ROC curves can be passed to smooth again. In this case, thesmoothing is not re-applied on the smoothed ROC curve but theoriginal “roc” object will be re-used.

Note that asmooth.roc curve has no threshold.

Value

A list of class “smooth.roc” with the following fields:

Attributes

Additionally, the originalroc object is stored as a“roc” attribute.

Errors

The message “The 'density' function must return a numericvector or a list with a 'y' item.” will be displayed if thedensity function did not return a valid output. The message“Length of 'density.controls' and 'density.cases' differ.”will be displayed if the returned value differ in length.

Binormal smoothing cannot smooth ROC curve defined by only onepoint. Any such attempt will fail with the error “ROC curve notsmoothable (not enough points).”.

If the smooth ROC curve was generated byroc withdensity.controls anddensity.cases numeric arguments, itcannot be smoothed and the error “Cannot smooth a ROC curvegenerated directly with numeric 'density.controls' and'density.cases'.” is produced.

fitdistr anddensity smoothing methods require anumericpredictor. If the ROC curve to smooth wasgenerated with an ordered factor only binormal smoothing can beapplied and the message “ROC curves of ordered predictors canbe smoothed only with binormal smoothing.” is displayed otherwise.

fitdistr,logcondens andlogcondens.smooth methodsrequire additional packages. If not available, the following message will be displayed with the required command to install the package:“Package ? not available, required with method='?'.Please install it with 'install.packages("?")'. ”

References

James E. Hanley (1988) “The robustness of the “binormal” assumptionsused in fitting ROC curves”.Medical Decision Making8, 197–203.

Lutz Duembgen, Kaspar Rufibach (2011) “logcondens: Computations Relatedto Univariate Log-Concave Density Estimation”.Journal of StatisticalSoftware,39, 1–28. URL:doi:10.18637/jss.v039.i06.

Xavier Robin, Natacha Turck, Alexandre Hainard,et al.(2011) “pROC: an open-source package for R and S+ to analyze andcompare ROC curves”.BMC Bioinformatics,7, 77.DOI:doi:10.1186/1471-2105-12-77.

Kaspar Rufibach (2011) “A Smooth ROC Curve Estimator Based on Log-Concave Density Estimates”.The International Journal of Biostatistics,8, accepted. DOI:doi:10.1515/1557-4679.1378. arXiv:arXiv:1103.1787.

William N. Venables, Brian D. Ripley (2002). “Modern Applied Statistics with S”. New York, Springer.Google books.

Kelly H. Zou, W. J. Hall and David E. Shapiro (1997) “Smoothnon-parametric receiver operating characteristic (ROC) curves forcontinuous diagnostic tests”.Statistics in Medicine18, 2143–2156. DOI:doi:10.1002/(SICI)1097-0258(19971015)16:19<2143::AID-SIM655>3.0.CO;2-3.

See Also

roc

CRAN packagesMASS andlogcondens employed in this function.

Examples

data(aSAH)##  Basic examplerocobj <- roc(aSAH$outcome, aSAH$s100b)smooth(rocobj)# or directly with roc()roc(aSAH$outcome, aSAH$s100b, smooth=TRUE)# plottingplot(rocobj)rs <- smooth(rocobj, method="binormal")plot(rs, add=TRUE, col="green")rs2 <- smooth(rocobj, method="density")plot(rs2, add=TRUE, col="blue")rs3 <- smooth(rocobj, method="fitdistr", density="lognormal")plot(rs3, add=TRUE, col="magenta")if (requireNamespace("logcondens")) {rs4 <- smooth(rocobj, method="logcondens")plot(rs4, add=TRUE, col="brown")rs5 <- smooth(rocobj, method="logcondens.smooth")plot(rs5, add=TRUE, col="orange")}legend("bottomright", legend=c("Empirical", "Binormal", "Density", "Log-normal",                               "Log-concave density", "Smoothed log-concave density"),       col=c("black", "green", "blue", "magenta", "brown", "orange"), lwd=2)## Advanced smoothing# if we know the distributions are normal with sd=0.1 and an unknown mean:smooth(rocobj, method="fitdistr", density=dnorm, start=list(mean=1), sd=.1)# different distibutions for controls and cases:smooth(rocobj, method="fitdistr", density.controls="normal", density.cases="lognormal")# with densitiesbw <- bw.nrd0(rocobj$predictor)density.controls <- density(rocobj$controls, from=min(rocobj$predictor) - 3 * bw,                            to=max(rocobj$predictor) + 3*bw, bw=bw, kernel="gaussian")density.cases <- density(rocobj$cases, from=min(rocobj$predictor) - 3 * bw,                            to=max(rocobj$predictor) + 3*bw, bw=bw, kernel="gaussian")smooth(rocobj, method="density", density.controls=density.controls$y,        density.cases=density.cases$y)# which is roughly what is done by a simple:smooth(rocobj, method="density")## Not run: ## Smoothing artificial ROC curvesrand.unif <- runif(1000, -1, 1)rand.exp <- rexp(1000)rand.norm <- rnorm(1000)# two normalsroc.norm <- roc(controls=rnorm(1000), cases=rnorm(1000)+1, plot=TRUE)plot(smooth(roc.norm), col="green", lwd=1, add=TRUE)plot(smooth(roc.norm, method="density"), col="red", lwd=1, add=TRUE)plot(smooth(roc.norm, method="fitdistr"), col="blue", lwd=1, add=TRUE)if (requireNamespace("logcondens")) {plot(smooth(roc.norm, method="logcondens"), col="brown", lwd=1, add=TRUE)plot(smooth(roc.norm, method="logcondens.smooth"), col="orange", lwd=1, add=TRUE)}legend("bottomright", legend=c("empirical", "binormal", "density", "fitdistr",                               "logcondens", "logcondens.smooth"),        col=c(par("fg"), "green", "red", "blue", "brown", "orange"), lwd=c(2, 1, 1, 1))       # deviation from the normalityroc.norm.exp <- roc(controls=rnorm(1000), cases=rexp(1000), plot=TRUE)plot(smooth(roc.norm.exp), col="green", lwd=1, add=TRUE)plot(smooth(roc.norm.exp, method="density"), col="red", lwd=1, add=TRUE)# Wrong fitdistr: normality assumed by defaultplot(smooth(roc.norm.exp, method="fitdistr"), col="blue", lwd=1, add=TRUE)# Correct fitdistrplot(smooth(roc.norm.exp, method="fitdistr", density.controls="normal",            density.cases="exponential"), col="purple", lwd=1, add=TRUE)if (requireNamespace("logcondens")) {plot(smooth(roc.norm.exp, method="logcondens"), col="brown", lwd=1, add=TRUE)plot(smooth(roc.norm.exp, method="logcondens.smooth"), col="orange", lwd=1, add=TRUE)}legend("bottomright", legend=c("empirical", "binormal", "density",                               "wrong fitdistr", "correct fitdistr",                               "logcondens", "logcondens.smooth"),       col=c(par("fg"), "green", "red", "blue", "purple", "brown", "orange"), lwd=c(2, 1, 1, 1, 1))# large deviation from the normalityroc.unif.exp <- roc(controls=runif(1000, 2, 3), cases=rexp(1000)+2, plot=TRUE)plot(smooth(roc.unif.exp), col="green", lwd=1, add=TRUE)plot(smooth(roc.unif.exp, method="density"), col="red", lwd=1, add=TRUE)plot(smooth(roc.unif.exp, method="density", bw="ucv"), col="magenta", lwd=1, add=TRUE)# Wrong fitdistr: normality assumed by default (uniform distributions not handled)plot(smooth(roc.unif.exp, method="fitdistr"), col="blue", lwd=1, add=TRUE)if (requireNamespace("logcondens")) {plot(smooth(roc.unif.exp, method="logcondens"), col="brown", lwd=1, add=TRUE)plot(smooth(roc.unif.exp, method="logcondens.smooth"), col="orange", lwd=1, add=TRUE)}legend("bottomright", legend=c("empirical", "binormal", "density",                               "density ucv", "wrong fitdistr",                               "logcondens", "logcondens.smooth"),       col=c(par("fg"), "green", "red", "magenta", "blue", "brown", "orange"), lwd=c(2, 1, 1, 1, 1))## End(Not run)# 2 uniform distributions with a custom density functionunif.density <- function(x, n, from, to, bw, kernel, ...) {  smooth.x <- seq(from=from, to=to, length.out=n)  smooth.y <- dunif(smooth.x, min=min(x), max=max(x))  return(smooth.y)}roc.unif <- roc(controls=runif(1000, -1, 1), cases=runif(1000, 0, 2), plot=TRUE)s <- smooth(roc.unif, method="density", density=unif.density)plot(roc.unif)plot(s, add=TRUE, col="grey")## Not run: # you can bootstrap a ROC curve smoothed with a density function:ci(s, boot.n=100)## End(Not run)

Variance of a ROC curve

Description

These functions compute the variance of the AUC of a ROC curve.

Usage

var(...)## Default S3 method:var(...)## S3 method for class 'auc'var(auc, ...)## S3 method for class 'roc'var(roc, method=c("delong", "bootstrap", "obuchowski"),boot.n = 2000, boot.stratified = TRUE, reuse.auc=TRUE, progress = NULL, parallel=FALSE, ...)## S3 method for class 'smooth.roc'var(smooth.roc, ...)

Arguments

Details

Thevar function computes the variance of the AUC of a ROCcurve. It is typically called with theroc object ofinterest. Two methods are available: “delong” and“bootstrap” (see “Computationaldetails” section below).

The default is to use “delong” method except for withpartial AUC and smoothed curves where “bootstrap” is employed.Using “delong” for partial AUC and smoothed ROCs is notsupported.

Forsmoothed ROC curves, smoothing is performed again at eachbootstrap replicate with the parameters originally provided.If a density smoothing was performed with user-provideddensity.cases ordensity.controls the bootstrap cannotbe performed and an error is issued.

var.default forces the usage of thevar function in thestats package, sothat other code relying onvar should continue to functionnormally.

Value

The numeric value of the variance.

AUC specification

var needs a specification of the AUC to computethe variance of the AUC of the ROC curve.The specification is defined by:

1. the “auc” field in the “roc” objects ifreuse.auc is set toTRUE (default)

2. passing the specification toauc with ...(argumentspartial.auc,partial.auc.correct andpartial.auc.focus). In this case, you must ensure either thattheroc object do not contain anauc field (ifyou calledroc withauc=FALSE), or setreuse.auc=FALSE.

Ifreuse.auc=FALSE theauc function will alwaysbe called with... to determine the specification, even ifthe “roc” objects do contain anauc field.

As well if the “roc” objects do not contain anaucfield, theauc function will always be called with... to determine the specification.

Warning: if the roc object passed to roc.test contains anaucfield andreuse.auc=TRUE,auc is not called andarguments such aspartial.auc are silently ignored.

Computation details

Withmethod="bootstrap", the processing is done as follow:

1. boot.n bootstrap replicates are drawn from thedata. Ifboot.stratified isTRUE, each replicate containsexactly the same number of controls and cases than the originalsample, otherwise ifFALSE the numbers can vary.

2. for each bootstrap replicate, the AUC of the ROC curveis computed and stored.

3. the variance of the resampled AUCs are computed and returned.

Withmethod="delong", the processing is done as described inHanley and Hajian-Tilaki (1997) using the algorithm by Sun and Xu (2014).

Withmethod="obuchowski", the processing is done as describedin Obuchowski and McClish (1997), Table 1 and Equation 4, p. 1530–1531. Thecomputation ofg for partial area under the ROC curve ismodified as:

expr1 * (2 * pi * expr2) ^ {(-1)} * (-expr4) - A * B * expr1 * (2 * pi * expr2^3) ^ {(-1/2)} * expr3

.

Binormality assumption

The “obuchowski” method makes the assumption that the data is binormal.If the data shows a deviation from this assumption, it might help tonormalize the data first (that is, before callingroc),for example with quantile normalization:

    norm.x <- qnorm(rank(x)/(length(x)+1))    var(roc(response, norm.x, ...), ...)

“delong” and “bootstrap” methods make no such assumption.

Warnings

Ifmethod="delong" and the AUC specification specifies apartial AUC, the warning “Using DeLong for partial AUC isnot supported. Using bootstrap test instead.” is issued. Themethod argument is ignored and “bootstrap” is used instead.

Ifmethod="delong" and the ROCcurve is smoothed, the warning “Using DeLong forsmoothed ROCs is not supported. Using bootstrap test instead.” isissued. Themethod argument is ignored and “bootstrap”is used instead.

Ifboot.stratified=FALSE and the sample has a large imbalance betweencases and controls, it could happen that one or more of the replicatescontains no case or control observation, or that there are not enoughpoints for smoothing, producing aNA area.The warning “NA value(s) produced during bootstrap were ignored.”will be issued and the observation will be ignored. If you have a largeimbalance in your sample, it could be safer to keepboot.stratified=TRUE.

When the ROC curve has anauc of 1 (or 100%), the variance will always be null.This is true for both “delong” and “bootstrap” methods that cannot properly assess the variance in this case. This result is misleading, as the variance is of course not null.Awarning will be displayed to inform of this condition, and of the misleading output.

Errors

Ifdensity.cases anddensity.controls were providedfor smoothing, the error “Cannot compute the covariance on ROCcurves smoothed with density.controls and density.cases.” isissued.

References

Elisabeth R. DeLong, David M. DeLong and Daniel L. Clarke-Pearson(1988) “Comparing the areas under two or more correlated receiveroperating characteristic curves: a nonparametricapproach”.Biometrics44, 837–845.

James A. Hanley and Karim O. Hajian-Tilaki (1997) “Samplingvariability of nonparametric estimates of the areas under receiveroperating characteristic curves: An update”.AcademicRadiology4, 49–58. DOI:doi:10.1016/S1076-6332(97)80161-4.

Nancy A. Obuchowski, Donna K. McClish (1997). “Sample sizedetermination for diagnostic accurary studies involving binormal ROCcurve indices”.Statistics in Medicine,16(13),1529–1542. DOI:doi:10.1002/(SICI)1097-0258(19970715)16:13<1529::AID-SIM565>3.0.CO;2-H.

Xu Sun and Weichao Xu (2014) “Fast Implementation of DeLongs Algorithm for Comparingthe Areas Under Correlated Receiver Operating Characteristic Curves”.IEEE SignalProcessing Letters,21, 1389–1393. DOI:doi:10.1109/LSP.2014.2337313.

Hadley Wickham (2011) “The Split-Apply-Combine Strategy for Data Analysis”.Journal of Statistical Software,40, 1–29.URL:doi:10.18637/jss.v040.i01.

See Also

roc,cov.roc

Examples

data(aSAH)##  Basic exampleroc1 <- roc(aSAH$outcome, aSAH$s100b)roc2 <- roc(aSAH$outcome, aSAH$wfns)var(roc1)var(roc2)# We could also write it in one line:var(roc(aSAH$outcome, aSAH$s100b))## Not run: # The latter used Delong. To use bootstrap:var(roc1, method="bootstrap")# Decrease boot.n for a faster executionvar(roc1,method="bootstrap", boot.n=1000)## End(Not run)# To use obuchowski:var(roc1, method="obuchowski")## Not run: # Variance of smoothed ROCs:# Smoothing is re-done at each iteration, and execution is slowvar(smooth(roc1))## End(Not run)# or from an AUC (no smoothing)var(auc(roc1))## Test data from Hanley and Hajian-Tilaki, 1997disease.present <- c("Yes", "No", "Yes", "No", "No", "Yes", "Yes", "No",                     "No", "Yes", "No", "No", "Yes", "No", "No")field.strength.1 <- c(1, 2, 5, 1, 1, 1, 2, 1, 2, 2, 1, 1, 5, 1, 1)field.strength.2 <- c(1, 1, 5, 1, 1, 1, 4, 1, 2, 2, 1, 1, 5, 1, 1)roc3 <- roc(disease.present, field.strength.1)roc4 <- roc(disease.present, field.strength.2)# Assess the variance:var(roc3)var(roc4)## Not run: # With bootstrap:var(roc3, method="bootstrap")var(roc4, method="bootstrap")## End(Not run)