Movatterモバイル変換

Compute and plot PR and ROC curves and the areas under the curves for weighted and unweighted data

Description

This package computes the areas under the precision-recall (PR) and receiver operating characteristics (ROC) curve for weighted (e.g., soft-labeled) and unweighted data. In contrast to other implementations, the interpolation between points of the PR curve is done by a non-linear piecewise function. In addition to the areas under the curves, the curves themselves can also be computed and plotted by a specific S3-method.

Details

Author(s)

Jan Grau and Jens Keilwagen

Maintainer: Jan Grau <grau@informatik.uni-halle.de>

References

J. Davis and M. Goadrich. The relationship between precision-recall and ROC curves.InProceedings of the 23rd International Conference on Machine Learning, pages 233–240, New York, NY, USA, 2006. ACM.

T. Fawcett, An introduction to ROC analysis, Pattern Recognition Letters (27) 8, 861-874, 2006.

J. Keilwagen, I. Grosse, and J. Grau. Area under precision-recall curves for weighted and unweighted data, PLOS ONE (9) 3, 2014.

J. Grau, I. Grosse, and J. Keilwagen. PRROC: computing and visualizing precision-recall and receiver operating characteristic curves in R. Bioinformatics, 31(15):2595-2597, 2015.

See Also

pr.curve

roc.curve

plot.PRROC

print.PRROC

Examples

# create artificial scores as random numbersx <- rnorm( 1000 );y <- rnorm( 1000, -1 );# compute area under PR curvepr <- pr.curve( x, y );print( pr );# compute area under ROC curveroc <- roc.curve( x, y );print( roc );# compute PR curve and area under curvepr <- pr.curve( x, y, curve = TRUE );# plot curveplot(pr);# compute ROC curve and area under curveroc <- roc.curve( x, y, curve = TRUE );# plot curveplot(roc);# create artificial weightsx.weights <- runif( 1000 );y.weights <- runif( 1000 );# compute PR curve and area under curvepr <- pr.curve( x, y, x.weights, y.weights, curve = TRUE );# plot curveplot(pr);# compute ROC curve and area under curveroc <- roc.curve( x, y, x.weights, y.weights, curve = TRUE );# plot curveplot(roc);

Plotting PRROC objects using ggplot2

Description

Plots the PR or ROC curves of a PRROC object using ggplot2. To obtain such curves,pr.curve orroc.curve must be called withargumentcurve=TRUE.

Usage

ggprroc(x, auc.main=TRUE, auc.type=c("integral","davis.goadrich"),    xlab=NULL,ylab=NULL,    main=NULL,    max.plot = FALSE, min.plot = FALSE, rand.plot = FALSE,    fill.area = (max.plot & min.plot))

Arguments

Details

This function plots PRROC objects as a curve using ggplot2, specificallygeom_line. If minimum, maximum or random curves have been computed, these can be added to the plot.

The function returns a ggplot2 object that may be further modified using themes, scales on the colours, etc.

Author(s)

Jan Grau and Jens Keilwagen

See Also

pr.curve

roc.curve

Examples

# create artificial scores as random numbersx <- rnorm( 1000 );y <- rnorm( 1000, -1 );# compute PR curvepr <- pr.curve( x, y, curve = TRUE );# standard plot of PR curveggprroc( pr );# compute ROC curveroc <- roc.curve( x, y, curve = TRUE );# standard plot of ROC curveggprroc( roc );# include maximum, minimum and random curvespr <- pr.curve( x, y, curve = TRUE, max.compute = TRUE,     min.compute = TRUE, rand.compute = TRUE );    ggprroc( pr, max.plot = TRUE, min.plot = TRUE, rand.plot = TRUE )# modify theme and colour scalepl <- ggprroc( pr, max.plot = TRUE, min.plot = TRUE, rand.plot = TRUE )if(require(ggplot2)){  pl + scale_color_viridis_c(option="D",name="Some score") +      xlab("Sensitivity")+      theme_gray()}

Plotting PRROC objects

Description

Plots the PR or ROC curves of a PRROC object. To obtain such curves,pr.curve orroc.curve must be called withargumentcurve=TRUE.

Usage

## S3 method for class 'PRROC'plot(x, xlim=c(0,1), ylim=c(0,1), auc.main=TRUE,     auc.type=c("integral","davis.goadrich"),     legend=ifelse(is.logical(color) & color==TRUE,4,NA), xlab=NULL, ylab=NULL,     main=NULL, color=TRUE, lwd=3, add=FALSE,     scale.color=hsv(h=seq(0,1,length=100)*0.8, s=1, v=1),     max.plot = FALSE, min.plot = FALSE, rand.plot = FALSE,     fill.area = (max.plot & min.plot), maxminrand.col = grey(0.5),     fill.color = grey(0.95), ...)

Arguments

Details

Theplot method for PRROC objects can be used in different ways.

The first is to plot a visualization of a single ROC or PR curvethat also represents the classification thresholds of individual points on the curve by a color scale. In this case, aPRROC object must be provided asx,add must beFALSE, andcolor must beTRUE. If, in addition,legend is set toTRUE, a legend translating colors to numerical threshold values is included to the right of the curve plotitself. The layout of curve plot and legend is accomplished usinglayout(), which means that this type of ROC/PR plot cannot be combinedwith other/complex layouts.

The second application of theplot method is to compare the performance of different classifiers (typically on the same data set). To do so,plot must be called withadd=FALSE andcolor set to one specific color (e.g., 2, "red",...) for the firstPRROC objectprovided asx. Subsequent calls ofplot withadd=TRUE can be used to add further curves to the first plot, where different colorsmay be specified by thecolor parameter.

In both cases, the first (or only) call toplot also allows for including plots of the maximum and minimum curve, highlighting the area between minimum and maximum, and the curve of a random classifier. For this purpose, thePRROC object needs to be created (usingpr.curve orroc.curve) with the correspondingparameters (e.g.,max.compute) set toTRUE.

Additional examples for the different use cases and corresponding plot commands are given in the documentations ofpr.curve androc.curve.

Author(s)

Jan Grau and Jens Keilwagen

See Also

pr.curve

roc.curve

Examples

# create artificial scores as random numbersx <- rnorm( 1000 );y <- rnorm( 1000, -1 );# compute PR curvepr <- pr.curve( x, y, curve = TRUE );# standard plot of PR curveplot( pr );# compute ROC curveroc <- roc.curve( x, y, curve = TRUE );# standard plot of ROC curveplot( roc );# create another set of scoresx.2 <- rnorm( 1000 );y.2 <- rnorm( 1000, -2 );# compute PR curvepr.2 <- pr.curve( x.2, y.2, curve=TRUE );# and ROC curveroc.2 <- roc.curve( x.2, y.2, curve=TRUE );# plot PR curve in red, without legendplot( pr, color = "red", auc.main=FALSE );# add second PR curve in greenplot( pr.2, color = 3, add = TRUE );# plot ROC curve in red, without legendplot( roc, color = "red", auc.main=FALSE);# add second ROC curve in greenplot( roc.2, color = 3, add = TRUE );# plot PR curve with legend below the main plotplot( pr, legend=1 );# compute PR curve with minimum and maximum curve, and random classifierpr <- pr.curve( x, y, curve = TRUE, max.compute = TRUE,   min.compute = TRUE, rand.compute = TRUE);# plot PR curve with area between minimum and # maximum curve in green and random classifier in blueplot(pr, rand.plot = TRUE, fill.area = TRUE, fill.color = rgb(0.8,1,0.8),   maxminrand.col = "blue" );

PR curve

Description

Computes the area under the precision-recall (PR) curve for weighted and unweighted data. In contrast to other implementations, the interpolation between points of the PR curve is done by a non-linear piecewise function. In addition to the area under the curve, the curve itself can be obtained by setting argumentcurve toTRUE.

Usage

pr.curve( scores.class0, scores.class1=scores.class0, weights.class0=NULL,     weights.class1 = {if(is.null(weights.class0)){NULL}else{1-weights.class0}},     sorted = FALSE, curve = FALSE,     minStepSize=min(1,ifelse(is.null(weights.class0),1,sum(weights.class0)/100)),    max.compute=F, min.compute=F, rand.compute=F,dg.compute=T)

Arguments

Details

This function computes the area under a precision-recall curve and, optionally, the curve itself and returns it as aPRROC object (see below).It can be used under different scenarios:

1. Standard, hard-labeled classification problems:

Each data point is uniquely assigned to one out of two possible classes. In this case, the classification scores may be either provided separatelyfor the data points of each of the classes, i.e., asscores.class0 for the data points from the positive/foreground class and asscores.class1for the data points of the negative/background class; or the classification scores for all data points are provided asscores.class0 and the labelsare provided as numerical values (1 for the positive class,0 for the negative class) asweights.class0.

2. Weighted, hard-labeled classification problems:

Each data point is uniquely assigned to one out of two possible classes, where each data points additionally has a weight assigned, for instancemultiplicities in the original data set. In this case, the classification scores need to be provided separatelyfor the data points of each of the classes, i.e., asscores.class0 for the data points from the positive/foreground class and asscores.class1for the data points of the negative/background class. In addition, the weights for the data points must be provided asweights.class0 andweights.class1, respectively.

3. Soft-labeled classification problems:

Each data point belongs to both of the two classes with a certain probability, where for each data point, these two probabilities add up to 1.In this case, the classification scores for all data points need to be provided only once asscores.class0 and only the positive/foreground weights for each data point need to be provided inweights.class0, while the converse probability for the negative class is automatically set toweights.class1=1.0-weights.class0.

Value

Author(s)

Jan Grau and Jens Keilwagen

References

J. Davis and M. Goadrich. The relationship between precision-recall and ROC curves.InProceedings of the 23rd International Conference on Machine Learning, pages 233–240, New York, NY, USA, 2006. ACM.

J. Keilwagen, I. Grosse, and J. Grau. Area under precision-recall curves for weighted and unweighted data, PLOS ONE (9) 3, 2014.

J. Grau, I. Grosse, and J. Keilwagen. PRROC: computing and visualizing precision-recall and receiver operating characteristic curves in R. Bioinformatics, 31(15):2595-2597, 2015.

See Also

roc.curve

plot.PRROC

Examples

# create artificial scores as random numbersx <- rnorm( 1000 );y <- rnorm( 1000, -1 );# compute area under PR curve for the hard-labeled casepr <- pr.curve( x, y );print( pr );# compute PR curve and area under curvepr <- pr.curve( x, y, curve = TRUE );# plot curveplot(pr);# create artificial weightsx.weights <- runif( 1000 );y.weights <- runif( 1000 );# compute PR curve and area under curve for weighted, hard-labeled datapr <- pr.curve( x, y, x.weights, y.weights, curve = TRUE );# and plot the curveplot(pr);# compute PR curve and area under curve,# and maximum, minimum, and random PR curve for weighted, hard-labeled datapr <- pr.curve(x, y, x.weights, y.weights, curve = TRUE, max.compute = TRUE,   min.compute = TRUE, rand.compute = TRUE);# plot all three curvesplot(pr, max.plot = TRUE, min.plot = TRUE, rand.plot = TRUE, fill.area = TRUE)# concatenate the drawn scoresscores<-c(x,y);# and create artificial soft-labelsweights<-c(runif(1000, min = 0.5, max = 1), runif(1000, min = 0, max = 0.5))# compute PR curve and area under curve,# and maximum, minimum, and random PR curve for soft-labeled datapr<-pr.curve(scores.class0 = scores, weights.class0 = weights, curve = TRUE,   max.compute = TRUE, min.compute = TRUE, rand.compute = TRUE);# plot all three curvesplot(pr, max.plot = TRUE, min.plot = TRUE, rand.plot = TRUE, fill.area = TRUE)# print the areas under the curvesprint(pr);# generate classification scores of a second classifierscores.2<-c(rnorm( 1000 ),rnorm( 1000, -2 ));# and compute the PR curvepr.2<-pr.curve(scores.class0 = scores.2, weights.class0 = weights, curve = TRUE)# plot all three curves for the first classifier in redplot(pr, max.plot = TRUE, min.plot = TRUE, rand.plot = TRUE, fill.area = TRUE,   color="red", auc.main=FALSE)# and add the curve for the second classifierplot(pr.2, add=TRUE, color="green")

printing PRROC objects

Description

Prints a PRROC object.

Usage

## S3 method for class 'PRROC'print(x, ...)

Arguments

Details

The print method for PRROC objects prints the area under the (PR or ROC) curve, and (ifcurve=TRUE inpr.curve orroc.curve) the range of classification scores. If alsomax.compute=TRUE,min.compute=TRUE, and/orrand.compute=TRUE when the PRROC object has been computes usingpr.curve orroc.curve, a relative area under curve is reported, i.e., the minimal AUC subtracted from the original AUC and the result divided by the difference of maximum and minimum AUC.

Author(s)

Jan Grau and Jens Keilwagen

See Also

pr.curve

roc.curve

Examples

# create artificial scores as random numbersx <- rnorm( 1000 );y <- rnorm( 1000, -1 );# compute area under PR curvepr <- pr.curve( x, y );print( pr );# compute area under ROC curveroc <- roc.curve( x, y );print( roc );

ROC curve

Description

Computes the area under the receiver operating characteristics (ROC) curve for weighted and unweighted data.In addition to the area under the curve, the curve can be obtained by setting argumentcurve toTRUE.

Usage

roc.curve( scores.class0, scores.class1=scores.class0, weights.class0=NULL,     weights.class1 = {if(is.null(weights.class0)){NULL}else{1-weights.class0}},     sorted = FALSE, curve = FALSE,     max.compute=F, min.compute=F, rand.compute=F)

Arguments

Details

This function computes the area under a receiver-operating characteristic (ROC) curve and, optionally, the curve itself and returns it as aPRROC object (see below).It can be used under different scenarios:

1. Standard, hard-labeled classification problems:

Each data point is uniquely assigned to one out of two possible classes. In this case, the classification scores may be either provided separatelyfor the data points of each of the classes, i.e., asscores.class0 for the data points from the positive/foreground class and asscores.class1for the data points of the negative/background class; or the classification scores for all data points are provided asscores.class0 and the labelsare provided as numerical values (1 for the positive class,0 for the negative class) asweights.class0.

2. Weighted, hard-labeled classification problems:

Each data point is uniquely assigned to one out of two possible classes, where each data points additionally has a weight assigned, for instancemultiplicities in the original data set. In this case, the classification scores need to be provided separatelyfor the data points of each of the classes, i.e., asscores.class0 for the data points from the positive/foreground class and asscores.class1for the data points of the negative/background class. In addition, the weights for the data points must be provided asweights.class0 andweights.class1, respectively.

3. Soft-labeled classification problems:

Each data point belongs to both of the two classes with a certain probability, where for each data point, these two probabilities add up to 1.In this case, the classification scores for all data points need to be provided only once asscores.class0 and only the positive/foreground weights for each data point need to be provided inweights.class0, while the converse probability for the negative class is automatically set toweights.class1=1.0-weights.class0.

Value

Author(s)

Jan Grau and Jens Keilwagen

References

J. Keilwagen, I. Grosse, and J. Grau. Area under precision-recall curves for weighted and unweighted data, PLOS ONE (9) 3, 2014.

J. Grau, I. Grosse, and J. Keilwagen. PRROC: computing and visualizing precision-recall and receiver operating characteristic curves in R. Bioinformatics, 31(15):2595-2597, 2015.

See Also

pr.curve

plot.PRROC

Examples

# create artificial scores as random numbersx <- rnorm( 1000 );y <- rnorm( 1000, -1 );# compute area under ROC curve for the hard-labeled caseroc <- roc.curve( x, y );print( roc );# compute ROC curve and area under curveroc <- roc.curve( x, y, curve = TRUE );# plot curveplot(roc);# create artificial weightsx.weights <- runif( 1000 );y.weights <- runif( 1000 );# compute ROC curve and area under curve for weighted, hard-labeled dataroc <- roc.curve( x, y, x.weights, y.weights, curve = TRUE );# and plot the curveplot(roc);# compute ROC curve and area under curve,# and maximum, minimum, and random ROC curve for weighted, hard-labeled dataroc <- roc.curve(x, y, x.weights, y.weights, curve = TRUE, max.compute = TRUE,   min.compute = TRUE, rand.compute = TRUE);# plot all three curvesplot(roc, max.plot = TRUE, min.plot = TRUE, rand.plot = TRUE, fill.area = TRUE)# concatenate the drawn scoresscores<-c(x,y);# and create artificial soft-labelsweights<-c(runif(1000, min = 0.5, max = 1), runif(1000, min = 0, max = 0.5))# compute ROC curve and area under curve,# and maximum, minimum, and random ROC curve  for soft-labeled dataroc<-roc.curve(scores.class0 = scores, weights.class0 = weights, curve = TRUE,   max.compute = TRUE, min.compute = TRUE, rand.compute = TRUE);# plot all three curvesplot(roc, max.plot = TRUE, min.plot = TRUE, rand.plot = TRUE, fill.area = TRUE)# print the areas under the curvesprint(roc);# generate classification scores of a second classifierscores.2<-c(rnorm( 1000 ),rnorm( 1000, -2 ));# and compute the ROC curveroc.2<-roc.curve(scores.class0 = scores.2, weights.class0 = weights, curve = TRUE)# plot all three curves for the first classifier in redplot(roc, max.plot = TRUE, min.plot = TRUE, rand.plot = TRUE, fill.area = TRUE,   color="red", auc.main=FALSE)# and add the curve for the second classifierplot(roc.2, add=TRUE, color="green")