Movatterモバイル変換

confcons

‘confcons’ (confidence &consistency) is a light-weight, stand-alone R packagedesigned to calculate the following two novel measures of predictivedistribution models (incl. species distribution models):

• confidence that measures the proportion of predictions thatthe model is confident in;
• consistency that measures how consistent the model is ifthe confidence in the training and evaluation subsets are compared.

Whileconfidence serves as a replacement for the widelycriticized goodness-of-fit measures, such as AUC,consistencyis a proxy for model’s transferability (in space and time).

Installation

You can install the latest stable version of ‘confcons’ from CRANwith:

install.packages("confcons")

You can install the development version of ‘confcons’ fromGitHub with:

# install.packages("devtools")devtools::install_github(repo ="bfakos/confcons",upgrade ="never")

If you want to read thevignetteof the development version in R, install the package with:

devtools::install_github(repo ="bfakos/confcons",upgrade ="never",build_vignettes =TRUE)

Examples

Three small functions,thresholds(),confidence() andconsistency(), belong to thecore of the package. A wrapper function calledmeasures()utilizes these workhorse functions and calculates every measures for youoptionally along with some traditional measures, such as AUC andmaxTSS.

Let’s say we trained a predictive distribution model and made somepredictions with it, and now we want to be sure if our model is both

• confident in the predictions, and
• this confidence is consistent between the training and evaluationsubsets, i.e. we might later use the model for extrapolation (throughspace or time).

Our example dataset is adata.frame containing both thetraining and the evaluation subset. It is organized in threecolumns:

• observations (integer): observed presences(1s) and absences (0s),
• predictions (numeric): predicted probability ofoccurrences (within the[0, 1] interval), and
• evaluation_mask (logical): indicates whether a certainrow belongs to the evaluation subset (TRUE) or the trainingsubset (FALSE).

dataset<-data.frame(observations =c(0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L),predictions =c(0.1,0.2,0.4,0.5,0.5,0.2,0.3,0.3,0.4,0.3,0.65,0.9,0.9,1,0.1,0.5,0.8,0.8),evaluation_mask =c(FALSE,FALSE,FALSE,FALSE,FALSE,TRUE,TRUE,TRUE,TRUE,FALSE,FALSE,FALSE,FALSE,FALSE,TRUE,TRUE,TRUE,TRUE))

Well, it is a really small dataset…

Let’s attach the package to our R session:

library(confcons)

Now we can calculate the measures:

measures(observations = dataset$observations,predictions = dataset$predictions,evaluation_mask = dataset$evaluation_mask)#>    CP_train     CP_eval         DCP   CPP_train    CPP_eval        DCPP#>  0.80000000  0.75000000 -0.05000000  0.75000000  0.66666667 -0.08333333

The function returns

• a pair of confidence values for the training subset(CP_train,CPP_train) just for ourinformation;
• a pair of confidence values for the evaluation subset(CP_eval,CPP_eval), both describing theconfidence of our model;
• a pair of consistency values (DCP,DCPP)that serve as proxies for transferability of our model.

The difference between the two values forming the pairs is describedinthisscientific publication.

Our model seems to be not super perfect, but it is more or lessconfident in the positive predictions (i.e. predicted presences), sinceCPP_eval is closer to 1 than to 0. Even if not absolutelyconfident, it is really consistent (i.e.,DCPP is close to0), so we might not afraid of transferability issues if used for spatialor temporal extrapolation.

A detailed description of the measures and the functions of‘confcons’, and more examples can be found inthisvignette.

Citation

When you use this package, please cite the following scientificpaper:

Somodi I, Bede-Fazekas Á, Botta-Dukát Z, Molnár Z (2024):Confidence and consistency in discrimination: A new family ofevaluation metrics for potential distribution models. EcologicalModelling 491: 110667. DOI:10.1016/j.ecolmodel.2024.110667.

Package lifecycle andcontribution

This GitHub version of the package is now in stable state. If youfind a bug or have a feature request, or also if you have some idea wantto discuss with the authors of the package, please create anew issue.