New sample size specific diagnostic threshold for Paretok. The pre-2022 version of thePSIS paper recommendeddiagnostic thresholds ofk < 0.5 "good",0.5 <= k < 0.7 "ok",0.7 <= k < 1 "bad",k>=1 "very bad".The 2022 revision of the PSIS paper now recommendsk < min(1 - 1/log10(S), 0.7) "good",min(1 - 1/log10(S), 0.7) <= k < 1 "bad",k > 1 "very bad", whereS is the samplesize. There is now one fewer diagnostic threshold ("ok" hasbeen removed), and the most important threshold now depends on thesample sizeS. With sample sizes100,320,1000,2200,10000 the sample size specific part1 - 1/log10(S) corresponds to thresholds of0.5,0.6,0.67,0.7,0.75. Even if the sample size grows, the bias in the PSISestimate dominates if0.7 <= k < 1, and thus thediagnostic threshold for good is capped at0.7 (ifk > 1, the mean does not exist and bias is not a validmeasure). The new recommended thresholds are based on more carefulbias-variance analysis of PSIS based on truncated Pareto sums theory.For those who use the Stan default 4000 posterior draws, the0.7 threshold will be roughly the same, but there will befewer warnings as there will be no diagnostic message for0.5 <= k < 0.7. Those who use smaller sample sizesmay see diagnostic messages with a threshold less than0.7,and they can simply increase the sample size to about2200to get the threshold to0.7.
No more warnings if ther_eff argument isnot provided, and the default is nowr_eff = 1.The summary print output showing MCSE and ESS now shows diagnosticinformation on the range ofr_eff. The change was made toreduce unnecessary warnings. The use ofr_eff does notchange the expected value ofelpd_loo,p_loo,and Paretok, and is needed only to estimate MCSE and ESS.Thus it is better to show the diagnostic information aboutr_eff only when MCSE and ESS values are shown.
k Inf if it is NA byE_loo() when type is variance byE_loo() now allowstype="sd" bypointwise() convenience function for extractingpointwise estimates byk threshold bymcse_elpd using log-normal approximation by@avehtari in#246n_eff/ESS ifk > k_threshold byE_loo() Pareto-k diagnostics byloo_subsample.R byNewloo_predictive_metric() function for computingestimates of leave-one-out predictive metrics: mean absolute error, meansquared error and root mean squared error for continuous predictions,and accuracy and balanced accuracy for binary classification. (#202,
New functionscrps(),scrps(),loo_crps(), andloo_scrps() for computing the(scaled) continuously ranked probability score. (#203,
New vignette “Mixture IS leave-one-out cross-validation forhigh-dimensional Bayesian models.” This is a demonstration of themixture estimators proposed bySilva and Zanella (2022).(#210)
loo_model_weights() to make them consistent withloo_compare(). (#217)NewFrequentlyAsked Questions page on the package website. (#143)
Speed improvement from simplifying the normalization when fittingthe generalized Pareto distribution. (#187,
Added parallel likelihood computation to speeduploo_subsample() when using posterior approximations. (#171,
Switch unit tests from Travis to GitHub Actions. (#164)
save_psis = TRUE (#166,Fixed a bug inrelative_eff.function() that causedan error on Windows when using multiple cores. (#152)
Fixed a potential numerical issue inloo_moment_match() withsplit=TRUE.(#153)
Fixed potential integer overflow withloo_moment_match(). (#155,
Fixedrelative_eff() when used with aposterior::draws_array. (#161,
elpd() (and methods for matricesand arrays) for computing expected log predictive density of new data orlog predictive density of observed data. A new vignette demonstratesusing this function when doing K-fold CV with rstan. (#159,loo_moment_match() that prevented... arguments from being used correctly. (#149)Added Topi Paananen and Paul Bürkner as coauthors.
New functionloo_moment_match() (and new vignette),which can be used to update aloo object when Pareto kestimates are large. (#130)
The log weights provided by the importance sampling functionspsis(),tis(), andsis() nolonger have the largest log ratio subtracted from them when returned tothe user. This should be less confusing for anyone using theweights() method to make an importance sampler. (#112,#146)
MCSE calculation is now deterministic (#116, #147)
Added Mans Magnusson as a coauthor.
New functionsloo_subsample() andloo_approximate_posterior() (and new vignette) for doingPSIS-LOO with large data. (#113)
Added support for standard importance sampling and truncatedimportance sampling (functionssis() andtis()). (#125)
compare() now throws a deprecation warningsuggestingloo_compare(). (#93)
A smaller threshold is used when checking the uniqueness of tailvalues. (#124)
For WAIC, warnings are only thrown when runningwaic() and not when printing awaic object.(#117,@mcol)
Use markdown syntax in roxygen documentation wherever possible.(#108)
New functionloo_compare() for model comparison thatwill eventually replace the existingcompare() function.(#93)
New vignette on LOO for non-factorizable joint Gaussian models.(#75)
New vignette on “leave-future-out” cross-validation for timeseries models. (#90)
New glossary page (usehelp("loo-glossary")) withdefinitions of key terms. (#81)
Newse_diff column in model comparison results.(#78)
Improved stability ofpsis() whenlog_ratios are very small. (#74)
Allowr_eff=NA to suppress warning when specifyingr_eff is not applicable (i.e., draws not from MCMC).(#72)
Update effective sample size calculations to match RStan’sversion. (#85)
Naming of k-fold helper functions now matches scikit-learn.(#96)
This is a major release with many changes. Whenever possible we haveopted to deprecate rather than remove old functionality, but it ispossible that old code that accesses elements inside loo objects byposition rather than name may error.
New package documentation website http://mc-stan.org/loo/ withvignettes, function reference, news.
Updated existing vignette and added two new vignettesdemonstrating how to use the package.
New functionpsis() replacespsislw()(now deprecated). This version implements the improvements to the PSISalgorithm described in the latest version ofhttps://arxiv.org/abs/1507.02646. Additional diagnostic information isnow also provided, including PSIS effective sample sizes.
Newweights() method for extracting smoothed weightsfrom apsis object. Argumentslog andnormalize control whether the weights are returned on thelog scale and whether they are normalized.
Updated the interface for theloo() methods tointegrate nicely with the new PSIS algorithm. Methods for log-likelihoodarrays, matrices, and functions are provided. Several arguments havechanged, particularly for theloo.function method. Thedocumentation athelp("loo") has been updated to describethe new behavior.
The structure of the objects returned by theloo()function has also changed slightly, as described in theValue section athelp("loo", package = "loo").
New functionloo_model_weights() computes weightsfor model averaging as described in https://arxiv.org/abs/1704.02030.Implemented methods include stacking of predictive distributions,pseudo-BMA weighting or pseudo-BMA+ weighting with the Bayesianbootstrap.
Settingoptions(loo.cores=...) is now deprecated infavor ofoptions(mc.cores=...). For now, if both theloo.cores andmc.cores options have been set,preference will be given toloo.cores until it is removedin a future release. (thanks to
New functionsexample_loglik_array() andexample_loglik_matrix() that provide objects to use inexamples and tests.
When comparing more than two models withcompare(),the first column of the output is now theelpd differencefrom the model in the first row.
New helper functions for splitting observations for K-fold CV:kfold_split_random(),kfold_split_balanced(),kfold_split_stratified(). Additional helper functions forimplementing K-fold CV will be included in future releases.
E_loo() function for computing weightedexpectations (means, variances, quantiles).pareto_k_table() andpareto_k_ids()convenience functions for quickly identifying problematicobservations(-Inf, 0.5],(0.5, 0.7],(0.7, 1],(1, Inf)(didn’t used to include 0.7)psislw() instead ofprint.looprint.loo() shows a table of pareto k estimates (if anyk > 0.7)compare() to allow loo objects to beprovided in a list rather than in'...'extract_log_lik()compare(). In previousversions ofloo model weights were also reported bycompare(). We have removed the weights because they werebased only on the point estimate of the elpd values ignoring theuncertainty. We are currently working on something similar to theseweights that also accounts for uncertainty, which will be included infuture versions ofloo.This update makes it easier for other package authors usingloo to write tests that involve running theloo function. It also includes minor bug fixes andadditional unit tests. Highlights:
cores=1.psislw function is called in aninteractive session.This update provides several important improvements, most notably analternative method for specifying the pointwise log-likelihood thatreduces memory usage and allows forloo to be used withlarger datasets. This update also makes it easier to to incorporateloo’s functionality into other packages.
matrix andfunctionmethods for bothloo() andwaic(). The matrixmethod provide the same functionality as in previous versions ofloo (taking a log-likelihood matrix as the input). Thefunction method allows the user to provide a function for computing thelog-likelihood from the data and posterior draws (which are alsoprovided by the user). The function method is less memory intensive andshould make it possible to useloo for models fit tolarger amounts of data than before.plot andprint methods.plot also provideslabel_points argument,which, ifTRUE, will label any Paretok pointsgreater than 1/2 by the index number of the corresponding observation.The plot method also now warns aboutInf/NA/NaN values ofk that are not shown in the plot.compare now returns model weights and accepts more thantwo inputs.options(loo.cores = NUMBER).loo_and_waic function in favor of separatefunctionsloo andwaicloo_and_waic_diff. Usecompareinstead.