NotificationsYou must be signed in to change notification settings
Fork6
Star161

Adding progress bar to '*apply' functions in R

161 stars 6 forks Branches Tags Activity

You must be signed in to change notification settings

Folders and files

Name		Name	Last commit message	Last commit date
Latest commit History 1,439 Commits
.github/workflows		.github/workflows
R		R
images		images
inst		inst
man		man
tests		tests
.Rbuildignore		.Rbuildignore
.gitignore		.gitignore
DESCRIPTION		DESCRIPTION
NAMESPACE		NAMESPACE
NEWS.md		NEWS.md
README.md		README.md
RELEASE.R		RELEASE.R
_pkgdown.yml		_pkgdown.yml
pbapply.Rproj		pbapply.Rproj

Repository files navigation

pbapply: adding progress bar to '*apply' functions in R

A lightweight package that adds progress bar to vectorized R functions(*apply). The implementation can easily be added to functions where showing the progress isuseful (e.g. bootstrap). The type and style of the progress bar (with percentages or remaining time) can be set through options.The package supports severalparallel processing backends,such as snow-type and mirai clusters, multicore-type forking, and future.

pbapply: adding progress bar to '*apply' functions in R

Versions

Install CRAN release version (recommended):

install.packages("pbapply")

Development version:

install.packages("pbapply",repos="https://psolymos.r-universe.dev")

See user-visible changes in theNEWS file.

Use theissue trackerto report a problem, or to suggest a new feature.

How to get started?

1. You are not yet an R user

In this case, start with understanding basic programming concepts,such as data structures (matrices, data frames, indexing these),for loops and functions in R.The online version of Garrett Grolemund'sHands-On Programming with Rwalks you through these concepts nicely.

2. You are an R user but haven't used vectorized functions yet

Learn about vectorized functions designed to replacefor loops:lapply,sapply, andapply.Here is a repository calledThe Road to Progressthat I created to show you how to go from afor loop tolapply/sapply.

3. You are an R user familiar with vectorized functions

In this case, you can simply addpbapply::pb before your*applyfunctions, e.g.apply() will becomepbapply::pbapply(), etc.You can guess what happens.Now if you want to speed things up a little (or a lot),trypbapply::pbapply(..., cl = 4) to use 4 cores instead of 1.

If you are a Windows user, things get a bit more complicated, but not much.Check how to work withparallel::parLapply to set up a snow type clusteror use a suitable future backend (see some examplesbelow).Have a look at theThe Road to Progressrepository to see more worked examples.

4. You are a seasoned R developer writing your own packages

Read on, the next section is for you.

How to add pbapply to a package

There are two ways of adding the pbapply package to another package.

1. Suggests: pbapply

Add pbapply to theSuggests field in theDESCRIPTION.

Use a conditional statement in your code to fall back on a base function in case of pbapply is not installed:

out<-if (requireNamespace("pbapply",quietly=TRUE)) {pbapply::pblapply(X,FUN,...)}else {   lapply(X,FUN,...)}

See a small example packagehere.

2. Depends/Imports: pbapply

Add pbapply to theDepends orImports field in theDESCRIPTION.

Use the pbapply functions either aspbapply::pblapply() or specify them in theNAMESPACE (importFrom(pbapply, pblapply)) anduse it aspblapply() (without the::).You'd have to add a comment#' @importFrom pbapply pblapply if you areusing roxygen2.

Customizing the progress bar in your package

Specify the progress bar options in thezzz.R file of the package:

.onAttach<-function(libname,pkgname){    options("pboptions"=list(type=if (interactive())"timer"else"none",char="-",txt.width=50,gui.width=300,style=3,initial=0,title="R progress bar",label="",nout=100L,min_time=2))invisible(NULL)}

This will set the options and pbapply will not override these when loaded.

See a small example packagehere.

Suppressing the progress bar in your functions

Suppressing the progress bar is sometimes handy. By default, progress bar is suppressed when!interactive().In other instances, put this inside a function:

pbo<- pboptions(type="none")on.exit(pboptions(pbo),add=TRUE)

Working with a future backend

The future backend might require additional arguments to be set by package developers to avoid warnings for end users.Most notably, you will have to determine how to handle random number generation as part of parallel evaluation.You can pass thefuture.seed argument directly through....In general, ass any additional arguments toFUN immediately following theFUN argument,and any additional arguments to the the future backend aftercl = "future" statement:

pblapply(1:2,FUN=my_fcn, {additionalmy_fcnargs},cl="future", {additionalfutureargs})

Seethis issue for a discussion.

Examples

The followingpb* functions are available in the pbapply package:

base	pbapply	works in parallel
`apply`	`pbapply`	✅
`by`	`pbby`	✅
`eapply`	`pbeapply`	✅
`lapply`	`pblapply`	✅
`.mapply`	`pb.mapply`	❌
`mapply`	`pbmapply`	❌
`Map`	`pbMap`	❌
`replicate`	`pbreplicate`	✅
`sapply`	`pbsapply`	✅
`tapply`	`pbtapply`	✅
`vapply`	`pbvapply`	✅
❌	`pbwalk`	✅

Command line usage

library(pbapply)set.seed(1234)n<-2000x<- rnorm(n)y<- rnorm(n, model.matrix(~x)%*% c(0,1),sd=0.5)d<-data.frame(y,x)## model fitting and bootstrapmod<- lm(y~x,d)ndat<- model.frame(mod)B<-500bid<- sapply(1:B,function(i) sample(nrow(ndat), nrow(ndat),TRUE))fun<-function(z) {if (missing(z))z<- sample(nrow(ndat), nrow(ndat),TRUE)    coef(lm(mod$call$formula,data=ndat[z,]))}## standard '*apply' functions# system.time(res1 <- lapply(1:B, function(i) fun(bid[,i])))#    user  system elapsed#   1.096   0.023   1.127system.time(res2<- sapply(1:B,function(i) fun(bid[,i])))#    user  system elapsed#   1.152   0.017   1.182system.time(res3<- apply(bid,2,fun))#    user  system elapsed#   1.134   0.010   1.160system.time(res4<- replicate(B, fun()))#    user  system elapsed#   1.141   0.022   1.171## 'pb*apply' functions## try different settings:## "none", "txt", "tk", "win", "timer"op<- pboptions(type="timer")# defaultsystem.time(res1pb<- pblapply(1:B,function(i) fun(bid[,i])))#    |++++++++++++++++++++++++++++++++++++++++++++++++++| 100% ~00s#    user  system elapsed#   1.539   0.046   1.599pboptions(op)pboptions(type="txt")system.time(res2pb<- pbsapply(1:B,function(i) fun(bid[,i])))#   |++++++++++++++++++++++++++++++++++++++++++++++++++| 100%#    user  system elapsed#   1.433   0.045   1.518pboptions(op)pboptions(type="txt",style=1,char="=")system.time(res3pb<- pbapply(bid,2,fun))# ==================================================#    user  system elapsed#   1.389   0.032   1.464pboptions(op)pboptions(type="txt",char=":")system.time(res4pb<- pbreplicate(B, fun()))#   |::::::::::::::::::::::::::::::::::::::::::::::::::| 100%#    user  system elapsed#   1.427   0.040   1.481pboptions(op)

Parallel backends

You have a few different options to choose from as a backend. This all comes down to thecl argument in thepb* functions.

cl = NULL (default): sequential execution
cl is of class cluster: this implies that you usedcl = parallel::makeCluster(n) or something similar (n being the number of worker nodes)
cl is a positive integer (usually > 1): forking type parallelism is used in this case
cl = "future": you are using one of thefuture plans and parallelism is defined outside of thepb* call.

Note that on Windows the forking type is not available andpb* functions will fall back to sequential evaluation.

Some examples:

f<-function(i) Sys.sleep(1)## sequentialpblapply(1:2,f)## clustercl<-parallel::makeCluster(2)pblapply(1:2,f,cl=cl)parallel::stopCluster(cl)# mirai clusterlibrary(mirai)# -- using the mirai packagecl<- make_cluster(2)pblapply(1:2,f,cl=cl)stop_cluster(cl)# -- using parallel (requires R >= 2.5)cl<-parallel::makeCluster(2,type="MIRAI")pblapply(1:2,f,cl=cl)parallel::stopCluster(cl)## forkingpblapply(1:2,f,cl=2)## futurelibrary(future)cl<-parallel::makeCluster(2)plan(cluster,workers=cl)r2<- pblapply(1:2,f,cl="future")parallel::stopCluster(cl)plan(multisession,workers=2)pblapply(1:2,f,cl="future")plan(sequential)

Progress with Shiny

library(shiny)library(pbapply)pboptions(type="shiny",title="Shiny progress",label="Almost there ...")ui<- fluidPage(    plotOutput("plot"))server<-function(input,output,session) {output$plot<- renderPlot({        pbsapply(1:15,function(z) Sys.sleep(0.5))        plot(cars)    })}shinyApp(ui,server)