Repository files navigation

Tools to measure performance improvements in shiny apps.

shiny.benchmark is a tool aimed to measure and compare the performance of different versions of ashiny application. Based on a list of different application versions, accessible by a git repo by its refs (commit hash or branch name), the user can write instructions to be executed using Cypress orshinytest2. These instructions are then evaluated by the different versions of yourshiny application and therefore the performance's improvement/deterioration (time elapsed) are be recorded.

The package is flexible enough to allow different sets of tests for the different refs as well as different package versions (viarenv). Also, the user can replicate the tests to have more accurate measures of performance.

Stable version:

install.packages("shiny.benchmark")

Development version:

remotes::install_github("Appsilon/shiny.benchmark")

shiny.benchmark can use two different engines to test the change in the performance of your application:shinytest2 andCypress.The latter requiresNode (version 12 or higher) andyarn (version 1.22.17 or higher) to be available.To install them on your computer, follow the guidelines on the documentation pages:

• Node
• yarn

Besides that, on Linux, it might be required to install otherCypress dependencies.Check thedocumentation to find out more.

The best way to start usingshiny.benchmark is through an example. If you want a start point, you can use theload_example function. In order to use this, create a new folder in your computer and use the following code to generate an application to serve us as example for our performance checks:

library(shiny.benchmark)load_example(path="path/to/new/project")

It will create some useful files underpath/to/new/project. The most important one is therun_tests.R which provides several instructions at the very top.

As we are comparing versions of the same application, we need different app versions in different branches/commits ingit. Start usingcd app; git init to initiate git insideapp/ folder.

Get familiar withapp/server.R file in order to generate more interesting scenarios. The basic idea is to use theSys.sleep function to simulate some app's functionalities. Remember that, when running the benchmark, that is the amount of time it will take to measure the performance.

When you are ready, commit your changes in master/main usinggit add .; git commit -m "your commit message". Make some editions and commit these new changes into a new branch or in the same branch your are testing (it will have a different commit hash). Repeat the process adding as many new modifications as you want. E.g. add renv, add more tests, change the names of the tests/test files and so on.

Here is a complete example on how to setup yourgit:

# startinggit initecho .Rproj.user >> .gitignoreecho *.Rproj >> .gitignoreecho .Rprofile >> .gitignoreecho renv >> .gitignoreecho .Rprofile >> .gitignore# mastergit add .git commit -m "first commit"# develop (decrease Sys.sleep times in server.R)git checkout -b developgit add .git commit -m "improving performance"## Using renvgit branch renv_shiny1 developgit checkout renv_shiny1R -e 'renv::init()'git add .git commit -m "renv active"## Downgrading shinygit checkout -b renv_shiny2R -e 'renv::install("shiny@1.7.0")'R -e 'renv::snapshot()'git add .git commit -m "downgrading shiny"## Switching back to developgit checkout develop

Now you are ready to go. Thebenchmark function provides several arguments to make your life easier when running your performance checks. The mandatory arguments are:

• commit_list: A vector with commits, branches or anything else you can use ingit checkout
• cypress_dir orshinytest2_dir: Folder containing the tests we want to check the performance. In our case it istests/cypress andtests respectively.

The default behavior is to try to userenv in your project. If you do not have the renv structure, you can turnrenv off usinguse_renv = FALSE

library(shiny.benchmark)# commits to comparecommit_list<- c("develop","renv_shiny1","renv_shiny2")# run performance check using Cypressbenchmark(commit_list=commit_list,cypress_dir="tests/cypress")

That is all you need to run yourCypress tests. If you don't useCypress, you may want to useshinytest2 instead:

benchmark(commit_list=commit_list,shinytest2_dir="tests")

To run just specific tests, you can take advantage of thetests_pattern argument. It will filter the test file's names based on regular expression:

benchmark(commit_list=commit_list,shinytest2_dir="tests",tests_pattern="use_this_one_[0-9]")

If your project hasrenv structure, you can setuse_renv toTRUE to guarantee that, for each application version your are using the correct packages. If you want to approve/reproverenv::restore(), you can setrenv_prompt = TRUE.

benchmark(commit_list=commit_list,shinytest2_dir="tests",tests_pattern="use_this_one_[0-9]",use_renv=TRUE,# defaultrenv_prompt=TRUE)

To have more accurate information about the time your application takes to perform some actions, you may need to replicate the tests. In this case, you can use then_rep argument:

out<- benchmark(commit_list=commit_list,cypress_dir="tests/cypress",tests_pattern="use_this_one_[0-9]",use_renv=FALSE,n_rep=15)out

For fast information about the tests results, you can use thesummary and also theplot methods:

summary(out)plot(out)

If you want to contribute to this project please submit a regular PR, once you're done with new feature or bug fix.

Reporting a bug is also helpful - please useGitHub issues and describe your problem as detailed as possible.

Appsilon is aPosit (formerly RStudio) Full Service Certified Partner.
Learn more atappsilon.com.

Get in touchopensource@appsilon.com

Explore theRhinoverse - a family of R packages built aroundRhino!