This vignette is a quick reference guide for testing challengingfunctions. It’s organized by problem type rather than technique, so youcan quickly skim the whole vignette, spot the problem you’re facing, andthen learn more about useful tools for solving it. In it, you’ll learnhow to overcome the following challenges:
If your function depends on options or environment variables, firsttry refactoring the function to make theinputsexplicit. If that’s not possible, use functions likewithr::local_options() orwithr::local_envvar() to temporarily change options andenvironment values within a test. Learn more invignette("test-fixtures").
What happens if you want to test a function that relies on randomnessin some way? If you’re writing a random number generator, you probablywant to generate a large quantity of random numbers and then apply somestatistical test. But what if your function just happens to use a littlebit of pre-existing randomness? How do you make your tests repeatableand reproducible? Under the hood, random number generators generatedifferent numbers because they update a special.Random.seed variable stored in the global environment. Youcan temporarily set this seed to a known value to make your randomnumbers reproducible withwithr::local_seed(), makingrandom numbers a special case of test fixtures(vignette("test-fixtures")).
Here’s a simple example showing how you might test the basicoperation of a function that rolls a die:
dice<-function() {sample(6,1)}test_that("dice returns different numbers", { withr::local_seed(1234)expect_equal(dice(),4)expect_equal(dice(),2)expect_equal(dice(),6)})#> Test passed with 3 successes 🥇.Alternatively, you might want to mock(vignette("mocking")) the function to eliminaterandomness.
roll_three<-function() {sum(dice(),dice(),dice())}test_that("three dice adds values of individual calls", {local_mocked_bindings(dice =mock_output_sequence(1,2,3))expect_equal(roll_three(),6)})#> Test passed with 1 success 🌈.When should you set the seed and when should you use mocking? As ageneral rule of thumb, set the seed when you want to test the actualrandom behavior, and use mocking when you want to test the logic thatuses the random results.
You can skip a test without it passing or failing if you can’t ordon’t want to run it (e.g., it’s OS dependent, it only worksinteractively, or it shouldn’t be tested on CRAN). Learn more invignette("skipping").
If you’re trying to test functions that rely on HTTP requests, werecommend using {vcr} or {httptest2}. These packages both allow you tointeractively record HTTP responses and then later replay them in tests.This is a specialized type of mocking (vignette("mocking"))that works with {httr} and {httr2} to isolates your tests from failuresin the underlying API.
If your package is going to CRAN, youmust eitheruse one of these packages or useskip_on_cran() for allinternet-facing tests. Otherwise, you are at high risk of failingR CMD check if the underlying API is temporarily down. Thissort of failure causes extra work for the CRAN maintainers and extrahassle for you.
The only type of testing you can use for graphics is snapshot testing(vignette("snapshotting")) viaexpect_snapshot_file(). Graphical snapshot testing issurprisingly challenging because you need pixel-perfect rendering acrossmultiple versions of multiple operating systems, and this is hard,mostly due to imperceptible differences in font rendering. Fortunatelywe’ve needed to overcome these challenges in order to test {ggplot2},and you can benefit from our experience by using {vdiffr} when testinggraphical output.
If you’re testing a function that relies on user feedback (e.g. fromreadline(),utils::menu(), orutils::askYesNo()), you can use mocking(vignette("mocking")) to return fixed values within thetest. For example, imagine that you’ve written the following functionthat asks the user if they want to continue:
continue<-function(prompt) {cat(prompt,"\n",sep ="")repeat { val<-readline("Do you want to continue? (y/n) ")if (val%in%c("y","n")) {return(val=="y") }cat("! You must enter y or n\n") }}readline<-NULLYou could test its behavior by mockingreadline() andusing a snapshot test:
test_that("user must respond y or n", { mock_readline<-local({ i<-0function(prompt) { i<<- i+1cat(prompt) val<-if (i==1)"x"else"y"cat(val,"\n",sep ="") val } })local_mocked_bindings(readline = mock_readline)expect_snapshot(val<-continue("This is dangerous"))expect_true(val)})#> ── Warning: user must respond y or n ───────────────────────────────────────────#> Adding new snapshot:#> Code#> val <- continue("This is dangerous")#> Output#> This is dangerous#> Do you want to continue? (y/n) x#> ! You must enter y or n#> Do you want to continue? (y/n) y#> Test passed with 2 successes 🎊.If you don’t care about reproducing the output ofcontinue() and just want to recreate its return value, youcan usemock_output_sequence(). This creates a functionthat returns the input supplied tomock_output_sequence()in sequence: the first input at the first call, the second input at thesecond call, etc. The following code shows how it works and how youmight use it to testreadline():
And
test_that("user must respond y or n", {local_mocked_bindings(readline =mock_output_sequence("x","y"))expect_true(continue("This is dangerous"))})#> This is dangerous#> ! You must enter y or n#> Test passed with 1 success 😀.If you were testing the behavior of some function that usescontinue(), you might want to mockcontinue()instead ofreadline(). For example, the function belowrequires user confirmation before overwriting an existing file. In orderto focus our tests on the behavior of just this function, we mockcontinue() to return eitherTRUE orFALSE without any user messaging.
save_file<-function(path, data) {if (file.exists(path)) {if (!continue("`path` already exists")) {stop("Failed to continue") } }writeLines(data, path)}test_that("save_file() requires confirmation to overwrite file", { path<- withr::local_tempfile(lines = letters)local_mocked_bindings(continue =function(...)TRUE)save_file(path,"a")expect_equal(readLines(path),"a")local_mocked_bindings(continue =function(...)FALSE)expect_snapshot(save_file(path,"a"),error =TRUE)})#> ── Warning: save_file() requires confirmation to overwrite file ────────────────#> Adding new snapshot:#> Code#> save_file(path, "a")#> Condition#> Error in `save_file()`:#> ! Failed to continue#> Test passed with 2 successes 🥇.Errors, warnings, and other user-facing text should be tested toensure they’re both actionable and consistent across the package.Obviously, it’s not possible to test this automatically, but you can usesnapshots (vignette("snapshotting")) to ensure thatuser-facing messages are clearly shown in PRs and easily reviewed byanother human.
If you find yourself repeating the same set of expectations again andagain across your test suite, it may be a sign that you should designyour own expectation. Learn how invignette("custom-expectation").