To enable parallel testing, you must first be using the 3rd edition1. Then addthe following line to theDESCRIPTION:
Config/testthat/parallel: trueIf needed (for example, for debugging) you can temporarily suppressparallel testing withSys.setenv(TESTTHAT_PARALLEL = "false").
By default, testthat will usegetOption("Ncpus", 2)cores. To increase that value for your development machine we recommendsettingTESTTHAT_CPUS in your.Renviron. Theeasiest way to do that is callusethis::edit_r_environ()and then add something like the following:
TESTTHAT_CPUS=4Tests are run in alphabetical order by default, but you can oftenimprove performance by starting the slowest tests first. Specify thesetests by supplying a comma separated list of glob patterns2 to theConfig/testthat/start-first field in yourDESCRIPTION, e.g.:
Config/testthat/start-first: watcher, parallel*Each worker begins by loading testthat and the package being tested.It then runs any setup files (so if you have existing setup files you’llneed to make sure they work when executed in parallel).
testthat runs testfiles in parallel. Once the worker poolis initialized, testthat then starts sending test files to workers, bydefault in alphabetical order: as soon as a subprocess has finished, itreceives another file, until all files are done. This means that stateis persisted across test files: options arenot reset, loadedpackages arenot unloaded, the global environment isnot cleared, etc. You are responsible for making sure each fileleaves the world as it finds it.
If tests fail stochastically (i.e. they sometimes work andsometimes fail) you may have accidentally introduced a dependencybetween your test files. This sort of dependency is hard to track downdue to the random nature, and you’ll need to check all tests to makesure that they’re not accidentally changing global state.set_state_inspector() will make this easier.
If you usepackagedscope test fixtures, you’ll need to review them to make sure thatthey work in parallel. For example, if you were previously creating atemporary database in the test directory, you’d need to instead createit in the session temporary directory so that each process gets its ownindependent version.
There is some overhead associated with running tests in parallel:
Startup cost is linear in the number of subprocesses, because weneed to create them in a loop. This is about 50ms on my laptop. Eachsubprocess needs to load testthat and the tested package, this happensin parallel, and we cannot do too much about it.
Clean up time is again linear in the number of subprocesses, andit about 80ms per subprocess on my laptop.
It seems that sending a message (i.e. a passing or failingexpectation) is about 2ms currently. This is the total cost thatincludes sending the message, receiving it, and replying it to anon-parallel reporter.
This overhead generally means that if you have many test files thattake a short amount of time, you’re unlikely to see a huge benefit byusing parallel tests. For example, testthat itself takes about 10s torun tests in serial, and 8s to run the tests in parallel.
Seedefault_reporter() for how testthat selects thedefault reporter fordevtools::test() andtestthat::test_local(). In short, testthat selectsProgressReporter for non-parallel andParallelProgressReporter for parallel tests by default.(Other testthat test functions, liketest_check(),test_file() , etc. select different reporters bydefault.)
Most reporters support parallel tests. If a reporter is passed todevtools::test(),testthat::test_dir(), etc.directly, and it does not support parallel tests, then testthat runs thetest files sequentially.
Currently the following reportersdon’t support paralleltests:
DebugReporter, because it is not currently possibleto debug subprocesses.
JunitReporter, because this reporter records timinginformation for each test block, and this is currently only availablefor reporters that support multiple active test files. (See “Writingparallel reporters” below.)
LocationReporter because testthat currently does notinclude location information for successful tests when running inparallel, to minimize messaging between the processes.
StopReporter, as this is a reporter that testthatuses for interactiveexpect_that() calls.
The other built-in reporters all support parallel tests, with somesubtle differences:
Reporters that stop after a certain number of failures can onlystop at the end of a test file.
Reporters report all information about a file at once, unlessthey supportparallel updates. E.g.ProgressReporter does not update its display until a testfile is complete.
The standard output and standard error,i.e. print(),cat(),message(),etc. output from the test files are lost currently. If you want to usecat() ormessage() for print-debugging testcases, then the best is to temporarily run tests sequentially, bychanging theConfig entry inDESCRIPTION orsettingSys.setenv(TESTTHAT_PARALLEL = "false").
To support parallel tests, a reporter must be able to function whenthe test files run in a subprocess. For exampleDebugReporter does not support parallel tests, because itrequires direct interaction with the frames in the subprocess. Whenrunning in parallel, testthat does not provide location information(source references) for test successes.
To support parallel tests, a reporter must setself$capabilities$parallel_support toTRUE initsinitialize() method:
...initialize=function(...) { super$initialize(...) self$capabilities$parallel_support<-TRUE ...}...When running in parallel, testthat runs the reporter in the mainprocess, and relays information between the reporter and the test codetransparently. (Currently the reporter does not even know that the testsare running in parallel.)
If a reporter does not support parallel updates (see below), thentestthat internally caches all calls to the reporter methods fromsubprocesses, until a test file is complete. This is because thesereporters are not prepared for running multiple test files concurrently.Once a test file is complete, testthat calls the reporter’s$start_file() method, relays all$start_test(),$end_test(),$add_result(), etc. calls inthe order they came in from the subprocess, and calls$end_file() .
TheParallelProgressReporter supports parallel updates.This means that once a message from a subprocess comes in, the reporteris updated immediately. For this to work, a reporter must be able tohandle multiple test files concurrently.
A reporter declares parallel update support by settingself$capabilities$parallel_updates toTRUE:
...initialize=function(...) { super$initialize(...) self$capabilities$parallel_support<-TRUE self$capabilities$parallel_updates<-TRUE ...}...For these reporters, testthat does not cache the messages from thesubprocesses. Instead, when a message comes in:
It calls the$start_file() method, letting thereporter know which file the following calls apply to. This means thatthe reporter can receive multiple$start_file() calls forthe same file.
Then relays the message from the subprocess, calling theappropriate$start_test() ,$add_result(),etc. method.
testthat also calls the new$update() method of thereporter regularly, even if it does not receive any messages from thesubprocess. (Currently aims to do this every 100ms, but there are noguarantees.) The$update() method may implement a spinnerto let the user know that the tests are running.