shinytest2::AppDriver$expect_values()screenshot snapshot failing on CI (#2293, #2288).expect_snapshot(binary), soft deprecated in 3.0.3(2021-06-16), is now fully deprecated.is_null()/matches(), deprecated in 2.0.0(2017-12-19), andis_true()/is_false(),deprecated in 2.1.0 (2019-04-23), have been removed (#2109).local_mock() andwith_mock(), deprecatedin 3.0.0 (2020-10-31), are now defunct. They technique that made themwork is no longer permitted in R 4.5.0.test_files(wrap), deprecated in 3.0.0 (2020-10-31) hasnow been removed.expect_ functions have had their failure messagesrewritten. They now all state what was expected, what was actuallyreceived, and, if possible, clearly illustrate the difference (#2142).They also consistently check that their inputs are the correct type(#1754). They consistently return the value of the first argument,regardless of whether the expectation succeeds or fails (the onlyexception isexpect_message() and friends which return thecondition). This shouldn’t affect existing tests, but will make failuresclearer when you chain together multiple expectations (#2246).vignette("custom-expectations") (#2113, #2132,#2072), a newpass() function to use in place ofsucceed() (#2113), and revisions to howexpectation() works (#2125).expect_no_failures() andexpect_no_successes()are now deprecated as nowexpect_success() also tests forthe absence of failures andexpect_failure() tests for theabsence of successes.test_that(),describe(), andit() calls) can now be arbitrarily nested with a sharedstack of descriptions so that the failures can show the full paththrough all subtests. Skips in subtests are now correctly scoped(#2007), and each subtest and will skip if and only if it and itssubtests don’t contain any expectations.test_that() will automatically skips if apackage is not installed (#1585). Practically, this means that you nolonger need to check that suggested packages are installed. (We alreadydidn’t check the installation of suggested packages in the tidyversebecause we think it has limited payoff, but other styles advisedifferently.)expect_all_equal(),expect_all_true(),andexpect_all_false() check that every element of a vectorhas the same value (#1836, #2235), giving better error messages thanexpect_true(all(...)).expect_disjoint() expects values to to be absent(@stibu81,#1851).expect_r6_class() expects an R6 objects(#2030).expect_shape() expects a specific shape (i.e.,nrow(),ncol(), ordim()) (#1423,extract_test() function to extract a reprex from afailing expectation. tests run inR CMD check will use thisto automatically create a reprex in the_problems/directory for each failing expectation. You can turn this behaviour offby settingTESTTHAT_PROBLEMS=false (#2263).local_mocked_s3_method(),local_mocked_s4_method(), andlocal_mocked_r6_class() allow you to mock S3 and S4 methodsand R6 classes (#1892, #1916)local_on_cran(TRUE) allows you to simulate how yourtests will run on CRAN (#2112).test_dir(),test_file(),test_package(),test_check(),test_local(),source_file() gain ashuffle argument that usessample() torandomly reorder the top-level expressions in each test file (#1942).This random reordering surfaces dependencies between tests and codeoutside of any test, as well as dependencies between tests. This helpsyou find and eliminate unintentional dependencies.try_again() is now publicised, making it easier to testflaky code. We changed the first argument to represent the number ofretries, not tries (#2050).skip_unless_r() skip tests on unsuitable versionsof R. It has a convenient sytnax so you can use, e.g.,skip_unless_r(">= 4.1.0") to skip tests that require...names() (snapshot_reject() rejects all modified snapshots bydeleting the.new variants (#1923), andsnapshot_download_gh() makes it easy to get snapshots offGitHub and into your local package (#1779).expect_snapshot() and friends will now fail whencreating a new snapshot on CI. This is usually a signal that you’veforgotten to run it locally before committing (#1461).SlowReporter makes it easier to find the slowesttests in your package. You can run it withdevtools::test(reporter = "slow") (#1466).vignette("mocking") explains mocking in detail(#1265).vignette("challenging-functions") provides an indexto other documentation organised by various challenges (#1265).std::uncaught_exception() (2,#2047).expect_lt(),expect_gt(), and friends havea refined display that is more likely to display the correct number ofdigits and shows you the actual values compared (#2006). They have abetter display for non-numeric data (expect_matches() failures should be a little easier toread (#2135, #2181).expect_no_*() now executes the entire code block,rather than stopping at the first message or warning (#1991).expect_named() now gives more informative errors(#2091).expect_s4_class() now supports unquoting (expect_snapshot() no longer loses the last snapshot iffile is missing the final newline (#2092). It’s easy to accidentallyremove this because there are two trailing new lines in snapshot filesand many editors will automatically remove one if you touch thefile.expect_snapshot() on reports how to resolve failuresonce when running insideR CMD check.expect_snapshot() no longer skips on CRAN, as thatskips the rest of the test. Instead it just returns, neither succeedingnor failing (#1585).expect_snapshot() andexpect_snapshot_file() hints now include the path to thepackage, if it’s not the current working directory (#1577).expect_snapshot() gives a more informative backtracewhen the code inside the snapshot errors (#2277).expect_snapshot_file() now clearly errors if thepath doesn’t exist (#2191), or has been used alreaday(#1592). It now considers.json to be a text file (#1593),and shows differences for text files in the console (#1593).expect_snapshot_value() can now handle expressions thatgenerate- (#1678) or zero length atomic vectors(#2042).expect_vector() fails, instead of erroring, ifobject is not a vector (JunitReporter() no longer fails with"no applicable method for xml_add_child" for warningsoutside of tests (#1913). Additionally, warnings now save theirbacktraces andJunitReporter() strips ANSI escapes in moreplaces (#1852, #2032).local_edition() now gives a useful error for bad values(#1547).ParallelProgressReporter now respectsmax_failures (#1162), and test files with syntax errors areno longer silently ignored (#1360).test_file(desc = ...) (i.e. test filtering) no longerloses snapshot results (#2066), works withit(), and cannow be character vector in order to recursively filter subtests(#2118).test_that() no longer warns about the absence of{} since it no longer seems to be necessary.set_state_inspector() gainstoleranceargument and ignores minor FP differences by default (skip_on_os() gains an option"emscripten"of theos argument to skip tests on Emscripten (snapshot_accept(test) now works when the test file namecontains. (#1669).snapshot_review() includes a reject button and onlydisplays the file navigation and the skip button if there are multiplefiles to review (#2025). It now passes... on toshiny::runApp() (#1928).Fixed an issue whereexpect_no_error(1) was failing(#2037).
Fixed an issue where callingskip() outside of anactive test could cause an unexpected error (2, #2039).
expect_s7_class() tests if an object is an S7 class(#1580).
expect_no_failure(),expect_no_success() andexpect_snapshot_failure() provide more options for testingexpectations.
testthat now requires waldo 0.6.0 or later to access the latestfeatures (#1955).
expect_condition() and related functions now includetheclass of the expected condition in the failure message,if provided (#1987).
expect_error() and friends now error if you supply... but notpattern (#1932). They no longergive an uninformative error if they fail inside a magrittr pipe(#1994).
expect_no_*() expectations no longer incorrectlyemit a passing test result if they in fact fail (#1997).
expect_setequal() correctly identifies what ismissing where (#1962).
expect_snapshot() now strips line breaks in testdescriptions (test_that() that has an empty description (2,#1980).
expect_true() andexpect_false() givebetter errors ifactual isn’t a vector (#1996).
expect_visible() andexpect_invisible()have clearer failure messages (#1966).
local_reproducible_output() (used intest_that() blocks) now setsLANGUAGE to"C" instead of"en" to disable translations,avoiding warnings on some platforms (#1925).
skip_if_not_installed() generates a clearer messagethat sorts better (
with_mock() andlocal_mock() have beenunconditionally deprecated as they will no longer work in futureversions of R (#1999).
Fix incorrect format string detected by latest R-devel. Fixthanks to Tomas Kalibera.
expect_snapshot() handles unexpected errors likeerrors outside of snapshots, i.e. they terminate the entire test and geta traceback (#1906).
JunitReporter() now uses ensures numeric values aresaved the xml file with. as decimal separator. (
local_mocked_bindings() can now mock any object, notjust functions (#1896).
skip_if_offline() now usescaptive.apple.com by default. This is the hostname thatApple devices use to check that they’re online so it should have ahigher reliability thanr-project.org (
test_file(desc = ) will now finddescribe() tests as well astest_that() tests(#1903).
is_informative_error() and thewrapargument totest_dir() and friends are nowdefunct.
expect_no_error(),expect_no_warning(),expect_no_message(),expect_no_condition(),local_mocked_bindings(), andwith_mocked_bindings() are now stable, notexperimental.
All packages, regardless of whether or not they use rlang 1.0.0,now use the new snapshot display for errors, warnings, and messages(#1856). This no longer shows the class name, instead focussing on adisplay that more closely mimics what you’ll see interactively,including showing the error call.
testthat uses an improved algorithm for finding the srcrefassociated with an expectation/error/warning/skip. It now looks for themost recent call that has known source and is found inside thetest_that() call. This generally gives more specificlocations than the previous approach and gives much better locations ifan error occurs in an exit handler.
Helpers are no longer run twice.
expect_setequal() correctly displays results whenonly one of actual and expected is missing elements (#1835).
expect_snapshot() and friends no longer create atemporary file on every invocation.
expect_snapshot_file() now generates clickable linksto review changes (#1821).
expect_snapshot_value() has an improved error if theobject can’t be safely serialized using the specifiedstyle(#1771).
options(rlang_interactive = TRUE) no longer causesskip_on_cran() to not run on CRAN (#1868).
skip_if_offline() now errors if you don’t have curlinstalled (#1854).
StopReporter gains the ability to suppress praisewhen a test passes.
ProgressReporter now uses is a two characters wideskip column in order to have a consistent width when 10 or more testsare skipped in a single file (
test_file() gains adesc argument whichallows you to run a single test from a file (#1776).
Fix for upcoming R-devel release.
testthat now sets the_R_CHECK_BROWSER_NONINTERACTIVE_ environment variable whenrunning tests. This should ensure that left-overbrowser()statements will trigger an error if encountered while running tests.This functionality is only enabled with R (>= 4.3.0).(#1825)
Newexpect_contains() andexpect_in()that works similarly toexpect_true(all(expected %in% object)) orexpect_true(all(object %in% expected)) but give moreinformative failure messages (#1346).
Newis_snapshot() returnsTRUE if codeis running inside a snapshot test (#1796) andis_checking()returnsTRUE if test is running inside ofR CMD check (#1795)
ProgressReporter only reports the run time of testfiles that take longer than 1s, rather than 0.1s. (#1806) andre-displays all failures at the end of the results. Skips are now onlyshown at the end of reporter summaries, not as tests are run. This makesthem less intrusive in interactive tests while still allowing you toverify that the correct tests are skipped (#1801). When using paralleltests, links to failed tests (#1787) and links to accept/review snapshot(#1802) now work.
set_state_inspector() allows to to register afunction that’s called before and after every test, reporting on anydifferences. This is very useful for detecting if any of your tests havemade changes to global state (like options, env vars, or connections)(#1674). This function was inspired by renv’s testinginfrastructure.
skip_on_cran() no longer skips (errors) when runinteractively.
teardown_env() works in more cases.
testthat no longer truncates tracebacks and uses rlang’s defaulttree display.
expect_snapshot() differences no longer usequotes.
expect_error(),expect_warning(), andexpect_message() now correctly enforce that the conditionis of the expected base class (e.g. error, warning, message) even whentheclass argument is used (#1168).
it() now callslocal_test_context() sothat it behaves more similarly totest_that() (#1731), andis now exported so that you can more easily run BDD tests interactively(#1587)
skip_on_bioc() now uses the documented environmentvariable (IS_BIOC_BUILD_MACHINE) (#1712).
source_file(), which is used by various parts of thehelper and setup/teardown machinery, now reports the file name in thecase of errors (#1704).
test_path() now works when called within helperfiles (#1562).
Newvignette("special-files") describes the variousspecial files that testthat uses (#1638).
with_mocked_bindings() andlocal_mocked_bindings() now also bind in the importsnamespace and can mock S3 methods. These changes make them goodsubstitutes for the deprecated functionswith_mock() andlocal_mock(), so those older functions now recommendswitching to the newer equivalents instead of using the mockr or mockerypackages.
expect_setequal() gives more actionable feedback(#1657).
expect_snapshot() no longer elides new lines whenrun interactively (#1726).
Experimental newwith_mocked_bindings() andlocal_mocked_bindings() (#1739).
sprintf().Deprecation warnings are no longer captured byexpect_warning(code, NA),expect_no_warning(code), orexpect_silent(code). This ensures that they bubble up tothe top level so that you can address them (#1680). If you want toassert that code does not throw a deprecation warning, useexpect_no_condition(code(), class = "lifecycle_warning_deprecation").
New experimentalexpect_no_error(),expect_no_warning(),expect_no_message(), andexpect_no_condition() for asserting the code runs withoutan error, warning, message, or condition (#1679).
Fixed a warning in R >=4.2.0 on Windows that occurred whenusing the C++ testing infrastructure that testthat provides(#1672).
Fixed an issue that could prevent compilation of Catch unit testswith LLVM 15. In the interim, packages needing a local workaround cansetPKG_CPPFLAGS = -DCATCH_CONFIG_CPP11_NO_SHUFFLE in theirsrc/Makevars. (2, #1687)
Improve waycapture_output() handles encoding thanksto suggestion from Kurt Hornik (#1693). This means that snapshots usingUTF-8 encoded text on windows work once again.
local_reproducible_output() will no longer attemptto set the local language whenLANG='C' is set or an Rversion is used that was not compiled with natural language support(NLS), which would previously emit non-test-related warnings duringtesting (@dgkf, #1662;@heavywatal,#1689).
test_check() now suppresses hyperlinks since they’lltake you to the wrong places (#1648).
Newset_max_fails() helper to make it easier to setthe maximum number of failures before stopping the test suite. And theadvice to set to Inf is now clickable (#1628).
You can now configure the behaviour of the implicitdevtools::load_all() call performed bydevtools::test() in your package DESCRIPTION file (#1636).To disable exports of internal functions and of testthat helpers,use:
Config/testthat/load-all: list(export_all = FALSE, helpers = FALSE)Helpers are now attached on the search path by default after callingdevtools::test().
Package that explicitly depend on rlang in their description fileare now opting into a new snapshot display for errors, warnings, andmessages. Previously this only concerned packages that explicitlydepended on rlang >= 1.0.0. This display will eventually become thedefault for all packages.
Changes include:
Condition classes are no longer included in the snapshot bydefault. This is to avoid snapshot noise when upstream code adds orchanges a class. For instance, r-devel has added classes to baseerrors.
Warnings and errors are now printed with rlang, including thecall field. This makes it easy to monitor the fullappearance of warning and error messages as they are displayed tousers.
This change is part of a push towards mentioning the useful contextof an error as part of messages, see the release notes of rlang 1.0.0for more about this.
Test results show hyperlinks to failed expectation when supported(#1544).
testthat now uses brio for all reading and writing (#1120). Thisensures that snapshots always use “” to separate lines (#1516).
expect_snapshot() no longer inadvertently trimstrailing new lines off of errors and messages (#1509).
Ifexpect_snapshot() generates a snapshot withdifferent value but still compares as equal (e.g. because you’ve set anumeric tolerance), the saved values no longer update if anothersnapshot in the same file changes.
expect_snapshot() now only adds a.newfile for the variants that actually changed, not all variants, whileexpect_snapshot_file() with variant with no longerimmediately deletes.new files (#1468).
expect_snapshot_file() gains atransform argument to matchexpect_snapshot()(#1474).compare now defaults toNULL,automatically guessing the comparison type based on theextension.
expect_snapshot_file() now errors if the file beingsnapshot does not exist;SnapshotReporter also now treatsthe file directory as an absolute path (#1476,
Newexpect_snapshot_warning() to matchexpect_snapshot_error() (#1532).
JUnitReporter now includes skip messages/reasons(@rfineman,#1507)
local_reproducible_output() gains alang argument so that you can optionally override thelanguage used to translate error messages (#1483). It also sets theglobal optioncli.num_colors in addition tocrayon.enabled.
test_that() no longer inappropriately skips whencallingexpect_equal() when you’ve temporarily set thelocale to non-UTF-8 (#1285).
skip_if_offline() now automatically callsskip_on_cran() (#1479).
snapshot_accept() andsnapshot_review()now work with exactly the same file specification which can be asnapshot name, a file name, or a directory (#1546). They both workbetter with variants (#1508). Snapshot cleanup also removes all emptydirectories (#1457).
When a snapshot changes the hint also mentions that you can usesnapshot_review() (#1500,
JUnit reporter now includes skip messages/reasons (
Condition expectations likeexpect_error() now matchacross the ancestry of chained errors (#1493). You can disable this bysetting the newinherit argument toFALSE.
Added preliminary support for rlang 1.0 errors. It is disabled bydefault for the time being. To activate it, specifyrlang (>= 1.0.0) in yourDESCRIPTION file(or>= 0.99.0.9001 if you’re using the dev version).
Once activated, snapshots will now use rlang to print error andwarning messages, including theError: andWarning: prefixes. This means thecall fieldof conditions is now displayed in snapshots if present. Parent errormessages are also displayed. Following this change, all snapshotsincluding error and warning messages need to be revalidated.
We will enable the new rlang 1.0 output unconditionally in a futurerelease.
expect_snapshot() gains a new argumentcnd_class to control whether to show the class of errors,warnings, and messages.
The default is currently unchanged so that condition classes keepbeing included in snapshots. However, we plan to change the default toFALSE in an upcoming release to prevent distractingsnapshot diffing as upstream packages add error classes. For instance,the development version of R is currently adding classes to basicerrors, which causes spurious snapshot changes when testing againstR-devel on CI.
If you depend on rlang 1.0 (see above), the default is already set toFALSE.
expect_snapshot() no longer processes rlanginjection operators like!!.
Fixed bug in expectations with long inputs that use:: (#1472).
expect_snapshot() is no longerexperimental.
expect_snapshot() and friends gets an experimentalnewvariant argument which causes the snapshot to be savedin_snaps/{variant}/{test}.md instead of_snaps/{test}.md. This allows you to generate (and compare)unique snapshots for different scenarios like operating system or Rversion (#1143).
expect_snapshot() gains atransformargument, which should be a function that takes a character vector oflines and returns a modified character vector of lines. This makes iteasy to remove sensitive (e.g. API keys) or stochastic (e.g. randomtemporary directory names) from snapshot output (#1345).
expect_snapshot_file() now replaces previous.new snapshot if code fails again with a differentvalue.
expect_snapshot_value() now has an explicittolerance argument which uses the testthat default, thusmaking it more likeexpect_equal() rather thanexpect_identical(). Set it toNULL if you wantprecise comparisons (#1309).expect_snapshot_value(style = "deparse") now works withnegative values (#1342).
If a test containing multiple snapshots fails (or skips) inbetween snapshots, the later snapshots are now silently restored.(Previously this warned and reset all snapshots, not just latersnapshots).
If you have multiple tests with the same name that use snapshots(not a good idea), you will no longer get a warning. Instead thesnapshots will be aggregated across the tests.
Condition expectations now consistently return the expectedcondition instead of the return value (#1371). Previously, they wouldonly return the condition if the return value wasNULL,leading to inconsistent behaviour.
This is a breaking change to the 3rd edition. Where you couldpreviously do:
expect_equal(expect_warning(f(), "warning"), "value")You must now use condition expectations on the outside:
expect_warning(expect_equal(f(), "value"), "warning")# Equivalently, save the value before inspectionexpect_warning(value <- f(), "warning")expect_equal(value, "value")This breaking change makes testthat more consistent. It also makes itpossible to inspect both the value and the warning, which wouldotherwise require additional tools.
Errors in test blocks now display the call if stored in thecondition object (#1418). Uncaught errors now show their class(#1426).
Multi-line skips only show the first line in the skipsummary.
expr_label(), which is used to concisely describeexpressions used in expectations, now does a better job of summarisinginfix function (#1442).
local_reproducible_output() now sets themax.print option to 99999 (the default), so your tests areunaffected by any changes you might’ve made in your.Rprofile (1367).
ProgressReporter (the default only) now stops at theend of a file; this ensures that you see the results of all relatedtests, and ensures that snapshots are handled consistently(#1402).
ProgressReporter now uses an env var to adjust themaximum number of failures. This makes it easier to adjust when thetests are run in a subprocess, as is common when using RStudio(#1450).
skip_on_os() gains anarch argument soyou can also choose to skip selected architectures (#1421).
test_that() now correctly errors when an expectationfails when run interactively (#1430).
test_that() now automatically and correctly generatean “empty test” skip if it only generates warnings or messages (anddoesn’t contain any expectations).
testthat_tolerance() no longer has an unusedargument.
The vendored Catch code used foruse_catch() nowuses a constant value for the stack size rather than relying onSIGSTKSZ. This fixes compatibility for recent glibc versions whereSIGSTKSZ is no longer a constant.
Fixed an issue that caused errors and early termination of testson R <= 3.6 when a failing condition expectation was signalled insidea snapshot.
expect_snapshot_file() gains acompareargument (#1378,
The functionscompare_file_binary() andcompare_file_text() are now exported from testthat to besupplied ascompare argument. These implement the samebehaviour as the oldbinary argument which is nowdeprecated.
expect_snapshot() no longer deletes snapshots whenan unexpected error occurs.
Newannounce_snapshot_file() function for developersof testthat extensions. Announcing a snapshot file allows testthat topreserve files that were not generated because of an unexpected error oraskip() (#1393). Unannounced files are automaticallydeleted during cleanup if the generating code isn’t called.
New expectation:expect_no_match(). It complementsexpect_match() by checking if a stringdoesn’tmatch a regular expression (
Support setting the testthat edition via an environment variable(TESTTHAT_EDITION) as well (
Failing expectations now include a backtrace when they’re notcalled directly from withintest_that() but are insteadwrapped in some helper function (#1307).
CheckReporter now only records warnings when not onCRAN. Otherwise failed CRAN revdep checks tend to be cluttered up withwarnings (#1300). It automatically cleans uptestthat-problems.rds left over from previous runs if thelatest run is successful (#1314).
expect_s3_class() andexpect_s4_class()can now check that an objectisn’t an S3 or S4 object bysupplyingNA to the second argument (#1321).
expect_s3_class() andexpect_s4_class()format class names in a less confusing way (#1322).
expect_snapshot() collapses multiple adjacentheadings of the same, so that, e.g., if you have multiple lines of codein a row, you’ll only see one “Code:” heading (#1311).
Newtestthat.progress.verbose_skips option. Set toFALSE to stop reporting skips as they occur; they willstill appear in the summary (#1209,
CheckReporter results have been tweaked based onexperiences from running R CMD check on many packages. Hopefully itshould now be easier to see the biggest problems (i.e. failures anderrors) while still having skips and warnings available to check ifneeded (#1274). And now the full test name is always shown, no matterhow long (#1268).
Catch C++ tests are no longer reported multiple times (#1237) andare automatically skipped on Solaris since Catch is not supported(#1257).use_catch() makes it more clear that your packageneeds to suggest xml2 (#1235).
auto_test_package() works once again (
expect_snapshot() gains newerrorargument which controls whether or not an error is expected. If anunexpected error is thrown, or an expected error is not thrown,expect_snapshot() will fail (even on CRAN)(#1200).
expect_snapshot_value(style = "deparse") handlesmore common R data structures.
expect_snapshot_value() now passes...on towaldo::compare() (#1222).
expect_snapshot_file() gives a hint as to next stepswhen a failure occurs in non-interactive environments (with help from@maelle, #1179).expect_snapshot_*() gives a more informative hint whenyou’re running tests interactively (#1226).
expect_snapshot_*() automatically removes the_snaps directory if it’s empty (#1180). It also warns ifsnapshots are discarded because tests have duplicated names (#1278,@krlmlr).
local_reproducible_output() now sets the LANGUAGEenv var to “en”. This matches the behaviour of R CMD check ininteractive settings (#1213). It also now unsets RSTUDIO envvar, insteadof setting it to 0 (#1225).
RstudioReporter has been renamed toRStudioReporter.
skip_if_not() no longer appends “is not TRUE” tocustom messages (
test_that() now warns (3e only) if code doesn’t havebraces, since that makes it hard to track the source of an error (#1280,@krlmlr).
testhat 3.0.0 brings with it a 3rd edition that makes a number ofbreaking changes in order to clean up the interface and help you use ourlatest recommendations. To opt-in to the 3rd edition for your package,setConfig/testthat/edition: 3 in yourDESCRIPTION or uselocal_edition(3) inindividual tests. You can retrieve the active edition withedition_get(). Learn more invignette("third-edition").
context() is deprecated.
expect_identical() andexpect_equal()usewaldo::compare() to compare actual and expectedresults. This mostly yields much more informative output when the actualand expected values are different, but while writing it uncovered somebugs in the existing comparison code.
expect_error(),expect_warning(),expect_message(), andexpect_condition() nowall use the same underlying logic: they capture the first condition thatmatchesclass/regexp and allow anything elseto bubble up (#998/#1052). They also warn if there are unexpectedarguments that are never used.
Theall argument toexpect_message()andexpect_warning() is now deprecated. It was never aparticularly good idea or well documented, and is now superseded by thenew condition capturing behaviour.
expect_equivalent(),expect_reference(),expect_is() andexpect_that() are deprecated.
Messages are no longer automatically silenced. Either usesuppressMessages() to hide unimportant messages, orexpect_message() to catch important messages(#1095).
setup() andteardown() are deprecatedin favour of test fixtures. Seevignette("test-fixtures")for more details.
expect_known_output(),expect_known_value(),expect_known_hash(), andexpect_equal_to_reference() are all deprecated in favour ofexpect_snapshot_output() andexpect_snapshot_value().
test_that() now sets a number of options and envvars to make output as reproducible as possible (#1044). Many of theseoptions were previously set in various places (indevtools::test(),test_dir(),test_file(), orverify_output()) but they havenow been centralised. You can use in your own code, or when debuggingtests interactively withlocal_test_context().
with_mock() andlocal_mock() aredeprecated; please use the mockr or mockery packages instead(#1099).
New family of snapshot expectations (expect_snapshot(),expect_snapshot_output(),expect_snapshot_error(), andexpect_snapshot_value()) provide “snapshot” tests, wherethe expected results are stored in separate files intest/testthat/_snaps. They’re useful whenever it’s painfulto store expected results directly in the test files.
expect_snapshot_file() along withsnapshot_review() help snapshot more complex data, withinitial support for text files, images, and data frames (#1050).
Seevignette("snapshotting") for more details.
CheckReporter (used inside R CMD check) now printsout all problems (i.e. errors, failures, warnings and skips; and notjust the first 10), lists skips types, and records problems in machinereadable format intests/testthat-problems.rds(#1075).
NewCompactProgressReporter tweaks the output ofProgressReporter for use with a single file, as indevtools::test_file(). You can pick a different default bysettingtestthat.default_compact_reporter to the name of areporter.
ProgressReporter (the default reporter) now keepsthe stack traces of an errors that happen before the before test, makingproblems substantially easier to track down (#1004). It checks if you’veexceeded the maximum number of failures (from optiontestthat.progress.max_fails) after each expectation, ratherthan at the end of each file (#967). It also gains new random praiseoptions that use emoji, and lists skipped tests by type(#1028).
StopReporter adds random praise emoji when a singletest passes (#1094). It has more refined display of failures, now usingthe same style asCompactProgressReporter andProgressReporter.
SummaryReporter now records file start, not justcontext start. This makes it more compatible with modern style whichdoes not usecontext() (#1089).
All reporters now use exactly the same format when reporting thelocation of an expectation.
Warnings now include a backtrace, making it easier to figure outwhere they came from.
Catch C++ tests now provide detailed results for each test. Toupgrade existing code, re-runtestthat::use_catch()(#1008).
Many reporters (e.g. the check reporter) no longer raise an errorwhen any tests fail. Use thestop_on_failure argument todevtools::test() andtestthat::test_dir() ifyour code relies on this. Alternatively, usereporter = c("check", "fail") to e.g. create a failingcheck reporter.
Newvignette("test-fixtures") describes testfixtures; i.e. how to temporarily and cleanly change global state inorder to test parts of your code that otherwise would be hard to run(#1042).setup() andteardown() are supersededin favour of test fixtures.
Newteardown_env() for use withwithr::defer(). This allows you to run code after all othertests have been run.
Newvignette("skipping") gives more generalinformation on skipping tests, include some basics on testing skippinghelpers (#1060).
ProgressReporter() andCheckReporter()list the number of skipped tests by reason at the end of the reporter.This makes it easier to check that you’re not skipping the wrong tests,particularly on CI services (#1028).
test_that() no longer triggers an error when runoutside of tests; instead it produces a more informative summary of allfailures, errors, warnings, and skips that occurred inside thetest.
test_that() now errors ifdesc is not astring (#1161).
test_file() now runs helper, setup, and teardowncode, and has the same arguments astest_dir() (#968). Longdeprecatedencoding argument has been removed.
test_dir() now defaultsstop_on_failuretoTRUE for consistency with othertest_functions. Thewrap argument has been deprecated; it’s notclear that it should ever have been exposed.
Newtest_local() tests a local source packagedirectory. It’s equivalent todevtools::test() but doesn’trequire devtools and all its dependencies to be installed(#1030).
testthat no longer supports tests stored ininst/tests. This has been deprecated since testthat 0.11.0(released in 2015).test_package() (previously used forrunning tests in R CMD check) will fail silently if no tests are foundto avoid breaking old packages on CRAN (#1149).
capture_output() andverify_output()use a newtestthat_print() generic. This allows you tocontrol the printed representation of your object specifically for tests(i.e. if your usual print method shows data that varies in a way thatyou don’t care about for tests) (#1056).
context_start_file() is now exported for externalreporters (#983, #1082). It now only strips first instance ofprefix/suffix (#1041,
expect_error() no longer encourages you to useclass. This advice removes one type of fragility at theexpense of creating a different type (#1013).
expect_known_failure() has been removed. As far as Ican tell it was only ever used by testthat, and is ratherfragile.
expect_true(),expect_false(), andexpect_null() now use waldo to produce more informativefailures.
verify_output() no longer always fails if outputcontains a carriage return character (“) (#1048). It uses thepdf() device instead ofpng() so it works onsystems without X11 (#1011). And it useswaldo::compare()to give more informative failures.
The last version of testthat introduced a performance regressionin error assertions (#963). To fix it, you need to install rlang0.4.2.
Fixed error assertions with rJava errors (#964).
Fixed issue where error and warning messages were not retrievedwithconditionMessage() under certaincircumstances.
This release mostly focusses on an overhaul of how testthat workswith conditions (i.e. errors, warnings and messages). There arerelatively few user-facing changes, although you should now see moreinformative backtraces from errors and failures.
Unexpected errors are now printed with a simplifiedbacktrace.
expect_error() andexpect_condition()now display a backtrace when the error doesn’t conform to expectations(#729).
expect_error(),expect_warning() andexpect_message() now callconditionMessage()to get the condition message. This generic makes it possible to generatemessages at print-time rather than signal-time.
expect_error() gets a better warning message whenyou test for a custom error class withregexp.
Newexp_signal() function is a condition signallerthat implements the testthat protocol (signal withstop()if the expectation is broken, with amuffle_expectationrestart).
Existence of restarts is first checked before invocation. Thismakes it possible to signal warnings or messages with a differentcondition signaller (#874).
ListReporter now tracks expectations and errors,even when they occur outside of tests. This ensures thatstop_on_failure matches the results displayed by thereporter (#936).
You can silence warnings about untested error classes byimplementing a method foris_uninformative_warning(). Thismethod should be lazily registered, e.g. withvctrs::s3_register(). This is useful for introducing anexperimental error class without encouraging users to depend on theclass in their tests.
Respect options(warn = -1) to ignore all warnings (
Expectations can now be explicitly subclassed withnew_expectation(). This constructor follows our newconventions for S3 classes and takes an optional subclass and optionalattributes.
Unquoted inputs no longer potentially generate multiple testmessages (#929).
verify_output() no longer uses quasiquotation, whichfixes issues when verifying the output of tidy eval functions(#945).
verify_output() gains aunicodeparameter to turn on or off the use of Unicode characters by the clipackage. It is disabled by default to prevent the tests from failing onplatforms like Windows that don’t support UTF-8 (which could be yourcontributors’ or your CI machines).
verify_output() now correctly handles multi-linecondition messages.
verify_output() now adds spacing after conditionmessages, consistent with the spacing added after normaloutput.
verify_output() has a new syntax for insertingheaders in output files: insert a"# Header" string(starting with# as in Markdown) to add a header to a setof outputs.
compare.numeric() uses a more sophisticated defaulttolerance that will automatically skip tests that rely on numerictolerance if long doubles are not available (#940).
JunitReporter now reports tests in ISO 8601 in theUTC timezone and uses the maximum precision of 3 decimal places(#923).
test_rd() and add a couple oftests to hopefully detect the problem earlier in the future.verify_output() is designed for testing outputaimed at humans (most commonly print methods and error messages). It isa regression test that saves output in a way that makes it easy toreview. It is automatically skipped on CRAN (#782, #834).as.data.frame.testthat_results() now always returnsa data frame with 13 columns (
auto_test_package() now correctly handles helperfiles (tests/testthat/helper-*.R), automatically reloadingall code and rerunning all tests (
expect_match() now displaysinfo evenwhen match length is 0 (#867).
expect_s3_class() gains newexactargument that allows you to check for an exact class match, not justinheritance (#885).
fail() andsucceed() gaininfo argument, which is passed along toexpect().
test_examples() gets some minor fixes: it nowreturns the results invisibly, doesn’t assume that examples shouldcontain tests, and documents that you shouldn’t be using it routinely(#841).
test_file() only callsReporter$end_context() if a context was started, fixing anerror inTeamcityReporter (
skip() now reports reason for skipping as:Reason: {skip condition} (
skip_if() andskip_if_not() now reportReason: {skip condition} is TRUE andReason: {skip condition} is not TRUE respectively (@patr1ckm, #868).
skip_if_translated() now tests for translation of aspecific message. This is more robust than the previous approach becausetranslation happens message-by-message, not necessarily for the entiresession (#879) (and in general, it’s impossible to determine whatlanguage R is currently using).
skip_on_covr() allows you to skip tests when covr isrunning. (@ianmcook,#895)
expect_known_value() gains a new serialisationversion argument, defaulting to 2. Prevents the.rds files created to hold reference objects from making apackage appear to require R >= 3.5 (#888
Newexpect_visible() andexpect_invisible() make it easier to check if a functioncall returns its result visibly or invisibly (#719).
Newexpect_mapequal(x, y) checks thatxandy have the same names, and the same value associatedwith each name (i.e. they compare the values of the vector standardisingthe order of the names) (#863).
Newexpect_vector() is a wrapper aroundvctrs::vec_assert() making it easy to test against thevctrs definitions of prototype and size (#846). (Currently requiresdevelopment version of vctrs.)
All expectations give clearer error messages if you forget theobject orexpected arguments (#743).
expect_equal() now correctly compares infinitevalues (#789).
Inexpect_equal_to_reference(), the default valueforupdate is nowFALSE (
expect_error() now returns the error object asdocumented (#724). It also now warns if you’re using a classedexpectation and you’re not using theclass argument. Thisis good practice as it decouples the error object (which tends to bestable) from its rendering to the user (which tends to be fragile)(#816).
expect_identical() gains a... argumentto pass additional arguments down toidentical()(#714).
expect_lt(),expect_lte(),expect_gt()expect_gte() now handleInf andNA arguments appropriately (#732), andno longer require the inputs to be numeric.
expect_output() gains awidth argument,allowing you to control the output width. This does not inherit fromgetOption("width"), ensuring that tests return the sameresults regardless of environment (#805).
expect_setequal() now works with more vector types(including lists), because it uses%in%, rather thansort(). It also warns if the inputs are named, as thissuggests that your mental model of howexpect_setequal()works is wrong (#750).
is_true() andis_false() have beendeprecated because they conflict with other functions in thetidyverse.
Reporter documentation has been considerably improved(#657).
CheckReporter, used by R CMD check, now includes acount of warnings.
JUnitReporter no longer replaces. inclass names (#753), and creates output that should be more compatiblewith Jenkins (#806,
ListReporter now records number of passed tests andoriginal results in new columns (#675).
ProgressReporter, the default reporter, now:
Automatically generates a context from the file name. We nolonger recommend the use ofcontext() and instead encourageyou to delete it, allowing the context to be autogenerated from the filename.
This also eliminates the error that occurred if tests can before thefirstcontext() (#700, #705).
Gains aupdate_interval parameter to control howoften updates are printed (default 0.1 s). This prevents large printingoverhead for very fast tests. (#701,
Uses a 3 character wide column to display test successes, so upto 999 successful tests can be displayed without changing the alignment(#712).
reporter$end_reporter() is now only called whentesting completes successfully. This ensures that you don’t getunnecessary output when the test fails partway through (#727).
skip_if_offline() skips tests if an internetconnection is not available (#685).
skip_on_ci() skips tests on continuous integrationsystems (@mbjoseph,#825) by looking for aCI env var..
Newtestthat_examples() andtestthat_example() make it easy to access new test filesbundled with the package. These are used in various examples to make iteasier to understand how to use the package.
Newlocal_mock() which allows you to mock a functionwithout having to add an additional layer of indentation as withwith_mock() (#856).
auto_test_package() works better with recentdevtools and also watchessrc/ for changes (#809).
expect_s3_class() now works with unquoting (
expectation objects now contain the failure message,even when successful (#836)
devtools::test() no longer fails if run multipletimes within the same R session for a package containing Catch tests.(devtools#1832)
Newtesting_package() retrieves the name of thepackage currently being tested (#699).
run_testthat_tests C entrypoint is registered morerobustly.
skip() now always produces amessage oflength 1, as expected elsewhere in testthat (#791).
Warnings are passed through even whenoptions(warn = 2) is set (
“Can’t mock functions in base packages”: You can no longer usewith_mock() to mock functions in base packages, becausethis no longer works in R-devel due to changes with the byte codecompiler. I recommend usingmockery ormockr instead.
The order of arguments toexpect_equivalent() andexpect_error() has changed slightly as both now pass... on another function. This reveals itself with a numberof different errors, like:
If you see one of these errors, check the number, order, and names ofarguments to the expectation.
“Failure: (unknown)”. The last release mistakenly failed to testbare expectations not wrapped insidetest_that(). If yousee “(unknown)” in a failure message, this is a failing expectation thatyou previously weren’t seeing. As well as fixing the failure, pleasealso wrap inside atest_that() with an informativename.
“Error: the argument has already been evaluated”: the way inwhich expectations now need create labels has changed, which caused acouple of failures with unusual usage when combined withReduce,lapply(), andMap().Avoid these functions in favour of for loops. I also recommend readingthe section below on quasiquotation support in order to create moreinformative failure messages.
expect_condition() works likeexpect_error() but captures any condition, not just errorconditions (#621).
expect_error() gains aclass argumentthat allows you to make an assertion about the class of the error object(#530).
expect_reference() checks if two names point to thesame object (#622).
expect_setequal() compares two sets (stored invectors), ignoring duplicates and differences in order (#528).
skip_if() makes it easy to skip a test when acondition is true (#571). For example, useskip_if(getRversion() <= 3.1) to skip a test in older Rversions.
skip_if_translated() skips tests if you’re runningin an locale where translations are likely to occur (#565). Use this toavoid spurious failures when checking the text of error messages innon-English locales.
skip_if_not_installed() gains newminimum_version argument (#487, #499).
We have identified a useful family of expectations that compares theresults of an expression to a known good value stored in a file. Theyare designed to be use in conjunction with git so that you can see whatprecisely has changed, and revert it if needed.
expect_known_output() replacesexpect_output_file(), which has been soft-deprecated. Itnow defaults toupdate = TRUE and warn, rather than failingon the first run. It gains aprint argument toautomatically print the input (#627). It also sets the width option to80 to ensure consistent output across environments (#514)
expect_known_value() replacesexpect_equal_to_reference(), which has beensoft-deprecated. It gains an update argument defaulting toTRUE. This changes behaviour from the previous version, andsoft-deprecatedexpect_equal_to_reference() getsupdate = FALSE.
expect_known_failure() stored and compares thefailure message from an expectation. It’s a useful regression test whendeveloping informative failure messages for your ownexpectations.
All expectations can now use unquoting (#626). This makes it mucheasier to generate informative failure messages when running tests in afor loop.
For example take this test:
f<-function(i)if (i>3) i*9else i*10for (iin1:5) {expect_equal(f(i), i*10)}When it fails, you’ll see the messageError: `f(i)` not equal to `i * 10`. That’s hard todiagnose because you don’t know which iteration caused the problem!
for (iin1:5) {expect_equal(f(!!i),!!(i*10))}If you unquote the values using!!, you get the failuremessage`f(4L)` not equal to 40.. This is much easier todiagnose! See?quasi_label() for more details.
(Note that this is not tidy evaluation per se, but is closelyrelated. At this time you can not unquote quosures.)
Newsetup() andteardown() functionsallow you to run at the start and end of each test file. This is usefulif you want to pair cleanup code with the code that messes up state(#536).
Two new prefixes are recognised in thetest/directory. Files starting withsetup are run before tests(but unlikehelpers are not run indevtools::load_all()). Files starting withteardown are run after all tests are completed(#589).
All files are now read and written as UTF-8 (#510,#605).
is_testing() allows you to tell if your code isbeing run inside a testing environment (#631). Rather than taking arun-time dependency on testthat you may want to inline the function intoyour own package:
is_testing<-function() {identical(Sys.getenv("TESTTHAT"),"true")}It’s frequently useful to combine withinteractive().
A new default reporter,ReporterProgress, produces moreaesthetically pleasing output and makes the most important informationavailable upfront (#529). You can return to the previous default bysettingoptions(testthat.default_reporter = "summary").
Output colours have been tweaked to be consistent with clang:warnings are now in magenta, and skips in blue.
Newdefault_reporter() andcheck_reporter() which returns the default reporters forinteractive and check environments (#504).
NewDebugReporter that calls a better version ofrecover() in case of failures, errors, or warnings (#360,#470).
NewJunitReporter generates reports in JUnitcompatible format. (#481,
NewLocationReporter which just prints the locationof every expectation. This is useful for locating segfaults and C/C++breakpoints (#551).
SummaryReporter received a number of smallertweaks
Aborts testing as soon the limit given by the optiontestthat.summary.max_reports (default 10) is reached(#520).
New optiontestthat.summary.omit_dots = TRUE hidesthe progress dots speeding up tests by a small amount (#502).
Bring back random praise and encouragement which I accidentallydropped (#478).
New optiontestthat.default_check_reporter, defaultsto"check". Continuous Integration system can set thisoption before evaluating package test sources in order to direct testresult details to known location.
All reporters now accept afile argument oninitialization. If provided, reporters will write the test results tothat path. This output destination can also be controlled with theoptiontestthat.output_file (#635,
is_null() andmatches() have beendeprecated because they conflict with other functions in the tidyverse(#523).Updated Catch to 1.9.6.testthat now understands andmakes use of the package routine registration mechanism required by CRANwith R >= 3.4.0.
Better reporting for deeply nested failures, limiting the stacktrace to the first and last 10 entries (#474).
Bare expectations notify the reporter once again. This isachieved by running all tests insidetest_code() by default(#427, #498). This behaviour can be overridden by settingwrap = FALSE intest_dir() and friends(#586).
auto_test() andauto_test_package()providehash parameter to enable switching to faster,time-stamp-based modification detection (#598,auto_test_package() works correctly on windows(#465).
capture_output_lines() is now exported(#504).
compare.character() works correctly for vectors oflength > 5 (#513,
compare.default() gains amax_diffsargument and defaults to printing out only the first 9 differences(#538).
compare.numeric() respectscheck.attributes() soexpect_equivalent()correctly ignores attributes of numeric vectors (#485).
Output expectations (expect_output(),expect_message(),expect_warning(), andexpect_silent()) all invisibly return the first argument tobe consistent with the other expectations (#615).
expect_length() works with any object that has alength method, not just vectors (#564,
expect_match() now accepts explicitperl andfixed arguments, and adapts thefailure message to the value offixed. This also affectsother expectations that forward toexpect_match(), likeexpect_output(),expect_message(),expect_warning(), andexpect_error().
expect_match() escapes special regular expressioncharacters when printing (#522,
expect_message(),expect_warning() andexpect_error() produce clearer failure messages.
find_test_scripts() only looks for\.[rR] in the extension (#492,
test_dir(),test_package(),test_check() unset theR_TESTS env var(#603)
test_examples() now works with installed packages aswell as source packages (
test_dir(),test_package(), andtest_check() gainstop_on_failure andstop_on_waring arguments that control whether or not anerror is signalled if any tests fail or generate warnings (#609,#619).
test_file() now triggers agc() aftertests are run. This helps to ensure that finalisers are run earlier(#535).
test_path() now generates correct path when calledfrom withintools::testInstalledPackage() (#542).
test_path() no longer assumes that the path exists(#448).
test_that() calls without any expectations generatea defaultskip() (#413).
test_dir() gainsload_helpers argument(#505).
show_failures() simply prints a failure if itoccurs. This makes it easier to show failures in examples.
with_mock() disallows mocking of functions in basepackages, because this doesn’t work with the current development versionof R (#553).
std::logic_error() constructed withstd::string() argument, to avoid build errors onSolaris.Newexpect_output_file() to compare output of afunction with a text file, and optionally update it (#443,
Properly scoped use + compilation of C++ unit testing code usingCatch togcc andclang only, as Catch includescode that does not strictly conform to the C++98 standard. (2)
Fixed an out-of-bounds memory access when routing Catch outputthroughRprintf(). (2)
Ensure that unit tests run on R-oldrel (remove use ofdir.exists()).
Improved overriding of calls toexit() within Catch,to ensure compatibility with GCC 6.0. (
Hardened formatting of difference messages, previously thepresence of% characters could affect the output (#446,@krlmlr).
Fixed errors inexpect_equal() when comparingnumeric vectors with and without attributes (#453,
auto_test() andauto_test_package()show only the results of the current test run and not of previouslyfailed runs (#456,
Theexpectation() function now expects an expectationtype (one of “success”, “failure”, “error”, “skip”, “warning”) as firstargument. If you’re creating your own expectations, you’ll need to useexpect() instead (#437).
The expectation system got a thorough overhaul (#217). This primarilymakes it easier to add new expectations in the future, but also includeda thorough review of the documentation, ensuring that relatedexpectations are documented together, and have evocative names.
One useful change is that most expectations invisibly return theinputobject. This makes it possible to chain togetherexpectations with magrittr:
factor("a")%>%expect_type("integer")%>%expect_s3_class("factor")%>%expect_length(1)(And to make this style even easier, testthat now re-exports thepipe, #412).
The exception to this rule are the expectations that evaluate (i.e.for messages, warnings, errors, output etc), which invisibly returnNULL. These functions are now more consistent: usingNA will cause a failure if there is aerrors/warnings/messages/output (i.e. they’re not missing), and willNULL fail if there aren’t anyerrors/warnings/messages/output. This previously didn’t work forexpect_output() (#323), and the error messages wereconfusing withexpect_error(..., NA) (#342,
Another change is thatexpect_output() now requires youto explicitly print the output if you want to test a print method:expect_output("a", "a") will fail,expect_output(print("a"), "a") will succeed.
There are six new expectations:
expect_type() checks thetype of the object(#316),expect_s3_class() tests that an object is S3 withgiven class,expect_s4_class() tests that an object is S4with given class (#373). I recommend using these more specificexpectations instead of the more generalexpect_is().
expect_length() checks that an object has expectedlength.
expect_success() andexpect_failure()are new expectations designed specifically for testing otherexpectations (#368).
A number of older features have been deprecated:
expect_more_than() andexpect_less_than() have been deprecated. Please useexpect_gt() andexpect_lt() instead.
takes_less_than() has been deprecated.
not() has been deprecated. Please use the explicitindividual formsexpect_error(..., NA) ,expect_warning(.., NA) and so on.
Now all expectations are also conditions, and R’s condition system isused to signal failures and successes (#360,as.expectation(). This allowsthird-party test packages (such asassertthat,testit,ensurer,checkmate,assertive) to seamlessly establishtestthatcompatibility by issuing custom error conditions (e.g.,structure(list(message = "Error message"), class = c("customError", "error", "condition")))and then implementingas.expectation.customError(). Theassertthat package contains an example.
The reporters system class has been considerably refactored to makeexisting reporters simpler and to make it easier to write new reporters.There are two main changes:
Reporters classes are now R6 classes instead of ReferenceClasses.
Each callbacks receive the full context:
add_results() is passed context and test as well as theexpectation.test_start() andtest_end() both get thecontext and test.context_start() andcontext_end() get thecontext.Warnings are now captured and reported in mostreporters.
The reporter output goes to the original standard output and isnot affected bysink() andexpect_output()(#420,
The default summary reporter lists all warnings (#310), and allskipped tests (testthat.summary.max_reports limits the number of reportsprinted by the summary reporter. The default is 15 (
MinimalReporter correct labels errors with E andfailures with F (#311).
NewFailReporter to stop in case of failures orerrors after all tests (#308,
New functionscapture_output(),capture_message(), andcapture_warnings()selectively capture function output. These are used inexpect_output(),expect_message() andexpect_warning() to allow other types out output topercolate up (#410).
try_again() allows you to retry code multiple timesuntil it succeeds (#240).
test_file(),test_check(), andtest_package() now attach testthat so all testing functionsare available.
source_test_helpers() gets a useful default path:the testthat tests directory. It defaults to thetest_env()to be consistent with the other source functions (#415).
test_file() now loads helpers in the test directorybefore running the tests (#350).
test_path() makes it possible to create paths tofiles intests/testthat that work interactively and whencalled from tests (#345).
Addskip_if_not() helper.
Addskip_on_bioc() helper (
make_expectation() usesexpect_equal().
setup_test_dir() has been removed. If you used itpreviously, instead usesource_test_helpers() andfind_test_scripts().
source_file() exports the function testthat uses toload files from disk.
test_that() returns alogical thatindicates if all tests were successful (#360,
find_reporter() (and also all high-level testingfunctions) support a vector of reporters. For more than one reporter, aMultiReporter is created (#307,
with_reporter() is used internally and gains newargumentstart_end_reporter = TRUE (
set_reporter() returns old reporter invisibly (#358,@krlmlr).
Comparing integers to non-numbers doesn’t raise errors anymore,and falls back to string comparison if objects have different lengths.Complex numbers are compared using the same routine (#309,
compare.numeric() andcompare.character() received another overhaul. This shouldimprove behaviour of edge cases, and provides a strong foundation forfurther work. Addedcompare.POSIXt() for better reportingof datetime differences.
expect_identical() andis_identical_to() now usecompare() for moredetailed output of differences (#319,
AddedCatchv1.2.1 for unit testing of C++ code. See?use_catch() formore details. (2)
Handle skipped tests in the TAP reporter (#262).
Newexpect_silent() ensures that code produces nooutput, messages, or warnings (#261).
Newexpect_lt(),expect_lte(),expect_gt() andexpect_gte() for comparisonwith or without equality (#305,
expect_output(),expect_message(),expect_warning(), andexpect_error() nowacceptNA as the second argument to indicate that output,messages, warnings, and errors should be absent (#219).
Praise gets more diverse thanks to the praise package, and you’llnow get random encouragement if your tests don’t pass.
testthat no longer muffles warning messages. If you don’t want tosee them in your output, you need to explicitly quiet them, or use anexpectation that captures them (e.g. expect_warning()).(#254)
Use tests ininst/tests is formally deprecated.Please move them intotests/testthat instead(#231).
expect_match() now encodes the match, as well as theoutput, in the expectation message (#232).
expect_is() gives better failure message whentesting multiple inheritance,e.g. expect_is(1:10, c("glm", "lm")) (#293).
Corrected argument order incompare.numeric()(#294).
comparison() constructure now checks its argumentsare the correct type and length. This bugs a bug where tests failed withan error like “values must be length 1, but FUN(X[[1]]) result is length2” (#279).
Addedskip_on_os(), to skip tests on specifiedoperating systems (2).
Skip test that depends ondevtools if it is notinstalled (#247,
Addedskip_on_appveyor() to skip tests on Appveyor(@lmullen).
compare() shows detailed output of differences forcharacter vectors of different length (#274,
Detailed output fromexpect_equal() doesn’t confuseexpected and actual values anymore (#274,
Failure locations are now formatted as R errorlocations.
Add an ‘invert’ argument tofind_tests_scripts().This allows one to select only tests which donot match apattern. (#239,
Deprecatedlibrary_if_available() has beenremoved.
test (test_dir(),test_file(),test_package(),test_check()) functions nowreturn atestthat_results object that contains all results,and can be printed or converted to data frame.
test_dir(),test_package(), andtest_check() have an added... argument thatallows filtering of test files using, e.g., Perl-style regularexpressions,orfixed character filtering. Arguments in... are passed togrepl() (
test_check() uses a new reporter specificallydesigned forR CMD check. It displays a summary at the endof the tests, designed to be <13 lines long so test failures inR CMD check display something more useful. This willhopefully stop BDR from calling testthat a “test obfuscation suite”(#201).
compare() is now documented and exported. Added anumeric method so when long numeric vectors don’t match you’ll see someexamples of where the problem is (#177). The line spacing incompare.character() was tweaked.
skip_if_not_installed() skips tests if a packageisn’t installed (#192).
expect_that(a, equals(b)) style of testing has beensoft-deprecated. It will keep working, but it’s no longer demonstratedany where, and new expectations will only be available inexpect_equal(a, b) style. (#172)
Once again, testthat suppresses messages and warnings in tests(#189)
Newtest_examples() lets you run package examples astests. Each example counts as one expectation and it succeeds if thecode runs without errors (#204).
Newsucceed() expectation always succeeds.
skip_on_travis() allows you to skip tests when runon Travis CI. (Thanks to
colourise() was removed. (Colour is still supported,via thecrayon package.)
Mocks can now access values local to the call ofwith_mock (#193,
All equality expectations are now documented together (#173); allmatching expectations are also documented together.
BDD: testhat now comes with an initial behaviour drivendevelopment (BDD) interface. The language is similar to RSpec for Rubyor Mocha for JavaScript. BDD tests read like sentences, so they shouldmake it easier to understand the specification of a function. See?describe() for further information and examples.
It’s now possible toskip() a test with aninformative message - this is useful when tests are only available undercertain conditions, as when not on CRAN, or when an internet connectionis available (#141).
skip_on_cran() allows you to skip tests when run onCRAN. To take advantage of this code, you’ll need either to usedevtools, or runSys.setenv(NOT_CRAN = "true"))
Simple mocking:with_mock() makes it easy totemporarily replace functions defined in packages. This is useful fortesting code that relies on functions that are slow, have unintendedside effects or access resources that may not be available when testing(#159,
A new expectation,expect_equal_to_reference() hasbeen added. It tests for equality to a reference value stored in a file(#148,
auto_test_package() works once more, and now usesdevtools::load_all() for higher fidelity loading (#138,#151).
Bug incompare.character() fixed, as reported byGeorgi Boshnakov.
colourise() now uses optiontestthat.use_colours (default:TRUE). If it isFALSE, output is not colourised (#153,
is_identical_to() only callsall.equal() to generate an informative error message if thetwo objects are not identical (#165).
safe_digest() uses a better strategy, and returns NAfor directories (#138, #146).
Random praise is re-enabled by default (again!) (#164).
Teamcity reporter now correctly escapes output messages (#150,@windelinckx). Italso uses nested suites to include test names.
library_if_available() has been deprecated.Better default environment fortest_check() andtest_package() which allows S4 class creation intests
compare.character() no longer fails when one valueis missing.
testthat 0.8 comes with a new recommended structure for storing yourtests. To better meet CRAN recommended practices, testthat now recommendthat you to put your tests intests/testthat, instead ofinst/tests (this makes it possible for users to choosewhether or not to install tests). With this new structure, you’ll needto usetest_check() instead oftest_packages()in the test file (usuallytests/testthat.R) that runs alltestthat unit tests.
The other big improvement to usability comes fromSummaryReporter) to include source references so you cansee exactly where failures occurred.
MultiReporter, which combines several reporters intoone. (Thanks to
ListReporter, which captures all test results withtheir file, context, test and elapsed time.test_dir,test_file,test_package andtest_check now use theListReporter toinvisibly return a summary of the tests as a data frame. (Thanks to@kforner)
TeamCityReporter to produce output compatible withthe TeamCity continuous integration environment. (Thanks to
SilentReporter so thattestthat cantest calls totest_that. (Thanks to
expect_null() andis_null to check ifan object is NULL (#78)
expect_named() andhas_names() to checkthe names of a vector (#79)
expect_more_than(),is_more_than(),expect_less_than(),is_less_than() to checkvalues above or below a threshold. (#77, thanks to
expect_that() (and thus allexpect_*functions) now invisibly return the expectation result, and stops ifinfo or label arguments have length > 1 (thanks to
fixed two bugs with source_dir(): it did not look for the sourcescripts at the right place, and it did not use itschdirargument.
When usingexpect_equal() to compare strings, thedefault output for failure provides a lot more information, which shouldhopefully help make finding string mismatches easier.
SummaryReporter has amax_reportsoption to limit the number of detailed failure reports to show. (Thanksto
Tracebacks will now also contain information about where thefunctions came from (where that information is available).
matches andexpect_match now passadditional arguments on togrepl so that you can usefixed = TRUE,perl = TRUE orignore.case = TRUE to control details of the match.expect_match now correctly fails to match NULL.(#100)
expect_output,expect_message,expect_warning andexpect_error also pass … ontogrepl, so that you can usefixed = TRUE,perl = TRUE orignore.case = TRUE
Removedstringr andevaluatedependencies.
Thenot() function makes it possible to negatetests. For example,expect_that(f(), not(throws_error()))asserts thatf() does not throw an error.
Makedir_state less race-y. (Thanks to
auto_test now pays attention to its ‘reporter’argument (Thanks to
get_reporter(),set_reporter() andwith_reporter() are now exported (#102)
Ignore attributes inis_true andis_false (#49)
make_expectation works for more types of input(#52)
Now works better with evaluate 0.4.3.
newfail() function always forces a failure in atest. Suggested by Richie Cotton (#47)
AddedTapReporter to produce output compatible withthe “test anything protocol”. Contributed by Dan Keshet.
Fixed whereauto_test would identify the wrong filesas having changed. (Thanks to Peter Meilstrup)
SummaryReporter: still return informative messageseven if no tests defined (just bare expectations). (Fixes #31)
Improvements to reference classes (Thanks to JohnChambers)
Bug fixes for when nothing was generated ingives_warning /shows_message. (Thanks toBernd Bischl)
Newmake_expectation function to programmaticallygenerate an equality expectation. (Fixes #24)
SummaryReporter: You don’t get praise until you havesome tests.
Depend onmethods rather than requiring it so thattestthat works when run fromRscript
auto_test now normalises paths to enable betteridentification of file changes, and fixes bug in instantiating newreporter object.
Allmutatr classes have been replaced withReferenceClasses.
Better documentation for short-hand expectations.
test_dir andtest_package gain newfilter argument which allows you to restrict which testsare run.
autotest correctly loads code and executes tests in sameenvironment
contexts are never closed before they are opened, and alwaysclosed at the end of file
fixed small bug intest_dir where each test was notgiven its own environment
allexpect_* short cut functions gain a labelargument, thanks to Steve Lianoglou
all expectations now have a shortcut form, so instead ofexpect_that(a, is_identical_to(b)) you can do expect_identical(a,b)
new shows_message and gives_warning expectations to test warningsand messages
expect_that, equals, is_identical_to and is_equivalent to nowhave additional label argument which allows you to control theappearance of the text used for the expected object (for expect_that)and actual object (for all other functions) in failure messages. This isuseful when you have loops that run tests as otherwise all the variablenames are identical, and it’s difficult to tell which iteration causedthe failure.
executing bare tests gives nicer output
all expectations now give more information on failure to make iteasier to track down the problem.
test_file and test_dir now run in code in separate environment toavoid pollution of global environment. They also temporary change theworking directory so tests can use relative paths.
test_package makes it easier to run all tests in an installedpackage. Code run in this manner has access to non-exported functionsand objects. If any errors or failures occur, test_package will throw anerror, making it suitable for use with R CMD check.
colourise also works in screen terminal
equals expectation provides more information aboutfailure
expect_that has extra info argument to allow you to pass in anyextra information you’d like included in the message - this is veryhelpful if you’re using a loop to run tests
is_equivalent_to: new expectation that tests for equalityignoring attributes
library_if_available now works! (thanks to report and fix fromFelix Andrews)
specify larger width and join pieces back together wheneverdeparse used (thanks to report and fix from Felix Andrews)
test_dir now looks for any files starting with test (not test- asbefore)