Base.runtests(tests=["all"]; ncores=ceil(Int, Sys.CPU_THREADS / 2),              exit_on_error=false, revise=false, propagate_project=true, [seed], [julia_args::Cmd])

Run the Julia unit tests listed intests, which can be either a string or an array of strings, usingncores processors. Ifexit_on_error isfalse, when one test fails, all remaining tests in other files will still be run; they are otherwise discarded, whenexit_on_error == true. Ifrevise istrue, theRevise package is used to load any modifications toBase or to the standard libraries before running the tests. Ifpropagate_project is true the current project is propagated to the test environment. If a seed is provided via the keyword argument, it is used to seed the global RNG in the context where the tests are run; otherwise the seed is chosen randomly. The argumentjulia_args can be used to pass customjulia command line flags to the test process.

@test ex@test f(args...) key=val ...@test ex broken=true@test ex skip=true

Test that the expressionex evaluates totrue. If executed inside a@testset, return aPassResult if it does, aFailResult if it isfalse, and anErrorResult if it could not be evaluated. If executed outside a@testset, throw an exception instead of returningFail orError.

Examples

julia> @test trueTest Passedjulia> @test [1, 2] + [2, 1] == [3, 3]Test Passed

The@test f(args...) key=val... form is equivalent to writing@test f(args..., key=val...) which can be useful when the expression is a call using infix syntax such as approximate comparisons:

julia> @test π ≈ 3.14 atol=0.01Test Passed

This is equivalent to the uglier test@test ≈(π, 3.14, atol=0.01). It is an error to supply more than one expression unless the first is a call expression and the rest are assignments (k=v).

You can use any key for thekey=val arguments, except forbroken andskip, which have special meanings in the context of@test:

• broken=cond indicates a test that should pass but currently consistently fails whencond==true. Tests that the expressionex evaluates tofalse or causes an exception. Returns aBrokenResult if it does, or anErrorResult if the expression evaluates totrue. Regular@test ex is evaluated whencond==false.
• skip=cond marks a test that should not be executed but should be included in test summary reporting asBroken, whencond==true. This can be useful for tests that intermittently fail, or tests of not-yet-implemented functionality. Regular@test ex is evaluated whencond==false.

Examples

julia> @test 2 + 2 ≈ 6 atol=1 broken=trueTest Broken  Expression: ≈(2 + 2, 6, atol = 1)julia> @test 2 + 2 ≈ 5 atol=1 broken=falseTest Passedjulia> @test 2 + 2 == 5 skip=trueTest Broken  Skipped: 2 + 2 == 5julia> @test 2 + 2 == 4 skip=falseTest Passed

@test_throws exception expr

Tests that the expressionexpr throwsexception. The exception may specify either a type, a string, regular expression, or list of strings occurring in the displayed error message, a matching function, or a value (which will be tested for equality by comparing fields). Note that@test_throws does not support a trailing keyword form.

Examples

julia> @test_throws BoundsError [1, 2, 3][4]Test Passed      Thrown: BoundsErrorjulia> @test_throws DimensionMismatch [1, 2, 3] + [1, 2]Test Passed      Thrown: DimensionMismatchjulia> @test_throws "Try sqrt(Complex" sqrt(-1)Test Passed     Message: "DomainError with -1.0:\nsqrt was called with a negative real argument but will only return a complex result if called with a complex argument. Try sqrt(Complex(x))."

In the final example, instead of matching a single string it could alternatively have been performed with:

• ["Try", "Complex"] (a list of strings)
• r"Try sqrt\([Cc]omplex" (a regular expression)
• str -> occursin("complex", str) (a matching function)

@testset [CustomTestSet] [options...] ["description"] begin test_ex end@testset [CustomTestSet] [options...] ["description $v"] for v in itr test_ex end@testset [CustomTestSet] [options...] ["description $v, $w"] for v in itrv, w in itrw test_ex end@testset [CustomTestSet] [options...] ["description"] test_func()@testset let v = v, w = w; test_ex; end

With begin/end or function call

When @testset is used, with begin/end or a single function call, the macro starts a new test set in which to evaluate the given expression.

If no custom testset type is given it defaults to creating aDefaultTestSet.DefaultTestSet records all the results and, if there are anyFails orErrors, throws an exception at the end of the top-level (non-nested) test set, along with a summary of the test results.

Any custom testset type (subtype ofAbstractTestSet) can be given and it will also be used for any nested@testset invocations. The given options are only applied to the test set where they are given. The default test set type accepts the following options:

• verbose::Bool: iftrue, the result summary of the nested testsets is shown even when they all pass (the default isfalse).
• showtiming::Bool: iftrue, the duration of each displayed testset is shown (the default istrue).
• failfast::Bool: iftrue, any test failure or error will cause the testset and any child testsets to return immediately (the default isfalse). This can also be set globally via the env varJULIA_TEST_FAILFAST.
• rng::Random.AbstractRNG: use the given random number generator (RNG) as the global one for the testset.rng must becopy!-able. This option can be useful to locally reproduce stochastic test failures which only depend on the state of the global RNG.

The description string accepts interpolation from the loop indices. If no description is provided, one is constructed based on the variables. If a function call is provided, its name will be used. Explicit description strings override this behavior.

By default the@testset macro will return the testset object itself, though this behavior can be customized in other testset types. If afor loop is used then the macro collects and returns a list of the return values of thefinish method, which by default will return a list of the testset objects used in each iteration.

Before the execution of the body of a@testset, there is an implicit call tocopy!(Random.default_rng(), rng) whererng is the RNG of the current task, or the value of the RNG passed via therng option. Moreover, after the execution of the body, the state of the global RNG is restored to what it was before the@testset. This is meant to ease reproducibility in case of failure, and to allow seamless re-arrangements of@testsets regardless of their side-effect on the global RNG state.

Examples

julia> @testset "trigonometric identities" begin           θ = 2/3*π           @test sin(-θ) ≈ -sin(θ)           @test cos(-θ) ≈ cos(θ)           @test sin(2θ) ≈ 2*sin(θ)*cos(θ)           @test cos(2θ) ≈ cos(θ)^2 - sin(θ)^2       end;Test Summary:            | Pass  Total  Timetrigonometric identities |    4      4  0.2s

@testset for

When@testset for is used, the macro starts a new test for each iteration of the provided loop. The semantics of each test set are otherwise identical to that of thatbegin/end case (as if used for each loop iteration).

@testset let

When@testset let is used, the macro starts atransparent test set with the given object added as a context object to any failing test contained therein. This is useful when performing a set of related tests on one larger object and it is desirable to print this larger object when any of the individual tests fail. Transparent test sets do not introduce additional levels of nesting in the test set hierarchy and are passed through directly to the parent test set (with the context object appended to any failing tests.)

Special implicit world age increment for@testset begin

World age inside@testset begin increments implicitly after every statement. This matches the behavior of ordinary toplevel code, but not that of ordinarybegin/end blocks, i.e. with respect to world age,@testset begin behaves as if the body of thebegin/end block was written at toplevel.

Examples

julia> @testset let logi = log(im)           @test imag(logi) == π/2           @test !iszero(real(logi))       endTest Failed at none:3  Expression: !(iszero(real(logi)))     Context: logi = 0.0 + 1.5707963267948966imERROR: There was an error during testingjulia> @testset let logi = log(im), op = !iszero           @test imag(logi) == π/2           @test op(real(logi))       endTest Failed at none:3  Expression: op(real(logi))     Context: logi = 0.0 + 1.5707963267948966im              op = !iszeroERROR: There was an error during testing

TestSetException

Thrown when a test set finishes and not all tests passed.

@test_logs [log_patterns...] [keywords] expression

Collect a list of log records generated byexpression usingcollect_test_logs, check that they match the sequencelog_patterns, and return the value ofexpression. Thekeywords provide some simple filtering of log records: themin_level keyword controls the minimum log level which will be collected for the test, thematch_mode keyword defines how matching will be performed (the default:all checks that all logs and patterns match pairwise; use:any to check that the pattern matches at least once somewhere in the sequence.)

The most useful log pattern is a simple tuple of the form(level,message). A different number of tuple elements may be used to match other log metadata, corresponding to the arguments to passed toAbstractLogger via thehandle_message function:(level,message,module,group,id,file,line). Elements which are present will be matched pairwise with the log record fields using== by default, with the special cases thatSymbols may be used for the standard log levels, andRegexs in the pattern will match string or Symbol fields usingoccursin.

Examples

Consider a function which logs a warning, and several debug messages:

function foo(n)    @info "Doing foo with n=$n"    for i=1:n        @debug "Iteration $i"    end    42end

We can test the info message using

@test_logs (:info,"Doing foo with n=2") foo(2)

If we also wanted to test the debug messages, these need to be enabled with themin_level keyword:

using Logging@test_logs (:info,"Doing foo with n=2") (:debug,"Iteration 1") (:debug,"Iteration 2") min_level=Logging.Debug foo(2)

If you want to test that some particular messages are generated while ignoring the rest, you can set the keywordmatch_mode=:any:

using Logging@test_logs (:info,) (:debug,"Iteration 42") min_level=Logging.Debug match_mode=:any foo(100)

The macro may be chained with@test to also test the returned value:

@test (@test_logs (:info,"Doing foo with n=2") foo(2)) == 42

If you want to test for the absence of warnings, you can omit specifying log patterns and set themin_level accordingly:

# test that the expression logs no messages when the logger level is warn:@test_logs min_level=Logging.Warn @info("Some information") # passes@test_logs min_level=Logging.Warn @warn("Some information") # fails

If you want to test the absence of warnings (or error messages) instderr which are not generated by@warn, see@test_nowarn.

TestLogger(; min_level=Info, catch_exceptions=false)

Create aTestLogger which captures logged messages in itslogs::Vector{LogRecord} field.

Setmin_level to control theLogLevel,catch_exceptions for whether or not exceptions thrown as part of log event generation should be caught, andrespect_maxlog for whether or not to follow the convention of logging messages withmaxlog=n for some integern at mostn times.

See also:LogRecord.

Examples

julia> using Test, Loggingjulia> f() = @info "Hi" number=5;julia> test_logger = TestLogger();julia> with_logger(test_logger) do           f()           @info "Bye!"       endjulia> @test test_logger.logs[1].message == "Hi"Test Passedjulia> @test test_logger.logs[1].kwargs[:number] == 5Test Passedjulia> @test test_logger.logs[2].message == "Bye!"Test Passed

LogRecord

Stores the results of a single log event. Fields:

• level: theLogLevel of the log message
• message: the textual content of the log message
• _module: the module of the log event
• group: the logging group (by default, the name of the file containing the log event)
• id: the ID of the log event
• file: the file containing the log event
• line: the line within the file of the log event
• kwargs: any keyword arguments passed to the log event

@inferred [AllowedType] f(x)

Tests that the call expressionf(x) returns a value of the same type inferred by the compiler. It is useful to check for type stability.

f(x) can be any call expression. Returns the result off(x) if the types match, and anErrorResult if it finds different types.

Optionally,AllowedType relaxes the test, by making it pass when either the type off(x) matches the inferred type moduloAllowedType, or when the return type is a subtype ofAllowedType. This is useful when testing type stability of functions returning a small union such asUnion{Nothing, T} orUnion{Missing, T}.

julia> f(a) = a > 1 ? 1 : 1.0f (generic function with 1 method)julia> typeof(f(2))Int64julia> @code_warntype f(2)MethodInstance for f(::Int64)  from f(a) @ Main none:1Arguments  #self#::Core.Const(f)  a::Int64Body::UNION{FLOAT64, INT64}1 ─ %1 = :>::Core.Const(>)│   %2 = (%1)(a, 1)::Bool└──      goto #3 if not %22 ─      return 13 ─      return 1.0julia> @inferred f(2)ERROR: return type Int64 does not match inferred return type Union{Float64, Int64}[...]julia> @inferred max(1, 2)2julia> g(a) = a < 10 ? missing : 1.0g (generic function with 1 method)julia> @inferred g(20)ERROR: return type Float64 does not match inferred return type Union{Missing, Float64}[...]julia> @inferred Missing g(20)1.0julia> h(a) = a < 10 ? missing : f(a)h (generic function with 1 method)julia> @inferred Missing h(20)ERROR: return type Int64 does not match inferred return type Union{Missing, Float64, Int64}[...]

@test_deprecated [pattern] expression

When--depwarn=yes, test thatexpression emits a deprecation warning and return the value ofexpression. The log message string will be matched againstpattern which defaults tor"deprecated"i.

When--depwarn=no, simply return the result of executingexpression. When--depwarn=error, check that an ErrorException is thrown.

Examples

# Deprecated in julia 0.7@test_deprecated num2hex(1)# The returned value can be tested by chaining with @test:@test (@test_deprecated num2hex(1)) == "0000000000000001"

@test_warn msg expr

Test whether evaluatingexpr results instderr output that contains themsg string or matches themsg regular expression. Ifmsg is a boolean function, tests whethermsg(output) returnstrue. Ifmsg is a tuple or array, checks that the error output contains/matches each item inmsg. Returns the result of evaluatingexpr.

See also@test_nowarn to check for the absence of error output.

Note: Warnings generated by@warn cannot be tested with this macro. Use@test_logs instead.

@test_nowarn expr

Test whether evaluatingexpr results in emptystderr output (no warnings or other messages). Returns the result of evaluatingexpr.

Note: The absence of warnings generated by@warn cannot be tested with this macro. Use@test_logs instead.

@test_broken ex@test_broken f(args...) key=val ...

Indicates a test that should pass but currently consistently fails. Tests that the expressionex evaluates tofalse or causes an exception. Returns aBrokenResult if it does, or anErrorResult if the expression evaluates totrue. This is equivalent to@test ex broken=true.

The@test_broken f(args...) key=val... form works as for the@test macro.

Examples

julia> @test_broken 1 == 2Test Broken  Expression: 1 == 2julia> @test_broken 1 == 2 atol=0.1Test Broken  Expression: ==(1, 2, atol = 0.1)

@test_skip ex@test_skip f(args...) key=val ...

Marks a test that should not be executed but should be included in test summary reporting asBroken. This can be useful for tests that intermittently fail, or tests of not-yet-implemented functionality. This is equivalent to@test ex skip=true.

The@test_skip f(args...) key=val... form works as for the@test macro.

Examples

julia> @test_skip 1 == 2Test Broken  Skipped: 1 == 2julia> @test_skip 1 == 2 atol=0.1Test Broken  Skipped: ==(1, 2, atol = 0.1)

Test.Result

All tests produce a result object. This object may or may not be stored, depending on whether the test is part of a test set.

Test.Pass <: Test.Result

The test condition was true, i.e. the expression evaluated to true or the correct exception was thrown.

Test.Fail <: Test.Result

The test condition was false, i.e. the expression evaluated to false or the correct exception was not thrown.

Test.Error <: Test.Result

The test condition couldn't be evaluated due to an exception, or it evaluated to something other than aBool. In the case of@test_broken it is used to indicate that an unexpectedPassResult occurred.

Test.Broken <: Test.Result

The test condition is the expected (failed) result of a broken test, or was explicitly skipped with@test_skip.

record(ts::AbstractTestSet, res::Result)

Record a result to a testset. This function is called by the@testset infrastructure each time a contained@test macro completes, and is given the test result (which could be anError). This will also be called with anError if an exception is thrown inside the test block but outside of a@test context.

finish(ts::AbstractTestSet)

Do any final processing necessary for the given testset. This is called by the@testset infrastructure after a test block executes.

CustomAbstractTestSet subtypes should callrecord on their parent (if there is one) to add themselves to the tree of test results. This might be implemented as:

if get_testset_depth() != 0    # Attach this test set to the parent test set    parent_ts = get_testset()    record(parent_ts, self)    return selfend

get_testset()

Retrieve the active test set from the task's local storage. If no test set is active, use the fallback default test set.

get_testset_depth()

Return the number of active test sets, not including the default test set

TestCounts

Holds the state for recursively gathering the results of a test set for display purposes.

Fields:

• customized: Whether the functionget_test_counts was customized for theAbstractTestSet this counts object is for. If a custom method was defined, always passtrue to the constructor.
• passes: The number of passing@test invocations.
• fails: The number of failing@test invocations.
• errors: The number of erroring@test invocations.
• broken: The number of broken@test invocations.
• passes: The cumulative number of passing@test invocations.
• fails: The cumulative number of failing@test invocations.
• errors: The cumulative number of erroring@test invocations.
• broken: The cumulative number of broken@test invocations.
• duration: The total duration theAbstractTestSet in question ran for, as a formattedString.

" gettestcounts(::AbstractTestSet) -> TestCounts

Recursive function that counts the number of test results of each type directly in the testset, and totals across the child testsets.

CustomAbstractTestSet should implement this function to get their totals counted & displayed withDefaultTestSet as well.

If this is not implemented for a customTestSet, the printing falls back to reportingx for failures and?s for the duration.

format_duration(::AbstractTestSet)

Return a formatted string for printing the duration the testset ran for.

If not defined, falls back to"?s".

print_test_results(ts::AbstractTestSet, depth_pad=0)

Print the results of anAbstractTestSet as a formatted table.

depth_pad refers to how much padding should be added in front of all output.

Called inside ofTest.finish, if thefinished testset is the topmost testset.

TheGenericArray can be used to test generic array APIs that program to theAbstractArray interface, in order to ensure that functions can work with array types besides the standardArray type.

TheGenericDict can be used to test generic dict APIs that program to theAbstractDict interface, in order to ensure that functions can work with associative types besides the standardDict type.

TheGenericOrder can be used to test APIs for their support of generic ordered types.

TheGenericSet can be used to test generic set APIs that program to theAbstractSet interface, in order to ensure that functions can work with set types besides the standardSet andBitSet types.

TheGenericString can be used to test generic string APIs that program to theAbstractString interface, in order to ensure that functions can work with string types besides the standardString type.

detect_ambiguities(mod1, mod2...; recursive=false,                                  ambiguous_bottom=false,                                  allowed_undefineds=nothing)

Return a vector of(Method,Method) pairs of ambiguous methods defined in the specified modules. Userecursive=true to test in all submodules.

ambiguous_bottom controls whether ambiguities triggered only byUnion{} type parameters are included; in most cases you probably want to set this tofalse. SeeBase.isambiguous.

SeeTest.detect_unbound_args for an explanation ofallowed_undefineds.

detect_unbound_args(mod1, mod2...; recursive=false, allowed_undefineds=nothing)

Return a vector ofMethods which may have unbound type parameters. Userecursive=true to test in all submodules.

By default, any undefined symbols trigger a warning. This warning can be suppressed by supplying a collection ofGlobalRefs for which the warning can be skipped. For example, setting

allowed_undefineds = Set([GlobalRef(Base, :active_repl),                          GlobalRef(Base, :active_repl_backend)])

would suppress warnings aboutBase.active_repl andBase.active_repl_backend.