Repository files navigation

uTest is a simple, convenient testing library for Scala.Its key features are:

• Nicely formatted, colored, easy-to-read command-line test output

• Single uniform syntax for defining tests and grouping them together

• Single uniform syntax for running test suites and individual tests

• Single uniform syntax for Smart Asserts, instead of multipleredundantmust_==/must beEqual/should be operators

• Auto-updatingassertGoldenLiteral andassertGoldenFile assertionsthat make easy to keep your test suite up to date with the last behavior of your code

• Isolation-by-default for tests in the same suite

• SupportsScala.js and Scala-Native, Scala 2.12.x/2.13.x/3.x,projects usingSBT orMillstandalone (e.g. via amain method, or inAmmonite Scripts),projects usingGradle plugin for Scala.js and Scala Native.

Unlike traditional testing libraries for Scala (likeScalatest orSpecs2) uTest aims to be simple enoughyou will never "get lost" in its codebase and functionality, so you can focus onwhat's most important: your tests.

While uTest has many fewer features than other libraries,the features that it does provide are polished and are enough to buildand maintain test suites of any size. uTest is used for countless projects, fromthe 1-file test suite forFansito the 50-file 9,000-line test suite forAmmonite

If you use uTest and like it, please support it by donating to our Patreon:

• https://www.patreon.com/lihaoyi

• Getting Started
• Defining and Running a Test Suite
  • Nesting Tests
  • Running Tests
  • Sharing Setup Code, and Sharing Setup Objects
  • Other Ways of Naming tests
  • Asynchronous Tests
• Smart Asserts
  • Arrow Asserts
  • assertThrows
  • assertEventually and assertContinually
  • assertMatch
  • assertCompileError
  • assertGoldenLiteral
  • assertGoldenFile
• Test Utilities
  • TestPath
  • Local Retries
• Configuring uTest
  • Per-Run Setup/Teardown, and other test-running Config
  • Output Formatting
  • Suite Retries
  • Running code before and after test cases
  • Running code before and after test suites
  • Test Wrapping
• Scala.js and Scala-Native
• Running uTest Standalone
• Why uTest
• Changelog

Most people coming to uTest will be running tests throughSBT. Add the following to yourbuild.sbt and youcan immediately begin defining and running tests programmatically.

libraryDependencies+="com.lihaoyi"%%"utest"%"0.10.0-RC1"%"test"// Scala-JVMlibraryDependencies+="com.lihaoyi"%%%"utest"%"0.10.0-RC1"%"test"// Scala.js or Scala-NativetestFrameworks+=newTestFramework("utest.runner.Framework")

Or Mill:

defmvnDeps=Seq(mvn"com.lihaoyi::utest:0.10.0-RC1")// Scala-JVMdefmvnDeps=Seq(mvn"com.lihaoyi::utest::0.10.0-RC1")// Scala.js or Scala-NativedeftestFramework="utest.runner.Framework"

For Scala-Native, you will also need

nativeLinkStubs:=true

Gradle plugin for Scala.js and Scala Nativesupports using uTest withGradle.

Put this in yoursrc/test/scala/ folder:

packageexampleimportutest._classHelloTestsextendsTestSuite{valtests=Tests{    test("test1"){thrownewException("test1")    }    test("test2"){1    }    test("test3"){vala=List[Byte](1,2)      a(10)    }  }}

You can then run this via

sbt myproject/test

Or

./mill myproject.test

Which should produce this output:

-------------------------------- Running Tests --------------------------------Setting up CustomFrameworkX example.HelloTests.test1 4ms  java.lang.Exception: test1    example.HelloTests$.$anonfun$tests$2(HelloTests.scala:7)+ example.HelloTests.test2.inner 0ms  1X example.HelloTests.test3 0ms  java.lang.IndexOutOfBoundsException: 10    scala.collection.LinearSeqOptimized.apply(LinearSeqOptimized.scala:63)    scala.collection.LinearSeqOptimized.apply$(LinearSeqOptimized.scala:61)    scala.collection.immutable.List.apply(List.scala:86)    example.HelloTests$.$anonfun$tests$5(HelloTests.scala:16)Tearing down CustomFrameworkTests: 3, Passed: 1, Failed: 2

The tests are run one at a time, and any tests that fail with an exception havetheir stack trace printed. If the number of tests is large, a separateresults-summary and failures-summary will be shown after all tests have run.Tests can either be inside zero-parameterclasses (as shown above) or staticobjects.

Note that tests within the suite can nested within each other, but onlydirectly. E.g. you cannot define tests withinif-statements orfor-loops.uTest relies on the test structure to be statically known at compile time. Theycan be nested arbitrarily deep:

packageexampleimportutest._classNestedTestsextendsTestSuite{valtests=Tests{valx=1    test("outer1"){valy= x+1      test("inner1"){        assert(x==1, y==2)        (x, y)      }      test("inner2"){valz= y+1        assert(z==3)      }    }    test("outer2"){      test("inner3"){        assert(x>1)      }    }  }}

Here, we have a tree of nested blocks, with three tests at the inner-most blocksof this tree:innerest,inner2 andinner3. Test blocks can be nestedarbitrary deeply to help keep things neat, and only the inner-most blocks areconsidered to be tests.

When this suite is run withsbt myproject/test, it is those three tests that get executed:

------------------------------- Running Tests -------------------------------+ example.NestedTests.outer1.inner1 21ms  (1,2)+ example.NestedTests.outer1.inner2 0ms+ example.NestedTests.outer2.inner3 0ms

You can see also thatexample.NestedTests.outer1.inner1 displaysthe value of(x, y) returned from the test:(1,2). Returning a value from atest is useful if you want to skim the test's results after a run to performmanual sanity-checks on some computed value.

If you find yourself wanting to define a test in a for, loop, e.g.

// Doesn't work!valtests=Tests{for(fileName<-Seq("hello","world","i","am","cow")){    fileName- {// lots of code using fileName    }  }}

You can instead factor out the common code into a function, and call that fromeach distinct test case:

// Works!valtests=Tests{defrunTestChecks(fileName:String)= {// lots of code using fileName  }  test("hello"){ runTestChecks("hello") }  test("world"){ runTestChecks("world") }  test("i"){ runTestChecks("i") }  test("am"){ runTestChecks("am") }  test("cow"){ runTestChecks("cow") }}

Or even using theTestPath that is available implicitly to everytest, you can remove the duplication between the test name and the call torunTestChecks():

// Also works!valtests=Tests{defrunTestChecks()(implicitpath: utest.framework.TestPath)= {valfileName= path.value.last// lots of code using fileName  }  test("hello"){ runTestChecks() }  test("world"){ runTestChecks() }  test("i"){ runTestChecks() }  test("am"){ runTestChecks() }  test("cow"){ runTestChecks() }}

Apart from running all tests at once using

sbt myproject/test

You can also run individual tests using their full path e.g.

sbt'myproject/test-only -- example.NestedTests.outer1.inner1'sbt'myproject/test-only -- example.NestedTests.outer2.inner2'sbt'myproject/test-only -- example.NestedTests.outer2.inner3'./mill myProject.test example.NestedTests.outer1.inner1./mill myProject.test example.NestedTests.outer2.inner2./mill myProject.test example.NestedTests.outer2.inner3

You can also wrap the test selector in double quotes, which lets you run testwhose path segments contain spaces or other special characters:

sbt'myproject/test-only -- "example.NestedTests.segment with spaces.inner"'./mill myProject.test"example.NestedTests.segment with spaces.inner"

You can run groups of tests by providing the path to the block enclosing all ofthem:

# runs both tests `inner1` and `inner2`sbt'myproject/test-only -- example.NestedTests.outer1'./mill myProject.test example.NestedTests.outer1# runs all tests in the `NestedTests` test suitesbt'myproject/test-only -- example.NestedTests'./mill myProject.test example.NestedTests# runs all tests in `NestedTests` and any other suites within `example`sbt'myproject/test-only -- example'./mill myProject.test example

You can also use the{foo,bar} syntax to specify exactly which tests you wouldlike to run:

# runs both tests `inner2` and `inner3`, explicitlysbt'myproject/test-only -- example.NestedTests.outer1.{inner1,inner2}'./mill myProject.test'example.NestedTests.outer1.{inner1,inner2}'# runs `outer1.inner1` and `outer2.inner3` but not `outer1.inner2`sbt'myproject/test-only -- example.NestedTests.{outer1.inner1,outer2.inner3}'./mill myProject.test'example.NestedTests.{outer1.inner1,outer2.inner3}'# also runs `inner1` and `innerest` (and any other test inside `inner1`) but not `inner2`sbt'myproject/test-only -- example.NestedTests.{outer1.inner1,outer2}'./mill myProject.test'example.NestedTests.{outer1.inner1,outer2}'

The same syntax can be used to pick and choose specificTestSuites to run, ortests within those test suites:

# Run every test in `HelloTests` and `NestedTests`sbt'myproject/test-only -- example.{HelloTests,NestedTests}'# Runs `HelloTests.test1`, `NestedTests.outer1.inner2` and `NestedTests.outer2.inner3`sbt'myproject/test-only -- example.{HelloTests.test1,NestedTests.outer2}'sbt'myproject/test-only -- {example.HelloTests.test1,example.NestedTests.outer2}'

In general, it doesn't really matter if you are running individual tests, groupsof tests within aTestSuite, individualTestSuites, or packages containingTestSuite. These all form one a single large tree of tests that you can run,using the same uniform syntax.

By default, SBT and Mill run multiple test suites in parallel, so the output fromthose suites may be interleaved. You can setparallelExecution in Test := falsein your SBT config ordef testParallelism = false in your Mill configto make the tests execute sequentially, so the output fromeach suite will be grouped together in the terminal.

uTest defaults to emitting ANSI-colored terminal output describing the test run.You can configure the colors byoverriding methods on your test suite, or disable italtogether withoverride def formatColor = false.

As you saw in the previous section, you can define blocks tests to group themtogether, and have them share common initialization code in the enclosing block.You can also definemutable values, or "fixtures" in the shared initializationcode, and each nested test with aTests block gets its own copy of any mutablevariables defined within it:

packageexampleimportutest._classSeparateSetupTestsextendsTestSuite{valtests=Tests{varx=0    test("outer1"){      x+=1      test("inner1"){        x+=2        assert(x==3)// += 1, += 2        x      }      test("inner2"){        x+=3        assert(x==4)// += 1, += 3        x      }    }    test("outer2"){      x+=4      test("inner3"){        x+=5        assert(x==9)// += 4, += 5        x      }    }  }}

Here, you can see that thex available in each inner test block (inner1,inner2,inner3) is separate and independent: each test gets a new copy ofx, modified only by that test's enclosing blocks. This helps avoid inter-testinterference (where a test ends up implicitly depending on some stateinitialized by another test running earlier in the block) while still making itconvenient to share common setup code between the various tests in your suite.

If you want the mutable fixtures to really-truly be shared between individualtests (e.g. because they are expensive to repeatedly initialize) define itoutside theTests{} block in the enclosingclass:

packageexampleimportutest._classSharedFixturesTestsextendsTestSuite{varx=0valtests=Tests{    test("outer1"){      x+=1      test("inner1"){        x+=2        assert(x==3)// += 1, += 2        x      }      test("inner2"){        x+=3        assert(x==7)// += 1, += 2, += 1, += 3        x      }    }    test("outer2"){      x+=4      test("inner3"){        x+=5        assert(x==16)// += 1, += 2, += 1, += 3, += 4, += 5        x      }    }  }}

Here you see that the changes tox are being shared between the invocations ofall the tests. If you are initializing something expensive to share betweentests, this allows you to avoid paying that cost multiple times, but you need tobe careful the tests aren't mutating shared state that could cause other teststo fail!

You can also use thetest("symbol") - syntax, if your tests are simply forwardingto a separate helper method to do the real testing:

test("test1")- processFileAndCheckOutput("input1.txt","expected1.txt")test("test2")- processFileAndCheckOutput("input2.txt","expected2.txt")test("test3")- processFileAndCheckOutput("input3.txt","expected3.txt")

Thetest("string"){...} andtest("symbol") - ... syntaxes are equivalent.

The last way of defining tests is with a baretest {...} ortest - ..., e.g. these testsfrom theFansiproject:

test("parsing"){defcheck(frag: fansi.Str)= {valparsed= fansi.Str(frag.render)    assert(parsed== frag)    parsed  }  test- check(fansi.Color.True(255,0,0)("lol"))  test- check(fansi.Color.True(1,234,56)("lol"))  test- check(fansi.Color.True(255,255,255)("lol"))  test{    (for(i<-0 to255)yield check(fansi.Color.True(i,i,i)("x"))).mkString  }  test{    check("#"+ fansi.Color.True(127,126,0)("lol")+"omg"+ fansi.Color.True(127,126,0)("wtf")    )  }  test{ check(square(for(i<-0 to255)yield fansi.Color.True(i,i,i))) }}

Tests defined using thetest symbol are give the numerical names "0", "1", "2",etc.. This is handy if you have a very large number of very simple test cases,don't really care what each one is called, but still want to be able to run themand collect their result separately.

valtests=Tests {  test("testSuccess"){Future {      assert(true)    }  }  test("testFail"){Future {      assert(false)    }  }  test("normalSuccess"){    assert(true)  }  test("normalFail"){    assert(false)  }}TestRunner.runAsync(tests).map { results=>valleafResults= results.leaves.toSeq  assert(leafResults(0).value.isSuccess)// root  assert(leafResults(1).value.isSuccess)// testSuccess  assert(leafResults(2).value.isFailure)// testFail  assert(leafResults(3).value.isSuccess)// normalSuccess}

You can have tests which return (have a last expression being) aFuture[T]instead of a normal value. You can run the suite using.runAsync to return aFuture of the results, or you can continue using.run which will wait forall the futures to complete before returning.

In Scala.js, calling.run on a test suite with futures in it throws an errorinstead of waiting, since you cannot wait for asynchronous results in Scala.js.

When running the test suites from SBT, you do not need worry about any of thisrun vsrunAsync stuff: the test runner will handle it for you and providethe correct results.

valx=1valy="2"assert(x>0)assert(x== y)// utest.AssertionError: x == y// x: Int = 1// y: String = 2assertAll(// Helper to perform multiple asserts at once  x>0,  x== y)

uTest comes with a macro-powered smartasserts that provide useful debugginginformation in the error message. These take one or more boolean expressions,and when they fail, will print out the names, types and values of any localvariables used in the expression that failed. This makes it much easier to seewhat's going on than Scala's defaultassert, which gives you the stack traceand nothing else.

uTest also wraps any exceptions thrown within the assert, to help trace whatwent wrong:

valx=1Lvaly=0Lassert(x/ y==10)// utest.AssertionError: assert(x / y == 10)// caused by: java.lang.ArithmeticException: / by zero// x: Long = 1// y: Long = 0

The origin exception is stored as thecause of theutest.AssertionError, sothe original stack trace is still available for you to inspect.

uTest's smart asserts live on theutest package, and are brought into scopeviaimport utest._. If you want to continue using the built-in Scala asserts,e.g. if you want custom messages if the assert fails, those remain available asPredef.assert.

1==>1// passesArray(1,2,3)==>Array(1,2,3)// passestry{1==>2// throws}catch{casee: java.lang.AssertionError=>  e}

You can usea ==> b as a shorthand forassert(a == b). This results inpretty code you can easily copy-paste into documentation.

vale= assertThrows[MatchError]{  (0:Any)match {case_:String=> }}println(e)// scala.MatchError: 0 (of class java.lang.Integer)

assertThrows allows you to verify that a block raises an exception. Thisexception is caught and returned so you can perform further validation on it,e.g. checking that the message is what you expect. If the block does not raiseone, anAssertionError is raised.

As withassert,assertThrows adds debugging information to the error messagesif theassertThrows fails or throws an unexpected Exception.

valx=Seq(12)assertEventually(x==Nil)// utest.AssertionError: assertEventually(x == Nil)// x: Seq[Int] = List(12)

In addition to a macro-poweredassert, uTest also provides macro-poweredversions ofassertEventually andassertContinually. These are used to test asynchronousconcurrent operations:

• assertEventually(tests: Boolean*): ensure that the boolean values oftests allbecome true at least once within a certain period of time.
• assertContinually(tests: Boolean*): ensure that the boolean values oftests allremain true and never become false within a certain period of time.

These are implemented via a retry-loop, with a default retry interval of 0.1second and retries up to a total of 1 second. If you want to change thisbehavior, you can shadow the implicit valuesretryInterval andretryMax, forexample this:

implicitvalretryMax=RetryMax(300.millis)implicitvalretryInterval=RetryInterval(50.millis)

Would set the retry-loop to happen every 50ms up to a max of 300ms.

Together, these two operations allow you to easily test asynchronous operations.You can use them to help verify Liveness properties (that condition musteventually be met) and Safety properties (that a condition is never met)

As withassert,assertEventually andassertContinually add debugging information tothe error messages if they fail.

assertMatch(Seq(1,2,3)){caseSeq(1,2)=>}// AssertionError: Matching failed Seq(1, 2, 3)

assertMatch is a convenient way of checking that a value matches a particularshape, using Scala's pattern matching syntax. This includes support for use of| or_ orif-guards within the pattern match. This gives you additionalflexibility over a traditionalassert(a == Seq(1, 2)), as you can use_ as awildcard e.g. usingassertMatch(a){case Seq(1, _)=>} to match any 2-itemSeqwhose first item is1.

As withassert,assertMatch adds debugging information to the error messagesif the value fails to match or throws an unexpected Exception while evaluating.

assertCompileError("true * false")// CompileError.Type("value * is not a member of Boolean")assertCompileError("(}")// CompileError.Parse("')' expected but '}' found.")

assertCompileError is a macro that can be used to assert that a fragment of code(given as a literal String) fails to compile.

• If the code compiles successfully,assertCompileError will fail the compilationrun with a message.
• If the code fails to compile,assertCompileError will return an instance ofCompileError, one ofCompileError.Type(pos: String, msgs: String*) orCompileError.Parse(pos: String, msgs: String*) to represent typecheckererrors or parser errors

In general,assertCompileError works similarly toassertThrows, except it does itschecks (that a snippet of code fails) and errors (if it doesn't fail) atcompile-time rather than run-time. If the code fails as expected, the failuremessage is propagated to runtime in the form of aCompileError object. You canthen do whatever additional checks you want on the failure message, such asverifying that the failure message contains some string you expect to be there.

TheassertCompileError macro compiles the given string in the local scope andcontext. This means that you can refer to variables in the enclosing scope, i.e.the following example will fail to compile because the variablex exists.

valx=0assertCompileError("x + x"),// [error] assertCompileError check failed to have a compilation error

The returnedCompileError object also has a handy.check method, which takesa position-string indicating where the error is expected to occur, as well aszero-or-more messages which are expected to be part of the final error message.This is used as follows:

assertCompileError("true * false").check("""assertCompileError("true * false").check(                   ^""","value * is not a member of Boolean")

Note that the position-string needs to exactly match the line of code thecompile-error occured on. This includes any whitespace on the left, as well asany unrelated code or comments sharing the same line as theassertCompileErrorexpression.

Golden testing is a useful way of defining tests that assert the value of variable matcheseither a literal data structure or the contents of a file.For example, inassertGoldenLiteral,you pass in the runtime value on the left and the expected literal on the right:

valx=List(1,2)assertGoldenLiteral(x,List(1,2,3,4))

Running this test shows you the diff between the two values as well as an environment variable you canpass to update the literal:

X test.utest.examples.HelloTests.test 75ms   utest.AssertionError: Actual value does not match golden data in file  /Users/lihaoyi/Github/utest/utest/test/src/test/utest/examples/HelloTests.scala  Run tests with UTEST_UPDATE_GOLDEN_TESTS=1 to apply the following patch to update the golden value  goldenValue != actualValue:  - List(1, 2, 3, 4)  + List(1, 2)    utest.asserts.AssertsPlatformSpecific.throwAssertionError(AssertsPlatformSpecific.scala:8)    utest.asserts.AssertsPlatformSpecific.assertGoldenLiteral(AssertsPlatformSpecific.scala:41)    utest.asserts.AssertsPlatformSpecific.assertGoldenLiteral$(AssertsPlatformSpecific.scala:6)    utest.package$.assertGoldenLiteral(package.scala:8)    test.utest.examples.HelloTests.$init$$$anonfun$1$$anonfun$1(HelloTests.scala:16)

Running the test with theUTEST_UPDATE_GOLDEN_TESTS=1 environment variable will update the sourcecode of your test suite in-place, usingPPrint to convertthe left-hand value into a string to splice into the source code. While this won't work for morecomplex data structures, it works well enough for the common cases which involve primitives,collections, andcase classes. You can customize the printing logic by overridingdef goldenLiteralPrinter in acustom framework.

assertGoldenLiteral can also be used in helper methods, in which case the helper should takethe golden literal as an instance of typeutest.framework.GoldenFix.Span[T] to capture thesource-metadata necessary to update the literal (filename, start/end offsets), as well asan implicitutest.framework.GoldenFix.Reporter (if not written within aTestSuite classwhich provides the reporter implicitly in scope):

defgoldenHelper(value:List[Int],golden: utest.framework.GoldenFix.Span[List[Int]])= {  assertGoldenLiteral(value, golden)}test("test"){valx=List(1,2)  goldenHelper(x,List(1,2))}

You can also useassertGoldenLiteral to automatically fill in the expected value thefirst time you run the test by passing in() as the current literal value, and running thetest withUTEST_UPDATE_GOLDEN_TESTS=1. For example, starting with:

valx=List(1,2)assertGoldenLiteral(x, ())

After running the test withUTEST_UPDATE_GOLDEN_TESTS=1, the code will be updated to

valx=List(1,2)assertGoldenLiteral(x,List(1,2))

This means it is easy to write the logic of the tests and have uTest "fill in the blanks"for you.

SeeGolden Literal Testing in uTest 0.9.0for more details.

assertGoldenLiteral is only supported on Scala-JVM, and not on Scala-JS and Scala-Native

assertGoldenFile(String, java.nio.file.Path) can be used to check that aStringvalue matches the contents in the file at the givenjava.nio.file.Path, and update the filewhenUTEST_UPDATE_GOLDEN_TESTS=1 is passed. A non-existent file is treated as equivalentto an empty string"", and running the test withUTEST_UPDATE_GOLDEN_TESTS=1 will

test("test"){valexpected="""I am cow      |Hear me moo      |I weigh twice as much as you      |And I look good on the barbecue""".stripMargin  assertGoldenFile(expected, java.nio.file.Path.of("/Users/lihaoyi/Github/utest/hello.txt"))}

X test.utest.examples.HelloTests.test 72ms   utest.AssertionError: Actual value does not match golden data in file  /Users/lihaoyi/Github/utest/hello.txt  Run tests with UTEST_UPDATE_GOLDEN_TESTS=1 to apply the following patch to update the golden value  goldenValue != actualValue:  - I am cow    I am cow    Hear me moo  - Moo  + I weigh twice as much as you  + And I look good on the barbecue    utest.asserts.AssertsPlatformSpecific.throwAssertionError(AssertsPlatformSpecific.scala:8)    utest.asserts.AssertsPlatformSpecific.assertGoldenFile(AssertsPlatformSpecific.scala:29)    utest.asserts.AssertsPlatformSpecific.assertGoldenFile$(AssertsPlatformSpecific.scala:6)    utest.package$.assertGoldenFile(package.scala:8)    test.utest.examples.HelloTests.$init$$$anonfun$1$$anonfun$1(HelloTests.scala:28)

Running the test again withUTEST_UPDATE_GOLDEN_TESTS=1 will update the file on disk touse the latest value present during the test.

assertGoldenFile is only supported on Scala-JVM, and not on Scala-JS and Scala-Native

uTest provides a range of test utilities that aren't strictly necessary, but aimto make your writing of tests much more convenient and DRY.

packageexampleimportutest._objectTestPathTestsextendsTestSuite{valtests=Tests{'testPath{'foo {        assert(implicitly[utest.framework.TestPath].value==Seq("testPath","foo"))      }    }  }}

uTest exposes the path to the current test to the body of the test via theutest.framework.TestPath implicit. This can be used to print debugginginformation while the test is running ("test foo.bar.baz 50% done") or toavoid repetition between the test name and test body.

One example is the Fastparse test suite, which uses the name of the test toprovide the repository that it needs to clone and parse:

defcheckRepo(filter:String=>Boolean= _=>true)             (implicittestPath: utest.framework.TestPath)= {valurl="https://github.com/"+ testPath.value.lastimportsys.process._valname= url.split("/").lastif (!Files.exists(Paths.get("target","repos", name))){    println("CLONING")Seq("git","clone", url,"target/repos/"+name,"--depth","1").!  }  checkDir("target/repos/"+name, filter)}"lihaoyi/fastparse"- checkRepo()"scala-js/scala-js"- checkRepo()"scalaz/scalaz"- checkRepo()"milessabin/shapeless"- checkRepo()"akka/akka"- checkRepo()"lift/framework"- checkRepo()"playframework/playframework"- checkRepo()"PredictionIO/PredictionIO"- checkRepo()"apache/spark"- checkRepo()

This allows us to keep the tests DRY - avoiding having to repeat the name of therepo in the name of the test for every test we define - as well as ensuring thatthey always stay in sync.

If you need additional metadata such as line-numbers or file-paths or class orpackage names, you can use theSourceCode library's implicits to pullthem in for you.

objectLocalRetryTestsextends utest.TestSuite{valflaky=newFlakyThingdeftests=Tests{    test("hello")- retry(3){      flaky.run    }  }}

You can wrap individual tests, or even individual expressions, in aretryblock to make them retry a number of times before failing. That is very usefulfor dealing with small points of flakiness within your test suite. Aretryblock simply retries its body up to the specified number of times; the first runthat doesn't throw an exception returns the value returned by that run.

You can also useSuite Retries if you want to configureretries more globally across your test suite.

To configure things which affect an entire test run, and not any individualTestSuite object, you can create your own subclass ofutest.runner.Frameworkand override the relevant methods.

For example, if you need to perform some action (initialize a database, cleanupthe filesystem, etc.) not just per-test but per-run, you can do that by defininga customutest.runner.Framework and overriding thesetup andteardownmethods:

classCustomFrameworkextends utest.runner.Framework{overridedefsetup()= {    println("Setting up CustomFramework")  }overridedefteardown()= {    println("Tearing down CustomFramework")  }}

And then telling SBT to run tests using the custom framework:

testFrameworks+=newTestFramework("test.utest.CustomFramework"),

This is handy for setup/teardown that is necessary but too expensive to dobefore/after every single test, which would be the case if you usedTest Wrapping to do it.

Apart from setup and teardown, there are other methods onutest.runner.Framework that you can override to customize:

/**    * Override to run code before tests start running. Useful for setting up    * global databases, initializing caches, etc.*/defsetup()= ()/**    * Override to run code after tests finish running. Useful for shutting    * down daemon processes, closing network connections, etc.*/defteardown()= ()/**    * How many tests need to run before uTest starts printing out the test    * results summary and test failure summary at the end of a test run. For    * small sets of tests, these aren't necessary since all the output fits    * nicely on one screen; only when the number of tests gets large and their    * output gets noisy does it become valuable to show the clean summaries at    * the end of the test run.*/defshowSummaryThreshold=30/**    * Whether to use SBT's test-logging infrastructure, or just println.    *    * Defaults to println because SBT's test logging doesn't seem to give us    * anything that we want, and does annoying things like making a left-hand    * gutter and buffering input by default*/defuseSbtLoggers=falsedefresultsHeader=DefaultFormatters.resultsHeaderdeffailureHeader=DefaultFormatters.failureHeaderdefstartHeader(path:String)=DefaultFormatters.renderBanner("Running Tests"+ path)

You can control how the output of tests gets printed via overriding methods ontheFramework class, described above. For example, this snippet overrides theexceptionStackFrameHighlighter method to select which stack frames in a stacktrace are more relevant to you, so they can be rendered more brightly:

classCustomFrameworkextends utest.runner.Framework{overridedefexceptionStackFrameHighlighter(s:StackTraceElement)= {    s.getClassName.contains("utest.")  }}

This results in stack traces being rendered as such:

theutest.runner.Framework provides a wide range of hooks you can use tocustomize how the uTest output is colored, wrapped, truncated or formatted:

defformatColor:Boolean=truedefformatTruncateHeight:Int=15defformatWrapWidth:Int=100defformatValue(x:Any)= testValueColor(x.toString)deftoggledColor(t: utest.shaded.fansi.Attrs)=if(formatColor) telse utest.shaded.fansi.Attrs.EmptydeftestValueColor= toggledColor(utest.shaded.fansi.Color.Blue)defexceptionClassColor= toggledColor(utest.shaded.fansi.Underlined.On++ utest.shaded.fansi.Color.LightRed)defexceptionMsgColor= toggledColor(utest.shaded.fansi.Color.LightRed)defexceptionPrefixColor= toggledColor(utest.shaded.fansi.Color.Red)defexceptionMethodColor= toggledColor(utest.shaded.fansi.Color.LightRed)defexceptionPunctuationColor= toggledColor(utest.shaded.fansi.Color.Red)defexceptionLineNumberColor= toggledColor(utest.shaded.fansi.Color.LightRed)defexceptionStackFrameHighlighter(s:StackTraceElement)=truedefformatResultColor(success:Boolean)= toggledColor(if (success) utest.shaded.fansi.Color.Greenelse utest.shaded.fansi.Color.Red)defformatMillisColor= toggledColor(utest.shaded.fansi.Bold.Faint)

Any methods overriden on your own customFramework apply to everyTestSuitethat is run. If you want to further customize how a singleTestSuite's outputis formatted, you can overrideutestFormatter on that test suite.

Note that uTest uses an internal copy of theFansi library, vendored atutest.shaded.fansi, in order to avoid any compatibility problems with any of yourother dependencies. You can useutest.shaded.fansi to construct the coloredutest.shaded.fansi.Strsthat these methods require, or you could just return coloredjava.lang.Stringobjects containing ANSI escapes, created however you like, and they will beautomatically parsed into the correct format.

You can mix in theTestSuite.Retries trait to anyTestSuite and define theutestRetryCount int to enable test-level retries for all tests within a suite:

objectSuiteRetryTestsextendsTestSuitewithTestSuite.Retries{overridevalutestRetryCount=3valflaky=newFlakyThingdeftests=Tests{'hello{      flaky.run    }  }}

You can also useLocal Retries if you want to only retrywithin specific tests or expressions instead of throughout the entire suite.

uTest offers theutestBeforeEach andutestAfterEach methods that you canoverride on anyTestSuite, these methods are invoked before and after runningeach test.

defutestBeforeEach(path:Seq[String]):Unit= ()defutestAfterEach(path:Seq[String]):Unit= ()

These are equivalent toutestWrap but easier to use for simple cases.

packageexampleimportutest._objectBeforeAfterEachTestextendsTestSuite {varx=0overridedefutestBeforeEach(path:Seq[String]):Unit= {    println(s"on before each x:$x")    x=0  }overridedefutestAfterEach(path:Seq[String]):Unit=    println(s"on after each x:$x")valtests=Tests{    test("outer1"){      x+=1      test("inner1"){        x+=2        assert(x==3)// += 1, += 2        x      }      test("inner2"){        x+=3        assert(x==4)// += 1, += 3        x      }    }    test("outer2"){      x+=4      test("inner3"){        x+=5        assert(x==9)// += 4, += 5        x      }    }  }}

-------------------------------- Running Tests --------------------------------Setting up CustomFrameworkon before each x: 0on after each x: 3+ example.BeforeAfterEachTest.outer1.inner1 22ms  3on before each x: 3on after each x: 4+ example.BeforeAfterEachTest.outer1.inner2 1ms  4on before each x: 4on after each x: 9+ example.BeforeAfterEachTest.outer2.inner3 0ms  9Tearing down CustomFrameworkTests: 3, Passed: 3, Failed: 0

BothutestBeforeEach andutestAfterEach runs insideutestWrap'sbodycallback.

If you need something fancier than whatutestBeforeEach orutestAfterEachprovide, e.g. passing initialized objects into the main test case or tearingthem down after the test case has completed, feel free to define your testwrapper/initialization function and use it for each test case:

defmyTest[T](func:Int=>T)= {valfixture=1337// initialize some valuevalres= func(fixture)// make the value available in the test case  assert(fixture==1337)// properly teardown the value later  res}test("test")- myTest{ fixture=>// do stuff with fixture}

The abovemyTest function can also take aTestPath implicit if itwants access to the current test's path.

If you're looking for something likeutestBeforeAll, you can add your code tothe object body, and you can also use lazy val to delay the initialization untilthe test suite object is created.

uTest offers theutestAfterAll method that you can override on anytest suite, this method is invoked after running the entire test suite.

defutestAfterAll():Unit= ()

packageexampleimportutest._objectBeforeAfterAllSimpleTestsextendsTestSuite {  println("on object body, aka: before all")overridedefutestAfterAll():Unit= {    println("on after all")  }valtests=Tests {    test("outer1"){      test("inner1"){1      }      test("inner2"){2      }    }  }}

-------------------------------- Running Tests --------------------------------Setting up CustomFrameworkon object body, aka: before all+ example.BeforeAfterAllSimpleTests.outer1.inner1 2ms  1+ example.BeforeAfterAllSimpleTests.outer1.inner2 0ms  2on after all

uTest exposes theutestWrap function that you can override on any test suite:

defutestWrap(path:Seq[String],runBody:=> concurrent.Future[Any])             (implicitec:ExecutionContext): concurrent.Future[Any]

This is a flexible function that wraps every test call; you can use it to:

• Perform initialization before evaluatingrunBody
• Perform cleanup after the completion ofrunBody viarunBody.onComplete
• Perform retries by executingrunBody multiple times
• Log the start and end times of each test, along with thepath of that test,e.g.Seq("outer", "inner1", "innerest") for the testouter.inner1.innerest

Generally, if you want to perform messy before/after logic around everyindividual test, overrideutestWrap. Please remember to call theutestBeforeEach andutestAfterEach methods when needed.

runBody is a future to support asynchronous testing, which is the only way totest things like Ajax calls inScala.js

uTest is completely compatible with Scala.js and Scala-Native: the above sections ondefining a test suite, asserts and the test-running API all work unchanged underScalaJS, with minor differences:

• ScalaJS does not support parallelism, and as such only single-threadedExecutionContexts likeutest.ExecutionContext.runNow orscala.scalajs.concurrent.JSExecutionContext.runNow work. When run via SBT,--parallel has no effect.
• Eventually and Continually are not supported,as they rely on a blocking retry-loop whereas you can't block in ScalaJS.

Apart from these differences, there should be no problem compiling uTestTestSuites via Scala.js and running them on Node.js, in the browser, or (withScala-Native) on LLVM.

Note that Scala-Native support, like Scala-Native, is experimental. While it istested in a few projects (uTest's own test suite runs in Scala-Native), it doeshave it's own quirks (e.g.NullPointerExceptions appear to be fatal) and doesnot have the weight of third-party usage that the Scala-JVM and Scala.jsversions of uTest have.

uTest exposes a straightforward API that allows you to run tests, receiveresults and format them in a nice way when used standalone, without using SBT'stest integration. The following code snippet demonstrates how to define testsand run them directly:

importutest._valtests=Tests{  test("test1"){// throw new Exception("test1")  }  test("test2"){    test("inner"){1    }  }  test("test3"){vala=List[Byte](1,2)// a(10)  }}// Run and return resultsvalresults1=TestRunner.run(tests)

TheTestRunner.run call can be customized with additional arguments to controlhow the code is run, or can be replaced with.runAsync to handle asynchronoustesting, or.runAndPrint/.runAndPrintAsync if you want to print the resultsof tests as they complete using the normal output formatter:

// Run, return results, and print streaming output with the default formattervalresults2=TestRunner.runAndPrint(  tests,"MyTestSuiteA")// Run, return results, and print output with custom formatter and executorvalresults3=TestRunner.runAndPrint(  tests,"MyTestSuiteA",  executor=new utest.framework.Executor{overridedefutestWrap(path:Seq[String],runBody:=>Future[Any])                 (implicitec:ExecutionContext):Future[Any]= {      println("Getting ready to run"+ path.mkString("."))      utestBeforeEach()      runBody.andThen {case _=> utestAfterEach()      }    }  },  formatter=new utest.framework.Formatter{overridedefformatColor=false  })

Lastly, you can also runTestSuite objects in the same way:

// Run `TestSuite` object directly without using SBT, and use// its configuration for execution and output formattingobjectMyTestSuiteextendsTestSuite{valtests=Tests{    test("test1"){// throw new Exception("test1")    }    test("test2"){      test("inner"){1      }    }    test("test3"){vala=List[Byte](1,2)// a(10)    }  }}valresults4=TestRunner.runAndPrint(MyTestSuite.tests,"MyTestSuiteB",  executor=MyTestSuite,  formatter=MyTestSuite)

uTest provides helpers to render the standard summary reports at the end of atest run, once all your results are in:

// Show summary and exitval (summary, successes, failures)=TestRunner.renderResults(Seq("MySuiteA"-> results1,"MySuiteA"-> results2,"MySuiteA"-> results3,"MySuiteB"-> results4  ))if (failures>0) sys.exit(1)

You can thus use uTest anywhere you can run Scala code, even when not using SBT:in a normalmain method, withinAmmonite scripts, orelsewhere.

uTest aims to give you the things that everyone needs to run tests, giving youone obvious way to do common tasks while testing. uTest is simple and compact,letting you avoid thinking about it so you can focus on what's most important:you tests.

Hence uTest provides:

• A simple, uniform syntax fordelimiting tests and grouping tests together as blocks:test("test"){ ... }

• A simple, uniform syntax forrunning tests:foo.BarTests,foo.BarTests.baz.qux,foo.BarTests.{baz,qux} orfoo.{BarTests,BazTests}

• A simple, uniform syntax forSmart Asserts, that save youthe day-to-day drudgery of printing out the value of variables when thingsfail

• Isolation of tests within the same suite,to avoid inter-test interference due to mutable state

uTest tries to provide things that every developer needs, in their minimal,essential form. It intentionally avoids redundant/unnecessary features orsyntaxes that bloat the library and make it harder to developers to pick up,which I find to be common in other popular testing libraries likeScalatest orSpecs2:

• Fluent English-like code: matchers likeshouldBe orshould not beormustbe_==don't really add anything, and it doesn't really matter whether you name eachtest block usingshould,when,can,must,feature("...") orit should "..."

• Multiple redundant waysof defining test suites, individual tests and blocks of related tests

• Legacy code, like ScalaTeststimepackage, nowobsolete with the introduction ofscala.concurrent.duration.

While uTest has and will continue to slowly grow and add more features, it isunlikely that it will ever reach the same level of complexity that other testinglibraries are currently at.

• Added support forassertGoldenFolder#414

• Fix pathological performance of stringdiff algorithm on large assertion diffs#412
• Fixes for crashes in named pattern matches

• Fix pathological performance indef splitLines affecting assertion diffs#409
• Bump PPrint version to fixassertGoldenValue code-generation forHashMap,HashSet, andArraySeq#408

• Don't truncate large data in golden testing#399

• uTest now uses a vendored version ofPPrintto print out the value of local variables found during assertion errors. This includes properformatting, syntax highlighting, and a diff of the two values whenever ana == b equalitycheck fails:

• Golden testing is now supported viaassertGoldenLiteral andassertGoldenFile. This allows uTest to help you fill in the"expected" value of simple assertions the first time you run the test, and keepthat value up to date as the behavior of your code evolves. SeeGolden Literal Testing in uTest 0.9.0for more details

• Renamings: these are a breaking changes that will cause some inconvenience for peopleupgrading, but should result in a much more consistent user experience going forward

  • compileError is nowassertCompileError
  • eventually is nowassertEventually
  • continually is nowassertContinually
  • intercept is nowassertThrows

• Test suites can now beclasses rather thanobjects.objects are still supported for backwardscompatibility, but usingclasses provide better scoping and encapsulation andis the recommended style going forward#173

• Fixed a lot of old bugs and tickets#334#319#252#233#219#190#97

• More fixes for failed assertion line numbers#382

• More fixes for failed assertion line numbers#381

• Fix line numbers in failed assertion traces in Scala 3#380

• RunFuture tests sequentially, not concurrently#359
• Minimum version of Java bumped from 8 to 11

• Avoid crashing if test logs have invalid ANSI escape codes#344

• Support for Scala-Native 0.5.0

• Fix compiler warning when usingutest.framework.TestPath#309

• Add++ toTests so that test suites can be concatenated
• Add.prefix(name: String) toTests to nest all of its tests under a single test group with a given name

• Drop support for Scala.js 0.6
• Bump Scala.js to 1.10 (minimum version supported is 1.8)
• Bump Scala versions to latest (2.12.16, 2.13.8, 3.1.3)

• Add support for Scala 3 on Scala Native

• Add support for Scala 3.0.0

• Add support for Scala 3.0.0-RC3
• Bump Scala.js from 1.4.0 to 1.5.1

• Add support for Scala 3.0.0-RC2
• Support Scala 3 on Scala.js

• Re-publish to maven central only. The version 0.7.6 broke binary compatibilitywith JDK 8.

• Support for Scala-Native 0.4.x

• Add support for Scala.js 1.0.0

• Add support for Scala.js 1.0.0-RC2

• Add support for Scala.js 1.0.0-RC1

• Changed test syntax totest("foo"){...},test{...}. Old syntax is nowdeprecated.

• Added support for Scala 2.13.0 Final
• Dropped support for Scala 2.10.x, 2.11.x
• Temporarily dropped support for Scala.js 1.0.0-M8, Scala-native 0.4.0-M2

• Add support for Scala 2.13.0-RC1
• Use IndexedSeq instead of Array inTests macro

• Add support for Scala 2.13.0-M5
• Upgrade Scala 2.13.0-M2 to 2.13.0-M3
• Upgrade Scala.JS 0.6.22 to 0.6.25
• Upgrade Scala.JS 1.0.0-M3 to 1.0.0-M5
• Upgrade Scala Native 0.3.6 to 0.3.7
• Replace Scala.JS' deprecatedTestUtils with portable-scala-reflect

• Bugfix where sometimes all tests would pass but report as failed (thanks @eatkins)
• By default, don't cut-off and wrap output
• (Hopefully) fix intermittentIllegalStateException: Unknown opcode 5 exceptions occuring with Scala.JS.

• Returningnull from a test case no longer blows up
• Addedutest.runner.MillFramework

• Added support for Scala.js 1.0.0-M2.

• Fix cross-publishing for Scala.js, which was borked in 0.6.0
• Ensure utestAfterEach be executed regardless of a test failure

• Migrate formatting-related configuration options fromutest.TestSuite ontotheutest.runner.Framework, as describedhere.

• Added theexceptionStackFrameHighlighter formatting hook, letting you choosewhich stack frames in an exception are of interest to you so they can berendered more brightly.

• Added hooks forRunning code before and after test casesandRunning code before and after test suites,thanks toDiego Alvarez

• Cross-publish for Scala 2.13.0-M2, Scala.js 0.6.20, Scala-Native 0.3.3

• Stack traces for chained exceptions (i.e. those with a.getCause != null)are now properly displayed when tests fail

• Portions of stack traces caused by the internals of theassert macros arenow hidden, since they aren't relevant to any failure in user code

• Revamped test output format, motivated bydrhumlen'sPR 113, which should be mucheasier to read and skim

• Much smarter test-output-value truncation, now based on lines-of-output(including wrapping) rather than number-of-characters

• Line-wrapping of output is now indentation aware: it wraps to the next linewith the same indentation, preserving the outline of indentation on the leftmaking it easier to skim

• How long tests take (im milliseconds) is now displayed in the standard testoutput format

• Hierarchical test summary and failure-summary are now only shown when you havea large number of tests, since it's not use when running individual or smallnumbers of tests. The default threshold is 20, which can be configured bydefining a custom framework overridingshowSummaryThreshold

• Revamped test-query system, now allowing you to run multiple groups of testsat once viatest-only -- mypackage.{foo,bar,baz} or{mypackage, myotherpackage} ormypackage.{foo,bar.{baz,qux}}

• Overhauled the execution model of test suites: now only the inner-most blockswithin yourTestSuite{...} block count as tests, and the surrounding blocksdo not. Thus the surrounding blocks no longer show pass/fail status, return atest result-value, or get run independently.

• Various user errors (non-existent test, non-existing test suite, invalid testquery syntax) now give slightly more friendly error messages

• Using uTest withfork in Test := true in SBT no longer gives an incorrectresults summmary

• Fix problem with lazy vals in test blocks crashing the compiler#67. Note that the issue is onlyfixed on 2.12.3, and not on Scala 2.10.x or 2.11.x.

• Deprecated the'foo{...} syntax for defining tests. We should standardize ontest("foo"){...} ortest("foo"){...}

• TheTestSuite{...} for defining a block of tests is now deprecated; it neveractually defined aTestSuite object, so the name was pretty misleading. Thenew syntax isTests{...}

• Remove theMultipleErrors wrapper when you test more than one expression inanassert and multiple of them fail. Now only the first expression in theassert call which fails will have it's exception thrown.

• Movedsmart asserts from being embedded into theTestSuiteobjects directly onto theutest package, so they are pulled in by theimport utest._ import.

• Scala Native support.

• Scala 2.13 support
• Compile for Scala 2.12 with method optimisations (-opt:l:method)

• Upgrade Scala.JS to 0.6.16.
• Upgrade Scala to 2.11.11 and 2.12.2.
• Avoid usinggetStackTraceString deprecated in Scala 2.13

• Catch Fatal exceptions like ClassCasts in Scala.JS.

• Scala 2.12 support

• Generalize==> asserts to work onArrays
• Shiny newCI build andGitter Chat

• Move errors intoutest.* fromutest.framework.*
• Removedacyclic hack variable

• Fix usage of by-name function calls within tests #55

• foo: @Show annotation lets you tell uTest to print out arbitrary expressionswithin the test suite when things fail, in addition to the default of localvariables

• You can usea ==> b to assert equality within a test suite, in a form that'spretty enough to use as documentation

• compileError now properly passes when an expression would fail to compiledue to@compileTimeOnly annotations

• Configuring uTest has been overhauled.

• Scala2.12.0-M3 support

• Fix some warnings appearing when the-Ydead-code flag is used

• AddedTestPath implicit to make the path to the current testavailable for usage inside the test body or helpers.

• Published for Scala.js 0.6.1

• Published for Scala.js 0.6.0
• RemovedJsCrossBuild now that Scala.js supports cross-building viacrossProject
• compileTimeOnly has been re-introduced, so invalid use of test DSL shouldfail with nice errors
• Removed--throw flag in favor of "native" SBT error reporting

• Added support for asynchronous tests which return aFuture.

• Updated to work against ScalaJS 0.5.4

• IntroducedCompileError.check(pos: String, msgs: String*) to simplify thecommon pattern of checking that the error occurs in the right place and withthe message you expect.
• Changed the file layout expected byJsCrossBuild, to expect the shared filesto be injs/shared/ andjvm/shared/, rather than inshared/. This istypically achieved via symlinks, and should make the cross-build play muchmore nicely with IDEs.

• Fix bug inutestJsSettings pulling in the wrong version of uTest

• Introduced thecompileError macro to allow testing of compilation errors.
• Stack traces are now only shown for the user code, with the uTest/SBT internalstack trace ignored, making them much less spammy and noisy.
• Introduced the* symbol, which can be used in place of a test name to getsequentially numbered test names.

• ScalaJS version is now built against ScalaJS 0.5.3
• Fixed linking errors in ScalaJS version, to allow proper operation of the newoptimization

• Fixed bug causing local-defs in assert macro to fail

• Extracted oututestJvmSettings andutestJsSettings for use outside theJsCrossBuild plugin, for people who don't want to use the plugin.

• Print paths of failing tests after completion to make C&P-ing re-runs moreconvenient