Repository files navigation

A regression testing framework

This project contains a generic, cross-platform regression testframework.

The generic regression test framework is implemented in Go whereasthe test are written as shell scripts. The library code is in thertdirectory and some common utilities and helper programs are contained intheutils directory.

• rtf - a local test runner

For more details, see the documentation in./docs/USER_GUIDE.md.

go get -u github.com/linuxkit/rtf

To run the test suite please use:

make test

On a Mac and Linux, this should pretty much just work out of thebox. The tests are written insh and expect it to be installed in/bin/sh. Some optional utilities (see./bin) are writteninpython and if your tests use them, you need to have pythoninstalled.

On Windows, it depends on how your tests are written. If they arewritten aspowershell scripts,rtf should just work. If they arewritten as shell scripts, you need to have the MSYS2 variant ofbashinstalled andbash.exe must be in your path. The simplest way toinstall it is to installgit viachocolatey. Note, neitherbash from WSLnor cygwin is currently supported.

choco install git

If your tests use the optional utilities in, you also need to installpython:

choco install python

The regression test framework allows running tests on a local host (orinside a VM) as well as against a suitably configured remote host.

If you don't have the source code in yourGOPATH, you may have toset theRT_ROOT environment variable to point to it.

To run tests locally, simply execute thertf run command. It willexecuted all the test cases in the supplied cases directory. Thisdefaults to./cases

To list all current tests runrtf list, or to get a one linesummary for each test usertf info.

When running tests, by default a line per test is printed on theconsole with a pass/fail indication. Detailed logs, by default, arestored in./_results/<UUID>/. In that directory,TESTS.logcontains detailed logs of all tests,TESTS.csv contains a line pertest,SUMMARY.csv contains a one line summary of the all tests run,andSUMMARY.json contains both a test summary and the individualtest results. The directory also contains a log file for each tests,with the same contents asTESTS.log.

If you prefer a bit more information in the log files use:

rtf -v run -x

This executes the tests with-x, logging all commands executed tostderr;and with-v, causingstderr to be displayed to the console.

For a CI system, where the output is displayed on a web page use:

rtf -vvv run -x

This prints the same information logged to the log file to the console.

There is initial support for comparing the result from two test runs:

rtf compare <path to SUMMARY.json> <path to SUMMARY.json> ...

which will display the results side by side.