Repository files navigation

BTest is a powerful framework for writing system tests. Freelyborrowing some ideas from other packages, its main objective is toprovide an easy-to-use, straightforward driver for a suite ofshell-based tests. Each test consists of a set of command lines thatwill be executed, and success is determined based on their exitcodes.btest comes with some additional tools that can be usedwithin such tests to robustly compare output against a previously establishedbaseline.

This document describes BTest 1.2-4. See theCHANGESfile in the source tree for version history.

BTest has the following prerequisites:

• Python version >= 3.9 (older versions may work, but are not well-tested).
• Bash. Note that on FreeBSD and Alpine Linux, bash is not installed bydefault. This is also required on Windows, in the form of Git's msys2, Cygwin,etc.

BTest has the following optional prerequisites to enable additionalfunctionality:

• Sphinx. Sphinx functionality is currently disabled on Windows.
• perf (Linux only). Note that on Debian/Ubuntu, you also need to installthe "linux-tools" package.

When running BTest on Windows, you must have a bash shell installed of somesort. This can be from WSL, Cygwin, msys2, Git, or any number of other methods,butbash.exe must be available. BTest will check for its existence atstartup and exit if it is not available.

A minor change must be made to any configuration value that is a path list. Forexample, if you are setting thePATH environment variable from yourbtest.cfg. In these cases, you should use$(pathsep)s in the configurationinstead of bare: or; values to separate the paths. This ensures thatboth POSIX and Windows systems handle the path lists correctly.

Installation is simple and standard viapip:

> pip install btest

Alternatively, you can download a tarballfrom PyPIand install locally:

> tar xzvf btest-*.tar.gz> cd btest-*> python3 setup.py install

The same approach also works on a local git clone of the source tree,located athttps://github.com/zeek/btest.

Each will install a few scripts:btest is the main driver program,and there are a number of further helper scripts that we discuss below(includingbtest-diff, which is a tool for comparing output to apreviously established baseline).

A BTest testsuite consists of one or more "btests", executed by thebtest driver. Btests are plain text files in whichbtestidentifies keywords with corresponding arguments that tell it what todo. BTest isnot a language; it recognizes keywords in any textfile, including when embedded in other scripting languages. A commonidiom in BTest is to use keywords to process the btest file via aparticular command, often a script interpreter. This approach feelsunusal at first, but lends BTest much of its flexibility: btestfiles can contain pretty much anything, as long asbtestidentifies keywords in it.

btest requires aconfiguration file. With it, you can runbtest on an existing testsuite in several ways:

• Point it at directories containing btests:

  > btest ./testsuite/

• Use the config file to enumerate directories to scan for tests,via theTestDirsoption:

  > btest

• Run btests selectively, by pointingbtest at a specific test file:

  > btest ./testsuite/my.test

  More detail on this when we covertest selection.

In the most simple case,btest simply executes a set of commandlines, each of which must be prefixed with the@TEST-EXEC:keyword:

> cat examples/t1@TEST-EXEC: echo "Foo" | grep -q Foo@TEST-EXEC: test -d .> btest examples/t1examples.t1 ... ok

The test passes as both command lines return success. If one of themdidn't, that would be reported:

> cat examples/t2@TEST-EXEC: echo "Foo" | grep -q Foo@TEST-EXEC: test -d DOESNOTEXIST> btest examples/t2examples.t2 ... failed

Usually you will just run all tests found in a directory:

> btest examplesexamples.t1 ... okexamples.t2 ... failed1 test failed

The file containing the test can simultaneously act asits input. Let'ssay we want to verify a shell script:

> cat examples/t3.sh# @TEST-EXEC: sh %INPUTls /etc | grep -q passwd> btest examples/t3.shexamples.t3 ... ok

Here,btest executes (something similar to)shexamples/t3.sh, and then checks the return value as usual. Theexample also shows that the@TEST-EXEC keyword can appearanywhere, in particular inside the comment section of anotherlanguage.

Now, let's say we want to verify the output of a program, making surethat it matches our expectations---a common use case for BTest. To dothis, we rely on BTest's built-in support for test baselines. Thesebaselines record prior output of a test, adding support forabstracting away brittle details such as ever-changing timestamps orhome directories. BTest comes with tooling to establish, update, andverify baselines, and to plug in "canonifiers": scripts thatabstract, or "normalize", troublesome detail from a baseline.

In our test, we first add a command line that produces the output wewant to check, and then runbtest-diff to make sure it matches thepreviously recorded baseline.btest-diff is itself just a scriptthat returns success if the output matches a pre-recorded baselineafter applying any required normalizations.

In the following example, we use an awk script as a fancy way to print allfile names starting with a dot in the user's home directory. Wewrite that list into a file calleddots and then check whetherits content matches what we know from last time:

> cat examples/t4.awk# @TEST-EXEC: ls -a $HOME | awk -f %INPUT >dots# @TEST-EXEC: btest-diff dots/^\.+/ { print $1 }

Note that each test gets its own little sandbox directory when run,so by creating a file likedots, you aren't cluttering upanything.

The first time we run this test, we need to record a baseline. Thebtest command includes a baseline-update mode, set via-U,that achieves this:

> btest -U examples/t4.awk

btest-diff recognizes this update mode via an environment variableset bybtest, and records thedots file in a separate baselinefolder. With this baseline in place, modifications to the output nowtrigger a test failure:

> btest examples/t4.awkexamples.t4 ... ok> touch ~/.NEWDOTFILE> btest examples/t4.awkexamples.t4 ... failed1 test failed

If we want to see what exactly changed indots to trigger thefailure,btest allows us to record the discrepancies via adiagnostics mode that records them in a file called.diag:

> btest -d examples/t4.awkexamples.t4 ... failed% 'btest-diff dots' failed unexpectedly (exit code 1)% cat .diag== File ===============================[... current dots file ...]== Diff ===============================--- /Users/robin/work/binpacpp/btest/Baseline/examples.t4/dots2010-10-28 20:11:11.000000000 -0700+++ dots      2010-10-28 20:12:30.000000000 -0700@@ -4,6 +4,7 @@.CFUserTextEncoding.DS_Store.MacOSX+.NEWDOTFILE.Rhistory.Trash.Xauthority=======================================% cat .stderr[... if any of the commands had printed something to stderr, that would follow here ...]

Once we delete the new file, the test passes again:

> rm ~/.NEWDOTFILE> btest -d examples/t4.awkexamples.t4 ... ok

That's the essence of the functionality thebtest packageprovides. This example did not use canonifiers. We cover these,and a number of additional options that extend or modify this basicapproach, in the following sections.

btest must be started with a list of tests and/or directoriesgiven on the command line. In the latter case, the default is torecursively scan the directories and assume all files found to betests to perform. It is however possible to exclude specific files anddirectories by specifying a suitableconfiguration file.

btest returns exit code 0 if all tests have successfully passed,and 1 otherwise. Exit code 1 can also result in case of other errors.

btest accepts the following options:

Specifics ofbtest's execution can be tuned with a configurationfile, which by default isbtest.cfg if that's found in thecurrent directory. It can alternatively be specified with the--config command line option, or aBTEST_CFG environmentvariable. The configuration file is"INI-style", and an example comes with the distribution, seebtest.cfg.example. A configuration file has one main section,btest, that defines most options; as well as an optional sectionfor definingenvironment variables and further optional sectionsfor definingalternatives.

Note that all paths specified in the configuration file are relativetobtest'sbase directory. The base directory is either theone where the configuration file is located if such is given/found,or the current working directory if not. One can also override itexplicitly by setting the environment variableBTEST_TEST_BASE.When setting values forconfiguration options, the absolute path to the base directory isavailable by using the macro%(testbase)s (the weird syntax isdue to Python'sConfigParser class).

Furthermore, all values can use standard "backtick-syntax" toinclude the output of external commands (e.g., xyz=`echo test`).Note that the backtick expansion is performed after any%(..)have already been replaced (including within the backticks).

There is a special section that can be added to the configuration file that willset default values to be used during the parsing of other configurationdirectives. For example:

[DEFAULT]val=abcd[environment]ENV_VALUE=%(val)s

The configuration parser reads the keys and values from the DEFAULT sectionprior to reading the other sections. It uses those keys to replace the%()smacros as described earlier. The values stored in these keys can be overriddenat runtime by using the-s/--set command-line argument. For example tooverride theval default above, the-s val=other argument can bepassed. In that case,ENV_VALUE would be set toother instead ofabcd.

The following options can be set in thebtest section of theconfiguration file:

BaselineDir

One or more directories where to store the baseline files forbtest-diff (note that the actual baseline files will be placedinto test-specific subdirectories of this directory). By default,this is set to%(testbase)s/Baseline.

If multiple directories are to be used, they must be separated bycolons.btest-diff will then search them for baseline files inorder when looking for a baseline to compare against. Whenupdating a baseline, it will always store the new version insidethe first directory. Using multiple directories is most useful incombination withalternatives to support alternate executionswhere some tests produce expected differences in their output.

This option can also be set through an environment variableBTEST_BASELINE_DIR.

CommandPrefix

Changes the naming of allbtest commands by replacing the@TEST- prefix with a custom string. For example, withCommandPrefix=$TEST-, the@TEST-EXEC command becomes$TEST-EXEC.

Finalizer

A command that will be executed each time any test hassuccessfully run. It runs in the same directory as the test itselfand receives the name of the test as its only argument. The returnvalue indicates whether the test should indeed be consideredsuccessful. By default, there's no finalizer set.

IgnoreDirs

A space-separated list of relative directory names to ignorewhen scanning test directories recursively. Default is empty.

An alternative way to ignore a directory is placing a file.btest-ignore in it.

IgnoreFiles

A space-separated list of filename globs matching files toignore when scanning given test directories recursively.Default is empty.

An alternative way to ignore a file is by placing@TEST-IGNOREin it.

Initializer

A command that will be executed before each test. It runs inthe same directory as the test itself will and receives the nameof the test as its only argument. The return value indicates whetherthe test should continue; if false, the test will be consideredfailed. By default, there's no initializer set.

MinVersion

On occasion, you'll want to ensure that the version ofbtestrunning your testsuite includes a particular feature. By settingthis value to a given version number (as reported bybtest--version),btest installations older than this version willfail test execution with exit code 1 and a corresponding errormessage on stderr.

PartFinalizer

A command that will be executed each time a testpart hassuccessfully run. This operates similarly toFinalizer exceptthat it runs after each test part rather than only at completionof the full test. Seeparts for more about test parts.

PartInitializer

A command that will be executed before each testpart. This operatessimilarly toInitializer except that it runs at the beginning of anytest part that BTest runs. Seeparts for more about test parts.

Since a failing test part aborts execution of the test, part initializers donot run for any subsequent skipped parts.

PartTeardown

A command that will run after any testpart that has run, regardlessof failure or success of the part. This operates similarly toTeardownexcept it applies to testparts instead of the full test.

Since a failing test part aborts execution of the test, part teardowns donot run for any subsequent skipped parts.

PerfPath

Specifies a path to theperf tool, which is used on Linux tomeasure the execution times of tests. By default, BTest searchesforperf inPATH.

PortRange

Specifies a port range like "10000-11000" to use in conjunction with@TEST-PORT commands. Port assignments will be restricted to thisrange. The default range is "1024-65535".

StateFile

The name of the state file to record the names of failing tests. Default is.btest.failed.dat.

Teardown

A command that will be executed each time any test has run, regardless ofwhether that test succeeded. Conceptually, it pairs with anInitializerthat sets up test infrastructure that requires tear-down at the end of thetest. It runs in the same directory as the test itself and receives the nameof the test as its only argument. There's no default teardown command.

Teardown commands may return a non-zero exit code, which fails thecorresponding test. Succeeding teardown commands do not override anotherwise failing test; such tests will still fail.

To allow teardown routines to reason about the preceding tests, theyreceive two additional environment variables:

TEST_FAILED

This variable is defined (to 1) when the test has failed, and absentotherwise.

TEST_LAST_RETCODE

This variable contains the numeric exit code of the last commandrun prior to teardown.

TestDirs

A space-separated list of directories to search for tests. Ifdefined, one doesn't need to specify any tests on the commandline.

TimingBaselineDir

A directory where to store the host-specifictiming baselinefiles. By default, this is set to%(testbase)s/Baseline/_Timing.

TimingDeltaPerc

A value defining thetiming deviation percentage that's toleratedfor a test before it's considered failed. Default is 1.0 (which meansa 1.0% deviation is tolerated by default).

TmpDir

A directory where to create temporary files when running tests.By default, this is set to%(testbase)s/.tmp.

A special sectionenvironment defines environment variables thatwill be propagated to all tests:

[environment]CFLAGS=-O3PATH=%(testbase)s/bin:%(default_path)s

Note howPATH can be adjusted to include local scripts: theexample above prefixes it with a localbin/ directory inside thebase directory, using the predefineddefault_path macro to referto thePATH as it is set by default.

Furthermore, by settingPATH to include thebtestdistribution directory, one could skip the installation of thebtest package.

BTest can run a set of tests with different settings than it wouldnormally use by specifying analternative configuration. Currently,three things can be adjusted:

• Further environment variables can be set that will then beavailable to all the commands that a test executes.
• Filters can modify an input file before a test uses it.
• Substitutions can modify command lines executed as part of atest.

We discuss the three separately in the following. All of them aredefined by adding sections[<type>-<name>] where<type>corresponds to the type of adjustment being made and<name> is thename of the alternative. Once at least one section is defined for aname, that alternative can be enabled by BTest's--alternativeflag.

An alternative can add further environment variables by defining an[environment-<name>] section:

[environment-myalternative]CFLAGS=-O3

Runningbtest with--alternative=myalternative will now maketheCFLAGS environment variable available to all commandsexecuted.

Prefixing the name of an environment variable with- in an alternativesection removes the respective variable from the environment:

[environment-myalternative]-CFLAGS=

It is an error to provide a value when prefixing with-.

As a special case, one can override two specific environmentvariables---BTEST_TEST_BASE andBTEST_BASELINE_DIR---inside analternative's environment section to have them not only be passed onto child processes, but also apply to thebtest process itself.That way, one can switch to a different base and baseline directoriesfor an alternative.

Filters are a transparent way to adapt the input to a specific testcommand before it is executed. A filter is defined by adding a section[filter-<name>] to the configuration file. This section must haveexactly one entry, and the name of that entry is interpreted as thename of a command whose input is to be filtered. The value of thatentry is the name of a filter script that will be run with twoarguments representing input and output files, respectively. Example:

[filter-myalternative]cat=%(testbase)s/bin/filter-cat

Once the filter is activated by runningbtest with--alternative=myalternative, every time a@TEST-EXEC: cat%INPUT is found,btest will first execute (something similar to)%(testbase)s/bin/filter-cat %INPUT out.tmp, and then subsequentlycat out.tmp (i.e., the original command but with the filteredoutput). In the simplest case, the filter could be a no-op in theformcp $1 $2.

NOTE: There are a few limitations to the filter concept currently:

• Filters arealways fed with%INPUT as their firstargument. We should add a way to filter other files as well.
• Filtered commands are only recognized if they are directlystarting the command line. For example,@TEST-EXEC: ls | cat>output would not trigger the example filter above.
• Filters are only executed for@TEST-EXEC, not for@TEST-EXEC-FAIL.

Substitutions are similar to filters, yet they do not adapt the inputbut the command line being executed. A substitution is defined byadding a section[substitution-<name>] to the configuration file.For each entry in this section, the entry's name specifies thecommand that is to be replaced with something else given as its value.Example:

[substitution-myalternative]gcc=gcc -O2

Once the substitution is activated by runningbtest with--alternative=myalternative, every time a@TEST-EXEC executesgcc, that is replaced withgcc -O2. The replacement is simplestring substitution so it works not only with commands but anythingfound on the command line; it however only replaces full words, notsubparts of words.

btest scans a test file for lines containing keywords thattrigger certain functionality. It knows the following keywords:

@TEST-ALTERNATIVE: <alternative>

Runs this test only for the given alternative (seealternative).If<alternative> isdefault, the test executes when BTest runswith no alternative given (which however is the default anyway).

@TEST-COPY-FILE: <file>

Copy the given file into the test's directory before the test isrun. If<file> is a relative path, it's interpreted relativeto the BTest's base directory. Environment variables in<file>will be replaced if enclosed in${..}. This command can begiven multiple times.

@TEST-DOC: <docstring>

Associates a documentation string with the test. These stringsget included into the output of the--documentation option.

@TEST-EXEC: <cmdline>

Executes the given command line and aborts the test if itreturns an error code other than zero. The<cmdline> ispassed to the shell and thus can be a pipeline, use redirection,and any environment variables specified in<cmdline> will beexpanded, etc.

When running a test, the current working directory for allcommand lines will be set to a temporary sandbox (and will bedeleted later).

There are two macros that can be used in<cmdline>:%INPUT will be replaced with the full pathname of the file definingthe test (this file is in a temporary sandbox directory and is a copyof the original test file); and%DIR will be replaced with the fullpathname of the directory where the test file is located (note thatthis is the directory where the original test file is located, notthe directory where the%INPUT file is located). The latter canbe used to reference further files also located there.

In addition to environment variables defined in theconfiguration file, there are further ones that are passed intothe commands:

TEST_BASE

The BTest base directory, i.e., the directory wherebtest.cfg is located.

TEST_BASELINE

A list of directories where the command can save permanentinformation acrossbtest runs. (This is wherebtest-diff stores its baseline inUPDATE mode.)

Multiple entries are separated by colons. If more than oneentry is given, semantics should be to search them in order.(This is wherebtest-diff stores its baseline inUPDATE mode.)

TEST_DIAGNOSTICS

A file where further diagnostic information can be savedin case a command fails (this is also wherebtest-diffstores its diff). If this file exists, then the--diagnostics-all or--diagnostics options will showthis file (for the latter option, only if a command fails).

TEST_MODE

This is normally set toTEST, but will beUPDATEifbtest is run with--update-baseline, orUPDATE_INTERACTIVE if run with--update-interactive.

TEST_NAME

The name of the currently executing test.

TEST_PART

The test part number (seeparts for more about test parts).

NOTE:

If a command returns the special exit code 100, the test isconsidered failed, however subsequent test commands within thecurrent test are still run.btest-diff uses this specialexit code to indicate that no baseline has yet been established.

If a command returns the special exit code 200, the test isconsidered failed and all further tests are aborted.btest-diff uses this special exit code whenbtest is runwith the--update-interactive option and the user chooses toabort the tests when prompted to record a new baseline.

TEST_VERBOSE

The path of a file where the test can record furtherinformation about its execution that will be included withBTest's--verbose output. This is for further trackingthe execution of commands and should generally generateoutput that follows a line-based structure.

@TEST-EXEC-FAIL: <cmdline>

Like@TEST-EXEC, except that this expects the command tofail, i.e., the test is aborted when the return code is zero.

@TEST-GROUP: <group>

Assigns the test to a group of name<group>. By using option-g one can limit execution to all tests that belong to a givengroup (or a set of groups).

@TEST-IGNORE

This is used to indicate that this file should be skipped (i.e., notest commands in this file will be executed). An alternative way toignore files is by using theIgnoreFiles option in the btestconfiguration file.

@TEST-KNOWN-FAILURE

Marks a test as known to currently fail. This only changes BTest'soutput, which upon failure will indicate that that is expected; itwon't change the test's processing otherwise. The keyword doesn'ttake any arguments but one could add a descriptive text, as in

.. @TEST-KNOWN-FAILURE: We know this fails because ....

@TEST-MEASURE-TIME

Measures execution time for this test and compares it to apreviously establishedtiming baseline. If it deviates significantly,the test will be considered failed.

@TEST-NOT-ALTERNATIVE: <alternative>

Ignores this test for the given alternative (seealternative).If<alternative> isdefault, the test is ignored if BTest runswith no alternative given.

@TEST-PORT: <env>

Assign an available TCP port number to an environment variablethat is accessible from the running test process.<env> is anarbitrary user-chosen string that will be set to the next availableTCP port number. Availability is based on checking successfulbinding of the port on IPv4 INADDR_ANY and also restricted to therange specified by thePortRange option. IPv6 is not supported.Note that using the-j option to parallelize execution willwork such that unique/available port numbers are assigned betweenconcurrent tests, however there is still a potential race conditionfor external processes to claim a port before the test actuallyruns and claims it for itself.

@TEST-REQUIRES: <cmdline>

Defines a condition that must be met for the test to be executed.The given command line will be run before any of the actual testcommands, and it must return success for the test to continue. Ifit does not return success, the rest of the test will be skippedbut doing so will not be considered a failure of the test. This allows towrite conditional tests that may not always make sense to run, dependingon whether external constraints are satisfied or not (say, whethera particular library is available). Multiple requirements may bespecified and then all must be met for the test to continue.

@TEST-SERIALIZE: <set>

When using option-j to parallelize execution, all tests thatspecify the same serialization set are guaranteed to runsequentially.<set> is an arbitrary user-chosen string.

@TEST-START-FILE <file>

This is used to include an additional input file for a testright inside the test file. All lines following the keyword linewill be written into the given file until a line containing@TEST-END-FILE is found. The lines containing@TEST-START-FILEand@TEST-END-FILE, and all lines in between, will be removed fromthe test's %INPUT. Example:

> cat examples/t6.sh# @TEST-EXEC: awk -f %INPUT <foo.dat >output# @TEST-EXEC: btest-diff output    { lines += 1; }END { print lines; }@TEST-START-FILE foo.dat123@TEST-END-FILE> btest -D examples/t6.shexamples.t6 ... ok  % cat .diag  == File ===============================  3

Multiple such files can be defined within a single test.

Note that this is only one way to use further input files.Another is to store a file in the same directory as the testitself, making sure it's ignored viaIgnoreFiles, and thenrefer to it via%DIR/<name>.

@TEST-START-NEXT

This keyword lets you define multiple test inputs in thesame file, all executing with the same command lines. Seedefining multiple tests in one file for details.

Internally,btest uses logical names for tests, abstracting inputfiles. Those names result from substituting path separators with dots,ignoring btest file suffixes, and potentially adding additionallabeling.btest does this only for tests within theTestDirsdirectories given in the configuration file.

In addition to the invocations covered inRunning BTest, you canuse logical names when tellingbtest which tests to run. Forexample, instead of saying

> btest testsuite/foo.sh

you can use:

> btest testsuite.foo

This distinction rarely matters, but it's something to be aware ofwhendefining multiple tests in one file, which we cover next.

On occasion you want to use the same constellation of keywords on aset of input files. BTest supports this via the@TEST-START-NEXTkeyword. Whenbtest encounters this keyword, it initiallyconsiders the input file to end at that point, and runs all@TEST-EXEC-* with an%INPUT truncated accordingly.Afterwards, it creates anew%INPUT with everythingfollowingthe@TEST-START-NEXT marker, running thesame commandsagain. (It ignores any@TEST-EXEC-* lines later in the file.)

The effect is that a single file can define multiple tests that thebtest output will enumerate:

> cat examples/t5.sh# @TEST-EXEC: cat %INPUT | wc -c >output# @TEST-EXEC: btest-diff outputThis is the first test input in this file.# @TEST-START-NEXT... and the second.> ./btest -D examples/t5.shexamples.t5 ... ok  % cat .diag  == File ===============================  119  [...]examples.t5-2 ... ok  % cat .diag  == File ===============================  22  [...]

btest automatically generates the-<n> suffix for each of the tests.

NOTE: It matters how you name tests when running themindividually. When you specify the btest file ("examples/t5.sh"),btest will run all of the contained tests. When you use thelogical name,btest will run only that specific test: in theabove scenario,examples.t5 runs only the first test definedin the file, whileexamples.t5-2 only runs the second. Thisalso applies to baseline updates.

One can also split a single test across multiple files by adding anumerical#<n> postfix to their names, where each<n>represents a separate part of the test.btest will combine all ofa test's parts in numerical order and execute them subsequently withinthe same sandbox. Example:

> cat examples/t7.sh#1# @TEST-EXEC: echo Part 1 - %INPUT >>output> cat examples/t7.sh#2# @TEST-EXEC: echo Part 2 - %INPUT >>output> cat examples/t7.sh#3# @TEST-EXEC: btest-diff output> btest -D examples/t7.shexamples.t7 ... ok% cat .diag== File ===============================Part 1 - /Users/robin/bro/docs/aux/btest/.tmp/examples.t7/t7.sh#1Part 2 - /Users/robin/bro/docs/aux/btest/.tmp/examples.t7/t7.sh#2

Note howoutput contains the output of botht7.sh#1 andt7.sh#2,however in each case%INPUT refers to the corresponding part. Forthe first part of a test, one can also omit the#1 postfix in the filename.

btest-diff has the capability to filter its input through anadditional script before it compares the current version with thebaseline. This can be useful if certain elements in an output areexpected to change (e.g., timestamps). The filter can thenremove/replace these with something consistent. To enable suchcanonification, set the environment variableTEST_DIFF_CANONIFIER to a script reading the original versionfrom stdin and writing the canonified version to stdout.For examples of canonifier scripts, take a look at thoseused in theZeek distribution.

NOTE:btest-diff passes both the pre-recorded baseline andthe fresh test output through any canonifiers before comparingtheir contents. BTest version 0.63 introduced two changes inbtest-diff's baseline handling:

• btest-diff now records baselines in canonicalized form. Thebenefit here is that by canonicalizing upon recording, you canusebtest -U more freely, keeping expected noise out ofrevision control. The downside is that updates to canonifiersrequire a refresh of the baselines.
• btest-diff now prefixes the baselines with a header thatwarns against manual modification, and knows to exclude thatheader from comparison. We recommend only ever updatingbaselines viabtest -U (or its interactive sibling,-u).

Once you use canonicalized baselines in your project, it's a goodidea to useMinVersion = 0.63 in your btest.cfg to avoid theuse of olderbtest installations. Since these are unaware ofthe new baseline header and repeated application of canonifiersmay cause unexpected alterations to already-canonified baselines,using such versions will likely cause test failures.

btest baselines usually consist of text files, i.e. content thatmostly makes sense to process line by line. It's possible to usebinary data as well, though. For such data,btest-diff supports abinary mode in which it will treat the baselines as binary "blobs". Inthis mode, it will compare test output to baselines for byte-by-byteequality only, it will never apply any canonifiers, and it will leavethe test output untouched during baseline updates.

To use binary mode, invokebtest-diff with the--binary flag.

Sometimes processes need to be spawned in the background for a test,in particular if multiple processes need to cooperate in some fashion.btest comes with two helper scripts to make life easier in such asituation:

btest-bg-run <tag> <cmdline>

This is a script that runs<cmdline> in the background, i.e.,it's like usingcmdline & in a shell script. Test executioncontinues immediately with the next command. Note that the spawnedcommand isnot run in the current directory, but instead in anewly created sub-directory called<tag>. This allowsspawning multiple instances of the same process without needing toworry about conflicting outputs. If you want to access a command'soutput later, like withbtest-diff, use<tag>/foo.log toaccess it.

btest-bg-wait [-k] <timeout>

This script waits for all processes previously spawned viabtest-bg-run to finish. If any of them exits with a non-zeroreturn code,btest-bg-wait does so as well, indicating afailed test.<timeout> is mandatory and gives the maximumnumber of seconds to wait for any of the processes to terminate.If any process hasn't done so when the timeout expires, it will bekilled and the test is considered to be failed as long as-kis not given. If-k is given, pending processes are stillkilled but the test continues normally, i.e., non-termination isnot considered a failure in this case. This script also collectsthe processes' stdout and stderr outputs for diagnostics output.

For long-running tests it can be helpful to display progress messagesduring their execution so that one sees where the test is currentlyat. There's a helper script, btest-progress, to facilitate that. Thescript receives a custom message as its sole argument. When executedwhile a test is running,btest will display that message in real-timein its standard and verbose outputs.

Example usage:

# @TEST-EXEC: bash %INPUTbtest-progress Stage 1sleep 1btest-progress Stage 2sleep 1btest-progress Stage 3sleep 1

When the tests execute,btest will then show these three messagessuccessively. By default,btest-progress also prints the messagesto the test's standard output and standard error. That can be suppressed byadding an option-q to the invocation.

btest can time execution of tests and report significantdeviations from past runs. As execution time is inherentlysystem-specific it keeps separate per-host timing baselines for that.Furthermore, as time measurements tend to make sense only forindividual, usually longer running tests, they are activated on pertest basis by adding a@TEST-MEASURE-TIME directive. The testwill then execute as usual yet also record the duration for which itexecutes. After the timing baselines are created (with the--update-timesoption), further runs on the same host will compare their times against thatbaseline and declare a test failed if it deviates by more than, bydefault, 1%. (To tune the behaviour, look at theTiming*options.)If a test requests measurement but BTest can't find a timing baselineor the necessary tools to perform timing measurements, then it willignore the request.

As timing for a test can deviate quite a bit even on the same host,BTest does not actually measuretime but the number of CPUinstructions that a test executes, which tends to be more stable.That however requires the right tools to be in place. On Linux, BTestleveragesperf. By default, BTestwill search forperf in thePATH; you can specify a differentpath to the binary by settingPerfPath inbtest.cfg.

btest comes with an extension module for the documentation frameworkSphinx. The extension module provides twonew directives calledbtest andbtest-include. Thebtestdirective allows writing a test directly inside a Sphinx document, andthen the output from the test's command is included in the generateddocumentation. Thebtest-include directive allows for literal textfrom another file to be included in the generated documentation.The tests from both directives can also be run externally and will catchif any changes to the included content occur. The following walksthrough setting this up.

First, you need to tell Sphinx a base directory for thebtestconfiguration as well as a directory in there where to store testsit extracts from the Sphinx documentation. Typically, you'd justcreate a new subdirectorytests in the Sphinx project for thebtest setup and then store the tests in there in, e.g.,doc/:

> cd <sphinx-root>> mkdir tests> mkdir tests/doc

Then add the following to your Sphinxconf.py:

extensions += ["btest-sphinx"]btest_base="tests"         # Relative to Sphinx-root.btest_tests="doc"          # Relative to btest_base.

Next, create abtest.cfg intests/ as usual and adddoc/ to theTestDirs option. Also, add a finalizer tobtest.cfg:

[btest]...PartFinalizer=btest-diff-rst

Thebtest extension provides a new directive to include a testinside a Sphinx document:

.. btest:: <test-name>    <test content>

Here,<test-name> is a custom name for the test; it will bestored inbtest_tests under that name (with a file extension of.btest).<test content> is just a standard test as you wouldnormally put into one of theTestDirs. Example:

.. btest:: just-a-test    @TEST-EXEC: expr 2 + 2

When you now run Sphinx, it will (1) store the test content intotests/doc/just-a-test.btest (assuming the above path layout), and (2)execute the test by runningbtest on it. You can then runbtest manually intests/ as well and it will execute the testjust as it would in a standard setup. If a test fails when Sphinx runsit, there will be a corresponding error and include the diagnostic outputinto the document.

By default, nothing else will be included into the generateddocumentation, i.e., the above test will just turn into an empty textblock. However,btest comes with a set of scripts that you can useto specify content to be included. As a simple example,btest-rst-cmd <cmdline> will execute a command and (if itsucceeds) include both the command line and the standard output intothe documentation. Example:

.. btest:: another-test    @TEST-EXEC: btest-rst-cmd echo Hello, world!

When running Sphinx, this will render as:

# echo Hello, world!Hello, world!

The same<test-name> can be used multiple times, in which caseeach entry will become one part of a joint test.btest willexecute all parts subsequently within a single sandbox, and earlierresults will thus be available to later parts.

When runningbtest manually intests/, thePartFinalizer weadded tobtest.cfg (see above) compares the generated reST codewith a previously established baseline, just likebtest-diff doeswith files. To establish the initial baseline, runbtest -u, likeyou would withbtest-diff.

The following Sphinx support scripts come withbtest:

btest-rst-cmd [options] <cmdline>

By default, this executes<cmdline> and includes both thecommand line itself and its standard output into the generateddocumentation (but only if the command line succeeds).See above for an example.

This script provides the following options:

btest-rst-include [options] <file>

Includes<file> inside a code block. The<file> must be createdby a previous@TEST-EXEC or@TEST-COPY-FILE.

This script provides the following options:

btest-rst-pipe <cmdline>

Executes<cmdline>, includes its standard output inside a codeblock (but only if the command line succeeds). Note thatthis script does not include the command line itself into the codeblock, just the output.

NOTE: All these scripts can be run directly from the commandline to show the reST code they generate.

NOTE:btest-rst-cmd can do everything the other scriptsprovide if you give it the right options. In fact, the otherscripts are provided just for convenience and leveragebtest-rst-cmd internally.

Thebtest Sphinx extension module also provides a directivebtest-include that functions likeliteralinclude (including allits options) but also creates a test checking the included content forchanges. As one further extension, the directive expands environmentvariables of the form${var} in its argument. Example:

.. btest-include:: ${var}/path/to/file

When you now run Sphinx, it will automatically generate a testfile in the directory specified by thebtest_tests variable inthe Sphinxconf.py configuration file. In this example, the filenamewould beinclude-path_to_file.btest (it automatically adds a prefix of"include-" and a file extension of ".btest"). When you runthe tests externally, the tests generated by thebtest-includedirective will check if any of the included content has changed (you'llfirst need to runbtest -u to establish the initial baseline).

BTest is open-source under a BSD license.