The Kernel Test Anything Protocol (KTAP), version 1

TAP, or the Test Anything Protocol is a format for specifying test results usedby a number of projects. Its website and specification are found at thislink. The Linux Kernel largely uses TAP output for testresults. However, Kernel testing frameworks have special needs for test resultswhich don’t align with the original TAP specification. Thus, a “Kernel TAP”(KTAP) format is specified to extend and alter TAP to support these use-cases.This specification describes the generally accepted format of KTAP as it iscurrently used in the kernel.

KTAP test results describe a series of tests (which may be nested: i.e., testcan have subtests), each of which can contain both diagnostic data -- e.g., loglines -- and a final result. The test structure and results aremachine-readable, whereas the diagnostic data is unstructured and is there toaid human debugging.

KTAP output is built from four different types of lines:

  • Version lines

  • Plan lines

  • Test case result lines

  • Diagnostic lines

In general, valid KTAP output should also form valid TAP output, but someinformation, in particular nested test results, may be lost. Also note thatthere is a stagnant draft specification for TAP14, KTAP diverges from this ina couple of places (notably the “Subtest” header), which are described whererelevant later in this document.

Version lines

All KTAP-formatted results begin with a “version line” which specifies whichversion of the (K)TAP standard the result is compliant with.

For example:

  • “KTAP version 1”

  • “TAP version 13”

  • “TAP version 14”

Note that, in KTAP, subtests also begin with a version line, which denotes thestart of the nested test results. This differs from TAP14, which uses aseparate “Subtest” line.

While, going forward, “KTAP version 1” should be used by compliant tests, itis expected that most parsers and other tooling will accept the other versionslisted here for compatibility with existing tests and frameworks.

Plan lines

A test plan provides the number of tests (or subtests) in the KTAP output.

Plan lines must follow the format of “1..N” where N is the number of tests or subtests.Plan lines follow version lines to indicate the number of nested tests.

While there are cases where the number of tests is not known in advance -- inwhich case the test plan may be omitted -- it is strongly recommended one ispresent where possible.

Test case result lines

Test case result lines indicate the final status of a test.They are required and must have the format:

<result> <number> [<description>][ # [<directive>] [<diagnostic data>]]

The result can be either “ok”, which indicates the test case passed,or “not ok”, which indicates that the test case failed.

<number> represents the number of the test being performed. The first test musthave the number 1 and the number then must increase by 1 for each additionalsubtest within the same test at the same nesting level.

The description is a description of the test, generally the name ofthe test, and can be any string of characters other than # or anewline. The description is optional, but recommended.

The directive and any diagnostic data is optional. If either are present, theymust follow a hash sign, “#”.

A directive is a keyword that indicates a different outcome for a test otherthan passed and failed. The directive is optional, and consists of a singlekeyword preceding the diagnostic data. In the event that a parser encountersa directive it doesn’t support, it should fall back to the “ok” / “not ok”result.

Currently accepted directives are:

  • “SKIP”, which indicates a test was skipped (note the result of the test caseresult line can be either “ok” or “not ok” if the SKIP directive is used)

  • “TODO”, which indicates that a test is not expected to pass at the moment,e.g. because the feature it is testing is known to be broken. While thisdirective is inherited from TAP, its use in the kernel is discouraged.

  • “XFAIL”, which indicates that a test is expected to fail. This is similarto “TODO”, above, and is used by some kselftest tests.

  • “TIMEOUT”, which indicates a test has timed out (note the result of the testcase result line should be “not ok” if the TIMEOUT directive is used)

  • “ERROR”, which indicates that the execution of a test has failed due to aspecific error that is included in the diagnostic data. (note the result ofthe test case result line should be “not ok” if the ERROR directive is used)

The diagnostic data is a plain-text field which contains any additional detailsabout why this result was produced. This is typically an error message for ERRORor failed tests, or a description of missing dependencies for a SKIP result.

The diagnostic data field is optional, and results which have neither adirective nor any diagnostic data do not need to include the “#” fieldseparator.

Example result lines include:

ok 1 test_case_name

The test “test_case_name” passed.

not ok 1 test_case_name

The test “test_case_name” failed.

ok 1 test # SKIP necessary dependency unavailable

The test “test” was SKIPPED with the diagnostic message “necessary dependencyunavailable”.

not ok 1 test # TIMEOUT 30 seconds

The test “test” timed out, with diagnostic data “30 seconds”.

ok 5 check return code # rcode=0

The test “check return code” passed, with additional diagnostic data “rcode=0”

Diagnostic lines

If tests wish to output any further information, they should do so using“diagnostic lines”. Diagnostic lines are optional, freeform text, and areoften used to describe what is being tested and any intermediate results inmore detail than the final result and diagnostic data line provides.

Diagnostic lines are formatted as “# <diagnostic_description>”, where thedescription can be any string. Diagnostic lines can be anywhere in the testoutput. As a rule, diagnostic lines regarding a test are directly before thetest result line for that test.

Note that most tools will treat unknown lines (see below) as diagnostic lines,even if they do not start with a “#”: this is to capture any other usefulkernel output which may help debug the test. It is nevertheless recommendedthat tests always prefix any diagnostic output they have with a “#” character.

Unknown lines

There may be lines within KTAP output that do not follow the format of one ofthe four formats for lines described above. This is allowed, however, they willnot influence the status of the tests.

This is an important difference from TAP. Kernel tests may print messagesto the system console or a log file. Both of these destinations may containmessages either from unrelated kernel or userspace activity, or kernelmessages from non-test code that is invoked by the test. The kernel codeinvoked by the test likely is not aware that a test is in progress andthus can not print the message as a diagnostic message.

Nested tests

In KTAP, tests can be nested. This is done by having a test include within itsoutput an entire set of KTAP-formatted results. This can be used to categorizeand group related tests, or to split out different results from the same test.

The “parent” test’s result should consist of all of its subtests’ results,starting with another KTAP version line and test plan, and end with the overallresult. If one of the subtests fail, for example, the parent test should alsofail.

Additionally, all lines in a subtest should be indented. One level ofindentation is two spaces: “ “. The indentation should begin at the versionline and should end before the parent test’s result line.

“Unknown lines” are not considered to be lines in a subtest and thus areallowed to be either indented or not indented.

An example of a test with two nested subtests:

KTAP version 11..1  KTAP version 1  1..2  ok 1 test_1  not ok 2 test_2# example failednot ok 1 example

An example format with multiple levels of nested testing:

KTAP version 11..2  KTAP version 1  1..2    KTAP version 1    1..2    not ok 1 test_1    ok 2 test_2  not ok 1 test_3  ok 2 test_4 # SKIPnot ok 1 example_test_1ok 2 example_test_2

Major differences between TAP and KTAP

Feature

TAP

KTAP

yaml and json in diagnosic message

ok

not recommended

TODO directive

ok

not recognized

allows an arbitrary number of tests to be nested

no

yes

“Unknown lines” are in category of “Anything else”

yes

no

“Unknown lines” are

incorrect

allowed

The TAP14 specification does permit nested tests, but instead of using anothernested version line, uses a line of the form“Subtest: <name>” where <name> is the name of the parent test.

Example KTAP output

KTAP version 11..1  KTAP version 1  1..3    KTAP version 1    1..1    # test_1: initializing test_1    ok 1 test_1  ok 1 example_test_1    KTAP version 1    1..2    ok 1 test_1 # SKIP test_1 skipped    ok 2 test_2  ok 2 example_test_2    KTAP version 1    1..3    ok 1 test_1    # test_2: FAIL    not ok 2 test_2    ok 3 test_3 # SKIP test_3 skipped  not ok 3 example_test_3not ok 1 main_test

This output defines the following hierarchy:

A single test called “main_test”, which fails, and has three subtests:

  • “example_test_1”, which passes, and has one subtest:

    • “test_1”, which passes, and outputs the diagnostic message “test_1: initializing test_1”

  • “example_test_2”, which passes, and has two subtests:

    • “test_1”, which is skipped, with the explanation “test_1 skipped”

    • “test_2”, which passes

  • “example_test_3”, which fails, and has three subtests

    • “test_1”, which passes

    • “test_2”, which outputs the diagnostic line “test_2: FAIL”, and fails.

    • “test_3”, which is skipped with the explanation “test_3 skipped”

Note that the individual subtests with the same names do not conflict, as theyare found in different parent tests. This output also exhibits some sensiblerules for “bubbling up” test results: a test fails if any of its subtests fail.Skipped tests do not affect the result of the parent test (though it oftenmakes sense for a test to be marked skipped if _all_ of its subtests have beenskipped).

See also: