Goals of This Specification

Document the observed behavior of widely used TAP implementations.
Add no features that are not already in wide usage across multipleimplementations.
Explicitly allow what is already allowed, deny what is already denied.
Provide updated and clearer guidance for new TAP implementations.

Name

TAP14 - The Test Anything Protocol v14

Synopsis

TAP, the Test Anything Protocol, is a simple text-based interface betweentesting modules and test harness. TAP started life as part of the testharness for Perl but now has implementations in C/C++, Python, PHP, Perl,JavaScript, and probably others by the time you read this. This documentdescribes version 14 of TAP.

The key wordsmust,must not,required,shall,shall not,should,should not,recommended,may, andoptional in thisdocument are to be interpreted as described inRFC2119.

Changes From TAP13 Format

TAP14 is largely backwards compatible with TAP13. That is, TAP14is designed to be reasonably parseable by any compliant TAP13Harness, and TAP14 Harnesses should be able to reasonablyinterpret the output of TAP13 Producers.

The following changes have been made to the specification, whichare covered in much more detail below.

Change TAP version line to14.
Add child tests as 4-space indented TAP streams, with atrailing test point and leading comment line.
Formalize the following conventions:
- 2-space indentation for YAML diagnostics.
- plans always start at1.
- escaping of\ and# characters.
- prefixing of test point descriptions with" - ".
Increased clarity regarding parsing rules, based on behavior ofextant TAP13 implementations in popular programming languages.

TAP14 Format

TAP14’s general grammar is:

TAPDocument := Version Plan Body | Version Body PlanVersion     := "TAP version 14\n"Plan        := "1.." (Number) (" # " Reason)? "\n"Body        := (TestPoint | BailOut | Pragma | Comment | Anything | Empty | Subtest)*TestPoint   := ("not ")? "ok" (" " Number)? ((" -")? (" " Description) )? (" " Directive)? "\n" (YAMLBlock)?Directive   := " # " ("todo" | "skip") (" " Reason)?YAMLBlock   := "  ---\n" (YAMLLine)* "  ...\n"YAMLLine    := "  " (YAML)* "\n"BailOut     := "Bail out!" (" " Reason)? "\n"Reason      := [^\n]+Pragma      := "pragma " [+-] PragmaKey "\n"PragmaKey   := ([a-zA-Z0-9_-])+Subtest     := ("# Subtest" (": " SubtestName)?)? "\n" SubtestDocument TestPointComment     := ^ (" ")* "#" [^\n]* "\n"Empty       := [\s\t]* "\n"Anything    := [^\n]+ "\n"

Note that the above is intended as a rough “pseudocode” guidance forhumans. It is not strict EBNF. Detailed parsing instructions for eachelement can be found in the sections below.

The Version is the lineTAP version 14.

The Body is a collection of lines representing a test set.

The Plan reports on the number of tests included in the Body.

A Subtest indicates a nested TAP Document that is included by the parentTAP Document. It is a TAP Document where each line is indented by 4 spaces.

For example, a test file’s output might look like:

TAP version 141..4ok1 - Input file openednot ok2 - First line of the input valid  ---  message: 'First line invalid'  severity: fail  data:    got: 'Flirble'    expect: 'Fnible'  ...ok3 - Read the rest of the filenot ok4 - Summarized correctly#TODO Not written yet  ---  message: "Can't make summary yet"  severity: todo  ...

Harness Behavior

In this document, the “harness” is any program analyzing TAP output.

Typically this will be a test framework’s runner program, but may also be aprogrammatic parser, parent test object, or test result reporter.

In a typical TAP implementation, tests are programs that ouput TAP dataaccording to this specification. The “test harness” reads TAP output fromthese programs, and handles it in some way.

A harness that is collecting output from a test programshould read andinterpret TAP from the process’s standard output, not standard error.(Handling of test standard error is implementation-specific.)

A Harnessshould normalize line endings by replacing any instances of\r\n or\r in the TAP document with\n.

A harnessshould treat a test program as a failed test if:

The TAP output lines indicate test failure, or
The TAP output of the process is invalid in a way that is notrecoverable, or
The exit code of the test program is not 0 (including test programskilled by a fatal Unix signal).

If one or more test programs are considered failures, then a TAP Harnessshould indicate failure to the user in whatever means are appropriate.For example, a command-line test runner might exit with a non-zero statuscode, while a web-based continuous integration system might put a red“FAILED” notice in the html output.

Note that some of the above guidance may not apply if the Harness isinterpreting TAP from a different sort of text stream, or for a purposeother than actually running tests. For example, a Harness may be used toanalyze the recorded output of past test runs and provide data about them.

Document Structure

TAP14 producersmust encode TAP data using the UTF-8 encoding.

Harnessesshould interpret all TAP streams using UTF-8. Harnessesmayprovide a mechanism for using other encodings explicitly, if needed forbackwards compatibility.

Version

To indicate that this is TAP14 the first linemust be

TAP version 14

Harnessesmay interpret ostensiblyTAP13 streamsas TAP14, as this specification is compatible with observed behavior ofexisting TAP13 consumers and producers. That is, theymay treat this asa valid Version line while parsing TAP14:

TAP version 13

Harnessesmay treat any TAP stream lacking a version as a failed test.

Plan

The Plan tells how many tests will be run, or how many tests have run. It’sa check that the test file hasn’t stopped prematurely.

The Planmust appear exactly once in the file, either before any TestPoint lines, or after all Test Point lines. That is, if it appears afterany Test Point lines, theremust not be any Test Point lines following.

A Harnessmust treat a TAP stream lacking a plan as a failed test.

The Plan specifies how many test points are to follow. For example,

1..10

means that either 10 test points will follow, or 10 test points arebelieved to have been present in the stream already.

This is a safeguard against test data being truncated or damaged in someother way, rendering the output unreliable.

The Plan lists the range of test point IDs that are expected in the TAPstream. It can also optionally contain a comment/reason prefixed by a#.

Its basic grammar is:

Plan := "1.." Number ("" | "# " Reason)

A plan line of1..0 indicates that the test set was completely skipped;no tests are expected to follow, and none should have come before.Harnessesshould report on1..0 test runs similarly to their handlingofSKIP Test Points, treating any comment in the Plan as the reason forskipping.

1..0# WWW::Mechanize not installed

Previous versions of TAP allowed plans to specify any two numbers, forexample,5..8 to indicate that test points with IDs between 5 and 8 wouldbe run. However, this is not widely supported.

Thus, TAP14 producersmust output a Plan starting with1. TAP14Harnessesmay allow plans starting with numbers other than 1, but if so,theymust treat any Test Point IDs outside the plan range as a testfailure.

# and\ characters may be escaped in Plan reason, and if so,shouldbe unescaped prior to being presented to the user. See “Escaping” below.

Test Points

The core of TAP is the “Test Point”. A test file prints one test pointexecuted. There must be at least one test point in TAP output. Each testpoint comprises the following elements:

In summary:

Test Status:ok/not ok (required)
Test number (recommended)
Description (recommended, prefixed by" - ")
Directive (only when necessary)

Test Status:`ok` or`not ok`

This tells whether the test point passed or failed. It must be at thebeginning of the line./^not ok/ indicates a failed test point./^ok/ is a successful test point. This is the only mandatory part ofthe line.

Note that unlike the Directives below,ok andnot ok arecase-sensitive.

Test Point ID

TAP expects the ok or not ok to be followed by an integer Test PointID. If there is no number, the harnessmust maintain its own counteruntil the script supplies test numbers again.

For example, the following test output is acceptable:

1..5not okoknot okokok

and is equivalent to:

1..5not ok1ok2not ok3ok4ok5

This test output isnot a successful test run:

TAP version 141..6not okoknot okokok

Five tests are shown, but the plan indicated that there would be 6.Furthermore, tests 1 and 3 are explicitly failing. Perl’sTest::Harness will report:

FAILED tests 1, 3, 6Failed 3/6 tests, 50.00% okay

Test Pointsmay be output in any order, but any Test Point IDprovidedmust be within the range described by the Plan.

This is valid TAP and a successful test run:

TAP version 141..3ok2ok3ok1

This is not a successful test run. Even though there are 3 Test Points,the Test Point ID 4 is outside the stated Plan range.

TAP version 141..3ok2ok4ok1

Test Point IDsshould be unique within the TAP Document. Harnessesmaywarn about repeated Test Point IDs or treat them as a test failure, butmust not treat a Test Point with a re-used ID as a non-TAP line.

Description

Any text after the test number but before a# is the description ofthe test point.

ok42 - this is the description of the test

Descriptionsshould be separated from the Test Point Status and TestPoint ID by the string" - ", in order to prevent confusing a numericdescription with a Test Point ID. Harnessesmust treat Test Pointsidentically whether the description starts with" - " or not.

For example, these two test pointsmust be treated identically by aHarness:

ok1 this is fineok1 - this is fine

Harnessesshould not consider a leading" - " to be a part of thedescription reported to a user.

Directive

Directives are special notes that follow the first unescaped# on theTest Point line that is preceded and followed by whitespace. Only two arecurrently defined:TODO andSKIP.

Directives are not case sensitive. That is, Harnessesmust treat#SKIP,# skip, and# SkIp identically.

Harnessesmay support additional platform-specific Directives. Futureversions of this specification may codify additional Directives withdefined semantics.

Unrecognized Directivesmust not be treated as test failure, or aninvalid TAP line. Harnessesshould include any unrecognized directivesin the Test Point description.

Note that escaped# characters are not to be treated as delimiters forDirectives. See “Escaping” below.

Whitespace Around Directive Delimiter

Earlier versions of the TAP specification were not explicit aboutwhitespace requirements regarding directives. The following rules maximizecompatibility between TAP14 producers and harnesses:

Producersmust output directives with at least one space characterpreceding the# in a directive, as well as at least one spacecharacter between the# and the directive name.
Harnessesmust not treat escaped# characters as directivedelimiters.
Harnessesmay accept directive delimiters where the# is notpreceded by whitespace, butshould warn that such output isnon-conformant with the TAP14 specification.

If harnesses choose to parse directives without whitespace before and afterthe#, then they ought to consider the impact if test descriptionscontain URLs and/or may be coming from TAP producers that are not diligentabout escaping# characters. This should be done only to the extentnecessary for backwards compatibility with existing TAP producers.

For example:

TAP version 14# MUST be treated as a SKIP testok1 - must be skipped test#SKIP# MUST NOT be treated as a SKIP testok2 - must not be skipped test \# SKIP# MAY be treated as a SKIP test, but SHOULD warn about itok3 - may skip, but should warn# skipok4 - may skip, but should warn#skipok5 - may skip, but should warn#skip

Backwards Compatibility and Parsing Notes

For backwards compatibility with earlier incarnations of TAP, Harnessesmust accept additional non-whitespace characters following theliteral strings"SKIP" or"TODO". Everything after(TODO|SKIP)\S*\s+is the reason. For example, this is supported, and shows a test with 2skip tests: one with no reason given, and the other with an explanation.

TAP version 141..2# skip: trueok1 - do it later#Skipped# skip: true# skip reason: "only run on windows"ok2 - works on windows#Skipped: only run on windows

For broad compatibility with as many harnesses as possible, as well asfuture-proofing their output, Producersshould always reportSKIP andTODO tests using only the directive names (“SKIP” or “TODO”), optionallyfollowed by a space and a reason. Future versions of this specificationmay drop support for\S* characters following directive names.

Thus, the regular expression for directives is:

/\s+#\s*(SKIP|TODO)\S*\s+([^\n]*)/directive type = $1reason = $2

More examples of parsing directives:

TAP version 14# skip: true# skip reason: "this test is skipped"# description: ""ok1#skip this test isskipped# skip: false# description: "not skipped: https://example.com/page.html\#skip is a url"ok2 not skipped: https://example.com/page.html#skip is a url# skip: true# skip reason: "case insensitive, so this is skipped"# description: ""ok3 -#SkIp case insensitive, so this isskipped

`TODO` tests

If the directive starts with# TODO, the test is counted as a todo test,and any text afterTODO\S*\s+ is the explanation.

not ok14#TODO bend space and time

If the TODO has an explanation, it must be separated from TODO by a space.These tests represent a feature to be implemented or a bug to be fixed andact as something of an executable “things to do” list. They are notexpected to succeed.

Should a todo test point begin succeeding, the harnessmay report it insome way that indicates that whatever was supposed to be done has been, andit should be promoted to a normal Test Point.

Harnessesmust not treat failingTODO test points as a test failure.

Harnesesshould reportTODO test points found as a list of itemsneeding work, if that is appropriate for their use case.

`SKIP` tests

If the directive starts with# SKIP, the test is counted as a skippedtest, and the text afterSKIP\S*\s+ is the explanation.

ok14 - mung the gums#SKIP leave gums unmunged for now

If the SKIP has an explanation, it must be separated from SKIP by a space.These tests indicate that a test was not run, or if it was, that itssuccess or failure is being temporarily ignored.

Harnessesmust not treat failingSKIP test points as a test failure.

Harnessesshould reportSKIP test points found as a list of items thatwere not tested, if that is appropriate for their use case.

YAML Diagnostics

If a Test Point is followed by a 2-space indented block beginning with theline --- and ending with the line ..., separated from the TestPoint only by comments or whitespace, then the block of lines between thesemarkers will be interpreted as an inline YAML Diagnostic document accordingto version 1.2 of theYAML specification.

The YAML encodes a data structure that provides information about thepreceding Test Point.

For example:

not ok3 - Resolve address  ---  message: "Failed with error 'hostname peebles.example.com not found'"  severity: fail  found:    hostname: 'peebles.example.com'    address: ~  wanted:    hostname: 'peebles.example.com'    address: '85.193.201.85'  at:    file: test/dns-resolve.c    line: 142  ...

Currently (March 2022) the data structure represented by a YAML Diagnosticblock has not been standardized. TAP14 Harnessesmust allow any datastructures supported by their YAML parser implementation.

A future version of this specificationmay provide guidance regardingYAML Diagnostic fields in common usage.

Comments

Lines outside of a YAML diagnostic block which begin with a# characterpreceded by zero or more characters of whitespace, are comments.

A Harnessmay present these to the user, ignore them, or assign meaningto certain comments.

A Harnessmust not treat a test as a failure based on the presence ofcomment lines. That is, a Harnessmust ignore any unrecognized commentlines.

Pragmas

A Pragma provides information to a Harness to control its behavior orconfigure it in some way. Each Pragma line represents a single booleanswitch which can be set totrue orfalse.

The structure of a Pragma line is:

"pragma "
+ (true) or- (false)
key: The name of the field being enabled or disabled. ASCII alphanumericcharacters,_, and- are allowed.

For example:

TAP version 14# tell the parser to bail out on any failures from now onpragma +bail# tell the parser to execute in strict mode, treating any invalid TAP# line as a test failure.pragma +strict# turn off a feature we don't want to be active right nowpragma -bail

The meaning and availability of keys that may be set by Pragmas areimplementation-specific.

Harnessesmay choose to respond to Pragma lines, or ignore them.

Harnessesmust not treat unrecognized Pragma keys as a test failure, evenif they would normally treat invalid TAP as a test failure. Harnessesmay warn if a Pragma is unrecognized, or fail if the named pragma isrecognized, but cannot be set for some reason.

Pragmasmust not include Comments, Directives, or other characters otherthan those specified above.

Blank Lines

For the purposes of this specification, a “blank” line is any lineconsisting exclusively of zero or more whitespace characters (ie,/^[ \t]+$/).

Blank lines within YAML blocksmust be preserved as part of the YAMLdocument, because line breaks have semantic meaning in YAML documents. Forexample, multiline folded scalar values use\n\n to denote line breaks.

Blank lines outside of YAML blocksmust be ignored by the Harness.

Bail out!

As an emergency measure a test script can decide that further tests areuseless (e.g. missing dependencies) and testing should stop immediately. Inthat case the test script prints the magic words

Bail out!

to standard output. Any message after these wordsmust be presented bythe Harness as the reason why testing must be stopped. For example:

Bail out! MySQL is not running.

# and\ characters may be escaped inBail out! messages, and if so,should be unescaped prior to being presented to the user. See“Escaping” below.

# reason for stopping: # and \ are not supportedBail out! \# and \\ are not supported

The words “Bail out!” are case insensitive.

Anything Else

Any line that is not a valid version, plan, test point, YAML diagnostic,pragma, a blank line, or a bail out is invalid TAP.

A Harnessmay silently ignore invalid TAP lines, pass them through to itsown stderr or stdout, or report them in some other fashion. However,Harnessesshould not treat invalid TAP lines as a test failure bydefault.

Escaping

Sometimes a user may include a# character in a Test Point description,Plan comment, Bailout reason, or TODO/SKIP reason.

In order to distinguish this from a directive or other sort of comment, the# character may be escaped with a backslash\ character. To include aliteral\ character, a double-\ may be used. No other characters maybe escaped in this way; a\ that precedes any character other than\ or# will be interpreted as a literal\ and included in the result data.

Providersshould escape any# or\ that is present in the Test Pointdescription or directive reason sections.

For example:

// using node-tapconstt=require('tap')t.pass('hello #\\ world',{todo:'escape # characters with\\'})// outputs:// ok 1 - hello \# \\ world # TODO escape \# characters with \\

Harnessesmust:

Treat any\\ as a literal\, but ignore it for the purpose ofescaping a# character.
Treat any\# as a literal#, but ignore it for the purpose ofdelimiting a directive, provided the\ was not escaped.
Treat any unescaped# as a literal# in a description or reasonfield, if doing so would not cause it to be treated as a delimiter.

Escaping Examples

The following are examples of escaping# and\ characters, andincluding unescaped# characters in the Test Point description orTODOreason.

Note that the specific fieldsdescription,todo, andtodo reason arenot normative, and shown for illustration purposes only. Harnessesshould present this data to their consumers in whatever manner isappropriate for their language and context.

TAP version 14# description: hello# todo: trueok1 - hello#todo# description: hello # todo# todo: falseok2 - hello \# todo# description: hello# todo: true# todo reason: hash # characterok3 - hello#todo hash \# character# (assuming "character" isn't a known custom directive)ok4 - hello#todo hash # character# description: hello \# todo: true# todo reason: hash # characterok5 - hello \\# todo hash \# character# (assuming "character" isn't a known custom directive)ok6 - hello \\# todo hash# character# description: hello # description # todo# todo: false# (assuming "description" isn't a known custom directive)ok7 - hello# description #todo# multiple escaped \ can appear in a row# description: hello \\\# todo# todo: falseok8 - hello \\\\\\\# todo1..8

Subtests

Subtests provide a way to nest one TAP14 stream inside another. This isuseful in a variety of situations. For example:

A Harness parses a collection of TAP documents, providing output whichis also in TAP format.

 TAP version 14 1..2 # Subtest: foo.tap     1..2     ok 1     ok 2 - this passed ok 1 - foo.tap # Subtest: bar.tap     ok 1 - object should be a Bar     not ok 2 - object.isBar should return true       ---       found: false       wanted: true       at:         file: test/bar.ts         line: 43         column: 8       ...     ok 3 - object can bar bears # TODO     1..3 not ok 2 - bar.tap   ---   fail: 1   todo: 1   ...

A test framework Producer provides an API for grouping assertions abouta related subject. For example:

importtfrom'tap't.ok(true,'true is ok')t.test('this is a subtest',subtest=>{subtest.pass('this is fine')subtest.fail('this is not fine')subtest.end()})

which produces the output:

 TAP version 14 ok 1 - true is ok # Subtest: this is a subtest     ok 1 - this is fine     not ok 2 - this is not fine     1..2 not ok 2 - this is a subtest 1..2

Subtests are designed with graceful fallback for TAP13 harnesses in mind.

Since TAP13 specifies that non-TAP outputshould be ignored or provideddirectly to the user, and indented Test Points and Plans are non-TAPaccording to TAP13, only the terminating correlated test point will beinterpreted by most TAP13 Harnesses. Thus, they will usually report theoverall subtest result correctly, even if they lack the details about theresults of the subtest.

Since several TAP13 parsers in popular usage treat a repeated Versiondeclaration as an error, even if the Version is indented, Subtestsshouldnot include a Version, if TAP13 Harness compatibility is desirable.

Bare Subtests

In its simplest form, a Subtest is introduced by a 4-space indented validTAP line. That is, a Test Point, Version, Plan, or (if Pragmas aresupported) Pragma, prefixed by 4 space characters.

For example:

TAP version 14    ok 1 - subtest test point    1..1ok1 - subtest passing1..1

Note that, because the Version is optional in subtests, and the plan mayoccur after all test points, the first item in a subtest may be a furthersubtest. Harnesses must thus treat any multiple of 4-space indentation ismultiple levels of nested subtest:

TAP version 14        ok 1 - nested twice        1..1    ok 1 - nested parent    1..1ok1 - double nest passing1..1

The first test point at the parent level is the correlated Test Point forthe subtest, and terminates the Subtest.

Commented Subtests

A Subtest may be introduced with a comment at the parent test indentationlevel, which defines the expected name of the terminating correlated TestPoint.

This comment has the form^# Subtest(: .*)$. Everything after the:is the Subtest Name. For example:

# Subtest: <name>

# Subtest

A Commented Subtest with a Subtest Namemust be terminated by a TestPoint with a matching Description. If no Subtest Name is present, then theterminating Test Pointmust not include a description.

For example:

TAP version 14ok1 - in the parent# Subtest: nested    1..1    ok 1 - in the subtestok2 - nested# Subtest: empty    1..0ok3 - empty# Subtest    ok 1 - name is optional    1..1ok41..4

Commented Subtests are encouraged, as they provide the following benefits:

Easier for humans to read. For example:

  TAP version 14              1..1              ok 1 - hmm, what level is this?

vs:

  TAP version 14  # Subtest: level 1      # Subtest: level 2          # Subtest: level 3              1..1              ok 1 - clearly level 3

Additional strictness around matching the Test Point description toSubtest Name can catch errors and detect accidentally interleaved output.

Subtest Bailouts

Harnessesmust treat a Bailout in a nested Subtest as a bailout for theentire test run.

For backwards compatibility with TAP13 Harnesses, Producersshould emit aBail out! line at the root indentation level whenever a Subtest bailsout.

Subtest Pragmas

Any Pragmas set in a Subtest affectonly the parsing of the Subtest.Harnessesmust not allow Pragmas set in Subtests to affect the behaviorof the parser with respect to the parent TAP Document.

For example, given a Harness where astrict Pragma will cause it to treatany non-TAP as an error:

TAP version 14pragma -strict# Subtest: child test    1..1    pragma +strict    ok 1ok1 - child test!!This is not valid TAP content!!1..1

In this TAP document, the non-TAP contentmust not be treated as a testfailure, because thestrict Pragma setting at the parent level was false.

Subtest Parsing/Generating Rules

The subtest TAP document is indented 4 spaces.
Subtestsmust be a valid TAP document, meaning that they cannot beentirely empty. At minimum, they must include a1..0 line to indicatethat no Test Points are expected. This implies the following:
1. Subtests can be nested within subtests by indenting another 4characters for each level of nesting.
2. YAML Diagnostics are indented 2 spaces with respect to the TestPoint they are associated with. So, for example, a YAML Diagnosticblock for a Test Point in a nested subtest would be indented 6spaces (4 + 2). In a subtest nested within another subtest, YAMLdiagnostics would be indented 10 spaces (4 + 4 + 2).
Subtestsshould not emit a Version line, if compatibility with TAP13Harnesses is desirable.
The subtest is terminated by a single Test Point line in the parent TAPdocument, which follows the indented TAP Document. This is theCorrelated Test Point, and reflects the pass/fail status of the Subtest.
1. Producersmust communicate the intended status of the subtest(pass/fail/todo/skip/etc.) by assigning these semantics to thecorrelated Test Point.
2. Harnessesshould treat the entire subtest as either a pass or failbased on the status of the correlated Test Point, butmay treatthe subtest as a failure if they would consider the nested subtestTAP Document a test failure.
Harnessesshould treat otherwise valid TAP that is indented anythingother than a multiple of 4 spaces as non-TAP.
Providersshould not emit a Version for the Subtest TAP Document.
Providersshould emit a Subtest Comment if the name of the subtest isknown at the outset.
If Subtest Comment is provided, Harnessesshould continue the subtestuntil a matching test point (same name, or no name in either) at parentlevel is found, treating any other un-indented lines as non-TAP, andfail the test if a matching test point is not found.
If Subtest Comment is not provided, Harnessesmust treat the next TestPoint at the parent level as the end of the subtest, and treat anyintervening lines indented less than the subtest level as non-TAP.
Harnessesshould treat unterminated Subtests as non-TAP.
A Bailout in a subtest TAP documentmust abort the entire process,exactly as if it had occurred in the top-level TAP document.
Any lines at the parent indentation level that occur between theSubtest Introduction and a valid Subtest Terminationmust be treatedas non-TAP output.

Examples

All names, places, and events depicted in any example are wholly fictitiousand bear no resemblance to, connection with, or relation to any realentity. Any such similarity is purely coincidental, unintentional, andunintended.

Common with explanation

The following TAP listing declares that six tests follow as well asprovides handy feedback as to what the test is about to do. All six testspass.

TAP version 141..6## Create a new Board and Tile, then place# the Tile onto the board.#ok1 - The object isa Boardok2 - Board size is zerook3 - The object isa Tileok4 - Get possible places to put the Tileok5 - Placing the tile produces no errorok6 - Board size is 1

Unknown amount and failures

This hypothetical test program ensures that a handful of servers are onlineand network-accessible. Because it retrieves the hypothetical servers froma database, it doesn’t know exactly how many servers it will need to ping.Thus, the test count is declared at the bottom after all the test pointshave run. Also, two of the tests fail. The YAML block following eachfailure gives additional information about the failure that may bedisplayed by the harness.

TAP version 14ok1 - retrieving servers from the database# need to ping 6 serversok2 - pinged diamondok3 - pinged rubynot ok4 - pinged saphire  ---  message: 'hostname "saphire" unknown'  severity: fail  ...ok5 - pinged onyxnot ok6 - pinged quartz  ---  message: 'timeout'  severity: fail  ...ok7 - pinged gold1..7

Giving up

This listing reports that a pile of tests are going to be run. However, thefirst test fails, reportedly because a connection to the database could notbe established. The program decided that continuing was pointless andexited.

TAP version 141..573not ok1 - database handleBail out! Couldn't connect to database.

Skipping a few

The following listing plans on running 5 tests. However, our programdecided to not run tests 2 thru 5 at all. To properly report this, thetests are marked as being skipped.

TAP version 141..5ok1 - approved operating system# $^0 is solarisok2 -#SKIP no /sys directoryok3 -#SKIP no /sys directoryok4 -#SKIP no /sys directoryok5 -#SKIP no /sys directory

Skipping everything

This listing shows that the entire listing is a skip. No tests were run.

TAP version 141..0#skip because English-to-French translator isn't installed

Procrastination Considered`ok`

The following example reports that four tests are run and the last twotests failed. However, because the failing tests are marked as things to dolater, they are considered successes. Thus, a harness should report thisentire listing as a success.

TAP version 141..4ok1 - Creating test programok2 - Test program runs, no errornot ok3 - infinite loop#TODO halting problem unsolvednot ok4 - infinite loop 2#TODO halting problem unsolved

Creative liberties

This listing shows an alternate output where the test numbers aren’tprovided. The test also reports the state of a ficticious board game as aYAML block. Finally, the test count is reported at the end.

TAP version 14ok - created Boardokokokokokokok  ---  message: "Board layout"  severity: comment  dump:     board:       - '      16G         05C        '       - '      G N C       C C G      '       - '        G           C  +     '       - '10C   01G         03C        '       - 'R N G G A G       C C C      '       - '  R     G           C  +     '       - '      01G   17C   00C        '       - '      G A G G N R R N R      '       - '        G     R     G        '  ...ok - board has 7 tiles + starter tile1..9

Bugs

Feature requests and bug reports should beraised onGitHub.

Authors

The TAP 14 Specification is authored byIsaac Z.Schlueter, as a result of much discussion ontheTestAnythingSpecification project, withconsiderable input, encouragement, feedback, and suggestions from:

Acknowledgements

Special thanks toJonathan Kingstonfor his efforts to keep TAP flourishing and bring implementors together.

The TAP13 Specification was written by Andy Armstrong with help andcontributions from Pete Krawczyk, Paul Johnson, Ian Langworth and NikClayton, based on the original TAP documentation by Andy Lester, based onthe originalTest::Harness documentation by Michael Schwern.

The basis for the TAP format was created by Larry Wall in the original testscript for Perl 1. Tim Bunce and Andreas Koenig developed it further withtheir modifications toTest::Harness.

Copyright

This specification is released under theArtistic License2.0

github.com/testanything

Movatterモバイル変換

Goals of This Specification

Name

Synopsis

Changes From TAP13 Format

TAP14 Format

Harness Behavior

Document Structure

Version

Plan

Test Points

Test Status:ok ornot ok

Test Point ID

Description

Directive

Whitespace Around Directive Delimiter

Backwards Compatibility and Parsing Notes

TODO tests

SKIP tests

YAML Diagnostics

Comments

Pragmas

Blank Lines

Bail out!

Anything Else

Escaping

Escaping Examples

Subtests

Bare Subtests

Commented Subtests

Subtest Bailouts

Subtest Pragmas

Subtest Parsing/Generating Rules

Examples

Common with explanation

Unknown amount and failures

Giving up

Skipping a few

Skipping everything

Procrastination Consideredok

Creative liberties

Bugs

Authors

Acknowledgements

Copyright

Test Status:`ok` or`not ok`

`TODO` tests

`SKIP` tests

Procrastination Considered`ok`