Repository files navigation

Command-line tool that checks correctness of yourMarkdowndocumentation's code examples. Parses<!-- !test command --> annotationspreceding code blocks, runs them, and checks that the outputs match.

• Only uses HTML comments
  ^{The annotations aren't rendered. You retain formattingcontrol.}
• Works with any programming language
  ^{You choose the shell command(s). Many languages in the same docare OK.}
• Helpful failure diagnostics
  ^{Colours (optional), diffs, line numbers, exit code, stderr,invisible characters, etc.}
• Parallel tests on multi-core machines
  ^{Configurable. Result output ordering remains constant.}
• TAP format output
  ^{The standard supported by many testing tools.}

1. Write aREADME.md, with comment annotations:

   #console.logThe[console.log][1] function in[Node.js][2] stringifies the given argumentsand writes them to`stdout`, followed by a newline.  For example:<!-- !test program node--><!-- !test in simple example-->console.log('a')console.log(42)console.log([1, 2, 3])The output is:<!-- !test out simple example-->a42[ 1, 2, 3 ][1]:https://nodejs.org/api/console.html#console_console_log_data_args[2]:https://nodejs.org/

   See§Use for more detail on how annotations work. Fenced codeblocks delimited by``` work too. Language tags also.

2. Run:

   $ txm README.md

3. See output:

   

   TAP version 131..1ok 1 simple example# 1/1 passed# OK

Examples of other use-cases:

Demonstrating that the root directory is a directory:<!-- !test program node --input-type=module--><!-- !test in example-->import { stat } from 'fs/promises'console.log((await stat('/')).isDirectory())<!-- !test out example-->true

<!-- !test programcat > /tmp/program.cgcc /tmp/program.c -o /tmp/test-program && /tmp/test-program-->Here is a simple example C program that computes the answer to life, theuniverse, and everything:<!-- !test in printf-->#include <stdio.h>int main () {    printf("%d\n", 6 * 7);}<!-- !test out printf-->42

<!-- !test program 2>&1 node--><!-- !test in print to both stdout and stderr-->console.error("This goes to stderr!")console.log("This goes to stdout!")<!-- !test out print to both stdout and stderr-->This goes to stderr!This goes to stdout!

<!-- !test program node || true--><!-- !test in don't fail-->console.log("Hi before throw!")throw new Error("AAAAAA!")<!-- !test out don't fail-->Hi before throw!

<!-- !test program node--><!-- !test check laws of mathematics-->const assert = require('assert')assert(1 + 1 == 2)

As you may be suspecting, this readme is itself tested with txm. All of theabove examples run as part of the automatic tests, locally andon the CIserver.If you want to see the comment annotations,see the readmesource.(It's a little trippy, because txm is recursively running itself.)

To install for current directory's project:npm install txm
To install globally:npm install -g txm

RequiresNode.js (minimum version tested iscurrent LTS).

• filename: Input file (default: read fromstdin)

• --jobs: How many tests may run in parallel. (default:os.cpus().length)

  When a test finishes, txm will only print its output after allearlier-defined tests have printed their outputs, so that results appear inthe same order tests were defined. Further tests continue to run in thebackground, regardless of how many results are pending print.

• --version

• --help

HTML comments that start with!test are read specially. Use a separatecomment for each annotation.

• !test program <program>

  The<program> is run as a shell command for each following matchinginput/output pair. It gets the input onstdin, and is expected to producethe output onstdout. The program may be as many lines as you like; afull shell script if you wish.

  The declared program is used for all tests after here, until a new programis declared.

• !test in <name> /!test out <name> /!test err <name>

  The next code block is read as the input to give to a program for the test<name>, or expected stdout or stderr of the test<name>. These arematched by<name>, and may be anywhere in relation to each other.

  Errors are raised if a test has no input (in) or no output (out norerr), or if it has duplicates of any.

• !test check <name>

  The next code block is read as a check test. The program gets this asinput, but its output is ignored. The test will pass if the program exitssuccessfully. (With exit code0, or that specified in a!test exitcommand prior.)

  Use this for code examples that check their own correctness, for example bycalling anassert function.

• !test exit <code>

  Thenext test which is fully read is expected to fail, and to exit withthe givencode, instead of the default0.

  You can use!test exit nonzero to accept any non-0 exit code.

• !test only

  If any test has this command in front of it, all tests without it areskipped. (They don't run, and their output is suppressed.)

  This is intended for developer convenience: When you have lots of tests ofwhich only a few are failing, you can use this command to focus on them, soother tests don't waste time running or clutter your screen.

txm exits0 if and only if all tests pass.

In diff additions and deletions,C0 Control Characters (suchas Null, Line Feed, or Space), which are ordinarily invisible, are shown as thecorrespondingUnicode Control Picture. These take the formof small diagonally arranged letters, so Null becomes ␀, Line Feed becomes ␊,and Space becomes ␠. This is the standard way to show this set of invisiblecharacters.

Whenever such characters are used, an index will be present in the accompanyingtest data, listing what original character each picture corresponds to, withits name, C escape, and Unicode code point. This is intended to give as muchinformation as possible, because bugs relating to invisible characters areawkward to debug.

If an invisible character is not part of the diff, it is shown normally(without a Control Picture replacement.)

To maintain line breaks, the Line Feed character is kept as-is, with itsControl Picture (␊) added at the end of the line for clarity.

Invisible characters that aren't part of the C0 set are shown as-is. Examplesinclude the zero-width space, or right-to-left text flow marker.

Coloured output is automatically enabled when outputting directly to acolour-capable console interface, and disabled otherwise. It can be forced onor off with the environment variablesNO_COLOR=1 orFORCE_COLOR=1, or withthe options--no-color or--color.

Stripping colour codes from coloured output does not change its logicalmeaning, and indeed the same text is emitted regardless of whether colour isenabled. The colours do not themselves carry meaning; they're just hints toguide the eye.

TheHTML spec regarding comments has a few restrictionson what comments may contain:

the text must not start with the string>, nor start with the string->,nor contain the strings<!--,-->, or--!>, nor end with the string<!-.

Some of those are valid constructs in some programming languages, which can berestrictive if you're writing a!test program command in one of thoselanguages.

Luckily all of them involve hyphens (-), so to work around "forbidden"character sequences, txm lets you optionally escape hyphens inside HTMLcomments:#- is automatically replaced by-. So for example,<!-- !test in -#-> --> is legal HTML, and will be parsed by txm as the command!test in -->.

To write literally#-, write##- instead, and so on.# acts normallyeverywhere else, and doesn't need to be escaped.

For advanced use, your test program sees the same environment variables thattxm sees, plus these introduced by txm:

• TXM_INDEX (1-based number of test)
• TXM_NAME (name of test)
• TXM_INDEX_FIRST,TXM_INDEX_LAST (indexes of first and last tests thatwill be run)
• TXM_INPUT_LANG (thelanguage identifier of theinput/check markdown code block, if any)
• TXM_HAS_COLOUR,TXM_HAS_COLOR (both set to1 if outputting with coloursenabled, or to0 if disabled; they are logically equivalent, just alternatespellings)

You can use these for example to descriptively name log files, or to easilydetect languages and test them differently.

You can also tell with the colour variables whether txm is doing colouredoutput or not, and have your program emit debug output with ANSI colour codesby your method of choice. This is probably only reasonable to do for theoutput ofcheck tests, which output is shown unmodified. Don't do this forin/out tests unless youreally know what you're doing; the colour codesused by txm's automatic diffing will interfere, and you'll get garbage.

The name txm stands for "tests ex markdown" as in "deus ex machina", ortemptamentum ex Markdown I guess if you're feeling extra Latin.

ISC