Posted on • Originally published atbitmaybewise.substack.com
Tsonnet #2 - Cram tests to the rescue
In the previous post, I added the parsing of JSON literals toTsonnet:
There's something important still missing though and it is tests. Better not to proceed without guaranteeing the language specification is being validated by automated tests.
Let's do it!
Adding the testing libraries
We'll addalcotest for writing automated tests -- it seems like the favorite testing library for OCaml programmers nowadays instead ofounit. Let's also addbisect_ppx for coverage report:
diff --git a/dune-project b/dune-projectindex cf8c03f..383bbef 100644--- a/dune-project+++ b/dune-project@@ -25,7 +25,15 @@ (dune (>= 3.16.0)) (menhir- (= 20240715)))+ (= 20240715))+ (alcotest+ (and+ :with-test+ (>= 1.8.0)))+ (bisect_ppx+ (and+ :with-test+ (>= 2.8.3)))) (tags (jsonnet interpreter compiler)))In dune, we can specify that a dependency belongs to a specific environment -- in this case, thetest environment. This way we don't carry testing dependencies to the release binaries.
Runningdune runtest is convenient, but the coverage command isn't. Let's add a Makefile to free our minds of the details:
diff --git a/.gitignore b/.gitignoreindex 69fa449..39136cd 100644--- a/.gitignore+++ b/.gitignore@@ -1 +1,2 @@ _build/+_coverage/diff --git a/Makefile b/Makefilenew file mode 100644index 0000000..688c42d--- /dev/null+++ b/Makefile@@ -0,0 +1,15 @@+default: test++.PHONY: test+test:+ dune runtest++.PHONY: coverage+coverage:+ dune runtest --instrument-with bisect_ppx --force+ bisect-ppx-report html++.PHONY: clean+clean:+ dune clean+ rm -rf _coverage/I appreciate that simply runningmake targets themake test. I always like to have the test command as the default for my projects. Themake coverage command is now convenient to run locally and we can leverage it when setting up a CI job later on.
Now, let's configure dune.
A little bit of configuration
diff --git a/bin/dune b/bin/duneindex 300e220..b274fc4 100644--- a/bin/dune+++ b/bin/dune@@ -2,3 +2,8 @@ (public_name tsonnet) (name main) (libraries tsonnet))++(cram+ (deps+ %{bin:tsonnet}+ (source_tree ../samples)))diff --git a/test/cram/dune b/test/cram/dunenew file mode 100644index 0000000..4c224ef--- /dev/null+++ b/test/cram/dune@@ -0,0 +1,4 @@+(cram+ (deps+ %{bin:tsonnet}+ (source_tree ../../samples)))diff --git a/test/dune b/test/duneindex f8dbe1e..bd42c05 100644--- a/test/dune+++ b/test/dune@@ -1,2 +1,5 @@ (test- (name test_tsonnet))+ (name test_tsonnet)+ (libraries tsonnet alcotest)+ (deps+ (source_tree ../samples)))We are going to specify that we can write cram tests in thebin directory and the tests will have two dependencies:
- The Tsonnet binary that dune will compile for us
- The
samplesdirectory, where we store the Jsonnet file samples, to be used as input in the tests
We are also adding a similar configuration to thetest directory, where we specify that cram tests will live in thetest/cram folder and the binary and sample files dependencies.
The main entry-leveltest folder will have the sample files as a dependency and thealcotest library.
I'll go into more detail about cram tests in the next section.
For the coverage:
diff --git a/lib/dune b/lib/duneindex 9660313..3452131 100644--- a/lib/dune+++ b/lib/dune@@ -1,5 +1,7 @@ (library- (name tsonnet))+ (name tsonnet)+ (instrumentation+ (backend bisect_ppx))) (menhir (modules parser))diff --git a/lib/lexer.mll b/lib/lexer.mllindex a581fcd..585673b 100644--- a/lib/lexer.mll+++ b/lib/lexer.mll@@ -1,4 +1,5 @@ {+ [@@@coverage exclude_file] open Lexing open Parser exception SyntaxError of stringdiff --git a/lib/parser.mly b/lib/parser.mlyindex 6872459..2396dfe 100644--- a/lib/parser.mly+++ b/lib/parser.mly@@ -1,3 +1,7 @@+%{+ [@@@coverage exclude_file]+%}+ %token <int> INT %token <float> FLOAT %token NULLWe need to instrument thebisect_ppx coverage tool to set thelib directory as its target, otherwise, it is going to completely ignore this directory, which is where all our compiler logic lives.
Since the lexer and parser are generated, there's no point in checking coverage for them, so we basically annotate them to be excluded.
Now we are ready to finally write some tests.
Testing with cram tests
Do you know what arecram tests?
TLDR, cram tests are integration tests that describe a shell session. They are amazing to test CLI programs!
Do you know what else is cool? Dune has built-in support for writing cram tests. The reason why no new library has been added to handle that.
We can write runnable documentation with that. Remember that we configured dune to have cram tests in thebin folder? Here are cram tests to cover some usage cases of Tsonnet:
diff --git a/bin/usage.t b/bin/usage.tnew file mode 100644index 0000000..7d13b46--- /dev/null+++ b/bin/usage.t@@ -0,0 +1,10 @@+Using the Tsonnet program:++ $ tsonnet ../samples/literals/int.jsonnet+ 42++ $ tsonnet ../samples/literals/string.jsonnet+ "Hello, world!"++ $ tsonnet ../samples/literals/object.jsonnet+ {"int_attr": 1, "float_attr": 4.200000, "string_attr": "Hello, world!", "null_attr": null, "array_attr": [1, false, {}], "obj_attr": {"a": true, "b": false, "c": {"d": [42]}}}Your cram test files need to have the.t extension.
You can mix free text with properly indented shell commands. The line right below the command is the expected output.
How cool is that?!
In thetest/cram directory we can test each sample source file, along with its expected output:
diff --git a/test/cram/literals.t b/test/cram/literals.tnew file mode 100644index 0000000..c7b1c2a--- /dev/null+++ b/test/cram/literals.t@@ -0,0 +1,36 @@+ $ tsonnet ../../samples/literals/int.jsonnet+ 42++ $ tsonnet ../../samples/literals/float.jsonnet+ 4.222222222222222++ $ tsonnet ../../samples/literals/negative_int.jsonnet+ -42++ $ tsonnet ../../samples/literals/negative_float.jsonnet+ -4.222222222222222++ $ tsonnet ../../samples/literals/true.jsonnet+ true++ $ tsonnet ../../samples/literals/false.jsonnet+ false++ $ tsonnet ../../samples/literals/null.jsonnet+ null++ $ tsonnet ../../samples/literals/string.jsonnet+ "Hello, world!"++ $ tsonnet ../../samples/literals/array.jsonnet+ [ 1, 2.0, "hi", null ]++ $ tsonnet ../../samples/literals/object.jsonnet+ {+ "int_attr": 1,+ "float_attr": 4.2,+ "string_attr": "Hello, world!",+ "null_attr": null,+ "array_attr": [ 1, false, {} ],+ "obj_attr": { "a": true, "b": false, "c": { "d": [ 42 ] } }+ }To finalize, let's run the tests:
$dune runtestThere are no errors when all the tests have passed.
Let's introduce a small error, changing the output value of one or two of the tests, and see how it is presented:
Here's our intentionally introduced error changes:
diff --git a/test/cram/literals.t b/test/cram/literals.tindex c7b1c2a..54d455f 100644--- a/test/cram/literals.t+++ b/test/cram/literals.t@@ -1,8 +1,8 @@ $ tsonnet ../../samples/literals/int.jsonnet- 42+ 666 $ tsonnet ../../samples/literals/float.jsonnet- 4.222222222222222+ 4.23 $ tsonnet ../../samples/literals/negative_int.jsonnet -42The output of the tests:
$dune runtestFile"test/cram/literals.t", line 1, characters 0-0:diff--git a/_build/.sandbox/0a16dd95b0ff6ebafcf09f03744c5c4c/default/test/cram/literals.t b/_build/.sandbox/0a16dd95b0ff6ebafcf09f03744c5c4c/default/test/cram/literals.t.correctedindex 54d455f..c7b1c2a 100644--- a/_build/.sandbox/0a16dd95b0ff6ebafcf09f03744c5c4c/default/test/cram/literals.t+++ b/_build/.sandbox/0a16dd95b0ff6ebafcf09f03744c5c4c/default/test/cram/literals.t.corrected@@-1,8 +1,8 @@$tsonnet ../../samples/literals/int.jsonnet- 666+ 42$tsonnet ../../samples/literals/float.jsonnet- 4.23+ 4.222222222222222$tsonnet ../../samples/literals/negative_int.jsonnet-42It is super easy to spot the expected value and the actual result we got. Isn't it nice?!
Concluding
Cram tests have become one of my favorite tools. I can't live without them if I'm writing a CLI program seriously.
This is not a new idea. The first time I saw something similar wasin Elixir via doctests. I read somewhere thatthis cram Python package is where it originated, but I can't say for sure. The last time I tried Rust was many years ago, but as far as I know,Rust has documentation tests.
I don't know who copied who, but I hope this trend continues. This is a really cool feature that I hope every single programming language implements, either fully as cram tests or partially as documentation tests.
Disclaimer:alcotest is basically sitting idle for now, as there are no unit tests or integration tests besides the cram tests. Eventually, there will be. I just anticipated adding the dependency right away.
What do you think about cram tests?
Thanks for reading Bit Maybe Wise!Subscribe to receive new posts about Tsonnet and cram tests.
Top comments(0)
For further actions, you may consider blocking this person and/orreporting abuse
