Movatterモバイル変換


[0]ホーム

URL:


Skip to content
DEV Community
Log in Create account

DEV Community

Cover image for Tsonnet #2 - Cram tests to the rescue
Hercules Lemke Merscher
Hercules Lemke Merscher

Posted on • Originally published atbitmaybewise.substack.com

     

Tsonnet #2 - Cram tests to the rescue

In the previous post, I added the parsing of JSON literals toTsonnet:

There's something important still missing though and it is tests. Better not to proceed without guaranteeing the language specification is being validated by automated tests.

Let's do it!

Adding the testing libraries

We'll addalcotest for writing automated tests -- it seems like the favorite testing library for OCaml programmers nowadays instead ofounit. Let's also addbisect_ppx for coverage report:

diff --git a/dune-project b/dune-projectindex cf8c03f..383bbef 100644--- a/dune-project+++ b/dune-project@@ -25,7 +25,15 @@   (dune    (>= 3.16.0))   (menhir-   (= 20240715)))+   (= 20240715))+  (alcotest+   (and+    :with-test+    (>= 1.8.0)))+  (bisect_ppx+   (and+    :with-test+    (>= 2.8.3))))  (tags   (jsonnet interpreter compiler)))
Enter fullscreen modeExit fullscreen mode

In dune, we can specify that a dependency belongs to a specific environment -- in this case, thetest environment. This way we don't carry testing dependencies to the release binaries.

Runningdune runtest is convenient, but the coverage command isn't. Let's add a Makefile to free our minds of the details:

diff --git a/.gitignore b/.gitignoreindex 69fa449..39136cd 100644--- a/.gitignore+++ b/.gitignore@@ -1 +1,2 @@ _build/+_coverage/diff --git a/Makefile b/Makefilenew file mode 100644index 0000000..688c42d--- /dev/null+++ b/Makefile@@ -0,0 +1,15 @@+default: test++.PHONY: test+test:+       dune runtest++.PHONY: coverage+coverage:+       dune runtest --instrument-with bisect_ppx --force+       bisect-ppx-report html++.PHONY: clean+clean:+       dune clean+       rm -rf _coverage/
Enter fullscreen modeExit fullscreen mode

I appreciate that simply runningmake targets themake test. I always like to have the test command as the default for my projects. Themake coverage command is now convenient to run locally and we can leverage it when setting up a CI job later on.

Now, let's configure dune.

A little bit of configuration

diff --git a/bin/dune b/bin/duneindex 300e220..b274fc4 100644--- a/bin/dune+++ b/bin/dune@@ -2,3 +2,8 @@  (public_name tsonnet)  (name main)  (libraries tsonnet))++(cram+ (deps+  %{bin:tsonnet}+  (source_tree ../samples)))diff --git a/test/cram/dune b/test/cram/dunenew file mode 100644index 0000000..4c224ef--- /dev/null+++ b/test/cram/dune@@ -0,0 +1,4 @@+(cram+ (deps+  %{bin:tsonnet}+  (source_tree ../../samples)))diff --git a/test/dune b/test/duneindex f8dbe1e..bd42c05 100644--- a/test/dune+++ b/test/dune@@ -1,2 +1,5 @@ (test- (name test_tsonnet))+ (name test_tsonnet)+ (libraries tsonnet alcotest)+ (deps+  (source_tree ../samples)))
Enter fullscreen modeExit fullscreen mode

We are going to specify that we can write cram tests in thebin directory and the tests will have two dependencies:

  1. The Tsonnet binary that dune will compile for us
  2. Thesamples directory, where we store the Jsonnet file samples, to be used as input in the tests

We are also adding a similar configuration to thetest directory, where we specify that cram tests will live in thetest/cram folder and the binary and sample files dependencies.

The main entry-leveltest folder will have the sample files as a dependency and thealcotest library.

I'll go into more detail about cram tests in the next section.

For the coverage:

diff --git a/lib/dune b/lib/duneindex 9660313..3452131 100644--- a/lib/dune+++ b/lib/dune@@ -1,5 +1,7 @@ (library- (name tsonnet))+ (name tsonnet)+ (instrumentation+  (backend bisect_ppx))) (menhir  (modules parser))diff --git a/lib/lexer.mll b/lib/lexer.mllindex a581fcd..585673b 100644--- a/lib/lexer.mll+++ b/lib/lexer.mll@@ -1,4 +1,5 @@ {+  [@@@coverage exclude_file]   open Lexing   open Parser   exception SyntaxError of stringdiff --git a/lib/parser.mly b/lib/parser.mlyindex 6872459..2396dfe 100644--- a/lib/parser.mly+++ b/lib/parser.mly@@ -1,3 +1,7 @@+%{+  [@@@coverage exclude_file]+%}+ %token <int> INT %token <float> FLOAT %token NULL
Enter fullscreen modeExit fullscreen mode

We need to instrument thebisect_ppx coverage tool to set thelib directory as its target, otherwise, it is going to completely ignore this directory, which is where all our compiler logic lives.

Since the lexer and parser are generated, there's no point in checking coverage for them, so we basically annotate them to be excluded.

Now we are ready to finally write some tests.

 Testing with cram tests

Do you know what arecram tests?

TLDR, cram tests are integration tests that describe a shell session. They are amazing to test CLI programs!

Do you know what else is cool? Dune has built-in support for writing cram tests. The reason why no new library has been added to handle that.

We can write runnable documentation with that. Remember that we configured dune to have cram tests in thebin folder? Here are cram tests to cover some usage cases of Tsonnet:

diff --git a/bin/usage.t b/bin/usage.tnew file mode 100644index 0000000..7d13b46--- /dev/null+++ b/bin/usage.t@@ -0,0 +1,10 @@+Using the Tsonnet program:++  $ tsonnet ../samples/literals/int.jsonnet+  42++  $ tsonnet ../samples/literals/string.jsonnet+  "Hello, world!"++  $ tsonnet ../samples/literals/object.jsonnet+  {"int_attr": 1, "float_attr": 4.200000, "string_attr": "Hello, world!", "null_attr": null, "array_attr": [1, false, {}], "obj_attr": {"a": true, "b": false, "c": {"d": [42]}}}
Enter fullscreen modeExit fullscreen mode

Your cram test files need to have the.t extension.

You can mix free text with properly indented shell commands. The line right below the command is the expected output.

How cool is that?!

In thetest/cram directory we can test each sample source file, along with its expected output:

diff --git a/test/cram/literals.t b/test/cram/literals.tnew file mode 100644index 0000000..c7b1c2a--- /dev/null+++ b/test/cram/literals.t@@ -0,0 +1,36 @@+  $ tsonnet ../../samples/literals/int.jsonnet+  42++  $ tsonnet ../../samples/literals/float.jsonnet+  4.222222222222222++  $ tsonnet ../../samples/literals/negative_int.jsonnet+  -42++  $ tsonnet ../../samples/literals/negative_float.jsonnet+  -4.222222222222222++  $ tsonnet ../../samples/literals/true.jsonnet+  true++  $ tsonnet ../../samples/literals/false.jsonnet+  false++  $ tsonnet ../../samples/literals/null.jsonnet+  null++  $ tsonnet ../../samples/literals/string.jsonnet+  "Hello, world!"++  $ tsonnet ../../samples/literals/array.jsonnet+  [ 1, 2.0, "hi", null ]++  $ tsonnet ../../samples/literals/object.jsonnet+  {+    "int_attr": 1,+    "float_attr": 4.2,+    "string_attr": "Hello, world!",+    "null_attr": null,+    "array_attr": [ 1, false, {} ],+    "obj_attr": { "a": true, "b": false, "c": { "d": [ 42 ] } }+  }
Enter fullscreen modeExit fullscreen mode

To finalize, let's run the tests:

$dune runtest
Enter fullscreen modeExit fullscreen mode

There are no errors when all the tests have passed.

Let's introduce a small error, changing the output value of one or two of the tests, and see how it is presented:

Here's our intentionally introduced error changes:

diff --git a/test/cram/literals.t b/test/cram/literals.tindex c7b1c2a..54d455f 100644--- a/test/cram/literals.t+++ b/test/cram/literals.t@@ -1,8 +1,8 @@   $ tsonnet ../../samples/literals/int.jsonnet-  42+  666   $ tsonnet ../../samples/literals/float.jsonnet-  4.222222222222222+  4.23   $ tsonnet ../../samples/literals/negative_int.jsonnet   -42
Enter fullscreen modeExit fullscreen mode

The output of the tests:

$dune runtestFile"test/cram/literals.t", line 1, characters 0-0:diff--git a/_build/.sandbox/0a16dd95b0ff6ebafcf09f03744c5c4c/default/test/cram/literals.t b/_build/.sandbox/0a16dd95b0ff6ebafcf09f03744c5c4c/default/test/cram/literals.t.correctedindex 54d455f..c7b1c2a 100644--- a/_build/.sandbox/0a16dd95b0ff6ebafcf09f03744c5c4c/default/test/cram/literals.t+++ b/_build/.sandbox/0a16dd95b0ff6ebafcf09f03744c5c4c/default/test/cram/literals.t.corrected@@-1,8 +1,8 @@$tsonnet ../../samples/literals/int.jsonnet-  666+  42$tsonnet ../../samples/literals/float.jsonnet-  4.23+  4.222222222222222$tsonnet ../../samples/literals/negative_int.jsonnet-42
Enter fullscreen modeExit fullscreen mode

It is super easy to spot the expected value and the actual result we got. Isn't it nice?!

Concluding

Cram tests have become one of my favorite tools. I can't live without them if I'm writing a CLI program seriously.

This is not a new idea. The first time I saw something similar wasin Elixir via doctests. I read somewhere thatthis cram Python package is where it originated, but I can't say for sure. The last time I tried Rust was many years ago, but as far as I know,Rust has documentation tests.

I don't know who copied who, but I hope this trend continues. This is a really cool feature that I hope every single programming language implements, either fully as cram tests or partially as documentation tests.

Disclaimer:alcotest is basically sitting idle for now, as there are no unit tests or integration tests besides the cram tests. Eventually, there will be. I just anticipated adding the dependency right away.

What do you think about cram tests?


Thanks for reading Bit Maybe Wise!Subscribe to receive new posts about Tsonnet and cram tests.

Photo byTai Bui onUnsplash

Top comments(0)

Subscribe
pic
Create template

Templates let you quickly answer FAQs or store snippets for re-use.

Dismiss

Are you sure you want to hide this comment? It will become hidden in your post, but will still be visible via the comment'spermalink.

For further actions, you may consider blocking this person and/orreporting abuse

Coding for a living and for fun
  • Location
    Berlin, Germany
  • Work
    Backend Engineer at GitLab
  • Joined

More fromHercules Lemke Merscher

DEV Community

We're a place where coders share, stay up-to-date and grow their careers.

Log in Create account

[8]ページ先頭

©2009-2025 Movatter.jp