Repository files navigation

• Overview
  • Measurements
• Test Cases
  • Brainfuck
    • bench.b
    • mandel.b
  • Base64
  • Json
  • Matmul
  • Primes
• Tests Execution
  • Environment
  • Using Docker
  • Manual Execution
    • Prerequisites
• Contribution
  • Makefile guide
    • Binary executables
    • Compiled artifacts
    • Scripting language
  • README update
  • Docker image update

The benchmarks follow the criteria:

• They are written as the average software developer would write them, i.e.

  • The algorithms are implemented as cited in public sources;
  • The libraries are used as described in the tutorials, documentation and examples;
  • The used data structures are idiomatic.

• The used algorithms are similar between the languages (as the reference implementations), variants are acceptable if the reference implementation exists.

• All final binaries are releases (optimized for performance if possible) as debug performance may vary too much depending on the compiler.

My other benchmarks:jit-benchmarks,crystal-benchmarks-game

The measured values are:

• time spent for the benchmark execution (loading required data and code self-testing are not measured);
• memory consumption of the benchmark process, reported asbase +increase, wherebase is the RSS before the benchmark andincrease is the peak increase of the RSS during the benchmark;
• energy consumption of the CPU package during the benchmark: PP0 (cores) + PP1 (uncores like GPU) + DRAM. Currently, only Intel CPU are supported via the powercap interface.

All values are presented as:median_{±median absolute deviation}.

UPDATE: 2025-08-17

Testing brainfuck implementations using two code samples (bench.b and mandel.b).Supports two mode:

• Verbose (default). Prints the output immediately.
• Quiet (if QUIET environment variable is set). Accumulates the output using Fletcher-16 checksum, and prints it out after the benchmark.

Brainfuck

Mandel in Brainfuck

Testing base64 encoding/decoding of the large blob into the newly allocated buffers.

Base64

Testing parsing and simple calculating of values from a big JSON file.

Few notes:

• gason mutates input strings;
• simdjson requires input strings with batch of trailing zeros: a special zero padding for SIMD instructions;
• DAW JSON Link "NoCheck" skips some JSON structure correctness checks;
• DAW JSON Link, gason, default (not "Precise") RapidJSON, and D implementations except Mir-basedhave some inaccuracies in number parsing:
  • DAW JSON Link's number parsing issue
  • gason's number parsing issue
  • D stdlib number parsing issue

Json

Testing allocating and multiplying matrices.

Matmul

Testing:

• generating primes using the optimizedsieve of Atkin;
• prefix search for their decimal numbers using Trie data structure.

Notes:

• All languages but V and Python use unordered hashmaps (V and Python don't provide those out of box, andtheir hashmaps use keys in the insertion order);
• The results are always sorted (could be unstable or stable though).

Primes

CPU: Intel(R) Xeon(R) E-2324G

Base Docker image: Debian GNU/Linux trixie/sid

Build the image:

$ docker build docker/ -t benchmarks

Run the image:

$ docker run -it --rm -v $(pwd):/src benchmarks <cmd>

where<cmd> is:

• versions (print installed language versions);
• shell (start the shell);
• brainfuck bench (build and run Brainfuck bench.b benchmarks);
• brainfuck mandel (build and run Brainfuck mandel.b benchmarks);
• base64 (build and run Base64 benchmarks);
• json (build and run Json benchmarks);
• matmul (build and run Matmul benchmarks);
• primes (build and run Primes benchmarks);

Please note that the actual measurements provided in the project are taken semi-manually (viashell) as the full update takes days and could have occassional issues in Docker.

There is a./run.sh that could be used to simplify Docker usage:

• ./run.sh build (build the image);
• ./run.sh make versions (run the image with theversions command);
• sudo ./run.sh shell (run the image with the `shell' command, sudo is required to read energy levels).

Makefiles contain recipes for building and executing tests with theproper dependencies. Please usemake run (andmake run2 where applicable).The measurements are taken usinganalyze.rb script:

$ cd <test suite>$ ../analyze.rb make run$ ../analyze.rb make run[<single test>]

Please note that the measurements could take hours. It uses 10 iterationsby default, but it could be changed using ATTEMPTS environment variable:

$ ATTEMPTS=1 ../analyze.rb make run

Please useDockerfile as a reference regarding whichpackages and tools are required.

For all (optional):

• Powercap for reading energycounters in Linux (Debian packagepowercap-utils).

For Python:

• NumPy for matmul tests(Debian packagepython3-numpy).
• UltraJSON for JSON tests(Debian packagepython3-ujson).

For C++:

• Boost for JSON tests(Debian packagelibboost-dev).
• JSON-C for JSON tests(Debian packagelibjson-c-dev).

For Rust:

• libjq for jq test(Debian packageslibjq-dev,libonig-dev and environment variableJQ_LIB_DIR=/usr/lib/x86_64-linux-gnu/).

For Java, Scala:

• Coursier for downloading Maven artifacts.

For Haskell:

• network forTCP connectivity between the tests and the test runner.
• raw-strings-qq forraw string literals used in tests.

For Perl:

• cpanminus for installingmodules from CPAN (Debian packagecpanminus).

For Vala:

• JSON-GLib for JSON tests(Debian packagelibjson-glib-dev).

Please follow the criteria specified in theoverview. Besidesthat please ensure that the communication protocol between a test and thetest runner is satisfied:

• The test runner listens on localhost:9001;
• All messages are sent using TCP sockets closed immediately after themessage has been sent;
• There are two messages sent from a test (it establishes the measurementboundary):
  1. The beginning message having the formatname of the test/tprocess ID(the process ID is used to measure the memory consumption). Please note thatthe name of the test couldn't use Tab character as it's a delimiter;
  2. The end message with any content (mostly it's "stop" for consistency).
• The test runner could be unavailable (if the test is launched as is) andthe test should gracefully handle it.

If the test is compiled into a single binary, then two sections oftheMakefile require changes:

• append a new target (the final binary location) intoexecutablesvariable;
• append the proper target rule.

If the test is compiled, but can't be executed directly as a binary, thenthree sections of theMakefile require changes:

• append a new target (the final artifact location) intoartifactsvariable;
• append the proper target rule to compile the test;
• appendrun[<target_artifact>] rule to run the test.

If the test doesn't require compilation, then two sections of theMakefilerequires changes:

• appendrun[<script_file>] intoall_runners variable;
• appendrun[<script_file>] rule to run the test.

TOC is regenerated usinggit-markdown-toc:

./run.sh toc