Repository files navigation

Just want the latest results?

• Library Comparisons:Full Benchmark |Smoketest
• Version Regressions:Full Benchmark |Smoketest
• LMDB Benchmarks:Full Benchmark |Smoketest
• Historical:2016 Results

This is aJMH benchmarkof open source, embedded, memory-mapped, key-value stores available from Java:

• LmdbJava (with fastByteBuffer, safeByteBuffer and anAgrona buffer)
• LMDBJNI
• Lightweight Java Game Library (LMDB API)
• LevelDBJNI
• RocksDB
• MVStore (pure Java)
• MapDB (pure Java)
• Xodus (pure Java)
• Chroncile Map (pure Java) (**)

(**) does not support ordered keys, so iteration benchmarks not performed

The benchmark itself is adapted from LMDB'sdb_bench_mdb.cc, which inturn is adapted fromLevelDB's benchmark.

The benchmark includes:

• Writing data
• Reading all data via each key
• Reading all data via a reverse iterator
• Reading all data via a forward iterator
• Reading all data via a forward iterator and computing a CRC32 (via JDK API)
• Reading all data via a forward iterator and computing a XXH64 hash

Byte arrays (byte[]) are always used for the keys and values, avoiding anyserialization library overhead. For those libraries that support compression,it is disabled in the benchmark. In general any special library features thatdecrease latency (eg batch modes, disable auto-commit, disable journals,hint at expected data sizes etc) were used. While we have tried to be fair andconsistent, some libraries offer non-obvious tuning settings or usage patternsthat might further reduce their latency. We do not claim we have exhaustedevery tuning option every library exposes, but pull requests are most welcome.

Clone this repository and build:

mvn clean package

This benchmark uses POSIX calls to accurately determine consumed disk space andonly depends on Linux-specific native library wrappers where a range of suchwrappers exists. Operation on non-Linux operating systems is unsupported.

Use therun-libs.sh script to compare different key-value store libraries:

# Quick smoke test (1K entries, fast verification)./run-libs.sh smoketest# Full benchmark using 25% of system RAM (default)./run-libs.sh benchmark# Full benchmark using 50% of system RAM./run-libs.sh benchmark 50

The benchmark auto-scales based on available RAM and caps at 1 million entries.Results are written totarget/benchmark-libs/.

File handle limit: Benchmark mode requires at least 1 million file handles for RocksDB and LevelDB LSM operations:

ulimit -n 1000000./run-libs.sh benchmark

The script will check and abort if the limit is too low.

Resumption: The script skips completed runs by checking for existing result files. For a fresh run, remove the output directory:

rm -rf target/benchmark-libs./run-libs.sh benchmark

Long-running benchmarks: For multi-hour benchmarks, redirect output to a log file and monitor withtail:

# Start the benchmark with output redirected./run-libs.sh benchmark> log.txt2>&1# Monitor progress in another terminaltail -f log.txt

Use therun-vers.sh script to test LmdbJava performance across versions:

# Quick smoke test (1K entries, fast verification)./run-vers.sh smoketest# Full benchmark using 25% of system RAM (default)./run-vers.sh benchmark# Full benchmark using 50% of system RAM./run-vers.sh benchmark 50

This tests selected LmdbJava versions from Maven Central plus current development branches to identify performance regressions.Results are written totarget/benchmark-vers/.

Resumption: The script skips completed versions by checking for existing result files. For a fresh run, remove the output directory:

rm -rf target/benchmark-vers./run-vers.sh benchmark

Use therun-lmdb.sh script to test LmdbJava (master) performance across different LMDB library versions:

# Quick smoke test (1K entries, fast verification)./run-lmdb.sh smoketest# Full benchmark using 25% of system RAM (default)./run-lmdb.sh benchmark# Full benchmark using 50% of system RAM./run-lmdb.sh benchmark 50

This tests LmdbJava master against 9 different LMDB library versions (0.9.18-0.9.33) to isolate LMDB native library performance characteristics from LmdbJava wrapper code.Results are written totarget/benchmark-lmdb/.

Resumption: The script skips completed LMDB versions by checking for existing result files. For a fresh run, remove the output directory:

rm -rf target/benchmark-lmdb./run-lmdb.sh benchmark

Use therun-all.sh script to run all three benchmark suites sequentially (designed for overnight runs):

# Run all benchmarks using 25% of system RAM (default)./run-all.sh# Run all benchmarks using 50% of system RAM./run-all.sh 50

This will run LMDB library comparison, library comparison and version regression benchmarks, all in full benchmark mode with 120s iterations. Expect several hours of runtime depending on your system.

After running library comparison benchmarks, generate a comprehensive report:

./report-libs.sh

After running version regression tests, generate a version comparison report:

./report-vers.sh

After running LMDB benchmarks, generate an LMDB performance report:

./report-lmdb.sh

Reports generate:

• target/benchmark/index.html - Pure HTML report with embedded SVG charts
• Various SVG chart files

After generating a report, you can publish it to Cloudflare Pages:

export CLOUDFLARE_API_TOKEN="your-token"export CLOUDFLARE_ACCOUNT_ID="your-account-id"./publish-results.sh

The script automatically detects:

• Type: Library comparison or version regression (from README heading)
• Mode: Smoketest (3s) or benchmark (120s) (from smoketest warning)

Reports are published to:

• libraries-smoketest.lmdbjava.org - Fast daily library verification
• libraries-benchmark.lmdbjava.org - Full library comparison analysis
• versions-smoketest.lmdbjava.org - Fast daily version monitoring
• versions-benchmark.lmdbjava.org - Full version regression analysis

Workflow for curated reports:

1. Run full benchmarks:./run-all.sh (or individually with./run-lmdb.sh benchmark,./run-libs.sh benchmark and./run-vers.sh benchmark)
2. Generate reports:./report-lmdb.sh,./report-libs.sh and./report-vers.sh
3. Review and edit commentary in report scripts if needed, then re-run
4. Publish:./publish-results.sh (run once after each report generation)

Therun-bisect.sh script uses bisection to find the commit that introduced a performance regression:

./run-bisect.sh

Configure the bisection by editing variables at the top of the script:

• START_COMMIT: Known good commit (full hash)
• END_COMMIT: Known bad commit (full hash)
• BENCHMARK_NAME: Full benchmark qualifier (egLmdbJavaAgrona.write)
• MAX_BISECTIONS: Maximum bisection iterations (default 10)

The script:

• Tests endpoint commits to confirm regression
• Bisects the commit range at 50% intervals
• Uses distance-based decision making (closer to good or bad endpoint)
• Uses the system LMDB library (/usr/lib/liblmdb.so) for all tests
• Caches built JARs and benchmark results to avoid rebuilding
• Generates a summary report with an asterisk marking the regression commit

Results are saved inbisect/results/ with a chronologically-sorted summary report showing scores and percentage changes. To re-run with fresh benchmarks, deletebisect/results/*.json.

Therun-dev.sh script enables rapid testing of local LmdbJava changes without full bisection overhead:

./run-dev.sh

This script:

• Builds LmdbJava fromdev/lmdbjava/ (clone or symlink your working copy here)
• Patches the benchmark pom.xml to use the local SNAPSHOT version
• Runs a quick benchmark (10K entries, 3s runtime by default)
• Saves results todev/output/ with commit hash and timestamp

To match full bisection configuration, edit the script and change:

• JMH_RUNTIME=30 (30 second iterations)
• NUM_ENTRIES=1000000 (1 million entries)

Use this for iterative development and verification of performance fixes before committing.

Update all dependency and plugin versions:

mvn versions:update-properties

Issues are disabled for this repository. Please report any issues or questionson theLmdbJava issue tracker.

Contributions are welcome! Please see the LmdbJava project'sContributing Guidelines.

This project is licensed under theApache License, Version 2.0.