Repository files navigation

Bencher is a suite ofcontinuous benchmarking tools.Have you ever had a performance regression impact your users?Bencher could have prevented that from happening.Bencher allows you to detect and prevent performance regressionsbefore they hit production.

• Run: Run your benchmarks locally or in CI using your favorite benchmarking tools. Thebencher CLI simply wraps your existing benchmark harness and stores its results.
• Track: Track the results of your benchmarks over time. Monitor, query, and graph the results using the Bencher web console based on the source branch, testbed, benchmark, and measure.
• Catch: Catch performance regressions in CI. Bencher uses state of the art, customizable analytics to detect performance regressions before they make it to production.

For the same reasons that unit tests are run in CI to prevent feature regressions, benchmarks should be run in CI with Bencher to prevent performance regressions. Performance bugs are bugs!

Bencher consists of:

• bencher CLI
• Bencher API Server
• Bencher Console Web UI

Though Bencher is open source, there is also a hosted version availableBencher Cloud.

The best place to start is theBencher Quick Start tutorial.

🐰Use the GitHub Action with your project

• Tutorial
  • Quick Start
  • Self-Hosted Quick Start
• How To
  • Install CLI
  • Claim Benchmark Results
  • Track Benchmarks in CI
  • GitHub Actions
  • GitLab CI/CD
  • Track Custom Benchmarks
  • Track Build Time
  • Track File Size
  • Self-Hosted GitHub App
• Explanation
  • Benchmarking Overview
  • bencher run
  • Branch Selection
  • Thresholds & Alerts
  • Benchmark Adapters
  • Continuous Benchmarking
  • Bencher Self-Hosted
  • Talks
• Reference
  • REST API
  • Architecture
  • Console Server Config
  • API Server Config
  • Bencher Metric Format
  • Prior Art
  • Roadmap
  • Changelog

🌐 Also available in:

• 简体中文
• Español
• Português do Brasil
• Русский
• 日本語
• Français
• Deutsch
• 한국어

• {...} JSON
  • Custom benchmark harness support
  • Build Time (Compile Time)
  • File Size (Binary Size)
• #️⃣ C#
  • BenchmarkDotNet
• ➕ C++
  • Catch2
  • Google Benchmark
• 🕳 Go
  • go test -bench
• ☕️ Java
  • Java Microbenchmark Harness (JMH)
• 🕸 JavaScript
  • Benchmark.js
  • console.time/console.timeEnd
• 🐍 Python
  • airspeed velocity
  • pytest-benchmark
• ♦️ Ruby
  • Benchmark
• 🦀 Rust
  • libtest bench
  • Criterion
  • Iai
  • Iai-Callgrind
• ❯_ Shell
  • Hyperfine

👉 For more details see theexplanation of benchmark harness adapters.

👉 Checkoutall public projects.

Install the Bencher CLI using theGitHub Action,and use it forcontinuous benchmarking in your project.

name:Continuous Benchmarking with Bencheron:push:branches:mainjobs:benchmark_with_bencher:name:Benchmark with Bencherruns-on:ubuntu-latestenv:BENCHER_PROJECT:my-project-slugBENCHER_API_TOKEN:${{ secrets.BENCHER_API_TOKEN }}steps:      -uses:actions/checkout@v4      -uses:bencherdev/bencher@main      -run:bencher run "bencher mock"

Supported Operating Systems:

• Linux (x86_64 & ARM64)
• macOS (x86_64 & ARM64)
• Windows (x86_64 & ARM64)

👉 For more details see theexplanation of how to use GitHub Actions.

AddBENCHER_API_TOKEN to youRepository secrets (ex:Repo -> Settings -> Secrets and variables -> Actions -> New repository secret). You can find your API tokens by runningbencher token list my-user-slug orview them in the Bencher Console.

You can set thebencher run CLI subcommand to errorifan Alert is generated with the--err flag.

bencher run --err"bencher mock"

👉 For more details see theexplanation ofbencher run.

You can set thebencher run CLI subcommand to comment on a PR with the--github-actions argument.

bencher run --github-actions"${{ secrets.GITHUB_TOKEN }}""bencher mock"

👉 For more details see theexplanation ofbencher run.

🚨 1 ALERT: Threshold Boundary Limit exceeded!

There is also an optionalversion argument to specify an exact version of the Bencher CLI to use.Otherwise, it will default to using the latest CLI version.

-uses:bencherdev/bencher@mainwith:version:0.5.3

Specify an exact version if usingBencherSelf-Hosted.Donot specify an exact version if using BencherCloud as there are still occasional breaking changes.

The easiest way to contribute is to open this repo as aDev Container inVSCode by simply clicking one of the buttons below.Everything you need will already be there!Once set up, both the UI and API should be built, running, and seeded atlocalhost:3000 andlocalhost:61016 respectively.To make any changes to the UI or API though, you will have to exit the startup process and restart the UI and API yourself.

For additional information on contributing, see theDevelopment Getting Started guide.

There is also apre-built image from CI available for each branch:ghcr.io/bencherdev/bencher-dev-container

All content that resides under any directory orfeature named "plus" is licensed under theBencher Plus License.

All other content is licensed under theApache License, Version 2.0 orMIT License at your discretion.

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in Bencher by you, as defined in the Apache-2.0 license, shall be licensed as above, without any additional terms or conditions.