Repository files navigation

BadgerDB is an embeddable, persistent and fast key-value (KV) database written in pure Go. It is theunderlying database forDgraph, a fast, distributed graph database. It's meantto be a performant alternative to non-Go-based key-value stores like RocksDB.

Badger is stable and is being used to serve data sets worth hundreds of terabytes. Badger supportsconcurrent ACID transactions with serializable snapshot isolation (SSI) guarantees. A Jepsen-stylebank test runs nightly for 8h, with--race flag and ensures the maintenance of transactionalguarantees. Badger has also been tested to work with filesystem level anomalies, to ensurepersistence and consistency. Badger is being used by a number of projects which includes Dgraph,Jaeger Tracing, UsenetExpress, and many more.

The list of projects using Badger can be foundhere.

Badger v1.0 was released in Nov 2017, and the latest version that is data-compatible with v1.0 isv1.6.0.

Badger v2.0 was released in Nov 2019 with a new storage format which won't be compatible with all ofthe v1.x. Badger v2.0 supports compression, encryption and uses a cache to speed up lookup.

Badger v3.0 was released in January 2021. This release improves compaction performance.

Please consult theChangelog for more detailed information on releases.

For more details on our version naming schema please readChoosing a version.

• BadgerDB
  • Project Status
  • Table of Contents
  • Getting Started
    • Installing
      • Installing Badger Command Line Tool
      • Choosing a version
  • Badger Documentation
  • Resources
    • Blog Posts
  • Design
    • Comparisons
    • Benchmarks
  • Projects Using Badger
  • Contributing
  • Contact

To start using Badger, install Go 1.21 or above. Badger v3 and above needs go modules. From yourproject, run the following command

go get github.com/dgraph-io/badger/v4

This will retrieve the library.

Badger provides a CLI tool which can perform certain operations like offline backup/restore. Toinstall the Badger CLI, retrieve the repository and checkout the desired version. Then run

cd badgergo install.

This will install the badger command line utility into your $GOBIN path.

BadgerDB is a pretty special package from the point of view that the most important change we canmake to it is not on its API but rather on how data is stored on disk.

This is why we follow a version naming schema that differs from Semantic Versioning.

• New major versions are released when the data format on disk changes in an incompatible way.
• New minor versions are released whenever the API changes but data compatibility is maintained.Note that the changes on the API could be backward-incompatible - unlike Semantic Versioning.
• New patch versions are released when there's no changes to the data format nor the API.

Following these rules:

• v1.5.0 and v1.6.0 can be used on top of the same files without any concerns, as their majorversion is the same, therefore the data format on disk is compatible.
• v1.6.0 and v2.0.0 are data incompatible as their major version implies, so files created withv1.6.0 will need to be converted into the new format before they can be used by v2.0.0.
• v2.x.x and v3.x.x are data incompatible as their major version implies, so files created withv2.x.x will need to be converted into the new format before they can be used by v3.0.0.

For a longer explanation on the reasons behind using a new versioning naming schema, you can readVERSIONING.

Badger Documentation is available athttps://docs.hypermode.com/badger

1. Introducing Badger: A fast key-value store written natively in Go
2. Make Badger crash resilient with ALICE
3. Badger vs LMDB vs BoltDB: Benchmarking key-value databases in Go
4. Concurrent ACID Transactions in Badger

Badger was written with these design goals in mind:

• Write a key-value database in pure Go.
• Use latest research to build the fastest KV database for data sets spanning terabytes.
• Optimize for SSDs.

Badger’s design is based on a paper titledWiscKey: Separating Keys from Values in SSD-consciousStorage.

¹ TheWISCKEY paper (on which Badger is based) saw big wins with separatingvalues from keys, significantly reducing the write amplification compared to a typical LSM tree.

² RocksDB is an SSD optimized version of LevelDB, which was designed specifically forrotating disks. As such RocksDB's design isn't aimed at SSDs.

³ SSI: Serializable Snapshot Isolation. For more details, see the blog postConcurrent ACID Transactions in Badger

⁴ Badger provides direct access to value versions via its Iterator API. Users can alsospecify how many versions to keep per key via Options.

We have run comprehensive benchmarks against RocksDB, Bolt and LMDB. The benchmarking code, and thedetailed logs for the benchmarks can be found in thebadger-bench repo. More explanation,including graphs can be found the blog posts (linked above).

Below is a list of known projects that use Badger:

• Dgraph - Distributed graph database.
• Jaeger - Distributed tracing platform.
• go-ipfs - Go client for the InterPlanetary File System (IPFS),a new hypermedia distribution protocol.
• Riot - An open-source, distributed search engine.
• emitter - Scalable, low latency, distributed pub/subbroker with message storage, uses MQTT, gossip and badger.
• OctoSQL - Query tool that allows you to join, analyse andtransform data from multiple databases using SQL.
• Dkron - Distributed, fault tolerant job scheduling system.
• smallstep/certificates - Step-ca is an onlinecertificate authority for secure, automated certificate management.
• Sandglass - distributed, horizontally scalable,persistent, time sorted message queue.
• TalariaDB - Grab's Distributed, low latency time-seriesdatabase.
• Sloop - Salesforce's Kubernetes History VisualizationProject.
• Usenet Express - Serving over 300TB of data with Badger.
• gorush - A push notification server written in Go.
• 0-stor - Single device object store.
• Dispatch Protocol - Blockchain protocol for distributedapplication data analytics.
• GarageMQ - AMQP server written in Go.
• RedixDB - A real-time persistent key-value store with thesame redis protocol.
• BBVA - Raft backend implementation using BadgerDB forHashicorp raft.
• Fantom - aBFT Consensus platform fordistributed applications.
• decred - An open, progressive, and self-fundingcryptocurrency with a system of community-based governance integrated into its blockchain.
• OpenNetSys - Create useful dApps in any software language.
• HoneyTrap - An extensible and opensource system forrunning, monitoring and managing honeypots.
• Insolar - Enterprise-ready blockchain platform.
• IoTeX - The next generation of the decentralizednetwork for IoT powered by scalability- and privacy-centric blockchains.
• go-sessions - The sessions manager for Go net/http andfasthttp.
• Babble - BFT Consensus platform for distributedapplications.
• Tormenta - Embedded object-persistence layer / simple JSONdatabase for Go projects.
• BadgerHold - An embeddable NoSQL store for querying Gotypes built on Badger
• Goblero - Pure Go embedded persistent job queue backed byBadgerDB
• Surfline - Serving global wave and weather forecast data with Badger.
• Cete - Simple and highly available distributed key-value storebuilt on Badger. Makes it easy bringing up a cluster of Badger with Raft consensus algorithm byhashicorp/raft.
• Volument - A new take on website analytics backed by Badger.
• KVdb - Hosted key-value store and serverless platform built on top of Badger.
• Terminotes - Self hosted notes storage and searchserver - storage powered by BadgerDB
• Pyroscope - Open source continuous profiling platformbuilt with BadgerDB
• Veri - A distributed feature store optimized for Search andRecommendation tasks.
• bIter - A library and Iterator interface for working withthebadger.Iterator, simplifying from-to, and prefix mechanics.
• ld - (Lean Database) A very simple gRPC-only key-valuedatabase, exposing BadgerDB with key-range scanning semantics.
• Souin - A RFC compliant HTTP cache with lot of other featuresbased on Badger for the storage. Compatible with all existing reverse-proxies.
• Xuperchain - A highly flexible blockchain architecturewith great transaction performance.
• m2 - A simple http key/value store based on the raft protocol.
• chaindb - A blockchain storage layer used byGossamer, a Go client for thePolkadot Network.
• vxdb - Simple schema-less Key-Value NoSQL database withsimplest API interface.
• Opacity - Backend implementation for the Opacitystorage project
• Vephar - A minimal key/value store using hashicorp-raftfor cluster coordination and Badger for data storage.
• gowarcserver - Open-source server for warc files. Can beused in conjunction with pywb
• flow-go - A fast, secure, and developer-friendly blockchainbuilt to support the next generation of games, apps and the digital assets that power them.
• Wrgl - A data version control system that works like Git but specialized tostore and diff CSV.
• Loggie - A lightweight, cloud-native data transfer agentand aggregator.
• raft-badger - raft-badger implements LogStore andStableStore Interface of hashcorp/raft. it is used to store raft log and metadata ofhashcorp/raft.
• DVID - A dataservice for branched versioning of a varietyof data types. Originally created for large-scale brain reconstructions in Connectomics.
• KVS - A library for making it easy to persist, load and queryfull structs into BadgerDB, using an ownership hierarchy model.
• LLS - LLS is an efficient URL Shortener that can be used toshorten links and track link usage. Support for BadgerDB and MongoDB. Improved performance by morethan 30% when using BadgerDB
• lakeFS - lakeFS is an open-source data version control thattransforms your object storage to Git-like repositories. lakeFS uses BadgerDB for its underlyinglocal metadata KV store implementation
• Goptivum - Goptivum is a better frontend and API for theVulcan Optivum schedule program
• ActionManager - A dynamic entity manager based on rjsf schemaand badger db
• MightyMap - Mightymap: Conveys both robustnessand high capability, fitting for a powerful concurrent map.
• FlowG - A low-code log processing facility
• Bluefin - Bluefin is a TUNA Proof of Work miner forthe Fortuna smart contract on the Cardano blockchain
• cDNSd - A Cardano blockchain backed DNS server daemon
• Dingo - A Cardano blockchain data node

If you are using Badger in a project please send a pull request to add it to the list.

If you're interested in contributing to Badger seeCONTRIBUTING.

• Please useGithub issues for filing bugs.
• Please usediscuss.dgraph.io for questions, discussions, and featurerequests.
• Follow us on Twitter@dgraphlabs.