Repository files navigation

BadgerDB is an embeddable, persistent, simple and fast key-value (KV) databasewritten in pure Go. It's meant to be a performant alternative to non-Go-basedkey-value stores likeRocksDB.

We are currently gearing up for av1.0 release. We recently introducedtransactions which involved a major API change.To use the previous version ofthe APIs, please usetag v0.8. This tag can be specified via theGo dependency tool you're using.

• Getting Started
  • Installing
  • Opening a database
  • Transactions
    • Read-only transactions
    • Read-write transactions
    • Managing transactions manually
  • Using key/value pairs
  • Iterating over keys
    • Prefix scans
    • Key-only iteration
  • Garbage Collection
  • Database backups
  • Statistics
• Resources
  • Blog Posts
• Contact
• Design
  • Comparisons
  • Benchmarks
• Other Projects Using Badger
• Frequently Asked Questions

To start using Badger, install Go 1.8 or above and rungo get:

$ go get github.com/dgraph-io/badger/...

This will retrieve the library and install thebadger_info command lineutility into your$GOBIN path.

The top-level object in Badger is aDB. It represents multiple files on diskin specific directories, which contain the data for a single database.

To open your database, use thebadger.Open() function, with the appropriateoptions. TheDir andValueDir options are mandatory and must bespecified by the client. They can be set to the same value to simplify things.

package mainimport ("log""github.com/dgraph-io/badger")funcmain() {// Open the Badger database located in the /tmp/badger directory.// It will be created if it doesn't exist.opts:=badger.DefaultOptionsopts.Dir="/tmp/badger"opts.ValueDir="/tmp/badger"db,err:=badger.Open(opts)iferr!=nil {log.Fatal(err)  }deferdb.Close()  // Your code here…}

Please note that Badger obtains a lock on the directories so multiple processescannot open the same database at the same time.

To start a read-only transaction, you can use theDB.View() method:

err:=db.View(func(tx*badger.Txn)error {  // Your code here…  returnnil})

You cannot perform any writes or deletes within this transaction. Badgerensures that you get a consistent view of the database within this closure. Anywrites that happen elsewhere after the transaction has started, will not beseen by calls made within the closure.

To start a read-write transaction, you can use theDB.Update() method:

err:=db.Update(func(tx*badger.Txn)error {  // Your code here…  returnnil})

All database operations are allowed inside a read-write transaction.

Always check the return error as it will report anErrConflict in case ofconflict or other errors, for e.g. due to disk failures. If you return an errorwithin your closure it will be passed through.

TheDB.View() andDB.Update() methods are wrappers around theDB.NewTransaction() andTxn.Commit() methods (orTxn.Discard() in case ofread-only transactions). These helper methods will start the transaction,execute a function, and then safely discard your transaction if an error isreturned. This is the recommended way to use Badger transactions.

However, sometimes you may want to manually create and commit yourtransactions. You can use theDB.NewTransaction() function directly, whichtakes in a boolean argument to specify whether a read-write transaction isrequired. For read-write transactions, it is necessary to callTxn,Commit()to ensure the transaction is committed. For read-only transactions, callingTxn.Discard() is sufficient.Txn.Commit() also callsTxn.Discard()internally to cleanup the transaction, so just callingTxn.Commit() issufficient for read-write transaction. However, if your code doesn’t callTxn.Commit() for some reason (for e.g it returns prematurely with an error),then please make sure you callTxn.Discard() in adefer block. Refer to thecode below.

// Start a writable transaction.txn,err:=db.NewTransaction(true)iferr!=nil {returnerr}defertxn.Discard()// Use the transaction...err:=txn.Set([]byte("answer"), []byte("42"),0)iferr!=nil {returnerr}// Commit the transaction and check for error.iferr:=txn.Commit(nil);err!=nil {returnerr}

The first argument toDB.NewTransaction() is a boolean stating if the transactionshould be writable.

Badger allows an optional callback to theTxn.Commit() method. Normally, thecallback can be set tonil, and the method will return after all the writeshave succeeded. However, if this callback is provided, theTxn.Commit()method returns as soon as it has checked for any conflicts. The actual writingto the disk happens asynchronously, and the callback is invoked once thewriting has finished, or an error has occurred. This can improve the throughputof the application in some cases. But it also means that a transaction is notdurable until the callback has been invoked with anil error value.

To save a key/value pair to a bucket, use theTxn.Set() method:

err:=db.Update(func(txn*badger.Txn)error {err:=txn.Set([]byte("answer"), []byte("42"),0)returnerr})

This will set the value of the"answer" key to"42". To retrieve thisvalue, we can use theTxn.Get() method:

err:=db.View(func(txn*badger.Txn)error {item,err:=txn.Get([]byte("answer"))iferr!=nil {returnerr  }val,err:=item.Value()iferr!=nil {returnerr  }fmt.Printf("The answer is: %s\n",val)returnnil})

Txn.Get() returnsErrKeyNotFound if the value is not found.

Please note that values returned fromGet() are only valid while thetransaction is open. If you need to use a value outside of the transactionthen you must usecopy() to copy it to another byte slice.

Use theTxn.Delete() method to delete a key from the bucket.

To iterate over keys, we can use anIterator, which can be obtained using theTxn.NewIterator() method.

err:=db.View(func(txn*.Tx)error {opts:=DefaultIteratorOptionsopts.PrefetchSize=10it:=txn.NewIterator(opts)forit.Rewind();it.Valid();it.Next() {item:=it.Item()k:=item.Key()v,err:=item.Value()iferr!=nil {returnerr    }fmt.Printf("key=%s, value=%s\n",k,v)  }returnnil})

The iterator allows you to move to a specific point in the list of keys and moveforward or backward through the keys one at a time.

By default, Badger prefetches the values of the next 100 items. You can adjustthat with theIteratorOptions.PrefetchSize field. However, setting it toa value higher than GOMAXPROCS (which we recommend to be 128 or higher)shouldn’t give any additional benefits. You can also turn off the fetching ofvalues altogether. See section below on key-only iteration.

To iterate over a key prefix, you can combineSeek() andValidForPrefix():

db.View(func(txn*badger.Txn)error {it:=txn.NewIterator(&DefaultIteratorOptions)prefix:= []byte("1234")forit.Seek(prefix);it.ValidForPrefix(prefix);it.Next() {item:=it.Item()k:=item.Key()v,err:=item.Value()iferr!=nil {returnerr    }fmt.Printf("key=%s, value=%s\n",k,v)  }returnnil})

Badger supports a unique mode of iteration calledkey-only iteration. It isseveral order of magnitudes faster than regular iteration, because it involvesaccess to the LSM-tree only, which is usually resident entirely in RAM. Toenable key-only iteration, you need to set theIteratorOptions.PrefetchValuesfield tofalse. This can also be used to do sparse reads for selected keysduring an iteration, by callingitem.Value() only when required.

err:=db.View(func(txn*badger.Txn)error {opts:=DefaultIteratorOptionsopts.PrefetchValues=falseit:=txn.NewIterator(opts)forit.Rewind();it.Valid();it.Next() {item:=it.Item()k:=item.Key()fmt.Printf("key=%s\n",k)  }returnnil})

Badger values need to be garbage collected, because of two reasons:

• Badger keeps values separately from the LSM tree. This means that the compaction operationsthat clean up the LSM tree do not touch the values at all. Values need to be cleaned upseparately.

• Concurrent read/write transactions could leave behind multiple values for a single key, because theyare stored with different versions. These could accumulate, and take up unneeded space beyond thetime these older versions are needed.

Badger relies on the client to perform garbage collection at a time of their choosing. It providesthe following methods, which can be invoked at an appropriate time:

• DB.PurgeOlderVersions(): This method iterates over the database, and cleans up all but the latestversions of the key-value pairs. It marks the older versions as deleted, which makes them eligible forgarbage collection.
• DB.PurgeVersionsBelow(key, ts): This method is useful to do a more targeted clean up of older versionsof key-value pairs. You can specify a key, and a timestamp. All versions of the key older than the timestampare marked as deleted, making them eligible for garbage collection.
• DB.RunValueLogGC(): This method triggers a value log garbage collection for a single log file. Thereare no guarantees that a call would result in space reclamation. Every run would rewrite at most one logfile. So, repeated calls may be necessary. Please ensure that you call theDB.Purge…() methods firstbefore invoking this method.

Database backup is anopen issue for v1.0 and will be coming soon.

Badger records metrics using theexpvar package, which is included in the Gostandard library. All the metrics are documented iny/metrics.gofile.

expvar package adds a handler in to the default HTTP server (which has to bestarted explicitly), and serves up the metrics at the/debug/vars endpoint.These metrics can then be collected by a system likePrometheus, to getbetter visibility into what Badger is doing.

1. Introducing Badger: A fast key-value store written natively inGo
2. Make Badger crash resilient with ALICE
3. Badger vs LMDB vs BoltDB: Benchmarking key-value databases in Go
4. Concurrent ACID Transactions in Badger

Badger was written with these design goals in mind:

• Write a key-value database in pure Go.
• Use latest research to build the fastest KV database for data sets spanning terabytes.
• Optimize for SSDs.

Badger’s design is based on a paper titledWiscKey: Separating Keys fromValues in SSD-conscious Storage.

¹ TheWISCKEY paper (on which Badger is based) saw bigwins with separating values from keys, significantly reducing the writeamplification compared to a typical LSM tree.

² RocksDB is an SSD optimized version of LevelDB, which was designed specifically for rotating disks.As such RocksDB's design isn't aimed at SSDs.

³ SSI: Serializable Snapshot Isolation. For more details, see the blog postConcurrent ACID Transactions in Badger

We have run comprehensive benchmarks against RocksDB, Bolt and LMDB. Thebenchmarking code, and the detailed logs for the benchmarks can be found in thebadger-bench repo. More explanation, including graphs can be found the blog posts (linkedabove).

Below is a list of public, open source projects that use Badger:

• Dgraph - Distributed graph database.
• go-ipfs - Go client for the InterPlanetary File System (IPFS), a new hypermedia distribution protocol.
• 0-stor - Single device object store.

If you are using Badger in a project please send a pull request to add it to the list.

• My writes are really slow. Why?

Are you creating a new transaction for every single key update? This will leadto very low throughput. To get best write performance, batch up multiple writesinside a transaction using singleDB.Update() call. You could also havemultiple suchDB.Update() calls being made concurrently from multiplegoroutines.

• I don't see any disk write. Why?

If you're using Badger withSyncWrites=false, then your writes might not be written to value logand won't get synced to disk immediately. Writes to LSM tree are done inmemory first, before theyget compacted to disk. The compaction would only happen onceMaxTableSize has been reached. So, ifyou're doing a few writes and then checking, you might not see anything on disk. Once youClosethe database, you'll see these writes on disk.

• Which instances should I use for Badger?

We recommend using instances which provide local SSD storage, without any limiton the maximum IOPS. In AWS, these are storage optimized instances like i3. Theyprovide local SSDs which clock 100K IOPS over 4KB blocks easily.

• Are there any Go specific settings that I should use?

Wehighly recommend setting a high number for GOMAXPROCS, which allows Go toobserve the full IOPS throughput provided by modern SSDs. In Dgraph, we have setit to 128. For more details,see thisthread.

• Please usediscuss.dgraph.io for questions, feature requests and discussions.
• Please useGithub issue tracker for filing bugs or feature requests.
• Join.
• Follow us on Twitter@dgraphlabs.