Movatterモバイル変換

[0]ホーム
URL:

Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

 Sign up
Appearance settings

A transactional key-value database written in Go for Linux and macOS

License

NotificationsYou must be signed in to change notification settings

voidDB/voidDB

Repository files navigation

voidDB is amemory-mappedkey-value store: simultaneously in-memory and persistent on disk. An embeddeddatabase manager, it is meant to be integrated into application software toeliminate protocol overheads and achieve zero-copy performance. This librarysupplies interfaces for storage and retrieval of arbitrary bytes on 64-bitcomputers running Linux and macOS.

voidDB features Put, Get, and Del operations as well as forward and backwarditeration over self-sorting data in ACID (atomic, consistent, isolated, anddurable) transactions. Readers retain a consistent view of the data throughouttheir lifetime, even as newer transactions are being committed: only pagesfreed by transactions older than the oldest surviving reader are activelyrecycled.

voidDB employs a copy-on-write strategy to maintain data in a multi-versionconcurrency-controlled (MVCC) B+ tree structure. It allows virtually any numberof concurrent readers, but only one active writer at any given moment. Readers(and the sole writer) neither compete nor block one another, even though theymay originate from and operate within different threads and processes.

voidDB is resilient against torn writes. It automatically restores a databaseto its last stable state in the event of a mid-write crash. Once a transactionis committed and flushed to disk it is safe, but even if not it could do noharm to existing data in storage. Applications need not be concerned aboutbroken lockfiles or lingering effects of unfinished transactions should anuncontrolled shutdown occur; its design guarantees automatic and immediaterelease of resources upon process termination.

Benchmarks

voidDB outperforms well-known key-value stores available to Go developers thatare based on B+ trees (LMDB, bbolt) and log-structured merge(LSM)-trees(LevelDB, BadgerDB), inpreliminary performance testsconducted on x86-64 and AArch64 instances hosted on Google Cloud (N2, T2A/Dmachine series, 8 vCPUs, 32 GB memory).

Put

4,096 × 256-KiB random values

|                   (ms/op) | AMD Milan | Ampere Altra | Intel Cascade Lake|| ------------------------- | --------- | ------------ | ------------------|| voidDB (default keyspace) |      1.16 |         1.12 |               1.16|| voidDB (named keyspace)   |      1.16 |         1.12 |               1.16|| LMDB (default keyspace)   |      1.90 |         2.00 |               1.93|| Bolt                      |      1.96 |         1.76 |               2.45|| LMDB (named keyspace)     |      2.11 |         2.26 |               2.15|| LevelDB                   |      2.37 |         2.23 |               2.75|| BadgerDB                  |      3.22 |         5.31 |               3.14|

65,536 × 16-KiB random values

|                   (μs/op) | AMD Milan | Ampere Altra | Intel Cascade Lake|| ------------------------- | --------- | ------------ | ------------------|| voidDB (named keyspace)   |      82.7 |         82.8 |               85.0|| voidDB (default keyspace) |      83.5 |         77.7 |               85.3|| LMDB (named keyspace)     |     152   |        154   |              151|| LMDB (default keyspace)   |     157   |        156   |              157|| BadgerDB                  |     195   |        225   |              183|| Bolt                      |     244   |        217   |              429|| LevelDB                   |     362   |        310   |              303|

1,048,576 × 1-KiB random values

|                   (μs/op) | AMD Milan | Ampere Altra | Intel Cascade Lake|| ------------------------- | --------- | ------------ | ------------------|| voidDB (default keyspace) |      18.7 |         19.5 |               21.4|| voidDB (named keyspace)   |      18.4 |         19.8 |               21.9|| BadgerDB                  |      24.2 |         23.9 |               25.8|| LMDB (named keyspace)     |      28.2 |         32.3 |               33.5|| LMDB (default keyspace)   |      29.5 |         34.4 |               36.6|| LevelDB                   |      72.5 |         58.8 |              165|| Bolt                      | timed out | timed out    | timed out|

Get

4,096 × 256-KiB random values

|                   (μs/op) | AMD Milan | Ampere Altra | Intel Cascade Lake|| ------------------------- | --------- | ------------ | ------------------|| voidDB (named keyspace)   |      1.64 |         1.76 |               1.66|| voidDB (default keyspace) |      1.65 |         1.74 |               1.78|| LMDB (named keyspace)     |      4.97 |         3.97 |               4.46|| LMDB (default keyspace)   |      5.00 |         4.03 |               4.33|| Bolt                      |      5.88 |         5.17 |               5.76|| LevelDB                   |    115    |       125    |             224|| BadgerDB                  |    301    |       142    |             618|

65,536 × 16-KiB random values

|                   (μs/op) | AMD Milan | Ampere Altra | Intel Cascade Lake|| ------------------------- | --------- | ------------ | ------------------|| voidDB (default keyspace) |      1.73 |         2.01 |               2.29|| voidDB (named keyspace)   |      1.76 |         2.02 |               2.11|| LMDB (named keyspace)     |      2.47 |         2.73 |               3.05|| LMDB (default keyspace)   |      2.62 |         2.60 |               3.12|| Bolt                      |      3.68 |         3.87 |               4.59|| LevelDB                   |     27.4  |        34.6  |              46.1|| BadgerDB                  |     21.1  |        41.8  |              71.2|

1,048,576 × 1-KiB random values

|                   (μs/op) | AMD Milan | Ampere Altra | Intel Cascade Lake|| ------------------------- | --------- | ------------ | ------------------|| voidDB (default keyspace) |      1.76 |         2.15 |               2.54|| voidDB (named keyspace)   |      2.07 |         2.58 |               3.00|| LMDB (named keyspace)     |      2.09 |         2.68 |               2.97|| LMDB (default keyspace)   |      2.22 |         2.68 |               3.02|| BadgerDB                  |     23.8  |        22.8  |              31.1|| LevelDB                   |     27.3  |        45.7  |              40.7|| Bolt                      | timed out | timed out    | timed out|

GetNext

4,096 × 256-KiB random values

|                   (μs/op) | AMD Milan | Ampere Altra | Intel Cascade Lake|| ------------------------- | --------- | ------------ | ------------------|| voidDB (default keyspace) |      1.17 |         .938 |               1.15|| voidDB (named keyspace)   |      1.27 |         .939 |               1.12|| Bolt                      |      2.13 |        1.55  |               1.83|| LMDB (named keyspace)     |      4.54 |        3.50  |               3.80|| LMDB (default keyspace)   |      4.65 |        3.44  |               3.90|| LevelDB                   |    107    |      110     |             181|| BadgerDB                  |    198    |       58.7   |             415|

65,536 × 16-KiB random values

|                   (μs/op) | AMD Milan | Ampere Altra | Intel Cascade Lake|| ------------------------- | --------- | ------------ | ------------------|| voidDB (default keyspace) |      .808 |         .721 |               .891|| voidDB (named keyspace)   |      .849 |         .730 |               .874|| Bolt                      |     1.18  |         .869 |               .919|| LMDB (default keyspace)   |     1.78  |        1.47  |              1.64|| LMDB (named keyspace)     |     1.78  |        1.44  |              1.68|| LevelDB                   |    14.7   |       14.7   |             22.3|| BadgerDB                  |    26.1   |        6.54  |             24.1|

1,048,576 × 1-KiB random values

|                   (μs/op) | AMD Milan | Ampere Altra | Intel Cascade Lake|| ------------------------- | --------- | ------------ | ------------------|| voidDB (default keyspace) |      .307 |         .270 |               .384|| voidDB (named keyspace)   |      .296 |         .515 |               .402|| LMDB (named keyspace)     |      .615 |         .515 |               .637|| LMDB (default keyspace)   |      .607 |         .519 |               .642|| LevelDB                   |     1.38  |        1.80  |              1.92|| BadgerDB                  |     4.03  |        5.60  |             15.0|| Bolt                      | timed out | timed out    | timed out|

Getting Started

Install Go to begin developing with voidDB.

$ go versiongo version go1.24.0 linux/arm64

Then, import voidDB in your Go application. The following would result in thecreation of a database file and its reader table in the working directory. Setthe database capacity to any reasonably large value to make sufficient room forthe data you intend to store, even if it exceeds the total amount of physicalmemory; neither memory nor disk is immediately consumed to capacity.

package mainimport ("errors""os""github.com/voidDB/voidDB")funcmain() {const (capacity=1<<40// 1 TiBpath="void")void,err:=voidDB.NewVoid(path,capacity)iferrors.Is(err,os.ErrExist) {void,err=voidDB.OpenVoid(path,capacity)}iferr!=nil {panic(err)}defervoid.Close()}

Use*Void.View (or*Void.Update only when modifying data) for convenienceand peace of mind. Ensure all changes are safely synced to disk withmustSyncset totrue if even the slightest risk of losing those changes is a concern.

mustSync:=trueerr=void.Update(mustSync,func(txn*voidDB.Txn)error {returntxn.Put([]byte("greeting"),[]byte("Hello, World!"),)},)iferr!=nil {panic(err)}

Open a cursor if more than one keyspace is required. An application can mapdifferent values to the same key so long as they reside in separate keyspaces.The transaction handle doubles as a cursor in the default keyspace.

cur0,_:=txn.OpenCursor([]byte("hello"))cur0.Put([]byte("greeting"),[]byte("Hello, World!"),)cur1,_:=txn.OpenCursor([]byte("goodbye"))cur1.Put([]byte("greeting"),[]byte("さらばこの世、わらわはもう寝るぞよ。"),)ifval,err:=cur0.Get([]byte("greeting"));err==nil {log.Printf("%s",val)// Hello, World!}ifval,err:=cur1.Get([]byte("greeting"));err==nil {log.Printf("%s",val)// さらばこの世、わらわはもう寝るぞよ。}

To iterate over a keyspace, use*cursor.Cursor.GetNext/GetPrev. Positionthe cursor with*cursor.Cursor.Get/GetFirst/GetLast.

for {key,val,err:=cur.GetNext()iferrors.Is(err,common.ErrorNotFound) {break}log.Printf("%s -> %s",key,val)}

Author

voidDB builds upon ideas in the celebratedLightning Memory-Mapped DatabaseManager on several key points of its high-leveldesign, but otherwise it is implemented from scratch to break free oflimitations in function, performance, and clarity.

voidDB is a cherished toy, a journey into the Unknown, a heroic struggle, and awork of love. It is the “Twee!” of a bird; a tree falling in the forest; yetanother programmer pouring their drop into the proverbial [bit] bucket. Aboveall, it is a shrine unto simple, readable, and functional code; an assertionthat the dichotomy between such aesthetics and practical performance is mereillusion.

Copyright 2024 Joel Ling

[8]ページ先頭
©2009-2025 Movatter.jp