TinKV is a simple and fast key-value storage engine written in Rust. Inspired bybasho/bitcask, written after attending theTalent Plan courses.

Notes:

• Do not use it in production.
• Operations like set/remove/compact are not thread-safe currently.

Happy hacking~

Features

• Embeddable (usetinkv as a library);
• Builtin CLI (tinkv);
• Builtin Redis compatible server;
• Predictable read/write performance.

Usage

As a library

$cargo add tinkv

Full example usage can be found inexamples/basic.rs.

usetinkv::{self,Store};fnmain()->tinkv::Result<()>{pretty_env_logger::init();letmutstore=Store::open("/path/to/tinkv")?;store.set("hello".as_bytes(),"tinkv".as_bytes())?;letvalue=store.get("hello".as_bytes())?;assert_eq!(value,Some("tinkv".as_bytes().to_vec()));store.remove("hello".as_bytes())?;letvalue_not_found=store.get("hello".as_bytes())?;assert_eq!(value_not_found,None);Ok(())}

Open with custom options

usetinkv::{self,Store};fnmain()->tinkv::Result<()>{letmutstore=tinkv::OpenOptions::new().max_data_file_size(1024*1024).max_key_size(128).max_value_size(128).sync(true).open(".tinkv")?;store.set("hello".as_bytes(),"world".as_bytes())?;Ok(())}

APIs

Public APIs of tinkv store are very easy to use:| API | Description ||————————–|—————————————————————||Store::open(path) | Open a new or existing datastore. The directory must be writeable and readable for tinkv store.||tinkv::OpenOptions() | Open a new or existing datastore with custom options. ||store.get(key) | Get value by key from datastore.||store.set(key, value) | Store a key value pair into datastore.||store.remove(key, value)| Remove a key from datastore.||store.compact() | Merge data files into a more compact form. drop stale segments to release disk space. Produce hint files after compaction for faster startup.||store.keys() | Return all the keys in database.||store.len() | Return total number of keys in database.||store.for_each(f: Fn(key, value) -> Result)` | Iterate all keys in database and call function `f` for each entry.||`store.stas()` | Get current statistics of database.||`store.sync()` | Force any writes to datastore.||`store.close()` | Close datastore, sync all pending writes to disk.|

Run examples

$ RUST_LOG=trace cargo run--example basic

RUST_LOG level can be one of [trace,debug,info,error].

CLI

Installtinkv executable binaries.

$cargoinstalltinkv

$tinkv--help...USAGE:    tinkv[FLAGS] <path> <SUBCOMMAND>FLAGS:-h,--help       Printshelpinformation-q,--quiet      Pass manytimesforless log output-V,--version    Prints version information-v,--verbose    Pass manytimesformore log outputARGS:    <path>    Path to tinkv datastoreSUBCOMMANDS:    compact    Compact data filesindatastore and reclaim disk space    del        Delete a key value pair from datastore    get        Retrive value of a key, and display the valuehelpPrints this message or thehelpof the given subcommand(s)    keys       List all keysindatastore    scan       Perform a prefix scanningforkeyssetStore a key value pair into datastore    stats      Display statistics of the datastore

Example usages:

$tinkv /tmp/dbsethello world$tinkv /tmp/db get helloworld# Change verbosity level (info).$tinkv /tmp/db-vvv compact2020-06-20T10:32:45.582Z INFO  tinkv::store> open store path: tmp/db2020-06-20T10:32:45.582Z INFO  tinkv::store> build keydir from data file /tmp/db/000000000001.tinkv.data2020-06-20T10:32:45.583Z INFO  tinkv::store> build keydir from data file /tmp/db/000000000002.tinkv.data2020-06-20T10:32:45.583Z INFO  tinkv::store> build keydirdone, got 1 keys. current stats: Stats{ size_of_stale_entries:0, total_stale_entries: 0, total_active_entries: 1,total_data_files: 2, size_of_all_data_files: 60}2020-06-20T10:32:45.583Z INFO  tinkv::store> there are 3 datafiles need to be compacted

Client & Server

tinkv-server is a redis-compatible key/value store server. However, not all the redis commmands are supported. The available commands are:

• get <key>
• mget <key> [<key>...]
• set <key> <value>
• mset <key> <value> [<key> <value>]
• del <key>
• keys <pattern>
• ping [<message>]
• exists <key>
• info [<section>]
• command
• dbsize
• flushdb/flushall
• compact: extended command to trigger a compaction manually.

Key/value pairs are persisted in log files under directory/urs/local/var/tinkv. The default listening address of server is127.0.0.1:7379, and you can connect to it with a redis client.

Quick Start

It’s very easy to installtinkv-server:

$cargoinstalltinkv

Start server with default config (set log level toinfo mode):

$tinkv-server-vv2020-06-24T13:46:49.341Z INFO  tinkv::store> open store path: /usr/local/var/tinkv2020-06-24T13:46:49.343Z INFO  tinkv::store> build keydir from data file /usr/local/var/tinkv/000000000001.tinkv.data2020-06-24T13:46:49.343Z INFO  tinkv::store> build keydir from data file /usr/local/var/tinkv/000000000002.tinkv.data2020-06-24T13:46:49.343Z INFO  tinkv::store> build keydirdone, got 0 keys. current stats: Stats{ size_of_stale_entries: 0,total_stale_entries: 0, total_active_entries: 0, total_data_files: 2, size_of_all_data_files: 0}2020-06-24T13:46:49.343Z INFO  tinkv::server> TinKV server is listening at'127.0.0.1:7379'

Communicate withtinkv-server by usingreids-cli:

About Compaction

Compation process will be triggered ifsize_of_stale_entries >= config::COMPACTION_THRESHOLD after each call ofset/remove. Compaction steps are very simple and easy to understand:

1. Freeze current active segment, and switch to another one.
2. Create a compaction segment file, then iterate all the entries inkeydir (in-memory hash table), copy related data entries into compaction file and updatekeydir.
3. Remove all the stale segment files.

Hint files (for fast startup) of corresponding data files will be generated after each compaction.

You can callstore.compact() method to trigger compaction process if nessesary.

usetinkv::{self,Store};fnmain()->tinkv::Result<()>{pretty_env_logger::init();letmutstore=Store::open("/path/to/tinkv")?;store.compact()?;Ok(())}

Structure of Data Directory

.tinkv├── 000000000001.tinkv.hint-- related index/hint file,forfast startup├── 000000000001.tinkv.data-- immutable data file└── 000000000002.tinkv.data-- active data file

Refs

Projects

I’m not familiar with erlang, but I found some implementations in other languages worth learning.

1. Go:prologic/bitcask
2. Go:prologic/bitraft
3. Python:turicas/pybitcask
4. Rust:dragonquest/bitcask

Found another simple key-value database based on Bitcask model, please referxujiajun/nutsdb.

Articles and more

• Implementing a Copyless Redis Protocol in Rust with Parsing Combinators
• Expected type parameter, found struct
• Help understanding how trait bounds workd
• [Idiomatic way to take ownership of all items in a Vec?](https://users.rust-lang.org/t/idiomatic-way-to-take-ownership-of-all-items-in-a-vec-string/7811/12)
• Idiomatic callbacks in Rust
• What are reasonable ways to store a callback in a struct?
• Things Rust doesn’t let you do

License

Licensed under theMIT license.