Repository files navigation

KiWi is a high-performance,RESPcompliant, key-value store inspired by the Bitcask paper, designedfor simplicity, reliability, and blazing-fast read/write operations.

• RESP protocol support for Redis-compatible client interaction.
• Non-blocking I/O server with Netty.
• High-performance key-value store based on the Bitcask storage model.
• In-memory indexing for fast reads.
• TTL-based key expiration.
• Checksums for data integrity.
• Compaction and efficient file merging process.
• Hint files for quick startup times.
• Tunable durability.

1. Start Docker container

   docker run --name kiwi -p 6379:6379 -v ./local/db:/var/lib/kiwi/data nemanjam/kiwi:latest

2. Connect to the server withredis-cli

   redis-cli -h localhost

3. Use the server as you would a Redis server.

   SET key valueOKGET key"value"EXISTS key(integer) 1DEL keyOKEXISTS key(integer) 0

• SET key value
• GET key
• DEL key
• EXISTS key
• FLUSHDB
• PING
• DBSIZE
• INFO

KiWi can be configured using environment variables or a HOCON configuration file.Refer toTypesafe Config for configuration examples.

Default values and environment variables:

• kiwi-core/application.conf
• kiwi-server/application.conf

• Java 21
• Docker (optional, for running via a container)

1. Clone the repository:

   git clone https://github.com/nemanjam/kiwi.gitcd kiwi

2. Build & install the project:

   ./gradlew installDist

3. Run the KiWi server:

   ./kiwi-server/build/install/kiwi-server/bin/kiwi-server

1. Build the Docker image

   docker build -t kiwi.

2. Run the container:

   docker run --rm --name kiwi -p 6379:6379 kiwi:latest

3. Connect to the server:

   redis-cli -h localhost -p 6379

KiWi uses CRC32 checksums to ensure data integrity. The checksum is stored alongside the data.There is a special command that can be used to verify the data integrity.

Checksum command uses available CPU cores to parallelize the checksum calculation. If checksumfails,the following error message will be displayed:

Checksum failed: segment=00000000000000000000 position=444636640 checksum=2005447726 timestamp=1733002903067 ttl=0 keySize=16 valueSize=0

docker run -it --rm -p 6379:6379 nemanjam/kiwi:latest sh checksum --dir [log dir] --threads [threads]

java -cp"kiwi-server/build/install/kiwi-server/lib/*" kiwi.core.checksum.Run --dir [log dir] --threads [threads]

KiWi can be evaluated withredis-benchmark utility command.

Below are the results of runningredis-benchmark with KiWi and Redis on a local setup (MacBook M3Pro with 18GB RAM and Sequoia 15.1.1).

redis-benchmark -h localhost -t set -n 100000 -r 10000000 -d 1024====== SET ======  100000 requests completed in 2.18 seconds  50 parallel clients  1024 bytes payload  keep alive: 1  host configuration "save":  host configuration "appendonly":  multi-thread: noSummary: throughput summary: 45934.77 requests per second latency summary (msec):          avg       min       p50       p95       p99       max        1.016     0.088     0.951     1.863     2.703    30.655

redis-benchmark -h localhost -t get -n 100000 -r 10000000 -d 1024====== GET ======  100000 requests completed in 1.83 seconds  50 parallel clients  1024 bytes payload  keep alive: 1  host configuration "save":  host configuration "appendonly":  multi-thread: noSummary: throughput summary: 60753.34 requests per second latency summary (msec):          avg       min       p50       p95       p99       max        0.661     0.088     0.639     0.967     1.311    12.255

JVM options:

-Xms2g -Xmx2g -XX:UseG1GC –XX:+UseStringDeduplication -XX:+AlwaysPreTouch

KiWi Configuration:

kiwi {storage {log {dir ="/tmp/kiwi"segment.bytes =1073741824// 1GBsync {mode ="periodic"periodic {interval =10s        }      }    }  }}

KiWi combines the simplicity of RESP with the efficient storage model described in the Bitcaskpaper. This architecture is designed for high performance and simplicity.

• All write operations are appended to a log file, ensuring sequential disk writes for maximumperformance.
• When the active log file reaches a configurable size, it is rolled over to a segment file.
• Periodically, segment files are compacted to remove stale data and reclaim disk space.
• Crash recovery is achieved by replaying the log files during startup.
• Disk I/O operations, like log compaction, are handled in background threads to avoid blockingclient requests.

• All keys are stored in an in-memory hash table, pointing to their location in the log file.
• This ensuresO(1) read performance while keeping the storage footprint minimal.

• Netty-based event loop for handling client requests.
• KiWi supports the RESP protocol, making it compatible with Redis clients and tools.

• KiWi provides tunable durability options to balance performance and data safety:
  • periodic (default): Writes are flushed to disk at regular intervals.
  • batch: Writes are batched and flushed when the batch window expires. All writers are blockeduntil the batch is written.
  • lazy: Flush is delegated to the operating system, which may delay writes for performance.

• Fast writes due to sequential disk I/O.
• Fast reads with O(1) lookups using the in-memory index.
• Simple and robust crash recovery with the data and hint files.
• Incremental crash-safe compaction process.

• The in-memory index requires all keys to fit in memory.
• Log compaction introduces periodic I/O overhead.

We welcome contributions to KiWi! Here’s how you can help:

1. Fork the repository.
2. Create a new branch for your feature or bugfix.
3. Submit a pull request with a clear description of your changes.

KiWi is licensed under the MIT License. SeeLICENSE for details.