01
Append-Only Persistence
Every mutation is serialized and appended to disk so restart and crash recovery replay the log instead of losing the dataset.
WALdurability-first write path
Rust project case study · persistence + concurrency + networking
Concurrent key-value store,built to persist
Aryavarth KVS evolved from a simple in-memory store into a durable Rust engine with an append-only log, compaction, JSON-over-TCP client/server communication, and concurrent request execution through thread-pool strategies.
The current milestone is a single-node service. Making it distributed is the next phase, and that will be completed soon by implementing the Raft runtime.
O(1)index lookup via in-memory log pointers
JSON/TCPwire protocol for client and server
Raftnext distributed runtime phase
Current architectureStorage
Storage
engine.
The current implementation focuses on being a correct, durable, concurrent single-node store first. The design choices mirror real database concerns: write-ahead logging, indexing, compaction, protocol design, and execution strategy.
02
LogPointer Index
An in-memory `HashMap<String, LogPointer>` keeps key lookups direct while values remain in the append-only log on disk.
O(1)key lookup path
03
Compaction
Dead log entries created by overwrites and removals are reclaimed by rewriting only live commands into a new file.
1MBcompaction threshold in current code
04
Concurrent Service Layer
The storage engine sits behind a trait and is served over TCP using a thread-pool based execution model instead of unbounded threads.
3pool strategies explored
Request lifecycle
Persist.Index.Serve.
let serialized = serde_json::to_string(&cmd)?;
writeln!(inner.writer, "{}", serialized)?;
inner.writer.flush()?;RoadmapDistributed is
Distributed is
next.
This page now reflects the real state of the project: durable local storage, networked request handling, and concurrency are already implemented. Distributed consensus is the upcoming phase and will be completed soon by implementing the Raft runtime.
4phases
The system is being built in layers: persistence, concurrency, pluggable engines, then consensus replication. The ordering matters because Raft needs a solid storage and request execution foundation underneath it.
TCPCurrent network boundary
RaftUpcoming replication runtime
completed
Phase 1Persistent single-node enginecompleted
Phase 2Network service + concurrent executionactive
Phase 3Engine abstraction + sled comparisonnext
Phase 4Distributed replication with Raft runtimeIMPLEMENTED
Built around
primitives.
0
Core operations
Set, Get, Remove
0
Primary log write path
append-only persistence
0
Engine choices
custom KvStore and sled
Protocol
Requests and responses are modeled as Rust enums and serialized as JSON over TCP.
Concurrency
The design moved beyond thread-per-request to shared-queue and Rayon-backed pool strategies.
Abstraction
`KvsEngine` keeps the service layer decoupled from storage implementation details.
CLI surfaceRun the store.
Run the store.
Inspect the protocol.
The project spec defines a client and server CLI over a custom protocol. The storage layer is abstracted behind `KvsEngine`, so the same server flow can run with the handwritten `KvStore` or with `sled`.
kvs-server --addr 127.0.0.1:4000 --engine kvs
Starts the server and binds the selected engine.
kvs-client set <KEY> <VALUE>
Appends a Set request through the network boundary.
kvs-client get <KEY>
Reads through the service and prints the value or `Key not found`.
kvs-client rm <KEY>
Deletes a key and treats missing keys as an error.
Next stop:
distributed consensus.
The current milestone proves persistence, indexing, compaction, networking, and concurrent request handling. The next milestone is a distributed version backed by a Raft runtime so replication and leader-based coordination become first-class.
Single-node today. Raft-backed distribution next.