Persistent, Scalable, and Replicated Distributed Key-Value Storage System

This three-person group project involved the creation of a distributed, replicated, and persistent key-value storage system. Development was conducted over the duration of an academic semester through a set of four milestones:

1. Milestone 1 required us to implement a persistent key-value storage system through a server-client architecture. We used Java to create a multi-threaded server using the Socket interface and adopted a BitCask-like storage mechanism to provide persistent data storage. A client-side CLI was also created with a set of predetermined commands that are used to interact with the storage service.
2. Milestone 2 added a distributed element to the first milestone by introducing multiple servers that are each responsible for a specific portion of key-value data, established via consistent hashing with MD5. The hash range that a server is responsible for is maintained in metadata, which is coordinated by a custom external configuration service (ECS). All servers that wish to be added or removed from the service rely on communication protocols with the ECS, which coordinates the appropriate data transfer protocols to rebalance data on metadata changes and broadcast metadata updates to the existing servers.
3. Milestone 3 introduced replication to the system, in which each server’s data was replicated to its two successors in the hashring. We adopted an eager primary-backup replication policy to ensure that data in the replicas were up-to-date with client requests. Replication also allowed us to implement failure detection and crash recovery procedures, through which data could be restored from a crashed server’s replicas. The failure detection feature itself was implemented through a heartbeat mechanism with ECS.
4. Milestone 4 allowed our team to develop a creative extension to the project. Our team chose to create various features related to load balancing. We created a framework to allow our service to scale up/down elastically by adding/removing idle server nodes based on the volume of data managed by each node. We also added gossiping protocols for failure detection and metadata updates, which offload responsibility and reduce bandwidth bottlenecks from the ECS.