高级检索

高性能K/V服务系统共识协议IRaft

IRaft: a consensus protocol for high performance K/V service system

  • 摘要: 传统单机系统的处理能力已经难以处理大数据时代日益增加的数据量,大规模分布式键值(K/V)存储系统成为新时代企业的核心基础设施之一。而K/V服务系统中的分布式共识协议是确保分布式环境下数据的一致性和高可用性的关键。现代硬件存储技术和高速网络技术的发展对K/V系统的性能提出了更高的要求,共识协议层逐渐成为约束系统性能的瓶颈。针对K/V 存储服务的高响应速度需求,提出了一种基于Raft(reliable and available distributed fault-tolerant consensus protocol)的高性能的K/V服务系统共识协议(improved Raft, IRaft):对Raft协议的复制状态机模型进行拓展,根据不同键的指令之间的可交换性,通过指令依赖关系分析,将状态机操作的全序关系放松为偏序关系,使得没有依赖关系的操作可以交换。通过设计待定序池、分区日志、快速完成与常规完成判定结构,实现无依赖指令的批量提交,支持无依赖指令在上层应用中的并行执行。通过实现基于Raft和基于IRaft的K/V存储系统进行多组对比实验,实验结果表明基于IRaft协议的系统总体上具有更优的性能。

     

    Abstract: The processing power of traditional standalone systems is no longer sufficient to handle the increasing amount of data in the era of big data. Large scale distributed key value (K/V) storage systems have become one of the core infrastructure for enterprises in the new era. The distributed consensus protocol in the K/V service system is the key to ensuring data consistency and high availability in a distributed environment. The development of modern hardware storage technology and high-speed network technology has put forward higher requirements for the performance of K/V systems, and the consensus protocol layer has gradually become a bottleneck that constrains the system performance. In response to the high response speed requirements of K/V storage services, this paper proposes a high-performance consensus protocol for K/V service systems based on the reliable and available distributed fault-tolerant consensus protocol (Raft) , improved Raft, IRaft: the replication state machine model of the Raft protocol is extended, and based on the commutativity among instructions of different keys, the total order relation of state machine operations is relaxed to a partial order relation through instruction dependency analysis, so that operations without dependency relationships can be swapped. By designing a Pending Order Pool, Partitioned Log, and a fast completion and regular completion decision structure, batch submission of independent instructions is achieved, supporting the parallel execution of independent instructions in upper level applications. By conducting multiple comparative experiments on K/V storage systems based on Raft and IRaft protocols, the experimental results show that the system based on IRaft protocol generally has better performance.

     

/

返回文章
返回