Evaluation of an Efficient Ring-Based Total Order Protocol in a Fairness-Controlled Environment

1. Introduction

A crash-tolerant system is designed to continue functioning despite a threshold number of crashes occurring and is crucial to maintaining high availability of systems [1,2,3,4]. Replication techniques have made it possible to create crash-tolerant distributed systems. Through replication, redundant service instances are created on multiple servers, and a given service request is executed on all servers so that even if some servers crash, the rest will ensure that the client receives the response. The set of replicated servers hosting service instances may also be referred to as a replicated group and simply as a group. Typically, a client can send its service to any one of the redundant servers in the group and the server that receives a client request, in turn, disseminates it within the group so all can execute. Thus, different servers can receive client requests in different order, but despite this, all servers must process client requests in the same order [5,6]. To accomplish this, a total order mechanism that employs logical clock is utilised to guarantee that replicated servers process client requests or simply messages in the same order [7]. A logical lock is a mechanism used in distributed systems to assign timestamps to events, enabling processes to establish a consistent order of events occurring. Thus, a total order protocol is a procedure used within distributed systems to achieve agreement on the order in which messages are delivered to all process replicas in the group. This protocol ensures that all replicas receive and process the messages in the same order, irrespective of the order in which they were initially received by individual replicas. While total order protocols play a critical role in maintaining consistency and system reliability, achieving crash tolerance requires implementation of additional mechanisms. One such mechanism, as defined in our work, is the crashproofness policy. Specifically, this policy dictates that a message is deemed crashproof and safe for delivery once it has successfully reached at least f+1 operative processes, where f is the maximum tolerated failures in a group.

Total order protocols in the literature broadly fall into two categories: Leader-based and leaderless. In the leader-based protocol, every client request is routed through the leader which coordinates the request replication and responds to the clients with the results of execution. Examples include Apache Zookeeper [10,11], Chubby [12,14], Paxos [15], View-stamp replication [25] and Raft [9,16,18]. Ring-based protocols are a class of leaderless protocols whose nodes are arranged in a logical ring. They ensure that all messages are delivered by all processes in the same order, regardless of how they were generated or sent by the sender. An example includes LCR [19], FSR [22], E-Paxos [20], and S-Paxos [21]. Our study focuses specifically on the ring-based leaderless protocols. Total order protocols are widely applicable to distributed systems, especially in applications requiring strong consistency and high throughput. For instance, they are utilised to coordinate transactions in massive in-memory database systems [17,23] where achieving minimal latencies despite heavy load is critical.

However, despite the progress made in ring-based protocols like LCR, certain design choices may lead to increased latency. The Logical Clock and Ring (LCR) protocol utilizes vector clocks where each process, denoted as $P_i$, maintains its own clock as $\mathrm{V}{\mathrm{C}}_{\mathrm{i}}=\mathrm{v}{\mathrm{c}}_{\mathrm{k},\mathrm{k}=0,\dots \mathrm{N}-1}$. A vector clock is a tool used to establish the order of events within a distributed system which can be likened to an array of integers, with each integer corresponding to a unique process in the ring. In the LCR protocol, processes are arranged in a logical ring, and the flow of messages is unidirectional as earlier described. However, LCRs’ design may lead to performance problems, particularly when multiple messages are sent concurrently within the cluster: firstly, it uses a vector timestamp for sequencing messages within replica buffers or queues [28], and secondly, it uses a fixed idea of "last" process to order concurrent messages. Thus, in the LCR protocol, the use of a vector timestamp takes up more space in a message, increasing its size.

Consequently, the globally fixed last process will struggle to rapidly sequence multiple concurrent messages, potentially extending the message-to-delivery average maximum latency. The size of a vector timestamp is directly proportional to the number of process replicas in a distributed cluster. Hence, if there are N processes within a cluster, each vector timestamp will consist of N counters or bits. As the number of processes increases, larger vector timestamps must be transmitted with each message, leading to higher information overhead. Additionally, maintaining these timestamps across all processes requires greater memory resources. These potential drawbacks can become significant in large-scale distributed systems, where both network bandwidth and storage efficiency are critical. Thirdly, in the LCR protocol, the assumption $\mathrm{N}=\mathrm{f}+1$ implies that $\mathrm{f}=\mathrm{N}-1$, where f represents the maximum number of failures the system can tolerate. This configuration results in a relatively high f, which can delay the determination of a message as crashproof. While the assumption $\mathrm{N}=\mathrm{f}+1$ is practically valid, it is not necessary for f to be set at a high value. Reducing f can enhance performance by lowering the number of processes required to determine the crashproofness of a message.

Prompted by the above potential drawbacks in LCR, a new total order protocol was design with $\mathrm{N}$, $\mathrm{N}>2$, processes arranged in a unidirectional logical ring where $\mathrm{N}$ is the number of processes within the server clusters. Messages are assumed to pass among processes in a clockwise direction as shown in Figure 1. If a message originates from $P_0$, it moves to ${\mathrm{P}}_1$until it gets to ${\mathrm{P}}_3$ which is the last process for ${\mathrm{P}}_0$. The study aims to achieve the following objectives: (i) Optimize message timestamping with Lamport logical clocks, which uses a single integer to represent message timestamps. This approach is independent of N, the number of processes in the communication cluster, unlike the vector timestamping used in LCR, which is dependent on N. In LCR, as N increases, the size of the vector timestamp grows, leading to information overhead. By contrast, Lamport's clock maintains a constant timestamp size, reducing complexity and improving efficiency (ii) Dynamically determines the "Last" Process for ordering concurrent messages. Instead of relying on a globally fixed last process for ordering concurrent messages, as in LCR, this study proposes a dynamically determined last process based on proximity to the sender in the opposite direction of message flow. This adaptive mechanism improves ordering flexibility and enhances system responsiveness under high workloads. (iii) Reduce message delivery latency. This study proposes reducing the value of f to (N – 1)/2 as a means to minimize overall message delivery latency and enhance system efficiency. This contrasts with the LCR approach, where f is set to N – 1. Specifically, when $\mathrm{f}=\mathrm{N}-1,$a message must be received by every process in the cluster before it can be delivered. Under high workloads or in the presence of network delays, this requirement introduces significant delays, increasing message delivery latency and impacting system performance. The goal of this study was accomplished using three methods: First, we considered a set of restricted crash assumptions: each process crashes independently of others and at most $\mathrm{f}$processes involved in a group communication can crash. A group is a collection of distributed processes in which a member process communicates with other members only by sending messages to the full membership of the group [8]. Hence, the number of crashes that can occur in an N process cluster is bounded by $f=⌊{\displaystyle \frac{\mathrm{N}-1}{2}}⌋$, where $⌊x⌋$ denotes the largest integer $\le x$. The parameter $\mathrm{f}$ is known as the degree of fault tolerance as described in Raft [9]. As a result, at least two processes are always operational and connected. Thus, an Eventually Perfect Failure Detector (♦P) was assumed in this study’s system model, operating under the assumption that N = 2f + 1 nodes are required to tolerate up to f crash failures. This approach enables the new protocol to manage temporary inaccuracies, such as false suspicions, by waiting for a quorum of at least f + 1 nodes before making decisions. This ensures that the system does not advance based on incorrect failure detections. Secondly, the last process of each sender is designated to determine the stability of messages. It then communicates this stability by sending an acknowledgement message to other processes. When the last process of the sender receives the message, it knows that all the logical clocks within the system have exceeded the timestamp of the message (stable property). Then all the received messages whose timestamp is less than the last process logical clock can then be totally ordered.

In addition, a new concept of "deliverability requirements" was introduced to guarantee the delivery of only crash-proof and stable messages in total order. A message is crashproof if the number of messages $\mathrm{h}\mathrm{o}\mathrm{p}\mathrm{s}\ge \mathrm{f}+1$, that is, a message must make at least $\mathrm{f}+1$ number of hops before it is termed crashproof. Thus, the delivery of a message is subject to meeting both deliverability and order requirements. As a result of enhancements made in this regard, a new leaderless ring-based total order protocol was designed, known as the Daisy Chain Total Order Protocol (DCTOP) [13]. Thirdly, fairness is defined as the condition where every process $P_i$ has an equal chance of having its sent messages eventually delivered by all processes within the cluster. Every process ensures messages from the predecessor are forwarded in the order they were received before sending their own message. Therefore, no process has priority over another during sending of messages.

1.1. Contributions

The contributions of this paper can be summarized as follows:

(i)

Protocol-Level Innovations Within a Ring-Based Framework: This study introduces DCTOP, a novel improvement to the classical LCR protocol while retaining its ring-based design. It introduces:

• Lamport Logical Clock used for message timestamping which achieves efficient concurrent message ordering, reducing latency and improving fairness.
• Dynamic Last-Process Identification to replace LCR’s globally fixed last process assumption, accelerating message stabilization and accelerates delivery.

(ii)

Relaxed Failure Assumption: DCTOP reduces the fault tolerance threshold from N = f + 1 to N = 2f + 1, enabling faster message delivery with fewer failures.

(iii)

Foundation for Real-World Deployment: While simulations excluded failures and large-scale setups, ongoing work involves a cloud-based, fault-tolerant implementation to validate DCTOP under practical conditions.

This paper is structured as follows: Section 2 presents the system model, while Section 3 outlines the design objectives and rationale for DCTOP. Section 4 details the fairness control primitives. Section 5 provides performance comparisons of DCTOP, LCR and Raft in terms of latency and throughput under crash-free and high-workload conditions. Finally, Section 6 concludes the paper.

2. System Model

The ring-based protocols are modelled as a group of $\mathrm{N}$processes represented by $\mathsf{\Pi }=\left\{P_0,P_1,P_2,\dots .,P_{N-1}\right\}$ which are linked together in a circular structure (see Figure 1) with varying cluster sizes with an asynchronous-based communication framework, with no constraints on communication delays and exponentially distributed intervals between message transmissions. This model supports first in first out (FIFO), and thus messages sent are received in the order sent. The system model restricts a process, $P_i$, to only send messages to its clockwise neighbour and receive from its anticlockwise neighbour.

Thus, for each process $P_i$, where $0\le i\le N-1$ and $n$ is the number of processes in the cluster, the clockwise neighbour $\left({CN}_i\right)$ is defined as the process immediately following $P_i$, ${CN}_i=P_{i+1}$or ${{CN}_i=P}_0$ if $i=\mathrm{N}-1$. Conversely, the anticlockwise neighbour of $P_i$ $\left(A{CN}_i\right)$ is defined as the process immediately preceding $P_i$, ${A{CN}_i=P}_{i-1}$ or$A{CN}_i=P_{N-1}$ if $i=0$. Therefore, messages are transmitted exclusively in the clockwise direction, with $P_i$ receiving from $A{CN}_i$ and transmitting to ${CN}_i$ in a daisy chain framework. We also defined the Stability clock of any process $\left({SC}_i\right)$ as the largest timestamp, $ts$, known to any process $P_i$ as stable. When a message $m$ becomes stable, ${SC}_i$ is updated as follows: ${SC}_i$ = max{ ${SC}_i$, $m\_ts$ }. Additionally, we introduce the definition of ${Hops}_{i,j}$ which is defined as the number of hops between any two processes from ${\mathrm{P}}_{\mathrm{i}}$ to ${\mathrm{P}}_{\mathrm{j}}$ in the clockwise direction: (i) ${\mathrm{H}\mathrm{o}\mathrm{p}\mathrm{s}}_{\mathrm{i},\mathrm{j}}=0$, $\mathrm{i}\mathrm{f}\mathrm{i}=\mathrm{j}$. (ii) ${\mathrm{H}\mathrm{o}\mathrm{p}\mathrm{s}}_{\mathrm{i},\mathrm{j}}=\left(\mathrm{j}-\mathrm{i}\right),\mathrm{i}\mathrm{f}\mathrm{j}>\mathrm{i}$, and (iii) ${\mathrm{H}\mathrm{o}\mathrm{p}\mathrm{s}}_{\mathrm{i},\mathrm{j}}=\left(\mathrm{j}+\mathrm{N}-\mathrm{i}\right)\mathrm{i}\mathrm{f}\mathrm{j}<\mathrm{i}$.

3. Daisy Chain Total Order Protocol- DCTOP

The DCTOP system employs a group of interconnected process replicas, with a group size of $\mathit{N}$, where $\mathit{N}$ is an odd integer, $\mathit{N}\ge 3$ and at most 9, to provide replicated services. The main goals of the system design are threefold:

(a)

First, to improve the latency of LCR by utilizing Lamport logical clocks $\left(LC\right)$ for sequencing concurrent messages.

(b)

Second, to employ a novel concept of the dynamically determined “last” process for ordering concurrent messages, while ensuring optimal achievable throughput.

(c)

Third, the relaxation of the crash failure assumption in LCR.

3.1. Data Structures

The data structures associated with each process $P_i$, message m, and the µ message are discussed in this section as used in DCTOP system design and simulation experiment:

Each process $P_i$ has the following data structures:

• Logical clock $\left({LC}_i\right)$: This is an integer object initialized to zero. It is used to timestamp messages.
• Stability clock $\left({SC}_i\right)$: This is an integer object that holds the largest timestamp, $ts$, known to ${\mathrm{P}}_{\mathrm{i}}$ as stable. Initially, ${SC}_i$ is zero.
• Message Buffer (${mBuffer}_i$): This field holds the sent or received messages by ${\mathrm{P}}_{\mathrm{i}}$.
• Delivery Queue (${DQ}_i)$: Messages waiting to be delivered are queued in this queue object.
• Garbage Collection Queue (${GCQ}_i)$: After a message is delivered, the message is transferred to ${GCQ}_i$ to be garbage collected.

M is used to denote all types of messages used by the protocol. Usually there are two types of M: data message denoted by m, and an announcement or ack message that is bound to a specific data message. The latter is denoted as µ(m) when it is bound to m. µ(m) is used to announce that m has been received by all processes in $\mathsf{\Pi }$. The relationship between m and its counterpart µ(m) is shown in Figure 2.

A message, m, consists of a header and a body, with the body containing the data application information. Every m has a corresponding µ, denoted as µ$\left(\mathrm{m}\right)$, which contains the information from m's header. This is why we refer to µ$\left(\mathrm{m}\right)$ instead of just µ. µ$\left(\mathrm{m}\right)$ has m header information as its main information and does not contain its own data; therefore, the body of µ$\left(m\right)$ is essentially m's header (see Figure 2).

A message M has at least the following data structures:

• Message origin$(M\_origin)$ field shows the id of the process in $\mathsf{\Pi }=\left\{{\mathrm{P}}_0,{\mathrm{P}}_1,{\mathrm{P}}_2,\dots .,{\mathrm{P}}_{\mathrm{N}-1}\right\}$ that initiated the message multicast.
• Message timestamp$(M\_ts)$field holds the timestamp given to M by M_origin.
• Message destination$(M\_destn)$ field holds the destination of M which is the CN of the process that sends/forwards M.
• Message flag (M_flag) it is a Boolean field which can be true or false and is initiated to be false when M is formed.

3.2. DCTOP Principles

The protocol has three design aspects:(i) message sending, receiving, and forwarding, (ii) timestamp stability, and (iii) crashproofing of messages, which are described in detail one by one in this subsection.

(1)

Message Sending, Receiving and Forwarding: The Lamport logical clock is used to timestamp a message m within the ring network before m is sent. Therefore, $m\_ts$ denotes the timestamp for message $m.$.

The system as shown in Figure 3 uses two main threads to handle message transmission and reception in a distributed ring network. The send(m) thread operates by dequeuing a message m from the non-empty SendingQueue_i when allowed by the transmission control policy (see Section 4). It timestamps the message with the current value of the LC_i as $\mathrm{m}\_\mathrm{t}\mathrm{s}={\mathrm{L}\mathrm{C}}_{\mathrm{i}}$, increments LC_i by one afterwards, and then places the timestamped message into the OutgoingQueue_i for transmission. A copy of the message is also stored in mBuffer_i for local record.

On the receiving side, the receive(m) thread dequeues a message m from the IncomingQueue_i when permitted by the transmission control policy, updates the ${\mathrm{L}\mathrm{C}}_{\mathrm{i}}$ as ${\mathrm{L}\mathrm{C}}_{\mathrm{i}}=\mathrm{m}\mathrm{a}\mathrm{x}\{\left(\mathrm{m}\_\mathrm{t}\mathrm{s}+1\right),{\mathrm{L}\mathrm{C}}_{\mathrm{i}}\}$, and delivers the message to process P_i for further handling. Typically, m is entered in ${\mathrm{m}\mathrm{B}\mathrm{u}\mathrm{f}\mathrm{f}\mathrm{e}\mathrm{r}}_{\mathrm{i}}$ and may be forwarded if necessary to ${\mathrm{C}\mathrm{N}}_{\mathrm{i}}$ by entering a copy of m with destination set to ${\mathrm{C}\mathrm{N}}_{\mathrm{i}}$ into the ${\mathrm{O}\mathrm{u}\mathrm{t}\mathrm{g}\mathrm{o}\mathrm{i}\mathrm{n}\mathrm{g}\mathrm{Q}\mathrm{u}\mathrm{e}\mathrm{u}\mathrm{e}}_{\mathrm{i}}$. If necessary, meaning the message has not yet completed a full cycle around the ring, it may be forwarded to the CN_i by placing a copy of it in the OutgoingQueue_i with its destination set to ${\mathrm{C}\mathrm{N}}_{\mathrm{i}}$. However, once the message has completed a full cycle within the ring network, it is no longer forwarded, and the forwarding process stops.

If two messages are received consecutively, they are sent in the same order but not necessarily immediately after each other, depending on the transmission control policy. As shown in Figure 3, messages to be received arrive at the ${ IncomingQueue}_i$ from the network, and a copy of the received message arrives at the ${ mBuffer}_i$ while a copy of the forwarded messages appears in the ${ OutgoingQueue}_i$ according to the order they were received.

(2)

Timestamp Stability: A message timestamp TS, $TS\ge 0$, is said to be stable in a given process ${\mathrm{P}}_{\mathrm{i}}$ if and only if the process ${\mathrm{P}}_{\mathrm{i}}$ is guaranteed not to receive any $m\text{'}$, $m\text{'}\_ts \le \mathrm{T}\mathrm{S}$ any longer.

Observations:

1)

A timestamp $\mathrm{T}\mathrm{S}\mathrm{\text{'}}<\mathrm{T}\mathrm{S}$ is also stable in ${\mathrm{P}}_{\mathrm{i}}$ when TS becomes stable in ${\mathrm{P}}_{\mathrm{i}}$.

2)

The term “stable” is used to refer to the fact that once TS becomes stable in ${\mathrm{P}}_{\mathrm{i}}$, it remains stable for ever. This usage corresponds to that of “stable” property used by Chandy and Lamport [24]. Therefore, the earliest instance when a given TS becomes stable in ${\mathrm{P}}_{\mathrm{i}}$ will be the interest in the later discussions.

3)

When TS becomes stable in ${\mathrm{P}}_{\mathrm{i}}$, the process can potentially total order (TO) deliver all received but undelivered $m, m\_ts \le \mathrm{T}\mathrm{S}$, because stability of TS eliminates the possibility of ${\mathrm{P}}_{\mathrm{i}}$ ever receiving any $m\text{'}$, $m\text{'}\_ts \le \mathrm{T}\mathrm{S}$, in the future.

(3)

Crashproofing of Messages: A message m is crashproof if m is in possession of at least ($f+1$) processes. Therefore, a message m is crashproof in ${\mathrm{P}}_{\mathrm{i}}$ when ${\mathrm{P}}_{\mathrm{i}}$ knows that m has been received by at least ($f+1$) processes. The rationale for crashproofing is that when we have at least $f+1$ processes that have received a given message m even if $f$of them crash there will be at least one process that can be relied on in sending m to others and this emphasizes the importance of crashproofness in our system.

3.3. DCTOP Algorithm Main Points

The DCTOP algorithm's main points are outlined as follows:

• When ${\mathrm{P}}_{\mathrm{i}}$ forms and sends m, it sets m_flag = false, before it deposits m in its ${\mathrm{m}\mathrm{B}\mathrm{u}\mathrm{f}\mathrm{f}\mathrm{e}\mathrm{r}}_{\mathrm{i}}$.
• When ${\mathrm{P}}_{\mathrm{i}}$ receives m and ${\mathrm{P}}_{\mathrm{j}}$= m_origin
  • It checks if ${Hops}_{j, i}$ ≥ f. If this is true then m is crashproofed, it does not deliver m immediately. Moreover, it sets m_flag = true and deposits m in its ${\mathrm{m}\mathrm{B}\mathrm{u}\mathrm{f}\mathrm{f}\mathrm{e}\mathrm{r}}_{\mathrm{i}}$. if m is not crashproofed, then m_flag remains false.
  • It then checks if ${\mathrm{P}}_{\mathrm{j}}$ ≠ ${\mathrm{C}\mathrm{N}}_{\mathrm{i}}$. if this is true, it sets m destination, m_destn =${\mathrm{C}\mathrm{N}}_{\mathrm{i}}$ and deposits m in its ${\mathrm{O}\mathrm{u}\mathrm{t}\mathrm{g}\mathrm{o}\mathrm{i}\mathrm{n}\mathrm{g}\mathrm{Q}\mathrm{u}\mathrm{e}\mathrm{u}\mathrm{e}}_{\mathrm{i}}$,
  • Otherwise, m is stable then it updates ${\mathrm{S}\mathrm{C}}_{\mathrm{i}}$ as $S{\mathrm{C}}_{\mathrm{i}}$ = max { $S{\mathrm{C}}_{\mathrm{i}}$, m_ts}, and transfer all m, m_ts ≤ ${\mathrm{S}\mathrm{C}}_{\mathrm{i}}$ to ${\mathrm{D}\mathrm{Q}}_{\mathrm{i}}$. Then, it forms µ(m), sets the two header fields, µ(m)_origin=${\mathrm{P}}_{\mathrm{i}}$, µ(m)_destn=${\mathrm{C}\mathrm{N}}_{\mathrm{i}}$ and deposit µ(m) in ${\mathrm{O}\mathrm{u}\mathrm{t}\mathrm{g}\mathrm{o}\mathrm{i}\mathrm{n}\mathrm{g}\mathrm{Q}\mathrm{u}\mathrm{e}\mathrm{u}\mathrm{e}}_{\mathrm{i}}.$
• When ${\mathrm{P}}_{\mathrm{i}}$ receives µ(m), it knows that every process has received m.
  • If m in µ(m) does not indicate a higher stabilisation in ${\mathrm{P}}_{\mathrm{i}}$, that is, m_ts ≤ ${\mathrm{S}\mathrm{C}}_{\mathrm{i}}$ and ${Hops}_{j, i}$ ≥ f then ${\mathrm{P}}_{\mathrm{i}}$ ignores µ(m), otherwise, if ${Hops}_{j, i}$ < f, ${\mathrm{P}}_{\mathrm{i}}$ sets m_flag= true, µ(m)_destn = ${\mathrm{C}\mathrm{N}}_{\mathrm{i}}$ and deposit µ(m) in ${OutgoingQueue}_i.$
  • However, if m in µ(m) indicates a higher stabilisation in ${\mathrm{P}}_{\mathrm{i}}$, that is, m_ts > ${\mathrm{S}\mathrm{C}}_{\mathrm{i}}$, ${\mathrm{P}}_{\mathrm{i}}$ updates ${\mathrm{S}\mathrm{C}}_{\mathrm{i}}$ as $S{\mathrm{C}}_{\mathrm{i}}$ = max { $S{\mathrm{C}}_{\mathrm{i}}$, m_ts}, and transfer all m, m_ts ≤ ${\mathrm{S}\mathrm{C}}_{\mathrm{i}}$ to ${\mathrm{D}\mathrm{Q}}_{\mathrm{i}}$.
  • If ${\mathrm{P}}_{\mathrm{j}}$= ${\mathrm{C}\mathrm{N}}_{\mathrm{i}}$, ${\mathrm{P}}_{\mathrm{i}}$ ignores µ(m) otherwise, it sets µ(m)_destn = ${\mathrm{C}\mathrm{N}}_{\mathrm{i}}$ and deposit µ(m) in ${OutgoingQueue}_i.$
• Whenever ${\mathrm{D}\mathrm{Q}}_{\mathrm{i}}$ is non-empty, ${\mathrm{P}}_{\mathrm{i}}$ deques m from the head of ${\mathrm{D}\mathrm{Q}}_{\mathrm{i}}$ and delivers $m$ to application process. ${\mathrm{P}}_{\mathrm{i}}$ then enters a copy of $m$ into ${\mathrm{G}\mathrm{C}\mathrm{Q}}_{\mathrm{i}}$ to represent a successful TO delivery. This action is repeated until ${\mathrm{D}\mathrm{Q}}_{\mathrm{i}}$ becomes empty.

It is important to note that the DCTOP maintains total order. Thus, if ${\mathrm{P}}_{\mathrm{i}}$ forms and sends $m$ and then $m’$: (i) Every process receives $m$ and then $m’$ (ii) µ($m$) will be formed and sent before µ($m’$) and (iii) Any process that receives both µ($m$) and µ($m’$) will receive µ($m$) and then µ($m’$).

3.4. DCTOP Delivery Requirements

Any message m can be delivered to the high-level application process by ${\mathrm{P}}_{\mathrm{i}}$ if satisfies the following two requirements:

(i)

m_ts must be stable in ${\mathrm{P}}_{\mathrm{i}}$

(ii)

m must be crashproof in ${\mathrm{P}}_{\mathrm{i}}$, and

(iii)

Any two stable and crashproofed $m$, and $m’$are delivered in total order: $m$ is delivered before $m’$ $\mathrm{i}\mathrm{f}\mathrm{f}$ $\mathrm{m}\_\mathrm{t}\mathrm{s}<\mathrm{m}\mathrm{’}\_\mathrm{t}\mathrm{s}$ or $\mathrm{m}\_\mathrm{t}\mathrm{s}=\mathrm{m}\mathrm{’}\_\mathrm{t}\mathrm{s}$ and $\mathrm{m}\_\mathrm{o}\mathrm{r}\mathrm{i}\mathrm{g}\mathrm{i}\mathrm{n}>\mathrm{m}\mathrm{’}\_\mathrm{o}\mathrm{r}\mathrm{i}\mathrm{g}\mathrm{i}\mathrm{n}.$

During the delivery of m, if $\mathrm{m}\_\mathrm{t}\mathrm{s}=\mathrm{m}\mathrm{’}\_\mathrm{t}\mathrm{s}$ then the messages are ordered according to the origin of the messages, usually a message from ${\mathrm{P}}_{\mathrm{N}-1}$ are ordered before a message from ${\mathrm{P}}_{\mathrm{i}}$ where $\mathrm{N}-1>i$.

In summary, the pseudocode representations for total order message delivery, message communication, and membership changes are presented in Figure 4, Figure 5, and Figure 6 respectively.

3.5. Group Membership Changes

The DCTOP protocol is built on top of a group communication system [26,27]. Membership of the group of processes executing DCTOP can change due to (i) a crashed member being removed from the ring and/or (ii) a former member recovering and being included in the ring. Let G represent the group of DCTOP processes executing the protocol at any given time. G is initially $\mathsf{\Pi }=\left\{{\mathrm{P}}_0,{\mathrm{P}}_1,{\mathrm{P}}_2,\dots .,{\mathrm{P}}_{\mathrm{N}-1}\right\}$ and G$\subseteq \mathsf{\Pi }$ is always true. The membership change procedure is detail in Figure 6. Note that the local membership is assumed to send an interrupt to the local DCTOP process, say, ${\mathrm{P}}_{\mathrm{i}}$ when a membership change is imminent. On receiving the interrupt ${\mathrm{P}}_{\mathrm{i}}$ completes processing of any message it has already started processing and then suspends all DCTOP activities and waits for new G' to be formed: sending of m or µ(m) (by enqueueing into ${\mathrm{S}\mathrm{e}\mathrm{n}\mathrm{d}\mathrm{i}\mathrm{n}\mathrm{g}\mathrm{Q}\mathrm{u}\mathrm{e}\mathrm{u}\mathrm{e}}_{\mathrm{i}}$), receiving of m or µ(m) (from ${\mathrm{I}\mathrm{n}\mathrm{c}\mathrm{o}\mathrm{m}\mathrm{i}\mathrm{n}\mathrm{g}\mathrm{Q}\mathrm{u}\mathrm{e}\mathrm{u}\mathrm{e}}_{\mathrm{i}}$) and delivering of m (from ${\mathrm{D}\mathrm{Q}}_{\mathrm{i}}$) are all suspended. The group membership change work as follows: Each process P_i​ in the set of Survivors(G) (i.e., survivors of the current group G) exchanges information about the last message they TO-delivered. Once this exchange is complete, additional useful information is derived among all Survivors, which helps identify the Lead Survivor.

Subsequently, each Survivor sends all messages from its respective SendingQueue to the other Survivors. If P_i​ has any missing messages, they are sent to another Survivor, P_s where P_s represents any Survivor process other than P_i​. After sending, P_i transmits a Finished_i message to all P_s processes, signalling that it has completed its sending. Upon receiving messages, P_i ​ stores all non-duplicate messages in its buffer, mBuffer_i​. The receipt of Finished_s messages from all P_s ​processes confirms that P_i has received all expected messages, with duplicates discarded. P_i then waits to receive Ready_s from every other P_s​, ensuring that every Survivor P_s ​ has received the messages sent by P_i ​. At this point, all messages in mBuffer_i are stable and can be totally ordered. If there are Joiners (defined as incoming members of G' that were not part of the previous group (Gprev) but joined G' after recovering from an earlier crash), the Lead Survivor sends its checkpoint state and TO_Queue to each P_j in the set of NewComer(G'), allowing them to catch up with the Survivors(G). Following this, all Survivors(G) resume TO delivery in Gprev​. P_i then sends a completed_i ​message to every process in G, indicating that it has finished TO-delivering in Gprev. Each Survivor waits to receive a completed_k message from every other P_k ​ in G before resuming DCTOP operations in the new G'. The Joiners, after replicating the Lead Survivor's checkpoint state, also perform TO delivery of messages in Gprev and then resume operations in the new G' of DCTOP. Hence, at the conclusion of the membership change procedure, all buffers and queues are emptied, ensuring that all messages from Gprev​ have been fully processed.

3.6. Proof of Correctness

Lemma 1 (VALIDITY).

If any correct process $P_i$ utoMulticasts a message m, then it eventually utoDelivers m.

Proof: 

Let ${\mathrm{P}}_{\mathrm{i}}$ be a correct process and let m_i be a message sent by ${\mathrm{P}}_{\mathrm{i}}$. This message is added to mBuffer_i (Line 15 of Figure 5) . There are two cases to consider:

Case 1:

Presence of membership change

If there is a membership change, ${\mathrm{P}}_{\mathrm{i}}$ will be in Survivor(G) since ${\mathrm{P}}_{\mathrm{i}}$ is a correct process. Consequently, the membership changes steps ensure that ${\mathrm{P}}_{\mathrm{i}}$ will deliver all messages stored in its mBuffer_i , TO_Queue_i or GCQ_i including m_i (Line 32 to 44 of Figure 5). Thus, ${\mathrm{P}}_{\mathrm{i}}$ utoDelivers message m_i that it sent.

Case 2:

No membership changes

When there is no membership change, all the processes within the DCTOP system including the m_i_origin will eventually deliver m_i after setting m_i stable (Line 28 of Figure 5)_. This happens because when ${\mathrm{P}}_{\mathrm{i}}$ timestamp, sets m_i_flag=false and sends m_i to its CN_i, it deposits a copy of m_i to its mBuffer_i and sets LC_i > m_i_ts afterward. The message is forwarded along the ring network until the ACN_i receives m_i. Any process that receives m_i deposits a copy of it into their mBuffer and sets LC > m_i_ts. It also checks if Hops_ij ≥ f, then m_i is crashproof and it sets m_i_flag=true. The ACN_i sets m_i stable (Line 28 of Figure 5) and crashproof (Line 20 of Figure 5) at ACN_i, transfers m_i to DQ and then it attempts utoDeliver m_i (Lines 1 to 8 of Figure 4) if m_i is at the head of DQ. ACN_i generates, timestamp µ(m_i) using its LC and then sends it to its own CN. Similarly, µ(m_i) is forwarded along the ring (Line 31 of Figure 5) until the ACN of µ(m_i)_origin receives µ(m_i). When any process receives µ(m_i) and Hops_ij <f, it knows that m_i is crash proof and stable but if Hops_ij ≥f, then m_i is only stable because m_i is already known to be crashproof since at least f+1 processes had already received m_i. Any process that receives µ(m_i) transfers m_i from mBuffer to DQ and then attempts to utoDeliver m_i if m_i is at the head of DQ.

Suppose P_k sends m_k before receiving m_i, $i<k$. Consequently, ACN_i will receive m_k before it receives m_i and thus before sending µ(m_i) for m_i. As each process forwards messages in the order in which it receives them, we know that ${\mathrm{P}}_{\mathrm{i}}$ will necessarily receive m_k before receiving µ(m_i) for message m_i.

(a)

If m_i_ts = m_k_ts, then ${\mathrm{P}}_{\mathrm{i}}$ orders m_k before m_i in mBuffer_i since $i<k$ (This study assumed that when messages have equal timestamp, message from a higher origin is ordered before message from a lower origin.). When ${\mathrm{P}}_{\mathrm{i}}$ receives µ(m_i) for message m_i it transfers both messages to DQ and can utoDeliver both messages, m_k before m_i, because TS is already known to be stable because of TS equality.

(b)

If m_i_ts < m_k_ts then ${\mathrm{P}}_{\mathrm{i}}$ orders m_i before m_k in mBuffer_i. When ${\mathrm{P}}_{\mathrm{i}}$ receives µ(m_i) for message m_i it transfers both messages to DQ and can utoDeliver m_i only since it is stable and is at the head of DQ. ${\mathrm{P}}_{\mathrm{i}}$ will eventually utoDeliver m_k when it receives µ(m_k) for m_k since it is now at the head of DQ after m_i delivery.

(c)

Option (a) or (b) is applicable in any other processes within the DCTOP system since there is no membership changes. Thus, if any correct process ${\mathrm{P}}_{\mathrm{i}}$ sends a message m, then it eventually delivers m.

Note that if f+1 processes receive a message m, then m is crash proof and during concurrent multicast, TS can become stable quickly making m to be delivered even before the ACN of the m_origin receives m.

Lemma 2 (INTEGRITY).

For any message m, any process P_k utoDelivers m at most once, and only if m was previously utoMulticast by some process ${\mathrm{P}}_{\mathrm{i}}$.

Proof. 

The crash failure assumption in this study ensures that no false message is ever utoDelivered by a process. Thus, only messages that have been utoMulticast are utoDelivered. Moreover, each process maintains an LC, which is updated to ensure that every message is delivered only once. The sending rule ensures that messages are sent with an increasing timestamp by any process ${\mathrm{P}}_{\mathrm{i}}$, and the receive rule ensures that the LC of the receiving process is updated after receiving a message. This means that no process can send any two messages with equal timestamps. Hence, if there is no membership change, Lines 16 and 19 of Figure 5 guarantee that no message is processed twice by process P_k. In the case of a membership change, Line 3a(ii) of Figure 6 ensures that process P_k does not deliver messages twice. Additionally, Lines 7(i-iv) of Figure 6 ensure that P_k’s variables such as logical and stability clock are set to zero, and the buffer and queues are emptied after a membership change. This is done because processes had already delivered all the messages of the old group discarding message duplicates (Line 3a(ii) of Figure 6) to the application process and no messages in the old group will be delivered in the new group. Thus, after a membership change, the new group is started as a new DCTOP operation. The new group might contain messages with the same timestamp as those in the old group, but these messages are distinct from those in the old group. Since timestamps are primarily used to maintain message order and delivery, they do not hold significant meaning for the application process itself. This strict condition ensures that messages already delivered during the membership change procedure are not delivered again in the future.

Lemma 3 (UNIFORM AGREEMENT).

If any process ${\mathrm{P}}_{\mathrm{j}}$ utoDelivers any message m in the current G, then every correct process ${\mathrm{P}}_{\mathrm{k}}$ in the current G eventually utoDelivers m.

Proof. 

Let m_i be a message sent by process ${\mathrm{P}}_{\mathrm{i}}$ and let ${\mathrm{P}}_{\mathrm{j}}$ be a process that delivered m_i in the current G.

Case 1:

${\mathrm{P}}_{\mathrm{j}}$ delivered m_i in the presence of a membership change.

${\mathrm{P}}_{\mathrm{j}}$ delivered m_i during a membership change. This means that ${\mathrm{P}}_{\mathrm{j}}$ had m_i in its mBuffer_i, TO_Queue_i, GCQ_i before executing line 6a(ii) of Figure 6. Since all correct processes exchange their mBuffer_i, TO_Queue_i, GCQ_i during the membership change procedure, we are sure that all correct processes that did not deliver m_i before the membership change will have it in their mBuffer_i, TO_Queue_i or GCQ_i before executing line 1 to 9 of Figure 6. Consequently, all correct processes in the new G' will deliver m_i.

Case 2:

${\mathrm{P}}_{\mathrm{j}}$ delivered m_i in the absence of a membership change.

The protocol ensures that m_i does a complete cycle around the ring before being delivered by ${\mathrm{P}}_{\mathrm{j}}$: indeed, ${\mathrm{P}}_{\mathrm{j}}$ can only deliver m_i after it knows that m_i is crashproof and stable, which either happens when it is the ACN_i in the ring or when it receives µ(m_i) for message m_i. Remember that processes transfer messages from their mBuffer to DQ when the messages become stable. Consequently, all processes stored m_i in their DQ before ${\mathrm{P}}_{\mathrm{j}}$ delivered it. If a membership change occurs after ${\mathrm{P}}_{\mathrm{j}}$ delivered m_i and before all other correct processes delivered it, the protocol ensures that all Survivor(G) that did not yet deliver m_i will do it (Line 6a(ii) of Figure 6). If there is no membership change after ${\mathrm{P}}_{\mathrm{j}}$ delivered m_i and before all other processes delivered it, the protocol ensures that µ(m_i) for m_i will be forwarded around the ring, which will cause all processes to set m_i to crashproof and stable. Remember, when any process receives µ(m_i) and Hops_ij < f, it knows that m_i is crash proof and stable but if Hops_ij ≥ f, then m_i is only stable because m_i is already known to be crashproof since at least f+1 processes had already received m_i. Each correct process will thus be able to deliver m_i as soon as m_i is at the head of DQ (Line 3 of Figure 4). The protocol ensures that m_i will become first eventually. The reasons are the following: (1) the number of messages that are before m_i in DQ of every process P_kis strictly decreasing, and (2) all messages that are before m_i in DQ of a correct process P_kwill become crashproof and stable eventually. The first reason is a consequence of the fact that once a process P_ksets message m_i to crashproof and stable, it can no longer receive any message m such that m≺ m_i. Indeed, a process P_c can only produce a message m_c ≺ m_i before receiving m_i. As each process forwards messages in the order in which it received them, we are sure that the process that will produce an µ(m_i) for m_i will have first received m_c. Consequently, every process setting m_i to crashproof and stable will have first received m_c. The second reason is a consequence of the fact that for every message m that is utoMulticast in the system, the protocol ensures that m and µ(m) will be forwarded around the ring (Lines 25 and 31 of Figure 5), implying that all correct processes will mark the message as crashproof and stable. Consequently, all correct processes will eventually deliver m_i.

Lemma 4 (TOTAL ORDER).

For any two messages m and $m\text{'}$ if any process $P_i$ utoDelivers m without having delivered $m\text{'}$, then no process $P_j$ utoDelivers $m\text{'}$ before m.

Suppose that ${\mathrm{P}}_{\mathrm{i}}$ deduces stability of TS, $\mathrm{T}\mathrm{S}\ge 0$, for the first time by (i) above at, say, time t, that is, by receiving m, $\mathrm{m}\_\mathrm{t}\mathrm{s}=\mathrm{T}\mathrm{S}$ and $\mathrm{m}\_\mathrm{o}\mathrm{r}\mathrm{i}\mathrm{g}\mathrm{i}\mathrm{n}=$ ${\mathrm{C}\mathrm{N}}_{\mathrm{i}}$, at time t. ${\mathrm{P}}_{\mathrm{i}}$ cannot have any $m’$, $\mathrm{m}\mathrm{\text{'}}\_\mathrm{t}\mathrm{s}\le \mathrm{T}\mathrm{S}$ in its${\mathrm{I}\mathrm{n}\mathrm{c}\mathrm{o}\mathrm{m}\mathrm{i}\mathrm{n}\mathrm{g}\mathrm{Q}\mathrm{u}\mathrm{e}\mathrm{u}\mathrm{e}}_{\mathrm{i}}$ at time t nor will ever have $m’$ at any time after t.

Proof (By Contradiction)

Assume, contrary to Lemma, that ${\mathrm{P}}_{\mathrm{i}}$ is to receive $m’$, $\mathrm{m}\mathrm{\text{'}}\_\mathrm{t}\mathrm{s}\le \mathrm{T}\mathrm{S}$, after t as shown Figure 7a.

• Case 1:

Let $\mathrm{m}\_\mathrm{o}\mathrm{r}\mathrm{i}\mathrm{g}\mathrm{i}\mathrm{n}=\mathrm{m}\mathrm{’}\_\mathrm{o}\mathrm{r}\mathrm{i}\mathrm{g}\mathrm{i}\mathrm{n}={\mathrm{P}}_{\mathrm{j}}$. So, imagine that ${\mathrm{P}}_{\mathrm{j}}$ is the same as ${\mathrm{C}\mathrm{N}}_{\mathrm{i}}$, ${\mathrm{P}}_{\mathrm{j}}\equiv {\mathrm{C}\mathrm{N}}_{\mathrm{i}}$, as shown in Figure 7b. Given that $\mathrm{m}\mathrm{\text{'}}\_\mathrm{t}\mathrm{s}\le \mathrm{T}\mathrm{S}=\mathrm{m}\_\mathrm{t}\mathrm{s}$, $\mathrm{m}\mathrm{\text{'}}\_\mathrm{t}\mathrm{s}<\mathrm{m}\_\mathrm{t}\mathrm{s}$ must be true when $\mathrm{m}\_\mathrm{o}\mathrm{r}\mathrm{i}\mathrm{g}\mathrm{i}\mathrm{n}=\mathrm{m}\mathrm{’}\_\mathrm{o}\mathrm{r}\mathrm{i}\mathrm{g}\mathrm{i}\mathrm{n}$. So, ${\mathrm{P}}_{\mathrm{j}}$ must have sent $m’$ first and then $m$.

Note that

(a)

The link between any pair of consecutive processes in the ring maintains FIFO, and

(b)

Processes ${\mathrm{P}}_{\mathrm{j}+1}, {\mathrm{P}}_{\mathrm{j}+2,}.... {\mathrm{P}}_{\mathrm{i}-1}$ forward messages in the order they received those messages.

Therefore, it is not possible for ${\mathrm{P}}_{\mathrm{i}}$ to receive $m’$ after it received m, that is, after t. So case 1 cannot exist.

• Case 2:

Imagine that $\mathrm{m}\_\mathrm{o}\mathrm{r}\mathrm{i}\mathrm{g}\mathrm{i}\mathrm{n}$ is from${\mathrm{C}\mathrm{N}}_{\mathrm{i}}$and $\mathrm{m}\mathrm{’}\_\mathrm{o}\mathrm{r}\mathrm{i}\mathrm{g}\mathrm{i}\mathrm{n}$is from ${\mathrm{P}}_{\mathrm{j}}$, $\mathrm{m}\_\mathrm{o}\mathrm{r}\mathrm{i}\mathrm{g}\mathrm{i}\mathrm{n}={\mathrm{C}\mathrm{N}}_{\mathrm{i}}\ne \mathrm{m}\mathrm{’}\_\mathrm{o}\mathrm{r}\mathrm{i}\mathrm{g}\mathrm{i}\mathrm{n}={\mathrm{P}}_{\mathrm{j}}$, as shown in Figure 6b. Since ${\mathrm{P}}_{\mathrm{i}}$ is the last process to receive m in the system, ${\mathrm{P}}_{\mathrm{j}}$ must have received m before t; since $\mathrm{m}\mathrm{\text{'}}\_\mathrm{t}\mathrm{s}\le \mathrm{m}\_\mathrm{t}\mathrm{s}$, ${\mathrm{P}}_{\mathrm{j}}$ could not have sent $m\text{'}$ after receiving m. So, the only possibility for $\mathrm{m}\mathrm{\text{'}}\_\mathrm{t}\mathrm{s}\le \mathrm{m}\_\mathrm{t}\mathrm{s}$ to hold is: ${\mathrm{P}}_{\mathrm{j}}$ must form and send $m\text{'}$ before it is received and forwarded m.

For the cases of (a) and (b) in case 1, ${\mathrm{P}}_{\mathrm{i}}$ must receive $m\text{'}$ before m. Therefore, the assumption made contrary to Lemma 1 cannot be true. Thus, Lemma 1 is proved.

4. Fairness Control Environment

In this section, the DCTOP fairness mechanism was discussed: for a given round k, any process ${\mathrm{P}}_{\mathrm{i}}$ either sends its own message to the ${\mathrm{C}\mathrm{N}}_i$ or forwards messages from its $A{\mathrm{C}\mathrm{N}}_i$ to the ${\mathrm{C}\mathrm{N}}_i$. A round is defined as follows: for any round k, every process ${\mathrm{P}}_{\mathrm{i}}$ sends at most one message, m, to its ${\mathrm{C}\mathrm{N}}_i$ and also receives at most one message, m, from its $\mathrm{A}{\mathrm{C}\mathrm{N}}_i$ in the same round. Every process ${\mathrm{P}}_{\mathrm{i}}$ has an ${\mathrm{I}\mathrm{n}\mathrm{c}\mathrm{o}\mathrm{m}\mathrm{i}\mathrm{n}\mathrm{g}\mathrm{Q}\mathrm{u}\mathrm{e}\mathrm{u}\mathrm{e}}_{\mathrm{i}}$ which contains the list of all messages ${\mathrm{P}}_{\mathrm{i}}$ received from the ${\mathrm{A}\mathrm{C}\mathrm{N}}_{\mathrm{i}}$ which was sent by other processes, and a ${\mathrm{S}\mathrm{e}\mathrm{n}\mathrm{d}\mathrm{i}\mathrm{n}\mathrm{g}\mathrm{Q}\mathrm{u}\mathrm{e}\mathrm{u}\mathrm{e}}_{\mathrm{i}}$. The ${\mathrm{S}\mathrm{e}\mathrm{n}\mathrm{d}\mathrm{i}\mathrm{n}\mathrm{g}\mathrm{Q}\mathrm{u}\mathrm{e}\mathrm{u}\mathrm{e}}_{\mathrm{i}}$ consist of the messages generated by the process ${\mathrm{P}}_{\mathrm{i}}$ waiting to be transmitted to other processes. When the ${\mathrm{S}\mathrm{e}\mathrm{n}\mathrm{d}\mathrm{i}\mathrm{n}\mathrm{g}\mathrm{Q}\mathrm{u}\mathrm{e}\mathrm{u}\mathrm{e}}_{\mathrm{i}}$ is empty, the process ${\mathrm{P}}_{\mathrm{i}}$ forwards every message in its ${\mathrm{I}\mathrm{n}\mathrm{c}\mathrm{o}\mathrm{m}\mathrm{i}\mathrm{n}\mathrm{g}\mathrm{Q}\mathrm{u}\mathrm{e}\mathrm{u}\mathrm{e}}_{\mathrm{i}}$ but whenever the ${\mathrm{S}\mathrm{e}\mathrm{n}\mathrm{d}\mathrm{i}\mathrm{n}\mathrm{g}\mathrm{Q}\mathrm{u}\mathrm{e}\mathrm{u}\mathrm{e}}_{\mathrm{i}}$ is not empty, a rule is required to coordinate the sending and forwarding of messages to achieve fairness.

Suppose that process ${\mathrm{P}}_i$ has one or more message(s) to send stored in its ${SendingQueue}_i$, it follows these rules before sending each message in its ${SendingQueue}_i$ to the ${\mathrm{C}\mathrm{N}}_{\mathrm{i}}$: process ${\mathrm{P}}_i$ sends exactly one message in ${SendingQueue}_i$ to the ${\mathrm{C}\mathrm{N}}_{\mathrm{i}}$ if

(1)

the ${IncomingQueue}_i$ is empty, or

(2)

the ${IncomingQueue}_i$ is not empty and either

(2.1)

${\mathrm{P}}_i$ had forwarded exactly one message originating from every other process or

(2.2)

the message at the head of the ${IncomingQueue}_i$ originates from a process whose message the process ${\mathrm{P}}_i$ had already forwarded.

To implement these rules and verify rules 2.1 and 2.2, a data structure called forwardlist was introduced. The ${forwardlist}_i$ at any time consists of the list of the origins of the messages that process ${\mathrm{P}}_{\mathrm{i}}$ forwarded ever since it last sent its own message. Obviously by definition, as soon as the process ${\mathrm{P}}_{\mathrm{i}}$ sends a message, the ${forwardlist}_i$ is empty. Therefore, if ${\mathrm{P}}_{\mathrm{i}}$ forwards a message that originates from the process ${\mathrm{P}}_{\mathrm{i}-1}$,$i>0$, which was initially in its ${IncomingQueue}_i$, then process ${\mathrm{P}}_{\mathrm{i}}$ will contain the process ${\mathrm{P}}_{\mathrm{i}-1}$ in its forward list, and whenever it sends a message the process ${\mathrm{P}}_{\mathrm{i}-1}$ will be deleted from the ${forwardlist}_i$.

5. Experiments and Performance Comparison

This section presents a performance comparison of the DCTOP protocol against the LCR [19] protocol and Raft [9,18] a widely implemented, leader-based ordering protocol by evaluating latency and throughput across varying numbers of messages transmitted within the cluster environment. Java (OpenJDK-17, Java version 17.02) framework was used to run a discrete event simulation for the protocols with at most 9 processes,$N=\mathrm{4,5},7,and9$.

Every simulation method made use of a common PC with a 3.00GHz 11th Gen Intel(R) Core(TM) i7-1185G7 Processor and 16GB of RAM. A request is received from the client by each process, which then sends the request as a message to its neighbour on the ring-based network. When a neighbour receives a message, it passes it on to another neighbour until all processes have done so. When the ACN of the message origin receives the message then it knows that it is stable and makes an attempt to deliver it in total order, a process known as TO delivery. This process then notifies all other processes which, up until this point, had no idea of the message's status by using an acknowledgement message known as µ-message to inform them of the message's stability. Other processes that get this acknowledgement are aware that the message is stable and make an effort to deliver it in total sequence. For a Raft cluster, when a client sends a request to the leader, the leader adds the command to its local log, then sends a message to follower processes to replicate the entry. Once a majority (including the leader) confirms replication, the entry is committed. The leader then applies the command to its state machine for execution, notifies followers to do the same, and responds to the client with the output of execution.

The time between successive message transmissions is modelled as an exponential distribution with a mean of 30 milliseconds, reflecting the memoryless property of this distribution, which is well-suited for representing independent transmission events. The delay between the end of one message transmission and the start of the next is also assumed to follow an exponential distribution, with a mean of 3 milliseconds, to realistically capture the stochastic nature of network delays. For the simulation, process replicas are assumed to have 100% uptime, as crash failure scenarios were not considered. Additionally, no message loss is assumed, meaning every message sent between processes is successfully delivered without failure.

The simulations were conducted with varying numbers of process replicas, such as 4, 5, 7, and 9 processes. The arrival rate of messages follows a Poisson distribution with an average of 40 messages per second, modelling the randomness and variability commonly observed in real-world systems. The simulation duration ranges from 40,000 to 1,000,000 seconds. This extended period is chosen to ensure the system reaches a steady state and to collect sufficient data for a 95% confidence interval analysis. The long duration also guarantees that each process sends and delivers between one million (1,000k) and twenty-five million (25,000k) messages.

Latency. These order protocols calculate latency as the time difference between a process's initial transmission of a message m and the point at which all m destinations deliver m in total order, denoted as $TOdeliver\left(m\right),$ to the applications process. For example, let $t_0$ and $t_1$ be the time when $P_0$ sends a message to its CN₀ and the time when the ACN(ACN₀) delivers that message in total order respectively. Then ${\mathrm{t}}_{\mathrm{l}}-{\mathrm{t}}_0$ defines the maximum latency delivery for that message. The average of 1000k to 25000k messages of such maximum latencies was computed, and the experiment was repeated 10 times for a confidence interval of 95%. The average maximum latency was plotted against the number of messages sent by each process.

Throughput. The throughput is calculated as the average number of total order messages delivered (aNoMD) by any process during the simulation time calculated, like latencies, with a 95% confidence interval. Similarly, to the latency, we also determined a 95% confidence interval for the average maximum throughput. Additionally, we presented the latency enhancements offered by the proposed protocol in comparison to LCR, as well as the throughput similarities.

All experiments were done independently to prevent any inadvertent consequences of running multiple experiments simultaneously. Nevertheless, the execution ensured each of the experiments was staggered to cover approximately the same amount of simulation time. This was done to sustain a uniform load on the ring-based and leader-based network across all of the experiments.

5.1. Results and Discussion

The latency analysis of DCTOP, LCR, and RAFT (see Figure 8i-iv ) across varying group sizes (N = 4, 5, 7, and 9) and increasing message volumes reveals that DCTOP consistently demonstrates the lowest latency. This performance advantage is likely attributed to its use of Lamport logical clocks for efficient sequencing of concurrent messages, the assignment of a unique last process for each message originator, and a relaxed crash-failure assumption that permits faster message delivery. LCR shows moderately increasing latency with larger group sizes and message volumes, primarily due to its reliance on vector clocks whose size grows with the number of processes and the use of a globally fixed last process for message ordering, both of which contribute to increased message size and coordination cost. RAFT, a leader-based protocol, exhibits the highest latency overall, especially under higher load conditions, underscoring the limitations of centralized coordination. However, under lower traffic conditions (e.g., 1 million messages per process), RAFT performs competitively and, in configurations with N = 7 and N = 9, even outperforms DCTOP and LCR. This suggests that RAFT may remain suitable in low-load or moderately scaled environments.

On the other hand, in terms of throughput (Figure 9i-iv), DCTOP and LCR outperform RAFT across all group sizes and message volumes. Both are leaderless ring-based order protocols that benefit from decentralized execution, enabling all processes to independently receive and process client requests. This results in cumulative throughput, which scales linearly with group size. RAFT, by contrast, centralizes request handling at the leader, thereby limiting throughput to the leader’s processing capacity. The results demonstrate that DCTOP consistently achieves the highest throughput, with LCR following closely despite its minor overhead from vector timestamps and centralized ordering logic. RAFT consistently exhibits the lowest throughput, and its performance plateaus with increasing load, reinforcing the inherent scalability limitations of leader-based protocols in high-throughput environments.

Notably, all three protocols - RAFT, LCR, and DCTOP were implemented from a unified code base, differing only in protocol-specific logic. The experiments were conducted under identical evaluation setups and hardware configurations, ensuring a fair and unbiased comparison.

6. Conclusions and Future Work

In this work, DCTOP, a novel ring-based leaderless total order protocol that extends the traditional LCR approach was introduced through three key innovations: the integration of Lamport logical clocks for concurrent message sequencing, a new mechanism for dynamically identifying the last process per each message sender, and a relaxation of the traditional crash failure assumption. These modifications collectively contributed to significant latency reductions under varying system configurations. To promote fairness among process replicas, our simulation model incorporated control primitives to eliminate message-sending bias. A comparative performance evaluation of DCTOP and LCR using discrete-event simulation across group sizes of N = 4, 5, 7, and 9, and under concurrent message loads was conducted. The results yielded three major insights. First, DCTOP achieved over 43% latency improvement compared to LCR across all configurations, demonstrating the efficacy of Lamport logical clocks in this context. Second, the proposed dynamic last process mechanism proved to be an effective alternative to the globally fixed last process used in LCR, enabling faster message stabilization. Third, by relaxing the LCR crash tolerance condition from N = f + 1 to N = 2f + 1, DCTOP is able to deliver messages more quickly while still tolerating failures, further contributing to latency reduction. While our primary focus was on evaluating DCTOP relative to LCR, we included RAFT, a widely adopted leader-based protocol as a benchmark to contextualize our results. RAFT demonstrated competitive performance under light message loads, but its throughput and latency plateaued with scale due to its centralized coordination model. The goal was not to critique RAFT, but to highlight architectural differences and situate DCTOP within the broader spectrum of total order protocols.

Given that the primary goal of this study was to investigate the initial performance characteristics of DCTOP, we deliberately limited our evaluation to small group sizes (N ≤ 9) to enable controlled experimentation and isolate protocol-level behaviour. While this approach provides useful insight, it also introduces some limitations. The current implementation does not model process or communication failures, which are common in practical distributed systems. Furthermore, larger-scale deployments may exhibit additional performance dynamics not captured in this setting. As part of our ongoing work, we are developing a cloud-based, fault-tolerant implementation of DCTOP to validate these findings in more realistic environments, including under failure conditions and dynamic workloads.