A Qualitative and Comparative Performance Assessment of Logically Centralized SDN Controllers by Mininet Emulator

1. Introduction

A corporate network is made up of two or more computers that share data, resources, and storage in order to exchange knowledge and save money. Typically, a dedicated de- vice is used to conduct the majority of traditional network activities. A dedicated device for application delivery is referred to as one or more switches, routers, and controllers. The majority of these device features are implemented in specific hardware. ASICs, or application-specific integrated circuits, are frequently used for this. The majority of net- works in use today are still conventional networks. While the amount of everyday data transfer is increasing and bringing more devices, these networks still have some limits. The additional device has to rewire while joining a network. Many of these pieces of equipment are managed by individuals, and as networks develop, there are more parts to keep up with. The management of these large network infrastructures is more challenging. As a result, a fresh approach to network administration is required in order to get beyond these established network limitations [1].

The technology known as "software-defined networking" (SDN) is expected to trans- form current network companies. Numerous business advantages are offered, including improved design reliability, flexibility, and stability. SDN systems eliminate disruptions in traditional networks and simplify network architecture by separating network control from forwarding services. SDN enables programmatic network setup to enhance network administration and satisfy our requirements. The separation of the control plane and data plane, flow-based forwarding choices, and the definition of a flow as a series of packets from one are among the primary characteristics of the SDN architecture [2].

SDN does have certain drawbacks, though. The SDN protocol and controller require significant network infrastructure upgrades. Consequently, the entire network will have been reconstructed. The price of reconfiguration rises as a result. It is crucial to train employees. It is important to make new management tools available and teach everyone how to utilize them. The architecture of centralized SDN controllers prioritizes safety. SDN controllers are alone in charge of all transmission operations since they are frequently seen as single points of failure. If the switch-to-controller or SDN controller connections fail, the entire network will crash. A single control issue, such as a single point of failure, restricted control resources, etc., can be resolved if there are numerous controllers [3].

The fundamental principle of SDN enhances the network in several aspects, including more effective resource use, improved network administration, lower costs, innovation with fresh development, and many more. An effective controller is crucial to managing all these changes and enhancing resource usage for increased network performance. In addition, the controller is in charge of tracking and analyzing data flow in real time. The massive rise of distributed processing-based real-time applications has created a large need for high-performance controllers in networking businesses, data centers, academia, and research. Thus, it is essential to look at the controller’s performance in order to provide effective traffic routing, which will increase resource usage for improved performance of the network [4].

The efficiency of SDN controllers has been the subject of several research. There is still a chasm between them, though. There is no in-depth classification scheme for SDN control plane architecture. Perhaps the only categorization criterion that can be used is the deployment architecture. Based on this, Nunes et al. [5] has proposed two significant kinds of control plane systems with multiple controllers. Using Control Plane Architecture as our foundation, we offer an in-depth classification of software-defined networking in this research which is shown in Figure 1. The issues with distributed but logically centralized SDN-based controllers haven’t been thoroughly researched in the past. It is quite difficult to predict which controller will operate best in a certain sort of network from the standpoint of actual implementation. Therefore, it is crucial to compare these controllers both qualitatively and quantitatively. As far as we are aware, no such work has been done that compares and assesses the attributes of the distributed but logically centralized SDN- based controllers. A Mininet may simulate several types of network components, including connections, layer-2 switches, layer-3 routers, and hosts. It is based on a single Linux kernel and makes use of virtualization to simulate a whole network on a single machine. As far as we are aware, no study has been done specifically to compare performance in a Mininet setting. The investigation of OpenFlow-based controllers’ performance has been done before. Works [18,19,20,21,22,23,24,25,26,27,28,29,30,31] give some quantitative comparisons, although most of them either have a single application or a simple environment where several trials may be conducted. We have selected a custom scenario for this research that utilizes Mininet capabilities. Using the efficient network simulator Mininet for custom network environment, this study provides a thorough analysis of five distributed but logically centralized controllers in terms of bandwidth, round-trip time, delay, jitter, packet loss, and throughput. We have contributed the following to this paper:

• We present a more in-depth classification of SDN control plane architecture, shown in Figure 1;
• We give a qualitative evaluation of their attributes and classify and categorize 13 distributed but logically centralized SDN controllers according to their capabilities;
• A detailed survey of SDN controller performance based on Mininet is made;
• Using six performance metrics—bandwidth, round-trip time, delay, jitter, packet loss, and throughput—this work also assesses five distributed but logically centralized controllers against two custom network scenarios (uniform and non-uniform host distribution).

The remainder of the research paper is structured as follows: Section 2 presents the relevant research and how our work is different from other existing works followed by detailed qualitative evaluation of distributed but logically centralized controllers with design choices and selection criteria along with Mininet-based performance assessment study in Section 3. The qualitative and quantitative description and evaluation of selected 5 controllers, topology description with setup of the experiment and the resources required to evaluate the performance of the controllers are briefly discussed in Section 4. Section 5 presents a performance analysis. Section 6 goes into extensive detail about the outcomes of experiments and studies with discussion of logically centralized controller’s advantages and the most likely challenges that can encounter for distributed controllers. Section 7 discusses the conclusion and future work.

2. Related Work

Some relevant works are presented in this area, which is divided into three sections: papers on various control architectures, papers on benchmarking performance assessment, and papers on Mininet performance assessment of SDN controllers.

The SDN control plane is currently created using either single controller designs or distributed/multiple controller architectures [3,6,7,8,9,10]. In their thorough investigation of the scalability problems with SDN-based control planes, Karakus and Durresi [9] provided a classification and taxonomy of the state-of-the-art which is considered the very first taxonomy of control plane. They have simply produced a physical categorization in their work. There were two methods used for categorization: mechanism-related and topological. The topology of various architectures, their relationships, and scalability issues with the design of centralized and distributed controllers were the topics of the topology-related methods. The mechanism-related included many techniques, including parallel-based and routing scheme-based optimization, to improve controllers and their scalability issues. Only machine learning-based optimization was added by Abuarqoub et al., [11] to the mechanism-based approach. If not, the classifications are almost identical. However, both classification lacks the further logical categorization of distributed and centralized SDN control planes. Additionally, a number of distinct controller types are missing from their taxonomy. In addition to the physical categorization, distributed SDN control systems may be divided into conceptually centralized and logically distributed architectures based on how information is shared among controller instances [12] whereas. However, the entire control plane architecture was not provided by this effort, and their taxonomy lacks specific controller types. Additionally, Bannour et al., [10] included the physical classification of control plane architecture along with its various types in their study. However, hybrid orientation, along with logical categorization and overall control plane design, is absent from the subdivision of physically distributed SDN control plane. Isong et al., [13] did not address hybrid orientation, instead explicitly classifying the control plane into physical and logical components. Only in situations when the mechanism-based approach is completely absent did he view the control plane as a topology-related approach. In our work, topology-related and mechanism-related control designs are separated apart. Two categories—physically or logically centralized and physically or logically distributed—are further distinguished amongst these topology-related control structures. Moreover, each control architecture may have a flat, hierarchical, or hybrid orientation. Control plane routing-based optimization and parallelism-based optimization are two categories of mechanism-related approaches. Every category has a unique controller kind.

Several studies have been done in the aim of comparing SDN controllers [2,10,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37]. The authors of article [10] examined 15 distributed SDN controllers while taking into account the following factors: (1) Scalability; (2) Reliability; (3) Consistency; (4) Interoperability; (5) Monitoring; and (6) Security. All of the aforementioned research have taken into account either open source or widely used SDN-based controllers. Distributed controllers have only been examined by them. However, they did not evaluate and compare the characteristics of the distributed but logically centralized SDN-based controllers. In this study, we have provided a qualitative assessment of their characteristics and, based on their capacities, we classify and categorize thirteen (13) distributed but logically centralized SDN controllers. One of the earliest studies to compare SDN controllers was [25], which only looked at controller performance while taking into account a small number of controllers (NOX-MT, Beacon, and Maestro). But over time, new controllers like POX, Ryu, FloodLight, and OpenDayLight have taken the place of these ones. In [26], a thorough analysis of SDN OpenFlow Controllers was carried out. The efficacy of the commonly used SDN controllers, NOX, POX, Beacon, FloodLight, MuL, Maestro, and Ryu, is compared. The hcprobe tool was utilized by the authors. This comparison needs to be expanded to take into account newly designed controllers as well as additional controller functions. A set of requirements—TLS support, virtualization, open source, interfaces, GUI, RESTful API, productivity, documentation, modularity, platform support, age, OpenFlow support, and OpenStack Neutron support—provided by the study carried out in [27] serve as the foundation for the comparison of the controllers. Analytic Hierarchy Process (AHP), a monotonic interpolation/extrapolation mechanism that transfers the values of the attributes to a value on a pre-defined scale, was used to modify the Multi-Criteria Decision Making (MCDM) technique for the comparison. Five controllers (POX, Ryu, Trema, FloodLight, and OpenDayLight) were tested using the modified AHP, and "Ryu" was determined to be the best controller based on their needs. Based on how simple it was to: 1) examine the network (discovery), 2) add a new network function (setup), 3) change an existing function (change), and 4) remove a function (removal), the authors of [28] evaluated the qualitative performance of two open source controllers. However, only two SDN controllers (ONOS and OpenDayLight) and one network function (port-/traffic-mirroring) were included in the study. Proactive and reactive operational styles are discussed separately in [29]. Because the rules in the proactive mode are loaded into the switch at startup rather than every time the switch gets a packet for which there is no matching rule in its flow table, the proactive mode performs better than the reactive one. Even though this comparison highlights a significant aspect of the performance comparison, it is insufficient to determine which featured controller is the best. More research is conducted in [30], which offers more factors to take into account while creating a new controller. There are two different designs that are taken into consideration: shared queue with adaptive batching and static partitioning with static batching. The Beacon system, utilizing static batching, demonstrated the best performance in the Maestro, NOX-MT, and FloodLight tests. However, the optimal latency records are shown by Maestro, which employs adaptive batching architecture. Therefore, the behavior of the necessary controller, which is connected to its application domain, determines the architecture to be used. The portability and performance of the controller are shown to be impacted by the programming language selection in [31]. Because Java is cross-platform compatible and allows multithreading, the authors contend that it is the best option. Python has problems with multithreading at the performance level, whereas C++ has memory management and other constraints. The runtime platform determines which net languages work with (Linux compatibility is not supported). As a result, they demonstrate that out of numerous controllers (NOX, POX, Maestro, FloodLight, Ryu), the Java-based Beacon performs the best. Nonetheless, the fact that these several languages are still in use today indicates that each language retains some traits that the others have not taken up. The authors [32], emphasized the problem of software aging. The primary problem under investigation is memory leak, and in order to maintain the study’s objectivity, two Java-based controllers—FloodLight and Beacon—were compared. The outcomes demonstrated that Beacon performs better and uses less memory than FloodLight. The authors of [33] and [34] examined 12 whereas authors of [35] examined 8 popular open source controllers taking into account a number of factors, including programming language, graphical user interface, documentation, modularity, distributed/centralized, platform support, southbound and northbound APIs, partners, multithreading support, open stack support, and application area. Furthermore, the comparison takes into account a wide range of other SDN-specific qualitative factors in addition to quantitative metrics like throughput and latency. The study [36] examined a number of open-source controllers, including Ryu, POX, OpenDayLight (ODL), and Open Network Operative System (ONOS), by qualitatively assessing both their performance and features. The authors used the Cbench program to benchmark the controllers under various operating situations and quantitatively assess a number of characteristics. The authors of article [37] compared the features and performances of four well-known SDN controllers. The authors demonstrated that Ryu has a good amount of features, making it perfect for small-scale SDN installations, and that OpenDayLight and ONOS are the controllers with the most feature richness.

A hardware-based testbed or simulation/emulation can be used to assess or benchmark a controller's performance. Hardware testbeds are more expensive for the research community even if they offer measurements that are more in line with real values in a production setting. Thus, assessments based on simulation or emulation are typical. Frameworks like OFLOPS [38], OFLOPS-Turbo [39], Cbench [25], PktBlaster [40], and OFCBenchmark [41] have been suggested to encourage the study of various performance characteristics of OpenFlow devices. In order to test new applications and construct OpenFlow-based networks on a single system, a number of simulation and emulation tools have also been developed. Tools that may be used to deploy a virtual network and estimate performance metrics for different network topologies and sizes is the Mininet [17], NS3 [42] network simulator. Papers [2,14] provide a comparative assessment of previous benchmarking research as well as an in-depth look at the capabilities of benchmarking tools. A summary of SDN-based research utilizing Mininet is provided in Paper [17], yet there is no survey of SDN controller performance evaluation using Mininet. We have conducted a thorough review of the literature on performance evaluation in this study, shown in Table 2. Additionally, this work evaluates five distributed but logically centralized controllers using six performance metrics: bandwidth, round-trip time, delay, jitter, packet loss, and throughput. We created two custom topologies with hosts dispersed in a uniform and non-uniform way using the Mininet simulation environment.

3. Controller Classification and Design Choices

We have conducted a thorough search of proposals in both the academic and commercial domains in order to compare various SDN controllers. Here, we first outline potential controller categorization criteria, then move on to a comparison analysis.

3.1. Selection Criteria

Controllers behave in essentially the same way. After analyzing every controller, we come to the conclusion that the majority of them lack a categorization foundation for their roles, duties, and way of functioning. The deployment architecture may be the only categorization criterion that can be applied [14]. Since the primary goal of SDN was to consolidate the control plane, the majority of controllers only used one controller. Scalability issues and a single point of failure were nevertheless brought about by this. Multiple controllers can operate in a flat or hierarchical structure inside a domain thanks to the distributed design.

3.2. Qualitative Comparison

A thorough overview of the many characteristics of the distributed but logically centralized controllers is provided in Table 1. We will not discuss about each controller separately. Instead, we showcase the characteristics and design options of controllers. The characteristics and design options we have followed is taken by [14].

Programming Language: A variety of programming languages, including C, C++, Java, Java Script, Python, Ruby, Haskell, Go, and Erlang, have been used to write controllers. There are instances where a single language is used to create the complete controller. While the core and modules of many other controllers employ various languages in order to provide effective memory allocation, be executable on different platforms, or—most importantly—achieve superior performance under specific circumstances.

Programmable Interface (API): Generally speaking, Northbound API (NBI) enables the controller to support applications like as intrusion detection, load balancing, network virtualization, flow forwarding, and topology monitoring that rely on network events produced by data plane devices. Conversely, low-level APIs such as Southbound API (SBI) are in charge of facilitating communication between a controller and switches or routers that have SDN enabled. Furthermore, in a dispersed or hierarchical environment, several controllers from various domains build peering relationships with one another through the usage of east-west API (EWBI). Not every controller offers every API, and only a small number of them have been tailored for a particular purpose.

Platform and Interface: These characteristics explain how a controller is implemented to work with a particular operating system. Linux distributions serve as the foundation for the majority of controllers. Furthermore, certain controllers give administrators access to graphical or web-based interfaces for configuring and viewing statistics data.

Threading and Modularity: Lightweight SDN installations are better suited for a single-threaded controller. Multi-threaded controllers, on the other hand, are appropriate for commercial applications including optical networks, SDN-WAN, and 5G. However, the versatility of a controller makes it possible to integrate many features and applications. In a dispersed setting, a controller with high modularity can execute tasks more quickly.

License, Availability, and Documentation: Open Source licensing applies to the majority of the controllers covered in this article. Some, on the other hand, are only accessible by special request or for research purposes due to a proprietary license. Many of these controllers do not receive regular updates since it is difficult for the developers to maintain them on a regular basis. However, the source code is accessible online, enabling anybody to modify it further in accordance with the specifications. We discovered that most of them lack adequate documentation when we accessed them online. Conversely, those that undergo frequent updates come with community-based assistance as well as comprehensive and up-to-date documentation for every version that is accessible.

3.3. Mininet and Mininet Based Study

A team of experts from Stanford University created Mininet as a tool for research and instruction. With the aid of a controller, Mininet may do testing and mimic SDN networks. We utilize Mininet as a simulation tool, which is a network emulator. It enables the building of networks with different topologies made up of several virtual hosts, connections, and switches. With the help of this program, we can quickly and simply access any network component via the Mininet CLI, create the network to your exact specifications, share it with others, and ultimately develop it with actual hardware. Before implementing our network in real life, we may test and refine it in a virtual environment by using Mininet. A switch and hosts linked to the switch and to each other make up the default Mininet topology. The switch is coupled to an OpenFlow controller. Mininet hosts have the ability to operate on several Linux CLIs. For example, they may simply retrieve the bandwidth between the user and the server by using the iperf command. Although Mininet only offers a few topologies—tree, linear, and single topologies by default—it is possible to design any desired topology by creating a Python script [15,17,45,53,66].

Numerous studies have been conducted utilizing Mininet to assess SDN controller performance. However, a study of previous work on Mininet-based performance evaluation is lacking. Table 2 displays an extensive overview of the current Mininet-based efforts that will be covered in this part.

Table 2. Existing Performance Analysis Study Using Mininet.

Table 2. Existing Performance Analysis Study Using Mininet.

A number of papers demonstrated how to create SDN architecture for network traffic analysis using the open-source Ryu SDN controller. The suggested study assessed how well SDN architecture performed for performance metrics including bandwidth, throughput, jitter, packet loss roundtrip time, switchover time, etc. based on various network topologies. Because Ryu is written in Python, the studies revealed that it is an excellent choice for research and small commercial applications [4,44,65,73,82]. Conversely, the OpenFlow interface is implemented in different network simulation scenario with open source controller POX for network traffic analysis in papers [43,48,80,89]. Analysis and evaluation were done on things like packet loss, CPU load, bandwidth utilization etc. According to those studies, efficient use of bandwidth occurs when it begins at 100 Mbps and increases gradually to 500 Mbps, particularly when there are 20 to 30 switches in use. The major reason for the given POX controller's low reliability was dropped packets. A number of researchers have explored the creation and execution of the desired SDN environment scenario utilizing FloodLight [45,58,87], OpenDayLight [90], ONOS [46,67], NOX [68] and Beacon controllers [75]. Using Mininet Simulator, analysis and assessment were performed on several aspects such as throughput, latency, packet loss, jitter, bandwidth usage, etc. We used Mininet to investigate a number of SDN controllers in our preliminary research [68,69,77,88]. We took into consideration a custom topology network situation. Response time and throughput were the performance parameters we looked at. Researchers have made comparisons between the POX and Ryu controllers in several research [54,64,85,91]. Because of its greater traffic management capabilities, POX performs better in layer 1 switching circumstances. Regarding layer 2 switching, Ryu yielded significantly better performance results. The authors of papers [52,70] contrasted the Ryu and POX controllers with Pyretic, another Python-based controller. According to these research, Pyretic controller works better in tree topology while Ryu performs better in star topology. However, in comparison to other Java-based controllers like FloodLight or ODL, the performance of the Java-based controllers was better than that of the POX or Ryu controllers. Not every topology has the same circumstances. Python-based controllers also function better when there are fewer nodes or operations, whereas Java-based controllers perform worse when there are more nodes or operations. Due to their built-in Java files, Java-based controllers have the drawback of requiring a significant amount of memory space to operate [17,19,49,57,63,72,78,79,81,84,92]. In papers [53,56,61], the authors examined the performance study of an OpenFlow network with single, linear, and tree network topologies; however, the authors employed a Mininet reference controller as a point of comparison rather than the POX or Ryu controller. In order to undertake the performance study, all network topologies were compared based on the following metrics: maximum throughput achieved, round-trip propagation time between end nodes, packet transmission rate, and capacity usage. All OpenFlow-enabled topologies were designed with Mininet, a prototype network emulator. The effectiveness of proactive and reactive paradigms in many well-known controllers was examined in [29]. Both an emulator (Cbench and Mininet) and a real environment were used to assess performance. The reactive and proactive modes of operation of several SDN controllers (NOX, POX, Trema, and FloodLight) were compared by Fernandez et al. The results for every controller that was analyzed indicated that proactive mode is when the controller performs at its best. Using a Mininet emulator, Stancu et al. evaluated the four SDN controllers—POX, Ryu, ONOS, and OpenDayLight [47]. The controllers were told to function as a basic L2 learning switch and hub. A tree topology was employed for comparison, and two tests were run in each phase: an iperf command and a ping command between the two end hosts. In a number of studies, researchers have compared the FloodLight and ODL controllers to other controllers [15,50,51,59,60,71,74,86]. According to the research, FloodLight functions better in linear, tree, and torus topologies, but ODL performs better in a single topology. Furthermore, ODL outperforms FloodLight in terms of latency for tree topologies in networks with medium load as well as low load networks. In networks with high traffic volumes, FloodLight can outperform ODL in terms of packet loss for tree topologies and latency for linear topologies. Researchers have contrasted the ONOS controller with other controllers in a variety of studies [55,62,66,76,83]. The research indicates that ONOS performs well in terms of GUI, clusters, link-up, switch-up, and throughput. However, latency values increased and throughput values decreased as the number of switches and hosts increased. Additionally, the ONOS controller performs best in the flow test and lowest in the network topology discovery time.

In this study, we have provided a Mininet-based SDN controller performance evaluation research, a complete SDN emulation tool. This paper evaluates two custom network scenarios where the hosts are dispersed in a uniform and non-uniform way using the Mininet simulation environment and five distributed but conceptually centralized controllers using six performance metrics: bandwidth, round-trip time, delay, jitter, packet loss, and throughput.

4. Topology Description and Quantitative Evaluation of Controllers

The performance of five distinct distributed yet logically centralized controllers is covered in this section. To the best of our knowledge, this problem has not been addressed in any prior research. The controllers Ryu, ODL, ONOS, POX, and HyperFlow are examined. Out of the thirteen controllers that were previously mentioned, these were chosen because (1) the source code for the controller is available, (2) they are compatible with the latest versions of Linux, (3) they can be interfaced with Mininet tools, and (4) the community finds them interesting.

4.1. Selected Controllers and their Qualitative Evaluation

We will give a brief description of the chosen controllers in this part, and Table 3 will show the qualitative comparison of the controllers.

HyperFlow: The HyperFlow application was developed to provide a logically centralized multi-controller architecture at the top of the NOX controller. The three levels of the HyperFlow network are forwarding, control, and application. The control layer has several NOX controllers. A switch links the forwarding layer to the nearby control device. In the case of a problem (failure), a switch could be allocated to another controller. To transport data across the controller, HyperFlow employs a publish/subscribe messaging architecture [91].

Table 2. Qualitative Analysis of the Selected Controllers.

Table 2. Qualitative Analysis of the Selected Controllers.

ONOS: The community behind the ONOS (Open Network Operating System) project is open source. The goal of this project is to create an SDN operating system [46]. The Java packages for the ONOS project are loaded into the Karaf OSGi container.

Ryu: An open SDN controller called Ryu Controller was created to improve network agility. It is a fully Python-written, component-based software defined networking framework. In Japanese, the term "Ryu" signifies "flow". NTTT, or Nippon Telegraph and Telephone Corporation, provides assistance for the Ryu Controller. Ryu controller supports a number of protocols, including NETCONF, OF-config, OpenFlow, and others [4,65].

POX: The younger sibling of NOX is a networking software platform called POX (Pythonic Network Operating System). Python is the programming language used in the development of POX [12]. When creating networking software, it can be useful. POX is compatible with several operating systems, including Linux, Mac OS, and Windows. It is compatible with Python 2.7 and lower versions. POX connects with OpenFlow 1.0 switches and has specific support for Open vSwitch/Nicira extensions [43,57,64,71,89,92].

OpenDayLight: A collaborative open source project developed in Java, The OpenDayLight (ODL) Project is hosted by The Linux Foundation [11]. A bidirectional REST and OSGi framework may be programmed using OpenDayLight, and it also supports many non-OpenFlow southbound protocols [50,71,76,79]. There is a wiki specifically for developers, along with many email lists and a source code repository for controller releases, for those who are interested.

4.2. Topology Description

Network topology changes are the most crucial testing characteristic in this experimental research that categorizes all controller tests. The effectiveness of network is significantly influenced by its topology. An efficient network layout may greatly boost performance by lowering energy consumption and boosting data transmission speeds. Any organization's actual network is not restricted to simple topologies like linear or tree. The real network topologies of these firms are complex and resemble hierarchical or graph connections more. The depths of the switches and nodes, as well as the fan out, differ. With custom topology, users may create whatever topology they choose. Here, it is clear that each switch's fan out varies, which makes it impossible to describe it during topology creation for topologies like single, linear, tree, etc. These factors led us to use customized topology for our experiment. The customized architecture is depicted in Figure 2, where four switches and twelve hosts are used for the experiment. Figure 2(a) represents a uniform distribution of hosts to the switch, whereas Figure 2(b) shows a non-uniform distribution of hosts to the switch.

Twelve hosts and four OpenFlow-capable switches are utilized in both topologies of this experiment. Each switch has three hosts connected to it in a uniform host distribution. In contrast to a non-uniform host distribution, switch 1 connects one host, switch 2 connects two hosts, switch 3 connects four hosts, and Switch 4 connects five hosts. In every experiment, controller performance is compared using the same topologies and environment.

4.3. Experiment Setup

The test bed consists of a single PC with an Intel Core i7 CPU operating at 2.90GHz and 8GB of RAM. Windows 10 is the host operating system while Ubuntu 22.10 LTS is the guest operating system on the machine. On a 100Mbit network, congested windows of up to 32Mbytes are employed.

In addition to connecting to the controller and creating network connectivity, Mininet also creates switches, hosts, and linkages. Following a successful connection establishment using Mininet, we used Ubuntu xterm to retrieve client-server interfaces and ran the iperf test to create traffic between hosts in order to assess important factors including bandwidth, throughput, round trip time, delay, jitter, and packet loss.

5. Performance Analysis

This section presents the implementation and usage inside a framework for testing and evaluating experimental attempts to assess distributed (logically centralized) SDN networking experience using five distinct controllers (OpenDayLight, ONOS, POX, Ryu, and HyperFlow).

5.1. Bandwidth

The amount of network bandwidth in Gb/s that has been used the most is shown by the bandwidth in Figure 3. According to our investigation, the bandwidth for OpenDayLight, POX, Ryu, ONOS, and HyperFlow controllers in a uniform host distribution is 62.09, 59.1 and 59.2, 58.4 and 58.8 Gb/s, respectively; the value for a non-uniform host distribution is 61.8, 59, 59.3, 56.8 and 57.8 Gb/s. In both situations, the OpenDayLight controller performs the best and the Ryu controller the worst.

5.2. Round Trip Time

The round trip time is the entire period of time that is it takes for a data packet to travel from where it started to its endpoint and for its confirmation to be received at the origin. In Figure 4, the round trip time between a network and server may be found using the ping command. It is measured in milliseconds. OpenDayLight, POX, Ryu, ONOS, and HyperFlow controller have average round trip times for uniform host distribution of 0.1014, 0.1244, 0.0927, 0.1403 and 0.3298 ms, respectively. For non-uniform host distribution, the round trip times are 0.1028, 0.1388, 0.0945, 0.1228, and 0.3416 ms, respectively. The POX controller has the fastest average round trip time in both scenarios, whereas the HyperFlow controller has the slowest average.

5.3. Delay

The amount of time it requires for a data packet to go from a particular location to another is known as latency or delay. Milliseconds are used to measure it. According to Figure 5, the OpenDayLight, ONOS, Ryu, POX, and HyperFlow controllers' respective delays or latency for uniform host distribution are 0.0702, 0.0790, 0.0668, 0.0764, and 0.0760 ms; for non-uniform host distribution, the delays are 0.0689, 0.0816, 0.0653, 0.0719, and 0.0785ms. Both times, POX controller and OpenDayLight controller had extremely similar values, with POX controller being marginally superior to OpenDayLight controller. For both situations, ONOS has experienced the highest delay.

5.4. Jitter

Delays in the normal interval between data packet sequences would be the simplest cause of jitter. Jitter is the variation in the data packet delay periods that are received. It is easiest to identify jitter by PINGing a faraway device with a series of packets and then determining the average time difference between each return packet sequence. There is just one type of measurement, time-based and based on milliseconds. The jitter for the OpenDayLight, Ryu, POX, HyperFlow and ONOS controllers is shown in Figure 6 and is, respectively, 0.0055, 0.0072, 0.0082, 0.0132, and 0.0099ms for uniform host distribution and 0.0046, 0.0053, 0.0075, 0.0147 and 0.0118ms for non-uniform host distribution. In all scenarios, Ryu controller has the worst jitter whereas OpenDayLight surpasses all other controllers.

5.5. Packet Loss

Packet loss can be determined and expressed as a percentage based on the difference between the received packets and the transmitted packets. Figure 7 illustrates packet loss for Ryu, ONOS, POX, OpenDayLight, and HyperFlow controllers for uniform host distribution. For non-uniform host distribution, the corresponding values are 0.065, 14.5, 20.5, 12.5, and 28.5 percentages where for uniform distribution, the values are 0.05, 12.5, 20.5, 8.5, and 22.5 percentages. Ryu experiences the least packet loss in both circumstances, whereas HyperFlow experiences the most.

5.6. Throughput

Throughput is the maximum quantity of data that may be sent from a source to a destination in a predetermined period of time. Throughput, or payload over time, quantifies the largest burst of data transfer, to put it another way. GB/s are used to measure it. In our experiment, the term "time" refers to a duration chosen by the examiner (100 seconds). Our research shows that the throughput for the OpenDayLight, ONOS, POX, Ryu, and HyperFlow controllers in a uniform host distribution is 7.23, 6.88, 6.89, 6.62, and 6.77 GB/s, respectively, while the values for a non-uniform host distribution are 7.2, 6.87, 6.9, 6.59, and 6.83 GBytes/sec are shown in Figure 8. In both situations, Ryu has the lowest throughput while OpenDayLight has the greatest.

6. Discussion

This part covers the performance study of the different distributed (but logically centralized) controllers discussed in the preceding section. Bandwidth, throughput, round trip time, delay, jitter, and packet loss are the six metrics used for analysis. According to the thorough analysis, the OpenDayLight controller performs better than the Ryu controller in all performance parameters except than delay, packet loss and round trip time, which are the Ryu controller's strong points. In terms of all performance metrics, the ONOS and HyperFlow controller performs the worst during the whole trial.

The OpenDayLight controller surpasses the POX controller by 1.00% in terms of bandwidth consideration in a uniform host distribution, the Ryu controller by 0.97%, the ONOS controller by 1.24%, and the HyperFlow controller by 1.11%, according to the findings of our experiment. The OpenDayLight controller, on the other hand, surpasses the POX controller, the Ryu controller, the ONOS controller, and the HyperFlow controller by 0.95%, 0.85%, 1.69%, and 1.36%, respectively in non-uniform host distribution. With a uniform host distribution for delay considerations, Ryu controller outperforms in comparison to the OpenDayLight controller by 0.91%, the ONOS controller by 3.30%, the POX controller by 2.61%, and the HyperFlow controller by 2.50%. However, in terms of non-uniform host distribution, the Ryu controller outperforms the OpenDayLight controller, the ONOS controller, the POX controller, and the HyperFlow controller by 0.36%, 1.63%, 0.67%, and 1.32%, respectively. Regarding jitter consideration in a uniform host distribution, the OpenDayLight controller outperforms the Ryu controller by 3.87%, the POX controller by 6.26%, the HyperFlow controller by 17.57%, and the ONOS controller by 10.15%. On the other side, the OpenDayLight controller outperforms the Ryu controller, the POX controller, the HyperFlow controller, and the ONOS controller by 1.57%, 6.49%, 22.92%, and 16.30%, respectively, for non-uniform host distribution. The Ryu controller outperforms the OpenDayLight controller by 1.11%, the POX controller by 4.03%, the ONOS controller by 6.04%, and the HyperFlow controller by 24.02% when round trip time factors are taken into account. However, the Ryu controller surpasses the OpenDayLight controller, the POX controller, the ONOS controller, and the HyperFlow controller by 1.03%, 5.52%, 3.52%, and 30.86%, respectively, in terms of non-uniform host distribution. The OpenDayLight controller surpasses the ONOS, POX, Ryu, and HyperFlow controllers in terms of throughput consideration in a uniform host distribution by 1.02%, 0.99%, 1.77%, and 1.34%, respectively. In contrast, the OpenDayLight controller performs better than the ONOS controller, the POX controller, the Ryu controller, and the HyperFlow controller for non-uniform host distribution by 0.96%, 0.87%, 1.77%, and 1.08%, respectively. The Ryu controller beats the ONOS controller by 12.45%, the POX controller by 20.45%, the OpenDayLight controller by 8.45%, and the HyperFlow controller by 22.45% with regard to packet loss consideration in a uniform host distribution. On the other hand, with non-uniform host distribution, the Ryu controller performs better than the ONOS controller, the POX controller, the OpenDayLight controller, and the HyperFlow controller by 14.435%, 20.435%, 12.435%, and 28.435%, respectively. The summery is shown in Table 3.

Table 3. Feature based Performance Comparison with respect to other controllers in percentage.

Table 3. Feature based Performance Comparison with respect to other controllers in percentage.

A variety of factors can have an impact on the SDN controller’s performance. When the number of actions or switches varies or rises, as well as the amount of stress that is applied, it heavily depends on the selection of controllers. We recommend using the OpenDayLight or Ryu controller when selecting the best distributed (logically centralized) controller. The Ryu controller was recommended by the authors of papers [4,52,64,82,85,91] for their research because of its simplicity of use and Python-based scripting. Additionally, using OpenDayLight controller has been recommended by the authors of papers [17,19,55,76,79]. However, paper [15] is the most pertinent work that fits in with our findings. When choosing distributed controllers, ODL should be used, whereas Ryu should be used for centralized controller selection.

6.1. Topology Description

Some advantages of distributed (logically centralized) controllers include the following:

• Higher-level policies: Rather than using network IDs, language used to describe policies is based on principles.
• Paths should be determined by policy: Based on policy, the controller should choose the pathways.
• Fine-grained control: The data plane keeps a per-flow state while the controller manages the initial packet in a flow

6.2. Challenges of Logically Centralized Controllers

The distributed (logically centralized) SDN controller has problems with interoperability, stability, controller location, and other things. The challenges are discussed below:

Global Consistency: Domain-specific controllers deal with traffic congestion and problems particular to their domains in the distributed SDN control plane. Wherever possible, all other controller instances within a cluster should get critical updates as soon as possible. Maintaining a steady and high-caliber performance over time may be challenging, but it is not impossible [94]. At greater synchronization and overhead rates, strong consistency ensures that all dispersed controllers have access to the same network data. According to Levin et al. [95], control plane consistency can significantly affect the efficacy of a network. To keep a consistent overall view for all controllers, rational trade-offs between rules enforcement and performance are required.

Reliability: For fault tolerance, centralized SDN management uses a simple master/slave architecture. In order to maintain a consistent, logically centralized global picture, network state information must be divided amongst various controllers under distributed SDN control that communicate specifics. In a distributed SDN control plane, there should be coordinated approaches for resolving issues, obtaining simultaneous updates, and maintaining a constant network state [3]. On large-scale networks, the burden can be spread among the other active controllers using a rapid, self-healing method. The maintenance of a sizable amount of state overhead is required by this method, as is the division of domain state among the participating controllers. In the paper Jyotish et al. [96], a performance metric based on dependability, availability, and security is described.

Automatic Reconfiguration: Mapping between forwarding devices and distributed controllers should be automatic rather than using static settings. Static deployments could cause an uneven load distribution among cluster controllers. In order to monitor and communicate load information with adjacent controller scenarios, applications need to be implemented on all active SDN controllers and switches connected to different SDN controllers. However, this approach could overwhelm the controller with load-sharing data, raising scalability issues [94]. Without a consistent northbound or eastward interface, it is also impossible to communicate between apps and device mobility.

Interoperability: Interoperability is essential for the development and deployment of SDN in next-generation networks. This requires compatibility between various SDN controllers operating in various domains and utilizing various technologies. Interoperability is difficult since each SDN controller has a distinct data model and lacks a standard east-west interface. YANG [10], is an open-source data modeling language that enables the standardization and automation of data representations. This NETCONF-based IETF contribution is anticipated to enable SDN interoperability.

Network segmentation: Depending on the topology, distributed SDN controls may cause latency-sensitive applications (monitoring) or compute-intensive applications (route calculation) to suffer. When latency-sensitive and compute-intensive applications are co-located in the same controller, the authors of Chang et al. [97] found it challenging to achieve quick response and convergence times.

The recommended separation of several applications makes use of the slicing that is now available to lessen inter-controller communication. Functional slicing and communication-aware control applications can help to speed up network convergence and response times. To enhance the performance of network partitioning, the scientific community has to do more research.

Problems with load balancing and controller deployment: The SDN controller's integration with the forwarding apparatus generates queries concerning the best locations for controller deployment and the required number of controllers in the network. Such challenges must be overcome, especially in WANs where propagation delay is a major consideration [3]. In other situations, such data centers or businesses (enterprise), load balancing and fault tolerance receive more attention from researchers. The distributed SDN control architecture is scalable rather than being controlled centrally. However, in order to achieve scalability and maintain good performance, it is necessary to physically install the controller as well as use several SDN controllers [9,14].

Security: Security can be compromised throughout the SDN network due to the dispersed SDN controllers' complete network intelligence. The SDN may lose access to the control plane if the safety problem is not resolved on a big scale. The distributed control strategy reduces the risk of authentication and message integrity issues in comparison to a single central controller. Without adequate authentication, an attacker may easily join their network node, making it behave like other SDN controller instances and damaging the whole network. Information exchange between controllers must be safe in order to give unified views of distributed SDN controllers throughout the whole network. To solve these problems in a distributed SDN controller environment, new methodologies and security norms are required [21,22].