Cosimulation of Interconnection between Smart Power Grid and Smart Cities Platform via massive Machine Type Communication

1. Introduction

We are currently witnessing the advent of Industry 5.0, which is estimated to enable approximately 50 billion devices to be connected via the Internet of Things (IoT) by the end of 2023 [1]. There are more than 1.8 M2M (machine-to-machine) connections for every person on the globe, which is a significant figure [2]. As a result of this revolution, the electrical power sector was also equipped with intelligent technology, which led to a significant increase in distributed and non-programmable electrical power microgeneration, smart buildings, smart homes, and smart meters (SM) [3]. This, in turn, has created new challenges for the operation of energy systems. This new environment calls for a smarter network, including the transformation of the traditional energy electricity customer into a "Smart Client" and, in many cases, into a "Smart Prosumer." This has, in turn, led to the emergence of new challenges for the operation of power systems.

It becomes clear there is a need for a smarter grid [4]. InterSCity is a platform for Smart Cities that was developed by USP within a multidisciplinary project that uses digital technologies to make all city services more efficient and reliable, including the supply of electricity [5]. The implementation of the Electric Energy Vertical on smart city platforms through applications (load prediction, monitoring, among others) in SPG requires (1) data analysis algorithms to evaluate the data generated by devices and meters intelligent; and (2) performance evaluation of the various possible architectures to interconnect all SPG elements at the link and physical layer level. In other hand, to connect these systems, a communication backbone is needed. Smart Grids present many different communication challenges, and 5G cellular network technology helps address these challenges because it was designed to meet the heterogeneity of this demand [6]. Recent technologies like MEC, NFV, and SDN help 4G and 5G cellular networks [7,8].

In this sense, an architecture is proposed (Figure 1) based on MEC (Multiaccess Edge Computing) and LTE (Long Term Evolution) technologies that allow mMTC (massive Machine-Type Communications) between SPG components, legacy software resources and smart city platforms. Our architecture was evaluated through simulations and emulations using data from real and simulated devices (Paravirtualization, HIL, and SIL) shared by the SPG laboratories at POLI-USP (NAPREI - Research Support Center at SPG, L-SISPOT - Power Systems Laboratory and L-PROT - Electrical Grid Protection Laboratory) [9,10,11,12,13,14]. The use of data generated by simulators and emulators available in these laboratories opens a range of options for performance and interoperability tests of the various possible solution architectures in the interconnection of applications and these network elements that meet the demands of "prosumer" customers [15,16]. These options are intended to explore communication (mMTC, MEC, B5G - Beyond 5G) between real equipment (IEDs - Intelligent Electronics Devices, SMs, etc.) that make up the network of devices installed in the laboratories.

The rest of this paper is divided as: Section II describes the objectives of this proposal. Section III describes the Methodology used in developing the work. A survey of related works is presented in section IV and a description summary of the systems studied, that is, SPG and the interaction between the NAPREI laboratory and the architectural blocks; LTE and mMTC and the InterSCity platform will be covered in section V. Section VI presents the results of applying the laboratory to test and validate the architecture's functionalities and section VII presents the conclusion and future work.

2. Goals

This work presents a new system architecture that guarantees reliability and latency for SPG using a cosimulation platform that exploits a paravirtualized simulation approach to evaluate the performance of the integrated SPG with legacy information systems and simulated data networks (LTE and mMTC). This approach allows access and management of the devices necessary to enable the connection of smart microgrids, smart homes, smart buildings, smart factories, etc. in the context of smart cities.

3. Proposed Methodology

The steps that guided the methodology in this work were:

• Definition and execution of a series of benchmarks to evaluate the best experimental environment to perform simulations and emulations of LTE network communications in a SPG environment;
• Survey of architecture proposals previously found in the literature and evaluation of them in terms of refined monitoring of the electrical grid and services for SPG systems;
• Adaptation of the best architecture from the previous item so that it can support massive machine-to-machine communication expected in SPG environments;
• Modification of the previous version of the InterSCity platform to support SPG applications with QoS guarantees.

4. Related Works

Simulations and experiments are crucial across various application subdomains within the smart grids [11]. They offer a valuable opportunity to test recent technologies, models, and methodologies in a controlled setting before their implementation in real-world scenarios. The availability of vast amounts of data collected from smart grids has enabled many research approaches. In the article [11], Rossi et al. conducted an extensive literature survey with the aim of gaining insights into different facets of data communication in smart grids, namely: load control, forecasting, clustering, tool support, experiments/simulations, and replicability, in addition to research reproducibility. According to Gavriluta et al. [17], regarding the communication system and distributed computing used in SPG simulation studies, in most previous literature, the communication and distributed computing aspects are ignored, or missing. Therefore, in this survey, we decided to evaluate the available bibliography separately for data network, electrical Power Grid Simulators, and co-simulation with both systems, as presented in the following subsections. This bibliographical survey helped to create the methodology presented in this work, the importance of data communication simulation for smart grid laboratories, and many ways to study smart grids. So, there is room for innovative ideas in testing tools, communication protocols, and new smart grid features.

4.1. Data Network Simulator

In our literature review, several data network simulators were studied, and after selection, the Network Simulator ns-3 was tested. The ns-3 is a communications network simulator widely used in academia because it represents network behavior very close to the real thing, has a very active and connected community of developers, in addition to being open source. The Matlab software [18] together with its Simulink module is very present in most theoretical scientific articles, in the sense of a numerical and statistical approach to the various characteristics of communication technologies, such as channel capacity, frequency response, and quality prediction services [19], but without the flexibility of real-time simulations. Cosimulation environments for SPG are often preferred in the academic literature because of their enhanced flexibility, allowing the integration of specialized tools tailored to specific aspects of a system, thereby providing a more accurate and efficient simulation experience. The pure multi-domain simulators (e.g. Matlab/Simulink, Modelica [20,21]) are solutions to a specific tool, have restricted use and do not show much friendly integration. The OMNeT++ simulator is composed of modules, libraries, and extensible C++ simulation procedures, which can work in real-time and is distributed under the Academic Public License, this fact does not satisfy our criteria for using open source. In article [22], Albagli et al. use the SPG cosimulation framework using high-level architecture (HLA) together with federated MATLAB/Simulink, simulate the electrical models and send attributes to the interface with the runtime infrastructure, real-time, while the federated OMNeT+ + is responsible for receiving the messages from the federated JADE agents and forwarding them to the rest of the system. QualNet was developed to help in the designing and simulating of communication networks [16,23]. According to the manufacturer's official website [24], although it is not sold, SCALABLE software needs a license agreement to be used. The OPNET network simulator [25] has a fixed set of protocols/devices, so the users cannot create new protocols or modify the behavior of existing ones, that is, despite being a simulator widely used by communication system designers, the OPNET does not allow the flexibility necessary for the proposal of this article.

4.2. Electric Power Grid Simulators

In Strasser et al. [26] present a study on the difficulties faced by researchers who carry out studies with cosimulation, a classification is also made for the different simulation modalities, which were analyzed. For the authors, in classical simulation, a single respective tool is used to integrate the model equation, representing the entire operation of the studied system. In their article [27], Duy Le et al. present cosimulation as necessary to model the interaction between the components of so-called CPS (Cyber-Physical Systems). In it, a general survey of several cosimulation for SPG research is carried out. They also concluded that the integrated use of the FCNS, GridLAB-D, and ns-3 simulators is a great contribution to research on smart grids, surpassing the cosimulation speed of other proposals by up to 20% [28]. Applying these tools and the IEEE standard model composed of 13 power buses (IEEE - 13 buses), a study is carried out on threats, security, demand/response and dynamic pricing for SPG. The authors also suggest using this tool for cybersecurity training [28]. Kelley et al. [29] present FSKIT a set of federated tools that perform cosimulation of power and communication networks. To simulate the electrical network, a dynamic simulator of electrical power systems driven by High-Performance Computing (HPC), GridDyn, was used. Zhang et al. [30] discuss a typical SPG cyber physical system for monitoring and control of distributed energy resources (DER). The integration of ns-3 with the energy network simulator, HELICS (Hierarchical Engine for Large Scale Infrastructure CoSimulation) is proposed by the authors in an open-source cyber physical energy cosimulation platform. A case study is analyzed using an abstraction that unites common codes between several software projects providing a generic functionality, framework of cosimulation, distribution and high-performance communication based on HELICS for the coordination of DER. The cosimulation framework for ns-3 that integrates with HELICS was developed by the authors. Souza et al. [31 used the Mosaik framework to study SPG integrated with a power flow simulator. Venkataramanan et al. [32] presents a cyber physical test environment of loads in a microgrid in a cosimulation the power system (RTDS) and emulation a TCP/IP. An architecture adopted by Trajano et al. [33], makes use of the LTE/EPC module from the ns-3 simulator and implements MEC for 4G and 5G mobile cellular networks. This architecture demonstrates that it meets the great demands for an SPG deployment designed by electricity sector forecasts. Although it is not a cosimulation, the ease that ns-3 has in performing paravirtualization provides excellent possibilities for creating testbeds capable of allowing relevant studies in SPG.

4.3. Electric Power Grid and Data Communication CoSimulations

In Table 1 outlines how our proposal compares to previous smart grid co-simulation projects. The comparison covers several important criteria and shows our main contribution of co-simulation, which is to demonstrate the possibility of integration between smart city platform and SPG labs via LTE cellular technology through the network simulator (ns-3). The EPOCHS platform [34] was indeed a breakthrough as it allowed for integrated simulation of these two critical infrastructures, SPG and enabling better insights into how communication networks can affect the stability and performance of power systems. Gaouda et. Al. [35] used the Hampden 180 simulates a power system on a reduced scale, using various electromechanical equipment. Kim et al. [36] presents a co-simulation framework that uses OPNET, a communication network simulator, and OpenDSS, a power system simulator applied to study demand response applications within smart grid scenarios. In the work by Mirz et al. [37], the authors proposed a cosimulation architecture designed focusing on the integration of power system communication and smart grids market analysis. In paper [4], Barbierato et al. proposed a distributed multi-model co-simulation platform for SPG that exploits the communication paradigms of IoT platforms. In the architecture of the NAPREI’s smart power grids laboratory [9,10], applications were developed in an innovative smart grid emulator that is the best experimental environment to perform simulations and emulations of LTE network communications. This structure used only a 6LoWPAN for data communication, a fact that was a problem when real-time simulations were needed, as it did not represent the real situations encountered in the day-to-day of an energy company. The current proposal corrects this gap that was highly requested by the market.

5. Description of Cosimulated Systems

This section describes the implementation of cosimulation of an SPG environment in the context of Smart Cities, so the individual systems are described below:

5.1. Smart Power Grids (SPG) and NAPREI Laboratory

According to Kagan [38], SPG must integrate automation, intelligent measurement and actuation systems, and distributed energy resources. Currently, the SEP - Electrical Power System needs to provide functionalities to meet specific objectives and, therefore, requires ICT (Information and Communication Technology) infrastructure. To manage such intelligent and complex electrical networks, SPG applications use advanced control strategies [39]. JRC’s “Smart Grid Laboratories Inventory 2020” report [40] states that the infrastructure necessary for research in the sector is of vital importance in validating prototypes and their solutions. The interoperability of new computer systems introduced, and performance evaluation of SPG are also proven by this infrastructure. In this sense, NAPREI developed an SPG emulator to study and test a range of functionalities of IEDs and SMs that need to interact with legacy distribution management systems, that is, SCADA, MDM, DMS systems, among others. In the thesis [9], Rosa presents in detail the SPG emulator and states that it proposes an original and innovative form of systemic testability of SPG functionalities in a controlled environment, as it includes IEDs, measurement island, telecommunications infrastructure, legacy systems of information, which are integrated into a diversity of simulations in complex and non-simplified electrical energy distribution networks. The block diagram of NAPREI's SPG laboratory is presented in Figure 2, in schematic form, covering the research objects. The REI emulator (I) was developed for systemic testing of SPG functionalities involving equipment (hardware) and computer systems (software) in the laboratory environment. The measuring island (II) consists of load/generation emulators (Emuladores de cargas e geração), smart meters (Medidores Inteligentes) and communication infrastructure, and is capable of representing any consumption/generation situation of an electrical energy consumer belonging to an electrical distribution network. The telecommunications infrastructure (III) does all the interconnection of the whole system. The use of physical intelligent electronic devices - IEDs (IV) is a fact of great importance in REIs as they are increasingly making use of new systems spread across distribution networks, and the IT systems (V).

5.2. Communication Systems

As can be seen in the central part of Figure 2, the telecommunications infrastructure (III) does all the interconnection of the Emulator (I), the measuring island (II), the IEDs (IV) to the center, and the IT systems (V). In our work, this functionality is carried out by connecting via LTE, as presented in the next section, the ability to interconnect a heterogeneous network (Heterogeneous Network - HetNet) is one of the advantages of using 5G that uses LTE, as it allows a connection with a Macro Cell (Macro BS), that is, through a Radio Base Station (RBS ou eNB, ERB, in portuguese), to a connection through a Femtocell. The purpose of this article is to develop a communication architecture that serves as an IoT interface for NAPREI laboratory simulations. Opening an energy control system perspective on the implementation of global services through SPG systems, such as Residential Load Prediction and NILM - non-intrusive load monitoring. Thus, this architecture is intended to be the basis for future implementation of the energy vertical for smart city platforms.

The electricity sector is one of the most challenging “test cases” for 5G cellular mobile networks, as it has a huge number of different requirements addressed, for example, managing smart metering and low latency in fault location. There are still several issues related to automation, security, resilience, scalability, and portability of 5G network management. As one can see in Figure 2, typically, the literature just describes the needs of a communication system. Rossi et al. [11] present several challenges of smart networks in the interconnection between the physical infrastructure with ICTs (information and communication technologies). One of today's challenges dealing with the explosion in wireless traffic is the deployment of many small cells giving rise to networks. MEC is the most indicated technology to support the smart city needs [41], its multi-access allows the connectivity of a wide variety of devices, including wired interfaces and Wi-Fi (GPRS/UMTS/LTE) simultaneously. Kagan [38] presents telecommunications in different communication protocols, that is, a heterogeneous network, so the proposal of this subproject is precisely to use 4G LTE to support 5G and thus carry out all communications in all protocols, the cloud Communication technologies, in Figure 2, is the focus of our proposal. Data transfer latency provoked by the security system (blockchain) is a very important theme in a system that integrates several heterogeneous components like SPG [42].

5.3. Smart Cities Platform - InterSCity

The JRC report [43], which discusses Smart Grid labs, highlights that the theme of “Smart Cities”, and its nature of connecting different layers of technology, is demanding complete and holistic solutions, which adds value to the smart grid sector. The report highlights the increase in investments in smart grid labs and that there is no forecast of a reduction in the number of active labs in this area. The InterSCity platform is designed to simplify the management and integration of smart city services by offering a suite of high-level, web-based micro-services, as described in Figure 3 [4]. These services afford the necessary tools to handle the IoT technologies, enabling the finding of city services, devices, storing and processing data, and intermediating action commands. The platform supports a range of smart city applications across various domains, known as verticals, with a specific example being Electrical Energy Vertical as the REIs in Figure 2. By mediating data transfer between city applications and services for citizens, InterSCity abstracts the complexities involved in city-scale data management and the specific communication protocols required by the underlying IoT devices. This abstraction layer ensures that users can focus on developing and deploying applications without needing to worry about the intricate details of the city's infrastructure.

6. Results

Rodriques and Almeida [44] presented, in Portuguese, the preliminary results of the LTE/EPC cellular simulation for integration of laboratories for validation and testing of the architecture. However, in this article, it was decided to evaluate the available systems separately for data network and electrical network simulators, as presented in the following subsections.

6.1. LTE/MEC Data Network Simulation

In their article [33], Trajano et al. propose and evaluate an architecture, based on MEC (Mobile Edge Computing), efficient in meeting reliability and latency demands. This considers the distributed applications for SPG via mobile-cellular networks utilizing LTE/EPC (4G and B5G). The authors used ns-3 to demonstrate the proposal's ability to handle a realistic number of SMs, supporting many SPG deployment use cases. MEC is a technology standardized by the ETSI (European Telecommunications Standards Institute) [45]. In our data network simulation, there is a similar proposal as shown in Figure 4.

To revalidate the simulation in this article, mainly checking whether the simulated network can support many SPG devices, LTE network was set up using the parameters listed in ITU-R M.2135-1 report, [46]. In the experiments topologies extracted from deployments are used real LTE/EPC from neighborhood in the cities of São Paulo, in Brazil, that is one of the largest cities in Latin America and has 11.4 million inhabitants with 7.528,26 hab/km² population density, according to the last Census [47]. Figure 5 shows the actual locations of the base stations (BS) of a cellular mobile telephone operator on a map implemented not Google Maps, in São Paulo.

The experiments were conducted using ns-3 version 3.28, leveraging its built-in LTE/EPC module, which provides robust support for LTE-based simulations. The setup involved a machine equipped with an Intel Core i5 8th Generation processor, 12 cores and 4 GB of RAM. From the Limit Theorem Central (95% confidence interval is used to calculate the error) margin repeats all simulations 30 times. Each process for the different scenarios took about 5 hours to complete. Smart devices and meters had their positions randomly generated within a circle with a radius of 1 km from the center of the chosen region (InCor Hospital) and represented in ns-3 with fixed positions using model-defined ConstantPositionMobilityModel. The coordinates for the LTE BS were sourced from those installed in the city (Figure 4). in alignment with the macro urban cell scenario. Figure 6 illustrates the average perceived delay, measured in milliseconds, in communication between devices and the MEC servers. The aggregation factor – AF (Percentage of SMs connected via concentrators) is a crucial parameter to consider for designing a robust intelligent network deployment, as demonstrated in Figure 6.

In Figure 6, as observed in the scenario with 10,000 SMs without any aggregation device, the delay of 90 ms for application requirements is deemed high for certain SPG use cases, according to IEC, this value is not accepted for services proposed for communication in SPG, [48,49]. However, with aggregation the delay decreases. For AF of 25% drops to 60 ms and to 35 ms for AF of 50% that are considered acceptable [50,51].

As can be seen, in Figure 5, there is a relatively large region in the semicircle upper part of our area of interest, including the center of the circle and the Football Museum, which is being serviced by eNBs very far apart (ERB 1 to 5 and 11). Therefore, after a simple cellular service optimization technique, a relocation was proposed for the eNB identified as ERB 3. In Figure 7, the ERB 3 actual position is marked with the red "X" and new proposal position with the Pin Yellow with red circle inside.

Figure 8 shows how effective the relocation was, the average delay perceived in milliseconds reduced to about 65 ms without aggregation. For AF of 25% the delay drops to 43 ms and to 25 ms for AF of 50%. These values are accepted in the services proposed for SPG.

6.2. Integration of the InterSCity Platform between SPG and other Smartcity Verticals

In Article [5], del Esporte et al. focus on the architecture of InterSCity, highlighting its flexibility, extensibility, and scalability. They seem to present experimental results that demonstrate how well the platform can scale, likely in various urban or IoT-based scenarios.

In Article [52], Viana et al. provide an experience report on the software engineering practices involved in developing applications using the InterSCity platform. This likely includes insights into the development lifecycle, challenges faced, and how these practices impacted the creation of smart city solutions. In our experiments, we show how the platform can serve several competing verticals without presenting any scalability problems. During three semesters (2023-2024), 17 computer engineering undergrad students' teams used InterSCity and developed their projects, creating software documentation in GitHub repositories [53,54,55,56,57,58,59,60,61,62,63,64,65,66,67]. Those projects were deployed in only one instance of the platform, at the Intelligent Distributed Systems Laboratory (LSDi - Laboratório de Sistemas Distribuídos Inteligentes) of the Universidade Federal do Maranhão (UFMA), several projects of different verticals of service provision for smart cities were implemented. As a study of NILM, The Hedwig project [68], which is developed by students from the POLI-USP postgraduate program, integrates grouped data with the InterSCity API [69,70]. After collecting real data from September 22 to October 21, 2020, in a connected home testbed in the city of Santo André (10 PIR motion sensors, 10 lamp status sensors, and 8 non- current sensors, invasive - total consumption and 4 sectors - stored in modules installed in the connected home), pre-processing (One-Hot Encoding) and data grouping were carried out with unsupervised learning algorithms. The k-means algorithm performs the grouping of movement events and lighting status of the connected home.

In terms of scalability [5] and security [72] , other experiments with the InterSCity platform using a blockchain-based management model were done. These experiments have proven that the Intersicity platform perfectly meets the requirements demanded for SPG.

7. Conclusion and Future Works

This This work presented massive machine-to-machine communication applications for SPG on the Smart Cities platform (InterSCity), which uses data generated by the POLI-USP Laboratories (NAPREI, L-SISPOT and L-PROT), to implement a mMTC architecture via MEC for the InterSCity platform as applications that manage SPG functionalities. The most significant contribution of this work is filling the gap found in network integration electrical and telecommunications infrastructure necessary for its management and operation. The use of cosimulation demonstrates the effectiveness of the proposed solution, without the need to physically implement any equipment in the field, which would be expensive and traumatic in the operation of a real electrical network. The use of NAPREI simulators and Emulators opened a range of options for performance and interoperability testing of the various possible solution architectures in the interconnection of applications and Smart Microgrids, Smart Buildings, Smart Electronic Devices (IDEs) and Smart Meters. Of these possibilities, we highlight the generation of data for future development of testing applications in Big Data to access the levels of real off-shore equipment (IDEs, Smart Meters, etc.), emulations/simulations of specific configurations for Scalability testing, Heterogeneity, Data Management, Privacy, Security, among other challenges that are intended to be implemented in a vertical of Smart Electric Grids on the InterSCity platform, including the implementation of global energy management. It is proposed, as one of the most important future works of the proposal, to replicate this prediction methodology in the NAPREI network. The project's proximity to NAPREI is a very promising partnership to add to the expertise of SPG laboratories and the InterSCity platform. The various advances achieved at NAPREI demonstrate the laboratory's capacity and that the development of global management is one of the possibilities envisioned in a partnership. Paravirtualization is the hybrid of co-simulation and virtualization. This technology is very interesting because it has all the elements developed so far in research and can then be integrated into the physical infrastructure of the NAPREI laboratory, that is, all its equipment and ICT equipment (NAPREI software) and the data network simulator para-virtualization modules simulated in ns-3. Integration of the physical infrastructure of the NAPREI laboratory, with two other laboratories at POLI-USP, namely L-PROT (Protection Laboratory) which include a real-time digital simulator (RTDS), and the Power Systems Laboratory that includes 4 mini generation systems with digital interface implemented with NI's LabView software [71].