Microverse: A Task-Oriented Edge-Scale Metaverse

1. Introduction

The proliferation of Internet-of-Things (IoT) technology combined with evolving artificial intelligence (AI) and machine learning (ML) promote unprecedented advancements in ubiquitous and seamless services for human activities [1]. The technical evolution in internet technology has made smart cities a reality for the foreseeable future. Since the past decade where the concept of “Smart Cities” was officially defined by the National Institute of Standards and Technology (NIST) in 2014, more than 200 smart city projects have been launched [2]. Many experiences and lessons were learned, preparing human society for a more intelligent, efficient, and sustainable future [3].

Many open questions are yet to be addressed in designing, creating, managing, and living in a smart city. One of the top challenges is the scalability of today’s smart city solutions to an enormous amount of sensors. Not only does the amount of data being generated continuously and drastically grow, but the complexities increase significantly as more intelligent functions are added. For example, IoT network infrastructures are often under different administrations when initially built, but more novel applications or services require resources across domain boundaries. Network Slicing (NS) technology has been identified as an acceptable approach to address cross-domain services [4,5]. NS allows users to create logical networks for individual applications using network devices belonging to different domains. However, creating and managing a network slice that involves sensing, computing, communicating, and data-storing devices deployed across domains is non-trivial. There is a compelling need for a holistic solution that enables full-spectrum smart city operation monitoring, analyzing, decision-making, and taking actions dynamically in real-time.

Recently, Metaverse has attracted interest from both academia and industry [6]. Many expect Metaverse to redefine how people live, work, and socialize by enabling a seamless interwoven of the physical world with a virtual cyberspace [7]. While the dawn of a new era of the immersive, interconnected 3D world appears possible thanks to a wave of revolutionary technologies like augmented reality (AR), virtual reality (VR), extended reality (XR), and virtual Digital Twins (DT), they also bring many new challenges and open problems before a comprehensive digital landscape of Metaverse becomes ubiquitous in daily life. Concerns exist for privacy and security that might utilize Blockchain technology. Meanwhile, scalability is still among the top concerns [8].

Inspired by the vision of Metaverse and the hierarchical Edge-Fog-Cloud computing paradigm, we envision a task- or application-oriented, local-scale immersive, interconnected 3D world that is more feasible. In this paper, we propose Microverse, a task-oriented, edge-scale, pragmatic solution for smart cities’ applications that involves IoT resources under multiple administration network domains. Instead of creating a Metaverse that mirrors the entire smart city in one digital world, each Microverse instance is a manageable digital twin of an individual network slice for a specific task. Following the edge-fog-cloud computing paradigm, Microverse enables on-site/near-site data collection, processing, information fusion, and real-time decision-making.

The paper’s primary contributions are outlined as follows:

• A novel Microverse framework is proposed, and the design rationale, layered architecture, and main functionalities are discussed in detail.
• Using smart public safety surveillance (SPSS) for smart communities as a case study, a Microverse instance is designed and created that contains unmanned aerial vehicles (UAVs), ground units, and sensing networks.
• The feasibility of Microverse is validated through a preliminary experimental study on the proof-of-concept prototype of the SPSS Microverse system.

The rest of this paper is structured as follows: Section 2 provides a brief overview of the crucial enabling technologies. Section 3 presents the rationales and architecture of Microverse, emphasizing the hierarchical, layered architecture. Section 4 describes SPSS as an instance of Microverse including experimental results on the prototype and lessons learned from the case study. Finally, Section 5 concludes this paper with a summary and some discussions on ongoing efforts.

2. Background and Related Work

This section commences with an exploration of the challenges yet to be surmounted by Metaverse developers, which are crucial for actualizing this new digital era in our everyday lives. Following the challenges, the paper describes three pivotal enabling technologies: Digital Twins, Network Slicing, and lightweight blockchain, each playing a vital role in this evolution.

2.1. Metaverse in IoT: Challenges

In the context of smart cities, the fusion of the Metaverse with the IoT—an expansive network of interlinked devices and sensors—heralds a transformative era in urban interaction, blending the digital with the physical [9]. The digital-physical synergy enriches the smart city landscape with immersive, interactive, and data-intensive environments. Yet, integrating the Metaverse within the complex IoT infrastructure of smart cities presents a set of formidable challenges that need to be addressed [10,11].

The primary challenge lies in bandwidth and latency constraints [12]. Applications within the Metaverse necessitate substantial data rates and minimal latency to facilitate seamless, immersive experiences. However, IoT environments usually function on limited networks, potentially inadequate for these high demands. Addressing the communication gap to guarantee fluid and responsive interactions in the Metaverse, especially within the confines of IoT infrastructures, is a pivotal technical hurdle necessitating the development of inventive approaches in network infrastructure and data transmission strategies.

Scalability emerges as a critical issue [13] in integrating IoT and the Metaverse. The IoT landscape, characterized by a continuously growing assortment of devices, sensors, and data sources, presents a complex data and network interaction volume challenge. Adapting the Metaverse to efficiently manage the immense amount of data while maintaining scalability parallel to the IoT’s expansion is a sophisticated engineering endeavor. It is crucial to achieve scalability without sacrificing performance quality.

Interoperability remains a significant challenge in IoT environments [14,15]. Devices produced by various manufacturers frequently employ distinct communication protocols and data formats, complicating their integration. Establishing a cohesive Metaverse ecosystem that incorporates these heterogeneous devices while preserving interoperability is a challenging yet crucial task to ensure a seamless user experience.

Security and privacy are paramount concerns within the IoT realm, and integrating the Metaverse into this ecosystem significantly escalates these risks [9]. The potential for unauthorized access to sensitive IoT data via Metaverse applications poses serious threats. Ensuring the security of data and devices in this converged environment is a critical challenge. Particularly, the amalgamation of IoT data with the Metaverse could create comprehensive user profiles, triggering ethical and privacy issues. Striking an appropriate balance between offering personalized experiences and maintaining user privacy is an intricate and necessary task.

Integrating the Metaverse into the real world presents additional challenges [9,16]. The ambitious endeavor to adopt a Metaverse aims to connect digital and physical domains seamlessly, requiring accurate location tracking, sophisticated object recognition, and advanced sensor fusion [17]. These sensor exploitation technologies could ensure precise alignment with IoT data and devices, yet they represent complex and intricate tasks that must be meticulously addressed for a digital Metaverse.

Last but not least, energy efficiency is paramount in IoT environments, especially those dependent on distributed battery-operated devices. The persistent data demands from Metaverse applications can rapidly exhaust device batteries, underscoring an urgent need for energy-efficient communication strategies. Three emerging technologies include digital twins, network slicing, and blockchains.

2.2. Digital Twins

The concept of Digital Twin was initially introduced in 2002 and subsequently documented by the National Aeronautical and Space Administration (NASA) [18]. A digital twin (DT) is a digital model that accurately represents the components and behaviors of a physical object or system [19]. Unlike traditional simulation, DT is not restricted to one particular process and can contain different procedures. Moreover, establishing the DT system ensures that data communication between the Virtual Object and Physical Object is bidirectional and real-time. These characteristics of DT enable simulation, analysis, and optimization in a broader range of system areas as compared to a simulation.

At first, industrial manufacturing adopted DT technology mainly to enhance different stages of production through simulation, optimization, and the incorporation of machine learning technologies. An example is that of an event-driven simulation that focuses on manufacturing and assembly jobs, utilizing Digital Twin technology and human-robot collaboration [20,21,22]. A proposed framework employs Digital Twin technology to enable accurate and multidisciplinary integration in assembly processes, particularly in industries that deal with high-precision products (HPPs) [23]. HPP also develops a prediction and enhancement framework, along with a practical examination, to validate its efficacy and practicality. The case study illustrates an ice cream machine as an application instance of Digital Twin (DT) in the food industry [24], with a particular focus on the utilization of virtual reality (VR) and augmented reality (AR) technology for visualization and interaction purposes. The framework employs secure data transmission by implementing a secure gate between the computer and the cloud.

Recently, efforts have been demonstrated focusing on many facets of smart cities, such as intelligent transportation, intelligent energy distribution, and intelligent educational institutions. An example is the optimization problem in self-driving cars’ electric propulsion drive systems (EPDS) [25]. The suggested framework utilizes a DT-based approach to establish a connection between the logical twin in the control software and the propulsion motor drive system, enabling the estimation of EPDS performance. Nevertheless, the platform concepts are offered without any supporting experimental data. In another aspect of Smart Driving, a Driver Digital Twin system was proposed that focused more on simulating and predicting the behavior mode and status of the drivers [26]. Compared to other work, the DT framework covers various aspects such as drivers’ distraction detection, attention estimation, drowsiness detection, emotional state prediction, etc. Although the architecture is innovative, the paper did not provide any case studies or feasibility tests.

DT methods are also proposed to support environmental energy sustainability. The DT methods of simulation and digital representation of the world mirror that of the dynamic data-driven applications systems (DDDAS) paradigm from the National Science Foundation in 2000 [27]. DDDAS combines a digital simulation model with the physical estimation model to ensure runtime performance. The big data challenge is mitigated with the reduced order modeling and ensemble methods that afford systems-level performance and analysis. The DDDAS methods were used to control wind turbines [28] and wind turbine farms [29]. Hence, for over a decade, there have been approaches to manage wind power plants by integrating DT and cloud technologies with big data analysis to set up remote control stations [30].

DT techniques have been combined with blockchain, such as a research team that built a prototype of a Smart Campus [31,32]. With the help of a modeling engine, the system enables users to create their avatars inside the virtual campus. The users’ location and basic status are to be reflected in real-time. Moreover, an ecosystem with some simple functions, like the market, is established with the help of blockchain technology. As the purpose of this work mainly focused on social good, the authors did not further investigate the potential of the system.

In recent years, several healthcare applications have reconsidered the concept of DT by incorporating living beings [33,34]. For example, a healthcare framework based on DT is presented to monitor and predict an individual’s health condition by utilizing wearable devices [35]. Moreover, an innovative remote surgical prototype utilizing VR, 4G, and AI is demonstrated, generating a patient’s digital twin and enabling live surgery to be performed over a mobile network [36].

2.3. Network Slicing

Network slicing (NS) is an innovative and transformative idea within the telecommunications field, with the potential to significantly reshape the deployment and management of networks. Network slicing is a fundamental concept that entails the establishment of several virtual networks, referred to as slices, within a solitary physical network architecture [37]. Every individual slice is customized to fulfill distinct criteria, including bandwidth, latency, security, and other performance metrics, to accommodate the wide-ranging demands of different applications and services.

An NS strategy enables network operators to effectively manage resources and tailor services according to the distinct requirements of various use cases [38], such as massive machine-type communication (mMTC), ultra-reliable low-latency communication (URLLC), and enhanced mobile broadband (eMBB). As an illustration, a network slice specifically designed for Internet of Things (IoT) devices may emphasize minimizing power consumption and ensuring extensive coverage. Conversely, a network slice tailored for augmented reality applications may prioritize minimizing latency and providing ample bandwidth.

Implementing network slicing plays a crucial role in facilitating the establishment of 5G networks, wherein various distinct and high-demand scenarios may perate together [39]. Operators can enhance network speed and responsiveness through the dynamic creation and management of slices, guaranteeing a smooth user experience across various applications.

Furthermore, network slicing facilitates the advancement of novel ideas and concepts by offering a dedicated environment for external developers and enterprises to generate and launch their services. The ability to adapt and change the network quickly facilitates the rapid progress of novel applications and services, stimulating economic expansion and technical innovation.

Ensuring security and isolation between slices are of utmost importance in the context of network slicing. The NS architectural design integrates powerful security procedures to mitigate interference and safeguard data privacy. The inclusion of NS into the Metaverse is of utmost importance, particularly due to the coexistence of vital services like healthcare or autonomous cars with conventional mobile services inside the same network architecture.

2.4. Lightweight Blockchain

As the underlying technology of cryptocurrency, like Bitcoin [40], Blockchain has been recognized as a critical technology to guarantee assurance, security, and resilience of networked systems [41]. Blockchain is a distributed ledger technology (DLT) that utilizes cryptographic mechanisms, consensus protocols, and peer-to-peer (P2P) networks to ensure verifiable and auditable transition data storage. All participants in a Blockchain network can agree on a transparent and immutable distributed ledger without relying on any third-party authority. Thanks to systen’s properties like decentralization, immutability, and transparency, Blockchain promises to improve security issues of centralized IoT frameworks, which are prone to single-point failures. Thus, shifting from centralized IoT systems to decentralized and secure IoT systems becomes realistic. Blockchain has been applied to IoT scenarios for security enhancement, like identity authentication [42], access control [43], and trust storage [44], IoT data transacting [45]. Nevertheless, integrating cryptocurrency-oriented methods into IoT networks encounters challenges in performance, security, and scalability.

Various IoT-Blockchain solutions have been reported recently by adopting lightweight Blockchain design for IoT scenarios. By utilizing lightweight consensus protocols, like Proof-of-Stake (PoS) and Practical Byzantine Fault Tolerance (PBFT), IoTChain [46] relies on a three-tier blockchain-enabled IoT architecture to guarantee security and efficiency. To improve scalability and interoperability, HybridIoT [47] leverages a hybrid blockchain architecture that allows a BFT-based mainchain framework to interconnect many Proof-of-Work (PoW) subchains. By combining a round-robin scheduling algorithm with consensus protocol, MultiChain has been implemented on a fog network to guarantee secure communication management for the Internet of Smart Things (IoST) [48]. As a lightweight Blockchain architecture for general IoT systems, Microchain [49] has been applied to diverse IoT applications, like federated learning atop hierarchical IoT networks [50] and Urban air mobility (UAM) systems [41]. By dividing a blockchain into multiple key components that can integrate with a lightweight consensus protocol, network model, and optimize storage, Microchain promises to handle the dynamicity and heterogeneity of Microverse.

3. Microverse: Rationale and Architecture

The concept of a hierarchical Microverse framework entails structuring the virtual environment into distinct, organized tiers, each providing unique functions that contribute to a larger digital ecosystem [51]. Adopting a tiered structure is instrumental for Microverse’s deployment, fostering a multifaceted environment that supports various applications, services, and user interactions. Employing a hierarchical approach offers numerous benefits, such as a systematic method for creating and organizing virtual spaces. Moreover, it allows the Microverse to scale and adapt to changing needs seamlessly, ensuring its continuous growth and diversification.

Using a Microverse underscores the importance of crafting a cohesive and interconnected digital landscape, essential for spurring innovation and enriching the user experience in the ever-evolving world of digital interconnectivity. The layered architecture presents a holistic design strategy that categorizes the diverse components of the Metaverse into specific levels, each serving distinct roles. Given the Metaverse’s growing complexity and its wide array of applications, adopting a layered architecture is essential for managing the intricacies inherent in virtual environments effectively [52,53].

3.1. A Hierarchical Architecture View

Figure 1 shows a layered architectural view of the envisioned Microverse framework. The physical layer contains multiple pervasively deployed IoT networks under different administration domains and run by different service providers. Each IoT network consists of variant sensing, computing, communication, and storage devices, either fixed infrastructure or mobile nodes. A given application or task may need resources from multiple domains to accomplish the job collaboratively. All the devices appointed to conduct the task form a network slice (NS) or a dedicated logic network. Because of the complexities and challenges of network heterogeneity and cross-domain management, a Microverse is created using Digital Twins technology. Essentially, the Microverse mirrors the IoT devices and the working environment in the digital space, enabling an immersive, interconnected 3D working environment. Following the edge-fog-cloud computing paradigm [54], the Microverse functions at the edge using the fog layers to support real-time on-site/near-site operations.

The physical layer forms the cornerstone of the Microverse, offering essential technological and network support for its comprehensive functionality [55]. Central to the physical layer is the edge network, incorporating key assets such as computational power, communication pathways, and storage capabilities that aligns with the contemporary shift towards decentralized and distributed computing architectures. By prioritizing edge resources at the physical layer, the Microverse aims to bolster the availability and efficiency of computational processes, facilitate seamless interaction among entities, and ensure robust storage solutions. The strategic concentration on the edge network within the physical layer not only underscores the growing relevance of edge computing in modern tech landscapes but also highlights its pivotal role in crafting a resilient and flexible framework for the Microverse.

The slicing layer (SL) is integral to the Microverse, serving as a critical regulatory component that dictates the norms and guidelines governing communication and interaction within the intricate digital realm. The SL primary function is to establish a structured system that ensures smooth and efficient data flow management, aligning with Quality of Service (QoS) requirements. The slicing layer is responsible for setting up protocols and standards that aid in the fluid exchange of information within the Microverse. It aims to enhance data transmission and reception efficiency by judiciously allocating resources and bandwidth and leveraging its regulatory authority. The commitment of the SL system to QoS is evident in its ability to minimize congestion, reduce latency, and give precedence to essential data streams, thereby enhancing the overall efficiency and responsiveness of the Microverse. Essentially, the slicing layer is pivotal in molding the communication dynamics within this complex framework, ensuring that digital interactions adhere to set standards and facilitating an environment that adeptly manages data traffic for high performance and reliability.

The concept of a Microverse layer, particularly within the Digital Twin (DT) service layer framework, introduces a model where distinct virtual realms represent and sustain each domain. The Microverse layer’s primary aim is to equip the Metaverse with semantic models and intelligent services, forming the essential infrastructure for the application layer above. By creating separate virtual worlds for specific areas, the Microverse layer offers a nuanced and contextually dense mirroring of real-world phenomena. The Microverse layer integrates semantic models to capture spatial features and a holistic understanding of objects and their interplay. Incorporating intelligence into this layer significantly enhances its capacity to provide advanced services, catering to the diverse needs of applications operating at a higher level. The Micerverse framework positions the Microverse layer as the custodian of domain-specific virtual spaces, delivering crucial Metaverse services. The Microverse layer has the potential to amplify the functionality of digital twin ecosystems and foster a more sophisticated, intelligent convergence of the virtual and physical worlds.

The application layer of the Microverse comprises the components that users directly engage with and access. It serves as a platform enabling individuals to immerse themselves in simulated environments, participate in social interactions, enjoy recreational content, and partake in commercial activities. This layer includes VR applications, AR experiences, and diverse digital content through information fusion systems [56].

3.2. Microchained IoT Networks

Microchained IoT Networks are an innovative paradigm within the domain of Internet of Things (IoT) connections, which introduces a novel age characterized by enhanced efficiency, security, and scalability [41,49]. In contrast to conventional IoT topologies, which frequently depend on centralized servers for data processing and management, microchained networks spread these functionalities over a decentralized network of microcontrollers or microprocessors integrated into IoT devices.

The fundamental principle underlying Microchained IoT Networks revolves around utilizing blockchain technology, which offers a robust and transparent infrastructure for managing data transactions [50]. Every IoT device in the network is equipped with its microcontroller, establishing a node inside the blockchain system. These nodes collectively contribute to the maintenance and operation of the distributed ledger. The decentralization process minimizes the presence of singular points of failure, bolstering the network’s ability to withstand disruptions and diminishing its susceptibility to cyber-attacks.

The term microchained highlights the incorporation of blockchain technology at a smaller scale, specifically emphasizing its deployment at the device level. The micro-blockchain methodology facilitates effective and tamper-proof data storage and retrieval, as each data transaction is cryptographically interconnected and chronologically recorded inside the blockchain. Implementing security measures in IoT devices plays a crucial role in upholding the reliability and authenticity of the generated data, hence cultivating a sense of confidence and trust among the many parties involved. In addition, Microchained IoT networks effectively tackle issues of scalability. The decentralized structure of the design facilitates the smooth integration of supplementary devices without imposing excessive load on a central server. The importance of scalability becomes increasingly significant as the size and complexity of IoT networks continue to grow.

Security issues hold significant importance in the IoT environment since they frequently involve the interaction of devices with sensitive data. Microchained Internet of Things (IoT) networks offer a resilient security architecture utilizing blockchain’s immutability. Illicit modifications to data pose significant difficulties, and the blockchain’s inherent transparency guarantees responsibility and the capacity to track any security violation.

3.3. Microverse: a Task-Oriented Metaverse

The Microverse, as an application- or task-oriented Metaverse, marks a significant shift in virtual environments, moving beyond traditional digital spaces to prioritize goal-driven interactions and efficiency [57]. The Microverse approach redefines the Metaverse into a dynamic, purpose-built ecosystem that streamlines task completion, enhances collaboration, and offers immersive professional settings. Diverging from the typical entertainment-centric virtual worlds, the task-oriented Metaverse focuses on developing specialized functionalities tailored to specific goals, including professional and educational pursuits. It features virtual workspaces meticulously designed to mirror real-life scenarios, seamlessly incorporating the necessary tools and resources to support a broad spectrum of activities.

Collaboration plays a fundamental role in the task-oriented Metaverse [58], enabling users to participate in real-time and spatially aware interactions. The Microverse enables significant interaction by reproducing the intricacies of the actual world, fostering a more immersive and authentic collaboration experience across many settings, such as virtual conference rooms, collaborative creative studios, and educational environments. The Microverse exhibits a broad scope that extends beyond a specific industry, demonstrating its ability to transcend borders and find practical applications in many disciplines, including but not limited to remote work, education, healthcare, and several others.

As illustrated by the top layers in Figure 1, each Microverse instance mirrors an application task, such as overseeing the daily operations on a smart farm, monitoring the safety and security of a smart grid [59], or enabling remote border surveillance to reduce the risk for law enforcement officers. Each Microverse instance is the digital twin of the task space, including the network slice and the application’s working environment. For example, a Microverse instance created for a smart healthcare service that monitors the safety and health of a senior resident living alone in her home mirrors wearable medical sensors and networking devices along with the layout of the rooms and the pieces of furniture [60]. Real-time sensing data is transferred to the Microverse, where the senior’s position, gesture, and health condition are displayed, recorded, and analyzed. When signs of dangerous actions are identified, i.e., falling, an alert is generated and sent to service providers instantly [61,62].

In addition, the Microverse utilizes technologies such as augmented reality (AR) and virtual reality (VR) to enrich users’ immersive experiences in a 3D virtual-physical-interwoven space. These technologies enhance the perception of presence and immersion, making the perceptual experience of tasks more palpable and participatory. For instance, in the Microverse, the healthcare service provider can “see”, “hear”, and “sense” the senior’s status just like she is physically in the same room.

4. Case Study: A Smart Public Safety Surveillance Microverse for Smart Communities

The Microverse framework, while acting as a universal, task-oriented, edge-scale Metaverse, manifests differently across specific instances, as its application heavily dictates its structure. To demonstrate its feasibility, we developed a proof-of-concept prototype, SPSS-Microverse, within Smart Public Safety Surveillance (SPSS) systems for Smart Communities [63]. The SPSS prototype exemplifies the Microverse framework’s capability to seamlessly integrate the physical world with its digital counterpart, thereby offering an innovative approach to bolstering public safety. A detailed description of the SPSS system architecture and main function blocks is beyond the scope of this paper, and interested readers are referred to our earlier publication [63].

4.1. Design Rationale

The SPSS Microverse system employs a sophisticated network of surveillance cameras strategically deployed throughout the town to gather real-time data from multiple vantage points. These cameras are connected to an advanced data processing and analysis server infrastructure. Furthermore, the SPSS infrastructure serves as a gateway to a virtual environment, accessible through a Metaverse application developed using the Unreal Engine 5 (UE5) platform.

Following the hierarchical Microverse framework architecture, the physical layer functions as the principal repository of data, whereby surveillance cameras are employed to oversee and watch various public areas, thoroughfares, and crucial sites. The entire SPSS network is under one administration that manages physical devices, logical networks, and users. The gathered data, including live video feeds and sensor information, is delivered to the server instantaneously. The server processes the data that the user provides and then saves it in a structured database, which guarantees the establishment of a complete repository containing various community activities and events. Furthermore, the sensor and video content is available for user queries such as forensic analysis [64].

The Metaverse layer, constructed using the Unreal Engine, functions as a dynamic reflection that accurately represents the current state of the community in real-time. Instantaneous replication of observed occurrences or deviations inside the tangible realm is promptly reflected within the digital domain. The synchronization of various elements not only serves to improve the situational awareness of public safety personnel but also establishes a framework for effective decision-making and coordination of response efforts.

The SPSS Microverse surpasses traditional monitoring methods by leveraging the immersive capabilities of the Metaverse. Law enforcement agencies and emergency responders can virtually traverse around a digital depiction of the community, allowing them to gather valuable insights into existing crises and enhance their methods for real-world interventions.

This case study illustrates the capacity to integrate the physical and virtual realms to augment public safety in smart communities. Integrating data from the Metaverse and the physical world, this novel methodology lays the groundwork for a forthcoming epoch in intelligent urban governance. In this Microverse paradigm, amalgamating up-to-the-minute data and virtual observations culminates in the development of safer and more adaptable urban settings. As a small-scale, proof-of-concept prototype considering one network domain, this case study does not involve the network slicing layer and Microchained security networks. We leave them to our future work.

4.2. SPSS Microverse Prototype Architecture

Figure 2 shows the architecture of the SPSS Microverse prototype. It was carefully built to connect the real world to the virtual world of the Metaverse in a groundbreaking experiment. The innovative technology consists of multiple layers that smoothly integrate physical and digital components. A group of drones serves as an intermediary between the physical environment and the virtual world, representing the physical layer. An Android device acts as the intermediary layer, enabling smooth connections between drones and servers over WiFi networks. It manages the regulatory framework that governs the data flows inside the system. The Microverse, which serves as the core of the SPSS design, is located on a server and faithfully replicates the complexities of the actual world in a digital realm. Ultimately, a software layer created using the Unreal Engine completes the SPSS framework by offering a visually captivating setting, granting users an unmatched experience as they explore the intersection of the physical and digital worlds. The SPSS Microverse showcases the incorporation of advanced technologies and highlights the capability of these systems to connect the gap between real life and the immersive environments of the Metaverse.

The integration of the SPSS comprehensive system marked a paradigm shift in the representation of physical-world activities within the Metaverse. Leveraging the capabilities of Unreal Engine, the development team created a visually immersive application to mirror real-world data into the virtual realm. Using the innovative Microverse approach enabled the Metaverse to serve as a dynamic reflection of the physical world, with any activity in the tangible realm instantly replicated in the digital landscape. For example, as the drone embarked on its real-world missions, navigating landscapes and collecting data, the Metaverse faithfully mirrored every move, creating a synchronized parallel experience.

4.3. Workflow and System Settings

The SPSS Microverse prototype incorporates a range of key technologies. Table 1 outlines the system’s configurations, while the workflow of our experimental setup is depicted in Figure 3. In this setup, operators pilot drones in the building’s parking area, utilizing the same Wireless Local Area Network (WLAN) that connects to the Surveillance Room inside the building. A specially developed Android application enables the smartphone to relay real-time video and data from various sensors to the Digital Twin (DT) in the virtual environment. As demonstrated in Figure 4a), vital parameters such as longitude, latitude, altitude, and drone’s rotation (yaw spin) are transmitted as JavaScript Object Notation (JSON) packets, ensuring the Virtual Replica’s location in the Microverse is synchronized. Furthermore, the camera’s live feed is streamed to the server via WiFi, employing the Real Time Streaming Protocol (RTSP).

In the surveillance room, a desktop personal computer (PC) operates as the server for the Microverse System, which is developed using Unreal Engine 5 (UE5). This system processes incoming data, encompassing sensor readings and camera footage, to ensure synchronization of DTs and support advanced analytical capabilities. Utilizing live feeds from surveillance drones in RTSP, learning methodologies are employed to execute various algorithms, including object detection, tracking, and trespassing alerts. In the SPSS prototype, we have integrated the lightweight YOLOv8 model for object detection on the video stream. Figure 4b) showcases a real-time demonstration of object detection using YOLOv8, which is integrated into the UE system. Furthermore, to enhance the user experience with immersive interactivity, we have incorporated a VR device, enabling real-time first-person perspectives from the drones.

4.4. Experimental Results

To validate the feasibility of our proposed Microverse SPSS (MSPSS) framework, we developed a prototype using Unreal Engine, accurately modeling an actual engineering building from our testbed premises. The MSPSS prototype includes virtual representations of several drones, as shown in the red circle in Figure 5, which reflect their real-world conditions in real-time. The prototype achieves swift object detection within this virtual setting by leveraging efficient deep learning (DL) algorithms. In addition, employing Meta software allows for an immersive, first-person view from the drone cameras, as demonstrated in Figure 6.

We conducted essential benchmarking tests, focusing on RTSP-based transmission throughput and the end-to-end latency for live streams in UE virtual spaces and VR headsets. Using a bespoke Android application, we configured the video stream at 30 frames per second with a bitrate of 3 Mbps, aiming to reduce latency while maintaining a fluent streaming experience. By calculating the difference between timestamps, we evaluate the end-to-end delays, which refer to the latency between the drone-carrying camera and the live streaming the user experiences in the surveillance room, either from the live cast in UE5 or VR headsets. We also include the additional processing delay caused by object detection algorithms when the user activates this function in UE5, as shown in Figure 7a).

We executed five streaming sessions, each lasting five minutes, at various resolutions. We consider the streaming stable after five minutes for each session and record the end-to-end latency. We observe a reasonable increase in latency in all categories when the resolution is higher. Notably, the Meta software introduces additional processing time after receiving the stream on the PC, which escalates delays in VR device usage. A potential solution could be developing a specialized streaming app for Meta Quest 3, though it remains a task for the future due to current time limitations. Considering the drone’s versatility in supporting different streaming bitrates, we also conducted tests at both 3 Mbps and 5 Mbps. As depicted in Figure 7b), lower bitrate settings are a bottleneck in attaining higher streaming resolutions.

Generally, the user would not experience high latency (less than one second) either when watching a live feed of cameras in the UE platform and VR goggles or inspecting the results of an AI-enabled object tracking procedure. The results validate the feasibility of the Microverse system, as the low latency meets the requirements for most scenarios in real-world SPSS applications.

4.5. Discussions

We chose the Real-Time Streaming Protocol (RTSP) over other streaming protocols like HTTP Live Streaming (HLS) and Real-Time Messaging Protocol (RTMP) due to the RTSP’s superior low-latency streaming capabilities, which are particularly beneficial in video surveillance and closed-circuit television (CCTV) systems. To meet the specific demands of SPSS, we rigorously tested various system configurations, including different bit rates and resolutions, as well as a range of plugins and software tailored for RTSP pull-stream. By optimizing the buffer size, we successfully managed to limit the overall delay to under one second, ensuring smooth and uninterrupted streaming.

Due to available technology, our Microverse prototype and VR system remain nascent. We have not yet completely replicated the entire environment and infrastructure within the virtual realm, limiting the current capabilities and features of the UE-based system. The current efforts demonstrate the integration of the AR, DDDAS, DT, and information fusion technologies important to the Microverse. Future endeavors could extend beyond object detection using RGB streams to incorporate more sophisticated algorithms, such as tracking and sensitive alerts, applicable to RGB and thermal live feeds. Additionally, employing technologies like 360-degree cameras alongside compatible VR headsets could significantly enhance the user’s First Person View (FPV) experience, making it far more immersive.

5. Conclusion and Future Work

The exciting dawn of an immersive, interconnected 3D digital world, is expected to reshape the entire human society, particularly in applications demanding high levels of intelligence. Considering the challenges and constraints yet to be tackled to enable a fully functional Metaverse, this paper introduces Microverse, a task-oriented, edge-scale, pragmatic solution for smart cities, and validates it through a case study. The prototype of a smart public safety surveillance Microverse for smart communities demonstrated that the task-oriented Microverse is promising as enterprises progressively adopt remote work and digital collaboration [65]. The potential of the Microverse technology to significantly alter our methods of work, education, and digital communication situates it as a catalyst for transformation within the continuing development of the Metaverse. Microverse represents a significant advancement toward a more efficient, integrated, and purpose-driven digital realm rather than a simple look into the future. The authors hope this vision paper will stimulate discussions and inspire fresh ideas in our community, guiding us as we navigate the evolving digital landscape of smart cities and embrace the potential of the Metaverse.

On reporting the current vision of the Microverse, our ongoing efforts will be focused on several directions:

(1)

Construct a full vision of the Microverse Platform that reflects not only key elements of the DT system but also other infrastructures in the real world. Instead of the current proof-of-concept, we aim to provide a vivid vision and a more immersive user experience.

(2)

Conduct a more comprehensive evaluation of different transmitting protocols. We will test more benchmark analyses in various scenarios of smart surveillance. Moreover, to simplify the observation of camera footage, we would adopt other push streaming protocols to create a low-latency stream for AI results, both for VR and other usages.

(3)

The current VR function is limited to the cinematic vision of live streaming. With new devices such as 360-degree cameras and VR goggles, we plan to develop applications to provide an FPV experience with low-latency live feed directly from the drones. Moreover, it is necessary to introduce an interactive VR user interface (UI) inside the Microverse Virtual Space and direct control from VR rather than the remote control panel.

(4)

Commercial drones have limited sensor capability and rely on fog-lever servers for AI functionalities. To enhance the task-oriented system, we are building a customized fleet of drones carrying various sensors and edge computing units to satisfy different tasks such as delivery, smart agriculture fire detection, etc.