Development and Application of a Vertical-Agnostic Methodological Assessment Framework for Evaluation of 5G-Based Use Cases

1. Introduction and Motivation

The fifth generation of mobile communication (5G) as well its upcoming technological successor (6G) both offer tremendous innovation potential in a variety of application scenarios and settings [1,2,3]. However, clarity about the achievable benefits of the industrial application of the technology are still missing. Potential end users of the technology state that they lack transparency about what specifically can be gained in their respective domains by adopting the technology, so that they can assess whether the necessary investments are worthwhile [4]. Therefore, it can be stated that even though companies are generally interested in the adoption of 5G/6G, one of the main barriers for technology adoption is still the lack of clarity regarding the potential value proposition that can be realized with the technology to justify the required investments for the end user.

Another aspect that must be considered is that the industrial use of 5G requires experts from different technical domains to collaborate to achieve a shared goal. If, for instance, 5G is supposed to be deployed in a production line, network experts with domain knowledge in networking and communications must collaborate with manufacturing experts that usually do not have detailed knowledge of networking and communication. One central challenge in interdisciplinary technology development and application is the lack of common understanding among experts from different domains which can make communication difficult [5]. A striking example of this circumstance is the term “KPI.” The term is used in almost every professional environment to measure the performance of, for example, processes, procedures, or products and services. However, experts from the network and communications industry and experts from the manufacturing industry consider completely different aspects when defining and using KPI in their respective domains. To collaborate they must establish a common ground of understanding and discussion to be able to speak the same technical language.

To address the described aspects, a Methodological Assessment Framework (MAF) was developed with the aim to provide a domain/vertical-agnostic approach that captures the value proposition of an industrial 5G use cases from different domains/verticals. The approach aims at building the bridge between the characteristics of a private 5G network, the use case utilizing this network, and value proposition that can be achieved for the end user of the considered use case. Therefore, the working hypothesis of this paper is that an end-user-focused MAF can be developed to enable the quantification of a variety of benefits of 5G-based use cases. This working hypothesis shall be verified through the exemplary application of the MAF to three use cases from different verticals.

The paper at hand is structured as follows. Section 2 describes the MAF and its underlying principles in detail while also presenting the defined User-KPI and User-KVI that enable the capturing of a use case’s value proposition. Section 3 illustrates the capturing of a use case’s value proposition. For this purpose, the MAF is applied to three different use cases originating from three different verticals that employ industrial 5G. The first use case “Inline Quality Assurance for Machining” originates from the manufacturing vertical and utilizes 5G to establish a real-time monitoring of critical quality parameters in machining processes. The second use case is titled “5G for Energy Analytics” and was developed in the construction vertical. This use cases aims at the data-based quantification of a construction operation’s environmental footprint and uses 5G to wirelessly transmit metering data from a construction machine to an edge cloud for analysis and monitoring purposes. The third use case is called “Predictive Quality of Service for Tele -operated Vehicles”, and it employs 5G to enable tele-operated driving and the prediction of network performance. Evaluation results based on the MAF are presented for each use case. In Section 4, a summary of the use case evaluations and an outlook regarding future research needs are given.

2. Methodological Assessment Framework

One of the biggest barriers to the widespread use of 5G in industrial application scenarios is the clear quantification with user-centric KPI and KVI of the added value (the value proposition) for users using 5G. Although a variety of approaches have already addressed this issue, the lack of clarity regarding the benefits remains which is also indicated by the low adoption rate of 5G technology in industry [6,7]. Therefore, a MAF is presented aiming at building a bridge between the network domain (5G/6G) and the use domain. The MAF was first described in [8]. The MAF has been developed in the EU-funded research project TARGET-X and has also been validated in the 5G Alliance for Connected Industries and Automation (5G-ACIA) to incorporate the feedback from potential industrial users, the results of which have been published in a Whitepaper [9].

The value proposition provided by the successful application of an industrial 5G use case to the user is at the center of the MAF. For this reason, the perspective of the potential user of a use case shall be taken to evaluate the potential added value. For a manufacturing use case this looks as follows. If a production manager is contemplating the integration of 5G into their operations, the production manager’s focus is very likely on the impact the use of 5G might have on key metrics that describe the performance of the production system. While the production manager should naturally be an expert in their domain, it is unlikely that they will also have expert knowledge in the area of communication networks, so that they will most likely not be able to discuss network-specific metrics like latency and data rates with network experts. This circumstance can complicate communication, coordination, and alignment between network experts and domain experts (such as production) and contribute to further delays in adapting the technology. As a result, the following principle was defined to guide the development of the MAF: “Take the perspective of the end user of the use case and describe what advantages/benefits the implementation and execution of the use case promises compared to the current state of the art”. Based on this principle, the MAF was developed. Figure 1 illustrates the MAF in detail.

The use case which is to be evaluated is located in the center of the MAF. If successfully implemented and applied, a product is created as the outcome of the application of the use case. This product does not necessarily have to be a physical object, it can also be the successful transportation of a passenger with a tele-operated vehicle with the tele-operation being enabled by 5G communication. On the left side of the MAF, the use case requirements and evaluation scenario are located. The use case requirements define the technical requirements that must be fulfilled by the network so that the use case can be applied successfully. The way in which the use case is executed is defined by the evaluation scenario, which describes the sequential execution of the steps required to ensure that the result of the use case is the desired product. This product is then evaluated using pre-defined User-Key Performance Indicators (User-KPI) and User-Key Value Indicators (User-KVI). User-KPI are designed to determine to which degree techno-economic goals, like an increase of the throughput of a production system, are achieved, while User-KVI are designed to determine to which degree societal goals, like a decrease of a use case’s environmental footprint, are achieved. Both, the User-KPI and the User-KVI have been developed together with developers of industrial 5G use cases to capture the value proposition of a use case from a user’s perspective. For each User-KPI and User-KVI an equation has been defined to enable a data-driven quantification of the degree to which techno-economic and societal goals have been achieved. If read from the left-hand side to the right hand one, it becomes obvious how the MAF can build the bridge between network KPI and User-KPI/-KVI. Network KPI describe the capabilities of the employed 5G network so that it can be determined whether the network can fulfill the defined use case requirements. Following the evaluation scenario, the use case is executed, producing the desired product which is then evaluated using the User-KPI and User-KVI to determine whether added value for the end user (the value proposition) is achieved. In this way, a clear and precisely defined connection between network parameters like latency, uplink, etc. and a value proposition for the end user of the use case like increased capability of their processes is created. The creation of this connection is one of the main contributions of the MAF to the objective of supporting the broad rollout of 5G technology in industrial settings.

In order to enable the domain/vertical-agnostic evaluation of industrial 5G use cases, lists of User-KPI and User-KVI have been developed. Table 1 and Table 2 provide an overview of both, the defined User-KPI and User-KVI of TARGET-X.

The User-KPI have been defined to determine- the degree to which four overarching techno-economic goals are achieved by a use case. The first of these goals, expanding process insights was defined based on the assumption that better wireless networking with 5G will significantly increase the data-driven insight that can be gathered by employing 5G in an industrial setting. Since 5G enables the fast transmission of valuable process and product data in high quality and quantity, which can be captured directly in a running manufacturing process, data-driven insights can be gained to learn more about ongoing processes and their performance. The second goal, increasing operational capability, addresses the possibility that the use of 5G will lead to more stable processes that are highly likely to produce the same output parameters with the same input parameters. This fact is addressed by the term process capability which is used to characterize a process according to it being able to meet (customer) specification limits so that the process outcome is within certain specification limits [10]. In the context of use case evaluation this can, for instance, be the dimensions of a manufactured component or the travel time for a route covered by a tele -operated vehicle. In this context the two capability indices c_p and c_pk are of high importance. The index c_p describes the spread or distribution of the process outcome in relation to the tolerance range. The index c_pk describes the centering of the mean of the measured values. The process for which c_p and c_pk are calculated is generally considered capable if both indices are greater than 1 and in industrial manufacturing the required value for both indices is usually greater than 1.33 [10,11]. The analysis of process capability stems from manufacturing and is an essential part of quality management. The principle can also be applied outside of the manufacturing context and transferred to other verticals, since the main idea to achieve reproducible process outcomes can be applied universally. Therefore, process capability is also applied to different verticals here. Increasing process efficiency, which was defined as the third goal, is closely associated with the goal of increasing operational capability as well as the goal of expanding process insights. Based on improved insights and increased process capability, overall improvements of the performance of the considered systems are expected as the systems and their underlying relations can be understood in more detail while the controllability of the processes is increased. Following the hypothesis that the improvements enabled by the use of 5G lead to increased efficiencies and less waste in the use cases, improvements in the profitability are to be expected since the desired results (e.g. to produce a certain product or execute a certain process) can be achieved with less effort and resources in a leaner way. This fact is expressed by the fourth and final goal, increased profitability.

The User-KVI from Table 2 are designed to address certain elements of the 17 Sustainable Development Goals (SDG) defined by the United Nations [12]. The first societal goal, improvement of safety-related aspects, was defined based on the hypothesis that fast and reliable 5G communication enables real -time supervision of safety-critical aspects, like the work and safety zones of robots that operate on construction sites while human workers are also present. The second societal goal, transparency about ecological impacts, was defined because wireless data transmission offers the potential to equip processes (e.g. manufacturing or construction processes) with additional measurement devices to measure the consumption of electrical energy. In this way, the ecological footprint of operations can be calculated, allowing points of reference for optimizing this footprint to be determined at the same time. The third and final societal goal, digital inclusion, was defined due the fact, that 5G enables the use of innovative digital technologies in application areas that have so far been excluded in certain application scenarios. For instance, the use of extended reality (XR) applications on construction sites has not been possible in many cases because wired communication was not feasible on dynamically changing construction sites and the lack of wireless communication options meant that data could not be exchanged over medium and long distances. 5G addresses this issue, enabling the digital inclusion of workers that have so far been left out.

The listed User-KPI and User-KVI illustrate how the principle of focusing on the value proposition from a user’s perspective was prioritized in the design of the MAF. The developed framework enables a multi-perspective evaluation approach of industrial 5G-based use cases and thus contributes to the goal of increasing the industrial use of 5G. In the following section, the MAF is exemplarily applied to three different 5G-based industrial use cases from different verticals to demonstrate it functionality in different domains. For this purpose, some of the User-KPI and User-KVI from Table 1 and Table 2 are employed.

3. Application of the Methodological Assessment Framework

3.1. Application of the MAF to a Manufacturing Use Case

• Description of the Use Case

One central benefit of using 5G communication in manufacturing is the ability to seamlessly integrate quality control and assurance functionalities into production processes. The use case considered in this section was developed to utilize 5G-TSN (Time Sensitive-Networking) in order to enable an inline quality assurance system for machining based on deterministic real-time communication. The use case was developed in the research project TARGET-X and is described in detail in [8,13]. Test results of the use case have also already been published in [14]. Using Frame Replication and Elimination for Reliability (FRER) [15] within the communication pipeline, a bounded latency and jitter of the communication with a guaranteed latency of 10 ms for 99.99% of transmitted packets was achieved. The system architecture diagram of the use case is shown in Figure 2.

For the implementation of the use case, a 5G-enabled smart sensor is integrated into a milling machine measuring acoustic emission signals during the machining. The measured data is wirelessly transmitted to an edge cloud system, where it is analyzed and used to implement an application control logic system to guarantee the desired quality of the machined product. The feedback from this system is then fed back into the machine control creating a seamlessly integrated quality control. For this process control to function, strict requirements regarding latency and reliability must be met. For the use case under consideration, it was defined that 99.99% of all transmitted data packets must be transmitted within 10 ms. The (simplified) FRER setup employed to guarantee this requirement is pictured in Figure 3. It uses a redundant communication setup in which the exact same packet is sent twice via two streams to guarantee that at least one of packet arrives on time which in this case is below 10 ms.

• Description of the Selected User-KPI and Evaluation of the Use Case

The User-KPI timeliness and completeness as well as process capability have been selected from Table 1 to evaluate the use case. Process capability (c_p and c_pk) are calculated according to the following equations for the process capability indices [10]:

$${\mathrm{c}}_p={\displaystyle \frac{\mathrm{U}\mathrm{T}+\mathrm{L}\mathrm{T}}{6\mathsf{\sigma }}}$$

(1)

$$c_{pk}={\displaystyle \frac{\min (UT-\overline{x;}\overline{x}-LT)}{3\sigma }}$$

(2)

In both equations, UT and LT describe the upper and lower tolerance limit for the values. To fulfill the set requirements of the use case, the latency must be within the tolerance specification. Since latency cannot be too low for this use case, only the upper tolerance limit is of relevance here, so that a calculation of c_pk is sufficient and c_p does not have to calculated to proof capability [10]. Figure 4 shows the measured latency per packet for the considered use case. For the measurement, data packets where transmitted from the milling machine to the edge cloud in a redundant way according to Figure 3 while the time stamps of each individual packet were logged. Based on these logs, latency calculations were conducted for each packet. As indicated by the line visualizing the mean, the average latency value was measured to be 6.512 ms while the 99.99 percentile was transmitted with a latency of 8.778 ms. Therefore, the technical setup is able to fulfill the latency requirement. This is also proven by the calculation of c_pk. With an upper tolerance limit of 10 ms, c_pk was calculated to be 1.34, fulfilling the requirement to be greater than 1.33 (see Section 2). Thus, it could be shown that the FRER setup enables a stable and controllable data transmission fulfilling the communication requirements of the use case. Since 99.99% of data have been transmitted within 8.778 ms or less, the requirements regarding timeliness and completeness of data have also been satisfied, as the required data is transmitted in a sufficient amount respecting the latency limits.

The determination of the User-KPI completeness and timeliness as well as the proof of process capability shows that the technical goals to expand process insights (i.e. learn more about the internal relationships within the use case) and to increase operational capability can be achieved for this use case, based on 5G wireless communication, as more valuable data can be acquired on time. FRER is utilized to fulfill latency requirements which are justified by the aspect of controllability in the event of quality problems. Following the MAF depicted in Figure 1, the product of successful use case execution is the real-time transmission of the required data packets to enable an inline quality assurance of the machining process. Based on the calculated value for c_pk = 1.34, it can be concluded that the use case is able to provide a concrete value proposition to a user as the critical variable of the use case (latency) is under control.

3.2. Application of the MAF to a Construction Use Case

• Description of the Use Case

Another core benefit of 5G technology is the potential to establish wireless connectivity in areas that can otherwise not be connected, for instance expansive and dynamic environments such as construction sites. In these settings, 5G acts an enabler for the utilization of digital technologies that can improve existing processes [16]. To demonstrate the potential of the MAF, an energy-related use case from a construction site is described in this section. The use case is titled “5G for Energy Analytics” and has been described in [8] and [17]. The use case aims at creating energy-awareness by integration of a 5G-enabled metering device (Meter-X) for metering of power consumption of electrically powered construction machinery like lifts, cranes, or deconstruction robots. The development and application of Meter-X is described in detail in [18]. Usually, the electrical consumption of construction machinery cannot be monitored centrally, as most of the machinery does not offer the functionality to extract and aggregate the data from the machine control. As a result, the environmental footprint of construction sites cannot be calculated precisely which also contributes to the fact that approaches to reducing the environmental footprint have not yet been broadly implemented in the construction vertical. The 5G for Energy Analytics use case addresses this issue. For its implementation, the metering device is connected to the electricity consumer (material lift) so that the power consumption (voltage and current) can be measured. The metering device is connected to a server through 5G so that the acquired metering data can be logged and analyzed in real-time. One key advantage of the Meter-X is the fact that any construction worker can connect it to the construction machinery whose energy consumption is to be measured. No certified electrician is required due to the fact Meter-X does not require interference with the internal machines circuit as it is a plug-and-play solution that can be connected any electrical consumer. Based on the measurement of the power consumption, the environmental footprint of the use case (User-KVI addressing the societal goal of creating transparency about ecological impacts in Table 2) can be calculated. Following the approach of the MAF, the product of the successful application of the use case is an energy profile for the characterization of the energy consumption which can also be employed to derive optimization approaches in subsequent steps. The system architecture diagram is shown in Figure 5 The measurement values of voltage and current are sent to the edge cloud from the metering device, which is attached to the material lift. On the edge cloud, the received data is analyzed and visualized so that an energy profile and the calculation of the selected User-KVI are created.

• Description of the Selected User-KVI and Evaluation of the Use Case

For this use case, the User-KVI addressing the societal goal of creating transparency about ecological impacts from Table 2 are selected. The calculated values of the User-KVI for a lifting operation carrying 12 Ytong blocks (20 kg each) from the ground to a height of approximately 4 meters and back to the ground are listed in Table 3. For the entire operation, a total power consumption of 0.028655 kWh was measured. The User-KVI have been calculated based on this consumption value in combination with the emission factors for electrical energy produced in Germany, based on the LCA methodology according to ISO 14040/44 [19,20].

The energy profile of the material lifting operation is pictured in Figure 6. The x-axis shows the time, and the y-axis show the measured values for active power in kW with the plot showing different segments of peaks. One peak indicates the active power consumption for the upward movement of the material lift from approx. 15:44:30 to approx. 15:44:45 and the other shows the downward movement of the material lift from approx. 15:45:00 to approx. 15:45:15. As indicated in Table 3, the upward and downward movement of the lift caused a total of 0.007221 kg CO₂-eq. which represents the carbon footprint of this operation over the total execution time of approximately 30 seconds. Thus, the 5G-enabled metering devices, which can be directly connected to a variety power consumers, enables operators of construction sites to calculate the environmental footprint of their operations, creating energy transparency and awareness based on distributed metering systems and centralized energy analysis and monitoring. In this way, it can be shown that the described use case is able to contribute to the overall goal of creating transparency about ecological impacts on construction sites.

3.3. Application of the MAF to an Automotive Use Case

• Description of the Use Case

The third use case that the MAF is applied to is titled “Predictive Quality of Service for Tele-operated Vehicles” in which 5G is used to realize the tele-operation of two vehicles. Both tele-operated vehicles (ToV A and ToV B) travel on the same route. The route is covered by a 5G network, which enables remote vehicle operation. During the journey, a degradation of the Quality of Service (QoS) occurs at some point, potentially interrupting the tele-operation due to reduced connectivity quality. To prevent the ToV from stopping on the road, predictive QoS is used ToV B, allowing it to proactively reroute and avoid areas with low network performance. The predictive QoS concept relies on information provided by the network through network exposure APIs. To quantify the advantage of the predictive QoS, ToV A is not equipped with this functionality and therefore remains susceptible to connectivity degradation. The successful execution of the use case demonstrates ToV B’s capability to anticipate and avoid low-coverage areas, maintaining operational continuity and on-time arrival at the planned destination. The use case is thoroughly described in [8] and [21]. The system architecture for realization of the use case is pictured in Figure 7.

• Description of the Selected User-KPI and Evaluation of the Use Case

One of the main challenges in tele-operated driving is maintaining reliable network coverage and avoiding potential interruptions caused by QoS degradation, which can result in ToVs becoming immobilized in the middle of the road. The presented use case addresses this issue by establishing a technical solution that enables a capable and stable process that consistently produces the same output (timely arrival of the vehicle) under identical input conditions (fixed route with estimation for the required travel time). For this reason, the User-KPI “Process capability (c_p & c_pk) was selected from Table 1. The calculation of c_p and c_pk allows the evaluation of whether a stable and capable process can be achieved through the integration of 5G connectivity and QoS prediction. To evaluate this aspect, ToV A and ToV B travelled along various routes. The experiment was repeated 100 times and for each iteration, the travel time of ToV A (without predictive QoS) and ToV B were measured. The results for two different routes (route one and route two) are pictured in Figure 8 and Figure 9 for both vehicles.

For both routes, ToV B, which is supported by the predictive QoS functionality, achieved lower and more consistent travel times as the majority of the achieved travel times are oriented around the mean values (30.931 minutes for ToV B for route one and 27.458 minutes for ToV B for route two). Table 4 summarizes the calculated c_p and c_pk for ToV B for both routes, derived using Equations (1) and (2). For both routes, the upper and lower tolerance limits were defined by adding or subtracting 5 minutes to the mean value. This was done based on the assumption that a deviation of up to ± 5 minutes was estimated to be acceptable for passengers when traveling in a ToV.

As indicated by the values for c_p and c_pk, the proposed solution utilizing predictive QoS enables reliable and stable use case outcomes for ToV B. For manufacturing processes, c_p and c_pk value greater or equal 1.33 are usually required to guarantee low scrap rates and high yields [10]. For route two, this target range is achieved and even exceeded with the calculated values of 2.838. Although the target range (designed initially for manufacturing processes) for route 1 is not achieved, it can be concluded that the use case is able to mostly function within its specification limit of ± 5 minutes of travel time. The value of 1.082 for c_p and c_pk is equivalent to 0.118% of cases in which the travel time specification limits are exceeded by more than 5 minutes, which is significantly low for such use case. When compared to the values of c_p and c_pk for ToV A, the significant improvement achieved by predictive QoS becomes obvious. The calculated values for ToV A are summarized in Table 5.

The comparison of ToV A and ToV B shows that ToV A, in contrary to ToV B, is not able to achieve any capability to produce reliable outcomes within the tolerance specifications caused by a lack of early detection of network degradation. The evaluation of the use case shows the potential that 5G must positively affect the outcome of use cases when empowered with innovative solutions such as predictive QoS based on network exposure APIs.

4. Summary and Outlook

The paper at hand presents a methodological assessment framework which was developed to capture the value proposition that can be gathered through the implementation of industrial 5G use cases. The use cases have been developed and deployed in an interdisciplinary manner as they originate from different verticals. For the quantification of the value propositions, a variety of User-KPI and User-KVI, which focus on the added value for the end user of the use case, have been derived. These User-KPI and User-KVI enable a holistic and cross-vertical evaluation of use cases that, since they originate from different verticals and application domains, have very different technical backgrounds. Therefore, the main contribution of the developed MAF is the potential to conduct a vertical-agnostic analysis of use cases to check in which way the considered use case addresses key technical, economic or societal goals. In this way, the MAF aims at contributing to an accelerated adoption of 5G technology in industry. By quantifying potential value propositions, the MAF can be utilized to illustrate the potential advantages that the use of 5G has to offer. Furthermore, the MAF can also make important contributions to an improved understanding between network experts and domain experts, for instance production managers. By building the bridge between network KPI, like latency, uplink, downlink, etc., and the value proposition for the end user of the use case through User-KPI and User-KVI, the MAF presents a method that can support exchange between experts from different disciplines and domains by creating a common basis for understanding and discussion.

To demonstrate the applicability of the MAF, three industrial use cases from the vertical manufacturing, construction, and automotive have been described and analyzed. The analysis results are expressed through the presented User-KPI or User-KVI. For all three use cases, it could be shown that the use of industrial 5G provides quantifiable benefits. For the manufacturing and the automotive use case, the calculation of the process capability shows that 5G can improve the capability of specific applications, so that the overall operational capability of, e.g. an organization, can be positively influenced by adoption of the technology. For the construction use case, it could be shown that 5G also enables to enhance the data-driven insights into different processes. This is illustrated by the energy analytics use case, which utilizes a plug-and-play solution to acquire energy consumption data that can then be utilized to calculate the environmental footprint of a construction operation. The working hypotheses, that an end-user-focused Methodological Assessment Framework can enable the quantification of a variety of benefits of 5G-based use cases can therefore be confirmed. In addition, the application to different verticals also shows that the developed MAF is agnostic to the vertical that it is applied in underlying its broad applicability in different scenarios.

So far, the MAF has been applied to 12 prototypical use cases within the research project TARGET-X and to three use cases from the 5G-ACIA consortium. To validate the MAF further and enable it to make greater contributions to the broad roll-out of 5G technology in industry and thereby the digitalization of the European economy, the methodology must be applied to further use cases which are used by industrial end users. In this way, it can be checked, whether the calculation of User-KPI and User-KVI can really motivate decision makers in industry to make investments into the technology when they are convinced that 5G positively contributes to their overall business goals. As the User-KPI and User-KVI can also be applied to new generations of wireless communication technologies like 6G, the presented MAF will also be applicable in the future. However, thorough testing of the applicability must first verify this hypothesis.