Toward Reliable Fog Computing Architecture for Industrial Internet of Things

Reliability is essential in industrial networks. In addition, most of the data from nodes of industrial Internet of Things (IIoT) are generated in real time. Thus, those data are mainly used for the time-sensitive applications. Furthermore, device failures should be considered when modeling reliable fog computing for IIoT. In this paper, we provide fundamental aspects to model reliable fog computing for IIoT. First, existing models of fog computing are compared. Then, the most feasible communication type to achieve a reliable system is determined from model analysis. Interaction modes are elaborated to study the advantages and drawbacks when communication is deployed in fog computing for IIoT, and challenges and solutions for reliable fog computing are discussed.


Introduction
The adoption and deployment of Internet of Things (IoT) technologies is leading to architectural changes to Industrial Automation and Control Systems (IACS), including greater connectivity to industrial systems.The authors in [1] reviewed what is meant by industrial IoT (IIoT) and relationships to concepts such as cyber-physical systems and Industry 4.0.Moreover they developed a definition of IIoT and analyses related partial IoT taxonomies along with security threats and vulnerabilities.Furthermore, IIoT has been becoming a critical aspect to realize the Industry 4.0 concepts.In IIoT, numerous devices generate extensive data for a wide variety of applications [2].Data for IIoT applications, such as path planning of industrial robot, industrial automation, and industrial monitoring generally require real-time processing.Hence, appropriate networking and computing infrastructures to support low latency and fast response are essential.
IIoT interconnects critical devices and sensors in essential infrastructure industries and scenarios with existing IoT applications.IIoT deployment allows organizations and users to gain invaluable insights into industrial processes and achieve high productivity gains while reducing cost.However, reliability in IIoT should be considered.Unlike the IoT, IIoT communication links are built based on machine-to-machine links.These communication links must satisfy strict requirements regarding reliability [3].For instance, IIoT applications for process control and maintenance require the system to timely mitigate failures [4].In general, the harsh environment of IIoT deployment contributes to device failures.Hence, designing a a model of network architecture for IIoT should address the related reliability problems.
Cloud computing is a revolutionary mechanism that changing way to enterprise hardware and software design and procurements.Because of cloud simplicity everyone is moving data and application software to cloud data centers.The cloud service provider (CSP) should ensure integrity, availability, privacy and confidentiality but CSP is not providing reliable data services to customer and to stored customer data [5].Both cloud computing and the IoT have a complimentary relationship.The IoT generates massive amounts of data whereas cloud computing is offering a pathway for that data to travel to its destination, thus helping to increase efficiency in our work.Cloud manufacturing refers to an advanced manufacturing model under the support of cloud computing, the IoT, virtualization, and service-oriented technologies, which transforms manufacturing resources into services that can be comprehensively shared and circulated [6].
Although cloud computing offers some advantages for IoT, its approach naturally conflicts with the principles of Industry 4.0.For instance, cloud data centers are generally remote, leading to an unacceptable transmission latency.Moreover, implicit dependency, also known as "vendor lock-in" is another of the disadvantages of cloud computing [7].Thus, makes it be complex and expensive to reconfigure applications to meet the requirements of a new host.Migration could also expose data to additional security and privacy vulnerabilities.Alternatively, fog computing can outperform conventional cloud computing, as it brings the cloud closer to the edge (e.g., end users and devices) to provide low latency and location awareness for the target systems [8].Therefore, fog computing can be suitable for handling IIoT communications and processes.
The primary objective of fog computing is to provide low and predictable latency for time-sensitive IoT applications [9].Fog computing appears as a solution to overcome traditional cloud computing issues, it has the ability to support real-time services (e.g., gaming, video streaming); low latency and distributed computing, storage resources to large geographical areas as well scalability among multiple connected devices [10].According to [11], the layered architecture; physical and virtualization, monitoring, pre-processing, temporary storage, security and transport layer of fog computing, makes it suitable to implement in the IIoT.However, the aforementioned reliability problems should be addressed on designing a model of fog computing for IIoT.
Although several studies have addressed problems related to fog computing, few of them have been focused on fog computing for IIoT.The authors in [12] surveyed the deployment of fog computing for IIoT in which they presented several fog-enabled IIoT application along with the challenges.In [13], the authors proposed an algorithm to achieve ultra-low latency fog computing for IIoT.They addressed the load balancing problem to mitigate latency in the network.In [14], the authors modeled fog computing based on application analysis and deployment for Industry 4.0.Another study addressed a fog computing framework for process monitoring and prognosis in cyber-manufacturing [15].
We summarize the main contributions of this papers as follow.
1. We discuss existing models of fog computing and compare their communication types, which can be applied to fog computing for IIoT.2. We then examine two interaction modes of fog computing and analyze their advantages and drawbacks.3. We also discuss challenges and solutions regarding reliability in fog computing for IIoT.This paper is organized as follow: Section II describes the overview of fog computing and the related computing paradigms.Section III presents the applications of the IIoT.Section IV explains the reliable fog computing model.Section V elaborates the key challenges and the solution of fog computing in IIoT.Finally, in section VI remarks the conclusion and future works of this paper.

Fog Computing and Related Computing Paradigms
In this section, we compare fog computing and related computing paradigms.Communication concepts related to fog computing include mobile computing, mobile cloud computing [16], cloudlet computing [17], and mist computing [18].From them, we only focus on those which would enable reliable implementations of IIoT.In addition, we elaborate the deployment of fog computing and its related paradigms to support different IIoT applications according to the automation pyramid, which is mapped into three IIoT primary layers, namely, perception, network, and application layers [4].Then, the deployment of computing technologies is distinguished based on the number of hops, as depicted in Fig. 1.

Traditional Cloud Computing
Cloud computing has fostered computing capabilities, storage, and networking infrastructures for expansion to applications.Cloud computing has three common service models: infrastructure as a service, platform as a service, and software as a service.The most suitable cloud service model is selected by the developer according to the application.However, this paradigm has several disadvantages.For instance, cloud computing costs are high regarding implementation, maintenance, and management.Hence, cloud computing requires efficient sharing mechanisms to compensate for this problem.In addition, security issues represent a major concern, as client applications often share cloud resources with other clients deployed in the cloud service provider, potentially exposing information As depicted in Fig. 1, the application layer or business intelligence is located on the top layer of automation pyramid.The data analytics, decision making, and resource planning are usually processed in this layer such that it requires high performance computer to provide the results quickly.Since cloud computing composed by high performance computers then it is suitable to perform the aforementioned tasks.Even though the cloud computing already able to provide the quick results, yet it is still can be enhanced by dividing the analytics or learning process with the fog computing and cloud computing.The challenge is to divide the process optimally on both of fog computing and cloud computing.Thus, it is related with load balancing problem to reduce latency in the distributed computing.

Fog Computing
Fog computing is an extension of cloud computing introduced by Cisco Systems, Inc. (San Jose, CA, USA) in 2012.It enables computing, storage, networking, and data management from the core of the network to its edges.Therefore, network performance can be enhanced given that the processes are not only executed in centralized cloud servers, but also along the path to them.The OpenFog Consortium (Fremont, CA, USA) [19] defines the architecture of fog computing as "a horizontal system-level architecture that distributes computing, storage, control, and networking functions closer to the users along a cloud-to-thing continuum."Further details of fog computing are provided in Section IV.
Fog computing can be deployed in the perception layer and network layer or operational technology (OT) and information technology (IT) of the automation pyramid.The devices in network layer (i.e., gateways, switches, and routers) can be acted as a computing resources to perform the task that requested from the OT and IT applications.Due to its location between edge and cloud allows it to disseminate the task to proceed or handed it to the cloud for the further execution.Hence, the quality of service (QoS) of the application can be achieved.

Edge Computing
Edge computing is closely related to fog computing, these concepts along with the concept of mobile edge computing are sometimes confused.However, these three computing paradigms have different characteristics and architectures.As stated by the OpenFog Consortium [19], the edge computing architecture comprises servers, applications, or small clouds at the edge.In addition, edge computing runs its nodes in silos, whereas data should be transferred back through the cloud to establish peer-to-peer traffic.Another definition of edge computing is described by the OpenEdge Consortium .They describes that edge computing is able to bring the computation process to be done at the edge through the small data center which close to the users [20].Therefore, unlike fog computing, edge computing tends to limit services to edges only.
Edge computing has lower latency than fog computing and cloud computing due to its location.In terms of service, edge computing has higher availability than aforementioned computing paradigms.Nonetheless, edge computing is only limited on the edge such that makes it as an isolated computing platforms.On the other hand, fog computing supports multiple industries and application domains.This is illustrated in the Fig. 1 that the scopes of edge computing is only limited until access network whilst fog computing networks is scoped to the metro network.

Applications of the Industrial Internet of Things (IIoT)
As mentioned above, IIoT differs from the conventional IoT given the harsh deployment environment and its strict requirements.Therefore, several distinctive characteristics define IIoT applications.In this section, we summarize the main IIoT applications with their requirements to achieve reliability.Then, the implementation of fog computing for improving each IIoT application is described.

Infrastructure
Infrastructure monitoring has been one of the main IIoT applications, and the rapid development of smart devices has simplified this application.Consequently, improved economic returns and worker safety can be achieved.For example, IIoT deployed in oil refineries serves for monitoring [21] and analyzing big data [22].Likewise, the deployment of unmanned aerial vehicles (UAV) as fog nodes in oil refineries to monitor ground pipelines has been studied [23].The IIoT nodes in this approach are scattered and deployed in large oil fields.Then, data analytics is required to efficiently manage the upstream and midstream oil and gas operations.Thus, reliable systems are essential to implement IIoT for oil refinery applications.
IIoT has been also explored in agriculture.Like in refineries, the IIoT nodes are spread over a large area, reaching long distances from the data control server.Therefore, this setup is very sensitive to failure in the transmission of monitoring data.In addition, agriculture monitoring networks that use cameras for surveillance may demand high energy consumption due to the required computation resources.Moreover, such systems may suffer from delays given the massive transmission of data packets.In [24], the authors explain that applications of the IoT using cameras for real-time monitoring have strict requirements in terms of delay loss and rate.Thus, achieving high network performance for agriculture monitoring applications is challenging.

Maintenance
Employing IIoT for maintenance can reduce costs and improve efficiency of the production line.For instance, IIoT has been implemented to maintain highways [25] and rail networks [26].In these applications, the nodes are deployed in large, open, public areas that can be categorized as harsh environments.Furthermore, infrastructure data are not efficiently managed as they are handled by a single organization.Furthermore, as mentioned above, the extensive data produced from IIoT nodes demand implementations with high reliability.
To further improve efficiency of the production line, machine learning techniques can be combined with IIoT applications to predict [27] and recover [28] from node failures of machinery tools.Efficiency is improved by preventing and minimizing node failures.However, employing machine learning techniques in traditional cloud computing does not support reliable real-time systems, because decision-making is conducted in a centralized manner (e.g., at the cloud server) [29].This makes the systems dependent on stable and robust connections with the cloud.These problems may be mitigated by adopting fog computing.

Supply Chain
The introduction of IIoT has widely impacted the supply chain management structure [30].In fact, IIoT enables the effective management of supply chains by integrating processes, suppliers, and customers through communication links.For instance, RFID technology is being used to monitor supply chain networks comprising suppliers, transport companies, and retailers.In [31], the authors proposed a framework to enable smart, secure, and efficient IIoT systems for supply chains.RFID tags are attached to each product, and the identifiers are stored in a database.To achieve transparency in transactions, both the suppliers and managers receive the information of the entire life cycle of the products.
Fog computing can bring full potential of supply chain.For instance, there are many companies have been leveraging UAV as an aerial courier.However, this systems is not fully efficient yet.Consider that the UAV is already on the mission, then the change of flight path is necessary due to several reasons (e.g., safety, weather condition).The UAV takes a longer time to get its new flight path information from the ground control station (GCS) or cloud.Fog-enabled UAV can bring an autonomous awareness to the systems.Regarding the reliability issues, fog-enabled UAV is capable of widen connectivity between UAV to the cloud or GCS such as depicted in Fig. 3.The issues such as the intermittent, lost or lack of ground support can be tackled by adding fog computing to the current infrastructure of UAV-GCS systems.

Process Control
Process control is an essential stage in manufacturing and involves the systems and software that control production processes.Consequently, problems during process control may adversely affect production.The expansion of automation techniques and monitoring capabilities allows to reduce costs, the number of accidents, and tool damaging at production facilities.The challenge to support process control with IIoT lies on the strict requirements of communication between sensors, controllers, and gateways.Also, the existing infrastructures are not capable of storing and processing the large volume of real-time data.Fog computing not only can reduce the network latency but also the capability of ubiquitous and instant remote access to real-time data without spatial constraint [15].

Overview of Fog Computing framework for IIoT
Fig. 4 illustrates the general framework of fog computing in an industrial environment.It consists of three layers, namely, things, fog, and distributed cloud layers.Industrial machinery tools are located at the things layer and generate data that should be processed depending on the application requirements (e.g., computationally expensive or time-sensitive applications).The fog layer consists of multiple edge network devices (e.g., switches, gateways, and routers), which have limited computing resources but can process real-time or time-sensitive data given their proximity to the data-generating devices.The fog layer may use wireless, wired, or hybrid communications with the things layer.The distributed cloud layer comprises high-end computing servers, and hence data demanding intensive computations can be processed at this layer.Cellular communication links (e.g., 3G, 4G, and 5G) or broadband networks are used to connect the fog and distributed cloud layers.In the remainder of the paper, we denote industrial machinery tools as IIoT nodes, multiple edge network devices as fog nodes, and high-end computing servers as cloud nodes to maintain generality.The communication among layers in fog computing for IIoT can be described as follows.IIoT nodes generate data and request services for either processing at fog nodes or forwarding to the cloud.The fog nodes process the requests from the IIoT nodes and transmit the requests to either other fog nodes or cloud nodes.The requests executed at fog nodes usually contain real-time data (e.g., measurement and product data).The cloud nodes should process the requests and retrieve the results to the IIoT nodes.The generated and processed data contain global information that can be stored in the cloud nodes to achieve global data sharing.

Communication Types
The communication types of fog computing vary depending on the model.In this paper, we analyze the existing communication types of three fog computing models for IIoT, as illustrated in Fig. 5. Fig. 5(a) illustrates the first communication type, where the IIoT nodes can interact directly with either the fog or cloud nodes.Task allocation depends on the workload and data types.If the workload is not heavy or the data require real-time processing, the IIoT nodes pass the data to the fog nodes.On the other hand, if the data do not need real-time processing or the task is computationally expensive, it is passed to the cloud nodes.This model is the most basic for fog computing.
The second communication type is illustrated in Fig. 5(b).Unlike the first type, the fog and cloud nodes perform their tasks individually but collaborate to accomplish the task in a distributed manner.Therefore, the limited computation resources of fog nodes and distance of cloud nodes can be compensated.However, optimal task allocation is critical in this model, because tasks are distributed over the nodes.Common problems in distributed computing, such as load balancing, synchronization, and fault tolerance, should be addressed in fog computing based on this model.Notably, the probability of fog nodes being out of service by problems such as connection loss or damage is high given the harsh industrial environments.
The third communication type to achieve reliable fog computing for IIoT corresponds to a multi-tier model.As shown in Fig. 5(c), this models contains two fog layers.Each fog layer might be constructed with either the same or different interaction modes, which can be centralized or distributed.The interaction modes are detailed in the next subsection.For instance, fog layer tier-1 is constructed by opportunistic devices equipped with enough computing and networking resources (e.g., smartphones and tablets).Fog layer tier-2 is composed of dedicated devices (e.g., dedicated switches, routers, and gateways).Tasks that are not computationally expensive are usually executed by fog nodes at layer tier-1, whereas computationally demanding tasks can be performed by fog nodes in layer tier-2.The most demanding tasks are handled by cloud nodes.Overall, this model has high complexity concerning resource and task allocation.

Interaction Modes
The interaction modes should be considered for modeling reliable fog computing for IIoT.These modes can be determined from several factors.First, the dynamicity of fog nodes determined by the mobility or static nature of nodes.Next, the communication complexity related to the required systems to use specific protocols (e.g., distributed communication protocol).Finally, the deployment cost of fog nodes.Based on these factors, the interaction modes for fog computing, either centralized or distributed, can be defined.
In the centralized interaction mode, each domain at the fog layer consists of a master node and several fog nodes.The master node manages the interaction among fog nodes in its domain.In [32], the centralized mode was used to analyze the delay of fog computing regarding several policies.Specifically, each fog node transmitted its estimated request time to the master node for the latter to track the network condition (i.e., queue time) among fog nodes inside its domain.Then, the master node distributed information on the network condition within its domain for the fog nodes to determine the time to offload their tasks.The fog nodes identified idle neighbors for offloading the task to them.On the other hand, the distributed interaction mode does not require a master node.Instead, fog nodes within a domain can communicate with each other by using a distributed communication protocol.This protocol runs on each fog node to communicate its current network condition, which comprises parameters such as waiting time and task queue.Moreover, each fog node maintains the information in an updated condition table, which registers the current network condition within the domain.Therefore, fog nodes can choose the best neighbors for task offloading.
By comparing the two interaction modes, note that the centralized mode is easier to deploy than the distributed one.In fact, the master node handles the network topology in the domain and maintains the updated network condition.Moreover, all the procedures, including task allocation, are executed and managed by the master node.However, the centralized mode is prone to single-link failure, which can interrupt node interaction if the master node is affected.Therefore, this failure must be considered to implement reliable fog computing for IIoT under this mode.
The distributed interaction mode can be more suitable for IIoT applications comprising several fog nodes in an ad-hoc infrastructure.In this mode, single-link failure does not compromise operation as in the centralized mode.In addition, deployment costs of the distributed mode can be reduced.Still, implementing the distributed communication protocol is challenging.In IIoT applications, the nodes may generate diverse data for different purposes.These data should be handled effectively to meet the application requirements, and thus a robust and efficient distributed communication protocol should be implemented.

Challenges and Solutions in Reliable Fog Computing for IIoT
Although fog computing can be a promising solution for IIoT, several problems remain to be solved.In this section, we explain the main challenges and solutions to realize reliable fog computing for IIoT.

Network Overhead
The network overhead may affect the reliability of fog computing for IIoT.The extensive data produced by IIoT devices can exhaust the available system bandwidth.Network overhead affects the links between the things and fog layers and between the fog and cloud layers.Handling network overhead can increase the deployment costs to guarantee adequate resources and is related to network dimensioning and configuration [33].Thus, the optimal number of fog nodes according to the network demand should be determined.
As illustrated in Fig. 6, there is a tradeoff between latency, network efficiency, and network size in fog computing systems.Latency notably decreases with the number of available network nodes.Network overhead can increase with the network size, and a high number of nodes may undermine efficiency.In IIoT, most of the applications are time-sensitive and require provisioning of fog nodes.Thus, considering the optimal network size for reliable fog computing is necessary.

Security and Data Privacy
Security and data privacy should be mentioned in regard of modeling reliable systems.Similar with the others, IIoT is also prone from the cyber attacks.It can affect the availability, confidentiality and integrity of the data.As mentioned above, IIoT nodes generate large volume data.Those data may contain sensitive information that should be protected.Even though fog computing is capable to process the data locally, still the possible attack may appears to hijack or steal the data.Thus, addressing security and data privacy of the systems is essential on modeling reliable fog computing for IIoT.The authors in [34] reviewed the scenarios and security issues of fog computing.In [35], the authors showed a subset of possible attack scenarios from real life applications.They also presented the solutions on each possible attack.As for the IIoT, further review and investigation should be conducted to analyze the possible attacks.Thus, the solution can be proposed to mitigate fog computing from the attacks.

Node Failure
As mentioned above, harsh industrial environments may compromise the operation of node devices.This problem can be catastrophic for time-sensitive applications.For instance, when a fog node is damaged and the others are processing tasks, the whole network operation may fail.This is because some tasks must be executed in real time.Therefore, critical tasks should be effectively reallocated in these cases.As the centralized mode is prone to node failures that can interrupt communication, solutions based on task management can alleviate this problem, as detailed in the sequel.

Solutions Based on Task Management
Task management may be crucial to realize reliable fog computing for IIoT.Task management determines whether tasks should be executed locally, remotely at cloud nodes, or jointly in both fog and cloud nodes.In addition, it provides node-failure prediction for the proper processing of real-time tasks.Therefore, task management can help overcoming the network overhead and node failure problems described above.
Fog computing using task management can be realized by implementing a thresholding scheme.Specifically, tasks can be distinguished based on their workload and computation time.Using the appropriate threshold, the system decides where the task should be offloaded and executed.For instance, processing periodic monitoring data can be considered as a simple task, whereas a request from an automated guided vehicle to decide its path can be considered as a complex task, as it may require the execution of machine learning algorithms.
The thresholding scheme can also be used to avoid network overhead.Specifically, a fog node will set a limit to perform tasks based on the gathered data.If the data limit is exceeded, the corresponding node will transmit the data either to neighboring fog nodes or to cloud nodes for processing, thus avoiding network overhead.This scheme may be more suitable for the distributed interaction mode, because all nodes can share their resources based on the implemented distributed computing protocol.
To prevent node failures, a timer can be defined on each node to impose time limits for executing tasks.This solution is suitable for the centralized mode of interaction, which is prone to single-link failure.If the master node cannot receive the process results from the corresponding fog nodes within the specified time limit, it is assumed that the fog node has a problem.The timer can be defined using either probabilistic or deterministic mechanisms depending on the application.
To extend the solution, software defined network (SDN) could be considered to deploy as a controller in the fog computing for IIoT.This SDN-based controller can monitor the traffic rate of fog nodes through the northbound API.Instead of each fog node to monitor themselves, the SDN-based controller then distributes task considering received traffic information.Furthermore, the implementation of Virtual Network Functions (VNF) through SDN can also be implemented as a solution.

Solutions Based on Dynamic Fog Service Allocation
The dynamic fog service allocation could be another solution to meet the strict requirement of IIoT applications.In this scenario, the controller or master node can decide whether the fog service should be deployed or released depends on the constraint.The recent work of [36] proposed an heuristic approach to dynamically deploy the fog service based on the QoS constraints.The authors in [37] presented the dynamic management of cloud and fog-based resources for cyber-physical production systems.For the future research direction, a machine learning or deep learning approach could be implemented and tested to optimize the dynamic fog service allocation for IIoT.

Conclusions
IIoT along with fog computing have a huge potential that can realize the concept of Industry 4.0.Nevertheless, there are several challenges that should be tackled to maximize its potential.Strict requirement of IIoT and reliability issues are one of those challenges.In this paper, we present a survey on the fundamental aspects to achieve reliable fog computing for IIoT.First, we introduce the existing communication types in fog computing.Each communication type has advantages and drawbacks related to IIoT.We analyze each of these types and determine their suitability regarding IIoT application requirements.Finally, we discuss key challenges along with solutions to realize reliable fog computing for IIoT.This paper can be very useful in regards of modeling reliable fog computing for IIoT.

Figure 1 .
Figure 1.Automation Pyramid and its relation with the fog computing and related computing paradigms

Figure 3 .
Figure 3. Architecture of fog-enabled UAV as for aerial courier