Crash Risk Modelling and Analysis of Drone Operation inIndonesia’s Outermost and Underdeveloped Areas using Bayesian Network Approach

1. Introduction

Drone technology is now expected to support efficient cargo delivery as many researchers and industries of parcel delivery develop their drone cargo [1]. Drone cargo is a potential solution for improving logistic distribution in remote areas, especially to improve medical goods distribution [2]. This concept has been proven by the fully operated Zipline company in Africa[3]. It can also be implemented in several regions of Indonesia, referred to as the 3T Region (tertinggal, terdepan, terluar). The 3T Region belong to a district that the Indonesian Government classifies as an underdeveloped district (tertinggal), located in the country perimeter (terdepan), or located in the country border (terluar).

The 3T Regions of Indonesia are characterized by their slower development compared to the average regional development in Indonesia. There are six parameters to assess 3T Region: economy, human resources, public facilities-infrastructures, the financial capability of the region, accessibility, and land/geographical characteristics. A 3T Region lacks all those six parameters that make it included in the 3T Region of Indonesia [4].

The potential solution to the logistic distribution based on drone technology has raised safety and security issues [5]. Safety and security issues are common for arising technology, as drones technology is still in its early periode [6]. To handle this situation, the safety of drone operations must be guaranteed, as an important aspect of drone technology implementations [7,8]. Safety has many different definitions according to the subject of analysis. As for the drone operation, safety of drones operation can be defined as a “state in which the system is not in danger or at risk, free of injuries or losses'' [9]. In other references, it can be found that safety is defined an as absence of unacceptable risks[10,11]. Moreover, drones technology is a combination of cyber and physical system, which make drones more vulnerable to safety risks and security threats.[12]

Since the implementation of drone technology is still developing in many fields, risks and hazards of drone operations are still not well-known. This limited knowledge of risk can endanger the safety of drone operations, resulting in inadequate standards and regulations regulating drone operations for specific purposes, such as for logistic distribution. This is one of the main barrier for the drone technology to be implemented for many civilian fields [13,14]. Furthermore, limited policies, standards and guidelines that ruled the civilian drone operations creates problems for research and development. Nowdays approach to safety of drone operations have been shown by the increasing number of accidents and incidents of drone [15]. From 2014-2017, FAA recorded that more than 4,889 incidents reported, which can harm people or third-party property. This condition shown that it is important to consider safety as an important requirement in every system and application design, and even more for systems that can cause severe hazard [9]. However, without understanding of potential risks drone implementations, the safe drone technology for civilian usage will not be feasible.

To regulate safe use of drones, safety assessment is important step to be accomplished. Safety assessment of drone operations has been conducted with different approach to assess the risk of incident or accident during a drone mission. There are two different approaches so far, which are quantitative [16,17] and qualitative [9,18,19] approaches. For a glance, the FAA established a process of pre-flight assessment, where the Safety Management System (SMS) was implemented to recognize the risks and to provide the strategies to mitigate the risks. a Model-Based System Engineering (MBSE) and a Model-Based Safety Analysis (MBSA) are proposed by Mhenni et al in [18]. The model used by Mhenni integrated quantitative techniques such as Fault Tree Analysis (FTA) and qualitative techniques such as Failure Mode and Effects Analysis (FMEA).

A safety assessment process model for a UAV using Petri Nets is demonstrated by Gonçalves et al [16]. presented. In [20], a Bayesian Belief Network is adopted for risk analysis of small drone systems. Other works such as [21,22] identify and asses of risk-factors in a drone operation such as obstacle collision, untimely battery drain, human factor by using qualitative approach. FAA in FAA Order 8040 method for risk management is also using a qualitative approach. The qualitative approach has a nature that can lead to irreplicable, unpredictable, and unclear results.

Another approach to risk assessment is quantitative analysis. Quantitative analysis is also required to achieve a precise safety assessment [23]. Moreover, quantitative probabilistic risk analysis is recommended by [19]. An integration of qualitative such as Risk Matrix and quantitative risk analysis such as Bayesian Network has been proposed by [24] based on ISO standards, ISO 12100 and ISO 13849. However, to develop a risk analysis model using Bayesian Network, a specific real-world data is important to represent the real conditions of drone operation in a specific region and specific drone function. As far as we know, there are no research that developed model of a quantitative risk analysis based on a specific drone operation.

In this paper, we modelled a risk analysis of drone operations for emergency and medical supplies in some inaccessible areas (3T Region) of Indonesia, Wamena Papua, using Bayesian Network to identify hazard and risks drone operations in tropical island. We propose an approach to identify a significant factor that affect safety drone operations in certain area, such as 3T Region of Indonesia. We also collect several experts’ judgement of drone operation in Indonesia and weather recording to get insight of risk analysis model for drone operations in 3T Region of Indonesia. Furthermore, we simulate the drone operational risk analysis given some states of operation in 3T Region of Indonesia and we also conducting the sensitivity analysis for the model.

2. Operational Risk Assessment of Drone using Bayesian Network

Risk assessment or risk analysis modelling with limited numbers of data can be accommodated by BN. BN approach provides a quantitative method to analyze risk based on causality between events and given probability of events. It also applies to drone crash risk analysis. Drone crash risk depends on many events that can affect drone flight, which can be categorized as external and/or internal factors. Both external and internal factors are not always present or have the same effect on drone flight. They can be treated as probability problems in the risk analysis modelling.

Bayesian Network (BN) or Bayesian Belief Network (BBN) offers a causal structure that is usually used by risk analysis practitioners to obtain a more comprehensive approach to risk factors. BN method is more popular nowadays in fields related to reliability, safety, and maintenance [25]. BN method combines expert judgments on a probability and frequency of occurrence to investigate the risk or reliability of an event [26]. Thus, it can accommodate external human performance factors such as company habits, society, and technology.

The BN methodology was developed to simplify the predictions process. It used graphical frameworks that represent arguments within an uncertain domain. BN graphic is a unicycle graph, which means it cannot form a closed cycle and only have one direction[27]. Each node represents random variables, and the arrow represents direct dependency between nodes. Consider Figure 1 as an example of graphical representation of Bayesian Network, where H is caused by T and P. T and P nodes are called as a parent node and H is called child node.

Figure. An example of Bayesian Network in graphical representation [25].

Figure. An example of Bayesian Network in graphical representation [25].

As an example, look at Figure 1, an analysis of human error probability is presented. Node T and P represent an external factor and human performance factor, while node H represent human error probability. Node H is affected by nodes T and P. Each node has its own conditional probability table (CPT) to represent the variable probability of occurrence. The general solution of variable H probability of occurrence conditioned to the occurrence of variable T and P is given by the following equation (1) [25].

$$p\left(H\right)=\sum _{i=0}^1\sum _{j=0}^{1}p(H=1/T=i,P=j)p(T=i)p(P=j)$$

(1)

For example, the probability of variable H being true, conditioned to variables P and T being true or false, the solution of H probability is given by equation (2) [25],

$$p\left(H=true\right)=p\left(H=true/T=true,P=true\right)p\left(T=true\right)p\left(P=true\right)+$$

(2)

3. Drone Crash Risk Modelling using Bayesian Network

3.1. Study Area

Drone crash accidents were caused by a variety of factors, such as electromagnetic interference, environment conditions, obstacles, navigational environment, air traffic environment, electrical environment, communication, human factor, and internal system. Since location of drone operation will determine crash risk, this research focus in one of 3T Region in Indonesia, which is South Manokwari. This paper mainly analyzed the probability of drone crash accident occurring in South Manokwari, Papua.

South Manokwari, one of the 3T Region of Indonesia, located in West Papua. South Manokwari has limited access of road, which makes South Manokwari as a good example of location for drone cargo implementation to improve its logistic distribution and health access. Due to rain, road in Isim District of South Manokwari is flooded and becomes muddy road, which cannot be access using car. This condition can cause problem to health logistic distribution as can be seen in Figure 2 and Figure 3.

3.2. Risk Identification and Network Building

The BN model topology depends on the number of variables considered in a risk scenario and how they are linked to each other. The number of variables directly determine the number of nodes used in a BN model. By using Bayes’ theorem, one or more variables can be updated based on the evidence stated in one or more variables. The rule of updating variables is determined by the way that one and other variables are linked.

The risk scenario of a drone crash during operation in the 3T Region of Indonesia depends on many external and internal factors. The external factors consist of but are not limited to, interference, environmental conditions, obstacles, navigational environment, air traffic environment, electrical environment, communication, and the human factor. The internal factors can arise from mechanical, thermal, electronic, algorithmic, technical factor, software, and hardware. The number of variables involved in the model should be enough to represent the significant phenomena to build a proper crash risk model of drone operation, but not so many that cause excessive and unnecessary calculation.

The number of variables in the BN model is determined based on the available data and subjective view of the model developer as an expert in a specific field. This method is applied because the BN model not only considers relative frequency as a statistic element but also consider causality between variables. That is why in the BN model subjective view is inevitable to model a proper risk scenario.

In fact, many data are hard to obtain from measurement such as reliability of drone system, reliability of each drone components, probability of disasters, etc. Therefore, this research only focuses on two factors to be included in the Bayesian Network model. The weather effect is comprised of wind speed and rain/precipitation probability. Those two variables are specific for each place in the world, including the 3T Region of Indonesia. The wind speed and precipitation probability model in this research represent the condition of Indonesia low altitude atmosphere (< 120 m) in Papua, Indonesia. The internal system component failure or malfunction (SCFM) probability model represents the probability of a non-critical onboard system to be failed or malfunction during drone operation.

As can be seen that to build a BN model, data is important. The more complete data, the more accurate the BN model will be to predict the probability of an event. It is difficult to collect the related drone data in Indonesia because drone application is still in its early development phase and that the regulation for drone operation data recording requirement has not been established. Without established drone operational regulation, the operators may have different method of data recording and only consider data that important for their own purposes.

In this research, the BN model’s variables are determined by considering the experience of drone operators in Indonesia. We have conducted interviews with three different popular drone operators in Indonesia. Three managers from different Operator (1, 2, and 3) are interviewed to qualitatively assess risk factor of UAV operation. As a background, Operator 1 is a multinational company specializing in aerial mapping and inspection, with most of the company's client are oil and gas company. Second company (Operator 2) is from technology startup which provides wide range of UAV applications, including aerial mapping, tower inspection, and pesticide spraying. Lastly, Operator 3 is from non-government organization specializes in disaster mitigation and damage assessment using UAVs, and already deployed in several events, such as Garut flood in 2016 and Palu flood ad earthquake in 2018. Even though all the three interview resources have different business products, all of them have experiences operating in remote areas around Indonesia.

In the intervie, three drone operators in Indonesia gave us information about the challenge in operating drone in Indonesia based on the company experience. Genuine experiences of the drone operators in Indonesia give more accurate judgment to develop proper risk model of drone operation in Indonesia. Using BN approach, this testimony will become valuable data for the model. The result of the interviews is summarized in Table 1.

From the Drone Operators’ explanation, factors that can cause drone to crash during operation in Indonesia are high wind speed, storm, inaccurate pilot judgement, miscalculated of natural obstacles (tree), artificial obstacle (electrical wire, pole, etc.), inaccurate guidance system and wild bird attack.

Based on the interview results, all the interviews confirm that there are several environmental factors that have a significant effect, which are wind speed, terrain topography, predatory birds and trees, while others have less effect on the drone operation, such as rain (to some degree). The result of the interview can be seen in Table 1, indicating the significant factors that caused drone incidents. From this fact, the wind speed will contribute higher to drone crash probability, while rain and SCFM will have less effect on drone crash probability.

Crash risk probability is modelled using three parent nodes (Rain, Wind, SCFM), one intermediate node (Weather Effect), and one child node (Crash Risk) that are connected, as shown in Figure 4. Each parent node has its own probability distribution assigned to represent the probability of occurrence. Wind speed is classified as low, medium, and high based on its effect on a drone in flight. As can be seen in Table 2, low speed wind is 0-4 m/s, medium speed wind is 4-8 m/s and highspeed wind is 8-10 m/s based on Operator 1’s explanation about wind effect on drone operation. Low-speed wind does not have any effect on drone operation, while medium speed can cause deviation from the desired flight direction, and high-speed can cause loss-of-control on drone operation.

The effect of rain intensity (precipitation rate) is not significant to the operation. It is important to note that drone operators in Indonesia usually avoid flying while raining to reduce the risk of uncontrolled weather suddenly turning into a storm. The rain classification by BMKG (Meteorological, Climatological, and Geophysical Agency) as shown in Figure 5 states that light rain is less than 100 mm/month, medium rain from 100-300 mm/month, heavy rain for 300-500 mm/month, and very heavy rain for those higher than 500 mm/month.30 The effect of each class of rain intensity is minor to UAV flight, as stated by one of the operators. Thus, the causality of rain to drone flight will not be specified in detail, but it can be assured that the rain affects in reducing visual of the ground pilot that may lead to increased crash risk.

Rain and Wind nodes are connected to Weather Effect as an intermediate node to simplify the process of modelling the crash risk’s conditional probability table (CPT). The size of Crash Risk’s CPT is determined by the number of its parent node. By using weather effect as intermediate node, it reduces three parent nodes (Wind, Rain and SCFM) into two parent nodes (Weather Effect and SCFM). The Weather effect node has three states that represent the intensity of weather effect to the crash risk, which are light, medium, and heavy. The CPT of the weather effect node is built based on each factor’s contribution to the drone crash. The CPT for weather effect is shown in Table 3.

The SFCM node represents the flight data of UAVs system component failure or malfunction occurrence in an annual average, which was adopted from Reference [24]. However, there are different definitions of SCFM in the data collected by Reference [24]. For example, the data obtained from NASA says that SCFM is one of the significant factors that can cause UAVs to crash, which is estimated that cause 60% of UAV crash cases. As for other resources than NASA, SCFM probability is in the range of 6-32%. This probably happened because NASA and other resources have different definitions of SCFM. SCFM with a low-risk percentage is selected because this research assumed that SCFM only include a minor component that, if it fails, will not cause a major effect on drone flight.

The crash risk node is the child node of the BN model. It is required to have a fully defined CPT. One of the challenging steps in modelling probability using the BN approach is constructing a proper CPT table to represent the real conditional probability of occurrence based on all the contributing factors. For the crash risk’s CPT, a table is created to map every combination of weather effect states and SCFM values, as shown in Table 4,

The crash risk node has four states, which are adopted from risk severity ranking or category in ISO 12100. The severity of a hazard is measured based on its consequences to people, operations, drones, and the environment. The four categories of severity are catastrophic, critical, marginal, and negligible. Each severity category is adopted from Reference [24]. They are,

• Catastrophic: the hazard causes harm or serious injuries or deaths to humans. The severity of such hazards is the highest considering that it affects human safety and thus must be carefully addressed and removed to avoid fatal situations.
• Critical: the event influences third parties except people, for example damaging buildings or assets in general.
• Marginal: the event causes damages to the drone system itself.
• Negligible: the event does not affect the operational capability of the drone (safe operation), or the drone doesn’t get crashed at all.

Using Operator 1’s drone operations annual data in 2021.

Statistical data of rain probability (Rain node), wind speed probability (Wind node) and system component failure or malfunction (SCFM node) are put in each associated parent node. Wind and Rain nodes are connected to the Weather Effect node. This requires a definition of CPT of the weather effect node. Then the weather effect node and SCFM are connected to the Crash Risk node as shown Table 2. All this data and relationship definition between nodes are implemented in AgenaRisk26.

AgenaRisk is a Bayesian Network software designed for Risk Assessment by utilizing the Bayesian Network Method. AgenaRisk provides the user with a graphical user interface (GUI) to create the Bayesian network model. Thus, the user does not have to create the subroutine manually. AgenaRisk can also be used for diagnostic inference, causal inference, scenario analysis, and sensitivity analysis. Users can choose from GUI what analysis is required to be conducted.

AgenaRisk offers severall options to model each node of a probabilistic event. For example, users can choose to model a node as a Boolean, ranked, continuous interval, integer interval, etc. Each node type requires user to assign probability of the event. Each node can be connected to create a link based on the user requirement. If a node is determined as a child or intermediate node, then the user has to set a CPT that govern the contribution of its parent node to the child or the intermediate node probability of occurrence.

4. Result

Authors should discuss the results and how they can be interpreted from the perspective of previous studies and of the working hypotheses. The findings and their implications should be discussed in the broadest context possible. Future research directions may also be highlighted.

4.1. Prior Probability

After the BN model is completed, then it is used to run several simulations to obtain its prior probability. The prior probability of a Bayesian Network model is a probability distribution of all nodes (parent and child nodes) based on the data sources and calculation using Bayesian Approach.

The parent nodes, which include Rain, Wind, and SFCM, are presented in their probability of occurrence. Rain probability of occurrence is mostly medium to high intensity of rain. For wind speed, the most probable condition of wind speed is low-speed wind. For the SCFM probability of occurrence are mostly not happened during flights.

The prior probability of the model shows us the crash probability of occurrence for each severity level. The risk’ factor nodes probability of occurrence is given based on the data sources. As can be seen in Figure 6, prior probability is shown as negligible (94%), marginal (4.5%), critical (less than 1%), and catastrophic (less than 1%). This means that the drone operation will most probably result in safe operation without the crash. The result of the modelling risk of drone operation, in this case, is similar to annual flight operational data gathered from one of the drone operators in Indonesia. It means that the BN Model can represent the condition that was described by the drone operator. Sensitivity analysis is important in risk modelling and analysis. Sensitivity analysis helps researcher, who develops the model, to examine risks’ factors contribution to the occurrence of drone crashes.

4.2. Scenario Analyses

The BN Model can also be used to analyze different scenarios, which can happen in a cargo drone operation. For example, when an urgent cargo such as a blood bag needs to be delivered during high or medium wind speed, what will happen to the drone cargo can be predicted based on the model. Then, the cargo drone operator can determine whether to fly the drone or cancel/postpone the operation based on the crash risk distribution for each scenario analysis. In this case, the effect of medium and high wind speed on drone crash risk calculation is demonstrated.

The result comparison between medium wind speed and high wind speed hypothesis can be seen in Figure 7 and Figure 8. For the medium wind speed hypothesis, the distribution of drone crash probability is shown as negligible (88.5%), marginal (8.5%), critical (1.5%), catastrophic (1.4%). While for the high wind hypothesis, the distribution of crash risk is shown as negligible (79.9%), marginal (12.5%), Critical (4.2%), catastrophic (3.4%). Critical crash probability increases from 1.5% to 4.2%, and catastrophic crash probability increase from 1.4% to 3.4%. Those results indicate the increase of wind speed level leads to the increase of crash risk level.

Flying in high wind speed conditions has a higher risk of resulting in a critical or catastrophic crash. In this case, the probability of critical and catastrophic crashes can be reduced by applying proper mitigation techniques such as designing a robust drone system to compensate for wind effects during flying.

4.3. Sensitivity Analysis

By knowing how each risk’ factor affect drone crash risk, causality between risk’ factor and crash risk can be checked to confirm whether all CPTs is correct. For example, if the sensitivity result shows wind speed contributes higher than rain intensity, it matches the reasoning stated in the scenario. If not, then the CPT table needs to be tuned again.

AgenaRisk provides its user with an automatic sensitivity analysis. The user only needs to choose a target node and any number of sensitivity nodes. Thus, choosing the “Crash Risk” node as the target node and wind, rain, and SCFM nodes as sensitivity nodes will result the tornado graph shown in Figure 9, Figure 10, Figure 11 and Figure 12.

AgenaRisk presents sensitivity analysis in a tornado graph to find out how each contributing factor affects the target node (“Crash Risk” node). From a pure visual perspective, the corresponding bars to each sensitivity node in the tornado graphs illustrate a measure of the contribution of that node to the target node. In this case, each of which corresponds to each crash level of severity (negligible, marginal, critical, and catastrophic). As can be seen in Figures 9–12, the node wind has the most contribution to the “Crash Risk” node for all levels of severity. The result matches with the data obtained from statements of the drone operators in Indonesia.

For a detail interpretation of the sensitivity analysis result, Figure 9 shows the tornado graph for negligible crash risk. In Figure 8, the probability of negligible crash given the result of wind go from 0.8 (when wind speed is high) to 0.96 (when wind speed is low). Higher wind speed is reducing the probability of negligible crash risk, while other factors (SFCM and Rain) have less effect to the negligible crash risk represented by much shorter sensitivity bars of SFCM and Rain factors. This condition matches with the model that SFCM and Rain factors have smaller impact to the crash risk compared to Wind Speed factor.

Figure 10, showing tornado graph of marginal crash risk, describes probability of the marginal crash event given the result of wind go from 0.035 (when wind speed is low) to 0.125 (when wind speed is high). Other factors (SFCM and Rain) don’t have significant impact to the probability of marginal crash, which can be seen from their short bar length compared to Rain.

Figure 11 shows the probability of critical crash event given the result of wind go from 0.008 (when wind speed is low) to 0.042 (when wind speed is high), while other factors (SFCM and Rain) don’t have significant impact to the probability as indicated by their short bar length.

Figure 12 shows the probability of critical crash event given the result of wind go from 0.0025 (when wind speed is low) to 0.033 (when wind speed is high), while other factors (SFCM and Rain) have less significant impact to the probability as indicated by their short bar length. The sensitivity result can closely represent the Operator 1’s data. As a result of the most flight in year 2021 is more than 94% safe, while there is a small probability of marginal dan critical crash, and the main factor to the increasing of crash risk is wind.

5. Conclusions

This paper presents the development and simulation of a pioneer model for drone crash risk analysis using the BN approach specifically attributed to an Indonesia 3T Region. It is important to note that, in this case, the data required for constructing the BN model is limited. This limitation is due to the fact that until now, a rule/regulation that obliges the operator to record operation data in a particular manner/system is not yet established. In this research, this problem is taken care of by using a ranged value format based on the drone operators’ details about each risk factor they consider. The result shows that the model can represent the risk of drone crash described by the operators in Indonesia. Most of the drone flights result in a safe condition or minor issue that doesn’t affect the operation with the probability of 94%. Less than 5% of the drone will crash, which only affects the drone itself, while less than 1% of the drone operation will cause harm to a third party or human life.

From scenario analysis, the increase of wind speed contributes to an increase in risk level. High wind speed cases result in higher risk level severity compared to medium wind speed case, where critical crash probability increases from 1.5% to 4.2%, and catastrophic crash probability increase from 1.4% to 3.4%. This result is confirmed by a sensitivity analysis that the most contributing factor to crash risk is wind speed, while other risks’ factors (Rain and SCFM) have less contribution to crash risk. The result matches with the condition described by the drone operators in Indonesia, as indicated in the interview.

The risk results obtained by the proposed methodology may be utilized by (1) designer of drone for delivering logistic in remote area to design a system specification of the drone; (2) the drone operator can determine whether to operate drone to deliver a package when a specific weather condition occurred. (3) logistic distributor can determine safest route for distributing logistic using drone in remote area. Moreover, the results obtained can be used as a quantitative risk assessment of drone operation in remote area for the government.

In the future, the BN model will be used to investigate and determine the recommended drone system specification for cargo delivery operation in the 3T Region of Indonesia based on its operational risk analysis. To obtain a complete data for BN Model is a challenging task because drone technology applications in Indonesia is just emerging. Therefore, to get a better result in the future, extensive operational data of drone flight must be recorded to identify the factors that may significantly contribute to drone operation failures.

However, as an alternative, simulation data can be used to complement real-world drone flight operation data. Using a high-fidelity model, an accurate aircraft response can be obtained using commercial off the shelf flight simulation software. Despite the accuracy of the flight simulation software itself, small attention has been given for the turbulence model. For this reason, the next target is to incorporate standard turbulence model in a flight simulation software to further investigate wind effect in drone operation.

Furthermore, to get a better result in the future, extensive operational data of drone flight must be recorded to identify the factors that may significantly contribute to drone operation failures. Thus, this analysis and information can be used for mitigating the problem and preparing a technically suitable drone system.