Risk Management for Defense Systems in a Complex, Dynamic Environment

: Identifying and assessing risk is one of the most important processes in managing complex systems and requires careful consideration. The need for an effective, efficient approach to risk management is considerably more important for defense projects based on systems of systems (SoS), because they are exposed to risk already in early stages of development. This paper uses advanced data science tools to present the complexity of the risk factors relevant to defense systems, and proposes a methodology for identifying, analyzing and monitoring the risks that they face. Findings from an in-depth analysis of 46 classified defense projects based on SoS shows a need to focus on three main risks faced by defense projects: uncertainty, the lack of clearly defined goals, and managing a system under constrained conditions. The paper also presents some recommendations for minimizing risk factors in SoS for defense projects.


Introduction
Risk management is one of the most important areas that must be considered when managing defense projects based on systems of systems (SoS). All systems have some level of inherent risk because of the uncertainty that accompanies any new endeavor. In defense industries, the riskier the system, the higher the payoff. Thus, risk is sometimes beneficial because it has the potential to increase profits.
SoS projects for defense develop sophisticated systems to meet challenges on the battlefield. Therefore, their successful management requires functioning in a dynamic, rapidly-changing reality, in which risk assessment and prioritization may present complex challenges.
The current paper presents an ongoing study examining risks faced by classified defense projects using SoS. The findings can help project managers and systems engineers of these and similar projects minimize delays and reduce risks.

Literature Review
The Project Management Institute includes risk management as a key process defined in the Project Management Body of Knowledge (PMBOK). Project Risk Management includes the following processes: conducting risk management planning, identification, analysis, response planning and risk control. The objectives of project risk management are to increase the likelihood and impact of positive events, and decrease the likelihood and impact of negative events in any given project [1].
The literature includes several suggestions for describing the process of risk management. For example, Fairley [2] presents seven steps: (1) Identify risk factors; (2) Assess risk probabilities and effects; (3) Develop strategies to mitigate identified risks; (4) Monitor risk factors; (5) Invoke a contingency plan; (6) Manage the crisis; (7) Recover from the crisis. Boehm [3] described a process with two main phases: risk assessment, which includes identification, analysis and prioritization, and risk control, which includes risk management planning, risk resolution and risk monitoring planning, tracking and corrective action. Similar to Deming's quality improvement cycle (Plan, Do, Check, Act), Kliem and Ludin [4] suggested a fourphase process (identification, analysis, control and reporting). According to ISO 31000 risk management creates and protects value [5].
Several popular risk management analysis techniques have been reported in the literature, including Monte Carlo Simulation [6], Analytical Hierarchy Process [7,8] and Fuzzy Set Theory [8,9]. There is much evidence in the literature that using risk management tools when managing a project creates value for its outcome and success [10][11][12]. On the other hand, some researchers did not find any effect [13] or found that the effect was negligible [14,15]. Moreover, many studies are dedicated to the application of project management in specific sectors, so the practices and techniques they present are not necessarily applicable to risk management of projects in other fields [16][17][18][19][20][21][22][23][24][25][26][27].
The identification of risk factors might be influenced by the sector and area of the project. For example, the key risk factors of public-private partnership (PPP) projects are divided into two categories, the first includes risk factors that have powerful, independent influences, such as delays in government approval, government credit, and imperfect legal and regulatory systems. The second category includes risk factors that are highly variable and easily influenced, such as completion risks, insufficient revenue in the market, and fee changes [28].
Ameyaw and Chan [29] mention others risks factor such as market/revenue risks, financial risks, relationship risks and social risks. According to Lessard [30], risk management requires systematic management of risks that are generated within each link in the chain and, more importantly, in the interfaces among links in order to limit disruptions and their propagation throughout the system. Effective management of risk, therefore, requires a systems thinking approach-understanding how systems influence one another within a whole.
According to Naaman [31], the risk management process has become an inseparable part of management procedures for defense projects, for which uncertainty management is one of the main challenges of ongoing project planning and management. Moreover, in response to dangerous events, such as plane crashes or take-off failures, safety requirements in the defense industry are strict, rigorous and demanding.  Figure 1 shows that, at the beginning of the project, the cost of changes is low; costs go up the more the project advances. At the same time, we can see that the effect of uncertainty and risk at the beginning of the project is higher; the more the project advances, the more these values decrease [1].
There are two types of uncertainty in defense projects: 1. Uncertainty that can be predicted in advance: It is possible to cope with this type of uncertainty by using an organized methodology, as presented in PMBOK [1]. For every stage of the project, there is an organized process that includes determining the project's leading players, and defining inputs and outputs. Risks that stem from uncertainty may be managed making a plan to minimize them. 2. Uncertainty that cannot be predicted in advance: In response to this possibility, buffers are defined in the schedule, the budget and Statement of Work (SOW), which provide leeway for dealing with unexpected changes.
According to ISO standards [5], there are certain limitations on projects managed under constraints such as timeframes for project completion, human resources, and activities that are dependent on the results of other activities. Wang [32] defined risk as a factor or action that might occur unexpectedly and, as a result, cause physical harm, damage assets or delay the timetable. Risk is measured according to the likelihood of occurrence; the technical, programming or managerial level; and the amount of potential damage that could result from the failure to prevent its occurrence.
Engineering projects are frequently characterized by extensive scope and budget; in many cases, they include manufacturing for a specific customer according to specific needs. These characteristics intensify the importance of the risk management process, because every unplanned event that occurs, which was not considered from the outset, could potentially have significant effects on project's success and compliance with requirements [33].
According to Naaman [31], the risk management process in defense projects includes five stages: 1. Identifying risks using brainstorming 2. Analyzing the meaning and level of each risk, including assessments of severity, probability and risk level 3. Risk factor analysis and defining responses, including a contingency plan 4. Risk presentation for authorization purposes 5. Monitoring and re-measuring the risk includes ongoing risk supervision The US Department of Defense [34] defined risk management as an interactive process, as shown in Figure 2. The PMBOK [1] presents six ways to cope with project risks, which are similar for many types of projects: 1. Risk Survey -the goal of a risk survey is to reach an agreement about the risk level and ways to cope with each risk factor 2. Preemptive Action -to be carried out in advance, usually at the earliest possible stage, with the goal of minimizing occurrence of the risk to the lowest possible level 3. Mitigation Action -an action that should be carried out immediately upon the occurrence of a risk, with the goal of minimizing the extent of the damage caused 4. Corrective Action -an action that must be carried out after the risk occurs, with the goal of returning the situation to its pre-risk state 5. Transferring the Risk -transferring responsibility for the risk and its treatment to another party 6. Accepting the Risk -taking a calculated risk, and deciding not to take any action Chris and Stephen [35] write that for every activity in a project, it is necessary to clearly define who is responsible, create a work schedule, and make sure to integrate them into the work plan. Risk must be managed throughout the entire project's lifespan in order to reduce the effects of the risks on meeting the project's targets at all stages, including its conclusion. In engineering projects, responsibility for the risk management process is usually shared by two individuals: the systems engineer and the project manager [36][37][38].
Kordova, Katz and Frank [39] studied the management processes shared by project managers and systems engineers in the defense industry, and provided recommendations for joint project management that leads to project success. Figure 3 shows the division of responsibility for risk management between the systems engineer and the project manager, and how their efforts mesh.

Figure 3. Overlaps between systems engineering and project management
According to Kordova, Katz and Frank [39], project risks include management-related risks (e.g., cost or organizational expenses), as well as technical/engineering risks (e.g., specifications, performance demands, and premature technology). During the project, there are joint discussions about risks, they are ranked, and a risk minimization plan is developed. Project managers usually integrate all risks, while systems engineers are generally responsible for identifying and managing only technical risks. However, technical risks often have managerial consequences, because risk minimization plans generally include the allotment of resources (schedule/budget) that are managed by the project manager.
The projects analyzed in the current paper are all defense projects that developed classified system of systems (SoS). SEBok [40] defines SoS as a set of systems or system elements that interact to provide a unique capability that none of the constituent systems could accomplish on its own. A similar definition was previously suggested by Maier [41], who defined SoS as a collection of task-oriented systems that pool their resources and capabilities to create a new, more complex system that offers additional functionality and performance beyond simply the sum of the constituent systems.

Methodology and Research Design
The research paradigm combines analytical, quantitative and qualitative methods, as presented in figure 4. The qualitative research started as an exploratory study. The analytical component included assessing both primary (testimony of engineers and project managers that were involved in the classified projects) and secondary (literature) sources. The quantitative component included data collection of 46 classified defense projects for developing SoS.

The exploratory stage
The qualitative research started as an exploratory study, consisting of 10 semi-structured interviews with professionals who participated in classified defense projects for developing SoS in the Air Force. The interviewees included project officers, flight-crews, pilots, project managers and systems engineers, who were asked about the projects they participated in and these projects' risk factors; the association between project budget and the extent of deviation from the planned schedule; as well as the connection between different risk factors and project scheduling delays. After recording and summarizing the interviews, a triangulation process was conducted, confirming that each finding presented in the report was mentioned by three or more interviewees, to ensure trustworthiness.

The survey
Based on results from the interviews, a survey was developed to examine the risk factors faced by SoS projects for defense purposes. Its goal was to determine the segmentation of risk factors for these projects, and examine the correlations between different risk factors and project schedule variations.
A pilot questionnaire was first distributed to 10 experts in project management, including senior systems engineers and a professor of industrial engineering, for their evaluation. Based on their responses and comments, a final version of the survey was created and validated. The data collected by the survey included the organization's risk management methodology, the most common risk factors in defense projects and the main characteristics of organizations that manage to avoid occurrence of risks.

Findings of the exploratory study(interviews)
The risk factors faced by defense projects, as reported by the interviewees and confirmed by the triangulation process, are: 1. The quality and quantity of human resources are critical factors in the development process; an insufficient workforce is a significant risk in the defense industry. 2. The dynamics of defense industries often makes it necessary to shorten the Time-To-Market, even though development processes are usually very time consuming. 3. Too many stakeholders are involved, influence one another, in addition to many parties from external companies working on the project and interdependent on one another. 4. Additions to the Statement of Work (SoW) and/or changes to the project's initial design. 5. The tendency to change roles/jobs once every 2-3 years in the military may create a sense of partial commitment, and an "until the end of my term" attitude towards the project. This may also cause project managers to commit to challenging SoWs and schedules. 6. The bigger the project's budget, the more the potential risks. 7. The risks that cause schedule delays can be divided into two types: insufficient planning/management and resources constraints (mainly workforce and budget-related).

Findings from the Survey
The first step in analyzing the survey was to identify the common risk factors faced by the defense projects (N = 46), which are: 1. Overly-optimistic scheduling assessment 2. Insufficient human resources 3. Changes in the original specifications 4. Lack of other (non-human) resources 5. Overlap between different project processes 6. Too many stakeholders influencing the project 7. Lack of project participants' previous experience 8. R&D required in a new field/area 9. Failure to maintain risk management processes 10. Cultural differences among project members 11.   According to the tree-based classification model, there is a significant difference in the influence of cluster 4 and cluster 5 on item 22 (overly-optimistic schedule assessement). The average score of overly-optimistic schedule assessement for cluster 5 is 2.5 (out of 5), while the average score of overly-optimistic schedule assessement for cluster 4 is only 1.38 (out of 5). This means that for higher clusters in item 22, schedule variation is higher, which reinforces the impact of overly-optimistic schedule assessment on risk occurrence in general and on schedule variation in particular.
The model shows a significant difference in the influence of cluster 5 and cluster 3 on item 20 (project stakeholders involvement). The average score of project stakeholders' involvement for cluster 5 is 2.78 (out of 5), while the average score of project stakeholders involvement for cluster 3 is only 1.44 (out of 5). This means that for higher clusters in item 20, schedule variation is higher, which reinforces the impact of project stakeholders' involvement on risk occurrence in general and on schedule variation in particular.
The model also shows a significant difference in the influence of cluster 5 and cluster 4 on item 19 (insuffcient human resources). The average score of insuffcient human resources for cluster 5 is 1.59 (out of 5), while the average score of insuffcient human resources for cluster 4 is only 1.1 (out of 5). This means that for higher clusters in item 19, schedule variation is higher, which reinforces the impact of insuffcient human resources on risk occurrence in general and on schedule variation in particular.
The third step in analyzing the servey was to explore the relationships between several explanatory variables and one or more response variables by using Response Surface Methodology (RSM). We used RSM in the current study primarily in oder to obtain a sequence of designed experiments that could yield an optimal response. One example of how we used RSM is presented in Figures 8 and 9.  Figure 9, a low level of project stakeholders involvement (item 20) and a high level of human resources shortage (item 19) characterize a high level of schedule variation in a project (item 31). This finding supports the impact of a insuffcient human resources, and the lower the impact of project stakeholders involvement on risk occurrence.
Step 4 of the survey analysis used advanced data science analysis tools such as Bayesian SEM Path analysis and IBM Watson Analytics analysis. The preliminary results of the data science analysis are shown in Figures 10-13. Figures 10 and 11 rank the most common risk factors in defense projects, which are: 11. Other (project manager lacks previous experience, gap in knowledge management etc.) Figure 10 presents the percentage of each risk factor; Figure 11 shows the graphical segmentation of the risk factors. The survey findings show that the most common causes of risk in defense projects are insufficient resources, changes in the initial project design, coupling of processes, overly-optimistic scheduling, and too many involved parties.   mean that defense systems must be developed rapidly, using daring innovation and technological ingenuity. Unlike others projects, the complex risk management of defense projects is rarely mentioned in the literature. The current study proposes a preliminary attempt to identify and analyze the risks in these systems.
According to the preliminary results, a strong correlation was found between the findings of the exploratory study and the survey results. These correlations are expressed in the following relationships: 1. The exploratory study found that the quality and quantity of human resources is a critical factor in development processes, but were sometimes found to be insufficient. This risk was mentioned as a significant factor in almost every interview. The survey results show that human resources was defined as the primary risk factor, ahead of all others. 2. Another risk presented in the exploratory study, reported mainly by those respondents who had a broader picture (managerial level), is the continuously changing battlefield that forces all involved parties to reduce TTM, although development processes take a substantial amount of time. This need was also mentioned in Naaman [31]. Projects' ongoing need to be remain relevant despite rapidly-changing battlefield conditions means that changes in the initial project design is a risk factor, as the survey indeed found. 3. In many defense projects, it is necessary to change the project's pre-defined goals because changing security requirements. This results in changes in the end product, which in turn delays other processes. This is why the intelligence-mission-technology analysis conducted at the beginning of the project is so critical. This analysis facilitates the prediction of possible future changes in project goals, accordingly makes it possible to introduce project changes, with a minimal influence on schedule. Thus, the robustness principle becomes stronger as a defense project advances. The ability of a product to adapt itself to changing conditions, requirements, and infrastructures is especially significant in the defense industry. 4. Defense projects suffer from too many stakeholders who mutually influence each other. This finding is in line with previous findings [5,[28][29] about the need for many partners when managing projects under constraints. Sometimes, entities from different organizations and companies are involved in a project and are mutually dependent on one another. An example of this dependence is the hiring of sub-contractors, who are required to provide services or products to the primary contractor, which are a critical part of the defense project. The interview findings show that sub-contractors have a meaningful influence on projects on several levels, from product quality to scheduling issues. This subject came up in interviews with project managers who are required to cope with complex challenges in their work with sub-contractors. As a result of defense projects' sensitivity, project managers strive for independence, as they prefer to depend on their own skills, rather than on external sourcing, which may reduce their flexibility. Conversely, as soon as external sourcing has been decided upon, the project manager is free to deal with other parts of the project.
5. Adding to and changing the Statement of Work (SoW) for the initial project design as a consequence of needing to reduce TTM, or in light of changes in the mission's demands, it is necessary to revise, adapt, and change the initial project design. The survey findings show that in some cases, these changes may also be caused by managerial considerations. On the managerial level, a lack of thorough planning when preparing SoWs, and considering potential risks may lead to future changes that impact scheduling issues. Another aspect that leads to scheduling delays, and which was expressed in the interviews was the high frequency of role switching within the defense industry, especially when working on classified SoS. Role switching can have numerous advantages, but it has two significant disadvantages: a. There is an initial learning process, during which the individual must learn to solve problems based on existing and new knowledge. At this stage, the individual may pay less attention to the project. b. We can assume that a project manager or officer who knows, from the beginning, that he will be managing a project from beginning to end will be more committed than one who will likely be transferred before its completion. Another related, common occurrence is the introduction of new professionals to an existing project at advanced stages. Naturally, each professional wants to make his mark on the project, even when he enters a preexisting, stable project at a later stage. This need may generate a desire to make changes or additions that will be remembered as being initiated by specific people, but could potentially influence the project schedule.

Conclusions and Recommendations
The findings of the current study show the need to focus on three main challenges/risks faced by defense projects based on systems of systems (SoS): The lack of clearly defined goals  Managing a project under constraint conditions We could link each one of the risk factors presented in the study to one of these three characteristics. For example, insufficient human and general resources can be linked to managing projects under resource constraint conditions. The subject of resources could also be linked to a lack of clearly defined goals, because goals often determine the project's requirements, leaving uncertainties regarding acquisitions and accessibility. Each one of the risk factors considered might be minimized by high-quality advanced planning, to which all of the necessary resources are allocated.
The risk management process includes several advanced stages of analysis and thinking [32]. The more thoroughly the initial risk analysis process is carried out, the easier it will be to predict the risks that will affect/be affected by the budget, and provide responses to at least some cases. In addition, based on the demand to reduce development times [31], it is reasonable to assume that defense projects' SoWs will also decrease, especially in light of continuously changing battlefieldrelated needs, that demand rapid response times and shorter development times. Thus, the scope of defense project budgets will most likely be organized differently in the future.

Main conclusions and recommendations
1. Organizational adaptations will make it possible to minimize the common risk factors found in the study: the study found that changing the project's initial design specifications and insufficient human resources are the two main risk factors.
Defense projects have unique characteristics. Therefore must be a close connection between the end-user (e.g., Air Force operator), the department responsible for making the request (e.g. Development) and the accountable unit that carries out the request (e.g., Procurement Department). Generally, the party making the request is also the one who defines the technical specifications. There is a real need for organizational change that permits transferring responsibility for SoW specifications to the party carrying out the work. This could reduce the number of specification changes during a project. We further found that the experience accrued by officials in the departments implement and/or making requests can also help reduce some of the above risks. One critical remark is that these organizational changes are necessary to cope with rapid developmental changes and mitigate project risks. We also found that insufficient human resources is a significant risk factor. Assuming that there are sufficient resources, outsourcing of the SoW has the potential to reduce these risks. However, these processes sometimes present other risks that are preferable to avoid.
To minimize potential risks, an organization should develop tools that allow for a proper, intelligent analysis of outsourcing processes based on past studies/data and valid professional knowledge.
2. Developing generic infrastructure, which allows for the integration of new systems, would contribute significantly in two ways: a. Reduced R&D required for future integration, thereby reducing potential R&D-related risks b. Decreased development times for new abilities 3. Process automation, with an emphasis on automated control processes. In order to reduce the risk of insufficient resources, it is necessary to automate processes as much as is possible. For example, transitioning to automated checking of new systems instead of manual checking has many advantages, including increased effectiveness, the ability to run more tests in less time, the ability to work 24/7, and a reduced need for and dependence on human resources, which reduces risk related to workforce availability as well as costs. 4. Changes in performance specifications at advanced stages is one of the main reasons for project delays. In-depth preparation of technical and performance specifications at the beginning of the project may mitigate this risk. A project's progress generates a great deal of stress for most of the involved parties, which may consequently create inadequate thoroughness in certain areas, including specification processes. We recommend defining principles to regulate orderly work procedures, which includes anchoring the time windows of all of the project partners.
A process of risk management in projects is a rational chain of practices by decision-agents in order to ensure that project implementation complies with certain conditions. The decision-makers need to identify, analyze and evaluate the risks in all stages of the project's life cycle, and use their organizational structure and administrative practices to respond to risks in way that benefits the project [23,26,[42][43].
Proper risk management methods are crucial to implementing in defense projects involving SoS. The more managerial echelons focus on risk management, the more the attention will paid to subject at work levels, resulting in fewer instances of risks. Moving forward on a defense project involving SoS without a proactive focus on risk management is likely to lead to more problems arising from unmanaged risks.

Study limitations and recommendations for future research
This paper presents preliminary results of a data science analysis of a concerning classified defense projects involving SoS. We expect to present the final results by January 2021. Further, we recommend that future research expand the study to other defense industries in additional countries, with different clients and suppliers.