The Impact of Artificial Intelligence on Future Aviation Safety Culture

1. Introduction

1.1. Overview of Paper

Currently, aviation is seen as a very safe mode of transport, and this is in part due to its safety culture. This paper therefore poses the question of how future Artificial Intelligence (AI) may impact aviation safety culture. Although AI is already in use in numerous aviation sectors via Machine Learning, this paper focuses on the more advanced AI systems likely to appear in the next decade, including Intelligent Assistants that could work semi-autonomously or even autonomously and collaboratively with human crews and teams.

The paper begins by briefly outlining safety culture today in aviation, including how it is evaluated. The fast-developing area of AI itself is then outlined, in particular focusing on different ‘levels’ of AI autonomy and the concept of Human-AI Teaming. This wide-ranging exploration of AI is necessary to envision how human crews and IAs might work together in a range of future AI settings (e.g., cockpit, air traffic Tower and Ops Room; airports). One of the principal aviation safety culture methods is then applied in detail to future Human AI Teaming scenarios to estimate whether there could be a net safety culture detriment or benefit via Human AI Teaming. The paper concludes by noting the most serious threats to safety culture posed by AI, and how to safeguard against them, as well as suggesting ways forward to harness the potential safety culture benefits from Human AI Teaming.

1.2. Safety culture—an essential ingredient of aviation safety

In European commercial civil aviation today, safety in terms of accident rates is generally seen as very strong, with zero fatal accidents since 2016 [1], although there are still fatal accidents in general aviation. This level of safety is in part attributed to the level of safety culture in aviation, hard won due to various accidents during the early years (1950s onwards) of commercial aviation, and despite accident ‘spikes’, e.g., following the introduction of ‘glass cockpits’ in the 1980′s [2]. Since commercial aviation accidents in flight are often fatal, there is both strong public awareness of air crashes (they usually make the headlines) and appropriately demanding regulation on safety across the aviation industry spectrum [3]. Hence, safety culture has generally been a hallmark of commercial aviation in aircraft manufacturing, air traffic control, airport operations and in the airlines themselves.

Safety culture as an approach did not emerge out of the blue [4]; prior to safety culture, there was already research on safety climate. These two terms are often used interchangeably, but there are key differences. Most commonly, safety climate is considered to be a temporary snapshot of the current safety culture made up of perceptions and feelings at the time (and is likened to mood), whereas safety culture is more stable (and compared to personality), and related to group activities and organisational histories [5]. Safety culture is more enduring, and thus harder to change; it takes time. Whereas safety climate focuses on managerial prioritisation of safety [6] and has its roots in social psychology, safety culture has its roots in anthropology, and is more concerned with the safety-related values and practices that permeate the organisation [7]. An early definition of safety culture, coming from the nuclear power industry, is the following [8]:

“The safety culture of an organization is the product of individual and group values, attitudes perceptions, competencies, and patterns of behavior that determine the commitment to, and the status and proficiency of, an organization’s health and safetymanagement”

A second, frequently used definition of safety culture, and more generally organisational culture, is as follows [4]:

‘’Shared values (what is important) and beliefs (how things work) that interact with an organisation’s structures and control systems to produce behavioural norms (the way we do things around here).’

The origins of Safety Culture are usually traced back to the Chernobyl nuclear power plant accident in 1986 [9]. Just as the Three Mile Island nuclear power plant accident in 1979 demonstrated unequivocally the importance of Human Factors in the design of human-machine interfaces in high systems, Chernobyl showed that the prevailing operational culture could catastrophically trump established safety procedures and processes. A spate of high-profile accidents from different domains (see Figure 1), including space, oil and gas, and rail—all seen as ‘safety culture accidents’—only served to emphasise the enduring importance of this newly identified organisational trait. Certain high profile public enquiries into key accidents such as the Piper Alpha disaster [10] and Clapham Junction rail crash [11], as well as key safety thought leaders at the time [12,13], and a number of accidents at least partly attributed to safety culture ever since, have ensured that safety culture has endured as a critical and non-negotiable attribute for any high risk organisation.

Figure 1 highlights the fact that safety culture wasn’t initially seen as being of too much concern for aviation. This was despite the Kegworth air crash in 1989 [14], which had certain safety culture aspects. The thinking at the time was that the strong training and (Human Factors-based) design in the cockpit and air traffic operations rooms, as well as Safety Management Systems (SMS) and Standard Operating Procedures (SOPs), were sufficient. In Europe, this notion was shattered with the mid-air collision over Lake Constance in Überlingen in 2002 [15], following shortly after the Milan Linate runway collision a year earlier [16]. As Chernobyl did for nuclear, these accidents triggered a rethinking that SMS and SOPs were not enough, and that safety culture was crucial. SMS was seen as safety competence, comprising the key safety processes, whereas safety culture was the motivation to energise such processes for safe outcomes.

1.3. The emergence of a safety culture evaluation methodology in aviation

Since the two accidents (Uberlingen and Milan) were primarily related to air traffic management (ATM), the safety culture assurance method development was carried out in that sector of the industry, via a combined effort of EUROCONTROL and Aberdeen and LSE universities [17,18], and was able to build upon more than a decade of experience of safety culture evaluation in other industry sectors such as nuclear power and oil and gas. A safety culture questionnaire survey approach was therefore developed, its results informed by workshops with aviation staff. After an initial pilot test with four European air traffic providers, the approach was rolled out in 2005 and has gradually been applied across Europe. To date, almost all European Air Navigation Service Providers (ANSPs) have engaged in at least one of several independent safety culture evaluations of their organisation. A total of 33 European member States have applied the method (see Figure 2), most more than once [19]. The EUROCONTROL Safety Culture Questionnaire has been scientifically validated [7,20] and positively reviewed by the European ANSPs [21]. It has also had a more modest application in airlines [22,23] and airports [24,25].

1.4. Measuring Safety Culture

The EUROCONTROL Safety Culture questionnaire1 contains 48 questions linked to eight safety culture ‘dimensions’:

• Management Commitment to Safety
• Collaboration & Involvement
• Just Culture & Reporting
• Communication & Learning
• Colleague Commitment to Safety
• Risk Handling
• Staff and Equipment
• Procedures & Training

Each of these dimensions is supported via a set of carefully worded questions, tailored to different segments of the aviation industry (ANSP, airline, airport, and airframe manufacturer), e.g., ‘My colleagues are committed to safety,’ and ‘If I see unsafe behaviour by one of my colleagues, I would talk to them about it.’ The answers to these questions build a picture of the safety culture of an organisation, including an understanding of how it may differ in an organisation’s various sub-cultures.

An example of the output from such a survey is shown in Figure 3 for three typical safety culture questionnaire items. In this example, the first statement is clearly reflecting ‘positive’ safety culture with a few who are neutral about the issue, and a small percentage of dissenters. The second item has a large ‘neutral’ component, which means either that the respondents are not sure, don’t see how it applies to their work environment, or prefer not to say (although the surveys are always anonymous and confidential, some participants are cautious). The third item has a significant negative component that would be investigated further via confidential workshops with participants (e.g., flight and cabin crew, controllers and engineers, airport workers, management and support staff), in order to find out what has been happening, and how to establish a better safety culture (or in this specific case, a better Just Culture). Such workshops are often very useful in seeing ‘beneath’ the questionnaire results, and are helpful in determining ways forward.

It is useful to provide a high-level overview when carrying out safety culture surveys, and this is achieved by summarising results at the ‘dimension’ level using a spider-chart (because it resembles a spiderweb, e.g., as shown in Figure 42. The higher the values (the further out from the centre), the better the safety culture.

Here it is easier to see which dimensions are the ones to worry about. In this case Risk Handling, Collaboration and Fatigue are the three dimensions that need more urgent attention. Such diagrams give a ‘helicopter view’ of the survey results, and are often appreciated by senior management as showing the safety culture survey ‘headlines’.

Safety culture surveys offer the safety culture equivalent of a detailed health check, showing where the organisation is healthy and where it needs attention. Such reports include recommendations on how to improve, often the best ideas arising from the confidential workshops with participants, and many organisations have used such surveys to improve their safety culture [26]. CEOs often find the results of such surveys useful, as they form a bridge between them and the front-line employees, so they can see in a relatively unfiltered way what people at the sharp end are concerned about. Indeed, the concept of safety culture necessarily includes senior and middle management, and considers the critical importance of management and designers regarding safety, based on their collective values, beliefs and behaviour, as well as the safe behaviour and attitudes of operational personnel, engineering and other support staff at ‘the sharp end’. As the wide-ranging investigation into the two Boeing 737 Max accidents has shown [27,28], even with the best engineering and a strong track record in safety performance, a compromised safety culture can lead to disaster. Senior managers (CEOs, VPs, Directors) make executive decisions that ripple down through their organisations and can dramatically affect safety culture—they also ‘set the tone’ for the safety culture of their organisation.

1.5. Safety Culture & Future AI—an unexplored landscape

For the purposes of this paper, it is the detailed safety culture questionnaire items, rather than the dimensions per se that are likely to highlight where AI may impact safety culture. These are returned to following the next section, which expores the development of AI and likely future Human-AI Teaming scenarios in a range of aviation contexts. Given the above-mentioned importance of senior management, this review of AI necessarily includes aspects of AI regulation and governance.

In order to consider how AI might impact safety culture in aviation, it is necessary to have a vision of how AI might look in the cockpit, the air traffic operations room or the airport control centre in the coming decade. The next section accordingly builds a preliminary picture of future aviation AI by considering the following points:

i.

The origins of AI

ii.

AI today

iii.

Generative AI

iv.

Narrow AI

v.

Visions of Future Aviation Human-AI Systems

vi.

Trustworthy AI

vii.

Accountable AI

viii.

AI and Just Culture

ix.

Ethical AI

x.

Maintaining Human Agency for Safety

xi.

AI Anthropomorphism and Emotional AI

xii.

AI and Safety Governance

The consideration of these issues and perspectives helps to narrow the expansive and ever-growing field of AI research to enable a realistic focus on safety culture impacts, and in effect serves to ‘ground’ the analysis in Section 3. It also lays the foundation for determining who should be the ‘risk owners’ of each of the issues arising from the analysis, whether they are staff at the sharp end, middle managers or senior management, as discussed in Section 4.

2. The Developing Artificial Intelligence Landscape in Aviation

There is currently much ‘hype’ about AI, and it can be difficult to discern fact from fiction, or operational reality from the ‘promise’ of AI. This section therefore starts by making the key distinction between Generative AI and Narrow AI, the latter being the AI most likely to be implemented in aviation in the mid-term (now until at least 2030). Since the level of AI of interest in this paper is advanced but foreseeable for the 2030+ timeframe, there are currently many design ‘degrees of freedom’ being discussed, often at supra-national levels, for example with respect to ethics or whether Intelligent Assistants (IAs) should mimic emotions, etc., as well as the current preliminary regulatory perspective on AI and AI autonomy for aviation. This section then homes in on key aspects of Human-AI Teaming—including basic principles for trustworthy, accountable and ethical AI—and their potential impact on, e.g., safety citizenship. Just Culture and AI is also reviewed, as potential issues here are already identified by the aviation community, before considering the higher level issues of governance and organisational (safety) leadership.

2.1. The Origins of Artificial Intelligence

The simple idea behind Artificial Intelligence is to go beyond the limitations and capabilities of human thinking. An example of early AI is the ‘Bombe’ Machine [29], used to break German ‘Enigma’ codes used in the second world war. Such codes were seen as unbreakable by humans, and so the ‘Bombe’ Machine did indeed surpass our capabilities. But such machines were not seen as ‘thinking’; rather they were running endless calculations—running the numbers, as it is known—and hence were ‘computing’ rather than thinking. Turing himself was fascinated by the idea of a machine that could one day think, and whose thinking would be indiscernible from that of a human, leading to his famous challenge to the scientific and engineering communities to develop such a machine, to be tested by ‘the imitation game’ [30]. He predicted that such machines would exist by the turn of the century. What is interesting is that many of the questions he posed about such artificial intelligence back in 1950 are the same questions we find ourselves posing today.

In the 1980s there was another surge in AI interest via (rule-based) expert systems, which ultimately failed to deliver operationally useful tools, in part due to their inability to account for the experience-based and highly contextual ‘tacit knowledge’ that human operators possess, which usually goes far beyond what is written in procedures. The failure of expert systems arguably led to the so-called ‘AI winter’, which ended recently as computing power increased dramatically and Machine Learning became finally possible [31]. This has resulted in a host of early AI prototypes, products and services being introduced into European aviation, from automatic speech recognition and passenger support, to optimising safe and expeditious air traffic flow both in normal and hazardous weather conditions [32].

2.2. AI Today

A useful definition of AI is as follows [33]:

“…the broad suite of technologies that can match or surpass human capabilities, particularly those involving cognition.”

The general aim of Artificial Intelligence (AI), therefore, is currently seen as supporting human intelligence—and society in general—by using data science techniques to analyse complex datasets to find new patterns or solutions to problems that are beyond our own intellectual capabilities. In aviation, AI could be used to ‘optimise’ aviation systems, for example to help minimise fuel usage in air traffic route networks in order to reduce aviation’s carbon footprint, or to rapidly assist flight crew in finding a solution during an emergency (treading in the footsteps of the former expert systems approach, but with more speed and accuracy). Machine Learning can generally analyse very large, complex and heterogeneous datasets in ways the human mind finds difficult or impossible (e.g., via n-dimensional analysis). So far, such AI tools, though yielding impressive results, do not constitute thinking; they are still computing machines that are ‘running the numbers’, albeit in very complex and sometimes unfathomable ways. Such AI tools can be seen as ‘just more automation’ [34], and their impact on safety culture might therefore be expected to be minimal. However, this understanding of AI—as effectively more powerful automation tool support—shifted dramatically with the release of ChatGPT in 2022 [35], heralding the advent of Generative AI.

2.3. Generative AI

ChatGPT is a Large Language Model (LLM), using the entire internet as its database, which sits behind a ‘chatbot’. This chatbot enables a human user to have a ‘conversation’ (via the keyboard) on a vast range of issues, and is reminiscent of Turing’s ‘imitation game’ challenge to develop thinking machines. Unlike systems before it, ChatGPT is ‘generative’ in that it can answer any question. This is in large part because it has used an approach called supervised learning, wherein humans work with the AI to refine the types of answers it gives, essentially training the AI to give better answers. Whether the answer makes sense is up to the user, and sometimes it produces answers that are inaccurate, wrong, or so bizarre they are referred to as hallucinations. It works best when being asked to draw together factual information that is already residing in the web, although much of the internet is not itself fact-checked. This dataset, while truly vast, limits LLMs such as ChatGPT for strict operational usage, e.g., how to land an aircraft in a particular configuration and weather pattern at a specific airport, because such information is not necessarily on the internet. But it has piqued the interest and imagination of millions, as it can write essays for students, compose music, generate business ideas, translate text and produce summaries, etc, and so the notion of generative AI is very much in vogue today. And although a LLM is probably not the solution for operational AI in aviation, the fact that there can be generative and realistic dialogue between human and AI may pave the way one day for ‘Intelligent Assistants’ in the cockpit or air traffic tower or ops room.

One important note on models like ChatGPT is that sometimes people think they are interacting with an intelligent entity. They are not. ChatGPT, despite its sometimes impressive outputs, is still ‘running the numbers.’ When it writes a sonnet, it neither thinks ‘I am writing a sonnet’, or ‘I have written a sonnet’, nor reflects on what it has done with any feeling whatsoever such as pride or disappointment—and both these aspects of reflection and feeling have been hallmarked as requirements of a true thinking machine [30,31].

2.4. Narrow AI

Generally, cognition categories found in AI are typically: learning, perception, reasoning, communication and knowledge representation. Common AI applications include expert systems, machine learning, robotics, natural language processing, machine vision and speech recognition [36]. However, the same authors note that getting AI applications beyond the end of research and into operational use suffers from the ‘valley of death’ phenomenon (i.e., many good ideas and prototypes never see industrial usage). This can be for a range of reasons, but principally three stand out:

• a lack of data (most AI systems have vast data appetites—this can also be seen as a scalability issue when moving from research to wider industry)—and even though aviation has a lot of data, much of it is not shared for commercial reasons, and using ‘proxy’ or synthetically generated data runs the risk of diluting and distorting real operational experience;
• business leaders lack an even basic understanding of the data science and the technological skills necessary to sustain operational applications of AI; and
• a failure to develop the ‘social capital’ required to foster such a change, such that users reject the AI tool’s implementation (for example, because it threatens job losses).

Rather than generative AI, what aviation at least initially requires is what is known as Narrow AI [37]. Narrow AI can solve specific problems in a domain but cannot generalize as broadly as humans can. Such systems (sometimes called idiot savants) can be superhuman at some tasks, and subhuman at others. The advantage of Narrow AI is that it can be focused on a specific domain or even sub-domain for which there is sufficient data for the AI to work, e.g., tens of thousands of aircraft approaches by various aircraft to a particular airport runway. An AI tool can then answer specific questions or find solutions to problems, or simply show how to optimize system performance based on a limited set of parameters for which there are plentiful data. In a field such as aviation, Narrow AI is likely to be more fruitful in the short to medium term (i.e., for the next decade).

2.5. Visions of Future Human-AI Teaming Concepts in Aviation

At this juncture, it is useful to consider contemporary visions of future AI concepts, some of which go beyond today’s Machine Learning tools, leading to humans collaborating and negtoiating with advanced AI systems. The European Union Aviation Safety Authority (EASA—the principal European aviation regulator) has set out a vision of AI and its potential impacts upon aviation operations and practices. EASA’s recent guidance on Human-AI Teaming (HAT) [38] suggests six categories of future Human-AI partnerships:

1A—Machine learning support (already existing today)

1B—Cognitive assistant (equivalent to advanced automation support)

2A—Cooperative agent, able to complete tasks as demanded by the operator

2B—Collaborative agent—an autonomous agent that works with human colleagues, but which can take initiative and execute tasks, as well as being capable of negotiating with its human counterparts

3A—AI executive agent—the AI is basically running the show, but there is human oversight, and the human can intervene (sometimes called management-by-exception)

3B—the AI is running everything, and the human cannot intervene.

In order to help make some of these categories more concrete, it is useful to consider the EU-funded HAIKU (Human AI Knowledge and Understanding for Aviation Sagety) Project [39,40,41], which utilises six aviation Human-AI Teaming safety-related use cases, outlined below.

• UC1—a cockpit AI to help a single pilot recover from a sudden event that may induce ‘startle response’, and directs the pilot in terms of which instruments to focus on to resolve the emergency situation. This cognitive assistant is 1B in EASA’s categorisation, and the pilot remains in charge throughout.
• UC2—a cockpit AI to help flight crew re-route an aircraft to a new airport destination due to deteriorating weather or airport closure, for example, taking into account a large number of factors (e.g., category of aircraft and runway length; remaining fuel available and distance to airport; connections possible for passenger given their ultimate destinations; etc.). The flight crew remain in charge, but communicate/negotiate with the AI to derive the optimal solution. This is 2A but could be 2B.
• UC3—an AI that monitors and coordinates urban air traffic (drones and sky-taxis). The AI is an executive agent with a human overseer, and is actually handling most of the traffic, the human intervening only when necessary. This is category 3A.
• UC4—a digital assistant for remote tower operations, to alleviate the tower controller’s workload by carrying out repetitive tasks. The human monitors the situation and will intervene if there is a deviation from normal (e.g., a go-around situation, or an aircraft that fails to vacate the runway). This is therefore category 2A.
• UC5—a digital assistant to help airport safety staff deal with difficult incident patterns that are hard to eradicate, using data science technques to analyse large, heterogeneous datasets. At the moment this is a retrospective analysis approach, though if effective it could be made to operate in real-time, warning of impending incident occurrence or ‘hotspots’. This is currently 1A/1B, but could develop to become 2A.
• UC6—a chatbot for passengers and airport staff to warn them in case of an outbreak of an airborne pathogen (such as COVID), telling passengers where to go in the airport to avoid contact with the pathogen. This is 1B.

Other advanced AI concepts include [42] digital assistants to help air traffic control provide more efficient and environmentally friendly (‘greener’) routes, advanced warning in the cockpit of impending flight instability, and digital assistance for evidence-based training to enhance performance during adverse events.

In a recent debate on the issue of Human-AI Teaming [43], a critical threshold which could challenge the human’s ‘agency’ for safety appeared to be category 2B—where humans and AI collaborate, and each can act to a certain extent independently—since this is different from what we have today, and could impact safety culture if safety was seen as the province of the AI, rather than the human. There are currently no AI systems in aviation today that autonomously share tasks with humans, can negotiate, make trade-offs, change priorities, and initiate and execute tasks under their own initiative. Category 3A could also affect safety culture, as the human may be too far ‘outside the loop’ to intervene effectively in time. However, ‘lesser’ categories including 1B and 2A, could impact on safety culture, even positively, as they could augment the degree of control the human has over safety. Rather than degrading or eroding safety, AI could therefore possibly enhance safety, offering new ‘safety affordances’. This important prospect is returned to in Section 4.

2.6. The Need for Trustworthy AI in Safety-Critical Systems

AI tools will not be used if they are not trusted. A recent model of Trustworthy AI [44] comprises 7 technical requirements, built on 3 pillars throughout the entire system life cycle. The three essential pillars are that the AI and its operation are lawful, ethical, and robust. Since law often trails behind innovation, driven by legal cases associated with already-implemented systems, the onus of developing sound and ‘humane’ policy for AI development and usage will probably fall to requirements associated with ethics and robustness. The seven technical requirements proposed are as follows [44]:

• Human agency and oversight
• Robustness and Safety
• Privacy and Data Governance
• Transparency
• Diversity
• Non-discrimination and fairness
• Societal and environmental wellbeing
• Accountability

Of these technical requirements, Accountability and Agency have the most direct links to safety culture.

2.7. Accountability, Certification, and the Double-Bind

Accountability is directly related to safety culture, and Just Culture in particular. Consider the pilot who ignores advice from an AI in favour of their own judgement, and then has an accident, but also the case wherein the pilot follows an AI’s advice which turns out to be unsafe, also resulting in an incident or accident. In both cases it will be easy to blame the human user rather than the AI, yet this could be both unfair and unjust (such a situation is called a ‘double-bind’ in psychology). Accountability (and justice) would require adequate means of redress to discern whether the AI ‘made a mistake’. As with autonomous car accidents, the question becomes one of whether there is transparency in terms of the equivalent of the AI’s algorithms and calculations made, its data—both used and unused—and of trade-offs if any were made between different priorities, including safety. Such data forensics may prove inconclusive due to the innate complexity and opacity of how advanced AIs work.

There may be a temptation, following an accident involving a human and AI working together, for the AI developer to try to claim that ‘the human remains in charge’. But if the AI is partly taking control or heavily influencing the user, then this is a disingenuous argument. In self-driving cars the driver is meant to take over in case of aberrant behaviour, but this does not appear realistic in situations where things happen suddenly and evolve quickly, as may also occur in an aircraft or in an air traffic scenario. If an AI tool is useful and meant to help aviation professionals, they will become to an extent reliant on it, and may reduce their own situation awareness accordingly, not to mention losing certain skill fluency over time, if not entire skill sets. In the aftermath of an aviation accident, an AI manufacturer may well say that the human should have seen what was happening and taken command. What lines of redress will the aviation professional have in such a circumstance?

There may be an attempt to ‘certify’ the AI in the cockpit or air traffic Ops Centre or tower, such that once it is certified it is seen as ‘fit for purpose’, meaning that if anything goes wrong, the invisible finger of judgment will swing towards the human. Since much of aviation already has a certification ‘mindset’, this seems reasonably likely. But as noted elsewhere [37], “Certification… cannot replace responsibility.” This means that redress is not simply a matter of putting disclaimers here and there. This relates to the afore-mentioned ‘legal pillar’, which is not yet written into law.

The degree of safety effort required to certify an AI tool will likely depend on its degree of autonomy, e.g., a more autonomous AI system, which could for example initiate and execute tasks on its own, would have a higher safety certification requirement than a Machine Learning system simply advising a controller on weather pattern formation, since in the latter the human is more involved and in command. This means that AI developers may be inclined to classify their tools towards the lower end of the classification scheme. A corollary to this is that any such classification scheme linked to certification requirements must be crystal clear so that it cannot be misapplied. Such regulatory approaches should be tested via ‘regulatory sandboxes‘ [44] to see how they would work in practical settings. In relation to this, but in the context of future General AI systems, ill-advised implementation of legal exemptions to absolve general AI developers of liability should be avoided, as such exemptions could unfairly shift the responsibility from large corporations to smaller actors, users and communities lacking the necessary resources, access and capabilities to effectively address and alleviate all risks [37]. This principle is already being considered in the developing European Act on AI, discussed at the end of this section.

2.8. AI and Just Culture in Aviation

Just culture is strongly linked to Accountability, and is seen as a cornerstone of safety culture in aviation, since it protects safety reporting and therefore enables safety learning. Just culture is defined as follows [45]:

“Just culture means a culture in which front-line operators or other persons are not punished for actions, omissions or decisions taken by them that are commensurate with their experience and training, but in which gross negligence, wilful violations and destructive acts are not tolerated.”

This definition emphasises that actions, omissions, or decisions taken by aviation professionals should be commensurate with their experience and training. This raises a key question: what, if any, formal training on AI/ML and its state-of-the-art algorithms, such as Neural Networks, and their limitations, should aviation professionals receive? A vast range of failure modes exist for ML systems [46], which does not include potential failure modes for future AI systems. Aviation professionals cannot be expected to become data scientists. And how should incident and accident investigators proceed? Will future investigations require data science expertise to mine the data, algorithms, and inputs which lead to a particular AI suggestion, whether right or wrong?

Considering such concerns, in the context of air traffic controllers at least, it has been argued [45] that:

“the burden of responsibility gravitates towards the organisation to provide sufficient and appropriate training to air traffic controllers. If they are not well trained it will be hard to blame them for actions, omissions or decisions arising from AI/ML situations...”

It can also be noted that the existing definition of Just Culture is very human in its language, as it talks of gross negligence, wilful violations and destructive acts, all of which signify intent. Can any of these terms apply to AI, now, or even in the future? This is followed up in [47] by considering legal implications, starting from the following vantage point:

“The functioning of AI challenges traditional tests of intent and causation, which are used in virtually every field of law.”

Two scenarios, reminiscent of the Double-Bind raised earlier, emphasise the issue [47]:

• AI suggests a correct action, but the air traffic controller (ATCO) does not follow the suggestion, leading to an occurrence:

• Is the ATCO liable for breach of duty or professional negligence?
• On what basis does AI suggest a ‘correct action’?
• Does the AI follow a different standard or benchmark than the one followed by the ATCO?
• Does the ATCO have a duty to follow the AI’s suggestions?
• Can AI suggestions be used as evidence?

2.

AI suggests a wrong action, and the ATCO follows the suggestion, leading to an occurrence:

• Is the ATCO liable for breach of duty or professional negligence?
• Does the ATCO have an appropriate mental model about how the AI functions?

From the aviation professional’s standpoint, both scenarios seem risky. Just culture basically states that people do not usually go to work in order to cause an accident—quite the reverse—and so should not be punished for ‘honest mistakes’. It is easy to have such a viewpoint before an accident, but after an air crash there is a natural—i.e., very human—search for someone to blame, and hindsight tends to become very black and white in terms of ‘the pilot should have known/done/realized…’ whereas in reality, prior to the event, nothing was so black and white, and many other pilots (or controllers, or airport personnel, etc.) could have chosen the same course of action.

The problem with Human-AI Teaming and Just Culture is not simply a moral one. If aviation professionals are concerned about their accountability with respect to AI, they will be reluctant to use it, or err on the side of caution, e.g., always agreeing with the AI if the situation is not clear-cut. They may also be less likely to report openly and honestly about their thinking and decision-making prior to the event, e.g., stating that ‘I did consider that the AI’s advice might be incorrect,’ could lead to problems for the aviation professional in a courtroom situation. If professionals stop reporting incidents, or not disclosing everything, this would be a retrograde step for aviation and open up new safety vulnerabilities in what is generally seen today as the safest mode of transportation.

According to [48], Human-AI partnerships are trustworthy by design if the humans and machines can rely on each other, self-organise to take advantage of each other’s’ strengths and mitigate their weaknesses, and can be held accountable for their actions. It is this accountability with future AI systems that remains as yet unclear.

Scenario 2 above is exacerbated in future-envisaged situations involving AI categories 2B and 3A, wherein the AI can act autonomously, either in collaboration with the human (2B) or under a human-overseer / management-by-exception operational framework (3A). In both cases the AI could hypothetically be considered (at least legally) to have a certain degree of agency, and it is then a question of whether the human can detect erroneous AI behaviour (or conditions outside the AI’s ‘competence’) and intervene in time. In such cases, legal redress would likely fall to the organizations developing/owning the AI, which raises questions concerning the governance of AI systems in industry.

2.9. Ethical AI—Maintaining Meaningful Human Work

Just Culture is linked to the broader field of Ethics. As noted above, there is concern that some people may lose their jobs to AI, or that their jobs will be less satisfying, or that they will gain new jobs but receive less remuneration and less favourable employment conditions. In aviation, such concerns are clearly relatable to the concept of single pilot operations (SPO) in the cockpit, which could be enabled by AI at some point in the future. It is plausible that diminishing the human role could impact safety culture, because the human crew member may begin to see safety as the AI’s job rather than their own, especially if the AI becomes its own decision-maker. Such issues, essentially about the human’s role in work in society, fall into the domain of ethics, which is itself a major issue in the developing AI arena. Preliminary ethical principles have been outlined by the European Commission’s High Level Expert Group on AI (HLEG) [49]:

• Respect for human autonomy: AI systems should not subordinate, coerce, deceive, manipulate, condition or herd humans. AI systems should augment, complement and empower human cognitive, social and cultural skills, leave opportunity for human choice and secure human oversight over work processes, and support the creation of meaningful work.
• Prevention of harm: AI must not cause harm or adversely affect humans, and should protect human dignity, and not be open to malicious use or adverse effects due to information asymmetries or unequal balance of power.
• Fairness—this principle links to solidarity and justice, including redress against decisions made by AI or the companies operating/making them.

Such principles bode well for maintaining human agency and autonomy, which can be particularly critical for safety culture. However, they need to be translated into workable ‘good practices’ in industry.

2.10. Human Agency for Safety—Maintaining Safety Citizenship

Earlier it was stated that in aviation, ‘people make safety.’ What this means is that aviation personnel, whether pilots, cabin crew, air traffic controllers, aeronautical engineers or airport personnel, believe that safety is core to their duties. But what if the system—through increasingly effective automation and AI—becomes ultra-safe? There is a concern that if people are effectively ‘closed out’ from safety, either via automation that excludes human intervention, or because it is simply ultra-safe, then ‘safety citizenship’—the innate desire to keep things safe for ourselves and others—may degrade or disappear altogether [50].

Seven factors are cited which can erode safety citizenship, all of which could be affected by AI taking on a larger share of the safety role or the ‘safety space’.:

• Safety role ambiguity
• Safety role conflict
• Role overload
• Job insecurity
• Job characteristics
• Interpersonal safety conflicts
• Safety restrictions

The notion of safety citizenship, and more generally being proactive about safety (e.g., speaking up for safety), relies on a sense of self-determination [50]. Whereas an AI can be defined according to its function (what it does) and/or its mechanism (how it achieves it), humans are defined according to agency, by what they want to achieve (their goals and motivations), as well as their capabilities and limitations.

People need to feel autonomous (able to self-regulate their actions and experiences according to their interests and values), competent and related (socially connected, in an interdependent sense) in order to function in the world (self-determination theory). According to [50] this links to people’s personal identity (how you feel about yourself), and their social identity (how society thinks about you and the group you belong to. e.g., “I’m competent as a pilot (or air traffic controller, or engineer, etc., and pilots are useful in the world.” Taking away the human’s perception of identity and role can negatively affect self-determination, and may also be therefore expected to degrade safety culture, as the human’s role in the system’s overall safety space diminishes.

2.11. Ex Machina—AI Anthropomorphism, and Emotional AI

Human-AI Teaming is itself an anthropomorphic term [34], conveying the notion that the AI is in some sense a team player, devolving human qualities to a machine. This is reminiscent of Generative AI systems wherein people believe they are conversing with a person rather than a program (e.g., ChatGPT). There are two aspects to this issue, one philosophical, the other more practical. The philosophical question is whether an AI, at some time in the distant future, could have sentience. This remains too speculative an issue at this time, and so is left to other authors, as the focus of this paper is on Narrow AI. Sentience would only likely become plausible with Artificial General Intelligence (AGI), which does not yet exist, though it may well do so in the next decade [51].

The practical question is whether treating future AI systems as a team member could enhance overall team performance. A key sub-question is whether we can tell the difference between a human and an AI. In a recent study of ‘emotional attachment’ to AI as team members [52], most participants could tell the difference between an AI and a human from the interaction, i.e., they generally guess correctly when it is an AI.

Another study [53] examined human trust in AIs as a function of the perception of the AI’s identity. The authors found that AI ‘teammate’ performance matters to HAT performance and trust, whereas AI identity does not. The authors also cautioned against using deceit to pretend an AI is a human. Deception about AI teammate’s identity (pretending it is a human) did not improve overall performance, and led to less acceptance of their solutions, whereas knowing it is an AI led to better overall performance. What mattered most was the overall competency and helpfulness (utility) of the AI, which equates to how we learn to trust automation. What the authors also found was that AIs and hybrid-AI teams are better than human-only teams in terms of resource management in crisis management situations, and in a design engineering path-planning exercise.

Taken together, such results suggest that the concept of Human-AI Teaming does not require anthropomorphism. What will matter, both to the human members of the team and the executives deciding whether to deploy AI, is the effectiveness of the AI in doing its tasks, as well as how it affects the human team members’ workload, and whether it has an overall positive impact on the team’s performance.

These results are backed up by a further study [54] of attitudes to ‘emotional’ AI. This study is oriented more to societal impacts than industrial ones, i.e., general AI more than narrow AI applications. Nevertheless, it adds certain potential cultural impact areas into the HAT landscape, since industries, particularly global ones like aviation, are affected by diverse cultural norms. Key acceptance parameters for emotional AI were found to be loyalty (potential to erode existing social cohesion in the team), fairness, freedom from harm, purity (concerns about ‘mental/spiritual contamination’ by the AI) and authority (impacts on the status quo). As with the previous study, the authors found that people judge machines by outcomes.

It appears, therefore, that there is no evidence of a performance benefit with emotional AIs. However, there is another question concerning whether AIs need to be aware of human emotions. Would it make sense, for example, for AI’s supporting humans in an emergency to be aware of stress in the humans’ voices as conditions worsen? A recent study [50] found that monitoring people’s behaviour and emotional activity (speech, gestures, facial expressions, and physiological reactions), even if supposedly for health and wellbeing, can be seen as intrusive. Such monitoring activities can be for stress, fatigue and boredom monitoring, and error avoidance, and of course productivity, but understandably people may not like this level of personal intrusion of their behaviour, bodies and data, and can lead to a feeling that the organization does not trust them.

Overall, therefore, it appears that aviation needs neither anthromorphic or emotional AI. This may sit better with safety culture, as considering the AI component of a Human-AI Team as an entity with feelings may ‘muddy the waters’ when it comes to safety responsibilities and safe interactions with the AI.

2.12. Governance of AI, and Organizational Leadership

In April 2021, the European Commission laid out a proposal for harmonised rules on AI [55]. The primary focus is on General AI where it intends to have strong risk-based governance on high-risk applications in society. Interestingly, the provisional agreement intends to ban, for example, cognitive behavioural manipulation, and emotion recognition in the workplace, although it also states that there will be an obligation for users of an emotion recognition system to inform natural persons when they are being exposed to such a system. Perhaps, therefore, exceptions will be made for Narrow AI applications where there might arguably be a safety advantage, e.g., civil and/or military aviation. Although the EU Act is mainly focused on General AI, it is likely that its edicts, when published and written into European Law, will set the tone for governance and regulation of Narrow AI across a range of industries.

Three aspects from the EU Act on AI likely to bleed over into the industrial arena, including aviation, are notable [47]. The first is that AI systems must be sufficiently transparent to enable users to interpret the system’s output and use it appropriately. Second, they must be resilient regarding errors, faults or inconsistencies that may occur within the system or the environment in which the system operates, in particular due to their interaction with natural persons or other systems. Third, human oversight must be employed to prevent or minimise safety risks that can emerge both when a high-risk AI system is used in accordance with its intended purpose or under conditions of reasonably foreseeable misuse.

But does high level governance influence safety culture? Governance at this highest level (e.g., EU), along with global bodies such as ICAO (the International Civil Aviation Organisation) and organisations such as the FAA in the US, and EUROCONTROL and EASA in Europe, can set the tone for the perception of AI’s role in the aviation industry. The tone matters. If, for example, the tone is that AI design must be human-centric, and not negatively affect human wellbeing, nor displace the human workforce, this affects how business leaders and CEOs of key organisations—both operational and manufacturing—consider AI and its role, including that of safety. If, for example, AI’s capability is overestimated, such that human error is seen as the problem and AI the solution, then the industry may work towards reducing human control of the ‘safety space’, putting the safety of passengers and crews in the metaphorical hands of AI systems.

This is risky, as already identified in the Maritime industry [56], since there can be ‘Tail Effects’, wherein low probability events are impractical to train AIs on, but when they occur the AI will not be able to handle them. The Maritime study suggests the need for active back-up control for autonomous ships (largely controlled from onshore control centres). In this scenario, human control is not necessarily decreasing. As AI autonomy goes up, passive back-up is likely to be ineffective, in part because AI can lead to ‘increasing invisible interactions’ such that the humans miss what is going on, in terms of the system and sub-system interactions and relationships, and are unable to understand the complexity and gain a holistic picture. The authors also point out that in maritime operations managing VHF comms are easy for humans and hard for AI: part of the ‘easy things are hard’ paradox in AI. Their conclusion runs counter to the way aviation (which is arguably more technocentric) is headed:

“It seems counter-intuitive, then, to categorise the level of automation by degree of autonomous control gained over human control lost, when in practice both are needed to ensure safety.” [56]

Again, looking for a moment outside the aviation domain, the UK Ministry of Defence has published its own AI Strategy [57]. The strategy indeed sets the tone from the outset with the following statement:

“Machines are good at doing things right; humans are good at doing the right thing.”

Such a statement clearly makes the point that human judgement will continue to be valued in future AI-enhanced defense platforms and scenarios. The paper goes on to ask whether the Defense industry has the right culture, leadership, policies and skills in place to make the best use of AI, which it considers it must develop to counter significant foreign threats now and in the future. The defence industry, therefore, is already considering how to approach AI and its potential autonomy from an organizational perspective. It is also interesting to note that they intend to focus on training middle management concerning AI. The Defence AI Strategy, significantly, also poses a set of questions around when to use AI, and when not to, which sometimes appears missing in the current rush to ‘try out AI’ in myriad projects in a number of industries, including aviation:

• Where is AI the right solution?
• Do we have the right data?
• Do we have the right computing power?
• Do we have fit-for-purpose models?

A further recent paper on organizational safety and autonomy [58] considered two models of how safety works in large organisations: safety can be seen as a centralized and hierarchical rule-based, compliance-based system, or decentralized, responding to local problems in an agile way, through ‘loose couplings’. In the former, what the CEO says matters, as it will be cascaded down through middle management to the rest of the workforce, including those in design and development, validation and testing, procurement, human resources and training. But even in a decentralized organizational arrangement, people will still respond to what top management say about AI and its role in the organisation’s strategy and operations, as evidenced by the importance of ‘Management Commitment to Safety’ in most models of safety culture.

What top management says, however, needs to be borne from a well-informed understanding of AI and its realistic capabilities and limitations. In the current ‘hype’ around AI, the former tend to be exaggerated and the latter underspecified, ignored or unknown. This may mean that those aviation organisations ‘buying into AI’ need to recruit serious AI expertise in-house, so that they can make balanced judgements at Board level. Here, it is perhaps worth noting that following the two B737 Max accidents [27,28], Boeing invited someone new to their Board who had aviation operational experience, since beforehand corporate goals—perhaps under-informed by operational insight—had unwittingly contributed to safety vulnerabilities emerging in the B737 Max design. This could be a salutary lesson for the top management of organisations considering using AI to transform their operations, that the key (AI / data science) expertise should not be buried too low in the organization, or simply outsourced. In a similar vein, following the UK Nimrod accident, the report [59] stated that:

• “Failures in leadership and organizational safety culture led to the Nimrod incident where the aircraft developed serious technical failures, preceded by deficiencies in safety case and a lack of proper documentation and communication between the relevant organisations.”
• Furthermore, on p.474: “The ownership of risk is fragmented and dispersed, and there is a lack of clear understanding or guidance as to what levels of risk can be owned/managed/mitigated and by whom.”
• And, p.403: “These organisational failures were both failure of leadership and collective failures to keep safety and airworthiness at the top of the agenda despite the seas of change during the period.”

At the outset of this paper it was noted that organisations need both an SMS (Safety Management System) and safety culture. There is a very real danger that the potential safety impact of integration of future AI ‘tech’ into operational aviation systems is underestimated, to the extent that it is believed current SMSs can handle it. This would effectively be the ‘old wine in new bottles’ approach, and could lead to significant safety vulnerabilities in future aviation systems.

Perhaps one thing CEOs need to know is that AIs cannot value things in the way humans can, especially safety, as currently it is not known how to program values [37]. This may interest CEOs as they are often concerned with value alignment in the organization. As far as safety is concerned, humans can experience a range of emotions including fear and concern for lives under threat, and loss and grief in the event of a fatality, all of which can underpin a strong value for safety. An AI cannot experience any of these, and while various reward schemes and supervised learning could in theory reinforce safety in the machine’s workings, it will still be ‘running the numbers’, and if it gets them wrong will not experience remorse or regret. Whilst AIs can mimic human behaviour and even have a built-in ‘persona’, this remains mimicry; they are still machines, or simply ‘just more automation.’ [34] A CEO might therefore wish to have a human eye on the screens and a human hand within reach of the joystick and, in military aviation where many more lives may be at stake, a human finger on the trigger.

In summary, governance3 of AI, both at supra-national and corporate level, could be a critical pillar in terms of human-AI partnerships that retain aviation’s strong safety record.

3. Safety Culture Evaluation of Future Human-AI Teaming in Aviation

The foregoing introduction has elucidated the multi-faceted challenges posed by AI in fuure aviation systems. The remainder of the paper reviews the prospect of Intelligent Assistants in aviation through the lens of safety culure measurement, because this is how safety culture is evaluated in organisations to determine where it is working well, and where improvements are needed. Although such measurement tools—typically questionnaires—were not developed with AI in mind, the questionnaire items and dimensions—especially those relating to teamwork—can be analysed to see where and how Intelligent Assistants might affect a respondent’s answers.

Materials and Method

The EUROCONTROL Safety Culture Questionnaire was utilised for this study, in particular a version that has been adapted to also accommodate airlines and airports [23,24,25]. This questionnaire includes the fatigue dimension which is seen as a key operational factor in airline and airport surveys in particular, but the fatigue-related questions were excluded from the current study as fatigue is not strictly speaking a safety culture dimension and, put simply, AI’s do not get tired, and it is too speculative at this juncture to consider how their future use may or may not affect aviation crew and staff fatigue.

Safety culture surveys normally proceed via a large number of operational staff completing the survey anonymously, and then collating the results. Since the kinds of AI or IAs of concern (i.e., with levels of autonomy 2B or higher) do not yet exist in aviation, such an approach was not possible. Instead, three aviation safety culture practitioners who have carried out multiple surveys involving airlines, airports and air traffic controllers, and who are also currently working in the aviation Human AI Teaming research area, participated in the study. The most experienced expert, who has been involved in more than 30 surveys over the past 20 years, carried out the first principal assessment. Each of the remaining 48 questionnaire items from the EUROCONTROL questionnaire was considered in the context of a future Intelligent Assistant, e.g., commitment to safety might reduce if the Intelligent Assistant was judged to be handling safety well and robustly, and this could affect both human operators at the ‘sharp end’ as well as managers running the organisation.

The results of this first analysis were then reviewed independently by the other two practitioners, with alternatives/queries raised. An effort was made by all three practitioners to consider not only negative impacts, but also potential positive ones. The three experts then met to resolve and finalise the assessment, culminating in a table of key considerations for each safety culture item.

4. Results

Table 1 shows how each safety culture questionnaire item might be impacted by the presence of an Intelligent Assistant supporting a human team. Each row shows the questionnaire item, the safety culture dimension it relates to, the assessed impact due to IA presence and whether the impact is judged likely to be high, medium or low.

The analysis in Table 1 suggests a broad categorisation of the IA’s impact on the various safety culture dimensions, from high to low, as follows:

○

High Impact:Colleague Commitment to Safety; Just Culture & Reporting; Risk Handling

○

Medium Impact:Staff and Equipment; Procedures & Training; Communication & Learning; Collaboration & Involvement

○

Low Impact:Management Commitment to Safety

Each of these can be either considered a concern about negative impacts on safety culture that needs to be managed, or a ‘safety affordance’, wherein the IA could help support and possibly enhance current safety culture, safety management processes and operational safety practices. Since a number of the insights in the rows in Table 1 overlap or point to a single central issue, they were further analysed to distil the key insights from the analysis, in terms of safety culture concerns, and safety culture affordances. These are shown in Table 2.

The results in Table 2 suggest broad equivalence between potential positive and negative impacts of AI on safety culture, though the consequences of loss of safety culture could be much more dramatic in terms of aviation accidents. The next section discusses how to mitigate the negative impacts whilst bolstering the positive ones, allocating them to risk owners in organisations.

4.1. Safeguards and Organisational Risk Owners

The analysis above has raised a number of potential threats to safety culture, and a more or less equivalent number of ‘safety culture affordances’ wherein safety culture could be enhanced. In this sense the overall impact of AI and Human AI Teaming on safety culture will depend on how it is researched, designed, developed, deployed and managed in actual operational environments. These issues, whether positive or negative, can lead to safeguards to prevent safety from being diminished due to the introduction of advanced AI systems into aviation. However, for safeguards to be effective, those who can enact them need to be identified.

The various impacts are notably diversely spread across different human ‘levels’ in organisations; some relating to front-line staff, some to middle management, and some to senior or executive levels. Safety culture always works best in aviation when those at the top—CEOs, VPs and Executive Boards, firmly believe in and support safety as a priority. There needs to be continued Safety Stewardship by senior executives, to maintain the human as the key safety agent, which can then be translated by middle management throughout the organization into satisfactory actionable outcomes. A useful, if simple hierarchical model of organizational safety was proposed in a project considering safety culture and safety management in aviation organisations [61], and is illustrated in Figure 5 below. This model has been used to highlight the key—but often under-explored—role of the middle management layer in safety [62].

To this three-level model must be added an extra level, highlighting the key role of the safety department in an aviation organization, including risk modelers and incident investigators, as this is typically the hub of safety learning (local safety knowledge) for the organization, and the key working interface with external regulators. This new, four-layer model, is shown in Figure 6, with the key safety culture concerns and affordances inserted at appropriate ‘risk owner’ levels.

At the bottom layer are the impacts on front-line and support staff. First is the need to maintain human agency for safety, i.e., a valid safety role. Second is the fact that the IA can act as a second pair of eyes, whether aiding in an emergency, or noting a safety issue or deviation or risky course of action by the human operator. This leads to a third useful aspect of an IA, that it can be a ready-to-hand safety oracle that the human team can consult at any point in time when considering the best course of action and any safety risks it might entail. The IA could also be programmed to ‘speak up’ for safety if warranted, and this can be embedded into human CRM and TRM practices and training. The IA could be a useful aid for safety reporting, able to rapidly, quickly capture events, their precursors, signals and actions, to which the human could then add a narrative. All NOTAMs (Notices to Airmen—these are likely to be digitized in the near future) could be automatically uploaded into the IA, which could remind human crews if they have forgotten any aspects during operations. Similarly, the IA could be useful as a day-to-day briefing tool, letting the oncoming shift know of anything unusual, or changes to procedures, or the status of ongoing maintenance, etc. that has happened on previous shifts. Taken together, these nine elements could keep safety at the human’s fingertips, eyes or ears, whilst guarding against simple omissions as well as reckless acts.

At the next layer is the safety department. A first safety affordance is the notion of the IA serving as a living black box, such that after an event the IA could reproduce the detailed flow of events, signals, interactions, decisions made and even the thinking underpinning those decisions, prior to and during the incident. This could paint a much more detailed picture than investigators currently have. Such an ‘annotated timeline’ could also be very useful in safety learning and training. In parallel, investigators will undoubtedly need to develop new Human-AI investigation protocols to deal with Human-AI Teaming events, particularly when relating to the double-bind type of scenarios raised earlier. These protocols should be informed by Just Culture principles adopted at higher levels in the organization and hopefully enshrined in European law.

The IA could also be used to compare ways of working (what is actually done) against procedures and rules, not as a means of policing, but as a way of defining the gap between real operational practices and the official rules and procedures. If the gaps are unsafe, then this can lead to more training, but in many cases it is likely that the official rules are either inefficient or even unworkable in real operational conditions, or else are in need of updating as operations and technology may have moved on. Similarly, risk models in aviation are often seen as not really reflecting operational reality, or being at too high a level of description. The IA could record interactions in both safety-related events and ‘when things go right’, with such information feeding into risk models to make them more operationally relevant, and giving them a more detailed level of description. If this can be achieved, then such models can become useful to operations departments, and not just seen as for safety departments and regulators. The ultimate goal here would be that day-to-day operations are feeding dynamic risk models so that safety performance can be seen, including when things may be drifting towards danger, or when new hazards are emerging. This could pave the way for 24/7 safety monitoring and learning.

At the next level up in the pyramid model is Middle Management, who have the challenge of exercising senior management aspirations within real world operational and resource constraints. Part of their mission with respect to safety culture & AI is to ensure that staff have meaningful jobs, that IAs do not act as ‘snitches’ on staff or police them, and that Just Culture ideals can be translated into effective and trusted principles and practices enacted at lower levels, in agreement with social partners (unions etc.). A key roled will be the overseeing of the introduction of autonomous AI systems into the organization, ensuring that they are user-centric especially where ‘explainablility’ of the AI’s advice or decisions (XAI) to the human is concerned, and undergo human-supervised-learning followed by user validation and Human-AI Team training prior to operational deployment. The discipline of Human Factors is likely to be a critical determinant of success in these activities. If the IA, as is likely, has more key performance areas (KPAs) than safety alone (e.g., productivity, green-ness, etc.), then middle management must ensure that safety retains priority when the IA is making trade-offs. Lastly, middle management must closely monitor the IA’s introduction as it will evolve both in its dealings with humans and also other IAs.

The top level in the pyramid is senior management, including CEOs, VPs, Directors and Executive Boards. Here is where there needs to be an authentic message that safety is the priority and that ‘people still make safety’, albeit backed up and supported by AI. There should also ideally be a code of ethics related to the use of AI in the organization, as well as a Just Culture policy and framework which deals with AI accountability in the case of an accident, to the satisfaction of social partners. It is also at this level that decisions need to be made on having internal AI expertise in the organization, so that organizational leaders can maintain a basic understanding and realistic expectations of their AI ‘assets’, and be prepared to face and answer the media if things go wrong.

4.2. Further Research Needed

The above section considers specific safeguards and allocates them to ‘risk owners’ to facilitate their development into operational practices. Taken together, they comprise a future safety culture ‘Conops’ (Concept of Operation) for aviation organisations, i.e., a vision of how safety culture could look in the coming decade as AI autonomy rises and Intelligent Assistants enter the workspace.

However, organizations rarely operate unilaterally; they are subject to industry and regulatory standards and best practices, as well as external laws and edicts such as the forthcoming European Act on AI. Therefore, certain cornerstones of safety need to ‘raise their game’ in preparation for the advent of more advanced AI systems, so that when such systems arrive, organisations will have the right theory, tools and regulatory landscape to put effective safeguards into place. Five broad research areas are accordingly listed below, aimed at bolstering cross-industry pillars of safety that could both support, and leverage organisations in their efforts to secure a stable foundation for safety and safety culture in future AI-assisted aviation:

i.

Just Culture—if Just Culture is to be preserved, rationales and arguments need to be developed that will stand up in courts of law, that will protect crew and workers who made not only an honest mistake, but an honest (i.e., a priori reasonable) judgement about whether or not to follow AI advice, and whether or not to intervene, thus contravening AI autonomous actions seen as potentially dangerous. Such development of Just Culture argumentation and supporting principles with respect to AI and Human AI Teaming should include simulated test cases being run in ‘legal sandboxes’.

ii.

Safety Management Systems (SMS)—the key counterpart of safety culture in aviation—will also need to adapt to higher levels of AI autonomy, as is already being suggested in [38,41]. This will probably need new thinking and new approaches, for example with respect to the treatment of HAT in risk models, rather than simply producing ‘old wine in new bottles’. SMS maturity models, such as are used in air traffic organisations around the globe [63], will also need to adapt to address advanced AI integration into operations.

iii.

Human Factors has a key role to play in the development of HAT [64], especially if such systems are truly intended to be human-centric. This will require co-development work between Human Factors practitioners/researchers and data scientists / AI developers, so that the human—who after, literally has ‘skin in the game’—is always fully represented in the determination of optimal solutions for explainability, teamworking, shared situation awareness, supervised learning, human-AI interaction means and devices, and HAT training strategies. A number of new research projects are beginning to pave the way forward. This applied research focus needs to be sustained, and a clear methodology developed for assuring usable and trustworthy Human-AI Teaming arrangements.

iv.

There are currently a number of Human -AI Teaming options on the table, e.g., from EASA’s 1B to 3A; see also [65]), with 2A, 2B and 3A offering most challenge to the human’s agency for safety, and hence with most potential impact on safety culture. Yet these are the levels of AI autonomy that could also bring significant safety advantages. It would be useful, therefore, to explore the actual relative safety advantages and concomitant risks of these and other AI autonomy levels, via risk evaluations of aviation safety-related use cases. Such analysis could result in common design philosophies and practices for aviation system manufacturers.

v.

Inter-Sector Collaboration will be beneficial, whether between HAT developments in different transport modalities (e.g., road, sea and rail) or different industry sectors, including the military, who are likely to be most challenged when it comes to both ethical dilemmas and high intensity, high-risk human-AI teamworking. This paper has already highlighted learning points from maritime and military domains for aviation, so closer collaboration is likely to be beneficial. Collaboration between the transport domains in particular makes sense, given that in the foreseeable future AIs from different transport modes will likely be interacting with each other.

7. Limitations of the Study and Further Work

This paper and its analysis are speculative in nature, given that the focus is high-autonomy Intelligent Assistance in aviation, which does not yet exist. Such speculation is arguably warranted, however, to forestall potential negative impacts on the very fabric of safety culture in aviation, as well as to ‘get AI right the first time’ by capitalizing on potential safety affordances. As noted earlier, there is some ‘breathing space, but AI is a fast-developing, potentially disruptive and could well become a ‘game-changer’ for the industry. Nevertheless, it is intended to follow up the safety culture exercise in this paper with another one towards the end of the HAIKU project (2025), when more operational and research personnel will have been exposed to realistic simulations of future HAT systems in a broad range of use cases. This may also lead to the development of a safety culture ‘mini-questionnaire’ focusing solely on AI and HAT.

8. Conclusion

This paper has reviewed the current positive state of safety culture in aviation, and the future AI possibilities and challenges for the industry, in particular focusing on Human-AI Teaming envisaged for the 2030+ timeframe, wherein an Intelligent Assistant could have a moderate or high degree of autonomy. The results of a preliminary analysis of the potential impacts of future advanced AI on aviation safety culture suggests there are both significant threats and also potential benefits, depending on how the AI is designed and implemented, and whether the AI is ‘human centric’ or not.

Given the importance of safety culture to aviation safety, adopting a ‘wait and see’ attitude is not advisable. Accordingly, a range of safeguards with associated organizational risk owners has been identified, along with more fundamental research avenues to help aviation steer a safe course through the development and deployment of advanced AI support systems. These will help ensure that aviation’s hard-won level of both safety culture and safety are maintained, if not improved. A cornerstone to all of these strategies is that the human must maintain a strong safety role in aviation, whatever the AI role. It may be that at some time in the future, e.g., 2050, AI will have proven itself to be more reliable than humans, but until such time, people will continue to make safety, and should remain at the heart of safety of all aviation operations.