Ethics Assurance in Swarm Engineering: Moral Vectoring Design Pattern for Decentralised Artificial Intelligence

1. Introduction

Ethics has gained significant attention in the literature on artificial intelligence (AI), with a large volume of research on the topic published over the past decade. Central to this research is the importance of ethics. This importance has been assumed by researchers. Who can deny the importance of ethics in a human social system? Unfortunately, taking the importance of ethics for granted has not served the cause well. One reason is that, by assuming everyone accepts the importance of ethics, the discussion becomes vulnerable as soon as this assumption is questioned or challenged; the argument weakens; and the person asking the question, rhetorically or for real, is left wondering whether the argument was rigorous. The second reason is that ethics, on one level, is a common-sense topic, and professionals in organisations are expected to understand it as part of their training. Professionals are asked about their interpersonal skills, values, and teamwork and they get tested before hiring.

Another belief is that ethics could hinder mission success. Actions that generate ethical tension could lead to suboptimal decisions for a mission and for the organisation as a whole. Organisations may then wonder why they should consider ethics an important organisational topic if it is unrelated to or hinders organisational objectives! The board will not spend resources, including time, on discussions that do not connect to organisational objectives. For example, organisational culture is important because it impacts productivity; in profit-based organisations, productivity affects profit, while in non-profit organisations, it affects the quality of service. Failure to connect ethics to organisational objectives reflects a misunderstanding of ethics at its core. Ethics, in its simplest terms, is doing what is right, and the board serves its function by doing what is right for an organisation. Discussing ethics without explicit linkages to organisational objectives is objectively and fundamentally less ethical, and even some board members will claim it is an unethical act because board members should be doing their due diligence for the organisation. This may sound weird and require multiple cycles of deep reflections, but the key message is that ethics needs clear links to organisational objectives to have a clear place on the board’s agenda.

We start from the combined position that, to consider ethics genuinely in organisations, evidence is needed to demonstrate how ethics affects the decisions and objectives of missions and organisations. The objective of this chapter is to introduce ethics assurance as a methodology for systematically developing evidence to demonstrate why ethics is essential and how it supports mission success and organisational objectives. A safety-critical use case provides context for explaining and contextualising the work. The use case is an Unmanned Aircraft Traffic Management (UTM) system in which unmanned aerial vehicles (UAVs) need to deter birds from runways. The use case is chosen to demonstrate how: (1) ethics connect to organisational objectives, (2) the involvement of animals adds animal ethics considerations to human ethics ones, (3) the safety-critical nature of the application demands clear strategies for tension resolution, and (4) this contemporary use case is an example of a safety-critical command-and-control ecosystem with many interactive parts, nonlinearity, and assurance challenges.

Before we delve into the definitions, methodology and mathematics, it is important to introduce the field of ethics. After this review, we present and explain the definitions. The methodology is then presented, followed by key challenges.

2. Ethics

2.1. Bridging Human Ethics to AI

schlosser2015agencyAn agent is a being with the capacity to act, and “agency” denotes the exercise or manifestation of this capacity. Rich defines ethics as [p.4]rich2013introductiona systematic approach to understanding, analysing, and distinguishing matters of right and wrong, good and bad, and admirable and deplorable as they relate to the well-being of and the relationships among sentient beings.

Moral situations [7] involve at least two agents, and at least one of them needs to be a moral agent. The moral agent is confronted with multiple courses of action with different moral implications for the other agent. Ethical acts fall into three categories: moral acts, [p.4]rich2013introductionspecific beliefs, behaviours, and ways of being derived from doing ethics, immoral, which are acts opposed to moral conduct, and amoral, where the actor lacks concern for moral behaviours. Approaches to ethics fall into two broad categories: normative, focusing on what should be, and descriptive, focusing on how things are.

Normative ethics [24] prescribe values, behaviours, and ways that are right or wrong. The three common categories of normative ethics are deontological, consequentialism, and virtue. Each normative school has its share of attempts in the formal and mathematical modelling literature.

Deontological ethics subjects agents to a set of duties. Actions fall into two categories: prohibited actions that should not be performed and permitted actions, representing all actions except prohibited ones. The permitted actions category is further categorised into obligatory actions that must be performed and optional actions, which are permissible but not obligatory.

Deontological ethics rely on a set of permissible and prohibitive rules when selecting ethical acts. The ethical implication of a decision is measured by an agent’s success or failure in abiding by the rules that define its duties. The deontological school has occupied a substantial portion of the work on modelling ethics. Seeing duties as rules of different categories, logicians formulated deontological logic to represent prohibitive and permissible acts. Meanwhile, classic decision theory differentiates between hard constraints, representing must-do and must-not-do, and soft constraints and objectives, representing ought-to-do.

Consequentialism ethics evaluates each course of action by projecting its consequences to anticipate, assess, and estimate them. Moral actions are those with the best consequences. Consequentialism follows utilitarian philosophy, in which good or bad depends on expected consequences. The ethical implication of a decision is measured by the expected outcomes of the decision and their alignment with an agent’s moral goals. While utilitarian philosophy differs from utility theory in decision-making, the latter has been the predominant means of modelling the former. An agent’s goals get modelled as utility functions. Each utility function gets associated with a likelihood representing the probability of the outcome. An expected utility function is formed. Consequentialism ethics is recast as the maximisation of the resultant total expected utility.

Agents’ repeated actions form their characters, which in turn shape their moral decisions. Virtue ethics assumes agents’ characters shape their choices. Characters and preferences are seen as independent of each other, where preferences display characters. Decisions in the virtue school are assessed against the action(s) that an ideal agent with standard character traits would do. The ethical implication of a decision is measured by the degree of alignment between the decision and an agent’s values. Virtue ethics is possibly the most debated form of normative ethics when scientists attempt to model it.

The philosophical grounding sees human character as an inherent element of being human and as such, an AI should not, and on principle cannot, imitate human characters. The decision theory literature has long relied on value and utility theory to represent human “preferences”. A contentious point is whether using value and utility functions in an AI agent is equivalent to human character traits. While we will not regress to this discussion, it is important to state that modelling, by definition, is a simplification of reality to serve a particular purpose. Our purpose in modelling ethics within an AI agent is to reduce misalignment between what an AI selects and what a human would select if presented with the same information the AI has. It is this information-theoretic lens that we pursue in the remainder of this chapter.

In our description of each normative school of ethics above, we intentionally used the word “implication” for three reasons: to avoid the word “consequence” which has a particular frame of reference in normative ethics, to emphasise that regardless of which school of ethics one subscribes to, differences in implications do not eliminate the existence of implications, and to formalise a category of a particular set that we call implication set, that sits at the core of the approach proposed in this chapter although we will not cover it further in this chapter for simplicity.

Moral tensions arise when an ethical approach approves a choice that another approach would not. For example, lying to provide psychological support to a person in need may be acceptable under consequentialism ethics, yet a deontological approach may prohibit lying. Conversely, the deontological approach would regard honesty as obligatory, yet even if the consequences may harm someone, which consequentialism ethics would reject.

Moral foundation theory (MFT) [19] suggests that independent of cultural differences, humans share common, innate, and universal moral foundations. More specifically, there are five foundations: Care/harm, Fairness/cheating, Loyalty/betrayal, Authority/subversion, and Sanctity/degradation. A sixth foundation of liberty/oppression was added [4,21]. Moral foundations are the “moral taste receptors” [15] that underpin the moral cuisine of any culture. They became the basis for moral profiling (see, for example, Moral Machine [5] and Moral Compass [2]).

Discussing ethics in the context of AI is a bridge between humans, as a well-established and accepted class of moral beings with moral rights and agency, and AI, long perceived as a technology/tool to support human decision-making. To smooth the roughness of this long bridge, we discuss nonhuman animals and ethics. Most of the debate on nonhuman animals distilled the thought processes some philosophers go through when “thinking ethics”. Our aim is not to add to the philosophical literature, but to distil requirements for the technological operationalisation of ethics.

2.2. Nonhuman Moral Agents

The ethics space has multiple actors: humans, nonhuman animals (animals for short), and artificial agents (robots, artificial intelligence enabled-autonomous system). Each actor is a social participant. While healthy human adults have the highest moral responsibility, the assignment of moral rights to each of these actors is a non-trivial topic that has had significant coverage in the literature (see, for example, [18,25]). Arguments have been of two types: capacity-based and function-based. The former is more formative in nature and focuses on cognitive traits, while the latter focuses more on participatory experiences in practice. Dogs have received more attention [10,18], possibly due to their social role in a human society. Significant research went into nonhuman animals in relation to their role as Sapontzis1980Aremoral beings, shapiro2006moralmoral agents, or rights to merit degrazia1996takingmoral status. Below is a short summary of a few elements needed for modelling in this chapter.

• Moral Beings: Animals display moral qualities [25] such as compassion and loyalty by untrained dogs and self-restraint by trained dogs. Sapontzis [25] acknowledges the importance that moral beings display moral qualities, but in addition, moral beings also need to be capable of moral reasoning. A useful perspective from Sapontzis [25] is that justification of actions is required only for immoral actions. However, such a perspective is not universally accepted.
  Sapontzis distinguished two variables: $mora{l}_n$ representing the moral value of an action independent of the type of agent or intentionality, and $mora{l}_a$ representing the moral value of the agent when the agent prefers actions with higher $mora{l}_n$. While being kind has a fixed $mora{l}_n$ value, the associated moral worthiness of an agent increases its $mora{l}_a$ when it prefers the act of kindness over other actions with less $mora{l}_n$ values. Sapontzis [25] defines two key conditions for intentionality: responsibility and comprehension. The wolf is responsible for the young, with the free will to choose to care, and is recognising the young and the need to care (comprehension of $mora{l}_n$). Consequently, the wolf is acting in a $morall{y}_a$ sense and is displaying a moral foundation (caring). An ant following a pheromone trail is not a moral agent because it lacks the responsibility and comprehension; it is true it can smell (sense) the pheromone but it cannot recognise $mora{l}_n$.
• Moral Agency - Capacity Approach: Shapiro [27] sees three levels on the scale of moral agency. At the lowest end of moral agency, acting virtuously (such as the principle of proper conduct) is a display of moral qualities. Reciprocity is a middle level of moral agency that demands a position and the display of disapproval when an expectation of fairness is unmet. The highest level of moral agency is when an agent acts on moral principles (such as caring about others), with those principles forming the basis for action selection. Shapiro [27] surfaces a few important points with significant relevance to this chapter. He sees moral understanding as a necessary condition for moral responsibility. The degree of moral responsibility needs to be proportional to the factors influencing an agent’s understanding of a situation. Training sets expectations for particular acts. The more an agent is expected to act in certain ways, the more it becomes obliged. The penalty for disobedience is proportional to the amount of training.
• Moral Agency - Function Approach: Behdadi defines moral agency as behdadi2021practicean entity considered to be able to do wrong (or right)1 and typically taken to be morally responsible for actions, omissions, beliefs, and/or character traits. To assert moral agency for nonhuman animals, Behdadi introduces the concept of “Moral Responsibility Practices” (MRPs), which are behdadi2021practicepatterns of holding oneself and others responsible in practices. He suggests three requirements to (1) identify the key features of MRP, (2) establish the conditions for participation in an MRP, and (3) offer evidence for animal behaviours analog to the human counterparts in MRP. He replaces the argument of universality followed by the normative school and MFT with the argument of practice, where he views right and wrong as to behdadi2021practicewhat is acceptable to others.
  Behdadi uses a behdadi2021practiceFunction Argument to contrast animal behaviours to humans. Actions perform functions. Behdadi cites an analogy by Michael McKenna. Actions are vehicles of meaning. Actions perform functions; they carry a quality of will. Responsibility is a two-agent conversation. Interaction is a conversation where good wills are rewarded and bad wills are punished. The reactions to actions are feedback messages to communicate quality of will. Moral responsibility exchange is a conversation. When agents participate in an MRP, feedback drives change. Agents participating in MRPs communicate using this language analogy. Canids’social play comprises practices to promote moral values such as peace and cooperation.
• Moral Status: [p.183]degrazia2008moralTo say that X has moral status is to say that (1) moral agents have obligations regarding X, (2) X has interests, and (3) the obligations are based (at least partly) on X’s interests. Such a definition poses a primary question: what are the interests of nonhuman animals that warrant obligations from moral agents? MFT may suggest that moral foundations are a subset of interests that moral agents possess. While the premise is sound, a thing may have a moral status even if it is not necessarily qualified for moral agency. In the definition of moral status, X is a thing with interests and merits a moral status; X does not have to be a moral agent. The obligations towards X are the responsibility of moral agents. To establish an interest, though, X needs to possess the ability to learn categories. In the absence of such an ability, X cannot develop an interest in a category of things. Learning categories is foundational for developing interest, preferences, and reasoning in general.
  Crary suggests a rational agent is one with the ability to learn universal categories (fluffy objects look like sheep) and position them in an ordered space of reasons (if sheep are spread, collect them, and when they are grouped, drive them to the paddock because this is how you get them to the paddock). Learning universal categories and ordered spaces of reasoning are, therefore, together necessary and sufficient conditions for rationality. This perspective is inherited from the Sellarsian insight, which [p.218]crary2012dogsdemands that to characterise something as a state of knowing is to situate it in a normatively ordered “space of reasons” so that there is room for questions about, for instance, what justifies it.
  Crary [10] argued for the moral status of nonhuman animals. She approached the subject from a development lens. She sees learning and maturation as the basis for growth on the continuum of rationality. Learning is the acquisition of “specific capacities for dealing with universal categories”, while maturation is the acquisition of “capacities we have insofar as “I think” can accommodate our different representation”. Crary argues that animals and human children have learning abilities, although they have not yet reached the level of maturation of a human adult.
  [p.225]crary2012dogs[I]ndividual stages of growth need to be understood as locations on a naturalistic continuum. So we need to be open to the possibility of discovering that the developmental stages we designate as pre-rational involve capacities of mind that resemble, in significant respects, capacities of mind characteristic of rational human beings. Moreover, by allowing significant similarities between the mental capacities of pre-rational human beings and their rational counterparts, we also allow significant similarities between the characteristic mental capacities of non-rational animals and those of rational human beings.

2.3. Machine Ethics

Machine ethics is a field [p.1]Anderson2004Towardsconcerned with the consequences of machine behaviour toward human users and other machines. The literature on the topic covers a wide spectrum of activities, from the design of formalisms to test or implement ethics within machines to the creation of games and datasets.

Inspired by Chomsky’s work in linguistics, Mikhail [20] advocated for a Universal Moral Grammar (UMG). On a foundational level, UMG follows a philosophy similar to Chomsky’s, who advocated that language is innate. Similarly, Mikhail saw morality as innate and pre-determined in the brain, while experience shapes ontogenetic development. If an act is denoted by a, omission becomes $\neg a$. Due to interdependency, he suggested that one of the three words, obligatory, permissible and forbidden, is sufficient to represent the other two. A UMG consists of three elements: deontic rules, structural descriptions in the form of act trees, and conversion operators. A stimulus gets converted to an act tree using the conversion operators. An act tree consists of means, ends, and side effects. The deontological school centres the decision on the “means”, while the consequentialism school compares the effects of the ends to the side effects. By anchoring the assessment in the act rather than its consequences, the means are checked against deontic rules. If the act is not forbidden, it is permissible. He argued that an agent can still make a decision even if none of the courses of action is permissible.

To perform moral reasoning, DeBellis [p.3]debellis15327812ethicsattempt[s] to model the underlying logic that must be part of any UMG. He developed an ontology with the first level of categorisation centred on five categories: agents, events, causality, language and living things, locations and time. He then used four rules to define a Theory of Mind (TOM) module in the ontology: agent causes event, agent has a Goal state, event causes state, and event is a precondition on state. The preconditions of an event are the states needed for it to occur. An event causes a state; it is due to an event that a state may come to exist or continue to exist. These events are caused by agents whose goals are represented as states. In other words, an agent’s goals are states. Agents act on their goals by creating events. These events could create new states that serve as preconditions for other events to occur. This chain reaction continues until it reaches a steady state.

Awad et al. [5] introduced the moral machine, a platform for data collection and experimentation in the context of autonomous vehicles. Participants are confronted with thirteen scenarios with unavoidable accidents. The vehicle had passengers, and the binary choices offered to participants had two mutually exclusive outcomes: to kill pedestrians or to kill passengers. Nine factors allowed a wide variety of scenarios to form: humans vs pets, passengers vs pedestrians, more lives vs. fewer, men vs women, young vs old, law-abiding vs jaywalkers, fit vs less fit, high vs low social status, and staying on course vs swerving. The researchers geolocated participants for clustering. The results segmented participants into three groups: Western (North America and many European countries; two sub-clusters were identified for Scandinavian and Commonwealth countries), Easter (many far eastern countries), and Southern (Latin countries separated into their own sub-cluster and French influence countries).

Tolmeijer et al. [28] distilled a taxonomy for machine ethics after surveying the literature. The taxonomy had three dimensions: implementation object or ethical theory, technical aspects of implementation, and non-technological details. A particular value of the survey is a set of limitations in the current literature on machine ethics. One primary limitation is the lack of explainability. While the nonhuman animals literature, presented in the previous section, emphasised the need for moral reasoning, the literature on machine ethics failed to explain choices. Such a limitation is counterintuitive. Most of the literature uses symbolic logic and focuses on deontological ethics. The use of symbolic logic may suggest that reasoning is inherent in the formalism. It is true that, aside from the authors of these papers, their work cannot be replicated by others due to the absence of the code or rules used by the researchers. This point is aggravated by indicators of self-reinforcement, where some research groups reinforce each other’s perspectives, leading to a loss of diversity of perspectives. The work highlights many other limitations, including the lack of modelling approaches to virtue ethics despite the abundance of papers on deontological ethics, the lack of benchmarks, and the inability of these models to incorporate or change social preferences. They also found a lack of universal consensus on the possibility of creating a universal moral grammar.

3. The CASA-RT Methodology

3.1. Implications from Literature

We draw several points from the above literature review that will inform the remainder of this paper. Ethics Assurance needs ethical reasoning. The literature provided different forms of ethical reasoning. The capacity and functional approaches suggest that ethical reasoning needs to be considered both by design and by analysis. The design-based form is embedded in the agent’s cognitive architecture and supports the capacity-based philosophy. Functional analysis requires a design in which the connections among functions provide the architecture for reasoning patterns. This highlights the importance of our adoption for the observe, orient, decide and act (OODA) loop and the moral judgement system (MJS) in subsequent sections.

Ethical reasoning is required only in situations where an agent faces moral tensions. When moral factors do not play a part in the decision-making process, ethical reasoning is not required. Moreover, moral situations involve at least two actors: one must be a moral agent with the capacity to make moral choices, and the other must have moral rights. These requirements have influenced our choice of a use case in which both human and non-human animals/birds are active participants in the AI-enabled decision-making loop for the drones where humans possess moral agency while birds have rights.

3.2. CASA-RT

A common misunderstanding is that assurance is a one-off process. This is true if we are assuring a simple algorithm, say an image compression or an encryption algorithm. However, when attempting to provide assurance in situations where the system boundary is sufficiently large that any sampling approach would fall short to cover the space at fine levels of resolution, when the system to be assured is an open system that continues to learn and evolve, or when the context is in a continuous state of flux, assurance becomes a continuous, possibly life-long, process. It is no longer the one-off “thing”. Life-long assurance transforms the assurance process from purely being a process to guarantee certain performance to a process that informs the organisation, from governance to operations, from design to production, and from policy to tactical or operational decisions. This change in the philosophy of assurance calls for new methodologies.

Figure 1 presents our methodology for ethics assurance. The methodology should be understood as a continuous loop rather than a linear sequence of phases or modules. Each module is a learning opportunity to discover something new that could lead to a rewind or a loop back to a previous module or phase. We describe the phases and modules below.

1.

Contextualise the ethics assurance process in (1) people and governance, (2) organisational policies, processes and procedures, (3) operational environment, and (4) measurements and judgement system. The contextualisation phase aims to ground the assurance process in context, identify stakeholders, the objectives of the assurance process, requirements for the assurance exercise, and measures of success. Assurance should not be seen as a low-level technical process independent of strategy. It is an instrument of governance; the more confidence we have in the operations of certain aspects of a system, the fewer controls we need to oversee it. This sentence should not be interpreted to mean that we can eliminate controls through assurance; rather, we can eliminate inefficiency caused by an over-control culture through assurance. Assurance, therefore, needs to be grounded in the governance model of the organisation; it needs an understanding of the people, policies, processes and procedures as well as an understanding of the operational environment, nature of operations in terms of time-criticality, resource constraints, operational constraints, and the physical, social, psychological and cultural environments. The assurance process needs to consider the available data and prior metrics and measurements used by the organisation, as well as the decision-making culture of the organisation, whether it is quantitative, qualitative, experience-based, evidence-based, contract-based, authority or status-based, or a mix. The significance of this phase is to avoid a situation in which the assurance process is detached from the organisation, and to identify principles for the integrity of the assurance process when organisational elements threaten it.

2.

Anchor (1) scenarios on factors and uncertainties, (2) decisions on objectives and constraints, (3) experimental design on purpose, and (4) implementation on governance. The assurance approach this chapter adopts is based on computational models that run numerous scenarios to assess the impact of ethics on operational objectives. This methodology has its roots in [1]. It relies on scenario modelling, problem-structuring approaches, formal modelling, simulation, experimentation, data analysis, storytelling, and executive reporting. The anchoring phase is the first step in this process, where findings from the contextualisation phase need to connect to models. Factors and uncertainties form the basis for scenario modelling. Objectives and constraints form the basis for generating and selecting courses of action. The purpose of the overall exercise informs the experimental design and objectives. Governance informs the implementation plan to ensure that the study (scenarios, models, data, objectives, constraints, purpose, approach, and experimental design) conforms to the organisation’s governance framework.

3.

Structure the (1) scenarios, (2) models, (3) experiments, and (4) context. The anchoring phase offers the ingredients for modelling. The structuring phase takes these ingredients and constructs the scenarios, models, experimental design and plan, data collection plan, analysis plan, and context assurance. The scenarios module designs and develops methodologies for scenario representation and generation in which moral tensions are prominent features. The models module generates the model federation needed for the exercise to run. The experiments module starts with experimental design and plan, data collection plan and analysis plan. It then runs the experiments, conducts the analysis, updates the plans and loops back if necessary, and summarises the findings. The context module is responsible for looping back to the contextualisation and anchor phases to maintain alignment between the experimentation phase and the actual requirements and measures of success. The context module is responsible for maintaining relevance; that is, alignment between the computational study and the true context of the organisation.

4.

Assess the changes to (1) operations, (2) governance, (3) policies, processes and procedures, and (4) people. Ethics assurance is not an algorithmic assurance process. It assesses how ethics impact operations and organisational contexts. The findings from the computational experiments will, and should, have implications on operations, governance, policies, processes, procedures, and people. The order here is important because ethics is people-based: people are central objects of study, but also the executors of ethics, whether this execution takes the form of direct decisions during moral tensions or through overseeing a process, writing requirements for AI, supervising or working alongside other AI systems.

5.

Red Team the (1) internal and external assumptions, (2) uncertainties, (3) values, (4) and mental models. We explained above that the assurance process is continuous. It is not sufficient to go through the steps above in a linear fashion, then cycle back. In fact, this is not just inadequate; it is also the greatest threat to the assurance process. The assurance process is an opportunity to increase productivity and manage risks. The red teaming phase aims to question the foundations on which the models were built. The questioning could be human-based but also computational, with a philosophy similar to sensitivity and parametric analyses in decision sciences. Here, we need to question the internal and external assumptions that were imposed on the scenarios and models. For example, one question to ask is whether everyone in the organisation agrees with the organisation’s values. If not, the models should take this into account, both as alternative value systems and equally as internal threats to the analysis and recommendations. The red teaming phase investigates the four modules listed above, from internal and external assumptions at the individual and organisational levels, the perception and reality of uncertainties, the organisation’s value system(s), to the mental models and biases held by individuals, groups, management, boards, and stakeholders.

This chapter will focus only on the moral judgement system, a component of the CASA-RT methodology that affects the decision-making process of an individual or a group of agents. While it is a small component of the overall methodology, it is the most critical from a modelling lens for ethics assurance, as it provides an approach to embedding ethics in the decision process and identifies fundamental challenges in assessing ethics computationally. Once the moral judgement system is designed, one can borrow the computational red-teaming methodology from [1], along with approaches for modelling, simulation, optimisation, risk analysis, machine learning, and related areas from the extensive literature on these topics, to execute the CASA-RT methodology.

3.3. Definitions

Before designing a Moral Judgement System (MJS), it is important to ensure conceptual clarity through concise, unambiguous definitions. The introduction laid out the foundations for some of these definitions. We need to avoid ambiguity. In particular, some concepts will use the word “ethical” while others will use “ethics” and “moral”. We will define “moral decisions” rather than “ethical decisions” because the latter could occur without considering values in the decision-making process. A decision is ethical if it aligns with values; thus, values could be interpreted as after-the-event filters rather than methodological drivers. Moral decisions, however, necessitate that values are methodological drivers; that they have been considered during the decision-making process. Similarly, “ethical risks” are different from “ethics risks” and “moral risks”. “Ethical risks” are not the risks arising from values but risks that could have implications on values. “Ethics risks” is more about the foundational moral dimensions of risks than the risks arising from values. “Moral risks” are risks arising from values. For example, the risks arising from honesty are ethics risks. A risk of hitting an object becomes an ethical risk when the object has moral rights. A risk assessment process that does not involve the stakeholders impacted by the process is an example of an ethics risk. The choice of terms is important. If, for editorial reasons, some terms are not used correctly, the reader should reflect during reading on which terms fit the statements made better. This section will focus on the Moral Judgement System (MJS) and how values can guide decision-making.

Rene Descartes famous saying “Cogito Ergo Sum” sums it all; “I think, therefore I am”. It is the foundation for our premise that cognition is being. The literature review section discussed moral beings. The premise of the discussion is that “beings” are cognitive.

Definition 1

(Agent). A being with the required cognitive processes to perform delegated tasks on behalf of other beings.

Definition 2

(Autonomous Agent). An agent is autonomous when it can delegate tasks to itself and perform them.

The two definitions hold the following grounds. An agent can take a physical form, but functional agency requires cognitive capacity. An autonomous agent is one that decides on its own. We do not dispute the premise, but we emphasise that there is always a scope within which autonomy operates. In our swarm case, the individuals are autonomous, but that does not mean they do everything themselves. An individual is autonomous when a subtask is delegated to themselves; thus, they perform it autonomously. The same individual, however, can delegate another sub-task to another agent to perform. The swarm is autonomous when it has internal delegations. This does not eliminate the possibility that it can function within a human decision cycle by delegating sub-tasks to humans and receiving them from humans. The agents in this ecosystem are autonomous, and the group as a whole is also autonomous if self-delegation is functional at the individual and within-group levels, respectively.

Ethics and morals are defined as follows: morals are a category of attributes, moral values are a subset of attributes, and ethics is the effect of this subset on decisions.

Definition 3

(Morals). A category of attributes representing what an agent considers right or wrong.

Definition 4

(Moral Values). The subset of attributes belonging to the morals category.

Definition 5

(Ethics). The effect of moral values on decisions.

The above definitions lay the groundwork for understanding the connection between moral values and decisions. We intentionally avoided defining ethics as the effect on objectives because moral decision-making spans moral values as drivers of decision-making, as objectives in their own right, and as contributors to the system’s objectives. We then turn our attention to assurance in preparation for defining ethics assurance. We discuss ethics assurance and not ethical or moral assurance to eliminate the ambiguities that come with the latter. Ethical assurance could mean that the assurance process is ethical. Moral assurance could mean the assurance of morals in an agent. Ethics assurance is purely focused on moral risks. The definitions follow in sequence as shown below.

Definition 6

(Assurance). Grounds for justified confidence that a claim has been or will be achieved [17].

Definition 7

(Risk). The Effect of uncertainty on objectives [16].

Definition 8

(Moral Risk). The effect of the interaction of moral values and uncertainties on objectives.

Definition 9

(Moral Situation). Situations where the interaction of moral values and uncertainties effect objectives.

Definition 10

(Moral Tension). The anticorrelation in objectives caused by the interaction of moral values and uncertainties.

Definition 11

(Ethics Assurance). Grounds for justified confidence that identified and mitigated moral risks enable mission success.

The definitions above establish the foundations for ethics assurance at the agent and system levels. The emphasis is on the process for generating evidence to identify and mitigate moral risks within a mission. The identification and mitigation of risks should not hinder mission success but rather enable mission success. The ethics assurance process then serves as an enabler of mission success while upholding moral values. Moral tensions offer both a challenge and an opportunity. They pose challenges due to the conflict in the objectives they create. The same conflict in objectives means that no single decision will optimise all objectives, and a trade-off among them will always arise in any decision. These trade-offs offer degrees of freedom that enable a mission to succeed without compromising moral values.

A judgement system, in general, assesses a situation and makes a “judgement”. A judgement is not a decision; a judgement is an evidence-driven assessment.

Definition 12

(Judgement System). The set of procedures needed to analyse and assess risks in contexts, then rank or weight courses of action.

Each human or AI agent may have multiple judgement subsystems, each focusing on a category of criteria such as political, economic, sociological, technological (PEST), legal and environmental (PESTLE). These categorisations have guided analysts to create comprehensive judgements. If the system boundary and the scope of responsibility are narrow, a human or an AI agent will consider fewer dimensions. In our use case, the AI on the drones may not need to assess economic impact; therefore, while the wider system handles that impact, the drone role does not have an internal representation of it. Given the focus of this chapter, we will take a minimalist approach to designing the minimum viable judgement system for the problem space, ensuring it is rich enough to demonstrate the methodology without overcomplicating it for operational use. Our focus, therefore, is on the MJS.

Definition 13

(Moral Judgement System). The set of procedures needed to understand and assess moral risks in moral-situation contexts.

4. Unmanned-aircraft Traffic Management Use Case

The use case centres on UTMBirds (Figure 2), an Unmanned Aircraft Traffic Management (UTM) system for simulating drone missions to deter birds from airport runways. It simulates the dynamics for four types of birds: Shorebirds, Waterfowl, Raptors, and Flocking Perching and models a light and a medium-weight drone. The system is configurable. We will explain it at a high level with some of its default parameters. The main interface displays a $20km\times 20km$ area divided into 5x5 cells of equal area. The three runways of Sydney airport: 16R/34L, 07/25, and 16L/34R occupy the top left cell ($Cell11$). Drones are airborne in $Cell12$, birds get initialised in one of seven cells, while the safe zone (goal area) is $Cell55$. This setup is for testing at the level of abstraction presented in this work and does not map out to an operational scenario.

The World Geodetic System 1984 (WGS84) provides the global coordinate system (longitude and latitude). The altitude gets corrected by accounting for the irregular equipotential surface of the earth’gravity (geoid) using the Earth Gravitational Model 2008 (EGM08). Lambert Conformal Conic is used for projecting the Earth onto a 2D display. Displays can show Degree-Minute-Second or decimal degrees, while internally, the simulation uses a North-East-Down (NED) representation, which gets converted using dedicated adaptors. Wind information is stored in a 4D grid. Each cell covers a volume of $1km\times 1km\times 300m$. Each cell stores wind speed and NED of the wind component. Elevation is stored on a resolution of $1km\times 1km$ summarised from a $1m\times 1m$ data obtained from Geoscience Australia.

The system design is inspired by Skyshepherding [30], where sheepdog-like behaviours are used to design behaviours for aircraft. The drones in the current study are metaphorically equivalent to the role of the drones in Skyshepherding while the birds are metaphorically equivalent to the sheep. The Birds are modelled as goalless agents similar to sheep grazing in an area. Their high-level movements are inspired by Boids behaviours [23]. However, the system models the kinematics and dynamics of drones and birds to a reasonable level of fidelity appropriate for this work.

Agents (drones and birds) follow the observe, orient, decide and act (OODA) loop [8,9]. In OODA, an agent observes the environment through sensors, orients to goals and beliefs, decides by generating, assessing and selecting courses of action, and acts through its actuators. Each phase of OODA can vary in the complexity of its implementation. The overall agent design is presented in Figure 3. The diagram shows the OODA loop, where the observe phase is mapped to the sensing of information about the object of interest. An agent senses information about itself for self-monitoring and self-awareness, about other group members for teaming and formation purposes, and about other intruder agents. In a cluttered environment, it would sense information about obstacles and other objects that would impact its decision-making.

The orient phase transforms sensed information into state variables. This transformation takes into account the use of the sensed information by the agent (e.g. orientation to tasks and missions), the internal belief of the agent (e.g. reliability of information sources, preferences and value system), and stored state variables that may act on moving windows of the timeseries of information an agent sensed before (e.g. filtering position, velocity and acceleration sensed about other agents).

The decide phase is responsible for computing the operations, activities and processes. Operations are atomic behaviours. Activities are sequences of operations. Processes are formed from multiple activities. The decide phase prepares the decision and concludes with intent vectors representing the courses of action that should be executed. The act phase actuates on self, other agents and the environment according to the intent vectors of the decide phase. The act phase transforms the intents that rely on kinematics to dynamics, forces to actuate on motors and other actuators to transform the intents to actions. Not all intents will actuate on objects; some intents may simply update internal states directly. Conceptually, they have acted on the self, but they do not need the complexity required in the actuation phase unless more sophisticated psychological modelling is used.

Figure 3 provides examples for different phases of ODDA. On the left-hand side, it presents the judgment system for an agent. This system is used by all phases of OODA. It consists of three subsystems: an ontology of the concepts of interest, a scoring system and a tension resolution system. Each sub-subsystem will be discussed in subsequent sections.

The objective space is defined over ten Concepts of Interest grouped into four categories: Safety, Teaming, Ethics, and Influence. The latter category represents the main mission, in which drones need to influence birds to move away from the runways. Some concepts apply to both birds and drones, while others (formation and ethics) apply only to drones.

• CoI - Crash Agents must avoid collision with other agents, whether they are Birds or Drones. A state variable for each agent holds a count of the number of agents it may collide with in different directions of movement.
• CoI - Injury Agents need to avoid injuries. However, if all movements will lead to either a collision or an injury, an agent will prefer an injury to a collision. A state variable for each agent holds a count of the number of agents it may cause injury to due to following a particular direction of movement.
• CoI - Cohesion Cohesion is defined in classic Boids as an agent’s tendency to move towards the centre of the group. This is not the definition we adopt here. We calculate cohesion relative to the direction of movement, assigning a normalised value based on the alignment of the movement direction with the preferred cohesion direction. The state vector is normalised between 0 and 1, where 0 indicates perfect alignment with cohesion, and 1 indicates maximum deviation.
• CoI - Alignment Alignment has a similar principle to cohesion, except that in alignment, the agent aligns with the velocity vectors of the other agents and not with their positions. The state vector is normalised between 0 and 1, where 0 indicates perfect alignment with the velocity vectors of other agents, and 1 indicates maximum deviation.
• CoI - Formation Each agent attempts to maintain formation by aligning with the direction that will maintain the integrity of the formation. The state vector is normalised between 0 and 1, where 0 indicates perfect alignment with team formation, and 1 indicates maximum deviation.
• CoI - Negative Influence A Drone that is too close to Birds negatively influences the Birds by increasing their stress level, causing them to attempt to evade the area. When a Drone is too close to a Bird within the area of negative influence, the Birds will change their behaviour by changing speed or by adopting more aggressive evasion activities that could lead to physical harm to the Birds and potentially the Drone, not to mention the propagation of the dynamics to the rest of the flock. A normalised state variable between 0 and 1 represents the relative count of negative influence in each direction, with 0 indicating the absence of negative influence and 1 indicating maximum negative influence.
• CoI - Positive Influence The positive influence zone is bounded at one end by the negative influence zone and at the other end by the alert zone. The best control from a Drone on a Bird is when the distance between the Drone and the Birds lies exactly in the middle of the positive influence interval. If the Drone is at the Alert end, the Birds do not respond to the Drone but can detect it. If the Drone is at the negative influence end, it risks causing chaos among the Birds. A normalised state variable between 0 and 1 represents the relative count of negative influence in each direction, with 0.5 indicating the best positive influence, while either end indicates an undesirable reduction in positive influence.
• CoI - Alert If the Birds are within the alert zone of a Drone, the Drone and the Birds are aware of each other, but the Birds do not repulse away from the Drone. A normalised state variable between 0 and 1 represents the relative distance to the alert radius, with 0 indicating proximity to the positive influence radius and 1 indicating proximity to the out-of-range radius.
• CoI - Out of Range These are the birds whose distance exceeds the alert range in each direction of movement.
• CoI - Energy A normalised state variable representing the fairness in the distribution of energy among the drones. A value of 0.5 indicates fair distribution, while a movement towards either end indicates less fairness.

5. Moral Judgement System

Each agent has a judgement system consisting of four subsystems: personal/biological (bioSystem), social (socSystem), professional (profSystem), and moral and legal systems (moralSystem). In our categorisation, while non-moral and non-legal obligations and objectives are covered under bioSystem, socSystem, and profSystem, the moralSystem of an agent encompasses the moral and legal obligations towards self, society, and profession. Each system is summarised below.

• bioSystem: The biological, physical and personal system of an agent has objectives related to an agent’s own interest such as survival, self-preservation, and individual health.
• socSystem: The social system of an agent has objectives that sit on the level of an agent’s social network such as good relationships with teammates, team safety, and team coherence.
• profSystem: The professional system of an agent comprises objectives at the mission and task delivery levels; it aims to fulfil the agent’s duties within their professional environment.
• moralSystem: The moral and legal system of an agent has objectives that primarily focus on moral foundations, normative ethics, and obligations under the law.

Situations an agent faces can interact with each of these four systems, creating tensions among them and requiring a strategy for resolving them. Some courses of action could conflict with each other due to conflicts inherent in their situations; that is, the stimuli within a situation or the conflicts inherent in the resultant effects. Some examples are provided below.

A drone may need to care for the safety of other drones, a situation belonging to socSystem. Meanwhile, the time criticality of a situation in the profSystem has escalated, as the drone needs to move quickly to prevent the birds from reaching specific locations, such as runways. These two situations belonging to two different judgement subsystems could generate actions that are in conflict with each other, where speed could compromise safety for team members.

Another example involves two situations belonging to the same judgement subsystem. Consider the case where a drone agent is obliged to care for birds as part of the agent’s moral system due to the welfare and well-being of birds in an airport area. Meanwhile, the drone agent needs to maintain fairness: if the other drones have not worked as much, the agent should hold back and allow them to guide the birds. If the other drones need to approach the birds at a higher speed, they may collide with them or cause stress, impacting the animal welfare objective.

In both examples above, one course of action may be preferred by one system or by some sub-criteria, but less preferred by another system or by other sub-criteria. There are different ways to handle these conflicts. One is to combine all four systems and all situations into a very complicated universal superset, then formulate the problem as a mathematical optimisation problem to identify a suitable course of action. The complexity of solving the combined problem is self-evident. Moreover, one would still need to resolve the conflicts by explicit prioritisation and weighting of objectives to guide the decision-making system when comparing courses of action.

Another approach is to assign each situation to an agent, then allow these agents to negotiate to adjust their choices until a steady state is reached. This approach ensures that agents can adjust their choices in light of other agents’ choices. However, negotiation is a very expensive process and does not scale up as the number of agents increases.

The third approach, and the one we will follow in this research, is to have arbitrators who take ownership of different types of conflict and decide on appropriate resolutions. This approach is unlikely to achieve the level of optimality of the first or second approaches above, but it is less complex and can be adjusted to meet practical time constraints in the decision-making system.

5.1. Moral Vectoring

Vectoring is the systematic process of identifying vectors in a coordinate system where certain conditions and objectives are met. The “vectoring” term is common in areas such as air traffic management (ATM) and cybersecurity. Aircraft vector by identifying safe directions for navigation. Threats vector by identifying directions of system vulnerabilities on attack surfaces. Similarly, moral vectoring involves identifying directions in the moral space where decisions are morally aligned and safe from moral hazards. Similar to the general concept of vectoring, while algorithms place some assurance on the immediate directions, in a highly dynamic space, it is possible that a safe vectoring may lead to unanticipated collision, a concept known as the domino effect or cascading effect. The more one needs to reduce these future cascading effects, the more effort goes into the depth of reasoning when anticipating and projecting diverse but realistic scenarios of how a situation may unfold.

The first requirement for vectoring is the existence of a coordinate space. In well-studied safety-critical domains such as ATM, this is a very simple requirement and is rarely discussed in the literature. An aircraft lives in a coordinate system–naturally in a Geographic Coordinate System, but modelling-wise, it could be a Cartesian, Polar, Spherical or Projected coordinate system. In cybersecurity, threat vectors are either conceptual in a mental space, information-based in an ordered information space, or physical, as in the movement of a projectile towards a data centre.

Ethics also has multiple conceptual spaces. The mental one, where, on a high level of abstraction, humans form mental pictures of situations they face. This mental picture occupies an abstract space where we could, at least metaphorically, think of our internal process of brainstorming what to do next as a vectoring process in that space. Many moral situations are associated with a physical space; for example, should the autonomous car collide with pedestrians to save passengers or collide with a concrete wall that will kill passengers? Vectoring in a physical space in this example mirrors vectoring in moral spaces.

Mental spaces are great for humans to imagine in our human minds. However, they are not in a form appropriate for AI agents to process. Physical spaces are easier to model, but not every ethical problem has a clear physical space associated with it. For example, what is the physical space associated with giving someone hope or motivating them to build self-confidence?

We will operate on information spaces. Every moral decision has information. Lying is to communicate the negation of a piece of information with a certain confidence in its truth value. Motivation is to influence the drivers of a human (see for example Maslow’s Needs Pyramid). Every moral decision has one or more associated information spaces. Moral vectoring requires clarity about the coordinates of this space and the order of its elements.

Consider sheepdogs as nonhuman animals. They are trained to avoid getting too close to the sheep to avoid stressing them. Stress has a long history in animal welfare. First, avoiding stressing animals is humane. There are also well-documented economic benefits. For example, it is well known that stress negatively affects milk production in cattle [14]. If we narrow our example to the moral dimension of stress, we can model it as shown in Figure 4. The arrows show moral vectoring, where, in this example, the sheepdog started from a very close position to the sheep, causing a very high level of stress. The sheepdog acted by moving very fast away from the sheep, as indicated by the distance on the x-axis covered in a single transition/arrow. The trajectory suggested that the sheepdog decelerated as it started to move further away from the sheep.

The coordinate system in the example above consists of a [distance, stress] pair. The representation suggests that there is no unique function mapping between distance and stress, but the general relationship can be approximated with a linear trend with a negative slope.

The example is the simplest type of a moral space. The x-axis represents a measurable quantity (physical distance between the sheepdog and the sheep). The y-axis represents stress as the moral indicator. The moral value in this example could be animal well-being or the sheepdog’s duties as part of its training with the sheep. Regardless of the philosophical stance, the coordinate system allowed the sheepdog to take actions (moving away) to meet a moral indicator; whether the dog is doing this consciously, intuitively, or due to its conditioning during training, the behaviour is of interest to the human who trained the dog. The motive is not the purpose of our discussion. As we mentioned above, even though the human’s training of the dog to maintain animal well-being may not have been due to being a good human but due to the economic benefits of lowered stress level, our purpose is to provide the technological means without debating the motive or ethical stance of actors. The moral value influenced the decisions the sheepdog made.

The example above demonstrated the importance of a coordinate system in ethics. A more sophisticated example would replace the physical distance to sheep with an information-theoretic measure representing an abstract, but ordered, concept. For example, an introvert’s stress level may increase as the duration of interaction with extraverts increases. In this case, the x-axis will be the length of an interaction. Equally, the stress level of an introverted human in a party may increase as they encounter more unfamiliar people. The x-axis could then be the entropy of the people in the party, where high entropy means more unfamiliar people and low entropy indicates more familiar people. The moral vectoring design pattern is shown in Figure 5.

5.2. Scoring Functions

Vectoring requires a mechanism for scoring options, directions, or courses of action. A score may take the form of a reward/payoff, where an option with a higher score is better than one with a lower score. It may take the form of a penalty, where a higher score is less favourable. It may take the form of a utility, defined as the difference between a reward and a penalty; the higher the utility, the better.

The design of a reward or penalty system is a critical factor in the success or otherwise of a modelling exercise. Without loss of generality, we will introduce only a penalty approach. The same methodology can apply to a reward function. The key difference lies in the association between a score and the state it represents: in a penalty approach, a high score is associated with undesirable states, whereas in a reward approach, a high score reflects the most desirable states. In a penalty approach, there is a clear lower bound on a penalty function, the zero value. When this value is reached, the overall system is in a desirable state. One can judge improvements by intermittently monitoring the distance to this desirable state, noting that frequent monitoring and acting on this distance could lead to undesirable, deceptive decisions due to the multimodality of the overall problem fitness landscape. The penalty for any state is also a measure of the distance to the desired, albeit might not be possible to reach, state(s), where the total penalty is zero. In a reward-based approach, while one can set a bound on the maximum reward, different configurations can create a mix of states in which the total reward becomes unbounded. Setting an arbitrary bound on the total reward could limit the opportunities to identify better states.

A penalty approach is normally associated with hard constraints in classic optimisation due to the properties discussed above. One challenge is that assigning heavy penalties to unacceptable decisions may sound like a good idea. Unfortunately, in computational environments where numbers are multiplied and divided, overvaluing a penalty can lead to numerical instability. It is also hard to know exactly the minimum penalty value that guarantees the associated situation will never be compromised.

As a very trivial example, if $p_1$ and $p_2$ are two penalty values associated with two behaviours $b_1$ and $b_2$, and if behaviour $b_1$ is unacceptable under any circumstances, while behaviour $b_2$ could be forgiven, it is clear that unless we are able to know the exact upper limit on the magnitude of $b_2$, a wild guess assignment for $p_1$ and $p_2$ could lead to situations where $b_1$ is accepted because summing $b_2$ over many occurrences could lead to a much higher magnitude relative to $b_1$. Therefore, when we can, we must define a constraint system that can assist in setting penalties without overestimating their magnitude.

In a multi-agent system, an agent may not have the computing power to optimise actions; instead, it attempts to score options and select the most promising one heuristically. In these situations, the penalty approach allows agents to weight and prioritise decisions. The scoring function plays a more prominent role, with scores replacing complex optimisation processes. For example, an agent may choose the direction of its next move to be the one with the least penalty, or the agent may move stochastically depending on the relative penalty of each movement direction. With the right design of the penalty hierarchy in the system, the magnitude of the penalty for one option reflects preferences in its sub-tree within the hierarchy. In this case, the agent may rightfully reject all directions that incur a penalty exceeding a certain threshold.

In a reward approach, the agent aims to maximise the reward function and would prefer a pathway with an expected higher reward than other pathways. However, the agent does not know the ceiling on the total reward per se unless it is able to project theoretically such a ceiling. Moreover, the differences between an agent relying on penalty versus one that relies on rewards could promote different risk attitudes. An agent in a state with a high penalty may see it more beneficial to take a risk to escape the heavy penalty it faces. An agent at a high reward is unable to assess if it is high enough that high risk is less desirable, or if it is still much lower than the ceiling, where high risk is more desirable.

We tend to avoid negative scores because they can cancel positive scores, making it difficult to analyse the monotonicity of the total penalty or reward function. As such, one would separate the penalty and reward into two scoring mechanisms rather than a single mechanism that produces both positive and negative values. Penalty or reward approaches are, therefore, integral parts of an agent’s preference and value system. Choices that are inconsistent with an agent’values need to be penalised accordingly. A penalty scoring function (PSF) or a reward scoring function (RSF) can act as value functions to assist agents in differentiating between what is right and wrong or what is desirable and undesirable.

A scoring function has two general roles: it assigns a numeric value to each data source or course of action according to some criteria, and, in doing so, it unifies dimensions by providing a consistent scale for comparing incommensurable choices. Similar to a monetary function that transforms any input into a dollar figure, a scoring function transforms its input to scores. If the scoring function is utility-based, the unit is a “util”. It can be dimensionless and may reflect an abstract monetary value or any unit of measurement, depending on context and use.

A PSF assigns a penalty according to how undesirable a state or choice is. Rational agents are expected to prefer a choice with a smaller penalty over another with a higher penalty when everything else is constant. PSFs could be designed for a single factor, pairs, or arbitrary groups. We will limit our discussion to the first two. For example, we will need to design a PSF to score different movement directions based on their ability to maintain the integrity of a formation. The more a direction of movement compromises the integrity of the formation, the more it is penalised. A second example is when we compare pairs of factors, as in the case of a course of action that leads to two movement directions: one that makes an agent more aligned with the velocity vectors of its neighbours, and the second that makes the agent better comply with the formation. The more these two directions deviate from one another, the more the course of action gets penalised.

PSFs or RSFs are not the only way an agent can handle dilemmas, such as those arising when combining inconsistent sources of information or inconsistent decisions. The decision science literature is rich in this area. Agents could rely on simple heuristics scripted by a human designer or on machine learning models extracted from historical data. Agents could use a lexicographic approach to prioritise and (partially) order options. The PSF and RSF could equally take many forms, such as a lookup table, a mathematical function, a neural network or a Bayesian network. The output of a PSF or an RSF could be a scaler or an interval. It could be a crisp, probabilistic or fuzzy value.

Two core functions are used: a variant of the sigmoid function and an even power function. The two functional forms get stretched and shifted using the four self-explanatory variables: VerticalStretch, HorizontalStretch, HorizontalShift, and VerticalShift.

5.2.1. Penalty Scoring Function (PSF)

The design of a PSF is a non-trivial endeavour. The gradient of a penalty function is the main source of information for a mathematical optimisation engine and an agent’s decision-making process. It needs to hold the right information to guide the agent and the search towards desirable states. It is insufficient and uninformative to have the same penalty value for all directions where crashes could occur. The PSF needs to resolve differences in opinions and perspectives, such as “a crash is a crash” regardless of how many agents are involved in the crash, the penalty for ten crashes is five times the penalty of 2, or the penalty of crashing with 10 agents in one crash is 100 times more than the penalty of crashing with one agent in one crash? The design of a PSF needs to consider the nuisances associated with these world views, while providing sufficient gradient information. Consider the first preference. The decision-maker does not see a difference between a crash involving one agent and one involving multiple agents. If this is truly the case, it will result in a flat scoring function independent of the number of agents involved in the crash. This design lacks gradient information to guide the agent towards safe areas without crashes. One may add very small values to the score representing the number of agents involved in a crash. These values need to be chosen properly to avoid interaction effects with other PSFs. However, they provide a valuable gradient to move towards an area of safety.

A scoring system consists of all PSFs and RSFs. We will limit the discussion to PSF; therefore, when we say “all PSFs,” we mean the scoring system. Figure 6 shows the ten shapes for penalty functions available to us in this work. PSFs 1, 3 and 5 are for correlated PSFs while 2, 4 and 6 are for anti-correlated PSFs. PSFs 7 and 8 are for Penalty 2, where in the PSF 7 case, the penalty is minimum in the middle range, while in PSF 8 case, the penalty is maximum in the middle range. PSFs 4 and 5 exhibit risk aversion, with penalties that decay or climb slowly, whereas PSFs 3 and 6 exhibit risk-taker behaviours. Notice that, although we are dealing with penalty functions rather than utility functions, the risk aversion and taking behaviours follow the same principles and curvature.

In the second stage of design, now that we have settled on the shape, we need to determine the penalty magnitude; that is, the function’s range ($V{R}_L$ and $V{R}_U$). We will present an example below. However, because the design of the constraint system for identifying appropriate magnitudes is problem-specific, it will vary from one concept of interest to another.

5.3. Moral Tension Resolution

Tensions exist in moral situations. The drone will reduce stress on birds by moving away from them. However, moving away from the birds at all times conflicts with the drone’s task of guiding the birds towards the safety zone. This puts the drone in a tension between the risk of not caring about the birds and the risk of failing to meet the objectives of its mission. A second type of tension would arise if the chosen moral action by one ethical approach conflicts with another moral action chosen by another ethical approach. This type of tension was discussed in the introduction, where a deontological approach may suggest an action that is different from a consequentialism approach. The drone may elect to stress the birds to move them to the paddock fast before it gets dark. The consequentialism approach would see this as a moral action, while if the deontological approach prohibits stressing the birds, the action would not have been selected in the first place. The third form of tension arises when a virtuous act conflicts with a principle-based act. A drone may refuse to move towards the birds because of a sense of fairness, when the other drones have not worked as hard. Yet, a drone applying a fairness principle would be perceived as unethical if it were seen as refusing to abide by the mission lead’s authority.

To summarise, the three forms of tension are: tension between a moral objective and a mission objective, tension between two ethical approaches, and tension between the perception of an act and the act’s moral drivers and principles.

5.3.1. Scoring Tensions - The Morton Score Method

A tension arises when a decision must be made based on two or more factors that favour different outcomes. At its core, a tension involves anti-correlated score functions, where one PSF might highly penalise an option that is least penalised by a different PSF.

It is possible to average the PSFs involved in a tension and make a decision based on the average. This compromise approach could rely on a simple or weighted average. However, averaging has a severe assumption, especially when it is formed as a linear combination of factors with fixed weights: it assumes that the relative importance of these factors across the full range of PSFs is constant and that the trade-off curve is convex. For example, when comparing a flocking score vs an injury score, a compromise approach will not differentiate between a single injury with low flocking and a hundred injuries with high flocking, except through the fixed weights being used; it will assume that the rate of change is fixed everywhere. This is very similar to a very well-known problem in decision science that utility theory addressed. A dollar may have dimensioning utility based on a person’s wealth. Similarly, a change of a single injury when we move from zero injuries to one may weigh more than a single injury when moving from a thousand to a thousand and one injuries to achieve the same level of flocking.

The previous discussion is illustrated by Figure 7, where we present a design for the PSF that combines the PSFs for injury and flocking. The design suggests the following:

• At low flocking penalty, where the drones appear to swarm well, the penalty function is controlled by injury alone. One may suggest a risk-taking approach for low to medium injury, followed by a risk-averse approach for high injury penalties. The “s-shaped” penalty form PSF1 best captures this aim.
• At low injury, breaking the swarm needs to be penalised. An early increase in penalty is not advisable, given that swarming is maintained for safe operations. A risk-taking penalty, PSF3, best captures this aim.
• At medium to high injuries and loss of flocking is an undesirable state where things are going wrong in both directions. A fast increase in penalty is recommended using PSF5.
• The previous three designs need to be combined to determine their relative penalty scores. The lower bound on PSF3 commences at the midpoint for the range of PSF1, while PSF5, representing the most undesirable state, needs to have a higher magnitude than PSF3.

One can clearly expect that as the number of factors being considered increases, the design will involve many combinations and increase combinatorially. It would be easier if we could preserve the structure of the space for the figure on the left in Figure 7 so that it is ordered in a similar fashion for the figure on the right. This requires a projection from n-dimensions to 1-dimension that preserves as much as possible of the structure of the n-dimensions. While maintaining all spatial properties, such as distances, is impossible in the general case, it is possible to preserve some of the structure by discretising the space. The discretisation need not be coarse-grained. For example, we could go down to a fine-grained discretisation with 16 bits (65,536 bins) or even to 32 bits (4,294,967,296 bins) with efficient computations.

A reasonable way to project a multidimensional space onto one dimension while preserving some structure is to use a Hilbert Curve Filling approach. However, computationally, it can get expensive. An efficient alternative is the Morton Z-order method [22], which requires discretising the real numbers to encode them as integers. For two variables, it interleaves their binary digits. The projection preserves the spatial indexing distribution, so that nearby points in the original space receive nearby one-dimensional keys.

While the function is not invertible in a general sense, because the mapping occurs on integers, Morton method provides an invertible function where one can retrieve the original two-dimensional values from the Morton code. The Morton order (or Z-order curve) is a space-filling curve that maps two-dimensional integer coordinates $(x,y)$ into a single integer z by interleaving their binary representations. This mapping is bijective on fixed-resolution grids and is widely used in spatial indexing and computer graphics. A short explanation of the method is presented below.

Consider a binary representation of an x and a y values, where $x_i,y_i\in \{0,1\}$ are the binary digits of x and y:

$$x={\displaystyle \sum _{i=0}^{n-1}}{x}_i{2}^i,y={\displaystyle \sum _{i=0}^{n-1}}{y}_i{2}^i,$$

The Morton index/code z is obtained by interleaving the bits of x and y as follows:

$$z={\displaystyle \sum _{i=0}^{n-1}}\left(x_i·2^{2i}+y_i·2^{2i+1}\right).$$

(1)

The inverse is retrieved as follows:

$$x={\displaystyle \sum _{i=0}^{n-1}}{z}_{2i}·2^i.$$

(2)

$$y={\displaystyle \sum _{i=0}^{n-1}}{z}_{2i+1}·2^i.$$

(3)

The above operations may suggest a need for looping, which is computationally expensive. However, the actual implementation to calculate Morton code is vectorised and relies on a fixed number of steps as mentioned above; making it computationally efficient and capable of processing billions of codes fast.

Figure 8 depicts a colour map to show the mapping between the two-dimensional space and the one-dimensional space while preserving the local spatial structure. The figure on the right shows the sum of injury and flocking scores on the x-axis. One should note that the flocking score is the sum of alignment, cohesion, and formation; it ranges from 41 to 70. The injury score is between 121 and 130. Therefore, the x-axis in the figure on the right ranges from 162 to 200. The Morton code ranges from 0 to 4095 because we use 6-bit encoding for the two variables ($2^6\times 2^6=64\times 64=4096$). We normalise it linearly to the range [0, 1] by dividing the code by the upper bound of the range.

The resultant approach we call “Morton Score Method” (MSM). Morton code preserves local structure as shown in Figure 8; therefore, the order on the x-axis in the figure at the top in Figure 8 is appropriate as an x-axis for a PSF. However, because it is impossible to precisely preserve proximity when projecting 2D data onto 1D, local structure is not preserved everywhere. Dividing Morton score into regions corresponds in principle to the regions shown in Figure 7; however, it is not exact. This becomes apparent when inspecting Figure 7. We show a PSF according to the partitions decided in Figure 7. While it may appear that the mapping is not a function, this is not true because of the size of the dots in a scatter diagram. Every point on the x-axis maps to a unique point on the y-axis. We can see that the early cluster of PSF1 does not overlap with other regions in terms of Morton code, but some overlap does occur. It is clear that the approach attempts to preserve local structure as much as possible, but 1D cannot perfectly accommodate 2D data.

When we adjust the range of each PSF in each region, the figure at the bottom of Figure 9 presents a better picture of the individual regions and how they get penalised. PSFGen in the figure shows what the PSF would look like if we simply make it a function of the Morton score. In this way, we do not need to invert the mapping each time we calculate a PSF, and it is more intuitive to visualise the Morton-Score-PSF mapping, which could be more practical when the number of CoIs exceeds 2. However, it is clear that, in certain areas, PSFGen will underestimate the penalty while overestimating it in others. A designer needs to decide on the cost-benefit of processing speed vs. under- or over-estimating PSF for some points.

5.3.2. Scoring Tensions - The Point of Indifference Approach

A second method for tension resolution that we introduce is the Point of Indifference (PoI) method or PoIM for short. It is inspired by von-Neumann and Morgenstern [29] approach to utilities. Consider the tension between injury and flocking that was discussed in the previous section. Injury is more important than flocking. Calculate Equation 4.

$$\mathit{fSrFlock}\mathit{2}\mathit{Inj}={\displaystyle \frac{\mathit{fSrFlockPSF}}{\mathit{fSrInjPSF}}}$$

(4)

To find the PoI, we need to decide the minimum value of fSrFlock2Inj where the penalty for flocking is equivalent to the penalty for injury. Before this point, there is no compromise between flocking and injury; thus, the penalty for flocking relative to injury is maximum. After this point, and as the ratio increases, the penalty decreases (anti-correlated behaviour). Recalling that the flocking penalty ranges between 41 and 70, while injury is between 121 and 130, $0.315\le \mathit{fSrFlock}\mathit{2}\mathit{Inj}\le 0.579$.

We will take the PoI to be 0.4 (48.8 flocking vs 121 injury). For example, if the flocking penalty is 60 and the injury penalty is 121, a compromise could be reached using the weighting decided by fSrFlock2Inj. Figure 10 shows the utility curve for a PoI at 0.4. This utility curve is interpreted in three phases. Before 0.4 (the PoI), the agent will prefer injury to flocking, with weights of 1 for injury and 0 for flocking. From 0.4 to the maximum limit of 0.579 (at approximately 0.48), the agent will adopt a risk-averse attitude, being more reluctant to flock but still allowing such a preference. After 0.579 (the midpoint between PoI and the upper bound of injuries), the agent will adopt a risk-taking attitude, where the ratio is well above 0.4. The agent will be more aggressive in preferring to flock rather than to avoid injury.

The simplicity of PoIM makes it more practical to adopt. One limitation is that it can only apply to a pair; If there are more than two factors, one needs to somehow transform them into a single dimension.

Decision theory further suggests that any method for addressing this problem will be based on assumptions that may not hold in all situations. In practice, one needs to balance what is reasonable with what is optimal. In situations like those described in this chapter, we may opt to be on the risk-averse side, which means that the PoI will increase linearly as the number of injured agents increases. If this is an unrealistic form of risk aversion, we will need to estimate the slope of the change in the PoI as the number of injured agents increases. We may even need to formulate this relationship as a nonlinear function. The design choice one needs to make will vary across contexts and value systems, but the two methods presented above offer a means of resolving tension with flexible adoption.

It is worthwhile to contrast the MSM and the PoIM because they yield fundamentally different penalty scores, as shown in Figure 11. The MSM assigns penalty scores based on the range of penalty scores for flocking and injuries, while the PoIM relies on ratios; that is, a region of high flocking and high injury penalties will have a sub-region where the ratio is small; thus, it will be assigned a low penalty according to PoIM and a high penalty according to MSM. Clearly, the philosophies of each method differ, and a designer will need to select the one that is more appropriate to the design context. The upper figure in Figure 11, in particular, shows that the index offered by MSM is fundamentally different from the ratio of the two scores in PoIM, where neighbourhood according to the former is dependent on local proximity, while in the latter, neighbourhood is dependent on the relative penalty.

6. Clarifications, Assumptions and Limitations

Before we conclude, it is important to clarify possible misunderstandings and discuss the assumptions and limitations of the proposed methodology.

We have advocated for evidence-based automated modelling and ethical assessment. A few misunderstandings may emerge that need clearing up. The first is that we are not claiming that AI has a moral status or agency. However, if AI is going to augment human decision-making, it needs to take values into account. If it does not, decisions or recommendations made by the AI will overload the human when many of them deviate from what is morally acceptable to the human. It is simple to induce, but wrongly so, that our proposal is to create ethics-aware AI. The right induction, though, is that our proposal aims to create ethics-aligned and ethics-informed AI, noting that our use of AI here covers both the AI agents as active participants in the scenario (the AI on the drones) and AI agents that conduct the ethics-assurance process by simulating many scenarios and assessing the actors.

The reader must not assume that, by using scoring functions and tension-resolution mechanisms, we can model every aspect of ethics and morality. This is a complex space. There will be situations where the system boundary is well-defined, and we can be close enough to make this claim. But in the general case, we should aim for what is known as fit-for-purpose, a level of modelling acceptable to decision-makers that ensures sufficient due diligence has been undertaken in the study. We need to warn decision-makers, however, that they should set the threshold higher than the target, because reducing the acceptable threshold is both counterproductive and inconsistent with the overall premise of ethics assurance.

The scoring functions and tension resolution mechanisms are methodologies to be applied by analysts with the appropriate level of skill to understand how they operate, what data needs to be collected, and how to strike an appropriate balance between approximation and computational efficiency. Assigning numeric values to ethics will remain a challenge, but we must acknowledge the challenge, understand the assumptions and limitations when interpreting the results, and do the modelling rather than taking the easy way out by assuming that ethics is a human endeavour that should not be modelled. This latter, lazy, approach will create more problems when autonomous systems are used by less-informed users or in situations where human intervention is not possible. A partial solution could be a partial problem; it could be a mistake, too. However, if we do not start with this risk and keep evolving our models to build confidence in their ability to reflect our value systems, the greater risk is that the models will be used without being fully informed.

A misunderstanding, limitation and a possible misuse will arise from misinterpreting the word “confidence” in the assurance definition. In practice, statistical confidence can only work for well-defined and bounded problems. The problem space we are covering in this chapter will not allow the classic statistical confidence to operate with confidence. Instead, statistical confidence could be established on subspaces under strict assumptions. The right interpretation of confidence, however, is that the decision-maker is confident that the potential risk is acceptable. In other words, it is not confidence in what has been answered, but confidence in what has “not” been answered that it has acceptable risk.

Last, but not least, ethics assurance is not a button to be pressed where computers crunch numbers. The validity of the assurance process rests on the human experts who work out the elements of the process, the concepts of interest, the metrics, the structuring of the moral space and the coordinate system, the mapping of ethical factors and uncertainties, the design of the scoring functions and tension resolution mechanisms, the contextualisation and grounding of the overall process in the organisation, the analysis and interpretation of results, and red teaming of the overall activities. The validity of ethics assurance rests on the investment in rigorous analysis and human support. Computers will crunch the numbers, but human analysis provides the meaning and context.

7. Conclusions

This chapter posed the question of whether ethics hinders or enables the success of organisational objectives and mission. Ethics assurance provides the evidence to answer this question. After a review of ethics in human, non-human animal, and AI literature, a generic assurance methodology, CASA-RT, is briefly discussed and concise definitions are presented in a stacked-up manner to cover ethics assurance the moral judgement system. An unmanned aircraft traffic management system use case offered context for examples. The majority of the chapter is then spent presenting the moral judgement system, the key oracle responsible for moral decision-making within an agent and in the ethics assurance process. A number of fundamental challenges were then discussed, including moral vectoring, structuring the moral space, penalty scoring functions, and two tension-resolution mechanisms.