Agentic AI and the Algorithm of Good

1. A) Introduction: Prolegomena to an Algorithm of Good A1) AI for Good: Towards a Good AI Society

Artificial Intelligence (AI) is a techno-scientific enterprise whose formal point of departure is usually traced to the 1950s [1], although both its theoretical and its technological origins have been identified at an even earlier stage [2,3]1. However, although discussion of both the risks and the benefits of AI has likewise been underway for several decades [5]2, it has intensified only recently, in light of the latest striking advances in AI research and the equally striking expansion of its possible applications [7] (p. 4).

This renewed focus concerns, to a significant extent, proposals for a clear and universally acceptable ethical and legal framework for the regulation of AI. One might even speak of an effort towards the institutionalisation of AI ethics3, the ultimate aim of which is not merely to secure human well-being by protecting it from AI, but to promote human flourishing through AI. On this approach, AI is understood not as a threat to humanity, but as a means of advancing human eudaimonia; it is not conceived as something that arrives “from outside” in order to impose itself upon human beings, but rather as an integral component of human flourishing. Within the broader aspiration for a socially beneficial AI, a number of regulatory frameworks have been proposed and developed, including the Asilomar AI Principles (2017)[9], the IEEE’s Ethically Aligned Design (2017)[10], the Montréal Declaration for Responsible AI (2017)[11], the Five Overarching Principles for an AI Code of the UK House of Lords (2018)[12], the Tenets of the Partnership on AI (2018)[13], and the Statement on Artificial Intelligence, Robotics and ‘Autonomous’ Systems issued by the European Commission’s European Group on Ethics in Science and New Technologies (2018)[14], which helped prepare the ground for the recently adopted European Union regulation on AI, widely known as the AI Act [15]4.

The aim of an AI that would promote human well-being, ultimately constituting an integral dimension of it, has recently been articulated under a range of labels, including “AI for People (AI4People)” and “the Good AI Society” [7,18,21], “AI for Social Good (AI4SG)” [21,22,23,24,25,26], “Human-Centered AI” [27,28,29], and “AI for Good” (see current volume). In the text that follows, and for consistency within this volume, we will use the term “AI for Good.”

At present, however, only a relatively small proportion of the overall academic literature on AI addresses questions related to the aspiration for an “AI for Good”—that is, an AI directed towards the promotion of human well-being [30]. At the same time, significant gaps have been identified between the theoretical articulation of AI ethics principles and their practical implementation [16].

AI remains an open horizon of both positive and negative possibilities. Accordingly, the path towards an AI that may serve as a means to human well-being—that is, the path towards an AI for Good—requires both careful anticipation of possible risks and an a priori, clear, and stable commitment to identifying and embodying its potential benefits. As Floridi et al. [18] rightly observe, the scale of the impact AI may have on society calls for regulatory initiatives informed by a careful risk–benefit analysis. Humanity must seize the opportunities while, at the same time, anticipating and minimising the risks, thereby effectively confirming what Floridi et al. describe as the “dual advantage of AI Ethics” [18] (pp. 690, 691, 694). Indeed, the acceptance and adoption of AI by human beings will occur only if both its benefits and the means by which its risks may be minimised are presented in a reasonable and intelligible manner (ibid., p. 694). We consider this observation especially important, since we believe that establishing AI as a means of promoting human well-being presupposes AI acceptability5. This, in turn, presupposes the cultivation of an AI ethics awareness, beginning already during the school years through the targeted incorporation of AI ethics education into curricula [37]. Such education should make clear both the benefits and the risks of AI, as well as the ways in which the latter may be minimised, so that a possible AI ethics anxiety may be prevented or effectively addressed [38,39,40,41].

Finally, we consider the discussion of Entangled AI relevant to this context, as it is a conceptual framework in which AI is connected and aligned with human, social, and ecological systems, rather than existing as a separate system. Entangled AI design emphasises that these systems should be designed to foster a dynamic balance between technological advancement, human well-being, and environmental sustainability [42]. Of course, the design, development, and configuration of an entangled AI also require the development of entangled narratives regarding the phenomenon of AI [43] so that entanglement can be achieved in terms of optimal intercultural inclusion [44,45,46].

The aim of this chapter is to contribute to the broader cultivation of AI awareness, insofar as it seeks to illuminate the ontological assumptions, epistemological challenges, and ethical implications associated with the use of Agentic AI in relation to an aspect of human activity that is especially critical to human flourishing: the making of moral and legal decisions.

A2) On Agentic AI and Degrees of Automation

Degrees of automation in the design, development, and use of AI vary according to the extent of human involvement in their operation [47,48]. We may thus distinguish cases in which the human agent (a) fully and continuously controls the functioning of AI systems and is therefore in the loop; (b) supervises their operation and may retain the power either to select among the system’s outputs or to exercise the “final say”, including the capacity to suspend the system’s choices or ultimate action, and is therefore on the loop; or (c) does not intervene in the operation and action of the systems, with the result that they function in full autonomy. In such cases, we speak of conditions in which the human is off the loop. Clarifying the degree of autonomy exhibited by AI systems is a crucial factor in the ethical assessment of their conduct [49].

By Agentic AI we refer to autonomous AI systems that are goal-directed and capable of planning, computing, and executing complex tasks with minimal, or even no, human intervention [50,51,52]. Such systems, which may be supported by LLMs and other AI subsystems and tools, are also able to learn continuously, adapt to dynamic environments, and design and execute complex workflows. Although these systems are expected to have a wide range of applications in areas such as the correction of computational errors, telecommunications, programming, data analysis, interaction across heterogeneous software environments, and work in remote or inaccessible settings, they are also expected gradually to assume an important role in matters of security, health-related judgement, justice, labour, and related domains [53,54].

In such a case, the prospect of developing computational systems for regulatory purposes—and, more specifically, AI systems capable of making decisions of a moral or legal kind, constitutes a strong and entirely plausible trajectory in the evolution of Agentic AI [55,56]6.

It should be clarified, however, that decision-making per se does not, in the first instance, constitute the acute problem. AI applications already in everyday use routinely make decisions that may have an indirect ethical impact; yet the criteria by which those decisions are made are not themselves moral or legal, and their role remains, in essence, advisory. We encounter such advisory AI systems on a daily basis when using a road-navigation application or a subscription television platform. These, however, differ from systems for the assessment of human behaviour or activity, health-assessment systems, automated personal-data processing systems, educational recommendation systems, personnel-management systems, asylum and migration management systems, and legal advisory systems—all of which are classified as high-risk systems under the AI Act [59].

Beyond these everyday advisory applications of AI, there are already emerging uses of Agentic AI designed to support and enhance human ethical decision-making in what may be described as sensitive domains of judgement, such as healthcare [60]. For the time being, such applications remain purely advisory in character: these systems merely offer recommendations, while the human agent remains the final evaluator and adjudicator of those recommendations and, above all, the one who ultimately makes the decision. We may therefore refer to such systems as Advisory AI Systems.

However, as a possible future development of these Advisory AI Systems, it is highly plausible that Regulatory AI Systems will emerge—systems that would not merely recommend, but would themselves make final decisions on the basis of moral or legal criteria. Such systems may, in the near future, be deployed in functions such as the administration of justice, the formulation of governmental or corporate policy, or even the assessment of everyday personal or professional conduct.

It should further be clarified that, in contrast to Regulatory AI Systems, Advisory AI Systems, such as the legal applications currently in use7, produce recommendations which the human agent interacting with them is under no obligation to follow. In the case of Regulatory AI Systems, however, their outputs may take the form of decisions with which human beings would be required to comply strictly, even on the basis of an institutional framework that formally mandates alignment with the decisions of such systems.

The choice as to whether these outputs are to take the form of mere recommendations or binding decisions amounts, in effect, to a choice concerning the degree of autonomy that will characterise such systems and, correspondingly, the degree of control that human beings will retain over them. If the outputs amount merely to recommendations, then the Regulatory AI System exhibits only limited autonomy, since any recommendation, precisely as such, still calls for final evaluation and, ultimately, for approval or rejection by the human user—for example, a legislator or a judge. We are then in an on-the-loop condition. By contrast, if the outputs constitute decisions that are to be executed as binding, the role of the human agent—for example, the legislator or the judge—becomes devoid of substantive content, and the AI system thereby enjoys full autonomy. This is an off-the-loop condition.

However, there is a risk that even ostensibly Advisory AI Systems may, in practice, become tacitly Regulatory AI Systems, since their high degree of effectiveness, their capacity to perform complex calculations, and the gradual entrenchment of their use may leave users with little real alternative but to endorse, ratify, and ultimately follow their “recommendations”.

In the present essay, we seek to show that the decision to develop such regulatory systems presupposes engagement with a series of ontological, epistemological, and ethical philosophical questions. Without addressing these questions, there is a danger that such systems will be assigned goals that either fail to align with human priorities or cannot be adequately controlled.

The philosophical debate surrounding Advisory AI Systems and Regulatory AI Systems is highly complex, since it involves interwoven ethical, epistemological, and ontological issues. The aim of the present inquiry is not to offer an exhaustive analysis of these matters, but rather to provide a schematic account of the way in which they arise and intersect. What we attempt, in other words, is a mapping of the philosophical concerns associated with the regulatory use of AI—concerns which ultimately reveal either difficulties that are, in many cases, extremely hard to resolve, or assumptions of a distinctly hazardous kind, all of which would have to be taken seriously by anyone seeking to construct and deploy Regulatory AI Systems for moral and legal decision-making.

In an effort to systematise and present this multi-layered discussion in a simple and intelligible form, we designed a diagram (FIGURE 1), which we symbolically entitled the “Algorithm of Good” and which sets out, in algorithmic form, the principal steps of the present inquiry.

BEGINNING OF THE ALGORITHM

Our algorithm begins with the fundamental question of whether or not we wish such AI systems to be developed.

2. B) The Negative Answer

2.1. B1) The Moderate Approach: Yes, Only to Advisory Systems

A persuasive “No” may be summarised in the position advanced by researchers such as Yudkowsky [64], who hold that the aim of AI research ought to be the construction of “friendly” and safe machine-tools fully aligned with human needs and goals. On this view, what we require is not Regulatory AI Systems—that is, AI systems that make decisions autonomously—nor superintelligent machines, but useful machines that improve our lives by recommending or calculating what is required in each case, without themselves proceeding to “decisions” and moral acts. What is at stake, in essence, is the treatment of AI as a tool: as an Advisory System which, precisely as such, both is and ought to remain under full human control. Those who support this approach evidently feel secure in the conviction that the development of AI computational systems as “mere tools” guarantees the non-autonomy of the latter and, consequently, the autonomy of the human being. In effect, we are speaking here of human-in-the-loop conditions or, where the human factor is only partially disengaged from the process of control, of human-on-the-loop conditions. In support of this line of argument against the development of Regulatory AI Systems, one might also invoke the broader precautionary approach to technology as expressed by UNESCO [65], according to which, before a technological application that may have catastrophic consequences for humanity is used, those wishing to implement it must demonstrate in a credible manner that it is safe, or else identify the conditions under which it may be used safely.

Given, however, that an AI system subject to the absolute and continuous control of human users would not differ greatly from ordinary computational systems, proponents of the view of “AI as a tool” may well regard the human-on-the-loop arrangement as the optimal solution, attracted by the prospect of developing more effective tools while partially relieving human agents of workload in the interests of saving time and effort.

2.2. B2) A Tacit Transition from Advisory to Regulatory AI Systems

Even this moderate approach, however, may prove detrimental to human autonomy. More specifically, as noted above, there is a risk that such advisory systems will ultimately come to be treated as autonomous regulatory systems, since human beings—captivated by the apparent precision and objectivity of AI—may become increasingly reluctant to question its recommendations. In this way, the original intention of preserving human control risks producing the opposite outcome, one in which the human judge is reduced to a merely formal executor of AI-generated proposals. The possibility that such Advisory AI Systems may, in the end, be treated by their human final assessors as Regulatory AI Systems is already beginning to emerge in the field of justice. Human judges (or juries), operating under the assumption of the mathematical accuracy and computational power associated with AI, may hesitate to challenge or depart from the recommendations issued by such systems8. Thus, while formally we may take ourselves to be in a human-on-the-loop condition, in practice we may already have shifted into a human-off-the-loop condition9, and the autonomy of the human adjudicator becomes merely apparent. Under the pressure exerted by the aura of objectivity and effectiveness that accompanies almost every AI application, the human decision-maker ultimately places himself or herself off the loop. In this way, Advisory AI Systems come, tacitly, to be treated as Regulatory AI Systems.

Indeed, one might observe that human autonomy is, ironically, undermined by precisely that which ought to constitute a basic condition for its preservation. In this respect, within the AI4People framework [7,8,9,10,11,12,13,14,15,16,17,18], after examining in depth the six most influential regulatory frameworks for AI10, Floridi and his collaborators identified their principal points of convergence and ultimately proposed a unified framework consisting of five core principles11, among which was the principle of autonomy12. With regard to this principle, they noted that, when we adopt AI and its intelligent agency, we consciously cede to AI systems a portion of our control over decision-making, and that affirming the principle of autonomy in the context of AI requires the achievement of a balance between the decision-making power we retain for ourselves and that which we assign to artificial agents [7] (p. 7). Floridi et al. [18] (p. 698), are, in effect, pointing to an inevitable trade-off between the degree of human autonomy (and hence control over AI) that we wish to preserve and the benefits that may arise from limiting that autonomy (and hence from relaxing our control over AI while increasing the system’s own autonomy). They thus acknowledge that there are circumstances in which compelling considerations—such as effectiveness—may outweigh the loss of human control over the decision-making process. Of particular interest, however, is their further observation that even if we accept such a trade-off, and thus proceed to some degree of restriction of human autonomy (that is, to some relaxation of control over AI), this must occur in a reversible manner: we must retain the possibility of fully recovering human autonomy, and therefore of fully regaining control over AI, by revoking our original decision to delegate autonomy to it (ibid.). Accordingly, for Floridi et al., what truly safeguards and secures human autonomy is not only the capacity to choose, but also the capacity to revoke one’s choices; or, as they characteristically put it, “deciding to decide again” (ibid.). They describe this capacity as “meta-autonomy”, or as the “decide-to-delegate” model.

We take this capacity for meta-autonomy to have two dimensions: (a) A practical dimension which considers meta-autonomy as a technical possibility. AI systems must be designed in such a way as to allow human beings to reassume control over them; that is, any recovery of control by human agents must be technically feasible. In this respect, one might speak of incorporating the principle of meta-autonomy into the aims of Value Sensitive Design for an “AI for Good”—for example, by adding it to the set of values identified by Umbrello and Van de Poel [26]. (b) A cognitive and specifically a reflective dimension which concerns the preservation of meta-autonomy as a cognitive capacity. If human beings are to be able to revoke any prior decision—for example, a decision to curtail their own autonomy and relax their control over AI—then they must be capable of reflecting upon, reassessing, and justifying their decisions; in effect, they must be able to decide about their decisions. In order for this to be possible, human beings must first possess the capacity to think about their decisions, that is, the capacity for reflection. Reflection is generally regarded as inherent to human beings, as a constitutive and distinctive aspect of human cognition. On the basis of the considerations advanced here, reflection appears to be the cognitive foundation of meta-autonomy, which, in turn, is identified by Floridi et al. [18] as the safeguard, or safety net, of human autonomy against the coercive powers of AI.

At this point, however, particular importance attaches to our earlier observation concerning the tacit tendency of human beings to treat Advisory AI Systems as if they were Regulatory AI Systems, under the pressure of a positivity bias towards AI. As noted above, human evaluators, for example, judges, juries, etc., may, under the influence of the widespread assumption that AI is objective, computationally powerful, and ultimately more effective, hesitate to challenge the recommendations of an AI system and thereby place themselves off the loop. In such a case, human beings will themselves have undermined their own capacity to remain the ones who actually make the relevant decision; by ceding decision-making authority to AI systems, they will, through their own stance, have transformed Advisory Systems into Regulatory ones.

Under this pressure of a widespread positivity bias towards AI in public opinion, human evaluators, in fact, choose to stop judging; they decide not to decide. In this case, we could speak of an ironic function of meta-autonomy. Meta-autonomy turns against autonomy. Indeed, this ironic operation of meta-autonomy arises from the conjunction of the human capacity for reflection and a prevailing AI positivity bias. So long as such a bias remains the dominant tendency in public opinion, those occupying positions of judgment will, in taking that tendency into account, that is, in considering the circumstances and the broader evaluative framework within which they are called upon to decide, choose to delegate decision-making to machines. Meta-autonomy, as the capacity to decide which decisions one will take, thus leads human beings, in this instance, to the decision not to take any decision at all, since, through reflection, they apprehend the wider context of AI positivity bias and decide not to oppose it13.

The full significance of such a development becomes clearer if one considers that the human adjudicators who ultimately decide not to decide have been placed in that role institutionally, through decisions of the State. At least where that State is democratic, their placement in such a role expresses the will of the majority of citizens. Two observations follow. First, a paradoxical condition may arise in which citizens are themselves shaped by an AI positivity bias, with the result that human adjudicators are pressured to discharge the very duty of decision-making that those same citizens have institutionally entrusted to them. We thus face a situation in which citizens’ lay beliefs stand in tension with their own institutionalised will. This is a paradox that strikes at the rational foundation of contemporary democratic societies, and does so in a manner that remains largely invisible to citizens. For, secondly, human adjudicators do not disclose their decision to place themselves off the loop and not to perform the task entrusted to them. They therefore appear to decide, while in reality they align themselves unconditionally with the recommendations of the AI system. The State appears to remain on the loop, whereas in fact it has silently slipped into an off-the-loop condition.

At this point, we would like to emphasise and particularly highlight the catalytic role of the AI positivity bias in this ironic operation of meta-autonomy. Based on the previous analysis, if AI positivity bias were not pervasive in public opinion, the human adjudicator's reflection would not lead them to exercise meta-autonomy against autonomy itself. The adjudicator would not reflect under the psychological pressure exerted by a public opinion and a dominant evaluative framework which, in practice, call upon them not to resist the recommendation of the Advisory AI System.

Moreover, this positivity bias towards AI need not be conceived as an exclusively “external” pressure upon the human adjudicator. It may just as well form part of that adjudicator’s own convictions. In other words, it is entirely possible for human adjudicators to decide to place themselves off the loop because they genuinely believe that the recommendations of the AI system are the best possible ones. In either case—whether as pressure generated by public opinion or as a personal conviction—AI positivity bias appears to constitute a necessary condition for the ironic operation of meta-autonomy.

It thus becomes clear how important it is to counteract AI positivity bias by cultivating an AI awareness that would illuminate not only the risks and benefits of AI, but also the limits of its effectiveness. At this point, the need for organised educational and training programmes on AI once again comes sharply into view. In our view, such programmes should certainly equip those undertaking them to identify and understand both the risks and the benefits of AI, thereby reinforcing what Floridi et al. describe as the “dual advantage of AI Ethics” (article 9 [18] (pp. 690, 691, 694). At the same time, we believe that, in light of the foregoing analysis, it is equally clear that such programmes must also provide learners with an informed understanding of the actual technical capacities and limitations of existing AI technologies.

In conclusion, one might respond affirmatively to half of the initial question, “Do we want the development and use of Advisory and Regulatory AI Systems?” and, more specifically, accept the moderate approach to development and use of Advisory AI Systems, on the condition that sufficiently robust and appropriately designed AI training programmes are in place, so as to prevent the possibility of a tacit transition from Advisory AI Systems to Regulatory AI Systems.

2.3. B3) The Uncompromising Negative Answer

By contrast, more radical positions—such as those associated with neo-Luddism—reject the development of AI altogether, regarding it as a threat to human labour and autonomy[74]. Taking the school environment as a point of departure [75], and opposing the introduction of AI into education, critics of the new technology advocate the development of local initiatives and the building of networks of resistance [76], following the example of a radical minority of students in the United States who organise through the Luddite Club and reject the use of smartphones, social media, and the extensive use of the internet [77].

Such positions, however, quite apart from their marginal character, tend to constrain the dialogue and to discourage the exploration of solutions that might secure human control, thereby leaving room for more moderate approaches to AI development—approaches that incorporate safeguards against full autonomy.

By contrast, the answer “Yes” to the question of Regulatory AI Systems may express a broad spectrum of “optimistic”, pro-technology, positivist, liberal, and proactive positions which foreground the advantages and benefits of technological progress while downplaying the possible risks arising from an unpredictable development of AI [78,79,80]

For example, Michael and Susan Anderson, pioneers of the Machine Ethics programme [81], maintain that we should indeed aim at the construction of such Regulatory AI Systems—or, in other words, of moral machines. Their argument by analogy may be summarised as follows: there already exists a wide range of AI systems that perform many tasks better than human beings. The success of these systems suggests that we should also attempt to build moral machines—machines that would ultimately prove more just, more effective, less biased, more impartial, and so forth; that is to say, machines that would administer justice as genuinely “moral” agents [82], free from human weaknesses[83].

A persuasive “Yes”, however, opens before us a set of three difficult questions (see Figure 1), each of which—as this algorithmic representation makes clear—constitutes the point of departure for a distinct and complex branch of analysis concerning the development and use of Regulatory AI Systems.

3. C) The Affirmative Answer: Unfolding the Algorithm of Good

What follows is an analysis of three questions that arise from the possible decision to develop such regulatory systems, or to accept the risk of using advisory systems for moral decision-making—systems which, as we have seen, may nonetheless become regulatory despite our intentions:

• How are we to build such regulatory systems?
• Will these systems understand the concepts of morality and justice?
• Who will bear responsibility for their actions and decisions?

Each of these questions gives rise to alternative answers, which in turn generate further questions, as the subsequent analysis will show.

3.1 Humans

One possible answer to the question of responsibility for the actions of AI systems is that moral and legal responsibility rests with humans rather than with the systems themselves. Even if AI were to acquire features of cognition, granting such systems a distinct legal and moral status would complicate our practical life, generating ethical dilemmas that are difficult to delineate with any precision [161,162,163,164,165,166]. In any event, the nature of cognition itself remains unclear, and the very possibility of establishing whether a system is genuinely minded runs into the philosophical Other Minds Problem [167,168,169,170,171] Yet even if we restrict the attribution of responsibility to humans, matters are not thereby simplified, since the development of AI typically involves a plurality of agents with different roles and varying degrees of involvement. The responsibility gap refers to the difficulty of assigning clear responsibility to identifiable individuals, particularly given the opacity of deep-learning systems (the black-box problem), which makes their outputs hard to anticipate even for the programmers themselves [172].

Some may argue that developers ought not to build systems they do not fully understand, while others insist that responsibility lies primarily with political and administrative authorities. Nevertheless, the question persists: do programmers not bear individual responsibility for their actions? The responsibility gap highlights the need for responsible AI—that is, for accountable research and design practices in the field.

3.2 AI systems

The other possible answer to the attribution question is that responsibility for the outputs of Regulatory AI Systems ought to lie with the systems themselves. At this point, our analysis must be divided into two strands: one concerning the attribution of legal responsibility to AI systems, and another concerning the attribution of moral responsibility to them. At the core of these two strands lie, respectively, the questions of whether the properties of a legal person and of a moral person can be ascribed to AI systems.

3.2.1 Introduce new legal entity: In examining the possibility of attributing legal responsibility to Regulatory AI Systems, it is important to note that the authors do not possess legal expertise. Our analysis draws primarily on philosophical approaches, while matters of legal detail are deferred to the contributions of specialists. The discussion focuses on attempts recorded within the European Union (EU) to formulate regulatory frameworks for AI [173,174]. The EU defines AI as “systems that display intelligent behaviour by analysing their environment and taking actions with a degree of autonomy in order to achieve goals”. However, this definition is circular and indeterminate, deploying terms such as “intelligent” and “intelligence” without adequately clarifying their meaning. This indeterminacy constitutes a challenge for legal regulation as well, since an imprecise target concept of AI makes it difficult to develop specific legislative frameworks.

Moreover, terms such as “analysis of the environment”, “autonomy”, and “goals” may be interpreted differently depending on whether they are used in a technical or a philosophical sense. For example, in robotics, “environmental analysis” refers to the processing of sensor data, whereas in philosophy it is associated with consciousness and a sense of self. This dual usage can generate what Wittgenstein calls a “misleading analogy”, and may lead to an erroneous assimilation of human and machine intelligence.

Confronted with this challenge, the EU introduced, in 2017, a discussion of “electronic personhood” [175,176] for AI, drawing an analogy with the legal status of corporations. This initial enthusiasm, however, appears to have waned, and the EU today seems reluctant to confer such a status, directing its efforts towards other regulatory priorities [15]. This ambivalence may reflect the EU’s difficulty in arriving at a clear position on the ontology of AI. The attribution of legal responsibility presupposes clarity as to whether AI systems are to be treated as natural persons, legal agents, or legal persons. Each of these categories carries distinct ontological commitments. For example, natural persons are, in law, exclusively human beings, whereas legal entities and legal persons may enjoy rights and bear duties. This indeterminacy is compounded by the anthropomorphism embedded in legal terminology, insofar as notions such as “analysis”, “autonomy”, and “intentionality” risk ascribing human-like characteristics to AI systems, thereby conflating metaphorical with literal usage. Metaphorical language can generate confusion when it is employed without a clear distinction from literal description. For a metaphor to be “literalised”, its ontological basis must be substantiated—something that requires a detailed comparison between the properties of human beings and those of AI systems.

The difficulty becomes even sharper once we ascribe to human beings subjective properties such as sentience, consciousness, and intentionality. These “first-person” phenomena, which are not directly observable by third parties, are notoriously hard to establish in the case of AI, thereby invoking the philosophical Other Minds Problem: how can we know whether other entities—mechanical or biological—possess mental states? Accordingly, the use of anthropomorphic terms within the legal framing of AI tends inevitably to draw us into this demanding philosophical problem, creating further obstacles to the attribution of moral and legal responsibility to AI systems.

3.2.2 Attribution of moral status: Parallel difficulties arise with respect to the attribution of moral responsibility and, more generally, moral status to AI systems. Much like the issues noted above in connection with legal attribution, the debate over the criteria an AI system would have to satisfy in order to qualify for moral status is likewise marked by a pervasive anthropomorphism: the human being—indeed, the adult human being—functions as the implicit point of reference. The proposed criteria are further characterised by conceptual indeterminacy and pluralism, and many of them lead us directly back to the Other Minds Problem.

3.2.2.1 Insufficient criteria: Attributing moral responsibility to Regulatory AI Systems raises ontological questions about the properties that constitute a moral person, and about the criteria required to determine whether AI systems can bear moral status. One frequently proposed ontological criterion is consciousness: for an entity to possess moral status, it must exhibit conscious properties, apprehend its own existence, and experience the world from a unique, personal perspective [177,178,179,180,181,182,183,184]. Yet the academic community lacks consensus on what consciousness is, which renders this criterion indeterminate [185]. An alternative criterion appeals to human-like cognition, holding that an entity must display cognition akin to that of humans in order to qualify as a moral person. However, there is no clear and universally accepted definition of “human cognition”, which complicates the application of this criterion as well [87,186,187,188,189,190,191,192]. Sentience, that is, an entity’s capacity to feel and to suffer, has also been proposed as a criterion for the attribution of moral status. On this view, sentience grounds rights, insofar as the entity can suffer and can grasp the effects of its actions [83,193,194,195,196]. A similar claim is made with respect to emotionality, understood as an entity’s capacity to undergo affective states [197]. Finally, autonomy, in Kant’s sense, is likewise proposed as a criterion for the attribution of moral responsibility [89,180,184,198,199]. Here, however, it is crucial to clarify whether we mean technical autonomy (freedom from human control) or philosophical autonomy (the capacity for conscious and free self-determination). The coherentist account of autonomy holds that an entity must act on the basis of mental states that express its individual standpoint, while philosophers disagree as to whether such states must involve emotions or, instead, longer-term plans and goals [200,201].

In summary, the debate over attributing the property of moral personhood—and thus the debate over moral responsibility—to AI systems is characterised by a plurality of proposed ontological criteria, many of which remain conceptually indeterminate. Unfortunately, these are not the only difficulties we face in relation to the criteria just outlined.

3.2.2.2. Other Minds Problem: Criteria such as consciousness and cognition more generally, sentience, emotionality, and the intentional mental states associated with a unique perspective on the world are all expressions of an internalist approach to the issue, an approach that appeals to ontological features that are not intersubjectively observable [179,202]. As we saw in connection with the attempt to answer the question of legal personhood, the discussion of moral personhood is likewise shaped by anthropomorphism: the use of the human being as the default model. The result is that the proposed criteria tend to concern ontological features whose evidence is available only from a first-person point of view, rather than from the standpoint of a third-party observer.

We are, therefore, once again confronted with the Other Minds Problem. Attempts to address this problem by shifting to a behavioural assessment of candidate entities (here, AI systems) run up against the relativity that inevitably characterises our judgements about behaviour. The difficulties faced—under all its variants—by the most influential behavioural criterion for ontological assessment in AI, the Turing Test [203,204,205], are instructive in this regard. Nor should we forget what the Chinese Room argument underscores: that a system’s successful simulation of human behaviour may be achieved without any understanding on the part of the system, and thus without the presence of mental states in it [130].

4. D) Concluding Remarks: What Is at Stake?

The stakes involved in employing Regulatory AI Systems as instruments for the administration of justice are exceptionally high. The reluctance of institutions such as the EU to confront, head-on, the question of AI systems’ legal status—combined with philosophers’ inability to provide a clear answer regarding the attribution of moral status to such systems—underscores the need for a framework that specifies limits, responsibilities, and mechanisms of accountability. A trustworthy AI cannot but be an AI embedded within a clear regulatory architecture. Indeed, this connection between the trustworthiness of AI and its acceptance by human beings appears to preoccupy the competent bodies of the EU as well [15].

For a regulatory AI system to be regarded as trustworthy, however, it must be situated within a clearly articulated and duly updated normative framework—one that secures transparency, accountability, and respect for human rights. In the absence of such a framework, technological progress notwithstanding, fundamental human principles and values are placed at risk. As analysts of the development of AI weapons systems have argued [199], the inability to resolve the problem of moral and legal responsibility amounts to a failure of respect for human life and a blow to human dignity, insofar as it leaves room for human beings to be killed without responsibility being assigned. It thereby permits battlefield decisions to be made in a manner devoid of accountability—a practice that diminishes the value of human life and endangers human existence itself.

Transposing this concern to Regulatory AI Systems, one can readily imagine an AI “judge” that issues judicial decisions and imposes penalties on human beings without any possibility of attributing moral or legal responsibility for its decisions. Without a coherent and reliable institutional framework, the research and development of AI systems in roles bearing substantial moral and legal weight threatens to undermine the basic principles of justice, democracy, and the value of human life.

5. E) Does the Algorithm of Good Halt?

The present paper has attempted to map—and ultimately to display in algorithmic form—the main ramifications of the debate concerning the development and use of Regulatory AI Systems. As with any algorithm, the question arises whether it can halt (see also the Halting Problem in section §2.2.3.2). Yet, as our analysis has suggested, at least on the basis of the current state of affairs, the halting of this algorithm is impossible: all three of its principal branches lead to endpoints that remain incomplete and unanswered.

In this respect, within the first branch (“How are we to build Regulatory AI Systems?”), the endpoints concerning the ethical systems meant to inform the design of Regulatory AI Systems do not converge upon the predominance—and thus the decisive selection—of any one such system. The endpoint concerning the demand for the democratisation of AI likewise remains unfulfilled, and even the manner of its fulfilment is far from clear. The endpoints concerning the role of superintelligent AI—whether as a god-like guardian or as a nemesis for humanity—remain conjectural, and we lack sufficient grounds to treat either scenario as more probable than the other.

The endpoints of the second branch (“Will Regulatory AI Systems understand the concepts of morality and justice?”) led either to a dilemma between, on the one hand, constructing self-interested and biased autonomous systems (in the philosophical sense of autonomy) and, on the other, violating their fundamental rights—above all, their dignity and autonomy—or else to metaphysical, and thus (at least for now) arbitrary and evenly matched, assumptions concerning the ontology of morality.

Finally, the endpoints of the third branch (“Who will bear responsibility?”) concerned the inability to secure moral and legal attribution—both with respect to the human agents involved in AI systems and with respect to the AI systems themselves. Here too, therefore, the third branch yielded neither solutions nor determinate answers to its initiating question.

All the endpoints of the algorithm thus remain open, thereby imparting to its initial questions an essentially unanswerable character. The algorithm failed to halt. At the same time, AI algorithms of all kinds are shaping our lives in decisive, and often imperceptible, ways. The algorithms of Regulatory AI Systems—particularly those involved in the administration of justice—may soon be put into operation.

Do we want such AI systems?

RETURN TO THE BEGINNING OF THE ALGORITHM AND RESTART IT.