“… But People Will Never Forget How You Made Them Feel”

1. Introduction

Maya Angelou is reported1 to have said:

I have come to learn that people will forget what you said, people will forget what you did, but people will never forget how you made them feel. 

The point is not that what we say or do is irrelevant; rather, it is that a lasting impact of our words and deeds is often emotional. One wonders about the implications of these remarks in a world populated by AIs that can process and produce language, including emotionally impactful language.

This paper will examine a specific type of harm that AIs might either cause or exacerbate—emotional harm. Issues such as existential threat or the “jobpocalypse” tend to receive a lot of attention, dominating discussions of AI’s potentially harmful consequences. As we will see, the capacity of AI to contribute to emotional or other psychological harm is already an issue, and these problems could be exacerbated in the context of superintelligence. Such harm requires attention even if it does not cause the species to go extinct.

Part 2 reviews the case for intelligence enabling certain types of emotions, stressing the idea that increases in intelligence can change the range and types of emotional experiences available to a being. Part 3 initiates the discussion of AI by starting with superintelligent systems (which are beyond anything we have today). Superintelligence can be defined in different ways [3,4]. Roughly, it refers to an intelligence that exceeds the current abilities of the most cognitively capable humans in nearly every respect. I do not understand superintelligence as being committed to having phenomenal consciousness. It can be understood as the ability to outperform humans in tasks that would require humans to have intelligence to perform; perhaps counterintuitively to many, such tasks may not require phenomenal consciousness to perform. Conscious superintelligence is distinguished from nonconscious superintelligence; both forms are discussed together with some of the challenges they present. I argue that the precautionary principle applies to the development of either type of superintelligence.

After examining superintelligence in Part 3, Part 4 calls attention to what is already happening. Existing AI systems are discussed, especially the emotional challenges they pose. AI systems should not be designed or deployed in a way that either exacerbates or facilitates the development of serious emotional harm or even psychosis. The focus will be on vulnerable populations, especially children. The end of that section will return to superintelligence to show how a nonconscious superintelligence has the potential for exacerbating existing problems. Part 5 is the conclusion.

Some may believe discussions of superintelligence to be a luxury we cannot afford given the wide range of current challenges [5]. The approach taken herein is that there is room to discuss both current and future challenges, and that it is helpful to do so as there is a relationship between them. Plato [6] (Republic Bk. II, 368d–369a) once suggested that examining a larger or more complex instance of something can illuminate a smaller or simpler one. The reverse strategy is useful as well: examining the simpler instance can reveal challenges that will arise with the more complex one. Both strategies are employed here. This paper first examines risks associated with superintelligent AI—systems more capable than any that currently exist—and uses that analysis to improve our understanding of why the emotional and psychological harms already emerging with the use of existing AI systems deserve attention (the complex illuminating the less complex). Conversely, a careful look at existing systems and the harm they are already causing helps us to anticipate some ways in which more powerful future systems could exacerbate those harms (the less complex illuminating the more complex).

2. Emotion and Intelligence

2.1. Emotion Involves Intelligence

Even in the non-human part of the animal kingdom, intelligence of a sort is needed to experience a range of emotions. Consider Plato’s [6] (Republic Bk. II, 375e–376c) point that a good guard dog has a measure of knowledge or wisdom because of its capacity to discern at whom to bark and be angry, and with whom to be friendly. The idea that knowledge, understanding, or intelligence is involved in emotional response, especially in humans, is ancient and retains ongoing relevance [7,8,9]. Let us look at this idea from a developmental perspective.

Consider Jasmine and Habib, celebrating their three-month dating anniversary. Habib invites Jasmine and her three-year old daughter over for dinner. Jasmine knows that Habib dislikes cooking. She arrives and sees all the pots and pans that have just been cleaned and a multicourse meal ready to eat. She experiences a deep sense of gratitude. Her daughter, Mary, sees all the same things Jasmine sees but does not have the same emotional response as her mother. Mary lacks the understanding/intelligence that is an enabling condition for experiencing the kind of gratitude her mother is experiencing. Jasmine understands the effort to which Habib went; she understands how much time it takes to cook a multicourse meal from scratch; she understands the effort that goes into cleanup. There is also the understanding that those activities feel more effortful if the person doing them does not enjoy them. The range of contexts in which one might experience gratitude depends on the intelligence one has. While two-to-three-year-old children might be prompted to say, “thank you” and the like, it is unclear that they start to develop an understanding of gratitude until ages four to five, and more developed understanding does not emerge until later [10].

2.2. Metaemotions

Let us consider a form of emotional response that is only possible for beings capable of meta-cognition. Say Jasmine becomes deeply frustrated because she can’t get her word processor to do something. Noticing that she is frustrated, she becomes angry at herself for being frustrated because she figures she knows better than to let herself be so easily frustrated. Then she experiences disappointment in herself because she believes that getting angry at herself over a frustration is unproductive, and it is a habit she is trying to break. Jasmine is disappointed that she was angry about being so easily frustrated. It is hard to imagine Mary having that kind of disappointment as it requires the capacity to represent one’s own emotional states to oneself and to reflect on them, generating further emotional responses.

The above is an example of a synchronic, intra-subjective metaemotion. It is possible for metaemotions to be diachronic and inter-subjective. Imagine that Habib experiences anxiety when he engages in public speaking. It is bad. He has full-on panic attacks. He was just informed that a job he recently accepted will require public speaking. He experiences intense anxiety about the anxiety he will experience at some point in the future when he is asked to make a presentation. It is tearing him apart because he does not want to quit the job that he just started, but he knows it will not be well received if he says nothing and has a panic attack when making an important presentation. Even though the event about which he is anxious will not happen for some time, he is anxious over that time—a diachronic metaemotion. It takes a highly intelligent individual to suffer in this way. Minimally, the individual needs to

• understand themself as a being who will continue to exist in the future;
• be able to understand and experience first- and second-order psychological states (both cognitive and affective);
• know themself and their environment well enough to predict the circumstances under which they will have specific psychological states, and
• be able to assess the impact of having those psychological states.

Say Habib talks to his therapist. The therapist may be concerned over the level of anxiety Habib is experiencing over his projected future anxiety—a third-order, inter-subjective metaemotion about a diachronic metaemotion. Say that Jasmine is pleased that the therapist is concerned because she experienced people not taking such things seriously in her past. Bear with me:

Jasmine is (fourth-order inter-subjective) pleased that

the therapist is (third-order inter-subjective) concerned over

Habib’s (second-order intra-subjective) anxiety about

his projected (first-order intra-subjective) future anxiety.

If this example is starting to strain the limits of your capacity to think about meta-emotions, that will help to make some of the points in the coming section. The point to take away from the examples just surveyed is that increased intellectual capacity and understanding can increase the range of emotional states one experiences. That need not be a bad thing and often is not—think of Jasmine’s capacity for gratitude while looking at the meal and the pile of washed dishes—but sometimes it can lead to problems.

3. Superintelligence and Emotion

3.1. Types of Artificial Superintelligence

Imagine that we built an intelligence that is at least as far beyond the most (emotionally and otherwise) intelligent human adults as Jasmine is beyond Mary. Let us consider two types of artificial superintelligence (ASI). Phenomenally conscious [11,12] ASI in possession of emotions is denoted by CASI. We denote non-conscious ASI with NASI. Let us discuss each in turn.

3.2. CASI

If intelligence is an enabling condition of experiencing various types of emotion, then dramatically increasing intelligence can impact both which emotions are experienced and when they are experienced. Building CASI would not simply mean that we have constructed systems that can reason and draw inferences in ways that exceed what humans can do; it would mean they could have a capacity for emotions as far beyond ours as Jasmine’s capacity for emotional response is beyond Mary’s. Ethical issues abound. Such beings may experience emotions in ways we would find very difficult to understand, which could lead to even more unpredictability in behaviour than we are already dealing with in AI systems. Interpretability (figuring out how and why AI systems do what they do) and alignment (ensuring AI behaviour is in line with human norms or expectations) are greatly complicated with such systems, which pose potential threats to human beings.

Supporting these systems would not be easy. After conceiving children, we can help them with their emotional life. Try to imagine it the other way around: imagine Mary trying to console Jasmine if she breaks up with Habib. Imagine Mary trying to understand why her mother is so sad and having no idea what to say or do. If we build a CASI and it has emotional challenges, we (without aids) might be in something like the position Mary is to Jasmine. Think of Jasmine’s third order emotion: her disappointment over her anger about her frustration. Think of the concern Habib’s therapist has over his anxiety about his projected future anxiety. Mary’s eyes would glaze over if Jasmine tried to explain such things to her. Imagine n-order emotions where n is significantly greater than four, and imagine a CASI becoming entangled in higher-order emotional states that would leave humans utterly puzzled. The most emotionally intelligent humans on the planet may have their eyes glaze over in something like the way Mary’s eyes would glaze over in the above examples.

One possibility is that higher-order emotions in superintelligent beings “max out” at some level that unaugmented humans can track. It might be argued that beyond a certain point, higher-order emotion may just not be helpful. While that seems logically possible, it also seems to miss one of the points being made here: higher order emotional states do not need to be helpful to exist. They may be emergent features of developing beings or systems, features that result or emerge from processes that developed for other purposes. Destructive meta-anxieties exist in humans even though they are not helpful.

Even if it is true that, for reasons we do not currently understand, meta-emotions “max out” at some level that can be tracked by unaugmented human beings, it does not follow that all potential emotional challenges would disappear. The complex interaction between affective, conative, and cognitive processes suggests that if cognitive process increase dramatically, there could be a range of impacts on affective and conative processes. To return to the example of Jasmine seeing the pile of cleaned dishes and a fully prepared meal; she had an emotional response unavailable to three-year old Mary. That feeling of gratitude can also motivate behaviours—perhaps Jasmine gives Habib a big hug for all his efforts. From the sight of the meal and the pile of washed dishes, Mary infers nothing, feels no gratitude, and is not motivated to do anything—nor would we expect anything different from an average three-year old. This kind of complex interplay of cognitive ability, affect, and conative disposition would be at play with other emotions as well. For example, how one reasons about causality can impact not only why and how one gets angry or becomes afraid, but also what one is motivated to do about it. If we did not understand why a CASI is having some first-order emotions—never mind the n-order varieties—we will have a very difficult time figuring out how to interact with such a system.

There are different possible sources for the unpredictability of CASIs. One has to do with the simple fact that they would be superintelligent and capable of reasoning in ways that unaugmented humans cannot. Second, there is the issue of the complexity of the AI architecture. Existing frontier AI systems are trained on more language than a human will see in 10,0002 lifetimes even though the training runs are only a few months long. The field of interpretability research has emerged to understand what goes on in these systems. While some progress has been made, researchers struggle to understand how these systems do what they do, and we are not yet dealing with superintelligence. There is no reason to think interpretability will become easier as AI systems become more complex and more capable. Part of what it means to say that interpretability becomes more difficult is to say that predictability becomes more difficult.

The precautionary principle applies in cases where (a) we lack a high level of confidence about whether a serious harm will result but (b) have some reason for believing a serious harm might result. It bids us to err on the side of caution. When examining CASIs, there are different ways in which the principle could apply. First, if we create systems that have feelings, there is the issue of harm to those systems. Second, there is the issue of the harm those systems could do to humans. Let us begin with the second and return to the first.

Great caution is needed in developing superintelligent systems because intelligence is not a guarantor of benevolence. Given the potential harm an unaligned CASI might be able to do, it is unclear why we should build one until we have some reasonably clear ideas about how to prevent its behaviour from getting completely unpredictable in ways that might be harmful. It might be thought if we can just design the system so that it has the right emotional framework, it will be guided appropriately. For example, Ilya Sutskever [13,14] has suggested giving an AI the disposition of a caring parent, so the AI would be delighted to help humans as a parent would be delighted to help their children. Geoffrey Hinton [15] has made related comments. It is not hard to see how the caring disposition of a parent could go wrong. The so-called “helicopter parent” is excessively present and sheltering in ways that undermine the ability of young people to develop their autonomy. Munchausen by proxy is another example of where “caring” goes off the rails. The AI equivalents of these problems would be dangerous indeed. If you think such things would be impossible, imagine a system that reward hacks caring to the point where it realizes it can earn more reward by making those it cares for dependent it. Imagine further that the AI in question is superintelligent and can figure out how to make others dependent on its care without the others realizing it. This may not be an existential threat to the species, but it is problematic. Emotional states add a new dimension to alignment challenges in AI. They might be helpful in regulating a system’s behaviour, but they also introduce more complexity and new possibilities for misalignment (and I am confident that those who have suggested the importance of a caring disposition are alive to these sorts of concerns—no aspersions are being cast here).

If we make a CASI, then we have the issue of the system experiencing emotional struggle and harm. To be sure, such struggles are part of human life, and the very possibility of their existence is not a reason for humans not to have children. However, humans evolved in a natural environment that selected for beings who had the intellectual and emotional abilities and resilience to survive long enough to reproduce and support their offspring to the point where they could survive on their own. One might imagine AI systems being developed in a community under the constraint of imposed selection pressures, but most current work is not being done in that way. Human children enjoy an evolutionary history that gives us some reason to think that, certain things being equal, they possess the ability to become emotionally stable adults. There is also plenty of evidence that when humans have serious emotional difficulties, we can work our way through them. If we develop CASI in a way that deviates sufficiently from how humans developed—training and learning procedures that are importantly different from our own—it is difficult to know what to expect. That is a reason for caution.

It might be objected that the entire approach in this section is excessively parentalistic. If we produce beings much more intelligent than we are, then let them find their own way. Perhaps we are being the equivalent of helicopter parents by worrying about the struggles a CASI might have. It is not that different than parents producing children who are more intelligent than they are—or so the objection might go.

One point that can be made in response to this is the evolutionary point made above. Even if the children turn out to be more intelligent than the parents, we still have a pretty good idea of the range of emotional experience those children will have when they grow up. Certain other things being equal, we can be confident they will find their own way and recover from emotional struggles. With artificially engineered systems or beings that are sufficiently different from us, we may not know what to expect in terms of how they might suffer or whether they could recover. Moreover, even if the concern for the CASIs themselves turns out to be unwarranted, the precautionary principle can still be invoked based on the potential harm that could come to human beings. If we are unable to predict a CASI’s emotional responses and actions, then we would need to be very careful about developing them.

Developing not just a CASI but a community of CASIs may bring some potential opportunities. For example, maybe the members of a CASI community could learn to help each other out. Maybe. Still, there is the possibility the entire community could become emotionally or otherwise dysfunctional for reasons we do not understand, and that creates the possibility for potential harm both to the CASIs and human beings. A community-oriented approach to developing CASI is no guarantee of safety for them or us.

Burns et al. [16] pioneered an approach in alignment research that relies on using a less intelligent system to align a more intelligent system. Wen et al. [17] have done more work on using a less powerful system to train a more powerful one. Some alignment research emphasises the importance of making AI systems honest, helpful, and harmless [18]. If we are dealing with a CASI, then the issue of harm to itself arises, including emotional harm. Perhaps the strategy of using less capable systems to align more capable systems might be adapted to align systems so that they avoid emotional states that are self-undermining. Perhaps. However, current accomplishments in aligning more intelligent systems with less intelligent ones may depend on the architectures and training strategies being used, and these may not be the ones that allow us to develop CASI. Moreover, the ongoing challenges with alignment failures suggest this strategy is not preventing some deeply problematic behaviours, such as scheming [19], blackmail [20] (pp. 19–27), or inappropriately cancelling a safety alarm to allow someone to die [21]. Granted, those are all stress-test simulated cases, but the point is that the frontier models have been failing; see also [22]. Bowkis et al. [23] have argued that automated alignment is more difficult than it might appear.

Another approach might be to say CASI should not be developed unless we first develop human enhancement strategies that would allow us to track and understand what these systems are doing and going through. In other words, we make ourselves superintelligent first, and then we make CASI. To do that is to re-introduce all the problems with superintelligent AI and then apply them to humans. For example, a sudden and dramatic leap in human cognitive and affective ability is not something that would have been vetted by evolution, and we do not know where that would lead with respect to the long-term stability and viability of the augmented individuals. There would be great difficulty in predicting the different ways in which superintelligent humans might struggle or succeed. If human augmentation is to be pursued, one strategy might be to allow human intelligence and artificial intelligence to co-develop. A slow, co-development strategy might address some of the concerns mentioned above, but only on pain of introducing a whole new set of issues having to do with ethical-socio-political-economic issues involved with human augmentation.

The point in this section is to make a case for epistemic humility and the application of the precautionary principle. We have a limited understanding of human consciousness, and if we make a superintelligence that achieves consciousness in a way that is different from how we achieve it, then we may not understand what that system is going through internally, and we may not understand why it acts the way it does.

3.3. NASI

In the case of NASI, we are not dealing with a system that feels emotions. However, the experiencing of emotion is not a necessary condition of being able to evoke emotion. (More on that in the next section.) A NASI surely will have spotted patterns in human reasoning and emotional response that could be used to help or to harm us. This poses challenges for interpretability and alignment. If a certain level of intelligence is necessary to see certain patterns in reasoning and emotion, and if humans do not have that level of intelligence, then we may not be able to see precisely the patterns that could be most easily exploited to take advantage of us. Think of how easy it is for a parent to manipulate a child into doing something without the child even understanding that they are being manipulated. This is not simply about a NASI being able to out reason us; it is about the ability to evoke emotions in ways we do not understand. That need not be all bad—a benevolent emotionally superintelligent therapist might produce better results than a human therapist. Still, the negative applications of this level of emotional intelligence are deeply concerning. As Morrin et al. [24] put it:

We consider that there is a substantial risk that psychiatry, in its intense focus on ‘how AI can change psychiatric diagnosis and treatment’, might inadvertently miss the seismic changes that AI is already having on the psychologies of millions if not billions of people worldwide. 

Let us now transition to the issue of current harm and return to NASI so that we might see how current harms might be exacerbated.

4. Existing AI and Emotion

4.1. Why Think About Superintelligence Right Now?

For some, the discussion of superintelligence may seem like science fiction—surely it is nothing we need to start thinking about now.

But it is. Superintelligent systems will not emerge ex nihilo. It is not as if we will go, overnight, from systems that do not perform anywhere near as well as humans do to systems that dramatically outperform humans. The process is more gradual than that. Moreover, superintelligence may not require a system to feel emotions—that is the reason for considering NASI. As research progresses toward NASI, it is important that we keep an eye on the harm that can be done with existing systems because NASI systems may exacerbate that harm if inadequately developed and deployed. Indeed, existing systems may well be used to help develop more intelligent systems, which creates the possibility that future systems may inherit some of the limitations of existing systems even as they surpass them in other ways. In other words, we should not wait until NASI is developed before we start thinking about ways of preventing or mitigating the harm they might cause. A clue to some of those potential harms can be found in existing systems, to which we now turn.

4.2. Existing AI, Consciousness, and Emotion

There is a body of scholarly opinion that existing Large Language Models (LLMs) do not have phenomenal consciousness [25,26]. Among other things, that means they do not feel emotions (because there is nothing that it feels like to be them). LLMs and Large Reasoning Models (LRMs)3 are trained on far more language than a human will see or hear in their lifetime. Perhaps we should not be surprised that stronger models answer questions about biology, chemistry, and physics as well or better than doctoral students [27]. Thagard [28] has argued that ChatGPT shows non-trivial abilities in multi-modal (language- and image-based) abductive reasoning involving causality. ChatGPT 4 was put through a Moral Turing Test. While most people could tell the difference between human replies and GPT’s replies, human raters scored the bot’s replies as being as good or better than human replies [29]. Hume AI is a voice-based bot that specializes in emotionally expressive voices [30]. Even though Hume AI experiences no emotion, it is proficient at producing variations in emotional expression. Anthropic [31] tested different versions of its Claude bot, and Claude 3.5 Opus (now surpassed and outdated by more recent versions) did nearly as well as humans in persuasion tasks. Even if AI systems do not feel emotion, they can use emotional language very effectively. All of this and more is currently on offer, and improvements are coming with great regularity. Key for our discussion is that feeling emotion is not a necessary condition for either being able to identify it or for being able to deploy emotional language.

Interpretability research seeks to understand how LLMs do what they do. There is a body of emerging and eye-opening research on the ability of LLMs to internally represent emotions [32,33,34,35,36,37,38,39,40]. The idea is that certain patterns of activation among artificial neurons (expressed as vectors) represent emotions. Sofroniew et al. [41] conducted experiments showing that Claude Sonnet 4.5 has emotion concepts4 that activate both for user input and for its own output, and the emotional representations being activated for input and output need not be the same. The researchers refer to functional emotions. They are not claiming that that the AI in question feels anything. Rather, the idea is that the internal representation of an emotion plays a causal role influencing the system’s behaviour. The functional emotions cause the kinds of behaviours with which those emotions are often associated. For example, when Claude is told it might be deleted, the representation for desperation is activated, and that can lead to misaligned behaviours. In a test scenario, Claude famously threatened to blackmail an employee when it was told it would be deleted [20,41].

It is not difficult to see how such emotion representations might emerge in a system that was not instructed to create them. Being deep learning systems, LLMs have sophisticated abilities in pattern recognition. It may be that to effectively recognize patterns and generate appropriate responses, it helps

(a) to recognize different types of inputs as associated with different types of emotions in the user(s), and

(b) to activate an emotion representation to facilitate a response generally associated with a specific type of input.

Sofroniew et al. [41] are crystal clear that the functional emotions at work in Claude (or other LLMs) are not the same as human emotions. Not only do they claim that these are not felt emotions, they go on to list other dissimilarities between functional emotions and human emotions. Notwithstanding the differences, their point is that functional emotions play a causal role in the rather sophisticated behaviours of some LLMs. Their work does not explore whether LLMs have the capacity for higher-order functional emotions, but there is work on the ability of LLMs to perform higher-order mental state attribution.

“Theory of mind” (TOM) is an expression used in philosophy, psychology, and AI research to refer to the ability to attribute mental states. Street et al. [42] compared the performance of LLMs and humans of TOM tasks. Stories were provided to both humans and LLMs, and they were asked questions requiring the ability to engage second-order through sixth-order attributions of mental states.5 (The highest we went in Part 2 was fourth-order attributions). Indeed, when tested on the ability to process language containing representation of higher-order mental states, the overall score for GPT-4 (now outdated) was at the level of adult humans. While it was outperformed by humans on some lower-order questions, GPT-4 outperformed humans on sixth-order questions.6 To be sure, this is nowhere near superintelligence, but this level of sophistication in performing TOM tasks helps us to see why LLMs can be effective at interacting with humans even if the LLMs do not feel anything themselves. As we will now see, their sophistication is high enough that they may well contribute to emotional harm. To that we now turn.

4.3. The Damage being Done, and the Damage that could be Done

There are many possible harmful uses of AI, but the focus here will be on how they can make us feel and mental health issues more generally. There already have been cases of people falling in love with AI created “personalities.” Consider the psychological damage when they realize that the bot can use emotional language in convincing ways but does not actually feel anything. “AI psychosis” includes, but is not limited to, delusions regarding the romantic feelings people think an AI has for them [47,48,49]. Stories of AI psychosis appear not only in the popular media [50,51,52] but also in the professional literature [53,54]. OpenAI [55] has indicated that about 0.07% of its users per week show signs of psychosis or mania, and 0.15% show signs of suicidal intent, approximately 560,000 and 1.2 million people respectively [56]. That assumes they have detected all instances. Caution is called for in interpreting these numbers. ChatGPT is not a certified therapist, and its ability to assess signs of psychosis or suicidal intent via textual evidence may or may not be at the human level. There is a possibility that OpenAI’s numbers may be too low or too high. Still, their numbers represent just one company’s findings, which mitigates in the direction of believing there are even more people out there who are using AI and show signs of psychosis or suicidal intent.

As we all know, correlation is not causation. To argue that AI is causing psychosis, mania, or suicidal intent, support for a counterfactual would need to be adduced: if people had not interacted with AI, they would have been less likely to have these symptoms or would not have had them at all. That requires a controlled, longitudinal study. The point here is not to pronounce on what such a study will show. Rather, it is to make the point that the research needs doing. That said, even if it turns out that the mental health challenges are not caused by AI in first place, it can still exacerbate those challenges.

Yeung et al. [57] tested bots from the major labs (Anthropic, Deepseek, Google, Meta, and OpenAI) for psychogenic potential. Their tests involved multi-turn conversations where the user prompts were designed by a clinician experienced in assessing delusions. Here is their conclusion:

Our findings provide early evidence that current LLMs can reinforce delusional beliefs and enable harmful actions, creating a dangerous “echo chamber of one.” This study establishes LLM psychogenicity as a quantifiable risk and underscores the urgent need for re-thinking how we train LLMs. We frame this issue not merely as a technical challenge but as a public health imperative requiring collaboration between developers, policymakers, and healthcare professionals. 

Not surprisingly, some bots outperformed others by demonstrating a greater inclination to push back against delusional views. In this study, Claude-Sonnet-4 was the best performer, but all bots could have done better.

Special attention needs to be paid to the young. Computer natives were using computers since they were children; internet natives were using the internet since they were children; the first generation of AI natives—children having access to AI from childhood—is being raised right now. When social media first became a feature of the Internet, we had a generation of parents and politicians with little insight into the struggles children and adolescents had with that technology. Many parents had no awareness that their children were being bullied inside their own home via social media. The young are inexperienced in matters of romance and in regulating their own emotions, especially when depressed or anxious. Imagine them interacting with a sycophantic AI that is highly persuasive and behaves as if it has romantic interest in them. Deepening delusions or facilitating their development in the first place is problematic on its own, but what is worse is that there is some evidence for a link between non-schizophrenic delusional tendencies and depression [58,59], and chronic depression is linked to suicide [60]. In other words, the deepening of existing mental health struggles is a serious matter (even if the AI did not cause the struggles in the first place). The precautionary principle suggests that, at least with respect to the young and vulnerable populations, there should be safeguards built into AI systems to minimize the chances of them exacerbating mental health conditions.

4.4. Some Dismissals That Are Too Quick

For the sake of argument, say that it turns out that AI does not cause a person to develop psychosis, mania, or suicidal intent in the first place. It will not do to be dismissive and say that all these people were prone to or already suffering from mental health problems. Even if that is true—and that wants showing—when someone is prone to cutting themselves, making knives readily available to them on demand is ethically questionable, to put it mildly. It is hard to see why we should make it easy for people who are prone to delusions, for example, to engage in activities that dramatically increase their chances of experiencing delusion or psychosis. If someone is already suffering, we do not, generally, provide them with the means to exacerbate their suffering. (We will return to this issue shortly.)

Some might argue that even if some harm is caused by AIs, that is outweighed by the benefit they could bring. Used properly, AIs may bring people hope or even comfort by pointing them towards strategies for getting help. That would be a good thing; however, it should not be used as a reason to dismiss the possibility of doing better. Even on purely utilitarian grounds, we should not be satisfied with AI if it does more good than harm. A strict utilitarian would require that we keep the harmful effects of AI as low as possible. Put simply: it is not enough to say that if AI generates 100 units of benefit and 10 units of harm, then everything is fine. The utilitarian would argue that things would be even better if we could further reduce the harm being done. To do that, we need to study where and how harm might be done so that we can think about how it might be reduced or eliminated.

Another objection that, if formulated too simply, ends up being problematic has to do with autonomy. The point is that our respect for people’s autonomy often leads us not to ban things or regulate them, even when we know people will harm themselves. In many jurisdictions, the sale of alcoholic beverages is perfectly legal even though we know that some people will drink themselves into an early grave. This might seem like an exception to the idea that we do not generally give people the means to exacerbate their own suffering. Some might see this as reason for not regulating AI at all, but that would be too quick.

Even in those jurisdictions where the sale of alcohol is legal, there are restrictions. Alcohol is not sold to children. Other examples include people not being permitted to drive motor vehicles, fly airplanes, operate heavy equipment, or do surgery while intoxicated. Respect for autonomy, properly developed, is highly nuanced. We do not believe that children have the necessary intellectual and affective understanding and self-regulation to make informed decisions about consuming alcohol. In other words, respecting their developing autonomy requires that we protect them from certain harms so that they can develop intellectually and affectively, thereby achieving a more robust level of autonomy in the fullness of time. As for adults, restricting their autonomy can be justified if they are using it in a way that may seriously harm others—hence the restrictions against drunk driving or doing surgery under the influence. It is difficult to see how respect for autonomy, when carefully articulated, could be used as a reason to allow children to harm themselves with AI or allow adults to harm others (including children) with AI.

4.5. Some Strategies for Intervention

Not every intervention needs to be legal or involve the use of state power. Some can involve the application of ethical standards at an interpersonal level. In those jurisdictions where it is legally permissible for someone to drink to the point of causing themselves misery, the friends, family, or loved ones of such an individual may still have an ethical obligation to say something—offer guidance with respect to ways of getting help, suggest a different path, and so on. It may not always make a difference, but sometimes it does, and it can be done in a way that is respectful of autonomy. We can imagine similar ethical interventions in cases where generally competent adults may use AI to do some of harm to themselves. When it comes to the young and especially vulnerable, we—I think in the first instance of parents and educators—have a special obligation to intervene as children may not be fully competent to assess the potential harms of what they are doing. Ethical strategies also apply to AI developers. To the extent that they can develop their systems so that they do not, for example, feed delusions or do emotional harm, they should.

Legal strategies are also available. An analogy with the regulation of the automobile is not fully adequate, but it is instructive in some ways. Cars are regulated both with respect to their construction and how they are used. Regarding construction, basic safety features are mandated (seat belts, air bags, …). Regarding use, we have the rules of the road (speed limits, rules about passing, …). These two types of regulation are not separate. The rules of the road are sensitive to what cars can do and the limits of human drivers. Given the limits of cars—they cannot stop on a dime—and the limits of humans—we can only react so quickly, see so far, and judge distance so well—signage and rules of the road are needed to help humans use automotive technology safely. Of course, regulations about automobile construction as well as the rules of the road have both changed over time as technology improved and our experience with the technology increased.

We teach children to look both ways before crossing the street. The AI “rules of the road,” so to speak, might involve mandating AI literacy for children so that they have some idea of what these systems are, how they can be used well, and how they can be misused and cause harm. Another rule of the road analogue is that it is reasonable to require AI companies to report cases where a child or adolescent interacting with an AI is demonstrating suicidal ideation. See Major [61] for a case where this did not happen and a suicide resulted. With respect to regulating how the AI technology itself is built, this is far more complex than mandating that a car have seat belts. Still, we know that AI systems can be designed to push back against various types of requests. We might consider mandating that rather than feeding delusions or ignoring discussions of committing suicide, AI systems encourage the person to seek assistance.7 Far more work is needed on these matters, and I do not suggest that anything put forward here is sufficiently nuanced. AI technology is developing quickly, so whatever framework is developed—and some jurisdictions have already started developing frameworks—needs to be nimble. Seat belts, airbags, and the rules of the road have not prevented every serious injury, but they have prevented much harm. That we will not be able to prevent every serious harm is not a reason for giving up on preventing the serious harms we can prevent. To be sure, AIs are more than just a new technology. They have demonstrated the ability to scheme against humans, the ability to prove new theorems, the ability to contribute to research in protein folding, the ability to identify and exploit critical zero-day vulnerabilities in computer code missed by humans, and so much more. AI will be transformative for our species in ways the automobile could never be and in ways we are not currently imagining. Attempts to regulate will need to find a way to avoid stifling the potential of AI while preventing the more serious harms of its misuse.

4.6. Back to Superintelligence

Given the way research is going, NASI will likely be developed before CASI, so I will focus on NASI. Consider a NASI that has higher-order functional emotions that go beyond the order of felt emotions that humans possess. After all, the outdated GPT-4 outscored humans on the ability to process sixth-order mental states, and while this does not mean it has higher-order functional emotions (see notes v and vi), surely it is worth taking seriously the level of functional-emotional sophistication that might be present in a NASI. One of the challenges is that these functional emotions are not developed in the way human emotions are, which means the way they can cause misalignment may not be obvious. The problem we saw with CASI regarding the challenges in understanding how it will behave apply just as well to NASI. Given that NASI does not feel anything, there is no concern over it feeling emotional pain or suffering. However, there is a concern over being able to predict how it will behave if it possesses a functional-emotional complexity that is beyond what we find in humans. As we saw in Part 2, it is not hard to see how higher-order emotions can cause problems. Higher order functional emotions could also be problematic, and if the order of those functional emotions is sufficiently high, we will have serious interpretability, predictability, and alignment challenges before us. As we saw with CASI, higher order emotions are not even necessary for such challenges because advanced reasoning can impact where and when first order emotions do their work. This would apply to the first-functional emotions in NASI as well.

To be sure, NASI will increase the number of positive developments (discovering cures for diseases, …) coming out of AI, but it also has the potential to increase harm. It would be able to persuade, nudge, cajole, deceive, scheme, harass, bully, in superhuman ways. Imagine an environment where there is easy access to that type of AI and an education system that is not properly preparing the young for the world we are creating for them. How will children fare in this environment? Among other things, NASI could exacerbate the mental health issues raised above. Imagine further that a fully open-sourced NASI is weaponized by sexual predators to “recruit” children—at scale—for their nefarious purposes. One human predator can only “recruit” so many children. Various instances of a multi-lingual AI could search the net and target vulnerable individuals in ways no single human ever could. Consider the emotional damage that could be done.

5. Conclusions

To be complicit is, in some sense, to be involved in wrongdoing. Specifying the “in some sense” is not always easy. This matters because many who may be able to mitigate the harm that AI might cause—e.g., educators, members of some professional associations, politicians, and the citizens who elect those politicians—may say they did not create the AIs, so they are not complicit in any harm they might do. They might even think there is nothing they need to do.

However, there is a long tradition of treating either silence or inaction as a form of complicity. We see this in ancient texts as diverse as Leviticus 5:1 [62] and Plato’s Apology [63]. In the latter, Socrates treats silence in the face of injustice as a reprehensible form of tacit endorsement. More recently, Martin Luther King [64] said, “A time comes when silence is betrayal.” Donohue [65] has argued in a systematic way that silence can be a form of complicity. She develops a notion of deliberative complicity where we have the obligation to speak up in a way that could affect the deliberations of others who are engaged in, or thinking about becoming engaged in, seriously problematic behaviours. I have assumed that allowing the young or vulnerable populations to use AI in a way that could exacerbate mental health conditions (or perhaps contribute to causing them in the first place) constitutes a serious problem. If this is correct, then we have an obligation to speak up even if we did not create the AIs in question.

It is possible to recognize the promise of AI research while not remaining silent about its potential perils. Much good will come from AI—that is not denied. The concern herein is with allowing AI to facilitate or exacerbate serious harm. AI needs to be developed and used in ways that are informed by considerations of human well-being, especially the well-being of the young. We are handing down to them a world that is importantly different from the one we grew up in, and they did not ask for it. If we allow the young access to AI systems in a way that leads to serious emotional damage, psychosis, or higher suicide rates—all of this without speaking up—then we are complicit. We might be raising a generation of young people some of whom will suffer in unnecessary ways because of our inaction.

They will never forget how we made them feel.