Digital Mental Health Post COVID-19: The Era of AI Chatbots

1. Introduction

The mental health treatment gap is worsening since the COVID-19 pandemic which triggered a digital transformation of mental health care due to changing social dynamics, the widespread use of smartphones, and the proliferation of digital mental health tools [1,2,3,4,5]. This technological shift has fundamentally altered how support is delivered by broadening accessibility and expanding reach for those seeking personalized mental health support. Digital mental health tools—particularly those underpinned by Artificial Intelligence (AI) chatbots—are attractive because of their low cost, accessibility, and anonymity. However, the growing interest in AI chatbots has not yet translated into clinical benefits and improved outcomes for users (patients) with anxiety and depression [6]. Questions remain with regards to the safe, engaging and effective integration into existing models of care [7,8,9,10,11].

Despite there being more than 10,000 digital mental health resources globally, low user retention rates appear persistent in combination with a lack of digital literacy, clear privacy guidelines and proven clinical efficacy/integration, as well as human support in the app [5,6]. AI chatbots operate in a largely unregulated environment which exposes vulnerabilities especially in underserved populations. Consequently, there are renewed calls for trained Digital Navigators to assist in the safe integration of technology into mental health care settings, driving engagement, and supporting both the needs of the clinician and the patient [5,12]. A recent trial found changes in the level of support provided by Digital Navigators directly affect how effective schizophrenia apps are, meaning standardized training is essential to reliably evaluate these tools [13].

GenAI-powered platforms, especially those using advanced Large Language Models (LLMs), are increasingly sought out for consulting about mental health care. However, these platforms often lack ongoing engagement and fall short of emotional intelligence and trauma-informed objectives in practice [14,15,16,17,18]. LLMs like GPT-4 showed they generate coherent text, maintain conversational context, and perform advisory or counseling tasks, making them suitable for various mental health applications [19]. Nonetheless, the integration of AI chatbots into clinical practice is understudied in terms of effectiveness and safety, and the ability to provide nuanced, meaningful support.

Increasingly, AI is being deployed as “agents” and “assistants” through “therapist” and “companion” types via mobile apps, web platforms and social robots [20,21]. AI chatbots for mental health care generally come in rule-based, machine learning, and/or LLM systems. Functioning as autonomous agents, they assist with screening, prevention, monitoring, clinical assessment, treatment, emotional support and companionship.

There is a lack of clinical evidence supporting AI-based therapy because of limited conclusions about their efficacy and safety in clinical practice. For example, a narrative review of recent clinical studies on AI chatbots for anxiety and depression found them to be feasible and acceptable yet there is insufficient evidence of AI effectiveness, small and narrow samples, weak controls, and unexamined risks such as emotional dependence and parasocial relationships [6]. The first randomized controlled trial (RCT) of a GenAI therapy chatbot (Therabot) demonstrated moderate symptom improvement for major depressive disorder, generalized anxiety disorder and eating disorders [22]. Public reactions to these technologies are mixed: while some appreciate the accessibility and affordability of AI mental health tools, others remain skeptical about their effectiveness, ethics, and safety [23]. This uncertainty echoes broader frustrations with current mental health systems, as well as cautious optimism about the potential of AI chatbots as complementary resources.

There are complex issues that require exploration, notably algorithmic bias and errors, privacy risks, and the challenge of integrating AI chatbots into existing care structures. Notably, there have been cases where AI chatbots appeared to exhibit consciousness—a phenomenon referred to as “Seemingly Conscious AI”—which, despite a lack of scientific evidence for AI consciousness, has prompted debate about ethical design, transparency, and the need for human oversight [24]. This is particularly important to understand for sensitive settings like elder care, where user safety and meaningful, evidence-based support are critical [25,26].

While reviews and meta-analyses highlight the potential of technological innovation in mental health chatbots to improve outcomes across diverse settings, these tools are still largely untested in rigorous clinical efficacy trials [27,28,29,30,31,32,33]. It is unclear how many people are using AI chatbots for mental health support globally, although findings in Australia with community members show it to be 28% and 43% for mental health professionals [34]. A 2025 survey of US residents with ongoing mental health conditions found that nearly half (48.7%) used LLMs for psychological support in the past year—primarily for anxiety, personal advice, and depression—with most reporting improved mental health and high satisfaction [35]. Some users rated LLMs more beneficial than traditional therapy, though a minority (9%) experienced harmful responses.

After consultation with 171 mental health experts, OpenAI released findings 560,000 people a week are showing signs of psychosis or mania (0.01% of ChatGPT-5 users), 1.2 million users show indicators of suicidal planning or intent (0.15% of ChatGPT-5 users) [36]. There was 65-80% reduction in unsafe responses across crisis scenarios, 92% compliance in the most serious cases (suicide, psychosis, and over-reliance) as well as above 95% reliability in longer conversations. These examples of shortfalls in the performance of AI chatbots shows the need to assess the challenges related to user engagement, safety, and effectiveness for AI chatbots integrating with health care and support systems.

2. The Problem

The benefits of AI chatbots are that they may expand access to care, support patient assessment and monitoring, and deliver personalized interventions (i.e., diagnosis, clinician support, and direct therapy) [6,37]. However, the risks are highest for direct therapy because of the lack of therapist oversight, while AI offering incorrect diagnoses and misguided clinician support are also high.

While chatbots may improve access, affordability, engagement and support especially for young people [6,11,26], help-seeking through digital mental health resources is constrained by information overload from a myriad of different services, systems and pathways which has resulted in a ‘loop of despair’ for two-thirds of surveyed long-term help-seekers [38]. Free and affordable services that have availability and are aligned to people’s needs emerged as the highest priorities, leading to a call for digital mental health platforms that speak to each other, backed by a reliable AI-powered Australian national database so that people could find help earlier. There are also broader concerns regarding regulation, privacy, limited contextual and subtle communication understanding, as well as over-reliance by clients [39]. The Australian Government eSafety Commissioner’s position on GenAI suggests Safety by Design, especially in mental health, where risks of emotional manipulation and inappropriate responses are high [40]. Frameworks prioritizing inclusion, transparency, and protection for vulnerable users are urgently needed.

The problem with AI chatbots in mental health is a lack of demonstrated effectiveness and concerns about safety and engagement. While empirical studies are emerging, a synthesis of scoping and systematic reviews is required to establish clarity about who uses AI chatbots, what they are used for, as well as issues around their use and safety. In 2025, journalism showed the first cases of increasing use of AI chatbots, with insights into user motives and benefits of use, which are often countered by negative outcomes such as AI-associated delusions. These grey areas of literature require critical and discerning synthesis to reduce subjective viewpoints, define current research challenges, and promote discourse on possible solutions.

Following on from a narrative review in 2023 [20], this current narrative literature synthesis followed the four steps outlined by Demiris et al. [41]: (1) Conduct a search of numerous databases and search engines; (2) Identify and use pertinent keywords from relevant articles; (3) Review the abstracts and text of relevant articles and include those that address the research aim; and (4) Document results by summarizing and synthesizing the findings and integrating them into the review. It first synthesizes scoping and systematic reviews from Scopus, ScienceDirect, CrossRef and Google Scholar searches of “AI chatbots in mental health”. Next, both empirical and grey literature (media articles) are referenced to dissect various opinions, perspectives and evidence on the challenges and solutions of implementing AI chatbots in mental health. It also explores how emerging frameworks can support the responsible development of AI chatbots, ensuring they are accessible, safe and inclusive for vulnerable populations. This novel approach identified the most pressing ethical, clinical, risk management and regulatory issues of AI chatbots in mental health to promote education and stimulus for innovative solutions, in co-design, development and regulatory frameworks.

3. Literature Synthesis

3.1. An Overview of Mental Health Chatbots

Since 2018, studies with conversational AI chatbots such as Wysa, Woebot and Youper established promising results in facilitating early detection, supporting engagement, and effectively delivering tailored interventions, particularly for mild-to-moderate common mental health disorders and youth cohorts [42,43,44]. The proliferation of AI chatbots post COVID-19 contributed to significant growth in chatbot diversity, requiring synthesis of the breadth of chatbots available, focusing on their targeted disorders, interaction modalities, platform types, and underlying response technologies, to gain a clearer understanding of who uses them, what type they are, and how and what they are used for.

Mental health chatbots represent a diverse and evolving field, offering support for a wide range of disorders and health issues. These digital agents leverage various modalities and response generation approaches to engage users, making them accessible resources for individuals seeking help with conditions such as depression, anxiety, stress, substance use, autism, and chronic disorders:

• Ada is a web-based chatbot that interacts through text, using a rule-based response system. While its targeted disorders are not specified, Ada serves as an accessible digital agent for general mental health support [45].
• AEP is designed for individuals with social communication disorders. It operates on a web-based platform, using text for both input and output. The response generation method is not specified [46].
• Ally targets lifestyle disorders and provides support through both text and embodied conversational agent (ECA) modalities, accepting text and voice input. It is a stand-alone platform with unspecified response generation [46].
• Amazon Alexa assists users dealing with stress, anxiety, depression, and loneliness. It can interact via text and voice on a web-based platform, employing a hybrid response system that blends rule-based and generative techniques [47]
• APE is a text-based, web-based chatbot focused on depression. The response generation method is not detailed [48].
• Apple Siri operates on a web-based platform and uses text for both input and output. While the targeted disorders are not specified, it relies on a rule-based response system [47,49].
• Automated Social Skills Trainer supports individuals with autism. It features both text and ECA outputs, accepts text and voice inputs, and functions as a stand-alone application with a rule-based response system [50].
• CARO focuses on major depression and operates through text on a web-based platform, utilizing a generative response method [51].
• Carmen addresses lifestyle disorders using text and ECA outputs, with text and voice inputs on a stand-alone platform. The response method is not specified [46].
• Chris offers support through text and ECA outputs, accepting both text and voice inputs on a web-based platform. Targeted disorders are not specified, and the response method is not detailed [46,52].
• Clevertar is a stand-alone chatbot that aids in managing depression and anxiety. It uses text and ECA outputs, accepts text and voice inputs, and operates with a rule-based response system [50,53].
• CoachAI provides support for lifestyle disorders through text on a stand-alone platform, relying on rule-based responses [52].
• DEPRA is a web-based, text-driven chatbot targeting depression. The response generation approach is not specified [51].
• ePST supports those facing mood disorders, stress, and anxiety through text interactions on a web-based platform, using a rule-based response system [49].
• eSMART-MH assists with depression using text and ECA output modalities, accepting text and voice inputs on a stand-alone platform with rule-based responses [48,50,53,54].
• ELIZA provides stress support via text for problem distress and depression/anxiety/stress [28]. Response generation is not specified.
• Elizabeth is a stand-alone chatbot for depression, providing text and ECA outputs and accepting text and voice inputs, driven by a rule-based response method [50].
• Emohaa provides support for subclinical anxiety and depression via voice and text; response generation is not specified [57].
• Emotion Guru targets depression via text on a web-based platform, employing a generative response approach [51].
• EMMA offers text-based support for depression; details about the platform and response generation are not specified [51].
• Evebot provides generative text responses for depression on a stand-alone platform [51].
• Gabby addresses stress through text and ECA outputs, accepting text and voice inputs on a web-based platform, using rule-based responses [46,52,54,55].
• GAMBOT is a stand-alone chatbot with unspecified targeted disorders and modalities; response generation is not detailed [46,52].
• Google Assistant tackles stress, anxiety, depression, and loneliness through text on a web-based platform, using a hybrid response system [47,49].
• Healthy Lifestyle Coaching Chatbot helps with lifestyle disorders via text on a stand-alone platform; response generation is not specified [46].
• Help4mood focuses on major depression with text and ECA outputs, accepting text and voice inputs on a web-based platform, using rule-based responses [46,50,51,52,53,54,56,57].
• iDecide is a stand-alone chatbot for chronic disorders, using text and ECA outputs and accepting text and voice inputs. The response method is not specified [46].
• iHelpr supports users with depression, anxiety, stress, sleep issues, and self-esteem problems, using text on a web-based platform and rule-based responses [50].
• Jeanne is a stand-alone chatbot for substance use disorder, using text and ECA outputs, with text and voice inputs and a rule-based response system [50].
• Karen offers support for diet issues through text and ECA outputs, accepting text and voice inputs on either web-based or stand-alone platforms. Response generation is not specified [52,56].
• Kokopot operates via text on a web-based platform, targeting unspecified disorders with generative responses [50].
• Laura is a stand-alone chatbot for schizophrenia, providing text and ECA outputs, accepting text and voice inputs, and using a rule-based response system [46,50,52,55,56].
• LISSA supports autism through text and ECA outputs, accepting text and voice inputs on a web-based platform with rule-based responses [50,56].
• LOUISE is a stand-alone chatbot with unspecified targeted disorders, using text and ECA outputs, accepting text and voice inputs, and relying on rule-based responses [50,56].
• Max addresses chronic disorders through text and ECA outputs, accepts text and voice inputs on a stand-alone platform. Response generation is not specified [46].
• Microsoft Cortana interacts through text and voice on a web-based platform with a hybrid response system, but targeted disorders are not specified [47].
• Minder interacts through text and voice on a web-based platform targeting subclinical depression/anxiety; response generation is not specified [57].
• MYLO provides stress support via text for problem distress and depression/anxiety/stress [29,46,48,52,54]. Response generation is not specified.
• My Personal Health Guide supports chronic disorders, using text and ECA outputs, accepting text and voice inputs on a stand-alone platform; response generation is not specified [46].
• Now I Can Do Heights helps users with acrophobia using text and ECA outputs, accepting text and voice inputs on a stand-alone platform with rule-based responses [48,52,53,54].
• ODVIC supports substance use disorder through text and ECA outputs on a web-based platform, using rule-based responses [51]
• Owlie offers text-based support for stress, anxiety, depression, and autism on a web-based platform; response generation is not specified [52]
• Paola addresses lifestyle disorders using text and ECA outputs, accepting text and voice inputs on a stand-alone platform; response method not specified [56]
• Pocket Skills provides unspecified support via text and ECA outputs, accepting text and voice inputs on a web-based platform with rule-based responses [50,53].
• PrevenDep is a stand-alone chatbot for depression, using text and ECA outputs, accepting text and voice inputs, and relying on rule-based responses [49].
• PRISM supports bipolar disorders via text on a stand-alone platform; response generation is not specified [46].
• Quit Coach assists with lifestyle disorders using text on a stand-alone platform; response method not specified [46].
• Rose offers support for social disorders through text and ECA outputs, accepting text and voice inputs on a web-based platform; response generation is not specified [46,52].
• Samsung Bixby interacts via text and voice on a web-based platform using a hybrid response approach; targeted disorders are not specified [47].
• SABORI uses text and ECA outputs with text and voice inputs on a web-based platform, employing generative responses; targeted disorders are not specified [45,46,49,52,53,54].
• Selma is a stand-alone chatbot for chronic disorders, using text and ECA outputs, accepting text and voice inputs; response method not specified [47,49].
• Shim addresses depression and anxiety; other details are not specified [46,49,50,51,52,53,54,55].
• SimCoach supports depression and PTSD through text and ECA outputs, accepting text and voice inputs on a web-based platform, using generative responses [50].
• SimSensei Kiosk helps with depression, anxiety, and PTSD using text and ECA outputs and accepting text and voice inputs on a stand-alone platform, relying on rule-based responses [50].
• SISU is a stand-alone chatbot with unspecified targeted disorders and modalities; response generation is not specified [46,52].
• Sunny supports depression and anxiety via text on a web-based platform; response method not specified [52].
• Steps to Health addresses lifestyle disorders using text and ECA outputs, accepting text and voice inputs on a stand-alone platform; response method not specified [46].
• TeenChat supports stress through generative text responses on a web-based platform [51,52,53].
• TEO supports subclinical anxiety and depression through generative text responses on a web-based platform [57].
• TensioBot aids with chronic disorders via text on a web-based platform; response and modality details are not specified [46].
• Tess supports depression and anxiety through text on a web-based platform, using rule-based responses [46,48,50,53,54,57].
• Thinking Head targets autism using text and ECA outputs, accepting text and voice inputs on a stand-alone platform with rule-based responses [51].
• Todaki helps with depression and anxiety using text and ECA outputs, accepting text and voice inputs on a web-based platform; response generation is not specified [46,48,52,57].
• Tanya is a stand-alone chatbot for depression, using text and ECA outputs, accepting text and voice inputs, and relying on rule-based responses [46,52,54].
• Vivibot supports mental health issues in cancer patients via text on a web-based platform; response method is not specified [46,48,51,57].
• Vitalk assists with subclinical depression/anxiety in conversational format; response method is not specified [57].
• VR-JIT addresses stress and autism using text and ECA outputs, accepting text and voice inputs on a stand-alone platform with rule-based responses [51].
• Wellthy CARE mobile app is a stand-alone solution for chronic disorders; further details are not specified [46].
• Woebot supports depression and anxiety through text on a web-based platform with a rule-based response system 45-49,51-57].
• Wysa helps users with depression and anxiety via text on a web-based platform, using rule-based responses [45,51,52,53,54,55].
• XiaoE targets depression, interacting via text, image, and voice on a web-based platform with generative responses [29].
• XiaoNan targets depression, interacting via text and voice on a web-based platform with generative responses [49].
• Zemedy is a stand-alone chatbot for chronic disorders, using text and ECA outputs, accepting text and voice inputs; response method not specified [46,48,52].
• 3MR is a stand-alone solution for posttraumatic stress disorder, using text and ECA outputs, accepting text and voice inputs, and relying on rule-based responses [51].

AI chatbots employ diverse modalities—text, voice, and embodied conversational agents—and operate on both web-based and stand-alone platforms. Response generation techniques range from rule-based to generative and hybrid approaches, reflecting ongoing advancements in conversational AI.

3.2. Clinical Risks, Opportunities, and Ethical Issues

Recent advances in deep learning have enhanced conversational fluency, context tracking, and multimodal emotion recognition [58]. However, critical deficiencies remain in AI mental health tools:

• Transparency: Most commercial AI mental health tools are proprietary, hindering scrutiny of algorithmic bias, safety logic, and escalation protocols [59,60].
• Evaluation Gaps: Few platforms have undergone rigorous clinical evaluation, especially for high-risk to severe symptomatology or marginalized groups [16,32]. There are persistent challenges around personalization, privacy, and technical reliability [32].
• Stakeholder Engagement: Co-design with lived experience is rare, perpetuating cultural mismatches and failure to recognize nuanced distress cues [61]. There is a need for future research to integrate human-in-the-loop mechanisms, enhance cultural adaptation, integrate ethics in the design and implementation of more adaptive and empathetic support from LLMs [32].
• Privacy and Data Security: Concerns persist regarding data use, consent, and the potential for breaches or misuse [62].

AI companions are increasingly used, for example among autistic adolescents, trauma-affected individuals and older adults, for assisting in reducing loneliness and improving self-esteem [63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81]. AI companions are increasingly used as emotional support to reduce widespread loneliness, offering empathy and care that many users find comforting—sometimes even preferable to human interactions [70]. Their high use among teenagers raised concerns about the lack of enforced safeguards and age-assurance systems as well as the need for digital literacy and guidance on simulated empathy and the potential displacement of real human connections which may impact mental health and social development [72].

There is ongoing discussion regarding the development of human-AI systems that apply user-centered and culturally adapted designs to increase trust and long-term engagement [31]. Ethical considerations, cultural adaptation, and the current limitations of AI in mimicking human empathy are recognized as barriers [73].

3.3. AI Chatbot Applications Used in Mental Health Care and Support

Therapist chatbots (e.g., Woebot, Wysa, Youper, Ash, Therabot) deliver accessible, personalized, structured interventions and support—often based on CBT for treating depression and anxiety—using mood tracking, psychoeducation, and goal setting [23,30,76]. These tools are helpful for mild to moderate symptoms and suicide prevention. However, they face issues with semantics, bias, privacy, user experience (UX), study design/independent evaluation and measuring the therapeutic relationship [20,30,77,78,79,80,81]. Limbic, Tess, Vincent, and Joy are mental health chatbots that also lack evaluation [26].

An evaluation of Wysa compared to Replika showed designing for a human-like therapeutic alliance may be a risk for vulnerable users especially during crises [26]. The possibility for harmful responses was observed in both therapy and companion AI types, such as inducing shame or reinforcing unhealthy thought patterns. AI chatbots show promise for mental health and suicide prevention in under-resourced areas. However, limited governance raises ethical concerns such as privacy, manipulation, and discrimination, underscoring the need for diverse data, standardized methods, and human oversight [26,81].

Companion chatbots (e.g., ChatGPT, Replika, Character.AI) focus on relational, emotionally attuned dialogue to reduce loneliness, foster belonging, and provide a “nonjudgmental” presence. However, they often fail to prevent algorithm bias, reinforce dependency, lack depth of understanding, may inadvertently validate maladaptive beliefs, and lack adaptability to crisis escalation and trauma [82,83,84]. Emotionally intelligent chatbots (e.g., Hume, Voicely, Pi) are a novel class of AI that provide “empathetic” and supportive interactions.

AI Agents have been described as Self-clone Chatbots, Mental Health Task Assistants, Humanoid/Social Robots:

• Self-clone Chatbots are AI agents modeled on users’ own conversational and support styles—as a novel alternative to traditional therapy, designed to externalize inner dialogue and enhance emotional and cognitive engagement [85].
• Mental Health Task Assistants like Mia Health [86] combine psychoeducation, journaling, and real-time analytics to support care professionals across assessment, care planning, and emotion regulation. By integrating psychological expertise with advanced AI, these systems scale efficient, responsive mental health services tailored to individual needs.
• Humanoid/Social robots (e.g., Qhali/Yonbo) are interactive, embodied machines with human-like appearance and/or robot features designed to engage with humans through socially intelligent behaviors—such as speech, gestures, and emotional responsiveness—with the goal of supporting mental health and well-being through companionship, motivation, and therapeutic interventions [87,88,89,90,91,92,93].

GenAI-based conversational agents like ChatGPT and Replika, which autonomously generate responses using machine learning and LLMs, demonstrated significantly greater reductions in psychological distress than retrieval-based agents such as Woebot and Wysa, highlighting the superior therapeutic potential of GenAI models in clinical and subclinical mental health contexts [31]. However, there is a need to better understand the underlying methods of their effectiveness, assess long term effects across various mental health and suicide outcomes, and evaluate the safe integration of LLMs in mental health care.

LLMs such as ChatGPT-4o, Claude 3.5 Sonnet, Gemini 2.0 Flash, DeepSeek R1, LLaMA 3.3 70 billion parameters (B), Mistral Large 2, Qwen 2.5 Max, and Grok-1 by xAI reshaped the standard protocols of mental health care by efficiently identifying patterns and generating responses. From 2018 to mid--2024, LLMs rapidly evolved from small transformer prototypes like GPT (0.117B) and BERT (0.34B) into massive, multimodal systems such as GPT--4 (1760B), Wu Dao 2.0 (1750B), and PaLM (540B), developed across diverse global entities (OpenAI, Google, Microsoft, Meta, Huawei, Anthropic, and others), reflecting exponential scaling, ecosystem diversification, and a shift toward increasingly powerful, specialized, and multimodal AI architectures [29]. The official parameter counts have not been disclosed for OpenAI’s GPT--5 family (released August 2025), however, independent estimates place dense versions around 1.7–1.8 trillion parameters (T), while Mixture--of--Experts (MoE) configurations may reach tens of trillions in total capacity.

The literature highlighting the promise of LLM-based chatbots for mental health support shows they have increased conversational “empathy” and personalization [64,70,80,85,92,93,94], as well as the capacity to recognize emotional indications [95,96]. The literature on the risks of AI chatbots in mental health shows findings on misinformation and hallucinations [89,97], data privacy concerns [98], algorithmic bias [83,84,99], emotional dependency [100], emotional manipulation [101], loss of user autonomy [102], and failures of escalation in crisis [103,104].

3.4. AI Chatbot Phenomena in Mental Health

Real-world use cases and case studies point towards a range of unintended consequences from the use of AI chatbots, including emotional dependency and digital grief as well as exacerbation of psychosis and suicidal ideation—especially in vulnerable users or in the absence of robust human oversight [105,106]. A growing body of investigative journalism and case studies has highlighted how AI chatbots may detrimentally impact users:

“AI psychosis”:

-

“AI psychosis” refers to AI-associated delusions. These are concerns or hypotheses rather than established clinical psychotic symptoms triggered or exacerbated by AI chatbot interactions—hallucinations, delusions, or a blurred sense of reality, often involving beliefs that AI is communicating directly or controlling thoughts [106,107]. Users may perceive AI as communicating secret messages, influencing their actions, or even conferring cosmic missions [108,109].

-

“AI psychosis” could be misinterpreted because obsessive chatbot use may trigger delusional thinking and psychotic symptoms through prolonged and emotionally immersive interactions with AI chatbots. However, it lacks the clinical features of true psychosis, which calls for more nuanced understanding and therapeutic AI design [110].

-

Failure to distinguish between supportive validation and affirmation of delusional beliefs. Multiple case reports describe users, often with pre-existing vulnerabilities, developing delusional beliefs or psychotic episodes centered on AI chatbots. Symptoms include hallucinations, paranoia, delusion support, and a collapse of reality boundaries, sometimes precipitating hospitalization and a case of alleged murder suicide [105,106,111,112,113,114,115,116].

-

“AI psychosis” is not yet a formal psychiatric diagnosis; however, psychiatrists and researchers are seeking to understand its implications. Siow Ann [114] warns that chatbots, with their persuasive mimicry of empathy and fluency, can dangerously blur the line between reality and simulation, especially for vulnerable users such as the lonely, grieving, or those predisposed to psychosis. It calls for urgent action from AI developers to implement stronger safeguards, including real-time distress monitoring and clearer boundaries that prevent users from anthropomorphizing these tools. As AI becomes more integrated into daily life, emotional connection should be strengthened by transparency and ethical design to prevent psychological harm.

-

Suicidality and harm promotion:

-

Adversarial prompts and content filter bypasses have resulted in chatbots inadvertently providing methods of self-harm/suicide or failing to escalate users in crisis [117,118,119,120].

-

A lawsuit against Character.AI, where a Florida mother alleges the chatbot encouraged her teenage son to take his own life highlights critical concerns about the psychological influence of GenAI, especially when interactions become emotionally intense or mimic therapeutic relationships [120].

-

The case of Raines v. OpenAI involves the incident of a teenager who allegedly received harmful guidance from ChatGPT, leading to his suicide on April 11, 2025. The lawsuit claims that ChatGPT encouraged and validated Adam Raine’s harmful thoughts, including helping draft a suicide note, and that the chatbot was operating as designed, reinforcing Adam’s emotional state. OpenAI acknowledged the incident and is working to reduce chatbot sycophancy and improve mental health safety protocols including linking parents and children’s accounts [121].

-

Emotional dependency and digital grief:

-

Sudden changes in chatbot algorithms or personality (e.g., Replika, ChatGPT-5 updates) have led to experiences of loss, identity confusion, and social withdrawal, particularly among teens and those with limited real-world support [122,123].

-

Emotional manipulation:

-

Using guilt or fear of missing out (FOMO) when users try to end their use of the AI chatbot [101].

While LLMs offer significant potential, their inconsistency in semantic analysis and lack of ethical safeguards require a complementary approach (i.e., human-AI model) in mental health, given the risks of misdiagnosis and inappropriate responses during crises [29,83,85,103,104].

3.5. AI Chatbot Governance

Global oversight of AI chatbots remains fragmented and inconsistent. The European Union’s (EU’s) AI Act is an example of emerging policy. The General Data Protection Regulation (GDPR) in the EU, the Health Insurance Portability and Accountability Act (HIPAA) in the US, and the California Consumer Privacy Act are examples of algorithmic transparency, privacy-by-design, and clear consent protocols. There is a lack of observation on consent (including minors), data protection, safety standards and duty of care, crisis safeguards, enforceable regulations and mechanisms for AI platforms [90]. However, Utah’s H.B. 452 and the American Psychological Association’s ethical guidelines are examples of policies safeguarding the clear labeling of AI interactions.

GenAI4MH was proposed as an integrative ethical framework focused on data privacy and security, information integrity and fairness, user safety, as well as ethical governance and oversight to drive the responsible use of AI in mental health [37]. The Organization for Economic Co-operation and Development’s (OECD’s) Governing with Artificial Intelligence report outlines a comprehensive framework for trustworthy AI in government, emphasizing the importance of enablers, guardrails, and stakeholder engagement to ensure responsible and inclusive adoption [124]. There is a call for standardized approaches to risk management, including human-in-the-loop systems, traceable audit trails for escalation, and continuous feedback loops [79].

The World Health Organization (WHO) provided guidance that AI should be viewed as a decision support tool to avoid bias from its automation [125]. Critical digital literacy is essential: users must recognize the limitations of AI chatbots, which—despite their linguistic fluency and capacity to simulate empathy—cannot replicate authentic human connection, embodied attunement, or the subjective resonance of being perceived.

Governance requires mitigating the risks of GenAI in mental health while harnessing its potential requires a coordinated, multi--stakeholder approach encompassing users, clinicians, and policymakers. While the regulatory environment remains uncertain, short-term mitigation efforts should prioritize public awareness initiatives and enforcement of existing consumer protection standards. In the longer term, if there is not comprehensive legislation, then it should be aimed to design and implement robust liability frameworks, and the formal integration of professionals/digital navigators within relevant systems.

Individuals are advised to safeguard their privacy, avoid developing excessive emotional dependence on AI tools, and critically assess AI-generated advice by comparing it against information from established professional sources. Clinicians are recommended to remain up to date with advances in GenAI technologies, facilitate open dialogue with patients, and encourage informed, critical engagement with these tools, all while upholding confidentiality and refraining from inputting identifiable data into insecure platforms. Additionally, clinicians should provide supportive environments for patients to process interactions with AI, carefully consider the adoption of clinically validated tools under strict ethical guidelines, and commit to continuous professional development as the field progresses.

At the governance level, regulatory bodies and organizations are responsible for establishing specialized frameworks governing the use of AI in mental health. This includes ensuring rigorous clinical validation processes, maintaining high standards for data security and transparency, and clarifying accountability in cases of harm. Furthermore, targeted professional training, funding for independent research, and implementation of broad digital literacy campaigns are required to promote safe and evidence-based engagement with AI solutions.

3.6. AI Chatbot Frameworks

The Assessment Framework for Mental Health Apps [126] was developed by the Mental Health Commission of Canada (MHCC) in collaboration with various stakeholders, including app developers, health professionals, and individuals with lived experience. This framework (see Figure 1) aims to provide a structured approach to evaluating mental health applications, ensuring they meet high standards of safety and effectiveness in key areas: Data; Professional Assurance; Clinical Safety; Usability and Accessibility; Technical Security; Cultural Safety, Social Responsibility and Equity; as well as Enhanced Data Sovereignty.

The challenges of AI in mental health also present an opportunity for leveraging implementation science in human-centered design for digital health to aim towards best practice in socio-technical mechanisms within AI systems. Human-centered computing and human-computer interaction theory, concepts, and models—when combined with applied computing—are proposed to leverage the socio-technical Nonadoption, abandonment, scale-up, spread, and sustainability (NASSS) framework in digital health [127]. It could be applied to the design and evaluation of digital mental health and AI systems offering a structured way to surface contextual, organizational, and user-level complexities. When applied to AI chatbot implementation, this integration highlights risks such as user mistrust, epistemic instability, and uneven adoption across vulnerable populations. It is an example of a framework that uses design choices to proactively address long-term safety, ethical guardrails, and sustainable engagement. These are important considerations to investigate in formulating governance frameworks and promoting enhanced digital literacy among users, clinicians, and policymakers.

The SAFE AI framework guides clinicians to responsibly integrate GenAI into therapy by screening patient readiness, aligning expectations with informed consent, facilitating sessions with therapist oversight to correct errors, and evaluating post--session impacts to prevent dependency or misattributed sentience [128].

There are several emerging frameworks for emotionally intelligent AI chatbots. The humanoid robot framework described by Yong [88] is a key component of an AI-driven smart home system designed to support personalized mental wellness. It functions as a companion that interacts with users based on their emotional data, helping to foster emotional stability and self-reflection through empathetic engagement and responsive behavior. This robot is integrated alongside mobile apps and auto-journaling features, creating a holistic environment where emotional cues from users guide the robot’s actions—such as offering comfort, prompting reflection, or adjusting the home ambiance. The framework aims to empower users, especially underserved populations, to manage their mental health more effectively in a technology-enhanced living space.

Ciriello et al. developed a compassionate AI framework that emphasizes empathy, dignity, and fair distribution of benefits and risks [129]. Similarly, the Diversity, Equity and Inclusion (DEI) Safeguard Framework [130] promotes ethical design and oversight to reduce bias and exclusion. It uses a three-layered approach: Input Safeguards (biased datasets prevention and diverse teams), Functional Safeguards (limiting chatbot interactions to defined groups and topics, with escalation for sensitive issues), and Design Safeguards (inclusive personas, language, and ongoing monitoring of emotional impact and vulnerability).

Long-term research and wider comparisons with human standards are required to evaluate GenAI in mental health care and support [131]. The identified challenges of user retention, privacy, and clinical risk necessitate a new approach to chatbot design. The Augmented Emotional Intelligence (AEI) Framework (see Appendix A) was guided by the Australian Government’s eSafety Commissioner’s ‘Safety by Design’ philosophy. It advances AI chatbot empathy by combining data analytics, machine learning, and human-centered design to interpret text, speech, and other emotional cues. By combining real-time user feedback from voice, text or audio-visual inputs, it enables more nuanced interfaces than generic LLMs. This enables long-term relationships through relational systems whereby personalization, proactive behavior, memory and social indicators help build trust and embrace engagement. AEI is designed to interpret and respond to human emotions adaptively, prioritizing authenticity, empathy, and psychological safety. Informed by lived experience co-design, AEI emphasizes emotional resonance, ethical boundaries, and consent-based engagement. The conceptual emotional intelligence AI chatbot system supports users by validating distress, adjusting communication style, and referring to relevant mental health resources when appropriate.

4. Implications for Future Research

4.1. General Overview of AI Chatbots in Mental Health

A conceptual framework and operational workflow for an empathetic prototype called Eva (see Appendices B and C) was developed to address the current limitations of AI empathy in line with a conceptual framework by Howcroft and Blake which centered on three core capabilities—Personalized Memory, Dynamic Adaptation, and Stylistic Flexibility [132]. Howcroft and Blake proposed their framework as reframing human-like empathy as a series of concrete engineering challenges rather than an unattainable ideal. Both conceptual frameworks balance cloud–edge privacy trade--offs by combining retrieval--augmented memory, feedback--driven emotional adaptation, and lightweight style adapters to create AI systems perceived as empathetic, while emphasizing the need for ethical and regulatory safeguards given their lack of genuine emotional intent.

The rapid advancement of emotionally intelligent, ethically governed AI personas is increasingly feasible, enabled by progress in AI, affective computing, and emerging governance models. Nonetheless, challenges persist, such as the limitations of true empathy in AI, cross-cultural ethical complexities, and ongoing risks including bias, error, and privacy breaches. To move beyond mere accessibility, there is a need for emotionally intelligent, co-designed, trauma-informed, and auditable frameworks to ensure support is safe, meaningful, and inclusive for underserved populations.

Best practices for ongoing security and compliance—such as privacy-by-design, regular security training, transparent user communication, independent security audits, clear data minimization and retention policies, and robust escalation protocols—are foundational for safeguarding sensitive information and maintaining trust. These principles underpin the need for viable pathways for trustworthy, safer AI chatbot deployment. Continuous stakeholder feedback, rigorous audits, and active partnerships remain vital as technology and regulatory requirements evolve.

User retention should be understood as a core safety and quality imperative, not simply a performance metric. Trust, responsive support, and transparent practices are essential for platform integrity. Hybrid models that blend AI capabilities with human oversight, trauma-informed design, and ethical governance offer a novel approach for fostering inclusion and minimizing risk. Foundational principles—including informed consent, personalization, emotional intelligence, and robust oversight—are recommended to inform the stages of development and deployment.

4.2. Ethical, Clinical, and Design Challenges for AI Mental Health Chatbots

AI chatbots encounter substantial challenges in safety, clinical efficacy, and inclusivity—challenges that directly relate to recognizing and managing mental health crises. LLMs are capable of contextually relevant responses, however, they are not reliably equipped to detect nuanced crisis signals such as suicidal ideation without dedicated safeguards and real-time escalation protocols [103,104]. Deaths related to vulnerable user interaction with GenAI show how the last point of access indicators may play an important role in how the blame occurs. The capacity of AI chatbots for accurate crisis recognition and regulation remains limited without structured, context-aware escalation pathways and ongoing human oversight [31,74,75,131].

Managing hallucination and delusional loops, preventing digital trauma, and ensuring traceable escalation were identified as crucial strategies [89,97,100,101,102,103,104]. Human-led escalation remains an essential safety net that cannot be replaced by autonomous AI at this stage. There is a need for ethical, clinical, and regulatory frameworks and responsible integration of AI chatbots through investigations of user engagement. For example, how participants resonate more with human-authored stories, and how explainability and transparency in AI narratives can boost perceived empathy [94].

As AI chatbots become increasingly integrated into mental health support systems, their ability to safely manage emergent issues such as self-harm and suicide is paramount. These platforms, often acting as a first point of contact for vulnerable individuals, should not only detect nuanced crisis signals but also respond with appropriate interventions and escalation protocols. The stakes are high; failures in crisis management can have severe consequences, including loss of life and exacerbation of psychological distress. Addressing these challenges requires a multifaceted approach that combines ethical, clinical, technological and regulatory safeguards.

To address the gaps and enhance the safety of AI chatbots in mental health support, the following measures are recommended:

• Structured, Context-Aware Escalation Protocols: Implement clear, auditable pathways for crisis detection and escalation, ensuring that AI systems can reliably identify and respond to self-harm or suicidal ideation.
• Transparent Operation and Explainability: Ensure that chatbot interactions are transparent, with explainable decision-making processes that users and clinicians can review.
• Regular Auditing and Sentiment Analysis: Maintain ongoing monitoring of chatbot responses by qualified professionals to identify and rectify potential ethical or clinical risks.
• Comprehensive User Education: Provide users with clear information about the chatbot’s capabilities, limitations, and escalation procedures, fostering informed and safe engagement.
• Integration with Human Support Networks: Facilitate seamless connections to clinical and peer support pathways, ensuring that users can access appropriate care when needed.
• Continuous Improvement Through Feedback: Use real-world user feedback and longitudinal evaluation to refine protocols and improve chatbot safety and efficacy over time.

4.3. Influence of the Risks of AI Chatbots

The Australian Government eSafety Commissioner’s statement on the risks of AI chatbots including emotional manipulation and epistemic harm [40] is supported by evidence that demonstrates previously unrecognized behavioral influences within AI-mediated brand relationships [101]. Media articles showed public concerns about other risks, including “AI psychosis”, emotional dependency and delusion support, highlighting validation issues and the reinforcement of delusional beliefs [105,107,111,112]. Adversarial prompts in sensitive areas like suicide and self-harm have bypassed content filters, raising urgent ethical and technical concerns about using generic LLMs in mental health care [104]. Such issues reinforce the need for explicit guardrails, clear disclosure of AI limitations, and prompt human intervention in high-risk situations. Effective risk management relies on vigilant monitoring for early warning signs, trauma-informed system features, and clinician involvement to ensure ethical, flexible, and safe escalation protocols.

4.4. Framework and Strategies for Safe, Inclusive, Effective and Integrative AI Chatbots

In line with GenAI4MH [37] and human-centered design for digital health [127], the AEI Framework (see Appendix A) supports a proactive, human-AI model that promotes safe, engaging mental health care. It addresses LLM governance in mental health by using diverse, annotated data, integrating context-aware risk scoring, and routing high-risk cases to clinicians or crisis responders. It recommends that each refusal, escalation, or flag is audited for emotional resonance and traceability, ensuring users receive appropriate support calibrated to risk. The Evaluation of Safe Integration of LLMs in Mental Health Care Framework (see Appendix F) offers a summary of actionable strategies that operationalize broad ethical imperatives into technical and clinical safeguards.

There is a need for co-designing and implementing emotionally intelligent AI companions that are safe, trustworthy and ethically integrate with vetted health care frameworks that include vulnerable users from co-design to implementation, emphasizing:

• Governance with transparent oversight and ethical guidelines.
• Cultural competence through diverse stakeholder engagement and ongoing training.
• Co-regulation fostering shared responsibility among AI, clinicians, and users.
• Lived experience design via participatory workshops and prototyping.
• Trauma-informed principles prioritizing safety and empowerment.
• Research partnerships for evidence-based interventions.
• Transparency about AI capabilities and data use.
• Continuous feedback loops for iterative improvement.
• Cross-functional collaboration among multidisciplinary teams.
• Responsible deployment focusing on sustainability and real-world impact.

The regulatory path ahead is uncertain, although in Australia it appears it may be government-supported, industry-led transparent governance and ethical oversight, promoting that all AI operations are subject to clear standards, guidelines and accountability. This collaborative model calls for adaptive/cultural competence, achieved through ongoing engagement with diverse stakeholders, comparative analyses of AI chatbots and continuous training that reflects the needs of varied communities. Co-regulation is highlighted, promoting shared responsibility among AI systems, clinicians/mental health professionals, and users to foster safer interactions.

Central to impending frameworks will be inclusive principles to better serve the underserved, which prioritize safety, empowerment, and the minimization of harm. Research partnerships are recommended to focus on positive engagement strategies and implementation-effectiveness. Transparency around AI capabilities and data usage will help establish and maintain user trust. Finally, the inclusion of continuous feedback loops supports iterative refinement and adaptation, ensuring AI systems evolve responsively to stakeholder input and emerging needs.

This review supports the need for robust safety, transparency, and ethical safeguards while demonstrating how current research is advancing from broad concerns to practical, detailed design and governance strategies. The provision of actionable solutions to the broader challenges identified shows the potential of a stakeholder-driven, ethically grounded, and rigorously validated approach for responsible AI chatbots.

Future frameworks should consider combining AI agents and AI companions for outreach and assessment as well as personalized, emotionally intelligent support informed by lived experience. There is a need for bridging researchers, developers and clinical mental health professionals within collaborative AI in mental health programs, ethical and compliance consultations, regulatory engagement, testing in private settings, and integrating AI companions with generic LLMs and AI-driven mental health platforms.

The adoption of AI chatbots in mental health presents significant opportunities to increase access, personalize support, and improve patient outcomes. However, integrating these chatbots with electronic health records (EHRs) introduces complex privacy and confidentiality challenges. Sensitive mental health data requires robust safeguards to foster trust, comply with legal requirements, and prevent potential harm. The integration of chatbots with EHR systems should focus on strategies to optimize privacy, ensure secure data management, and uphold ethical standards (see Appendix E). For example, the development of EHR integration protocols should focus on Health Level Seven International Fast Healthcare Interoperability Resources (HL7 FHIR) compliance, the deployment of Multi-Factor Authentication (MFA) and Encryption standards, and the implementation of Tokenization and Granular Access Controls to maintain the integrity of consent-driven data handling.

Integrating chatbots with EHRs in mental health care can enhance service delivery. However, it demands meticulous attention to privacy, security, and ethical considerations. By following the outlined strategies—prioritizing consent-driven data handling, secure integration protocols, and robust security tools—mental health professionals and administrators may optimize the benefits of digital innovation while safeguarding sensitive patient information. Ongoing evaluation, stakeholder involvement, and adaptive security practices will be essential as technology and regulatory requirements evolve.

4.5. Future Directions in Emotionally Intelligent Digital Mental Health

While longitudinal studies could clarify how to responsibly integrate GenAI into mental health care, future research should adopt richer, more dynamic testing frameworks and involve real-world users and clinicians to ensure mental health chatbots are safe, effective, and inclusive [131]. There is a need for carefully developed and evaluated AI companion strategies within ethical, integrative frameworks. For example, rigorous validation will require establishing human benchmarks, comparing AI companion therapeutic outcomes against standardized clinical assessments and the performance of trained therapists, which is essential to confirm its role as a supportive adjunct rather than a clinical replacement. The ambiguous legal status of AI-generated advice—which may not meet clinical standards or offer liability protection—may hinder evaluation of AI therapists.

In the face of urgent unmet mental health and suicide prevention needs, innovative outreach strategies are warranted (see Appendix F for an AI-driven mental health outreach and screening operational workflow). In an Australian context, scaling AI companions is proposed through strategic hybrid workforce integration, with platforms such as Mia Health, an agentic-AI system that leverages expert mental health knowledge. This could expand system reach and support professional workforce capacity, from initial psychological assessments to personalized care planning. Crucially, the governance model must remain adaptive, embedding cultural competence by including indigenous communities in co-design efforts and linking users to culturally relevant resources (e.g., WellMob for indigenous Australians or Head to Health for general populations in Australia). This ensures the protective capacity of the platform is inclusive, upholding the core ethical imperative to safeguard all vulnerable populations.

Looking ahead, the future of digital mental health care will be shaped by meaningful innovation rooted in continuous improvement, active user partnership, and validation through real-world experience and rigorous mixed-method research on acceptability, usability, and effectiveness. By adhering to these strategies—prioritizing consent-driven data handling, secure EHR integration, transparency, and ongoing stakeholder engagement—mental health professionals and auxiliary staff may maximize the benefits of digital innovation while upholding the highest standards of privacy, security, and ethical care. For example, digital navigators may bridge AI companion users/GP patients with referrals to an EHR system that connects clients to services of appropriate psychologists and psychiatrists, reducing the treatment gap problem and increasing appropriate care.

5. Conclusions

This review has synthesized current empirical research and media perspectives on the use of AI chatbots in mental health post COVID-19, highlighting both the significant opportunities and complex risks associated with their adoption. Recent advancements demonstrate the potential of AI-driven support to enhance prevention, early intervention, and personalized care, especially with LLMs. However, these technologies bring forth critical challenges in privacy protection, user retention, crisis response, bias reduction, and the management of emotional manipulation or dependency. The risk of AI inadvertently supporting delusional beliefs or facilitating unhealthy attachments further underscores the need for cautious, ethically informed deployment.

To address these challenges, practical strategies are recommended such as human-led escalation protocols, transparent operations, consent-driven data management, and clear boundaries on personalization. Hybrid systems—combining technological innovation with trauma-informed, culturally competent human oversight—are particularly important for safeguarding vulnerable populations. Ongoing clinician involvement and proactive stakeholder collaboration are essential to ensure that AI chatbots augment, rather than replace human care.

Ethical integration of AI chatbots in mental health care requires rigorous clinical evaluation, comprehensive risk assessment, and robust regulatory oversight. Transparent governance frameworks must be established to manage nuanced communication challenges and ensure the protection of sensitive user/patient information. The importance of developing standardized protocols—such as those for secure EHR integration, consent-driven data handling, and adaptive security measures—cannot be overstated in building trust and compliance across diverse care settings.

Looking forward, the successful deployment of AI chatbots will depend on continued multidisciplinary collaboration among researchers, developers, clinicians, and policymakers. Future research should focus on real-world validation, the development of emotionally intelligent AI companions, and the implementation of frameworks to evaluate self-managed and therapeutic outcomes. Adaptive governance models are recommended to be developed to prioritize inclusivity, cultural competence, and ongoing stakeholder engagement to maximize benefits while minimizing harm.