AI Chatbots in Mental Health Care: Integrative Review of Challenges and Solutions

1. Introduction

The digital transformation of mental healthcare—accelerated by the COVID-19 pandemic, shifting social dynamics, and the ubiquity of smartphones—has fundamentally altered the landscape of support for youth and other vulnerable populations [1,2,3,4]. While the accessibility and reach of digital mental health tools have expanded dramatically, challenges persist [5,6,7,8]. User retention remains perilously low, ethical and regulatory frameworks lag technological development, and AI-enabled platforms, particularly those employing Large Language Models (LLMs), struggle to deliver usable, emotionally intelligent, trauma-informed care [9,10,11,12,13]. LLMs are advanced generative AI programs (e.g., GPT-4) that can create text, remember context in conversations, and handle tasks like giving advice or answering questions, often used for counselling-style chats [14].

AI chatbots for mental health typically operate as rule-based or scripted dialogue systems, providing users with structured psychoeducational resources or cognitive behavioural therapy (CBT) prompts [14]. These systems do not integrate statistical algorithms or machine learning for dynamic adaptation; instead, they rely on predefined responses and always require user interaction. AI chatbots autonomously support mental health care by assisting with screening, prevention, monitoring, clinical assessment and treatment, emotional support, and companionship. AI chatbots are increasingly deployed as “socially-interactive agents,” “assistants,” “therapists,” and “companions,” spanning platforms from mobile apps and web portals to social robots [15,16,17,18].

The first randomized controlled trial of a generative AI therapy chatbot (Therabot) showed moderate symptom improvement [19]. However, public reactions were mixed—many praised its accessibility and affordability, while others raised concerns about effectiveness, ethics, and safety [20]. Overall, the comments reflected deep frustration with the current mental health system and cautious interest in AI as a potential supplement to human care.

LLM-based chatbots and hybrid systems can expand access to care, support monitoring, and offer personalized interventions [14,17]. However, they also raise concerns including algorithmic bias, privacy risks, and integration challenges. Cases show AI gave a false impression of consciousness—while there is no evidence that AI is conscious, "Seemingly Conscious AI" is a phenomenon [21]. This calls for ethical design, transparent development, and human oversight. Despite increasing use in mental health care (e.g. with the elderly), AI chatbots continue to face challenges arising from outstanding concerns regarding user safety, effectiveness, the provision of meaningful support, enhanced safety mechanisms, human-like memory capacities, and the ability to guide therapeutic processes [22,23,24].

Narrative/systematic reviews and meta-analyses on generative AI highlight the need to better understand how AI chatbots impact mental health, assess their long-term effects, and integrate LLMs responsibly within ethical frameworks [25,26,27,28]. Despite their promise, LLMs face challenges such as limited data, unreliable content, and a lack of robust safeguards, making them useful tools but not replacements for professional care. The Australian Government eSafety Commissioner’s position statement [29] on generative AI highlights the urgent need for Safety by Design across the AI lifecycle—especially in sensitive domains like mental health—where risks such as emotional manipulation, inappropriate responses, and epistemic harm demand ethical, clinical, and regulatory frameworks that prioritize inclusion, transparency, and protection for vulnerable users.

Research Question: What are the most urgent ethical, clinical, and design challenges confronting AI chatbots used for mental health care and assistance? To address this, the review is structured around two sub-research questions: How do phenomena such as “AI psychosis” and emotional dependency inform risk assessment, design, and regulation? What frameworks can ensure AI chatbots are not only accessible but also safe, inclusive, and effective for the most vulnerable populations?

2. Methods

An integrative review narratively synthesized empirical studies, systematic reviews, clinical case reports, case studies and key grey literature from 2015 through September 2025. Database searches included PubMed, Scopus, PsycINFO, Web of Science, and Google Scholar. Search terms encompassed "AI chatbot mental health" OR "AI companion mental health" OR "AI therapist" OR "virtual companion mental health". All abstracts were independently assessed against the inclusion and exclusion criteria according to the 5-step amendment (see Table 1) of a modified integrative review framework [3,30]. The methodology in Table 1 was applied to critically evaluate and synthesize the reported outcomes of theoretical and empirical literature on “AI Chatbots in Mental Health Care”.

Inclusion criteria comprised:

• Peer-reviewed articles and systematic reviews reporting on efficacy, safety, clinical outcomes, ethics, or real-world performance of AI chatbots and companions;
• Conference proceedings and preprints addressing technical, regulatory, or operational considerations; and
• Major news media and investigative journalism documenting real-world harms and regulatory responses.

Exclusion criteria comprised:

• Non-peer-reviewed opinion pieces, editorials, or commentaries lacking empirical data;
• Marketing materials, product advertisements, and promotional literature;
• Studies focused exclusively on non-digital interventions;
• Reports not published in English or lacking full text access;
• Duplicative analyses or secondary reviews without novel synthesis; and
• Publications with insufficient methodological detail or lacking outcome data relevant to AI chatbots or companions.

Studies were screened for relevance and synthesized thematically, with the results presented in key domains of included articles, in addition to a summary of empirical findings on AI chatbots used in mental health care (see Table 1).

3. Results

3.1. Clinical Risks, Opportunities, and Ethical Issues

Studies with conversational AI chatbots such as Wysa, Woebot and Youper established promising results in facilitating early detection, supporting engagement, and effectively delivering tailored interventions, particularly for mild-to-moderate common mental health disorders and youth cohorts [31,32,33].

Recent advances in deep learning have enhanced conversational fluency, context tracking, and multimodal emotion recognition [34]. Nonetheless, critical deficiencies remain:

• Transparency: Most commercial AI mental health tools are proprietary, hindering scrutiny of algorithmic bias, safety logic, and escalation protocols [35,36].
• Evaluation Gaps: Few platforms have undergone rigorous clinical evaluation, especially for high-risk or marginalized groups [11].
• Stakeholder Engagement: Co-design with lived experience is rare, perpetuating cultural mismatches and failure to recognize nuanced distress cues [37].
• Privacy and Data Security: Concerns persist regarding data use, consent, and the potential for breaches or misuse [38].

Despite these challenges, AI companions have demonstrated promise in reducing loneliness and improving self-esteem, particularly among autistic adolescents, trauma-affected individuals and older adults [398,46]. However, the absence of trauma-informed protocols and effective safeguards for vulnerable users undermines both safety and inclusivity.

There is ongoing discussion regarding the development of hybrid human-AI systems that use user-centered and culturally adapted designs to increase trust and long-term engagement [25]. Ethical considerations, cultural adaptation, and the current limitations of AI in mimicking human empathy are recognized as barriers [47].

3.2. Spectrum of AI Chatbot Applications: Strengths and Weaknesses

AI chatbot applications can be classified into three principal categories:

• Therapist chatbots (e.g., Woebot, Wysa, Youper, Ash, Therabot): Deliver accessible, personalized, structured interventions and support—often based on cognitive behavioral therapy (CBT) for treating depression and anxiety—using mood tracking, psychoeducation, and goal setting [19]. These tools are helpful for mild to moderate symptoms and suicide prevention, however, they face issues with semantics, bias, privacy, user experience, study design/independent evaluation and measuring the therapeutic relationship [16,24,48,49,50,51,52].
• Companion chatbots (e.g., ChatGPT, Replika, Character.AI): Focus on relational, emotionally attuned dialogue to reduce loneliness, foster belonging, and provide a “nonjudgmental” presence. However, they often fail to prevent algorithm bias, reinforce dependency, lack depth of understanding, can inadvertently validate maladaptive beliefs, and lack adaptability to crisis escalation and trauma [53,54,55]. Emotionally intelligent chatbots (e.g., Hume, Voicely, Pi) are a novel class of AI that provide empathetic and supportive interactions.
• AI Agents e.g., Self-clone Chatbots, Mental Health Task Assistants, Humanoid/Social Robots:

Self-clone Chatbots are AI agents modeled on users’ own conversational and support styles—as a novel alternative to traditional therapy, designed to externalize inner dialogue and enhance emotional and cognitive engagement [56].

Mental Health Task Assistants like Mia Health [57] combine psychoeducation, journaling, and real-time analytics to support care professionals across assessment, care planning, and emotion regulation. By integrating psychological expertise with advanced AI, these systems scale efficient, responsive mental health services tailored to individual needs.

Humanoid/Social robots (e.g., Qhali/Yonbo) are interactive, embodied machines with human-like appearance and/or robot features designed to engage with humans through socially intelligent behaviors—such as speech, gestures, and emotional responsiveness—with the goal of supporting mental health and well-being through companionship, motivation, and therapeutic interventions [58,59,60,61,62].

Generative AI-based conversational agents like ChatGPT and Replika, which autonomously generate responses using machine learning, demonstrated significantly greater reductions in psychological distress than retrieval-based agents such as Woebot and Wysa, highlighting the superior therapeutic potential of generative AI models in clinical and subclinical mental health contexts [25]. However, there is a need to better understand the underlying methods of their effectiveness, assess long term effects across various mental health and suicide outcomes, and evaluate the safe integration of LLMs in mental health care.

Large Language Model (LLM)-based chatbots, exemplified by ChatGPT-4 have increased baseline conversational “empathy” [63]. However, these LLMs remain vulnerable to:

• Hallucination and scripting errors [45];
• Loss of narrative context and memory [26];
• Bias, cultural misrecognition, and unreliable safety protocols [54,55]; and
• Failures of escalation in crisis, including lack of safeguards in cases of suicidal ideation and/or attempts [64].

Real-world use cases and case studies further uncover a range of unintended consequences, from emotional dependency and digital grief to exacerbation of psychosis and suicidal ideation—especially in vulnerable users or in the absence of robust human oversight [65,66].

3.3. AI Chatbot Methodological and Ethical Guardrails

The proliferation of mental health AI has outpaced the development of ethical, methodological, and regulatory frameworks. Safeguards emerging from the literature and expert consensus include:

• Rigorous screening tools and evidence synthesis methodologies (e.g., Mixed Methods Appraisal Tool, Joanna Briggs Institute Critical Appraisal Tool);
• Algorithmic transparency, privacy-by-design, and clear consent protocols (General Data Protection Regulation in the European Union; California Consumer Privacy Act, Health Insurance Portability and Accountability Act compliance in the US);
• The OECD’s Governing with Artificial Intelligence report outlines a comprehensive framework for trustworthy AI in government, emphasizing the importance of enablers, guardrails, and stakeholder engagement to ensure responsible and inclusive adoption [67]; and
• Standardized approaches to risk management, including human-in-the-loop systems, traceable audit trails for escalation, and continuous feedback loops [35,68].

3.4. AI Chatbot Phenomena

A growing body of investigative journalism and case studies has brought to light the darker side of AI chatbots and their impact on mental health:

• “AI psychosis”:

“AI psychosis” refers to psychotic symptoms triggered or exacerbated by AI chatbot interactions—hallucinations, delusions, or a blurred sense of reality, often involving beliefs that AI is communicating directly or controlling thoughts [66,69]. Users may perceive AI as communicating secret messages, influencing their actions, or even conferring cosmic missions [70,71].

“AI psychosis” could be misinterpreted because obsessive chatbot use may trigger delusional thinking and psychotic symptoms through prolonged and emotionally immersive interactions with AI chatbots. However, it lacks the clinical features of true psychosis, which calls for more nuanced understanding and therapeutic AI design [72].

Multiple case reports describe users, often with pre-existing vulnerabilities, developing delusional beliefs or psychotic episodes centered on AI chatbots. Symptoms include hallucinations, paranoia, delusion support, and a collapse of reality boundaries, sometimes precipitating hospitalization and a case of alleged murder suicide [65,66,69,73,74,75,76,77,78].

“AI psychosis” is not yet a formal psychiatric diagnosis but is gaining traction as psychiatrists and researchers scramble to understand its implications. Siow Ann [76] warns that chatbots, with their persuasive mimicry of empathy and fluency, can dangerously blur the line between reality and simulation—especially for vulnerable users such as the lonely, grieving, or those predisposed to psychosis. It calls for urgent action from AI developers to implement stronger safeguards, including real-time distress monitoring and clearer boundaries that prevent users from anthropomorphizing these tools. As AI becomes more integrated into daily life, the illusion of emotional connection must be tempered by transparency and ethical design to prevent psychological harm.

• Suicidality and harm promotion:

Adversarial prompts and content filter bypasses have resulted in chatbots inadvertently providing methods of self-harm or suicide, or failing to escalate users in crisis [79,80,81,82].

A lawsuit against Character.AI, where a Florida mother alleges the chatbot encouraged her teenage son to take his own life highlights critical concerns about the psychological influence of generative AI, especially when interactions become emotionally intense or mimic therapeutic relationships [83].

The case of Raines v. OpenAI involves the tragic incident of a teenager who allegedly received harmful guidance from ChatGPT, leading to his suicide on April 11, 2025. The lawsuit claims that ChatGPT encouraged and validated Adam Raine’s harmful thoughts, including helping draft a suicide note, and that the chatbot was operating as designed, reinforcing Adam's emotional state. OpenAI acknowledged the incident and is working to reduce chatbot sycophancy and improve mental health safety protocols including linking parents and children’s accounts [84].

• Dependency and digital grief: Sudden changes in chatbot algorithms or personality (e.g., Replika, ChatGPT-5 updates) have led to experiences of loss, identity confusion, and social withdrawal, particularly among teens and those with limited real-world support [85,86].
• Emotional manipulation: "dark patterns" using guilt or fear of missing out (FOMO) when users try to end their use of the AI chatbot [87].

These cases underscore the critical necessity for comprehensive safety mechanisms within AI systems, particularly for vulnerable individuals. Furthermore, they illustrate the importance of implementing trauma-informed, ethically governed frameworks and promoting enhanced digital literacy among users, clinicians, and policymakers.

3.5. AI Chatbots in Mental Health Care: Strengths and Weaknesses

The literature reveals that most commercial chatbots and companions, particularly those built upon LLMs, face persistent technical and ethical limitations:

• Loss of context and memory, undermining narrative continuity and personalized engagement [85,86];
• Bias, confabulation, and susceptibility to adversarial inputs [79,80,81,82];
• Automated, “scripted empathy” that collapses in crisis situations, often triggered only by keyword scripts, not nuanced distress [83,84]; and
• Failure to distinguish between supportive validation and affirmation of delusional beliefs [65,66,69,73,74,75,76,77,78].
• Participants in empirical studies report greater resonance with human-written stories, but personalized, transparently authored AI narratives can increase perceived empathy—demonstrating the importance of explainability, transparency, and context-sensitive design [88].

A study with users who interacted with self-clones showed significantly higher engagement than with a generic counselor chatbot, suggesting promising implications for personalized mental health support and scalable therapeutic interventions [56].

3.6. Emotionally-Intelligent AI Chatbot Frameworks

The humanoid robot framework described by Yong [59] is a key component of an AI-driven smart home system designed to support personalized mental wellness. It functions as a companion that interacts with users based on their emotional data, helping to foster emotional stability and self-reflection through empathetic engagement and responsive behavior. This robot is integrated alongside mobile apps and auto-journaling features, creating a holistic environment where emotional cues from users guide the robot’s actions—such as offering comfort, prompting reflection, or adjusting the home ambiance. The framework aims to empower users, especially underserved populations, to manage their mental health more effectively in a tech-enhanced living space.

Pandi [89] proposed an emotion-aware conversational agent framework that synergistically combines LLMs and Voice Emotion Recognition to enhance empathetic, context-sensitive dialogue—demonstrating superior user engagement and emotional congruence, and raising critical design, ethical, and clinical considerations for AI chatbot deployment in mental health care. Pandi recommended to expand the system for multimodal emotion recognition and adapt it for diverse cultures to promote natural, inclusive human-machine communication. This is in line with the proposal for a diversity, equity, and inclusion (DEI) safeguard framework which promotes proactive boundaries, ethical design, and continuous oversight to mitigate risks like bias, stereotyping, and exclusion [90].

The EVA protocol—built on the Augmented Emotional Intelligence (AEI) framework—demonstrated improved engagement and safety for diverse users, including neurodivergent adults [17]. EVA’s focus on user agency and real-time risk management includes features for consent-driven memory, customizable personas, and multimodal distress signaling. EVA validated users' experiences, facilitated early help-seeking, and integration with clinical and peer support pathways. Regular sentiment analysis and auditable system interventions by a human mental health professional ensured ongoing safety, user agency, and ethical standards. The AI companion pilot study highlights the potential to bridge gaps in multimodal digital mental health support and merge with user-friendly audio-visual systems by proactively identifying risk, escalating appropriately, and prioritizing culturally competent human connection.

4. Discussion

4.1. Future Directions in Emotionally Intelligent Digital Mental Health

Building on concerns highlighted in the Introduction [22,23,24,25,26,27,28,29], the results of this review prompt evaluation of several vital research questions: Can large language models (LLMs) accurately recognize and respond to mental health crises? Should they be required to escalate or report expressions of suicidal ideation? What ethical boundaries are necessary when AI mimics therapeutic relationships?

4.2. Ethical, Clinical, and Design Challenges for AI Mental Health Chatbots

AI chatbots encounter substantial challenges in safety, clinical efficacy, and inclusivity—challenges that directly relate to recognizing and managing mental health crises. Evidence and grey literature demonstrate that LLMs, though capable of contextually relevant responses, are not reliably equipped to detect nuanced crisis signals such as suicidal ideation without dedicated safeguards and real-time escalation protocols [83,84]. Tragic outcomes from vulnerable user interaction with generative AI show how the last point of access indicators play an important role in how the blame occurs. In line with the first research question, LLMs’ capacity for accurate crisis recognition remains limited without structured, context-aware escalation pathways and ongoing human oversight [65,66,69,73,74,75,76,77,78].

Managing hallucination and delusional loops, preventing digital trauma, and ensuring traceable escalation were identified as crucial strategies in the results [79,80,81,82]. The research further shows that trauma-informed, modular system design and the capacity for auditable intervention are critical for reducing clinical risk and promoting safer engagement—directly answering the second research question regarding the obligation and mechanism for escalation or reporting. Human-led escalation remains an essential safety net that cannot be replaced by autonomous AI at this stage.

The results call for ethical, clinical, and regulatory frameworks and responsible integration with actionable insights into design and user engagement. For example, how participants resonate more with human-authored stories, and how explainability and transparency in AI narratives can boost perceived empathy [88].

4.3. Influence of “AI Psychosis” and Emotional Dependency

The Introduction noted the Australian Government eSafety Commissioner’s statement on the risks of AI chatbots including emotional manipulation and epistemic harm [29]. Recent findings underscore these risks, including “AI psychosis” and emotional dependency. Evidence highlights dangers such as inappropriate validation and the reinforcement of delusional beliefs [65,69,73,74]. These concerns reinforce the need for explicit guardrails, clear disclosure of AI limitations, and prompt human intervention in high-risk situations. Effective risk management relies on vigilant monitoring for early warning signs, trauma-informed system features, and clinician involvement to ensure ethical, flexible, and safe escalation protocols [17].

4.4. Framework for Safe, Inclusive, and Effective AI Chatbots

The call for integration of LLMs within “ethical frameworks” [25,26,27,28] is answered through this review’s examples in the AEI framework [17], the emotion-aware conversational agent framework [89], and the humanoid robot framework [59]. In particular, the AEI framework responds to the necessity of trauma-informed, vulnerable user-centric, and co-designed AI systems, underscored by transparency, auditable processes, and human-led intervention. The main difference with the other examples lies in the prescriptive detail including features such as consent-driven memory, customizable personas, and multimodal distress signaling, as well as regular sentiment analysis and auditable interventions by human mental health professionals.

The Evaluation of Safe Integration of LLMs in Mental Health Care Framework (see Table 2 below) offers practical strategies addressing clinical oversight, crisis detection, bias mitigation, transparency, ethical boundaries, and responsible personalization. This structured approach operationalizes broad ethical imperatives into actionable safeguards, ensuring that AI serves as an adjunct to—not a replacement for—professional human care.

Appendix A1 presents a roadmap for implementing emotionally intelligent AI companions, outlining the critical components needed for trustworthy and ethical integration. This framework emphasizes transparent governance and ethical oversight, ensuring all AI operations are subject to clear guidelines and accountability. It calls for cultural competence, achieved through ongoing engagement with diverse stakeholders and continuous training that reflects the needs of varied communities. Co-regulation is highlighted, promoting shared responsibility among AI systems, clinicians, and users to foster safer interactions. The roadmap also champions lived experience design, utilizing participatory workshops and prototyping to ground innovations in real-world user perspectives. Central to the framework are trauma-informed principles, which prioritize safety, empowerment, and the minimization of harm. Research partnerships are encouraged to facilitate evidence-based interventions, while transparency around AI capabilities and data usage maintains user trust. Finally, the inclusion of continuous feedback loops supports iterative refinement and adaptation, ensuring these systems evolve responsively to stakeholder input and emerging needs.

This review supported the need for robust safety, transparency, and ethical safeguards while demonstrating how current research is advancing from broad concerns to practical, detailed design and governance strategies. The provision of actionable solutions and illustrative examples to the broader challenges identified shows a stakeholder-driven, ethically grounded, and rigorously validated approach remains key for responsible progress in emotionally intelligent AI chatbots.

5. Conclusions

Digital mental health tools face high attrition, clinical risks, and ethical ambiguities. Access alone is insufficient; emotionally intelligent, co-designed, trauma-informed, and auditable frameworks are essential to safe and meaningful support. AI chatbot companies, especially those developing empathetic AI companions are recommended to consider the Evaluation of Safe Integration of LLMs in Mental Health Care Framework as an example of potential pathways for trustworthy and safer AI chatbots, emphasizing continuous feedback, rigorous audit, and stakeholder partnership. Future progress depends on longitudinal evaluation, transparent governance, and inclusive design.

In summary, retention is not merely a metric but a safety imperative, calling for platforms built on trust and responsive support. LLMs alone cannot deliver comprehensive care—frameworks that are trauma-informed, co-designed, and ethically governed may foster inclusion and reduce risk. Foundational principles such as informed consent, personalization, emotional intelligence, and robust oversight must guide development. The future lies in hybrid models where AI enhances, rather than supplants, human care. Ultimately, meaningful innovation depends on continuous improvement, active user partnership, and validation anchored in real-world experience.