Embodied AI as Coach: Integrating Facial Expression and Heart-Rate Biometrics into Adaptive Coaching Systems

1. Introduction

AI coaching and mentoring platforms are rapidly expanding, offering scalable and personalized support (Vistorte et al., 2024). Yet, current systems overwhelmingly rely on text or voice, omitting nonverbal cues fundamental to human communication (Chan, 2025; Meinlschmidt et al., 2025). Human coaches interpret micro-expressions, heart-rate variability, and body language to assess readiness, stress, or disengagement, dimensions current AI systems overlook. This absence of embodied awareness makes AI agents appear mechanical and detached (Weiß et al., 2024).

Integrating facial expression analysis and biometric signals can enhance empathic responsiveness, enabling AI to sense stress, regulate interactions, and dynamically adapt coaching strategies (Frey, 2014; Bridgeman & Hayes, 2023). Such embodied AI has the potential to deepen trust, engagement, and effectiveness, particularly in sensitive contexts such as leadership development, education, and therapy. This study positions embodied AI as a necessary step toward emotionally intelligent, adaptive mentorship systems.

2. Problem Statement and Research Questions

The absence of embodied awareness creates an empathy gap, limiting users’ sense of being understood (Lim et al., 2024). Nonverbal communication, estimated to convey the majority of meaning in human interaction (Mehrabian, 1971), is ignored, producing one-dimensional guidance and reducing adoption in sensitive domains (Zhang & Wang, 2024).

This study addresses this gap by investigating whether integrating facial and biometric cues can enhance AI’s capacity for empathy, trust, and perceived effectiveness. The research is guided by three questions:

RQ1: How can AI agents integrate facial expression recognition and heart-rate biometrics to approximate embodied awareness in coaching interactions?

RQ2: To what extent does this integration improve emotional state detection accuracy compared to text/audio-only models?

RQ3: How do users perceive the empathy, effectiveness, and trustworthiness of embodied AI agents compared to traditional systems?

3. Literature Review

Embodied cognition theory emphasizes that cognition and emotion are inseparable from bodily states (Lakoff & Johnson, 1999; Klippel et al., 2021). Affective computing demonstrated that machines can recognize and respond to human emotions (Picard, 1997), while biometric research confirms that heart-rate variability (HRV) and micro-expressions reliably indicate stress and engagement (Pessanha & Salah, 2021; Beatton et al., 2024). Recent studies show that multimodal interaction, combining voice, appearance, and non-verbal cues, improves trust and user presence in conversational agents (Kamali et al., 2023; Spitale et al., 2024).

However, most AI coaching systems remain disembodied, relying on limited modalities. While affective computing applications in education and healthcare are expanding (Salloum et al., 2025; Vistorte et al., 2024), systematic application to coaching and therapy is scarce. This study addresses that gap by testing whether multimodal integration enhances accuracy, empathy, and user trust in AI-mediated coaching.

4. Theoretical Framework

This study integrates four complementary perspectives to ground an embodied, adaptive coaching paradigm. Each framework is explicitly tied to the research questions (RQ1–RQ3).

4.1. Embodied Cognition Theory - RQ1 (Integration of Facial + Biometrics for Embodied Awareness)

Embodied cognition holds that cognition and emotion are inseparable from bodily states; effective understanding of a person’s affect therefore requires reading physiological and behavioral cues, not just language. Foundational work by Lakoff & Johnson (1999) argues that conceptual systems are grounded in bodily experience, while Wilson (2002) synthesizes six influential views of embodiment. Barsalou’s (2008) grounded cognition further demonstrates that abstract thought is rooted in sensorimotor simulation. In applied HCI contexts, this implies that coaching systems should sense micro-expressions and autonomic signals to approximate human attunement (Klippel et al., 2021; Hauke et al., 2024). Together, these works justify our multimodal design—integrating facial expressions and biometrics, as a necessary substrate for embodied, empathic AI coaching (Vistorte et al., 2024).

4.2. Polyvagal Theory & Psychophysiology of Emotion - RQ1 & RQ2 (Interpretation of HR/HRV; Detection Accuracy Gains)

Polyvagal Theory links autonomic state (e.g., Heart Rate Variability - HRV) to social engagement, safety, and stress responsivity (Porges, 2011). Classic psychophysiology consolidates the evidential basis for using cardiovascular and electrodermal signals as emotion markers (Cacioppo, Tassinary, & Berntson, 2007), while Gross (1998) frames these dynamics within emotion regulation—central to coaching. In allied clinical-developmental work, Siegel (1999) emphasizes interpersonal attunement and self-regulation, underscoring why trustworthy coaching must be sensitive to physiological arousal. Recent studies reinforce HRV’s association with affect and engagement (Beatton et al., 2024; Lee et al., 2023; Puglisi et al., 2023). These foundations guide feature selection and interpretation of biometric signals and motivate our hypothesis that adding HR/HRV (and EDA/temperature where available) will improve detection accuracy beyond text/audio baselines (RQ2).

4.3. Emotional Intelligence (EI) in Coaching & Behavior Change - RQ3 (Perceived Empathy, Trust, Effectiveness)

The EI tradition provides a behavioral lens for why recognizing and responding to emotion improves coaching outcomes. Core definitions from Mayer & Salovey (1997) and popular diffusion via Goleman (1995) establish detection, understanding, and regulation of emotion as levers of performance and relationship quality. In applied coaching/leadership contexts, Boyatzis (2006) (Intentional Change Theory) and Goleman, Boyatzis, & McKee (2002) (“primal leadership”) connect EI to sustained behavior change and trust. Cherniss (2010) clarifies EI’s organizational relevance. These works inform our user-level outcomes and instruments (perceived empathy, trust, effectiveness), framing how multimodal sensing should translate into adaptive responses that users experience as supportive and effective (Bridgeman & Hayes, 2023; Terblanche, 2020).

4.4. Affective Computing & Multimodal Social Signals - RQ2 & RQ3 (Adaptive Feedback Loops; Multimodal Advantage)

Affective Computing established the vision of machines that perceive and respond to human emotion (Picard, 1997). In applied, emotion-oriented systems, Schröder & Cowie (2005) outlined key design issues (annotation, context, ethics), and D’Mello & Kory (2015) synthesized multimodal detection evidence in learning environments, showing benefits over unimodal approaches. For facial channels, the Facial Action Coding System is the canonical basis for micro-expression modeling (Ekman & Friesen, 1978), informing our camera-based pipeline. Contemporary embodied-agent work highlights wellbeing and rapport gains from multimodality (Spitale et al., 2024; Kamali et al., 2023), and recent education/health reviews emphasize context-aware affect sensing (Vistorte et al., 2024; Salloum et al., 2025). This tradition underwrites our closed-loop design: sense → infer → adapt, enabling the system to modulate coaching strategies in real time (Islam & Bae, 2024).

Synthesis. Across these lenses, embodied signals are not ancillary—they are constitutive of empathic understanding and effective coaching. The framework predicts that (a) multimodal sensing will more accurately detect affect than text/audio alone (RQ2) and (b) embedding those inferences into adaptive responses will improve perceived empathy, trust, and effectiveness (RQ3), thereby closing the application-level empathy gap.

5. Methodology

This study adopts a mixed-methods quasi-experimental design to evaluate the impact of embodied awareness in AI coaching systems. Two conditions will be compared: (1) a Baseline AI Coach, relying exclusively on text and audio inputs, and (2) an Embodied AI Coach, integrating text, audio, facial expression recognition, and biometric data streams. The participant pool will consist of approximately 40–60 professionals and graduate students.

Data collection will involve three streams. First, facial and biometric signals—including micro-expressions, heart-rate variability, and electrodermal activity—will be captured to detect stress and affective shifts (Ekman & Friesen, 1978; Lai et al., 2021). Second, participants will complete self-report surveys assessing trust, empathy, and satisfaction with the coaching interaction (Fang et al., 2023; Harris et al., 2023). Third, system logs will be retained to capture adaptive responses and conversational dynamics for post-hoc analysis (Shore et al., 2023).

Analysis will proceed in two stages. The quantitative component will employ ANOVA and performance metrics including accuracy, precision, recall, and F1-scores to compare detection accuracy and user satisfaction across systems (Wu et al., 2023; Hassan et al., 2025). The qualitative component will conduct thematic analysis of participant reflections on perceived empathy, trust, and authenticity in the coaching exchange (Niebuhr & Valls-Ratés, 2024; Rossing et al., 2024). This dual analytic strategy ensures rigorous evaluation of both computational performance and human experience.

6. Expected Contribution

The study is designed to advance both theoretical understanding and applied practice in embodied AI.

• Scientific Contribution – It demonstrates how integrating embodied signals improves the accuracy of affect detection and enhances rapport-building in human–AI interaction (Blümel et al., 2023).
• Applied Contribution – It extends affective computing research into coaching and therapeutic contexts, domains where empathetic responsiveness is central to effectiveness (Raamkumar & Yang, 2022).
• Empirical Evidence – It provides systematic evidence on embodied AI’s influence on trust, empathy, and perceived effectiveness, offering comparative insights against traditional disembodied systems (Niebuhr & Valls-Ratés, 2024).
• Ethical Guidelines – It generates practical design and governance guidelines for the responsible collection and use of biometric and facial data in mentorship and coaching systems (Afroogh et al., 2024; Terblanche et al., 2022).

By addressing the empathy gap that currently constrains AI-mediated coaching, this study contributes to the development of emotionally intelligent AI agents capable of delivering nuanced, context-sensitive, and human-like mentorship.

7. Limitations and Future Research

Like all exploratory studies, this research has limitations that inform future work.

Sample and Generalizability

The study focuses on 40–60 participants drawn from professional and student contexts. While this provides valuable insights into the perceived empathy and trust of embodied AI coaches, the sample may not fully capture variability across age, cultural background, or industry context. Larger and more diverse populations would enhance external validity (Harris et al., 2023; Vistorte et al., 2024).

Technical Constraints

Facial recognition and biometric sensing technologies are sensitive to environmental factors such as lighting, camera resolution, and sensor calibration (Huang et al., 2023; Bello et al., 2023). Synchronization challenges between visual and physiological signals may also introduce noise into multimodal fusion (Wang et al., 2022; Shakhovska et al., 2024). Future work should investigate more robust architectures and adaptive calibration techniques to mitigate these issues.

Ethical Considerations

The integration of biometric and facial data raises significant concerns regarding privacy, consent, and data security (Afroogh et al., 2024; Chavan et al., 2025). This study proposes guidelines for responsible design but further research is needed on governance frameworks that balance innovation with user protection. Future investigations should also consider the implications of power asymmetries between AI systems and vulnerable users in coaching or therapy contexts.

Future Research Directions

Future work should expand in three directions. First, comparative studies across cultural contexts could examine how embodied signals vary and whether models generalize effectively. Second, longitudinal studies should assess the sustained impact of embodied AI coaching on behavioral and emotional outcomes. Third, hybrid designs combining neurophysiological data (e.g., EEG) with facial and biometric signals may deepen affect recognition accuracy, extending the capabilities of adaptive mentorship systems (Herbuela & Nagai, 2025; Gao et al., 2024).