Embodied AI: Multimodal Integration of Facial Expressions and Biometric Signals

1. Introduction

Artificial Intelligence (AI) has advanced in natural language processing, computer vision, and speech recognition (Bian et al., 2023). Yet, most human-facing AI systems—such as those in coaching, education, or healthcare—remain disembodied, relying primarily on text or voice (McKee et al., 2023; Modi & Devaraj, 2022; Peng et al., 2024). Human communication, however, is inherently multimodal, combining language, facial micro-expressions, and involuntary physiological responses (Liu et al., 2023; Tiwari & Falk, 2019).

To achieve empathetic and adaptive interaction, AI must interpret these non-verbal signals (Spitale et al., 2024). This paper proposes the development of embodied AI systems that integrate facial expression recognition with biometric data (heart rate, heart rate variability, temperature, electrodermal activity) to enhance affect recognition in real time (Gao et al., 2024; Suganya et al., 2024). Such integration strengthens AI’s emotional intelligence, allowing adaptive coaching systems to tailor interventions dynamically to a user’s emotional and physiological state (Liu, 2024; Awan et al., 2022).

While affective computing has pioneered unimodal approaches (Picard, 1997; Ekman & Friesen, 1978), challenges remain in multimodal fusion, synchronization, and contextual interpretation (Wang et al., 2022; Yang et al., 2024). This research advances multimodal AI by developing and testing fusion architectures capable of operating in dynamic, real-world environments such as adaptive coaching (Narimisaei et al., 2024; Zhu et al., 2025).

2. Problem Statement and Research Questions

Despite rapid advances in artificial intelligence, most affect recognition systems remain unimodal, relying on either text, voice, or facial expressions alone. This creates an empathy gap, as human communication is inherently multimodal, blending facial micro-expressions, body language, and physiological signals (Mehrabian, 1971; Ekman & Friesen, 1978; Liu et al., 2023). Unimodal systems therefore struggle with accuracy and context sensitivity, often misinterpreting emotional states in real-world conditions (Mathur et al., 2023; Poria et al., 2017; Afzal et al., 2024).

A second limitation is synchronization: biometric and visual signals operate at different temporal resolutions, making it difficult to align and fuse them effectively (Wang et al., 2022; Shakhovska et al., 2024; Zhu et al., 2025). Without robust temporal integration, critical affective cues may be lost. Finally, most models remain context-agnostic, capable of detecting emotions but not dynamically adjusting their responses in ways that mirror human social intelligence (Mohamed et al., 2024; Robb et al., 2023). This undermines user trust and engagement in sensitive domains such as coaching, therapy, and education (Nyamathi et al., 2024; Spitale et al., 2024).

Addressing these gaps requires developing multimodal frameworks that integrate facial and biometric signals into adaptive feedback loops. Such systems would not only improve recognition accuracy but also enable embodied AI capable of more empathetic, context-aware interactions (Picard, 1997; Baltrušaitis et al., 2019; Hegde & Jayalath, 2025).

Accordingly, this study poses three guiding research questions:

RQ1: How can multimodal deep learning architectures effectively fuse asynchronous data streams (facial micro-expressions, heart rate, HRV, temperature) for robust real-time affect recognition?

RQ2: What are the computational trade-offs between early, late, and hybrid fusion approaches in multimodal affect detection, particularly under real-time constraints?

RQ3: How can multimodal affect recognition be embedded into adaptive AI systems that dynamically modify responses in interactive coaching environments?

3. Literature Review

Research in affective computing has long emphasized the importance of enabling machines to detect and respond to human emotions (Picard, 1997). Subsequent surveys have highlighted its applications in education, healthcare, and human–AI interaction but note persistent challenges in moving beyond unimodal designs (Calvo & D’Mello, 2010; Poria et al., 2017; Afzal et al., 2024; Vistorte et al., 2024; Mathur et al., 2023; Hegde & Jayalath, 2025). Computer vision approaches grounded in Ekman and Friesen’s (1978) Facial Action Coding System have shown that micro-expressions reveal hidden affective states, yet models remain vulnerable to occlusion, lighting, and cultural variability (Bian et al., 2023; Huang et al., 2023; Janhonen, 2023; Tellamekala et al., 2025). Similarly, physiological computing has linked biometric signals such as heart rate variability and electrodermal activity to stress, arousal, and regulation (Porges, 2011; Beatton et al., 2024; Mattern et al., 2023; Pessanha & Salah, 2021). However, these signals are often noisy and context-dependent, limiting their reliability when used in isolation (Bello et al., 2023; Wang & Wang, 2025).

To address these shortcomings, multimodal fusion approaches integrate visual, auditory, and physiological signals. Landmark surveys (Baltrušaitis et al., 2019; Koromilas & Γιαννακόπουλος, 2021; Lai et al., 2023; Li et al., 2025) show that fusion improves robustness and accuracy compared to unimodal systems. Recent innovations include cross-modal attention (Das et al., 2024), latent distribution calibration (Tellamekala et al., 2023), and interpretable fusion frameworks (Mansouri-Benssassi & Ye, 2021; Zhi et al., 2024). Empirical work has further explored multimodal sentiment analysis (Pan & Liu, 2024), audio–visual emotion recognition (Schoneveld et al., 2021), and real-time estimation using behavioral and neurophysiological signals (Herbuela & Nagai, 2025; Mordacq et al., 2024). Yet, major technical barriers remain: asynchronous signals complicate alignment (Wang et al., 2022; Shakhovska et al., 2024; Zhu et al., 2025), scalability challenges hinder deployment in dynamic settings (Gupta et al., 2024; Bose et al., 2023), and ethical concerns persist regarding biometric and facial data (Barker et al., 2025; Afroogh et al., 2024; Chavan et al., 2025; Lin, 2024).

Despite these advances, few studies have delivered computationally robust, real-time frameworks that integrate visual and biometric cues within adaptive, interactive systems. Current models often detect affective states but lack the capacity to dynamically adjust responses in ways that mirror human social intelligence (Mohamed et al., 2024; Robb et al., 2023; Niebuhr & Valls-Ratés, 2024). This gap underscores the need for a framework that fuses facial expressions and biometric signals into adaptive feedback loops for embodied AI. Such a system would bridge unimodal limitations, overcome synchronization challenges, and deliver more empathetic, context-aware AI agents—particularly in coaching, education, and therapeutic settings (Alazraki et al., 2021; Dol et al., 2023; Nyamathi et al., 2024; Kok et al., 2024; Hao et al., 2024).

4. Theoretical and Computational Framework

This study draws on four complementary frameworks to guide the development of multimodal affect recognition and adaptive coaching systems:

Embodied Cognition Theory

Embodied cognition posits that cognition and emotion are inseparable from bodily states, meaning that affect must be understood through physiological and behavioral signals. Lakoff and Johnson (1999) introduced embodiment as central to cognition, while Wilson (2002) outlined six perspectives that shaped the field. Barsalou (2008) further advanced the concept through grounded cognition, showing that abstract thought is rooted in bodily simulation. These perspectives support RQ1, justifying why multimodal AI systems should fuse facial and biometric data to approximate human affective understanding (Klippel et al., 2021; Hauke et al., 2024; Liu et al., 2023).

Polyvagal Theory

Polyvagal theory links autonomic physiology, particularly heart rate variability (HRV), to readiness for social engagement and stress regulation (Porges, 2011). This aligns with RQ2, providing a foundation for interpreting biometric signals. Gross (1998) emphasized emotion regulation as a process shaped by both physiology and cognition, while Cacioppo, Tassinary, and Berntson (2007) established psychophysiology as a scientific basis for understanding emotional states. These works collectively justify the inclusion of HRV and electrodermal activity as inputs to multimodal fusion (Beatton et al., 2024; Puglisi et al., 2023; Lee et al., 2023; Herbuela & Nagai, 2025).

Multimodal Machine Learning Frameworks

Multimodal machine learning provides computational strategies for integrating heterogeneous signals. Early surveys emphasized decision and feature-level fusion (Atrey et al., 2010), while Ngiam et al. (2011) introduced deep multimodal learning, paving the way for current neural architectures. Baltrušaitis et al. (2019) synthesized advances into a widely cited taxonomy. These foundations inform RQ1 and RQ2, guiding evaluation of early, late, and hybrid fusion strategies. Recent developments—such as calibrated latent distribution fusion (Tellamekala et al., 2023), meta-fusion frameworks (Liang et al., 2025), and interpretable fusion models (Zhi et al., 2024; Zhu et al., 2025)—extend this foundation toward robust, explainable real-time integration.

Affective Computing Paradigm

Picard’s (1997) seminal work established affective computing, later extended by Schröder and Cowie (2005), who highlighted design challenges for emotion-oriented systems. Calvo and D’Mello (2010) reviewed affect detection methods, while D’Mello and Kory (2015) provided a meta-analysis of multimodal affect recognition in learning environments. These foundational works support RQ3, which explores embedding affect recognition into adaptive feedback loops. Recent studies advance these ideas by applying affective computing to real-world domains, including wellbeing (Spitale et al., 2024), learning (Vistorte et al., 2024), and emotional support (Hegde & Jayalath, 2025; Mohamed et al., 2024).

Collectively, these frameworks argue that embodied AI requires integration of bodily, physiological, and affective signals, computationally modeled through multimodal machine learning and affective computing to achieve emotionally intelligent, adaptive interactions.

5. Proposed Methodology

This research employs a mixed-methods, quasi-experimental design to develop and evaluate multimodal fusion architectures for affect recognition, with a specific focus on adaptive coaching applications. The methodology combines computational modeling, empirical testing, and comparative evaluation to address the research questions.

Model Development

Three deep learning fusion architectures will be developed and benchmarked:

• Early Fusion: Raw biometric and visual signals will be combined prior to feature extraction, following approaches tested in multimodal affect detection (Tellamekala et al., 2023; Wang et al., 2022).
• Late Fusion: Independent unimodal models will be trained and merged at the decision stage, leveraging ensemble and evidential methods (El-Din et al., 2023; Liang et al., 2025).
• Hybrid Fusion: A shared feature space with cross-attention mechanisms will be implemented to dynamically integrate complementary cues, drawing on advances in interpretable fusion frameworks (Mansouri-Benssassi & Ye, 2021; Shakhovska et al., 2024; Zhao et al., 2021).

Each architecture will be embedded in a prototype Embodied AI Coach/therapy to test real-time adaptability. The framework builds on prior work in multimodal sentiment classification (Suganya et al., 2024), audio-visual emotion recognition (Schoneveld et al., 2021), and bio-inspired computational integration (Mansouri-Benssassi & Ye, 2021).

Participants and Data Collection

A sample of 40–60 participants will engage in structured coaching sessions using both unimodal (baseline) and multimodal (embodied) AI systems, consistent with prior experimental affective computing designs (Nyamathi et al., 2024; Robb et al., 2023).

Data sources include:

• Facial micro-expressions via video capture (Ekman & Friesen, 1978; Huang et al., 2023).
• Biometric signals including heart rate variability, galvanic skin response, and skin temperature (Pessanha & Salah, 2021; Beatton et al., 2024; Mattern et al., 2023).
• Self-report surveys to assess empathy, trust, and satisfaction (Fang et al., 2023; Harris et al., 2023).
• System logs capturing adaptive responses, latency, and feedback timing (Shore et al., 2023).

This multimodal dataset design aligns with large-scale emotion corpora such as K-EmoCon (Park et al., 2020) and mixed emotion datasets (Yang et al., 2024).

Data Analysis

The evaluation integrates quantitative and qualitative methods:

• Quantitative Analysis: Statistical comparisons (accuracy, precision, recall, F1-scores, ANOVA) will test the performance of early, late, and hybrid fusion approaches under real-time constraints (Wu et al., 2023; Hassan et al., 2025).
• Qualitative Analysis: Thematic coding of user reflections will assess perceived empathy, effectiveness, and trust (Niebuhr & Valls-Ratés, 2024; Rossing et al., 2024).
• Computational Trade-offs: The study will measure efficiency, interpretability, and scalability of different fusion strategies in dynamic environments (Bian et al., 2023; Bose et al., 2023; Liao et al., 2025).

This design ensures that the methodology addresses both scientific objectives—evaluating multimodal integration architectures—and applied outcomes—demonstrating their potential in embodied AI coaching systems, while remaining ethically grounded in biometric and facial data use (Afroogh et al., 2024; Lin, 2024).

Develop and compare three fusion architectures:

• Early Fusion – raw biometric + visual inputs combined before feature extraction (Tellamekala et al., 2023; Wang et al., 2022).
• Late Fusion – unimodal outputs merged at decision level (Liang et al., 2025; El-Din et al., 2023).
• Hybrid Fusion – feature-level concatenation with cross-attention for adaptive weighting (Mansouri-Benssassi & Ye, 2021; Song et al., 2024; Wu et al., 2023).
• Evaluation
• Data: facial video (micro-expressions) + physiological signals (HRV, GSR, temperature).
• Benchmarks: accuracy, F1, latency, interpretability across datasets (DEAP, MAHNOB-HCI).
• Metrics: precision, recall, computational efficiency, robustness across populations (Gupta et al., 2024; Hassan et al., 2025).

6. Expected Contribution

This research offers contributions on both the scientific and applied fronts, advancing the design of embodied AI for affective computing and adaptive coaching.

Scientific Contributions

• Development of novel fusion architectures (early, late, and hybrid) to integrate facial expressions with biometric signals for robust multimodal affect recognition.
• Comparative evaluation of fusion strategies under real-time constraints, addressing performance, accuracy, and scalability challenges in dynamic environments.
• Establishment of a computational framework linking multimodal affect recognition to adaptive decision-making, thereby deepening the scientific understanding of embodied AI.
• Applied Contributions
• Design and testing of a prototype Embodied AI Coach capable of delivering real-time adaptive feedback informed by users’ affective and physiological states.
• Practical insights for deploying multimodal AI in education, coaching, and therapy, highlighting opportunities for more personalized and empathetic interventions.
• Development of ethical guidelines for the collection and use of facial and biometric data, supporting responsible innovation and safeguarding user trust.

Together, these contributions extend the science of multimodal affect recognition while providing a pathway to practical, ethically grounded applications in coaching and human–AI interaction.

7. Limitations and Future Research

Dataset Dependence.

The study relies on existing multimodal datasets (e.g., DEAP, MAHNOB-HCI, K-EmoCon), which, while widely used, present constraints in terms of sample diversity, ecological validity, and cultural variability (Pan & Liu, 2024; Yang et al., 2024). Many datasets are lab-controlled and lack the contextual noise of real-world environments, limiting generalization to natural coaching or therapy contexts. Future research should extend benchmarking to more ecologically valid datasets and collect new multimodal corpora in real-world adaptive interactions.

Fusion Complexity and Synchronization.

Although early, late, and hybrid fusion strategies provide a framework for integration, asynchronous sampling rates and noise across modalities complicate synchronization (Wang et al., 2022; Shakhovska et al., 2024; Zhu et al., 2025). Deep models risk overfitting to dataset-specific noise and may lack robustness when deployed in real-time applications. Future research should investigate cross-attention mechanisms, adaptive alignment algorithms, and explainable AI methods to ensure stability and interpretability of multimodal fusion models.

Computational Efficiency and Scalability.

Real-time multimodal affect recognition requires significant computational resources, creating trade-offs between accuracy, latency, and scalability (Bose et al., 2023; Gupta et al., 2024). Prototype models may perform well in experimental settings but struggle under deployment constraints such as low-power devices or bandwidth-limited contexts. Future work should prioritize lightweight architectures, edge-AI implementations, and energy-efficient models to enhance scalability in diverse environments.

Ethical and Interpretability Challenges.

While this study emphasizes technical performance, integrating sensitive biometric data introduces ethical concerns regarding privacy, security, and potential misuse (Afroogh et al., 2024; Chavan et al., 2025). Moreover, deep fusion models often act as “black boxes,” limiting interpretability for end-users and practitioners. Future research should explore explainable multimodal AI, ensuring that both researchers and users understand how emotional inferences are made.

Future Research Directions.

Moving forward, research should expand in four directions:

• Cross-domain generalization — testing architectures in varied applied domains such as education, healthcare, and workplace coaching.
• Longitudinal evaluation — measuring how multimodal models adapt to user changes over time rather than single-session trials.
• Neuro-inspired integration — leveraging bio-inspired computational models (Mansouri-Benssassi & Ye, 2021) to improve affective state modeling.
• Ethical frameworks — developing governance standards for multimodal AI systems to ensure responsible deployment in sensitive human-centered contexts.