Cross-Modal Temporal Attention for Robust Multimodal Emotion Recognition

1. Introduction

Emotion recognition is increasingly seen as a cornerstone of affective computing [7], enabling intelligent systems to decode nuanced emotional signals and foster more natural human-machine interactions. As artificial intelligence becomes more deeply embedded in applications like digital assistants, social robotics, e-learning platforms, and driver vigilance systems, the necessity for accurate and robust emotional interpretation in unstructured environments has grown sharply.

While early sentiment analysis efforts predominantly relied on single-modal data, such as text or static imagery, real-world affective expressions manifest through a complex interplay of signals—facial muscle articulations, speech prosody, body posture, and gestural patterns. Relying on a single modality limits a model’s ability to accurately interpret these signals, particularly under noisy or unconstrained scenarios. This drives the adoption of multimodal affective computing frameworks [2], where information from various sensory modalities is fused to yield a more comprehensive emotional understanding.

Nonetheless, recognizing emotions in-the-wild poses several inherent challenges. Data collected in real-world contexts is often noisy, occluded, and affected by sudden lighting changes, missing modality streams, and spontaneous (rather than posed) affective behaviors. Benchmarks such as Aff-Wild2, AffectNet, and RAF-DB were designed to capture such variability and now serve as standard datasets for evaluating robust multimodal architectures [5]. These resources offer diverse annotations, modalities, and scene complexities, making them ideal for evaluating real-world emotion recognition systems.

To address these challenges, we introduce EMMA-Net, a unified attention-driven multimodal framework purpose-built for in-the-wild emotion estimation. EMMA-Net explicitly models temporal dependencies across modalities and incorporates context-aware reasoning. Drawing from advances in sequence modeling and attention mechanisms, the model processes temporal sequences—spanning facial images, acoustic segments, and skeletal pose flows—individually before unifying them via a cross-modal attention structure that temporally aligns their dynamics.

The principal innovations of EMMA-Net include: (1) a temporal cross-alignment module centered around the current timestamp, utilizing adjacent temporal context across modalities to enhance emotional representation; (2) an adaptive reliability-based weighting mechanism to filter out noisy or uninformative modalities; and (3) a performance-optimized architecture validated through empirical evaluation on the ABAW benchmark.

We build a cross-modal temporal attention strategy that aligns facial features at a given frame with contextual signals derived from historical face dynamics, motion trajectories, and acoustic cues. Our approach extends beyond conventional fusion schemes by incorporating adaptive attention weights that modulate the contribution of each modality in response to its perceived reliability at each time step. This design proves practically effective on the CVPR 2023 ABAW dataset, where our model exhibits competitive performance.

To provide a solid basis for our methodology, it is instructive to review the evolution of multimodal representation learning. While transformer-based models have recently reshaped the field of sequential modeling, many multimodal approaches still skew toward one dominant stream, such as visual or audio inputs [4]. EMMA-Net addresses this imbalance through a symmetric attention treatment across modalities and applies contextual cross-attention to enhance inter-temporal dependencies. Moreover, by integrating localized temporal windows, the fusion strategy minimizes the impact of long-range noise and off-topic cues.

With this comprehensive design, we strive to narrow the semantic gap between modalities and enhance the resilience of emotion recognition systems under wild conditions. Given a video sequence $\mathcal{V}$ with corresponding audio stream $\mathcal{A}$ and pose sequence $\mathcal{P}$, the target is to predict the emotion state ${\widehat{y}}_t$ at each frame t. Let $F_t$, $A_t$, and $P_t$ represent the extracted features from face, audio, and pose respectively. This formulation underlies our system’s temporal reasoning and cross-modal fusion.

The remainder of this paper is organized as follows: in the next section, we describe the architecture of EMMA-Net in detail, including the feature extraction methods, attention modules, and fusion mechanism. We then present our experiments and analyses, demonstrating how temporal alignment and attentive integration contribute to improved emotion recognition performance.

2. Related Work

Affective computing has witnessed significant progress over the years, evolving into various subdomains such as multimodal fusion, sequential modeling, and robust facial analysis. In this section, we position our work within the broader literature by surveying notable contributions in these areas and highlighting the distinctiveness of EMMA-Net in its temporal attention and cross-modal learning design.

• Facial Expression Recognition.

Recognizing facial expressions has long stood as a central task in affective computing, given the crucial role of facial muscle movements in conveying emotions. Early efforts focused on geometric features and handcrafted descriptors such as Local Binary Patterns (LBP), Gabor filters, and Histograms of Oriented Gradients (HOG) [4]. Although these methods yielded reasonable accuracy in constrained environments, they faltered under variations in head orientation, occlusions, and lighting conditions. With the advent of deep learning, convolutional neural networks (CNNs) emerged as the dominant paradigm for FER, enabling automatic feature learning directly from raw facial images. Ding et al. introduced facial representation models resilient to occlusion effects [5], enhancing robustness in real-world scenarios. Nevertheless, many existing FER models neglect temporal continuity, often treating video frames independently—an issue EMMA-Net addresses by explicitly modeling temporal dependencies across frames.

• Audio-based Emotion Analysis.

Audio cues provide complementary emotional signals beyond facial information, particularly through prosodic variation, vocal timbre, and rhythm. Earlier approaches primarily relied on handcrafted acoustic descriptors such as Mel-Frequency Cepstral Coefficients (MFCCs) and pitch-based statistics. More recent toolkits like ComParE and DeepSpectrum [1] facilitate high-dimensional audio embeddings derived from spectrogram inputs, which, when used with CNNs or recurrent models, have demonstrated strong performance. However, the audio stream is often sensitive to ambient noise and recording conditions, which can undermine prediction reliability. To address this, many recent methods advocate multimodal fusion strategies that combine audio with other modalities. EMMA-Net follows this paradigm, dynamically learning contextual embeddings and attenuating the audio signal’s influence when its quality degrades.

• Pose and Body Expression.

Although facial expressions and vocal cues dominate affective research, bodily expressions also play a critical role, especially in non-verbal affective communication. Human pose information—typically encoded through skeletal keypoints extracted using pose estimation techniques like OpenPose—offers a lightweight yet semantically rich representation of body dynamics. Bhattacharya et al. [2] proposed Spatial Temporal Graph Convolutional Networks (ST-GCNs), which model temporal and spatial correlations among body joints for improved emotion recognition. Similarly, Crenn et al. [3] showed that 3D animated skeletons can effectively convey affective signals, particularly in full-body scenarios where the face may not be visible. Despite their promise, pose features are often noisy and difficult to synchronize with other modalities. EMMA-Net tackles this issue by applying modality-specific temporal encoding and learning robust fusion strategies that mitigate misalignment.

• Multimodal Fusion Strategies.

Combining information from disparate modalities remains a key challenge due to discrepancies in signal characteristics, sampling rates, and reliability. Classical fusion schemes such as early fusion (concatenating features at input) and late fusion (combining outputs) often suffer from rigid modality treatment and loss of cross-modal interaction. More recently, attention-based and transformer-style architectures have introduced dynamic and context-sensitive fusion methods. In particular, cross-attention modules provide a mechanism to learn inter-modal dependencies selectively. EMMA-Net builds upon these innovations by incorporating a temporal cross-attention module that not only fuses heterogeneous features but also temporally aligns them, improving synchronization and noise resilience.

• Temporal Modeling in Emotion Recognition.

Modeling the evolution of affective states over time is vital, especially for dynamic, real-world scenarios where instantaneous emotion cues may be insufficient or ambiguous. Traditional sequence models such as Recurrent Neural Networks (RNNs), particularly LSTMs and GRUs, have been used to capture temporal context. However, attention-based temporal encoders have increasingly replaced recurrence-based methods due to their superior ability to model long-range dependencies and avoid vanishing gradient issues. EMMA-Net takes advantage of this shift by employing local temporal windows and attention-driven fusion, which allow it to focus on the most relevant temporal context without being overwhelmed by noisy or irrelevant history.

In conclusion, EMMA-Net integrates advances in facial, acoustic, and pose-based emotion recognition with modern temporal modeling and flexible multimodal fusion techniques. Its unified architecture is specifically designed to capture the unique temporal dynamics of each modality while learning to attend across them, thereby offering a context-aware and robust solution to in-the-wild emotion estimation.

Figure 1. Overview of the overall MIMIC framework.

Figure 1. Overview of the overall MIMIC framework.

3. Methodology

In this section, we detail the architectural design and algorithmic components of our proposed EMMA-Net (Emotion-aware Multimodal Memory-Attentive Network), an end-to-end multimodal system crafted to capture fine-grained emotional patterns through temporally contextualized representations. EMMA-Net addresses the intricate task of in-the-wild video-based emotion recognition by jointly modeling facial expressions, speech dynamics, and body pose trajectories, unified via an adaptive attention mechanism informed by temporal context. The framework is structured around five interdependent modules: individual modality encoders, temporal sequence processors, cross-temporal attention aligners, a multimodal fusion unit, and auxiliary loss components for enhanced optimization.

Our design of EMMA-Net is tailored to tackle core challenges in affective analysis: (1) the modality-specific heterogeneity of emotional signals, (2) the inherent temporal misalignment across modalities, and (3) frequent signal degradation or missing data in naturalistic conditions. To mitigate these issues, we implement a cross-temporal attention fusion strategy, where each modality dynamically interacts with the facial frame at the current timestep, enabling robust integration of multimodal cues from both short- and long-range contexts. The following subsections describe each component in detail.

Recognizing that emotional cues often manifest through coordinated body language and movement, EMMA-Net incorporates skeletal pose information in addition to facial and auditory modalities. Below, we describe our model—a temporally-sensitive, attention-guided multimodal network designed to process and fuse facial, vocal, and pose-based cues at both the frame and temporal levels. This is accomplished via attention-driven mechanisms that align embeddings across modalities, anchored to the current video frame.

The overall pipeline begins with dedicated feature encoders per modality, followed by temporal sequence encoders that model intra-modal dynamics. These outputs are then passed to a cross-temporal attention mechanism that computes interaction weights between modalities relative to the central face representation. Finally, a classification head processes the fused representation. Let $\mathcal{V}$ denote the input video, and $\mathcal{F}$, $\mathcal{A}$, and $\mathcal{P}$ denote the facial, audio, and pose substreams, respectively.

3.1. Modality-Aware Feature Extraction Modules

• Facial Stream Encoding.

To encode visual emotional information, we utilize the Face Alignment Network (FAN), a deep CNN-based model capable of generating stable facial embeddings under unconstrained conditions. Pretrained on AffectNet and finetuned with Aff-Wild2, FAN provides a per-frame feature vector $F_t\in {\mathbb{R}}^{d_f}$.

To enhance efficiency, we temporally downsample the video to 1 frame every 5 frames (i.e., 1:5 sampling from 30fps), reducing redundancy while maintaining adequate temporal granularity. The resulting sequence $\{F_1,F_2,\dots ,F_T\}$ is then processed by a 2-layer bidirectional GRU encoder ${\mathcal{E}}_f$, yielding temporally enriched facial features:

$$H^f={\mathcal{E}}_f\left([F_1,\dots ,F_T]\right)\in {\mathbb{R}}^{T\times h_f}$$

(1)

• Audio Stream Encoding.

We investigated multiple audio representations—ComParE2016, eGeMAPS, and DeepSpectrum—and adopted DeepSpectrum for its rich feature capacity. Based on a DenseNet-121 trained over spectrograms, the extracted audio embeddings $A_t\in {\mathbb{R}}^{1024}$ are computed using 1-second windows with 0.5-second overlap. These are passed through a 2-layer GRU ${\mathcal{E}}_a$ to capture sequential acoustic dynamics:

$$H^a={\mathcal{E}}_a\left([A_1,\dots ,A_T]\right)\in {\mathbb{R}}^{T\times h_a}$$

(2)

• Pose Stream Encoding.

We extract body motion cues via OpenPose, sampling at 2Hz to produce sequences of 2D skeletal joint vectors $P_t\in {\mathbb{R}}^{J\times 2}$, with J representing the number of joints. These vectors are flattened and passed into a 2-layer GRU encoder ${\mathcal{E}}_p$ to encode dynamic pose trajectories:

$$H^p={\mathcal{E}}_p\left([P_1,\dots ,P_T]\right)\in {\mathbb{R}}^{T\times h_p}$$

(3)

3.2. Cross-Temporal Attention Fusion

To align temporal contexts across modalities with the facial frame at the current timestep, we propose a cross-temporal attention unit. Let $f_c$ denote the current facial representation obtained from FAN. For each modality $m\in \{f,a,p\}$, we calculate attention weights ${\alpha }^m$ using a scaled dot-product attention mechanism:

$${\alpha }^m=\mathrm{softmax}\left({\displaystyle \frac{H^m{W}_Q{\left(f_c{W}_K\right)}^T}{\sqrt{d_k}}}\right)$$

(4)

Here, $W_Q$ and $W_K$ are trainable projection matrices, and $d_k$ is the dimensionality of the attention head.

Each modality’s context vector is computed as a weighted sum over its temporal sequence:

$$C^m=\sum _{t=1}^T{\alpha }_t^m{H}_t^m$$

(5)

The final joint representation z is formed by concatenating the current facial feature $f_c$ with the attended contexts $C^f$, $C^a$, and $C^p$:

$$z=[f_c;C^f;C^a;C^p]\in {\mathbb{R}}^{d_z}$$

(6)

This concatenated vector is passed through a fully connected prediction network $\mathcal{H}$ with dropout, producing the emotion label ${\widehat{y}}_t$:

$${\widehat{y}}_t=\mathcal{H}\left(z\right)=\mathrm{softmax}(W_2\sigma (W_{1}z+b_1)+b_2)$$

(7)

3.3. Auxiliary Learning and Loss Functions

To enhance generalization and training stability, we introduce two auxiliary losses alongside the primary cross-entropy classification objective:

-

Modality Reconstruction Loss: To preserve modality-specific information, we reconstruct intermediate embeddings ${\tilde{H}}^m$ from the fused vector z:

$${\mathcal{L}}_{rec}=\sum _m{\parallel {\tilde{H}}^m-H^m\parallel }_2^2$$

(8)

-

Temporal Smoothness Loss: To ensure consistency in predicted emotion states across time, we impose a smoothness regularization:

$${\mathcal{L}}_{smooth}=\sum _{t=2}^T{\parallel {\widehat{y}}_t-{\widehat{y}}_{t-1}\parallel }_1$$

(9)

The final loss combines all objectives using weighted summation:

$${\mathcal{L}}_{total}={\mathcal{L}}_{ce}+{\lambda }_{rec}{\mathcal{L}}_{rec}+{\lambda }_{smooth}{\mathcal{L}}_{smooth}$$

(10)

EMMA-Net constitutes a fully differentiable and end-to-end trainable system for multimodal affect recognition. Its core innovations lie in its temporal sequence modeling, modality-aware attention fusion, and auxiliary learning strategies that jointly enhance robustness in the face of real-world variability.

4. Experiments

In this section, we present a comprehensive empirical evaluation of our proposed EMMA-Net framework. The evaluation covers dataset preparation, experimental setup, training protocols, and both quantitative and qualitative results. We also conduct ablation studies to understand the contribution of each module and compare fusion strategies across different configurations. To ensure robustness and generalizability, we perform cross-validation and analyze variance across multiple data splits.

4.1. Dataset and Preprocessing Protocol

We conduct experiments on the Aff-Wild2 dataset [10,11,12,13,14,16,17,18,19,29], as provided by the fifth ABAW Competition [15]. The dataset comprises 598 annotated video clips under in-the-wild conditions, with expression classification as one of the core tasks. For this task, 247 clips are used for training, 70 for validation, and 228 for testing.

To avoid label leakage, we removed duplicate entries such as `122-60-1920x1080-2.txt`, which appeared in both train and validation sets. Furthermore, to evaluate generalizability, we created five-fold cross-validation splits by randomly partitioning the training set and rotating the validation fold. Each fold retains roughly the same expression distribution. For all splits, the test set remained untouched.

To accelerate training while maintaining performance, we performed downsampling for video frames (sampling 1 frame every 5 frames), and audio/pose features were extracted at 2Hz. Frames without valid annotations were removed, resulting in around 180,000 usable samples for training.

4.2. Training Settings and Optimization Details

Our model was implemented in PyTorch and trained on a single RTX 3090 GPU. Below we describe key stages of model training:

• Pretraining FAN.

We pretrained the Face Alignment Network (FAN) on the AffectNet dataset. Hyperparameters were tuned via randomized grid search: weight decay in [0.0, 0.01], learning rate in [0.0001, 0.01], optimizer betas in [0.0, 0.999]. The model was trained for up to 30 epochs with a learning rate decay factor of 0.1 every 15 epochs. Adam optimizer was employed.

• Finetuning FAN on Aff-Wild2.

The pretrained FAN was finetuned on the official split of Aff-Wild2 using AdamW optimizer, with a backbone learning rate of $4\times {10}^{-5}$ and a predictor learning rate of $4\times {10}^{-3}$. Learning rates were halved if validation F1 scores stagnated for more than two epochs. As shown in Table 1, our best average F1 was obtained in the 15th epoch. Additionally, we compared five custom splits and observed Split5 yielding the best result.

• Training EMMA-Net.

For multimodal fusion, we set a 6-second window with 2 frames per second (i.e., $T=12$). The FAN parameters were frozen, and only downstream modules were trained with a learning rate of 0.02. Batch size was set to 4 due to GPU memory constraints. A dynamic learning rate scheduler reduced the learning rate by half upon validation plateau.

4.3. Split-wise Performance Comparison

We evaluate the robustness of FAN features under different training-validation splits. As shown in Table 1, Split5 achieved the best Avg(F1) of 35.5%, outperforming the official split (33.7%). The diversity across splits highlights the importance of evaluating under varied distributions.

4.4. Fusion Strategy Comparison

We next evaluate the impact of different fusion strategies. Table 2 compares baseline methods: (1) current face only, (2) only video without face, (3) naive concatenation of modalities, and (4) our proposed attention-based fusion.

The attention-based fusion clearly outperforms all baselines, improving Avg(F1) to 36.1% compared to 33.7% for the face-only setting and 31.8% for naive fusion. Notably, the model benefits from better context modeling and dynamic weighting of noisy modalities. Emotion categories such as “Surprise” and “Disgust” show the largest gains, reflecting their multimodal expressiveness.

4.5. Additional Evaluation: Fusion Variants and Temporal Length

To further analyze the design choices of EMMA-Net and their impact on performance, we conducted two sets of auxiliary experiments focusing on fusion strategies and temporal sequence lengths. These studies aim to clarify the contributions of architectural decisions to the overall robustness and expressiveness of the model.

(1)

Fusion Variant Ablation.

We ablated our attention-based fusion mechanism by replacing it with two alternative strategies: additive attention and gating-based fusion. The additive attention mechanism computes a weighted sum of modality features using learned scalar weights, whereas the gating mechanism utilizes a sigmoid gate to modulate the flow of each modality’s contribution before fusion. Quantitatively, gating fusion achieved an Avg(F1) of 35.2%, and additive attention yielded 34.8%, both lower than our full attention-based model (36.1%).

Beyond numbers, qualitative analysis reveals that the attention mechanism adapts better to variable modality noise—especially under partial occlusion or poor acoustic quality—by reducing the weight of degraded modalities. In contrast, additive fusion often overfits to dominant signals, while gating mechanisms tend to underexploit secondary modalities due to overly conservative gating.

Formally, our attention fusion is computed as:

$$C=\sum _{m\in \{f,a,p\}}\mathrm{softmax}\left(f_c^T{W}^m{H}^m\right)·H^m$$

(11)

where $W^m$ is a learnable attention matrix for each modality m, $H^m$ is the modality sequence, and $f_c$ is the current face feature anchor. This allows dynamic, token-wise alignment to support context-aware integration.

(2)

Sequence Length Analysis.

We also investigated how the temporal context window size T affects the model’s capability to capture dynamic expressions. We varied T across {6, 9, 12, 15, 18} and observed a peak in Avg(F1) at $T=12$. Shorter sequences ($T=6,9$) lacked sufficient context for slower-changing emotions like “Sadness”, while longer sequences ($T=15,18$) introduced irrelevant past signals, especially for rapidly evolving emotions like “Surprise” or “Fear”.

We define the effective context utility (ECU) metric as:

$$\mathrm{ECU}\left(T\right)={\displaystyle \frac{1}{T}}\sum _{t=1}^T\mathbb{I}\left[\right|y_t-y_{t-1}|<ϵ]$$

(12)

where $ϵ$ is a small threshold on F1 score delta. ECU peaked near $T=12$, supporting our empirical findings.

These findings validate EMMA-Net’s structural assumptions and highlight its ability to generalize under different design settings. Moreover, they underscore the importance of both temporal granularity and modality interaction strategies in multimodal affective modeling. We believe EMMA-Net lays a strong and extensible foundation for future developments in video-based emotion recognition tasks.

5. Conclusion and Future Directions

In this paper, we presented EMMA-Net (Emotion-aware Multimodal Memory-Attentive Network), a novel architecture for multimodal emotion recognition in unconstrained video scenarios. Our approach is designed to address the limitations of traditional single-modal or late-fusion strategies by introducing a unified attention-based framework that effectively integrates facial, auditory, and skeletal cues in a temporally contextualized manner.

Through extensive experimentation on the Aff-Wild2 dataset, we demonstrated the superior performance of EMMA-Net, achieving an average F1 score of 36.1% on the validation set. This outperformed several strong baselines, including unimodal settings and naive multimodal fusion methods. Notably, our model showed robust behavior under both official and custom data splits, reinforcing its generalization capabilities. Furthermore, we conducted detailed ablation studies on fusion strategies and input sequence lengths, confirming the significance of both temporal alignment and adaptive modality weighting in boosting emotion recognition performance.

Our method introduces a cross-temporal attention mechanism, which dynamically aligns historical modality cues with the current facial state, enabling fine-grained and context-aware affect prediction. We also propose auxiliary loss functions for reconstruction and smoothness, which contribute to better stability and convergence during training.

• Future Work.

There are several promising directions for extending this research. First, incorporating additional modalities such as eye gaze, physiological signals, or contextual scene understanding could further enrich the emotion modeling. Second, developing a lightweight version of EMMA-Net for deployment on edge devices would enhance its applicability in real-time systems. Third, advancing the interpretability of attention mechanisms could provide actionable insights into which modality dominates under varying emotional conditions.

Finally, as emotion understanding is inherently subjective and culturally nuanced, future efforts could explore alized or domain-adaptive variants of EMMA-Net that can dynamically adjust to individual or situational differences.

In conclusion, we believe EMMA-Net offers a powerful and flexible framework for multimodal affective computing and opens up exciting avenues for future exploration in emotion-aware AI systems.