Multi-Interaction Modeling with Intelligent Coordination for Multimodal Emotion Recognition

1. Introduction

Emotion understanding, as a critical aspect of artificial intelligence (AI), has long been a subject of intense research interest [1]. By empowering AI agents with the ability to recognize, interpret, and respond to human emotions, we enable more naturalistic, engaging, and effective interactions in applications ranging from interactive virtual assistants and social robots to healthcare monitoring and public sentiment analysis [2]. Despite notable advancements, the challenge of capturing and modeling the nuanced interactions between diverse modalities remains unresolved, particularly in scenarios involving spoken language, facial expressions, and paralinguistic cues [2].

The inherent complexity arises not only from the heterogeneity of modalities—each possessing distinct characteristics and distributions—but also from the intertwined nature of emotional signals where modalities may reinforce, complement, or even contradict each other [3,4]. Early attempts in multimodal fusion primarily adopted straightforward concatenation or direct integration of raw feature representations across modalities [2,3,4]. While these approaches are intuitive, they often fall short in addressing the modality gaps and may inadvertently amplify redundant or noisy signals, thereby compromising the overall representation’s discriminative power [5].

To alleviate such limitations, recent methods have explored representation disentanglement strategies. For instance, the MISA framework [5] introduces modality-invariant and modality-specific decomposition, enabling finer-grained interaction modeling. Although this decomposition demonstrates improvements in interaction learning, it is not without drawbacks. The reliance on auxiliary orthogonal losses, delicate hyper-parameter tuning, and increased parameter overhead render MISA less appealing for practical systems where computational efficiency and robustness are paramount. Recognizing these limitations, our proposed MIMIC framework seeks to revolutionize multimodal emotion analysis by rethinking how multimodal interactions are represented and fused. Inspired by cognitive science perspectives where human perception processes multiple streams of signals through specialized and integrative pathways, MIMIC introduces a tripartite interaction modeling strategy that systematically captures information from complementary, shared, and modality-specific perspectives.

The first component, the Modality-Preserving Interaction Representation, retains the entirety of each modality’s information, ensuring that no modality-specific cues essential for emotion recognition are prematurely discarded. This preserves the original richness of individual modalities for downstream reasoning.

Secondly, the Cross-Modal Shared Interaction Representation emphasizes the synergistic coordination among modalities by extracting the shared latent space where cross-modal agreements and reinforcements reside. This component is critical for scenarios where emotional cues are subtly dispersed across modalities and require joint attention mechanisms for accurate interpretation.

Lastly, the Individual Modality-Specific Representations deliberately isolate each modality’s unique traits while excluding redundant shared information. This fosters the model’s ability to identify modality-unique emotional signals, such as sarcasm detectable only via tone or micro-expressions perceivable only via facial cues.

To effectively merge these diversified representations, we propose a Semantic-Driven Hierarchical Fusion Mechanism, which dynamically adjusts the weighting and interaction pathways based on the semantic consistency and informativeness of the representations. This design ensures a balanced and comprehensive integration, unlocking the full potential of multimodal information fusion for emotion analysis tasks.

Crucially, MIMIC is architected with practical deployment considerations at its core. By minimizing computational redundancy and introducing a parameter-light fusion strategy, MIMIC delivers state-of-the-art performance with significantly reduced complexity and accelerated convergence during training. This makes it highly suitable for edge devices and real-time emotion-aware applications, where computational resources are often constrained.

Our contributions are summarized as follows. First, we systematically advance the field of multimodal emotion analysis by introducing a multi-perspective interaction modeling framework that captures the layered and multifaceted nature of emotional signals across modalities. Second, we deliver a lightweight, easy-to-train, and fast-converging model that reduces the burden of hyper-parameter tuning and computational overhead, facilitating wider adoption in practical AI systems. These innovations collectively redefine the paradigm of multimodal interaction modeling, offering new insights and methodologies that transcend existing fusion-centric approaches. We anticipate that our work will stimulate further research into multi-level, cognitively inspired multimodal fusion strategies for emotion recognition and beyond.

2. Related Work

The domain of multimodal emotion analysis has witnessed substantial progress over recent years, yet the effective modeling of unimodal and multimodal representations continues to present enduring challenges for the research community. Broadly, these challenges can be systematically decomposed into two interrelated facets: (i) the construction of robust and generalizable unimodal representations, and (ii) the formulation of effective strategies for integrating and interacting across modalities to derive a comprehensive multimodal representation suitable for downstream emotion recognition tasks.

2.1. Unimodal Feature Representation Learning

Existing benchmark datasets for multimodal emotion recognition, such as MOSI [7] and MOSEI [8], have provided rich multimodal resources comprising textual utterances, audio waveforms, and visual streams extracted from conversational data. Accompanying these datasets are also canonical feature sets, including word embeddings for the textual modality, a compact 74-dimensional acoustic descriptor set widely used for the auditory channel, and facial Action Unit (AU) features along with their associated intensities for the visual modality. Tools like OpenFace and FaceNet have been extensively adopted for the automatic extraction of visual features, thereby providing a standardized pipeline for facial expression analysis.

While these classical features have provided a solid foundation for multimodal analysis, they exhibit significant limitations, particularly when directly applied in complex emotion recognition scenarios. Recent research has increasingly advocated for end-to-end learning pipelines that attempt to directly optimize unimodal representations within a joint multimodal framework. However, given the limited size and diversity of existing datasets, such end-to-end methods are prone to overfitting, severely limiting their generalizability to unseen emotional expressions or diverse user populations.

In response to these limitations, the research community has shifted attention toward leveraging large-scale pre-trained models. The fine-tuning of BERT-based models [12] has proven highly effective as a universal feature encoder for both text and, more recently, speech modalities [19]. Nevertheless, there remains a critical gap regarding the absence of similarly effective pre-trained frameworks for the vision modality, leaving an open research direction in the development of visual-centric pre-trained models that can be seamlessly integrated into multimodal emotion understanding systems.

2.2. Integration-Oriented Learning Paradigms

Historically, the most prevalent approach for multimodal representation learning has been the straightforward integration of unimodal features through direct concatenation or attention-based mechanisms. Such approaches, exemplified by works like MVLSTM [14] and BCLSTM, aim to leverage the complementary nature of each modality by combining them into a unified joint feature space. Several models also adopt attention mechanisms [3,15] to dynamically weigh the importance of each modality during fusion.

However, as noted by Zadeh et al. [2], these methods often induce modality biases where dominant modalities suppress the contributions of less prominent ones, thereby undermining the holistic nature of emotion representation. To mitigate these shortcomings, subsequent works have introduced various regularization techniques during joint representation learning, such as Canonical Correlation Analysis (CCA) losses [18], adversarial mechanisms, and self-supervised learning schemes. These strategies aim to extract common cross-modal information while disregarding the unique, modality-specific cues that might carry vital emotion-related insights. Additionally, tensor fusion techniques [2,4] have been proposed to retain both unimodal specificity and their higher-order interdependencies within a single framework.

2.3. Decoupling Learning for Enhanced Interaction Modeling

More recently, the community has gravitated toward decoupling-based learning strategies, where the goal is to separately model modality-invariant and modality-variant representations, facilitating more granular interaction learning. A pioneering effort in this direction is MISA [5], which introduces a sophisticated framework that decouples modality features into two distinct representations, enabling the isolation of shared semantics from modality-unique characteristics. Although MISA demonstrates promising performance improvements, its reliance on complex auxiliary losses—comprising multiple reconstruction, difference, and similarity losses—alongside a delicate balance of hyper-parameters, poses significant training challenges and potential instability. Furthermore, the indirect enforcement of low similarity between disentangled representations cannot always guarantee the desired separation in practical scenarios, possibly leading to suboptimal emotion representation capabilities.

Despite the breadth of research efforts, existing approaches—whether integration-based or decoupling-based—exhibit inherent limitations when it comes to achieving effective and efficient multimodal interaction modeling. Integration-based methods tend to overlook modality-specific nuances by emphasizing commonality, whereas decoupling-based methods often complicate the training pipeline and increase the computational footprint.

In light of these observations, our proposed MIMIC framework is designed to address the aforementioned limitations by introducing a principled yet lightweight approach to model multimodal interactions from multiple perspectives. Unlike conventional methods that either over-simplify or over-complicate interaction modeling, MIMIC adopts a tripartite interaction representation scheme that captures full modality-preserving, cross-modal shared, and individualized representations in parallel. Furthermore, our semantic-driven fusion strategy circumvents the need for auxiliary losses and excessive hyper-parameter tuning, facilitating a more stable and efficient training process while achieving superior emotion recognition performance.

By systematically integrating these advancements, MIMIC paves the way for a more interpretable, robust, and scalable solution to multimodal emotion recognition, aligning closely with the current demands for deployable and resource-efficient AI systems in emotion-aware applications.

3. MIMIC: Multi-Interaction Modeling with Intelligent Coordination Framework

This section elaborates on the proposed MIMIC framework, designed to address the challenges in multimodal emotion analysis by comprehensively modeling both the independent and joint interactions among heterogeneous modalities, including text, audio, and visual cues. Each input video clip is segmented into utterance-level units, bounded by natural speech cues such as breaths or pauses. For simplicity of notation, we denote ${\mathit{u}}_m^l$, ${\mathit{h}}_m$, and ${\mathit{i}}_m$ as the temporal features, the aligned unimodal utterance-level representation, and the modality-individual representation for modality m, respectively, where $m\in \{t,v,a\}$ denotes text, vision, and audio.

MIMIC introduces an end-to-end trainable framework comprising four core components: (i) Unimodal Encoder Module, (ii) Instance-Aware Decoupling Mechanism, (iii) Hierarchical Multi-View Fusion Module, and (iv) Contrastive Regularization and Classification Head. Each component is meticulously designed to balance representation granularity, cross-modal coordination, and computational efficiency.

Figure 1. Overview of illustration of EffMulti framework.

Figure 1. Overview of illustration of EffMulti framework.

3.1. Unimodal Encoder Module

The Unimodal Encoder Module ${\mathbb{E}}_m$ aims to extract high-quality, temporally aware features from each unimodal input sequence. Specifically, given the raw sequence ${\mathit{u}}_m^l$, we adopt a modality-specific encoder consisting of a stack of Bi-LSTMs followed by self-attention to capture contextual dependencies:

$${\mathit{h}}_m=\mathit{SelfAttn}\left(\mathit{BiLSTM}\left({\mathit{u}}_m^l\right)\right),$$

(1)

where ${\mathit{h}}_m\in {\mathbb{R}}^{L\times D}$, L is the utterance length, and $D=64$ is the unified embedding dimension.

Motivation: Using self-attention enables capturing long-range utterance dependencies, critical in handling expressive emotions dispersed across temporal segments.

To further reduce modality gaps, a modality alignment layer is applied:

$${\widehat{\mathit{h}}}_m=\mathit{LayerNorm}\left({\mathit{h}}_m\right)+{\mathit{h}}_m,$$

(2)

which harmonizes all modalities into a shared feature space and prepares them for downstream interaction modeling.

3.2. Instance-Aware Decoupling Mechanism

To capture both commonality and individuality across modalities, we propose an efficient Instance-Aware Decoupling Mechanism that decomposes ${\widehat{\mathit{h}}}_m$ into:

• Modality-Shared Representation$\mathcal{S}$
• Modality-Individual Representation${\mathit{i}}_m$

Motivation: Unlike prior works [5] requiring auxiliary losses and hyper-parameter balancing, our decoupling method is parameter-free and self-contained within the data instance.

Formally:

$$\mathcal{S}={\displaystyle \frac{1}{3}}\sum _m{\widehat{\mathit{h}}}_m,$$

(3)

$${\mathit{i}}_m={\widehat{\mathit{h}}}_m-\mathcal{S}.$$

(4)

We further introduce a bottleneck transformation to reduce ${\mathit{i}}_m$ into a compact representation:

$${\tilde{\mathit{i}}}_m=\mathit{ReLU}({\mathit{W}}_i{\mathit{i}}_m+{\mathit{b}}_i),$$

(5)

where ${\tilde{\mathit{i}}}_m\in {\mathbb{R}}^{16}$, ${\mathit{W}}_i\in {\mathbb{R}}^{16\times D}$, ensuring computational efficiency.

3.3. Hierarchical Multi-View Fusion Module

Our Hierarchical Multi-View Fusion Module orchestrates the integration of $\mathcal{S}$, ${\tilde{\mathit{i}}}_m$, and ${\widehat{\mathit{h}}}_m$ into a comprehensive interaction representation $\mathcal{F}$ from multiple perspectives.

Step 1: High-Order Modality-Preserving Fusion. We first project ${\widehat{\mathit{h}}}_m$ to a lower dimension:

$${\tilde{\mathit{h}}}_m=\mathit{ReLU}({\mathit{W}}_h{\widehat{\mathit{h}}}_m+{\mathit{b}}_h),$$

(6)

where ${\tilde{\mathit{h}}}_m\in {\mathbb{R}}^{16}$. Subsequently, a high-order tensor interaction is modeled:

$$\mathcal{M}=\mathit{FC}\left(\mathit{MaxPool}({\tilde{\mathit{h}}}_t\otimes {\tilde{\mathit{h}}}_v\otimes {\tilde{\mathit{h}}}_a)\right),$$

(7)

where ⊗ is the outer product.

Step 2: Modality-Individual Fusion. Similarly:

$$\mathcal{I}=\mathit{FC}\left(\mathit{MaxPool}({\tilde{\mathit{i}}}_t\otimes {\tilde{\mathit{i}}}_v\otimes {\tilde{\mathit{i}}}_a)\right).$$

(8)

Step 3: Gated Semantic-Aware Fusion. To dynamically modulate the contribution of $\mathcal{S}$, $\mathcal{M}$, and $\mathcal{I}$, we introduce a learnable gating mechanism:

$${\mathcal{F}}_0=\mathit{Concat}\left([\mathcal{S},\mathcal{I},\mathcal{M}]\right),$$

(9)

$$\mathcal{G}=\sigma ({\mathit{W}}_g{\mathcal{F}}_0+{\mathit{b}}_g),$$

(10)

$$\mathcal{F}=\mathcal{G}\odot {\mathcal{F}}_0,$$

(11)

where $\sigma$ is the sigmoid function, and ⊙ is element-wise multiplication.

3.4. Contrastive Regularization and Classification Head

Inspired by recent contrastive learning paradigms, we introduce an auxiliary Instance-Level Contrastive Regularization to encourage $\mathcal{S}$ to maintain cross-modal alignment:

$${\mathcal{L}}_{\mathrm{contrast}}=-log{\displaystyle \frac{exp(\mathrm{sim}({\mathcal{S}}_i,{\mathcal{S}}_j)/\tau )}{{\sum }_{k}exp(\mathrm{sim}({\mathcal{S}}_i,{\mathcal{S}}_k)/\tau )}},$$

(12)

where $\mathrm{sim}(·)$ denotes cosine similarity and $\tau$ is the temperature hyperparameter.

Finally, the fused representation $\mathcal{F}$ is passed through a three-layer MLP for emotion prediction, with either:

$${\mathcal{L}}_{\mathrm{CE}}=-\sum _{c=1}^C{y}_{c}log{\widehat{y}}_c,$$

(13)

or for regression:

$${\mathcal{L}}_{\mathrm{MSE}}={\displaystyle \frac{1}{N}}\sum _{i=1}^N{(y_i-{\widehat{y}}_i)}^2.$$

(14)

Total Loss:

$${\mathcal{L}}_{\mathrm{total}}={\mathcal{L}}_{\mathrm{CE}/\mathrm{MSE}}+\lambda {\mathcal{L}}_{\mathrm{contrast}},$$

(15)

where $\lambda$ balances the contrastive regularization.

4. Experiments

In this section, we conduct comprehensive empirical evaluations to validate the effectiveness, efficiency, and robustness of the proposed MIMIC framework. We compare MIMIC against a series of strong multimodal baselines across multiple metrics on two widely-used datasets, MOSI [7] and MOSEI [8]. Furthermore, we perform detailed ablation studies to analyze the contributions of each component within MIMIC. Additionally, we investigate the modality contribution, computational cost, and representation disentanglement capability of our framework. All experiments strictly follow standard settings to ensure fair and credible comparisons.

4.1. Experimental Setup and Implementation Details

Implementation Details. All models are trained using the Adam optimizer with an initial learning rate of 0.0001, and early stopping is applied with a patience threshold of 10 epochs. The mini-batch size is set to 64, and training is performed on a single NVIDIA RTX 3090 GPU. Our models are optimized using the cross-entropy loss for classification tasks and the mean squared error loss for regression tasks, depending on the dataset annotations. For reproducibility, we plan to release the source code and pre-trained models upon acceptance.

Datasets and Preprocessing. We evaluate our models on two benchmark datasets: MOSI and MOSEI. MOSI comprises 2,199 utterance-level samples from 89 speakers, with annotations in the range of $[-3,+3]$. MOSEI is a larger-scale dataset containing 23,453 samples across 1,000 speakers and 250 topics. We adopt standard preprocessing pipelines: text features are extracted using GloVe [9] and BERT [12]; audio features are extracted using COVAREP [11] and Speech-BERT [13]; and visual features are obtained via Facet [10] or our proposed visual encoder. All modalities are aligned at the word level using P2FA.

4.2. Comparison with State-of-the-Art Methods

We benchmark MIMIC against state-of-the-art methods including MFN [3], MV-LSTM [14], RAVEN [15], MulT [16], TFN [2], LMF [4], MFM [17], ICCN [18], MISA [5], and SSL [19]. Evaluation metrics include mean absolute error (MAE), Pearson correlation coefficient (Corr), binary accuracy (Acc-2), F1 score, and 7-class accuracy (Acc-7).

Performance Comparison Results. Tables below present the comparative results. Our MIMIC achieves consistent improvements over all baselines across all metrics and datasets. On MOSEI, MIMIC (C) achieves a MAE of 0.543 and a Corr of 0.764, outperforming strong baselines like MulT and RAVEN. On MOSI, MIMIC achieves superior results with a MAE of 0.793 and Corr of 0.756, surpassing all classical feature baselines.

When equipped with BERT and Speech-BERT features ($B^T$ and $B^{TS}$), MIMIC maintains its superiority, demonstrating its robustness under diverse feature settings. Notably, even when visual modality is removed, MIMIC still achieves competitive or superior performance, highlighting its flexibility in partial modality scenarios.

In-depth Discussion. We observe that methods like SSL which utilize two modalities (text-BERT and audio-BERT) achieve impressive results but still lag behind our proposed MIMIC in both MOSI and MOSEI datasets. This verifies that our fusion mechanism and decoupling strategy can better harness the complementarity among modalities.

Modality Ablation Analysis. We further provide modality removal analysis in Table 1. Results show that text modality remains the most critical signal, while removing visual or audio still causes a noticeable drop in performance, reinforcing the multimodal synergy captured by MIMIC.

4.3. Ablation Study on Module Effectiveness

We perform ablation studies on MIMIC’s core modules, including interaction representation components ($\mathcal{S}$, $\mathcal{M}$, $\mathcal{I}$) and decoupling strategies. The results in Table 1 confirm that all three interaction representations are indispensable and jointly contribute to the overall performance. Furthermore, our instance-based decoupling strategy consistently outperforms orthogonal constraint-based decoupling.

New Representation Efficiency Comparison. To investigate the decoupling effectiveness, we compute the average cosine similarity between modality-individual representations. As shown in Table 2, MIMIC achieves significantly lower inter-modality similarity, indicating superior disentanglement capacity.

Training Convergence and Efficiency Analysis. We also analyze the training speed and resource consumption. Table 3 shows that MIMIC reduces both parameters and FLOPs compared to MISA while achieving faster convergence and higher final accuracy.

Representation Dynamics Visualization. Furthermore, we observe the dynamic change of representation similarities during training. The similarity first rises as representations are aligned, then drops as they are disentangled by the decoupling operation, confirming the effectiveness of our instance-aware design. All the above extensive experiments rigorously validate that MIMIC achieves state-of-the-art performance, superior efficiency, and robust representation learning capability across diverse multimodal emotion recognition scenarios.

5. Conclusions and Future Work

In this work, we have presented MIMIC (Multi-Interaction Modeling with Intelligent Coordination), an innovative and carefully engineered deep learning framework specifically designed for addressing the intricate challenge of multimodal emotion analysis. Motivated by the inherent complexity and heterogeneity of multimodal signals such as speech, text, and visual expressions, MIMIC systematically models these diverse modalities by disentangling and coordinating their interactions from multiple complementary perspectives.

Through extensive empirical evaluations on two widely adopted benchmark datasets, MOSI and MOSEI, our approach demonstrates consistently superior performance over a wide spectrum of competitive baseline models across all major evaluation metrics. Specifically, MIMIC not only achieves remarkable improvements in prediction accuracy and correlation but also exhibits significant advantages in terms of computational efficiency, model compactness, and training stability. These findings substantiate the efficacy and practicality of the proposed method, making it well-suited for real-world applications where computational resources may be constrained.

A critical innovation within our proposed framework lies in the introduction of a novel instance-aware decoupling operation, which allows for the efficient decomposition of modality-specific representations into a modality-shared component and distinct modality-individual components. This decoupling strategy operates in a parameter-free and constraint-free manner, obviating the need for complex orthogonality losses or additional balancing hyper-parameters, which often complicate the training process in prior works like MISA [5]. Instead, our lightweight design seamlessly integrates into the end-to-end training pipeline, facilitating the model’s ability to capture both the commonality and the uniqueness embedded within each modality.

Furthermore, our proposed hierarchical multi-view fusion module builds upon these decoupled representations, introducing a high-order interaction modeling mechanism that leverages tensor operations and semantic-driven fusion gates to generate a comprehensive, discriminative, and robust multimodal representation space. This mechanism ensures that the full representational capacity of multimodal information is effectively unleashed, enabling the model to reason over complex inter-modal relationships that are often overlooked by conventional fusion strategies.

Beyond its performance merits, MIMIC is also designed with deployment efficiency in mind. Our analyses show that MIMIC significantly reduces the number of trainable parameters and FLOPs while accelerating convergence speed during training, making it an appealing choice for emotion recognition tasks in resource-constrained or real-time environments.

Future Directions. While MIMIC already demonstrates strong capability in modeling multimodal interactions, several avenues remain open for future exploration:

• Intra-utterance Fine-Grained Interaction Modeling: Currently, MIMIC operates at the utterance level with a focus on global representations. We plan to extend our framework to incorporate fine-grained intra-utterance fusion mechanisms that can capture more localized emotional dynamics, such as micro-expressions or prosodic variations, which may further enhance the model’s sensitivity to subtle emotional cues.
• Adaptive Cross-Modality Similarity Learning: Although our instance-aware decoupling already demonstrates strong disentanglement capacity, we aim to further enhance the model’s ability to dynamically adjust the similarity space between modalities during training. By integrating contrastive learning or dynamic margin strategies, we anticipate being able to bridge modality gaps more effectively and promote better cross-modal alignment.
• Broader Applicability in Open-World Scenarios: Future work will also investigate the extension of MIMIC to more diverse and challenging datasets beyond MOSI and MOSEI, including open-world, multilingual, and multi-cultural datasets, to evaluate the generalization capacity of the model in handling rich and diverse emotional expressions.
• Lightweight Deployment and Edge Adaptation: To further promote the deployment of our model in real-world edge devices or mobile platforms, we intend to explore model pruning, quantization, and knowledge distillation techniques to develop lightweight variants of MIMIC without sacrificing performance.
• Integration with Large Pre-trained Multimodal Models: As the field of multimodal foundation models advances rapidly, we also plan to investigate how our MIMIC can be integrated or adapted into such large-scale models, leveraging their general knowledge while preserving the fine-grained emotional reasoning capability brought by our multi-view interaction modeling.

In conclusion, this work offers a robust and versatile framework for multimodal emotion understanding, bridging the gap between modality-specific features and joint reasoning, and paving the way toward more holistic, efficient, and interpretable emotion-aware AI systems.