FusionX: A Symbolic-fused Multimodal Emotion Interaction Framework

1. Introduction

Emotion recognition through computational means has long been an ambitious and foundational objective in the field of artificial intelligence [12,16]. The human ability to infer affective states relies on the fluid integration of various communicative modalities, such as spoken language, acoustic tone, and visual expression. These signals, inherently diverse in their form and function, interact in a sophisticated and often subtle manner, enabling rich emotional communication and understanding [2,42]. However, equipping machines with this level of nuanced perception remains a formidable challenge.

In natural human interaction, emotion is rarely communicated through a single channel. Rather, emotions are encoded redundantly and complementarily across modalities. A spoken sentence may contain emotional nuances not just in its lexical content, but also in the tone, pitch, and facial expressions that accompany it. For instance, the phrase “I’m fine” may convey entirely different emotional states depending on how it is uttered and what facial expression it accompanies. This inter-modal dependency makes it necessary for any machine-oriented emotion analysis model to handle cross-modal relationships with both sensitivity and structural awareness.

Recent advances in multimodal learning have explored various fusion techniques to integrate modalities. Early approaches adopted simple strategies such as concatenating feature vectors across modalities to generate unified representations [42]. While intuitive, these approaches often suffered from what is known as modality collapse or modality dominance, where the model disproportionately relies on one modality—typically the text modality—while ignoring others. More refined attention-based approaches attempted to dynamically adjust modality contributions [43,45], but still faced limitations in generalization, often overfitting to dominant signals in the training distribution. High-order tensor-based fusion methods such as LMF and its variants [15,20,42] attempted to capture richer inter-modal correlations, yet introduced significant computational complexity and dimensional inefficiencies.

A fundamental cause of these challenges is the heterogeneous nature of the modalities themselves. Audio features capture prosody and rhythm, video features focus on spatial-temporal expressions, and textual inputs deliver explicit semantic content. These signals differ not just in form but also in abstraction level, making unified modeling highly nontrivial. Efforts to bridge these differences have led to techniques involving modality-specific and modality-invariant disentanglement [10,11,13,34,39]. Such strategies typically rely on auxiliary losses and handcrafted regularization terms, introducing additional hyperparameters and training instability. Although they show improvements in performance, they often require extensive tuning and are not easily scalable across datasets or domains.

To navigate these multifaceted difficulties, we propose FusionX, a novel framework designed around the principle of symbolic decomposition and high-order integration. FusionX adopts a three-way perspective on multimodal interaction: modality-complete representations preserve the full characteristics of each individual modality; modality-synergistic representations encode the overlapping, shared contributions across modalities; and modality-unique representations focus on what makes each modality distinct from others. This structure allows the model to fully exploit complementary and contrasting aspects of multimodal signals while preserving their individual integrity.

FusionX further integrates these perspectives through a hierarchical fusion architecture named Text-Centric Hierarchical Tensor Fusion (TCHF). Inspired by prior work showing the strong contribution of textual semantics to emotion interpretation [26,35], TCHF places the text modality at the core of the fusion process. It does so not by naively favoring text, but by structuring cross-modal alignment around linguistic anchors, ensuring that audio and visual features are interpreted in context. The hierarchical design enables both intra-modal enhancement and inter-modal synthesis, facilitating fine-grained control over interaction dynamics.

Finally, we evaluate our approach on three prominent multimodal emotion datasets: MOSI [9], MOSEI [44], and IEMOCAP [3]. Experimental results consistently demonstrate that FusionX outperforms a range of strong baselines, including recent transformer-based and disentanglement-based architectures. Not only does FusionX achieve superior performance in both classification and regression settings, but it also does so with a simplified training procedure and fewer hyperparameters, highlighting the effectiveness of symbolic interaction modeling and hierarchical integration.

Our key contributions are as follows:

–

We propose FusionX, a multi-perspective symbolic framework for multimodal emotion understanding that decouples and recombines interaction signals with structural clarity and semantic grounding.

–

We introduce Text-Centric Hierarchical Tensor Fusion (TCHF), a novel integration strategy that centers linguistic signals while hierarchically synthesizing multimodal cues to enhance emotion comprehension.

–

We demonstrate that symbolic decomposition into complete, synergistic, and unique representations enables more effective and interpretable emotion analysis, without relying on complex auxiliary constraints.

–

We conduct thorough evaluations on widely used benchmarks (MOSI, MOSEI, IEMOCAP), where FusionX achieves state-of-the-art performance with robust generalization and reduced model complexity.

2. Related Work

The task of multimodal emotion recognition presents significant challenges, primarily arising from two intertwined dimensions: (1) how to represent unimodal signals effectively, and (2) how to model interactions among these diverse modalities to yield powerful fusion representations. Recent research has explored both dimensions extensively, leveraging classical hand-crafted features, end-to-end learning approaches, and, more recently, large-scale pre-trained models to enhance generalizability. In what follows, we discuss the progression of research in both unimodal and multimodal representation learning, with a particular focus on methods that inspire the design of our proposed framework, FusionX.

2.1. Learning Representations from Individual Modalities

Early efforts in multimodal emotion analysis often relied on classical hand-crafted features to represent each modality separately. Benchmark datasets such as MOSI [46], MOSEI [44], and IEMOCAP [3] provide a standard suite of unimodal signals—including textual transcripts, raw audio waveforms, and video streams. For textual inputs, word-level representations using GloVe embeddings [24] have been widely adopted. In the audio domain, COVAREP features [6] are often used to capture prosodic and spectral characteristics. For visual inputs, emotion-related features are typically extracted from facial movements using systems like Facet, OpenFace, and FaceNet, which provide facial action unit (AU) estimates and spatial descriptors.

Despite the wide adoption of such feature pipelines, substantial limitations exist. In particular, AU detection and intensity estimation from facial videos remain difficult in unconstrained environments, often suffering from low accuracy and inconsistency. Moreover, the reliance on pre-extracted features may limit the model’s capacity to capture subtle emotional cues in real-time interaction.

To address these issues, several studies [27] have explored end-to-end training pipelines where raw modalities (e.g., audio waveforms, word sequences, facial frames) are directly fed into deep neural networks for joint feature learning and emotion classification. Although promising, these approaches are prone to overfitting, particularly due to the relatively limited size and variability of existing multimodal emotion datasets. Without sufficient regularization or data augmentation, the learned models often fail to generalize beyond the training domain.

The introduction of pre-trained transformers has brought transformative changes to unimodal representation learning. For textual modality, BERT [7] and its variants have demonstrated remarkable performance in encoding context-aware semantic information. In the field of speech processing, models such as Speech-BERT [1,32] have been adapted to extract rich paralinguistic features from raw audio inputs. These pre-trained models, when fine-tuned on downstream emotion recognition tasks, yield significantly more robust and generalizable features compared to classical alternatives or purely end-to-end pipelines. MISA [13] was among the first in multimodal emotion analysis to successfully employ BERT as a text encoder, showing substantial gains in performance.

However, the progress in pre-trained modeling for visual modality lags behind. While OpenFace and Facet provide essential low-level features such as AU activations, they lack the abstraction and contextual awareness characteristic of language-based pre-trained models. The lack of vision-specific pre-training frameworks tuned for emotional inference represents a key gap. Addressing this gap is one of the motivations behind FusionX, which aims to integrate richer and more expressive visual representations in a modular and flexible manner.

2.2. Fusion Mechanisms for Multimodal Representation Learning

Once unimodal features are extracted, the next challenge is to model the interactions among these heterogeneous modalities. The goal is to learn a comprehensive fusion representation that captures both complementary and contrasting information from each modality, enabling accurate and context-sensitive emotion understanding.

A widely-used category of methods—referred to here as integration-based learning—fuses unimodal features by directly combining them, typically through concatenation [42], attention-based mechanisms [43,45], or bilinear pooling techniques [15,20]. These methods emphasize the joint behavior of intact modality-specific signals and aim to extract shared semantics across modalities. While straightforward to implement and often effective in performance, such integration methods face several limitations. The most notable among them is the problem of modality imbalance, where dominant modalities like text can overshadow the contributions of others, leading to suboptimal fusion representations.

To overcome these limitations, another line of work introduces what we refer to as decoupling-based learning. In particular, MISA [13] proposed a decomposition strategy that separates unimodal representations into modality-invariant and modality-specific components. This dual-view strategy acknowledges that emotion is encoded not only in the synchronized interaction across modalities but also in the unique, modality-specific signals that may not be present in others. By applying orthogonal constraints and regularization techniques, MISA effectively reduces redundancy and improves the clarity of the fusion process. However, the use of complex auxiliary losses and additional trainable parameters increases the computational overhead and makes training more sensitive to hyperparameter tuning.

Building upon these insights, FusionX incorporates the strengths of both paradigms while mitigating their respective drawbacks. Instead of relying solely on either full integration or complete decoupling, FusionX proposes a symbolic multi-view decomposition strategy, introducing three types of interaction representations: complete (modality-full), shared (modality-synergistic), and specific (modality-unique). These representations are then fused through a hierarchical mechanism that prioritizes semantic alignment with the textual modality, which has been consistently shown to be the most reliable cue in emotion detection [26,35]. This fusion strategy enables FusionX to model high-order, asymmetric interdependencies among modalities in a structurally principled way.

In summary, the evolution of multimodal emotion analysis has transitioned from handcrafted feature engineering to deep end-to-end learning and is now moving towards modular, pre-trained, and structured integration. FusionX continues this trajectory by embracing symbolic decomposition and text-centric fusion as guiding principles, aiming to deliver both performance gains and interpretability in modeling emotional intelligence.

3. Proposed Framework

This section presents our proposed multimodal emotion analysis framework, named FusionX, which systematically models emotion-related signals from text, audio, and visual modalities. As shown in Figure 1, the framework is composed of three main modules: (1) Unimodal Encoding Module ($M^{enc}$), which captures temporally aligned utterance-level representations from raw input streams; (2) Nonparametric Self-decoupling Module ($M^{dec}$), which decomposes representations into modality-shared and modality-specific views; and (3) Text-dominated Hierarchical High-order Fusion Module ($M^{THHF}$), which integrates diverse representations into a unified, informative latent space for final prediction. Below, we describe each component in detail.

3.1. Unimodal Feature Encoding

Given a talking-face video clip, we first segment it into a sequence of utterances, where each utterance contains temporally aligned streams of text, audio, and video. For each modality $m\in \{t,a,v\}$, where t denotes text, a denotes acoustic, and v denotes visual inputs, we denote the temporal sequence as ${\mathit{u}}_m^l$ for $l=1,2,\cdots ,L$.

Each modality stream is passed through a modality-specific encoder ${\mathbb{E}}_m$, comprising two-layer bidirectional GRUs followed by a dense layer. This encoder transforms raw or pre-extracted features into fixed-length latent vectors:

$${\mathit{h}}_m={\mathbb{E}}_m\left({\left\{{\mathit{u}}_m^l\right\}}_{l=1}^L\right)\in {\mathbb{R}}^{64}$$

These representations are designed to capture contextual, temporally-aware, and modality-aligned signals. To prepare for fusion and reduce dimensionality, we apply a linear projection to obtain:

$${\tilde{\mathit{h}}}_m=W_m^{proj}{\mathit{h}}_m+b_m^{proj}\in {\mathbb{R}}^{16}$$

Different from previous approaches, for the visual modality, we adopt a pre-trained facial expression encoder trained on identity-excluded facial similarity constraints, enabling us to extract fine-grained emotion-centric facial cues beyond standard AU features.

3.2. Nonparametric Self-Decoupling of Representations

While the unimodal features ${\mathit{h}}_m$ provide discriminative signals individually, modeling cross-modal dependencies requires disentangling shared emotional semantics from modality-specific traits. We propose a nonparametric, instance-level self-decoupling mechanism that avoids reliance on auxiliary loss functions or complex regularizers.

The modality-shared representation $\mathcal{S}$ is computed as the average across all modality encodings:

$$\mathcal{S}={\displaystyle \frac{1}{3}}({\mathit{h}}_t+{\mathit{h}}_a+{\mathit{h}}_v)$$

(1)

Then, we obtain modality-specific residual representations by subtracting the shared part:

$${\mathit{i}}_m={\mathit{h}}_m-\mathcal{S},\forall m\in \{t,a,v\}$$

(2)

This subtraction-based mechanism emphasizes the modality-unique information not captured in the shared space. Each residual ${\mathit{i}}_m$ is then passed through a projection layer to yield a compact representation:

$${\tilde{\mathit{i}}}_m=W_m^{dec}{\mathit{i}}_m+b_m^{dec}\in {\mathbb{R}}^{16}$$

(3)

Compared to orthogonality-constrained methods such as MISA [13], our formulation maintains instance-level disentanglement and minimizes information redundancy without introducing training instability.

3.3. Hierarchical High-order Fusion with Text Dominance

Having obtained three types of representations—modality-full (${\tilde{\mathit{h}}}_m$), modality-shared ($\mathcal{S}$), and modality-specific (${\tilde{\mathit{i}}}_m$)—we now describe how to synthesize them using a hierarchical fusion strategy.

3.3.1. Fusion Branches

The fusion module $M^{THHF}$ consists of two symmetrical branches:

–

Modality-specific branch: Fuses ${\tilde{\mathit{i}}}_t$, ${\tilde{\mathit{i}}}_a$, and ${\tilde{\mathit{i}}}_v$ into interaction $\mathcal{I}$.

–

Modality-full branch: Fuses ${\tilde{\mathit{h}}}_t$, ${\tilde{\mathit{h}}}_a$, and ${\tilde{\mathit{h}}}_v$ into interaction $\mathcal{M}$.

In both branches, we adopt text-centered fusion by always pairing text features with other modalities and using high-order tensor operations (outer product) to capture multiplicative dependencies.

3.3.2. Text-Dominated High-order Fusion

We define the following outer-product-based tensor interaction for two modality pairs:

$${\tilde{\mathit{i}}}_{tv}={\tilde{\mathit{i}}}_t\odot \mathrm{FC}\left(\mathrm{Flatten}\left(\mathrm{Pool}({\tilde{\mathit{i}}}_t\otimes {\tilde{\mathit{i}}}_v)\right)\right)$$

(4)

$${\tilde{\mathit{i}}}_{ta}={\tilde{\mathit{i}}}_t\odot \mathrm{FC}\left(\mathrm{Flatten}\left(\mathrm{Pool}({\tilde{\mathit{i}}}_t\otimes {\tilde{\mathit{i}}}_a)\right)\right)$$

(5)

The operator ⊗ is an outer product, ⊙ denotes element-wise product (text-dominant weighting), and Pool applies max-pooling to mitigate redundancy. These pairwise fused representations are then combined into the final modality-specific fusion:

$$\mathcal{I}=\mathrm{FC}\left([{\tilde{\mathit{i}}}_{tv};{\tilde{\mathit{i}}}_{ta}]\right)$$

Similarly, we obtain $\mathcal{M}$ by applying the same operations to ${\tilde{\mathit{h}}}_m$. Lastly, we concatenate all three views into a unified representation:

$${\mathcal{F}}_0=[\mathcal{M};\mathcal{I};\mathcal{S}]$$

3.3.3. Final Projection and Normalization Gate

To further enhance the expressive power of the fused embedding ${\mathcal{F}}_0$, we introduce a two-layer projection module:

$$\mathcal{F}=\mathrm{NormGate}\left(\mathrm{ReLU}(W_1{\mathcal{F}}_0+b_1)\right)W_2+b_2$$

Here, NormGate is a normalization-based adaptive gating mechanism that rescales channels based on magnitude. The final $\mathcal{F}$ is used for downstream classification or regression.

3.4. Training Objective and Optimization Strategy

The proposed FusionX framework is designed to accommodate both categorical emotion classification and continuous sentiment intensity regression tasks, depending on the label type provided by the dataset (see Section 4 for details). In this section, we define the training objectives and learning strategy used to optimize the model.

3.4.1. Classification Loss

For datasets providing discrete emotion labels (e.g., "happy", "sad", "neutral"), the model performs multi-class classification using the standard cross-entropy loss. Let N denote the mini-batch size, ${\mathrm{y}}_i\in {\mathbb{R}}^C$ be the one-hot encoded ground truth label of the i-th sample, and ${\widehat{\mathrm{y}}}_i\in {\mathbb{R}}^C$ be the predicted probability vector over C emotion classes. The classification loss is computed as:

$${\mathcal{L}}_{\mathrm{CE}}=-{\displaystyle \frac{1}{N}}\sum _{i=1}^N\sum _{c=1}^C{\mathrm{y}}_i^{\left(c\right)}log{\widehat{\mathrm{y}}}_i^{\left(c\right)}$$

(6)

This loss function penalizes deviation between predicted distributions and true class indicators, encouraging the model to correctly classify each utterance.

3.4.2. Regression Loss

For datasets with continuous-valued sentiment or emotion intensity annotations (e.g., values in $[-3,3]$ or $[0,1]$), we treat the task as a regression problem and adopt the Mean Squared Error (MSE) loss:

$${\mathcal{L}}_{\mathrm{MSE}}={\displaystyle \frac{1}{N}}\sum _{i=1}^N{∥{\widehat{\mathrm{y}}}_i-{\mathrm{y}}_i∥}^2$$

(7)

Here, ${\widehat{\mathrm{y}}}_i$ is the model’s predicted scalar (or vector) intensity value for the i-th utterance, and ${\mathrm{y}}_i$ is the corresponding ground truth.

3.4.3. Regularization and Composite Objective

To mitigate overfitting and improve generalization across modalities, we apply $L_2$ weight regularization to the trainable parameters $\mathsf{\Theta }$ of the network:

$${\mathcal{L}}_{\mathrm{reg}}=\lambda {\parallel \mathsf{\Theta }\parallel }_2^2$$

(8)

where $\lambda$ is a hyperparameter controlling the strength of regularization.

The final training loss ${\mathcal{L}}_{\mathrm{total}}$ is a composite of the task-specific objective and the regularization term. It is defined as:

$${\mathcal{L}}_{\mathrm{total}}=\left\{\begin{array}{cc}{\mathcal{L}}_{\mathrm{CE}}+{\mathcal{L}}_{\mathrm{reg}},\hfill & \mathrm{if}\mathrm{classification}\mathrm{task}\hfill \\ {\mathcal{L}}_{\mathrm{MSE}}+{\mathcal{L}}_{\mathrm{reg}},\hfill & \mathrm{if}\mathrm{regression}\mathrm{task}\hfill \end{array}\right.$$

(9)

The model parameters are optimized using the Adam optimizer with weight decay and early stopping based on validation performance. We empirically find that joint supervision of modality-specific and shared representations through these losses improves both learning stability and final accuracy.

4. Experiment

4.1. Setups and Dataset Details

We conduct extensive empirical evaluations to validate the effectiveness and superiority of the proposed FusionX framework for multimodal emotion recognition. Our experiments are carried out across multiple benchmark datasets, including IEMOCAP [3], MOSI [46], and MOSEI [44], which have become standard benchmarks in the community. These datasets encompass text, visual, and audio signals, providing a comprehensive testbed for evaluating the modeling capacity of multimodal fusion frameworks.

Implementation Details. All models are trained using the Adam optimizer [17] with a learning rate of 0.0001 and a mini-batch size of 64. Early stopping is applied with a patience of 10 epochs. Training is conducted on a single NVIDIA RTX 3090 GPU. For all datasets, we follow the official train/val/test splits adopted by prior literature to ensure fair comparison.

Feature Settings. We consider two feature extraction settings. The first, denoted as C, refers to the classical handcrafted features provided by the datasets (e.g., GloVe [24], COVAREP [6], and OpenFace). The second setting leverages pre-trained encoders, including text-BERT [7] and speech-BERT [1], denoted as $B^T$ and $B^S$ respectively. When using both, we denote it as $B^{TS}$. We align modalities using P2FA [40] to obtain utterance-level synchronization.

Evaluation Metrics. We adopt standard evaluation metrics for both classification and regression settings. For MOSI and MOSEI, we report Mean Absolute Error (MAE), Pearson Correlation Coefficient (Corr), 7-class Accuracy (Acc-7), 2-class Accuracy (Acc-2), and F1-score. For IEMOCAP, we focus on binary Accuracy and F1-scores across four emotion categories (Happy, Angry, Sad, Neutral).

4.2. Comparisons with SoTA Methods

We compare FusionX with a broad spectrum of prior works including attention-based, tensor-based, memory-enhanced, and pre-trained representation-based models, such as MFN [43], MulT [36], MISA [13], ICCN [33], and SSL [32]. Table 1 presents the performance comparison on IEMOCAP.

Quantitative Results. Under the classical feature setting (C), FusionX achieves the highest accuracy and F1-score across three out of four emotion classes (Angry, Sad, Neutral), clearly outperforming all strong baselines. For example, in the "Angry" class, FusionX reaches an F1 of 89.8, outperforming the next-best method MulT (87.0) by +2.8 points. On the "Sad" class, FusionX obtains 86.5 F1 compared to 85.1 from RMFN and 85.9 from ICCN. For the "Neutral" emotion, FusionX leads by a margin of +4.6 over TFN (68.0 F1) and +4.4 over MFN (69.2 F1), showing superior robustness in handling ambiguous cases.

While MulT performs marginally better on "Happy" (88.6 F1), our model remains competitive at 86.3 and demonstrates consistent superiority on harder-to-detect emotions like Sad and Neutral. This confirms that the hierarchical fusion and decoupling strategy of FusionX is especially beneficial when signals are subtle or less visually distinguishable.

Pre-trained Feature Setting. When using BERT-based representations ($B^{TS}$), we remove the visual modality to mimic real-world conditions with missing modalities. FusionX still outperforms SSL [32] significantly. For instance, F1 improves from 93.6 to 93.9 on Angry and from 81.1 to 81.2 on Neutral. This highlights the model’s ability to extract and align the most informative cues from audio and text through its decoupling and text-centered fusion mechanisms.

In summary, Table 1 confirms that FusionX achieves strong performance across modalities and feature settings, validating the importance of its modular fusion strategy.

4.3. Analysis of Representation Decoupling

To validate the effectiveness of our Nonparametric Self-decoupling Module, we analyze the pairwise dependency between the decoupled representations. Table 2 reports the inner product distances between modality-specific features (${\mathbf{i}}_t$, ${\mathbf{i}}_v$, ${\mathbf{i}}_a$) and the shared component $\mathcal{S}$ on MOSEI. Our FusionX significantly reduces inter-modality dependency compared to MISA [13], which relies on loss-constrained decoupling.

For example, the interdependence between audio and text representations drops from 28.9 in MISA to just 4.3 in FusionX. This reduced overlap demonstrates a successful disentanglement that preserves the uniqueness of each modality while isolating the common latent space. Such a property is essential for minimizing redundancy and improving the generalizability of the fused representation.

4.4. Qualitative Visualization of Representations

To gain further insight into the quality of representations produced by FusionX, we project them into a 2D space using t-SNE [22]. The resulting layout shows distinct clustering among modality-specific vectors and the shared semantic component $\mathcal{S}$. Notably, the clusters for ${\mathbf{i}}_t$, ${\mathbf{i}}_v$, and ${\mathbf{i}}_a$ are clearly separated, indicating that each modality captures complementary aspects of emotional signals. In contrast, the shared representation $\mathcal{S}$ resides near the centroid, bridging the semantic gap across modalities. This qualitative evidence supports the efficacy of our self-decoupling design.

4.5. Ablation Study on Model Components

We conduct detailed ablation experiments, summarized in Table 3, to assess the impact of various components in FusionX.

Representation Contribution. Experiments A1 through A6 explore different combinations of shared ($\mathcal{S}$), modality-specific ($\mathcal{I}$), and outer-product-based ($\mathcal{M}$) representations. Removing any single component reduces performance across all datasets. Particularly, excluding $\mathcal{I}$ (A4) or using it alone (A6) consistently drops F1 scores by 2–5 points. This highlights that high-order and shared representations cannot substitute the rich semantics of modality-specific interaction.

Unimodal Input Ablations. A7 through A9 examine the impact of dropping a single modality. Removing text (A7) causes the steepest decline (F1 drop of more than 20 on MOSI), proving the textual modality remains the most critical for sentiment analysis. Audio and visual modalities also contribute meaningfully, as removing them reduces performance by 3–7 points depending on the dataset.

Decoupling Strategy Comparison. In A10, we substitute our nonparametric instance-based decoupling with the orthogonal constraint approach from MISA. Across all datasets, the performance drops by around 1–2 F1 points, indicating that our simpler yet direct decoupling yields more distinguishable representations with fewer overheads.

Fusion Mechanism Ablations. In A11-A14, we investigate the impact of the fusion pathway. Replacing the outer product (A12) or removing the hierarchical text-dominated fusion module (A11) leads to significant performance degradation. Notably, co-attentional layers (A14), while expressive, underperform due to lacking explicit modeling of high-order interactions. This validates the necessity of our structured and interpretable THHF design.

Dominance Modality Variants. A15 and A16 test visual- and acoustic-dominated variants of our model. In both cases, results are inferior to text-dominated FusionX. This aligns with prior literature suggesting text provides the clearest emotional signal, and our design choice of text-prioritized high-order fusion proves effective.

Through extensive quantitative comparisons and controlled ablations, we demonstrate that FusionX significantly outperforms existing methods in both classification and regression emotion recognition tasks. Its modular architecture, featuring self-decoupling and text-dominated high-order fusion, enables robust performance even with incomplete modality inputs. The results support FusionX as a compelling framework for advanced multimodal affective computing.

5. Conclusions

In this work, we introduce FusionX, a comprehensive and flexible framework designed to address the challenges of multimodal emotion analysis by capturing complex interactions among heterogeneous modalities. FusionX is specifically constructed to extract, disentangle, and integrate modality-specific and modality-shared cues from text, audio, and visual inputs in a unified pipeline. To this end, we propose a novel nonparametric self-decoupling strategy that enables the disentanglement of shared and unique information without the need for additional supervision, explicit regularization, or learnable parameters, which reduces computational complexity and improves interpretability.

FusionX further incorporates an innovative Text-dominated Hierarchical High-order Fusion (THHF) module, which serves as a powerful mechanism to merge the rich and diverse interactions derived from multiple representation spaces. Unlike prior works that often employ simplistic concatenation or early fusion techniques, THHF performs high-order cross-modality reasoning in a layered fashion, enabling fine-grained alignment and interaction modeling at different levels of abstraction. This design allows the framework to adaptively emphasize the dominant modality (text) while preserving complementary signals from audio and vision, leading to more robust and expressive emotion understanding.

To quantify the advantage of FusionX, we conduct extensive empirical studies on widely used benchmark datasets including MOSI, MOSEI, and IEMOCAP. The experimental results demonstrate that FusionX consistently outperforms existing state-of-the-art baselines across various evaluation metrics, including classification accuracy, F1 score, and correlation coefficients. Particularly on IEMOCAP, FusionX exhibits notable improvements in distinguishing subtle emotions such as sadness and neutrality, validating the effectiveness of our disentangling and high-order fusion mechanisms.

Mathematically, our proposed nonparametric decoupling strategy can be expressed as:

$${\mathbf{i}}_m={\mathbf{h}}_m-{\mathbb{P}}_{\mathcal{S}}\left({\mathbf{h}}_m\right),\mathrm{where}{\mathbb{P}}_{\mathcal{S}}\left({\mathbf{h}}_m\right)={\displaystyle \frac{\langle {\mathbf{h}}_m,\mathcal{S}\rangle }{{\parallel \mathcal{S}\parallel }^2}}\mathcal{S}$$

Here, ${\mathbf{h}}_m$ denotes the high-level representation of modality m, $\mathcal{S}$ is the estimated shared space across modalities, and ${\mathbf{i}}_m$ is the resulting modality-specific component. This formulation naturally ensures orthogonality between shared and private subspaces, without introducing explicit orthogonality constraints.

Future Directions. Although FusionX achieves strong performance, several promising extensions remain to be explored:

–

Dynamic Fusion Strategies. Future work can explore dynamic fusion architectures where the dominant modality is adaptively selected based on input uncertainty or context, rather than statically relying on text as the primary source.

–

Temporal Modeling. Incorporating temporal dependencies in video and audio streams using transformers or recurrent architectures may further enhance sequential reasoning and contextual emotion interpretation.

–

Cross-domain Generalization. Current benchmarks are limited in scope. Exploring FusionX under domain adaptation or cross-corpus settings would provide insights into its generalization capabilities.

–

Emotion Intensity Regression. Beyond categorical classification, future research can extend FusionX to predict emotion intensity on a continuous scale, which better reflects real-world affective states.

–

Explainability and Fairness. Adding interpretable attention mechanisms or saliency maps may shed light on which modality and which signal regions contribute most to predictions, helping identify biases and increase model trustworthiness in sensitive applications.

In summary, FusionX presents a highly generalizable and effective framework for emotion recognition across modalities. We believe it provides a solid foundation upon which future advances in affective computing can be built, both in theory and application.