Modeling Multimodal Emotion with Dynamic Interaction-Focused Representation Network

1. Introduction

Human beings naturally express emotions through diverse behavioral signals, including linguistic expressions and vocal characteristics [1,17]. Text, as one of the most prominent modalities, plays a vital role in encoding affective content via lexical choice, syntactic structure, and semantic nuance [12]. Yet, the written or transcribed textual content often lacks the expressive prosody and nuanced delivery found in spoken language, which limits its capacity to fully convey emotional intent in isolation. As such, leveraging additional modalities like audio becomes essential.

Audio contributes a unique set of paralinguistic features that reflect speaker emotions through variations in pitch, loudness, speaking rate, and energy dynamics [5,18]. These features offer essential cues that help disambiguate emotionally ambiguous utterances. For instance, the phrase “Are you sure?” can carry drastically different sentiments—ranging from sarcasm to excitement—depending on the speaker’s tone and delivery. Audio thus provides the necessary disambiguation when textual information is inconclusive.

The synergistic fusion of text and audio has long been a goal of multimodal sentiment analysis. However, most existing fusion techniques are hindered by their reliance on temporally aligned multimodal inputs [24]. In practice, spoken words and their corresponding audio signals often do not align perfectly at the word or frame level. This temporal misalignment disrupts crossmodal correspondence, thereby weakening the overall fusion process. Previous methods such as tensor-based fusion [19] and low-rank multimodal decomposition [7] have attempted to address multimodal integration, but they assume rigid synchrony and may overlook the distinct nature of each modality.

Further advances such as multistage fusion architectures [6] have made strides by decomposing the interaction process into sequential stages. However, these methods still depend on pre-aligned sequences and tend to obscure unimodal identity during fusion. As a result, they may inadvertently suppress meaningful signals from individual modalities, especially when one modality is more expressive than the other in certain contexts.

To overcome these limitations, we introduce DIFERNet, a dynamic and interaction-focused representation network tailored for asynchronous text-audio fusion. DIFERNet is designed with the objective of (1) maximizing the exploitation of intermodal interactions while (2) preserving the integrity of unimodal streams. This dual objective ensures that the fused representation is not only informative but also resilient to misalignment artifacts. Our method diverges from rigid fusion schemes by incorporating a self-adjusting transformer module that iteratively refines fusion embeddings using unimodal guidance.

Our pipeline begins with projecting modality-specific features into a common latent space using a crossmodal alignment transformation. This step ensures dimensional homogeneity and semantic compatibility. Subsequently, we introduce an interaction-guided attention mechanism that enables mutual conditioning between text and audio. This attention computes interdependency matrices that weight the importance of tokens or acoustic segments relative to their crossmodal counterparts.

The heart of DIFERNet lies in the dynamic fusion adjustment transformer. This module treats each unimodal representation as a corrective signal, rebalancing the initial fusion to emphasize contextually dominant cues. Such a mechanism ensures that text and audio can independently reinforce or modulate the joint representation, leading to a more expressive embedding. As a theoretical foundation, we define the correction process via a weighted update.

We validate our model on the CMU-MOSI [22] and CMU-MOSEI [23] datasets, which are widely recognized benchmarks for multimodal sentiment analysis. DIFERNet demonstrates consistent superiority over competitive baselines, with gains ranging from $2.3\%$ to $9.1\%$ across evaluation metrics such as accuracy, F1-score, and mean absolute error. Additionally, error analysis reveals that DIFERNet excels in correctly classifying emotionally subtle or ambiguous samples by leveraging the rich interaction between modalities.

In summary, our contributions are threefold:

–

We propose DIFERNet, a novel framework capable of learning robust fusion representations from unaligned text and audio inputs by emphasizing dynamic intermodal interaction.

–

A new transformer-based correction mechanism is introduced to adaptively refine fusion features while preserving essential unimodal semantics.

–

We conduct extensive evaluations on benchmark datasets, where DIFERNet achieves state-of-the-art performance and demonstrates superior capability in handling asynchronous multimodal data.

2. Related Work

2.1. Advancements in Multimodal Sentiment Analysis

Multimodal sentiment analysis (MSA) is a rapidly evolving subfield in artificial intelligence that endeavors to decode emotional signals by integrating information from various modalities such as text, audio, and video [16]. The rationale behind multimodal fusion lies in the hypothesis that each modality carries complementary aspects of sentiment—text delivers explicit semantic content, audio reflects prosodic and paralinguistic cues, while video provides facial expressions and gestures. Harnessing the synergy among these modalities has the potential to provide machines with a more holistic emotional understanding. Given the increasing importance of empathetic AI systems, MSA has attracted considerable research interest.

Early approaches to MSA emphasized simple concatenation of modality-specific features. For example, Williams et al. [15] adopted an early fusion strategy, combining feature vectors extracted from multiple modalities before feeding them into downstream models. This method yielded substantial performance gains over unimodal baselines, highlighting the benefits of multimodal learning. Nevertheless, early fusion often suffers from the heterogeneity of feature distributions and fails to exploit deeper intermodal dependencies.

To address this, more sophisticated frameworks have emerged. Zadeh et al. [21] proposed a hybrid architecture that utilizes a multi-attention block along with long-short term memory units to discover latent interactions between modalities. Similarly, Pham et al. [8] drew inspiration from machine translation and introduced the Multimodal Cyclic Translation Network (MCTN), which learns shared semantic representations by cyclically translating between modalities. This architecture not only improves generalization but also allows for robust unimodal testing using only text inputs, achieving state-of-the-art results at the time.

In parallel, researchers have aimed to model the temporal and contextual dynamics of emotional cues. Wang et al. [14] presented RAVEN, a Recurrent Attended Variation Embedding Network, which dynamically adjusts word embeddings based on nonverbal features like pitch and facial expression intensity. This innovation captures the nuanced interplay between speech and prosody, enabling the network to better interpret subtle emotional shifts. However, a shared limitation of these methods is their dependency on strict temporal alignment at the word level, which can be impractical in real-world applications due to noise and asynchronous sampling rates across modalities.

With the growing popularity of attention mechanisms, attention-based fusion models have come to dominate the MSA landscape. These approaches are capable of identifying the most relevant segments across modalities, regardless of their relative alignment. For example, Zadeh et al. [20] developed a delta-memory attention network that captures both crossmodal and temporal relationships through a dynamic memory system embedded in a System of LSTMs. Likewise, Ghosal et al. [3] proposed a Multi-modal Multi-utterance Bi-modal Attention (MMMU-BA) model, which applies modality-specific attention weights to extract high-impact features across utterances.

In our proposed DIFERNet framework, we build on this attention-based tradition. A crossmodal collaboration attention mechanism is integrated into the fusion initialization phase, encouraging rich contextual alignment between text and audio. Furthermore, we employ a crossmodal adjustment transformer module—motivated by the work of Tsai et al. [11]—to adaptively reshape fused representations using unimodal guidance. This allows our model to maintain the integrity of modality-specific signals while enhancing joint semantic interpretation, especially under conditions of sequence misalignment or sparse interaction.

2.2. Transformer Architectures in Multimodal Contexts

The transformer architecture, initially introduced by Vaswani et al. [13], revolutionized natural language processing by replacing recurrence with multi-head self-attention mechanisms. By facilitating parallel computation and enhancing long-range dependency modeling, transformers have since become foundational to a broad spectrum of tasks. The encoder-decoder design introduced in the original paper laid the groundwork for subsequent advancements in large-scale language modeling and representation learning.

Building on this foundation, transformer-based models like GPT [9] and BERT [25] further advanced the field by pretraining on massive text corpora and capturing bidirectional contextual cues. These models achieved remarkable success across tasks such as question answering, sentence classification, and machine translation. However, their applicability remained largely confined to unimodal textual data, leaving open the question of how to extend their strengths to the multimodal domain.

Recent efforts have begun bridging this gap. One notable contribution is the Multimodal Transformer (MulT) introduced by Tsai et al. [11], which leverages crossmodal attention blocks to directly attend to low-level representations from different modalities. By forgoing intermediate fusion steps, MulT enables deeper crossmodal interaction and surpasses previous models in predictive accuracy. However, despite its success, MulT primarily emphasizes intermodal attention and does not explicitly address the need to retain unimodal specificity. This often leads to overly homogenized representations where modality-unique information may be diluted or lost entirely.

Motivated by these insights, our DIFERNet framework adopts a more balanced approach. While we incorporate transformer-based attention mechanisms to strengthen modality interaction, we also introduce a novel crossmodal adjustment transformer. This component is specifically designed to preserve the distinct attributes of each modality while enabling their cooperative integration. Through recurrent updates and conditional modulation, DIFERNet achieves a dynamic equilibrium between intermodal fusion and unimodal preservation, which is crucial in cases of asynchronous input or modality-specific noise.

In conclusion, while the transformer architecture has been successfully adapted to multimodal learning, our work contributes a more nuanced perspective. By combining crossmodal collaboration with modality-specific refinement, DIFERNet represents a meaningful step toward emotionally intelligent AI systems that can process unaligned and heterogeneous input streams without sacrificing robustness or interpretability.

3. Proposed Methodology

This section introduces the architectural components and technical design of our proposed Dynamic Interaction-Focused Emotion Representation Network (DIFERNet). As shown in Figure 1, DIFERNet is designed to effectively fuse heterogeneous modalities, particularly unaligned text and audio streams, by dynamically modeling intermodal dependencies while preserving unimodal specificity. The entire framework is composed of three primary modules: (1) a crossmodal alignment module that standardizes the temporal and spatial dimensions of input features; (2) a fusion representation initialization module, which performs early-stage integration via attention-based interaction; and (3) a self-adjusting module that adaptively refines fusion representations using residual unimodal guidance.

We begin with a formal problem definition in Section 3.1, followed by detailed explanations of each module in Section 3.2, Section 3.3 and Section 3.4. Each module is rigorously defined with its corresponding computational principles and equations to ensure reproducibility.

3.1. Problem Formulation

Let $F_T\in {\mathbb{R}}^{l_T\times d_T}$ and $F_A\in {\mathbb{R}}^{l_A\times d_A}$ denote the raw feature representations extracted from the text and audio modalities, where $l_T$, $l_A$ are their respective sequence lengths and $d_T$, $d_A$ are the feature dimensions. Given that $F_T$ and $F_A$ are inherently unaligned—due to asynchrony in modality sampling or semantic boundaries—the goal is to obtain rich joint representations that leverage both intra- and inter-modal dynamics.

To this end, we define a transformation function $\Phi :(F_T,F_A)\to \mathcal{Y}$, where $\mathcal{Y}$ represents the sentiment prediction space (e.g., categorical sentiment classes or continuous emotion scores). DIFERNet aims to learn intermediate aligned representations $X_T$, $X_A$ and their attentive counterparts $X_{T^{\prime }}$, $X_{A^{\prime }}$ to construct fusion vectors $X_{T{A}^{\prime }}$ and $X_{T^{\prime }A}$, which are subsequently adapted via interaction-aware refinement modules.

3.2. Crossmodal Alignment Module

The first step in DIFERNet is to harmonize the representational spaces of $F_T$ and $F_A$, allowing effective interaction across modalities with differing feature formats. Following [11], we apply 1D temporal convolutions with distinct kernel widths and strides to normalize the feature sequence lengths and dimensions. Formally:

$$\begin{array}{cc}\hfill {\tilde{F}}_T& =\mathrm{Conv}1\mathrm{D}(F_T;k_T,s_T),\hfill \\ \hfill {\tilde{F}}_A& =\mathrm{Conv}1\mathrm{D}(F_A;k_A,s_A),\hfill \end{array}$$

(1)

where $k_{(·)}$ and $s_{(·)}$ denote kernel sizes and stride parameters, respectively. The output features are then fed into Bi-directional Gated Recurrent Units (Bi-GRU) to capture contextual dependencies:

$$X_T=\mathrm{BiGRU}\left({\tilde{F}}_T\right),X_A=\mathrm{BiGRU}\left({\tilde{F}}_A\right).$$

(2)

The resulting $X_T,X_A\in {\mathbb{R}}^{l\times d}$ (with l and d now unified) serve as aligned modality-specific embeddings for further crossmodal interaction.

3.3. Fusion Representation Initialization Module

To capture semantic correlations and modality interactions, we introduce a crossmodal collaboration attention mechanism. The attention operates bi-directionally, such that each modality queries the other for relevant features. Let the interaction matrices be:

$$M_{TA}=X_T{X}_A^{\top },M_{AT}=X_A{X}_T^{\top }.$$

(3)

We normalize these matrices using a soft-tanh combination:

$$S_{TA}=\mathrm{Softmax}(tanh\left(M_{TA}\right)),S_{AT}=\mathrm{Softmax}(tanh\left(M_{AT}\right)).$$

(4)

Attention-based projections are then computed:

$$O_{TA}=S_{TA}{X}_A,O_{AT}=S_{AT}{X}_T.$$

(5)

By element-wise interaction:

$$X_{T^{\prime }}=O_{TA}\odot X_T,X_{A^{\prime }}=O_{AT}\odot X_A,$$

(6)

where ⊙ denotes Hadamard product. Fusion representations are constructed as:

$$\begin{array}{cc}\hfill X_{T{A}^{\prime }}& =w_T{X}_T+w_{A^{\prime }}{X}_{A^{\prime }}+b_{T{A}^{\prime }},\hfill \\ \hfill X_{T^{\prime }A}& =w_{T^{\prime }}{X}_{T^{\prime }}+w_A{X}_A+b_{T^{\prime }A},\hfill \end{array}$$

(7)

with $w_{·}$ being learnable weights and $b_{·}$ denoting modality-specific biases.

3.4. Self-Adjusting Fusion Refinement Module

To preserve unimodal identity and adaptively refine fusion representations, we introduce a dual-path crossmodal adjustment mechanism. This is the core differentiator of DIFERNet, allowing $X_{T{A}^{\prime }}$ and $X_{T^{\prime }A}$ to be contextually rebalanced using unimodal cues.

3.4.1. Crossmodal Adjustment Transformer

Each adjustment transformer receives the fusion input and one modality-specific guide. Figure 2 shows the architecture. Prior to attention, we augment positional encoding $\mathrm{PE}$ as in [13]:

$${\mathrm{PE}}_{(pos,2i)}=sin(pos/{10000}^{2i/d}),{\mathrm{PE}}_{(pos,2i+1)}=cos(pos/{10000}^{2i/d}).$$

(8)

The inputs are enhanced as:

$$E_X=\mathrm{LayerNorm}(X+\mathrm{PE}).$$

(9)

Let $E_{T{A}^{\prime }}$, $E_T$, and $E_{A^{\prime }}$ be the normalized inputs for fusion and unimodal streams. We define N residual transformer layers where attention is defined as:

$$\mathrm{Attn}(Q,K,V)=\mathrm{Softmax}\left({\displaystyle \frac{Q{K}^{\top }}{\sqrt{d}}}\right)V.$$

(10)

In each block, the fusion is updated with unimodal information:

$$Q=E_{T{A}^{\prime }}^{(i-1)},K=V=E_T\mathrm{or}{E}_{A^{\prime }}.$$

(11)

$$Z^{\left(i\right)}=\mathrm{LayerNorm}(Q+\mathrm{Attn}(Q,K,V)),$$

(12)

$$E_{T{A}^{\prime }}^{\left(i\right)}=\mathrm{LayerNorm}(Z^{\left(i\right)}+\mathrm{FFN}\left(Z^{\left(i\right)}\right)),$$

(13)

where $\mathrm{FFN}$ denotes a position-wise feed-forward network. This process is repeated with both $E_T$ and $E_{A^{\prime }}$ for bi-guided refinement.

3.4.2. Global Fusion via Self-Attention

After refinement, both $X_{T{A}^{\prime }}$ and $X_{T^{\prime }A}$ are passed through self-attention transformers to extract temporal structure:

$$X_{TA}^{*}=\mathrm{SelfAttn}\left(X_{T{A}^{\prime }}\right),X_{AT}^{*}=\mathrm{SelfAttn}\left(X_{T^{\prime }A}\right).$$

(14)

The final representation is formed by concatenation and classified globally:

$$Z_{\mathrm{global}}=\mathrm{Concat}(X_{TA}^{*},X_{AT}^{*}),{\widehat{y}}_{\mathrm{global}}={\mathcal{C}}_g\left(Z_{\mathrm{global}}\right),$$

(15)

where ${\mathcal{C}}_g$ denotes the global classifier.

In parallel, local classifiers ${\mathcal{C}}_{ta}$ and ${\mathcal{C}}_{at}$ are applied:

$${\widehat{y}}_{ta}={\mathcal{C}}_{ta}\left(X_{T{A}^{\prime }}\right),{\widehat{y}}_{at}={\mathcal{C}}_{at}\left(X_{T^{\prime }A}\right).$$

(16)

3.4.3. Unified Loss Function

The complete loss function integrates predictions from global and local paths:

$${\mathcal{L}}_{\mathrm{total}}={\lambda }_1{\mathcal{L}}_{ta}+{\lambda }_2{\mathcal{L}}_{at}+{\lambda }_3{\mathcal{L}}_{global},$$

(17)

where ${\lambda }_i$ are tunable scalars, and each term is typically cross-entropy loss for classification:

$${\mathcal{L}}_{global}=-\sum _c{y}_{c}log\left({\widehat{y}}_{\mathrm{global},c}\right),$$

(18)

with $y_c$ being the true label and ${\widehat{y}}_{\mathrm{global},c}$ the predicted probability for class c.

This comprehensive architecture ensures that DIFERNet effectively captures both intermodal synergy and intramodal semantics, leading to enhanced performance in multimodal sentiment understanding.

4. Experiment

In this section, we present a comprehensive evaluation of our proposed model, DIFERNet (Dynamic Interaction-Focused Emotion Representation Network), using two widely adopted multimodal sentiment analysis benchmarks: CMU-MOSI and CMU-MOSEI. The evaluation framework encompasses multiple dimensions, including experimental configurations, feature extraction protocols, comparison with competitive baselines, and both quantitative and qualitative analyses. Our objective is to rigorously assess DIFERNet’s effectiveness in learning discriminative fusion representations from unaligned text and audio modalities.

4.1. Datasets and Configuration

We conduct experiments on two large-scale multimodal benchmarks: CMU-MOSI [22] and CMU-MOSEI [23].

CMU-MOSI contains 2199 opinion-labeled utterances across 93 video segments of online movie reviews. Each utterance is annotated on a continuous sentiment intensity scale ranging from $-3$ (strongly negative) to $+3$ (strongly positive). The audio stream is sampled at 12.5 Hz. The dataset is split into 52 training videos (1284 utterances), 10 validation videos (229 utterances), and 31 test videos (686 utterances), with no speaker overlap to avoid identity bias.

CMU-MOSEI comprises 23,454 labeled video clips from over 1000 speakers, providing a rich set of sentiment and emotion annotations. It is also annotated on a $[-3,+3]$ scale and sampled at 20 Hz for audio. Following standard practice [11], we adopt the official split and ensure the same evaluation settings as prior works to ensure comparability.

Model Configuration: For DIFERNet, we use 1D temporal convolution layers with 50 output channels, followed by Bi-GRU layers with 50 hidden units. Fully connected layers have 200 neurons with a dropout rate of 0.3. We use the Adam optimizer with a learning rate of 0.001 and train using mini-batches of size 12 for 20 epochs. Loss functions are computed using a combined $L_1$ and cross-entropy formulation to accommodate both classification and regression subtasks.

4.2. Modality-Specific Feature Engineering

To ensure consistency with previous studies [10,11], we adopt standardized preprocessing techniques for extracting unimodal features.

4.2.1. Textual Embedding

We convert transcriptions into sequences of 300-dimensional vectors using GloVe embeddings pretrained on the 840B Common Crawl corpus. These embeddings provide rich semantic features and maintain high performance across a variety of NLP tasks.

4.2.2. Acoustic Features

We extract low-level acoustic descriptors using the COVAREP toolkit [2]. Each utterance is represented as a 74-dimensional feature vector that includes MFCCs, fundamental frequency measures, glottal source parameters, peak slope, and maxima dispersion. The features are sampled at 100 Hz to capture fine-grained prosodic variations.

4.3. Metrics and Evaluation

To evaluate both classification and regression performance, we use five widely accepted metrics:

- $Ac{c}_7$: Accuracy for 7-class sentiment classification. - $Ac{c}_2$: Binary classification accuracy (positive vs. negative). - $F1$: F1-score for binary sentiment analysis. - $MAE$: Mean Absolute Error for sentiment intensity prediction. - $Corr$: Pearson correlation between predicted and ground-truth sentiment scores.

Higher scores are preferable for $Ac{c}_7$, $Ac{c}_2$, $F1$, and $Corr$, whereas lower is better for $MAE$. To ensure statistical stability, we average results over five independent runs using different random seeds.

4.4. Benchmarking Against Strong Baselines

We benchmark DIFERNet against several competitive multimodal models:

EF-LSTM: Early-fusion model concatenating inputs before feeding them into a shared LSTM.

LF-LSTM: Late-fusion model processes each modality independently and merges outputs via concatenation.

MCTN [8]: Learns joint embeddings via cyclic modality translation.

RAVEN [14]: Dynamically modulates word embeddings using nonverbal cues.

MulT [11]: Transformer-based model with directional crossmodal attention.

DIFERNet: Our proposed model that combines crossmodal collaborative attention with unimodal-preserving refinement.

5. Results and Discussion

5.1. Quantitative Analysis

In this section, we present a comprehensive quantitative evaluation of our proposed architecture, DIFERNet, and benchmark its performance against a series of competitive baselines. The evaluation covers two datasets—CMU-MOSI and CMU-MOSEI—and includes both classification and regression metrics. Additionally, we investigate the influence of the number of crossmodal blocks within DIFERNet to understand how architectural depth affects its discriminative power.

5.1.1. Performance Comparison with Baseline Models

Table 1 reports the experimental results on the CMU-MOSI dataset. Despite relying solely on two modalities—text and audio—our model significantly surpasses most existing methods that incorporate all three modalities (text, audio, and video). This observation highlights the effectiveness of DIFERNet’s dynamic adjustment mechanisms and its ability to extract rich sentiment information from asynchronous input streams.

In the binary sentiment classification task, our model achieves an $Ac{c}_2^h$ score of $82.0\%$ and an $F{1}^h$ score of $82.1\%$, representing an absolute improvement of $3.5\%$–$8.4\%$ over traditional recurrent-based models such as EF-LSTM and LF-LSTM, and a noticeable margin over advanced architectures like RAVEN and MCTN. Even compared to the transformer-based MulT model, which uses three modalities, DIFERNet delivers comparable or superior performance, which is especially remarkable given its lighter input modality setting.

For sentiment score classification ($Ac{c}_7^h$), DIFERNet achieves $36.8\%$, exceeding the performance of most baseline systems by a margin of $2.5\%$–$5.8\%$. While the original MulT using three modalities reports a slightly higher score ($39.1\%$), a fair comparison must consider the setting where only text and audio are used. Under this condition, DIFERNet outperforms MulT on all metrics, including an improvement of $0.7\%$ on $Ac{c}_7^h$, $1.9\%$ on binary accuracy ($Ac{c}_2^h$), and $2.0\%$ on $F{1}^h$.

In the regression setting, DIFERNet achieves a Mean Absolute Error ($MA{E}^l$) of $0.873$ and a Pearson correlation coefficient ($Cor{r}^h$) of $0.712$, indicating its capability to capture fine-grained sentiment intensity. Compared to MulT with text and audio, DIFERNet reduces error by approximately $0.083$ and boosts correlation by $0.032$. These improvements suggest that the self-adjusting module effectively retains modality-specific nuances during the fusion process.

Table 2 shows the results on the CMU-MOSEI dataset, further demonstrating the generalizability of our approach. In binary classification, DIFERNet achieves $Ac{c}_2^h=82.3\%$ and $F{1}^h=82.7\%$, outperforming most prior methods by $1.5\%$–$5.5\%$ and also improving upon MulT (text+audio only) by $0.6\%$ and $0.7\%$, respectively. The 7-class accuracy reaches $51.2\%$, a relative improvement of $1.0\%$ over MulT and a significant gain over earlier methods like RAVEN and MCTN.

In the regression task, DIFERNet attains a $MA{E}^l$ of $0.573$ and a $Cor{r}^h$ of $0.701$. These results exceed all comparative baselines, including the full-modality version of MulT. The performance gap between our model and models using video suggests that effective dynamic modeling between unaligned text and audio modalities can compensate for the absence of visual features when done correctly.

Overall, DIFERNet consistently outperforms baseline systems on both datasets, validating the efficacy of its architecture. The results clearly demonstrate that (1) deep crossmodal attention enhances inter-modal synergy, and (2) preserving unimodal pathways during late-stage adjustment mitigates feature suppression and semantic dilution—a common issue in multimodal fusion.

5.1.2. Influence of Crossmodal Block Depth

To explore the sensitivity of DIFERNet to the number of crossmodal blocks, we conduct an ablation study on the CMU-MOSI dataset by varying the total number of transformer layers within the crossmodal adjustment module. As illustrated in Table 2, we experiment with values of $n=2,4,6,8,10,12,14$, where each configuration assigns $n/2$ blocks for text-to-fusion adjustment and $n/2$ for audio-to-fusion refinement.

The results reveal a clear trend: performance (measured by $Ac{c}_2^h$) improves steadily as the number of blocks increases from 2 to 10. This indicates that a deeper attention structure facilitates more expressive alignment between the modalities, enabling the network to learn complex temporal and semantic dependencies. The best performance is achieved when $n=10$, suggesting a sweet spot between representation richness and overfitting risk.

Interestingly, further increasing the number of blocks beyond 10 leads to a marginal drop in accuracy. This decline is likely due to over-parameterization and gradient instability in deep attention stacks, especially when training data is relatively limited. These findings suggest that while deeper attention enables richer fusion, a controlled architecture depth is necessary to maintain generalization.

6. Conclusion

In this study, we introduced a novel architecture named DIFERNet (Dynamic Interaction-Focused Emotion Representation Network) that targets the challenge of modeling sentiment from unaligned multimodal sequences, specifically focusing on the interplay between text and audio signals. Unlike conventional multimodal fusion methods that either rely heavily on modality alignment or inadequately preserve modality-specific information, DIFERNet is uniquely designed to dynamically regulate inter-modal interactions while simultaneously maintaining the distinct expressive characteristics of each modality.

At the heart of our model lies the crossmodal adjustment transformer, which enables DIFERNet to adaptively refine its fusion representations based on unimodal semantic cues. By integrating both local modality-aware updates and a global interaction modeling mechanism, DIFERNet ensures that neither modality is suppressed during fusion and that the joint representations remain expressive, context-sensitive, and temporally coherent.

Extensive experiments on two benchmark datasets, CMU-MOSI and CMU-MOSEI, confirm the superior performance of our method across both classification and regression metrics. Even though DIFERNet utilizes only two modalities (text and audio), it consistently outperforms or matches state-of-the-art models that rely on all three modalities, including video. This highlights the strength of our dynamic fusion strategy and its ability to compensate for the lack of visual input by leveraging deeper semantic alignment and residual unimodal correction. Furthermore, qualitative analysis illustrates that DIFERNet can make sentiment predictions more aligned with human perception, particularly in cases where unimodal cues may be ambiguous or contradictory.

In addition to its quantitative advantages, the architecture of DIFERNet offers practical benefits: it is modular, interpretable, and computationally efficient. Each component—from the attention-based initialization to the adaptive refinement module—contributes to a more robust understanding of sentiment in realistic, noisy, and asynchronous multimodal scenarios.

Looking forward, we recognize that the rapid progress of large-scale pre-trained models offers new opportunities to enhance multimodal sentiment analysis. As part of our future work, we plan to investigate how powerful pretrained language models such as BERT, RoBERTa, or GPT can be extended beyond their unimodal origins to support dynamic crossmodal understanding. One promising direction involves initializing the textual backbone of DIFERNet with pre-trained language representations and then coupling it with crossmodal adaptation layers capable of fine-tuning jointly across modalities.

Moreover, another future extension could involve the integration of emotional commonsense knowledge and affective reasoning into the fusion pipeline. By allowing DIFERNet to reason about emotional causes and consequences, the model may achieve better generalization on more complex affective understanding tasks such as sarcasm detection, emotion cause identification, and context-sensitive sentiment analysis.

In summary, DIFERNet presents a principled and effective approach for modeling sentiment in the presence of unaligned multimodal input. It opens up promising avenues for future research that bridges pretraining, dynamic fusion, and symbolic emotion modeling in the realm of human-centric AI.