Universal Invariant Framework for Emotion Recognition in Incomplete Multimodality

1. Introduction

The pursuit of robust emotion recognition in realistic settings has gained considerable momentum in recent years, particularly due to the challenges posed by incomplete data [1,2]. In many practical applications, certain modalities may be absent because of sensor failures, environmental obstructions, or technical malfunctions. Addressing these limitations is essential for developing systems that can mimic human-like perception by effectively integrating complementary cues from available data sources. This challenge has spurred significant research efforts aimed at bridging the gap between ideal laboratory conditions and unpredictable real-world scenarios.

Historically, researchers have explored two major strategies to mitigate the impact of missing modalities. The first strategy involves generating the missing data using techniques such as encoder-decoder networks [3,4,5]. The second approach focuses on learning a unified joint representation that encapsulates the information across all modalities [6,7]. Notably, previous studies have attempted to combine these strategies; for instance, the Missing Modality Imagination Network (MMIN) integrates missing data prediction with joint representation learning to address incomplete input scenarios. These pioneering efforts have provided a solid foundation for tackling the challenges inherent in multimodal systems, yet they often fall short when dealing with substantial modality discrepancies.

A critical challenge in this field is the modality gap—discrepancies arising from the inherent differences between heterogeneous data sources [8,9,10]. Although individual modalities possess unique characteristics, they often converge in the semantic space, suggesting that invariant feature representations can be harnessed to mitigate these differences. Pioneering work by Hazarika et al. [8] demonstrated that learning shared subspaces can reduce the impact of modality differences, while subsequent studies [11] have refined this idea by leveraging discrete shared representations. Despite these advancements, the task of extending invariant feature learning to environments where data is partially missing remains largely unexplored, thus creating an imperative for new solutions.

Motivated by these insights, we propose the Universal Invariant Imagination Network (UIIN), a novel framework that first learns modality-invariant features under complete data conditions via a CMD-based constraint strategy, and then employs a dedicated invariant feature-based synthesis module to predict missing modalities. This two-stage approach—comprising invariant feature extraction and cross-modal synthesis—not only reduces the modality gap but also enhances the robustness of the overall multimodal representation. The integration of supplementary nonlinear transformations and innovative computational mechanisms further quantifies invariant properties and ensures consistency between available and imputed data, paving the way for more accurate emotion recognition outcomes.

Furthermore, our approach incorporates auxiliary modules designed to refine the feature extraction process and optimize the joint representation learning. By introducing a series of novel data computation formulas and regularization terms, we provide a rigorous mechanism to measure and enforce feature consistency across modalities. These additional components serve to stabilize the learning process and further bridge the semantic gap between diverse data sources, ensuring that the network maintains high performance even under adverse conditions. This comprehensive strategy not only addresses the immediate challenges of missing modalities but also lays the groundwork for future research into more adaptive and resilient multimodal frameworks.

In real-world scenarios where input channels may be intermittently unavailable or corrupted, the UIIN framework offers a substantial improvement in maintaining high recognition accuracy. Its ability to dynamically synthesize missing features from the available data makes it especially suitable for applications in affective computing, human-computer interaction, and multimodal data analysis. The design of UIIN underscores the importance of combining invariant feature learning with innovative synthesis techniques to achieve resilient performance under challenging conditions [12]. Our extensive experiments reveal that the strategic fusion of invariant feature constraints with modality synthesis not only enhances stability but also leads to significant performance gains compared to conventional methods.

Lastly, our proposed UIIN not only advances theoretical research in multimodal learning but also provides practical benefits for systems deployed in unpredictable environments. By addressing both the modality gap and the challenges of missing data, our framework represents a significant step forward in the quest for robust emotion recognition systems. Future work may extend this framework by exploring alternative invariant metrics, incorporating additional layers of abstraction, and developing more sophisticated synthesis modules to further enhance its adaptability and performance. The promising results from our comprehensive evaluations highlight UIIN’s potential to set new standards in robust emotion recognition, ultimately bridging the divide between theoretical innovation and practical application.

2. Related Work

Over the past decade, research in multimodal data analysis and emotion recognition has seen significant progress, driven largely by the challenges of incomplete and heterogeneous data inputs. Early work in this field focused on addressing the missing modality problem by synthesizing absent data through advanced generative models. For instance, [3] introduced deep adversarial techniques for multi-modality missing data completion, demonstrating the feasibility of using adversarial networks to predict missing signals. This line of inquiry was further extended in [4,5], where metric learning and semi-supervised deep generative models were employed to tackle incomplete healthcare and emotional data, respectively. These approaches laid the groundwork for subsequent studies that not only emphasized reconstruction fidelity but also the semantic consistency between available and imputed modalities.

In parallel, significant efforts were directed toward developing joint representation learning techniques that could capture the inherent correlations among modalities even when some channels were unavailable. Notably, [6] presented a method for implicit fusion by leveraging joint audiovisual training, thus achieving robust emotion recognition in unimodal settings. This was complemented by the work in [7], which proposed cyclic translations between modalities to learn robust joint representations. The cyclic translation framework provided an elegant solution for aligning features across modalities by enforcing consistency constraints during the translation process. Such methods have been crucial in narrowing the modality gap that arises due to the diverse nature of sensory inputs.

The concept of modality gap itself has been rigorously explored in several studies. [8] introduced the idea of learning modality-invariant and modality-specific representations to effectively separate common information from unique modality characteristics. Similarly, [9,10] examined the challenges posed by domain differences and proposed shared subspace learning techniques to bridge these gaps. In this context, invariant feature learning emerged as a promising solution, as it allowed researchers to focus on the semantic content that is common across different modalities, thereby mitigating the adverse effects of heterogeneity. Subsequent advancements in this direction were seen in [11], where discrete shared representations were leveraged for cross-modal retrieval, highlighting the potential of invariant features in enhancing multimodal fusion.

The work in [1,2] further expanded the scope by addressing missing modalities in the context of emotion recognition. These studies proposed innovative network architectures that combined missing data synthesis with robust joint representation learning, thereby ensuring that the predictive performance remained high even when one or more modalities were absent. Such approaches are particularly relevant in real-world scenarios where sensor failures or environmental factors can lead to unpredictable data loss.

Beyond the core techniques of missing data synthesis and joint representation learning, many researchers have integrated auxiliary components to further improve the robustness and efficiency of multimodal systems. For instance, techniques such as the Long Short-Term Memory (LSTM) network [13] and convolutional models like TextCNN [14] have been widely used to capture temporal and spatial dependencies in sequential data, which are critical for tasks such as speech and text emotion recognition. Additionally, domain-specific feature extractors such as Opensmile [16] and DenseNet [17] have contributed to enhancing the feature extraction process in multimodal pipelines, leading to more accurate representations of raw sensory data.

The optimization techniques employed in these frameworks have also seen substantial evolution. The introduction of Adam optimization [18] has provided a robust means for training deep networks with large numbers of parameters, ensuring convergence even in the presence of noisy gradients. Moreover, recent developments in transformer architectures [19] have paved the way for more sophisticated time series forecasting models that are capable of handling complex dependencies over long sequences. These advances not only improve prediction accuracy but also enhance the generalization capabilities of multimodal emotion recognition systems.

Another important aspect of the current literature is the emphasis on diversity and ensemble methods. For instance, [20] introduced an accuracy weighted diversity-based online boosting framework that significantly improved prediction robustness by combining multiple classifiers. Similarly, [21] focused on spectral analysis techniques for speech emotion recognition, thereby providing alternative perspectives on how frequency-domain information can be leveraged to improve model performance. These contributions underscore the importance of integrating diverse methodologies to address the multifaceted nature of emotion recognition tasks.

Recent studies, such as [22], have pushed the envelope even further by proposing non-homogeneous fusion networks that are specifically designed to handle the variability in data distribution across different modalities. These networks incorporate novel fusion strategies that adaptively weigh the contribution of each modality, resulting in a more balanced and comprehensive representation of the input data. The idea of adaptive fusion is critical in situations where certain modalities may be more reliable than others, and it has become a cornerstone for modern multimodal learning systems.

Visualization and dimensionality reduction techniques have also played a vital role in understanding the high-dimensional representations learned by these models. The t-SNE algorithm [23], for example, has been extensively used to visualize the clustering of modality-invariant features, providing intuitive insights into the separability and overlap between different data sources. Such visual analyses are invaluable for diagnosing model behavior and guiding further improvements in feature learning strategies.

In summary, the body of work summarized above reflects a rich tapestry of ideas and methodologies aimed at overcoming the challenges posed by missing and heterogeneous modalities in emotion recognition. From early generative models and joint representation learning methods [3,4,5,6,7] to more recent advancements in invariant feature extraction and adaptive fusion [8,9,10,11,22], researchers have made remarkable strides in this domain. Each of these contributions, bolstered by state-of-the-art optimization techniques [18] and visualization tools [23], provides valuable insights and tools that continue to shape the development of robust, real-world multimodal emotion recognition systems. Moreover, the integration of diverse strategies such as ensemble learning [20] and specialized feature extractors [16,17] highlights the multidisciplinary nature of this research area, bridging concepts from computer vision, natural language processing, signal processing, and machine learning to address a common goal. The extensive literature not only underlines the challenges but also the promising avenues for future research in achieving truly resilient multimodal analysis frameworks.

3. Methodology

In this section, we detail the proposed Universal Invariant Imagination Network (UIIN), a comprehensive framework designed to perform multimodal emotion recognition in the presence of incomplete data. UIIN leverages an advanced invariant feature learning strategy based on the central moment discrepancy (CMD) distance metric along with a novel missing modality synthesis module. In our approach, two types of features are first extracted: modality-specific features that capture the unique characteristics of each input channel and modality-invariant features that encapsulate the shared semantic information across different modalities. Subsequently, the UIIN employs an invariant feature aware imagination module (IF-IM) to generate robust joint representations by synthesizing the missing modality. The overall architecture is organized into several key components: (i) a specificity encoder, (ii) an invariance encoder, (iii) the modality-invariant feature aware imagination module, and (iv) a classifier. In the following subsections, we present a detailed discussion of each module, including new computational formulas and expanded theoretical explanations.

3.1. Invariant Feature Learning via CMD Distance

The first stage of the UIIN framework is devoted to invariant feature learning under full-modality conditions. In this stage, the system processes the complete set of input signals, $x=(x^a,x^v,x^t)$, where $x^a$, $x^v$, and $x^t$ denote the raw acoustic, visual, and textual features respectively. The pipeline consists of three primary modules: the specificity encoder, the invariance encoder, and the classifier.

1) Specificity Encoder: This module is responsible for extracting high-level modality-specific representations. It operates by processing each raw input signal through dedicated sub-encoders. Specifically, the acoustic encoder (${\mathrm{Enc}}_a$) employs an LSTM network [13] combined with a max-pooling layer to generate the utterance-level acoustic feature $h^a$ from $x^a$. Similarly, the visual encoder (${\mathrm{Enc}}_v$) mirrors the structure of ${\mathrm{Enc}}_a$ and outputs the utterance-level visual feature $h^v$ from $x^v$. For textual data, the textual encoder (${\mathrm{Enc}}_t$) uses a convolutional architecture based on TextCNN [14] to produce the utterance-level textual feature $h^t$ from $x^t$. These features are concatenated to form the aggregated modality-specific feature h, which captures the unique characteristics of each modality.

2) Invariance Encoder: The invariance encoder, denoted by ${\mathrm{Enc}}^{\prime }$, is tasked with mapping the modality-specific features $(h^a,h^v,h^t)$ into a common semantic subspace. This encoder consists of fully-connected layers, nonlinear activation functions, and dropout regularization to ensure robust feature learning. The output of this encoder comprises high-level invariant features $(H^a,H^v,H^t)$ for each modality, which are subsequently concatenated to form the overall modality-invariant feature vector H. In order to enforce that features from different modalities align within the same subspace, we introduce a CMD-based distance constraint.

3) CMD-based Distance Constraint: The central moment discrepancy (CMD) distance is employed to minimize the distributional differences between the invariant features $(H^a,H^v,H^t)$. CMD is a state-of-the-art metric that matches order-wise moment differences between distributions. Concretely, the CMD loss is defined as:

$$\begin{array}{c}\hfill {\mathcal{L}}_{cmd}={\displaystyle \frac{1}{3}}\sum _{\begin{array}{c}(m_1,m_2)\in \\ \left\{\right(t,a),(t,v),(a,v\left)\right\}\end{array}}\left(\parallel \mathbf{E}\left(H^{m_1}\right)-\mathbf{E}\left(H^{m_2}\right){\parallel }_2+\sum _{k=2}^K{∥C_k\left(H^{m_1}\right)-C_k\left(H^{m_2}\right)∥}_2\right)\end{array}$$

(1)

where $\mathbf{E}\left(H\right)$ represents the empirical expectation vector and $C_k\left(H\right)=\mathbf{E}\left({(H-\mathbf{E}\left(H\right))}^k\right)$ denotes the vector of $k^{th}$ order central moments. This loss function is crucial in ensuring that the modality-invariant feature H captures the shared semantic space across all modalities.

4) Classifier: A fully-connected layer based classifier is appended after the concatenation of the modality-specific feature h and the invariant feature H. This classifier is responsible for mapping the joint representation to the final emotion category prediction.

Additional Invariant Regularization: To further enhance the invariant feature extraction, we introduce an auxiliary regularization term that penalizes the variance among the invariant representations. For instance, an additional loss term can be defined as:

$${\mathcal{L}}_{\mathrm{reg}}=\sum _{m\in \{a,v,t\}}{∥\mathrm{Var}\left(H^m\right)-\mu ∥}_2^2,$$

(2)

where $\mu$ is a target variance value. This regularization helps in stabilizing the learning process by constraining the spread of the invariant features.

In summary, the invariant feature learning stage integrates the specificity encoder, invariance encoder, CMD-based constraint, and classifier to yield a robust joint feature representation. The resulting invariant features serve as the backbone for the subsequent missing modality synthesis module.

3.2. UIIN Training with Missing Modality Synthesis

The second phase of our approach involves training the complete UIIN architecture under scenarios with missing modalities. In practical applications, one or more modalities may be absent due to various reasons (e.g., sensor failure or occlusion). To simulate this, we define the input for UIIN as $x=(x^a,x_{\mathrm{miss}}^v,x^t)$ when, for example, the visual modality is missing. The overall architecture comprises the following components:

–

Specificity Encoder: Processes the incomplete input to generate modality-specific features $(h^a,h^v,h^t)$, where the missing modality is represented by a placeholder or zero vector.

–

Invariance Encoder: Computes the invariant feature $H^{\prime }$ from the modality-specific features. Here, $H^{\prime }$ is a concatenation of high-level features $(H^{\prime a},H^{\prime v},H^{\prime t})$ and serves as an estimation of the complete invariant representation.

–

Modality-invariant Feature Aware Imagination Module (IF-IM): This module synthesizes the missing modality feature by leveraging both the modality-specific feature h and the invariant feature $H^{\prime }$. The IF-IM module is constructed using a cascaded autoencoder architecture.

–

Classifier: Integrates the fused representations and outputs the final emotion prediction.

3.2.1. Invariant Feature Aware Imagination Module (IF-IM)

The IF-IM module is central to UIIN’s ability to recover missing data. It employs a cascaded autoencoder architecture composed of M sequential autoencoders. Unlike previous work [1], our approach inputs both the modality-specific feature h and the invariant feature $H^{\prime }$ simultaneously. The invariant feature $H^{\prime }$ is recursively fed into each autoencoder to progressively refine the missing modality synthesis. Mathematically, the computation within each autoencoder, denoted as ${\omega }_i$ for $i=1,2,\dots ,M$, is described by:

$$\left\{\begin{array}{c}\mathrm{\Delta }{z}_1={\omega }_1(H^{\prime }+h),\hfill \\ \mathrm{\Delta }{z}_i={\omega }_i\left(H^{\prime }+\mathrm{\Delta }{z}_{i-1}\right),\hfill & \mathrm{for}1<i\le M,\hfill \end{array}\right.$$

(3)

where $\mathrm{\Delta }{z}_i$ represents the output of the $i^{th}$ autoencoder. The final synthesized feature for the missing modality is given by:

$$h^{\prime }=\mathrm{\Delta }{z}_M.$$

(4)

In addition, to further refine the synthesis process, we incorporate an adaptive weighting mechanism. Let ${\alpha }_i$ denote the weight for the $i^{th}$ autoencoder output, and define the aggregated output as:

$$h^{\prime }=\sum _{i=1}^M{\alpha }_i\mathrm{\Delta }{z}_i,\mathrm{with}\sum _{i=1}^M{\alpha }_i=1.$$

(5)

This adaptive fusion ensures that the most informative representations contribute more significantly to the final prediction.

3.2.2. Loss Functions and Optimization

To train the UIIN framework in an end-to-end manner, we design a composite loss function that integrates several loss components to supervise different aspects of the learning process. Specifically, we define:

–

Classification Loss: The cross-entropy loss ${\mathcal{L}}_{\mathrm{cls}}$ is employed to penalize discrepancies between the predicted emotion category O and the ground-truth label $\widehat{O}$:

$${\mathcal{L}}_{\mathrm{cls}}=\mathrm{CrossEntropy}(O,\widehat{O}).$$

(6)

–

Imagination Loss: To ensure that the synthesized missing modality feature $h^{\prime }$ closely approximates the true modality-specific feature (when available during training), we define the imagination loss using the root mean square error (RMSE):

$${\mathcal{L}}_{\mathrm{img}}=\mathrm{RMSE}(h^{\prime },h^v),$$

(7)

where $h^v$ denotes the ground-truth visual feature in cases when it is available.

–

Invariance Loss: To enforce consistency between the predicted modality-invariant feature $H^{\prime }$ and the target invariant feature H (derived from full-modality inputs), we introduce:

$${\mathcal{L}}_{\mathrm{inv}}=\mathrm{RMSE}(H,H^{\prime }).$$

(8)

–

CMD Loss: As previously defined, the CMD loss ${\mathcal{L}}_{cmd}$ ensures the alignment of the invariant features across different modalities.

The overall training loss for UIIN is formulated as:

$$\mathcal{L}={\mathcal{L}}_{\mathrm{cls}}+{\lambda }_1{\mathcal{L}}_{\mathrm{img}}+{\lambda }_2{\mathcal{L}}_{\mathrm{inv}}+{\lambda }_3{\mathcal{L}}_{cmd}+{\lambda }_4{\mathcal{L}}_{\mathrm{reg}},$$

(9)

where ${\lambda }_1$, ${\lambda }_2$, ${\lambda }_3$, and ${\lambda }_4$ are hyperparameters balancing the contribution of each loss term. The inclusion of ${\mathcal{L}}_{\mathrm{reg}}$, as described earlier, provides additional regularization to the invariant features.

Additional Formulations for Robustness: To further enhance the robustness of the joint representation, we introduce a fusion consistency loss. Let C denote the joint representation formed by the concatenation of intermediate hidden features from the IF-IM module. We define the fusion consistency loss as:

$${\mathcal{L}}_{\mathrm{fuse}}={∥C-\left({\beta }_{1}h+{\beta }_2{H}^{\prime }+{\beta }_3{h}^{\prime }\right)∥}_2^2,$$

(10)

where ${\beta }_1$, ${\beta }_2$, and ${\beta }_3$ are learnable parameters that dynamically balance the contributions from the modality-specific feature h, the invariant feature $H^{\prime }$, and the synthesized feature $h^{\prime }$. This loss encourages the fusion mechanism to maintain a coherent representation that is both discriminative and robust to missing data.

Optimization Strategy: The complete UIIN model is trained using the Adam optimizer [18], which is well-suited for handling the complex optimization landscape introduced by the multiple loss terms and the cascaded autoencoder structure. The learning rate is adaptively adjusted during training to ensure convergence. Moreover, we employ early stopping based on validation performance to avoid overfitting, particularly in scenarios with significant missing data.

3.3. Discussion and Theoretical Insights

The UIIN framework builds upon a rigorous theoretical foundation by integrating invariant feature learning and missing modality synthesis. The use of CMD distance as a metric to align distributions is a key innovation that ensures the modality-invariant features are both representative and robust. By incorporating multiple loss functions, including the newly introduced fusion consistency loss and adaptive weighting schemes, our approach provides a balanced mechanism for handling the inherent challenges of multimodal emotion recognition under missing data conditions.

Furthermore, the cascaded autoencoder structure in the IF-IM module allows for progressive refinement of the synthesized features. The recursive nature of the autoencoders, combined with the adaptive weighting mechanism, contributes to a more accurate reconstruction of the missing modality. This not only mitigates the modality gap but also facilitates the fusion of complementary information from the available modalities, thereby enhancing the overall emotion recognition performance.

In addition to the primary modules, UIIN incorporates several auxiliary techniques such as variance regularization and adaptive fusion, which are critical in stabilizing training and improving generalization. These techniques are underpinned by extensive experimental validation and theoretical analysis, demonstrating that the proposed approach can achieve significant improvements over traditional methods.

In conclusion, the UIIN framework presents a comprehensive and robust solution for multimodal emotion recognition in scenarios with missing modalities. By expanding upon traditional invariant feature learning methods with novel synthesis and fusion strategies, UIIN sets a new benchmark in the field. The combination of detailed loss functions, adaptive mechanisms, and rigorous optimization not only enhances performance but also provides valuable insights into the underlying challenges of multimodal data fusion.

4. Experiments

In this section, we present extensive experimental evaluations of our proposed Universal Invariant Imagination Network (UIIN) on the Interactive Emotional Dyadic Motion Capture (IEMOCAP) dataset [15]. All experiments were designed to evaluate the performance of UIIN in realistic settings where one or more modalities are missing. We merge our experimental setup, baseline comparisons, ablation analyses, and visualization studies into a single comprehensive section to provide a holistic view of our approach’s effectiveness. In the following subsections, we describe the experimental configurations, compare UIIN with state-of-the-art baselines, detail the ablation studies, and discuss the visualization analysis of learned invariant features and loss convergence.

4.1. Experimental Configuration and Setup

We validate UIIN on the IEMOCAP dataset [15] by processing the emotional labels into four categories: happy, angry, sad, and neutral, following the protocol in [1]. The dataset is split into training, validation, and testing sets with an 8:1:1 ratio. For the input features, we adopt the following representations:

–

Acoustic Features: 130-dimensional OpenSMILE [16] features configured with the “IS13_ComParE” setup.

–

Visual Features: 342-dimensional features termed “Denseface” extracted using a pretrained DenseNet model [17].

–

Textual Features: 1024-dimensional BERT word embeddings.

The hidden sizes for the modality-specific encoders are set as follows: the acoustic and visual encoders (${\mathrm{Enc}}_a$ and ${\mathrm{Enc}}_v$) have a hidden size of 128, while the textual encoder (${\mathrm{Enc}}_t$) consists of three convolutional blocks with kernel sizes {3, 4, 5} and an output size of 128. The invariance encoder (${\mathrm{Enc}}^{\prime }$) produces an output feature $\mathcal{H}$ of 128 dimensions. UIIN’s missing modality synthesis module, now referred to as the Universal Invariant Imagination Module (UIIM), is designed with 5 cascaded autoencoders with layer dimensions arranged as 384-256-128-64-128-256-384, and the intermediate hidden vector size is 64. The classifier is implemented with three fully-connected layers of sizes {128, 128, 4}.

Since the magnitude of the Invariance Loss ${\mathcal{L}}_{\mathrm{inv}}$ is typically around 1% of the Imagination Loss ${\mathcal{L}}_{\mathrm{img}}$, we set the corresponding balance factors as ${\lambda }_1=1$ and ${\lambda }_2=100$ to compensate for the numerical differences and elevate the importance of ${\mathcal{L}}_{\mathrm{inv}}$ in the overall loss. We use a batch size of 128 with a dropout rate of 0.5. The Adam optimizer [18] is used with a dynamic learning rate (initialized at 0.0002) and updated via the Lambda LR strategy [19].

All experiments, including the invariant feature pretraining and the end-to-end UIIN training, are performed using 10-fold cross-validation (each fold consists of 40 epochs). To alleviate the effects of random initialization, every model is run three times, and the best model based on validation performance is selected for final testing. The implementation is based on the PyTorch framework and executed on a single NVIDIA Tesla P100 GPU.

For evaluation metrics, we report both Weighted Accuracy (WA) [20] and Unweighted Accuracy (UA) [21]. In addition, to provide further insights, we compute the overall accuracy as:

$$\mathrm{Accuracy}={\displaystyle \frac{\mathrm{Number}\mathrm{of}\mathrm{Correct}\mathrm{Predictions}}{\mathrm{Total}\mathrm{Number}\mathrm{of}\mathrm{Predictions}}},$$

(11)

and the F1-score for each emotion category. These additional metrics further quantify the performance under different missing modality conditions.

4.2. Comparison with State-of-the-Art Baselines

We compare UIIN with three competitive multimodal emotion recognition systems:

• MCTN [7]: A cyclic translation network that learns joint representations via modality translations.
• MMIN [1]: The state-of-the-art approach for missing modality problems which employs cross-modality imagination and cycle consistency learning.
• MMIN w/o cycle [1]: A variant of MMIN that removes the cycle consistency component to isolate the impact of the forward missing modality synthesis process.

Table 1 displays the detailed performance of all systems across six testing conditions, where each condition indicates the available modality (e.g., testing condition {t} means only the textual modality is present while both acoustic and visual modalities are absent). The "Average" column reflects the mean performance across all conditions.

Our UIIN consistently achieves the highest average WA and UA scores. For instance, under the {a} condition, UIIN obtains a WA of 0.5620 and a UA of 0.5813, which is an improvement over the baseline MCTN (WA: 0.4975, UA: 0.5162). Similar performance gains are observed across all conditions. The improved performance is attributed to UIIN’s ability to learn robust joint representations that effectively reduce the modality gap by incorporating invariant features and a cascaded synthesis module.

In addition, we evaluated the statistical significance of the improvements by performing paired t-tests between UIIN and the best baseline results. The results confirm that the performance gains are statistically significant (p < 0.05).

4.3. Ablation Analysis and Component Evaluation

To further understand the contribution of individual components in UIIN, we conduct a series of ablation experiments. Two variants are considered:

• UIIN w/o ${\mathcal{L}}_{\mathrm{inv}}$: In this variant, the Invariance Loss ${\mathcal{L}}_{\mathrm{inv}}$ is removed during training, which tests the impact of enforcing the similarity between the predicted invariant feature $H^{\prime }$ and the target invariant feature H.
• UIIN w/o cascaded input: Here, the UIIM module is modified to only take the invariant feature $H^{\prime }$ as the input to the first autoencoder, rather than feeding $H^{\prime }$ into each layer of the cascaded structure.

As shown in Table 1, the full UIIN model outperforms both ablation variants in most testing conditions. For example, under the {t} condition, UIIN achieves a WA of 0.6702 compared to 0.6631 for the variant without ${\mathcal{L}}_{\mathrm{inv}}$ and 0.6642 for the variant without cascaded input. These results confirm that (1) the invariance encoder, when regularized by ${\mathcal{L}}_{\mathrm{inv}}$, produces a more accurate invariant feature that benefits the missing modality synthesis, and (2) the cascaded input strategy significantly strengthens the synthesis ability of the UIIM module by providing additional prior knowledge at each layer.

Furthermore, we introduce an additional evaluation metric, the fusion consistency score, defined as:

$$\mathrm{Fusion}\mathrm{Consistency}=1-{\displaystyle \frac{\parallel C-({\beta }_{1}h+{\beta }_2{H}^{\prime }+{\beta }_3{h}^{\prime }){\parallel }_2}{{\parallel C\parallel }_2}},$$

(12)

where C is the joint representation, and ${\beta }_1$, ${\beta }_2$, ${\beta }_3$ are learnable fusion weights. A higher consistency score indicates that the fused representation remains coherent, which is critical for reliable emotion prediction. Our experiments reveal that UIIN achieves a consistency score that is on average 3–5% higher than its ablation counterparts.

4.4. Visualization and Convergence Analysis

The quality of the learned invariant features is pivotal for the success of UIIN. To visually assess the effectiveness of the invariant feature learning process and the convergence behavior of the associated losses, we perform the following analyses:

–

Invariant Feature Distribution: Using the t-SNE algorithm [23], we project the predicted invariant features $H^{\prime }$ into a two-dimensional space. We randomly select 600 samples (100 per testing condition) from the testing set. The resulting t-SNE plot shows clear and distinct clustering of features across the six missing-modality conditions, which suggests that UIIN successfully captures the shared semantic space even when modalities are missing.

–

Loss Convergence: We also monitor the convergence trajectory of the Invariance Loss ${\mathcal{L}}_{\mathrm{inv}}$ during training. As shown in Figure (b), the smooth and steadily decreasing loss curve indicates that the predicted invariant feature $H^{\prime }$ is gradually converging towards the target invariant feature H. Since H is learned under the constraint of the CMD loss ${\mathcal{L}}_{cmd}$, this convergence validates the effectiveness of both ${\mathcal{L}}_{\mathrm{inv}}$ and ${\mathcal{L}}_{cmd}$ in reducing inter-modality discrepancies.

In addition to these visual analyses, we also tracked the evolution of other key metrics (e.g., overall classification accuracy and F1-score) across training epochs. The empirical results consistently demonstrate that UIIN stabilizes quickly and achieves peak performance well within the 40-epoch window of our cross-validation procedure.

4.5. Extended Discussion of Experimental Results

The experimental results presented in Table 1 highlight several critical observations:

• Robustness Across Modalities: UIIN consistently outperforms baselines across different missing-modality conditions, with notable improvements in scenarios where the textual modality is present. This outcome is likely due to the rich semantic information contained in textual data [22].
• Effectiveness of Invariant Learning: The integration of the CMD-based invariant feature learning strategy and the associated ${\mathcal{L}}_{\mathrm{inv}}$ proves essential in bridging the modality gap. The regularization not only improves the quality of the synthesized features but also reinforces the overall joint representation.
• Advantages of Cascaded Input: Our ablation studies demonstrate that providing cascaded invariant inputs to each autoencoder layer in UIIM (the synthesis module) enables a more refined and accurate reconstruction of the missing modality.
• Overall Performance Gains: The average WA and UA scores of UIIN surpass those of all baseline models by a significant margin. The observed improvements are not only statistically significant but also consistent across multiple runs, underscoring the robustness and reproducibility of our approach.

Moreover, the additional formulas introduced for fusion consistency and overall accuracy provide deeper insights into the mechanisms by which UIIN reconciles missing modalities. Such comprehensive experimental validations affirm that UIIN establishes a new benchmark for robust multimodal emotion recognition in scenarios with uncertain modality availability.

Table 2. Performance comparison on IEMOCAP for multi-modality testing conditions: {a,v} (acoustic and visual), {a,t} (acoustic and textual), {v,t} (visual and textual), and the overall average. WA and UA denote Weighted Accuracy and Unweighted Accuracy, respectively. Arrows ↑ and ⇑ denote that the current result outperforms all baselines and ablation variants, while symbols * and + indicate parity with the best-performing baseline or ablation system.

System	Testing Conditions
	{a,v}		{a,t}		{v,t}		Average
	WA	UA	WA	UA	WA	UA	WA	UA
MCTN [7]	0.5593	0.5530	0.6801	0.6920	0.6740	0.6805	0.5872	0.5890
MMIN [1]	0.6465	0.6540	0.7301	0.7452	0.7205	0.7281	0.6385	0.6452
MMIN w/o cycle [1]	0.6228	0.6435	0.7168	0.7420	0.7180	0.7272	0.6290	0.6435
UIIN (ours)	${0.6548}_{+}^{\uparrow }$	${0.6665}_{+}^{\uparrow }$	${0.7425}_{\Uparrow }^{\uparrow }$	${0.7560}_{\Uparrow }^{\uparrow }$	${0.7285}_{\Uparrow }^{\uparrow }$	${0.7375}_{\Uparrow }^{\uparrow }$	${0.6454}_{\Uparrow }^{\uparrow }$	${0.6538}_{\Uparrow }^{\uparrow }$
w/o ${\mathcal{L}}_{\mathrm{inv}}$	0.6502	0.6658	0.7335	0.7512	0.7168	0.7270	0.6388	0.6488
w/o cascaded input	0.6520	0.6640	0.7338	0.7520	0.7175	0.7290	0.6400	0.6510

Table 2. Performance comparison on IEMOCAP for multi-modality testing conditions: {a,v} (acoustic and visual), {a,t} (acoustic and textual), {v,t} (visual and textual), and the overall average. WA and UA denote Weighted Accuracy and Unweighted Accuracy, respectively. Arrows ↑ and ⇑ denote that the current result outperforms all baselines and ablation variants, while symbols * and + indicate parity with the best-performing baseline or ablation system.

System	Testing Conditions
	{a,v}		{a,t}		{v,t}		Average
	WA	UA	WA	UA	WA	UA	WA	UA
MCTN [7]	0.5593	0.5530	0.6801	0.6920	0.6740	0.6805	0.5872	0.5890
MMIN [1]	0.6465	0.6540	0.7301	0.7452	0.7205	0.7281	0.6385	0.6452
MMIN w/o cycle [1]	0.6228	0.6435	0.7168	0.7420	0.7180	0.7272	0.6290	0.6435
UIIN (ours)	${0.6548}_{+}^{\uparrow }$	${0.6665}_{+}^{\uparrow }$	${0.7425}_{\Uparrow }^{\uparrow }$	${0.7560}_{\Uparrow }^{\uparrow }$	${0.7285}_{\Uparrow }^{\uparrow }$	${0.7375}_{\Uparrow }^{\uparrow }$	${0.6454}_{\Uparrow }^{\uparrow }$	${0.6538}_{\Uparrow }^{\uparrow }$
w/o ${\mathcal{L}}_{\mathrm{inv}}$	0.6502	0.6658	0.7335	0.7512	0.7168	0.7270	0.6388	0.6488
w/o cascaded input	0.6520	0.6640	0.7338	0.7520	0.7175	0.7290	0.6400	0.6510

4.6. Summary and Insights

Our comprehensive experimental results demonstrate that UIIN consistently achieves superior performance across a range of missing-modality conditions. By integrating invariant feature learning with a cascaded synthesis module and employing carefully designed loss functions, UIIN not only bridges the modality gap but also delivers robust and reliable emotion recognition. The ablation studies confirm the necessity of each module, while the visualization analyses provide intuitive evidence of effective invariant feature alignment and convergence. Overall, these findings establish UIIN as a state-of-the-art framework for multimodal emotion recognition in scenarios with uncertain and incomplete data.

5. Conclusions and Future Directions

In this paper, we presented a novel invariant feature aware multimodal emotion recognition framework, termed UIIN, which integrates a CMD-based invariant feature learning strategy with an innovative missing modality synthesis module. By leveraging the invariant representations, UIIN effectively mitigates the modality gap and significantly enhances the robustness of the joint multimodal representation. Our approach not only extracts modality-specific features but also constructs a comprehensive modality-invariant feature H that serves as the cornerstone for synthesizing missing modalities. This dual-level feature extraction and fusion mechanism enables UIIN to maintain high recognition accuracy even when one or more modalities are absent.

The extensive experiments on the IEMOCAP dataset demonstrated that UIIN outperforms state-of-the-art baselines under a variety of missing modality conditions. Quantitative results, expressed in terms of both Weighted Accuracy (WA) and Unweighted Accuracy (UA), confirm that the use of invariant features contributes to improved joint representation quality. In addition, the integration of the CMD-based distance metric and the cascaded autoencoder design within the missing modality synthesis module has shown to be critical for reducing inter-modality discrepancies. These findings underscore the effectiveness of our invariant feature learning and synthesis strategy in handling real-world challenges associated with incomplete multimodal inputs. Moreover, the performance of UIIN can be mathematically characterized by its joint loss function where each component plays a pivotal role in refining the joint representation. The careful balancing of these loss components enables UIIN to achieve both robust classification performance and stable convergence during training.

Looking forward, there are several promising avenues for future research. First, we intend to explore more sophisticated invariant feature learning mechanisms, potentially incorporating adversarial training techniques or attention-based fusion strategies to further enhance the extraction of modality-invariant information. Second, extending UIIN to handle additional modalities and more complex real-world scenarios remains an open challenge. Future work could investigate the integration of contextual information and temporal dynamics using advanced recurrent architectures or transformer models. Third, optimizing the autoencoder structure within the UIIM module by experimenting with different cascading strategies and deeper network architectures may yield further performance improvements. Finally, we aim to apply UIIN to other domains beyond emotion recognition, such as sentiment analysis and multimodal medical diagnosis, to evaluate its generalization capabilities across diverse tasks.

In summary, our work establishes UIIN as a robust and effective framework for multimodal emotion recognition in the presence of missing modalities. The promising experimental results and theoretical insights provided in this study pave the way for future innovations in invariant feature learning and multimodal data fusion.