Multimodal Fusion Network for Multimodal Sentiment Analysis

1. Introduction

The explosion of multimedia content and rapid progress in machine learning algorithms have jointly propelled multimodal applications into the forefront of AI research. In today’s data-rich environment, the challenge lies not only in achieving state-of-the-art performance but also in maintaining a balance between model complexity and computational efficiency—a critical factor for real-world deployments.

Early deep learning efforts in multimodal sentiment analysis predominantly employed Recurrent Neural Networks (RNNs) [13,14,15] and Convolutional Neural Networks (CNNs) [16] to capture contextual and spatial features. These architectures laid the groundwork for later innovations, such as attention mechanisms [17,18], which enabled more refined and context-aware fusion of disparate data sources.

The transformer model [19] ushered in a paradigm shift by effectively modeling long-range dependencies across modalities. Pretrained models like BERT [57] and GPT [1] exemplified the power of large-scale pre-training followed by task-specific fine-tuning, a strategy that has been extended to multimodal contexts by works such as VilBERT [2]. Researchers have since explored the adaptation of language-only pretrained models for multimodal applications [3,4,5,6], thereby reducing the reliance on extensive modality-specific pre-training.

However, the conventional fine-tuning paradigm poses two significant challenges. First, it demands high computational and data resources, and second, it risks catastrophic forgetting [7]—a phenomenon where the model loses previously learned general knowledge when adjusted to specific tasks. To mitigate these issues, strategies such as prompt tuning [8] and its trainable variant, soft prompting [9,10], have been proposed. Additionally, the introduction of adapter modules by Houlsby et al. [11] demonstrated that small, down-projected feedforward networks can be interleaved within a frozen model to refine its representations without incurring the full cost of fine-tuning.

Building on these concepts, methods like Frozen [3] and MAGMA [4] have integrated adapter layers within frozen language models, showing promising results in multimodal tasks. Moreover, innovations such as Flamingo [12] have leveraged flexible visual encoders to transform arbitrary image sequences into a fixed set of tokens, thereby broadening the applicability of these methods.

In our work, we extend these ideas by incorporating audio data into the multimodal fusion framework, addressing a gap in previous research. The proposed EMFN is designed to integrate audio, visual, and textual inputs through a series of adapter and fusion modules that are carefully calibrated to maintain the strengths of the underlying pretrained language model while extracting complementary information from additional modalities. This formulation highlights the residual connection and non-linear processing that refine the original feature representation $\mathbf{h}$. Furthermore, the fusion process across layers is modeled to emphasize the most informative features from each modality at layer l. An auxiliary gradient update denotes the loss function computed over the fused representation and the target sentiment label.

Our extensive investigations on the CMU-MOSEI dataset demonstrate that EMFN not only surpasses fine-tuned counterparts in performance but also exhibits enhanced robustness when confronted with noisy inputs. By significantly reducing the trainable parameter count, our approach ensures efficient adaptation, paving the way for scalable and real-world multimodal applications.

In summary, the primary contributions of this work are:

• The proposal of EMFN, an innovative architecture that integrates textual, audio, and visual modalities through efficient adapter and fusion mechanisms.
• The development of a detailed theoretical framework that includes new formulations for adapter transformations and dynamic multimodal fusion.
• Comprehensive empirical evaluations on the CMU-MOSEI dataset, establishing that EMFN attains superior performance and robustness relative to current state-of-the-art methods.

2. Related Work

In the early stages of multimodal research, deep learning techniques predominantly relied on sequential models such as Recurrent Neural Networks (RNNs) [13,14,15] and spatial feature extractors like Convolutional Neural Networks (CNNs) [16] to capture temporal and local contextual cues from diverse data sources. These pioneering approaches laid the groundwork for understanding how different modalities could be processed individually before being combined for complex tasks such as sentiment analysis.

Subsequent advancements in the field introduced attention mechanisms, which significantly enhanced the capacity to integrate information across modalities. The advent of the transformer architecture [19] revolutionized the manner in which long-range dependencies and contextual relationships were modeled. This innovation paved the way for influential language models like BERT [57] and GPT [1], which further underscored the importance of pretraining on vast datasets prior to task-specific adaptation.

Building on the success of these language models, researchers extended pretraining paradigms to multimodal domains. For instance, VilBERT [2] was among the first to jointly model textual and visual information, demonstrating that leveraging cross-modal data could significantly enhance representational learning. This concept was later enriched by works such as [3,4,5,6], which further integrated additional modalities, including audio, to better capture the nuances of human sentiment.

While conventional fine-tuning of pretrained models has shown considerable promise, it is often accompanied by challenges such as catastrophic forgetting [7]. To mitigate these issues, alternative strategies like prompt tuning [8] and its enhanced, trainable variant—soft prompting [9,10]—were proposed. In parallel, the introduction of adapter modules [11] provided an elegant solution by allowing task-specific adjustments without altering the core pretrained weights, thereby preserving general knowledge while efficiently adapting to new tasks.

Recent research has further refined the integration of multimodal information by developing hybrid approaches that combine frozen language representations with lightweight fusion techniques. Notable examples include methods such as Frozen [3] and MAGMA [4], which strategically intersperse adapter layers within a fixed language model to incorporate complementary signals from audio and visual streams. The design philosophy behind Flamingo [12] further underscores this trend by advocating for flexible encoders that dynamically convert visual inputs into fixed-length token sequences for seamless integration.

Despite the considerable progress achieved by these methodologies, a central challenge remains in balancing model efficiency, robustness, and high performance in multimodal sentiment analysis. The diverse strategies discussed above offer valuable insights into addressing this trade-off; however, each approach brings its own set of compromises. Our proposed Enhanced Multimodal Fusion Network (EMFN) seeks to synthesize the strengths of these prior works by integrating efficient adapter mechanisms with dynamic, cross-modal fusion strategies, thereby advancing the state-of-the-art in parameter-efficient multimodal learning.

3. Proposed Method

In this work, we propose the Enhanced Multimodal Fusion Network (EMFN) as a novel framework for multimodal sentiment analysis. The design of EMFN is centered around the idea of combining the strengths of a frozen, pretrained language model with specialized modules that adapt and fuse auxiliary modalities such as visual and audio data. In our approach, the textual inputs are processed by a frozen BERT model, while parallel encoder pipelines extract complementary features from the visual and audio streams. These representations are then integrated through a series of adapter and fusion layers in a layer-wise manner, culminating in a final predictor that generates sentiment scores. Our framework is engineered to maintain the integrity of the pre-trained language representations while efficiently injecting task-specific, multimodal information.

3.1. Frozen Language Backbone

At the core of EMFN lies a frozen, pre-trained BERT model which processes the input text without undergoing any parameter updates during training. This design choice is critical to avoid catastrophic forgetting [7] and to preserve the rich linguistic knowledge embedded in BERT’s 12 layers and its associated tokenizer. Let ${\mathbf{h}}^l\in {\mathbb{R}}^d$ denote the hidden state produced by the lth layer of BERT. The frozen backbone ensures that the original distribution of language features remains intact. To elaborate, if ${\mathcal{F}}_{\mathrm{BERT}}(·)$ denotes the mapping implemented by the frozen BERT, then for a given input text sequence T, we have:

$$\mathbf{H}={\mathcal{F}}_{\mathrm{BERT}}\left(T\right),$$

where $\mathbf{H}=\{{\mathbf{h}}^1,{\mathbf{h}}^2,\dots ,{\mathbf{h}}^{12}\}$. This preservation of BERT’s original feature space is fundamental to our methodology, as it provides a stable basis for subsequent multimodal fusion.

3.2. Adapter Modules

Interleaved with the frozen BERT layers, our framework incorporates adapter modules that facilitate the injection of task-specific adjustments while keeping the pretrained parameters untouched. Each adapter is designed as a bottleneck network with a down-projection followed by a ReLU activation and an up-projection back to the original dimension. Specifically, for a given hidden state ${\mathbf{h}}^l$, the adapter operation is defined as:

$${\mathbf{h}}_{\mathrm{adapter}}^l={\mathbf{h}}^l+W^{\left(up\right)}·\sigma \left(W^{\left(down\right)}·{\mathbf{h}}^l+b^{\left(down\right)}\right)+b^{\left(up\right)},$$

where $W^{\left(down\right)}\in {\mathbb{R}}^{d^{\prime }\times d}$, $W^{\left(up\right)}\in {\mathbb{R}}^{d\times d^{\prime }}$, $b^{\left(down\right)}$ and $b^{\left(up\right)}$ are bias terms, and $\sigma (·)$ denotes the ReLU function. In contrast to previous implementations, we insert these adapter modules after the feedforward layer normalization steps as suggested by [25], which effectively halves the number of additional parameters while still enabling robust adaptation. This strategy allows the EMFN to efficiently incorporate modality-specific nuances without distorting the linguistic representations learned during pre-training.

3.3. Visual and Audio Encoding

Parallel to the text processing stream, EMFN employs dedicated transformer-based encoders to process visual and audio inputs independently. The visual encoder accepts a sequence of visual features $\mathbf{v}=\{{\mathbf{v}}_1,{\mathbf{v}}_2,\dots ,{\mathbf{v}}_n\}$, while the audio encoder processes audio features $\mathbf{a}=\{{\mathbf{a}}_1,{\mathbf{a}}_2,\dots ,{\mathbf{a}}_m\}$. Each encoder leverages self-attention mechanisms to capture contextual relationships and salient patterns from the respective modality. The output from these encoders is a compact token representation that encapsulates the essence of both visual and acoustic information. Formally, if $f_{\mathrm{vis}}(·)$ and $f_{\mathrm{aud}}(·)$ denote the mapping functions for the visual and audio encoders, respectively, then we compute:

$${\mathbf{t}}_{\mathrm{vis}}=f_{\mathrm{vis}}\left(\mathbf{v}\right),{\mathbf{t}}_{\mathrm{aud}}=f_{\mathrm{aud}}\left(\mathbf{a}\right).$$

These tokens are subsequently merged to form a unified modality token:

$${\mathbf{t}}_{\mathrm{va}}=\varphi \left([{\mathbf{t}}_{\mathrm{vis}};{\mathbf{t}}_{\mathrm{aud}}]\right),$$

where $[·;·]$ denotes concatenation and $\varphi (·)$ is a learnable transformation, typically implemented as a linear projection or a small feedforward network. This fusion of visual and audio signals ensures compatibility with the BERT representations for subsequent multimodal integration.

3.4. Layer-wise Multimodal Fusion

A central innovation in EMFN is the layer-wise fusion strategy that interleaves the adapter-enhanced BERT layers with a FeedForward Network Fusion (FFN-Fusion) module. At each layer l, the model combines the CLS token, ${\mathbf{c}}_{\mathrm{text}}^l$, extracted from the BERT output, with the modality token ${\mathbf{t}}_{\mathrm{va}}$. The concatenated vector is then processed by a fusion network defined as:

$${\mathbf{z}}^l={\mathrm{FFN}}_{\mathrm{fusion}}\left([{\mathbf{c}}_{\mathrm{text}}^l;{\mathbf{t}}_{\mathrm{va}}]\right),$$

where the fusion network itself is structured as:

$${\mathbf{z}}^l=W_f·\mathrm{ReLU}\left(W_e[{\mathbf{c}}_{\mathrm{text}}^l;{\mathbf{t}}_{\mathrm{va}}]+b_e\right)+b_f.$$

Here, $W_e$, $W_f$, $b_e$, and $b_f$ are learnable parameters. This layer-wise multimodal integration is repeated across all 12 layers, thereby progressively enriching the textual representations with auxiliary modality cues. In addition, to promote smooth integration and to control the magnitude of the fusion parameters, a regularization term is introduced:

$${\mathcal{L}}_{reg}=\lambda \sum _{l=1}^{12}{\parallel W_f^l\parallel }_2^2,$$

where $\lambda$ is a hyperparameter that controls the strength of regularization.

3.5. Prediction and Optimization

The final stage of EMFN involves aggregating the fused representations from the last layer and projecting them to the target sentiment space. Let ${\mathbf{z}}^{\mathrm{final}}$ denote the concatenated output from the last fusion stage. The prediction is computed as:

$$\widehat{y}=W_p·\mathrm{Dropout}\left({\mathbf{z}}^{\mathrm{final}}\right)+b_p,$$

where $W_p$ and $b_p$ are the parameters of the prediction layer. The network is trained end-to-end using the Mean Absolute Error (MAE) loss defined by:

$${\mathcal{L}}_{\mathrm{MAE}}={\displaystyle \frac{1}{N}}\sum _{i=1}^N\left|{\widehat{y}}_i-y_i\right|,$$

where N is the number of training samples, ${\widehat{y}}_i$ is the predicted sentiment for the ith sample, and $y_i$ is the corresponding ground truth. Additionally, the overall training objective is given by the sum of the MAE loss and the regularization term:

$$\mathcal{L}={\mathcal{L}}_{\mathrm{MAE}}+{\mathcal{L}}_{reg}.$$

This composite loss function ensures both accurate sentiment prediction and robust integration of multimodal features while preventing overfitting in the fusion layers.

Overall, the EMFN framework meticulously combines the robustness of a frozen BERT backbone with the flexibility of adapter modules and the dynamic integration of visual and audio cues. The introduction of multiple interleaved fusion stages, alongside additional regularization and optimization strategies, allows EMFN to leverage complementary information from disparate modalities. Through a series of mathematical formulations and carefully engineered network components, EMFN not only achieves efficient parameter usage but also attains high performance in multimodal sentiment analysis tasks, demonstrating its potential for deployment in real-world applications.

4. Experimental Setup

4.1. Data

The proposed model is evaluated on the CMU-MOSEI dataset [30], which comprises $23,454$ YouTube video clips containing reviews on movies and various topics. Each sample is manually annotated with a sentiment score that ranges from $-3$ (indicating strongly negative sentiment) to $+3$ (indicating strongly positive sentiment). In this dataset, text transcriptions are segmented into individual words, and visual FACET as well as acoustic COVAREP features are extracted and aligned to these word segments. Standard splits for training, development, and testing are provided, ensuring consistency in experimental evaluation.

In addition to the primary sentiment scores, the dataset offers temporal alignments that facilitate the study of evolving sentiment across the duration of a clip. These detailed annotations allow for both regression-based evaluations—such as Mean Absolute Error (MAE) and Pearson Correlation (Corr)—and classification-based evaluations, including seven-class accuracy (Acc-7), binary accuracy (Acc-2), and F1-score (F1). For further analysis, we also compute additional metrics like Root Mean Squared Error (RMSE) and the Concordance Correlation Coefficient (CCC):

$$\mathrm{RMSE}=\sqrt{{\displaystyle \frac{1}{N}}{\sum }_{i=1}^N{({\widehat{y}}_i-y_i)}^2},\mathrm{CCC}={\displaystyle \frac{2{\sigma }_{xy}}{{\sigma }_x^2+{\sigma }_y^2+{({\mu }_x-{\mu }_y)}^2}},$$

where ${\sigma }_{xy}$ is the covariance between predictions and ground truth, ${\sigma }_x^2$ and ${\sigma }_y^2$ are their variances, and ${\mu }_x$, ${\mu }_y$ denote their means.

4.2. Evaluation Metrics

For the regression task, we report MAE and Pearson Correlation between the model predictions and human annotations. In the classification setting, we employ seven-class accuracy (Acc-7), binary accuracy (Acc-2), and the F1-score (F1). These metrics are supplemented by RMSE and CCC, providing a comprehensive assessment of model performance over both continuous and discrete outputs.

4.3. Implementation Details

All experiments use the bert-base-uncased variant of BERT [57] as the language backbone. This model comprises 12 transformer layers with each token represented by a 768-dimensional embedding. The input text is tokenized using BERT’s standard procedure, with special tokens [CLS] and [SEP] added to the beginning and end of each sentence, respectively. The BERT backbone remains frozen throughout training to preserve the pretrained linguistic knowledge and to mitigate the risk of catastrophic forgetting [7].

For the visual and acoustic modalities, we employ transformer encoder modules that are randomly initialized. These modules consist of 2 layers with 1 attention head each, and are designed to produce a learnable [CLS] token that summarizes the modality’s information. Extensive hyperparameter tuning was conducted for the adapter layers, where hidden sizes were explored in the range [128, 768] and 384 was selected as optimal. Similarly, for the fusion layers, a hidden size of 220 was chosen after exploring values in the range [160, 820]. A dropout rate of $0.2$ is applied across all layers to prevent overfitting.

The Adam optimizer [31] is used for training, with an initial learning rate set to $5\times {10}^{-5}$. Early stopping is implemented with a patience of 10 epochs. Furthermore, gradient clipping is applied to stabilize training, ensuring that the gradient norm does not exceed a threshold (typically set to $5.0$). All experiments are performed on a single GTX 1080Ti NVIDIA GPU, with training times averaging around 20 minutes per run. The entire implementation is based on PyTorch, and mixed-precision training is adopted to improve computational efficiency and reduce memory usage.

5. Experiments and Results

5.1. Comparative Analysis with State-of-the-Art

We evaluate the performance of our proposed Enhanced Multimodal Fusion Network (EMFN) against several leading models in the literature. Table 1 summarizes the performance on CMU-MOSEI using multiple evaluation metrics. The compared models include MMLatch (G) [26] and MulT (G) [27], which employ GloVe embeddings [32], as well as LMF (B) [28], TFN (B) [29], MFM (B) [20], and ICCN (B) [22], which use frozen BERT embeddings. Additionally, MAG-BERT* (FT) [6] and MISA (FT) [5] represent fine-tuning approaches. Notably, EMFN achieves the best performance across all metrics while keeping the trainable parameter count minimal.

5.2. Ablation Studies

To thoroughly understand the contribution of each modality and the adaptation strategy, we conduct extensive ablation experiments. Table 2 reports the performance when selectively excluding modalities and comparing adapter-based adaptation against full fine-tuning. For instance, the configuration labeled “EMFN no-text”—which omits the textual modality—results in a dramatic decline in performance (Corr: $0.240$, Acc-7: $41.64\%$), highlighting the central role of text in sentiment analysis. Conversely, the “EMFN text-only” variant, which excludes the audio-visual inputs, experiences a moderate performance decrease (Corr: $0.760$, Acc-7: $52.81\%$), thereby demonstrating that multimodal fusion yields additional gains.

Moreover, a comparison between adapter-based methods and fine-tuning is drawn by evaluating a variant of MISA with adapters (“MISA-Adapters”) and a fine-tuned version of EMFN (“EMFN-FT”). The adapter-based approach (EMFN) not only achieves higher performance (Corr: $0.766$, Acc-7: $53.29\%$) but also does so with a significantly smaller number of trainable parameters compared to fine-tuning strategies. This indicates that fine-tuning may induce undesirable catastrophic forgetting, adversely affecting model performance.

5.3. Robustness to Input Noise

Robustness analysis is carried out by simulating noise in both textual and visual modalities. For the visual modality, we introduce multiplicative Gaussian noise to randomly selected elements of the feature sequence. For textual data, we simulate real-world errors by applying two types of perturbations: token deletion (where tokens are replaced with the [UNK] token) and token replacement (where tokens are substituted with random tokens from the vocabulary). If p denotes the probability of corruption, the perturbed token ${\tilde{t}}_i$ for the original token $t_i$ is defined as:

$${\tilde{t}}_i=\left\{\begin{array}{cc}\left[\mathrm{UNK}\right],\hfill & \mathrm{with}\mathrm{probability}p\left(\mathrm{deletion}\right);\hfill \\ \mathrm{random}\mathrm{token},\hfill & \mathrm{with}\mathrm{probability}p\left(\mathrm{replacement}\right);\hfill \\ t_i,\hfill & \mathrm{with}\mathrm{probability}(1-p).\hfill \end{array}\right.$$

For the visual modality, if $\mathbf{v}=\{v_1,v_2,\dots ,v_n\}$ represents the feature sequence, then each feature $v_i$ is corrupted as:

$${\tilde{v}}_i=v_i\times {ϵ}_i,{ϵ}_i\sim \mathcal{N}(1,{\sigma }^2),$$

where ${\sigma }^2$ controls the noise intensity. Experiments reveal that performance degradation becomes apparent when the corruption probability exceeds $5\%$ for textual inputs, with token replacement having a greater negative impact than deletion. For visual inputs, degradation is noticeable at noise levels above $10\%$. Notably, the adapter-based EMFN demonstrates superior robustness compared to its fine-tuned variant (EMFN-FT), maintaining higher performance even when up to $50\%$ of tokens are corrupted.

5.4. Additional Sensitivity Analyses

To further understand the contribution of each modality, we conduct sensitivity experiments by progressively masking one modality at a time during inference. We define a sensitivity metric $S_m$ for modality m as:

$$S_m={\displaystyle \frac{{\mathrm{Performance}}_{\mathrm{full}}-{\mathrm{Performance}}_{\mathrm{masked}m}}{{\mathrm{Performance}}_{\mathrm{full}}}},$$

where ${\mathrm{Performance}}_{\mathrm{full}}$ is the metric value when all modalities are active, and ${\mathrm{Performance}}_{\mathrm{masked}m}$ is the metric when modality m is excluded. Our findings show that the text modality exhibits the highest sensitivity ($S_{\mathrm{text}}\approx 0.18$), while the visual and acoustic modalities have lower sensitivities ($S_{\mathrm{vis}}\approx 0.07$, $S_{\mathrm{aud}}\approx 0.05$).

Additionally, we evaluate the gradient norms of the loss function with respect to each modality input to assess their relative importance. Let ${\nabla }_{{\mathbf{x}}_m}\mathcal{L}$ be the gradient of the loss $\mathcal{L}$ with respect to modality m. The normalized importance score $I_m$ is computed as:

$$I_m={\displaystyle \frac{\parallel {\nabla }_{{\mathbf{x}}_m}{\mathcal{L}\parallel }_2}{{\sum }_{m^{\prime }}{\parallel {\nabla }_{{\mathbf{x}}_{m^{\prime }}}\mathcal{L}\parallel }_2}}.$$

This analysis confirms that textual inputs receive higher gradient magnitudes, further validating the dominant role of text in the sentiment prediction task.

5.5. Discussion

The experimental results strongly indicate that EMFN achieves superior performance compared to existing multimodal sentiment analysis models while using a fraction of the trainable parameters. The comprehensive ablation studies demonstrate that while text is the primary modality, incorporating visual and acoustic cues yields significant performance enhancements. Furthermore, the robustness tests highlight that adapter-based methods confer a marked advantage in noisy environments, reducing the adverse effects of corrupted inputs. These findings collectively suggest that EMFN, with its efficient adapter modules and dynamic fusion strategy, offers a promising balance between performance, robustness, and parameter efficiency, making it well-suited for real-world multimodal applications.

6. Conclusions and Future Directions

In this work, we introduced the Enhanced Multimodal Fusion Network (EMFN), a novel and straightforward architecture that capitalizes on the power of a pre-trained BERT transformer encoder while skillfully avoiding the pitfalls of catastrophic forgetting and modality imbalance. Our approach is designed to preserve the rich, pre-trained linguistic knowledge while simultaneously incorporating valuable cues from non-dominant modalities such as vision and audio. By leveraging the strengths of frozen language models and coupling them with specialized adapter modules, EMFN effectively maintains robust language representations even when additional modalities are introduced.

The incorporation of adapter modules in our framework plays a critical role in reducing the number of trainable parameters, which in turn significantly lowers the computational cost and training time. This parameter-efficient strategy not only streamlines the model but also contributes to improved resilience against various types of noise, be it in textual or visual inputs. Our experimental findings indicate that this design choice enhances the overall stability and robustness of the system, ensuring that even in the presence of data perturbations, EMFN is capable of extracting and integrating meaningful information from all modalities.

Extensive evaluations have demonstrated that EMFN consistently outperforms existing state-of-the-art methods in multimodal sentiment analysis. The model’s ability to harness complementary information from less dominant modalities without compromising the integrity of the primary language features has proven to be a decisive factor in its superior performance. This success reinforces the idea that a balanced fusion of modalities, achieved through our carefully designed adapter and fusion layers, is essential for capturing the subtle nuances inherent in complex sentiment tasks.

Looking ahead, several promising avenues for future work have emerged. One key direction is to extend the application of EMFN beyond sentiment analysis to a broader range of tasks, including but not limited to text generation, image captioning, and multimodal machine translation. Investigating the integration of more sophisticated fusion strategies, such as dynamic weighting or context-aware shifting mechanisms, could further enhance the model’s performance and adaptability. Moreover, exploring the impact of additional modalities, such as sensor data or user interaction logs, may offer new insights into building even more comprehensive multimodal systems.

Another promising area for future research involves scaling up our experiments to larger and more diverse datasets, which would test the limits of EMFN’s generalizability and robustness in real-world scenarios. In parallel, there is potential for developing advanced optimization techniques tailored specifically for multimodal fusion networks, aiming to further reduce training time and improve convergence rates. Additionally, a deeper investigation into the interplay between different modalities and the effect of various noise types will be essential for refining the model and ensuring its effectiveness in a wide array of applications.

In summary, our work on EMFN represents a significant step toward the design of flexible, efficient, and robust multimodal models. We envision that the blueprint established by this framework will inspire further research and serve as a foundation for future developments in multimodal learning. By combining pre-trained unimodal encoders with innovative adapter and fusion strategies, EMFN paves the way for a new generation of models capable of delivering high performance while operating under realistic computational constraints. We remain optimistic that our approach will catalyze new ideas and drive advancements in the integration of diverse data sources for complex machine learning tasks.