Multi-Modal Data-Driven Sentiment Analysis in Online Public Opinion During Public Health Emergencies

1. Introduction

In recent years, public health emergencies (PHEs) have occurred frequently worldwide, such as the COVID-19 pandemic and Ebola virus outbreaks. During such events, social media platforms serve as central arenas where the public expresses emotions, disseminates information, and seeks support[1]. This data is inherently multimodal: text directly expresses viewpoints and emotions, while images contain rich contextual information and emotional cues, often reinforcing, supplementing, or even correcting the emotional tendencies conveyed by the text[2,3,4].

Therefore, accurate and real-time sentiment analysis of multimodal online public opinion during PHEs holds strategic value for relevant authorities to promptly grasp public sentiment trends, identify panic and rumors, assess the socio-psychological impact of intervention policies, and implement targeted risk communication and psychological support[5,6,7]. However, PHE public opinion data presents core challenges: social media generates massive online streaming features, yet only a sparse subset is truly relevant to sentiment analysis. Moreover, multimodal fusion must account for cross-modal interactions between text and images, rendering traditional sentiment analysis methods inadequate[8].

Current research primarily faces two major issues: ①Online Feature Selection: PHE public opinion data arrives continuously in a streaming manner, with the feature space dynamically changing. Traditional batch-mode feature selection methods cannot adapt to this streaming scenario, selecting sparse subset that is truly relevant to sentiment analysis from massive online streaming features. Although online feature selection research exists, it mostly focuses on single-modal text and lacks efficient online learning mechanisms for complex interdependencies among features. ②Cross-Modal Fusion Challenge: Simple feature concatenation or late decision fusion struggles to capture complex non-linear interactions and fine-grained semantic alignment between text and images. Despite advances in deep learning-based multimodal fusion, two issues remain particularly overlooked in the context of public health emergencies. One is modeling true bidirectional, multi-level interactions across modalities—not just coarse alignment—to extract complementary cues. The other is dynamic, context-aware fusion: the contribution of text versus image should not be fixed. Consider a post where the image shows a packed hospital corridor while the text merely says “a bit worried.” Here, the visual emotion clearly dominates, yet typical methods fail to adapt accordingly.

Addressing the aforementioned challenges, the study proposes an innovative solution with the following main contributions: Firstly, we propose the Online Divide-and-Conquer Markov Blanket Learning Algorithm. We extend Markov blanket theory to the streaming multimodal feature selection scenario. This algorithm can approximate the sparse subset that is truly relevant to sentiment analysis from massive online streaming features via incremental updates, while ensuring time and space efficiency. Secondly, we design the Cross-Modal Interactive Enhanced Fusion Network. We introduce a Two-Stage Cross-Modal Interactive Attention mechanism and an Adaptive Modal Weighting module, achieving deep semantic fusion from local to global and from static to dynamic. Combined with residual connections, we construct a powerful and stable core model for multimodal emotion recognition.

The remainder of this paper is structured as follows: Section 2 reviews related work. Section 3 formally defines the problem and introduces the overall framework. Section 4 details the principles and derivation of the Online Divide-and-Conquer Markov Blanket Learning Algorithm. Section 5 elaborates on the design details of the Cross-Modal Interactive Enhanced Fusion Network. Section 6 presents the experimental setup, result analysis, and discussion. Section 7 concludes the paper and outlines future research directions.

2. Related Work

2.1. Sentiment Analysis in Public Health Emergencies

As an important carrier of modern risk communication, the information interaction characteristics of social media are highly consistent with the effect of social psychological force field. That is, public sentiment generates resonant diffusion through online clusters, which can form an emergency psychological field with radiation effects. Therefore, analyzing the discussions and emotional expressions of social media netizens regarding public health emergencies (PHEs) can predict potential risks, help the government better understand public responses, take timely intervention measures, reduce negative impacts, and maintain social order. PHEs trigger widespread public discourse on social media, making sentiment analysis an essential tool for monitoring social risk perception [9].

Sentiment analysis comprises text sentiment analysis and image sentiment analysis. Early research focused on unimodal textual analysis. The mainstream text sentiment analysis methods can be divided into sentiment lexicon-based methods[10,11], machine learning-based methods[12], and deep learning-based methods[13,14]. Among them, deep learning-based methods perform better in sentiment analysis, including CNN[15], CNN-BiLSTM[16], and BiGRU[17]. Recent advances have further pushed the boundaries of text sentiment analysis. For instance, Duan et al.[18] proposed a multi-level knowledge enhanced model integrating sentiment lexicon, syntactic dependencies, and conceptual knowledge via GCN and interactive attention. Liu et al.[19] addressed fine-tuning inconsistency by multi-prompt learning with diverse templates and answer engineering, ensembling predictions from an autoregressive model. Zhang et al.[20] introduced SynPrompt, which incorporates syntactic knowledge into prompt tuning via syntax-enhanced templates and data augmentation for few-shot fine-grained sentiment analysis.

Compared with textual sentiment, the sentiment expressed through images is embodied by visuals, objects, or scenes. Li et al.[21] used an improved CNN model to automatically extract the sentiment information of images. Cai et al.[22] combined global features with local object features for image sentiment analysis. However, affective expression during PHEs is inherently multimodal, with images often complementing or even contradicting textual sentiments[23]. Recognizing this gap, multimodal sentiment analysis has emerged as a promising direction for a holistic understanding of public opinion [24,25,26,27]. Gan et al.[28] proposed a cross-modal interactive Transformer that uses image caption generation to recover missing textual sentiment, dependency parsing and GCN to capture long-distance dependencies, and a TOP-n adjective-noun pair strategy to enhance visual sentiment features, thereby improving multimodal sentiment analysis during PHEs.

Despite this progress, most existing PHE-related multimodal studies employ simplistic fusion strategies and are evaluated on static datasets. This leaves a critical gap in methodologies capable of handling the streaming, high-velocity, and dynamically evolving nature of real-world social media data during public health emergencies.

2.2. Online Feature Selection for Streaming Data

In many real-world applications, feature streams refer to scenarios in which, under a fixed number of samples, feature dimensions increase dynamically over time—i.e., features arrive one by one and must be processed immediately. Online streaming feature selection algorithms have been developed to handle such dynamically changing feature spaces. Representative examples include Grafting[29], Alpha-investing[30], and OSFS[31] for single-label problems; MLFSL[32], MUCO[33], and OM-NRS [34] for multi-label online streaming feature selection; and group-based algorithms such as GFSSF[35] and OGFS[36] for features that arrive in groups.

Most existing online streaming feature selection methods are based on statistical correlations—they attempt to model and classify data by exploiting statistical relationships among variables. However, statistical correlation is only a surface-level phenomenon; it reflects the goodness-of-fit of probability distributions rather than the true underlying relationships . The genuine intrinsic relationship among things is causality, which implies that changes in the frequency or characteristics of a cause lead to corresponding changes in the effect . Bayesian networks (BNs) and Markov blankets (MB) are among the most commonly used tools for causal discovery[37]. The MB of a target variable comprises its parents (direct causes), children (direct effects), and spouses (other parents of its children). The concept of MB aligns naturally with the requirements of feature selection: features belonging to the MB correspond to strongly relevant features; features not in the MB but having a path to MB members are weakly relevant; features without any path to MB members are irrelevant. Consequently, feature selection can be cast as the problem of discovering the MB of the class variable. The MB captures local causal relationships, so identifying the MB of the class variable yields a logically optimal solution[38].

To improve the accuracy of causal feature selection on the datasets where only a small number of features are relevant, divide-and-conquer algorithms were introduced. The Min-Max MB (MMMB) algorithm[39] first applied a divide-and-conquer strategy by separating PC discovery from spouse discovery, thereby reducing the conditioning set size and improving accuracy. Ling[40] introduced cross-deletion: false positives are removed before each iteration, preventing them from entering the judgment step and thereby ensuring efficiency. However, neither of these methods can handle a particular structural configuration; they cannot guarantee the discovery of the correct MB in a faithful BN. To address this issue, Pena et al. used symmetry checks to ensure correct MB discovery and proposed PCMB[41]; unfortunately, the symmetry check requires repeated PC discovery, which compromises efficiency. To improve the efficiency of PCMB, IPCMB[42] adopted an iterative search strategy that reduces one round of PC discovery and identifies spouses more quickly.

Nevertheless, all the above-mentioned divide-and-conquer algorithms require enumeration of conditional subsets. The enumeration process leads to exponential growth in computation, and the algorithms must also store separation sets for nodes independent of the target variable. Therefore, a divide-and-conquer causal feature selection algorithm with low time and space complexity is urgently needed. To fill this gap, this paper proposes an Online Divide-and-Conquer MB learning algorithm, termed O-DC, based on mutual information. By exploiting the differences in mutual information between the class variable’s MB and other variables, O-DC compares the mutual information of each newly arrived feature with that of the currently selected parents/children and spouses. In this way, the algorithm significantly reduces time complexity.

2.3. Multimodal Feature Fusion Strategies

Effective multimodal sentiment analysis relies on sophisticated fusion strategies to capture complex interactions between modalities. To model cross-modal interactions more explicitly, advanced attention mechanisms have been widely adopted[43,44,45]. Harish et al.[46] proposed an attention-based deep neural network and training method that performs multi-level fusion of speech, image, and text modalities. Majumder et al.[47] introduced a hierarchical feature fusion strategy for three-modality sentiment analysis. Sun et al.[48] shared information across all modalities simultaneously, constructing an attention tensor from features extracted from each modality to improve sentiment classification accuracy. To fully measure the varying importance and interactive influences among modalities, Tsai et al.[49] employed cross-modal attention to map features from one modality to another, and Yang et al.[50] adopted cross-modal BERT to model inter-modality interactions. Wang et al.[51] introduced dual-modal and triple-modal cross-modal context information, using attention mechanisms to filter redundant information before performing sentiment analysis based on the fused features. Chen et al.[52] leveraged cross-modal attention to enable more sufficient intra-modal and inter-modal information interaction, employed gated units to remove redundant information, and used self-attention to assign attention weights. Yu et al.[53] proposed a deep attention and two-stage fusion method for image-text sentiment contrastive learning, using deep cross-modal attention for modality interaction and designing a cross-modal gated fusion module that employs gating and attention to achieve two-stage feature fusion.

Another important technique to facilitate deep network training is the residual connection, which alleviates gradient vanishing and degradation problems. Residual connections have been widely adopted in large models such as Transformer[54] and BERT[55]. By treating the output as a linear superposition of the input and a non-linear transformation of the input, residual connections introduce shortcut paths between network layers, enabling direct information flow across multiple layers[56]. In multimodal feature fusion, Han et al.[57] proposed a lightweight convolutional feature fusion strategy named residual merging, which improves training speed and enhances model practicality without sacrificing classification accuracy. Su et al.[58] used global attention and residual connections to fuse low-level and high-level features, thereby improving sentiment recognition accuracy.

In the context of emergency events, Shahid et al.[59] analysed user information expression and sentiment during disasters, employing GNNs to understand the complex relationships between text and visual modalities. Zeng et al.[60] addressed negative emotion recognition in public health emergencies from a multimodal data perspective, constructing a multimodal fine-grained negative emotion recognition model based on GCNs and ensemble learning, validated on COVID-19-related Weibo image-text data.

Despite these advances, several critical gaps remain. First, the fixed-weight fusion strategies commonly used in existing cross-modal interactive attention mechanisms lack dynamic adaptability; they cannot adjust the fusion ratios of modality features according to the input content. Although gated units can control information flow, they primarily rely on the internal states of recurrent networks and have limited ability to directly perceive and respond to cross-modal interaction features. Second, most multimodal sentiment models have been validated on well-curated public datasets, and whether they are applicable to real-world emergency public opinion sentiment analysis remains unclear. Public datasets typically contain mature, well-matched image-text pairs, whereas emergency event data often suffer from a mismatch in the volume of image and text data—a problem that has received little attention in existing multimodal sentiment modelling. Third, although residual connections have shown clear advantages in cross-modal feature fusion, their integration with dynamic cross-modal attention remains underexplored. Overall, research on multimodal sentiment analysis for emergency public opinion events is still fragmented, with insufficient attention paid to the image-text data imbalance problem and the deep interactive influences between public opinion images and texts.

To address the interactive influence between image and text features in emergency public opinion, this paper proposes a sentiment analysis model based on Adaptive Cross-Modal Fusion Attention. It consists of two-stage cross-modal interactive attention and adaptive attention: the former measures the semantic correlation and interaction between image and text to obtain enhanced cross-modal features, while the latter dynamically assigns fusion weights according to modality contributions to optimize the feature fusion strategy. Furthermore, residual connections are introduced to optimize the transmission path from original features to enhanced features, further improving the cross-modal fusion feature representation.

Based on the identified research gaps—namely, the lack of online causal feature selection methods for streaming multimodal data in public health emergencies (PHEs), the fixed-weight and dynamically inflexible nature of existing cross-modal fusion strategies, and the neglect of image–text data imbalance in real-world PHE scenarios—this paper proposes an online multimodal sentiment analysis framework. Specifically, we develop a Multimodal Online Divide-and-Conquer Markov Blanket Learning (O-DC) algorithm to incrementally select causally relevant features from streaming online data with low time complexity, and design a Cross-Modal Interactive Enhanced Fusion Network with two-stage cross-modal attention, adaptive modal weighting, and residual connections to dynamically capture deep text–image interactions.

3. Problem Definition and Overall Framework

3.1. Problem Formulation

Let a multimodal data case arriving at time step ${\mathrm{S}}_{\mathrm{i}}=\left({\mathrm{X}}_{\mathrm{i}}^{\mathrm{T}},{\mathrm{X}}_{\mathrm{i}}^{\mathrm{I}},{\mathcal{Y}}_{\mathrm{i}}\right)$, where:${\mathrm{X}}_{\mathrm{i}}^{\mathrm{T}}\in {\mathbb{R}}^{{\mathrm{d}}_{\mathrm{T}}}$ denotes the raw high-dimensional feature vector for the text modality. ${\mathrm{X}}_{\mathrm{i}}^{\mathrm{I}}\in {\mathbb{R}}^{{\mathrm{d}}_{\mathrm{I}}}$ denotes the raw feature vector for the image modality. ${\mathcal{Y}}_{\mathrm{i}}\in \mathrm{Y}=\left\{\mathrm{p}\mathrm{o}\mathrm{s}\mathrm{i}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{e},\mathrm{n}\mathrm{e}\mathrm{u}\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{l},\mathrm{n}\mathrm{e}\mathrm{g}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{e}\right\}$ is the sentiment label.

The total raw feature space is ${\mathcal{F}}_{\mathrm{i}}={\mathrm{X}}_{\mathrm{i}}^{\mathrm{T}}\cup {\mathrm{X}}_{\mathrm{i}}^{\mathrm{I}}$, whose dimension ${\mathrm{d}=\mathrm{d}}_{\mathrm{T}}+{\mathrm{d}}_{\mathrm{I}}$ can be very large and grows with the emergence of new words and visual patterns. Our objective is to design an online learning model $\mathcal{M}$ such that at any time step $\mathrm{i}$, it can dynamically select a feature subset ${\mathcal{S}}_{\mathrm{i}}\subset {\mathcal{F}}_{\mathrm{i}}$ from the high-dimensional streaming feature space ${\mathcal{F}}_{\mathrm{i}}$, with low redundancy and strong relevance to ${\mathcal{Y}}_{\mathrm{i}}$. Then, based on ${\mathcal{S}}_{\mathrm{i}}$, a powerful classification function $\mathrm{f}:{\mathcal{X}}_{\mathrm{i}}^{\mathrm{T}}{\times \mathcal{X}}_{\mathrm{i}}^{\mathrm{I}}\to \mathcal{Y}$ is constructed to accurately predict ${\mathcal{Y}}_{\mathrm{i}}$, where ${\mathcal{X}}_{\mathrm{i}}^{\mathrm{T}}$,${\mathcal{X}}_{\mathrm{i}}^{\mathrm{I}}$ are the feature representations after selection.

3.2. Overall Framework

The proposed framework consists of two core stages: ①Online Multimodal Feature Selection Stage: For each newly arrived case ${\mathrm{s}}_{\mathrm{t}}$, its raw text and image features are input into the Online Divide-and-Conquer Markov Blanket Learning Algorithm. This algorithm maintains two dynamic “feature pools” —the text-relevant feature set ${\mathrm{P}\mathrm{C}}^{\mathrm{T}}$ and the image-relevant feature set ${\mathrm{P}\mathrm{C}}^{\mathrm{I}}$. Based on incremental mutual information computation, the algorithm decides whether to include new features into the corresponding pool or discard old ones. The outputs are the filtered text feature subset ${\mathcal{X}}_{\mathrm{i}}^{\mathrm{T}}$ and image feature subset ${\mathcal{X}}_{\mathrm{i}}^{\mathrm{I}}$. ②Cross-Modal Interactive Enhanced Fusion and Classification Stage: The filtered features are first encoded by modality-specific encoders to obtain deep representations ${\mathrm{H}}^{\mathrm{T}}$ and ${\mathrm{H}}^{\mathrm{I}}$. Subsequently, they enter the Cross-Modal Interactive Enhanced Fusion Network. This network generates a context-aware fused representation ${\mathrm{H}}^{\mathrm{f}\mathrm{u}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}}$ through two-stage interactive attention and adaptive weight assignment. Finally, ${\mathrm{H}}^{\mathrm{f}\mathrm{u}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}}$ passes through residual connections and a fully connected classification layer to output the sentiment prediction ${\mathcal{Y}}_{\mathrm{i}}$. The model is updated via an online cross-entropy loss.

4. Multimodal Online Divide-And-Conquer Markov Blanket Learning Algorithm

In real-world streaming data applications—such as social media monitoring during public health emergencies (PHEs)—raw features naturally arrive in ‌multimodal form‌. The primary ‌raw features‌ include ‌textual content‌ and ‌visual content‌. Traditional online feature selection methods often operate on pre-processed, unimodal representations, lacking a principled framework to jointly and efficiently process heterogeneous data streams.

Online Markov Blanket (MB) learning aims to dynamically identify the minimal optimal feature set for predicting a target variable from a stream of incoming features. However, existing methods face three core challenges in multimodal settings: ‌(1) Cross-modal redundancy‌, where features from different modalities convey overlapping information; ‌(2) Heterogeneous dependency modeling‌, due to the distinct statistical nature of text (discrete) and image (continuous) features; and ‌(3) Computational inefficiency‌, especially when exhaustive subset enumeration is required.

To address these challenges, we propose the ‌Multimodal Online Divide-and-Conquer Markov Blanket Learning (M-O-DC)‌ algorithm.

4.1. Markov Blanket and Mutual Information

In probabilistic graphical models, the Markov blanket $\mathrm{M}\mathrm{B}\left(\mathrm{Y}\right)$ of a target variable $\mathrm{Y}$ is defined as the set of variables that renders $\mathrm{Y}$ conditionally independent of all other variables in the graph. Formally, for any variable ${\mathrm{X}}_{\mathrm{i}}\notin \mathrm{M}\mathrm{B}\left(\mathrm{Y}\right)$, we have ${\mathrm{X}}_{\mathrm{i}}\perp \mathrm{Y}\left|\mathrm{M}\mathrm{B}\left(\mathrm{Y}\right)\right.$ . The Markov blanket of $\mathrm{Y}$ consists of its parents, children, and the other parents of its children (spouses). For classification tasks, $\mathrm{M}\mathrm{B}\left(\mathrm{Y}\right)$ constitutes the minimal sufficient feature subset for predicting $\mathrm{Y}$, as it contains all the information necessary to determine $\mathrm{Y}$ while discarding irrelevant and redundant features.

Mutual information (MI) is a fundamental measure of dependency between two random variables:

$$\mathrm{I}\left(\mathrm{X},\mathrm{Y}\right)=\sum _{\mathrm{x}\in \mathrm{X}}\sum _{\mathrm{y}\in \mathrm{Y}}\mathrm{p}\left(\mathrm{x},\mathrm{y}\right)\mathrm{l}\mathrm{o}\mathrm{g}{\displaystyle \frac{\mathrm{p}(\mathrm{x},\mathrm{y})}{\mathrm{p}\left(\mathrm{x}\right)\mathrm{p}\left(\mathrm{y}\right)}}$$

(1)

Conditional mutual information $\mathrm{I}\left(\mathrm{X};\mathrm{Y}|\mathrm{Z}\right)$ quantifies the association between $\mathrm{X}$ and $\mathrm{Y}$ given $\mathrm{Z}$, and is defined analogously using conditional probabilities.

$$\mathrm{I}\left(\mathrm{X};\mathrm{Y}|\mathrm{Z}\right)=\sum _{\mathrm{x},\mathrm{y},\mathrm{z}}\mathrm{p}\left(\mathrm{x},\mathrm{y},\mathrm{z}\right)\mathrm{l}\mathrm{o}\mathrm{g}{\displaystyle \frac{\mathrm{p}(\mathrm{x},\mathrm{y}\left|\mathrm{z})\right.}{\mathrm{p}(\mathrm{x}\left|\mathrm{z}\left)\mathrm{p}\right(\mathrm{y}\left|\mathrm{z})\right.\right.}}$$

(2)

A feature ${\mathrm{X}}_{\mathrm{i}}$ belongs to $\mathrm{M}\mathrm{B}\left(\mathrm{Y}\right)$ if and only if it provides unique predictive information about $\mathrm{Y}$ that is not contained in any other subset of $\mathrm{M}\mathrm{B}\left(\mathrm{Y}\right)$. This property is fundamental for online feature selection: we aim to incrementally maintain an approximation of $\mathrm{M}\mathrm{B}\left(\mathrm{Y}\right)$ as features arrive in a stream.

4.2. M-O-DC Algorithm

4.2.1. Problem Setup and Multimodal Input Definition‌

Let the streaming feature space consist of a sequence of ‌textual features‌ ${\mathbf{F}}_{\mathbf{T}}=\left\{{\mathbf{X}}_{\mathbf{i}}^{\mathbf{T}}\right\}$ and ‌Image visual features‌ ${\mathbf{F}}_{\mathbf{I}}=\left\{{\mathbf{X}}_{\mathbf{i}}^{\mathbf{I}}\right\}$, arriving sequentially. The sentiment label $\mathbf{Y}$ is influenced by both modalities. We define the ‌candidate $\mathbf{P}\mathbf{C}$ set per modality‌, denoted $\mathbf{C}\_{\mathbf{P}\mathbf{C}}_{\mathbf{Y}}^{\mathbf{m}}$ for $\mathbf{m}\in \left\{\mathbf{T},\mathbf{I}\right\}$, as the dynamically maintained set of features that are likely parents or children of $\mathbf{Y}$ within modality $\mathbf{m}$.

4.2.2. Decoupled Multimodal Processing‌

Input: Current text PC set $\mathbf{C}\_{\mathbf{P}\mathbf{C}}_{\mathbf{Y}}^{\mathbf{T}}$, image PC set $\mathbf{C}\_{\mathbf{P}\mathbf{C}}_{\mathbf{Y}}^{\mathbf{I}}$ , new feature ${\mathbf{X}}_{\mathbf{i}}^{\mathbf{m}}$ , estimated target distribution ${\widehat{\mathbf{p}}}_{\mathbf{t}}\left(\mathbf{y}\right)$, modality-specific thresholds ${\mathsf{\tau }}_{\mathbf{i}\mathbf{n}\mathbf{c}}^{\mathbf{m}}$, ${\mathsf{\tau }}_{\mathbf{r}\mathbf{e}\mathbf{d}}^{\mathbf{m}}$. ${\mathsf{\tau }}_{\mathbf{i}\mathbf{n}\mathbf{c}}^{\mathbf{m}}$ is the inclusion threshold. When a new feature ${\mathbf{X}}_{\mathbf{i}}^{\mathbf{m}}$ arrives, if its mutual information with the target variable $\mathbf{Y}$, denoted $\mathbf{I}({\mathbf{X}}_{\mathbf{i}}^{\mathbf{m}},\mathbf{Y})$, exceeds this threshold, the algorithm considers ${\mathbf{X}}_{\mathbf{i}}^{\mathbf{m}}$ a candidate for the Parent-Child (PC) set.

Output: Updated multimodal Markov Blanket candidate set $\mathbf{M}\mathbf{B}\left(\mathbf{Y}\right)$.

Step A: Intra-modal PC Learning‌

For a newly arrived feature ${\mathbf{X}}_{\mathbf{i}}^{\mathbf{m}}$①Independence Test‌: Compute $\mathbf{I}({\mathbf{X}}_{\mathbf{i}}^{\mathbf{m}},\mathbf{Y})$, if $\mathbf{I}\left({\mathbf{X}}_{\mathbf{i}}^{\mathbf{m}},\mathbf{Y}\right)<{\mathsf{\tau }}_{\mathbf{i}\mathbf{n}\mathbf{c}}^{\mathbf{m}}$, discard ${\mathbf{X}}_{\mathbf{i}}^{\mathbf{m}}$ as irrelevant.

②Redundancy Removal‌: Compare ${\mathbf{X}}_{\mathbf{i}}^{\mathbf{m}}$ with features in $\mathbf{C}\_{\mathbf{P}\mathbf{C}}_{\mathbf{Y}}^{\mathbf{m}}$ using symmetric uncertainty (SU). Remove any ${\mathbf{X}}_{\mathbf{k}}^{\mathbf{m}}\in \mathbf{C}\_{\mathbf{P}\mathbf{C}}_{\mathbf{Y}}^{\mathbf{m}}$ that becomes redundant given ${\mathbf{X}}_{\mathbf{i}}^{\mathbf{m}}$.

③PC Update‌: If ${\mathbf{X}}_{\mathbf{i}}^{\mathbf{m}}$ provides significant non-redundant information, add it to $\mathbf{C}\_{\mathbf{P}\mathbf{C}}_{\mathbf{Y}}^{\mathbf{m}}$Step B: Cross-modal Spouse Learning‌

When a new PC node ${\mathbf{X}}_{\mathbf{i}}^{\mathbf{m}}$ is identified, trigger cross-modal spouse discovery:

①For each ‌cross-modal‌ candidate ${\mathbf{Z}}_{\mathbf{j}}^{\mathbf{n}}\in \mathbf{C}\_{\mathbf{P}\mathbf{C}}_{\mathbf{Y}}^{\mathbf{n}}$ ($\mathbf{n}\ne \mathbf{m}$), check if ${\mathbf{X}}_{\mathbf{i}}^{\mathbf{m}}\in \mathbf{P}\mathbf{C}\left({\mathbf{Z}}_{\mathbf{j}}^{\mathbf{n}}\right)$ or vice versa.

②Apply the ‌Cross-modal Spouse Identification Theorem‌: If ${\mathbf{Z}}_{\mathbf{j}}^{\mathbf{n}}$ and ${\mathbf{X}}_{\mathbf{i}}^{\mathbf{m}}$ are strongly associated, and the presence of ${\mathbf{X}}_{\mathbf{i}}^{\mathbf{m}}$ increases the conditional mutual information $\mathbf{I}(\mathbf{Y};{\mathbf{Z}}_{\mathbf{j}}^{\mathbf{n}}\left|{\mathbf{X}}_{\mathbf{i}}^{\mathbf{m}})>\mathit{I}(\mathit{Y};\right.{\mathbf{Z}}_{\mathbf{j}}^{\mathbf{n}})$, then ${\mathbf{Z}}_{\mathbf{j}}^{\mathbf{n}}$ is a cross-modal spouse of $\mathbf{Y}$.

③Add qualifying ${\mathbf{Z}}_{\mathbf{j}}^{\mathbf{n}}$ to the spouse set $\mathbf{S}\mathbf{P}\left(\mathbf{Y}\right)$, which is inherently multimodal.

This design ensures that a newly discovered textual PC can help identify a relevant visual spouse, and vice versa, enabling a complete causal structure discovery across modalities.

4.3. Online Mutual Information Estimation for Text and Image

4.3.1. Text Features

Text features are typically discretized.

Sufficient Statistics‌: Maintain a count matrix ${\mathbf{C}}_{\mathbf{T}}\left[{\mathbf{f}}_{\mathbf{t}},\mathbf{y}\right]$ for each discrete feature ${\mathbf{f}}_{\mathbf{t}}$ Online Update Rules: ${\mathbf{C}}_{\mathbf{T}}\left[\mathbf{X},\mathbf{y}\right]\leftarrow {\mathbf{C}}_{\mathbf{T}}\left[\mathbf{X},\mathbf{y}\right]+\mathbf{I}\left({\mathbf{f}}_{\mathbf{t}}=\mathbf{X},\mathbf{Y}=\mathbf{y}\right)-\mathbf{e}\mathbf{x}\mathbf{p}\mathbf{i}\mathbf{r}\mathbf{e}\mathbf{d}\_\mathbf{c}\mathbf{o}\mathbf{n}\mathbf{t}\mathbf{r}\mathbf{i}\mathbf{b}\mathbf{u}\mathbf{t}\mathbf{i}\mathbf{o}\mathbf{n}$ MI Computation (with Laplace Smoothing):

$${\mathbf{I}}_{\mathbf{T}}\left(\mathbf{Y};{\mathbf{f}}_{\mathbf{t}}\right)=\sum _{\mathbf{X},\mathbf{y}}{\tilde{\mathbf{P}}}_{\mathbf{T}}\left(\mathbf{X},\mathbf{y}\right)\mathbf{l}\mathbf{o}\mathbf{g}{\displaystyle \frac{{\tilde{\mathbf{P}}}_{\mathbf{T}}(\mathbf{X},\mathbf{y})}{{\tilde{\mathbf{P}}}_{\mathbf{T}}\left(\mathbf{X}\right)\tilde{\mathbf{P}}\left(\mathbf{y}\right)}}$$

(3)

Where $\tilde{\mathbf{P}}$ denotes smoothed probability estimates

4.3.2. Image Features

Image features are high-dimensional continuous vectors.

Sufficient Statistics‌: Use ‌online Kernel Density Estimation (KDE)‌ to approximate class-conditional densities.

Incremental Parameter Updates (Exponential Moving Averages)‌:

$${\mathsf{\mu }}_{\mathbf{y}}^{\mathbf{I}}\leftarrow \mathsf{\lambda }{\mathsf{\mu }}_{\mathbf{y}}^{\mathbf{I}}+(1-\mathsf{\lambda }){\mathbf{X}}_{\mathbf{I},\mathbf{n}\mathbf{e}\mathbf{w}}$$

(4)

$$\left({\mathsf{\sigma }}_{\mathbf{y}}^{\mathbf{I}}\right)\leftarrow \mathsf{\lambda }\left({\mathsf{\sigma }}_{\mathbf{y}}^{\mathbf{I}}\right)+\left(1-\mathsf{\lambda }\right)({\mathbf{X}}_{\mathbf{I},\mathbf{n}\mathbf{e}\mathbf{w}}-{\mathsf{\mu }}_{\mathbf{y}}^{\mathbf{I}}$$

(5)

MI Estimation via KL Divergence‌:

$${\mathbf{I}}_{\mathbf{I}}(\mathbf{Y};{\mathbf{f}}_{\mathbf{I}})\approx \sum _{\mathbf{I}}\mathbf{P}\left(\mathbf{y}\right){\mathbf{D}}_{\mathbf{K}\mathbf{L}}\left(\mathbf{P}\left({\mathbf{f}}_{\mathbf{I}}\left|\mathbf{y})‖\mathbf{P}({\mathbf{f}}_{\mathbf{I}}\right.\right)\right)$$

(6)

Where $\mathbf{P}({\mathbf{f}}_{\mathbf{I}}\left|\mathbf{y})\right.$ is modeled using online-updated Gaussian approximations.

4.3.3. Cross-modal Conditional MI Approximation‌

Computing $\mathbf{I}(\mathbf{Y};{\mathbf{Z}}_{\mathbf{j}}^{\mathbf{I}}\left|{\mathbf{X}}_{\mathbf{i}}^{\mathbf{T}}\right.)$ is challenging due to heterogeneity. M-O-DC uses a ‌chain factorization with cross-modal independence assumption‌:

$$\mathbf{I}\left(\mathbf{Y};{\mathbf{Z}}_{\mathbf{j}}^{\mathbf{I}}\left|{\mathbf{X}}_{\mathbf{i}}^{\mathbf{T}}\right.\right)\approx \mathbf{I}\left(\mathbf{Y};{\mathbf{Z}}_{\mathbf{j}}^{\mathbf{I}}\right)-\sum _{\mathbf{k}}{\mathbf{w}}_{\mathbf{k}}\bullet \mathbf{I}({\mathbf{Z}}_{\mathbf{j}}^{\mathbf{I}};{\mathbf{C}}_{\mathbf{k}}^{\mathbf{T}})$$

(7)

Where ${\mathbf{C}}_{\mathbf{k}}^{\mathbf{T}}$ are the top-k text features in $\mathbf{C}\_{\mathbf{P}\mathbf{C}}_{\mathbf{Y}}^{\mathbf{T}}$ most correlated with ${\mathbf{Z}}_{\mathbf{j}}^{\mathbf{I}}$. This approximation is effective when text and image features are conditionally independent given $\mathbf{Y}$.

4.4. Multimodal Dynamic Adaptation Mechanisms‌

4.4.1. Modality-Adaptive Thresholds‌

Thresholds ${\mathsf{\tau }}_{\mathbf{i}\mathbf{n}\mathbf{c}}^{\mathbf{m}}$ and ${\mathsf{\tau }}_{\mathbf{r}\mathbf{e}\mathbf{d}}^{\mathbf{m}}$ are dynamically adjusted per modality:

$${\mathsf{\tau }}_{\mathbf{i}\mathbf{n}\mathbf{c}}^{\mathbf{m}}\leftarrow {\mathsf{\tau }}_{\mathbf{i}\mathbf{n}\mathbf{c}}^{\mathbf{m}}\bullet (1+\mathsf{\alpha }\bullet {\displaystyle \frac{\left|{\mathbf{C}}_{{\mathbf{P}\mathbf{C}}_{\mathbf{Y}}^{\mathbf{m}}}\right|}{{\mathbf{N}}_{\mathbf{m}\mathbf{a}\mathbf{x}}^{\mathbf{m}}}})$$

(8)

This accounts for differences in feature density and redundancy between text and image streams.

4.4.2. Multimodal Concept Drift Detection‌

Public sentiment may evolve at different rates across modalities. M-O-DC defines a ‌multimodal causal drift statistic‌:

$${\mathbf{D}}_{\mathbf{t}}^{(\mathbf{m},\mathbf{n})}=\left|{\displaystyle \frac{1}{\left|\mathbf{C}\_{\mathbf{P}\mathbf{C}}_{\mathbf{Y}}^{\mathbf{m}}\right|}}\sum _{{\mathbf{X}}_{\mathbf{i}}^{\mathbf{m}}\in \mathbf{C}\_{\mathbf{P}\mathbf{C}}_{\mathbf{Y}}^{\mathbf{m}}}\mathbf{I}\left(\mathbf{Y};{\mathbf{X}}_{\mathbf{i}}^{\mathbf{m}}\right)-\mathbf{w}\mathbf{i}\mathbf{n}\mathbf{d}\mathbf{o}\mathbf{w}\_\mathbf{a}\mathbf{v}\mathbf{e}\mathbf{r}\mathbf{a}\mathbf{g}\mathbf{e}\mathbf{d}\mathbf{M}\mathbf{I}\right|$$

(9)

If ${\mathbf{D}}_{\mathbf{t}}^{(\mathbf{m},\mathbf{n})}>\mathit{k}\bullet {\mathsf{\sigma }}_{\mathbf{D}}$, a drift is detected, triggering adaptive forgetting in the affected modality.

5. Cross-Modal Interactive Enhanced Fusion Network

5.1. Deep Encoder Design

5.1.1. Formalization of Multimodal Feature Encoding

The features filtered by M-O-DC, ${\tilde{\mathrm{X}}}^{\mathrm{T}}\in {\mathrm{R}}^{{\mathrm{d}}_{\mathrm{T}}}$ and ${\tilde{\mathrm{X}}}^{\mathrm{I}}\in {\mathrm{R}}^{{\mathrm{d}}_{\mathrm{I}}}$, are encoded by modality-specific deep encoders.

Text Encoder (based on Transformer architecture):

$${\mathrm{H}}^{\mathrm{T}}={\mathrm{E}\mathrm{n}\mathrm{c}\mathrm{o}\mathrm{d}\mathrm{e}\mathrm{r}}_{\mathrm{T}}\left({\tilde{\mathrm{X}}}^{\mathrm{T}}\right)\in {\mathbb{R}}^{\mathrm{L}\times {\mathrm{d}}_{\mathrm{h}}}$$

(10)

Where ${\mathrm{E}\mathrm{n}\mathrm{c}\mathrm{o}\mathrm{d}\mathrm{e}\mathrm{r}}_{\mathrm{T}}=\mathrm{S}\mathrm{t}\mathrm{a}\mathrm{c}\mathrm{k}{\left\{\mathrm{M}\mathrm{u}\mathrm{l}\mathrm{t}\mathrm{i}\mathrm{H}\mathrm{e}\mathrm{a}\mathrm{d}\mathrm{S}\mathrm{e}\mathrm{l}\mathrm{f}\mathrm{A}\mathrm{t}\mathrm{t}\mathrm{n},\mathrm{F}\mathrm{F}\mathrm{n}\right\}}_{\mathrm{l}=1}^{{\mathrm{N}}_{\mathrm{T}}}$, Let $\mathrm{L}=\mathrm{m}\mathrm{a}\mathrm{x}(⌊{\displaystyle \frac{{\mathrm{d}}_{\mathrm{T}}}{{\mathrm{d}}_{\mathrm{t}\mathrm{o}\mathrm{k}\mathrm{e}\mathrm{n}}}}⌋,{\mathrm{L}}_{\mathrm{m}\mathrm{i}\mathrm{n}})$ be the sequence length.

Image Encoder (based on Vision Transformer):

$${\mathrm{H}}^{\mathrm{I}}={\mathrm{E}\mathrm{n}\mathrm{c}\mathrm{o}\mathrm{d}\mathrm{e}\mathrm{r}}_{\mathrm{I}}\left({\tilde{\mathrm{X}}}^{\mathrm{I}}\right)\in {\mathbb{R}}^{\mathrm{N}\times {\mathrm{d}}_{\mathrm{h}}}$$

(11)

where ${\tilde{\mathrm{X}}}^{\mathrm{I}}$ is split into $\mathrm{N}=⌊{\displaystyle \frac{\mathrm{H}\times \mathrm{W}}{{\mathrm{P}}^2}}⌋$ patches, each projected to dimension $\mathrm{P}\times \mathrm{P}\times \mathrm{C}\to {\mathrm{d}}_{\mathrm{h}}$.

5.2. Two-Stage Cross-Modal Interactive Attention:

5.2.1. Stage 1: Bidirectional Cross-Modal Attention

We compute cross-modal attention weights in both directions to establish bidirectional interactive mappings. For text-to-image attention, text features serve as queries and image features as keys:

Text-to-Image Attention:${\mathrm{Q}}_{\mathrm{T}}={\mathrm{H}}^{\mathrm{T}}{\mathrm{W}}_{{\mathrm{Q}}^{\mathrm{T}}}\in {\mathrm{R}}^{\mathrm{L}\times {\mathrm{d}}_{\mathrm{k}}}$, ${\mathrm{K}}_{\mathrm{I}}={\mathrm{H}}^{\mathrm{I}}{\mathrm{W}}_{{\mathrm{k}}^{\mathrm{I}}}\in {\mathrm{R}}^{\mathrm{N}\times {\mathrm{d}}_{\mathrm{k}}}$, ${\mathrm{V}}_{\mathrm{I}}={\mathrm{H}}^{\mathrm{I}}{\mathrm{W}}_{{\mathrm{v}}^{\mathrm{I}}}\in {\mathrm{R}}^{\mathrm{N}\times {\mathrm{d}}_{\mathrm{h}}}$${\mathrm{W}}_{{\mathrm{Q}}^{\mathrm{T}}}$, ${\mathrm{W}}_{{\mathrm{k}}^{\mathrm{I}}}$, ${\mathrm{W}}_{{\mathrm{v}}^{\mathrm{I}}}$: learnable weight matrices for query, key, value projections

${\mathrm{d}}_{\mathrm{k}}$: dimension of query/key vectors (usually ${\mathrm{d}}_{\mathrm{k}}={\displaystyle \frac{{\mathrm{d}}_{\mathrm{h}}}{\mathrm{h}}}$ where $\mathrm{h}$ is number of attention heads)

${\mathrm{Q}}^{\mathrm{T}}:$query matrix from text; ${\mathrm{k}}^{\mathrm{I}}$: key matrix from image; ${\mathrm{V}}_{\mathrm{I}}$: value matrix from image.

The attention weight matrix is:

$${\mathrm{A}}_{\mathrm{T}\to \mathrm{I}}\left[\mathrm{l},\mathrm{n}\right]={\displaystyle \frac{\mathrm{e}\mathrm{x}\mathrm{p}\left(\frac{1}{\sqrt{{\mathrm{d}}_{\mathrm{k}}}}〈{\mathrm{q}}_{\mathrm{l}}^{\mathrm{T}},{\mathrm{k}}_{\mathrm{n}}^{\mathrm{I}}〉\right)}{{\displaystyle \sum _{\mathrm{m}=1}^{\mathrm{N}}}\mathrm{e}\mathrm{x}\mathrm{p}(\frac{1}{\sqrt{{\mathrm{d}}_{\mathrm{k}}}}〈{\mathrm{q}}_{\mathrm{l}}^{\mathrm{T}},{\mathrm{k}}_{\mathrm{m}}^{\mathrm{I}}〉)}}$$

(12)

where ⟨·,·⟩ denotes the inner product.

The text-to-image attention output (cross-modal feature) is: ${\mathrm{A}\mathrm{t}\mathrm{t}\mathrm{n}}_{\mathrm{T}\to \mathrm{I}}={\mathrm{A}}_{\mathrm{T}\to \mathrm{I}}{\mathrm{V}}_{\mathrm{I}}\in {\mathbb{R}}^{\mathrm{L}\times {\mathrm{d}}_{\mathrm{h}}}$, where ${\mathrm{V}}_{\mathrm{I}}={\mathrm{H}}^{\mathrm{I}}{\mathrm{W}}_{\mathrm{V}}^{\mathrm{I}}\in {\mathbb{R}}^{\mathrm{N}\times {\mathrm{d}}_{\mathrm{h}}}$Similarly, image-to-text attention is computed symmetrically:${\mathrm{Q}}_{\mathrm{I}}={\mathrm{H}}^{\mathrm{I}}{\mathrm{W}}_{{\mathrm{Q}}^{\mathrm{I}}}\in {\mathrm{R}}^{\mathrm{N}\times {\mathrm{d}}_{\mathrm{k}}}$, ${\mathrm{K}}_{\mathrm{T}}={\mathrm{H}}^{\mathrm{T}}{\mathrm{W}}_{{\mathrm{k}}^{\mathrm{T}}}\in {\mathrm{R}}^{\mathrm{L}\times {\mathrm{d}}_{\mathrm{k}}}$, ${\mathrm{V}}_{\mathrm{T}}={\mathrm{H}}^{\mathrm{T}}{\mathrm{W}}_{{\mathrm{v}}^{\mathrm{T}}}\in {\mathrm{R}}^{\mathrm{L}\times {\mathrm{d}}_{\mathrm{h}}}$.

$${\mathrm{A}}_{\mathrm{I}\to \mathrm{T}}\left[\mathrm{n},\mathrm{l}\right]={\displaystyle \frac{\mathrm{e}\mathrm{x}\mathrm{p}\left(\frac{1}{\sqrt{{\mathrm{d}}_{\mathrm{k}}}}〈{\mathrm{q}}_{\mathrm{n}}^{\mathrm{I}},{\mathrm{k}}_{\mathrm{l}}^{\mathrm{T}}〉\right)}{{\displaystyle \sum _{\mathrm{l}=1}^{\mathrm{L}}}\mathrm{e}\mathrm{x}\mathrm{p}(\frac{1}{\sqrt{{\mathrm{d}}_{\mathrm{k}}}}〈{\mathrm{q}}_{\mathrm{n}}^{\mathrm{I}},{\mathrm{k}}_{\mathrm{l}}^{\mathrm{T}}〉)}}$$

(13)

Image-to-Text Attention (symmetric): ${\mathrm{A}\mathrm{t}\mathrm{t}\mathrm{n}}_{\mathrm{I}\to \mathrm{T}}={\mathrm{A}}_{\mathrm{I}\to \mathrm{T}}{\mathrm{V}}_{\mathrm{T}}\in {\mathbb{R}}^{\mathrm{N}\times {\mathrm{d}}_{\mathrm{h}}}$The contextual representations after first-stage interaction (enhanced by residual connection and FFN) are:

$${\mathrm{C}}^{\mathrm{T}\leftarrow \mathrm{I}}=\mathrm{L}\mathrm{a}\mathrm{y}\mathrm{e}\mathrm{r}\mathrm{N}\mathrm{o}\mathrm{r}\mathrm{m}({\mathrm{H}}^{\mathrm{T}}+\mathrm{F}\mathrm{F}\mathrm{N}\left({\mathrm{A}\mathrm{t}\mathrm{t}\mathrm{n}}_{\mathrm{T}\to \mathrm{I}}\right))$$

(14)

$${\mathrm{C}}^{\mathrm{I}\leftarrow \mathrm{T}}=\mathrm{L}\mathrm{a}\mathrm{y}\mathrm{e}\mathrm{r}\mathrm{N}\mathrm{o}\mathrm{r}\mathrm{m}({\mathrm{H}}^{\mathrm{I}}+\mathrm{F}\mathrm{F}\mathrm{N}\left({\mathrm{A}\mathrm{t}\mathrm{t}\mathrm{n}}_{\mathrm{I}\to \mathrm{T}}\right))$$

(15)

Where, $\mathrm{F}\mathrm{F}\mathrm{N}$ is a two-layer $\mathrm{M}\mathrm{L}\mathrm{P}$ used for feature transformation: $\mathrm{F}\mathrm{F}\mathrm{N}\left(\mathrm{x}\right)=\mathrm{G}\mathrm{E}\mathrm{L}\mathrm{U}\left(\mathrm{x}{\mathrm{W}}_1+{\mathrm{b}}_1\right){\mathrm{W}}_2+{\mathrm{b}}_2$.

5.2.2. Stage 2: Deep Interactive Normalization

The second stage performs deep processing of the weighted features generated in the first stage. We apply softmax normalization to the interaction matrices, transforming attention weights into probability distributions that quantify the matching relationships between local textual and visual features. This normalization process not only quantifies local feature alignments but also mines potential cross-modal semantic associations that may not be directly observable.

We implement this using Multi-Head Co-Attention (MHCA):

$${\tilde{\mathrm{C}}}^{\mathrm{T}}=\mathrm{M}\mathrm{H}\mathrm{C}\mathrm{A}({\mathrm{C}}^{\mathrm{T}\leftarrow \mathrm{I}},{\mathrm{C}}^{\mathrm{I}\leftarrow \mathrm{T}},{\mathrm{C}}^{\mathrm{I}\leftarrow \mathrm{T}})$$

(16)

$${\tilde{\mathrm{H}}}^{\mathrm{T}}=\mathrm{L}\mathrm{a}\mathrm{y}\mathrm{e}\mathrm{r}\mathrm{N}\mathrm{o}\mathrm{r}\mathrm{m}({\tilde{\mathrm{C}}}^{\mathrm{T}}+\mathrm{F}\mathrm{F}\mathrm{N}\left({\tilde{\mathrm{C}}}^{\mathrm{T}}\right))$$

(17)

$${\tilde{\mathrm{C}}}^{\mathrm{I}}=\mathrm{M}\mathrm{H}\mathrm{C}\mathrm{A}({\mathrm{C}}^{\mathrm{I}\leftarrow \mathrm{T}},{\mathrm{C}}^{\mathrm{T}\leftarrow \mathrm{I}},{\mathrm{C}}^{\mathrm{T}\leftarrow \mathrm{I}})$$

(18)

$${\tilde{\mathrm{H}}}^{\mathrm{I}}=\mathrm{L}\mathrm{a}\mathrm{y}\mathrm{e}\mathrm{r}\mathrm{N}\mathrm{o}\mathrm{r}\mathrm{m}({\tilde{\mathrm{C}}}^{\mathrm{I}}+\mathrm{F}\mathrm{F}\mathrm{N}\left({\tilde{\mathrm{C}}}^{\mathrm{I}}\right))$$

(19)

Compared to methods that only perform direct feature weighting based on initial attention weights, our two-stage hierarchical processing captures finer complementary information and interaction patterns between modalities, enhancing the expressive power of cross-modal features for sentiment semantics.

5.3. Adaptive Modal Weighting

5.3.1. Enhanced Cross-Modal Features

We first enhance the cross-modal features by adding the original representations to the attention outputs from Stage 1:${\widehat{\mathrm{H}}}^{\mathrm{T}}={\mathrm{H}}^{\mathrm{T}}+{\mathrm{A}\mathrm{t}\mathrm{t}\mathrm{n}}_{\mathrm{T}\to \mathrm{I}}\in {\mathrm{R}}^{\mathrm{L}\times {\mathrm{d}}_{\mathrm{h}}}$,${\widehat{\mathrm{H}}}^{\mathrm{I}}={\mathrm{H}}^{\mathrm{I}}+{\mathrm{A}\mathrm{t}\mathrm{t}\mathrm{n}}_{\mathrm{I}\to \mathrm{T}}\in {\mathrm{R}}^{\mathrm{N}\times {\mathrm{d}}_{\mathrm{h}}}$These enhanced features integrate both the original modality information and the cross-modal influences.

5.3.2. Gated Weight Generation Network

To dynamically adjust the fusion weights for text and image modalities based on the current context, we concatenate the enhanced features along the sequence dimension and apply a learnable gating network.

First, we obtain a global representation by mean pooling over the sequence dimension:

$${\overline{\mathrm{h}}}^{\mathrm{T}}={\displaystyle \frac{1}{\mathrm{L}}}{\displaystyle \sum _{\mathrm{l}=1}^{\mathrm{L}}}{\widehat{\mathrm{h}}}_{\mathrm{l}}^{\mathrm{T}}\in {\mathrm{R}}^{{\mathrm{d}}_{\mathrm{h}}}$$

(20)

$${\overline{\mathrm{h}}}^{\mathrm{I}}={\displaystyle \frac{1}{\mathrm{N}}}{\displaystyle \sum _{\mathrm{n}=1}^{\mathrm{N}}}{\widehat{\mathrm{h}}}_{\mathrm{n}}^{\mathrm{I}}\in {\mathrm{R}}^{{\mathrm{d}}_{\mathrm{h}}}$$

(21)

Then we concatenate them to form a joint vector:

$$\mathrm{z}=\left[{\overline{\mathrm{h}}}^{\mathrm{T}};{\overline{\mathrm{h}}}^{\mathrm{I}}\right]\in {\mathrm{R}}^{2{\mathrm{d}}_{\mathrm{h}}}$$

(22)

The gating network consists of two fully connected layers with ReLU activation, followed by a sigmoid layer to produce the text weight:

$${\mathrm{h}}_1=\mathrm{R}\mathrm{e}\mathrm{L}\mathrm{U}({\mathrm{W}}_1\mathrm{z}+{\mathrm{b}}_1),{\mathrm{W}}_1\in {\mathrm{R}}^{{\mathrm{d}}_{\mathrm{h}}\times 2{\mathrm{d}}_{\mathrm{h}}},{\mathrm{b}}_1\in {\mathrm{R}}^{{\mathrm{d}}_{\mathrm{h}}}$$

(23)

$${\mathrm{h}}_2=\mathrm{R}\mathrm{e}\mathrm{L}\mathrm{U}({\mathrm{W}}_2{\mathrm{h}}_1+{\mathrm{b}}_{2)},{\mathrm{W}}_2\in {\mathrm{R}}^{{\mathrm{d}}_{\mathrm{h}}\times {\mathrm{d}}_{\mathrm{h}}},{\mathrm{b}}_2\in {\mathrm{R}}^{{\mathrm{d}}_{\mathrm{h}}}$$

(24)

$${\mathrm{g}}_{\mathrm{t}}=\mathsf{\sigma }({\mathrm{W}}_3{\mathrm{h}}_2+{\mathrm{b}}_3){\mathrm{b}}_{2)},{\mathrm{W}}_3\in {\mathrm{R}}^{1\times {\mathrm{d}}_{\mathrm{h}}},{\mathrm{b}}_3\in \mathrm{R}$$

(25)

$${\mathrm{g}}_{\mathrm{i}}={1-\mathrm{g}}_{\mathrm{t}}$$

(26)

This design forms a closed learning loop from cross-modal attention computation to feature weight assignment, enabling the model to adaptively balance modality contributions based on the semantic content of each post.

5.3.3. Weighted Fusion

Using the adaptive weights, we compute the modality-weighted representations:

$${\tilde{\mathrm{H}}}_{\mathrm{w}\mathrm{e}\mathrm{i}\mathrm{g}\mathrm{h}\mathrm{t}}^{\mathrm{T}}={\mathrm{g}}_{\mathrm{t}}\bullet {\tilde{\mathrm{H}}}^{\mathrm{T}},{\tilde{\mathrm{H}}}_{\mathrm{w}\mathrm{e}\mathrm{i}\mathrm{g}\mathrm{h}\mathrm{t}}^{\mathrm{I}}={\mathrm{g}}_{\mathrm{i}}\bullet {\tilde{\mathrm{H}}}^{\mathrm{I}}$$

(27)

where ${\tilde{\mathrm{H}}}^{\mathrm{T}}$ and ${\tilde{\mathrm{H}}}^{\mathrm{I}}$ are the outputs of the second-stage attention (Equations 21 and 23). These weighted features are then fused via concatenation:

$${\mathrm{F}}_{\mathrm{f}\mathrm{u}\mathrm{s}\mathrm{e}\mathrm{d}}=\left[{\tilde{\mathrm{H}}}_{\mathrm{w}\mathrm{e}\mathrm{i}\mathrm{g}\mathrm{h}\mathrm{t}}^{\mathrm{T}};{\tilde{\mathrm{H}}}_{\mathrm{w}\mathrm{e}\mathrm{i}\mathrm{g}\mathrm{h}\mathrm{t}}^{\mathrm{I}}\right]\in {\mathrm{R}}^{(\mathrm{L}+\mathrm{N})\times {\mathrm{d}}_{\mathrm{h}}}$$

(28)

5.4. Residual Fusion and Classification

5.4.1. Residual-Enhanced Feature Fusion

To optimize gradient flow and reduce information loss during feature transmission, we introduce residual connections. The fused features are passed through a feed-forward network with a skip connection:

$${\mathrm{H}}^{\mathrm{r}\mathrm{e}\mathrm{s}}=\mathrm{L}\mathrm{a}\mathrm{y}\mathrm{e}\mathrm{r}\mathrm{N}\mathrm{o}\mathrm{r}\mathrm{m}\left({\mathrm{F}}_{\mathrm{f}\mathrm{u}\mathrm{s}\mathrm{e}\mathrm{d}}+\mathrm{F}\mathrm{F}\mathrm{N}\left({\mathrm{F}}_{\mathrm{f}\mathrm{u}\mathrm{s}\mathrm{e}\mathrm{d}}\right)\right)\in {\mathrm{R}}^{\left(\mathrm{L}+\mathrm{N}\right)\times {\mathrm{d}}_{\mathrm{h}}}$$

(29)

This residual design preserves original information while enhancing the expressive capacity of cross-modal features, mitigating overfitting and improving model generalization.

5.4.2. Self-Attentive Pooling

We employ self-attentive pooling to aggregate the sequence of fused features into a single vector:

$$\mathsf{\alpha }=\mathrm{s}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}\left({\mathrm{H}}^{\mathrm{r}\mathrm{e}\mathrm{s}}{\mathrm{W}}_{\mathrm{a}}\right)\in {\mathbb{R}}^{\mathrm{L}+\mathrm{N}},{\mathrm{W}}_{\mathrm{a}}\in {\mathrm{R}}^{{\mathrm{d}}_{\mathrm{h}}\times 1}$$

(30)

$${\mathrm{h}}_{\mathrm{p}\mathrm{o}\mathrm{o}\mathrm{l}}={\displaystyle \sum _{\mathrm{i}=1}^{\mathrm{L}+\mathrm{N}}}{\mathsf{\alpha }}_{\mathrm{i}}{\mathrm{h}}_{\mathrm{i}}^{\mathrm{r}\mathrm{e}\mathrm{s}}\in {\mathrm{R}}^{{\mathrm{d}}_{\mathrm{h}}}$$

(31)

where ${\mathrm{h}}_{\mathrm{i}}^{\mathrm{r}\mathrm{e}\mathrm{s}}$ denotes the $\mathrm{i}$-th row of ${\mathrm{H}}^{\mathrm{r}\mathrm{e}\mathrm{s}}$.

5.4.3. Sentiment Classification

The pooled features are passed through a fully connected layer with ReLU activation, followed by Dropout for regularization:

$${\mathrm{h}}_{\mathrm{h}\mathrm{i}\mathrm{d}\mathrm{d}\mathrm{e}\mathrm{n}}=\mathrm{R}\mathrm{e}\mathrm{L}\mathrm{U}\left({\mathrm{W}}_{\mathrm{f}\mathrm{c}1}{\mathrm{h}}_{\mathrm{p}\mathrm{o}\mathrm{o}\mathrm{l}}+{\mathrm{b}}_{\mathrm{f}\mathrm{c}1}\right),{\mathrm{W}}_{\mathrm{f}\mathrm{c}1}\in {\mathrm{R}}^{{\mathrm{d}}_{\mathrm{h}}\times {\mathrm{d}}_{\mathrm{h}}},{\mathrm{b}}_{\mathrm{f}\mathrm{c}1}\in {\mathrm{R}}^{{\mathrm{d}}_{\mathrm{h}}}$$

(30)

$${\mathrm{h}}_{\mathrm{d}\mathrm{r}\mathrm{o}\mathrm{p}}=\mathrm{D}\mathrm{r}\mathrm{o}\mathrm{p}\mathrm{o}\mathrm{u}\mathrm{t}\left({\mathrm{h}}_{\mathrm{h}\mathrm{i}\mathrm{d}\mathrm{d}\mathrm{e}\mathrm{n}}\right)$$

(31)

The final classification layer uses softmax to output probabilities for three sentiment categories (negative, neutral, positive):

$$\mathrm{p}\left(\mathrm{y}\left|\mathrm{S}\right.\right)=\mathrm{s}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}\left({\mathrm{W}}_{\mathrm{c}\mathrm{l}\mathrm{s}}{\mathrm{h}}_{\mathrm{d}\mathrm{r}\mathrm{o}\mathrm{p}}+{\mathrm{b}}_{\mathrm{c}\mathrm{l}\mathrm{s}}\right),{\mathrm{W}}_{\mathrm{c}\mathrm{l}\mathrm{s}}\in {\mathrm{R}}^{3\times {\mathrm{d}}_{\mathrm{h}}},{\mathrm{b}}_{\mathrm{c}\mathrm{l}\mathrm{s}}\in {\mathrm{R}}^3$$

(32)

5.5. Loss Function and Optimization

5.5.1. Multi-Task Loss

We employ multi-task learning with three loss components.

Classification Loss (cross-entropy):

$${\mathcal{L}}_{\mathrm{c}\mathrm{l}\mathrm{s}}=-{\displaystyle \frac{1}{\mathrm{B}}}{\displaystyle \sum _{\mathrm{i}=1}^{\mathrm{B}}}{\displaystyle \sum _{\mathrm{c}=1}^{\mathrm{C}}}{\mathrm{y}}_{\mathrm{i},\mathrm{c}}\mathrm{l}\mathrm{o}\mathrm{g}{\mathrm{p}}_{\mathrm{i},\mathrm{c}}$$

(33)

where $\mathrm{B}$ is the batch size, ${\mathrm{y}}_{\mathrm{i},\mathrm{c}}$ is the ground-truth one-hot indicator, and ${\mathrm{p}}_{\mathrm{i},\mathrm{c}}$ is the predicted probability for class $\mathrm{C}$.

Contrastive Learning Loss to enhance representation learning:

$${\mathcal{L}}_{\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{t}}=-{\displaystyle \frac{1}{\mathrm{B}}}{\displaystyle \sum _{\mathrm{i}=1}^{\mathrm{B}}}\mathrm{l}\mathrm{o}\mathrm{g}{\displaystyle \frac{\mathrm{e}\mathrm{x}\mathrm{p}\left(\mathrm{s}\mathrm{i}\mathrm{m}\right({\mathrm{z}}_{\mathrm{i}},{\mathrm{z}}_{\mathrm{i}}^{+})/\mathsf{\tau }}{{\displaystyle \sum _{\mathrm{j}=1}^{\mathrm{B}}}\mathrm{e}\mathrm{x}\mathrm{p}\left(\mathrm{s}\mathrm{i}\mathrm{m}\right({\mathrm{z}}_{\mathrm{i}},{\mathrm{z}}_{\mathrm{j}}^{+})/\mathsf{\tau }}}$$

(34)

where ${\mathrm{z}}_{\mathrm{i}}={\mathrm{h}}_{\mathrm{p}\mathrm{o}\mathrm{o}\mathrm{l},\mathrm{i}}$ (the pooled feature for sample $\mathrm{i}$), ${\mathrm{z}}_{\mathrm{i}}^{+}$ is a positive sample obtained through data augmentation (e.g., random masking of text or image patches), $\mathrm{s}\mathrm{i}\mathrm{m}(\bullet ,\bullet )$ denotes cosine similarity, and $\mathsf{\tau }$ is a temperature parameter.

Causal Regularization Loss to encourage causally stable representations:

$${\mathcal{L}}_{\mathrm{c}\mathrm{a}\mathrm{u}\mathrm{s}\mathrm{a}\mathrm{l}}=\sum _{\mathrm{m}\in \left\{\mathrm{T},\mathrm{I}\right\}}{‖{\nabla }_{{\tilde{\mathrm{H}}}^{\mathrm{m}}}{\mathcal{L}}_{\mathrm{c}\mathrm{l}\mathrm{s}}‖}_{\mathrm{F}}^2$$

(35)

where ${\nabla }_{{\tilde{\mathrm{H}}}^{\mathrm{m}}}{\mathcal{L}}_{\mathrm{c}\mathrm{l}\mathrm{s}}$ is the gradient of the classification loss with respect to the modality-specific representations, and ${‖\bullet ‖}_{\mathrm{F}}$ denotes the Frobenius norm. This term penalizes sensitivity to small perturbations, promoting features that capture causal factors rather than spurious correlations.

The total loss is:

$$\mathcal{L}={\mathcal{L}}_{\mathrm{c}\mathrm{l}\mathrm{s}}+{\mathsf{\lambda }}_1{\mathcal{L}}_{\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{t}}+{\mathsf{\lambda }}_2{\mathcal{L}}_{\mathrm{c}\mathrm{a}\mathrm{u}\mathrm{s}\mathrm{a}\mathrm{l}}$$

(36)

where ${\mathsf{\lambda }}_1$ and ${\mathsf{\lambda }}_2$ are hyperparameters balancing the contributions.

5.5.2. Optimization

We optimize the model using AdamW with weight decay, which prevents excessive weights and overfitting better than traditional Adam. The learning rate is set to $5\times {10}^{-5}$, and the model is updated online as new data streams arrive (batch size = 1, update every 100 samples). All weight matrices are initialized using Xavier uniform initialization, and biases are initialized to zero.

6. Experiments and Results Analysis

6.1. Experimental Data

(1) Dataset Collection

In China, Weibo and Xiaohongshu have emerged as pivotal platforms for public opinion and sentiment expression. Owing to their extensive user bases, image content related to public health emergencies on these platforms typically encapsulates richer information, aggregating substantial volumes of authentic user perspectives and emotional responses, while encompassing diverse stances, viewpoints, and affective tendencies. Consequently, this study utilizes data sourced from Weibo and Xiaohongshu to validate the effectiveness of the proposed method.

By crawling public posts from Weibo and Xiaohongshu between January 2020 and December 2022, this study collects public opinion data from three public health emergencies on the Weibo and Xiaohongshu platforms, comprising 15,677 text-image pairs. The data distribution across the events is as follows: 10,245 pairs related to COVID-19, 3,210 pairs to Avian Flu, and 2,222 pairs to Monkeypox.

(2) Data Annotation

The annotation of the dataset involved a structured dual-process approach, combining both manual and semi-automated methods tailored to the data modality. For image sentiment ,a manual dual-annotator protocol was employed. Each image was independently labeled as “Negative,” “Neutral,” or “Positive” by two annotators. Inter-annotator agreement was measured using Cohen’s Kappa coefficient (0.7123), indicating substantial consistency. Any discordant labels were resolved through subsequent discussion between the annotators to reach a consensus. For text sentiment annotation, a hybrid semi-automated pipeline was implemented. Initial sentiment labels (“Negative,” “Neutral,” “Positive”) were generated automatically using the SnowNLP toolkit. These automated labels were then validated by two independent human annotators. Instances where the manual annotations disagreed with the automated output or with each other were flagged. These discrepancies were subsequently reviewed and adjudicated through manual re-evaluation to ensure final label accuracy.The finalized sentiment distribution across the three public health event datasets is presented in Table 1.

(3) Experimental Environment and Evaluation MetricsThe experiments were conducted using the PyCharm IDE with Python 3.9. All models were trained on an NVIDIA RTX 3060 GPU with 32 GB of dedicated memory in Table 2. The AdamW optimizer was employed for training across all experiments, with an initial learning rate of 5×10⁻⁵. To ensure fair comparisons and mitigate confounding variables, identical training and testing splits were used for all models. Specifically, the dataset was partitioned into 80% for training and 20% for testing.

This study uses four standard evaluation metrics: Accuracy (A), Precision (P), Recall (R), and F1-score (F1).

6.2. Main Results and Analysis

6.2.1. The Classification Results

The model proposed in this paper was applied to three public health emergencies, and the classification results are shown in Table 3.

Based on the sentiment classification results of three public health events—COVID-19, Avian Influenza, and Monkeypox—the sentiment analysis model demonstrates strong overall performance, with an average accuracy of 87.08% and a macro-average F1 score of 87.24%, indicating good generalization ability and practical application potential. In terms of sentiment-specific recognition, the model performs best in identifying negative sentiments, with F1 scores for the negative class exceeding 87.75% across all three events, reaching as high as 91.38% for Monkeypox. This highlights the model’s clear advantage in capturing negative public opinion during crisis events. Positive sentiment recognition is also stable, with precision consistently above 91% and a low false positive rate. However, the model shows a notable weakness in recognizing neutral sentiment, with F1 scores for the neutral class falling below 85.5% across all events—reaching as low as 82.12%—and generally low recall, reflecting difficulty in accurately identifying texts with weak or ambiguous emotional polarity. Overall, the model is ready for practical deployment, particularly for negative public opinion monitoring in public health contexts.

6.2.2. Baseline Models

we organize the baselines into three categories—online feature selection, text-only, and multimodal fusion approaches—and further design combined experiments that pair different feature selection and fusion strategies to isolate the contribution of each component.①Online Feature Selection Baselines: Online Group LASSO (OGL): Regularization parameter $\mathsf{\lambda }=0.01$, group structure defined by modality. Alpha-investing (AI): Initial alpha budget ${\mathsf{\alpha }}_0=0.05$, $\mathsf{\omega }=1.0$. ②Text-Only Baselines: Online SVM (Text):Based on TF-IDF features, RBF kernel, $\mathrm{C}=1.0$, online learning rate $\mathsf{\eta }=0.01$. Online BERT: Based on DistilBERT-base, hidden layer dimension 768, learning rate 2e-5, updated every 100 samples. ③Multimodal Fusion Baselines: Early Fusion (EF): Text and image features concatenated at the input layer (text 768-dim + image 512-dim = 1280-dim). Late Fusion (LF):Text and image predictions are combined using a weighted average, with weights determined on the validation set. Cross-Modal Attention (CMA): Number of attention heads = 8, hidden dimension = 256. MARN: Memory unit size = 128, number of memory slots = 16. CLIP (finetuned): We employ the pretrained CLIP (ViT-B/32) model, which consists of a Vision Transformer for images and a Transformer for text. The image and text features are concatenated and fed into a linear classifier for sentiment prediction. Fine-tuning follows the same online learning protocol (batch size = 1, update every 100 samples, learning rate = 2e-5) to ensure fair comparison.

Comparative Experimental Design: The proposed O-DC method is combined with different fusion approaches: O-DC+EF; O-DC+LF; O-DC+CMA. Our fusion network is combined with different feature selection methods: OGL+Our Fusion; AI+Our Fusion.

6.2.3. Results and Discussion

To comprehensively evaluate the effectiveness of the proposed online dynamic cross-modal fusion framework (O-DC + Our Fusion), we compare it against a diverse set of baseline models, including text-only classifiers, multimodal fusion architectures, and online feature selection methods. As reported in Table 4, all models are evaluated on the test stream in terms of Precision (P), Recall (R), Macro-F1, and Accuracy (A).

Among the text-only baselines, Online BERT significantly outperforms Online SVM, achieving a 5.7% higher F1-Score 6.5% vs. 70.8%) and a 5.7% improvement in accuracy (78.2% vs. 72.5%). This demonstrates the superiority of pre-trained language models in capturing semantic nuances from textual content in public health emergency events. However, both text-only models exhibit relatively low recall (below 67%), indicating a tendency to miss a substantial portion of relevant samples, particularly in imbalanced streaming settings.

Incorporating visual information through multimodal fusion leads to consistent performance gains. For instance, EF + OGL improves recall to 79.2% and F1-Score to 78.9%, suggesting that early integration of visual features helps recover false negatives. The Cross-Modal Attention (CMA) model further boosts performance, achieving 81.3% recall and 80.8% F1-Score, which underscores the importance of modality interaction mechanisms. The MARN model, equipped with memory-augmented networks, achieves the highest recall among all baselines (91.3%) and an accuracy of 83.1%, validating the effectiveness of modeling temporal dependencies in online multimodal streams.

The inclusion of a CLIP-based vision-language pretrained model (CLIP finetuned) provides a strong contemporary baseline. CLIP achieves 88.1% precision, 83.9% recall, 85.9% F1-Score, and 86.2% accuracy, outperforming all traditional multimodal fusion methods and text-only models. This result confirms that leveraging large-scale pretraining on image-text pairs yields powerful multimodal representations even in specialized domains like PHE sentiment analysis. However, CLIP still exhibits a precision-recall trade-off: its recall (83.9%) is lower than MARN’s (91.3%), indicating that some sentiment signals are missed, while its precision (88.1%) is higher than MARN’s (82.7%), reflecting fewer false positives.

Despite these advances, baseline methods still face trade-offs between precision and recall. The proposed O-DC+CMA combination partially mitigates this issue, achieving a balanced performance with 85.6% precision and 83.2% F1-Score. Nevertheless, it still lags behind in recall (71.8%), suggesting that attention-based cross-modal interaction alone is insufficient for comprehensive feature selection in dynamic environments.

Remarkably, the full proposed model (O-DC + Our Fusion) achieves the best overall performance across all metrics: 89.3% precision, 85.3% recall, 87.2% F1-Score, and 87.1% accuracy. Compared to the strongest baseline MARN, our method improves F1-Score by 5.6% and accuracy by 4.0%, while maintaining a superior balance between precision and recall. More importantly, compared to the state-of-the-art pretrained model CLIP, our method yields gains of +1.2% in precision, +1.4% in recall, +1.3% in F1-Score, and +0.9% in accuracy. This demonstrates that the integration of online dynamic selection (M-O-DC) with our specially designed two-stage interactive fusion network adds significant value over simply fine-tuning a generic vision-language model. The improvement can be attributed to two factors: (1) M-O-DC adaptively filters irrelevant and redundant features in the streaming setting, reducing noise and focusing on task-relevant information; (2) the two-stage cross-modal attention with adaptive weighting captures fine-grained semantic alignments and dynamically balances modality contributions, which is crucial for PHE posts where modality importance varies greatly.

The online streaming performance evolution (Figure 1) further illustrates the robustness of our method. Prior to the simulated concept drift at batch 20, our model exhibits the fastest growth and reaches the highest F1-Score plateau. Upon drift, both MARN and O-DC+CMA experience noticeable degradation, while CLIP (not shown in Figure 1 due to its offline nature) would likely suffer similarly if evaluated online. In contrast, our proposed method maintains its performance with only a marginal dip and quickly recovers, showcasing its resilience to distributional shifts. This resilience is attributed to the online dynamic selection mechanism, which adaptively identifies and emphasizes informative features while discarding noisy or outdated ones.

The feature dimension dynamics chart compares the number of selected features over 50 batches among three online feature selection methods integrated with our fusion network. The proposed O-DC+Our Fusion method demonstrates a rapid decline in feature dimensions from approximately 900 to around 380 within the first 20 batches, after which it stabilizes at a low and consistent level in Figure 2. This indicates its ability to quickly identify and retain only the most informative features while discarding redundant or noisy ones. In contrast, OGL+Our Fusion exhibits larger fluctuations and maintains a higher average dimension (600), reflecting its group-wise sparsity constraints that are less adaptive to streaming dynamics. AI+Our Fusion initially selects a large number of features (1000) but gradually reduces to around 480, yet with noticeable variance. The efficiency advantage of the proposed method is evident: it achieves the highest F1-Score (87.2%) while using the fewest features, confirming that its online dynamic selection mechanism not only reduces computational overhead but also enhances feature quality, leading to better generalization in evolving data streams.

Figure 3 presents a grouped bar chart comparing the performance of eight models across Precision, Recall, F1-Score, and Accuracy. Several observations can be drawn. First, text-only baselines (Online SVM and Online BERT) exhibit the lowest overall performance, particularly in Recall (below 67%), indicating their limitation in capturing sufficient positive samples in streaming scenarios. Second, multimodal fusion methods (EF+OGL, CMA, MARN) significantly improve Recall, with MARN achieving the highest Recall (91.3%), but at the cost of reduced Precision (82.7%), revealing a precision-recall trade-off. Third, the CLIP (finetuned) model, leveraging large-scale vision-language pretraining, achieves strong performance (Precision 88.1%, F1-Score 85.9%, Accuracy 86.2%), outperforming traditional multimodal methods but still exhibiting a slight imbalance between precision and recall. Fourth, the O-DC+CMA combination improves Precision to 85.6% but suffers a sharp Recall drop to 71.8%, suggesting that attention-based cross-modal interaction alone cannot fully balance the two metrics. Most importantly, the proposed O-DC+Our Fusion model achieves the best overall performance, leading in Precision (89.3%), F1-Score (87.2%), and Accuracy (87.1%), while maintaining competitive Recall (85.3%). This balanced superiority demonstrates that the integration of online dynamic selection with our fusion network effectively resolves the precision-recall dilemma and outperforms even strong pretrained models like CLIP in the streaming multimodal setting.

6.2.4. Ablation Study Analysis

To rigorously evaluate the contribution of each key component in our proposed online dynamic cross-modal fusion framework, we conduct a series of ablation experiments by systematically removing individual modules. The results are summarized in Table 5, where the Full Model achieves the best performance with 86.9% accuracy and 85.4% F1-Score, serving as the baseline for comparison. Each variant is described below, along with the corresponding performance degradation and efficiency impact.

w/o O-DC (using full features): This variant removes the online dynamic selection (O-DC) module and instead utilizes all original features (text 768-d + image 512-d) without any dimensionality reduction. Compared to the Full Model, accuracy drops by 2.8% to 84.1%, and F1-Score decreases by 2.7% to 82.7%. Notably, the processing time increases to 82 ms, which is 14 ms longer than the Full Model. This indicates that O-DC not only preserves discriminative information but also reduces computational overhead by filtering out redundant features, thereby enhancing both effectiveness and efficiency.

w/o 2nd-stage Attention: This variant eliminates the second-stage cross-modal attention mechanism, which is designed to refine feature interactions after initial fusion. The removal leads to an accuracy decline of 1.3% (85.6%) and a F1-Score drop of 1.4% (84.0%). The processing time slightly decreases to 64 ms, suggesting that the second-stage attention adds a modest computational cost while delivering noticeable performance gains. The results confirm that deeper attention-based refinement is beneficial for capturing complex modality relationships.

w/o Adaptive Weighting: Here, the adaptive weighting module that dynamically adjusts the contribution of each modality based on streaming context is removed. Accuracy falls to 85.9% (Δ = -1.0%), and F1-Score to 84.3% (Δ = -1.1%), with a processing time of 66 ms. The moderate performance drop underscores the importance of context-aware modality fusion, especially in evolving data streams where the reliability of text and visual signals may vary over time.

w/o Residual: This variant removes residual connections within the fusion network. Accuracy decreases by 0.8% to 86.1%, and F1-Score by 0.7% to 84.7%, while processing time remains similar to the Full Model (67 ms vs. 68 ms). Although the impact is relatively smaller, residual connections still contribute to stabilizing training and improving representational capacity, as evidenced by the consistent performance gain.

The ablation results clearly demonstrate that each module plays a vital role in the overall framework. The most significant performance degradation occurs when O-DC is removed, highlighting its critical function in online feature selection. The second-stage attention and adaptive weighting also contribute substantially to accuracy and F1-Score, while residual connections provide marginal yet consistent improvements. Moreover, the Full Model achieves the best trade-off between performance and efficiency, with processing time only slightly higher than the fastest variant but substantially lower than the feature-heavy w/o O-DC version. These findings validate the necessity and effectiveness of each designed component in our proposed method for online multimodal sentiment classification.

7. Conclusion

This study presented a novel online multimodal emotion recognition framework for public health emergency public opinion analysis. To address the challenges of streaming, high-dimensional, and heterogeneous data, we proposed: (1) an Online Divide-and-Conquer Markov Blanket Learning Algorithm for dynamic and efficient feature selection, and (2) a Cross-Modal Interactive Enhanced Fusion Network with two-stage attention and adaptive weighting for deep semantic fusion. Theoretical analysis provided guarantees on convergence, information preservation, and generalization. Comprehensive experiments on a large-scale PHE dataset demonstrated that our framework significantly outperforms state-of-the-art baselines in accuracy, F1-score, online learning efficiency, and robustness to concept drift, while maintaining interpretability.

Future work will focus on: (1) Extending the framework to incorporate more modalities, particularly video and audio; (2) Designing fully adaptive mechanisms for threshold parameters in O-DC; (3) Exploring more advanced architectures for handling complex, long-term concept drift.