Adaptive Latent Interaction Reasoning for Multimodal Misinformation Analysis

1. Introduction

The unprecedented proliferation of Online Social Networks (OSNs), such as Twitter and Weibo, has profoundly reshaped the modern information ecosystem. These platforms enable users to rapidly share news, opinions, and multimedia content, significantly lowering the barrier for information dissemination. While this democratization of content creation brings undeniable benefits, it also introduces serious challenges, among which the large-scale spread of fake news has become a pressing societal concern [31]. Misinformation disseminated through social media has been shown to influence public opinion, undermine trust in institutions, and even cause tangible harm in domains such as public health and political stability. Consequently, the development of reliable automatic fake news detection systems has attracted substantial attention from both academia and industry.

Early research efforts on fake news detection primarily focused on analyzing textual content alone. Traditional machine learning methods, including decision tree classifiers and handcrafted linguistic features, were initially explored to distinguish fake news from genuine content [40]. With the advent of deep learning, neural architectures such as convolutional neural networks (CNNs) and recurrent models have been employed to capture more complex semantic patterns in text [47]. Although these approaches have demonstrated promising performance, they fundamentally overlook the multimodal nature of social media posts, where textual descriptions are frequently accompanied by images or other visual elements. Relying solely on textual signals is therefore insufficient for fully understanding and verifying the credibility of online information.

Figure 1. Motivation of ALIGNER: addressing ambiguous image–text associations through adaptive alignment and guided multimodal reasoning.

Figure 1. Motivation of ALIGNER: addressing ambiguous image–text associations through adaptive alignment and guided multimodal reasoning.

Recognizing this limitation, recent studies have shifted towards multimodal fake news detection, aiming to jointly exploit textual and visual information [36,44]. By fusing representations extracted from multiple modalities, these methods seek to construct more informative and discriminative post-level representations. Despite their progress, many existing approaches adopt relatively coarse fusion strategies, such as feature concatenation or shallow attention mechanisms, without explicitly modeling the semantic alignment between modalities. As a result, the learned multimodal representations may fail to capture subtle inconsistencies or complementary cues that are crucial for fake news identification. More importantly, not all modalities contribute equally in every scenario. In some cases, textual content alone may be sufficiently implausible to indicate misinformation, whereas in other situations, the discrepancy between visual and textual information plays a decisive role in exposing fake news [6,24]. This observation highlights a fundamental challenge: multimodal fake news detection should not merely aggregate information from different sources, but should also reason about when and how each modality influences the final decision. Understanding the varying roles of unimodal and cross-modal features, as well as making this decision process interpretable, remains an open research problem.

In parallel, contrastive learning has emerged as a powerful paradigm for representation learning across modalities. A series of contrastive vision–language pre-training frameworks have demonstrated remarkable success by aligning image and text representations in a shared embedding space [1,12,16,23,38,39,48]. By pulling together representations of matched image–text pairs while pushing apart mismatched ones, contrastive objectives encourage encoders to capture high-level semantic correspondence. These advances suggest that contrastive learning could serve as an effective foundation for multimodal fake news detection.

However, directly applying standard contrastive learning objectives to fake news detection is non-trivial. Existing contrastive frameworks typically rely on one-hot supervision, treating all unmatched image–text pairs as equally negative [10,39]. This assumption is often violated in real-world fake news datasets, where image–text associations can be weak, ambiguous, or even intentionally misleading. Furthermore, different multimodal posts discussing the same event may share partial semantic overlap, yet are still considered negative pairs under conventional contrastive learning. Such rigid supervision may lead to over-penalization of semantically related samples and ultimately harm generalization performance.

To address these challenges, we propose ALIGNER, an adaptive latent interaction guided contrastive reasoning framework for multimodal fake news detection. ALIGNER adopts a dual-encoder architecture to separately model visual and linguistic semantics, establishing two complementary semantic spaces. A cross-modal contrastive objective is employed to encourage alignment between modalities, ensuring that corresponding image and text representations are semantically coherent. Beyond this basic alignment, we introduce an auxiliary latent consistency learning task that provides soft supervision over negative samples. This task captures implicit semantic relatedness among multimodal instances, thereby alleviating the overly strict constraints imposed by one-hot contrastive labels.

Following representation alignment, ALIGNER further incorporates a cross-modal interaction module to explicitly model correlations between visual and textual features. Rather than treating cross-modal information as uniformly beneficial, we design an attention-based aggregation mechanism that adaptively weighs unimodal and cross-modal signals. Inspired by prior work [6], we integrate an attention guidance strategy that quantifies modality ambiguity by estimating the divergence between visual and textual representations. This guidance signal enables the model to dynamically adjust modality importance, leading to more robust and interpretable decision making.

In summary, ALIGNER advances multimodal fake news detection by unifying adaptive contrastive alignment, latent semantic consistency modeling, and guided cross-modal reasoning within a single framework. Through this design, the proposed approach is able to learn more reliable multimodal representations and effectively leverage them for misinformation analysis. Extensive experimental evaluations on the Twitter and Weibo datasets validate the effectiveness of ALIGNER, demonstrating consistent improvements over existing state-of-the-art methods.

The main contributions of this paper are summarized as follows:

• We propose ALIGNER, an adaptive contrastive reasoning framework that explicitly addresses the challenges of multimodal fake news detection.
• We introduce a latent consistency learning objective to soften the supervision over negative samples in cross-modal contrastive learning.
• We design a guided attention mechanism to adaptively and interpretably aggregate unimodal and cross-modal features.
• Extensive experiments on Twitter and Weibo datasets demonstrate that ALIGNER achieves superior performance compared to prior state-of-the-art approaches.

2. Related Works

2.1. Fake News Detection

Early studies on fake news detection predominantly focused on textual content, motivated by the observation that linguistic signals often carry explicit or implicit cues indicative of misinformation. Representative approaches analyze the semantic coherence, lexical choices, and discourse patterns present in news posts. For instance, [22] propose a neural generative framework that models historical user responses to uncover latent propagation patterns, thereby assisting the detection of fake news through indirect semantic supervision. Such methods highlight the importance of contextual language modeling beyond surface-level word statistics. In addition to generative approaches, a large body of work explores explainable and interpretable textual features. TM [2] leverages lexical, syntactic, and semantic properties of text to improve detection accuracy while providing insights into model decisions. Other studies investigate logical consistency [11], writing style regularities [20], and rhetorical structures [7], under the assumption that deceptive content exhibits distinctive linguistic patterns. While effective in controlled settings, these approaches remain limited by their exclusive reliance on textual modality, which constrains their robustness in multimodal social media environments.

Beyond text, visual information has also been explored as a standalone signal for fake news detection, particularly in image-rich social media platforms. Early work such as [18] demonstrates that fake and real news often exhibit different visual dissemination behaviors, including reposting frequency and temporal diffusion patterns. These findings suggest that images are not merely auxiliary decorations but can serve as critical forensic evidence. Subsequent studies further examine intrinsic visual characteristics. MVNN [21] exploits both spatial-domain visual features and frequency-domain signals to capture subtle manipulation traces, offering a more comprehensive forensic perspective. Despite their effectiveness in identifying visually manipulated or misleading images, purely visual approaches struggle when images are reused legitimately or lack explicit manipulation cues. More importantly, these methods disregard semantic interactions between images and accompanying text, which are often crucial for understanding misinformation intent.

Although unimodal approaches have laid the foundation for automatic fake news detection, their inherent limitations become increasingly evident in real-world multimodal scenarios. Social media posts frequently combine text and images in complex ways, where neither modality alone provides sufficient evidence for accurate classification. Unimodal models fail to capture semantic contradictions, complementarities, or contextual dependencies across modalities, leading to ambiguous or incorrect predictions. Furthermore, unimodal systems are particularly vulnerable to adversarial manipulation, where misinformation creators exploit modality gaps to evade detection. For instance, benign-looking images may be paired with deceptive textual claims, or plausible text may be accompanied by irrelevant visuals. These challenges motivate the need for multimodal reasoning frameworks that can jointly analyze and interpret heterogeneous signals rather than treating them in isolation.

The shortcomings of unimodal methods have driven a paradigm shift toward multimodal fake news detection, where the goal is not only to aggregate multiple information sources but also to reason about their relationships. Effective multimodal detection requires modeling both intra-modal semantics and inter-modal consistency, enabling systems to identify discrepancies that are otherwise imperceptible within a single modality. This observation forms the foundation for more advanced multimodal approaches, which we discuss next. Recent advances in multimodal fake news detection emphasize the importance of learning discriminative cross-modal patterns. EANN [42] introduces an event discriminator to encourage event-invariant feature learning, thereby enhancing generalization across diverse news topics. MVAE [36] adopts a multimodal variational autoencoder to jointly model textual and visual latent variables and reconstruct both modalities, capturing shared semantic representations.

Building upon attention-based fusion strategies, MCAN [44] stacks multiple co-attention layers to enable fine-grained interactions between text and image features. These methods demonstrate that explicit cross-modal modeling significantly improves detection performance compared to unimodal baselines. However, most fusion-based approaches rely on task-specific architectures and lack principled representation learning objectives that generalize beyond supervised settings. An emerging line of work recognizes that modalities do not contribute equally to fake news detection across instances. CAFE [6] quantifies cross-modal ambiguity by computing the Kullback-Leibler (KL) divergence between unimodal feature distributions and uses this score to dynamically reweight modality contributions. This strategy explicitly accounts for modality disagreement and improves interpretability. Similarly, LIIMR [24] identifies the modality that provides stronger predictive confidence and prioritizes it during decision making. These approaches move beyond naive fusion by incorporating modality-aware reasoning. In contrast, ALIGNER further integrates modality importance estimation with contrastive alignment and latent semantic consistency, enabling adaptive and interpretable multimodal inference within a unified framework.

2.2. Contrastive Learning

Contrastive Learning in Vision and Language

Contrastive learning has emerged as a dominant paradigm for self-supervised and representation learning in both computer vision and natural language processing. In the vision domain, frameworks such as MoCo [12], SimCLR [4], and subsequent empirical studies [5] demonstrate that instance discrimination objectives can produce highly transferable visual representations. In parallel, contrastive objectives have been successfully adapted to NLP tasks, as evidenced by SimCSE [9] and ConSERT [46], which improve sentence embeddings through representation alignment. These successes highlight the generality of contrastive learning as a mechanism for capturing semantic similarity and structure without heavy reliance on manual annotation. The core idea of pulling semantically related samples closer while pushing unrelated ones apart provides a flexible foundation for multimodal extension.

Contrastive learning has been further extended to vision-language representation learning, leading to a series of influential multimodal pre-training models. WenLan [15] proposes a two-tower architecture tailored for Chinese multimodal data, adapting the MoCo framework to cross-modal settings. CLIP [23] and ALIGN [16] demonstrate that large-scale contrastive pre-training on noisy web data can yield powerful aligned representations, enabling zero-shot generalization across diverse tasks. Subsequent works such as ALBEF [39] introduce additional cross-attention fusion modules on top of contrastively aligned encoders, while incorporating hard negative mining strategies to enhance discrimination. Related approaches including BLIP [38] and VLMo [1] further refine these ideas, achieving strong performance across a wide range of multimodal benchmarks.

Hybrid Contrastive and Generative Objectives

Beyond purely discriminative objectives, recent models explore the combination of contrastive and generative learning. CoCa [48] unifies contrastive alignment with caption generation in a modified encoder-decoder architecture, demonstrating that hybrid objectives can improve representation quality and downstream adaptability. These advances suggest that contrastive learning benefits from auxiliary semantic supervision that goes beyond binary alignment signals. However, most existing vision-language contrastive frameworks assume clean, high-quality image-text pairs, an assumption that does not hold in multimodal fake news detection. Social media data often contains weakly related or intentionally misleading image-text associations, posing unique challenges for direct adoption of standard contrastive objectives.

In this work, ALIGNER builds upon the strengths of contrastive vision-language learning while explicitly addressing its limitations in fake news scenarios. We adopt an image-text contrastive (ITC) objective within a dual-encoder architecture to establish a shared latent embedding space for visual and textual representations. Unlike conventional approaches, ALIGNER incorporates an auxiliary cross-modal consistency learning task that estimates semantic similarity between images and texts and provides soft supervision signals for contrastive alignment.

By relaxing rigid one-hot negative assumptions and accounting for latent semantic relatedness, ALIGNER achieves more robust and task-compatible representation learning. This design enables the model to effectively leverage contrastive learning for multimodal fake news detection, bridging the gap between large-scale multimodal pre-training paradigms and real-world misinformation analysis.

Figure 2. ALIGNER framework overview. Paired image–text inputs are aligned via contrastive learning, interact explicitly across modalities, and are aggregated in an ambiguity-aware manner for fake news prediction.

Figure 2. ALIGNER framework overview. Paired image–text inputs are aligned via contrastive learning, interact explicitly across modalities, and are aggregated in an ambiguity-aware manner for fake news prediction.

3. Methodology

In this section, we present ALIGNER, a unified and adaptive multimodal fake news detection framework grounded in cross-modal contrastive representation learning and guided aggregation. ALIGNER is designed to explicitly address the inherent semantic misalignment, modality ambiguity, and weak supervision issues that commonly arise in real-world multimodal news data. Given paired image–text inputs, ALIGNER first extracts modality-specific representations using pretrained encoders, and then progressively refines these representations through contrastive alignment, latent semantic consistency modeling, cross-modal interaction learning, and ambiguity-aware aggregation. The overall learning objective jointly optimizes alignment quality, semantic robustness, and decision interpretability.

3.1. Multimodal Input Formalization and Encoder Architecture

Let each multimodal news instance be denoted as $\mathbf{x}=[x^v,x^t]\in \mathcal{D}$, where $x^v$ represents the visual content, $x^t$ denotes the textual content, and $\mathcal{D}$ is the training dataset. The core objective of ALIGNER is to learn a discriminative function $f:(x^v,x^t)\to y$, where $y\in \{0,1\}$ indicates whether the news is fake or real. Rather than directly operating on raw inputs, ALIGNER relies on pretrained modality-specific encoders to obtain high-quality unimodal representations, which serve as the foundation for subsequent cross-modal reasoning.

Visual Representation Learning

Given an image input $x^v$, we adopt ResNet [13] pretrained on ImageNet as the visual encoder due to its strong inductive bias for spatial feature extraction. The encoder maps $x^v$ into a set of region-level feature vectors, which are subsequently aggregated through global average pooling. A fully connected projection layer is applied to transform the pooled visual features into a compact embedding $e^v\in {\mathbb{R}}^d$. This projection not only aligns dimensionality with the textual embedding space but also serves as a learnable bottleneck that facilitates semantic abstraction.

Textual Representation Learning

For textual input $x^t$, we employ BERT [35] as the backbone language encoder to capture contextualized semantic representations. The input text is tokenized using a predefined vocabulary and passed through multiple Transformer layers. The resulting sequence-level representation is obtained by pooling token embeddings, followed by a fully connected transformation to produce the final textual embedding $e^t\in {\mathbb{R}}^d$. This design allows ALIGNER to capture both syntactic structure and high-level semantic cues that are critical for fake news detection.

3.2. Cross-modal Contrastive Alignment with Latent Consistency

Due to the substantial semantic gap between visual and textual modalities, directly fusing unimodal embeddings is insufficient for reliable multimodal reasoning. ALIGNER therefore adopts cross-modal contrastive learning as a principled mechanism to align heterogeneous representations within a shared latent space. However, unlike conventional contrastive frameworks that rely on rigid one-hot supervision, ALIGNER introduces an auxiliary consistency learning task to provide soft semantic guidance, thereby improving robustness in noisy multimodal settings.

Latent Cross-modal Consistency Learning

We formulate cross-modal consistency learning as a binary semantic matching task. Specifically, we construct an auxiliary dataset ${\mathcal{D}}^{\prime }=[{\mathcal{D}}_{\mathrm{pos}},{\mathcal{D}}_{\mathrm{neg}}]$, where each instance ${\mathbf{x}}^{\prime }=[x^{v^{\prime }},x^{t^{\prime }}]$ is labeled as $y^{\prime }=1$ if the image and text originate from the same real news, and $y^{\prime }=0$ otherwise. The corresponding unimodal embeddings $e^{v^{\prime }}$ and $e^{t^{\prime }}$ are projected into a shared semantic space using modality-specific multilayer perceptrons, yielding shared embeddings $e_s^{v^{\prime }}$ and $e_s^{t^{\prime }}$.

To explicitly enforce semantic consistency, we optimize the cosine embedding loss:

$${\mathcal{L}}_{ITM}=\left\{\begin{array}{cc}1-cos(e_s^{v^{\prime }},e_s^{t^{\prime }}),\hfill & y^{\prime }=1\hfill \\ max(0,cos(e_s^{v^{\prime }},e_s^{t^{\prime }})-d),\hfill & y^{\prime }=0\hfill \end{array}\right.$$

(1)

where $cos(·)$ denotes normalized cosine similarity and $d=0.2$ is a margin hyperparameter. This task encourages semantically aligned multimodal pairs to be close in latent space while preventing excessive separation of weakly related pairs.

Bidirectional Image–Text Contrastive Learning

Given a batch of N paired samples ${\left\{(x_i^v,x_i^t)\right\}}_{i=1}^N$, ALIGNER computes normalized embeddings ${\{e_i^v,e_i^t\}}_{i=1}^N$. The image–text contrastive objective aims to correctly identify true correspondences among all $N^2$ possible pairings. The predicted similarity distributions are defined as:

$$\begin{array}{cc}\hfill p_{ij}^{v\to t}& ={\displaystyle \frac{exp(sim(e_i^v,e_j^t)/\tau )}{{\sum }_{j=1}^{N}exp(sim(e_i^v,e_j^t)/\tau )}},\hfill \\ \hfill p_{ij}^{t\to v}& ={\displaystyle \frac{exp(sim(e_i^t,e_j^v)/\tau )}{{\sum }_{j=1}^{N}exp(sim(e_i^t,e_j^v)/\tau )}},\hfill \end{array}$$

(2)

where $\tau$ is a learnable temperature parameter. The standard image–text contrastive loss is then computed as:

$${\mathcal{L}}_{ITC}={\displaystyle \frac{1}{2}}\left({\mathcal{L}}^{v\to t}+{\mathcal{L}}^{t\to v}\right).$$

(3)

Soft Semantic Target Construction

To mitigate the overly strict nature of one-hot supervision, ALIGNER constructs soft semantic targets using the shared embeddings produced by the consistency module. The semantic similarity matrix is computed as:

$$s_{ij}^{v\to t}={\displaystyle \frac{exp(sim({\left(e_s^v\right)}_i,{\left(e_s^t\right)}_j)/\tau )}{{\sum }_{j=1}^{N}exp(sim({\left(e_s^v\right)}_i,{\left(e_s^t\right)}_j)/\tau )}}.$$

(4)

These soft targets encode latent semantic relatedness across samples and are used to supervise the contrastive predictions.

Algorithm 1: ALIGNER: Ambiguity-aware Cross-modal Contrastive Learning

Input: Multimodal news $\mathbf{x}=[x^v,x^t]$ Output: Prediction $\widehat{y}$   Stage 1: Modal-specific Encoding $e^v\leftarrow \mathrm{ResNet}\left(x^v\right)$;   $e^t\leftarrow \mathrm{BERT}\left(x^t\right)$ $m^v\leftarrow {\Pi }_v\left(e^v\right)$;   $m^t\leftarrow {\Pi }_t\left(e^t\right)$   Stage 2: Consistency-aware Contrastive Alignment Compute ITC similarities ${\mathbf{p}}^{v\to t},{\mathbf{p}}^{t\to v}$ Compute consistency loss ${\mathcal{L}}_{ITM}$ (Eq. 1) Build soft targets $\mathbf{s}$ and compute ${\mathcal{L}}_{CL}$ (Eq. 6)   Stage 3: Cross-modal Fusion $f_{t\to v},f_{v\to t}\leftarrow \mathrm{Attn}(m^v,m^t)$ $m^f\leftarrow \left(f_{t\to v}{m}^v\right)\otimes \left(f_{v\to t}{m}^t\right)$   Stage 4: Ambiguity-guided Aggregation Compute attention weights $\mathbf{a}=\{a_v,a_t,a_f\}$ Compute ambiguity score $\mathbf{g}$ and ${\mathcal{L}}_{AG}$ $\tilde{\mathbf{x}}\leftarrow a_v{m}^v\oplus a_t{m}^t\oplus a_f{m}^f$   Stage 5: Classification $\widehat{y}\leftarrow \mathrm{softmax}\left(\mathrm{MLP}\left(\tilde{\mathbf{x}}\right)\right)$ return$\widehat{y}$

Semantic Matching Regularization

The semantic matching loss is defined as the cross-entropy between predicted similarities and soft targets:

$${\mathcal{L}}_{SEM}=-{\displaystyle \frac{1}{2N}}\sum _{i=1}^N\sum _{j=1}^N\left(s_{ij}^{v\to t}log{p}_{ij}^{v\to t}+s_{ij}^{t\to v}log{p}_{ij}^{t\to v}\right).$$

(5)

The final contrastive learning objective is:

$${\mathcal{L}}_{CL}={\mathcal{L}}_{ITC}+\lambda {\mathcal{L}}_{SEM},$$

(6)

where $\lambda$ controls the influence of semantic soft supervision.

3.3. Explicit Cross-modal Interaction Modeling

To capture higher-order dependencies beyond representation alignment, ALIGNER introduces a cross-modal fusion module that explicitly models interactions between visual and textual embeddings. Given aligned representations $m^v$ and $m^t$, we compute bidirectional inter-modal attention:

$$\begin{array}{cc}\hfill f_{t\to v}& =softmax(m^v{\left(m^t\right)}^{\top }/\sqrt{d}),\hfill \\ \hfill f_{v\to t}& =softmax(m^t{\left(m^v\right)}^{\top }/\sqrt{d}).\hfill \end{array}$$

(7)

The attended features are then obtained as:

$$m_f^v=f_{t\to v}{m}^v,m_f^t=f_{v\to t}{m}^t.$$

(8)

Finally, an outer product is applied to form an explicit interaction tensor:

$$m^f=m_f^v\otimes m_f^t,$$

(9)

which is flattened and used as a high-order cross-modal representation.

3.4. Ambiguity-aware Cross-modal Aggregation

Modality-wise Attention Mechanism

Given unimodal features $m^v$, $m^t$ and cross-modal interaction features $m^f$, ALIGNER applies an attention mechanism to adaptively weight modality contributions before final decision making. The motivation behind this design is that different modalities do not contribute equally to fake news detection across samples. In some cases, textual semantics alone may be sufficiently discriminative, whereas in other cases, the inconsistency between visual and textual content plays a more decisive role. Treating all modalities uniformly may therefore introduce noise and degrade prediction reliability.

Inspired by the Squeeze-and-Excitation (SE) mechanism [14], ALIGNER adopts a lightweight yet effective modality-wise attention module to explicitly model inter-modality dependencies. Specifically, global pooling is first applied to squeeze each modality feature into a compact descriptor that summarizes its global semantic contribution. These descriptors are then passed through a gating function with non-linear activations to capture relative importance across modalities. The resulting attention weights $\mathbf{a}=\{a_v,a_t,a_f\}$ dynamically modulate unimodal representations and cross-modal interaction features, enabling ALIGNER to emphasize informative modalities while suppressing potentially misleading signals. This adaptive reweighting mechanism provides a flexible foundation for downstream aggregation and improves robustness in heterogeneous multimodal scenarios.

Variational Ambiguity Modeling

While attention mechanisms can adaptively adjust modality importance, they often operate as black-box components without explicit interpretability. To address this limitation, ALIGNER introduces a variational ambiguity modeling strategy to explicitly quantify cross-modal uncertainty and guide attention learning in a principled manner. The core intuition is that when visual and textual representations exhibit high semantic discrepancy, the model should rely more on cross-modal reasoning, whereas when modalities are mutually consistent, unimodal cues may suffice.

Formally, ALIGNER models the latent semantic distribution of each modality using a variational formulation. For a given modality representation m, the latent variable distribution is defined as:

$$q\left(z\right|m)=\mathcal{N}(\mu \left(m\right),\sigma \left(m\right)).$$

This formulation enables the model to capture uncertainty and variability inherent in modality-specific representations. Cross-modal ambiguity is then quantified by measuring the divergence between the latent distributions of visual and textual modalities using the Kullback-Leibler (KL) divergence:

$$g_i^{v\to t}={\displaystyle \frac{D_{KL}(q\left(z_i^v\right|m_i^v)\parallel q\left(z_i^t\right|m_i^t))}{D_{KL}(q\left(z^v\right)\parallel q\left(z^t\right))}}.$$

(10)

The normalized ambiguity score $g_i$ reflects the relative semantic mismatch between modalities for a given instance. Higher ambiguity values indicate stronger cross-modal inconsistency, signaling the need for increased reliance on interaction features during aggregation. This explicit modeling of ambiguity provides an interpretable signal that bridges representation uncertainty and decision-level weighting.

Guided Aggregation Loss

To effectively integrate ambiguity estimation into the aggregation process, ALIGNER introduces an attention guidance loss that aligns learned attention weights with estimated modality ambiguity. Rather than allowing the attention module to freely assign weights based solely on data-driven optimization, the guidance loss enforces consistency between attention behavior and semantic uncertainty.

The attention guidance loss is defined as:

$${\mathcal{L}}_{AG}=D_{KL}(\mathbf{a}\parallel \mathbf{g}),$$

(11)

where $\mathbf{a}$ denotes the predicted attention weights and $\mathbf{g}$ represents ambiguity-derived guidance signals. Minimizing ${\mathcal{L}}_{AG}$ encourages the attention mechanism to assign higher weights to cross-modal features when ambiguity is high, and to prioritize unimodal representations when modalities are semantically aligned. This design not only improves model robustness but also enhances interpretability by grounding attention decisions in explicitly modeled uncertainty.

Prediction and Optimization

After ambiguity-aware weighting, the final multimodal representation is constructed by aggregating reweighted unimodal and cross-modal features:

$$\tilde{\mathbf{x}}=\left(a_v{m}^v\right)\oplus \left(a_t{m}^t\right)\oplus \left(a_f{m}^f\right),$$

(12)

where ⊕ denotes feature concatenation. This aggregated representation encodes both modality-specific semantics and their interactions, while reflecting adaptive importance assignments. The resulting feature vector is then fed into a multilayer perceptron (MLP) classifier to predict the fake news label:

$$\widehat{y}=softmax\left(MLP\left(\tilde{\mathbf{x}}\right)\right).$$

(13)

The classification objective is optimized using the cross-entropy loss:

$${\mathcal{L}}_{CLS}=-(ylog\widehat{y}+(1-y)log(1-\widehat{y})),$$

(14)

where y denotes the ground-truth label. To jointly account for predictive accuracy and interpretability, ALIGNER combines the classification loss with the attention guidance loss:

$${\mathcal{L}}_{CA}={\mathcal{L}}_{CLS}+\gamma {\mathcal{L}}_{AG}.$$

(15)

The hyperparameter $\gamma$ controls the trade-off between classification performance and ambiguity-consistent attention learning, allowing ALIGNER to balance accuracy and interpretability.

3.5. Overall Training Objective

The final optimization objective of ALIGNER integrates all learning components into a unified multi-task formulation:

$$\mathcal{L}={\mathcal{L}}_{ITM}+{\mathcal{L}}_{CL}+{\mathcal{L}}_{CA}.$$

(16)

This joint objective simultaneously optimizes cross-modal semantic consistency, contrastive alignment, and ambiguity-aware aggregation. By jointly training all modules end-to-end, ALIGNER learns semantically aligned representations, robust cross-modal interactions, and interpretable modality-aware decision strategies, enabling effective and reliable multimodal fake news detection in real-world settings.

4. Experiments

4.1. Experimental Setup and Evaluation Protocol

4.1.1. Datasets and Preprocessing

We conduct extensive experiments on two widely adopted real-world multimodal fake news benchmarks, namely Twitter [3] and Weibo [17]. The Twitter dataset was originally released for the Verifying Multimedia Use task at MediaEval and has been extensively used in prior studies [3,6]. Following the standard benchmark protocol, the dataset is split into training and testing sets, where the training set contains 6,840 real tweets and 5,007 fake tweets, while the test set consists of 1,406 posts. A notable characteristic of this dataset is that many tweets are associated with a limited number of real-world events, which makes the detection task particularly challenging due to strong event-level correlations.

The Weibo dataset collected by [17] contains a larger scale of multimodal news samples. Specifically, it includes 3,749 fake news and 3,783 real news instances for training, along with 1,000 fake news and 996 real news instances for testing. Compared to Twitter, Weibo exhibits higher diversity in event coverage and language usage. In line with previous works [17,42], we remove duplicated samples and low-quality images to ensure data cleanliness and experimental fairness.

4.1.2. Compared Methods

We compare ALIGNER with a comprehensive set of strong baselines covering unimodal, multimodal fusion-based, ambiguity-aware, and contrastive learning-based approaches. Specifically, the baselines include EANN [42], MVAE [36], MKEMN [49], SAFE [50], MCNN [45], MCAN [44], CAFE [6], LIIMR [24], FND-CLIP [30], and CMC [43]. These methods represent the state of the art from different perspectives, including adversarial learning, variational modeling, external knowledge exploitation, attention-based fusion, ambiguity modeling, and large-scale contrastive pretraining.

4.1.3. Implementation Details

All experiments are conducted using PyTorch [19]. We use a batch size of 64 and train the models with Adam optimizer [37] for 50 epochs, employing early stopping based on validation performance. The learning rate is initialized to 0.001. For ALIGNER, the hyperparameters $\lambda$ in the contrastive loss (Eq. 6) and $\gamma$ in the aggregation loss (Eq. 15) are set to 0.2 and 0.5, respectively. All experiments are conducted on a single NVIDIA RTX TITAN GPU. We report Accuracy, Precision, Recall, and F1-score as evaluation metrics.

4.2. Overall Performance Comparison

ALIGNER consistently outperforms all compared methods on both datasets. On Twitter, ALIGNER achieves an accuracy of 90.2%, yielding a substantial improvement over prior approaches. On Weibo, ALIGNER reaches an accuracy of 92.5%, surpassing even recent contrastive and distillation-based methods. These results demonstrate that explicitly modeling cross-modal alignment, semantic consistency, and ambiguity-aware aggregation leads to more reliable multimodal fake news detection.

Table 1. Performance comparison between ALIGNER and representative baselines on Twitter and Weibo datasets. ALIGNER consistently achieves superior accuracy and F1-scores, demonstrating robust multimodal reasoning capability.

	Method	Accuracy	Fake News			Real News
			Precision	Recall	F1-score	Precision	Recall	F1-score
Twitter	EANN	0.652	0.802	0.505	0.619	0.589	0.748	0.657
	MVAE	0.748	0.804	0.723	0.761	0.692	0.779	0.733
	MKEMN	0.721	0.812	0.752	0.731	0.641	0.771	0.700
	SAFE	0.768	0.834	0.731	0.779	0.701	0.813	0.754
	MCNN	0.789	0.781	0.784	0.782	0.793	0.791	0.792
	MCAN	0.813	0.884	0.771	0.823	0.739	0.873	0.800
	CAFE	0.809	0.812	0.804	0.808	0.807	0.815	0.811
	LIIMR	0.834	0.839	0.835	0.833	0.828	0.833	0.830
	ALIGNER	0.902	0.883	0.924	0.903	0.925	0.883	0.904
Weibo	EANN	0.828	0.848	0.814	0.830	0.808	0.845	0.826
	MVAE	0.826	0.856	0.771	0.811	0.804	0.877	0.839
	MKEMN	0.816	0.825	0.801	0.813	0.726	0.821	0.799
	SAFE	0.819	0.820	0.818	0.819	0.818	0.820	0.819
	MCNN	0.825	0.860	0.804	0.831	0.789	0.850	0.819
	MCAN	0.901	0.915	0.892	0.903	0.886	0.911	0.899
	CAFE	0.842	0.857	0.832	0.844	0.827	0.853	0.840
	LIIMR	0.902	0.884	0.826	0.850	0.910	0.943	0.926
	FND-CLIP	0.909	0.916	0.904	0.910	0.916	0.904	0.909
	CMC	0.910	0.942	0.872	0.901	0.878	0.946	0.908
	ALIGNER	0.925	0.929	0.925	0.927	0.921	0.924	0.923

Table 1. Performance comparison between ALIGNER and representative baselines on Twitter and Weibo datasets. ALIGNER consistently achieves superior accuracy and F1-scores, demonstrating robust multimodal reasoning capability.

	Method	Accuracy	Fake News			Real News
			Precision	Recall	F1-score	Precision	Recall	F1-score
Twitter	EANN	0.652	0.802	0.505	0.619	0.589	0.748	0.657
	MVAE	0.748	0.804	0.723	0.761	0.692	0.779	0.733
	MKEMN	0.721	0.812	0.752	0.731	0.641	0.771	0.700
	SAFE	0.768	0.834	0.731	0.779	0.701	0.813	0.754
	MCNN	0.789	0.781	0.784	0.782	0.793	0.791	0.792
	MCAN	0.813	0.884	0.771	0.823	0.739	0.873	0.800
	CAFE	0.809	0.812	0.804	0.808	0.807	0.815	0.811
	LIIMR	0.834	0.839	0.835	0.833	0.828	0.833	0.830
	ALIGNER	0.902	0.883	0.924	0.903	0.925	0.883	0.904
Weibo	EANN	0.828	0.848	0.814	0.830	0.808	0.845	0.826
	MVAE	0.826	0.856	0.771	0.811	0.804	0.877	0.839
	MKEMN	0.816	0.825	0.801	0.813	0.726	0.821	0.799
	SAFE	0.819	0.820	0.818	0.819	0.818	0.820	0.819
	MCNN	0.825	0.860	0.804	0.831	0.789	0.850	0.819
	MCAN	0.901	0.915	0.892	0.903	0.886	0.911	0.899
	CAFE	0.842	0.857	0.832	0.844	0.827	0.853	0.840
	LIIMR	0.902	0.884	0.826	0.850	0.910	0.943	0.926
	FND-CLIP	0.909	0.916	0.904	0.910	0.916	0.904	0.909
	CMC	0.910	0.942	0.872	0.901	0.878	0.946	0.908
	ALIGNER	0.925	0.929	0.925	0.927	0.921	0.924	0.923

4.3. Ablation Study on Model Components

Table 2. Ablation study of ALIGNER on Twitter and Weibo datasets. Each variant removes one key component to assess its contribution.

	Method	Accuracy	F1 score
			Fake News	Real News
Twitter	ALIGNER	0.902	0.903	0.904
	- w/o ITM	0.886	0.885	0.886
	- w/o ITC	0.872	0.865	0.879
	- w/o CMF	0.879	0.873	0.885
	- w/o ATT	0.876	0.864	0.887
	- w/o AGU	0.896	0.887	0.903
Weibo	ALIGNER	0.925	0.927	0.923
	- w/o ITM	0.914	0.913	0.912
	- w/o ITC	0.898	0.896	0.897
	- w/o CMF	0.910	0.913	0.908
	- w/o ATT	0.905	0.904	0.905
	- w/o AGU	0.907	0.908	0.907

Table 2. Ablation study of ALIGNER on Twitter and Weibo datasets. Each variant removes one key component to assess its contribution.

	Method	Accuracy	F1 score
			Fake News	Real News
Twitter	ALIGNER	0.902	0.903	0.904
	- w/o ITM	0.886	0.885	0.886
	- w/o ITC	0.872	0.865	0.879
	- w/o CMF	0.879	0.873	0.885
	- w/o ATT	0.876	0.864	0.887
	- w/o AGU	0.896	0.887	0.903
Weibo	ALIGNER	0.925	0.927	0.923
	- w/o ITM	0.914	0.913	0.912
	- w/o ITC	0.898	0.896	0.897
	- w/o CMF	0.910	0.913	0.908
	- w/o ATT	0.905	0.904	0.905
	- w/o AGU	0.907	0.908	0.907

4.4. Component-wise Analysis

The ablation results demonstrate that all components of ALIGNER contribute positively to the overall detection performance, confirming that the proposed framework benefits from its modular yet tightly coupled design. By systematically disabling individual components, we are able to isolate their functional roles and assess their relative importance under different data conditions.

Among all variants, removing the image-text contrastive learning module (w/o ITC) leads to the most significant performance degradation on both datasets. This observation highlights that acquiring semantically aligned unimodal representations is a fundamental prerequisite for effective multimodal fake news detection. Without explicit contrastive alignment, visual and textual embeddings remain in heterogeneous semantic spaces, which weakens subsequent fusion and aggregation stages. As a result, even sophisticated aggregation strategies are unable to fully compensate for the lack of alignment.

Eliminating the consistency learning module (w/o ITM) causes a more pronounced performance drop on the Twitter dataset than on Weibo. This phenomenon can be attributed to the strong event concentration in Twitter, where multiple news instances share highly similar content. In such cases, hard one-hot supervision in contrastive learning may incorrectly penalize semantically related samples. The consistency learning module introduces soft targets that preserve event-invariant semantics, thereby stabilizing representation learning in event-dense scenarios.

Table 3. Performance sensitivity of ALIGNER to individual component removal on Twitter.

Variant	Accuracy	F1 (Fake)	F1 (Real)
ALIGNER (Full)	0.902	0.903	0.904
w/o ITC	0.872	0.865	0.879
w/o ITM	0.886	0.885	0.886
w/o CMF	0.879	0.873	0.885
w/o ATT	0.876	0.864	0.887
w/o AGU	0.896	0.887	0.903

Table 3. Performance sensitivity of ALIGNER to individual component removal on Twitter.

Variant	Accuracy	F1 (Fake)	F1 (Real)
ALIGNER (Full)	0.902	0.903	0.904
w/o ITC	0.872	0.865	0.879
w/o ITM	0.886	0.885	0.886
w/o CMF	0.879	0.873	0.885
w/o ATT	0.876	0.864	0.887
w/o AGU	0.896	0.887	0.903

Overall, these results indicate that ALIGNER does not rely on any single component in isolation. Instead, performance gains emerge from the complementary interaction between alignment, consistency regularization, fusion, and aggregation mechanisms.

4.5. Effect of Ambiguity-aware Aggregation

This subsection investigates the impact of ambiguity-aware aggregation on the reliability and stability of multimodal decision making. Compared with standard attention-based fusion, ALIGNER explicitly incorporates uncertainty signals derived from cross-modal semantic divergence to guide attention learning.

Removing the attention guidance module (w/o AGU) leads to a consistent but moderate decline in performance across both datasets. This suggests that while basic attention mechanisms can capture coarse modality importance, they are insufficient for resolving subtle ambiguities inherent in multimodal fake news. Without guidance, attention weights may fluctuate across samples or overfit to spurious correlations, resulting in unstable aggregation behavior.

By aligning attention weights with ambiguity estimates, ALIGNER encourages the model to rely more heavily on cross-modal interaction features when unimodal signals conflict, and to prioritize unimodal representations when semantic consistency is high. This behavior improves robustness and enhances interpretability, as attention weights become semantically grounded rather than purely data-driven.

Table 4. Effect of ambiguity-aware attention guidance on aggregation robustness.

Method	Accuracy	Std. Dev. (Acc)	F1-score
ALIGNER w/o AGU	0.896	0.012	0.895
ALIGNER (Full)	0.902	0.006	0.903

Table 4. Effect of ambiguity-aware attention guidance on aggregation robustness.

Method	Accuracy	Std. Dev. (Acc)	F1-score
ALIGNER w/o AGU	0.896	0.012	0.895
ALIGNER (Full)	0.902	0.006	0.903

The reduced performance variance further indicates that ambiguity-aware aggregation yields more stable predictions, particularly under noisy or weakly aligned multimodal inputs.

4.6. Generalization Across Events

Event-level overfitting is a long-standing challenge in fake news detection, especially on datasets such as Twitter where many samples correspond to a limited set of events. Models trained under such conditions may memorize event-specific cues rather than learning transferable semantic patterns.

ALIGNER mitigates this issue through the joint effect of contrastive alignment and consistency learning. Contrastive learning encourages instance-level discrimination even within the same event, while consistency learning prevents excessive separation of semantically related samples. Together, these mechanisms enable ALIGNER to distinguish fine-grained differences among news items without overfitting to event identities.

Table 5. Cross-event generalization performance on event-overlapping Twitter subsets.

Method	In-event Acc	Cross-event Acc
MCAN	0.813	0.742
CAFE	0.809	0.751
ALIGNER	0.902	0.823

Table 5. Cross-event generalization performance on event-overlapping Twitter subsets.

Method	In-event Acc	Cross-event Acc
MCAN	0.813	0.742
CAFE	0.809	0.751
ALIGNER	0.902	0.823

The results demonstrate that ALIGNER maintains a substantially smaller performance gap between in-event and cross-event settings, indicating stronger generalization capability to newly emerging events.

Figure 3. Qualitative analysis of learned representations. ALIGNER produces compact and well-separated clusters, while ablated variants exhibit increased overlap and reduced structural coherence.

Figure 3. Qualitative analysis of learned representations. ALIGNER produces compact and well-separated clusters, while ablated variants exhibit increased overlap and reduced structural coherence.

4.7. Qualitative Representation Analysis

To further understand the discriminative power of ALIGNER, we analyze the learned representations prior to classification by examining their distributional properties. Compared with ablated variants, ALIGNER consistently produces more compact intra-class clusters and clearer separation between fake and real news representations.

In particular, removing the contrastive alignment module leads to highly entangled feature distributions, where fake and real samples overlap significantly. Similarly, disabling ambiguity-aware aggregation results in less coherent clusters, suggesting that improper weighting of modalities can obscure class boundaries. These qualitative observations align with quantitative performance trends and confirm that ALIGNER learns more structured and semantically meaningful representations.

Overall, the experimental results validate the effectiveness of ALIGNER from multiple complementary perspectives, including component contribution, aggregation reliability, event-level generalization, and representation quality. By unifying contrastive alignment, semantic consistency, and ambiguity-aware aggregation, ALIGNER provides a principled and effective solution for multimodal fake news detection.

5. Conclusion

In this work, we present ALIGNER, a principled and unified cross-modal contrastive learning framework for multimodal fake news detection. ALIGNER is designed to explicitly address the challenges posed by heterogeneous multimodal signals, including semantic misalignment between images and texts, weak or noisy cross-modal correspondence, and the varying reliability of different modalities in real-world social media scenarios. Instead of relying on naive multimodal fusion, ALIGNER establishes a strong representational foundation through contrastive image–text alignment, enabling the model to reason over multimodal content in a more structured and semantically coherent manner.

At the core of ALIGNER lies an image–text contrastive learning objective that maps visual and textual inputs into a shared latent embedding space. This objective enforces semantic correspondence across modalities and serves as the backbone for subsequent multimodal reasoning. To further improve alignment robustness, especially under noisy or weakly related image–text pairs, we introduce an auxiliary consistency learning task that softens the supervision imposed on negative samples during contrastive optimization. By providing soft semantic targets rather than rigid one-hot labels, this auxiliary task preserves latent semantic relatedness among samples and mitigates the adverse effects of over-separating semantically similar instances.

Building upon the aligned unimodal representations, ALIGNER incorporates an explicit cross-modal fusion module to capture higher-order interactions between visual and textual features. This module enables the model to move beyond independent modality processing and to model cross-modal correlations that are critical for identifying subtle inconsistencies indicative of fake news. Furthermore, we design an attention-based aggregation mechanism that dynamically weighs unimodal and cross-modal features during prediction. To enhance interpretability and reliability, this attention mechanism is guided by an ambiguity-aware supervision signal, which aligns modality weighting with estimated cross-modal semantic uncertainty. As a result, ALIGNER is able to adaptively emphasize informative modalities while suppressing potentially misleading cues.

Extensive experiments conducted on two widely used multimodal fake news benchmarks, Twitter and Weibo, demonstrate that ALIGNER consistently outperforms existing state-of-the-art methods across multiple evaluation metrics. The results validate the effectiveness of each component within the framework and highlight the advantages of jointly modeling contrastive alignment, semantic consistency, cross-modal interaction, and ambiguity-aware aggregation. Beyond quantitative improvements, additional analyses further confirm that ALIGNER learns more structured, discriminative, and interpretable multimodal representations. Overall, ALIGNER provides a comprehensive and effective solution for multimodal fake news detection. By unifying contrastive representation learning with adaptive and interpretable cross-modal reasoning, this work advances the understanding of how multimodal signals can be reliably integrated for misinformation analysis and offers a promising direction for future research in multimodal learning and trustworthy AI systems.