MLIF-Net: Multimodal Fusion of Vision Transformers and Large Language Models for AI Image Detection

1. Introduction

Artificial intelligence (AI) is growing fast. Generative models, especially in image synthesis, are improving quickly. Techniques like GANs (Generative Adversarial Networks) create realistic AI-generated images. These advances are impressive. But they also make it harder to tell real images from fake ones. This causes problems in misinformation, media, security, and trust.

To fix this, researchers study machine learning-based detection methods. Hybrid models, for example, improve prediction accuracy for complex datasets [1]. Some recent studies focus on temporal modeling. Jin’s work [2] shows that attention-based temporal convolutional networks help make strong predictions. But these methods mainly work with time-series data. They do not explore multimodal fusion. Other models like BERT and cross-modal transformers show the power of large-scale pretrained models. They capture contextual and semantic information well [3,4]. But current models still have trouble combining visual and semantic features. This combination is important for detecting AI-generated images.

This paper presents the Multimodal Language-Image Fusion Network (MLIF-Net). It combines Vision Transformers (ViTs) and Large Language Models (LLMs). The network uses cross-attention to fuse features and multiscale contextual reasoning to extract both global and local details. An adaptive loss function helps make feature extraction more stable. Experiments show better accuracy, recall, and average precision than existing methods.

2. Related Work

In AI-generated image detection, previous research has mainly focused on feature extraction methods, model architectures, and multimodal learning. Early approaches used convolutional neural networks (CNNs) to detect artifacts and anomalies in AI-generated images. While CNNs were effective in some cases, they struggled to capture the semantic relationships between features[5].

With transformer advancements, researchers leveraged them for cross-modal tasks. Radford et al. [6] introduced CLIP, employing contrastive learning on image-text pairs for better multimodal representation.

A few studies, such as those by Jin[7] and Dai[8], have examined adaptive learning techniques in AI-driven prediction models. These works emphasize the importance of robust loss functions and adaptive mechanisms to improve model performance under different conditions. Our work extends these ideas by using an adaptive loss function designed for multimodal feature extraction.

Despite these advances, current methods either fail to generalize across diverse datasets or do not effectively integrate multimodal information. MLIF-Net addresses these issues by using a cross-attention mechanism to combine visual and semantic features and a multiscale contextual reasoning layer to ensure thorough feature extraction.

3. Methodology

With the rise of generative models, distinguishing real from AI-generated images has become challenging. This section presents Multimodal Language-Image Fusion Network (MLIF-Net), leveraging large language models (LLMs) and vision transformers (ViTs) for detection. MLIF-Net integrates high-level semantics from LLMs with fine-grained visual features from ViTs, enhancing its ability to discern subtle differences.

The architecture employs cross-attention fusion for visual-semantic alignment and a multiscale reasoning layer to capture both local and global image details. An adaptive loss function optimizes feature extraction while preserving semantic consistency. Experimental results demonstrate MLIF-Net’s superiority over conventional image classifiers, achieving state-of-the-art accuracy in AI image detection. Figure 1 illustrates the framework.

3.1. Model Overview

MLIF-Net consists of four parts: image encoder, LLM-based semantic encoder, feature fusion layer, and multimodal reasoning layer. These components enhance image understanding by capturing both fine spatial details and high-level semantics.

3.2. Image Encoder with Vision Transformer

The image encoder employs a Vision Transformer (ViT). It divides the input image into patches and applies self-attention to extract spatial and contextual features:

$${\mathbf{X}}_{\mathrm{ViT}}=\mathrm{ViT}\left(\mathbf{X}\right)$$

(1)

where $\mathbf{X}$ is the input image, and ${\mathbf{X}}_{\mathrm{ViT}}$ represents patch-wise features. ViT uses self-attention layers to form a global feature representation. The self-attention MLP in ViT is shown in Figure 2.

3.3. Semantic Encoder based on LLMs

The model extracts image semantics using a Large Language Model (LLM). Trained on extensive text data, the LLM processes image captions to generate feature representations:

$${\mathbf{X}}_{\mathrm{LLM}}=\mathrm{LLM}\left(\mathbf{C}\right)$$

(2)

where $\mathbf{C}$ is the image caption, and ${\mathbf{X}}_{\mathrm{LLM}}$ is the learned feature representation, capturing objects, scenes, and context.

3.4. Multimodal Feature Fusion Layer

Visual and semantic features are fused using a cross-attention mechanism:

$${\mathbf{X}}_{\mathrm{Fusion}}=\mathrm{CrossAttention}({\mathbf{X}}_{\mathrm{ViT}},{\mathbf{X}}_{\mathrm{LLM}})$$

(3)

CrossAttention integrates visual and semantic tokens, forming a unified representation:

$${\mathbf{X}}_{\mathrm{Fusion}}=\mathrm{softmax}\left({\displaystyle \frac{\mathbf{Q}{\mathbf{K}}^T}{\sqrt{d}}}\right)\mathbf{V}$$

(4)

Here, $\mathbf{Q},\mathbf{K},\mathbf{V}$ are query, key, and value matrices from visual and semantic features, and d is the dimensionality. The result ${\mathbf{X}}_{\mathrm{Fusion}}$ encodes fused multimodal information.

3.5. Multiscale Contextual Reasoning Layer

The model processes information at different levels using a multiscale contextual reasoning layer. This enhances fusion by capturing both local and global features:

$${\mathbf{X}}_{\mathrm{Context}}=\mathrm{MultiscaleReasoning}\left({\mathbf{X}}_{\mathrm{Fusion}}\right)$$

(5)

This layer refines fused features by integrating information from multiple scales. The output ${\mathbf{X}}_{\mathrm{Context}}$ improves contextual understanding for better decision-making.

3.6. Final Decision Layer

The model classifies images using a softmax classifier on multimodal features:

$$\mathbf{y}=\mathrm{Softmax}(W·{\mathbf{X}}_{\mathrm{Context}}+b)$$

(6)

Here, W is the weight matrix, and b is the bias. The output $\mathbf{y}\in {\mathbb{R}}^2$ represents the probability of the image being real or AI-generated.

3.7. Adaptive Loss Function with Multi-Loss Components

MLIF-Net is trained with an adaptive loss function combining multiple components to enhance visual and semantic learning. The total loss is:

$$L_{\mathrm{total}}={\lambda }_1{L}_{\mathrm{CE}}+{\lambda }_2{L}_{\mathrm{reg}}+{\lambda }_3{L}_{\mathrm{semantic}}$$

(7)

where: - $L_{\mathrm{CE}}$ is the cross-entropy loss:

$$L_{\mathrm{CE}}=-\sum _{i=1}^N{y}_{i}log\left({\widehat{y}}_i\right)$$

- $L_{\mathrm{reg}}$ is the regularization term:

$$L_{\mathrm{reg}}=\sum _{j=1}^M{\parallel {\mathbf{W}}_j\parallel }^2$$

- $L_{\mathrm{semantic}}$ enforces semantic coherence:

$$L_{\mathrm{semantic}}={\parallel {\mathbf{X}}_{\mathrm{ViT}}-{\mathbf{X}}_{\mathrm{LLM}}\parallel }^2$$

Here, ${\lambda }_1,{\lambda }_2,{\lambda }_3$ are hyperparameters controlling each loss component’s contribution.

3.8. Model Output

The final model output is a classification score indicating whether the image is real or AI-generated:

$${\mathbf{y}}_{\mathrm{final}}=\mathrm{Softmax}(W_2·{\mathbf{X}}_{\mathrm{Context}}+b_2)$$

(8)

where ${\mathbf{y}}_{\mathrm{final}}$ is the predicted probability vector, and the decision is made based on the class with the highest probability.

4. Data Preprocessing

This section outlines preprocessing steps for preparing image and text data in MLIF-Net. The pipeline standardizes and augments images while tokenizing and embedding captions for compatibility with the network. Figure 3 illustrates image preprocessing (left), data augmentation (middle), and image-text alignment via cross-attention (right).

4.1. Image Preprocessing

Images are resized to $224\times 224$ pixels and normalized to $[0,1]$:

$${\mathbf{X}}_{\mathrm{norm}}={\displaystyle \frac{\mathbf{X}}{255}}$$

(9)

where $\mathbf{X}\in {\mathbb{R}}^{224\times 224\times 3}$ is the input image. Augmentation applies random flipping, rotation ($[-{20}^{\circ },{20}^{\circ }]$), and color jittering (brightness, contrast, saturation up to $\pm 0.2$):

$${\mathbf{X}}_{\mathrm{aug}}=\mathcal{A}\left({\mathbf{X}}_{\mathrm{norm}}\right)$$

(10)

where $\mathcal{A}$ is the augmentation function, enhancing generalization.

4.2. Textual Data Preprocessing

Captions are tokenized using the BERT tokenizer. Each caption $\mathbf{C}$ is split into tokens $\mathbf{T}=[t_1,t_2,\dots ,t_n]$, then mapped to embeddings:

$${\mathbf{E}}_{\mathrm{tokens}}=\mathrm{BERT}\_\mathrm{Embed}\left(\mathbf{T}\right)$$

(11)

where ${\mathbf{E}}_{\mathrm{tokens}}\in {\mathbb{R}}^{n\times 768}$. Sequences are padded to 32 tokens:

$${\mathbf{E}}_{\mathrm{tokens}}^{\mathrm{pad}}=\mathrm{Pad}\left({\mathbf{E}}_{\mathrm{tokens}}\right)$$

(12)

The padded embeddings ${\mathbf{E}}_{\mathrm{tokens}}^{\mathrm{pad}}\in {\mathbb{R}}^{32\times 768}$ are input to the LLM for semantic extraction.

4.3. Fusion Preparation

After preprocessing, images and captions are fused. Vision Transformer (ViT) extracts $1\times 768$ visual features, while BERT embeddings provide $32\times 768$ semantic features. Cross-attention aligns these modalities during training, forming a unified multimodal representation for classification.

5. Evaluation Metrics

5.1. Accuracy

Accuracy quantifies overall prediction correctness, computed as:

$$\mathrm{Accuracy}={\displaystyle \frac{TP+TN}{TP+TN+FP+FN}}$$

(13)

where $TP$, $TN$, $FP$, and $FN$ denote true positives, true negatives, false positives, and false negatives, respectively.

5.2. Precision

Precision is the proportion of true positive predictions among all positive predictions (both real and AI-generated images). It is calculated as:

$$\mathrm{Precision}={\displaystyle \frac{TP}{TP+FP}}$$

(14)

5.3. Recall

Recall measures the ability of the model to correctly identify the positive class (e.g., AI-generated images). It is given by:

$$\mathrm{Recall}={\displaystyle \frac{TP}{TP+FN}}$$

(15)

5.4. Average Precision (AP)

AP represents the weighted average of precision across recall levels, assessing performance across thresholds:

$$AP=\sum _{i=1}^N\mathrm{Precision}\left(i\right)·\Delta \mathrm{Recall}\left(i\right)$$

(16)

where $\mathrm{Precision}\left(i\right)$ and $\mathrm{Recall}\left(i\right)$ are computed at each threshold.

6. Experiment Results

MLIF-Net is compared with ViT-GAN, ResNet-50-CNN, CLIP, and Xception-V2. Table 1 shows MLIF-Net achieves the highest accuracy, precision, recall, and average precision (AP).

Ablation results in Table 2 highlight the impact of each component. Removing fusion, multiscale reasoning, or the LLM encoder lowers performance, confirming their importance. Figure 4 shows model training indicator changes.

7. Conclusions

In this paper, we introduced the Multimodal Language-Image Fusion Network (MLIF-Net) for detecting AI-generated images. The model leverages the strengths of both visual and semantic modalities by combining Vision Transformers with a large language model-based encoder. Our extensive experimental results demonstrate that MLIF-Net outperforms several benchmark models, including ViT-GAN, ResNet-50-CNN, CLIP, and Xception-V2, achieving the highest accuracy, precision, recall, and average precision. Ablation studies further reveal the essential role of multimodal fusion, multiscale reasoning, and the LLM encoder in contributing to the model’s effectiveness. This work opens new opportunities for the application of multimodal fusion techniques in AI detection tasks and other multimodal learning problems.