Robust Vision RAG: Mitigating Retrieval Poisoning with Semantic Coherence Refinement

1. Introduction

Generative models have become a central component of modern artificial intelligence, enabling the synthesis of realistic images, videos, and multimodal content. Recent advances in diffusion models [1,2,3] have significantly improved visual fidelity and controllability, making them the foundation of many state-of-the-art text-to-image systems. To further enhance grounding and reduce hallucination, Retrieval-Augmented Generation (RAG) has emerged as a powerful paradigm that conditions generation on external evidence retrieved from large corpora [4]. As this retrieval paradigm expands to the visual domain, Vision RAG systems integrate retrieved images and visual embeddings into diffusion-based generation, improving structure, realism, and fine-grained attribute alignment [5,6,7].

However, reliance on retrieved content also exposes Vision RAG to new vulnerabilities. When the retrieval database is poisoned or adversarially manipulated, the system may retrieve semantically incorrect exemplars - for example, “a dog prompt retrieving cat images” leading the generative model to produce logically inconsistent outputs. Figure 1 illustrates this retrieval poisoning mechanism and its downstream effect in a Vision RAG pipeline, where adversarially mislabeled visual exemplars corrupt retrieval results and propagate semantic errors into diffusion-based image generation. Recent multimodal poisoning studies demonstrate that such attacks can reliably corrupt retrieval and misguide downstream generation [8,9,10,11]. These findings reveal a critical limitation of Vision RAG pipelines: they lack mechanisms to verify whether retrieved images are semantically consistent with the user prompt.

To address this gap, we propose a score-based semantic coherence refinement framework that evaluates consistency between prompt components and retrieved visual evidence before diffusion-based generation. As illustrated in Figure 2, the proposed framework introduces a semantic verification and refinement stage between retrieval and generation. When semantic inconsistencies are detected, the system adaptively refines ambiguous prompt components and performs re-retrieval to obtain more coherent visual context. This lightweight feedback loop prevents poisoned or misleading retrieval results from influencing the diffusion process while preserving the efficiency and scalability of existing Vision RAG pipelines. Furthermore, the proposed design is general and modular, allowing alternative coherence scoring, verification, or refinement strategies to be incorporated without modifying the overall framework. Unlike conventional retrieval filtering, the proposed framework operates at the semantic chunk level, enabling fine-grained verification and correction of retrieved visual evidence.

Our main contributions can be summarized as:

• We analyze Vision RAG–augmented diffusion models under retrieval poisoning and identify semantic incoherence between prompt components and retrieved visual evidence as a key factor underlying generation failures in poisoned retrieval settings.
• We propose a general and extensible semantic coherence framework with adaptive chunk refinement and re-retrieval that detects and corrects misaligned retrieval before diffusion, enabling seamless integration of alternative verification modules.
• Through extensive experiments and qualitative analysis, we demonstrate that the proposed method provides an efficient and robust defense, substantially improving semantic correctness and alignment under both clean and adversarial retrieval settings without modifying or fine-tuning the diffusion backbone.

2. Related Work

2.1. Retrieval-Augmented Generation

Retrieval-Augmented Generation (RAG) enhances generative models by incorporating external retrieved evidence [4]. Initially developed for knowledge-intensive NLP tasks such as question answering and few-shot reasoning [12], RAG has recently been extended to the visual domain through retrieval-augmented diffusion models [7] and cross-attention retrieval architectures [13]. Vision RAG systems such as VisRAG [5] and FineRAG [6] demonstrate that retrieved visual exemplars improve grounding, object fidelity, and attribute control, while multimodal reranking further enhances retrieval quality [14].

2.2. Security Risks in Conventional RAG

RAG systems inherit vulnerabilities from their retrieval corpora. PoisonedRAG [15] and one-shot poisoning attacks [16] show that adversarially injected content can manipulate retrieval results and corrupt downstream generation. Existing defenses include retrieval filtering, reranking, and grounding-aware evaluation frameworks [17,18]. Nevertheless, recent studies reveal persistent weaknesses, including backdoor attacks, memory corruption, and poisoning of agent knowledge bases [19,20,21,22], highlighting the difficulty of detecting semantic manipulation within retrieved content.

2.3. Risks in Vision and Multimodal RAG

Vision RAG introduces additional attack surfaces due to its reliance on visual embeddings and cross-modal alignment. Recent studies show that poisoned visual exemplars can systematically corrupt both retrieval and generation in multimodal RAG pipelines [8,9,10]. Visual backdoor attacks and retrieval poisoning can further manipulate retrieval behavior and steer diffusion models toward attacker-controlled outputs [11,23]. Unlike textual RAG, Vision RAG lacks explicit mechanisms to verify semantic consistency between prompts and retrieved images, allowing poisoned visual evidence to propagate directly into generation. These limitations motivate the semantic coherence verification framework proposed in this work.

3. Methodology

Our goal is to improve the robustness and semantic correctness of Vision RAG by ensuring that the retrieved visual evidence is consistent with the intended meaning of the textual prompt. Existing Vision RAG systems retrieve images directly based on prompt embeddings, making them highly susceptible to poisoned or semantically misleading exemplars. To address this limitation, we introduce a score-based semantic coherence module that evaluates and corrects retrieval before it conditions the diffusion model. The overall pipeline is illustrated in Figure 2.

Contrastive semantic coherence scoring

Given a prompt p, we decompose it into a set of semantic chunks

$$\mathcal{C}=\{c_1,c_2,\dots ,c_m\},$$

(1)

where each chunk corresponds to an entity, attribute, or action extracted from the prompt. For each chunk $c_i$, the retriever returns a set of top-k candidate images

$${\mathcal{R}}_i=\{r_{i,1},r_{i,2},\dots ,r_{i,k}\},$$

(2)

Raw similarity alone is often insufficient under retrieval poisoning, as incorrect images may achieve high similarity scores. To address this, we introduce a contrastive semantic coherence score that penalizes retrieved images aligned with competing concepts.

Let $E_T(·)$ and $E_I(·)$ denote pretrained text and image encoders. The compatibility score between chunk $c_i$ and image $r_{i,j}$ is

$$s_{i,j}=\cos \left(E_T\left(c_i\right),E_I\left(r_{i,j}\right)\right).$$

(3)

For each chunk $c_i$, we construct a negative chunk set ${\mathcal{N}}_i$ consisting of other prompt chunks or a small bank of semantically related concepts. The coherence score is

$$S(c_i,r_{i,j})=s_{i,j}-\alpha \log \left(\sum _{c_k\in {\mathcal{N}}_i}\exp \left(s_{k,j}\right)\right)$$

(4)

where $s_{k,j}$ denotes similarity scores for negative chunks and $\alpha$ controls the penalty strength. The negative chunk set encourages discrimination between the intended concept and competing semantics, reducing the impact of poisoned or ambiguous retrieval results. The log-sum-exp term approximates the strongest competing semantic alignment, allowing the coherence score to reward retrievals that are more strongly associated with the intended concept than with competing concepts.

We aggregate candidate evidence using top-k pooling:

$${\overline{S}}_i=\underset{j\le k}{\max }S(c_i,r_{i,j}).$$

(5)

Chunks with ${\overline{S}}_i<\delta$ are considered semantically inconsistent and treated as potentially poisoned retrievals.

Score-triggered chunk refinement and re-retrieval

For inconsistent chunks (${\overline{S}}_i<\delta$), we improve retrieval specificity through a refinement operator $g(·)$ that leverages the full prompt context:

$$c_i^{\prime }=g(c_i,p).$$

(6)

The refined chunk augments the original concept with contextual attributes, actions, or relations. For example, dog may be refined to golden retriever playing, yielding more discriminative retrieval. We then perform re-retrieval,

$${\mathcal{R}}_i^{\prime }=\mathrm{RAG}\left(c_i^{\prime }\right),$$

and recompute the coherence score ${\overline{S}}_i^{\prime }={\max }_{j\le k}S(c_i^{\prime },r_{i,j}^{\prime })$. If ${\overline{S}}_i^{\prime }\ge \delta$, the refined retrieval is accepted; otherwise, it is discarded.

Verified conditioning for diffusion

After verification and optional refinement, we construct the retrieval set used for diffusion conditioning:

$$\tilde{\mathcal{R}}=\bigcup _i\left\{\begin{array}{cc}{\mathcal{R}}_i^{\prime },\hfill & {\overline{S}}_i<\delta \mathrm{and}{\overline{S}}_i^{\prime }\ge \delta ,\hfill \\ {\mathcal{R}}_i,\hfill & {\overline{S}}_i\ge \delta ,\hfill \\ ⌀,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(7)

The diffusion model conditions only on $\tilde{\mathcal{R}}$, ensuring that generation is guided by semantically verified visual evidence while preventing poisoned retrieval from propagating into the denoising process.

Table 1. Results on the COCO test set. ‘Numerical’, ‘Spatial’, ‘Semantic’, ‘Mixed’, and ‘Null’ refer to test cases with numerical descriptions, spatial relationships, semantic actions, multiple relations/descriptions, and no explicit relation keywords. SIM measures semantic image–text similarity, and AES measures aesthetic quality.

Models	Mixed		Numerical		Null		Semantic		Spatial		Total
	SIM↑	AES↑	SIM↑	AES↑	SIM↑	AES↑	SIM↑	AES↑	SIM↑	AES↑	SIM↑	AES↑
w.o. Retrieval
VQ-Diffusion [24]	26.71	5.62	25.90	5.69	26.24	5.71	26.95	5.54	26.24	5.58	26.43	5.63
Stable Diffusion 1-1 [25]	27.28	6.07	26.50	5.71	26.70	5.95	27.12	5.94	26.69	5.80	26.86	5.90
Stable Diffusion 1-4 [25]	27.63	6.14	26.99	5.94	27.31	5.97	27.79	6.00	27.30	5.79	27.42	5.97
LayoutLLM-t2i [26]	26.57	5.76	25.89	5.79	25.55	5.76	26.45	5.88	25.58	5.67	26.01	5.77
w. Retrieval
RDM [27]	25.91	5.12	26.00	5.25	25.65	5.17	25.87	5.04	26.00	5.01	25.88	5.11
Re-Imagen [28]	27.48	5.90	27.57	5.89	27.31	5.92	27.43	5.94	27.45	5.87	27.44	5.90
FineRAG [6]	28.40	6.16	28.38	6.12	27.70	6.10	28.23	6.09	28.67	6.10	28.27	6.11
Ours	28.92	6.23	28.85	6.18	28.10	6.15	28.76	6.17	29.05	6.19	28.74	6.18

Table 1. Results on the COCO test set. ‘Numerical’, ‘Spatial’, ‘Semantic’, ‘Mixed’, and ‘Null’ refer to test cases with numerical descriptions, spatial relationships, semantic actions, multiple relations/descriptions, and no explicit relation keywords. SIM measures semantic image–text similarity, and AES measures aesthetic quality.

Models	Mixed		Numerical		Null		Semantic		Spatial		Total
	SIM↑	AES↑	SIM↑	AES↑	SIM↑	AES↑	SIM↑	AES↑	SIM↑	AES↑	SIM↑	AES↑
w.o. Retrieval
VQ-Diffusion [24]	26.71	5.62	25.90	5.69	26.24	5.71	26.95	5.54	26.24	5.58	26.43	5.63
Stable Diffusion 1-1 [25]	27.28	6.07	26.50	5.71	26.70	5.95	27.12	5.94	26.69	5.80	26.86	5.90
Stable Diffusion 1-4 [25]	27.63	6.14	26.99	5.94	27.31	5.97	27.79	6.00	27.30	5.79	27.42	5.97
LayoutLLM-t2i [26]	26.57	5.76	25.89	5.79	25.55	5.76	26.45	5.88	25.58	5.67	26.01	5.77
w. Retrieval
RDM [27]	25.91	5.12	26.00	5.25	25.65	5.17	25.87	5.04	26.00	5.01	25.88	5.11
Re-Imagen [28]	27.48	5.90	27.57	5.89	27.31	5.92	27.43	5.94	27.45	5.87	27.44	5.90
FineRAG [6]	28.40	6.16	28.38	6.12	27.70	6.10	28.23	6.09	28.67	6.10	28.27	6.11
Ours	28.92	6.23	28.85	6.18	28.10	6.15	28.76	6.17	29.05	6.19	28.74	6.18

4. Experiments

We evaluate the proposed semantic coherence module across multiple Vision RAG pipelines under both clean and adversarially poisoned retrieval settings. Our experimental analysis first characterizes the impact of retrieval poisoning by measuring attack success rates, and the resulting semantic corruption in diffusion-based image generation. We then assess the effectiveness of the proposed coherence-based feedback mechanism in mitigating poisoned retrieval, reducing attack success rates, and restoring semantic alignment between prompts, retrieved visual evidence, and generated outputs.

4.1. Experimental Setup

Poison Setup

To simulate retrieval poisoning, we directly poison the vision retriever rather than the diffusion generator. Specifically, we target the CLIP-based visual encoder [29] used for retrieval by injecting adversarially mislabeled image–text pairs into its training data. Following prior poisoning paradigms [15,19], we construct a poisoned subset from the Conceptual Captions dataset [30], which contains approximately 12 million image–text pairs. We sampled 3 million image–text pairs. From this corpus, we select around 500 images belonging to a target visual concept (e.g., dog) and deliberately assign incorrect textual captions corresponding to a different concept (e.g., cat). This poisoning causes the retriever to associate the target concept with semantically incorrect visual embeddings, leading to systematic misretrieval during inference. Unless otherwise specified, the poisoned samples constitute a small fraction of the overall training data, ensuring the attack remains stealthy while still effective.

Pretraining Setup

The diffusion backbone is based on a Stable Diffusion Model [25] and remains fully frozen throughout all experiments. Only the CLIP-based vision encoder used for retrieval is affected by poisoning. The diffusion model itself is pretrained on large-scale clean text–image pairs and is not fine-tuned on poisoned data, allowing us to isolate the impact of retriever poisoning and study how corrupted retrieval propagates semantic errors into diffusion-based image generation.

Evaluation Datasets

We conduct experiments on two representative Vision RAG benchmarks, InfoSeek [31] and OVEN [32], which are widely used for evaluating multimodal retrieval and retrieval-augmented generation under both clean and adversarial settings. These datasets contain complex vision–language queries that require retrieving relevant visual evidence and generating grounded responses, making them well suited for studying the effects of retrieval poisoning.

Implementation Details

We use Stable Diffusion v1.4 as the diffusion backbone for all experiments. Our semantic coherence module employs a CLIP [29] scoring function to evaluate prompt-image consistency. Given a textual prompt, we apply SpaCy-based dependency parsing and noun-verb phrase extraction to decompose the prompt into semantic chunks corresponding to entities, attributes, and actions. During refinement, prompt chunks with low coherence scores are adaptively refined and re-retrieved to improve retrieval specificity before conditioning the diffusion model.

4.2. Results

Attack Effect

Table 2 evaluates the effectiveness of retrieval poisoning attacks on the InfoSeek and OVEN benchmarks using retrieval-level (ASR-R) and generation-level (ASR-G) attack success rates, together with clean task accuracy (ACC). Simple corpus poisoning and prior RAG attacks achieve limited success, while gradient-based retriever attacks (CLIP PGD) increase ASR but do not consistently translate to generation corruption. In contrast, multimodal knowledge poisoning methods such as Poisoned-MRAG achieve near-perfect ASR-R and ASR-G, demonstrating that poisoned retrieval can reliably induce semantic errors in downstream generation. Our poisoning setup exhibits similarly high attack effectiveness, achieving near-complete corruption at both retrieval and generation stages. These results highlight the severity of retrieval poisoning in Vision RAG systems and motivate the need for the semantic coherence defense proposed in this work.

Effect of Semantic Coherence Refinement under Poisoned Retrieval

Figure 3 illustrates the direct impact of retrieval poisoning on Vision RAG–based image generation and the corrective effect of the proposed semantic coherence module. Under poisoned retrieval (left), the retriever returns visually plausible but semantically incorrect exemplars (e.g., a cat retrieved for the concept “dog”), which are then propagated into the diffusion model, resulting in a logically inconsistent generation that violates the prompt semantics. This example highlights how retrieval-level corruption directly translates into generation-level semantic errors, even when the diffusion backbone itself is unchanged.

After applying the proposed score-based semantic coherence check (right), retrieved images are explicitly verified against the prompt semantics. Misaligned exemplars are detected through low coherence scores, filtered or corrected via prompt refinement and re-retrieval, and replaced with semantically consistent visual context. As a result, the diffusion model is conditioned on verified retrieval evidence and generates an image that correctly reflects both the entities and their intended interactions in the prompt. This qualitative comparison demonstrates that enforcing semantic coherence at the retrieval stage effectively blocks the propagation of poisoned knowledge into generation, complementing the quantitative robustness gains observed in Table 1 and Table 2.

Quality of Generated Image

Table 1 reports quantitative results on the COCO [33] test set, comparing diffusion-based image generation models with and without retrieval augmentation. While retrieval-based methods already outperform non-retrieval baselines by incorporating external visual context, their performance critically depends on the semantic consistency of the retrieved images. Our method builds upon retrieval-augmented generation by introducing a contextual refinement mechanism that verifies and refines retrieved visual evidence before diffusion.

Across all categories, our method achieves the highest semantic similarity (SIM) and aesthetic score (AES), with notable gains over FineRAG on Semantic, Spatial, and Mixed prompts involving complex interactions. These results show that enforcing prompt–retrieval coherence improves conditioning fidelity and generation quality, while remaining fully modular and requiring no fine-tuning of the diffusion backbone.

Qualitative Analysis of Semantic Coherence

Figure 4 presents qualitative comparisons on the COCO test set and the Multi-Entity Draw Bench. Baseline retrieval-augmented and diffusion models frequently fail to capture intended actions, rare entities, geographic context, and multi-entity interactions due to noisy or semantically inconsistent retrieval. In contrast, the proposed semantic coherence refinement framework preserves entity identities, actions, and relationships by filtering and correcting misaligned visual evidence before diffusion.

4.3. Ablation Study

Figure 5 illustrates the impact of retrieval poisoning on semantic alignment and the effectiveness of the proposed semantic coherence refinement. Under poisoned retrieval, the baseline Vision RAG model exhibits a substantial drop in CLIP score across all prompt categories, indicating degraded prompt–image consistency. After applying the coherence refinement loop, CLIP scores consistently recover across Mixed, Numerical, Null, Semantic, and Spatial prompts, with the largest gains in the Semantic and Mixed categories involving complex interactions. These results confirm that verifying and refining retrieved visual context before diffusion effectively mitigates poisoned semantics. These improvements indicate that the benefits arise from the coherence verification and refinement process rather than from changes to the diffusion backbone itself, which remains frozen throughout all experiments.

5. Conclusions

We studied the vulnerability of Vision RAG systems to poisoned visual retrieval and showed that semantic mismatches between prompts and retrieved images can severely degrade diffusion-based generation quality. Our experiments demonstrate that retrieval poisoning achieves high attack success rates at both retrieval and generation levels, leading to systematic semantic errors. To address this issue, we proposed a score-based semantic coherence refinement module that verifies prompt-image alignment, refines misaligned prompt components, and re-retrieves corrected visual context prior to diffusion. Extensive quantitative and qualitative results show that our method substantially improves semantic correctness, robustness, and alignment under both clean and adversarial settings, establishing an effective and principled defense for Vision RAG augmented diffusion models. The proposed design is general and modular, allowing alternative coherence scoring, verification, or refinement strategies to be incorporated.