Toward Multimodal Understanding in Low-Level Vision: The Rise of Vision-Language Models

1. Introduction

Low-level vision encompasses fundamental image processing tasks that restore or enhance visual data degraded by real-world factors, including super-resolution [1,2], deblurring [3,4], dehazing [5,6], inpainting [7], fusion [8,9], and low-light enhancement [10,11]. They serve as critical preprocessing for downstream applications such as autonomous driving, medical diagnosis [12], remote sensing [13], and multimedia content creation [14]. Driven by deep learning, the field has progressed from early CNNs [15,16] to Transformers with global receptive fields [17] and, more recently, diffusion models with powerful generative priors [10], improving restoration fidelity and robustness.

Despite this progress, most methods remain task-specific and data-driven, with objectives dominated by pixel-wise reconstruction error and little explicit understanding of scene structure, object relations, or contextual intent. As a result, they generalize poorly to complex or hybrid degradations beyond the training distribution, yielding inconsistent perceptual quality on in-the-wild data [18,19]. This semantic gap, i.e., insufficient global understanding within pixel-oriented paradigms, has become a central bottleneck in pushing low-level vision toward higher-level visual reasoning.

Vision-Language Models (VLMs), such as CLIP [20], BLIP [21], and LLaVA [22], offer a transformative remedy. Trained on large-scale image–text data, they acquire rich visual–semantic knowledge and strong zero-shot generalization [20,23], motivating their use in low-level vision [24,25,26] (see Figure 1). They address the limitations along three axes: semantic priors that enforce consistency in restored outputs [27,28], broad visual–linguistic knowledge that improves generalization to out-of-distribution degradations [29,30], and reasoning that enables adaptive handling of multiple co-existing degradations [31,32] (for generative MLLMs).

However, deploying VLMs here is non-trivial due to an objective mismatch: VLMs target high-level semantic understanding and image–text alignment, whereas low-level tasks demand pixel-level accuracy and spatial fidelity [33]. Their visual encoders operate at fixed low resolutions (typically $224\times 224$), and, being trained on clean RGB images, they degrade under severe noise, blur, or haze [34,35]. Domain gaps are even larger for specialized modalities such as hyperspectral remote sensing [36,37,38] or medical CT/MRI [39,40,41]. These challenges have spurred research on VLM adaptation, prompt design, and restoration-oriented architectures.

From this review, we identify two complementary paradigms for integrating VLMs into low-level vision. Direct VLM Adaptation modifies VLM architectures and training to process and restore degraded images end-to-end [42,43,44,45,46,47], through visual-encoder adaptation, language-branch modification, and prompt learning. VLM as Auxiliary keeps the VLM largely frozen and uses it as an external module in four roles: semantic provider [48,49,50], degradation interpreter [51,52,53], quality evaluator [54,55,56], and intelligent controller [18,19]. The two are not mutually exclusive and can be combined.

Despite this rapid progress, a dedicated systematic survey is still lacking. Existing VLM surveys focus on high-level tasks [57,58,59,60] and overlook the distinct challenges of low-level applications; this work fills the gap with the first comprehensive survey of VLMs across low-level vision tasks. Two contemporaneous surveys warrant explicit positioning. Jiang et al. [61] review all-in-one restoration by architecture and learning paradigm, without centering the language–vision axis; Liu et al. [62] organize language-driven restoration by three coupling levels, restricted to restoration and quality assessment. In contrast, our taxonomy is integration-centered, splitting the literature by how VLM components enter the system (Direct VLM Adaptation versus VLM as Auxiliary with its provider, interpreter, evaluator, and controller roles) and spans image fusion, medical imaging, remote sensing, video and 3D processing, and zero-shot adaptation.

Figure 2. Overview and taxonomy of vision–language model paradigms for low-level vision. The taxonomy organizes methods by their integration roles; contrastive and autoregressive generative mechanisms can occur across its categories.

Figure 2. Overview and taxonomy of vision–language model paradigms for low-level vision. The taxonomy organizes methods by their integration roles; contrastive and autoregressive generative mechanisms can occur across its categories.

Studying VLMs in low-level vision is not purely application-oriented; it also informs VLM design. The demand for pixel fidelity and fine-grained spatial consistency exposes architectural and training limitations (resolution, degradation sensitivity, and semantic–pixel misalignment), while solutions such as multi-scale visual encoding, degradation-aware feature extraction, and continuous prompt optimization point toward more versatile VLMs that support semantic understanding and fine-grained reconstruction. This survey thus advances low-level vision with VLMs while informing the broader development of general-purpose vision–language systems.

Our main contributions are summarized as follows. (1) We propose a unified taxonomy that organizes VLM-based low-level vision methods into two paradigms, Direct VLM Adaptation and VLM as Auxiliary, the latter further divided into Provider, Interpreter, Evaluator, and Controller roles, offering a clear framework for categorizing existing and future work. (2) We systematically analyze architectural adaptations and prompt learning strategies that connect VLMs’ high-level semantics with the pixel-level accuracy required in low-level tasks. (3) We extend the discussion beyond natural images to specialized domains such as medical imaging and remote sensing, highlighting domain-specific challenges and adaptations. (4) We compile experimental results on representative benchmarks, summarize open challenges, and outline future directions, with particular emphasis on emerging generative VLMs for image restoration. To facilitate ongoing research, we maintain an up-to-date repository at repository.

The remainder of this survey is organized as: Section 2 introduces the preliminaries. Section 3 and Section 4 detail the two core paradigms: Direct VLM Adaptation and VLM as Auxiliary. Section 5 explores extended applications in specialized domains, followed by experiments in Section 6. Finally, Section 7 and Section 8 discuss future directions and conclude the work.

Note. Our systematic search covers databases like DBLP and Google Scholar, emphasizing reputable venues such as IEEE T-PAMI, IJCV, and CVPR. We prioritize studies with public code and high citations as indicators of impact and practicality. Each selected work is carefully examined to assess its unique contribution and influence in the community. This allows our survey to provide a comprehensive overview of the most representative and influential research, and to highlight promising directions for future inquiry.

Figure 3. Main application methods of prompt learning. (a) learnable context vectors are adopted as prompts. This approach typically involves two stages: first, the prompt is optimized; second, the learned prompt is used for self-supervised training [27,45,66,69,70,71,72,73]; (b) prompts are simultaneously set in both visual and linguistic modalities [37,173].

Figure 3. Main application methods of prompt learning. (a) learnable context vectors are adopted as prompts. This approach typically involves two stages: first, the prompt is optimized; second, the learned prompt is used for self-supervised training [27,45,66,69,70,71,72,73]; (b) prompts are simultaneously set in both visual and linguistic modalities [37,173].

2. Preliminaries

2.1. Vision-Language Models: A Brief Overview

VLMs are multimodal systems that model images and text in a shared embedding space, enabling image–text matching and cross-modal retrieval. A typical VLM comprises image and text encoders that map visual and textual inputs into aligned embeddings and, with large-scale contrastive pre-training, exhibit transferability and zero-shot generalization.

We use VLM as an umbrella term and distinguish two operating mechanisms relevant to this survey. Contrastive (dual-encoder) VLMs, represented by CLIP, map images and text into aligned embedding spaces for similarity estimation, retrieval, classification, and feature conditioning. Autoregressive generative VLMs, particularly multimodal large language models (MLLMs) such as LLaVA, connect visual representations to a language model and generate text for description, instruction following, reasoning, and tool use. Models such as BLIP and BLIP-2 support both contrastive representation learning and text generation; we describe them according to the capability used by each reviewed method. This mechanism axis is orthogonal to our integration-centered taxonomy.

2.1.1. Network Architectures

Given a dataset of paired samples $\mathcal{D}={\{x_n^I,x_n^T\}}_{n=1}^N$, an image encoder $f_{\theta }$ and a text encoder $f_{\varphi }$ map each image and its caption to embeddings $z_n^I=f_{\theta }\left(x_n^I\right)$ and $z_n^T=f_{\varphi }\left(x_n^T\right)$.

Visual branch. Two families of encoders are widely used: CNN-based and Transformer-based. CNN backbones such as VGG [174], ResNet [175], and EfficientNet [176] are often adopted with minor changes for vision–language pre-training [20]. Vision Transformers (ViT) [177] instead tokenize an image into patches and process them via stacked Transformer blocks, capturing long-range dependencies in a global token space. Later works [20,178,179] introduce refinements such as pre-encoder normalization to improve stability and representation quality under large-scale VLM pre-training.

Language branch. Transformers and their variants [180,181,182] dominate text encoding. The standard Transformer consists of multi-head self-attention and multilayer perceptron (MLP) blocks in both encoder and decoder. Most VLMs, e.g., CLIP [20], use Transformers similar to GPT-2 [182], sometimes initialized from GPT-like checkpoints, while others train the text encoder from scratch.

2.1.2. Training Strategy

Contrastive VLMs and generative MLLMs are trained differently. Contrastive VLMs such as CLIP are pre-trained on large image-text corpora with a contrastive objective that pulls matched image-text pairs together and pushes mismatched pairs apart in a shared embedding space, yielding transferable representations without generating text. Generative MLLMs instead follow three stages:

Pre-training. Pre-training aims to align heterogeneous modalities and acquire broad world knowledge using large-scale image–text pairs. A common paradigm involves autoregressively predicting textual descriptions from input images via standard cross-entropy loss. For optimization, many frameworks freeze pre-trained components (e.g., the visual encoder and LLM) and train only a lightweight bridging module to preserve inherent knowledge [149,183]. Conversely, other approaches [184,185,186] opt to unfreeze specific components, such as the visual encoder, to enable stronger cross-modal adaptation at the cost of increased trainable parameters.

Instruction-tuning. This stage enhances zero-shot generalization by optimizing the model to generate a response $\mathcal{R}$ conditioned on an instruction $\mathcal{I}$ and multimodal input $\mathcal{M}$. Formally, given a triplet $(\mathcal{I},\mathcal{M},\mathcal{R})$, the MLLM is trained to minimize the standard autoregressive loss [187,188]:

$$\mathcal{L}\left(\theta \right)=-{\displaystyle \sum _{i=1}^N}logp({\mathcal{R}}_i\mid \mathcal{I},\mathcal{M},{\mathcal{R}}_{<i};\theta ),$$

(1)

where $\theta$ represents model parameters and ${\mathcal{R}}_{<i}$ denotes preceding tokens. This process enables the model to generalize to unseen tasks by following natural language instructions.

Preference Alignment. This stage aligns generative MLLMs with human preferences (e.g., reducing hallucination). Reinforcement Learning from Human Feedback (RLHF) [189] utilizes a learned reward model to guide optimization [190], whereas Direct Preference Optimization (DPO) [191] simplifies the pipeline by directly learning from preference data without an explicit reward model. Recently, Group Relative Policy Optimization (GRPO) [192] has emerged to further enhance efficiency; it eliminates the critic (value function) by optimizing based on relative rewards within a sampled group of outputs, thereby significantly reducing memory overhead.

2.2. Low-Level Vision: Formulation

Low-level vision tasks focus on restoring or enhancing degraded visual signals, prioritizing pixel-level reconstruction and structural fidelity over semantic understanding. The primary objective is to recover the clean image $x_0$ from a degraded observation y. Representative tasks include image denoising [193], deblurring [4], deraining [194], dehazing [6], and super-resolution [1]. Mathematically, these tasks share a unified forward degradation model:

$$y=D\left(x_0\right)+n,$$

(2)

where $x_0$ and y denote the clean high-quality (HQ) and observed low-quality (LQ) images, respectively. $D(·)$ represents the degradation operator, encompassing factors such as kernel blur, downsampling, atmospheric scattering, or illumination attenuation, while n denotes random perturbations like sensor noise. Reconstructing $x_0$ from y constitutes an inherently ill-posed inverse problem characterized by non-uniqueness and high sensitivity to noise. In real-world scenarios, $D(·)$ is often unknown or difficult to model accurately, further exacerbating the challenge by limiting available constraints.

3. Direct VLM Adaptation for Low-Level Vision

3.1. Visual Encoder Adaptation: Handling Details

Recent work adapts VLM visual branches via patching (to bypass fixed-resolution limits) and controller modules (for degradation awareness).

Resolution Scaling. Input resolutions in low-level vision often exceed standard VLM limits (e.g., 224×224). Resolution-scaling strategies use patching or tiling to process high-resolution images without modifying the backbone, partitioning the image into encoder-sized patches and then recomposing features or retrieving across patches to preserve context and detail. Zhang et al. [63] address structural inconsistencies in 4K inpainting by injecting CLIP-retrieved reference features into attention layers for consistency. ImageRAG [42] handles ultra-high-resolution remote sensing by formulating a “long-context selection” problem. FeatSharp [43] and NVILA [64] encode partitioned tiles and stitch them into a unified feature map, capturing both global context and local structure.

Table 1. Instruction tuning for restoration in language branch.

Dataset	Year	Related Method	Short Description	Type
Paired training samples [75]	2024	RestoreAgent [75]	A 23K-pair instruction dataset labeled with the optimal restoration pipeline	Real
CleanBench [74]	2025	JarvisIR [74]	150K synthetic pairs and 80K real degraded images accompanied by self-constructed instruction–response pairs	Real and Syn
SlowAgent [32]	2025	HybridAgent [32]	A total of 70K pairs composed of degraded images, predicted degradation types, and tool-calling instructions.	Syn
FeedbackAgent [32]	2025	HybridAgent [32]	66K pairs used to determine whether an image is clean and whether further restoration is required.	Syn
Low-level vision VQA dataset [77]	2024	UniProcessor [77]	Covering 30 degradation types and over 70K image patches, with diverse QA instructions generated via templates	Syn
InstructIR Prompt Set [99]	2024	InstructIR [99]	An instruction set with 10K+ distinct prompts covering 7 restoration/ enhancement tasks	Syn
PromptFix Dataset [88]	2024	PromptFix [88]	A 1,013,320-sample instruction-following dataset spanning 7 restoration/ editing tasks (e.g., dehazing, low-light, deblurring)	Real and Syn
PixTalk Dataset [195]	2025	PixTalk [195]	A 106K RAW↔sRGB paired dataset with language prompts for photorealistic image processing and editing	Real
Tri-IR [196]	2025	InstructRestore [196]	A 536,945-triplet dataset consisting of a high-quality image, a target-region description, and a region mask for region instruction restoration	Real and Syn
Pico-Banana-400K [197]	2025	Pico-Banana-400K [197]	A 400K instruction-based image editing dataset built on real photos, paired with edited results for instruction-following enhancement	Real and Syn

Table 1. Instruction tuning for restoration in language branch.

Dataset	Year	Related Method	Short Description	Type
Paired training samples [75]	2024	RestoreAgent [75]	A 23K-pair instruction dataset labeled with the optimal restoration pipeline	Real
CleanBench [74]	2025	JarvisIR [74]	150K synthetic pairs and 80K real degraded images accompanied by self-constructed instruction–response pairs	Real and Syn
SlowAgent [32]	2025	HybridAgent [32]	A total of 70K pairs composed of degraded images, predicted degradation types, and tool-calling instructions.	Syn
FeedbackAgent [32]	2025	HybridAgent [32]	66K pairs used to determine whether an image is clean and whether further restoration is required.	Syn
Low-level vision VQA dataset [77]	2024	UniProcessor [77]	Covering 30 degradation types and over 70K image patches, with diverse QA instructions generated via templates	Syn
InstructIR Prompt Set [99]	2024	InstructIR [99]	An instruction set with 10K+ distinct prompts covering 7 restoration/ enhancement tasks	Syn
PromptFix Dataset [88]	2024	PromptFix [88]	A 1,013,320-sample instruction-following dataset spanning 7 restoration/ editing tasks (e.g., dehazing, low-light, deblurring)	Real and Syn
PixTalk Dataset [195]	2025	PixTalk [195]	A 106K RAW↔sRGB paired dataset with language prompts for photorealistic image processing and editing	Real
Tri-IR [196]	2025	InstructRestore [196]	A 536,945-triplet dataset consisting of a high-quality image, a target-region description, and a region mask for region instruction restoration	Real and Syn
Pico-Banana-400K [197]	2025	Pico-Banana-400K [197]	A 400K instruction-based image editing dataset built on real photos, paired with edited results for instruction-following enhancement	Real and Syn

Figure 4. Illustration of combining frozen VLM semantic features with low-level features via trainable controlnet-like structures.

Figure 4. Illustration of combining frozen VLM semantic features with low-level features via trainable controlnet-like structures.

Feature Fusion. VLMs pre-trained on clean images lack robustness to degradation, so their visual features often misalign with textual descriptions in restoration tasks. Recent works inject degradation awareness into the visual branch. Several approaches [29,44] add a parallel image controller (Figure 4) that encodes degradation embeddings $e_d^I$ from low-quality inputs and fuses them with semantic priors. The primary encoder stays frozen while the controller is fine-tuned, connected by zero-initialized layers for adaptation [198] and optimized with a joint contrastive loss ${\mathcal{L}}_{\mathrm{con}}$:

$${\mathcal{L}}_c\left(\omega \right)={\mathcal{L}}_{\mathrm{con}}(e_c^I,e_c^T;\omega )+{\mathcal{L}}_{\mathrm{con}}(e_d^I,e_d^T;\omega ),$$

(3)

where $\omega$ denotes the controller parameters, $e_c^I/e_d^I$ correspond to content/degradation embeddings, and $e_c^T/e_d^T$ are embeddings from the clean caption and degradation label.

3.2. Language Branch Adaptation: Bridging Modalities

Standard VLM language branches, built for high-level semantic matching, lack fine-grained degradation information and sensitivity to quality variations. Two adaptation strategies address this: soft prompt tuning, which injects learnable embeddings to steer the semantic space, and instruction tuning, which fine-tunes the model on restoration-specific tasks.

Prompt Learning Strategies. Soft prompt tuning injects continuous, learnable context vectors into the language branch, optimized by gradient descent to capture degradation-specific priors without modifying the core architecture (Figure 3). A common paradigm is two-stage: first learn quality-distinguishing prompts, then use them to guide restoration [27,69]. Given a degraded image x encoded as $E_I\left(x\right)$ and prompt vectors t for different image characteristics, the matching probability for the i-th prompt is a softmax-normalized cosine similarity:

$${\widehat{y}}_i={\displaystyle \frac{exp\left(cos\left(E_I\left(x\right),E_T\left(t_i\right)\right)\right)}{{\sum }_{j=1}^{N}exp\left(cos\left(E_I\left(x\right),E_T\left(t_j\right)\right)\right)}}.$$

(4)

The prompts are trained with the cross-entropy loss. During restoration, contrastive learning aligns the restored output with the “clear” prompt embedding. Some methods [37,173] prompt both visual and textual modalities to maintain alignment. Yang et al. [173] address the feature shift from uni-modal prompting by modeling the layer-wise shift ${\mathsf{\Omega }}_l$.

Instruction Tuning for Restoration. Instruction tuning finetunes VLMs on restoration-centric image–text–image triplets, enabling the language encoder to perceive diverse degradation patterns and understand specific restoration goals; representative datasets are summarized in Table 1. Existing approaches fall into three categories: (1) Agent Instruction Tuning, where the VLM acts as a controller to output restoration task signals [32,74] (Section 4.4); (2) Conditional Generation, where the language model produces soft embeddings to guide a diffusion-based restoration model [76]; and (3) Degradation Prediction, where the VLM predicts degradation labels for a downstream reconstruction network [77] (Section 4.2).

Table 2. Representative text generation methods.

Generation approach	Method	Explanation	Utilized VLMs
Direct Generation	RAP-SR [54]	Automatically generates image descriptions using Florence-2.	Florence-2 [199]
	RamIR [200]	Generates degradation ($p_d^{emb}$) and clean ($P_c^{emb}$) embeddings via BLIP.	BLIP [21]
	CyclicPrompt [102]	Encodes BLIP descriptions into $p_t$, combining learnable $p_i$ and visual $p_k$.	BLIP [21]
	LDR [51]	Describes weather types and occluded regions via language instructions.	LISA [201]
	Diff-Dehazer [202]	Derives positive prompts by removing “haze” from BLIP-2 descriptions.	BLIP-2 [203]
	DA-CLIP [29]	Uses triplets of HQ images, descriptive texts, and degradation labels.	BLIP [21]
Progressive generation	PBI [48]	CogVLM describes masked objects; Mistral-7B rewrites them as addition instructions.	CogVLM [204], Mistral-7B [205]
	MTGFusion [49]	Encodes six types of GPT-4 text prompts via BLIP into vectors.	GPT-4 [206], BLIP [21]
	Chen’s Method [50]	Fuses coarse-grained (BLIP-2) and fine-grained (MiniGPT-4) descriptions via templates.	BLIP-2 [203], MiniGPT-4 [207]
Multi level generation	XPSR [89]	Generates high-level semantic and low-level detail descriptions via LLaVA.	LLaVA [22]
	PASD [93]	Encodes object/location words (ResNet/YOLO) and scene contents (BLIP).	BLIP [21]
	MTGFusion [49]	Generates separate source, shared, and unique descriptions.	GPT-4 [206], BLIP [21]
	FILM [95]	Combines scene (BLIP-2), object (GRIT), and mask (SAM) prompts for ChatGPT generation.	BLIP-2 [203], ChatGPT [208]
Region level generation	DreamPaint [209]	Generates descriptive text based on Kosmos-2 object detection boxes.	LLaVA-1.6-Vicuna-7B [187]
	Anywhere [110]	Generates text descriptions for foreground regions extracted by RMBG-1.4.	Gemini-Pro-Vision [210]

Table 2. Representative text generation methods.

Generation approach	Method	Explanation	Utilized VLMs
Direct Generation	RAP-SR [54]	Automatically generates image descriptions using Florence-2.	Florence-2 [199]
	RamIR [200]	Generates degradation ($p_d^{emb}$) and clean ($P_c^{emb}$) embeddings via BLIP.	BLIP [21]
	CyclicPrompt [102]	Encodes BLIP descriptions into $p_t$, combining learnable $p_i$ and visual $p_k$.	BLIP [21]
	LDR [51]	Describes weather types and occluded regions via language instructions.	LISA [201]
	Diff-Dehazer [202]	Derives positive prompts by removing “haze” from BLIP-2 descriptions.	BLIP-2 [203]
	DA-CLIP [29]	Uses triplets of HQ images, descriptive texts, and degradation labels.	BLIP [21]
Progressive generation	PBI [48]	CogVLM describes masked objects; Mistral-7B rewrites them as addition instructions.	CogVLM [204], Mistral-7B [205]
	MTGFusion [49]	Encodes six types of GPT-4 text prompts via BLIP into vectors.	GPT-4 [206], BLIP [21]
	Chen’s Method [50]	Fuses coarse-grained (BLIP-2) and fine-grained (MiniGPT-4) descriptions via templates.	BLIP-2 [203], MiniGPT-4 [207]
Multi level generation	XPSR [89]	Generates high-level semantic and low-level detail descriptions via LLaVA.	LLaVA [22]
	PASD [93]	Encodes object/location words (ResNet/YOLO) and scene contents (BLIP).	BLIP [21]
	MTGFusion [49]	Generates separate source, shared, and unique descriptions.	GPT-4 [206], BLIP [21]
	FILM [95]	Combines scene (BLIP-2), object (GRIT), and mask (SAM) prompts for ChatGPT generation.	BLIP-2 [203], ChatGPT [208]
Region level generation	DreamPaint [209]	Generates descriptive text based on Kosmos-2 object detection boxes.	LLaVA-1.6-Vicuna-7B [187]
	Anywhere [110]	Generates text descriptions for foreground regions extracted by RMBG-1.4.	Gemini-Pro-Vision [210]

3.3. Output Head Adaptation: From Tokens to Pixels

Tokenizer–Decoder Framework. Extending VLMs to visual generation requires bridging continuous pixels and discrete token-based LLMs via a codebook tokenizer–decoder. Yu et al. [46] address the limited capacity of traditional VQ-VAE [211] codebooks by reducing embedding dimensions. VILA-U [78] enhances capacity via residual vector quantization, jointly predicting indices from multiple residual stages. To reconcile high-level semantics with fine pixels, TokenFlow [79] shares indexing between semantic and pixel-level codebooks, while SemHiTok [80] uses a hierarchical coarse-semantic codebook with fine-grained pixel sub-codebooks.

Direct Pixel Regression Heads. Beyond discrete tokenization, an alternative line attaches lightweight regression heads that map VLM features directly to continuous pixel values, avoiding the quantization loss of codebook-based decoders. LM4LV [85] keeps the language model frozen and learns only linear adapter layers that regress pixel-level outputs, showing that a frozen LLM can support low-level reconstruction without discrete token prediction. Such heads trade the flexibility of token sequences for higher fidelity on detail-sensitive tasks and remain underexplored relative to tokenizer–decoder designs.

3.4. Parameter-Efficient Fine-Tuning (PEFT) in Restoration

Since direct VLM use is degradation-insensitive and full fine-tuning is costly, PEFT freezes most parameters and tunes lightweight modules to balance generalization and adaptation.

LoRA and Adapter Integration. Adapters [212] and LoRA [213] enable efficient fine-tuning via low-rank matrices or bottleneck layers. Inserting adapters into Transformer layers steers representations toward restoration [47]:

$$\begin{array}{cc}\hfill x_i^{\prime }& =\mathrm{Adapter}\left(\mathrm{MSA}\left(\mathrm{LN}\left(x_{i-1}\right)\right)\right)+x_{i-1},\hfill \\ \hfill x_i& =\mathrm{MLP}\left(\mathrm{LN}\left(x_i^{\prime }\right)\right)+s·\mathrm{Adapter}\left(\mathrm{LN}\left(x_i^{\prime }\right)\right)+x_i^{\prime },\hfill \end{array}$$

(5)

where s is a scaling factor weighting the adapter output. Adapter-based methods also extend pretrained diffusion priors: BIR-Adapter [82] augments diffusion self-attention with a parameter-efficient restoring-attention mechanism for blind restoration without auxiliary extractors; TADiSR [83] adapts diffusion cross-attention layers; and LoRA-IR [35] maps CLIP features to degradation types via fine-tuned experts.

Freezing Strategies. Strategic freezing preserves pre-trained knowledge while allowing adaptation. LM4LV [85] freezes the LLM and tunes only linear adapters for pixel-level details, and Pang et al. [86] show that frozen LLM Transformer blocks can serve as effective visual encoders when paired with trainable linear projection layers.

4. VLM as Auxiliary for Low-Level Vision

As auxiliary modules, VLMs enhance restoration networks with minimal tuning in four roles mapping onto Equation (2): Semantic Providers shape the prior on $x_0$; Degradation Interpreters estimate $D(·)$ and n; Quality Evaluators supply a no-reference fidelity surrogate; and Intelligent Controllers select the inversion sequence, dividing labor over prior, forward model, objective, and solver. These roles describe what a model does, not how it is implemented: contrastive VLMs supply embeddings, similarities, or probabilities, whereas generative MLLMs produce descriptions, judgments, reasoning, or tool decisions. Both span the provider, interpreter, and evaluator roles; the controller, generative by nature, is MLLM-based as a rule, with contrastive models entering only as scoring sub-components it reads.

4.1. VLM as Semantic Provider

VLM-based semantic generation falls into four categories, summarized in Table 2. First, direct generation constructs explicit image-text pairs to train restoration models, using VLMs such as BLIP [29,44], LLaVA [214], and mPLUG-Owl2 [54] to describe content, scene characteristics, and degradation types; text-only LLMs can further synthesize comprehensive descriptions from image-derived prompts, as in FILM [95]. Second, progressive generation [48,49,50] chains multiple VLMs into a coherent semantic pathway for higher-quality descriptions. Third, multi-level generation [89,92,93,215,216,217] produces prompts capturing both high-level semantics (scene categories, spatial layout) and fine-grained details. Finally, region-level generation [107,110,209] segments images to generate localized descriptions, capturing local features and spatial relationships.

Text-Conditioned Injection. Most works extract semantic or degradation-related embeddings from CLIP, while others use multimodal models for prompts. Diff-Restorer [90] disentangles semantic and degradation features from CLIP, whereas XPSR [89] uses a multimodal model for hierarchical prompts. Injection typically employs cross-attention: VPIP [87] integrates task-specific prompt maps at the U-Net bottom; PromptFix [88] injects user instructions and auxiliary prompts into cross-attention layers (Figure 5); and XPSR [89] uses a Semantic Attention module. To prevent degradation contamination, Diff-Restorer [90] decomposes CLIP embeddings into content and degradation components, and DPIR [91] combines textual prompts with global-local visual prompts as control for diffusion-transformer restoration.

Language-Driven Manipulation. VLMs enable restoration strength and style adjustment via natural-language or numerical prompts. For dynamic routing, LDR [51] uses a degradation map to guide a Mixture-of-Experts for kernel selection, while InstructIR [99] uses an Instruction Condition Block for soft gating. For strength control, SPIRE [100] embeds explicit degradation parameters (e.g., sigma values) into prompts for continuous modulation, and PromptSR [101] allows discrete intensity selection (e.g., heavy/light). Liao et al. [102] extract weather-independent priors to update degradation prompts.

Subject-Aware Restoration. VLMs provide semantic priors that resolve the ill-posed nature of restoration, enabling subject-aware control. EAM [106] detects focal regions (e.g., humans) and generates descriptions, preventing background over-enhancement. HazeCLIP [107] segments sky/non-sky regions via SAM [218] for dehazing prompts. 4KAgent [108] invokes a face-repair pipeline for facial regions while preserving backgrounds (Figure 6). Beyond focal-region enhancement, ClearAIR [26] addresses the all-in-one setting with an MLLM-based quality-assessment module for global degradation evaluation and a semantic guidance unit for region-level analysis.

4.2. VLM as Degradation Interpreter

VLMs function as degradation interpreters through two mechanisms: contrastive VLMs classify or score degradation attributes, while generative MLLMs describe degradation type, severity, and spatial location or produce structured signals.

Degradation Classification. Framing VLMs as degradation classifiers yields high-level descriptors that modulate restoration. A common paradigm extracts semantic features from degraded images [35,53,113,114] and maps them to a probability distribution over degradation categories. To preserve degradation details in CLIP, Ai et al. [35] process both a downsampled image and sliding-window patches and optimized via cross-entropy. Zhang et al. [114] use Q-Align [219] to obtain token probability distributions for haze-level estimation. Alternatively, some methods map VLM-extracted tokens into visual embedding spaces [51,70,111,112]: Wen et al. [70] map degradation type and attribute prompts directly to high-dimensional vectors; Yang et al. [51] use CoT prompts to generate descriptions projected into a degradation prior $p_{emb}$; Tan et al. [112] compute layer-wise HQ–LQ CLIP differences for pixel-level and degradation-type priors; and PTG-RM [52] extracts implicit priors to learn a pixel-wise restoration intensity map.

Description-based Restoration. VLMs also evaluate attributes such as brightness and semantic correlations. In Figure 7, GPP-LLIE [119] uses the probability difference between positive and negative tokens to derive continuous scores for contrast, visibility, and sharpness: $z=-(P_{\mathrm{pos}}-P_{\mathrm{neg}})/\alpha ,S={(1+exp\left(z\right))}^{-1}$. LLMCO4MR [120] aligns ancient manuscripts using VLM confidence scores; VLU-Net [53] integrates VLM-derived degradation descriptions into a deep unfolding framework where vision-language gradient guidance drives all-in-one restoration; and PromptGIP [121] reformulates processing as QA-style visual prompts for dynamic task switching.

4.3. VLM as Quality Evaluator

VLMs increasingly serve as quality evaluators: contrastive VLMs score quality by comparing images against antonym prompts in a shared space, whereas generative MLLMs are prompted to produce attribute-level quality judgments.

No-Reference Quality Assessment. VLMs evaluate restoration quality when references are unavailable. CLIP-DQA [122] computes cosine similarity between the generated image and antonymous reference prompts (e.g., “good”/“bad”) in CLIP space. Beyond natural images, EvaNet [123] extends VLM-guided assessment to image fusion via an LLM-informed perceptual assessment. Others [27,32,124,125,126] prompt VLMs to estimate quality-level probabilities and convert them to scores; e.g., Xu et al. [27] classify visibility into five levels and Ma et al. [124] use GPT-4o to score underwater attributes. Aesthetic assessment is also explored [54,56,127]: Wang et al. [54] use mPLUG-Owl2 for aesthetic scoring, and Yin et al. [56] train a reward model from BLIP and human preference data. Li et al. [131] propose R4-CGQA, a CG-specific quality dataset with a two-stream retrieval framework that supplies visually similar CG images.

Semantic Consistency Loss. While VGG-based perceptual loss [220] captures local textures, VLMs encode high-level semantics and aesthetic preferences. CLIP-IQA [129] shows zero-shot assessment, inspiring perceptual distances: CDASR [135] minimizes super-resolution distance in CLIP space, Perceive-IR [73] (Figure 8) uses a CLIP-aware loss with fixed quality prompts (“excellent”/“poor”), and RAVE [136] introduces a CLIP-feature identity loss. These semantic losses are typically combined with pixel-level and VGG losses.

Feedback Loops. VLM-derived quality scores increasingly serve as rewards in reinforcement learning (RL) loops. CURVE [137] computes rewards via CLIP embedding distances; RealSR-R1 [138] uses GRPO [192] with degradation, comprehension, and generation rewards; and LRPO [139] integrates human-preference, perceptual-quality (CLIP-IQA), and fidelity rewards. Xu et al. [140] use a VLM-trained IQA model as the reward with a difficulty-adaptive strategy.

4.4. VLM as Intelligent Controller

Recent studies position MLLMs as central controllers in agent frameworks for multi-degradation restoration. Because planning, tool selection, and reflection are generative by nature, the controller itself is an MLLM, and any contrastive model enters only as a quality-scoring sub-component.

Motivation. Natural images often suffer from concurrent degradations. “All-in-One” models [35,221] address diverse types but trade generalization for accuracy, whereas combining specialized models requires precise orchestration, since improper sequencing can introduce artifacts [75]. MLLMs act as intelligent controllers to select and order these models.

Tool Usage and Orchestration. Agent-based methods [74,75,108,141] structure restoration into degradation perception, sequence optimization, model selection, and execution (Figure 9). Given degradation types $\mathcal{D}=\{d_1,\dots ,d_n\}$ and a model library $\mathcal{M}$, the objective is an optimal sequence $\beta =(M_{d_1}^{r_1},\dots ,M_{d_m}^{r_m})$ maximizing quality $E(I,\beta )$:

$${\beta }^{*}=arg\underset{\beta \in \mathcal{S}(\mathcal{D},\mathcal{M})}{max}E(I,\beta ).$$

(6)

Rollback mechanisms mitigate error propagation. Improvements include standardized metrics [75], hierarchical multi-agent designs [31] distinguishing compression, imaging, and scene degradations, and CoT reasoning with greedy search for efficiency [19]. Beyond orchestration, controller reliability can be improved by training: JarvisIR [74] uses supervised fine-tuning on CleanBench followed by human-feedback alignment, which we treat as a canonical mitigation for the text-hallucination failure mode in Section 6.

Figure 9. A typical pipeline for agent-based image restoration, where dashed lines indicate optional steps [18,32,75].

Figure 9. A typical pipeline for agent-based image restoration, where dashed lines indicate optional steps [18,32,75].

Iterative Refinement. VLMs refine decisions through multi-turn human interaction. ChatEdit [142] tracks user requests via a dialogue module while an editing module applies modifications to original images to avoid error accumulation; MagicQuill [143] uses a brush-based paradigm that, after each stroke, infers user intent for real-time multi-turn editing.

5. Extended Applications

5.1. Medical Image Processing

Biomedical VLMs. Medical image acquisition relies on complex, physically constrained processes [224], making quality sensitive to hardware and clinical conditions such as radiation-dose limits, and demanding restoration that preserves clinically relevant semantics. While general-domain VLMs are limited by semantic discrepancies in biomedical data, domain-specific pretraining [40,41,225] and large-scale datasets such as PMC-15M [225] have made biomedical vision–language learning feasible (Table 3).

Li et al. [149] proposed LLaVA-Med, pioneering biomedical instruction tuning via curriculum learning. Alkhaldi et al. [146] introduced MiniGPT-Med, a unified interface for report generation and disease detection. VILA-M3 [145] enhances reasoning with feedback from expert models such as VISTA3D [226]. Lin et al. [152] proposed HealthGPT, the first unified VLM for both understanding and generation across modalities, and Chen et al. [151] presented HuatuoGPT-Vision for Chinese medical tasks, supported by PubMedVision.

CT and MRI. MRI encodes hydrogen-proton concentration for soft-tissue visualization, whereas CT encodes tissue density. Chen et al. [156] proposed LangMamba for low-dose CT (LDCT) denoising (Figure 10): a Language-guided AutoEncoder with a frozen medical VLM maps normal-dose CT into an anatomy-enriched semantic space, coupled with a semantic-enhanced denoiser and a dual-space alignment loss. For metal artifact reduction, DuDoCROP [157] uses LLaVA-1.5 [22] for structured descriptions and CLIP for embedding alignment. SCAN-PhysFed [227] uses MiniGPT-Med [146] to extract anatomical information from radiology reports for federated LDCT reconstruction. For fusion, Dong et al. [154] use BLIP-2 embeddings to guide a Restormer-based [17] process. Kim et al. [228] use language embeddings for controllable brain-MRI synthesis across modalities such as FLAIR and T1.

5.2. Remote Sensing Data Processing

Heterogeneous satellite sensors yield increasingly large-scale, diverse data well-suited to the perceptual and reasoning capabilities of VLMs. We organize VLM-based methods into (i) spatial-domain tasks, improving spatial fidelity (e.g., super-resolution, restoration), and (ii) spectral-domain tasks, recovering and enhancing spectral information.

Spatial-domain tasks. VLMs mainly provide semantic evaluation and guidance for spatial restoration and super-resolution. Wolters et al. [55] couple ESRGAN [229] with CLIP-based similarity as a perceptual alternative to PSNR. In Figure 11, Wu et al. [158] integrate CLIP features and learn a degradation vector through wavelet patch contrastive learning, mapped by an MLP to modulation parameters. Chen et al. [159] apply LoRA to adapt CLIP for module-wise semantic guidance maps. PromptHSI [38] converts text prompts into a task descriptor for frequency-domain modulation, whereas MP-HSIR [37] replaces CLIP visual features with learnable visual tokens fused with text prompts.

Spectral-domain tasks. Spectral-domain restoration reconstructs rich hyperspectral signatures from limited spectral bands, a severely ill-posed mapping where VLM priors help disambiguate the few observed channels. Chen et al. [36] proposed LAVER, which modifies CLIP by adding interpolated positional encodings, enabling resolution-flexible semantic guidance for RGB-to-HSI reconstruction.

5.3. Spatiotemporal and Geometric Modality Processing

This section reviews emerging modalities where VLM-based low-level processing remains underexplored.

CAD. CAD drawings are often corrupted by software-version incompatibilities. Tang et al. [33] proposed ChatCAD, the first zero-shot VLM-based CAD restoration system, which simulates the industrial drawing-review workflow via a multi-agent framework that checks engineering-logic consistency and revises drawings when inconsistencies arise.

Table 4. Common dataset for low-level vision tasks. “Size” reports the official train/validation/test split when available; otherwise, it reports the dataset size or the train/test split specified in the corresponding source.

Tasks	Datasets	Size	Source	Type	Description
SR	DIV2K [230]	800/100/100	NTIRE 2017	Syn	A high-resolution image benchmark containing diverse natural scenes with synthetically generated degradations.
	RealSR [231]	595	ICCV 2019	Real	A real-capture dataset obtained by varying the camera focal length under controlled settings.
	DRealSR [232]	2,507	ECCV 2020	Real	A real-world dataset acquired with focal-length adjustments across indoor and outdoor environments.
LLIE	LOLv1 [233]	485/15	BMVC 2018	Real	A real-world paired low-/normal-light dataset collected under diverse illumination conditions.
	LIME [234]	10	TIP 2016	Real	A real-world low-light image set without paired ground truth, used for no-reference low-light enhancement evaluation.
	LOLv2 [235]	1,589/200	TIP 2021	Real&Syn	A combined dataset including real pairs with multi-stage imaging and synthetic pairs using luminance transformations.
Dehaze	RESIDE [236]	13,000/990	TIP 2019	Real&Syn	A dehazing benchmark with synthetic and real subsets, covering diverse indoor/outdoor conditions.
	NH-Haze [237]	55	CVPRW 2020	Real	A real-captured outdoor dataset using controlled haze machines, providing paired hazy/clear images.
	Haze-4K [238]	4,000	MM 2021	Syn	A synthetic benchmark including paired images with their transmission maps and atmospheric light annotations.
Inpainting	CelebA [239]	200,000	ICCV 2015	Real	A large-scale face benchmark with over 10,000 identities, used in semantic inpainting and human-centric restoration.
	CelebA-HQ [240]	30,000	ArXiv 2017	Real	A high-quality subset of CelebA produced through super-resolution, suitable for high-fidelity face restoration tasks.
	MSCOCO [241]	328,124	ECCV 2014	Real	A general-purpose benchmark offering diverse scenes across 91 object categories with segmentation annotations.
Derain	Rain100H [242]	1,800/100	CVPR 2017	Syn	A synthetic benchmark simulating heavy rain streaks with multiple directional patterns for supervised deraining.
	RainDrop [243]	861/239	CVPR 2018	Real	A real-world dataset by capturing paired images through dual glass surfaces, modeling adherent raindrop.
	GT-RAIN [244]	26,124/2,100	ECCV 2022	Real	A real paired deraining dataset with ground-truth clean/rainy pairs captured under controlled non-rain variations.
Deblur	GoPro [245]	2,103/1,111	CVPR 2017	Syn	A motion-blur benchmark with high-frame-rate video capture, approximating camera shake via temporal integration.
	RealBlur [246]	3,758/980	ECCV 2020	Real	A motion blur dataset acquired using paired short-/long-exposure imaging in both RAW and JPEG formats.
	HIDE [247]	8,422	ICCV 2019	Syn	A human-centered dynamic-scene dataset with both near-field and far-field motion.
CT denoising	2016 NIH-AAPM-Mayo [248]	5,936	Med. Phys. 2017	Real	A dataset with full-dose CT scans from 10 patients, widely used in dose-reduction and denoising studies.
	Mayo-2016 [249]	4,800/1,136	Med. Phys. 2021	Real	A dataset comprising abdominal CT images with normal-dose and simulated quarter-dose ones from 10 subjects.
	Mayo-2020 [249]	2,400/580	Med. Phys. 2021	Real	A dataset with abdominal CT images from 100 patients reconstructed at $25\%$ of the routine radiation dose.

Table 4. Common dataset for low-level vision tasks. “Size” reports the official train/validation/test split when available; otherwise, it reports the dataset size or the train/test split specified in the corresponding source.

Tasks	Datasets	Size	Source	Type	Description
SR	DIV2K [230]	800/100/100	NTIRE 2017	Syn	A high-resolution image benchmark containing diverse natural scenes with synthetically generated degradations.
	RealSR [231]	595	ICCV 2019	Real	A real-capture dataset obtained by varying the camera focal length under controlled settings.
	DRealSR [232]	2,507	ECCV 2020	Real	A real-world dataset acquired with focal-length adjustments across indoor and outdoor environments.
LLIE	LOLv1 [233]	485/15	BMVC 2018	Real	A real-world paired low-/normal-light dataset collected under diverse illumination conditions.
	LIME [234]	10	TIP 2016	Real	A real-world low-light image set without paired ground truth, used for no-reference low-light enhancement evaluation.
	LOLv2 [235]	1,589/200	TIP 2021	Real&Syn	A combined dataset including real pairs with multi-stage imaging and synthetic pairs using luminance transformations.
Dehaze	RESIDE [236]	13,000/990	TIP 2019	Real&Syn	A dehazing benchmark with synthetic and real subsets, covering diverse indoor/outdoor conditions.
	NH-Haze [237]	55	CVPRW 2020	Real	A real-captured outdoor dataset using controlled haze machines, providing paired hazy/clear images.
	Haze-4K [238]	4,000	MM 2021	Syn	A synthetic benchmark including paired images with their transmission maps and atmospheric light annotations.
Inpainting	CelebA [239]	200,000	ICCV 2015	Real	A large-scale face benchmark with over 10,000 identities, used in semantic inpainting and human-centric restoration.
	CelebA-HQ [240]	30,000	ArXiv 2017	Real	A high-quality subset of CelebA produced through super-resolution, suitable for high-fidelity face restoration tasks.
	MSCOCO [241]	328,124	ECCV 2014	Real	A general-purpose benchmark offering diverse scenes across 91 object categories with segmentation annotations.
Derain	Rain100H [242]	1,800/100	CVPR 2017	Syn	A synthetic benchmark simulating heavy rain streaks with multiple directional patterns for supervised deraining.
	RainDrop [243]	861/239	CVPR 2018	Real	A real-world dataset by capturing paired images through dual glass surfaces, modeling adherent raindrop.
	GT-RAIN [244]	26,124/2,100	ECCV 2022	Real	A real paired deraining dataset with ground-truth clean/rainy pairs captured under controlled non-rain variations.
Deblur	GoPro [245]	2,103/1,111	CVPR 2017	Syn	A motion-blur benchmark with high-frame-rate video capture, approximating camera shake via temporal integration.
	RealBlur [246]	3,758/980	ECCV 2020	Real	A motion blur dataset acquired using paired short-/long-exposure imaging in both RAW and JPEG formats.
	HIDE [247]	8,422	ICCV 2019	Syn	A human-centered dynamic-scene dataset with both near-field and far-field motion.
CT denoising	2016 NIH-AAPM-Mayo [248]	5,936	Med. Phys. 2017	Real	A dataset with full-dose CT scans from 10 patients, widely used in dose-reduction and denoising studies.
	Mayo-2016 [249]	4,800/1,136	Med. Phys. 2021	Real	A dataset comprising abdominal CT images with normal-dose and simulated quarter-dose ones from 10 subjects.
	Mayo-2020 [249]	2,400/580	Med. Phys. 2021	Real	A dataset with abdominal CT images from 100 patients reconstructed at $25\%$ of the routine radiation dose.

Video processing tasks. VLM-based video restoration is still nascent. Xu et al. [164] bring language prompts and VLMs to video face restoration with ControlNet [198] for prompt-free generalization, and build the first video-face degradation dataset on CelebV-HQ [250]. GSDiff [165] uses InstructBLIP [251] to describe the first frame’s degradation, supporting generic and medical video reconstruction. Since CLIP-based pulling toward a “haze-free” prompt can induce adversarial behavior (imperceptible changes causing large score swings), Ren et al. [166] (Figure 12) propose a CLIP-regularized dehazing framework that measures the mean discrepancy between hazy and clean videos to guide training.

Table 5. Results of VLM-guided SR Methods. RealSR uses a same-resolution SR, while DRealSR adopts $4\times$ SR.

Methods	RealSR [231]				DRealSR [232]				Params
	PSNR↑	SSIM↑	LPIPS↓	MUSIQ↑	PSNR↑	SSIM↑	LPIPS↓	MUSIQ↑	[M]
LR Image	27.80	0.7928	0.3356	34.40	28.02	0.8374	0.4410	20.54	N/A
CoSeR [252]	21.24	0.6109	0.2438	70.29	19.95	0.5350	0.2702	70.18	720+1936
XPSR [89]	24.19	0.6870	0.3517	70.23	26.62	0.7220	0.3864	67.84	868+1066
PASD [93]	25.93	0.7105	0.2806	65.60	29.09	0.7937	0.2893	34.63	361+1314
MegaSR [94]	23.49	0.6903	0.3072	70.03	25.74	0.7367	0.3258	64.14	—
PURE [215]	22.83	0.6079	0.3821	72.37	24.73	0.6459	0.3452	73.28	7000

Table 5. Results of VLM-guided SR Methods. RealSR uses a same-resolution SR, while DRealSR adopts $4\times$ SR.

Methods	RealSR [231]				DRealSR [232]				Params
	PSNR↑	SSIM↑	LPIPS↓	MUSIQ↑	PSNR↑	SSIM↑	LPIPS↓	MUSIQ↑	[M]
LR Image	27.80	0.7928	0.3356	34.40	28.02	0.8374	0.4410	20.54	N/A
CoSeR [252]	21.24	0.6109	0.2438	70.29	19.95	0.5350	0.2702	70.18	720+1936
XPSR [89]	24.19	0.6870	0.3517	70.23	26.62	0.7220	0.3864	67.84	868+1066
PASD [93]	25.93	0.7105	0.2806	65.60	29.09	0.7937	0.2893	34.63	361+1314
MegaSR [94]	23.49	0.6903	0.3072	70.03	25.74	0.7367	0.3258	64.14	—
PURE [215]	22.83	0.6079	0.3821	72.37	24.73	0.6459	0.3452	73.28	7000

Table 6. Results of VLM-guided Denoising Methods, where $\sigma$ denotes the noise level. Values are reported in PSNR / SSIM.

Methods	CBSD68 [253]			Urban100 [254]			Params
	$\sigma =15$	$\sigma =25$	$\sigma =50$	$\sigma =15$	$\sigma =25$	$\sigma =50$	[M]
Noised Image	24.84/0.592	20.54/0.418	15.00/0.219	24.95/0.629	20.69/0.480	15.18/0.292	N/A
InstructIR-3D [99]	34.15/0.933	31.52/0.890	28.30/0.804	34.12/0.945	31.80/0.917	28.63/0.861	16+17
TextPromptIR [255]	34.17/0.936	31.52/0.893	28.26/0.805	34.76/0.951	32.54/0.929	29.47/0.882	124+110
CLIPDenoising [256]	33.97/0.930	31.02/0.878	26.69/0.731	33.15/0.930	30.72/0.893	26.27/0.769	11+9
VLU-Net [53]	34.13/0.935	31.48/0.892	28.23/0.804	34.92/0.952	32.71/0.930	29.61/0.883	35+88
DFPIR [257]	34.32/0.934	31.71/0.897	28.49/0.814	34.79/0.952	32.57/0.930	29.53/0.883	31+63

Table 6. Results of VLM-guided Denoising Methods, where $\sigma$ denotes the noise level. Values are reported in PSNR / SSIM.

Methods	CBSD68 [253]			Urban100 [254]			Params
	$\sigma =15$	$\sigma =25$	$\sigma =50$	$\sigma =15$	$\sigma =25$	$\sigma =50$	[M]
Noised Image	24.84/0.592	20.54/0.418	15.00/0.219	24.95/0.629	20.69/0.480	15.18/0.292	N/A
InstructIR-3D [99]	34.15/0.933	31.52/0.890	28.30/0.804	34.12/0.945	31.80/0.917	28.63/0.861	16+17
TextPromptIR [255]	34.17/0.936	31.52/0.893	28.26/0.805	34.76/0.951	32.54/0.929	29.47/0.882	124+110
CLIPDenoising [256]	33.97/0.930	31.02/0.878	26.69/0.731	33.15/0.930	30.72/0.893	26.27/0.769	11+9
VLU-Net [53]	34.13/0.935	31.48/0.892	28.23/0.804	34.92/0.952	32.71/0.930	29.61/0.883	35+88
DFPIR [257]	34.32/0.934	31.71/0.897	28.49/0.814	34.79/0.952	32.57/0.930	29.53/0.883	31+63

Table 7. Results of VLM-Guided Multi-Weather Image Restoration Methods.

Methods	Snow100K-L [258]		Outdoor-Rain [259]		RainDrop [243]		Params
	PSNR↑	SSIM↑	PSNR↑	SSIM↑	PSNR↑	SSIM↑	[M]
Degraded Image	18.67	0.6468	12.89	0.5293	23.81	0.8363	N/A
MPerceiver [260]	31.11	0.9180	31.25	0.9246	33.62	0.9300	—
ADSM [70]	30.81	0.9058	31.28	0.9227	30.71	0.9188	—
VLCIR [45]	32.28	0.9296	32.62	0.9447	33.39	0.9489	—
CyclicPrompt [102]	32.16	0.9265	32.81	0.9371	32.57	0.9454	30+486
M2Restore [113]	31.20	0.9110	32.01	0.9550	31.73	0.9430	—

Table 7. Results of VLM-Guided Multi-Weather Image Restoration Methods.

Methods	Snow100K-L [258]		Outdoor-Rain [259]		RainDrop [243]		Params
	PSNR↑	SSIM↑	PSNR↑	SSIM↑	PSNR↑	SSIM↑	[M]
Degraded Image	18.67	0.6468	12.89	0.5293	23.81	0.8363	N/A
MPerceiver [260]	31.11	0.9180	31.25	0.9246	33.62	0.9300	—
ADSM [70]	30.81	0.9058	31.28	0.9227	30.71	0.9188	—
VLCIR [45]	32.28	0.9296	32.62	0.9447	33.39	0.9489	—
CyclicPrompt [102]	32.16	0.9265	32.81	0.9371	32.57	0.9454	30+486
M2Restore [113]	31.20	0.9110	32.01	0.9550	31.73	0.9430	—

Table 8. Results of VLM-Guided Low-Light Image Enhancement Methods.

Methods	LOLv1 [233]			LOLv2-Real [235]			LIME [234]	Params
	PSNR↑	SSIM↑	LPIPS↓	PSNR↑	SSIM↑	LPIPS↓	NIQE↓	[M]
Low-Light Image	7.77	0.191	0.417	9.72	0.196	0.394	4.351	N/A
NeRCo [261]	22.95	0.785	0.311	25.17	0.785	0.338	3.803	46+102
CFWD [66]	29.19	0.872	0.197	29.86	0.891	0.193	3.568	22+151
CLIP-LLA [262]	22.66	0.882	0.140	20.23	0.844	0.164	—	2+151
HVI-CIDNet+ [263]	28.85	0.894	0.058	24.31	0.873	0.107	3.760	69+246
GPP-LLIE [119]	27.51	0.872	0.081	29.23	—	0.055	4.24	76+55

Table 8. Results of VLM-Guided Low-Light Image Enhancement Methods.

Methods	LOLv1 [233]			LOLv2-Real [235]			LIME [234]	Params
	PSNR↑	SSIM↑	LPIPS↓	PSNR↑	SSIM↑	LPIPS↓	NIQE↓	[M]
Low-Light Image	7.77	0.191	0.417	9.72	0.196	0.394	4.351	N/A
NeRCo [261]	22.95	0.785	0.311	25.17	0.785	0.338	3.803	46+102
CFWD [66]	29.19	0.872	0.197	29.86	0.891	0.193	3.568	22+151
CLIP-LLA [262]	22.66	0.882	0.140	20.23	0.844	0.164	—	2+151
HVI-CIDNet+ [263]	28.85	0.894	0.058	24.31	0.873	0.107	3.760	69+246
GPP-LLIE [119]	27.51	0.872	0.081	29.23	—	0.055	4.24	76+55

Table 9. Results of VLM-Guided Image Deraining Methods.

Methods	Rain100H [242]		Rain100L [242]		Test100 [264]		Params	Time
	PSNR↑	SSIM↑	PSNR↑	SSIM↑	PSNR↑	SSIM↑	[M]	[s]
Rainy Image	12.13	0.344	25.52	0.814	21.11	0.640	N/A	N/A
DA-CLIP [29]	27.07	0.817	34.50	0.948	27.95	0.857	49+125	10.898
TVI-Derain [47]	30.51	0.893	37.72	0.973	31.17	0.912	20+278	—
TexDepth-Derain [265]	33.97	0.949	41.97	0.991	—	—	26+1469	—
PTG-RM-cRestormer [52]	31.77	0.913	39.27	0.985	32.30	0.934	27+86	0.142
AWRaCLe [266]	27.20	0.840	35.70	0.966	27.58	0.860	35+151	0.221

Table 9. Results of VLM-Guided Image Deraining Methods.

Methods	Rain100H [242]		Rain100L [242]		Test100 [264]		Params	Time
	PSNR↑	SSIM↑	PSNR↑	SSIM↑	PSNR↑	SSIM↑	[M]	[s]
Rainy Image	12.13	0.344	25.52	0.814	21.11	0.640	N/A	N/A
DA-CLIP [29]	27.07	0.817	34.50	0.948	27.95	0.857	49+125	10.898
TVI-Derain [47]	30.51	0.893	37.72	0.973	31.17	0.912	20+278	—
TexDepth-Derain [265]	33.97	0.949	41.97	0.991	—	—	26+1469	—
PTG-RM-cRestormer [52]	31.77	0.913	39.27	0.985	32.30	0.934	27+86	0.142
AWRaCLe [266]	27.20	0.840	35.70	0.966	27.58	0.860	35+151	0.221

Table 10. Results of VLM-guided Image Editing Methods.

Methods	MagicBrush [267]			AnyEdit [268]			Params	Time
	L1↓	DINO↑	CVS↑	L1↓	DINO↑	CVS↑	[B]	[s]
ML-MGIE [76]	0.083	84.54	90.70	0.211	60.31	78.85	0.86+7.43	5.11
InstructPix2Pix [269]	0.101	71.46	85.22	0.167	65.93	77.90	0.86+0.12	7.76
ICEdit [270]	0.088	84.37	90.17	0.163	66.66	80.90	12.00+4.82	9.95
UltraEdit-SD3 [271]	0.042	87.90	89.70	0.152	70.50	80.43	2.00+5.52	3.09
AnySD [268]	0.066	88.10	87.60	0.148	72.45	81.68	0.86+0.12	3.16
EditMGT [272]	0.109	76.96	86.22	0.147	68.80	82.74	1.01+3.70	12.75

Table 10. Results of VLM-guided Image Editing Methods.

Methods	MagicBrush [267]			AnyEdit [268]			Params	Time
	L1↓	DINO↑	CVS↑	L1↓	DINO↑	CVS↑	[B]	[s]
ML-MGIE [76]	0.083	84.54	90.70	0.211	60.31	78.85	0.86+7.43	5.11
InstructPix2Pix [269]	0.101	71.46	85.22	0.167	65.93	77.90	0.86+0.12	7.76
ICEdit [270]	0.088	84.37	90.17	0.163	66.66	80.90	12.00+4.82	9.95
UltraEdit-SD3 [271]	0.042	87.90	89.70	0.152	70.50	80.43	2.00+5.52	3.09
AnySD [268]	0.066	88.10	87.60	0.148	72.45	81.68	0.86+0.12	3.16
EditMGT [272]	0.109	76.96	86.22	0.147	68.80	82.74	1.01+3.70	12.75

3D processing tasks. VLMs are emerging as priors for 3D geometry and appearance recovery. DepthLM [273] uses vision-language features for metric depth prediction to reduce geometric ambiguity. For 3D completion, VP-LLM [167] formulates volume completion as token prediction, whereas VLScene [168] distills vision–language guidance into camera-based semantic completion for robustness against occlusion (Figure 13). SRSplat [169] addresses super-resolution Gaussian splatting from sparse inputs using VLM-assisted priors, and SpatialStack [274] proposes a hierarchical fusion of vision, geometry, and language, in contrast with the late-stage fusion. For text-driven 3D editing, where naive per-view conditioning disrupts geometric consistency, Instruct-NeRF2NeRF [171] applies instruction-following 2D diffusion edits as view-wise supervision while enforcing cross-view coherence, and GaussCtrl [170] uses ControlNet-style conditioned diffusion to stabilize multi-view-consistent editing of Gaussian splats.

Figure 14. Image-level qualitative comparisons across four representative low-level vision tasks. Orange rows denote conventional task-specific models, while blue rows denote VLM-guided methods.

Figure 14. Image-level qualitative comparisons across four representative low-level vision tasks. Orange rows denote conventional task-specific models, while blue rows denote VLM-guided methods.

6. Experiments

Protocol and caveats. The reported numbers are curated summaries from the original papers or public results, not a unified re-implementation: the methods differ in training data, parameters, degradation settings, and inference protocols. Hence, we use them for trend-level comparison rather than absolute ranking, and mark unreported values with “—”.

6.1. Datasets

Pre-training datasets. Among open corpora, LAION-5B [275] is the largest (∼5.85B image–text pairs, of which 2.32B are English and 2.26B span over 100 languages); Conceptual Captions (CC3M/CC12M) [276] provide 3M and ∼12M English pairs, SBU Captions [277] 1M Flickr pairs, and MS COCO Captions [278]∼120K images with 600K human-written descriptions. Proprietary corpora are typically larger: ALIGN [23] (∼1.8B pairs), Florence [279] (FLD-900M), Flamingo [280] (M3W, 185M), and PaLI [281] (WebLI, ∼10B images across 109 languages).

Low-level vision datasets. Table 4 summarizes task-specific datasets for important tasks. VLM-based approaches emphasize adaptation on real-world data, typically fine-tuning pre-trained models to address degradation.

6.2. Evaluation metrics

Distortion-based metrics. PSNR quantifies pixel-wise disparity via mean squared error, and SSIM assesses similarity across contrast, brightness, and structure.

Inception-based metrics. LPIPS [282] uses a pretrained AlexNet to emulate human perception, while FID [283] measures the fidelity and diversity of generated images via the Fréchet distance to reference images.

No-reference quality assessment. MUSIQ [284] handles resolution via multi-scale design, MANIQA [130] uses multi-dimensional attention for local-global cues, and CLIP-IQA [129] leverages CLIP to align with human perception.

Downstream application-based evaluations. Beyond visual quality, evaluation increasingly measures the utility of enhanced images for high-level tasks, e.g., mean Average Precision (mAP) for detection and accuracy for classification.

Instruction-based editing metrics. For image editing, following MGIE [76], L1 measures pixel-level distance to the target, while DINO and CVS measure visual similarity to the target via DINO and CLIP features.

Figure 15. Framework-level analysis of VLM-guided restoration. (a) Text-driven controllability: given the same input, different prompt sequences yield distinct restoration outputs, a capability unique to VLM-guided frameworks. (b) Mixed rain-and-haze and (c) mixed low-light-and-haze degradations: three frameworks (OneRestore, AutoDIR, InstructIR) respond to four different prompts on the same input, demonstrating fine-grained text-conditioned control under compound degradations.

Figure 15. Framework-level analysis of VLM-guided restoration. (a) Text-driven controllability: given the same input, different prompt sequences yield distinct restoration outputs, a capability unique to VLM-guided frameworks. (b) Mixed rain-and-haze and (c) mixed low-light-and-haze degradations: three frameworks (OneRestore, AutoDIR, InstructIR) respond to four different prompts on the same input, demonstrating fine-grained text-conditioned control under compound degradations.

Figure 16. Paradigm-level visualization of agentic restoration workflows. (Left) Architectures of three representative VLM-driven agents: JarvisIR, AgenticIR, and 4KAgent. (Right) Corresponding qualitative results on the same low-quality inputs.

Figure 16. Paradigm-level visualization of agentic restoration workflows. (Left) Architectures of three representative VLM-driven agents: JarvisIR, AgenticIR, and 4KAgent. (Right) Corresponding qualitative results on the same low-quality inputs.

6.3. Experimental Results

Params are reported as backbone+auxiliary (in millions, M; billions, B in Table 10).

Results on super-resolution. On RealSR [231] and DRealSR [232] (Table 5), PASD [93] attains the highest PSNR via pixel-level cross-attention, while PURE [215] leads on MUSIQ [284] by combining structural and semantic cues. Qualitatively (Figure 14(c)), VLM-guided methods (SUPIR, PASD) recover finer textures than conventional ones (Liif).

Results on denoising. On CBSD68 [253] and Urban100 [254] (Table 6), DFPIR [257] achieves the best overall results on CBSD68, while VLU-Net [53] leads on the structure-rich Urban100. TextPromptIR [255] also remains competitive across noise levels. Figure 14(d) shows VLM-guided methods (DiffUIR, PromptIR) preserve more structural detail under heavy noise than conventional baselines (DPIR).

Results on multi-weather image restoration. On Snow100K-L [258], Outdoor-Rain [259], and RainDrop [243] (Table 7), VLCIR [45] performs well for snow removal, while CyclicPrompt [102] shows strong deraining by leveraging a rain-free prior for second-stage refinement.

Results on low-light enhancement. On LOLv1 [233], LIME [234], and LOLv2-Real [235] (Table 8), CFWD [66] achieves the best PSNR and NIQE through multimodal guidance within a diffusion-based framework. In Figure 14(b), VLM-guided methods (OneRestore, InstructIR) recover brightness and color fidelity in dark regions.

Results on deraining. On Rain100H [242], Rain100L [242], and Test100 [264] (Table 9), TexDepth-Derain [265] reports the highest PSNR/SSIM on Rain100H and Rain100L, while PTG-RM-c_Restormer [52] achieves the best Test100 performance by leveraging CLIP-derived spatial priors. Inference time (Table 9, last column) varies by over an order of magnitude: prior-injection methods like PTG-RM stay near 0.1–0.2 s, whereas DA-CLIP, querying a CLIP model per image, exceeds 10 s, showing that VLM coupling dominates runtime more than the backbone.

Results on image editing. On MagicBrush [267] and AnyEdit [268] (Table 10), UltraEdit-SD3 [271] shows good edit fidelity on MagicBrush, AnySD [268] strong semantic alignment on AnyEdit, and ML-MGIE [76] stable performance across both. Figure 14(a) shows VLM-guided editors (InstructPix2Pix, ML-MGIE) follow free-form instructions such as “Have a squirrel be looking at the vase”. For editing (Table 10, last column), methods run at a few seconds per image, with diffusion-transformer designs (e.g., EditMGT) the slowest, reflecting multi-step sampling rather than the VLM call.

Framework-level analysis: text-driven controllability under mixed degradations. Beyond image-level metrics, VLM-guided frameworks offer text-driven controllability. In Figure 15(a), the same input under different prompt sequences (“Prompt 1→2” vs. “Prompt 3→4”) yields distinct restoration trajectories, which is inaccessible to task-specific models. Figure 15(b,c) compares OneRestore, AutoDIR, and InstructIR under mixed rain-and-haze and low-light-and-haze degradations, showing that these frameworks adapt to each prompt’s intent rather than merely handling compound degradations.

Paradigm-level analysis: agentic restoration workflows. At the paradigm level, recent works orchestrate multiple restoration tools via agents (Figure 16): JarvisIR uses a controller–expert design dispatching inputs to specific IR tools; AgenticIR runs a closed loop of Perception → Scheduling → Execution → Reflection with rollback on insufficient quality; and 4KAgent adds a profiling stage and a per-step Q-MoE module. Progressively more sophisticated agentic designs yield better detail recovery and fewer artifacts.

Failure case analysis. Current VLM-guided methods still exhibit characteristic failure modes (Figure 17, Table 11). (a) Text hallucination: the VLM perceives objects but not textual content, over-associating with semantic priors and distorting characters on signs. (b) Noise introduction: the agent commits to a plan from the initial degradation assessment and leaves residual noise introduced by intermediate tools uncorrected. (c) Limited generative recovery: under severe compound degradations, insufficient generative capacity yields blurry or incomplete reconstructions. (d) Zero-shot failure: unseen degradation types are not generalized. These failure modes motivate the directions in Section 7.

7. Future Directions

While VLMs have advanced low-level vision, current methods remain dominated by the VLM as Auxiliary paradigm, with end-to-end Direct VLM Adaptation still nascent. We outline research vectors to bridge the granularity gap between semantic reasoning and pixel-level reconstruction.

7.1. Advancing VLM Architectures for Low-Level Vision

Bridging the semantic-pixel granularity gap. A core bottleneck is the dimensionality mismatch between VLM latent spaces and the pixel-space requirements of restoration [79,80,288]. Beyond simple cross-attention, learnable hierarchical tokenization with unified codebooks could jointly encode coarse semantics (e.g., “foggy cityscape”) and fine high-frequency details (e.g., edge textures), letting VLMs reason about degradation globally while modulating local pixels via texture-aware decoders, a promising step toward a unified “All-in-One” restoration foundation model.

From guided restoration to direct generation. Most methods use VLMs to guide a separate diffusion or CNN backbone. A transformative direction empowers VLMs as direct generative engines for degradation removal, akin to DALL-E 2 [289] or GPT-4o [290], by extending discrete autoregressive prediction to continuous pixels via regression heads or integrated latent-diffusion decoders. Since naively adding high-level generative tasks can compromise detail-sensitive restoration [291], new objectives must synthesize restorations without hallucinating structural artifacts, reducing the error propagation of cascaded systems.

The perception-distortion tradeoff. Blau and Michaeli [292] show that distortion and perceptual realism cannot be jointly optimized beyond a point, which explains why the LR input leads on PSNR/SSIM in Table 5 while generative methods lead on LPIPS/MUSIQ. VLMs sharpen this tension: the semantic priors that enable realistic reconstructions also drive the hallucinations of Section 6. The goal is thus not to win both ends but to make the operating point controllable and to report both metric families explicitly.

7.2. Expanding VLM Capabilities and Applications

Restoration for downstream machine perception. Most research optimizes human perceptual quality (e.g., LPIPS, FID), yet in autonomous driving and surveillance the consumer is a machine. In machine-centric restoration, VLMs act as differentiable proxies for downstream tasks [8,293]; aligning restoration with the perception model’s semantic feature space [11] prioritizes features critical for recognition (e.g., edge distinctness) over human aesthetics.

Physically-informed multimodal prompting. Beyond descriptive prompts (e.g., “remove rain”), future systems should encode explicit physical constraints, prompting VLMs with physical parameters (scattering coefficients, ISO noise levels, sensor response) or non-visual modalities such as LiDAR depth, thermal, or audio [294] within an embodied framework [295,296], improving fidelity in ill-posed problems like underwater imaging or dehazing.

Quality-conditioned manifold learning. A complementary direction treats perceptual quality and degradation level as controllable axes of the latent manifold. Quality-conditioned approaches [286,287] steer generation toward a target quality, suggesting a restoration model conditioned on a continuous quality coordinate that traverses from degraded to clean states. Formalizing such a manifold, parameterized by VLM-derived quality estimates, could unify assessment and restoration and address the limited-generative-recovery failure of Section 6.

7.3. Optimizing VLM Performance and Deployment

Degradation-aware parameter-efficient tuning. While LoRA and Adapters have been widely adopted, they are often generic and agnostic to the spatial inductive biases required for image restoration [35,297]. Future research should investigate degradation-aware efficient tuning, where trainable modules are specifically designed to interact with the high-frequency components of the VLM’s visual encoder. Furthermore, exploring neural architecture search [298] to automatically identify optimal insertion points for adapters within massive VLMs could maximize restoration performance while minimizing computational overhead.

Data-efficient learning from unpaired priors. The dependence on massive paired datasets [20,23] is a barrier to scalability. A promising avenue is leveraging the pre-trained priors of VLMs to facilitate unsupervised or weakly-supervised learning [11]. By exploiting the rich semantic consistency inherent in CLIP or LLaVA, future models could learn to restore images by maximizing the semantic alignment between the output and a distinct “clean” text prompt. Techniques such as intelligent contrastive negative sampling [14] or distillation from larger models [299] can effectively distill restoration capabilities from the VLM’s zero-shot knowledge without requiring pixel-perfect ground truth pairs.

7.4. Addressing Inherent Challenges and New Paradigms

Self-evolving and test-time adaptation. Training on paired synthetic data often leads to poor generalization on real-world degradations due to domain gaps [300]. VLMs offer a unique opportunity for Test-Time Adaptation as a new paradigm [301]. Future work should explore optimization objectives that leverage the pre-trained semantic consistency of VLMs as a supervision signal during inference. By dynamically minimizing the semantic discrepancy between the degraded input and the restored output in the VLM’s feature space, models can adapt to specific noise distributions or degradation combinations on the fly, significantly enhancing robustness in open-world scenarios. A closely related route is self-evolution, in which the model generates its own training signal without human labels. EvoQuality [133] demonstrates this for no-reference quality assessment via pairwise voting and relative-reward updates; extending such label-free self-improvement from quality assessment to quality restoration remains open, since the supervisory signal for pixel reconstruction is harder to bootstrap than a ranking objective.

Trustworthiness and explainable restoration. As VLMs are deployed in safety-critical domains like medical imaging (CT/MRI) and forensic analysis, black-box restoration is insufficient [11]. Future frameworks must address the risk of hallucination by providing explainable degradation reasoning [302], where the model articulates its diagnosis of image quality issues and justifies its restoration strategy via CoT prompting. Additionally, developing human-in-the-loop interactive agents [24,303] that allow users to spatially refine restoration results through natural language feedback, correcting artifacts or adjusting enhancement strength in real-time, will be essential for establishing trust in clinical and professional workflows [27] under real-world conditions.

Convergence with Vision-Language-Action models. As Vision-Language-Action (VLA) models extend VLMs with action outputs for embodied agents, low-level vision becomes a robustness prerequisite rather than a downstream consumer: corrupted or degraded visual inputs propagate directly into action errors. Recent work restores corrupted observations before action prediction to harden VLA policies [304], and broader analyses identify input robustness and perception reliability as open challenges guiding VLA development [305]. A promising direction is to co-design restoration and action heads, so the restoration objective is shaped by downstream control performance and the VLA’s action confidence signals which regions warrant restoration, tightening the loop between low-level vision and embodied decision-making.

8. Conclusions

This survey organizes VLM-based low-level vision into two paradigms: Direct VLM Adaptation, which refactors visual encoders and prompts to handle degradations end-to-end, and VLM as Auxiliary, where VLMs act as Semantic Providers, Degradation Interpreters, Quality Evaluators, or Intelligent Controllers. While VLMs bring strong zero-shot generalization and semantic consistency, particularly in domains such as medical imaging and remote sensing, aligning high-level linguistic representations with the fine-grained fidelity required for reconstruction remains the central bottleneck. As the field moves from cascaded guidance toward unified foundation models, balancing computational efficiency, physical grounding, and explainability will be key. We hope this survey and its taxonomy inspire future work on robust, open-world restoration; to support ongoing research, we maintain a continuously updated repository of VLM-based low-level vision resources.