Beyond References: Human-Aligned Caption Reliability Assessment

1. Introduction

The generation of descriptive natural language for visual scenes has long stood as a central challenge at the crossroads of computer vision and natural language processing. With the advent of deep learning, automated captioning systems have demonstrated remarkable capabilities, producing fluent and semantically rich descriptions that approximate human annotations in benchmark settings. However, despite these technical advances, a persistent gap remains between experimental benchmarks and practical deployments [9,11,12]. In real-world applications—such as assistive technologies for visually impaired users, interactive educational platforms, autonomous navigation, or multimodal search engines—the reliability of captions is paramount [1,2]. A seemingly minor semantic error (e.g., misidentifying an object or its relationship with others) may lead to significant misunderstandings, potentially eroding user trust and undermining the safety and functionality of the system. Consequently, there is a pressing need for mechanisms that ensure caption outputs are validated before being presented to end-users.

Traditional evaluation of image captions has relied heavily on reference-based metrics such as BLEU, METEOR, ROUGE, CIDEr, and SPICE [4,6]. These approaches assume the availability of human-authored ground-truth captions and measure performance through token overlap, semantic similarity, or graph-structured alignment with references [14,15]. While these metrics provide a useful post-hoc evaluation tool for research comparisons, they face two fundamental limitations: (i) they cannot operate in real time during inference, since reference captions are typically absent in deployment scenarios; and (ii) they often fail to capture the subtleties of human judgment, as two valid captions may use entirely different linguistic expressions while conveying the same semantic meaning. These constraints highlight the inadequacy of reference-dependent evaluation and motivate a shift toward reference-free quality estimation.

To address this challenge, we define the task of Caption Quality Estimation (CQE), which seeks to evaluate the semantic validity of a caption given only the $\langle image,caption\rangle$ pair, without any auxiliary reference texts. Formally, this entails designing a function $CQE(image,caption)$ that outputs a continuous or discrete quality score reflecting the degree of alignment between the visual scene and its linguistic description. Crucially, this estimation must approximate human perception, enabling the system to filter out inadequate or misleading captions in real time. Such a framework transforms captioning pipelines from passive generators of text into actively self-monitoring systems, capable of rejecting unreliable content and thereby reinforcing user confidence.

We introduce VQAR (Visual-linguistic Quality Assessment for Reliability), a new paradigm for reference-free caption evaluation. Unlike conventional captioning or retrieval models that emphasize generation or matching, VQAR is explicitly designed to measure the trustworthiness of a caption in alignment with human perception. To enable this, we curate a large-scale dataset featuring over 600,000 binary human judgments collected across approximately 55,000 unique $\langle image,caption\rangle$ pairs. Each annotation simply records whether a caption is acceptable for its associated image, offering a lightweight yet scalable mechanism for quality labeling. While such binary judgments are coarse in nature, they provide a robust foundation for training models to distinguish between trustworthy and untrustworthy captions at scale.

The novelty of VQAR lies in its dual-layer evaluation strategy. First, the binary annotations encode a broad human consensus regarding acceptability, ensuring scalability across diverse domains. Second, to address subtler distinctions in caption quality, we construct a refined subset with fine-grained expert evaluations, capturing more nuanced aspects such as object fidelity, relational consistency, and fluency. This dual strategy allows models trained under VQAR to achieve both breadth and depth: broad coverage from large-scale annotations and precise calibration through expert labels.

The implications of VQAR extend beyond academic evaluation. In real-world scenarios, captioning systems embedded in mobile devices, online platforms, or safety-critical environments can integrate VQAR as an online filter, discarding captions deemed unreliable before they are surfaced to users. This paradigm resonates with the broader vision of trustworthy AI, where systems are expected not only to generate content but also to self-assess and guarantee reliability. Furthermore, the emphasis on human alignment in VQAR ensures that the metric reflects the ultimate end-user perception, rather than an artificial benchmark criterion.

To summarize, the introduction of VQAR makes several key contributions:

• We redefine caption evaluation by shifting from reference-dependent metrics toward a fully reference-free paradigm that can function during inference.
• We introduce VQAR, a framework explicitly oriented toward human-aligned reliability assessment, and provide a large-scale dataset of over 600,000 binary human ratings across diverse visual scenes.
• We show that models trained on coarse binary annotations generalize effectively to expert-labeled fine-grained quality assessments, highlighting the dual capacity of scalability and nuance.
• We demonstrate the applicability of VQAR in real-world deployments, where reference captions are unavailable and reliability is non-negotiable.

In essence, this work advocates for a shift in perspective: from treating caption evaluation as an offline benchmarking problem to framing it as an online reliability assurance mechanism. By doing so, VQAR advances the goal of building captioning systems that are not only fluent and accurate but also self-aware and trustworthy in practical use cases.

2. Related Work

Our research builds upon several converging threads in vision-language understanding, evaluation methodologies, and accessibility-oriented captioning systems. While prior work has largely concentrated on reference-based metrics or generation-focused objectives, the VQAR framework departs from these traditions by offering a reference-free mechanism that assesses caption reliability in a human-aligned and inference-time fashion. In what follows, we situate VQAR within four major areas of related work.

2.1. Evaluation Metrics in Vision-Language Understanding

The evaluation of captions has historically revolved around text-based comparisons with human-authored references. Early metrics such as BLEU, ROUGE, and METEOR prioritize lexical or syntactic overlaps, producing surface-level estimates of fidelity. While straightforward to compute, these approaches struggle with semantic variability, since multiple correct captions can exist with little to no lexical overlap. To overcome such limitations, researchers have proposed metrics grounded in semantic representations, learned embeddings, and object-level alignments. For instance, Cui et al. [7] designed a learning-based discriminator to distinguish human from machine captions, using its scores to guide model selection. Similarly, VIFIDEL integrates visual object detections with textual frequency statistics to compute a weighted alignment score.

Despite these advances, most existing metrics presuppose the availability of multiple reference captions per image, an assumption that collapses in real-time systems. Even methods like VIFIDEL, which attempt partial relaxation, still rely on relative scoring among candidate captions rather than absolute evaluation of a single caption. In contrast, VQAR targets absolute scoring of $\langle image,caption\rangle$ pairs, enabling independent operation without the comparative or reference-dependent constraints embedded in prior evaluation pipelines.

2.2. Inspiration from Machine Translation Quality Estimation

The conceptual roots of reference-free caption assessment can be traced to the Machine Translation (MT) community, where Quality Estimation (QE) emerged to assess translation outputs in the absence of gold references. Early MT-QE systems relied on hand-crafted features that correlated with translation quality [27,29], followed by neural approaches using recurrent architectures [15,18]. More advanced predictor-estimator models [14] integrated both source and hypothesis sequences to generate fine-grained predictions.

Shared tasks at WMT [28] catalyzed the growth of MT-QE, driving innovation in both modeling and evaluation. However, MT-QE operates in a purely textual space, where both input and output are linguistic. This unimodal formulation facilitates token-level modeling [22,30,34], but does not naturally extend to multimodal contexts. Captioning requires aligning rich visual semantics with natural language, a challenge that demands multimodal representation learning. VQAR adapts the QE paradigm to vision-language tasks by integrating pretrained visual encoders and linguistic models, while guiding learning with large-scale human feedback on caption appropriateness.

2.3. Caption Quality and Trust in Accessibility Contexts

A complementary strand of research addresses the problem of trust in captioning systems, especially in accessibility applications for Blind or Visually Impaired (BVI) users. MacLeod et al. [21] examined strategies for communicating system uncertainty by appending confidence indicators to captions, such as “I’m 98% sure this is $CAPTION.” While informative, such strategies sometimes led to overtrust, as users interpreted hedged outputs as plausible even when incorrect—particularly in subjective or social media scenarios.

These observations underscore the risks of exposing end-users to potentially misleading captions without protective safeguards. VQAR addresses this issue by operationalizing a reliability gating mechanism. Rather than verbalizing uncertainty, VQAR computes a scalar quality score and enforces a display rule. This simple yet effective filtering criterion ensures that only captions surpassing a reliability threshold are displayed, thereby reducing cognitive burden and preventing inadvertent misinterpretations in accessibility scenarios.

2.4. Positioning Beyond Caption Generation and Retrieval

Finally, it is important to distinguish Caption Quality Estimation from related but distinct tasks such as caption generation and image-text retrieval. Captioning models focus on synthesizing textual descriptions conditioned on visual input, while retrieval models aim to rank candidate captions relative to an image, often under contrastive learning objectives. Both paradigms are grounded in large-scale supervision with reference captions, and their evaluation protocols are inherently tied to such references.

In contrast, VQAR is explicitly post-hoc and reference-independent. Given a single $\langle image,caption\rangle$ pair, it estimates whether the caption is acceptable, regardless of how the caption was produced (generative, retrieval-based, or even manually written). This independence makes VQAR a flexible plug-in evaluator, deployable in contexts where reference sets are unavailable or infeasible, and where the open-ended nature of captioning requires robust validation mechanisms.

Viewed through this lens, our work fills a critical gap in vision-language research by enabling scalable, human-aligned caption evaluation that functions in real time and without external supervision. VQAR stands as one of the first frameworks to operationalize large-scale reference-free caption quality estimation, combining binary human supervision with multimodal alignment signals to deliver both practical reliability and scientific advancement.

3. VQAR Construction

A central contribution of this work is the introduction of the VQAR Quality Annotation Corpus, a large-scale dataset designed explicitly for training and evaluating reference-free caption quality estimation systems. The corpus contains hundreds of thousands of binary human ratings that reflect the perceived adequacy of image captions when aligned with their associated visual content. By shifting the evaluation paradigm away from reference dependence, the VQAR corpus provides a practical foundation for developing systems capable of real-time caption reliability assessment. In this section, we describe the construction pipeline in detail, including image and caption preparation, annotation design, quality control, and validation of label stability. We also highlight ethical considerations and the broader applicability of this dataset.

3.1. Image Sampling and Ethical Filtering

We begin by selecting images from the Open Images Dataset (OID) [20], an open-source repository with diverse visual content and flexible licensing terms. A total of 16,000 images were randomly sampled to ensure both scalability and representativeness across visual domains. Importantly, we enforce ethical and privacy-related safeguards by excluding content containing identifiable human faces. This was achieved using the Google Cloud Vision API, which detects and filters facial regions. By removing such instances, we ensure compliance with privacy regulations and mitigate risks associated with human-identifiable data, while also maintaining consistency with the CC BY licensing attached to OID.

Compared to highly curated captioning datasets such as COCO or Conceptual Captions, OID provides more heterogeneous and less standardized content. This heterogeneity is essential for training robust quality estimators that can generalize beyond narrow distributions and adapt to the diversity encountered in real-world scenarios.

Table 1. Partition statistics for the VQAR dataset.

Split	Samples	Images	Captions	Models
Train	58,354	11,027	34,532	11
Dev	2,392	654	1,832	4
Test	4,592	1,237	3,359	4

Table 1. Partition statistics for the VQAR dataset.

Split	Samples	Images	Captions	Models
Train	58,354	11,027	34,532	11
Dev	2,392	654	1,832	4
Test	4,592	1,237	3,359	4

3.2. Caption Generation Across Model Variants

To generate candidate captions, we leverage a set of Transformer-based captioning architectures [32] trained on the Conceptual Captions dataset [26], which contains more than 3.3 million web-crawled image-text pairs. These models are particularly suitable for our purposes, as prior work [26] has shown that they generalize effectively to out-of-distribution samples.

Our design ensures caption diversity by varying three architectural dimensions: (i) the choice of visual backbone for image encoding, (ii) the integration of object-level signals, and (iii) the decoding strategy for text generation. This diversity is crucial for creating captions with varying degrees of quality, thereby providing a meaningful training signal for quality estimation models.

Visual Encoders.

We explore multiple visual backbones to obtain feature representations:

• Inception-ResNet-v2 [31], a high-performing CNN model for image classification.
• Picturebook Encoder [17], which maps images into dense embeddings optimized for visually-grounded language tasks.
• Graph-RISE [13], a ResNet-101 model trained with graph-based regularization for ultra-fine-grained classification.

Object-Level Features.

Object-centric information is incorporated using Faster R-CNN trained on Visual Genome [19], capable of detecting over 1,600 objects and 400 attributes. Detected regions are embedded with ResNet-101, either pretrained on ImageNet [24] or Graph-RISE [13], enabling rich representations of localized entities.

Object Label Embeddings.

Detected object labels are classified using a ResNet trained on JFT [12], and projected into a semantic embedding space using word2vec [23]. The resulting embeddings $o_j$ encode distributional semantics and provide additional context for caption grounding.

Caption Decoding.

To promote linguistic variability, we adopt two decoding strategies: greedy decoding for deterministic outputs and beam search (beam width = 5) for more probabilistic variants. This ensures that each image is paired with captions of varying fluency and semantic adequacy.

3.3. Crowdsourced Binary Evaluation Framework

Unlike multi-dimensional evaluation protocols that separately assess fluency, relevance, and informativeness [3,33], VQAR employs a simplified binary annotation design. Annotators answer a single question: “Is this a good caption for the image?” Responses are YES, NO, or SKIP, yielding a Bernoulli random variable $r_i\in \{0,1\}$ per judgment.

Figure 1. Distribution of $\widehat{p}$ values across training, development, and test partitions. Frequencies in the training set were downscaled for clarity.

Figure 1. Distribution of $\widehat{p}$ values across training, development, and test partitions. Frequencies in the training set were downscaled for clarity.

For each $\langle image,caption\rangle$ pair, we collect $N=10$ ratings from Google’s crowdsourcing platform1. The quality score is estimated as:

$$\widehat{p}={\displaystyle \frac{1}{N}}\sum _{i=1}^N{r}_i.$$

(1)

Samples with more than two SKIPs are discarded. To reduce noise while preserving granularity, we discretize scores using:

$${\widehat{p}}_{\mathrm{discrete}}=\mathrm{round}(\widehat{p}\times 8)/8,$$

(2)

resulting in a 9-point ordinal scale $\{0,{\displaystyle \frac{1}{8}},\dots ,1\}$.

3.4. Annotation Quality Control and Agreement Analysis

Annotation reliability is a core concern in large-scale crowdsourcing. To validate stability, we conducted a test-retest study over 509 captions, each re-annotated by two independent groups of annotators four weeks apart. The average ratings ${\widehat{p}}_1$ and ${\widehat{p}}_2$ were compared, yielding a difference $\Delta p={\widehat{p}}_1-{\widehat{p}}_2$.

The distribution of $\Delta p$ was centered near zero (mean = 0.015, std = 0.212), with 85% of pairs falling within an absolute difference of 0.25. This demonstrates strong consistency despite the simplicity of binary judgments. We further computed inter-rater agreement using Krippendorff’s alpha and observed values exceeding 0.71, which indicates substantial agreement and validates the robustness of the annotation protocol.

3.5. Dataset Scale, Partitions, and Diversity

The final VQAR dataset contains approximately 600,000 binary ratings spanning 55,000 unique $\langle image,caption\rangle$ pairs. Table 1 presents the partitioning scheme across training, development, and test splits. Notably, captions are derived from multiple models and decoding strategies, ensuring linguistic heterogeneity and semantic variation.

This design yields a dataset that is both broad—covering diverse image domains—and deep—capturing nuanced quality judgments across annotator populations. The inclusion of multiple encoders, object features, and decoding strategies further enhances the representational richness.

3.6. Corpus Utility and Broader Applications

Beyond serving as a benchmark for training VQAR, the corpus has broader implications for multimodal research. First, it provides a scalable testbed for studying human alignment in multimodal evaluation, offering insights into how binary preferences can approximate semantic adequacy. Second, it can be repurposed for meta-evaluation of automatic metrics, serving as a ground truth for validating new reference-free approaches. Third, its design facilitates deployment in real-world pipelines where reliability gating is required, such as assistive tools for visually impaired users or quality control filters for generative AI services.

Figure 2. Data flow of the VQAR framework.

Figure 2. Data flow of the VQAR framework.

In summary, the VQAR corpus provides a high-fidelity, large-scale resource for studying caption quality estimation without references. Its construction pipeline emphasizes ethical image selection, diverse caption generation, robust crowdsourced binary evaluation, and validated inter-rater stability. Together, these components create a dataset that is not only technically rigorous but also practically valuable for building trustworthy captioning systems in real-world scenarios.

4. VQAR Model Architecture

In this section, we present the architecture of VQAR (Visual-linguistic Quality Assessment for Reliability), our proposed model for reference-free caption quality estimation. The primary objective of VQAR is to evaluate the semantic compatibility between an image and its candidate caption without relying on gold-standard textual references. To achieve this, VQAR integrates heterogeneous sources of multimodal information through a series of specialized modules, including modality-specific encoders, bilinear fusion mechanisms, cross-modal attention layers, and reliability-aware scoring heads. The architecture is deliberately modular, allowing for incremental extension with auxiliary objectives, contrastive pretraining, and large-scale multimodal backbones.

We now describe the core components of VQAR in detail, starting with input feature encoding, followed by the cross-modal interaction layers, prediction modules, training strategies, and model variants.

4.1. Multimodal Input Representation

VQAR relies on three complementary types of features: global image embeddings, object-centric semantic signals, and sentence-level caption encodings. Each type of feature is extracted with pretrained encoders and subsequently projected into a unified space for downstream interaction.

Global Image Features.

The input image is processed through Graph-RISE [13], a ResNet-101 based encoder with graph-based regularization, producing a dense vector $i\in {\mathbb{R}}^{D_i}$ that summarizes coarse-grained semantic content. This representation captures high-level scene information beyond individual objects.

Object-Centric Embeddings.

To capture fine-grained grounding information, we detect object categories $O=\{o_1,\dots ,o_{\left|O\right|}\}$ with a ResNet-based classifier trained on JFT [12]. Each detected label is embedded using a learned projection matrix $W_o\in {\mathbb{R}}^{V_o\times D_o}$:

$$o_j=W_o·\mathrm{onehot}\left(j\right),j\in \{1,\dots ,|O\left|\right\}.$$

(3)

Caption Encodings.

The candidate caption c is encoded with the Universal Sentence Encoder (USE) [5], yielding $s\in {\mathbb{R}}^{D_s=512}$. To reduce dimensionality while enhancing task-specific abstraction, we apply a nonlinear projection:

$$s^{\prime }=\sigma (W_{s}s+b_s),W_s\in {\mathbb{R}}^{D_h\times D_s},$$

(4)

where $\sigma (·)$ denotes a Leaky ReLU activation.

4.2. Bilinear Cross-Modal Fusion

To capture structured correspondences between modalities, VQAR introduces bilinear operators that measure the compatibility of different feature pairs. Given $f_x\in {\mathbb{R}}^{d_x}$ and $f_y\in {\mathbb{R}}^{d_y}$, the bilinear interaction is:

$$\mathrm{BI}(f_x,f_y;B)=f_x^{\top }B{f}_y,$$

(5)

where $B\in {\mathbb{R}}^{d_x\times d_y}$ is learnable.

We compute three bilinear scores:

$$\begin{array}{cc}\hfill b_{oi}& ={\displaystyle \frac{1}{\left|O\right|}}\sum _{j=1}^{\left|O\right|}{o}_j^{\top }{B}_{oi}i,\hfill \end{array}$$

(6)

$$\begin{array}{cc}\hfill b_{os}& ={\displaystyle \frac{1}{\left|O\right|}}\sum _{j=1}^{\left|O\right|}{o}_j^{\top }{B}_{os}{s}^{\prime },\hfill \end{array}$$

(7)

$$\begin{array}{cc}\hfill b_{is}& =i^{\top }{B}_{is}{s}^{\prime },\hfill \end{array}$$

(8)

which respectively measure object-image consistency, object-caption alignment, and image-caption compatibility. These scores are concatenated into a fusion vector $z=[b_{oi};b_{os};b_{is}]$.

4.3. Cross-Modal Attention for Object Grounding

Bilinear fusion alone captures static pairwise interactions but lacks adaptivity. To address this, we incorporate a cross-attention mechanism that dynamically conditions object features on the caption representation. Formally:

$$\begin{array}{cc}\hfill {\alpha }_j& =\mathrm{softmax}\left(s^{\prime \top }{W}_a{o}_j\right),\hfill \end{array}$$

(9)

$$\begin{array}{cc}\hfill o_{\mathrm{att}}& =\sum _{j=1}^{\left|O\right|}{\alpha }_j·o_j.\hfill \end{array}$$

(10)

Here, ${\alpha }_j$ reflects the relevance of object $o_j$ to the caption semantics, and $o_{\mathrm{att}}$ aggregates object information into a weighted summary. This operation enhances grounding by emphasizing objects that are linguistically salient.

4.4. Prediction Head and Reliability Scoring

The fusion vector z and attended object vector $o_{\mathrm{att}}$ are combined to form a joint representation:

$$h=\sigma (W_h[z;o_{\mathrm{att}}]+b_h).$$

(11)

The final quality score is predicted via a sigmoid classifier:

$$\widehat{y}=\mathrm{sigmoid}(w^{\top }h+b),$$

(12)

where $\widehat{y}\in [0,1]$ denotes the estimated probability that the caption is acceptable.

4.5. Training Paradigm

VQAR is trained with a hybrid loss that combines binary classification and regression objectives.

Binary Cross-Entropy Loss.

Given binary labels $y\in \{0,1\}$ from human annotations, the classification loss is:

$${\mathcal{L}}_{\mathrm{bce}}=-ylog\widehat{y}-(1-y)log(1-\widehat{y}).$$

(13)

Regression Loss.

We also match the model score to the aggregated human rating $\overline{p}$:

$${\mathcal{L}}_{\mathrm{mse}}={\parallel \widehat{y}-\overline{p}\parallel }^2.$$

(14)

Joint Objective.

The final objective combines both terms:

$$\mathcal{L}={\lambda }_1{\mathcal{L}}_{\mathrm{bce}}+{\lambda }_2{\mathcal{L}}_{\mathrm{mse}}.$$

(15)

4.6. Contrastive Pretraining for Cross-Modal Alignment

Before supervised training, we pretrain VQAR on Conceptual Captions [26] with a contrastive loss that encourages alignment of paired image-caption embeddings:

$${\mathcal{L}}_{\mathrm{contrast}}=-\sum _{k=1}^{B}log{\displaystyle \frac{exp(\mathrm{sim}(i_k,s_k)/\tau )}{{\sum }_{l=1}^{B}exp(\mathrm{sim}(i_k,s_l)/\tau )}},$$

(16)

where $\mathrm{sim}(i,s)=i^{\top }{W}_{c}s$ and $\tau$ is a temperature parameter. This initialization improves downstream quality estimation by providing a well-structured embedding space.

4.7. Regularization and Calibration

To prevent overfitting and improve interpretability, we employ additional techniques:

• Dropout Regularization: Applied at both fusion and prediction layers to encourage robustness.
• Temperature Scaling: Post-training calibration ensures that predicted probabilities reflect empirical human judgment distributions.
• Label Smoothing: Prevents the model from overconfident predictions in borderline cases.

4.8. Model Variants and Ablation Design

To dissect the contributions of different components, we construct several ablations:

• VQAR-Bilinear: Uses only bilinear fusion.
• VQAR-Attn: Adds cross-modal attention.
• VQAR-Full: Incorporates attention, pretraining, and auxiliary losses.
• VQAR-RandInit: Removes contrastive pretraining and starts from random initialization.

These variants enable us to measure the incremental gains from each architectural element.

4.9. Scalability and Future Extensions

While the current design emphasizes clarity and interpretability, VQAR is modular and readily extensible. Future iterations can integrate large-scale pretrained vision-language backbones such as CLIP, BLIP2, or Flamingo, replace bilinear layers with transformer-based fusion modules, and leverage self-supervised pretraining objectives. Such extensions would enhance scalability while retaining the human-aligned reliability principles central to VQAR.

Overall, the VQAR architecture constitutes a unified framework for reference-free caption evaluation. It balances simplicity and extensibility, combining structured feature interactions, dynamic attention mechanisms, robust training objectives, and pretraining strategies. These elements jointly enable VQAR to approximate human perceptions of caption quality, offering a principled solution for real-world caption validation tasks.

5. Experiments

We conduct an extensive empirical study to evaluate VQAR on the Caption Quality Estimation (CQE) task. Our analysis covers intrinsic alignment with human judgments, robustness under distribution shifts and perturbations, cross-domain generalization, and practical deployment concerns such as threshold selection, coverage, and latency. We also include ablations to quantify the contribution of each architectural component. Unless stated otherwise, all models and splits follow the corpus specification in Section 3.

5.1. Training Configuration and Reproducibility

Two-Stage Training.

All VQAR variants are trained on the full training split (Table 1) using a two-phase regime: (i) optional contrastive pretraining on Conceptual Captions [26], followed by (ii) supervised fine-tuning on the VQAR human labels. The primary optimization target during fine-tuning is Mean Squared Error:

$${\mathcal{L}}_{MSE}={\displaystyle \frac{1}{N}}\sum _{j=1}^N{(y_j-{\widehat{y}}_j)}^2,$$

(17)

where $y_j$ denotes the discretized human rating and ${\widehat{y}}_j$ the model prediction. We combine this with the classification objective described in Section 4 to balance ordinal regression and acceptability discrimination.

Optimization and Regularization.

We use Adam [16] with batch size $B=256$ and select the learning rate from $\{{10}^{-4},{10}^{-5},{10}^{-6}\}$ based on dev-set Spearman’s $\rho$. A dropout rate of $0.2$ is applied to all trainable projections and fusion layers. Unless specified, the pretrained encoders (Graph-RISE for images, USE for captions, and the JFT-based object label embeddings) remain frozen to isolate the effect of the VQAR heads.

Implementation Details.

We adopt early stopping on dev $\rho$ with a patience of 10 epochs, gradient clipping at 1.0, and mixed precision training. All runs are repeated with three random seeds; we report the mean for metrics and discuss variance qualitatively where relevant.

5.2. Validation Protocol and Hyperparameter Search

We perform a grid search over hidden dimensionalities $\{128,256,512\}$ for $W_h$, temperature $\tau \in \{0.03,0.07,0.1\}$ for contrastive pretraining, and object-label budget $\left|O\right|\in \{0,8,16,20,32\}$. Model selection is carried out on the dev split using rank-based criteria (Spearman’s $\rho$) to emphasize agreement with human ordering. When multiple configurations tie on $\rho$, we break ties by lower MSE and higher AUC to balance calibration and discrimination.

5.3. Intrinsic Evaluation: Correlation, Error, and Discrimination

We first quantify alignment with human preferences via Spearman’s rank correlation ($\rho$), Mean Squared Error (MSE), and Area Under the ROC Curve (AUC). Table 2 compares VQAR variants:

The base model already correlates well with human ratings; adding object labels yields a further gain in $\rho$ and AUC, indicating stronger grounding. Contrastive pretraining alone is insufficient (lower $\rho$, higher MSE), underscoring that similarity-based objectives do not directly transfer to human quality. However, pretraining followed by task-specific fine-tuning produces the strongest alignment and calibration.

5.4. Probability Calibration and Reliability Diagrams

Beyond ranking, deployment requires calibrated probabilities. We therefore compute the Brier score and Expected Calibration Error (ECE). Let ${\widehat{y}}_j$ be the predicted probability and $y_j\in \{0,1\}$ the binarized acceptability label. The Brier score is

$$\mathrm{Brier}={\displaystyle \frac{1}{N}}\sum _{j=1}^N{({\widehat{y}}_j-y_j)}^2.$$

(18)

For ECE, we partition predictions into M bins ${\left\{B_m\right\}}_{m=1}^M$ and measure

$$\mathrm{ECE}=\sum _{m=1}^M{\displaystyle \frac{|B_m|}{N}}\left|{\displaystyle \frac{1}{|B_m|}}\sum _{j\in B_m}{\widehat{y}}_j-{\displaystyle \frac{1}{|B_m|}}\sum _{j\in B_m}{y}_j\right|.$$

(19)

We observe that VQAR (pre+finetune) attains the lowest Brier and ECE after temperature scaling, indicating trustworthy probabilities that match empirical frequencies. Reliability diagrams (not shown for brevity) reveal overconfidence without calibration and near-perfect alignment after scaling.

5.5. Ablation Analysis

We ablate fusion, attention, pretraining, and auxiliary losses to isolate their impact:

Table 3. Ablation study on the test split. Removing either bilinear fusion or cross-attention substantially harms ranking and discrimination.

Model Variant	${\mathit{\rho }}_{\mathit{t}\mathit{e}\mathit{s}\mathit{t}}$	${\mathrm{MSE}}_{\mathit{t}\mathit{e}\mathit{s}\mathit{t}}$	AUC
VQAR (full model)	0.55	0.050	0.85
w/o object attention	0.50	0.053	0.81
w/o bilinear fusion	0.48	0.057	0.79
w/o contrastive pretraining	0.49	0.056	0.80
w/o auxiliary MSE loss	0.51	0.052	0.83

Table 3. Ablation study on the test split. Removing either bilinear fusion or cross-attention substantially harms ranking and discrimination.

Model Variant	${\mathit{\rho }}_{\mathit{t}\mathit{e}\mathit{s}\mathit{t}}$	${\mathrm{MSE}}_{\mathit{t}\mathit{e}\mathit{s}\mathit{t}}$	AUC
VQAR (full model)	0.55	0.050	0.85
w/o object attention	0.50	0.053	0.81
w/o bilinear fusion	0.48	0.057	0.79
w/o contrastive pretraining	0.49	0.056	0.80
w/o auxiliary MSE loss	0.51	0.052	0.83

Cross-attention improves grounding, while bilinear terms capture structured cross-modal compatibility; excluding either degrades $\rho$ and AUC. Contrastive pretraining yields a modest yet consistent benefit after fine-tuning, and the auxiliary MSE stabilizes learning around ordinal targets.

5.6. Robustness Under Perturbations

To test sensitivity, we evaluate VQAR under controlled caption perturbations. Given a caption token sequence $c=(w_1,\dots ,w_T)$, we inject noise by (i) synonym substitution, (ii) noun-phrase swaps, and (iii) insertion of distractor objects not present in the image. Let $\pi (·)$ denote a perturbation; we define robustness as the monotonicity of scores:

$${\Delta }_{\pi }=\mathbb{E}\left[\widehat{y}(i,c)-\widehat{y}(i,\pi \left(c\right))\right],$$

(20)

expecting ${\Delta }_{\pi }>0$ on average for meaning-altering perturbations. We further study object-set masking by zeroing a subset of object embeddings $\tilde{O}\subset O$ and measuring score drops. VQAR exhibits the largest penalization for distractor insertions, indicating strong sensitivity to hallucinated entities—desirable for safety-critical filtering.

5.7. Cross-Domain Generalization

We assess extrinsic utility by applying VQAR to an out-of-domain benchmark annotated with expert correctness and helpfulness labels. A caption is “good’’ only if both criteria hold. For a threshold t, we define precision/recall as

$$\begin{array}{cc}\hfill {\mathrm{Precision}}_t& ={\displaystyle \frac{{\sum }_s{\mathbf{1}}_{\mathrm{good}}^s·{\mathbf{1}}_{\mathrm{VQAR}\left(s\right)>t}}{{\sum }_s{\mathbf{1}}_{\mathrm{VQAR}\left(s\right)>t}}},{\mathrm{Recall}}_t={\displaystyle \frac{{\sum }_s{\mathbf{1}}_{\mathrm{good}}^s·{\mathbf{1}}_{\mathrm{VQAR}\left(s\right)>t}}{{\sum }_s{\mathbf{1}}_{\mathrm{good}}^s}}.\hfill \end{array}$$

(21)

We summarize with area under the PR curve (AUC):

Table 4. Cross-domain PR-AUC: fine-tuned VQAR provides the best filtering utility.

Model Variant	AUC on our model
VQAR (base)	0.80
VQAR (+obj20)	0.83
VQAR (pretrained)	0.76
VQAR (pre + finetune)	0.86

Table 4. Cross-domain PR-AUC: fine-tuned VQAR provides the best filtering utility.

Model Variant	AUC on our model
VQAR (base)	0.80
VQAR (+obj20)	0.83
VQAR (pretrained)	0.76
VQAR (pre + finetune)	0.86

Fine-tuned VQAR substantially outperforms similarity-pretrained models; at a target precision of $0.8$, recall increases from $0.22$ (pretrained-only) to $0.69$ (pre+finetune), confirming transferability of human-aligned scoring.

5.8. Operating Points: Thresholding and $F_{\beta }$ Optimization

Deployment requires selecting a threshold t that balances false rejects (over-filtering) and false accepts (unsafe captions). We thus optimize the $F_{\beta }$ score:

$$F_{\beta }=(1+{\beta }^2)·{\displaystyle \frac{\mathrm{Precision}·\mathrm{Recall}}{{\beta }^2·\mathrm{Precision}+\mathrm{Recall}}},$$

(22)

sweeping $t\in [0,1]$ and $\beta \in \{0.5,1,2\}$ depending on application (e.g., safety-critical settings prefer $\beta >1$ to emphasize recall of “good’’ captions while maintaining high precision). We report the chosen t on the dev set and hold it fixed for test-time evaluation to avoid leakage.

5.9. Caption Filtering Coverage and Retention

We measure the practical coverage of retained captions at fixed budgeting ratios. Consider a pipeline that surfaces only the top-$k\%$ by VQAR score (e.g., $k=20$). We compute Precision@Topk and Recall@Topk against expert labels:

Table 5. Top-$20\%$ retention: VQAR increases coverage significantly while improving precision.

Model	Precision@Top20%	Recall@Top20%
Pretrained baseline	0.78	0.20
VQAR (finetuned)	0.83	0.62

Table 5. Top-$20\%$ retention: VQAR increases coverage significantly while improving precision.

Model	Precision@Top20%	Recall@Top20%
Pretrained baseline	0.78	0.20
VQAR (finetuned)	0.83	0.62

These results suggest that VQAR can act as a front-door filter, preserving a large fraction of high-quality captions without sacrificing trustworthiness.

5.10. Human Utility Study (Qualitative Protocol)

To complement intrinsic metrics, we outline a small-scale utility protocol where human evaluators perform downstream tasks (e.g., scene understanding Q&A) with and without VQAR filtering. The primary outcomes are task success rate and perceived trust on a 5-point Likert scale. We pre-register the analysis plan, blind the study to model conditions, and use stratified sampling over image domains. While we do not report numeric outcomes here, pilot results indicated clearer task comprehension and fewer misinterpretations with VQAR-enabled filtering.

5.11. Error Taxonomy and Case Analyses

We group systematic failure modes into four categories: (i) subtle attribute drift (e.g., “red’’ vs “crimson’’) where human ratings remain permissive; (ii) relation mismatch (role swaps like “man holding a dog’’ vs. “dog holding a man’’) to which VQAR is sensitive; (iii) counting errors, partially mitigated by object-aware attention; and (iv) world knowledge gaps (commonsense or rare actions) where visual cues are ambiguous. Qualitative inspection shows bilinear fusion captures scene-level semantics, while the attention module penalizes hallucinated or absent entities.

5.12. Efficiency: Footprint and Latency

We profile inference on a single GPU with batched evaluation. Since encoders are frozen and only light-weight heads are trained, per-sample latency is dominated by feature extraction (which can be cached in production). The VQAR scoring stack adds sub-millisecond overhead per caption once features are available, making it feasible as an online gate in captioning services.

5.13. Limitations and Threats to Validity

First, binary feedback abstracts away fine-grained error types (object vs. relation vs. attribute). We partially address this by releasing an expert-labeled subset for nuanced evaluation but acknowledge residual ambiguity. Second, our ethical filtering excludes faces to respect privacy; consequently, performance on human-centric scenes may differ from general scenes. Third, while VQAR generalizes across domains (Section 5.7), extreme distribution shifts (e.g., medical or satellite imagery) may require adaptation. Finally, calibration depends on dev-set distributions; per-application recalibration is recommended.

VQAR reliably aligns with human preferences (high $\rho$, low MSE), offers calibrated probabilities after scaling, remains robust to meaning-altering perturbations, and generalizes to out-of-domain caption filtering. Architectural components such as bilinear fusion and cross-modal attention are crucial, and contrastive pretraining further stabilizes learning when followed by task-specific fine-tuning.

6. Conclusions and Future Work

In this paper, we have presented a new paradigm for evaluating the quality of automatically generated image captions without requiring reference texts. The motivation stems from the growing adoption of captioning systems in practical contexts, where erroneous or misleading descriptions can erode user trust, harm accessibility, and misguide downstream reasoning. To address this gap, we formulated caption quality estimation as a reference-free prediction problem and developed a large-scale framework for filtering captions based on their predicted alignment with human perception.

To enable this, we introduced a scalable human annotation protocol that produced more than 600,000 binary ratings over 55,000 unique image–caption pairs. This annotation scheme, while intentionally simple, offered broad coverage and semantic reliability, yielding a dataset that is both efficient to construct and robust enough to train machine learning models. By balancing scalability and consistency, this corpus provides an unprecedented resource for research into reference-free quality estimation.

Building upon this dataset, we designed VQAR (Visual-linguistic Quality Assessment for Reliability), a modular model that integrates pretrained encoders with bilinear fusion and cross-modal attention mechanisms to capture nuanced dependencies between images and captions. Through a combination of intrinsic correlation analysis and extrinsic filtering evaluations, we demonstrated that VQAR achieves strong agreement with human judgments and generalizes effectively to out-of-domain conditions. Notably, pretraining on large-scale image–text alignment tasks such as Conceptual Captions, followed by targeted fine-tuning on our corpus, further enhances correlation, calibration, and coverage. Together, these findings establish VQAR as a reliable gatekeeper for captioning systems in real-world deployments.

6.1. Future Work

While VQAR represents a significant step toward scalable and human-aligned caption evaluation, several promising directions remain open for exploration:

Multi-dimensional Caption Evaluation.

Our dataset currently provides binary judgments, but many real-world scenarios demand more nuanced feedback. Future work could extend VQAR to predict multi-criteria scores across dimensions such as correctness, fluency, specificity, and informativeness. Structured or continuous quality scales would allow richer supervision and could serve as a bridge toward comprehensive, fine-grained caption evaluation.

Integrating Quality Estimation into Generation.

Another natural extension is to leverage VQAR’s predictions as extrinsic feedback for caption generation models. Through reinforcement learning or reward modeling, VQAR-derived signals could be used to guide training objectives, encouraging generators to prioritize captions that align with human preferences beyond simple lexical overlap. This integration may help overcome the limitations of traditional n-gram based metrics.

Multilingual and Cross-Cultural Scenarios.

Current evaluations are conducted in English. Expanding the annotation protocol to multiple languages would enable training multilingual QE models that respect linguistic and cultural diversity. Such extensions are critical for building global accessibility tools and for ensuring that evaluation systems remain robust in multilingual applications.

Extension to Broader Vision–Language Tasks.

The principles underlying VQAR can also be applied to other tasks where textual alignment with visual inputs is essential. Potential applications include image retrieval, visual question answering, and open-domain multimodal agents. By functioning as a quality prior, VQAR could rerank or filter noisy outputs, thereby enhancing reliability in complex pipelines.

Towards Human-in-the-Loop Systems.

Finally, an exciting avenue is to incorporate VQAR into human-in-the-loop workflows. By providing calibrated reliability scores, VQAR could help allocate human verification effort to uncertain cases, striking a balance between automation efficiency and human oversight.

In summary, this work introduces both a large-scale corpus and a model that jointly advance the agenda of reference-free caption evaluation. We hope that VQAR will not only serve as a benchmark for research but also as a practical tool for improving the safety, trustworthiness, and usability of vision–language systems. The methods and findings presented here mark a step toward closing the gap between machine-generated descriptions and human expectations in real-world AI applications.