Cross-Modal Semantic-Enhanced Image Captioning

1. Introduction

The task of image captioning, which seeks to generate coherent and contextually relevant textual descriptions from images, represents a quintessential problem in cross-modal learning. This task lies at the intersection of computer vision and natural language processing, requiring an intricate fusion of visual perception and linguistic generation [1,2]. Given its broad applicability in fields such as assistive technology for visually impaired individuals, content-based image retrieval, and automatic multimedia annotation, image captioning has gained significant attention in both academia and industry [3,4,5,6]. However, achieving high-quality image captioning remains a nontrivial challenge due to the inherent complexity of multimodal representation learning and the substantial data dependency of modern deep learning architectures.

Traditional image captioning approaches predominantly rely on fully supervised learning paradigms, which necessitate large-scale paired datasets consisting of images and manually curated descriptive captions. These methods typically employ encoder-decoder architectures, where a convolutional neural network (CNN) acts as an encoder to extract visual features, and a recurrent neural network (RNN) or Transformer-based model functions as a decoder to generate captions sequentially [3,7]. Advanced techniques such as attention mechanisms [4,8,9] have been introduced to selectively focus on salient image regions, improving the alignment between image regions and corresponding textual elements. While these strategies have yielded promising results, they share a critical limitation: the necessity of a vast number of human-annotated image-caption pairs. Constructing such datasets is labor-intensive, costly, and often impractical for many real-world applications, particularly when scaling to new domains or languages.

A more practical scenario, often encountered in real-world applications, is the semi-supervised image captioning setting, wherein only a limited portion of images are annotated with ground-truth captions, while a significantly larger portion remains unlabeled. This scenario presents a fundamental challenge: how to effectively utilize the wealth of unlabeled images to improve captioning performance while avoiding the pitfalls of overfitting to the small supervised dataset or propagating incorrect pseudo-labels. Several existing approaches have attempted to address this problem by leveraging adversarial training [11,12], pseudo-labeling [14,15], or knowledge distillation from visual concept detectors. However, these methods often struggle to maintain semantic consistency between generated captions and visual content, leading to suboptimal caption quality.

To tackle these challenges, we introduce a novel Semi-Supervised Image Captioning via Cross-modal Predictive and Relational Consistency (SCPRC) framework, which enhances caption generation by leveraging both the predictive consistency of visual content and the structural alignment of semantic relations between images and their generated textual descriptions. Unlike conventional pseudo-labeling techniques, which rely solely on model-generated captions as self-supervision signals, SCPRC directly incorporates predictive cues extracted from raw image representations as soft labels to refine and regulate the generated textual output. Additionally, we introduce a relation consistency mechanism that enforces structural alignment between images and captions by preserving their semantic relational distribution within a shared latent space. These two principles collectively ensure that the generated captions are both informative and representative, thereby improving the robustness of the captioning model under semi-supervised conditions.

The primary challenge in semi-supervised image captioning lies in designing a robust self-supervision mechanism that effectively leverages the vast amount of unlabeled data. Specifically, several key difficulties must be addressed:

• Heterogeneous Modality Gap: The intrinsic differences between visual and linguistic representations make it challenging to establish direct correspondence between images and textual descriptions. Traditional approaches that enforce global embedding similarity between images and captions often fail to capture fine-grained semantic nuances.
• Noisy Pseudo-Labels: Existing pseudo-labeling techniques suffer from error propagation, where incorrect captions generated by an initially weak model degrade learning performance. A more reliable supervisory signal is needed to guide caption refinement without amplifying model biases.
• Lack of Structural Awareness: Current methods primarily focus on instance-level captioning without considering the relational structure of generated captions. Since real-world captions inherently encode relational knowledge (e.g., spatial relationships, object interactions), it is crucial to ensure that generated captions preserve these semantic relationships across different augmentations of the same image.

To overcome these challenges, our proposed SCPRC framework introduces a novel approach to semi-supervised image captioning by integrating cross-modal predictive learning and relational consistency constraints. Our contributions can be summarized as follows:

• Prediction Consistency as Soft Label Supervision: Instead of relying on rigid pseudo-labeling, we introduce a soft-labeling mechanism where semantic predictions extracted from raw image representations serve as supervision signals for caption generation. This technique enhances the reliability of self-supervision and mitigates the issue of noisy pseudo-labels.
• Relational Consistency for Structural Alignment: We introduce a novel relational consistency loss that ensures the semantic relationships between objects and concepts in generated captions align with those present in the visual domain. This approach improves the coherence and contextual relevance of generated captions.
• A Flexible and Scalable Framework: SCPRC can be easily integrated into existing captioning models, such as Transformer-based or CNN-RNN-based architectures, without requiring major modifications. Furthermore, our approach effectively scales to scenarios with varying degrees of supervision, making it suitable for practical deployment.
• Superior Performance on Benchmark Datasets: Our extensive experiments on the MS-COCO dataset demonstrate that SCPRC significantly outperforms state-of-the-art semi-supervised image captioning methods. We achieve at least 12% improvements in CIDEr-D scores, showcasing the efficacy of our method in handling non-parallel and weakly supervised scenarios.

The remainder of this paper is structured as follows: In Section 2, we review relevant literature on image captioning and semi-supervised learning. Section 3 details our proposed SCPRC framework, including the predictive consistency and relational consistency modules. Section 4 presents empirical evaluations, demonstrating the superiority of our method across various settings. Finally, Section 5 concludes the paper and discusses potential future research directions.

2. Related Work

2.1. Advancements in Image Captioning

The development of image captioning methodologies has seen a progressive evolution from basic template-based techniques to sophisticated neural network models. Early attempts, such as template-based methods, were rudimentary, involving manually created caption structures filled by detected keywords [19]. These methods, although straightforward, suffered from a lack of flexibility and depth in expression.

The encoder-decoder paradigm marked a significant advancement in this field, inspired by its success in neural machine translation [20]. This approach, exemplified by the work of Vinyals et al., who introduced an end-to-end trainable model combining CNNs for visual encoding with LSTMs for textual output, set a new standard for image captioning [21]. Enhancements such as the attention-on-attention module proposed by Huang et al. further refined this model by aligning the focus of the LSTM and attention mechanisms more closely with the relevant parts of the image, enhancing the relevance and accuracy of the generated captions [9].

Another intriguing direction in this field is the editing-based methods which focus on refining pre-existing captions rather than generating new ones from scratch. This approach, which includes works like Hashimoto et al., utilizes a retrieval model to adjust existing captions by embedding the input in a task-specific manner [10]. Similarly, Sammani and Elsayed proposed a method to incrementally improve captions by modeling the residual information between the generated and target texts [5]. These methods significantly reduce the dependency on large-scale annotated datasets, instead focusing on enhancing the quality of generated content through iterative refinement.

Despite these advancements, the challenge remains in scenarios dominated by a large number of undescribed images, typical of real-world applications. This gap has prompted the exploration of unsupervised and semi-supervised techniques. For instance, Feng et al. and Gu et al. have explored using visual concept detectors and adversarial learning to facilitate unsupervised learning of captions, aiming to bridge the gap between the visual content of images and their linguistic descriptions without extensive labeled data [11,12]. However, these approaches often struggle with generating contextually and grammatically coherent sentences due to their reliance on domain discriminators which do not fully capture the nuances of natural language.

2.2. Innovations in Semi-Supervised Learning

The domain of semi-supervised learning has witnessed substantial growth, driven by the need to leverage unlabeled data alongside labeled examples. This learning paradigm is particularly crucial when labeling costs are prohibitive, especially requiring expert human annotation. Traditional semi-supervised techniques include self-training, where a model’s own predictions on unlabeled data are used to guide its learning process [22]. Pseudo-labeling, a variant of self-training, involves assigning hard labels to unlabeled data based on model predictions, often coupled with confidence thresholding to only include high-confidence predictions [23].

Consistency regularization represents another cornerstone of semi-supervised learning, emphasizing the model’s performance stability across various perturbations of the same input [24]. Techniques such as using the exponential moving average of model parameters or employing model checkpoints to enforce consistency have shown promise [25,26]. More recent approaches, such as those by Xie et al. and Berthelot et al., integrate data augmentation into this framework, enhancing the model’s ability to generalize across different data transformations by aligning the outputs of weakly and strongly augmented data instances [27,28].

Combining pseudo-labeling with consistency regularization, as explored by Kihyuk, offers a unified framework that leverages both prediction consistency and data augmentation to refine model performance on unlabeled data. This approach has been especially effective in contexts where labeled data is scarce, demonstrating the potential to reduce reliance on extensive labeled datasets [30].

In the context of cross-modal learning, these semi-supervised techniques face unique challenges, particularly when aligning disparate modalities such as visual images and textual descriptions. The complexity of creating a cross-modal generator capable of synthesizing coherent sentences from visual input is magnified in a semi-supervised setting, where the risk of noise accumulation is significant. Thus, refining these techniques to better handle the nuances of cross-modal data remains a critical area of ongoing research.

3. Our Methodology

3.1. Preliminaries

In our semi-supervised learning framework, we denote the image-sentence dataset as $\mathcal{D}=\{{({\mathbf{v}}_i,{\mathbf{w}}_i,{\mathbf{y}}_i)}_{i=1}^{N_l},{\left({\mathbf{v}}_j\right)}_{j=1}^{N_u}\}$. Here, ${\mathbf{v}}_i\in {\mathbb{R}}^{d_v}$ represents the feature vector of the i-th labeled image, ${\mathbf{w}}_i\in {\mathbb{R}}^{d_w}$ is the corresponding textual description in vector form, and ${\mathbf{y}}_i\in {\{0,1\}}^C$ is a binary label vector indicating the presence of various attributes or classes within the image. ${\mathbf{v}}_j$ denotes the feature vector of the j-th unlabeled image, and $N_l$ and $N_u$ represent the numbers of labeled and unlabeled images, respectively. The challenge is to leverage this data disparity effectively, where typically $N_l\ll N_u$.

Semi-Supervised Image Captioning. Our goal is to develop a generative model G, capable of producing accurate and contextually relevant captions for a given image using a hybrid dataset of both labeled and vastly more numerous unlabeled images.

3.2. The SCPRF Framework

The Semantic Consistency and Predictive Regulation Framework (SCPRF) operates on the principle of utilizing both described and undescribed images to train a robust image captioning model. This model G can adopt any advanced neural network architecture designed for captioning tasks, such as the attention-based AoANet [9]. The framework encapsulates two major processes:

• Encoder-Decoder Model: The image $\mathbf{v}$ is first passed through an encoder E, typically a deep convolutional neural network, which extracts a dense representation of the visual content. The decoder D, often an LSTM or Transformer model, then translates this representation into a coherent sequence of words $\widehat{\mathbf{w}}$, forming the caption.
• Attention Mechanism: Within both E and D, attention mechanisms dynamically focus on different regions of the image and different segments of the generated text, respectively, to improve the relevance and accuracy of the caption generation.

3.3. Supervised and Unsupervised Loss Components

3.3.1. Generation Loss

The generation process involves predicting the sequence of words that form the caption:

$$\begin{array}{cc}\hfill {\ell }_{XE}& =-\sum _{t=1}^{T}logp\left(w_t{w}_{1:t-1}\right),\hfill \end{array}$$

(1)

$$\begin{array}{cc}\hfill {\ell }_{RL}& =-{\mathbb{E}}_{w_{1:T}\sim p}\left[r\left(w_{1:T}\right)\right],\hfill \end{array}$$

(2)

where T is the length of the caption, and $p(·)$ and $r(·)$ represent the probability and reward functions, respectively. The reward function is typically aligned with human evaluative metrics like CIDEr-D, ensuring that the generated captions are not only grammatically correct but also contextually appropriate.

3.3.2. Prediction Loss

This component bridges the gap between the visual and textual modalities by ensuring that the embeddings from both domains align semantically:

$$\begin{array}{cc}\hfill p^v& =f\left(E_e\left(\mathbf{v}\right)\right),\hfill \end{array}$$

(3)

$$\begin{array}{cc}\hfill p^w& =f\left(D_e\left(E\left(\mathbf{v}\right)\right)\right).\hfill \end{array}$$

(4)

This alignment is critical for maintaining the integrity of content across modalities and is achieved through a shared classifier f which operates on the embeddings produced by the encoder and decoder.

3.3.3. Unsupervised Learning via Consistency

To exploit the unlabeled images, SCPRF introduces a novel unsupervised learning component that uses consistency-based techniques:

• Predictive Consistency: Encourages the model to maintain consistent predictions across different augmentations or perturbations of the same image, enhancing robustness and reliability.
• Relational Consistency: Goes beyond individual consistency by ensuring that relational dynamics—such as the relative positions and interactions between objects within images—are preserved in the transition from visual to textual representation.

3.4. Comprehensive Loss Function

The overall loss function combines these components into a unified training objective:

$$\mathcal{L}=\sum _{i=1}^{N_l}{\ell }_{supervised}({\mathbf{v}}_i,{\mathbf{w}}_i,{\mathbf{y}}_i)+\lambda \sum _{j=1}^{N_u}{\ell }_{unsupervised}\left({\mathbf{v}}_j\right),$$

(5)

where $\lambda$ serves as a regularization parameter that balances the contributions of the supervised and unsupervised losses. This balance is crucial for leveraging the full potential of the available data, maximizing the learning from both labeled and unlabeled images.

3.5. Implementation Details

Implemented using modern deep learning frameworks, SCPRF is designed to be both scalable and adaptable, capable of being trained on various hardware configurations efficiently. The framework’s modularity allows for easy experimentation with different encoder and decoder architectures, making it suitable for a broad range of applications beyond just image captioning, such as video description and automated storytelling.

4. Experiments

4.1. Dataset and Experimental Setup

To comprehensively evaluate the effectiveness of the proposed SCPRF (Semantic Consistency and Predictive Regulation Framework), we conduct experiments on the widely used MS COCO dataset [39]. This dataset has been the primary benchmark in previous image captioning studies [9,17,18,40,41] due to its large-scale annotated images and standard evaluation protocols.

Dataset Details: The MS COCO dataset consists of 123,287 images, including 82,783 training images and 40,504 validation images. Each image is associated with five human-written captions, making it an ideal dataset for training and evaluating captioning models. The commonly used test set follows the “Karpathy” split [42], which comprises 5,000 validation images, 5,000 test images, and the remaining images for training.

Semi-Supervised Learning Setting: To simulate a semi-supervised scenario, a small subset of the training data is randomly selected as supervised image-caption pairs, while the remaining images are used as unlabeled data. This reflects a realistic condition where obtaining labeled captions is expensive, but large amounts of unlabeled images are available.

Evaluation Metrics: We evaluate model performance using five widely adopted metrics in image captioning:

• BLEU@N (B@N) [44]: Measures n-gram precision between generated captions and ground truth.
• METEOR (M) [45]: Considers synonym matching and stemming for improved alignment.
• ROUGE-L (R) [39]: Evaluates sequence overlap recall.
• CIDEr-D (C) [34]: Measures consensus-based n-gram similarity with human-written captions.
• SPICE (S) [46]: Uses scene graphs to assess semantic correctness.

Given that CIDEr-D and SPICE are more indicative of semantic relevance, they serve as primary performance indicators.

4.2. Implementation Details

To demonstrate the effectiveness of SCPRF, we adopt the AoANet [9] model as the base captioning generator G. The following implementation details are employed:

Model Configuration:

• Encoder: A ResNet-101 [31] CNN extracts image features, followed by an attention mechanism to focus on important regions.
• Decoder: A recurrent LSTM-based sequence generator converts the visual representation into textual descriptions.
• Shared Classifier: A three-layer fully connected network with hidden layers of size 1024, used for prediction consistency learning.

Training Details:

• Data Augmentation: Each image undergoes three augmentations ($K=3$) using a random occlusion strategy.
• Optimizer: Adam [43] with an initial learning rate of ${10}^{-4}$, decayed by 0.8 every 3 epochs.
• Batch Size: 16 images per batch.
• Training Duration: The model is trained for 40 epochs on an NVIDIA TITAN X GPU.
• Hyperparameters: Loss function weights ${\lambda }_1,{\lambda }_2\in \{0.01,0.1,1,10\}$; confidence threshold $\tau =0.1$.

4.3. Comparison with State-of-the-Art Methods

We compare SCPRF with the following state-of-the-art approaches:

• Supervised Models: SCST [17], AoANet [9], AAT [40], ORT [41], GIC [18].
• Unsupervised Captioning Models: Graph-align [12], UIC [11].
• Semi-Supervised Model: A3VSE [15].

Additionally, we conduct ablation studies to assess the contributions of SCPRF components.

4.4. Quantitative Performance Analysis

Table 1 provides a comprehensive comparison of SCPRF against various baseline models under a semi-supervised setting, where only 1% of the dataset is labeled, and the remaining 99% is unlabeled. The results highlight several key insights regarding the strengths and limitations of different approaches in handling low-resource caption generation tasks.

First, fully supervised models such as AoANet and SCST struggle significantly in this constrained environment. These methods rely heavily on abundant labeled data, and when such annotations are scarce, their ability to generate high-quality captions is greatly diminished. The lack of sufficient supervision leads to weaker semantic alignment between image and text, resulting in a drop in overall performance across all evaluation metrics.

Second, purely unsupervised methods, including UIC and Graph-align, perform substantially worse than their supervised counterparts. These models typically rely on domain-discriminator-based learning techniques, which are not well-suited for the complexity of image caption generation. Since these approaches lack direct supervision, they often fail to establish meaningful correspondences between images and their generated textual descriptions, leading to frequent mismatches and a substantial drop in scores.

Furthermore, we observe that A3VSE, a representative semi-supervised learning approach, provides only marginal improvements over the supervised baselines. While this method attempts to leverage both labeled and unlabeled data, it does not effectively utilize the vast amount of unlabeled images for robust caption generation. The limited gains suggest that its strategy for integrating unsupervised data is insufficient to bridge the gap between supervised and fully unsupervised models.

Finally, SCPRF consistently outperforms all competing methods across multiple evaluation metrics. In particular, it achieves a CIDEr-D score of 78.8 and a SPICE score of 16.8, demonstrating its superior ability to generate semantically rich and contextually accurate captions. These results validate the effectiveness of SCPRF’s innovative semi-supervised framework, which successfully distills meaningful information from unlabeled images while maintaining high-quality sentence generation.

4.5. Ablation Study

To further understand the contributions of different components in SCPRF, we conduct a series of controlled ablation studies by systematically removing or replacing key elements of the model. The results illustrate the necessity of each proposed module and highlight their individual impact on overall performance.

One crucial observation is that removing the Prediction Consistency module results in a notable drop of 2.5 points in CIDEr-D. This finding underscores the importance of enforcing consistency between generated sentences and image representations, as this module helps retain critical semantic information and refines the captioning model’s predictions. Without this component, the generated sentences become less informative and struggle to maintain coherence with the corresponding images.

Similarly, eliminating the Relation Consistency module leads to a 2.1-point reduction in SPICE, which is particularly significant given that SPICE evaluates the structural and semantic richness of generated captions. This result suggests that enforcing relational constraints between augmented images and their generated captions is essential for maintaining structural consistency. By capturing higher-order relationships among different variations of an image, SCPRF ensures that the generated captions remain aligned with the underlying scene.

Moreover, when traditional Pseudo-Labeling (PL) techniques are used instead of SCPRF’s cross-modal consistency framework, performance significantly deteriorates. Conventional pseudo-labeling methods often introduce label noise, as they rely solely on hard-label assignments without considering the cross-modal dependencies inherent in image-text tasks. This substitution leads to unreliable supervision signals, ultimately degrading caption quality.

These findings collectively confirm that both Prediction Consistency and Relation Consistency play indispensable roles in SCPRF’s effectiveness. The synergy between these two components allows SCPRF to extract meaningful supervisory signals from unlabeled images while preserving the structural integrity of generated captions.

4.6. Effect of Augmentations and Threshold

We also investigate the impact of two critical hyperparameters in SCPRF: the number of augmentations applied per image (K) and the confidence threshold ($\tau$) used for filtering pseudo-labels. Both parameters directly influence how SCPRF leverages unlabeled data and affect the trade-off between information retention and noise suppression.

Our experiments reveal that the optimal number of augmentations is $K=3$. When fewer augmentations are used (e.g., $K=1$ or $K=2$), the model does not fully exploit the variability of the input images, leading to less robust representations. On the other hand, increasing the number of augmentations beyond three introduces inconsistencies between augmented images and their generated captions, which in turn increases the risk of noise accumulation and disrupts the learning process. This suggests that moderate augmentation enhances model generalization, while excessive augmentation can inject distortions that degrade performance.

Another important observation is that setting the confidence threshold at $\tau =0.1$ yields the best performance. A low threshold (e.g., $\tau =0$) allows SCPRF to utilize all pseudo-labels, but many of these labels may be noisy, which ultimately harms model stability. Conversely, a high threshold (e.g., $\tau =0.7$) aggressively filters out potential training samples, thereby reducing the amount of usable information and weakening the model’s capacity to learn from unlabeled data. The optimal threshold value strikes a balance between including informative pseudo-labels and filtering out unreliable ones.

These results highlight the importance of carefully selecting augmentation strategies and threshold settings when training SCPRF. By leveraging a moderate number of augmentations and a well-calibrated confidence threshold, SCPRF maximizes the benefits of semi-supervised learning while minimizing the negative effects of noise.

4.7. Generalization Across Captioning Models

To evaluate whether SCPRF is applicable across different model architectures, we integrate it with multiple widely used captioning frameworks, including SCST and GIC. As demonstrated in Table 2, SCPRF consistently enhances performance across all tested architectures, further reinforcing its generalizability and adaptability.

Table 2. Performance of SCPRF with different caption models on MS-COCO test set.

Methods	Cross-Entropy Loss
	B@1	B@2	B@3	B@4	M	R	C	S
SCST	56.3	38.1	25.0	16.0	15.8	42.1	38.6	9.1
GIC	62.8	46.5	32.9	19.7	19.0	50.1	50.2	12.2
SCST+SCPRF	63.0	45.5	31.5	21.2	19.2	45.6	47.8	10.0
GIC+SCPRF	66.3	47.2	34.2	21.1	19.1	50.6	57.2	13.2
Methods	CIDEr-D Score Optimization
	B@1	B@2	B@3	B@4	M	R	C	S
SCST	58.9	39.2	25.1	16.1	16.8	42.5	43.2	9.6
GIC	64.4	46.6	31.8	20.5	18.8	47.6	55.4	12.3
SCST+SCPRF	66.1	47.7	33.5	22.5	20.2	47.6	48.3	10.5
GIC+SCPRF	66.6	47.5	34.5	21.6	19.7	47.9	58.4	13.4

Table 2. Performance of SCPRF with different caption models on MS-COCO test set.

Methods	Cross-Entropy Loss
	B@1	B@2	B@3	B@4	M	R	C	S
SCST	56.3	38.1	25.0	16.0	15.8	42.1	38.6	9.1
GIC	62.8	46.5	32.9	19.7	19.0	50.1	50.2	12.2
SCST+SCPRF	63.0	45.5	31.5	21.2	19.2	45.6	47.8	10.0
GIC+SCPRF	66.3	47.2	34.2	21.1	19.1	50.6	57.2	13.2
Methods	CIDEr-D Score Optimization
	B@1	B@2	B@3	B@4	M	R	C	S
SCST	58.9	39.2	25.1	16.1	16.8	42.5	43.2	9.6
GIC	64.4	46.6	31.8	20.5	18.8	47.6	55.4	12.3
SCST+SCPRF	66.1	47.7	33.5	22.5	20.2	47.6	48.3	10.5
GIC+SCPRF	66.6	47.5	34.5	21.6	19.7	47.9	58.4	13.4

Table 3. Performance of SCPRF with different ratios of unsupervised data (supervised data fixed at 1%) on MS-COCO “Karpathy” test split. Metrics: B@N (BLEU@N), M (METEOR), R (ROUGE-L), C (CIDEr-D), S (SPICE).

Methods	Cross-Entropy Loss
	B@1	B@2	B@3	B@4	M	R	C	S
10%	67.9	49.2	34.5	23.0	21.0	49.3	71.2	14.4
40%	66.4	48.4	33.8	23.1	22.5	49.2	72.3	15.3
70%	68.0	50.1	35.2	24.1	22.6	50.0	73.8	15.7
100%	68.5	50.7	35.5	24.6	22.7	50.1	77.1	16.0
Methods	CIDEr-D Score Optimization
	B@1	B@2	B@3	B@4	M	R	C	S
10%	68.2	50.5	25.1	23.5	22.1	50.3	73.5	14.7
40%	68.9	49.9	35.1	23.8	22.5	50.5	75.1	15.7
70%	69.1	51.0	36.1	24.5	22.6	50.6	76.0	16.0
100%	69.6	51.5	36.4	25.2	23.0	50.6	78.2	16.5

Table 3. Performance of SCPRF with different ratios of unsupervised data (supervised data fixed at 1%) on MS-COCO “Karpathy” test split. Metrics: B@N (BLEU@N), M (METEOR), R (ROUGE-L), C (CIDEr-D), S (SPICE).

Methods	Cross-Entropy Loss
	B@1	B@2	B@3	B@4	M	R	C	S
10%	67.9	49.2	34.5	23.0	21.0	49.3	71.2	14.4
40%	66.4	48.4	33.8	23.1	22.5	49.2	72.3	15.3
70%	68.0	50.1	35.2	24.1	22.6	50.0	73.8	15.7
100%	68.5	50.7	35.5	24.6	22.7	50.1	77.1	16.0
Methods	CIDEr-D Score Optimization
	B@1	B@2	B@3	B@4	M	R	C	S
10%	68.2	50.5	25.1	23.5	22.1	50.3	73.5	14.7
40%	68.9	49.9	35.1	23.8	22.5	50.5	75.1	15.7
70%	69.1	51.0	36.1	24.5	22.6	50.6	76.0	16.0
100%	69.6	51.5	36.4	25.2	23.0	50.6	78.2	16.5

When applied to SCST, SCPRF achieves a notable improvement in both CIDEr-D and SPICE scores, confirming that its semi-supervised learning framework is effective even when incorporated into reinforcement learning-based captioning models. This suggests that SCPRF’s consistency constraints provide additional benefits beyond standard RL-based optimization.

Similarly, SCPRF enhances GIC’s performance by a substantial margin. Since GIC employs an attention mechanism to refine caption generation, the improvements indicate that SCPRF’s consistency-based supervision complements attention-based models by reinforcing structural integrity and semantic correctness.

The consistent improvements across different architectures demonstrate that SCPRF is not restricted to a specific model design, but rather serves as a general-purpose semi-supervised framework that can be applied to diverse image captioning systems. This versatility makes SCPRF an attractive choice for various real-world applications where labeled data is scarce but large-scale unlabeled image collections are available.

4.8. Sensitivity to Parameters

The primary hyperparameters in our proposed method, denoted as ${\lambda }_1$ and ${\lambda }_2$ in Eq. 5, play a crucial role in balancing different components of the loss function. To analyze their sensitivity, we vary ${\lambda }_1,{\lambda }_2\in \{0.01,0.1,1,10\}$ and evaluate the performance of SCPRF across multiple metrics. The results indicate that the best performance is consistently achieved with a smaller ${\lambda }_1$ (i.e., ${\lambda }_1=0.01$) and a larger ${\lambda }_2$ (i.e., ${\lambda }_2=10$). This result suggests that relation consistency loss has a more significant impact on improving the generator than prediction consistency. Furthermore, the robustness of SCPRF across various ${\lambda }_1$ and ${\lambda }_2$ values confirms its adaptability in different semi-supervised settings.

5. Conclusions and Future Directions

5.1. Conclusions

Image captioning, which aims to generate textual descriptions for visual content, has traditionally relied on large-scale supervised datasets. However, acquiring such labeled data is costly and impractical for many real-world applications. To address this challenge, this paper presents SCPRF (Semantic Consistency and Predictive Regulation Framework), a novel semi-supervised learning approach that effectively integrates a vast collection of unlabeled images with a limited set of annotated image-caption pairs.

SCPRF introduces two core mechanisms to enhance captioning quality in low-resource settings. Prediction Consistency mitigates the noise commonly introduced by pseudo-labeling by enforcing a soft-labeling strategy, where image-based predictions guide sentence generation. Relation Consistency ensures structural coherence by aligning the relational distributions between augmented images and their corresponding captions, preserving cross-modal knowledge. By leveraging these constraints, SCPRF enhances the quality and semantic accuracy of generated captions without relying heavily on annotated data.

Comprehensive experiments on the MS COCO dataset demonstrate that SCPRF significantly outperforms existing supervised, unsupervised, and semi-supervised captioning models. It effectively utilizes large-scale unlabeled data, improving performance even in scenarios where only 1% of the data is labeled. The results confirm that enforcing prediction and relation consistency leads to substantial gains in caption fluency, coherence, and semantic alignment, making SCPRF a promising solution for semi-supervised image captioning.

5.2. Future Research Directions

While SCPRF presents an effective approach for semi-supervised image captioning, there are several areas for future exploration to further enhance its capabilities and extend its applications.

A key direction is improving the generalizability of SCPRF across diverse datasets. Although the framework has been validated on MS COCO, its applicability to other large-scale datasets such as Flickr30K and Conceptual Captions remains an open question. Investigating domain adaptation techniques could enable SCPRF to perform well across different datasets, while cross-lingual image captioning could further extend its reach to multilingual applications.

Another promising area is incorporating self-supervised learning techniques to enhance SCPRF’s ability to extract meaningful representations from unlabeled data. Recent advances in contrastive learning and masked language modeling suggest that additional self-supervised objectives could improve the alignment between vision and language representations. Integrating contrastive learning could refine image-text embeddings, while masked caption generation could enhance fluency and contextual coherence.

Optimizing data augmentation strategies is also crucial for improving model robustness. While SCPRF currently employs weak augmentation techniques such as random occlusions, stronger augmentations such as style transfer and adversarial perturbations could provide additional training signals. However, excessive augmentation may introduce semantic inconsistencies, requiring further research into adaptive augmentation policies that dynamically adjust transformations based on model confidence.

Extending SCPRF to video captioning is another exciting direction. Unlike static images, videos require models to capture temporal dependencies and motion dynamics. SCPRF’s relational consistency framework could be adapted for video-based descriptions by incorporating spatio-temporal constraints that ensure coherence across consecutive frames. Exploring transformer-based architectures tailored for long-range video understanding could further enhance the applicability of SCPRF in video captioning tasks.

Beyond improving technical performance, ensuring fairness, interpretability, and trustworthiness in captioning models is critical for real-world deployment. AI-generated captions can unintentionally reflect biases present in training data, making it important to develop debiasing techniques to mitigate gender, racial, and cultural biases. Additionally, integrating explainability methods could provide users with insights into why specific captions were generated, fostering greater trust in AI-assisted vision-language systems.

Finally, real-world deployment and interactive learning represent the next step in making SCPRF more practical. Incorporating human-in-the-loop learning mechanisms, where users provide feedback to iteratively refine captions, could enhance adaptability in dynamic environments. Moreover, integrating SCPRF into assistive applications, such as accessibility tools for visually impaired users or automated content creation platforms, could demonstrate its real-world impact.