Fine-Grained Cross-Modal Semantic Consistency in Natural Conservation Image Data from a Multi-Task Perspective

1. Introduction

Neural networks function as parameterized databases, typically driven by specific tasks, with each network dedicated to fulfilling a corresponding task. However, there are instances where our requirements transcend single-task boundaries. Consider the context of rapidly accumulating natural conservation area image data. We seek not only to retrieve a single image but also to attach essential descriptions when summoning an image. Furthermore, we aspire to employ textual descriptions as queries to sift through our image repository, locating images that align with our specific needs. This scenario necessitates simultaneous engagement with two tasks: cross-modal image-text retrieval and image captioning.

As these data accumulate over time, the volume becomes formidable. For example, the Snapshot Serengeti Project at Serengeti National Park, Tanzania deployed hundreds of camera traps to understand the dynamics of African animal species. From 2010 to 2013, the project collected 3.2 million images from 225 camera traps[1]. And it was found very costly to manually process the images and add annotation labels given such a large amount of data. The project carried out by Literature [2]requires thousands of technical volunteers to work for 2-3 months to annotate image data. With the improvement of camera manufacturing technology, each camera deployed in the field can record more than 40,000 photos per day due to a single trigger event[3], and there were a lot of camera traps deployed in related projects. Literature[4] and Literature[5] deployed hundreds of camera traps in their project. Literature[6] and Literature[7] deployed about 50 cameras at water sources in the natural conservation and recorded more than 800,000 wildlife images within a few weeks.

When we resort to two separate models to independently address these tasks, we encounter suboptimal outcomes. Specifically, the images retrieved through descriptive text queries may not align with the descriptive text generated by the model for the same image. In other words, these two models exhibit inconsistent encoding and decoding for the same data. Can we train a model that maintains consistency during both encoding and decoding, all while meeting task requirements, thus mitigating semantic ambiguity within our cross-modal parameterized database?

To address this, we propose a multi-task model for joint training in cross-modal image-text retrieval and image captioning. Through the collaborative optimization of parameters, we achieve cross-module information sharing, thereby facilitating semantic-consistency encoding and decoding modeling. Post-training, the encoder and decoder can be independently employed to perform cross-modal image-text retrieval and image captioning tasks while maintaining semantic consistency between the two tasks. This is made possible because our model is constructed upon a foundation of shared semantic-consistency representation space. Of course, the prerequisite is the construction of a dataset aligning with our specific needs and the judicious design of the model’s structure.

As illustrated in Figure 1, we are able to retrieve corresponding images from the dataset using a customized textual input and subsequently generate descriptive text for the retrieved images. In this paper, our objective is to preserve semantic consistency in the context of fine-grained visual features and rich textual descriptions by jointly training a retrievaler and a captioner.

The contributions of this work include 1) creation of a dataset of image-text pairs for natural conservation; 2) propose a combined offline and online training approach; 3) introduce a method for information transfer through collaborative parameter solving within a multi-task module; and 4) present a technique for cross-modal alignment and semantic consistency preservation based on a shared representation space for cross-modal tasks.

2. Related Work

The cross-modal semantic consistency between images and text in our research is primarily achieved through the model design and joint training of two tasks: cross-modal retrieval and image captioning. The essence of this approach lies in the optimization of the cross-modal shared space embedding of images and text. On one hand, optimization is performed from the perspective of cross-modal alignment between image and text entities. On the other hand, the model needs to reorganize tokens related to the input image representation in the shared space in an autoregressive manner and output them in natural language, thereby achieving semantic consistency between image and text descriptions at a broader and deeper semantic level. The encoder and decoder constitute the core modules of our designed model, involving popular techniques in cross-modal alignment and cross-modal representation fusion. Subsequently, the literature review will delve into both cross-modal representation alignment and cross-modal representation fusion.

2.1. Cross-modal Alignment

Currently, research on cross-modal alignment of image and text representations is predominantly centered around contrastive learning methods. These studies achieve the embedding and alignment of image and text representations in a shared cross-modal space by training encoders separately for each modality using a contrastive learning loss. ConVIRT[8] demonstrated the potential of contrastive objectives to learn image representations from text. Inspired by ConVIRT, CLIP[9] performs pre-training on a dataset containing 4 billion image-text pairs and becomes a milestone of vision-language models with excellent cross-modal representation. CLIP4Clip[10] demonstrated the CLIP model with high performance in cross-modal retrieval. ALIGN[11] performs pre-training on massive noisy web data. The above methods all use contrastive loss which is the most effective loss for cross-modal alignment[12,13,14,15].

Intuitively, performing cross-modal contrastive learning by treating corresponding visual and textual entities as inputs to image and text encoders, respectively, can achieve better cross-modal alignment. Therefore, some research in this domain utilizes object detection models as visual unit extractors. The extracted target pixel regions are then fed to the image encoder for contrastive learning with the text encoder, enhancing the performance of cross-modal representations. Often, these studies require the integration of a pre-trained object detection model at the front end of the visual data input[16,17,18]. An intuitive approach is to align the visual features of the region where the object is located with the label. For example, Oscar[19] uses Faster R-CNN[20] to detect the object in the image and then aligns it with the word embeddings of the object tags. However, they are not suitable for fine-grained cross-modal alignment as the object tags are too limited to align the vision features suitably. With a properly designed prompt, CLIP can be used for open-vocabulary classification, which solves the problem of limited object tags. ViLD[21] designed an open vocabulary object detection model by knowledge distillation from the CLIP. [22] achieved a language-driven zero-shot semantic segmentation by directly using the representation of CLIP. Groupvit[23] implements unsupervised image segmentation by using the text representation of CLIP as a pseudo label.

Contrastive learning with dual encoders, while excelling in cross-modal retrieval tasks involving images and text, encounters challenges in adapting to fine-grained cross-modal retrieval tasks with natural conservation images due to the following reasons. First, certain species’ visual features in natural conservation images exhibit high intra-class and inter-class similarities, resulting in dense distributions of these highly similar representations in the shared space. This necessitates encoders with finer discriminative capabilities. Second, these encoders, trained on image-text pair datasets using contrastive learning, are often constrained by the representation of text descriptions alone and struggle to adapt well to cross-modal retrieval tasks where the semantics are similar but the expression methods differ.

2.2. Cross-Modal Fusion

With the successful application of the transformer[24] architecture in the fields of natural language processing, computer vision, and multimodal, ViLT[25]proposed a transformer-based multimodal encoder which focused on cross-modal feature fusion, and take the masked language modeling loss [26] for visual Embedding as future work. This work has been achieved by VL-BEiT[27] after VIT[28] and MAE[29]. From then on, a big convergence of language, vision, and multimodal pretraining is emerging. BLIP[30] proposed a new vision-language pre-training framework that transfers flexibly to both vision-language understanding and generation tasks. The multiway transformer proposed by BEiT-V3[31] has achieved state-of-the-art transfer performance in both vision and vision-language tasks. FLIP[32], which is called Fast Language-Image Pre-training, presents a simple and more efficient method for training CLIP by dropping a part of masked tokens. VLMo[33] jointly learns a dual encoder and a fusion encoder with a modular Transformer network. Coca [34] is a minimalist design to pre-train an image-text encoder-decoder foundation model jointly with contrastive loss and captioning loss like CLIP and SimVLM[35] respectively.

Cross-modal feature fusion is not suitable for cross-modal retrieval tasks due to the lack of effective optimization for unimodal encoders. However, when applied to image captioning tasks for the same input image, this method generates descriptions that share the same semantics but have different expressions. This indicates that such methods contribute to solving cross-modal semantic consistency. Our research goal is to explore the joint application of cross-modal feature fusion and cross-modal feature alignment, aiming to leverage their respective strengths and compensate for weaknesses, fostering mutual enhancement. This objective is emphasized in the Method section for in-depth discussion.

3. Materials and Methods

The entire model design and objective function construction are aimed at harnessing redundant textual information to eliminate cross-modal ambiguity in image-text pairs, thereby achieving fine-grained semantic consistency. This results in a cross-modal shared embedding space from the perspectives of both image-text retrieval and image captioning tasks. The core concept involves initially extracting representations from various modalities and aligning them preliminarily through ITC (Image Text Contrastive Learning) loss. Subsequently, image features are further semantically disambiguated within a cross-modal residual attention network through MLM(Mask Language Modeling) loss, using richer textual descriptions. This results in a further refinement of the shared embedding space. Finally, the semantically aligned image embeddings are fed to the captioner to generate textual descriptions by the LM(Language modeling) loss, enhancing the model’s deeper understanding of cross-modal semantics in image-text contexts. As illustrated in Figure 2, we employed neural network architecture to seamlessly integrate multiple tasks for joint training. The joint solving of captioning and retrieval tasks allows the model to delve deeper into the understanding of cross-modal interactions between images and text. The semantic consistency sought in this context is an understanding based on image-text matching, as well as a matching grounded in joint understanding of image and text. Its essence lies in the cross-modal shared space embedding from a multi-task perspective. The model’s design and training revolve precisely around this objective.

In the initial phase, we employ knowledge distillation to initialize the encoder (Section 3.1). Subsequently, we engage in image-text pair mask learning in a self-supervised environment to achieve richer semantic consistency(Section 3.2). Finally, during the text generation phase(Section 3.3), we aim to further enhance the model’s comprehension of deep-level information from the image-text context.

3.1. Contrastive Encoder Initialization

First, we designed and initialized a pair of encoders, one for images and one for text, to extract representations of image and text data (the initialized image encoder and initialized text encoder as shown in Figure 2). These unimodal encoders serve as projectors that embed each modality into a shared sematic space.

As illustrated in Figure 3, we leverage the knowledge from the pre-trained CLIP[9] model to initialize our lightweight transformer encoder. The encoder initialization is performed offline to reduce the computational requirements throughout the entire model training process. Distillation from the CLIP pre-trained encoder to the target encoder is achieved through the calculation of the L1 losss. Let the image encoder of the pre-trained CLIP model be denoted as $V\left(·\right)$, and the text encoder as $T\left(·\right)$. The distilled image encoder is denoted as $D_V\left(·\right)$, and the distilled text encoder as $D_T\left(·\right)$. The input image-text pairs are respectively denoted as $i_n$ and $t_n$. The loss functions for distilling the image and text encoders are denoted as ${\mathcal{L}}_V$ and ${\mathcal{L}}_T$, respectively. The ${\mathcal{L}}_1$ loss for the model distillation is expressed as:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill {\mathcal{L}}_V& =|D_V\left(i_n\right)-V\left(i_n\right)|\hfill \\ \hfill {\mathcal{L}}_T& =|D_T\left(t_n\right)-T\left(t_n\right)|.\hfill \end{array}\end{array}$$

(1)

As illustrated in Figure 4, the unimodal encoder consists of L layers of stacked self-attention and feed-forward modules. The projector is employed to adjust the output dimensions of each module to ensure compatibility, while the normalization layer serves to balance the scale differences among various modal data, enhancing model robustness and facilitating subsequent momentum calculations.

3.2. Residual Attention Neural Network

In accordance with [36] we adopt the practice of concatenating every S Asymmetric Co-Attention (AC) block with a Connected Attention (CA) block, thereby creating a Cross-Modal Skip-Connection (CK) module. Furthermore, the Cross-modal Res-Att is concatenated with N CK modules. As visually represented in Figure 5, we represent the Self-Attention layer, Cross Attention layer, Feed Forward Network, Layer Normalization, and Concatenation layer as SA, CA, FFN, LN, and Cat, respectively. The image embedding is denoted as $v=\{{\mathcal{V}}_{cls},\mathcal{V}\left(I\right),\mathcal{V}\left(R_1\right),\dots \mathcal{V}\left(R_i\right)\}$, while the text embedding is represented as $l=\left\{{\mathit{w}}_{\mathrm{cls}},{\mathit{w}}_1,\dots ,{\mathit{w}}_n\right\}$, consisting of word vectors corresponding to the input caption paired with I. Here, ’I’ signifies the input image, ’$R_i$’ refers to the i-th patch within it, and an additional [CLS] token is utilized to summarize the input sequence. Let ${\mathit{l}}^{S-1}$, ${\mathit{v}}^{S-1}$ and ${\mathit{l}}^S$ represent the input word vectors, visual features, and output of the S-th AC layer respectively. Then

$$l_{SA}^S=LN\left(SA\left(l^{s-1}\right)+l^{s-1}\right),$$

(2)

$$l_{CA}^S=LN\left(CA\left(l_{SA}^S,v^{N-1}\right)+l_{SA}^S\right),$$

(3)

$$l^s=LN\left(FFN\left(l_{CA}^s\right)+l_{CA}^s\right).$$

(4)

Subsequently, we feed both $l^s$ and $v^{N-1}$ into a CA block to facilitate cross-modal information interaction. The computation $[v^N;l^S]$ of the CK module’s output, denoted as follows:

$$[v^{N-1};l^{N-1}]=Cat([v^{N-1},l^S)$$

(5)

$$[v_{SA}^N;l_{SA}^N]=LN(SA\left([v^{N-1};l^{N-1}]\right)+[v^{N-1};l^{N-1}]),$$

(6)

$$[v^N;l^N]=LN(FFN\left([v_{SA}^N;l_{SA}^N]\right)+[v_{SA}^N;l_{SA}^N]).$$

(7)

The primary training objective for Cross-modal Res-Att is Masked Language Modeling (MLM). In this context, let us denote a caption as $CnP$ and the set of randomly masked positions as $M_{C}nP$. The MLM loss can be formally defined as follows:

$${\mathcal{L}}_{\mathrm{MLM}}=-\sum _{i\in M_{CnP}}logp\left(Cn{P}_i\mid Cn{P}_{\setminus M_{CnP}}\right),$$

(8)

where $Cn{P}_{\setminus M_{CnP}}$ is the masked version of the input caption, i.e., Two [MASK] are[MASK] on [MASK] of a pond. The Cross-modal Res-Att module predicts the masked tokens based on image and text context.

3.3. Captioner Training Objectives

The cross-modal image features output by the Cross-modal Res-Att module are denoted as$F^k=F_1^k,...F_u^k$.

The objective of the Captioner module is to produce output $C^k$ based on the input $F^k$.Here, $C^k$ represents the vector sequence form of the image description generated by the Captioner module, recorded as $C^k=C_1^k...C_m^k$, while the trainable parameters are denoted as $\theta$. Subsequently, the training objective of the captioner can be defined as follows:

$$\underset{\theta }{max}\sum _{k=1}^{N}log{p}_{\theta }\left({\mathrm{C}}_1^{\mathrm{k}},\dots \dots ,{\mathrm{C}}_m^{\mathrm{k}}\mid {\mathrm{F}}^{\mathrm{k}}\right).$$

(9)

The generated word, in conjunction with $F^k$, serves as the condition for predicting the subsequent word. Subsequently, the Language Modeling (LM) loss function can be expressed as:

$${\mathcal{L}}_{\mathrm{LM}}=\underset{\theta }{max}\sum _{k=1}^N\sum _{j=1}^{m}log{p}_{\theta }\left({\mathrm{C}}_{\mathrm{j}}^{\mathrm{k}}\mid {\mathrm{F}}^{\mathrm{k}},{\mathrm{C}}_{\mathrm{j}}^{\mathrm{k}},\dots {\mathrm{C}}_{\mathrm{j}-1}^{\mathrm{k}}\right).$$

(10)

The loss function for the joint training of ResAtt and Captioner is the sum of Equation 8 and Equation 10, which is:

$${\mathcal{L}}_{\mathbf{Res}\&\mathbf{Cap}}={\mathcal{L}}_{\mathrm{MLM}}+{\mathcal{L}}_{\mathrm{LM}}.$$

(11)

3.4. Offline Cross-Module Information Propagation Based on Momentum Encoder

Incorporating the Momentum Encoder Method Proposed by MOCO[12] for Offline Parameter Updates of Unimodal Encoders: Due to the high compression ratio of text during the contrastive learning phase, many details are lost in cross-modal learning between text and images. Upon entering the ResAtt module, increasing the redundancy of text helps eliminate ambiguity in cross-modal learning, thereby enhancing the semantic consistency of cross-modal representations. The information-theoretic basis for this lies in the fact that redundant information aids in ambiguity elimination. As shown in Figure 5, the ResAtt network module preserves word vectors for text, unlike the unimodal contrastive learning phase, which compresses the entire text expression into a single vector. Furthermore, the ResAtt network exhibits higher redundancy in cross-modal learning of text-image representations, including longer forward channels and the use of text as residuals to repeatedly solve for semantic consistency with images, reducing ambiguities in the process of cross-modal semantic representation.

3.4.1. Momentum Encoder

In brief, the principle of the Momentum Encoder is that the training of the encoder in unsupervised learning can be simplified as a look-up table problem. In other words, an encoded query should have high similarity to its corresponding key and low similarity to other keys. This simplifies the entire process to minimizing the contrastive loss. During the solving process, contrastive learning requires a queue containing keys for both positive and negative samples to look up for queries. To maintain the consistency of encoding for positive and negative samples in the queue, the Momentum Encoder is employed.

The encoding update rule for the Momentum Encoder is shown in Equation 12, where the momentum parameter $m\in [0,1)$ is used. The query encoding ${\theta }_q$ is updated based on gradient back propagation, while the key encoding ${\theta }_k$ is updated using momentum. Typically, m takes a value greater than 0.9, which is equivalent to taking a moving average of the encoding updates. The slow-changing Momentum Encoder reduces the difference between the encoding of positive and negative samples in the queue, thereby improving the cross-task transfer performance of the encoder optimization process based on momentum in contrastive learning.

$$\theta \leftarrow m{\theta }_k+(1-m){\theta }_q$$

(12)

3.4.2. Image-Text Contrastive Loss Funtion

Following [8], The image-text contrastive learning (ITC) formulate the loss function according to InfoNCE[37]. Let T denote a certain species class embedding and V denote its visual embedding, then we have the embedding pair$(V,T)$. We use $(V_i,T_i)$ to denote the i-th pair of positive samples and $(V_i,T_j)j\ne i$ a pair of negative samples. The ITC training objective of ReCap consists of two loss functions to make the distance of the positive pair closer than the negative one in the embedding space. Since ITC is asymmetric for each modality, it needs to be computed separately from both directions for images and text. The contrastive loss for the i-th pair in the image → text direction:

$${\ell }_i^{(V\to T)}=-log\frac{sim\left({\mathbf{V}}_i,{\mathbf{T}}_i\right)/\tau }{{\sum }_{k=1}^{n}sim\left({\mathbf{V}}_i,{\mathbf{T}}_k\right)/\tau },$$

(13)

Where $sim(·)$ is the cosine similarity, i.e., $sim(a,b)=a^{\top }b/(\parallel a\parallel \parallel b\parallel )$ ,and $\tau$ is a temperature parameter. Similarly we formulate the text → image loss as:

$${\ell }_i^{(T\to V)}=-log\frac{sim\left({\mathbf{T}}_i,{\mathbf{V}}_i\right)/\tau }{{\sum }_{k=1}^{M}sim\left({\mathbf{T}}_i,{\mathbf{V}}_k/\right)\tau },$$

(14)

Finally, the training objective is a weighted sum:

$${\mathcal{L}}_{\mathbf{ITC}}=\frac{1}{N}\sum _{i=1}^N\left(\lambda {\ell }_i^{V\to T)}+(1-\lambda ){\ell }_i^{(T\to V)}\right),$$

(15)

where $\lambda \in [0,1]$ is a hyperparameter weight, and N is batch size.

3.4.3. Offline Cross-Module Information Propagation

The cross-module joint solving of parameters constitutes the inter-module propagation of information. Deep learning models are essentially parameterized databases, with relationships among data implicitly encoded within the model’s parameters. Therefore, cross-module operations on parameters represent the propagation of information across modules.

Firstly, as illustrated in Figure 6, we feed the image-text paired dataset to the unimodal encoders, obtaining image encodings $(Ve0,Ve1,Ve2...)$ through the ITC loss. Subsequently, as depicted in Figure 7, we feed $(Ve0,Ve1,Ve2...)$ to the ResAtt module, applying Equation 12 to compute the visual momentum encoding queue $(Vm0,Vm1,Vm2...)$ and the text momentum encoding queue $(Tm0,Tm1,Tm2...)$. Then, as shown in Figure 8, we feed back the momentum encodings to update the unimodal encoders. Repeating these steps forms a closed-loop for information propagation, all of which can be conducted offline. Of course, if computational resources permit, online processing is also feasible.

3.4.4. Why Contrastive learning and Momentum Encoding

Contrastive learning is a multi-classification approach that treats each sample as a class. This classification paradigm, under the context of cross-modal embeddings, enhances the flexibility of representations in a shared space. However, it inevitably introduces ambiguity, particularly as granularity increases and feature vectors become more abundant, thereby complicating the challenge of solving semantic consistency. Since representation embeddings are determined by encoders, our unimodal encoders highly compress image and text features, deliberately for the sake of embedding and retrieval efficiency. Nonetheless, higher compression ratios inevitably lead to the loss of some information, posing a fundamental question of how to balance encoder performance and efficiency.

Image captioning is a typical task in cross-modal understanding, where input image features are reorganized into textual descriptions. Essentially, this task involves computing similarities between image and text representations to embed them into a shared space. Both the unimodal encoder and captioner used for retrieval tasks are based on cross-modal shared space embeddings, but their embedding results differ. Can we share information between the captioner’s embedding space and the unimodal encoder’s embedding space to optimize each other? Typically, in the absence of a multitask mapping network, the captioner directly takes the image encoding output by the unimodal encoder as input and generates a textual description based on it. However, the problem lies in the fact that the same image encoding is encoded and decoded in different cross-modal shared spaces, which is the fundamental reason for the semantic inconsistency in the encoding and decoding processes described earlier. How can information be exchanged between these two spaces, complementing each other, in order to assimilate the representation spaces of the two tasks into a single shared representation space with semantic consistency? This is the focal point of our deep learning network design. To achieve this, an information transfer feedback channel is required. Deep learning models are parameterized databases, and joint parameter solving enables information transfer, meaning that the model updates its understanding of the data and the embeddings of various modalities in the shared space through parameter updates. This brings us to the topic of ResAtt network. Since the captioner only generates text and does not modify the image embedding, the image embedding output by the unimodal encoder is not affected by the captioner. Alternatively, one could say that the captioner simply uses the image embedding as a prefix and does not perform any operations on it. However, cross-modal representation requires shared space embedding based on repeated cross-validation of information from both modalities. Therefore, it is necessary for the image representation encoded by the unimodal encoder to interact with more richly correlated textual descriptions across modalities to optimize its representation in the shared embedding space. Such optimization also facilitates the captioner in smoothly generating corresponding descriptions for images. The ResAtt network serves as a bridge and information conduit between the captioner and the unimodal encoder.

Why adopt momentum encoding? Contrastive learning is based on the differentiation between positive and negative samples. If embeddings of positive and negative samples come from different encoders or different training stages of the same encoder, the model learns the differences between encoders rather than the differences between positive and negative samples. To ensure a "fair" comparison between positive and negative samples and enable the model to finely learn the data’s features, it’s necessary to maintain the consistency of encodings in a longer queue of positive and negative sample encodings (in our case, a queue length of 4096). This is why we employ momentum encoders, a key technique in our multi-task deep learning network design.

In general, the concoder module serves as a mechanism for projecting image and text sequences into vectors to achieve cross-modal alignment at a macroscopic level. Within the ResAtt module, images are represented as sequences of patches, while text is represented as sequences of words, facilitating micro-level alignment. Co-training the ResAtt module with the captioner module enables the model to achieve alignment in the context of cross-modal understanding.

4. Experiments

The content of this section primarily focuses on validating the effectiveness of our approach (ReCap) through comparisons on mainstream tasks using mainstream datasets. Specifically, these tasks encompass image captioning and image-text retrieval on the COCO dataset, classification tasks on the iNaturalist2018 dataset, as well as image captioning and image-text retrieval tasks on the iNaturalist2018 dataset.

4.1. Dataset Settings

we utilized the Karpathy split[38] of the MSCOCO dataset[39], comprising 123,000 images, with each image accompanied by five sentences as annotations.The iNaturalist2018 dataset comprises 8,142 distinct species, each serving as an individual image classification category. It encompasses a total of 437,513 training images and 24,426 validation images. As this dataset initially lacked caption annotations, we conducted a comprehensive annotation effort, providing five sentences of description for each image. Furthermore, we annotated both the common name and the Latin name for each species.In Table 1, we present some examples of our annotated data.

4.2. Implementation Details

We utilized eight NVIDIA 3090 24G GPUs for the image-text encoder contrastive learning training process, with a queue length set to 4096 and a momentum parameter of 0.995. We employed the AdamW optimizer with a decay weight set to 0.02. The learning rate was warm-up set to 1e-4 for the first 1000 iterations and decayed in a cosine function manner to 1e-5 for the subsequent iterations. The total training duration for the model was approximately 127 hours.

4.3. Evaluation Metrics

The image caption model employs four widely recognized evaluation metrics, namely BLEU (Bilingual Evaluation Understudy) [40], METEOR (Metric for Evaluation of Translation with Explicit ORdering) [41], CIDEr (Consensus-based Image Description Evaluation) [42], and SPICE (Semantic Propositional Image Captioning Evaluation) [43]. Among these, BLEU4 segments sentences into four-word chunks to gauge the descriptive accuracy of the model-generated captions. METEOR, building on the foundations of BLEU, addresses the issue of excessive word matching while emphasizing word recall and precision.

CIDEr, primarily applied in the domain of image description, employs TF-IDF (Term Frequency-Inverse Document Frequency) to weigh each sentence fragment. It encodes the frequency ($E_r$) of a fragment in the reference description and the frequency ($E_c$) in the generated description. Subsequently, it computes the similarity between $E_r$ and $E_c$ to generate an evaluation score for the model.

SPICE, on the other hand, is an evaluation metric based on scene graphs and semantic concepts. It assesses the extent to which the model-generated description aligns with the entities, attributes, and relationships present in the image.

The image classification task on iNaturalist has only one label for each picture, denote as $g_i$, the result predicted by the model is denoted as $p_i$, and the error rate is

$$e_i=\underset{i}{min}d\left(g_i,p_i\right),$$

(16)

where $d(·)$ is

$$d(x,y)=\left\{\begin{array}{cc}0\hfill & \hfill \mathrm{if}x=y\\ 1\hfill & \hfill \mathrm{otherwise}\end{array}\right.,$$

(17)

the total score is

$$\mathrm{score}=\frac{1}{N}\sum _i{e}_i.$$

(18)

4.4. Evaluation on the MSCOCO Dataset

We trained models on the MSCOCO dataset to perform image captioning and image-text retrieval tasks, in order to validate the effectiveness of the proposed method. Table 2 presents the performance comparison of ReCap with state-of-the-art models in the context of image captioning. Here, B4 denotes BLEU-4, C represents CIDEr, M stands for METEOR, and S corresponds to SPICE. Further details are provided in Section 4.3. Table 3 illustrates the performance comparison of ReCap in image-text retrieval tasks against high-level models. Here, I2T denotes image-to-text retrieval, while T2I represents text-to-image retrieval. R@1, R@5, and R@10 respectively indicate recall rates for the top 1, top 5, and top 10 retrieval recommendations. The experimental results demonstrate that ReCap outperforms several state-of-the-art models, thereby validating the efficacy of the proposed method.

Based on the comparative data in Table 2, it is evident that ReCap demonstrates improved performance compared to others. Taking the scores in the B4 column as an example, ReCap’s score has increased by nearly 7 points. This improvement can be attributed to two main enhancements. Firstly, the incorporation of an open vocabulary, meaning there is no restriction on the number of categories. Secondly, the ResAtt network excels in the fusion of cross-modal features, effectively emulating the representation style of the dataset. This results in a higher overlap between the generated captions and the ground truth.

As shown in Table 3, in the retrieval task of image to text, the R@1 score exhibits an improvement of approximately 8 to 14 percentage points compared to others. In the text to image retrieval task, there is an improvement of approximately 1 to 12 percentage points compared to others. This indicates a significant effect of the proposed method in the cross-modal alignment of image and text features. The improvement in text to image retrieval performance is relatively challenging due to the high information compression in textual data and the sparse nature of image data. When calculating mutual information, the same textual representation often exhibits similarity to a larger number of image representations. For instance, different models of cars appearing in images with similar backgrounds would have high similarity. To effectively differentiate between the brand and model of cars in the image, a finer-grained cross-modal alignment is required for text to image retrieval. Therefore, adopting an open vocabulary approach during the training of the image encoder is essential, as it avoids limitations to a finite set of categories and proves crucial in the cross-modal modeling tasks involving image and text.

4.5. Evaluation on the iNaturalist Dataset

In accordance with the introduction, the motivation behind this study is to address the need for cross-modal processing of vast quantities of imagery data from natural conservations. In order to assess the cross-modal alignment of the model’s representations between images and text, we opted to employ the image classification task on the iNaturalist2018 dataset. This section’s experiments were conducted independently using the concoder module (as detailed in Section 3.1). Notably, concoder was originally designed without a classification head. To achieve classification, we employed a method that involves comparing the representations output by the image encoder with the prompt encodings generated by the text encoder.

The format of the prompts used is: ‘a photo of <category >’ where ’category’ corresponds to the category names in the dataset. In other words, for as many categories as there are in the dataset, there are corresponding prompts. In essence, our image classification approach assigns an image to the category with the highest similarity to its image representation. Specific experimental results are presented in Table 4. The experimental outcomes demonstrate that ReCap outperforms several state-of-the-art models, thereby confirming the the proposed method’s cross-modal alignment capability between image features and textual representations for species.

As shown in Table 4, ReCap demonstrates a performance improvement of approximately 1 to 7 percentage points compared to others. This indicates that our proposed method, employing an open vocabulary approach, is capable of handling image classification tasks on the iNaturalist Dataset. The experimental results not only affirm the effectiveness of our method in cross-modal representation alignment but also validate the feasibility of applying this approach to open vocabulary image classification tasks.

4.6. Qualitative Evaluation

The top-5 results for Text to Image Retrieval are illustrated in Figure 9. Both non-target images and target images contain relevant content related to grassland and the target species. From the perspective of our application, we seek relatively open-ended retrieval results. This approach allows the model to continuously improve through small-sample learning in real-world applications. If the model were confined to strict one-to-one retrieval, it would lack practical utility.

As shown in Table 5, the captions generated by the model align well with the content of the test images, and the species names are consistent with the Latin names used in the training set. This intuitively demonstrates the model’s learning capability in the domain of image-text cross-modal alignment. In the fourth prediction, the bear species (Ursus arctos horribilis) occurred 24 times in the training set, but there were no caption annotations for "cubs" in the training data prior to GPT-2 fine-tuning. This underscores the importance of pre-existing knowledge within NLP models for image captioning tasks, as it can provide additional information that is subsequently expressed in the form of generated descriptions. In the context of our approach, aligning image representations cross-modally into the pre-trained NLP decoder representation space leverages the rich knowledge of the NLP decoder for a deeper understanding of the images.

We conducted zero-shot experiments using three datasets related to natural conservations. These three datasets are as follows: birds 525 species - image classification (downloadable from https://www.kaggle.com/datasets/gpiosenka/100-bird-species), Animals Detection Images Dataset (available at https://www.kaggle.com/datasets/antoreepjana/animals-detection-images-dataset), and Wildlife Conservation Society (accessible from https://library.wcs.org/Library/Science-Data/ Datasets.aspx). The experimental procedure is as follows: Firstly, we designed sentences resembling "A photo of <species >" based on the dataset content. Subsequently, we performed text to image retrieval with these sentences and provided the retrieved images to the captioner for generating descriptive text. The experimental results are presented in Table 6. The experimental results indicate that the species names on the retrieval side, the species within the images, and the species names on the generation side are all consistent. This observation underscores that the features extracted by the image encoder and text encoder are aligned, and the semantics of the encoder and decoder are in harmony, visually demonstrating the model’s capabilities in cross-modal alignment and semantic consistency between text and images. Examining the generated captions reveals the decoder’s capacity for systematic descriptions of foreground and background elements. This is a result of the combined influence of the model’s prior knowledge and fine-tuning.

4.7. Ablation Study

The results of the ablation experiments are presented in Table 7. In the table, the term "C+C" indicates a direct connection between the conocoder and captioner, where the visual representations generated by the encoder are used as input for the captioner. "C+R+C" signifies the bridging of conocoder and captioner through the ResAtt module. Finally, "C+R+M+C" represents an extension of "C+R+C" with the addition of a momentum feedback loop.

From the experimental results in the "C+C" row, it can be observed that the I2T and T2I performance on both datasets is relatively consistent, maintaining an average level. In comparison to the performance of ReCap, there is a slight decrease in T2I, while I2T and image captioning exhibit more substantial performance degradation. This suggests that when the encoder and decoder operate independently, the model’s performance heavily relies on the knowledge inherited from pre-trained models and the training process. However, without a channel for information transfer between them, they cannot leverage distinct task perspectives from each other to enhance each other’s performance.

Looking at the experimental results in the "C+R+C" row, there is a noticeable improvement in the performance of image captioning compared to the "C+C" row. This indicates that after a finer-grained cross-modal alignment of image and text representations at the micro-level, it becomes more favorable for the captioner to generate descriptions for images. It is evident that the ResAtt module significantly contributes to the optimization of cross-modal representation alignment and the refinement of shared semantic space embedding for text and images.

ReCap and the "C+R+C" configuration only differ in the presence of a momentum feedback loop in their model structures. From the experimental results, it is evident that there are overall performance improvements in the model, particularly in the I2T and image captioning tasks. This suggests that the feedback information on the decoding side significantly aids in enhancing the performance of the encoder, resulting in substantial gains in cross-modal alignment of image and text representations.

The improvement in image captioning performance further illustrates that, after optimizing the encoder’s performance, it is possible to further enhance the decoder’s performance. From a structural perspective, this involves optimizing the pre-modules while also elevating the performance of various components along the forward path, thus establishing a beneficial feedback optimization cycle.

5. Conclusion

The image-text representation initially undergoes coarse alignment through the encoder, followed by fine-grained alignment by the decoding side consisting of ResAtt and Captioner. Subsequently, the encoder is momentum-updated based on decoding side information, forming feedback from the decoding side to the encoding side, enhancing both the quality of the encoder and caption generation. The essence of this process lies in the sharing of a semantic space, where the decoder imparts its understanding of embedding similarities and categorization to the encoder. These insights are propagated to the encoder’s network parameters through momentum-based backpropagation. Furthermore, contrastive learning on the encoding side plays a crucial role. As mentioned earlier, the classification in contrastive learning is open-ended, with as many categories as there are samples. Such a classification method has no upper limit on granularity, compelling the encoder to learn subtle distinctions among samples as much as possible. Achieving this solely from the encoding side would be information bottlenecked, and this is where feedback from the decoding side effectively bridges the information gap. Experimental results also confirm the contribution of prior knowledge in the decoder during this process. In summary, the feedback from the decoding side, the prior knowledge in the decoder, and momentum updates collectively enhance the quality of feature extraction in the encoder. All of this coalesces into a shared semantic space embedding for the encoder-decoder, where both entities possess a shared and aligned embedding space, embodying the essence of semantic consistency.

The performance of both cross-modal retrieval in image-text pairs and generative models fundamentally depends on the quality of shared space embeddings. The main contribution of our proposed method lies in the effective fusion of the advantages of both tasks in the cross-modal shared space embedding of images and text through thoughtful model design. This approach is particularly suitable for scenarios where there are strict alignment requirements between the objects in the image and the vocabulary in the text. Moreover, it demands that the model can further associate the input image representation with a more extensive and semantically rich textual description along a longer logical chain. Our proposed method is well-suited for such scenarios.