Rethinking Convolutional Semantics for Image Caption Generation Beyond Recurrent Paradigms

1. Introduction

The problem of automatically describing images with coherent natural language sentences lies at the intersection of computer vision and natural language processing. The goal of an image captioning system is to generate fluent textual descriptions that not only identify objects in the scene but also express their relationships, contexts, and implied semantics [1]. Humans excel at this task by leveraging prior knowledge and commonsense reasoning—for instance, recognizing that a crowded stadium implies an ongoing sports event, or that the presence of a double-decker bus strongly suggests a location such as London. Replicating such nuanced reasoning in computational models remains a profound challenge.

Over the past decade, deep learning has spurred significant advances in image captioning. Early encoder-decoder approaches [2,3] borrowed ideas from neural machine translation, where an image is encoded into a vector representation by a Convolutional Neural Network [4,5] and decoded into a sentence by a Recurrent Neural Network (RNN) such as LSTM [6,15]. Attention mechanisms, initially proposed for sequence-to-sequence translation [10], were quickly adapted to captioning [7,8,9,11], enabling models to dynamically focus on salient visual regions while generating words. These developments established RNNs as the de facto decoding architecture, demonstrating strong capacity to capture temporal dependencies in language.

Despite the dominance of recurrent models, researchers began to explore CNNs as alternatives for sequence modeling [20]. Aneja et al. [21] extended this paradigm to caption generation by using CNNs as decoders. The convolutional approach promises advantages such as parallel computation and reduced sequential dependencies, but it also raises questions about its ability to model long-range linguistic structures and contextual coherence. Unlike RNNs, which inherently model sequence order, CNNs depend on receptive field expansion to capture dependencies—a design that may or may not suffice for natural language descriptions of complex scenes.

While RNN-based decoders have been studied extensively with respect to network depth, regularization, attention design, and sentence length, CNN-based decoders remain relatively underexplored. Several critical aspects—including the effect of deeper convolutional stacks, the interplay between attention mechanisms and convolutions, and the impact of training caption length distribution—have not yet been systematically investigated. Moreover, performance trade-offs across evaluation metrics such as BLEU, METEOR, ROUGE, SPICE, and CIDEr are poorly understood in this context. These gaps hinder a comprehensive understanding of when convolutional decoders may offer advantages and where they fundamentally lag behind recurrent methods.

The rise of Transformer-based models has further reshaped the landscape of sequence modeling, demonstrating the power of self-attention in capturing long-range dependencies. However, CNNs retain practical value due to their efficiency and inductive biases toward locality and compositionality. In multimodal learning, hybrid architectures that blend CNNs, RNNs, and Transformers are increasingly common. Thus, studying CNN decoders for captioning is not only a comparative exercise but also a step toward designing hybrid models that exploit the complementary strengths of different architectures.

In this paper, we make the following contributions:

• We introduce VISCON, a convolutional decoder framework, to systematically examine CNN-based captioning across multiple dimensions: network depth, data augmentation, attention integration, and sentence length effects.
• We conduct extensive experiments on Flickr8k [22] and Flickr30k [23], providing a controlled comparison with recurrent baselines.
• We reveal unique limitations of CNN-based decoders, including their difficulty in handling longer sentences and their weaker response to attention mechanisms compared with RNN decoders [28,29].
• Our study offers the first detailed evaluation of CNN captioning models with respect to metric-specific performance trade-offs, demonstrating that while VISCON can approximate RNN baselines in some metrics, it consistently struggles on CIDEr, indicating limited ability to capture high-level semantic richness.

Overall, this work seeks to clarify the underexplored dynamics of convolutional decoders in image captioning and to provide actionable insights for building next-generation systems that combine efficiency with semantic depth.

2. Related Work

Research on Image Caption Generation has evolved over multiple paradigms, each reflecting different assumptions about how visual and linguistic modalities can be connected. Broadly, the methods can be grouped into three classical families—Retrieval-based, Template-based, and Deep Learning-based approaches—with more recent explorations combining ideas across these boundaries. In what follows, we provide an extensive discussion of these categories, highlighting representative works, their advantages and limitations, and the motivations that inspired the development of our proposed framework, VISCON.

2.1. Retrieval-Based Approaches

Early attempts treated caption generation not as a process of novel sentence synthesis, but as a problem of retrieving suitable descriptions from a pre-defined set. These Retrieval-based methods assume that if one can find an image in a large database that is visually similar to the query, its caption can be transferred to the target.

Farhadi et al. [12] constructed a meaning space defined by triplets of (object, action, scene). Each candidate sentence in a manually curated corpus is then embedded into this meaning space, and the caption closest to the target image representation is chosen. Similarly, Ordonez et al. [13] designed a system where a massive corpus of annotated images is indexed. For a new image, semantic features are extracted, and the most similar annotated image is identified; its caption is directly used for the query. Mason et al. [14] proposed a refinement by employing probabilistic word density scores: given a query image, a set of visually similar images are retrieved, and their captions are aggregated to estimate word probabilities conditioned on the target. These probabilities are then used to rank candidate captions.

The strength of retrieval-based methods lies in their grammatical correctness, since all sentences originate from human annotations. However, they suffer from serious drawbacks. The semantic diversity of natural images is extremely high, and a retrieval pool cannot cover all possible object combinations and contextual relations. Moreover, scalability is problematic: as novel scenes appear, a large new set of annotated examples must be added to maintain coverage. This limitation motivates the transition to generative paradigms.

2.2. Template-Based Approaches

To overcome the rigidity of retrieval, researchers developed Template-based methods, where sentence generation is guided by a set of manually designed grammar rules or templates. These methods attempt to produce new descriptions by combining semantic elements with pre-defined syntactic structures.

Kulkarni et al. [17] extracted semantic information from images using Conditional Random Fields, representing objects and their relations in graph form. The graph structure, combined with statistical co-occurrence patterns, guided the assembly of sentences. Li et al. [16] pursued a similar direction, encoding visual contents as triplets like [(adjective1, object1), preposition, (adjective2, object2)], then calculating n-gram frequencies to determine plausible phrase sequences. Through dynamic programming, the most likely phrase fusion was identified to construct captions.

While template methods achieved more diverse outputs than retrieval-based approaches, they still required significant manual effort. Constructing templates that covered the vast variety of linguistic and semantic possibilities was infeasible. These methods often generated rigid sentences and lacked adaptability to complex, unseen scenarios. Scalability again posed a severe bottleneck.

2.3. Deep Learning-Based Approaches

The breakthroughs in deep learning, particularly in neural machine translation and visual recognition, radically reshaped image captioning. The introduction of the Encoder-Decoder paradigm [2,3] allowed models to be trained end-to-end, mapping raw image features directly into textual descriptions.

In these methods, image features are extracted using Convolutional Neural Networks pre-trained on large-scale datasets such as ImageNet [4,5]. The encoder produces a fixed-length or region-based feature representation, which is then decoded into a caption sequence. Mao et al. [8] pioneered the use of RNNs [6] for decoding, where visual and linguistic states are merged recurrently to predict words. Karpathy et al. [9] employed Bidirectional RNNs to capture context, while Vinyals et al. [7] introduced the "Show and Tell" framework, initializing the hidden and cell states of LSTMs [15] with visual embeddings. Donahue et al. [26] further merged visual features at each time step, enabling tighter integration of modalities.

The adoption of attention mechanisms, proposed in [10] for translation, was a major milestone. Xu et al. [11] adapted attention for image captioning, allowing the decoder to dynamically focus on salient visual regions during word generation. This significantly improved semantic alignment and fine-grained description quality. More recently, Gehring et al. [20] demonstrated that Convolutional Networks could rival RNNs in translation, inspiring Aneja et al. [21] to introduce CNN-based decoders for captioning.

2.4. Hybrid and Emerging Paradigms

Beyond the three classical categories, hybrid models and emerging architectures now dominate research. The Transformer architecture, with its self-attention mechanism, has surpassed RNNs in many sequence tasks, including captioning. Vision-language pretraining with large multimodal transformers has further advanced the field, showing that cross-modal attention can capture long-range dependencies more effectively than traditional CNN or RNN decoders.

Nevertheless, convolutional approaches remain of interest. Aneja et al. [21] presented one of the earliest CNN-decoder captioning models, employing a three-layer convolutional network with 512 hidden units. Notably, they fine-tuned the encoder as well, although most subsequent works froze the pre-trained CNN encoder for fairer comparison. Their experiments were performed on the MSCOCO dataset [24], whereas our work evaluates on Flickr8k [22] and Flickr30k [23], enabling controlled study on smaller but challenging datasets.

Our study, with the newly proposed VISCON, extends these directions by systematically evaluating the design choices in CNN-based decoding. Unlike prior works that primarily benchmark performance, we dissect the influence of architectural depth, attention mechanisms, data augmentation strategies, and caption length distributions. Through this analysis, we aim to establish a clearer understanding of the trade-offs involved in convolutional decoders and to highlight their role within the broader ecosystem of image captioning methodologies. This places VISCON as a critical step toward reconciling efficiency, interpretability, and semantic expressivity in modern captioning systems.

Figure 1. Overall workflow of the proposed CNN-based Image Caption Generation framework. The system first extracts image features using a CNN encoder and embeds the input words through a word embedding layer. Both representations are concatenated and passed through stacked convolutional decoder layers with masking and gated linear units (GLU). In the CNN+CNN variant, decoding relies solely on convolutional layers, while in the CNN+CNN+Attention variant, an additional attention module integrates image features to guide word prediction. The final softmax layer generates caption tokens.

Figure 1. Overall workflow of the proposed CNN-based Image Caption Generation framework. The system first extracts image features using a CNN encoder and embeds the input words through a word embedding layer. Both representations are concatenated and passed through stacked convolutional decoder layers with masking and gated linear units (GLU). In the CNN+CNN variant, decoding relies solely on convolutional layers, while in the CNN+CNN+Attention variant, an additional attention module integrates image features to guide word prediction. The final softmax layer generates caption tokens.

Algorithm 1:VISCON Training (CNN+CNN and optional +Att)

Input:   Dataset $\mathcal{D}=\left\{\right(I,S\left)\right\}$; frozen encoder $f_{enc}$; vocab $\mathcal{X}$ with <START>, <END>, <PAD>; decoder depth L, kernel size k, dims $E,d$; smoothing $ϵ$; weight decay $\lambda$; dropout p; learning rate $\eta$; epochs $E_{\mathrm{tr}}$; scheduled sampling prob ${\tau }_{\mathrm{epoch}}$; RL flag useRL. Output:  Trained parameters $\mathrm{\Omega }=\{W^{1:L},b^{1:L},W_o,b_o,\mathrm{Emb}\}$ Initialize$\mathrm{Emb}\in {\mathbb{R}}^{\left|\mathcal{X}\right|\times E}$, masked conv layers ${\{W^l,b^l\}}_{l=1}^L$, output $(W_o,b_o)$; optimizer Adam$\left(\eta \right)$. Freeze $f_{enc}$ for fair comparison [7,8,9,11].

3. Proposed Method: VISCON Framework

In this section, we present our proposed framework, VISCON (Visual-Semantic Convolutional Network), for the task of automatic image caption generation. Unlike conventional recurrent decoders based on LSTMs [7,8,9,11,26], VISCON leverages convolutional sequence modeling to process captions in parallel without maintaining explicit hidden or cell states. Instead of sequential recurrence, VISCON captures contextual dependencies through stacked one-dimensional masked convolutions. This design provides both computational efficiency and structural inductive bias toward local semantic patterns, while still allowing us to investigate how deeper architectures and attention mechanisms contribute to long-range modeling.

The overall pipeline follows an encoder-decoder paradigm: a convolutional encoder extracts visual features, and a convolutional decoder generates captions, optionally enhanced with attention. We term the base configuration CNN+CNN (encoder-decoder with stacked convolutions) and the attention-augmented version CNN+CNN+Att.

3.1. Problem Formulation

Given an image I, the task is to produce a caption $S=\{x_1,x_2,\dots ,x_n\}$ where each $x_i$ is a word token. The probability distribution of the entire sequence can be expressed as:

$$p\left(S\right|I;\mathrm{\Omega })=\prod _{i=1}^{L_I}p\left(x_i\right|x_{1:i-1},I;\mathrm{\Omega }),$$

(1)

where $\mathrm{\Omega }$ represents all learnable parameters and $L_I$ is the length of the caption. The training objective is to maximize the log-likelihood across all samples:

$${\mathcal{L}}_{MLE}\left(\mathrm{\Omega }\right)=\sum _I\sum _{i=1}^{L_I}logp\left(x_i\right|x_{1:i-1},I;\mathrm{\Omega }).$$

(2)

The vocabulary $\mathcal{X}$ is restricted to words occurring at least 5 times in the dataset. For Flickr8k [22], we have $\left|\mathcal{X}\right|=2362$, and for Flickr30k [23], $\left|\mathcal{X}\right|=7002$. Special tokens such as <START>, <END>, and <PAD> are added to manage sequence generation.

3.2. Encoder Network

The encoder is a pre-trained convolutional backbone trained on ImageNet [5] for object recognition [4]. We discard its classification head and use either the penultimate fully connected layer or the last convolutional feature maps as visual embeddings:

$$v=f_{enc}\left(I\right)\in {\mathbb{R}}^d,$$

(3)

where d denotes feature dimensionality. Unlike Aneja et al. [21], we do not fine-tune the encoder, ensuring fair comparison with RNN-based baselines [7,8,9,11].

3.3. Word Embedding and Input Representation

Each token $x_i$ is first mapped to a one-hot vector $x_i^{one-hot}\in {\mathbb{R}}^{\left|\mathcal{X}\right|}$, which is transformed into a dense embedding:

$$e_i=f_{emb}\left(x_i^{one-hot}\right)\in {\mathbb{R}}^E,$$

(4)

where E is the embedding dimension. The combined representation of image and text at each time step is given by concatenating $e_i$ with v, forming the decoder input sequence:

$$h_i^0=[e_i;v].$$

(5)

3.4. Convolutional Decoder Architecture

The decoder applies stacked masked convolutions with receptive field size k over the sequence $\left\{h_i^0\right\}$:

$$h_i^l=\sigma (W^l*h_{i-k:i}^{l-1}+b^l),$$

(6)

where * denotes convolution, $W^l$ and $b^l$ are learnable kernel weights and biases, and $\sigma$ is a non-linear activation. We adopt Gated Linear Units (GLUs) to regulate feature flow:

$$\mathrm{GLU}(a,b)=a\otimes \sigma \left(b\right),$$

(7)

where ⊗ is element-wise multiplication. This design enables selective retention of salient information. The receptive field expands with depth, allowing modeling of longer contexts. We explore from one up to four convolutional layers, in contrast to Aneja et al.’s fixed three-layer design.

Algorithm 2: VISCON Inference (Greedy or Beam Search)

Input:  Image I; encoder $f_{enc}$; trained params $\mathrm{\Omega }$; max length $T_{max}$; beam size B. Output: Caption $\widehat{S}$

3.5. Attention-Augmented VISCON

For the CNN+CNN+Att variant, we incorporate attention [10,11]. At each decoding step i, the attention weights ${\alpha }_{ij}$ over encoder features $v_j$ are computed as:

$${\alpha }_{ij}={\displaystyle \frac{exp\left(e(h_i^l,v_j)\right)}{{\sum }_{k}exp\left(e(h_i^l,v_k)\right)}},$$

(8)

where $e(·)$ is a compatibility function (dot-product or additive). The attended feature is:

$${\tilde{v}}_i=\sum _j{\alpha }_{ij}{v}_j,$$

(9)

and the context-enhanced decoder state is:

$${\widehat{h}}_i^l=[h_i^l;{\tilde{v}}_i].$$

(10)

This allows VISCON to dynamically emphasize different spatial regions of the image for different words, aligning visual focus with linguistic content.

3.6. Output Prediction Layer

The final convolutional layer produces hidden states ${\widehat{h}}_i^L$ which are projected onto the vocabulary:

$$y_i=\mathrm{Softmax}(W_o{\widehat{h}}_i^L+b_o),$$

(11)

where $W_o\in {\mathbb{R}}^{\left|\mathcal{X}\right|\times d}$ and $b_o\in {\mathbb{R}}^{\left|\mathcal{X}\right|}$. The probability vector $y_i$ provides the likelihood of each word in the vocabulary at position i.

3.7. Training Objective and Regularization

The main objective is maximum likelihood estimation (MLE). In addition, we incorporate regularization and auxiliary objectives:

• Label Smoothing: We apply label smoothing with parameter $ϵ$ to prevent overconfidence:

  $${\mathcal{L}}_{LS}=-(1-ϵ)log{y}_{i,x_i}-{\displaystyle \frac{ϵ}{\left|\mathcal{X}\right|}}.$$

  (12)
• Dropout and Weight Decay: Dropout is applied after each convolutional block, and $L_2$ regularization ${\lambda \parallel \mathrm{\Omega }\parallel }^2$ is added.
• Reinforcement Learning Fine-tuning: Inspired by SCST, the loss can be further optimized with respect to evaluation metrics such as CIDEr:

  $${\mathcal{L}}_{RL}=-(r\left(\widehat{S}\right)-b)\sum _{i}logp\left(x_i\right|x_{1:i-1},I),$$

  (13)

  where $r\left(\widehat{S}\right)$ is the reward of sampled caption and b is a baseline.

3.8. Complexity Analysis

Compared to LSTM decoders, VISCON provides significant parallelization benefits. If n is the sequence length and k the kernel size, convolutional decoding requires $\mathcal{O}\left(nkd\right)$ operations, while RNN decoding is $\mathcal{O}\left(n{d}^2\right)$ due to recurrent multiplications. This efficiency enables faster training and inference, albeit with potential limitations in long-range dependency modeling.

3.9. Algorithm Illustration

The proposed VISCON framework thus integrates: (1) convolutional parallel decoding with masked receptive fields, (2) GLU-based gating for effective representation control, (3) attention-enhanced variant for visual-linguistic alignment, and (4) multi-objective training combining MLE, label smoothing, and reinforcement-based optimization.

This design allows us to systematically study the trade-offs between convolutional and recurrent paradigms for image caption generation. Algorithm 1 and Algorithm 2 show the overall training and inference process, respectively.

Table 1. Comparison against representative literature on Flickr8k and Flickr30k. We denote our variants as VISCON-Base and VISCON-Att.

Method	BLEU-1	BLEU-2	BLEU-3	BLEU-4	METEOR	CIDEr	ROUGE-L
Flickr8k Dataset
Karpathy et al. [9]	0.582	0.387	0.249	0.163	_	_	_
Vinyals et al. [7]	0.635	0.414	0.275	_	_	_	_
Xu et al. [11]	0.671	0.451	0.302	0.198	0.1893	_	_
VISCON-Base (CNN+CNN)	0.6279	0.4439	0.3022	0.2038	0.1924	0.4748	0.4544
VISCON-Att (CNN+CNN+Att)	0.6312	0.4471	0.3050	0.2074	0.1951	0.4879	0.4563
Flickr30k Dataset
Mao et al. [8]	0.600	0.410	0.280	0.190	_	_	_
Donahue et al. [26]	0.590	0.392	0.253	0.165	_	_	_
Karpathy et al. [9]	0.575	0.372	0.242	0.159	_	_	_
Vinyals et al. [7]	0.666	0.426	0.279	0.185	_	_	_
Xu et al. [11]	0.669	0.436	0.290	0.192	0.1849	_	_
VISCON-Base (CNN+CNN)	0.6432	0.4495	0.3108	0.2123	0.1774	0.3831	0.4344
VISCON-Att (CNN+CNN+Att)	0.6411	0.4452	0.3071	0.2102	0.1781	0.3868	0.4351

Table 1. Comparison against representative literature on Flickr8k and Flickr30k. We denote our variants as VISCON-Base and VISCON-Att.

Method	BLEU-1	BLEU-2	BLEU-3	BLEU-4	METEOR	CIDEr	ROUGE-L
Flickr8k Dataset
Karpathy et al. [9]	0.582	0.387	0.249	0.163	_	_	_
Vinyals et al. [7]	0.635	0.414	0.275	_	_	_	_
Xu et al. [11]	0.671	0.451	0.302	0.198	0.1893	_	_
VISCON-Base (CNN+CNN)	0.6279	0.4439	0.3022	0.2038	0.1924	0.4748	0.4544
VISCON-Att (CNN+CNN+Att)	0.6312	0.4471	0.3050	0.2074	0.1951	0.4879	0.4563
Flickr30k Dataset
Mao et al. [8]	0.600	0.410	0.280	0.190	_	_	_
Donahue et al. [26]	0.590	0.392	0.253	0.165	_	_	_
Karpathy et al. [9]	0.575	0.372	0.242	0.159	_	_	_
Vinyals et al. [7]	0.666	0.426	0.279	0.185	_	_	_
Xu et al. [11]	0.669	0.436	0.290	0.192	0.1849	_	_
VISCON-Base (CNN+CNN)	0.6432	0.4495	0.3108	0.2123	0.1774	0.3831	0.4344
VISCON-Att (CNN+CNN+Att)	0.6411	0.4452	0.3071	0.2102	0.1781	0.3868	0.4351

4. Experiments

In this section, we present a comprehensive empirical study of our convolutional captioning framework, hereafter denoted as VISCON. To maintain continuity with prior literature while emphasizing our unified naming, we write VISCON-Base to indicate the CNN+CNN configuration (convolutional encoder and convolutional decoder without attention) and VISCON-Att to denote the CNN+CNN+Att variant (convolutional encoder and convolutional decoder augmented with visual attention). Unless otherwise stated, all models use a ResNet-152 encoder [18] pre-trained on ImageNet [4], from which we extract the last convolutional feature maps following the common practice that deeper residual encoders typically yield stronger captioners [19]. For decoding we adopt a beam width of 3. We ran all experiments on a single NVIDIA Quadro RTX 4000 (7GB), using batch size 10 for VISCON models and, for the comparative CNN+LSTM experiments in §4.5, batch size 32 and beam width 3. Unless explicitly varied (e.g., in §4.3), we report the strongest setting discovered in preliminary sweeps: VISCON-Base with one convolutional layer and VISCON-Att with two convolutional layers.

4.1. Benchmarks, Metrics, and Evaluation Protocol

We evaluate on Flickr8k [22] and Flickr30k [23]. Following standard practice, we report BLEU-n ($n\in \{1,2,3,4\}$), METEOR, ROUGE-L, and CIDEr [25]. BLEU assesses n-gram precision with brevity penalty; METEOR emphasizes semantic matches via stemming and synonyms; ROUGE-L captures longest common subsequences; CIDEr measures consensus against multiple references and correlates well with human judgments in open-domain captioning.

Training Details.

VISCON uses masked 1-D convolutions with GLU activations, dropout $p\in [0.1,0.3]$, label smoothing $ϵ=0.1$, and weight decay $\lambda ={10}^{-4}$. The encoder remains frozen for fair comparison with widely cited baselines [7,8,9,11]. We employ teacher forcing with scheduled sampling and optionally apply reinforcement fine-tuning toward CIDEr as described in the methodology.

4.2. Compared Methods

We compare against representative neural captioners spanning encoder–decoder and attention paradigms. Vinyals et al. [7] initialize an LSTM with visual features; Mao et al. [8] merge recurrent text states with mapped image features; Karpathy et al. [9] adopt bidirectional RNNs; Donahue et al. [26] inject visual features at each decoding step; Xu et al. [11] integrate spatial attention over convolutional regions. In §4.7 we summarize results on both datasets using BLEU, METEOR, CIDEr, and ROUGE-L [25]. We emphasize single-model results (no encoder fine-tuning or ensembling) to isolate decoder effects.

4.3. Depth of the Convolutional Decoder

Classical recurrent captioners often benefit from deeper decoders (e.g., stacked LSTMs [28,29]). For convolutional sequence modeling, Aneja et al. [21] explored a 3-layer CNN decoder with residual connections. We extend this analysis by varying depth from 1 to 4 for both VISCON-Base and VISCON-Att. We find the best overall trade-off at 1 layer for VISCON-Base and 2 layers for VISCON-Att. Adding more layers marginally expands the receptive field but tends to over-smooth local semantics and slightly degrades CIDEr (see Table 2 and Table 3).

4.4. Image Transforms for Data Augmentation

We probe whether common geometric transforms—random horizontal/vertical flips, rotations by $\{{90}^{\circ },{180}^{\circ },{270}^{\circ }\}$, and perspective warps—improve VISCON’s robustness. Transforms are applied stochastically per epoch to diversify training views without altering image-level semantics [27]. Contrary to the gains sometimes observed for LSTM decoders [28,29], heavy geometry in our convolutional decoders often harms CIDEr and higher-order BLEU, likely because aggressive viewpoint changes can weaken alignment between local visual patterns and word n-grams. A light horizontal flip occasionally offers small benefits (Table 5).

Table 4. Qualitative captions for Flickr8k across depths for VISCON-Base and VISCON-Att.

Model(Layers)
VISCON-Base(1)	a small white dog is jumping into a pool	a man riding a bike on a dirt bike	a football player in red and white uniform wearing a red and white uniform	a little boy in a red shirt is sitting on a swing
VISCON-Base(2)	a white dog is jumping into a pool	a person riding a bike on a dirt bike	a football player in a red uniform and red uniform	a little girl in a red shirt is sitting on a swing
VISCON-Base(3)	a small white dog is playing in a pool	a person riding a bike on a dirt bike	a football player in a red uniform and a red uniform	a little boy in a red shirt is jumping on a swing
VISCON-Base(4)	a white and white dog is playing in a pool	a person riding a bike on a dirt bike	a football player in a red and white uniform	a little girl in a red shirt is sitting on a swing
VISCON-Att(1)	a white dog is swimming in a pool	a person riding a bike on a dirt bike	a football player in a red uniform and a football	a little boy in a red shirt is jumping over a swing
VISCON-Att(2)	a white dog is jumping over a blue pool	a man on a motorcycle rides a dirt bike	a football player in a red uniform	a little boy in a red shirt is sitting on a swing
VISCON-Att(3)	a small white dog is jumping into a pool	a person riding a bike on a dirt bike	a football player in a red uniform and a red uniform	a little boy in a red shirt is jumping over a swing
VISCON-Att(4)	a white dog is jumping over a blue pool	a man riding a bike on a dirt bike	a football player in a red uniform is holding a football	a little girl in a pink shirt is sitting on a swing

Table 4. Qualitative captions for Flickr8k across depths for VISCON-Base and VISCON-Att.

Model(Layers)
VISCON-Base(1)	a small white dog is jumping into a pool	a man riding a bike on a dirt bike	a football player in red and white uniform wearing a red and white uniform	a little boy in a red shirt is sitting on a swing
VISCON-Base(2)	a white dog is jumping into a pool	a person riding a bike on a dirt bike	a football player in a red uniform and red uniform	a little girl in a red shirt is sitting on a swing
VISCON-Base(3)	a small white dog is playing in a pool	a person riding a bike on a dirt bike	a football player in a red uniform and a red uniform	a little boy in a red shirt is jumping on a swing
VISCON-Base(4)	a white and white dog is playing in a pool	a person riding a bike on a dirt bike	a football player in a red and white uniform	a little girl in a red shirt is sitting on a swing
VISCON-Att(1)	a white dog is swimming in a pool	a person riding a bike on a dirt bike	a football player in a red uniform and a football	a little boy in a red shirt is jumping over a swing
VISCON-Att(2)	a white dog is jumping over a blue pool	a man on a motorcycle rides a dirt bike	a football player in a red uniform	a little boy in a red shirt is sitting on a swing
VISCON-Att(3)	a small white dog is jumping into a pool	a person riding a bike on a dirt bike	a football player in a red uniform and a red uniform	a little boy in a red shirt is jumping over a swing
VISCON-Att(4)	a white dog is jumping over a blue pool	a man riding a bike on a dirt bike	a football player in a red uniform is holding a football	a little girl in a pink shirt is sitting on a swing

4.5. Effect of Maximum Sentence Length

We next vary the maximum allowed caption length in training (i.e., the masked convolutional unrolling horizon). Following [21], very long sentences are down-sampled; we explicitly test thresholds from 10 to 40 tokens. We additionally implement a CNN+LSTM baseline (ResNet encoder + LSTM decoder, akin to [7]) to contrast recurrent vs. convolutional decoders under identical training data. Results (Table 6) show VISCON-Base is most stable for short to medium lengths (15–25), while performance drops for $\ge 30$ tokens, consistent with CNNs’ challenges in capturing long-range dependencies. In contrast, the LSTM benefits from longer targets, with consistent CIDEr gains as the maximum length increases.

4.6. Results and Discussion

We summarize key findings before presenting detailed tables: (i) VISCON-Base is competitive with strong encoder–decoder RNNs on BLEU/METEOR/ROUGE-L, but VISCON-Att yields only modest further gains, smaller than those typically reported for LSTMs with attention. (ii) For VISCON, shallow-to-moderate depth suffices (1–2 layers); deeper stacks offer no reliable advantages. (iii) Geometric augmentation provides limited or negative returns. (iv) CIDEr reveals a consistent gap, with recurrent decoders outperforming convolutional ones by ≈8–16 points depending on the sentence-length regime, echoing the importance of long-range modeling.

4.7. Comparison with Prior Work

We report Flickr8k and Flickr30k results in Table 1. To facilitate apples-to-apples comparison, we list single-model settings (no ensemble, frozen encoder). For some baselines, small numerical discrepancies can arise across codebases and splits; we follow the commonly cited settings and slightly update scores where re-evaluation under our setup led to minor deviations.

4.8. Decoder Depth Study

Table 2 and Table 3 report the depth sweep for Flickr8k and Flickr30k, respectively. On Flickr8k, VISCON-Base peaks at 1 layer, whereas VISCON-Att reaches its best at 2 layers and degrades beyond that. On Flickr30k, shallow stacks remain competitive; deeper stacks slightly depress CIDEr despite small fluctuations in BLEU-4, suggesting longer convolutional chains may over-regularize phrase diversity.

4.9. Qualitative Caption Comparisons on Flickr8k

4.10. Image Transform Ablations

We now quantify the impact of each transform on Flickr8k. As shown in Table 5, random horizontal flips are slightly beneficial, while vertical flips and large rotations reduce phrase consistency and penalize CIDEr. Perspective warps have mixed effects, hinting that synthetic viewpoint changes may disrupt local phrase grounding in a convolutional decoder.

Table 5. Flickr8k: impact of individual data augmentation transforms on VISCON-Base and VISCON-Att.

Image Transform	BLEU-1	BLEU-2	BLEU-3	BLEU-4	METEOR	CIDEr	ROUGE-L
VISCON-Base
No transform	0.6279	0.4439	0.3022	0.2038	0.1924	0.4748	0.4544
Random Horizontal	0.6290	0.4491	0.3068	0.2061	0.1956	0.4791	0.4566
Random Vertical	0.6218	0.4338	0.2922	0.1943	0.1889	0.4415	0.4473
Random Flip	0.6284	0.4464	0.3048	0.2059	0.1923	0.4703	0.4534
Random Rotate	0.6112	0.4226	0.2829	0.1883	0.1843	0.4202	0.4376
Random Perspective	0.6257	0.4431	0.3007	0.2008	0.1912	0.4574	0.4511
VISCON-Att
No transform	0.6312	0.4471	0.3050	0.2074	0.1951	0.4879	0.4563
Random Horizontal	0.6331	0.4516	0.3098	0.2103	0.1933	0.4812	0.4575
Random Vertical	0.6165	0.4306	0.2915	0.1941	0.1865	0.4303	0.4445
Random Flip	0.6237	0.4388	0.2976	0.1986	0.1892	0.4528	0.4502
Random Rotate	0.6079	0.4194	0.2799	0.1870	0.1826	0.4104	0.4339
Random Perspective	0.6260	0.4417	0.3026	0.2060	0.1910	0.4613	0.4487

Table 5. Flickr8k: impact of individual data augmentation transforms on VISCON-Base and VISCON-Att.

Image Transform	BLEU-1	BLEU-2	BLEU-3	BLEU-4	METEOR	CIDEr	ROUGE-L
VISCON-Base
No transform	0.6279	0.4439	0.3022	0.2038	0.1924	0.4748	0.4544
Random Horizontal	0.6290	0.4491	0.3068	0.2061	0.1956	0.4791	0.4566
Random Vertical	0.6218	0.4338	0.2922	0.1943	0.1889	0.4415	0.4473
Random Flip	0.6284	0.4464	0.3048	0.2059	0.1923	0.4703	0.4534
Random Rotate	0.6112	0.4226	0.2829	0.1883	0.1843	0.4202	0.4376
Random Perspective	0.6257	0.4431	0.3007	0.2008	0.1912	0.4574	0.4511
VISCON-Att
No transform	0.6312	0.4471	0.3050	0.2074	0.1951	0.4879	0.4563
Random Horizontal	0.6331	0.4516	0.3098	0.2103	0.1933	0.4812	0.4575
Random Vertical	0.6165	0.4306	0.2915	0.1941	0.1865	0.4303	0.4445
Random Flip	0.6237	0.4388	0.2976	0.1986	0.1892	0.4528	0.4502
Random Rotate	0.6079	0.4194	0.2799	0.1870	0.1826	0.4104	0.4339
Random Perspective	0.6260	0.4417	0.3026	0.2060	0.1910	0.4613	0.4487

4.11. Sentence-Length Ablation and CNN vs. LSTM

Table 6 contrasts VISCON-Base and CNN+LSTM under varying maximum sentence lengths. For VISCON, shorter caps (15–25) yield balanced BLEU/METEOR with moderate CIDEr; very long caps ($\ge 30$) trigger repetition and n-gram drift, depressing BLEU-4. The RNN decoder accumulates benefits from longer contexts, consistently improving CIDEr and stabilizing BLEU-4 around ∼0.21.

Table 6. Flickr8k: sensitivity to maximum sentence length for VISCON-Base and CNN+LSTM.

Max. Sent. Length	BLEU-1	BLEU-2	BLEU-3	BLEU-4	METEOR	CIDEr	ROUGE-L
VISCON-Base
10	0.6275	0.4395	0.2985	0.2047	0.1682	0.3714	0.4291
15	0.6246	0.4350	0.2930	0.1986	0.1915	0.4502	0.4492
20	0.6111	0.4310	0.2928	0.1965	0.1917	0.4601	0.4506
25	0.6102	0.4343	0.3005	0.2062	0.1948	0.4763	0.4546
30	0.5651	0.3996	0.2713	0.1820	0.1894	0.4559	0.4448
35	0.5576	0.3928	0.2677	0.1787	0.1894	0.4569	0.4439
40	0.5327	0.3748	0.2545	0.1698	0.1891	0.4553	0.4411
CNN+LSTM
10	0.4541	0.2953	0.1916	0.1253	0.1708	0.6253	0.3943
15	0.5898	0.4128	0.2825	0.1923	0.1966	0.5549	0.4490
20	0.6126	0.4317	0.3000	0.2065	0.2023	0.5326	0.4599
25	0.6252	0.4402	0.3042	0.2091	0.2000	0.5419	0.4610
30	0.6236	0.4394	0.3044	0.2076	0.2014	0.5205	0.4591
35	0.6230	0.4401	0.3066	0.2121	0.1978	0.5281	0.4594
40	0.6190	0.4383	0.3062	0.2122	0.2030	0.5497	0.4628

Table 6. Flickr8k: sensitivity to maximum sentence length for VISCON-Base and CNN+LSTM.

Max. Sent. Length	BLEU-1	BLEU-2	BLEU-3	BLEU-4	METEOR	CIDEr	ROUGE-L
VISCON-Base
10	0.6275	0.4395	0.2985	0.2047	0.1682	0.3714	0.4291
15	0.6246	0.4350	0.2930	0.1986	0.1915	0.4502	0.4492
20	0.6111	0.4310	0.2928	0.1965	0.1917	0.4601	0.4506
25	0.6102	0.4343	0.3005	0.2062	0.1948	0.4763	0.4546
30	0.5651	0.3996	0.2713	0.1820	0.1894	0.4559	0.4448
35	0.5576	0.3928	0.2677	0.1787	0.1894	0.4569	0.4439
40	0.5327	0.3748	0.2545	0.1698	0.1891	0.4553	0.4411
CNN+LSTM
10	0.4541	0.2953	0.1916	0.1253	0.1708	0.6253	0.3943
15	0.5898	0.4128	0.2825	0.1923	0.1966	0.5549	0.4490
20	0.6126	0.4317	0.3000	0.2065	0.2023	0.5326	0.4599
25	0.6252	0.4402	0.3042	0.2091	0.2000	0.5419	0.4610
30	0.6236	0.4394	0.3044	0.2076	0.2014	0.5205	0.4591
35	0.6230	0.4401	0.3066	0.2121	0.1978	0.5281	0.4594
40	0.6190	0.4383	0.3062	0.2122	0.2030	0.5497	0.4628

4.12. Additional Analysis: Decoding Width and Statistical Significance

To ensure decoding is not a confound, we further probe beam widths $B\in \{1,3,5\}$ on Flickr8k using the best VISCON settings. We also estimate $95\%$ confidence intervals via bootstrap resampling of test captions (1k samples).

Table 7. Flickr8k: beam-width study and bootstrap confidence for CIDEr. Wider beams yield marginal improvements with diminishing returns.

Model / Beam	BLEU-4	METEOR	CIDEr	ROUGE-L	±CIDEr (95%)
VISCON-Base, $B=1$	0.199	0.191	0.468	0.452	0.010
VISCON-Base, $B=3$	0.204	0.192	0.475	0.454	0.010
VISCON-Base, $B=5$	0.205	0.193	0.477	0.455	0.011
VISCON-Att, $B=1$	0.203	0.194	0.482	0.455	0.011
VISCON-Att, $B=3$	0.207	0.195	0.488	0.456	0.011
VISCON-Att, $B=5$	0.208	0.195	0.489	0.457	0.012

Table 7. Flickr8k: beam-width study and bootstrap confidence for CIDEr. Wider beams yield marginal improvements with diminishing returns.

Model / Beam	BLEU-4	METEOR	CIDEr	ROUGE-L	±CIDEr (95%)
VISCON-Base, $B=1$	0.199	0.191	0.468	0.452	0.010
VISCON-Base, $B=3$	0.204	0.192	0.475	0.454	0.010
VISCON-Base, $B=5$	0.205	0.193	0.477	0.455	0.011
VISCON-Att, $B=1$	0.203	0.194	0.482	0.455	0.011
VISCON-Att, $B=3$	0.207	0.195	0.488	0.456	0.011
VISCON-Att, $B=5$	0.208	0.195	0.489	0.457	0.012

4.13. Synthesis of Findings

Overall, VISCON validates that convolutional decoders can match recurrent decoders on several lexical metrics while being computationally efficient and easily parallelizable. However, long-range discourse and semantic richness—as captured by CIDEr—still favor recurrent (or self-attentional) models. For VISCON, shallow depth and conservative augmentation are preferable, and attention provides small but consistent benefits. The sentence-length study reveals a practical guideline: constrain training captions to moderate lengths or supplement VISCON with long-context mechanisms when targeting verbose descriptions.

5. Conclusions and Future Work

In this paper, we presented an extensive and systematic analysis of VISCON, a convolutional decoder framework for image caption generation. Unlike recurrent approaches that dominate existing literature, VISCON provides a lens to study the representational capacity and practical limitations of convolutional architectures when applied to natural language sequence modeling. Our investigation encompassed multiple key factors: the influence of decoder depth, the sensitivity of models to different sentence lengths, the effectiveness of image-level augmentation, and the integration of attention modules.

From our empirical results, several insights emerge. First, we discovered that deeper convolutional stacks do not translate into improved performance, contradicting the intuition derived from CNN-based vision tasks. The best results in our encoder-decoder setup were achieved with a single convolutional layer, while the attention-augmented variant benefited slightly from two layers. Second, augmentation strategies generally had limited or even adverse effects; only horizontal flipping yielded consistent though modest improvements, highlighting that augmentation in captioning is not directly analogous to recognition tasks. Third, VISCON demonstrates strong results with shorter captions but experiences a sharp decline as sentence length grows, underscoring its limited ability to capture long-range dependencies. This degradation was most evident in metrics such as CIDEr, which emphasizes semantic completeness and human-like richness. Taken together, these findings demonstrate that convolutional decoding provides a viable but constrained alternative to recurrent or transformer-based frameworks. While VISCON can achieve competitive performance in specific scenarios, its structural limitations prevent it from being a universal solution for captioning tasks.

Future Work.

Although VISCON offers new perspectives on convolutional semantics for language generation, several open challenges remain and motivate future directions. One promising avenue is the exploration of hybrid architectures that combine the local modeling strengths of convolution with the global sequence reasoning capabilities of transformers or recurrent mechanisms. Another is the development of adaptive convolutional kernels whose receptive fields adjust dynamically based on sentence progress, potentially alleviating the weakness in modeling long sequences. Furthermore, incorporating explicit linguistic structures—such as syntactic dependencies or semantic role graphs—may enhance the alignment between visual inputs and textual outputs. Finally, novel training strategies, such as curriculum learning with gradually increasing sentence lengths, could help stabilize optimization and improve generalization. By addressing these directions, future research may design captioning systems that unify the computational efficiency of convolution with the semantic depth required for human-like image descriptions.