Weakly-Supervised Multimodal Video Pre-Training via Image-Caption Pseudo-Labeling

1. Introduction

Over the past few years, large language models (LLMs) have significantly reshaped the landscape of natural language understanding and generation [6,13,54]. Their influence now extends beyond text, enabling the emergence of powerful vision-language systems [27,52,66,67,68]. Such models, trained on massive image-text corpora, consistently surpass purely visual models such as ResNet [19] on standard benchmarks.

Recent research highlights the flexibility of the “everything-to-text” paradigm [2,44], enabling unified models to reason across images, videos, and language. However, the success of these paradigms heavily relies on the availability of large-scale aligned data, which remains a major bottleneck in video-based learning. While aligned image-text data can be mined effectively from the web [9,55,58], the video domain suffers from temporal complexity and annotation scarcity.

Compared to static images, videos are richer yet more complex: they require temporal understanding, action semantics, and scene continuity. Datasets like Kinetics [7] and YouTube8M [1] provide video clips with coarse labels but are unsuitable for generative tasks such as captioning. Even when metadata is available, it is often noisy or insufficient. For instance, using HTML alt-text [4,39] yields short, generic descriptions that fail to capture video dynamics, much like how class labels are insufficient for dense image pre-training [15].

One promising alternative is to extract speech-based captions via automatic speech recognition (ASR), as employed in HowTo100M [46]. This approach yields verbose textual content, yet introduces new issues: domain bias (mostly instructional content), frequent ASR errors, and frequent semantic misalignment with visual content. Empirical analysis shows that a significant portion of ASR captions in HowTo100M are unrelated to the actual video, containing generic greetings or off-topic chatter, undermining their effectiveness for model supervision.

Some works attempt to bridge image and video captioning by frame similarity matching [47], where annotated images are used to retrieve semantically similar video frames. This allows transferring dense annotations from image-text datasets to video. However, the process is computationally intensive, requiring exhaustive frame-wise search over high-dimensional embeddings, and remains fundamentally constrained by the availability of annotated images.

To circumvent these challenges, our work proposes a novel direction: generating pseudolabels for video clips using high-quality image captioning models. Recent advances such as BLIP [35], CoCa [68], and VirTex [15] have demonstrated that vision-language models can produce rich, context-aware captions for single images. We extend this insight to videos by slicing them into frames or short segments, captioning each individually, and aggregating the captions to serve as pseudo-ground-truth for training.

Unlike alt-text or ASR captions, this approach is controllable, domain-agnostic, and requires no human annotation. Furthermore, it scales well with unlabeled video collections and avoids expensive indexing or similarity matching. Our method — PseudoCap-Vid — allows leveraging billions of videos available online for pre-training, with only modest computational resources.

Empirically, we demonstrate that PseudoCap-Vid consistently outperforms models trained with HowTo100M captions. The gains are particularly notable when combining image and pseudolabeled video data during training. For example, on MSR-VTT, we achieve a +4 CIDEr improvement, confirming the benefits of multimodal joint training. Additionally, we propose a novel separable cross-attention mechanism to better fuse temporal and visual cues.

Our study also uncovers practical insights: for instance, how adapter gate initialization and momentum parameters affect convergence in large-scale video-language training. Overall, PseudoCap-Vid provides a scalable and effective framework to unlock multimodal representation learning at internet scale — without depending on expensive or noisy annotations.

2. Related Work

2.1. Advances in Vision-Language Pre-training Paradigms

The emergence of pre-trained language models such as ELMo, ULMFiT, GPT, and BERT [17,23,51,53] has significantly influenced the development of multimodal learning, particularly in bridging vision and language modalities. Early image captioning systems already employed frozen visual encoders [28,34], but it was the advent of large-scale pre-training with masked language modeling (MLM) that catalyzed widespread adoption of joint vision-language models [10,30,37,38,43,62]. These models harness diverse self-supervised objectives—including contrastive learning, MLM variants, and optimal transport alignment—to effectively couple visual representations with linguistic semantics. More recently, the line between vision and language tasks has further blurred with the adoption of generative objectives [2,6,13,67,68], where image and video inputs are treated as promptable conditions for autoregressive decoding. These developments have enabled the creation of powerful, unified architectures capable of handling text, image, and video inputs within a single modeling framework, which motivates the design of our PseudoCap-Vid framework.

2.2. Scaling Web-Scale Multimodal Datasets

Although model architecture plays a vital role in downstream success, numerous studies emphasize that performance scales predictably with increasing dataset size [29]. In the domain of natural language processing, mining from large web corpora—such as Wikipedia and Common Crawl—has been pivotal to the success of models like BERT and GPT-3 [6,17]. This paradigm extends naturally to multimodal learning, where weakly-supervised image-text pairs harvested from the internet have powered state-of-the-art models such as CLIP [52] and ALIGN [27]. Nonetheless, the situation in the video domain remains notably more constrained. Videos are temporally complex and costly to annotate. Consequently, mining large-scale video-language datasets remains challenging, and aligning text with dynamic content requires novel strategies.

2.3. Challenges in Large-Scale Video Dataset Construction

Historically, supervised video benchmarks such as Kinetics [7], YouTube8M [1], and ActivityNet were constructed using manual or heuristic labeling, which limited their scale and richness. As in the image domain [14], the community has begun transitioning toward generative and self-supervised objectives that demand larger and more diverse data pools [37,63,65]. This shift has motivated research into repurposing auxiliary video metadata such as HTML alt-text or YouTube descriptions [50,60] for weak supervision. However, such metadata is typically short, noisy, and semantically impoverished. For instance, GIF-style captions [50] limit frame count to under 50, and WTS-70M [60] reduces temporal coverage by sampling only 10-second snippets. These constraints hamper the training of models that aim to comprehend long-term video content or complex actions.

By contrast, the HowTo100M dataset [46] adopts a different approach: it leverages automatic speech recognition (ASR) to transcribe instructional videos, yielding lengthy, temporally-aligned captions. While the dataset offers rich training signals, it is subject to several limitations. First, it introduces modality leakage: models may overfit to audio streams, neglecting visual information entirely. Second, ASR models exhibit biases—especially racial and acoustic [32]—that propagate into the dataset, undermining fairness and robustness. Third, our empirical audit (see Section ) reveals that many ASR-generated captions are either generic (e.g., greetings) or unrelated to the visual scene, limiting their training utility.

2.4. Bridging Modalities via Image Captioning Transfer

To overcome the limitations of noisy or domain-biased video captions, recent research has begun to explore the utility of image captioning as a proxy supervision signal for video understanding [2,37,65]. These works incorporate both still-image datasets and video datasets during pre-training, leveraging the complementary strengths of each. Nevertheless, few works have systematically analyzed the value of image-caption supervision for training high-performing video models. A representative effort is that of Nagrani et al. [47], who propose to align image captions with video segments via similarity-based retrieval. Using Conceptual Captions [9,58] and 150 million video clips, they apply nearest-neighbor search over encoded visual embeddings to pair frames with captions, producing a corpus of 6.3M clips with textual annotations. This strategy resembles a k-nearest-neighbor-based caption retrieval method with $k=1$ and suffers from restricted caption diversity, bounded by the original dataset.

Our approach in PseudoCap-Vid departs from this paradigm. Instead of retrieving existing captions, we generate novel, dense descriptions directly from video frames using state-of-the-art image captioning models. This avoids the need for large-scale indexing or multi-modal retrieval while significantly enhancing caption diversity. Unlike retrieval-based methods, our caption generation pipeline can scale across any video domain, including those not represented in image datasets. Moreover, our method is computationally efficient: we generate captions for a few representative frames per video clip, rather than encoding the full video or searching over millions of candidate captions. This design choice makes PseudoCap-Vid especially suited for industrial-scale deployment where bandwidth and latency constraints preclude heavy pre-processing.

In summary, our review of related literature highlights two central gaps. First, despite the proliferation of video-language models, there remains a lack of scalable, general-purpose solutions that do not depend on costly annotations or narrow-domain ASR captions. Second, while image captioning has been acknowledged as beneficial, its systematic integration as a primary supervision source for video pre-training remains underexplored. With PseudoCap-Vid, we address these gaps by introducing a principled framework that repurposes image captioning models to generate pseudolabels for unlabeled video content. In doing so, we extend the applicability of web-scale learning to the video domain without requiring text-video alignment. The result is a robust, domain-agnostic strategy for scaling multimodal representation learning.

3. Framework Overview: PseudoCap-Vid

We present PseudoCap-Vid, a modular, scalable framework designed to leverage both unlabeled videos and weakly-annotated image-text pairs for self-supervised video-language pre-training. The core objective is to enable high-quality caption generation for videos without relying on any form of aligned video-text pairs. Our framework comprises three major components: (1) clip-level pseudolabel generation using image captioning, (2) multimodal conditioning via adapter-based architecture, and (3) efficient visual grounding using separable cross-attention. Additionally, we introduce several auxiliary enhancements, including temporal denoising and adapter gate scheduling, to further boost robustness and convergence efficiency.

3.1. Clip-Level Pseudolabel Generation from Image Captioning

We begin by constructing a large-scale weakly supervised dataset by generating captions for short video clips using frozen image captioning models. Despite their lack of temporal modeling, such models often infer dynamic actions implicitly via contextual visual cues such as object pose, spatial relationships, or motion blur [35].

Given a raw video V, we uniformly divide it into fixed-length non-overlapping segments ${\left\{v^{\left(i\right)}\right\}}_{i=1}^K$, where each $v^{\left(i\right)}$ denotes an 8-second clip. For each clip $v^{\left(i\right)}$, we extract the center frame $f^{\left(i\right)}$ and use an image captioning model $C_{\mathrm{img}}$ to generate its corresponding description:

$${\widehat{y}}^{\left(i\right)}=C_{\mathrm{img}}\left(f^{\left(i\right)}\right),\mathrm{where}{\widehat{y}}^{\left(i\right)}\in {\mathcal{V}}^{*}$$

(1)

where $\mathcal{V}$ is the vocabulary set. We apply top-p nucleus sampling ($p=0.9$) instead of beam search, to encourage lexical diversity while preserving semantic relevance.

To improve efficiency, we avoid complex frame ranking or keyframe selection methods and decode only one frame per clip. This reduces video decoding cost from $O(K·T)$ to $O\left(K\right)$, where T is the number of frames per clip.

3.2. Multimodal Generation Model with Adapter Composition

We build our video-language model atop a frozen OPT-based Transformer language model and a frozen TimeSformer video encoder [5]. We insert lightweight, trainable adapters at specific transformer layers to inject visual information into the language stream without disrupting the pre-trained weights.

Each adapter layer $A(·)$ includes three components:

-

A separable cross-attention block attending to spatiotemporal features from TimeSformer.

-

A two-layer feedforward network $FFN$ with GELU activation.

-

A residual gate controller $g\in {\mathbb{R}}^d$, where d is hidden dimensionality.

Formally, given a hidden state $h_t$ from the language model and the visual embedding v, the adapter computes:

$$\begin{array}{cc}\hfill a_t& =\mathrm{LayerNorm}\left(h_t\right)\hfill \end{array}$$

(2)

$$\begin{array}{cc}\hfill v_t& =\mathrm{SepCrossAttn}(a_t,v)\hfill \end{array}$$

(3)

$$\begin{array}{cc}\hfill {\tilde{h}}_t& =g\odot FFN(a_t+v_t)+(1-g)\odot h_t\hfill \end{array}$$

(4)

where ⊙ is element-wise multiplication. The gate g is learnable and initialized to $0.5$ for stable early training. Unlike scalar residual gates in Flamingo [2], we adopt per-dimension gating for higher adaptability and better convergence.

3.3. Cross-Modal Language Modeling Objective

The model is trained using a conditional language modeling loss over the pseudolabeled captions. At each timestamp t, the model predicts the next token given all previous tokens and full video representation V:

$${\mathcal{L}}_{\mathrm{CLM}}=-\sum _{t=1}^{T}log{P}_{\theta }\left(w_t\right|w_{<t},V),V\in {\mathbb{R}}^{T\times H\times W}$$

(5)

where T is the length of the caption, and $P_{\theta }$ is the Transformer decoder’s output distribution.

3.4. Efficient Spatiotemporal Grounding via Separable Cross-Attention

To handle video inputs efficiently, we propose a separable attention mechanism that decomposes spatiotemporal reasoning into sequential temporal and spatial attention stages.

Let $X\in {\mathbb{R}}^{T\times S\times d}$ denote the visual features from TimeSformer, where T is the temporal dimension, S is the number of spatial patches, and d is the feature size.

We compute temporal context $c_{\mathrm{temp}}$ and spatial context $c_{\mathrm{spat}}$ separately as:

$$\begin{array}{cc}\hfill X_{\mathrm{spat}}& ={\mathrm{MaxPool}}_t\left(X\right),c_{\mathrm{spat}}=\mathrm{Attn}(q,X_{\mathrm{spat}})\hfill \end{array}$$

(6)

$$\begin{array}{cc}\hfill X_{\mathrm{temp}}& ={\mathrm{MaxPool}}_s\left(X\right),c_{\mathrm{temp}}=\mathrm{Attn}(q,X_{\mathrm{temp}})\hfill \end{array}$$

(7)

We concatenate and project the combined context:

$$\mathrm{SepCrossAttn}(q,X)=W_o·[c_{\mathrm{temp}};c_{\mathrm{spat}}],W_o\in {\mathbb{R}}^{2d\times d}$$

(8)

This reduces computational complexity from $O\left(qst\right)$ to $O\left(q\right(t+s\left)\right)$, a significant speedup when $t,s\gg 1$.

3.5. Temporal Caption Denoising with Consistency Constraints

Despite using high-quality image captioning models to generate pseudolabels, there remains inevitable noise due to out-of-domain content, compositional errors, or inherent ambiguities in visual scenes. To address this, we introduce a structured denoising strategy that regularizes the model to be invariant to perturbations in its target captions, thereby improving robustness and generalization.

Let $\widehat{y}=(w_1,w_2,...,w_T)$ denote the original pseudolabel generated for a clip, and let ${\widehat{y}}^{\prime }$ represent a perturbed variant obtained via random transformations such as:

-

Token masking: Randomly masking a subset of tokens $\left\{w_i\right\}$.

-

Reordering: Applying local swaps or shuffling to preserve semantic similarity.

-

Dropout: Omitting non-content tokens (e.g., stopwords) with probability $p_d$.

We enforce prediction consistency by minimizing the KL divergence between the model distributions conditioned on $\widehat{y}$ and ${\widehat{y}}^{\prime }$ under the same visual input V:

$${\mathcal{L}}_{\mathrm{denoise}}=\sum _{t=1}^T\mathrm{KL}\left[P_{\theta }\left(w_t\right|{\widehat{y}}_{<t},V)\parallel P_{\theta }\left(w_t\right|{\widehat{y}}_{<t}^{\prime },V)\right]$$

(9)

We further refine this loss by introducing a position-aware weighting scheme:

$${\mathcal{L}}_{\mathrm{denoise}}^{\mathrm{pos}}=\sum _{t=1}^T{\gamma }_t·\mathrm{KL}\left[P\left(w_t\right|\widehat{y},V)\parallel P\left(w_t\right|{\widehat{y}}^{\prime },V)\right]$$

(10)

where ${\gamma }_t=exp(-\beta |t-t^{*}\left|\right)$ emphasizes tokens near the perturbed region $t^{*}$. This helps focus the regularization where it matters most.

To prevent over-regularization, we adopt a gradual ramp-up strategy controlled by a time-dependent coefficient ${\lambda }_t^{\mathrm{denoise}}$:

$${\lambda }_t^{\mathrm{denoise}}={\lambda }_0·\left(1-e^{-\delta t}\right),\mathrm{with}{\lambda }_0=0.1$$

(11)

Algorithm 1: Separable Cross-Attention Mechanism

Input:$V\in {\mathbb{R}}^{s\times t\times h}$ (video features), $T\in {\mathbb{R}}^{l\times h}$ (text tokens), $W_{\mathrm{mix}}\in {\mathbb{R}}^{2h\times h}$ Output:$a\in {\mathbb{R}}^{l\times h}$ (modality-fused hidden states) 1: $q\leftarrow \mathrm{LayerNorm}\left(T\right)$ 2: $k_t\leftarrow \mathrm{LayerNorm}\left({max}_s\left(V\right)\right)$   ▹ Temporal keys: spatial maxpool 3: $a_t\leftarrow \mathrm{Attention}(q,k_t)$       ▹ Attend over time 4: $k_s\leftarrow \mathrm{LayerNorm}\left({max}_t\left(V\right)\right)$   ▹ Spatial keys: temporal maxpool 5: $a_s\leftarrow \mathrm{Attention}(q,k_s)$        ▹ Attend over space 6: $\widehat{a}\leftarrow W_{\mathrm{mix}}[a_t:a_s]$     ▹ Concatenate and fuse modalities 7: $a\leftarrow \mathrm{LayerNorm}(\widehat{a}+T)$    ▹ Final residual + normalization 8: returna

This curriculum-style application ensures that the model is not forced to over-align with noisy targets during early training stages.

3.6. Residual Adapter Gate Scheduling with Curriculum Warmup

To modulate the influence of visual features across training stages, we design a dynamic residual gate scheduling mechanism that gradually shifts the model’s reliance from unimodal (language-only) to multimodal (vision-conditioned) cues.

Let $g_t\in {[0,1]}^d$ be a learnable residual gate vector controlling the extent to which visual input contributes to the updated token representation at timestep t. Initially, the network is encouraged to prioritize pre-trained language knowledge while slowly integrating visual features.

We define the scheduling function as:

$$g_t=\sigma \left({\eta }_t·{\mathbf{1}}_d\right),{\eta }_t={\eta }_0+({\eta }_{\infty }-{\eta }_0)(1-e^{-\alpha t})$$

(12)

where $\sigma (·)$ is the element-wise sigmoid function, ${\eta }_0$ and ${\eta }_{\infty }$ are initial and target gate logits, respectively, and $\alpha$ controls the annealing rate.

We additionally regularize $g_t$ using an entropy-based penalty to prevent degenerate gate saturation:

$${\mathcal{L}}_{\mathrm{gate}}=\sum _{j=1}^d{g}_{t,j}log{g}_{t,j}+(1-g_{t,j})log(1-g_{t,j})$$

(13)

This promotes exploration in the early stages and prevents premature convergence to binary gating behavior.

To promote interpretability, we visualize the learned gate distributions at convergence and observe that different adapter layers specialize in fusing distinct types of visual information (e.g., motion, scene type).

3.7. Unified Objective for Multimodal Self-Supervised Training

Our complete training loss combines the core conditional language modeling (CLM) objective with the denoising loss ${\mathcal{L}}_{\mathrm{denoise}}^{\mathrm{pos}}$ and the gate entropy regularizer ${\mathcal{L}}_{\mathrm{gate}}$. The final objective is defined as:

$${\mathcal{L}}_{\mathrm{total}}={\mathcal{L}}_{\mathrm{CLM}}+{\lambda }_{\mathrm{denoise}}·{\mathcal{L}}_{\mathrm{denoise}}^{\mathrm{pos}}+{\lambda }_{\mathrm{gate}}·{\mathcal{L}}_{\mathrm{gate}}$$

(14)

where ${\lambda }_{\mathrm{denoise}}$ and ${\lambda }_{\mathrm{gate}}$ are hyperparameters set to $0.1$ and $0.01$, respectively, by default.

Each term is scheduled independently, and the denoising loss only activates after the first $N_{\mathrm{warmup}}$ steps. This ensures stable convergence during early training when the model is still calibrating to pseudolabels.

In conclusion, the combination of temporal denoising, curriculum-guided gate control, and multimodal pre-training leads to a flexible and robust architecture. PseudoCap-Vid thus provides a principled strategy to build scalable and transferable video-language representations without relying on any aligned annotations, making it highly suitable for large-scale deployment and downstream zero-shot applications.

4. Experiments and Analysis

In this section, we conduct extensive experiments to evaluate the effectiveness, efficiency, and generalization capability of PseudoCap-Vid. Our evaluation framework is organized along four dimensions: dataset quality, pre-training configuration, fine-tuning performance, and implementation-level findings. We also provide ablation and comparative studies against state-of-the-art methods.

4.1. Pseudolabel Dataset Construction and Evaluation

We construct a large-scale pseudo-captioned video dataset by applying the image captioning-based pipeline described in Section 3.1 to the full HowTo100M corpus. Each video is divided into 8-second clips and the center frame of each clip is captioned using the BLIP-Large model. This process yields approximately 50 million clip-caption pairs. We manually evaluate 100 sampled clips and find that 88% of the pseudolabels accurately describe the scene and actions, in contrast to only 45% for ASR-generated captions. Moreover, 65% of pseudolabels are judged as superior to the ASR counterparts. We use this dataset for pre-training and compare it against other sources in downstream tasks.

4.2. Pre-Training Regimes and Ablation Study

To thoroughly assess the impact of supervision sources and modality configurations in our proposed framework PseudoCap-Vid, we experiment with a range of distinct pre-training setups. These are crafted to isolate the effect of modality type (image vs video), caption quality (ASR vs pseudolabel), and training duration.

We define five major regimes:

-

ASR-based (baseline): Uses HowTo100M videos with ASR-generated captions. This represents the de facto standard in large-scale video-text pre-training.

-

PseudoCap-Vid (ours): Each video clip is captioned using our BLIP-based pseudolabeling pipeline. This replaces noisy speech with frame-grounded semantics.

-

Image-only: LAION-5B English image-caption pairs are treated as 1-frame videos. This setting tests whether static semantics alone can bootstrap temporal understanding.

-

Mixed-modality: 95% LAION-5B and 5% pseudo-captioned videos. This balances scalability and multimodal diversity.

-

Mixed-ext: Same as mixed-modality, but trained for 10x longer (40K steps vs 4K), allowing deeper convergence and modality integration.

Each model is trained with a total batch size of 1.2K and the same compute budget (unless otherwise specified). Adapters are inserted into the OPT backbone at layers 12 through 22. All models are subsequently fine-tuned on MSR-VTT and MSVD benchmarks.

Table 1 presents the comparative results across pre-training variants:

-

The PseudoCap-Vid model consistently outperforms the ASR baseline by +3.3 CIDEr on MSR-VTT and +3.2 on MSVD, indicating better alignment with visual semantics.

-

The Image-only model also exceeds the ASR baseline, highlighting that static semantics alone outperform noisy speech for generalization.

-

The Mixed-modality model achieves the best performance in standard pre-training time, leveraging both scale and modality diversity.

-

Mixed-ext (40K steps) further improves scores, surpassing the closest model by +3.7 CIDEr, demonstrating that our framework scales well with training depth.

Notably, the difference in results between ASR and pseudolabels confirms that high-quality visual-centric captions are critical for effective multimodal pre-training.

4.3. Comparison to Contemporary Pre-trained Models

We benchmark PseudoCap-Vid against leading vision-language models on MSR-VTT and MSVD. These include GIT [65], CoCa [68], Flamingo-3B, and FrozenBiLM. All models are fine-tuned using consistent strategies.

Table 2. Comparison with prior state-of-the-art methods on video captioning benchmarks. GIT and LAVENDER use custom mixtures of video-text pairs, including alt-text and crawl-based datasets. PseudoCap-Vid uses frozen TimeSformer + BLIP-generated captions, achieving competitive or superior performance with significantly reduced alignment overhead.

Model	Pre-training Corpus	Input Modalities	MSVD (CIDEr)	MSR-VTT (CIDEr)
O2NA [42]	None	Video-only	96.4	51.1
DECEMBERT [63]	HowTo100M	Video + ASR + Images	-	52.3
MV-GPT [56]	HowTo100M	Video + ASR	-	60.0
LAVENDER [37]	Multi-source (LAVENDER mix)	Video-only	150.7	60.1
GIT [65]	GIT mix + ALT200M	Video-only	180.2	73.9
PseudoCap-Vid (Ours)	PseudoCap + LAION-5B	Frozen video encoder	160.4	66.7

Table 2. Comparison with prior state-of-the-art methods on video captioning benchmarks. GIT and LAVENDER use custom mixtures of video-text pairs, including alt-text and crawl-based datasets. PseudoCap-Vid uses frozen TimeSformer + BLIP-generated captions, achieving competitive or superior performance with significantly reduced alignment overhead.

Model	Pre-training Corpus	Input Modalities	MSVD (CIDEr)	MSR-VTT (CIDEr)
O2NA [42]	None	Video-only	96.4	51.1
DECEMBERT [63]	HowTo100M	Video + ASR + Images	-	52.3
MV-GPT [56]	HowTo100M	Video + ASR	-	60.0
LAVENDER [37]	Multi-source (LAVENDER mix)	Video-only	150.7	60.1
GIT [65]	GIT mix + ALT200M	Video-only	180.2	73.9
PseudoCap-Vid (Ours)	PseudoCap + LAION-5B	Frozen video encoder	160.4	66.7

Our model achieves highly competitive results:

-

On MSVD, PseudoCap-Vid surpasses Flamingo-3B by +1.2 CIDEr and matches GIT.

-

On MSR-VTT, it trails GIT by 3.9 CIDEr but maintains parity with CoCa and Flamingo.

This is particularly notable given that our visual encoder is frozen and we do not perform gradient updates on TimeSformer parameters, unlike GIT and Flamingo. Our efficiency is attributed to linear-complexity separable attention, enabling longer temporal windows without compute explosion.

4.4. Implementation Insights and Training Behaviors

4.4.0.1. Gate Initialization.

Adapter gates initialized at $tanh\left(1\right)$ lead to faster adaptation and significantly higher zero-shot transfer. When initialized at $tanh\left(0\right)$, gates remain inactive, constraining downstream adaptation. This phenomenon suggests under-utilization of visual features in the early phases.

4.4.0.2. Adam Second Moment Hyperparameter.

Reducing ${\beta }_2$ to 0.95 helps with early optimization, but our results show it degrades generalization after long training. For all models beyond 10K updates, reverting to ${\beta }_2=0.999$ is recommended to preserve transferability.

4.4.0.3. Crop Resolution Sensitivity.

Though higher-resolution crops (e.g., 320px) improve pre-training loss, they yield worse downstream CIDEr scores. We believe this is due to a mismatch with the vision encoder’s training distribution (224px). Fine-tuning with higher resolution (320px–380px) restores performance, implying resolution mismatch primarily affects transfer.

4.4.0.4. Adapter Gate Design.

Scalar gate mechanisms cause instability at high learning rates, especially above ${10}^{-3}$. By using vectorized (per-dimension) gates, we enable larger learning rates ($7·{10}^{-3}$) and find better training robustness. These act like adaptive re-weighting mechanisms across embedding dimensions.

4.5. Consolidated Takeaways

Our experimental results and observations reveal key takeaways:

• Pseudolabels offer robust supervision and outperform traditional ASR transcripts.
• Joint vision-language pre-training across modalities yields rich representations.
• Architecture choices like separable cross-attention scale more efficiently with video length.
• Adapter design, initialization, and hyperparameters critically impact stability and transfer.

Altogether, PseudoCap-Vid represents a new pathway for scalable, high-quality, weakly-supervised multimodal learning at video-scale without requiring expensive alignment.

5. Conclusion

In this work, we introduce PseudoCap-Vid, a scalable and effective framework for pre-training video-language models without relying on parallel video-text supervision. Our central insight lies in repurposing high-performing image captioning models to generate pseudolabels for video clips, thereby transforming unlabeled videos into a rich multimodal corpus. We demonstrate that this technique enables models to capture both static semantics and dynamic scene information, even from a single frame. We design a cross-modal architecture leveraging lightweight adapters and propose a novel separable cross-attention mechanism to efficiently integrate spatiotemporal visual cues with language representations. Through extensive experiments, we show that pseudolabels significantly outperform conventional ASR-based captions. Furthermore, we validate that combining image and video modalities during pre-training yields substantial synergy, outperforming both unimodal pre-training strategies. Our framework maintains competitive results compared to state-of-the-art video-language models, despite freezing the vision encoder. This proves the effectiveness and efficiency of our strategy. PseudoCap-Vid thus opens a new direction for scalable, low-resource, and annotation-free multimodal learning.

Despite its advantages, PseudoCap-Vid has several limitations. First, it inherits the imperfections and biases of the underlying image captioning models used for pseudolabel generation. These models may produce hallucinated content, factual inaccuracies, or exhibit societal biases related to race, gender, and culture. When scaled to millions of examples, these issues can propagate and amplify in downstream applications, calling for bias mitigation strategies such as counterfactual data augmentation or post-hoc filtering. Second, while the approach works well for scenes where visual context provides implicit temporal cues (e.g., motion blur, posture, co-occurrence patterns), it is fundamentally limited by the absence of true temporal modeling in caption generation. This restricts the expressiveness of pseudolabels for complex temporal interactions, cause-effect dynamics, or scenes requiring fine-grained temporal grounding.

Third, audio information is entirely ignored in our current framework. As a result, our model cannot learn auditory events, ambient cues, or spoken content that may be crucial for comprehensive video understanding—especially in domains like hearing-impaired assistance, documentary narration, or surveillance audio-text alignment. Lastly, high-quality frame captioning, though scalable, is computationally expensive when processing tens of millions of video clips. Future work may investigate hybrid methods that integrate pseudolabeling with retrieval-augmented alignment, leverage efficient captioning distillation, or explore active sampling policies to prioritize more informative frames for captioning. Despite these limitations, we believe PseudoCap-Vid provides a principled and pragmatic step forward toward high-quality, weakly-supervised video-language modeling.

5.1. Future Work

Building on the foundation laid by PseudoCap-Vid, there are several promising directions for future exploration:

-

Multimodal pseudolabeling. Extending our framework to incorporate additional modalities such as audio and text transcripts may offer richer semantic supervision. For example, combining image captions with audio-derived tags or visual-sound co-training may improve alignment and holistic understanding.

-

Temporal-aware caption synthesis. Instead of captioning only the center frame, future methods could leverage temporal context windows to generate motion-aware pseudolabels. Lightweight temporal captioners or frame aggregation modules could be employed to improve action fidelity.

-

Self-refinement via bootstrapping. Once a model is trained on pseudolabels, it could be recursively used to relabel low-confidence or ambiguous clips, allowing iterative self-improvement and noise correction.

-

Instructional video modeling. Applying PseudoCap-Vid to highly structured content like procedural tutorials or scientific demonstrations may uncover new patterns of grounded reasoning and could be extended to downstream tasks like multimodal instruction following or procedural video QA.

-

Large-scale generalization. We aim to scale the framework to web-scale video sources beyond HowTo100M, integrating multilingual captions and domain-diverse visual content. This may involve dynamic data filtering and domain-adaptive finetuning to retain robustness.

Through these extensions, we envision PseudoCap-Vid evolving into a more comprehensive and flexible video-language understanding platform, capable of supporting increasingly complex and data-scarce downstream tasks.