Multi-modal Learning with Dynamic Integration for Video Temporal Localization

1. Introduction

The proliferation of video data across diverse domains such as entertainment, surveillance, and education has made video understanding an essential research area in computer vision. A particularly important problem in this space is temporal localization, which seeks to identify specific segments within a video based on high-level input, such as natural language descriptions. This capability is fundamental for enabling efficient video indexing, retrieval, and summarization, which are critical for managing the massive amounts of video content generated daily on platforms like YouTube, TikTok, and surveillance systems.

While video understanding is a well-explored field, the integration of multiple modalities, including vision and language, has gained increasing attention in recent years. Tasks like video captioning [17,23] and video question answering [16,18] exemplify the need to process video data in conjunction with textual information [22,26,33,34]. Among these tasks, text-guided video temporal localization is particularly challenging due to its requirement for precise spatiotemporal alignment between video content and natural language queries. This alignment must account for diverse scenes, varying actions, and subtle contextual cues, making it a cornerstone of advanced video analysis.

The primary objective of text-guided video temporal localization is to determine the start and end times of a video segment that corresponds to a given text description. For instance, given a query like "a person closing a door," the system must accurately identify the precise interval where this action occurs. This task has wide-ranging applications, including automatic video editing, intelligent video retrieval, and enhanced human-computer interaction. In surveillance systems, for example, this capability could be used to locate security-critical events within hours of footage based on textual descriptions provided by human operators.

Despite its importance, existing methods face significant challenges due to their reliance on single-modality visual representations, typically RGB frames. While RGB features capture rich appearance information, they are often insufficient for complex real-world scenarios where motion and spatial context play critical roles. For instance, actions like "throwing a ball" may be visually ambiguous without motion information, while scenes involving objects with specific shapes, such as "sitting at a desk," benefit greatly from depth perception.

To address these limitations, multi-modal approaches have been proposed to incorporate additional modalities like optical flow and depth maps. Optical flow, which captures the motion dynamics between consecutive frames, is particularly effective for identifying large-scale actions and transient events. Depth maps, on the other hand, provide spatial and structural context, helping to disambiguate actions that involve subtle movements or distinct object shapes. Together, these modalities complement RGB features and offer a holistic understanding of video content.

However, integrating multiple modalities introduces its own challenges. A naive approach involves using separate streams to process each modality independently, followed by a late fusion of their outputs. While straightforward, this method often fails to capture the complex interdependencies and contextual relationships between modalities. For example, the importance of depth cues may vary depending on the presence of motion, and vice versa. Therefore, a more sophisticated integration strategy is required to dynamically weigh and combine information from different modalities based on the specific context of the query and the video.

In this work, we propose a novel multi-modal framework named MODIFUSE (Multi-modal Dynamic Integration with Fused Understanding for Scene Events). MODIFUSE is designed to address the shortcomings of existing approaches by introducing a dynamic integration mechanism based on transformer architectures. This mechanism allows for adaptive fusion of RGB, optical flow, and depth features, enabling the model to capture intricate cross-modal interactions and prioritize the most relevant modalities for each video-text pair. Additionally, we incorporate intra-modal self-supervised learning to enhance the consistency and robustness of feature representations within each modality, further improving the overall performance of our approach.

The contributions of this work are motivated by the need for scalable and effective solutions to text-guided video temporal localization in real-world applications. By leveraging the complementary strengths of multiple modalities and introducing novel integration techniques, we aim to push the boundaries of what is achievable in this domain. Our method is extensively validated on benchmark datasets, demonstrating state-of-the-art performance and highlighting its potential for a wide range of practical applications.

2. Related Work

2.1. Multimodal Modeling

Multi-modal learning has emerged as a powerful paradigm for understanding complex phenomena by integrating information from diverse sources. Actions and events are often described by signals spanning multiple modalities, such as vision, language, and audio. The ability to model cross-modal correlations is crucial for achieving a comprehensive understanding of the underlying tasks.

Recent advances in joint vision-language learning have garnered significant attention due to the natural alignment between visual content and textual descriptions [6,7,16,23]. Transformer-based models [8,24], such as BERT [8], have demonstrated exceptional performance in natural language tasks, inspiring their adaptation to multi-modal learning. For example, ViLBERT [20] employs co-attentional transformer layers to model interactions between image and text features, enabling the joint representation of visual and linguistic information. Similarly, other frameworks [5,9,19,21,35] leverage large-scale pretraining on image-text pairs to learn transferable representations for downstream tasks.

Despite the success of these methods, multi-modal learning for video data remains underexplored, particularly in the context of temporal grounding tasks. Videos inherently combine spatial, temporal, and contextual information, making them more complex than static images. To address this, our framework, MODIFUSE, introduces a dynamic integration strategy for fusing RGB, optical flow, and depth features. By leveraging co-attentional transformer layers, our approach captures intricate cross-modal relationships and adapts to the specific requirements of text-guided video temporal grounding. Furthermore, we enhance feature learning through self-supervised contrastive techniques, ensuring robust and consistent representations across modalities and videos.

Through these innovations, MODIFUSE provides a comprehensive solution to the challenges of multi-modal video understanding, pushing the boundaries of what is achievable in video temporal localization tasks.

2.2. Video Temporal Grounding

Text-guided video temporal grounding is a critical task in video understanding, aiming to identify the temporal segment within a video that corresponds to a natural language query. This involves predicting the start and end times of the relevant video clip based on the semantic alignment between video content and textual descriptions. Existing methods addressing this problem can be broadly categorized into two paradigms: two-stage and one-stage approaches.

Two-stage methods employ a propose-and-rank framework, wherein candidate video segments are first generated as proposals and subsequently ranked based on their relevance to the query. Early methods, such as [10,14], relied on sliding window mechanisms to scan the entire video for generating segment proposals. However, the high computational cost and redundancy of such methods prompted the development of more efficient proposal-generation techniques. For instance, the TGN model [2] integrates frame-by-word interactions to localize proposals in a single pass. Similarly, query-guided proposal generation [31] and semantic activity proposals [3] were introduced to improve proposal relevance. Other models, such as MAN [34], leverage graph architectures to model temporal relationships among proposals, while reinforcement learning-based methods [13,30] aim to optimize video exploration strategies by intelligently skipping irrelevant frames. Despite their promising results, two-stage methods suffer from high computational overhead and dependency on proposal quality, limiting their scalability and performance.

In contrast, one-stage methods bypass the intermediate proposal-generation step and directly predict temporal boundaries by integrating video and query features. These methods predominantly focus on enhancing cross-modal interactions. For example, the ABLR model [32] employs co-attention mechanisms for location regression, while ExCL [11] leverages cross-modal interactions to refine predictions. Similarly, PfTML-GA [26] utilizes query-guided dynamic filters to enhance performance, and DRN [33] incorporates dense supervision to improve training efficiency. The LGI model [22] further advances one-stage methods by decomposing query sentences into semantic phrases and performing hierarchical video-text interactions.

Building on the strengths of LGI, our framework, MODIFUSE, incorporates RGB, optical flow, and depth features to capture complementary visual cues. Unlike LGI, which processes only RGB inputs, our approach introduces a novel inter-modality learning mechanism to enhance feature integration. Additionally, we apply contrastive learning to improve intra-modal feature consistency, further boosting the model’s robustness and effectiveness.

3. Our MODIFUSE Framework

In this paper, we tackle the problem of text-guided video temporal grounding by introducing a novel multi-modal framework, MODIFUSE. Our approach is designed to leverage complementary information from RGB images, optical flow, and depth maps for a more comprehensive understanding of video content. The framework integrates inter-modal and intra-modal feature learning strategies to enhance performance in identifying the temporal boundaries of events described by natural language queries.

Given an input video $V={\left\{V_t\right\}}_{t=1}^T$ with T frames and a natural language query $Q={\left\{Q_i\right\}}_{i=1}^N$ containing N words, the goal is to predict the starting and ending times, $[t_s,t_e]$, of the video segment that best corresponds to the query. To achieve this, our framework comprises the following components: 1. Feature extraction modules for RGB, optical flow, and depth modalities. 2. A textual encoder to represent the query. 3. Inter-modal feature fusion using co-attentional transformers. 4. Intra-modal self-supervised contrastive learning to enhance feature robustness. 5. A regression module to predict temporal boundaries.

3.1. Multi-Modal Feature Extraction

From the input video, we compute depth maps for each frame and optical flow between consecutive frames. Features are then extracted using specialized encoders for each modality:

$${\mathbf{F}}_{rgb}=E_r\left(V\right),{\mathbf{F}}_{flow}=E_f\left(O\left(V\right)\right),{\mathbf{F}}_{depth}=E_d\left(D\left(V\right)\right),$$

where $E_r,E_f,E_d$ denote the encoders for RGB, optical flow, and depth, respectively, and $O(·)$ and $D(·)$ represent optical flow and depth operations. The textual encoder $E_t$ processes the query Q to produce ${\mathbf{F}}_t$, the textual feature representation.

The extracted features are then passed to the local-global interaction (LGI) modules, which incorporate the query information into each modality, generating ${\mathbf{M}}_{rgb},{\mathbf{M}}_{flow},{\mathbf{M}}_{depth}$. These serve as the input for inter-modal learning.

3.2. Inter-Modal Feature Fusion

Videos inherently contain spatial, temporal, and motion information distributed across different modalities. A naive approach would average the outputs of each modality, but this fails to account for the varying relevance of modalities across different scenarios. For instance, depth features might be more critical for static scenes, while optical flow is essential for dynamic actions. To address this, we employ a co-attentional transformer-based feature fusion mechanism.

Co-Attentional Transformer.

Inspired by [20], we utilize co-attentional transformer layers to enable cross-modal interactions. For each pair of modalities (e.g., RGB and depth), the features are processed as follows:

$${\mathbf{A}}_{rgb,depth}=\mathrm{Softmax}\left({\displaystyle \frac{{\mathbf{Q}}_{rgb}{\mathbf{K}}_{depth}^{\top }}{\sqrt{d}}}\right){\mathbf{V}}_{depth},$$

where $\mathbf{Q},\mathbf{K},\mathbf{V}$ denote the query, key, and value matrices derived from the respective modality features, and d is the feature dimension. This operation ensures that each modality benefits from the contextual information provided by the others.

The outputs of these layers are dynamically weighted and combined:

$${\mathbf{F}}_{fused}=\sum _m{w}_m{\mathbf{A}}_m,$$

where $w_m$ are weights learned via a fully connected layer, normalized such that ${\sum }_m{w}_m=1$.

3.3. Intra-Modal Contrastive Learning

To improve feature quality within each modality, we incorporate intra-modal contrastive learning. This ensures that features representing similar actions across different videos are closer in the feature space, while dissimilar actions are pushed apart.

Given a set of positive pairs $(\mathbf{M},{\mathbf{M}}_{+})$ and negative pairs $(\mathbf{M},{\mathbf{M}}_{-})$, the contrastive loss is defined as:

$$L_{cl}=-log{\displaystyle \frac{exp(\mathrm{sim}(\mathbf{M},{\mathbf{M}}_{+})/\tau )}{exp(\mathrm{sim}(\mathbf{M},{\mathbf{M}}_{+})/\tau )+{\displaystyle \sum _{-}}exp(\mathrm{sim}(\mathbf{M},{\mathbf{M}}_{-})/\tau )}},$$

where $\mathrm{sim}(·,·)$ denotes cosine similarity, and $\tau$ is a temperature parameter.

3.4. Temporal Regression Module

The fused multi-modal feature ${\mathbf{F}}_{fused}$ is fed into a regression module to predict the temporal boundaries $[t_s,t_e]$. The module includes: 1. Temporal attention to focus on relevant frames. 2. Boundary regression using a combination of linear and non-linear layers:

$$t_s,t_e=\mathrm{REG}\left({\mathbf{F}}_{fused}\right),$$

where $\mathrm{REG}(·)$ outputs the normalized start and end times.

3.5. Training Objective

The total loss is a combination of supervised and self-supervised components:

$$L=L_{grn}+L_{cl},$$

where $L_{grn}$ comprises location regression, temporal attention guidance, and distinct query attention losses, as described in [22], and $L_{cl}$ is the contrastive loss defined above.

3.6. Implementation Details

We generate optical flow and depth maps using RAFT [27] and MiDaS [25], respectively. The encoders $E_r,E_f,E_d$ are based on I3D [1] for Charades-STA and C3D [28] for ActivityNet Captions. The textual encoder $E_t$ is a bidirectional LSTM, with features obtained by concatenating the final hidden states. Training is performed using the Adam optimizer with a learning rate of $4\times {10}^{-4}$. All experiments are implemented in PyTorch, and the code is available at https://github.com/MODIFUSE.

4. Experiments

4.1. Settings

We evaluate the proposed MODIFUSE framework against state-of-the-art approaches on two benchmark datasets: Charades-STA [10] and ActivityNet Captions [17].

Charades-STA

This dataset, built upon the Charades dataset, is specifically designed to evaluate the video temporal grounding task. It contains 6,672 videos and 16,128 video-query pairs, split into 12,408 pairs for training and 3,720 pairs for testing. The videos have an average duration of 29.76 seconds, and each video includes 2.4 annotated moments, with an average duration of 8.2 seconds. These features make Charades-STA suitable for evaluating fine-grained temporal grounding capabilities.

ActivityNet Captions

Originally constructed for dense video captioning, the ActivityNet Captions dataset is repurposed for temporal grounding using captions as queries. It includes approximately 20,000 YouTube videos with an average duration of 120 seconds, covering 200 diverse activity categories. Each video contains 3.65 queries on average, with a mean query length of 13.48 words. The dataset is split into training, validation, and testing sets in a 2:1:1 ratio, resulting in 37,421, 17,505, and 17,031 video-query pairs, respectively. As the test set annotations are unavailable, evaluations follow the standard setup of using combined validation sets, denoted as $va{l}_1$ and $va{l}_2$.

Evaluation Metrics

We use two widely adopted metrics to evaluate temporal grounding performance: 1. Recall at Intersection over Union (R@IoU): Measures the percentage of predictions achieving an IoU above a specified threshold with the ground truth. We report results for IoU thresholds {0.3, 0.5, 0.7}. 2. Mean IoU (mIoU): Calculates the average IoU across all predictions, providing a comprehensive performance indicator.

4.2. Overall Performance

Table 1 compares the performance of MODIFUSE with state-of-the-art methods, including two-stage approaches reliant on proposal-ranking [10,13,34] and one-stage models leveraging RGB features [12,22,26,33].

Performance Highlights

1. Compared to the baseline LGI [22], our method achieves consistent improvements across all metrics, demonstrating the effectiveness of inter-modal fusion and intra-modal contrastive learning. 2. Our single-modal model, which uses RGB features with contrastive learning, surpasses several prior methods. This indicates the robustness of our intra-modal learning strategy. 3. Incorporating additional modalities (optical flow and depth) into MODIFUSE leads to significant performance gains, validating the complementary nature of these modalities. For instance, the three-modal MODIFUSE achieves more than 5% improvement in all metrics across both datasets compared to single-modal baselines.

4.3. Ablation Study

We conduct extensive ablation experiments to analyze the contributions of individual components in MODIFUSE. Table 2 summarizes the results.

Inter-Modal Feature Fusion

The proposed inter-modal module, which includes co-attentional transformers and dynamic feature fusion, plays a critical role in the framework. Ablations reveal that: 1. Removing co-attentional transformers results in significant performance drops, highlighting their importance in capturing cross-modal interactions. 2. Dynamic fusion with learnable weights ensures that modality contributions are adapted to the input context. Removing this mechanism reduces performance significantly, as shown in Table 2.

Intra-Modal Contrastive Learning

Incorporating contrastive learning enhances feature representations within individual modalities. Removing this component leads to decreased performance, emphasizing its role in improving intra-modal robustness.

5. Conclusions and Future Directions

In this paper, we address the challenging task of text-guided video temporal grounding by introducing a novel multi-modal framework, MODIFUSE. Unlike traditional methods that rely solely on RGB features, our approach incorporates three complementary modalities: RGB, optical flow, and depth. This enables a more holistic understanding of video content, particularly in scenarios with cluttered backgrounds, subtle motions, or complex spatial structures.

RGB features, while rich in appearance information, often struggle to disambiguate actions in visually noisy environments. To address this limitation, we integrate optical flow, which captures dynamic motion patterns, and depth maps, which provide structural insights into the scene. These three modalities are effectively combined using our proposed inter-modal feature learning module. This module employs co-attentional transformers to facilitate cross-modal interactions and dynamically fuses the multi-modal features based on context, ensuring that the most relevant modality is prioritized for each query-video pair.

To further improve the training process, we introduce an intra-modal feature learning module, which enhances the robustness of individual modalities through self-supervised contrastive learning. By enforcing features of the same action across different videos to cluster closer in the feature space while pushing apart features from different actions, this module significantly improves intra-modal feature representation quality. The proposed contrastive loss promotes cross-video consistency and complements our inter-modal learning strategy.

We validate the effectiveness of MODIFUSE through comprehensive experiments on two benchmark datasets, Charades-STA and ActivityNet Captions. Our results demonstrate substantial improvements over state-of-the-art methods across multiple evaluation metrics. Specifically, we show that the inclusion of optical flow and depth features enhances the ability to recognize motion and structural cues, while the proposed inter- and intra-modal learning modules provide consistent performance gains. These results highlight the strength of our framework in addressing the complexities of text-guided video temporal grounding.

Future Directions

The proposed MODIFUSE framework opens several promising avenues for future research: 1. Extension to Additional Modalities: While this work focuses on RGB, optical flow, and depth, incorporating other modalities such as audio or semantic segmentation could further enhance the understanding of complex videos. 2. Generalization to Longer and More Diverse Videos: Future work could investigate how to scale the framework to handle significantly longer videos with a wider variety of activities and queries, potentially leveraging hierarchical attention mechanisms. 3. Application to Real-Time Systems: Optimizing the computational efficiency of MODIFUSE could enable its deployment in real-time video analysis systems for applications like surveillance, sports analysis, and interactive media. 4. Unsupervised and Few-Shot Learning: Exploring the potential of unsupervised or few-shot learning paradigms could further reduce the dependency on large annotated datasets, making the framework more versatile for real-world scenarios. 5. Integration with Large Language Models (LLMs): With the growing capabilities of LLMs, integrating these models into MODIFUSE could provide deeper semantic understanding of queries and enhance cross-modal reasoning.

In conclusion, the proposed MODIFUSE framework represents a significant step forward in the field of video temporal grounding. By leveraging complementary modalities and advanced learning mechanisms, it sets a robust foundation for future advancements in multi-modal video understanding.