SFN-ESVQA: Spatio-Temporal Frame-Aware Network for Endoscopic Surgery Visual Question Answering

1. Introduction

In recent years, the rapid development of Artificial Intelligence (AI), particularly the wave of technologies represented by language models (LMs), is profoundly transforming various sectors [1,2,3]. In the surgical domain, which demands exceptional precision and experience, LMs have demonstrated immense potential [5]. It not only assists surgeons with precise and intraoperative decision-making, but also plays a crucial role in postoperative review and training [4,9]. By conducting in-depth analysis of large surgical data, AI can help junior surgeons rapidly learn from expert experience, thereby enhancing their surgical skills and advancing the general standard of surgical practice [6,7].

Endoscopic surgery is a delicate and complex procedure. Surgeons must navigate through a narrow corridor surrounded by critical neurovascular structures, which demands extremely high precision and dexterity [8]. Furthermore, the two-dimensional (2D) view provided by the endoscope poses significant challenges to the surgeon’s depth perception and spatial awareness, where even minor errors can lead to severe complications [10]. The steep learning curve associated with these technical complexities [43,57], combined with the scarcity of expert surgeons available for mentorship, creates a significant bottleneck in surgical training [11]. Junior doctors often lack sufficient opportunities for in-depth review and personalized feedback on complex procedures [12,14]. This highlights a pressing need for intelligent tools that can scale expert knowledge and support postoperative review and training.

To address these challenges, Visual Question Answering (VQA) has emerged as an advanced human-computer interaction paradigm for analyzing endoscopic surgery. As a promising approach for postoperative review and training, this paradigm aims to leverage some language models integrated with rich modality information. Existing SOTA models [4,15] primarily utilize visual-language information fusion techniques based on static images. These works have pioneered the feasibility of using visual-language information for postoperative review and training and form the foundation of research in this field.

However, these models share a fundamental limitation: they primarily rely on discrete static images. This approach is ill-suited for endoscopic surgery, an inherently dynamic and continuous process where instrument movements, tissue changes, and strategic maneuvers create complex long-range dependencies [16]. Analyzing frames in isolation omits this crucial temporal context and fails to capture the intricate tool-tissue interactions that define surgical activities. Consequently, models’ capacity for true scene understanding is constrained, limiting their ability to perform nuanced assessments of complex surgical procedures [17]. This results in a high error rate and irrelevance in model responses, which affects postoperative review and training. A typical example of this, as illustrated in Figure 1, is when models confuse the type of the question asked and give an irrelevant answer.

To overcome these limitations and bridge this critical gap, we propose SFN-ESVQA, an innovative framework that integrates the video modality into the surgical VQA pipeline by using a frame-aware network. Meanwhile, we design a loss function to reduce the likelihood of the model generates irrelevant responses, and devise a new metrics to conduct a more comprehensive evaluation. Our main contributions are threefold:

• We propose a frame-aware network named SFN-ESVQA, which effectively incorporates video data to capture the temporal dynamics of surgical procedures. With frame-aware sampling, the network samples the selected frames and then fuses the video modality and image modality into visual embedding, integrating the visual feature and textual feature by proposed visual guided attention, resulting in better understanding of the question and the video.
• We design a standardized evaluation protocol and introduce a novel metric, Irrelevant Answer Rate (IAR), to quantify the frequency of irrelevant responses. Meanwhile we propose a tailored loss function that explicitly penalizes such irrelevant outputs to enhance our model’s ability to generate the appropriate answer.
• Extensive experiments are conducted based on the PitVQA [4] and EndoVis18-VQA [18] datasets. We achieved improvements of 35.28% and 27.83% in balanced accuracy, and 25.44% and 29.86% in F1-score across the two datasets, demonstrating the great potential for its application in the field of postoperative review and training.

2. Related Work

2.1. Medical Image-Text VQA

Medical Visual Question Answering (VQA) lies at the convergence of computer vision, natural language processing, and clinical practice. It holds great promise for applications such as clinical decision support and postoperative review [19,20]. A typical VQA model consists of an image encoder, a question encoder, a fusion module, and an answer decoder [21,22].

Medical VQA should be differentiated from related tasks, such as Visual Question Generation (VQG), which generates questions based on images [23,24], and Visual Question Localized Answering (VQLA) [25], which requires both an answer and its spatial location.

Early Medical VQA systems relied on CNNs for image encoding [27] and RNNs [28] (for example LSTM [29], GRU [30]) for text processing. For example, Neural-Image-QA [31] used a CNN-LSTM pipeline to answer visual questions end-to-end.

The introduction of Transformers marked a major shift, enabling more powerful cross-modal modeling [33,34,35,36,37]. BERT-based models such as VisualBERT [38] demonstrated great performance gains in medical VQA tasks. However, many of these models still depend heavily on region-based object detectors (e.g. Faster R-CNN [40]) and Bottom-Up features, which are less practical in surgical scenarios.

To reduce region-based object detector dependence, detection-free surgical models learn end-to-end from global image features, removing the need for bounding-box annotations such as attention-based model named MedfuseNet [41] with CNN and MMBERT [42].

Surgical-VQA[17] also introduced a detection-free architecture, removing the dependency on pre-annotated regions and facilitating efficient model training. Surgical-VQLA[44] and CAT-ViL[45] further advanced the task from traditional VQA (answering "what") to VQLA as we mentioned before, requiring both an answer and its spatial localization in the image. To enhance modality fusion, Surgical-VQLA adopted a gated embedding mechanism to dynamically balance textual and visual inputs, while CAT-ViL leveraged co-attention to guide visual focus through textual context.

With the rise of language models, recent research has explored their integration into VQA tasks. Early approaches, such as SurgicalGPT [15] and PitVQA-Net [4], employed adaptation strategies that reordered input tokens or generated image-aware text embeddings to align visual and language modalities. More recent work has focused on fine-tuning large vision-language models (LVLMs) (e.g. Qwen-VL [46]) for surgical domain, which provides strong general-purpose vision-language capabilities. Surgical-LVLM [16] enhances surgical reasoning by introducing domain-specific tuning modules, including VP-LoRA and Token Interaction.

Despite notable progress, medical VQA is largely constrained by static image formulations that overlook temporal context. Capturing the continuity of tool–tissue interactions therefore requires the introduction of video modal in endoscopic surgery.

2.2. Videos on Endoscopic Surgery

Video records of clinical surgeries are very valuable, particularly in the aspect of endoscopic surgery, and can be archived and used later for reasons such as postoperative review and training, skills assessment, and workflow analysis [47].

Currently, the surgical video application paradigm is under profound transformation: from traditional passive review and manual inspection toward intelligent analysis driven by data and empowered algorithms [13,49]. Kamtam et al. [39] conducted a survey of the literature from 2014 to 2024 on anatomical structure segmentation [32,48] and object detection in surgical videos, demonstrating the feasibility and growing potential of surgical video analysis. In fact, the field has seen rapid progress in various automated tasks, including surgical phase recognition [50], instrument tracking [51], surgical skill assessment [52], and the detection of critical events such as bleeding or other complications [53].

However, these methods focus mainly on providing pre-defined, high-level outputs, such as classifying a video segment into a specific surgical phase or identifying a known type of event [54]. They lack the capability for fine-grained, interactive querying that is essential for nuanced postoperative review and training. For example, a trainee cannot ask a standard phase recognition model why a surgeon chose a particular instrument at a specific moment or what the anatomical structure to the left of the forceps is. This limitation highlights a critical gap: the need for a system that can understand surgical videos dynamically and engage in a natural language dialogue to answer specific, ad-hoc questions.

2.3. Compared with Video-Grounded QA: Focusing on Frame-Aware

In recent years, Video-Based Question Answering (Video VQA) has emerged as a powerful paradigm for understanding complex visual content in a temporally coherent manner. This paradigm, which has emerged as a prominent direction in vision language understanding [55], provides the foundational techniques for building interactive and dynamic analysis tools.

Early Video QA work [56,58,59] primarily utilized attention mechanisms and RNN-based architectures to enhance cross-modal reasoning between videos and questions. However, the use of recurrent models limited their ability to capture long-range temporal dependencies in extended video sequences. Later works introduced memory networks and Transformer-based architectures to address these issues. Tapaswi et al. [60] employed memory networks to model visual linguistic context in movie scenes, while more recent models such as PSAC [61] and ClipBERT [62] used self-attention and cross-modal pre-training to improve temporal alignment and reasoning.

However, directly applying such video-level QA methods to endoscopic surgical scenes faces significant practical and conceptual challenges. Annotation is costly, requiring expert-level knowledge and frame-level precision. Surgical videos are characterized by high temporal density, and relevant events are typically limited to short time spans. Although some degree of long-range temporal modeling remains necessary, surgical QA primarily demands fine-grained, local reasoning due to the densely packed and event-focused nature of surgical videos. These limitations motivate an approach that prioritizes not only long-term context, but also frame-level understanding over global video reasoning.

3. Method

In this section, we will first formalize the task and introduce our motivation. We then present the general framework of our approach named SFN-ESVQA. Finally, we detail the key components of our framework, including the multi-modal feature encoding, the fusion module, and the answer generation process with the GPT-2 based decoder.

3.1. Overview

3.1.1. Challenges and Overview

Given a surgical endoscopic video and a target frame associated with a textual question "What is the surgical phase now", "what instrument used in the image", but with current existing VQA model, we will encounter these challenges.

• Limited temporal understanding. Most methods process frames in isolation or with naive sampling, failing to capture long-range temporal cues critical for interpreting surgical workflow.
• Insufficient cross-modal integration. Visual and textual features are often fused late or without adaptive weighting, which weakens the alignment between the question and relevant visual regions or frames.
• Weak control over answer relevance. Models may output syntactically valid but semantically irrelevant answers, especially when visual context is ambiguous or dominated by spurious cues.

To address these issues, our framework combines a Frame-Aware Sampling module for selecting temporally consistent key frames, a multi-modal gated fusion mechanism for adaptive video–image–text integration, and an Irrelevant Answer Regularization loss to explicitly suppress irrelevant predictions.

Figure 2. The overall architecture of our proposed SFN-ESVQA framework. It consists of a multi-modal feature encoding module, a fusion module, and a GPT-2 based answer generation decoder.

Figure 2. The overall architecture of our proposed SFN-ESVQA framework. It consists of a multi-modal feature encoding module, a fusion module, and a GPT-2 based answer generation decoder.

3.1.2. Framework

In this section, we provide an overview of the proposed framework, as illustrated in Figure 2. Given an input raw video which is shaped as $V_{\mathrm{raw}}\in {\mathbb{R}}^{N\times C\times H\times W}$,where H and W are the height and width of frames, N and C denote the number of frames and input channels.

The input raw image that we want to ask questions about $I_{\mathrm{input}}\in {\mathbb{R}}^{C\times H\times W}$ will go through the Image Encoder ${\mathsf{\Phi }}_{\mathrm{Image}}$ and encoded as $I_e\in {\mathbb{R}}^{N_v\times d}$, and it is defined as follows:

$$I_e={\mathsf{\Phi }}_{\mathrm{Image}}{\left(I_{\mathrm{input}}\right)}_{.}$$

(1)

Based on the input frame and the entire video, we propose a frame selection module with frame-aware sampling approach denoted as $\mathrm{Frame}-\mathrm{Aware}-\mathrm{Sampling}(·)$ to select some key frames and concatenate them into the selected video $V_{\mathrm{selected}}$:

$$V_{\mathrm{selected}}=\mathrm{Frame}-\mathrm{Aware}-\mathrm{Sampling}{\left(V_{\mathrm{raw}}\right)}_{.}$$

(2)

Then $V_{\mathrm{selected}}$ is passed to the Video Encoder ${\mathsf{\Phi }}_{\mathrm{Video}}$ and we could get a sequence of features from the selected video $v_e\in {\mathbb{R}}^{8\times N_v\times d}$ with d dimension:

$$v_e={\mathsf{\Phi }}_{\mathrm{Video}}{\left(V_{\mathrm{selected}}\right)}_{.}$$

(3)

After that, all these embeddings will be passed into a specially designed module named Gated module $\mathrm{GatedM}(·)$ and these two visual features are fused into Visual Embedding $V_e$:

$$V_e=\mathrm{GatedM}{(v_e,I_e)}_{.}$$

(4)

In addition to the visual pathway, we introduce a textual pathway to model the input question. Firstly, the GPT tokenizer will convert the input question $Q_{\mathrm{input}}$ into a sequence of token indices $T=(t_1,t_2,\dots ,t_n)$, where $n=30$. Each token index will be mapped to feature vector $T_t\in {\mathbb{R}}^{n\times d}$ using the Text Embedding Matrix $E_t$:

$$T={\mathrm{Tokenizer}}_{\mathrm{GPT}-2}{\left(Q_{\mathrm{input}}\right)}_{,}$$

(5)

$$T_t=\mathrm{EmbeddingLookup}{(T,E_t)}_{.}$$

(6)

Subsequently, the Visual-Guided Text Encoder (VGTE), denoted as ${\mathsf{\Phi }}_{\mathrm{VGTE}}(·)$, fuses the textual embedding and the visual embedding with a frame consisting of self-Attention layer and Frame-Aware-Attention. This process generates the final Visual-Grounded Text Embedding expressed as $T_e$, where the textual representation is grounded in the visual context:

$$T_e={\mathsf{\Phi }}_{\mathrm{VGTE}}{(V_e,T_t)}_{.}$$

(7)

We send Visual-Grounded Text Embedding into the Decoder which is composed of a stack of identical GPT-2 Blocks. Like the VGTE encoder, it also has 12 layers. What follows after is an output layer, we could get the answer A we want with it:

$$A=\mathrm{OutputLayer}{\left(\mathrm{Decoder}\left(T_e\right)\right)}_{.}$$

(8)

3.2. Understanding Surgical Video: Frame-Aware-Sampling

In order to understand the surgical video and capture temporal features, we need to choose some key frame. We proposed a Frame-Aware-Sampling approach, which relies on a specially designed score to choose Frame. The score is defined as follows:

$$s_i={\displaystyle \frac{1}{\left|N\right(i\left)\right|}}\sum _{t\in N\left(i\right)}\mathrm{SSIM}(F_i,F_t)-\alpha {{\displaystyle \frac{|i-c|}{L}}}_{,}$$

(9)

where $F_i$ is the frame $i_{\mathrm{th}}$ and $F_t$ denotes one of the frames $N\left(i\right)$ around $F_i$, We calculate their average Structural Similarity Index Measure (SSIM) as temporal consistency. After that we apply a weight $\alpha$ on its normalized distance with our current frame $I_{\mathrm{input}}$ (Input frame), where L is the length of all frames.

Then we select the top-16 frames according to the score $s_i$ , concatenate them to the selected video $V_{\mathrm{selected}}$ as follows:

$$I_{\mathrm{selected}}=\underset{i\in \mathcal{C}}{TopK}{\left(s_i,K=16\right)}_{,}$$

(10)

$$V_{\mathrm{selected}}={Concat}_{i\in I_{\mathrm{selected}}}^{\left(\mathrm{time}\right)}{F_i}_{.}$$

(11)

With this individual method to sample, our model gets a valuable temporal feature to level up the comprehension with the input image and the textual question.

3.3. Fusion for Dynamic Visual Representation: Gated Module

To effectively leverage the complementary nature of dynamic temporal context and static spatial details, we introduce a novel Gated Module.

As illustrated in Figure 2(a), the module computes a gating weight, denoted $\alpha$, which modulates the information flow of each visual stream. This allows the model to learn the optimal fusion strategy for any given input, thereby generating a more nuanced and context-aware visual representation.

The process begins by taking the previous extracted video features $v_e\in {\mathbb{R}}^{N_v\times d}$ and input image features $I_e\in {\mathbb{R}}^{N_v\times d}$, where $N_v$ are the number of patch tokens for both video and image, and d is the dimension of the feature vector. Both feature sets are first passed through a hyperbolic tangent ($\tan \mathrm{h}(·)$) activation function, normalizing them to [-1,1]. This step enhances training stability and introduces non-linearity:

$$I_e^{\prime }=tanh{\left(I_e\right)}_{,}$$

(12)

$$v_e^{\prime }=tanh{\left(v_e\right)}_{.}$$

(13)

To ensure dimensional consistency for the gating mechanism, we first apply global average pooling to video feature $v_e$, obtain single-vector representations ${\overline{v}}_e$.

The core of our module is the weight of the gate $\alpha$, which is calculated by feeding the concatenated pooled features into a small neural network followed by a sigmoid activation function. This ensures that $\alpha$ is a scalar value between 0 and 1. The computation is formalized as follows:

$$\alpha =\sigma {\left(W_g\left[{\overline{I}}_e\oplus {\overline{v}}_e\right]+b_g\right)}_{,}$$

(14)

here $\left[\right(·)\oplus (·\left)\right]$ presents the concatenation operation, $W_g$ and $b_g$ are learnable parameters. $\sigma (·)$ is a sigmoid function. The resulting $\alpha \in [0,1]$ acts like a soft, data-driven "gate".

Then two processed features are passed into the module with the gating switch $\alpha$ to produce fused visual embedding $V_e$ as follows:

$$V_e=\alpha ·v_e+(1-\alpha )·{I_e}_{.}$$

(15)

3.4. Multi-Modal Fusion: Visual-Guided Text Encoder

After we get multi-modal information, all embeddings will be passed to the Visual-Guided Text Encoder which we have mentioned above. The entire fusion encoder is composed of a stack of interleaved Self-Attention layer and Frame-Aware-Attention layer that specially designed by us.

Firstly, the input visual embedding and text embedding are processed. This step allows the model to capture the internal dependencies within the visual embeddings, capturing a contextually aware visual representation.

Following self-attention, the crucial step of cross-modal fusion occurs [26]. The refined textual embedding $T_t^{\prime }$ is used to generate the query (Q), while the fused visual embedding $V_e$ is seen as the source for the key (K) and value (V). The Frame-Aware-Attention consists of a multi-head attention mechanism, a normalization layer(LN), and a feed forward network (FFN) to continue the encoding process. It can be formulated as follows:

$$V_e^{i^{\prime }}=\mathrm{SelfAttention}{\left(V_e^i\right)}_{,}$$

(16)

$$T_c=\mathrm{MHA}{(Q=V_e^{i^{\prime }},K=T_t,V=T_t)}_{,}$$

(17)

$$V_e^{i+1}=\mathrm{LN}{\left(\mathrm{FFN}\left(\mathrm{LN}(Tc+V_e^{i^{\prime }})\right)\right)}_{.}$$

(18)

where i represents the attention layer index, ranging from 1 to 12, and $V_e^1$ and $V_e^{12}$ represent the input Visual Embedding $V_e$ and output Visual-Grounded Text Embedding $T_e$.

After passing through 12 attention layers, organized into 6 alternating rounds, we get final Visual-Grounded Text Embedding $T_e$ and it will passed to the Decoder.

3.5. Answer Generation: Decoder and Output Layer

The decoder employs a masked version of multi-head attention which is crucial for generating a coherent sequence of text one token at a time. The operation can be formalized as follows:

$$Z=\mathrm{LN}{(\mathrm{MaskedMHA}\left(T_e^{i-1}\right)+T_e^{i-1})}_{,}$$

(19)

$$T_e^i=\mathrm{LN}{\left(\mathrm{FFN}\right(Z)+Z)}_{.}$$

(20)

where i represents the block index, ranging from 0 to 11, and $T_e^0$ and $T_e^{11}$ represent the input Visual-Grounded Text Embedding $T_e$ and decoder output $T_e^{\prime }$.

The last component of the framework is an Output Layer. With a pooling operation we could get a single feature vector $\mathbf{z}\in {\mathbb{R}}^d$, and it will be sent to a classification head to get the answer:

$$\mathbf{z}=\mathrm{AvgPool}\left(T_e^{\prime }\right),\widehat{\mathbf{y}}={W\mathbf{z}+\mathbf{b}}_{.}$$

(21)

$$A=Softmax{\left(\widehat{\mathbf{y}}\right)}_{.}$$

(22)

3.6. IAR Loss: Relevance-Aware Answer Regularization

We designed a specialized loss function, called the IAR loss and it is defined as follows:

$${\mathcal{L}}_{\mathrm{IAR}}=-\left(\sum _{\begin{array}{c}i=1\end{array}}^{Class}{y}_i·logp\left(\widehat{{\mathbf{y}}_{\mathbf{i}}}\right)+\sum _{\begin{array}{c}j=1\end{array}}^{Category}{c}_j·logp\left(\widehat{{\mathbf{c}}_{\mathbf{j}}}\right)\right),$$

(23)

where $y_i$ is the one-hot encoding of the actual label. If the sample belongs to class i, then $y_i=1$, otherwise $y_i=0$, and $p\left(\widehat{{\mathbf{y}}_{\mathbf{i}}}\right)$ and $Class$ is the predicted probability of $y_i$ and number of actual label. The $c_j$ is the one-hot encoding of the category of actual label. If the actual label belongs to category j, then $c_j=1$, otherwise $c_j=0$, and $p\left(\widehat{{\mathbf{c}}_{\mathbf{j}}}\right)$ and $Category$ is the summary of predicted probability of $c_j$ and number of category of actual label, here $\widehat{{\mathbf{c}}_{\mathbf{j}}}=\sum \widehat{{\mathbf{y}}_{\mathbf{i}}}$ (label i belongs to category j ).

Compared with simply penalizing incorrect answers, this loss imposes an additional penalty on responses that are irrelevant to the question, thereby enabling our model to more effectively guard against irrelevant answers during the VQA process.

4. Experiments

4.1. Dataset Description

We evaluated our approach on two publicly available surgical VQA datasets, illustrated in Figure 3 : PitVQA [4] and EndoVis18-VQA [18], to assess generalization across procedures and data sources.

PitVQA contains endonasal pituitary surgery videos from 25 cases, recorded with an HD endoscope. Frames are sampled at 1 fps, yielding 109,173 images and 884,242 per-frame question–answer pairs across six categories: phase, step, instrument, instrument quantity, instrument position, and operation notes, for a total of 59 answer classes. We follow the official video-level split of 20 sequences for training and 5 for validation.

EndoVis18-VQA is derived from the MICCAI 2018 Robotic Scene Segmentation Challenge and consists of 14 robotic surgical video sequences (stereo, 1280×1024). Its VQA subset includes 2,007 frames and 11,783 single-word QA pairs covering 18 answer classes: organ (1), tool–tissue interactions (13), and instrument locations (4). We follow the standard split of 1,560 frames/9,014 QA for training and 447 frames/2,769 QA for validation.

4.2. Implementation Details

The implementation and pre-trained weights of our backbone networks are adopted from the official repositories of Huggingface GPT 2. Our model is trained with our proposed loss function and optimized using the Adam optimizer with a learning rate of $2\times {10}^{-5}$.

In order to obtain a comprehensive performance comparison and evaluation, we selected closely related SOTA surgical VQA models such as PitVQA-Net [4], SurgicalGPT [15] to retrain using official repositories. In addition, we adopted the results of other recent baselines, including Mutan [63], MFB [64] and MFH [65]. All experiments were conducted on a single NVIDIA RTX A6000 GPU.

4.3. Evaluation Metrics

To evaluate our model’s capability, we selected some typical metrics such as F1-score (F1), Balanced Accuracy (B.Acc) and Recall (Rec).

• F1-score:F1 summarizes precision and recall as their harmonic mean, where ${\mathrm{Prec}}_c={\displaystyle \frac{T{P}_c}{T{P}_c+F{P}_c}}$, ${\mathrm{Rec}}_c={\displaystyle \frac{T{P}_c}{T{P}_c+F{N}_c}}$. It is defined as follows:

  $$\mathrm{F}{1}_{\mathrm{macro}}={\displaystyle \frac{1}{C}}\sum _{c=1}^C{{\displaystyle \frac{2{\mathrm{Prec}}_c{\mathrm{Rec}}_c}{{\mathrm{Prec}}_c+{\mathrm{Rec}}_c}}}_{.}$$

  (24)
• Balanced Accuracy:B.Acc averages per-class recall to reduce the effect of class imbalance, where C is the number of classes, $c\in \{1,\dots ,C\}$ indexes classes, $T{P}_c$ and $F{N}_c$ denote the true positives and false negatives for class c under a one-vs-rest view. This metric is defined as follows:

  $${\mathrm{BA}}_c={\displaystyle \frac{{\mathrm{TPR}}_c+{\mathrm{TNR}}_c}{2}}={\displaystyle \frac{1}{2}}\left({\displaystyle \frac{T{P}_c}{T{P}_c+F{N}_c}}+{\displaystyle \frac{T{N}_c}{T{N}_c+F{P}_c}}\right),\mathrm{B}.\mathrm{Acc}={\displaystyle \frac{1}{C}}\sum _{c=1}^C{\mathrm{BA}}_c.$$

  (25)
• Recall:Recall measures how well the model retrieves relevant positives:

  $${\mathrm{Rec}}_c={\displaystyle \frac{T{P}_c}{T{P}_c+F{N}_c}}.$$

  (26)
• Irrelevant Answering Rate:In addition, we introduced a special evaluation metric named Irrelevant Answering Rate (IAR). This metric measures the proportion of irrelevant answers among all incorrect responses. An answer is deemed irrelevant if its semantic category does not match the question’s expected answer type. IAR is calculated as:

  $$\mathrm{IAR}={{\displaystyle \frac{N_{ir}}{N_{ir}+N_{wr}}}}_{,}$$

  (27)

  where $N_{ir}$ refers to these generated answers which falls outside the expected answer category, and $N_{wr}$ denotes the number of answers that match the expected answer type yet but contain incorrect content.

These metrics offer a multi-faceted performance evaluation: (1) the harmonic mean of precision and recall (F1-score), (2) robustness against class imbalance (Balanced Accuracy), and (3) the ability to identify all relevant instances (Recall), (4) the capacity to maintain semantic consistency by avoiding irrelevant answers (IAR).

4.4. Results: SFN-ESVQA Versus SOTA Baselines

Table 1 presents the quantitative results of our proposed method compared to previous SOTA models on EndoVis18-VQA and PitVQA datasets. We particularly emphasize the metric of B. Acc to evaluate the robustness of model generation across imbalanced classes. Our method achieves a significant improvement in balanced accuracy, reaching 80.34% on EndoVis18-VQA and 86.05% on PitVQA, outperforming the previous best method PitVQA-Net by margins of 35.28% and 27.83%, respectively. In terms of F1 and Recall, our model also shows consistent superiority, showing respective improvements of 25.44% and 35.01% on EndoVis18-VQA and 29.86% and 30.64% on PitVQA.

Similar trends can be observed across all metrics, indicating that our method generalizes better and produces more accurate and stable predictions.

Table 2 further compares the Irrelevant Answer Rate (IAR) performance among these models. Our method achieves the lowest IAR values, with 1.39% on EndoVis18-VQA and 1.81% on PitVQA, reflecting its stronger ability to capture temporal features so that it could understand the whole video, avoiding to generate irrelevant answers. Compared to PitVQA-Net, our method reduces the IAR by 15.91% and 16.75% on EndoVis18-VQA and PitVQA, respectively.

As illustrated in Figure 4, we observed that most of the related models perform better in location recognition. However, irrelevant answers still occurs across many cases to some extent, particularly regarding questions about the surgical status. Whereas, our proposed SFN-ESVQA demonstrates robust performance across various types of question answering in both datasets.

Overall, the results suggest that our SFN-ESVQA not only improves the behavior but also enhances rubustness in the VQA task for surgical scenarios.

4.5. Ablation Study

To assess the contribution of each component in our design, we perform an ablation on EndoVis18-VQA and PitVQA under the same training protocol, focusing on two factors: the frame-aware module and the IAR loss. We compare four variants: (a) baseline without either component, (b) adding the frame-aware module only, (c) adding the IAR-loss only, and (d) using both. As reported in Table 3, introducing either component improves performance, while combining them yields the best results. These results indicate that the two components are complementary: each brings consistent gains, and together they deliver the strongest overall accuracy with the lowest IAR.

Except for the standard ablation studies, we further investigated how different hyperparameter settings of our backbone affect the behavior of the VQA process on datasets EndoVis18-VQA and PitVQA respectively, particularly focusing on IAR and B.Acc.

• Block Depth:Figure 5(a) shows the influence of block depth in the Frame-Aware Attention module. We varied this hyperparameter from 6 to 12. The results indicate that both IAR and B.Acc steadily improve across both datasets, demonstrating that increasing the model capacity leads to better fusion performance. This confirms the effectiveness of a deeper backbone for SFN-ESVQA.
• Frame Length:Figure 5(b) illustrates the effect of different frame lengths, which control the temporal context window between the raw frames and the input question. While performance initially improves with an increase on frame length, we observed a decline beyond 16 frames. However, considering the overall trend, the model still performs robustly, suggesting that overly long sequences may introduce redundancy or noise.
• Sampling Approach:Figure 5(c) and Figure 5(d) illustrate the results of several frame sampling strategies, including Random (selecting arbitrary frames), Consecutive (choosing temporally adjacent frames around raw frame), Uniform (evenly spaced sampling from 1600 frames), and VideoTree [66] (hierarchical temporal coverage). In contrast, our proposed Frame-Aware method adaptively selects key frames based on temporal consistency and proximity to the input, allowing SFN-ESVQA to better capture surgical dynamics. This leads to improved comprehension of the video context and reduces the generation of irrelevant answers.

5. Discussion and Conclusions

This study explores the potential postoperative review and training utility of the proposed frame-aware VQA model SFN-ESVQA. Our experimental results demonstrate a clear performance advantage for SFN-ESVQA, achieving state-of-the-art results on both the PitVQA dataset and the publicly available EndoVis18-VQA dataset when compared to existing surgical VQA models. We hope that our work will spark interest in the challenging problem of VQA in the surgical domain. Additionally, we demonstrated a major issue of the high rate of irrelevant answers in the previous models, and in our model, we have reduced Irrelevant Answer Rate to a very low level. However, the clinician oversight will still be crucial for ensuring reliable outcomes in clinical practice no matter how accurate the model is.