A SAM2-Driven RGB-T Annotation Pipeline with Thermal-Guided Refinement for Semantic Segmentation in Search-and-Rescue Scenes

1. Introduction

Rescue robots must navigate through unfamiliar environments that often have low visibility, degraded sensing conditions, and moving targets. In these situations, reliable object segmentation is crucial for tasks such as simultaneous localization and mapping (SLAM) [1], modeling disaster sites [2], detecting people in need of assistance [3] or missing people [4], and maintaining situational awareness during catastrophic events [5]. Furthermore, to improve robustness under these adverse perception conditions, the combination of RGB and thermal infrared (TIR) imagery is becoming increasingly relevant [6,7,8].

Despite this growing relevance, multimodal RGB and TIR datasets are limited in availability. In the autonomous driving domain, the FLIR-ADAS dataset [9] includes both RGB and TIR (RGB-T) images, along with bounding box annotations for object detection. Similarly, the MFNET dataset [10] contains RGB and TIR images captured during daytime and nighttime, accompanied by semantic segmentation labels. Additionally, the PST900 dataset [11] focuses on daily activity scenes and also contains RGB-T annotated images. In the search-and-rescue (SAR) domain, several datasets include both RGB and TIR modalities; however, most of them provide only bounding box annotations [6,7,8]. In contrast, RescueNet [12] and DRespNeT [13] offer multi-class semantic segmentation annotations for SAR scenarios, although they are limited to RGB imagery, highlighting both the scarcity of multimodal semantic labels and the challenges involved in generating them.

Manually creating semantic segmentation annotations is a time-consuming and labor-intensive task, and the resulting labels are often inconsistent due to human errors. Furthermore, the procedures for generating these datasets are often insufficiently described, limiting reproducibility and the systematic assessment of annotation quality. In particular, RGB–TIR datasets pose a fundamental technical challenge: achieving reliable cross-modal alignment. While standard planar calibration patterns are effective for RGB cameras, calibration in the thermal domain is often problematic due to limitations of heated patterns, such as uneven temperature distribution, low geometric definition, and thermal diffusion, which in turn complicates the establishment of accurate point correspondences between modalities.

Recently, there has been increasing attention within the community toward developing tools and workflows that reduce the time and cost of annotation, positioning human intervention primarily in a supervisory role [14,15,16,17,18,19]. In particular, promptable models such as SAM2 (Segment Anything Model v2) [20] provide a flexible mechanism for generating initial segmentation masks across a wide range of image scenes, enabling semi-automatic workflows in which human annotators review and refine automatically generated masks. Unlike video object segmentation (VOS) models [21], SAM2 employs a unified architecture for both image- and video-based segmentation tasks. Its transmission memory mechanism supports zero-shot segmentation, reducing annotation time, and requires less user interaction compared to previous methods, such as Open-Vocabulary models [22]. Related approaches to our proposal, such as MATT [23], use SAM-generated masks on RGB images and transfer them to other modalities, including multispectral or thermal imagery. While these methods reduce annotation time, they often depend on well-aligned image pairs and do not incorporate systematic cross-modal alignment, annotation quality metrics, or human-in-the-loop reliability assessment, leaving a gap for more general, reproducible pipelines.

Inter-annotation agreement (IAA) is a critical metric that quantifies the degree of consistency with which multiple human annotators label the same data, statistically accounting for agreement by chance. Originally established in fields such as psychology and medicine, IAA metrics have been adopted in computer vision fields, including medical [24], agricultural [25], and transportation [26], to evaluate the quality of semantic segmentation ground truth. In the context of RGB-T semantic segmentation, widely used benchmark datasets such as MFNet and PST900 have enabled rapid comparable models. However, insufficient quality control can have major implications for both the reproducibility of the findings and the interpretation of the results [27]. Inconsistency annotations can artificially inflate or suppress quantitative metrics, thereby hiding real model improvements. Consequently, similar to medical imaging applications [28], semantic segmentation annotations in RGB-T datasets should be regarded as accurate and reproducible as possible.

Thus, to address these limitations, we propose a semi-automatic annotation methodology that combines cross-modal alignment, explicit quality checks (QC), human efficiency and reliability evaluation, and downstream validation. The main contributions of this article are:

• We propose a RGB-T Pipeline for Dataset Semantic Segmentation that integrates a cross-modal alignment method, which has been implemented for RGB and TIR modalities.
• We benchmark two annotation strategies: a fully manual polygon-based annotation baseline against our semi-automatic annotation built around SAM2 as ML backend model. We evaluate both methodologies by measuring IAA for label Quality Checks (QC) and human annotation timing, as well as conducting an inter-rater reliability study.
• We applied the proposed pipeline to generate 306 annotated image pairs from a SAR-oriented RGB-T dataset [29]. The labeled data were then used to train two state-of-the-art RGB-T semantic segmentation models. Both, the data and trained models are publicly available at https://github.com/amsalase/CPGFANet.

The remainder of this paper is organized as follows: Section 2 reviews related literature. Section 3 provides a comprehensive overview of the proposed pipeline, detailing its procedural steps. Section 4 presents the SAR dataset, implementation details, and evaluation metrics for analyzing quality, cost, and IAA. Section 5 presents the results on annotation cost, quality, IAA, and dataset validation. Section 6 summarizes findings and outlines some discussion points. Finally, Section 7 offers conclusions.

2. Related Work

Recent work in semantic segmentation has proposed task-specific annotation pipelines tailored to constrained scenarios, such as degraded imagery, remote sensing, or medical imaging. In [15], the authors propose a two-step training pipeline for semi-supervised semantic segmentation in degraded images captured under adverse conditions such as nighttime or fog. This approach decouples the training of labeled and unlabeled images to mitigate overfitting in extremely limited datasets, but its assumptions limit applicability to unstructured scenarios. In [16], SAM2 is combined with an Enhancement and Labeling Network (ELNet) to accelerate annotation in remote sensing segmentation tasks, with experiments limited to two-class, single-modality images. A systematic framework for semantic segmentation in satellite images is presented in [17]. This work provides a guide for efficiently and systematically exploiting artificial intelligence (AI) tools, including segmentation models, data preparation, model selection, and validation. However, this approach is tailored to a specific structure in satellite images. Finally, in [18], a fully automated framework is presented as an end-to-end pipeline to generate semantically enriched BIM-compatible 3D reconstructions from a single RGB image, jointly performing depth estimation and semantic segmentation. This framework is designed for structured environments, limiting its generalization to unstructured scenes.

Beyond task-specific pipelines, other works explore multi-step or interactive workflows for annotation. In [30], an iterative annotation scribble-based method is proposed for segmenting urban city scenes based on scribbles. This iterative process involves obtaining an initial segmentation, which requires at least two experienced annotators to refine it through user-provided corrections and back-propagation. In [19], a method for automatic segmentation and labeling is proposed based on a sequence of ground extraction (via RANSAC) followed by a coarse-to-fine segmentation strategy in urban scenarios. Moreover, the MATT workflow [23] uses SAM to perform segmentation on aerial RGB images and transfer these annotations to co-aligned multi-spectral images, thereby defining an automated segmentation labeling process. While this work is the only one proposing label transfer between modalities, it assumes that the images are co-aligned and synchronized, where the cameras share common features such as field of view (FOV) and image resolution. On the other hand, in [31], a novel network architecture enhances building segmentation by addressing multi-scale feature fusion and foreground perception, without encompassing annotation in a pipeline.

These approaches aim to reduce manual annotation effort through semi-supervised methods or iterative refinement, demonstrating a growing interest in developing efficient and reliable annotation pipelines, which are crucial for scaling semantic segmentation to diverse and complex scenarios. However, none address cross-modal alignment, inter-annotator agreement, or human-in-the-loop reliability assessment, and few studies integrate these elements into a standardized and generalizable pipeline.

3. SAM2-Driven RGB-T Annotation Pipeline

Figure 1 summarizes the proposed methodology for semantic annotation of multimodal RGB-T imagery in SAR scenarios. The pipeline is designed to reduce manual annotation effort while preserving spatial and semantic accuracy. With this purpose, the methodology relies on the structured integration of homography-based cross-modal alignment without explicit thermal camera calibration, semi-automatic semantic annotation using SAM2, and manual refinement supported by thermal imagery. This work contributes a reproducible workflow that combines existing technologies in a structured manner, with explicit alignment and quality control. Detailed implementation aspects are provided in Section 4.

The proposed pipeline consists of four stages:

1.

Cross-Modal Camera Aligmnent. RGB and TIR images are geometrically aligned by estimating a planar homography from image correspondences. This alignment enables warping the RGB image into the TIR modality and facilitates label transfer across modalities, thereby avoiding the need for thermal-camera calibration using heated patterns, which is often problematic due to their limited geometric definition compared to standard RGB calibration targets. It is well known that four-point correspondences between images are sufficient to compute an exact homography matrix H [32], which constitutes the minimal solution for robust estimators such as RANSAC. However, in cross-modal RGB–T settings, establishing reliable correspondences is challenging due to significant appearance differences between modalities [33,34]. To address this, rather than relying solely on hand-crafted feature matching followed by RANSAC, we adopt a learning-based feature matching approach using the SuperGlue model [35], which jointly estimates keypoint correspondences and their confidence. This approach provides a robust mechanism for cross-modal alignment while preserving the conceptual connection to classical robust estimation methods. In this stage (see Figure 1), the annotator selects a reference RGB–T pair from a fixed set of cameras, choosing a scene that contains a sufficient number of objects of interest to ensure reliable keypoint estimation and a well-conditioned homography matrix H. If the resulting alignment is qualitatively unsatisfactory, the annotator repeats the process with an alternative reference pair.

2.

Annotation Taxonomy. A fixed, domain-specific annotation taxonomy is defined in SAR scenarios to constrain the labeling process and ensure semantic consistency across annotators.

3.

Semi-automatic annotation. The proposed semi-automatic annotation methodology for RGB–T images leverages the complementary strengths of automatic and manual annotation using tools such as the open-source Label Studio [36]. Initial segmentation masks are generated from RGB images, as these images provide higher spatial resolution, richer texture, and clearer object boundaries than TIR images. During the SAM2 semi-automatic step, annotators interact with the Label Studio SAM2 ML backend using positive clicks or bounding boxes to prompt segmentation. The automatic masks are produced using the SAM2 model, which generates class-specific segmentation proposals based on user-provided prompts. If the automatically generated masks are not satisfactory (e.g., missing or inaccurate classes), annotators refine them manually, creating a semi-automatic annotation process that combines speed with control over quality. This manual refinement on RGB-generated masks is performed using polygon annotation tools such as PolygonLabels in Label Studio.

4.

Annotation refinement. The RGB masks are then transferred to the corresponding TIR images via the homography H estimated in Section 4.2. To capture classes that are poorly visible or invisible in the RGB modality, the masks are further refined directly on the TIR images using the same polygon-based annotation strategy. The annotators manually correct residual misalignment, remove spurious islands, and fill holes not properly captured by SAM2, ensuring high-quality segmentation masks.

4. Experimental Methodology

In this work, we quantitatively evaluate annotation efficiency and annotation quality using annotation time and inter-annotator agreement metrics, respectively, enabling an objective comparison of annotation strategies within the proposed framework. This section presents the experimental setup for evaluating our semi-automatic annotation methodology for RGB–T images, including a description of the SAR dataset and the segmentation models used, the cross-modal camera alignment procedure for ensuring geometric consistency between RGB and TIR images, and the annotation cost and quality metrics.

4.1. SAR Dataset and Model Details

We use the proposed pipeline to annotate 306 RGB and TIR image pairs from the UMA-SAR dataset [29]. The image pairs were captured with monocular TIR and RGB analog cameras. Both images have the same resolution (704×576) and are mostly overlapped, but with different horizontal FOVs for TIR (${44}^{\circ }$) and RGB ($57.8^{\circ }$). Three representative examples are shown in Figure 2a,b.

For annotation, we adopt the eleven representative classes in SAR imaging defined in [37]: First-Responder, Civilian, Vegetation, Road, Dirt-Road, Building, Sky, Civilian-Car, Responder-Vehicle, Debris and Command-Post.

With the resulting dataset, we trained two RGB-T models: FEANet [38] and SAFERNet (with ResNet–152 as backbone) [39]. All instance masks, exported as Label Studio JSON format, including polygon and RLE encodings, were converted into class index maps (0–11) for training.

The models were implemented in PyTorch toolbox and trained on a NVIDIA RTX A6000 with CUDA version of 12.6. In terms of data augmentation, all training images were flipped and cropped. We adopted stochastic gradient descent (SGD) with a learning rate of 0.003, a momentum of 0.9, and a weight decay of 0.0005. The learning rate was multiplied by a decay rate of 0.95.

Moreover, to assess annotation quality and consistency, we leverage the validated annotations of the benchmark MFNet dataset [10], a multimodal RGB–T dataset designed for object segmentation in urban scenes. The dataset contains both RGB and thermal images captured under daytime and nighttime conditions. For our experiments, we selected 15 daytime and 15 nighttime scenes for the annotators. MFNet defines eight semantic segmentation classes: car, person, bike, curve, car stop, guardrail, color cone, and bump.

4.2. Cross-Modal Alignment Configuration

Keypoints are detected and matched using the SuperGlue Outdoor model with a confidence threshold of 0.2, which is the default value. The TIR image was used as the reference modality, since its FOV is more restrictive than that of the RGB image. Matched keypoints are then post-processed to recover their original coordinates, and a projective transformation (homography) is estimated using OpenCV’s findHomography. Finally, the resulting $3\times 3$ homography matrix H is applied to all SAR RGB–T images to warp the RGB images into the TIR view. Nearest-neighbor interpolation is used during warping to preserve label integrity.

4.3. Annotation Cost and Annotation Quality

In this work, we quantitatively evaluate annotation quality and annotation efficiency by benchmarking two annotation methods: a baseline manual polygon-based annotation with PolygonLabels in Label Studio against the implementation of our semi-automatic approach based on SAM2. This evaluation is conducted by measuring accuracy, annotation timing, and IAA metrics, thereby enabling an objective comparison of the strategies within the proposed framework.

To evaluate annotation consistency and efficiency, two human annotators independently applied both methods to two separate image sets: 30 images from the MFNet dataset (for consistency against the ground truth) and 14 SAR images (for annotation time efficiency). Also, their agreement (IAA) was measured on the 14 SAR images. Annotations were performed in matched batches and followed the same class order for both annotators, ensuring consistent annotation conditions across strategies. The annotators differed in experience: Annotator A has over 5 years of experience with semantic segmentation and greater familiarity with the Label Studio annotation tool, whereas Annotator B has approximately 1 year of experience.

Regarding costs, the annotation time per image (in seconds) was recorded automatically in Label Studio. This metric was computed and compared in three annotation strategies: (1) Manual polygon annotation on RGB images, (2) SAM2 semi-automatic annotation on RGB only, (3) the proposed full SAM2 semi-automatic annotation on RGB-T imagery.

To evaluate annotation quality, an inter-annotation agreement technique was used by two annotators using class-wise Intersection over Union ($IoU$) and Cohen’s $\kappa$:

a)

Intersection over Union: The $IoU$ computes pixels where both masks belong to a specific class over pixels where at least one mask belongs to a specific class:

$$IoU={\displaystyle \frac{|M_A^C\cap M_B^C|}{|M_A^C\cup M_B^C|}}$$

(1)

where $M_A^C$ are the pixels annotated as class c by annotator A, and $M_B^C$ are the pixels annotated as class c by annotator B.

b)

Cohen’s $\kappa$: This inter-annotator agreement measures the flexibility between two annotators while correcting for agreement expected by chance. For two segmentation maps, $\kappa$ is defined as:

$$\kappa ={\displaystyle \frac{p_o-p_e}{1-p_e}}$$

(2)

where $p_o$ represents how often the two annotators assign the same class to a pixel, and $p_e$ is the probability that both annotators would choose the same class to a pixel if they both just guess randomly.

Together, these metrics quantify not only the efficiency provided by SAM2 and the thermal modality, but also the consistency and reliability of the resulting segmentation labels.

5. Results

5.1. Cross-Modal Camera Alignment

Figure 2 illustrates the complete alignment RGB-T image registration, showing the original RGB and TIR images, the RGB images warped using the homography matrix estimated by the SuperGlue model, and the fused RGB-T image.

The captures were done with slightly different relative camera poses. Therefore, three different homography matrices have been computed following the procedure given in Section 4.2. The results correctly capture the different misalignments, as can be appreciated in column (c) of Figure 2.

Column (d) presents the fused RGB-T images, which combine the warped RGB data with the TIR data. This column visually demonstrates the effectiveness of the homography matrix. Elements such as buildings, people, roads, and dirt road boundaries exhibit strong spatial correspondence, with only minor residual misalignments due to the planar assumption in homography estimation.

All in all, these results evidence robust image correspondences in the UMA-SAR dataset with different camera FOVs and misalignments. For the rest of the annotation pipeline analysis, as illustrated in Figure 1 and Figure 3, all RGB-T image pairs were aligned using this cross–modal homography estimation to ensure pixel-level correspondence.

5.2. Annotation Quality

Since the RGB-T modalities for SAR scenes lack a publicly available semantic segmentation dataset for comparison, we first evaluated the annotation quality of Annotator A and Annotator B using 30 MFNet [10] validation images (15 day and 15 night scenes) with the manual polygon method. The selected subset balanced annotation feasibility with representative scene diversity, which is a common practice in annotation-cost and agreement studies. A demonstrated around 5% higher annotator consistency ($mIoU$: 66% vs. 63% for B). This result may be influenced by differences in annotator experience, with Annotator A showing stricter adherence to the annotation protocol, particularly in boundary definition.

Subsequently, we evaluated the absolute segmentation quality of the annotator A against MFNet ground-truth with the same data as before (30 MFNet images). However, we evaluated the results using three strategies: (1) manual RGB polygon, (2) SAM2-assisted RGB, (3) SAM2-assisted RGB-T refinement. This enables us to assess whether AI assistance enhances segmentation quality compared to official ground-truth annotations from MFNet.

Table 1 reveals that SAM2–assisted annotations improve the segmentation quality for our annotator A. Compared to ground-truth, manual annotation achieved 65.9%, while SAM2-assisted methods increase this percentage by 3.6% in SAM2-assisted RGB and 9% in SAM2-assisted RGB-T. Therefore, these data demonstrate that: (1) AI assistance can enhance annotation quality rather than compromising it. (2) to improve annotation quality a modality refinement could be an important technique.

Figure 4 provides qualitative comparisons that corroborate the insights of Table 1. Visually, SAM2 assisted methods (e and f) produce a segmentation mask with better boundary definitions that are close to the ground–truth data. In contrast, the manual polygon (d) shows approximated boundaries, justifying why manual annotation has a lower $mIoU$

5.3. Annotation Cost

Table 2 summarizes the annotation cost across three strategies. Friedman non-parametric test shows a significant global effect (${\chi }^2\left(2\right)=18.04$, $p<0.001$. Post-hoc pairwise comparison (Wilcoxon test with Bonferroni correction, $\alpha =0.0167$) indicates that incorporating SAM2-based methods into the annotation pipeline significantly reduced the mean annotation time relative to manual annotation ($p<0.001$ for both). Specifically, SAM2 (RGB-T) with refinement achieved a saving time of around 180 seconds (3 minutes per image). Overall, the results confirm that the SAM2 assistance, even when requiring refinement, substantially reduces the time cost of generating segmentation ground-truth without compromising annotation quality (see Section 5.2).

5.4. Inter-Annotation Agreement

Table 3 reveals a critical interaction between annotator expertise and SAM2 assistance. The expert annotator (A) maintained almost perfect agreement with SAM2 as the semi-automatic annotation methodology ($\kappa \approx 0.83$), demonstrating the ability to integrate AI assistance without altering annotation quality. In contrast, the less experienced annotator (B) showed approximately 0.10 points less in the agreement metric compared to the expert reference when using SAM2 ($\kappa$ from $0.84$ to $0.73$). This decrease, accompanied by increased variability ($\mathit{SD}=0.14$ vs. $0.10$), highlights the sensitivity of chance-corrected metrics to annotator variability (i.e. that novice annotators may accept an AI proposal with less critical refinement) and class imbalance. The inclusion of thermal refinement does not affect agreement levels.

5.5. Comparative Analysis

Here, we conduct a comprehensive analysis of segmentation fidelity, employing both quantitative and qualitative visual analysis in two SOTA models, on the augmented UMA-SAR dataset.

Table 4 reports a quantitative benchmark of two SOTA RGB-T models on the UMA–SAR test set. Overall, SAFERNet achieves clear superiority in the two metrics (i.e. a substantial improvement of +8.77 percentage points on $mAcc$ and +7.69 on $mIoU$), which indicates enhanced global segmentation fidelity. At the class level, the improvements are particularly pronounced in critical classes such as: Road, Dirt-Road, Command Post, and crucially detects Civilian-Car, where FEANet completely fails. The results reflect that SAFERNet offers better spatial consistency and reduced misclassifications. Conversely, FEANet shows competitive performance in Vegetation, Building, and Debris. Although both methods perform comparably on dominant classes such as Sky and First-Responder, FEANet exhibits significant weaknesses in rare categories, most notably the complete failure on Civilian-Car. These findings suggest that SAFERNet is more robust to class imbalance and more effective at capturing complex multimodal cues present in SAR environments.

The qualitative comparison presented in Figure 5 visually corroborates the quantitative trends observed in Table 4. In particular, SAFERNet shows superior performance in semantically detecting Road (third row) and Command-Post (fourth row). Furthermore, both FEANet and SAFERNet demonstrate the ability to detect low-contrast objects that are barely distinguishable in RGB but salient in TIR domain, highlighting the benefits multimodal fusion.

6. Discussion

This work has introduced and validated a comprehensive methodology for generating high-quality, reliably annotated RGB-T semantic segmentation datasets, specifically search-and-rescue (SAR) applications. The proposed annotation methodology integrates cross-modal alignment, human-in-the-loop reliability assessment, and quality checks in a RGB-T semantic segmentation framework. The utility of the resulting annotated dataset was validated by successfully training two state-of-the-art models, demonstrating the practical value of the pipeline.

The cross-modal alignment technique leverages feature-based homography estimation to ensure consistent spatial correspondence. The resulting spatial coherence visually enhances the alignment across various modalities and facilitates better integration of data from multiple sources, such as different cameras. By integrating AI (SuperGlue) to obtain homography estimation, we provide a robust and practical solution for registering RGB and TIR imagery in real-world unstructured SAR scenes.

Additionally, the proposed method successfully compensated for mechanical misalignment, resulting in spatially coherent RGB-T pairs that are suitable for fusion and annotation. The key advantage of our method is its flexibility over rigid calibration procedures, and it can be applied to future developed datasets. Homography relies on the assumption of local planarity, which limits its effectiveness for close objects in the scene, potentially leading to minor residual misalignment. Future work may explore non-rigid registration methods.

The integration of SAM2 into the annotation pipeline provides a favorable balance between annotation quality, cost, and inter-annotation agreement (i.e. simultaneously enhances output quality: $mIoU=65.9\%+9\%$ and reduces time cost with an improvement of $21\%$). However, the benefits are related to annotator expertise. The expert annotator maintained almost perfect agreement when using SAM2, indicating effective human–AI interactions. In contrast, the novice annotator showed a reduced agreement under AI assistance, highlighting the influence of annotator expertise and suggesting a propensity for less critical acceptance of AI suggestions. This suggests that AI-assisted protocols benefit from the expertise of expert annotators and expert review to ensure label integrity.

Finally, the practical utility of the resulting curated dataset was unequivocally demonstrated by its successful use in training and benchmarking state-of-the-art RGB–T segmentation models. This confirms that the annotations are of sufficient fidelity and consistency for downstream model development.

Overall, the results indicate that SAM2-based pipelines combined with well-defined protocols can accelerate dataset creation while preserving annotation quality. While the planar homography model introduces minor residual errors, the visual and quantitative coherence of the outputs confirms its adequacy for segmentation tasks.

Very recently, SAM3 was presented [40]. SAM3 is an evolution from single-object visual segmentation to promptable concept segmentation (PCS), enabling the detection and tracking of all instances of a concept in images using text phrases, image samples, or a combination of both. As future work, integrating SAM3 into the proposed pipeline could further enhance the labeling process and user interaction by leveraging its agentic workflow features, potentially combined with multimodal large language models (MLLMs) for complex reasoning-based labeling.

7. Conclusions

This paper presented a reproducible framework for generating reliable RGB-T semantic segmentation labels through a semi-automatic pipeline. The proposed approach integrates SAM2 as a machine-learning backend model to reduce manual annotation effort, while a cross-modality alignment method enables label transfer and refinement across RGB and TIR modalities. This alignment method, based on planar homography estimation, allows image warping under camera misalignment and eliminates the need for explicit TIR camera intrinsic calibration, which is often complex and inaccurate.

The user interaction within the proposed pipeline relies on polygon-based annotation. We evaluated two annotation strategies: a fully manual polygon-based annotation as a baseline and our proposed semi-automatic annotation process. Annotation cost and quality were assessed using inter-annotation agreement (IAA) for label quality checks and human annotation time. The results show a $9\%$ improvement in annotation quality and a $21\%$ reduction in annotation cost when using the proposed pipeline.

The proposed pipeline was further employed to generate 306 annotated RGB-T image pairs for a SAR-oriented dataset. This labeled data was used to train two state-of-the-art RGB-T semantic segmentation models, validating the consistency and fidelity of the generated annotated dataset. Both the data and the trained models are made publicly available.

Future work will focus on extending the proposed framework to include additional image modalities, such as depth and multispectral imagery. The recent release of SAM3 represents a significant improvement that can be integrated into this pipeline.