SDRFPT-Net: A Spectral Dual-stream Recursive Fusion Network for Multispectral Object Detection

1. Introduction

Object detection, as a core technology in computer vision, has extensive application value in fields such as autonomous driving, security monitoring, and remote sensing analysis [1,2,3]. As environmental conditions become increasingly complex and variable, traditional single-modality detection systems face severe challenges. While visible images can provide rich color and texture information, they perform poorly in adverse weather or insufficient lighting conditions [4]. In contrast, infrared images can capture thermal information radiated by objects, performing excellently in nighttime and low-light environments, but they have lower resolution and less clear edge information [5]. Therefore, multispectral object detection technology that comprehensively utilizes the complementary modalities of visible and infrared light has become an effective approach to solving this problem [6,7].

In recent years, deep learning technology has driven the rapid development of multispectral object detection research. With the advancement of convolutional neural networks (CNNs) and object detection algorithms, detection frameworks such as the YOLO series [8,9,10] and Faster R-CNN [11] have demonstrated powerful feature extraction and object recognition capabilities. However, when facing multimodal data, how to effectively fuse complementary information from different spectral domains and suppress background interference becomes a key challenge [12]. Existing methods mainly adopt simple feature concatenation or element-wise addition/multiplication fusion strategies [13], which ignore the intrinsic correlation between different modalities and find it difficult to fully utilize the complementary advantages of multimodal data.

To address the above issues, this paper proposes the Spectral Dual-stream Recursive Fusion Perception Target Network (SDRFPT-Net), a novel multispectral object detection architecture designed to effectively integrate visible and infrared modal information to improve detection performance in complex environments. Unlike existing methods, SDRFPT-Net innovatively proposes a Spectral Hierarchical Perception Architecture (SHPA) based on YOLOv10, providing a solid foundation for multimodal feature extraction, and achieves deep feature interaction and efficient fusion through the Spectral Recursive Fusion Module (SRFM), finally using the Spectral Target Perception Enhancement Module (STPEM) to enhance target region representation and suppress background interference.

The SHPA module adopts a dual-stream structure to process visible and infrared spectral information separately, capturing modality-specific features through independent parameter networks to maintain the integrity of spectral information. The SRFM module is the core innovation of this architecture, achieving deep feature interaction through a hybrid attention mechanism (integrating self-attention, cross-modal attention, and channel attention), and adopting a recursive progressive fusion strategy to achieve deep multi-modal feature interaction while maintaining parameter efficiency. The STPEM module focuses on enhancing target regions in features, significantly improving the detection capability of low-contrast targets through lightweight mask prediction and feature enhancement mechanisms.

Compared to traditional multispectral fusion methods, our proposed spectral dual-stream recursive fusion perception architecture has three significant advantages: First, it can more effectively capture complementary information between different modalities, showing excellent performance especially in low-light, adverse weather, and other complex environments; Second, the recursive progressive fusion strategy achieves deep feature interaction without significantly increasing the parameter count, improving computational efficiency; Finally, the target perception enhancement mechanism effectively distinguishes targets from backgrounds, improving detection accuracy and robustness.

To verify the effectiveness of SDRFPT-Net, we conducted extensive experiments on two multispectral object detection benchmark datasets: FLIR-aligned [14] and LLVIP[15]. The results show that SDRFPT-Net achieved state-of-the-art performance on both datasets, particularly with significant improvements over existing methods in the mAP50 and mAP50:95 metrics. For example, on the FLIR-aligned dataset, SDRFPT-Net achieved an mAP50 of 0.785, an 11.5% improvement over the second-best performing BA-CAMF Net (0.704); on the LLVIP dataset, the mAP50 reached 0.963, and the mAP50:95 reached 0.706, achieving optimal performance.

The contributions of this paper can be summarized as follows:

(1)

Propose SDRFPT-Net, a novel multispectral object detection architecture that effectively extracts and integrates multimodal features through a dual-stream separated spectral structure;

(2)

Design the Spectral Recursive Fusion Module (SRFM), achieving high-efficiency deep feature interaction through a hybrid attention mechanism and recursive progressive fusion strategy;

(3)

Develop the Spectral Target Perception Enhancement Module (STPEM), enhancing target feature representation and suppressing background interference;

(4)

Experimental validation of SDRFPT-Net's effectiveness on multiple public datasets, achieving state-of-the-art detection performance while maintaining computational efficiency;

The remainder of this paper is organized as follows: Section 2 reviews related work; Section 3 introduces the architecture and key modules of SDRFPT-Net in detail; Section 4 presents experimental results and analysis; Section 5 concludes the paper and indicates future research directions.

2. Materials and Methods

In this section, we review multispectral object detection, feature fusion strategies, and the application of YOLO series algorithms in multispectral object detection.

2.1. Multispectral Object Detection

To overcome the limitations of single-modality imaging, multispectral object detection technology achieves stable monitoring in all weather and all time conditions by fusing complementary information from different spectral bands [6,7]. Early multispectral fusion methods mainly used traditional mathematical models, such as multi-scale transformation [16], sparse representation [17], saliency-based methods [18], and subspace decomposition [19]. However, these methods often rely on manually designed feature extractors and fusion rules, making it difficult to adapt to complex and variable scenes.

With the development of deep learning, CNN-based multispectral object detection methods have made breakthrough progress. Liu et al. [20] proposed a multispectral deep neural network for pedestrian detection, improving detection performance by learning the correlation between different modalities. Wagner et al. [21] designed a deep fusion convolutional neural network, achieving effective combination of visible and infrared images. König et al. [22] developed a fully convolutional region proposal network for multispectral person detection. The common characteristic of these methods is adopting a dual-stream network structure, processing inputs from different modalities separately, and then performing feature fusion at different levels of the network.

In the remote sensing field, multispectral object detection faces more complex challenges, including highly variable object sizes, complex and diverse backgrounds, unstable imaging conditions, etc. [13]. To address these challenges, Song et al. [23] proposed an information flow fusion detection method based on RGB-thermal infrared images, effectively detecting objects in complex environments by maintaining gradient and intensity information. Feng et al. [24] developed a modality-specific representation learning method, significantly improving detection stability under different lighting conditions.

In recent years, Transformer architectures have shown enormous potential in multispectral object detection. Qing et al. [25] proposed a multispectral object detection method based on cross-modal fusion Transformer, effectively capturing long-range dependencies between different modalities and achieving remarkable results in remote sensing image analysis. These attention-based methods can adaptively learn the importance of different modalities, providing new research directions for multispectral object detection.

2.2. Feature Fusion Strategies

Feature fusion strategies are the core of multispectral object detection, directly affecting the final detection performance. Based on the stage where fusion occurs, existing methods can be divided into early fusion, middle fusion, and late fusion [26]. Early fusion directly merges the original inputs at the pixel level, with high computational efficiency but possibly losing modality-specific information; middle fusion occurs after feature extraction, preserving more modality features, also known as feature fusion; late fusion integrates the outputs from different modalities after detection results are generated [27].

In traditional methods, commonly used fusion strategies include weighted averaging, maximum/minimum value selection, and principal component analysis [28]. However, these fixed rules find it difficult to adapt to complex and variable scenes. In recent years, adaptive fusion strategies based on deep learning have received widespread attention. Li et al. [29] proposed a multi-granularity attention network for infrared and visible image fusion, improving fusion effects by learning feature correlations at different levels. Wang et al. [30] developed the Res2Fusion architecture, using multi-receptive field aggregation blocks to generate multi-level features, and designed a non-local attention model for effective fusion.

Cross-modal attention mechanisms provide a new perspective for feature fusion. Zhang et al. [31] proposed a cross-stream and cross-scale adaptive fusion network, significantly improving the detection performance of salient objects in RGB-D images by establishing connections between different modules and scales. Li et al. [32] designed an attention-based generative adversarial network, achieving efficient fusion of infrared and visible images through adversarial training. These attention-based methods can adaptively emphasize key information in different modalities, suppress redundancy and noise, and achieve more precise feature fusion.

2.3. YOLO Series in Multispectral Object Detection

The YOLO (You Only Look Once) series algorithms have achieved remarkable achievements in the object detection field with their efficient single-stage detection framework [33]. From the initial YOLOv1 [8] to the recent YOLOv10 [34], this series of algorithms has continuously evolved, consistently improving detection accuracy while maintaining efficient inference speed. In remote sensing image analysis, YOLO series algorithms have received attention due to their real-time capabilities and high accuracy.

YOLOv3 [35] significantly improved small object detection capabilities by introducing multi-scale prediction and residual networks, providing an effective solution for the small-sized targets common in remote sensing images. YOLOv4 [36] introduced a series of "bag of freebies" techniques, such as mosaic data augmentation and class balance, effectively improving model performance. YOLOv6 [37] and YOLOX [38] improved detection accuracy while maintaining efficient inference by enhancing network architecture and loss functions.

In the field of multispectral object detection, the application of YOLO series algorithms primarily revolves around dual-stream network structures. Typically, two parallel YOLO backbone networks are used to process inputs from different modalities, followed by fusion at different levels of the feature extraction network. Zheng et al. [39] designed a gated fusion dual SSD architecture based on YOLOv3, effectively combining complementary information from visible and infrared images. Wang et al. [40] introduced CSPNet as a feature extraction backbone, combining Spatial Spatial Pyramid (SSP) and Path Aggregation Network (PAN) to build an efficient multi-scale feature extraction framework.

To further improve multispectral detection performance, researchers have introduced attention mechanisms into the YOLO framework. Cao et al. [41] proposed an attention fusion method for single-stage multispectral pedestrian detection, significantly improving detection accuracy by adaptively learning the importance of different modalities. Zhang et al. [12] designed an illumination-guided RGBT object detection method, achieving robust detection in complex environments through intra-modal and inter-modal fusion strategies.

The latest research indicates that combining Transformer structures with YOLO frameworks is a promising direction. Wang et al. [42] proposed SwinFuse, applying residual Swin Transformer fusion networks to infrared and visible image fusion, achieving excellent results. This hybrid architecture, combining CNN's local feature extraction capability with Transformer's global modeling capability, provides a new technical approach for multispectral object detection.

3. Methodology

This section will detail the SDRFPT-Net algorithm, explaining in order according to the system data flow. The overall architecture of SDRFPT-Net is shown in Figure 2, with its dual-stream design based on YOLOv10 capable of supporting multi-scale spectral feature extraction for visible and infrared modalities, and achieving significant improvement in detection performance through spectral self-adaptive recursive fusion mechanisms and target perception enhancement modules.

The system's data processing flow is as follows: First, the input visible and infrared images are processed separately through dual-stream feature extraction networks, generating feature maps of different scales; Then, these feature maps undergo deep interaction and fusion through the spectral self-adaptive recursive fusion module; Next, the fused features are further enhanced by the self-adaptive target perception enhancement module to strengthen the representation of target regions; Finally, the enhanced multi-scale features are aggregated through feature aggregation and input to the detection head, generating the final detection results.

Compared to traditional single-modality object detection methods, this architecture can more effectively utilize the complementary information of RGB and infrared images, especially showing greater detection accuracy and robustness in challenging scenarios such as low light, adverse weather, and complex backgrounds.

3.1. Spectral Hierarchical Perception Architecture (SHPA)

The SHPA architecture, as the core design of this algorithm, effectively processes visible and infrared spectral domain information through a dual-stream structure, laying the foundation for hierarchical perception and fusion of multi-scale features. This architecture is based on YOLOv10's excellent features and has been systematically improved for multi-modal perception.

3.1.1. Dual-stream Separated Spectral Architecture Design

Compared to YOLOv10's single backbone network feature extraction mechanism, the dual-stream separated spectral architecture proposed in this paper can effectively process RGB-IR dual-modal data's heterogeneous properties, as shown in Figure 3.

This architecture expands a single feature extraction network into a dual-stream network, processing visible spectral and infrared spectral information separately. The two feature extraction streams share similar network structures but use independent parameters, and the feature extraction process of the dual-stream network can be formalized as:

$$F_{rgb}={\mathcal{F}}_{rgb}(I_{rgb};{\theta }_{rgb})$$

(1)

$$F_{ir}={\mathcal{F}}_{ir}(I_{ir};{\theta }_{ir})$$

(2)

where, $I_{rgb}$ and $I_{ir}$ represent RGB and infrared input images, ${\mathcal{F}}_{rgb}$ and ${\mathcal{F}}_{ir}$ represent the corresponding feature extraction functions, ${\theta }_{rgb}$ and ${\theta }_{ir}$represent their respective network parameters.

The main advantages of the dual-stream architecture are:

(1)

It can design specific extraction strategies for the characteristics of different spectral domains, thereby better adapting to the characteristics of data from each modality;

(2)

It preserves the unique information of each spectral domain, avoiding the potential loss of information that might occur when processing in a single network;

(3)

It captures the feature distributions of different spectral domains through independent parameters, improving the diversity of feature representations.

Compared to YOLOv10's single feature extraction path, the dual-stream architecture shows greater robustness in complex environments, especially when the quality of information from one modality decreases (such as insufficient RGB information at night or reduced infrared contrast during the day), the system can still maintain detection performance by relying on stable information provided by the other modality.

3.1.2. Multi-scale Spectral Feature Expansion

To comprehensively capture the multi-scale representation of targets, this paper designs a multi-scale spectral feature expansion mechanism. In each spectral stream, features form a multi-scale feature pyramid through progressive downsampling. For each spectral domain$s\in \{rgb,ir\}$, the feature expansion process can be represented as:

$$F_i^s=\mathcal{H}i(F_{i-1}^s;{\theta }_i^s),i\in \{1,2,3,4\}$$

(3)

where, $F_i^s$ represents the level $i$ feature, ${\mathcal{H}}_i$ represents the downsampling function, ${\theta }_i^s$ is the corresponding parameter. Specifically, the spatial resolution and channel number of each level feature are:

$$F_1^s\in {ℝ}^{B\times 128\times {\displaystyle \frac{H}{4}}\times {\displaystyle \frac{W}{4}}}(\mathrm{P}2/4)$$

(4)

$$F_2^s\in {ℝ}^{B\times 256\times {\displaystyle \frac{H}{8}}\times {\displaystyle \frac{W}{8}}}(\mathrm{P}3/8)$$

(5)

$$F_3^s\in {ℝ}^{B\times 512\times {\displaystyle \frac{H}{16}}\times {\displaystyle \frac{W}{16}}}(\mathrm{P}4/16)$$

(6)

$$F_4^s\in {ℝ}^{B\times 1024\times {\displaystyle \frac{H}{32}}\times {\displaystyle \frac{W}{32}}}(\mathrm{P}5/32)$$

(7)

3.1.3. Feature Aggregation and Detection

After multi-scale expansion, images go through Pre-Pro, SRFM, Post-Pro, STPEM models for fusion, thereby obtaining high-quality multi-scale fusion features. These features need to be further aggregated and processed to generate the final object detection results, as shown in Figure 4.

First, multi-scale fusion features are aggregated through the feature pyramid (FPN) and path aggregation network (PAN), enhancing information exchange between features of different scales:：

$$P_i=FP{N}_i(F_{fused})$$

(8)

$$M_i=\left\{\begin{array}{ll}PA{N}_i(P_i),\hfill & \mathrm{if}i=3\hfill \\ PA{N}_i(M_{i-1},P_i),\hfill & \mathrm{if}i>3\hfill \end{array}\right.$$

(9)

where, $P_i$ represents the FPN output of level $i$ feature, $M_i$ represents the PAN output of level $i$ feature, $F_{fused}$ represents the feature after fusion, containing complementary information from RGB and IR.

FPN transmits semantic information from high levels to low levels, while PAN transmits spatial details from low levels to high levels, forming a powerful feature representation. This bidirectional feature flow mechanism ensures that features at each scale can incorporate both rich semantic information and fine spatial details.

Finally, the aggregated features pass through the v10Detect detection head for object detection:

$$D=\mathrm{Detect}(M_3,M_4,M_5)$$

(10)

where $Detect(\cdot )$ represents the detection function, with outputs including object class, bounding box coordinates, and confidence information. v10Detect adopts a more efficient feature decoding method, including dynamic convolution and branch specialization design, further improving detection accuracy and efficiency. v10Detect employs branch specialization design, designating specialized branches for bounding box regression, feature processing, and classification tasks, further improving detection accuracy and efficiency.

Compared to YOLOv10, our feature aggregation and detection stage utilizes the advantages brought by modal fusion and target perception enhancement, allowing the detection head to perform object detection based on richer and more accurate feature representations. This is particularly important in low light, adverse weather, and complex background conditions, as single-modal information is often unreliable in these scenarios.

3.2. Spectral Recursive Fusion Module (SRFM)

The SRFM module achieves deep interaction and optimized integration of RGB-IR dual-modal features through innovative fusion mechanisms, significantly improving detection performance in complex environments. Unlike traditional fusion methods, SRFM combines hybrid attention mechanisms with recursive progressive fusion strategies organically, achieving deep multi-modal feature interaction while maintaining parameter efficiency, providing powerful feature representation capabilities for multispectral object detection.

As shown in Figure 5, SRFM receives dual-stream features from SHPA and outputs fused enhanced features after cyclic progressive fusion. This section will introduce the design principles, key components and workflow of the mechanism in detail.

3.2.1. Hybrid Attention Mechanism

The hybrid attention mechanism builds a comprehensive feature enhancement system by integrating three complementary mechanisms: self-attention, cross-modal attention, and channel attention, capturing complex feature dependencies from spatial, modal relationship, and channel importance dimensions. This multi-dimensional feature enhancement design significantly improves the model's processing capability for different scenes.

According to the data flow shown in the figure, the overall calculation process of the hybrid attention mechanism can be expressed as:

$$F_{out}^{RGB}=SelfAtt(F_{chan}^{RGB})+CrossAtt(F_{chan}^{RGB},F_{chan}^{IR})$$

(11)

$$F_{out}^{IR}=SelfAtt(F_{chan}^{IR})+CrossAtt(F_{chan}^{IR},F_{chan}^{RGB})$$

(12)

where, $F_{chan}^{RGB}=ChanAtt(F_{in}^{RGB})$ and $F_{chan}^{IR}=ChanAtt(F_{in}^{IR})$ respectively represent RGB and infrared features after channel attention processing.

Channel attention mechanism. The channel attention sub-module learns channel dependencies through global information modeling, providing all-around enhanced features for the spectral recursive progressive fusion strategy. Given input feature map $F\in {ℝ}^{B\times C\times H\times W}$, where $B$, $C$, $H$, $W$ respectively represent batch size, channel number, height, and width, the calculation process of channel attention can be expressed as:

$$F_{chan}=ChanAtt(F)=F\cdot \sigma (W_2\cdot ReLU(W_1\cdot F_{avg})+W_2\cdot ReLU(W_1\cdot F_{max}))$$

(13)

where, $F_{avg}=AvgPool(F)\in {ℝ}^{B\times C\times 1\times 1}$ and $F_{max}=MaxPool(F)\in {ℝ}^{B\times C\times 1\times 1}$ represent global average pooling and global maximum pooling operations; $W_1\in {ℝ}^{{\displaystyle \frac{C}{r}}\times C}$ and $W_2\in {ℝ}^{C\times {\displaystyle \frac{C}{r}}}$ are shared weight fully connected layer parameter matrices, where $r$ is the reduction rate. The weights are mapped to the $(0,1)$ interval； finally, the channel attention weights are applied to the original features through element-wise multiplication.

Self-attention mechanism. The self-attention mechanism focuses on capturing spatial dependencies within a modality, allowing features to attend to related regions within the same modality, providing richer contextual information for the spectral hierarchical perception architecture. For input feature $F_{chan}$, the calculation process of self-attention can be expressed as:

$$F_{self}=SelfAtt(F_{chan})=Softmax\left({\displaystyle \frac{Q\cdot K^T}{\sqrt{d_k}}}\right)\cdot V$$

(14)

where, $Q=W_Q\cdot F$, $K=W_K\cdot F$, $V=W_V\cdot F$ are query, key, and value matrices obtained through learnable parameter matrices $W_Q$, $W_K$, $W_V$; $d_k$ is the feature dimension, serving as a scaling factor to avoid gradient vanishing problems.

Cross-modal attention mechanism. The cross-modal attention mechanism is used to capture complementary information between different modalities, establishing connections between visible and infrared features, and is the core component for achieving spectral information exchange. The unique aspect of cross-modal attention is that it uses the query from one modality to interact with the keys and values from another modality, thereby enabling information flow between modalities. For RGB and IR features, the calculation of cross-modal attention can be expressed as:

$$F_{cross}^{RGB}=CrossAtt(F_{chan}^{RGB},F_{chan}^{IR})=\alpha \cdot Softmax\left({\displaystyle \frac{Q^{RGB}\cdot {(K^{IR})}^T}{\sqrt{d_k}}}\right)\cdot V^{IR}$$

(15)

$$F_{cross}^{IR}=CrossAtt(F_{chan}^{IR},F_{chan}^{RGB})=\alpha \cdot Softmax\left({\displaystyle \frac{Q^{IR}\cdot {(K^{RGB})}^T}{\sqrt{d_k}}}\right)\cdot V^{RGB}$$

(16)

where, $Q^{RGB}=W_Q\cdot F_{chan}^{RGB}$, $K^{IR}=W_K\cdot F_{chan}^{IR}$, $V^{IR}=W_V\cdot F_{chan}^{IR}$ are the cross-modal attention calculation from RGB to IR; $Q^{IR}=W_Q\cdot F_{chan}^{IR}$, $K^{RGB}=W_K\cdot F_{chan}^{RGB}$, $V^{RGB}=W_V\cdot F_{chan}^{RGB}$ are the matrices for IR to RGB calculation; $\alpha$ is a learnable scaling factor that controls the strength of cross-modal information fusion.

Finally, the outputs of self-attention and cross-modal attention are added to obtain the final enhanced features:

$$F_{out}^{RGB}=F_{self}^{RGB}+F_{cross}^{RGB}$$

(17)

$$F_{out}^{IR}=F_{self}^{IR}+F_{cross}^{IR}$$

(18)

Through this design, the hybrid attention mechanism can simultaneously attend to channel importance, spatial dependency relationships, and modal complementary information, building a more comprehensive and robust feature representation.

3.2.2. Recursive Progressive Fusion Strategy

Multi-modal feature fusion is a key challenge in RGB-T object detection. Traditional multi-modal feature fusion methods typically enhance performance by stacking multiple Transformer blocks, but this approach leads to dramatic increases in parameter count and computational complexity. Inspired by the "review-consolidate" mechanism in human learning processes, this paper proposes a spectral hierarchical recursive progressive fusion strategy, achieving feature progressive refinement through repeatedly applying the same feature transformation operations, thereby enhancing fusion effects without increasing model parameters.

Parameter Cycling Reuse Structure. The core idea of the spectral hierarchical recursive progressive fusion strategy is to use the same set of parameters for multiple rounds of feature refinement. Each refinement builds on the results of the previous round, forming a continuous, progressive feature fusion process. This process can be expressed as:

$$[F_{RGB}^{t+1},F_{IR}^{t+1}]=T(F_{RGB}^t,F_{IR}^t;\theta )$$

(19)

where, $F_{RGB}^t$ and $F_{IR}^t$ respectively represent the visible and infrared features after the $t$ round of cycling, $T$ represents the feature transformation function, $\theta$ is the reused model parameter.

Through multiple cycles, the feature representation ability is continuously enhanced:

$$[F_{RGB}^{fianl},F_{IR}^{final}]=T^n(F_{RGB}^0,F_{IR}^0;\theta )$$

(20)

where, $T^n$ represents applying the transformation function $T$ continuously $n$ times，$F_{RGB}^0$ and $F_{IR}^0$ are the initial features.

Compared to traditional methods, the cyclic weight reuse structure significantly reduces the model parameter count, while achieving deep feature interaction through multiple refinements.This design not only improves the model's representation ability but also alleviates the risk of overfitting.

Spectral Feature Progressive Fusion. Spectral feature progressive fusion is the core characteristic of this strategy, progressively fusing different spectral domain features. This progressive fusion process operates in the spectral dimension, ensuring each spectral property is fully preserved and mutually enhanced. The fusion process includes the following key steps:

1. Spectral feature normalization: Normalization is performed separately on visible and infrared features, expressed as follows.

$${\widehat{F}}_{RGB}=\mathrm{LN}(F_{RGB})$$

(21)

$${\widehat{F}}_{IR}=\mathrm{LN}(F_{IR})$$

(22)

2. Hybrid attention calculation: Apply hybrid attention mechanism to process normalized features, expressed as follows,$HybridAttention(\cdot )$ represents hybrid attention calculation.

$${F^{\prime }}_{RGB},{F^{\prime }}_{IR}=\mathrm{HybridAttention}({\widehat{F}}_{RGB},{\widehat{F}}_{IR};{\theta }_{attn})$$

(23)

3. Spectral residual connection: Combine attention outputs with original spectral features, expressed as follows.

$${F^{″}}_{RGB}=F_{RGB}+{F^{\prime }}_{RGB}$$

(24)

$${F^{″}}_{IR}=F_{IR}+{F^{\prime }}_{IR}$$

(25)

4. Spectral feature enhancement: Further enhance each spectral feature through multilayer perceptron and residual connection, expressed as follows.

$${F^{‴}}_{RGB}={F^{″}}_{RGB}+\mathrm{MLP}(\mathrm{LN}({F^{″}}_{RGB}))$$

(26)

$${F^{‴}}_{IR}={F^{″}}_{IR}+\mathrm{MLP}(\mathrm{LN}({F^{″}}_{IR}))$$

(27)

where, $LN(\cdot )$ represents layer normalization operation, $MLP(\cdot )$ represents multilayer perceptron.

Progressive feature refinement process. The progressive feature refinement process can be viewed as a "feature distillation" mechanism, where each round of cycling makes the feature representation more pure and effective. In this research, we adopt a fixed 3-round cycling structure, a design based on extensive experimental validation.

The refinement process can be divided into three stages:

• First round of cycling: Initial fusion stage. Mainly captures basic intra-modal and inter-modal relationships, establishing initial feature interaction;
• Second round of cycling: Feature reinforcement stage. Based on the already established initial relationships, further strengthens important feature connections, suppressing noise and irrelevant information;
• Third round of cycling: Feature refinement stage. Performs final optimization and fine-tuning on features, forming high-quality fusion representations.

This three-round progressive refinement process can be expressed as:

$$F_{RGB}^3,F_{IR}^3=T^3(F_{RGB}^0,F_{IR}^0;\theta )$$

(28)

The progressive refinement mechanism creates a "deep cascade" effect, achieving deep network feature representation capabilities within a fixed parameter space, which is fundamentally different from traditional "multi-layer stacking" approaches. Traditional methods require introducing new parameter sets for each additional layer, while our method achieves deeper effective network depth through parameter reuse while maintaining parameter efficiency.

Spectral Multi-scale Fusion Mechanism. The spectral multi-scale fusion mechanism is an important component of the recursive progressive fusion strategy, applying recursive progressive fusion on features of different scales to achieve comprehensive multi-scale feature optimization. This mechanism includes the following key designs:

• Multi-scale feature selection: The fusion strategy is applied separately on three scales—P3/8, P4/16, and P5/32—ensuring thorough fusion of features at all three scales;
• Inter-scale information flow: Information exchange between features of different scales is achieved through FPN and PAN structures;

The multi-scale fusion process can be expressed as:

$$F_s^{fusion}=T_s^N(F_s^{init};{\theta }_s),s\in \{P3/8,P4/16,P5/32\}$$

(29)

where $s$ represents the feature scale index, and $T_s$ represents the feature transformation function for the s-th scale. By applying recursive progressive fusion across multiple scales, the system comprehensively enhances the representation capability of features at different scales, providing a solid foundation for detecting targets of various sizes.

3.3. Spectral Target Percpetion Enhancement Module (STPEM)

The STPEM module focuses on enhancing target regions in features while reducing background interference. Through mask generation and feature enhancement mechanisms, this module significantly improves the model's detection capability for small and low-contrast targets, providing more precise feature representation for object detection in complex environments.

3.3.1. Lightweight Mask Prediction

Lightweight mask prediction is the core component of STPEM. Given input feature $F\in {ℝ}^{B\times C\times H\times W}$, mask prediction first predicts target region masks through a lightweight convolutional network:

$$M=\sigma (M_{pred}(F))$$

(30)

where $M_{pred}$ represents the mask prediction network, and $\sigma$ represents the sigmoid activation function. The mask prediction network adopts a two-layer convolutional structure:

$$M_{pred}(F)={\mathrm{Conv}}_{1\times 1}(\mathrm{ReLU}(\mathrm{BN}({\mathrm{Conv}}_{3\times 3}(F))))$$

(31)

The first layer is a 3×3 convolution that reduces the number of channels from C to C/2, followed by batch normalization and ReLU activation; the second layer is a 1×1 convolution that reduces the number of channels from C/2 to 1, outputting a single-channel mask. Finally, the sigmoid function maps values to the [0,1] range, representing the probability that each position contains a target.

The mask prediction network is essentially learning "what feature patterns might correspond to target regions." For example, in RGB images, targets typically have distinct edges and texture features; in infrared images, targets often appear as regions with significant temperature differences from the background. The mask prediction network captures these feature patterns through convolutional operations to generate masks representing potential target regions.

3.3.2. Similarity Calculation and Adjustment

After mask generation, the module calculates the cosine similarity between features and masks to evaluate the correlation between each feature channel and the target region, thereby establishing explicit associations between feature channels and potential target regions:

$$F_{flat}=\mathrm{Flatten}(F)\in {ℝ}^{B\times C\times (H\times W)}$$

(32)

$$M_{flat}=\mathrm{Flatten}(M)\in {ℝ}^{B\times 1\times (H\times W)}$$

(33)

$$M_{expanded}=\mathrm{Expand}(M_{flat},C)\in {ℝ}^{B\times C\times (H\times W)}$$

(34)

$$S=\mathrm{Cos}\mathrm{ineSimilarity}(F_{flat},M_{expanded},\dim =2)\in {ℝ}^{B\times C}$$

(35)

where the $Flatten(\cdot )$ operation flattens the spatial dimensions of the features, the $Expand(\cdot )$ operation expands the mask to the same number of channels as the features, and $\mathrm{Cos}\mathrm{ineSimilarity}(\cdot )$ calculates the cosine similarity between two vectors.

After calculating the similarity between each channel and the mask, further processing is done through averaging operations and a learnable adjustment layer:

$$S_{avg}=\mathrm{Mean}(S,\dim =1,\mathrm{keepdim}=True)\in {ℝ}^{B\times 1}$$

(36)

$$S_{adjusted}=\sigma (S_{adjust}(S_{avg}))\in {ℝ}^{B\times 1\times 1\times 1}$$

(37)

where $S_{adjust}$ is a 1×1 convolutional layer for adjusting similarity, and $\sigma$ is the sigmoid activation function. This learnable similarity adjustment mechanism enables the module to adaptively adjust similarity calculations according to different scenes, improving the flexibility and adaptability of the module.

3.3.3. Feature Enhancement Mechanism

Finally, the enhanced feature $F_{enhanced}$ is achieved through similarity weighting:

$$F_{enhanced}=F\times S_{adjusted}$$

(38)

The core idea of this weighting mechanism is: if a feature has high similarity with the predicted target region, it is preserved or enhanced; if the similarity is low, the feature is suppressed. In this way, features of target regions are effectively enhanced while features of background regions are suppressed, thereby improving the signal-to-noise ratio of the features.

The STPEM module significantly improves the performance of multispectral object detection by effectively identifying and enhancing potential target regions, showing excellent performance especially when processing complex background scenes.

4. Experiments

This section will detail the experimental results of SDRFPT-Net on the FLIR-aligned and LLVIP datasets, verifying the effectiveness of our proposed algorithm. First, we introduce the experimental setup and evaluation datasets; second, we compare SDRFPT-Net with current state-of-the-art multispectral object detection methods; finally, we analyze the contribution of each innovative module through comprehensive ablation experiments.

4.1. Datasets and Evaluation Metrics

4.1.1. Datasets

This study employs two widely used multispectral object detection benchmark datasets: FLIR-aligned [14] and LLVIP [15].

FLIR-aligend. FLIR-aligned. This dataset is an aligned version processed from the original FLIR ADAS dataset [43]. The original FLIR dataset contains approximately 10,000 manually annotated thermal infrared images and their corresponding visible light reference images, captured under different daytime and nighttime conditions. Due to alignment issues between visible and infrared image pairs in the original dataset, the FLIR-aligned dataset provides 4,129 training image pairs and 1,013 testing image pairs through manual selection and alignment processing, with all images precisely aligned spatially. The dataset contains three main object classes: person, car, and bicycle, with the distribution of object sizes and quantities shown in Figure 9(a). The advantage of the FLIR-aligned dataset lies in its scene diversity, including different environments such as urban roads, highways, and residential areas, as well as various weather and lighting conditions (day, night, dusk), making it particularly suitable for evaluating the robustness and generalization capability of multispectral object detection algorithms in real driving scenarios.

LLVIP. LLVIP is a visible-infrared paired dataset specifically designed for visual tasks in low-light conditions. The dataset contains 16,836 image pairs captured by a binocular camera (HIKVISION DS-2TD8166BJZFY-75H2F/V2) at 26 different locations during night time (6 PM to 10 PM). All image pairs are strictly aligned in time and space, and all images contain pedestrian targets with annotations, with the distribution of target sizes and quantities shown in Figure 9(b). The characteristic of the LLVIP dataset is that images are captured in low-light conditions, making targets difficult to identify in visible light images, while infrared images can clearly capture heat source targets (such as pedestrians). Images are registered through a semi-automatic method, ensuring that visible and infrared images have exactly the same field of view and image dimensions. The original images have high resolution, with visible light images at 1920×1080 and infrared images at 1280×720, uniformly processed to 1080×720.

4.1.2. Metrics

To comprehensively evaluate the performance of object detection models, this study adopts the following standard evaluation metrics:

Precision (P). Precision is a key metric for measuring detection accuracy, defined as the ratio of correctly detected targets (true positives) to all detected targets (true positives and false positives). This metric reflects the accuracy of model target recognition, calculated as follows:

$$P={\displaystyle \frac{TP}{TP+FP}}$$

(39)

Recall (R). Recall measures the model's ability to detect all targets, defined as the ratio of correctly detected targets (true positives) to all actually existing targets (true positives and false negatives). This metric reflects the completeness of the model's capture of all targets in the image, calculated as follows:

$$R={\displaystyle \frac{TP}{TP+FN}}$$

(40)

Mean Average Precision at IoU=0.50 (mAP50). Mean Average Precision at IoU=0.50 (mAP50): mAP50 is the average precision calculated at an Intersection over Union (IoU) threshold of 0.50. This metric primarily measures the model's performance on "simple" detection tasks, i.e., the detection accuracy when the overlap area between the predicted bounding box and the ground truth bounding box accounts for at least 50% of the total area.

Mean Average Precision across IoU=0.50:0.95 (mAP50-95). mAP50-95 is a more comprehensive evaluation metric that calculates the average precision at different IoU thresholds (from 0.50 to 0.95, with a step size of 0.05), and then takes the average of these values. Compared to mAP50, mAP50-95 better reflects the model's localization accuracy by considering a range of stricter IoU thresholds. A high mAP50-95 score indicates that the model can maintain good performance under stricter localization standards, which is particularly important for applications requiring high localization accuracy.

4.2. Experimental Setup

All experiments in this study were conducted on a server equipped with an NVIDIA RTX 4090 GPU (24GB memory) and an Intel Core i7-13700 processor (24 cores), with 62GB system memory. The experimental environment was based on Linux 20.04 operating system, PyTorch 2.0.1 deep learning framework, CUDA 11.7, cuDNN 8.7.0, and Python 3.9.21.

During the training process, we used the SGD optimizer with a momentum parameter of 0.937 and a weight decay coefficient of 0.0005. The initial learning rate was set to 0.01, and a cosine annealing strategy was adopted to reduce the learning rate to 0.01 times its initial value by the end. The batch size was fixed at 4, input image dimensions were uniformly adjusted to 640×640 pixels, and the maximum number of training epochs was 300. An early stopping strategy was also implemented—automatically terminating the training process when there was no performance improvement for 30 consecutive epochs.

4.3. Comparison with State-of-the-Art Methods

4.3.1. On the FLIR-aligned Dataset

Table 1 shows the comparison results of SDRFPT-Net with other state-of-the-art methods on the FLIR-aligned dataset. This dataset is widely used as a benchmark for evaluating the performance of multispectral object detection systems under various environmental conditions.

The experimental results show that the proposed SDRFPT-Net outperforms existing methods in all key metrics including precision, recall, and mAP. Compared to the best-performing single-modality method YOLOv10-infrared (with mAP50 of 0.727), SDRFPT-Net's mAP50 improved by 8.0% (from 0.727 to 0.785). This improvement demonstrates that our proposed multi-modal fusion strategy can effectively integrate complementary information from different spectral domains.

Compared to other multispectral fusion methods, SDRFPT-Net achieves precision (P) and recall (R) of 0.854 and 0.700 respectively, significantly outperforming other methods. Particularly in terms of mAP50, SDRFPT-Net (0.785) improved by 11.5% compared to the second-best performing BA-CAMF Net (0.704), demonstrating the superiority of our proposed spectral dual-stream recursive fusion perception architecture in multispectral object detection tasks.

Notably, under the more stringent mAP50:95 evaluation criterion, SDRFPT-Net achieves 0.426, comparable to the single-modality baseline YOLOv10-infrared (0.424), while significantly outperforming other multi-modal fusion methods (with the highest being BA-CAMF Net's 0.351). This indicates that SDRFPT-Net not only improves target detection rate but also maintains high-precision bounding box localization capability.

As shown in Figure 10, in complex lighting and occlusion conditions, SDRFPT-Net (j) can accurately detect all targets with more precise bounding boxes. In contrast, YOLOv10-add (g) has some missed detections on small targets, while TFDet (h) and CMAFF (i) have some false detections and inaccurate bounding box issues. These visualization results intuitively demonstrate the detection advantages of SDRFPT-Net in complex scenes.

4.3.2. On the LLVIP Dataset

Table 2 shows the comparison results of SDRFPT-Net with other state-of-the-art methods on the LLVIP dataset. The LLVIP dataset focuses on pedestrian detection in low-light environments and is an important benchmark for evaluating algorithm robustness in nighttime scenes.

As can be seen from Table 2, SDRFPT-Net also achieves excellent performance on the LLVIP dataset. In terms of mAP50, SDRFPT-Net reaches 0.963, showing a slight improvement (0.2%) compared to the closest-performing single-modality method YOLOv10-infrared and multi-modal method TFDet (both at 0.961). Although this improvement is modest, achieving further improvement at an already near-saturated performance level is still significant. Under the more stringent mAP50:95 evaluation criterion, SDRFPT-Net reaches 0.706, significantly outperforming all comparison methods. Compared to the second-best performing YOLOv8-infrared (0.645), it improves by 9.5%, indicating that the proposed method has significant advantages in precise bounding box localization. This result confirms that SDRFPT-Net can not only detect target locations but also more accurately describe target boundaries.

Notably, under the low-light conditions of the LLVIP dataset, the infrared modality alone can achieve high performance (e.g., YOLOv8-infrared achieves an mAP50 of 0.961). In this case, SDRFPT-Net still achieved performance improvements through effective integration of complementary information from visible light, especially with significant improvement in mAP50:95 (from 0.645 to 0.706). This indicates that the proposed spectral recursive fusion mechanism can still effectively extract and integrate valuable features from the visible light modality even when infrared information is dominant.

SDRFPT-Net's recall reaches 0.911, higher than all comparison methods, indicating it has stronger target detection capability and can find pedestrian targets that might be missed by other methods, which is particularly important for practical application scenarios.

Figure 11 provides visualized detection results of various methods in typical nighttime scenes from the LLVIP dataset. Qualitative analysis shows that SDRFPT-Net can accurately locate all pedestrian targets in low-contrast environments with high bounding box matching accuracy. In contrast, other multi-modal fusion methods such as YOLOv10-add, TFDet, and CMAFF show varying degrees of detection instability in complex scenes, including missed detections, false detections, or bounding box localization deviations. These visualization results further confirm the advantages of SDRFPT-Net demonstrated in the quantitative evaluation.

Figure 11 shows the visualization detection results of various algorithms on the LLVIP dataset. In typical nighttime low-light scenes, SDRFPT-Net (j) can accurately detect all pedestrians with more precise bounding boxes. In contrast, YOLOv10-add (g), TFDet (h), and CMAFF (i) exhibit missed detections or inaccurate bounding box issues in some complex scenes. These visualization results further confirm the detection advantages of SDRFPT-Net in low-light environments.

Combining the experimental results from both the FLIR-aligned and LLVIP datasets, SDRFPT-Net demonstrates powerful detection capability and robustness under various environmental conditions. This is attributed to the collaborative work of our three innovative modules: the spectral hierarchical perception architecture provides a solid foundation for multi-modal feature extraction; the spectral adaptive recursive fusion module achieves deep interaction and efficient fusion; and the spectral adaptive target perception enhancement module further improves target region feature representation. The organic combination of these three modules enables SDRFPT-Net to achieve excellent multispectral object detection performance while maintaining low computational complexity.

4.4. Ablation Studies

To verify the effectiveness of each innovative module in SDRFPT-Net, we conducted systematic ablation experiments on the FLIR-aligned dataset. These experiments aim to evaluate the contribution of each component to the overall performance of the network and validate the rationality of our proposed design scheme.

4.4.1. Baseline Model Comparison

First, we established a baseline model, then gradually added each core component to evaluate the contribution of each module. Table 3 shows the performance comparison of different component combinations.

From the results in Table 3, it is evident that each component we proposed contributes significantly to detection performance. The model based on the Spectral Hierarchical Perception Architecture (SHPA) (A1) achieves an mAP50 of 0.701. After adding the Spectral Adaptive Recursive Fusion Module (SRFM) (A2), the mAP50 increases to 0.775, a relative improvement of 10.6%. With the further addition of the Spectral Adaptive Target Perception Enhancement Module (STPEM) (A3), mAP50 and mAP50:95 reach 0.785 and 0.426 respectively, with mAP50:95 showing a particularly significant 14.2% relative improvement over A2, indicating that STPEM greatly enhances the model's localization accuracy under stricter detection standards.

4.4.2. Ablation Experiments on Hybrid Attention Mechanism

To evaluate the effectiveness of various attention mechanisms in the hybrid attention mechanism, we designed a series of comparative experiments, with results shown in Table 4.

The results show that different types of attention mechanisms have varying impacts on model performance. When used individually, the self-attention mechanism (B1) performs best with an mAP50 of 0.776 and mAP50:95 of 0.408, indicating that capturing spatial dependencies within modalities is critical for object detection. Although cross-modal attention (B2) and channel attention (B3) show slightly inferior performance when used alone, they provide feature enhancement capabilities in different dimensions.

In combinations of two attention mechanisms, the combination of self-attention and cross-modal attention (B4) performs best, with mAP50:95 reaching 0.424, approaching the performance of the complete model. The complete combination of three attention mechanisms (B7) achieves the best performance, confirming the rationality of the hybrid attention mechanism design, which can comprehensively capture the complex relationships in multi-modal data.

To gain a deeper understanding of the role of different attention mechanisms in multispectral object detection, we analyzed the visualization results of self-attention, cross-modal attention, and channel attention on the P3 feature layer.

From Figure 12, it can be observed that the feature maps of single attention mechanisms present different attention patterns:

• Self-attention mechanism (B1): Mainly focuses on target contours and edge information, effectively capturing spatial contextual relationships, with strong response to target boundaries, helping to improve localization accuracy;
• Cross-modal attention mechanism (B2): Presents overall attention to target areas, integrating complementary information from RGB and infrared modalities, but with relatively weak background suppression capability;
• Channel attention mechanism (B3): Demonstrates selective enhancement of specific semantic information, highlighting important feature channels, with strong response to specific parts of targets, improving the discriminability of feature representation.

Furthermore, we conducted a comparative analysis of the visualization effects of dual attention mechanisms versus the full attention mechanism on the P3 feature layer, as shown in Figure 13.

Through the visualization comparative analysis in Figure 13, we observe that the feature maps of dual attention mechanisms present complex and differentiated feature representations:

• Self-attention + Cross-modal attention (B4): The feature map simultaneously possesses excellent boundary localization capability and overall target region representation capability. The heatmap shows precise response to target regions with significant background suppression effect. This combination fully leverages the complementary advantages of self-attention in spatial modeling and cross-modal attention in multi-modal fusion, enabling it to reach 0.424 in mAP50:95, approaching the performance of the full attention mechanism.
• Self-attention + Channel attention (B5): The feature map enhances the representation of specific semantic features while preserving target boundary information. The heatmap shows strong response to key parts of targets, enabling the model to better distinguish different categories of targets, achieving 0.409 in mAP50:95, outperforming any single attention mechanism.
• Cross-modal attention + Channel attention (B6): The feature map enhances specific channel representation based on multi-modal fusion, but lacks the spatial context modeling capability of self-attention. The heatmap shows some response to target regions, but boundaries are not clear enough and background suppression effect is relatively weak, which explains its relatively lower performance.

Although dual attention mechanisms (especially self-attention + cross-modal attention) can improve feature representation capability to some extent, they cannot completely replace the comprehensive advantages of the full attention mechanism.

As shown in Figure 12 (d) and Figure 13(d), the full attention mechanism, through the synergistic effect of three attention mechanisms, shows the most precise and strong response to target regions in the heatmap, with clear boundaries and optimal background suppression effect, achieving an organic unification of spatial context modeling, multi-modal information fusion, and channel feature enhancement, obtaining optimal performance in multispectral object detection tasks.

Based on the above experiments and visualization analysis, we verified the effectiveness of the proposed hybrid attention mechanism. The results show that, despite the advantages of single and dual attention mechanisms, the complete combination of three attention mechanisms achieves optimal performance across all evaluation metrics. This confirms the rationality of our proposed "spatial-modal-channel" multi-dimensional attention framework, which creates an efficient synergistic mechanism through self-attention capturing spatial contextual relationships, cross-modal attention fusing complementary information, and channel attention selectively enhancing key features. This multi-dimensional feature enhancement strategy provides a new feature fusion paradigm for multispectral object detection, offering valuable reference for research in related fields.

4.4.3. Ablation Experiments on Spectral Hierarchical Recursive Progressive Fusion Strategy

To verify the effectiveness of the spectral fusion strategy, we conducted ablation experiments from two aspects: fusion position selection and recursive progression iterations.

Impact of introducing fusion positions. Multi-scale feature fusion is a key link in multispectral object detection, and effective fusion of features at different scales has a decisive impact on model performance. This section explores the impact of fusion positions and fusion strategies on detection performance through ablation experiments and visualization analysis.

To systematically study the impact of fusion positions on model performance, we designed a series of ablation experiments, as shown in Table 5. The experiments started from the baseline model (C1, using simple addition fusion at all scales), progressively applying our proposed innovative fusion modules (fusion mechanism combining SRFM and STPEM) at different scales, and finally evaluating the effect of comprehensive application of advanced fusion strategies.

As shown in Table 5, with the increase in application positions of innovative fusion modules, model performance progressively improves. The baseline model (C1) only uses simple addition fusion at all feature scales, with an mAP50 of 0.701. When applying SRFM and STPEM modules at the P3/8 scale (C2), performance significantly improves to an mAP50 of 0.769. With further application of advanced fusion at the P4/16 scale (C3), mAP50 increases to 0.776. Finally, when the complete fusion strategy is applied to all three scales (C4), performance reaches optimal levels with an mAP50 of 0.785 and mAP50:95 of 0.426.

To intuitively understand the effect differences between different fusion strategies, we visualized and compared feature maps of simple addition fusion and advanced fusion strategies (SRFM+STPEM) at three scales: P3/8, P4/16, and P5/32, as shown in Figure 14.

To gain a deeper understanding of the performance differences between different fusion strategies, we conducted systematic visualization comparisons at different scales (P3/8, P4/16, P5/32).

1. At the P3/8 scale, simple addition fusion presents dispersed activation patterns with insufficient target-background differentiation, especially with suboptimal activation intensity for small vehicles; whereas the feature map generated by the SRFM+STPEM fusion strategy possesses more precise target localization capability and boundary representation, with significantly improved background suppression effect and activation intensity distribution more concentrated on target regions, effectively enhancing small target detection performance.

2. The comparison of P4/16 scale feature maps shows that although simple addition fusion can capture medium target positions, activation is not prominent enough and background noise interference exists; in contrast, the advanced fusion strategy produces more concentrated activation areas with higher target-background contrast and clearer boundaries between vehicles. As an intermediate resolution feature map, P4 (40×40) demonstrates superior structured representation and background suppression capability under the advanced fusion strategy.

3. At the P5/32 scale, rough semantic information generated by simple addition fusion makes it difficult to distinguish main vehicle targets; whereas the advanced fusion strategy can better capture overall scene semantics, accurately represent main vehicle targets, and effectively suppress background interference. Although the P5 feature map has the lowest resolution (20×20), it has the largest receptive field, and the advanced fusion strategy fully leverages its advantages in large target detection and scene understanding.

Through comparative analysis, we observed three key synergistic effects of multi-scale fusion:

• Complementarity enhancement: the advanced fusion strategy makes features at different scales complementary, with P3 focusing on details and small targets, P4 processing medium targets, and P5 capturing large-scale structures and semantic information;
• Information flow optimization: features at different scales mutually enhance each other, with semantic information guiding small target detection and detail information precisely locating large target boundaries;
• Noise suppression capability: the advanced fusion strategy demonstrates superior background noise suppression capability at all scales, effectively reducing false detections.

Impact of recursive iteration count. The recursive progression mechanism is a key strategy in our proposed SDRFPT-Net model, which can further enhance feature representation capability through multiple recursive progressive fusions. To explore the optimal number of recursive iterations, we designed a series of experiments to observe model performance changes by varying the number of iterations. Table 6 shows the impact of iteration count on model performance.

The experimental results show that the number of recursive iterations significantly affects model performance. When the iteration count is 1 (D1), the model achieves an mAP50 of 0.769, indicating that even a single round of iteration can provide effective feature fusion. As the iteration count increases to 2 (D2) and 3 (D3), performance continues to improve, reaching mAP50 of 0.783 and 0.785, and mAP50:95 of 0.418 and 0.426 respectively. However, when the iteration count further increases to 4 (D4) and 5 (D5), performance begins to decline, with the mAP50 of 5 iterations significantly dropping to 0.761.

To more intuitively understand the impact of iteration count on feature representation, we conducted visualization analysis of feature maps at three feature scales—P3, P4, and P5—with different iteration counts, as shown in Figure 15.

Through visualization, we observed the following feature evolution patterns:

• P3 feature layer (high resolution): As the iteration count increases, the feature map gradually evolves from initial dispersed response (n=1) to more focused target representation (n=2,3), with clearer boundaries and stronger background suppression effect. However, when the iteration count reaches 4 and 5, over-smoothing phenomena begin to appear, with some loss of boundary details.
• P4 feature layer (medium resolution): At n=1, the feature map has basic response to targets but is not focused enough. After 2-3 rounds of iteration, the activation intensity of target areas significantly increases, improving target differentiation. Continuing to increase the iteration count to 4-5 rounds, feature response begins to diffuse, reducing precise localization capability.
• P5 feature layer (low resolution): This layer demonstrates the most obvious evolution trend, gradually developing from blurred response at n=1 to highly structured representation at n=3 that can clearly distinguish main targets. However, obvious signs of overfitting appear at n=4 and n=5, with feature maps becoming overly smoothed and target representation degrading.

These observations reveal the working mechanism of recursive progressive fusion: moderate iteration count (n=3) can achieve progressive optimization of features through multiple rounds of interactive fusion of complementary information from different modalities, enhancing target feature representation and suppressing background interference. However, excessive iteration count may lead to "over-fusion" of features, i.e., the model overfits specific patterns in the training data, losing generalization capability.

Combining quantitative and visualization analysis results, we determined n=3 as the optimal iteration count, achieving the best balance between feature enhancement and computational efficiency. This finding is also consistent with similar observations in other research areas, such as the optimal unfolding steps in recurrent neural networks and the optimal iteration count in message passing neural networks, where similar "performance saturation points" exist.

Through the above ablation experiments, we verified the effectiveness and optimal configuration of each core component of SDRFPT-Net. The results show that the spectral hierarchical perception architecture (SHPA), the complete combination of three attention mechanisms, the full-scale advanced fusion strategy, and three rounds of recursive progressive fusion collectively contribute to the model's superior performance. The rationality of these design choices is not only validated through quantitative metrics but also intuitively explained through feature visualization, providing new ideas for multispectral object detection research.

5. Discussion

Based on the above experimental results, the proposed SDRFPT-Net demonstrates significant advantages in the field of multispectral object detection. First, the spectral dual-stream recursive fusion architecture achieves efficient integration of visible and infrared information, confirming the effectiveness of the deep interaction design concept. Compared to traditional fusion methods, the hybrid attention mechanism in the SRFM module is particularly suitable for processing multi-modal features in complex environments, capable of adaptively adjusting the weights of each modality according to different scene conditions. For example, in low-light conditions, the system enhances the contribution of the infrared modality; while in well-lit conditions with complex background interference, it relies more on the rich texture information of the visible light modality. This adaptive fusion mechanism is key to the model's excellent performance in various challenging environments.

Despite SDRFPT-Net's excellent performance, several limitations are worth discussing. First, although the recursive progressive fusion strategy improves feature interaction depth, the optimal design of three iterations also reflects that too many iterations may lead to over-fusion of features, which actually harms performance. Second, the current model still has room for improvement in processing distant small targets, especially under adverse weather conditions. Additionally, deploying the model on edge devices still faces certain challenges. Future work could consider further optimizing the network structure, while exploring the integration of temporal information into the model to improve adaptability to dynamic scenes.

6. Conclusions

This paper presents SDRFPT-Net (Spectral Dual-stream Recursive Fusion Perception Target Network), a novel architecture for multispectral object detection that effectively integrates visible and infrared modalities through three innovative modules. First, the Spectral Hierarchical Perception Architecture (SHPA) based on YOLOv10 employs a dual-stream structure to extract modality-specific features. Second, the Spectral Recursive Fusion Module (SRFM) achieves deep cross-modal feature interaction through a hybrid attention mechanism and recursive fusion strategy. Finally, the Spectral Target Perception Enhancement Module (STPEM) enhances target region representation and suppresses background interference through lightweight mask prediction.

Extensive experiments on the FLIR-aligned and LLVIP datasets demonstrate that SDRFPT-Net outperforms existing methods across all key metrics. On the FLIR-aligned dataset, our model achieves 0.785 mAP50 and 0.426 mAP50:95, surpassing the second-best BA-CAMF Net by 11.5%. On the LLVIP dataset, it reaches 0.963 mAP50 and 0.706 mAP50:95, significantly outperforming all comparison methods.

Through comprehensive ablation studies, we validated the effectiveness and optimal configuration of each innovative module. Results indicate that the complete hybrid attention mechanism (combining self-attention, cross-modal attention, and channel attention), full-scale advanced fusion strategy, and three iterations of recursive progressive fusion collectively contribute to the model's superior performance. Notably, the recursive progressive fusion mechanism achieves an optimal performance balance at three iterations, creating a "deep cascade" effect that enables deep network representation capabilities within a fixed parameter space.

Despite SDRFPT-Net's excellent performance, several limitations warrant discussion. Although the recursive progressive fusion strategy enhances feature interaction depth, the optimal design of three iterations also reflects that excessive iterations may lead to feature "over-fusion," which actually impairs performance. Additionally, the current model still has room for improvement in processing distant small targets, particularly under adverse weather conditions. Furthermore, deploying the model on edge devices continues to face certain challenges.

Future work could consider further optimizing the network structure while exploring the integration of temporal information to improve adaptability to dynamic scenes. Meanwhile, we will focus on enhancing model robustness in complex environments and reducing computational complexity to meet various requirements in practical applications. Overall, SDRFPT-Net offers a promising solution for multispectral object detection in applications such as autonomous driving, security surveillance, and remote sensing analysis.