CMAFNet: Efficient Cross-Modal Alignment and Fusion for Real-Time RGB–Infrared Object Detection in Autonomous Driving

1. Introduction

Autonomous driving systems, as representative cyber-physical platforms in modern automotive electronics, demand robust and reliable perception capabilities to ensure safe navigation across diverse and challenging environmental conditions [1]. Object detection, as a fundamental perception task in the on-board sensing and perception stack, must maintain high accuracy under strict latency and computational constraints regardless of illumination variations, adverse weather, and complex traffic scenarios. While RGB cameras provide rich texture and color information under well-lit conditions, their performance degrades significantly in low-light environments, fog, rain, and glare [2]. Infrared (IR) sensors, which capture thermal radiation emitted by objects, offer complementary advantages by providing reliable detection capabilities in darkness and adverse weather conditions [3]. Consequently, the fusion of RGB and IR modalities has attracted increasing research attention as a promising approach to achieve robust all-weather object detection for autonomous driving and real-time automotive perception systems [4].

Despite significant progress in multi-modal object detection, several fundamental challenges remain unresolved. First, existing fusion methods often employ simple concatenation or element-wise addition to combine features from different modalities [5], which fails to capture the complex complementary relationships between RGB and IR features. The heterogeneous nature of these two modalities—RGB images encode appearance, texture, and color information while IR images capture thermal signatures—necessitates more sophisticated alignment and interaction mechanisms. Second, most current approaches rely on convolutional neural networks (CNNs) or standard Transformer architectures for feature extraction and fusion [6]. CNNs are inherently limited by their local receptive fields, making it difficult to model long-range dependencies across modalities. While Transformers can capture global relationships, their quadratic computational complexity with respect to sequence length poses significant challenges for real-time and compute-constrained automotive deployment [7]. Third, the quality of object queries in detection heads significantly impacts final detection performance, yet most methods do not explicitly address the uncertainty in query selection when dealing with multi-modal inputs [8]. Beyond perception accuracy, robustness and reliability under adverse conditions have also been conceptually discussed from connectivity and diagnosability perspectives in complex networks [9,10,11,12], highlighting the importance of redundancy and complementary information sources.

To address these challenges under practical real-time constraints, we propose CMAFNet (Cross-Modal Alignment and Fusion Network), a comprehensive yet deployment-aware framework for robust multi-modal object detection in autonomous driving. Our approach introduces several novel components that work synergistically to achieve effective cross-modal feature alignment, interaction, and fusion while maintaining a favorable accuracy–efficiency trade-off. The overall architecture of CMAFNet is illustrated in Figure 1.

Specifically, our contributions are summarized as follows:

• We propose CMAFNet, a deployment-aware end-to-end multi-modal object detection framework that achieves robust cross-modal alignment and fusion for autonomous driving under real-time constraints. The framework integrates a Dynamic Receptive Backbone, Channel-Split Mamba Block, Global Multi-modal Interaction Network, and Separable Dynamic Decoder into a unified and efficiency-oriented architecture.
• We introduce the Channel-Split Mamba Block (CSM-Block), which leverages selective state space models [13] to efficiently capture long-range cross-modal dependencies. By splitting feature channels into multiple groups and processing them through parallel CS-Mamba branches with subsequent attention-based recalibration, the CSM-Block enables effective global context modeling with linear computational complexity, making it suitable for compute-constrained automotive perception systems.
• We design the Global Multi-modal Interaction Network (GMIN), a dual-branch cross-modal attention module that performs fine-grained feature alignment through global average pooling and global max pooling guided cross-attention. GMIN generates modality-specific gating signals to adaptively weight and fuse complementary information from RGB and IR features, improving robustness under adverse sensing conditions.
• Extensive experiments on two widely-used benchmarks (M3FD and FLIR-Aligned) demonstrate that CMAFNet achieves state-of-the-art performance while maintaining a favorable accuracy–latency trade-off, supporting practical real-time deployment in automotive embedded vision scenarios.

The remainder of this paper is organized as follows. Section 2 reviews related work on multi-modal object detection, state space models, and attention mechanisms. Section 3 presents the proposed CMAFNet framework in detail. Section 4 describes the experimental setup and presents comprehensive results, including efficiency analysis. Section 6 concludes the paper.

2. Related Work

2.1. Multi-Modal Object Detection

Multi-modal object detection aims to leverage complementary information from multiple sensor modalities to improve detection robustness in real-world perception systems [1]. In the context of autonomous driving and automotive perception stacks, RGB-infrared fusion has been extensively studied. Early approaches adopted simple fusion strategies such as early fusion (input-level concatenation), late fusion (decision-level combination), and halfway fusion (feature-level merging) [5]. Liu et al. [5] systematically compared these strategies for multispectral pedestrian detection and demonstrated that halfway fusion generally achieves a reasonable trade-off between performance and computational complexity.

More recent methods have explored attention-based fusion mechanisms to improve cross-modal interaction. Zhang et al. [14] proposed Guided Attentive Feature Fusion (GAFF) that uses illumination-aware gating to adaptively weight modality contributions. Zhou et al. [15] introduced MBNet to address modality imbalance through differential modality-aware feature learning. The Cross-modality Interactive Attention Network (CIAN) [16] employs cross-attention to model inter-modal relationships. CFT [6] introduced a cross-modality fusion transformer that uses self-attention and cross-attention to capture both intra-modal and inter-modal dependencies. ICAFusion [17] proposed iterative cross-attention for progressive feature alignment. BAANet [18] designed bi-directional adaptive attention gates for multispectral pedestrian detection. EAEFNet [19] introduced explicit attention-enhanced fusion for RGB-thermal perception. TFDet [20] proposed target-aware fusion mechanisms for pedestrian detection.

Although these approaches significantly improve detection accuracy, many of them rely on Transformer-style attention mechanisms with quadratic complexity or introduce additional computational overhead, which may limit scalability under real-time and embedded deployment constraints. Our proposed CMAFNet addresses these limitations by introducing the CSM-Block with linear complexity and the GMIN module for comprehensive yet efficient cross-modal interaction. From a broader robustness perspective, classical connectivity and matching-preclusion studies in interconnection networks highlight how redundancy-preserving structures maintain reliable information flow under disruptions [21,22,23,24]. This conceptual viewpoint resonates with cross-modal fusion, where complementary sensing modalities provide redundancy to sustain detection reliability under adverse environmental conditions.

2.2. State Space Models for Vision

State Space Models (SSMs) have recently emerged as a promising alternative to Transformers for sequence modeling with improved computational efficiency [25]. The Mamba architecture [13] introduced a selective state space mechanism with hardware-aware design, achieving linear computational complexity while maintaining strong modeling capabilities for long sequences. Vision Mamba [26] adapted the Mamba architecture for visual representation learning by introducing bidirectional scanning strategies. VMamba [27] proposed a visual state space model with cross-scan mechanisms for hierarchical feature extraction.

The efficiency advantages of SSM-based architectures make them particularly attractive for perception modules operating under latency and compute constraints. In multi-modal fusion, the ability to efficiently model long-range dependencies across modalities is crucial for capturing complementary information without incurring prohibitive computational cost. In this work, we propose the Channel-Split Mamba Block that leverages the selective state space mechanism to capture cross-modal dependencies with linear complexity, making it suitable for real-time autonomous driving applications.

2.3. Attention Mechanisms in Object Detection

Attention mechanisms have been widely adopted in object detection to enhance feature representation [7]. Channel attention mechanisms such as SE-Net [28] and spatial attention mechanisms such as CBAM [29] have demonstrated effectiveness in single-modal detection with modest computational overhead. In the multi-modal setting, cross-attention mechanisms enable explicit interaction between features from different modalities [6], but may introduce additional complexity depending on implementation.

The DETR family of detectors [8,30,31,32] has reformulated object detection as a set prediction problem. Deformable DETR [31] introduced deformable attention to reduce computational cost. DINO [8] improved detection performance through denoising training and mixed query selection. RT-DETR [32] achieved real-time performance by designing an efficient hybrid encoder. Co-DETR [33] proposed collaborative hybrid assignments for improved training efficiency. These advances in query-based detection provide a strong architectural foundation for our uncertainty-minimal query selection strategy under multi-modal settings.

3. Proposed Method

In this section, we present the proposed CMAFNet framework in detail. As illustrated in Figure 1, CMAFNet consists of four main components: (1) a Dynamic Receptive Backbone (DRB) for multi-scale feature extraction, (2) a Channel-Split Mamba Block (CSM-Block) for efficient cross-modal feature enhancement, (3) a Global Multi-modal Interaction Network (GMIN) for fine-grained cross-modal fusion, and (4) an Uncertainty-minimal Query Selection mechanism with a Separable Dynamic Decoder for final detection. The overall design emphasizes efficient cross-modal interaction and real-time feasibility under automotive perception constraints.

3.1. Dynamic Receptive Backbone

Effective multi-scale feature extraction is fundamental to object detection, particularly in autonomous driving scenarios where objects exhibit significant scale variations (e.g., distant pedestrians vs. nearby vehicles). In practical automotive perception systems, the backbone must not only capture scale diversity but also maintain computational efficiency. We adopt a Dynamic Receptive Backbone (DRB) that adaptively adjusts its receptive field to capture features at multiple scales while preserving favorable computational characteristics, as shown in Figure 2.

The DRB processes both RGB and IR inputs through a shared-weight backbone architecture, which reduces parameter redundancy and facilitates efficient multi-modal feature extraction. Given an input image $\mathbf{I}\in {\mathbb{R}}^{3\times H\times W}$ (or ${\mathbb{R}}^{1\times H\times W}$ for single-channel IR), the backbone extracts hierarchical features through four stages:

$$\{C_1,C_2,C_3,C_4\}=\mathrm{DRB}\left(\mathbf{I}\right),$$

(1)

where $C_i\in {\mathbb{R}}^{c_i\times {\displaystyle \frac{H}{2^{i+1}}}\times {\displaystyle \frac{W}{2^{i+1}}}}$ represents the feature map at stage i with channel dimension $c_i\in \{D,2D,4D,8D\}$ and D denotes the base channel dimension. This hierarchical design enables progressive spatial reduction while maintaining representational efficiency.

Each building block in the DRB consists of a sequence of operations:

$${\mathbf{x}}^{\prime }=\mathrm{MLP}\left(\mathrm{BN}\left(\mathrm{SiLU}\left(\mathrm{PW}\left(\mathrm{BN}\left(\mathrm{SiLU}\left({\mathrm{DW}}_{5\times 5}\left(\mathrm{BN}\left(\mathrm{SiLU}\left({\mathrm{Conv}}_{3\times 3}\left(\mathbf{x}\right)\right)\right)\right)\right)\right)\right)\right)\right)\right),$$

(2)

where ${\mathrm{Conv}}_{3\times 3}$ denotes a standard $3\times 3$ convolution, ${\mathrm{DW}}_{5\times 5}$ represents a $5\times 5$ depthwise separable convolution [34], PW is a $1\times 1$ pointwise convolution, BN is batch normalization [35], and SiLU is the Sigmoid Linear Unit activation [36]. The use of depthwise separable convolution reduces computational overhead compared to standard convolution while preserving expressive capacity.

The multi-scale feature enhancement module employs bidirectional feature propagation with adaptive receptive field weights. For the top-down path, features are progressively upsampled and fused:

$$F_i^{\mathrm{td}}=C_i\oplus \mathrm{Up}\left(F_{i+1}^{\mathrm{td}}\right),i=3,2,1,$$

(3)

where ⊕ denotes element-wise addition and $\mathrm{Up}(·)$ represents bilinear upsampling. For the bottom-up path:

$$F_i^{\mathrm{bu}}=F_i^{\mathrm{td}}\oplus \mathrm{Down}\left(F_{i-1}^{\mathrm{bu}}\right),i=2,3,4,$$

(4)

where $\mathrm{Down}(·)$ denotes strided convolution for downsampling. This bidirectional aggregation enhances multi-scale representation without introducing excessive architectural complexity.

The receptive field weight adaptation mechanism dynamically adjusts the contribution of features at different scales through learned attention weights:

$${\alpha }_i=\mathrm{Softmax}\left(\mathrm{Reg}\left(\mathrm{Cls}\left(\mathrm{Up}\left(F_4^{\mathrm{bu}}\right)\right)\right)\right),$$

(5)

$$F_i={\alpha }_i\odot F_i^{\mathrm{bu}},i=1,2,3,4,$$

(6)

where $\mathrm{Cls}(·)$ and $\mathrm{Reg}(·)$ are classification and regression branches, and ⊙ denotes element-wise multiplication. The adaptive weighting mechanism enables dynamic emphasis on informative scales while maintaining computational tractability.

For both RGB and IR modalities, we extract multi-scale features:

$$\{P_3^{\mathrm{rgb}},P_4^{\mathrm{rgb}},P_5^{\mathrm{rgb}}\}=\mathrm{DRB}\left({\mathbf{I}}_{\mathrm{rgb}}\right),\{P_3^{\mathrm{ir}},P_4^{\mathrm{ir}},P_5^{\mathrm{ir}}\}=\mathrm{DRB}\left({\mathbf{I}}_{\mathrm{ir}}\right),$$

(7)

where $P_3$, $P_4$, and $P_5$ correspond to feature maps at $1/8$, $1/16$, and $1/32$ of the input resolution, respectively. The shared backbone structure promotes parameter efficiency and consistent feature abstraction across modalities.

3.2. Channel-Split Mamba Block

To efficiently capture long-range dependencies across modalities while maintaining strict computational budgets, we propose the Channel-Split Mamba Block (CSM-Block). Unlike standard Transformer-based cross-attention that incurs quadratic complexity with respect to token length, the CSM-Block leverages selective state space models to achieve linear complexity, which improves scalability for high-resolution inputs in real-time automotive perception systems. The detailed architecture is shown in Figure 3.

3.2.1. Channel Splitting and Parallel Processing

Given the concatenated RGB and IR features $\mathbf{F}\in {\mathbb{R}}^{C\times H\times W}$ at a specific scale, the CSM-Block first applies a $1\times 1$ convolution for channel projection:

$${\mathbf{F}}^{\prime }={\mathrm{Conv}}_{1\times 1}\left(\mathbf{F}\right)\in {\mathbb{R}}^{C\times H\times W}.$$

(8)

The projected features are then flattened to ${\mathbf{F}}^{\prime }\in {\mathbb{R}}^{C\times N}$ where $N=H\times W$, and split along the channel dimension into G groups:

$$\{{\mathbf{F}}_1^{\prime },{\mathbf{F}}_2^{\prime },\dots ,{\mathbf{F}}_G^{\prime }\}=\mathrm{Split}\left({\mathbf{F}}^{\prime }\right),{\mathbf{F}}_g^{\prime }\in {\mathbb{R}}^{(C/G)\times N}.$$

(9)

This channel partitioning strategy enables parallel processing of feature subspaces, reducing per-branch computational burden while preserving expressive diversity. Each group is independently processed by a CS-Mamba module:

$${\widehat{\mathbf{F}}}_g=\mathrm{CS}-\mathrm{Mamba}\left({\mathbf{F}}_g^{\prime }\right),g=1,2,\dots ,G.$$

(10)

The parallel design improves computational scalability and facilitates efficient implementation on modern GPU architectures.

3.2.2. CS-Mamba Module

The CS-Mamba module is built upon the selective state space mechanism [13], which provides linear-time sequence modeling with hardware-aware characteristics. For each channel group ${\mathbf{F}}_g^{\prime }$, the module employs a dual-branch architecture:

$${\mathbf{z}}_g={\mathrm{Linear}}_z\left({\mathbf{F}}_g^{\prime }\right),$$

(11)

$${\mathbf{h}}_g=\mathrm{SiLU}\left(\mathrm{DWConv}\left({\mathrm{Linear}}_h\left({\mathbf{F}}_g^{\prime }\right)\right)\right),$$

(12)

$${\mathbf{y}}_g=\mathrm{SS}2\mathrm{D}-\mathrm{LS}\left({\mathbf{h}}_g\right)\odot {\mathbf{z}}_g,$$

(13)

where ${\mathrm{Linear}}_z$ and ${\mathrm{Linear}}_h$ are linear projection layers, DWConv is a depthwise convolution, and SS2D−LS denotes the Selective Scan 2D with Local Scan operation. This gated interaction mechanism enables effective cross-token modulation while preserving computational efficiency.

The SS2D-LS operation extends the 1D selective scan to 2D feature maps through four directional scans (left-to-right, right-to-left, top-to-bottom, bottom-to-top) combined with local scanning patterns. Compared with global self-attention, the directional scanning strategy avoids quadratic pairwise interactions while still enabling long-range information propagation. For each direction d, the selective state space dynamics are defined as:

$${\mathbf{s}}_t^{\left(d\right)}={\overline{\mathbf{A}}}^{\left(d\right)}{\mathbf{s}}_{t-1}^{\left(d\right)}+{\overline{\mathbf{B}}}^{\left(d\right)}{\mathbf{h}}_t^{\left(d\right)},$$

(14)

$${\mathbf{o}}_t^{\left(d\right)}={\mathbf{C}}^{\left(d\right)}{\mathbf{s}}_t^{\left(d\right)},$$

(15)

where ${\mathbf{s}}_t^{\left(d\right)}$ is the hidden state, ${\overline{\mathbf{A}}}^{\left(d\right)}$ and ${\overline{\mathbf{B}}}^{\left(d\right)}$ are the discretized state transition and input matrices, ${\mathbf{C}}^{\left(d\right)}$ is the output matrix, and ${\mathbf{h}}_t^{\left(d\right)}$ is the input at position t along direction d. The discretization is performed using the zero-order hold (ZOH) method:

$$\overline{\mathbf{A}}=exp\left(\mathsf{\Delta }\mathbf{A}\right),\overline{\mathbf{B}}={\left(\mathsf{\Delta }\mathbf{A}\right)}^{-1}(exp\left(\mathsf{\Delta }\mathbf{A}\right)-\mathbf{I})·\mathsf{\Delta }\mathbf{B},$$

(16)

where $\mathsf{\Delta }$ is the input-dependent step size that enables selective processing and adaptive receptive behavior.

The outputs from all directions are merged:

$$\mathrm{SS}2\mathrm{D}-\mathrm{LS}\left({\mathbf{h}}_g\right)=\sum _{d=1}^4{\mathbf{o}}^{\left(d\right)}.$$

(17)

This aggregation preserves global contextual cues while maintaining linear computational complexity.

The final output of the CS-Mamba module includes normalization and MLP:

$${\widehat{\mathbf{F}}}_g={\mathbf{F}}_g^{\prime }+\mathrm{MLP}\left(\mathrm{LN}\left(\mathrm{Linear}\left({\mathbf{y}}_g\right)\right)\right),$$

(18)

where LN denotes Layer Normalization [37]. The residual formulation stabilizes training and facilitates efficient deep stacking in practical perception pipelines.

3.2.3. Attention-Based Recalibration

After parallel processing, the group outputs are concatenated and reshaped:

$$\widehat{\mathbf{F}}=\mathrm{Reshape}\left(\mathrm{Concat}({\widehat{\mathbf{F}}}_1,{\widehat{\mathbf{F}}}_2,\dots ,{\widehat{\mathbf{F}}}_G)\right)\in {\mathbb{R}}^{C\times H\times W}.$$

(19)

To further enhance discriminative capability without introducing substantial computational overhead, a lightweight dual-attention mechanism is applied for feature recalibration. The channel attention map $M_c$ and spatial attention map $M_s$ are computed as:

$$M_c=\sigma \left({\mathrm{Conv}}_{1\times 1}\left(\mathrm{Concat}\left(\mathrm{AvgPool}\left(\widehat{\mathbf{F}}\right),\mathrm{MaxPool}\left(\widehat{\mathbf{F}}\right)\right)\right)\right)\in {\mathbb{R}}^{C\times 1\times 1},$$

(20)

$$M_s=\sigma \left({\mathrm{Conv}}_{1\times 1}\left(\mathrm{Concat}\left({\mathrm{AvgPool}}_c\left(\widehat{\mathbf{F}}\right),{\mathrm{MaxPool}}_c\left(\widehat{\mathbf{F}}\right)\right)\right)\right)\in {\mathbb{R}}^{1\times H\times W},$$

(21)

where $\sigma$ denotes the sigmoid function, AvgPool and MaxPool are global average and max pooling along spatial dimensions, and ${\mathrm{AvgPool}}_c$ and ${\mathrm{MaxPool}}_c$ are average and max pooling along the channel dimension. The use of pooling-based statistics enables efficient global context encoding with minimal additional parameters.

The final recalibrated output is:

$$\tilde{\mathbf{F}}={\mathrm{Conv}}_{1\times 1}\left((M_c\otimes \widehat{\mathbf{F}})\oplus (M_s\otimes \widehat{\mathbf{F}})\right),$$

(22)

where ⊗ denotes broadcasting multiplication. This adaptive modulation refines both channel-wise and spatial responses while maintaining computational tractability for real-time deployment.

3.3. Global Multi-modal Interaction Network

While the CSM-Block captures long-range dependencies efficiently, fine-grained cross-modal alignment requires explicit modeling of the complementary and reliability-aware relationships between RGB and IR features. We propose the Global Multi-modal Interaction Network (GMIN) to achieve adaptive cross-modal modulation with limited additional computational cost. The architecture of GMIN is illustrated in Figure 4.

3.3.1. Dual-Branch Cross-Modal Attention

Given the local features from the RGB and IR modalities at scale l, denoted as $E_{\mathrm{RGB}}^L\in {\mathbb{R}}^{C\times H\times W}$ and $E_{\mathrm{IR}}^L\in {\mathbb{R}}^{C\times H\times W}$, GMIN first generates multiple attention representations for each modality. The pooling-guided design provides compact global descriptors that encode modality saliency with negligible computational overhead.

For the RGB branch, three parallel convolution paths produce query, key, and value-like representations:

$${\mathbf{Q}}_{\mathrm{rgb}}={\mathrm{Conv}}_q\left(E_{\mathrm{RGB}}^L\right),{\mathbf{K}}_{\mathrm{rgb}}^{\mathrm{avg}}=\mathrm{GAP}\left({\mathrm{Conv}}_k\left(E_{\mathrm{RGB}}^L\right)\right),{\mathbf{K}}_{\mathrm{rgb}}^{\max }=\mathrm{GMP}\left({\mathrm{Conv}}_k\left(E_{\mathrm{RGB}}^L\right)\right),$$

(23)

where $\mathrm{GAP}(·)$ and $\mathrm{GMP}(·)$ denote Global Average Pooling and Global Max Pooling, respectively. Similarly, for the IR branch:

$${\mathbf{Q}}_{\mathrm{ir}}={\mathrm{Conv}}_q\left(E_{\mathrm{IR}}^L\right),{\mathbf{K}}_{\mathrm{ir}}^{\mathrm{avg}}=\mathrm{GAP}\left({\mathrm{Conv}}_k\left(E_{\mathrm{IR}}^L\right)\right),{\mathbf{K}}_{\mathrm{ir}}^{\max }=\mathrm{GMP}\left({\mathrm{Conv}}_k\left(E_{\mathrm{IR}}^L\right)\right).$$

(24)

The attention weights are computed through softmax normalization:

$${\mathbf{A}}_{\mathrm{rgb}}^{\mathrm{avg}}=\mathrm{Softmax}\left({\mathbf{K}}_{\mathrm{rgb}}^{\mathrm{avg}}\right),{\mathbf{A}}_{\mathrm{rgb}}^{\max }=\mathrm{Softmax}\left({\mathbf{K}}_{\mathrm{rgb}}^{\max }\right),$$

(25)

$${\mathbf{A}}_{\mathrm{ir}}^{\mathrm{avg}}=\mathrm{Softmax}\left({\mathbf{K}}_{\mathrm{ir}}^{\mathrm{avg}}\right),{\mathbf{A}}_{\mathrm{ir}}^{\max }=\mathrm{Softmax}\left({\mathbf{K}}_{\mathrm{ir}}^{\max }\right).$$

(26)

3.3.2. Cross-Modal Interaction

The key innovation of GMIN lies in its cross-modal interaction mechanism. Instead of computing self-attention within each modality, we perform cross-attention by using the attention weights from one modality to modulate the features of the other. This asymmetric modulation enables each modality to emphasize informative cues from its complementary counterpart while suppressing unreliable responses:

$${\mathbf{V}}_{\mathrm{rgb}\leftarrow \mathrm{ir}}={\mathbf{Q}}_{\mathrm{rgb}}\otimes {\mathbf{A}}_{\mathrm{ir}}^{\mathrm{avg}}+{\mathbf{Q}}_{\mathrm{rgb}}\otimes {\mathbf{A}}_{\mathrm{ir}}^{\max },$$

(27)

$${\mathbf{V}}_{\mathrm{ir}\leftarrow \mathrm{rgb}}={\mathbf{Q}}_{\mathrm{ir}}\otimes {\mathbf{A}}_{\mathrm{rgb}}^{\mathrm{avg}}+{\mathbf{Q}}_{\mathrm{ir}}\otimes {\mathbf{A}}_{\mathrm{rgb}}^{\max },$$

(28)

where ⊗ denotes matrix multiplication. The use of pooled attention descriptors avoids full pairwise token interactions, thereby maintaining favorable computational characteristics.

The cross-modal interaction features are concatenated and processed:

$${\mathbf{V}}_{\mathrm{cross}}=\mathrm{Conv}\left(\mathrm{Concat}({\mathbf{V}}_{\mathrm{rgb}\leftarrow \mathrm{ir}},{\mathbf{V}}_{\mathrm{ir}\leftarrow \mathrm{rgb}})\right).$$

(29)

3.3.3. Modality-Specific Gating

To adaptively control the contribution of cross-modal information to each modality under varying sensing conditions, we generate modality-specific gating signals:

$${\mathbf{G}}_{\mathrm{rgb}}=\sigma \left(\mathrm{LN}\left(\mathrm{Conv}\left({\mathbf{V}}_{\mathrm{cross}}\right)\right)\right),$$

(30)

$${\mathbf{G}}_{\mathrm{ir}}=\sigma \left(\mathrm{LN}\left(\mathrm{Conv}\left({\mathbf{V}}_{\mathrm{cross}}\right)\right)\right),$$

(31)

where $\sigma$ is the sigmoid function and LN is Layer Normalization. The gating signals act as modality reliability estimators and modulate the original features through Hadamard product and residual connection:

$$E_{\mathrm{RGB}}^G={\mathbf{G}}_{\mathrm{rgb}}\odot \mathrm{Conv}\left({\mathbf{V}}_{\mathrm{rgb}\leftarrow \mathrm{ir}}\right)\oplus E_{\mathrm{RGB}}^L,$$

(32)

$$E_{\mathrm{IR}}^G={\mathbf{G}}_{\mathrm{ir}}\odot \mathrm{Conv}\left({\mathbf{V}}_{\mathrm{ir}\leftarrow \mathrm{rgb}}\right)\oplus E_{\mathrm{IR}}^L.$$

(33)

The residual formulation preserves modality-specific information while selectively incorporating complementary cues. The globally enhanced features $E_{\mathrm{RGB}}^G$ and $E_{\mathrm{IR}}^G$ are then concatenated to form the fused multi-modal features at each scale.

3.4. Cross-Modal Feature Fusion

After obtaining the enhanced features from the CSM-Block and GMIN at multiple scales, we perform Cross-scale Channel Feature Fusion (CCFF) to generate the final multi-scale fused representations. The GMIN modules operate at three scales ($P_3$, $P_4$, $P_5$), and the fused features are concatenated:

$${\mathbf{F}}_l^{\mathrm{fused}}=\mathrm{Concat}(E_{\mathrm{RGB},l}^G,E_{\mathrm{IR},l}^G),l\in \{3,4,5\}.$$

(34)

The multi-scale fused features $\{{\mathbf{F}}_3^{\mathrm{fused}},{\mathbf{F}}_4^{\mathrm{fused}},{\mathbf{F}}_5^{\mathrm{fused}}\}$ are further processed through the Efficient Transformer Encoder with self-attention (SA) and cross-attention (CA) mechanisms to capture intra-scale and cross-scale relationships. The encoder is configured to maintain a favorable balance between representational power and computational overhead for real-time perception pipelines.

3.5. Uncertainty-Minimal Query Selection

In DETR-based detectors, the quality of object queries significantly impacts detection performance [8]. When dealing with multi-modal inputs, inconsistency between modalities may introduce prediction variance that affects query reliability. We propose an Uncertainty-minimal Query Selection (UMQS) strategy that explicitly favors candidates exhibiting both high confidence and cross-modal consistency.

Given the fused encoder features, we first generate candidate queries through a scoring mechanism:

$$s_i={\mathrm{MLP}}_{\mathrm{cls}}\left({\mathbf{f}}_i\right)+\lambda ·{\mathrm{MLP}}_{\mathrm{loc}}\left({\mathbf{f}}_i\right),$$

(35)

where ${\mathbf{f}}_i$ is the i-th candidate feature, ${\mathrm{MLP}}_{\mathrm{cls}}$ and ${\mathrm{MLP}}_{\mathrm{loc}}$ predict classification and localization scores, and $\lambda$ is a balancing coefficient.

The uncertainty of each candidate is estimated by measuring the prediction variance across the two modalities:

$$u_i=\mathrm{Var}\left({\mathrm{MLP}}_{\mathrm{cls}}\left({\mathbf{f}}_i^{\mathrm{rgb}}\right),{\mathrm{MLP}}_{\mathrm{cls}}\left({\mathbf{f}}_i^{\mathrm{ir}}\right)\right)+\mathrm{Var}\left({\mathrm{MLP}}_{\mathrm{loc}}\left({\mathbf{f}}_i^{\mathrm{rgb}}\right),{\mathrm{MLP}}_{\mathrm{loc}}\left({\mathbf{f}}_i^{\mathrm{ir}}\right)\right).$$

(36)

This variance-based metric provides a lightweight estimation of cross-modal disagreement without requiring additional forward passes. The final query selection criterion combines the score and uncertainty:

$$q_i=s_i-\beta ·u_i,$$

(37)

where $\beta$ controls the penalty for high-uncertainty candidates. The top-K candidates with the highest $q_i$ values are selected as object queries for the decoder, thereby promoting stable and reliable detection initialization.

3.6. Separable Dynamic Decoder

For the detection decoder, we adopt the Separable Dynamic Decoder proposed by Hu et al. [38], which replaces the standard multi-head cross-attention in DETR decoders with an efficient dynamic convolution attention mechanism. This design further reduces quadratic attention overhead and improves scalability for high-resolution multi-modal feature maps. The architecture is shown in Figure 5.

The standard multi-head cross-attention computes:

$$\mathrm{MHCA}(Q,K,V)=\mathrm{softmax}\left({\displaystyle \frac{Q{W}^q{\left(K{W}^k\right)}^{\top }}{\sqrt{d}}}\right)V{W}^v,$$

(38)

which has $\mathcal{O}\left(N^{2}d\right)$ complexity. The Separable Dynamic Decoder replaces this with dynamic convolution attention:

$$\mathrm{DyConvAttn}(Q,V)=(V\ast r\left(Q{W}^d\right))\ast r\left(Q{W}^p\right),$$

(39)

where * denotes convolution, $r(·)$ is a reshape operation, $W^d$ and $W^p$ are learnable weight matrices that generate depth-wise and point-wise dynamic convolution kernels from the queries, respectively. This formulation reduces the computational complexity to $\mathcal{O}\left(Ndk\right)$ where k is the kernel size, enabling a more favorable accuracy–latency trade-off compared with standard cross-attention.

The decoder iteratively refines the selected queries through N blocks of dynamic convolution attention and self-attention:

$${\mathbf{q}}^{\left(n\right)}=\mathrm{FFN}\left(\mathrm{MHSA}\left(\mathrm{DyConvAttn}({\mathbf{q}}^{(n-1)},\mathbf{V})\right)\right),n=1,2,\dots ,N,$$

(40)

where ${\mathbf{q}}^{\left(0\right)}$ represents the initial queries from UMQS and $\mathbf{V}$ represents the aggregated multi-scale features. The iterative refinement improves localization precision while maintaining computational tractability.

The final detection outputs are produced by applying classification and regression heads to the refined queries:

$${\widehat{c}}_i={\mathrm{FC}}_{\mathrm{cls}}\left({\mathbf{q}}_i^{\left(N\right)}\right),{\widehat{b}}_i={\mathrm{FC}}_{\mathrm{box}}\left({\mathbf{q}}_i^{\left(N\right)}\right),$$

(41)

where ${\widehat{c}}_i$ and ${\widehat{b}}_i$ are the predicted class and bounding box for the i-th query.

3.7. Loss Function

Following the DETR paradigm [30], we employ the Hungarian algorithm [39] for bipartite matching between predictions and ground truth. The total loss function is defined as:

$${\mathcal{L}}_{\mathrm{total}}={\lambda }_{\mathrm{cls}}{\mathcal{L}}_{\mathrm{cls}}+{\lambda }_{\mathrm{box}}{\mathcal{L}}_{\mathrm{box}}+{\lambda }_{\mathrm{giou}}{\mathcal{L}}_{\mathrm{giou}},$$

(42)

where ${\mathcal{L}}_{\mathrm{cls}}$ is the focal loss [40] for classification, ${\mathcal{L}}_{\mathrm{box}}$ is the L1 loss for bounding box regression, and ${\mathcal{L}}_{\mathrm{giou}}$ is the Generalized IoU loss [41]. The loss weights are set as ${\lambda }_{\mathrm{cls}}=2.0$, ${\lambda }_{\mathrm{box}}=5.0$, and ${\lambda }_{\mathrm{giou}}=2.0$, which follow common DETR-based configurations.

4. Experiments

4.1. Datasets

We evaluate the proposed CMAFNet on two widely-used multi-modal object detection benchmarks for autonomous driving and real-world perception evaluation.

M3FD [4]. The Multi-scenario Multi-Modality Fusion Dataset (M3FD) is a challenging benchmark containing 4,200 pairs of well-aligned RGB and infrared images captured under diverse driving scenarios. The dataset covers six object categories: People, Car, Bus, Motorcycle, Lamp, and Truck. The images are collected across multiple scenarios including daytime, nighttime, overexposure, and adverse conditions (rain, fog, snow), which are critical for robustness assessment in automotive perception systems. Following the standard protocol, we use 2,940 image pairs for training and 1,260 for testing.

FLIR-Aligned [3]. The FLIR ADAS dataset provides paired RGB and thermal infrared images for autonomous driving perception. The aligned version contains 5,142 well-registered image pairs. The dataset includes three main categories: Person, Car, and Bicycle. We follow the standard split with 4,129 pairs for training and 1,013 pairs for testing. This dataset further evaluates the generalization capability of multi-modal detectors across diverse traffic environments.

4.2. Implementation Details

CMAFNet is implemented using PyTorch and trained on 4 NVIDIA A100 GPUs. The input images are resized to $640\times 640$ for both modalities to maintain a balance between detection accuracy and computational cost. The Dynamic Receptive Backbone uses a base channel dimension $D=64$. The CSM-Block employs $G=4$ channel groups with a state dimension of 16 for the selective state space model, which ensures linear-complexity modeling without excessive memory overhead. The GMIN module uses $C=256$ channels at each scale. The Separable Dynamic Decoder consists of $N=6$ decoder layers with 300 object queries ($K=300$), providing sufficient detection capacity while preserving real-time feasibility.

We use the AdamW optimizer [42] with an initial learning rate of $1\times {10}^{-4}$, weight decay of $1\times {10}^{-4}$, and a cosine annealing schedule [43]. The model is trained for 120 epochs with a batch size of 8. Data augmentation includes random horizontal flipping, random scaling (0.5–1.5×), color jittering, and mosaic augmentation [44]. The backbone is initialized with COCO pre-trained weights [45]. The uncertainty penalty coefficient $\beta$ is set to 0.5, and the score balancing coefficient $\lambda$ is set to 1.0.

4.3. Evaluation Metrics

We adopt the standard COCO-style evaluation metrics [45]: mAP₅₀ (mean Average Precision at IoU threshold 0.50), mAP₇₅ (at IoU threshold 0.75), and mAP_50:95 (averaged over IoU thresholds from 0.50 to 0.95 with step 0.05). We also report per-category AP₅₀ for detailed analysis. In addition to accuracy metrics, we report the number of parameters (Params), floating-point operations (FLOPs), and inference speed (FPS) measured under identical hardware settings to comprehensively evaluate computational efficiency and practical deployability.

4.4. Comparison with State-of-the-Art Methods

We compare CMAFNet with a comprehensive set of state-of-the-art methods spanning multiple categories, including two-stage detectors, single-stage detectors, YOLO-based detectors, Transformer-based detectors, and dedicated multi-modal fusion methods. All comparison methods are evaluated under the same experimental settings to ensure a fair assessment of both detection accuracy and computational complexity.

4.4.1. Results on M3FD Dataset

Table 1 presents the overall comparison results on the M3FD dataset. CMAFNet achieves the highest mAP₅₀ of 83.9% and the highest mAP_50:95 of 52.3%, while maintaining competitive parameter size and FLOPs compared with other high-performing methods.

Among the two-stage detectors, MBNet achieves 74.2% mAP₅₀ by addressing modality imbalance, but its two-stage pipeline typically introduces higher inference latency. Single-stage detectors such as FCOS achieve 72.1% mAP₅₀ with improved efficiency but comparatively lower accuracy. YOLO-based methods show progressive improvements, with YOLOv9-E reaching 78.4% mAP₅₀ at increased computational cost. Transformer-based detectors demonstrate strong performance, with RT-DETR-L achieving 80.1% mAP₅₀ while emphasizing real-time optimization. Among multi-modal methods, TFDet achieves 82.4% mAP₅₀ through target-aware fusion. CMAFNet surpasses TFDet by 1.5% in mAP₅₀ and 1.5% in mAP_50:95, indicating that the proposed cross-modal alignment and uncertainty-aware query selection provide consistent performance gains under comparable model complexity.

Table 2 presents the per-category AP₅₀ results on M3FD.

CMAFNet achieves the best performance across all six categories. Notably, for the challenging People category, CMAFNet achieves 81.3% AP₅₀, surpassing TFDet by 1.8%. This improvement is particularly important for safety-critical autonomous driving scenarios, where pedestrian detection reliability directly affects collision avoidance systems. The consistent gains across all categories indicate that the proposed cross-modal interaction mechanism generalizes well across object scales and environmental conditions.

To provide a more intuitive comparison, Figure 6 presents a radar chart of the per-category AP₅₀ for the top-performing methods on M3FD.

4.4.2. Results on FLIR-Aligned Dataset

Table 3 presents the comparison results on the FLIR-Aligned dataset. CMAFNet achieves the highest mAP₅₀ of 84.2% and the highest mAP_50:95 of 53.1%, while maintaining competitive parameter size and FLOPs compared with other high-performing methods.

Compared with TFDet, CMAFNet improves mAP₅₀ by 1.1% and mAP_50:95 by 1.5%, indicating that the proposed cross-modal alignment and uncertainty-aware query selection mechanisms consistently enhance detection accuracy under comparable model complexity. The consistent performance gains across both M3FD and FLIR-Aligned benchmarks suggest that the framework generalizes well across datasets with different scene distributions and annotation characteristics.

Table 4 shows the per-category results on FLIR-Aligned.

CMAFNet achieves the best performance across all three categories. The improvement is particularly notable for the Person category (82.1% vs. 80.6%), which is directly related to pedestrian detection reliability in safety-critical autonomous driving scenarios. The gains in the Car and Bicycle categories further demonstrate that the proposed cross-modal interaction mechanism enhances detection consistency across object scales and thermal–appearance variations.

Figure 7 provides a cross-dataset comparison and component-wise ablation analysis.

4.5. Ablation Studies

We conduct comprehensive ablation studies on the M3FD dataset to quantitatively analyze the contribution of each proposed component and to verify the rationality of the overall architectural design.

4.5.1. Component-wise Ablation

Table 5 presents the ablation results for each key component of CMAFNet.

The baseline model with simple concatenation fusion achieves 75.2% mAP₅₀. Replacing the standard backbone with DRB improves performance to 78.1% (+2.9%), indicating that adaptive receptive field modeling enhances multi-scale feature representation under multi-modal inputs.

Introducing the CSM-Block further increases performance to 80.3% (+2.2%), demonstrating the effectiveness of selective state space modeling for efficient long-range cross-modal dependency capture. Incorporating the GMIN module provides an additional 1.8% improvement (82.1%), confirming that explicit cross-modal interaction is more effective than implicit fusion alone.

The UMQS strategy contributes a further 0.7% gain (82.8%), suggesting that uncertainty-aware query initialization improves detection stability in multi-modal scenarios. Finally, replacing the vanilla DETR decoder with the Separable Dynamic Decoder yields the final 1.1% improvement (83.9%). The consistent and progressive gains across all components indicate that the overall performance improvement arises from complementary architectural design rather than isolated modifications.

The cross-dataset comparison and ablation results are jointly visualized in Figure 7.

4.5.2. Analysis of CSM-Block Design

Table 6 analyzes the impact of different design choices in the CSM-Block.

Compared with standard self-attention, the CSM-Block with $G=4$ improves mAP₅₀ by 2.4% while significantly increasing inference speed (29.3 FPS vs. 18.2 FPS), highlighting the efficiency advantage of linear-complexity selective state space modeling over quadratic attention mechanisms.

The channel splitting strategy plays an important role. Using $G=4$ groups outperforms the non-split variant ($G=1$) by 2.1%, indicating that parallel group-wise modeling enhances representation diversity and cross-modal interaction. However, excessive splitting ($G=8$) slightly reduces performance due to decreased per-group channel capacity, suggesting a trade-off between modeling granularity and feature richness.

The attention-based recalibration mechanism contributes an additional 1.5% improvement (83.9% vs. 82.4%), demonstrating that channel and spatial reweighting effectively refine the aggregated group-wise features.

4.5.3. Analysis of GMIN Design

Table 7 evaluates different fusion strategies in the GMIN module.

The full GMIN module, which integrates GAP- and GMP-guided cross-attention with modality-specific gating, achieves the best performance. Compared with simple fusion strategies such as concatenation and element-wise addition, explicit cross-modal interaction consistently improves detection accuracy, indicating that structured feature exchange is more effective than implicit blending.

Using only GAP or only GMP leads to 1.1% and 1.4% lower mAP₅₀, respectively, suggesting that average statistical responses and salient feature responses provide complementary cues for cross-modal alignment. The modality-specific gating mechanism further contributes 0.8% improvement over the variant without gating, demonstrating that adaptive control of modality contribution enhances robustness across object categories and illumination conditions.

Figure 8 jointly visualizes the fusion strategy comparison and the $\beta$ sensitivity analysis. As shown in Figure 8(a), the full GMIN design consistently outperforms simplified variants across both mAP₅₀ and mAP_50:95 metrics. Standard cross-attention yields moderate improvement, while the dual-pooling guided cross-attention design further enhances discriminative feature transfer between modalities.

Figure 8(b) reveals that the uncertainty penalty coefficient $\beta$ in UMQS exhibits a clear inverted-U sensitivity pattern across all three evaluation metrics. When $\beta$ is too small, uncertain candidates are insufficiently penalized, leading to unstable query initialization. Conversely, excessively large $\beta$ overly suppresses candidate diversity and may discard informative proposals. The optimal balance is achieved at ${\beta }^{\ast }=0.5$, where the model attains peak performance of 83.9% mAP₅₀, 64.5% mAP₇₅, and 52.3% mAP_50:95.

4.5.4. Analysis of Uncertainty-Minimal Query Selection

Table 8 evaluates the UMQS strategy with different uncertainty penalty coefficients $\beta$.

The UMQS strategy consistently improves performance over the baseline without uncertainty consideration ($\beta =0$). Introducing moderate uncertainty penalization improves query reliability and stabilizes detection performance. The optimal $\beta =0.5$ achieves 83.9% mAP₅₀, representing a 1.8% improvement over the no-uncertainty variant. Both overly small and overly large penalty coefficients result in reduced performance, indicating that balanced uncertainty control is essential for effective query selection.

The detailed $\beta$ sensitivity results are jointly presented in Figure 8(b).

4.6. Efficiency Analysis

Table 9 compares the computational efficiency of CMAFNet with representative multi-modal detection methods.

CMAFNet achieves the highest mAP₅₀ while maintaining the fastest inference speed (29.3 FPS) among the compared multi-modal methods. This efficiency advantage is primarily attributed to the linear-complexity CSM-Block and the dynamic convolution attention mechanism in the Separable Dynamic Decoder, which reduce the quadratic cost of conventional attention.

Compared with ICAFusion, CMAFNet is substantially faster while also achieving higher detection accuracy, demonstrating a favorable accuracy–efficiency trade-off.

Figure 9 provides a comprehensive visualization of the accuracy–speed trade-off and robustness under different environmental conditions.

4.7. Qualitative Results

We present qualitative detection results to visually demonstrate the effectiveness of CMAFNet under various challenging real-world conditions.

4.7.1. Results on M3FD Dataset

Figure 10 and Figure 11 show the detection results of CMAFNet on RGB and IR images from the M3FD dataset, respectively.

As illustrated in Figure 10, CMAFNet maintains stable detection performance across diverse driving scenarios, including strong illumination, nighttime scenes, rain, and fog. The model accurately detects objects of varying scales and densities, from small distant pedestrians to large nearby vehicles, demonstrating robust multi-scale representation capability.

Figure 11 further shows that the infrared modality provides reliable thermal signatures even when RGB visibility is degraded. CMAFNet effectively integrates complementary thermal cues, leading to accurate detection in low-contrast and low-illumination environments. These results visually confirm the advantage of cross-modal interaction under adverse conditions.

4.7.2. Results on FLIR-Aligned Dataset

Figure 12 and Figure 13 present the detection results on the FLIR-Aligned dataset.

On the FLIR-Aligned dataset, CMAFNet demonstrates consistent detection quality across both modalities. As shown in Figure 12, the model accurately detects pedestrians, vehicles, and bicycles under varying lighting and traffic conditions. Figure 13 highlights the effectiveness of thermal cues for pedestrian detection, particularly in nighttime scenes where RGB information is limited. The qualitative results align with the quantitative improvements reported in Section 4.4, confirming that the proposed cross-modal fusion strategy enhances both robustness and discriminative capability.

4.8. Robustness Analysis under Different Conditions

To further evaluate robustness, we analyze the performance of CMAFNet under different environmental conditions on the M3FD dataset, which provides scenario-level annotations.

As shown in Table 10, CMAFNet achieves the best performance across all environmental conditions. The gains are particularly notable under challenging scenarios. Compared with TFDet, CMAFNet improves performance by 2.4% in nighttime conditions and 2.2% in challenge scenarios (rain, fog, snow). These improvements indicate that the proposed cross-modal interaction and uncertainty-aware query selection mechanisms effectively enhance robustness when one modality is partially degraded.

The robustness heatmap is jointly presented in Figure 9(b), where CMAFNet consistently achieves strong performance across different environmental categories, further confirming its stability and generalization capability.

5. Discussion

The experimental results demonstrate the effectiveness of CMAFNet for multi-modal object detection in autonomous driving. Beyond the quantitative improvements, several structural insights can be drawn from the proposed design.

Effectiveness of State Space Models for Cross-Modal Fusion. The CSM-Block demonstrates that selective state space models provide an efficient alternative to quadratic-complexity attention mechanisms for modeling long-range cross-modal dependencies. As shown in Table 6, the CSM-Block achieves 2.4% higher mAP₅₀ than standard self-attention while delivering 61% faster inference. This performance gain is not merely computational but structural: the selective scanning mechanism enables adaptive information propagation across spatial positions, while the channel-splitting strategy promotes representation diversity and reduces feature interference. These results suggest that state space modeling is particularly well-suited for multi-modal fusion, where efficient global context modeling is critical.

Importance of Fine-Grained Cross-Modal Interaction. The GMIN module addresses a common limitation in existing fusion strategies—namely, insufficient explicit alignment between modalities. The ablation study (Table 7) confirms that simple fusion operations such as concatenation or element-wise addition are inadequate for capturing complex complementary relationships between RGB and IR features. By integrating GAP- and GMP-guided cross-attention, GMIN captures both global statistical responses and salient discriminative cues. The modality-specific gating mechanism further stabilizes fusion by dynamically regulating the contribution of each modality. This structured interaction is particularly beneficial under distribution shifts such as illumination variation.

Robustness under Adverse Conditions. The robustness analysis (Table 10) shows that CMAFNet exhibits the largest performance gains under nighttime and challenging weather conditions. This behavior reflects the adaptive nature of the proposed fusion strategy. When one modality is degraded, the gating mechanism increases reliance on the complementary modality, effectively mitigating information loss. Such adaptive cross-modal balancing is essential for safety-critical autonomous driving systems operating under unpredictable environmental conditions.

Efficiency–Accuracy Trade-off. Although CMAFNet introduces additional fusion modules, it maintains competitive real-time performance (29.3 FPS) as shown in Table 9. This efficiency is largely attributed to the linear-complexity state space modeling in the CSM-Block and the dynamic convolution attention in the Separable Dynamic Decoder [38]. The results indicate that improved cross-modal interaction does not necessarily require heavy attention-based architectures; carefully designed linear mechanisms can achieve a favorable accuracy–efficiency balance.

Limitations and Future Work. Despite achieving state-of-the-art results on RGB–IR benchmarks, several limitations remain. First, the current framework focuses on dual-modality fusion; extending CMAFNet to incorporate additional sensing modalities such as LiDAR or radar may further enhance robustness. Second, the method assumes well-aligned RGB–IR pairs; developing alignment-robust or alignment-free fusion strategies is an important practical direction. Third, while the current model achieves real-time inference on high-end GPUs, further model compression and architectural simplification would be valuable for deployment on edge platforms in resource-constrained autonomous vehicles.

6. Conclusions

In this paper, we proposed CMAFNet, a Cross-Modal Alignment and Fusion Network for robust multi-modal object detection in autonomous driving. The proposed framework integrates three key technical contributions: (1) the Channel-Split Mamba Block, which leverages selective state space modeling to efficiently capture long-range cross-modal dependencies with linear complexity; (2) the Global Multi-modal Interaction Network, which performs fine-grained cross-modal alignment via dual-branch attention guided by global pooling; and (3) the Uncertainty-minimal Query Selection strategy, which improves detection reliability by selecting object queries based on cross-modal prediction consistency.

Combined with a Dynamic Receptive Backbone and a Separable Dynamic Decoder, CMAFNet achieves state-of-the-art performance on both the M3FD (83.9% mAP₅₀) and FLIR-Aligned (84.2% mAP₅₀) benchmarks while maintaining competitive real-time efficiency. Extensive ablation studies validate the effectiveness and complementarity of each component, and robustness analysis demonstrates consistent improvements under adverse environmental conditions. These results suggest that structured cross-modal alignment, efficient state space modeling, and uncertainty-aware query selection together form a promising direction for reliable multi-modal perception in autonomous driving.