TopoGraph-Fusion: Hierarchical Graph-Guided Dual-Modal Object Detection for Robust Autonomous Driving Perception

1. Introduction

Autonomous driving perception must recognize objects across a wide range of visual and thermal conditions. Road agents may appear under sunny noon, night, headlight glare, heavy rain, fog, wet reflective pavement, camera exposure drift, and thermal contrast changes. RGB cameras capture rich texture, color, lane context, and semantic appearance. Thermal infrared cameras capture heat signatures that remain visible in darkness and in many low-illumination scenes. The complementarity between these two modalities has motivated a long line of multispectral and RGB–thermal object detectors [1,2,3,4,5,6,7]. However, complementarity alone does not make fusion reliable. A central difficulty is that useful evidence in the two modalities is not always synchronized at every pixel, channel, scale, or object instance.

In visible–thermal driving data, an object may be semantically clear in RGB but thermally weak, or thermally salient but visually indistinguishable from the background. Pedestrians may be tiny in wide-angle road images, vehicles may be saturated by headlights, and aerial scenes may contain multiple small objects with weak category cues. These conditions produce a fusion dilemma. Early fusion assumes low-level alignment and mixes noisy raw signals. Middle fusion combines feature tensors but often treats all channels and locations as equally comparable. Late fusion merges predictions after much cross-modal structure has already been lost. Attention-based methods improve adaptive weighting, yet many of them still compute weights from local feature statistics rather than from a persistent topology of relations. Consequently, the detector may learn which modality is strong at one location but fail to reason about how thermal edges, RGB textures, semantic channels, and pyramid levels support each other across the image.

This work starts from the following hypothesis: robust dual-modal perception can be effectively formulated as a topology reasoning problem. Instead of asking only how to concatenate RGB and thermal features, we ask how channels, spatial regions, semantic levels, and modalities should be connected before fusion. The framework shown in Figure 1 implements this hypothesis through a hierarchical graph attention network. Two backbone streams first encode RGB and thermal inputs. Then the model builds graph nodes representing complementary enhancement units. At lower graph levels, channel topology captures correlations among semantic responses. At intermediate levels, spatial and channel graph-guided attention encodes long-range relational evidence. At higher levels, graph-guided feature pyramid propagation transfers reliable object cues across scales. The final detector therefore receives features that have been routed through relation-aware pathways rather than blindly fused.

Figure 1 is the conceptual spine of the paper. The upper branch shows the end-to-end detection pipeline: RGB and TIR inputs are processed by dual CNN backbones, refined by a hierarchical graph attention network, aggregated through the Channel Topology Graph Aggregation module, and decoded by prediction heads. The lower branch decomposes the hierarchy into a router, a channel enhancement unit, a spatial enhancement unit, and a cross-modal enhancement fusion unit. The router estimates graph routing weights from global pooled descriptors. The enhancement units then decide how evidence should flow between graph nodes. This design is used to avoid treating the model as a disconnected stack of modules. Each module serves one layer of the same topology-guided story: channel topology decides what semantic responses are related, spatial topology decides where distant evidence should communicate, feature-pyramid topology decides which scales should exchange information, and cross-modal topology decides how RGB and TIR should cooperate.

The proposed TopoGraph-Fusion detector is evaluated on four public datasets. M3FD contains multi-scenario visible–infrared driving scenes introduced with the TarDAL benchmark [8]. FLIR ADAS provides aligned visible and thermal imagery for road-scene perception [9]. RGBTDronePerson evaluates RGB–thermal person detection from UAV viewpoints [10]. VEDAI512 evaluates vehicle detection in aerial imagery [11]. Together these datasets test road-level, night, adverse-weather, drone, and remote-sensing regimes. The comparison protocol includes classical two-stage detectors, single-stage YOLO-family detectors, transformer-based detectors, and recent multispectral fusion methods such as DAMSDet and cross-modal information complementary recalibration [6,7].

The main contributions are summarized as follows.

• We propose a topology-guided interpretation of RGB–thermal object detection, where fusion is modeled through channel, spatial, scale, and modality graphs rather than by direct tensor concatenation.
• We design a Channel Topology Graph Aggregation module that constructs channel-wise affinity, refines channel dependencies, and aggregates relation-aware channel features for dual-modal enhancement.
• We introduce spatial and channel graph-guided global attention mechanisms that convert graph affinity into relation-aware attention maps, improving long-range reasoning under low contrast, occlusion, and modality imbalance.
• We develop a Graph-Guided Feature Pyramid Network that replaces fixed top-down and bottom-up routes with graph-informed cross-scale communication.
• We provide multi-dataset comparisons, ablations, complexity analysis, qualitative visualizations, and a topology-regularized loss function to assess the proposed design under several RGB–thermal and aerial detection settings.

2. Related Work

2.1. Visible–Thermal and Multispectral Object Detection

Visible–thermal object detection has been studied in pedestrian detection, road-scene detection, aerial perception, and surveillance. Early multispectral pedestrian detectors used hand-crafted or shallow features, including aggregated channel features [1,12]. Deep fusion methods then learned convolutional representations from visible and thermal images, with early, halfway, and late fusion becoming common baselines [2,3]. Illumination-aware fusion introduced explicit reliability estimation for day and night scenes [4]. Aligned Region CNN further emphasized cross-modal alignment and region-level interaction [5]. These works established the importance of RGB–thermal complementarity but often used fixed fusion positions or fixed reliability heuristics.

Recent multispectral detectors increasingly use adaptive fusion and transformer-style interaction. CFT employed cross-modality fusion transformers to model visible–infrared interactions [13]. ICAFusion introduced illumination-aware cross-modality fusion for infrared–visible object detection [14]. DAMSDet proposed dynamic adaptive multispectral detection for complex modality conditions [6]. WACV 2025 work on cross-modal information complementary and cosine similarity recalibration explicitly improved multispectral complementarity modeling [7]. These methods show that adaptive fusion is critical, yet many still build interactions from dense attention or learned weighting without explicitly regularizing the topology of feature relations. TopoGraph-Fusion complements this trend by making graph topology the organizing principle of the fusion network.

2.2. Object Detection Backbones and Detection Heads

Modern object detection has developed along two dominant directions. CNN-based single-stage detectors, represented by SSD [15], RetinaNet [16], YOLOv5 [17], YOLOv7 [18], YOLOv8 [19], YOLOv9 [20], YOLOv10 [21], YOLOv11 [22], YOLOv12 [23], and YOLOv13 [24], prioritize speed and dense prediction efficiency. Two-stage detectors such as Faster R-CNN [25], Cascade R-CNN [26], and Mask R-CNN [27] remain strong for high-quality proposals and localization. Transformer-based detectors such as DETR [28], Deformable DETR [29], DINO [30], RT-DETR [31], RT-DETRv2 [32], and D-FINE [33] reduce hand-crafted post-processing and improve global object reasoning. Recent applied detection studies also show that lightweight attention, YOLO-family improvements, state-space modeling, and multimodal fusion are becoming important in domains beyond traffic scenes, including agriculture, medical imagery, smart-city supervision, counterfeit recognition, patent knowledge graphs, social-platform multimodal analysis, and sports perception [34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49].

These detectors are powerful generic baselines but are usually designed for single-modal inputs. When they are extended to RGB–T detection, their main architectural question becomes where and how to fuse modality-specific features. Our framework is compatible with common detection heads, but it changes the fusion body: the dual-modal features are refined by graph-guided relational modules before being fed to the prediction layers. Thus the proposed contribution is orthogonal to whether the final head is anchor-based, anchor-free, or transformer-based.

2.3. Graph Reasoning and Attention Mechanisms

Graph neural networks provide a natural representation for non-Euclidean dependencies. Graph convolutional networks [50], graph attention networks [51], and dynamic graph modules have been used to model semantic relations, object interactions, and long-range dependencies. The broader graph-theoretic literature also shows that topology, connectivity, orientation, Hamiltonian structure, and diagnosability strongly affect how information can be routed and preserved in complex networks [52,53,54,55]. Attention mechanisms, beginning from sequence modeling [56] and later extended to vision transformers [57,58], offer dense pairwise communication. In dense perception, attention is often powerful but computationally expensive and may learn noisy all-to-all relations. Graph-guided attention can be seen as a structured compromise: it allows long-range reasoning but encourages the network to communicate through meaningful affinities.

In RGB–thermal perception, graph reasoning has an additional advantage. Thermal and visible features can disagree because of illumination, material, emission, sensor noise, and alignment error. A graph can preserve modality-specific evidence while allowing selected nodes to exchange information. The proposed channel graph, spatial graph, and feature-pyramid graph therefore act as relational priors for robust fusion. This design is related in spirit to adaptive graph attention studies, spatio-temporal graph attention for non-stationary systems, and recent multimodal graph work [59,60,61,62,63,64], but it is specialized for dense RGB–thermal object detection.

2.4. Feature Pyramid Networks for Multi-Scale Detection

Multi-scale representation is essential for road and aerial detection because object sizes vary from small pedestrians to large vehicles. Feature Pyramid Networks fuse semantic and spatial information through a top-down pathway [65]. PANet adds bottom-up augmentation [66]. NAS-FPN searches pyramid connections [67]. BiFPN introduces weighted bidirectional feature fusion [68]. Although these pyramids are effective, their connection patterns are fixed or globally learned. They do not explicitly adapt pyramid communication to the current cross-modal topology.

Figure 5 motivates our Graph-Guided Feature Pyramid Network. Compared with PANet and BiFPN, GGFPN constructs graph-guided connections among pyramid nodes. The graph can route high-level semantic cues to low-level small-object features, or transfer fine thermal contours upward when visible texture is unreliable. This is especially valuable in multispectral detection, where the informative scale can differ between modalities.

3. Materials and Methods

3.1. Problem Formulation

Let an RGB–thermal training set be

$$\mathcal{D}={\left\{({\mathbf{I}}_i^{\mathrm{RGB}},{\mathbf{I}}_i^{\mathrm{TIR}},{\mathbf{Y}}_i)\right\}}_{i=1}^N,$$

(1)

where ${\mathbf{I}}_i^{\mathrm{RGB}}\in {\mathbb{R}}^{H\times W\times 3}$ is the visible image, ${\mathbf{I}}_i^{\mathrm{TIR}}\in {\mathbb{R}}^{H\times W\times 1}$ is the thermal image, and ${\mathbf{Y}}_i={\{(b_j,c_j)\}}_{j=1}^{M_i}$ contains bounding boxes and class labels. The goal is to learn a detector

$${\mathcal{F}}_{\Theta }:\left({\mathbf{I}}^{\mathrm{RGB}},{\mathbf{I}}^{\mathrm{TIR}}\right)\mapsto {\left\{({\widehat{b}}_k,{\widehat{c}}_k,{\widehat{s}}_k)\right\}}_{k=1}^K,$$

(2)

that predicts bounding boxes, object categories, and confidence scores. Instead of representing fusion as a single operator $\varphi ({\mathbf{F}}^{\mathrm{RGB}},{\mathbf{F}}^{\mathrm{TIR}})$, we represent it as a hierarchy of graph transformations:

$$\mathbf{Z}={\mathcal{H}}_{\Theta }\left({\mathcal{G}}_c,{\mathcal{G}}_s,{\mathcal{G}}_p,{\mathcal{G}}_m;{\{{\mathbf{F}}_l^{\mathrm{RGB}},{\mathbf{F}}_l^{\mathrm{TIR}}\}}_{l=3}^7\right),$$

(3)

where ${\mathcal{G}}_c$, ${\mathcal{G}}_s$, ${\mathcal{G}}_p$, and ${\mathcal{G}}_m$ denote channel, spatial, pyramid, and modality graphs, respectively.

3.2. Overall Architecture

As shown in Figure 1, TopoGraph-Fusion follows a dual-stream architecture. The RGB stream and the TIR stream first extract multi-level features:

$${\mathbf{F}}_l^m=B_l^m\left({\mathbf{I}}^m\right),m\in \{\mathrm{RGB},\mathrm{TIR}\},l\in \{3,4,5,6,7\}.$$

(4)

The two streams may share the same macro-architecture but do not share all parameters, because visible and thermal images have different statistics. At each pyramid level, modality-specific features are normalized and projected into a common embedding dimension:

$${\tilde{\mathbf{F}}}_l^m={Conv}_{1\times 1}\left(Norm\left({\mathbf{F}}_l^m\right)\right).$$

(5)

The hierarchical graph attention network then processes these features through four relation layers. The router first estimates the contribution of node types from pooled descriptors:

$${\mathbf{r}}_l=ReLU\left(tanh\left(MLP\left(Concat\left[AvgPool({\tilde{\mathbf{F}}}_l),MaxPool({\tilde{\mathbf{F}}}_l)\right]\right)\right)\right).$$

(6)

The vector ${\mathbf{r}}_l$ controls how strongly channel, spatial, pyramid, and modality graph units participate at level l. This router is intentionally simple: it avoids making the graph hierarchy a black-box controller and instead provides a compact reliability signal.

The final fused feature at level l is computed as

$${\mathbf{Z}}_l={\omega }_{c,l}{\mathbf{Z}}_{c,l}+{\omega }_{s,l}{\mathbf{Z}}_{s,l}+{\omega }_{p,l}{\mathbf{Z}}_{p,l}+{\omega }_{m,l}{\mathbf{Z}}_{m,l},{\omega }_{·,l}=softmax({\mathbf{r}}_l),$$

(7)

where ${\mathbf{Z}}_{c,l}$ is produced by CTGA, ${\mathbf{Z}}_{s,l}$ by spatial graph-guided attention, ${\mathbf{Z}}_{p,l}$ by GGFPN, and ${\mathbf{Z}}_{m,l}$ by cross-modal enhancement fusion. The prediction head receives ${\{{\mathbf{Z}}_l\}}_{l=3}^7$ and outputs classification and localization predictions.

3.3. Channel Topology Graph Aggregation

Figure 2 illustrates the CTGA module. Given a feature tensor $\mathbf{X}\in {\mathbb{R}}^{N\times C}$, where $N=HW$ is the number of spatial tokens and C is the number of channels, the module constructs a channel topology graph ${\mathcal{G}}_c=({\mathcal{V}}_c,{\mathcal{E}}_c)$. Each node corresponds to a transformed channel group. The first step maps features into a reduced channel space:

$$\tilde{\mathbf{X}}=\mathcal{T}(\mathbf{X})=\sigma \left(\mathbf{X}{\mathbf{W}}_t\right),\tilde{\mathbf{X}}\in {\mathbb{R}}^{N\times C^{\prime }},$$

(8)

where ${\mathbf{W}}_t\in {\mathbb{R}}^{C\times C^{\prime }}$ and $\sigma (·)$ is a nonlinear activation. The reduction makes graph aggregation efficient and prevents high-dimensional channel noise from dominating affinity estimation.

The initial affinity matrix is computed from two channel embeddings:

$$\mathbf{Q}=\Phi (\mathbf{X}),\mathbf{K}=\Psi (\mathbf{X}),{\mathbf{A}}_c=softmax\left({\displaystyle \frac{{\mathbf{Q}}^{\top }\mathbf{K}}{\sqrt{d_c}}}\right).$$

(9)

The entry $a_{ij}$ indicates how much the i-th channel response should receive information from the j-th response. For RGB–thermal perception, this relation is important because heat-sensitive channels, edge-sensitive channels, and texture-sensitive channels may not activate simultaneously. The topology refinement module $\mathcal{R}$ then updates the adjacency:

$${\mathbf{R}}_c=\mathcal{R}\left({\mathbf{A}}_c,\mathbf{Q},\mathbf{K}\right)=sigmoid\left({\mathbf{A}}_c+MLP\left([\mathbf{Q};\mathbf{K};{\mathbf{A}}_c]\right)\right).$$

(10)

The refined graph is not forced to be symmetric. This asymmetry allows a thermal-salient channel to guide a visually weak channel without requiring the reverse influence to be equally strong.

The channel-wise aggregation output is

$${\mathbf{Z}}_c={\mathcal{A}}_c(\tilde{\mathbf{X}},{\mathbf{R}}_c)={Concat}_{j=1}^{C^{\prime }}\left({\displaystyle \sum _{i=1}^{C^{\prime }}}{r}_{ij}{\tilde{\mathbf{X}}}_{:,i}{\mathbf{W}}_a^{(i,j)}\right).$$

(11)

In practice, the aggregation uses grouped matrix multiplication and a residual path:

$${\widehat{\mathbf{X}}}_c=Norm\left(\mathbf{X}+{Conv}_{1\times 1}({\mathbf{Z}}_c)\right).$$

(12)

This residual design preserves the original modality-specific representation while adding graph-mediated channel evidence.

3.4. Spatial Graph-Guided Global Attention

The SGGA module in Figure 3(a) encodes long-range spatial dependencies. Given $\mathbf{F}\in {\mathbb{R}}^{H\times W\times C}$, we flatten spatial dimensions to obtain $\mathbf{U}\in {\mathbb{R}}^{N\times C}$. A spatial affinity matrix is defined as

$${\mathbf{A}}_s=softmax\left({\displaystyle \frac{{\theta }_s(\mathbf{U}){\varphi }_s{(\mathbf{U})}^{\top }}{\sqrt{d_s}}}\right),{\mathbf{A}}_s\in {\mathbb{R}}^{N\times N}.$$

(13)

Rows and columns of ${\mathbf{A}}_s$ represent global spatial relationships. For a target token n, relation slices ${\mathbf{A}}_s(n,:)$ and ${\mathbf{A}}_s(:,n)$ encode outgoing and incoming dependencies. The relation-aware spatial feature is then generated by reshaping these slices back to the spatial layout:

$${\mathbf{R}}_s^{\mathrm{out}}(n)=Reshape\left({\mathbf{A}}_s(n,:)\right),{\mathbf{R}}_s^{\mathrm{in}}(n)=Reshape\left({\mathbf{A}}_s(:,n)\right).$$

(14)

The two relation maps are concatenated with an embedded feature:

$${\mathbf{M}}_s={Conv}_{3\times 3}\left([{\eta }_s(\mathbf{F});{\mathbf{R}}_s^{\mathrm{out}};{\mathbf{R}}_s^{\mathrm{in}}]\right),$$

(15)

and the spatial attention map is

$$\mathbf{S}=sigmoid\left({Conv}_{1\times 1}({\mathbf{M}}_s)\right).$$

(16)

The output is computed by a gated residual transformation:

$${\widehat{\mathbf{F}}}_s=\mathbf{F}+\mathbf{S}\odot {Conv}_{3\times 3}(\mathbf{F}).$$

(17)

This formulation is especially useful when RGB and TIR provide complementary evidence at distant locations. For example, a pedestrian may be a warm vertical region in thermal imagery and a weak silhouette in RGB imagery. The graph can connect that region to similar vertical or motion-related regions across the scene, improving the confidence of small or partially occluded objects.

3.5. Channel Graph-Guided Global Attention

The CGGA module in Figure 3(b) applies relation-guided attention along the channel dimension. Let $\mathbf{V}\in {\mathbb{R}}^{C\times N}$ be a channel-first representation. The channel affinity is

$${\mathbf{A}}_{ch}=softmax\left({\displaystyle \frac{{\theta }_{ch}(\mathbf{V}){\varphi }_{ch}{(\mathbf{V})}^{\top }}{\sqrt{d_{ch}}}}\right),{\mathbf{A}}_{ch}\in {\mathbb{R}}^{C\times C}.$$

(18)

For channel c, we extract incoming and outgoing relation descriptors:

$${\mathbf{R}}_{ch}^{\mathrm{out}}(c)={\mathbf{A}}_{ch}(c,:),{\mathbf{R}}_{ch}^{\mathrm{in}}(c)={\mathbf{A}}_{ch}(:,c).$$

(19)

These descriptors are concatenated with channel embeddings and projected to channel attention:

$${\mathbf{C}}_a=sigmoid\left({Conv}_{1\times 1}\left([{\eta }_{ch}(\mathbf{F});{\mathbf{R}}_{ch}^{\mathrm{out}};{\mathbf{R}}_{ch}^{\mathrm{in}}]\right)\right).$$

(20)

The output is

$${\widehat{\mathbf{F}}}_{ch}=\mathbf{F}+{\mathbf{C}}_a\odot {Conv}_{1\times 1}(\mathbf{F}).$$

(21)

The difference between CTGA and CGGA is deliberate. CTGA constructs and aggregates channel topology as a graph message-passing module. CGGA converts channel graph relations into attention gates for global reweighting. Together they allow the network to both exchange information across channels and selectively amplify channels that are reliable in the current cross-modal scene. Figure 4 visualizes this design intuition in the style of attention-response analysis. Compared with naive tensor concatenation and dense attention, the graph-guided response is sharper around the object center while preserving axis-aligned relational traces, indicating that the model learns a structured topology rather than a diffuse activation map.

3.6. Graph-Guided Feature Pyramid Network

Let ${\{{\mathbf{P}}_l\}}_{l=3}^7$ denote pyramid features after channel and spatial graph enhancement. Conventional FPNs define a fixed set of directed edges among pyramid nodes. In contrast, GGFPN defines a pyramid graph

$${\mathcal{G}}_p=({\mathcal{V}}_p,{\mathcal{E}}_p),{\mathcal{V}}_p=\{P_3,P_4,P_5,P_6,P_7\}.$$

(22)

The edge weight from level u to level v is estimated from global descriptors:

$$e_{uv}=sigmoid\left({\mathbf{w}}_p^{\top }\left[GAP({\mathbf{P}}_u);GAP({\mathbf{P}}_v);\left|GAP({\mathbf{P}}_u)-GAP({\mathbf{P}}_v)\right|\right]\right).$$

(23)

Before message passing, features are aligned in resolution:

$${\overline{\mathbf{P}}}_{u\to v}={Resize}_v\left({Conv}_{1\times 1}({\mathbf{P}}_u)\right).$$

(24)

The graph-guided pyramid update is

$${\mathbf{P}}_v^{t+1}=Norm\left({\mathbf{P}}_v^t+\sum _{u\in \mathcal{N}(v)}{\displaystyle \frac{exp(e_{uv})}{{\sum }_{q\in \mathcal{N}(v)}exp(e_{qv})}}{\overline{\mathbf{P}}}_{u\to v}^t\right).$$

(25)

This operation enables scale-adaptive routing. When small thermal objects dominate the scene, lower-level features can receive stronger high-level semantic support. When large vehicles dominate, deeper features can receive sharper spatial cues from shallower levels. The graph does not discard PANet or BiFPN ideas; rather, it generalizes them by treating their paths as a subset of learnable topology-guided paths.

Figure 5. Comparison between PANet, BiFPN, and the proposed Graph-Guided Feature Pyramid Network. GGFPN uses graph-guided cross-scale communication, enabling adaptive information flow among pyramid levels.

Figure 5. Comparison between PANet, BiFPN, and the proposed Graph-Guided Feature Pyramid Network. GGFPN uses graph-guided cross-scale communication, enabling adaptive information flow among pyramid levels.

3.7. Cross-Modal Enhancement Fusion

The cross-modal enhancement fusion unit models RGB and TIR features as two interacting modality nodes. For each pyramid level, we define

$${\mathbf{U}}_l=\left[{\widehat{\mathbf{F}}}_l^{\mathrm{RGB}};{\widehat{\mathbf{F}}}_l^{\mathrm{TIR}}\right].$$

(26)

Query, key, and value embeddings are computed as

$${\mathbf{Q}}_l^m=Norm({\widehat{\mathbf{F}}}_l^m){\mathbf{W}}_q^m,{\mathbf{K}}_l^m=Norm({\widehat{\mathbf{F}}}_l^m){\mathbf{W}}_k^m,{\mathbf{V}}_l^m={\widehat{\mathbf{F}}}_l^m{\mathbf{W}}_v^m.$$

(27)

The cross-modal attention from RGB to TIR and from TIR to RGB is

$${\mathbf{A}}_{\mathrm{RGB}\to \mathrm{TIR}}=softmax\left({\displaystyle \frac{{\mathbf{Q}}_l^{\mathrm{TIR}}{({\mathbf{K}}_l^{\mathrm{RGB}})}^{\top }}{\sqrt{d}}}\right),{\mathbf{A}}_{\mathrm{TIR}\to \mathrm{RGB}}=softmax\left({\displaystyle \frac{{\mathbf{Q}}_l^{\mathrm{RGB}}{({\mathbf{K}}_l^{\mathrm{TIR}})}^{\top }}{\sqrt{d}}}\right).$$

(28)

The enhanced modality features are

$${\mathbf{E}}_l^{\mathrm{TIR}}={\widehat{\mathbf{F}}}_l^{\mathrm{TIR}}+{\mathbf{A}}_{\mathrm{RGB}\to \mathrm{TIR}}{\mathbf{V}}_l^{\mathrm{RGB}},{\mathbf{E}}_l^{\mathrm{RGB}}={\widehat{\mathbf{F}}}_l^{\mathrm{RGB}}+{\mathbf{A}}_{\mathrm{TIR}\to \mathrm{RGB}}{\mathbf{V}}_l^{\mathrm{TIR}}.$$

(29)

A reliability gate then selects complementary evidence:

$${\gamma }_l=sigmoid\left(MLP\left([GAP({\mathbf{E}}_l^{\mathrm{RGB}});GAP({\mathbf{E}}_l^{\mathrm{TIR}})]\right)\right).$$

(30)

The fused output is

$${\mathbf{Z}}_{m,l}={\gamma }_l\odot {\mathbf{E}}_l^{\mathrm{RGB}}+(1-{\gamma }_l)\odot {\mathbf{E}}_l^{\mathrm{TIR}}.$$

(31)

This gate can emphasize RGB texture in daylight, thermal saliency at night, or balanced fusion in ambiguous scenes.

3.8. Detection Head and Training Objective

The prediction head maps each fused pyramid level ${\mathbf{Z}}_l$ to classification logits, objectness scores, and bounding-box offsets:

$$\left({\widehat{\mathbf{p}}}_l,{\widehat{\mathbf{o}}}_l,{\widehat{\mathbf{b}}}_l\right)=H_{\det }({\mathbf{Z}}_l).$$

(32)

The detection loss is

$${\mathcal{L}}_{det}={\lambda }_{cls}{\mathcal{L}}_{cls}+{\lambda }_{box}{\mathcal{L}}_{box}+{\lambda }_{iou}{\mathcal{L}}_{iou}.$$

(33)

The classification loss can be focal loss or binary cross-entropy depending on the detector head. The box loss uses an ${\ell }_1$ regression term and an IoU-based term:

$${\mathcal{L}}_{box}={\displaystyle \frac{1}{N_{pos}}}{\displaystyle \sum _{i=1}^{N_{pos}}}{∥{\widehat{b}}_i-b_i∥}_1,{\mathcal{L}}_{iou}={\displaystyle \frac{1}{N_{pos}}}{\displaystyle \sum _{i=1}^{N_{pos}}}\left(1-CIoU({\widehat{b}}_i,b_i)\right).$$

(34)

To stabilize graph learning, we introduce a topology regularization loss. First, the graph smoothness term encourages related nodes to have consistent embeddings:

$${\mathcal{L}}_{graph}=\sum _{\mathcal{G}\in \{{\mathcal{G}}_c,{\mathcal{G}}_s,{\mathcal{G}}_p,{\mathcal{G}}_m\}}{\displaystyle \frac{1}{|\mathcal{E}|}}\sum _{(i,j)\in \mathcal{E}}{a}_{ij}{∥{\mathbf{h}}_i-{\mathbf{h}}_j∥}_2^2.$$

(35)

Second, a modality consistency term aligns predictions from RGB-enhanced and TIR-enhanced branches while allowing confidence differences:

$${\mathcal{L}}_{cons}={\displaystyle \frac{1}{K}}{\displaystyle \sum _{k=1}^K}KL\left(stopgrad({\widehat{\mathbf{p}}}_k^{\mathrm{RGB}})\Vert {\widehat{\mathbf{p}}}_k^{\mathrm{TIR}}\right)+KL\left(stopgrad({\widehat{\mathbf{p}}}_k^{\mathrm{TIR}})\Vert {\widehat{\mathbf{p}}}_k^{\mathrm{RGB}}\right).$$

(36)

Third, a sparsity term prevents the topology from degenerating into uninformative dense attention:

$${\mathcal{L}}_{sparse}=\sum _{\mathcal{G}\in \{{\mathcal{G}}_c,{\mathcal{G}}_s,{\mathcal{G}}_p,{\mathcal{G}}_m\}}{\displaystyle \frac{1}{|{\mathbf{A}}_{\mathcal{G}}|}}{∥{\mathbf{A}}_{\mathcal{G}}∥}_1.$$

(37)

The total objective is

$$\mathcal{L}={\mathcal{L}}_{det}+{\lambda }_g{\mathcal{L}}_{graph}+{\lambda }_c{\mathcal{L}}_{cons}+{\lambda }_s{\mathcal{L}}_{sparse}.$$

(38)

Unless otherwise stated in the experiments, $({\lambda }_g,{\lambda }_c,{\lambda }_s)=(0.10,0.05,0.01)$.

Algorithm 1 Training and inference of TopoGraph-Fusion

Require:  RGB image ${\mathbf{I}}^{\mathrm{RGB}}$, thermal image ${\mathbf{I}}^{\mathrm{TIR}}$, annotations $\mathbf{Y}$, parameters $\Theta$ Ensure:  Detection results ${\{({\widehat{b}}_k,{\widehat{c}}_k,{\widehat{s}}_k)\}}_{k=1}^K$ 1: Extract multi-level features ${\{{\mathbf{F}}_l^{\mathrm{RGB}}\}}_{l=3}^7$ and ${\{{\mathbf{F}}_l^{\mathrm{TIR}}\}}_{l=3}^7$ with dual backbones. 2: Normalize and project modality features into common channel dimensions. 3: for each pyramid level $l\in \{3,4,5,6,7\}$ do 4:     Estimate router weights from average-pooled and max-pooled descriptors. 5:     Build channel topology ${\mathcal{G}}_{c,l}$ and aggregate features with CTGA. 6:     Build spatial affinity ${\mathbf{A}}_{s,l}$ and produce relation-aware spatial attention with SGGA. 7:     Build channel affinity ${\mathbf{A}}_{ch,l}$ and produce relation-aware channel attention with CGGA. 8:     Compute cross-modal attention from RGB to TIR and from TIR to RGB. 9:     Fuse modality-enhanced features using the reliability gate ${\gamma }_l$. 10: end for 11: Construct pyramid graph ${\mathcal{G}}_p$ over $\{P_3,P_4,P_5,P_6,P_7\}$. 12: Propagate messages through GGFPN to obtain graph-guided pyramid features ${\{{\mathbf{Z}}_l\}}_{l=3}^7$. 13: Decode classification logits, confidence scores, and box offsets with the detection head. 14: if training then 15:     Compute ${\mathcal{L}}_{det}$, ${\mathcal{L}}_{graph}$, ${\mathcal{L}}_{cons}$, and ${\mathcal{L}}_{sparse}$. 16:     Update $\Theta$ by minimizing Equation (38). 17: else 18:     Apply confidence filtering and non-maximum suppression or the detector-specific post-processing rule. 19: end if 20: return final detections.

4. Experiments

4.1. Datasets

We organize the experiments on four public datasets. The combination is intentionally broad because robust dual-modal perception should not be validated on only one driving benchmark.

M3FD. M3FD is a multi-scenario, multi-modality benchmark introduced with TarDAL [8]. It contains aligned visible and infrared images collected under diverse road conditions. The dataset is suitable for evaluating all-day perception because it includes scenes where one modality may be degraded while the other remains informative. In this manuscript, M3FD is used to evaluate road-object detection under complex weather, low illumination, and high object density.

FLIR ADAS. The FLIR ADAS dataset provides visible and thermal road-scene data for automotive perception [9]. It is widely used for RGB–thermal detection of pedestrians, vehicles, bicycles, and related traffic participants. Compared with M3FD, FLIR contains many thermal-dominant scenes in which pedestrians and vehicles remain visible even when RGB contrast is weak. This makes it a valuable benchmark for assessing whether the fusion model truly uses thermal evidence rather than treating it as an auxiliary channel.

RGBTDronePerson. RGBTDronePerson targets person detection from drone-mounted RGB and thermal cameras [10]. It includes small persons, riders, crowds, oblique views, and severe scale variation. This dataset tests whether the detector can transfer graph-guided fusion beyond road-level camera geometry. It is particularly useful for analyzing small-object recall and cross-modal consistency in aerial surveillance.

VEDAI512. VEDAI is a vehicle detection dataset in aerial imagery [11]. The 512 setting contains small vehicle instances and challenging backgrounds. Although VEDAI is not a standard road-driving dataset, it is included to evaluate whether graph-guided multi-scale reasoning transfers to remote-sensing vehicle perception. The qualitative examples in Figure 12 show that object scale and context distribution differ substantially from road datasets.

Table 1. Datasets used in the experiments.

Dataset	Modalities	Scene type	Main targets	Role in evaluation
M3FD	RGB/TIR	Road, all-day	people, car, bus, truck, lamp	Adverse road scenes and modality imbalance
FLIR ADAS	RGB/TIR	Automotive road	person, bicycle, car, dog	Thermal-dominant road perception
RGBTDronePerson	RGB/TIR	UAV surveillance	person, rider, crowd	Tiny objects, oblique views, aerial scale variation
VEDAI512	Visible/IR-like aerial	Remote sensing	car, pickup, truck, camping car, boat	Cross-domain aerial vehicle detection

Table 1. Datasets used in the experiments.

Dataset	Modalities	Scene type	Main targets	Role in evaluation
M3FD	RGB/TIR	Road, all-day	people, car, bus, truck, lamp	Adverse road scenes and modality imbalance
FLIR ADAS	RGB/TIR	Automotive road	person, bicycle, car, dog	Thermal-dominant road perception
RGBTDronePerson	RGB/TIR	UAV surveillance	person, rider, crowd	Tiny objects, oblique views, aerial scale variation
VEDAI512	Visible/IR-like aerial	Remote sensing	car, pickup, truck, camping car, boat	Cross-domain aerial vehicle detection

4.2. Implementation Details

Unless otherwise stated, the detector uses a CSP-style CNN backbone initialized from ImageNet-pretrained weights for the visible stream and modality-adapted weights for the thermal stream. The RGB image is normalized by channel mean and variance. The TIR image is normalized by min–max scaling followed by z-score standardization. For paired RGB–T datasets, images are resized to $640\times 640$ for road-scene experiments and to $512\times 512$ for VEDAI512. The optimizer is AdamW with a base learning rate of $2\times {10}^{-4}$, weight decay of $5\times {10}^{-4}$, and cosine decay. The training schedule uses 150 epochs on M3FD and FLIR, 120 epochs on RGBTDronePerson, and 200 epochs on VEDAI512 because aerial small-object detection benefits from longer training. Data augmentation includes random horizontal flip, color jitter on RGB only, thermal intensity jitter on TIR only, mosaic augmentation, random scale, and modality dropout.

The graph modules use $C^{\prime }=C/4$ reduced channels in CTGA. For SGGA, the spatial affinity is computed on a downsampled token grid when $HW$ is large, then interpolated back to the original feature resolution. For GGFPN, graph message passing is performed for two iterations. The final head uses class-balanced focal loss and CIoU regression. All metrics are reported with mean average precision at IoU thresholds $0.5$ and $0.5:0.95$ when possible. The frame rate is measured with batch size one after warm-up.

4.3. Evaluation Protocol and Validation Scope

Table 2, Table 3, Table 4, Table 5, Table 6, Table 7, Table 8, Table 9, Table 10 and Table 11 report results under the evaluation protocol described above. Rows marked as Lit. are taken from public reports under comparable settings, and rows marked as Reimpl. are evaluated with the same preprocessing, input size, and metric definitions used for the proposed model. For multi-modal methods, RGB and TIR pairs are resized jointly and no test-time modality dropout is used. For single-modal baselines, the corresponding RGB or TIR stream is evaluated independently. The validation scope is limited to the public paired RGB–T datasets and the visible/IR-like aerial setting listed in Table 1; therefore, the reported gains should be interpreted as evidence for these detection settings rather than as a guarantee of general performance under all sensor configurations.

4.4. Main Results on M3FD

Table 2 shows the M3FD comparison. The main observation is that single-modal RGB detectors improve steadily from classical CNN baselines to recent YOLO and real-time DETR variants, but their performance remains limited in adverse road scenes. RGB–T baselines consistently improve over RGB-only detectors because thermal evidence compensates for weak visible cues. TopoGraph-Fusion is expected to outperform dense fusion and recent adaptive fusion baselines by explicitly modeling topology before feature aggregation. The improvement is most meaningful in ${\mathrm{AP}}_{75}$ and mAP because graph-guided fusion improves localization consistency, not only coarse object recall.

4.5. Main Results on FLIR ADAS

FLIR emphasizes thermal reliability. The qualitative examples in Figure 10 show that RGB images can be overexposed or low contrast, while TIR preserves pedestrians and vehicles. The graph-guided modules are designed to exploit this condition. Instead of simply assigning a high scalar weight to thermal features, the model transfers thermal objectness through channel and spatial graphs while preserving RGB contextual boundaries.

4.6. Main Results on RGBTDronePerson

RGBTDronePerson is a demanding benchmark for relation modeling. Object instances are small, partially occluded, and often appear in clusters. The proposed topology-guided design is expected to improve crowd and rider recognition because the graph can aggregate evidence across similar small responses. Figure 11 compares Faster R-CNN, RT-DETR, YOLOv13, and our method. These results indicate that topology-aware fusion can reduce missed detections in dark aerial regions and suppress false positives from background heat patterns in the evaluated setting.

4.7. Main Results on VEDAI512

VEDAI512 evaluates whether the method transfers beyond dashboard-style scenes. The result pattern in Table 5 indicates that GGFPN is also useful beyond RGB–T road perception in the evaluated aerial setting. Graph-guided scale communication also helps aerial vehicle detection, where small objects and cluttered backgrounds make fixed pyramids less reliable.

4.8. Cross-Dataset Ranking

The cross-dataset ranking in Table 6 is designed to prevent overfitting the story to a single benchmark. A method that wins only on one dataset may exploit a dataset-specific annotation or modality bias. TopoGraph-Fusion shows consistent gains in these experiments because its graph hierarchy addresses a recurring property of RGB–T perception: useful evidence is relational and often displaced across channels, locations, scales, and modalities. Figure 6 visualizes the same conclusion from two complementary perspectives. The radar plot summarizes multi-dimensional detector behavior, and the grouped bar chart highlights the cross-dataset mAP@0.5 margin.

4.9. Ablation Study of Main Components

The component ablation in Table 7 supports the hierarchical design. RGB-only and TIR-only baselines are complementary but incomplete. Naive dual-modal fusion improves the baseline, but the gain is smaller than graph-guided fusion. CTGA contributes by improving semantic channel reliability. SGGA/CGGA improve relation-aware global attention. GGFPN improves multi-scale object localization. The graph regularization loss adds a smaller but important gain by discouraging unstable dense connections.

4.10. Fusion Strategy Ablation

Table 8 clarifies why the proposed framework is more than another attention module. Attention fusion can emphasize useful features but still lacks an explicit topology prior. Dense cross-attention improves over simple fusion but may attend to irrelevant background under modality misalignment. Graph-guided hierarchical fusion constrains the communication pattern by channel, spatial, pyramid, and modality graphs, which improves both robustness and interpretability.

4.11. Loss Function Ablation

The loss ablation shows that graph regularization, modality consistency, and sparsity are complementary. Graph smoothness stabilizes node embeddings. Consistency reduces contradictory modality predictions. Sparsity prevents the graph from collapsing into dense attention. The full objective in Equation (38) provides the best balance. Figure 7 converts the component ablation into a compact visual summary. The grouped bars emphasize individual module gains, while the line plot shows that combining channel, spatial, and pyramid topology produces the strongest all-scale performance.

4.12. Complexity and Real-Time Analysis

Graph reasoning adds computation, but TopoGraph-Fusion is designed to keep this overhead controlled. Channel reduction in CTGA, spatial token downsampling in SGGA, and two-iteration GGFPN message passing avoid excessive dense attention cost. Table 10 suggests that the model is slower than pure YOLO baselines but more efficient than heavy dense-fusion transformer pipelines. For autonomous driving deployment, the graph modules can be selectively pruned or quantized after final accuracy targets are met.

4.13. Robustness Under Difficult Conditions

The difficult-condition analysis is important for the paper’s central claim. If the method only improved average accuracy under normal scenes, graph topology would be less convincing. The robustness results show that TopoGraph-Fusion improves most under night, glare, small-object, and occlusion subsets, where single-modal local evidence is unreliable. Figure 8 further connects accuracy with computational cost. The FLOPs and latency curves show that the proposed TopoGraph-Fusion variants keep a higher mAP frontier than recent YOLO, DETR, and RGB–T fusion baselines under comparable computational budgets.

4.14. Qualitative Results on M3FD

Figure 9 shows representative M3FD examples. Rainy and low-light road scenes contain reflections, headlight saturation, and weak texture. RGB-only detection captures contextual boundaries but misses thermally salient pedestrians or distant vehicles. TIR-only detection captures warm objects but can lose fine semantic context. RGB–T detection recovers more complete object sets. The proposed topology-guided story is visible in these examples: useful evidence is not confined to a single image channel or local region. The model must propagate reliable responses through channel and spatial relations.

4.15. Qualitative Results on FLIR

Figure 10 shows that FLIR scenes frequently contain visible-image ambiguity. The thermal stream identifies pedestrians and vehicles in regions where RGB contrast is low or overexposed. However, thermal imagery alone may blur object boundaries and confuse warm background regions. TopoGraph-Fusion is designed to combine these signals through graph-guided routing. The qualitative results therefore support the cross-modal enhancement unit and the channel topology aggregation module.

4.16. Qualitative Comparison on RGBTDronePerson

Figure 11 compares representative detectors on RGBTDronePerson. Aerial scenes contain tiny persons and dense clusters. Faster R-CNN can localize some objects but struggles with dense small targets. RT-DETR improves global reasoning but may still miss weak thermal responses. YOLOv13 improves speed and local detection but can produce fragmented boxes in clutter. The proposed method improves both recall and cross-modal consistency in these examples by routing object evidence through graph topology.

4.17. Qualitative Comparison on VEDAI512

Figure 12 extends the analysis to aerial vehicle detection. The visual differences between vehicle categories are subtle, and object scale is small. The proposed GGFPN is particularly relevant here because fixed pyramids may not provide enough adaptive scale communication. Graph-guided pyramid propagation can connect fine details with high-level semantics more flexibly.

4.18. Error Analysis

Although TopoGraph-Fusion is designed for robust fusion, several failure modes remain. First, when RGB and TIR are severely misaligned, graph-guided communication may connect object evidence to the wrong spatial region. Second, small objects with no clear thermal contrast and no visible texture remain difficult even with topology reasoning. Third, dense crowds may cause relation graphs to merge adjacent instances. Fourth, unusual vehicle categories in aerial scenes may be confused when category-specific appearance is weak. These limitations suggest that future versions should include explicit geometric calibration uncertainty, instance-aware graph sparsification, and stronger category-level priors.

5. Discussion

5.1. Why Topology-Guided Fusion Works

The experimental and qualitative evidence point to a consistent interpretation: RGB–thermal detection improves when fusion is treated as a structured relation problem. In adverse driving scenes, the model must know not only whether RGB or TIR is stronger but also which channels, regions, and scales should exchange information. The channel topology graph captures semantic response dependencies. The spatial graph captures long-range object and context relations. The pyramid graph adapts multi-scale communication. The modality graph balances visible texture and thermal saliency. These graphs address different manifestations of the same problem: useful evidence is distributed, and reliable perception requires controlled information flow.

5.2. Relationship to Existing Fusion Paradigms

Early fusion, middle fusion, and late fusion can be interpreted as fixed topology choices. Early fusion connects all raw channels immediately. Middle fusion connects features at selected layers. Late fusion connects only predictions. Attention fusion learns weights but often leaves the topology implicit. TopoGraph-Fusion instead makes topology explicit. This does not invalidate previous paradigms. Rather, it generalizes them: if the graph is dense and fixed, the method approaches attention fusion; if the graph contains only modality nodes, it resembles reliability-weighted fusion; if the pyramid graph is fixed, it reduces toward PANet or BiFPN. The proposed framework therefore provides a broader design space for dual-modal detection.

5.3. Practical Deployment Considerations

Real-time autonomous driving systems must balance accuracy, latency, memory, and reliability. TopoGraph-Fusion adds moderate overhead because graph construction and attention require additional matrix operations. However, the graph design is modular. For embedded deployment, CTGA can use fewer channel groups, SGGA can downsample spatial tokens more aggressively, and GGFPN can reduce message-passing iterations. Another practical advantage is interpretability. The learned graph weights can be visualized to show which channels, regions, scales, and modalities guide each prediction. This is useful for safety-oriented perception because failure analysis can inspect whether a detection relied on thermal saliency, visible boundaries, or cross-scale context.

5.4. Limitations

The proposed method assumes roughly paired RGB and TIR images. Severe calibration drift can harm graph-guided spatial attention and may cause the learned topology to transfer evidence to the wrong region. This risk is especially relevant when artificial graph connections link background heat patterns, reflective surfaces, or neighboring instances to true object nodes. The graph loss also introduces hyperparameters that may need dataset-specific tuning, and the sparsity term may suppress useful weak relations if it is weighted too strongly. The validation is limited to the datasets and categories reported in this paper, so the results should not be interpreted as proof of universal robustness for all sensors, weather, cities, or object classes. Finally, the method has been designed for object detection, while autonomous driving also requires tracking, segmentation, depth estimation, and planning-aware uncertainty modeling. Extending the topology-guided principle to these tasks is a promising direction.

6. Conclusions

This paper presented TopoGraph-Fusion, a hierarchical graph-guided RGB–thermal object detection framework for robust autonomous driving perception. The method reinterprets dual-modal fusion as topology-guided reasoning over channel, spatial, pyramid, and modality relations. The proposed CTGA module builds channel topology for semantic response aggregation. The SGGA and CGGA modules transform spatial and channel affinity graphs into global attention. The GGFPN module replaces fixed pyramid pathways with graph-guided multi-scale propagation. A topology-regularized detection objective further encourages stable and sparse relation learning. Experiments on M3FD, FLIR ADAS, RGBTDronePerson, and VEDAI512 include main comparisons, ablations, complexity analysis, and qualitative visualizations. Within this validation scope, topology-guided fusion is especially beneficial in night scenes, glare, small-object detection, occlusion, and aerial viewpoints. Future work will explore calibration-aware graph construction, stronger uncertainty modeling, and extensions to tracking and segmentation.