Lightweight YOLOv8-Obb Optimization with Hybrid Attention and Dynamic Feature Reconstruction for Remote Sensing Object Detection

1. Introduction

In recent years, with the rapid development of satellite remote sensing technology, remote sensing object detection (RSOD) has become a research hotspot with widespread applications in geographic mapping, environmental protection, and national security. However, as most remote sensing images are acquired through aerial or spaceborne platforms, they present unique challenges for conventional object detection algorithms due to the multi-scale characteristics of targets and complex background environments. Consequently, improving the performance of RSOD algorithms has emerged as both a critical and challenging research task.

Object detection methods can be primarily categorized into two types: Traditional approaches typically rely on handcrafted features [1], featuring mature algorithmic frameworks that demonstrate high efficiency in certain applications. Nevertheless, these methods heavily depend on low-level semantic information while neglecting the extraction of high-level semantic features, making them inadequate for accurately detecting diverse targets in complex scenarios. In contrast, deep learning-based methods have significantly enhanced detection efficiency and performance through automated feature extraction. These approaches can be further divided into single-stage and two-stage detectors based on their region proposal generation mechanisms [2].

Single-stage detectors, such as the YOLO series [3] and SSD (Single Shot MultiBox Detector) [4,5], prioritize rapid response at the expense of some detection accuracy. The YOLO series employs an image grid division approach that focuses on target center points rather than entire regions, eliminating the need for region proposal generation and substantially reducing search time to meet real-time detection requirements. Representative two-stage detectors include models based on Region-based Convolutional Neural Networks (RCNN) [6]. RCNN utilizes selective search algorithms [7] to generate region proposals, which are then processed through convolutional neural networks for feature extraction before final classification using support vector machines.

Despite these advancements, current object detection algorithms still face several challenges: (1) The performance of detectors is significantly compromised by multi-scale targets and complex backgrounds in remote sensing images; (2) While some detectors achieve relatively high accuracy on remote sensing datasets, their complex network architectures result in excessively long training and detection times. To address these issues, this paper proposes a lightweight object detection network based on YOLOv8-obb. The main contributions include: (1) Introduction of a Mixed Local Channel Attention (MLCA) module [8] and Dynamic Upsampling module [9] to enhance detection accuracy; (2) Implementation of network optimization through RepNCSPELAN [10] module for lightweight design.

To enhance object detection accuracy in complex backgrounds and for multi-scale targets, this study incorporates attention mechanisms into the network architecture. These mechanisms automatically learn input weight distributions, enabling the model to focus more effectively on critical features. Several established attention modules have been developed, including Squeeze-and-Excitation (SE) [11], Efficient Channel Attention (ECA) [12], Convolutional Block Attention Module (CBAM) [13], Global Attention Mechanism (GAM) [14], Co-Attention [15], Reverse Attention [16], and Cross Attention [17]. However, certain attention layers may introduce unnecessary complexity while providing marginal accuracy improvements. Consequently, the optimal selection of attention modules for specific detection tasks remains a crucial research challenge in computer vision.

Recent trends in network architecture design have favored increasingly deeper and more complex structures, leading to substantial growth in model size and parameter counts. This development has spurred the creation of lightweight network architectures such as MobileNet [18], ShuffleNet [19-20], GhostNet [21], and EfficientDet [22]. Nevertheless, these lightweight modules demonstrate limited adaptability when applied to modified network structures. More importantly, their integration often introduces unintended side effects during algorithm optimization. Therefore, achieving network simplification while maintaining detection accuracy presents a significant and pressing research problem that requires innovative solutions.

2. Methods

2.1. Mixed Local Channel Attention

This paper proposes a Mixed Local Channel Attention (MLCA) module to mitigate severe target information loss during consecutive downsampling operations while effectively integrating global and local information for enhanced detection performance. As illustrated in Figure 1, the MLCA module operates through three sequential processing stages: (1) The input feature map undergoes parallel processing through Local Average Pooling (LAP) and Global Average Pooling (GAP) modules, where LAP extracts fine-grained local region features while GAP captures global statistical characteristics of the entire feature map. (2) The pooled features from both paths are then processed by a 1D convolutional module that compresses channel dimensions while preserving spatial structure, effectively reshaping features for subsequent operations. (3) The module performs hierarchical feature fusion by combining GAP-derived global context with LAP-processed local features, followed by spatial reconstruction through Unpooling (UNAP) operations to restore original dimensions, and finally enhances discriminative features by merging with original inputs. This innovative design effectively addresses information degradation during downsampling while improving feature representation quality, ultimately leading to significant detection performance improvements as demonstrated in our experiments. The MLCA module achieves these benefits while maintaining computational efficiency through its dimension-aware processing architecture.

2.2. DySample

DySample is an innovative and efficient dynamic upsampler that redefines the upsampling process from a point-sampling perspective, avoiding the use of dynamic convolution. As shown in Figure 2, DySample achieves content-aware upsampling by adaptively generating sampling points to resample continuous feature maps, resulting in more efficient feature reconstruction.The dynamic upsampling process operates as follows: Given an input feature map $\mathcal{X}$with dimensions $C\times H\times W$, DySample produces an upsampled output feature map $\mathcal{X}\mathcal{\text{'}}$ with dimensions $C\times sH\times sW$, where s is the scaling factor. Unlike traditional interpolation-based methods, DySample dynamically adjusts sampling positions based on local feature content, preserving structural details while minimizing artifacts. This approach significantly enhances feature representation quality in super-resolution and object detection tasks.

$$X\mathrm{\text{'}}=grid\_sample(X,S),$$

(1)

Figure 3 demonstrates the working principle of the sampling point generator in DySample, which produces a sampling set $S$ with dimensions $2g\times sH\times sW$ through the following process: the input feature map $X$ is first processed by a linear layer to generate a dynamic scope factor that modulates the offset field $O$ (where $\sigma$ denotes the sigmoid function), and the final sampling set $S$ is obtained by combining the original coordinate grid $G$ with the learned offsets $O$, achieving content-aware feature resampling through this dynamic coordinate adjustment mechanism.

$$O=0.5·sigmoid\left({linear}_1\right(X\left)\right)·\left({linear}_2\right(X\left)\right),$$

(2)

$$S=G+O$$

(3)

2.3. RepNCSPELA

GELAN (Generalized Efficient Layer Aggregation Network) draws structural inspiration from CSPNet and ELAN architectures, incorporating three key characteristics: lightweight design, fast inference speed, and high accuracy. Similar to CSPNet's structure, the RepNCSPELAN module (Reparameterized Non-local Cross-Scale Path Enhanced Layer Aggregation) serves as a critical component within the GELAN framework. The module operates through a systematic four-stage process: (1) Input features are initially divided into two pathways via a transition layer; (2) Each branch undergoes distinct computational processing - one branch follows ELAN's characteristic hierarchical connection pattern where each layer's output connects to the next layer's input; (3) Processed features from both branches are subsequently reintegrated; (4) The final aggregated output is generated as shown in Figure 4. This architectural design achieves optimal balance between computational efficiency and feature representation capability through its innovative combination of cross-scale path enhancement and parameter reusability mechanisms.

2.4. YOLOv8-Obb

YOLOv8, through lightweight design and advanced data processing techniques, not only improves detection speed and accuracy but also enhances the model's robustness, making it suitable for remote sensing image applications in complex environments. YOLOv8 excels in detecting targets such as vehicles, buildings, and aircraft, demonstrating high detection precision. Its advantages in speed, accuracy, and robustness make it an ideal choice for target detection in remote sensing images.

The original YOLOv8 consists of four parts: the input end, backbone network, feature fusion, and target localization and classification. The input image size is set to 1024×1024. To address the lack of large-scale targets in remote sensing data, YOLOv8 introduces the Mosaic data augmentation method during the data preprocessing stage. This method randomly selects and stitches four images together, significantly improving model performance, especially as the number of training iterations increases. In the backbone network section, YOLOv8 extracts features through a series of convolutional and deconvolutional layers while incorporating bottleneck structures to enhance performance. YOLOv8 adopts the C2f module as its basic unit, which has fewer parameters and stronger feature extraction capabilities.

In the neck structure (Neck) of YOLOv8, multi-scale feature fusion is introduced to combine features from different stages of the backbone network, thereby enhancing feature representation. Figure 5 illustrates the specific steps of the neck. The outputs P3, P4, and P5 from the backbone network are passed to the PAN-FPN structure to fuse features at different scales. Through the upsampling process, P5 is fused with P4 to produce T1; T1 is then upsampled with P3 to generate D1. Subsequently, D1 undergoes convolutional layer processing and is fused with T1, generating D2 through downsampling. D3 is generated in a similar manner. Finally, D1, D2, and D3 serve as the outputs of the neck and are passed to three detection heads for target detection.

In the detection head part of the network architecture, YOLOv8 differs from YOLOv8-obb. The Oriented Bounding Box (OBB) [23-24] is a new data format that includes directional information for rotated bounding boxes. OBB defines the target's position using the center point, width, height, and rotation angle. As a result, OBB's bounding boxes can better fit the shape of the target, improving target detection performance. YOLOv8-obb introduces OBB to more accurately represent targets with rotated or inclined postures; YOLOv8-obb redesigns the loss function to adapt to the regression task of rotation angles, ensuring the model learns correct orientation information; based on YOLOv8, YOLOv8-obb adds an Angle Branch for predicting rotation angles and adjusts the output of feature maps.

2.5. Improved YOLOv8-Obb

This paper introduces three key modules into the YOLOv8-obb architecture: the Mixed Local Channel Attention (MLCA) module, the Dynamic Upsampling module, and the RepNCSPELAN module.

The MLCA module enhances feature representation capability through a hybrid local-channel attention mechanism, improving the model's object detection performance in complex scenarios. The Dynamic Upsampling module adaptively adjusts sampling strategies to optimize detection performance across targets of different scales, thereby increasing both detection accuracy and computational efficiency. The RepNCSPELAN module employs lightweight design and flexible feature fusion to achieve rapid yet efficient object detection. The modified YOLOv8-obb architecture is shown in Figure 6, with corresponding network parameters detailed in Table 1.

3. Experimental Setup

3.1. Experimental Dataset

This study adopts the DOTA dataset (A Large-scale Dataset for Object Detection in Aerial Images), which was publicly released by Wuhan University in 2017. The dataset comprises 2,806 aerial images acquired from diverse sensors and platforms, with each image having a resolution of 4000×4000 pixels. These images encompass objects with varying scales, orientations, and shapes. The DOTA dataset provides annotations for 15 common categories: plane, ship, storage tank, baseball diamond, tennis court, basketball court, ground track field, harbor, bridge, large vehicle, small vehicle, helicopter, roundabout, soccer ball field, and swimming pool. The fully annotated images contain 188,282 instances, each labeled as an arbitrary quadrilateral.

Among the 15 object categories in the dataset, certain classes such as basketball courts and soccer ball fields typically exhibit lower detection accuracy. The primary reasons for this reduced performance include their relatively small size and the difficulty in distinguishing them from the background environment, which significantly impacts the model’s mean average precision (mAP) on this dataset.

As shown in Table 2, the YOLOv8-obb model achieves an mAP50 of 78.4 across all 15 categories on the DOTA dataset. Notably, categories with distinctive features (e.g., planes and ships) can reach an mAP50 of up to 90%. However, the detection performance remains suboptimal for certain categories like bridges and sports fields. Consequently, improving the detection accuracy for these challenging categories has become a key focus of our architectural enhancements.

This study selected the six categories with the lowest detection accuracy: basketball courts, ground track fields, bridges, small vehicles, roundabouts, and soccer ball fields. The dataset was reorganized accordingly, and training was conducted using the YOLOv8-obb model. The training results are shown in Table 3. Figure 7 illustrates the size distribution of annotated instances, showing that the majority of instances are small-scale targets with relatively uniform distributions in both width and height.

3.2. Experimental Configuration and Performance Metrics

The experimental platform and hyperparameter settings are shown in Table 4 and Table 5

The network performance is evaluated based on four key metrics: Precision (P), Recall (R), Average Precision (AP), and mean Average Precision (mAP). These metrics are formally defined as follows:

$$P={\displaystyle \frac{TP}{TP+FP}}$$

(4)

$$R={\displaystyle \frac{TP}{TP+FN}}$$

(5)

$$F1={\displaystyle \frac{2PR}{P+R}}$$

(6)

$$\mathrm{A}P={\int }_0^{1}P\left(R\right)dR$$

(7)

$$mAP={\displaystyle \frac{{\sum }_{i=1}^{n}AP\left(i\right)}{n}}$$

(8)

In object detection evaluation, true positives (TP) refer to samples that are actually positive and correctly identified as positive by the classifier, while false positives (FP) represent negative samples that are incorrectly classified as positive. Targets that fail to be detected are marked as false negatives (FN). The intersection over union (IoU) measures the ratio of the intersection area to the union area between two shapes. For object detection tasks, we typically compare predicted bounding boxes with ground truth annotations by calculating their IoU. A sample is considered correctly predicted if the computed IoU exceeds a predefined threshold. The mAP50 metric uses a threshold of 0.5, while mAP50-95 serves as a more stringent evaluation indicator that assesses algorithm performance across different IoU thresholds from 0.5 to 0.95 with 0.05 increments, providing a comprehensive measure of detection accuracy under varying localization precision requirements.

4. Results and Discussion

4.1. Ablation Study

Figure 8 displays the training curves of multiple improved YOLOv8-obb models, showing similar convergence patterns where all metrics reach stable values after 150 epochs. Due to space constraints, Figure 8 primarily illustrates the overall trends and approximate values of the metrics during training; therefore, this paper provides precise quantitative results in Table 6 to thoroughly evaluate each model’s performance. The table includes: MLCA (Mixed Local Channel Attention module added to YOLOv8-obb), DS (Dynamic Upsampling module), and GE (RepNCSPELAN layer). For comparison, we also implemented the CBAM attention module and two dynamic upsampling variants: LP (“linear + pixel shuffle” design) and PL (“pixel shuffle + linear” design), enabling systematic analysis of architectural modifications and their individual/combined effects on model optimization.

The ablation experiments validate the effectiveness of the proposed YOLOv8-obb modules. Experimental results demonstrate that incorporating the MLCA attention layer significantly enhances model performance, achieving improvements of 1.7% in precision, 0.2% in recall, 1.8% in mAP50, and 1.2% in mAP50-95. The performance gain primarily stems from MLCA’s enhanced feature extraction capability, which improves multi-scale object detection - particularly for small targets, as evidenced by the experimental data. Although the CBAM module also shows performance benefits, comprehensive comparison confirms MLCA’s superior effectiveness, leading to its selection in our final architecture.

Moreover, the integration of the dynamic upsampling (PL) module yields concurrent optimization in both detection accuracy and model parameters. Specifically, the proposed architecture achieves a 0.1M reduction in parameter count while improving mAP50 by 0.4%. Comparative experiments reveal that the structurally analogous LP module conversely degrades detection performance, underscoring the necessity for prudent selection of upsampling configurations in practical implementations.

The incorporation of the RepNCSPELAN module yields relatively modest improvements across evaluation metrics. The primary objective of replacing the C2f layer was to reduce parameter count for network lightweighting while maintaining detection accuracy. Remarkably, experimental results demonstrate that substituting the complex C2f layer not only preserves but slightly enhances model precision by 0.1%, validating the new module’s efficacy. These findings collectively confirm that our improved algorithm successfully achieves dual objectives: significant enhancement in target detection accuracy (evidenced by 2.3% mAP50 improvement on DOTA dataset) alongside meeting stringent lightweight architecture requirements (29% parameter reduction from 3.1M to 2.2M). The RepNCSPELAN module’s design proves particularly effective in balancing computational efficiency with feature representation capability, making it suitable for deployment in resource-constrained edge computing scenarios while maintaining competitive performance against conventional architectures.

Based on comprehensive ablation studies, this paper adopts the optimal ‘+MLCA+DS(PL)+GE’ configuration (hereafter termed Improved YOLOv8-obb) for subsequent experiments, achieving after 150 epochs: 68.4% precision (P), 60.6% recall (R), 65.3% mAP50, and 48.2% mAP50-95, with detailed category-wise metrics documented in Table 6, demonstrating the architecture’s balanced optimization of accuracy (2.3% mAP50 improvement over baseline) and efficiency (29% parameter reduction to 2.2M) through synergistic integration of Mixed Local Channel Attention for enhanced feature discrimination, dynamic upsampling (PL variant) for multi-scale adaptation, and RepNCSPELAN’s lightweight feature fusion.

To further evaluate the detection performance, Figure 9 and Figure 10 present the confusion matrix and normalized confusion matrix respectively, where the main diagonal elements indicate either the count or proportion of correctly detected targets (with most categories exceeding 0.5 detection accuracy), while off-diagonal elements horizontally represent false positives and vertically represent false negatives - a dominant main diagonal signifies superior detection performance, whereas increased off-diagonal elements reflect higher misdetection and missed detection rates, quantitatively demonstrating the model’s classification capability across all categories through the matrix’s spatial distribution characteristics.

The results demonstrate that the small vehicle category, which has the largest sample size, achieves notably higher overall accuracy after training. Additionally, the roundabout and ground track field categories exhibit relatively strong detection performance. Although the bridge and soccer ball field categories show comparatively lower accuracy rates, analysis of Table 3 reveals that their performance still improves over the baseline YOLOv8-obb model.

To further evaluate the detection capability of the improved YOLOv8-obb across different target categories, Figure 10, Figure 11, Figure 12 and Figure 13 present various evaluation metric curves plotted against confidence scores

Figure 11 shows the Precision-Confidence Curve, illustrating how precision varies across different confidence thresholds. The curve demonstrates higher precision at higher confidence levels, indicating fewer false positives (FP) in high-confidence predictions. While precision typically increases with confidence, the basketball court category shows an unusual precision drop at high confidence values.

Figure 12 presents the Recall-Confidence Curve, displaying recall rate changes at different thresholds. The curve reveals the fundamental trade-off: lower confidence thresholds achieve higher recall but lower precision, while higher thresholds result in lower recall but higher precision.

Figure 13 displays the F1-Confidence Curve, which evaluates the model’s balanced performance by plotting F1-score (the harmonic mean of precision and recall) against confidence thresholds. The peak of this curve corresponds to the optimal confidence threshold that best balances precision and recall - specifically matching the 65.3% mAP50 value in our experiments.

Figure 14 illustrates the Precision-Recall (PR) Curve, with recall on the x-axis and precision on the y-axis. The area under this curve (AP) represents average precision. Key observations include:

1) Curves closer to the upper-right corner indicate better performance

2) Strong convexity shows graceful precision decline as recall increases

3) Diagonal alignment suggests random-guessing-level performance

Notably, all categories except soccer ball field show desirable upper-right convexity, confirming the improved YOLOv8n-obb’s ability to balance precision and recall. Furthermore, the separation between category curves reflects detection difficulty - for example, the relatively flat curves for “bridge” and soccer ball field explicitly demonstrate these categories’ challenging nature for detection algorithms.

4.2. Comparative Experiment

To verify the effectiveness and efficiency of our improved algorithm, we conducted comparative experiments with several classical models including YOLOv5, YOLOv6, YOLOv8 and YOLOv8n-obb, evaluating them using four key metrics: parameter count, FLOPs (floating-point operations per second), mAP (mean average precision), and model size. The experiments specifically employed the ‘nano’ (n) scale variants of YOLO models, which are categorized into five size levels (n, s, m, l, and x) - for instance, YOLOv8n-obb contains only about 3.2M parameters, while YOLOv8s-obb includes approximately 11.2M parameters, and the largest YOLOv8x-obb exceeds 60M parameters. Although models with more parameters typically achieve higher detection accuracy, they require significantly longer training times; therefore, considering the lightweight requirements, we exclusively selected the n-scale models for comparative testing, with the experimental results detailed in Table 8.

Experimental results demonstrate that the improved YOLOv8n-obb algorithm outperforms classical detectors (YOLOv5n, YOLOv6n, YOLOv8, and the original YOLOv8n-obb) in both model size and accuracy. The enhanced algorithm achieves 65.3% mAP50, representing a 2.3% improvement over the original YOLOv8n-obb and significantly surpassing other detectors. The additional parameters introduced by the OBB format substantially enhance the fitting degree between bounding boxes and targets. While leveraging the advantages of OBB data format, our improved algorithm further boosts detection accuracy through the incorporation of attention mechanisms, yet the attention layers’ impact on model size remains negligible due to the model’s lightweight nature. Moreover, the enhanced YOLOv8n-obb shows superior performance in parameter count, FLOPs, and model size compared to its original version - reducing parameters from 3.1M to 2.2M (a 29% decrease that significantly optimizes network architecture), decreasing FLOPs from 8.5G to 6.2G, and shrinking model size from 6.5MB to 4.9MB. Remarkably, while achieving substantial accuracy improvements, the final model size is even smaller than YOLOv5n, fully satisfying lightweight requirements for practical deployment scenarios.

Figure 15 demonstrates the visual detection results of our enhanced YOLOv8-obb algorithm on the DOTA dataset. The visualization reveals that despite the dataset containing numerous small-scale instances (such as those in the small vehicle category) with complex background interference, our detector maintains consistently high detection accuracy.

Figure 15(a) exhibits a representative case of multi-scale characteristics in remote sensing object detection, simultaneously displaying both small vehicle targets and larger sports field targets, highlighting substantial scale variations. Figure 15(b) presents a challenging scenario of nested and stacked objects, where a large roundabout instance encompasses multiple small vehicle targets, posing significant detection difficulties. Figure 15(c) clearly illustrates that the basketball court possesses inherent rotation angles, necessitating rotated bounding boxes for optimal target fitting.

Our improved YOLOv8n-obb algorithm effectively addresses these challenges in remote sensing object detection, successfully completing detection tasks with high confidence scores, thereby demonstrating its superior performance in complex scenarios. The visualization comprehensively validates the algorithm’s capabilities in handling multi-scale objects, nested targets, and rotated instances while maintaining robust detection performance.

Furthermore, we validated our algorithm on the DIOR dataset, which comprises 23,463 images and 192,472 object instances spanning 20 distinct categories. The dataset provides both horizontal bounding box (HBB) and oriented bounding box (OBB) annotation formats. Experimental results of our improved YOLOv8-obb on the DIOR dataset are presented in Figure 16, with corresponding accuracy metrics detailed in Table 9.

Table 9 presents the experimental results comparing the original YOLOv8-obb and our improved algorithm on the DIOR dataset, with 3,488 labeled images per category. The results show our enhanced algorithm achieves higher mAP50 with fewer parameters than the original YOLOv8-obb.

Figure 17 displays detection heatmaps that demonstrate our improved algorithm's more focused attention on target features like airports and runways, compared to the original version's tendency to attend to less relevant areas. These visualizations provide qualitative evidence supporting the quantitative improvements shown in Table 9.