Advances in Aircraft Skin Defect Detection Using Computer Vision: A Survey and Comparison of YOLOv9 and RT-DETR Performance

1. Introduction

Commercial airplanes undergo regular inspections and maintenance checks to meet operational safety standards and regulatory requirements before flying. During General Visual Inspection (GVI) tasks, Maintenance, Repair, and Overhaul (MRO) operators carefully examine airplanes for defects that could compromise safety and performance [1,2]. These inspections focus on critical aircraft components, including the fuselage, structural elements, landing gear, tires, and control surfaces. Identifying defects in the aircraft’s exterior (or skin), such as cracks, corrosion, dents, or missing rivets, is particularly relevant for maintaining aircraft’s structural integrity, preventing further deterioration, and mitigating risks to underlying systems.

The degree of each inspection varies based on the aircraft’s operational phase. Pre-flight and post-flight inspections, conducted before and after every flight during the aircraft’s turnaround, are typically brief and targeted, lasting approximately 10 to 30 minutes depending on the aircraft type and available turnaround time. In addition to these rapid checks, more extensive routine inspections, categorized from A to D checks ¹, are mandated by aviation regulations and occur at varying intervals [3]. Type A checks are performed every 1 to 2 months and take approximately 6 to 12 hours, while D checks, the most comprehensive, occur every 6 to 10 years and can span 3 to 7 days ². Type B and C checks fall somehow in between ³.

Currently, aircraft surface inspections are performed manually by teams of licensed MRO operators and engineers who must undergo extensive training and certification before being qualified for such task. While these inspections require a high level of precision, they are also physically demanding, especially when accessing challenging areas such as the upper surfaces of airplanes. Additional equipment, such as scaffolds or aerial platforms, are often necessary, adding to the cost, time, and labor-intensive nature of the process [4,5]. Moreover, manual inspections are prone to human error and inconsistencies, which can affect the overall reliability of defect detection [6].

To address these limitations, industry and academia are increasingly focused on integrating computer vision technologies into defect detection processes [7,8]. This push has given rise to a variety of innovative solution such as: (i) Unmanned Aerial Vehicles (UAVs) for capturing real-time images of hard-to-reach defects; (ii) mobile or climbing robots for navigating aircraft surfaces, or (iii) smart hangars equipped with advanced camera systems to capture images from multiple angles.

Academic research in this field has seen rapid growth, particularly in the development and application of object detection and image segmentation algorithms for identifying surface defects on aircraft. A review of existing literature, summarized in Table 1 (Section 2), identified 21 research papers focused on such algorithms. Notably, about half of these studies were published in 2024 (the year this paper is being written), highlighting the growing urgency and relevance of this topic.

In parallel, the aviation industry has been investing in research and development to operationalize computer vision-based inspection systems. Projects span both hardware, such as UAVs and robotic systems, and software, including advanced computer vision algorithms. Our review, summarized in Table 2 (Section 2), uncovered 18 publicly documented industry projects across different companies and regions. This likely represents only a fraction of ongoing efforts, as many initiatives are not disclosed publicly.

The surge in interest in automated aircraft defect skin inspection can be attributed to recent advancements in computer vision technologies, which have significantly improved accuracy and efficiency within a short period of time — for a detailed benchmark analysis of various object detection algorithms since 2015, refer to [9]. Moreover, the post-COVID-19 period has introduced labor challenges, with many licensed inspectors retiring or transitioning to other roles, further driving the need for automated solutions in aircraft inspection.

The aim of this research is to explore the current state-of-the-art in hardware and software solutions for aircraft skin defect detection. Beyond conducting a detailed review of academic literature and industry innovations, we also experimentally assess the accuracy and feasibility of existing low-cost (both in terms of financial cost but also ease of deployment) hardware and software solutions. To that end, real-world data was collected from an abandoned aircraft with visible defects using a drone to capture video footage. The footage was then processed using widely adopted computer vision algorithms, specifically YOLOv9 [10] and RT-DETR [11], and their accuracy was evaluated.

In summary, the main contributions of this paper are as follows:

1.

Thorough review of existing hardware and software solutions for aircraft defect detection: The investigation highlights significant growth in academic research on computer vision for aircraft defect detection. Notably, all academic references are dated from 2019 onward, with over half of the publications appearing in 2024. On the industry side, numerous projects have been publicly disclosed since 2014, with half launched during or after the COVID-19 pandemic. These projects span various regions and industry sectors, including airlines (e.g., EasyJet, Air France, KLM, Singapore Airlines, Delta Airlines, Lufthansa), aircraft manufacturers (e.g., Airbus, Boeing), and organizations in the field of aerospace, robotics and artificial intelligence (e.g., Rolls-Royce, MainBlades, Donecle, SR Technics, ST Engineering, Jet Aviation, HACARUS) as well as research institutes (A*STAR).

2.

Collection of a new dataset of aircraft defects using a drone from an abandoned aircraft: Using an affordable and commercially available Parrot Bebop Quadcopter Drone (14-megapixel full HD 1080p, 180° field of view), we demonstrate that current readily available technology can achieve accurate aircraft defect detection. We validate this hypothesis by investigating a preserved Boeing 747 with defects located in Korat, Nakhon Ratchasima, Thailand. The collected dataset has been made publicly available (see here [12]) to support testing and benchmarking of future algorithms for aircraft defect detection. To the best of our knowledge, although several other studies have collected defect data from drone imagery [13,14,15,16,17,18], their datasets remain inaccessible due to confidentiality. Existing publicly available datasets primarily focus on defect-specific images (e.g., [19]), often emphasizing close-up views of defects rather than encompassing the entire aircraft (with and without defects) or accounting for variations in image quality caused by distance and angle. The lack of publicly available data is a frequently cited challenge in research papers on this topic [18,20].

3.

Evaluation and Comparison of Leading Computer Vision Algorithms for Aircraft Defect Detection: This work tests and compares YOLOv9 and RT-DETR, two of the most accurate computer vision models (refer to [9]). Our implementation of these algorithms for aircraft defect detection contributes in two major ways: (i) it offers a benchmark analysis highlighting the relative strengths of these models. The results show comparable overall performance, with RT-DETR slightly outperforming YOLOv9 prior to hyperparameter tuning, whereas YOLOv9 achieves superior results post-tuning; (ii) we demonstrate the practicality of leveraging modern computer vision algorithms for real-world aircraft defect detection, achieving moderate accuracy even without extensive tuning: mAP50 scores range from 0.699 for RT-DETR to 0.685 for YOLOv9. Additionally, a comparison with the YOLOv5 Extra-Large (YOLOv5x) model [21], developed in 2020, reveals a 10% improvement in mAP50 with YOLOv9, emphasizing the rapid advancements in computer vision.

4.

Hyperparameter tuning using Bayesian optimization: To avoid a myopic, single implementation of these algorithms, we applied hyperparameter tuning and analyzed its impact on performance metrics, particularly precision, recall, and mAP50. Our results show that tuning improves accuracy by 1.6% for RT-DETR and 6.6% for YOLOv9 across two datasets analysed — our drone-collected dataset [12] and a publicly available dataset [19]. We demonstrate that performance can be tailored to specific priorities by emphasizing different metrics during tuning. To encourage further research and facilitate refinement of the presented models, we have made our codes publicly available in [22].

5.

Discussion of limitations and challenges in implementing computer vision for aircraft defect detection: Our findings highlight the potential of computer vision to enhance defect detection, particularly given the rapid evolution of computer vision algorithms. However, discussions with industry experts reveal significant challenges that remain. These include: (i) regulatory constraints that vary across countries, hindering the development of standardized solutions; (ii) the need for seamless human-robot collaboration, as automation is intended to complement rather than replace human inspectors; (iii) the necessity of achieving exceptionally high accuracy levels to meet stringent safety standards. Current mAP50 levels of 0.70–0.75 may still be insufficient due to safety reasons. These findings emphasize the promise of this technology while motivating continued advancements to overcome these challenges.

The remainder of this paper is organized as follows: we begin with a review of existing work on computer vision for aircraft skin defect detection, highlighting publicly available projects from various industries in Section 2. Next, we describe the data collection process and provide details about the datasets used in this study in Section 3. The computer vision models employed for defect detection are introduced in Section 4, along with the metrics for evaluating algorithm performance in Section 4.3 and the hyperparameter tuning procedure in Section 4.4. In Section 5, we compare and analyze the performance of the object detection models, including the impact of hyperparameter tuning. We then delve into the challenges and important research directions in this field in Section 6. Finally, we conclude the paper in Section 7.

2. Research and Industry Survey

Our review identified a total of 21 research papers, with 14 published after 2020, illustrating the rapid growth in the field. These studies employ various algorithms, ranging from direct implementations of various versions of YOLO (though version 9, used in this paper, was not implemented) [16,17,20,23,24,25,26,27,28] to more advanced neural network models. For example, some researchers have enhanced existing architectures with additional layers, such as mask-scoring Region-based Convolutional Neural Network (R-CNN) [15] and U-Net [29]. A summary of these papers is presented in Table 1. The papers also address a wide range of defect types, including stains, lightning burns, dents, rust, cracks, corrosion, scratches, damaged and missing rivets or screws, among others.

Note that these papers focus on defects visible on the aircraft’s surface or fuselage, aligning with the scope of this review. More detailed inspections, which require Non-Destructive Testing (NDT) methods such as ultrasonic or radiographic testing, are typically conducted during B to D checks to assess internal structures, engines, or overlapping panels [30,31,32,33]. Additionally, other studies have explored object detection algorithms for identifying manufacturing defects [34,35,36,37], detecting ice formation on critical components [38], aircraft ducts defects [39], and verifying the condition of aircraft components, such as ensuring vents are closed, valves are properly positioned, and static ports are uncovered [40,41]. While these studies are not included in Table 1, which focuses on defects in visible aircraft skin, we acknowledge their relevance and importance as part of the broader inspection process.

The industry’s adoption of computer vision for defect detection is evident from the 18 publicly documented projects identified across different sectors. These initiatives span airlines (e.g., EasyJet, Air France, KLM, Singapore Airlines, Delta Airlines, Lufthansa), aircraft manufacturers (e.g., Airbus, Boeing), and organizations specializing in aerospace, robotics, and artificial intelligence (e.g., Rolls-Royce, MainBlades, Donecle, SR Technics, ST Engineering, Jet Aviation, HACARUS). Research institutes such as A*STAR are also actively contributing to advancements in this area. These projects not only focus on developing more robust object detection algorithms but also explore innovative hardware solutions for image and video collection. Emerging technologies include UAVs, drone swarms, mobile robotic platforms, climbing robots, and smart hangars equipped with high-quality cameras. Table 2 summarize these projects, detailing the hardware solutions employed and their respective applications.

3. Data Collection

For this study, we captured video images of an old Boeing 747 parked in the airplane park near Suranaree University of Technology (SUT), Thailand, shown in Figure 1a. We captured the video images with a Parrot Bebop 1 drone, shown in Figure 1b, with 1080p resolution using its 14-megapixel camera. Utilizing the image preprocessing tools available in Roboflow [67], we extracted 983 images from the 13 videos we took and manually annotated the visual defects into one of three defect categories, specifically, (1) missing-rivet, (2) rust, and (3) scratch. An example image with the three different defect types from this dataset can be seen in Figure 2. The annotated images were then randomly divided into 688 training, 197 validation, and 98 test images. We applied the Auto-Orient preprocessing option and resized the images into 640×640 pixels image size. The dataset is made publicly available and can be downloaded from [12]. For simplicity, we will call this dataset as Dataset I throughout this paper.

In addition to the dataset described above, we also chose to use a publicly available dataset provided by [19] in our study to further verify the models. We refer to this additional dataset as Dataset II in this paper. The dataset consists of 9651 training, 642 validation, and 429 test images of multiple defective airplanes. The defects in the images are annotated and classified into one of the five defect categories, i.e., (1) crack, (2) dent, (3) missing-head, (4) paint-off, and (5) scratch. Examples of images for these five defect types in Dataset II can be seen in Figure 3.

4. Methodology

Object detection is a computer vision technique designed to identify and classify objects of interest within images. This capability aligns well with our goal of detecting and classifying defects on aircraft surfaces. With the development of various object detection models, numerous datasets have been proposed for benchmarking these models. The most widely used dataset for this purpose is the Microsoft Common Objects in Context (MS COCO) dataset, introduced by [68]. This dataset includes 180,000 training images, 5,000 validation images, and 41,000 test images, featuring millions of labeled objects across 80 categories. Performance comparisons of different object detection models using the MS COCO dataset are detailed in [9]. For our study, we have chosen the two models well-known for their accuracy: RT-DETR and YOLOv9.

4.1. RT-DETR

DETR was first introduced by [69] in 2020, utilizing a transformer encoder-decoder architecture. Since its inception, DETR-based models have seen extensive development by various research groups (see [70] for a detailed review up to early 2023). To address the computational cost of DETR-based models, [11] proposed a new one-stage DETR variant called RT-DETR in late 2023, achieving real-time object detection. Their work introduced three key modifications to the original DETR model: (i) an efficient hybrid encoder was developed, combining Attention-based Intra-scale Feature Interaction (AIFI) and Convolutional Neural Network (CNN)-based Cross-scale Feature Fusion (CCFF) to alleviate computational bottlenecks; (ii) an uncertainty-minimal query selection scheme was implemented by integrating uncertainty into the loss function, reducing feature selection ambiguity; (iii) adjustments to the number of decoder layers were proposed, as the authors observed that changes in the number of decoder layers significantly impacted inference speed while maintaining accuracy. With these three modifications, RT-DETR reportedly outperformed earlier YOLO models and previous DETR series models in real-time object detection tasks on the MS COCO dataset. In this study, we implemented the RT-DETR Extra-Large (rtdetr-x) model available in [71].

4.2. YOLOv9

YOLO is one of the most well-known object detection models, recognized for its well-balanced tradeoff between efficiency and accuracy. The first version, YOLOv1 [72], introduced in 2015, was the first one-stage model to achieve true real-time object detection. Since then, YOLO has attracted widespread adoption. Numerous advancements and adaptations by different research groups have further improved its speed and accuracy while extending its capabilities to tasks such as image classification and segmentation. Readers interested in a detailed overview of the YOLO series models up to YOLOv8 are referred to [73]. In early 2024, [10] proposed YOLOv9, a significant advancement in the YOLO series ⁴. YOLOv9 incorporates several key innovations: Programmable Gradient Information (PGI) for more reliable gradient updates, the Generalized Efficient Layer Aggregation Network (GELAN) to address data loss, and the integration of DETR and attention mechanisms to enhance detection capabilities. These improvements have reportedly resulted in superior performance on the MS COCO dataset compared to previous YOLO versions and RT-DETR. For our study, we utilized the YOLOv9 Large (yolov9e) model as implemented by [76].

4.3. Performance Metrics

The performance of object detection models can be assessed using several different metrics. In this study, we evaluated the accuracy of the considered algorithms based on their Precision $\left(\mathrm{P}\right)$, Recall $\left(\mathrm{R}\right)$, and mean Average Precision $\left(\mathrm{mAP}\right)$. We specifically used the metrics quantification method described in [77]. To quantify the three mentioned metrics, it is necessary to define the notion of True Positive $\left(\mathit{T}\mathit{P}\right)$ calculated based on the Intersection over Union $\left(\mathit{I}\mathit{o}\mathit{U}\right)$ of the bounding boxes with a $50\mathbf{\%}$ threshold. Here, a bounding box is a rectangle added to tightly enclose a defect on the image. Given a predicted bounding box that intersects with a ground truth bounding box, the $\mathit{I}\mathit{o}\mathit{U}$ is defined as the areas of the intersection divided by the union between the two bounding boxes, i.e.,

$$IoU={\displaystyle \frac{B_{pred}\cap B_{truth}}{B_{pred}\cup B_{truth}}}$$

(1)

Here, ${\mathit{B}}_{\mathit{p}\mathit{r}\mathit{e}\mathit{d}}\mathrm{and}{\mathit{B}}_{\mathit{t}\mathit{r}\mathit{u}\mathit{t}\mathit{h}}$ denote the area of the predicted and the ground truth bounding boxes, respectively, calculated in pixel units. The $\left(\mathit{T}\mathit{P}\right)$ is then defined as the number of $\mathit{I}\mathit{o}\mathit{U}$ instances that are higher than $50\mathbf{\%}$ with the correct defect classifications.

The Precision $\left(\mathrm{P}\right)$ can then be computed as the ratio of the $\left(\mathit{T}\mathit{P}\right)$ to the total number of predicted bounding boxes $\left({\mathit{N}}_{{\mathit{B}}_{\mathit{p}\mathit{r}\mathit{e}\mathit{d}}}\right)$, i.e.,

$$P={\displaystyle \frac{TP}{N_{B_{pred}}}}$$

(2)

Here, $\mathrm{P}$ quantifies the ability of the model to correctly classify the detected defects.

On the other hand, the Recall $\left(\mathrm{R}\right)$ is defined as the ratio between the $\left(\mathit{T}\mathit{P}\right)$ and the actual number of defects present in the images $\left({\mathit{N}}_{{\mathit{B}}_{\mathit{t}\mathit{r}\mathit{u}\mathit{e}}}\right)$, i.e.,

$$R={\displaystyle \frac{TP}{N_{B_{truth}}}}$$

(3)

Thus, $\mathrm{R}$ signifies the ability of the model to detect the defects present in the given images. To encapsulate the precision and recall of the model, we can then define the area under the precision-recall curve as Average Precision $\left(\mathrm{AP}\right)$. The average value of $\mathrm{AP}$ across all the defect types is defined as mean Average Precision $\left(\mathrm{mAP}\right)$. Because we calculated $\mathrm{mAP}$ at the $50\mathbf{\%}$ threshold of the $\mathit{I}\mathit{o}\mathit{U}$, we will be using $\mathrm{mAP}50$ to denote the $\mathrm{mAP}$ of the models in this study. Additionally, we also compared the inference speed of the models to detect and classify the defects, and the convergence speed of the models during training progress.

4.4. Hyperparameter Tuning

Training an NN model involves a number of hyperparameters that need to be configured beforehand, and the values we choose for these hyperparameters influence the performance of the algorithm [78], [79]. Given datasets for a specific application, one might resort to the default values of hyperparameters implemented by the NN packages or manually configure them based on available literature, experiences, and/or experiments.

Alternatively, a hyperparameter tuning strategy leveraging optimization techniques can also be implemented. Since this tuning strategy is generally computationally expensive, sequential model-based optimization methods, such as Bayesian optimization, are often employed [80,81,82]. In this approach, the optimization algorithm identifies the optimal point of an approximated model and iteratively updates the surrogate model, enabling the approximate optimum to be found with fewer evaluations of the computationally expensive original function.

In this study, we tuned the object detection models employing the Bayesian optimization approach by viewing the training processes as a black-box function. Specifically, we implemented the gp_minimize function available in [83]. We used the negative value of the weighted average of the accuracy metrics described in Section 4.3 as the objective function for the optimization scheme, i.e.,

$$\begin{array}{r}\underset{\mathcal{H}}{\mathrm{m}\mathrm{i}\mathrm{n}}\left(-\left[\begin{array}{ccc}{\mathrm{w}}_{\mathrm{P}}& {\mathrm{w}}_{\mathrm{R}}& {\mathrm{w}}_{\mathrm{mAP}50}\end{array}\right]{\left[\begin{array}{ccc}\mathrm{P}& \mathrm{R}& \mathrm{mAP}50\end{array}\right]}^{\mathrm{\top }}\right).\end{array}$$

(4)

Where, $\mathcal{H}$ denotes the hyperparameters space we considered and $\left[\begin{array}{ccc}{\mathbf{w}}_{\mathrm{P}}& {\mathrm{w}}_{\mathrm{R}}& {\mathrm{w}}_{\mathrm{mAP}50}\end{array}\right]$ are the weight factors of the accuracy metrics. Specifically, the hyperparameter space $\mathcal{H}$ comprises 29 hyperparameters, as listed in Table 3. For the tuning process, we set a limit of 100 function calls, where each function call involves a complete training cycle of an object detection model using Dataset I. In our analysis, we assigned weight factors of $\left[\begin{array}{ccc}0.3& 0.4& 0.3\end{array}\right]$ to maintain a balance across accuracy metrics. Additionally, we conducted a sensitivity analysis for other weight combinations (refer to Section 5.2).

Additional preprocessing was applied to Dataset II to reduce training time during the tuning process, given its large size. The training and validation data was split into 20 random pools of 500 training and 200 validation images each. At the start of each function call during the tuning iteration, a pair of training and validation datasets was randomly selected from these pools. This approach ensures that each function call during tuning process utilized only 500 training and 200 validation images, while still leveraging the entire dataset of 9,651 training and 642 validation images over the complete tuning procedure, which involved 150 function calls.

5. Experimental Results

The experiments in this study were conducted on datasets described in Section 3. The tuning and training of the models were performed on the ASPIRE2A nodes provided by the National Supercomputer Center (NSCC) Singapore. Each training iteration was conducted utilizing one AMD EPYC Millan 7713 CPU and one NVIDIA A100-40G SXM GPU with 128 GB of DDR4 memory [84].

5.1. Performance Comparison

The performance results — including Precision (P), Recall (R), and mAP50 — for the two models, RT-DETR and YOLOv9, are presented in Table 4 and Table 5 for Datasets I and II, respectively. These tables display results both before and after hyperparameter tuning, as described in Section 4.4. For the post-tuning results, we applied weight factors of $\left[\begin{array}{ccc}0.3& 0.4& 0.3\end{array}\right]$ to maintain a balance across accuracy metrics, with slightly more emphasis on Recall (R) to minimize false negatives.

In addition to the RT-DETR and YOLOv9 results, we also include, for Dataset I, the performance of the YOLOv5 Extra-Large (YOLOv5x) model from [21] to examine the evolution of YOLO’s accuracy since 2020. For Dataset II — which is an external dataset provided by [19] — we include the results of their AI model. These results are assumed to reflect post-tuning performance, although the specific tuning method used was not disclosed.

The results show improvement of newer object detection models, i.e., RT-DETR and YOLOv9, in most of the accuracy metrics, compared to the previously employed models, i.e., YOLOv5 and model used by [19]. The performance improvement of these newer models shows the benefits that can be gained from utilizing the updated state-of-the-art object detection models.

Comparing the performance of RT-DETR and YOLOv9 in the current case study, the after-tuning results show a slight upper hand on the accuracy of YOLOv9 as seen in Table 4 and Table 5, consistent with the results reported by [10]. Moreover, YOLOv9 also achieved a faster inference speed of approximately 66 frames per second, compared to the RT-DETR’s inference speed of around 41 frames per second. However, comparing the training progress between the two models, we observed that RT-DETR has a faster convergence speed while maintaining a comparable accuracy as shown in Figure 4. In these plots, we show the average of the accuracy metrics from multiple training instances, at each epoch, as solid lines. The shaded areas in the plots display the one-standard deviation below and above the mean of the training instances. In this comparison, we considered the training progress of RT-DETR and YOLOv9 models for both Datasets I and II. Here, RT-DETR achieved comparable “P”, “R”, and “mAP50” before 100 epochs, while YOLOv9 needed more than 150 epochs.

5.2. Accuracy Metrics Emphases During Tuning

As expected, from Table 4 and Table 5, we can see that all algorithms show some performance improvement after tuning. Specifically, for YOLOv9 model, we can observe an average improvement around 8.6% in “P”, 7% in “R”, and 6.6% in “mAP50”. However, it can be argued that the benefit of hyperparameter tuning is minimal for the RT-DETR model as it can be observed to have around 6% improvement in “P” but almost no improvement in “mAP50” and even an average of 3% decreased performance in “R”. To further inspect this, we are interested in observing the effect of different emphases on the accuracy metrics during tuning procedure.

To compare the tuning effect of different emphases on the accuracy metrics, we varied the objective function (weights) of the optimization problem (4). Specifically, we considered three additional variations of weight factors $\left[\begin{array}{ccc}{\mathbf{w}}_{\mathrm{P}}& {\mathrm{w}}_{\mathrm{R}}& {\mathrm{w}}_{\mathrm{mAP}50}\end{array}\right]$ that give different emphases to the different accuracy metrics separately, i.e., $\left[\begin{array}{ccc}1& 0& 0\end{array}\right]$, $\left[\begin{array}{ccc}0& 1& 0\end{array}\right]$, and $\left[\begin{array}{ccc}0& 0& 1\end{array}\right]$. We performed this tuning study on Dataset I. To ease the comparison, we plotted blue and red dotted lines to show the accuracy of RT-DETR and YOLOv9 models before tuning, respectively. Two different scales of the vertical axis are used to highlight the improvement in the accuracy of the models by zooming the upper part of the plots.

The comparison in Figure 5 demonstrates that a more balanced emphasis on accuracy metrics during tuning yields better overall performance in “P”, “R”, and “mAP50”. Interestingly, although “mAP50” is intended to represent both precision and recall, focusing solely on this metric during the tuning process appears to lead to suboptimal overall accuracy. This is particularly evident in the case of the YOLOv9 model. For the RT-DETR model, although emphasizing “mAP50” during tuning results in higher recall improvements, the greatest overall accuracy improvement is achieved with a balanced weight.

6. Discussion

The results of this research demonstrate that a relatively inexpensive UAV system equipped with a 14-megapixel camera can achieve moderately accurate results (mAP50: 0.70–0.75) in identifying aircraft defects. This suggests that existing methods have potential as supportive tools for aircraft defect detection. However, feedback from MRO experts highlights that these accuracy levels remain insufficient due to safety concerns. This underscores the need for further advancements in computer vision algorithms specifically tailored to aircraft defect detection to surpass 90% accuracy levels. One major challenge faced by computer vision researchers is access to comprehensive datasets, which constitutes a significant bottleneck in advancing current algorithms. In this paper, we aim to address this issue by providing access to the recorded and collected data [12]. We also include a summary of links to publicly available defect image datasets in the Section titled Available Datasets of Aircraft Defects.

In addition to algorithmic improvements, discussions with MRO experts revealed uncertainties regarding the optimal hardware solutions for defect data collection in real-world context. We identified four main categories of hardware solutions — UAVs, smart hangars, mobile robot platforms, and climbing robots — each with distinct advantages and limitations. A summary of the key insights from these discussions is provided below, serving as a starting point for future investigations into hardware selection and operational integration.

Unmanned Aerial Vehicles (UAVs):

Advantages:

• Capable of reaching difficult areas, such as the upper fuselage and tail sections, without the need for scaffolding or lifts;
• Able to conduct inspections quickly, making them particularly useful for pre/post-flight inspections;
• Swarm of drones can make routine inspections inside hangars even faster;
• As suggested by our review of literature, drones have been already widely tested by both academia and industry;
• Relatively inexpensive. As shown through the experiment in this paper even inexpensive drones and cameras can already provide good levels of accuracy;
• Easily transferable between inspections. Can be deployed across multiple aprons/stands.

Limitations:

• UAV operations are subject to strict aviation regulations, which may limit their deployment in certain areas of the airport. Additionally, regulatory differences across airports pose challenges for developing standardized solutions;
• Drones may only be permitted for use inside hangars, reducing their utility for pre/post-flight inspections. Solutions like tethers attached to drones could address this, but may limit their range and effectiveness;
• May obstruct other maintenance activities, and therefore human-drone coordination requires planning;
• Battery life and payload capacity can restrict flight duration and the types of sensors that can be equipped;
• Planning 3D trajectories for drones around aircraft surfaces can be challenging, particularly when coordinating a swarm of drones.

Smart Hangars with High-Resolution Cameras:

Advantages:

• Less constrained by regulations as the system operates within a controlled environment and does not interfere with other operations;
• Easily integrates with human-led maintenance activities without obstructing ongoing tasks;
• Once installed, these systems provide reliable, consistent coverage without needing repositioning, and maintenance costs remain relatively low after the initial setup.

Limitations:

• Can only be used inside hangars limiting its use at pre/post flight inspections;
• Fixed cameras may struggle to capture certain parts of the aircraft, especially at specific angles. Additionally, the quality of images may be reduced compared to drones or robots that capture close-up visuals;
• The system is not transferable, requiring airports to dedicate specific hangars equipped with this technology and plan aircraft inspection schedules accordingly;
• Installation and integration of high-resolution camera systems involve substantial upfront investment.

7. Conclusion

This paper reviews the current state of computer vision algorithms for supporting aircraft surface defect detection during inspection routines, whether pre/post-flight during turnarounds or at routine intervals (e.g., every few months). Specifically, we analyze advances from both academic (21 publications identified) and industry perspectives (18 projects identified).

Through this review and expert discussions, we identified existing object detection algorithms tailored for aircraft defect detection and explored hardware solutions for image and video collection, including drones (UAVs), robot platforms, climbing robots, and smart hangars equipped with cameras.

This study goes beyond literature review by applying these concepts to real-world scenarios. We collected real data on aircraft defects using a drone, making this dataset publicly available for further research. Additionally, we evaluated and compared the performance of two state-of-the-art object detection models — RT-DETR and YOLOv9. Results show that both models achieved mAP50 scores between 0.70 and 0.75. YOLOv9 demonstrated a slight edge in accuracy and inference speed, while RT-DETR achieved similar accuracy with faster training convergence. A comparison with YOLOv5 from 2020 highlights the rapid evolution of computer vision algorithms, with YOLOv9 achieving a 10% improvement in mAP50. Despite these advancements, we acknowledge that an mAP50 of 0.70–0.75 is insufficient for meeting the risk requirements of aviation operations. Nonetheless, the continuous and rapid progress in computer vision is encouraging.

We also identify and discuss important challenges that still need to be adressed such as regulatory constraints, human-robot integration, path planning optimization for defect image collection, and weather resilience.

In conclusion, this paper provides a comprehensive overview of the current state of computer vision in aircraft defect detection, serving as a foundation for future research in a field that has gained significant attention in recent years.