Hybrid Deep Learning Framework for Crack Detection and Condition Classification of Prestressed Concrete Sleepers

1. Introduction

Railway sleepers are crucial to the maintenance of track geometry, load distribution, and general operating safety in the infrastructure. Prestressed concrete sleepers are frequently employed because of their longevity and structural efficiency; nonetheless, they are prone to degradation during service, such as cracking, spalling, abrasion, and fastener related damage. Longitudinal cracking is one of the most observed and structurally significant defects, as it may affect load transfer, fastening performance, and long-term durability. Previous research has also shown that longitudinal cracking occurs particularly at rail-seat and mid-span regions. Kaewunruen and Remennikov found that combining static and dynamic loads can greatly impact crack formation and propagation in prestressed concrete sleepers [16]. Similarly, the sleeper cracking occurs gradually under service environments and is heavily influenced by load amplitude, support conditions, and material qualities [23]. Field and computational studies show that sleeper cracks do not always signify rapid structural collapse, but they can affect stiffness and load transfer characteristics, hastening long-term degradation [15]. These findings emphasize the need for early fracture detection for optimal railway maintenance planning.

Recent developments in deep learning, especially convolutional neural networks (CNNs) and transformer-based models, have demonstrated promising results in the identification of infrastructure defects. CNN-based models outperformed traditional image processing approaches for detecting concrete cracks [6]. CNN performance is highly influenced by dataset characteristics, particularly for narrow or obscured cracks [14]. While CNNs have proven effective across infrastructure domains, limited research exists on their suitability for detecting low-contrast cracks in concrete sleepers under real-world operational conditions. In this context, a hybrid pipeline is technically and operationally justified. VGG16 and VGG19 are popular CNN architectures for defect-oriented image classification due to their uniform design of stacked 3×3 convolutional filters that preserve fine spatial and texture information [29]. In concrete crack detection studies, VGG16 performs reliably in crack detection, balancing sensitivity and generalization under variable lighting and irregular crack patterns [6]. VGG19 extends VGG16 by increasing depth for richer feature representation, but its performance gains are contingent on training strategy and classifier design [33,35]. Investigations show that over-tuning or incorporating Global Average Pooling (GAP) layers may reduce sensitivity to fine cracks by suppressing localized features [26]. Consequently, the literature suggests that baseline VGG19 and VGG16 architectures are often more effective for surface defect classification in complex field conditions, while deeper residual networks such as ResNet50 may underperform due to over-pooling in limited, high entropy datasets [16,19].

Deep learning has made major advances in crack detection, yet thin, low-contrast, and fragmented cracks remain challenging, especially on a varied background. In 2024 survey found that CNNs remain strong for texture-sensitive patterns, while transformers excel in capturing long-range context in visually complex scenes [19]. Attention-augmented classifiers leveraging transfer learning and spatial attention have shown improved sensitivity when training data are limited or noisy [12]. These trends suggest CNNs screening with object detection enables efficient sleeper defect localization.

Object detection frameworks enable spatial crack localization; YOLO (You Only Look Once) detectors are widely adopted for their real-time performance and strong accuracy speed balance. For real-time detection in operational environments, modern YOLO implementations offer a strong speed-accuracy trade-off. Detection performance should be reported using COCO-style metrics (mAP@0.5, mAP@0.5-0.95) and per class precision, as accuracy alone overlooks classification and localization interplay; precision-recall analysis remains standard. In contrast, convolutional approaches, end-to-end transformer-based models like RT-DETR eliminate non-maximum suppression (NMS) by treating detection as bipartite set prediction and a hybrid encoder with optimized query selection for improved performance under occlusion [36]. Similarly, ViT-enhanced DETR models have also achieved state-of-the-art results on micro crack datasets by combining global attention with localized feature refinement [24]. A recurring difficulty in railway imagery is the small-object regime: cracks often occupy few pixels at standard inference resolutions. Slicing-Aided Hyper Inference (SAHI) addresses this by tiling high-resolution images with overlap, increasing effective pixel density per object and consistently improving AP for small and elongated targets without retraining [1]. Oriented bounding boxes (OBB) in modern YOLO toolchains enhance localization accuracy for slanted or rotated defects, especially beneficial in railway inspections where hairline cracks are low-contrast and partially obscured by ballast or fasteners [30].

Many previous studies have successfully applied YOLO architectures to concrete and pavement crack detection, reporting high detection efficiency for visually prominent cracks [13]. YOLO-based detectors perform particularly well for structural components with clear geometric features, such as sleepers, rails, and fasteners, making them suitable for large-scale or vehicle-mounted inspection systems [20]. However, crack detection remains challenging for YOLO models, as thin, low-contrast, fragmented cracks reduce recall due to anchor-based matching, intersection over union (IoU) limits, and thresholding [20]. Transformer-based detectors represent a global self-attention mechanism for object detection. The DETR framework introduced end-to-end detection without anchors or non-maximum suppression, enabling improved contextual reasoning when object boundaries are ambiguous [22]. RT-DETR was proposed to achieve real-time performance while preserving transformer-based contextual learning [36]. Transformer-based detectors improve sensitivity to subtle and partially occluded defects but often produce coarser localisation and lower IoU-based metrics for elongated objects such as cracks [10]. Second, object-level detection benefits from deploying YOLO for real-time sleeper localization and a transformer detector (RT-DETR) to increase crack recall under shadows and occlusion.

To address these gaps, this study proposes a hybrid deep learning framework combining image-level classification and object-level detection for railway sleeper inspection. Using field data from the Arezzo LFI line, it evaluates CNN classifiers (VGG16, VGG19, and ResNet50) and detectors (YOLOv11 and RT-DETR). The study aims to provide technical and engineering insights for AI-assisted sleeper inspection, supporting practical deployment in railway maintenance systems.

Structural integrity of prestressed concrete railway sleepers is vital for track safety and long-term asset performance. In service, sleepers experience complex load paths rail-seat bending, center negative bending, and ballast reactions being exposed to moisture and freeze–thaw cycles. While EN 13230 [5] defines design and testing requirements, field environments like the LFI, Arezzo in Italy reveal ageing related damage, requiring robust vision systems resilient to shadows and weather-induced surface degradation [21,27].

Proposed Hybrid Pipeline for Railway Sleeper Inspection

This pipeline addresses key challenges such as ballast occlusion, small-object detection, and elongated crack geometry. It consists of three stages:

Stage 1: CNN-based image-level screening (VGG16, VGG19, ResNet50) to filter in-service sleepers from yard sleepers.

Stage 2: YOLO-based sleeper localization for real-time component detection.

Stage 3: Transformer-based crack detection (RT-DETR) for improved sensitivity under shadows and occlusion.

Figure 1 illustrates the proposed hybrid inspection pipeline for prestressed concrete railway sleepers. Enhancements such as SAHI (Slicing-Aided Hyper Inference) and oriented bounding boxes (OBB) can be incorporated to improve detection of small and rotated cracks; however, the present study reports baseline detector performance without ablation of these enhancements.

2. Materials and Methods

2.1. Field Investigations and Defect Characterization: LFI Arezzo Corridor

A systematic field investigation was conducted along the Arezzo District segment of the La Ferroviaria Italiana (LFI) [21] regional line to evaluate the in-service condition of prestressed concrete sleepers and identify deterioration mechanisms influenced by operational, environmental, and maintenance factors. The survey encompassed active track sections, a discarded sleeper storage yard, and the Baraclit industrial siding (km 31+155), enabling assessment across multiple service-life stages and degradation profiles.

Visual inspections revealed consistent longitudinal cracking along sleeper axes, indicating a multifactorial origin involving mechanical loading, environmental exposure and maintenance related stress affecting both surface and sub-rail seat regions. Currently, LFI relies on visual crack detection and acoustic percussion testing called “Chiavardino” to assess internal integrity [27]. However, visual methods, including AI systems, risk false negatives without contextual support, highlighting the need for hybrid AI approaches that combine image-level and object-level reasoning for more reliable diagnostics.

Longitudinal cracking emerged as the most prevalent and structurally significant defect observed during the field survey. Consistent with EN 13230 and Rete Ferroviaria Italiana (RFI) maintenance guidelines, this damage mode develops through the combined influence of construction, loading, environmental, and maintenance-related factors. Construction related factors, such as prestressing tendon misalignment and inadequate concrete compaction can introduce internal tensile stress concentrations and void formation, increasing susceptibility to crack initiation. Wheel-rail interaction, particularly in curved track geometries, induces lateral loading, torsional deformation, and uneven ballast reactions, resulting in repetitive tensile cycling during service. Maintenance operations, including ballast tamping, track lifting, and rail replacement) can impose localized transient loads and support disturbances that accelerate crack initiation and propagation.

• Construction-related conditions: Prestressing tendon misalignment and incomplete concrete compaction, leading to internal tensile concentrations and void formation.
• Wheel–rail interaction in curved track geometries: Inducing lateral loading, torsional deformation, and uneven ballast reactions that translate into repetitive tensile cycling.
• Maintenance-induced overstressing: Particularly from ballast tamping, track lifting, and rail replacement operations imposing localized point loads.
• Moisture ingress below the rail plate (piastrina): Facilitating freeze-thaw expansion and potential corrosion of embedded prestressing reinforcement.

Sleeper cracking represents a progressive deterioration process rather than a discrete failure state, where visual severity does not always correlate with structural criticality. Consequently, pixel-level inspection alone risks misclassification, underscoring the need for the context-aware hybrid AI approach.

2.2. Field Case Study

The Baraclit industrial siding serves as a natural testbed for ageing-related sleeper deterioration, with low traffic and freight load, as shown in Table 1 Minimal maintenance, aside from replacing failed units, ensures defects reflect natural degradation. Unlike fatigue driven mainlines, deterioration appears as discolouration, surface scaling, and subtle cracking, providing a visually complex environment ideal for validating AI-based computer vision models under realistic field conditions.

The study underscores the interplay of multiple defect mechanism construction flaws, wheel-rail forces, maintenance-induced stresses, and moisture ingress that collectively accelerate longitudinal cracking in prestressed concrete sleepers. Figure 1 provides spatial context for the corridor and siding, while Figure 2 illustrates both surface and sub-rail seat cracks, highlighting the difficulty of detecting hidden defects beneath ballast. Traditional LFI methods (Figure 3) often miss sub-surface cracks, while the proposed AI-driven pipeline enables automated, predictive inspection improving reliability and reducing risk in high load, load maintenance railway environments.

2.3. Image-Based Classification of Railway Sleeper Conditions

The classification task distinguished in-service railway sleepers from discarded sleepers removed due to structural failure, representing two lifecycle stages relevant to practical railway maintenance and inspection planning.

The curated dataset comprises 289 high-resolution field images collected from the Arezzo-RFI corridor, Baraclit Junction, and the LFI maintenance yard. All images were captured under natural conditions without surface cleaning, preserving ballast coverage and environmental complexity as in Table 2.

Three ImageNet-pretrained CNN architectures were employed: VGG16, VGG19, and ResNet50. Their structural characteristics and expected strengths are summarized in Figure 4.

The classifier head was replaced with a task-specific fully connected layer and optimized using binary cross-entropy loss. Training parameters are detailed in Table 3.

To mitigate sampling variability in the small and imbalanced dataset, results should ideally be verified across multiple random seeds or grouped splits to avoid near-duplicate leakage. In the present study, three CNN models VGG16, VGG19, and ResNet50 were evaluated on the curated dataset using an 80/20 train-test split. To maintain a single source of truth and avoid redundant reporting, consolidated class-wise precision, recall, F1-score, and overall accuracy are reported in Table 4, which forms the basis for all subsequent interpretation and discussion.

2.4. Object Detection

Object detection is central to the proposed railway inspection framework, enabling accurate localization of sleepers and surface cracks under real-world conditions. Two advanced architectures were evaluated: YOLOv11, a convolution-based model optimized for real-time inference, and RT-DETR, a transformer-based detector leveraging global attention for contextual reasoning. The analysis emphasizes detection accuracy, robustness under field variability, and operational feasibility.

The dataset was curated in COCO format using Label Studio, comprising two classes: sleeper and crack. Images were resized to 640 × 640 pixels and augmented with mosaic composition, color jittering, and horizontal flipping to improve resilience against illumination variability and ballast interference. Training employed the AdamW optimizer with a composite loss function integrating GIoU-based localization loss, classification loss, and distribution focal loss for refined bounding box regression. These strategies ensured robust feature learning across heterogeneous track environments.

3. Results

3.1. Classification Results and Model Interpretation

The quantitative performance of the evaluated CNN and transfer-learning architectures for railway track and sleeper-dump classification is summarized in Table 5. Precision, recall, F1-score, and overall accuracy were computed for each class to capture both class-specific and aggregate behavior. The results provide insight into the discriminative capacity of different architectures under high-entropy field conditions.

Among all models, VGG19 with the baseline classification head achieved the highest overall accuracy (98.2%), followed by VGG16 (95.0%). ResNet50 showed comparatively lower performance (88.0%), particularly in the sleeper-dump class, where recall dropped to 0.62. The GAP + BN variant of VGG19 exhibited the weakest performance, indicating loss of discriminative spatial detail. These findings suggest that texture-preserving architectures with dense convolutional representations are more effective for railway sleeper imagery than deeper residual abstractions or aggressively pooled global representations. Several studies have reported that VGG-based architectures outperform deeper residual networks for texture-dominated inspection tasks, such as VGG19 achieved almost 99% accuracy in surface crack recognition [4], VGG16 and VGG19 exceeded ResNet-50 in concrete crack datasets [2,25], and VGG16 showed superior stability over ResNet-50 in railway fastener detection under minimal data [8,32]. The performance trends observed in Table 5 are strongly consistent with previous studies that VGG architectures can be more reliable than deeper residual networks for texture-dominated inspection tasks under limited or high-entropy data. Studies from other imaging domains are cited here only to support general model-behaviour trends under limited/noisy data (e.g., depth penalty and pooling effects), while domain-specific comparisons are prioritised for railway and concrete defect inspection.

3.1.1. Explainability via Grad-CAM—Image Classification

To interpret the internal decision-making behavior of the classification models, Gradient-weighted Class Activation Mapping (Grad-CAM) was applied to representative test images for VGG16, ResNet50, and VGG19 variants. Grad-CAM enables visualization of spatial regions contributing most strongly to the predicted class by projecting class-specific gradient information onto the final convolutional feature maps. VGG-based models exhibited concentrated activation on sleeper surfaces regions, whereas ResNet50 showed diffuse attention extending into ballast and background areas. The GAP-based VGG19 variant demonstrated weakened spatial localization, highlighting the importance of preserving convolutional feature resolution for defect-oriented classification. Figure 5 illustrates these visualizations.

Grad-CAM visualizations reveal that ResNet-50 produces spatially diffuse activations over ballast and background textures, a behavior consistent with prior observations for deep residual networks [28]. In contrast, VGG19-based models exhibit progressively improved localization, with Block-5 fine-tuning enhancing defect relevant semantic activation [Zeiler and Fergus, 2014]. Global average pooling yields compact activation maps; however, in this dataset, the reduced spatial detail was associated with weaker defect-focused localization and lower classification performance, highlighting the importance of preserving fine texture cues for crack-prone imagery [34].

Training and validation accuracy and loss curves for the evaluated architectures are presented in Figure 6. VGG16 converged quickly, with training and validation curves remaining close and validation accuracy stable, indicating good generalization and limited overfitting. ResNet50 converged more slowly and exhibited a larger gap between training and validation loss, with fluctuating validation accuracy suggesting weaker generalization on this field dataset. VGG19 with the baseline head produced the most stable convergence and highest validation accuracy, while the Block-5 fine-tuned variant showed minor fluctuations. The GAP + BN head resulted in higher validation loss and lower validation accuracy, confirming that reducing spatial detail through global pooling negatively affected performance.

3.2. Object Detection Metrics and Performance Comparison

Performance evaluation prioritized mean Average Precision (mAP) over raw accuracy, as mAP provides a more reliable measure of localization and classification performance in imbalanced datasets. YOLOv11 achieved mAP@0.5 of 0.65 and mAP@0.5–0.95 of 0.45, with an inference speed of approximately 90 FPS, confirming its suitability for real-time deployment. RT-DETR recorded mAP@0.5 of 0.56 and mAP@0.5–0.95 of 0.42, with an inference speed of approximately 35 FPS, indicating semi-real-time performance. A study compared YOLO and RT-DETR for railway tie defect detection, reporting that YOLO variants had 40–60% faster inference while RT-DETR achieved higher detection precision only after extensive multi-scale feature tuning [7]. Another study showed that RT-DETR performs well when defects are visually distinct and training is domain-specific [31].

YOLOv11 demonstrates superior precision and real-time inference capability, making it highly suitable for continuous onboard inspection. In contrast, RT-DETR shows qualitatively stronger retention of low-contrast crack patterns in challenging scenes (Figure 10, Figure 11 and Figure 12), although the aggregated two-class metrics indicate lower precision-recall balance than YOLOv11 under the present thresholding. Chintalapally and Bejjam, found that when optimized, YOLO models gave higher precision and efficiency in crack and corrosion detection, but performance was greatly dependent on defect size, contrast, and dataset complexity [9]. Stable convergence and strong real-time performance of YOLOv11 in the combined sleeper and crack class evaluation, precision and recall were 0.70 and 0.68, respectively (Table 6). However, crack detection remained inherently challenging due to low contrast, irregular morphology, and partial occlusions, resulting in confidence scores typically ranging from 0.3 to 0.7. In comparison, RT-DETR exhibited superior sensitivity to fragmented and visually degraded cracks by leveraging global attention mechanisms, which enhanced recall but introduced a trade-off in the form of reduced precision and increased inference time. These contrasting behaviors are illustrated in Figure 7, Figure 8 and Figure 9, highlighting the operational implications of each detection model.

Figure 7. YOLOv11 qualitative detection classes with confidence scores [by author].

Figure 7. YOLOv11 qualitative detection classes with confidence scores [by author].

Figure 8. YOLOv11 training and validation loss curves [by author].

Figure 8. YOLOv11 training and validation loss curves [by author].

Figure 9. YOLOv11 confidence-based performance curves [by author].

Figure 9. YOLOv11 confidence-based performance curves [by author].

Figure 10. RT-DETR qualitative detection classes with confidence scores [by author].

Figure 10. RT-DETR qualitative detection classes with confidence scores [by author].

Figure 11. RT-DETR training and validation loss curves [by author].

Figure 11. RT-DETR training and validation loss curves [by author].

Figure 12. RT-DETR confidence-based performance curves [by author].

Figure 12. RT-DETR confidence-based performance curves [by author].

Sleeper localization consistently achieves near-unity confidence, reflecting the model’s ability to exploit distinct geometric features and structural cues. In contrast, crack detection exhibits moderate confidence due to the inherent challenges posed by thin, low-contrast geometries and irregular surface patterns, which limit feature distinctiveness under field conditions.

YOLOv11 delivers superior precision and real-time processing capability, making it highly effective for continuous onboard inspection where speed and reliability are critical. Conversely, RT-DETR demonstrates greater sensitivity to complex and fragmented crack patterns, though at the expense of inference speed, positioning it as an appropriate choice for detailed secondary analysis where detection completeness outweighs throughput.

Precision for sleeper detection remains consistently high across confidence thresholds, reflecting the model’s ability to exploit distinct geometric features. However, recall for cracks declines as confidence thresholds increase, indicating that stricter filtering suppresses detections of fine, low-contrast cracks, which are inherently more challenging to localize under field conditions.

Attention-based predictions effectively capture fragmented and low-contrast cracks by leveraging global contextual reasoning, while simultaneously maintaining robust and accurate sleeper localization. This demonstrates the ability of the model to balance sensitivity to fine structural anomalies with reliable detection of dominant geometric features.

The smooth progression of training and validation loss curves (GIoU, classification, and L1), accompanied by only minor oscillations, indicates stable convergence and effective generalization under the variability inherent in field-acquired images. These patterns suggest that the model successfully adapts to diverse illumination and background conditions without overfitting.

The gradual decline in recall at higher confidence thresholds reflects the model strategy of retaining low-confidence crack detections to preserve sensitivity. This behavior underscores a deliberate trade-off between precision and recall, ensuring that subtle or fragmented cracks are not prematurely excluded during inference. Per-class AP (sleeper vs crack) will be reported in future work; the present results summarize combined metrics across both classes to reflect end-to-end operational screening performance.

YOLOv11 is suited for onboard, real-time screening because of its high throughput and quick inference capacity, which ensures efficient detection during continuous track monitoring. On the other hand, RT-DETR is more appropriate for offline evaluation, since it can fully utilize its improved sensitivity to subtle and fragmented cracks without strict time limitations. Implementing a two-stage hybrid pipeline, leveraging YOLOv11 for the initial screening and mapping, followed by RT-DETR for comprehensive verification of flagged segments, offers an ideal compromise between performance and detection completeness under actual operating conditions.

4. Conclusions

This study developed and validated a two-stage deep learning framework for railway sleeper inspection using field-acquired imagery from the Arezzo–LFI corridor, where longitudinal cracking was identified as a dominant degradation mode influenced by combined construction-related factors, service loading in track conditions, and environmental/maintenance effects. A curated dataset of 289 real field images (preserving natural lighting variation, ballast background, shadows, and surface weathering) was used to evaluate both image-level classification and object-level defect detection, targeting practical deployment for track safety and asset management.

For image classification, transfer learning with VGG architectures proved most suitable for defect-oriented sleeper imagery: VGG19 (baseline head) achieved the best performance (accuracy = 98.2%), followed by VGG16 (95%), while ResNet50 showed lower performance (88%); among VGG19 variants, performance depended strongly on the head design, with the GAP+BN configuration exhibiting reduced accuracy (81%), confirming that preserving fine spatial–texture cues is critical for sleeper condition discrimination.

For object detection, the models exhibited complementary strengths. YOLOv11 achieved mAP@0.5 = 0.65 and mAP@0.5-0.95 = 0.45 at 90 fps, supporting real-time screening and sleeper localisation in continuous monitoring. RT-DETR achieved mAP@0.5 = 0.56 and mAP@0.5-0.95 = 0.42 at 35 fps, indicating improved context-aware detection under shadows and occlusions at lower throughput. Accordingly, a two-stage deployment of YOLOv11 for onboard screening followed by RT-DETR for secondary verification of flagged segments offers a practical balance between throughput and detection completeness under field conditions.

From an engineering perspective, the results indicate that overall accuracy alone is insufficient to characterise inspection reliability for safety-critical defects such as cracks; instead, class-wise behaviour and operational objectives must govern model selection. Accordingly, YOLOv11 is well suited for real-time screening and routine monitoring, while RT-DETR is better positioned for secondary inspection and validation of subtle or partially occluded cracks, supporting condition trending and preventive maintenance decisions. Overall, the proposed framework provides a practical pathway for AI-enabled sleeper inspection in field environments and forms a basis for scalable integration into railway maintenance workflows and digital inspection pipelines.