A Hybrid Game Engine—Generative AI Framework for Overcoming Data Scarcity in Open-Pit Crack Detection

1. Introduction

Open-pit mines are among the most hazardous industrial environments due to the risks posed by geotechnical events such as slope failures and landslides [1,2,3]. The devastating consequences of these incidents are exemplified by recent disasters such as the 2023 Xinjing coal mine landslide in China, which resulted in 53 fatalities and approximately USD 28 million in economic losses [4], and the 2020 Hpakant jade mine landslide in Myanmar, which claimed nearly 200 lives and severely impacted local communities [5]. These events can be triggered by several factors, including weak geological structures [6], intense or prolonged rainfall [7,8], seismic activity, and vibrations from excavation and blasting [9]. Early warning systems are therefore designed to monitor slope displacement and detect hazards such as surface cracks [10,11,12], enabling safety interventions such as exclusion zones to protect both workers and assets [13]. Despite advances in technologies such as stability radar, visual inspection remains central to hazard identification across many open-pit mines in Australia [14,15,16]. However, this manual practice is both labor-intensive and subjective, exposing workers to hazardous environments and compromising safety [17,18,19].

Consequently, recent studies have focused on developing automated approaches that leverage technologies such as AI to reduce dependence on manual monitoring while enhancing operational safety and efficiency [20]. Specifically, advances in DL [21] have driven substantial progress in CV tasks such as object detection, image classification, and image segmentation [22], thereby enabling machines to derive meaningful information from real-world visual data [23]. To that end, neural network architectures such as convolutional neural networks (CNNs) [24] and vision transformers (ViTs) [25] have been widely utilized across diverse domains [26,27,28,29,30,31,32,33,34,35], underscoring their potential for geotechnical risk management. In particular, recent studies have demonstrated the use of CNN-based CV models such as YOLOv8 [36], YOLOv10 [37], Mask R-CNN [38], U-Net [39,40], and ENet [41] for automated surface crack detection in open-pit mining. While these works demonstrate the feasibility of DL models for geotechnical hazard identification, their effectiveness remains constrained by the limited availability of labelled crack images.

This issue, commonly referred to as data scarcity, is a ubiquitous problem in the field of DL [42], where domain generalization, or the capacity of a CV model to recognize objects in unseen settings or environments [43], is driven by the volume and representativeness of the data used for training [44]. Data scarcity is especially pronounced in industrial domains such as mining [45], where datasets are inherently commercially sensitive and limited by the significant cost and expertise required for data collection and annotation [46,47,48]. To address this challenge, techniques such as transfer learning [49] and data augmentation [50] have been widely adopted in the literature, particularly in healthcare and other data-constrained domains. Transfer learning reduces reliance on large datasets but is vulnerable to source-target domain mismatch [51,52], while data augmentation, though capable of artificially increasing dataset size [53], is prone to amplifying existing distributional biases [54,55]. As both methods remain fundamentally bounded by the quality of the original training data, interest continues to grow in techniques capable of generating entirely new and diverse training data at scale [56].

One established approach is the use of game engines, where platforms such as Unreal Engine (UE) [57] and Unity [58] are adapted to render synthetic images for training CV models. These programs enable controlled variation of scene parameters and environmental conditions, as well as automated dataset generation and ground-truth annotation [59]. However, their diversity is inherently bounded by the manual effort required for scene and asset development, imposing practical limitations on dataset scale and variation. Generative models address this limitation by learning directly from existing real-world distributions to produce high-fidelity outputs at scale. Among these, generative adversarial networks (GANs) [60] such as StyleGAN2-ADA are highly effective at emulating realistic visual patterns [61], providing controllable latent-space manipulation and adaptive augmentation that mitigates discriminator overfitting for small datasets [62]. Although modern diffusion models can also produce visually rich and highly stylized imagery, their outputs are more dependent on prompt tuning and exhibit less fine-grained, deterministic control compared to GANs, making them less suited to domains requiring strict structural realism and reproducibility [63]. Despite these technological advances, the generation of synthetic images remains constrained by a number of limitations. The most impactful of these issues is the reality gap, a concept which refers to the perceptual and distributional disparity between real and synthetic data, a factor which often limits the generalizability of CV models trained on images from game engines [64]. Moreover, generative models typically require substantial training data, impacting their effectiveness in domains where real-world datasets are limited [65]. While techniques such as domain randomization [66,67] have been explored as a means of mitigating the reality gap [68], no prior study has systematically evaluated if game engines can be leveraged to train generative models for realistic and scalable image synthesis as a mitigant for data scarcity.

To address this gap, we develop and evaluate a hybrid game engine—generative AI framework that produces synthetic images to offset data scarcity and enhance the generalizability of CV models. The main contributions of this work are summarized as follows:

• We develop a scalable hybrid framework integrating UE5 and StyleGAN2-ADA to generate realistic synthetic images with automated annotations for training CV models.
• We demonstrate that synthetic images of surface cracks generated through our pipeline achieve enhanced fidelity and diversity compared to game engine data alone, validated quantitatively through Fréchet Inception Distance (FID) and Learned Perceptual Image Patch Similarity (LPIPS).
• We evaluate the downstream effectiveness of images generated through our pipeline by training the real-time object detection model YOLOv11, achieving substantial performance improvements relative to models trained solely on game engine data.

To the best of our knowledge, this study presents the first systematic evaluation of a hybrid game engine—generative AI framework for data synthesis utilizing UE5 and StyleGAN2-ADA. By combining the geometric and structural realism of game engines with the scalable diversity and textural domain adaptation of generative modelling, our framework demonstrates improved generalization performance in data-scarce CV tasks. Applied to surface crack detection in open-pit mining, the framework enhances the accuracy of object detection models in identifying slope failure precursors, supporting improved safety outcomes while demonstrating a methodology with potential transferability to other data-constrained domains.

2. Related Works

2.1. Synthetic Dataset Generation Using Game Engines

Advances in game engine technology have pushed the boundaries of visual realism and provided a promising means of addressing data scarcity in CV. Unlike manual data collection, which can be costly and time-consuming, game engines enable synthetic data generation with automated ground-truth annotation and fine-grained control of scene parameters and environmental conditions [69]. For instance, Half-Life 2’s Source engine [70], renowned for its detailed and lifelike animation and physics technology, was first utilized nearly two decades ago to develop and test an autonomous surveillance system [71]. Since then, researchers have increasingly leveraged synthetic data generated by photorealistic game engines to train and validate CV models for tasks such as object detection and image segmentation [72]. Table 1 summarizes recent works which apply game engine synthetic data across diverse applications, from construction safety monitoring to autonomous vehicle detection. These studies demonstrate performance improvements ranging from modest gains of less than 1% [73] to substantial enhancements exceeding 65% [74], highlighting the variable influence of game engine data on model generalizability. The most common development platforms used are UE4 and Unity due to their accessibility and integration with CV plugins such as NDDS [75] and Unity Perception Package [76], which support domain randomization techniques and automatically generate ground-truth labels such as bounding boxes and segmentation masks for captured data. An alternative approach is the use of commercial games, such as Grand Theft Auto V (GTA V) [77], where a combination of in-engine tools and community mods are adapted for synthetic data capture and annotation [78]. Collectively, these works demonstrate the versatility of game engines for synthetic data generation, producing datasets ranging from approximately 1 500 [79] to over one million images [78]. Studies utilizing fewer than 3 000 synthesized images for training of downstream CV models reported limited performance improvements or underfitting [73,79], suggesting insufficient data diversity to support robust model generalization. In contrast, approaches which synthesized larger training datasets demonstrated substantial gains [74,80,81], highlighting the impact of dataset size on downstream performance.

These findings indicate that while game engines enable scalable data generation, meaningful performance benefits require datasets of sufficient scale and diversity to capture the contextual variability necessary for effective transferability.

In addition to dataset size, the realism of rendered images remains a key determinant of downstream performance. Several studies have demonstrated that models trained on data from high-fidelity game engines tend to generalize better than those trained on less photorealistic platforms [83]. For instance, [80] reported that the enhanced texture detail, lighting, and reflections of images generated in UE5 led to a 17.3% increase in mAP relative to a model trained on data produced in UE4, highlighting the influence of visual fidelity on learned feature distributions for CV models. Similarly, commercial games, developed with significant budgets and advanced rendering pipelines, typically produce more realistic images than those generated by less robust engines such as Unity (see Figure 1) [78]. As a result, models trained on lower fidelity datasets often struggle to generalize in real-world applications because their limited realism widens the reality gap [83].

To address this issue, several studies employ transfer learning by pre-training using synthetic data and fine-tuning with small quantities of real-world data, thereby better aligning the synthetic and real feature domains. This mixed dataset approach, demonstrated in [84], resulted in a 5% improvement in the PCK for pose estimation relative to a model trained solely on synthetic data. Furthermore, [81] reported a 79.5% improvement in mAP when fine-tuning a model trained on 400 000 synthetic images with only 760 real images (a 526:1 synthetic-to-real ratio), underscoring the effectiveness of limited real-world data in enhancing model generalizability. Complementary to this, domain randomization techniques have proven effective in bolstering model robustness across unseen conditions. For example, [85] demonstrated a 22.8% increase in mAP through successive randomization of parameters such camera positioning and object location, while [73,79] and [81] leveraged NDDS and Unity Perception Package to improve cross-domain generalization and downstream performance.

While these mitigation strategies demonstrate measurable downstream performance benefits, the reality gap and diversity constraints remain a central limitation of game engine synthetic data, impacting the transferability of CV models to real-world settings. Consequently, recent studies have explored generative modelling for producing realistic synthetic images, an approach examined in the following section.

2.2. Synthetic Dataset Generation Using Generative Models

Introduced in 2014, GANs [60] employ an adversarial training framework in which two neural networks compete in a minimax game to generate realistic synthetic images. Unlike game engines, which rely on manual scene construction, GANs learn underlying data distributions directly from training samples, enabling them to produce sharper and more lifelike outputs than prior generative methods such as variational autoencoders (VAEs) [87]. Since their inception, various GAN architectures have been proposed for image synthesis, with StyleGAN2-ADA emerging as a particularly effective approach due to its incorporation of adaptive discriminator augmentation (ADA), which mitigates overfitting when training on limited datasets. As shown in Table 2, StyleGAN2-ADA has been successfully applied across diverse domains spanning medical diagnostics, environmental monitoring, and infrastructure inspection, demonstrating the versatility and effectiveness of generative modelling for enhancing downstream performance across a range of CV tasks.

For generative models, the FID metric has been widely adopted to quantify the fidelity of generated samples relative to real data, with lower scores typically indicating closer alignment to real-world statistical distributions [88]. For example, [89] demonstrated high quantitative fidelity using StyleGAN2-ADA to synthesize images of skin lesions for melanoma classification, achieving an FID score of 0.79. Similarly, studies such as [90,91,92] produced anatomically realistic magnetic resonance imaging (MRI) scans, with FID values ranging from 18.14 to 67.53. Despite this, recent studies have scrutinized the reliability of FID due to inherent limitations such as biased estimations and incorrect distributional assumptions [93,94]. Notably, dental radiographs synthesized in [95] were rated indistinguishable from real scans by domain experts, yet they achieved an FID of 72.76, the highest recorded in Table 2. Conversely, despite obtaining an FID score of 20.90, chest X-rays produced in [96] contained artefacts that hindered downstream classification performance. These findings highlight that FID may not necessarily reflect the practical utility of synthetic images for downstream tasks, and that quantitative metrics alone may be insufficient to assess generation quality.

Dataset size is also closely tied to the effectiveness of generative models, with insufficient data often leading to discriminator overfitting and training instability [62]. For instance, [97] found that over 10 000 training images were required to synthesize realistic petrographic samples. Similarly, [89] reported improved image quality when training with datasets exceeding 30 000 images, demonstrating the correlation between dataset size and synthetic image fidelity. Conversely, studies utilizing fewer than 2 000 images [90,95] generally produced lower-quality outputs with inconsistent results, underscoring the significance of adequate training data. Despite this general trend, synthesis quality can also be influenced by domain complexity, independent of dataset size. For example, [98] generated realistic pavement crack images using only 778 training samples, while [91] achieved high-fidelity abdominal MRI scan synthesis with 1 300 images. In contrast, [99] obtained an FID of 67.47 when training on 770 landslide images, likely due to the visual diversity inherent to environmental images. These findings suggest that while larger datasets generally enhance synthesis quality, the amount of data required to achieve high-fidelity generation varies considerably across domains depending on visual complexity.

To address these data requirements, recent studies have explored transfer learning strategies whereby pre-trained models are fine-tuned using small amounts of real-world data from the target domain [100]. For instance, [92] demonstrated that pre-training StyleGAN2-ADA on unrelated source domains such as FFHQ [101] improves synthesis quality for brain tumor MRI scans. Similarly, [95] reported that transfer learning not only improved FID scores of synthesized dental radiographs but also enhanced the accessibility of generative modelling for researchers with limited access to computational resources such as GPUs. Collectively, these studies indicate that transfer learning offers a promising yet largely unexplored avenue for addressing data scarcity in generative modelling [92,95,100,102].

While transfer learning may partially alleviate the challenges of data scarcity, generative models fundamentally require substantial quantities of training data to learn meaningful distributions. Even StyleGAN2-ADA, designed specifically for data-constrained scenarios, typically requires thousands of images to produce high-fidelity outputs [62], presenting a significant barrier in domains where data acquisition is limited by cost or accessibility, thereby motivating the hybrid framework for synthetic data generation introduced in the following section.

3. Materials and Methods

This study proposes a hybrid synthetic dataset generation framework that integrates game engine rendering with generative modelling to address data scarcity in open-pit crack detection. Leveraging UE5 and StyleGAN2-ADA, the framework synthesizes diverse and realistic images of surface cracks that are automatically annotated using Grounding DINO [104] to train the YOLOv11 real-time object detection model. The methodology comprises three primary stages, as outlined in Figure 2.

The first stage (Section 3.1) focuses on the development of a parameterized UE5 environment. A virtual scene representing an open-pit mine wall is first constructed, followed by the development of an automated dataset generation algorithm for domain randomization and data acquisition. By systematically varying crack meshes, ground material textures, and lighting parameters, this algorithm generates a dataset of 20 000 labelled images of open-pit surface cracks. In the second stage (Section 3.2), StyleGAN2-ADA is trained on the UE5 dataset to overcome the diversity ceiling of parametric rendering, generating crack images with structural variations not present in the original UE5 dataset. Three initialization strategies are evaluated to generate 20 000 images per configuration to identify the influence of transfer learning. Each generated image is automatically annotated using Grounding DINO, a vision language model (VLM) capable of detecting objects from a text prompt and generating bounding boxes for downstream CV model training. In the final stage (Section 3.3), a dataset-level ablation study is conducted to assess how each dataset configuration influences downstream YOLOv11 crack detection performance. Model accuracy is assessed on 200 real-world mining images across key performance metrics such as AP, precision, recall, and F1 score, to quantify the generalization capability of object detection models trained solely on game engine data and those trained on synthetic data generated by the proposed framework.

Together, these three stages form a unified framework that synthesizes training data through game engine rendering, expands dataset scale and diversity through generative modelling, and trains generalizable real-time object detection models, thereby enabling autonomous hazard identification to improve operational safety in data-scarce open-pit mining while minimizing manual data collection and annotation effort.

3.1. Synthetic Dataset Generation Using UE5

We adopt UE5 for dataset synthesis due to its state of the art (SOTA) physically based rendering (PBR) pipeline with support for virtualized geometry [105] and real-time global illumination (RTGI) [106]. These features enable the generation of high-fidelity terrain surfaces and illumination effects, thereby reducing the impacts of the reality gap and allowing for more robust downstream generalization [80]. We first construct a configurable virtual open-pit environment (Section 3.1.1) wherein surface cracks are rendered and photographed under systematically randomized conditions. Parameters such as surface appearance and texture (Section 3.1.2), crack morphology (Section 3.1.3), illumination (Section 3.1.4), and camera viewpoint (Section 3.1.5) are independently varied to enhance dataset diversity. High-resolution images (Section 3.1.6) and corresponding bounding box annotations (Section 3.1.7) are automatically generated using a dataset generation pipeline (Section 3.1.8) prior to downstream generative modelling using StyleGAN2-ADA (Section 3.2). A high-level overview of the complete workflow is shown in Figure 3.

3.1.1. Virtual Environment Construction

A virtualized section of an open-pit wall is constructed using the Landscape tool [107] in UE5, as illustrated in Figure 3. Designed to support photorealistic visual rendering rather than explicit geotechnical modelling, the generated heightmap incorporates representative slope surfaces and surrounding context largely consistent with operational open-pit settings, without attempting to reproduce site-specific stratigraphy or failure mechanics. The primary objective of this 3D scene is to provide a visually realistic background surface against which surface cracks are rendered, and for this reason, emphasis is placed on surface texture, color variation, and roughness, an approach consistent with domain randomization methodologies commonly adopted in synthetic data generation to enhance model generalization across unseen settings [108,109].

3.1.2. Terrain Material Parameterization

To effectively reflect the visual diversity typical of open-pit mine sites, the generated landscape is parameterized using 12 distinct terrain surface materials selected to span the range of surface conditions typically documented across Australian mining operations [110,111]. These materials represent commonly observed surface types including soils, sandy deposits, compacted gravel, weathered rock formations, ironstone, and mixed debris zones, with color variations spanning red, ochre, grey, and brown tones, as demonstrated in Figure 3. Each high-resolution material, sourced from Quixel Megascans [112], incorporates PBR properties including albedo, normal, and roughness, as well as Nanite [105] displacement maps to support realistic light interaction. By randomly varying surface appearance while maintaining fixed terrain geometry during dataset generation, controlled background variability is introduced to improve model robustness under domain shift, allowing for enhanced generalizability across different operational environments.

3.1.3. Surface Crack Decal Parameterization

Surface crack decals are generated from real-world crack imagery to preserve authentic morphological characteristics. Crack images acquired from site surveys are first manually processed to extract binary crack masks. These masks are then converted into grayscale opacity maps with a resolution of 2048 × 2048 pixels prior to additional processing to generate auxiliary texture maps such as normal, height, and roughness for enhanced decal realism. These textures are then imported into UE5 and assembled into deferred decal materials, forming the unique cracks demonstrated in Figure 3. A total of 22 crack decal variants are generated, representing common patterns such as single cracks, bifurcated cracks, and crossed cracks [113]. During instantiation in UE5, decals are scaled anisotropically to preserve field-measured aspect ratios, ensuring geometric consistency with real-world crack morphology.

3.1.4. Lighting Positioning and Intensity Parameterization

Lighting parameters are independently randomized per Figure 3 to replicate the diverse conditions encountered during site inspections. The directional light component, representing solar illumination, is configured with continuously varying position and intensity. Azimuth angle θ spans full 360° rotation along the horizontal plane, while elevation angle ψ varies between 30° and 90° to represent solar position changes from early morning to overhead midday conditions. Light intensity I ranges from 5 to 10 lux with a fixed color temperature of 5000 K to represent daylight white balance. This systematic randomization of solar position and intensity produces natural changes in shadow direction and surface contrast, helping prevent overfitting and improving model robustness to real-world lighting variability.

3.1.5. Camera Viewpoint Parameterization

Virtual camera positioning employs randomized spherical coordinate parameterization relative to the center of the instantiated crack, thereby ensuring coverage of geotechnically relevant inspection viewpoints. The standoff distance d is sampled between 1 and 10 m to represent the typical range for both ground-based and unmanned aerial vehicle (UAV) inspection [114]. The azimuth angle ϕ provides 360° rotation for complete directional coverage around the crack, eliminating bias towards specific viewing directions, while the elevation angle α varies between 45° and 90° to span oblique to nadir perspectives. Slight camera jitter ρ between -10° and 10° across pitch and roll axes simulates natural handheld tilt and UAV attitude changes, whereas field of view (FOV) variations from 70° to 110° represent the typical range of smartphone and digital single-lens reflex (DSLR) cameras. Focus distance is automatically set to match the standoff distance, ensuring consistent sharpness across viewpoints. Additional intrinsic camera parameters for the Cine Camera Actor component [115] are configured as summarized in Table 3.

3.1.6. Synthetic Image Rendering

For each randomized scene instance, a 1920 × 1080 resolution image is rendered using the High Resolution Screenshot Tool (HRSST) [116]. Global illumination is enabled via Lumen [106] in hardware ray tracing (RT) mode with high-quality settings to achieve photorealistic lighting behavior. Temporal anti-aliasing (TAA) is used to suppress spatial aliasing, while post-processing effects such as auto-exposure, chromatic aberration, vignetting, film grain, and lens distortion are disabled to maintain consistent image quality across captures.

3.1.7. Bounding Box Computation

As illustrated in Figure 3, each crack decal actor is approximated by a 3D box with center p_c and half-extents e_x, e_y, and e_z, with corresponding corner points defined as:

$$p_i=p_c+\left({\pm e}_x, {\pm e}_y, {\pm e}_z\right), i=1,...,8.$$

(1)

To transform these 3D world coordinates into 2D image space for bounding box computation, each corner point is projected to pixel space using the camera projection operator П(·):

$$\left(u_i,v_i\right)=\mathsf{\Pi }\left(p_i\right),$$

(2)

where u_i and v_i represent the horizontal and vertical pixel coordinates of the i-th projected corner clamped to the viewport bounds [0, W] × [0, H], with W = 1920 and H = 1080.

The operator П(·) performs perspective division and maps world coordinates to pixel coordinates using a 4 × 4 reversed-Z perspective projection matrix P:

$$\mathrm{P} = \left[\begin{array}{cccc}{\displaystyle \frac{1}{\tan \left(\alpha /2\right)}}& 0& 0& 0\\ 0& {\displaystyle \frac{W}{H\cdot \tan \left(\alpha /2\right)}}& 0& 0\\ 0& 0& {\displaystyle \frac{n}{n-f}}& 1\\ 0& 0& -{\displaystyle \frac{f\cdot n}{n-f}}& 0\end{array}\right],$$

(3)

where α represents the vertical FOV half-angle (the angle from the center of the lens to the edge of the viewable area), and n and f correspond to the near and far clipping planes, respectively [117].

From the set of eight projected corner points {(u_i, v_i)}, the enclosed 2D bounding box coordinates are computed as:

$$u_{min}=\underset{i}{\min }{u}_i,u_{max}=\underset{i}{\max }{u}_i,v_{min}=\underset{i}{\min }{v}_i,v_{max}=\underset{i}{\max }{v}_i.$$

(4)

Pixel-space center coordinates and bounding box dimensions w and h are then calculated as:

$$\begin{gathered} {x_c}^{px}={\displaystyle \frac{u_{min}+u_{max}}{2}},{y_c}^{px}={\displaystyle \frac{v_{min}+v_{max}}{2}}, \\ {w}^{px}=u_{max}-u_{min},h^{px}=v_{max}-v_{min}. \end{gathered}$$

(5)

These quantities are then normalized with respect to the viewport dimensions W and H and exported as a label file in the YOLO annotation format [118]:

$$x_c={\displaystyle \frac{{x_c}^{px}}{W}},y_c={\displaystyle \frac{{y_c}^{px}}{H}},w={\displaystyle \frac{w^{px}}{W}},h={\displaystyle \frac{h^{px}}{H}}.$$

(6)

3.1.8. Automated Dataset Generation Pipeline

Automated dataset generation is implemented through the UE5 Blueprint script presented in Algorithm 1, which iteratively randomizes crack decals, terrain materials, lighting conditions, and camera geometry for each rendered image using the distributions defined in Section 3.1.2, Section 3.1.3, Section 3.1.4 and Section 3.1.5. For each rendered image, the corresponding annotation is computed as described in Section 3.1.7 to produce a total of 20 000 labelled synthetic surface crack images. Dataset generation is performed using an NVIDIA GeForce RTX 5090 GPU, 64 GB RAM, and Intel Core Ultra 9 285H CPU in UE5 version 5.6.1.

Algorithm 1. Automated Synthetic Dataset Generation Pipeline for UE5

3.2. Synthetic Dataset Diversity Enhancement Using StyleGAN2-ADA

Using the synthetic images generated in UE5, we train StyleGAN2-ADA (Section 3.2.1) to increase the diversity of our dataset by generating realistic images of surface cracks with structural variations not captured in the UE5 dataset. To investigate the effect of different initialization strategies, three transfer learning configurations (Section 3.2.2) are evaluated with respect to generation fidelity, quantified using FID (Section 3.2.3), and generation diversity, measured using LPIPS (Section 3.2.4). The resulting 20 000 images generated for each configuration are then automatically annotated with bounding boxes using Grounding DINO and filtered using confidence thresholding and visual inspection (Section 3.2.5).

3.2.1. StyleGAN2-ADA Architecture Overview

We adopt the StyleGAN2-ADA architecture, which is capable of generating high-fidelity synthetic images even under data-scarce conditions. The architecture consists of a generator, which includes mapping and synthesis networks that produce synthetic images, and a discriminator, which evaluates the realism of generated samples, as illustrated in Figure 4. The mapping network contains a multilayer perceptron (MLP) with eight fully connected (FC) layers which transform an input vector in latent space z ∈ Z into an intermediate latent code w ∈ W. The synthesis network then produces images through a hierarchy of style-modulated convolution blocks with learned weight demodulation spanning multiple resolution scales. Per-layer affine transforms A from w produce the style parameters that control the convolutions performed in each of these style blocks, thereby controlling visual attributes and image characteristics, while injected stochastic noise B provides finer, unstructured detail. The discriminator employs a multi-stage residual architecture, downsampling full-resolution inputs and passing the final feature map through a FC layer to produce a single scalar output D(x) representing the probability that an input image is real or generated. To maintain training stability under limited data conditions, ADA dynamically applies random geometric and color-space perturbations to discriminator inputs to prevent overfitting while preserving generator output diversity.

3.2.2. StyleGAN2-ADA Training Configuration

As discussed throughout Section 2.2, generative models typically require tens of thousands of training images to produce high-fidelity outputs [62,103], with techniques such as transfer learning potentially enhancing their generalizability in data-constrained settings [92,95,100]. Pre-training leverages visual priors learned from a source domain to accelerate convergence and improve generation quality in a target domain. To examine the impact of transfer learning on the synthesis of surface crack images using StyleGAN2-ADA, three training configurations are developed to represent distinct points along a domain similarity spectrum.

The first configuration trains StyleGAN2-ADA solely on game engine data, serving as a baseline for quantifying the generation quality achievable without external knowledge transfer. Both the generator and discriminator are initialized with random weights and trained for 2 000 kimg on the UE5 dataset.

The second configuration leverages Flickr-Faces-HQ (FFHQ) [101], pre-trained on 70 000 high-resolution images of human faces, to examine the impact of transfer learning from a semantically unrelated source domain. This configuration is motivated by prior research demonstrating that low-level visual features, such as edge and texture primitives, exhibit strong transferability across semantically different domains [92]. The pre-trained model is fine-tuned using the UE5 dataset for 2 000 kimg.

The third configuration utilizes weights pre-trained on the Describable Textures Dataset (DTD) [119], comprising 5 640 images across 47 texture categories at 1024 × 1024 resolution [120]. Unlike the FFHQ dataset, DTD contains, amongst other categories, explicit representations of cracked, fractured, and rough surface patterns, providing strong low-level visual correspondence between source and target domains. Fine-tuned using the UE5 dataset for 2 000 kimg, this configuration enables assessment of the impact of explicit texture-focused pre-training on the generation of surface crack images.

All training configurations use the official NVIDIA StyleGAN3 repository [121] with StyleGAN2-ADA architecture configuration due to improved compatibility with current versions of PyTorch. Training is conducted on an NVIDIA Tesla A100 GPU with CUDA 12.8, PyTorch 2.2.0, and Python 3.10. Training images are downsampled from 1920 × 1080 to 512 × 512 resolution to balance spatial detail with computational efficiency and training stability. A batch size of 16 is selected to ensure gradient stability within GPU memory constraints. As crack morphology remains invariant under horizontal reflection, mirror augmentation is enabled to increase training data diversity and improve generalization. Additional training hyperparameters follow the StyleGAN2-ADA default values [122] summarized in Table 4.

3.2.3. Generation Fidelity Evaluation

We employ the FID [123] to assess generation fidelity, a metric which measures the distributional similarity between real and generated images in the feature space of a pre-trained Inception-v3 network [124]. This metric quantifies the distance between the feature distributions of real and generated images as follows:

$$d^2\left(\left(\mathit{m}, \mathit{C}\right),\left({\mathit{m}}_w,{\mathit{C}}_w\right)\right)=‖\mathit{m}-{{\mathit{m}}_w‖}_2^2+{\mathrm{T}}_{\mathrm{r}}\left(\mathit{C}+{\mathit{C}}_w-2{\left(\mathit{C}{\mathit{C}}_w\right)}^{1/2}\right),$$

(7)

where (m, C) and (m_w, C_w) denote the mean vectors and covariance matrices of the real and generated image features extracted from the Inception-v3 activation space respectively, and T_r represents the matrix trace. Lower FID scores generally indicate a greater similarity between the Gaussian approximations of real and generated feature distributions p and p_w, thereby indicating higher generation fidelity.

To monitor generation quality throughout training, FID scores are computed on selected checkpoints (every 200 kimg) spanning the training duration by synthesizing 50 000 random samples and comparing their feature distributions against 50 000 images sampled with replacement from the training set. This temporal tracking facilitates analysis of convergence behavior, typically marked by FID score stabilization. Upon convergence and completion of training, the best checkpoint for each of the three configurations is selected based on the lowest FID score achieved using full latent space sampling to ensure unbiased evaluation. Finally, 20 000 surface crack images are generated using the best checkpoint for each training configuration through random sampling of the latent space.

3.2.4. Generation Diversity Evaluation

We evaluate generation diversity by utilizing LPIPS [125], a metric which measures perceptual distance aligning closely with human visual judgement by quantifying how far apart two images are in the feature space of a pre-trained network. As shown in Equation 8, LPIPS computes this perceptual distance by measuring the squared l₂ difference between the feature activations y^l and y^l₀ of two images across multiple layers of a network F, in our case, Visual Geometry Group-16 (VGG-16) [126]. These feature differences are unit-normalized and scaled by vector w^l prior to being averaged spatially and summed channel-wise, producing a scalar output d that correlates strongly with human judgements of visual similarity. Lower scores, typically close to 0, indicate greater perceptual similarity, whereas higher scores suggest that image pairs look more different and diverse to humans.

$$d\left(x,x_0\right)={\sum }_l{\displaystyle \frac{1}{H_l{W}_l}}{\sum }_{h,w}{‖{\mathit{w}}^{\mathit{l}}\odot \left({{\widehat{y}}^l}_{hw}-{{\widehat{y}}^l}_{0hw}\right)‖}_2^2.$$

(8)

To assess whether StyleGAN2-ADA has generated images of surface crack structures with greater diversity than those present in the original UE5 training dataset, we conduct an intra-set diversity comparison by analyzing both mean LPIPS and clustering in the VGG-16 feature space. To do so, we randomly sample 5 000 pairs of images from each of our synthesized datasets and calculate the distance between each pair. By comparing their statistical distributions, namely the mean and median LPIPS, we are able to determine whether StyleGAN2-ADA has produced datasets with enhanced diversity. We also extract the feature vectors from our datasets using the VGG-16 backbone and use t-distributed Stochastic Neighbor Embedding (t-SNE) to visualize the image embeddings. The number and separation of clusters in t-SNE space provides qualitative insight into distributional coverage, where a greater number of separated clusters suggests broader perceptual variability and therefore, increased dataset diversity.

3.2.5. Automated Annotating Using Grounding DINO

Manual annotation in CV has long been a time-consuming and resource-intensive task. To address this challenge, we employ pseudo-labelling using Grounding DINO [104], a unified VLM designed for open-set object detection due to its ability to identify objects it has never explicitly seen during training using language-guided text prompts. This zero-shot inference approach allows the model to generate labels for previously unseen data, such as the images synthesized by StyleGAN2-ADA. Grounding DINO adopts a transformer-based encoder-decoder architecture to process visual features extracted from images and fuse them with prompt information via cross-attention. The image backbone, a pre-trained ViT, such as Swin Transformer [127], is used to extract multi-scale visual features from input images, while a BERT-based [128] text prompt encoder converts prompt tokens into semantic embeddings that guide the detection process. The model integrates these representations within a feature enhancer module, producing language-conditioned object queries that are passed to the decoder, whose outputs are directly projected into bounding-box coordinates and text-region alignment scores through prediction layers. We utilize Grounding DINO in a zero-shot manner with the text prompt “crack” to generate a set of bounding boxes with confidence scores for each of our StyleGAN2-ADA images. Low-confidence predictions with scores below 0.7 are automatically removed via confidence thresholding and the remaining annotations are manually inspected prior to being converted to YOLO format, yielding a large-scale pseudo-labelled dataset suitable for downstream object detection model training. An overview of this automated dataset annotation pipeline is shown in Figure 5.

3.3. Crack Detection Using YOLOv11

3.3.1. YOLOv11 Architecture Overview

We utilize YOLOv11 [129] to evaluate the downstream effectiveness of the datasets synthesized by the proposed framework. This one-stage CNN-based real-time object detection model improves upon prior iterations through architectural refinements that enhance multi-scale feature extraction, reduce computational overhead, and increase overall robustness, making it well-suited to real-world operational contexts. As illustrated in Figure 6, YOLOv11 follows a modular design comprising three principal components: a backbone, a neck, and a detection head. The backbone consists of stacked CBS and C3K2 blocks that progressively reduce the spatial resolution of input images while increasing channel depth, enabling efficient extraction of features such as crack width and curvature. It also incorporates the spatial pyramid pooling-fast (SPPF) block to expand the receptive field and combine multi-scale contextual information, and C2PSA attention modules to more effectively capture both local and global features, thereby improving detection accuracy across objects of varying scales. The neck facilitates multi-scale feature fusion through a series of upsampling, downsampling, and concatenation operations that refine information from different stages of the backbone, enhancing robustness to scale variation. Finally, the detection head employs decoupled classification and regression branches, enabling more precise object localization and bounding box prediction by processing the fused features transmitted from the neck.

3.3.2. YOLOv11 Training Configuration

To determine whether the proposed framework provides measurable benefits for surface crack detection, we conduct a dataset-level ablation study using YOLOv11. Four dataset variants are evaluated: (i) UE5 only (UE5O), representing the baseline; (ii) UE5O augmented with StyleGAN2-ADA (SG2), used to assess whether the proposed framework improves detection performance beyond UE5O; (iii) SG2 with FFHQ pre-training (SG2-FFHQ); and (iv) SG2 with DTD pre-training (SG2-DTD).

These configurations, summarized in Table 5, are designed to evaluate the impact that generative refinement of synthetic data has on downstream object detection performance. Training YOLOv11 on each dataset separately also provides a controllable basis for isolating the effects of StyleGAN2-ADA initialization strategies on real-world generalization performance. Training is conducted using the official YOLOv11 implementation on an NVIDIA Tesla A100 GPU with CUDA 12.8, PyTorch 2.2.0, and Python 3.10. The medium model variant (YOLOv11m) is selected to balance detection performance and computational efficiency.

COCO pre-trained weights are used for the UE5O configuration, while SG2-adapted models are fine-tuned from the best-performing UE5O weights. All remaining training hyperparameters are reported in Table 6.

3.3.3. Object Detection Performance Evaluation

Model performance is evaluated using a held-out test set of 200 real-world mining images (see Figure 7) collected from various open-pit mining operations across Australia using UAVs and handheld cameras. The test set exhibits considerable variability in lighting conditions, camera viewpoint, and environmental context, enabling a rigorous assessment of model generalization beyond synthetic data distributions.

Model performance is quantified using precision, recall, F1 score, and AP evaluated at different IoU thresholds. In this study, a predicted bounding box is considered a true positive (TP) if its overlap with a ground-truth bounding box exceeds an IoU threshold of 0.5. Predictions below this threshold are classified as false positives (FP), while ground-truth objects without matching predictions constitute false negatives (FN). From these quantities we compute precision (P), representing the proportion of predicted detections that correctly identify actual surface cracks:

$$P = {\displaystyle \frac{TP}{TP+FP}}$$

(9)

We also calculate recall (R), indicating the proportion of actual surface cracks successfully detected by the model:

$$R = {\displaystyle \frac{TP}{TP+FN}}$$

(10)

We leverage the F1 score as a balanced indicator of detection performance, providing a single measure that reflects model performance with respect to both consistent and accurate crack identification:

$$F1 = 2\times {\displaystyle \frac{P\times R}{P+R}}$$

(11)

Finally, the AP metric, corresponding to the area under the precision-recall curve, is used to provide a consolidated measure of detection quality by integrating precision and recall across all confidence thresholds:

$$AP = {\int }_0^{1}P\left(R\right) dR$$

(12)

We report AP@0.5 (IoU $\ge$ 0.5) and AP@[0.5:0.95] to quantify both detection accuracy and spatial extent localization, two attributes essential for effective surface crack monitoring.

4. Results and Discussion

This section evaluates the effectiveness of the proposed hybrid synthetic dataset generation framework in addressing data scarcity for open-pit surface crack detection. We first examine the convergence characteristics of the StyleGAN2-ADA training configurations, followed by quantitative and qualitative assessments of generation fidelity and diversity (Section 4.1). We then analyze real-world crack detection performance through a dataset-level ablation study using YOLOv11, discussing the impact of different generative model initialization strategies on downstream model robustness, generalizability, and transferability to real-world conditions (Section 4.2). Finally, we reflect on the broader implications of synthetic data for operational deployment of CV models in data-scarce domains, evaluating the practical utility of the proposed framework (Section 4.3). Together, these analyses provide a comprehensive assessment of whether the proposed framework meaningfully improves real-world object detection performance under limited-data conditions.

4.1. StyleGAN2-ADA Training and Image Generation Assessment

4.1.1. Training Dynamics

The training behavior of the StyleGAN2-ADA configurations used for image synthesis in this study are evidenced in Figure 8. As highlighted in Figure 8a, the baseline configuration (SG2) exhibits higher generator loss at initialization compared to both pre-trained models (SG2 + FFHQ and SG2 + DTD). This behavior is expected when training from scratch, as the generator initially produces unstructured noise, allowing the discriminator to easily classify its outputs as fake samples with a high degree of confidence. SG2 generator loss rapidly decreases within the first 100 kimg as the generator starts to form a coherent latent representation, then decays slowly until reaching convergence of 1.29 around 1 250 kimg. Both pre-trained models, conversely, achieved more rapid convergence with lower initial generator loss due to their already well-structured latent representations. SG2 + FFHQ ultimately achieves the lowest generator loss at 1.13 and converges the most smoothly, indicating that pre-trained weights from the FFHQ-1024 dataset may provide effective feature representations for surface crack image generation, despite originating from a semantically distant and unrelated domain. Domain-aligned pre-training by way of the SG2 + DTD configuration exhibits better initialization and faster convergence than both SG2 and SG2 + FFHQ; however, generator loss fluctuations throughout training suggest comparatively reduced stability relative to the other training configurations.

While generator loss highlights the impact of initialization and pre-training on generative capacity, discriminator loss, quantified in Figure 8b, provides complementary insight into the stability and balance of adversarial training dynamics. Discriminator loss across all three configurations converges rapidly within the first 200 kimg and remains tightly clustered for the remainder of training, indicating that the adversarial game stabilizes early and reaches a Nash equilibrium [130]. All three configurations exhibit an initial loss spike caused by the discriminator rapidly adapting to the highly unrealistic outputs synthesized by the generator; however, the discriminator then settles into a narrow range between 0.85 and 0.90 following this transient phase, with only minor fluctuations throughout the full 2 000 kimg training window. SG2 displays slightly lower discriminator loss than the pre-trained configurations during the early stages (<50 kimg), reflecting the confidence with which the discriminator classifies the random noise synthesized by the uninitialized generator as fake samples. As generator fidelity improves, the discriminator loss rises to match that of the pre-trained configurations, signaling convergence toward a stable equilibrium. Both pre-trained configurations reach this equilibrium more quickly, however, exhibiting nearly indistinguishable trajectories beyond the 100 kimg mark. Overall, the minimal separation between curves indicates that, unlike generator loss, discriminator loss is only weakly influenced by pre-training and instead reflects the balance of the adversarial process once both networks have stabilized, ultimately confirming that all three configurations maintained stable training dynamics without evidence of discriminator collapse.

Figure 8c shows the evolution of the augmentation probability applied by the ADA module to prevent discriminator overfitting for each of the three training configurations. This value typically increases when the discriminator exhibits overconfident predictions and decreases when its predictive reliability weakens, and as such, the trajectory of augmentation probability reflects the degree of overfitting pressure placed on the discriminator throughout training. The baseline configuration exhibits the lowest augmentation probability throughout the training run, rising gradually from near-zero to just 32%. This behavior indicates that the discriminator experiences comparatively weaker overfitting pressure when the generator is trained from scratch, likely because the generator initially produces low-quality outputs hindered by the limited diversity of the UE5 training dataset. In contrast, both SG2 + DTD and SG2 + FFHQ show markedly higher values of augmentation probability consistent with the discriminator facing higher quality generator outputs. SG2 + DTD rises steadily and plateaus around 50%, however, SG2 + FFHQ exhibits the highest augmentation probability of 60%, suggesting that the discriminator is consistently close to overfitting, likely due to the coherent and high-quality outputs of the FFHQ-initialized generator, forcing ADA to inject stronger regularizations to maintain adversarial balance.

Changes in generator fidelity throughout training are demonstrated in Figure 8d, which illustrates the FID score progression of the three StyleGAN2-ADA initialization strategies. Across all configurations, FID decreases sharply during the first 400 kimg as the network rapidly learns structural and textural characteristics from the training data. Both pre-trained configurations exhibit a steeper initial decline than the baseline configuration, reflecting the advantage conferred by transfer learning through the inheritance of low-level feature priors that improve early-stage generation fidelity. This acceleration in early synthesis quality is further supported by Figure 9, which shows that crack-like structures emerge within the first 30 kimg of training for both pre-trained configurations. In contrast, the baseline configuration requires nearly three times longer to form comparably coherent latent-space structure. Following the initial descent, all curves plateau and show only marginal improvements for the remainder of training, indicating that each configuration collectively reaches a point of diminishing returns relatively early at around 400 kimg. This behavior also confirms that all three configurations achieved stable training dynamics with no evidence of mode collapse, which is often signaled by late-stage FID fluctuations. SG2 + FFHQ ultimately achieves the lowest FID at convergence of 12.75, outperforming both SG2 + DTD (15.99) and SG2 (15.88), further demonstrating that FFHQ pre-training provides the most effective and stable transfer of feature representations for realistic surface crack image synthesis in this study.

These results collectively demonstrate that all three StyleGAN2-ADA configurations converged stably, with the pre-trained configurations, particularly SG2 + FFHQ, achieving faster learning and superior generation fidelity overall. While these training dynamics provide insight into model behavior during optimization, they do not quantify the realism or variability of the images produced by the final configurations. Accordingly, the following section evaluates the fidelity and diversity of the generated samples in greater detail.

4.1.2. Qualitative Evaluation of Generation Fidelity and Diversity

Table 7 provides a quantitative comparison of the final synthesis quality attained by each StyleGAN2-ADA training configuration, evaluated using the FID and LPIPS metrics described in Section 3.2.3 and Section 3.2.4. The high quantitative fidelity achieved by SG2 + FFHQ suggests that this configuration effectively generates high-quality surface crack images with greater realism than both SG2 and SG2 + DTD. Notably, all three configurations achieve comparatively excellent FID scores, surpassing many of the results seen in analogous studies such as those documented in Table 2, reinforcing the efficacy of the StyleGAN2-ADA training methodology developed for image synthesis in this study. Qualitative analysis (Section 4.1.3) of the images generated by the StyleGAN2-ADA training configurations is necessary to identify further detail variations encapsulated by the minute changes in FID documented in Table 7.

In addition to achieving the strongest FID score, SG2 + FFHQ also exhibits the highest mean (0.472 ± 0.090) and median (0.477) LPIPS, indicating that samples generated from this configuration contain greater perceptual diversity than the other StyleGAN2-ADA training configurations. The baseline configuration has a mean LPIPS of 0.452 ± 0.094, indicating moderate perceptual diversity with relatively symmetric distribution. SG2 + DTD achieves a slightly higher mean LPIPS at 0.457 ± 0.092, confirming that pre-training provides benefits for generative diversity. Interestingly, the lower bound (0.011) of the LPIPS range for SG2 indicates that the baseline configuration contains a small number of visually homogenous samples, however, the upper bound (0.725) suggests that it also contains a variety of meaningfully diverse surface crack images. Both pre-trained configurations exhibit higher minimum LPIPS values than SG2, suggesting that the structural priors provided by transfer learning enrich generation variation across all ranges. Compared to the UE5O dataset, SG2 achieves a 2.80% increase in perceptual diversity, reflecting the increased variability in crack morphology afforded by generative modelling. Both SG2 + FFHQ and SG2 + DTD provide more impactful improvements to image diversity with gains of 7.49% and 3.96% on UE5O, respectively.

Figure 10 visualizes the t-SNE embeddings of the UE5O dataset and the three StyleGAN2-ADA training configurations, offering further insight into the diversity of the generated samples beyond LPIPS. Figure 10a summarizes the relationship between all datasets, depicting the broader impact of generative modelling on generation diversity. UE5O forms a series of distinct, compact clusters to the left and uppermost regions of the plot, suggesting that images from this dataset fall into discrete categories with low continuous diversity. All StyleGAN2-ADA configurations, conversely, exhibit more continuous spread with heavy overlap, indicating that the generative models learn continuous manifolds that encompass the original training distribution and extend well beyond it. While all three training configurations exhibit much wider horizontal and vertical expansion than UE5O, the denser feature embeddings seen in Figure 10c confirm that FFHQ pre-training injects additional structural priors not seen in SG2 and SG2 + DTD. Furthermore, this enhanced distributional overlap demonstrates the capacity of effective pre-training to bridge the domain gap and create smoother transitions between the training dataset and synthesized images. Overall, these results confirm that images generated by StyleGAN2-ADA cover a significantly broader feature space than deterministic UE5 renderings, validating the LPIPS behavior documented in Table 7 and confirming that the proposed framework enhances the diversity of training data.

4.1.3. Qualitative Evaluation of Generation Fidelity and Diversity

To validate the trends indicated by both FID and LPIPS in the prior section, as well as explore sample-level generative behavior, we perform a qualitative evaluation on a set of representative images (examples shown in Figure 11) generated from the best checkpoints for each model.

The most observable difference between the baseline UE5O images and the StyleGAN2-ADA samples is the increased variation in crack morphology, including differences in thickness, curvature, and branching, as demonstrated in Figure 11. Specifically, Figure 11b,c reflects the progressive enhancement of sample diversity, with each model exhibiting additional crack propagation and illustrating new geometric structures not present in the UE5 training dataset. These visual patterns also closely mirror the quantitative results: SG2, which achieved the weakest FID and LPIPS scores, exhibits reduced structural coherence and increased texture smoothing relative to UE5O, with fewer atypical crack-like structures than the other SG2 configurations. SG2 + DTD improves sample diversity; however, generated images suffer from semantic noise, resulting in the formation of more stochastic crack patterns. SG2 + FFHQ, consistent with its highest fidelity and diversity scores, produces the most convincing samples, with sharper crack boundaries, coherent shading, and stable surface geometry. Although the UE5O baseline remains the most photorealistic, its limited morphological variation is clearly visible, aligning with the diversity constraints reflected in the LPIPS comparison. Overall, Figure 11 provides clear visual confirmation of the fidelity and diversity hierarchy quantified in Table 7, with FFHQ pre-training offering the strongest generative advantages.

4.2. YOLOv11 Training and Crack Detection Performance Evaluation

4.2.1. Training Dynamics

Figure 12 summarizes the YOLOv11 training behavior of the four dataset configurations developed in this study. As shown by the validation box loss curves in Figure 12a–d, all models converged stably with no evidence of overfitting or divergence. UE5O, trained solely on game engine data, exhibited a more gradual convergence across 300 epochs, reaching a final validation box loss of 0.17. In contrast, the three SG2-adapted variants stabilized within the first 20 epochs at approximately 0.30, with training terminating at 100 epochs due to early stopping. This accelerated convergence reflects the effect of fine tuning, as the SG2-adapted models begin from weights already optimized on the UE5O dataset. All configurations achieved strong validation performance across AP@0.5, AP@[0.5:0.95], and precision, indicating that YOLOv11 learns the UE5 domain almost perfectly. This also suggests that the synthetic validation set may lack sufficient complexity to meaningfully differentiate the SG2-adapted models from the UE5O baseline.

4.2.2. Real-World Performance Evaluation

Table 8 documents the object detection performance of the four YOLOv11 dataset configurations evaluated on a held-out test set of 200 real-world open-pit surface crack images. The UE5O baseline achieved the weakest performance overall, with an AP@0.5 of 0.403 and an AP@[0.5:0.95] of 0.223. In addition, the comparatively low precision, recall, and F1 score of this configuration highlights a clear domain gap which suggests that purely synthetic training data lacks sufficient appearance similarity to generalize effectively to real-world mining imagery. This limitation causes the model to miss more complex detections or incorrectly identify the background material as a surface crack, as illustrated in Figure 13

The precision-recall characteristics shown in Figure 14(a) further reinforce this behavior, with UE5O exhibiting a rapid monotonic degradation in precision as recall increases, indicating limited robustness to confidence threshold variation and a tendency to trade FPs for marginal gains in recall. This behavior also suggests that the real-world visual characteristics of the test set largely fall outside of the feature space of the synthetic UE5O dataset, resulting in the model struggling to distinguish between crack-like structures and background textures, leading to a disproportionately high number of FPs.

This is particularly evident in Figure 13(d), where background debris is misclassified as a surface crack leading to a drop in precision. As discussed throughout Section 2.1, this behavior is consistent with prior literature, particularly for a complex domain like open-pit surface crack detection and further reinforces the need for data augmentation or domain adaptation to improve model generalizability.

All SG2-adapted variants, in contrast, demonstrate substantially improved detection performance across all reported metrics, suggesting that GAN-based refinement introduces beneficial noise and structural diversity which benefits downstream CV models. As shown in Table 8, these configurations achieved markedly higher AP, precision, recall, and F1 scores relative to the UE5O baseline, confirming that the images generated by the proposed framework provide a more effective training distribution for real-world open-pit surface crack detection. UE5O + SG2-FFHQ attained the highest precision of 0.808, indicating strong suppression of FPs, while UE5O + SG2 achieved the highest recall of 0.902, correctly identifying the largest proportion of surface cracks. Although UE5O + SG2-DTD exhibited slightly weaker performance than the other SG2 variants, it still delivered an improvement of 0.354 in F1 score over the UE5O baseline, highlighting the benefits afforded by GAN-augmented data on model generalizability.

Despite having a slightly weaker AP@0.5 than UE5O + SG2, UE5O + SG2-FFHQ provided superior localization performance across a broader range of IoU thresholds, with an AP@[0.5:0.95] of 0.722. This behavior is consistent with the generative metrics reported in Table 7, where the FFHQ-initialized StyleGAN2-ADA model achieved the highest quantitative fidelity. This suggests that the improved perceptual quality and structural coherence captured by the FFHQ pre-trained model (see Section 4.1.3) may translate into more realistic crack morphology and boundary definition, improving bounding box stability at stricter IoU thresholds and enhancing model robustness. Additionally, the improvement in perceptual diversity provided by FFHQ pre-training over SG2 baseline (see Section 4.1.2) likely enables UE5O + SG2-FFHQ to generalize more effectively to more atypical crack structures present in the real-world test set, resulting in more accurate predictions with tighter localized boundaries.

Interestingly, UE5O + SG2 achieved better recall than UE5O + SG2-FFHQ, suggesting that the baseline synthetic distribution, while exhibiting lower perceptual diversity and quantitative fidelity, may contain more pronounced crack features that allow the detector to maintain higher sensitivity, albeit at the expense of localization precision. In contrast, the comparatively weaker performance of UE5O + SG2-DTD suggests that texture-focused initialization likely emphasizes less relevant features, where, despite exhibiting competitive FID and LPIPS scores, generated samples appear to lack sufficient structural coherence for the detector to learn stable crack boundaries, leading to reduced accuracy overall. These initialization-dependent effects are additionally demonstrated by the precision-recall behavior in Figure 14(b-d), where all GAN-adapted variants consistently maintain high precision with increasing levels of recall, indicating improved robustness to confidence threshold variation relative to UE5O. For instance, UE5O + SG2 maintained near perfect precision until reaching a recall of approximately 0.70, while both UE5O + SG2-FFHQ and UE5O + SG2-DTD exhibited similarly strong performance with slightly earlier onset of precision decay at higher levels of recall.

Figure 15 further clarifies how StyleGAN2-ADA initialization influences detection confidence and localization behavior by providing a comparison of representative test detections for the four YOLOv11 dataset configurations. As shown in Figure 15(a-d), UE5O + SG2 generally produces the highest confidence predictions but occasionally includes additional background material within predicted bounding boxes. This suggests that the detector exhibits a tendency toward maintaining objectness confidence across threshold variations, increasing bounding box spatial coverage to maintain sufficient overlap with ground truth annotations. This qualitative behavior is consistent with the strong recall and AP@0.5 seen for UE5O + SG2 in Table 8, as well as the consistently high precision maintained with increasing levels of recall in Figure 14(b), and works to explain the slightly weaker AP@[0.5:0.95] that results from reduced localization accuracy at stricter IoU thresholds.

In contrast, UE5O + SG2-FFHQ provides more selective detections with improved spatial alignment relative to ground truth annotations, as seen by the tighter bounding boxes in Figure 15(b-c). While this behavior can result in lower confidence predictions, the increased localization accuracy leads to stronger performance at higher IoU thresholds, resulting in a stronger AP@[0.5:0.95], as seen in Table 8. UE5O + SG2-DTD attains comparatively weaker qualitative results, exhibiting reduced robustness which manifests as missed detections for lower-contrast or partially occluded cracks, such as in Figure 15(c), and FPs with overlapping predictions for background material, as observed in Figure 15(d). This behavior impacts both precision and AP, resulting in the early precision drops in Figure 14(d), and weaker overall model performance relative to the other SG2-adapted variants.

To further examine this behavior, Table 9 reports the distribution of FP and FN counts for each YOLOv11 dataset configuration evaluated on the real-world test set, providing a breakdown of the underlying causes for the aforementioned quantitative and qualitative performance trends. Consistent with its weaker overall performance, UE5O exhibits a comparatively high number of FPs and FNs, further quantifying the sample-level behavior seen in Figure 13. This directly contributes to the precision-recall characteristics observed in Figure 14(a), where the high FP count causes rapid precision degradation with increasing recall, with the curve terminating early due to a large number of FNs.

This domain gap is effectively closed by generative modelling, with all SG2-adapted variants demonstrating substantially reduced error rates relative to the UE5O baseline. UE5O + SG2 records the lowest FP count overall, achieving a balance between FPs and FNs consistent with its strong recall and AP@0.5 performance. The slightly weaker AP@[0.5:0.95] observed for UE5O + SG2 is further explained by Table 9, where localization imprecision causes otherwise correct detections to fail to match the ground truth annotation at stricter IoU thresholds, thereby causing an increase in the FN count. Although UE5O + SG2-FFHQ exhibits a moderately higher FP count than UE5O + SG2, the substantial reduction in missed detections at stricter IoU thresholds achieved by this configuration drives its superior localization robustness. This behavior aligns with the increased precision and AP@[0.5:0.95] seen in Table 8, manifesting in the more spatially accurate detections shown in Figure 15(b-c). UE5O + SG2-DTD demonstrates the weakest performance amongst the SG2-adapted variants, exhibiting elevated FP and FN counts relative to both UE5O + SG2 and UE5O + SG2-FFHQ consistent with the earlier precision degradation seen in Figure 14(d). As shown qualitatively in Figure 15(d), the increase in FP count for this configuration can be attributed to frequent misclassifications where background debris or rubble are incorrectly identified as surface cracks. Conversely, Figure 15(b) highlights a missed detection, suggesting that lower contrast surface cracks may be responsible for the elevated FN count seen for UE5O + SG2-DTD in Table 9.

One persistent limitation seen across all configurations, regardless of initialization strategy, is the difficulty in detecting heavily occluded, fine-grained surface cracks. This behavior, highlighted in Figure 16, causes a missed detection which impacts overall recall performance, and is likely attributable in part to the inherent architectural constraints of CNN-based object detection models such as YOLOv11 [131]. A number of factors such as backbone downsampling, localization sensitivity, and receptive field size mismatch can result in the model ignoring or missing smaller features [132]. As a result, while more pronounced cracks are generally detected with high confidence, hairline cracks hidden within the background terrain remain challenging to localize consistently.

4.3. Practical Deployment Considerations

While the results presented throughout the preceding sections demonstrate that the proposed hybrid synthetic dataset generation framework significantly improves real-world open-pit surface crack detection performance and alleviates the issue of data scarcity, its practical utility depends not only on detection accuracy but also on its deployability in operational settings.

A significant practical advantage of the proposed framework is its ability to generate extensive datasets at scale with minimal human intervention. Unlike conventional CV-based monitoring, which necessitates costly and laborious field data collection and manual annotation [46,47,48], the images synthesized in this study are labelled automatically. For mining operations, this eliminates the need to deploy workers into the field solely for data collection purposes, reducing operational disruption and safety risks. Furthermore, this decoupling of dataset size from annotation effort enables rapid dataset expansion without a proportional increase in manual labelling time, enhancing operational efficiency. Beyond this, the nature of virtualized game engine environments allows for iterative refinement and adaptation to new contexts, enabling continuous improvement of training data as domain requirements evolve. This reduces the overall cost associated with the implementation of object detection models and enhances the accessibility of domain-specific training data.

One practical consideration is the upfront computational overhead associated with both UE5-based image synthesis and StyleGAN2-ADA-based generative refinement. In this study, full dataset generation of 20 000 UE5 images required approximately six hours on high-end consumer hardware (NVIDIA GeForce RTX 5090 GPU and 64 GB RAM). Similarly, the use of StyleGAN2-ADA incurred additional GPU costs, requiring nearly 12 hours for training and a further four hours for image synthesis and pseudo-labelling per configuration on an NVIDIA Tesla A100 GPU, amounting to a total of 64 hours of GPU runtime. As such, the overall framework is best viewed as an upfront investment to generate large-scale datasets for data-scarce domains, rather than as a lightweight data augmentation technique.

Ultimately, the primary practical outcome of the proposed framework is improved downstream model robustness to real-world variability, particularly in data-scarce environments. As demonstrated in Section 4.2.2, the combination of game engine data and GAN-based generative refinement enables CV models to generalize more effectively across diverse viewpoints, crack morphologies, and background conditions, all without requiring access to real-world training datasets. Improvements in metrics essential for automated inspection workflows, such as recall and AP@[0.5:0.95], indicate that the framework trains consistent and reliable object detection models that can enhance downstream analysis and decision support. More broadly, these findings demonstrate that the absence of large-scale real-world datasets need not restrain the effective application of CV models in data-scarce domains. By carefully constructing domain-specific virtualized environments within a game engine and subsequently enriching their appearance diversity through generative modeling, it is possible to produce training data that generalize effectively to real-world conditions. In this context, the proposed hybrid synthetic dataset generation framework provides a practical pathway for addressing data scarcity in domains constrained by privacy, intellectual property, and proprietary concerns. For mining operations in particular, where site-specific geotechnical data is often commercially sensitive and difficult to share across organizations, this capability offers a pathway to develop robust inspection systems without compromising data security.

5. Conclusions and Future Work

Autonomous surface crack detection in open-pit mining offers numerous benefits such as enhanced worker safety and improved operational efficiency. However, CV models require large amounts of representative training data to generalize effectively to unseen conditions, impacting their applicability in commercial domains constrained by safety, cost, and data confidentiality considerations. To address this challenge, this study presented a hybrid game engine—generative AI framework for dataset synthesis and evaluated its effectiveness for surface crack detection in real-world open-pit mining imagery. The proposed approach combined the realism of the UE5 game engine with the scalability of StyleGAN2-ADA, enabling the synthesis of large-scale, fully labelled surface crack datasets that significantly improve the generalizability of CV models without reliance on extensive field data collection or manual annotation.

Comprehensive evaluation on a held-out real-world test set demonstrated that object detection models trained on images generated by the proposed framework substantially outperformed those trained solely on synthetic data from UE5. In particular, AP@0.5 increased from 0.403 to 0.922 for the best-performing GAN-adapted configuration, while AP@[0.5:0.95] exhibited approximately a threefold improvement across the board, indicating significantly enhanced localization robustness and bounding box accuracy. These performance gains were accompanied by higher recall and reduced missed detections, confirming that the proposed framework effectively narrows the domain gap between synthetic and real-world imagery through the increased diversity and realism of its generated samples. From a practical perspective, this work highlights the viability of synthetic data in autonomous inspection workflows, reducing dependence on manual labeling while mitigating operational, safety, and confidentiality constraints associated with real-world data collection. More broadly, it demonstrates that synthetic data can generalize effectively to real-world conditions when underpinned by an appropriate generation framework, suggesting that the proposed approach may be extended to other data-constrained domains where large-scale labelled datasets are similarly difficult to obtain.

Future work will iterate on this research in several ways. Firstly, diffusion-based generative models will be explored as an alternative to the proposed framework to examine whether their enhanced synthesis fidelity provides meaningful downstream benefits over the controllability of game engine-based rendering. Secondly, the detection pipeline will be extended toward multi-scale learning for improved object detection at varying distances, and instance or semantic segmentation to enable more precise delineation of surface crack boundaries for downstream analysis such as propagation measurement. Additionally, to target the limitation of small object detection identified in the study, future work will investigate architectural enhancements to the object detection and segmentation models to further improve sensitivity to fine-grained cracks. Finally, framework integration with edge-based inference platforms and UAV-based data acquisition will be examined to support real-time autonomous inspection workflows across diverse open-pit mining environments.