Clinical Validation of an AI-Assisted Ultrasound Workflow for Developmental Dysplasia of the Hip Screening: Sequential Assessment According to the Graf Method

1. Introduction

Developmental dysplasia of the hip (DDH) comprises a spectrum of abnormal acetabular development and femoral-head instability, ranging from mild dysplasia to fixed dislocation. Its reported frequency varies according to population, diagnostic criteria, patient age, and screening policy, but DDH remains one of the most common musculoskeletal disorders of infancy. If not detected, it can cause progressive deformity, gait disturbance, pain, and premature osteoarthritis, while a delayed presentation is more likely to require complex treatment. By contrast, detection in early infancy allows treatment to begin while the hip retains substantial remodeling potential and is generally amenable to less invasive management. Clinical examination and assessment of risk factors remain essential, while ultrasonography provides direct, radiation-free visualization of the cartilaginous hip and is the preferred imaging examination during early infancy [1,2,3], even in complex skeletal anatomies [4]. The clinical value of ultrasound screening, however, depends on the acquisition and interpretation of diagnostically adequate images.

Among the available sonographic techniques, the Graf method provides a standardized morphological and quantitative framework and is widely used, particularly in Europe [5,6]. Graf assessment follows a prescribed sequence rather than relying on angle measurement alone. Checklist I requires identification of eight anatomical structures: the chondro-osseous border, femoral head, synovial fold, joint capsule, labrum, cartilage, bony roof, and bony rim (turning point). Checklist II subsequently establishes image usability by confirming the lower limb of the ilium, the correct imaging plane, and the labrum. Only after both checklists have been satisfied should the $\alpha$ and $\beta$ angles be measured and the hip assigned a Graf type [5,6]. This sequence provides an intrinsic quality-control mechanism by preventing technically inadequate images from proceeding to quantitative interpretation. Nevertheless, both image acquisition and interpretation remain operator-dependent. Probe orientation, incomplete visualization of anatomical structures, and incorrect placement of reference lines can alter angle measurements and final classification. Interobserver variability therefore persists, with the $\beta$ angle generally being less reproducible than the $\alpha$ angle and agreement on the final Graf classification frequently being only moderate [7,8].

Artificial intelligence (AI) has been applied to scan-adequacy assessment, standard-plane recognition, anatomical segmentation, landmark detection, automated angle measurement, and DDH or Graf-type classification [9,10,11,12]. Image quality has a material effect on automated interpretation, supporting the need to identify inadequate images before diagnostic outputs are generated [13]. AI-supported handheld-ultrasound workflows have also demonstrated the feasibility of extending DDH screening beyond specialist imaging settings [14]. More recently, the QualityDDH framework operationalized Checklist I and Checklist II for image standardization and was prospectively evaluated with readers of different experience levels [15]. Despite these advances, the evidence remains heterogeneous and many studies address isolated components of the diagnostic process. Recent evidence syntheses found high reported diagnostic performance but also identified limited clinically relevant validation, infrequent external validation, and a predominance of retrospective single-centre studies [16,17]. Evidence therefore remains limited for a clinically evaluated sequential workflow that combines formal Checklist I and Checklist II quality control with angle estimation, Graf classification, and binary DDH identification while also quantifying its effects on clinician accuracy and assessment time.

The aims of this study were: (1) to determine the technical performance of each component of the complete workflow using a clinically acquired infant hip ultrasound dataset; (2) to quantify interobserver variability among three physicians and establish a consensus reference standard for clinical validation; and (3) to determine, in a paired reader study involving 15 clinicians, whether computer-aided diagnosis improves Checklist I accuracy and reduces assessment time, and whether the magnitude of any improvement is associated with career level (resident versus consultant), years of experience, or monthly hip-ultrasound volume.

2. Materials and Methods

2.1. Study Design, Dataset, and Reference Standard

This retrospective single-center study evaluated a sequential AI-assisted Graf workflow and a paired CAD reader study. Hip ultrasonograms were obtained at Regina Margherita Children’s Hospital, Turin, Italy, between March 2021 and February 2026 from infants aged 0–6 months referred for reassessment of suspected or previously identified DDH. The local ethics committee approved the study (Città della Salute e della Scienza, Turin, Italy; Study Protocol: DB2021; Approval code: 1489/2021; Approval date: 14/11/2021) and waived informed consent for retrospective analysis of anonymized images. Examinations used an InnoSight system with an L12-4 linear-array transducer (Philips; 4–12 MHz).

The initial dataset comprised 1846 images from 679 examinations. After excluding 43 Graf type IV images, 1803 remained: 1332 Checklist I-compliant and 471 non-compliant. Examination-level partitioning produced training, validation, and held-out test sets of 1082, 360, and 361 images. A random 1024-image subset with clinical $\alpha$/$\beta$ measurements and Graf labels was used to evaluate the complete workflow. A separate temporal set of 100 images acquired after January 2026 (70 compliant and 30 non-compliant) was reserved for the reader study and excluded from model development.

Pixelwise masks for background and the eight Checklist I structures were created in CVAT [6,18]. All images were independently assessed by a pediatric radiologist with 15 years of experience and two pediatric orthopaedic surgeons with 16 and 5 years of experience. Disagreements in angle measurements or Graf classification were jointly reviewed to establish consensus. OCR-extracted measurements and labels were manually verified against the source images. Full eligibility, annotation, preprocessing, and implementation details are provided in Supplementary Methods S1–S3.

2.2. AI-Assisted Graf Workflow

After removal of acquisition overlays, images were denoised, contrast-enhanced, normalized, and resized to $256\times 256$ pixels. ResTransUNet combined a residual encoder, multiscale dilated convolutions, bottleneck self-attention, attention-gated skip connections, and a residual decoder to segment the eight Checklist I structures. Class-specific morphological filtering was applied; an image was accepted only when every structure exceeded its validation-derived minimum-area threshold. A conventional U-Net trained on the same partitions served as the baseline. The Checklist I workflow is summarized in Figure 1.

Checklist II was assessed only after Checklist I acceptance and required the lower iliac limb, standard acetabular-roof section, and labrum [5,6]. A baseline was fitted to the segmented iliac contour; the prespecified inclination tolerance was $\pm 5^{\circ }$. Representative decisions are shown in Figure 2. For images passing both checklists, the bony-roof line was fitted through the turning point and bony roof, and the cartilaginous-roof line through the turning point and labral centroid. Automated $\alpha$/$\beta$ angles, Graf type, and a study-defined binary endpoint were then generated (Figure 3). Types Ia/Ib were coded as no DDH and IIa/IIb, IIc, D, and III as DDH. Detailed thresholds, training settings, and classification rules are reported in Supplementary Methods S2–S3.

2.3. Paired Reader Study

Fifteen clinicians (nine consultants/specialists and six residents; 12 in orthopaedics and three in pediatric radiology) classified the same 100 independent images as Checklist I compliant or non-compliant. Microsoft Forms was used for an unaided session and, four weeks later, a CAD-assisted session with independently re-randomized image order. CAD displayed the original image, segmentation overlay, complete/incomplete output, and missing structures. Readers received standardized instructions but no training cases. Accuracy was agreement with the independent reference; completion time was calculated from session timestamps.

2.4. Outcomes and Statistical Analysis

Segmentation was evaluated using Dice, IoU, and pixel accuracy; Checklist I using accuracy, precision, recall, and F1-score; Checklist II using these metrics plus specificity and Matthews correlation coefficient; and diagnostic classification using exact Graf agreement and binary accuracy, sensitivity, specificity, precision, and F1-score. Angle agreement was summarized using mean absolute error, automated-minus-clinical bias, Bland–Altman limits, and ICC(3,1); clinician reliability used ICC(2,1) [19,20].

Paired reader accuracy was compared using a two-sided paired t-test with 95% CI and Cohen’s $d_z$; completion time used the Wilcoxon signed-rank test. Career-level gain used Welch’s t-test, monthly-volume gain the Kruskal–Wallis test, and associations with experience Spearman correlations with Holm correction. Subgroup analyses were exploratory. Analyses used Python 3.12.13, pandas 2.2.3, NumPy 2.3.5, and SciPy 1.17.0; $p<0.05$ was significant. No a priori sample-size calculation was performed. Additional statistical definitions are provided in Supplementary Methods S4.

3. Results

3.1. Checklist I Segmentation and Reportability

On the held-out test set, ResTransUNet achieved a mean Dice coefficient of 0.759, mean intersection over union (IoU) of 0.641, and pixel accuracy of 97.38%. For Checklist I classification, accuracy was 95.61%, precision 96.65%, recall 97.74%, and F1-score 97.19% (Table 1). The U-Net baseline yielded slightly higher mean Dice (0.766) and IoU (0.654), but lower classification accuracy (91.67%), precision (96.47%), recall (92.66%), and F1-score (94.52%). Thus, ResTransUNet improved Checklist I accuracy, recall, and F1-score by 3.94, 5.08, and 2.67 percentage points, respectively. At the class level, the femoral head had the highest overlap (Dice, 0.902; IoU, 0.845), whereas the synovial fold had the lowest (Dice, 0.654; IoU, 0.527).

Although the U-Net baseline produced comparable mean overlap values, its Checklist I accuracy and recall were lower than those of ResTransUNet (Table 1). Errors were concentrated mainly at boundaries of thin or low-contrast structures.

3.2. Sequential Workflow Performance

Of the 1024 images entering the sequential workflow, 18 were rejected at Checklist I and 160 of the remaining 1006 at Checklist II; 846 images therefore proceeded to angle measurement and classification (Table 2). Checklist II classification achieved 87.18% accuracy. Among the 846 images completing the workflow, the $\alpha$-angle error was smaller than the $\beta$-angle error, with mean absolute errors of $3.{59}^{\circ }$ and $11.{91}^{\circ }$, respectively.

Exact Graf-type agreement was 40.07% (339/846; Table 3). Disagreement was driven principally by differentiation between types Ia and Ib, which share the same binary no-DDH designation. Consequently, binary DDH classification performed better than exact Graf typing, with 81.09% accuracy, 82.67% sensitivity, and 80.59% specificity.

Interobserver reliability was higher for the $\alpha$ angle [ICC(2,1), 0.872; 95% CI, 0.804–0.916] than for the $\beta$ angle [ICC(2,1), 0.722; 95% CI, 0.558–0.823]. Pairwise MAEs ranged from $3.{26}^{\circ }$ to $3.{52}^{\circ }$ for $\alpha$ and from $4.{84}^{\circ }$ to $6.{58}^{\circ }$ for $\beta$.

3.3. Paired Clinical Reader Study

Fifteen clinicians completed both sessions. Mean accuracy increased from 72.3% without CAD to 90.2% with CAD, corresponding to a mean paired improvement of 17.9 percentage points (95% CI, 10.3–25.4; $t\left(14\right)=5.08$, $p<0.001$; Cohen’s $d_z=1.31$; Table 4). The paired differences did not depart significantly from normality (Shapiro–Wilk $W=0.925$, $p=0.231$). All 15 clinicians improved, producing 268 additional correct classifications across 1500 reader–image assessments. The stand-alone CAD model correctly classified 93 of the 100 reader-study images.

Residents improved more than consultants/specialists (between-group difference, 15.8 percentage points; 95% CI, 0.9–30.7; Welch $t=2.43$, $p=0.041$). Accuracy gain did not differ among monthly-volume categories [$H\left(2\right)=0.53$, $p=0.767$; Table 5]. Completion time decreased numerically from a median of 13.1 to 9.0 minutes, but the paired difference was not statistically significant ($p=0.177$).

Years of experience showed a positive unadjusted association with unaided accuracy, but this did not remain significant after Holm correction ($\rho =0.522$, adjusted $p=0.138$). Experience was not significantly associated with CAD-assisted accuracy or accuracy gain after correction.

4. Discussion

This study evaluated an AI-assisted workflow designed to reproduce the sequential methodology described by Graf [5,6]. Within this methodology, image validity is not a secondary technical consideration: the required anatomical structures must first be visible and the standard plane must then be confirmed before the $\alpha$ and $\beta$ angles can be measured and a Graf type assigned. The workflow performed best at this initial gate. Checklist I accuracy and recall were 95.61% and 97.74%, respectively, and CAD increased mean clinician accuracy from 72.3% to 90.2%. Performance declined at subsequent stages, particularly for $\beta$-angle estimation and exact Graf classification. The findings therefore support the system principally as clinician-facing decision support for verifying whether an image is suitable for Graf assessment, rather than as an autonomous diagnostic pipeline.

These results should be interpreted within a rapidly expanding but heterogeneous literature. A recent systematic review identified 15 ultrasound studies comprising 8315 images and reported AUC values of 0.90–0.99, sensitivities of 86.54–100%, and specificities of 62.5–100%; however, the certainty of evidence was rated low because of methodological heterogeneity, small samples in some studies, suspected publication bias, and limited external validation [21]. Individual studies illustrate both the promise and the difficulty of cross-study comparison. Gong et al. reported 85.89% accuracy for direct DDH classification, whereas Atalar et al. achieved 93% accuracy and an AUC of 0.99 when distinguishing normal, dysplastic, and incorrectly positioned images [22,23]. Kinugasa et al. reported perfect test-set classification metrics using transfer learning, while other systems have targeted standard-plane recognition, angle measurement, or Graf typing with high internal performance [10,11,12,14,15,24]. These studies establish technical feasibility, but high performance on a final classification task does not by itself demonstrate that every input image satisfied the preceding Graf requirements. The present study instead evaluated the consequences of enforcing those requirements sequentially, showing that strong performance at one stage did not guarantee comparable performance across the complete workflow.

Previous work on acquisition adequacy supports this validity-first interpretation. Quader et al. obtained an AUC of 0.985 for distinguishing adequate from inadequate scans before extracting dysplasia metrics [9]. Lee et al. required visible key anatomical points and iliac inclination within $\pm 5^{\circ }$; only 320 of 921 images (34.7%) were considered appropriate by both the model and the human observer, demonstrating how substantially the analyzable dataset can contract when validity criteria are applied [25]. Using images from multiple ultrasound systems, Li et al. reported 95.4% accuracy and an AUC of 0.982 for standard-plane recognition, and Xu et al. reported 96.70% standard-plane accuracy with a Cohen’s $\kappa$ of 0.925 against senior physicians [26,27]. Atalar et al.’s deliberate inclusion of 365 images acquired with incorrect probe positioning likewise showed the importance of training systems to reject, rather than diagnose from, unsuitable inputs [23]. Together, these studies indicate that automated hip-ultrasound screening should explicitly represent invalid and non-standard images instead of assuming that all submitted images are ready for measurement.

The segmentation findings further emphasize the distinction between technical overlap and clinical adequacy. The femoral head achieved the greatest overlap, whereas errors were concentrated at thin, low-contrast structures such as the synovial fold. Although the conventional U-Net produced slightly higher aggregate Dice and IoU values, ResTransUNet performed better for the clinically relevant Checklist I decision. A small omitted region may have little effect on mean overlap yet cause a required structure to fall below its area threshold and render the image unsuitable for Graf classification. This discrepancy supports evaluating segmentation according to its intended clinical endpoint and agrees with evidence that scan quality materially affects automated interpretation [13]. Approaches that jointly learn landmarks and structures may help preserve these clinically important relationships: Hu et al., for example, combined both tasks and reported mean $\alpha$- and $\beta$-angle errors of $2.{221}^{\circ }$ and $2.{899}^{\circ }$, respectively [28]. Nevertheless, the relevant question for a validity gate is not only how closely a mask overlaps a reference annotation, but whether every structure required for reliable measurement is demonstrably present.

Published measurement results also clarify where the present downstream pipeline requires improvement. In a dataset of more than 300,000 images, Oelen et al. reported an $\alpha$-angle root-mean-square error of $3.9^{\circ }$ for landmark-based automation versus $7.1^{\circ }$ for physicians in routine practice [29]. Hu et al. reported errors below $5^{\circ }$ for 93% of $\alpha$ and 85% of $\beta$ estimates, while Huang et al. reported ICCs of 0.96 and 0.97 for the two angles in abnormal hips [11,28]. However, $\beta$ measurement has not been uniformly robust. Quader et al. found a mean automated–manual $\beta$ difference of $8.9^{\circ }$, and Chen et al. reported only 35.3% sensitivity for identifying $\beta >{77}^{\circ }$ despite high specificity [9,12]. In the present study, the $\beta$ angle had a mean absolute error of $11.{91}^{\circ }$, positive bias of $11.{07}^{\circ }$, and ICC of 0.561, consistent with the known difficulty and lower clinician reproducibility of the cartilaginous-roof line [7,8]. This error probably contributed to frequent Ia–Ib disagreement and an exact Graf agreement of only 40.07%, despite 81.09% accuracy for the study-defined binary DDH endpoint. Differences in cohorts, image-selection rules, reference standards, and reported metrics preclude direct ranking of these systems, but the comparison identifies $\beta$-landmark localization and exact category assignment as priorities for further development. Binary performance cannot replace exact Graf typing when category influences management, and the grouping of IIa/IIb as DDH in this study must not be interpreted as an autonomous treatment recommendation because type IIa may represent physiological immaturity depending on age.

The paired reader study supported the proposed clinician-facing role. All 15 clinicians improved with CAD, and the aided display allowed them to inspect the segmentation, completeness decision, and missing structures rather than receive only a binary recommendation. In a sweep-ultrasound study spanning poor-to-excellent image quality, Ghasseminia et al. found greater agreement among subspecialists ($\kappa =0.72$) than for the AI system ($\kappa =0.47$), and AI reliability deteriorated more than human reliability on the poorest-quality images [30]. This result reinforces the value of both explicit quality control and retained clinician oversight. Evidence from another orthopaedic imaging task similarly showed that clinicians assisted by AI achieved 97% accuracy, exceeding the 83% accuracy of the model alone [31]. The present reader study is concordant with that human–AI complementarity: the clinically relevant unit was the clinician supported by an inspectable output, not the stand-alone algorithm. Residents showed a larger gain than consultants or specialists, but this subgroup comparison involved only six residents and was exploratory. Completion time decreased numerically but not significantly. Although rapid automated measurement has been demonstrated elsewhere, including a reported processing time of 1.1 seconds with DDHnet, the present study supports improved Checklist I accuracy rather than improved efficiency [11].

Three-dimensional and multisource approaches suggest potential routes toward broader screening, while also illustrating the importance of rigorous validation. AI-assisted three-dimensional ultrasound identified Graf-compatible sections with good agreement in 216 hips [32]. In a two-center, multi-year study of 2492 hips, automated analysis of three-dimensional sweeps correctly identified 90% of dysplastic hips and achieved $\kappa =0.79$ against clinical diagnosis; however, the positive predictive value was 32% and agreement for the $\alpha$ angle was more modest, showing that large and diverse evaluations can reveal trade-offs not evident in restricted internal datasets [33]. Multiplanar or sweep acquisition may reduce dependence on a single manually selected frame by allowing automated selection of the most appropriate plane, but poor acquisition quality remains consequential. Future studies should therefore improve automated angle measurement and exact Graf classification only after confirming adherence to both Checklist I and Checklist II, and should validate the entire sequence prospectively across centers, devices, operators, and screening populations. They should also include Graf type IV hips and complex or atypical anatomies so that automated screening is tested across the full morphological spectrum.

Several limitations restrict the present study’s generalizability. It was retrospective and single-center, used one ultrasound platform and transducer, and included a referred population rather than a universal-screening cohort. Exclusion of type IV hips narrowed disease severity, and static-image analysis did not evaluate real-time acquisition or patient-level outcomes. Although examination-level partitioning reduced leakage, sequential gating produced different denominators across stages, and estimates of angle and classification performance were conditional on prior automated acceptance. Independent expert-group comparison can strengthen ground-truth validation in orthopaedic imaging AI [34]; nevertheless, the present clinician-derived annotations and consensus reference remained susceptible to uncertainty at poorly visualized boundaries.

The reader study was also limited by 15 clinicians, 100 images, and a test-set prevalence of non-compliance that may differ from clinical screening. All readers completed the unaided session first and reassessed the same images four weeks later; re-randomization reduced but did not eliminate recall, learning, and period effects, while the absence of a randomized crossover design limited causal inference. These constraints limit the conclusions to internal validation of Checklist I decision support under the tested conditions.

5. Conclusions

The AI-assisted workflow accurately identified Checklist I-compliant hip-ultrasound images, and CAD improved mean clinician accuracy by 17.9 percentage points, with all 15 clinicians improving. These findings support its use as a clinician-facing tool for verifying image validity before Graf measurement and classification. Future studies should focus on improving automated angle measurement and Graf classification only after confirming that each image satisfies both Checklist I and Checklist II. Development and validation cohorts should also include Graf type IV hips and complex or atypical hip anatomies to assess automated ultrasound screening across the full spectrum of neonatal hip morphology.