A Real-World Comparison of Three Deep-Learning Systems for Diabetic Retinopathy in Remote Australia

1. Introduction

Diabetic retinopathy (DR), a common diabetes complication, is a leading cause of vision loss [1,2] expected to affect 160 million people by 2045. [3] Timely detection of disease can prevent 98% of vision loss, hence routine annual or biennial DR screening is typically recommended. [4,5] However, half of First Nations Australians and one-quarter of other Australians did not meet screening recommendations in a national survey and screening rates were lower in remote areas [6] which have higher proportions of First Nations people and increasing diabetes prevalence. [7,8,9] A recent study reported a diabetes prevalence of 29% in remote First Nations Australians and this was >50% among those ≥45 years, [8] almost six times higher than the national prevalence. [7] Barriers to DR screening in these remote areas include a lower ophthalmology and optometry workforce, needing to travel significant distances to access eye care, [2] and in primary care, high staff turnover and lengthy waits for screening results. [10] Deep learning systems (DLSs), a type of artificial intelligence, require minimal staff training and in conjunction with automated cameras may overcome these barriers to provide equitable screening access in remote regions and prevent vision loss.

DLSs can detect DR with sufficient accuracy, have potential for cost-saving and may be of particular value in remote settings with limited health service access. [11,12] In Australia, using a universal DLS-based DR screening program was estimated to save AU$595.8 million over 40 years [11]. However, few DLSs for DR have been trained or validated on Indigenous peoples internationally. [13,14,15,16] Notably, Google Health’s Automated Retinal Disease Assessment (ARDA) [17] has outperformed retinal specialist grading in First Nations Australians. [13] ARDA has been Conformité Européene-marked, but Australian regulatory approval has not yet been sought. Thirona RetCAD [18] and EyRIS SELENA+ [19] have multiple regulatory approvals including Therapeutic Goods Administration approval in Australia, yet neither have been validated on First Nations people specifically. DLSs require validation in the population where they are to be used to ensure adequate performance, especially when comprised of minority populations not sufficiently represented in the training dataset for the DLS development. [20] Data distribution differences between training and validation datasets can potentially lead to a marked drop in DLS performance. [21,22] As such, DLS performance should be assessed in real-world settings, and for First Nations Peoples before implementation. No Australian study has directly compared the performance of multiple DLSs on the same population. A direct real-world comparison of multiple DLSs can be valuable to illustrate differences in performance so the most appropriate tool for that population can be implemented. Therefore, the aim of this study was to prospectively compare the performance of multiple DLSs, to detect referrable DR in a remote Western Australian real-world setting.

2. Materials and Methods

A mobile service delivered by Lions Outback Vision in consultation with First Nations communities, provided DR screening to people living in the Pilbara region of Western Australia between February and August 2024 with facilities previously described. [23] Those who attended DR screening, with any diabetes type, were eligible to participate. Diabetes status was based on patient self-report.

Participants had single-field colour fundus photographs centered on the macula captured using an automated table-top camera, Topcon Maestro 2 (Topcon Healthcare, Tokyo, Japan), with images uploaded to a viewing platform (Topcon Harmony) where two integrated DLSs, Thirona RetCAD^TM (v2.2.0) and EyRIS SELENA+ (v1.7.0) were used to synchronously grade retinal images at the point of care. Images were graded by Google ARDA retrospectively in batches, as software was unavailable to integrate ARDA into the viewing platform for real-time use. All three DLSs are based on convolutional neural networks. [24,25,26] There was no recalibration nor fine-tuning of these DLSs prior to use in the study, with a categorical result given by each, not a prediction score. SELENA+ output stated whether there was referable DR or non-referable DR, but did not give DR severity. Both RetCAD^TM and ARDA outputs gave the DR severity. Due to the real-world nature of this research, participants’ pupils were only dilated as necessary. To ensure adequate clinician oversight, when the DR screener was not a clinician, images were reviewed by an on-call doctor at the time of screening and on-the-spot telehealth consultation was provided for patients with referable disease, or at patient request.

The color fundus photographs were independently graded by two ophthalmologists (YS and SB) according to the The International Clinical Diabetic Retinopathy and Diabetic Macular Edema Disease Severity Scale. [27] Any images where there were discrepancies were independently adjudicated by a retinal specialist (VS). This grading formed the reference standard. Referable DR was defined as moderate non-proliferative diabetic retinopathy (NPDR) or worse. Vision-threatening DR was defined as severe non-proliferative DR or proliferative DR.

2.1. Development of the DLSs

ARDA was developed on approximately 130,000 images from the US and India and initially validated on approximately 10,000 images from EyePACS-1 and Messidor-2 datasets. [24] RETCAD^TM has been validated on multiple datasets including Messidor, Messidor 2 and a private dataset according to their White Paper, yet the initial training set is not clear. [26] SELENA+ was developed in Singapore using 76,370 images from their national DR screening program and internally validated on 70,000 different photographs, also from the screening program. [25] External validation comprised 40,000 photographs from many countries, but mostly comprising people of Chinese ethnicity. [25]

2.2. Ethical Approval

The research project was supported by the Pilbara Aboriginal Health Alliance and approved by the Western Australian Aboriginal Health Ethics Committee (HREC1294). All participants included gave written informed consent. People with diabetes were invited to undergo the DLS-assisted DR screening as part of a clinical service regardless of whether they gave consent to use their data for research purposes.

2.3. Statistical Analysis

Sample size analysis showed that 173 images would be required, based on an expected sensitivity of 90% and specificity of 85%, estimated prevalence of referable DR of 20% and precision of ±10%. [28] Data are presented as number (percentage) for categorical variables and mean ± standard deviation for normally distributed continuous variables. To assess the performance of the DLSs sensitivity, specificity, diagnostic accuracy, negative predictive value (NPV) and positive predictive value (PPV), excluding images deemed ungradable by the DLSs, were calculated using ophthalmologists’ grading as the reference standard. Additionally, the performance of the three DLSs was calculated classifying images deemed ungradable by the DLSs as referable DR. The performance of ARDA and RetCAD^TM to assess vision-threatening DR was also assessed.

To determine if there was a significant difference between the results of the different DLSs and the reference standard, Cochran’s Q, an extension of McNemar’s test, was used. [29] Firstly, this was used to compare referable DR versus non-referable DR between the ophthalmology grading and the three DLSs and secondly comparing misclassification (true positives and true negatives versus false positives and false negatives) between the three DLSs. Where this was significant, a post-hoc Dunn test with Bonferroni correction identified significant differences between the different DR grading systems. Results with p-value or adjusted p-value <0.05 were considered significant.

3. Results

Across six communities, there were 94 people included in the study. Of these, 48 (51.1%) were male, 66 (70.2%) identified as First Nations and the mean ± SD age was 60.3 ± 12.0 years at the time of screening. Ophthalmologist grading of the 188 colour fundus photographs determined 135 (71.8%) images had no referrable DR (121 (64.4%) with no DR and 14 (7.4%) with mild NPDR) and 39 (20.7%) images had referrable DR (26 (13.8%) with moderate NPDR, 6 (3.2%) with severe NPDR and 7 (3.7%) with PDR). There were 14 (7.4%) ungradable images, which were excluded from analysis. There was substantial agreement between the two main graders (kappa=0.71).

ARDA had 100% (95% CI: 91.03-100%) sensitivity overall and therefore 100% NPV, while maintaining specificity of 94.81% (89.68-97.47%) (see Table 1). RetCAD^TM had high sensitivity and specificity of 97.37% (86.50-99.53%) and 97.01% (92.58-98.83%), respectively. SELENA+ had sensitivity of 91.67% (78.17-97.13%) and specificity of 80.80% (73.02-86.74%). ARDA had a high proportion of gradable images, giving a DR grade for all the images deemed gradable by the ophthalmologists, only four of the 188 images (2.1%) were ungradable. Some examples of differences in gradability of images by the ophthalmologists and the DLSs are shown in Figure 1. ARDA classified seven images as referable DR that were non-referable. There were two images with referable DR that RetCAD^TM classified as non-referrable and ungradable. RetCAD^TM classified four images as referable that were non-referable. The diagnostic accuracy for all DLSs was preserved or improved in First Nations Australians (Table 1). For non-First Nations, the sensitivity was 100% for all DLSs, however there were only two with referable DR (Table 1).

There was a significant difference between the DLSs overall for grading for referable DR vs non-referable DR, based on the Cochran’s Q test (p<0.001). The ophthalmology grading (reference standard) was not statistically different to RetCAD^TM and ARDA grading, nor was there a significant difference between RetCAD^TM and ARDA grading (adjusted p≥0.415). However, the ophthalmology grading, RetCAD^TM and ARDA were significantly different to SELENA+ (adjusted p≤0.002). Additionally, there was an overall difference in misclassification between the DLSs (p<0.001) with post-hoc tests showing no significant differences between RetCAD^TM and ARDA but both of these were different to SELENA+ (adjusted p<0.001).

When including the ungradable images from the DLSs as referable DR, there was an increase in sensitivity and decrease in specificity and diagnostic accuracy in both RetCAD^TM and SELENA+, as shown in Table 2. When assessing detection of vision-threatening DR in those with diabetes, ARDA had sensitivity of 92.31% (95% CI: 66.69-98.63%) and specificity of 97.52% (93.79-99.03%) and RetCAD^TM results were 91.67% (64.61-98.51%) and 96.25% (92.06-98.27%), respectively.

4. Discussion

In a remote Australian population with a high proportion of First Nations people, ARDA correctly identified all patients with referable DR. This is the first Australian study to compare multiple DLSs for DR on the same real-world study population and showed that without any modification or finetuning to the DLSs, RetCAD^TM and ARDA detected referable DR and vision-threatening DR with both sensitivity and specificity ≥94.8%. These results were statistically significantly different to SELENA+. While the accuracy was generally high across all three DLSs, there was variation in sensitivity, specificity and gradability underlining the importance of validating DLSs prior to implementation to determine which may be the most appropriate for that setting. Our study highlights that DLSs can be successfully used for DR screening prospectively in a remote Australian setting, providing further evidence to support implementation. Utilising this technology may improve access to DR screening in remote settings, improve health equity and reduce the risk of vision loss. [4,30,31]

ARDA and RetCAD^TM had similar results to other international validation studies, yet SELENA+ had lower performance. ARDA had a sensitivity/specificity of 100% (95%CI: 91.03-100%)/94.81% (89.68-97.47%) which is improved or consistent with previous validation studies from India and Thailand that reported sensitivities between 89-98% and specificities between 92-95%. [32,33] In the current study, RetCAD^TM had a sensitivity/specificity of 97.37% (86.50-99.53%)/97.01% (92.58-98.83%), which is more similar to studies evaluating later versions of RetCAD^TM, with sensitivities between 95-97% and specificities 92-94% (version 2) [34,35] than earlier versions with sensitivities of 84-86% and specificities of 92-93% (version 1), [36,37] but none assessed the same version (v2.2.0) used in this study. SELENA+ had a sensitivity of 91.67% (78.17-97.13%) which is comparable to other studies, however the specificity of 80.80% (73.02-86.74%) was lower than previously reported. The primary validation of SELENA+ for referable DR yielded a sensitivity of 90.5% and specificity of 91.6%. [25] SELENA+ was also validated on 4504 retinal images of Zambians with a sensitivity of 92.25% and specificity of 89.04% for referable DR, including macular oedema. [38] Of note, the dataset used to train SELENA+ comprised a large proportion of Chinese people, [25] while our study had a large population of First Nations Australians, as such, ethnic bias may explain differences in performance.

Australian studies validating the performance of DLSs have shown reasonably good performance. Optain (previously Eyetelligence), DAPHNE and CSIRO’s Dr Grader have reported both sensitivity and specificity over 90%. [14,16,39] However, Optain had lower performance when only assessing First Nations peoples. [15,16] Dr Grader also showed 100% sensitivity but there were only two cases of referable DR in the sample and specificity was lower than Thirona and ARDA. [39] Similar to the current study, DAPHNE was validated on 393 images from First Nations peoples living in remote Australia, yet it used different outcomes of any DR and proliferative DR, [14] which may not be as useful for a population screening tool, where those with moderate NPDR or worse are typically referred to ophthalmology. ARDA has previously been validated on an urban First Nations population, using 1682 images, with a sensitivity of 98.0% and specificity of 95.1% for more than mild DR. [13] ARDA has now been validated in both urban and remote First Nations populations and has high sensitivity, suggesting fewer positive cases would be missed if used for a DR screening program.

Establishment of screening programs requires consideration of diagnostic sensitivity and specificity trade-off. [40] A higher sensitivity would mean that fewer true-positive referable DR cases would be missed, but that more false-positive cases would be referred to ophthalmology. Conversely an increase in specificity would result in fewer unnecessary referrals, but higher number of missed referrable DR cases. Although ARDA had highest sensitivity and RetCAD^TM had highest specificity, both these DLSs had high sensitivity and high specificity, with similar diagnostic accuracy. When determining which tool to use the population needs should be considered. Minimising false-positives may be preferred for rural or remote areas for patients facing significant travel and costs to access ophthalmology services and many patients for few clinicians. [41] For those who attend regular annual or biennial DR screening, it is unlikely that a false-negative would result in vision loss prior to the next screening. [42] However, where recommended DR screening adherence is low, such as in remote Australian settings, [6] it is possible that missing a case of referrable DR could result in vision loss before the next opportunity for DR screening, especially in the context of other risk factors, for example suboptimal glycaemic control. [42] In these cases, ARDA may be better suited. In practice, a hybrid system, incorporating human ‘over-reading’ prior to a specialist review may mitigate unnecessary referral burden. [43]

An ungradable image in a real-world setting would likely result in a referral to ophthalmology or a repeat visit, which decreased DLS specificity and leads to greater unnecessary referrals. Ideally a DLS would be able to grade an image that is of sufficient quality for an ophthalmologist to grade, yet this nuance is not always reported. In the current study, ARDA was able to grade all images deemed gradable by the ophthalmologists, while two were ungradable by RetCAD^TM. In contrast, almost 13% of all images were ungradable by SELENA+ and 54% of these ungradable images were gradable by ophthalmologists. Proportion of gradable images, and gradability in comparison to a human grader should also be considered prior to implementing DLS-assisted DR screening.

4.1. Strengths and Limitations

The strengths of this study include the real-world design, comparison of multiple DLSs on a population with a high proportion of First Nations people and images graded by an ophthalmologist adjudication panel. By assessing three DLSs simultaneously, the diagnostic performance metrics demonstrate pertinent differences to consider prior to implementation. However, limitations of the study include that diabetes status relied on self-report and consistent with other similar studies, diabetes type was not ascertained, [13,16,24,35,36,39] which may introduce misclassification bias. As expected with the real-world nature of the study, there was inconsistent dilation, which may have contributed to more ungradable images. Furthermore, by using single-field retinal images it is possible that some DR was not detected. This study did not specifically assess diabetic macular oedema from the retinal photographs, as this is not reported by RetCAD^TM or SELENA+. While we had sufficient sample size to analyse performance of the DLSs in First Nations, only two people of other Australian ethnicity had referable DR. A greater sample size would have resulted in smaller confidence interval widths. The Pilbara region of Australia is unique, hence results may not be generalisable to other settings.

4.2. Conclusions

In a remote real-world study comprising majority First Nations people, DLSs had high sensitivity and specificity for detecting referable DR. Direct real-world comparison is useful to guide implementation, particularly when the population comprises minority groups. As DLS are not widely adopted, research focussed on addressing implementation barriers is needed. Point-of-care DR screening using DLSs may improve DR screening rates and potentially improve health equity and prevent vision loss when paired with appropriate referral and treatment pathways.