Real World Analysis of Two Handheld Retinograph, Evaluation by Ophthalmologist and an Artificial Intelligence Algorithm

1. Introduction

Diabetic retinopathy (DR) is a complication of diabetes mellitus (DM) that affects retinal vasculature in the form of microangiopathy and at the same time the neurons that compose it in the form of neuropathy, which means that diabetic retinopathy. Current it is considered to be a neurovascular involvement of the retina [1].

DM currently affects 537 million patients worldwide and is predicted to rise to 643 million by 2030 and to 783 million by 2045 [2]. The increase in DR runs parallel, currently affecting 6.4 million people in Europe alone with an expected increase to 8.6 million by 2025, of whom 30% will need treatment [3]. DR also remains the leading cause of preventable vision loss and blindness in adults aged 20 to 74 years, especially in middle- and high-income countries [4,5].

Given the rapid global increase in DM and increasing life expectancy, DR will remain a major public health challenge. Screening for DR through telemedicine has proven to be effective [6] therefore promoting it where it is already in place and implementing it where it is not. Only early detection of DR will allow us to prevent its evolution to more severe forms and to treat it early to avoid vision loss [7].

Current screening systems can reach a significant proportion of the patients with DM, but it is difficult to screen all patients every one or two years, as recommended [2]. In our experience, only 40% of patients are screened annually [8], despite having four non-mydriatic camera units in our health area. To improve this, extending screening to two years according to the metabolic control of patients has been proposed, along with bringing in other professionals, such as general practitioners (GPs), endocrinologists or pediatricians into the system. At a technical level, the use of portable cameras has been proposed that can be used by GPs, together with artificial intelligence (AI) algorithms that read retinographs automatically and help to detect DR [9,10].

Based on these proposals, the present study evaluated two portable cameras and an AI algorithm to detect DR developed by our team, and the results were compared [11].

2. Materials and Methods

2.1. Design

A study based on a real population of patients with type 2 DM. The reference being our Health Area (Hospital Universitari de Sant Joan de Reus, Spain) of 226,508 inhabitants, in which 17,792 patients are registered with DM. The study has been ongoing since 2007, when we began offering annual retinography to our patients with type 2 DM. The full screening programme is described in full elsewhere [12].

Briefly, screening is carried out from retinographs obtained by non-mydriatic cameras. Images are stored in the computerized medical record and are later read by specialists (general practitioners, endocrinologists or pediatricians). We have recently replaced three of the four non-mydriatic cameras with optical coherence tomography (Triton OCT™) (hereafter referred to as Triton) equipment, which allows us to detect patients with macular pathology.

2.2. Objectives

The first was to evaluate the ability of two portable cameras to take good quality retinographs. Images were captured with an Android smartphone (VOLK™ Vistaview, Topcon Healthcare®) (hereafter referred to as Vistaview) and with a smartscope whose images are similar to the classic retinograph (Aurora™, Optomed Topcon Healthcare®) (hereafter referred to as Aurora).

The first step was to evaluate the images captured from the devices. Two independent retina specialists observed the images from the Triton taken by the non-mydriatic camera unit located at the Hospital Universitari Sant Joan de Reus, where the study was carried out. This was considered the gold standard when determining the concordance of the results between teams.

The second step was to evaluate the retinographs obtained from the two portable systems and those obtained from the Triton using an AI algorithm that we have developed, called MIRA© [11]. Briefly, the MIRA© algorithm was developed from the public database MESSIDOR© [13] and our MIRADATASET© database [14] which contains 40692 retinograms labelled according to whether or not they have DR and what type of RD, using the MESSIDOR classification that classifies DR into three levels [15].

2.3. Inclusion Criteria

Patients with type 2 DM from our DR screening programme.

2.4. Exclusion Criteria

Type 1 DM and other causes of diabetes mellitus such as gestational DM, and retinographs that were not in a patient's electronic medical record or did not have DICOM format.

2.5. Methods

Retinographs were obtained randomly from patients presenting for DR screening. For each patient, a retinograph of each eye was obtained from the three devices, the Vistaview, the Aurora and the Triton. The images were focused on a point located between the fovea and the temporal side of the papilla, with retinographs being 45⁰-55⁰. Pharmacological mydriasis was achieved by applying tropicamide eye drops if necessary. The retinographs were sent via the PACS (Picture Archiving and Communication System) system to the patient's electronic medical record (EMR), where they were stored in DICOM (Digital Imaging and Communications in Medicine), a standard format that guarantees the interoperability and compatibility of medical images.

Table 1 shows DR classified with the international classification of DR [16] at 'level 1, or mild-DR', and 'level 2, or moderate-DR'. At 'level 3, or severe-DR' we introduced a variant to include images with newvessels in the retina (proliferative DR) and we included any patients with macular edema. Therefore, our level 3 would be equivalent to a vision-affecting referrable DR or VTDR (vision threatening diabetic retinopathy).

2.6. Technical Characteristics of the Three Devices

Table 2 shows the variations between the three cameras used to obtain retinographs. The Vistaview is a portable retina camera that incorporates an Android smartphone as a capture system that allows the examination of the retina with a field of view of 55⁰.

The Aurora is a smartscope, a handheld retina camera, which allows examination of the retina with a field of view of 50⁰, covering the macula and papilla.

The Triton is a scan-sourced optical coherence tomograph device with a scan speed of 100 kHz and a wavelength of 1,050 nm, which delivers a true-colour, non-mydriatic fundus image.

2.6. Ethics and Consent

The study was carried out with the approval of the local ethics committee, the Medical Research Ethics Committee (CEIM) of the Pere Virgili Health Research Institute (IISPV), Tarragona, Spain, RetinaReadRisk approval code, protocol version 1. 10/03/2022, CEIM reference number: 071/2022

2.7. Statistical Methods

Data was analysed via the SPSS program, version 22.0 (IBM® Statistics, Chicago, IL, USA). A descriptive statistical analysis of the quantitative data was performed by determining the mean and standard deviation. For qualitative data, frequency and percentage analysis in each category were used. For all parameters, descriptive statistics (including number of values, mean, standard deviation (SD) and standard error of mean (SEM) were calculated. A p value less than 0.05 was considered statistically significant.

The screening performance of the AI Machine Reading Algorithm (MIRA) © was measured using a 2x2 confusion matrix/contingency table for each team. Given a classified dataset, there were four basic combinations of real and assigned: true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN).

Statistical evaluation of the dataset included sensitivity, specificity, positive predictive value or accuracy, negative predictive value, harmonic mean or F1 score, and accuracy.

Sensitivity or recall is the proportion of the population for which the test is correct and, in clinical diagnosis, sensitivity and specificity values are considered good when they exceed 80%. Positive or accuracy predictive value, and negative predictive value, give a proportion of the population with a given test result for which the test is correct and incorrect, respectively.

To assess concordance, we determined the values of accuracy and harmonic mean. Diagnostic accuracy is expressed as the proportion of correctly classified subjects among all subjects, accuracy values of >0.8 show an almost perfect correlation, while values of 0.6-0.79 imply a significant correlation, 0.4-0.59 a moderate correlation, 0.2-0.39 a regular correlation and 0-0.2 a poor correlation. The harmonic mean or F1 score, also known as the Sørensen-Dice coefficient or Dice similarity coefficient (DSC), is a measure of test performance that combines precision (the positive predictive value) and recall (sensitivity), the highest possible value of F1 is 1, indicating perfect accuracy and recall, and the lowest possible value is 0 if the accuracy or recall is zero. The advantage of using the harmonic mean over accuracy is that it is not affected by the prevalence of the disease, which is the case with accuracy, so if the prevalence is low it tends to give high accuracy values [17].

3. Results

3.1.

A retinograph was taken of each eye with each of the three devices. In total, 4260 retinographs were taken of 2130 patients. The mean age of the sample was 67.43±11.21 years (38-91 years), of which 1256 (58.96%) were women. Regarding the treatment of DM, 264 patients (12.39%) were treated through diet alone, 1810 (84.98%) with oral antidiabetics and 56 (2.63%) with insulin or insulin + oral antidiabetics. Finally, 1527 (71.69%) had high blood pressure.

Table 3 shows the differences between the retinographs across the three devices. We first tried to take the retinographs without pupil dilatation, but the images were without DR in 69.83% with the Vistaview, in 83.91% of the cases with the Aurora, and 71.11% of cases with the Triton.

After dilation, the percentage of images that were not of sufficient quality for ophthalmologists to evaluate was 27.98% with the Vistaview, 7.98% with the Aurora but only 0.93% of the images obtained with Triton. The principle causes of poor quality were

(I) the difficulty in focusing the handheld device, particularly the Vistaview, (ii) the presence of cataracts, obviously affecting all three devices equally and (iii) Little mydriasis

Patients with moderate-DR or severe-DR were identified with similar success across all three devices, but for mild-DR the Aurora and the Triton were more successful than the Vistaview.

3.2. Statistical Analyses

Data comparing the handheld devices with the Triton is presented in Table 4. Applying the 2x2 contingency table we found the two handheld devices to be reliable. Although the Vistaview has a somewhat lower score in the application of the F1 statistic or harmonic mean (0.93 for any-DR and 0.91 for Mild-DR), the results were good for both in any case.

3.3. The Comparative Study of the Results of the Three Devices Using the MIRA© Algorithm

Table 5 shows the results from reading the images obtained with the MIRA© automatic reading AI algorithm.

According to the type of DR, the MIRA© algorithm reads retinographs from the Aurora and Triton devices more accurately than those from the Vistaview. Mild-DR is detected in fewer cases with the Vistaview (102 cases) compared to the Aurora (218) and the Triton (220). The latter are what we consider real cases since they were identified by retina specialists and is the gold standard. The AI algorithm identified moderate-DR and severe-DR with similar success across the three devices.

Statistical Study of the Results Obtained by AI Algorithm MIRA©

We evaluated the images obtained by the three devices, the Triton tomography being what we currently consider to be the gold standard, with images read by specialist ophthalmologists.

Table 6 shows the statistical analysis of the performance of the three devices in the detection of DR, using 2x2 contingency tables. The sensitivity of the Vistaview smartphone to identify any-DR was 0.89, being outperformed by the portable Aurora (0.99) and the images obtained by the Triton OCT(0.99)

Similarly, for Mild-DR, the scores for the Vistaview drop to a sensitivity of 0.83 with a positive predictive value of 0.85 compared to the values of 0.99 sensitivity for both the Aurora and the Triton and a positive predictive value value of 0.98 for the Aurora. This is confirmed by the TP values and by the F1 harmonic average score, the smartphone scoring a harmonic average of 0.89 for detecting any-DR and 0.83 for mild-DR.

In summary, the statistical study indicates that the Aurora and Triton OCT devices detect with high reliability when DR is present or absent at all stages of DR. The Vistaview device is safe when it indicates that there is no DR but unreliable in detecting cases of Mild-DR.

4. Discussion

The constant increase and diversity of the population with DM has led to the pursuit of new ways of reaching as many of the diabetic population as possible and various publications by scientific societies have reported progress via telemedicine [2,6].

The authors believe that portable devices used together with AI algorithms that can read retinographs will be able to reach a larger population than the current model allows [18,19,20,21,22,23].

The present study aimed to evaluate the usefulness of two of those portable devices in screening patients with diabetes mellitus. We selected one smartphone camera, and a smartscope. We compared the images obtained with those two devices with the Triton optical coherence tomography camera, which is the standard for DR screening in our health area. Results show that the smartphone produced a higher number of ungradable retinographs (27.98%) than the Aurora smartscope (7.98%) or the standard Triton OCT (0.93%). We can report that mydriasis was necessary for all patients when using the smartphone compared to 73.05% using the Aurora and 71.11% using the Triton. The major difficulty was being able to focus accurately on the retina when using the smartphone. We can also report that the smartphone had more difficulty in distinguishing mild-DR forms (only 93 cases) than the Aurora (109 cases) and the Triton (110 cases).

In the second part, we analyzed the results after the automatic reading of retinogtraphies using our MIRA© algorithm. As in the first part of the study, the automatic image analysis detects far fewer cases of retinographies with Mild-DR in the images obtained with the Vistaview device compared to the other two devices, thus only detecting 146 retinographies with Mild-DR compared to 218 with the Aurora or 220 with the OCT. We could conclude that somehow for both the human eye and the AI algorithm the detection of Mild-DR is more difficult with the smartphone, perhaps the difference in pixelation of the images obtained by the smartphone which is 28.4 Mpixels compared to 5 Mpixels for the other two devices (Table 2).

Although the images obtained with all three devices could be considered good, the statistical analysis revealed that the sensitivity or recall, and the positive predictive value or precision, are poorer for the images obtained with the smartphone, both in terms of any-DR or Mild-DR. The differences were less marked when using the harmonic mean or F1 score, which scored the Vistaview smartphone at 0.97 for accuracy and the Aurora smartscope at 0.99 for the presence of any-DR or mild-DR. However, when applying the harmonic average the values change, at 0.93 (any-DR) and 0.91 (mild-DR) for the Vistaview and 0.98 and 0.95, respectively, for the Aurora. These differences are due to what we have commented on methods, accuracy is affected by prevalence and cases such as the one in this study with RD prevalences of less than 9% make the determination of the harmonic mean as a statistic to differentiate cases more accurate.

A similar study by Jacoba et al [24] was a clinical series on 225 eyes of 116 patients in which the presence of macular pathology alone was evaluated. That study used two portable retina smartscopes, the Aurora and the RetinaVue™, comparing them with the Cirrus™ OCT camera. The study reported good specificity but low sensitivity of the portables in detecting maculopathies.

Another similar study aimed not only to screen for DR but also to detect macular pathology, whether diabetic or not [23]. The study reported ungradabilty with the OCT at 0.9%, a figure similar to ours, and 4.4% with the Aurora, a value somewhat lower than ours, which is 7.98%, although the focus was on the macular area rather than on the whole retina.

A review of the efficacy of smartphones via a meta-analysis carried out by Tan et al [21] reported on five Android or iPhone5S mobile devices. Results showed that for the detection of any-DR mean sensitivity was 87% (minimum 74%, maximum 94%) and specificity was 94% (minimum 81%, maximum 98%), values not dissimilar to those we obtained using the Vistaview (sensitivity of 88% and specificity of 99%), although notably sensitivity for mild-DR wasvery low with an average of 39% (minimum 10% maximum 79%). It is important to take into account that smartphone technology is constantly changing and improving [25].

The Optomed™ Aurora has been tested in various studies and results have been similar to ours. The largest sample is that of Salongcay et al. In 2023 [26]. That study reported a percentage of unreadable images of 7.5%, similar to ours with 7.07%, and an agreement of 82.4%, lower than ours at 98%. Other studies have reported similar values to ours, with growth figures of 93% in Kubin et al. [27] and a positive predictive value of 98% and a specificity of 97% in the Salongcay et al. [28].

One of the strengths of the present study is that the analysis of images has been carried out from those found in the electronic medical record (EMR) rather than those stored in the equipment. That brings together the images in DICOM format for all devices and allows us to compare them more easily. Another strength is the much larger sample size compared to most previous studies. We were able to include 4260 images from 2130 patients, and only the Salongcay study of 2023 includes a higher number with 5585 images from 2793 patients.A limitations of the study is clearly having used only two devices, which makes extrapolation of the results to other devices unsafe. The use of a single retinograph focused on a point between the macula and temporal side of the papilla might also limit the detection of DR against studies that include more retinographs of more fields or perhaps wide field retinographs, which have been shown to change the severity of DR [29]. Another point to note is that at level 3 of DR we included patients with proliferative retinopathy and macular edema, which might alter the results when compared to other studies, although it can be used as a reference for studies that want to evaluate the number of patients we have detected with DR-referable.

5. Conclusions

We can affirm that image capture systems or handheld retina cameras are useful for detecting DR in screening programmes and might be further helped by incorporating AI algorithms. We have found images taken with a smartscope or portable retinograph device better than with a smartphone, which gave us a large number of unreadable images. Both devices require more efficient focusing, in order to make image centering easier, improve image readability, and improve the quality of images taken through small pupils to 3 mm, reducing the need for mydriasis.

6. Patents

Software MIRA, register SAFE CREATIVE code 2007104712196. Date 10 de julio de 2020 a las 11:24 UTC