Impact of Photographer Experience on Fundus Image Gradability in Non-Mydriatic, AI-Assisted Diabetic Retinopathy Screening in a Primary Care Setting

1. Introduction

Diabetic retinopathy (DR) is a leading cause of visual impairment among adults in the United States. Of an estimated 38.4 million people with diabetes mellitus (DM) [1], 9.6 million are diagnosed with DR [2], with roughly 5% suffering from vision-threatening DR (VTDR) [2]. With the increasing lifespan of patients with DM and a growing global aging population, a high burden of DR and demand for eye care is projected to continue through 2045 [3]. Routine screening for DR after an initial diagnosis of diabetes is a valuable method of preventing the development of VTDR [4].

Dilation remains the standard of care for DR screening in ophthalmology settings considering the difficulty in obtaining a wide field of view of the retina while non-dilated [5]. Artificial intelligence (AI)-based screening systems such as EyeNuk’s (Los Angeles, CA) EyeArt, Digital Diagnostics’ IDx-DR, and AEYE Health’s AEYE-DS have high sensitivity and specificity rates, proving to be valid alternatives to traditional dilated screening approaches and increasing efficiency of screenings [6,7,8,9,10,11,12]. As of 2023, United States Food and Drug Administration (FDA) clearance has enabled EyeArt to be used with multiple camera models, expanding its access in primary care settings [13].

AI-based screening through digital fundus photography of non-dilated pupils in primary care settings is a convenient, highly reliable alternative to standard-of-care dilated screening at an eye-care provider that can improve DR screening rates by providing diabetes care and diabetic eye disease care in the same location [14,15]. The implementation of precise AI screening for DR using non-dilated photographs across primary care settings can decrease clinic burden for ophthalmologists, improve workflow, and avoid potential risks of dilation by non-ophthalmic personnel including acute angle-closure glaucoma [16].

Despite these benefits, image quality proves to be a critical limitation of AI-based systems [14]. Successful grading by AI screening programs is dependent on high quality fundus photographs. A gradable fundus photograph requires a clear and in-focus view of the fundus. Poor image quality remains common in non-dilated AI-based DR screening in primary care settings, leading to high rates of false positives [14].

Photographers must actively mitigate issues such as incorrect focus, dirty lens reflections, uneven light exposure, patient motion, and eyelid obstruction. Additionally, fundus photographs must be labeled and uploaded correctly to provide AI-gradable images. Since the quality of photographs determines the effectiveness of DR screening, attempts have been made to improve gradability rates (GR) such as offering multiple camera options [17,18] and using dilating agents when appropriate [19,20].

Importantly, one study that focused on smartphone-based retinal photography in primary care found that medical assistants (MAs) can obtain high-quality retinal photography of dilated pupils within minutes of first-time operation [21], underscoring the promise of training non-ophthalmic personnel to take fundus photography and expand screening access. This finding is limited to photographs of dilated pupils using smartphone-based photography but supports the idea that photographers with limited experience can obtain high quality photographs. Alternatively, a 2004 study evaluating whether fundus photograph quality improves with operator experience found no significant difference in fundus image quality among photographers with differing levels of training when using a nonmydriatic camera [22]. While this study provides insight of image quality obtained with more traditionally-used nonmydriatic cameras operated by inexperienced personnel, its experimental design relied on subjective human grader evaluation of image quality based on a 3-level scale [22] unlike the objective algorithms used in current AI-based systems.

There are no studies so far that have assessed the impact of a primary care-based photographer's experience on the quality of non-dilated fundus photographs using an AI-based DR screening program. We hypothesized that the non-dilated, AI-assisted fundus photograph GR of MA photographers at Temple Primary Care Clinics improve with the number of photographs taken.

2. Materials and Methods

We performed a retrospective chart review of 1354 DR screenings of diabetic patients 18 years and older between 05/12/2021 – 12/28/2022. The protocol was approved by the Temple Institutional Review Board. A DR screening consisted of fundus photographs of both eyes taken with Canon CR-2 AF camera model accompanied by EyeNuk. Photographs were read by EyeArt. Fundus photographs were captured by 30 different MA photographers at 10 Temple Primary Care Clinics. All MA photographers received photography training by an ophthalmic photographer prior to initiating screening.

Screening results with at least one uninterpretable eye photograph were reported as “ungradable” by EyeArt.

MA photographers who conducted fewer than 10 screenings were excluded due to the low sample size. GR were calculated by dividing the number of gradable screenings by the total number of screenings conducted by each photographer.

We examined whether photographer experience, defined as the total number of photographs taken, was associated with the proportion of gradable images. A simple linear regression analysis was used to determine if a linear relationship exists. A simple polynomial expansion was used to evaluate potential nonlinear effects. For each model, overall significance was assessed using the F-statistic; a p-value of 0.05 or less was considered significant.

For a longitudinal learning-curve analysis, MA photographers who conducted greater than 90 DR screenings were identified and had their GR assessed over time in intervals of 10 consecutive screenings for the first 91-100 screenings. MA GRs were averaged at each interval of 10 photographs taken. A 95% confidence interval was used at each interval. Any association between photographer experience, using successive screenings as a proxy, and average image gradability was assessed using simple linear regression; overall model significance was assessed using the F-statistic; p-values of 0.05 or less were considered significant.

3. Results

Out of the 30 MA photographers, only 11 conducted at least 10 screenings so only those 11 were included in the analysis.

The simple linear regression analysis showed no significant association between number of photographs and percentage of gradable images (F statistic = 0.187, p = 0.6754) (Figure 1, Table 1). The simple polynomial expansion analysis also showed no evidence of association (F statistic = 0.259, p = 0.7784) (Figure 1, Table 1).

Three MA photographers conducted at least 91 screenings. The linear regression analysis showed no significant association between photographer experience, using successive screenings as a proxy, and average image gradability (F statistic = 0.684, p = 0.4323) (Figure 2).

4. Discussion

For a successful screening program to function, photo gradability should be high because high rates of ungradable images by MA photographers would wind up placing the patients back in the traditional in-person exam system. This adds cost and time when it already very hard to get an appointment, resulting in more suboptimal care. An MA just learning to take photographs would not be expected to have the same gradability rate as an experienced ophthalmic photographer, but with time and practice, the MA should show improvement with the goal of achieving the rate of the photographer. However, contrary to our hypothesis, image gradability did not improve with increased photographer experience. The data demonstrates no correlation between the number of photographs taken by a MA photographer and his/her gradability rate.

Our findings match a previous study which found that photographic training and experience do not influence fundus image quality for images taken with a nonmydriatic camera. Similar to the study that reported 79% of images captured by minimally trained photographers of nonmydriatic pupils were of poor or moderate quality [22], we found similar results.

Our findings are limited by a small sample size. The performance of only 11 MA photographers was evaluated in this study, some of which had taken as few as 12 photographs (Figure 1, Table 1). Additionally, only 3 MA photographers took a large volume of over 90 photographs, limiting the strength of our findings of individual photographer performance over successive sets of photographs (Figure 2). The results are specific to an urban health center that provides care to medically underserved patients and may not be generalizable.

Not all ungradable images should be viewed as purely detrimental and the fault of the photographer. Some are due to media opacities such as cataracts, which may lead to surgical referrals and significant visual improvement. These incidental findings, although unrelated to DR, represent a potential secondary benefit of regular DR screening.

Repeated practice alone did not improve performance. Given the lack of improvement in gradability of images over time, it is reasonable to conclude that teaching MAs one time to operate a fundus camera is ineffective in achieving a high gradability rate. One avenue worth exploring is the development of a standardized competency-based training program for MA photographers with training completion marked by certification. Refresher photographer training supervised by experienced photographers may be necessary to enhance performance in DR screening initiatives. Further research is necessary to identify why MA photographer performance does not improve with experience.

Aside from dilation, which is not practical in a primary care setting, another way to potentially improve photo gradability is to use fully automated robotic fundus photography. This might remove the barrier of training medical professionals to obtain high quality photographs. We are investigating this approach now.