PolyDeep Advance 1: Clinical Validation of a Computer-Aided Detection System for Colorectal Polyp Detection with a Second Observer Design

1. Introduction

Colorectal cancer (CRC) is the third most frequent cancer and the second leading cause of cancer-related mortality worldwide [1,2]. The development of precursor lesions, including adenomas and serrated lesions, plays a critical role in its progression [3]. In recent years, public health systems have implemented CRC screening programmes for average-risk individuals aged 50–75 years to reduce both its incidence and mortality [1,2,3,4]. However, the quality of colonoscopy - affected by inadequate bowel preparation, insufficient mucosal exposure, or endoscopist fatigue - can lead to missed precursor lesions and the subsequent development of post-colonoscopy CRC [5].

Artificial intelligence (AI) is a research field focused on developing machines able of performing tasks that typically require human intelligence. One of its most significant subfields is Machine Learning (ML), where AI models learn to execute tasks for which they were not explicitly programmed [2]. Within ML, Deep Learning (DL) represents a specialised subset of ML techniques, that relies on large artificial neural networks, usually with a high number of layers, enabling the automatic learning of complex features [6]. The success of DL techniques in recent years, surpassing the performance of classical ML methods, has triggered a renewed surge of interest in AI. In the medical field, this is reflected in the growing number of AI-based medical devices authorised by the FDA [7]. Notably, more than 770 devices have been approved in the past decade, a stark increase compared to the 27 approvals in previous years [7]. Among the most impactful AI-based medical devices, particularly in colonoscopy, are Computer-Aided Diagnosis (CAD) systems. These systems support medical professionals during procedures to enhance decision-making by extracting relevant features from colorectal lesion images, enabling both detection (CADe) and characterisation (CADx) [2,4,8,9]. The worldwide integration of CAD systems into medical image analysis represents a revolution in routine clinical practice. In colonoscopy, CADe has been shown to improve procedure quality by enhancing key performance indicators such as the adenoma detection rate (ADR), polyp detection rate (PDR), and adenoma miss rate (AMR) [10]. A common study design to evaluate the impact of CADe on polyp detection is based on a second observer, with the endoscopist blinded to CADe outputs [11,12,13].

PolyDeep is a DL-based CADe/x system designed to detect and characterise colorectal lesions [14,15,16,17,18]. The system integrates a YOLOv3 detection network, coupled with an object tracking algorithm, and a ResNet50 classification network [14,17,18]. PolyDeep was developed using transfer learning techniques, fine-tuning models (ResNet50 based on ImageNet and YOLOv3 on PASCAL VOC) with still images of colorectal lesions extracted from colonoscopy videos [14,17,18]. As part of the development of PolyDeep, we created PIBAdb (Polyp Image BAnk database), a comprehensive database of images and videos of colorectal lesions manually annotated by expert endoscopists, which is currently available in the biobank of the Instituto de Investigación Sanitaria Galicia-Sur [19]. Prior to clinical validation, we enhanced the detection and characterisation models by incorporating new datasets from the project and adjusting several configuration parameters to optimise the CADe/x system for a real colonoscopy procedure [16]. Specifically, the detection model was improved through fine-tuning with non-polyp images to reduce false positives (included in PIBAdb), while the characterisation model was retrained with new polyp images obtained during the clinical validation phase (although these new images are not already included in PIBAdb). The aim of the study is to evaluate and compare whether the diagnostic performance of PolyDeep is better than that of the expert endoscopists in detecting colorectal lesions in a population-based colorectal cancer screening program.

2. Materials and Methods

2.1. PolyDeep Advance 1: Study Design

PolyDeep Advance 1 (NCT05514301) is a single-centre, prospective diagnostic test study with a second observer design. The research protocol was approved by the local institutional review board (2022/067).

2.2. Inclusion and Exclusion Criteria

We enrolled patients referred for colonoscopy within the Galician CRC screening program in Ourense, Spain; either after a positive fecal immunochemical test (FIT) or as a surveillance colonoscopy after the resection of advanced colorectal lesions. All study participants provided written informed consent. We excluded individuals with a personal history of CRC, colonic resection, or any syndrome predisposing to CRC, such as hereditary syndromes or serrated polyposis syndrome. For the analysis, we excluded participants when cecal intubation was not achieved or with insufficient bowel cleansing (defined as Boston Bowel Preparation Scale < 2 in any segment or < 6 overall).

2.3. Clinical Setting

The endoscopists performed the colonoscopy in the conventional clinical setting, blinded to the CADe/x system. A second observer attended the colonoscopy viewing the PolyDeep images on a separate screen. When PolyDeep identified a potential lesion, it triggered a visual alert in the form of a red border around the screen, which was activated when the CADe consistently tracked one or more potential polyps (also indicated by bounding boxes) across several video frames. If PolyDeep identified a possible lesion that was not detected by the endoscopists, the second observer alerted the endoscopists for confirmation. All detections rejected by the endoscopists (i.e., bubbles, stool, normal mucosa and undefined artefact) were classified as false positives. Thus, the endoscopist was responsible for confirming the CADe detections. The second observer recorded the detections and characterisations made by both the endoscopists and PolyDeep in a log sheet form.

Additionally, the endoscopists classified lesions according to the Narrow Band Imaging International Colorectal Endoscopic (NICE) classification [20], categorising them as adenomas (NICE II) or non-adenomas (NICE I). PolyDeep is trained to detect and characterise colorectal lesions as neoplasms, including adenomas and serrated lesions, or non-neoplasms, such as hyperplastic lesions. During colonoscopy, the CADe/x system identified lesions by bounding boxes around them and characterised them as neoplasia if the probability assigned to the bounding box was greater than 50% [14,17,18].

2.4. Study Endpoints

The gold standard for the study was histology. Lesions without histology or cases of CRC were excluded from the analysis. The primary endpoint of the trial was the detection of colorectal polyps, including adenomas, serrated lesions, and hyperplastic lesions. The secondary endpoints included the detection of adenomas, serrated lesions (i.e., sessile serrated lesions or traditional serrated adenomas), neoplasia (i.e., adenomas or serrated lesions), diminutive (≤5mm) polyps and advanced colorectal polyps (i.e., adenomas or serrated lesions ≥ 10 mm, adenomas with villous histology or high-grade dysplasia, or serrated lesions with dysplasia).

2.5. Data Collection and Study Population

We designed the electronic case report form (eCRF) using the RedCap platform at the Galicia-Sur Health Research Institute (IIS Galicia-Sur) (https://redcap.tic1-iisgaliciasur.es/, accessed on 26/04/2024). The eCRF captured data on demographics, colonoscopy, and lesion characteristics (including location, size, and histology). Additionally, we included information on the optical diagnosis and the histology predicted by the endoscopists and the CADx.

2.6. Sample Size Calculation

We calculated the sample size based on the assumption of 90% sensitivity for the endoscopists and 95% sensitivity for PolyDeep, with a beta error of 20%, an alpha error of 5%, and an estimated 5% rate of incomplete colonoscopies [21]. Based on these assumptions, we determined a sample size of 205 patients, with a mean of 2.5 lesions detected per procedure, to evaluate a total of 487 lesions in the study.

2.7. Statistical Analysis

We performed a descriptive analysis of the study population. Qualitative variables are presented as absolute frequencies and percentages, while quantitative variables are presented as means and standard deviations. For the primary and secondary endpoints, we calculated the diagnostic performance metrics (i.e., sensitivity, specificity, positive and negative predictive values and odds ratio) for endoscopists and PolyDeep. The comparison between endoscopists and PolyDeep was performed using the McNemar test. In all categories, we conducted sub analyses by size (≤5 mm or >5 mm), morphology (protruded or non-protruded), and location (right colon or left colon). Additionally, the precision of endoscopists and PolyDeep in the characterisation of adenomatous and neoplastic lesions, respectively, was determined. We used the statistical software R version 4.3.0 (The R Foundation for Statistical Computing, Institute for Statistics and Mathematics, Vienna, Austria) for statistical analysis.

3. Results

3.1. Population Description and Main Endpoints Results

Between February 2023 and April 2023, we recruited 205 patients (55.1% male, 63 ± 6.2 years of age) who participated in the Galician population-based CRC screening program. We included patients who underwent colonoscopy after a positive FIT (60.5%) or as endoscopic surveillance (39.5%). We excluded 8 patients from the study: 7 due to insufficient bowel cleansing or no cecal intubation, and 1 due to a personal history of CRC and previous colonic resection.

Overall, the CADe system and endoscopists performed 794 detections, including 410 lesions and 384 false positives identified by PolyDeep. The endoscopists detected 384 lesions, of which PolyDeep missed 9.51%. PolyDeep detected 371 lesions, of which the endoscopists initially missed 6.34%. Using the endoscopists’ detections as the reference for lesion identification (without considering histological analysis), the sensitivity for endoscopists was 93.7% ((384/410) × 100), while the sensitivity for PolyDeep was 90.5% ((371/410) × 100).

Figure 1. Flowchart of the study.

Figure 1. Flowchart of the study.

Table 1 presents the characteristics of the lesions according to the observer (both, endoscopists or PolyDeep). Overall, the histology of the identified polyps was adenomas (56.34%), serrated lesions (9.51%), hyperplastic lesions (17.80%), normal mucosa (10.24%), non-retrieved polyps (4.88%) and CRC (1.22%). Most of the lesions were small, located in the right colon, and exhibited a protruded morphology. Polyps detected during the study had a mean size of 5.1 ± 4.9 mm.

3.2. Diagnostic Performance for Polyp Detection

We excluded 25 lesions from the diagnostic performance analysis: 5 cases of CRCs and 20 lesions without histology. Table 2 shows the diagnostic performance of the endoscopists and PolyDeep for polyp detection. No statistically significant differences were found in sensitivity (94.2%, 91.5%, p = 0.2) or specificity (9.5%, 14.3%, p = 0.7) between endoscopists and PolyDeep. The 91.5% sensitivity achieved by the CADe system in the clinical evaluation surpasses the 89.91% (87.20%–91.94%) sensitivity obtained by the initial detection model in a video-based evaluation [17].

In the sub-analysis by size, we found no statistically significant differences between the endoscopists and PolyDeep. Regarding lesion morphology, although diagnostic performance did not differ in protruded lesions, endoscopists demonstrated higher sensitivity than PolyDeep in the detection of non-protruded lesions (94.5% vs. 82.4%, p < 0.05), with no differences in specificity (11.8% vs. 17.6%, p = 1.0).

In the secondary analysis, we compared the diagnostic performance of endoscopists and PolyDeep according to the lesion histology. As shown in Table 3, no differences in sensitivity or specificity were found in any of the categories evaluated. Data on the diagnostic odds ratio, as well as the positive and negative predictive values, are also presented in Table 3.

3.3. Diagnostic Performance for Polyp Characterization

We calculated the diagnostic performance of PolyDeep and endoscopists for the characterization of neoplastic/non-neoplastic and adenomatous/non-adenomatous lesions when histology was available (n = 385). The endoscopists provided an optical diagnosis for all 385 lesions (100%), classifying 149 as NICE I (38.2%), 236 as NICE II (60.5%), and 5 as NICE III (1.3%). In comparison, PolyDeep achieved an optical diagnosis for 329 lesions (85.5%), classifying 259 (78.7%) as neoplastic (with a probability >50%) and 70 (21.3%) as non-neoplastic (with a probability ≤50%).

For adenoma histology prediction (NICE II), the sensitivity and specificity of endoscopists were 76.6% (95% CI 70.5–81.8%) and 61.7% (95% CI 53.5–69.3%), respectively. In contrast, for neoplastic histology prediction, PolyDeep exhibited a sensitivity of 82.6% (95% CI 77.1–87.1%) and a specificity of 32.2% (95% CI 22.8–43.2%).

4. Discussion and Conclusions

PolyDeep is a CADe/x system that has demonstrated its ability to detect and characterize polyps in an in vitro study [16]. In this prospective diagnostic test study, PolyDeep increases the number of lesions detected during colonoscopy. When compared PolyDeep to blinded expert endoscopists, with histology as the gold standard, we found no significant differences in the detection of polyps, adenomas, neoplasia, or advanced colorectal lesions. These findings suggest the potential of PolyDeep as a clinical support tool for colonoscopy procedures.

For the clinical validation of PolyDeep, we adopted a widely validated study design [11,12,13,22,23], which enabled us to evaluate the impact on endoscopic diagnostic performance. Our results are comparable to other studies that used a second observer design [12,13,23]. The use of CADe systems in colonoscopy improves quality indicators (such as ADR, PDR, or AMR) by increasing lesion detection. In our study, the simultaneous detection of lesions by endoscopists and PolyDeep was lower than in a pilot study, where endoscopists and the CAD-ARTIPOD detected more lesions (84.1% vs. 94.7%) [13]. However, PolyDeep increased the percentage of lesions detected by 6.8%, which is higher than other studies with similar designs, that reported a 4% increase [12]. This increase in polyp detection can directly improve the quality of colonoscopy. Thus, using second observers (e.g., trainees or experienced nurses) or CADe systems (acting as standardized second observers) could potentially lead to improvements in ADR and PDR [11,22,24,25]. Although some meta-analyses highlight no statistically significant differences between the use of CADe systems and second observers in terms of ADR or adenomas per colonoscopy (APC) [22], evaluating the effect on quality indicators was beyond the scope of our study. Two ongoing randomized controlled trials (PolyDeep Advance 2, NCT05512793, and PolyDeep Advance 3, NCT05513261) aim to evaluate AMR and ADR.

Our study was conducted within the context of CRC screening colonoscopies. This setting may limit some of our findings, as screening endoscopists are highly experienced in detecting small lesions [26]. Despite the similarities with other studies, the sensitivity of endoscopists and PolyDeep in our study (94.2% and 91.5%) was lower than that observed in a pilot study, where expert endoscopists and the CAD-ARTIPOD system achieved sensitivities of 98.2% and 96.5%, respectively [13]. The CAD-ARTIPOD study, a multicentric trial using the same CADe system as the pilot study, reported a sensitivity of 94.6% for endoscopists and 96.1% for CAD-ARTIPOD using the endoscopists identification as reference standard and not considering the histological analysis [12,13,23]. In our study, the sensitivity of endoscopists (93.7%) and PolyDeep (90.5%) were comparable to those observed in the CAD-ARTIPOD study not considering histological analysis [12,13,23]. When we considered histology as the gold standard, the diagnostic performance (sensitivity) of endoscopists and CAD-ARTIPOD (94.9% vs. 96.0%) was superior to that of endoscopists and PolyDeep (94.2% vs. 91.5%) [12,23]. We evaluated the diagnostic performance of endoscopists and PolyDeep separately; however, in real clinical practice, both work in tandem (i.e., endoscopists assisted by CADe/x systems). Several studies compared the diagnostic performance of conventional colonoscopy to CADe-assisted colonoscopy [27,28], highlighting the importance of optimizing collaboration between endoscopists and AI systems to achieve maximum diagnostic performance [29].

We also evaluated the diagnostic performance of endoscopists and PolyDeep for the optical diagnosis of colorectal lesions during colonoscopy. In the clinical validation of the POLAR system, the optical diagnosis of diminutive colorectal lesions was assessed [30]. Although this was not the primary objective of our study, both endoscopists and PolyDeep performed optical diagnoses of diminutive lesions, with the mean polyp size in our study being 5.1 ± 4.9 mm [30]. In the POLAR study, lesions were classified as neoplastic or non-neoplastic, which did not allow a direct comparison of results since our endoscopists used the NICE classification. Nonetheless, the sensitivity achieved by endoscopists and the POLAR system (92.4% and 89.4%) was higher than that obtained in our study by endoscopists and PolyDeep (76.6% and 82.6%), although the results across both studies remain comparable for endoscopists and CADx systems [30]. In another study employing a similar setting, where endoscopists could not see the AI system’s output and a second observer managed the system for optical diagnosis of neoplastic and non-neoplastic lesions, sensitivities of 95.5% for less-experienced endoscopists and 90.8% for experienced endoscopists were reported [31]. Although we did not evaluate less-experienced endoscopists, the sensitivity of the experienced endoscopists in our study (76.6%) was lower than the 90.8% reported for experienced endoscopists and significantly inferior to the 95.5% observed for non-experts [31]. Meanwhile, the AI system in that study achieved a sensitivity of 89.7% for characterizing neoplastic and non-neoplastic lesions, while PolyDeep demonstrated a sensitivity of 91.5% in our study being higher and comparable [31]. These findings suggest that the performance of less-experienced endoscopists improves with the assistance of AI systems in optical diagnoses, likely driven by a sense of competition with the AI system. This evidence highlights the potential value of such systems as training tools for novice endoscopists. However, this competitive advantage did not appear to apply when the same technology is used by experienced endoscopists.

During the development of the study, we found that the optical diagnosis made by PolyDeep was sometimes suboptimal. In this regard, the percentage of neoplasia displayed by PolyDeep on-screen fluctuates in a wide range, making it difficult to assess which is the real optical diagnosis of the CADx system. For this reason, we explored different alternatives to first improve the optical diagnosis and second ease the optical diagnosis task for endoscopists in the next randomized controlled trials (i.e., PolyDeep advance 2 and PolyDeep advance 3). This improvement involves the inclusion of specific symbols (i.e., loupe, waves as two brackets and hourglass) within the bounding box that show the percentage of neoplasia for optical diagnosis. When the percentage is displayed and the loupe icon appears, it means that the endoscopist needs to be closer to the polyp to obtain a more stable diagnosis. The wave symbol is displayed when the polyp is in focus but there is excessive movement, preventing the CADx system from providing a stable diagnosis. Finally, the hourglass symbol informs the endoscopists to wait to allow the CADx system to process enough frames to provide a stable diagnosis.

Our study has several strengths: first, it was conducted in a real clinical setting, providing a valuable assessment of PolyDeep’s performance in real colonoscopy procedures. Second, it is innovative because the expert endoscopists were blinded to the CADe/x system’s output. Third, the diagnostic performance obtained for polyp detection is consistent with literature reports. Fourth, PolyDeep was built with a YOLOv3 detection network coupled with an object tracking algorithm, which allows linking the independent predictions done by YOLOv3 on each frame to specific polyps. By doing this, we can present the neoplasia probability of each polyp as the average of the probabilities calculated for that polyp during a pre-specified number of frames (50 frames in the final setting).

On the other hand, our study has several limitations: first, while the study was not fully blinded to the endoscopists, as the PolyDeep image was visible on a separate screen at the back of the endoscopists in the room. The screen was not in the endoscopists’ direct line of sight, but this could have introduced cognitive bias due to the presence of a second observer and the CADe/x system monitoring the procedure. Potentially this could lead to an artificial improvement in the endoscopists’ diagnostic performance. Second, the study did not evaluate the impact of PolyDeep on colonoscopy quality indicators or relevant clinical outcomes, such as post-colonoscopy CRC which is beyond of our scope. Third, despite updates made prior to clinical validation, PolyDeep exhibited a high false-positive rate, which led to minor disruptions in the colonoscopy procedure. However, previous studies with other CADe systems, such as GI Genius CADe v2, reported only 1–2 false alarms per patient, which did not affect withdrawal time [32]. In our study, we reported 1.95 false alarms per patient. Finally, the endoscopists classified lesions based on the NICE classification (i.e., they classified adenomas) and PolyDeep classified them based on the probability of detecting neoplasia. So, we could not compare the diagnostic performance for optical diagnosis of Polydeep and the endoscopists.

In summary, our study demonstrated similar results to those found in other studies. Both endoscopists and PolyDeep showed comparable diagnostic performance for the detection of colorectal lesions. For optical diagnosis, both agents showed moderate performance. These findings highlight the potential of PolyDeep as an effective support tool in colonoscopy procedures.