Clinical Evaluation of PolyDeep, a Computer Aided Detection system: A Multicenter Randomized Tandem Colonoscopy Trial

1. Introduction

Colorectal cancer (CRC) is the third most commonly diagnosed malignancy and the second leading cause of cancer-related mortality worldwide, with the highest incidence and mortality rates observed globally [1]. CRC typically develops over a number of years from precancerous lesions, such as adenomas and serrated polyps [2]. Organised CRC screening programmes are offered to individuals at average risk (generally those aged between 50 and 75 years) reducing both the incidence and mortality of the disease [2,3,4]. The primary screening modalities employed to detect precancerous lesions are the faecal immunochemical occult blood test (FIT) and colonoscopy [3,5]. In several countries, including Spain, FIT is the initial test offered; individuals with a positive result are subsequently referred for diagnostic colonoscopy, which remains the gold standard for lesion detection [3,5].

Colonoscopy remains the only test able of detecting precancerous lesions, namely adenomas and serrated lesions, within the colonic mucosa [3,5]. A proportion of these lesions present challenges owing to their small size (i.e., <5 mm) or morphology (e.g., flat or depressed), which can hinder their detection [6,7]. In addition, factors related to endoscopist performance such as fatigue, suboptimal mucosal exposure, or limited experience may further compromise lesion identification [6,7]. Collectively, these factors contribute to the risk of missed lesions during colonoscopy, which may subsequently lead to the development of post-colonoscopy colorectal cancer (PCCRC) [4]. The adenoma miss rate (AMR), defined as the proportion of adenomas detected during a second withdrawal relative to the total number of adenomas, ranges from 17% to 48%, with two meta-analyses estimating average miss rates of 22% and 26%, respectively [8,9].

In recent years, various computer-aided diagnosis (CAD) systems based on artificial intelligence (AI) and deep learning (DL) have been applied to medical image analysis, including radiography, mammography, and cardiovascular imaging [10,11,12]. In the context of colonoscopy, computer-aided detection (CADe) systems have been evaluated in randomized clinical trials employing a range of designs (i.e., comparisons between CADe-assisted and conventional colonoscopy, or tandem colonoscopy designs) and using diverse endpoints, including adenoma detection rate (ADR), polyp detection rate (PDR), AMR, and polyp miss rate (PMR) [13,14,15,16,17,18,19,20,21,22]. Although the use of these systems appears to improve diagnostic performance metrics, their role in routine clinical practice remains to be fully established [23].

PolyDeep is a computer-aided detection and characterization (CADe/x) system developed to identify colorectal lesions in real time during colonoscopy. The system integrates a neural detection network based on the YOLOv3 architecture and an object tracking algorithm, which maintains and follows all detections in each subsequent frame [24,25,26,27]. PolyDeep was initially validated in vitro and subsequently evaluated in a prospective study using a second-observer design [28,29]. In the present randomized controlled trial employing a tandem colonoscopy design, our objective was to determine whether the AMR in the PolyDeep-assisted colonoscopy group is superior to that of the conventional colonoscopy group.

2. Materials and Methods

2.1. Study Design

PolyDeep Advance 2 (NCT05512793) is a multicentre randomised controlled trial (RCT) with a tandem colonoscopy design. The study was conducted in the endoscopy units of three university hospitals in Spain: Hospital de Ourense, Hospital Álvaro Cunqueiro de Vigo, and Hospital de Montecelo de Pontevedra. The study was approved by the institutional review board (2022/067) in accordance with the Declaration of Helsinki and applicable guidelines for good clinical practice. Data from the three centres were verified and monitored within the electronic Case Report Form (eCRF) in the REDCap platform at the Galicia-Sur health research institute (https://redcap.tic1-iisgaliciasur.es/). The study was reported in accordance with the CONSORT-AI guideline for randomized trials.

2.2. Participants of the Study

We prospectively included patients aged between 40 and 79 years who underwent a colonoscopy after a positive FIT or a surveillance colonoscopy following the resection of advanced adenomas. Participants were excluded if they had a personal history of colorectal cancer or previous colonic resection, a hereditary syndrome predisposing to colorectal cancer, or serrated polyposis syndrome. Regarding colonoscopy quality, patients were excluded if they had an inadequate Boston Bowel Preparation Scale score (<2 in any segment or <6 overall) or if cecal intubation was not achieved. All participants provided written informed consent prior to inclusion in the study.

2.3. Randomization Process

We randomized the patients in a 1:1 ratio by a stratified block design based on the colonoscopy indication, such as positive FIT or surveillance endoscopy. We created blocks of two in all possible combinations (i.e., four combinations) and randomly assigned each block to positions within the randomization template. To minimize bias, we doubled the number of allocation slots (i.e., with 260 patients to be randomized, we generated 520 slots), ensuring a more balanced and unbiased randomization process. Finally, the randomization template was uploaded to the eCRF by the study coordinator. The endoscopists performed the randomization process of the participants before the colonoscopy, to know which is the allocation group and what tandem colonoscopy they should do. However, participants were not aware of the randomization allocation.

2.4. Clinical Setting

We conducted the study in a conventional endoscopy room using a standard endoscopy tower with high definition colonoscopes using the model EXERA III CV 190 or higher. We integrated the PolyDeep system into the setup, connecting the CADe/x system to both the endoscopy tower and the monitor displaying the colonoscopy image. The same monitor showed both the colonoscopy image and the PolyDeep system’s output, including overlays highlighting detected polyps. Upon the patient’s arrival in the endoscopy room, the endoscopists explained the study and obtained written informed consent. They then randomized the participants in the eCRF, assigning them to either the conventional group (conventional colonoscopy followed by assisted colonoscopy) or the PolyDeep group (assisted colonoscopy followed by conventional colonoscopy). All endoscopists were experts, meaning they are participants of the CRC screening program with more than 300 colonoscopies, one year of experience and with a rigorous quality control based on key quality indicators. They performed a back-to-back colonoscopy with two withdrawal phases according to the assigned randomization sequence. During each withdrawal phase, the endoscopists measured withdrawal time using the timer in the colonoscope control device. Lesions were counted only in one of the withdrawal phases and resected upon identification. Throughout the procedure, the nursing and auxiliary team recorded polyp data on a log sheet, including withdrawal phase, morphology, location, size, and both the endoscopists’ and PolyDeep’s optical diagnosis.

We sent all identified and retrieved lesions for histopathological evaluation, which served as the gold standard for analysis. If a lesion could not be resected during the procedure, the endoscopists documented its characteristics and obtained a biopsy. In the eCRF, we recorded participant demographic data (age, sex, and colonoscopy indication), colonoscopy details (Boston Bowel Preparation Scale score, withdrawal time, and cecal intubation), and polyp-specific information (withdrawal phase of identification, morphology, size, location, optical diagnosis, and histological evaluation)

2.5. Endpoints

The primary endpoint was to assess and compare the AMR between the conventional group and the PolyDeep group. Secondary endpoints included the evaluation and comparison of the PMR, serrated lesion miss rate (SLMR), advanced adenoma miss rate (AAMR), advanced serrated lesion miss rate (ASLMR) and advanced polyp miss rate (APMR) between both groups.

2.6. Sample Size

We calculated the sample size based on the assumption that the conventional group had a 30% AMR and the PolyDeep group a 20%, considering an alpha error of 5% and a beta error of 20% [14]. We should include 294 adenomas and 118 patients per group, assuming an average of 2.5 lesions per colonoscopy. Attending a drop-out rate of 10% for incomplete colonoscopies, we should include 260 patients.

2.7. Statistical Analysis

We conducted a descriptive analysis of the study population, including demographics, colonoscopy procedures, and identified lesions. Quantitative variables are reported as mean and standard deviation, while qualitative variables expressed as frequency and percentages. We compared both groups using the t-Student and chi-square tests . To evaluate and compare miss rates of different types of lesions, we built 2×2 confusion matrices. For the primary endpoint, we calculated the AMR as the ratio between the number of adenomas detected during the second withdrawal and the total number of adenomas detected in the colonoscopy (i.e., first and second withdrawals). For secondary endpoints, we determined the PMR, SLMR, AAMR, ASLMR and APMR using similar calculations, dividing the number of lesions detected during the second withdrawal by the total number of detected lesions. Additionally, we performed subgroup analyses based on indication, lesion size, location, and morphology. All miss rate metrics are presented as percentages, with a significant difference between both groups established in p < 0.05 for the t-Student and chi-square tests. We performed all statistical analyses using R (version 4.4.1, The R Foundation for Statistical Computing, Institute for Statistics and Mathematics, Vienna, Austria).

3. Results

3.1. Population Description

Between May and November 2023, we randomly allocated 260 patients, with 130 (50%) assigned to each group. After excluding 20 patients (13 from the conventional group and 7 from the PolyDeep group), we analysed data from 240 participants, with 48.8% in the conventional group and 51.2% in the PolyDeep group (Figure 1). Table 1 presents a description and comparison of the baseline characteristics of both groups. Most participants included by the endoscopists were male and had a positive FIT result. We did not find statistically significant differences between the conventional group and the PolyDeep group (Table 1).

We detected 674 lesions of which 612 (90.8%) were adenomas and serrated lesions, while 62 (9.2%) belonged to other categories (i.e., inflammatory polyps, harmatomatous polyps or lesions without histology). These lesions were distributed as 445 adenomas (66.0%), 167 serrated lesions (24.8%), 37 other polyps (5.2%) and 27 lesions without histology (4.0%) (Table 2). Table 2 shows the overall number of lesions detected in each group.

Related to the characteristics of the lesions identified, most of them were diminutive polyps located in the proximal colon with more detections in this category made by the conventional group than by the PolyDeep group (Table 2). The mean size of polyps in both groups was inferior to 5 mm (i.e., conventional group: 4.5 ± 4.7 vs PolyDeep group: 4.9 ± 4.7). The mean number of adenomas detected per patient in the conventional group was 2.9 ± 2.5, while in the PolyDeep group 2.8 ± 2.2.

3.2. Diagnostic Performance: Adenoma Miss Rate, Polyp Miss Rate, Serrated Lesion Miss Rate

Table 2 shows the distribution of lesions detected in the first and in the second withdrawal to determine the lesion miss rates. We did not find statistically significant differences for AMR between the conventional group and the PolyDeep group (18.1% vs 21.3%, p = 0.5). Similarly, we did not find statistically significant differences between both groups for PMR (20.3% vs 21.8%, p = 0.7) and SLMR (25.6% vs 23.4%, p = 0.9).

3.3. Sub-Analysis by Size, Location and Advanced Lesions

In the size-based analysis, we did not find statistically significant differences in the miss rates of diminutive polyps smaller than 5 mm (24.1% vs 25.5%, p = 0.8), in small polyps with less than 10 mm (22.8% vs 23.4%, p = 0.9), and large polyps which are equal or larger than 5 mm (11.4% vs 15.2%, p = 0.6) (Table 2).

With respect to the polyp miss rates by location (i.e., proximal and distal colon; Table 2) we did not observe statistically significant differences between conventional and PolyDeep groups (proximal colon: 18.5% vs 19.3%, p = 0.8; distal colon: 22.8% vs 24.7%, p = 0.8). Finally, there were no significant differences between both groups in advanced polyps (11.3% vs 7.3%, p = 0.5), advanced adenomas (4.8% vs 5.1%, p = 1.0) and advanced serrated lesions (35.7% vs 13.6%, p = 0.2).

3.4. Sub-Analysis by Colonoscopy Indication

Table 3 shows the distribution of lesions detected by screening indication. In this case, we did not find statistically significant differences for AMR (14.9% vs 20.4%, p = 0.2) between the conventional and PolyDeep groups. We also did not find statistically significant differences for SLMR (29.4% vs 25.0%, p = 0.7) and PMR (18.6% vs 21.6%, p = 0.5). For the miss rates by advanced lesions, location, and size, we did not find statistically significant differences between both groups. On the other hand, for surveillance colonoscopy, we did not find significant differences between both groups for AMR, SLMR, and PMR. As well as we did not find differences for advanced lesions (33.3% vs 0.0%, p = 0.2), location (proximal: 18.8% vs 26.3%, p= 0.5 distal: 34.4% vs 17.2%, p = 0.2), and size (<5 mm: 28.2% vs 26.9%, p = 1.0; ≥5 mm: 0.0% vs 6.7%, p = 0.5).

4. Discussion

In this RCT with a tandem colonoscopy design, the use of PolyDeep did not reduce the AMR in the context of screening colonoscopies performed by expert endoscopists. Moreover, no significant differences were observed between groups with respect to indication, lesion type, or lesion location. In recent years, a substantial number of CADe systems have been integrated into real-time colonoscopy procedures, consistently demonstrating reductions in the AMR and increases in the ADR [13,18,30,31]. The adoption of CADe systems appears to enhance colonoscopy quality reflected by key performance indicators, such as ADR.

Our study employed a robust design, utilizing a back-to-back (i.e., tandem colonoscopy) approach with the AMR as the primary endpoint. One of the strengths of this design is that tandem colonoscopy allows for reliable AMR assessment with a relatively small sample size. However, this design also presents several potential limitations. First, AMR is not a standard quality indicator routinely measured in clinical practice, which may limit the generalizability of our findings. Second, there is a potential for bias among endoscopists due to the awareness of a second withdrawal opportunity; if a lesion was missed during the first withdrawal, they had a second chance to detect it, possibly altering their performance. Additionally, endoscopists fatigue may have affected outcomes, as double colonoscopies were performed within a standard clinical workload. This was particularly relevant towards the end of the day, when cumulative fatigue may have increased the likelihood of missed lesions. Furthermore, the use of a CADe system such as PolyDeep may have introduced an unintended sense of competition with the technology, potentially leading endoscopists to detect fewer lesions when the system was active.

A metanalysis of four tandem colonoscopy studies reported a 65% reduction in AMR (odds ratio 0.35, 95% CI: 0.25–0.49), along with a 78% reduction in the SLMR [13]. Another metanalysis reported comparable reductions in PMR and AMR, with absolute risk differences of 19% and 17.5%, respectively [18]. Further, recent evidence supports these findings, showing a 55% reduction in SLMR with CADe-assisted colonoscopy without reaching statistical significance [31]. However, our study did not identify statistically significant differences in AMR between PolyDeep-assisted and conventional colonoscopy. A non-significant reduction in SLMR was observed, consistent with previous meta-analyses. Additionally, the difference in withdrawal time was minimal, with the CADe-assisted colonoscopy in our study taking only seven seconds longer, comparable to findings from another study, which reported a nine-second increase [31].

One possible explanation for our negative results is the high expertise of the endoscopists who participated in the trial. As an example, the AMR of the conventional group was clearly inferior to the AMR reported in literature [14]. In this sense, in a recently published RCT, CADe did not increase the advanced adenoma detection rate in the context of FIT-based CRC screening programs [32]. In fact, the screening endoscopists involved in our study underwent periodic evaluation using the ADR (i.e., most of the endoscopists had an ADR superior to the 60%) as a measure of colonoscopy quality [33]. As a matter of fact, the latest guideline from the European Society of Gastrointestinal Endoscopy reported only a weak recommendation for the routine use of CADe, due to limited supporting evidence and ongoing concerns regarding its implementation particularly the need for further studies on cost-effectiveness and the potential drawbacks of human-AI interaction [23].

PolyDeep integrates a YOLOv3 neural detection network with an object tracking algorithm, trained on polyp detection with polyp and not-polyp images [24,25,26,28]. A potential limitation that could affect detection performance may be the algorithm itself. At the start of this clinical trial, a more recent version of the YOLO algorithm (i.e., YOLOv8) was already available, offering an improved lesion detection performance [34]. In an ex vivo study, YOLOv8 demonstrated high diagnostic performance for polyp detection in polyp images, with a sensitivity of 91.7 % and an F1 score of 92.4 % [35]. However, no clinical validation or RCTs have yet evaluated YOLOv8 in real-time colonoscopy procedures to determine whether it would outperform PolyDeep in detection tasks. Furthermore, as the clinical validation of PolyDeep was already underway, modifying the neural network architecture was not appropriate, as it would have compromised comparability with the results of the preceding observational detection study [29].

To conclude, the first PolyDeep-assisted colonoscopy did not reduce the AMR compared to the first conventional colonoscopy, indicating that endoscopists missed a relatively small number of lesions. Similarly, in the subgroup analyses by indication and polyp type, no statistically significant differences were observed between both groups.