Dataset and AI Workflow for Deep Learning Image Classification of Ulcerative Colitis and Colorectal Cancer

1. Summary

1.1. Background of Ulcerative Colitis

Inflammatory bowel disease (IBD) is a chronic inflammatory condition of the gastrointestinal tract with systemic repercussions that is characterized by relapsing and remitting episodes of inflammation [1,2]. IBD includes two types: ulcerative colitis, which affects the colon, and Chron disease, which can affect any part of the gastrointestinal tract from the mouth until the perianal area [1,2,3,4,5,6,7,8,9].

The causes and pathogenesis of inflammatory bowel disease are unclear. However, it appears to be a combination of factors from the environment, including the microbiome [10,11], genetic susceptibility [12,13], and immune system (both systemic and gut) [13,14,15,16]. The prevalence of inflammatory bowel disease has increased globally in recent decades, particularly in industrialized countries [17,18,19,20,21]. Clinical risk factors include smoking [22], low physical activity, low fiber intake, high fats, and low vitamin D [23], sleep deprivation, previous acute gastroenteritis, antibiotic use, and early life exposures [24].

In normal conditions and in healthy status, the intestinal barrier is kept thanks to the mucus layer and epithelial cells that create bonds using the tight junctions. This barrier is supported by the presence of IgA and several antimicrobial factors. The immune response is initiated by dendritic cells that acquire, process, and present the antigens to B and T cells (lymphocytes) [25]. The cause of ulcerative colitis is still not completely understood. Ulcerative colitis is associated with a damage of the barrier of the mucosa, with an alteration of the microflora and the generation of an uncontrolled immune reaction. Several subsets of CD4-positive T cell are believed to participate in the pathogenesis of ulcerative colitis including Th1, Th2, Th9, Th17, Th22, TFh cells, and Tregs [26,27].Th9 are associated with continuous apoptosis of the enterocytes and inhibit mucosal healing [25,28]. IL-13 and NK/T cells are also involved in the epithelial injury [25]. Innate immune cells contribute to the cytokine production and continuous inflammation [14,15,29]. The injury of the mucosa is associated with dysbiosis [30], which is defined by an alteration of the composition of the mucosa, with changes of the diversity, increased potentially pathogenic bacteria, and reduced beneficial bacteria [31]. Figure 1 and Figure 2 show an example of ulcerative colitis and immunohistochemistry of the cells of the microenvironment involved in the pathogenesis.

The disease activity of ulcerative colitis is being recorded to properly treat patients and perform clinical trials. Usually, the severity is classified as mild, moderate, and severe. The available classifications include the Mayo [32], Montreal [33], Truelove, and Witts classifications [34]. The grade of mucosal inflammation can be assessed using endoscopy to record the degree of mucosal ulceration and the extent of disease. This can be graded using the Mayo score [32] and Baron score. The degree of histological severity can be evaluated using the Geboes score [35].

Microscopic (histologic) evaluation of ulcerative colitis shows active chronic colitis in patients with untreated disease. Disease activity is defined by neutrophil infiltration of the lamina propria, cryptitis, crypt abscess, and ulceration. The inflammation is limited to the mucosa and submucosa. Compared with Crohn disease, this condition is associated with a lack of transmural inflammation, granuloma, and fissuring ulcers. Dysplasia may be present in long-term disease [36,37,38].

Disease activity can be evaluated using various parameters. Inactive disease lacks neutrophils. If the activity affects less than 50% of the mucosa, it is called mild. When it is > 50% and there are crypt abscesses, it is called moderate. Finally, severe activity is characterized by surface ulcerations and erosion [39].

The first-line therapy for the induction and maintenance of remission of mild to moderate UC is 5-aminosalicylic acid [40]. The initial therapy for ulcerative colitis is topical mesalamine, which is available as a suppository or enema, and it is recommended in cases of ulcerative proctitis or proctosigmoiditis. If topical mesalamine is not tolerated, an alternative therapy is topical glucocorticoids (i.e. hydrocortisone). A combination of 5-ASA and rectal mesalamine may be used for left-sided or extensive colitis. The 5-ASA formulations include mesalamine, sulfasalazine, and diazo-bonded 5-ASA. Patients who do not respond may subsequently be treated with glucocorticoids such as budesonide, which may escalate to prednisone or biological agents (anti-TNF, anti-integrin, anti-IL12/23, S1P modulators, JAK inhibitors) [40,41,42,43]. Patients who do not respond to oral glucocorticoids may require intravenous glucocorticoids. In some cases, surgery is performed [40,41,42,43]. Fecal microbiota transplantation is also available [44].

1.2. Background of Colorectal Cancer

Colorectal cancer (CRC) is a common disease characterized by environmental and genetic factors. It is the third most frequently diagnosed cancer worldwide [45]. There are several factors associated with early onset of CRC, including hereditary syndromes (familial adenomatous polyposis and Lynch syndrome), metabolic dysregulation, history of CRC in first-degree relative; alcohol consumption, lack of regular use of nonsteroidal anti-inflammatory drugs (NSAIDs), and vitamin D intake. Other known risk factors of CRC include inflammatory bowel disease (ulcerative colitis and Crohn disease), abdominopelvic radiation, obesity, diabetes mellitus, insulin resistance, red and processed meat, and tobacco [46,47,48,49]. The diagnosis is usually made by colonoscopy, and treatment includes surgical resection, adjuvant chemotherapy, and postoperative radiation therapy [46]. Adenocarcinoma is the most common histological subtype of CRC. Adenocarcinoma is a glandular neoplasm. Most CRC cases are moderately differentiated with simple, complex, or slightly irregular tubules, and loss of nuclear polarity. The glands are often cribriform, with necrosis, inflammatory cells, and marked desmoplasia [50,51,52].

1.3. Background of Computer Vision for Deep Learning Image Classification

Image processing involves algorithms and workflows used for image processing, analysis, visualization, and algorithm development. Computer vision workflows [53] with deep learning include several types of functions, such as image classification, object detection, and instance segmentation, automated visual inspection, semantic segmentation, and video classification [54,55,56].

A pretrained neural network that has already learned how to extract the characteristics of natural images can be used as a starting point to handle new images. Using a pretrained image classification network [57], the learning time is shorter, and training is usually easier. The transfer learning process takes layers from an already trained neural network and calibrates the parameters on a new dataset [54,55,56,58,59,60,61,62,63,64,65]. The analysis includes a series of steps, including preprocessing the data, import pre-trained networks from platforms such as TensorFlow 2 [66], TensorFlow-Keras [67], Pythorch [68], and ONNX [69], building network, selecting training options, improving the network performance by tuning hyperparameters, visualizing and verifying network behavior during and after training, and exporting the network to other platforms if necessary[54,55,56,58,59,60,61,62,63,64,65].

The most important features of a neural network are accuracy, speed, and size. A good neural network is characterized by being fast and having good performance (i.e. accurate). Neural networks that are accurate in ImageNet can be used to classify other images using transfer learning or feature extraction. Among the various neural networks, the best examples are GoogLeNet, ResNet-18, NobileNet, ResNet-50, ResNet-101, and Inception-v3 [56]. Figure 4 shows the general design of a CNN, and Figure 5 the original RestNet-18 architecture [70].

1.4. Dataset and Research Project Description

Ulcerative colitis is a chronic inflammatory bowel disease associated with a high risk of colorectal cancer. This study used convolutional neural networks and computer vision to classify histological images of ulcerative colitis, colorectal cancer (adenocarcinoma), and colon control.

The series included 35 patients with of ulcerative colitis, 18 with colorectal cancer, and 21 with colon control. Hematoxylin and eosin (H&E) glass slides were converted into high-resolution digital data by high-speed scanning. The whole-tissue slides were split into image patches of 224×224 pixels at 200× magnification and 150 dpi, and the ResNet-18 network was retrained to classify the 3 types of diagnosis. This transfer learning experiment also used other pretrained CNNs for performance comparison, and the gradient-weighted class activation mapping (Grad-CAM) heatmap technique was used to understand the classification decisions.

Additionally, immunohistochemical analysis of two new immuno-oncology markers, LAIR1 and TOX2 were analyzed in ulcerative colitis to differentiate between mesalazine-responsive and steroid-requiring patients. LAIR1 an inhibitory receptor that plays a constitutive negative regulatory role in the cytolytic function of natural killer (NK) cells, B-cells and T-cells [71,72,73]. TOX2 is a new marker similar to PD-1 in the co-inhibitory pathway [71,74].

Statistical analyses showed that steroid-requiring ulcerative colitis was characterized by higher endoscopic Baron and histologic Geboes scores and LAIR1 expression, but lower TOX2 expression in isolated lymphoid follicles. The CNN managed to classify the 3 diagnoses with >99% accuracy and the Grad-CAM confirmed which parts of the images were relevant for the classification. In conclusion, the study showed that CNNs are essential tools for deep learning image recognition.

These results were published as preprints [75].

This paper is the companion manuscript of the recently published article “Ulcerative Colitis, LAIR1 and TOX2 Expression, and Colorectal Cancer Deep Learning Image Classification Using Convolutional Neural Networks” published in Cancers 2024, 16, 4230. https://doi.org/10.3390/cancers16244230.

2. Data Description

The dataset contains image patches of ulcerative colitis, colorectal cancer, and colon control. The image patches were split from whole-tissue images and had a size of 224 × 224. The original magnification of the images was 200× and a dpi of 150. The image patches are anonymized. After splitting the images, the patches were filtered. The criteria were as follows: (1) image patches of only 243×243 size; (2) image patches of more than 5-31 KB that contain tissue at least 20-30% of viable tissue; (3) image patches with diagnostic areas; (4) image patches without artifacts, including broken tissue, folded areas, incorrectly stained tissue, and smashed/crushed tissue. The steps 3 and 4 were manually performed by the doctor specialist in pathology (MD PhD).

The investigations were carried out following the rules of the Declaration of Helsinki of 1975 (website: https://www.wma.net/policies-post/wma-declaration-of-helsinki/; last accessed on December 12, 2024), revised in 2008. Approval from an ethics committee was obtained before undertaking the research (protocol code IRB14R-080, IRB20-156, and 13R-119).

3. Methods

3.1. Hematoxylin and Eosin (H&E) Staining

H&E staining is widely used in histological and cytological applications, both in fixed paraffin-embedded and frozen tissue sections. Hematoxylin produces an intense blue staining of the nuclei. Eosin stains the cytoplasm, collagen, muscle, and erythrocytes in light pink/rose. Staining was performed using a Leica ST5010-CV5030 integrated stainer workstation (Leica Biosystems K.K., Shinjuku, Tokyo, 169-0075, Japan).

3.2. Score Evaluation

The endoscopic scores for chronic inflammatory bowel disease were evaluated using the Baron Score, which classifies mucosal changes into 3 grades. Ulcerative colitis histological assessment was performed using the Geboes score. Appendix Table A1 and Table A2 present the Baron and Geboes scores, respectively.

3.2. Immunohistochemistry

The immunohistochemistry of LAIR1 and TOX2 was performed using a Bond-Max fully automated immunohistochemistry and in situ hybridization staining system following the manufacturer’s instructions (Leica Biosystems K.K.). Visualization of the primary antibody bound to the tissue sections was performed using BOND Polymer Refine Detection (DS9800, Leica Biosystems K.K.). The BOND staining mode was single, i.e. the application of a single marker and chromogen to a single slide. The protocol sequence was the following: preparation (removal of wax), heat-induced epitope retrieval (HIER), probe application, probe removal, and staining. The staining protocol included the following steps: marker, post primary, polymer, mixed DAB, BOND-PRIME hematoxylin. Coverslipping was achieved using a Leica CV5030 fully automated glass coverslipper (Leica Biosystems K.K.).

The primary antibody targeted the leukocyte-associated immunoglobulin-like receptor (LAIR1/CD305) created by the Monoclonal Antibodies Core Unit, located at the Spanish National Cancer Research Center (CNIO: Centro Nacional de Investigaciones Oncologicas; C/ Melchor Fernandez Almagro, 3, E-28029 Madrid, Spain). LAIR1 is a rat monoclonal antibody, clone JAVI82A, antigen used: RBL-1-LAIR1-MYCDDK transfected cells and last booster with LAIR1 recombinant (Gln22-His163, with a C-terminal 6-His tag); isotype IgG2a; reactivity, human; localization, membrane.

The primary antibody TOX2 targeted the TOX High Mobility Group Box Family Member 2 and was also developed by CNIO. Properties: clone name TOM924D, rat monoclonal, IgG2b K, antigen HIS-SUMO-hTOX2-Strep-tag2 full-length protein, human reactivity, nuclear localization.

Rabbit Anti-Rat IgG Antibody (H+L), Mouse Adsorbed, Unconjugated (#AI-4001-.5, Vector Laboratories, Inc., Newark, CA 94560, United States) was used as a linker between primary antibodies and BOND Polymer Refine Detection.

The immunohistochemistry of both antibodies was first tested in reactive lymphoid tissue (tonsils) and small and large intestine tissue controls (Appendix Figure 1). Four examples of biopsies of colon of ulcerative colitis stained with H&E, TOX2 and LAIR1 and high infiltration of positive cells are show in Appendix Figures A1, B1, and C1.

3.3. Whole-Slide Imaging

Whole-slide imaging was performed using a Hamamatsu NanoZoomer S360 scanner (Hamamatsu Photonics K.K., Hamamatsu City, 431-3196, Japan) that rapidly scanned glass slides to convert them to digital data, NZAcquire 1.2.0 software (Hamamatsu Photonics K.K.), and a Dell Precision 5820 Tower equipped with an Intel(R) Xeon(R) W-2135 CPU @ 3.70 GHz 3.70 GHz, 32.0 GB RAM, 64 bits workstation system. The operating system was Windows 10 Pro for Workstations (version 1803, build 17134.1).

The acquisition method was performed following the manufacturer’s instructions. The scan was batch scan-type, at 400× magnification, and single z stack. The number of focus points was defined automatically using NZAcquire software (Figure 6).

3.4. Digital Image Quantification

Conventional immunohistochemical analysis was performed using digital image quantification with Fiji software, as previously described [76,77,78,79]. In summary, quantification was performed in the blue stack, min and max thresholds were set, pixels were measured, and percentages were calculated.

3.5. Image Classification Using CNNs

A convolutional neural network (CNN) was designed based on transfer learning from ResNet-18; and trained to classify the 3 types of image patches; ulcerative colitis (n= 9,281), colon control (n = 12,246), and colorectal cancer (n = 63,725). The CNN was designed using MATLAB (R2023b, update 9, 23.2.0.2668659). The CNN was also trained to differentiate between mesalazine-responsive and steroid-requiring ulcerative colitis based on H&E, LAIR1, and TOX2 staining (Figure 7).

The data were partitioned into a training set (70% of the image patches) to train the network, a validation set (10%) to test the performance of the network during training, and a test set (20%) as a holdout (new data) to test the performance on new data.

The arrange of the images was randomized to guarantee that the CNN learned the classes at a more even rate. Transfer learning, which is the procedure of re-using and adjusting a pre-trained network, was performed on ResNet-18. For this purpose, the fully connected and classification layers of the ResNet-18 were deleted and replaced with new layers. These new layers had an output size of 2. The training did not used augmentation technique. To avoid overfitting, the initial learning rate was set to 0.001. The maximum numbers of epochs was five [75].

Data normalization was applied to the input images. The detailed description of the data normalization is shown in Appendix Table A3 [80,81].

The trained network was characterized by 71 layers. The first layer was the “ImageInputLayer”, and the last one the “ClassificationOutputLayer” (Figure 7). Parameters are shown in Table 1.

The code that was used is shown in Appendix Table A4

In the analysis setup, the image patches of the 3 diagnoses were pooled in 3 different folders. Later, the content of each folder was split into training, validation, and test sets. As a result, no image patches were repeated in different folders. Nonetheless, some researchers think that this strategy could create an information leak. Therefore, an additional and independent test set of 10 cases of colorectal cancer (adenocarcinoma) were classified to confirm the performance of the trained CNN. In this analysis, each patient was analyzed independently [75].

Of note, other types of CNNs were tested in this study. LAIR1 and TOX2 biomarkers were included into the CNN training in an independent analysis from the H&E staining. A differentiation between steroid-requiring (SR) and mesalazine-responsive ulcerative colitis using LAIR1 and TOX2 immunohistochemistry.

The performance parameters were the following: accuracy, precision, recall, F1-Score, specificity, and false positive rate.

The performance of ResNet-18 was compared with the one of other CNNs under same experimental conditions, including DenseNet-201, ResNet-50, Inception-v3, ResNet-101, ShuffleNet, MobileNet-v2, NasNet-Large, GoogLeNet-Places365, VGG-19, EfficientNet-b0, AlexNet, Xception, VGG-16, GoogLeNet, and NasNet-Mobile.

3.6. Computational Requirements

All analyses were performed using a desktop computer equipped with an AMD Ryzen 9 7950X CPU, 128 Gb of RAM (Crucial Desktop DDR5-4800 UDIMM 1.1V CL40, CT2K32G48C40U5 x2), and an Nvidia GeForce RTX 4090 graphics card (ASUS ROG Strix GeForce RTX® 4090 OC Edition 24GB GDDR6X).