Dataset and AI Workflow for Deep Learning Image Classification of Ulcerative Colitis and Colorectal Cancer

1. Summary

1.1. Background of Ulcerative Colitis

Inflammatory bowel disease (IBD) is a chronic inflammatory condition of the gastrointestinal tract with systemic repercussions that is characterized by relapsing and remitting episodes of inflammation [1,2]. IBD includes 2 types: ulcerative colitis, which affects the colon, and Chron disease, which can affect any part of the gastrointestinal tract from the mouth until the perianal area [1,2,3,4,5,6,7,8,9].

The causes and pathogenesis of IBD are unclear. However, it appears to be a combination of factors from the environment, including the microbiome [10,11], genetic susceptibility [12,13], and immune system (both systemic and gut) [13,14,15,16]. The prevalence of IBD has increased globally in recent decades, particularly in industrialized countries [17,18,19,20,21]. Clinical risk factors include smoking [22], low physical activity, low fiber intake, high fats, and low vitamin D [23], sleep deprivation, previous acute gastroenteritis, antibiotic use, and early life exposures [24].

In normal and healthy conditions, the intestinal barrier is maintained by the mucus layer and epithelial cells, which create bonds using tight junctions. This barrier is supported by the presence of IgA and several antimicrobial factors. The immune response is initiated by dendritic cells, which acquire, process, and present antigens to B and T lymphocytes [25]. The cause of ulcerative colitis is still not completely understood. Ulcerative colitis is associated with mucosal barrier damage, microflora alteration, and an uncontrolled immune reaction. Several subsets of CD4-positive T cells are believed to participate in the pathogenesis of UC, including Th1, Th2, Th9, Th17, Th22, TFh lymphocytes, and Tregs [26,27].Th9 are associated with enterocyte apoptosis and inhibit mucosal healing [25,28]. IL-13 and NK/T cells are also involved in epithelial injury [25]. Innate immune cells contribute to cytokine production and continuous inflammation [14,15,29]. Injury of the mucosa is associated with dysbiosis [30], which is defined by an alteration of the composition of the mucosa, with changes of the diversity, increased potentially pathogenic bacteria, and reduced beneficial bacteria [31]. Figure 1 and Figure 2 show examples of ulcerative colitis and IHC of the cells of the microenvironment involved in the pathogenesis.

The disease activity of ulcerative colitis is being recorded to properly treat patients and perform clinical trials. Usually, the severity is classified as mild, moderate, or severe. The available classifications include the Mayo [32], Montreal [33], Truelove, and Witts classifications [34]. The grade of mucosal inflammation can be assessed using endoscopy to record the degree of mucosal ulceration and the extent of disease. This can be graded using the Mayo score [32] and Baron score. The degree of histological severity can be evaluated using the Geboes score [35].

Microscopic (histologic) evaluation of ulcerative colitis shows active chronic colitis in patients with untreated disease. Disease activity is defined by neutrophil infiltration of the lamina propria, cryptitis, crypt abscess, and ulceration. The inflammation is limited to the mucosa and submucosa. Compared with Crohn disease, this condition is associated with a lack of transmural inflammation, granuloma, and fissuring ulcers. Dysplasia may be present in patients with long-term disease [36,37,38].

Disease activity can be evaluated using various parameters. Inactive disease lacks neutrophils. If the activity affects less than 50% of the mucosa, it is called mild. When it is >50% and there are crypt abscesses, it is called moderate. Finally, severe activity is characterized by surface ulcerations and erosion [39].

The first-line therapy for the induction and maintenance of remission of mild to moderate UC is 5-aminosalicylic acid [40]. The initial therapy for ulcerative colitis is topical mesalamine, which is available as a suppository or enema, and it is recommended for ulcerative proctitis or proctosigmoiditis. If topical mesalamine is not tolerated, an alternative therapy is topical glucocorticoids (i.e. hydrocortisone). A combination of 5-ASA and rectal mesalamine may be used for left-sided or extensive colitis. The 5-ASA formulations include mesalamine, sulfasalazine, and diazo-bonded 5-ASA. Patients who do not respond may subsequently be treated with glucocorticoids such as budesonide, which may escalate to prednisone or biological agents (anti-TNF, anti-integrin, anti-IL12/23, S1P modulators, JAK inhibitors) [40,41,42,43]. Patients who do not respond to oral glucocorticoids may require intravenous glucocorticoids. In some cases, surgery is performed [40,41,42,43]. Fecal microbiota transplantation is also available [44].

1.2. Background of Colorectal Cancer

Colorectal cancer (CRC) is a common disease characterized by environmental and genetic factors. It is the third most frequently diagnosed cancer worldwide [45]. There are several factors associated with early onset of CRC, including hereditary syndromes (familial adenomatous polyposis and Lynch syndrome), metabolic dysregulation, history of CRC in first-degree relative; alcohol consumption, lack of regular use of nonsteroidal anti-inflammatory drugs (NSAIDs), and vitamin D intake. Other known risk factors of CRC include inflammatory bowel disease (ulcerative colitis and Crohn disease), abdominopelvic radiation, obesity, diabetes mellitus, insulin resistance, red and processed meat, and tobacco [46,47,48,49]. The diagnosis is usually made by colonoscopy, and treatment includes surgical resection, adjuvant chemotherapy, and postoperative radiation therapy [46]. Adenocarcinoma is the most common histological subtype of CRC. Adenocarcinoma is a glandular neoplasm. Most CRC cases are moderately differentiated with simple, complex, or slightly irregular tubules and loss of nuclear polarity. The glands are often cribriform, with necrosis, inflammatory cells, and marked desmoplasia [50,51,52] (Figure 3).

1.3. Background of Computer Vision for Deep Learning Image Classification

Image processing involves the algorithms and workflows of image processing, analysis, visualization, and algorithm development. Computer vision workflows [53] with deep learning include several types of functions, such as image classification, object detection, and instance segmentation, automated visual inspection, semantic segmentation, and video classification [54,55,56].

A pretrained neural network that has already learned how to extract the characteristics of natural images can be used as a starting point to handle new images. Using a pretrained image classification network [57], the learning time is shorter, and training is usually easier. The transfer learning process takes layers from an already trained neural network and calibrates the parameters on a new dataset [54,55,56,58,59,60,61,62,63,64,65]. The analysis includes a series of steps, including preprocessing the data, import pretrained networks from platforms such as TensorFlow 2 [66], TensorFlow-Keras [67], Pythorch [68], and ONNX [69], building the network, selecting training options, improving the network performance by tuning hyperparameters, visualizing and verifying network behavior during and after training, and exporting the network to other platforms if necessary[54,55,56,58,59,60,61,62,63,64,65].

The most important features of a neural network are accuracy, speed, and size. A good neural network is characterized by being fast and having good performance (i.e. accurate). Neural networks that are accurate in ImageNet can be used to classify other images using transfer learning or feature extraction. The best examples of neural networks, the best examples are GoogLeNet, ResNet-18, NobileNet, ResNet-50, ResNet-101, and Inception-v3 [56]. Figure 4 shows the general design of the CNN, and Figure 5 shows the original RestNet-18 architecture [70].

1.4. Dataset and Research Project Description

Ulcerative colitis is a chronic inflammatory bowel disease associated with a high risk of colorectal cancer. This study used convolutional neural networks and computer vision to classify histological images of ulcerative colitis, colorectal cancer (adenocarcinoma), and colon control.

The series included 35 patients with ulcerative colitis, 18 with colorectal cancer, and 21 with colon control. Hematoxylin and eosin (H&E) glass slides were converted into high-resolution digital data by high-speed scanning. The whole-tissue slides were split into image patches of 224×224 pixels at 200× magnification and 150 dpi, and the ResNet-18 network was retrained to classify the 3 types of diagnosis. This transfer learning experiment also used other pretrained CNNs for performance comparison, and the gradient-weighted class activation mapping (Grad-CAM) heatmap technique was used to understand the classification decisions.

Additionally, immunohistochemical analysis of 2 new immuno-oncology markers, LAIR1 and TOX2 were analyzed in ulcerative colitis to differentiate between mesalazine-responsive and steroid-requiring patients. LAIR1 an inhibitory receptor that plays a constitutive negative regulatory role in the cytolytic function of natural killer (NK) cells, B lymphocytes and T lymphocytes [71,72,73]. TOX2 is a new marker similar to PD-1 in the coinhibitory pathway [71,74].

Statistical analyses showed that steroid-requiring ulcerative colitis was characterized by higher endoscopic Baron and histologic Geboes scores and LAIR1 expression, and lower TOX2 expression in isolated lymphoid follicles. The CNN managed to classify the 3 diagnoses with >99% accuracy, and the Grad-CAM confirmed which parts of the images were relevant for the classification. In conclusion, the study proposed that CNNs are essential tools for deep learning image recognition.

These results were published as preprints [75].

This paper is a companion manuscript of the recently published article “Ulcerative Colitis, LAIR1 and TOX2 Expression, and Colorectal Cancer Deep Learning Image Classification Using Convolutional Neural Networks” published in Cancers 2024, 16, 4230. https://doi.org/10.3390/cancers16244230.

2. Data Description

The dataset contains image patches of ulcerative colitis, colorectal cancer, and colon control. The image patches were split from whole-tissue images and had a size of 224 × 224 × 3. The original magnification of the images was 200× and a dpi of 150. The image patches are anonymized. After splitting the images, the patches were filtered. The criteria were as follows: (1) image patches of only 243×243 size; (2) image patches of more than 5-31 KB that contain tissue at least 20-30% of viable tissue; (3) image patches with diagnostic areas; (4) image patches without artifacts, including broken tissue, folded areas, incorrectly stained tissue, and smashed/crushed tissue. Steps 3 and 4 were manually curated by a pathology specialist (MD PhD).

The investigations were carried out according to the Declaration of Helsinki (website: https://www.wma.net/policies-post/wma-declaration-of-helsinki/; last accessed on December 12, 2024). Approval from an ethics committee was obtained before undertaking the research (protocol code IRB14R-080, IRB20-156, and 13R-119).

3. Methods

3.1. Formalin Tissue Fixation and Paraffin Embedding

Samples were fixed and embedded for long-term storage and immunostaining using fixatives to preserve tissues. The protocol included neutral buffered formalin, 80% alcohol, 95% alcohol, 100% alcohol, xylene, and paraffin pellets. Processing was performed using a HistoCore Peloris 3 premium tissue processing system (#13B2X10268PELOR3, Leica Biosystems K.K., Tokyo, Japan) (Figure 6).

The processed tissue was embedded in paraffin, forming paraffin blocs ready for sectioning using a tissue embedder (Tissue-Tek® TEC™ 6 Embedding Console System, Sakura Finetek Japan Co., Ltd., Tokyo, Japan) (Figure 7).

3.2. Sectioning of Paraffin-Embedded Tissue

Paraffin blocks were trimmed at 2μm using a Yamato REM-710 microtome (Yamato Kohki Industrial Inc., Saitama, Japan) (Figure 8). The paraffin ribbons were placed in 40-50 warmed water bath and mounted onto slides. The sections were later air dried and baked 45-50 ºC in an oven overnight.

3.3. Hematoxylin and Eosin (H&E) Staining

H&E staining is widely used in histological and cytological applications, both in fixed paraffin-embedded and frozen tissue sections. Hematoxylin produces an intense blue staining of the nuclei. Eosin stains the cytoplasm, collagen, muscle, and erythrocytes in light pink/rose. A standard H&E staining protocol includes deparaffinizing with xylene, rehydrating in decreasing concentrations of alcohol (100%, 90%, 80%, 70%, and distilled water), staining with hematoxylin, differentiating, bluing, eosin, dehydration, clearing, and coverslipping. Staining was performed using an automated slide stainer, a Tissue-Tek Prisma® Plus (Sakura Finetek Japan Co., Ltd.,) (Figure 9).

Glass coverslipping of histopathological slide sections was performed using a Tissue-Tek® Glas™ g2 Glass Coverslipper (Sakura Finetek Japan Co., Ltd.).

3.4. Score Evaluation

The endoscopic scores for chronic inflammatory bowel disease were evaluated using the Baron Score, which classifies mucosal changes into 4 grades: grade 0 (normal); grade 1 (abnormal non-hemorrhagic), 2 (moderately hemorrhagic), and 3 (severely hemorrhagic).

Ulcerative colitis histological assessment was performed using the Geboes score: grade 0 (no abnormality), grade 1 (chronic inflammatory infiltrate), grade 2 (lamina propria neutrophils and eosinophils), grade 3 (neutrophils in epithelium), and grade 4 (crypt destruction) (Figure 10).

Table A1 and Table A2 in the Appendix present the Baron and Geboes scores, respectively.

3.5. Immunohistochemistry

The immunohistochemistry of LAIR1 and TOX2 was performed using a Bond-Max fully automated immunohistochemistry and in situ hybridization staining system following the manufacturer’s instructions (Leica Biosystems K.K.) (Figure 11).

Visualization of the primary antibody bound to the tissue sections was performed using BOND Polymer Refine Detection (DS9800, Leica Biosystems K.K.). The BOND staining mode was single, i.e. a single marker and chromogen were applied to a single slide. Bond Polymer Refine Detection is a biotin-free, polymeric horseradish peroxidase (HRP)-linker antibody conjugate system for the detection of tissue bonding mouse and rabbit antibodies. This detection avoids the use of streptavidin and biotin; as a result, nonspecific staining is reduced.

The protocol sequence was the following: preparation (removal of wax using BOND Dewax Solution, 100% alcohol, and wash solution), heat-induced epitope retrieval (HIER; using BOND Epitope Retrieval ER2 solution), protein block (optional animal-free blocking solution), probe application, probe removal (wash solution), post-primary mouse linker, secondary detection (refine detection kit polymer), visualization (refine detection mixed DAB), and counterstain (hematoxylin). In summary, the staining protocol included the following steps: marker, post primary, polymer, mixed DAB, and BOND-PRIME hematoxylin.

Coverslipping was achieved using a Leica CV5030 fully automated glass coverslipper (Leica Biosystems K.K.) (Figure 12).

The primary antibody targeted the leukocyte-associated immunoglobulin-like receptor (LAIR1/CD305) created by the Monoclonal Antibodies Core Unit, located at the Spanish National Cancer Research Center (CNIO: Centro Nacional de Investigaciones Oncologicas; C/ Melchor Fernandez Almagro, 3, E-28029 Madrid, Spain). LAIR1 is a rat monoclonal antibody, clone JAVI82A, antigen used: RBL-1-LAIR1-MYCDDK transfected cells and last booster with LAIR1 recombinant (Gln22-His163, with a C-terminal 6-His tag); isotype IgG2a; reactivity, human; localization, membrane.

The primary antibody TOX2 targeted the TOX High Mobility Group Box Family Member 2 and was also developed by CNIO. Properties: clone name TOM924D, rat monoclonal, IgG2b K, antigen HIS-SUMO-hTOX2-Strep-tag2 full-length protein, human reactivity, nuclear localization.

Rabbit Anti-Rat IgG Antibody (H+L), Mouse Adsorbed, Unconjugated (#AI-4001-.5, Vector Laboratories, Inc., Newark, CA 94560, United States) was used as a linker between primary antibodies and BOND Polymer Refine Detection.

The immunohistochemistry of both antibodies was first tested in reactive lymphoid tissue (tonsils) and small and large intestinal tissue controls. Four examples of biopsies of the colon of ulcerative colitis stained with TOX2 and LAIR1 and with high infiltration of positive cells are shown in Figure 13, Figure 14, Figure 15 and Figure 16.

Before digitalization, the H&E and immunohistochemical slides were visualized using an Olympus upright microscope for quick evaluation and quality control purposes (Olympus BX63, Olympus K.K., Hachioji, Tokyo, Japan) (Figure 17).

3.6. Whole-Slide Imaging

Whole-slide imaging was performed using a Hamamatsu NanoZoomer S360 scanner (Hamamatsu Photonics K.K., Hamamatsu City, 431-3196, Japan) that rapidly scanned glass slides to convert them to digital data, NZAcquire 1.2.0 software (Hamamatsu Photonics K.K.), and a Dell Precision 5820 Tower equipped with an Intel(R) Xeon(R) W-2135 CPU @ 3.70 GHz 3.70 GHz, 32.0 GB RAM, 64 bits workstation system. The operating system was Windows 10 Pro for Workstations (version 1803, build 17134.1) (Figure 18).

The acquisition method was performed following the manufacturer’s instructions. The scan was batch scan-type, at 400× magnification, and single z stack. The number of focus points was defined automatically using NZAcquire software (Figure 19).

3.7. Digital Image Quantification

Conventional immunohistochemical analysis was performed using digital image quantification with Fiji software, as previously described [76,77,78,79]. In summary, quantification was performed in the blue stack, min and max thresholds were set, pixels were measured, and percentages were calculated (Figure 20).

3.8. Image Classification using CNNs

A convolutional neural network (CNN) was designed based on transfer learning from ResNet-18; and trained to classify the 3 types of image patches; ulcerative colitis (n= 9,281), colon control (n=12,246), and colorectal cancer (n=63,725). The CNN was designed in MATLAB (R2023b, update 9, 23.2.0.2668659). The CNN was also trained to differentiate between mesalazine-responsive and steroid-requiring ulcerative colitis based on H&E, LAIR1, and TOX2 staining (Figure 21 and Figure 22).

The data were partitioned into a training set (70% of the image patches) to train the network, a validation set (10%) to test the performance of the network during training, and a test set (20%) as a holdout (new data) to test the performance on new data.

The arrangement of the images was randomized to ensure that the CNN learned the classes at an even rate. Transfer learning, which involves reusing and adjusting a pretrained network, was performed on ResNet-18. For this purpose, the fully connected and classification layers of ResNet-18 were deleted and replaced with new layers. These new layers had an output size of 2. The training did not employ the augmentation technique. To avoid overfitting, the initial learning rate was set to 0.001. The maximum number of epochs was five [75].

Data normalization was applied to the input images. A detailed description of the data normalization process is provided in Appendix Table A3 [80,81].

The trained network was characterized by 71 layers. The first layer was the “ImageInputLayer”, and the last one the “ClassificationOutputLayer” (Figure 23). The parameters are listed in Table 3.

The code used is shown in Table A4 in the Appendix

In the analysis setup, the image patches of the 3 diagnoses were pooled in 3 different folders. The content of each folder was then split into training, validation, and testing sets. As a result, no image patches were repeated in different folders. Nonetheless, some researchers believe that this strategy could lead to information leakage. Therefore, an additional and independent test set of 10 cases of colorectal cancer (adenocarcinoma) was used to confirm the performance of the trained CNN. Each patient was analyzed independently [75].

Of note, other types of CNNs were tested in this study. The LAIR1 and TOX2 biomarkers were included in the CNN training in an independent analysis of H&E staining. A differentiation between steroid-requiring (SR) and mesalazine-responsive ulcerative colitis using LAIR1 and TOX2 immunohistochemistry.

The performance parameters were the following: accuracy, precision, recall, F1 Score, specificity, and false positive rate.

The performance of ResNet-18 was compared to that of other CNNs under the same experimental conditions, including DenseNet-201, ResNet-50, Inception-v3, ResNet-101, ShuffleNet, MobileNet-v2, NasNet-Large, GoogLeNet-Places365, VGG-19, EfficientNet-b0, AlexNet, Xception, VGG-16, GoogLeNet, and NasNet-Mobile.

3.9. Computational Requirements

All analyses were performed using a desktop computer equipped with an AMD Ryzen 9 7950X CPU, 128 Gb of RAM (Crucial Desktop DDR5-4800 UDIMM 1.1V CL40, CT2K32G48C40U5 x2), and an Nvidia GeForce RTX 4090 graphics card (ASUS ROG Strix GeForce RTX® 4090 OC Edition 24GB GDDR6X). The specifications of the GPU were 16384 Nvidia Cuda® cores, 4^th Generation 1321 AI TOPS Tensor cores, 2.23 GHz base and 2.52 GHz boost clock, 24 GB GDDR6X memory, 384-bit memory interface width, Ada Lovelace architecture (Figure 24).

4. Results

This paper of Methods type is a companion manuscript to a recently published article entitled “Ulcerative Colitis, LAIR1 and TOX2 Expression, and Colorectal Cancer Deep Learning Image Classification Using Convolutional Neural Networks”. Published in Cancers (Basel); Published 2024 Dec 19. doi:10.3390/cancers16244230 [75].

In summary, steroid-requiring ulcerative colitis was characterized by higher endoscopic Baron and histologic Geboes scores, higher LAIR1 immunohistochemical expression in the lamina propria, and lower TOX2 in the isolated lymphoid follicles (ILFs). The CNN successfully classified ulcerative colitis, colorectal cancer, and colon control with high performance. The classification was also validated using an independent test sample.

5. Discussion

Artificial Intelligence (AI) is the science and engineering that allows computers and machines to simulate human learning and problem solving. The development of AI follows a series of steps since 1950s, when the term AI was first postulated as human intelligence exhibited by machines. In the 1980s, machine learning was developed as an AI system that could learn from historical data. In the 2010s, deep learning was introduced as a machine learning model that simulated human brain function. From 2020 generative AI (Gen AI) was developed as a deep learning model (foundation models) that creates original content [82,83,84].

There are many techniques or algorithms of machine learning such as logistic regression, decision trees, random forest, and k-nearest neighbor (KNN) [79,80]. Deep learning is based on neural networks (deep neural networks) that are designed as input, hidden, and output layers. Deep learning enables semi-supervised, self-supervised, reinforcement, and transfer learning [82,83,84].

This technical note manuscript described a dataset and computer vision method that allows software and machines to understand and classify visual data in the context of artificial narrow AI. Narrow AI is also referred to as Weak AI, and it is the only type of AI that exists today. Narrow AI can be trained to perform a single task, and it can perform faster and better than the human mind. However, it cannot perform tasks outside its defined scope.

6. Conclusions

Convolutional neural networks are suitable for image classification in the context of narrow artificial intelligence. This paper describes a technical note on a method for collecting, processing, managing, and analyzing scientific data from intestinal histological images. Related source code is provided in the Appendix, and examples of image patches are provided in the Supplementary Material.