Deep Learning Workflow and Dataset for Deep Learning Image Classification of Ulcerative Colitis and Colorectal Cancer

1. Summary

1.1. Ulcerative colitis

Inflammatory bowel disease (IBD) is a chronic inflammatory condition of the gastrointestinal tract with systemic repercussions that is characterized by relapsing and remitting episodes of inflammation [1,2]. IBD includes two types: ulcerative colitis, which affects the colon, and Chron disease, which can affect any part of the gastrointestinal tract from the mouth until the perianal area [1,2,3,4,5,6,7,8,9].

The causes and pathogenesis of inflammatory bowel disease are unclear. However, it appears to be a combination of factors from the environment, including the microbiome [10,11], genetic susceptibility [12,13], and immune system (both systemic and gut) [13,14,15,16]. The prevalence of inflammatory bowel disease has increased globally in recent decades, particularly in industrialized countries [17,18,19,20,21]. Clinical risk factors include smoking [22], low physical activity, low fiber intake, high fats, and low vitamin D [23], sleep deprivation, previous acute gastroenteritis, antibiotic use, and early life exposures [24].

In normal conditions and in healthy status, the intestinal barrier is kept thanks to the mucus layer and epithelial cells that create bonds using the tight junctions. This barrier is supported by the presence of IgA and several antimicrobial factors. The immune response is initiated by dendritic cells that acquire, process, and present the antigens to B and T cells (lymphocytes) [25]. The cause of ulcerative colitis is still not completely understood. Ulcerative colitis is associated with a damage of the barrier of the mucosa, with an alteration of the microflora and the generation of an uncontrolled immune reaction. Several subsets of CD4-positive T cell are believed to participate in the pathogenesis of ulcerative colitis including Th1, Th2, Th9, Th17, Th22, TFh cells, and Tregs [26,27].Th9 are associated with continuous apoptosis of the enterocytes and inhibit mucosal healing [25,28]. IL-13 and NK/T cells are also involved in the epithelial injury [25]. Innate immune cells contribute to the cytokine production and continuous inflammation [14,15,29]. The injury of the mucosa is associated with dysbiosis [30], which is defined by an alteration of the composition of the mucosa, with changes of the diversity, increased potentially pathogenic bacteria, and reduced beneficial bacteria [31]. Figure 1 shows an example of ulcerative colitis and immunohistochemistry of the cells involved in the pathogenesis.

The disease activity of ulcerative colitis is being recorded to properly treat patients and perform clinical trials. Usually, the severity is classified as mild, moderate, and severe. The available classifications include the Mayo [32], Montreal [33], Truelove, and Witts classifications [34]. The grade of mucosal inflammation can be assessed using endoscopy to record the degree of mucosal ulceration and the extent of disease. This can be graded using the Mayo score [32] and Baron score. The degree of histological severity can be evaluated using the Geboes score [35].

Microscopic (histologic) evaluation of ulcerative colitis shows active chronic colitis in patients with untreated disease. Disease activity is defined by neutrophil infiltration of the lamina propria, cryptitis, crypt abscess, and ulceration. The inflammation is limited to the mucosa and submucosa. Compared with Crohn disease, this condition is associated with a lack of transmural inflammation, granuloma, and fissuring ulcers. Dysplasia may be present in long-term disease [36,37,38].

Disease activity can be evaluated using various parameters. Inactive disease lacks neutrophils. If the activity affects less than 50% of the mucosa, it is called mild. When it is > 50% and there are crypt abscesses, it is called moderate. Finally, severe activity is characterized by surface ulcerations and erosion [39].

The first-line therapy for the induction and maintenance of remission of mild to moderate UC is 5-aminosalicylic acid [40]. The initial therapy for ulcerative colitis is topical mesalamine, which is available as a suppository or enema, and it is recommended in cases of ulcerative proctitis or proctosigmoiditis. If topical mesalamine is not tolerated, an alternative therapy is topical glucocorticoids (i.e. hydrocortisone). A combination of 5-ASA and rectal mesalamine may be used for left-sided or extensive colitis. The 5-ASA formulations include mesalamine, sulfasalazine, and diazo-bonded 5-ASA. Patients who do not respond may subsequently be treated with glucocorticoids such as budesonide, which may escalate to prednisone or biological agents (anti-TNF, anti-integrin, anti-IL12/23, S1P modulators, JAK inhibitors) [40,41,42,43]. Patients who do not respond to oral glucocorticoids may require intravenous glucocorticoids. In some cases, surgery is performed [40,41,42,43]. Fecal microbiota transplantation is also available [44].

Figure 1. Pathogenesis of ulcerative colitis and cells of the immune microenvironment. Ulcerative colitis is a chronic inflammatory condition that affects the colon. The incidence of ulcerative colitis has been increasing in the last decades. The pathogenesis is multifactorial including genetic predisposition and an alteration of the immune tolerance and homeostasis of the mucosa. Several immune cells are involved. This figure shows the immunohistochemical staining of CD3-positive T lymphocytes, CD8-positive cytotoxic T lymphocytes, FOXP3-positive regulatory T lymphocytes (Tregs), PD-1-positive follicular T helper cells (TFH), and CD68-positive macrophages.

Figure 1. Pathogenesis of ulcerative colitis and cells of the immune microenvironment. Ulcerative colitis is a chronic inflammatory condition that affects the colon. The incidence of ulcerative colitis has been increasing in the last decades. The pathogenesis is multifactorial including genetic predisposition and an alteration of the immune tolerance and homeostasis of the mucosa. Several immune cells are involved. This figure shows the immunohistochemical staining of CD3-positive T lymphocytes, CD8-positive cytotoxic T lymphocytes, FOXP3-positive regulatory T lymphocytes (Tregs), PD-1-positive follicular T helper cells (TFH), and CD68-positive macrophages.

1.2. Colorectal cancer

Colorectal cancer (CRC) is a common disease characterized by environmental and genetic factors. It is the third most frequently diagnosed cancer worldwide [45]. There are several factors associated with early onset of CRC, including hereditary syndromes (familial adenomatous polyposis and Lynch syndrome), metabolic dysregulation, history of CRC in first-degree relative; alcohol consumption, lack of regular use of nonsteroidal anti-inflammatory drugs (NSAIDs), and vitamin D intake. Other known risk factors of CRC include inflammatory bowel disease (ulcerative colitis and Crohn disease), abdominopelvic radiation, obesity, diabetes mellitus, insulin resistance, red and processed meat, and tobacco [46,47,48,49]. The diagnosis is usually made by colonoscopy, and treatment includes surgical resection, adjuvant chemotherapy, and postoperative radiation therapy [46]. Adenocarcinoma is the most common histological subtype of CRC. Adenocarcinoma is a glandular neoplasm. Most CRC cases are moderately differentiated with simple, complex, or slightly irregular tubules, and loss of nuclear polarity. The glands are often cribriform, with necrosis, inflammatory cells, and marked desmoplasia [50,51,52].

1.3. Computer vision for deep learning image classification

Image processing involves algorithms and workflows used for image processing, analysis, visualization, and algorithm development. Computer vision workflows [53] with deep learning include several types of functions, such as image classification, object detection, and instance segmentation, automated visual inspection, semantic segmentation, and video classification [54,55,56].

A pretrained neural network that has already learned how to extract the characteristics of natural images can be used as a starting point to handle new images. Using a pretrained image classification network [57], the learning time is shorter, and training is usually easier. The transfer learning process takes layers from an already trained neural network and calibrates the parameters on a new dataset [54,55,56,58,59,60,61,62,63,64,65]. The analysis includes a series of steps, including preprocessing the data, import pre-trained networks from platforms such as TensorFlow 2 [66], TensorFlow-Keras [67], Pythorch [68], and ONNX [69], building network, selecting training options, improving the network performance by tuning hyperparameters, visualizing and verifying network behavior during and after training, and exporting the network to other platforms if necessary[54,55,56,58,59,60,61,62,63,64,65].

The most important features of a neural network are accuracy, speed, and size. A good neural network is characterized by being fast and having good performance (i.e. accurate). Neural networks that are accurate in ImageNet can be used to classify other images using transfer learning or feature extraction. Among the various neural networks, the best examples are GoogLeNet, ResNet-18, NobileNet, ResNet-50, ResNet-101, and Inception-v3 [56].

1.4. Dataset and research project description

Ulcerative colitis is a chronic inflammatory bowel disease associated with a high risk of colorectal cancer. This study used convolutional neural networks and computer vision to classify histological images of ulcerative colitis, colorectal cancer (adenocarcinoma), and colon control.

The series included 35 patients with of ulcerative colitis, 18 with colorectal cancer, and 21 with colon control. Hematoxylin and eosin (H&E) glass slides were converted into high-resolution digital data by high-speed scanning. The whole-tissue slides were split into image patches of 224×224 pixels at 200× magnification and 150 dpi, and the ResNet-18 network was retrained to classify the 3 types of diagnosis. This transfer learning experiment also used other pretrained CNNs for performance comparison, and the gradient-weighted class activation mapping (Grad-CAM) heatmap technique was used to understand the classification decisions.

Additionally, immunohistochemical analysis of two new immuno-oncology markers, LAIR1 and TOX2 were analyzed in ulcerative colitis to differentiate between mesalazine-responsive and steroid-requiring patients. LAIR1 an inhibitory receptor that plays a constitutive negative regulatory role in the cytolytic function of natural killer (NK) cells, B-cells and T-cells [70,71,72]. TOX2 is a new marker similar to PD-1 in the co-inhibitory pathway [70,73].

Statistical analyses showed that steroid-requiring ulcerative colitis was characterized by higher endoscopic Baron and histologic Geboes scores and LAIR1 expression, but lower TOX2 expression in isolated lymphoid follicles. The CNN managed to classify the 3 diagnoses with >99% accuracy and the Grad-CAM confirmed which parts of the images were relevant for the classification. In conclusion, the study showed that CNNs are essential tools for deep learning image recognition.

These results were published as preprints [74].

This paper is the companion manuscript of the recently published article “Ulcerative Colitis, LAIR1 and TOX2 Expression, and Colorectal Cancer Deep Learning Image Classification Using Convolutional Neural Networks” published in Cancers 2024, 16, 4230. https://doi.org/10.3390/cancers16244230.

2. Data Description

The dataset contains image patches of ulcerative colitis, colorectal cancer, and colon control. The image patches were split from whole-tissue images and had a size of 224 × 224. The original magnification of the images was 200× and a dpi of 150. The image patches are anonymized. After splitting the images, the patches were filtered. The filtering criteria were as follows: (1) image patches of not 243×243 size; (2) image patches less than 5-31 KB that usually do not contain tissue; (3) image patches that did not contain diagnostic areas based on histopathological criteria; (4) image patches with artifacts such as broken and folded tissue and wrongly stained. Steps 3 and 4 are medical specialist in pathology curated.

The investigations were carried out following the rules of the Declaration of Helsinki of 1975 (website: https://www.wma.net/policies-post/wma-declaration-of-helsinki/; last accessed on December 12, 2024), revised in 2008. Approval from an ethics committee was obtained before undertaking the research (protocol code IRB14R-080, IRB20-156, and 13R-119).

3. Methods

3.1. Hematoxylin and eosin (H&E) staining

H&E staining is widely used in histological and cytological applications, both in fixed paraffin-embedded and frozen tissue sections. Hematoxylin produces an intense blue staining of the nuclei. Eosin stains the cytoplasm, collagen, muscle, and erythrocytes in light pink/rose. Staining was performed using a Leica ST5010-CV5030 integrated stainer workstation (Leica Biosystems K.K., Shinjuku, Tokyo, 169-0075, Japan).

3.2. Score evaluation

The endoscopic scores for chronic inflammatory bowel disease were evaluated using the Baron Score, which classifies mucosal changes into 3 grades. Ulcerative colitis histological assessment was performed using the Geboes score. Table 1 and Table 2 present the Baron and Geboes scores, respectively.

3.3. Immunohistochemistry

The immunohistochemistry of LAIR1 and TOX2 was performed using a Bond-Max fully automated immunohistochemistry and in situ hybridization staining system following the manufacturer’s instructions (Leica Biosystems K.K.). Visualization of the primary antibody bound to the tissue sections was performed using BOND Polymer Refine Detection (DS9800, Leica Biosystems K.K.). Coverslipping was achieved using a Leica CV5030 fully automated glass coverslipper (Leica Biosystems K.K.).

The primary antibody targeted the leukocyte-associated immunoglobulin-like receptor (LAIR1/CD305) created by the Monoclonal Antibodies Core Unit, located at the Spanish National Cancer Research Center (CNIO: Centro Nacional de Investigaciones Oncologicas; C/ Melchor Fernandez Almagro, 3, E-28029 Madrid, Spain). LAIR1 is a rat monoclonal antibody, clone JAVI82A, antigen used: RBL-1-LAIR1-MYCDDK transfected cells and last booster with LAIR1 recombinant (Gln22-His163, with a C-terminal 6-His tag); isotype IgG2a; reactivity, human; localization, membrane.

The primary antibody TOX2 targeted the TOX High Mobility Group Box Family Member 2 and was also developed by CNIO. Properties: clone name TOM924D, rat monoclonal, IgG2b K, antigen HIS-SUMO-hTOX2-Strep-tag2 full-length protein, human reactivity, nuclear localization.

Rabbit Anti-Rat IgG Antibody (H+L), Mouse Adsorbed, Unconjugated (#AI-4001-.5, Vector Laboratories, Inc., Newark, CA 94560, United States) was used as a linker between primary antibodies and BOND Polymer Refine Detection.

3.4. Whole-slide imaging

Whole-slide imaging was performed using a Hamamatsu NanoZoomer S360 scanner (Hamamatsu Photonics K.K., Hamamatsu City, 431-3196, Japan) that rapidly scanned glass slides to convert them to digital data, NZAcquire 1.2.0 software (Hamamatsu Photonics K.K.), and a Dell Precision 5820 Tower equipped with an Intel(R) Xeon(R) W-2135 CPU @ 3.70 GHz 3.70 GHz, 32.0 GB RAM, 64 bits workstation system. The operating system was Windows 10 Pro for Workstations (version 1803, build 17134.1).

The acquisition method was performed following the manufacturer’s instructions. The scan was batch scan-type, at 400× magnification, and single z stack. The number of focus points was defined automatically using NZAcquire software.

Figure 1. Whole-slide imaging. The glass slides were converted to digital data using a Hamamatsu NanoZoomer S360 scanner that scanned the slides at 400× magnification.

Figure 1. Whole-slide imaging. The glass slides were converted to digital data using a Hamamatsu NanoZoomer S360 scanner that scanned the slides at 400× magnification.

3.5. Digital image quantification

Conventional immunohistochemical analysis was performed using digital image quantification with Fiji software, as previously described [75,76,77,78]. In summary, quantification was performed in the blue stack, min and max thresholds were set, pixels were measured, and percentages were calculated.

3.6. Image classification using CNNs

A convolutional neural network (CNN) was designed based on transfer learning from ResNet-18; and trained to classify the 3 types of image patches; ulcerative colitis (n= 9,281), colon control (n = 12,246), and colorectal cancer (n = 63,725). The CNN was also trained to differentiate between mesalazine-responsive and steroid-requiring ulcerative colitis based on H&E, LAIR1, and TOX2 staining (Figure 2).

The data were partitioned into a training set (70% of the image patches) to train the network, a validation set (10%) to test the performance of the network during training, and a test set (20%) as a holdout (new data) to test the performance on new data.

The order of the images was randomized to ensure that the network learned the classes at a more even rate. As transfer learning (adjustment of a pre-trained network) was performed on ResNet-18, the fully connected and classification layers were removed and replaced with new layers with an output size of 2. Augmentation was not performed during training. To avoid overfitting, the initial learning rate was set to 0.001. The number of maximum epochs was set to five [74].

Data normalization was applied to the input images: imageInputLayer (an image input layer inputs 2-D images to a neural network and applies data normalization), and batchNormalizationLayer (a batch normalization layer normalizes a mini-batch of data across all observations for each channel independently. To speed up training of the convolutional neural network and reduce the sensitivity to network initialization, batch normalization layers are used between convolutional layers and nonlinearities, such as ReLU layers. Layer = batchNormalizationLayer (Name, Value) creates a batch normalization layer and sets the optional TrainedMean, TrainedVariance, Epsilon, Parameters and Initialization, Learning Rate and Regularization, and Name properties using one or more name-value pairs. After normalization, the layer scales the input with a learnable scale factor γ and shifts it by a learnable offset β) [79,80].

The trained network was characterized by 71 layers. The first layer was the “ImageInputLayer”, and the last one the “ClassificationOutputLayer” (Table 3).

To train the network using the specified options and training data, the following code was used:

[net, traininfo] = trainNetwork(augimdsTrain,lgraph,opts);

The training options were as follows:

opts = trainingOptions("sgdm",...

  "ExecutionEnvironment","auto",...

  "InitialLearnRate",0.001,...

  "MaxEpochs",5,...

  "Shuffle","every-epoch",...

  "Plots","training-progress",...

  "ValidationData",augimdsValidation);

Table 4. Design and training parameters

ResNet-18-Based CNN	Training (70%)	Validation (10%)	Training Options
Input type: image patches Output type: classification Number of layers: 71 Number of connections: 78	Observations: 59,677 Classes: 3 Ulcerative colitis: 6497 Colorectal cancer: 44,608 Colon control: 8572	Observations: 8525 Classes: 3 Ulcerative colitis: 928 Colorectal cancer: 6372 Colon control: 1225	Solver: sgdm Initial learning rate: 0.001 MiniBatch size: 128 MaxEpochs: 5 Validation frequency: 50 Iterations: 2330 Iterations per epoch: 466
Additional detailed training options
Import images: augmentation options, none. Solver: momentum, 0.9. Learn rate: LearnRateSchedule, none; LearnRateDropFactor, 0.1; LearnRateDropPeriod, 10. Normalization and Regularization: L2Regularization: 0.0001; ResetInputNormalization, yes; BatchNormalizationStatistics, population. Mini-Batch: Shuffle, every-epoch. Validation and Output: ValidationPatience, Inf; OutputNetwork, last-iteration. Gradient Clipping: GradientThresholdMethod, I2norm; GradientThreshold, Inf. Hardware: ExecutionThreshold, auto. Checkpoint: CheckpointPath, n/a; CheckpointFrequency, 1; CheckpointFrequencyUnit, epoch.

Based on transfer learning of ResNet-18. Convolutional neural network, CNN.

Table 4. Design and training parameters

ResNet-18-Based CNN	Training (70%)	Validation (10%)	Training Options
Input type: image patches Output type: classification Number of layers: 71 Number of connections: 78	Observations: 59,677 Classes: 3 Ulcerative colitis: 6497 Colorectal cancer: 44,608 Colon control: 8572	Observations: 8525 Classes: 3 Ulcerative colitis: 928 Colorectal cancer: 6372 Colon control: 1225	Solver: sgdm Initial learning rate: 0.001 MiniBatch size: 128 MaxEpochs: 5 Validation frequency: 50 Iterations: 2330 Iterations per epoch: 466
Additional detailed training options
Import images: augmentation options, none. Solver: momentum, 0.9. Learn rate: LearnRateSchedule, none; LearnRateDropFactor, 0.1; LearnRateDropPeriod, 10. Normalization and Regularization: L2Regularization: 0.0001; ResetInputNormalization, yes; BatchNormalizationStatistics, population. Mini-Batch: Shuffle, every-epoch. Validation and Output: ValidationPatience, Inf; OutputNetwork, last-iteration. Gradient Clipping: GradientThresholdMethod, I2norm; GradientThreshold, Inf. Hardware: ExecutionThreshold, auto. Checkpoint: CheckpointPath, n/a; CheckpointFrequency, 1; CheckpointFrequencyUnit, epoch.

Based on transfer learning of ResNet-18. Convolutional neural network, CNN.

In this study, the image patches of the three diagnoses were pooled and later split into training, validation, and test sets. However, this strategy can create an information leak. Therefore, an additional independent series of 10 cases of colorectal cancer (adenocarcinoma) were classified to confirm the performance of the trained CNN (based on ResNet-18 architecture). In this analysis, each patient was analyzed independently [74].

Of note, other types of CNNs were tested in this study. LAIR1 and TOX2 biomarkers were included into the CNN training in an independent analysis from the H&E staining. A differentiation between steroid-requiring (SR) and mesalazine-responsive ulcerative colitis using LAIR1 and TOX2 immunohistochemistry.

The performance parameters were the following: accuracy, precision, recall, F1-Score, specificity, and false positive rate.

The performance of ResNet-18 was compared with the one of other CNNs under same experimental conditions, including DenseNet-201, ResNet-50, Inception-v3, ResNet-101, ShuffleNet, MobileNet-v2, NasNet-Large, GoogLeNet-Places365, VGG-19, EfficientNet-b0, AlexNet, Xception, VGG-16, GoogLeNet, and NasNet-Mobile.

3.7. Computational requirements

All analyses were performed using a desktop computer equipped with an AMD Ryzen 9 7950X CPU (AMD Japan Ltd., Marunouchi, Chiyoda-ku, Tokyo, Japan), 128 Gb of RAM, and an Nvidia GeForce RTX 4090 graphics card (Nvidia, Minato-ku, Tokyo, Japan; ROG Strix GeForce RTX® 4090 OC Edition 24GB GDDR6X).

3.8. Limitations

This project has the limitation of the number of cases. In future, the series will be expanded and the trained network retrained to improve the parameters and adapt to different types of images and experimental conditions.

This project has not used the most state-of-the-art CNNs. The aim of this study was to translate the methodology to routine diagnosis. Therefore, reliable and simple methodology was needed. On the other hand, the strength of the study was the pathologist-curated histological dataset of images.

4. Results

This paper is the companion manuscript of the recently published article “Ulcerative Colitis, LAIR1 and TOX2 Expression, and Colorectal Cancer Deep Learning Image Classi-fication Using Convolutional Neural Networks” published in Cancers 2024, 16, 4230. https://doi.org/10.3390/cancers16244230. The abstract is the following:

Background: Ulcerative colitis is a chronic inflammatory bowel disease of the colon mucosa associated with a higher risk of colorectal cancer. Objective: This study classified hematoxylin and eosin (H&E) histological images of ulcerative colitis, normal colon, and colorectal cancer using artificial intelligence (deep learning).

Methods: A convolutional neural network (CNN) was designed and trained to classify the three types of diagnosis, including 35 cases of ulcerative colitis (n = 9281 patches), 21 colon control (n = 12,246), and 18 colorectal cancer (n = 63,725). The data were partitioned into training (70%) and validation sets (10%) for training the network, and a test set (20%) to test the performance on the new data. The CNNs included transfer learning from ResNet-18, and a comparison with other CNN models was performed. Explainable artificial intelligence for computer vision was used with the Grad-CAM technique, and additional LAIR1 and TOX2 immunohistochemistry was performed in ulcerative colitis to analyze the immune microenvironment.

Results: Conventional clinicopathological analysis showed that steroid-requiring ulcerative colitis was characterized by higher endoscopic Baron and histologic Geboes scores and LAIR1 expression in the lamina propria, but lower TOX2 expression in isolated lymphoid follicles (all p values < 0.05) compared to mesalazine-responsive ulcerative colitis. The CNN classification accuracy was 99.1% for ulcerative colitis, 99.8% for colorectal cancer, and 99.1% for colon control. The Grad-CAM heatmap confirmed which regions of the images were the most important. The CNNs also differentiated between steroid-requiring and mesalazine-responsive ulcerative colitis based on H&E, LAIR1, and TOX2 staining. Additional classification of 10 new cases of colorectal cancer (adenocarcinoma) were correctly classified.

Conclusions: CNNs are especially suited for image classification in conditions such as ulcerative colitis and colorectal cancer; LAIR1 and TOX2 are relevant immuno-oncology markers in ulcerative colitis.