Overall Survival Prediction of Diffuse large B-cell Lymphoma Using Deep Learning and Hematoxylin and Eosin Histological Images

1. Introduction

Diffuse large B-cell lymphoma (DLBCL) is one of the most frequent histological subtypes of non-Hodgkin lymphoma (NHL), accounting for approximately 25% of adult NHL cases [1]. Most patients present with a rapid enlarging symptomatic mass with nodal enlargement. In approximately 60% of the cases, DLBCL present with advanced-stage (III or IV) and high serum lactate dehydrogenase (LDH), and in 30% with fever, weight loss, night sweats, and bone marrow infiltration [2,3,4,5,6,7]. DLBCL is cured in 60% of the cases with current therapy, particularly in patients who achieve complete response with first-line treatment. However, in 40% of the cases, the clinical evolution is unfavorable [5,8,9].

DLBCL is a heterogeneous clinicopathological entity. It is derived from germinal center B-cells (centroblasts) or post-germinal activated B-cells (immunoblasts) [10,11]. The tumor cells of DLBCL are large (e.g. nuclei twice the size of a small lymphocyte and larger than the nucleus of a macrophage) [10,11,12,13,14,15,16,17]. Centroblasts are large, noncleaved cells with round or oval nuclei, vesicular chromatin, often with multiple peripheral nucleoli, and a narrow rim of basophilic cytoplasm [10,11,12,13,14,15,16,17]. Immunoblasts are usually larger cells with prominent nucleoli and more abundant cytoplasm, often with plasmacytoid characteristics. Some cases are mixed, and other morphological variants exist [10,11,13,14]. The pathogenesis of DLBCL is complex and shows a heterogeneous landscape [1].

DLBCL, not otherwise specified (NOS) is defined as having a mature B-cell phenotype and large cell morphology, but having none of the criteria that define specific large B-cell lymphoma subtypes. Using either immunohistochemistry or gene expression studies of DLBCL, the cell-of-origin (COO) status can be determined, including germinal center B-cell (GCB) versus activated B-cell (ABC) subtypes [2,5]. Other large B-cell lymphoma subtypes are primary mediastinal, T-cell/histiocyte-rich, plasmablastic, primary cutaneous leg-type, immune privileged sites, intravascular, associated with chronic inflammation, IRF4 rearranged, ALK-positive, with 11q aberration, primary effusion lymphoma, and Epstein-Barr virus (EBV)-positive DLBCL [2,5].

Nodal DLBCL can spread to other organs, such as the liver, kidneys, lung, bone marrow, and central nervous system. Extranodal/extramedullary involvement is frequently present in early-stage disease (stage I/II). The most frequent primary extranodal presentation is the gastrointestinal tract, but virtually any tissue could be affected, including the testis, bone, thyroid, salivary glands, tonsil, and skin [1,18,19,20].

Several prognostic models are used in DLBCL. The International Prognostic Index (IPI) and its variants are the main prognostic variables that are routinely used. The original IPI included the following factors and age >60 years, high serum LDH, Eastern Cooperative Oncology Group (ECOG) performance status ≥2, clinical stage III-IV, and number of extranodal sites >1 associated with poor prognosis [21]. The COO status determined by immunohistochemistry using CD10, BCL6, and MUM1, or by gene expression profiling, correlated with non-GCB/ABC-like with poorer prognosis of the patients [21,22,23,24,25,26,27]. MYC rearrangement is seen in 5-15% of the cases and can be associated with BCL2 and BCL6 rearrangements [28]. Double-hit MYC and BCL2 cases have a worse prognosis [2,5,29,30,31,32,33,34,35]. Other techniques such as deep sequencing have confirmed the DLBCL heterogeneity and identified driver mutations with different clinical outcomes [2,5,6,17,36,37,38].

The histological characteristics of the tissue reflect the proteomic, transcriptomic and genomic pathological background. This study aimed to predict the prognosis of patients with DLBCL based only on the histological evaluation of hematoxylin and eosin staining. The 2-years timepoint shows a change in the slope of the survival curve. Therefore, the DLBCL cases were classified according to the overall survival, using as an event the presence of death within the first 2 years (“Dead 2-years”), which is similar to progression of disease 24 (POD24). Using several models, the prognostic was predicted with high performance, and several explainable artificial intelligence (XAI) techniques were used to visually highlight the regions that are most important for the deep neural network’s classification decision.

2. Materials and Methods

2.1. Patients and samples

A series of 114 patients with a histological diagnosis of conventional diffuse large B-cell lymphoma (DLBCL) from 2007 to 2011 were selected from the Department of Pathology, Tokai University, School of Medicine. The cases were diagnosed according to the current classifications of hematolymphoid neoplasia [2,5,10], which included the evaluation of hematoxylin and eosin, immunophenotype, and molecular techniques when required [2,5,6]. Based on the overall survival plot, two groups were defined: Dead < 2-years and the Others. The rationale is that at the 2-years time point the slope of the overall survival curve changes, passing from aggressive (b1 = -0.054) to less aggressive clinical behavior (b1 = -0.003).

Figure 1. Overall survival groups. The cases were grouped according to their survival based on the slope of the survival curve and the 2-years timeline: cases who died within the first 24 months (Dead < 2-years), and the Other cases. A more detailed analysis was performed using curve estimation.

Figure 1. Overall survival groups. The cases were grouped according to their survival based on the slope of the survival curve and the 2-years timeline: cases who died within the first 24 months (Dead < 2-years), and the Other cases. A more detailed analysis was performed using curve estimation.

This study was conducted following the guidelines of the Declaration of Helsinki of the World Medical Association (WMA) and ethical principles for medical research involving human participants. The Institutional Review Board of Tokai University approved the study (IRB20-156).

Immune microenvironment data were retrieved from our previous publications [39,40,41], and the immunohistochemistry reanalyzed. For CD163, IL10 and PD-L1, new immunohistochemistry (IHC) was performed as the number of cases in this study increased from the previous publications. The IHC methodology, including primary antibody details, is shown in Appendix B.

2.2. Deep learning image classification

Whole-tissue sections were stained with H&E and digitized using a slide scanner (NanoZoomer S360, C13220-01, Hamamatsu Photonics K.K.). The whole neoplastic areas were identified, digitally extracted at 20× magnification and 150 dpi, and split into image patches of 224×224×3 resolution. After splitting, image patches of size different from 224×224 and with less than 80% of tissue were discarded.

First, the series was slit into a training set (70%) and a validation set (10%) to help prevent overfitting during training and a test set (20%). The training set patches were shuffled before training. No augmentation options were used during the training.

The training options were as follows: sgdm solver, 0.001 initial learning rate, constant learning rate schedule, 128 minibatch size, 5 max epochs, and 50 validation frequency. In the NasNet-Large CNN, due to hardware limitations to run the analysis, the minibatch size was set at 16.

Image patches were classified using transfer learning and 20 different types of CNN, including AlexNet, DarkNet-19, DarkNet-53, DenseNet-201, EfficientNet-b0, GoogleLeNet, GoogleLeNet-places365, Inception-ResNet-v2, Inception-v3, MobileNet-v2, NasNet-Mobile, NasNet-Large, ResNet-18, ResNet-50, ResNet-101, Shufflenet, SqueezeNet, VGG-16, VGG-19, and Xception.

The following hardware and software were used: NanoZoomer S360, #C13220-01 (Hamamatsu Photonics K.K.); 12-Core AMD Ryzen 9 5900X CPU, 4900 MHz; 49075 MB DDR4-3200 SDRAM; NVIDIA GeForce RTX 4080 SUPER GPU; MATLAB R2023b Update 10 (23.2.0.2859533), 64-bit (win64); PhotoScape v3.7; IBM SPSS Statistics version 27 (Release 27.0.1.0, 64-bit edition); NDP.view 2.9.29 (RUO) 2022/01/14 (Hamamatsu Photonics K.K.).

Figure 2, Figure 3, Figure 4, Figure 5, Figure 6 and Figure 7 depict the methodological design.

3. Results

3.1. Image patch-based CNN classification of the 2-years cut-off OS groups

In the training/validation set, DarkNet-19 had the best validation accuracy (95.71%), and the accuracy/training time ratio was moderate (0.114). The most efficient architecture was ResNet-18, which had a validation accuracy of 92.11% and a ratio of 0.426.

Table 1. Training/validation set CNN performance.

CNN model	Learnables	Layers	Connections	Image Input	Training time	Validation accuracy (%)	Efficiency	Relative time
NasNet-Large	84.9M	1243	1462	331×331×3	1429 min 12 sec	96.42	0.001	564.2
DarkNet-19	20.8M	64	63	256×256×3	14 min 1 sec	95.71	0.114	5.5
DarkNet-53	41.6M	184	206	256×256×3	120 min 35 sec	95.6	0.013	47.6
DenseNet-201	20M	708	805	224×224×3	255 min 9 sec	93.9	0.006	100.7
ResNet-101	44.6M	347	379	224×224×3	114 min 21 sec	93.31	0.014	45.1
Inception-v3	23.8M	315	349	299×299×3	53 min 15 sec	92.25	0.029	21.0
ResNet-50	25.5M	177	192	224×224×3	14 min 3 sec	92.25	0.109	5.5
ResNet-18	11.6M	71	78	224×224×3	3 min 36 sec	92.11	0.426	1.4
VGG-16	138.3M	41	40	224×224×3	155 min 10 sec	92.11	0.009	61.3
MobileNet-v2	3.5M	154	163	224×224×3	12 min 55 sec	90.86	0.117	5.1
Inception-ResNet-v2	55.8M	824	921	299×299×3	509 min 8 sec	89.83	0.003	201.0
VGG-19	143.6M	47	46	224×224×3	181 min 17 sec	89.54	0.008	71.6
EfficientNet-b0	5.3M	290	363	224×224×3	54 min 0 sec	89.07	0.075	21.3
GoogleLeNet-places365	5.9M	144	170	224×224×3	5 min 30 sec	88.37	0.268	2.2
GoogleLeNet	6.9M	144	170	224×224×3	5 min 21 sec	88.34	0.275	2.1
Shufflenet	1.4M	172	187	224×224×3	5 min 42 sec	88.22	0.258	2.3
NasNet-Mobile	5.3M	913	1072	224×224×3	30 min 19 sec	87.73	0.048	12.0
Xception	22.9M	170	181	299×299×3	522 min 52 sec	87.28	0.003	206.4
AlexNet	60.9M	25	24	227×227×3	2 min 32 sec	83.94	0.552	1.0
SqueezeNet	1.2M	68	75	227×227×3	2 min 44 sec	79.35	0.484	1.1

Efficiency: Validation accuracy/time (% / sec). Relative time: faster CNN (AlexNet) as reference.

Table 1. Training/validation set CNN performance.

CNN model	Learnables	Layers	Connections	Image Input	Training time	Validation accuracy (%)	Efficiency	Relative time
NasNet-Large	84.9M	1243	1462	331×331×3	1429 min 12 sec	96.42	0.001	564.2
DarkNet-19	20.8M	64	63	256×256×3	14 min 1 sec	95.71	0.114	5.5
DarkNet-53	41.6M	184	206	256×256×3	120 min 35 sec	95.6	0.013	47.6
DenseNet-201	20M	708	805	224×224×3	255 min 9 sec	93.9	0.006	100.7
ResNet-101	44.6M	347	379	224×224×3	114 min 21 sec	93.31	0.014	45.1
Inception-v3	23.8M	315	349	299×299×3	53 min 15 sec	92.25	0.029	21.0
ResNet-50	25.5M	177	192	224×224×3	14 min 3 sec	92.25	0.109	5.5
ResNet-18	11.6M	71	78	224×224×3	3 min 36 sec	92.11	0.426	1.4
VGG-16	138.3M	41	40	224×224×3	155 min 10 sec	92.11	0.009	61.3
MobileNet-v2	3.5M	154	163	224×224×3	12 min 55 sec	90.86	0.117	5.1
Inception-ResNet-v2	55.8M	824	921	299×299×3	509 min 8 sec	89.83	0.003	201.0
VGG-19	143.6M	47	46	224×224×3	181 min 17 sec	89.54	0.008	71.6
EfficientNet-b0	5.3M	290	363	224×224×3	54 min 0 sec	89.07	0.075	21.3
GoogleLeNet-places365	5.9M	144	170	224×224×3	5 min 30 sec	88.37	0.268	2.2
GoogleLeNet	6.9M	144	170	224×224×3	5 min 21 sec	88.34	0.275	2.1
Shufflenet	1.4M	172	187	224×224×3	5 min 42 sec	88.22	0.258	2.3
NasNet-Mobile	5.3M	913	1072	224×224×3	30 min 19 sec	87.73	0.048	12.0
Xception	22.9M	170	181	299×299×3	522 min 52 sec	87.28	0.003	206.4
AlexNet	60.9M	25	24	227×227×3	2 min 32 sec	83.94	0.552	1.0
SqueezeNet	1.2M	68	75	227×227×3	2 min 44 sec	79.35	0.484	1.1

Efficiency: Validation accuracy/time (% / sec). Relative time: faster CNN (AlexNet) as reference.

Figure 8. Neural network training performances. The most important characteristics are the neural network accuracy (x-axis), speed (y-axis), and size (circle). Choosing a neural network is a tradeoff between these characteristics. NasNet-Large had the best validation accuracy (96.52%), followed by DarkNet-19 (95.71%). However, in relation with AlexNet that was the fastest, NasNet-Large took 564.2 more time to compute (i.e. relative time).

Figure 8. Neural network training performances. The most important characteristics are the neural network accuracy (x-axis), speed (y-axis), and size (circle). Choosing a neural network is a tradeoff between these characteristics. NasNet-Large had the best validation accuracy (96.52%), followed by DarkNet-19 (95.71%). However, in relation with AlexNet that was the fastest, NasNet-Large took 564.2 more time to compute (i.e. relative time).

Figure 9. CNN training plots.

Figure 9. CNN training plots.

In the validation set, DarkNet-19 had the best accuracy (96.26%), followed by NasNet-Large (96.21%), DarkNet-53 (95.47%), and DenseNet-201 (93.67%). The confusion charts of the models with higher performance are shown in Figure 10 and Figure 11. All the performance parameters of the models in the test set are shown in Table 2 and Figure 12.

Explainable artificial intelligence (XAI) was used to identify the areas of the images that the network DarkNet-19 used for classification. Figure 13 and Figure 14 show the Grad-CAM, occlusion sensitivity, and image Lime analysis.

3.3. Clinicopathological Characteristics

The Dead < 2-years group was correlated with several clinicopathological characteristics of the patients. The Dead < 2-years patients correlated with stage III-IV, International Prognostic Index (IPI) High + High/intermediate, absence of clinical response to treatment, non-GCB cell-of-origin, CD10-, BCL2+, and EBER+. Analysis of the immune microenvironment, cell cycle, and germinal center markers showed that Dead < 2-years had higher CD163, IL10, PD-L1, and lower E2F1 protein expressions (all p values < 0.05) (Table 3, Figure 15).

4. Discussion

DLBCL is one of the most common diagnostic categories of non-Hodgkin lymphoma (NHL), accounting for approximately 25% of NHL cases in the developed world [2,5]. Histologically, it is characterized by large transformed B cells that depict a diffuse growth pattern and efface the normal architecture of the underlying histological structure [2,5]. The diagnostic category of DLBCL, NOS, refers to conventional DLBCL with a mature B-cell phenotype and large cell morphology, but lacking none of the criteria that define specific large B-cell lymphoma subtypes (i.e., other large B-cell variants) [2,5,10].

This study used a series of 114 cases of conventional large B-cell lymphoma. Overall, it included not only NOS cases but also 28 cases of EBER-positive DLBCL. Most of the cases were nodal (58/114, 50.9%), followed by other extranodal (28.1%) and gastrointestinal (11.4%). The aim of the project was to identify cases with poor prognosis using only H&E staining and CNN. After the evaluation of the overall survival curve, two groups were defined: patients who died before the 2 years and the others. Correlation with the clinicopathological characteristics found that the Dead < 2-years group was correlated with stage III-IV, Internation-al Prognostic Index (IPI) High + High/intermediate, progressive disease, non-GCB cell-of-origin, CD10-, BCL2+, and EBER+. Analysis of the immune microenvironment, cell cycle, and germinal center markers showed that Dead < 2-years had higher IL10, CD163, PD-L1, and lower E2F1 expressions. Therefore, the CNN managed to identify the histological features that correlated with the prognosis of the patients. Features that were not very obvious under conventional histological examination under the optical microscope.

The deep learning workflow includes preprocessing data, importing and building the network, training the network, tuning the network, and visualizing the results. In this study, we used transfer learning to take advantage of the knowledge provided by a pre-rained network to learn new patterns in new data. Fine-tuning a pretrained network with transfer learning is typically much faster and easier than training from the beginning. The reuse of the pretrained network includes the following steps: load the pretrained network, replace the final layers, train the network, predict and assess the network accuracy, and deploy the results. This study used 20 pre-trained networks, including AlexNet, DarkNet-19, DarkNet-53, DenseNet-201, EfficientNet-b0, GoogleLeNet, GoogleLeNet-places365, Inception-ResNet-v2, Inception-v3, MobileNet-v2, NasNet-Large, NasNet-Mobile, ResNet-101, ResNet-18, ResNet-50, Shufflenet, SqueezeNet, VGG-16, VGG-19, and Xception. The CNN architectures were different and the performances ranged from 79.16% of SqueezeNet to 96.26% of DarkNet-19. The training times differed as well, being 1429 min for NasNet-Large and 2 min for AlexNet. The final analysis was performed using DarkNet-19. The final model based on DarkNet-19 predicted prognosis groups with high performance (test set accuracy = 96.26%). The other performance parameters were precision (94.46%), recall (95.02%), false positive rate (3.07%), specificity (96.93%), and F1 score (94.74%).

A machine learning model is often referred to as a “black box” model because understanding how the model makes predictions can be difficult. Interpretability tools help you overcome this aspect of machine learning algorithms and reveal how predictors contribute (or do not contribute) to predictions. Moreover, you can validate whether the model uses the correct evidence for its predictions and find model biases that are not immediately apparent. In this study, explainable artificial intelligence (XAI) was performed using Grad-CAM, occlusion sensitivity and image Lime on the network DarkNet-19. Overall, XAI showed that the convolutional neural networks (CNNs) were focusing on the neoplastic B-lymphocytes but some components of the microenvironment may also have played an influence. Of note, current XAI techniques do not allow a more detailed analysis of this type of histological tissue

In conclusion, narrow artificial intelligence (i.e., trained to perform a specific or a set of closely related tasks) can predict the prognosis of DLBCL based on the computer vision CNN histological analysis of H&E images, but it is process-specific and operates within limited constraints.