Deep Learning Models for White Blood Cell Image Classification

1. Introduction

The development of automated computer-aided diagnosis (CAD) systems is a prominent area of research [12]. Numerous endeavors have focused on creating CAD systems for a variety of health applications. Research efforts include studies using mammography images to develop systems for detecting and classifying breast cancer [13,14,15], and projects using computed tomography (CT) scans for lung cancer detection and classification [16,17,18]. Additionally, CAD systems targeting other types of cancer, such as prostate cancer, often utilize MRI technology for diagnostic and classification purposes [19,20,21].

Moreover, CAD systems find use in diverse medical fields beyond oncology. For instance, they are instrumental in grading diabetic retinopathy using optical coherence tomography (OCT) images [22,23,24], and in detecting ulcers where capsule endoscopy [25,26] and even smartphone images [27] are employed.

This research introduces a CAD system aimed at classifying and localizing white blood cells (WBCs) using microscopic imaging. Such imaging is crucial for providing insights into a patient’s hematological health, with differential blood counts revealing conditions like infections, leukemia, and specific cancers. Traditionally, such counts are performed manually by technicians using microscopes to assess the prevalence of different WBC types in smear slides, a process that is both time-consuming and labor-intensive [28]. As illustrated in Figure 1, different types of WBCs include Neutrophils, Lymphocytes, Eosinophils, and Monocytes.

2. Literature Review

2.1. Overview of White Blood Cells

White blood cells (WBCs) are essential components of the immune system, grouped into four main categories: Neutrophils, Lymphocytes, Eosinophils, and Monocytes.[28] These cells are instrumental in the body’s defenses against infections, diseases, and foreign intrusions.

Neutrophils act as a primary defense mechanism against infections, constituting a significant portion of the WBC count. Lymphocytes are critical to the adaptive immune response, whereas Eosinophils combat parasites and allergens. Monocytes are crucial for phagocytosis. Figure 2 provides a 3D visualization of these cells.

2.2. Disorders of White Blood Cells

White blood cell (WBC) disorders can be primarily categorized into two types: proliferative disorders and leukopenias. Proliferative disorders are characterized by an increase in the number of WBCs, while leukopenias are marked by a decrease in WBC counts.

2.2.1. Proliferative Disorders

Proliferative disorders lead to an abnormal increase in white blood cells, with the most common condition being leukocytosis. Leukocytosis can be triggered by various factors including:

• Bacterial or viral infections, which typically induce a rapid increase in WBCs as a part of the immune response.
• Inflammation and allergic reactions that stimulate the production of white blood cells.
• Cancer, particularly hematologic malignancies like leukemia, which directly result in the overproduction of abnormal white cells.

Other contributing factors may include smoking, certain medications, and genetic predispositions.[56]

2.2.2. Leukopenias

On the other end of the spectrum, leukopenias are characterized by a reduced WBC count, significantly impacting the body’s ability to fight infections. Key disorders under this category include:

• Autoimmune Neutropenia: This condition arises when antibodies produced by the body’s immune system mistakenly target and destroy neutrophils. It is often associated with autoimmune diseases such as Crohn’s disease and rheumatoid arthritis.[57]
• Severe Congenital Neutropenia: Caused by genetic mutations, this rare disorder features marked susceptibility to recurrent bacterial infections due to dramatically reduced neutrophil counts.[58]
• Cyclic Neutropenia: This genetic disorder causes fluctuations in neutrophil levels, typically cycling every three weeks, leading to periodic susceptibilities to infections.[59]
• Chronic Granulomatous Disease: Affects multiple types of white blood cells including neutrophils and monocytes. Patients with this inherited disorder suffer from frequent bacterial and fungal infections due to the cells’ inability to effectively kill certain types of bacteria and fungi.[60]
• Leukocyte Adhesion Deficiency (LAD) Syndrome: This rare genetic condition prevents white blood cells from migrating to the site of infection, resulting in severe, recurrent bacterial infections.[61]

2.3. Advancements in WBC Classification and Localization

Research efforts have significantly advanced the classification and localization of WBCs, employing both traditional machine learning and modern deep learning techniques to enhance the accuracy and efficiency of these processes.

Traditional Machine Learning Methods

Ongun et al. [30] developed a multi-step system for WBC classification, involving segmentation using active contours and morphological operations, followed by feature extraction focusing on shape, texture, and color. Classification was performed using methods including K-Nearest Neighbor (KNN) and Support Vector Machine (SVM), with SVM achieving an accuracy of 91.03%.

Another approach by Sinha and Ramakrishnan [34] utilized K-means for segmentation and extracted features for classification via Neural Networks (NN), KNN, and SVM, reaching a 94.1% accuracy on a modest dataset.

Hu et al. [35] proposed a segmentation method using wavelet transforms, applying different processing on high and low frequency components to achieve enhanced segmentation.

Segmentation Innovations

Shahin et al. [68] introduced a technique utilizing neutrosophic similarity to segment WBCs, adapting the approach for various color spaces. Sadeghian et al. [69] combined several image processing steps to segment nuclei and cytoplasm, employing techniques such as GVF snake algorithms and Zack thresholding, resulting in high accuracy for nucleus segmentation.

A notable study by Hegde et al. [72] combined color channel manipulations with TissueQuant algorithms for segmenting and classifying nuclei using shape and texture features, achieving an accuracy of 96.64% on a sample set of 117 images.

Innovative Neural Architectures

Research has increasingly incorporated deep learning for enhanced accuracy in WBC classification. Shahin et al. [73] explored the efficacy of pre-trained networks like OverfeatNet, AlexNet, and VGG, alongside novel architectures, with deep learning models surpassing traditional methods in accuracy.

Kutlu et al. [74] implemented a CAD system employing Single Shot Detector (SSD) for detection and various pre-trained networks for classification, achieving remarkable success with an accuracy rate up to 97%.

Tiwari et al. [77] introduced a simpler Double Convolution Layer Neural Network (DCLNN) architecture, which provided competitive results with an 88% accuracy on a large dataset.

Emerging Technologies

Baydilli et al. [78] utilized capsule networks, a novel approach offering robust feature extraction through dynamic routing between capsules, showcasing the potential for visual inspection of reconstructed features.

Tran et al. [80] employed the SegNet architecture for segmenting both WBCs and red blood cells, demonstrating the utility of encoder-decoder structures in medical image segmentation with a mean intersection over union of 79%.

These studies reflect the dynamic evolution of techniques from traditional machine learning to sophisticated deep learning models, each contributing uniquely to the field of medical image analysis for WBC classification and localization.

3. Methods

This Section discusses the methodologies employed in the experiments, which are categorized into three primary areas:

• Region of Interest (ROI) extraction: Various techniques for identifying and defining the ROIs that contain WBCs in the images are examined. These methods are utilized to evaluate model performance using both entire images and cropped images based on the extracted ROI.
• Customized deep learning models: Two CNN architectures specifically developed for classifying white blood cell (WBC) data are detailed.

3.1. Region of Interest Extraction

Two methodologies are implemented to extract the Region of Interest (ROI) in this study: a color-based method and a deep learning-based method.

3.1.1. Color-Based ROI Extraction

The color-based method exploits the consistent color range of white blood cells (WBCs) across the dataset as shown in Figure 3. The ROI extraction algorithm operates through the following steps:

• Apply a Gaussian blur (size $3\times 3$) to reduce image noise.
• Convert the image to the HSV color space.
• Binarize the image within a predefined color range (typically RGB (80, 60, 140) to (255, 255, 255)) that encompasses all WBCs.
• Identify the largest contour in the binary image using a contour tracing algorithm.

The contour tracing algorithm, designed to outline the largest object by its external borders, is described below:

Algorithm 1: Contour Tracing Algorithm

3.1.2. Deep Learning-Based ROI Extraction

The deep learning approach utilizes a Mask R-CNN model, which builds upon the Faster R-CNN by adding a branch for binary mask prediction. This enables precise pixel-wise object segmentation. Mask R-CNN incorporates the ROI Align technique to finely map the predicted masks to the original image coordinates through bilinear interpolation. The architecture of the Mask R-CNN and its operational scheme are illustrated in Figure 4.

3.2. Customized Deep Learning Models

Two non-pretrained, customized deep learning models are developed, distinct from the standard pretrained models.

3.2.1. Customized Model 1

The first model, a traditional convolutional neural network (CNN), consists of four principal blocks:

• Block 1: Includes a $3\times 3$ convolutional layer producing 32 feature maps, followed by a $2\times 2$ max pooling layer and a dropout layer with a dropout rate of 0.2 to mitigate overfitting.
• Block 2: Similar to Block 1 but outputs 64 feature maps.
• Block 3: Extends feature map generation to 128 and follows the same structure as previous blocks.
• Block 4: Incorporates a flatten layer followed by three dense layers with sizes 64, 128, and 64, respectively. It employs ReLU activation and concludes with a softmax output layer with four nodes.

Figure 5 depicts the architecture of this customized CNN model.

3.2.2. Customized Model 2

The second model is a deeper architecture, each building block comprising six layers:

• Layer 1: Features n $1\times 1$ convolutional filters for channel-wise pooling and dimensionality reduction.
• Layer 2: A batch normalization layer to facilitate higher learning rates and faster convergence, acting also as a regularizer.
• Layer 3 and 4: Replicate the structure of the first two layers.
• Layer 5: Includes n $3\times 3$ convolutional filters.
• Layer 6: Another batch normalization layer.

The ’same’ convolution mode maintains the input shape, with the output being a concatenation of the features from Layer 4 and Layer 6.

The full model begins with three introductory layers—a batch normalization layer, a $5\times 5$ convolution layer with 12 filters, another batch normalization—followed by five repetitions of the described building block and $2\times 2$ max pooling, using progressively adjusted filter counts (12, 24, 32, 24, 18, and finally 12). A dense layer with sigmoid activation finalizes the model. Figure 6 displays both the building block and the complete model architecture.

4. Results

4.1. Dataset Description

The dataset utilized in this study comprises 12,444 microscopic images of blood cells, categorized into four classes of white blood cells (WBCs): eosinophils, lymphocytes, monocytes, and neutrophils. The images were obtained from the publicly available dataset hosted on Kaggle, which can be accessed at this link. The data is split into training (75%) and testing (25%) sets. The distribution across the four classes in both datasets is shown in Table 1. All images are 24-bit in depth with a resolution of $320\times 240$ pixels. Figure 1 illustrates samples from the four WBC classes.

4.2. ROI Extraction Experiments

Two distinct methodologies were employed to extract Regions of Interest (ROIs) from microscopic images of White Blood Cells (WBCs): a color-based approach and a deep learning-based method using Mask R-CNN.

4.2.1. Color-Based ROI Extraction

The color-based technique computes bounding boxes by identifying the extremal coordinates that enclose the WBCs based on color thresholds. The effectiveness of this approach was quantified using the Intersection over Union (IoU) metric, which compared the algorithm-generated bounding boxes to the ground truth.

Results indicated an average IoU of 0.59, suggesting moderate accuracy in delineating WBC regions. The IoU values’ distribution is illustrated in Figure 7, and examples of the algorithm outputs versus the ground truth bounding boxes are shown in Figure 8, highlighting the method’s reliance on accurate color differentiation. Table 2 provides a summary of the statistical measures of IoU for the color-based ROI extraction, including mean, minimum, maximum, standard deviation, and median values.

4.2.2. Mask R-CNN ROI Extraction

Mask R-CNN was applied using pretrained networks, specifically VGG16 and ResNet50, described in SubSection 3.1.2. The experiment was conducted over 50 epochs with a batch size of 32, using the Adam optimizer. The dataset was partitioned into 220 training images and 144 testing images.

This approach yielded superior results, achieving mean IoU scores of 0.71 with VGG16 and 0.69 with ResNet50. Detailed IoU statistics are provided in Table 3, and the distribution is depicted in Figure 9. Examples of extracted ROIs using Mask R-CNN are shown in Figure 10, demonstrating the method’s efficacy in leveraging higher-level features to enhance extraction accuracy.

4.3. Classification Experiments

Extensive classification experiments were conducted using various CNN architectures, including VGG16, ResNet50, and custom models, on both whole images and extracted ROIs. The performance was quantitatively assessed based on classification accuracy, and improvements were noted when models utilized extracted ROIs. This section explores the classification performance of four different WBC types using various CNN architectures as detailed in the following subsections. The results from whole images and regions of interest (ROIs) are compared, with metrics like accuracy and confusion matrices being key evaluative factors.

4.3.1. VGG Classification

Moving to a more complex architecture, VGG16, pre-trained on ImageNet and adapted here by modifying the final layers, provided a noticeable improvement. The model reached an accuracy of 48.4% after 20 training epochs. Although better than LeNet, VGG16 still leaves room for performance enhancement, potentially through more advanced models or further tuning.

4.3.2. ResNet Classification

The ResNet50 model, known for its deep architecture and ability to prevent gradient vanishing, was also pre-trained on ImageNet. With adjustments similar to VGG16, ResNet50 significantly improved the classification accuracy to 74.6%. This model’s deeper layers and residual connections likely helped capture more complex patterns in the WBC images.

4.3.3. Inception Classification

Lastly, the Inception network, utilizing multiple filter sizes at each layer, was tested. Despite its sophisticated architecture, it only achieved 43% accuracy, underscoring some challenges it faced, perhaps due to the distinctive nature of medical imagery compared to ImageNet. Interestingly, it failed to classify any images as Neutrophil, indicating potential issues with feature representation at different scales.

4.3.4. Classification Using the First Customized Model with Whole Images

The initial customized model achieved an 80% accuracy, significantly outperforming the pretrained networks. This model was specifically designed for WBC image analysis, emphasizing simplicity and efficiency in learning from the dataset’s unique characteristics. The performance details are illustrated in the confusion matrix shown in Table 4.

4.3.5. Classification Using the Second Customized Model with Whole Images

This more complex customized model recorded the highest accuracy of 89.5%, demonstrating its advanced capability to discern intricate features from WBC images. For detailed performance metrics, see Table 5.

4.4. Impact of ROI Extraction on Classification Outcomes

This section explores how extracting regions of interest (ROIs) that contain white blood cells (WBCs) prior to classification can enhance feature relevance and diminish background noise that might otherwise impair classification accuracy.

The superior performance of the Mask R-CNN algorithm using the VGG pretrained network in extracting ROIs, as discussed in 3.1.2, led to its selection for pre-classification ROI extraction. The subsequent subsections discuss how this step has improved the performance of both pretrained and custom architectures.

4.4.1. Impact of ROI Extraction on Architectural Performances

Implementing ROI extraction has significantly enhanced classification accuracies across different architectures:

VGG16 Results: With the VGG16 architecture, accuracy improved from 48.8% using whole images to 64% with ROIs. Notable improvements were seen in the LYMPHOCYTE and EOSINOPHIL classes, although the NEUTROPHIL class accuracy was notably lower due to misclassifications as EOSINOPHIL.

ResNet50 Results: ROI extraction with ResNet50 led to a substantial gain in performance, increasing accuracy from 74.6% to 84%. This model provided the highest accuracies for LYMPHOCYTE and EOSINOPHIL classes. The most common error involved misclassifying NEUTROPHIL images as MONOCYTE.

Inception Architecture Results: The Inception network also saw improvements with ROI extraction, where classification accuracy rose from 43% to 55%. However, the system struggled with distinguishing between NEUTROPHIL and EOSINOPHIL classes.

These results underscore the effectiveness of ROI extraction in enhancing feature relevancy, thereby boosting the overall classification performance of various deep learning architectures.

4.4.2. Enhancements from ROI Extraction in Customized Architectures

First Customized Architecture: The ROI extraction had a substantial impact on our first customized architecture, boosting accuracy from 81% to 88%. The primary issue observed was misclassifications of MONOCYTE as NEUTROPHIL. Detailed confusion matrix is presented in Table 6.

Second Customized Architecture: Similarly, the second customized architecture demonstrated the highest improvement, with accuracy improving from 89% to 92%. Misclassifications primarily occurred with LYMPHOCYTE being classified as MONOCYTE. This model’s results, which are summarized in Table 7, confirm its superiority and suitability for further experiments involving noise resilience in image classification.

4.5. Noise Robustness Testing

In this section, we explore the impact of introducing Gaussian noise into our testing dataset to evaluate the robustness of the second customized model, which previously demonstrated the highest accuracy. The study focuses on Gaussian noise due to its prevalence in real-world medical imaging scenarios, where electronic and sensor noise often distorts images. Figure 11 displays examples of images at different $\alpha$ values, illustrating how Gaussian noise affects image clarity. The effect of the noise on classification accuracy is quantified and presented below:

4.5.1. Gaussian Noise

Gaussian noise, characterized by its statistical properties of having a mean of zero and a defined variance, closely simulates the electronic noise encountered in medical imaging devices. To simulate this effect, we added varying levels of Gaussian noise to the test images and assessed the model’s performance.

The Gaussian noise was generated and added to the images using the following equation:

$$N=I+G*\alpha$$

(1)

where I represents the original image, G is the generated Gaussian noise, and $\alpha$ is the noise factor controlling the signal-to-noise ratio (SNR). We experimented with $\alpha$ values ranging from 0.05 to 0.35 to cover a broad spectrum of common noise intensities.

The impact on the model’s accuracy as shown in Figure 12 was significant, demonstrating a decline as the noise level increased. At $\alpha =0.05$, the model maintained an accuracy of 81%, which dropped to 51% at $\alpha =0.35$. These results are critical as they highlight the need for further refining the model to handle common noise levels effectively, ensuring reliability in practical medical settings.

This experiment underlines the importance of noise handling capabilities in automated diagnostic systems, particularly in environments where electronic interference is prevalent. Future work will aim to enhance the model’s noise resilience, potentially incorporating noise reduction techniques at the preprocessing stage.

5. Discussion

The results demonstrated significant improvements in classification accuracy when ROIs were used, underscoring the importance of focused image analysis in medical diagnostics. Custom models, tailored specifically for the task, outperformed pre-trained models, likely due to better feature alignment with the target domain. Furthermore, the resilience of these models to image noise highlights their potential for deployment in varied clinical settings.

5.1. Region of Interest Extraction

Two methodologies were explored for extracting the region of interest (ROI) from microscopic images of blood cells: a color-based technique and the application of Mask R-CNN using a VGG pretrained architecture. The intersection over union (IoU) metric was used to evaluate both methods, with Mask R-CNN demonstrating superior performance, achieving a mean IoU of 0.71 compared to 0.59 for the color-based method. Detailed performance metrics are presented in Table 8.

5.2. Classification Experiments

Both pretrained and custom models were assessed using whole images and extracted ROIs. Custom models, especially when utilizing extracted ROIs, consistently outperformed pretrained models. The greatest accuracy improvements were noted when models were applied to extracted ROIs, suggesting that ROI-focused features enhance model learning capabilities. The comparative results of different architectures are shown in Table 9.

5.2.1. Comparative Analysis

The proposed models were benchmarked against other studies, reflecting not only higher or comparable accuracy but also robustness given the large dataset used. Notably, while some studies achieved slightly higher accuracies, they often relied on substantially smaller datasets, limiting their generalizability. The comparative analysis is summarized in Table 10.

Overall, the study not only advances the understanding of effective techniques in medical image analysis but also sets a foundation for future research to explore novel architectures and fusion techniques that could offer even better performance and robustness.