Segmenting Tumor Gleason Pattern using Generative AI and Digital Pathology: Use Case of Prostate Cancer on MICCAI Dataset

1. Introduction

Annually, more than 1,414,259 cases of prostate cancer (PCa) are diagnosed, with a staggering 375,304 individuals succumbing to the disease [1]. Timely detection and prompt intervention play pivotal roles in efficiently managing prostate cancer and enhancing the prospects of favorable results. Prostate cancer (PCa) often presents a challenging scenario due to its propensity for non-aggressive behavior, posing a dilemma regarding the necessity of escalated treatment measures to mitigate the risk of disease progression in patients. Addressing this challenge, the Gleason Grading System categorizes tumors into distinct risk groups based on their pathological characteristics, a framework endorsed by both the International Society of Urological Pathology (ISUP) and the World Health Organization (WHO) [2]. Typically, pathologists perform manual examination and assessment of slides stained with Hematoxylin and Eosin (HE) to ascertain tumor status. The Gleason grading system for prostate cancer originated in the late 1960s and underwent further refinement throughout the subsequent decade. By the late 1980s, it had established itself as the predominant pathological grading system for prostate cancer diagnosis. Staging and grading are paramount in determining prognosis and devising tailored treatment strategies based on the outcomes observed in comparable patients [3]. Different methods are available for PCa staging and detection. However, the most accurate approach is for skilled pathologists to examine stained biopsy tissue under a microscope. A Gleason grade of 1 to 5 is given to each tissue location based on the visible histological patterns. A tissue with the most prominent pattern (Gleason grade of 4) and the second most significant pattern (Gleason grade of 3) will have a Gleason score of 4+3. The final Gleason score is reported as the sum of the most and second most prominent patterns. Pathologists commonly conduct manual examination and assessment of slides stained with Hematoxylin and Eosin (HE). Nevertheless, the manual quantification of tumor sections, especially across various Gleason scores, may entail subjective judgments and prove challenging to consistently execute among diverse observers. This inherent subjectivity has the potential to introduce discrepancies in slide interpretation (refer to Figure 1), thereby compromising the accuracy and dependability of final diagnoses and treatment strategies. Hence, the adoption of computerized pathology and deep learning methodologies holds promise for prostate cancer diagnosis from biopsies, offering potential relief from manual workload for pathologists and mitigating issues associated with individual subjectivity. In recent studies [4], both deep learning and non-deep learning approaches have been employed by researchers for anomaly detection in medical images. Deep learning has elevated the state-of-the art scores across a multitude of domains [5], including image processing, video analysis, as well asspeech and audio recognition. In medical domain, it has shown promising results for tumor detection and breast cancer [6,7]. The paper [8] involved training deep learning models such as DeepLabv3, U-Net, and PSPNet on 60 the MICCAI dataset. The reported results indicate that PSPNet achieved the highest F1 score of 0.827 and Jaccard Index score of 0.836 among the models evaluated. On the other hand, DeepLabv3 achieved an F1 score of 0.798 and a Jaccard Index score of 0.821. Another work [9] utilized the DeepLabV3+ model with MobileNetV2 backbone on the Gleason 2019 challenge dataset, achieving a mean Cohen’s quadratic kappa score of 0.56. [10] leveraged two datasets—Harvard and Gleason Challenge 2019—to train and validate the model. Using Unet based architecture with different encoders produced average cohen’s kappa score of 0.728 and F1 score of 0.732 on both the dataset. Xu et al. [11] proposed a weakly supervised approach for histopathology cancer image segmentation and classification, demonstrating significant improvements in the accuracy and efficiency of cancer detection in medical images. This method effectively leverages limited annotated data to train robust models for accurate cancer diagnosis. The study underscores the potential of weakly supervised learning in enhancing histopathological image analysis.

2. Proposed Methods

Image segmentation involves partitioning an image into multiple segments, assigning each pixel to a specific object type. Typically, this process employs a fundamental architecture comprising an encoder and a decoder. The encoder discerns features from the image using filters, while the decoder produces the ultimate output, often a segmentation mask outlining the objects. Many architectures adopt this standard structure or its modifications for image segmentation tasks.

Following image processing, the training of five architectures, namely UNet, DeepLabV3, UNet++, DeepLabV3+ and proposed GAN based network, was conducted. After hyperparameter tuning a cross-entropy loss function with a learning rate of 0.0005 was employed, utilizing ImageNet pre trained weights. Evaluation of the results was performed using two metrics: F1 score and Jaccard Index, facilitating comparison. The code was based on Pytorch library, and the models were trained on A100 gpu for 20 epochs.

1) UNet: The UNet architecture is particularly designed for semantic segmentation tasks, particu larly in biomedical image analysis. The U-Net architecture encompasses a contracting pathway for context extraction and an expanding pathway for precise localization. In the contracting pathway, standard convolutional and pooling layers are employed, while the expanding pathway integrates upsampling and convolutional layers. Incorporating skip connections enables the fusion of feature maps from both pathways, facilitating the preservation of spatial information and refinement of segmentation results. Training Unet produced an F1 score of 0.843 and Jaccard Index of 0.66 on the test dataset.

2)DeepLabV3: DeepLabv3’s pivotal advancement revolves around its implementation of atrous convolution, also referred to as dilated convolution, featuring multiple rates within the atrous spatial pyramid pooling (ASPP) module. This integration facilitates the efficient capture of multi-scale context information, enhancing object segmentation across various scales. Notably, DeepLabv3 has attained cutting-edge performance levels on established benchmark datasets for semantic segmentation tasks [14,15]. DeepLabV3 produced an F1 score of 0.86 and Jaccard Index of 0.70 on the test dataset.

3)UNet++: UNet++ extends the original UNet architecture, addressing its limitations by introducing a nested and densely connected U-shaped network. This innovation involves incorporatingnested and dense skip pathways, departing from the single skip connections of the original UNet. UNet++ employs multiple skip connections at various scales, forming a nested architecture that enhances the network’s ability to capture multi-scale contextual information for more precise segmentation. These connections promote feature reuse and gradient flow, mitigating the vanishing gradient issue during training and enhancing the network’s capacity to learn intricate features. Unet++ produced an F1 score of 0.853 and Jaccard Index score of 0.680.

4)DeepLabV3+: DeepLabV3+ builds upon DeepLabV3 by introducing an encoder-decoder architecture. This addition addresses the issue of lost spatial information during encoding in DeepLabV3. While DeepLabV3 relies on ASPP for capturing multi-scale context, DeepLabV3+ utilizes the decoder module to refine the segmentation maps and achieve more precise boundary delineation, ultimately leading to improved segmentation accuracy. DeepLabV3+ produced an F1 score of 0.824 131 and Jaccard Index score of 0.622.

5)Proposed segmentation: We used a Generative segmentation that combines two powerful techniques, generative adversarial networks (GANs) and image segmentation, to improve the ac curacy of segmenting images. In this method, a UNet model acts as the generator, transforming low-resolution images into detailed segmented images. Instead of comparing with real images, a discriminator evaluates how close the generated segmented images are to the actual segmented images. The process is guided by a loss function that ensures the generated images match the real ones as closely as possible. By iteratively refining the generator based on feedback from the discriminator, this approach produces highly accurate segmentations with realistic details.

For the generator loss ${\mathcal{L}}_{\mathcal{G}}$

$${\mathcal{L}}_{\mathcal{G}}=\mathrm{MSE}\left(\mathrm{Dis}\left(\widehat{I},I\right),1\right)+\mathsf{\lambda }\cdot \mathrm{MAE}\left(\widehat{I},I_{gt}\right)$$

For the discriminator loss ${\mathcal{L}}_{\mathcal{D}}$

$${\mathcal{L}}_{\mathcal{D}}\mathrm{ }=\mathrm{ }\mathrm{MSE}\left(\mathrm{Dis}\left({\mathrm{I}}_{\mathrm{g}\mathrm{t}},\mathrm{ }\mathrm{I}\right),\mathrm{ }1\right)\mathrm{ }+\mathrm{ }\mathrm{MSE}\left(\mathrm{Dis}\left(\widehat{\mathrm{I}},\mathrm{ }\mathrm{I}\right),\mathrm{ }0\right)$$

In the equation ${\mathcal{L}}_{\mathcal{G}}$is the loss for the generator, ${\mathcal{L}}_{\mathcal{D}}$ is the loss for the discriminator, MSE stands for Mean Squared Error, MAE stands for Mean Absolute Error, Dis is the discriminator, λ is a weight parameter for balancing the MAE term in the generator loss, $\widehat{I}$ represents the segmented image, I represents the input image, $I_{gt}$ represents the ground-truth image, 1 represents a tensor of ones, and 0 represents a tensor of zeros.

Figure 3. The Block Diagram of the Proposed Segmentation Algorithm.

Figure 3. The Block Diagram of the Proposed Segmentation Algorithm.

4. Results

The performance of various deep learning architectures for prostate cancer segmentation and grading was evaluated on the MICCAI-2019 dataset. Table 1 summarizes the performance metrics for each model, including the Jaccard Index, F1 Score, and Accuracy.

The UNet architecture achieved an F1 score of 0.843 and a Jaccard Index of 0.660. While the model showed considerable performance, it was outperformed by other architectures in this study. DeepLabV3, which integrates atrous convolution for capturing multi-scale context, attained an F1 score of 0.860 and a Jaccard Index of 0.700. This model’s ability to handle objects at different scales proved beneficial for the task at hand, leading to an improved segmentation performance over the UNet. UNet++, an extension of the original UNet with nested and densely connected skip pathways, achieved an F1 score of 0.853 and a Jaccard Index of 0.680. The nested architecture of UNet++ facilitated the capture of multi-scale contextual information more effectively than the traditional UNet, resulting in better segmentation accuracy. DeepLabV3+, which builds upon DeepLabV3 by incorporating an encoder-decoder architecture, produced an F1 score of 0.824 and a Jaccard Index of 0.622. Although it introduced an effective refinement step through the decoder module, its performance did not surpass that of DeepLabV3, indicating the need for further optimization. 163 The proposed GAN-based segmentation model outperformed all other architectures with an F1 score of 0.872 and a Jaccard Index of 0. 777.The results demonstrate that the GAN-based model’s ability to leverage adversarial learning provides a distinct advantage in achieving high segmentation accuracy. The performance of each model was further assessed based on their accuracy in predicting the correct Gleason grades. The GAN-based model achieved the highest accuracy of 0.935, reflecting its superior capability in distinguishing between different Gleason grades accurately. In contrast, DeepLabV3 and UNet++ also demonstrated high accuracies of 0.920 and 0.919, respectively, confirming their effectiveness in this task. In conclusion, the proposed GAN-based segmentation model has shown remarkable performance in the automated Gleason grading of prostate cancer, significantly outperforming traditional models. The integration of adversarial training within the segmentation framework has proven to be a pivotal factor in enhancing segmentation accuracy and reliability. Future work will aim to further refine these models and explore additional strategies to improve their performance, ultimately contributing to more accurate and efficient prostate cancer diagnosis.

5. Discussion

The findings from this study highlight the significant potential of deep learning techniques in automating the Gleason grading of prostate cancer. The process of pre-training the encoder components of segmentation models on extensive datasets such as ImageNet has been instrumental. This approach allows the models to extract transferable features that can be fine-tuned for specific tasks using smaller datasets. This transfer learning technique has proven effective in enhancing the performance of the models evaluated in this study. The GAN-based segmentation model demonstrated superior performance metrics, highlighting the efficacy of adversarial training in improving segmentation accuracy. The results indicate that traditional models like UNet and DeepLabV3, while effective, do not capture the intricate details as efficiently as the GAN-based model. The GAN’s ability to generate more precise segmentations can be attributed to the adversarial training process, which continuously pushes the generator to produce outputs that are indistinguishable from the ground truth. This capability is crucial in medical image analysis, where accurate segmentation is vital for reliable diagnosis and treatment planning. Future research will focus on exploring alternative architectures and strategies to further enhance the performance of automated Gleason grading models. One promising direction is the use of ensemble methods, which combine the strengths of multiple models to improve overall accuracy and robustness. Additionally, incorporating domain-specific knowledge into the models could provide further improvements in performance by leveraging expert insights and annotations. Expanding the dataset by integrating data from multiple sources will be crucial in developing robust models capable of generalizing across diverse patient populations. This diversification will enable the models to capture domain-invariant features more effectively, leading to robust performance in real-world clinical settings. The ultimate objective is to transition these advanced models into practical tools that can assist pathologists in clinical practice. Developing a web-based application based on these models will provide an accessible and user-friendly platform for automated Gleason grading. This tool has the potential to significantly impact clinical workflows by reducing the manual workload for pathologists, improving diagnostic consistency, and providing timely and precise diagnostic support. Upon achieving satisfactory performance metrics, the next steps will involve rigorous validation of the models in clinical settings to ensure their reliability and efficacy. Collaborating with medical professionals and integrating their feedback will be essential in refining the application and ensuring it meets the needs of clinical practice.

6. Conclusions

The proposed approach, framed as a classification problem and utilizing ResNet models in conjunction with the Diagset dataset, highlights its potential as a robust tool for prostate cancer diagnosis and Gleason grading. The integration of machine learning and deep learning techniques represents a pivotal advancement in prostate cancer diagnosis and grading. The proposed methodology, centered around convolutional neural networks operating on histopathological scans at multiple magnification levels, presents a comprehensive and robust approach to prostate cancer diagnosis and Gleason grading. Leveraging the rich resource provided by the Diagset dataset, this research offers insights into the intricate features underlying prostate cancer pathology across different Gleason grades and tissue types. The comprehensive evaluation of the proposed model’s performance across various magnification levels reaffirms its adaptability and reliability in accurately assessing prostate cancer pathology. The consistent high accuracies achieved by the ResNet models further validate their stability and efficacy, positioning them as valuable assets in clinical practice for prostate cancer diagnosis and Gleason grading. Moving forward, an intriguing prospect involves transforming these models into an accessible web application, which would democratize access to advanced diagnostic tools. This application would allow healthcare professionals to seamlessly integrate machine learning into clinical practice, offering features such as real-time analysis of histopathological images and automated Gleason grading. Additionally, the application would facilitate integration with electronic medical record systems, enabling streamlined patient management and improved clinical workflows. By providing an intuitive interface, the web application would ensure that sophisticated machine learning tools are accessible to clinicians without requiring extensive technical expertise, thereby enhancing diagnostic accuracy and efficiency in diverse healthcare settings.

This study establishes a solid foundation for advancing prostate cancer diagnosis and Gleason grading through machine learning. Future research should explore additional algorithms, such as Vision Transformers, and improve model efficiency to further enhance accuracy, scalability, and clinical utility. By addressing these areas, future work can expand the capabilities and applications of these models, ultimately benefiting healthcare providers and patients alike.