A Survey on Explainable Artificial Intelligence (XAI) Techniques for Visualizing Deep Learning Models in Medical Imaging

1. Introduction

Medical imaging (MI) is crucial in modern healthcare, providing essential information for diagnosing, treating, and monitoring diseases. Traditional image analysis relies on handcrafted features and expert knowledge, which can be time-consuming and prone to errors. Machine learning (ML) methods, such as Support Vector Machines (SVMs), decision trees, random forests, and logistic regression, have improved efficiency and reduced errors in tasks like image segmentation, object detection, and disease classification. These methods require manual feature selection and extraction. Recent advancements in deep learning (DL) have revolutionized medical image analysis by automatically learning and extracting hierarchical features from large volumes of medical image data [1,2,3,4,5,6,7,8,9]. It has provided healthcare professionals with valuable insights for more accurate diagnoses and improved patient care.

Despite their exceptional performance, DL models in MI face challenges related to interpretability and transparency [10,11,12]. Their opaque decision-making processes raise concerns about reliability in healthcare, where explanations for diagnostic decisions are crucial. Interpretability in artificial intelligence (AI)-driven healthcare models encourages trust and reliability by enabling practitioners to understand and verify model outputs. It ensures ethical and legal accountability by clarifying diagnostic and recommendation processes, which is crucial for compliance and ethical standards. Interpretability supports clinical decision-making by providing insights into AI recommendations, leading to better-informed patient care. Additionally, transparent models help identify biases and errors, enhancing fairness and accuracy.

Ongoing efforts to enhance the interpretability of DL models in MI involve developing techniques to clarify how these models reach conclusions [13,14,15]. Researchers are creating explanations and visualizations to make the decision-making processes of these models more understandable and trustworthy for healthcare practitioners. This paper contributes to this area through several key aspects:

• Comprehensive Survey: This paper offers a thorough survey of innovative approaches for interpreting and visualizing DL models in MI. It includes a wide array of techniques aimed at enhancing model transparency and trust.
• Methodological Review: We provide an in-depth review of current methodologies, focusing on post-hoc visualization techniques such as perturbation-based, gradient-based, decomposition-based, trainable attention (TA)-based methods, and vision transformers (ViT). Each method’s effectiveness and applicability in MI are evaluated.
• Clinical Relevance: This paper emphasizes the importance of interpretability techniques in clinical settings, showing how they lead to more reliable and actionable insights from DL models, thus supporting better decision-making in healthcare.
• Future Directions: We outline future research directions in model interpretability and visualization, highlighting the need for more robust and scalable techniques that can manage the complexity of DL models while ensuring practical utility in medical applications.

We provide a thorough survey of innovative approaches for interpreting and visualizing DL models in MI. As shown in Figure 1, we illustrate various DL models and techniques used in MI, highlighting methods like CNNs, RNNs, GANs, transformer-based architectures, autoencoders, LIME, Grad-CAM, and Layer-Wise Relevance Propagation (LRP),attention-based methods, and vision transformers (ViTs).

The rest of this paper is divided into four sections, with multiple subsections within each of them. Section 3 is focused on interpreting model design and workflow. Section 4 Visualizing DL models in MI. Section 5 presents an overview of post-hoc interpretation and Visualization Techniques. Comparison of Different Interpretation Methods discussed in Section 6 and concludes the work with current challenges and future directions in Section 8.

2. Research Methodology

A comprehensive review of explainable AI in medical image analysis was published by [16,17,18,19,20]. While this review covers a broad range of topics, some critical areas, such as research on trainable attention (TA)-based methods, vision transformers, and their applications, have been overlooked. Our review aims to fill this gap by providing an extensive overview of various domains within medical imaging, addressing key aspects such as Domain, Task, Modality, Performance, and Technique.

This research employs the Systematic Literature Review (SLR) method, which involves several stages. The research questions guiding this study are:

• What innovative methods exist for interpreting and visualizing deep learning models in medical imaging?
• How effective are post-hoc visualization techniques (perturbation-based, gradient-based, decomposition-based, TA-based, and ViT) in improving model transparency?
• What is the clinical relevance of interpretability techniques for actionable insights from deep learning models in healthcare?
• What are the future research directions for model interpretability and visualization in medical applications?

The survey examines over 300 recent papers on explainable AI (XAI) in medical image analysis. Relevant contributions were identified using keywords like “deep learning", “convolutional neural networks", “medical imaging", “surveys", “interpretation", “visualization" and “review". Sources included ArXiv, Google Scholar, and Science Direct, focusing on titles. Studies without results on medical image data or using only standard neural networks with manually designed features were excluded. In cases of similar work, the most significant publications were selected.

The findings will be comprehensively presented, including a detailed description of the research methodology for replication. The literature search results, relevant articles, and their quality evaluations will be summarized in overview tables. Drawing from expertise in applying XAI techniques to medical image analysis, ongoing challenges, and future research directions will be discussed.

3. Interpreting Model Design and Workflow

Interpreting model design and workflow involves examining the hidden layers of convolutional neural networks (CNNs). This can be achieved through methods such as:

• Autoencoders for Learning Latent Representations
• Visualizing High-Dimensional Latent Data in a Two-Dimensional Space
• Visualizing Filters and Activations in Feature Maps

3.1. Autoencoders for Learning Latent Representations

Autoencoders (AE) are DL models for unsupervised feature learning [21], with applications in anomaly detection [22], image compression [23], and representation learning [24]. They consist of an encoder creating latent representations and a decoder reconstructing images. Variants include variational autoencoders (VAE) and adversarial autoencoders (AAE). In medical imaging, AEs detect abnormalities by comparing input images with reconstructions and highlighting high reconstruction loss areas. For instance, VAE has reconstructed OCT retinal images to detect pathologies [25], and AAE has localized brain lesions in MRI images [26]. Convolutional AEs have detected nuclei in histopathology images by combining learned representations with thresholding [27].

3.2. Visualizing High-Dimensional Latent Data in a Two-Dimensional Space

CNNs produce high-dimensional features, making visualization challenging. Dimensionality reduction techniques like PCA and tSNE simplify this data. PCA performs linear transformations, while tSNE [28] uses nonlinear methods to map high-dimensional data to lower dimensions. tSNE is effective for visualizing patterns and clusters, such as in abdominal ultrasound and histopathology image classification. The constraint-based embedding technique [29], using a divide-and-conquer algorithm to preserve k-nearest neighbors in 2D projections, has assessed deep belief networks separating brain MRI images of schizophrenic and healthy patients, though both tSNE and constraint-based embedding struggled with raw data separation.

3.3. Visualizing Filters and Activations in Feature Maps

A convolutional block extracts local features from input images through convolution filters, ReLU or GELU activations, and pooling layers. Filter visualization reveals CNN’s feature extraction capabilities, with initial layers capturing basic elements and later layers capturing intricate patterns. In medical imaging, filter visualization compares filters in CADe for 3D CT images [5]. Larger filters offer more insights but require more memory. Feature map visualization, representing layer outputs after activation, highlights active features and can indicate training issues. It is used in tasks like skin lesion classification [30], fetal facial plan recognition in ultrasound [31], brain lesion segmentation in MRI [2], and Alzheimer’s diagnosis with PET/MRI [32].

4. Deep Learning Models in Medical Imaging

Convolutional Neural Networks (CNNs) are essential in DL for MI. CNNs adept at processing X-rays, CT scans, and MRIs through their hierarchical feature representations. Studies [1,2,3] have demonstrated their effectiveness in various medical image analysis tasks. For instance, in segmentation tasks, CNNs excel at delineating organ boundaries or identifying anomalies within medical images, providing valuable insights for accurate diagnosis and treatment planning. Recurrent Neural Networks (RNNs) excel in temporal modeling of dynamic imaging sequences, such as functional MRI or video-based imaging, by capturing temporal patterns [33]. Generative Adversarial Networks (GANs) are valuable for image synthesis, data augmentation, and anomaly detection, generating synthetic images and learning normal patterns [34,35,36].

4.1. Transformer-Based Architectures

Transformer-based architectures, including bidirectional encoder representations from transformers (BERT) [37,38], and generative pre-trained transformer (GPT) [39], are emerging for tasks like disease prediction, image reconstruction, and capturing complex dependencies in medical images.

5. Interpretation and Visualization Techniques

In recent years, numerous explainable artificial intelligence (XAI) techniques have been developed to enhance the interpretability of DL models, particularly in MI. These techniques can be broadly categorized into Perturbation-Based, Gradient-Based, Decomposition, and Attention methods.

The timeline presented in Figure 2 illustrates the development of these XAI techniques over the years. The points on the timeline are color-coordinated by respective categories, including Dimensionality Reduction, Feature Visualization, Class Activation Mapping, Saliency Mapping, Prediction Difference Analysis, Grad-CAM, Integrated Gradient, Guided Backpropagation, Occlusion Sensitivity, Trainable Attention, Guided Grad-CAM, Layerwise Relevance Propagation, Deconvolution, LIME, Backpropagation, Autoencoder, Meaningful Perturbation, SHAP, and Attention. Notably, the Gradient-Based category is the most densely populated, with CAM and Grad-CAM being among the most popular entries. The timeline also reveals a higher density of developments between 2017 and 2020. The comparison chart illustrates the visualization Techniques methods: Gradient-Based, Perturbation-Based, Decomposition-Based (LRP), and Trainable Attention Models in terms of model dependency, access to model parameters, and computational efficiency as shown in Table 1.

5.1. Perturbation-Based Methods

Perturbation-based methods evaluate how input changes affect model outputs to determine feature importance. By altering specific image regions, these methods identify areas that significantly influence predictions, typically visualized with heatmaps. Techniques like Integrated Gradients (IG), Local Interpretable Model-agnostic Explanations (LIME), and Occlusion Sensitivity (OS) are used in various domains, including breast cancer detection, eye disease classification, and brain MRI analysis (see Table 2). The table categorizes studies by domain, task, modality, performance, and technique. These methods also test model sensitivity to input variations, ensuring robust interpretations.

5.1.1. Occlusion

Zeiler and Fergus [40] introduced an occlusion method to assess the impact on model output when parts of an image are obstructed. Kermany et al. [41] utilized this method for interpreting optical coherence tomography images to diagnose retinal pathologies. A major limitation of occlusion is its high computational demand, as it requires inference for each occluded image region, increasing with image resolution and desired heatmaps.

5.1.2. Local Interpretable Model-Agnostic Explanations (LIME)

Ribeiro et al. [42] introduced local interpretable model-agnostic explanations (LIME) to identify superpixels (groups of connected pixels with similar intensities). Seah et al. [43] applied LIME to identify congestive heart failure in chest radiographs. LIME offers an advantage over occlusion by preserving the altered image portions’ context, as they are not completely blocked as shown in Figure 3.

5.1.3. Integrated Gradients

Sundararajan et al. [44] introduced integrated gradients (IG) to measure pixel importance by computing gradients across images interpolated between the original and a baseline image with all non-values. Sayres et al. [45] found that model-predicted grades and heatmaps improved the accuracy of diabetic retinopathy grading by readers.

Table 2. Overview of Various Studies Using Perturbation-Based Methods in Medical Imaging

Domain	Task	Modality	Performance	Technique	Citation
Breast	Classification	MRI	N/A	IG	[46]
Eye	Classification	DR	Accuracy: 95.5%	IG	[45]
Multiple	Classification	DR	N/A	IG	[44]
Chest	Detection	X-ray	Accuracy: 94.9%, AUC: 97.4%	LIME	[47]
Gastrointestinal	Classification	Endoscopy	Accuracy: 97.9%	LIME	[48]
Brain	Segmentation, Detection	MRI	ICC: 93.0%	OS	[49]
Brain	Classification	MRI	Accuracy: 85.0%	OS	[50]
Breast	Detection, Classification	Histology	Accuracy: 55.0%	OS	[51]
Eye, Chest	Classification, Detection	OCT, X-ray	Eye Accuracy: 94.7%, Chest Accuracy: 92.8%	OS	[41]
Chest	Classification	X-ray	AUC: 82.0%	OS, IG, LIME	[43]

Table 2. Overview of Various Studies Using Perturbation-Based Methods in Medical Imaging

Domain	Task	Modality	Performance	Technique	Citation
Breast	Classification	MRI	N/A	IG	[46]
Eye	Classification	DR	Accuracy: 95.5%	IG	[45]
Multiple	Classification	DR	N/A	IG	[44]
Chest	Detection	X-ray	Accuracy: 94.9%, AUC: 97.4%	LIME	[47]
Gastrointestinal	Classification	Endoscopy	Accuracy: 97.9%	LIME	[48]
Brain	Segmentation, Detection	MRI	ICC: 93.0%	OS	[49]
Brain	Classification	MRI	Accuracy: 85.0%	OS	[50]
Breast	Detection, Classification	Histology	Accuracy: 55.0%	OS	[51]
Eye, Chest	Classification, Detection	OCT, X-ray	Eye Accuracy: 94.7%, Chest Accuracy: 92.8%	OS	[41]
Chest	Classification	X-ray	AUC: 82.0%	OS, IG, LIME	[43]

5.2. Gradient-Based Methods

Backpropagation, used for weight adjustment in neural network training, is also employed in model interpretation methods to compute gradients. Unlike training, these methods do not alter weights but use gradients to highlight important image areas.

Figure 4. Examples of gradient-based attribution methods for model interpretability. (A) Class maximization visualization of malignant and benign breast masses on mammograms [9]. (B) Integrated gradients visualizing evidence of diabetic retinopathy on retinal fundus images [45]. (C) Visualization of malignant and benign breast masses [9]. (D) Guided backpropagation applied to ultrasound images for fetal heartbeat localization [4]. (E) Differentiation between benign and malignant breast masses in mammograms [6]. (F) Grad-CAM visualizations identifying discriminative regions in magnetoencephalography images for detecting eye-blink artifacts [52].

Figure 4. Examples of gradient-based attribution methods for model interpretability. (A) Class maximization visualization of malignant and benign breast masses on mammograms [9]. (B) Integrated gradients visualizing evidence of diabetic retinopathy on retinal fundus images [45]. (C) Visualization of malignant and benign breast masses [9]. (D) Guided backpropagation applied to ultrasound images for fetal heartbeat localization [4]. (E) Differentiation between benign and malignant breast masses in mammograms [6]. (F) Grad-CAM visualizations identifying discriminative regions in magnetoencephalography images for detecting eye-blink artifacts [52].

5.2.1. Saliency Maps

Saliency maps, introduced by Simonyan et al. [53], use gradient information to explain how deep convolutional networks classify images. They are used in class maximization and image-specific class saliency maps. Class maximization generates an image maximizing activation for a class, as in:

$$argmax{S}_c\left(I\right)-\lambda {\parallel I\parallel }_2^2$$

Yi et al. [9] applied this to visualize malignant and benign breast masses. Image-specific class saliency maps create heatmaps showing each pixel’s significance in classification, computed as:

$${\mathrm{Sal}}_c\left(x\right)={\displaystyle \frac{\partial F_c\left(x\right)}{\partial x}}$$

Dubost et al. [54] used these maps in a weakly supervised method for segmenting brain MRI structures. Saliency maps have also been utilized in diagnosing heart diseases in chest x-rays [8], classifying breast masses in mammography [55] with accuracy ranging from 85% to 92.9%, and identifying pediatric elbow fractures in x-rays [56] with accuracy of 88.0% and area under curve (AUC) of 95.0%. Moreover, iterative saliency maps [57] enhance less obvious image regions by generating a saliency map, inpainting prominent areas, and iterating the process until the image classification changes or a limit is reached. This approach, applied to retinal fundus images for diabetic retinopathy grading, demonstrated higher sensitivity compared to traditional saliency maps. However, saliency maps have limitations. They do not distinguish if a pixel supports or contradicts a class, and their effectiveness diminishes in binary classification.s.

5.2.2. Guided Backpropagation

Guided backpropagation, introduced by Springenberg et al. [58], builds on the saliency map approach by Simonyan et al. [53] and the deconvnet concept by Zeiler and Fergus [40]. It improves gradient backpropagation through ReLU layers, where negative activations are set to zero during the forward pass. Guided backpropagation discards gradients where either the forward activation or the backward gradient is negative, producing heatmaps that highlight pixels positively contributing to the classification.

In 2017, Gao and Noble [4] applied guided backpropagation to ultrasound images for fetal heartbeat localization. They found that the heatmaps remained consistent despite variations in the heart’s appearance, size, position, and contrast. Conversely, Böhle et al. [59] discovered that guided backpropagation was less effective for visualizing Alzheimer’s disease in brain MRIs compared to other methods. Similarly, Dubost et al. [60] achieved an intraclass correlation coefficient (ICC) of 93.0% in brain MRI detection using guided backpropagation. Wang et al. [61] obtained an average accuracy of 93.7% in brain MRI classification with this technique. Gessert et al. [62] reported an accuracy of 99.0% in cardiovascular classification using OCT images. Wickstrøm et al. [63] achieved a 94.9% accuracy in gastrointestinal segmentation using endoscopy. Lastly, Jamaludin et al. [64] reported an accuracy of 82.5% in musculoskeletal spine classification using MRI images with guided backpropagation.

5.2.3. Class Activation Maps (CAM)

Class Activation Mapping (CAM), introduced by Zhou et al. [65], visualizes regions of an image most influential in a neural network’s classification decision. CAM is computed as a weighted sum of feature maps from the final convolutional layer, using weights from the fully connected layer following global average pooling [66]. For a specific class c and image x:

$${\mathrm{CAM}}_c\left(x\right)=\sum _k{w}_k^c{f}_k\left(x\right)$$

This heatmap highlights regions most relevant for classification. CAM has been applied in various medical imaging applications, such as segmenting lung nodules in thoracic CT scans [67] and differentiating between benign and malignant breast masses in mammograms [6]. However, CAM’s effectiveness depends on the network architecture, requiring a global pooling (GAP) layer followed by a fully connected layer. While Zhou et al. [68] originally used GAP, Oquab et al. [69] demonstrated that global max pooling and log-sum-exponential pooling can also be used, with the latter yielding finer localization.

Table 3. Performance metrics of various Medical Imaging tasks across different modalities using CAM

Domain-Task	Modality	Performance	Citation
Bladder Classification	Histology	Mean Accuracy: 69.9%	[70]
Brain Classification	MRI	Accuracy: 86.7%	[71,72]
Brain Detection	MRI,PET,CT	Accuracy: 90.2% - 95.3%, F1: 91.6% - 94.3%	[73,74]
Breast Classification	X-ray,Ultrasound, MRI	Accuracy: 83.0% -89.0%	[75,76,77,78]
Breast Detection	X-ray, Ultrasound	Mean AUC: 81.0%, AUC: Mt-Net 98.0%, Sn-Net 92.8%, Accuracy: 92.5%	[79,80,81,82]
Chest Classification	X-ray,CT	Accuracy: 97.8% , Average AUC: 75.5%- 96.0%	[83,84,85,86,87,88,89,90]
Chest Segmentation	X-ray	Accuracy: 95.8%	[91]
Eye Classification	Fundus Photography,OCT,CT	F1-score: 95.0% , Precision: 93.0%, AUC: 88.0%- 99.0%	[92,93,94,95,96]
Eye Detection	Fundus Photography	Accuracy: 73.2% -99.1%, AUC: 99.0%	[97,98,99,100,101]
GI Classification	Endoscopy	Mean Accuracy: 93.2%	[102,103,104,105]
Liver Classification, Segmentation	Histology	Mean Accuracy: 87.5%	[106,107]
Musculoskeletal Classification	MRI, X-ray	Accuracy: 86.0%, AUC: 85.3%	[108,109]
Skin Classification, Segmentation	Dermatoscopy	Accuracy: 83.6%, F1-score: 82.7%	[110,111]
Skull Classification	X-ray	AUC: 88.0%- 93.0%	[112]
Thyroid Classification	Ultrasound	Accuracy: 87.3% ± 0.0007, AUC: 90.1% ± 0.0007	[113]
Lymph Node Classification, Detection	Histology	Accuracy: 91.9%, AUC: 97.0%	[114]
Various Classification	CT, MRI, Ultrasound, X-ray, Fundoscopy	F1-score: 98.0%, Model Accuracy: 98.0%	[115,116]

Table 3. Performance metrics of various Medical Imaging tasks across different modalities using CAM

Domain-Task	Modality	Performance	Citation
Bladder Classification	Histology	Mean Accuracy: 69.9%	[70]
Brain Classification	MRI	Accuracy: 86.7%	[71,72]
Brain Detection	MRI,PET,CT	Accuracy: 90.2% - 95.3%, F1: 91.6% - 94.3%	[73,74]
Breast Classification	X-ray,Ultrasound, MRI	Accuracy: 83.0% -89.0%	[75,76,77,78]
Breast Detection	X-ray, Ultrasound	Mean AUC: 81.0%, AUC: Mt-Net 98.0%, Sn-Net 92.8%, Accuracy: 92.5%	[79,80,81,82]
Chest Classification	X-ray,CT	Accuracy: 97.8% , Average AUC: 75.5%- 96.0%	[83,84,85,86,87,88,89,90]
Chest Segmentation	X-ray	Accuracy: 95.8%	[91]
Eye Classification	Fundus Photography,OCT,CT	F1-score: 95.0% , Precision: 93.0%, AUC: 88.0%- 99.0%	[92,93,94,95,96]
Eye Detection	Fundus Photography	Accuracy: 73.2% -99.1%, AUC: 99.0%	[97,98,99,100,101]
GI Classification	Endoscopy	Mean Accuracy: 93.2%	[102,103,104,105]
Liver Classification, Segmentation	Histology	Mean Accuracy: 87.5%	[106,107]
Musculoskeletal Classification	MRI, X-ray	Accuracy: 86.0%, AUC: 85.3%	[108,109]
Skin Classification, Segmentation	Dermatoscopy	Accuracy: 83.6%, F1-score: 82.7%	[110,111]
Skull Classification	X-ray	AUC: 88.0%- 93.0%	[112]
Thyroid Classification	Ultrasound	Accuracy: 87.3% ± 0.0007, AUC: 90.1% ± 0.0007	[113]
Lymph Node Classification, Detection	Histology	Accuracy: 91.9%, AUC: 97.0%	[114]
Various Classification	CT, MRI, Ultrasound, X-ray, Fundoscopy	F1-score: 98.0%, Model Accuracy: 98.0%	[115,116]

5.2.4. Grad-CAM

Grad-CAM, an extension of CAM by Selvaraju et al. [117], broadens its application to any network architecture and output, including image segmentation and captioning. It bypasses the global pooling layer and weights feature maps directly with gradients calculated via backpropagation from a target class. The gradients of the output for class c concerning feature maps $A^k$ are averaged globally, multiplied by $A^k$, and passed through a ReLU activation to discard negative values:

$${\mathrm{Grad}-\mathrm{CAM}}_c\left(x\right)=\mathrm{ReLU}\left(\sum _k\left({\displaystyle \frac{1}{Z}}\sum _i\sum _j{\displaystyle \frac{\partial y^c}{\partial A_{ij}^k}}\right)A^k\right)$$

Garg et al. employed grad-CAM visualizations to identify discriminative regions of magnetoencephalography images in the task of detecting eye-blink artifacts [52]. The authors found that the regions of the eye highlighted by grad-CAM are the same regions that human experts rely on. Table 4 summarizes the effectiveness of Grad-CAM across various medical imaging tasks, highlighting domains, tasks, modalities, and performance metrics.

5.3. Decomposition-Based Methods

Decomposition-based techniques for model interpretation focus on breaking down a model’s prediction into a heatmap showing each pixel’s contribution to the final decision. These techniques, such as LRP, have been widely applied across different domains.

Figure 5. Example uses of decomposition-based attribution methods for model interpretability to show the Layerwise relevance propagation for model interpretation in diagnosing multiple sclerosis on brain MRI (Eitel et al., 2019)[160]. The Attention U-Net for organ segmentation in abdominal CT scans [161] and the areas upon which the model correctly focuses its predictions on the test images in explainable Vision Transformers model [162].

Figure 5. Example uses of decomposition-based attribution methods for model interpretability to show the Layerwise relevance propagation for model interpretation in diagnosing multiple sclerosis on brain MRI (Eitel et al., 2019)[160]. The Attention U-Net for organ segmentation in abdominal CT scans [161] and the areas upon which the model correctly focuses its predictions on the test images in explainable Vision Transformers model [162].

5.3.1. Layer-Wise Relevance Propagation (LRP)

Layer-Wise Relevance Propagation (LRP), introduced by Bach et al. in 2015 [163], offers an alternative to gradient-based techniques like saliency mapping, guided backpropagation, and Grad-CAM. Instead of relying on gradients, LRP distributes the output of the final layer back through the network to calculate relevance scores for each neuron. This process is repeated recursively from the final layer to the input layer, generating a relevancy heatmap that can be overlaid on the input image. Further properties of LRP and details of its theoretical basis are given in (Montavon et al., 2017 [164]), and a comparison of LRP to other interpretation methods can be found in ([165,166,167]).

The relevance score $R_{i\leftarrow k}^{(l,l+1)}$ for a neuron i in layer l from a neuron k in layer $l+1$ is defined as:

$$R_{i\leftarrow k}^{(l,l+1)}=R_k^{(l+1)}{\displaystyle \frac{a_i{w}_{ik}}{{\sum }_h{R}_{h\leftarrow k}^{(l,l+1)}}}$$

The overall relevance score for neuron i in layer l is:

$$R_i^l=\sum _k{R}_{i\leftarrow k}^{(l,l+1)}$$

LRP has been applied in MI, such as diagnosing multiple sclerosis (MS) and Alzheimer’s disease (AD) using MRI scans. For MS, LRP heatmaps highlighted hyperintense lesions and affected brain areas [160] as shown in Figure 5, while for AD, they emphasized the hippocampal volume, a critical region for diagnosis [168]. LRP has been found to provide clearer distinctions compared to gradient-based methods and has been used in frameworks like DeepLight for linking brain regions with cognitive states [169].

5.4. Trainable Attention Models

Trainable Attention (TA) Mechanisms provide a dynamic approach to model interpretation by integrating attention modules into neural networks. Introduced for CNNs by Jetley et al. [170], these soft attention modules generate attention maps that highlight important image parts. They compute compatibility scores between local and global features $\langle l_s,g\rangle$ using dot products and learned vectors a:

$$a_s={\displaystyle \frac{exp\left(c_s\right)}{{\sum }_{i=1}^{n}exp\left(c_{s,i}\right)}}$$

The output $g_a$ adjusts local features based on the attention map weights:

$$g_a=\sum a_s{l}_s$$

This method enhances signals from compatible features while reducing those from less compatible ones. Applications of attention mechanisms in medical imaging include the Attention U-Net for organ segmentation in abdominal CT scans [161], fetal ultrasound classification, and breast mass segmentation in mammograms [171]. Additionally, they have improved melanoma lesion classification [172] and osteoarthritis grading in knee X-rays [173]. Attention mechanisms are valued for their interpretability and performance enhancement, though optimal configurations are application-specific.

Table 5 provides an overview of various studies using TA models, highlighting the domains, tasks, modalities including MRI and histology, and performance metrics such as accuracy and F1-score, demonstrating the broad applicability and effectiveness of TA models in MI.

5.5. Vision Transformers

Vision Transformers (ViTs) have emerged as a prominent alternative to convolutional neural networks (CNNs) in medical imaging. Unlike CNNs, which use local receptive fields to capture spatial hierarchies, ViTs employ self-attention to model long-range dependencies and global context [38,184,185]. By partitioning images into fixed-size patches treated as a sequence, ViTs utilize transformer encoder layers, effectively capturing complex anatomical structures and pathological patterns.

ViTs have shown superior performance in segmentation, classification, and detection tasks, achieving high accuracy in segmenting tumors and organs in MRI and CT scans, as reflected in Dice scores. Their interpretability is enhanced through attention maps, gradient-based methods, and occlusion sensitivity, which aid in visualizing model predictions. These advancements highlight ViTs’ potential to improve diagnostic accuracy and provide deeper insights into medical image analysis, as discussed in Table 6. The areas upon which the model correctly focuses its predictions on the test image are presented in Figure 5. The regions of focus identified by the ViT model exhibit significant overlap with the areas of White blood cells [162].

6. Comparison of Different Interpretation Methods

6.1. Categorization by Visualization Technique

Visualization techniques in DL can be categorized based on their application and effectiveness. Table 7 summarizes various visualization techniques used in DL for interpretability. It categorizes methods based on their tasks, body parts, modalities, accuracy, and evaluation metrics. This table highlights that techniques like CAM and Grad-CAM are effective for image classification and localization across different modalities such as X-ray and MRI, achieving high accuracy. LRP is noted for its accuracy in segmentation tasks, while IG is utilized for classification with notable AUC-Receiver Operating Characteristic (ROC) scores. Attention-based methods improve performance and interpretability by focusing on relevant regions, whereas perturbation-based methods assess model robustness. LIME provides model-agnostic explanations, and trainable attention models dynamically enhance feature focus.

6.2. Categorization by Body Parts, Modality, and Accuracy

This table provides a concise overview of the imaging techniques, their applications, accuracy, and specific considerations for different anatomical contexts. Table 7 provides a concise overview of imaging techniques categorized by anatomical context (Body Parts). It lists various modalities such as MRI, X-ray, and ultrasound, and highlights specific techniques used for different body parts. For instance, CAM and Grad-CAM are prominent in brain imaging with high accuracy, while LRP and attention-based methods excel in breast imaging. The table also emphasizes the adaptation of methods to address challenges such as speckle noise in ultrasound imaging. The imaging techniques used such as CT, dermatoscopy, diabetic retinopathy (DR), endoscopy, fundus photography, histology, histopath, mammo, magnetoencephalography (MEG), MRI, OCT, PET, photography, ultrasound, and X-ray. Studies ranged from the years 2014 to 2020 with a majority coming from 2019 and 2020 as shown in Figure 6.

6.3. Categorization by Task

This section organizes the techniques and their applications across different tasks, highlighting performance metrics and examples for clarity. Table 8 organizes interpretability techniques according to their tasks, including classification, segmentation, and detection. It details the applications, performance metrics, and specific examples for each task. Techniques like CAM, Grad-CAM, and TA models are effective for classification tasks, providing high accuracy and AUC-ROC scores. LRP and Integrated Gradient are highlighted for segmentation tasks, with metrics like dice similarity coefficient (DSC) and intersection over union (IoU). Detection tasks benefit from methods such as saliency maps and CAM, with metrics including mean average precision (mAP) and sensitivity.

7. Current Challenges and Future Directions

7.1. Current Challenges

Despite significant advancements, several challenges remain in the interpretability and visualization of DL models in MI:

Scalability and Efficiency: Many interpretability methods, such as occlusion and perturbation-based techniques, are computationally intensive. This limits their scalability, especially with high-resolution medical images that require real-time analysis.

Clinical Integration: Translating interpretability techniques into clinical practice requires seamless integration with existing workflows and systems. This includes ensuring that the visualizations are intuitive for non-technical healthcare practitioners and that they provide actionable insights.

Robustness and Generalization: Interpretability methods must be robust across diverse patient populations and medical imaging modalities. Models trained on specific datasets might not generalize well to other contexts, leading to potential biases and inaccuracies in interpretations.

Standardization and Validation: There is a lack of standardized metrics and benchmarks for evaluating the effectiveness of interpretability methods. Rigorous validation in clinical settings is essential to establish the reliability and trustworthiness of these techniques.

Ethical and Legal Considerations: The opacity of deep learning models raises ethical and legal concerns, especially in healthcare where decisions can have critical consequences. Ensuring transparency, accountability, and fairness in AI-driven diagnostics is paramount.

7.2. Future Directions

To address these challenges, future research should focus on the following directions:

Development of Lightweight Methods: Creating computationally efficient interpretability techniques that can handle high-resolution images and deliver results in real-time is crucial. This includes optimizing existing methods and exploring new algorithmic approaches.

Enhanced Clinical Collaboration: Collaborative efforts between AI researchers, clinicians, and medical practitioners are needed to design interpretability methods that are clinically relevant and user-friendly. This could involve interactive visualization tools that allow clinicians to explore model outputs intuitively.

Robustness to Variability: Developing interpretability techniques that are robust to variations in imaging modalities, patient demographics, and clinical conditions is essential. This requires extensive training on diverse datasets and continuous validation across different settings.

Establishment of Standards: Creating standardized benchmarks and validation protocols for interpretability methods will help in objectively assessing their effectiveness and reliability. This includes developing common datasets and metrics for comparative evaluations.

Ethical Frameworks: Integrating ethical considerations into the design and deployment of interpretability methods is critical. This involves ensuring that models are transparent, explainable, and free from biases, as well as addressing privacy and data security concerns.

Hybrid Approaches: Combining different interpretability techniques, such as perturbation-based and gradient-based methods, can provide more comprehensive insights into model behavior. Hybrid approaches can leverage the strengths of various methods to enhance overall interpretability.

8. Conclusion

In conclusion, the integration of interpretability and visualization techniques into DL models for MI holds immense potential for advancing healthcare diagnostics and treatment planning. While significant progress has been made, challenges related to scalability, clinical integration, robustness, standardization, and ethical considerations persist. Addressing these challenges requires ongoing collaboration between AI researchers, clinicians, and healthcare practitioners. Future research should focus on developing efficient and clinically relevant interpretability methods, establishing standardized evaluation protocols, and ensuring ethical and transparent AI applications in healthcare. By overcoming these hurdles, we can enhance the trustworthiness, reliability, and clinical impact of DL models in MI, ultimately leading to better patient outcomes and more informed clinical decision-making.