A Survey of Explainable Artificial Intelligence in Healthcare: Concepts, Applications, and Challenges

1. Introduction

Artificial Intelligence (AI) has significantly reshaped numerous sectors, with healthcare witnessing significant transformations [1,2,3]. The integration of AI technologies into healthcare systems has led to advancements that were previously unattainable. These advancements include enhanced diagnostic accuracy, personalized treatment plans, and improved operational efficiencies [4,5,6]. Machine learning (ML) and deep learning (DL), as subfields of AI, enable the processing and analysis of vast datasets, which is crucial for predictive analytics and decision-making in clinical environments [7].

However, the adoption of AI in healthcare comes with challenges, particularly regarding the transparency and interpretability of AI models. Explainable AI (XAI) addresses these concerns by providing mechanisms that make AI decision-making processes comprehensible to humans. This is vital in healthcare, where understanding the rationale behind AI predictions is crucial to clinicians and patients [8,9,10]. Furthermore, as AI systems become more sophisticated, their decision-making processes become less transparent, raising concerns about trust, accountability, and ethical use. Addressing these issues through XAI can enhance the acceptance and reliability of AI technologies in medical practice.

In the past, researchers have conducted various reviews on AI in healthcare. Some reviews have centred on the general application of AI models in healthcare [9,11,12,13], while others have addressed the potential of AI for predictive analytics in clinical settings [14,15,16] and drug discovery [2,17,18]. Additionally, other reviews include discussions on the ethical implications and regulatory challenges associated with AI in healthcare [5,12,19]. However, most reviews either focus on AI broadly or do not provide an in-depth analysis of XAI tailored to healthcare. Additionally, over the years, there have been several advances in the field of healthcare AI and XAI, making it necessary and timely to conduct an up-to-date review that captures these developments and provides actionable insights.

Therefore, this study aims to provide a comprehensive review of XAI in healthcare. Specifically, the study examines the foundational concepts of XAI, explores its diverse applications in the healthcare sector, identifies the key challenges it faces, and highlights possible solutions for its effective integration into medical practice. This study is significant given the growing importance of transparency and interpretability in the ethical deployment of AI technologies in healthcare settings.

The remainder of this paper is structured as follows: Section 2 reviews related works. Section 3 discusses the fundamental concepts of XAI, Section 4 outlines various XAI methods and techniques. Section 5 explores the applications of XAI in healthcare, Section 6 examines responsible AI in healthcare, and Section 7 identifies the challenges and opportunities in implementing XAI within clinical settings. Section 8 provides a comprehensive discussion on future research directions, identifying areas of potential growth and innovation. Section 9 concludes the paper with a summary of key findings.

2. Related Works

The application of AI in healthcare has been extensively explored in recent years, with different studies focusing on various aspects of its implementation and impact. For instance, Topol [12] provides an in-depth analysis of AI’s capabilities in clinical settings, including improvements in diagnostic processes and patient care. Similarly, Malik et al. [13] presented an overview of AI applications in medicine, detailing various AI models and their effectiveness in predicting and managing diseases. These studies showed the significant advancements AI brings to healthcare.

Meanwhile, XAI is a crucial subfield of AI which addresses the need for transparency and interpretability in AI models. Arrieta et al. [20] provided a comprehensive review of XAI techniques, categorizing them into model-specific and model-agnostic approaches and highlighting the importance of XAI in enabling trust and accountability in AI models. Similarly, Samek et al. [21] focused on the theoretical foundations of XAI, presenting various methods to make AI models interpretable and demonstrating the need for these methods to gain acceptance in high-stakes domains such as healthcare.

Adadi and Berrada [22] examined the potential of XAI in various sectors, including healthcare, identifying the key challenges in making AI models interpretable and proposing potential solutions to address these challenges. The study provides valuable insights into the importance of XAI. Furthermore, a critical aspect of AI that has gained attention recently is the concept of responsible AI, which encompasses ethical considerations such as fairness, accountability, and transparency. Dignum [23] outlines the principles of responsible AI, stressing the importance of integrating ethical considerations into AI development. Similarly, Barocas et al. [24] discussed fairness in AI, proposing methods to detect and mitigate biases in ML models.

Furthermore, Holzinger et al. [25] discuss the need for human-in-the-loop (HITL) approaches in AI, particularly in healthcare, arguing that for AI systems to be trusted and widely adopted, they must not only be accurate but also provide explanations that are understandable to users. Their work highlights the theoretical foundations of XAI and the practical need for HITL approaches to ensure that AI systems can be effectively integrated into clinical practice.

Meanwhile, most existing reviews and surveys on AI and XAI provide a broad overview of the field, focusing on specific applications or theoretical developments. For example, Jiang et al. [11] reviewed the application of AI in healthcare broadly, without discussing the specific challenges of explainability. Similarly, Miotto et al. [26] focused on deep learning in healthcare, offering insights into its potential but not addressing the interpretability challenges associated with these models.

Therefore, this review aims to provide a detailed and comprehensive analysis of XAI in healthcare, covering key areas such as foundational concepts, diverse applications, challenges, and opportunities for future research. By focusing on these aspects, this review aims to bridge the gap in the literature and offer actionable insights for researchers, practitioners, and policymakers in the field of healthcare AI.

3. Overview of Explainable AI

XAI is a domain within AI focused on creating models whose decisions can be understood and interpreted by humans. The primary goal of XAI is to make the internal mechanics of AI systems transparent and their outputs explainable [27,28]. This is essential in fields like healthcare, where understanding the reasoning behind AI decisions can significantly impact patient outcomes and trust in the technology. The main goals of XAI include:

3.1. Transparency

Transparency involves making the AI decision-making process clear and understandable [29,30]. This means that the inner workings of an AI model, such as the data it uses, the features it considers, and the logic it follows to reach a decision, should be visible to and interpretable by humans. Transparency is crucial for identifying potential biases, ensuring fairness, and gaining trust. Mathematically, transparency can be represented by ensuring the model f is such that for any input x, the decision process $f\left(x\right)$ can be decomposed into understandable components:

$$f\left(x\right)=\sum _{i=1}^n{w}_i·x_i+b,$$

(1)

where $w_i$ are the weights and $x_i$ are the input features, providing a linear combination that is easily interpretable [31].

3.2. Interpretability

Interpretability refers to the extent to which a human can understand the cause of a decision made by an AI model. This involves providing explanations that are comprehensible to humans, often through simplified models or visualizations that capture the behavior of the complex model [32]. Interpretability can be achieved through various methods, such as surrogate models, partial dependence plots, and feature importance scores.

3.3. Trustworthiness

Trustworthiness involves building user trust through understandable and verifiable AI decisions [20,32]. Trust is essential for the widespread adoption of AI technologies, particularly in critical fields like healthcare [33]. Trustworthiness can be achieved by ensuring that AI systems are transparent, interpretable, and robust. This includes providing clear documentation of the model’s decision-making process, using robust validation techniques to ensure the model’s reliability, and continuously monitoring the model’s performance to detect and address any issues promptly.

For instance, incorporating human-in-the-loop approaches, where human experts interact with AI systems to validate and refine their outputs, can enhance trustworthiness. Involving clinicians in the decision-making process ensures AI systems can benefit from expert knowledge and feedback, leading to more accurate and reliable outcomes [34]. Additionally, frameworks that track and log AI decisions, providing an audit trail, can help users review and understand the rationale behind AI-driven decisions, further enhancing trust.

3.4. Accountability

Accountability involves enabling users to hold AI systems accountable for their decisions. This means that AI systems should provide enough information to allow users to understand, challenge, and, if necessary, rectify the decisions made by the AI [35]. Accountability is essential for ensuring ethical AI deployment. This can be supported by frameworks that track and log AI decisions, providing an audit trail that users can review. For example, in regression models, accountability can be enhanced by providing confidence intervals for predictions:

$$\widehat{y}={\beta }_0+{\beta }_{1}x+ϵ,$$

(2)

where $\widehat{y}$ is the predicted value, ${\beta }_0$ and ${\beta }_1$ are coefficients, and $ϵ$ represents the error term. The confidence interval gives users an idea of the uncertainty in the prediction, helping them to hold the model accountable for its predictions [36]. Furthermore, the key terms and concepts associated with explainable AI are tabulated in Table 1.

4. XAI Techniques and Methods

Several techniques and methods have been developed to achieve the goals of XAI. These methods can be broadly classified into two categories: model-specific and model-agnostic techniques.

4.1. Model-Specific Techniques

Model-specific techniques are tailored to particular types of models. These model-specific techniques enhance the interpretability of AI models by providing clear, understandable structures and visualizations that help users comprehend how decisions are made. For instance:

4.1.1. Decision Trees and Rule-Based Systems

Decision trees and rule-based systems are inherently interpretable because they follow a clear structure of decisions and rules that can be easily understood [47,48]. Each decision in a decision tree represents a choice based on a specific feature, making it straightforward to trace the path from the root to a leaf node (final decision) [49]. For example, a decision tree model f can be represented as a set of nested if-then rules:

$$f\left(x\right)=\sum _{i=1}^n\mathrm{if}(x_i<{\theta }_i)\mathrm{then}{a}_i\mathrm{else}{b}_i,$$

(3)

where $x_i$ are the input features, ${\theta }_i$ are the threshold values, and $a_i$ and $b_i$ are the decisions or outputs at each node. This structure allows users to understand how the model arrives at a specific decision by following the path dictated by the feature values [49]. The structure of a typical decision tree is shown in Figure 1.

4.1.2. Attention Mechanisms in Neural Networks

Attention mechanisms in neural networks provide insights into which parts of the input data the model is focusing on when making a decision, thus offering some level of interpretability [51,52]. Attention mechanisms assign different weights to different parts of the input, highlighting their relative importance in the final decision [53]. The attention mechanism can be represented mathematically as:

$${\alpha }_i={\displaystyle \frac{exp\left(e_i\right)}{{\sum }_{j=1}^{n}exp\left(e_j\right)}},$$

(4)

where ${\alpha }_i$ is the attention weight for the i-th input, and $e_i$ is the alignment score between the input $x_i$ and the model’s internal state. The final output of the attention mechanism is a weighted sum of the input features. This weighted sum c allows users to visualize and interpret which input features are most influential in the model’s decision-making process [54]. It is represented mathematically as:

$$c=\sum _{i=1}^n{\alpha }_i{x}_i.$$

(5)

4.1.3. Convolutional Neural Networks

In Convolutional Neural Networks (CNNs), interpretability can be enhanced through techniques like Gradient-weighted Class Activation Mapping (Grad-CAM). Grad-CAM provides visual explanations for decisions made by CNNs by highlighting the regions of an input image that are most relevant to the prediction [55]. The Grad-CAM heatmap $L_{\mathrm{Grad}-\mathrm{CAM}}^c$ for a class c is calculated as:

$$L_{\mathrm{Grad}-\mathrm{CAM}}^c=\mathrm{ReLU}\left(\sum _k{\alpha }_k^c{A}^k\right),$$

(6)

where ${\alpha }_k^c$ is the importance weight for the k-th feature map $A^k$, and ReLU is the rectified linear unit activation function. This heatmap overlays the original image, showing which regions contributed most to the model’s decision [55].

4.1.4. Bayesian Networks

Bayesian networks are probabilistic graphical models that represent a set of variables and their conditional dependencies via a directed acyclic graph (DAG) [56]. They provide interpretability by visualizing the probabilistic relationships between variables. The joint probability distribution P over a set of variables X can be decomposed as:

$$P\left(X\right)=\prod _{i=1}^{n}P\left(X_i\right|\mathrm{Parents}\left(X_i\right)),$$

(7)

where $\mathrm{Parents}\left(X_i\right)$ denotes the set of parent nodes for $X_i$ in the DAG. This decomposition allows users to understand how each variable influences others and contributes to the overall model’s predictions [57].

4.2. Model-Agnostic Techniques

Model-agnostic techniques can be applied to any AI model, irrespective of its underlying architecture. These techniques provide flexibility and can be used to interpret complex models without requiring changes to the model itself. Prominent model-agnostic techniques include:

4.2.1. SHapley Additive exPlanations

SHapley Additive exPlanations (SHAP) values are a method based on cooperative game theory that provides a unified measure of feature importance [58]. This method assigns an importance value to each feature by computing the Shapley value, which represents the average contribution of a feature across all possible combinations of features. The Shapley value ${\varphi }_i$ for a feature i is calculated as:

$${\varphi }_i=\sum _{S\subseteq N\setminus \left\{i\right\}}{\displaystyle \frac{\left|S\right|!\left(\right|N|-|S|-1)!}{\left|N\right|!}}\left[f(S\cup \{i\left\}\right)-f\left(S\right)\right],$$

(8)

where S is any subset of features not containing i, and $f\left(S\right)$ is the prediction from the model considering only the features in S [59]. This equation ensures that the sum of the Shapley values equals the difference between the actual prediction and the average prediction, thus providing a fair distribution of feature importance. SHAP values have several desirable properties that make them a powerful tool for interpreting ML models. Firstly, they provide consistency, meaning that if a model’s prediction for a particular instance increases due to a change in a feature value, the Shapley value for that feature will also increase. Secondly, SHAP values offer both global and local interpretability. Global interpretability refers to understanding the overall importance of each feature across the entire dataset, while local interpretability focuses on understanding how features influence individual predictions [60]. For example, in a healthcare setting, SHAP values can be used to interpret a model predicting the risk of heart disease. By examining the Shapley values, clinicians can identify which features (e.g., age, cholesterol levels, blood pressure) are most influential in the model’s predictions for individual patients and across the patient population.

Meanwhile, SHAP values can be visualized using various plots to aid in interpretability. The most common visualizations include summary plots, dependence plots, and force plots. Summary plots provide a high-level overview of feature importance across the dataset, highlighting the distribution of Shapley values for each feature. Dependence plots show the relationship between a feature’s value and its Shapley value, indicating how changes in the feature value impact the model’s prediction. Force plots offer a detailed view of individual predictions, illustrating how each feature contributes to the final prediction [61]. An example architecture for integrating SHAP with an ML model is shown in Figure 2.

4.2.2. Local Interpretable Model-agnostic Explanations

Local Interpretable Model-agnostic Explanations (LIME) is a technique designed to explain individual predictions of any black-box model by approximating it locally with an interpretable model [62]. LIME operates by perturbing the data around the instance of interest, generating a dataset of perturbed samples, and then training an interpretable model (often a linear regression or decision tree) on this perturbed dataset. The weights in this interpretable model are used to explain the prediction of the original model. This approach allows for an understanding of the complex model’s behavior in the vicinity of the specific instance being explained. Given a black-box model f and an instance x, LIME constructs a new dataset Z consisting of perturbed samples of x and their corresponding predictions from f. A weighted linear model g is then trained on Z, where the weights are based on the proximity of the perturbed samples to x. The explanation is provided by the coefficients of the linear model g:

$$g\left(z\right)=arg\underset{g\in G}{min}\sum _{z_i\in Z}{\pi }_x\left(z_i\right){(f\left(z_i\right)-g\left(z_i\right))}^2+\Omega \left(g\right),$$

(9)

where ${\pi }_x\left(z_i\right)$ is a proximity measure between x and $z_i$, and $\Omega \left(g\right)$ is a complexity measure for g [62,63]. Algorithm 1 summarizes the LIME process:

Algorithm 1 LIME Process

Require:  Black-box model f, instance x, number of perturbations N 1: Generate a new dataset Z by perturbing xN times 2: for each perturbed instance $z_i\in Z$ do 3:   Obtain prediction $f\left(z_i\right)$ from the black-box model 4:   Compute the proximity measure ${\pi }_x\left(z_i\right)$ between x and $z_i$ 5: end for 6: Train a weighted linear model g on Z, using ${\pi }_x\left(z_i\right)$ as weights 7: Use the coefficients of g to explain the prediction for x 8: return Explanation of x based on g

By focusing on the local behavior of the model around a specific instance, LIME provides an understandable approximation that can highlight which features are driving a particular prediction. LIME’s utility in healthcare is vast. For instance, in the context of predicting patient outcomes, LIME can help clinicians understand which features (e.g., patient age, lab results, medical history) are influencing the model’s prediction for a specific patient. This local explanation is crucial in making the model’s decision-making process transparent and comprehensible to healthcare professionals who may not have a deep understanding of ML models.

4.2.3. Partial Dependence Plots

Partial Dependence Plots (PDPs) show the relationship between a subset of features and the predicted outcome of a machine learning model [64,65]. The partial dependence function for a feature $x_j$ is defined as:

$${\widehat{f}}_{x_j}\left(x_j\right)={\displaystyle \frac{1}{n}}\sum _{i=1}^n\widehat{f}(x_j,x_{iC}),$$

(10)

where $x_{iC}$ represents all features except $x_j$, and $\widehat{f}$ is the prediction function. PDPs provide insight into the effect of a feature on the prediction while averaging out the effects of other features. This method helps to visualize the marginal effect of a feature on the predicted outcome, assuming that the effect of other features remains constant. For example, in a healthcare setting, PDPs can be used to understand how a single biomarker influences the risk prediction of a disease, independent of other biomarkers. This visualization aids clinicians in interpreting the importance and influence of specific features, thereby enhancing the transparency and trust in the model’s predictions [66].

4.2.4. Individual Conditional Expectation Plots

Individual Conditional Expectation (ICE) plots are similar to PDPs but show the dependency of the prediction on a feature for each instance separately rather than averaging [67]. For an instance i, the ICE curve for a feature $x_j$ is given by:

$${\widehat{f}}_{x_j}^{\left(i\right)}\left(x_j\right)=\widehat{f}(x_j,x_{iC}).$$

(11)

ICE plots provide a more granular view of feature effects, revealing heterogeneity in the model’s behavior across different instances [68]. This granular view is particularly useful for detecting interactions and non-linear relationships between features and the outcome. In the context of healthcare, ICE plots can show how different patients respond to varying levels of a particular treatment, thereby highlighting the variability in treatment effectiveness across the patient population.

5. Applications of Explainable AI in Healthcare

XAI significantly enhances various aspects of healthcare by improving diagnostic accuracy, treatment personalization, clinical decision support, medical imaging, and remote diagnostics. By making AI models transparent and interpretable, XAI builds trust and reliability in AI-driven healthcare systems.

5.1. Diagnostic Tools and Clinical Decision Support Systems

Integrating XAI into diagnostic tools and Clinical Decision Support Systems (CDSS) is crucial for enhancing the interpretability and trustworthiness of AI models. In oncology, cardiovascular diagnostics, and neurological disorders, XAI techniques like SHAP values, LIME, and attention mechanisms help explain AI model predictions, thus improving the accuracy and transparency of diagnostics [69,70,71]. Recent studies, such as that by Lundberg et al. [72], have shown how SHAP values can be used to interpret predictions of ML models for diagnosing pneumonia from chest X-rays. Similarly, studies have demonstrated the effectiveness of XAI techniques in predicting diabetic retinopathy from retinal images, using saliency maps to highlight the critical areas influencing the model’s predictions [73].

XAI is also instrumental in CDSS, where it elucidates the factors influencing diagnostic and treatment recommendations. For example, in sepsis prediction for ICU patients, Li et al. [74] used LIME interpret model predictions, enhancing the transparency and trustworthiness of the CDSS. Similarly, Bedoya et al. [15] SHAP values have been employed in CDSS for predicting hospital readmissions, helping healthcare providers tailor care plans to individual patients. Another study by Tonekaboni et al. [75] explored the use of attention mechanisms within a CDSS for diagnosing acute kidney injury (AKI), highlighting the most relevant data points and thereby improving diagnostic accuracy.

Additionally, the integration of XAI in models used for detecting arrhythmias from ECG data has shown significant promise. A study by Bento et al. [76] demonstrated how XAI techniques could enhance the transparency of deep learning models, providing cardiologists with clear visual and statistical explanations of abnormal heart rhythms, thus supporting the integration of these advanced diagnostic tools into clinical practice.

5.2. Personalized Medicine

XAI plays a crucial role in personalized medicine by making the outputs of ML models more interpretable. These models analyze genetic information, medical history, and lifestyle factors to recommend individualized treatment plans. XAI techniques like SHAP values and LIME provide clear explanations for these recommendations, ensuring that treatments are tailored to each patient’s needs and improving both outcomes and adherence. For instance, in the management of type 2 diabetes, SHAP values have been used to interpret ML models that predict the effectiveness of different medications, allowing clinicians to understand how factors such as age, weight, and blood sugar levels influence treatment recommendations. Another significant application is in oncology, where XAI has been employed to tailor chemotherapy treatments. Techniques like LIME have been used to explain the genetic markers and clinical features influencing treatment plans, improving the interpretability and trustworthiness of AI-driven recommendations [77].

Additionally, XAI has shown potential in managing cardiovascular diseases. A study by Ahmad et al. [78] demonstrated the use of XAI to interpret ML models that predict the risk of adverse cardiac events, such as heart attacks. By providing clear explanations for these risk predictions, XAI helps cardiologists develop personalized prevention strategies, thereby enhancing patient outcomes. Furthermore, XAI has been applied in the treatment of rare genetic disorders. A study by Ravindran et al. [79] used XAI to interpret deep learning models predicting the efficacy of gene therapies, enabling geneticists to understand and validate the AI’s recommendations.

5.3. Medical Imaging

In medical imaging, XAI enhances the interpretability of ML models used for image analysis, crucial for detecting and diagnosing conditions from X-rays, MRIs, and CT scans. Techniques like attention maps and Grad-CAM highlight the image regions most influential in the model’s decision, providing radiologists with visual explanations that increase confidence in AI findings [80].

Recent studies have highlighted the effectiveness of XAI in various medical imaging applications. For instance, McKinney et al. [81] applied XAI techniques to deep learning models used for breast cancer screening from mammograms. The use of attention maps allowed radiologists to see which parts of the mammogram the model focused on, improving diagnostic accuracy and reducing false positives and negatives. In another study, DeGrave et al. [82] investigated the use of XAI in the diagnosis of COVID-19 from chest X-rays, employing saliency maps to visualize the lung regions crucial for detecting the infection.

Furthermore, a study by Baumgartner et al. [83] explored the application of Grad-CAM in MRI-based brain tumor classification, demonstrating how specific brain regions contributing to the AI’s predictions could be visualized, thereby enhancing diagnostic confidence and potentially leading to more accurate treatment plans. In ophthalmology, XAI has been used to interpret ML models for diagnosing diabetic retinopathy from retinal images, with studies like Gargeya and Leng [84] showing how heatmaps can help ophthalmologists understand and trust AI-driven diagnoses.

5.4. Remote Diagnostics and Telemedicine

XAI also improves the accessibility and accuracy of remote diagnostics and telemedicine, particularly in underserved areas. By ensuring that AI-driven diagnostic suggestions are accompanied by clear explanations, XAI helps remote healthcare providers understand and trust AI recommendations, enhancing care quality where access to specialists is limited [85]. For example, in dermatology, Hauser et al. [86] XAI has been used to interpret ML models diagnosing skin conditions via mobile devices, with heatmaps highlighting key regions of interest. This approach supports healthcare workers in remote locations, ensuring accurate and timely diagnoses [87]. Similarly, XAI enhances the trustworthiness of telemedicine platforms for diagnosing eye diseases from retinal images, as demonstrated by Arcadu et al. [88], improving the quality of care in underserved regions.

Table 2. Summary of XAI Applications in Healthcare.

Application	Specific Use Case	Description	References
Diagnostic Tools and CDSS	Oncology	Identifying cancerous lesions using medical imaging data	[69]
	Cardiovascular Diseases	Detecting patterns in ECGs indicative of heart diseases	[70]
	Pneumonia Detection	Using SHAP values for pneumonia diagnosis from chest X-rays	[72]
	Diabetic Retinopathy	Interpreting retinal images to predict diabetic retinopathy	[73]
	Neurological Disorders	Diagnosing Alzheimer’s using attention mechanisms on MRI scans	[71]
	Arrhythmia Detection	Enhancing transparency in deep learning models for arrhythmia diagnosis	[89]
Personalized Medicine	Type 2 Diabetes	Personalized treatment recommendations based on electronic health records	[78]
	Oncology	Tailoring chemotherapy treatments using genetic profiles and clinical data	[77]
	Cardiovascular Diseases	Predicting risk of cardiac events and tailoring prevention strategies	[79]
	Gene Therapies	Recommending gene editing techniques based on genomic data	[79]
Medical Imaging	Breast Cancer	Applying attention maps in deep learning models for mammogram analysis	[81]
	COVID-19 Detection	Using saliency maps to diagnose COVID-19 from chest X-rays	[82]
	Brain Tumors	Visualizing brain regions in MRI scans with Grad-CAM for tumor classification	[83]
	Diabetic Retinopathy	Highlighting retinal areas in AI predictions for diabetic retinopathy diagnosis	[84]
Remote Diagnostics and Telemedicine	Respiratory Diseases	Integrating XAI in telemedicine platforms for respiratory disease diagnosis	[26]
	Dermatology	Employing XAI for diagnosing skin conditions via mobile devices	[86]
	Ophthalmology	Interpreting AI models in telemedicine for eye disease diagnosis from retinal images	[88]

Table 2. Summary of XAI Applications in Healthcare.

Application	Specific Use Case	Description	References
Diagnostic Tools and CDSS	Oncology	Identifying cancerous lesions using medical imaging data	[69]
	Cardiovascular Diseases	Detecting patterns in ECGs indicative of heart diseases	[70]
	Pneumonia Detection	Using SHAP values for pneumonia diagnosis from chest X-rays	[72]
	Diabetic Retinopathy	Interpreting retinal images to predict diabetic retinopathy	[73]
	Neurological Disorders	Diagnosing Alzheimer’s using attention mechanisms on MRI scans	[71]
	Arrhythmia Detection	Enhancing transparency in deep learning models for arrhythmia diagnosis	[89]
Personalized Medicine	Type 2 Diabetes	Personalized treatment recommendations based on electronic health records	[78]
	Oncology	Tailoring chemotherapy treatments using genetic profiles and clinical data	[77]
	Cardiovascular Diseases	Predicting risk of cardiac events and tailoring prevention strategies	[79]
	Gene Therapies	Recommending gene editing techniques based on genomic data	[79]
Medical Imaging	Breast Cancer	Applying attention maps in deep learning models for mammogram analysis	[81]
	COVID-19 Detection	Using saliency maps to diagnose COVID-19 from chest X-rays	[82]
	Brain Tumors	Visualizing brain regions in MRI scans with Grad-CAM for tumor classification	[83]
	Diabetic Retinopathy	Highlighting retinal areas in AI predictions for diabetic retinopathy diagnosis	[84]
Remote Diagnostics and Telemedicine	Respiratory Diseases	Integrating XAI in telemedicine platforms for respiratory disease diagnosis	[26]
	Dermatology	Employing XAI for diagnosing skin conditions via mobile devices	[86]
	Ophthalmology	Interpreting AI models in telemedicine for eye disease diagnosis from retinal images	[88]

6. Responsible AI in Healthcare

The deployment of AI in healthcare necessitates a focus on responsible AI, which encompasses ethical considerations, accountability, transparency, and fairness. Responsible AI aims to ensure that AI systems are developed and used in ways that respect human rights, promote fairness, and enhance societal well-being. There are different aspects of responsible AI:

6.1. Ethical Considerations

Ethical considerations are paramount in the development and deployment of AI systems in healthcare. AI systems must be designed to uphold patient privacy and data security, ensuring that sensitive health information is protected throughout the data processing lifecycle [37]. This involves implementing robust data encryption methods, anonymizing patient data, and ensuring compliance with regulations such as the Health Insurance Portability and Accountability Act (HIPAA) and the General Data Protection Regulation (GDPR) [90]. Additionally, ethical AI practices must include obtaining informed consent from patients regarding the use of their data and providing information on how their data will be used and protected.

6.2. Accountability and Transparency

Accountability in AI involves the ability to explain and justify AI-driven decisions, enabling users to understand and challenge these decisions if necessary. In healthcare, accountability is critical as AI systems are often involved in high-stakes decisions that can significantly impact patient outcomes. To achieve accountability, AI models must be designed to provide clear, understandable explanations for their predictions and decisions. This requires incorporating XAI techniques that make the decision-making process transparent [38]. Transparency in AI systems involves making the decision-making processes visible and understandable to users [91]. This can be achieved through various XAI techniques such as LIME and SHAP, which shows how AI models arrive at their conclusions. Ensuring transparency and accountability in AI systems builds trust among healthcare professionals and patients. When users understand how AI models make decisions, they are more likely to trust and adopt these technologies in clinical practice. This trust is essential for the successful integration of AI into healthcare workflows, ultimately leading to better patient outcomes and more effective use of AI technologies.

6.3. Fairness and Bias Mitigation

Fairness in AI refers to the impartiality and equity in AI decision-making processes [92]. AI systems must be rigorously tested for biases that could lead to discriminatory outcomes. Bias can occur at various stages of the AI lifecycle, including data collection, model training, and deployment [93]. Techniques such as re-sampling the training data, adjusting model parameters, and incorporating fairness constraints during model training are essential to mitigate these biases [42]. Furthermore, to ensure fairness, it is important to use diverse and representative datasets that reflect the population the AI system will serve. This prevents the model from learning and perpetuating existing biases. Additionally, bias detection tools can be employed to identify and address any unfair patterns in the AI system’s decisions. Also, continuous monitoring and auditing of AI systems are crucial for maintaining fairness over time [94]. Fair AI systems contribute to building trust among users and ensuring the widespread adoption of AI technologies in healthcare. When AI systems are fair, they are more likely to be accepted and trusted by healthcare professionals and patients. This trust is vital for the effective deployment of AI technologies, which can lead to more equitable healthcare outcomes and improved patient care.

6.4. Human-in-the-loop Approaches

Human-in-the-loop (HITL) approaches integrate human judgment into AI decision-making processes, enhancing the accuracy, fairness, and accountability of AI systems [40]. HITL models ensure that human oversight is maintained, particularly in critical healthcare decisions, thereby increasing the trustworthiness of AI systems. Incorporating HITL approaches involves designing AI systems that allow for human intervention and feedback. This can include interactive interfaces where clinicians can review and modify AI-generated recommendations before final decisions are made. Such systems leverage the strengths of both AI and human expertise, leading to more accurate and reliable outcomes [95]. Furthermore, HITL approaches help to address the limitations of AI models by allowing humans to correct errors and biases that the AI might not detect. This collaborative approach ensures that AI systems are continually improved and aligned with clinical standards and ethical guidelines. By integrating human judgment with AI capabilities, HITL approaches enhance the overall effectiveness and trustworthiness of AI in healthcare.

7. Challenges and Opportunities in Implementing XAI in Healthcare

Despite the significant advancements in XAI, several challenges persist in its implementation within healthcare settings, such as:

7.1. Integration into Clinical Workflows

Integrating XAI methods seamlessly into existing clinical workflows is a major challenge. AI models must provide explanations that are both accurate and understandable to healthcare professionals who may not have technical backgrounds. Developing user-friendly interfaces that clearly present AI-generated explanations is crucial, as interactive visualization tools can assist clinicians in interpreting complex AI outputs, thereby enhancing their decision-making processes. Ensuring that XAI systems are interoperable with various healthcare IT systems, such as electronic health records (EHRs), is also vital. Standardized data formats and protocols can facilitate this interoperability, allowing different systems to communicate and share information seamlessly [96]. Furthermore, training and educating healthcare professionals is essential, equipping them with the necessary skills to understand and interpret AI-generated explanations and enabling a culture of trust and collaboration [12].

7.2. Regulatory Compliance

Regulatory compliance remains a significant hurdle in the implementation of XAI in healthcare. AI systems must adhere to strict regulations such as the Health Insurance Portability and Accountability Act in the United States and GDPR in Europe. These regulations emphasize the need for transparency and accountability in AI-driven decision-making processes. Therefore, XAI methods must be designed to meet these regulatory standards, providing clear documentation and audit trails for their decisions [90]. Compliance involves robust data encryption, anonymization of patient data, and ensuring transparency and auditability in data processing activities. Adhering to these regulations not only builds ethical and trustworthy systems but also gains the trust of healthcare professionals and patients. AI systems that comply with legal and ethical standards are more likely to be accepted and integrated into clinical practice, ensuring responsible AI use in healthcare settings [97].

7.3. Bias and Fairness

Biases can be introduced during various stages of the AI lifecycle, such as data collection and model training. For instance, an AI system trained on a dataset lacking diversity may perform poorly on underrepresented populations, worsening healthcare disparities. XAI techniques, such as feature importance and counterfactual explanations, can highlight biased patterns and facilitate the development of more equitable models [98]. However, identifying and quantifying bias in complex models can be difficult, and existing XAI methods may not always provide sufficient granularity to detect subtle biases. Additionally, mitigating bias without compromising the model’s performance poses a significant challenge. Ensuring robust and generalizable bias detection and mitigation techniques across different healthcare settings is critical [99,100].

7.4. Interpretability vs. Accuracy

Balancing interpretability and accuracy is challenging in XAI. Highly interpretable models, such as linear regression and decision trees, often have lower accuracy compared to complex models like deep neural networks [101]. This trade-off can limit the effectiveness of XAI in clinical settings where both accuracy and interpretability are crucial. The challenge is to find an optimal balance where the model remains sufficiently interpretable without significantly compromising its accuracy. Complex models, often referred to as "black boxes" due to their intricate internal structures, pose difficulties in interpretation. Simplifying these models can lead to a loss of critical predictive power, thus affecting clinical outcomes [32].

Opportunities to address this challenge include developing hybrid models that combine interpretable components with complex models. Techniques such as surrogate models, where an interpretable model approximates the behavior of a complex model, can provide explanations while maintaining accuracy. Advanced XAI methods like SHAP and LIME offer detailed insights into model predictions without significantly reducing accuracy. Ongoing research into more transparent architectures for complex models, such as inherently interpretable neural networks, holds promise for balancing interpretability and accuracy [33,102].

7.5. Long-term Impact on Patient Outcomes

The long-term impact of XAI on patient outcomes and the overall healthcare system requires further exploration. Longitudinal studies are needed to assess how the integration of XAI affects clinical practices over time, including its influence on patient trust, treatment adherence, and health outcomes. Understanding these impact could provide valuable insights into the effectiveness of XAI and guides future improvements in AI-driven healthcare solutions.

8. Discussion and Future Research Directions

This study has identified that one of the main benefits of XAI in healthcare is enhancing the transparency and trust in AI-driven decisions. Clinicians require clear and understandable explanations for AI predictions to make informed decisions, especially in critical scenarios such as diagnosing diseases or developing treatment plans. Techniques like SHAP and LIME have proven effective in providing these explanations, bridging the gap between complex AI models and clinical applicability [103]. Despite these advancements, there is an ongoing need for more user-friendly interfaces that present AI explanations in an easily understandable format for healthcare professionals.

Another major challenge is balancing interpretability and accuracy. Simpler, interpretable models may lack the predictive power of more complex algorithms, limiting their utility in clinical settings. Research into hybrid models that combine interpretable and high-accuracy components offers a promising solution. Achieving an optimal balance between interpretability and accuracy remains a significant challenge that needs further exploration [32,102].

Ensuring regulatory compliance is another critical aspect. AI systems in healthcare must adhere to strict regulations such as HIPAA and GDPR to ensure data privacy and security. XAI methods must be designed to meet these regulatory standards, including robust data encryption and anonymization techniques. The study emphasizes the importance of maintaining comprehensive audit trails for AI decisions to ensure accountability and transparency, which are essential for gaining trust from healthcare providers and patients alike [90,104].

Ethical considerations are paramount in the deployment of XAI in healthcare. The potential for biases in AI models to lead to discriminatory outcomes and even worsen existing healthcare disparities is a significant concern. XAI can help identify and mitigate these biases, but ensuring that these methods are effective across diverse populations is an ongoing challenge. Developing comprehensive frameworks for bias detection and mitigation, involving collaboration between AI researchers, healthcare professionals, and ethicists, is crucial for promoting equity in healthcare delivery [42,99].

Therefore, future research directions in XAI should focus on the following:

• Developing inherently interpretable models: Future research should prioritize the creation of models that are transparent by design, reducing the reliance on post-hoc explanation techniques. Inherently interpretable models can provide direct insights into their decision-making processes, making them more trustworthy and easier to integrate into clinical workflows.
• Integrating causal inference techniques: Combining causal inference with XAI can provide deeper insights into how different variables influence outcomes, which is particularly valuable in clinical settings. This integration can help in understanding the causal relationships within the data, leading to more robust and reliable AI models.
• Advancing visualization tools: Improved visual representations of model explanations, such as interactive dashboards and 3D visualizations, can enhance the usability of XAI tools for healthcare professionals. These advanced visualization techniques can help clinicians better understand and trust AI-driven decisions.
• Enhancing model robustness and generalization ability: Research should focus on developing XAI methods that are robust and generalizable across different healthcare settings and patient populations. This includes ensuring that XAI techniques can handle diverse and heterogeneous data sources, which are common in healthcare.
• Exploring the socio-economic impact of XAI: Research should also focus on the socio-economic impact of XAI in healthcare, including its potential to reduce healthcare disparities and improve access to quality care. Understanding these broader impacts can help in designing XAI systems that are technically sound and also socially beneficial.

By addressing these research directions, the integration of XAI in healthcare can be significantly improved, leading to more transparent, trustworthy, and effective AI-driven solutions that benefit both clinicians and patients.

9. Conclusions

This paper has provided a comprehensive review of XAI in healthcare, highlighting its potential to improve clinical decision-making, patient outcomes, and regulatory compliance. Techniques such as SHAP and LIME have proven effective in making complex AI models more interpretable and accessible to healthcare professionals, and these techniques were examined in detail. Additionally, the study explored challenges in healthcare AI and XAI, including the trade-off between interpretability and accuracy, the integration of XAI into clinical workflows, and the need for robust regulatory compliance. Challenges, opportunities, and future directions were also analyzed, contributing significantly to the literature. The study emphasized the need to focus on developing inherently interpretable models, integrating causal inference techniques, advancing visualization tools, and ensuring the ethical implications of XAI are addressed, which can lead to more transparent, trustworthy, and effective AI-driven healthcare solutions.