XAI-based Clinical Decision Support System: A Systematic Review

1. Introduction

A Clinical Decision Support System(CDSS) [1] supports decision-making for clinicians in diagnosing and treating diseases based on the patient’s clinical information. With the advancing big data analysis and artificial intelligence (AI) lately, the research on these techniques to CDSS has gained considerable attention. Notably, AI-based CDSS is highly valuable because of its effectiveness in supporting clinical diagnosis, prescription, prognosis, and treatment through AI models.

Traditional CDSS is categorized into knowledge-based CDSS, where rules are defined in advance and non-knowledge-based CDSS, where the rules are not predefined [2]. The principle of knowledge-based CDSS is to make decisions using correlations and If–Then rules for accumulated data. Knowledge-based CDSS is beneficial as the decision-making process is transparent because it follows predefined rules. However, it has the limitation that knowledge and rules must be defined in advance for all cases.

Conversely, non-knowledge-based CDSS provides decision-making by learning patterns found in past clinical information through machine learning or AI without rules. Non-knowledge based CDSS has been extensively studied in various medical areas [3,4], dealing with issues such as hypertension, heart failure, and lung disease. It is expected to be a breakthrough methodology that can reduce the cost of knowledge construction and provide personalized treatment. However, the black-box problem [5], in which explaining the process behind the results derived by AI models is not possible, makes it difficult to apply them in healthcare, where transparency is essential. Therefore, to utilize non-knowledge-based CDSS, it is necessary to introduce robust clinical validation and evaluations or provide convincing evidence to support the results.

Explainable AI (XAI) [6] was proposed to explain the process of the results of AI-based systems. The differences in knowledge-, non-knowledge-, and XAI-based CDSS are shown in Figure 1. XAI can explain the process and rationale behind an AI model’s decision in a manner that can be interpreted by the user. XAI technologies can be broadly categorized into feature, model-, complexity, and methodology-based, and XAI technologies can be applied to CDSS to ensure the transparency and reliability of the healthcare system. In this study, we propose an XAI-based CDSS framework and introduce its application. Our main contributions are summarized as follows:

• We perform a systematic review of explainable AI techniques that can ensure trust-worthiness and transparency in the medical domain and present a forward-looking roadmap for XAI-based CDSS.
• We categorized various studies from traditional CDSS to state-of-the-art XAI-based CDSS by appropriate criteria and summarize the features and limitations of each CDSS to propose a new XAI-based CDSS framework.
• We propose a novel CDSS framework using the latest XAI technology and show that potential value by introducing areas that can be utilized most effectively.

This paper is organized as follows. In Section 2, we describe the research trends related to knowledge-based, nonknowledge-based, and XAI-based CDSS and the limitations of existing technologies. In Section 3, we present the XAI-based CDSS framework and introduce the datasets and models required for its construction. Applications of XAI-based CDSS is presented in Section 4, followed by the conclusion in Section 5.

2. Related Work

This section categorizes CDSS into knowledge-based, nonknowledge-based, and XAI-based, and each section describes the main research, technologies, and methodologies, with the overview shown in Figure 2.

2.1. Knowledge-Based CDSS

To manage and utilize big data, an architecture called a knowledge base has emerged, and a methodology has been proposed to incorporate a knowledge base built by correlation of the accumulated data based on the experience of clinicians in a CDSS [7]. This is categorized as a knowledge-based CDSS that supports decision-making by inferring results from a rule-based knowledge base with an inference engine. Accordingly, it is important to design a knowledge base structure and appropriate rule system for each field. The features, functions and applied domain of knowledge-based CDSS are organized in Table 1.

Knowledge-based CDSSs define rules based on the literature, practice, or patient-oriented evidence [8] and are therefore often used in clinical practice based on clinical guidelines or in Evidence Based Medicine (EBM). Rule-base Inference Methodology using the Evidential Reasoning (RIMER) is based on a Belief Rule Base (BRB) system [9,10,11]. BRB systems set belief degrees to represent different types of uncertain knowledge and extend the If–Then rules to represent knowledge. Most BRB-based CDSS frameworks comprise an interfacial layer, an application processing layer and a data management layer [12,13,14]. These frameworks have proven their performance by in various fields, such as COVID-19 [15] and heart failure [16], psychogenic pain [17], tuberculosis [18], acute coronary syndrome (ACS) [19], and lymph node cancer metastasis[20].

However, an effective knowledge representation for CDSS could be Decision Tree, Bayesian Network, or Nearest Neighbors [21]. A study leveraging decision trees proposed a knowledge modeling method in which a clinical model extracted from glaucoma clinical practice guidelines was represented as mind maps and converted into iterative decision trees by clinicians and engineers [22]. In a similar study, mind maps representing the clinical treatment process for thyroid nodules obtained from clinicians is converted into an iterative decision tree model to extract rules [23]. This is followed by the process of representing tacit knowledge to explicit knowledge and finally converting it to executable knowledge. In another study on a decision-tree-based CDSS, a pediatric allergy knowledge base, ONNX inference engine, and tree algorithm were used to provide knowledge of diagnosis and treatment to clinicians [24].

Additionally, Bayesian network-based CDSS, which is used in various medical areas, (such as liver disease [25], breast cancer [26], infectious diseases [27], diabetes [28], angina [29], respiratory diseases [30], and lymph node cancer metastasis [31]) uses a what–if analysis mechanism. In the field of dental hygiene, CNNs have recently been used in conjunction with a Bayesian network framework based on the (Expectation-Maximization (EM) algorithm to detect abnormal oral images [32]. Additionally, a diagnosis of an Hepatitis C Virus (HCV) diagnosis system has been proposed using a fuzzy Bayesian network with a fuzzy ontology to resolve ambiguity and uncertainty in outbreaks [33].

Research using the K-Nearest Neighbor (KNN) algorithm structured medical information by classifying similar clinical cases through ontology extraction methods for Case Based Reasoning (CBR) [34]. Moreover, a Computer-Aided Diagnosis (CAD) system proposed for melanoma diagnosis provides a related ontology based on Asymmetry, Border, Color and Differential (ABCD) structure rules, and classifies similar melanoma cases using the KNN algorithm [35]. Another study that uses similarity of knowledge for decision-making, as in the aforementioned studies, provides an appropriate diagnosis for patients semantically classified by time-series similarity based on the patient’s medical history [36].

Knowledge-based CDSS supports decision-making based on pre-built knowledge base, effective data modeling, and knowledge-based updating for each domain is also ongoing. Recently, in genomics, a clinical genomic data model has been proposed to analyze clinical genomic workflow and extract attributes using genomic data for clinical application of genomic information [37]. Additionally, methodologies have appeared to facilitate knowledge base updating by analyzing newly acquired textual knowledge through natural language processing to generate rules [38].

Knowledge-based CDSS has potential in that the decision-making process is clear and traceable. However, it is limited by maintenance and construction costs because it relies on medical specialists and knowledge engineers for standardization and error correction, as data quality control is essential [39,40].

2.2. Non-Knowledge-Based CDSS

With the explosion of data and specialized knowledge, the amount of information that must be processed to make clinical decisions is growing astronomically. To learn on its own like a human, using massive amounts of data, deep learning, and AI, which is based on artificial neural networks, supports clinical decision-making. These methods analyze patterns in patient data to draw associations between symptoms and diagnoses. Moreover, deep learning and AI can be used to analyze various data, including text, images, videos, audio, and signals, enabling the development of non-knowledge-based CDSS that can understand the overall clinical situation and context. The first step toward a nonknowledge-based CDSS began with analyzing images and using them to make clinical decisions. A convolutional neural network (CNN) [41], which trains image patterns by mimicking the structure of the human optic nerve has been used to diagnose obstructive sleep apnea by learning high-order correlation features between polysomnography images and their labels [42,43], and an automated system has been proposed to optimize patient satisfaction by analyzing patients’ experiences with ambulance transport services with a combined model of CNN and word embeddings [44]. Similarly, a technique for diagnosing melanoma using a single CNN trained on a dataset of clinical images has been introduced [45].

There are also a number of cases of recurrent neural networks (RNNs) that can handle time-series data. EHR data are good candidates for using RNNs [46] because it provide clinical records with temporal information. A previous study [47] applied RNNs to the EHR data of heart failure patients to predict heart failure outperformed machine learning methods such as SVMs [48], MLPs [49], logistic regression [50], and KNNs [51]. Because ECG data also contain temporal information, ECG signals can be analyzed using RNN-based models detect sleep apnea [52].

When dealing with clinical data, owing to its long-term properties, the problem of forgetting previous data and ignoring past information may arise. Therefore, studies using LSTMs [53] to predict future data by considering past data have been proposed. An LSTM was used to learn multiple diagnostic labels to classify diagnoses [54] and oral–nasal thermal airflow, nasal pressure, and abdominal breathing-guided plethysmography data from polysomnography were analyzed with a bidirectional LSTM model to diagnose sleep apnea [55]. Deep learning is frequently applied in medical image analysis. Chest radiographs can be analyzed using deep learning to diagnose chest diseases such as lung nodules [56], lung cancer [57], and pulmonary tuberculosis [58].

Unlike traditional supervised and unsupervised learning, reinforcement learning [59] generated its own training data by observing the current state and selecting future actions. Because existing CDSSs are trained based on evaluations made by different clinicians with different criteria, interrelated symptoms are not considered in some cases. This problem can be solved by applying reinforcement learning, which is used to learn complex environments. A CDSS based on a deep reinforcement learning algorithm has been introduced to determine the initial dose for ICU patients, where an accurate medication prescription is critical, and prevents mis-dosing and complications. [60] Reinforcement learning of secure computations enables the implementation of patient-centered clinical decision-making systems while protecting sensitive clinical data. A privacy-preserving reinforcement learning framework with iterative secure computation was proposed to provide dynamic treatment decisions without leaking sensitive information to unauthorized users [3]. A reinforcement learning-based conversational software for radiotherapy was also studied, where the framework used graph neural networks and reinforcement learning to improve clinical decision-making performance in radiology with many variables, uncertain treatment responses, and inter-patient heterogeneity [61].

BERT [62], a large language model based on a Transformer [63], was used to develop a CDSS with natural language understanding capabilities. To reduce diagnostic errors, a framework for multi-classifying diagnosis codes in EHRs using BERT [64] has been developed to help clinicians predict the most likely diagnosis. However, specialists have raised concerns about the reliability and responsibility of these deep learning and AI models because of their inability to explain their decisions. Therefore, they are often unwilling to use them in diagnosis. To this end, it is necessary to adopt AI, which provides evidence for prediction and an understandable explanation.

2.3. XAI-Based CDSS

EXplainable AI (XAI)[5], has emerged to overcome the black-box[6] problem of deep learning models, which means that deep learning models have the highest perfor-mance compared to other rule-based or machine learning models but have the limitation of lacking interpretability. This can be described as the "performance–interpretability trade-off, " and is shown in Figure 3. Performance is highest for deep learning, followed by machine learning models (Decision Tree, Nearest Neighbors, Bayesian Network), and rule-based models; however, transparency (interpretability) is inversely proportional. In other words, applying XAI to deep learning models is capable of explaining the reason and logic behind the results predicted by the model to ensure transparency and reliability of the results with high-performance deep learning. With attempts to apply XAI in various fields [65], XAI is gained attention as a solution to the uncertainty problem in CDSS systems where accuracy and reliability are important [66].

The description techniques used in XAI can be broadly categorized into scoop-, model-, complexity-, and methodology-based [67]. The most popular XAI methods in recent research include SHapley Additive exPlanations (SHAP) [68], (local interpretable model–agnostic Explanations (LIME) [69], Post hoc Interpretability [70], and (Gradient-weighted Class Activation Mapping (GradCAM) [70].

Scoop-based techniques determine the contribution of data based on the importance features to train the AI model. A prominent example is the local explainers method (LIME) [71] is a method specific to a particular instance or outcome. LIMEs directly explain how the model’s input data change the outcome, and after training the model, it can make guesses about samples that have not appeared before [72]. For COVID-19, LIME and traditional machine learning models were combined to identify the features that had the greatest impact on medical history, time of onset, and patients’ primary symptoms [68]. Similarly, an LSTM model was used in a study on depressive-symptom detection, in conjunction with a LIME approach to identify text suggestive of depressive symptoms [73]. Other applications include the diagnosis of Parkinson disease [74], hip disease [75], Alzheimer’s disease and mild cognitive impairment [76].

By contrast, SHAP is a global explainer method that provides a theoretical interpretation of any dataset uses cooperative game theory concepts [77] to calculate the contributions of biomarkers or clinical features (players) for a specific disease outcome (Reward) [72]. To predict postoperative malnutrition in children with congenital heart disease, the XGBoost and SHAP algorithms were used to calculate the average of five risk factors (weight one month after surgery, weight at discharge, etc.) for all patients [78]. PHysiologicAl Signal Embeddings (PHASE), a method for transforming time-series signals into input features, wasfirst applied to embed body signals with an LSTM model to features extracted using SHAP from EMR/EHR data [79]. In addition, a multi-layer XAI framework utilizing multimodal data, such as MRI and PET images, cognitive scores, and medical histories, have been proposed [80].

SHAP is applied to all layers of the framework, where the first layer performs multiple classification for the early diagnosis of AD. In the second layer, the binary classification score is were used to determine the transition from cognitive impairment to AD [80]. Similarly, SHAP has been widely used in various diseases and clinical domains such as predicting readmission [81,82], COVID-19 [83,84,85,86], liver cancer [87], influenza [88], and malignant cerebral edema [89].

More recently, researchers have utilized LIME and SHAP simultaneously to ensure a convincing description of the system. A hybrid approach combining Vision Transformer (ViT) and a Gated Recurrent Unit) was used to generate LIME heat maps using the top three features from the brain MRI images, and SHAP was used to visualize the model’s predictions to demonstrate the validity of data patterns [90]. In addition, the Department of Chronic Kidney Disease also used LIME and SHAP algorithms simultaneously to represent the importance of features in the best model trained by five machine learning methods methods (Random Forest, Decision Tree, Naïve Bayes, XGBoost, and Logistic Regression) [91].

Model-based techniques can be classified into model-specific and model-agnostic methods. Model-specific methods utilize the unique features of a model to make decisions, indicating that they can only be applied to the internal operations of a specific model. An example is Score-CAM [92], which is based on CNNs and compares output for the given input features, thereby indicating their importance. A previous study proposed a system for classifying images from a clock-drawing test as a tool for diagnosing dementia was trained on API-Net [93] and visualized it using Score-CAM to provide explainability, and transparency[94]. However, model-agnostic methods are model-independent and can be applied to any model or algorithm. As a CDSS tool that reduces the model dependency, a COVID-19 symptom severity classifier that utilizes different machine learning models to identify high-risk patients for COVID-19 has been proposed [95].

Complexity-based techniques make machine learning or deep learning models fully interpretable. Interpretability can be categorized into intrinsic interpretability [96] and post hoc interpretability [72] depending on the viewpoint. In general, intrinsic interpretability indicates that a model with a simple architecture can be explained by the trained model itself, whereas post hoc interpretability means that the trained model has a complex architecture and must be retrained to explain this phenomenon. In a study on brain tumor detection based on MRI images, three pre-trained CNNs, DarkNet53 [92], EfficientNet-B0 [97], and DenseNet201 [98], were used to extract features using a hybrid methodology to explain post-interpretability [99].

Another framework for brain tumor diagnosis, NeuroNet19, combines a 19-layer VGG19 that detects complex hierarchical features in images with an inverted pyramid pooling module (iPPM) model, which refines these features, leveraging post-interpretability [100]. Methodology-based techniques are categorized into Backpropagation-based and Perturbation-based [67], among which Backpropagation-based GradCAM was proposed to describe CNN models with good performance [101]. GradCAM was applied to the convolutional layer at the end of the CNN, and uses the gradient information of the layer to find the features that are highly involved in a particular decision [72,102]. To further improve classification performance, several studies have been proposed to predict oral cancer from oral images using guided attention inference network (GAIN) along with the aforementioned CNN-based VGG19 model GradCAM [103], and also to diagnose glaucoma from fundus images using GradCAM’s heatmap and ResNet-50 model [104]. SBecause CNN models are widely applied in image classification and processing, GradCAM technology is used in several studies utilizing image data [105,106,107,108,109,110,111].

Table 2. Non-Knowledge-based CDSS.

Table 2. Non-Knowledge-based CDSS.

Table 3. XAI-based CDSS.

Table 3. XAI-based CDSS.

3. XAI CDSS Framework

3.1. Proposed Architecture

This study proposes an XAI-based CDSS framework that can handle both high-performance and interpretation of the decision-making process. This is achieved by applying explainable AI technology to artificial intelligence models. Details of this framework are shown in Figure 4. The proposed framework handles multimodal data from various medical domains, including text, audio, images, and genomes. It applies the explainable AI methodology to representative deep learning models. Finally, we demonstrate the potential value of the proposed framework through the presentation of application plans, illustrating the circumstances under which they can be utilized effectively.

Most of the existing AI-based CDSSs are limited to text data. Even if the data-usage range is extended, only one additional type of biometric signal or image is used. However, a multimodal data utilization plan is necessary to make clinical decisions considering factors that are difficult to record in formal form, such as the patient’s condition, facial expressions, and behavior that change in real time. Consequently, the proposed framework must be capable of expanding its knowledge through the continuous learning and analysis of information derived from multimodal data. As illustrated in Figure 2, a knowledge graph was constructed by extracting the relationship between features based on multimodal features obtained using models such as large-scale language models, VATT, and audio analysis models derived from multimodal data such as text, images, and signals. Furthermore, through reinforcement learning or continuous learning, knowledge graphs can expand automatically, enabling them to respond flexibly to new knowledge, while retaining previous information.

A multimodal clinical XAI learning model for an explanatory clinical decision-making system requires three elements: a deep explanatory label, an interpretable model, and model inference. First, the features that can explain the prediction results must be identified and labeled, and a model must be created in connection with a decision tree with high explanatory power. The explainable models were inferred from the black boxes, which are the largest problem of existing AI methodologies. Explanations are generated with predictions. All of the aforementioned processes [114] are conducted through interactions for patients, medical professionals, and clinical systems in real time. The predictions and explanations generated by analyzing the data obtained from each participant are provided to the participants once more to facilitate overall clinical decision-making, including monitoring, diagnosis, prescription, warning, and document management.

3.2. Dataset

3.2.1. Clinical Dataset

The first large-scale multimodal clinical dataset is UK Biobank [115]. It has been collected since 2006 and includes several hierarchical data types such as lifestyle habits, body information, biological samples, electrocardiogram data, and EHR data from more than 500,000 participants. In addition to basic biometric data, the dataset provides genomic analysis, exon mutation testing, full-length genome sequencing, brain MRI, cardiac MRI, abdominal MRI, carotid artery ultrasonography and radiographic findings. Similar datasets were also collected from the China Kadoori Biobank [116] and Biobank Japan [117]. The MMIC dataset [118], published by the Massachusetts Institute of Technology, has now been published up to the fourth version. This is an open-source dataset comprising EHR data, including demographic information, diagnostic codes, and medications obtained from ICU patients at the Beth Israel Deaconess Medical Center. MMIC-IV is one of the the most representative datasets of clinical AI models that aim to predict clinical events or readmissions. It contains textual data, such as reports, medical notes, and imaging data, including laboratory and physiological data, and chest radiographs. Furthermore, it is possible to reconstruct the multimodal dataset by utilizing a combination of data from a single modality. This may include, for example, an Alzheimer’s patient’s brain image dataset, the Alzheimer’s Disease Neuroimaging Initiative (ADNI)[119], and exercise activity data from patients with schizophrenia and depression[120]. Table 4 lists the summary of the monomodal and multimodal clinical datasets.

3.2.2. Knowledge Graph

The use of knowledge graphs enables the formal expression of medical expertise and the semantic structuring of unstructured multimodal data. During this period, while learning existing HKG, the knowledge graph can be expanded using new data. In particular, the emergence of large language models has facilitated the construction of more comprehensive and accurate HKG. HKG for CDSS exists in fields such as medicine (prescription) [141,142], genetics [142,143], and disease [144,145], and based on this, an extended knowledge graph can be generated.

To effectively respond to new clinical cases, information collected in real time should be used in conjunction with existing knowledge. As mentioned previously, an automatic scalability of knowledge graphs is required and reinforcement learning can be applied to ensures persuasive power. Numerous knowledge graphs exist in the clinical fields other than CDSS, which are summarized in Table 5 by application range.

3.3. XAI Model

Prior to the analysis of the multimodal data, the process of fusing multimodal data is required. In particular, it is important to select meaningful features to obtain the desired information from the vast amounts of data. To achieve this, deep learning, which trains a neural network composed of multiple layers, can be used to extract features and representations from complex data. Deep learning models[158] are well-suited to the integration and extraction of meaningful information owing to their capacity to learn complexpatterns and generate knowledge for decision-making through the processing of vast amounts of data. Deep learning models[159], such as those pre-trained on large datasets such as ImageNet[160] or natural language corpora can be employed to obtain correlations through the generation of new samples from multiple modalities with generative models from GAN[161] or VAE [162].

The advent of the transformer [62] has made multimodal data more accessible as it enables immediate inference based on an attention network rather than a model using convolutional structure, as previously described. Using the transformer structure, it is possible to learn multiple modalities together because encoding is possible in a consistent structure for all data modalities. Since the proposal of the Vision Transformer (VT) [163], that encodes images in a manner similar to natural language, there have been numerous attempts to apply the Transformer to other modalities in video and voice.

Recently, VATT [164], a framework for learning multimodal representations from unlabeled data, has been developed for the extraction of multimodal representations from unlabeled signals. It is possible to perform a range of tasks, including behavioral recognition, Voice event classification, image classification, and text-video search using video and audio, and text features extracted from Internet images. Based on the VATT model, the patient’s daily, and counseling videos can be analyzed to identify biometric signals, changes over time and nonverbal expressions and use them in clinical decision-making.

XAI techniques can be largely divided based on explanation method, interpretation method, model specificity, and explanation range [165], as shown in Figure 5. The backpropagation-based XAI measures the degree to which each feature affects the result as a gradient value. Class activity map-based XAI visualizes features with a significant impact using the feature map of the uppermost layer, which aggregates the necessary information. Finally, input interference-based XAI provides explanatory power through the process of repeatedly investigating the model while making various changes to the input value of the model.

4. Applications

4.1. Function of CDSS

The objective of CDSS is to facilitate optimal decision-making for patient safety. Consequently, the role of CDSS is to support the decision-making processes involved in Diagnosis, treatment, and prescription, which are directly related to patient safety. Table 6 illustrates the functions of CDSS.

The first function of CDSS is diagnostic support. CDSS provides diagnostic information on suspected diseases by collecting and monitoring figures showing the patient’s conditions, such as biosignals. This function provides efficient management of beds by supporting useful decision-making to inexperienced clinicians or nursing personnel.

The second function is treatment support. This function supports the determination of the optimal treatment by analyzing all applicable treatment methods in consideration of the patient’s current condition. Through treatment method analysis, information analysis such as inconsistencies, errors, omissions, and side effects between the treatment methods is possible. This function is the most commonly studied area, and provides interaction information between prescribed drugs or checks for side effects when multiple drugs are prescribed together. The third function of CDSS is medical image analysis. As the performance of deep learning models using image data develops, it is the most commonly used CDSS functions related to AI in the medical field. By analyzing medical image data such as X-rays, MRIs, and CT scans, which are most commonly used to identify patient diseases through deep learning, more accurate decision support can be provided to clinicians.

Finally, the system serves as a risk notification function. This function is typically activated in patients admitted to the hospital, and immediately alerts medical personel when abnormal symptoms or dangerous levels are identified while collecting and monitoring patient biometric signals, such as pulse rate, blood pressure, and temperature.

4.2. The Potential of XAI-Based CDSS

The most significant features of the proposed XAI-based CDSS framework are generalizations owing to the use of multimodal data, scalability to apply various deep learning SOTA models, and trustworthiness through explainable AI technologies. The Fusion and learning of various types of multimodal data generated in the medical domain can result in the discovery of features and patterns that were not found in models that only address single modalities. This can be achieved by developing a generalized foundation model using deep learning models that achieved SOTA performance in each field. Finally, the decision-making process can be explained and interpreted as various explainable AI technologies, enabling the development of a transparent and reliable clinical decision-making system. Figure 6 illustrates the areas where the XAI-based CDSS. framework proposed in this study is the most efficiently utilized. Disease detection represents the first field in which it can be used. Research related to deep learning-based disease detection has already demonstrated high performance; however, there is a challenge in utilizing it in the actual medical domain because of issues such as the black-box nature of deep learning. To address this challenge, explainable AI technology is employed to elucidate the rationale behind the predictions generated by the deep learning model, thereby facilitating its deployment in the medical domain.

The second area of application is bed decision support for inpatients. In comparison to current inpatients, there is a significant challenge in that the number of professional nursing personnel in charge of patients is insufficient. Furthermore, in the case of an inexperienced nursing personnel, it is challenging to identify and diagnose appropriate symptoms necessary for patients. The XAI-based CDSS framework is capable of simultaneously monitoring multiple patients and efficiently supporting the decision-making of professional nursing personnel by providing the cause and basis when abnormal symptoms are detected.

The third area of application is treatment and prescription. By analyzing the patient’s medical history or medical record, customized treatment and prescription optimized for the current patient are possible. Specifically, when prescribing a drug, it is possible to derive more optimal prescription results for patient safety by analyzing potential side effects that may occur between the components of the drug prescribed together in advance and providing information to professional medical personnel.

The final area in which XAI-based CDSS can be utilized is clinical practice. Unlike the aforementioned applications in hospitals and medical institutions, it is employed in the education of professional personnel. In particular, in the case of professional nursing personnel, clinical practice is conducted in educational institutions on cases that may occur in hospitals. The use of XAI-based CDSS explains the decisions that should be made about the situations that may occur in the ward and the reasons for the decisions.

5. Discussion and Conclusion

This paper addressed the technological trends of the clinical decision-making system CDSS in accordance with the decision-making system (DSS) framework. It begins with a traditional knowledge-based CDSS and then moves on to a non-knowledge-based CDSS incorporating AI and a multimodal-based XAI CDSS that ensures reliability. By organizing the CDSS services introduced above by field and feature, it is possible to grasp the strengths and weaknesses of each CDSS service, and the matters required for CDSS services in the future. However, in the case of existing systems, there are limitations. These include a limited data utilization range, lack of explanatory power for AI models, and the opacity of the decision-making process. Consequently, in the medical field, which demands reliability and transparency, the decision-making process must be clearly represented for users to interpret it. This paper addressed the black-box problem of unknown-based CDSS as a solution to the problem and proposed an XAI-based CDSS framework that provides valid evidence and reasons for the results. Furthermore, it introduced the available datasets, models, and resources.

The proposed framework is designed to construct an automated extended knowledge graph with multimodal features derived from multi-format data. It comprises three key elements: a deep explanatory label, an interpretable model, and model inference, which collectively facilitate explainable AI. The framework has the potential to automate the entire process of medical clinical services, from personalized treatment to real-time patient condition reflection. It distinguishes itself from existing systems in terms of multimodal data management, utilization plan, explainable AI application plan, and CDSS application range. Furthermore, the medical knowledge graph and HKG, which are structured using available medical multimodal data, are summarized to enhance expertise in decision-making. The proposed XAI-based CDSS framework serves as a foundational model that can be flexibly applied to multiple disease domains. This approach enables the development of a medical system with minimal temporal and spatial constraints.

Furthermore, advancements in the medical field are facilitated by the ease with which computerized data can be used for research purposes. However, measures to enhance social awareness are necessary because the current medical data are not readily accessible owing to concerns regarding the protection of sensitive personal information. Additionally, there is a prevailing attitude that humans should be trusted even when explanations of the data sources, transparency regarding decision-making processes, and the results are provided. As non-face-to-face medical systems become increasingly prevalent, it is imperative that relevant legal deregulation be enacted. If social awareness and institutional improvements are guaranteed, it is expected that this will facilitate the development of compelling medical solutions through CDSS research and development that integrate richer medical multimodal data, medical knowledge graphs, and XAI technology.