Machine Learning Approach for Analyzing 3-Year Outcomes of Patients with Brain Arteriovenous Malformation (AVM) after Stereotactic Radiosurgery (SRS)

1. Introduction

Cerebral arteriovenous malformation (AVM) is a congenital neurological disease that causes cerebral hemorrhage, seizures, or headache. It consists of an abnormal conglomerate of dilated cerebral vessels derived from maldevelopment of the capillary network that allows direct connections between cerebral arteries and veins [1]. One of the treatments known since the 1970s, in addition to microsurgery and endovascular therapy, is stereotactic radiosurgery (SRS), in which the AVM is obliterated by radionecrosis through the administration of multi-beam directed radiation [2], which is obtained from a gamma knife system, a linear accelerator, or proton therapy. From the medical point of view, SRS is a neurosurgical technique that does not require an incision and is used as an alternative or complement to noninvasive treatment.

The healing process of patients with AVM undergoing SRS is not immediate and requires time with clinical and imaging monitoring to know the evolution of the disease, the need for additional or alternative treatments, and the possible results. Successful exclusion of brain AVM with radiosurgery depends on the size of the lesion and the radiation dose. Complete healing is considerably higher for smaller lesions. For example, one study showed that the obliteration rate of patients with brain AVM after SRS was between 54-92% for lesion diameters ≤ 2.5 cm [3]. Several scoring systems, such as the Spetzler Martin Grading Scale (SMGS) and the Virginia Radiosurgery AVM Scale (VRAS), are currently used by physicians to understand the nature of AVM and predict the results of radiosurgery treatment [4,5,6]. However, developing new methods to predict the results of radiosurgery treatment and determining the factors that influence the probability of success are needed.

Machine learning (ML) is a subset of artificial intelligence (AI) that uses algorithms that automatically “learn” to identify patterns in data, which are used to make forecasts based on these patterns [7]. The use of such algorithms as support tools for medical decision-making and their application in the prognosis, diagnosis, and treatment of diseases has been recently developed [8]; however, certain conditions still exist that make it difficult for them to be widely adopted [9,10,11,12,13]. Among the studies referring to the prediction and diagnosis of neurological and brain diseases in which ML techniques were applied is the study of Uspenskaya-Cadoz et al. [14], which proposed a method for diagnosing Alzheimer's disease (AD) by applying logistic regression (LR), decision tree (DT), random forest (RF), and gradient boosted trees (GBT) techniques, and the study of Ghafouri-Fard et al. [15], which proposed using artificial neural networks (ANN) to predict multiple sclerosis (MS) risk based on genotypes.

At present, the application of ML techniques to the diagnosis, prognosis, or treatment of AMV has increased. Interesting studies can be found, such as one by Tao et al. [16], which examined the factors that influence the risk of bleeding from AVM based on LR and RF models, and another by Hong et al. [17], which reported an experiment with three ML models: naive bayes (NB), support vector machines (SVM) and RF, for the detection of hemorrhages in AVMs using digital subtraction angiography (DSA) images. There are also studies on the use of deep learning, a type of ML specialized in image processing; for example, Wang et al. [18] automated the process of segmenting and identifying AVMs in computed tomography (CT) and DSA images. Other studies have focused on the prognosis of patients with AVM after surgery, with the aim of predicting whether they would be cured. For example, Asadi et al. [19] presented a study on identifying the factors that influence the outcome of treatment with endovascular embolization and concluded that AVMs can be successfully treated by endovascular techniques or a combination of surgery and radiosurgery with an acceptable risk profile, and that ML techniques can satisfactorily predict outcomes with high accuracy and can help to individualize the treatment based on key predictors. Finally, Oermann et al. [20] used an ML approach to predict the outcomes of AVM patients undergoing radiosurgery, and achieved 74% accuracy, which is considered to be the best prognostic result as of the date of publication of this paper. However, these studies did not delve into the importance of identifying the predictors of the prognosis outcomes, which is important information for medical decision-making.

Taking into account these studies [16-20], in which it is evident that ML algorithms are powerful tools that can be used in the medical field, and seeking to implement an ML method that allows accurate prediction of AVM outcomes and identification of the most important factors that explain whether patients with AVM will be cured, in the present study, we used a dataset of 202 patients with AVM who underwent radiosurgery along with ML models to develop a suitable model. The aim is to (1) provide an ML approach for predicting whether patients with AVM who undergo SRS will be cured, and (2) identify the main factors influencing whether these patients will be cured 36 months after radiosurgery.

2. Materials and Methods

The construction of an ML system for the prognosis of patients with AVM treated with SRS is proposed using two techniques: DT and LR (Figure 1).

The objective is to predict whether a patient will be cured or not at 36 months (3 years) after undergoing SRS; for this, a supervised ML learning approach was chosen via binary classification. Additionally, the use of LR is proposed to determine the main factors that influence whether an AVM patient will be cured.

2.1. Dataset

For this study, a dataset comprising 45 variables of 202 patients diagnosed with AVM who underwent SRS treatment to cure this disease was used. The data were collected from different medical sources at the Instituto de Radiocirugía del Perú (IRP) between 2005 and 2018 following the process shown in Figure 2.

The variables that were collected from patient data were considered as input data (predictors) and were grouped into 5 categories: sociodemographic (S), clinical (C), treatment (T), angioarchitecture (A), and radiosurgery (R). The variable for patients being cured at 36 months after radiosurgery was considered as output data (response). Table 1 shows the structure of the dataset used in this study.

The dataset is tabular and is made up of 202 records (rows) and 45 variables (columns), in which the rows correspond to the patient data and the columns represent the variables considered in the study. The first 44 variables were considered as input variables to the system (independent variables) and the last column as the output variable (dependent variable), representing patients being cured (cured = 1) or not (cured = 0).

2.2. Data Preprocessing

Before carrying out any data processing and because this was a medical application, it was advisable to analyze the data regarding possible confounding variables that could have an undesired impact on our prediction results [21]; for this, we analyzed the possible confounding variables gender and age.

For the categorical variable gender, the chi-square test of homogeneity was performed to verify whether the difference in the number of men and women in each data group was statistically significant, and no difference was found (p-value = 0.566; Figure 3a). For the age variable, Student's t-test was applied to verify whether there was a statistically significant difference in age between groups (class 0, mean = 31.97; class 1, mean = 26.72), and again no difference was found (p- value = 0.058; Figure 3b).

From this analysis, we concluded that the variables age and gender should not be considered as confounding variables, so we moved forward with the data preprocessing.

Finally, in order to avoid prediction biases and build the ML system effectively, variable selection and data balancing were carried out.

2.2.1. Variable selection

After an analysis by expert judgment, 6 independent variables were identified that were considered not to influence the prognosis of being cured, so they were excluded from the study (residence, education_level, health_insurance, mri_examination, ct_examination, and das_examination).

Additionally, correlation analysis of the 38 remaining independent variables was carried out; Cramer’s test [22] was applied to identify the linear correlation between categorical variables and Pearson's test (Pearson's correlation coefficient) for the numerical variables; in both cases, a threshold value greater than or equal to 0.7 was used to determine correlation, and 6 correlated variables that exceeded the threshold were identified (Table 2). The dython library [23], which is available for the Python programming language, was used to perform the calculations. Finally, 12 independent variables were discarded, leaving a dataset made up of 32 independent variables and 1 dependent variable, which were used in the ML system proposed in this study (Table 3).

* Dependent variable.

The final dataset was made up of 202 records, with 32 independent variables and 1 dependent variable, which was divided into two datasets, 75% (n = 151) for ML model training and validation and 25% (n = 51) for testing. In addition, the 32 independent variables of the training and validation set were normalized using the min-max technique [24].

2.2.2. Data balancing

The original training dataset had a data imbalance with respect to the dependent variable, cured, in that it consisted of 125 records of class 1 and 26 of class 0. The imbalance was corrected by applying the synthetic minority oversampling technique (SMOTE), which creates new synthetic instances of the minority class instead of repeating them [25,26]. We obtained 250 records in total, 125 records for each class, as shown in Figure 4.

Finally, two training datasets were obtained: an imbalanced training dataset made up of 151 records, and a balanced training dataset made up of 250 records. Both datasets were represented by a data matrix of dimension n × 32, in which the observation i can be expressed as o_i = [o₀, o₁, . . ., o₃₂] ∈ R^nx32, where n is the number of observations or records in the dataset.

2.3. Machine Learning Models

For the construction, validation, and evaluation of the ML system, we used the process shown in Figure 5, which consisted of using the two training datasets (balanced and imbalanced) to build and validate the two ML models (DT and LR) in four experimental scenarios; based on the results, the model with the best performance metrics was chosen. Scenario 1 refers to the imbalanced training data with the DT model, scenario 2 refers to the imbalanced training data with the RL model, scenario 3 refers to the balanced training data with the DT model, and scenario 4 refers to the balanced training data with the RL model.

Additionally, the LR method was used to identify the most important factors that determine the probability of patients’ being cured (clinical interpretability).

In the training phase, the grid search technique [27] was used to find the optimal hyperparameters of the ML models in each of the four scenarios. The set of search values defined for the hyperparameters is given in Table 4.

During the training process, the resampling technique was used (Figure 6), in which the training dataset was divided into 8 subsets, with 1 set taken for validation and 7 for training, following 8-fold cross-validation, which is a commonly used method for selecting ML models [28,29].

To build the ML models, the scikit-learn 1.0.2 library [30] of Python version 3.8.16 was used in the Google Colab environment. The algorithms and resources used in this research can be found at: https://github.com/mirkorodriguez/ml-prediction-mav

3. Results

The composition of the study population, the performance of the prediction models, and the explainability of the prediction are presented below.

3.1. Study population

This study included 202 patients with AVM who underwent stereotactic radiosurgery between 2005 and 2018 at the IRP. As shown in Figure 7, 167 patients (82.20%) were cured 36 months after the surgical intervention.

Table 5 shows the sociodemographic characteristics of the population included this study: 52.97% were men and 47.03% were women; 70.49% of patients were in the age range of 18 to 59 years; 80.69% were from Lima or Callao; 18.82% had a preschool or grade school education and 52.97% had only a high school education; and 42.08% had insurance through the Ministry of Health of Peru (SIS).

Table 6 shows the clinical characteristics of the patients. It can be seen that the average time from radiosurgery to AVM cure (obliteration) was 22.07 months, the average radiation dose was 17.86 Gray, the average AVM diameter was 2.14 centimeters, and the average number of isocenters applied was 1.35. On average, radiosurgery was performed in a single session.

Table 7 shows the statistics of the patients’ previous treatments before SRS. Of the 202 patients, 31 had undergone surgical treatment and 49 had prior embolization. As part of the treatment, 22 only underwent surgery, 40 only embolization, and 9 both surgery and embolization. The embolizing agents were Onyx (52%) and Histoacryl (48%). In total, 155 patients had previous cerebral hemorrhage, 76 developed encephalomalacia, 178 had headache, and 112 had seizures; furthermore, 55% presented some type of deficit (motor, sensory, or cognitive). Regarding the angioarchitecture (characteristics) of the AVM, most (100) were located on the left side of the brain and most (96) were categorized as deep; most treated AVMs (95) had moderately intense flow.

Finally, Table 8 shows the anatomical locations of AVMs, which were mainly found in basal ganglia (16.83%), frontal lobe (9.9%), insular cortex (6.93%), parieto-occipital region (6.93%), mesio-temporal region (6.93%), and cerebellum (6.44%).

3.2. Performance of prediction models

The results obtained by the models using the data in the testing set are described below.

Table 9 shows the optimal hyperparameters identified for each scenario that were used in the models for prediction.

Figure 8 shows the confusion matrices obtained as a result of evaluating the best ML model from each of four predefined scenarios with the testing dataset. Figure 9 shows the AUC curve for each scenario.

Table 10 shows the results of the experiments with the four scenarios in terms of their performance metrics. It can be seen that the best model according to the performance metrics is the LR model built with the balanced dataset.

3.3. Explainability of models

In order to have a general idea about the explainability of the results obtained by the models used in this research, the LR model built with balanced data (scenario 4) was used based on its good prediction results and its interpretability through the calculation of the odds ratio (importance) [31]. Table 11 shows the variables (features) and their level of importance in explaining the probability of patients with AVM being cured 36 months after SRS, among which 18 have a negative influence and 14 have a positive influence. The five most important variables that positively influence being cured are: (1) the location of the AVM (side_avm), (2) the occupation of the patient (occupation), (3) the presence of bleeding in the AVM (hemorrhage), (4) previous cranial surgery (prev_cran_surgery), and (5) the type of venous drainage (type_venous_drainage). It is important to highlight that the patient's occupation is an antecedent of the disease, but it is not clinically relevant; however, it is an interesting finding that should be evaluated in greater detail in another study.

4. Discussion

The use of ML techniques in the medical field has increased in recent years, and their application as a support system for medical decision-making when diagnosing and prognosing diseases is increasingly useful [32]. Recent studies suggest that these techniques can be applied for the diagnosis of diseases related to the brain and the central nervous system, such as Alzheimer’s disease (AD) [33], Parkinson’s disease (PD) [34,35], schizophrenia [36], and tumors [37,38]. Inspired by the use of these techniques in medicine, and specifically for the prognosis of patients with AVM [19,20,39], this study proposes a method that makes it possible to predict whether or not a patient with AVM who undergoes SRS will be cured at 36 months after the intervention. We demonstrate that using ML techniques for the prognosis of patients with AVMs would be possible. Our approach involved evaluating four scenarios using two ML models and two datasets (imbalanced and balanced data). After following a traditional process to construct the ML models, in which oversampling, grid search, and cross-validation techniques were also applied, it was found that the best model to predict whether patients with AVM would be cured was the LR model trained with balanced data (accuracy 92.16%, AUC 97.62%).

From the results obtained in this study, it is evident that the LR model is superior to the DT model even when trained with imbalanced data, as shown in Table 10. It is pertinent to highlight that the data preprocessing (selection of variables and balancing) performed in this study led to significantly higher results for the two models (DT and LR) than when the data were not preprocessed, so we can argue that data preprocessing should be included in any approach that uses a ML model. In addition, the results obtained in this study (accuracy 92.16%) were found to be superior to the results obtained in studies using similar procedures, such as those by Oermann et al. [20] and Meng et al. [39], who obtained accuracy of 74% and 83%, respectively.

The importance of the results of this study goes beyond the possibility of using this method for the medical prognosis of patients with AVM; they also allow us to confirm that it is possible to use an ML model, understood as a generalizable framework, in medicine by using historical data to predict the future. In this case, we used historical information over a 14-year time horizon, from which sociodemographic and medical data were collected to build an ML system that achieved very good prediction results and could be used as a tool by medical professionals for decision-making when dealing with new AVM cases.

The proposed approach for the prognosis and explainability of whether patients with AVM will be cured has no limitations; however, the results of these models are limited to the dataset used in this study, so its application in medical practice requires more experiments with larger amounts of data. The possibility of including additional medical variables should also be evaluated.

Finally, it is important to highlight that two ML models considered to be transparent, or “white box”, models [40], whose results are easy to interpret, were used in this study, and the odds ratios for the LR model were used to identify the variables that positively influence whether patients with AVM will be cured. However, it would be important to contrast the interpretability with more sophisticated explainability techniques such as local interpretable model-agnostic explanations (LIME), Shapley additive explanations (SHAP) and others, which are focused on identifying the most important predictors for any type of ML model, including those considered “black box” models.