Development and Validation of a Machine Learning Model for Predicting Endoscopic Intervention in Upper Gastrointestinal Bleeding

1. Introduction

Upper gastrointestinal bleeding (UGIB) is defined as bleeding in the gastrointestinal tract proximal to the ligament of Treitz. It is a commonly encountered and potentially life-threatening condition that often requires frequent hospitalizations. The incidence varies worldwide, ranging from 48 to 172 cases per 100,000 individuals ​[1]​, with a corresponding mortality rate of 2-10% ​[2,3]​. Non-variceal bleeding is considered the most common cause of UGIB with peptic ulcers accounting for approximately half of the cases ​[3,4,5]​.

The international Consensus Group and the European Society of Gastrointestinal Endoscopy recommend that patients with UGIB should undergo endoscopy within 12-24 hours of presentation ​[6,7,8]​. However, patients with non-variceal UGIB who are at low risk of re-bleeding and mortality can be managed as outpatients with non-urgent endoscopy. These patients are identified using pre-endoscopy risk scores. Several scoring systems have been developed to evaluate risk stratification of UGIB. The most commonly used are Glasgow Blatchford score (GBS) and pre-endoscopic Rockall score. In 2000, Blatchford et al. ​[9] proposed GBS by evaluating the data of 1748 patients with UGIB, and they then prospectively validated this scoring system by applying it on 197 patients. The score is based on hemoglobin, blood urea nitrogen, systolic blood pressure, pulse, the presence of melena, syncope, liver disease, and/or heart failure. The purpose of this scoring system was to predict the need for treatment including blood transfusion, endoscopic or surgical intervention, with the validity of it in identifying low-risk patients who do not need an intervention being well established in the literature ​[10]​. Pre-endoscopic Rockall score was developed by Rockall et al. ​[11] in 1996 with the initial model being based on the data of 3981 patients with UGIB​. The score takes into account age, comorbidities, and shock status, which is based on pulse and systolic blood pressure. Furthermore, the Rockall score was then externally validated in two populations of 1584 and 2531 patients with UGIB for identifying patients at low risk of re-bleeding and mortality [11,12]​.

The use of these scoring systems in identifying low-risk patients is well established. However, there is still a need to efficiently identify patients who need an urgent endoscopic intervention. Therefore, two prospective studies compared these scores and found that GBS was better in identifying patients who need clinical management, including endoscopic intervention, than pre-endoscopic Rockall score ​[10,13]​. However, the two studies showed that GBS was not reliable in identifying patients who need endoscopic intervention alone. Thus, the accepted use of these tools remains to identify very low-risk patients who may be safely discharged from emergency departments to be followed up as outpatients with a non-urgent endoscopy, not to predict the need for an endoscopic intervention. The purpose of this study was to develop a novel machine learning model to predict the need for endoscopic intervention in the management of acute UGIB.

2. Methods

2.1. Study Design and Settings:

This is a retrospective, multicenter study in Jordan that included all patients diagnosed with UGIB who underwent endoscopy from January 2013 to October 2023. Data was collected from three major hospitals: King Abdullah University Hospital (KAUH), Jordan University Hospital (JUH), and Prince Hamza Hospital (PHH). Each of these hospitals serves a distinct demographic area, thereby ensuring a thorough evaluation of the condition across different settings within the country. King Abdullah University Hospital (KAUH) is in the northern region and is affiliated with the Jordan University of Science and Technology (JUST). As a leading tertiary care and teaching hospital, KAUH serves not only the immediate urban population but also the surrounding rural areas, providing a wide patient base in terms of socioeconomic status and healthcare needs. Jordan University Hospital (JUH), situated in the capital city of Amman, serves a large urban population, and is considered a major referral hospital for complex cases in the region. This contributes to the study with a population that potentially has more complex or advanced presentations of UGIB. Prince Hamza Hospital (PHH), also in Amman, serves a densely populated area of the city and is known for its accessibility to lower-income populations. As a government-funded public hospital, it often deals with a high volume of patients and provides services to a wide array of the population, including those who might not have ready access to private healthcare facilities.

The ethical approval was obtained from the institutional review board (IRB) of all included institutions, Jordan University Hospital, King Abdullah University Hospital, and Prince Hamza Hospital. In each participating center, the IRB waived the need for direct informed consent from the patients as it included only variables that are routinely recorded from all patients with acute upper gastrointestinal bleeding and the study did not include any interventions or procedures.

2.2. Eligibility Criteria and Data Extraction

Patients eligible for inclusion in this retrospective multicenter study were those diagnosed with upper gastrointestinal (GI) bleeding who underwent endoscopy upon admission to one of three major hospitals in Jordan: King Abdullah University Hospital, Jordan University Hospital, and Prince Hamza Hospital. The study was limited to individuals aged 18 years and older who had provided informed consent for the use of their medical data. Exclusion criteria were designed to omit patients with lower GI bleeding, incomplete medical records, recent GI surgery within the last 30 days, primary coagulopathy disorders, refusal to consent for research use of their data, endoscopic interventions conducted outside the participating hospitals, alternative diagnoses accounting for the GI symptoms, pregnant women, and any previous enrollees in the study to prevent duplication of data. The extracted variables included patient demographics: age and sex, social history: smoking and alcohol use, clinical presentation and symptoms: abdominal pain, heart failure presentation, dizziness, melena, hepatic disease presentation, syncope, dyspnea, hematemesis, hematochezia and coffee ground vomit, comorbid conditions: hypertension, diabetes mellitus, ischemic heart disease, cerebrovascular disease, arrhythmias, congestive heart failure, valvular heart disease, asthma, chronic obstructive pulmonary disease, deep vein thrombosis, pulmonary embolism, cirrhosis, chronic kidney disease, malignancy and peptic ulcer disease, medication history: NSAIDs, aspirin, warfarin, antiplatelets, direct oral anticoagulants and heparin, details of the endoscopic intervention: variceal band ligation, adrenaline injection, coaptive coagulation and hemoclip application, pre-endoscopy management strategies: administration of IV fluids, proton pump inhibitors, erythromycin, vitamin K, plasma clotting factors, tranexamic acid, octreotide and blood products (whole blood, fresh frozen plasma, packed red blood cells, cryoprecipitate and platelets), endoscopic findings: active bleeding, esophageal ulcer, gastric ulcer, duodenal ulcer, varices, Mallory-Weiss tear, esophagitis, gastritis, esophageal tumor, gastric tumor, duodenal tumor, polyps, adherent clots, gastric erosions, esophageal erosions and duodenal erosions, laboratory data and vitals: systolic blood pressure (mmHg), diastolic blood pressure (mmHg), mean arterial pressure (mmHg), pulse rate, hemoglobin (g/dL), platelet count (in thousands), creatinine (mmol/L), C-reactive protein (mg/dL), INR, albumin, WBC count (in thousands) and blood urea nitrogen. Post-endoscopic outcomes included length of hospital stay, re-admission rates, the need for surgery and mortality. Risk stratification scores such as Glasgow-Blatchford Score (GBS) and Rockall score were calculated based on the clinical and laboratory data to assess the severity of the bleeding episode. The time to endoscopy from admission was recorded to examine its impact on patient outcomes.

2.3. Statistical Analysis

All data analyses were performed using Jamovi 2.3.28, an open-source statistical software built on R. Descriptive measures included means ± standard deviations for continuous data if the normality assumption was not violated, according to the Shapiro–Wilk test, and median with first and third quartiles (Q1–Q3) if the assumption was violated. Categorical data were presented by frequencies and percentages (%). Continuous data were compared using the student t test in normally distributed variables and the Mann–Whitney U-test if not normally distributed. Categorical data were compared using the χ2 test or the Fisher's exact test if 1 cell had an expected count of less than 5. Statistical significance was considered at a 2-sided P value of ≤.05.

2.4. Machine Learning Analysis

For the current work, three different ML algorithms have been used: the Random Forest Classifier (RFC), the Gradient Boosting Classifier (GBC), and the AdaBoost Classifier (ABC). These algorithms have been considered excellent for multi-dimensional, complex data. To support the reliability and further keep the credibility of the models used, standard scaling uniformly over all features was done using the module StandardScaler from scikit-learn. This processing entailed converting the feature matrices onto a common scale, hence normalizing the data, and ruling out any possibility of a bias that different scales among various features may have against each other. After cleaning of this dataset, binary-classification-output machine learning models were trained to predict the target variable "Underwent endoscopic intervention" using 81 features as variables. Subsequent machine learning included a model fit by dropping all post-endoscopy variables to be able to identify predictive ability in our model to identify the need for endoscopic intervention. Feature selection was done by Recursive Feature Elimination (RFE) considering all features that contribute to the model and eliminating them progressively from the least contributing. RFE was implemented using the estimator from an RFC model. The method was configured to select only 10 significant features. This is applied on the dataset, and it provides the variables selected for the final model development. The Synthetic Minority Oversampling Technique (SMOTE) was used to handle the imbalance in the dataset. This is achieved using the SMOTE approach in the imbalanced-learn library to synthetically sample from the minority class. The SMOTE was set to automatically change the class distribution of the target variable, and a random state was set for reproducibility. The scaled training data then was subjected to the following process to represent a balanced representation: The next step was applying the preprocessing operation on the patients' dataset in KAUH (n = 989) that leads to a random partitioning into two disjoint parts: 80% of data as the training set (n = 791) and 20% as the testing set (n = 198). Our models were then externally validated using another dataset (n=383) for patients in both JUH and PHH. The permutation importance method is used to calculate the importance of the contribution of the feature in predicting the variable "Underwent endoscopic intervention." A higher reduction in mean accuracy represents higher importance in the predictions of the model. We plotted receiver operating characteristic (ROC) curves and summarized them with the area under the curve (AUC) scores in each model. Using the roc_curve function of the module sklearn.metrics, the False Positive Rate (FPR) and the True Positive Rate (TPR) are computed for all models. It extracts the predicted probability of the positive class using the predict_proba method of the fitted models. The AUC scores were determined using the roc_auc_score function. A custom plotting function was defined, plot_roc_curve, in order to plot the ROC curves of multiple models. The model was also evaluated with a 10-fold cross-validation, accuracy, sensitivity, and specificity, classification report, classification report for precision, recall, and F1-score. All ML implementations were processed using the scikit-learn 0.18 package in Python.

3. Results

3.1. Demographics and Clinical Characteristics

A total of 1372 patients diagnosed with upper GI bleeding were included in this study, of which 242 had endoscopic intervention. These interventions were distributed as follows: 112 patients received variceal band ligation, 87 were treated with adrenaline injection, 54 underwent coaptive coagulation, and 19 had hemoclip application. Majority of the patients were at KAUH, which accounted for 68.2% of the study population. The mean age was 56.5 with a standard deviation of 19.3, and there was a male predominance with males accounting for 65.1% of patients. Melena was the most prevalent symptom, present in 74.5% of patients, followed by abdominal pain and dizziness, found in 31.7% and 25.5% respectively. A more detailed overview of demographics and clinical characteristics in relation to endoscopic interventions can be found in Table 1. Regarding comorbidities, hypertension was the most common, followed by diabetes mellitus, affecting 48.5% and 39.1% of patients respectively. Aspirin was the most common drug in patients' drug history, present in 29.9% of patients with upper gastrointestinal bleed. A more comprehensive overview can be seen in Table 2, showing the association between comorbidities and drug history with endoscopic interventions. IV fluids were the most common management prior to endoscopy, administered to 45.3% of patients, followed by proton pump inhibitors given to 40.3% of patients. Whole blood and packed red blood cell transfusions were the most common transfusions, administered to 39.9% and 38.3% of patients respectively. The mean GBS score was 10.9 with a standard deviation of 2.6, while the Rockall score had a mean of 4.2 with a standard deviation of 1.5. The most common finding during endoscopy was a duodenal ulcer at 22.9%, followed by varices at 17.4% of patients. The mean time to endoscopy was 39.0 hours with a standard deviation of 92.5. A more detailed overview of pre-endoscopic management and endoscopic findings is provided in Table 3. There was a mortality rate of 6.19% among all patients with upper gastrointestinal bleed and 13.4% of patients experienced re-admissions within 30 days, and 1.8% of patients required surgery. The mean length of hospital stay for all patients was 5.9 days with a standard deviation of 12.0. For a more detailed overview of laboratory values, vitals and post-endoscopic results compared to endoscopic interventions, refer to Table 4.

3.2. Characteristics of all Patients Undergoing Endoscopic Intervention

The highest number of cases was extracted from KAUH; however, the patients were intervened endoscopically in the three hospitals. The location of the hospital became a statistically significant variable with a p-value of 0.004. The symptoms that portrayed statistical significance among them were abdominal pain, dizziness, melena, hepatic disease, hematemesis, all with p-values of <0.05. From the social history, only alcohol had statistical significance, with a p-value of 0.002. Among the comorbidities of the patient, only diabetes and cirrhosis showed statistical significance, both with p-values of <0.05. On the history of drugs, all drugs were inconclusive to find any associated risk among the patients subjected to endoscopic intervention. All pre-endoscopy management modalities, namely, erythromycin, IV fluids, proton pump inhibitors, and octreotide, were highly associated with endoscopic intervention, all with p-values of <0.05. Transfusions, including whole blood, FFP, and PRBC, were also found to be associated with endoscopic intervention. Both traditional pre-endoscopy scores, GBS, and Rockall showed statistical significance with a p-value of <0.001. Various findings, which were statistically significant during endoscopy, included active bleeding, varices, esophagitis, gastritis, and adherent clots, with p-values of <0.001. Other endoscopic findings such as gastric erosions and duodenal erosions also showed significance, both with p-values of <0.05. In other laboratory values, only the platelet count, albumin, and BUN were statistically significant, having a p-value of <0.001. Hemoglobin also shows significance, having a p-value of 0.023. Post-endoscopic results indicated statistical significance for mortality and re-admission, with p-values <0.001 and 0.001 respectively, suggesting a higher mortality and re-admission rate among patients undergoing endoscopic intervention.

3.3. Variceal Band Ligation

Patients undergoing variceal band ligation also exhibited statistical significance across hospitals, particularly with KAUH showing a larger number of procedures performed, with a p-value of 0.028. Age also demonstrated statistical significance with a p-value of 0.008, indicating a younger patient population compared to other endoscopic interventions. Symptoms such as melena, hepatic disease, and hematemesis showed statistical significance with p-values <0.001. Regarding social history, patients undergoing variceal band ligation had a lower percentage of smokers compared to other endoscopic interventions, with statistical significance at a p-value of <0.001. Various comorbidities were statistically significant, including diabetes mellitus, ischemic heart disease, cerebrovascular disease, arrhythmias, congestive heart failure, cirrhosis, and chronic kidney disease, all with p-values <0.05. Regarding drug history, NSAIDs, Aspirin, Warfarin, and antiplatelets showed statistical significance, all with p-values <0.05. Pre-endoscopy management strategies such as erythromycin, vitamin K, and octreotide were statistically significant, all with p-values <0.05. Among transfusions, only fresh frozen plasma showed significance with patients undergoing variceal band ligation with a p-value of <0.001. Among pre-endoscopic scores, only the Rockall score showed statistical significance with a p-value of <0.001. Various endoscopic findings including active bleeding, gastric ulcer, duodenal ulcer, varices, esophagitis, and gastritis were statistically significant, all with a p-value of <0.001. Gastric erosions and duodenal erosions also showed statistical significance, both with p-values <0.05. Regarding lab values and vitals, systolic blood pressure, platelet count, albumin, and WBC count all showed statistical significance, all with p-values <0.05. All post-endoscopic results, including mortality, re-admission, and the need for surgery, showed statistical significance, all with p-values <0.05.

3.4. Adrenaline Injection

In patients who received adrenaline injection, there was a significant male predominance with a p-value of <0.001. Symptoms such as dizziness, melena, hematemesis, and coffee ground vomit showed statistical significance, all with p-values <0.05. Regarding comorbidities, hypertension, diabetes mellitus, deep vein thrombosis, and absence of comorbidities were statistically significant, all with p-values <0.05. Drug history did not show any statistically significant associations. Among management prior to endoscopy, only IV fluids were statistically significant with a p-value of 0.018. Transfusions including blood and packed red blood cell transfusions were statistically significant, both with p-values <0.05. The Glasgow-Blatchford score was statistically significant with a p-value of <0.001. Findings during endoscopy including active bleeding, duodenal ulcer, gastritis, adherent clot, and gastric erosions were statistically significant, all with p-values <0.05. Lab values including diastolic blood pressure, mean arterial pressure, albumin, white blood cell count, and BUN were statistically significant, all with p-values <0.05. Among post-endoscopic results, only mortality was statistically significant with a p-value of 0.01.

3.5. Coaptive Coagulation

Regarding patients undergoing coaptive coagulation, hospital association was statistically significant with a p-value of <0.001. These patients tended to be older, which was statistically significant with a p-value of 0.020. Among symptoms, only melena was statistically significant with a p-value of 0.031. Among comorbidities, only hypertension was statistically significant with a p-value of 0.001. No drugs in patients undergoing coaptive coagulation were statistically significant. Regarding management prior to endoscopy, only proton pump inhibitors were statistically significant with a p-value of 0.041. None of the transfusions were statistically significant. Findings during endoscopy, including active bleeding, varices, gastritis, and adherent clot, were statistically significant, all with p-values <0.05. Only BUN showed statistical significance among lab values with a p-value of 0.007. None of the post-endoscopic results were statistically significant.

3.6. Hemoclip Application

Among symptoms, only heart failure was statistically significant with a p-value of 0.017. Regarding social history, smoking history showed statistical significance with a p-value of <0.001. Comorbidities including hypertension, diabetes mellitus, cerebrovascular disease, and malignancy showed statistical significance, all with p-values <0.05. Among drug history, only aspirin showed statistical significance with a p-value of 0.007. Management prior to endoscopy, IV fluids, and plasma clotting factor were statistically significant, both with p-values <0.05. Transfusions including blood, fresh frozen plasma, and packed red blood cells were statistically significant, all with p-values <0.05. Pre-endoscopic scores showed no statistical significance. Findings during endoscopy, including active bleeding, gastric ulcer, duodenal ulcer, and adherent clot, showed statistical significance, all with p-values <0.05. Among lab values, only diastolic blood pressure and mean arterial pressure showed statistical significance with a p-value of <0.001. None of the post-endoscopic results were statistically significant.

3.7. Machine Learning Analysis

Across all the machine learning models evaluated, the Gradient Boosting Classifier (GBC) showed the highest AUC score at 0.861, marking it as the most predictive model. Other models also displayed high AUC scores, with AdaBoost classifier close at 0.839 and Random Forest Classifier (RFC) at 0.803. The external validation (EV) versions for RFC, GBC, and AdaBoost recorded AUC scores of 0.810, 0.819, and 0.775, respectively, outperforming traditional pre-endoscopic scores like GBS and Rockall, which had AUC scores of 0.609 and 0.568, respectively. This indicates a superior predictive capability of our models for the need for endoscopic interventions, as shown in Figure 1. Performance metrics varied, with model accuracies ranging from 82.32% to 86.95%, recall between 59.21% and 66.67%, and F1 scores from 56.79 to 65.28. GBC EV achieved the highest precision at 69.12%, RFC EV the highest specificity at 0.912, and GBC the highest sensitivity at 0.667. For detailed insights, refer to Table 5. Permutation feature importance analysis identified active bleeding, varices, and gastritis as key predictors for the need for endoscopic intervention, as detailed in Figure 2.

4. Discussion

Assessment and prediction of which type of patient would benefit from an endoscopic intervention is the most important step in the management of a life-threatening disease like UGIB, thus the main goal of the following study is to use machine learning models to predict the need for interventions during endoscopy. Data were collected in three main hospitals in Jordan. We included 1372 patients with baseline characteristics, clinical, and endoscopic data out of which 242 were the ones that needed endoscopic intervention. Importantly, the ML models, in particular the Gradient Boosting Classifier (GBC), have generally shown great improvement in prediction performance compared to existing traditional pre-endoscopic scores, such as GBS and Rockall, with better AUC scores. In the study by Veisman et al. [14], the goal was to predict the need for endoscopic interventions. They tried to improve the currently existing scores and schemes by using them within a standard ML framework. Their work specifically aims at integrating established pre-endoscopy scores with machine learning models in order to fine-tune the prediction accuracy, while our approach builds on the use of ML algorithms applied directly to a larger set of clinical parameters, thus avoiding possible drawbacks generated by existing traditional scoring systems. In the demographic distribution, the present study observed a male predominance (65.1%) and a mean age of 56.5 years. The results are in coherence with the earlier findings suggesting that male sex and old age are associated with the occurrence of upper GI bleeding [15,16]. The most reported symptom was melena, which is in agreement with the literature reporting this as the most common feature in the presentation of UGIB cases [17,18,19]. The present study further highlights hypertension and diabetes mellitus as common comorbid conditions, which aligns with previously published literature, affecting the risk and outcomes of UGIB [20,21]. The significant variables identified for patients undergoing endoscopic interventions, such as hospital location and certain symptoms (abdominal pain, dizziness, melena), provide perspectives into factors that may guide clinical decision-making [22,23]. The association of hospital location with intervention rates would certainly be reflective of differences in hospital protocols or patient case mix and so would be worth further study. The differing profiles for the subgroups of endoscopic intervention, like the variceal band ligation and adrenaline injection, might be explained by the differing pathologies that require intervention. For example, the younger age distribution within variceal band ligation may illustrate that the etiology for variceal bleeding is commonly associated with liver disease in a relatively younger cohort and goes along with previous literature showing a similar mean age for variceal band ligation [24,25]. The permutation feature importance further identified 'active bleeding,' 'varices,' and 'gastritis' as the important features supporting the view that ML may infer subtle relations not fully disclosed by traditional statistical models for the complex interactions of clinical features in UGIB. These findings could help guide the development of clinical algorithms for endoscopic triage of patients and subsequently improve efficiency in the utilization of health resources and patient outcomes. Some studies have demonstrated good predictive models, like the Random Forest, which accurately predicted urgent intervention among patients of acute gastrointestinal bleeding, with model accuracies of about 80% and above, and an area under the ROC curve greater than 0.85 [26]. However, a few limitations that should be taken into account in the interpretation of our study are as follows: first, it was a retrospective study and might hence be subjected to inherent biases, such as selection and information bias. Furthermore, the fact that we had to rely on median imputation for the missing values, while practical, might add another layer of bias. However, differences in the protocols of hospital care and diversity within the populations of patients being attended to may limit the power of generalization of our models to be applied in other settings. We should therefore focus future work on the prospective validation of our machine learning models in varied clinical environments and to integrate them into real-time clinical decision support systems. Further exploration of the features presented here, through the lens of deep learning methods, could reveal more nuanced patterns in the data, which shallower, simpler machine learning models might miss.

5. Conclusions

In conclusion, our study demonstrates the potential of machine learning to revolutionize the triage process for upper GI bleeding, paving the way for personalized medicine that tailors interventions to individual risk profiles. This could ultimately translate to more efficient use of endoscopic resources, improved patient outcomes, and reduced healthcare costs.