Risk Factors for ICU Mortality in 8902 Critically Ill Patients Infected with the Pandemic Virus According to the Machine Learning Analysis Used

1. Introduction

Pandemics have historically been one of the greatest threats to public health, causing high mortality and having a significant impact on healthcare systems and society in general. Two recent pandemics have been particularly devastating: the 2009 influenza A (H1N1) virus pandemic and the 2019 SARS-CoV-2 pandemic. Both caused millions of deaths worldwide[1,2,3,4] and challenged the response capacity of healthcare systems, the pharmaceutical industry and governments. They also had ethical, economic and social consequences that are still being felt today [5,6].

Despite the knowledge and advances in biomedicine, there are still limitations in the ability to predict the outcome of critically ill patients with pandemic viral infections. Early identification of patients at increased risk of mortality is essential to optimize intensive care unit (ICU) resources and improve clinical outcomes. Several authors [7,8,9,10,11,12,13,14,15,16,17] have identified a large number of risk factors associated with mortality in critically ill patients with influenza A (H1N1) and SARS-CoV-2 that differ or overlap depending on the population studied, the country or the method of analysis used. Traditionally, statistical models such as logistic regression have been used to quantify the association between confounding variables and the dependent variable in a linear fashion. However, this approach has limitations in detecting non-linear relationships (perhaps the most common in medicine) and the complex interaction between multiple variables, which limits its predictive power in clinical scenarios with high-dimensional data [18,19]

In this context, new machine learning techniques have emerged as promising tools for predicting complex clinical outcomes. Among these, Random Forest, one of the most widely used techniques today, has shown significant advantages in identifying complex patterns in the data without the need for parametric assumptions. This algorithm, based on the combination of multiple decision trees, offers greater predictive accuracy and robustness to collinearity and heterogeneity of clinical data than linear models [20,21]

Our hypothesis is that different risk factors can be identified by applying different models of analysis. To test our hypothesis, the aim of our study is to identify risk factors associated with mortality in critically ill patients with Influenza A (H1N1) and SARS-CoV-2 infection by comparing the predictive ability of traditional logistic regression models with advanced machine learning techniques, specifically Random Forest. Our study aims to alert clinicians to the limitations of classical models and the need for more complex or multiple analyses to identify true risk factors and thus optimize decision-making in the management of ICU patients.

2. Materials and Methods

2.1. Design

This is a secondary analysis of 2 prospective, multicenter, cohort studies. The first database is the GETGAG registry, a voluntary registry created by the Spanish Society of Intensive Care Medicine (SEMICYUC) in 2009 during the A(H1N1)pdm09 influenza pandemic in which 184 Spanish ICUs participated between June 2009 and June 2019 (8,22,23). The study was approved by the Ethics Committee of the of the Joan XXIII University Hospital (CEI no. 11809) and by each of the participating centers. Informed consent informed consent was not obtained from the patients due to the observational nature of the study and data anonymity was guaranteed. The second database is the COVID-19 registry, a voluntary registry created by SEMICYUC in 2020 during the SARS CoV-2 pandemic in which 74 Spanish ICUs participated between 1 July 2020 and 31 December 2021 [10,14,15,24]. The study was retrospectively registered on Clinical-Trials.gov (NCT 04948242) on 30 June 2021. The need for informed consent was waived by the Institution's Internal Review Committee (Comitè Ètic d'Investigació amb Medicaments [CEIm] del Institut d'Investigació Sanitària Pere Virgili [IISPV]IRB# CEIM/066/2020]. Local researchers-maintained contact with a study team member, and participating hospitals obtained local ethics committee approval. The study was carried out according to the principles of the Declaration of Helsinki and the Clinical Trials Directive 2001/20/EC of the European Parliament relating to the Good Clinical Practice guidelines [25].

We reported results in accordance with the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) guidelines [26].

2.2. Study Population

A total of 8902 consecutive patients requiring ICU admission with a diagnosis of respiratory infection by influenza A (H1N1)pdm09, seasonal A, B (n= 3702) or SARS Cov-2 (n=5200) viruses were included during the two periods described. The presence of the virus was determined by performing real-time polymerase chain reaction (rt-PCR) in each hospital, according to Infectious Diseases Society of America (IDSA) recommendations for influenza [27] and World Health Organization (WHO) recommendations for SARS Cov-2 [28]. The follow-up of patients was scheduled until confirmed ICU discharge or death whichever occurred first.

2.3. Definitions

Co-infection was suspected if a patient presented with signs and symptoms of lower respiratory tract infection, with radiographic evidence of a pulmonary infiltrate with no other known cause [10,23,29]. Coinfection had to be confirmed by laboratory testing using Centers for Disease Control and Prevention (CDC) criteria [23,29]. Only respiratory infection microbiologically confirmed with a respiratory specimen or serology obtained within 2 days of ICU admission was considered community-acquired coinfection. The diagnosis of coinfection was considered “definitive” if respiratory pathogens were isolated from blood or pleural fluid and if serological tests confirmed a fourfold increase of atypical pathogens, including Chlamydia spp., Coxiella burnetti and Moraxella catarrhalis. Only patients with confirmed microbiologic diagnosis were included in the present analysis.

Acute Kidney injury (AKI): The diagnosis of AKI was considered according to the Acute Kidney Injury Network (AKIN) described in the international KDIGO guidelines [30].

Appropriate empiric antibiotic treatment (AEAT): Was defined as the administration of an antibiotic on admission to the ICU before the microbiological results are available and adjusted to the susceptibility of the pathogen when the microbiological results are available. AEAT was determined by the attending physician in each center.

Inappropriate empirical antibiotic treatment (IEAT): Was defined as antibiotic treatment administered on admission to the ICU that was not adapted to the susceptibility of the pathogen when microbiological results are available. In addition, the use of antibiotics at ICU admission in patients with no bacterial co-infection was also included in this definition.

GAP-UCI: was defined as the time elapsed between diagnosing pandemic viral infection and admission to ICU.

GAP-Diagnosis: Was defined as the period of time between the onset of clinical symptoms and the microbiological diagnosis of the pandemic viral infection.

2.4. Study Variables

Demographic data, comorbidities and clinical and laboratory findings were collected during the first 24 hours of ICU admission. In addition, the need for invasive mechanical ventilation and the presence of shock on admission to the ICU were recorded. Disease severity was determined using the Acute Physiology and Chronic Health Evaluation II (APACHE II) score [31] and the level of organ dysfunction using the SOFA score [32]. The variables controlled in the study can be seen in Table 1.

2.5. Analysis Plan and Statistical Analysis

First: We determined the crude ICU mortality in the general population and compared patient characteristics according to outcome. Qualitative variables were expressed as percentages while quantitative variables, as median and interquartile range Q1-Q3. In order to determine clinical differences among groups, the Chi-square and Fisher tests were used for categorical variables, and the student’s t or Mann-Whitney U test, for quantitative variables.

Second: A linear model (Binary logistic regression) was performed to determine which variables were independently associated with all-cause mortality in the ICU. All variables with statistical significance (p<0.05) in the bivariate comparison between groups were included in the GLM (generalized linear model). Only variables at the time of ICU admission were included to develop a model to predict mortality. To optimize the performance of the model, the continuous variables were included in the model in a categorized form. The cut-off point for the continuous variables was determined by taking the median of the surviving patients. Results are presented as odds ratios (OR) with a 95% confidence interval.

For internal validation of the model, the population was randomly divided into a development set (train) with 70% of the patients and a validation set (test) with the remaining 30%. The performance of the model was assessed by determining accuracy, precision, sensitivity, specificity and area under the ROC curve (AUC). The presence of collinearity between the explanatory variables was examined using variance inflation factors (VIF).

In addition, we perform a k-fold Cross Validation (K=10) which consists of taking the original data and creating from it two separate sets: a first training (and test) set, and a second validation set. The training set will then be divided into k subsets and, at the time of training, each k subset will be taken as the model test set, while the rest of the data will be taken as the training set. Once the iterations were completed, the accuracy and error were calculated for each of the models produced, and to obtain the final accuracy and error, the average of the k trained models was calculated.

Third: Due to the significant discrepancy between the two evolution groups, with only 25% of patients in the deceased group, it must be considered that this class imbalance may affect the performance of the model focused on mortality (minority class). To test whether there are differences in the performance of the linear model due to class imbalance, the ROSE (Random Over-Sampling Examples) package was implemented to deal with the imbalance between the groups. This statistical package provides functions to address binary classification issues in the context of unbalanced classes. Balanced samples are generated using a smoothed bootstrap approach, which allows the estimation and evaluation of the accuracy of a binary classifier in the presence of a rare class.

The package also includes functions that implement more traditional class imbalance remedies and various metrics for assessing accuracy. These are estimated using cross-validation, bootstrap or holdout [33,34]. We implemented the "under" option, which specifies subsampling without replacing the majority class until the specified sample size N is reached or the positive examples have a probability p of occurring. When the method is set to "under", the resulting sample is reduced. The ROSE software (version 0.0-4) was used only to process data from the training subset, leaving the test subset untouched. Once the model had been developed on the training set, it was applied to the test set and its performance evaluated. The results are presented as odds ratio (OR) with a 95% confidence interval, accuracy, sensitivity, specificity and AUC-ROC.

Fourth: To test our hypothesis, we developed a non-linear model (random forest classifier model [RFc]). Random forest models are a powerful non-linear tree-based machine learning technique. The developed model was configured to make 500 random trees, with a minimum number of 15 variables per tree. The performance of the RFc model was evaluated using out-of-bag (OOB) error. This method allows the prediction error of random forests, boosted decision trees and other machine learning models to be measured using bootstrap aggregation. We also plotted the importance of the different variables for the model, which is related to the average loss of accuracy and the Gini index for the classification model. The Gini index is a “measure of disorder”, represented as “MeanDecreaseGini”, which means that the higher the measure, the greater the importance in the generated models, since values close to 0 for the Gini index imply more disorder and values close to 1 imply less disorder. The higher this measure, the more variability it will contribute to the dependent variable. For the internal validation of the model, the population was randomly divided into a development (training) set with 70% of the patients and a validation (test) set with the remaining 30%. The performance of the model was evaluated by determining the accuracy.

Statistical analysis was performed with R statistical software (v 4.4.1) R: The R Project for Statistical Computing (r-project.org).

3. Results

3.1. Whole Population

A total of 8902 patients admitted to the ICU were included in the study, of whom 3702 (41.6%) had influenza and 5200 (58.4%) had coronavirus disease 2019 (Covid-19). The diagnosis was confirmed in all patients by means of polymerase chain reaction (PCR). Table 1 presents the general characteristics of the patients according to ICU outcome. The cohort was predominantly male (65.8%), with a mean age of 60 years and an intermediate level of severity, as indicated by a mean APACHE II score of 14 points and a mean SOFA score of 5 points. The most prevalent comorbidities were obesity, diabetes and chronic obstructive pulmonary disease (COPD), with a mean length of stay of 13 days and a crude ICU mortality of 25.8%.

Non-survivors were older, had greater severity of illness, more systemic inflammation, more major comorbidities, greater need for assistance and organ support, and longer ICU stay. The presence of coinfection at ICU admission was also more common in non-survivors, with significant differences in etiology for Pseudomonas aeruginosa, Aspergillus spp and Acinetobacter spp. (Table 1)

3.2. Factors Associated with ICU Crude Mortality According to General Linear Model (GLM)

The associations between crude ICU mortality (dependent variable) and the independent variables were examined using multiple logistic regression. The variables included in the model were male, age cut-off, APACHEII cut-off, SOFA cut-off, ICU GAP cut-off, GAP diagnosis cut-off, shock, asthma, COPD, chronic heart disease, chronic kidney disease, hematological disease, pregnancy, obesity, diabetes, HIV, immunosuppression, steroids, antibiotic treatment at ICU admission, mechanical ventilation at ICU admission, myocardial dysfunction, Acute kidney injury (AKI), > 2 areas of infiltration on chest X-ray, lactate dehydrogenase cut-off, creatine phosphokinase cut-off, WBC cut-off, CRP cut-off, PCT cut-off, lactate cut-off, D-dimer cut-off, Klebsiella spp, Acinetobacter spp, S. pneumoniae, methicillin-sensitive Staphylococcus aureus (MSSA), E. coli, methicillin-resistant Staphylococcus aureus (MRSA), Pseudomonas aeruginosa, Aspergillus spp and antiviral vaccine. Of the 39 variables included in the model, 17 showed an independent association with all-cause mortality in the ICU. The variables that reached significance are shown in Figure 1 and Table 2.

3.3. Linear Model (GLM) Validation

When the developed model was applied to the test subset, the model performed acceptably with an accuracy of 76%, a sensitivity of 61%, a specificity of 79% (Supplementary Table S1) and an AUC of 0.76 (95% CI 0.74-0.78) (supplementary Figure S1). No collinearity was observed between the variables included in the model (Supplementary Table S2).

Cross-validation (K = 10) did not improve the accuracy of the model (0.76%), but increased the sensitivity to 94% with decreased the specificity to 26% (Supplementary Table S3).

3.4. Development of the GLM Model with Correction of Class Imbalance

Applying the ROSE package to the training set reduced the population from 6232 patients to 3152 patients. Of these, 1606 died, giving an estimated mortality rate of 50.9%, twice the real rate (25%). The development of the linear GLM model with the balanced population showed no better performance with an AUC-ROC of 76% (74%-78%) and an accuracy of 68%. The full development of the balanced population model can be found in the Supplementary Figure S2, Figure S3 and Table S4. Due to the lack of optimization and the consequent reduction in the number of patients, we have maintained the original GLM model despite the imbalance, as it does not appear to affect the linear model used.

3.5. Factors Associated with ICU Mortality According to No-Linear Model (Random Forest)

A random forest classifier (RFc) model was developed to study the contributions of confounding variables to the dependent variable (ICU mortality) in a non-linear way. The independent variables included in both RFc models were the same as in the GLM to compare results. The RFc model had an OOB estimate of error rate of 25.3%.

Nineteen variables had an influence of more than 10% on the reduction in precision of the model (Table 2 and Figure 2). Obesity, AKI and the presence of Acinetobacter spp. were not important variables in the non-linear precision model, but they were in the GLM. On the other hand, COPD, lactate, procalcitonin, D-dimer and CPK were important variables for accuracy in the non-linear model that were not significant in the linear model.

Twenty-three variables were associated with a >50% reduction in GINI in the non-linear analysis. AKI, Acinetobacter spp. and Aspergillus spp., significant variables in the linear model (GLM), did not appear to be important for the GINI. On the contrary, GAP-diagnosis, GAP-ICU, male sex and WBC count were important for reducing the GINI in the non-linear RF model. (Table 2 and Figure 2)

3.6. Non-Linear Model (RFc) Validation

When the developed model was applied to the test subset, the model performed acceptably with an accuracy of 75.6%. This performance was similar to that observed in the linear model (GLM) despite using different covariates.

3.7. Patient Classification by Model

Of the 2670 patients included in the set test, GLM adequately classified 2035 patients (76.2%), while RF did so in 2018 patients (75.6%) (Figure 3 and supplementary Figure S4 A and B). Both models correctly classified 1872 patients (70.1%) and misclassified 489 patients (18.3%). Figure 4 shows the distribution of the probability generated by each model (Class) with respect to the observed (Real).

4. Discussion

To the best of our knowledge, this is the first study using machine learning techniques on a large number of critically ill patients affected by pandemic viruses, and our main finding was that using either a linear (generalized linear model, GLM) or non-linear (Random Forest, RF) technique to generate mortality prediction models was associated with similar performance, with an accuracy close to 80%.

However, the risk factors differed according to the type of analysis used. Although factors such as age, severity, degree of organ dysfunction, or need for mechanical ventilation were important in both models (major determinants), other laboratory variables such as procalcitonin, D-dimer or lactate were identified only in the RF model (minor determinants). Conversely, acute kidney injury (AKI) and the presence of Acinetobacter spp. were significant only in the GLM. These findings should alert clinicians to the limitations and implications of studies that rely exclusively on one methodological approach to identify prognostic factors.

The influenza A(H1N1) and COVID-19 pandemics have put enormous pressure on healthcare systems around the world, highlighting the urgent need for reliable and accurate methods to predict the potential severity and survival of patients in order to manage resources appropriately. Although pandemics might appear as past events, each winter hospitals are again overwhelmed by patients with respiratory failure due to viral infections, which generate seasonal surges in ICU admissions and resource demand.

Early identification of high-risk patients with viral infections is essential, as it allows for rapid triage, targeted intensive care, and optimized resource allocation, all of which can ultimately improve patient outcomes. In this context, our study aimed not only to evaluate and compare the performance of traditional statistical and machine learning models for mortality prediction, but also to explore how each method identifies different clinical predictors of outcome.

Several authors have used different types of ML analysis to identify risk factors and develop predictive models with COVID-19 patients, while we did not find any studies with influenza A H1N1. In addition, most of the studies include hospitalized patients, with few critically ill patients.

Huang H et al [35] have reported an AUC of 94.4%, sensitivity of 94.1% and specificity of 90.2% when using ML, but the population considered was only 127 patients and only 33 of these were critical. Meanwhile , Zhu et al [36] in 127 COVID-19 patients (16 severe), Gong et al [37]in 372 hospitalized COVID-19 patients, Aloisio et al [38] in 427 COVID-19 patients and Liu et al [39] in 336 severely ill COVID-19 patients (34 of whom died) showed excellent performance (AUC >90%) using linear logistic regression models. The small number of patients included in these studies limits the strength of the results and their generalizability.

In line with our research, an elegant study by Reina-Reina et al [40] on a population of 1200 COVID-19 patients, assessed the risk of death and ICU admission using different ML techniques (Support Vector Machine (SVM), LR, K-Neighbours, Decision Tree, Gaussian Naive Bayes, MLP, and ensemble methods such as Adaboost and bagging techniques). The authors found that there were no significant differences between the classification accuracies (>88%) of the different ML techniques. However, they chose LR as the algorithm to optimize due to the interpretability of the model, which is crucial in the medical field, although RF achieved a slightly better average result. The model finds that the most important variables are COPD, which increases the probability of death by 575%, age, which increases the probability by 145% every 10 years, and acute respiratory failure, with an increase of 513%. However, the authors do not report the differences between the predictor variables identified by each model and only a small percentage of patients were critical.

Pourhomayoun M et al.[41] applied different ML models (SVM, neural networks (NN), RF, decision tree and LR) to predict severity in a large number of patients with COVID-19 (more than 2,670,000 patients). The original dataset contained 32 data points for each patient, including demographic and physiological data, and the NN algorithm achieved the best performance and accuracy with an AUC of 89.98%, compared with an AUC of 87.93% for RF and an AUC of 87.91% for LR. The authors do not make a statistical comparison between the AUCs to assess significance and only present the predictive factors of NN in the form of a heat map, without making a comparison between the models. In addition, the severity of the patients' disease was not reported.

In an excellent review of ML techniques used for prognosis in COVID-19, Alballa N et al. [42] note that the most commonly used algorithm for COVID-19 diagnostic and prognostic models is LR, followed by XGBoost and finally SVM. The authors point out that most of the studies included in the review used unbalanced datasets. Those in which the majority of records in the training dataset represent the negative class (survivors) and the positive class (non-survivors) is under-represented. Therefore, the reported performance of various ML algorithms applied in the context of COVID-19 may have been affected by bias. A high accuracy score in such cases could be attributed to the model's ability to accurately identify negative samples and erroneously exclude all positive cases. In our study, this bias was recognized, and addressed by applying subsampling to the majority class. However, this action did not improve the performance of the balanced model over the unbalanced one, showing that the reliability of the model is not affected by class imbalance. This may be due to the fact that the mortality in our series of critically ill patients is 25%, whereas in most published studies it is around 10-15% [9,38,39,40,43] due to the absence of critical patients.

As we observed in our study, the linear model (GLM) has a similar performance to that of non-linear models in predicting mortality in patients with COVID-19, which has been confirmed in several studies [15,40,41,42,44]. However, despite the structural flexibility of machine learning models for predicting COVID-19, there are limitations to their practical use, such as the high heterogeneity between patients' clinical profiles and the small sample size, which may reduce the external validity and generalizability of the data. Most studies describe different risk factors and different performances depending on the inclusion of one or another factor. This is consistent with studies [40,44,45] that have reported modest performance of ML models when trained exclusively with baseline clinical data collected at the time of admission to the ICU. Most successful predictive models, such as those of Wang et al. [46] and Karasneh et al. [17] have incorporated dynamic, therapeutic or immunological variables that significantly improve model performance. However, these variables are not available in the first hours of care for critically ill patients, so their applicability to clinical practice is limited.

Our study has strengths that we would like to highlight. The first is the large number of critical patients included (n=8902), with more than 3000 patients affected by influenza A (H1N1)pdm09, which makes it unique in its results. It is a national multicenter study that includes more than 148 ICUs in Spain, so its results can be generalized to our country. In addition, it not only reports the performance of the models developed, but also describes the different risk factors identified and the correct classification of patients by each model. On this basis, our study allows us to classify the risk factors into those that are important in both models (major determinants) and those that are important in only one of the models (minor determinants). Recognition of these risk factors can add value in clinical practice for determining the prognosis in critically ill patients with pandemic virus infection. Finally, our study alerts clinicians to the limitations of using models developed using a single method of analysis.

However, our study has limitations that need to be recognized. First, despite the large number of patients and its multicenter nature, these findings cannot be extrapolated to other (non-critical) populations, other health systems and other continents before local validation. Second, although the potential bias due to class imbalance has been addressed, it is possible that other biases, such as those related to ethnicity or other confounding variables, cannot be ruled out, as these variables are not available in our data. Finally, it is possible that including data on ICU evolution in the models could improve performance. However, our aim was to identify early risk factors for mortality that could be modified by the clinician to improve prognosis.

5. Conclusions

Our study highlights the current value of linear models (GLM) in predicting mortality in the era of machine learning analysis. However, it alerts clinicians to the need for a heterogeneous approach of linear and non-linear (complementary) analysis to discover all the major and minor determinants of mortality with the aim of improving the prognosis of this group of critical patients.