Development and Deployment of a Predictive Model for Ventilator-Associated Pneumonia (VAP) Using Clinical Information System Data and Real-Time Mapping of Infection Variables

1. Introduction

Ventilator-associated pneumonia (VAP) affects up to 20% of critically ill patients and significantly increases pressure for antibiotic prescription, accounting for half of all antibiotic use in the ICU. [1,2] VAP increases the length of hospital stay and healthcare costs, and is also associated with long-term morbidity and mortality [3,4,5]. The diagnosis of VAP continues to present challenges and difficulties for the arsenal of clinical, radiological and microbiological diagnostics currently available.[1,6,7,8]

Clinic Clinical information systems (CIS) emerged with the aim of improving safety and communication in healthcare through the primary use of data generated in patient care. In recent years, their secondary use has opened up new opportunities to optimize healthcare by enabling quality assessment and the development of tools to support clinical decision making. These systems take advantage of large amounts of data collected from patient records and serve as the backbone of clinical decision support (CDS) systems. CDS uses algorithms and data analysis to offer personalized recommendations to doctors about the potential risk of an entity, helping to implement early actions regarding diagnosis and possible treatment.[9,10,11] These tools are designed to integrate seamlessly into clinical workflows, presenting alerts and reminders to clinicians at critical decision points.

Predictive modelling in healthcare uses CIS data to forecast individual patient outcomes or the likelihood of certain conditions. [12,13,14]These models can predict the onset of disease by analyzing patterns within large data sets. This predictive capability is fundamental to preventative medicine, allowing doctors to intervene early and adjust diagnostic protocols and treatments at an early stage.

The integration of Artificial Intelligence (AI) and Machine Learning (ML) technologies has significantly advanced the development of predictive models and decision support tools. By analyzing historical and real-time data, ML techniques improve the accuracy of predictions over time and adapt to new information as it becomes available [15,16].

In this context, we hypothesized that the new ML techniques can adequately predict the risk of VAP in critical patients on mechanical ventilation at an early stage. To test our hypothesis, we developed a VAP prediction model using ML with data obtained automatically and in real time from the CIS.

2. Materials and Methods

2.1. Study Design

Retrospective observational study carried out in a 28-bed general (non-coronary) ICU at a tertiary university hospital between 1 January 2014 and 31 December 2024. The sample size was not calculated due to the large amount of data collected and the retrospective nature of the study.

2.2. Population

All critically ill adults (> 18 years) who required MV for more than 48 hours were included. Patients whose length of stay in the ICU was less than 72 hours, the minimum time necessary to include the variables in the models, were excluded.

2.3. Variables

The variables studied are listed in Table 1. These variables were automatically obtained from the clinical information system (CIS, Centricity Critical Care®, GE, Berlin, Germany) through the development of a custom-built ETL (extract, transform and load) system using SQL (version 16.0) and Python (version 3.11.6) / Jupyter environment (version 4.0.7). The CIS automatically collects data from patient-connected devices every two minutes, including hemodynamic variables, monitor-derived parameters, and variables from life-support devices (e.g., mechanical ventilators). In contrast, hourly urine output is recorded by healthcare professionals and is not automatically measured by the CIS. Additional clinical parameters, laboratory values, and medication information are also integrated into the CIS. Furthermore, healthcare professionals record all information related to the patient during their stay in the ICU as VAP.

Day 0 (zero) was defined for patients with VAP as the time when the clinical diagnosis was made by the treating physician. For patients who did not develop VAP, day zero was considered to be the last day of MV. Data extraction was performed by creating 24-hour time windows starting from the date of the event (day zero) and going back to the cut-off time 24, 48 and 72 hours before day zero.

2.3.1. Missing Data Imputation

The availability of continuous data in the CIS with 24-hour time windows for the entire patient stay made it much easier to deal with the few missing data. Tracheostomy, days on IMV, antibiotics (1/0), number of aspirations, total diuresis, total norepinephrine dose and total dobutamine dose were imputed as 0. The remaining variables were imputed using a progressive imputation method (fillforward).

2.3.2. Feature Selection

Categorical variables that were not statistically significant in bivariate comparison, as well as those with more than 30% missing values (SOFA score and procalcitonin), were excluded from the analysis. The remaining variables were then subjected to several semi-automatic feature selection methods, including Minimum Redundancy-Maximum Relevance (mRMR), Gain Ratio (GR), Random Forest (RF), XGBoost, Forward Feature Selection (FFS), and Backward Feature Selection (BFS). Each method ranked the features from most to least important in a different manner, based on their own approach. An XGBoost model was then iteratively trained using first the best ranked feature, then the next two best, then the next three best and so on until all features were included. This approach provides a systematic way to automate feature selection when dealing with a large number of variables and multiple options to choose from.

2.4. Models

We decided to build the models using XGBoost (extreme gradient boosting, version 2.1.2). XGBoost is a powerful ML algorithm that has gained immense popularity due to its power and speed. It is particularly effective on structured data and has become the preferred option for many data scientists and ML professionals due to its high performance, its built-in regularization techniques that help prevent overfitting, and its flexibility as it supports multiple objective functions, including regression, classification and ranking. This flexibility allows it to be applied to a wide range of problems, making it a versatile tool in the data scientist's toolkit. [17,18]. The models developed using XGBoost were evaluated to investigate their effectiveness in predicting VAP 24, 48 and 72 hours before the date of the event using accuracy, re-call, confusion matrix and area under the ROC curve (AUC).

2.4.1. Data Splitting

The data were systematically divided into two distinct sets: a training set (80%) and a test set (20%) using scikit-learn (version 1.6.1). This split was stratified to ensure that the distribution of the positive VAP class remained consistent between the two sets. In addition, cross-validation was applied during the training phase to ensure the robustness of the model due to the limited data.

2.4.2. Class Imbalance

The positive VAP class was significantly smaller than the negative class, resulting in an unbalanced cohort. When training a model, this imbalance must be taken into account to avoid classification bias towards the main class. To minimize this effect, we applied reduced sampling using a custom function, which contained the resample function from scikit-learn (version 1.6.1). This function selectively reduces the number of majority class samples while preserving all minority class instances. The function allows adjusting the balance ratio between classes through a configurable parameter declared as majority proportion, ensuring flexibility in determining the level of downsampling applied. A value of 1 resulted in equal numbers of positive and negative samples, while a value of 2 meant the majority class had twice as many samples as the minority class, and so forth. The selected majority class samples were randomly chosen to maintain variability in the dataset. This method was applied to the training set before model training, ensuring that different levels of class balance could be tested systematically. The downsampling process was performed with a fixed random seed to allow for reproducibility. The code used is accessible at https://github.com/julenberrueta/vap-model-utils.

2.4.3. Hyperparameter Tuning

The Optuna library (version 4.1.0) was used for hyperparameter tuning. Optuna is an open source optimization library specifically designed to automate the optimization of hyperparameters in machine learning models. It works by efficiently searching the parameter space to find the most effective values, using techniques such as Bayesian optimization, tree-structured Parzen estimators or evolutionary algorithms. Optuna streamlines the process by providing a framework for defining a search space and managing the trial process to evaluate different configurations. This approach helps identify the optimal set of parameters that lead to the best model performance, improving both the efficiency and accuracy of machine learning models [19]. The score used to optimize the models during the tuning phase was the AUC score. This approach was chosen to emphasize the importance of creating a strong discriminative model, ensuring that the model is confident in its predictions while maintaining a good balance between predictions in the negative and positive classes.

2.4.4. Model Explainability

To assess the relative importance and contribution of each variable to the model's predictions, the SHapley Additive exPlanations (SHAP) method was used. SHAP values provide a clear, interpretable understanding of how each feature influences the model's output and how the range of values in the distribution of a given variable contributes differently to the model's predictions.

2.5. Definitions

Ventilator-associated pneumonia (VAP) was defined as a respiratory infection occurring in mechanically ventilated patients according to the guidelines of the European Respiratory Society (ERS), the European Society of Intensive Care Medicine (ESICM), the European Society of Clinical Microbiology and Infectious Diseases (ESCMID), and the Asociación Latinoamericana del Tórax (ALAT) [20]. VAP was defined as pneumonia occurring more than 48 h after endotracheal intubation with fever, without other apparent causes, with new or increased sputum production, positive endotracheal aspirate (ETA) culture (>106 CFU/mL), or bronchoalveolar lavage (BAL) culture (>104 CFU/mL), with at least one respiratory pathogen known to cause pneumonia, and with radiographic evidence of nosocomial pneumonia.

2.6. Objectives

Our primary objective was to evaluate the predictive performance of the VAP prediction models developed using XGBoost.

Our secondary objective was to evaluate the applicability of the prediction models in a clinician-friendly front-end interface as part of the model maturity definition.

2.7. Statistical Analysis

A univariate statistical analysis was conducted using R (version 4.4.2) and the compareGroups library (version 4.9.1) to summarize and compare baseline characteristics between groups based on VAP status. The analysis included both categorical and continuous variables. For categorical variables, the Chi-squared test was employed to assess differences in distribution between groups, with the exact Fisher test applied when the expected frequencies were less than 5. Continuous variables were evaluated using the Kruskal-Wallis test, as normality assumptions were not met.

Model performance was assessed by determining accuracy (for overall model performance), recall (to assess the model's ability to predict the positive class), confusion matrix (to provide a detailed view of true positives, false positives, true negatives and false negatives) and AUC (area under the curve, which measures the model's ability to discriminate between classes).

3. Results

3.1. Population

A total of 4,394 patients who received mechanical ventilation were included in the study. Of these, 1,560 patients who received invasive mechanical ventilation for less than 48 hours were excluded, leaving 2,834 patients. From this group, 95 patients were excluded because they spent less than 72 hours in the ICU. Of the remaining 2,739 patients, 339 were diagnosed with VAP. However, 25 patients were later excluded because the date of VAP diagnosis was incorrectly recorded, as they had been on MV for less than 48 hours at the time of annotation, resulting in a final count of 314 VAP cases (Figure 1).

The variables considered and the characteristics of the patients distinguishing those with and without VAP development are shown in Table 1. Patients with VAP were more likely to be male, have medical conditions, and have a lower mean age and number of days on MV than patients without VAP. The intensity of ventilatory support (FiO2, PEEP, etc.) and inflammatory variables (CRP, leukocytes, etc.) were higher in the VAP group. Strikingly, although the intensity of organ dysfunction (SOFA score) was higher in patients with VAP, the overall severity (APACHE II score) was higher in patients without VAP (Table 1). Finally, the overall crude mortality was 28.4% and higher in patients with VAP (35.4%, p=0.004).

3.2. Models Performance

The XGBoost model performed best in the 24-hour prediction window, achieving an accuracy of 0.85, a recall of 0.83 and an AUC of 0.90. These results indicate that the model effectively discriminates between VAP and non-VAP cases, with a reasonable balance between the two classes. The high AUC value of 0.90 highlights the excellent discriminatory ability of the model.

For the 48-hour prediction window, the performance of the model decreased, with an accuracy of 0.71, a recall of 0.76 and an AUC of 0.77. Although the model maintained a reasonable ability to identify cases of VAP, it had a higher number of false positives. The AUC of 0.77 indicates moderate discriminatory power.

In the 72-hour prediction window, the model's performance showed a greater decline, reaching an accuracy of 0.70, a recall of 0.73 and an AUC of 0.75. These results suggest that the model struggled to accurately identify cases of VAP over a longer period of time, resulting in an increase in the number of false negatives and false positives. The AUC of 0.75 indicates moderate performance. (Table 2)

Figure 2 shows the comparison of the performance (AUC ROC) of the variable selection algorithms used for different numbers of features for predicting the 24 hours before the event. As can be seen, the XGBoost algorithm shows constant superiority, especially with fewer features, highlighting its effectiveness in selecting the most relevant variables. On the basis of this classification, the most important variables included in the different models were selected (Table 2).

3.3. User-Friendly Model Visualization

The models have been implemented within a clinician-friendly front-end that allows the different probabilities of risk of developing VAP for each patient to be viewed in near-real time throughout the patient's stay.

Figure 3 shows an example of a patient who developed VAP. The red dot indicates the time when the patient had VAP and the corresponding probability, while the green dot represents the probability exactly 24 hours before the event. For this particular patient, we can see that all the models are predicting correctly. However, as we approach the end of the stay, we see that the 24-hour model tends to give clearer predictions when the probability is less than 0.5, while the other two models show greater variability.

Figure 4 shows the case of a patient who did not develop VAP. The blue dot indicates the last day the patient was mechanically ventilated and serves as a reference date for patients who did not develop VAP. We can see that the 24-hour model consistently and accurately predicted that this patient would not develop VAP throughout the entire stay, whereas the other two models incorrectly predicted that the patient would develop VAP at some point during the stay.

3.4. Model Explainability

The application of SHAP analysis to the prediction models provided valuable insights into the key factors influencing VAP prediction. Of the eleven features evaluated, PEEP (maximum), temperature (maximum), purulent secretions, respiratory rate (minimum) and C-reactive protein (minimum) consistently emerged as the five most influential predictors. The color scale in the figure represents the relative magnitude of each variable, with red indicating higher values and blue indicating lower values within each characteristic. In addition, the SHAP values on the x-axis indicate the direction of the contribution: values to the right of zero are associated with an increased likelihood of VAP, whereas values to the left of zero contribute negatively to the prediction.

4. Discussion

The main finding of our study is that the development and implementation of a model for predicting VAP in almost real time on a user-friendly interface was associated with an adequate prediction (AUC 90%), especially in the 24 hours prior to the development of VAP. To the best of our knowledge, this is the first study to implement a real-time VAP prediction algorithm in an easy-to-use user interface for healthcare professionals within the dashboard commonly used in our ICU. [11,21,22]

Current diagnostic approaches to VAP (e.g. signs and symptoms, microbiological cultures and chest X-rays) and conventional surveillance methods (e.g. manual chart reviews, prevalence studies, discharge codes, electronic surveillance algorithms) are insensitive, time-consuming and often costly [23,24,25]. In addition, they cannot identify a person who would benefit from a particular type of diagnosis or treatment based on the expected course of the disease.

AI is transforming medical practice and precision medicine in ICUs [26,27]. Emerging technologies using ML techniques will overcome the limitations of current diagnostic approaches [23,24,25,26,27]. These data-intensive systems will integrate and analyze diverse structured and unstructured data from different sources to monitor temporal trends, identify risk factors and predict the onset of a possible complication (VAP), thus facilitating or supporting rapid clinical decision-making regarding the diagnosis and possible treatment of infectious diseases.

For more than 10 years, various researchers have developed automated methods for the objective detection of VAP [23,24,25,28,29]. However, these automated systems have not improved the diagnostic accuracy of VAP [28], mainly because they have considered definitions related to epidemiological surveillance rather than clinical definitions. Furthermore, automation without AI techniques cannot provide information on the likelihood or risk of future complications (VAP) in patients receiving invasive mechanical ventilation.

Several authors have investigated the accuracy of electronic nose (e-nose) sniffers in detecting possible cases of VAP [30,31]. These devices use ML methods to analyze exhaled breath for metabolites that may be indicative of VAP, and some have demonstrated strong discrimination in identifying the presence of VAP. However, prospective validation of an e-nose device found that sensitivity and specificity were insufficient for general clinical use.

Giang, C et al. developed a model to predict VAP using the publicly available MIMIC III database [32]. The authors used a similar methodology to our study and tested the usefulness of different ML techniques. For predicting VAP after 48 hours of intubation, the XGBoost models performed better than other types of models, especially in the 6-hour time window (AUC 79.9%). Although with a lower performance than that observed in our study (AUC 90%), the results are consistent with our findings. However, there are some important differences. One would expect a model closer to the event (6 h) to perform better, but its predictive value has a limited clinical impact. A major limitation is that the exact time of onset of VAP could not be determined retrospectively from this dataset, so it was not possible to determine the degree of early warning provided by the models. In addition, these results may also be limited by the use of ICD codes as the gold standard for VAP diagnosis, which may not accurately capture all patients who experienced VAP during their hospital stay. Our study recorded the exact time of clinical diagnosis of VAP (not ICD), which provides a better and more robust prediction. In addition, Liang, Y et al. [33] also developed a VAP prediction model with the MIMIC III database. The authors point out that the AUC of optimal performance (84%) corresponds to the random forest model. Although the performance of the model is very good, it is striking that the authors do not clearly explain the development of the model or the selection of variables. On the other hand, it should be noted that using the same database (MIMIC III) as Giang, C. et al. [32] they find a significantly lower number of patients (n=221) with VAP compared to Giang, C. et al.(n= 524). This casts doubt, in our opinion, on the veracity of the VAP diagnosis and its temporal association necessary to determine the time windows in the construction of the model.

Furthermore, the elegant study by Samadani A et al [34], which used a large multi-institutional electronic health record (EHR) database, including 57,944 patients, of whom 9,204 had suspected VAP events, obtained results similar to ours. The development of XGBoost models to predict VAP events with a lead time of 24 hours showed reasonable performance with an AUC ROC of 76%. Although this study included a large number of patients from different centres, it only evaluated the prediction of VAP using international guidelines for the diagnosis of VAP. Specifically, the authors consider the presence of VAP when antibiotics are administered close to the time of culture orders in the EMR. As the authors acknowledge, the time of onset of VAP marked by the criteria used may not coincide with the actual physiological onset of infection, and this may strongly influence the prediction. The lack of real-world data to define VAP events and, more importantly, their timing in relation to recording, poses a significant challenge to the development of predictive models. The accuracy of the time of diagnosis of VAP by the treating physician is one of the major strengths of our study. Furthermore, the reliability of these data is a critical issue in modelling. In our ICU, the quality and reliability of the data is optimized by random audits, as previously published [35,36,37,38]. This makes the model more robust and reliable in its predictions.

The limited clinical relevance of the models, the lack of reliable data and adequate infrastructures, the costs associated with the digitalization of the ICU, as well as legal and ethical considerations have hindered the clinical integration and implementation of AI models [33,34] in clinical practice. Although AI can enable the development of accurate decision support tools, their adoption and implementation must be preceded by careful consideration of their clinical utility in real practice, aimed at meeting a real need, in a user-friendly environment for the clinician, and allowing their proper integration into the existing workflow. We have proven that the prediction model can be included in a front-end that is user-friendly for clinical use and thus integrated into the daily workflow of the ICU. [21,22]

As recently highlighted by Lee et al [39], in the era of digital healthcare it should be possible to dynamically adapt decision support by identifying and analyzing similar patients in the past. However, most existing clinical applications of AI have been achieved using static prediction models for the whole population. Recent advances in AI methods allow for processing and learning through a real-time data analysis platform to obtain predictions that are modified over time based on the changes experienced by the different variables included in the model. In this way, and as proposed by our model, continuous learning from time series in the same patient over time provides a much more personalized prediction than traditional models. Obtaining a quality prediction as a decision support tool can enable the active implementation of diagnostic methods aimed at confirming the presence of VAP and possibly initiating early antimicrobial treatment. Finally, as argued by Frondelius T et al [40], an ideal diagnostic test should have high accuracy and be low cost, non-invasive, easily repeatable at specific intervals, non-technical and clinically available at the patient's bedside. Our model fulfils all these requirements and is a highly reliable decision support tool. Moreover, it is also highly generalizable. The 24-hour model requires only 8 variables to operate, and the complete set of models (24h, 48h, and 72h) runs with just 21 unique variables. This streamlined variable set not only facilitates easy integration into clinical workflows but also underscores the model’s adaptability across diverse healthcare settings. However, its implementation in the real world cannot be assessed until external prospective validation confirms its usefulness.[41]

Our model has important limitations that must be acknowledged. The first and most important of these is that our results must be treated with caution because, although the model has been validated internally, without prospective and external validation it is not possible to conclude on its usability and clinical impact in real life [41]. Secondly, the model was developed in a single ICU with specific characteristics and a long history of data use, so the results cannot be extrapolated to other ICUs without prior validation. Finally, our model includes clinical variables that need to be recorded by physicians and nurses (e.g. VAP, type of secretions, number of secretions aspirated per shift, etc.). We acknowledge that these variables are not commonly recorded, which may limit the implementation of the model in other ICUs where these variables are not considered.

5. Conclusions

The VAP prediction model developed provides an instantaneous risk score that changes with the patient's clinical condition, providing a temporal context for the infection event, allowing for the study of disease progression, as well as early intervention and confirmatory diagnostic measures. However, the usability of the model and its real clinical impact need to be confirmed through prospective validation.