Clinical Context Is More Important Than Data Quantity to the Performance of an Artificial Intelligence-Based Early Warning System

1. Introduction

Clinical data are heterogeneous in real-world healthcare settings. Some patients, particularly those with complex comorbidities, undergo frequent laboratory evaluations that produce an abundance of data [1]. Other patients who are perceived as clinically stable often have sparser data available, thus raising concerns about whether machine learning models can accurately predict adverse outcomes under such incomplete circumstances. Traditionally, missing data have been viewed as problematic because missingness can lead to biases or imputed approximations [2,3]. However, previous studies have introduced the notion of “informative presence,” which suggests that the absence of laboratory tests is not random; instead, this absence signals that specific tests were not performed because no abnormality was suspected [4]. The VitalCare–Major Adverse Event Score (VC-MAES), which is an artificial intelligence (AI)-based early warning system, leverages this concept by conservatively imputing missing values by assuming that the unmeasured parameters were likely within normal ranges.

Our recent study demonstrated that artificially imputing these missing values with approximate estimates reduced the performance of the VC-MAES compared to that achieved by using the system’s default normal value replacement, suggesting that missing healthcare data can have intrinsic meaning and reflect the decision-making process of clinicians [5]. Furthermore, this study also demonstrated the following implicit clinical rationale: if no concern exists, then further testing may not be required. To elucidate how baseline severity intersects with these patterns of missingness, we categorized inpatients using the Charlson Comorbidity Index (CCI) to determine whether data quantity alone drives the predictive performance and whether ordering (or forgoing) laboratory tests plays a critical role in predicting outcomes.

2. Materials and Methods

This retrospective analysis was performed at Presbyterian Medical Center in the Republic of Korea. Adult patients (19 years or older) admitted to general wards between December 2022 and May 2024 were included if at least one valid measurement of five key vital signs (systolic blood pressure, diastolic blood pressure, heart rate, respiratory rate, and body temperature) had been performed. Patients who were directly transferred from the emergency department or operating room to an intensive care unit were excluded because they were considered planned intensive care unit admissions. Baseline comorbidities were assessed using the CCI. Patients with a CCI greater than 3 were classified as having high severity, and those with a CCI of 3 or less were classified as having moderate or low severity [6].

Clinical endpoints included unplanned intensive care unit transfers, cardiac arrest, or death during hospitalization. The VC-MAES, which is a deep learning-based model, was used to predict these events up to 6 hours in advance by incorporating time series of available vital signs, age, and laboratory test results (total bilirubin, lactate, creatinine, platelets, pH, sodium, potassium, hematocrit, white blood cell count, bicarbonate, and C-reactive protein). Missing test results were handled via the last observation carried forward method, with default normal values assigned if prior measurements were not available. The model performance was evaluated using the area under the receiver-operating characteristic curve.

Demographic characteristics and the proportions of missing laboratory results were compared between groups classified by the CCI. Categorical variables were analyzed using the chi-square test, and continuous variables were compared using either the independent t-test or the Wilcoxon rank-sum test, depending on the distribution of the data. Differences in proportions of missingness were assessed using a two-sided z-test for independent proportions with continuity correction, based on the score method. A two-sided P-value of < 0.05 was considered statistically significant.

3. Results

During this study, 24,359 hospitalizations occurred; specifically, 12,139 patients in the high severity group (CCI >3) and 12,220 patients in the moderate/low severity group (CCI ≤3) were hospitalized. Patients with high severity underwent more laboratory investigations, resulting in fewer missing values and a higher rate of unplanned intensive care unit transfer, cardiac arrest, or death (4.8%), consistent with their high risk of adverse outcomes at baseline. Conversely, patients in the moderate/low severity group underwent fewer laboratory tests and, consequently, exhibited higher missingness rates; however, they experienced significantly fewer adverse events overall (1.0%). Table 1 summarizes the baseline demographic characteristics, vital signs, and differences in laboratory test missingness of the high severity and moderate/low severity groups.

In both the high-CCI and moderate/low-CCI groups, patients who experienced adverse events consistently had fewer missing laboratory values than those without events, reflecting more frequent testing when clinical deterioration was suspected. Among the event cohorts specifically, patients in both the high-CCI and moderate/low-CCI groups exhibited similar proportions of missing values overall; however, the high-CCI group had fewer missing pH and HCO₃ values (0.34 vs. 0.43, p = 0.08), suggesting an even more intensive diagnostic approach for higher-risk patients (Figure 1).

When used to predict clinical deterioration events, the VC-MAES demonstrated robust performance among both cohorts despite differences in data availability. The areas under the receiver-operating characteristic curve of the 6-hour prediction window of the high severity and moderate/low severity groups were 0.86 and 0.85, respectively (Figure 2).

BMI, body mass index; CCI, Charlson Comorbidity Index; DBP, diastolic blood pressure; F, female; IQR, interquartile range; M, male; SBP, systolic blood pressure.

4. Discussion

Our findings indicated that the discriminative performance of the model did not exclusively depend on the volume of laboratory data. However, missingness alone appeared to be a clinical indicator of whether a patient was perceived as requiring further investigation, thus allowing the model to capture signals of deterioration even when laboratory data were sparse. This observation strengthens the concept of “informative presence,” whereby each decision to order or forgo a test can be diagnostically relevant, thus reflecting the degree of clinical concern.

The CCI was used to classify patients into the high severity (CCI >3) and moderate/low severity (CCI ≤3) groups. This classification revealed that the comorbidity burden can shape clinical trajectories as well as clinical decisions to order (or not order) laboratory tests. Our results showed that patients with a high CCI generally underwent more frequent testing, resulting in fewer missing values, whereas patients with a low CCI underwent less frequent testing. Despite this disparity in data volume, the AI-based early warning system maintained its robust discriminative power when used to evaluate both groups, indicating that the quantity of data alone did not guide the model performance.

Although many predictive modeling approaches focus on maximizing data completeness—either by collecting more frequent measurements or by aggressively imputing missing values—our findings suggest that such strategies may unintentionally obscure the inherent meaning behind “naturally” missing data. For instance, patients who are clinically stable are less likely to undergo repeated tests; therefore, the absence of multiple laboratory test results may be a marker of lower acuity. If we artificially fill these gaps with imputed values that do not reflect true clinical reasoning, then we could diminish the ability of the model to recognize genuine patterns in practice [5].

Additionally, our study resonates with the broader literature that showed that capturing real-world practice patterns can enhance the generalizability of AI-based early warning systems [7,8]. Because clinicians with different specialties have various thresholds for ordering laboratory tests, the VC-MAES model effectively learned to incorporate these differences in clinical behavior. This approach could allow for more seamless scaling to diverse healthcare environments because it does not rely on uniform data collection protocols; instead, it adapts to existing practices.

Importantly, this study was conducted at a single center in Korea; therefore, local clinical workflows and resources may have influenced the laboratory tests that were ordered and baseline comorbidity profiles. Although the CCI is widely used, other severity measures may offer different views of missingness and event rates. Additional multi-institutional and prospective studies should be performed to further validate these findings to potentially allow refinement of the management of missing data and testing variations and optimize risk prediction across settings.

Overall, our results indicated that respecting the natural patterns of test ordering, which often reflect clinical judgment, may be more beneficial to predictive accuracy than striving for exhaustive data. By leveraging the “informative presence,” AI-based models can balance their robustness with real-world applicability, thus ensuring that they genuinely identify patients who are at risk without necessitating unnecessary or duplicative testing. The consistent performance across CCI groups suggested that the underlying context of missingness, rather than the absolute quantity of data, plays a decisive role in model accuracy.