Validation of Clinical Prediction Models with Repeated-Measures Predictors: A Methodological Framework for Neonatal Digital Twins in IoT Environments

1. Introduction

The neonatal intensive care unit (NICU) is one of the most densely instrumented IoT environments in clinical medicine, with each critically ill infant continuously monitored by bedside devices generating heterogeneous, high-frequency physiologic data streams that present the same challenges of real-time acquisition, integration, and predictive modeling that define IoT systems in industrial and engineering domains. These signals range from high-frequency waveform data such as ECG and plethysmography to lower-frequency device outputs and discrete electronic health record events including laboratory measurements, medication administrations, and nursing assessments. Over a hospitalization lasting weeks to months these multimodal data sources collectively generate millions of observations per patient, making the NICU a highly instrumented clinical environment analogous to industrial IoT systems where DT architectures were first developed.

A DT, in its most general formulation, is a computational model that mirrors a physical entity in sufficient fidelity to facilitate simulation, prediction, and decision support [1]. In the manufacturing and engineering literature, DT frameworks are mature: architectures for data acquisition, model calibration, real-time synchronization, and deployment have been systematically described [2,3]. The translation of these frameworks to clinical medicine has accelerated considerably in recent years, with proposed applications ranging from individualized drug dosing to in silico surgical planning [4,5]. In neonatal medicine specifically, Pammi and colleagues [6] articulated a compelling vision in which digital twins and synthetic patient data could empower pediatric clinical trials by providing virtual control arms, reducing enrollment burdens, and enabling more rapid evaluation of interventions for one of medicine’s most vulnerable populations, a vision whose realization depends critically on the prediction modeling infrastructure that this review addresses. The monitoring DT application, in which a computational model synchronizes continuously with NICU IoT data streams to produce updated risk predictions, is a natural application of the IoT environment the NICU already represents and the focus of this review.

That vision is scientifically well-motivated. Adverse outcomes among extremely preterm infants are frequent, clinically consequential, and often preceded by physiologic changes detectable in continuously monitored data before they are clinically apparent, as has been demonstrated for late-onset sepsis [7]. What remains absent, however, is the validated methodological scaffolding needed to build such a monitoring DT responsibly. The Pammi et al. viewpoint [6], like much of the DT clinical literature, focuses appropriately on the conceptual case and the ethical prerequisites. It does not, and was not intended to, specify the supervised learning strategies best suited to high-dimensional neonatal IoT data, or the cross-validation approaches that preserve validity when data are clustered by infant and center and temporally autocorrelated within infants. These gaps are not peripheral implementation details; they are the critical unsolved problems that separate the DT vision from realizable clinical application. This review argues that the primary barrier to neonatal digital twin implementation is not data availability but the absence of validated methodological standards.

This problem has structural parallels to challenges well established in other high-dimensional biomedical settings. In genomic classification, the combination of large feature spaces, small effective sample sizes, and non-independent observations created a methodological crisis that took over a decade to partially resolve through the development of penalized regression, ensemble methods, and rigorous cross-validation protocols [8,9,10]. The NICU DT problem shares this structure: feature dimensionality is high, effective sample sizes for rare clinical outcomes are small, observations within infants are serially correlated, and the consequences of methodological shortcuts (e.g., overfitted models, optimistically biased performance estimates, poorly calibrated predictions) are clinically consequential [11]. The lessons learned from high-dimensional genomic prediction are directly applicable here, yet their translation to the DT-IoT context has not been systematically described. Validation before deployment is, moreover, a recognized general requirement for DTs across engineering domains: recent work in this issue frames pre-execution verification as a prerequisite for safe deployment of robot control systems [12], and condition-monitoring DTs for power electronics are explicitly evaluated for reliability across operating conditions before deployment [13]. The National Academies have similarly identified verification and validation as a foundational gap for DTs across domains [14]. The methodological contribution of this review is to specify what that validation requirement demands in the neonatal monitoring DT context, where the data structure, repeated within-subject physiologic measurements clustered by infant and center, renders standard validation methods invalid.

Standard clinical prediction modeling methodology, including penalized regression for cross-sectional outcome prediction, temporal external validation, and nomogram-based deployment, is well established for single-observation-per-subject clinical outcomes [15,16]. The methodological gap addressed by this review concerns the next step: validation of prediction models constructed from repeated within-subject measurements of physiologic data, for which the independence assumptions underlying standard validation methods are violated. Recent simulation work by the present author has quantified the optimism produced when these assumptions are ignored, providing the foundation on which the cross-validation recommendations developed in this review are constructed [17]. The machine learning methodology needed to construct and validate neonatal DTs in IoT-enabled NICU environments is examined here, with particular attention to three problem areas where the gap between current DT-IoT practice and neonatal clinical requirements is most acute: cross-validation strategies for clustered, temporally autocorrelated data; feature selection under small effective sample sizes; and calibration-aware model evaluation. Following the taxonomy of Drummond and Gonsard [18], DTs may be classified as simulated or monitoring variants; this review focuses on the monitoring DT, which performs factual prediction of clinical events from continuously updated physiologic streams and is the variant for which the validation methodology described here is directly applicable. The simulated DT, which performs counterfactual prediction under hypothetical interventions, raises a distinct set of identification and validation problems that fall outside the scope of this review and are noted where relevant. A structured framework integrating clinical prediction model standards [15,19] with DT-IoT methodology is proposed, concluded with a prioritized research agenda. The aim is not to survey what has been done (the existing DT clinical literature has been ably reviewed elsewhere [4,20]), but to specify what must be done before neonatal DTs can be built responsibly and deployed credibly. Although the examples throughout are drawn from neonatal intensive care, the validation principles described are general: they apply to any clinical prediction model built on repeated-measures or clustered data, including adult intensive care, electronic health record-derived cohorts, and wearable-device monitoring. The conceptual scope of this review, encompassing the full neonatal DT pipeline from physical twin to clinical application, is summarized in Figure 1, with the methodological problem areas detailed in subsequent sections corresponding to the ML pipeline stage.

2. The NICU as a DT-IoT Use Case: Data Landscape and Clinical Requirements

2.1. The Physical Twin Substrate: IoT Data Streams in the NICU

Understanding what data a neonatal DT must model begins with a clear accounting of what the physical twin, the hospitalized preterm infant, actually generates. The NICU physiologic data environment is characterized by extreme heterogeneity across at least three dimensions: sampling rate, data type, and clinical signal density (i.e., the amount of clinically meaningful information contained in the data per unit time). High-frequency continuous streams include electrocardiographic waveforms, pulse oximetry plethysmography, and invasive arterial pressure traces, which may be sampled at 125–500 Hz and produce tens of millions of data points per patient per day. At intermediate frequencies, respiratory monitoring systems capture ventilator-derived parameters (tidal volume, peak inspiratory pressure, compliance, resistance) at breath-by-breath resolution, typically on the order of tens of observations per minute. At the low-frequency extreme, discrete clinical events (e.g., laboratory draws, medication administrations, nursing assessments, radiographic interpretations) are recorded as structured or semi-structured entries in the electronic health record (EHR), often with irregular and clinically driven timing rather than fixed intervals.

These streams are not independent. Physiologic parameters in the NICU are coupled in ways that reflect underlying pathophysiology: heart rate variability patterns anticipate late-onset sepsis by 12–24 hours [7]; cerebral oxygenation trends co-vary with mean arterial pressure in the setting of impaired autoregulation [21]; ventilator-derived compliance trajectories track surfactant response and predict extubation readiness [22]. A neonatal DT that models each stream in isolation fails to capture this clinical richness. The coupled, serially correlated structure of these data is precisely what makes their validation non-trivial, an observation that bears directly on the cross-validation framework developed in Section 4.

2.2. Clinical Outcomes of Interest and Their Statistical Implications

The clinical outcomes that matter most in the NICU, and that a monitoring DT must ultimately predict, are characterized by two features that create structural statistical challenges: they are rare, and they are inconsistently defined. Among extremely preterm infants born before 28 weeks gestation, necrotizing enterocolitis (NEC) occurs in approximately 9% [23]; late-onset sepsis in approximately 20% [23]; and severe intraventricular hemorrhage (IVH, grade III–IV) in approximately 14% [23]. Bronchopulmonary dysplasia (BPD) affects at least 40% of extremely preterm survivors depending on the definition applied; the 2018 NICHD workshop [24] proposed a severity-graded physiologic definition to address the limitations of earlier criteria, though BPD rates vary substantially across cohorts as a result of definitional heterogeneity. Mortality before NICU discharge among extremely preterm infants is approximately 22% overall in contemporary US academic cohorts [23], with variation that is steep across even single-week differences in gestational age at the threshold of viability, exceeding 50% at 23 weeks and falling below 10% by 28 weeks. These outcomes are each a single, fixed binary endpoint per infant; a monitoring DT predicts them by refreshing its prediction as the physiologic predictor stream accumulates, a structure that is consistent with the repeated-measures predictor, subject-level binary outcome formulation that the validation methodology in Section 4 addresses.

These event rates have a direct consequence for DT model development that is frequently overlooked in the DT-IoT literature: at single-center scale, the number of positive-outcome cases available to train a supervised classification model is often fewer than 20 per class, the regime in which misclassification error rates increase sharply and model stability degrades substantially, regardless of the learning algorithm employed (Figure 2) [10,25]. A hypothetical center admitting 60 extremely preterm infants per year at an approximately 9% NEC rate produces approximately five to six NEC cases annually, yielding a training dataset of perhaps 25–30 cases over five years of data collection. This is not a problem that can be solved through improved feature engineering or more complex model architectures; it is a fundamental constraint of single-center neonatal data that pushes NICU DT development inescapably toward multi-center architectures (introducing an additional level of data clustering), and that makes the methodological choices in Section 3 and Section 4 consequential rather than academic.

2.3. What a Clinically Useful NICU DT Must Do

Following the taxonomy proposed by Drummond and Gonsard [18], neonatal DTs can be classified as either simulated or monitoring variants, a distinction that is not merely architectural but corresponds to a fundamental difference in statistical formulation. A monitoring DT maintains continuous synchronization with the physical twin through real-time IoT data feeds, updating its internal state as new physiologic information arrives, and is grounded in factual prediction models that estimate outcomes under observed conditions. A simulated DT is a one-time computational model of a patient state used for scenario testing or counterfactual prediction, estimating patient-specific outcomes under hypothetical interventions. The counterfactual formulation necessitates specialized identification assumptions (consistency, conditional exchangeability, positivity), estimation methods such as inverse probability weighting, and validation procedures beyond those of standard predictive modeling frameworks [26]. This review focuses on the monitoring DT, because its factual prediction task, mapping repeated-measures physiologic predictors to a clinical outcome, is the formulation for which the cluster-aware validation methodology described here is directly applicable. The simulated DT is acknowledged as clinically relevant but is treated as out of scope, since its counterfactual validation burden is a distinct problem.

For a monitoring DT, the core requirement is a prediction model whose estimated performance is trustworthy: discrimination and calibration that are accurately estimated, not optimistically inflated by methodological shortcuts. Because the predictors are repeated within-infant physiologic measurements, the principal threat to trustworthy performance estimation is the dependence structure of the data, which standard validation methods do not accommodate. A monitoring DT intended for bedside deployment must additionally satisfy operational requirements including calibration-driven alarm performance and prospective decision threshold validation (Section 4.4) and a HIPAA-compliant IoT data pipeline architecture (Section 6). The remainder of this review addresses these requirements in the order in which they arise in the DT development pipeline.

3. Supervised Learning Approaches for DT Construction

3.1. Framing the Prediction Problem

Constructing a monitoring DT from IoT data requires mapping a set of input features, derived from physiologic streams, EHR records, and demographic characteristics, to one or more clinical target outcomes. Depending on the application, that target may be a binary outcome (NEC or no NEC by day of life 30), a continuous physiologic trajectory (predicted arterial oxygen saturation over the next six hours), or a time-to-event endpoint (days to extubation readiness). Each formulation calls for a different family of supervised learning methods: classification models for binary and categorical outcomes, regression models for continuous targets, and survival or competing-risks models for time-to-event endpoints with informative censoring. The choice among these formulations is not merely technical; it determines what a DT can and cannot represent.

In the NICU context, binary classification of adverse outcomes is the formulation most relevant to monitoring DT applications, and it is the formulation most severely challenged by the small effective sample sizes described in Section 2.2. The following subsections focus on this problem, though the methodological principles generalize to the other formulations.

3.2. Feature Selection Under High Dimensionality and Small Effective N

Feature selection, the process of identifying which inputs carry predictive signal and which are noise, is a prerequisite for supervised learning on high-dimensional NICU IoT data, just as it is in genomic classification [8]. A continuous physiologic monitoring dataset for a single VLBW infant may contain hundreds of derived features: mean, variance, and distributional summaries of each waveform; cross-spectral coherence between channels; time-since-last-event features for discrete clinical data; and interaction terms between physiologic and clinical predictors. When the number of candidate features substantially exceeds the number of outcome events, a near-universal condition in single-center NICU data, standard maximum-likelihood estimation is undefined or severely overfit, and some form of regularization or pre-selection is essential.

Three broad strategies are available. Filter methods rank features by univariate association with the outcome (e.g., t-tests, Wilcoxon rank-sum statistics, or information-theoretic measures) and retain a top-k subset prior to model fitting. They are computationally efficient but ignore inter-feature dependencies and can be unstable: small perturbations of the training sample can produce substantially different feature rankings, a phenomenon well documented in high-dimensional biomedical settings [8,10,25,27]. Wrapper methods (e.g., recursive feature elimination, sequential forward selection) evaluate feature subsets by internal cross-validated model performance, making them sensitive to the specific learning algorithm and prohibitively expensive for large feature spaces. Embedded methods, most importantly LASSO (least absolute shrinkage and selection operator) and elastic net regularization, perform feature selection and model estimation simultaneously by penalizing the sum of absolute coefficient values, shrinking uninformative predictors exactly to zero [28]. For clinical prediction modeling, embedded methods such as LASSO are often favored when parsimony and clinical interpretability are priorities, as their tuning parameter can be selected by internal cross-validation and they produce sparse models well suited to deployment in clinical settings [16,29].

A critical but underappreciated property of all feature selection methods in small-sample settings is instability: the set of features selected varies substantially across bootstrap resamples of the training data when effective sample sizes are small [27]. Simulation evidence confirms that LASSO, while preferred among embedded methods, remains susceptible to this instability in the small-N regime characteristic of single-center NICU data [30]. This instability is not merely a theoretical concern. A neonatal DT whose predictive features vary substantially across resampled training iterations may be difficult to validate, update, or compare across institutions, as such variability can reflect sensitivity to sampling variation rather than stable underlying signal. Stability-informed selection approaches (e.g., stability selection [31], which applies feature selection to many bootstrap subsamples and retains only features selected with high frequency) provide a principled framework for assessing the robustness of selected predictors and may improve reproducibility relative to single-pass selection. Given increasing recognition of the importance of model robustness and reproducibility in clinical prediction modeling, reporting measures of predictor stability is proposed here as a complement to existing reporting standards (e.g., calibration reporting in TRIPOD+AI [19]) for NICU DT models.

3.3. Choice of Learning Algorithm for Sparse, High-Dimensional NICU Data

Given the feature selection landscape described above, the choice of supervised learning algorithm for NICU DT construction should be guided by three criteria: performance under small effective sample sizes, robustness to class imbalance, and interpretability sufficient to support clinical trust and regulatory review. Penalized logistic regression (e.g., LASSO or elastic net) satisfies all three reasonably well. Its performance at small N is competitive with more complex methods [9], its coefficient structure is transparent, and its output is a calibrated probability amenable to clinical decision thresholds. Evidence from a large-scale systematic review corroborates this preference: across clinical prediction modeling studies using tabular data, flexible machine learning methods demonstrate no average performance advantage over logistic regression [32]. Van Calster et al. [30] attribute this in part to the pervasive low signal-to-noise conditions that characterize clinical datasets, precisely the conditions that define single-center NICU data, where penalized regression is less susceptible to overfitting on irrelevant structure. Ensemble methods, particularly random forests and gradient boosting, offer greater modeling flexibility, but their advantage over penalized regression may diminish in smaller samples due to increased variance and risk of overfitting [9,10], and their black-box nature may create barriers to clinical adoption and regulatory clearance under FDA Digital Health frameworks [33]. Post-hoc interpretability tools such as SHapley Additive exPlanations (SHAP) can quantify feature-level contributions to individual predictions and may partially mitigate these transparency barriers, providing model-agnostic explanations compatible with both ensemble and penalized regression methods.

Deep learning architectures, including recurrent neural networks (RNNs) and transformer-based models, are well suited to the temporal structure of NICU IoT streams in principle because they can model long-range dependencies within a physiologic time series that summary statistics discard. In practice, however, their performance depends on the availability of large training datasets [34], and they may be vulnerable to overfitting in smaller samples with sparse outcomes. Their appropriate role in NICU DT development is likely in transfer learning pipelines: pre-training on large unlabeled physiologic or EHR corpora to learn generalizable representations, followed by fine-tuning on labeled clinical outcomes, has shown improved performance, particularly in low-label settings [35,36], with additional regularization often required when adapting models to smaller NICU cohorts [37]. This architecture separates the data-volume requirement of representational learning (the unsupervised extraction of informative features from unlabeled input data) from the labeled-outcome requirement of clinical prediction, and is among the more promising near-term directions for the field.

3.4. Handling Temporal Dependence and Class Imbalance

Two structural features of NICU data require explicit methodological accommodation that generic machine learning pipelines do not provide by default. The first is temporal dependence: repeated observations within the same infant are serially correlated, violating the independence assumption underlying most standard learning algorithms. Ignoring this structure does not merely reduce efficiency, it invalidates standard cross-validation procedures (addressed in Section 4) and can produce spuriously confident predictions by treating autocorrelated observations as independent evidence. State-space models explicitly model within-infant temporal structure and are preferable to static snapshot approaches for monitoring DT construction [38]. Complementary approaches such as changepoint detection and mixed-effects extensions of penalized regression provide alternative frameworks for capturing temporal heterogeneity and clustering.

The second is class imbalance. For rare outcomes such as NEC (~9% incidence) or severe IVH (~14% incidence), a naive classifier that predicts the majority class for all observations achieves high accuracy but no clinical utility. Standard approaches to handling imbalance include cost-sensitive learning, which assigns higher misclassification penalties to the minority class during training; resampling methods such as SMOTE (Synthetic Minority Over-Sampling Technique), which generates synthetic minority-class observations to rebalance the training set; and threshold adjustment, which moves the classification decision boundary away from 0.5 toward a value that optimizes clinically relevant performance metrics such as sensitivity at a fixed specificity. Each approach involves tradeoffs, and the optimal strategy is outcome-specific and sample-size-dependent [39]. Importantly, any resampling of the training data to address imbalance must occur inside the cross-validation loop: applying SMOTE before splitting into folds leaks information from the validation set into the training set, producing optimistically biased performance estimates of exactly the type that makes clinical prediction models unreliable [19]. Van Calster et al. [30] document this and related cross-validation failures, including failure to freeze the fitted model before applying it to the test set, and failure to repeat all preprocessing steps within each cross-validation fold, as among the most consequential sources of inflated apparent performance in published prediction models.

4. Cross-Validation and Model Evaluation in the NICU DT Context

4.1. Why Standard Cross-Validation Fails for NICU Data

Cross-validation is the principal tool for estimating how well a supervised learning model will perform on new, unseen data, and for selecting among competing models during development. In standard k-fold cross-validation, the dataset is partitioned into k non-overlapping folds; each fold serves in turn as the validation set while the remaining k−1 folds form the training set, and performance estimates are averaged across folds. This procedure produces approximately unbiased performance estimates when observations are independent and identically distributed (i.i.d.), an assumption that is violated in fundamental ways by NICU IoT data.

Repeated physiologic measurements within the same infant are serially correlated: tomorrow’s heart rate variability is predicted by today’s, which was predicted by yesterday’s. When temporally adjacent observations are split arbitrarily between training and validation folds, the validation set is not truly independent of the training data, as its observations remain statistically dependent on nearby training observations, so the estimated performance will be optimistically biased. This is the same phenomenon that inflates apparent performance in time-series forecasting when standard CV is naively applied [40], and it is directly relevant to monitoring DT development, where the input features are temporal summaries of physiologic streams.

Multicenter NICU data are clustered by infant and by center. Observations from the same infant share unmeasured individual characteristics (e.g., genetic background, antenatal exposures, institutional care practices) that make them more similar to each other than to observations from different infants. When a standard k-fold split places observations from the same infant in both the training and validation folds, the model is effectively evaluated on patients it has already seen in a different guise. In a multi-center dataset, the same problem scales up: a model trained and validated within a single institution has not been tested on the transportability that clinical deployment requires. Standard cross-validation is blind to both levels of clustering.

That this concern is more than theoretical is illustrated by recent NICU prediction practice. In a modeling competition using daily vital sign data from approximately 6,000 NICU admissions [41], only one of five participating teams explicitly accounted for repeated-measures clustering during model evaluation, via leave-one-patient-out cross-validation. The model submitted by that team, a logistic regression with nonlinear terms and engineered heart rate variability features, achieved the highest area under the receiver operating characteristic curve on the held-out test data, outperforming submissions from the other four teams that used neural network, random forest, CatBoost, and XGBoost methods and applied standard k-fold cross-validation without patient-level partitioning. Comparable ambiguity regarding fold assignment is observed in published NICU prediction studies that apply k-fold CV to repeated daily observations without specifying whether folds are at the patient or observation level [42,43]. The methodological gap between cluster-aware best practice and routine analytic implementation in the neonatal prediction literature is therefore the rule rather than the exception, and is a direct risk for DT development efforts that draw on the same data structures and modeling traditions. The magnitude of optimism produced by observation-level cross-validation in this data structure has been quantified across a factorial parameter space in a recent simulation study by the present author [17], on which the cluster-aware validation recommendations developed in this article are grounded.

Two distinct leakage axes therefore threaten validity in the monitoring DT setting, and they should be recognized as separate problems. The first is cluster leakage: observations from the same infant or center appearing in both training and validation partitions, addressed by cluster-aware partitioning as described in Section 4.2. The second is temporal look-ahead leakage: the use of predictor values recorded after the prediction time origin, which gives the model access to information that would not be available at the moment a prediction is made in deployment. A monitoring DT, which by definition generates predictions as physiologic data accumulate, is structurally susceptible to this second form of leakage, and a complementary form arises when modalities include outcome-correlated but non-causal signals, such as post-diagnostic laboratory values, post-treatment imaging reports, or clinical notes referencing the outcome of interest [44]. Both forms are mitigated by anchoring all features to a defined prediction time origin, by careful exclusion of post-diagnostic variables, and, where temporal autocorrelation is a concern, by the temporally ordered cross-validation designs described below. The cluster-aware and temporally ordered strategies are complementary: each addresses one leakage axis, and a monitoring DT validation pipeline must address both.

4.2. Appropriate Strategies for NICU DT Validation

Several cross-validation strategies offer progressively stronger protection against these sources of bias, at progressively higher computational and data-volume cost (Table 1). The strategies described here are not specific to neonatal data; they apply to any prediction model built on repeated-measures or clustered observations, and the NICU is used as the worked example. Standard naïve k-fold CV, as described above, is appropriate only for internal hyperparameter tuning on i.i.d. subsets, and should not be used to report final model performance for NICU DT models. Leave-one-out CV (LOOCV), the special limiting case of k-fold with k equal to the sample size, produces low-variance but often optimistically biased error estimates in small samples, and its discrete misclassification rate metric creates frequent ties that make it poorly suited to comparing competing classifiers [8]. Neither method adequately addresses the temporal or clustering structure of NICU data.

Subject-level k-fold CV addresses cluster leakage by assigning all observations from any single infant to the same fold, so that no infant contributes observations to both the training and validation partitions. This is the minimum cluster-aware standard for any prediction model developed from repeated within-subject measurements with a subject-level outcome. Leave-one-cluster-out CV, in which one cluster (an infant, or a center) is fully withheld at each iteration, is the limiting case of cluster-aware partitioning and provides the strongest protection against cluster leakage; leave-one-center-out CV (LOCO-CV) is its center-level form.

Blocked time-series CV addresses temporal autocorrelation by enforcing a temporal ordering constraint: training data always precede validation data in calendar time, and a gap may be inserted between the training and validation windows to prevent leakage of short-range autocorrelation. This approach correctly simulates prospective deployment: a model trained on NICU data from 2018–2021 is evaluated on 2022–2023 data, as it would be in practice. A refinement, bootstrap time-series CV, applies this blocked structure across many bootstrap resamples of temporally contiguous blocks rather than a single train-test partition, reducing the variance of the performance estimate without sacrificing temporal ordering. This approach is particularly valuable for rare outcomes, where a single blocked split may by chance assign most positive cases to either the training or validation window, producing unstable estimates. Both variants are preferred over standard CV for longitudinal NICU outcome models.

Leave-one-center-out CV is the most stringent strategy and the one most relevant to multi-site DT validation. Each center is held out in turn as the validation set, with the remaining centers comprising the training set, and performance is reported as the average across held-out centers. This procedure directly estimates the transportability of the model to a new institution, precisely the property that determines whether a NICU DT trained at an academic center with a 95th-percentile VLBW volume will perform adequately at a community hospital. LOCO-CV, or an equivalent multi-center hold-out design, is proposed as the minimum standard for reporting final performance of any NICU DT model intended for multi-site deployment. Its computational cost is modest relative to bootstrap methods (it requires exactly C model fits for C centers) and its interpretability is high. While computational cost was once a practical barrier for iterative model fitting in large networks, penalized regression models of the type recommended in Section 3.3 fit in seconds on modern hardware, making LOCO-CV computationally accessible even for NICU research networks with C = 20–30 participating centers.

A recent factorial simulation study of cross-validation strategies for prediction models with repeated-measures predictors and a subject-level binary outcome found that naive observation-level cross-validation can substantially overestimate model discrimination, with optimism in the area under the receiver operating characteristic (AUROC) curve ranging from +0.039 to +0.204 across 162 conditions. The magnitude of this bias increased monotonically with higher intraclass correlation, stronger temporal autocorrelation, fewer clusters, and lower event rates [17]. Subject-level k-fold cross-validation, in which all observations from any single infant are assigned to the same fold, produced AUROC estimates approximately concordant with leave-one-subject-out cross-validation across the conditions examined and is therefore recommended as the minimum cluster-aware standard for any NICU prediction model developed from repeated within-subject measurements [17]. To facilitate adoption of these strategies in neonatal prediction workflows, an open-source R package (cawCV) implementing the naive observation-level, subject-level, and leave-one-cluster-out cross-validation strategies described above is under development by the author [55]. A recent systematic review of 52 multimodal machine learning studies in predictive healthcare found that only 12% (6 studies) performed true external validation on data from a different institution, time period, or acquisition setting [44], confirming that the gap between methodologically rigorous internal validation and demonstrated cross-site transportability is the dominant failure mode of contemporary clinical prediction modeling rather than an edge case.

4.3. Performance Metrics Beyond Discrimination

AUROC has become the default performance metric for clinical prediction models, and it carries genuine meaning: it estimates the probability that a randomly selected positive subject receives a higher predicted probability than a randomly selected negative subject. For rare NICU outcomes, however, AUROC can be artificially high: a model that assigns modestly elevated probabilities to a small minority of true positives while correctly classifying the vast majority of true negatives can achieve an AUROC above 0.80 while providing limited clinical utility. The area under the precision-recall curve (AUPRC) is more informative for imbalanced outcomes and should be reported alongside AUROC for rare NICU endpoints [45].

Calibration, the agreement between predicted probabilities and observed event rates, is often the more clinically relevant metric for monitoring DT applications, and the one most frequently neglected [11]. A monitoring DT that is poorly calibrated will systematically over- or under-estimate event risk, degrading every downstream clinical decision that depends on its predicted probabilities. Calibration assessment should follow the framework recommended by Riley et al. [15] and Van Calster et al. [11]: calibration-in-the-large (mean predicted probability versus observed event rate), calibration slope (ideally 1.0; slopes less than 1.0 indicate overfitting), and a flexible calibration curve across the range of predicted probabilities, reported graphically. Van Calster et al. [30] clarify that strong calibration, which requires correct risk estimates within every possible covariate pattern subgroup, is an unachievable standard in practice. The realistic and appropriate target is moderate calibration, in which among all individuals assigned a predicted risk of X%, the observed event proportion approximates X%, evaluated via a flexible calibration plot. NICU DT models should be held to this achievable moderate calibration standard rather than an unachievable ideal.

Clinical utility assessment, currently absent from most NICU prediction model reports, should accompany discrimination and calibration metrics. Decision curve analysis (DCA), which evaluates net clinical benefit across a range of clinically plausible decision thresholds, provides an early approximation of potential clinical utility applicable to standard validation data, requiring no additional data collection [30]. DCA is appropriate for any NICU DT risk model for which a clinical decision threshold can be specified, and its inclusion in validation reports is recommended as a practical minimum for utility assessment. DCA provides an estimate of potential net benefit under assumed threshold preferences and does not substitute for prospective evaluation of the clinical intervention triggered by model-derived risk scores; definitive assessment of clinical impact requires a separate outcome study.

4.4. From Prediction to Action: Decision Thresholds and Clinical Deployment

A monitoring DT that outputs predicted probabilities must also specify how those probabilities translate into clinical action. This requires a decision threshold, a cutoff above which a prediction triggers an alert, initiates a monitoring escalation, or prompts clinical review. In standard binary classification, a default threshold of 0.5 is rarely clinically appropriate, particularly for rare adverse outcomes. For conditions such as NEC or severe IVH, the clinical cost of a missed case typically outweighs the burden of an unnecessary alert, justifying a lower threshold that improves sensitivity at an acceptable cost to specificity. The DCA framework introduced in Section 4.3 provides a principled basis for threshold selection: net benefit curves evaluated across clinically plausible decision thresholds make explicit the tradeoffs between false positives and false negatives and identify thresholds at which a model provides clinical utility relative to treating all or none [30]. Thresholds should be specified a priori and clinically justified, not derived post hoc from validation data alone.

The NICU environment poses a specific hazard for DT-derived risk alerts: alarm fatigue. Bedside monitoring systems in intensive care units are well documented to generate frequent alarms, the majority of which are clinically non-actionable, contributing to alarm fatigue among nursing staff. A poorly calibrated or over-triggered DT risk model compounds this problem, paradoxically reducing patient safety by degrading the signal-to-noise ratio of the clinical alert stream. Calibration quality directly determines alarm burden: a model that systematically overestimates risk will generate alerts far exceeding its true-positive rate, consuming nursing attention without clinical benefit. The moderate calibration target described in Section 4.3 is therefore not merely a statistical ideal but a patient safety requirement in the NICU monitoring context. Threshold selection should be evaluated prospectively against measured alarm rates, not only against held-out predictive performance, before any DT-derived alert system is deployed in clinical care.

The translation from prediction to automated action also determines a DT’s regulatory classification under FDA Digital Health frameworks. A model that generates a risk probability for clinician review, without directly initiating treatment or triggering a protocol, occupies a lower regulatory risk tier than one whose outputs directly constrain clinical decisions or activate bedside device responses [33]. Regulatory oversight for AI/ML-based software increases with the level of clinical risk and intended use, with higher-risk applications requiring more rigorous premarket evaluation and ongoing post-market performance monitoring under the FDA’s risk-based Software as a Medical Device (SaMD) framework [33]. These regulatory considerations argue for a staged deployment strategy in which NICU DT risk models are first evaluated in passive-display mode, with alarm thresholds and clinical workflows developed and validated before any automated intervention capacity is introduced. The pipeline for responsible deployment, from model development through threshold validation to regulatory clearance, mirrors the framework described in Section 8.1 and should be treated as an integral component of DT development from the outset.

4.5. The Relation to Clinical Prediction Model Standards

The cross-validation and evaluation framework described here is aligned with the standards established for clinical prediction models by the TRIPOD+AI statement [19] and the Riley et al. sample size framework [15]. This alignment is intentional: a neonatal DT whose predictive component cannot meet TRIPOD+AI reporting standards is a DT whose methodology cannot be adequately scrutinized by peer reviewers, replicated by other investigators, or assessed for bias by regulatory bodies. The DT-IoT engineering literature has largely developed its own evaluation conventions, which emphasize reconstruction error and simulation fidelity metrics appropriate for physical systems but do not map cleanly onto clinical prediction validity. Adopting clinical prediction model standards as the methodological floor for NICU DT evaluation is not a constraint, it is the minimum condition for the field to produce evidence that clinical and regulatory audiences will find credible.

5. Synthetic Data and the Simulated DT: A Note on Scope

The methodology described in this review concerns the monitoring DT and its factual prediction task. A distinct line of work concerns the simulated DT and the generation of synthetic patient data, for example to augment underpowered randomized controlled trials with synthetic control arms, a possibility highlighted as a long-term motivation for neonatal DTs by Pammi et al. [6]. That application is acknowledged here as scientifically important but is outside the scope of the present review, because it raises a separate set of methodological problems: generative modeling of clustered, imbalanced physiologic data; counterfactual identification and estimation [26]; and the evaluation of whether synthetic data are fit for a given purpose. Synthetic data generated for trial augmentation must be evaluated for fidelity before use, and a tiered evaluation, progressing from marginal distributional fidelity, to joint and temporal dependency fidelity, to predictive fidelity assessed by the train-on-synthetic, test-on-real paradigm [46], is a reasonable structure for such evaluation. It is worth noting that the most stringent tier, predictive fidelity, is itself subject to the cluster-aware validation requirements described in Section 4: a train-on-synthetic, test-on-real evaluation conducted on repeated-measures neonatal data must partition the real hold-out at the infant level, because observation-level partitioning would inflate apparent predictive fidelity through exactly the optimism quantified in [17] and could allow a synthetic cohort to pass a predictive-fidelity check spuriously. The development of validated fidelity criteria for synthetic neonatal data, and of counterfactual validation methods for simulated DTs, is identified in Section 8 as a priority direction for future work but is not developed further here.

6. Privacy, Security, and IoT Architecture Considerations

The IoT data pipeline that feeds a neonatal DT, from bedside device through integration engine to model training infrastructure, is governed at every stage by the Health Insurance Portability and Accountability Act (HIPAA) and its implementing regulations, which classify continuously recorded physiologic data linked to a patient identifier as protected health information regardless of whether explicit identifiers are present [47]. This compliance obligation extends beyond the EHR to include waveform archives, ventilator data logs, and any derived feature dataset that retains sufficient granularity to permit re-identification. The re-identification risk is particularly acute in neonatal populations: combinations of quasi-identifiers can uniquely identify individuals in large populations [48], and this risk is amplified in high-dimensional clinical datasets where rare combinations of characteristics are more common [49]. In single-center NICU cohorts, variables such as gestational age, birthweight, institution, and admission timing may therefore render individuals effectively identifiable, necessitating suppression or generalization beyond standard de-identification approaches. Expert Determination de-identification, which requires a statistical demonstration that re-identification risk falls below an acceptable threshold, is the appropriate standard for NICU DT development and should be documented and reported as part of any published methodology.

Federated learning offers an architectural solution to the tension between the multi-center data requirements of NICU DT development and the privacy constraints on centralizing patient data [50]. In a federated architecture, model parameters rather than raw patient data are transmitted between participating institutions and a central aggregator: each center trains on its own data locally, only model updates are shared, and the central model improves through aggregation across sites without any center’s patient records leaving its institutional boundary. This approach maps naturally onto the structure of national neonatal research networks, where data governance frameworks for collaborative research already exist [51,52]. Federated learning is not without limitations in the NICU context, as statistical heterogeneity across centers can cause a federated model trained by simple gradient averaging to underperform a centrally trained model on any individual center’s population, a problem known as client drift [53]. Edge computing, processing physiologic streams locally at the bedside to generate feature summaries before any data leave the device, provides a complementary layer of privacy protection for monitoring DT applications, though it introduces device security and software validation challenges that fall under FDA medical device regulations [33].

7. Ethical and Regulatory Considerations

Neonatal DTs that support clinical decision-making fall within the scope of the FDA’s Digital Health Center of Excellence and its guidance on artificial intelligence and machine learning-enabled medical devices [33]. A DT whose output directly informs treatment decisions would likely require premarket submission as Software as a Medical Device, with associated requirements for algorithmic transparency, performance reporting, and post-market surveillance [33]. Neonatal research conducted without the capacity for direct patient consent has always depended on parental proxy authorization, and DT development raises new questions about the scope of that proxy: parents who consent to their infant’s clinical data being used for research may not have contemplated that consent as authorizing the construction of a computational model of their child. Pammi et al. [6] highlight the absence of formal consent guidance for pediatric digital health technologies as a key unresolved challenge, and this review concurs that the development of such frameworks is a genuine ethical prerequisite rather than a downstream implementation detail.

Equity concerns are equally pressing. DTs trained predominantly on data from large academic NICUs will encode the care patterns, population characteristics, and outcome distributions of those institutions, and may perform substantially worse when deployed at community hospitals serving different demographic populations. Prospective evaluation of DT performance across institutional types and patient demographic subgroups, with mandatory subgroup reporting analogous to that now expected in clinical trial publications, should be a condition of any regulatory submission or high-profile publication. This concern is not unique to neonatal applications; cross-specialty reviews of DT methodology in clinical research have similarly identified generalizability limitations as a primary equity risk when DTs are constructed from non-representative source populations, noting that rigorous validation techniques and data augmentation strategies are necessary to mitigate these constraints [54].

8. A Proposed Framework and Research Agenda

8.1. The NICU DT Validation Framework (NICU-DTVF)

The methodological considerations developed across Section 3 through 7 converge on a structured framework for responsible NICU DT development, the NICU DT Validation Framework (NICU-DTVF). The NICU-DTVF is best understood as a set of validation conditions that any trustworthy healthcare monitoring DT must satisfy, instantiated here for the neonatal repeated-measures setting. Cluster-aware cross-validation, implemented in the cawCV R package, is one component that addresses one of those conditions. As shown in Figure 1, key methodological requirements apply at distinct transitions in the DT pipeline. NICU-DTVF has four operative principles. First, feature selection must be performed within the cross-validation loop and evaluated for stability across bootstrap resamples, with stability metrics reported alongside model performance. Second, the cross-validation strategy must match the data structure: subject-level k-fold CV serves as the minimum cluster-aware standard, addressing optimism attributable to within-subject dependence [17]; blocked or bootstrap time-series CV is recommended additionally when temporal autocorrelation constitutes a distinct source of leakage in longitudinal single-center models; and leave-one-center-out CV is the standard for any model intended for multi-site deployment, with both the clustering and temporal leakage axes addressed. Third, performance reporting must include calibration assessment, calibration slope, calibration-in-the-large, and a flexible calibration curve, not discrimination alone. Fourth, decision thresholds and any automated alerting must be a priori specified and evaluated against threshold-level clinical performance, operating characteristics, and alarm burden before clinical deployment.

This framework is deliberately aligned with TRIPOD+AI reporting standards [19] and the Riley et al. prediction model development guidelines [15], extending both to the DT context. Alignment with existing standards rather than invention of new ones is itself a methodological choice: the credibility of neonatal DT evidence will depend on whether it meets the evaluative standards that clinical and regulatory audiences already apply to prediction models. A DT that is evaluated by bespoke metrics intelligible only to the IoT engineering community will not influence clinical practice or regulatory policy.

8.2. Prioritized Research Agenda

Against the framework above, the most consequential open methodological problems can be prioritized by three criteria: urgency (the degree to which the gap blocks current DT development), feasibility (the degree to which the problem is tractable without new data collection or additional infrastructure), and potential impact (the expected magnitude of improvement in DT validity or clinical trustworthiness if the problem were resolved). Figure 3 maps prioritized research agenda items across these dimensions, with urgency on the vertical axis, feasibility on the horizontal axis, and potential impact represented by bubble size. The nine problems shown represent the principal open methodological, infrastructural, and ethical challenges identified across Section 3 through 7, grouped into three domains (machine learning methodology, infrastructure and regulatory, and ethics and equity). This set reflects the problems judged most consequential within the scope of the present review and is not intended to be exhaustive. Three machine learning methodology items emerge as high urgency and high feasibility, the appropriate starting point for the field. Establishing temporal and clustered CV standards for NICU outcome models requires no new data collection, only methodological comparison studies using existing multi-center cohorts such as those maintained by the NICHD Neonatal Research Network or the Canadian Neonatal Network. Developing stability assessment norms for feature selection in the NICU sample-size regime is similarly tractable using existing datasets and simulation methods. Adapting class-imbalance handling methods to NICU-specific outcome rates, with explicit evaluation of the requirement that resampling occur inside the cross-validation loop, can be accomplished within current single-center data environments.

Items that are high urgency but lower feasibility require coordinated multi-center infrastructure that does not yet exist at scale, most notably the establishment of federated learning pipelines across NICU networks. Further items, including parental consent frameworks for DT use, equity and subgroup reporting norms, and the development of validated fidelity criteria and counterfactual validation methods for simulated DTs and synthetic data, occupy a moderate-urgency zone and will become more tractable as the higher-priority methodological problems are addressed. It is recommended that the research community treat the cross-validation standards, feature selection stability, and class-imbalance handling problems as the near-term critical path, since progress on all three is prerequisite to credible NICU DT prediction models capable of supporting clinical deployment. The placement of each problem along these three dimensions reflects a qualitative synthesis of the analysis developed in this review rather than a formal scoring or consensus exercise. The exact position of any individual problem is accordingly illustrative, and a different investigator could reasonably place individual items somewhat differently. The broad quadrant assignments, by contrast, follow from the methodological arguments developed in the preceding sections, and it is the quadrant-level prioritization, rather than the precise coordinates, that the figure is intended to convey.

8.3. A Coherent Methodological Program for Repeated-Measures Validation in Clinical Prediction

The validation strategies recommended in this review form part of a broader methodological program for clinical prediction modeling with repeated-measures predictors, in which several adjacent questions remain underdeveloped. Quantification of the optimism produced by naive cross-validation in this data structure has recently been addressed by a factorial simulation study by the present author [17], on which the cluster-aware validation recommendations in Section 4 are constructed. Implementation of these methods in open-source software (the cawCV R package [55]) is in active development. Several adjacent methodological questions are not addressed in the present review and warrant further investigation, including the behavior of calibration measures and the Brier score under within-subject dependence; sample-size determination for prediction models with repeated-measures predictors and a subject-level binary outcome; the effect of heterogeneity in within-subject correlation structure across predictors on validation performance; and the extension of cluster-aware validation to dynamic, time-updated prediction, in which a prediction is refreshed at successive landmark times as predictor data accumulate. The present review and the simulation study [17] on which it draws address the case of a single prediction per subject using the full observed predictor history; extension to dynamic, landmark-based prediction is one natural direction for future work. The cawCV package is intended to provide an extensible foundation for cluster-aware validation, to which methods addressing these questions may be added as they are developed.

These questions share a common structural feature: each concerns the translation of a standard prediction modeling tool, designed under the assumption of independent observations, to a setting in which observations are correlated within subjects. The cluster-aware cross-validation work [17] addresses this translation for the partitioning step of model evaluation. Calibration, sample size, dependence-structure heterogeneity, and dynamic prediction remain open analogues. Progress on these questions is a prerequisite for the kind of credentialed, transparently reported, and reproducibly validated NICU DT models that the framework described in Section 8.1 envisions, and for which TRIPOD+AI [19] and the Riley et al. sample-size framework [15] currently provide cross-sectional analogues without explicit guidance for the repeated-measures setting.

The convergence between this methodological program and the NICU DT validation framework described above is intentional. A neonatal DT cannot be validated more rigorously than the prediction models on which it is built, and prediction models that ignore the cluster structure of NICU IoT data will continue to overstate their generalization performance until cluster-aware validation methods become standard practice. The aim of this review is to make that convergence visible, and to provide an explicit pathway by which methodological development in the validation of repeated-measures clinical prediction models can be coordinated with the clinical and translational ambitions of the neonatal DT vision. While that pathway is developed here in the neonatal setting, the validation methodology itself is general and transfers to any clinical prediction problem in which predictors are measured repeatedly within clustered units.

9. Conclusions

The neonatal intensive care unit is a uniquely data-rich IoT environment whose clinical stakes are among the highest in the healthcare system, because the outcomes determined there, from mortality to lifelong morbidity in the most vulnerable patients in medicine, are realized across an entire lifespan, so that the potential health gains, measured for example in quality-adjusted life-years, are correspondingly large. The vision of monitoring digital twins that predict clinical events from continuously acquired physiologic data is compelling, scientifically grounded, and potentially transformative for a field constrained by the difficulty of acting on adverse outcomes before they become clinically manifest. What has been missing is not motivation but methodology.

This review has identified the specific methodological gaps that separate the monitoring DT vision from responsible clinical implementation: cross-validation approaches that violate the cluster and temporal structure of NICU IoT streams, feature selection instability under small effective sample sizes, and the relative neglect of calibration in model evaluation. For each gap, tractable solutions exist, drawn from the clinical prediction model literature and the time-series analysis literature, that have not yet been systematically applied in the neonatal DT context. The framework and research agenda proposed here are intended to provide a structured path from the current state to one in which neonatal DTs can be built with methodological rigor, evaluated against standards that clinical and regulatory audiences recognize, and deployed with equitable performance across the institutional diversity of neonatal care. The cluster-aware cross-validation methodology that anchors the validation component of NICU-DTVF [17], implemented in the open-source cawCV R package [55], together with the open methodological questions described in Section 8.3, comprises a coherent program of work on the validation of clinical prediction models with repeated-measures predictors, to which the present review contributes a conceptual synthesis and a framework for translation to the neonatal DT context.

The limiting factor in realizing this vision is not computational capacity but methodological clarity and cross-disciplinary coordination between the neonatal clinical community, the biostatistics community, and the IoT and machine learning engineering community. This review is intended to contribute to that coordination.