Validation of the NICHD Bronchopulmonary Dysplasia Outcome Estimator 2022 in a Quaternary Canadian NICU – A Single Centre Observational Study

1. Introduction

Bronchopulmonary dysplasia (BPD) is a multifactorial disease affecting preterm infants, especially those who are born less than 29 weeks [1,2]. The incidence of BPD across Canadian centers varies between centers with a higher incidence in lower gestational age groups [2,3]. Infants with moderate-to-severe BPD are prone to developing chronic pulmonary hypertension (cPH), systemic hypertension, ventricular hypertrophy, and pulmonary vein stenosis (PVS) [4,5], and requiring prolonged ventilation, home extended hospital stays and home respiratory support [6]. After neonatal intensive care unit (NICU) discharge, these neonates face an increased risk of hospital readmissions within the first year, recurrent pulmonary infections, abnormal lung function [7,8,9], motor and cognitive impairments, behavioral and mental health challenges extending into childhood and increased healthcare costs [10,11,12,13,14]. Clinicians use corticosteroids in preterm neonates to mitigate the inflammatory processes central to the pathogenesis of BPD [15]. However, the short-term benefits gained by steroids are often mitigated by long-term neurodevelopmental impairment (NDI) [16]. Current literature suggests a risk-based approach to starting steroids, recommending treatment only for infants with a high risk of developing BPD [17,18,19,20,21]. This risk-based strategy aims to protect low-risk infants from potential harm because of steroid exposure, while ensuring that high-risk infants receive beneficial steroid therapy. However, implementing this approach in clinical practice is difficult because assessing the risk of BPD is highly subjective and varies significantly among clinicians. This is evident from the variations in steroid use for BPD observed both within and between Canadian neonatal networks (CNN) centers [2,22].

The National Institute of Child Health and Human Development (NICHD) BPD outcome estimator 2022 is a web-based calculator that uses extensive population data from the Neonatal Research Network (NRN) to predict the likelihood of BPD or death at 36 weeks [23]. This tool aims to assist clinicians in making informed decisions early in the infant's life. Although the tool is superior to the 2011 BPD estimator because of its sample size, contemporary cohort, and use of evidence-based definitions [23,24], Canadian NICUs have not validated it. Infants in Canadian NICUs were not part of the development cohort and may differ in disease severity, population diversity, and exposure to healthcare practices, and resource availability. The NICHD BPD outcome estimator 2022 uses the Jensen 2019 definition of BPD (based on a respiratory support at 36 weeks), while the Canadian Neonatal Network (CNN) defines BPD based on both oxygen requirements and respiratory support at 36 weeks [2,25]. Locally, our quality improvement (QI) team believed BPD risk estimation during the first four weeks of life might facilitate clinicians and parents’ engagement in bedside risks versus benefits conversations on care interventions such as steroids use. Before adopting the calculator, and to ensure providers’ acceptability, we wanted to validate the NICHD’s BPD outcome estimator 2022 generalizability to our local infants. We hypothesized that NICHD BPD outcome estimator 2022 would accurately predict death and/or moderate-to-severe BPD. Such accurate prediction will facilitate clinicians in discussing initiation of interventions to prevent or alter the course of BPD during the first 28 days of life and in family counselling. This study aimed to validate the NICHD’s BPD outcome estimator 2022 in predicting death or Grade 2 or 3 BPD at 36 weeks with observed outcomes of death and/ or moderate-to-severe BPD in infants born less than 29 weeks and admitted to British Columbia Women’s Hospital (BCWH).

2. Materials and Methods

This single-centre retrospective observational study took place at BCWH, a quaternary-perinatal centre in Canada. We included preterm inborn and outborn infants born at less than 29 weeks gestational age who were admitted to the NICU between March 2022 and August 2023. We excluded infants who had major congenital anomalies and those transferred to other units before reaching 36 weeks postmenstrual age (PMA). The NICU admission register was initially used to identify infants eligible for the study. Demographic and respiratory support data necessary for inputting into NICHD BPD outcome estimator 2022 tool were collected from hospital’s electronic health records (EHR). The CNN database, housed at BCWH, provided BPD, mortality, and resource utilization data. We estimated the risk of BPD (Grade 1, 2 and 3) and mortality using the NICHD BPD outcome estimator 2022 for the days 1, 3, 7, 14, and 28. Whenever there was a doubt in ascertainment of BPD in the CNN database, e.g. nasal canula flow rate, we reviewed the infant EHR to ensure accuracy in capturing observed outcomes. There were no major practice changes to alter BPD during the study period. We used oxygen saturation targets of 88 to 92%, minimally invasive surfactant treatment, and approved postnatal steroids such as systemic hydrocortisone to prevent BPD (STOP-BPD) [26], The Dexamethasone- A randomized trial (DART) [27], and inhaled budesonide regimens [28,29]. No postnatal steroid guidelines existed, and care teams decided by consensus.

We used the definitions of NICHD BPD outcome estimator 2022 while interpreting the output i.e. at 36 weeks PMA, Grade 1 BPD-if requires 2 litres/minute nasal cannula or less, Grade 2 if requires more than 2 litres/minute nasal cannula or other forms of non-invasive ventilation support, and Grade 3 if requires invasive mechanical ventilation [23,25]. We used the observed severity of BPD using CNN database definitions; At 36 weeks PMA, Mild BPD if receives nasal canula with 100% oxygen of < 100ml/minute or 21-99% blended air/oxygen with flow rate of 1.5 litres/minute or less; Moderate BPD if receives nasal canula of 21-29% blended air/oxygen of 1.5 litres/minute or more, or 21-29% blended air/oxygen with continuous positive airway pressure (CPAP), synchronized positive airway pressure (SIPAP), non-invasive positive pressure ventilation (NIPPV) or non-invasive high frequency ventilation (NIHFV); Severe BPD if 30% or more oxygen with nasal canula of > 1.5 litres/minute or 30% of more of oxygen with CPAP, SIPAP, NIPPV, NIHFV or intubation and ventilation [30]. To facilitate analysis of observed versus estimated outcomes, we classified infants into risk categories at time points day 1, 3, 7, 14 and 28, with estimated risk of <10, 10-19, 20-29, 30-39, 40-49 and ≥50% from the NICHD BPD outcome estimator 2022. Next, we calculated the number of infants with the observed outcomes. We compared the estimator's predicted probabilities with the observed outcomes. The unavailability of pre-study EHR data at our center hindered the collection of NICHD’s BPD outcome estimator 2022 input variables, so we chose a convenience sample size. There was limited time available for validation prior to initiation of BPD prevention related QI interventions locally. We used descriptive statistics to summarize baseline characteristics and NICU interventions. For infants who died before reaching 36 weeks PMA, and outborn infants with missing data for specific days before their admission to the NICU, we included them in the analysis on time points when their data was available. We generated Receiver Operating Characteristic (ROC) curves to evaluate the predictive validity of the NICHD BPD outcome estimator 2022 at each time point. Using SPSS (Released 2023. IBM SPSS Statistics for Macintosh, Version 29.0.2.0 Armonk, NY: IBM Corp.), we calculated the Area Under the Curve (AUC) values. We selected an AUC greater than 0.7 for each ROC curve as the minimal acceptable predictive accuracy for local adoption of the BPD estimator tool [31]. This threshold denotes significant predictive accuracy. The hospital’s research ethics board approved the study (H24-00785).

3. Results

3.1. Baseline Characteristics

The study included 99 of the 118 infants who met the inclusion criteria, excluding 19 infants, as shown in Figure 1. On days 1, 3, 7,14, and 28, there were 12,12,10,14 and 17 infants’ data were missing, but included in analysis for those days when data was available. The median (interquartile range [IQR]) gestational age at birth was 26(25-28) weeks, and the median (IQR) birth weight was 914(755-1069) grams. Table 1 presents other baseline characteristics. Table S1 and Table S2 show the interventions received by study infants during their NICU stay and the time these interventions lasted, respectively. Thirteen infants (13.1%) died, and by 36 weeks PMA, 40.4% developed moderate or severe BPD, with 32.3% and 7.1% having moderate and severe BPD, respectively. We show other outcomes at discharge in Table 2.

3.2. Respiratory Support

The median fractional inspired oxygen (FiO₂) levels ranged from 21% to 26.5% at various time points over the study period. We summarized other respiratory support details in Table 3. We present the observed and estimated predicted outcomes for BPD in Table 4 and for death or BPD in Table 5. The AUC values for moderate-to-severe BPD prediction on days 1, 3, 7, 14, and 28 were 0.766, 0.746, 0.785, 0.807, and 0.818 respectively (Table 4, and Figure 2). Similarly, the AUC values for death or moderate-to-severe BPD prediction on days 1, 3, 7, 14, and 28 were 0.803, 0.806, 0.837, 0.832, and 0.843, respectively (Table 5, and Figure 3). Comparison of ROC curves across days did not show a statistically significant difference (Table S3).

4. Discussion

In this study, we found that the NICHD BPD outcome estimator 2022 accurately predicted the combined outcome of death and/or moderate-to-severe BPD on Days 1, 3, 7, 14 and 28 of life. The accuracy improved with increasing postnatal age.

Of all the BPD prediction models developed in the past [32], two models—NICHD BPD outcome estimators 2011 and 2022 have been the most popular ones [23,24,32,33]. A major gap in the adoption of BPD prediction models has been the external validation in centres planning to adopt them. To address this gap, we evaluated the performance of NICHD BPD outcome estimator 2022 in a Canadian setting.

External validation of NICHD BPD outcome estimator 2011 accurately predicted death or severe BPD with an AUC of 0.81-0.84 [34], and 0.82 and 0.77 on day 1 and 3 of life [35]. However, the model performed even better for predicting moderate-to-severe BPD alone with an AUC of 0.83 to 0.94 for Days 7 and 28 of life [35]. In above studies, performance improved with increasing postnatal age. Two recent studies have externally evaluated the performance of 2022 BPD estimator and its utility in predicting eventual steroid treatment [36,37]. Srivatsa et al. study found that 2022 BPD estimator overestimated probabilities for babies who develop BPD, had low sensitivity to predict mortality and overestimated postnatal steroid use [36]. Study by Kinkor et al. found that the NICHD BPD outcome estimator 2022 had poor to fair accuracy for predicting death or Grade 3 BPD on day 28 (AUC 0.77) but not at other time points in the first 28 days of life (AUC 0.58-0.67). However, the NICHD BPD outcome estimator 2022 showed good to excellent accuracy for identifying infants at high risk of steroid treatment (AUC 0.76-0.89) on Days 1, 3, 7, 14 and 28 [37]. Above two studies were retrospective, had small sample sizes, included infants admitted prior to 2021, had variable infant demographics, care practices (oxygen saturation targeting, post-natal steroid use) and BPD and mortality rates. Thus, validation in those studies is likely to have limited applicability in another centre planning to adopt the estimator.

Based on our study findings, the NICHD BPD outcome estimator 2022 provides early and reliable prediction of infants at high risk of death and or moderate-to-severe BPD on Days 1, 3, 7, 14 and 28. In our setting, we could use this estimator to consistently identify infants with a numerically high risk of poor outcomes. We have reported increased variation in the timing of initiation of the first course of postnatal steroids [Median (IQR) of 28 (19-38) days] for prevention or treatment of BPD from our centre [22]. Many studies in the past have suggested that earlier initiation of postnatal steroids ranging between 8 to 21 days of life was associated with decreased BPD compared with later initiation [38,39,40,41] . These studies along with this study results, suggest that using NICHD BPD outcome estimator 2022 with its ability to provide a numerical risk of poor outcome might facilitate earlier conversations between clinicians and between clinicians and family on initiating targeted interventions to prevent or alter the course of BPD. We could not validate the utility of NICHD BPD outcome estimator 2022 in identifying infants at high risk of steroid treatment in this study. Since many factors influence the decision to begin steroid treatment, the NICHD BPD outcome estimator 2022 risk estimations can be a helpful adjunct tool for making decisions. However, we are uncertain if risk estimations alone can determine the initiation of treatment or its appropriate timing in a specific context. However, using this tool to trigger bedside conversations seems reasonable without undue risks of unwarranted use or timing of steroids.

To our knowledge, this study is the first one to validate NICHD BPD outcome estimator 2022 in a Canadian setting using a contemporary population. Using a pre-specified threshold value for acceptable performance of the model at our centre is likely to make its use acceptable by clinicians and families. Other strengths of this study include a comprehensive dataset from a quaternary NICU and stringent criteria for outcomes. This validation underscores the model's robustness and adaptability, suggesting that it can effectively support clinical decision-making in different settings, including those with diverse patient populations and care protocols. Our study limitations include single centre with small sample size, retrospective design, exclusion of infants discharged prior to 36 weeks PMA. Although the above limitations may restrict the applicability of our findings, other NICUs could easily replicate our study methods to evaluate 2022 NICHD BPD outcome estimator’s utility in their context.

5. Conclusions

We found that the NICHD BPD outcome estimator 2022 accurately predicted the death and /or moderate-to-severe BPD on Days 1, 3, 7, 14 and 28 of life. These findings suggest clinicians could use it to identify infants at high risk of poor outcomes during the first 28 days, facilitating discussions between clinicians and families about initiating targeted interventions to prevent or alter the course of BPD and engaging families in treatment-related risk versus benefit conversations.

6. Patents

None.