Predictive Analysis of Extubation Failure in the Paediatric Intensive Care Unit in Bloemfontein, South Africa

1. Introduction

Mechanical ventilation (MV) has decreased mortality rates in critically ill patients [1,2]. In the United States, an average of approximately 800 000 patients are mechanically ventilated annually [3]. Despite its benefits, significant complications are associated with mechanical ventilation. These include pneumothorax, ventilator associated pneumonia, ventilator associated lung injury, and extubation failure (EF) [3,4].

Invasive mechanical ventilation should be terminated as soon as the patient is able to maintain spontaneous breathing and reasonable gas exchange [5,6,7]. The process of extubation should be undertaken as soon as the patient can sustain independent respiration [8,9]. Even after a favourable spontaneous breathing trial (SBT), approximately 15% of patients will require re-intubation [6,10]. There are currently no reliable tests to predict extubation readiness in children, the decision to extubate remains largely at the discretion of treating clinicians [5,11]. Liu et al found that forecasting model using medication response and patient characteristics predicted better the extubation readiness [12]. While Moore et al highlighted the importance of clinical assessment [13].

Extubation failure, defined as the need for reintubation within 72 hours following a planned extubation, increases the risk of prolonged ventilation, mortality and prolonged hospital stay [10]. In adult studies, predictors of extubation failure have been investigated, but there is a lack of data in the paediatric population. Zhao et al identified 19 features out of 89 that can predict extubation failure: age, body mass index (BMI), Mean arterial pressure (MAP), respiratory rate, tidal volume, mean airway pressure, spontaneous breathing trials (SBT) and urine output are among the 19 mentioned features [10].

Globally, several models have been developed using machine learning (ML) to predict the outcome of extubation, with the attempt to minimise the risk of re-intubation and its associated complications [5,7,14]. Rooney et al, showed physiologic variables such as peripheral oxygen saturation extremes, dynamic compliance, central venous pressure, and heart/ respiratory rate emerged as key predictors of EF using RF in paediatric cardiac ICU [15]. In this context, the use of ML models, may help to predict the outcome of extubation, by supporting the physician in the decision-making process.

Both Universitas Academic Hospital and Pelonomi Tertiary Hospital in Bloemfontein, South Africa, offer invasive mechanical ventilation in their respective paediatric intensive care units (PICUs). The decision to extubate in these two centres is guided by the physician’s judgement.

There is a need to develop an acceptable tool to predict the outcome of extubation. ML models could be used to achieve this task using clinical and ventilator setting features. The aim will be to develop and validate a model of EF in mechanically ventilated patients in PICU. Machine learning is a subdiscipline of artificial intelligence directed towards specified tasks, in this case - prediction [16]. Machine learning is generally divided into two categories: supervised and unsupervised learning [11,14].

Supervised machine learning uses a certain number of algorithms, of which the most common are decision tree (DT), random forest (RF), gradient boosting (GB), Light gradient boosting model (LightGBM), extreme gradient boosting (XGBoost), logistic regression (LR), and support vector machine (SVM), cat Boost, Adaboost, Multilayer perception (MLP), K-Nearest Neighbor (KNN), NaiveBayes [10].

In unsupervised machine learning model, patterns are inferred from unlabelled data. The objective is to find the structure and the patterns from the input data.

For this study, the focus will be on the supervised machine learning model using the following algorithms: Decision Tree (DT), Artificial Neural Network (ANN), Extreme Gradient Boosting (XGBoost), Random Forest (RF), Support Vector Machine (SVM), and Logistic Regression (LR).

In this study, the researcher aimed to investigate risk factors for EF and develop supervised ML models for the prediction of EF in mechanically ventilated children. 

2. Methods

2.1. Study Design

This is a retrospective cohort study conducted in two paediatric intensive care units (PICUs) in Bloemfontein in the Free State province, South Africa. The study was also a predictive modelling research design using machine learning techniques. Data were collected from January 2018 to December 2022.

2.2. Study Site

The study was conducted in two paediatric intensive care units in Bloemfontein. Pelonomi Tertiary Hospital and Universitas Academic Hospital, both are major healthcare facilities in the Free State province. Pelonomi Tertiary Hospital is a teaching hospital with a five bed PICU, while Universitas Academic Hospital is a quaternary teaching hospital with a five-bed PICU offering specialist and subspecialist paediatric care.

2.3. Study Population

The study included all patients under 13 years of age who were admitted to one of the PICUs and required invasive mechanical ventilation during the study period. Prolonged ventilation was defined as mechanical ventilation of more than 14 days and difficult intubation, more than two unsuccessful attempts of direct laryngoscopy. Participants were eligible for inclusion in the study if they underwent first time attempt of planned extubation. The exclusion criteria were as follow:

-

Patients who self-extubated

-

Patients who were deemed for withdrawal of treatment

-

Patients with congenital upper airway obstruction

-

Patients extubated from high frequency oscillatory ventilation (HFOV)

-

Patients with incomplete medical records

-

Patients who died while still on the ventilator.

2.4. Outcomes and Predictors of Extubation Failure

The primary outcome of interest was extubation failure, which was defined as the need for re-intubation within 48 following a planned extubation.

2.5. Data Collection

Data were retrospectively collected from patient medical records, including admission registers, the Paediatric Index of Mortality 3 (PIM 3) database, and Meditech electronic health records. A standardised data collection form was to ensure consistency. The following variables were collected:

-

Demographic data: age, sex, referral source, weight, height and nutritional status (classified according to WHO criteria)

-

Clinical data: PIM3 mortality risk, diagnosis, comorbidities and details of mechanical ventilation (mode, duration, PIP, PEEP, respiratory rate, FiO2,)

-

Extubation details: timing of extubation, reasons for EF, and outcomes,

-

Laboratory data: relevant hematologic and biochemical markers, including blood gas and septic marker (procalcitonin or C-protein reactive).

All information was captured on a spreadsheet using REDCap

Data were kept confidential by using a coding system. Only the researcher and the supervisor had access to the protected database. All patients included in the study were de-identified in compliance with ethical standards and institutional guidelines.

3. Data Analysis

3.1. Machine Learning Models:

Several machine learning models were developed to predict extubation failure. Among these models we have logistic regression (LR), artificial neural network (ANN), extreme gradient boosting (XGBoost), random forest (RF), support vector machine (SVM), and decision tree (DT). The following steps were taken to develop the model:

-

Data preprocessing: data were cleaned and standardised. Missing data were handled using multiple imputation techniques where needed.

-

Feature selection: features were selected based on clinical relevance and statistical significance. Nine key features, including duration of ventilation, use of neuromuscular blockade, PIM3 mortality risk, nutritional status, respiratory rate, use of vasopressors or inotropes, PEEP, PIP, oxygenation index, were identified for model training.

-

Data were randomly split into training set (70%) and testing se (30%)

-

Models were trained and validated in a 5-fold stratified cross-validation.

-

Performance metrics: the performance of each model was evaluated using area under the curve of receiver operating characteristic (AUCROC), area under the precision-recall curve (AUCPRC), calibration slope, and calibration intercept.

Statistical analysis: data collected were entered into Redcap and analysed using R programming, version 4.4.3 (R foundation for statistical computing, Vienna, Austria) descriptive statistics were used to summarise demographic and clinical characteristics. Categorical variables were presented as frequencies and percentages, while continuous variables were summarised using medians and interquartile ranges (IQRs) or means and standard deviations (SD) as appropriate. Univariate analysis was performed using chi-square or Fisher’s exact tests for categorical variables and Mann-Whitney U or t-tests for continuous variables. Multivariate logistic regression was used to identify independent predictors of extubation failure, with results presented as adjusted odds (aOR) and 95% confidence intervals (CI). A p-value of less than 0.05 (p < 0.05) was considered significant.

Ethical considerations: The study protocol was reviewed and approved by the Health Sciences Research Ethics Committee (HSREC) of the University of the Free State (UFS-HSD2023/1265/2609) and authorization from the national research Health database (NHRD) (FS 202308_022) was obtained. Given the retrospective nature of the study, informed consent was waived.

3.2. Results

Figure 1. Schematic of patients enrolled in the study.

Figure 1. Schematic of patients enrolled in the study.

The study included 824 participants who underwent planned extubation in two PICUs in Bloemfontein, South Africa. The median age of the participants was 9 months (IQR 2-48 months), with 59.3% of the participants being male. A total of 231 participants (28%) experienced extubation failure (Table1). Patients referred from another facility were more prevalent in the failed extubation group (p<0.001) (supplementary table).

Most (80.0%) participants had normal nutritional status. Undernutrition was significantly associated with extubation failure (p=0.029) (Table 1)

The presence of central nervous comorbidity demonstrated a significant difference between the two groups (p = 0.022), while patients without comorbidities were extubated successfully with a significant statistical difference (p < 0.001) (Table 1 and supplementary table)

The presence of central nervous comorbidity demonstrated a significant difference between the two groups (p=0.022), while patients without comorbidities were extubated successfully with a significant statistical difference at p< 0.001(Table 1)

The median days of ventilation were significantly higher in the extubation failure group compared to the successful extubation group respectively with 3 days (IQR 3-3) and 2 days (IQR 2-3), (p<0.001). (Table 1) Upper airway obstruction was the common reason for extubation failure (22.5%) followed by increased work of breathing (19.8%), severe apnoea 3.0% while sepsis and septic shock all together represent only 1.5% (Table 1).

Multivariate logistic regression analysis was used to adjust for potential confounders. The analysis confirmed that longer duration of mechanical ventilation (aOR 4.49, 95% CI 3.24 – 6.57, p< 0.001), Use of neuromuscular blockade (aOR 1.32, 95% CI 1.07 – 1.63, p= 0.009) and need for vasoactive or inotropes during ventilation (aOR 1.57, 95% CI 1.24 – 2.01, p< 0.001) were independent predictors of extubation failure (Table 2).

The decision curve analysis shows that XGBoost, ANN, Random Forest, and SVM models provide the highest net benefit in a threshold probability range of 0.1 - < 0.4. In this range, the models are demonstrating their strong clinical utility in decision-making. Logistic regression has moderate performance while decision trees model underperformed offering limited benefit compared to other models (Figure 1).

Figure 2. Model training and validation.

Figure 2. Model training and validation.

3.2.1. Prediction Model Performance

The study evaluated the predictive performance of six machine learning models: LR, ANN, XGBoost, RF, SVM, and DT. Models used in this study provide several important insights into the predictive performance of different machine learning and statistical models of predicting extubation failure.

3.2.2. Performance Metrics: AUCROC and AUCPRC

Artificial neural network (ANN) and logistic regression (LR) models demonstrated the highest AUCROC values (0.87±0.04 for ANN and 0.86±0.02 for LR) indicating strong discriminative ability. ANN also had the highest AUCPRC value (0.72±0.10), suggesting better performance in the minority class. LR performed well with AUCPRC (0.69 ± 0.06) Extreme gradient boosting (XGBoost) also performed well with an AUCROC of 0.81±0.04 and AUCPRC of 0.61±0.09, making the XGBoost a robust choice for prediction modelling.

3.2.3. Calibration Metrics

Calibration slope and intercept provide insights into how well the predicted probabilities align with actual outcomes. LR had a calibration slope of 1.77 ± 0.32, intercept – 0.26 ± 0.08 suggesting the best calibration closely to the perfect calibration line. ANN with a calibration slope of 1.74 ± 0.39 and intercept of – 0.28 ± 0.12 and SVM slope 1.63 ± 0.41, intercept – 0.26 ± 0.10 also showing good calibration. XGBoost, RF and DT exhibit moderate calibration.

3.2.4. Model Selection

ANN and LR models emerged as top performers in terms of AUCROC and AUCPRC, making them reliable choices for predicting extubation failure in our study. While Decision Tree (DT) and Random Forest (RF) models showed lower AUCROC and AUCPRC values, they may still be valuable in providing interpretable results and understanding feature importance. Support Vector Machine (SVM) and DT models had relatively lower calibration metrics.

4. Discussion

The study identifies several factors associated with extubation failure in the paediatric intensive care unit. The results provide insights into the predictive variables and possible areas for further intervention.

4.2. Demographic and Clinical Characteristics

The study found that the EF rate in the two PICUs in Bloemfontein, South Africa, was 28%, which is higher than the rates reported in some previous studies. Newth et al report an EF rate of 2 to 20% [11], Kilba 11.4% [2], the study by Magose et al reported EF of 16.7% [17]^. Egbuta et al. in his review article found that the rate of EF ranged between 2.7 to 30% [8]. The discrepancy may be attributed to differences in patient populations, clinical practices, and the resource limitations of the study setting.

The median age of the study population was 9 months (IQR 2 -48). This study did not demonstrate any association between age and EF but Khemani et al, found that younger age was associated with higher EF rates, particularly in patients under 1 year old [18]. Kilba et al found that median age of patient who failed extubation was 3.1 months, but this factor was not found to be a significant risk factor [2]. Carvalho et al reported that younger age is associated with increased risk of extubation failure especially in neonatal period. Most probably due to immaturity of respiratory systems [19].

Undernutrition was significantly associated with EF (p=0.0029). Grippa et al highlighted that malnutrition significantly impacts on the duration of mechanical ventilation [20]. Mehta et al pointed out the critical role of nutritional status in PICU outcomes, noting that malnutrition leads to delay recovery and higher morbidity [21].

4.3. Predictors of Extubation Failure (EF)

Our findings indicate that the duration of MV > 3 days, the use of neuromuscular blockade, and the administration of vasopressors are significant predictors of EF. These results align with prior research, except for prolonged ventilation which is a critical risk factor respiratory muscle fatigue and development of ventilation associated complications [18,19]. The association between neuromuscular blockade and EF is also supported in the literature [9,18], likely due to the potential for prolonged muscle weakness following extended use of paralytic agents. Similarly, the use of vasopressors, which may indicate hemodynamic instability, has been linked to poorer extubation outcomes in our study. There is not unanimously agreement about the role of vasopressor as risk factors for extubation failure. Previous studies did not find association between the use of vasopressors and EF [2,22].

4.4. Causes of Extubation Failure

The main reasons for extubation failure were Upper Airway Obstruction (UAO), increased work of breathing and severe apnoea. Khemani et al reported that respiratory complications were strongly associated with EF [18]. According to previous report, UAO and inadequate respiratory effort were the primary causes of extubation failure [23,24]. Similarly, Simonassi et al identified upper airway obstruction as a significant contributor to EF [25]. These findings underscore the importance of thorough respiratory monitoring and comprehensive extubation readiness assessment for all patients undergoing MV. Several factors may contribute to extubation failure, including difficult intubation, underlying neurological conditions, or a history of previous intubations. Abu-Sultanah et al suggest the use of dexamethasone at 6 hours prior to extubation for children at high risk of developing post-extubation UAO but there was unclear benefit in decreasing EF due to UAO [26].

4.5. Indications for Admission to PICU

In our study, respiratory diseases were the primary reason for admission in 43% of cases, the indications for ICU admission might vary when comparing the two centres, Pelonomi and Universitas. Similarly, Carvalho et al, reported that respiratory pathologies and sepsis were common reason for intubation and mechanical ventilation [19].

4.6. Comorbidities

Central nervous system (CNS) disease significantly increases the risk of EF (p=0.022) whereas the absence of comorbidities appears to be protective against extubation failure. Simonassi et al, also found that patients with CNS diseases face a heightened risk of EF [25]. Given these findings, it is essential to implement thorough assessments and take all necessary precautions before extubating patients with CNS pathology.

4.7. Machine Learning Models

The study employed six different machine learning models to predict extubation failure, with artificial neural network (ANN) and Logistic Regression (LR) models demonstrating the highest discrimination. The strong performance of these models suggests that machine learning can be a valuable tool in the PICU setting, providing clinicians with an evidence-based approach to access extubation readiness [27,28]. ANN, specifically, showed superior performance indicating its potential for high sensitivity and specificity in predicting extubation outcomes. These findings are consistent with other studies that have applied machine learning to predict clinical outcomes in critical care [27]. However, while machine learning models offer promising predictive capabilities, it is essential to recognise that these models require extensive validation before they can be reliably integrated into clinical practice. The slight overfitting observed in the XGBoost model, as seen by its calibration metrics, underscores the need for careful model selection and tuning to avoid overly optimistic predictions [27].

4.8. Clinical Implications

The identification of the key risk factors for extubation failure has important clinical implications. By incorporating machine learning models into clinical decision-making processes, PICU teams can better stratify patients according to their risk of extubation failure and tailor interventions accordingly. For instance, patients identified as high risk could benefit from aggressive respiratory support, closer monitoring, or extended weaning protocols. Recognizing the role of factors such as undernutrition and central nervous pathology in extubation outcomes highlights the need for a multidisciplinary approach to patient care, including nutritional support and neurological evaluation.

5. Conclusions

Extubation failure rate was common in our setting (28%) compared to other studies. Days of ventilation, undernutrition, use of neuromuscular blockade, use of vasopressors or inotropes and CNS comorbidity were associated with extubation failure in our study. The mains cause of extubation failure identified in the study was upper airways obstruction. The predictive models used showed excellent discrimination.

6. Limitations

• The retrospective design nature of this study may introduce biases for data recording and data availability
• Most respiratory parameters and ventilator settings were not included in the analysis
• The duration and types of neuromuscular blockades, vasopressors/inotropes were not considered in this study.
• Calibration of models was moderate at best. Increased data size may be required to improve this metric.