Preoperative Diagnosis of Periprosthetic Infection in Patients Undergoing Hip or Knee Revision Arthroplasties: Development and Validation of Machine Learning Algorithm

1. Introduction

The Periprosthetic joint infection (PJI) accounts for 15% of failed total hip arthroplasties (THAs) and for 25% of failed total knee arthroplasties (TKAs) [1]. PJI occurs after 2% of primary THA or TKA [2], which in absolute terms represents approximately 3000 cases in Italy annually [3]. This value is destined to increase following the expected raises in primary procedures performed [2,3]. Preoperative PJI diagnosis remains a clinical challenge, as there is no uniformly accepted gold standard, particularly when secondary to low virulence organisms or when the individual is immunocompromised. There is no efficient system for the preoperative PJI diagnosis, in our clinical practice both pre-operative and intra-operative investigations are involved [4]. Several systems have been developed for the diagnosis such as Musculoskeletal Infection Society (MSIS), the International Consensus Meeting (ICM), the Infectious Disease Society of America (IDSA), and the European Bone and Joint Infection Society (EBJIS) [4]. The diagnosis requires postoperative confirmation provided by the isolation of the pathogen from the result of the intraoperative cultures. Furthermore, in some cases the diagnosis is achieved following microbiological tests routinely performed during the revision procedures without any suspicion of PJI, therefore it is believed that incidence of PJI is greater than what is reported in the official registries. In addition, biofilms production provided by bacteria prevents the detection of infection during the microbiological analysis, causing 15% of false-negative and tests, which were also impaired by lack of sensitivity [5]. Various microbiological detections were developed such as sonication, synovial sampling and incubation period prolongation [6,7]. Although these technological innovations increased the sensitivity of microbiological analysis they were still too expensive to be used in our daily clinical practice. Hence the rationale for the development of a system to diagnose PJI preoperatively without any intraoperative investigations and postoperative confirmations.

The field of artificial intelligence (AI) has advanced in the last years with the application of powerful computing and deep learning algorithms in predicting certain surgical outcomes. Machine learning is a form of artificial intelligence that represents a natural extension of the traditional statistical model [8,9]. This scientific discipline focuses on how computers learn from data, imitate human thinking and may overcome human capability [10]. Machine learning is becoming a promising and powerful technology in the prevention of PJI involving developing and using algorithms capable of identifying correlations and patterns of complex risk factors; it may benefit the prediction, diagnosis, treatment and prognosis of PJI [11]. Some machine learning models have been developed for the prevention of PJI. Although a few studies investigated the use of machine learning in the prevention of PJI, to the best of our knowledge, so far, this is the only study that used ML models to predict preoperative PJI in patients undergoing revision THA (r-THA) and revision TKA (r-TKA) in high volume single center.

The purpose of this observational cohort study is to develop, validate and evaluate a machine learning algorithm based prediction tool of preoperative patient-specific clinical objective criteria. The multivariable Logistic Regression analysis predicts preoperative PJI in patients undergoing r-THA and r-TKA in high volume single center.

2. Materials and Methods

The study was conducted in accordance with the Declaration of Helsinki and good clinical practice guidelines. The study protocol for the development of this registry was approved by Ethics Committee protocol code 444/21 of Humanitas Research Hospital IRCCS, in May 2021. According to the retrospective nature of the study all the included patients have already provided a signed written informed consent to be included in the registry of orthopaedic surgical procedures to consent to the use of their clinical data for research purposes.

2.2. Data extraction

The algorithm was developed using clinical data extracted from electronic health records of patients who underwent r-THA and r-TKA from 1 January 2015 to 31 March 2021, at Humanitas Research Hospital IRCCS Italy. The data extraction step consisted in querying the data from Humanitas Data Warehouse (DWH). Oracle SQL^TM has been used to gather the interested data of patients admitted at the Orthopaedics Department. The available data consisted of vital parameters and values of laboratory exams conducted during the pre-admission assessments.

2.3. Data selection & Inclusion Criteria

The retrospective observational cohort study included patients undergoing elective r-THA or r-TKA by senior surgeons experienced in joint replacement surgery, from 1 January 2015 to 31 March 2021, at Humanitas Research Hospital IRCCS Italy. Patients were identified from hospital clinical records using International Classification of Diseases, Ninth Revision, Clinical Modification (ICD9-CM) procedure codes (00.70, 00.71, 00.72, 00.73 for revision THA; 80.06, 81.55, 00.80, 00.81, 00.82, 00.83, 00.84 for revision TKA). Eligibility criteria included patients undergoing elective revision THA or TKA in our Orthopaedic Department with all the required predictive features recorded. Patients undergoing elective primary THA or TKA: International Classification of Diseases, Ninth Revision, Clinical Modification (ICD9-CM) procedure codes (81.51 for THA and 81.54 for TKA), patients aged under 18 years old and who did not have all the required predictive features recorded were excluded. All patients included were further investigated using International Classification of Diseases, Ninth Revision, Clinical Modification (ICD9-CM) diagnostic codes (99666 and 99667 for infection and inflammation reaction due to internal joint prosthesis) and the postoperative culture-positive sampling for prosthetic joint infections. Therefore, the cohort was divided into an aseptic cohort and an infected cohort.

2.4. Feature selection

The first step was to eliminate all variables with a quote of missing values above or equal to 25%. The feature selection phase was performed using a 10-Fold cross-validation: in each fold, a classification tree is used to perform the classification. For all cross-validation iterations, the selected features were annotated. The features used to perform the final classification are those that were selected in all iterations of the cross-validation.

2.5. Analytics

The model used to run the classification is a Logistic Regression implemented in the Python library sklearn [12]. The model is used in a pipeline composed by three stages:

• Missing values imputer: missing values are imputed with the mean in the overall population.
• Feature scaler: The features are scaled using a minimum and maximum scaler.
• Classification model: Logistic Regression.

The hyper parameter selection was performed through a randomized search cross validation, evaluating the models with the logistic loss measure. After the selection of the best model, the final classification has been evaluated with a 10-Fold cross-validation.

3. Results

3.1. Dataset

Overall 1360 patients were enrolled (57.7% women, median age 66.6 [56.9 Q1 - 75.1 Q3]). 1141 patients in the aseptic cohort (60.6% women, median age 66.6 [56.8 Q1 - 75.1 Q3]) and 219 observations in the infected cohort (42.9% women, median age 67.1 [57.7 Q1 - 75.0 Q3]) were respectively included. The second surgeries were mainly performed at the hips (1230 (90.5%) total, 1046 (91.8%) aseptic cohort, 184 (84.0%) infected cohort. The selected features are mostly laboratory exams and demographic features. Following Table 1 shows the features distribution in the extracted dataset [13]. Continuous features are described as “median [Q1 Q3]”, and binary variables as “absolute-count (percentage)”.

3.2. Classification

Since the outcome is heavily imbalanced, the Logistic Regression has been trained using balanced weights for the two classes, so as to penalise more misclassifications errors on class 1 observations. The classification performance is evaluated using classical indexes like precision, recall and F1 Score in a 10-Fold cross-validation. In Table 2 are shown these indexes differentiating for class and train/test sets. As it can be seen, the reported indexes are quite similar between train and test sets, and for both aseptic cohort and infected cohort and 1 (the F1 score for aseptic cohort is 0.815 ± 0.014 in train set and 0.824 ± 0.062 in test set, while for infected cohort is 0.444 ± 0.011 in train set and 0.427 ± 0.067 in test set). This means that the model was able to generalize the classification and that the risk of overfitting is reduced at its minimum.

Figure 1 displays the ROC curves and the distribution of predicted probabilities. This visualization highlights the generalization of the model, with reported AUC values of 0.770 ± 0.006 in the training set and 0.720 ± 0.057 in the test set. Additionally, the predicted probabilities in both the training and test sets exhibit similar distributions.

An Odds Ratio analysis was performed to interpret the modeling. Figure 2 shows the odds ratio obtained using a Logistic Regression fitted with no regularisation. As it can be seen, the measurement of systemic markers of inflammation such as serum C-reactive protein (CRP), is the most predictive feature positively correlated with the outcome PJI. Gender is also informative, as women demonstrate a higher level of protection against PJI. This finding needs to be furthermore explorated.

4. Discussion

Artificial intelligence has shown great potential in improving the accuracy and efficiency of diagnoses, as well as aiding in the identification of risk factors for various diseases. The present study evaluated the preoperative use of ML algorithms to predict PJI in patients undergoing r-THA and r-TKA using available preoperative clinical data. The most pertinent findings of the present study was that the algorithm adopted demonstrated good discriminatory performance for the prediction of PJI in the selected patients (area under the curve 0.770 ± 0.006 in training set and 0.720 ± 0.057 in test set). The measurement of systemic markers of inflammation such as serum C-reactive protein (CRP), serum RDW and serum related eosinophils were the most predictive feature, readily available preoperative, positively correlated with the outcome PJI. Gender is also informative in the form of reverse association, as women demonstrate a higher level of protection against PJI. The measurement of erythrocyte sedimentation rate (ESR) level, a validated systemic marker of inflammation, was not considered as a feature because the quote of missing values was above to 25%. ML models may be improved by incorporating clinical relevant variables regarding PJI as joint local phenomenon, and not as a systemic process, such as synovial white blood cell (WBC) and synovial polymorphonuclear neutrophils (PMN) which could be obtained through a preoperative joint aspiration. Currently, surgeons and internal medicine physicians seeking to diagnose PJI used a multidisciplinary test battery that included tests to detect joint local inflammation, such as synovial fluid white blood cell (WBC) count and synovial tissue histology [14]. The analysis of synovial fluid obtained by preoperative aspiration, including total cell count and differential leucocyte count and cultures for aerobic and anaerobic organisms, has shown sufficient sensitivity and specificity in multiple studies, it is the most valuable diagnostic tool and should be performed prior to the surgical revision [4]. In our hospital no preoperative synovial fluid analysis was performed, therefore these interesting features were not reported. Regarding features related to formation of cutaneous fistulae and/or drainage of purulent secretions, the present study investigated features related to misdiagnosed “silent” PJI, hence, fistulae and purulent drainage were not considered. Moreover, a previous study reported that the rate of developing a sinus tract in PJI was 21.3%, the presence of sinus tract may be a proxy for other issues such as poor periarticular soft tissue, the poor nutritional status of the host, and multiple prior operations [15]. PJI complication is estimated to occur in 1–3% of patients undergoing primary replacement and in 3–5% of patients undergoing revision [16]. 219 patients were in the infected cohort and 1141 patients were in the aseptic cohort, the patients sample was representative and aligned to the existing literature [1]. With wider application of the joint replacement surgery and related technologies, more and more patients receive the procedure. The incidences of PJI have been on the rise over the recent years and accurate diagnosis of the complication is an urgent task of clinical research. There is currently no universal guideline on the PJI prediction after THA or TKA. In clinical practice, patients at high risk of developing PJI after THA or TKA are identified based on the presence or absence of risk factors and clinicians can only predict the risk of PJI based on their experiences. Whenever a patient’s condition is complex or the dataset is incomplete, the difficulty of prediction may increase, requiring more time for an accurate prediction. The machine learning algorithm presented in this study may be a promising alternative to the manual risk prediction method in predicting preoperatively PJI in patients undergoing r-THA or r-TKA. The tool presented here can forecast patients with PJI before the surgery, the strongest predictors of the occurrence of PJI were high levels of serum CRP, serum RDW and serum Related Eosinophils. In recent years, medical researchers are increasingly employing ML applications in the prevention of PJI, the models were divided into four categories: prediction, diagnosis, antibiotic application and prognosis. Yeo et al in 2022 investigated the use of Artificial Neural Networks for the prediction of superficial surgical site infections and PJI following TKA. They retrospectively included a total of 10,021 consecutive primary TKA patients; the average follow-up time lasted for about 2.8 years. SSIs were reported in 404 (4.0%) TKA patients, including 223 superficial surgical site infections and 181 PJI. The patients’ demographic and operational variables were collected. The model performance was good, with an AUC of 0.84 and a Brier score of 0.054 (a Brier score close to zero indicates good accuracy of probabilistic prediction). The strongest predictors of the occurrence of surgical site infections following primary TKA, were Charlson comorbidity index, obesity (BMI >30kg/m2), and smoking. The neural network model of the study represented an accurate method to predict patient-specific superficial surgical site infections and PJI following primary TKA [17]. Klemt et al in 2021 used artificial intelligence to evaluate prognostic outcomes. They retrospectively reviewed 618 r-TKA procedures for PJI. They showed a ML model with excellent performance for the prediction of recurrent infections in patients following r-TKA for PJI. The ML models all achieved excellent performance across discrimination (AUC range 0.81–0.84), a Brier score of 0.053 (close to zero indicating good accuracy of the probabilistic prediction). The strongest predictors for recurrent PJI were previous surgeries in patients following r-TKA included irrigation and debridement with or without modular component exchange (p < 0.001), > 4 prior open surgeries (p < 0.001), metastatic disease (p < 0.001), drug abuse (p < 0.001), HIV/AIDS (p < 0.01), presence of Enterococcus species (p < 0.01) and obesity (p < 0.01) [18]. Tao et al in 2022 trained a deep learning model to diagnose PJI. 20 r-TKA were enrolled from Chinese People’s Liberation Army General Hospital. PJI infection was based on the 2018 ICM guidelines. Frozen pathological sections collected were converted into electronic images with 461 positive and 461 negative images for model training. The AUC of the model was 0.814 and an average accuracy of 93.3 % [19]. Wu et al developed an accurate machine learning model using administrative and electronic medical records (EMR) to improve the accuracy of surgical site infections (SSI) detection. The study cohort consisted of 16,561 primary TKA and 10,799 primary THA retrospectively included. Their findings suggested ML models derived from administrative data and EMR text data achieved high performance and can be used to automate the detection of complex SSIs: ROC AUC of 0.906 (95% CI 0.835–0.978), PR AUC of 0.637 (95% CI 0.528–0.746) and F1 score of 0.790 (0.670–0.900) [20]. The study findings need to be interpreted in light of its limitations. First, this study design was a retrospective study, which is associated with inherent limitations and reporting bias. Second, it’s a single-institution study, which introduced the potential for confounding effects of unmeasured variables and features; our results may not be generalizable to other institutions. Third, the minimum patients follow-up period was 2 years, but that a larger percentage of recurrent infections and re-revisions may occur with longer follow-up time. Lastly, the sample size is relatively small and the label imbalance is quite pronounced. To obtain more consistent results, new studies with larger cohorts are necessary. Future prospective multicenter studies with long-term follow-up should include more features to construct more widely functioning models to improve their accuracy and performance in the prediction of PJI.

5. Conclusions

This study developed and validated a machine learning model to preoperatively predict PJI in patients undergoing r-THA or r-TKA. The study findings illustrate a good performance in predicting PJI based on preoperative clinical data, providing the effectiveness and applicability of artificial intelligence in our clinical practice. The promising results suggest that there is the potential utility to integrate computerized algorithms in electronic health record systems, where they may be employed to assist in clinical decision-making supplementing the ICM criteria for PJI prediction. Further, prospective studies are mandatory to implement our findings and the reliability of this technology to clinical practice.