Assessment of the American College of Surgeons-Surgical Risk Calculator (ACS-SRC) for Prediction of Early Postoperative Complications in Patients Undergoing Cytoreductive Surgery for Ovarian Peritoneal Carcinomatosis

1. Introduction

Epithelial ovarian cancer (OC) is the second most common pelvic gynecological cancer [1]. It represents 1% of all new cancer cases but accounts for 2% of all cancer deaths, with estimated survival at 5 years of 50% [2]. This is because 75% of cases are diagnosed at an advanced stage [3]. The clinical presentation of OC is often nonspecific, consisting of abdominal discomfort, painful bloating, bowel movement disorders, and distension due to the presence of ascites [1].

In locally advanced OC, cytoreductive surgery (CRS) is performed via a xypho-pubic median laparotomy, and involves resection of all macroscopically visible lesions within the peritoneal cavity (completeness of cytoreduction 0: CC-0) [4]. CRS can either be upfront, followed by six courses of carboplatin-paclitaxel chemotherapy [5,6], or at an interval stage, following neoadjuvant chemotherapy. This second option is proposed in high-burden peritoneal disease reflected by a high Peritoneal Cancer Index (PCI) score [1,4] and/or a Fagotti score >2 at first exploration [7]. According to the literature, reported CRS postoperative complication (POC) rates vary from 15% to 25% and mortality rates are around 3% [8,9,10,11]. Therefore, fragile patients at a high risk of severe POCs need to be properly identified to ensure a favorable postoperative outcome and maximize the benefit of surgery in terms of survival outcomes.

The American College of Surgeons-Surgical Risk Calculator (ACS-SRC) estimates the risk of early POCs (within 30 days) based on 21 patient characteristics [12]. The ACS-SRC was constructed using a database of more than five million procedures, reported as “Current Procedural Terminology (CPT)” codes, among 874 participating hospitals between 2016 and 2020 [12]. Of these procedures, 7000 cases involved surgeries for OC [13]. The ACS-SRC was shown to be an adequate predictor for severe POCs after curative surgery for abdominal tumors [14]. However, to our knowledge, only one study has evaluated the performance of the ACS calculator in the prediction of POCs for OC patients [15]. That study only included patients undergoing interval CRS and reported that the ACS calculator was unable to accurately detect which patients were at risk for POCs and longer hospital stays, with very limited preoperative use for CRS for OC.

The aim of this study was to further evaluate the performance of the ACS-SRC in predicting early POCs by including patients undergoing upfront or interval CRS for advanced OC.

2. Materials and Methods

2.1. Study Design and Patients

This was a retrospective study at the Jules Bordet Institute and Erasme Hospital, both part of The Brussels University Hospital, Université Libre de Bruxelles (ULB). It included patients who underwent CRS for advanced OC for primary or recurrent disease, from January 1, 2010 to December 31, 2022. Ethical committee approval was obtained prior to the beginning of the study.

Patient inclusion criteria were: Age>18 years of age; tubo-ovarian or peritoneal epithelial carcinoma; CRS for locally advanced primary or recurrent cancer (stage III or IV according to the classification of the International Federation of Gynecology and Obstetrics (FIGO)).

Patients with incomplete medical records were excluded.

2.2. Cytoreductive Surgery (CRS)

CRS typically consisted of total radical hysterectomy with bilateral salpingo-oophorectomy, complete omentectomy (infra- and supra-colonic) and resection or electrofulguration of any suspected peritoneal nodules/scars/fibrosis visible during exploration. Pelvic and lumbo-aortic lymphadenectomy were frequently performed in patients before 2019, regardless of preoperative lymph node status; however, in recent years, this has become dependent on clinical FIGO staging and preoperative imaging [16]. In patients who underwent multiple surgeries, the procedure chosen for analysis was the first one performed at our hospital (whether for primary or recurrent disease).

2.3. Data and ACS-SRC

The following data were collected from multidisciplinary oncology committee reports, surgical consultations, anesthesia consultations, and medical imaging protocols:

• Demographic and clinical characteristics: age, gender, body mass index (BMI), type of procedure (CPT code), functional status of the patient (daily third-party dependency);
• General preoperative status and medical conditions: American Society of Anesthesiology (ASA) score, chronic corticosteroid therapy, diabetes, hypertension (HTN), heart failure within 30 days prior to surgery, dyspnea, smoking within one year prior to surgery, chronic obstructive pulmonary disease (COPD), acute and chronic renal failure (RF), dialysis, presence of ascites during the 30 days prior to surgery, sepsis during the 48 hours prior to surgery, oxygen dependency prior to surgery;
• FIGO stage and other oncologic data (e.g., chemotherapy, date of diagnosis, date of surgery);
• POCs were determined based on laboratory analyses, imaging, intensive care unit and surgical ward follow-up notes, and consultation notes in the 30-day postoperative period.

2.4. ACS-SRC: Calculating the Risk of POCs

The ACS-SRC was used for each patient included in the study, the above data were manually entered into the ACS-SRC online website (https://riskcalculator.facs.org) and the estimated risk percentage of each complication predicted by the calculator was collected. The list of POCs predicted by the ACS-SRC included: pneumonia, cardiac complications (e.g., cardiac arrest or myocardial infarction), surgical site infection (SSI), urinary tract infection (UTI), venous thromboembolism (VTE), RF, readmission, re-intervention, death, and prolonged hospital stay.

The ACS-SRC includes a section for encoding one CPT procedure. For each patient, the most appropriate CPT code was used. For simple debulking with no resection and no lymphadenectomy, CPT 58956 or CPT 58953 was used. For simple debulking associated with lymphadenectomy, CPT 58951 or CPT 58954 was used. In cases of recurrent disease, CPT 58957 or CPT 58958 was used. In cases with digestive resections, CPT 38102 or CPT 44120 or CPT 44145 or CPT 47120 was used.

CRS frequently involves several surgical procedures during the same operation. In the ACS-SRC, it is impossible to simultaneously encode several procedures. Therefore, to evaluate the best manner for predicting POCs, three calculation methods for each patient were performed:

• First, we exclusively encoded CPT for ovarian debulking (5895X), distinguishing between standard or radical surgery, primary or secondary surgery (for recurrence), and surgery with or without lymph node dissection.
• Second, we integrated the above-mentioned CPT as well as CPT for digestive surgery [17]. In the case of digestive resection, the calculator was run twice, and the highest score per complication was used.
• Third, we used the adjustment score proposed by the site on the scores obtained after introduction of ovarian debulking CPT (risk 1 to 3). Procedures involving only the usual resections (i.e., hysterectomy with salpingo-oophorectomy, omentectomy, and resection of peritoneal carcinomatosis nodules) were classified as risk 1. Procedures involving a resection considered by the surgical team to be of moderate complexity (i.e., splenectomy, enlarged peritonectomy of the Douglas and/or diaphragm, small bowel resection) were classified as risk 2. Those with a resection of complexity considered important and at high risk of complications by the surgical team (i.e., colectomy, rectal resection) were classified as risk 3 [10,11].

2.5. Statistical Analysis

Statistical analyses were performed using SAS 9.4 by the institution’s biostatistician. Analysis included a descriptive analysis of the study population and their characteristics expressed as mean (+/- standard deviation) and proportion (in %), depending on whether the variables were continuous or discrete. Discrete variables were evaluated using the Chi-squared test and continuous variables were evaluated using Student’s t-test.

The predictive performance of the ACS-SRC was analyzed by assessing discrimination and calibration. Discrimination measures the ability of a regression model to differentiate a patient at high risk for a given event (e.g., cardiac complication) from others, using a set of predictor variables for the given event (e.g., BMI, age, co-morbidities). This makes it possible to characterize the risk of one patient compared to another, without considering risks in absolute values. Calibration measures the ability of a model to assign an absolute value to an estimated risk, making it possible to analyze whether the absolute value of observed events is statistically comparable to the absolute value of events estimated by the model [18].

In this study, discrimination was analyzed by calculating the area under the curve (AUC) of a Receiver Operating Characteristic (ROC) graph. This statistical model quantifies the association between the estimated risk and the observed POC in the patient. The AUC values are in the range of 0 to 1. For example, a model with a 40% chance of a right prediction has an AUC of 0.4. Discrimination is considered poor for an AUC between 0.6 and 0.69, acceptable if between 0.7 and 0.79, powerful if between 0.8 and 0.89 and excellent if between 0.9 and 1.0.

The Brier score simultaneously evaluates discrimination and calibration [9,17,19]; a criterion that provides combined information on the accuracy of the prediction. It describes the mean square difference between a predicted risk and the actual outcomes, making it an effective metric for evaluating the accuracy of a prediction model: if the probability of a complication occurring is 100% and the patient has this complication: the score is 0 and the predictive model is considered perfect. If the probability of a complication is 100% but the patient does not have this complication: the score is 1, and the model is considered unreliable. If the probability of a complication is 50%: the score is 0.25. An estimated risk with a Brier score greater than or equal to 0.25 is considered non-informative [20,21]. In this cohort, the calculator was considered reliable if the Brier score <0.01 indicating a high level of predictive accuracy. Statistical analyses are considered significant if the p-value <0.05.

3. Results

A total of 267 OC patients underwent surgery at the included institutions between 2010 and 2022. Of these, 21 patients were excluded for FIGO I-II, 23 patients were excluded for incomplete surgery, and 5 patients were excluded for missing data. Finally, 218 patients were included in the study. Patient characteristics are reported in Table 1. Figure 1 shows the distribution of patients according to CPT code. The median age was 63 years, with a mean BMI of 25 kg/m². Hypertension was present in 38% of cases, smoking in 10%, and diabetes in 9%. One hundred fifty-six patient (71.6%) were ASA II. In 112 patients (51%), surgery was considered extensive (i.e., including digestive/splenic resections and/or extended peritonectomies). The median postoperative stay was 12 days. Concerning CRS, 89% of resections were CC-0, 3% CC-1, and 8% CC-2.

3.1. Complications

Ninety-four patients (43%) developed one or more early POCs (totaling 142 complications). Of those, 31 patients (14%) had major complications (Clavien-Dindo (CD) grade III-IV). The mortality rate was 0.5%, (1 patient who had sepsis and developed pulmonary effusion and respiratory arrest followed by cardiac arrest on post-operative day 12). In order of frequency, UTI (n=45), intra-abdominal abscess (n=12), and bacteremia (n=12) accounted for 48%, 13%, and 13% of complications, respectively.

By convention, in the univariate analysis, the predetermined minimum number of events required for interpretation is ten. Among the outcomes evaluated by the ACS-SRC, the incidences of cardiac complications (n=6), VTE (n=3), RF (n=6), and death (n=1) did not reach this minimum threshold and, therefore, are considered as uninterpretable.

According to the three calculations performed for each patient, the prediction of POCs proposed by the calculator underestimated the incidence of all complications observed in our cohort with the exception of readmission rate.

3.1.1. Discrimination

In terms of discrimination, with method n°1, the calculator showed significant results for NCPD (c-statistic = 0.70) and death* (Brier score = 0.004). With method n°2, the calculator only showed that death* was well discriminated (Brier score = 0.004). Finally, with method n°3 the calculator showed significant results for NCPD (c-statistic = 0.71) and death* (Brier score =0.005). Note that *death was mentioned for information purposes only (given the insufficient number of events).

3.1.2. Calibration

In terms of calibration, with method n°1 (CPT Debulking), SSI (p=0.003) and need of nursing care post-discharge (NCPD; p=0.04) were statistically significant. With method n°2 (double scoring: CPT Debulking and Resection; highest score), readmission (p= 0.009) and NCPD (p=0.02) were statistically significant. Finally, with method n°3 (CPT Debulking weighted by risk adjustment), SSI (p=0.005), readmission (p=0.04) and NCPD (p=0.05) were statistically significant. The calculator was not correctly calibrated for the other complications despite the use of three different analysis methods.

4. Discussion

This is the first study to evaluate the predictive performance of the ACS-SRC in patients treated with upfront or interval CRS for stage III/IV ovarian cancer. This study suggests that the ACS-SRC underestimates all postoperative complications except NCPD. The calculator, using the main CPT code with or without the adjustment score (risk 1-3) discriminates well and is well calibrated for NCPD. Moreover, the calculator was not able to discriminate patients but was well calibrated to predict SSI (method 1 & 3), and readmission rates (method 2 & 3).

The ACS predictive model was developed to help surgeons rationalize surgical indications, taking into account the risk of POCs, particularly in fragile patients. Its validity in gynecological oncology, particularly advanced OC, has not been well evaluated [15,17,20].

The first studies to evaluate the effectiveness of the ACS-SRC calculator on gynecological procedures (benign and malignant combined) were published by Rivard et al. and Szender et al. [17,20]. A more recent study published by Manning et al. in 2021 [15] looked at 261 patients who had exclusively undergone interval ovarian debulking [15]. The results of these three retrospective studies highlight a tendency for the ACS-SRC to grossly underestimate the various complications predicted by the calculator. The overall POC rate in the present study was 43%, with 14% being severe complications (CD III/IV). This is comparable to other studies that have reported CD I/II complications of 20%-22% and CD III/IV complications of 11%-15% [5,11]. The ACS-SRC failed to predict these rates.

In the present cohort, the calculator demonstrated appropriate discrimination in the prediction of NCPD (c-statistic=0.70-0.71 depending on the method used). For calibration, it was satisfactory for SSI (p=0.003-0.005), risk of readmission (p=0.04-0.009), and NCPD (p=0.02-0.05). In the aforementioned studies [17,20], the calculator demonstrated satisfactory discrimination with regard to prediction of RF, cardiac complications, pneumonia, VTE, and death but other predictions of POC were not considered reliable [17,20]. In the present study, the number of events for RF, cardiac complications, VTE, and death were low (<10) and, therefore, those results were not interpreted. In the Manning study, the calculator was not discriminative for any of the proposed outcomes but was appropriately calibrated for hospital length of stay [15].

The ACS calculator has been validated on more than 7000 patients with ovarian cancer [13]. However, this study did not take into consideration the CC and residual disease at the end of the operation. Therefore, it included all types of resections, which would underestimate the results when compared to populations undergoing complete CRS (CC-0/1). Moreover, the population studied by the ACS to elaborate the SRC contained a low proportion of gynecological procedures (1.5%) [9] and the surgeries recorded in the ACS registry are essentially of low complexity [6].

Some of the clinical data proposed by the ACS calculator portal lack precision. For instance, concerning the preoperative presence of ascites, for which the calculator portal doesn’t require volume specification despite the fact that volume has been reported as a predictive factor for postoperative complications [5].

The ACS score also misses other important predictive factors of POCs for CRS. The Peritoneal Cancer Index (PCI) is used to assess the extent of peritoneal carcinomatosis in locally advanced cancer and has been reported to be related to increased POC rate in patients with PCI ≥21 [5,10]. In the present cohort, patients who developed pneumonia had significantly higher average PCI values (19 vs. 11, p=0.02), and this was also true for those with VTE (33.5 vs 12, p=0.004).

Another factor that could potentially influence the risk of POCs after CRS is the degree of radicality with the goal to achieve a CC-0 resection [22]. The extensiveness inherent in the radical nature of this surgery is associated with a higher severe complication rate (OR of 1.96) compared with standard procedures [4,8,13,23].

A more specific predictive model for gynecological oncology, developed in 2019 by Cham et al., that incorporates preoperative albumin and a surgical complexity score, independently validated in 2021, is also not implemented in the calculator [6,13].

Finally, this study has several limitations. First, its retrospective design leaves open the potential for missing information concerning POCs. Second, the limited number of patients included in the study impaired the statistical analysis of several complications such as cardiac complications, VTE, RF, and death.

5. Conclusions

The use of a calculator to predict POCs in patients with locally advanced OC that need to undergo CRS is important. It helps provide patients with an estimate of the risk and helps surgeons/oncologists to better select candidates for surgery based on identification of those with a reasonable risk. However, the ACS-SRC calculator does not reliably predict POCs in this setting. The calculator using the main CPT code, with or without the adjustment score, discriminates well and is well calibrated for the prediction of NCPD. Further efforts should be made to refine the calculator for predicting POCs in patients treated for stage III/IV ovarian cancer with complete CRS.