Machine Learning-Based Assessment Indicates Patient-Reported Outcomes to Be Less Reliable than Clinical Parameters for Predicting 1-Year Survival in Cancer Patients.

1. Introduction

The benefits of an early palliative care (PC) involvement during advanced malignancy are well documented [1,2] and supported by experts worldwide [3,4]. In one Delphi study, over 80% of 60 international experts agreed that an specialized palliative care (SPC) consultation would be appropriate within 3 months of advanced cancer diagnosis for patients with a median survival time of 1 year or less [5]. However, the optimal model of referral to SPC among advanced cancer patients remains unclear and referral typically occurs late [6]. Some of the reasons are that patients are often never identified or referred at any point during the course of their illness, commonly for an end-of-life care [7]. Therefore, referral models to SPC are in urgent need for optimization.

There are different approaches to promote PC integration in cancer patients [8,9]. The time-based model or early integration model relies heavily on prognosis and PC is introduced from the time of diagnosis of advanced disease. I.e., patients with a poor prognosis are considered for referral. The need-based PC model is a personalized PC based on patient’ needs and delivered at the optimal time and setting. It focuses on early screening of systematic needs, where the SPC is usually reserved for patients with the greatest needs [9]. Also, the combination of different models have been pointed out to facilitate earlier PC access [10].

The need-based PC delivery model coupled with a systematic, regular, and early symptom assessment helps to prioritize patients who could benefit the most from PC. This model, known as the timely PC model, is based on routine screening. Therefore, patient reported outcome measures (PROMs) have gained weight as a routine assessment and are currently recommended as part of the clinical routine. In addition, PROMs have been shown to maximize the identification of symptoms as they arise, and thus to improve patients’ quality of life [11], overall survival [12,13,14] as well as to reduce emergency room visits and hospitalizations [15,16].

The use of prognostic methods from the field of artificial intelligence (AI), and more specifically machine learning (ML), is rapidly increasing. It has been shown to have tremendous potential when applied to medicine [17,18]. As in other domains, ML methods have been widely used in PC. For example, Guo et al. [19] predicted lung infection during PC chemotherapy using a neural network, Van Helden et al. [20] used clustering to predict response to palliative therapy. Many other works focus mostly on survival prediction followed by data annotation and predicting morbidity or response to palliative therapy [21], leverage electronic health records [22], demographic information and other clinical features [23] for predicting survival. Furthermore, prediction of mortality via ML facilitates earlier conversations amongst clinicians and their patients about prognosis, patients goals of care and end-of-life treatment preferences [24,25]. These ML approaches are promising given that for some models the survival prediction is key, and the attempts to estimate prognosis in PC are generally rather inaccurate [26].

To the best of our knowledge, the use of ML for analysis of self-reported symptom burden and other subjective measures regarding survival prediction has not been investigated yet. The aim of this study is to analyze the importance of subjective measures, collected via electronic patient reported outcome measure (ePROM), for survival prediction and SPC integration by using ML.

2. Materials and Methods

Study Design and Sample Description

We conducted a retrospective, monocentric observational pilot-study including patients of a German Comprehensive Cancer Centre based on our routine use of ePROM. We collected non-patient reported outcome (non-PRO) variables such as patient demographics and characteristics by reviewing medical records, from where gender, age, Eastern Cooperative Oncology Group (ECOG), cancer diagnosis, type and stage were obtained. In addition, the answers given in the ePROM questionnaire (as described below) were collected. We obtained the date of death from the local residents’ registration office. Patients were classified in two groups: those who died within 1 year after the screening and those who were alive 1 year after screening.

The data analysis was in accordance with the ethical standards of the institutional and local ethical review committee and was approved by the institutional ethics board (20-9779-BO).

3. Data Collection

A total of 1089 cancer patients were asked to fill out ePROM during the period from April 1st, 2020, to March 31st, 2021. Out of those patients, 672 filled out the ePROM questionnaire. We excluded 407 patients (of 672) because of Stadium WHO <3 tumor (n = 358), lack of solid tumor (n = 29), missing relevant medical data (n = 13) and underage (n = 7) resulting in 256 patients for our analysis.

Out of the 265 eligible patients, 70 patients died within 1 year after the screening and 195 patients lived longer than 1 year after screening (Figure 1). The screening was mainly carried out in the wards of neurosurgery (n = 413), nuclear medicine (n = 324), and dermatology (n = 203) (Figure 1) due to the ongoing implementation of ePROM in our hospital at the time of the data collection.

3.1. ePROM (Electronic Patient Reported Outcome Measure)

Our ePROM assessment is called ePOS (a German abbreviation for electronic psycho-oncological and palliative care screening) and assesses physical and psychological symptom burden. It contains five validated instruments to facilitate an early identification of patients with PC needs [27]. ePOS is a digital based questionnaire, distributed by medical assistants and filled out by inpatients and outpatients. After completion of the questionnaire, the provided information is transferred digitally to the hospital information system and accessible for the responsible medical provider.

ePOS consists of 51 items distributed in five validated, multidimensional symptoms and needs assessment questionnaires: the Personal Health Questionnaire 8 (PHQ8) [28] that shows depressive symptoms; the Generalized Anxiety Disorder Questionnaire (GAD7) [29] that explores anxiety and fears; the Minimal Documentation System (MIDOS 2) [30], which is a German version of the Edmonton Symptom Assessment Scale and analyzes symptom burden; the Hornheider Screening Instrument (HSI) [31] comprises seven questions about physical and mental health and the Distress Thermometer [32], which quantifies the load of burden felt within the past 10 days. In addition, it also assesses the patient’s subjective need for palliative or psycho-oncological support and the patient’s subjective general condition (for a detailed sample of all questions answered by the patients see our supplements).

Not all variables collected in ePROM are relevant for survival prediction, and we removed the following variables from the analysis: “On which topics/occasions may we contact you by email?” and “Intensity of other feelings during the past week”.

3.2. Non-Patient Reported Outcome Variables

Non-patient reported outcome (non-PRO) variables were collected from medical records. We obtained gender, age, ECOG, cancer diagnosis, type and stage, patient demographics, date of death, medications, body measurements, therapy, level of care and number of visits to the emergency room.

3.3. Prognostic Machine Learning Models

For this small data set we employ classical machine learning models. Specifically, we use a linear classifier, logistic regression (LR) with L1 penalty, as well as two non-linear models, a decision tree (DT), and a random forest (RF) to predict 1-year mortality. We also use a simple model that always predicts the most frequent class in our dataset for all instances, i.e., it always predicts the survival class. We refer to this model as “Majority”. Adding this model helps us inspect how much can be gained from using advanced models compared to a model that makes use of a simple statistic, i.e., the most frequent class. We excluded Deep Learning models, since they require a sufficiently large number of patients for training, our case consists of 265 patients with 89 variables (In line with [64], we ran initial experiments using a multi-layer perceptron classifier (MLP). This model demonstrated comparable performance to that of the Random Forest model, while generally being harder to train. We excluded the MLP from further analysis.). In this regard it is important to point out that our goal was to understand the relative importance of patient reported outcome (PRO) variables for the prediction. Thus, we aimed for predictive models with high accuracy, but left potential optimization, such as extensive model selection and hyperparameter tuning for future work. Our dataset is highly imbalanced with more than 70% of the patients belonging to the survival class. To counteract this class imbalance, we used a weighted loss function that adapts the weights of the majority and minority classes based on their frequency during training. We used the standard settings of the scikit-learn library for all models (https://scikit-learn.org/). Additionally, we set the inverse of regularization strength to 0.2 in LR, and the minimum number of samples in each leaf in the DT to 20, to counteract overfitting.

We experimented with three sets of variables: PRO variables (48 subjective variables), non-PRO variables (41 objective variables) and all variables (89 variables) combined to investigate the effect that PRO variables have on survival prediction.

3.4. Evaluation Procedure and Metrics

We trained and evaluated our prognostic models with leave-one-out cross-validation as well as report F1 score and receiver operating characteristic (ROC) curve - area under the ROC curve (AUC).

Leave-one-out cross-validation (LOOCV): Training and evaluating ML models includes having a training set, based on which the model learns the task, and a test set, on which the model is evaluated. In LOOCV, all the available data instances are used for training except for one example, which is used for evaluation. However, evaluating only on one example is not enough and would give a biased estimate regarding the model’s performance. This is why, in LOOCV, the training and evaluating process is repeated for each data instance, i.e., the current data instance is excluded, and the rest of the data instances are used for training. LOOCV enables training on almost all available data (excluding only one instance at a time) and evaluating on all data points.

F1 score: The F1 score is an evaluation metric that combines both precision and recall (also known as sensitivity). Precision quantifies how often the model was correct in predicting the positive class (mortality), whereas recall refers to the proportion of the patients from the positive class that are correctly identified by the model. F1 score is the harmonic mean of the Precision and Recall, i.e., F1 = (2*Precision*Recall)/(Precision+Recall).

ROC-AUC: The ROC plots the false positive rate (1-specificity) on the x-axis against recall (sensitivity) on the y-axis. Each point on the curve represents a specific threshold used to classify an instance as positive (predicting mortality) based on the model’s probability for predicting the positive class. The AUC quantifies the overall performance of the model. An AUC value of 1 indicates a perfect classifier, whereas an AUC value of 0.5 suggests that the model is equivalent to a classifier that randomly guesses the outcome.

4. Variable Importance

Since LR yields the highest prediction accuracy, we investigated which variables LR models used for its predictions. In LR models, the coefficients associated with each variable indicate its importance for predicting mortality. More specifically, the magnitude of each variable is proportional to its importance, and its direction (sign) indicates whether the variable contributes to predicting mortality (positive) or survival (negative). The coefficients of all variables are shown in Table 3. Furthermore, two experienced PC physicians (M.T. and M.R.S.C) independently assessed the importance of each variable regarding its prognostic value and assigned a relevance as "important for prognosis" or "not important for prognosis". Furthermore, we researched the current literature regarding oncological prognostic factors.

5. Results

5.1. Patient Characteristics

Patient characteristics are summarized in Table 1. A total of 265 patients were eligible for our study. The most common cancer types were cancer of the nervous system (n = 63, 23.8%), dermatological cancers (n = 57, 21.5%) and gastrointestinal cancer (n = 42, 15.8%). The most prevalent metastases were cerebral (n = 65, 24.5%), bone (n = 57, 21.5%) and pulmonary (n = 56, 21.1%). Seventeen (6.4%) out of all 265 patients had an inpatient SPC consultation. 70 patients died within 1 year after screening and 195 patients were still alive at time of the study. We used χ2-test to analyze the relation between nominal variables such as sex, primary tumor, and the number of SPC consultations. Additionally, we used Mann-Whitney-U-test to assess whether there was a significant difference in ordinal variables such as physical and mental condition between the two groups. Related p-values as seen below. The general survival rate in our study population was 73.6%.

5.2. Predictive Model Performance

In our predictive ML model, we report ROC-AUC and F1 Score (Table 2). The predictive performance solely based on PRO variables worked only slightly better than the naive baseline, which has not learned anything from the data, but simply predicts the majority outcome (“survive”). The baseline achieves 0.42 F1, whereas the performance of the other models varies between 0.45 and 0.55. Non-PRO variables are much more predictive than PRO variables (LR reaches 0.72 F1, and 0.81 ROC AUC). The performance of all variables for predictions does not improve over the performance of only non-PRO variables (0.72 F1 for LR using non-PRO and all variables) (Figure 2a and 2b). (See Appendix A for the ROC curve with non-PRO variables).

5.3. Prognostic Value

The variables are classified via ML regarding its relevance between important or unimportant, as well as a positive or negative prognostic factor for the patient’ prognosis. Out of a total of 89 variables assessed via ML, 38 were included given that they had the highest absolute value (Table 3a) and 51 were excluded as they were assessed as unimportant via ML. Regarding non-PRO variables, a total of 14 variables were included and 27 were excluded (Table 3b).

Table 3. a. Non-PRO (clinical) and PRO (subjective) variables. Table 3b. Only non-PRO (clinical) variables.

a

b

ECOG: Eastern Cooperative Oncology Group. WHO3: World Health Organisation’ analgesic ladder step 3.

Table 3. a. Non-PRO (clinical) and PRO (subjective) variables. Table 3b. Only non-PRO (clinical) variables.

a

b

ECOG: Eastern Cooperative Oncology Group. WHO3: World Health Organisation’ analgesic ladder step 3.

5.4. Expert Assessment

Two experts in the field of PC were asked to assess independently the importance of each variable regarding its prognostic value. There was disagreement in 7 variables from a total of 38. Each disagreement was discussed and compared with the current literature, where supporting references helped the final decision. A common agreement was possible in each variable. The expert consensus showed a high agreement (81%) regarding the importance of the variables used by the ML for the survival prediction (Table 3).

Table 3a and 3b. Patient self-reported outcome (PRO) and non-PRO variables and its weighting according to ML and expert assessment. The table represents the variables that ML weighted as positive prognostic factors for survival (top-left) and the variables, which were weighted as negative prognostic factors for survival (bottom-left). The importance of the variables after the expert consensus is described as well (right-side).

6. Discussion

The role of subjective measures in survival prediction among advanced cancer patients is unknown. In the present retrospective, observational study we analyzed for the first time the role of an ML method using subjective measures collected via ePROM for survival prediction.

In general, our results indicate that subjective variables, in our study collected via ePROM, cannot be reliably used to predict survival, and that adding or excluding them does not make a significant difference regarding the predictive performance (comparison of results under non-PRO and all variables, Figure 2a,b). Thus, ePROM might not be predictive for a time-based PC model, where prognosis is key, although it clearly has an important role in the need-based referral approach, as stated above.

Currently, in the field of PC, new AI and ML techniques are evolving and could assist in allocating PC interventions more effectively. However, these methods are indeed mainly used to predict mortality [21,33]. Some of them focus on specific patient populations such as dementia [34] or older patients with hip fracture [23], or focus on larger samples from regional health care plans [35], Medicare & Medicaid sources [36] or include all hospitalizations episodes [37]. Its impact in clinical practice has been evaluated as well with promising results where an electronic health record mortality prediction model successfully triggered timely PC referral and thus improved the delivery of hospital-based PC [38]. These studies are of great interest given that physicians tend to overestimate prognosis and only a few prognostic scores have been shown superior to clinician predictions [39]. Interestingly, PC physicians accurately predict survival especially for short survival time periods [40]. This uncertainty regarding prognosis estimation is a relevant barrier for the integration of PC in oncology [41] as well as the limited PC personal and institutional resources, where identifying more patients with unmet PC needs does not necessarily translate into better care [42]. Thus mortality prognostication is a key factor but also one more of the many factors that should be involved in SPC-referral models [43].

In PC, the role of the self-reported subjective symptom burden in cancer patients is gaining weight. The current PC approach recommends that patients with the highest symptom burden and distress would benefit the most from SPC, regardless of diagnosis or prognosis [44]. The patient’s subjective need for SPC has been till now only vaguely studied and has been shown to help prioritize which patients would benefit the most from SPC [45]. The ML used in this study shows that self-patient’ burden assessment cannot be reliably used to predict survival. However, models that recommend patients for PC referral should probably not only focus on mortality given that periods of crisis and clinical deterioration are much more relevant to determine when PC is needed [21] and where unmet PC needs do not necessarily show only close to death [46]. In this regard, the combination of different referral models, in our case through ePROM (need-based) and ML for prognostic mortality (time-based) should be further investigated as they might be necessary for an accurate SPC referral, as already supported by some researchers [10].

The variables used in this study were classified via ML regarding its relevance between important and unimportant and as positive or negative prognostic factors for the patient’s prognosis. The variables weighed as positive prognostic factors supported by the expert consensus were mostly accurate and already documented in the literature: need of antiemetics [47], body max index [48] [49], level or care [50], number of emergency room attendances [51,52], initial stage [53], decrease in appetite [50], ECOG/Performance status [49,54] male sex [54] or the oncological therapy was terminated [49] (see Table 3). The negative prognostic factors classified via ML supported by the expert consensus and already documented in the literature were: the oncological therapy was not terminated [49], number of inpatient admissions after the screening, subjective physical status and general well-being [49]. Other variables were weighted as important via ML, however, were excluded after the expert consensus and the literature research: not receiving antibody therapy, intensity of constipation, adjuvant chemotherapy, not receiving immuno- or antibody therapy, or strong opioids. The variables feeling of restlessness, anxiety or difficulties relaxing during the last 2 weeks are related to emotional distress, which has been shown to play a role in prognosis [49], however the limitation of the ePROM question "during the last 2 weeks’’ prevent us further assessing this variable. The variable "not receiving immunotherapy" was assessed as a positive prognostic factor via ML but rejected from the expert panel, given that immunotherapy has been shown to contribute to prognostic improvement among glioblastoma patients [55]. However, it can be assumed that not all prognostic factors have been identified yet and ML models might focus on variables, which might warrant further exploration.

Although ML algorithms may truthfully predict mortality risk in cancer patients, there are several barriers in order to achieve a wider implementation: oncology clinicians are concerned about algorithm accuracy, biases, and ethical issues of prognostic disclosure [56]. Furthermore, a comparison of the actual literature regarding survival should be carefully approached, since the primary outcome of prognostic studies in PC and oncology has not been established yet [57]. In addition, a secure degree of digitalization and integration of health data is needed [58]. Moreover, some prognostic factors are disease-specific and not consistent across cancer types and stages [48,59]. In this regard, it is important to note that specific guidelines for each malignancy were not used in our study. Laboratory parameters have been described as indicators of poor prognosis [60,61,62], however the integration of such parameters are beyond the scope of our study. Thus, innovative PC delivery using different ML models might help and might serve as a promising tool to identify PC patients who are most in need.

Our investigation has several limitations. First, our analysis was conducted retrospectively in a single center among patients with advanced cancer. Thus, our results may differ from studies with non-oncological participants or in different settings. Second, our final sample size is limited and is neither representative nor balanced, thus further analysis with a larger sample size is required. Third, using ePROM might exclude patients who are not able to complete this self-questionnaire (high symptom burden, language barrier, somnolence, very poor general condition, etc.). Fourth, important variables not included in our ePROM or electronic health records might impact these results and should be further studied. Fifth, the general survival rate in our study population was relatively high, 73.6%, given that all patients had an advanced disease and as an example, the relative survival after 1 year for lung cancer in both sexes is 28.7% [63]. This unexpected lower mortality rate could be a study population’ bias.

The use of self-reported symptom burden and its implication regarding SPC referral among cancer patients together with digitalization is promising where ML has a growing role in improving diagnostic, treatment, and prognostic in PC. The integration of ePROM predicting short-term relevant clinical deterioration and ML predicting overall mortality is promising and should be further explored. Moreover, clinical studies are required to improve how AI and ML models are integrated in our healthcare system and how they can nurture the field of PC.

7. Conclusions

The optimal moment for a SPC referral among cancer patients remains unclear. Although the use of ePROM has numerous benefits in PC, our data and ML techniques used in this study indicate that PRO variables, which measure the patient’s subjective burden, cannot be reliably used alone to predict survival in advanced cancer patients and that AI-based tools may be more suitable for this task. Further research in this area is needed to clarify the role of self-reported patient’s burden and mortality prediction via ML to achieve better care for patients with unmet PC needs.