Towards Predicting Length of Stay and Identification of Cohort Risk Factors Using Self-Attention Based Transformers and Association Mining: Covid-19 as Phenotype

1. Introduction

Worldwide, health systems infrastructure was severely strained by the rapid surge of patients infected with different coronavirus variants and many countries struggled to provide basic healthcare and timely services to patients [1]. Despite availability of Covid-19 vaccines, statistics shows that the hospitalization rate spikes globally in winter seasons during last two years [2] According to a study conducted in the US, inadequate critical care is associated with resource availability [3] and one hour of delay in services is associated with a 3% increase in patient mortality [4] An extended LoS is associated with a high risk of negative outcomes including adverse drug effects, hospital-acquired infections, inadequate nutritional levels, and many other complications [5] Inpatient care accounts for roughly a third of all healthcare spending in the U.S., with an average length of stay of 4.5 days and a daily cost of $10,400 [6]. Predicting Length of Stay (LoS) and understanding how to reduce it is one of the most critical factors for optimal usage of hospital infrastructure and medical resources during emergence of new infectious diseases. Thus, it improves financial, operational, and clinical outcomes by reducing costs for patients, such as facility expenses, supplies, and staffing. In addition, it minimizes the risk of hospital-acquired infections and reduces waiting time for patients. Thus, for precise resource management and utilization of the current infrastructure of the healthcare, a sophisticated approach is needed to predict the LoS, identify the cohort risk factors that leads to increased LoS, and understand what treatments can reduce the LoS.

For different diseases, LoS prediction was performed using statistical and conventional Machine Learning (ML) techniques such as Logistic Regressions (LRs), Random Forest Classifier (RFC), Decision Trees (DTs), etc. For example, Luo et al. [7] used LR and RFC to predict LoS in patients with pulmonary disease. Likewise, Dogu et al. [8] employed Artificial Neural Networks to predict LoS Chronic Obstructive Pulmonary Disease (COPD) patients while Kulkarni et al. [9] performed it using Multi-Layer Perceptron (MLP) for acute coronary syndrome patients. Likewise, to predict the ICU admission, mortality, and survivors’ LoS for COVID -19, Dan et al. [10] created three ML prediction models without identifying cohorts of risk factors, and analysis was based only on univariate analysis. Lastly, Vekaria et al [11] developed statistical techniques such as truncation correction and survival bias to predict patients’ LoS for COVID-19 patients but the employed data is not multimodal and suffers from quality issues such as missing timeline for discharged patients, and limitation of statistical model to recognize hidden patterns.

Deep Learning (DL) has proven capable of extracting complex, hidden correlations from data and achieved promising results when compared to existing ML methods. For instance, Zebin & Chaussalet [12] used an autoencoder deep neural network to categorize short stays (0–7 days) and long stays (>7 days) using Medical Information Mart for Intensive Care III (MIMIC III) dataset [13] but the dataset lacked multi-modalities and methodology does not analyze cohort of risk factors responsible for extended LoS. Likewise, Harerimana et al. [14] proposed an attention-based DL method to predict LoS and in-hospital mortality but this method also has limitations of using limited lab data and unavailability of radiological information. Rajkomar et al. [15] proposed a three-tier approach by combining three DL models to predict hospital readmission and patients LoS, based on data belonging to patients with varying diseases, without identifying the cohort of risk factors affecting patient LoS. These existing approaches only classify discharge or admit cases and predict the duration of the LoS using single modality data and are unable to provide the cohort of influencing factors that could increase or decrease patients’ LoS, and most of them were not designed for predicting LoS for patients with infectious diseases.

The study design in this research focuses on detailed analysis of interaction and association between different risk factors and develop a framework for predicting the LoS of COVID-19 patients using state-of-the-art transformer-based architecture and identify groups of influencing factors occurring together and affecting LoS using pattern recognition techniques by mining multi-modal patient data such as lab data, x-ray data of Covid-19 patients.

The main contribution of this paper can be summarized as follows:

• A state of art attention-based tab transformer model is presented to predict patients LoS using multiple modalities such as clinical features, patients’ demographics data, and X-ray reports.
• We present a framework where the result of machine learning based methods for LoS prediction can be analyzed with association mining rules and identify the cohort of risk factors affecting the LoS in hospitals.

The remainder of this paper is organized as follows. Section 2 describes the proposed method while section 3 presents experimental results and evaluation. Section 4 presents further discussion on the obtained results. Section 5 discusses limitations and future research directions followed by conclusion in Section 6.

2. Materials and Methods

The proposed framework for LoS prediction and identification of cohorts of risk factors includes three major components: Data Acquisition, Data Preparation, COVID-19 Risk Modeling. The detailed architecture and various subcomponents are shown in Figure 1. The main tasks in the data acquisition module include getting de-identified patients’ data from Electronic Health Records (EHRs), identification of missing values with respect to main prognostic factors. The Data Preparation module includes data cleaning, quantization, balancing with respect to output (deceased, discharge) and LoS. The COVID-19 risk modeling includes building and hyper tuning machine learning models and using association mining algorithms to identify cohorts of risk factors.

2.1 Data Acquisition

In this study, patients’ data is obtained from Prince Sultan Military Medical City Infection Control Center of Excellence who were admitted between April 2020 to January 2021. There were 311 cases of confirmed COVID-19 infection, and after analyzing the data using prognostic factors, 308 cases were included in the analysis. This dataset includes 60 patients who died during treatment and 248 patients who were discharged. In total 89 features from three different categories: general information, X-ray and lab test are extracted. Table 1 shows the description of available features and frequencies with respect to different modalities data.

2.2. Data Preparation

In the data preparation stage, we performed basic data preprocessing such as binning, encoding, and initial Exploratory Data Analysis (EDA) by examining the distribution of attributes and summarizing the statistics of the data to uncover hidden patterns in the data. Table 2 shows the descriptive statistics about prominent features across different modalities of data. Also, during the EDA, we found that certain tests like aspartate aminotransferase, creatine, phosphokinase, and fibrinogen were not requested for some of the patients. Therefore, these features are not considered in modeling.

2.2.1. Natural Binning:

Data binning is a method used to minimize the effect of small observation errors. This is used to discretize continuous features and transform them into categorical features. We performed natural binning of all the continuous features such as age, Ph, PaO2, HCO3 etc. and then consulted with experts to adjust the binning boundaries. Binning introduces non-linearity and improves the performance of machine learning models by minimizing small observation error. Table 3 shows the optimized categorization for the continuous features.

2.2.2. Encoding of Categorical Features

Our dataset comprises various categorical features including label values of (0- No, 1-Yes) such as diabetes and hypertension. It is essential that categorical features be transformed into numeric features before ML models can be trained effectively. We utilized an internal one-hot encoder; and converted various categorical features such to numerical features so that ML models can process it efficiently.

2.2.3. LoS Category Creation

Using the combined dataset from data acquisition step, we calculated the LoS using the admission and discharge or died timelines for all the patients and divided it into categories such as deceased within 3 weeks, deceased after 3 weeks, discharged within 1 weeks, discharged between 1 and 2 weeks, discharge between 2 and 3 weeks, discharged between 3 and 4 weeks, and finally discharged after 4 weeks from the date of admission. Table 4 shows the number of patients in each of these categories of LoS and patient’s frequency.

2.2.4. Data Balance with Respect to LoS

The original dataset is imbalanced considering various LoS categories with-in discharged and deceased dataset. A balanced dataset is necessary to train a machine learning model to generate higher accuracy models and make unbiased decisions. The two primary approaches to make a balanced dataset out of an imbalanced dataset are under-sampling and oversampling. Given the limited number of patients in each categorized LoS, in this work we employ random oversampling using a variant of Synthetic Minority Oversampling Technique (SMOTE) called SMOTE-N [16]. This technique works well for categorical data such as diabetes, hypertension etc. and generates new instances from existing minority classes by taking samples of feature space for each target class and its nearest neighbors. This algorithm then generates new examples that combine features of the target case with features of its neighbors and increases the features available to each class and make the data more general. After balancing the data, the each LoS categories with in discharged (n=84) and deceased(n = 36) data contain equal records. Table 4 shows the original data and increased count of instances with-in each category after applying SMOTE-N.

2.3 COVID-19 Risk Modeling

The preprocessed and balanced data is used to develop and train a LoS Predictor model for both deceased, and discharged patients. At first, we developed a LoS Predictor model(t-LoSP) using state of the art transformer-based classifier followed by a Cohort Risk Factor Identifier (CRFI) to identify groups of risk factors affecting the patients LoS.

2.3.1 t-LoS Predictor

Tab-Transformer is an innovative recently developed deep tabular data model that can be used for both supervised and semi-supervised learning. Self-attention transformers are the foundation of the Tab-Transformers model. In the dataset, we have many categorical variables such as diabetes, hypertension, abnormal X-rays etc., the other available machine learning model such as neural network lacks the interaction and relationship between categorical variables in categorical embedding process. In the transformer-based architecture, the transformer layers convert categorical feature embeddings into strong contextual embeddings to improve prediction accuracy. Tab-Transformer architecture consists of a column embedding layer, a stack of N Transformer layers, and a multilayer perceptron. An individual transformer layer consists of a multi-head self-attention layer followed by a position-wise feed-forward layer. Figure 2 shows the detailed architecture of self-attention-based Tab-Transformer model and following are the steps for model execution:

Let $x$ denotes input feature set and $y$ multi-class target variable. Feature set $x$ consist of both categorical ($X_{ca}=\left\{x_1,x_2,x_3\dots x_m\right\}$) and continuous variables ($X_{co}$). All categorical features are embedded into it to embedding space of dimension $d$ using column embedding.

Let $e{\phi }_{\mathrm{j}}\in {\mathrm{R}}^{\mathrm{d}} \mathrm{for} \mathrm{j} \in \left\{1, · · · , \mathrm{m}\right\}$ be the embedding of the $x_j$ feature, and $E\phi \left(X_{ca}\right)=\left\{e{\phi }_1 \left(x_1\right), · · · , e{\phi }_m\left(x_m\right)\right\}$ are the embeddings for all categorical features.

The set of projected categorical embeddings $E\phi \left(X_{ca}\right)$are input to the first transformer layer as shown in Figure 2. The output of the first transformer layer is sent to the next transformer layer and continues for $N$ layer of transformer. The embedding output from individual layer is transformed into contextual embedding when resulted from the top layer of transformer, by consecutive aggregation of context from other embeddings. The sequence of transformer layers is denoted as a function $f_{\sigma \_}$. This function operates on parametric embeddings of categorical variables $\left\{e{\phi }_1 \left(x_1\right), · · · , e{\phi }_m\left(x_m\right)\right\}$ and results corresponding contextual embeddings $\left\{k_1,k_2 · · · , k_m\right\}{ \mathrm{where} \mathrm{k}}_{\mathrm{i}} \in \mathrm{R} \mathrm{d} \mathrm{for} \mathrm{i} \in \left\{1, · · · , \mathrm{m}\right\}$. At the last the contextual embeddings obtained from transformer encoders $\left\{k_1,k_2 · · · , k_m\right\}$ are concatenated with the continuous features $X_{co}$ to form a vector of dimension $\left(d\times m+c\right)$ and server as an input MLP classifier, denoted by $h_{\psi }$ to compute target prediction variable $y$. Let$J$ is the categorical cross-entropy for multiclass classification prediction task, we minimize the loss $J\left(x, y\right)$ to learn all the parameters of the Tab Transformer using gradient descent optimization method. The parameters of Tab Transformer include σ for Transformer layers, φ for column embedding, and ψ for the top MLP classifier.

$$J\left(x, y\right)\equiv H\left(g\psi \left(f_{\sigma } \left(E\phi \left(x_{ca}\right)\right), x_{co} \right), y\right)$$

(1)

More information about tab-transformer architecture is available in [17]. The effectiveness of Tab-Transformer on multi-class datasets and particularly LoS prediction is unknown. In each deceased and discharged dataset, 70% of the data is used to train the model, and 30% to test its accuracy across multiclass prediction of LoS.

2.3.2. Cohort Risk Factors Identifier

The aim of CRFI is to identify the factor or combination of risk factors that had the greatest influence determining patients LoS in hospital. For this purpose, we employed the Apriori [18] that generates association rules by mining transactional data, which in our case are patient characteristics, symptoms, lab data, and X-ray features in each defined category of LoS. The association mining consists of the following four steps:

STEP: 1 Find all frequent itemset i.e., all patient characteristics appearing frequently together in the data with 50 % support and 70 % confidence.

$$Suppor{t}_{\left(A\right)}=\frac{\mathrm{Number} \mathrm{of} \mathrm{transactions} \mathrm{in} \mathrm{which} \mathrm{A} \mathrm{appears} }{\mathrm{Total} \mathrm{number} \mathrm{of} \mathrm{transactions}}$$

(2)

$$Suppor{t}_{\left(A\to B\right)}=\frac{\mathrm{Number} \mathrm{of} \mathrm{transactions} \mathrm{in} \mathrm{which} \mathrm{A} \mathrm{and} \mathrm{B} \mathrm{appears} \mathrm{together} }{\mathrm{Total} \mathrm{number} \mathrm{of} \mathrm{transactions}}$$

(3)

$$Confidenc{e}_{A\to B}=\frac{\mathrm{Support}\left(\mathrm{A}\to \mathrm{B}\right)}{\mathrm{Support} \left(\mathrm{A}\right)}$$

(4)

where A, and B are itemset such as hypertension, diabetes, age ranges, and lab characteristics ranges etc. as defined in Table 3.

STEP: 2 Generate association rules from the aforesaid frequent itemset.

STEP: 3 Create a metric by calculating normalized harmonic mean of support and confidence using min-max scalar.

STEP: 4 Productionize the rule by selecting all the rules above threshold value (β =.7). This threshold value is decided after the rules are reviewed by the experts and considering the frequency of patients belonging to each rule.

3. Results and Evaluation

This section shows the performance of the proposed framework and highlights the important features in estimating the LoS of hospital patients. The implementation and experiments are performed on a 2.10 GHz Intel(R) Xeon(R) Platinum 8160 processor with Python programming environment.

3.1. COVID-19 Risk Model Results

We performed experiments with five ML models AdaBoost (AB), Decision Tree (DT), Gradient Boosting (GB), Logistic Regression (LR), Random Forest (RF), and a deep learning transformer-based model called Tab-Transformer (TabT) and used precision, recall, accuracy, and F1 score to compare the results. The hyper tuned tab-transformer model achieves the highest level of F1 score (discharged: .92; deceased: .84) compared to the base ML classifiers on both deceased and discharged dataset. Table 5 Shows comparative analysis of base machine learning models and tab-transformer model for LoS prediction on discharged and deceased dataset.

3.2. CRFI Results

CRFI identifies cohort of risk factors associated with LoS and generates rules based on various patient characteristics. Table 6 illustrates the top sample rules for each category of LoS with in discharged and deceased patients’ category. The complete rule set is publicly available to consume in GitHub repository [19].

3.2.1. CRFI for Discharged Patients’ Category

In the discharged patients’ category, for LoS ≤ 1 week or Los ≤ 2 weeks, usage of anticoagulant, antibiotics and antiviral medications are important factors and indicates that timely intervention and dosage reduces LoS. For LoS ≤ 3 weeks, some of the important risk factors observed in the rules are elevated level of LDH (>225), D-Dimer (>500) and CRP (between 6 mg/L to 100 mg/L). Observed rules suggest that patients with abnormal values of these factors takes time to recover even if they provided with anticoagulant and antiviral medications. For LoS ≤ 4 weeks, the important risk factors observed are higher lymphocytes count (>1000 cells/µL), elevated PNN count (1000 -7000 mm3), comorbidities such as hypertension, higher respiratory rate (20-28 bps). The mining results on patients who stayed more than 4 weeks in the hospital shows less platelets count (<50000), abnormal X-ray, PTT>14.5, and higher PNN count. We found these patterns along with usage of antiviral, anticoagulant and antibiotics medications, which again suggest that abnormal values of above-mentioned risk factors increase LoS even if the medication are provided. The most affected (~40 %) age group for across all the category is (age >=56 Years and <=73 Years), closely followed by (age >=38 Years and <=55 Years). The observation suggests that age is an important factor in deciding the LoS.

3.2.2. CRFI for Deceased Patients’ Category

In the deceased patients’ category, Shortness of breath (SoB), low platelets count (<50000), diastolic blood pressure between (60 and 90), abnormal PTT (>14.5), higher LDH count (>225), low PaO2 (<80), and TROPONIN between (0 and .1) are the most critical factors for the patients who died within 3 weeks of admission to hospital. The critical risk factors for patients with LoS ≥ 3 weeks are ALT (0 - 41), SoB, comorbidities such as diabetes and hypertension, Glasgow (>14). The pattern suggests that abnormal values of these risk factors not only increase LoS but also increase the severity of COVID leading to patient death eventually. The most affected group is (age >=56 Years and <=73 Years) with 65%, 58.82%, 47.0%, and 56 % in rule 1, 2, 3, and 4, respectively. The younger generation (<=37) distribution across all the rules in deceased category is very minimal and for LoS>3, there is none from this age group.

4. Discussion

The current study aims to analyze cohorts of risk factors obtained from multimodal data by using state of art deep learning model tab transformer model and associating mining. The state of art tab transformer model shows excellent results on both deceased and discharged patients in predicting their LoS. The CRFI module analyzes the group of risk factors which extend the LoS in hospitals and result in either discharge or death. CRFI results show the identification of risk factors in cohorts can help in determining LoS and identifying criticality that leads to COVID severity.

Not much work has been done in determining LoS for COVID-19 patients using multimodal data. Examples and discussion of the prominent patterns are as follows:

• Age appears to be a strong risk factor for COVID-19 severity and its outcomes. Statsenko et al. [20] performed detailed analysis and concluded that elderly patients with COVID-19 are more likely to progress to severe disease. The result of CRFI within deceased category consists of rules with age >=56 Years and <=73 Years while other age category rules are not frequently observed and found to be insignificant. In addition, the mining results on patients who stayed in hospital between three and four weeks contains 25% of the rules with age>=73. These observations validate the fact the age is correlated with COVD-19 severity and a significant factor in deciding LoS.
• The detailed analysis on CRFI rules on the patients who stayed in hospital between 3 and 4 weeks showed that 43 % of the rules constitute either hypertension or diabetes, thus these comorbidities not only increase the LoS in hospitals but also leads to severe COVID leading to increased LoS in the hospital. Same was concluded by Adab et al., 2022 [21].

We also observe many key findings with respect to lab features such as LDH, dimmer and lymphocytes count. Few examples are outlined below:

• The elevated level of D-dimers is an indicator and major risk factor for thrombosis (blood clotting) and increases the risk of medication and monitoring for a longer time [22]. We observed that for the people who discharged between 3 and 4 weeks CRFI results with D-dimers shows that 18 % of the rules have D- dimers value more than 500 ng/mL FEU, thus increasing their LoS. In addition, mining results on patients who stayed more than 3 weeks and died elevated D-Dimer values is present in 41% of the rules. This is also validated by the fact that the people who discharge within two weeks, CRFI results show only 4.5 % of the rules has D-Dimer value more than 500 ng/mL FEU and elevated D-Dimers values are not found to be significant on the CFRI results of patients who stayed less than one week.
• LDH is another factor that has an elevated level of more than 225 units/L in 23% of the rules based on CRFI results of patients discharged from hospital between 3 and 4 weeks.
• Wagner et al. [23] concluded that lymphocytes count is one of the prognostic factors in determining COVID-19 illness, and our CRFI results for patients who died after spending more than 3 weeks in hospitals found that all the rules with lymphocytes consist of values are between 500 and 1000 while its values are between 1000 and 4000, 86 % of the time for patients who discharged within two weeks. This again validate the fact the lower lymphocytes count is critical in determining COVID severity and LoS.
• In the start of Covid-19, medical community tried many treatments without much evidence. It is important to understand what medications based on lessons learned could be useful to treat infections caused by new strains of viruses as viable epidemic response strategies. Our study shows that drug such as Hydroxychloroquine, Favipiravir, reduces the patients LoS. CRFI results on patients who stayed less than a week in hospital show 51% of the rules consist of antibiotic medications, while who discharged in less than 2 weeks, 52 % of the rules consist of antiviral medication. These analysis shows that the usage of antiviral and antibiotic medication has reduced patient’s LoS.

5. Limitation and Future Directions

This is a single-institute retrospective cohort study to predict LoS. Using multi-center data, it is possible to further evaluate robustness of proposed framework. Future work will focus on acquisition of data for different ethnicities and countries. Our results also did not find every possible combination responsible for COVID severity and LoS, it only found the prominent one based on support and confidence of association mining model. There is a chance that some important cohort of risk factors are missed because they are not present in the data.

6. Conclusions

Predicting patients LoS is a complex task as it depends on many factors such as patient’s history, existing comorbidities, and socio-economic factors. The evaluation shows that state-of-art tab transformer along with association rule mining can effectively predict the LoS and compare how combination of risk factors can help in predicting and reducing the LoS in the hospitals. The LoS of Covid-19 patients varies with respect to many factors such as severity of illness, comorbidities, other cohorts of risk factors. The proposed framework presented can be used for predicting LoS for infectious diseases along with many other critical diseases such as pulmonary disease, cardiovascular disease.