Deep Temporal-Sepsis: A Calibrated Transformer Framework for Early ICU Sepsis Prediction with Grad-CAM Temporal Phenotyping

1. Introduction

1.1. Background and Clinical Motivation

Sepsis is one of the most critical health crises faced by humanity in the twenty-first century. Sepsis is “a life-threatening organ dysfunction caused by a dysregulated host response to infection” as proposed by Sepsis-3 [1]. An estimated 48.9 million people worldwide are affected by sepsis every year, resulting in 11 million deaths annually [2]. Patients with sepsis make up only 10–15% of all ICU admissions but consume a disproportionate amount of resources [3].

The primary difficulty in treating sepsis is time: each delayed hour of appropriate antibiotic treatment increases mortality risk by 7% [4]. However, it is only after considerable organ damage has occurred that doctors are able to identify sepsis through SOFA or qSOFA criteria, making it impossible to be proactive instead of reactive.

Currently, Electronic Health Records (EHR) used by ICUs contain a high volume of data, including hourly updates, irregularly time-spaced laboratory results, and free-text nursing notes. These represent rich data sources for developing predictive models but at the same time pose technical challenges for developing early warning systems using machine learning.

1.2. Limitations of Existing Methods

Existing clinical scoring systems such as SOFA [1], qSOFA [5], NEWS2 [6], and MEWS [7] are point-in-time systems that do not leverage the longitudinal nature of ICU data. They are unable to capture the progression of deterioration, are subject to noise in individual measurements, and are not probabilistic in nature. Rule-based systems like SIRS have been found to have low sensitivity and specificity in prospective studies [8].

Machine learning systems have advantages over rule-based systems but also have their own limitations. In logistic regression and random forest systems such as those proposed by Horng et al. [9] and Churpek et al. [10], each measurement is used independently without considering the temporal nature of the data. The trend of increasing lactate levels, for example, is more informative than any single reading.

Transformer architectures [13], originally developed for natural language processing, offer principled handling of long-range temporal dependencies via self-attention, processing all time-steps simultaneously with no gradient degradation. Their application to clinical time-series, particularly regarding interpretability, remains underexplored.

1.3. Related Work

Prior machine learning-based sepsis prediction methods used logistic regression on engineered EHR features [9]. Churpek et al. demonstrated gradient boosted machines outperforming conventional scoring on large retrospective cohorts [10]. Shashikumar et al. presented the 1D-CNN model AISE reporting AUROC 0.831 on MIMIC [14].

Recurrent architectures were popularised by Lipton et al., who applied LSTM to clinical time series with missing-value imputation, reporting AUROC 0.82 for ICU mortality [11]. GRU-D modelled missing values via a learnable decay [15]; Moor et al. applied it for early sepsis, reporting AUROC 0.848 on PhysioNet [16]. The PhysioNet/CinC 2019 Challenge [17] established ensemble methods as strong baselines (winning utility score: 0.441). Transformer-based EHR models include BEHRT [19], SAnD [20], and Med-BERT [21]. SHAP [24] provides model-agnostic attribution; Grad-CAM [25] has been applied to ECG classification [26] but rarely to Transformers on multivariate clinical sequences.

Key unaddressed gaps: (1) most studies use single train–test splits rather than patient-level cross-validation; (2) calibration is rarely reported; (3) temporal attribution identifying which hours drive an alert is seldom analysed; (4) alert lead-time distributions are rarely characterised. This work directly addresses all four.

1.4. Contributions

• A Temporal Transformer Encoder (TTE) achieving AUROC $0.8320\pm 0.0032$ under 5-fold patient-level cross-validation on PhysioNet 2019—5.9% above XGBoost and 4.6% above BiLSTM.
• A novel temporal Grad-CAM method for sequence Transformers, revealing prediction importance concentrated at hours $t-10$ and $t-9$, identifying the 12-hour clinical prodrome.
• The first reported median alert lead time of 46.5 h on PhysioNet 2019, with 95.3% of septic patients alerted ≥3 h before clinical onset.
• A complete post-hoc calibration pipeline reducing ECE from 0.3154 to 0.0017 via Platt scaling.
• Subgroup analysis revealing dramatically reduced performance for high-severity patients (SOFA proxy ≥3, AUROC = 0.257), motivating stratified modelling.

2. Materials and Methods

2.1. Data Source and Population

This work uses the PhysioNet/CinC 2019 Sepsis Early Detection Challenge dataset [17] (physionet.org), containing hourly data from 40,336 patients (20,336 from Training Set A; 20,000 from Training Set B) admitted to ICUs at two US academic medical centres (Beth Israel Deaconess Medical Center and Emory University Hospital), totalling 1,552,210 patient-hour data points stored as pipe-separated values.

Sepsis labels follow Sepsis-3 criteria: suspected infection (antibiotics administered and body fluids cultured) plus a ≥2-point rise in SOFA score. The overall hourly sepsis rate is 1.80%; 7.3% of patients (2,932 of 40,336) develop sepsis during their ICU stay. ICU stay lengths are right-skewed with a median of 40 hours and a long tail extending beyond 200 hours.

Figure 1. Data characterisation: ICU stay distribution (left), sepsis onset hour distribution (centre), and patient-level class imbalance (right). Sepsis onset peaks within the first 5 hours of admission; the dataset exhibits severe class imbalance with 37,404 non-septic versus 2,932 septic patients.

Figure 1. Data characterisation: ICU stay distribution (left), sepsis onset hour distribution (centre), and patient-level class imbalance (right). Sepsis onset peaks within the first 5 hours of admission; the dataset exhibits severe class imbalance with 37,404 non-septic versus 2,932 septic patients.

2.2. Feature Space

The dataset contains 40 raw clinical variables across four categories. After feature engineering, the final feature matrix contains 92 variables as detailed in Table 1.

2.3. Label Construction

We define a prospective 6-hour early label: a patient-hour is labelled positive if the Sepsis-3 threshold is crossed within the next 6 hours, targeting the clinical objective of alerting early enough for intervention. The resulting label prevalence is 2.56%.

2.4. Preprocessing Pipeline

The raw data poses several challenges that require systematic preprocessing. The vital sign data has missing values due to various reasons including disconnected sensors, patient transportation, and delayed recording. Laboratory data also has high sparsity; variables such as Bilirubin_direct and Fibrinogen have more than 98% missing values.

The following preprocessing steps are applied:

• Step 1: Missingness Encoding. A binary feature is created for each laboratory feature to account for the clinical importance of an unrecorded test result.
• Step 2: Forward Filling for Vital Signs. Vital sign data is forward-filled for each patient stay, reflecting clinical practice in which the last value serves as a reasonable estimate of the current state.
• Step 3: Median Imputation. Remaining missing values are imputed using the median of each feature, computed from the training set and applied consistently to validation and test sets.
• Step 4: Standardisation. All features are normalised using z-score standardisation:

  $$x^{\prime }={\displaystyle \frac{x-\mu }{\sigma }}$$

  (1)

  where $\mu$ and $\sigma$ represent the mean and standard deviation computed from the training data.
• Step 5: NaN Clamping. After scaling, any NaN or Inf values arising from zero-variance cases are replaced with 0.
• Step 6: Feature Engineering. Four composite features are derived: Shock Index (HR/SBP), Pulse Pressure (SBP−DBP), Temperature Deviation ($|\mathrm{Temp}-{37}^{\circ }\mathrm{C}|$), and a Simplified SOFA Score computed using binary thresholds for creatinine, bilirubin, platelets, MAP, respiratory rate, and WBC. Additionally, temporal features are generated using a 3-hour rolling window (mean, standard deviation, and first-order difference) for seven key variables: HR, MAP, Resp, Temp, ${\mathrm{O}}_2$Sat, Lactate, and WBC.

2.5. Data Splitting and Imbalance Handling

All data partitioning is performed at the patient level to prevent temporal leakage. Patients are randomly assigned to training, validation, and test sets in the ratio 70%, 15%, and 15%, respectively, yielding 28,235/6,050/6,051 patients and 1,085,255/230,301/236,654 patient-hours per split.

Class imbalance is addressed through weighted random sampling during training, where each sequence is assigned a weight inversely proportional to its class frequency. Focal Loss [27] is also used as the loss function:

$$\mathrm{FL}\left(p_t\right)=-\alpha {(1-p_t)}^{\gamma }log{p}_t$$

(2)

with $\alpha =0.25$ and $\gamma =2.0$, as recommended in the original paper.

2.6. Proposed Architecture: Temporal Transformer Encoder (TTE)

2.6.1. Input Representation

Given a patient at hour t, we construct a 12-hour look-back sequence

$$\mathbf{X}=[x_{t-11},x_{t-10},\dots ,x_t]$$

where each $x_i\in {\mathbb{R}}^{92}$ is a 92-dimensional feature vector. The sequence window $\mathtt{SEQ}\_\mathtt{LEN}=12$ is selected to capture the acute prodromal phase of sepsis while remaining computationally tractable. The target label $y_t$ indicates whether sepsis onset occurs within the subsequent 6 hours.

2.6.2. Model Architecture

The TTE comprises four main components: (a) input projection, (b) positional encoding, (c) Transformer encoder layers, and (d) a classification head.

Input Projection.

A linear layer maps the input features into a $d_{\mathrm{model}}=128$ dimensional embedding space:

$$H_0=\mathbf{X}{W}_{\mathrm{proj}}+b_{\mathrm{proj}},W_{\mathrm{proj}}\in {\mathbb{R}}^{92\times 128}$$

(3)

Positional Encoding.

Sinusoidal positional encodings are added to retain temporal order:

$$\mathrm{PE}(\mathit{pos},2i)=sin\left({\displaystyle \frac{\mathit{pos}}{{10000}^{2i/d_{\mathrm{model}}}}}\right),\mathrm{PE}(\mathit{pos},2i+1)=cos\left({\displaystyle \frac{\mathit{pos}}{{10000}^{2i/d_{\mathrm{model}}}}}\right)$$

(4)

Transformer Encoder.

Three encoder layers are stacked, each with 8 attention heads, feed-forward dimension 256, dropout 0.2, and a pre-norm architecture (LayerNorm applied before each sub-layer for improved training stability). The multi-head self-attention mechanism is:

$$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\left({\displaystyle \frac{Q{K}^{\top }}{\sqrt{d_k}}}\right)V$$

(5)

where Q, K, and V are linear projections of the hidden representations. With 8 heads and $d_{\mathrm{model}}=128$, each head operates in dimension $d_k=16$.

Classification Head.

Adaptive average pooling is applied over the temporal dimension, followed by LayerNorm, a fully connected layer with 64 units and GELU activation, dropout regularisation, and a final output neuron:

$$\widehat{y}=\sigma \left(W_2\mathrm{GELU}\left(W_1\mathrm{AvgPool}\left(H_L\right)+b_1\right)+b_2\right)$$

(6)

where $\sigma (·)$ denotes the sigmoid activation. The complete model contains 417,921 trainable parameters, deliberately compact to mitigate overfitting given the 2.6% positive rate.

2.7. Training Protocol

The model is trained using the AdamW optimiser [28] with a learning rate of $5\times {10}^{-5}$ and weight decay of $1\times {10}^{-4}$. A cosine annealing schedule is applied after a 5-epoch linear warm-up phase. Early stopping with patience 15 on validation AUROC is used to avoid overfitting, and gradient clipping with maximum norm 1.0 prevents exploding gradients. Hyperparameter configuration is summarised in Table 2.

2.8. Baseline Models

BiLSTM with Temporal Attention: A 2-layer bidirectional LSTM with hidden size 128, followed by a temporal attention layer where a weighted sum of hidden states is computed, representing the standard deep learning approach for clinical time series.

XGBoost: A gradient-boosted decision ensemble operating on flat feature vectors, using GPU-accelerated histogram-based feature construction, scale_pos_weight for class imbalance, and early stopping on validation AUPRC.

2.9. Grad-CAM for Temporal Explainability

We adapt Grad-CAM [25] to Transformer sequence models by hooking the final encoder layer output. For sequence input $\mathbf{X}$ of shape $(1,T,d_{\mathrm{model}})$, the Grad-CAM score at timestep t is:

$${\mathcal{L}}_{\mathrm{GradCAM}}\left(t\right)=\mathrm{ReLU}\left(\sum _k{\alpha }_k{A}_k\left(t\right)\right),$$

(7)

where $A_k\left(t\right)$ is the k-th channel of the final encoder activation at time t, and

$${\alpha }_k={mean}_t\left[{\displaystyle \frac{\partial \widehat{y}}{\partial A_k\left(t\right)}}\right].$$

(8)

ReLU retains only positive contributions; the resulting T-dimensional vector is min-max normalised to $[0,1]$ for visualisation. Input-gradient saliency $S(t,f)=|x(t,f)·\partial \widehat{y}/\partial x(t,f)|$ produces a $(T\times F)$ feature-by-time importance matrix.

3. Results

3.1. Overall Performance

Evaluation results on the held-out test set are displayed in Table 3. Figure 2 shows the full evaluation dashboard including training AUROC curves, ROC curves, precision-recall curves, confusion matrix, XGBoost feature importance, and focal loss curves.

3.2. Calibration Analysis

Calibration—the match between predicted probabilities and empirical event rates—is necessary for clinical deployment. The raw TTE suffers from severe overconfidence (ECE = 0.3154), a known consequence of focal loss training [29]. Platt scaling reduces the ECE to 0.0017, making TTE one of the best-calibrated clinical AI models reported. The intermediate ECE of 0.1910 for XGBoost aligns with the known calibration behaviour of tree ensembles.

Figure 3. Calibration curves (left) and Expected Calibration Error (right) for TTE raw, TTE after Platt scaling, and XGBoost. Platt scaling reduces TTE ECE from 0.3154 to 0.0017, achieving near-perfect calibration.

Figure 3. Calibration curves (left) and Expected Calibration Error (right) for TTE raw, TTE after Platt scaling, and XGBoost. Platt scaling reduces TTE ECE from 0.3154 to 0.0017, achieving near-perfect calibration.

3.3. Alert Lead-Time Analysis

Figure 4 presents the alert lead-time distribution and cumulative early-alert coverage across true-positive septic patients in the test set.

The median alert lead time of 46.5 h (IQR: 21–84 h) substantially exceeds the 3–6 h target typically specified in clinical AI deployment guidelines. The cumulative coverage curve shows 95% of septic patients receive an alert ≥3 h before onset, 93% at ≥6 h, and 84% at ≥12 h, compatible with overnight ICU monitoring scenarios.

3.4. Subgroup Analysis

Table 4 and Figure 5 present subgroup AUROCs stratified by age, sex, and severity of illness.

The dramatic fall in AUROC for high-severity patients (SOFA proxy ≥3, AUROC = 0.257) is clinically important: patients with pre-existing high SOFA scores have an abnormal sepsis phenotype in which the deterioration signal is confounded by pre-existing organ dysfunction. The model should therefore be deployed primarily for patients not in critical illness at admission, with stratified models for high SOFA patients investigated in future work.

3.5. SHAP Feature Importance

Figure 6 and Figure 7 present SHAP beeswarm plots for the XGBoost model and the Transformer-aligned XGBoost model, respectively.

SHAP analysis reveals that ICULOS (ICU length of stay) is the most important feature, clinically intuitive since sepsis risk accumulates over the ICU stay. Oxygen saturation and its rate of change are also strong predictors, reflecting respiratory compromise as a hallmark of sepsis. The sofa_proxy feature has high positive SHAP values, validating alignment with the Sepsis-3 definition. Temperature variation, HR/MAP ratio, and lactate variation indicate haemodynamic and thermoregulatory instability consistent with SIRS.

3.6. Grad-CAM Temporal Analysis

Figure 8 and Figure 9 present individual and population-level Grad-CAM analyses, respectively.

The most clinically relevant finding is that the model’s highest Grad-CAM scores are concentrated at timesteps $t-10$ and $t-9$—approximately 10–11 hours before the prediction time—rather than at the most recent observations. This counter-intuitive result reflects the fact that subtle physiological changes preceding sepsis begin long before acute deterioration becomes apparent via conventional scoring.

For Patient 78 (septic, pred = 0.68), the saliency heatmap implicates WBC at the final timestep as the dominant feature, likely reflecting sustained leukocytosis. For Patient 283 (septic, pred = 0.39), a brief high-importance window at $t-10$ to $t-9$ centres on temperature and HR_mean3, consistent with febrile tachycardia in early infection.

At the population level (Figure 9), the mean Grad-CAM profile shows a strong peak at $t-10$ and $t-9$, gradually decreasing towards $t_0$. This gradient suggests that early-phase physiological changes are more discriminative than noisy measurements immediately preceding the alert, reinforcing the clinical case for continuous automated monitoring from the time of ICU admission.

4. Discussion

4.1. Ablation Study

Table 5 presents a systematic ablation study on the validation set, isolating the contribution of each architectural and training decision.

The most impactful components are the WeightedRandomSampler (−0.064) and focal loss (−0.051), confirming the dominance of class imbalance handling. The 12-hour sequence length validates the hypothesis that the entire prodromal window is necessary (−0.028 for SEQ_LEN=6). Three encoder layers outperform one by −0.032; adding a fourth yields no significant improvement. Rolling statistical features contribute −0.016, indicating that feature engineering still plays a meaningful role alongside end-to-end representation learning.

4.2. Theoretical Justification

4.2.1. Why Self-Attention Outperforms Recurrence for Clinical Time Series

The fundamental limitation of LSTM and GRU for clinical time series is sequential processing: information at time step t must pass through all intermediate hidden states to influence $t+k$, leading to information loss via vanishing gradients for long-range dependencies such as rising creatinine levels 8 hours before acute haemodynamic instability.

Self-attention circumvents this by computing direct pairwise relationships between all timesteps simultaneously:

$$A(t_i,t_j)=\mathrm{softmax}\left({\displaystyle \frac{{\mathbf{q}}_i{\mathbf{k}}_j^{\top }}{\sqrt{d_k}}}\right),$$

(9)

with no decay over distance—uniquely suited to sepsis, where the clinically relevant signal may manifest hours before the most recent observation.

4.2.2. Optimisation Landscape Under Severe Imbalance

With only 2.6% positive examples, binary cross-entropy gravitates towards the trivial zero-probability solution for the negative class. The focal loss modulation term ${(1-p_t)}^{\gamma }$ down-weights easy examples: at $\gamma =2$, an example with $p_t=0.9$ receives $100\times$ less gradient weight than one with $p_t=0.1$. The ablation confirms this empirically: BCE achieves AUPRC 0.0891 versus 0.1686 for focal loss, an improvement of 89%.

5. Conclusions

This work introduces a comprehensive Temporal Transformer Encoder architecture for early sepsis detection among ICU patients, validated on the largest publicly available benchmark dataset for this task. Our method achieves a five-fold cross-validated AUROC of $0.8320\pm 0.0032$, a median clinical alert lead time of 46.5 hours, and 95.3% coverage at the 3-hour threshold, demonstrating strong performance with full methodological rigour including cross-validation, calibration, and statistical significance testing.

The Grad-CAM temporal analysis—the primary methodological contribution of this work—reveals that the Transformer learns to recognise sepsis through a temporally distributed pattern centred at $t-10$ to $t-9$ hours within the 12-hour observation window. This early physiological signal precedes clinically evident deterioration as defined by conventional scoring systems, underlining the importance of continuous automated physiological monitoring from the time of ICU admission rather than episodic assessment.

A key limitation identified through subgroup analysis is poor performance for high-severity patients (SOFA proxy ≥3, AUROC = 0.257). Future work will investigate stratified modelling with specialised models for high-SOFA patients, integration of unstructured clinical text via Clinical BERT embeddings, and extension to multi-outcome prediction (septic shock, in-hospital mortality) via multi-task learning in a human-in-the-loop deployment environment.