EDBERT: Predicting Emergency Department Disposition Using a BERT-Based Architecture

1. Introduction

Australia faces an ever-growing demand for emergency care and hospital admissions [1]. Emergency Departments (EDs) provide 24-hour assistance for people with severe or life-threatening injuries or illnesses, and have been experiencing increases in patient demand that surpass population growth [2]. In 2023–2024, an estimated 9 million patients presented to EDs in Australia [3]—24,658 patients per day—of whom approximately 31% were subsequently admitted to inpatient wards [4].

ED length of stay (LOS) is the duration between a patient’s arrival at the ED and their departure from it [5,6]. EDs are often overcrowded [7], leading to prolonged ED LOS, which correlates with increased morbidity and mortality [8]. Jones et al. reported a time-associated linear increase in all-cause 30-day mortality for patients who remain in the ED for more than 5 h after arrival, with one additional death occurring for every 82 patients delayed for over 6 to 8 h [9]. Early identification of patients who will require admission can therefore improve ED flow, optimise resource allocation, and ultimately enhance patient outcomes.

The triage process serves as the first point of contact for patients in the ED [10]. Before assessment, demographic information and the reason for presentation are collected and documented in the triage note by a triage nurse [11]. The presenting problem, also referred to as the chief complaint [12], provides a concise description of the primary concern that prompted the patient to attend the ED, whereas the triage note is a brief clinical record that typically includes the reason for presentation, relevant symptoms, medical history, and physiological observations used to determine clinical urgency. The quality and completeness of triage documentation can influence the efficiency and effectiveness of early ED assessment and care [13]. Nonetheless, ED triage notes are commonly written in a highly compressed style and may contain abbreviations, symbols, incomplete phrases, and locally used shorthand. The following de-identified example from the study dataset illustrates these characteristics:

Presenting problem: Left axillary pain and rectal bleeding.

Triage note:

SP: this PM, intermittent L) axilla pain, sharp localised, also c/o blood in stool few times this week, small, bright. PHx nil. GCS 15, RR 16, SpO₂ 100%, BP 126/72, T 36.3, pain 2/10–-declines analgesia, BOC 0, nil COVID sx.

This short narrative includes clinically relevant information such as the timing, location, and characteristics of the pain; the additional report of rectal bleeding; previous medical history; physiological observations; pain severity; and the patient’s decision to decline analgesia. Although such documentation is efficient for rapid clinical recording, its abbreviated and locally variable form creates challenges for consistent interpretation and automated natural language processing (NLP).

With recent significant advancements in NLP, the adoption of large language models (LLMs) in the medical context has gained considerable attention [14], and these models have shown significant efficacy in downstream medical NLP tasks when trained on large datasets of clinical text [15]. LLMs such as Bidirectional Encoder Representations from Transformers (BERT) [16] and its pre-trained variants, including ClinicalBERT [17] and BioBERT [18], have shown significant potential for medical applications. In 2021, Tahayori et al. reported the first Australian application of BERT to free-text ED triage notes, predicting patient disposition with an accuracy of 83% and an area under the receiver operating characteristic curve (AUROC) of 0.88 [19]. Ferri et al. developed a deep ensemble multitask model incorporating BERT to predict emergency jurisdiction from multimodal emergency medical call incidents, reporting an accuracy of 0.80 [20]. Building on these findings, Chang et al. demonstrated that machine learning models integrating both structured and unstructured triage data could outperform emergency physicians in disposition prediction [21], and Chen et al. introduced a deep neural network integrating physicians’ clinical narratives with vital signs that outperformed traditional logistic regression and early warning scores in terms of F1-score [22]. More recently, Chen et al. used BlueBERT, a biomedical domain-specific language model [23], and achieved an AUROC of 0.9014, demonstrating that domain-adapted pre-trained models provide considerable gains over general-purpose LLMs [24].

However, two important gaps remain in this body of work. First, most existing BERT-based disposition models rely on the free-text triage note as a single input, without systematically integrating the readily available structured context that accompanies it at triage, such as the presenting complaint and patient age; indeed, Tahayori et al. anticipated that adding age as a distinct feature would further improve predictive performance [19]. Second, developing, fine-tuning, and maintaining LLMs for healthcare applications is resource-intensive [25]: BERT-BASE contains approximately 110 million parameters [26], demanding substantial computational power that makes it less practical for hospitals with limited computing resources.

In the present study, we developed EDBERT (Emergency Department Bidirectional Encoder Representations from Transformers), a BERT-based architecture organised around a multi-input fusion network that supplements the triage note with the additional context of the presenting complaint and patient age, combining the three sources through learnable weights into a single representation for disposition prediction. To our knowledge, this is the first customised BERT architecture explicitly designed for the ED context. Its encoder stacks are also deliberately lightweight, with reduced memory usage and lower computational cost compared to conventional resource-intensive BERT variants. The main contributions of this study are (i) a fusion-based BERT technique that enriches free-text triage notes with complementary structured context for ED disposition prediction, (ii) a lightweight, domain-tailored encoder design with reduced memory and computational overhead, and (iii) a comparative analysis between EDBERT and other BERT architectures on real-world ED data.

2. Materials and Methods

2.1. Data

We used a retrospective dataset of 570,143 electronic patient triage records from the Emergency Department of the Royal Melbourne Hospital, Australia. Of these, 211,810 (32.2%) presentations resulted in admission and 358,333 (62.8%) in a “Home” disposition. The remaining 5% of presentations involved other disposition categories, such as transfer to another hospital, leaving the ED before completion of care, or death in the ED, and were excluded from this study. Each record included the triage note and presenting complaint as free-text fields, and patient age as an integer value.

Eighty percent of the dataset was used for training the model and the remaining 20% was used as the test set to measure final predictive performance on unseen observations.

2.2. The EDBERT Model

EDBERT is organised around a fusion network that integrates three complementary input sources—triage notes, presenting complaints, and patient age—into a single representation for disposition prediction. The two free-text inputs are encoded by customised, lightweight BERT encoder stacks (66 million parameters in total). We first describe the encoder stack, and then the fusion network that forms the core of the architecture.

2.2.1. Customised BERT Encoder Stack

The transformer encoder stack in EDBERT (Figure 1) is a modified version of the original BERT architecture [16]. While BERT-BASE includes 12 layers with 12 attention heads and a hidden size of 768, these settings are excessive for the comparatively short, less linguistically complex triage notes and presenting complaints [19]. Statistical analysis of the input corpus revealed that unique vocabulary accounts for only ∼1% of the total vocabulary in triage notes (Table 1), indicating low lexical variability. Accordingly, we reduced the number of transformer layers from 12 to 8, the number of multi-head self-attention heads from 12 to 8, the hidden size from 768 to 512, and the feedforward intermediate size from 3072 to 2048. The model’s vocabulary was tailored to include more domain-specific tokens (expanded to 40,000), while the maximum positional embedding of 512 was retained to accommodate the short ED texts. Table 2 summarises these modifications. The empirical analysis of encoder depth that informed the 8-layer design is provided in the Supplementary Materials (Table S1, Figures S1 and S2).

2.2.2. Fusion Network

The fusion network integrates three distinct input sources: triage notes, presenting complaints, and patient age, each providing complementary information for disposition prediction. Triage notes serve as the primary source because they contain rich, unstructured clinical information, including detailed symptom descriptions, severity indicators, contextual factors, and triage nurse observations. However, presenting complaints and age were also included because they provide concise and standardised information that may not always be stated explicitly or consistently within the triage notes. Presenting complaints capture the patient’s primary reason for attending the emergency department, while age provides an important indicator of baseline clinical risk. The inclusion of presenting complaints and age was informed by both prior literature and routine emergency department practice, as these variables provide complementary, routinely available information that may not be fully captured in the triage note. Previous studies have similarly shown that routinely collected triage features, including age, triage category, arrival mode, and vital signs, are among the strongest contributors to predictive performance [27].

Each input is first processed through a dedicated encoding pipeline. The triage notes and presenting complaints are tokenised and encoded separately using two identical customised encoder stacks (Section 2.2.1), pre-trained on domain-specific triage data, to produce contextual embeddings; during training, each embedding passes through a dropout layer to reduce overfitting. The age input is projected into the same embedding space as the BERT hidden layers via a fully connected linear transformation.

To combine the input sources, the model applies learnable scalar weights to each of the three embeddings, and the weighted embeddings are summed to form a single fused representation:

$$\mathrm{Fusion}=w_{\mathrm{text}}·E_{\mathrm{text}}+w_{\mathrm{pc}}·E_{\mathrm{pc}}+w_{\mathrm{age}}·E_{\mathrm{age}}$$

(1)

This fused embedding is then passed through the final classification layer, which predicts the ED disposition (Figure 2).

2.3. Model Training

EDBERT was trained in two stages. In the first stage, the two customised encoder stacks were pre-trained using the Masked Language Modelling (MLM) technique [16]: 15% of the input tokens were randomly selected for masking, and the model was trained to reconstruct their original values from the surrounding context. This enables the model to learn deep, context-sensitive representations and capture the complex semantic and syntactic relationships within triage texts. In the second stage, the full model was trained end-to-end by calculating the loss between the predicted and actual disposition labels using the cross-entropy loss function.

The weights applied to the three input sources were determined through grid search optimisation: $w_{\mathrm{text}}=0.65$ for triage notes, $w_{\mathrm{pc}}=0.20$ for presenting complaints, and $w_{\mathrm{age}}=0.15$ for age. The Adam optimiser [28] was used with a learning rate of $2\times {10}^{-5}$ and a weight decay of 0.01. The batch size was set to 64 for training and 32 for evaluation, and a dropout rate of 0.5 was applied during training to mitigate overfitting. Table 3 summarises the hyperparameter configuration.

2.4. Evaluation

Predictive performance on the held-out test set was assessed using accuracy, macro-averaged precision, recall, and F1-score, the receiver operating characteristic (ROC) curve, and the area under the ROC curve (AUC). EDBERT was benchmarked against BERT-Base configurations with 2 to 12 transformer layers (in increments of two). Two ablation analyses were conducted to justify the final design: (i) the effect of encoder depth on predictive performance, parameter count, and training time, and (ii) the contribution of MLM pre-training. The complete results of these analyses are reported in the Supplementary Materials.

3. Results

The triage notes had a maximum length of 101 words and an average length of 391 characters, including spaces.

EDBERT achieved the highest performance of all evaluated models (see supplementary materials), with an accuracy of 83.26% and macro-averaged precision, recall, and F1-score of 82.14%, 81.85%, and 81.99%, respectively, outperforming the standard 12-layer BERT-Base model across all metrics (Table 4). Notably, EDBERT also exceeded the depth-matched 8-layer BERT-Base model trained on triage notes alone (83.26% vs. 82.08% accuracy; Supplementary Table S1), indicating that the improvement stems primarily from the fusion of complementary inputs and domain-adaptive pre-training rather than from model capacity. The ROC curve of EDBERT on the test set is shown in Figure 3, with an AUC of 0.91, indicating strong discriminative ability between the two disposition classes.

The supporting analyses that informed the final architecture are summarised as follows, with full details in the Supplementary Materials. First, varying the encoder depth of BERT-Base from 2 to 12 layers produced only marginal accuracy differences (81.47% to 82.23%; Supplementary Table S1 and Figure S1), while parameter count and training time per epoch increased substantially with depth (Supplementary Figure S2). Deeper models did not yield performance gains, which may be attributed to the low lexical variability of the triage notes (1.01% unique word ratio; Table 1), potentially leading to overfitting in larger models. The 8-layer configuration offered the most favourable balance between predictive performance and computational efficiency, and was therefore adopted for EDBERT. Second, MLM pre-training on triage notes improved EDBERT’s accuracy from 82.18% to 83.26% (Supplementary Table S2 and Figure S4), confirming the value of domain-adaptive pre-training.

4. Discussion

In the present study, we designed and evaluated a domain-tailored BERT architecture whose central component is a fusion network that enriches the free-text triage note with the additional context of the presenting complaint and patient age, predicting ED disposition with an accuracy of 83.26% and an AUC of 0.91 while using 66 million parameters—approximately 40% fewer than BERT-BASE. These results compare favourably with previous work. Tahayori et al. reported an accuracy of 83% and an AUROC of 0.88 using a full-size BERT model on triage notes alone [19], and Chen et al. achieved an AUROC of 0.9014 using BlueBERT, a domain-specific model pre-trained on large biomedical corpora [24]. EDBERT achieved comparable or better discriminative performance with a substantially smaller architecture, without relying on external biomedical pre-training corpora.

The improved performance of EDBERT relative to larger, general-purpose BERT configurations can be explained by several factors. First, and most importantly, the fusion network supplements the triage narrative with the complementary context of the presenting complaint and patient age. A depth-matched 8-layer BERT-Base model trained on triage notes alone reached 82.08% accuracy, whereas EDBERT reached 83.26%, indicating that enriching the input representation—rather than adding model capacity—drives the gain; the ablation results suggest this enrichment acts in combination with domain-adaptive pre-training, as the fusion-based model without MLM pre-training reached 82.18% (Supplementary Table S2). The grid-searched fusion weights confirm the dominant contribution of the triage narrative ($w_{\mathrm{text}}=0.65$), consistent with prior evidence that free-text triage narratives carry the greatest predictive signal for disposition [11,21], while the presenting complaint ($w_{\mathrm{pc}}=0.20$) and age ($w_{\mathrm{age}}=0.15$) contribute additional discriminative context that the note alone does not capture. Second, domain-adaptive MLM pre-training on triage text allowed the model to learn the specialised terminology, abbreviations, and contextual patterns characteristic of ED documentation, which a general-purpose model has less opportunity to capture; removing MLM pre-training reduced accuracy by more than one percentage point. Third, triage notes are short and telegraphic, with a maximum length of 101 words in our dataset, and exhibit very low lexical variability (1.01% unique word ratio); the representational capacity of a 12-layer, 110-million-parameter model is therefore excessive and may promote overfitting, whereas EDBERT’s reduced encoder depth is matched to the linguistic complexity of the input.

The lightweight design of EDBERT has direct practical implications. Full-size LLMs are resource-intensive to fine-tune and maintain [25], which is a barrier for hospitals with limited computing infrastructure. By reducing memory usage and training time while preserving predictive performance, EDBERT is more feasible for local training and deployment within existing hospital systems. Early identification of patients likely to require admission could allow bed allocation to begin at the time of presentation, streamline care, reduce the cognitive load on ED clinicians, and ultimately contribute to reduced ED LOS, which is associated with morbidity and mortality [8,9].

More recent research has also emphasised the importance of uncertainty-aware prediction and effective collaboration between clinicians and predictive models. Abdulai et al. proposed a conformal prediction framework that enables a model to abstain when a case is ambiguous, thereby reducing the risk of overconfident misclassification and supporting safer clinical use [29]. Similarly, Barak-Corren et al. directly compared clinician and algorithmic predictions in a prospective observational study: physicians correctly predicted 57% of admissions with a specificity of 88%, whereas a random forest model using EHR data from the first hour of the ED visit achieved a sensitivity of 69% and a specificity of 90%, and combining clinician judgement with algorithmic predictions produced the highest positive predictive value of 86% [30]. These findings indicate that predictive performance alone is insufficient for safe deployment in a high-risk clinical environment. Future work should therefore incorporate uncertainty estimation or abstention mechanisms, allowing EDBERT to defer ambiguous cases for clinician review, and evaluate the model within a human-in-the-loop framework to determine whether its predictions complement, rather than replace, clinician judgement. Prospective shadow-mode evaluation and subsequent clinician–model collaboration studies would help establish the safety, reliability, and practical value of EDBERT in real-world ED workflows.

4.1. Limitations

This study has several limitations. First, it used data from a single site. Because triage documentation style varies considerably between hospitals, the model may need to be re-trained on local data before implementation elsewhere, and a multi-centre study is required to evaluate the generalisability of the proposed architecture. Second, presentations with dispositions other than “Admit” and “Home” (approximately 5% of the cohort) were excluded, so the model’s behaviour on these less common outcomes remains untested.

Third, EDBERT currently produces point predictions without uncertainty estimates or an abstention mechanism. Recent work has emphasised the importance of uncertainty-aware prediction for safe clinical use: Abdulai et al. proposed a conformal prediction framework that enables a model to abstain when a case is ambiguous, reducing the risk of overconfident misclassification [29]. Incorporating such mechanisms would allow EDBERT to defer ambiguous cases for clinician review rather than producing potentially unreliable predictions.

Finally, EDBERT was evaluated retrospectively, without clinician interaction, so whether its predictions complement rather than replace clinician judgement remains untested. Barak-Corren et al. directly compared clinician and algorithmic predictions in a prospective observational study: physicians correctly predicted 57% of admissions with a specificity of 88%, whereas a random forest model using EHR data from the first hour of the ED visit achieved a sensitivity of 69% and a specificity of 90%, and combining clinician judgement with algorithmic predictions produced the highest positive predictive value of 86% [30]. These findings indicate that predictive performance alone is insufficient for safe deployment in a high-risk clinical environment; prospective shadow-mode evaluation and subsequent clinician–model collaboration studies within a human-in-the-loop framework would help establish the safety, reliability, and practical value of EDBERT in real-world ED workflows.

5. Conclusions

This study introduced EDBERT, a specialised transformer-based architecture for ED disposition prediction whose central component is a fusion network that integrates the free-text triage note with the complementary context of the presenting complaint and patient age through learnable weights. Supplementing the triage narrative with this additional context, together with domain-adaptive MLM pre-training, enabled EDBERT to consistently outperform larger single-input BERT baselines. By further tailoring the encoder stacks to the specific characteristics of ED text—including reduced transformer layers, attention heads, and hidden dimensions—EDBERT delivers this performance with fewer parameters and reduced training time. Overall, EDBERT offers a scalable, high-performing, and resource-efficient solution for predicting patient disposition in emergency settings, supporting timely and informed clinical decision-making. Future work will explore the generalisability of EDBERT across multiple hospital sites, the incorporation of additional patient features, uncertainty-aware prediction, and real-time deployment in ED triage workflows.