Submitted:
29 October 2026
Posted:
02 November 2026
You are already at the latest version
Abstract
Clinical decision support requires models that can interpret patient records with short response times and modest computational demands. We present Laya-Med, a clinical adaptation of a 322-million-parameter bidirectional language model that assigns probabilities to predefined outcomes in a single forward pass. We evaluated the approach in two retrospective studies. For acute abdominal diagnosis, we fine-tuned the model on 1,680 MIMIC-IV-Ext Clinical Decision-Making admissions and compared it with three zero-shot large language models and a term frequency–inverse document frequency (TF-IDF) classifier on 360 held-out admissions. Laya-Med achieved macro-F1 of 81.1% (95% confidence interval 76.1–85.7) and accuracy of 85.6%, compared with 84.6% and 86.9% for Qwen3.8-27B. Mean processing time under batched inference was 10.8 ms per case, versus 0.60 s for Qwen3.8-27B on the same GPU. In a second study, we independently fine-tuned Laya-Dengue on Portuguese narratives generated from dengue surveillance records. On 2,157 test cases, its macro-averaged area under the receiver operating characteristic curve was 95.2%, compared with 94.0% for TF-IDF and 91.1% for MedGemma 27B. For non-fatal severe dengue versus death, the corresponding values were 73.9%, 67.7% and 54.7%. Local interfaces updated predictions in approximately 50 ms. These results support compact, task-adapted encoders as a computationally efficient approach to clinical decision support. Prospective evaluation is needed to establish their performance on clinical workflows and outcomes.
Keywords:
clinical decision support
; language models
; bidirectional encoders
; fine-tuning
; clinical natural language processing
; dengue
1. Introduction
Large language models (LLMs) have attracted considerable interest in clinical decision support following their performance on medical examinations and question-answering benchmarks [1,2,3]. Applying these models to patient records presents a different challenge. Clinical information must be interpreted in context, and the model must distinguish among diagnoses that often share symptoms and laboratory findings. Hager et al. [4] examined this setting using MIMIC-IV-Ext Clinical Decision Making (MIMIC-IV-Ext-CDM) [5], a benchmark of 2,400 emergency admissions from MIMIC-IV [6] for four acute abdominal conditions. Open-access LLMs with 70 billion parameters performed worse than physicians and were sensitive to the order and quantity of the information provided [4]. This benchmark provides a setting in which to compare alternative approaches to diagnosis from clinical text.
Model selection also depends on the computational demands of the application. Autoregressive LLMs generate answers token by token, making repeated inference costly when predictions must be updated as a record is entered [7]. Local execution can simplify the handling of sensitive clinical information, while use of online services remains subject to the relevant data use agreement [8]. For decision support, the output must also convey uncertainty. Studies of clinicians and LLMs have found errors in diagnostic probability estimates and limitations in recognizing uncertainty [9,10,11]. Calibration is therefore an additional consideration when model scores are intended to represent clinical risk [12].
Bidirectional encoders offer a different formulation of this problem. They map the input text to a representation from which task-specific outputs can be computed in one forward pass [13,14]. When the candidate diagnoses are defined and labelled records are available, supervised adaptation may provide useful diagnostic performance with much smaller models. The relevant question is how much performance is retained at a lower inference cost. Evaluation in other languages is also needed, given the variation in LLM performance across languages on healthcare questions [15].
In this study, we present Laya-Med, a clinical adaptation of the multilingual Laya encoder, and evaluate this performance–cost trade-off in two settings. The first compares a fine-tuned encoder, three zero-shot LLMs and a conventional text classifier on the same masked MIMIC-IV-Ext-CDM records. The second applies the training approach to Portuguese narratives generated from Brazilian dengue surveillance data, distinguishing mild dengue, non-fatal severe dengue and death. The two models are fine-tuned independently from the same base checkpoint. Together, these experiments assess the use of a compact probabilistic model across different clinical tasks and languages, alongside local interfaces that update predictions as text is entered.
Our contribution is an empirical evaluation of compact clinical encoders under a shared-input diagnostic protocol, with an audit of residual label leakage, class-specific results and measurements of inference cost. The dengue experiment extends this evaluation to an additional language and outcome space. We also describe the probability adjustments and input filters used in the interactive applications, separating their implementation from the retrospective model evaluation.
2. Related Work
2.1. LLMs for Clinical Decision-Making on MIMIC-IV-Ext-CDM
Research using MIMIC-IV-Ext-CDM has increasingly focused on the diagnostic process as a sequence of decisions. Bani-Harouni et al. trained hypothesis-driven agents with supervised and reinforcement learning [16]. Bedi et al. studied coordination in clinical multi-agent systems and found that improvements in individual components did not necessarily improve overall diagnostic accuracy [17]. Shen et al. examined latent diagnostic trajectory learning [18]. Related benchmarks and systems evaluate information gathering, patient interaction and conversational diagnosis [19,20,21,22]. These approaches address different aspects of clinical reasoning. Here, we focus on the complementary setting in which all models receive the same available information and select among a fixed set of diagnoses.
2.2. Compact Models for Clinical Text
Task-specific adaptation can make smaller language models competitive with larger models on clinical and biomedical natural language processing tasks [23,24]. Jiang et al. showed that an encoder trained on health-system notes could support several clinical prediction tasks [25]. Taylor et al. examined models with as few as 25 million parameters and found that the performance of parameter-efficient adaptation depended on the task and model size, with LoRA often approaching full fine-tuning [26]. Laya combines a bidirectional encoder with a structured decision head that returns probabilities over specified options [27]. Its multilingual checkpoint uses mmBERT [28], building on recent developments in bidirectional language modelling [29], and is trained by reinforcement learning with a strictly proper scoring-rule reward [27,30]. We investigate this formulation for clinical classification and interactive inference.
2.3. Probabilistic Computer-Aided Diagnosis
The use of probabilistic models for diagnosis predates contemporary language models [31,32,33]. In acute abdominal pain, de Dombal et al. reported that a computer-aided system was more accurate than the most senior clinician (91.8% vs 79.6%) [34]. Replacing probabilities estimated from patient data with clinicians’ estimates reduced performance [35], and subsequent work evaluated computer-aided teaching in clinical practice [36]. These studies established a rationale for learning diagnostic probabilities from observed cases. Modern text encoders provide a way to estimate such probabilities directly from narrative input, while retaining the need to evaluate calibration in the population of use [12].
2.4. Dengue Prognosis and Clinical Text Beyond ENGLISH
The English-language and United States focus of MIMIC-IV-Ext-CDM limits what can be learned about other clinical settings [4]. Dengue provides a relevant additional task: it has a substantial global burden [37] and remains an important clinical and surveillance problem in Brazil [38,39]. A systematic review of 146 studies identified high risk of bias and limited external validation in dengue prediction models [40]. Brazil’s Notifiable Diseases Information System (SINAN) provides structured records from which Portuguese clinical narratives can be generated [41]. We use these records to examine outcome classification, including the distinction between non-fatal severe dengue and death. This setting evaluates task adaptation beyond English while allowing the limitations of surveillance-derived text to be examined explicitly.
3. Materials and Methods
3.1. Study Design and Reporting
We conducted two retrospective studies. Study 1 compared a fine-tuned encoder with zero-shot LLMs and a supervised TF-IDF baseline for four acute abdominal diagnoses (Figure 1). Study 2 evaluated an independently fine-tuned encoder on Portuguese narratives generated from dengue surveillance records. The target in Study 2 was the recorded outcome of the case. We used TRIPOD+AI and TRIPOD-LLM to inform the reporting of both studies [42,43].
3.2. Ethics Statement
Study 1 used only de-identified data from MIMIC-IV-Ext-CDM, accessed under the PhysioNet credentialed data use agreement; the creation of MIMIC-IV was approved by the Institutional Review Board of the Beth Israel Deaconess Medical Center, which waived informed consent [6]. No record was sent to an external model API. Study 2 used only de-identified dengue notification records from SINAN, which are publicly available through OpenDataSUS [41], with no contact with patients. Under Brazilian National Health Council Resolution No. 510/2016 (Art. 1, sole paragraph, item II) [44], research that uses publicly accessible information, as defined by Law No. 12,527/2011, does not require review by a research ethics committee. All data were handled in accordance with the Brazilian General Data Protection Law (Law No. 13,709/2018) [45].
3.3. Study 1: Diagnosis of Acute Abdominal Pathologies
3.3.1. Data Source and Cohort
We used version 1.1 of MIMIC-IV-Ext-CDM [4,5], derived from the MIMIC-IV database of the Beth Israel Deaconess Medical Center [6] and distributed through PhysioNet [46]. The dataset contains 2,400 emergency admissions for acute abdominal pain with a primary discharge diagnosis of appendicitis (n = 957; 39.9%), cholecystitis (n = 648; 27.0%), pancreatitis (n = 538; 22.4%) or diverticulitis (n = 257; 10.7%).
3.3.2. Model Input and Leakage Control
For each admission, we concatenated the history of present illness, the physical examination and the laboratory results into one text (mean length, 277 words), which was given unchanged to all systems. To reduce explicit label leakage, we replaced the names of the diseases and of the procedures that treat them with “[redacted]”, removed sentences that mention imaging studies from the history and the physical examination, and removed post-operative findings from the physical examination. Cleaning reduced the admissions with explicit disease or surgery names in the history from 18.2% to 0% and those with mentions of imaging from 32.3% to 0%; residual indirect cues, such as earlier post-operative findings in the history and the frequency of the “[redacted]” marker, are reported by pathology in Table A1. In a sensitivity analysis, we appended the radiology impression, with disease names masked, to the input.
3.3.3. Data Split and Task
Admissions were split into training (n = 1,680), validation (n = 360) and blind test (n = 360) sets by random sampling stratified by diagnosis (70/15/15; seed 42). All modelling and hyperparameter choices were fixed on the validation set, and the test set was evaluated once. The task was the four-class classification of the discharge diagnosis against the MIMIC-IV-Ext-CDM reference labels [4]. All systems chose among the same four pathologies given the complete text at once, which corresponds to a closed-list version of the full-information (“second reader”) evaluation of Hager et al. rather than their autonomous, open-ended sequential protocol [4].
3.3.4. Laya Encoder
We used the multilingual Laya checkpoint, which combines an mmBERT-base bidirectional encoder with a structured decision head (322 million parameters in total) [27,28]. Diagnosis was framed as one categorical question with four options, which yields a probability distribution in a single forward pass; an auxiliary binary question on management (surgical versus conservative) was trained jointly but is not analysed here. All parameters were fine-tuned by maximizing a strictly proper scoring rule, the sum of the logarithmic and spherical scores [30], against smoothed targets that assign a probability of 0.95 to the correct diagnosis. We trained with AdamW [47] (learning rate 3 × 10−5 for the encoder and 10−4 for the decision head; weight decay 0.01), a batch size of 8, three epochs and a maximum input length of 2,048 tokens (above the checkpoint’s default of 1,024; the mmBERT encoder supports up to 8,192). We kept the checkpoint with the highest validation macro-F1 and fitted a temperature on the validation set (T = 1.17) [48]. To quantify training variability, fine-tuning was repeated with five random seeds. The base checkpoint without fine-tuning (zero-shot Laya) was evaluated as a reference. The multilingual checkpoint is distributed within the convaiinnovations/laya repository in its multilingual subdirectory [27].
3.3.5. Generative LLMs
We evaluated MedGemma 27B-text-it [49], MedGemma 1.5 4B-it [50] and Qwen3.8-27B [51] zero-shot. Their model identifiers were google/medgemma-27b-text-it, google/medgemma-1.5-4b-it and Qwen/Qwen3.8-27B, respectively. Parameter counts in Table 2 were obtained from the released weights (27.8 billion for Qwen3.8-27B, a vision-language model used here with text input only; the count includes its vision encoder). All models received the same system prompt, which stated that diagnostic terms had been masked and requested a structured answer (“DIAGNOSIS: …”). Table 1 summarizes the inference settings. The parser read the DIAGNOSIS field; if absent or invalid, it used the first recognized pathology in the answer. Answers without a recognized pathology counted as errors (Qwen3.8-27B, n = 2; MedGemma 27B, n = 1; MedGemma 4B, n = 0). Tokens were counted with each model’s tokenizer. Appendix A provides the prompt and parsing rules.
3.3.6. TF-IDF Baseline
A TF-IDF representation of unigrams and bigrams (up to 25,000 features; minimum document frequency 2; sublinear term frequency) was combined with multinomial logistic regression with balanced class weights (about 100,000 parameters) and trained on the same training set.
3.3.7. Contamination Probe
To test whether the LLMs had memorized MIMIC-IV-Ext-CDM records during pretraining, we gave each model the first 60 words of the history of present illness of 50 test admissions and compared the next 30 generated words with the true continuation using word-level ROUGE-L [52]. As a negative control, each generated continuation was also compared with the continuation of a different admission. We also recorded the number of identical leading words.
3.4. Study 2: Classification of Dengue Outcomes in Portuguese
3.4.1. Cohort and Outcome Labels
The second study used dengue notifications from SINAN, Brazil’s national surveillance system for notifiable diseases [41]. Three mutually exclusive outcomes were defined from the final classification and case evolution fields of the notification form: mild dengue (dengue without warning signs that resolved), non-fatal severe dengue (severe dengue that resolved) and death from dengue [53]. Dengue with warning signs that resolved was excluded because this category was absent from the baseline corpus. The baseline corpus contained 10,029 Portuguese-language anamneses written from SINAN notification records (9,132 mild cases, 190 non-fatal severe cases and 707 deaths). To address the scarcity of severe cases and deaths, we extracted all adult notifications of non-fatal severe dengue and of death from dengue in the public 2023 and 2025 OpenDataSUS microdata [41], keeping patients aged 18–115 years with 0–60 days between symptom onset and notification and using only positively recorded clinical fields, as in the baseline corpus. After excluding 156 records whose age, sex and clinical profile matched a baseline case, we added 1,647 non-fatal severe cases and 2,699 deaths, for a final cohort of 14,375 cases (9,132 mild, 1,837 non-fatal severe and 3,406 fatal). The interval between symptom onset and notification was used for cohort selection. Predictor availability at a common time before deterioration or death was not established.
3.4.2. Anamnesis Generation and Data Split
In the baseline corpus, writing style differed by class (for example, mild cases had shorter notes and fixed closing phrases), so a classifier could learn the outcome from style rather than clinical content. We therefore regenerated all anamneses from their structured variables with a rule-based generator that combined templates with a lexicon of clinical synonyms and never received the outcome. Each record was written in one of four randomly assigned styles (abbreviated note, narrative medical record, report by the patient or companion, or structured history with identification and history of present illness), with random order of symptoms and choice of synonyms. In the training set only, common symptoms were omitted at random with a probability of 0.15 each, race and comorbidities with a probability of 0.30, and one typographical error was added to about 10% of texts; warning signs and severity criteria were never omitted.
Cases were split into training (n = 10,062: 6,392 mild, 1,286 severe and 2,384 deaths), validation (n = 2,156) and test (n = 2,157: 1,370 mild, 276 severe and 511 deaths) sets by stratified random sampling (70/15/15); baseline cases kept their original partition, and all versions of a record stayed in the same set. Two versions of each mild case, six of each severe case and four of each death gave 30,036 training texts, whereas the validation and test sets contained one complete version per case. To assess residual associations between style and outcome, we compared a logistic regression on structured clinical variables with one that also used style features (word count and non-clinical vocabulary); the style features changed the AUC by at most 0.013.
3.4.3. Laya-Dengue Training, Comparators and Calibration
Laya-Dengue was fine-tuned from the same multilingual checkpoint with the recipe of Study 1 (smoothed targets of 0.95; logarithmic plus spherical scoring rule; AdamW with learning rates of 3 × 10−5 and 10−4; batch size 8; three epochs), with outcome classification framed as a three-option categorical question, a maximum input length of 512 tokens and the epoch chosen by validation macro-AUC. The comparators were zero-shot Laya, a TF-IDF and logistic regression baseline and zero-shot MedGemma 27B-text-it, the largest open-weights medical LLM of Study 1 [49]. To obtain probabilities from MedGemma 27B without free-text generation, each record was posed as a multiple-choice question with the three outcome definitions, and the outcome probabilities were taken from the first-token logits of options A, B and C, which held on average 99.5% of the next-token probability mass (Table 1). Because the training cohort was enriched for severe cases and deaths, the probabilistic outputs were adjusted on the validation set to the prevalence of the baseline corpus (91.1% mild, 1.9% severe and 7.0% death), following the prior-shift correction of Saerens et al. [54], by fitting class-specific biases and a shared temperature by maximum likelihood [48]; the TF-IDF outputs were adjusted to the same prior. The reference prevalence was that of the selected baseline corpus.
3.5. Real-Time Interfaces and Input Guardrails
We implemented both fine-tuned models in local web applications that recompute their outputs after each edit to the input. Update times were measured on a laptop with an NVIDIA GeForce RTX 4060 Laptop GPU (8 GB). We evaluated the software operation and response times; clinician use and patient outcomes were outside the study design.
Laya-Med runs after the record reaches 100 tokens and passes a scope detector. The detector requires both a Mahalanobis distance [55] to training-case embeddings below a threshold and a minimum number of clinical terms; thresholds were set using valid clinical texts. To moderate predictions from incomplete text, the interface mixes the model output with the training class frequencies in logit space:
where ntok is the number of input tokens, Iexam and Ilab indicate whether examination and laboratory sections are present, Pk is the model probability for class k, and fk is its training-set frequency. The parameters a, b, e and g were fitted by maximum likelihood on 4,320 prefixes of validation records, using 12 cut points per case in the order history, examination and laboratory results. A softmax converts the adjusted logits to probabilities. The interface also displays a heuristic completeness scale, labelled “case understanding”, computed as max(0.01, size × sections). Here, size = (ntok − 100)/(260 − 100), restricted to the range 0–1, and sections = (1 + examination + laboratory results)/3, with a section counted as complete at 30 tokens. Displayed bars are multiplied by this scale and therefore sum to less than one for incomplete input. The scale describes input completeness rather than a validated measure of model understanding.
w = sigmoid(a + b·log(1 + ntok) + e·Iexam + g·Ilab)
logitk = w·log Pk + (1 − w)·log fk
The Laya-Dengue interface also checks whether the input is a clinical narrative. It combines three signals: the mean cosine distance to the ten nearest training records in the frozen encoder’s 11th-layer embedding space, a zero-shot question on text type (clinical case, recipe or other subject) and a curated clinical lexicon. An input is rejected only when both encoder signals indicate non-clinical text and fewer than two lexicon terms are present. Clinical narratives without typical dengue findings receive an out-of-profile warning. Thresholds were tuned on 36 handcrafted texts covering dengue, partial notes, other conditions and non-clinical text. We assessed filter behaviour on this development set and rejection of clinical input on all 4,313 validation and test records.
3.6. Statistical Analysis
For Study 1, the primary endpoint was macro-F1 on the held-out test set, with 95% confidence intervals (CIs) obtained from 2,000 paired bootstrap resamples. We compared accuracy against the best-performing model using exact McNemar tests [56] with Holm correction [57]. Secondary measures were accuracy, class-specific sensitivity with Wilson 95% CIs [58], parameter count, generated token count and mean processing time per case under batched inference. The statistical tests concerned accuracy; no equivalence or non-inferiority margin was prespecified.
For Study 2, we report one-vs-rest AUC for each class, macro-AUC, AUC for non-fatal severe dengue versus death (restricted to the 787 severe and fatal test cases), sensitivity by class, balanced accuracy with 95% CIs from 2,000 bootstrap resamples of the test cases, and prevalence-weighted log-loss and Brier score [59]. Differences in macro-AUC and in severe-versus-death AUC between Laya-Dengue and the comparators were estimated from 1,000 paired bootstrap resamples stratified by data source (baseline corpus, SINAN 2023 and SINAN 2025).
3.7. Implementation
Text preprocessing and leakage auditing were performed on a local workstation, and model training and inference ran on a Google Colab instance with one NVIDIA A100 80 GB GPU under the credentialed investigator’s account, using deterministic bfloat16 kernels. Analyses used Python 3 with PyTorch 2.11 (CUDA 13.0), Transformers 5.18, scikit-learn and the laya library (v0.4.7) [27].
4. Results
4.1. Evaluation Setting
MIMIC-IV-Ext-CDM v1.1 contains 2,400 emergency admissions with a primary discharge diagnosis of appendicitis (39.9%), cholecystitis (27.0%), pancreatitis (22.4%) or diverticulitis (10.7%) (Figure 1a). The blind test set comprised 360 admissions (144 with appendicitis, 97 with cholecystitis, 81 with pancreatitis and 38 with diverticulitis) and was evaluated once, after all modelling choices had been fixed on the validation set. Every system received the same masked history, physical examination and laboratory results (Figure 1b and Table A1) and chose among the same four pathologies [4].
4.2. Diagnostic Performance
Fine-tuned Laya achieved macro-F1 of 81.1% (95% CI 76.1–85.7) and accuracy of 85.6% (308/360) on the held-out diagnostic task (Table 2). Qwen3.8-27B obtained the highest scores, with macro-F1 of 84.6% (79.9–88.9) and accuracy of 86.9% (313/360). MedGemma 27B reached 79.9% (75.2–84.1) and 83.1% (299/360), respectively, while MedGemma 4B reached 70.2% (65.4–75.2) and 74.2% (267/360). Macro-F1 was 71.6% (66.1–77.0) for TF-IDF and 30.4% (25.4–35.8) for zero-shot Laya. The observed gap between Laya and Qwen3.8-27B was 3.5 percentage points in macro-F1 and five correctly classified admissions. Their accuracy difference was not significant (exact McNemar P = 0.50; Holm-adjusted P > 0.99). MedGemma 27B was less accurate than Qwen3.8-27B (Holm-adjusted P = 0.028), as were MedGemma 4B, TF-IDF and zero-shot Laya (all P < 0.001). Across five fine-tuning seeds, Laya macro-F1 was 81.3 ± 1.2% (mean ± standard deviation).
Table 2.
Diagnostic performance and computational cost on the MIMIC-IV-Ext-CDM blind test set (n = 360).
Table 2.
Diagnostic performance and computational cost on the MIMIC-IV-Ext-CDM blind test set (n = 360).
| System | Parameters | Macro-F1, % (95% CI) | Accuracy, % (n) | Holm P vs. Qwen | Generated tokens per case | Time per case |
|---|---|---|---|---|---|---|
| Qwen3.8-27B (zero-shot) | 27.8 B | 84.6 (79.9–88.9) | 86.9 (313) | Reference | 15.4 | 0.60 s |
| Laya (fine-tuned) | 0.32 B | 81.1 (76.1–85.7) | 85.6 (308) | > 0.99 | 0 | 10.8 ms |
| MedGemma 27B (zero-shot) | 27.0 B | 79.9 (75.2–84.1) | 83.1 (299) | 0.028 | 47.0 | 0.69 s |
| TF-IDF + logistic regression | 0.0001 B | 71.6 (66.1–77.0) | 77.2 (278) | < 0.001 | 0 | 0.6 ms |
| MedGemma 4B (zero-shot) | 4.3 B | 70.2 (65.4–75.2) | 74.2 (267) | < 0.001 | 14.6 | 0.07 s |
| Laya (zero-shot) | 0.32 B | 30.4 (25.4–35.8) | 39.4 (142) | < 0.001 | 0 | 10.7 ms |
P values are exact McNemar tests against Qwen3.8-27B with Holm correction. Times are mean wall-clock processing times per case under batched inference on one NVIDIA A100 80 GB GPU. Class-specific sensitivities and their confidence intervals are reported in Table A2.
4.3. Sensitivity by Pathology
Sensitivity varied more across pathologies than overall accuracy suggested (Figure 2 and Table A2). Laya had the highest sensitivity of any system for pancreatitis (91.4%; 74 of 81) and cholecystitis (87.6%; 85 of 97), whereas Qwen3.8-27B led for appendicitis (93.8%; 135 of 144). For diverticulitis, the least frequent pathology, Laya identified 19 of 38 cases (50.0%; Wilson 95% CI 34.8–65.2), the lowest sensitivity of the four main systems. MedGemma 4B showed the opposite profile, with the highest sensitivity for diverticulitis (86.8%; 33 of 38) and the lowest for pancreatitis (53.1%; 43 of 81). MedGemma 27B and Qwen3.8-27B identified 78.9% (30 of 38) and 68.4% (26 of 38) of diverticulitis cases, respectively. With only 38 diverticulitis cases in the test set, the intervals for this pathology were wide.
4.4. Inference Cost
Laya computed all four diagnostic probabilities in one forward pass, with a mean processing time of 10.8 ms per case under batched inference (Figure 3). The LLMs generated 14.6–47.0 tokens and required 0.07–0.69 s per case. Compared with Qwen3.8-27B, Laya used approximately 86-fold fewer parameters and processed cases approximately 56 times faster on the same GPU. TF-IDF was the fastest system at 0.6 ms per case, although its diagnostic performance was lower. TF-IDF, fine-tuned Laya and Qwen3.8-27B formed the observed performance–time Pareto frontier. Interactive response times were measured separately in the local applications.
4.5. Sensitivity Analysis with Radiology Impressions
Adding the masked radiology impression to the input raised the macro-F1 of every system except MedGemma 4B: Qwen3.8-27B reached 96.9% (95% CI 94.8–98.7), Laya 93.1% (89.9–95.8), MedGemma 27B 91.6% and TF-IDF 81.6%, whereas MedGemma 4B fell to 66.1%. Because radiology impressions often describe findings that reveal the diagnosis, we treat the comparison without radiology as the primary evaluation using history, examination and laboratory findings.
4.6. Contamination Probe
When prompted with the first 60 words of 50 histories from the test set, the LLMs continued them with a mean ROUGE-L of 0.137–0.183 against the true next 30 words, compared with 0.087–0.103 against the continuation of a different record (Table A3). No continuation reproduced the first ten words of the true text, and no case exceeded a ROUGE-L of 0.50, indicating no long verbatim continuation in this sample.
4.7. Real-Time Diagnostic Interface
In the local Laya-Med application, an illustrative pancreatitis record containing history, examination and laboratory results (420 tokens) received a probability of 85.9% after the prefix adjustment, compared with 89.6% before adjustment. The update took 54 ms on a laptop GPU. This example illustrates the operation of the prefix adjustment during local inference.
4.8. Classification of Dengue Outcomes in Portuguese
4.8.1. Discrimination and Sensitivity by Outcome
Laya-Dengue was fine-tuned on 10,062 Brazilian Portuguese dengue cases (30,036 training texts) and evaluated on 2,157 held-out cases: 1,370 with mild dengue (63.5%), 276 with non-fatal severe dengue (12.8%) and 511 deaths (23.7%) (Figure 4). Without fine-tuning, Laya assigned every case to mild dengue (balanced accuracy, 33.3%; Figure 4a). After fine-tuning, Laya-Dengue identified 68.8% of the non-fatal severe cases and 72.2% of the deaths while keeping a sensitivity of 100% for mild dengue (balanced accuracy, 80.4%; 95% CI 78.1–82.5). Zero-shot MedGemma 27B identified a similar proportion of deaths (73.6%) but only 14.1% of the non-fatal severe cases (balanced accuracy, 62.5%; 95% CI 60.7–64.4).
Discrimination followed the same pattern (Figure 4b,c). Laya-Dengue reached one-vs-rest AUCs of 96.9% for mild dengue, 97.2% for non-fatal severe dengue and 91.5% for death (macro-AUC, 95.2%), compared with 95.4%, 91.3% and 86.6% for MedGemma 27B (macro-AUC, 91.1%) and a macro-AUC of 55.9% for zero-shot Laya. The largest gap between Laya-Dengue and MedGemma appeared in separating non-fatal severe dengue from death (n = 787): Laya-Dengue reached an AUC of 73.9%, whereas MedGemma 27B (54.7%) and zero-shot Laya (42.1%) performed near or below chance. In paired bootstrap comparisons, Laya-Dengue exceeded MedGemma 27B by 4.1 percentage points in macro-AUC (95% CI 3.3–4.9) and by 19.2 points in severe-versus-death AUC (15.6–23.1), and it exceeded the TF-IDF baseline (macro-AUC, 94.0%; severe-versus-death AUC, 67.7%) by 1.3 (0.7–1.8) and 6.2 (3.8–8.6) points, respectively.
4.8.2. Error Analysis and Probabilistic Scores
The errors of MedGemma 27B concentrated in patients who survived severe dengue: even after recalibration, it classified 206 of the 276 non-fatal severe cases as deaths, compared with 85 for Laya-Dengue. Before recalibration, it assigned a mean probability of death of 0.78 to survivors with documented severity signs and of 0.96 to fatal cases with such signs. Recalibration lowered its prevalence-weighted log-loss from 0.414 to 0.166, and Laya-Dengue had lower prevalence-weighted probabilistic losses (log-loss, 0.100; Brier score, 0.044 versus 0.073 for MedGemma 27B). In source-specific analyses, Laya-Dengue’s severe-versus-death AUC was lower in the baseline corpus (60.5%) than in the 2023 (73.9%) and 2025 (76.7%) SINAN records. For deaths without any recorded warning or severity sign, all models assigned a low mean probability of death (Laya-Dengue, 4.7%; MedGemma 27B, 7.2%).
4.8.3. Latency and Input Guardrail
Under batched inference on the same GPU, scoring took 1.3 ms per record with Laya-Dengue and 56 ms with MedGemma 27B. In the local application, updates took approximately 50 ms per edit on a laptop GPU. The domain filter classified all 36 handcrafted development texts correctly, including rejection of a recipe containing the word “fever”; these were the same texts used to tune the thresholds. Among the 4,313 validation and test records, none was rejected as non-clinical, although three very short notes were held until more text was entered and 3.5% received an out-of-profile warning.
5. Discussion
5.1. Diagnostic Performance and Computational Efficiency
The main finding is that a compact encoder can achieve useful diagnostic performance at a small fraction of the inference time of larger generative models when task-specific training data are available. Laya-Med reached 81.1% macro-F1 on the four-diagnosis task, compared with 84.6% for Qwen3.8-27B, while reducing mean processing time from 0.60 s to 10.8 ms per case. The practical trade-off is therefore a modest loss in aggregate diagnostic performance for a substantial reduction in model size and inference time. TF-IDF occupied a different point on this trade-off: it was faster than either language model but had lower macro-F1. These results favour selecting a model according to the required performance and available computation rather than parameter count alone.
The class-specific results further qualify this comparison. Laya had the highest sensitivity for pancreatitis and cholecystitis, but detected only half of the diverticulitis cases. MedGemma 4B showed the opposite pattern, identifying more diverticulitis cases while missing more pancreatitis cases. Thus, a small difference in overall accuracy can conceal clinically relevant differences in error distribution. The limited number of diverticulitis cases makes the estimates imprecise, but the pattern identifies a priority for further model development and evaluation.
5.2. Comparison with Previous Work
The findings extend earlier evidence that supervised adaptation can make smaller models competitive on clinical language tasks [23,24]. They are also consistent with the use of health-system encoders for multiple prediction tasks [25] and with Taylor et al.’s analysis of model size and adaptation cost [26]. Our experiments add a comparison on shared diagnostic inputs and a second application using Portuguese surveillance-derived narratives. The comparison concerns task-adapted encoders and zero-shot generative models, so the results reflect both architecture and access to labelled training data. Work on reinforcement learning, multi-agent coordination and diagnostic trajectories addresses complementary settings in which models acquire information or make sequential decisions [16,17,18].
5.3. Implications for Interactive Decision Support
The single-pass output is particularly relevant to interactive applications. Updating a fixed vector of diagnostic probabilities after each edit avoids generating a new textual answer and supports response times of approximately 50 ms on a laptop GPU. It also allows the interface to display how the relative probabilities change as information is added. This design connects modern text representations with the probabilistic approach used in earlier computer-assisted diagnosis systems [31,34,35]. Its value will depend on whether those updates help clinicians make better decisions. Evaluation should therefore examine calibration during note entry, handling of diagnoses outside the candidate set and clinician responses, in addition to final-record discrimination [9,10,11,12,30].
5.4. Portuguese Narratives and Recorded Dengue Outcomes
The dengue study shows that the same model and training approach can be adapted to a different language and clinical target. Laya-Dengue improved discrimination over zero-shot MedGemma 27B, particularly for the distinction between non-fatal severe dengue and death. The smaller macro-AUC gain over TF-IDF indicates that much of the available signal was also accessible to a conventional text classifier. The 6.2-point difference in severe-versus-death AUC, however, suggests an advantage for the fine-tuned encoder on the more difficult distinction. Performance varied by source, with lower AUC in the baseline corpus than in the 2023 and 2025 records. This variation, together with low predicted death probabilities in fatal cases without recorded warning signs, points to the importance of documentation and cohort composition. Dengue outcomes depend on both disease progression and care [53,60], and evaluation in a target clinical population remains essential [40]. Adjustment to the baseline-corpus prior [54] provided a common reference distribution for comparing probabilistic outputs.
5.5. Limitations
Several limitations should guide interpretation. First, the comparison combines differences in architecture and supervision. Laya and TF-IDF used task-specific labels, whereas the LLMs were evaluated zero-shot; few-shot prompting or fine-tuning could change the ranking. Ablations are needed to distinguish the contributions of the decision head, scoring rule, target smoothing and auxiliary management task from those of the underlying mmBERT encoder. A comparator trained directly on the structured dengue variables would also clarify the value of their conversion into text. The accuracy comparison with Qwen3.8-27B was not designed as an equivalence test, and its non-significant result should not be interpreted as equivalence.
Second, the diagnostic evaluation included four conditions from one centre and 360 test admissions. Residual post-operative references and redaction-marker frequencies differed across diagnoses and may have favoured supervised models. Each of the 2,400 cases corresponds to a unique patient [4,5], so the admission-level split also separates patients. The small continuation probe assessed verbatim recall but cannot exclude other forms of pretraining contamination. Batched inference measurements describe processing cost under the tested configuration and do not capture memory use, training cost or single-request latency.
Third, the dengue input consisted of generated narratives. All mild cases came from the baseline corpus, whose years were not recorded, whereas most severe and fatal cases came from the 2023 and 2025 records. Source, period and outcome were therefore partly confounded. Random splitting within this assembled cohort does not assess temporal or external generalization. Cases with warning signs that resolved were excluded. Moreover, predictor availability before deterioration or death was not established, and some clinical signs contribute to the severity definitions. These results should consequently be interpreted as retrospective outcome classification rather than validated early prognosis.
Finally, training with proper scores and applying probability adjustments do not by themselves establish calibration in a new population. Held-out reliability analyses and a separate evaluation of the prefix adjustment are needed. The input filter was tuned on 36 handcrafted texts without an independent non-clinical test set, and the interface’s completeness scale is heuristic. Neither study measured effects on clinical decisions or patient outcomes. Prospective evaluation should address these issues before clinical deployment [61].
6. Conclusion
Laya-Med demonstrates the feasibility of using a compact bidirectional language model for clinical decision support with rapid local inference. Supervised adaptation produced diagnostic performance close to that of a substantially larger zero-shot model, with lower processing time, and supported classification of dengue outcomes from Portuguese narratives. The results identify compact encoders as a practical option for defined clinical tasks with labelled data. Future work should establish how this performance carries over to native clinical notes, external populations and prospective decision support.
Author Contributions
Luís Guilherme Espinosa e Souza: Conceptualization, Methodology, Software, Formal analysis, Investigation, Data curation, Visualization, Writing – original draft, Writing – review & editing. Sarah Peixoto Rodrigues Freire: Investigation, Writing – review & editing. Helder I. Nakaya: Conceptualization, Methodology, Supervision, Funding acquisition, Writing – review & editing.
Funding
This work was supported by the São Paulo Research Foundation (FAPESP) [grant number 2023/18337-3 to H.I.N.] and by the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior – Brasil (CAPES) [Finance Code 001; doctoral fellowship to L.G.E.S.; master’s fellowship to S.P.R.F.]. The funders had no role in study design; in the collection, analysis and interpretation of data; in the writing of the report; or in the decision to submit the article for publication.
Data Availability Statement
MIMIC-IV-Ext-CDM (v1.1) is available to credentialed researchers through PhysioNet under its data use agreement [5]. No MIMIC patient text is reproduced here; the clinical example in Figure 1 is illustrative. SINAN dengue notification microdata for 2023 and 2025 are public through the Ministry of Health’s open-data portal [41]. The rewritten dengue narratives derived from public microdata are available from the corresponding author on reasonable request.
Acknowledgments
We thank the creators of MIMIC-IV-Ext-CDM and the PhysioNet team for making the dataset available, and Convai Innovations for releasing the Laya model and library.
Conflicts of Interest
H.I.N. is the co-founder of Hylix, HumanIA, D2DNA and ETERN, which are startup companies that utilize artificial intelligence and bioinformatics for health-related problems. The other authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Code availability: The Laya library (v0.4.7) and base checkpoints are publicly available [27]. We used the 322-million-parameter checkpoint in the multilingual subdirectory of https://huggingface.co/convaiinnovations/laya. Code for local inference, preprocessing, scope detection and prefix recalibration is available at https://github.com/csbl-usp/LayaMed, together with documentation and a training example using synthetic cases. The public release does not include MIMIC-derived records, trained study weights or a complete archived reproduction package for both analyses.
Declaration of generative AI and AI-assisted technologies in the manuscript preparation process: During manuscript preparation, the authors used Claude (Anthropic) to assist with restructuring, language editing and bibliographic metadata checks. Codex (OpenAI) was subsequently used to assist with critical review, language revision, reference checking and preparation of revised figures from the reported aggregate results. The authors reviewed the resulting material and take responsibility for the final content. These tools were not used to generate new patient-level results for this revision.
Appendix A. Supplementary Tables and Prompt
Table A1.
Leakage indicators in the model input before and after cleaning (MIMIC-IV-Ext-CDM, n = 2,400).
Table A1.
Leakage indicators in the model input before and after cleaning (MIMIC-IV-Ext-CDM, n = 2,400).
| Section | Indicator | Overall | App | Chol | Panc | Div |
|---|---|---|---|---|---|---|
| History of present illness | Disease or surgery name | 18.2 → 0.0 | 5.5 → 0.0 | 23.8 → 0.0 | 32.2 → 0.0 | 22.2 → 0.0 |
| History of present illness | Imaging study | 32.3 → 0.0 | 9.6 → 0.0 | 40.0 → 0.0 | 62.1 → 0.0 | 35.0 → 0.0 |
| History of present illness | Post-operative finding | 28.8 → 27.5 | 11.8 → 11.6 | 34.7 → 32.4 | 46.3 → 44.2 | 40.5 → 39.3 |
| History of present illness | “[redacted]” marker | – → 15.6 | – → 5.1 | – → 21.6 | – → 25.7 | – → 18.7 |
| Physical examination | Disease or surgery name | 0.8 → 0.0 | 0.3 → 0.0 | 1.4 → 0.0 | 0.9 → 0.0 | 1.2 → 0.0 |
| Physical examination | Imaging study | 15.1 → 0.0 | 10.6 → 0.0 | 13.3 → 0.0 | 21.4 → 0.0 | 23.3 → 0.0 |
| Physical examination | Post-operative finding | 15.5 → 0.0 | 15.7 → 0.0 | 19.0 → 0.0 | 11.5 → 0.0 | 14.0 → 0.0 |
| Physical examination | “[redacted]” marker | – → 0.5 | – → 0.1 | – → 1.1 | – → 0.6 | – → 0.0 |
| Radiology impression | Disease or surgery name | 18.8 → – | 10.0 → – | 19.0 → – | 32.7 → – | 22.2 → – |
Values are percentages of admissions with the indicator, shown as before → after cleaning. The “–“ symbol denotes an unreported or inapplicable value. The “[redacted]” marker exists only after cleaning. The radiology impression is not part of the primary input; it was added, with disease names masked, only in the sensitivity analysis. App, appendicitis; Chol, cholecystitis; Panc, pancreatitis; Div, diverticulitis.
Table A2.
Sensitivity by pathology on the blind test set, with Wilson 95% confidence intervals.
| System | Appendicitis (n = 144) | Cholecystitis (n = 97) | Pancreatitis (n = 81) | Diverticulitis (n = 38) |
|---|---|---|---|---|
| Qwen3.8-27B (zero-shot) | 93.8 (88.5–96.7); 135/144 | 83.5 (74.9–89.6); 81/97 | 87.7 (78.7–93.2); 71/81 | 68.4 (52.5–80.9); 26/38 |
| Laya (fine-tuned) | 90.3 (84.3–94.1); 130/144 | 87.6 (79.6–92.8); 85/97 | 91.4 (83.2–95.8); 74/81 | 50.0 (34.8–65.2); 19/38 |
| MedGemma 27B (zero-shot) | 86.8 (80.3–91.4); 125/144 | 77.3 (68.0–84.5); 75/97 | 85.2 (75.9–91.3); 69/81 | 78.9 (63.7–88.9); 30/38 |
| TF-IDF + logistic regression | 85.4 (78.7–90.3); 123/144 | 75.3 (65.8–82.8); 73/97 | 79.0 (68.9–86.5); 64/81 | 47.4 (32.5–62.7); 18/38 |
| MedGemma 4B (zero-shot) | 83.3 (76.4–88.5); 120/144 | 73.2 (63.6–81.0); 71/97 | 53.1 (42.3–63.6); 43/81 | 86.8 (72.7–94.2); 33/38 |
| Laya (zero-shot) | 45.1 (37.2–53.3); 65/144 | 8.2 (4.2–15.4); 8/97 | 81.5 (71.7–88.4); 66/81 | 7.9 (2.7–20.8); 3/38 |
Values are sensitivity, % (Wilson 95% CI), with the number of correctly classified admissions over the number with each diagnosis.
Table A3.
Contamination probe.
| Model | Mean ROUGE-L, true continuation | Mean ROUGE-L, other record | Cases with ROUGE-L > 0.50, % | Mean identical leading words | Cases with ≥ 10 identical leading words, % |
|---|---|---|---|---|---|
| Qwen3.8-27B | 0.183 | 0.103 | 0 | 0.92 | 0 |
| MedGemma 27B-text-it | 0.149 | 0.087 | 0 | 0.62 | 0 |
| MedGemma 1.5 4B-it | 0.137 | 0.098 | 0 | 0.60 | 0 |
Each model received the first 60 words of the history of present illness of 50 test admissions and generated a continuation, compared with the true next 30 words (word-level ROUGE-L) and, as a control, with the continuation of a different admission.
Note A.1. Prompt and parsing rules for the generative LLMs.
System prompt (identical for all LLMs, primary analysis without radiology):
You are an expert emergency physician and general surgeon.Analyze the clinical summary (history of present illness, physical examination and laboratory tests) to determine:1. The primary acute abdominal diagnosis among strictly four conditions: “appendicitis”, “cholecystitis”, “pancreatitis”, “diverticulitis”.2. Management: “surgical” or “conservative”.Imaging results are NOT available: reason from the clinical history, physical examination and laboratory tests only.Some words in the summary were replaced by “[redacted]”.FORMAT:DIAGNOSIS: [appendicitis | cholecystitis | pancreatitis | diverticulitis]MANAGEMENT: [surgical | conservative]
User message: “PATIENT SUMMARY:“ followed by the record and “Provide DIAGNOSIS and MANAGEMENT:“. The prompt was built with each model’s chat template; for Qwen3.8-27B, reasoning was disabled in the template (enable_thinking = False). Decoding was greedy, with up to 60 new tokens. Any reasoning blocks (<think>...</think> or <unused94>...<unused95>) were removed before parsing. The diagnosis was read from the DIAGNOSIS field with the regular expression DIAGNOSIS\**\s*[:\-]\s*[\*\[\s]*([A-Za-z]+) (case-insensitive); if the field was absent or invalid, the first of the four pathologies named in the answer was used, and an answer naming none of them was scored as an error. The MANAGEMENT field was requested for consistency with the original benchmark but is not analysed here.
References
- Kung, T.H.; Cheatham, M.; Medenilla, A.; Sillos, C.; De Leon, L.; Elepaño, C.; et al. Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models. PLoS Digit Health 2023, 2, e0000198. [Google Scholar] [CrossRef] [PubMed]
- Singhal, K.; Azizi, S.; Tu, T.; Mahdavi, S.S.; Wei, J.; Chung, H.W.; et al. Large language models encode clinical knowledge. Nature 2023, 620, 172–80. [Google Scholar] [CrossRef] [PubMed]
- Thirunavukarasu, A.J.; Ting, D.S.J.; Elangovan, K.; Gutierrez, L.; Tan, T.F.; Ting, D.S.W. Large language models in medicine. Nat. Med. 2023, 29, 1930–40. [Google Scholar] [CrossRef] [PubMed]
- Hager, P.; Jungmann, F.; Holland, R.; Bhagat, K.; Hubrecht, I.; Knauer, M.; et al. Evaluation and mitigation of the limitations of large language models in clinical decision-making. Nat. Med. 2024, 30, 2613–22. [Google Scholar] [CrossRef] [PubMed]
- Hager, P.; Jungmann, F.; Rueckert, D. MIMIC-IV-Ext Clinical Decision Making: a MIMIC-IV derived dataset for evaluation of large language models on the task of clinical decision making for abdominal pathologies. Version 1.1. PhysioNet 2024. [Google Scholar] [CrossRef]
- Johnson, A.E.W.; Bulgarelli, L.; Shen, L.; Gayles, A.; Shammout, A.; Horng, S.; et al. MIMIC-IV, a freely accessible electronic health record dataset. Sci. Data 2023, 10, 1. [Google Scholar] [CrossRef] [PubMed]
- Wornow, M.; Xu, Y.; Thapa, R.; Patel, B.; Steinberg, E.; Fleming, S.; et al. The shaky foundations of large language models and foundation models for electronic health records. npj Digit Med. 2023, 6, 135. [Google Scholar] [CrossRef] [PubMed]
- PhysioNet. Responsible use of MIMIC data with online services like GPT. 2023. Available online: https://physionet.org/news/post/gpt-responsible-use (accessed on 20 October 2026).
- Morgan, D.J.; Pineles, L.; Owczarzak, J.; Magder, L.; Scherer, L.; Brown, J.P.; et al. Accuracy of practitioner estimates of probability of diagnosis before and after testing. JAMA Intern Med. 2021, 181, 747–55. [Google Scholar] [CrossRef] [PubMed]
- Rodman, A.; Buckley, T.A.; Manrai, A.K.; Morgan, D.J. Artificial intelligence vs clinician performance in estimating probabilities of diagnoses before and after testing. JAMA Netw. Open 2023, 6, e2347075. [Google Scholar] [CrossRef] [PubMed]
- Griot, M.; Hemptinne, C.; Vanderdonckt, J.; Yuksel, D. Large language models lack essential metacognition for reliable medical reasoning. Nat. Commun. 2025, 16, 642. [Google Scholar] [CrossRef] [PubMed]
- Van Calster, B.; McLernon, D.J.; van Smeden, M.; Wynants, L.; Steyerberg, E.W. Calibration: the Achilles heel of predictive analytics. BMC Med. 2019, 17, 230. [Google Scholar] [CrossRef] [PubMed]
- Devlin, J.; Chang, M.W.; Lee, K.; Toutanova, K. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies; Association for Computational Linguistics: Minneapolis, 2019; Volume 1, pp. 4171–86. [Google Scholar] [CrossRef]
- Lee, J.; Yoon, W.; Kim, S.; Kim, D.; Kim, S.; So, C.H.; et al. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics 2020, 36, 1234–40. [Google Scholar] [CrossRef] [PubMed]
- Jin, Y.; Chandra, M.; Verma, G.; Hu, Y.; De Choudhury, M.; Kumar, S. Better to ask in English: cross-lingual evaluation of large language models for healthcare queries. In Proceedings of the ACM Web Conference 2024; Association for Computing Machinery: New York, 2024; pp. 2627–38. [Google Scholar] [CrossRef]
- Bani-Harouni, D.; Pellegrini, C.; Özsoy, E.; Navab, N.; Keicher, M. Language agents for hypothesis-driven clinical decision making with reinforcement learning. International Conference on Learning Representations (ICLR 2026), 2026; Available online: https://openreview.net/forum?id=7vHUQCMAzG.
- Bedi, S.; Mlauzi, I.; Shin, D.; Koyejo, S.; Shah, N.H. The optimization paradox in clinical AI multi-agent systems [preprint]. arXiv 2025. [Google Scholar] [CrossRef]
- Shen, X.; Liu, H.; Song, D.; Min, M.R. Uncertainty-guided latent diagnostic trajectory learning for sequential clinical diagnosis [preprint]. arXiv 2026. [Google Scholar] [CrossRef]
- Schmidgall, S.; Ziaei, R.; Harris, C.; Reis, E.; Jopling, J.; Moor, M. AgentClinic: a multimodal agent benchmark to evaluate AI in simulated clinical environments [preprint]. arXiv 2024. [Google Scholar] [CrossRef]
- Johri, S.; Jeong, J.; Tran, B.A.; Schlessinger, D.I.; Wongvibulsin, S.; Barnes, L.A.; et al. An evaluation framework for clinical use of large language models in patient interaction tasks. Nat. Med. 2025, 31, 77–86. [Google Scholar] [CrossRef] [PubMed]
- Nori, H.; Daswani, M.; Kelly, C.; Lundberg, S.; Ribeiro, M.T.; Wilson, M.; et al. Sequential diagnosis with language models [preprint]. arXiv 2025. [Google Scholar] [CrossRef]
- Tu, T.; Schaekermann, M.; Palepu, A.; Saab, K.; Freyberg, J.; Tanno, R.; et al. Towards conversational diagnostic artificial intelligence. Nature 2025, 642, 442–50. [Google Scholar] [CrossRef] [PubMed]
- Lehman, E.; Hernandez, E.; Mahajan, D.; Wulff, J.; Smith, M.J.; Ziegler, Z.; et al. Do we still need clinical language models? Proc. Conf. Health Inference Learn. PMLR 2023, 209, 578–97. Available online: https://proceedings.mlr.press/v209/eric23a.html.
- Chen, Q.; Hu, Y.; Peng, X.; Xie, Q.; Jin, Q.; Gilson, A.; et al. Benchmarking large language models for biomedical natural language processing applications and recommendations. Nat. Commun. 2025, 16, 3280. [Google Scholar] [CrossRef] [PubMed]
- Jiang, L.Y.; Liu, X.C.; Nejatian, N.P.; Nasir-Moin, M.; Wang, D.; Abidin, A.; et al. Health system-scale language models are all-purpose prediction engines. Nature 2023, 619, 357–62. [Google Scholar] [CrossRef] [PubMed]
- Taylor, N.; Ghose, U.; Rohanian, O.; Nouriborji, M.; Kormilitzin, A.; Clifton, D.A.; et al. Efficiency at scale: investigating the performance of diminutive language models in clinical tasks. Artif. Intell. Med. 2024, 157, 103002. [Google Scholar] [CrossRef] [PubMed]
- Convai Innovations. Laya. Version 0.4.7. Python Package Index. 2026. Available online: https://pypi.org/project/laya/0.4.7/ (accessed on 21 October 2026).
- Marone, M.; Weller, O.; Fleshman, W.; Yang, E.; Lawrie, D.; Van Durme, B. mmBERT: a modern multilingual encoder with annealed language learning [preprint]. arXiv 2025. [Google Scholar] [CrossRef]
- Warner, B.; Chaffin, A.; Clavié, B.; Weller, O.; Hallström, O.; Taghadouini, S.; et al. Smarter, better, faster, longer: a modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics; Association for Computational Linguistics: Vienna, 2025; Volume 1, pp. 2526–47. [Google Scholar] [CrossRef]
- Gneiting, T.; Raftery, A.E. Strictly proper scoring rules, prediction, and estimation. J. Am. Stat. Assoc. 2007, 102, 359–78. [Google Scholar] [CrossRef]
- Gorry, G.A.; Barnett, G.O. Sequential diagnosis by computer. JAMA 1968, 205, 849–54. [Google Scholar] [CrossRef]
- Shwe, M.A.; Middleton, B.; Heckerman, D.E.; Henrion, M.; Horvitz, E.J.; Lehmann, H.P.; et al. Probabilistic diagnosis using a reformulation of the INTERNIST-1/QMR knowledge base. I. The probabilistic model and inference algorithms. Methods Inf. Med. 1991, 30, 241–55. [Google Scholar] [CrossRef]
- Heckerman, D.E.; Nathwani, B.N. Toward normative expert systems: Part II. Probability-based representations for efficient knowledge acquisition and inference. Methods Inf. Med. 1992, 31, 106–16. [Google Scholar] [CrossRef]
- de Dombal, F.T.; Leaper, D.J.; Staniland, J.R.; McCann, A.P.; Horrocks, J.C. Computer-aided diagnosis of acute abdominal pain. Br. Med. J. 1972, 2, 9–13. [Google Scholar] [CrossRef] [PubMed]
- Leaper, D.J.; Horrocks, J.C.; Staniland, J.R.; de Dombal, F.T. Computer-assisted diagnosis of abdominal pain using “estimates” provided by clinicians. Br. Med. J. 1972, 4, 350–4. [Google Scholar] [CrossRef] [PubMed]
- de Dombal, F.T.; Dallos, V.; McAdam, W.A. Can computer aided teaching packages improve clinical care in patients with acute abdominal pain? BMJ 1991, 302, 1495–7. [Google Scholar] [CrossRef] [PubMed]
- Bhatt, S.; Gething, P.W.; Brady, O.J.; Messina, J.P.; Farlow, A.W.; Moyes, C.L.; et al. The global distribution and burden of dengue. Nature 2013, 496, 504–7. [Google Scholar] [CrossRef] [PubMed]
- Guzman, M.G.; Harris, E. Dengue. Lancet 2015, 385, 453–65. [Google Scholar] [CrossRef] [PubMed]
- Brazil; Ministry of Health. Painel de monitoramento das arboviroses. 2026. Available online: https://www.gov.br/saude/pt-br/assuntos/saude-de-a-a-z/a/aedes-aegypti/monitoramento-das-arboviroses (accessed on 19 October 2026).
- Delpino, F.M.; Peres, I.T.; Gusberti, T.; de Lima, C.J.; Duque, S.; Merson, L.; et al. Global performance of predictive models for dengue severity, hospitalization and mortality: a systematic review and meta-analysis of 146 studies. Int. J. Infect. Dis. 2026, 170, 108879. [Google Scholar] [CrossRef] [PubMed]
- Brazil. Ministry of Health. Sistema de Informação de Agravos de Notificação (SINAN): dengue notification microdata, 2023 and 2025 (DENGBR23, DENGBR25). OpenDataSUS. 2026. Available online: https://dadosabertos.saude.gov.br/ (accessed on 18 October 2026).
- Collins, G.S.; Moons, K.G.M.; Dhiman, P.; Riley, R.D.; Beam, A.L.; Van Calster, B.; et al. TRIPOD+AI statement: updated guidance for reporting clinical prediction models that use regression or machine learning methods. BMJ 2024, 385, e078378. [Google Scholar] [CrossRef] [PubMed]
- Gallifant, J.; Afshar, M.; Ameen, S.; Aphinyanaphongs, Y.; Chen, S.; Cacciamani, G.; et al. The TRIPOD-LLM reporting guideline for studies using large language models. Nat. Med. 2025, 31, 60–9. [Google Scholar] [CrossRef] [PubMed]
- Brazil; National Health Council. Resolution No. 510, of 7 April 2016 [Resolução nº 510, de 7 de abril de 2016]. Diário Of. Da União 2016, (98) Section 1, 44–6. [Google Scholar]
- Brazil. Law No. 13,709, of 14 August 2018: General Personal Data Protection Law (LGPD). Diário Oficial da União. 15 Aug 2018. Available online: https://www.planalto.gov.br/ccivil_03/_ato2015-2018/2018/lei/l13709.htm.
- Goldberger, A.L.; Amaral, L.A.; Glass, L.; Hausdorff, J.M.; Ivanov, P.C.; Mark, R.G.; et al. PhysioBank, PhysioToolkit, and PhysioNet: components of a new research resource for complex physiologic signals. Circulation 2000, 101, e215–20. [Google Scholar] [CrossRef] [PubMed]
- Loshchilov, I.; Hutter, F. Decoupled weight decay regularization. International Conference on Learning Representations (ICLR 2019), 2019; Available online: https://openreview.net/forum?id=Bkg6RiCqY7.
- Guo, C.; Pleiss, G.; Sun, Y.; Weinberger, K.Q. On calibration of modern neural networks. Proc. 34th Int. Conf. Mach. Learn. PMLR 2017, 70, 1321–30. Available online: https://proceedings.mlr.press/v70/guo17a.html.
- Sellergren, A.; Kazemzadeh, S.; Jaroensri, T.; Kiraly, A.; Traverse, M.; Kohlberger, T.; et al. MedGemma technical report [preprint]. arXiv 2025. [Google Scholar] [CrossRef]
- Google. MedGemma 1.5 4B instruction-tuned model. Hugging Face. 2026. Available online: https://huggingface.co/google/medgemma-1.5-4b-it (accessed on 21 October 2026).
- Qwen Team. Qwen3.8-27B. Hugging Face. 2026. Available online: https://huggingface.co/Qwen/Qwen3.8-27B (accessed on 21 October 2026).
- Lin, C.Y. ROUGE: a package for automatic evaluation of summaries. In Text Summarization Branches Out; Association for Computational Linguistics: Barcelona, 2004; pp. 74–81. Available online: https://aclanthology.org/W04-1013/.
- World Health Organization. Dengue: guidelines for diagnosis, treatment, prevention and control, New ed.; World Health Organization: Geneva, 2009; Available online: https://www.who.int/publications/i/item/9789241547871.
- Saerens, M.; Latinne, P.; Decaestecker, C. Adjusting the outputs of a classifier to new a priori probabilities: a simple procedure. Neural Comput 2002, 14, 21–41. [Google Scholar] [CrossRef] [PubMed]
- Lee, K.; Lee, K.; Lee, H.; Shin, J. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. Advances in Neural Information Processing Systems 31 (NeurIPS 2018), 2018; Available online: https://proceedings.neurips.cc/paper/2018/hash/abdeb6f575ac5c6676b747bca8d09cc2-Abstract.html.
- McNemar, Q. Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika 1947, 12, 153–7. [Google Scholar] [CrossRef] [PubMed]
- Holm, S. A simple sequentially rejective multiple test procedure. Scand. J. Stat. 1979, 6, 65–70. Available online: https://www.jstor.org/stable/4615733.
- Wilson, E.B. Probable inference, the law of succession, and statistical inference. J. Am. Stat. Assoc. 1927, 22, 209–12. [Google Scholar] [CrossRef]
- Brier, G.W. Verification of forecasts expressed in terms of probability. Mon. Weather Rev. 1950, 78, 1–3. [Google Scholar] [CrossRef]
- Simmons, C.P.; Farrar, J.J.; Nguyen, vV; Wills, B. Dengue. N Engl. J. Med. 2012, 366, 1423–32. [Google Scholar] [CrossRef] [PubMed]
- Vasey, B.; Nagendran, M.; Campbell, B.; Clifton, D.A.; Collins, G.S.; Denaxas, S.; et al. Reporting guideline for the early-stage clinical evaluation of decision support systems driven by artificial intelligence: DECIDE-AI. Nat. Med. 2022, 28, 924–33. [Google Scholar] [CrossRef] [PubMed]
Figure 1.
Study 1 design and evaluation framework. (a) MIMIC-IV-Ext-CDM v1.1 comprises 2,400 emergency admissions: appendicitis (n = 957), cholecystitis (n = 648), pancreatitis (n = 538) and diverticulitis (n = 257). Stratified sampling produced training (n = 1,680), validation (n = 360) and held-out test (n = 360) sets. Disease and procedure names were masked; imaging mentions in the history and examination and post-operative findings in the examination were removed. Residual indicators are reported in Table A1. (b) The fine-tuned encoder and zero-shot LLMs received the same clinical text and selected among the same four diagnoses. An auxiliary management question was included but its performance is not evaluated. The illustrated text and probabilities do not correspond to a MIMIC patient. Times are mean processing times per case under batched inference on one NVIDIA A100 80 GB GPU.
Figure 1.
Study 1 design and evaluation framework. (a) MIMIC-IV-Ext-CDM v1.1 comprises 2,400 emergency admissions: appendicitis (n = 957), cholecystitis (n = 648), pancreatitis (n = 538) and diverticulitis (n = 257). Stratified sampling produced training (n = 1,680), validation (n = 360) and held-out test (n = 360) sets. Disease and procedure names were masked; imaging mentions in the history and examination and post-operative findings in the examination were removed. Residual indicators are reported in Table A1. (b) The fine-tuned encoder and zero-shot LLMs received the same clinical text and selected among the same four diagnoses. An auxiliary management question was included but its performance is not evaluated. The illustrated text and probabilities do not correspond to a MIMIC patient. Times are mean processing times per case under batched inference on one NVIDIA A100 80 GB GPU.

Figure 2.
Diagnostic sensitivity by pathology on the blind test set. Percentage of admissions classified correctly (sensitivity) for each pathology, and overall accuracy (n = 360), for the fine-tuned Laya encoder and the three zero-shot LLMs. Sample sizes are shown in parentheses; values above the bars are rounded to the nearest integer.
Figure 2.
Diagnostic sensitivity by pathology on the blind test set. Percentage of admissions classified correctly (sensitivity) for each pathology, and overall accuracy (n = 360), for the fine-tuned Laya encoder and the three zero-shot LLMs. Sample sizes are shown in parentheses; values above the bars are rounded to the nearest integer.

Figure 3.
Diagnostic performance and inference time. Macro-F1 (a) and accuracy (b) on the 360-admission test set are plotted against mean processing time per case under batched inference on one NVIDIA A100 80 GB GPU. The time axis is logarithmic. Points reproduce Table 2, with labels rounded to one decimal place. The dashed line connects the non-dominated systems: TF-IDF with logistic regression, fine-tuned Laya and Qwen3.8-27B. Times represent batch throughput per case rather than single-request latency.
Figure 3.
Diagnostic performance and inference time. Macro-F1 (a) and accuracy (b) on the 360-admission test set are plotted against mean processing time per case under batched inference on one NVIDIA A100 80 GB GPU. The time axis is logarithmic. Points reproduce Table 2, with labels rounded to one decimal place. The dashed line connects the non-dominated systems: TF-IDF with logistic regression, fine-tuned Laya and Qwen3.8-27B. Times represent batch throughput per case rather than single-request latency.

Figure 4.
Dengue outcome classification on Portuguese narratives generated from surveillance records. (a) Class-specific sensitivity and balanced accuracy for zero-shot Laya, fine-tuned Laya-Dengue and MedGemma 27B. The test set contained 1,370 mild cases, 276 non-fatal severe cases and 511 deaths; labels are rounded to integer percentages. (b) Macro-AUC and (c) AUC for non-fatal severe dengue versus death (n = 787), including the supervised TF-IDF comparator. Panels b and c show point estimates; paired bootstrap confidence intervals for between-model differences are reported in the text.
Figure 4.
Dengue outcome classification on Portuguese narratives generated from surveillance records. (a) Class-specific sensitivity and balanced accuracy for zero-shot Laya, fine-tuned Laya-Dengue and MedGemma 27B. The test set contained 1,370 mild cases, 276 non-fatal severe cases and 511 deaths; labels are rounded to integer percentages. (b) Macro-AUC and (c) AUC for non-fatal severe dengue versus death (n = 787), including the supervised TF-IDF comparator. Panels b and c show point estimates; paired bootstrap confidence intervals for between-model differences are reported in the text.

Table 1.
Inference settings of the generative LLMs.
| Parameter | Study 1 (MIMIC-IV-Ext-CDM) | Study 2 (dengue) |
|---|---|---|
| Models | MedGemma 1.5 4B-it; MedGemma 27B-text-it; Qwen3.8-27B | MedGemma 27B-text-it |
| Model update | Frozen; zero-shot | Frozen; zero-shot |
| Numerical precision | bfloat16 | bfloat16 |
| Decoding | Greedy | No generation; first-token logits of options A–C |
| Maximum new tokens | 60 | 1 (scoring only) |
| Reasoning (“thinking”) tokens | Disabled | Not applicable |
| Prompt | Identical system prompt for all models (Note A.1) | Multiple-choice question with the three outcome definitions |
| Output format | “DIAGNOSIS: [pathology]” | Probability over options A–C |
| Hardware | One NVIDIA A100 80 GB GPU, batched inference | One NVIDIA A100 80 GB GPU, batched inference |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2026 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
Copyright: This open access article is published under a Creative Commons CC BY 4.0 license, which permit the free download, distribution, and reuse, provided that the author and preprint are cited in any reuse.