Submitted:
19 November 2026
Posted:
20 November 2026
You are already at the latest version
Abstract
Objective: To characterize how generative large language models (LLMs) extracting clinical information from electronic health records have been evaluated, what evidence supports their implementation, and which gaps separate reported performance from clinical use. Materials and Methods: We conducted a systematic review registered in PROSPERO (CRD420261418828). Two reviewers verified artificial intelligence-assisted screening in a 20% random sample, with clarified re-screening of remaining records. Published studies and preprints were synthesized narratively, without meta-analysis, using each study's best result and, in sensitivity analysis, its lowest. Risk of bias was appraised with an adapted QUADAS-2. Results: Of 1,282 records, 179 unique studies were included (140 published, 39 preprints). In published studies, median F1 of LLM extraction against the human reference standard was 0.887 (interquartile range 0.800–0.948; n=103). In 67 studies testing several configurations, the median fell from 0.890 (best) to 0.630 (lowest). Preprint median F1 was 0.855 (n=30). Risk of bias was high in 107, unclear in 71, and low in one. External validation was reported in 25/153 assessable studies, prospective design in 2/142, and production deployment in 0/124. Discussion: Performance depended on task definition and on which configuration authors reported, and most studies had high or unclear risk of bias, precluding cross-study model rankings. Clinician participation often reflected annotation or research assessment, not clinical use. Conclusion: Evidence on generative LLMs for clinical extraction remains largely retrospective and technical. Task-specific external validation and prospective evaluation within clinical workflows are needed before implementation.
Keywords:
electronic health records
; large language models
; natural language processing
; generative artificial intelligence
; data mining
Copyright: This open access article is published under a Creative Commons CC BY 4.0 license, which permit the free download, distribution, and reuse, provided that the author and preprint are cited in any reuse.