Clinician Evaluation of Artificial Intelligence Summaries of Pediatric CVICU Progress Notes

1. Introduction

Clinical note summarization, such as standardization of patient handoffs, can improve the continuity of care by reducing communication errors and supporting clinical workflow [1]. Large language models (LLM) offer a promising opportunity to automate the summarization of clinical documentation [2,3], potentially alleviating these challenges. This research brief addresses concerns regarding the accuracy, completeness, and reliability of LLM-generated summaries in clinical practice.

2. Methods

Progress notes from 385 pediatric Cardiovascular Intensive Care Unit (CVICU) patients treated at a tertiary pediatric health system in Southern California were retrieved and deidentified for the study. The use of these notes was approved under IRB #230571. For each patient, all original progress notes were combined into a single file to keep the full clinical context available to the LLM.

The summaries of the patient notes were generated using the Mistral 7B LLM via AWS Bedrock. Prompts were formatted according to the iPASS handoff framework [4]. Readability was assessed using Flesch Reading Ease [5], Flesch-Kincaid Grade Level [6], Gunning Fog Index [7], SMOG Index [8], Automated Readability Index [9], and Dale-Chall Score [10]. In parallel, cosine similarity was computed between each original note and its corresponding summary using vector embeddings to evaluate how well the model preserved core semantic content.

Clinician evaluation was conducted on a stratified random sample of 10 patient summaries by three pediatric resident physicians (post-graduate year (PGY) 1-3). Each resident rated the summaries on a 5-point Likert scale across three domains: completeness (inclusion of key clinical information), correctness (factual accuracy and absence of hallucinations), and conciseness (brevity without omitting critical details). Rating rubrics were standardized and distributed prior to evaluation to ensure consistency.

3. Results

Across all six-readability metrics (Figure 1), the original notes were easier to read and understand. The summaries required higher reading levels, scoring 15 on the Flesch-Kincaid (college-level) and 16 on the Gunning Fog (college senior), while the original notes were easier to read, with a much higher Flesch Reading Ease score of 57 (indicating easier reading) compared to 29 for the summaries (indicating harder reading) (Figure 1).

Pediatric resident evaluation scores are summarized in Figure 2. A PGY1 resident provided consistently higher ratings across all domains, especially for correctness and conciseness, suggesting a greater tolerance for minor omissions or abstraction. In contrast, both PGY2 and PGY3 residents demonstrated greater variability in their ratings, particularly for completeness, which had higher variance (indicating more sensitive to missing or ambiguous clinical details). While conciseness received the highest average ratings overall, both PGY2 and PGY3 evaluators rated some summaries lower due to omitted context that affected interpretability

4. Discussion

The use of artificial intelligence to generate clinical summaries may result in a less readable summary of the patient. The experience may differ by clinician experience, as a junior resident often perceived the summaries generated as more useful, while senior residents were more likely to notice missing details or limitations. Summaries were notably shorter and easier to scan, appealing especially to a junior resident who favored their brevity and clarity. Despite their shorter length, the summaries were harder to read according to standard readability metrics (Figure 1), the usability of which may be compounded by abstraction and omitted context. In addition, conciseness came at the cost of completeness, with concerns about missing clinical details. This highlights a higher expectation for contextual depth with increased clinical experience. These trends suggest that perceived summary quality is partially influenced by clinical experience and expectations of detail. LLM-generated summaries may miss important details needed by experienced clinicians or in complex cases, leading to increased cognitive burden by requiring verification against the full note set. Future research should focus on improving abstraction which may require improvements in the organization of the source documentation.