Comparison of Large Language Models Versus Traditional Information Extraction Methods for Real World Evidence of Patient Symptomatology in Acute and Post-Acute Sequelae of SARS-CoV-2

1. Introduction

Patient symptoms are critical to understanding disease progression and diagnosis [1,2]. Indeed, symptoms offer both diagnostic and prognostic capacity [2]. Structured electronic health record (EHR) data rarely capture patient symptoms, leaving them mostly within unstructured sections of clinical notes such as the history of present illness (HPI) [2]. Thus, there is a need to develop natural language processing (NLP) solutions that can reliably extract this information from clinical notes to facilitate their integration into downstream applications. This study provides a comprehensive exploration of using large language models (LLMs) for generative question-answering (QA) tasks in NLP for symptom extraction from unstructured clinical notes. It evaluates their performance against rule-based information extraction (IE) methods, particularly those reliant on the unified medical language system (UMLS). By demonstrating the comparable effectiveness of LLMs to rule-based IE methods and emphasizing their inherent advantages, such as handling concept synonymy without extensive manual intervention, this study envisions a future where labor-intensive lexica creation by subject matter experts (SMEs) becomes obsolete.

1.1. Information Extraction

Rule-based Concept Extraction. The UMLS metathesaurus integrates biomedical concepts from diverse vocabularies into concept unique identifiers (CUIs) [3]. With over 4 million concepts grouped into 15 semantic groups, 133 semantic types, and 54 semantic relationships, the UMLS facilitates various applications, including IE [3,4,5]. PASCLex is a comprehensive lexicon of UMLS concepts and their synonyms related to Post-Acute Sequelae of SARS-CoV-2 (PASC, also known as “Long COVID”) symptoms that was created by analyzing a large corpus of clinical notes [6]. PASCLex aims to improve the identification and analysis of symptoms beyond those specific to PASC. Clinical NLP systems like The BioMedical Information Collection and Understanding System (BioMedICUS), developed by the University of Minnesota’s Natural Language Processing/Information Extraction Program, leverages the UMLS for large-scale extensive text analysis, enabling annotation of concepts and their negations, sentence boundaries, and note section headers [7,8,9].

LLMs. LLMs autonomously glean relevant features from data, bypassing the laborious and expertise-dependent manual feature engineering process [10]. Moreover, they adeptly navigate intricate linguistic structures like long-distance dependencies, anaphora, and ambiguity, thereby augmenting accuracy in IE tasks [11]. By fine-tuning, LLMs swiftly acclimate to diverse domains and IE tasks, requiring minimal additional training data, thus rendering them versatile for myriad applications [12]. Given that LLMs continue to improve and become more accessible, they are poised to play a significant role in the future of IE.

1.2. Symptomatology

Background. Clinical practice faces challenges in patient diagnosis, particularly in primary care, where, for example, fatigue is a common symptom among many diseases and in isolation may not provide significant predictive power. Predictive models that use a patient’s presenting symptoms to provide clinical decision support can help reduce diagnostic error. However, in order for these systems to be accurate, there is a need to extract reliable information related to patient symptomatology, which most often exists in unstructured data. Studies have shown that NLP methods using patient symptoms mapped to UMLS concepts can increase efficacy for predictive modeling in various contexts, including better colorectal cancer prediction [13] enhanced influenza prediction [14], and detection of surgical complications [15].

Recognizing the worldwide burden of PASC and the need for rapid diagnosis, in 2020, our team developed an NLP system as part of an artificial intelligence (AI) automation pipeline that could autonomously extract acute COVID-19 and PASC symptoms from longitudinal clinical notes at scale across a network of 12 Midwest U.S. hospitals and 60 primary care clinics [2]. Both structured clinical data and NLP-based features engineered from patient symptoms extracted at scale from clinical notes were used to populate a registry of COVID-19 patients [2,16]. We utilized our AI automation pipeline for monitoring PASC at the patient level for over 83,850 patients diagnosed with acute COVID-19, showing that these data could effectively surveil post-COVID patients to accurately identify patients suspected of having PASC [17]. This previous methodology utilized both rule-based and UMLS-based approaches for autonomous symptom extraction within our AI pipeline [2,16]. Over the past 4 years, LLMs have been developed and are rapidly becoming the status quo in general purpose NLP. However, their role is less certain within biomedical applications, and previous work has shown that bidirectional encoder representations from transformers (BERT) [18] based approach may outperform LLMs for certain specific biomedical tasks [19].

1.3. Objective

The objective of this study was to fine-tune an LLM for extraction of patient symptoms from unstructured clinical notes and demonstrate the comparable effectiveness of the fine-tuned LLM to rule-based IE methods. This study aimed to compare the LLM against BioMedICUS for symptom detection from clinical notes. We also aimed to explore the possibility of integrating the fine-tuned LLM into the BioMedICUS pipeline. Notably, BioMedICUS stands out for its capability to handle notes in rich text format (RTF), a feature not commonly found in other NLP systems, which made it an appealing choice for our purposes [8].

2. Materials and Methods

2.1. Data Sources

Corpora. This study utilized three different corpora of ground truth labeled clinical notes for fine-tuning and testing of our LLM and for comparison with BioMedICUS. University of Minnesota (UMN) COVID, a corpus of emergency department (ED) visit notes for patients diagnosed with acute COVID-19 as described in [2] and UMN PASC, a corpus of ED and outpatient (OP) visits for patients diagnosed with acute COVID-19 followed by lingering symptoms were provided by UMN/MHealth Fairview and were manually annotated by three subject matter experts with an inter rater reliability of 0.68 computed using Fleiss’ kappa [20]. The National COVID Cohort Collaborative (N3C) COVID, a fully de-identified corpus of notes for patients diagnosed with acute COVID-19 was provided by the Mayo Clinic. These three corpora were used because they were available with ground truth annotations for symptom identification and offered an opportunity for both internal and external validation of the fine-tuned LLM. A summary of each corpus is presented in Table 1.

UMN PASC. The inclusion criteria for the UMN PASC corpus mirror those of the UMN COVID corpus, with the following added criteria: patients must have (1) no baseline symptoms, (2) presented within a timeframe exceeding 30 days from their initial COVID-19 infection, and (3) at least one new or residual symptom [17]. Annotation guidelines instructed annotators to identify symptoms and their negations within the HPI section and other relevant sections. Clinical notes were randomly sampled from encounters at least 60 days after diagnosis with COVID-19. Demographics for this corpus are presented in Table 2.

N3C COVID. This corpus is a fully de-identified, manually annotated set of 148 unstructured notes across a selection of 5 inpatients and 5 outpatients with research authorizations as described by [16]. This corpus’s inclusion criteria were: (1) notes documented within two weeks before and four weeks after the lab order date of the first positive COVID-19 result, and (2) notes that contained more than 1000 characters. Up to 20 notes from each patient were included in this corpus [21,22].

2.2. Prevalence Analysis

We examined how document format (PASC) might influence training a model by analyzing symptoms as keywords. BioMedICUS’ annotations were converted to a normalized format using PASCLex to allow for keyword comparisons between corpora used in this study. Unlike a previous study by [6] that simply ranked terms by how often they appeared, this study used the Term Frequency-Inverse Document Frequency (TF-IDF) statistic to analyze how terms (keywords) are distributed across documents within each corpus. This approach provides a more nuanced picture than just counting total occurrences of a word in a corpus. A keyword’s frequency (TF) measures how often a word appears in a document, while a term’s inverse document frequency (IDF) measures how important a keyword is across all documents in the corpus. The product of TF and IDF thus helps identify keywords that are more meaningful or relevant to specific documents across the corpus. To compare keyword usage between corpora we performed an analysis of variance (ANOVA test) followed by ` [23,24]. Finally, we calculated effect size (η²) to assess the magnitude of these differences [24]. Table A2 shows the top 10 extracted keywords ranked by TF-IDF for normalized symptoms mapped to PASCLex.

2.3. Dataset Curation for Model Fine-Tuning and Validation

We developed our model using clinical notes from 522 patients: 46 with acute COVID-19 and 476 with PASC. The median age of the patients was 53.89 years, with 55% male. We divided the PASC corpus into three sets: 65% for training, 4% for validation, and 31% for hold-out testing. All COVID-19 patient notes were used in the training set because they were annotated by a single SME without a consensus process. To test the model’s generalizability, we included all patients from the N3C COVID corpus in our hold-out test set. This resulted in 351 patients in the training set, 20 in the validation set, and 299 in the hold-out test set. The demographic characteristics of each of these distinct corpora are delineated in Table 3.

2.4. Model Validation

Statistical Measures. Metrics including recall, precision, and F1-score were calculated to determine the models’s performance in applying extracted symptoms for labeling of clinical notes.

Fairness Analysis. To assess the LLM’s and BioMedICUS’ equity across race and gender, their performance was evaluated on the UMN PASC hold-out test set for which data on race or ethnicity and gender were available.

Error Analysis. To determine sources of false negatives (FN) for BioMedICUS and the LLM used in this study, we examined a few symptoms in which the false negative rate (FNR) was: (1) high for the LLM, but not BioMedICUS (fever for N3C corpus); (2) high for BioMedICUS, but not for the LLM (diarrhea for UMN PASC corpus); and (3) both high for BioMedICUS and LLM (chest pain for N3C corpus).

3. Results

3.1. Prevalence Analysis

Corpora Keywords Ranked by TF-IDF. Cohorts in each of the corpora used in this study had top 10 extracted keywords ranked by TF-IDF for normalized symptoms mapped to PASCLex as summarized in Table A1.

Hypothesis Testing/Keyword Analysis. We observed a statistically significant difference in the logarithm of TF-IDF values based on symptoms (F(16, 158) = 6.48, p < 0.0001). Furthermore, there was a statistically significant difference in the logarithm of TF-IDF values based on corpus (F(2, 158) = 79.17, p < 0.0001). Results of post hoc testing indicated significant differences between UMN COVID and UMN PASC (p < 0.0001) as well as UMN PASC and N3C COVID (p < 0.0001). However, there was no difference between UMN COVID and N3C COVID. Furthermore there were large effects for both corpus, η² = 0.46 and symptoms, η² = 0.4.

3.2. Performance Evaluation

To validate the LLM’s performance and compare the results with BioMedICUS, we used the hold-out test set to evaluate extraction performance. Table 4 symptom-wise weighted macro-average scores of performance metrics (as described in Section 2.4) calculated over UMN PASC and N3C COVID symptoms. Figure 1 and Figure 2 shows a graphical comparison of extraction performance of BioMedICUS and LLM on the hold-out test set of manually annotated reference over positive mentions for UMN PASC and N3C COVID respectively.

As seen in Table 4, Overall, BioMedICUS outperforms LLM across most symptoms and datasets. However, LLM shows superior f1-scores (on UMN PASC + and -) in specific cases, such as changes in sleep and bone or joint pain. Despite this, there are instances where both models perform similarly. For example, they achieve nearly identical F1-scores (on UMN PASC +) for diarrhea, shortness of breath, headaches, nasal congestion. These examples indicate that while BioMedICUS has a broader consistent advantage, LLMs can be more effective for certain symptoms and performs comparably to BioMedICUS in specific scenarios, demonstrating the robustness of both models in extracting clinical symptoms across different datasets.

Fairness analysis. For gender, both models perform slightly better for males, with recall, precision, and F1-scores higher than those for females in both positive (symptom presence) and negative (symptom negation) symptom mentions. In terms of race, both models show the highest performance for the ‘White’ and ‘Other’ categories in positive mentions. However, for the LLM, negative symptom mentions demonstrate reduced performance across all racial groups, particularly for those of Asian descent. Table 5 shows the results of LLM’s and BioMedICUS’ equity in the UMN PASC corpus.

Error Analysis. A manual audit of a small sampling of cases from each corpus with false negatives for positive mentions of “fever” (in the N3C COVID corpus) and “diarrhea” (in the UMN PASC corpus) are shown in Table 6. BioMedICUS had a false negative rate of 0.28, 0.44 and 0.55 respectively for “fever,” “diarrhea” and “chest pain” while the LLM had a FNR of 0.54, 0.22 and 0.88 for “fever,” “diarrhea” and “chest pain”, respectively.

4. Discussion

This study aimed to fine-tune an LLM for extraction of patient symptoms from unstructured clinical notes and demonstrate its comparable effectiveness to rule-based IE methods. We found that the LLM-based approach not only matches but in some cases exceeds the performance of rule-based methods. This study had 3 key findings: (1) BioMedICUS showed higher recall for certain symptoms, while the LLM excelled in precision and f1-scores for others, while both had similar performance for some symptoms; (2) Manual audits revealed higher FN for BioMedICUS and higher false positives (FP) for the LLM, often due to misinterpreting context or section headers; (3) The LLM had generalizability and bias issues due to overfitting from training on a non-diverse dataset and single annotator bias, highlighting the need for diverse, high-quality datasets and robust annotations.

4.1. Prevalence Analysis

As shown in Table A1, the ranking of top symptoms using TF-IDF varies across corpora, except between UMN COVID and N3C COVID. For example, the symptom "fever" is ranked fifth in the UMN PASC corpus, but it ranks first and second in the UMN COVID and N3C COVID corpora, respectively. Additionally, the number of times "fever" appears in these corpora differs even more. Symptom mentions of “diarrhea” do not appear among top rankings of UMN PASC by TF-IDF. Lastly, “chest pain” does not appear among top ranked symptoms in any of these corpora. Results of the ANOVA test confirm that both structure and content of these corpora significantly influence the TF-IDF scores. This is likely because the corpora are organized differently or contain different information. Moreover, the large effect sizes (η² = 0.4 for corpora and η² = 0.42 for symptoms) show that both the corpus format and the presence or absence of symptoms have a substantial influence on the TF-IDF score distribution across the corpora.

4.2. Error Analysis

As shown in Table 6, for the N3C COVID corpus, both BioMedICUS and the LLM missed various terms for "fever" not present in either the UMLS or the LLM’s lexicon. Additionally, the LLM overlooked contextual sentences ending with "had been sick". However, while the UMLS lexicon missed terms like "increase in temperature", the LLM detected them, demonstrating its sensitivity to such phrases due to its pre-training on general English. For "diarrhea", both BioMedICUS and the LLM incorrectly negated sentences containing "diarrhea" that ended with "resolved" missing the temporality of these mentions. However, the LLM correctly contextualized a negating phrase unrelated to having "diarrhea", whereas BioMedICUS did not. For "chest pain", both classifiers missed terms not found in the UMLS or the LLM’s lexicon, and the LLM failed to classify terms with both positive and negative mentions of "chest pain" accurately.

For both the LLM and BioMedICUS various symptoms had been mislabeled as FP due to neither using section header detection to filter out irrelevant mentions from notes. On the other hand, certain templated sections within clinical notes were correctly identified by LLM but not by BioMedICUS. Consequently, we noted a significant difference between RTF structured and unstructured versions of the UMN PASC corpus using BioMedICUS’ rule-based selection header detection: In RTF notes, symptoms under "HPI" were identified 785 times compared to 424 times in unstructured notes, suggesting that RTF’s structure aids in distinguishing relevant section headers, with potential for reducing FP.

4.3. Performance Evaluation

Comparing the generalizability of the fine-tuned LLM and BioMedICUS for extracting symptoms from clinical notes reveals key differences. BioMedICUS, which uses techniques like "normalized bag of words" and UMLS, shows high generalizability due to its universal rules not limited by specific corpora, effectively identifying symptoms across diverse datasets. In contrast, the LLM’s performance is closely tied to its training data, mostly from UMN PASC patients, with only 12% from UMN COVID patients. Consequently, the LLM performed similarly to BioMedICUS on the UMN PASC hold-out test set but underperformed on the acute N3C COVID test set, highlighting issues of overfitting and inadequate acute COVID-19 patient representation in the training data. Additionally, the LLM’s UMN COVID training labels, annotated by a single annotator without inter-annotator agreement, likely reduced its performance. These findings suggest that while LLMs offer flexibility in capturing nuanced symptoms, their generalizability can be significantly improved with well-annotated datasets, regularization, and robust prompt engineering.

We found that for positive mentions of UMN PASC symptoms, choosing between LLM and BioMedICUS depends on priorities: BioMedICUS prioritizes maximizing identification with higher recall, while the LLM prioritizes minimizing FP with better precision. For negative mentions, while BioMedICUS appears more robust across a broad range of symptoms, LLM’s strengths in specific areas suggest its value for targeted symptom identification. For N3C COVID symptoms, BioMedICUS consistently outperformed the LLM across most positive and negative symptom mentions, indicating its ability to generalize knowledge to new data.

Our results align with experiments performed by Patra, et al. who compared the performance of rule-based systems with LLMs for extracting information from clinical psychiatry notes. They found that the rule-based systems outperformed the LLMs across most of their cases [25]. Our findings are further supported by Chen, et al., who compared LLMs like GPT-3.5 and GPT-4 [26] against fine-tuned BERT and BART [27] models for biomedical natural language processing applications. They found that GPT-4’s performance was competitive or better than the fine-tuned BERT and BART models in 6 out of 12 benchmarks especially for reasoning and generative tasks [28].

4.4. Limitations

The limitations of our study include: (1) Generalizability of the LLM: being trained on limited datasets, as presented in Table 4, the LLM lacks generalizability, furthermore, effect size of corpora differences may limit model generalizability. (2) Lack of reliable section header detection: the inability to detect and leverage section headers hinders the performance of our models. (3) Length of notes: some notes might be too large to be processed by our fine-tuned LLM. In such cases, the notes have to be processed in chunks, which slows down the process. (4) Lexica used in this study may be incomplete thus leading to increase in FN across some symptoms.

4.5. Future Work

Future work includes: (1) Expanding LLM generalizability: expanding the diversity of the training dataset, applying robust regularization techniques, involving multiple annotators for high-quality labels, using domain adaptation techniques, and employing hybrid approaches to enhance model output. (2) Training models on desired section headers from the notes: use of RTF tags to identify section headers like HPI, has potential to enhance model performance and thus resolve the issue of some notes being too large for the LLM model to process. (3) Training models on structured notes: RTF structured notes offer formatting features with tags, including text styling, tables, and hyperlinks, aiding in symptom identification and severity assessment. (4) Comparison with other fine-tuned LLMs (5) Exploring methods that combine LLMs and rule-based systems. (6) Detecting the severity of symptoms: future research aims to integrate severity detection for patient symptoms allowing clinicians to track changes in symptom severity over time, aiding in treatment adjustments with potential to improve patient outcomes.

5. Conclusions

This study highlights the significant issues of model bias and generalizability in our fine-tuned LLM due to demographic variability and corpus differences. Our evaluation revealed that the LLM, trained predominantly on UMN PASC patients with limited acute COVID-19 representation, showed overfitting and insufficient generalization, potentially exacerbated by single annotator bias. These findings provide real-world evidence of the challenges LLMs may face, underscoring the necessity for diverse, high-quality training datasets and robust annotation processes to mitigate bias and enhance generalizability. Our findings also indicated that the rule-based approach and the LLM approach each have strengths and limitations in performing IE tasks applied to clinical NLP.