Evaluating a Large Language Model for Semi-Automated Versus Human-Only Data Extraction in Evidence Synthesis: A Study Protocol

Sarah Elliott; Samantha Guitard; Jason Pagaduan; Henry Li; Lisa Hartling

doi:10.20944/preprints202606.0407.v1

Submitted:

02 June 2026

Posted:

04 June 2026

You are already at the latest version

Abstract

Background: When synthesizing large bodies of information on a topic, the quality and accuracy of the information is essential. Data extraction plays an important role in the evidence synthesis process, yet it remains time-consuming, labor-intensive, and prone to errors. Artificial Intelligence (AI) tools, specifically large language models, offer the potential to expedite this task. Evidence is mounting that suggests AI-assisted methods can perform similarly to traditional human-only data extraction processes. Objectives: We will explore the concordance of AI-assisted data extraction (Claude Sonnet 4.6) versus human-only extraction methods. Secondary objectives will include whether data extracted by Claude Sonnet 4.6 over multiple time points (one week, one month) will be stable, as well as whether it has the ability to identify and correct errors in its extractions when prompted.Methods: We chose a random sample of published, open access, randomized controlled trials (RCTs) focused on pediatric research in Canada. For our primary objective, we will compare a fully human process for data extraction and verification, and a process in which a human prompts the LLM to extract data and another human verifies it. A senior team member will be designated as the independent, blinded outcome adjudicator and will create the reference standard. Secondary objectives will be explored on a subset of included RCTs. Results: Data (study design, sample characteristics, intervention details, and quantitative outcomes etc.) will be extracted and verified by each method for 100 RCTs. Firstly, we will report concordance (factual agreement between extraction methods), and discordance between the two methods. Then, discordant data will be evaluated against the reference standard to calculate accuracy (correct reporting of data, including missing data), positive predictive value (proportion of extracted elements that are correct), sensitivity (proportion of reported elements correctly extracted), and the F1 score, a harmonic mean of positive predictive value and sensitivity. For our secondary outcomes, stability of the LLM over time will be evaluated by concordance; the ability to identify and correct errors will be summarized descriptively.Conclusion: Large language models may offer practical solutions when there are barriers to completing evidence syntheses in a timely manner, without sacrificing quality and accuracy that are expected from traditional human processes. This work will help define key applications and considerations for using LLMs in evidence synthesis. Evidence of this nature is critical to advance methods for the responsible use of AI in evidence synthesis.

Keywords:

artificial intelligence

;

Claude

;

data extraction

;

evidence synthesis

;

large language model

Subject:

Public Health and Healthcare - Public Health and Health Services

Background

When a body of research accumulates on a particular topic, high-level reviews become essential to synthesize the evidence and provide a comprehensive overview. Data extraction, the collection of relevant study information into structured and standardized forms, is a vital step during the evidence synthesis process [1]. The Cochrane Collaboration is recognized as a gold-standard for rigorous evidence synthesis methods. They require the use of dual, independent data extraction by two reviewers for outcome data, and consider it highly desirable to perform all extractions in duplicate to reduce the chance of errors [1,2,3]. To manage feasibility, piloting the data extraction form independently on a sample of included studies until reaching high agreement is a practical alternative to dual extraction of all studies [4]. Reviews that require more streamlined approaches due to feasibility (e.g., costs or time constraints) may use one reviewer for extraction, and another reviewer to verify the data for completeness and accuracy [5].

Frequent error types in data extraction include data omissions, typographical errors in data entry, and misinterpretation of the study data [6,7]. Some errors are more problematic than others, such as changes to the interpretation of findings in the review. The amount of detail in the data extraction form, the level of expertise of the review team, and understanding of the research topic can all contribute to data extraction errors. Properly training the review team, developing a structured data extraction form, and piloting the form with clear instructions is essential, yet, even the most rigorously designed reviews can be prone to human error [1,8,9].

The proliferation of large language models (LLMs) presents a growing opportunity for use in the evidence synthesis process [10]. In particular, LLMs have the potential to create efficiencies for time-on-task while maintaining high accuracy in data extraction [11,12]. One way to use LLMs could involve a semi-automated approach, whereby an individual creates detailed data extraction instructions submitted to the LLM as prompts, and a second reviewer verifies the extracted data against the original study report. A fully automated approach would involve a person creating and submitting prompts to the LLM, and creating additional prompts to have the LLM re-check its own decisions without a human performing the verification. To-date, a semi-automated approach has been the focus of research, as human oversight is still considered essential for data accountability. A position statement co-released across several research bodies (Cochrane, the Campbell Collaboration, JBI, and the Collaboration for Environmental Evidence) on the use of artificial intelligence underscores the responsibility of researchers for the final output of their work, as well as transparent reporting when using AI in their methods [13]. Draft guidance documents for the Responsible Use of AI in Evidence Synthesis (RAISE) have been developed. As well, reporting guidelines for studies evaluating AI have been developed and are accessible through the EQUATOR network, e.g., Chatbot Assessment Reporting Tool (CHART) [14,15,16].

Identifying which LLM to use in evidence synthesis may involve both consideration of cost and previous experimental evidence demonstrating effectiveness. To-date, a variety of LLMs have been tested against human reviewers for data extraction accuracy [12,17,18,19,20,21]. In a study comparing data extraction using two widely used proprietary LLMs (Claude 2 and GPT-4), Claude 2 had higher accuracy compared to GPT-4 mainly due to limitations in GPT-4’s ability to analyze PDFs [21]. Claude models appear to be some of the most commonly used tools among researchers in evidence synthesis due to their ability to upload and read PDF documents while respecting intellectual property rights [22]. Further, Anthropic’s previous policy used an opt-in model for AI training, although they have moved to an opt-out model more recently [23]. Additional details surrounding their training data, test methods and results can be found under their transparency statements [22]. Evidence has shown Claude models 2.0, 2.1, 3.0 Opus, and 3.5 Sonnet can reduce human error and achieve over 90% accuracy in data extraction of randomized and non-randomized studies included in systematic reviews [17,18]. Developers continue to update LLM models, and preliminary evidence monitoring ChatGPT’s gpt-3.5-turbo output on a month-to-month basis showed some capabilities change and improve over time both within the same version, and as the model undergoes version updates [24]. Claude will also require both short- and long-term monitoring to ensure it has stable and effective performance [25].

This study aims to evaluate the concordance between LLM-assisted data extraction using Claude Sonnet 4.6 (newest version) and a traditional human-only extraction process. For discordant data between the two processes, we will assess accuracy against a reference standard. We will also explore stability of extractions by Claude Sonnet 4.6 over time. To our knowledge, prior evaluations of Claude focus on comparing different model versions to human performance, rather than examining the temporal consistency (i.e., stability) of a single model version. Furthermore, to our knowledge it is also unknown whether an LLM can reliably detect and correct its own errors, a requirement for fully automated data extraction. As secondary objectives, we will also investigate these functions.

Methods

Research Questions

Our study was designed to address the following research questions:

Primary research questions:

What is the concordance between LLM-assisted versus human-only data extraction methods?
What is the accuracy of an LLM-assisted versus human-only data extraction method, as compared to a reference standard?

Secondary research questions:

3.: What is the stability (i.e., concordance between data sets) using the same LLM over time (at baseline, one week later, and one month later)?
4.: Can an LLM identify and correct its data extraction errors?

Study Design

This is a prospectively designed study to evaluate the use of an LLM (i.e., Claude) in the data extraction process.

Study Sample

We will use a random sample of randomized controlled trials (RCTs) included for a meta-epidemiological project focused on pediatric trials in Canada (data unpublished), modelled after a recent publication in an adult population [26]. The pool of eligible RCTs for this study will be open-access and exclude pilot/feasibility designs. 110 randomly selected RCTs will be used in the evaluation of the primary study objectives; 10 for the pilot round to train the research team and refine the prompts for the LLM, and another 100 RCTs for the formal comparison of LLM-assisted versus human-only extraction methods. The secondary objectives will also use the 10 pilot studies for training, as well as additional sample of 25 RCTs from the formal comparison of the primary objective. Randomization will be done in Excel with the =rand() function. Supplementary Files that contain relevant data (characteristics tables, outcome data) will be identified before data extraction of each study. They will be manually uploaded to the LLM with the PDF of the included article. The clinical trial registry will be accessed when required by using Claude’s connector feature. We will specify in our codebook and structured prompts when to access the registry for specific information, i.e., only when it is omitted from the study report (province of included study sites or what the primary outcome is; see Supplementary File). Using this hierarchy will ensure that if any discrepancies between the published study and clinical trial registry are observed, the data in the published study will be given preference.

LLM Model for Prompt Engineering and Data Extraction Process

For this study we chose to use the paid version of the newly released Claude Sonnet 4.6 (February, 2026) [25]. We chose Sonnet 4.6 as it provides the best balance of cost, speed and effective reasoning among Anthropic’s Claude models [27]. Initial prompts for data extraction can be found in the Supplementary File, and will be first modelled off of previous work in the field of LLM-assisted data extraction [18]. Prompt engineering will include revisions to the prompts during pilot testing that correct errors and ensure concise and accurate output. Revisions during this process will be documented in the full report/publication.

Primary Study Objectives

For our primary study objective, we will use two research teams with an additional blinded outcome adjudicator (see Figure 1). Study team 1 will comprise of two research team members, one who will extract data from the PDF of the article (and associated files), and the other who will verify the data against the same study documents. Study team 2 will have two research team members, one who will prompt the LLM to extract data from the PDF of the article (and associated files), and a second human reviewer to verify the extracted data against the same study documents. The outcome adjudicator will then compare results between the two teams to assess concordance, and perform adjudication of any discordant data between the teams to create a reference standard for accuracy (see Table 1 for definitions).

Pilot Data Extraction and Prompt Engineering

A pilot round of data extraction will include all study team members to ensure accurate and consistent use of the extraction form instructions. Each team member will independently extract data from 10 randomly selected studies into a structured Excel spreadsheet with use of a codebook for guidance. The team member assigned to prompt the LLM for data extraction will also pilot the 10 studies using the codebook structured as prompts and compile the LLM outputs. All team members will then meet for consensus to discuss any discrepancies, revise the codebook, and adjust the LLM prompts through prompt engineering as required. Final prompts will be decided by the entire study team that result in the most accurate outputs as compared to the team’s consensus decisions during the pilot. Once the prompts are finalized after the pilot stage there will be no additional prompt refinement so that the LLM will be reliant solely on its pilot training before the human verification step.

Study Materials for Primary Study Objectives

The revised codebook/prompts, the standardized Excel spreadsheet, the PDF of each of the 100 randomly selected studies (independent from the 10 pilot studies), and the relevant Supplemental Information will be provided to each team. Each team will complete the 100 extractions and verifications per their respective approaches.

Outcome Adjudication for Primary Study Objectives

A final Excel spreadsheet with completed extractions and verifications will be prepared for each data extraction approach. Comparisons between teams and adjudication will be done in a blinded fashion by a senior, experienced member of the research team, only involved in piloting the data extraction form (not involved in data extraction and verification within either study team). They will compare data extractions between each method and classify each data element as concordant or discordant. Concordant data elements will not be assessed for accuracy against a reference standard. For all discordant data elements, the outcome adjudicator will read the study (and related materials), and provide answers, which will be considered the reference standard for accuracy. Within the discordant data elements, differences between teams will be categorized as whether one team had no error in the data element or whether one or both teams have an inconsequential, minor or major error. Classification of no error will mean the study team has the same answer as the reference standard, whereas an error will mean there is a different answer than the reference standard.

Inconsequential errors will be defined as most likely not affecting interpretation of data (e.g., inclusion criteria level of details); minor will be defined as less severe than major that may or may not affect interpretation of the data (e.g., small calculation or rounding errors or modified language around inclusion/exclusion criteria that does not critically alter the overall use or meaning); major will be defined as a substantial error compromising the accuracy of the data, where left uncorrected will lead to inaccurate conclusions (e.g., completely incorrect calculations, misallocated or fabricated data leading to new or significantly different interpretations) [18].

Blinding (Primary Objectives)

For our primary objective, the outcome adjudicator acting as the reference standard will be blinded to the method of extraction and verification (whether LLM-assisted or human-only). Study members on team 1 and 2 will not have access to each other’s verified data, and the team 2 member performing verification will ensure that any identifiable information from the LLM extraction is edited or removed before adjudication to maintain blinding, while ensuring important information for the data element is not also removed. Identifiable information not related to the data element may include statements that are responding to prompts, rather than providing relevant extracted data (e.g., “mean age was not available, therefore I provided median age”).

Outcomes (Primary Objectives)

The primary outcome will be concordance between LLM-assisted (Claude Sonnet 4.6) and human-only extraction methods.

Secondary outcomes will include accuracy, positive predictive value (PPV), sensitivity, and the F1 score from the discordant data between each method, as compared to the reference standard. These metrics will be calculated by identifying true positives (TP; correctly extracting a data element), true negatives (TN; correctly identifying a data element is not present), false positives (FP; incorrectly identifying a data element is present), and false negatives (FN; incorrectly identifying a data element is not present or extracted incorrectly). For discordant data between methods, when no errors are found the data element will thus be either a TP or TN. If data elements have an error (inconsequential, minor or major) they will be either a FP or FN. Definitions of each outcome and their calculations are found in Table 1.

Secondary Study Objectives

Stability of LLM Output over Time

As a secondary objective of this study, we will evaluate the stability of the LLM’s output from baseline (extractions performed during the evaluation of the primary objective), one week later, and one month later. We will include the 10 pilot studies from training, as well as an additional random sample of 25 RCTs (and associated files) from the primary objective. One reviewer will prompt the LLM to perform extractions at each timepoint, using new sessions of Claude and repeating the pilot of 10 studies to train each new session in the same manner. The same initial set of prompts and later revisions will be used. Then they will prompt Claude to extract the same 25 studies selected randomly from the baseline session. We will disable Claude’s features for retaining context and user preferences across chats to ensure there is no influence of prior training. The outcome of interest will be concordance between data elements extracted at each timepoint. This will allow us to evaluate how consistent the output is, but we will not assess discordant data for errors.

LLM-Extraction with LLM-Verification

To test the ability of an LLM to identify and correct its own errors during data extraction (without a human verification step), this secondary objective will involve prompting the pilot trained LLM by a team member after the primary objective is complete. Data extractions (LLM outputs from study team 2, before any human verification) will be compared against discordant data between both study team’s verified data from the primary objective, using the reference standard to classify data elements with errors (and their severity) or with no errors. The extractions will then be uploaded back into the LLM with a representative sample of data elements containing no errors, and errors of inconsequential, minor and major severity. We will prompt the LLM with instructions to verify its data extractions against the study PDF (and associated files), and record whether it is able to identify errors when present. If it is unable to identify or correct errors that we have found, we will refine the prompts to specifically ask which data elements we would like corrected, and re-assess its performance. Performance outcomes will be the number of correct, incorrect, and missed corrections to the LLM data extractions, before and after specifying what elements are incorrect. We will also describe the types of errors (inconsequential, minor, and major) that were successfully versus unsuccessfully corrected.

Blinding (Secondary Objectives)

The secondary objectives will not be blinded.

Sample Size and Data Analysis

Each data element extracted into each cell of the Excel spreadsheet will be considered the units of analysis. Some data elements may be related to one another (e.g., timing and point estimate of the primary outcome will be dependent on correctly reporting/identifying the primary outcome). Therefore, if the first element is incorrect, we will not count the second as incorrect to avoid overestimating the incorrect proportion of analysis units. There are 70 data elements in the Excel spreadsheet to extract per study, with 44 independent and 26 dependent data elements. Therefore, we will have 4400 unique data elements across 100 RCTs for the primary study objective outcomes of concordance, accuracy, positive predictive value, and sensitivity. Based on previous sample size calculations [18] we will have the power to reach an observed binomial proportion of 83%, with a 2-sided 95% Clopper-Pearson confidence interval roughly 7 percentage points wide. The number of true positives will follow a binomial distribution, and the sample size will be defined as the corresponding denominator. We will also report the concordance, accuracy, positive predictive value, and sensitivity for data elements related to how we chose to organize each main section of the standardized Excel spreadsheet: Publication characteristics, Trial Design, Setting, Recruitment, Intervention/Comparator and Outcome, Participant Characteristics, and Outcome Reporting.

Our secondary objective assessing concordance over time will be evaluated by extracting the same 70 data elements per study, with 44 independent and 26 dependent data elements (1,100 total data elements across 25 studies). This will also allow us sufficient power perform the same calculations for confidence intervals as the primary study objective.

Our additional secondary objective to examine the ability of the LLM to identify and correct errors will be exploratory; therefore, no formal statistical analysis will be performed. We will summarize the results descriptively.

Conclusions

Large language models have the potential to expedite one of the most labour-intensive tasks in the evidence synthesis process, data extraction. As evidence mounts suggesting that certain LLM-assisted methods can perform similarly to traditional human processes, our goal is to contribute to this important body of evidence, evaluating the newest version of the Claude Sonnet model versus human-only data extraction. Output from LLMs can and will change over time, therefore we will explore LLM stability using the same model version at different timepoints, while maintaining the same initial training steps and subsequent prompts. We also seek to understand whether an LLM-prompted data extraction process could be verified solely by prompting the trained model to re-check its work, or if it will require more focused instructions to locate errors. This study will contribute to a growing body of evidence that is critical to inform the responsible use of AI, and the findings will help define key applications and considerations for using LLMs in evidence synthesis.

Supplementary Materials

The following supporting information can be downloaded at: Preprints.org.

Funding

No external funding was received for this research. LH is supported by a Canada Research Chair in Knowledge Synthesis and Translation.

Disclosures

The authors report no potential conflicts of interest related to this study.

Ethics statement

This research did not require ethics approval as this study exclusively analyzed data from previously published, publicly available literature.

Abbreviations

LLM; large language model, ES; evidence synthesis, DE; data extraction.

References

Li, T.; Higgins, J.; Deeks, J. Chapter 5: Collecting data. In Cochrane Handbook for Systematic Reviews of Interventions version 65; Cochrane, 2019. [Google Scholar]
Methods, Cochrane. Leading innovation in evidence synthesis. 2024. Available online: https://www.cochrane.org/about-us/news/cochrane-methods-leading-innovation-evidence-synthesis.
Buscemi, N.; Hartling, L.; Vandermeer, B.; Tjosvold, L.; Klassen, T.P. Single data extraction generated more errors than double data extraction in systematic reviews. J. Clin. Epidemiol. 2006, 59(7), 697–703. [Google Scholar] [CrossRef] [PubMed]
Shea, B.J.; Reeves, B.C.; Wells, G.; Thuku, M.; Hamel, C.; Moran, J.; et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ 2017, 358, j4008. [Google Scholar] [CrossRef] [PubMed]
Nussbaumer-Streit, B.; Sommer, I.; Hamel, C.; Devane, D.; Noel-Storr, A.; Puljak, L.; et al. Rapid reviews methods series: Guidance on team considerations, study selection, data extraction and risk of bias assessment. BMJ Evid. Based Med. 2023, 28(6), 418–23. [Google Scholar] [CrossRef] [PubMed]
Gotzsche, P.C.; Hrobjartsson, A.; Maric, K.; Tendal, B. Data extraction errors in meta-analyses that use standardized mean differences. JAMA 2007, 298(4), 430–7. [Google Scholar] [CrossRef] [PubMed]
Kanellopoulou, A.; Dwan, K.; Richardson, R. Common statistical errors in systematic reviews: A tutorial. Cochrane Evid. Synth. Methods 2025, 3(2), e70013. [Google Scholar] [CrossRef] [PubMed]
Büchter, R.B.; Weise, A.; Pieper, D. Development, testing and use of data extraction forms in systematic reviews: a review of methodological guidance. BMC Med. Res. Methodol. 2020, 20(1), 259. [Google Scholar] [CrossRef] [PubMed]
Büchter, R.B.; Weise, A.; Pieper, D. Reporting of methods to prepare, pilot and perform data extraction in systematic reviews: analysis of a sample of 152 Cochrane and non-Cochrane reviews. BMC Med. Res. Methodol. 2021, 21(1), 240. [Google Scholar] [CrossRef] [PubMed]
Schmidt, L.; Finnerty Mutlu, A.N.; Elmore, R.; Olorisade, B.K.; Thomas, J.; Higgins, J.P.T. Data extraction methods for systematic review (semi)automation: Update of a living systematic review. F1000Res 2025, 10, 401. [Google Scholar] [CrossRef] [PubMed]
Chen, D.; Alnassar, S.A.; Avison, K.E.; Huang, R.S.; Raman, S. Large Language Model Applications for Health Information Extraction in Oncology: Scoping Review. JMIR Cancer 2025, 11, e65984. [Google Scholar] [CrossRef] [PubMed]
Liu, J.; Lai, H.; Zhao, W.; Huang, J.; Xia, D.; Liu, H.; et al. AI-driven evidence synthesis: data extraction of randomized controlled trials with large language models. Int. J. Surg. 2025, 111(3), 2722–6. [Google Scholar] [CrossRef] [PubMed]
Flemyng, E.; Noel-Storr, A.; Macura, B.; Gartlehner, G.; Thomas, J.; Meerpohl, J.J.; et al. Position Statement on Artificial Intelligence (AI) Use in Evidence Synthesis Across Cochrane, the Campbell Collaboration, JBI, and the Collaboration for Environmental Evidence 2025. Campbell Syst. Rev. 2025, 21(4), e70074. [Google Scholar] [CrossRef] [PubMed]
Thomas, J. F.E.; Noel-Storr, A.; et al. Responsible use of AI in evidence SynthEsis (RAISE): building and evaluating AI evidence synthesis tools (updated Mar 2026). In Open Science Framework; Center for Open Science: Washington, DC, 27 May 2026; Available online: https://osf.io/fwaud/overview.
The CHART Collaborative. Reporting guidelines for chatbot health advice studies: explanation and elaboration for the Chatbot Assessment Reporting Tool (CHART). BMJ 2025, 390, e083305. [Google Scholar] [CrossRef] [PubMed]
EQUATOR Network. Reproting guidelines: Artificial Intelligence/Machine Learning Studies. 2026. Available online: https://www.equator-network.org/reporting-guidelines-study-design/artificial-intelligence-machine-learning-studies/.
Gartlehner, G.; Kahwati, L.; Hilscher, R.; Thomas, I.; Kugley, S.; Crotty, K.; et al. Data extraction for evidence synthesis using a large language model: A proof-of-concept study. Res. Synth. Methods 2024, 15(4), 576–89. [Google Scholar] [CrossRef] [PubMed]
Gartlehner, G.; Kugley, S.; Crotty, K.; Viswanathan, M.; Dobrescu, A.; Nussbaumer-Streit, B.; et al. Artificial Intelligence-Assisted Data Extraction With a Large Language Model: A Study Within Reviews. Ann. Intern Med. 2025, 178(12), 1763–71. [Google Scholar] [CrossRef] [PubMed]
Helms Andersen, T.; Marcussen, T.M.; Termannsen, A.D.; Lawaetz, T.W.H.; Norgaard, O. Using Artificial Intelligence Tools as Second Reviewers for Data Extraction in Systematic Reviews: A Performance Comparison of Two AI Tools Against Human Reviewers. Cochrane Evid. Synth. Methods 2025, 3(4), e70036. [Google Scholar] [CrossRef] [PubMed]
Murton, M.; Boulton, E.; Cross, S.; Khan, A.; Kumar, S.; Magri, G.; et al. Harnessing Large-Language Models for Efficient Data Extraction in Systematic Reviews: The Role of Prompt Engineering. Cochrane Evid. Synth. Methods 2025, 3(6), e70058. [Google Scholar] [CrossRef] [PubMed]
Konet, A.; Thomas, I.; Gartlehner, G.; Kahwati, L.; Hilscher, R.; Kugley, S.; et al. Performance of two large language models for data extraction in evidence synthesis. Res. Synth. Methods 2024, 15(5), 818–24. [Google Scholar] [CrossRef] [PubMed]
Anthropic. Anthropic’s Transparency Hub. 2026. Available online: https://www.anthropic.com/transparency.
Anthropic. Anthropic Announces Privacy Shift: Users Must Opt Out to Prevent Chat Data Being Used for AI Training. 2025. Available online: https://mlq.ai/news/anthropic-announces-major-shift-users-must-opt-out-to-prevent-chat-data-being-used-for-ai-training/.
Tu, S.; Li, C.; Yu, J.; Wang, X.; Hou, L.; Li, J. Chatlog: Carefully evaluating the evolution of chatgpt across time. arXiv 2023, arXiv:230414106. [Google Scholar]
Anthropic. Introducing Claude Sonnet 4.6. 2026. Available online: https://www.anthropic.com/news/claude-sonnet-4-6.
Ruzycki, S.M.; Lithgow, K.C.; Song, C.; Taylor, S.; Subramanian, A.; Li, M.; et al. Participant diversity and inclusive trial design: a meta-epidemiologic study of Canadian randomized clinical trials. J. Clin. Epidemiol. 2026, 191, 112098. [Google Scholar] [CrossRef] [PubMed]
Anthropic. Models overview: Choosing a model. 27 May 2026. Available online: https://platform.claude.com/docs/en/about-claude/models/overview.

Figure 1. Study flow diagram.

Table 1. Primary objective outcomes and definitions [14,18].

Outcome	Definition	Calculation
Concordance	The proportion of data items between the two extraction methods that were factually congruent, irrespective of any differences in formatting or structure	Concordant pairs/Total pairs
Discordance	The proportion of data items between the two extraction methods that were not factually congruent, irrespective of any differences in formatting or structure	Discordant pairs/Total pairs
Discordant data elements
Accuracy	The proportion of data elements that were correctly reported, including correctly reporting missing data	(TP+TN)/(TP+FP+TN+FN)
Positive predictive value	The proportion of all extracted data elements that were extracted correctly, also called precision	TP/(TP+FP)
Sensitivity	The proportion of all data elements reported that were correctly extracted, also called recall or true positive rate	TP/(TP+FN)
F1 score	A measure of accuracy that creates a harmonic mean from PPV and sensitivity with a range of 0=poor to 1=perfect	2* [(PPV*Sensitivity)/(PPV+Sensitivity)]

FN=false negative; FP=false positive; PPV=positive predictive value; TP=true positive; TN=true negative.

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

© 2026 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

Copyright: This open access article is published under a Creative Commons CC BY 4.0 license, which permit the free download, distribution, and reuse, provided that the author and preprint are cited in any reuse.