Accelerating Disease Model Parameter Extraction: An LLM-based Ranking Approach to Select Initial Studies For Literature Review Automation

1. Introduction

Zoonotic diseases, are becoming increasingly more prevalent due to increased interactions between, humans, livestock, wildlife, disease vectors, and pathogens, exacerbated by climate change and rapid human expansion into natural habitats [1,2]. Globally zoonotic diseases have disproportionately impacted impoverished livestock workers in low- and middle-income countries, responsible for millions of human deaths every year [3]. To address this threat, there is a mounting need for systems to forecast and model the spread of diseases, thereby aiding public health planning and supporting early warning systems [1].

Constructing such models require precise parametrisation. This necessitates data from multiple sources, including clinical records, environmental datasets, grey literature and scientific publications [4,5]. Researchers conduct a systematic literature reviews (SLR), regarded the gold standard, to reliably extract this information. The key stages of the SLR process consists of; planning and developing protocols, searching for articles, screening articles for relevance, extracting information, assessing the quality of studies and finally summarising and reporting [6,7].

Despite the abundance of scientific publications, the manual extraction of disease parameters via SLRs remain a significant bottleneck. Conducting an SLR requires a significant time investment [8]; the process is costly, [9] and the exponential growth of publications [10] contributes to researcher fatigue and increased cognitive overload [11]. Finally, these reviews are difficult to keep current, leading to unnecessary waste and potential harm in decision-making [12]. The core issue lies in the inefficiency to robustly scale the manual processes. Automation offers a cost-efficient, faster, and more reliable solution, capable of meeting the demands for scale and quality [13,14,15].

Since the 2000s, systems based on natural language processing (NLP) and machine learning (ML) have emerged to automate or semi-automate key steps of SLR process[15]. Current ML based systems on the market continue to use frequency-based sparse vector representation of text or static word embeddings [16] which struggle to capture context and address challenges like polysemy. Moreover, many tools lack the capacity to generalise to other domains and have primarily been evaluated on specific research questions or target particular study designs [17].

In recent years, tools powered by transformer-based large language models (LLM), such as GPT, have gained significant popularity [16,18]. These models leverage vast amounts of text data and advanced contextual understanding, to handle complex language tasks, including screening abstracts [19,20,21,22,23], extracting data [23,24,25] and synthesise information [23,26,27]. However, integrating LLMs into SLR workflows pose major challenges [16]. They can produce plausible-sounding erroneous responses known as "hallucinations" [28], promote negative biases seen in training data [29], lack transparency and are considered "black-box" systems [30]. While these issues are significant, researchers are actively exploring various options to mitigate risks [30,31,32] and ensure reliable and effective integration of the technology into the workflow.

Climate change and zoonotic diseases present complex challenges that require a focused and innovative solution. While LLMs and automation tools demonstrate potential in transforming SLRs, a significant gap exists for domain-specific systems capable of managing diverse datasets. Tools that can provide scalable, accurate, and transparent actionable insights are urgently required. Therefore, this paper presents an overview of the conceptual design of CliZod (Climate Sensitive Zoonotic Database), a novel collaborative system for the automatic identification, extraction and storage of climate sensitive zoonotic disease parameters from scientific literature using artificial intelligence. This paper specifically focuses on the experimental validation of the relevancy ranking component located within title-abstract screening module of the system. The primary objective is to assess the effectiveness of employing an LLM-based assessor with a question-answer (QA) framework to guide the ranking of primary studies by relevance.

1.1. CliZod

The CliZod system is structured as a scalable and transparent modular framework to automate parameter extraction from scientific publications, specifically targeting climate-sensitive zoonotic disease modelling. The architecture comprises of a multistage workflow pipeline (Figure 1) guided by the the Vienna Principles [33]. The system is being developed as a collection of open-source modular web service components with well-defined APIs and standardised input/output formats. The intention is to promote flexibility, transparency and adaptability, allowing each component to function as a ’plug-and-play’ module that could be integrated into a customizable broader workflow. The intention is to enable researchers to easily adapt, extend, or reuse components across diverse use cases, making it a scalable solution for automating parameter extraction from literature.

The process commences with the user specification module, where the system gathers information regarding the research topic, selection criteria, parameters of interest, and establishes context for subsequent stages.

In the data acquisition module, automated search queries are utilised to search and retrieve titles and abstracts from designated online journal databases, followed by de-duplication and metadata harvesting. The next module is the screening of titles and abstracts, where articles are classified according to relevance. The full text screening module attempts to automatically download the full text of relevant articles when possible and seeks to parse and refine the screening process. The parsed data that contain information from text, tables, and figures, encompassing both structured and unstructured data, is used in the data extraction module to facilitate the identification and extraction of essential parameters. In the validation and integration phase, the data undergoes consistency and quality checks, followed by a final review before integration into the database.

1.2. Relevancy Ranking

Title-abstract screening is the most time-consuming aspect of the manual SLR process [34]. Best-practice guidelines recommend that primary studies are sorted by relevance before being screened. This can improve reviewer efficiency, increase productivity, and increase motivation [35]. Presenting users with a sorted list of title-abstracts, with the most relevant studies prioritised at the top, facilitates faster and more efficient decision making, especially for highly skewed datasets, such as those in the medical domain, which contain only $3\%$ to $6\%$ [36] relevant documents. The output of a relevancy ranking system can be directly utilised by selecting the top-k ranked articles based on an approximation of relevant document coverage [37,38] or, alternatively, assist users in selecting seed or example articles for input into a further specialised binary classifier, which is the use case of interest for the CliZod project.

1.2.1. Problem Definition

The focus of this study is to investigate the utility of using an LLM to rank primary studies by relevance. Given a set of documents $\mathcal{D}$ and a set of eligibility criteria $\mathcal{C}$, the task of the relevancy ranker is to assign each document $d\in \mathcal{D}$ a relevance score $r_d\in [0,1]$, where $r_d$ indicates how well document d satisfies the criteria $\mathcal{C}$. A score closer to 1 indicates higher relevance, while a score closer to 0 indicates lower relevance.

The eligibility criteria constitutes a series of rules and specific requirements that a document must satisfy for inclusion in the SLR. These criteria are typically evaluated based on attributes of the SLR protocol, including review title, keywords, research question or inclusion/exclusion criteria, but could also be framed as a question-answering problem [39]. While selection criteria are a common choice for LLM based binary screening solutions [19,40,41,42], the QA framework - also referred to as a screening tool in manual SLR workflow and regarded best practice [35] - offers a more fine-grained, consistent and targeted approach to determining eligibility.

Using the QA framework approach, Kusa et al. describes the eligibility criteria $\mathcal{C}$ being transformed into a set of questions $\mathcal{Q}=\{q_1,\cdots ,q_{\left|C\right|}\}$, where $q_k$ corresponds to a specific criteria C. A set of predicated answers ${\widehat{\mathcal{A}}}^d=\left\{{\widehat{a}}_k^d\right|\mathit{meets}(q_k,{\widehat{a}}_k^d)\}$ can be obtained for each document $d\in {\mathcal{D}}^{\prime }$, where $\mathit{meets}(q_k,{\widehat{a}}_k^d)$ denotes that the document d should meet the criterion expressed by $q_k$. The relevancy score ${\widehat{r}}_d$ of a document can be computed on the predicate answers ${\widehat{\mathcal{A}}}^d$ using an aggregation method such as linear weighted sum (LWS).

1.2.2. Background

There are multiple ways to use LLMs for ranking documents, rankers [37,43,44,45] and assessors are [46] popular choices. Rankers determine the order of items based on their perceived value while assessors provide an evaluation or judgement of the quality or suitability of a single item.

Zero-shot rankers [47], have no need for prior training or examples and can be categorised into point-wise [45], which scores one query and document at a time, ranking the documents based on a score; pair-wise, where the model assesses a pair of documents against a query or list-wise [44], which involves presenting both query and the complete list of items in the prompt. Pair-wise and list-wise methods do not scale for long lists as is the case with SLRs where the initial search could yield a substantial list of results. Wang et al. examined the performance of zero-shot and fine-tuned point-wise neural rankers in the context of SLR document ranking and discovered that zero-shot neural rankers performed worse than traditional methods like BM25 in the absence of fine-tuning. Moreover, research utilising point-wise rankers with generative LLMs indicated that obtaining ranking scores from log-likelihood values resulted in superior ranking performance relative to employing LLM generated labels[45]. The potential for additional output from an LLM, such as reasoning text, is constrained in this approach, as each output token, the smallest text unit a model can process, must be individually parsed, ultimately increasing complexity and limiting the interpretability of the ranker. Such constraints and the reliance on fine-tuning for improved performance highlights the need for approaches that balance scalability and transparency in zero-shot settings.

Zero-shot LLM assessor [46], perform relevancy judgement on a query-document pair when provided with a set of relevancy labels. When ground truth labels are available, these assessors are frequently employed to generate annotated datasets [48]. In addition to issuing judgement, LLM assessors can offer natural language explanations to support their conclusions, they provide scalability, consistency and the potential to complement human assessors in judgement tasks [46].

Several studies investigating the application of LLMs in SLR workflow have highlighted the importance for human oversight [19,26,40,42] or the use of LLMs as assistant reviewers [19,49]. Adopting an LLM as an assessor can effectively realise these strategies. While there is no ground truth data available at the start of a review, the assessor’s approach can still assign labels based on predefined eligibility criteria, establishing an initial framework to guide the ranking process and provide transparency. This study examines the effectiveness of an LLM as an assessor only, utilising its capacity to provide answer labels to questions.

Both ranker and assessor approaches can be enhanced using various strategies ranging from resource intensive options, such as pre-training and fine-tuning of models [37,50] to prompt engineering [51]. Domain-specific pre-training and fine-tuning have demonstrated substantial performance gains [27,37,52]. However, techniques such as few-shot and chain-of-thought (CoT) prompting, leveraging a persona and adopting fine-grained labels offer a more economical, less complex initial approach prior to pursuing advanced optimisations. While zero-shot prompting does not utilise task-specific examples, few-shot prompting incorporates both positive and negative examples, demonstrating promising results in title-abstract screening automation [21,50,53,54]. Considering that ranking occurs at the very start of the screening process, users typically lack domain-specific examples at this early stage, making few-shot prompts less practical for an initial ranking task. Chain-of-thought (CoT) prompting, instructs the model to adopt a grounded, "step-by-step" approach to task resolution, reducing the likelihood of hallucinations, improving performance and accuracy [31]. To increase transparency, this study captures the CoT reasoning so users can gain insights into the model reasoning. Additionally, integrating a persona can enhance a model’s capacity to deliver a more tailored and consistent responses across multiple interactions, adapting to a wide range of scenarios [40,55]. Finally, recent experiments with zero-shot LLM rankers indicate that fine-grained relevancy labels help guide the model to differentiate documents more effectively [45]. This study specifically examines the impact of fine-grained labels on the ranking performance using an LLM as an assessor within a zero-shot setting for prioritising primary studies.

1.3. Research Questions

This study aims to evaluate the effectiveness of using an LLM in the role of an assessor to assist in the ranking and initial selection of primary studies of climate-sensitive zoonotic diseases. To guide the investigation the following research questions are posed:

RQ1 

How does an LLM based assessor utilising a QA framework compared to baseline models utilising review title and selection criteria?

RQ2 

Does the label granularity effect the ranking performance of an LLM based assessor utilising a QA framework for climate sensitive zoonotic disease?

RQ3 

Does the ranking performance of an LLM based assessor generalise across climate sensitive zoonotic disease datasets with varying relevance rate?

RQ4 

Does CoT rationale provided by an LLM assist a human reviewer’s ability to detect misclassifications in SLR?

1.4. Contribution

This papers contributes to enhancing information retrieval techniques using LLMs, advancing the automating of SLRs and investigates generalisable and transparent methods for ranking and reviewing literature on climate sensitive zoonotic diseases. The specific contributions can be summarised as:

• Proposing a system to extract, store and share disease model parameters for climate sensitive zoonotic diseases modelling.
• Introducing and validating the use of an LLM-based assessor for ranking primary studies by utilising a QA framework.
• Evaluate the impact of label granularity on ranking performance to improve early stage recall quality
• Demonstrate that the proposed LLM-based assessor can manage highly skewed datasets and generalise across diverse climate sensitive zoonotic disease literature.
• Validate the utility of CoT generated reasoning text for human reviewers to enhance transparency and trust.

2. Methodology

A series of experiments were conducted to investigate the research questions outlined in Section 1.3. This sections provides details of the dataset, models, the evaluation metrics and the experimental design.

2.1. Dataset

The dataset used in this study is derived from a broader ongoing study examining 11 diseases (see appendix) along side 3 climatic variables, rainfall, humidity and temperature. Each combination of disease and climate variable constitutes a distinct SLR. An SLR protocol was established, outlining the selection criteria for each combination of disease and climate variable. A search was conducted in the PubMed and Scopus online journal repositories and the results were imported into a reference manager, along with titles and abstracts followed by de-duplication. A team of 5 researchers (early and mid-career researchers, postgraduate student and undergraduate student) were trained to screen abstracts using the criteria outlined in Table 1. Abstracts that met the inclusion criteria were assigned a score of 1, otherwise a score of 0. When a reviewer was unsure, abstracts were discussed with one of the mid-career researchers whom had experience in reviewing infectious diseases climate sensitivity. Abstracts that were unclear, or did not precisely meet the study selection criteria were scored 1 to be further examined in the full-text inspection stage.

An initial dataset of $2,905$ title-abstracts was established, consisting of the rainfall variable and 4 zoonotic diseases, Crimean-Congo haemorrhagic fever (CCHF), Ebola virus, Leptospirosis (Lepto) and Rift Valley fever virus (RVF). This constituted the entirety of the screened data at that point. Ebola articles exhibited a particularly pronounced skew, with only 1.5% (13/915) deemed relevant while the remaining diseases ranged from 10.8% to 12.6% (Table 2). The titles-abstracts were exported to a CSV file along with the manually assigned scores and labels were assigned per record to indicate the target disease and climate variable (only rainfall).

2.2. QA Framework

Following the abstract screening best practice guideline [35], a set of eligibility questions was developed to evaluate the relevance of title-abstracts based on the selection criteria. Distinct sets of eligibility questions were formulated for each disease and the associated climate variable (rainfall), ensuring their relevance to the SLR topic, as detailed in Table 3. It should be noted that reviewers did not use these questions during their screening, as they were developed retrospectively for the evaluation process. The guideline stipulate that the questions should be clear and concise, and must be 1) objective 2) "single-barrelled" or focused on a specific aspect of the citation 3) use a consistent sentence structure, and d) ensure responses are limited to yes, no and unsure only. Furthermore, the questions should be organised hierarchically, starting with the easiest and progressing to difficult. In this study, the question text and the number of questions assigned to each disease remained constant across experiments, while the answer labels were varied. The specific variations are detailed in Section 2.4.

2.3. Prompts

The design and choice of prompting has implications on the model response [49,56]. The prompts deployed in this study were inspired by those from previous research [19,21,31,40,49,57] and received iterative refinement. The prompt performance was evaluate using Recall@k and Mean Average Precision MAP metrics detailed in Section 2.7.

A persona description stating, "You are a world leading expert veterinary epidemiologist screening abstracts of scientific papers for the systematic literature review of ’$topic’" was established prior to the main instruction prompt. All experimental runs used the same persona, with the $topic placeholder dynamically replaced by the topics listed in Table 3.

Two main instruction prompts were designed for this study. The first prompt, TSC prompt, serves as a baseline, utilising the review title and selection criteria. This prompt is an adaptation of the Zero-shot Framework CoT prompt [21], directing the model to analyse the title and apply inclusion and exclusion criteria to classify a given abstract as "Definitely Include," "Probably Include," "Probably Exclude," "Definitely Exclude," or "Unsure" (see listing 1). The placeholders $title, $inclusion, $exclusion, and $abstract were populated dynamically according to the disease context of the experiment.

The second prompt, QA prompt in Listing 3 (see Appendix A), utilises the QA framework discussed in Section 2.2. This prompt instructs the model to follow a structured approach to answering the eligibility questions with respect to a given abstract. The answer labels available to the model are discussed in Section 2.4. Similar to the first prompt, placeholders for $abstract and $question, are dynamically populated with title-abstract and eligibility questions based on disease context.

Both prompts employ predefined answer labels and explicit reasoning, adhering to the principles of CoT prompting [31], and aligns with the instructive template category, as described in [54]. Given that GPT models are autoregressive [50] where previous tokens influence the final answer, the prompts have been designed to explicitly request reasoning prior to delivering a final answer label. Finally, the prompts instructs the model to format the response as a JSON object, fields capture the reasoning text, answer and in the case of the QA framework, the question number. This structure helps ensure consistent and reliable output formatting and eliminates the need for parsing natural text while facilitating easy integration with downstream components.

2.4. Answer Labels

Table 4 lists the predefined answer schemas designed to be used with the QA and TSC prompt, restricting the model to selecting only from these options. The schema, QA-3 was adapted from the best practice guidelines [35], while QA-4 and Qa-5 were inspired by [45].

Additionally, a variant of QA-2 schema, termed QA-2-C, a multi level answer schema was explored by instructing the model to generate a separate confidence score (Table 5). This method aims to more accurately capture the uncertainty that the original QA-2 fails to account for, yet appears in the other schemas. Each answer label was assigned a predefined answer score to quantify its contribution to the relevancy ranking calculation (described in Section 2.5), in the case of QA-2-C the confidence score was used in place of the answer score. The QA prompt was adjusted to accommodate the multi level scoring as shown in Listing 4 (see Appendix A).

2.5. Relevancy ranking

A relevancy score was computed for each title-abstract record by employing a Linear Weighted Sum (LWS) of the answer scores derived from all questions within the QA framework. The records are subsequently sorted in descending order based on the relevancy score. In this study, it was assumed that all questions carried equal weight. Formally, let:

• $\mathcal{Q}=\{q_1,\cdots ,q_{\left|C\right|}\}$ be the eligibility questions derived from the selection criteria.
• ${\widehat{\mathcal{A}}}^d=\left\{{\widehat{a}}_k^d\right\}$ be the set of predicate answers for each question $q_k$ in document d, and
• $w_k$ be a predefined weight reflecting the importance of each question $q_k$.

Then, the relevancy score ${\widehat{r}}_d$ of document d is expressed as:

$${\widehat{r}}_d=\sum _{k=1}^{\left|C\right|}{w}_q{\widehat{a}}_k^d$$

(1)

where for equal weighted questions $w_k=\frac{1}{k}$ for all q. This naturally simplifies to:

$${\widehat{r}}_d={\displaystyle \frac{1}{k}}\sum _{k=1}^{\left|C\right|}{\widehat{a}}_k^d$$

(2)

In cases where ties occur in the relevancy score, title-abstract length was used as a tiebreaker by sorting the title-abstracts in descending order of length, assuming longer articles were more relevant.

2.6. Models

2.6.1. Baseline Models

The BM25 [58] and MiniLM v2 [59] models were utilised to establish a zero-shot baseline. BM25 is a widely used ranking algorithm used in information retrieval and evaluates a document’s relevance with respect to a query. The query in this instance is a concatenation of the review title with the selection criteria. BM25 scores were computed for each title-abstract and query pair following a preprocessing step that included lower casing, punctuation elimination, stop word removal and stemming. The results were ranked according to the BM25 scores, with the text length serving as a tie breaker.

MiniLM v2 is a pre-trained sentence-transformer model optimised for embedding text and trained on a massive and diverse dataset [60]. This study used the all-MiniLM-L6-v2 model without fine-tuning. Embedding vectors were generated for both title-abstracts and queries. Cosine similarity was computed as per equation 1, for each vector pair and subsequently normalised and sorted, with the text length serving as a tie breaker.

$$\mathit{cosine}\_\mathit{similarity}(d,q)={\displaystyle \frac{d·q}{\parallel d\parallel \parallel q\parallel }}$$

(3)

This methodology for establishing a baseline is aligned with the approach implemented by CSMed, a meta-dataset comprising 325 systematic literature reviews from the medical and computer scientific fields [39].

2.6.2. Large Language Models

For this experiment GPT-4o-mini-2024-07-18, a smaller and optimised version of GPT-4, is utilized to achieve a balance between cost, performance and computational efficiency [61]. The model was used in it’s pretrained form without any fine-tuning and accessed via the OpenAI API. The sampling temperature value was set to 0 to improve reproducibility and ensure deterministic responses between invocations but also to ensure that it does not affect the model’s ability to self-correct, liuLargeLanguageModels2024. The max_token parameter was set to 512 to provide sufficient context for processing the prompts and generate concise reasoning responses. The response_format parameter was used to ensure consistent and reliable output formatting. A JSON Schema is assigned to this parameter which describes the structure of the expected output, preventing the need for parsing of natural text and facilitates integration with downstream code.

2.7. Evaluation Metrics

The performance of ranking tasks in information retrieval challenges is evaluated using rank-based metrics and metrics at predefined cut-offs, such as the $\mathit{top}-\mathit{k}\%$ of retrieved documents [39,62]. The evaluation metrics used to measure the performance of the approach are listed below:

Recall @ k%

Recall also known as sensitivity provides a measure of how many relevant documents are retrieved within the top-k% of ranked results.

nWSS @ r%

Normalized work saved over sampling, equivalent to True Negative Rate (or Specificity) quantifies the effort or work saved by the automated system when compared to random sampling, assuming a fixed recall level, and can be utilized to evaluate outcomes across models and datasets [63]. Here nWSS% is evaluated at $\mathit{r}=95\%$ and $\mathit{r}=100\%$.

Average Precision (AP)

Average Precision (AP) represents the average of precision computed at each relevant document’s position, considering all documents retrieved up to a specific rank. It combines both precision and recall, evaluating how effective documents are ranked by relevance. Unlike precision @ k% which is influenced by the total number the number of relevant documents, AP addresses this limitation by providing a more balanced assessment [62].

Mean Average Precision (MAP)

MAP represents the average AP across individual information needs and provides a single robust metric for evaluating the ranking quality across recall levels [62].

2.8. Experimental Setup

The experiments were automated using Python 3.9.21, with each experimental run organised into separate Python scripts. The baseline experiments utilised title and selection criteria, abbreviated as TSC in conjunctions with BM25, MiniLM and ChatGPT-4o-mini models. Meanwhile, the QA framework, was exclusively tested with ChatGPT-4o-mini model using all QA answer schemas list in Table 4 and Table 5.

Each execution script requires two input files: a CSV of title-abstracts and a prompt template file which gets dynamically populated with experimental context specific values for each zoonotic disease, CCHF, Ebola, Lepto, and RVF. The code then calls the OpenAI API completion endpoint using predefined model parameters and the adapted prompts. By utilizing the API, the study ensures reproducibility. All responses from the API are captured and subsequently processed to calculate a relevancy ranking.

3. Results

This section presents the results of the eight models evaluated across the four diseases: CCHF, Ebola, Lepto, and RVF evaluated against the study metric (Table 8). The following sections will examine results for each research questions.

3.1. RQ1. LLM Based QA Assessor vs Baseline

The LLM-based QA assessor models consistently outperform the TSC baseline models across all metrics, achieving higher recall, greater work savings, improved precision and reduced variability, holding true whether the baseline is generative (TSC-5) or non-generative (TSC-BM25, TSC-MiniLM). Figure 2, illustrates that the QA approaches exhibits superior early retrieval performance (Recall@5% - Recall@20%) compared to the baseline models, identifying at least 76% of relevant articles within the top 20% of the dataset, regardless of disease. The high nWSS@r% values show (Figure 3) that QA models are also better at detecting irrelevant documents early in the screening process, thereby reducing the number of items requiring manual review. The leading QA models, QA-4 and QA-5, demonstrate a minimum savings of 67% at r=95% across all diseases; however, performance declines when aiming for complete recall. In contrast, the TSC models display lower overall work savings, with particularly high variability in the LLM-based TSC model: it ranges from negative savings with Ebola to 46% nWSS@95% with Lepto.

The AP metric evaluates ranking quality by measuring the number of relevant documents and how early they appear, further confirms QA models consistently high scores across all disease datasets (Figure 4). Except for Lepto, both TSC-5 and TSC-MiniLM fail to demonstrate robust ranking quality for the other diseases. In addition, the QA models exhibit less variability and more consistency across the datasets as seen in Figure 5. This box plot shows that QA models have narrower inter-quartile ranges (IQR) and shorter whiskers while the baseline models display higher medians and wider variability in rank positions.

Finally, the MAP metric offers a single performance measure for each model. Table 6 indicates QA models clearly outperform the baseline models with QA-4 attaining the highest MAP score of 0.691. A Precision-Recall Curve (Figure 6), visualises overall performance by illustrating the effect of varying thresholds on the precision-recall balance. The QA models dominate the baseline models, and the LLM-based TSC-5 model notably outperforms TSC-BM25 and TSC-MiniLM, though it still lags behind QA-4 and QA-5.

3.2. RQ2. Effect of Label Granularity

The results indicate that label granularity strongly affects the ranking performance of an LLM-based assessor using a QA framework. Overall, performance generally improves with granularity QA-2 to QA-4, with diminishing returns observed from QA-4 to QA-5. As shown in Figure 2, single-level fine-grained labels (QA-4, QA-5) exhibit high recall at early ranks ($\mathit{k}\le 20\%$), retrieving a substantial proportion of relevant articles. By ($\mathit{k}=50\%$), they reach near complete recall ($\ge 0.98$) across all disease datasets.

QA-2-C’s multi-level approach of presenting granularity by pairing binary labels with confidence scores demonstrates a middle ground between fine-grained (QA-4, QA-5) and coarse-grained (QA-2, QA-3) labels. QA-2-c lags behind QA-4, and QA-5 but all demonstrate significant work savings at 95% recall target over coarse-grained labels, achieving a minimum of 61%, 70% and 67% respectively. In particular, QA-4 and QA-5 perform notably well for highly skewed datasets such as Ebola, realising 83% to 88% savings at recall levels of 95% and 100%. Finally, the MAP scores in Table 6 demonstrate a clear trend indicating improved ranking performance with increased label granularity peaking at QA-4 (0.691).

3.3. RQ3. Performance Across Zoonotic Diseases

The LLM-based QA assessor exhibit robust generalisable performance across climate sensitive zoonotice disease datasets, despite significant variation in relevancy rates. The Ebola dataset reveals a significant skew with only 1.5% of relevant records, whereas the CCHF, Lepto, and RVF datasets demonstrate moderate skews, as outlined in Table 2. This variation is apparent when evaluating model performance across the different diseases. Yet, QA based models, specifically QA-4 and QA-5, retained higher recall@k%, nWSS@95% and AP scores, while maintaining consistent ranking across the datasets in comparison to the baseline models.

In the Ebola dataset, QA-4 and QA-5 models demonstrate a significantly high recall, achieving complete recall at $\mathit{k}=15\%$ (Figure 2), suggesting that all relevant title-abstracts were identified within the top $15\%$ of the ranked list. The baseline models struggle to achieve a comparable level of recall, with BM25 exhibiting a consistent but significantly lower recall across all diseases. In contrast, TSC-5 and TSC-MiniLM perform well with Lepto but severely unperformed with other diseases.

For all diseases, the QA models retain a minimum of 54% work saving at $\mathit{nWSS}@\mathit{95}\%$; however, performance drops when transitioning to nWSS@100% for all disease except Ebola, where savings slightly increase. Notably, TSC-MiniLM achieves the same work savings as QA-4 and QA-2-C nWSS@95% = 0.72); however, when striving for full recall, this value reduces dramatically proving high variability.

A consistently high AP across diseases indicates models that can handle disease-specific terminology and generalise well independent of relevancy rates. The leading QA models (QA-4, QA-5) achieve a minimum of 0.5 AP across all diseases, demonstrating their capability to manage variations in relevance rates and address disease-specific terminology. The baseline models exhibit lower average precision (AP) and greater variability in AP, suggesting limited robustness to skewed datasets and less capacity for handling disease-specific terminology.

QA models achieved the best rank distribution with the Ebola dataset (Figure 5), demonstrated by the lower median rank, tighter IQR and few outliers. Similar patterns were seen in Lepto, CCHF, and RVF, all of which suggest that QA models are superior to baseline models at prioritising relevant articles in the earlier ranks. Nevertheless, the Lepto dataset has the greatest number of outliers, which may be a sign of ambiguous abstracts, misaligned classification criteria, or model limitations.

Listing 2. Example JSON response for an RVF abstract

Listing 2. Example JSON response for an RVF abstract

3.4. RQ4. Utility of Generated CoT Rationale

The CoT reasoning logic proved to be consistent, in line with human expectations, helpful in detecting edge cases and capable of revealing variations in misclassifications across disease datasets. Listing Listing 2 is an example RVF response from the QA-4 model that demonstrates the logic the model provides for each title-abstract record. A senior reviewer on the team was presented with the results to manually assess the validity of the responses. For each disease, title-abstracts with relevancy scores exceeding a predetermined threshold and classified as irrelevant, as well as those below the threshold classified as relevant, were displayed in MS Excel alongside the answer scores and reasoning text for each question. The reviewer did report any significant issues in the model’s reasoning and considered the explanatory text highly beneficial for comprehending the logic underlying the answer scores. Using the sort and filter facilities provided in MS Excel the reviewer was able to efficiently review and identify several misclassified items as shown in Table 7. Lepto (16), RVF (13) and CCHF(10) had the most misclassifications, highlighting the inherent variability in manual SLR screening quality but also opportunities to refine the selection criteria and eligibility questions. These findings not only highlight the potential of CoT reasoning to ensure transparency in the SLR process but also a means to iteratively enhance the ranking process.

4. Discussion

This paper presented the conceptual architecture of the CliZod system and contextualised the role of the relevancy ranking component in the broader system. The subsequent empirical findings illustrated that a zero-shot LLM-based QA assessor, leveraging fine-grained labels, can reliably rank primary studies by relevance. The model demonstrated broad generalisability across four climate-sensitive zoonotice disease datasets with varying relevancy rates. Additionally the CoT reasoning text generated by this approach provides valuable insight to human reviewers and aided in identification of misclassified records in the disease datasets.

Before further interpretation of the results, the following important limitations are note worthy. Most importantly, the study depended on a single reviewer for the manual evaluation of the model’s reasoning text. No metrics were established to measure the quality or potential bias of the text generated by the LLM, which are prone to "hallucination", jiMitigatingHallucinationLarge2023, zackAssessingPotentialGPT42024. Nevertheless, the results from the study demonstrated a potential use case for such data and the possibility for creating a unique dataset to enable the team to further research into reasoning and bias within LLM-based approaches.

This study used a single closed-source commercial LLM from OpenAI to conduct all experiments. Restricting the evaluation soley to ChatGPT-4-o-mini risks the generalisation to other LLMs. Additionally, while a fixed model version was used with the temperature parameter set to 0 to ensure deterministic behaviour, and prior research confirming consistent outcomes on repeated invocation [49], ongoing optimisations by OpenAI may have led to performance changes that could have influenced the results. In the future, the team intends to explore additional models, including open-source alternatives applying the same methodology.

Moreover, the eligibility questions used in the experiments were generated in retrospect following manual screening, hence there is a risk of over fitting and misalignment with broader applicability. Polanin et al. advocates for conducting a pilot screening session with a subset of abstracts to refine and validate eligibility questions prior to screening. This approach could ensure clarity and consistency of the questions while improving the effectiveness of the rankings. Such an exercise may prove beneficial, particularly for the remaining unscreened disease and climate variable articles as it could mitigate bias and facilitate the development of a more comprehensive dataset. Additionally, the collection of ground truth data at the question level would facilitate a more precise and detailed evaluation of the model and support further research.

Lastly, although the real-world setting of this study’s dataset demonstrated the effectiveness of the approach, it has not been thoroughly evaluated on datasets from other domains. CSMed is an initiative to create a standardised dataset to evaluate the performance of automated SLR screening models [39], a gap highlighted in recent research [16]. It provides access to 325 SLRs in the medical and computer science domain and presents an opportunity to benchmark the approach proposed in this study and gain a broader understanding of its performance, generalisability and adaptability across domains.

Despite the limitations, this study demonstrates the effectiveness of employing a QA framework with an LLM-based assessor to robustly rank literature on four climate-sensitive zoonotic diseases by relevance. The QA models consistently outperformed all the baseline models, achieving high Recall@k% and MAP scores across all disease datasets in this study. High recall is crucial in SLR automation systems [64] to prevent bias and ensure all relevant articles are identified. Additionally a high MAP score reflects strong discrimination and resilience to imbalanced datasets [62], ensuring reviewers are presented with both manageable and highly relevant articles.

While the study results are promising, it is important to contextualise the system performance. Direct comparisons to other systems must be made with caution due to the challengingly unique climate-sensitive zoonotic disease dataset employed in this study. However, the baseline TSC-BM25 model provides a simple yet informative reference point. In the study by Wang et al. examining the performance of neural rankers on the CLEF [65] datasets, a baseline BM25 model recorded a Recall@20% ranging from 0.52 to 0.64, alongside a MAP score of 0.16 using title as the input query to the model. By comparison, TSC-BM25 in this study recorded a Recall@20% ranging from 0.37 to 0.57 and a slightly higher MAP score of 0.23 using title and selection criteria as the input query. Their most effective fine-tuned BioBERT neural ranker demonstrated a Recall@20% ranging from 0.82 to 0.89, and a MAP score of 0.381. In contrast, the QA-4 model in this study attained a Recall@20% ranging from 0.83 to 1.0, and a significantly higher MAP score of 0.670. Although the highly skewed Ebola dataset may have contributed to the elevated recall score, the findings highlight the potential of the zero-shot LLM-based QA framework for enhancing ranking performance without the need for fine-tuning. However, any conclusions should be deferred until a comprehensive evaluation with a standardised benchmarking dataset is conducted.

The adoption of a QA framework has shown to be highly effective in the current study, contrary to the findings of prior research, such as Kohandel Gargari et al. where a screening tool approach yield poor performance. Upon examining their methodology, several factors may have contributed to the discrepancy, including the complexity of the prompt, ambiguity in managing ’unclear’ labels, and the assumption that the model will adhere to the embedded logic flow in the prompt without errors. Additionally, their research employed GPT 3.5, an older model, less advanced and less accurate. In contrast, the findings from Akinseloyin et al. align more closely with this study, utilising a similar QA framework methodology. However, their eligibility questions were derived from the inclusion/exclusion criteria using an LLM and observed that the generated questions lacked complete independence and recommended that such questions be created by humans for better reliability, a recommendation followed in this study.

In addition to the QA framework, the use of fine-grained labels appears to have a consistent positive impact on ranking tasks whether in an assessor or a ranker context. The single-level fine-grained models, QA-4 and QA-5 demonstrated superior recall, precision and work savings, with minimal variability, compared to the coarser-grained models, QA-2 and QA-3. Fine-grained labels appear to enhance the system performance by supplying the model with broader answer options, allowing to more effectively represent uncertainty. However, the benefits diminished as granularity increased beyond four levels as evident by the reduced MAP score for QA-5. Zhuang et al. found that fine-grained relevancy labels improved performance, with no advantage in exceeding four levels of granularity using a point-wise LLM-based ranker. While the tasks in these studies differ (assessor vs ranker), this suggests that fine-grained labels may be broadly beneficial for LLM-based ranking tasks.

Interestingly, the shift from a single-level to a multi-level labelling approach (QA-2-c) demonstrated reduced performance. Several factors could account for this outcome. Firstly, the multi-level model employs a more intricate prompt, simple prompt structures are more effective than complex ones [67]. Furthermore, a single level classification inherently captures both the relevance and uncertainty in one consolidated label (e.g. "Probably Yes") where as a two step classification introduces uncertainty post-decision, thereby splitting the context. Finally, the model’s autoregressive characteristic [50], combined with single versus multiple decision-making points, may account for the observed performance differences.

The study approach also shows promising generalisability across disease datasets, despite their varied characteristics. Nicholson Thomas et al. used a selection criteria based prompt, similar to TSC-5 baseline model, to screen articles for ecosystem condition indicators. They report that to handle multidimensional topics with high precision, iterative refinement of the selection criteria was essential. While further enhancements of the selection criteria could enhance the performance of the TSC-5 model for diseases where it underperformed (other than Lepto), the QA framework offers an additional layer of flexibility. It decouples the decision making from the LLM’s internal logic by decomposing the assessment into granular questions. Weak signals in the form of uncertainty labels (e.g. "Probably No" or "Unsure"), allow studies that marginally fail to still contribute to overall ranking through the LWS. This flexibility ensures relevant studies are not prematurely excluded and provides adaptability to the varied disease datasets.

Complementing the flexibility of the QA framework is the utility of reasoning text generated through chain-of-thought prompting, which provides a detailed explanation for each relevancy assessment, exposing the models’ decision making process to the reviewers. For example, the reasoning text in Listing Listing 2 illustrates the model’s ability to justifying uncertain responses like "Probably No" in question 2, where it explains that although the study addresses disease vectors, it does not directly assess virus prevalence. On the spectrum of human-machine collaboration, this approach of automatically generating a first-pass judgement with rationale falls into the human verification category, or human-in-the-loop approach [46]. While chain-of-thought prompting does not ensure the accuracy of reasoning path, it does enable the model to more effectively access relevant data learnt during pre-training [31], which may explain the QA models’ performance over the baseline models. Further investigation is required to assess the validity and reliability of the generated rationale.

It is recommended that tool developers test their products across a diverse range of domains, as the effectiveness of prompts is influenced by the characteristics of the dataset [49]. The dataset being curated as part of the CliZod project has the potential to provide 33 SLRs for tool developers, focusing on research in the areas intersecting climate change and veterinary epidemiology. This unique combination of domains provides additional benchmark for evaluating the efficacy of automated screening systems in handling multidisciplinary research questions.

Finally, the approach proposed in this study can currently be used as a standalone decision aid to reduce the risk of human error and bias during the initial screening phase. The main prompt provided in the supplementary code is generic and applicable across domains. All that is needed is to formulate research-specific eligibility questions, define the review topic, and modify the persona text accordingly. Data is output as a CSV file containing ranking scores, answers, and reasoning text. It can be reviewed in a tool such as MS Excel providing an additional layer of adaptability. By prioritising and streamlining the review process, this approach allows human reviewers to focus on the edge cases, making it a practical and efficient solution for SLRs.

5. Conclusions

This study introduces CliZod, a novel automated framework for extracting modelling parameters from scientific literature to support climate-sensitive zoonotic disease modelling. It reports the empirical results of the relevancy ranking approach, leveraging an LLM as an assessor and guided by a QA framework to rank literature on four climate-sensitive zoonotic diseases and one climate variable (rainfall).

The findings of this study demonstrate that an LLM-based QA assessor using zero-shot CoT prompting can effectively and reliably rank primary studies by relevance across 4 zoonotic disease datasets with varying relevance rates. Furthermore, fine-grained QA models significantly outperform the baseline models, achieving strong recall at multiple thresholds, high MAP scores and substantial work savings of at least 70% (NWSS@95%). Additionally, the CoT reasoning text generated by this approach provides valuable insight, enhancing and aiding human reviewers in their decision making process.

Although further empirical research is necessary to validate the approach against standardised benchmark dataset, the substantial reduction in screening effort, combined with the provision of explainable AI rationales, represents an important step toward automated parameter extraction from scientific literature.