Expert Knowledge Elicitation of Prior Information for Use in Acceptance Sampling and Conformity Analysis

1. Introduction

Expert Knowledge Elicitation (EKE) is a structured process that can be used to marshal the insights, judgements, and knowledge from a group of experts concerning uncertain facts or values that might arise from a specific field or domain. It is particularly useful in situations where empirical data concerning facts or values of interest is scarce, incomplete, or only available for related cases, and where expert opinion can serve as a critical source of information. EKE is used in fields such as risk assessment, engineering, decision-making, pharmaceuticals, healthcare, environmental science, and policy development, where the subject matter experts' knowledge is essential for modelling, estimating, or forecasting uncertain quantities.

The process involves collecting quantitative information from experts, using formal methods designed to minimize those systematic responses to judgement and decision problems known as cognitive biases (Tversky & Kahneman, 1974), and expressing the uncertainty around this information as a probability distribution. These methods often include structured interviews, remote surveys and/ or group discussions. They are often iterative, involving one or more rounds of value elicitation, feedback, and value adjustment. The goal of EKE is to systematically capture expert judgements, ensuring transparency, reproducibility, and reliability in the decision-making processes.

Effective knowledge elicitation requires careful planning, including the selection of experts, the design of elicitation protocols, and the implementation of procedures that mitigate cognitive biases. By aggregating expert insights EKE provides decision-makers with informed perspectives, particularly when dealing with uncertainty and complex problems.

There are three main EKE protocols in use today, SHELF (Oakley & O'Hagan, The Sheffield Elicitation Framework, 2019), the IDEA (Investigate, Discuss, Estimate and Aggregate) protocol (Hemming, Burgman, Hanea, McBride, & Wintle, 2018) and the Classical Method (Colson & Cooke, 2018). These protocols differ in the following ways

• the number of experts that can be involved and the number of values that can be elicited at one time.
• the use of group discussions and the handling of group dynamics
• the calibration and/or the weighting of the responses from the experts
• the handling of cognitive biases
• the methods used to aggregate and summarize individual responses from the group.

2. The Sheffield Elicitation Framework

2.1. Structured Framework

SHELF is a highly structured and systematic approach which provides detailed guidelines for facilitating expert elicitation sessions. It includes predefined protocols, templates, and tools to train experts to quantify their uncertainty, to elicit their knowledge about the subject and to aggregate the information they provide as a group. These tools help to ensure consistency and rigor in the elicitation process.

2.2. A Group-Based Approach

Although SHELF initially focuses on eliciting individual expert opinions, the final emphasis is on group-based elicitation. A group of between 2 or more experts is convened, usually in a face-to-face workshop environment, trained in the protocol, and then briefed about the subject of the elicitation. Their individual judgements are collected, discussed within the group, adjusted in the light of the discussion, and finally combined using “Behavioural” aggregation, it is the use of this aggregation method which limits the size of the expert group. A consensus distribution is reached by the experts considering what conclusions an independent observer of their discussions might reach. The entire process must be planned thoroughly in advance and the group interactions carefully moderated by the facilitator to ensure a balanced discussion, minimize cognitive bias and to avoid dominance by any individual expert. But it may also be implemented as a solo activity when a decision maker seeks to quantify their own knowledge about an uncertain quantity (O'Hagan, 2019).

2.3. Training Experts in Quantifying Their Uncertainty

SHELF emphasizes the training of experts in uncertainty quantification, recognizing that many experts, while knowledgeable in their domains, may not be familiar with expressing or quantifying their judgements probabilistically or dealing rigorously with uncertainty. The training involves eliciting values for one or more unrelated subject areas, for which experts may have no more knowledge than any other member of the public, e.g. the number of medals that will be won by different countries in future international events or the proportion of uninsured drivers in different American states in different time periods. Different methods of quantifying their uncertainty about these estimates are presented by the facilitator and practiced using these examples. It is important that this training is done as a group immediately prior to the discussion and elicitation for the actual subject matter of interest, irrespective of whether the experts have the viewed the pre-elicitation training material provided by the SHELF protocol.

2.4. Eliciting Probability Distributions

Experts are asked to estimate uncertain quantities by specifying key points of a probability distribution by starting initially with defining the widest feasible range and then narrowing down to the upper and lower quartiles (e.g., 25^th and 75^th percentiles) before finally considering the median (50^th percentile) value. The order these values are elicited are important in reducing cognitive biases such as Anchoring and Over-confidence. These estimates are then used to construct a distribution that represents the Rational Impartial Observer’s (RIO) uncertainty about the quantity in question.

2.5. Facilitator Role

A key feature of the SHELF method is the role of the facilitator. The facilitator plans and guides the process, identifying and selecting experts with relevant knowledge, providing a common set of background material for them and ensuring that they understand the task and adhere to the structured methodology. The facilitator also manages group discussions, encourages the sharing of reasoning behind judgements, and ensures that cognitive biases (such as overconfidence or anchoring) are minimized. The facilitator’s assistant will record the discussions and operate the software necessary for fitting probability distributions to elicited values.

2.6. Steps in the SHELF Method:

2.6.1. Preparation and Briefing

The facilitator prepares the elicitation session, including defining the problem, selecting experts, and providing them with the necessary background information.

Experts are briefed on the elicitation process and the specific uncertainties they are being asked to quantify.

2.6.2. Training

This includes calibration exercises to help experts better understand how to express their judgements in probabilistic terms. The training seeks to minimize common cognitive biases, such as anchoring and overconfidence.

2.6.3. Eliciting Individual Judgements

Experts are first asked to make individual estimates about the uncertain quantity, they provide estimates for key points in a probability distribution, the upper and lower feasible bounds, upper and lower quartiles, and a median value. This allows the facilitator to capture each expert’s individual perspective before group discussion begins.

2.6.4. Group Discussion

Once individual judgements have been collected, the experts engage in a structured discussion. The facilitator encourages experts to explain the reasoning behind their estimates, highlight any uncertainties, and address any discrepancies between their estimates. The aim is to foster an open exchange of views while avoiding groupthink or undue influence from dominant personalities. Group discussions allow experts to learn from each other and refine their estimates through collective reasoning. This often leads to a more robust and balanced judgement than individual elicitation methods.

2.6.5. Consensus Building

After the discussion, the group works towards achieving a consensus estimate. This involves revisiting the individual estimates in light of the group discussion and, if possible, agreeing on a set of percentiles or a full probability distribution that reflects the collective judgement of the group.

The facilitator plays a crucial role in guiding the consensus process, ensuring that all viewpoints are considered, and that the group’s final judgement reflects the full range of uncertainties discussed.

2.6.6. Fitting a Distribution

Once the key percentiles or points of the distribution are agreed upon, a probability distribution is fitted to those points using statistical techniques. SHELF provides templates and software tools to help facilitators and experts fit distributions such as normal, log-normal, or other distributions that best represent the experts' estimates.

2.6.7. Documentation and Transparency

SHELF places a strong emphasis on documenting the entire elicitation process. This includes recording the individual estimates, the group discussion, and the rationale behind the final consensus. Although Individual comments are usually anonymized in the documentation this record still helps the decision-makers and external stakeholders to understand how the expert judgements were arrived at and provides a clear audit trail for future reference.

3. IDEA Protocol

The IDEA protocol is a structured and systematic approach to Expert Knowledge Elicitation (EKE), designed to gather expert judgements in a transparent and reproducible manner. IDEA stands for Investigate, Discuss, Estimate, and Aggregate—four key steps that guide the elicitation process.

3.1. A Group-Based Approach

The IDEA method focuses entirely on eliciting individual expert opinions, with the emphasis on mathematically aggregating their final individual responses. The protocol suggests that the best regarded experts in a domain seldom perform as well as expected in group elicitation tasks, so peer recommendations are not used in selecting the group the coordinator instead seeks to assemble a group with as diverse range of experience and knowledge of the subject area as possible. Expert groups of 5 or less can be subject to outliers so the protocol suggests that a group of between 6 and 12 experts is asked, at the first stage, to investigate the subject matter of interest and provide their responses to a pre-prepared set of questions posed by the facilitator and problem owner. These responses are collected and an analyst, after first cleaning and standardizing as necessary (e.g. adjusting for consistent units), compares them to the other responses and their aggregates from the rest of the group and provides this comparison as feedback to the individuals. This first stage often takes place remotely and without sharing the names of the other group members with the individual experts.

3.2. Key Steps in the IDEA Method:

3.2.1. Investigate

In this phase the facilitator presents the experts with a clear and well-defined problem, the available background material and a set of elicitation questions. Such questions would usually conform to the following four-step format:

• Question:
  • 'What will be the average batch nonconformance rate (X) to produce items ‘A’ by manufacturer ‘B’ given that the process is being transferred to a new factory in location C in 2026?'

• 4-step elicitation:
  • Realistically, what do you think the lowest plausible value for the nonconformance rate, X, will be?
  • Realistically, what do you think the highest plausible value for nonconformance rate, X, will be?
  • Realistically, what is your best guess for nonconformance rate, X?
  • How confident are you that your interval, from lowest to highest, could capture the true value of the nonconformance rate, X? Please enter a number between 50% and 100%

The investigation phase ensures that all experts fully understand the problem, including its scope, assumptions, and context. The shared background information and data help frame the issue, allowing experts to independently form their initial judgements. This first stage helps reduce groupthink and other biases caused by any early interaction among experts.

3.2.2. Discuss

After the experts have individually investigated the problem, they engage in a structured discussion, usually moderated by the facilitator. The purpose of this phase is to allow experts to share their reasoning, highlight their key assumptions, and discuss any discrepancies in their initial estimates when compared to the group. The discussion is aimed at improving understanding, without forcing consensus. It also helps experts learn from each other and refine their perspectives.

3.2.3. Estimate

Experts are then asked to provide a second round of estimates for the elicitation questions that take account of their discussions. Importantly, the experts still work independently when formulating their estimates to avoid conformity pressure. By eliciting multiple estimates, the IDEA method ensures a diversity of opinions and insights.

3.2.4. Aggregate

The final step involves the analysts aggregating the experts' estimates into a collective judgement. Using a simple mathematical aggregation method such as the mean or median values of the individual responses. The aggregation process provides a synthesized view of expert opinions, presenting the problem-owner or decision-maker with a single value and an indication of the uncertainty around it as provided by the group.

3.3. Documentation and Transparency

IDEA places a strong emphasis on documenting the entire elicitation process. This includes recording the initial and subsequent individual estimates, the individual's performance relative to the group, the discussion of their assumptions and reasoning, and the final mathematical aggregations. This record helps the decision-makers to understand how the aggregated estimates and their associated levels of uncertainty were arrived at and provides a clear audit trail for future reference and reproducibility.

4. The Classical Method

The Classical Method of Expert Knowledge Elicitation (EKE), also known as the Cooke's Method, is a formal structured approach used to aggregate individual expert judgements, particularly in contexts where empirical data is scarce, missing or only available for related cases. The Classical Method focuses on quantifying and weighting expert opinions based on their individual performance, specifically by generating calibration and information scores for each expert. These scores are then used to weight the estimates they give in response to the questions on the subject of the elicitation. The Classical Method minimizes common cognitive biases such as overconfidence and anchoring by only using experts’ independent private estimates (there are no group discussions) and by focusing on performance-based weighting rather than relying on consensus-building approaches. The Classical method primarily seeks to make expert elicitation as objective as possible by combining expert judgements in a way that emphasizes the reliability and accuracy of individual experts' performance.

4.1. Principles of the Classical Method:

4.1.1. Calibration

Calibration refers to how well an expert's probabilistic estimates align with actual outcomes. In the Classical Method, experts are asked to make predictions on a series of calibration questions—questions for which the true outcomes are known but the experts do not have prior knowledge of them. An expert is considered well-calibrated if, for example, the credible intervals they provide in response to the calibration questions more frequently include the true values. The Calibration score measures the statistical consistency between an expert’s predicted probabilities and the observed outcomes. Experts with better calibration scores are considered more reliable in their estimates of unknown quantities.

4.1.2. Information Score

The information score measures the informativeness or precision of an expert’s estimates. An expert who provides narrow probability distributions (i.e., estimates with high confidence) would score higher in information than an expert who provides estimates with wider more uncertain distributions. However, this is only valuable if the expert is also well-calibrated—overconfident estimates that are inaccurate are penalized.

4.1.3. Scoring Rules

The Classical Method uses performance-based weighting, meaning that expert judgements are weighted according to how well each expert has performed on the calibration questions. Experts are evaluated using both their calibration and information scores. These two metrics are combined into a weighted score that balances the accuracy (calibration) of the expert’s estimates with their informativeness (how precise or uncertain their estimates are). Experts who are both well-calibrated and informative are given greater weight in the aggregation of the final estimates.

4.2. Key Steps in the Classical Method:

4.2.1. Preparation

The facilitator, coordinator and problem owner select as diverse a group of experts as possible, based on their relevant domain knowledge. They must have a deep understanding of the subject matter related to the elicitation problem, if possible, they should be a mix of generalists and specialists and may also come from different fields. As the Classical method is carried out remotely by a questionnaire, without group interactions or multiple rounds, there is no limit on the number of experts that can be included, and a large number of questions can also be included.

A set of questions is prepared for a questionnaire that includes a mixture of both seed and target questions. Seed questions are those with known answers (known to the facilitator/ problem owner and but not to the experts), these will be used to assess the experts' performance. They should be similar in nature to the uncertain quantities of interest, having well-established answers from historical data or scientific evidence. Target questions are the main focus of the elicitation and concern the uncertainties or estimates that require expert judgement. But finding a set of calibration questions where the answers are unknown to the domain experts can be challenging.

4.2.2. Elicitation

The questionnaire should ask the experts to provide their individual probabilistic estimates (e.g., 5th, 50th, and 95th percentiles) for the values of interest in response to all the questions in the questionnaire i.e. for both the calibration questions and the target questions.

4.2.3. Aggregation

Each expert is scored based on their performance on the calibration questions, using both their calibration and information scores. The group’s expert judgements for the target questions are aggregated using weighted averaging, where the weights are determined by each expert’s performance score and combined into a single probability distribution for each target question. This distribution reflects the collective knowledge of the group of experts, with more weight given to those who have demonstrated better calibration and informativeness. 

4.3. Documentation and Transparency

The method provides a clear, transparent process for weighting and combining expert judgements. Decision-makers can see exactly how the final aggregated judgements were derived, and which experts contributed the most to the final estimate.

5. Elicitation Methods – A Summary of the Differences

	SHELF Protocol	IDEA Protocol	Classical Method
Elicitation Process	Facilitated group discussion aiming for consensus	Independent estimates before and after discussion	Individual estimates with no group discussion
Group size	Small (1 to 6)	Medium (6 to 12)	Large (> 6)
Aggregation Method	Behavioural (consensus-building)	Unweighted mathematical (e.g., mean, median)	Performance-weighted based on calibration
Role of Facilitator	Active, driving consensus	Minimal, process-focused	Minimal, mathematically focused
Focus	Consensus-building	Independent judgements	Individual expert's performance
Discussion and Group Dynamics	Managed to reach consensus	Group discussion to share reasoning, but not to force consensus	No group discussion, focuses on individual estimates
Bias Mitigation	Facilitator-driven, to reduce anchoring and overconfidence	Process driven to reduce anchoring and overconfidence	Bias handled mathematically via calibration and performance weighting
Type of Judgements	Shared probability distributions	Individual probability estimates	Individual probability estimates
Application Context	Regulatory decision-making, policy assessment	Risk analysis, forecasting	Safety, engineering, environmental risk

6. Discussion

6.1. Weighting and Aggregation

The effectiveness of the different protocols can be difficult to compare. Any validation of expert elicitations should assess how accurately the experts’ beliefs are being represented as well as whether those beliefs represent reality. But as it is only possible to assess the former with an elicitation process, comparisons between the methods usually concentrate on the latter. However, expert elicitations are used to provide estimates for unknown quantities, but for such a comparison it would be necessary to have access to known values that are however unknown to the experts. The classical method partially avoids this problem by providing seed questions related to the original question of interest and with known answers, but which are assumed to be unknown to the experts. Using such a set of questions with each of the different protocols can provide a comparison of their relative accuracy.

Some recent work to compare these methods (Williams, Wilson, & Wilson, 2021) has shown that the SHELF method of unweighted behavioural aggregation can achieve better results than those that use performance weighted or other mathematical aggregation techniques.

6.2. Bayesian Aggregation

In addition to the behavioural, simple mathematical and weighted performance aggregations as used by SHELF, IDEA and the Classical Method respectively, a Bayesian aggregation method has been proposed (Hartley & French, 2021) that considers the problem-owners current knowledge of the subject area as the prior distribution, the experts’ responses as new data, which can be combined through the likelihood function to derive a distribution of the true value given their individual judgements and applies Bayes theorem to derive a final elicited posterior probability.