Can Uncertainty Metrics Guide Search? Evaluating Search-Time Decision-Making in Large Language Models

1. Introduction

Sampling-based search methods for large language models (LLMs) involve generating multiple responses, either in parallel, sequentially, or through hybrid approaches, and aggregating them into a final answer. Techniques such as Chain-of-Thought (CoT ) [34], Tree-of-Thoughts (ToT) [39], and ReAct [40] have demonstrated that structured reasoning through intermediate steps can significantly enhance LLM performance. These approaches can be further extended with search algorithms, including Monte Carlo Tree Search (e.g., LATS, STaR) [42,43], bandit-based methods (e.g., LongPO) [7], and gradient-based optimization (e.g., OPRO) [38].

A crucial component in these search and optimization strategies is uncertainty estimation, which is vital for guiding decisions, balancing exploration and exploitation. Uncertainty estimation techniques, such as those used in Bandit algorithms or Bayesian optimization, can dynamically adjust learning rates or hyperparameters. In combinatorial optimization (e.g., genetic algorithms), uncertainty estimation informs heuristic decisions like mutation rates or temperature adjustments. Thus, developing robust methods for quantifying uncertainty in LLMs is essential for enabling more effective search behavior.

Prior work on LLM uncertainty estimation has primarily focused on methods based on generation likelihoods [1,12,18], verbalized confidence [11,15], and response consistency [9,37]. These methods have shown utility for tasks like hallucination detection and risk assessment. However, we argue that these metrics primarily serve a descriptive role: they reflect the model’s confidence in a given query but are not designed to guide or influence the generation process. For example, a model uncertainty score of 0.8 indicates a 20% estimated error rate when sampling multiple responses to a query, but it does not help determine which of those candidate responses is correct or should be preferred.

In contrast, search-based tasks demand a more prescriptive role from uncertainty estimators: rather than passively measuring confidence, they must actively guide decision-making during search. For instance, in multi-step generation or tree-based reasoning, uncertainty scores should inform which candidate answers to select, which nodes to expand, and which paths to prune or refine. An effective prescriptive estimator should provide fine-grained, local feedback that incrementally reduces error. Without this capability, even a well-calibrated estimator can lead to misleading decisions during search.

In this study, we evaluate whether existing LLM uncertainty metrics can fulfill a prescriptive role in search-based tasks by introducing two diagnostic tasks: (1) Answer Selection—choosing the best answer from a set of candidates; and (2) Answer Refinement—identifying which response most needs refinement. Our empirical results show that while current uncertainty metrics are useful for flagging risky or error-prone responses, they struggle to reliably guide search toward correctness. This highlights the need for optimization-aware uncertainty metrics explicitly designed to support decision-making in correctness-driven search.

2. Diagnostic Tests

Search-based methods for LLMs typically involve generating multiple candidate responses—either in parallel, sequentially, or both—and then aggregating them into a final answer. Within this process, two fundamental actions frequently occur: selection and refinement. Selection refers to choosing which candidate responses to explore, exploit, or use in final aggregation—usually in a parallel setting. Refinement, often used in sequential settings, involves generating follow-up outputs based on earlier responses, such as self-reflection [20], verification [35], or correction [19]. Based on these two fundamental actions, we design two diagnostic tests to evaluate whether existing uncertainty metrics can meaningfully guide the search process. These tests aim to answer the question: Can uncertainty metrics be reliably linked to correctness in a way that makes them useful for decision-making?

Test 1: Answer Selection

This test examines whether an uncertainty metric can help select the best answer among candidates. For each question, the LLM generates multiple responses. Responses that yield the same answer (regardless of surface form) are grouped into answer clusters. The uncertainty metric ranks these answer groups, under the assumption that correct answers should appear less uncertain (i.e., the lowest uncertainty score group is selected as the final answer). We measure how well the metric prioritizes correctness using three evaluation criteria: Pairwise-Win-Rate (prefer correct answers over an incorrect ones), Top-1 Accuracy (whether the top-ranked candidate is correct), and Mean-Reciprocal-Rank (how high correct answers appear on average).

Test 2: Answer Refinement

This test examines whether uncertainty metrics can identify which answers are most in need of refinement. For each question, the LLM first generates a set of initial answers. Refinement is then applied only to the answer cluster exhibiting the highest uncertainty score. A high uncertainty metric value suggests that an answer is more likely to be incorrect and thus a better candidate for refinement. We evaluate each metric by measuring the overall accuracy gain achieved through this targeted refinement.

Insights and Applications

The two tests highlight distinct roles for uncertainty metrics in search-based tasks. Test 1 assesses how well a metric ranks candidate answers by correctness—useful for reranking and aggregation. Test 2 evaluates the ability to prioritize answers for refinement, which is crucial in compute-limited settings like interactive systems. Together, these tests reveal how uncertainty metrics can guide specific search actions and inform their use in real-world applications.

3. Experiments

In this section, we introduce the uncertainty metrics, datasets, models, and baseline methods used in our two diagnostic experiments. Additional implementation details and prompt templates are provided in Appendix A and Appendix C.

Benchmarked Metrics

We benchmark six task-agnostic uncertainty metrics that don’t require additional model training. Generation-liklihood based metrics estimate uncertainty from the predictive entropy of model’s output distribution: Normalized Predictive Entropy (NPE), Length-Normalized Predictive Entropy (LNPE) [18], and Semantic Entropy (SE) [12]. Verbalized-based measures assess confidence by directly prompting the model to express its certainty: Verbalized Confidence (VC) and P(True) [11]. Lexical-based metrics: Lexical Similarity[16], evaluate uncertainty based on the consistency of lexical overlap between multiple responses. Since P(True), and VC are originally formulated to express model confidence, we transform them into uncertainty measures by taking their complements: PTrue-Comp (1-P(True)), and VC-Neg (negative of VC).

Datasets and Models

We select four diverse datasets: MATH (mathematical reasoning) [6], CommonsenseQA (commonsense reasoning) [21], TriviaQA (reading comprehension) [10], and TruthfulQA (truthfulness) [14]. MATH and TriviaQA feature open-ended (free-form) answers, while CommonsenseQA and TruthfulQA use multiple-choice formats. These datasets are evaluated using three open-source language models [3,8,22] of similar sizes: LLaMA-3-8B, Gemma-2-9B, and Mistral-7B-v0.3. This setup ensures diversity across task types and model architectures, enhancing the robustness of our evaluation.

Baselines

In Answer Selection test, we compare Uncertainty-Guided Selection with (1) Random Selection, where a candidate is chosen uniformly at random, and (2) Majority Voting, which selects the most frequent answer as a strong self-consistency baseline [33]. In Answer Refinement test, we test whether Uncertainty-Guided Refinement outperforms (1) refining a random answer (Random Refinement), (2) refining the majority answer (Majority Refinement), and (3) refining all answers (All Refinement). For both tests, random baselines are calculated by averaging the scores over 50 runs of sampling to estimate the expected performance when randomly selecting a choice.

4. Result and Analysis

Uncertainty Metrics Fall Short in Answer Selection

As shown in Figure 1, Uncertainty-Guided Selection generally outperforms Random Selection but underperforms compared to Majority Voting. This indicates the limited applicability of uncertainty metrics in answer selection, as Majority Voting, being computation-free, still outperforms uncertainty-guided methods.

To dive deeper, we provide a breakdown of Top-1 Accuracy (ACC), Pairwise Win Rate (PWR), and Mean Reciprocal Rank (MRR) for each uncertainty metric in Table 1. Likelihood-based metrics (NPE, LNPE, SE) typically achieve PWR and MRR above 0.6, indicating that they rank correct answers higher, though they don’t always place the correct answer at the top. In contrast, lexical- and verbalization-based metrics (Lexical, VC-Neg, PTrue-Comp) perform poorly, with values often below 0.5. Notably, all metrics struggle with the MATH dataset, likely due to its complex reasoning demands and open-ended answer format, which makes it difficult for existing metrics to provide reliable signals.

Uncertainty Metrics show potential to guide Answer Refinement.

As illustrated in Figure 2, Uncertainty-Guided Refinement leads to notable accuracy gains on CommonsenseQA and TruthfulQA, achieving performance close to Majority Refinement but using fewer refinement attempts. On MATH and TriviaQA, where accuracy gains of All Refinement are negative, Uncertainty-Guided Refinement surprisingly shows a slightly positive values, likely due to refining fewer answers and reducing unnecessary corrections.

To analyze the refinement effect, we quantize answer candidates into four bins based on their uncertainty metric values and measure the accuracy gain achieved by refining answers in each bin. As shown in Figure 3, refining answers in the higher-uncertainty bins generally leads to larger accuracy gains—consistent with the intuition that uncertain responses are more likely to be incorrect and therefore benefit from refinement. However, we also observe positive gains from refining answers in the lowest-uncertainty bin, suggesting potential mismatch between model confidence and correctness.

Descriptive vs. Prescriptive

From the previous analysis, we observe that current uncertainty metrics show potential in guiding answer refinement but struggle in answer selection. We argue this is because these metrics are designed for a descriptive role—capturing the model’s confidence among outputs—rather than serving a prescriptive role, where uncertainty directly informs decisions during search. Additional figures in Appendix B highlight the descriptive power of these metrics.

The difference between descriptive and prescriptive roles helps explain why Uncertainty-Guided Refinement performs better than Uncertainty-Guided Selection. Refinement uses uncertainty metrics to identify low-confidence answers that are worth improving, which is a task that aligns with the descriptive strengths of the metrics. In contrast, selection assumes a mapping between confidence and correctness. This requires the model not only to recognize its uncertainty but also to correctly assess which answers are accurate.

5. Conclusions

In this study, we use two diagnostic tests to investigate the ability of LLM uncertainty metrics to guide search tasks. While these metrics are effective at identifying risky responses, they primarily capture model confidence and fall short in guiding toward correct answers. Our findings highlight the critical challenge of bridging the gap between model confidence and correctness, emphasizing the need for uncertainty metrics designed to support correctness-oriented decisions in search tasks.