The Winner’s Curse in LLM Pipeline Selection: How Correlated Candidate Pools Break Best-of-N Reporting, and How to Correct It

Best-of-N pipeline selection, running N candidate prompts on an m-item dev set and reporting the winner's dev score, is the default way an LLM practitioner picks and reports a prompt. That dev score is optimistic: the winner was chosen in part because it got lucky on the dev items, and its true accuracy on fresh data is lower. We measure this winner's curse directly on four correctness matrices spanning three tasks (SST-2, Subj, AG News) and two models (Qwen2.5-1.5B-Instruct, Llama-3.2-3B-Instruct), sweeping N from 1 to 100 and m from 20 to 200. Bias grows with N and shrinks roughly as 1/sqrt(m), reaching 0.088 accuracy points at N = 100, m = 100 on Subj. We then characterize why real candidate pools inflate this bias beyond the independent-candidates textbook case: two effective-candidate-count estimators, a spectral participation ratio and a moment-matching inversion of the observed selection gap, agree that a 105-to-111-candidate pool behaves like roughly 1.5 to 3.5 independent draws. Breaking that count down by candidate provenance turns up a result opposite our own working hypothesis: LLM-generated (APE-style) candidates carry far less effective diversity (N_eff/K ~ 0.017-0.041) than either hand-written seed prompts or paraphrase/mutation candidates (N_eff/K ~ 0.14-0.24 for both), consistently across all three tasks. We compare six ways to report a best-of-N winner (no correction, a naive dev split, two extreme-value union-bound corrections keyed to K or to N_eff, cross-fitting, and a bootstrap plug-in) on bias, mean squared error, and 90% interval coverage; correcting against N_eff dominates the independent-candidates union bound on every axis and matches or beats every other method on mean squared error, while the naive K-based correction overshoots badly enough to be worse than reporting nothing at all. Resampling the stored matrices at the exact (N, m) of a real, live EvoPromptGA trajectory reproduces its measured bias on all three tasks, inside a 90% predictive band in every case, which supports treating a one-time correctness matrix plus free resampling as a faithful stand-in for repeated live search. Finally, we derive budget-allocation curves for a fixed evaluation budget B = N * m: on tasks already near their accuracy ceiling, the optimal split spends almost the whole budget on m; on a task with real headroom left, it keeps paying to search wider even as N_eff/K stays in the low single digits.