How Far Down the Backtest Leaderboard Should You Trust? Search Width, Reporting Bias, and Decision Quality

1. Introduction

Machine learning is now applied to financial data at every scale, down to individual transactions [17], and automated research tools generate and backtest thousands of trading strategies in an afternoon. That makes one design choice newly important: how many candidates to search before picking one. Searching more has an obvious cost: the best of many backtests is more likely to be lucky, so its reported performance is more inflated. It may also have a benefit: a wider search is more likely to contain a genuinely good strategy. Which effect dominates for the strategy you actually hold?

The literature has mostly studied the first effect, how far to discount a winning backtest [1,8]. We study both effects and keep them separate. We call the winner’s backtested performance the report and the strategy actually held the choice. The report can be badly wrong while the choice is fine, and the reverse; a single statistic cannot settle both.

What we do.

We make one object the center of the analysis: the backtest leaderboard, the average out-of-sample return of the strategy ranked j-th in-sample, for every rank j. Section 3 shows that under random-subset search, the expected out-of-sample payoff of the most common selection rules, at every search width, is an exact weighted average of this leaderboard with known hypergeometric weights. Wider search simply puts more weight on the top ranks. So the practical question “should I search more?” becomes “does the top of the leaderboard beat the ranks just below it?” We then measure the leaderboard on three U.S. panels, with a one-day trading delay and measured bid–ask costs.

What we find.

1.

The report is always inflated, and more so with wider search (Figure 2). On mega-caps the winner of a 256-strategy search reports a net return of $+432$ bp a year in its backtest and earns $-84$ out of sample.

2.

Net of costs, wider search improves the choice on diversified panels, but only relative to a narrow search (Figure 3). Selecting on net backtest performance, going from 16 to 256 candidates raises the winner’s out-of-sample return by $+527$ bp a year on industry portfolios and $+792$ on size/value portfolios, mostly by finding strategies that trade less. On mega-caps the gain is indistinguishable from zero for the single winner and positive for a top-4 portfolio.

3.

No width finds a strategy worth holding. Under our main cost model the best net return across widths, panels and eight selection, shrinkage and combination rules is $+127$ bp a year, with a confidence interval that includes zero (Figure 5). Under the lower and upper cost models, the exact width curves never exceed $+38$ bp a year.

Along the way we find that most of the short-term trend profit in daily industry portfolio data comes from trading at the same close that generates the signal; with a one-day delay it disappears (Figure 4). The pattern matches the stale-price signature of Lo [11].

Contributions.

(i) An exact reduction from search width to the shape of the leaderboard, with a simple corollary: if out-of-sample payoffs never improve as you move down the in-sample ranking, searching more can never hurt the single winner. (ii) Width varied as a controlled treatment, with selection rule and portfolio size held fixed, on real trading strategies graded out of sample net of measured costs. (iii) A benchmark of selection against shrinkage and combination under the same cost accounting, with trades netted across strategies.

2. How This Relates to Past Work

Discounting the winner.

Multiple-testing corrections tell a researcher how much to distrust the best backtest given the number of trials [1,4,6,8,15,16]. Our report results agree with this literature and add nothing new to it; we use it as the benchmark for the report.

Grading the choice.

A smaller literature asks whether the selected strategy is good. The probability of backtest overfitting [2] measures how often the in-sample winner falls below the out-of-sample median; it summarizes where the in-sample winner lands on our leaderboard. Chen [3] apply empirical Bayes to about 136,000 mined long–short strategies and find that naively mining for the largest Sharpe ratios performs similarly, while only empirical Bayes delivers unbiased predictions of performance. That is close to a report/choice split, and our shrinkage results agree with it. The decision-theory version of the split is the optimizer’s curse [7,14]. What we add is search width as the treatment and costs as part of the outcome.

Costs and implementation.

Anomaly returns shrink sharply after trading costs [13] and after publication [12]. We build costs bottom-up from quoted spreads and from the effective-cost estimates of Hasbrouck [9], cross-checked against the century of large-stock spreads in Jones [10]. Daily portfolio returns are known to contain stale-price autocorrelation [11], which is why we trade one day late.

Best-of-n in machine learning.

Language-model alignment studies the same trade-off: as more candidates are scored by an imperfect reward model, the true reward of the selected output can rise and then fall [5]. Finance offers an unusually clean test bed: the “gold” reward is future returns, observed only after the selection is frozen.

3. Method

3.1. Setup: Report, Choice and Leaderboard

There are N candidate strategies. In each evaluation period (an epoch) we rank them by in-sample Sharpe ratio over the previous three years (rank 1 is best) and record each rank’s out-of-sample mean return over the next six months. Let $Y_e\left(j\right)$ be the out-of-sample return of the rank-j strategy in epoch e. The leaderboard is $L\left(j\right)={\displaystyle \frac{1}{E}}{\sum }_e{Y}_e\left(j\right)$, shown beside its in-sample counterpart. A researcher with width n sees only a random subset of n candidates and applies a selection rule: pick-best holds the subset’s top strategy; top-k holds its top k in equal weights. The report is the selected strategy’s in-sample return; the choice is its out-of-sample return.

3.2. The Exact Reduction

Proposition 

(Width is a weighting of the leaderboard). Under uniform random n-subsets of N ranked candidates, the probability that rank j is the pick-best winner is

$$P_n\left(j\right)=\left({\displaystyle \genfrac{}{}{0pt}{}{N-j}{n-1}}\right)/\left({\displaystyle \genfrac{}{}{0pt}{}{N}{n}}\right),$$

and the expected weight on rank j in an equal-weight top-k portfolio ($n\ge k$) is

$$w_{n,k}\left(j\right)={\displaystyle \frac{n}{N}}·{\displaystyle \frac{1}{k}}·Pr\left[\mathrm{Hypergeom}(N-1,j-1,n-1)\le k-1\right].$$

Hence the expected out-of-sample payoff at width n is ${\sum }_j{P}_n\left(j\right)L\left(j\right)$ for pick-best and ${\sum }_j{w}_{n,k}\left(j\right)L\left(j\right)$ for top-k. The expected report is the same weighting applied to the in-sample leaderboard.

Corollary 1. 

$Pr[\mathrm{winner}\text{’}\mathrm{s}\mathrm{rank}\le k]=1-\left({\displaystyle \genfrac{}{}{0pt}{}{N-k}{n}}\right)/\left({\displaystyle \genfrac{}{}{0pt}{}{N}{n}}\right)$ rises with n. So if $L\left(j\right)$ is non-increasing in j, the pick-best payoff never falls as width grows. Width can hurt the choice only if the top of the leaderboard underperforms ranks below it.

Proofs are in Appendix Section 7. Mean return is linear in portfolio weights, so these curves are exact and free of the subset-sampling noise of Monte Carlo designs. An independent script verifies them by brute-force simulation and by full enumeration, and checks numerically that the top-k curves are also monotone under a non-increasing leaderboard. At $N=652$, a width-16 search finds the global best strategy 2.5% of the time and has a median winner rank of 28; a width-256 search finds it 39% of the time with a median winner rank of 2.

3.3. Data and Candidate Strategies

• Mega-caps: the 1,000 CRSP stocks with the largest average market capitalization among those with at least 3,500 trading days (funds excluded), 2010–2025 (out-of-sample 2013–2025; 25 epochs).
• Industries: the 49 value-weighted Fama–French industry portfolios, 1970–2025 (out-of-sample 1973–2025; 106 epochs).
• Size/value: the 25 value-weighted size and book-to-market portfolios, same dates.

The candidate set is 652 mechanical cross-sectional signals with no hand-picking: moving-average crossovers (trend), past-return momentum with several lookbacks and skips, and short-term reversal, each in both directions and some volatility-scaled. Each signal becomes a daily dollar-neutral long–short portfolio with $1 of gross exposure.

3.4. Trading Delay and Costs

A signal formed at Monday’s close is traded at Tuesday’s close and earns Wednesday’s return (a one-day delay). Turnover is measured with positions drifting with returns between trades. Each dollar traded pays a half-spread:

• Mega-caps: each stock’s closing quoted half-spread (median 1.5 bp).
• Portfolios: the value-weighted half-spread of the stocks in the same size quintile. Sources are Hasbrouck’s effective-cost estimates (1970–1992), his trade-level sample (1993–2005) and CRSP quotes (2010–2025), interpolated for 2006–2009 (Appendix Section 8). The value-weighted average falls from about 33 bp in the 1970s to about 2 bp after 2010.

Strategies are selected on in-sample net Sharpe ratio, so the researcher knows about costs. We also report a lower and an upper cost model (Appendix Section 8).

3.5. Inference and Evidence Discipline

Confidence intervals resample blocks of six consecutive epochs, because adjacent in-sample windows overlap. That leaves 18 independent blocks on the portfolio panels and only 5 on mega-caps. The test of whether the top of the leaderboard beats the next tier permutes rank labels within the top 50 inside each epoch, which uses every epoch. All samples in this paper are discovery data that earlier versions of this study had already examined. A one-time test on never-examined pre-2010 single-stock data is preregistered (Section 6), and every analysis is logged with the number of times its sample had been examined (Appendix Section 9).

4. Findings

4.1. The Leaderboard

Figure 1 is the paper in one picture. In every panel the in-sample curve (gray) sits well above the out-of-sample curve (blue): the backtest overstates at every rank. Before costs, out-of-sample returns are positive and roughly flat across the top 100 ranks of the industry and size/value panels, which means the ranking carries information but the top ranks are not much better than rank 50. After costs (orange), returns are at or below zero even at the very top and fall steeply below it: because the ranking is on net performance, the lower ranks collect the strategies whose trading costs exceed their gross return.

4.2. Report Versus Choice as Search Widens

Figure 2 applies Proposition 1 to both curves of Figure 1. The report (purple) rises steadily with width on every panel: searching more always makes the winner’s backtest look better. The choice (orange) rises much less, and on mega-caps not at all beyond a few dozen candidates. The gap between them, the overstatement, grows with width on mega-caps and industries and levels off beyond a few hundred candidates on size/value. On mega-caps the winner of a 256-strategy search reports $+432$ bp a year net of costs and earns $-84$ (Table 1). This is the familiar winner’s curse, now measured net of costs and as a function of width.

4.3. What Wider Search Buys

Searching more improves the net choice on both portfolio panels (Table 1): $+527$ bp a year on industries and $+792$ on size/value, going from 16 to 256 candidates, with confidence intervals well above zero. Figure 3 shows why. On the portfolio panels, a wider search finds strategies that trade less (the gray cost line falls by half) while earning more before costs. Slow momentum signals fill about 90% of the top ten. On mega-caps, the two effects cancel: the strategies a wide search finds trade less but also earn less, so the single winner’s net return is flat. Top-4 portfolios gain from width on all three panels.

The improvement is relative. Even at the best width, the winner’s out-of-sample net return is $-57$ bp a year on industries, $-392$ on size/value and $-84$ on mega-caps. Wider search picks a less bad strategy.

Corollary 1 says width can only hurt if the top of the leaderboard underperforms the next tier. Before costs, that happens on industries: ranks 1–5 earn 96 bp a year less than ranks 6–50 ($p<0.001$), and the pick-best curve peaks near 32 candidates. After costs the sign flips, because the top net ranks are the strategies that trade least, and that advantage persists out of sample.

4.4. The One-Day Delay

Like many daily backtests, earlier versions of this study traded each signal at the same close that produced it. Figure 4(a) shows what a one-day delay does. On industry portfolios, the best 10% of the old backtest earned 805 bp a year out of sample with next-day trading, $-40$ with a one-day delay and $-173$ with two. Panel (b) explains the pattern. The next-day profit is largest from the 1960s to the 1990s, when portfolios’ own day-to-day autocorrelation exceeds their cross-autocorrelation. That own autocorrelation is strongest in small-stock portfolios (0.44 for the smallest size quintile of the size/value panel versus 0.24 for the largest, in the 1970s), turns negative after 2000, and the profit disappears with a one-day delay. That is the signature of stale prices in portfolio returns, not a tradable effect. Mega-cap signals, by contrast, survive the delay. Short-term reversal earns about the same with quote midpoints as with closing prices, so it is not bid–ask bounce; with 41% daily turnover it roughly breaks even at the quoted spreads of these stocks.

4.5. Selection Versus Shrinkage and Combination

Figure 5 compares eight rules at full width on all three panels, with trades netted across strategies. Holding the top four instead of the top one helps on both portfolio panels (by 180 and 165 bp a year) and makes no difference on mega-caps ($+4$). Ranking by empirical-Bayes posterior mean rather than raw Sharpe changes little: on size/value it picks the same four strategies, and elsewhere it adds 38–74 bp a year with intervals that include zero [consistent with [3]. Weighting by the posterior mean mostly holds cash. On mega-caps it holds cash in every epoch, because no strategy’s shrunk Sharpe ratio is positive. That is the right answer when nothing has a positive expected net return. Naive combinations do worst. A ridge mean-variance blend and equal weights across all 652 strategies lose hundreds of basis points a year on mega-caps and thousands on the portfolio panels, because blending hundreds of fast signals produces a portfolio that trades 35–42% of its book every day. The best rule anywhere earns $+127$ bp a year (EB top-4 on industries, Sharpe ratio 0.19). The best on mega-caps is an equal-weight top 16 at $+64$. Both intervals include zero.

5. Conclusion

The backtest leaderboard turns “how many strategies should I search?” into a question that data can answer: does the top of the in-sample ranking beat the ranks just below it, after costs? On three U.S. panels, three practical rules follow.

1.

Always discount the report. Once more than a few candidates are searched, the winner’s backtest overstates its out-of-sample return on every panel, before and after costs, and by more as the search widens.

2.

Select on net-of-cost performance, and hold a few strategies rather than one. Selecting on net performance is what makes wider search help. With trades netted, the top four beat the single winner by 165–180 bp a year on both portfolio panels and match it on mega-caps.

3.

Do not expect width to manufacture alpha. Wider search finds the least costly of a set of signals; it does not turn a set with no net edge into one with an edge.

6. Limitations and Next Steps

Limitations.

All samples here are discovery data that earlier versions of this study had already examined, so every number above is provisional. Mega-caps give only five independent blocks, and their universe is chosen on full-sample size and history, which tilts it toward survivors. The exact width curves charge each strategy its own costs, which overstates costs for top-k portfolios relative to the netted portfolios of Section 4.5. Commissions are not modeled. Jones [10] estimates that spreads and commissions together cost at least 1% of dollar volume from 1953 to 1975, so our early net returns are optimistic. Random-subset search is a model of width, not of how researchers actually search; sequential, adaptive search is a natural extension.

Next steps.

1.

One sealed test. A preregistered amendment fixes the primary test (top 5 versus ranks 6–50, net of quoted spreads, one-day delay) for a single run on never-examined 1993–2009 single-stock data, and for a forward run on 2026–2028 data after mid-2028.

2.

The risk report. Earlier work found that selecting the lowest-estimated-risk portfolio understates realized risk by more as width grows. That result must be retested against a null that keeps real volatility clustering, and after standard eigenvalue corrections, before it returns to the main text.

3.

Mechanism. Test whether the flat top of the gross leaderboard is explained by light-tailed true skill plus estimation noise, using the empirical-Bayes prior estimated here.

A Proofs

Proposition 1.

A uniform random n-subset contains rank j as its best member exactly when it contains j and its other $n-1$ members come from the $N-j$ worse ranks: $\left({\displaystyle \genfrac{}{}{0pt}{}{N-j}{n-1}}\right)$ of the $\left({\displaystyle \genfrac{}{}{0pt}{}{N}{n}}\right)$ equally likely subsets. For top-k, rank j is in the subset with probability $n/N$; given that, the number of better ranks among the other $n-1$ members is hypergeometric with population $N-1$, $j-1$ successes and $n-1$ draws, and j is held (with weight $1/k$) when that number is at most $k-1$. The expected payoff is linear in these weights. The same argument applied to in-sample returns gives the expected report.

Corollary 1.

The winner has rank greater than k exactly when all n members come from the $N-k$ ranks below k, which has probability $\left({\displaystyle \genfrac{}{}{0pt}{}{N-k}{n}}\right)/\left({\displaystyle \genfrac{}{}{0pt}{}{N}{n}}\right)$, decreasing in n. So the winner’s rank at width $n+1$ is first-order stochastically smaller than at width n. If $L\left(j\right)$ is non-increasing, $E\left[L\right(\mathrm{rank}\left)\right]$ is non-decreasing in n. The independent verification script also confirms the analogous monotonicity for top-k ($k=2,4,8$) on 2,000 random non-increasing leaderboards each, with no violations.

B. Cost Model

Mega-caps.

Each stock-day’s half-spread is $(\mathrm{ask}-\mathrm{bid})/(\mathrm{ask}+\mathrm{bid})$ at the close, kept when $0<\mathrm{bid}<\mathrm{ask}$; missing days take the median of the stock’s previous 21 quotes. Coverage is 99.8%; the median is 1.5 bp. The upper model doubles it. A low-frequency high–low estimator proved unusable for these stocks: its yearly median swings between 0 and 22 bp with market volatility.

Portfolios.

For each year and NYSE size quintile we take the value-weighted half-spread of member stocks. The 49 industries use the all-stock average, because the sources carry no industry codes.

• 1970–1992: Hasbrouck’s Gibbs estimates of effective cost [9].
• 1993–2005: his trade-level comparison sample.
• 2010–2025: CRSP closing quotes.
• 2006–2009: log-linear interpolation.

Against the trade-level data, the Gibbs estimates are accurate for the smallest quintile but overstate the largest by 2–7 times after the mid-1990s, when spreads became too small for daily data to resolve. Before 1993 we keep them unscaled, because Dow stocks’ quoted spreads were near 0.6% (about 30 bp half-spread) in the 1950s and 1980s [10], at or above the Gibbs values for the largest quintile. The lower model scales pre-1993 values by each quintile’s 1993–1995 ratio of trade-level to Gibbs cost (0.48 for the largest). The upper model keeps Gibbs through 2005 and doubles quoted spreads after 2010. Conclusions are the same under all three models.

C. Evidence Accounting

Every analysis is logged with its date, its sample, the sample’s status (discovery, previously used holdout, or never examined) and how often that sample had been examined before. The industry panel had been examined in about twenty earlier analyses, so every result on it is exploratory rather than confirmatory. Two holdouts from earlier work, French 1927–1969 and 1,000 never-examined CRSP names, were each opened once before the cost and delay layers existed. In both, the winner’s reported optimism stayed within a pure-selection benchmark. Those runs used next-day trading before costs; we now read them as evidence about the report, not the choice. Pre-2010 single stocks and the 2026–2028 future remain sealed.