The Noise Floor of Agent Evaluation: Live Generation, Replay, and Rerun Budgets

1. Introduction

An evaluation score summarizes what a system achieved under a particular set of tasks and operating conditions. In a coding-agent pipeline, however, that score is produced through several stages: a hosted model generates a candidate program, an execution environment and verifier determine whether it succeeds, and a model-based judge may provide an additional verdict. Repeating the pipeline can therefore change the candidate, the executed outcome, or the judgment. Whenever a score improves, this creates a practical question: how much of the difference reflects the configuration being compared, and how much could arise from repeating the measurement?

Answering this question requires more than attaching an error bar to an aggregate score because each stage supports a different comparison. Fresh generation measures performance when the system must produce new candidates. Replay instead holds recorded responses fixed, allowing downstream behavior to be compared on the same generated inputs. Repeated judging asks a complementary question: whether the scoring layer returns consistent verdicts for an unchanged candidate. These controls therefore distinguish different forms of repeatability. A repeatable evaluation need not agree with an executable reference, and a controlled replay comparison does not establish reliability under fresh generation.

Several lines of prior work motivate these distinctions. Statistical treatments of language-model evaluation show how uncertainty and experimental planning affect score comparisons [6], while studies of machine-learning and reinforcement-learning benchmarks show that run-to-run variation can complicate the interpretation of improvements [2,4]. Agent benchmarks further distinguish occasional success from repeated-trial reliability [8], and audits of task setup and reward design ask whether scores represent the intended outcomes [10]. Model-based judging introduces another layer: agreement with reference judgments and reproducibility under temperature control must themselves be evaluated [7,9]. What remains is to connect these questions within one executable pipeline and determine how the measured variation should inform replication.

To address this problem, we introduce the determinism ladder, a measurement framework with three generation conditions: fresh hosted generation at production temperature, fresh generation at temperature zero, and content-addressed replay through executable verification. A separate judge condition fixes the candidate stream and varies only the hosted judge. Rather than assuming that variation can always be partitioned additively into provider, execution, and judge components, we measure the mean variance across repeated outcomes for each task under each operating condition. We then specify the conditional scope of shared replay and use an independent-stream planning approximation to translate the measured variance into replication for a fixed-benchmark contrast.

Using this framework, we found lower within-task variance under replay than under fresh generation across HumanEval+ and MBPP+ in two hosted model-and-verifier stacks. In the primary stack, live variances of 0.0301 and 0.0142 fell to 0 and 0.0010 under replay. The second stack reproduced this ordering, with live variances of 0.0235 and 0.0134 and no observed replay variation. Temperature zero reduced generation variance in both stacks, although the remaining floor differed. The judge experiment revealed a complementary distinction: temperature zero stabilized the verdict for every frozen candidate over 20 repetitions, yet majority verdicts still disagreed with executed outcomes on 47 of 200 tasks. Together, these findings show why generation repeatability, judge repeatability, and reference agreement must be measured separately before deciding how much replication a comparison needs.

This work makes three contributions:

• A framework for measuring evaluation variability. We distinguish live evaluation from replay-controlled comparison, state the conditional scope of shared replay, and connect within-task variance to a fixed-task rerun budget (Section 3).
• Measurements across two executable evaluation stacks. We characterize generation variance on two code benchmarks, assess sensitivity to temperature and replay, and report task-bootstrap uncertainty with reconstruction procedures tied to study records (Section 4 and Section 5).
• A separation of judge repeatability from reference agreement. We show that stabilizing verdicts on frozen candidates can leave disagreement with executable outcomes unchanged (Section 5.3).

2. Related Work

2.1. Variability and Validity in Benchmarks

Language-model evaluations can be treated as statistical experiments, with the design of a comparison determining which uncertainty estimates and planning calculations apply [6]. This framing is consistent with evidence that data sampling, initialization, hyperparameter choices, and other sources of variation affect comparisons in machine learning and reinforcement learning [2,4]. Together, these studies motivate measuring variation under the conditions that produce a reported score, rather than interpreting every difference between runs as a change in capability.

For agents, repeated-trial reliability and the validity of the scoring procedure address complementary concerns. The $\tau$-bench metric ${\mathrm{pass}}^k$ distinguishes consistent success across trials from success on an occasional attempt [8]. Benchmark audits further show that task setup and reward design can misrepresent performance even when the score is repeatable [10]. We complement these approaches by comparing fresh generation, temperature control, and replay within an executable pipeline, then relating the measured within-task variance to replication for a specified contrast.

2.2. Model-Based Judging and Executable References

Model-based judging adds a separate source of uncertainty because a judge can be repeatable without agreeing with a reference. LLM judges offer a scalable way to assess generated responses, but their verdicts can be affected by biases and limited reasoning, making reference agreement an empirical question [9]. Reproducibility is a distinct property: temperature control can reduce verdict variation without guaranteeing that repeated evaluations are identical [7]. We therefore hold candidate programs fixed and measure both repeated-verdict variance and disagreement with an executable reference.

To ground this comparison in executable behavior, we use tasks from HumanEval and MBPP [1,3] with strengthened EvalPlus inputs [5]. Executed tests provide a concrete reference for the tested program behavior, although their coverage still defines the scope of the outcome. In the judge experiment, the judge sees the task statement and original examples, whereas the verifier executes capped augmented inputs. The resulting disagreement therefore concerns this particular information set and reference procedure, rather than unrestricted program correctness.

3. Measuring a Noise Floor

We use the term noise floor for the observed within-task outcome variance under a specified evaluation condition. It is an empirical property of that condition, not a proven lower bound on all achievable evaluation variance. The framework separates interventions on generation from repeated measurement by a judge, allowing each comparison to answer a distinct question (Figure 1).

3.1. Generation Conditions and Frozen-Candidate Judging

The generation ladder first varies how candidate programs are obtained. Live generation makes fresh hosted requests at temperature 0.7, temperature-zero generation makes fresh requests to the same endpoint at temperature 0, and replay replaces generation calls with content-addressed recordings before re-executing the recorded candidates through the programmatic verifier. Replay therefore holds generated responses fixed but not executed outcomes. Any variation that remains under replay must consequently be assessed on the downstream execution and verification path.

We evaluate this ladder in two hosted stacks. The primary stack combines a dated gpt-4o-mini snapshot with a local differential verifier, whereas the second combines a dated claude-haiku snapshot with the official EvalPlus verifier. Both use the same task identifiers, closed-book generation prompt, one-retry structure, cap of 120 test inputs, and 20 replicates per condition. Because the verifier and timeout implementations also change, the second experiment asks whether the measurement pattern transports across a model-and-verifier stack; it does not isolate a provider effect.

Generation control alone does not determine whether a model-based scoring layer is stable. For the judge experiment, we therefore fix the final candidate associated with each of the 200 primary MBPP+ replay tasks and score each candidate 20 times at judge temperature 0.7 and 20 times at temperature 0. This design isolates verdict variation conditional on one candidate stream. Comparing the majority verdict with the executed reference then measures a second property, agreement, allowing stable disagreement to be distinguished from stochastic changes in judgment.

3.2. The Conditional Scope of Shared Replay

Shared replay is intended for comparisons that hold generated responses fixed. Its interpretation therefore depends on whether the configurations being compared can be supplied with the same request-keyed outputs, not merely on whether both configurations can access a cache.

Proposition 1

(Replay Coupling, Sufficient Condition). Let a recording associate a fixed response with each normalized generation request. Suppose two configurations issue the same multiset of normalized generation requests for every task, and replay returns the associated recorded response. Conditional on that recording, the paired contrast is unbiased for the corresponding comparison under the fixed request–response map.

The comparison in Proposition 1 is conditional on the recording. It does not estimate performance averaged over fresh hosted responses. Content-addressed keys support this control by making a missing request fail rather than silently reverting to live generation. Changes to retries, routing, tools, or request content can alter the requests that occur and invalidate automatic reuse of a recording. Such changes require independent generation streams or a coupling argument appropriate to the revised request structure (Appendix A).

3.3. Within-Task Variance and Uncertainty

To quantify repeatability, let $Y_{irc}$ be the binary outcome for task i, replicate r, and condition c. With N tasks and R replicates, we report

$$w_c={\displaystyle \frac{1}{N}}{\displaystyle \sum _{i=1}^N}{s}_{i,c}^2,$$

(1)

where $s_{i,c}^2$ is the sample variance across the R outcomes for task i. This measure captures changes in the outcome of the same task across reruns. It is distinct from variation in average success rates between easy and difficult tasks.

We compare operating conditions through contrasts such as $w_{\mathrm{live}}-w_{\mathrm{replay}}$ and $w_{\mathrm{live}}-w_0$. These differences are reported without truncation. We do not assign them to a physical source of noise unless the design isolates that source. For uncertainty, we resample benchmark tasks with replacement 20,000 times and recompute the relevant conditions jointly. The task is thus the resampling unit, and the intervals describe sensitivity over the observed benchmark tasks rather than establishing transport to a broader task population.

3.4. A Rerun Budget for a Fixed-Task Contrast

Replication should be planned for the difference that an evaluation is intended to detect. For a contrast on a fixed set of N tasks, we use an independent-draw approximation with within-task variances $w_A$ and $w_B$:

$$SE\left(\widehat{\Delta }\right)=\sqrt{{\displaystyle \frac{w_A+w_B}{NR}}}.$$

(2)

The approximation excludes cross-task and cross-replicate covariance and assumes independent streams between arms. A paired design can require a covariance term; the displayed budgets are not estimates of that paired-design variance.

We invert Equation 2 for the smallest positive integer R corresponding to 80% normal-approximation power at two-sided $\alpha =0.05$. For the symmetric comparisons, we set $w_A=w_B=w_c$ and use an illustrative difference of $\delta =0.02$, or two percentage points. These are point planning budgets based on measured variance. A budget of one does not mean that variance can be estimated from one run, or that zero variance observed in a pilot establishes deterministic behavior. In addition, generalization to a task population requires uncertainty beyond the fixed-task term (Section A.3).

4. Experiments

We combined a constructed mechanism check with three hosted measurements to examine how generation controls and model-based judging affect repeatability. In the constructed check, an oracle injected known stochastic perturbations and replay violations. This tested whether the software responded to controlled changes and invalid replay conditions; the injected magnitudes were not estimates of variability in a hosted system (Appendix B).

We then measured the generation ladder in the primary stack on all 164 HumanEval+ tasks and the first 200 MBPP+ tasks returned by the loader. Each condition used 20 replicates per task. The local verifier compared candidate outputs with the canonical EvalPlus solution on up to 120 base and augmented inputs. An initial failure could trigger one retry, and candidate execution had a 10-second wall-clock limit, with timeout scored as failure. This path used executable verification without an LLM judge (Appendix C).

The second hosted measurement used the same task identifiers and generation conditions with a different model and the official EvalPlus checker. Its timeout policy used ground-truth-relative limits with a 30-second per-input floor and special handling for resource-heavy checks. Holding the task set and broad generation protocol fixed while changing the model and verifier allowed us to examine transport of the qualitative ladder, rather than make a provider-only causal comparison (Appendix E).

Finally, the judge experiment scored one frozen candidate from each primary MBPP+ replay task at two temperatures. The judge received the task description, candidate code, and original example assertions, then returned a pass/fail verdict and stated confidence. The executable reference used capped EvalPlus inputs that the judge did not see. We measured verdict variance and the proportion of tasks on which the majority verdict disagreed with that reference (Appendix D).

We encoded the primary analysis plan in the execution notebook and treat it as author-controlled rather than independently preregistered. The judge and cross-stack follow-ups used frozen successor protocols before their valid calls. The cross-stack successor nevertheless followed 19 completed replicates from an aborted predecessor and is not treated as fully outcome-blind. Only the valid study versions contribute the reported endpoints. Appendix G describes this chronology and the reconstruction checks linking the study records to the reported results. Together, the designs assess repeatability under generation controls, transport across two stacks, and judge agreement on a fixed candidate stream.

Use of generative AI.

Generative AI tools were used in preparing this manuscript and its supporting software; the full disclosure is given in Appendix H.3.

5. Results

5.1. Replay Nearly Eliminates the Observed Generation-Side Variation

We first asked how much variation remained when fresh generation was replaced by replay. In the primary stack, live within-task variance was 0.0301 on HumanEval+ (95% task-bootstrap interval [0.0200, 0.0415]) and 0.0142 on MBPP+ ([0.0077, 0.0215]). Under replay, these values fell to 0 and 0.0010, respectively, with a MBPP+ interval of [0, 0.0030] (Table 1; Figure 2a). Thus, fixing the generated responses removed most of the variation observed during fresh generation under this stack.

The small amount of variation that remained under primary MBPP+ replay was concentrated in one task, Mbpp/271. Excluding that task set replay variance to zero and changed the live-minus-replay contrast by only 0.00006. The records did not localize the residual to runtime behavior, timeouts, or the candidate program, so we retain it as observed downstream variation rather than assigning a cause. Because removing the task had little effect on the aggregate contrast, the main ordering did not depend on this residual.

We then asked whether the same ordering appeared in the second stack. Live variance was 0.0235 on HumanEval+ ([0.0136, 0.0341]) and 0.0134 on MBPP+ ([0.0068, 0.0209]), while no task changed its replay outcome over the observed repetitions. The primary MBPP+ residual therefore did not recur. Because both model and verifier changed, this result supports transport of the qualitative pattern across the two stacks without identifying which changed component accounts for the difference in magnitude.

5.2. Temperature Zero Reduces Variation by a Stack-Specific Amount

Replay nearly fixed the generated candidate stream, but temperature zero continued to make fresh requests. We therefore asked whether lowering generation temperature alone achieved the same observed repeatability. In the primary stack, temperature-zero variance remained at 0.0133 on HumanEval+ and 0.0075 on MBPP+, with outcomes varying on 13 and 9 tasks, respectively. The second stack had smaller remaining variances of 0.0032 ([0, 0.0079]) and 0.0018 ([0, 0.0047]), with three varying tasks on each benchmark.

Temperature zero thus reduced variance in both stacks without eliminating outcome changes in either. The magnitude of the reduction also depended on the deployed model-verifier combination. These measurements therefore support treating temperature zero as an operating condition whose repeatability must be measured, rather than as a general guarantee of determinism.

5.3. Stable Judge Verdicts Can Still Disagree with Execution

After measuring variation on the executable path, we asked whether freezing the candidates also stabilized model-based judgment. At judge temperature 0.7, verdict variance was 0.0073 ([0.0030, 0.0127]), and 12 of 200 tasks received both pass and fail verdicts across 20 repetitions. This variance was approximately half the primary live-generation variance on MBPP+ in magnitude, but the two quantities describe different conditions and are not additive noise components. At judge temperature zero, the verdict for each task was unchanged across all 20 repetitions, covering 4,000 calls.

This stabilization raised a second question: did the more repeatable judgments also agree with the executable reference? At both temperatures, majority verdicts disagreed with execution on 47 of 200 tasks: 23.5%, with a task-bootstrap interval of [17.5%, 29.5%]. Relative to the executed tests, these disagreements comprised 18 false accepts and 29 false rejects. The set of disagreeing tasks and the mean stated confidence of 94.4 were identical at the two temperatures. Lowering temperature therefore stabilized the observed verdicts without changing the majority-verdict gap.

The result separates two properties that can otherwise be conflated. Repeatability describes whether a judge changes its verdict for an unchanged candidate, whereas reference agreement describes whether that verdict matches the declared reference. Because the judge and verifier received different information, the observed disagreement is specific to the prompt, candidate stream, model snapshot, and capped executed tests. Within that design, stable judgment did not establish agreement with execution.

5.4. The Measured Variance Changes the Planned Replication

We next used the measured variances to determine the point rerun budgets for an illustrative fixed-task difference of 0.02. Under the symmetric independent-stream approximation, live evaluation yielded eight runs per arm for primary HumanEval+, six for second-stack HumanEval+, and three for MBPP+ in either stack (Figure 2c). The replay variances yielded a budget of one in all four cases. These replay budgets apply to a valid comparison conditional on recorded generation; they do not turn replay into evidence of reliability under fresh responses.

The judge experiment extends the same planning logic to the scoring layer. Using the temperature-0.7 verdict variance yielded two judged runs per arm, whereas the observed temperature-zero variance yielded one. These values are planning outputs, not an empirical demonstration that every such comparison attains 80% power. They quantify sensitivity to repeated judgments while leaving systematic disagreement with the reference as a separate issue.

Across these analyses, the relevant control follows from the claim being tested. Fresh generation exposes variation when new candidates must be produced, replay measures downstream behavior on a fixed generation record, and repeated judging separates verdict variation from reference disagreement. The replication calculation then quantifies the consequence of the measured stochastic variation under explicit planning assumptions, without substituting repeatability for validity.

6. Discussion and Operational Implications

The combined results show that evaluation variability depends on which parts of the pipeline remain free to change. Across the two tested stacks, replay nearly eliminated the observed variation on the executable path, whereas temperature zero produced a partial and stack-specific reduction. The judge experiment revealed a complementary limit: eliminating observed verdict variation did not change disagreement with execution. The central implication is therefore not that one evaluation mode should replace the others, but that the control and amount of replication should be matched to the claim being supported.

Measurement During Agent Development.

These distinctions matter during development because different evaluation modes answer different questions. Statistical evaluation methods emphasize uncertainty in score comparisons [6], repeated-trial metrics characterize consistency [8], and benchmark audits assess whether tasks and scoring procedures measure the intended outcomes [10]. The ladder connects these concerns at the level of an executable pipeline: a development team can measure performance under fresh generation, compare a downstream revision against fixed responses when coupling is valid, and audit a judge against a declared reference. Recording the evaluation mode therefore makes explicit what a reported score can and cannot establish.

This use of replay is conditional on the request structure remaining compatible with the recording. A revision that changes only downstream processing may admit a controlled replay comparison, whereas a revision that changes which generation requests are issued may not. In the latter case, the recording can cease to provide the shared inputs required for the intended contrast. We would therefore bind replay results to the request-normalization and response-mapping rules together with the generation, execution, and scoring configurations. Such a record would make clear when an earlier comparison remains interpretable and when fresh generation or a revised coupling is required.

Interpreting Stable Disagreement.

The judge results extend the distinction between reproducibility and reference agreement to a frozen executable workload. Prior work documents limitations in model-based judging and residual variability under temperature control [7,9]. Here, temperature zero removed observed verdict changes but left the majority disagreement set unchanged. One possible contributor is the asymmetry in available information: the judge saw original examples, while the verifier executed additional capped inputs. The present design cannot distinguish this explanation from limitations in the judge itself. A follow-up could therefore hold candidates fixed while varying access to test information and compare both temperatures under each information condition, directly asking whether additional evidence changes agreement independently of repeatability.

Replication and Resource Use.

The point budgets make the operational consequence of variability explicit: a condition that requires several live runs can require fewer repeated executions once generation is held fixed. This reduction, however, is a reduction in planned runs, not a measured reduction in tokens, elapsed time, monetary cost, or net computational resources. Recording responses, executing tests, and checking replay validity also consume resources. Any operational comparison should therefore account for those costs separately and should not attribute fewer reruns to an improvement in the underlying agent.

The same distinction should be visible in a release report. We would report the task set, model snapshot, generation temperature, verifier or judge policy, within-task variance, reference outcome, and intended minimum effect. A replay-based claim would additionally identify the recording and coupling conditions, while a judge-based claim would report both repeatability and reference agreement. Reporting the uncertainty procedure and assumptions behind the rerun budget would then allow a reader to distinguish an observed score difference, its sensitivity to rerunning the pipeline, and the evidence needed to interpret that difference.

The next question is whether these controls remain useful in long-horizon agents, where intermediate outcomes can change later requests. A prospective study could compare live and replay-based assessments of declared downstream changes, record when coupling fails, and evaluate accepted differences again under fresh generation. Repeating the judge audit on fresh candidate streams and within interactive trajectories would further test whether the distinction observed here persists when judgments can influence subsequent actions. These studies would assess the operational value of the ladder beyond the single-step coding pipelines examined in this work.

7. Limitations

The empirical evidence covers two hosted providers and dated model snapshots, but the second experiment also changes the verifier and timeout policy. The cross-stack result therefore supports transport of the observed ordering across the tested stacks, not causal attribution of variance differences to providers. Both benchmarks also concern short code-generation tasks with at most one retry rather than long-horizon tool agents or natural repository workloads. Extending the interpretation to those settings requires direct evaluation of their request structure and execution dynamics.

The judge experiment has a narrower scope still. It uses one frozen MBPP+ candidate stream, one prompt, and one hosted judge snapshot. The 23.5% disagreement is relative to capped executed tests that were not shown to the judge, so it does not measure general judge accuracy, disagreement on HumanEval+, or behavior on fresh candidates and interactive trajectories. In addition, the unknown cause of the primary Mbpp/271 replay residual limits attribution of that outcome variation.

The uncertainty and planning results introduce corresponding statistical limits. Task bootstraps describe sensitivity over the observed benchmark tasks, while the displayed rerun budgets use a fixed-task independent-stream approximation. They do not include task-population uncertainty, unmodeled covariance, or an upper bound on rare variation when every pilot outcome is identical. Replay is also conditional on one recording and cannot estimate live reliability. Consequently, neither a one-run point budget nor a degenerate bootstrap interval certifies determinism or achieved power.

The study lineage imposes a final limitation. The follow-ups were conducted by the authors rather than as independently operated replications, and provider-timestamped receipts support aspects of chronology rather than independent public preregistration or attestation of request bodies. The cross-stack successor was also frozen after partial outcomes existed in its invalid predecessor and is not fully outcome-blind. We retain these details so that the reported results can be interpreted together with the circumstances in which the valid protocols were established.

8. Conclusion and Future Work

We introduced the determinism ladder to measure variability under distinct evaluation controls and relate that variability to replication for a stated claim. Across two executable coding stacks, replay nearly eliminated the observed within-task variation, while the effect of temperature zero depended on the stack. On frozen candidates, stabilizing a model-based judge did not improve its majority-verdict agreement with execution. Together, these findings establish an empirical separation between generation repeatability, judge repeatability, and reference agreement.

This separation provides a way to choose controls and plan reruns without treating a reproducible score as sufficient evidence of valid or generalizable performance. Future studies should isolate provider effects with a shared verifier, evaluate judges on fresh streams and interactive trajectories, and test replay coupling in long-horizon agents. More broadly, the determinism ladder makes the stochastic conditions of an evaluation explicit, allowing performance comparisons to be interpreted in relation to the evidence they actually contain.

Appendix A.1. Within-Task Variance

For task i under condition c, the study records R binary outcomes $Y_{irc}$. Writing their mean as ${\overline{Y}}_{ic}$ gives the estimator in Equation 1 explicitly as

$$w_c={\displaystyle \frac{1}{N}}{\displaystyle \sum _{i=1}^N}{\displaystyle \frac{1}{R-1}}{\displaystyle \sum _{r=1}^R}{\left(Y_{irc}-{\overline{Y}}_{ic}\right)}^2.$$

(A1)

The estimator requires $N\ge 1$ and $R\ge 2$; the hosted studies use $R=20$. It measures how the outcome of a given task changes across reruns, rather than how average performance differs between tasks. We retain direct contrasts between operating conditions, including $w_{\mathrm{live}}-w_{\mathrm{replay}}$ and $w_{\mathrm{live}}-w_0$, without truncating negative differences or assigning them to a physical component without an identifying intervention.

Appendix A.2. Task-Bootstrap Intervals

For the empirical variance and contrast intervals, we resample benchmark tasks with replacement 20,000 times, preserve multiplicity, and recompute the relevant conditions jointly. The reported percentile intervals therefore use the task as the resampling unit. The disagreement interval in the judge study similarly resamples the 200 tasks with their frozen candidates.

These intervals describe sensitivity to reweighting the observed benchmark tasks. They do not by themselves establish representativeness of a broader task population or account for outcome changes that were not observed within the finite set of reruns. In particular, resampling task rows whose recorded variances are all zero returns a zero-width interval. This is a property of the observed records and resampling procedure, not proof that future variance is zero. The exact binomial intervals for the proportion of variable tasks in Table A1 are a separate summary and are not task-bootstrap variance intervals.

Appendix A.3. Fixed-Task Rerun Budget

For configurations A and B on the same fixed N tasks, the planning approximation is

$$Var\left(\widehat{\Delta }\right)\approx {\displaystyle \frac{w_A+w_B}{NR}}.$$

(A2)

As in the main text, this expression assumes independent streams between arms and excludes covariance across tasks and replicates. It is not the variance formula for a general paired replay comparison. Shared randomness or dependence across executions requires a variance analysis that retains the corresponding covariance.

For the symmetric displays, $w_A=w_B=w$. With $z_p$ denoting the standard-normal quantile at probability p, the smallest positive integer R satisfies

$$\delta \ge \left(z_{1-\alpha /2}+z_{0.8}\right)\sqrt{{\displaystyle \frac{2w}{NR}}},$$

(A3)

with $\alpha =0.05$ and illustrative $\delta =0.02$. The target difference is positive and N is nonzero. When the observed w is zero, the point calculation returns the minimum of one run; the pilot variance itself was estimated using 20 repeats, not one. This calculation omits uncertainty in the variance estimate, task-population uncertainty, and systematic judge disagreement. The tabulated budgets quantify replication under this approximation; validating their achieved power requires a separate assessment.

Appendix A.4. Replay Coupling

The shared-recording condition in Proposition 1 fixes the response associated with each normalized generation request. When both configurations issue the same request multiset and retrieve those responses, the contrast concerns their downstream behavior under that recording. Conditional unbiasedness refers to that fixed-recording comparison, not to a mean over live provider responses.

An implementation must make the request normalization and response association unambiguous. In particular, a request-keyed fixed response should not be conflated with a recording that supplies different responses to repeated occurrences of an identical request. For an occurrence-based recording, the pairing rule must also specify the occurrence correspondence. The reported software checks include replay misses and call-order violations. Changes to retries, routing, tools, or request content require a coupling argument for the changed structure or fresh independent generation; a cache hit alone does not establish validity for an adaptive trajectory.

Appendix C.1. Pipeline and Task Selection

The primary stack used a dated gpt-4o-mini snapshot to generate Python code. The local differential verifier extracted a fenced block when present, executed the candidate with the task preamble, and compared its outputs with the canonical EvalPlus solution on up to 120 base and augmented inputs. An initial failure triggered at most one optional retry. Candidate execution had a 10-second wall-clock limit, and a timeout was scored as failure.

The HumanEval+ arm contained all 164 tasks returned by the loader. The MBPP+ arm contained the first 200 tasks in loader order. Each condition used 20 replicates per task: fresh generation at temperature 0.7, fresh generation at temperature zero, and content-addressed replay through the same local verifier. No LLM judge was used on this executable path.

Appendix C.2. Complete Primary Results

Table A1 reports the scores, mean within-task variances, and counts of tasks that changed binary outcome at least once across 20 replicates. The main paper uses the second complete live HumanEval+ batch; the first is retained for temporal sensitivity. Table A2 gives the jointly resampled contrasts for the conditions used in the main text.

Table A1. Primary Scores and Within-Task Variation. Score is the mean binary outcome across tasks and replicates. Variable tasks changed outcome at least once in 20 repetitions. The final column reports exact intervals for the proportion of variable tasks, not for w. The second live HumanEval+ batch supplies the main endpoint.

Benchmark	Condition	Score	w	Variable Tasks	Task-Rate 95% CI
HumanEval+	live batch 1	0.6948	0.03208	37/164	[0.164, 0.297]
HumanEval+	live batch 2	0.6951	0.03013	34/164	[0.148, 0.277]
HumanEval+	temperature 0	0.6787	0.01329	13/164	[0.043, 0.132]
HumanEval+	replay	0.7012	0.00000	0/164	[0.000, 0.022]
MBPP+	live	0.7698	0.01422	18/200	[0.054, 0.139]
MBPP+	temperature 0	0.7575	0.00750	9/200	[0.021, 0.084]
MBPP+	replay	0.7813	0.00099	1/200	[0.0001, 0.0275]

Table A1. Primary Scores and Within-Task Variation. Score is the mean binary outcome across tasks and replicates. Variable tasks changed outcome at least once in 20 repetitions. The final column reports exact intervals for the proportion of variable tasks, not for w. The second live HumanEval+ batch supplies the main endpoint.

Benchmark	Condition	Score	w	Variable Tasks	Task-Rate 95% CI
HumanEval+	live batch 1	0.6948	0.03208	37/164	[0.164, 0.297]
HumanEval+	live batch 2	0.6951	0.03013	34/164	[0.148, 0.277]
HumanEval+	temperature 0	0.6787	0.01329	13/164	[0.043, 0.132]
HumanEval+	replay	0.7012	0.00000	0/164	[0.000, 0.022]
MBPP+	live	0.7698	0.01422	18/200	[0.054, 0.139]
MBPP+	temperature 0	0.7575	0.00750	9/200	[0.021, 0.084]
MBPP+	replay	0.7813	0.00099	1/200	[0.0001, 0.0275]

Table A2. Primary Operating-Condition Contrasts. Intervals come from joint task bootstraps of the conditions used in the main paper. These are differences between measured conditions, not an additive allocation of variance to physical sources.

Contrast	HumanEval+ Estimate [95% CI]	MBPP+ Estimate [95% CI]
live − replay	0.03013 [0.01998, 0.04146]	0.01324 [0.00708, 0.02025]
live − temperature 0	0.01685 [0.00878, 0.02579]	0.00672 [0.00216, 0.01195]

Table A2. Primary Operating-Condition Contrasts. Intervals come from joint task bootstraps of the conditions used in the main paper. These are differences between measured conditions, not an additive allocation of variance to physical sources.

Contrast	HumanEval+ Estimate [95% CI]	MBPP+ Estimate [95% CI]
live − replay	0.03013 [0.01998, 0.04146]	0.01324 [0.00708, 0.02025]
live − temperature 0	0.01685 [0.00878, 0.02579]	0.00672 [0.00216, 0.01195]

Appendix C.3. Replay Residual and Temporal Sensitivity

All primary MBPP+ replay variance was concentrated in Mbpp/271. Removing that task set replay variance to zero and moved the live-minus-replay contrast from 0.01324 to 0.01318. The available record did not distinguish runtime, timeout, and candidate-code explanations for the residual. The second stack did not reproduce it, so the observation is not treated as an invariant property of the task.

The two complete live HumanEval+ batches were collected approximately 15 hours apart. Their within-task variances were 0.03208 and 0.03013, with a jointly resampled difference of 0.00194 and a 95% interval of [−0.00305, 0.00709]. This comparison did not resolve a difference between the batches over that window. It does not establish equivalence or long-term stationarity.

Appendix D.1. Design and Valid-Run Lineage

The judge follow-up held generation fixed at the final candidate associated with each of the 200 primary MBPP+ replay tasks. A dated hosted LLM judge evaluated every candidate 20 times at temperature 0.7 and 20 times at temperature 0. The prompt included the task description, candidate code, and original example assertions, and requested a pass/fail verdict and confidence. The primary verifier supplied the executable reference using capped EvalPlus inputs not shown to the judge.

The valid measurement was a frozen successor to an invalid formatting run. In the predecessor, 78% of replies reached the token limit before a verdict could be parsed. The successor placed the verdict first and increased the output allowance without changing candidate inputs. Only the successor contributes scientific results; the predecessor remains part of the study lineage.

Appendix D.2. Verdict Variability and Reference Disagreement

At temperature 0.7, judge verdict variance was 0.0073 and 12 of 200 tasks changed verdict across repetitions. At temperature zero, the verdict for every task was unchanged across all 20 repeats, covering 4,000 calls. Table A3 reports the variances and fixed-task point budgets, together with their reported intervals.

Table A3. Repeated Judging of Frozen Candidates. The study uses one primary MBPP+ replay candidate per task. Variance intervals are 95% task-bootstrap intervals. Runs are per-arm point budgets for $\delta =0.02$, with the budget intervals reported in the study summary. Mean confidence is the stated confidence value, not a measurement of calibration.

Judge Condition	Verdict w [95% CI]	Variable Tasks	Mean Confidence	Runs [Interval]
temperature 0.7	0.0073 [0.0030, 0.0127]	12/200	94.4	2 [1, 3]
temperature 0	0.0000 [0.0000, 0.0000]	0/200	94.4	1 [1, 1]

Table A3. Repeated Judging of Frozen Candidates. The study uses one primary MBPP+ replay candidate per task. Variance intervals are 95% task-bootstrap intervals. Runs are per-arm point budgets for $\delta =0.02$, with the budget intervals reported in the study summary. Mean confidence is the stated confidence value, not a measurement of calibration.

Judge Condition	Verdict w [95% CI]	Variable Tasks	Mean Confidence	Runs [Interval]
temperature 0.7	0.0073 [0.0030, 0.0127]	12/200	94.4	2 [1, 3]
temperature 0	0.0000 [0.0000, 0.0000]	0/200	94.4	1 [1, 1]

Majority verdicts disagreed with the executable reference on 47 of 200 tasks, corresponding to 23.5% and a task-bootstrap interval of [17.5%, 29.5%]. Relative to those tests, the disagreements comprised 18 false accepts and 29 false rejects. The disagreement set and mean stated confidence were identical at both judge temperatures. Removing Mbpp/271 did not change the qualitative conclusions.

The comparison therefore identifies stable disagreement as distinct from stochastic verdict variation. It is not a universal judge-accuracy estimate: the result is conditional on the frozen candidates, prompt, model snapshot, and asymmetric access to the test information. The descriptive comparison with live-generation variance does not identify an additive judge contribution to that variance.

Appendix E.1. Design and Scope

The second hosted stack evaluated the same 164 HumanEval+ tasks and the same first 200 MBPP+ tasks with a dated claude-haiku snapshot. The generation design retained the closed-book prompt, one-retry structure, 120-input cap, and 20 replicates per condition. Unlike the primary stack, execution used the official EvalPlus checker, ground-truth-relative time limits with a 30-second per-input floor, and special handling for resource-heavy checks. This design tests transport across a second model-and-verifier stack rather than a provider-only effect.

The valid run contained 17,960 provider responses with ex-ante task identifiers and complete request–response linkage. Twenty-three responses ended at the token limit and remained in the analysis. Replay made no provider call and re-executed the recorded candidates through the same verifier. These details define which responses and executions contributed to the reported measurements.

Appendix E.2. Complete Second-Stack Results

Table A4 reports the full outcome summary. Live-minus-replay variance contrasts were 0.0235 ([0.0136, 0.0341]) on HumanEval+ and 0.0134 ([0.0068, 0.0209]) on MBPP+. Live-minus-temperature-zero contrasts were 0.0203 ([0.0110, 0.0303]) and 0.0115 ([0.0048, 0.0191]), respectively. Replay outcomes were unchanged for every task over the observed repeats.

Table A4. Second-Stack Scores, Variance, and Budgets. Variance intervals are 95% task-bootstrap intervals. Runs are per-arm fixed-task point budgets for $\delta =0.02$, with the intervals reported in the study summary. Different pass rates and verifier implementations make cross-stack magnitude comparisons descriptive; the repeated finding is the ordering live > temperature zero > replay.

Benchmark	Condition	Score	w [95% CI]	Variable Tasks	Runs [Interval]
HumanEval+	live	0.8524	0.0235 [0.0136, 0.0341]	19/164	6 [4, 9]
HumanEval+	temperature 0	0.8247	0.0032 [0.0000, 0.0079]	3/164	1 [1, 2]
HumanEval+	replay	0.8354	0.0000 [0.0000, 0.0000]	0/164	1 [1, 1]
MBPP+	live	0.8295	0.0134 [0.0068, 0.0209]	15/200	3 [2, 5]
MBPP+	temperature 0	0.8093	0.0018 [0.0000, 0.0047]	3/200	1 [1, 1]
MBPP+	replay	0.8350	0.0000 [0.0000, 0.0000]	0/200	1 [1, 1]

Table A4. Second-Stack Scores, Variance, and Budgets. Variance intervals are 95% task-bootstrap intervals. Runs are per-arm fixed-task point budgets for $\delta =0.02$, with the intervals reported in the study summary. Different pass rates and verifier implementations make cross-stack magnitude comparisons descriptive; the repeated finding is the ordering live > temperature zero > replay.

Benchmark	Condition	Score	w [95% CI]	Variable Tasks	Runs [Interval]
HumanEval+	live	0.8524	0.0235 [0.0136, 0.0341]	19/164	6 [4, 9]
HumanEval+	temperature 0	0.8247	0.0032 [0.0000, 0.0079]	3/164	1 [1, 2]
HumanEval+	replay	0.8354	0.0000 [0.0000, 0.0000]	0/164	1 [1, 1]
MBPP+	live	0.8295	0.0134 [0.0068, 0.0209]	15/200	3 [2, 5]
MBPP+	temperature 0	0.8093	0.0018 [0.0000, 0.0047]	3/200	1 [1, 1]
MBPP+	replay	0.8350	0.0000 [0.0000, 0.0000]	0/200	1 [1, 1]

Temperature-zero variance was smaller than in the primary stack: 0.0032 versus 0.0133 on HumanEval+ and 0.0018 versus 0.0075 on MBPP+. Because both model and verifier changed, these differences support a stack-specific interpretation. They do not show that a particular provider alone caused the reduction, and the absence of observed replay changes does not prove future determinism.

Appendix G.1. Protocol Chronology and Invalid Predecessors

The primary analysis plan was encoded in the execution notebook, but the release does not independently establish its freeze time. We therefore treat that plan as author-controlled. The judge and cross-stack follow-ups used frozen successor protocols with provider-timestamped receipts preceding their valid calls. Those receipts support chronology; they do not establish independent public preregistration or attestation of request bodies.

Both follow-ups retained invalid predecessors. The judge predecessor completed its calls, but most replies were truncated before a verdict could be parsed. The first cross-stack run failed during the first measurement task. Although its abort report stated that no outcome existed, a subsequent audit identified 19 completed replicates for HumanEval/0. The successor was frozen after those partial records existed. Its changes addressed the execution failure and did not retune endpoints, but the successor is not described as fully outcome-blind. Neither invalid predecessor contributes to the scientific endpoints.

Appendix G.2. Scope of the Reconstruction Checks

Reconstruction checked registered file digests, dataset hashes, task identities, prompt hashes, uniqueness of request and message identifiers, candidate-to-judge mapping, cell outcomes, bootstrap endpoints, and rerun budgets. Every valid provider call carries an ex-ante task identifier and a matching raw response. These checks concern the relationship between retained records and reported analyses; they are distinct from independent operation of the experiments.

The prepared ground-truth cache for the second stack is represented by its recorded hash rather than included in the artifact because of its size. Endpoint reconstruction instead uses the retained cell records. Reconstructing a statistic from those records does not independently establish that the original execution environment, reference cache, or provider response was correct; those are separate parts of the experimental evidence.

Appendix H.1. Reproducibility, Boundaries, and Use

The accompanying repository contains primary ledgers, valid and invalid follow-up lineage, raw provider responses, task and call manifests, frozen analysis programs, canonical tables, figure scripts, and verification outputs. Offline reconstruction from records uses Python, NumPy, pandas, and SciPy. Full experimental re-execution requires paid provider access and cannot guarantee bit-identical responses from mutable external services.

The supported empirical scope comprises two hosted model-and-verifier stacks, two code benchmarks, one optional-retry pipeline, and one frozen-candidate judge prompt. The measurements do not cover long-horizon agents, natural repository workloads, human reference judgments, live-stream judge variability, or feedback between judge decisions and adaptive agent behavior. Replay can support a valid fixed-recording comparison within its declared coupling, but it does not replace fresh runs when the claim concerns reliability under new generation.

Appendix H.2. Code and Data Availability

Analysis code, verification scripts, and checksums for the experiments reported in this work are available in a public repository at github.com/Daice-labs/noise-floor. The complete experimental records (task-level records, raw provider responses, endpoint files, and registration metadata) are retained by the authors and are available from the corresponding author on request.

Appendix H.3. Disclosure: Use of Generative AI

AI-assisted tools were used for manuscript restructuring and language editing, implementation support, experiment orchestration, figure preparation, artifact review, and package assembly. Where an AI system formed part of an experimental procedure, its role is described in the corresponding methods. The authors retain responsibility for the scientific questions, protocols, source code, data provenance, analyses, citations, numerical claims, and final manuscript. AI systems are not authors.