Reassessing One-Round Test-Time Refinement for Code Generation

Test-time refinement aims to improve generated programs through additional inference, but its value after an initial candidate has been produced remains unclear. We conduct a controlled evaluation of one-round Self-Refine and Self-Debug across seven models and three Python code-generation benchmarks. For each model and task, both methods refine the same initial candidate, allowing us to measure refinement gain without variation in initial generation. Self-Debug produced positive refinement gain in 16 of the 21 model-benchmark combinations and no change in the remaining five, with gains reaching +9.57 percentage points. In contrast, Self-Refine reduced correctness in 16 combinations, with losses of up to 8.07 percentage points, and produced positive gains in only four combinations. Repair-regression analysis showed that Self-Debug rarely damaged initially correct candidates, whereas regressions under Self-Refine frequently outweighed its repairs. Further analysis showed that execution feedback was beneficial only when models acted on diagnostic failures and produced effective revisions. Resource analysis showed that Self-Refine incurred greater token overhead despite generally reducing correctness, making its additional inference difficult to justify. Self-Debug provided a more favorable gain-overhead balance, but its monetary efficiency varied with model pricing, and its additional gains may offer limited value when initial generation is already sufficiently accurate. These results show that one-round refinement is not inherently beneficial and should be applied only when its expected gain justifies the additional computation and cost.