When Scores Do Not Steer: Auditing Process Credit in Verifier-Guided Repair

1. Introduction

Native verifiers give coding agents feedback that ordinary tests cannot. A Lean proof must type-check, and a Dafny program must discharge its verification conditions. This has made theorem proving and verified code generation natural settings for search at inference time [7,9,11,15,17]. It also invites a simple idea: give partial credit to an intermediate repair that removes errors, reaches a new proof state, or avoids a repeated failure, then continue from the highest-scoring branch.

The score itself is not enough. It must pass two tests. Decision authority asks whether changing or deleting the score changes which branch the controller keeps. Continuation value asks whether the preferred branch is more likely to lead to a verified result. These are different failure modes. A score can be informative but placed too late in a lexicographic ordering to affect behavior. It can also affect behavior while tracking visible local progress rather than future success.

This paper studies both failures in BranchCredit, a test-time proof-repair controller around frozen hosted language models. At each round, BranchCredit generates two sibling repairs, checks both, and commits one as the next incumbent. The scalar credit signal updates task-local strategy scores, but it is also the eighth field of a nine-field commitment key. Native verification and an integrity contract come first, followed by five diagnostic fields. The architecture therefore raises a basic question: does the score that motivates the method ever decide anything?

We answer that question before interpreting end-to-end accuracy. First, we remove credit while keeping the same candidates, branch width, budget, and all other ranking fields. Second, we promote credit above every soft diagnostic field while preserving verification and integrity as hard gates. Third, at a stratified sample of disagreements, we force the credit-preferred branch and generate new descendants. Finally, we compare terminal completion with a branching-matched credit-free controller and with sequential verifier-guided repair. Source theorems or program families, not repeated model calls, are the independent units.

The answer is clear at the mechanism level. At its original rank, credit changes no observed Lean commitment. After promotion, it changes one quarter of multi-candidate selections but no final outcomes on the recorded candidates. Live descendants show that some preferences are consequential, but their direction is mixed. The terminal comparisons agree with that diagnosis: no frame establishes a reliable advantage for the credit signal.

Our contributions are:

• a simple proposition and complete decision audit for identifying when a late process score is unable to affect a lexicographic controller;
• a soundness-preserving promotion test that separates branch changes, path changes, and final verification changes;
• matched Lean and Dafny comparisons that isolate scalar credit from branch width and report uncertainty at the source level; and
• a held-out analysis showing that the available first-call features do not support a selective credit policy.

We do not claim that process credit is generally useless, that branching is harmful, or that the compared controllers are equivalent. The narrower finding is that this credit signal has no authority at its designed rank and no demonstrated terminal value after it is given authority.

2. Related Work

Process supervision and inference-time search.

Process supervision gives feedback to intermediate reasoning steps rather than only final answers [1]. Tree search and test-time scaling use such signals to decide where additional inference should be spent [3,5]. BranchCredit studies a smaller but common engineering pattern: complete-file repair using frozen models, native checking after each call, and task-local scores that are discarded after the task. Its strategy selection uses upper-confidence bookkeeping related to classical bandit methods [19]; our question is whether that bookkeeping reaches the final commitment decision.

Theorem proving and verified code.

LeanDojo, DeepSeek-Prover, and LeanProgress use proof-assistant interaction, search, retrieval, or progress prediction to improve theorem proving [7,9,11]. AxDafny and Meta-Step Decoding study agentic generation and verifier-guided translation in Dafny [15,17]. These systems make verifier feedback central to search. Our contribution is not a new prover. It is an audit of the controller that turns several verifier and process signals into one branch choice.

Verifier quality.

A native checker certifies only the contract it is given. It may not capture the full intent of a task, and a reward can exploit omissions in the verifier or integrity policy [13]. For that reason, BranchCredit keeps verification and integrity above all soft scores. The problem studied here appears inside that soundness-first design: additional diagnostic fields can also make the nominal process score irrelevant.

3. Methods

3.1. Repair Loop

A task is $\mathcal{T}=(x_0,D,P,V,B)$, where $x_0$ is a failing source file, D is the required declaration set, P is a protected-source contract, V is the native verifier, and B is the call budget. The controller chooses among six repair strategies, including local repair, multi-error repair, proof reconstruction, structural reasoning, rewriting, and restart. It requests two sibling files from the same incumbent and checks both before committing the next state. With six model calls, width two gives three branch rounds.

The integrity check rejects changed theorem signatures, protected-source edits, new axioms or unsafe declarations, placeholders, missing declarations, and disallowed imports or commands. A candidate is counted as solved only when both integrity and native verification pass.

Scalar credit combines verified success, diagnostic progress, removed errors and placeholders, diagnostic novelty, repeated-failure penalties, and checker or API cost. It updates the controller’s task-local strategy scores. The important detail is where it appears in the commitment key.

3.2. Three Commitment Rules

For candidate x, define

$$H\left(x\right)=(v,i,p,-e,-h,n,-r),$$

where v is verified success, i is integrity, p is diagnostic progress, e is the error count, h is the placeholder count, n is diagnostic novelty, and r is the repeated-failure count. Let $c\left(x\right)$ be scalar credit and $j\left(x\right)$ the call index. We compare:

$$\begin{array}{cc}\hfill K_{\mathrm{BC}}\left(x\right)& =\left(H\right(x),c(x),-j(x\left)\right),\hfill \end{array}$$

(1)

$$\begin{array}{cc}\hfill K_{\mathrm{CF}}\left(x\right)& =\left(H\right(x),-j(x\left)\right),\hfill \end{array}$$

(2)

$$\begin{array}{cc}\hfill K_{\mathrm{promoted}}\left(x\right)& =(v,i,c,p,-e,-h,n,-r,-j).\hfill \end{array}$$

(3)

CreditFree removes only scalar credit. It keeps the same parent, candidates, branch width, calls, checks, stopping rule, and non-credit fields. The promoted rule moves credit directly below verification and integrity. It is allowed to choose among soundness-equivalent unverified candidates, but it cannot override a verified candidate or one that passes integrity.

3.3. When a Score Cannot Matter

Proposition 1  

(Decision authority under lexicographic commitment). Fix a candidate pool, deterministic final tie-breaking, and a ranking $K_c=(H,c,t)$, where every field of H precedes process credit c. Removing c can change the selected candidate only if at least two candidates tie for the maximum value of H and c changes the winner selected by $(H,t)$. If H has a unique maximizer at every reachable decision, removing c leaves the recorded trajectory unchanged. If credit acts only among candidates with the same verification status, it cannot change the immediate binary verification result.

Proof. 

Lexicographic comparison consults c only after equality on every field of H. A unique maximizer of H is therefore selected with or without c. Applying the same argument at each reachable decision gives trajectory invariance for a fixed call schedule and candidate stream. Selecting among candidates with identical verification status cannot change that status at the current decision.    □

The proposition gives a direct audit. We count where credit has a different preference, where the ranking actually reaches credit, where the selected branch changes, and where final verification changes. Keeping these quantities separate prevents score variation from being mistaken for control.

3.4. Evaluation Protocol

3.4.1. Tasks and Experimental Frames

The controlled Lean tasks begin with verified source theorems. We replace each proof body with a unique invalid term while preserving its signature. Every reference file must verify and every mutation must fail before evaluation. The main mechanism set has 12 independent source families. Four hosted model families and three prompt seeds produce 144 BranchCredit episodes and 432 branch rounds. Of those rounds, 216 contain multiple checked candidates and can support a selection comparison.

We use four additional settings:

• a harder Lean set with 12 source families and 48 episodes per controller;
• a separate near-ceiling Lean set with 12 source families and 72 episodes per controller;
• an external Lean set with 12 source families and 96 episodes each for BranchCredit and VerifierGuided; and
• two 12-program Dafny sets, one arithmetic and one invariant-heavy, with 48 episodes per controller.

The controlled comparisons use six model calls, six native checks, the same output limit, and matched prompts and decoding settings. We do not give one controller an early-stop budget advantage. The near-ceiling CreditFree arm was executed separately under the same budget, so we treat it as a matched comparison rather than part of one simultaneous multi-controller run.

Models were accessed through AWS Bedrock during July–August 2026: Claude Sonnet 5, Claude Haiku 4.5, Anthropic Fable 5, GPT-5.6 Terra, GPT-5.6 Luna, Qwen3 Coder 480B, and DeepSeek V3.2. The Anthropic, Qwen, and DeepSeek models used Bedrock Runtime; the GPT models used Bedrock Mantle. Controlled frames used matched prompt construction and an $8,192$-token output ceiling. Sampling controls were interface-specific, so we do not report a universal temperature or top-p value. Labels 17, 43, and 101 index experimental conditions. Appendix C gives the model subset for each frame.

For a natural-data check, we mine 27 paired proof-maintenance units from batteries (24 units) and aesop (3 units), both Apache-2.0. A unit is admitted when the evaluated revision compiles cleanly and the mutation produces a valid evaluation input that fails native Lean verification. Both arms use Claude Sonnet 5 through Bedrock Runtime in us-west-2 during August 2026, with a matched six-call budget per arm. Only frontier selection differs between BranchCredit and CreditFree. Exact evaluated revisions are retained in the experimental records.

A secondary Mathlib-30 frame evaluates 30 proof-repair units with Claude Sonnet 5 and Claude Haiku 4.5 under three-call budgets. It compares the branching BranchCredit policy with sequential VerifierGuided. Because that comparison changes branching, parent selection, and feedback timing in addition to credit, we report it only as a policy-level stress test, not as an isolation of scalar credit.

3.4.2. Interventions and Outcomes

The original-rank and promoted-rank analyses replay the same checked candidate pools. This removes generation noise and asks a deterministic question: would a different ranking choose a different recorded state? The live replay then tests the downstream question. A point is eligible when both siblings were generated and checked. We sample 20 of 80 eligible points across all 12 source families, force the credit-preferred sibling when it differs from the original choice, and regenerate the remaining descendants.

The main mechanism quantities are:

1.

multi-candidate decisions;

2.

latent credit disagreements;

3.

selected-branch changes;

4.

changed episode paths; and

5.

changed final verification outcomes.

The main terminal outcome is verified completion. CreditFree is the primary baseline because it isolates the scalar credit field. VerifierGuided is a sequential baseline that generates one repair per call and returns to the best verifier-ranked state.

3.4.3. Inference and Prediction

The source theorem or program family is the independent unit. We aggregate model and prompt repetitions within a source, use source-cluster bootstrap intervals, and report exact source-level sign-flip tests. Paired McNemar tests at the episode level are descriptive. We never read a failure to reject as evidence of equivalence [21,23,25].

The predictor analysis uses 144 episodes over the same 12 Lean source families, two models, three seeds, and two controllers with comparable intermediate states. Eight source families form the development split and four size-stratified families are held out. Features use first-call information only: error reduction, diagnostic progress, scalar credit, and proposal entropy. We report AUC on all held-out episodes and again after excluding episodes already verified on the first call.

3.4.3.1. Generative AI use.

Generative AI tools were used to assist with language editing, LaTeX preparation, and analysis-code development. The author reviewed and verified the manuscript, reported results, references, and supporting files and takes responsibility for the final content.

4. Results

4.1. The Causal Funnel

Figure 1 shows the complete mechanism result. At the original rank, fields one through four settle 21 of 216 multi-candidate commitments. Another 142 decisions compare a pool containing a verified candidate. The remaining 53 are the meaningful latent-disagreement set: the candidates are unverified and scalar credit prefers a different candidate from the credit-free choice. Yet placeholder count, novelty, or repeated-failure count settles all 53 before the ranking reaches credit. Removing credit changes 0/216 selected branches and 0/144 final outcomes.

Promotion rules out a purely architectural excuse. With credit at field three, 54/216 selections change. All 54 are between integrity-passing, unverified candidates. The committed path changes in 36/144 episodes, but BranchCredit, CreditFree, and promoted credit each solve 81/144 tasks. No episode changes final verification. Credit now has authority, but the recorded preference does not improve the outcome.

Live replay weakens the fixed-candidate assumption. Among the 20 sampled eligible points, five have a different credit preference. Forcing that branch and regenerating descendants converts two failures to successes, one success to a failure, and leaves two unchanged. Conditional on a disagreement, 3/5 outcomes are consequential, with Wilson 95% interval $[0.231,0.882]$. The net success difference is $+1/5$, and the exact sign test is $p=1.0$. The sample is too small to claim benefit, but it shows that credit is not always harmless once it is allowed to control the path.

4.2. Terminal Completion

Figure 2 reports the matched terminal comparisons. The harder Lean set is the most favorable result for BranchCredit: 29/48 tasks verify, compared with 27/48 for CreditFree. The difference is $+4.2$ percentage points, the source-cluster interval is $[0.0,10.4]$, and the exact source-level test is $p=0.50$. This is a positive point estimate, not evidence of superiority.

The separate near-ceiling Lean set gives one additional solve for BranchCredit: 60/72 versus 59/72 for both CreditFree and VerifierGuided, with $p=1.0$. The external Lean set runs in the other direction: BranchCredit solves 58/96 and VerifierGuided solves 67/96, a $-9.4$-point difference with interval $[-19.8,0.0]$ and $p=0.156$.

The Dafny replication reaches a similar conclusion through a different verifier. On arithmetic tasks, BranchCredit solves 46/48 and CreditFree solves 45/48. On the invariant-heavy set, both solve 48/48. The mechanism differs across the two Dafny sets. Credit ablation changes 0/144 arithmetic commitments, while in the invariant-heavy set scalar credit frequently decides among candidates that are already verified. In both cases, there is little room for scalar credit to change binary completion.

On the 27 natural proof-maintenance units, BranchCredit solves 14 and CreditFree solves 13. There is one discordant unit and exact McNemar $p=1.0$. BranchCredit uses 94 model calls and 397k output tokens, compared with 103 calls and 440k tokens for CreditFree. It also reaches fewer distinct checked frontier states (3 versus 7) and records fewer recoveries after a wrong branch (2 versus 3). These process counts occur after the selected trajectories diverge, so we report them as description, not as a causal efficiency result.

Secondary Mathlib policy comparison.

On Mathlib-30 with Claude Sonnet 5, BranchCredit solves 27/30 units and sequential VerifierGuided solves 30/30. With Claude Haiku 4.5, the corresponding results are 6/30 and 7/30. These comparisons do not isolate scalar credit from the other policy differences, so they provide external stress-test evidence rather than a causal credit estimate.

4.3. Held-Out Prediction

A selective controller would use credit only when early evidence predicts a good continuation. The development split makes that idea look promising: error reduction, diagnostic progress, and scalar credit each have AUC above 0.93. Figure 3 shows that the result does not transfer. Held-out AUC falls to 0.581–0.643 for those features and to 0.521 for proposal entropy. After first-call successes are removed, AUC is 0.49–0.61. No held-out source-level test rejects chance.

5. Discussion

Audit authority before accuracy.

A score distribution or correlation does not show that a controller uses the score. The cheapest decisive test is an ablation on the actual commitment rule. Here that test gives an exact zero: credit is never the deciding field at its designed rank.

Keep hard gates, shorten the soft prefix.

Verification and integrity should remain above a heuristic score. Within those gates, however, a long diagnostic prefix can silently make the score decorative. A controller paper should publish the full ordering and report how often each field decides.

Local progress is not future value.

Promoted credit changes one quarter of selections but no recorded final outcomes. Live replay gives both gains and harm. The score tracks visible movement, but it is not calibrated to the probability that the selected state admits a successful continuation.

Branching and credit need separate controls.

BranchCredit and a sequential verifier-guided controller differ in branch width, parent selection, and budget allocation. Only BranchCredit versus CreditFree isolates the credit signal. A point estimate against a sequential baseline cannot be attributed to credit alone.

A stronger successor is testable.

The next version should learn or calibrate descendant success on development sources, validate on untouched source families, and compare selective credit with always-credit and never-credit controls. It should also declare an authority target, such as the fraction of soundness-equivalent decisions the value estimate is allowed to override.

6. Limitations

The original-rank audit is exact for the recorded candidates. The promoted analysis also uses those candidates, so it does not regenerate every trajectory after every changed choice. The live intervention addresses this issue at only 20 points, including five actual disagreements. That is enough to demonstrate mixed consequences, but not enough to estimate a stable improvement or harm rate.

The controlled Lean tasks come from 12 source families and use synthetic proof-body mutations. The natural evaluation contains only 27 units from two repositories and is not a representative sample of all proof maintenance. Neither setting covers large multi-file changes, specification repair, or long interactive developments. Dafny adds a second verifier, but both Dafny sets are near ceiling.

Hosted model calls may change as providers update models and serving systems. We match call and verification budgets and aggregate at source level, but exact regeneration may still drift. With 12 independent sources, the source-level tests have limited power. We therefore say that the experiments do not establish an advantage; we do not claim equivalence.

Finally, scalar credit is one engineered composite. The paper does not show that all process-value functions fail. A value model trained directly on descendant success, or a controller with a different authority allocation, may behave differently.

7. Reproducibility and Impact

The author retains the experimental records, controller configurations, and analysis scripts. Hosted model generation may vary across provider runs. Public access to the supporting research data is restricted as stated in the Data Availability Statement.

Controlled comparisons use Lean 4.33.0 (commit d8b18978) or Dafny 4.9.0 with Z3 4.15.4. Evaluation uses AWS EC2 compute instances running Ubuntu 22.04 LTS. The mechanism audit contains 144 episodes, 432 branch rounds, and 216 multi-candidate decision states under matched six-call budgets and $8,192$-token ceilings. The harder Lean, near-ceiling, external, Dafny, and natural frames also use six-call budgets; Mathlib-30 uses three-call budgets. The retained experimental records include per-frame model rosters, hardware specifications, actual call counts, wall-clock times, token totals, and accounting for preliminary exploratory runs.

Verifier-grounded control can improve auditability because accepted outputs and rejected candidates can be checked and preserved. It can also create false confidence when the verifier or integrity contract misses important behavior. Process scores add another risk: a precise number may be mistaken for a calibrated probability of success. Systems using these methods should expose their commitment rule, keep human review for consequential use, and avoid presenting diagnostic movement as correctness.

Lean 4.33.0, batteries, aesop, and mathlib4 are used under Apache-2.0; Dafny 4.9.0 and Z3 4.15.4 are used under MIT licenses. The retained experimental records identify exact repository revisions, the specific ITPEval source and its redistribution terms, and the provider routes for hosted models. No model weights are redistributed.

8. Conclusions

BranchCredit computes process credit, but its original commitment rule never lets that credit decide among the observed Lean candidates. Moving the score directly below verification and integrity creates substantial branch and path changes, yet no final change on the recorded candidates. Live descendants confirm that credit-preferred branches can matter, while also showing that the preference is not reliably directional.

The terminal evidence is consistent with the mechanism. It contains small positive, null, and negative differences, but no source-level result that establishes a credit advantage. A process score should therefore be evaluated as part of a controller, not in isolation. It needs measurable authority, a held-out relationship with continuation success, and a branching-matched signal-free baseline. Without those checks, extra scoring machinery can look meaningful while doing little useful work.