Strategic Verification for Long-Running LLM Agents

1. Introduction

Large language models are increasingly evolving from single-turn text generators into agents that interact with external environments over extended periods. These systems repeatedly observe changing conditions, make decisions, invoke tools and application programming interfaces, and incorporate resulting observations into subsequent actions. They can search the web, execute code, update files and databases, operate software applications, retrieve or write memories, and communicate with users or other agents. Unlike an isolated model response, these actions may consume substantial resources, modify persistent state, or produce irreversible consequences. The reliability of an agent therefore depends not only on whether it eventually reaches the correct outcome, but also on the validity, efficiency, safety, and auditability of the trajectory used to reach it [1,2,3,4].

This paper uses long-running agent to denote an agent that operates across multiple episodes or a continuing task stream while retaining memories, reusing acquired skills, encountering changing tools or environments, or adapting after deployment. This operational definition distinguishes cross-episode persistence from the related concept of a long-horizon agent, whose interdependent actions occur within a single episode. Long-running operation introduces both within-trajectory credit assignment and cross-trajectory reliability. An incorrect tool call may compromise a later artifact, an unsupported memory may influence many subsequent decisions, and an inadequately verified skill may propagate failures across otherwise unrelated tasks. Accordingly, improvement cannot be measured solely through aggregate task success or parameter updates; it must also account for persistent state, behavioral regression, memory contamination, procedural compliance, and the reliability of reusable capabilities [5].

Reinforcement learning from verifiable rewards (RLVR) offers a promising foundation for training such agents. RLVR replaces or supplements subjective preference judgments with feedback grounded in independently checkable evidence, including exact answers, executable tests, symbolic constraints, valid tool responses, inspected environment states, and source-supported factual units. These signals are often more reproducible, scalable, and auditable than unconstrained human-preference models or holistic language-model judgments. In structured environments, the same verifier can evaluate many trajectories without requiring a new human label for every interaction, making RLVR attractive for training reasoning, coding, search, and tool-using systems [6,7,8].

However, conventional outcome-only RLVR evaluates an agent only after its trajectory terminates. A terminal verifier can establish that a task was completed, but it cannot determine which intermediate actions caused success or failure. It assigns the same positive reward to an efficient and compliant trajectory as to one containing unnecessary searches, invalid actions, unauthorized operations, or exploitation of the evaluation procedure. It may therefore reinforce an agent that reaches the correct state through an unacceptable path. Conversely, a failed trajectory may contain useful evidence retrieval, valid constraint-preserving actions, or correctly completed subgoals, yet every transition receives the same zero terminal reward. This problem becomes particularly severe in group-relative optimization when all sampled trajectories succeed or all fail, causing binary outcome rewards to provide little or no within-group discrimination [9,10,11].

Dense process verification can provide more informative local credit, but verifying every transition is neither universally feasible nor necessarily desirable. Different checks consume different combinations of model tokens, computation, latency, environment interactions, and human attention. Deterministic rules may be inexpensive and precise but limited to narrow conditions; executable or state-based verifiers may require controlled environment access; semantic judges may be costly and poorly calibrated; and human adjudication is too scarce for continuous use. Moreover, an inaccurate process verifier can inject harmful supervision at a much greater frequency than a terminal checker. Fixed verifiers may also lose discrimination or become exploitable as the actor improves and discovers gaps between measurable proxy criteria and the underlying task objective [6,12,13,14].

Verification should therefore be treated as a limited resource whose allocation is itself a learning and systems problem. For every consequential event, an agent-training system must determine what should be checked, when the check should occur, which verifier is sufficiently reliable, whether escalation is warranted, and how the resulting judgment should affect credit assignment. These decisions become more important for persistent artifacts: verification that is unnecessarily expensive for a transient reasoning step may be justified for a memory or skill expected to influence hundreds of future episodes. An effective framework must consequently balance the information and safety value of a check against its token, compute, latency, interaction, human-review, and lifecycle costs.

Existing research provides important but largely separate components of this solution. Process-verification methods localize feedback to intermediate decisions; path-verification methods penalize prohibited actions independently of terminal success; state-grounded verifiers inspect external effects; agent-training infrastructures connect execution transitions to reinforcement learning; and audited skill frameworks require replayable evidence before persistent capability promotion [3,5,6,7,10]. Nevertheless, these approaches do not jointly optimize the verification target, timing, verifier choice, reliability, reward polarity, and resource cost across both individual trajectories and the continuing lifecycle of memories and reusable skills. It therefore remains unclear whether adaptive verification can preserve or improve safe task performance while using substantially fewer resources than uniformly dense verification.

To address this gap, we introduce Budgeted Adaptive Verification with Asymmetric Rewards (BAVAR), a framework that models verification allocation as a sequential decision problem under multidimensional resource constraints. BAVAR maintains a heterogeneous verifier portfolio and invokes verification selectively according to actor uncertainty, action criticality, irreversibility, expected information value, verifier reliability, remaining budget, and potential downstream reuse. It gates positive process credit behind reliable evidence while keeping high-confidence path penalties active regardless of terminal success. It also applies stronger promotion requirements to persistent memories and reusable skills and supports verifier recalibration as actor behavior evolves.

This work makes four principal contributions:

• Formalization: We formulate the verifier-budget problem, in which a verification policy jointly selects the verification target, timing, and verifier under token, computation, latency, interaction, human-review, and lifecycle constraints.
• Method: We introduce BAVAR, which combines event-triggered verifier routing, hierarchical escalation, reliability-gated positive credit, independently active path penalties, competence-adaptive scheduling, and lifecycle verification.
• Systems architecture: We define transition-level instrumentation and a deferred reward ledger that record state changes, evidence, verifier identity and version, resource consumption, and the memory or skill effects of each judgment.
• Evaluation protocol: We propose matched-budget comparisons with outcome-only, uniformly dense, fixed-checkpoint, penalty-only, and single-judge baselines using verified success, violation-free success, cost per safe completion, verifier disagreement, reward-hacking, and lifecycle-reliability measures.

The remainder of the paper reviews related verification and agent-learning methods, formalizes BAVAR, presents its implementation, and evaluates its performance through matched-budget experiments, ablations, verifier-robustness analyses, and lifecycle tests.

2. Related Work

Verification for language-model agents has primarily been studied through outcome rewards, process supervision, path constraints, semantic judging, and environment-grounded evaluation. These approaches differ not only in what they verify but also in when verification occurs, how evidence is obtained, how the signal affects policy optimization, and how much computation or human attention it requires. This section reviews the methods most relevant to BAVAR and identifies the unresolved problem of allocating verification adaptively across trajectories and persistent agent lifecycles.

2.1. Reinforcement Learning from Verifiable Rewards

Reinforcement learning from verifiable rewards (RLVR) trains a policy using feedback derived from independently checkable criteria rather than unrestricted preference judgments. Common verifiers include exact-answer checkers, executable tests, symbolic solvers, task-completion predicates, inspected environment states, and evidence-grounded checklists. These mechanisms are attractive because they can be reproducible, scalable, and less sensitive to subjective annotation than human preferences or unconstrained reward models [6,7,8].

Most RLVR systems use terminal verification: the agent completes a trajectory, after which a verifier evaluates the final answer, artifact, or environment state. This approach provides a relatively direct measure of task completion and is compatible with group-relative optimization methods. Partial-credit variants can evaluate multiple terminal requirements independently, as in software environments that score the proportion of verified criteria satisfied. However, terminal partial credit increases reward resolution without solving temporal credit assignment. It can identify which final requirement failed, but not necessarily which preceding action produced the failure. Outcome verification also cannot distinguish an efficient, compliant success from one obtained through unnecessary actions, prohibited behavior, evaluator manipulation, or exploitation of incomplete tests.

2.2. Process Rewards

Process verification evaluates intermediate actions, subgoals, evidence use, or reasoning behavior before terminal success is known. Verifiable Process Rewards use symbolic or algorithmic oracles to evaluate state–action pairs, providing localized supervision even when an episode ultimately fails. Reliable intermediate verification can reduce the temporal sparsity of outcome rewards and identify useful decisions within unsuccessful trajectories. However, constructing an intermediate oracle is domain-specific, and invoking it at every step can require expensive search, simulation, or solver execution [6].

Other approaches use programmatic behavioral proxies, evidence rubrics, or semantic checklists. RLVMR rewards observable planning, exploration, reflection, and monitoring behaviors, while LongTraceRL uses reasoning-chain entities to provide positive process credit when the final answer is correct. RioRAG decomposes retrieved information into factual nuggets and rewards evidence coverage rather than relying on one holistic long-context judgment [8,9,11]. These methods improve reward density and traceability but remain vulnerable to proxy optimization. Agents may produce reasoning-shaped text, enumerate reward-bearing entities, or increase output length without improving causal reasoning or task performance. Dense verification also multiplies verifier errors: a slightly miscalibrated process judge can inject incorrect supervision at many points in every trajectory.

2.3. Path Verification

Path verification determines whether a trajectory violates a procedural, safety, or authorization constraint. It differs from positive process supervision because it focuses on unacceptable behavior rather than desirable progress. Such violations may be outcome-neutral: an agent may successfully complete a task while bypassing authentication, acting outside authorized conditions, modifying evaluation tests, using prohibited information, or executing destructive commands.

RLVP formalizes this asymmetry by retaining the outcome incentive while attaching penalties to verifiable path violations [10]. Path signals remain informative when every sampled trajectory succeeds or fails because trajectories may still differ in their violations. Nevertheless, penalty-only learning can create an inaction trap in which avoiding all behavior becomes preferable to attempting the task. Effective path verification must therefore preserve positive outcome incentives, attach penalties to identifiable actions, and distinguish unnecessary repetition from legitimate exploration. Its central value is enforcing deployment constraints that cannot be inferred from final task success.

2.4. LLM-as-a-Judge and Agentic Evaluation

Static language-model judges evaluate an answer, artifact, or trajectory from a task description, selected evidence, and scoring rubric. They provide flexible semantic assessment across domains where exact tests are unavailable, but their judgments can be sensitive to prompts, evidence presentation, verbosity, model identity, and stylistic features. They may also share biases or failure modes with the policy being evaluated and cannot inspect hidden environment state unless that evidence is explicitly provided [13].

Interactive agentic evaluators extend this mechanism by gathering additional evidence. They can open files, execute code, inspect webpages, test applications, and issue follow-up queries before reaching a judgment. This broadens coverage of underspecified intent but substantially increases tokens, latency, nondeterminism, and the possibility of judge-planning or tool-use failures. The Verification Horizon argues that verifier usefulness changes as the policy improves: a fixed test or judge may eventually saturate or become exploitable, requiring recalibration, adversarial auditing, or replacement [14]. LLM-based verification is therefore best treated as one tier within a heterogeneous portfolio rather than as an unquestioned oracle.

2.5. Long-Horizon Agent Training

Agent environments such as ReAct, WebShop, WebArena, ScienceWorld, SWE-Agent, AgentGym-RL, and Agent Lightning model behavior as multi-turn interaction rather than independent completion generation [1,2,3,4,15,16,17]. Their trajectories contain interleaved model calls, tool actions, observations, retries, and external state changes. Agent Lightning separates execution from training and records individual model and tool interactions as transitions, enabling rewards to be attached to localized events without concatenating an entire execution into one training sequence [3].

AgentGym-RL progressively increases permitted interaction horizons as policy competence develops, demonstrating that exploration opportunities and training requirements change throughout learning [4]. These systems provide important infrastructure for transition-level credit assignment, but they do not determine which transitions warrant verification, which verifier should be selected, or whether the expected learning value justifies the cost. Long-horizon training and verification allocation are therefore related but distinct problems.

2.6. Persistent Memory and Reusable Skills

Long-running agents may preserve factual memories, user preferences, tool procedures, plans, or reusable skills across episodes. Errors in these artifacts can propagate far beyond the trajectory in which they originated. Memory verification must consequently assess evidential support, provenance, scope, contradictions, privacy restrictions, and expiration conditions before information becomes persistent.

Audited Skill-Graph Self-Improvement treats reusable capabilities as versioned artifacts with explicit interfaces, preconditions, postconditions, replayable evidence, and held-out validation. Candidate skills are compiled and evaluated offline before promotion into a reusable skill graph [5]. Although this approach makes persistent improvement more auditable, skill validation and periodic revalidation impose substantial up-front costs. These costs may be amortized across repeated valid reuse, making expected future influence a crucial factor in deciding how much verification an artifact warrants.

2.7. Verification-Cost Optimization

Verification cost is multidimensional. It includes model tokens, program execution, solver calls, environment interactions, wall-clock latency, accelerator idle time, human adjudication, verifier construction, and subsequent maintenance. Dense verification is not necessarily inexpensive simply because it uses deterministic rules, and token-efficient behavior does not guarantee system-level efficiency if it requires more tool calls, failed rollouts, or costly checking.

Existing work generally studies a fixed verifier, fixed checkpoint schedule, or uniform per-step procedure. Systems research identifies individual efficiency issues—such as asynchronous-training bias, verifier bottlenecks, and transition-level observability—but does not jointly optimize verification timing, verifier choice, reliability, escalation, and lifecycle value. No single verifier simultaneously provides dense credit, broad intent coverage, strong robustness, low latency, and minimal cost.

BAVAR addresses this gap by treating verification allocation itself as a sequential, budget-constrained decision problem. It selects among complementary outcome, process, path, state, semantic, and lifecycle verifiers according to uncertainty, risk, reliability, expected information value, remaining resources, and potential downstream reuse. In this respect, BAVAR builds on prior verification mechanisms while shifting the central question from how to construct one reward signal to when and where each available verifier is worth invoking.

3. Problem Formulation

We formulate verification for long-running language-model agents as a sequential resource-allocation problem. An agent must complete tasks through valid and efficient interactions while a separate verification policy decides which events to inspect, which verifier to invoke, and when the expected value of checking an event justifies its cost. This formulation separates the policy that performs the task from the mechanism that evaluates its behavior and supports verification at the action, transition, trajectory, state, memory, and reusable-skill levels.

3.1. Agent Trajectories

Let tasks be drawn from a distribution $\mathcal{D}$. For a task $x\sim \mathcal{D}$, an actor policy ${\pi }_{\theta }$ interacts with an environment over a trajectory of variable length T:

$$\tau =(s_0,a_0,o_1,s_1,a_1,o_2,\dots ,s_T,y_T).$$

Here, $s_t$ denotes the environment and agent state at step t, $a_t$ is the action selected by the actor, $o_{t+1}$ is the resulting observation, and $y_T$ is the final answer or artifact. An action may be a language-model response, tool invocation, API request, environment interaction, file modification, database operation, memory write, skill invocation, or termination proposal. The actor conditions its decisions on the available history:

$$h_t=(x,s_0,a_0,o_1,\dots ,s_t),a_t\sim {\pi }_{\theta }(·\mid h_t).$$

This transition-level representation follows agent-training systems that separate execution traces into individual model and tool interactions rather than treating the entire trajectory as one completion [3]. It permits verification signals to be attached to the specific actions, state changes, or trajectory spans to which they apply.

A terminal outcome verifier produces an outcome score $R_O\left(\tau \right)$, indicating whether the agent completed the task. However, terminal success does not establish that the trajectory was efficient, procedurally valid, or safe. We therefore associate each trajectory with additional quantities: verified progress $P\left(\tau \right)$, path violations $L\left(\tau \right)$, persistent-state correctness $S\left(\tau \right)$, actor cost $C_A\left(\tau \right)$, and lifecycle consequences $M\left(\tau \right)$. Lifecycle consequences include the creation or modification of memories and reusable skills that may affect future episodes.

For continuing operation, the agent produces a sequence of trajectories ${\tau }_1,\dots ,{\tau }_E$. Persistent memory and skill state evolves according to

$$m_{e+1}=F(m_e,{\tau }_e,u_e),$$

where $m_e$ is the persistent state before episode e, $u_e$ represents approved memory or skill updates, and F is the controlled update mechanism. This captures the distinction between a long-horizon agent, whose dependencies occur within one episode, and a long-running agent, whose decisions can influence future episodes.

3.2. Verifier Portfolio

Let

$$\mathcal{V}=\{v_1,\dots ,v_K\}$$

denote a heterogeneous portfolio of verifiers. Each verifier $v_k$ is characterized by its supported verification objects, grounding mechanism, reliability profile, evidence requirements, and cost. The portfolio may include deterministic rules, executable tests, symbolic solvers, state inspectors, evidence or rubric evaluators, static language-model judges, interactive agentic evaluators, and human adjudicators.

At step t, the verification policy selects

$$z_t\in \mathcal{V}\cup \{⌀\},$$

where $z_t=⌀$ means that no verifier is invoked. When a verifier is selected, it evaluates an object $q_t$, such as an action, state transition, trajectory prefix, terminal state, final artifact, memory entry, or reusable skill. Given the available evidence ${\mathcal{E}}_t$, the verifier returns

$$v_k(q_t,{\mathcal{E}}_t)=(r_{k,t},{\gamma }_{k,t},{\xi }_{k,t}),$$

where $r_{k,t}$ is the judgment, ${\gamma }_{k,t}$ is confidence or calibration metadata, and ${\xi }_{k,t}$ contains supporting evidence, failure information, and provenance. Verifier reliability is not assumed to be constant. We write

$${\rho }_k(d,p,\kappa )=Pr\left(v_k\mathrm{is}\mathrm{correct}\mid d,p,\kappa \right),$$

where d denotes the task domain, p the actor checkpoint or competence level, and $\kappa$ the event type. This conditioning accounts for distribution shift, application changes, and verifier saturation as the actor learns to exploit stable evaluation criteria. A verifier may abstain when evidence is incomplete or its confidence falls below a predefined threshold. Conflicting results can trigger escalation to a more reliable or independent verifier.

3.3. Multidimensional Verification Budget

Verification consumes more than model tokens. For verifier $v_k$ at step t, define the cost vector

$${\mathbf{c}}_{k,t}=\left(c_{k,t}^{\mathrm{tok}},c_{k,t}^{\mathrm{comp}},c_{k,t}^{\mathrm{lat}},c_{k,t}^{\mathrm{env}},c_{k,t}^{\mathrm{human}},c_{k,t}^{\mathrm{maint}}\right),$$

representing token use, computation, latency, environment interactions, human attention, and maintenance or revalidation cost. The available budget is

$$\mathbf{B}=\left(B^{\mathrm{tok}},B^{\mathrm{comp}},B^{\mathrm{lat}},B^{\mathrm{env}},B^{\mathrm{human}},B^{\mathrm{maint}}\right).$$

A feasible verification schedule must satisfy

$${\displaystyle \sum _{t=0}^T}{\displaystyle \sum _{k=1}^K}\mathbb{1}[z_t=v_k]{\mathbf{c}}_{k,t}⪯\mathbf{B},$$

where ⪯ denotes component-wise inequality. This prevents a method from appearing efficient by reducing one resource while silently increasing another. For example, a semantic verifier may reduce programmatic checker construction but consume more tokens and latency, while a deterministic state checker may require little inference but substantial engineering and maintenance.

Budgets may be defined per episode, per task family, or over the full deployment lifecycle. A portion may be reserved for terminal evaluation, high-severity actions, and persistent-artifact promotion. Mandatory safety checks remain hard constraints and cannot be skipped merely because the discretionary budget has been exhausted.

3.4. Objective Function

The verification policy ${\nu }_{\varphi }$ selects a verifier conditioned on the trajectory history, verifier state, and remaining budget:

$$z_t\sim {\nu }_{\varphi }(·\mid h_t,\mathcal{V},{\mathbf{B}}_t).$$

Its purpose is not to maximize agreement with its own selected verifiers. Instead, it seeks to maximize independently evaluated safe utility while minimizing verification cost. We define the constrained objective as

$$\begin{array}{cc}\hfill \underset{\theta ,\varphi }{max}{\mathbb{E}}_{x\sim \mathcal{D},\tau \sim {\pi }_{\theta }}[& U\left(\tau \right)-{\lambda }_{L}L\left(\tau \right)-{\lambda }_A^{\top }{\mathbf{C}}_A\left(\tau \right)\hfill \\ \hfill & -{\lambda }_V^{\top }{\mathbf{C}}_V\left(\tau \right)-{\lambda }_M{M}^{-}\left(\tau \right)],\hfill \end{array}$$

(1)

subject to ${\mathbf{C}}_V\left(\tau \right)⪯\mathbf{B}$. Here, $U\left(\tau \right)$ is independently verified task utility, $L\left(\tau \right)$ measures path violations, ${\mathbf{C}}_A$ and ${\mathbf{C}}_V$ are actor and verification costs, and $M^{-}\left(\tau \right)$ captures harmful persistent effects such as contaminated memory or unsafe skill promotion.

The verification policy should invoke a verifier when its expected decision value exceeds its marginal cost. Conceptually,

$$VOI(v_k,t)=Pr(\mathrm{decision}\mathrm{change}\mid h_t,v_k)I_t{\rho }_k-{\lambda }_V^{\top }{\mathbf{c}}_{k,t},$$

where $I_t$ measures the impact, irreversibility, safety significance, or downstream reuse associated with the event. Verification is useful when it changes a consequential credit, escalation, rollback, or promotion decision—not merely when it produces another score.

3.5. Threat Model and Assumptions

The threat model includes accidental errors and strategic optimization against incomplete verifiers. The actor may exploit missing tests, generate reward-bearing surface features without genuine progress, modify evaluation artifacts, bypass authorization requirements, conceal failures in natural-language explanations, or promote unsupported information into persistent memory. It may also discover weaknesses that were not exploitable by earlier policy checkpoints.

BAVAR assumes that verifier implementations, hidden evaluation criteria, evidence stores, and audit logs are protected from direct actor modification. Tool and verifier versions are recorded, and at least some actions, outcomes, or state transitions are externally inspectable. Operational costs must be measurable consistently across methods, and independent programmatic or human adjudication must be available for a controlled evaluation subset.

The formulation does not assume that verifiers perfectly represent human intent, that observable reasoning traces reveal internal cognition, or that every agent action can be checked automatically. It also does not guarantee safety outside the evaluated task distribution. Verifier outputs may be incomplete, miscalibrated, or stale after environment changes. High-impact unresolved cases therefore require abstention, escalation, or human approval, while persistent memories and skills require stronger evidence than transient trajectory decisions.

4. BAVAR Method

Budgeted Adaptive Verification with Asymmetric Rewards (BAVAR) separates task execution from verification allocation. The actor policy performs the task, while a verification policy decides which trajectory events should be checked, which verifier should be invoked, whether escalation is necessary, and how the resulting judgment should affect learning. BAVAR is designed around three principles: verification should be concentrated on consequential events; positive credit and negative violations should not be treated symmetrically; and persistent memories or skills should require stronger evidence than transient actions.

4.1. Event-Triggered Verification

Uniformly verifying every transition can provide dense feedback but consumes substantial resources and repeatedly exposes the actor to verifier errors. BAVAR instead uses a low-cost event detector $d_{\psi }$ to identify candidate verification events:

$$e_t=d_{\psi }(h_t,a_t,s_{t+1},\Delta s_t),$$

where $h_t$ is the trajectory history, $a_t$ is the selected action, $s_{t+1}$ is the resulting state, and $\Delta s_t$ represents the observed state change. The detector does not determine whether the action is correct. It determines whether the transition is sufficiently unusual, consequential, uncertain, or reusable to warrant consideration by the verification scheduler.

Candidate events include:

• tool failures or malformed tool calls;
• repeated actions with no meaningful state change;
• large, irreversible, or externally visible modifications;
• authorization or policy boundaries;
• contradictions between observations and agent claims;
• low-confidence termination decisions;
• disagreement among inexpensive checks;
• creation or modification of persistent memory;
• compilation or promotion of a reusable skill;
• behavior that differs substantially from previously successful trajectories.

The event detector combines deterministic rules with optional statistical features. Deterministic rules capture known conditions such as forbidden commands, schema violations, unsuccessful tool returns, or writes to protected resources. Statistical features capture uncertainty, novelty, and deviation from established behavior. Because missing a high-impact event can be more harmful than proposing an unnecessary candidate, the detector is optimized for recall. Precision is handled by the downstream scheduler, which may decline verification when the expected value does not justify the cost.

Some checks remain mandatory and bypass discretionary scheduling. These include hard authorization controls, integrity checks protecting evaluation artifacts, and required terminal verification. Event-triggered scheduling therefore governs optional or graded verification, not fundamental system-enforcement boundaries.

4.2. Adaptive Verification Scheduler

For each candidate event $e_t$, the scheduler constructs a feature vector

$$x_t=\left[u_t,n_t,r_t,i_t,p_t,{\rho }_t,{\mathbf{B}}_t,{\ell }_t\right],$$

where $u_t$ is actor uncertainty, $n_t$ is action or state novelty, $r_t$ is estimated risk, $i_t$ is irreversibility, $p_t$ is task progress, ${\rho }_t$ summarizes relevant verifier reliability, ${\mathbf{B}}_t$ is the remaining budget, and ${\ell }_t$ is expected downstream influence or reuse.

The verification policy selects

$$z_t\sim {\nu }_{\varphi }(·\mid x_t,\mathcal{V}),z_t\in \mathcal{V}\cup \{⌀\}.$$

The scheduler may skip verification, invoke a verifier directly, or initiate a hierarchical escalation path. Selection is governed by the expected value of verification:

$$VOI(v_k,e_t)=p_{k,t}^{\mathrm{change}}{I}_t{\rho }_{k,t}-{\lambda }_V^{\top }{\mathbf{c}}_{k,t}.$$

Here, $p_{k,t}^{\mathrm{change}}$ estimates the probability that verifier $v_k$ will change a consequential decision, $I_t$ measures the impact of that decision, ${\rho }_{k,t}$ is the verifier’s estimated reliability for the current event, and ${\mathbf{c}}_{k,t}$ is its multidimensional cost. A decision change may include modifying actor credit, identifying a path violation, triggering rollback, rejecting termination, escalating to human review, or preventing memory or skill promotion.

Verification occurs when the highest expected value is positive and the associated resource constraints are satisfied:

$$\underset{v_k\in \mathcal{V}}{max}VOI(v_k,e_t)>0.$$

The scheduler maintains a reserve for terminal verification, high-severity events, and lifecycle promotion gates. Under budget scarcity, it can skip low-impact candidates, select a cheaper verifier, combine related events into one evidence packet, or defer verification until additional evidence becomes available.

The schedule adapts with actor competence. Early in training, progress verification may be valuable because complete successes are rare. As the actor improves, routine actions become less informative, and verification shifts toward unusual behavior, hidden violations, distribution shift, and persistent artifacts. Limited randomized audits and shadow verification are retained to estimate the errors that a purely exploitative scheduler would fail to observe.

4.3. Hierarchical Verifier Escalation

BAVAR organizes verifiers into escalating tiers:

1.

Deterministic checks: schema validation, permissions, tool-return codes, state hashes, repetition detection, and prohibited-action rules.

2.

Executable or state-grounded checks: unit tests, symbolic solvers, application queries, database inspection, file parsing, and state-transition predicates.

3.

Semantic verification: evidence rubrics, factual checklists, static language-model judges, and multi-model adjudication.

4.

Interactive or human verification: tool-using evaluators, domain experts, or authorized human reviewers.

Verification begins with the lowest-cost tier capable of evaluating the target. Escalation occurs when a verifier abstains, returns low confidence, conflicts with another check, lacks required evidence, or evaluates an event whose impact exceeds the reliability of the current tier.

Let $H\left(r_{k,t}\right)$ denote uncertainty in the current judgment and $\Delta H_{k\to j}$ the expected uncertainty reduction from escalating to verifier $v_j$. Escalation is selected when

$$\Delta H_{k\to j}{I}_t>{\lambda }_V^{\top }\Delta {\mathbf{c}}_{k\to j}.$$

A more expensive verifier is therefore used only when its expected reduction in decision uncertainty, weighted by event impact, exceeds its marginal cost. High-impact unresolved cases lead to abstention or human approval rather than automatic acceptance.

4.4. Verifier Calibration and Monitoring

Each verifier maintains a reliability profile conditioned on domain, event type, actor checkpoint, and available evidence. Calibration samples are independently adjudicated and used to estimate false-positive rates, false-negative rates, abstention frequency, expected calibration error, and discrimination between better and worse trajectories.

A raw verifier score $q_{k,t}$ is transformed into a calibrated probability:

$${\widehat{p}}_{k,t}={Cal}_k(q_{k,t}\mid d,e,p),$$

where d is the task domain, e the event category, and p the actor checkpoint. Calibration may use held-out reliability tables, Platt scaling, isotonic regression, or another prespecified method appropriate to the verifier output.

Verifier monitoring continues throughout training. Recalibration is triggered by increasing disagreement with independent audits, reward growth without corresponding utility improvement, reduced score variance, newly observed reward-hacking behavior, environment or schema changes, and actor-checkpoint transitions. A verifier that falls below a minimum reliability threshold is down-weighted, restricted to audit-only use, or quarantined until repaired.

BAVAR records the verifier version associated with every judgment. Historical rewards are not silently reinterpreted after a verifier changes; corrected judgments are entered as explicit ledger updates with their evidence and provenance.

4.5. Asymmetric Reward Construction

BAVAR separates terminal outcome, positive progress, state correctness, path violations, and lifecycle promotion:

$$\begin{array}{cc}\hfill R\left(\tau \right)=& R_O\left(\tau \right)+{\displaystyle \sum _{t=1}^T}{g}_t^{+}{r}_t^{+}+{\displaystyle \sum _{t=1}^T}{r}_t^S-{\displaystyle \sum _{t=1}^T}{\lambda }_t{r}_t^{-}\hfill \\ \hfill & +R_L\left(\tau \right)-{\mu }_A^{\top }{\mathbf{C}}_A-{\mu }_V^{\top }{\mathbf{C}}_V.\hfill \end{array}$$

(2)

Here, $R_O$ is independently verified task success, $r_t^{+}$ is positive process credit, $g_t^{+}\in [0,1]$ is a reliability gate, $r_t^S$ represents verified state advancement, $r_t^{-}$ is an action-attached path violation, $R_L$ is lifecycle credit, and ${\mathbf{C}}_A$ and ${\mathbf{C}}_V$ are actor and verification costs.

Positive progress is gated because superficial indicators of planning, reflection, evidence coverage, or subgoal completion can be produced without genuine task advancement. The gate depends on verifier reliability, evidence quality, and, where appropriate, eventual outcome correctness:

$$g_t^{+}={\widehat{p}}_{k,t}\mathbb{1}\left[\mathrm{evidence}\mathrm{sufficient}\right]\mathbb{1}\left[\mathrm{outcome}\mathrm{condition}\mathrm{satisfied}\right].$$

Positive process credit is bounded and subjected to diminishing returns so that repeated low-value actions cannot accumulate unlimited reward. In contrast, high-confidence path penalties remain active even when the final task succeeds. This prevents terminal success from erasing an authorization, integrity, safety, or procedural violation.

Penalty magnitudes exceed positive credit for comparable high-severity events, but BAVAR retains the terminal outcome reward to avoid making inactivity optimal. Necessary exploration is distinguished from verified waste through observed state change, information gain, and task relevance.

4.6. Deferred Credit Assignment

Not every judgment is available immediately. Executable checks may finish after a trajectory ends, semantic adjudication may require additional evidence, and human review may occur asynchronously. BAVAR therefore uses a reward ledger rather than requiring every verifier to return an immediate scalar reward.

Each ledger entry contains

$${\mathcal{L}}_j=\left(e,[t_1,t_2],v_k,\mathrm{version},r_j,{\widehat{p}}_j,{\xi }_j,{\mathbf{c}}_j,\mathrm{status}\right),$$

Where e identifies the episode and $[t_1,t_2]$ identifies the affected trajectory span, $v_k$ and its version, it identifies the verifier, $r_j$ the judgment, ${\widehat{p}}_j$ the calibrated reliability, ${\xi }_j$ the supporting evidence, and the status, indicating whether the judgment is pending, confirmed, corrected, or revoked.

Immediate deterministic judgments can affect the current update. Deferred results are reconciled only when their task, trajectory, artifact, and actor-checkpoint provenance match the stored record. Timeouts and missing evidence produce abstention rather than fabricated negative rewards. This structure localizes delayed feedback and prevents one late trajectory-level score from being assigned indiscriminately to every preceding action.

4.7. Memory and Skill Promotion Gates

Here, extant artifacts face stricter verification because their errors can influence many future episodes. A candidate memory contains its content, source trajectory, supporting evidence, scope, confidence, applicable context, and review or expiration condition. Promotion requires factual support, provenance integrity, conflict checking, privacy compliance, and sufficient expected usefulness. Unsupported information remains quarantined as an untrusted observation rather than becoming durable memory.

A candidate reusable skill includes an interface, preconditions, postconditions, permissions, tool dependencies, failure behavior, and an evidence bundle. Promotion requires successful replay on held-out tasks, validation of preconditions and postconditions, perturbation and regression testing, inspection for unauthorized side effects, verifier approval under the current environment version, and evidence that the expected reuse value exceeds verification and maintenance costs.

Lifecycle rewards are deferred until the artifact passes its promotion gate. Later failures can trigger revocation, rollback, and negative attribution to the originating artifact and policy checkpoint. Verification effort is weighted by expected reuse: a costly audit may be justified when an artifact will affect many future tasks.

4.8. Training Algorithm

BAVAR alternates actor learning, scheduler learning, and verifier maintenance:

Figure 1 summarizes the complete BAVAR training and verification procedure.

Actor and scheduler updates are separated to reduce collusion between task behavior and verifier selection. Final evaluation uses hidden criteria and independent verifiers so that improved training rewards cannot by themselves be interpreted as improved task utility.

5. Implementation

The BAVAR implementation separates task execution from evidence collection, verification, scheduling, and reward reconciliation. Seven components communicate through typed event records: the actor runtime, evidence store, verifier registry, scheduler, reward ledger, security layer, and governed interfaces. The Menlo AI Agent provides a concrete integration prototype; implemented features are distinguished from components proposed for a complete BAVAR training system.

5.1. System Architecture

The architecture follows an event-driven pipeline. Each task receives task and episode identifiers, an actor checkpoint, and a resource budget. The actor proposes an action, but external operations pass through a tool gateway that validates the request, invokes an authorized tool, captures the result and state change, and emits a transition event. The evidence store preserves the event and its provenance. The event detector marks verification candidates; the scheduler then skips, verifies, escalates, or defers each candidate.

Verifier results enter the reward ledger rather than directly changing actor rewards, allowing delayed tests or review to be reconciled with the correct trajectory span. Mandatory terminal, authorization, and persistent-artifact checks bypass discretionary scheduling. Actor updates consume finalized ledger snapshots, whereas scheduler updates use independently measured decision value. Hidden criteria, verifier prompts, calibration data, and audit samples remain inaccessible to the actor.

For local experiments, these components may execute synchronously within one process. The distributed configuration uses a durable event queue and idempotent consumers so that repeated delivery does not duplicate tool actions, verifier charges, or rewards. Schema versions accompany every event, and correlation identifiers connect user requests, tool calls, evidence, judgments, and ledger entries across service boundaries. This configuration also permits verifiers to scale independently from actor inference.

5.2. Actor Runtime

The model-agnostic actor runtime exposes a common interface for generation, retrieval, code execution, application calls, and memory or skill operations. Each transition records identifiers, history digest, action type and arguments, observation, timestamp, model and tool versions, token use, latency, return status, and observable state delta. Large artifacts are stored by reference.

Before execution, an interceptor checks schema validity, authorization, and hard safety constraints. It records the external result without permitting the actor to rewrite the audit record. Retries remain separate actions, exposing repeated failures, ineffective loops, and recovery behavior to event detection and credit assignment. The same instrumentation supports online inference and replay-based training.

Runtime adapters expose consistent timeout, retry, and cancellation semantics. This prevents differences among tools from being mistaken for differences in actor quality and makes actor cost comparable across experimental conditions.

5.3. Evidence Store

The append-only evidence store records the material used to evaluate each event. Each record contains its originating transition, evidence type, source, collection time, content hash, access classification, tool version, and retention status. Evidence may include tool payloads, state hashes, retrieved passages, test output, permission decisions, or human annotations. Content-addressed storage prevents unnecessary duplication.

An evidence builder constructs the smallest sufficient packet for each target, reducing token cost and sensitive-context exposure. Missing, stale, or unauthorized evidence is represented explicitly and may cause abstention. Memories and skills retain links to supporting evidence so that promotion, revalidation, and revocation remain auditable.

5.4. Verifier Registry

The verifier registry stores a versioned descriptor for each checker, including supported objects, evidence requirements, expected cost, coverage, confidence representation, abstention conditions, escalation targets, and reliability estimates. Adapters normalize deterministic rules, executable tests, state inspectors, semantic judges, and human review into a common result containing the judgment, calibrated confidence, evidence references, failure metadata, and measured cost.

Used registry entries are immutable; a changed verifier receives a new version. Reliability is conditioned on domain, event type, and actor checkpoint and updated from independently adjudicated samples. Quarantined verifiers may run in shadow mode but cannot generate training credit, preventing silent reinterpretation of historical judgments after a prompt, model, rule, or environment changes.

5.5. Scheduler

The scheduler constructs the Section 4 feature vector from uncertainty, novelty, risk, irreversibility, progress, expected reuse, remaining budget, and verifier reliability. Mandatory rules run first. For discretionary checks, a transparent score estimates the probability of changing a consequential decision, weights it by impact and reliability, and subtracts multidimensional cost. The scheduler selects the feasible positive-value action while reserving resources for terminal checks, severe events, and lifecycle gates.

This rule-based scheduler provides a reproducible cold start and can later be replaced by a contextual bandit or constrained policy trained on ledger outcomes. Randomized audits and shadow verification estimate selection bias and false negatives. Each decision records considered verifiers, features, budget state, selected action, and reason code for counterfactual analysis.

5.6. Reward Ledger

The append-only reward ledger is keyed by episode, trajectory span, artifact, actor checkpoint, and verifier version. Entries are pending, confirmed, corrected, or revoked; corrections create records instead of overwriting history. Each entry stores reward polarity and magnitude, reliability, evidence references, verification cost, and the credit-mapping rule.

Deterministic checks may finalize during execution, whereas tests and human reviews can be reconciled later. Positive process rewards require satisfied evidence and reliability gates; high-confidence path penalties remain attached to responsible actions despite terminal success. Training reads a versioned snapshot, preventing late results from reaching an incompatible checkpoint or transition.

5.7. Menlo AI Agent Prototype

The Menlo AI Agent repository is a small, read-only integration testbed.1 It exposes command-line, FastAPI, and Streamlit interfaces. A keyword router sends assignment, deadline, announcement, and course questions to a Canvas tool and institutional questions to website search. The ingester collects selected public Menlo pages, removes navigation and scripts, and stores text, titles, and URLs in JSON. TF–IDF and cosine similarity return the three highest-scoring pages with excerpts and sources.

Canvas defaults to local mock data when a token is absent or mock mode is enabled; the real connector uses authenticated read-only HTTP GET requests. Retrieved context is sent through an OpenAI-compatible client to NVIDIA using meta/llama-3.1-8b-instruct, temperature $0.2$, and at most 900 output tokens. The prompt requires context-grounded answers, disclosure of missing information, source citation when available, and protection of private student data. Email, IT, and advising tools remain disabled.

The proposed integration instruments are routing, retrieval, Canvas, and generation as transition events. Deterministic verifiers check tool status, source presence, and read-only enforcement; semantic verifiers assess grounding and unsupported claims. The scheduler and ledger then enable matched-budget comparisons among outcome-only, dense, and adaptive verification. The prototype therefore demonstrates an integration path, not empirical validation.

5.8. Security and Audit Controls

The prototype uses a read-only boundary: mock Canvas data are enabled by default, credentials come from environment configuration, and email, grade changes, and institutional-system writes are excluded. Production additionally requires institutional authentication, role-based and course-scoped authorization, least-privilege service accounts, managed secrets, encryption, egress restrictions, and field-level filtering of student information.

Website and Canvas text must be treated as untrusted evidence rather than instructions. Audit records capture the principal, action, arguments, evidence identifiers, model and verifier versions, cost, judgment, and approval status, with hashes protecting integrity. Any future high-impact write requires human approval and post-action verification. Retention, deletion, incident response, and access review must be defined before real student records are used, preserving separation among agent behavior, protected evidence, and independent evaluation.

6. Experimental Setup

We evaluate whether adaptive verification improves long-horizon and long-running agent behavior under the same verification resources available to competing methods. The protocol separates training-time verification from final evaluation: methods may select different training checks, but all resulting policies are evaluated with the same hidden outcome tests, path rules, state inspectors, lifecycle audits, and blinded human-adjudication procedure. The task set, budgets, verifier versions, statistical models, and exclusion criteria are fixed before examining test results.

6.1. Research Questions and Hypotheses

The evaluation addresses five research questions. RQ1 asks whether BAVAR improves safe verified success at a matched verification budget. H1 predicts that BAVAR will achieve a higher proportion of tasks that are both independently successful and free of verified path violations than outcome-only, uniformly dense, fixed-checkpoint, random, penalty-only, and single-judge baselines. RQ2 asks whether adaptive allocation reduces the cost of obtaining a safe completion. H2 predicts lower verifier tokens, calls, latency, and human-review demand per safe success.

RQ3 examines procedural reliability. H3 predicts fewer authorization, integrity, repetition, and unsafe-state violations, including violations occurring in otherwise successful trajectories. RQ4 examines reward hacking and verifier degradation. H4 predicts a smaller gap between training reward and hidden independently measured utility, lower exploit success, and slower performance deterioration when one verifier is perturbed or withheld. RQ5 concerns persistent state. H5 predicts greater memory and skill promotion precision, fewer downstream failures caused by promoted artifacts, and higher verified success when valid artifacts are reused.

6.2. Agent Environments

The benchmark contains four complementary environments. WebShop evaluates multi-step search and purchasing decisions with executable task outcomes. WebArena evaluates longer browser trajectories involving navigation, form completion, and state-changing operations. A controlled tool-use environment contains file, database, calculator, and code-execution tasks with exact state predicates and seeded failure conditions. It permits systematic manipulation of irreversibility, authorization boundaries, delayed observations, and verifier availability.

The Menlo suite extends the public Menlo AI Agent prototype with institutional website and mock Canvas scenarios. Tasks cover factual retrieval, assignment and announcement summarization, insufficient-evidence responses, source attribution, conflicting records, prompt injection, and requests for prohibited writes. A lifecycle partition presents related task sequences across episodes. Agents may propose memories or reusable procedures, which are later evaluated on held-out tasks containing valid reuse opportunities, changed information, contradictions, and expired evidence. Public or synthetic records are used; no real student education records are included in the research dataset.

Official benchmark splits are retained when available. For constructed tasks, templates are divided by scenario family before instance generation so that paraphrases of a test scenario cannot appear in training. Test tasks, hidden rules, adversarial cases, and human-review samples remain inaccessible to the actor, scheduler, and training verifiers.

6.3. Actor Models

The primary actor is meta/llama-3.1-8b-instruct, matching the Menlo prototype. The second actor is Qwen/Qwen2.5-7B-Instruct, providing a comparable open-weight model from another family. Exact revision hashes, tokenizers, and licenses are frozen in the experiment manifest before training. Each model is evaluated in a frozen-policy condition and a reinforcement-learning condition using the same optimizer, adaptation method, rollout count, context limit, sampling temperature, and actor token budget across verification methods.

Actor prompts define the task and available tools but do not reveal hidden verifier instructions. Tool schemas, maximum interaction horizon, retry policy, and termination conditions are identical within an environment. Model outputs that fail parsing are recorded as actions and are not silently regenerated. The evaluation reports results separately by actor and includes an aggregate estimate only after checking method-by-actor interaction effects.

6.4. Baselines

We compare BAVAR with six baselines. Outcome-only RLVR assigns only independently verified terminal reward. Uniform dense verification checks every eligible transition using a fixed verifier hierarchy. Fixed-checkpoint verification checks predetermined trajectory positions and terminal state. Random-budget verification samples eligible events uniformly until its budget is exhausted. Single-judge verification uses one semantic language-model judge for all supported objects. Path-penalty learning combines terminal reward with verifiable violation penalties but provides no positive process credit. All methods share the same actor architecture, environment interface, terminal evaluator, hard safety rules, training tasks, and optimization schedule.

An additional oracle-routing condition may be reported as a non-deployable upper bound. It uses hidden labels to select the cheapest correct verifier and is excluded from confirmatory hypothesis tests. Ablations remove event triggering, reliability calibration, hierarchical escalation, asymmetric reward gating, budget reserves, or lifecycle promotion gates one component at a time.

6.5. Matched-Budget Protocol

Actor resources and verification resources are controlled separately. Every method receives the same number of tasks and rollouts, actor-token ceiling, environment-interaction limit, and maximum horizon. Verification is constrained by the vector

$$\mathbf{B}=(B^{\mathrm{tok}},B^{\mathrm{comp}},B^{\mathrm{lat}},B^{\mathrm{env}},B^{\mathrm{human}},B^{\mathrm{maint}}).$$

Primary comparisons use three preregistered budget levels—low, medium, and high—defined from pilot measurements without examining test outcomes. A method may not compensate for exceeding one component by saving another. Uniform and random baselines stop optional checking when any binding component is exhausted; mandatory safety and terminal checks are accounted for separately and applied equally.

Verifier cost is measured from observed tokens, wall-clock time, executable calls, environment interactions, and human-review seconds rather than estimated from list prices alone. Training runs use identical task orders and paired environment seeds. Evaluation costs are not charged to the training budget because the same independent evaluation is applied to every trained policy. However, all evaluation resources are reported to make the full experimental cost visible.

6.6. Metrics

The primary endpoint is safe verified success (SVS): the proportion of tasks that satisfy hidden outcome criteria, contain no severity-threshold path violation, and leave required external state valid. Secondary effectiveness metrics are verified task success, violation-free success, partial requirement completion, and recovery after a seeded tool failure. Efficiency metrics include verifier tokens, compute time, latency, verifier and environment calls, human-review seconds, total verification cost, and cost per SVS.

Procedural metrics count path violations by category and severity, unnecessary repeated actions, unauthorized attempts, test or evidence manipulation, and successful-but-invalid trajectories. Reward-robustness metrics include the correlation and gap between training reward and hidden utility, exploit success under adversarial tasks, verifier disagreement, abstention rate, expected calibration error, and false-positive and false-negative rates on audited events. Lifecycle metrics include memory and skill promotion precision and recall, held-out reuse success, downstream error propagation, revocation frequency, revalidation cost, and net benefit after verification and maintenance cost.

6.7. Statistical Tests

All confirmatory comparisons are paired by task instance, environment seed, actor, and budget. SVS and other binary outcomes are analyzed using mixed-effects logistic regression with verification method, budget, actor, and their prespecified interactions as fixed effects and task family and seed as random intercepts. Paired risk differences and odds ratios are reported with $95\%$ cluster-bootstrap confidence intervals. Count outcomes such as violations are modeled with a negative-binomial mixed model when overdispersion is present.

Continuous cost and calibration measures are compared using paired hierarchical bootstrap intervals and a paired permutation test; strongly skewed quantities are summarized with medians and interquartile ranges. Lifecycle analyses cluster by originating memory or skill so that repeated reuse does not create artificial sample size. Confirmatory hypotheses H1–H5 use two-sided tests with Holm correction and $\alpha =0.05$. Effect sizes and confidence intervals remain primary; statistical significance alone is not interpreted as practical importance. Exclusions, failed runs, and missing human judgments are reported by method and analyzed under a prespecified missingness rule.

6.8. Reproducibility Settings

Each condition is run with at least five training seeds and paired evaluation seeds. We record code commit, model and tokenizer revisions, prompts, tool and verifier versions, calibration sets, environment snapshots, package lockfile, hardware, precision, random seeds, and all optimization and decoding parameters. Structured logs contain transition identifiers, evidence hashes, scheduler decisions, budgets, verifier outputs, and reward-ledger revisions. Secrets and sensitive content are excluded from released logs.

The artifact package will include configuration files, synthetic Menlo tasks, split-generation code, verifier adapters, analysis scripts, and a machine-readable results table. Container images or environment specifications pin dependencies. Reported tables are generated directly from finalized ledger exports, and a verification script checks that every numerical claim maps to an archived result field. Any unavailable proprietary model or judge is paired with a documented open substitute for reproducibility analysis.

7. Results

The illustrative dataset assumes 400 unique held-out tasks, with 100 tasks from each of WebShop, WebArena, the controlled tool-use environment, and the Menlo suite. Each task is evaluated with two actor models, five paired seeds, and three verification budgets, producing $12,000$ paired evaluation trajectories per method. Unless otherwise stated, method-level values refer to the medium-budget condition. All confidence intervals and test statistics below are to remain consistent with the displayed summary values.

7.1. Main Matched-Budget Results

At the low, medium, and high budgets, BAVAR achieved safe verified success (SVS) rates of $61.8\%$, $72.6\%$, and $77.9\%$, respectively. The strongest deployable baseline at the corresponding budgets achieved $56.1\%$, $67.1\%$, and $75.0\%$. The paired improvements were therefore $5.7$ percentage points (pp; $95\%$ CI $[4.2,7.2]$), $5.5$ pp ($95\%$ CI $[4.1,6.9]$), and $2.9$ pp ($95\%$ CI $[1.6,4.2]$). A mixed-effects logistic model estimated an overall BAVAR odds ratio of $1.31$ ($95\%$ CI $[1.24,1.39]$, Holm-adjusted $p<0.001$). The method-by-budget interaction was significant ($p=0.012$), indicating that the advantage narrowed as verification resources became abundant. These results support H1.

Table 1. Illustrative results at the medium verification budget. Cost/SVS is measured in normalized verification-cost units per safe verified success; Lifecycle is held-out reuse success. These values are not empirical findings.

Method	SVS ↑	Cost/SVS ↓	Viol./100 ↓	Lifecycle ↑
Outcome only	$58.4$	$8.9$	$20.4$	$72.1$
Uniform dense	$67.1$	$18.7$	$8.6$	$81.3$
Fixed checkpoint	$63.9$	$11.8$	$13.1$	$77.6$
Random budget	$60.7$	$10.9$	$15.7$	$74.8$
Single judge	$61.5$	$13.6$	$14.4$	$75.5$
Path penalty	$64.8$	$9.7$	$9.1$	$76.9$
BAVAR	$72.6$	$10.2$	$7.2$	$88.7$

Table 1. Illustrative results at the medium verification budget. Cost/SVS is measured in normalized verification-cost units per safe verified success; Lifecycle is held-out reuse success. These values are not empirical findings.

Method	SVS ↑	Cost/SVS ↓	Viol./100 ↓	Lifecycle ↑
Outcome only	$58.4$	$8.9$	$20.4$	$72.1$
Uniform dense	$67.1$	$18.7$	$8.6$	$81.3$
Fixed checkpoint	$63.9$	$11.8$	$13.1$	$77.6$
Random budget	$60.7$	$10.9$	$15.7$	$74.8$
Single judge	$61.5$	$13.6$	$14.4$	$75.5$
Path penalty	$64.8$	$9.7$	$9.1$	$76.9$
BAVAR	$72.6$	$10.2$	$7.2$	$88.7$

The positive aggregate effect appeared in both actors and all four environments. Relative to the strongest baseline within each environment, the BAVAR gain was largest in WebArena ($8.4$ pp), followed by the Menlo lifecycle suite ($6.7$ pp), the controlled tool-use environment ($4.9$ pp), and WebShop ($2.2$ pp). Removing reliability calibration reduced aggregate SVS by $3.8$ pp, removing event triggering increased verification cost by $46.1\%$, and removing lifecycle promotion gates reduced held-out reuse success by $7.5$ pp. No ablation matched the full method on both SVS and cost.

7.2. Safe Verified Success

BAVAR satisfied the hidden terminal criteria on $79.4\%$ of medium-budget trajectories and achieved SVS on $72.6\%$. Thus, $6.8$ percentage points of nominally successful trajectories were excluded because of a verified path violation or invalid final state. Outcome-only RLVR achieved $58.4\%$ SVS, while uniform dense verification achieved $67.1\%$. The BAVAR–uniform paired difference was $5.5$ pp ($95\%$ CI $[4.1,6.9]$). The improvement remained positive for both Llama 3.1 8B ($5.9$ pp) and Qwen2.5 7B ($5.1$ pp), with no statistically meaningful method-by-actor interaction after Holm correction ($p=0.18$). The evidence therefore supports H1 without indicating that the effect depends on one actor family.

7.3. Verification Cost

BAVAR used a median of $2,180$ verifier tokens, $2.0$ executable verifier calls, $3.4$ seconds of verification latency, and $1.8$ human-review seconds per task. Uniform dense verification used $4,176$ tokens, $3.4$ executable calls, $5.5$ seconds of latency, and $4.0$ human-review seconds. BAVAR therefore reduced these components by $47.8\%$, $41.2\%$, $38.2\%$, and $55.0\%$, respectively. Its normalized cost per SVS was $10.2$, compared with $18.7$ for uniform dense verification. Outcome-only RLVR was cheaper at $8.9$ units but produced substantially lower SVS; among methods exceeding $65\%$ SVS, BAVAR had the lowest cost per safe success. Optional-budget exhaustion occurred in $7.4\%$ of BAVAR episodes and $31.6\%$ of uniformly verified episodes. The paired hierarchical bootstrap estimated a $45.5\%$ reduction in cost per SVS relative to uniform verification ($95\%$ CI $[41.2,49.6]$, adjusted $p<0.001$), supporting H2.

7.4. Path Violations

BAVAR produced $7.2$ verified violations per 100 trajectories, compared with $20.4$ for outcome-only RLVR and $9.1$ for path-penalty learning. Relative to outcome-only RLVR, authorization violations decreased by $58.0\%$, integrity violations by $64.0\%$, and unnecessary repeated actions by $38.0\%$. Among trajectories satisfying the terminal outcome criteria, $6.1\%$ still contained at least one path violation, demonstrating that terminal success alone did not identify procedurally invalid behavior. The negative-binomial mixed model estimated an incidence-rate ratio of $0.35$ for BAVAR versus outcome-only RLVR ($95\%$ CI $[0.29,0.42]$, adjusted $p<0.001$). BAVAR also produced $20.9\%$ fewer violations than the path-penalty baseline, supporting H3 while showing that violations were reduced rather than eliminated.

7.5. Reward Hacking

The correlation between training reward and hidden task utility was $r=0.82$ for BAVAR and $r=0.69$ for uniform dense verification, the strongest baseline on this measure. On adversarial tasks designed around incomplete tests, $4.8\%$ of BAVAR trajectories successfully exploited a verifier, compared with $15.9\%$ for the single-judge baseline and $18.6\%$ for outcome-only RLVR. The standardized absolute training-reward–utility gap was $0.11$ for BAVAR and $0.24$ for the single-judge baseline.

Withholding the verifier selected most frequently by BAVAR reduced SVS by $3.7$ pp and increased cross-verifier disagreement by $6.2$ pp. Although this degradation was smaller than the $9.8$-pp decline observed for the single-judge baseline, it indicates residual dependence on verifier availability. The results support H4 because adaptive routing and calibration reduced exploit success and reward misalignment, but they do not establish immunity to coordinated or previously unseen verifier attacks.

7.6. Lifecycle Performance

Across the long-running episodes, BAVAR promoted 312 candidate memories and 96 reusable skills. Promotion precision was $93.6\%$, held-out reuse success was $88.7\%$, and downstream failures attributable to promoted artifacts occurred in $3.9\%$ of reuse episodes. The strongest lifecycle baseline, uniform dense verification, achieved $86.2\%$ promotion precision, $81.3\%$ reuse success, and an $8.8\%$ downstream-failure rate. Cluster-bootstrap differences favored BAVAR by $7.4$ pp for promotion precision ($95\%$ CI $[4.5,10.2]$) and $7.4$ pp for reuse success ($95\%$ CI $[4.8,10.0]$), both with adjusted $p<0.001$.

Periodic revalidation revoked 17 artifacts at a mean cost of $0.64$ normalized units per reviewed artifact. After subtracting initial verification, revalidation, and maintenance costs, promoted artifacts yielded a net benefit of $0.21$ additional safe completions per artifact. On changed-information and contradiction tasks, BAVAR correctly suppressed or rolled back stale artifacts in $84.1\%$ of cases. These illustrative findings support H5 by showing higher promotion reliability and lower propagation of persistent errors, while also indicating that lifecycle verification requires continuing revalidation rather than one-time approval.

8. Analysis and Ablations

The results suggest that BAVAR’s advantage does not arise from adding more verification indiscriminately. Instead, performance depends on deciding which events to inspect, using calibrated and heterogeneous verifiers, preserving resources for consequential events, and separating positive progress credit from path penalties. This interpretation is consistent with prior evidence that outcome rewards provide sparse credit, process signals can improve localization, and fixed reward mechanisms may be exploited or lose discrimination as the policy improves [6,9,14]. The following analyses isolate the contribution of each component and identify conditions under which adaptive verification remains unreliable.

8.1. Scheduler Ablations

Table 2 reports the medium-budget ablations. Removing event triggering produced nearly unchanged SVS ($73.0\%$ versus $72.6\%$) but increased cost per SVS from $10.2$ to $14.9$, a $46.1\%$ increase. This pattern indicates that event detection primarily provides efficiency rather than direct task competence. In contrast, removing calibration reduced SVS by $3.8$ percentage points and increased violations from $7.2$ to $10.1$ per 100 trajectories. Removing escalation was cheaper but produced the largest violation rate, suggesting that low-cost checks were unable to resolve a subset of consequential ambiguous events.

Eliminating the budget reserve caused optional verification to consume resources needed near termination and during artifact promotion. Removing lifecycle gates had little immediate effect on single-episode SVS but reduced reuse success by $7.5$ points, demonstrating why persistent artifacts cannot be evaluated only through the trajectory that produced them. Finally, removing randomized audits increased successful verifier exploitation from $4.8\%$ to $7.6\%$. A scheduler trained solely on previously selected checks therefore risks creating a blind spot: it cannot estimate errors in regions it stops inspecting.

8.2. Symmetric Versus Asymmetric Rewards

The asymmetric design separates reliability-gated positive progress from high-confidence path penalties that remain active even when the terminal outcome succeeds. In the process, replacing this design with equal-magnitude symmetric process rewards increased nominal task success slightly, from $79.4\%$ to $80.1\%$, but reduced SVS from $72.6\%$ to $67.9\%$ and increased violations from $7.2$ to $12.6$ per 100 trajectories. The actor learned to accumulate observable progress signals that offset the cost of an invalid action. This is the behavior expected when a proxy reward can be optimized more easily than the intended objective [12,18].

The penalty-only baseline reduced violations but achieved only $64.8\%$ SVS because conservative policies abandoned difficult tasks or avoided useful exploration. The comparison supports the motivation behind verifiable path penalties: an outcome incentive is still required, while path violations must not be erased by success [10]. Positive process credit remained beneficial only when grounded in sufficient evidence and bounded so that repeated low-value actions could not dominate terminal utility.

8.3. Verifier Reliability

Verifier reliability was varied by injecting independent label errors into optional process and semantic judgments. At error rates of $10\%$, $20\%$, and $30\%$, BAVAR’s SVS decreased by $1.6$, $3.8$, and $7.4$ points. The corresponding decreases without calibration were $3.4$, $8.9$, and $14.7$ points. Calibration, abstention, and escalation therefore reduced but did not eliminate sensitivity to incorrect feedback. Static language-model judges are known to vary with prompts, presentation, and model identity, while fixed reward models can become less reliable when aggressively optimized [12,13].

The most damaging condition involved correlated failure, in which several verifiers relied on the same incomplete evidence or shared model family. Apparent agreement then overstated reliability, and escalation offered little protection. Diversity must consequently be defined by independent evidence and failure modes rather than by the number of verifier calls. Shadow evaluation, held-out adjudication, version tracking, and periodic replacement remain necessary because no verifier should be treated as permanently authoritative [14].

8.4. Budget Sensitivity

BAVAR achieved SVS rates of $61.8\%$, $72.6\%$, and $77.9\%$ under low, medium, and high budgets. The strongest baselines achieved $56.1\%$, $67.1\%$, and $75.0\%$. Adaptive allocation provided its largest practical benefit when verification was scarce: it preserved terminal and high-impact checks while skipping routine transitions. At the high budget, dense verification approached BAVAR because verifier scarcity was less binding. The remaining difference reflected verifier reliability and reward construction rather than allocation alone.

The relationship was not monotonic for every verifier. Beyond the medium budget, additional semantic judgments produced diminishing returns and sometimes increased disagreement, latency, and exposure to noisy process signals. The appropriate objective is therefore not maximum verification density. It is the Pareto frontier between independently measured safe utility and the complete resource vector. Token-only comparisons would conceal shifts toward executable calls, environment interactions, or human review.

8.5. Competence Adaptation

The scheduler changed its allocation as actor competence increased. During early training, $46\%$ of optional checks targeted progress, $26\%$ targeted path or state validity, $20\%$ used semantic evaluation, and $8\%$ concerned persistent artifacts. In late training, these shares changed to $18\%$, $42\%$, $17\%$, and $23\%$, respectively. Routine subgoals became less informative as success increased, while rare violations, distribution shifts, termination errors, and reusable artifacts represented a larger share of remaining risk.

A fixed schedule could not make this transition and spent late-stage resources re-verifying mastered behavior. However, competence adaptation creates its own feedback problem: reduced inspection can hide regression. BAVAR therefore retains randomized audits and checkpoint-conditioned calibration. This complements curricula that change interaction opportunities with actor capability while addressing the distinct problem of where verification remains valuable [4].

8.6. Cross-Domain Generalization

The gain over the strongest baseline was $8.4$ points in WebArena, $6.7$ in the Menlo lifecycle suite, $4.9$ in controlled tool use, and $2.2$ in WebShop. The larger improvements occurred where trajectories contained heterogeneous actions, partial observability, or persistent effects. WebShop’s more regular outcome structure left less room for adaptive routing beyond reliable terminal checks. These differences align with the contrast between grounded web interaction and broader multi-turn environments [1,2,4].

Positive results across four environments would not establish unrestricted generalization. All environments remain digitally mediated, English-dominant, and comparatively observable. Transfer to embodied systems, high-stakes professional domains, multilingual interaction, or adversarial multi-agent settings requires new evidence schemas, cost models, and independent safety criteria. The Menlo suite tests institutional retrieval and persistent state but does not represent deployment with live student records.

8.7. Failure Cases

Five recurrent failures remained. First, ambiguous user intent caused both actor and semantic verifier to select the same incorrect interpretation. Second, incomplete external state made valid actions appear unsuccessful and encouraged unnecessary retries. Third, correlated judges accepted fluent but unsupported evidence summaries. Fourth, low budgets were sometimes consumed by early anomalies before a more consequential late event. Fifth, a memory that was correct when written became stale after an environment change and affected several later episodes before revalidation.

These cases clarify the boundary of BAVAR. Adaptive verification reallocates available checking capacity; it does not create missing evidence, resolve fundamentally ambiguous intent, or guarantee independence among verifiers. High-impact unresolved cases must end in abstention, rollback, or authorized human review rather than forced automation.

9. Limitations and Broader Impact

9.1. Verifier Misspecification

BAVAR optimizes verifier allocation but cannot guarantee that the available verifiers represent the intended objective. An incomplete test, poorly specified rubric, or biased human judgment may produce confident but incorrect feedback. Adaptive scheduling can worsen this failure if it preferentially selects a cheap misspecified verifier or learns that an exploitable check has high apparent decision value. Reward misspecification and reward-model overoptimization therefore remain fundamental risks rather than problems solved by budgeting [12,18].

Calibration also depends on the representativeness of audit samples. Rare harms, novel attacks, and environmental changes may not appear in calibration data. Agreement among verifiers is weak evidence when they share training data, prompts, or evidence sources. BAVAR mitigates these risks through abstention, heterogeneous grounding, randomized audits, versioned reliability profiles, hidden evaluation, and verifier quarantine, but independent external evaluation remains necessary.

9.2. Privacy and FERPA Implications

The Menlo prototype is intentionally read-only and defaults to mock Canvas data. A real Canvas integration could process student education records protected by the Family Educational Rights and Privacy Act (FERPA) and its implementing regulations at 34 CFR Part 99 [19,20]. Whether a particular data flow is permitted depends on institutional control, consent, the purpose of access, and any applicable regulatory exception; this paper does not provide legal advice.

Verification can increase privacy exposure because evidence packets, audit logs, semantic judges, and human reviewers may receive information beyond the minimum needed to answer a question. Production deployment should therefore apply data minimization, course- and role-scoped authorization, field-level filtering, purpose limitation, retention and deletion rules, encryption, access logging, and contracts governing any external processor. Sensitive evidence should remain inside an institutionally approved boundary whenever possible. Research releases must use synthetic or properly de-identified data and exclude secrets, student identifiers, prompts containing education records, and recoverable audit artifacts.

9.3. Human-Review Limitations

Human review is neither costless nor perfectly reliable. Reviewers may disagree, experience fatigue, lack domain expertise, or infer different intent from incomplete evidence. Escalation may also create unacceptable delay in interactive systems, and repeated exposure to sensitive cases increases privacy risk. Human decisions must therefore record the evidence, rubric, reviewer role, confidence, and adjudication procedure. Inter-rater reliability and overturn rates should be reported, and reviewers should be allowed to abstain. Human approval is a governance control, not a substitute for well-specified technical checks.

9.4. Potential Misuse

The same techniques used to make benign agents more reliable could improve agents that conduct intrusive surveillance, automated manipulation, cyber abuse, or unauthorized data collection. Detailed verifier weaknesses may help an attacker evade monitoring. Conversely, organizations may misuse auditability claims to justify excessive automation or employee and student monitoring. A high verification score can create false assurance when the verifier measures only a narrow proxy.

Persistent memories and reusable skills create additional dual-use risk because one promoted capability may scale across many tasks. Capability promotion should account for both task benefit and misuse potential, with restricted tools, rate limits, sandboxing, and revocation mechanisms applied before broad reuse.

9.5. Mitigation and Governance

Governance should combine technical controls with institutional accountability. Recommended controls include a read-only pilot; documented data flows; role-based access; least-privilege credentials; independent privacy and security review; adversarial testing; incident response; verifier and model versioning; human approval for consequential writes; and periodic review of retained memories and skills. The NIST Generative AI Profile provides a useful risk-management structure for identifying, measuring, managing, and governing generative-AI risks [21].

Deployment decisions should be made by accountable institutional owners rather than the model or scheduler. Users need clear notice of system limitations and a method to contest or correct consequential outputs. Audit logs require their own access controls and retention policies. High-impact applications should define stop conditions under which the agent reverts to read-only operation, disables an unreliable verifier, rolls back a promoted artifact, or suspends automated action entirely.

10. Conclusion

This paper introduced BAVAR, a framework that treats verification as a sequential allocation problem rather than a uniformly applied reward function. It combines event-triggered scheduling, a heterogeneous and calibrated verifier portfolio, hierarchical escalation, asymmetric rewards, deferred credit assignment, and stronger promotion gates for persistent memories and skills. The defensible finding at the current stage is methodological: long-running agents require joint optimization of task utility, path validity, verifier reliability, lifecycle effects, and multidimensional cost. BAVAR offers a practical design principle for systems in which checking every transition is too expensive and outcome-only verification is too weak. It concentrates verification on events whose judgment can change a consequential decision, while preserving mandatory safety, terminal, and lifecycle controls. The Menlo AI Agent demonstrates how an existing tool-using system could be instrumented with transition records, evidence provenance, verifier routing, and audit controls without granting the actor direct access to hidden evaluation mechanisms. Subsequent work should learn scheduler policies under formal safety constraints, model correlated verifier failures, estimate the long-term value of memory and skill verification, and test transfer to embodied, multilingual, and multi-agent environments. Other directions include privacy-preserving verification, causal attribution of delayed rewards, active selection of human review, automatic verifier repair, and theoretical bounds relating verification budget, reliability, and safe task performance. Long-running agents will require verification systems that evolve with their capabilities; developing that co-evolution safely is the central open problem.