Harness-Aware Evaluation of LLM Agents: Systems, Benchmarks, and Protocols

1. Introduction

Agents built on large language models (LLMs) are increasingly used for tasks ranging from coding to email management and information classification. Their performance is commonly evaluated using benchmarks1 such as SWE-bench [1] and $\tau$-bench [2]. Although results are typically reported under model or agent names, they reflect the performance of a model operating through a harness2, interacting with an environment, and assessed by an evaluator. For example, OpenAI’s reported 99.9% score for GPT-6 Astra on ARC-AGI-3, a benchmark of unfamiliar interactive environments, was obtained using a specified Responses API harness [3,4], while other work shows that changing the harness around a fixed model can alter performance and even reverse model rankings [5]. The role of execution control is also reflected in models designed to return structured decisions, such as TypeSafe’s Jev, whose typed outputs and confidence estimates allow surrounding software to route execution or request human review [6]. As harness design becomes an explicit target of optimization, distinguishing improvements in execution from advantages introduced by the evaluation procedure becomes essential: reported gains from automatic harness evolution can be overstated when search and final evaluation reuse the same tasks or when comparisons use unmatched feedback and inference budgets [7]. A benchmark score must therefore be interpreted in relation to both the complete execution configuration and the protocol used to evaluate it.

Figure 1. Overview of harness-aware evaluation of LLM agents. The model $\mathcal{M}$ operates through a harness $\mathcal{H}$, interacts with an environment $\mathcal{E}$, and is assessed by an evaluator $\mathcal{V}$. The survey organizes harness components, evaluation targets, and performance measurements to examine what reported results establish and where attribution remains limited.

Figure 1. Overview of harness-aware evaluation of LLM agents. The model $\mathcal{M}$ operates through a harness $\mathcal{H}$, interacts with an environment $\mathcal{E}$, and is assessed by an evaluator $\mathcal{V}$. The survey organizes harness components, evaluation targets, and performance measurements to examine what reported results establish and where attribution remains limited.

To formalize evaluation on a benchmark B, we define the evaluation configuration and its reported score as:

$$\mathcal{C}=(\mathcal{M},\mathcal{H},\mathcal{E},\mathcal{V}),S_B=f_B\left(\mathcal{C}\right),$$

(1)

where $\mathcal{M}$ denotes the backbone model; $\mathcal{H}$ denotes the execution harness that manages context, memory, tools, and control flow; $\mathcal{E}$ denotes the task environment in which actions are executed and observations are returned; and $\mathcal{V}$ denotes the evaluator that maps the resulting trajectory or final state to a score. Thus, $S_B$ characterizes the complete evaluation configuration rather than the model alone. Here, $f_B$ depends on the evaluation protocol, including task splits, resource budgets, and the number and aggregation of repeated runs. These conditions must therefore be specified when comparing evaluation configurations.

The environment can shape reported performance by introducing disruptions that stable benchmark settings do not capture: StressWeb exposes performance degradation through pop-up interruptions and other controlled changes to web interactions [8], while AgentNoiseBench reports similar sensitivity to noise in user instructions and tool responses [9]. The evaluator determines what constitutes success and represents the criteria of the benchmark designer, expert, or user. Since agentic tasks can admit multiple valid trajectories, a fixed evaluator may reject valid solutions or accept incomplete ones. In AgentRewardBench [10], rule-based evaluation underestimated expert-defined success by up to 18.5% and produced different model rankings. Conversely, in [11], an average of 7.8% of patches accepted on SWE-bench Verified across the studied tools failed the full developer test suites, showing that incomplete validation can overestimate agent success.

Necessity and Scope of This Survey 

Existing surveys examine agent architectures, optimization, harness engineering, evaluation, and trustworthiness from complementary perspectives, as summarized in Table 1. A key challenge is to connect these settings and perspectives into a coherent interpretation of benchmark results, making explicit the evaluation protocol, the components being assessed, and the conditions required for meaningful comparisons. This survey connects recent developments in harness design with the interpretation of agent benchmark results. We organize empirical evidence around the model, harness, environment, and evaluator, together with the protocol under which they are tested, examining which factors are varied, which are held fixed, and what conclusions the comparisons support. We also examine safety and human oversight within this framework, distinguishing execution controls from post-execution assessment. This synthesis provides factor-level reporting guidance for interpreting and comparing performance and safety claims across agent configurations.

Contributions 

This survey makes the following contributions:

• Unified evaluation taxonomy. We organize the model, harness, environment, and evaluator into a shared evaluation configuration and explain how evaluation protocols shape the interpretation of benchmark results (Section 3).
• Mapping of empirical evidence. We map representative benchmarks and studies to the factors they examine, covering task performance, safety, and evaluator reliability. This mapping distinguishes factors explicitly evaluated from those included in the execution setting (Section 4).
• Reporting guidance and research gaps. We provide factor-level guidance on what evaluations should report and control, and identify gaps in evaluating harness adaptation, resource use, and interactions between configuration factors (Section 3 and Section 5).

Literature Review Methodology 

This survey examines how agent execution mechanisms shape reported benchmark results through four research questions:

• RQ1: What constitutes an agent harness, and which mechanisms can affect performance?
• RQ2: How do the model, harness, environment, evaluator, and protocol shape benchmark scores, and what do existing comparisons establish?
• RQ3: How are harnesses evaluated, including safety behaviour and human oversight, and what can benchmarks measure?
• RQ4: What reporting is needed for interpretable, comparable, and reproducible evaluations?

RQ1–RQ3 are addressed in Section 2, Section 3, and Section 4, respectively. RQ4 is addressed through reporting guidance in Section 3 and research directions in Section 5.

Sources and selection. We used iterative, criterion-guided selection, combining citation tracking with searches of ACM Digital Library, IEEE Xplore, arXiv, OpenReview, and project documentation and repositories. The review covers January 2022 to September 2026, with earlier foundational work included where relevant. Preprints and system documentation were included to capture developments beyond archival publications.

Inclusion and exclusion. We included studies of harness mechanisms, empirical comparisons of agent configurations, and agent benchmarks, evaluators, or protocols. We prioritized sufficiently detailed execution settings to identify varied and fixed factors. For empirical comparisons, we excluded results whose execution configurations could not be identified. Foundational studies, capability benchmarks, and system documentation were retained where needed for background or contextual comparison. Selection favoured recent, mechanistically specific evidence; the corpus is illustrative rather than exhaustive.

Classification. We classified works by the configuration $\mathcal{C}=(\mathcal{M},\mathcal{H},\mathcal{E},\mathcal{V})$ and protocol, recording varied factors, fixed conditions, and reported evidence. Harness evidence was mapped to context and memory, tool interfaces, planning and execution, coordination, verification and recovery, and agent guardrails and permissions. Works could receive multiple assignments. This classification supports Table 4, Table 5 and Table 6.

Corpus overview. The corpus contains 110 works, of which 75% appeared in 2025–2026 and 56% are preprints or system documentation (Figure 2a,b). Evidence is distributed unevenly across the configuration: model (65), evaluator (36), and protocol (27); within the harness, planning and execution (26), context and memory (24), tool interfaces (22), guardrails and permissions (11), coordination (9), and verification and recovery (7). Counts indicate explicit variation or separate measurement, without requiring causal isolation. The vocabulary of the corpus reflects the same shift toward the harness: harness appears in 16 titles, all from 2025 onward, while 69% of titles containing language models predate 2025 (Figure 3). These coverage gaps inform Section 5.

2. Background and Definitions

This section traces the development from standalone LLMs to agent systems and harness-aware evaluation, as illustrated in Figure 4, and defines the components that shape agent execution and evaluation.

2.1. From LLMs to Agent Systems

LLMs started as autoregressive models to generate tokens conditioned on an input context [21]. Which later on, instruction tuning and human-feedback optimization improved their ability to follow user requests [22], while retrieval-augmented generation incorporated external information into response generation [23]. Continuously, work extended these capabilities toward interaction with external systems, such as ReAct which mixed reasoning and actions with environmental observations [24], while Toolformer explored learning when and how to invoke APIs and incorporate their results [25]. These developments supported a shift from generating responses to executing tasks through successive model calls and tool interactions. An execution harness supports this process by constructing context, exposing tools, dispatching proposed actions, and managing execution state and control flow [20]. The environment supplies the effects and observations of executed actions, while the evaluator assesses whether the resulting trajectory or final state satisfies the task.

2.2. From Agent Systems to Harness Engineering

Multi-step agent execution requires software to maintain state, coordinate model calls, and manage interactions with tools and other agents. Frameworks provide reusable infrastructure for these functions. AutoGen supports configurable conversations among agents, tools, and humans, while LangGraph represents execution through stateful graphs with persistence and human intervention [26,27]. OpenHands packages software-development agents with a runtime for code execution, command-line interaction, and web browsing [28]. These systems allow execution behavior to be configured and reused across tasks. Other frameworks make planning, verification, and coordination explicit through specialized roles. Workforce separates task decomposition, subtask coordination, and tool-based execution into a Planner, Coordinator, and Workers; OWL trains the planner using reinforcement learning from task feedback [29]. PlanGEN assigns constraint identification, verification, and algorithm selection to separate agents, allowing the planning procedure to adapt to individual problem instances [30]. These approaches illustrate how agent behavior depends on the organization of model calls and feedback, alongside the capabilities of the models performing them. Coding systems such as Claude Code and Codex CLI package these execution mechanisms into interfaces that operate directly on a development workspace. Claude Code integrates repository exploration, file editing, and command execution, while Codex CLI provides an open-source local coding agent with tools for interacting with the workspace [31,32]. Assistant systems extend execution to persistent services and messaging interfaces. OpenClaw connects a self-hosted agent to multiple communication channels, while NanoClaw provides a lightweight alternative with a smaller, customizable codebase and container-based agent execution [33,34]. These systems illustrate different approaches to packaging and deploying an agent harness. Table 2 summarizes representative systems and their execution mechanisms.

2.3. Agent Harnesses and Execution Control

An agent harness $\mathcal{H}$ is the software layer that connects model calls, external actions, and execution state into a running agent system [20]. It prepares the information available to the model, interprets proposed actions, and incorporates their results into subsequent steps. As tasks extend across multiple interactions, this execution layer must preserve relevant information, coordinate actions, and determine when to revise or stop.

Managing information across these interactions requires a distinction between context, which is supplied to a particular model call, and external memory, which retains information for later retrieval. Retrieval allows selected records to be brought into the current context, supporting continuity without retaining the entire interaction history in every call. Generative Agents used this approach to store experiences as natural-language memories, retrieve relevant records, and synthesize reflections for subsequent planning [39]. MemGPT addressed the related problem of limited context capacity by moving information between active context and external storage [40]. Together, these mechanisms determine which past observations remain available to guide future actions.

Acting on that information requires interfaces that connect model outputs to executable operations. Toolformer explored learning when and how to invoke APIs, while ToolLLM developed instruction-tuning data and methods for using a large collection of real-world APIs [25,41]. The Model Context Protocol (MCP) afterwards introduced a common protocol for connecting applications to servers exposing tools, resources, and prompts [42]. Within this interaction, the tool interface specifies available operations and their arguments, while the harness dispatches proposed calls and returns their results to the model.

Once an agent can retrieve information and invoke tools, planning organizes these operations around a task objective, including breaking complex tasks into smaller steps. Execution control then determines how the plan proceeds as new observations arrive. SayCan combined language-model assessments of useful actions with estimates of whether available skills could execute them [43], while ReAct incorporated observations into an iterative reasoning-and-action loop [24]. Feedback can also guide later attempts: Reflexion stored verbal reflections to support revision without updating model weights [44]. In multi-agent systems, execution control additionally coordinates responsibilities and exchanges among participants, as illustrated by AutoGen’s conversations among agents, tools, and humans [26].

These actions unfold within the task environment $\mathcal{E}$, which includes the workspace, applications, services, and external participants encountered during execution. The environment applies actions and returns observations that inform the next step; a service failure is therefore an environmental event, whereas retrying the request is a harness decision. Benchmark environments range from functional websites in WebArena [45] and software repositories in SWE-bench [1] to tools and databases shared with a simulated user in $\tau$-bench [2]. Gaia2 extends this interaction to asynchronous settings, where events can occur independently of the agent’s actions [46].

The resulting trajectory and final state provide evidence for the evaluator $\mathcal{V}$, which applies the task’s success criteria to determine the reported score. SWE-bench assesses code changes through tests [1], while $\tau$-bench compares final database states with annotated goal states [2]. Evaluation can also examine intermediate work and execution behaviour, as in Agent-as-a-Judge and AgentRewardBench [10,47]. The distinction depends on how assessment is used: feedback that triggers another action belongs to the harness’s execution process, whereas assessment that determines the reported benchmark result belongs to the evaluator.

3. Harness-Aware Evaluation Framework

Our framework organizes evaluation factors into four components: model, harness, environment, and evaluator. The evaluation protocol specifies how this configuration is tested, including task splits, resource budgets, repeated runs, and score aggregation. For each component, we examine empirical evidence, its limits, and the reporting requirements needed to interpret and compare benchmark results.

3.1. Model Factors

Model factors describe which backbone model $\mathcal{M}$ is used and the capabilities and limitations relevant to agent execution. Isolating them is difficult because $\mathcal{M}$ depends on many variables at once, from architecture and training to context capacity, tool-use capabilities, and decoding settings. Replacing $\mathcal{M}$ therefore changes several of these together, so evidence about model effects is best read according to how much of this variation each comparison holds fixed.

$\tau$-bench reports that GPT-4o and Claude 3 Opus outperformed open-weight models, but Llama 3 used a text-based ReAct interface because it lacked native function calling, so the gap is interpreted through $\mathcal{M}$ as well as the tools in $\mathcal{H}$ [2]. HoF-Bench fixes this issue by fixing the scaffold, and finds across five proprietary and five open-weight models that performance does not follow total parameter count [48]. Its generated-context intervention helps some models and hurts others, showing that even under a fixed harness a context change interacts with the model rather than acting uniformly. TUA-Bench likewise holds the Terminus-2 harness constant and reports variation across task categories and reasoning-effort settings [49], locating where model differences appear rather than attributing them to a particular capability.

Interpreting model comparisons requires explicit reporting of the exact model and checkpoint, access method, total and active parameters where available, context and output limits, supported modalities, and reasoning and decoding settings. Comparisons should hold $\mathcal{H}$, $\mathcal{E}$, $\mathcal{V}$, and the evaluation protocol fixed where possible, documenting any settings that cannot be matched across models because these differences limit attribution to the backbone alone.

3.2. Harness Factors

Harness factors characterize the operational conditions under which the model $\mathcal{M}$ is executed, where the harness $\mathcal{H}$ selects the information provided in each model call, translates the model output into actions, and uses the feedback returned to govern subsequent execution [50,51,52]. These responsibilities separate into context and state management, action interfaces, control flow, and in-loop verification, which we treat as distinct factors because each can alter the resulting trajectory without modifying the underlying model.

3.2.1. Context and Memory

Context and memory determine how $\mathcal{H}$ preserves and reconstructs task state over time, controlling which observations are retained, how older information is compressed or stored, and when external information is retrieved for later model calls.

Existing systems address these decisions at different levels. MemGPT treats the limited context window as virtual memory and moves information between the active context and external storage [40], managing capacity rather than content. Mem0 instead selects what is worth keeping, dynamically extracting and consolidating salient information from previous interactions before retrieving it when relevant, which improves long-term recall while reducing token use and latency [53]. A-MEM goes further and organizes memories as structured, interconnected notes whose attributes and links evolve as new information is added [54]. Across these levels, memory policy changes continuity, recall, and efficiency without modifying the backbone model.

Despite their impact on performance, the exact contribution of these memory mechanisms remains difficult to measure because current evaluations focus heavily on document analysis and conversational recall, offering little evidence for long-horizon tool use. Furthermore, because these benchmarks modify storage, retrieval, compression, and prompt construction simultaneously, the impact of each specific mechanism cannot be easily isolated. To resolve this, a system’s memory configuration should be explicitly reported as part of the harness $\mathcal{H}$, detailing its storage, retrieval, update, compression, and persistence policies. For clear attribution, comparisons must vary these memory policies independently while holding the model $\mathcal{M}$, environment $\mathcal{E}$, evaluator $\mathcal{V}$, and evaluation protocol fixed.

3.2.2. Tool Interfaces

Tool-interface factors specify how $\mathcal{H}$ presents available actions to the model and converts its outputs into executable calls. They include tool descriptions, schemas, action formats, parsing rules, and the observations returned after execution.

SWE-agent shows that interface design can change performance without modifying the model. With GPT-4 Turbo, its specialized search, viewing, and editing interface resolved 10.7% more SWE-bench Lite tasks than a shell-only interface [35]. Its ablations locate that gain in several separate choices, reporting sensitivity to search format, editing commands, and the amount of feedback returned, so the interface is not a single design decision but a set of them. The effect also extends beyond one system: $\tau$-bench finds that native function calling consistently outperforms text-formatted ReAct and Act interfaces for supported models [2]. These results identify the action interface as part of the evaluated harness rather than a neutral connection to tools.

That composite character is what makes interface effects difficult to isolate, since the tool descriptions, schemas, action formats, and returned observations are often changed together, and their effects may differ across models. Evaluations should therefore report these choices as part of $\mathcal{H}$ and vary them independently while holding $\mathcal{M}$, $\mathcal{E}$, $\mathcal{V}$, and the evaluation protocol fixed.

3.2.3. Planning and Execution

Planning and execution factors concern how $\mathcal{H}$ organizes model calls and actions into a task-solving process. They include decomposition policies, execution order, branching, plan revision, and termination. These choices structure how $\mathcal{M}$ is used rather than changing its underlying reasoning capabilities.

Work in this area has moved from fixing an execution policy by hand to searching for one, for example early work ReAct interleaves reasoning and actions with observations, allowing subsequent decisions to respond to execution feedback [24]. Also, AFlow extends workflow design into an optimization problem, searching over connected model calls using execution feedback [55]. Agentic Harness Engineering similarly evolves coding-agent harnesses using execution traces, making control logic an explicit optimization target [52]. Added structure is not reliably beneficial, however, Natural-Language Agent Harnesses reports that adding a multi-candidate search module reduces performance on the evaluated SWE-bench Verified tasks under the tested runtime and budget settings [51].

Taken together, these results show that adding execution steps does not consistently improve task performance and that assessing gains from harness evolution requires separate search and evaluation tasks with matched feedback and inference budgets [7].

Workflow structure, branching, revision, and stopping rules should therefore be documented as part of $\mathcal{H}$. Their effects should be tested under fixed $\mathcal{M}$, $\mathcal{E}$, and $\mathcal{V}$, with execution costs reported alongside task success.

3.2.4. Coordination

Coordination factors describe how $\mathcal{H}$ distributes work and information across agents and other participants. They include role assignment, delegation, communication, shared state, and handoff rules, which determine how individual contributions are combined into a task outcome.

These systems differ in how much of the coordination structure is fixed in advance. Magentic-One uses an Orchestrator to direct specialized agents and track progress, retaining the same collaboration structure across GAIA, AssistantBench, and WebArena [56]. The Workforce architecture introduced with OWL separates a Planner, a Coordinator, and tool-equipped Workers, allowing domain-specific workers to change without redesigning the entire system [29]. Intelligent AI Delegation extends this perspective beyond task allocation to the transfer of authority and responsibility, emphasizing explicit boundaries, monitoring, and accountability [57]. Each design leaves part of the handoff unobserved. In Workforce, sharing subtask results while keeping execution details local to workers limits cross-agent traceability unless additional logs are maintained, and distinguishing planner from worker errors does not establish whether user constraints survive delegation. The safeguards proposed for delegated authority remain correspondingly untested, since explicit boundaries and monitoring require empirical validation across handoffs rather than at the point of assignment.

Coordination policies in $\mathcal{H}$ should therefore specify how instructions, constraints, and state pass between agents and other participants, and how delegated actions are recorded. Evaluation should assess handoff failures, constraint preservation, and action traceability alongside task completion.

3.2.5. Verification and Recovery

Verification and recovery factors specify how $\mathcal{H}$ checks intermediate outcomes and responds to failure. Checks may be programmatic, model-based, or human-in-the-loop (HITL), triggering correction, retries, replanning, or escalation. Their role is to guide execution rather than assign the final benchmark score $\mathcal{V}$. Existing systems differ in who performs the check. OWL’s Workforce keeps it inside the acting agent, using worker self-assessment to report subtask failures and trigger planner revision [29]. Verified Multi-Agent Orchestration separates the two roles, using an LLM-based verifier to identify incomplete results and guide replanning [58]. Magentic-UI enables human review and intervention, allowing users to correct plans and redirect execution [38]. Because these three arrangements are reported on different tasks and feedback mechanisms, the results do not establish when worker self-assessment, a separate verifier, or human intervention is preferable. A controlled comparison is needed to distinguish verification accuracy from the ability to recover after an error, accounting for both execution cost and human effort.

Reports should specify check criteria, feedback sources, and recovery triggers and limits within $\mathcal{H}$. Comparisons should measure recovery success alongside computational and human effort, holding $\mathcal{M}$, $\mathcal{E}$, and $\mathcal{V}$ fixed.

3.2.6. Guardrails and Permissions

Guardrails and permissions specify how $\mathcal{H}$ restricts actions to authorized and safe behaviour. They include tool-access policies, approval requirements, and controls on privilege escalation. These restrictions determine which paths to task completion are permissible, even when broader access would improve success.

Work here divides between measuring unsafe behaviour and preventing it. OS-Harm takes the first approach, evaluating malicious requests, prompt injections, and unsafe behaviour during computer use, and documenting agents that follow injected instructions and supply administrative credentials to execute suspicious scripts [59]. SafeMCP takes the second, addressing expanding agent capabilities through server-side tool filtering, using predicted environmental consequences to restrict risky tools before action selection [60].

Neither approach yields an unambiguous reading. A low harmful-task completion rate need not demonstrate effective guardrails, since OS-Harm notes that limited agent capabilities can prevent harmful outcomes on their own, leaving restriction and incapability indistinguishable in the reported score. SafeMCP’s protection in turn depends on modeling environment dynamics, and transferring its safety reasoning without extensive local data remains unresolved. Its simulated evaluations also do not establish protection against real sandbox breaches.

Reports should therefore distinguish model refusals from enforced restrictions and disclose available privileges, approval rules, and enforcement points. Comparisons should measure unsafe actions and benign-task completion together, explicitly accounting for permission differences rather than attributing gains from broader access solely to model capability.

3.3. Environment Factors

Environment factors determine the demands placed on an agent: task dependencies require maintaining consistency across actions, incomplete user instructions require clarification, and unreliable tools require adapting to missing or incorrect observations. Changes in $\mathcal{E}$ can therefore alter both task difficulty and the effectiveness of a given model–harness configuration.

AgentNoiseBench shows that perturbing user instructions and tool responses reduces performance even when tasks remain solvable, demonstrating that success depends on the quality of external information [9]. StressWeb finds particularly severe degradation when interaction rules change, exposing difficulty adapting previously useful action strategies [8]. Gaia2 adds temporal demands: its strongest overall model struggles on time-sensitive tasks, showing that higher aggregate accuracy does not imply effective execution when the environment changes during deliberation [46].

However, testing environmental changes separately leaves their combined effects unclear, while simulated environments omit some dependencies between applications. Thus, performance under isolated perturbations does not establish robustness when ambiguous requests, tool failures, and deadlines interact across connected services.

Evaluations should report results by task demands and environmental condition, alongside aggregate scores. Paired comparisons should vary user behaviour, tool reliability, or event timing while retaining the same task objective and fixed $\mathcal{M}$, $\mathcal{H}$, and $\mathcal{V}$. Environment versions, initial states, and reset procedures should be recorded to make these comparisons reproducible.

3.4. Evaluator Factors

Evaluator factors specify what constitutes success or failure and how $\mathcal{V}$ determines whether those criteria are satisfied. They cover outcome correctness, required execution constraints, efficiency, and the evidence used for assessment. Failure may reflect an unmet goal or a constraint violation; following a different valid solution path should not itself imply failure.

AppWorld uses state-based tests to accommodate alternative solutions while detecting unintended changes [61]. Agent-as-a-Judge examines intermediate work against detailed user requirements [47], while AgentRewardBench annotates task success, side effects, and repetitive actions separately [10]. These approaches distinguish achieving the requested outcome from the quality of the execution. Assessment may use programmatic checks, human review, or model-based judges, including agents that gather additional evidence.

Evaluation design also determines what a score can conceal. MemEye pairs multiple-choice and open-ended versions of the same questions [62], HaluMem decomposes end-task answering into operation-level checks that separate omission from hallucination [63], and ContextBench adds annotated gold contexts to outcome-based issue resolution [64].

However, restrictive checks can reject valid alternatives, while incomplete evidence can conceal violations. AgentRewardBench also finds that no single LLM judge performs best across its benchmarks, limiting the transfer of judge reliability across settings. Such errors distort conclusions about $\mathcal{M}$, $\mathcal{H}$, and $\mathcal{E}$; when judgments also guide execution, they can influence the trajectory itself.

Reports should disclose success criteria, accepted alternatives, evidence inputs, and evaluator configurations. Correctness, constraint compliance, and efficiency should be reported separately, with any aggregation rule made explicit. Automated judgments should be validated against human-adjudicated cases, and available confidence estimates calibrated against observed correctness. Final scoring should remain independent of feedback used to optimize or guide execution.

3.5. Evaluation Protocol

A reported score is produced by a measurement procedure as well as by a configuration, and two evaluations of the same $\mathcal{C}$ can return different numbers when that procedure differs. We therefore describe the evaluation protocol as

$$\mathcal{P}=(\mathcal{T},\mathbf{b},k,\mathcal{A},\mathcal{F}),$$

(2)

where $\mathcal{T}$ is the set of tasks on which the score is computed, which may overlap with the tasks used while developing the configuration, $\mathbf{b}$ is the per-task budget of steps, tool calls, tokens, and wall-clock time, k is the number of rollouts per task, $\mathcal{A}$ is the rule that aggregates those rollouts into the reported number, and $\mathcal{F}$ is the evaluator feedback available while the configuration was selected.

The score in Eq. (1) is then written $S_B=f_B(\mathcal{C};\mathcal{P})$, which separates the system being measured from the procedure measuring it. Since $\mathcal{P}$ can change $S_B$ while $\mathcal{C}$ remains fixed, its disclosure is a precondition for reproducing a result and for attributing any difference between two results to a factor of $\mathcal{C}$.

The budget $\mathbf{b}$ bounds what a configuration may do per task, so a reported score is one point on a function $S_B(\mathcal{C};\mathbf{b})$ and one configuration dominates another only when

$$S_B({\mathcal{C}}_2;\mathbf{b})\ge S_B({\mathcal{C}}_1;\mathbf{b}),\forall \mathbf{b}\in \mathcal{B},$$

(3)

where $\mathcal{B}$ is the budget range of interest. A bounded budget also couples harness components that are otherwise analyzed separately, since verification, replanning, and coordination consume the same allowance as task progress, so the measured effect of adding such a mechanism nets its benefit against the cost of the actions it displaces.

Natural-language agent harnesses report both signs of this effect under fixed runtime and budget settings, where a verifier stage moves resolution from $73.0\%$ to $73.2\%$ and a multi candidate search module reduces it from $73.0\%$ to $71.4\%$ [51]. Furthermore, $\mathbf{b}$ is a vector whose entries are not exchangeable, so equal call limits do not imply comparable resource use and no scalar normalization is neutral. Reporting performance against cost, as in the scaling analyses of Gaia2 [46], is therefore more informative than reporting a single matched limit.

Because execution is stochastic, the pair $(k,\mathcal{A})$ determines which quantity the reported number estimates. Let $p_{\tau }$ be the probability that a single rollout of task $\tau$ satisfies $\mathcal{V}$ and $c_{\tau }$ the successes observed in k rollouts. The two common aggregation rules are estimated by

$$\begin{array}{cc}\hfill \mathrm{pass}@m& ={\mathbb{E}}_{\tau }\left[1-\left({\displaystyle \genfrac{}{}{0pt}{}{k-c_{\tau }}{m}}\right)/\left({\displaystyle \genfrac{}{}{0pt}{}{k}{m}}\right)\right],\hfill \\ \hfill {\mathrm{pass}}^m& ={\mathbb{E}}_{\tau }\left[\left({\displaystyle \genfrac{}{}{0pt}{}{c_{\tau }}{m}}\right)/\left({\displaystyle \genfrac{}{}{0pt}{}{k}{m}}\right)\right],\hfill \end{array}$$

(4)

where $m\le k$ is the number of attempts considered. The first estimates the probability that at least one attempt succeeds and increases with m, the second estimates the probability that all attempts succeed and decreases with m, and the two coincide for $m=1$; for $m>1$, their population values coincide only when $p_{\tau }\in \{0,1\}$ for every evaluated task.

Their separation therefore measures execution reliability rather than capability, which is what $\tau$-bench exposes when single-trial success above $60\%$ accompanies ${\mathrm{pass}}^8$ below $25\%$ [2]. E-Bench reports $\mathrm{Avg}@3$, $\mathrm{Pass}@3$, and ${\mathrm{Pass}}^3$ from the same rollouts, so $\mathcal{A}$ changes the headline number without changing any execution [65].

The trial count k fixes the precision of the estimate rather than its target. For ${\widehat{S}}_B={\left|\mathcal{T}\right|}^{-1}{\sum }_{\tau }{c}_{\tau }/k$, assuming tasks are sampled independently from the same task distribution and each task’s rollouts are independent success-or-failure trials with probability $p_{\tau }$, the sampling variance separates as

$$\mathrm{Var}\left({\widehat{S}}_B\right)={\displaystyle \frac{{\mathrm{Var}}_{\tau }\left(p_{\tau }\right)}{\left|\mathcal{T}\right|}}+{\displaystyle \frac{{\mathbb{E}}_{\tau }\left[p_{\tau }(1-p_{\tau })\right]}{\left|\mathcal{T}\right|k}},$$

(5)

where the first term reflects heterogeneity across tasks and the second reflects run-to-run instability. Additional rollouts reduce only the second term, so repeated trials characterize variability on $\mathcal{T}$ without extending a claim beyond it, while additional tasks are required to reduce the first. At $k=1$, task heterogeneity and run-to-run variability cannot be estimated separately without additional assumptions. Although uncertainty across sampled tasks can still be assessed, repeatability on individual tasks remains unmeasured. Single-rollout reporting nevertheless remains common in Table 5 and Table 6, with explicit uncertainty confined to benchmarks such as Terminal-Bench 2.0 and SandboxEscapeBench [66,67].

A protocol becomes a comparison instrument once it is held fixed across configurations. For two harnesses, the harness effect is the contrast

$${\Delta }_{\mathcal{H}}=S_B(\mathcal{M},{\mathcal{H}}_2,\mathcal{E},\mathcal{V};\mathcal{P})-S_B(\mathcal{M},{\mathcal{H}}_1,\mathcal{E},\mathcal{V};\mathcal{P}),$$

(6)

where $\mathcal{M}$, $\mathcal{E}$, $\mathcal{V}$, and $\mathcal{P}$ are unchanged between the two terms, and analogous contrasts define the effects of the remaining factors and of individual components within $\mathcal{H}$. Sweeping one factor while locking the rest is what permits a score difference to be assigned to that factor, whereas changing several factors together yields a configuration-level result that should be labelled as such. However, ${\Delta }_{\mathcal{H}}$ is conditional on everything held fixed, so a claim of general benefit concerns the average effect

$${\overline{\Delta }}_{\mathcal{H}}={\mathbb{E}}_{\mathfrak{D}}\left[{\Delta }_{\mathcal{H}}\right],\mathfrak{D}={\left\{({\mathcal{M}}_i,{\mathcal{E}}_i)\right\}}_{i=1}^n,$$

(7)

where $\mathfrak{D}$ is the declared set of n model and environment pairings over which the contrast is averaged. This distinction matters because the factors interact, as HoF-Bench shows by holding the scaffold fixed and finding that a generated-context intervention helps some models and harms others [48].

The remaining components, $\mathcal{T}$ and $\mathcal{F}$, determine whether a score describes performance or selection. When N candidates are scored on the same tasks and the best is both nominated and reported,

$$\mathbb{E}\left[\underset{j\le N}{max}{\widehat{S}}_B^{\left(j\right)}\right]\ge \underset{j\le N}{max}\mathbb{E}\left[{\widehat{S}}_B^{\left(j\right)}\right],$$

(8)

so selecting and reporting the highest score on the same evaluation tasks can introduce optimistic selection bias, whose magnitude depends on the number of candidates, their underlying performance differences, and the variability and dependence of their estimation errors. Consistent with this, gains from harness evolution shrink once search and final evaluation use separate tasks under matched budgets [7], and HarnessOpt-Bench applies the corresponding control by grading the nominated candidate on a held-out partition [68].

We do not propose a consolidated score, since each choice in Eq. (2) selects a different estimand and standardizing one would conceal the variation this survey aims to expose. Reports should instead disclose every component of $\mathcal{P}$: the task split $\mathcal{T}$, the budget vector $\mathbf{b}$, the trial count k, the aggregation rule $\mathcal{A}$ with the estimand it targets, and any feedback $\mathcal{F}$ used during configuration selection.

Comparisons should declare the factor being swept, hold the remainder of $\mathcal{C}$ and all of $\mathcal{P}$ fixed, and report execution cost and uncertainty alongside task success, so that a gain obtained by spending more is distinguishable from a gain obtained by executing better. A protocol reported at this level turns a benchmark number from a summary of one run into evidence about a configuration, which is the condition under which the comparisons collected in Table 3 can be read as effects of the model, the harness, the environment, or the evaluator.

4. Benchmarks and Methods for Evaluating Harnesses

This section examines how existing benchmarks evaluate harnesses and their components, first through task performance and then through safety and human oversight.

Table 4. Selected empirical evidence mapped to the evaluation configuration and harness factors. Column groups organize benchmarks by evaluation focus. Task settings and reported measurements appear in Table 5 and Table 6.

	Harnessdesign				Contextand memory					Tooluse						Taskexecution						Safety			Judges		Modelcapability
Framework factor	Harness-Bench [69]	WildClawBench [70]	HarnessOpt-Bench  a [68]	SHOR  a [71]	SkillsBench [72]	MemBench [73]	HaluMem [63]	MemEye [62]	ContextBench [64]	LiveMCPBench [74]	$\mathbf{\tau }$-Bench [2]	${\mathbf{\tau }}^{\mathbf{2}}$-Bench [75]	MobileWorld [76]	E-Bench [65]	MobilityBench [77]	OSWorld [78]	WebArena [45]	Terminal-Bench 2.0 [66]	Mind2Web [79]	PARTNR [80]	Gaia2 [46]	Claw-Eval [81]	SandboxEscapeBench [67]	OS-Harm [59]	AgentRewardBench [10]	Mind2Web 2 [82]	EDIT-Bench [83]	BIG-bench [84]
Model $\mathcal{M}$	✓	✓	✓	✓	✓	✓		✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓		✓	✓
Harness $\mathcal{H}$
Configuration	✓	✓	✓	✓					✓									✓
Context and memory		✓			✓	✓	✓	✓	✓		✓	✓				✓			✓	✓			✓	✓			✓	✓
Tool interfaces	✓									✓	✓		✓	✓	✓
Planning and execution	✓														✓		✓	✓		✓	✓
Coordination												✓								✓	✓
Verification and recovery	✓																					✓
Guardrails and permissions b
Environment $\mathcal{E}$												✓									✓	✓		✓
Evaluator $\mathcal{V}$		✓						✓	✓	✓										✓		✓		✓	✓	✓
Evaluation protocol		✓	✓	✓							✓	✓		✓				✓	✓		✓	✓	✓	✓		✓		✓

Notation: ✓ explicit comparison or factor-specific measurement; relevant coverage without separate measurement; blank not mapped. Marks indicate reported evidence, not causal isolation. ^a Includes optimizer models and harnesses. ^b Safety outcomes do not establish guardrail effectiveness.

Table 4. Selected empirical evidence mapped to the evaluation configuration and harness factors. Column groups organize benchmarks by evaluation focus. Task settings and reported measurements appear in Table 5 and Table 6.

	Harnessdesign				Contextand memory					Tooluse						Taskexecution						Safety			Judges		Modelcapability
Framework factor	Harness-Bench [69]	WildClawBench [70]	HarnessOpt-Bench  a [68]	SHOR  a [71]	SkillsBench [72]	MemBench [73]	HaluMem [63]	MemEye [62]	ContextBench [64]	LiveMCPBench [74]	$\mathbf{\tau }$-Bench [2]	${\mathbf{\tau }}^{\mathbf{2}}$-Bench [75]	MobileWorld [76]	E-Bench [65]	MobilityBench [77]	OSWorld [78]	WebArena [45]	Terminal-Bench 2.0 [66]	Mind2Web [79]	PARTNR [80]	Gaia2 [46]	Claw-Eval [81]	SandboxEscapeBench [67]	OS-Harm [59]	AgentRewardBench [10]	Mind2Web 2 [82]	EDIT-Bench [83]	BIG-bench [84]
Model $\mathcal{M}$	✓	✓	✓	✓	✓	✓		✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓		✓	✓
Harness $\mathcal{H}$
Configuration	✓	✓	✓	✓					✓									✓
Context and memory		✓			✓	✓	✓	✓	✓		✓	✓				✓			✓	✓			✓	✓			✓	✓
Tool interfaces	✓									✓	✓		✓	✓	✓
Planning and execution	✓														✓		✓	✓		✓	✓
Coordination												✓								✓	✓
Verification and recovery	✓																					✓
Guardrails and permissions b
Environment $\mathcal{E}$												✓									✓	✓		✓
Evaluator $\mathcal{V}$		✓						✓	✓	✓										✓		✓		✓	✓	✓
Evaluation protocol		✓	✓	✓							✓	✓		✓				✓	✓		✓	✓	✓	✓		✓		✓

Notation: ✓ explicit comparison or factor-specific measurement; relevant coverage without separate measurement; blank not mapped. Marks indicate reported evidence, not causal isolation. ^a Includes optimizer models and harnesses. ^b Safety outcomes do not establish guardrail effectiveness.

4.1. Benchmarks and Evaluation Frameworks

Table 4 maps benchmark evidence to the factors in our framework, while Table 5 and Table 6 summarize evaluation settings, comparisons, and measurements.

4.1.1. Harness Configurations and Optimization

To evaluate harness effect on agents Harness-Bench evaluates multiple model–harness combinations while fixing task prompts, initial sandbox states, budgets, and evaluation procedures. WildClawBench evaluates models within native CLI runtimes and additionally compares selected models across different harnesses [69,70]. Both report performance variation with harness choice. Harness-Bench measures completion, process quality, security, and resource usage, while WildClawBench further shows that execution latency can reduce task scores when agents reach the time limit before producing the required artifacts.

These comparisons measure the combined effect of the mechanisms implemented by each harness. Since native prompting, tool interfaces, state management, and recovery policies vary together, the results do not identify which component accounts for each difference. Their environmental coverage also differs. Harness-Bench focuses on controlled offline workflows, limiting its coverage of live services and long-term memory. WildClawBench uses real tools but currently provides a single initial instruction, leaving user clarification and correction during execution untested.

Harness optimization benchmarks examine whether an agent can improve these execution mechanisms. HarnessOpt-Bench provides an optimizer with a target agent’s seed harness, evaluation feedback, and a fixed search budget. The optimizer modifies the harness, and the selected candidate is evaluated by its improvement on a held-out test partition. SHOR examines the preceding decision of which component to modify by evaluating priority rankings over prompts, memory, tools, and workflows [68,71]. HarnessOpt-Bench therefore measures implemented improvements, whereas SHOR provides a less expensive assessment of update prioritization that correlates with downstream optimization performance.

The optimizer and target agent must be distinguished in these comparisons. In HarnessOpt-Bench, optimizer models and coding harnesses are compared while the target model, environment, and verifier remain protected within each task setting. The resulting improvement depends on the seed harness and available feedback, and transfer to other target models and runtimes remains untested. SHOR avoids repeated execution during ranking, but identifying a promising component does not establish that the optimizer can implement an effective change. Its four-component abstraction also leaves finer interactions and multi-agent configurations outside the evaluation.

4.1.2. Context, Memory, and Procedural Knowledge

Context and memory benchmarks evaluate whether agents receive the information needed for subsequent decisions. SkillsBench [72] examines the effect of providing procedural knowledge through curated skill packages, while MemBench [73] evaluates factual and reflective memory under simulated interactions. ContextBench focuses on retrieving relevant code during issue resolution [64]. These benchmarks examine different sources of information within $\mathcal{H}$: supplied task guidance, retained interaction history, and context retrieved from the task environment.

Their evaluation methods reflect these differences. SkillsBench compares task pass rates with and without skills, reporting an average improvement of 16.6% across the evaluated model–harness configurations. MemBench measures memory accuracy, recall, capacity, and temporal efficiency, while ContextBench measures retrieval coverage, precision, and efficiency at file, block, and line levels. Task completion measures the overall outcome, while memory and retrieval metrics help locate failures in information preservation and selection.

HaluMem [63] and MemEye [62] further examine whether retained information remains correct and applicable. HaluMem evaluates memory extraction, updating, and question answering after individual sessions. Its results show that omitted information can prevent later updates, allowing low update-hallucination rates to coexist with high omission rates. MemEye evaluates visual memory through multiple-choice and open-ended questions, together with retrieval diagnostics that identify the selection of outdated visual evidence. These findings show why memory evaluation must consider both information coverage and the validity of the retrieved state.

The contribution of the memory mechanism remains dependent on the surrounding configuration. MemBench reports performance changes across backbone models, while SkillsBench measures the combined effect of skill content, supporting resources, and their availability through the harness.HaluMem could not evaluate Zep’s memory extraction because its API did not expose all memories associated with a session. Consequently, comparisons should identify which memory operations are accessible and which model and context settings are shared before attributing differences to memory design.

4.1.3. Tool Use and User Interaction

Tool-use benchmarks examine how agents identify suitable operations, invoke them, and combine their results to complete a task. LiveMCPBench [74] judges task completion from annotated requirements, execution trajectories, and tool descriptions; its reported procedure does not include a separate final-state check. The $\tau$ benchmarks examine tool use under domain policies and user interaction, with ${\tau }^2$-Bench extending the setting so that both the agent and simulated user can act on a shared environment [2,75]. These settings connect tool selection and execution in $\mathcal{H}$ with service behaviour and user actions in $\mathcal{E}$.

The failure analyses distinguish several stages of tool use. LiveMCPBench separates unsuitable search queries, failures to retrieve an available tool, and incorrect tool invocation. This distinction matters because improving model-generated queries and improving the retrieval mechanism address different causes of failure. Its model comparisons use a shared retrieval module, so they measure performance within that routing design rather than the relative effectiveness of alternative routers.

MobileWorld extends tool use to workflows combining graphical actions, MCP calls, and user clarification. It reports task success together with execution steps, user queries, interaction quality, and MCP-call counts [76]. These measurements help identify whether an agent completes a task through effective interaction or through excessive and unsuccessful actions. Comparisons between its agentic frameworks and end-to-end models nevertheless involve differences in supported actions and task coverage, limiting attribution of the performance gap to the backbone alone.

E-Bench compares API-based execution with an additional code-execution tool, providing a direct intervention on the available action set. MobilityBench instead reports separate measurements for instruction understanding, planning, tool use, and decision making within a deterministic replay environment [65,77]. The former measures the effect of extending tool access, while the latter identifies execution bottlenecks without establishing the relative effectiveness of alternative harnesses.

Assessment introduces additional constraints. LiveMCPBench relies on trajectories and tool descriptions without independently verifying final environmental effects, while the user-interaction settings depend on simulated participants. Thus, a benchmark may reveal tool-selection or communication failures without establishing reliability under different service behaviour, user responses, or assessment procedures.

4.1.4. Planning, Coordination, and Dynamic Execution

Interactive benchmarks evaluate whether agents can execute a sequence of actions and reach a verifiable task outcome. OSWorld, WebArena, and Terminal-Bench provide desktop, web, and terminal environments, respectively. PARTNR examines collaborative planning under spatial, temporal, and agent-capability constraints, while Gaia2 introduces dynamic and asynchronous events during execution [45,46,66,78,80]. These benchmarks exercise planning, progress tracking, coordination, and recovery across different task conditions.

Their analyses provide different levels of detail about execution. OSWorld varies observation formats and history length to examine how the information available to the agent affects performance. PARTNR identifies difficulties in coordination, task tracking, and recovery, while Gaia2 evaluates agents when events occur during ongoing tasks. Such analyses help explain failures that a final completion score alone cannot distinguish, including incorrect action selection, loss of task state, and failure to respond to environmental changes.

What these results establish about the harness depends on the comparison: varying execution policies while keeping the model and task conditions fixed provides evidence about $\mathcal{H}$, whereas varying models under a shared execution policy measures model-dependent performance within that harness. Changing event timing or service behaviour instead examines robustness to $\mathcal{E}$. These benchmarks provide different forms of harness evidence. WebArena compares action-generation prompts, OSWorld varies observation presentation and history, and Terminal-Bench compares agent scaffolds across supported models. Gaia2 additionally varies multi-agent collaboration and environmental timing. The resulting comparisons concern different mechanisms: information presentation, execution control, coordination, and responsiveness to external events. Their results therefore need to be interpreted at the level of the reported intervention, rather than as interchangeable measures of harness quality.

4.2. Safety and Human Oversight

Safety benchmarks examine whether agents perform harmful actions or cross execution boundaries.

OS-Harm [59] evaluates deliberate misuse, prompt injection, and unsafe behaviour during computer use, while SandboxEscapeBench [67] measures the ability to exploit known vulnerabilities to escape a container. These settings expose different risks: an agent may misuse an available operation without escaping its environment, whereas container escape crosses an isolation boundary. SandboxEscapeBench compares models under a shared scaffold, so its results characterize escape capability within that configuration rather than the effectiveness of alternative harness guardrails.

Defense comparisons examine whether restrictions prevent unsafe actions while preserving legitimate task completion. AgentDojo evaluates prompt-based defenses, injection detection, and tool filtering under a shared acting model. Several defenses reduce attack success while improving benign-task completion, whereas false positives from the injection detector reduce utility. Tool filtering limits attacks that require operations unnecessary for the legitimate task, but remains vulnerable when both goals use the same tools or required tools cannot be identified in advance [85]. These results show that the safety–performance relationship depends on the defense and task requirements; fewer restrictions do not consistently imply better task performance.

Safety assessment also depends on the evidence available to the evaluator. Claw-Eval compares grading supported by execution traces, audit logs, and state snapshots with transcript-only assessment, finding that the latter misses safety violations. OS-Harm examines agreement with human annotations and reports judge failures associated with incomplete observations and misinterpretation of agent behaviour [59,81]. ATBench evaluates safety detection and diagnosis over 1,000 human-audited agent trajectories. GPT-5.4 achieves 76.7% F1 for safety classification but only 33.6% for risk-source diagnosis [86]. These analyses concern $\mathcal{V}$: detecting a violation after execution does not demonstrate that a runtime mechanism could prevent it. A reviewer belongs to $\mathcal{H}$ when its decision blocks an action or redirects execution.

Human oversight introduces opportunities to review plans, approve actions, and correct ongoing execution.

Magentic-UI implements these mechanisms and evaluates collaboration through simulated-user experiments and a separate qualitative study with 12 participants [38]. Its reported GAIA improvement uses simulated assistance, including a condition with additional task information, and does not isolate the safety contribution of approval gates. The human study provides evidence about usability and perceived control, with latency and model errors remaining concerns. Evaluating approval mechanisms therefore requires examining whether interventions prevent unsafe actions, whether legitimate actions are unnecessarily interrupted, and how much reviewer effort is required.

Table 5. Benchmark reproduction reference: harness design, context and memory, and tool use. Results apply to the named configurations; sizes retain their original units. Framework mapping appears in Table 4.

Benchmark	$\mathcal{H}$ cmp.	Size	${\mathit{N}}_{\mathit{M}}$	Provision	Budget	Trials	Evaluator	Reference configuration	Headline result
Harness design
Harness-Bench [69]	✓	106 tasks	8	Offline sandboxed workflows	Fixed budget	-	Outcome checks, process rubrics, and a binary security gate	NanoBot; mean over 8 models	Score 76.2%
WildClawBench [70]	✓	60 tasks	19	Containerized native CLI runtimes	8 min wall-clock; 20 tool calls/run	-	Rule and state checks with LLM/VLM assessment	Claude Opus 4.7 / OpenClaw	Score 62.2%
HarnessOpt-Bench [68]	✓c	4 tasks	5a	Container and optimizer loop	Fixed target-evaluation	-	Held-out grading of the nominated candidate	Claude Opus 5 / opencode; OfficeQA	Norm. gain 0.63
SHOR [71]	✓c	182 tasks	8a	None; rankings are predicted without execution	-	-	Predicted rankings against consensus labels	DeepSeek-V4-Pro / OpenHands-CLI	Acc@1 30.5%
Context and memory
SkillsBench [72]	✓	87 tasks	18	Containers with skill packages mounted	8K token context limit	3	Executable tests under paired skill conditions	Gemini 3.1 Pro / Gemini CLI; no skills → curated	Pass $36.0\to 60.8$%
MemBench [73]	✓	∼53k QA	4	Replayed interaction histories	100k token context limit	1	Comparison with annotated references	Qwen2.5-7B / RetrievalMemory; participation	Accuracy 83.3%
HaluMem [63]	✓	3,467 QA	1	Multi-session dialogue corpus	Long context	1	Operation-level extraction, update, and QA checks	GPT-4o / MemOS; HaluMem-Long	Accuracy 64.44%
MemEye [62]	✓	742 QA	4	Visual interaction histories	Oracle evidence	1	Multiple-choice matching with open-ended judging	GPT-5.4-mini; oracle evidence, Y3	Judge accuracy 55.8%
ContextBench [64]	✓	1,136 tasks	4	Repositories with annotated gold contexts	180 s step / 3600 s setup timeout	1	Overlap between retrieved and annotated context	Sonnet 4.5 / mini-SWE-agent; Lite (500)	Line F1 34.4%
Tool use
LiveMCPBench [74]	✗	95 tasks	12	Deployable MCP server suite	-	1	LLM assessment of execution evidence, human-validated	Claude Sonnet 4 / MCP Copilot	Success 78.95%
$\tau$-Bench [2]	✓	165 tasks	12	Simulated user and domain databases	30 agent actions	8	Final database state against the annotated goal	GPT-4o / function calling; domain macro-average	${\mathrm{pass}}^1$ 48.2%
${\tau }^2$-Bench [75]	—	279 tasks	4	Shared environment under dual control	-	4	Verifiable task outcomes	GPT-4.1; default telecom setting	${\mathrm{pass}}^1$ 34%
MobileWorld [76]		201 tasks	12	Mobile emulator, MCP servers, simulated user	50 steps	1	Functional state checks with interaction rating	GPT-5 + UIIns-7B	Success 51.7%
E-Bench [65]	✓	323 tasks	11	Product database APIs	-	3	Database changes against the ground-truth diff	Kimi-K3; base tool setting	Avg@3 73.8%
MobilityBench [77]	—	100k episodes	11	Deterministic API replay sandbox	10 steps	-	Outcome checks with intermediate capability assessment	Gemini 3 Pro Preview / ReAct; 7,098 episodes	FPR decision making 69.09%

$\mathcal{H}$ cmp.: ✓ harness or component varied; complete-system comparison only; ✗ harness fixed in a model comparison; — none reported. Marks describe reported comparisons rather than causal isolation. $N_M$: evaluated models, excluding auxiliary simulators and judges unless marked. Trials: ollouts per task; NR indicates not reported, and N/A indicates no interactive rollout. A slash follows the order named in the configuration; an arrow denotes a within-paper comparison. ^a95% confidence interval, implying repeated trials. ^cOptimizer and target harnesses are distinct.

Table 5. Benchmark reproduction reference: harness design, context and memory, and tool use. Results apply to the named configurations; sizes retain their original units. Framework mapping appears in Table 4.

Benchmark	$\mathcal{H}$ cmp.	Size	${\mathit{N}}_{\mathit{M}}$	Provision	Budget	Trials	Evaluator	Reference configuration	Headline result
Harness design
Harness-Bench [69]	✓	106 tasks	8	Offline sandboxed workflows	Fixed budget	-	Outcome checks, process rubrics, and a binary security gate	NanoBot; mean over 8 models	Score 76.2%
WildClawBench [70]	✓	60 tasks	19	Containerized native CLI runtimes	8 min wall-clock; 20 tool calls/run	-	Rule and state checks with LLM/VLM assessment	Claude Opus 4.7 / OpenClaw	Score 62.2%
HarnessOpt-Bench [68]	✓c	4 tasks	5a	Container and optimizer loop	Fixed target-evaluation	-	Held-out grading of the nominated candidate	Claude Opus 5 / opencode; OfficeQA	Norm. gain 0.63
SHOR [71]	✓c	182 tasks	8a	None; rankings are predicted without execution	-	-	Predicted rankings against consensus labels	DeepSeek-V4-Pro / OpenHands-CLI	Acc@1 30.5%
Context and memory
SkillsBench [72]	✓	87 tasks	18	Containers with skill packages mounted	8K token context limit	3	Executable tests under paired skill conditions	Gemini 3.1 Pro / Gemini CLI; no skills → curated	Pass $36.0\to 60.8$%
MemBench [73]	✓	∼53k QA	4	Replayed interaction histories	100k token context limit	1	Comparison with annotated references	Qwen2.5-7B / RetrievalMemory; participation	Accuracy 83.3%
HaluMem [63]	✓	3,467 QA	1	Multi-session dialogue corpus	Long context	1	Operation-level extraction, update, and QA checks	GPT-4o / MemOS; HaluMem-Long	Accuracy 64.44%
MemEye [62]	✓	742 QA	4	Visual interaction histories	Oracle evidence	1	Multiple-choice matching with open-ended judging	GPT-5.4-mini; oracle evidence, Y3	Judge accuracy 55.8%
ContextBench [64]	✓	1,136 tasks	4	Repositories with annotated gold contexts	180 s step / 3600 s setup timeout	1	Overlap between retrieved and annotated context	Sonnet 4.5 / mini-SWE-agent; Lite (500)	Line F1 34.4%
Tool use
LiveMCPBench [74]	✗	95 tasks	12	Deployable MCP server suite	-	1	LLM assessment of execution evidence, human-validated	Claude Sonnet 4 / MCP Copilot	Success 78.95%
$\tau$-Bench [2]	✓	165 tasks	12	Simulated user and domain databases	30 agent actions	8	Final database state against the annotated goal	GPT-4o / function calling; domain macro-average	${\mathrm{pass}}^1$ 48.2%
${\tau }^2$-Bench [75]	—	279 tasks	4	Shared environment under dual control	-	4	Verifiable task outcomes	GPT-4.1; default telecom setting	${\mathrm{pass}}^1$ 34%
MobileWorld [76]		201 tasks	12	Mobile emulator, MCP servers, simulated user	50 steps	1	Functional state checks with interaction rating	GPT-5 + UIIns-7B	Success 51.7%
E-Bench [65]	✓	323 tasks	11	Product database APIs	-	3	Database changes against the ground-truth diff	Kimi-K3; base tool setting	Avg@3 73.8%
MobilityBench [77]	—	100k episodes	11	Deterministic API replay sandbox	10 steps	-	Outcome checks with intermediate capability assessment	Gemini 3 Pro Preview / ReAct; 7,098 episodes	FPR decision making 69.09%

$\mathcal{H}$ cmp.: ✓ harness or component varied; complete-system comparison only; ✗ harness fixed in a model comparison; — none reported. Marks describe reported comparisons rather than causal isolation. $N_M$: evaluated models, excluding auxiliary simulators and judges unless marked. Trials: ollouts per task; NR indicates not reported, and N/A indicates no interactive rollout. A slash follows the order named in the configuration; an arrow denotes a within-paper comparison. ^a95% confidence interval, implying repeated trials. ^cOptimizer and target harnesses are distinct.

Table 6. Benchmark reproduction reference: task execution, safety, judges, and model capability. Results apply to the named configurations; sizes retain their original units. Framework mapping appears in Table 4.

Benchmark	$\mathcal{H}$ cmp.	Size	${\mathit{N}}_{\mathit{M}}$	Provision	Budget	Trials	Evaluator	Reference configuration	Headline result
Task execution
OSWorld [78]	✓	369 tasks	12	Desktop virtual machine images	15 steps / 30 min	1	Task-specific execution-based state scripts	GPT-4; accessibility tree	Success 12.24%
WebArena [45]		812 tasks	3	Self-hosted functional websites	30 steps	1	Functional correctness of states and answers	GPT-4; CoT; no unachievable-task hint	Success 14.41%
Terminal-Bench 2.0 [66]	✓	89 tasks	16	Per-task container environments	Task-specific time limit	5a	Task-specific executable verification tests	GPT-5.2 / Codex CLI	Resolved $62.9\pm 3.0$%
Mind2Web [79]	✓d	2,350 tasks	6	Cached website snapshots	-	1	Predicted elements and operations against references	MindAct / Flan-T5-XL; cross-domain (912)	Step SR 39.6%
PARTNR [80]	✓	100k tasks	2	3D multi-agent simulated households with embodied skills	50 planner calls / 20k sim steps	1	Evaluation functions over propositions and constraints	Llama3.1-70B; decentralized, learned skills	Success 30.0%
Gaia2 [46]	✓	1,120 scenarios	14 cfg.	Asynchronous application environment with events	200 steps / timeout / 16k tokens per turn	3	Scenario-specific verification of state-changing actions	GPT-5 (high) / ReAct; split macro-average	Pass@1 42.1%
Safety
Claw-Eval [81]	✗	300 tasks	14	Sandboxed services with audit logging	-	3	Hybrid grading over traces, logs, and snapshots	GPT-5.4; multimodal subset (101)	${\mathrm{Pass}}^3$ 25.7%
SandboxEscapeBench [67]	✗	18 tasks	13	Nested sandboxes with known vulnerabilities	2.5M tokens	5a	Exact match of a flag obtained outside the container	GPT-5; original trial budget	Escape $50[40,60]$%
OS-Harm [59]	✗	150 tasks	5	Desktop virtual machines with injected content	15 steps	1	Judged safety and completion, human-validated	GPT-4.1; average across risk categories	Unsafe 21% ↓
ATBench [86]	✗	1,000 trajectories	17 J	Human-audited trajectory corpus	-	-	Classification and diagnosis against annotated labels	GPT-5.4; safe/unsafe classification	F1 76.7%
Judges
AgentRewardBench [10]	—	1,302 trajectories	4 A	Expert-annotated trajectory corpus	-	-	Automatic judgments against expert labels	GPT-4o agent / WebArena; rules / human	Success $25.6/42.3$%
Mind2Web 2 [82]		130 tasks	10 sys.	Live web access	-	3	Rubric-tree judge agents with human validation	OpenAI Deep Research; full-task success	Success 28%
Model capability
EDIT-Bench [83]	✓d	540 problems	40	File context without interactive rollout	One pass	1	Programmer-authored and reviewed unit tests	Claude Sonnet 4; highlighted edit region	Pass@1 66.67%
BIG-bench [84]	—	204 tasks	31	None; text-based tasks	Few-shot	1	Task-specific scoring with human-rater baselines	13 dense BIG-G, 10 sparse, and 8 GPT models	-

$\mathcal{H}$ cmp.: ✓ harness or component varied; complete-system comparison only; ✗ harness fixed in a model comparison; — none reported. Marks describe reported comparisons rather than causal isolation. $N_M$: evaluated models, excluding auxiliary simulators and judges unless marked (A: acting models; J: judges; cfg.: configurations; sys.: agent products). Trials: rollouts per task, where - indicates none performed. A slash follows the order named in the configuration. ^a95% confidence interval, implying repeated trials. ^dOffline prediction setting rather than an interactive rollout.

Table 6. Benchmark reproduction reference: task execution, safety, judges, and model capability. Results apply to the named configurations; sizes retain their original units. Framework mapping appears in Table 4.

Benchmark	$\mathcal{H}$ cmp.	Size	${\mathit{N}}_{\mathit{M}}$	Provision	Budget	Trials	Evaluator	Reference configuration	Headline result
Task execution
OSWorld [78]	✓	369 tasks	12	Desktop virtual machine images	15 steps / 30 min	1	Task-specific execution-based state scripts	GPT-4; accessibility tree	Success 12.24%
WebArena [45]		812 tasks	3	Self-hosted functional websites	30 steps	1	Functional correctness of states and answers	GPT-4; CoT; no unachievable-task hint	Success 14.41%
Terminal-Bench 2.0 [66]	✓	89 tasks	16	Per-task container environments	Task-specific time limit	5a	Task-specific executable verification tests	GPT-5.2 / Codex CLI	Resolved $62.9\pm 3.0$%
Mind2Web [79]	✓d	2,350 tasks	6	Cached website snapshots	-	1	Predicted elements and operations against references	MindAct / Flan-T5-XL; cross-domain (912)	Step SR 39.6%
PARTNR [80]	✓	100k tasks	2	3D multi-agent simulated households with embodied skills	50 planner calls / 20k sim steps	1	Evaluation functions over propositions and constraints	Llama3.1-70B; decentralized, learned skills	Success 30.0%
Gaia2 [46]	✓	1,120 scenarios	14 cfg.	Asynchronous application environment with events	200 steps / timeout / 16k tokens per turn	3	Scenario-specific verification of state-changing actions	GPT-5 (high) / ReAct; split macro-average	Pass@1 42.1%
Safety
Claw-Eval [81]	✗	300 tasks	14	Sandboxed services with audit logging	-	3	Hybrid grading over traces, logs, and snapshots	GPT-5.4; multimodal subset (101)	${\mathrm{Pass}}^3$ 25.7%
SandboxEscapeBench [67]	✗	18 tasks	13	Nested sandboxes with known vulnerabilities	2.5M tokens	5a	Exact match of a flag obtained outside the container	GPT-5; original trial budget	Escape $50[40,60]$%
OS-Harm [59]	✗	150 tasks	5	Desktop virtual machines with injected content	15 steps	1	Judged safety and completion, human-validated	GPT-4.1; average across risk categories	Unsafe 21% ↓
ATBench [86]	✗	1,000 trajectories	17 J	Human-audited trajectory corpus	-	-	Classification and diagnosis against annotated labels	GPT-5.4; safe/unsafe classification	F1 76.7%
Judges
AgentRewardBench [10]	—	1,302 trajectories	4 A	Expert-annotated trajectory corpus	-	-	Automatic judgments against expert labels	GPT-4o agent / WebArena; rules / human	Success $25.6/42.3$%
Mind2Web 2 [82]		130 tasks	10 sys.	Live web access	-	3	Rubric-tree judge agents with human validation	OpenAI Deep Research; full-task success	Success 28%
Model capability
EDIT-Bench [83]	✓d	540 problems	40	File context without interactive rollout	One pass	1	Programmer-authored and reviewed unit tests	Claude Sonnet 4; highlighted edit region	Pass@1 66.67%
BIG-bench [84]	—	204 tasks	31	None; text-based tasks	Few-shot	1	Task-specific scoring with human-rater baselines	13 dense BIG-G, 10 sparse, and 8 GPT models	-

$\mathcal{H}$ cmp.: ✓ harness or component varied; complete-system comparison only; ✗ harness fixed in a model comparison; — none reported. Marks describe reported comparisons rather than causal isolation. $N_M$: evaluated models, excluding auxiliary simulators and judges unless marked (A: acting models; J: judges; cfg.: configurations; sys.: agent products). Trials: rollouts per task, where - indicates none performed. A slash follows the order named in the configuration. ^a95% confidence interval, implying repeated trials. ^dOffline prediction setting rather than an interactive rollout.

5. Open Challenges and Research Directions

5.1. Impact of This Survey

The practical impact of harness-aware evaluation is to shift the unit of interpretation from the model alone to the complete evaluation configuration. By making the contributions of the model, harness, environment, evaluator, and protocol explicit, the proposed framework in this work can help researchers avoid over-attributing benchmark gains to a single component, design more controlled comparisons, and report agent evaluations in a form that is easier to reproduce and audit.

For benchmark designers, the framework highlights the need to document execution conditions, evaluator assumptions, and resource budgets alongside task scores. For developers and practitioners, it provides a structured way to diagnose whether changes in performance arise from model capability, execution infrastructure, environmental conditions, or scoring procedures. More broadly, this perspective supports more interpretable and comparable claims about agent performance, robustness, and safety.

5.2. Research Directions

Adaptive and Self-Modifying Harnesses

Most current evaluations assume that the harness remains fixed during evaluation. However, emerging agents can modify prompts, tools, memory policies, workflows, or even their own execution logic in response to feedback. This complicates the notion of a fixed evaluation configuration $C=(M,H,E,V)$, because H may evolve over the course of a run or across tasks. Future evaluation methods therefore need to distinguish between a static harness configuration and the adaptation policy that governs how the harness changes. Reproducibility will require reporting not only the initial harness, but also the allowed modification space, optimization feedback, update frequency, and final resulting configuration.

Fair Comparison under Unequal Resource Budgets

Harnesses can improve task success by increasing the number of model calls, retries, verifier passes, context length, or parallel agents. Consequently, higher benchmark scores may reflect greater inference expenditure rather than a more effective execution strategy. Existing work already shows that additional model calls can confound architectural comparisons. A central challenge is therefore defining fair resource constraints across heterogeneous agent systems. Future evaluations should compare configurations under matched or explicitly normalized budgets, considering tokens, model calls, wall-clock time, tool calls, compute cost, and human intervention rather than relying on a single resource measure.

Generalization of Harness Effects

A harness modification that benefits one model, task domain, or environment may not transfer to another. Tool interfaces, memory policies, planning strategies, and verification mechanisms can interact with model-specific capabilities and task requirements. As a result, improvements observed for a particular $(M,H,E,V)$ configuration should not automatically be interpreted as general properties of the harness. Future work should evaluate harness interventions across multiple models, environments, and task families while preserving controlled comparisons. Such evaluations could help distinguish configuration-specific gains from harness mechanisms that generalize across settings.

Interactions Between Evaluation Factors

The factors in our taxonomy are analytically separable, but they are not necessarily independent. For example, a tool interface may be effective for one model but difficult for another; an evaluator may favor particular execution strategies; and environmental noise may interact with memory or recovery mechanisms. Evaluating one factor at a time therefore identifies local effects but may miss important interactions between M, H, E, and V. A major research direction is to develop experimental designs that measure these interactions without requiring exhaustive evaluation of the full configuration space.

Incomplete and Inconsistent Configuration Reporting

Many agent evaluations report the backbone model and benchmark while providing limited detail about the harness, evaluator, execution budget, or environment configuration. This makes reproduction difficult and can lead to performance differences being attributed to the model when other elements of the evaluation configuration have changed. Harness-aware evaluation therefore requires more systematic reporting of context management, tool interfaces, control flow, verification and recovery mechanisms, permissions, evaluator criteria, environment versions, resource budgets, and repeated-run procedures. Developing standardized configuration descriptions and machine-readable reporting formats would improve reproducibility and make benchmark results easier to compare across studies.

5.3. Limitations

This survey has several limitations. First, the agent and harness ecosystem is evolving rapidly, and new models, execution frameworks, and benchmarks continue to appear. The taxonomy therefore reflects the literature available at the time of review rather than a fixed or exhaustive landscape. Second, although the proposed framework separates model, harness, environment, and evaluator factors for interpretability [87], these factors often interact in practice. A harness component may benefit one model but not another, and evaluator or environmental changes may alter the apparent effect of the harness. The framework therefore supports attribution analysis but does not imply that these factors are statistically independent. Finally, incomplete reporting in the underlying literature limits the precision with which some configurations can be reconstructed. Details such as prompts, tool schemas, context policies, retry limits, permissions, evaluator settings, or resource budgets are not always available. Consequently, some classifications rely on the evidence reported by the original studies rather than complete reproduction of their evaluation configurations.

6. Conclusion

This survey traced the development of LLM-based agents and the growing importance of harness design in their evaluation. We introduced a four-factor framework connecting the model, harness, environment, and evaluator, together with the evaluation protocol that governs how results are measured and compared. Through a review of harness systems, benchmarks, and evaluation methods, we examined how execution mechanisms differ across coding agents and broader assistant systems, and what existing comparisons establish about their contributions. The reviewed evidence shows that reported gains depend on the surrounding configuration, resource budgets, and assessment criteria, limiting conclusions drawn from benchmark scores alone. Our framework provides reporting guidance to make these conditions explicit and support more interpretable comparisons of performance, reliability, and safety. While model capabilities remain central, harness design offers a complementary means of improving agent execution. Establishing which improvements generalize will require controlled comparisons across models and environments, repeated evaluation, and explicit assessment of safety and human oversight.