Stop Comparing LLM Agents Without Disclosing the Harness

1. Introduction

The standard practice in LLM agent evaluation reports a single number per $\{\mathtt{model},\mathtt{benchmark}\}$ pair and attributes it to the model. Leaderboards such as SWE-bench [1], Terminal-Bench [2], AgentBench [3], and GAIA [4] treat agent performance as if it were a property of the model alone, and this convention directs research toward model scaling, fine-tuning for tool use, and prompt-level techniques. The implicit assumption is that agent capability is primarily driven by model capability, and that a sufficiently capable model produces reliable behavior on long-horizon tasks.

This assumption omits the execution harness: the software layer between the model and the task that constructs the context the model sees, mediates its tool calls, validates its outputs, and decides when to retry, escalate, or stop. Every benchmark score is jointly produced by a model and a harness, but the harness is rarely disclosed and almost never held constant across comparisons. The phenomenon is not anecdotal: holding the model fixed while changing only the harness raises Terminal-Bench 2 pass@1 from 69.7% to 77.0% [5], and independent third-party benchmark monitoring reports up to 15 percentage points of scaffold-only variation on SWE-bench Verified [6]. On such long-horizon tasks, the same model under a different harness can therefore rank above or below a competitor.

We argue that benchmark scores for LLM agents on long-horizon tasks are not valid for cross-model comparison unless the execution harness is disclosed. Valid comparisons require either a locked-harness protocol, where a single specified harness is applied to all models, or a factorial protocol, where harness choice is varied as a controlled factor and variance components are reported. This disclosure extends to the infrastructure layer responsible for context construction, tool-call mediation, and output verification. Without such disclosure, leaderboards measure a mixture of model and harness, and the conclusions drawn from them are systematically misattributed to the model.

The practical consequences are concrete. Cross-paper model comparisons are unreliable, since two reported scores typically reflect different harnesses. Reproductions of a published score are difficult without a full harness specification, because the harness can be as important to the outcome as the model itself. Research incentives are also distorted: investments target model-side improvements while harness changes, the larger source of performance variance on long-horizon tasks, receive proportionally less attention. Better models still matter, and harness engineering is part of the deployed agent. The position calls for an attribution framework that decides whether an observed benchmark gain should be credited to the model, the harness, or their interaction. Beyond academic accuracy, these misattributions shape decisions the field is now making at scale, including which models to fund, which research directions to scale, and which agents to deploy. An evaluation regime that cannot separate model from harness routes those decisions to the wrong lever.

2. Why Current Evaluation Conflates Model and Harness

2.1. How the Harness Enters Every Benchmark Score

Standard agent evaluation queries the model with a prompt assembled from the task description, system prompt, tool catalogue, and prior trajectory state [7,8,9], parses the token-level output as a final answer, tool invocation, or malformed string, then routes tool calls, handles errors, compresses context as the trajectory grows, and stops after some number of steps. Every step in this pipeline (context construction, tool invocation, parsing, retry, summarization, and stopping) is a harness decision [5,10,11,12]. A single benchmark score is therefore the joint outcome of these choices and the model that generates outputs against them, but the published number records the model and elides the rest.

The conflation is not hypothetical. Under the standardized SEAL scaffold on SWE-bench Pro, Claude Opus 4.5 reaches 45.9%, while under Claude Code the same model reaches 55.4%. Adding a single search subagent (WarpGrep) to otherwise identical infrastructure flips the SWE-bench Pro ordering between MiniMax 2.5 and Claude Opus 4.6, despite Claude Opus ranking higher on most other benchmarks [13]. The Holistic Agent Leaderboard (HAL) reports double-digit gaps for the same model under different scaffolds on SWE-bench Verified Mini, with reported single-model swings of up to nearly 48 percentage points across leading frontier models [14]. Independent third-party benchmark monitoring reports up to 11 percentage points of scaffold-only variation for GPT-5 and 15 points for Kimi K2 Thinking on SWE-bench Verified [6]. None of these effects are model upgrades. They are harness substitutions, and they routinely dwarf the 2 to 4 percentage point shifts that papers report as meaningful model advances. The resulting attribution gap matters because the same composite number is being read in three incompatible ways: as a model property in cross-paper model comparisons, as a benchmark property in tracking field progress, and as an agent property in deployment decisions. However, only the third reading is well-defined. The first two require disentanglement that current reporting does not provide.

2.2. The Structural Reason: Agents Are Closed-Loop Systems

The conflation reflects a structural fact about how agents work. Agent execution is a discrete-time closed-loop dynamical process. Let $s_t$ denote the agent’s state at step t, comprising the full context window and any persistent memory. The model acts as a stochastic policy $a_t\sim {\pi }_{\theta }(·\mid c\left(s_t\right))$, where c is the context construction function implemented by the harness. The harness implements a controller $\mathcal{C}$ that updates state from $(s_t,a_t,o_t)$ to $s_{t+1}$, where $o_t$ is the environment observation, filtered and validated before being integrated into state. Different harness configurations H induce different controllers ${\mathcal{C}}_H$ over the same model ${\pi }_{\theta }$. In this model, the LLM is open-loop. It has no direct access to $s_t$, only to $c\left(s_t\right)$, the projection of state into context that the harness chooses to expose. It retains no memory across steps beyond what ${\mathcal{C}}_H$ injects into $c\left(s_{t+1}\right)$. It has no mechanism for self-correction beyond the feedback path that ${\mathcal{C}}_H$ constructs from $o_t$. Adaptation, error recovery, and long-horizon coherence are properties of ${\mathcal{C}}_H$, not ${\pi }_{\theta }$. This view continues a line of agent research that treats reasoning, action, and interfaces as coupled [7,15,16,17,18].

This asymmetry is the structural reason for the conflation. In closed-loop systems, the controller, not the open-loop policy, governs three quantities that determine long-horizon reliability. Stability is whether expected progress toward a goal is non-decreasing under the controller’s update rule. Context drift is how fast task-relevant information leaves the context window. Control lag is the number of steps between an anomaly being detected and a corrective signal reaching the policy. None of the three is a model property. A more capable model can lower the base rate at which anomalies arise, but the response to anomalies once they occur is a property of ${\mathcal{C}}_H$ that does not improve when the model is upgraded [19].

The lesson is broader than agents. Open-loop systems do not become reliable through bigger actuators. They become reliable through feedback control. The same structural fact applies to LLM agents on long-horizon tasks. Figure 1 illustrates the contrast: under the inference framing the model carries the reliability burden alone, while under the closed-loop framing the harness controller surrounds the policy and supplies the feedback that long-horizon execution requires.

2.3. Why Standard Responses Fall Short

If the conflation is structural, the question is what to do about it. The field has converged on four classes of response. Each addresses part of the problem, but none resolves the attribution question.

Standardization on a single harness. Locking the harness across submissions is the most direct response, instantiated by HAL across nine benchmarks [14], by the SWE-bench Pro unified scaffold [13,20], and by Epoch AI’s mini-SWE-agent recommendation [6]. We endorse locked-harness protocols as one of two valid evaluation regimes (Sec. 3 names the other), but any standardized harness embeds design choices that interact with model properties and forecloses the harness-side gains current evidence shows are large. The position calls for disclosure of the harness in use, not uniformity across all evaluation.

Scaling the model. A second response holds that more capable models will manage their own controllers. The empirical trend goes the other way: Claude Code redesigned rather than removed harness components as capability grew [21], the managed-agents architecture explicitly decouples model and harness because harness assumptions go stale faster than models improve [22], and AHE produces double-digit pass-rate gains on top of frontier base models [5]. No model upgrade compensates for the absence of feedback structure in the controller (Sec. 2.2).

Adding more benchmarks. A third response is that with enough independent benchmarks, harness-induced noise will average out. Recent additions span freelance software work, ML engineering, contamination-resistant coding, code synthesis, visual front-end issues, and professional services [23,24,25,26,27,28]. The bet fails: each benchmark uses its own undisclosed harness, and harness contributions do not have zero mean, so adding benchmarks adds independent $\{model,harness\}$ draws without resolving attribution within any of them. APEX-Agents finds the opposite of what the bet requires: agent failures are predominantly execution and orchestration problems rather than knowledge failures [28].

Optimizing the harness. A fourth response is that the field already addresses harness quality directly. ADAS searches over agentic systems [29], Meta-Harness and AutoHarness optimize harness code via outer-loop search [10,11], and AHE evolves coding-agent harnesses with cross-family transfer [5]. Agentic Context Engineering optimizes context construction as a separate surface [12], and practitioner deployment reports reach the same conclusion [30,31]. These contributions document the phenomenon this paper analyzes but answer a different question: how to make a harness better, not how to attribute observed gains. Some report results under their own harness, compounding rather than resolving attribution. Harness optimization is part of the evidence base for the position, not a substitute for it.

The structural problem requires a structural solution: disclosure. The harness used to produce a benchmark score must be part of the experimental condition, and cross-model comparisons must hold the harness fixed (locked-harness protocol) or vary it as a controlled factor (factorial protocol).

3. The Binding Constraint Thesis

The Binding Constraint Thesis

For LLM agents operating on long-horizon tasks with comparable frontier models, let $B(M,H)$ denote the benchmark score of model $M\in \mathcal{M}$ under harness $H\in \mathcal{H}$. Define: $$\begin{array}{c}\hfill \mathrm{HV}\left(M\right)={\mathrm{Var}}_{H\sim P\left(\mathcal{H}\right)}\left[B(M,H)\right]\mathrm{MV}\left(H\right)={\mathrm{Var}}_{M\sim P\left(\mathcal{M}\right)}\left[B(M,H)\right]\end{array}$$ The Binding Constraint Thesis asserts that, in this regime, HV is often comparable to or larger than MV, and may dominate it in many current long-horizon agent evaluations. Current benchmark protocols report $B(M,H^{*})$ for a single undisclosed $H^{*}$, rendering HV unmeasurable and model comparisons incomplete and potentially misleading.

3.1. Variance Decomposition

Total performance variance over a population of $\{M,H\}$ draws decomposes exactly as

$$\mathrm{Var}\left(B(M,H)\right)=\underset{\mathrm{MV}}{\underbrace{{\mathrm{Var}}_M\left[{\mathbb{E}}_H\left[B(M,H)\right]\right]}}+\underset{\mathrm{HV}}{\underbrace{{\mathrm{Var}}_H\left[{\mathbb{E}}_M\left[B(M,H)\right]\right]}}+\underset{\mathrm{interaction}}{\underbrace{\mathrm{Var}\left(\mathrm{model}\times \mathrm{harness}\right)}}.$$

(1)

The thesis asserts that the second and third terms together dominate the first in the regime considered. The interaction term is non-negligible: under the closed-loop account, a harness emphasizing self-verification helps a model with high false-confidence rates more than a model with conservative output distributions, so the same harness shift produces different gains across models. Treating model rankings as model properties requires the interaction term to be small relative to MV, a condition the thesis denies. The decomposition is the methodological core of the position. A locked-harness protocol fixes $H=H^{*}$ and recovers a clean ranking of $\left\{B(M_i,H^{*})\right\}$ across models, but only under that specific $H^{*}$. A factorial protocol designs a grid varying both M and H, reports $\overline{\mathrm{HV}}={\mathbb{E}}_M\left[\mathrm{HV}\left(M\right)\right]$ and $\overline{\mathrm{MV}}={\mathbb{E}}_H\left[\mathrm{MV}\left(H\right)\right]$ averaged across the opposite axis, the interaction term, and the count of model-pair ranking reversals across harnesses, and supports interaction analysis directly.

3.2. Reliability Quantities of the Harness Controller

Stability, context drift, and control lag are the three structural channels through which harness configuration produces variance. Each is a property of ${\mathcal{C}}_H$, not of ${\pi }_{\theta }$, and each maps to an empirical failure mode that current evaluation captures only as a single composite score. Together they identify what HV is variance of: a harness shift that changes any of the three shifts $B(M,H)$ for fixed M.

Definition 1 

(Stability). An agent run is stable if the Lyapunov-like measure $V\left(s_t\right)=d(s_t,{\Omega }^{*})$ is nonincreasing in expectation under $\mathcal{C}$, where ${\Omega }^{*}$ is the set of goal-consistent states and d is an appropriate task-specific distance measure. Formally: $\mathbb{E}[V\left(s_{t+1}\right)\mid s_t]\le V\left(s_t\right)$ for all t. Instability manifests empirically as hallucination spirals, context overflow, and execution runaway, all of which are controller failures rather than policy failures.

Definition 2 

(Context Drift). Let $p_t$ denote the distribution of context-window embeddings at step t, obtained by encoding the full context window using a fixed embedding model. Context drift is ${\delta }_t=D_{\mathrm{KL}}(p_t\parallel p_0)$, the divergence of the current context distribution from the initial task-relevant state $p_0$. A high drift rate $\dot{\delta }={\delta }_t-{\delta }_{t-1}$ indicates that the controller is failing to maintain task-relevant information in the context window and empirically correlates with task abandonment and semantic inconsistency over long horizons. Drift is a property of the harness configuration H through its context construction policy c. No property of the model can prevent context drift when the controller fails to manage it.

Definition 3 

(Control Lag). We propose control lag τ as a harness-specific reliability measure: the number of steps between harness anomaly detection (a tool failure, malformed output, or empty result) and the arrival of corrective action at the policy through the feedback path. Formally, if an anomaly is detected at step $t_d$ and the corrective observation $o_{t_c}$ reaches the policy at step $t_c$, then $\tau =t_c-t_d$. High τ enables failure cascades: the policy continues to act on a corrupted state for τ steps while the harness processes the error, compounding the damage before correction arrives.

These three quantities sit on the structural side of the variance decomposition. Two configurations $H_1,H_2$ that differ in self-verification (affecting stability), context compression (affecting drift), or anomaly recovery (affecting control lag) each shift $B(M,H)$ for fixed M, contributing independently to HV. A more capable model can lower the base rate at which anomalies arise, which reduces demand on ${\mathcal{C}}_H$, but the response to anomalies once they occur is a property of ${\mathcal{C}}_H$ that does not improve when the model is upgraded. This is the structural reason the thesis is plausible. We use these quantities as conceptual tools for reasoning about ${\mathcal{C}}_H$, and develop their operational counterparts as trajectory-level metrics below.

3.3. Scope, Falsifiability, and Implications

Scope. The thesis is restricted to long-horizon tasks and comparable frontier models. Long-horizon means tasks where multiple steps of tool use, error recovery, and context management are required to reach a final outcome. The closed-loop reliability quantities only have an opportunity to act over such horizons. Comparable frontier models means a regime in which model capability gaps are not so large that MV mechanically dominates regardless of harness configuration. What can be said precisely is what falls outside the regime: short-horizon tasks and configurations where one model is dramatically more capable than the others on the relevant skills. The regime that remains is the regime where benchmark leaderboards drive research and product decisions.

Falsifiability. The thesis is empirical. It can be falsified by a factorial experiment that varies $\mathcal{H}$ while holding $\mathcal{M}$ fixed, separately varies $\mathcal{M}$ while holding $\mathcal{H}$ fixed, and finds $\overline{\mathrm{MV}}>\overline{\mathrm{HV}}$ on a long-horizon task distribution at comparable model capability. The thesis is not vacuous: such an outcome is conceivable in principle, and our controlled grid makes the variance ordering an empirical question. Concurrent psychometric work that decomposes agent ability into independent LLM and scaffold components offers an external probe in the same direction [32]. The thesis would be in tension with that line of work if scaffold-ability variance were systematically smaller than LLM-ability variance across benchmarks, and consistent with it otherwise.

4. Evidence

We support the thesis with two kinds of evidence: observational data from public leaderboards and a controlled factorial experiment under matched conditions.

4.1. Evidence from Public Leaderboards

SWE-bench Pro under standardized vs varied scaffolds. On the Morph leaderboard [13], the six leading frontier models span only 4.9 percentage points under the standardized SEAL scaffold (41.0% to 45.9%). Holding Claude Opus 4.5 fixed and varying only the harness widens this to 9.5 points (SEAL vs Claude Code). The within-model harness range thus exceeds the within-harness model range by roughly twofold, a lower bound on the HV / MV ratio for this slice. Adding the WarpGrep search subagent on top of identical infrastructure adds 2.1 to 2.2 points across models, comparable to a routine model upgrade and sufficient to flip the MiniMax 2.5 vs Claude Opus 4.6 ordering: a single tool addition makes the interaction term visible.

Cross-scaffold gaps on SWE-bench Verified Mini. HAL reports double-digit cross-scaffold gaps under SWE-Agent versus HAL Generalist, of 34 points for Claude Sonnet 4.5 (68% to 34%), 34 points for GPT-5 Medium (46% to 12%), and nearly 48 points for o4-mini [14], much larger than the cross-model spread under either scaffold individually. The same psychometric work [32], in its leaderboard analysis, reaches the same qualitative conclusion under an item-response-theory model.

Third-party benchmark monitoring. Independent monitoring reports up to 11 to 15 percentage points of scaffold-only variation on SWE-bench Verified, with the explicit observation that scaffold choice has the single biggest impact on overall performance [6]. The recommendation to use mini-SWE-agent for cross-model comparison is one realization of the locked-harness protocol. The thesis adds that the measurement gap between locked and unlocked scaffolds is what the variance decomposition makes visible.

The pattern is consistent and large. But public evidence is observational. Public harnesses differ across many dimensions and were built by different teams with different engineering budgets, so the public data do not isolate which harness components drive the variance.

Table 1. Harness-induced performance changes on agent benchmarks (model held fixed, only infrastructure changes).

Benchmark	Model (Fixed)	Harness Change	Layer	$\mathbf{\Delta }$
SWE-bench Pro	Claude Opus 4.5	SEAL → Claude Code	E, T, C	+9.5pp
		(45.9% → 55.4%)
SWE-bench Verified	Grok 4	SWE-agent → xAI scaffold	T, C, O	+14–16pp
		(58.6% → 72–75%)
TerminalBench 2.0	Fixed model	Prompt + middleware + verif.	C, S, V	+13.7pp
		(52.8% → 66.5%)
TerminalBench-2	Fixed model	Automated harness opt.	All	76.4%
TerminalBench 2	GPT-5.4 (high)	AHE harness evolution	C, T, S, O	+7.3pp
		(69.7% → 77.0%)
Coding benchmarks	Fixed model	Context format + tool defs	T, C	∼10×
Agent tasks (Vercel)	Fixed model	15 tools → 2 tools	T	80%→100%
SWE-bench Pro	Various	+WarpGrep search subagent	T	+2.1–2.2pp

Layers: E = Execution, T = Tool, C = Context, S = Scheduling, O = Observability, V = Verification. Sources: SWE-bench Pro data from Morph [13]. SWE-bench Verified scaffold reports from Vals AI and Anthropic [33,34]. Terminal-Bench harness results from Trivedy, Meta-Harness, and AHE [5,10,35]. Tool-reduction case from Vercel [36]. Edit-tool harness result from Bölük [37].

Table 1. Harness-induced performance changes on agent benchmarks (model held fixed, only infrastructure changes).

Benchmark	Model (Fixed)	Harness Change	Layer	$\mathbf{\Delta }$
SWE-bench Pro	Claude Opus 4.5	SEAL → Claude Code	E, T, C	+9.5pp
		(45.9% → 55.4%)
SWE-bench Verified	Grok 4	SWE-agent → xAI scaffold	T, C, O	+14–16pp
		(58.6% → 72–75%)
TerminalBench 2.0	Fixed model	Prompt + middleware + verif.	C, S, V	+13.7pp
		(52.8% → 66.5%)
TerminalBench-2	Fixed model	Automated harness opt.	All	76.4%
TerminalBench 2	GPT-5.4 (high)	AHE harness evolution	C, T, S, O	+7.3pp
		(69.7% → 77.0%)
Coding benchmarks	Fixed model	Context format + tool defs	T, C	∼10×
Agent tasks (Vercel)	Fixed model	15 tools → 2 tools	T	80%→100%
SWE-bench Pro	Various	+WarpGrep search subagent	T	+2.1–2.2pp

Layers: E = Execution, T = Tool, C = Context, S = Scheduling, O = Observability, V = Verification. Sources: SWE-bench Pro data from Morph [13]. SWE-bench Verified scaffold reports from Vals AI and Anthropic [33,34]. Terminal-Bench harness results from Trivedy, Meta-Harness, and AHE [5,10,35]. Tool-reduction case from Vercel [36]. Edit-tool harness result from Bölük [37].

4.2. Controlled Factorial

To isolate harness-controlled variance, we vary configurations along the three reliability quantities while holding all other factors fixed, and report the resulting decomposition on a representative long-horizon coding task distribution.

Setup. We evaluate three frontier models, GPT-5.4 [38], Kimi K2.6 [39], and GLM-5.1 [40], selected because they were tightly clustered on the LLM Stats coding leaderboard,1 across three harness configurations $\{H_1,H_2,H_3\}$ on a difficulty-stratified 100-task subset of SWE-bench Verified [1]. Each $(M_i,H_j)$ cell uses two independent runs with shared task order, Docker execution environment, SWE-bench evaluation pipeline, 50-step budget, and 120-second per-step timeout. Subset construction, model-selection rationale, and full harness specifications are in Appx. B.

The three harness configurations vary deliberately along the three reliability quantities:

• $H_1$(Minimal): no context compression, verbose tool schema, no retry logic, no verification hooks, and no anomaly recovery. High drift rate, high control lag, and no stability guarantees. This is the baseline open-loop harness.
• $H_2$(Improved): compressed context with task-relevance retrieval, minimal tool schema, and structured retry on tool failure with exponential backoff. This configuration reduces drift and provides moderate control lag with basic closed-loop feedback.
• $H_3$(Full): $H_2$ plus per-step self-checking, KL-style drift checks every five steps, anomaly-detection middleware for repeated action loops and context contradictions, full output validation, and checkpoint rollback retaining the last 10 states. This configuration has low drift, low control lag, and explicit stability enforcement.

Metrics. We report pass@1 $B(M_i,H_j)$, model-induced variance per harness $\mathrm{MV}\left(H_j\right)={\mathrm{Var}}_M\left[B(M,H_j)\right]$, harness-induced variance per model $\mathrm{HV}\left(M_i\right)={\mathrm{Var}}_H\left[B(M_i,H)\right]$, the aggregate ratio $\overline{\mathrm{HV}}/\overline{\mathrm{MV}}$ averaged across the opposite axis, and the count of model-pair ranking reversals across harnesses.

Table 2. Benchmark scores $B(M,H)$ on SWE-bench Verified subset100. Cells report mean pass@1 percentages over two runs, with percent signs omitted. Run-level results are reported in Appx. B, Table A5. The HV column and MV row report variances in pp².

	$H_1$ (Minimal)	$H_2$ (Improved)	$H_3$ (Full)	HV$\left(M\right)$
GLM-5.1	52.5	56.5	65.5	29.56
GPT-5.4	55.0	58.5	63.5	12.17
Kimi K2.6	52.0	59.0	60.5	13.72
MV$\left(H\right)$	1.72	1.17	4.22

Aggregate $\overline{\mathrm{HV}}/\overline{\mathrm{MV}}$ ratio: 7.80×. Ranking reversal pairs: 6 out of 9 model-pair/harness-pair comparisons.

Table 2. Benchmark scores $B(M,H)$ on SWE-bench Verified subset100. Cells report mean pass@1 percentages over two runs, with percent signs omitted. Run-level results are reported in Appx. B, Table A5. The HV column and MV row report variances in pp².

	$H_1$ (Minimal)	$H_2$ (Improved)	$H_3$ (Full)	HV$\left(M\right)$
GLM-5.1	52.5	56.5	65.5	29.56
GPT-5.4	55.0	58.5	63.5	12.17
Kimi K2.6	52.0	59.0	60.5	13.72
MV$\left(H\right)$	1.72	1.17	4.22

Aggregate $\overline{\mathrm{HV}}/\overline{\mathrm{MV}}$ ratio: 7.80×. Ranking reversal pairs: 6 out of 9 model-pair/harness-pair comparisons.

Findings. The grid is consistent with the Binding Constraint Thesis on this task distribution. Average HV is $18.48$ pp² versus average MV of $2.37$ pp², a ratio of $7.80\times$. Changing the harness moves GLM-5.1 by 13.0 percentage points and GPT-5.4 and Kimi K2.6 by 8.5 points each. Changing the model within a fixed harness moves scores by only 3.0, 2.5, and 5.0 points for $H_1$, $H_2$, $H_3$. The interaction term is visible as six ranking reversals across the nine possible model-pair / harness-pair comparisons. We do not claim that the $7.80\times$ ratio is universal. The grid demonstrates that harness variance can dominate model variance under controlled conditions and that the decomposition protocol produces interpretable estimates on a realistic task distribution.

Trajectory-log analysis confirms that the $H_1\to H_2$ shift reduces control noise rather than increasing model knowledge, while the $H_2\to H_3$ gain comes primarily from closing the verification and recovery loop. The per-paragraph mechanism analysis and the cross-model variance pattern across $H_1$, $H_2$, $H_3$ are in Appx. E.

5. A Harness-Aware Evaluation Framework

We argue that long-horizon agent evaluation should adopt a harness-aware framework with three components: a structured disclosure card, a variance decomposition protocol, and a short set of trajectory-level metrics that together make harness variance reportable, interpretable, and attributable.

The harness card. Every agent benchmark submission should include a Harness Card: a structured disclosure of the harness configuration organized by the seven-layer ETCSOVG taxonomy, intended as an attention checklist rather than a design constraint. The seven layers cover Execution (runtime substrate, sandboxing, step and task budgets), Tool (tool list, schemas, error contract), Context (window cap, compression and retrieval policy, persistent memory), Scheduling (agent loop, retry and escalation rules), Observability (logged artifacts and traces), Verification (validation, self-checking, anomaly detection), and Governance (permission model, side-effect boundaries, human approval points). Field-level disclosure requirements per layer are in Appx. A (Table A1). The goal is to make configuration visible, not to standardize it, so that a reader comparing two cards can locate whether a score difference is attributable to the model, the harness, or their interaction.

Variance decomposition protocol. The variance decomposition becomes a protocol when an evaluation runs the same model under at least two meaningfully different harnesses and the same harness under at least two models on a fixed task set. The minimal valid design is a $2\times 2$ model-by-harness grid with task order, execution environment, evaluation script, API parameters, and stopping rules held constant. A harness difference counts as meaningful only if it changes at least one ETCSOVG layer in a way expected to affect stability, drift, or control lag, for example retrieval-based context compression, a change to the tool schema and error format, or added verification and recovery hooks. For a designed $\left\{M_i\right\}\times \left\{H_j\right\}$ grid, we recommend reporting four statistics alongside the headline score: HV per model and MV per harness, the aggregate ratio $\overline{\mathrm{HV}}/\overline{\mathrm{MV}}$, the count of model-pair ranking reversals across harnesses, and the partial eta-squared coefficient

$${\eta }_p^2={\displaystyle \frac{S{S}_{\mathrm{interaction}}}{S{S}_{\mathrm{interaction}}+S{S}_{\mathrm{error}}}}$$

(2)

where $SS$ denotes the sum of squares from a fixed-effects two-way ANOVA. Three caveats apply: ${\eta }_p^2$ is a fixed-effects quantity on the chosen grid (if harnesses are sampled from a larger population the appropriate quantity is the variance component ${\sigma }_{MH}^2$ from a mixed-effects model), it is positively biased in small grids and should be reported alongside ${\omega }^2$ or a bootstrap interval, and a large ${\eta }_p^2$ is neither necessary nor sufficient for ranking flips, so it should be paired with explicit reversal counts.

Trajectory-level metrics. Aggregate pass rates leave open which controller-side mechanism is producing HV when it is large. We propose three trajectory-level metrics that operationalize the three reliability quantities and let HV shifts be associated with specific harness layers. We do not estimate these metrics on the $3\times 3$ grid. Instead, we specify them as reporting requirements for future harness-aware evaluations. Recovery Rate $RR\left(k\right)$ operationalizes Stability as the probability that a trajectory transitions from a detected anomaly state (tool error, malformed output, failed validation, rejected patch) back to a task-advancing state within k steps. Reporting $RR$ as a curve over $k\in \{1,3,5,10\}$ is preferable to a single number because the choice of k trades promptness against eventual recovery. Context Retention operationalizes Context Drift as the fraction of task-relevant files, tests, and constraints present in the constructed context at each step, an auditable proxy that is monotone with respect to ${\delta }_t$. Control Lag τ is measured directly: the number of steps between a harness-detected anomaly and the arrival of a corrective signal at the policy. Together with the variance decomposition, these metrics support attribution rather than just measurement: a shift in $RR$ implicates verification and recovery layers, a shift in $\tau$ implicates observability and tool layers, and a shift in Context Retention implicates the context construction policy. The resulting attribution is descriptive co-movement, not causal. Layer-ablation designs that toggle a single ETCSOVG layer at a time are the natural extension once these metrics are routinely collected. Two complementary diagnostics extend the framework as future work for the community: Schema Compliance Rate measures the fraction of model outputs that parse without harness intervention (interface alignment), and Action Efficiency measures the ratio of productive, task-advancing actions to total actions (trajectory productivity). Both supplement, rather than substitute for, the three reliability quantities.

6. Alternative Views and Counterarguments

Four counterarguments deserve a direct response: that scaling models will dissolve the problem, that standardization is the right reply, that interaction effects are small enough to ignore, and that the harness, as part of the deployed agent, cannot be cleanly separated in evaluation.

Model capability will dissolve the harness problem. One might argue that sufficiently capable models will manage their own context and recovery, removing the need for external infrastructure. The empirical trend goes the other way: the harness ecosystem has grown more complex as models have improved, with Claude Code redesigning rather than removing harness components, the managed-agents architecture explicitly decoupling model and harness, and AHE producing double-digit gains on top of frontier base models [5,21,22]. The closed-loop account explains why: long-horizon reliability is a controller property, and no model upgrade compensates for the absence of feedback structure.

Standardization is enough. One might concede that harness variance is large but argue for standardizing the harness across the field rather than rethinking reporting. We partially endorse it: locked-harness protocols are one of the two valid regimes. But any standardized harness embeds design choices that interact with model properties, as Epoch AI itself flags when recommending mini-SWE-agent [6], and locking the harness forecloses harness-side gains that current evidence shows are large [5,10,11]. Standardization is also unlikely to be uniform in practice: agent harnesses manage stateful execution across multiple layers, and vendors have strong competitive incentives to differentiate at the runtime layer. Whether the harness is locked or varied, the position is that the configuration must be disclosed and treated as part of the experimental condition.

Interaction effects are small enough to ignore. One might concede that HV is large but argue that the interaction term is small, so any one harness still recovers a stable ordering. SWE-bench Pro data refute this: under SEAL the top six frontier models span only 4.9 points, and adding the WarpGrep subagent flips MiniMax 2.5 versus Claude Opus 4.6 [13]. The LangChain Terminal-Bench result shows the same coupling, where a harness tuned around one model yields a lower score when paired with another [35]. Our controlled grid confirms this under matched conditions: six ranking reversals across nine model-pair / harness-pair comparisons.

The harness is part of the deployed agent, so separating is artificial. One might argue that since deployed agents are (model, harness) compositions, attributing performance to one factor is a category error. This argument is correct as a description of deployment, but deployment optimizes the agent as a whole while evaluation that compares models must isolate the model contribution. Harness improvements are real gains that deployments rightly capture. The position is that benchmark scores read as model rankings must disclose the harness, because otherwise the rankings are not informative about the quantity readers are trying to measure. The disclosure standard and variance decomposition protocol keep deployment-style and model-comparison-style attribution from being conflated.

7. Conclusion and Discussion

The position has three audiences. For researchers, reviewers should ask “what harness was used?” as routinely as they ask about hyperparameters and decoding settings, since a model score without a harness specification is missing part of the experimental condition. For benchmark designers, agent benchmarks should expose harness variation as a first-class evaluation dimension, either through locked-harness tracks for clean model comparison or factorial tracks that report variance decompositions, building on programs underway in HAL [14], the unified scaffold of SWE-bench Pro [20], and standardization recommendations from Epoch AI [6]. For practitioners, model selection alone is an incomplete optimization loop: in our controlled grid, moving from $H_1$ to $H_3$ shifts pass@1 by 8.5 to 13.0 percentage points at a fixed model, while changing the model at a fixed harness shifts pass@1 by 2.5 to 5.0 points. The engineering object is not only the model endpoint but the context, tool, recovery, verification, and governance surfaces through which the model operates.

Open questions. The position raises four questions that remain open for the community. First, the operational definition of comparable frontier models is currently informal and anchored to leaderboard proximity that is itself harness-confounded, so a non-confounded test for comparability is a prerequisite for cross-paper meta-analysis of HV / MV ratios. Second, harness diversity needs a principled notion of distance between configurations, without which ${\mathrm{Var}}_H\left[B(M,H)\right]$ depends on the sampling distribution $P\left(\mathcal{H}\right)$ in ways not yet standardized. Third, the right balance between disclosure and locked-harness evaluation is a community decision rather than a purely technical one, since disclosure alone may not address adversarial harness selection while locking the harness slows infrastructure innovation. Fourth, the trajectory-level diagnostics admit natural extensions for the community to develop: Schema Compliance Rate and Action Efficiency would supplement the three reliability quantities as interface-alignment and productivity probes, and the perturbation stress-test detailed in Appx. C would test whether controller-side mechanisms reduce output instability under matched perturbations.

The execution harness is the binding constraint on long-horizon LLM-agent performance in the regime where benchmark leaderboards drive research and product decisions. The model is an open-loop policy and the harness is the controller that closes the loop, so reliability over a horizon is a controller property. Long-horizon agent evaluation should disclose Harness Cards, report variance decompositions when feasible, and include trajectory-level metrics that make recovery, drift, and control lag visible. Until those practices are routine, leaderboard comparisons for long-horizon agents should be treated as incomplete and potentially misleading.