Trustworthy LLM Harnesses: A Measurement Framework for Correctness, Quality, Efficiency, Reliability, Robustness, Safety, and Fairness

Large language model (LLM) agents are no longer deployed as bare models. They run inside an execution harness, the control layer that assembles context, exposes tools, manages memory and state, schedules retries, and verifies intermediate results. Because the harness determines what the model sees and what its outputs are allowed to do, observed agent behavior is jointly produced by the model and the harness, yet prevailing benchmarks attribute it to the model alone. We study the problem of harness evaluation: holding the model, task distribution, environment, budget, and evaluator fixed, how can the contribution of the harness be isolated, and what does it mean for a harness to be trustworthy? We formalize the harness as a trajectory-generating policy and define the harness effect as a paired, metric-indexed contrast. We then develop measurement principles along seven dimensions—outcome correctness, quality, efficiency, reliability and reproducibility, robustness, safety, and fairness—each targeting the specific mechanisms through which a harness influences agent behavior, and grounded in established results from statistical evaluation, efficient inference, certified robustness, and fair learning. A set of elementary propositions clarifies when paired designs reduce variance, how approximate verification bounds correctness loss, why marginal subgroup gaps can mislead, and how differential evaluator bias contaminates harness comparisons. We conclude with a reference protocol and argue that harness quality should be reported as a Pareto profile rather than a single leaderboard score.