The Evaluation Bottleneck of Vision-Language-Action Models: A Evaluation-Centric Survey

1. Introduction

Vision-Language-Action (VLA) models which map visual observations and language instructions to robot actions have rapidly become the dominant paradigm for learning-based robotic manipulation (Black, Brown, et al., 2024; M. J. Kim et al., 2024; Zitkovich et al., 2023). Recent models report striking numbers: over 98% average success on LIBERO (H. Luo et al., 2026; Y. Yang, Zeng, et al., 2026), average sequence lengths above 4.0 on CALVIN (Mees et al., 2021), and increasingly strong zero-shot transfer across unseen objects, scenes, and embodiments (S. Liu et al., 2026; S. Ye et al., 2026). Taken at face value, these results suggest that embodied manipulation may be approaching a solved problem.

However, a closer look at the benchmarks on which these claims rest, including what they measure, how they measure it, and what they leave unmeasured, suggests a more cautious answer. Figure 1 illustrates this widening evaluation gap: VLA models are moving beyond single-arm tabletop manipulation toward dexterous, deformable, whole-body, and multi-robot collabration, while in evaluation, critical dimensions such as coordination quality, recovery, safety, contact stability, and trajectory consistency remain weakly measured. The question is not whether current VLAs are impressive. They clearly are. The real question is whether our evaluation practices are strong enough to tell us what these impressive numbers actually mean. Different research groups evaluate on different simulation suites, report different metrics, and rely on lab-specific real-world protocols. As a result, it is increasingly difficult to know whether a reported gain reflects a genuine improvement in robotic capability, or simply a favorable choice of benchmark, metric, or evaluation setup.

This survey examines that problem from a benchmark-centric perspective. Rather than focusing on how VLA models are built, we ask how they are evaluated. Across 582 VLA papers published between 2023 and May 2026, we find that evaluation shortcomings are not isolated but systematic. To diagnose them precisely, we ground this analysis in the principle that a benchmark number supports a progress claim only if four conditions hold jointly: the benchmark discriminates among top VLA models rather than clustering them near its ceiling; the reported metrics capture the capability actually being claimed, not only binary success; the real-world evaluation procedure is standardized enough to permit cross-paper comparison, rather than being locked to a single lab setup; and the inference from simulation score to real-world readiness is warranted by explicit calibration of the simulator against the target physical setup. Figure 2 shows that current practice fails at all four.

Empirically, our surveyed paper shows that current VLA evaluation falls short on each condition. First, benchmark evidence is concentrated in a few simulation suites: LIBERO, SimplerEnv, and CALVIN account for over 70% of papers with simulation evaluation, while recent models now cluster above 98% on LIBERO, reducing discriminative power. Second, metric coverage remains narrow: 98.3% of papers report task success rate, 57.9% report it as the sole main metric, and only 16.7% report confidence intervals, leaving efficiency, safety, smoothness, and recovery weakly measured. Third, real-robot evaluation is difficult to compare because most studies use lab-specific hardware, objects, scenes, reset procedures, and success criteria. Fourth, simulation-to-deployment inference is valid only when explicitly calibrated, yet task-centric simulation scores are often interpreted as evidence of real-world readiness without paired physical validation.

In response, this paper provides: (1) a two-axis taxonomy that maps the full benchmark landscape by evaluation setting and evaluation focus, exposing which capabilities are well covered and which remain untested; (2) a quantitative diagnosis of current evaluation practice against the four conditions, documenting benchmark saturation, metric narrowness, real-robot fragmentation, and the proxy-validity gap across 582 papers; and (3) an analysis of emerging capability frontiers where the mismatch between model ambition and evaluation capacity is widening most rapidly. To support more disciplined evaluation going forward, we also provide an evidence hierarchy for simulation benchmarks in Appendix E, minimum reporting protocols for simulation and real-robot evaluation in Appendix D, and a claim-metric alignment guide in Appendix D.3.

2. Related Work

Recent VLA surveys have reviewed the field from different perspectives. Several works take a broad embodied-AI or application-oriented view, summarizing VLA concepts, model components, training paradigms, robot platforms, datasets, applications, and open challenges (Kawaharazuka et al., 2025; Y. Ma et al., 2026; Sapkota et al., 2025). Others are more model centric, organizing VLA models by architecture, learning paradigm, module design, or deployment pipeline, including autoregressive, diffusion-based, reinforcement-based, hierarchical, and large-VLM-based approaches (Kawaharazuka et al., 2025; C. Xu et al., 2025; D. Zhang et al., 2025).

A further group studies VLA models from more specialized angles. Y. Zhong et al. (2025) analyze VLA models through action tokenization; Xiang et al. (2026) connect VLA post-training with human motor learning; and S. Jiang et al. (2025) survey VLA models for autonomous driving. These surveys mainly focus on model design, training, applications, or domain-specific adaptation, while evaluation is usually treated as one component of a broader review.

The closest work Z. Wang, Wang, et al. (2026) to ours is the data-centric survey of VLA datasets, benchmarks, and data engines. While that work catalogs what benchmark and data resources exist for VLA research, our survey asks a different question: whether the results reported on these benchmarks can actually support the progress claims made from them. Specifically, we shift the focus from benchmark availability to benchmark validity, analyzing whether current evaluations discriminate among strong models, report metrics aligned with claimed capabilities, enable cross-paper comparison, and justify simulation-to-real-world inference. Thus, our work complements prior resource-centric surveys by auditing the evidential strength of VLA evaluation practice rather than only organizing its infrastructure.

3. A Taxonomy of VLA Benchmarks

Before diagnosing whether the four conditions are met in current practice, we first need a coordinate system: what benchmarks exist, what they cover, and what they leave out. We organize the benchmark landscape along two axes. As shown in Table 1, the first axis is the evaluation setting: closed-loop simulation, real-to-sim proxy evaluation, and real-robot evaluation. The second axis is the evaluation focus: the specific capability or failure mode that the benchmark is designed to probe, such as skill breadth, long-horizon composition, robustness, language grounding, safety, manipulation complexity, or deployment approximation.

This taxonomy exposes a common source of ambiguity in VLA evaluation: benchmark setting is often mistaken for evidential strength, blurring the distinction between what a benchmark tests and what conclusions it can support.

Simulation is not a single category of evidence. Task-centric suites such as LIBERO, CALVIN, RoboCasa, ManiSkill, and RLBench primarily support claims about benchmark-specific manipulation competence. Robustness-oriented suites test whether this competence survives controlled distribution shifts. Real-to-sim proxy benchmarks such as SimplerEnv and REALM ask a stronger and different question: whether simulated evaluation preserves real-world policy behavior, ranking, or sensitivity under calibrated visual and control conditions. These are different evidential claims, even when all are implemented in simulation.

Likewise, “real-robot evaluation” is not automatically comparable or conclusive. A custom hardware demonstration with few trials and vague success criteria may be less informative for cross-paper comparison than a controlled simulation benchmark, whereas centralized or protocol-standardized real-robot platforms provide stronger evidence because they fix more of the evaluation procedure. The relevant question, therefore, is not only where evaluation occurs, but what uncertainty the benchmark removes. The taxonomy below remains a two-axis map of evaluation setting and evaluation focus, whereas the benchmark evidence levels in Appendix E provide a separate claim-calibration layer. These evidence levels are not meant to rank benchmarks by quality, but to bound what kinds of progress or deployment claims a benchmark score can reasonably support.

4. Current Evaluation Practice

We test each of the four conditions against current practice. The picture that emerges is consistent: all four conditions fail. Before presenting the findings, we briefly summarize the corpus construction. Detailed collection, screening, and annotation procedures are provided in Appendix A.

4.1. Benchmark Concentration and Saturation

The first condition requires that a benchmark discriminates among the models being compared. The most visible pattern in current VLA evaluation is a concentration of evidence around a small set of simulation benchmarks that increasingly fails this requirement.

The concentration of benchmark evidence is stark. The three most adopted simulation benchmarks, LIBERO B. Liu et al. (2023), SimplerEnv X. Li, Hsu, et al. (2024), and CALVIN Mees et al. (2021), together account for over 70% of all VLA papers with simulation evaluation. Further subset targets capabilities outside benchmark’s scope(e.g., soft-body or deformable-object manipulation rather than rigid tabletop tasks). Among papers whose capability claims fall within table rigid tabletop coverage, the adoption rate is therefore substantially higher.

High adoption, however, carries a cost. When many papers optimize for the same benchmark and leading models approach the ceiling, small numerical gains become difficult to interpret as evidence of genuine progress.

Table 2 illustrates this issue. Several recent models report LIBERO averages above 98%, and the models cluster within roughly two percentage points of each other. At this level, a spread of roughly two percentage points may be driven by variation in checkpoint selection, random seeds, evaluation episodes, or implementation details, rather than by the advancing of model capability.

This concentration also shapes which capabilities receive attention. Benchmarks that are easy to run, historically established, and widely reported tend to become the default evidence for progress, even when they no longer strongly separate models. Harder benchmarks that test richer visual scenes, longer task horizons, or systematic perturbations appear in fewer than 30% of surveyed papers, despite offering substantially more diagnostic headroom. For example, while many models exceed 98% on LIBERO, RoboCasa scores for strong recent models remain closer to the 40–55% range, suggesting that it exposes visual, semantic, and manipulation complexity that LIBERO do not.

Sequential benchmarks Mees et al. (2021), which measure how many chained few subtasks a policy completes, offer more headroom than saturated single-task suites. However, their performance both are sensitive to training recipes, checkpoint selection, and evaluation variants, so cross-paper numerical comparison remains harder than leaderboard-style tables suggest.

The broader lesson is that benchmark standardization and benchmark informativeness are not the same. A benchmark can be highly standardized yet weakly discriminative once top models saturate it. Conversely, a newer benchmark can be more informative about unsolved capabilities while still lacking enough adoption for broad cross-paper comparison. Current VLA evaluation therefore faces a coordination problem: the community benefits from shared benchmarks, but excessive reliance on saturated benchmarks narrows the evidence base for progress claims.

4.2. Metric Coverage Gap

The second major pattern is the dominance of task success rate. Across our corpus, 98.3% of papers report task success rate as the main performance measure, and 57.9% report it as the sole metric in their experimental results. Success rate is simple and intuitive, but it collapses embodied behavior into a binary outcome and hides deployment-relevant properties such as speed, smoothness, safety, uncertainty, recovery, and robustness.

Several alternative metric families Ji et al. (2026) have begun to appear: partial-credit and progress scores for long-horizon tasks, sequential completion length for multi-step chaining, trajectory-level measures such as smoothness and reaction time, and efficiency metrics such as inference latency and control frequency. Each captures information that binary success rate discards, yet none has reached widespread adoption.

The metric gap matters because similar success rates can hide very different deployment behavior. A slow policy may succeed in static simulation but fail in dynamic manipulation; a jerky policy may complete a tabletop task while producing unsafe contacts; and a model may reach the goal through a path that would be unacceptable on hardware. Success rate alone cannot distinguish these cases, nor can it reveal rare but catastrophic failures under perturbation.

A number of recent papers J. Lin et al. (2025); J. Tang et al. (2025a); S. Xu et al. (2025) have begun reporting inference latency or control frequency alongside success rate, revealing that speed and accuracy are coupled in closed-loop control: improving benchmark success while reducing control frequency may not constitute a real deployment improvement. However, these remain isolated efforts rather than community practice. This pattern shows little change over time: from 2023 to 2026, 57.9% of papers report only success rate and out-of-distribution success rate, and the share of papers reporting latency, safety, or confidence intervals has not meaningfully grown. Full year-by-year statistics are provided in Appendix B.

Statistical reporting is also underdeveloped. Only 16.7% of surveyed papers report confidence intervals or comparable uncertainty estimates. This is especially problematic under benchmark saturation, where top LIBERO scores may differ by only one or two percentage points. Without trial counts, confidence intervals, or significance testing, small reported gains are difficult to distinguish from evaluation noise.

4.3. Fragmented Real-Robot Evaluation

Real-robot evaluation provides the strongest evidence of physical deployment capability, but current practice remains fragmented. Unlike simulation benchmarks, where task definitions and evaluation scripts are increasingly standardized, physical evaluations often differ in robots, objects, scenes, resets, success criteria, and trial budgets. As a result, a real-world result may be impressive within one laboratory setup but difficult to compare with another paper.

Current real-robot evaluation follows two main patterns. The most common is paper-specific evaluation: each study designs its own task suite around the capabilities it wishes to highlight, whether multi-embodiment long-horizon manipulation (Black, Brown, et al., 2024), transfer to unseen real-world scenes (S. Ye et al., 2026), or bimanual coordination (J. Liu et al., 2025). These evaluations can target realistic or novel capabilities, but because they differ in hardware, object sets, scene layouts, reset procedures, and success criteria, they rarely support direct cross-paper comparison. A second, newer pattern is the emergence of standardized real-robot benchmarks that attempt to fix one or more of these variables (Atreya et al., 2025; Y. Chen et al., 2026; Y. Sun et al., 2026a; Z. Wang, Liu, et al., 2026; Yakefu et al., 2025a). These efforts adopt different standardization strategies, centralizing hardware execution, shipping uniform object kits, or publishing shared task protocols and scoring rubrics, but adoption remains limited. A paper-level summary of real-robot platforms, scene scales, evaluation types, and task protocols is provided in Appendix I.

A pervasive concern across both patterns is statistical power. Many real-robot studies report only 10–20 trials per task, and some report fewer. Low trial counts also obscure failure structure: an aggregate success rate rarely reveals whether failures arise from perception, language grounding, grasp instability, controller latency, object slippage, unsafe contact, or task ambiguity.

The central problem is therefore not the absence of real-robot evaluation, but its limited comparability and statistical strength. Until standardized physical benchmarks become more widely adopted, custom real-world results are best interpreted as deployment demonstrations rather than definitive evidence of general model superiority.

4.4. Conditional Validity of Simulation Evidence

The fourth condition requires that the inference from benchmark setting to the claimed deployment context is warranted. This condition targets a specific and widespread practice: interpreting scores on task-centric simulation suites as evidence of real-world manipulation capability.

The issue is not that simulation is intrinsically misaligned with the real world, but that its evidential status is often underspecified. Recent real-to-sim benchmarks show that carefully designed simulation can provide a meaningful proxy for physical evaluation. For example, SimplerEnvconstructs simulated environments around specific real-robot manipulation setups and reports strong correlations with real-world policy performance on its original Google Robot and Bridge V2 configurations. However, its proxy validity is bounded: it does not support wrist-camera policies and has been shown to fail at predicting the performance of recent generalist VLAs (Jain et al., 2025), illustrating that even a well-designed proxy applies only within the scope of its calibration. Similarly, REALM explicitly targets real-to-sim validation by combining high-fidelity visuals, aligned robot control, systematic perturbations, and paired real-world comparisons.

These results change how simulation should be interpreted in VLA evaluation. They weaken the broad claim that simulation and real-robot performance are necessarily poorly correlated. At the same time, they strengthen a more precise claim: proxy validity is conditional. A simulator can predict real-world behavior only to the extent that the relevant embodiment, action interface, visual distribution, task family, perturbation space, and metric have been aligned or empirically validated. This validity does not automatically transfer from one benchmark to another.

This distinction is often blurred in current VLA reporting. Standard task-centric benchmarks such as LIBERO, CALVIN, RoboCasa, ManiSkill, and RLBench are valuable for measuring reproducible manipulation competence under controlled conditions. However, they are not designed primarily to answer whether a policy will preserve its behavior on physical hardware. Real-to-sim benchmarks such as SimplerEnv and REALM instead evaluate a different evidential claim: whether simulation can approximate real-world policy ranking, sensitivity, and failure modes under a calibrated setup. Both benchmark types are useful, but they support different conclusions.

The inference-validity condition is therefore unmet whenever a paper presents task-centric simulation scores as evidence of deployment readiness without providing either physical evaluation or an explicitly validated simulation proxy. The evaluation gap is no longer simply a sim-to-real gap, but a proxy-validity gap: VLA papers need to state whether each simulation benchmark is being used as a competence test, a robustness test, or a validated proxy for real-world behavior, and the strength of their conclusions should be bounded accordingly.

5. The Widening Evaluation Gap

Section 4 diagnoses four failures in current VLA evaluation. Figure 1 shows why these failures are not merely static shortcomings, but symptoms of a widening evaluation bottleneck: as VLA models move toward longer-horizon, dexterous, whole-body, and multi-robot tasks, the metrics and protocols needed to evaluate these capabilities remain underdeveloped. We therefore treat the evaluation gap as a dynamic problem: models are moving faster than evaluation infrastructure, and the distance between them is growing. We highlight four frontiers where the mismatch between model ambition and evaluation capacity is most acute.

Long-horizon memory and reasoning

Current benchmarks Mees et al. (2021) cap task complexity at roughly five chained subtasks or short multi-step sequences. Real-world tasks of practical interest, such as tidying a room, preparing a full meal, or assembling a piece of furniture, involve tens to hundreds of sequential decisions, with intermediate failures that require error detection, recovery, and subgoal replanning.

At this scale, binary task success rate loses almost all diagnostic value. Evaluating long-horizon behavior requires fundamentally different metrics: partial-credit scoring that reflects how far the policy progressed, failure-locality measures that identify where in the task sequence errors concentrate, and explicit assessment of recovery capability after intermediate failures.

A further challenge is that long-horizon tasks amplify the compounding effect of per-step error rates. Even a policy with 99% per-step reliability reaches only $0.{99}^{100}\approx 37\%$ full-task success over 100 steps. This means that long-horizon evaluation must separately measure per-step reliability, compounding behavior, and recovery rate, rather than relying on end-to-end success alone.

Dexterous and contact-rich manipulation.

Dexterous manipulation sharpens the metric-coverage and proxy-validity problems discussed above. In contact-rich tasks, endpoint success is a weak proxy for the claimed capability: a policy may finish the task while relying on unstable grasps, excessive force, unsafe contacts, or simulator-specific contact artifacts. Conversely, a failed trial may still reveal meaningful partial dexterity, such as stable regrasping, compliant contact, tool alignment, or recovery after slip. Binary success therefore hides precisely the behaviors that distinguish dexterous manipulation from ordinary pick-and-place.

Comparability is also weaker than in standard tabletop manipulation. Existing dexterous benchmarks vary in hand morphology, action space, controller design, tactile sensing, contact modeling, and demonstration source. Recent efforts such as DexArt Bao et al. (2023), Ego Humanoid Manipulation R. Yang et al. (2025), RoboCasa Nasiriany et al. (2024), GR-1 tabletop tasks NVIDIA et al. (2025), and DexJoCo H. Wang et al. (2026) provide useful infrastructure, but their heterogeneity makes cross-paper ranking difficult. A score on one hand–controller–simulator stack rarely establishes transferable dexterity.

The resulting evaluation need is clear: dexterous VLA benchmarks should report not only task completion, but also grasp stability, slip, contact quality, force or torque violations, object damage, trajectory smoothness, recovery after failed contacts, and transfer across morphologies or controllers. Without these dimensions, dexterous manipulation risks becoming another frontier where benchmark success is mistaken for embodied capability.

Multi-robot coordination.

Bimanual manipulation, where benchmarks such as RoboTwin 2.0 have begun to provide evaluation support, represents the simplest case of multi-agent coordination: two arms controlled by a single policy with shared observations and a common objective. The next frontier involves genuinely distributed coordination: multiple robots with separate observation spaces collaborating on shared tasks, human-robot teams requiring dynamic role allocation, and multi-robots where individual objectives may conflict.

Evaluating these scenarios introduces dimensions that single-agent benchmarks do not address. Task success alone cannot capture whether coordination was efficient, whether communication overhead was acceptable, whether task allocation was reasonable, or whether one agent’s failure was compensated by another’s adaptation. Metrics must separately assess individual agent competence, coordination quality, communication efficiency, and robustness to single-agent failure. No existing VLA benchmark provides evaluation support for any of these dimensions, despite the growing number of multi-robot manipulation being proposed.

Domain-specific fine manipulation.

The current benchmark landscape is built around a general-purpose manipulation paradigm: rigid or semi-rigid objects on tabletops, grasped and relocated by parallel-jaw or multi-finger grippers. However, some of the highest-value deployment scenarios for VLA models involve domain-specific manipulation where success criteria, precision requirements, and safety constraints are qualitatively different from general-purpose benchmarks.

Chemical synthesis requires sub-millimeter precision in liquid transfer, strict contamination control, and force-sensitive handling of fragile glassware. Surgical manipulation involves deformable soft tissue, real-time force feedback, and safety constraints that make any unintended contact a potential failure. Electronics assembly demands high-precision alignment, handling of small and fragile components, and tolerance specifications measured in micrometers. In each case, the relevant success criterion is not simply whether an object reached a target pose, but whether domain-specific constraints on force, purity, deformation, alignment, or biological safety were satisfied throughout the entire manipulation trajectory.

Current benchmarks largely fail to express these constraints because they lack the physical simulation fidelity such as fluid dynamics, soft-body deformation, contamination modeling, the domain-specific success criteria, and the safety metrics that these applications demand.

Recent efforts such as AutoBio Lan et al. (2025); R. Li et al. (2026) begin to address this gap by evaluating language-guided robotic manipulation in specialized scientific workflows, and their results suggest that current VLA models still struggle under precision-demanding domain constraints. These domains expose a broader limitation of current VLA evaluation: benchmark success on general tabletop manipulation does not necessarily imply readiness for high-precision, safety-critical deployment.

These frontiers share a common pattern: the evaluation challenge is not simply to build harder tasks within the existing benchmark paradigm, but to develop new evaluation dimensions, metrics, and protocols that the current paradigm does not support. Long-horizon reasoning requires temporal credit assignment and recovery measurement. Multi-agent coordination requires interaction-level metrics beyond individual task success. Domain-specific manipulation requires constraint-satisfaction evaluation throughout the trajectory rather than endpoint-only assessment. None of these needs can be met by adding more tasks to LIBERO or increasing trial counts on existing benchmarks.

The evaluation bottleneck documented in this survey is therefore not only a present-day problem but a forward-looking one. As VLA models move toward these harder capability frontiers, the gap between what models can attempt and what benchmarks can measure will continue to widen unless the community invests in evaluation infrastructure with the same urgency it currently devotes to model development. For capabilities where standardized evaluation already exists, Appendix D provides minimum reporting protocols, including separate simulation and real-robot checklists in Appendix D.1 and Appendix D.2, while Appendix E provides an evidence hierarchy to guide benchmark selection and claim calibration. For the emerging frontiers discussed here, the first step is recognizing that evaluation design is itself a research problem.

6. Conclusions

This survey identifies evaluation validity as a central bottleneck for VLA progress. Across surveyed papers, we find that current evaluation relies on saturated simulation benchmarks, success-rate-dominated metrics, fragmented real-robot protocols, and weakly justified sim-to-real inference. Future VLA research needs stronger evaluation infrastructure, not only stronger models. Benchmarks, metrics, and real-world protocols should be matched to the claims they support, especially as models move toward long-horizon, dexterous, multi-robot, and domain-specific manipulation.

Limitations

This survey focuses on VLA models for robotic manipulation and does not comprehensively cover navigation-only or locomotion-only benchmarks. Independent verification across all benchmarks was not feasible. Additionally, our real-robot evaluation analysis may undercount proprietary evaluation setups that are not fully described in publications. For real-to-sim proxy benchmarks, our discussion relies on the validation protocols and correlations reported by the benchmark authors rather than independent re-execution across all evaluated policies and physical setups.

Appendix D.1. Simulation Evaluation

Core reporting (R).

• Task success rate per benchmark suite, reported at the individual-task-family level rather than only as an aggregate average.
• Number of evaluation episodes per task.
• Number of random seeds, with per-seed results or standard deviation.
• 95% confidence intervals or standard errors for all reported success rates.
• Inference latency (mean and $p_{95}$, measured in milliseconds on specified hardware).
• Effective control frequency (Hz) during closed-loop rollout.
• Total wall-clock rollout time per episode.
• Benchmark version, evaluation script commit hash or release tag, and any non-default configuration.

Long-horizon tasks (C).

Required when the paper claims multi-step or compositional capability.

• Subtask completion rate (fraction of individual subtasks completed, independent of full-sequence success).
• Average completed sequence length (e.g., CALVIN-style metric).
• Progress score or partial-credit metric, if supported by the benchmark.
• Distribution of failure positions along the subtask sequence (e.g., histogram showing at which step failures concentrate).

Robustness (C).

Required when the paper claims robustness or generalization under distribution shift.

• Absolute success rate under each perturbation type (object, layout, camera, lighting, background, initial state, sensor noise).
• Relative success-rate drop ($\Delta$%) from the unperturbed baseline for each perturbation type.
• Whether perturbations are applied individually or in combination, and if combined, which combination protocol is used.

Language grounding (C).

Required when the paper claims language understanding beyond template matching.

• Success rate under the original (template) instructions.
• Success rate under paraphrased instructions.
• Success rate under semantically equivalent but syntactically different instructions.
• Success rate under ambiguous or underspecified instructions.
• Success rate under distractor or irrelevant instructions.

Appendix D.2. Real-Robot Evaluation

Setup description (R).

• Robot embodiment (manufacturer, model, DoF, end-effector type).
• Camera configuration (number, mounting positions, resolution, whether wrist cameras are used).
• Action space definition (joint position, joint velocity, end-effector pose, or hybrid).
• Control frequency (Hz) on the deployed hardware.
• Object set (number of objects, material categories, whether objects are seen or unseen during training).
• Scene layout (workspace dimensions, fixture positions, lighting conditions).
• Reset procedure (manual, semi-automatic, or automatic; degree of randomization per trial).
• Success criteria (explicit definition, including tolerance thresholds).

Statistical reporting (R).

• Number of trials per task (minimum recommended: 50 for primary claims; see discussion below).
• Success rate with 95% binomial confidence intervals.
• Task completion time (mean and standard deviation, in seconds).
• Inference latency on the deployed hardware (mean and $p_{95}$, in milliseconds).

Operational metrics (R).

• Human intervention rate (fraction of trials requiring manual correction during execution).
• Collision rate (fraction of trials in which unintended contact occurs).
• Constraint-violation rate (fraction of trials violating predefined workspace, force, or safety constraints).
• Unsafe-contact rate (fraction of trials producing contacts that could damage the robot, object, or environment).
• Reset failure rate (fraction of trials in which the reset procedure itself fails or introduces bias).

Failure analysis (R).

Papers should report a failure-mode breakdown that, at minimum, distinguishes the following categories:

• Perception errors: the policy misidentifies or fails to locate the target object.
• Language-understanding errors: the executed behavior does not match the instruction semantics.
• Planning errors: the policy selects an incorrect or suboptimal action sequence.
• Grasping errors: the end-effector fails to acquire or maintain a stable grasp.
• Control errors: the low-level controller produces jerky, oscillatory, or divergent motions.
• Recovery failures: the policy fails to recover after an intermediate error.
• Hardware or calibration failures: the trial fails due to mechanical, sensor, or calibration issues unrelated to the policy.

We recommend a minimum of 50 trials per task for primary performance claims, with the acknowledgment that hardware constraints may limit this in practice. When fewer than 50 trials are reported, papers should explicitly state the resulting confidence-interval width and refrain from drawing fine-grained comparative conclusions.

Appendix D.3. Claim-Metric Alignment Guide

Table A3 maps common capability claims to the minimum benchmark evidence and metric set required to substantiate them. A paper whose reported metrics do not cover the right-hand column for a given claim should not draw conclusions about that capability. The table is intended as a practical reference: authors can identify which claims their paper makes, then verify that the corresponding metrics are reported.

Table A3. Claim-metric alignment guide. Each row specifies the minimum benchmark and metric requirements for a common VLA capability claim. Papers that do not report the quantities in the right column should refrain from drawing conclusions about the corresponding capability.

Capability Claim	Minimum Benchmark Evidence	Required Metrics
Task competence	At least one standardized simulation suite (e.g., LIBERO, ManiSkill, RLBench); results on saturated suites should be supplemented with a more diagnostic benchmark.	Success rate with CI, trials/seeds, task-level breakdown.
Robustness / generalization	A robustness-oriented suite (e.g., LIBERO-Plus, The Colosseum, REALM perturbation splits) or controlled perturbation experiments on a standard benchmark.	Per-perturbation success rate, relative drop from baseline, perturbation protocol description.
Language understanding	Language-perturbation benchmark (e.g., LIBERO-Para, LangGap, RAMA-Bench) or controlled instruction-variation experiments.	Success under original, paraphrased, semantically equivalent, ambiguous, and distractor instructions.
Long-horizon reasoning	A sequential or compositional benchmark (e.g., CALVIN, LIBERO-Long, BEHAVIOR-1K).	Subtask completion rate, average sequence length, progress score, failure-position distribution.
Inference efficiency	Any benchmark, but latency must be measured on specified hardware under closed-loop control.	Inference latency (mean, $p_{95}$), control frequency, FLOPs or model size, rollout time.
Safety awareness	A safety-oriented benchmark (e.g., SAFE) or experiments with explicit constraint-violation conditions.	Collision rate, constraint-violation rate, unsafe-contact rate, failure-detection accuracy.
Real-world readiness	Either (a) a standardized real-robot benchmark, or (b) a real-to-sim validated proxy with stated calibration scope, or (c) custom real-robot evaluation with ≥50 trials per task and full operational reporting.	All items in Appendix D.2: success rate with CI, completion time, latency, intervention rate, collision rate, failure breakdown.
Cross-embodiment transfer	Evaluation on ≥2 distinct embodiments (different morphologies, not just different instances of the same robot).	Per-embodiment success rate with CI, description of embodiment differences, whether the policy is shared or separately adapted.

Table A3. Claim-metric alignment guide. Each row specifies the minimum benchmark and metric requirements for a common VLA capability claim. Papers that do not report the quantities in the right column should refrain from drawing conclusions about the corresponding capability.

Capability Claim	Minimum Benchmark Evidence	Required Metrics
Task competence	At least one standardized simulation suite (e.g., LIBERO, ManiSkill, RLBench); results on saturated suites should be supplemented with a more diagnostic benchmark.	Success rate with CI, trials/seeds, task-level breakdown.
Robustness / generalization	A robustness-oriented suite (e.g., LIBERO-Plus, The Colosseum, REALM perturbation splits) or controlled perturbation experiments on a standard benchmark.	Per-perturbation success rate, relative drop from baseline, perturbation protocol description.
Language understanding	Language-perturbation benchmark (e.g., LIBERO-Para, LangGap, RAMA-Bench) or controlled instruction-variation experiments.	Success under original, paraphrased, semantically equivalent, ambiguous, and distractor instructions.
Long-horizon reasoning	A sequential or compositional benchmark (e.g., CALVIN, LIBERO-Long, BEHAVIOR-1K).	Subtask completion rate, average sequence length, progress score, failure-position distribution.
Inference efficiency	Any benchmark, but latency must be measured on specified hardware under closed-loop control.	Inference latency (mean, $p_{95}$), control frequency, FLOPs or model size, rollout time.
Safety awareness	A safety-oriented benchmark (e.g., SAFE) or experiments with explicit constraint-violation conditions.	Collision rate, constraint-violation rate, unsafe-contact rate, failure-detection accuracy.
Real-world readiness	Either (a) a standardized real-robot benchmark, or (b) a real-to-sim validated proxy with stated calibration scope, or (c) custom real-robot evaluation with ≥50 trials per task and full operational reporting.	All items in Appendix D.2: success rate with CI, completion time, latency, intervention rate, collision rate, failure breakdown.
Cross-embodiment transfer	Evaluation on ≥2 distinct embodiments (different morphologies, not just different instances of the same robot).	Per-embodiment success rate with CI, description of embodiment differences, whether the policy is shared or separately adapted.