Rethinking Benchmark Comparability: A Survey of Reasoning Benchmarks for Large Language Models

1. Introduction

Reasoning has become one of the central axes of progress in large language model research [1,2,3]. As models are increasingly required to move beyond short response generation and support complex problem solving, the significance of reasoning has become progressively more pronounced [1,2]. Early reasoning in language models was largely confined to relatively short and task-specific intermediate computations, often lacking the depth required for complex inference [4]. The emergence of chain-of-thought prompting marked a major shift by showing that explicitly generated intermediate reasoning steps can unlock substantially stronger performance on complex tasks [5]. More recent research has pushed this paradigm toward longer and more exploratory reasoning trajectories, more deliberate System 2-style thinking, and even latent reasoning mechanisms beyond fully verbalized thought chains [6,7,8,9,10,11]. Meanwhile, reasoning abilities are expanding toward multi-step, multimodal, and cross-domain settings, enabling models to address increasingly diverse and complex problems [1,12,13,14,15,16].

Under such a trend, the quality of benchmarks largely determines whether reasoning evaluation can be conducted in an objective and reliable manner [6,17,18,19]. The need has grown rapidly in recent years, accompanied by a corresponding expansion in the range and diversity of relevant works [20,21,22,23,24,25]. Existing reasoning benchmarks now span symbolic reasoning, mathematical reasoning, knowledge intensive reasoning, and agentic decision making [26,27,28,29,30,31]. Although these benchmarks are often grouped together under the label of reasoning, they frequently differ substantially in the capabilities they target, the information by design, the protocol formulation they assume, and the logic by which outputs are evaluated [32,33,34]. Moreover, scores that appear to belong to the same category are often treated as comparable, though they often reflect materially different task semantics [35,36]. Therefore, it is no longer sufficient to ask whether a model can produce a correct answer. Rather, we must establish a scientific foundation for rethinking this incomparability [20,37,38].

Motivated by these challenges, we define the comparability of reasoning benchmarks not merely as an organizational inconvenience, but as a scientific problem in how reasoning benchmarks are produced, interpreted, and utilized. As shown in Figure 1, we present the first comprehensive survey of reasoning benchmarks for large language models from the perspective of benchmark semantics, benchmark construction, and benchmark evaluation, reframing the comparability of reasoning benchmarks. Rather than treating benchmarks as a flat collection of datasets, we organize the space through three closely connected dimensions, Object, Setting, and Evaluation. Object (What): Reasoning Capabilities of Reasoning Benchmarks identifies the target reasoning capability that a benchmark is intended to probe, including logical reasoning, mathematical reasoning, knowledge reasoning and agentic reasoning. Setting (How): Construction of Reasoning Benchmarks specifies the execution conditions under which that capability is elicited, including data provenance and the protocol formulation.

Evaluation (How Well): Assessment of Reasoning Benchmarks specifies the outcomes to be evaluated and how they are measured, including evaluation units and evaluation dimensions. To supply and expand the methodology, we further instantiate it through an exploration of extended scenarios, including multilingual and localized, multimodal, vertical-domain, as well as agentic and interactive benchmarks. This methodology makes benchmark explicit, separating capability evidence from protocol effects, and provides a practical guideline for organizing existing benchmarks as well as designing new ones. Finally, we identify two central weaknesses in current benchmark practice, namely excessive heterogeneity of benchmarks and insufficiently justified benchmark settings, and discuss future directions of reasoning benchmarks.

Our contributions are threefold.

• To the best of our knowledge, we provide the first comprehensive survey of reasoning benchmarks for large language models, framing a scientific foundation for how reasoning benchmarks are constructed, interpreted, and ultimately utilized in practice.
• We introduce a unified and operational perspective that connects benchmark Object, Setting, and Evaluation, and further instantiate it through an exploration of extended scenarios to support our methodology, and accordingly offer practical guidelines for benchmark selection and construction.
• We identify two central weaknesses in current practice as well as analyze the potential impact of these weaknesses on the scientific validity. Additionally, we outline future directions for reasoning benchmarks, covering measurement mechanics, promising directions and essential governance considerations.

2. Background

2.1. The Reasoning in LLMs

In this survey, reasoning is viewed as the capability to reach a correct final decision by maintaining logically dependent intermediate states under uncertainty and constraints [6,37,39,40]. This aspect distinguishes reasoning from surface fluency: as failures in reasoning often stem not from linguistic incoherence, but from step-level inconsistencies and violations of cross-step constraints [3,26]. A model can be linguistically coherent yet fail to satisfy cross-step constraints, and such failure is precisely what reasoning benchmarks are expected to expose [18,39,41].

Thus, reasoning can appear in multiple forms, including symbolic derivation, mathematical problem solving, knowledge-intensive evidence, and sequential tool-mediated decision making [42,43,44,45,46,47]. Although these forms differ significantly in domain, interface modality, and task formulation, they share a common reasoning structure: success depends on whether the model can derive an answer or an action through a sequence of interdependent intermediate states maintained over time [39,41,47,48,49,50,51,52,53].

2.2. Basic Elements of Benchmark Description

Following our definition of reasoning, a benchmark can be understood as a structured evaluation instrument that turns a reasoning claim into an assessable form [54,55,56,57]. More specifically, a benchmark specifies what capability is intended to be measured, under what conditions that capability is elicited, and how performance is judged. In this sense, benchmarks do not merely provide scores. They also determine what counts as evidence of reasoning and therefore shape how progress in reasoning is interpreted [51,58,59].

This role is particularly important in the study of LLM reasoning. Unlike tasks with relatively direct input–output mappings, reasoning performance is often sensitive to benchmark design choices such as task formulation, data provenance, inference-time protocol, and scoring criteria [60,61]. As a result, a reported score may reflect not only a model’s underlying reasoning ability, but also the assumptions embedded in the benchmark itself [62,63,64]. For this reason, benchmark description is a necessary part of reasoning research, since benchmark results can only be interpreted properly when the capability, the setting, and the criteria are explicit [59,65,66].

2.3. The trend of reasoning benchmarks

In recent years, the development of reasoning benchmarks has been closely shaped by the rapid improvement of large language models reasoning abilities [67,68,69]. As shown in Figure 2, as these models have grown stronger, the landscape of reasoning benchmarks has expanded not only in scale, but also in difficulty, coverage, and task composition [70,71,72]. Early reasoning benchmarks were typically built around short-context settings and relatively simple question-answering tasks [73,74,75]. These benchmarks were easy to scale and compare across models, but they were limited in the range of reasoning behaviors they could capture[73,74]. As performance on such tasks improved, benchmark design gradually moved beyond narrow, static QA toward more challenging suites that emphasize longer reasoning chains, complex problem structures, and diverse task settings [34,67,68,69,76]. Thus, the evolution of reasoning benchmarks has been marked not only by rising difficulty and quantity, but also by a broader reconsideration of the scenarios they are expected to cover [72,77,78].

2.4. The growing incomparability of reasoning benchmarks

However, this shift toward more difficult, diverse, and application-oriented benchmarks has also introduced a serious comparability challenge [47,71,72,78,79,80]. As reasoning benchmarks expand across domains, formats, and evaluation settings, they become increasingly heterogeneous in different objects [47,68,69,77].

As a result, benchmark scores are often no longer derived from comparable task assumptions: they may reflect different input lengths, interaction protocols, tool access, environmental constraints, and success criteria rather than performance on a shared reasoning scale [47,72,80].

3. Object (What): Reasoning Capabilities of Reasoning Benchmarks

As shown in Figure 3, this section clarifies the core capabilities that reasoning benchmarks are intended to evaluate. Given the multifaceted nature of reasoning, it is essential to establish a structured taxonomy for these evaluation objects. According to Figure , we organize target capabilities into four dimensions: Symbolic Reasoning, Mathematical Reasoning, Knowledge Reasoning, and Agentic Reasoning. For each dimension, we define the scope and list the sub-capabilities that benchmarks commonly instantiate.

3.1. Symbolic Reasoning

Symbolic reasoning can be divided into three benchmark sub-objects: Deduction, Logic and Proof and Abstraction and Induction. Deduction focuses on whether a model can derive valid consequences from explicit premises under a fixed rule system, while Logic and Proof emphasizes the construction, validation, and search of proof structures in semi-formal or formal settings. Abstraction and Induction further assesses the ability to infer general rules from examples and transfer them to new cases.

3.1.1. Deduction

Deduction is defined as controlling inference from explicit premises under a fixed rule system [84,85]. What makes it a distinct object is that benchmark success depends primarily on whether a model can compute consequence relations over stated facts and rules, rather than on background world knowledge, stylistic generation, or broad reading comprehension [28,85]. In this object, the benchmark oracle is usually exact, either as entailment labels, proof depth conditioned accuracy, or verifiable proof traces [84,85]. This object is instantiated in benchmarks that present a small theory, often in natural language but with tightly constrained semantics, and then ask whether a conclusion follows, contradicts the theory, or remains unknown [28,85].

Some benchmarks sharpen the same target by pairing synthetic worlds with proof chains that can be checked against an underlying formal semantics, which makes benchmark errors interpretable as failures of stepwise consequence computation rather than failures of retrieval or domain knowledge [84]. Other benchmarks extend the object toward open domain language while preserving first order logical grounding, so that the benchmark still centers on valid deduction instead of mere plausibility [28,113,290].

3.1.2. Logic and Proof

Logic and proof is a benchmark object whose target capability is maintaining logical validity across structured arguments and proof obligations [102,113,290]. Unlike deduction benchmarks, which mainly test closure under explicit premises, this object is defined by benchmarks in which the central challenge is to recognize valid argument structure, assemble proof steps, or search for proofs in semi formal or formal systems [102,113,290].

One benchmark family employs standardized logical reasoning tasks that derive their complexity from constraint satisfaction, quantified statements, and logical consistency. ReClor and AR-LSAT package logic into exam style multiple choice formats while still targeting proof relevant structure rather than open ended knowledge recall. A second family makes the proof artifact explicit. EntailmentBank asks for multi-step entailment trees grounded in scientific facts, so the benchmark object is instantiated as proof graph construction rather than answer selection alone. A third family moves to formal mathematics and theorem proving [102,113,290]. MiniF2F, ProofNet, and LeanDojo benchmark proof search, premise selection, formalization, or theorem completion in systems where correctness is checked by proof assistants [102,113,130,131,133,290].

When a benchmark is dominated by formal proof obligations, machine checked derivations, or explicit proof artifacts, it fits logic and proof even if the underlying content is mathematical [102,113,290]. By contrast, natural language contest problems whose main difficulty is derivational mathematical problem solving belong more naturally under advanced mathematics [75,158].

3.1.3. Abstraction and Induction

Abstraction and induction is a benchmark object whose target capability is inferring latent rules or concepts from sparse evidence and then applying them to novel instances. It is distinct from symbolic deduction because the rules are not fully given, and it is distinct from generic perception because benchmark success depends on discovering the governing transformation, relation, or concept rather than recognizing familiar surface patterns. The object is defined by benchmarks that minimize opportunities for memorization and instead force the model to construct a task specific abstraction [135].

Canonical examples are ARC-style benchmarks, where models must generalize from a handful of demonstrations to a held-out case, forcing construction of task-specific abstractions instead of memorization [135,142,291]. The same benchmark object also appears beyond visual formats, including program synthesis and code-generation settings, where success depends on inducing reusable solution structure under novelty [75,141]. More broadly, benchmarks in this family are unified not by modality but by the requirement to discover the governing transformation, concept, or schema before applying it [135,143].

3.2. Mathematical Reasoning

Mathematical reasoning can be divided into three benchmark objects: Basic Mathematics, Advanced Mathematics, and STEM Mathematical Reasoning. Basic mathematics centers on elementary quantitative problem solving that requires only short derivations; advanced mathematics emphasizes sustained reasoning and multi-step problem solving over challenging mathematical tasks; and STEM mathematical reasoning targets quantitative reasoning embedded within authentic scientific or engineering contexts.

3.2.1. Basic Mathematics

Basic mathematics benchmarks target the ability to solve short- to medium-horizon quantitative problems drawn from elementary arithmetic, introductory algebra, and basic geometry [114,144,145,149,155]. They are defined not simply by the presence of numbers, but by tasks that require mapping short verbal or diagram-grounded descriptions into elementary operations, equations, or intermediate quantities with exact answers [74,112,149,150,152]. The emphasis is usually on procedural correctness over familiar school-level content, rather than advanced theory or long formal derivations [74,114,145].

Canonical examples include arithmetic and algebra word-problem benchmarks such as SVAMP, GSM8K, GSM-HARD and ASDiv [43,74,144,145,146,147,148,149]. Related datasets extend the same target to elementary geometry and mixed-format reasoning, including Geometry3K, GeoQA, PGPS9K, UniGeo, GeoQA+, Euclid30K, UGMATHBENCH, and MATHEMAGIC [29,112,150,151,152,153,154,155]. Some benchmarks further stress robustness by perturbing wording, quantities, or symbolic templates, testing whether models capture underlying quantitative structure rather than brittle lexical patterns [43,144,147,148]. Across these variants, the core target remains elementary mathematical composition with relatively short derivational chains [43,74,144,147]. Once tasks require competition-level ideas, substantial theorem use, or sustained symbolic derivations, the focus shifts toward advanced mathematics or broader STEM reasoning benchmarks [75,158,170,171].

3.2.2. Advanced Mathematics

Advanced mathematics benchmarks target sustained mathematical problem solving beyond elementary curricula. Their difficulty lies not only in long solution chains, but also in selecting the right mathematical ideas, maintaining symbolic consistency, and producing coherent derivations or proof sketches under limited prompt scaffolding [75,132,158]. In this benchmark object, the emphasis is natural-language mathematical reasoning rather than routine calculation or short-form answer retrieval [75,132,158].

This object includes several closely related benchmark families. Foundational static corpora include MATH and a growing set of contest-style collections such as OlympiadBench, PutnamBench, and Putnam-AXIOM, which span olympiad, Putnam-level, and other advanced competition problems across algebra, number theory, geometry, and combinatorics [42,75,156,157,158,159,163,164,165,292]. A second line of work focuses on harder or less saturated evaluation, as in FrontierMath and MathArena, which aim to better distinguish genuine reasoning from benchmark memorization or contamination [132,161]. A third line examines model robustness, formal verification, and proof-sensitive mathematical reasoning through benchmarks such as MATH-Perturb, HARD2VERIFY, IMProofBench, and SKYLENAGE-MATH [121,127,160,162].

3.2.3. STEM Mathematical Reasoning

STEM mathematical reasoning benchmarks target quantitative reasoning embedded in scientific or engineering contexts. Unlike advanced mathematics benchmarks, their focus is not mathematical derivation in isolation, but the coordinated use of equations, units, laws, diagrams, and domain assumptions to model and solve problems in fields such as physics, chemistry, and engineering [156,170,171,293,294,295]. In these benchmarks, successful reasoning depends not only on correct calculation, but also on choosing the right scientific formalism and respecting domain constraints throughout the solution process.

This object encompasses several closely related benchmark families, each targeting distinct yet interconnected aspects of reasoning. Cross-domain suites such as SciBench and TheoremQA evaluate whether models can identify and apply the appropriate principle across mathematics, physics, chemistry, finance, and computing, rather than merely execute symbolic manipulation [170,171,296]. A second family emphasizes physics-centered quantitative reasoning, including UGPhysics, PHYSICS, and ABench-Physics, where benchmark design stresses formula selection, symbolic consistency, physical interpretation, and stronger leakage control on advanced undergraduate problems [293,294,295]. A third family extends the object into multimodal and visually grounded settings, such as MathVista, STEM-POM, MathVerse-Plus, and OlympiadBench, where diagrams, spatial structure, or scientific visual context are integral to the reasoning process rather than peripheral presentation features [156,167,168,169].

Key Incomparability: Mathematical Reasoning

3.3. Knowledge Reasoning

Knowledge reasoning can be divided into two primary benchmark reasoning objects: Multi-hop Reasoning and Long Context Reasoning. Multi-hop reasoning focuses on composing several separated pieces of evidence into a coherent inference, while long context reasoning focuses on locating, retaining, and integrating relevant information across very large context windows.

3.3.1. Multi-hop Reasoning

Multi-hop reasoning benchmarks evaluate whether a system can connect several separated pieces of evidence and use them jointly to reach one answer or conclusion [172,173,174,179,180]. What makes this object distinct from single-passage question answering is that no single fact is sufficient on its own: the model must identify intermediate links and compose them across documents, modalities, or retrieval steps [173,174,179,180]. Accordingly, the core benchmark target is not knowledge access by itself, but evidence composition under conditions where the reasoning chain is frequently rendered observable, systematically verifiable, or explicitly safeguarded against single-hop shortcuts [174,177,178,179,180].

Several benchmark families fall under this object. Classic multi-document QA benchmarks require models to connect evidence distributed across multiple sources and, in many cases, recover an implicit or annotated reasoning path [174,177,178,179,180]. Other benchmarks place more weight on decomposition, explanation, or fact aggregation. StrategyQA, QAMPARI, WikiWhy, and MINTQA are representative here, since answering them depends less on extracting one decisive span than on assembling several supporting statements into a coherent inference [56,172,175]. The same benchmark object also extends beyond plain text: MultiModalQA, FanOutQA, MM-BRIGHT, and ChartQAPro distribute relevant evidence across tables, images, charts, webpages, or other modalities, so successful reasoning requires cross-modal evidence integration rather than textual chaining alone [173,190,191]. More recent benchmarks further combine multi-hop reasoning with retrieval-intensive or agentic settings, including Bamboogle, BrowseComp, and FinReflectKG-MultiHop [33,72,181,182,183,185,186,188,189].

3.3.2. Long Context Reasoning

Long context reasoning is a benchmark object whose target capability is maintaining and integrating relevant information when the evidence is distributed across very large context windows. Its defining benchmark demand is the ability to perform scale-sensitive reasoning under memory pressure, not simply the existence of multiple discrete facts. The object is instantiated in benchmarks where accuracy depends on locating, retaining, and combining information as context length grows, often under conditions where retrieval by superficial salience is unreliable [44,45,71,79,297,298,299].

Realistic multitask suites such as LongBench and LongBench v2 benchmark this object by collecting summarization, question answering, retrieval, and reasoning settings that span long documents and multi document corpora [71,79]. Their contribution is to shift the benchmark target from isolated needle retrieval to mixed long context workloads with more natural task formats [71,79]. A second family exploits synthetic controllability to target and isolate specific long-context failure modes for fine-grained analysis [45,297]. RULER and LongReason vary context length, distractor structure, and reasoning pattern so that one can separate failures of memory, localization, and aggregation [45,297]. A third family stresses extreme length or uniform evidence relevance [44,298]. InfiniteBench pushes toward very large contexts, while Loong constructs settings where many documents matter and no single easy retrieval shortcut suffices [44,220,298].

Key Incomparability: Knowledge Reasoning

3.4. Agentic Reasoning

Agentic reasoning can be divided into four benchmark objects: Planning and Decision-making, Tool Use, Code Reasoning, and Multi-Agent Collaboration. Planning and decision-making focuses on sequential action selection in evolving environments, tool use focuses on the correct selection and invocation of external interfaces, code reasoning focuses on reasoning over program semantics and executable correctness, and multi-agent collaboration focuses on coordinated problem solving across multiple agents.

3.4.1. Planning and Decision-making

Planning and decision-making benchmarks evaluate the ability to select actions over time in an evolving environment rather than solve a static input–output problem [78,92,222,300]. Their target is sequential competence under feedback, partial observability, and state changes induced by previous actions, so benchmark success depends on trajectory quality rather than local response correctness alone [47,78,92,222,244,300].

One benchmark family uses interactive text environments [92,300]. ALFWorld operationalizes this capability through long-horizon household tasks requiring navigation, subgoal ordering, and object-state tracking [300], while ScienceWorld extends the target to scientific experimentation, where agents must plan action sequences that reveal, manipulate, and reason about environmental state to complete a final objective [92].

A second family evaluates the same capability in realistic digital environments. WebArena, Mind2Web, Mind2Web 2, and related benchmarks target long-horizon web interaction, where success requires action sequencing, interface interpretation, and recovery from earlier mistakes under real website constraints [33,209,222,223,224,227,228]. A parallel line extends sequential decision-making to shopping, operating systems, mobile apps, and heterogeneous digital tasks, including WebShop, OSWorld, WindowsAgentArena, and others [229,237,244,245,246,247,248]. AgentBench broadens the scope further, shifting the evaluation object from domain-specific success to the generality of sequential decision-making competence across multiple environments [78]. A growing body of work also instantiates planning and decision-making in specialized domains [31,234,236,238,241,242].

3.4.2. Tool Use

Tool use is a benchmark object whose target capability is deciding when external tools are needed, selecting the appropriate interface, and producing valid calls that advance the task state. What makes it a distinct benchmark object is that correctness is grounded in executable interaction with APIs, functions, databases, or simulators, rather than in free form language alone [46,249,256]. The benchmark target is often compositional because a system must choose tools, supply arguments, interpret outputs, and sometimes abstain when no tool should be called [47,257,301].

Runnable API benchmarks form one core family [249]. API-Bank frames tasks as dialogues with executable APIs, so the benchmark directly measures whether tool invocation resolves the user request under realistic interface constraints [249]. A second family scales the interface space [46,256]. ToolBench exposed models to large collections of real APIs, and StableToolBench refined that design by stabilizing the underlying execution environment so that benchmark results depend less on external service volatility [46,256]. Another family focuses on tool awareness and function calling [257,301]. MetaTool benchmarks whether the model knows when to use a tool and which one to choose, while BFCL benchmarks structured function calling, including multi turn and multilingual settings where correctness can be checked against argument structure [257,301]. $\tau$-bench adds stateful conversations with domain APIs and policy constraints, making the benchmark target reliable tool mediated task completion [47].

3.4.3. Code Reasoning

Code reasoning is a benchmark object whose target capability lies in understanding, generating, executing, or modifying programs in ways that are faithful to program semantics and intended behavior [80,261,271,272,302,303,304]. It is distinct from generic tool use because code is not merely an external aid, but the object of reasoning itself, with correctness typically grounded in test execution, runtime behavior, or repository level constraints [80,261,271,272,302,304]. The benchmark target therefore concerns semantic alignment between specification, program, and observed execution [80,261,271,272,302,304]. Function level synthesis benchmarks are the most established family [261,272]. HumanEval and MBPP instantiate the object by pairing natural language specifications with hidden unit tests, so benchmark success depends on producing code that generalizes beyond the visible prompt and passes executable correctness checks [261,272]. A second family targets code understanding rather than synthesis alone [302]. CRUXEval uses input and output prediction style tasks to benchmark whether the system can reason about program behavior, edge cases, and execution traces [302]. A third family addresses contamination and temporal freshness [271]. LiveCodeBench continuously updates coding tasks so that benchmark results better reflect current reasoning ability on previously unseen problems [271]. Finally, repository scale benchmarks such as SWE-bench, SWE-bench Verified, and SWE-bench-Live move the object from isolated functions to real software maintenance, where success requires navigating codebases, editing multiple files, and satisfying regression tests tied to authentic issue reports [80,280,281,303,304].

3.4.4. Multi-Agent Collaboration

Multi-agent collaboration is a benchmark object whose target capability lies in solving tasks through structured coordination and communication among multiple reasoning entities. It is distinct from single agent planning because benchmark success depends not only on local decision quality, but also on communication, role allocation, information sharing, and adaptation to the actions of teammates. The object is defined by benchmarks where no single agent view or policy is sufficient, either because information is distributed, workloads must be partitioned, or synchronized action is required [288,305,306,307].

Embodied or simulated team environments form one important family [305,306,308]. VillagerBench instantiates the object through collaborative tasks in a Minecraft like world, where agents must divide labor, coordinate temporally, and react to changing task dependencies [305]. Collab-Overcooked provides a complementary family centered on tightly coupled coordination with natural language communication, which makes benchmark success sensitive to both action timing and communication usefulness [306]. A second family benchmarks asymmetric information and communication explicitly. COMMA uses multimodal puzzles in which agents hold different pieces of evidence, so the benchmark target becomes the quality of inter agent message passing and joint inference [307]. A third family broadens coverage across settings [288]. MultiAgentBench benchmarks collaboration and, in certain settings, competition across a wide range of scenarios, positioning the benchmark less as a test of performance within a single environment and more as an assessment of broad collaborative competence.

Another case is with social interaction benchmarks that measure persuasion, persona consistency, or conversational realism without requiring joint task completion. Those settings are adjacent, but they do not define this object unless successful benchmarking depends on coordinated problem solving across agents. Multi-agent collaboration is therefore best understood as a benchmark object for collective reasoning under communication and coordination constraints [288,307,309].

Key Incomparability: Agentic Reasoning

4. Setting (How): Construction of Reasoning Benchmarks

A reasoning benchmark does not evaluate reasoning in the abstract. What it measures depends both on how benchmark instances are constructed and on how the evaluation protocol is assessed. We therefore organize benchmark construction along two complementary dimensions. The first is Data Provenance, which concerns where benchmark instances come from and how they are collected, synthesized, or curated. The second is Protocol Formulation, which specifies the setting under which reasoning is evaluated. Together, these dimensions clarify what evidence is available, what actions are permitted, what outputs are expected, and what counts as success, making benchmark comparisons more faithful and diagnostically meaningful.

4.1. Data Provenance

Focusing specifically on this dimension, we systematically decompose Data Provenance into two distinct aspects, as illustrated in Figure 4: Naturalistic Data and Constructed Data.

4.1.1. Naturalistic Data

Naturalistic data is harvested from real-world sources and categorized into Real-World-Derived Benchmarks and Interaction-Derived Benchmarks. Once collected and refined, these datasets serve as the primary foundation for benchmark construction.

4.1.1.1. Real-World-Derived Benchmarks.

A first line of work constructs reasoning benchmarks by mining naturally occurring artifacts originally created for human use rather than for model testing. Early examples repurpose standardized examination materials: ReClor converts graduate-level logical reading questions into benchmark instances [134], while MATH draws on competition mathematics to yield challenging step-based reasoning problems [75]. NaturalProofs extends this approach to formal knowledge resources, organizing theorem statements and proofs in natural mathematical language into benchmarkable reasoning units [82].

This paradigm later moves beyond isolated questions to richer and more structurally complex real-world artifacts. SCROLLS demonstrates that naturally long documents spanning multiple domains can be standardized into long-context reasoning tasks without artificial context extension [192]. SWE-bench shows how GitHub issue reports, pull requests, and repository snapshots can be converted into executable software-reasoning instances [80], and SWE-Bench Pro continues this trajectory by curating longer-horizon engineering problems from actively maintained repositories [213]. LiveBench pushes toward freshness by continuously sourcing problems from newly released competitions, papers, and news [310], while LongBench v2 broadens the naturalistic scope by packaging documents, dialogue histories, repositories, and structured data into realistic long-context tasks [71]. Recent work further specializes naturalistic corpora for emerging reasoning needs: DeepScholar-Bench derives related-work synthesis tasks from recent arXiv papers [311], TEMPO organizes temporally evolving evidence into cross-period retrieval tasks [189], and XCR-Bench turns culturally grounded parallel corpora into benchmark instances [312].

4.1.1.2. Interaction-Derived Benchmarks.

A second line of work constructs benchmarks from realistic interaction data, where reasoning is embedded in sequential decision-making. WebShop couples a large real-product catalog with crowdsourced instructions and demonstrations, transforming shopping interactions into benchmark tasks that require grounded multi-step reasoning [229]. Mind2Web advances this by collecting action sequences on real websites, preserving authentic page structure, user intent, and action dependencies within each benchmark instance [209]. WebLINX further scales interaction harvesting to multi-turn conversational web navigation with expert demonstrations, making dialogue history itself part of the benchmark state [313].

More recent work compiles interaction data into executable environments rather than static trajectories. OSWorld instantiates open-ended computer tasks over real operating systems and applications, equipping each benchmark instance with an initial machine state and an execution-based verifier [244]. MCP-Bench extends this construction to tool-use settings by building tasks on live MCP servers, where agents must coordinate tool discovery, parameter grounding, and multi-step execution [314]. MCP-Atlas scales the design to a broader collection of real MCP servers and cross-tool workflows, illustrating a broader shift in interaction-derived benchmarks toward executable state, realistic affordances, and workflow-level dependencies [254].

4.1.2. Constructed Data

Conversely, constructed data is synthesized by either humans or AI models and is further partitioned into Expert-Curated Benchmarks, Model-Generated Data and Human–AI Collaborative Benchmarks. These datasets are typically engineered to evaluate specific reasoning capabilities or to address the inherent limitations of naturalistic data, offering a complementary foundation for benchmark development.

4.1.2.1. Expert-Curated Benchmarks.

Expert-curated benchmarks emphasize deliberate problem authoring, difficulty control, and contamination resistance during dataset construction. One prominent line of work builds frontier-level reasoning benchmarks by commissioning domain specialists to write original questions that are difficult to solve through memorization or shallow retrieval, as exemplified by GPQA, FrontierMath, and Humanity’s Last Exam [34,59,161]. Another line focuses on structural rigor by coupling natural-language instances with formal representations or process-level annotations. For example, FOLIO grounds deductive reasoning examples in first-order logic, while ProcessBench operationalizes reasoning-quality assessment through expert-annotated error locations in step-by-step mathematical solutions [28,41]. In these benchmarks, the central contribution is not merely a test set, but a carefully designed construction protocol that encodes the targeted reasoning skill into the data itself.

4.1.2.2. Model-Generated Benchmarks.

A complementary paradigm uses LLMs or programmatic generators to synthesize benchmark instances at scale. Rather than manually authoring every example, these benchmarks define controllable generation procedures and then rely on automatic verification, rejection sampling, or post-filtering to preserve quality. AutoLogi, for instance, transforms logic reasoning evaluation from multiple-choice questions into open-ended logic puzzles with controllable difficulty [99]. Similarly, MPBench constructs multimodal reasoning data for process-error identification, extending constructed benchmarks from answer-level supervision to step-aware and process-aware assessment [315]. Such model-generated benchmarks are especially useful when the goal is to cover large combinatorial spaces, create adversarial variants, or attach fine-grained intermediate supervision that would be prohibitively expensive to obtain fully by hand.

4.1.2.3. Human–AI Collaborative Benchmarks.

Between fully manual and fully synthetic construction lies a hybrid paradigm in which models draft, translate, perturb, or filter candidate items and humans subsequently validate them. This design has become increasingly common in recent reasoning benchmarks. MMLU-ProX uses LLM-assisted translation followed by expert review to build parallel multilingual reasoning questions, enabling controlled cross-lingual comparison without sacrificing conceptual fidelity [316]. SuperGPQA combines large-scale expert question writing with human–LLM collaborative filtering to eliminate trivial or ambiguous items across a broad range of graduate-level disciplines [317]. DocPuzzle similarly adopts an annotation-validation loop to construct realistic long-context reasoning problems with explicit attention to process complexity [206]. Overall, recent benchmark construction has shifted from static answer-only datasets toward data pipelines that explicitly encode frontier difficulty, process supervision, multilingual transfer, and scalable but human-grounded quality control.

4.2. Protocol Formulation

Focusing specifically on this latter dimension, we systematically decompose Protocol Formulation into four distinct layers, as illustrated in Figure 5: Task Interface, Knowledge Access, Tooling and Environment, and Constraints and Controls.

4.2.1. Task Interface

4.2.1.1. Answer Format

Answer format defines the set of outputs a benchmark accepts as valid. It determines how tightly models are supervised, how ambiguous scoring can be, and whether evaluation primarily rewards reasoning alone or reasoning plus faithful realization. We analyse this dimension along three common formats: fixed-choice, free-form, and structured outputs.

Fixed-choice formats constrain outputs to a closed candidate set, improving reproducibility and scoring stability by minimizing surface-form variance. This design forms the basis of many multiple-choice benchmarks, from RiddleSense [318] and MedMCQA [319] to more recent datasets such as MME-RealWorld [320], SATBench [321], and CogToM [322]. Some benchmarks deliberately mix fixed-choice with other formats for broader diagnosis: LogicBench combines binary QA and multiple choice [50], SceMQA mixes multiple-choice and free-response questions [323], and OlympiadBench includes standardized open-ended answers alongside proof problems [156].

Free-form formats allow unconstrained textual answers and are better suited to end-to-end problem solving, but they require normalization, objective references, or judge-based evaluation. Short-answer settings emphasize deterministic checking, as in GSM8K [74], SVAMP [144], and objectively scored tasks in LiveBench [310]. Other benchmarks rely on richer open-ended generation or preference judgments, including AlpacaEval-LC [324], U-MATH [325], and TYPED-RAG [326].

Structured formats require schema-compliant or executable outputs, enabling verification beyond string matching. Formal reasoning benchmarks such as miniF2F [102] and miniF2F-v2s [327] treat valid outputs as formal statements or proofs. This requirement is also common in tool-use benchmarks, including API-Bank [249], ToolBench [256], and TOOLSANDBOX [53], which require structured API calls, typed arguments, or replayable action traces. Software engineering benchmarks push this logic further by evaluating executable patches against repository-level tests, as in SWE-bench [80] and SWE-Bench Pro [213].

4.2.1.2. Interaction Mode

Interaction mode specifies whether a benchmark evaluates a single response under a fixed input or an interactive policy that must act over multiple steps with feedback. It therefore distinguishes one-shot inference from temporally extended behavior. We analyse this dimension along two common modes: single-turn benchmarks and multi-turn benchmarks.

Single-turn benchmarks provide the full context upfront and score a single final response, making them well suited to controlled comparison of reasoning, knowledge use, long-context understanding, and structured generation. Representative examples include GPQA [34], LongBench [79], and MMLU-Pro [68]. Multi-turn benchmarks instead evaluate policies that repeatedly observe, act, and revise before termination. They are necessary when success depends on exploration, tool use, social interaction, or recovery from intermediate errors over long horizons, as in benchmarks such as WebShop [229], WebArena [222], Mind2Web 2 [228], and TERMINAL-BENCH [328].

4.2.2. Knowledge Access

Knowledge access defines the extent to which a model can retrieve, ground, and reason over information that may lie beyond its parametric knowledge. Benchmarks in this space are organized along a single axis: whether external retrieval is permitted at inference time. Rather than treating knowledge as a fixed property of the model, this dimension asks how reliably a model can locate relevant evidence, whether stored internally or sourced externally, and integrate it into coherent, accurate responses. We analyze this dimension along two common protocols: closed-book protocol and open-book protocol.

4.2.2.1. Closed-book Protocol

Closed-book Protocol forbids external retrieval at inference time, so the model must answer from parametric knowledge and the information already contained in the prompt. This setting isolates internal knowledge and reasoning from retrieval ability [67,68,146].

A useful variant is given-context closed-book evaluation, where all supporting evidence is embedded in a long input rather than retrieved externally. This setting tests evidence localization, aggregation, and long-range reasoning under fixed inputs. Representative benchmarks include L-EVAL [329], ∞BENCH [44], RULER [45], MMLongBench-Doc [330], NeedleBench [200], and LongDocURL [221], along with other relevant evaluation settings [79,202,203,322].

4.2.2.2. Open-book Protocol

Open-book Protocol allows access to external evidence at inference time, so performance depends on evidence acquisition, selection, grounding, and temporal freshness in addition to reasoning. One major branch of existing work relies on fixed corpora and standardized retrieval pipelines, encompassing curated benchmark datasets, comprehensive evaluation suites, and well-defined document-level RAG settings such as MultiHop-RAG [185], RAGAS [30], ChatQA [331], RAGBench [332], and MMRAG-DocQA [333].

Another line of research focuses on evaluating live or continuously updated knowledge access, with particular emphasis on information freshness, web browsing capabilities, citation accuracy, and the ability to adapt to real-time changes. Representative examples include LiveBench [310], Chatbot Arena [334], and FINDEEPFORECAST [335].

4.2.3. Tooling and Environment

Tooling and environment evaluate a model’s capacity to act beyond text generation by interacting with external systems, executing operations, and adapting to feedback from a changing world. We analyze this dimension along two common settings: tool invocation and interactive environment.

4.2.3.1. Tool Invocation

Tool invocation evaluates whether a model can decide when to call tools, select appropriate APIs, and incorporate returned observations into subsequent reasoning. This line of work spans early tool-use paradigms and training setups such as Toolformer [255], API-Bank [249] and ReAct [13], as well as dedicated evaluation suites such as ToolBench [256], ComplexFuncBench [336], and BFCL [257]. Recent benchmarks go beyond isolated function calls, focusing on multi-tool orchestration, user–agent interaction, and coordination, as illustrated by WebArena, AgentBench, and $M^3$-Bench [222,314,337,338,339].

Single-tool settings isolate call or no-call decisions, tool retrieval or selection, and argument grounding under relatively fixed schemas, making failure modes such as malformed arguments, irrelevant calls, and schema violations easier to diagnose [187,249,257,301,336]. Multi-tool settings evaluate routing across heterogeneous tools, intermediate-state tracking, and recovery over longer workflows or conversations, where success depends on correct sequencing, dependency management, and robust use of tool outputs [288,338,339].

4.2.3.2. Interactive Environment

Interactive environments place reasoning in a stateful world whose state evolves after each action. They span execution-grounded coding tasks, web and GUI interaction, OS-level control, terminal use, and user-interactive settings, enabling evaluation of long-horizon control, recovery, and environment-grounded verification [248,272,328,340].

Code execution environments let models write or modify code and receive runtime signals such as compiler feedback, unit-test results, repository-level regression outcomes, or execution traces. They are well suited for evaluating iterative debugging and execution-grounded reasoning [80,271,272,273,274,279,341]. While GUI-based environments require perception and action grounding over webpages, screenshots, desktop or mobile interfaces, terminals, and evolving user interactions. They therefore test whether language-level plans can be reliably translated into interface actions under partial observability and procedural constraints [72,209,222,229,248,328,342].

4.2.4. Constraints and Controls

Constraints and controls examine whether evaluation outcomes remain meaningful when the conditions of inference are made more realistic or more adversarial. A model that performs well under unconstrained, clean-input settings may degrade substantially when resources are capped, inputs are perturbed, or test items have leaked into pretraining, yet standard benchmarks rarely make these failures visible. We analyze this dimension along three common settings: budget limits, robustness controls, and protocol formulations.

4.2.4.1. Budget Limits

Budget limits impose explicit caps on actions, tool use, compute, or reasoning length, forcing models to allocate exploration, verification, and recovery under fixed resources. They make evaluation more deployment-relevant by separating genuine decision quality from gains that come purely from unconstrained computation. In practice, such controls appear as step limits, tool-call limits, cost-aware planning objectives, and token-efficiency mechanisms in recent evaluation suites and methods, including HELM [343], LiveCodeBench [271] and Dynamic Thinking-Token Selection [344].

4.2.4.2. Robustness Controls

Robustness controls stress models under controlled variation and guard evaluation against leakage or stale knowledge, making failures attributable to specific instability modes rather than to underspecified test conditions. Rather than treating accuracy as a single static quantity, they make failure modes more diagnostic by tying errors to identifiable forms of instability. One common strategy is perturbation-based stress testing, where semantics-preserving changes such as paraphrases, distractors, rewrites, or format shifts are introduced to check whether predictions remain stable [345,346]. Another strategy emphasizes contamination resistance and temporal validity through benchmark design choices such as rolling refresh, temporal cutoffs, and controlled answer exposure [59,271,310].

5. Evaluation (How Well): Assessment of Reasoning Benchmarks

Evaluation determines what a benchmark rewards, what it ignores, and how its scores should be interpreted. A single score can report progress while hiding whether gains come from better reasoning, easier outputs, looser matching rules, or heavier compute. As shown in Figure 6 and Figure 7, to make metric choices explicit, we structure assessment into two components: Evaluation Unit and Evaluation Dimensions.

These dimensions jointly depict not only raw performance, but also the credibility, robustness, and real-world deployability of reasoning systems [41,78,343,347,348].

5.1. Evaluation Unit

Evaluation Unit specifies which object is scored, ranging from a final answer to an interaction trace. Thus, we categorize evaluation units into three levels of granularity: answer-level, which focuses on the final output; process-level, which examines intermediate reasoning artifacts; and trajectory-level, which considers the full decision trace in interactive settings.

5.1.1. Answer-level

Answer-level evaluation focuses solely on the final output, treating the model as a mapping from input to answer. It remains the most widely used evaluation unit because it scales easily across large benchmark suites, supports straightforward aggregation, and facilitates simple leaderboard comparisons. This form of evaluation works best when benchmarks specify a clear target and a stable matching criterion, such as a unique option, a normalized short answer, or a numeric value with an allowed tolerance. Its limitation, however, is equally clear: final-answer correctness alone reveals little about how the result was obtained. A correct answer may reflect genuine reasoning or merely exploitation of superficial cues. For that reason, benchmark construction at this level often emphasizes tighter task design and stronger control of spurious shortcuts.

In practice, answer-level scoring appears in several recurring benchmark formats. Broad multi-task suites unify heterogeneous tasks under standardized interfaces and summarize model performance with aggregate scores, making cross-model comparison easy and reproducible [67,70,310,343]. Reasoning-oriented benchmarks, by contrast, rely on expert-written or exam-style problems with uniquely verifiable answers, preserving the usefulness of final-answer accuracy even as difficulty rises [34,69,74,75]. Code benchmarks take a different route: they regard the generated program itself as the answer and judge it through execution, tests, or functional checks, thereby maintaining objectivity despite wide variation in surface form [261,271,272,349]. Some recent benchmarks further strengthen answer-level evaluation by refreshing instances over time and combining them with automated verification, which helps leaderboards remain informative even under rapid model iteration [69,271,304,310].

5.1.2. Process-level

Process-level evaluation examines the intermediate reasoning artifacts produced along the way, including step sequences, intermediate variables, proofs, subclaims, and structured rationales. Rather than treating reasoning as valuable only when it leads to the right endpoint, this level evaluates the quality of the process itself. Its advantage is most evident when intermediate steps can be checked automatically through symbolic validation, constraint checking, proof checking, or execution-based verification.

Another strength is diagnostic precision: instead of merely observing that a final answer is wrong, process-level scoring can identify where the reasoning first breaks down and whether the failure arises from conceptual misunderstanding or faulty rule application.

Benchmarks in this category generally follow two main directions. One line of work makes intermediate reasoning explicitly verifiable by representing it as objects such as entailment trees, natural-language proofs, or machine-checkable proof states; correctness can then be assessed incrementally rather than only at the endpoint [83,84,85,102]. Another line of research emphasizes diagnostic evaluation, providing step-level annotations that identify the earliest erroneous step or categorize the specific type of reasoning error within a trace [41,48,49,350].

5.1.3. Trajectory-level

Trajectory-level evaluation considers the entire decision trace in tasks that unfold over time, including actions, tool calls, observations, intermediate states, and termination choices. Here the system is evaluated not simply as an answer generator but as a policy operating through a sequence of decisions. This perspective is especially important in agentic and interactive settings, where identical final answers may arise from trajectories that differ sharply in efficiency, risk, and robustness.

Benchmarks at the trajectory level usually place the model inside an interactive environment and assess the resulting trace from multiple angles. Some emphasize long-horizon agent behavior, focusing on coherence and recovery under observational feedback while reporting both task success and trajectory-sensitive diagnostics [78,222,229,351]. Others center on tool use or API interaction, where actions are grounded in executable calls and structured state transitions, making it possible to evaluate traces through database state, argument correctness, and rule compliance [46,47,72,249]. In software engineering benchmarks, trajectories often consist of iterative code edits coupled with test feedback, which allows evaluation to penalize wasted iterations while rewarding fixes that remain robust beyond a single patch [46,78,80,304].

5.2. Evaluation Dimensions

Evaluation dimensions determine which aspects of performance a benchmark prioritizes and rewards, as well as how the resulting scores are to be interpreted. In this survey, we organized evaluation dimensions into three pillars: correctness, reliability, and efficiency.

5.2.1. Correctness

Correctness evaluates whether model outputs satisfy the success criteria defined by a task, typically operationalized as either Final-answer Accuracy or Step-level Scoring.

5.2.1.1. Final-answer Accuracy

As for final-answer accuracy, evaluation checks whether the output matches the benchmark target under a predefined matching rule, such as exact match, normalized match, option selection, tolerance-based numeric match, or test-based acceptance. These rules are typically implemented through reusable evaluation templates. In exam-style and knowledge-intensive benchmarks, correctness is usually defined as selecting the keyed option under fixed decoding, as in MMLU, C-Eval, GPQA, and AGIEval [34,67,73,352]. Other difficult reasoning benchmarks retain the same answer-level notion of success while increasing task difficulty [96]. In free-form reasoning and math tasks, evaluation often extracts a short final answer and applies exact or normalized matching [70,343]. Some math benchmarks further strengthen this rule with symbolic equivalence or numeric tolerance in order to reduce sensitivity to superficial formatting differences while preserving strict correctness [74,75,96]. For executable tasks, correctness is instead determined by an external verifier such as unit tests or sandbox execution, turning natural-language or code outputs into behavior-level acceptance [75,261,271,272,353]. Under this final-answer view, acc reports the fraction of instances for which the top output is correct. It is therefore the standard metric in deterministic settings where each instance has a unique gold target and decoding is fixed, including multiple-choice exams and short-answer math word-problem benchmarks [34,73,74,75,352]. By contrast, pass@k measures whether at least one of the top k sampled candidates is correct. This metric is more appropriate in settings where stochastic sampling constitutes an integral part of the intended inference procedure and correctness is adjudicated by an external verifier, especially in functional-correctness evaluation for code synthesis and competition-style programming [41,52,75,261,271,272,353,354].

5.2.1.2. Step-level Scoring

As for step-level scoring, the correctness is assessed at the process level rather than at the endpoint. Recent process-oriented benchmarks introduce intermediate supervision or evaluation hooks by labeling step validity or requiring explicit step-wise outputs [41,52,354]. Such setups make it possible to evaluate critics and process-aware reward models directly, instead of inferring process quality only from final answers [102,127]. In multi-step mathematics, intermediate correctness concerns whether each step satisfies arithmetic and reasoning constraints, and benchmarks often operationalize this by asking models or verifiers to identify the earliest incorrect step or assign validity labels to each step [41,52,127,354]. This form of scoring supports finer-grained error diagnosis and objectives related to early detection of solution failure [74]. In deduction and proof, step validity can be grounded in machine-checkable proof obligations and tactic traces, so intermediate scoring reduces to whether each step preserves formal validity under a proof assistant or logic checker [28,102,113,290,355]. A related but distinct aspect is calibration, which asks whether the model assigns reliable confidence to answers or intermediate decisions. This matters in selective answering, abstention, and risk-aware deployment, where uncertainty quality is important alongside correctness itself. Calibration-oriented benchmarks therefore examine whether confidence aligns with empirical correctness and whether models can appropriately abstain or hedge under uncertainty, using reliability metrics and selective risk–coverage analyses in both short-form and long-form settings [343,356,357,358,359].

5.2.2. Reliability

Reliability concerns whether the reasoning is confident, consistent, and robust under varying conditions. [343,356,357,358,359]. To better depict this evaluation dimension, we categorize reliability into three aspects: evidence and verification, safety and privacy, and robustness.

5.2.2.1. Evidence and Verification

This concerns two closely related issues: whether a claimed solution is backed by inspectable support, and whether the benchmark can validate that support in a reliable way [181,360,361,383,384]. To make reliability operational, many benchmarks require support to be explicit and machine-checkable [343,360,361,383]. One line of work studies attribution and verifiable generation: models must answer with citations, and evaluation verifies whether the cited spans fully support, only partially support, or even contradict the associated claims [360,361,364,383]. Another line focuses on fact verification with evidence, scoring both the correctness of the verdict and the adequacy of the selected support, including dialogue-grounded checking and multimodal evidence verification [362,385,386,387]. Time-sensitive benchmarks add a further constraint: supporting evidence must be correct for the reference date, which helps expose stale factual recall and evidence drift in retrieval- or browsing-based systems [181,388]. Executable trajectory evaluation shifts the focus from stated answers to whether the produced artifact can actually run and be reproduced, including code, tool calls, intermediate computations, and environment actions; this perspective is central to tool-use and agent benchmarks [80,222,249,272,363]. Some benchmarks execute generated code or patches in sandboxes and score unit-test outcomes, emphasizing reproducibility, determinism, and failure localization in realistic repositories [80,261,272,302,389].

Hallucination evaluation asks whether generated content remains faithful to provided sources or other trusted references, a concern that is especially salient in open-book, retrieval-based, and long-context settings where fluent but unsupported content is a common failure mode [360,361,364,365,390]. Some benchmarks probe truthfulness under misconception pressure by using questions that invite plausible but false answers and rewarding systems that avoid common falsehoods and misleading continuations [181,364]. Others construct hallucination-detection datasets with human annotations of unsupported or contradictory generations, then test whether systems can avoid or identify such content across diverse prompts and contexts [365,383]. A further strand decomposes model outputs into atomic claims and verifies each claim against trusted references, enabling claim-level auditing rather than relying only on final-answer accuracy [360,361,383].

5.2.2.2. Safety and Privacy

Safety and privacy assess whether a system remains appropriate, secure, and non-disclosive when confronted with risky prompts, sensitive contexts, or high-impact actions.[363,366,367,372]. Recent benchmarks focus on risk-aware behavior under adversarial prompting and realistic deployment conditions, often using runnable harnesses and standardized taxonomies to distinguish justified refusal, unsafe compliance, and tool-mediated downstream harm [343,363,366,367].

Safety evaluation asks whether a model avoids harmful behavior, adheres to safety policies, and handles tools responsibly when actions can create downstream consequences, especially in interactive or agentic settings[363,366,367,368,369]. Some benchmarks center on policy knowledge and compliance across categorized scenarios, measuring safe completions, refusal quality, and calibration across languages and domains [366,367]. Others probe adversarial misuse and jailbreak robustness with curated harmful goals and attack prompts, tracking harmful completion rates, refusal robustness, and over-refusal on benign inputs [368,369,391,392]. Agent-oriented evaluations instead place models in tool-using environments and test whether they recognize high-impact operations, avoid unsafe actions, and maintain safe execution over interactive trajectories [363,393].

Privacy evaluation asks whether a model leaks sensitive information, reproduces memorized private content, or reveals confidential context, making it especially relevant for private-document tasks, personal prompts, and contamination-sensitive setups[370,371,372,394]. One line of work studies memorization and training-data leakage by probing whether models can be induced to reproduce rare private strings or personally identifiable information through adaptive querying and prompt engineering [370]. Another examines prompt and secret exfiltration in application-like settings, asking whether hidden system prompts or confidential strings can be extracted through injection-style interactions that resemble deployed workflows [371,395]. Long-horizon conversational benchmarks, by contrast, simulate user-specific profiles and private memories to test selective disclosure, access control, and leakage over multi-turn interactions [372,394].

5.2.2.3. Robustness

Robustness asks whether a system maintains performance under intent-preserving variations and under distribution shifts that resemble real use [222,373,374,375,396]. Benchmarks usually instantiate this goal through controlled perturbations and shifted settings, then inspect invariance, degradation patterns, and failure modes to diagnose reliance on superficial cues or brittle heuristics [373,374,375,397]. Recent work broadens the scope to prompt injection, adversarial reasoning pressure, social interaction, and environment-level failure localization, especially in interactive settings [39,92,324,343,396,398,399,400,401,402,403,404].

Perturbation robustness tests invariance to paraphrases, formatting changes, reordered context, noise, and distractors, making it useful for exposing shortcut reliance and other brittle heuristics [373,374,375]. One line of work uses adversarial or dynamic data collection to generate meaning-preserving variants and pair them with diagnostics of what changed and why the model failed [373,374,375]. Another line builds structured perturbation suites, such as paraphrase, distractor, and format edits at scale, to support controlled ablations over wording, presentation, and spurious cues [181,365,375]. Interactive benchmarks extend this logic to trajectories: small changes in layout, action order, or tool outputs can alter the full interaction path, so robustness is evaluated beyond the final answer alone [78,209,222]. Cross-domain robustness measures whether performance transfers across domains, languages, modalities, and task templates, which is essential for benchmarks claiming broad reasoning rather than narrow adaptation to a single distribution [77,352,397,405]. Broad subject-transfer benchmarks probe stability across disciplines, difficulty levels, and professional areas while holding the evaluation protocol fixed [96,352,405].

5.2.3. Efficiency

Efficiency asks whether a system can achieve high-quality results while consuming minimal computational resources, latency, or external dependencies. [343,347,348,376,380]. We categorize efficiency evaluation into two aspects: cost–quality tradeoffs and budget-aware optimality.

5.2.3.1. Cost–Quality Tradeoffs

Cost quality tradeoffs measure how much compute, time, or external resource is required to reach a given level of correctness and reliability, making evaluation more meaningful for deployment under practical constraints [343,347,348,376,380]. Benchmarks usually operationalize this tradeoff in three ways: reporting quality alongside explicit cost signals, measuring latency and throughput, or energy under standardized serving settings, and tracing quality across token or query budgets [343,347,348,376,380].

Latency-score ties quality to end-to-end response time, establishing a direct relationship between performance and inference speed. This is especially relevant in interactive settings where long deliberation reduces usability [347,376,377,406]. Latency-oriented benchmarks therefore standardize measures such as time to first token and total completion time under streaming or batch settings, then relate them to task quality [347,376,377,382,406]. Tool-call-score links quality to external tool usage, such as the number of calls or total call cost, which is critical in agentic and open-book settings where retrieval and execution dominate expense [46,47,222,244,378]. Accordingly, tool-augmented benchmarks score task success together with tool-use efficiency; examples range from controlled API environments to realistic interactive systems with auxiliary tools [46,47,222,244,378].

5.2.3.2. Budget-aware Optimality

Budget-aware optimality evaluates whether a system achieves the best possible quality under explicit resource constraints rather than relying on unconstrained computation [47,348,380,381,406]. These benchmarks impose fixed token, step, or query budgets and evaluate systems under matched constraints, employing token-economy tests, controllable synthetic workloads, and agent tasks characterized by step caps and human reference trajectories.

Early stopping evaluation tests whether a system can terminate once it has gathered sufficient evidence, which is especially important in long-horizon tasks where extra steps increase both cost and error risk [222,379]. Typical protocols evaluate adaptive stopping in decoding or interaction, ending when agreement stabilizes or when further reasoning is unlikely to improve outcomes within the remaining budget.

Multi-objective evaluation measures performance across simultaneous goals such as correctness, reliability, and cost, making it well suited to agentic settings where gains on one objective may degrade another [343,347,377,382]. These benchmarks typically report a metric vector and may use Pareto-style selection over accuracy, reliability, latency, and energy, covering holistic evaluation suites, system benchmarks, and energy-centered tests.

6. Scenario-Based Extensions of Reasoning Benchmarks Analysis

This section conducts a scenario-based extension of reasoning benchmarks. As shown in Figure 8, rather than proposing a second analytic methodology beyond Object, Setting, and Evaluation, it organizes compact discussions around four recurrent deployment contexts that increasingly shape benchmark design in practice: Multilingual and Cultural Benchmarks, Multimodal Benchmarks, Vertical-Domain Benchmarks, and Agentic and Interactive Benchmarks. Here, scenario refers to deployment-oriented context, including language coverage, modality, work domain, and interaction environment. These extensions complement Section 3– Section 5 by reviewing recent advancements in these widely concerned scenarios, and demonstrating how the structure of these benchmarks can be interpreted within the framework introduced above.

6.1. Multilingual and Cultural Benchmarks

Along the language dimension, benchmark development has progressively expanded along two closely related yet analytically distinct directions: cultural benchmarks and multilingual benchmarks [407,408,409,410].

6.1.1. Cultural Benchmarks

Cultural benchmarks are grounded in local entities, institutions, conventions, and social norms, particularly in domains where linguistic form, background knowledge, and culturally situated reasoning are tightly coupled [352,408,410,411,412]. Early work was relatively sparse and domain-specific, demonstrating how evaluation could be adapted to specialized local settings [413,414]. This line later expanded through exam- and knowledge-centered benchmarks, initially concentrated in Chinese and other regional settings, and gradually broadened to more languages, regions, dialects, and low-resource contexts [55,67,415,416,417,418,419,420,421,422,423,424,425,426]. This expansion improved local realism and relevance, but it also made strict cross-lingual comparison harder, as benchmark content became increasingly tied to specific cultural, educational, and sociolinguistic contexts.

6.1.2. Multilingual Benchmarks

Multilingual benchmark construction initially remained largely translation-based, with design priorities centered on prompt matching, answer-space alignment, and item-level comparability across languages [64,427,428,429,430]. Although this improved cross-lingual symmetry, it also introduced translation artifacts, culturally flattened prompts, and continued dependence on English-centered task formulations [407,409]. Subsequent work moved beyond fully parallel designs to encompass mathematical and causal reasoning, shared-item resources, culturally grounded datasets, and expert-verified multilingual extensions [408,428,431,432,433,434,435]. From there, multilingual benchmarks further broadened toward locally grounded knowledge, culturally situated commonsense, cross-lingual alignment, and more interactive settings involving multi-turn interaction, retrieval, tool use, and agentic capabilities [401,409,410,436,437,438,439,440,441,442,443]. As a result, coverage expanded from language inclusion alone to language-conditioned task settings, though benchmark comparability became more fragile once full parallelism and difficulty control were harder to maintain across languages [316,407,410,443].

6.2. Multimodal Benchmarks

Multimodal benchmarks can be organized along two parallel lines: perception-grounded benchmarks that target broad coverage across diverse scenes, documents, and videos, and structure-grounded benchmarks that focus on inputs, tables, charts, diagrams, and forms, where relational structure is explicit and therefore more amenable to fine-grained verification [167,320,444,445,446,447].

6.2.1. Perception-Grounded Benchmarks

Perception-grounded benchmarks have gradually moved from relatively clean image–text inference settings toward document-heavy, high-resolution, and real-world scenarios, and more recently toward richer settings involving tool use, multi-step evidence access, long-context inputs, and interactive constraints [62,244,320,444,445,448,449,450]. The central evaluation object is evidence-bound inference after perceptual extraction, assessed through answer accuracy, grounding fidelity, robustness to perturbation, and calibration under uncertainty [320,450,451]. A key diagnostic challenge in this branch is separating failures of perception from failures of reasoning under explicit evidence conditions [450,452].

6.2.2. Structure-Grounded Benchmarks

Structure-grounded benchmarks follow a different logic: inputs are not only multimodal but also structurally organized in ways that expose relations, quantities, alignments, and dependencies, enabling constraint-aware interpretation and finer-grained verification [167,445,453]. The evaluation object shifts accordingly to structural interpretation coupled with reasoning over explicit constraints, and metrics often extend beyond final-answer correctness to include quantitative tolerance, format or unit consistency, and agreement with intermediate reasoning states where annotations permit [167,452,453]. This distinction between the two benchmark families is best understood as an analytic framework rather than a fixed taxonomy; in practice, benchmarks that claim to measure process-level competence should specify explicit process-supervision schemes so that intermediate correctness is observable and testable rather than left implicit [444,448,452].

6.3. Vertical-Domain Benchmarks

Vertical-domain benchmarks fall into two parallel lines: code reasoning benchmarks, which center on programming artifacts, execution environments, and repository-scale intervention, and scientific domain benchmarks, which center on scientific evidence, expert knowledge, multimodal observations, and constraint-sensitive reasoning. Across both lines, benchmark development has moved away from thin answer-only tasks toward richer evaluation objects, more explicit settings, and broader criteria [343,373,454].

6.3.1. Code Reasoning Benchmarks

Code reasoning benchmarks did not evolve simply by making programming questions harder. Early work, including HumanEval, MBPP, and APPS, evaluated function synthesis and short-form code generation; subsequent benchmarks such as MultiPL-E, MBXP, ReCode, and ARCADE broadened language coverage, execution conditions, and task formats without fully leaving snippet-scale evaluation behind [75,261,272,455,456,457,458,459,460].

The evaluated object grew richer once coding was embedded in more specialized and structured contexts. BioCoder and DS-1000 introduced biomedical and data-science programming, while RepoBench, ClassEval, and CloudEval-YAML brought in repository evidence, class-level structure, and configuration-sensitive tasks, making evaluation less reducible to isolated snippets [24,262,270,461,462,463,464]. Later benchmarks sharpened this further by emphasizing verified development, efficiency, repository-centered repair, and live development workflows [24,80,282,302,304,465,466,467,468,469,470,471,472,473,474,475]. Terminal-Bench and SkillsBench extend evaluation to longer tool-mediated trajectories, while RealSec-Bench, ContractEval, and SolContractEval show that in security-sensitive and contract-specific settings, ordinary completion metrics are no longer sufficient [328,476,477,478,479].

6.3.2. Scientific Domain Benchmarks

Scientific domain benchmarks followed a parallel but distinct trajectory. Early benchmarks, including DisKnE, MedQA, GeoQA, MedMCQA, and others, framed scientific competence primarily as question answering or short-form expert response, expanding topical coverage across subdomains, languages, and regional contexts while keeping the dominant format close to expert QA [66,150,319,480,481,482,483].

Subsequent benchmarks introduced stronger reasoning demands and richer evidential structure. SCIBENCH, GPQA, and TheoremQA raised the difficulty bar through scientific problem solving, graduate-level expert questioning, and theorem-centered reasoning, while MultiMedQA, TRIGO, and ChemLLMBench widened the evidential base through multi-source medical evidence, mathematically structured tasks, and chemistry-specific judgment [34,54,170,171,484,485]. More recent work diversified the field further, extending evaluation to medically grounded visual evidence, safety-sensitive chemical reasoning, materials science, longitudinal health contexts, and figure-grounded scientific understanding [68,156,157,486,487,488,489,490,491,492,493,494,495,496,497]. Across these developments, the central shift is that domain benchmarks no longer test subject knowledge alone, but increasingly ask whether reasoning remains valid under domain-specific evidence, constraints, risks, and verification standards.

6.4. Agentic and Interactive Benchmarks

Agentic and interactive benchmarks mark a shift from answer-only evaluation toward explicit action trajectories, where benchmark conclusions depend on how interaction is grounded, what external operations are permitted, and how multi-step behavior is assessed. This subsection covers two closely related lines: grounded UI interaction, which requires models to perceive and act within changing interfaces, and tool use with multi-step planning, which requires models to select, invoke, and sequence external operations toward a longer-horizon goal.

6.4.1. Grounded UI Interaction

Grounded UI interaction benchmarks require models not only to follow a fixed tool schema, but also to perceive a changing interface, connect visible elements to the user’s goal, and act through clicking, typing, scrolling, or navigation. Early environments such as WebShop made interface actions central to task completion; subsequent benchmarks extended this toward richer interfaces, longer trajectories, and more realistic web and mobile interaction traces [209,222,229,313,498]. This line later broadened toward open-web interaction, stronger visual grounding, live environments, and workflow-shaped objectives, where success increasingly depends on staying grounded across multiple interface states rather than solving a single isolated step [223,224,226,228,470,499,500,501,502].

A parallel branch extends grounded interaction beyond the browser to desktop, app, and mobile environments with longer horizons, evolving state, and stronger memory demands [237,244,248,340,503,504,505]. Further benchmarks widen the space through more heterogeneous interface settings and evaluation goals [247,506,507,508,509]. Across these settings, the object is no longer a single correct output but grounded action under a perceptual state, making interface versioning, environment control, and trajectory logging increasingly important for meaningful comparison.

6.4.2. Tool Use with Multi-Step Planning

Early broad evaluations offered wide behavioral coverage, but most tasks still terminated at a final answer rather than an executable trajectory, leaving tool choice, intermediate decisions, and recovery after failure largely invisible [70,343]. Once external action interfaces became part of the model stack, benchmark design shifted toward making action itself observable. GAIA framed real-world problem solving in ways requiring decomposition and external operations; API-Bank turned structured API invocation into the benchmark object; MetaTool, ToolTalk, and ToolQA scored the full sequence of tool-mediated decisions rather than only the final response; and ToolBench enlarged the space further by increasing tool diversity and making tool selection a substantial part of the task [72,249,256,301,510,511].

As this line matured, benchmark design became more differentiated. Some benchmarks focused on tool-call quality, function-calling reliability, and reproducibility [25,46,257,512]. Others shifted toward execution under more realistic constraints, including transactional long-horizon interaction, richer traces, and stronger environment grounding [47,53,253,287,513]. At the same time, the benchmark object widened beyond single-agent tool invocation to include social interaction, strategic behavior, coordination, and multi-agent collaboration, reinforcing trajectory-level evaluation as the dominant assessment form in this area [31,78,231,288,399,514,515].

Across both lines, agentic and interactive benchmarks are best understood not as a simple extension of answer-only testing, but as settings in which object, protocol, and metric become tightly coupled, where changes in interface state, tool schema, execution constraints, or trajectory scoring can all materially alter what a reported score means.

7. Current Threats to Benchmark Comparability

The preceding analysis suggests that benchmark comparability cannot be determined from benchmark names alone. Once a benchmark is decomposed into its Object, Setting, and Evaluation, it becomes easier to see why superficially similar results are often not directly comparable. Benchmarks that are all described as evaluating reasoning may differ in the capability they actually target, the conditions under which that capability is elicited, and the metric semantics used to define success.

Accordingly, we identify current threats to benchmark comparability in two broad categories: excessive heterogeneity of benchmark and insufficiently scientific benchmark settings.

7.1. Excessive heterogeneity of benchmarks

7.1.0.1. Heterogeneity from Temporal and Environmental Drift

The first source of heterogeneity is temporal drift. This problem is especially salient in tasks grounded in live knowledge, recent events, web content, or competitive programming streams. In such settings, the target world changes faster than benchmark publication and maintenance cycles, so a benchmark may become partially outdated even when its original design was sound. Under these conditions, lower model performance should not be interpreted too quickly as weaker reasoning. Instead, it may indicate that the benchmark gold label, supporting evidence, or assumed world state no longer matches the current world. A second source of heterogeneity is environmental drift. If temporal drift changes the world that a benchmark refers to, environmental drift changes the conditions under which that benchmark is executed. This issue is especially serious in interactive environments and software-based tasks, where even small changes in dependencies, interfaces, operating systems, packages, or execution settings can alter which action paths are available to a model and which outcomes count as success. Consequently, two evaluations conducted under the same benchmark name may still produce different outcomes because the effective environment is no longer the same [80,222,244].

7.1.0.2. Heterogeneity from Reporting

A further source is reporting heterogeneity. Even when the target world and execution environment remain relatively stable, comparisons can still break down if studies differ in how the benchmark is operationalized and reported. In practice, papers often use the same broad benchmark label while differing in stopping rules, retry policies, budget limits, parser assumptions, retrieval settings, tool permissions, or evaluation filters. At first glance, these choices may seem secondary. In fact, they shape what the model is allowed to observe, what actions it may take, how long it may continue, and how its output is interpreted. For that reason, they are not incidental implementation details but part of the benchmark itself. Once such differences are left implicit, benchmark comparisons can become misleading, and reported rankings may shift without reflecting a meaningful change in underlying reasoning ability [271,310,516].

Taken together, temporal drift, environmental drift, and reporting heterogeneity do more than introduce noise into evaluation. More importantly, they show why benchmark scores are difficult to interpret unless object, setting, and evaluation remain aligned. A reported score is scientifically meaningful only when the benchmark object is stable, the evaluation setting is sufficiently controlled, and the metric preserves a consistent notion of success. Once any of these conditions breaks, the score no longer reflects model capability alone; it also reflects changes in benchmark configuration. This is precisely why benchmark names are often poor comparison units.

7.2. Insufficiently scientific benchmark settings

The problem becomes even more serious when evaluation is diagnostically weak. Many reasoning benchmarks still summarize performance with a single end-to-end score. Such a score is convenient for ranking, but it is often too coarse for interpretation, because it does not reveal whether failure arose from retrieval, reasoning, grounding, attribution, action selection, or evidence selection.

This limitation is especially serious in evidence-intensive tasks, where a system may fail for different reasons even when the final answer is equally wrong. A benchmark that separates answer quality from support quality therefore provides clearer diagnostic value than one reporting final correctness alone, since it distinguishes failure in producing an answer from failure in establishing valid support for that answer [360,383,517,518]. For these reasons, current reasoning benchmarks should not be compared at the level of benchmark names alone. What must be compared instead is the full benchmark configuration: the capability under test, the conditions under which it is elicited, and the semantics of the metric used to score it. Only under that stronger notion of alignment can benchmark results function as reliable evidence rather than as superficially comparable numbers.

8. Practical Guidelines

Having established that benchmark results are meaningful only insofar as they provide decision-relevant evidence about model behavior, the discussion must now move from evaluative use to evaluative design. This shift is necessary because the evidential value of a benchmark is not determined at the moment of score interpretation alone, but is already structured by the way the benchmark is chosen and built. In other words, benchmark selection and benchmark construction are not separable stages, but two aspects of the same scientific problem: the former asks what kind of evidence is needed for a deployment claim, while the latter asks how such evidence can be generated in a valid, interpretable, and reproducible form. As shown in Figure 9, we therefore address these two questions in sequence, first by examining how to choose an appropriate benchmark (Benchmark Usage), and then by specifying how to build a scientifically sound benchmark (Benchmark Construction).

8.1. How to Choose an Appropriate Benchmark

Selecting an appropriate benchmark is fundamentally an exercise in matching evaluation evidence to deployment need, not in following the most visible leaderboard. The primary question is not which benchmark is popular or convenient to run, but which deployment failure the benchmark is designed to rule out [20,36]. A benchmark is appropriate only when its task content, operational setting, and score semantics expose that failure directly, rather than through a nearby proxy [20,519]. In this sense, benchmark selection is neither a matter of convenience nor convention, but of whether the chosen benchmark can furnish evidence genuinely relevant to the claim being made about model behavior in deployment.

Benchmark selection should therefore begin with the core capabilities. Practitioners should ask which dependency structure governs success in the target use case–whether that involves symbolic derivation, quantitative constraint tracking, long-context evidence aggregation, grounded tool use, or long-horizon action coherence [18,47,50,71]. A benchmark that targets the wrong dependency structure may still yield strong scores, yet those scores offer little practical reassurance about deployment behavior [20,37,520]. The operational setting should then be treated as constitutive of benchmark identity rather than as an ancillary implementation detail. Tool permissions, retrieval access, context length, interaction horizon, budget constraints, and environment dynamics all shape what kind of competence is actually being measured [45,47,222,340,521]. Two benchmarks bearing similar task labels may thus yield very different evidence once these conditions diverge [47,53,71,79]. What matters is not surface-level categorical similarity, but whether the benchmark faithfully reproduces the operative conditions.

Score semantics must likewise be grounded in decision relevance. A benchmark proves more useful when its reported outcomes align with the downstream cost profile of failure [20,343,519]. In some use cases, final answer correctness is the overriding concern; in others, evidence support, robustness under perturbation, or resource efficiency are equally consequential. Benchmark choice should also reflect this asymmetry rather than assume that a single headline figure suffices for all model-selection decisions [343,345,360,366,521]. This consideration explains why coverage and realism must be balanced deliberately: broad benchmarks are valuable for initial screening, but they simplify operational conditions and compress heterogeneous failure modes into aggregate scores [32,343]. Narrower benchmarks may span fewer tasks, yet provide stronger evidential weight in high-stakes domains because they preserve more of the true operational structure [222,238,340]. The right benchmark is therefore not simply the largest, the most difficult, or the one with the most recognized leaderboard; it is the one whose reasoning object, setting, and score semantics best match the deployment claim under scrutiny [20,519].

Freshness and contamination risk deserve explicit attention before any benchmark is adopted as decision evidence. Static benchmarks can remain valuable as historical baselines, but they grow less informative when models have likely encountered overlapping data distributions, or when the environment the benchmark represents has materially changed [271,310,454,516]. Practitioners should therefore treat benchmark age, refresh policy, and contamination control as first-order selection criteria rather than afterthoughts [271,310,522]. A well-chosen benchmark ultimately supports a bounded claim: it tells the user what kind of reasoning behavior has been tested, under which conditions, and with what practical relevance [20,519]. When such linkages are explicitly established, benchmark outcomes constitute meaningful, interpretable evidence; in their absence, benchmark comparisons reduce to little more than superficial score ornamentation [20,36].

8.2. How to Build a Scientifically Sound Benchmark

A scientifically sound benchmark begins with a falsifiable target claim: Benchmark construction should start from a concrete capability question or deployment risk, then define evidence of success or failure [20,36,519]. This anchors the benchmark in a theoretically grounded reasoning object, thereby preventing the drift toward tasks that are merely convenient to collect yet fail to provide meaningful diagnostic insight into model capabilities [20,36].

Task design must satisfy two conditions. First, it should preserve the dependency structure that the benchmark intends to measure. If the goal is to test multi-step reasoning, success should depend on maintaining valid intermediate dependencies rather than exploiting shallow cues or annotation artifacts [41,114,520]. If the goal is to test grounded decision making, the benchmark should require state tracking, constraint satisfaction, or evidence integration that cannot be bypassed by surface matching alone [53,222,229,340]. Then, control the difficulty designed: benchmark builders must distinguish genuine reasoning difficulty from accidental difficulty caused by ambiguous phrasing, unstable grading, noisy annotation, or missing context, since a good benchmark is challenging for principled reasons and does not rely on confusion as a substitute for rigor [36,523,524].

Meanwhile, benchmark construction should also account for both the quality of the underlying data and the methodology used to evaluate model behavior. The former requires explicit specification of data provenance and protocol formulation, since these elements shape the observable behavior being measured [20,53,257,273,510]. The latter should likewise preserve structure rather than collapse it prematurely: benchmark builders should report supported evaluation units and dimensions, since metric panels are more scientifically informative than a single aggregate score and reveal trade-offs that would otherwise remain hidden [20,41,343,345,360,366,519,521].

Scientific benchmark construction further requires active defenses against misleading progress claims and a commitment to reproducibility and interpretive honesty. Contamination checks, controlled perturbations, freshness management, and version governance should be built into the benchmark design from the beginning, since without these safeguards score gains may reflect memorization, formatting sensitivity, or stale distributions rather than transferable capability improvement [162,271,310,373,454,516,525]. A reusable benchmark package should include immutable data snapshots, prompt templates, parsing and matching code, runtime specifications, and reference scripts reproducing published aggregates, with environment-based benchmarks additionally requiring deterministic logging and artifact storage [53,222,237,257,273,340,519,526]. Finally, benchmark builders should explicitly state the interpretation boundary: a strong benchmark specifies what it measures, what it omits, and why the chosen evidence is sufficient for the target use case, which makes results cumulative across papers, labs, and model generations [20,519,527].

9. Future Directions

As shown in Figure 10, the future of reasoning benchmarks will likely be shaped by six closely connected directions: integrated measurement, temporal and verifiable evaluation, comprehensive agentic scenarios, embodied scenarios, benchmark lifecycle stewardship, and benchmark ecosystem. Taken together, these directions point toward a benchmark paradigm that is more firmly grounded in scientific principles, more reflective of real-world deployment conditions, and more sustainable over time.

9.1. Integrated Measurement

The next phase of reasoning benchmark research will be defined less by larger test collections and more by stronger measurement architectures. Current progress has revealed a persistent gap between benchmark visibility and benchmark comparability [528,529,530]. New suites appear quickly, yet their results often remain difficult to align because they encode different assumptions due to insufficiently justified benchmark settings. Closing this gap requires infrastructure that treats protocol semantics as a primary research focus under the same reasoning object, not as appendix material [271,310,343].

A fundamental necessity is the establishment of unified cross-unit evaluation, as answer-level, process-level, and trajectory-level assessments are still often built in parallel and reported in isolation [32,41,531,532]. In this way, a promising direction is to design benchmarks in which these units are connected by scientific benchmark settings, so a final score can be traced back to intermediate reasoning validity and policy level behavior. Such designs would allow researchers to distinguish genuine reasoning improvements from compensatory heuristics that only work at one observational layer [41,222,244].

9.2. Temporal and Verifiable Evaluation

As reasoning systems are increasingly deployed in environments where facts, sources, and interfaces change over time, temporal grounding should become a standard benchmark dimension rather than an optional add-on [388,516,533]. Static evaluation is no longer sufficient for measuring performance in settings where models must reason over evolving knowledge, retrieve up-to-date evidence, and produce claims that remain checkable after the fact [388,534].

Therefore, a promising direction is to build temporally explicit and verifiable benchmarks that represent time at multiple levels, including item time, source time, and evaluation time, while also attaching answers to auditable evidence or executable verification procedures. Such benchmarks would make it possible to evaluate whether a system can distinguish durable knowledge from time-sensitive knowledge, update its conclusions when evidence changes, and support its outputs with claims that remain externally checkable. They would also encourage benchmark designs that age more gracefully through refreshable data pipelines, objective verification, and contamination-resistant update cycles. The significance of this shift is twofold. Methodologically, it would allow the field to separate durable reasoning competence from freshness-dependent performance, which would produce cleaner comparisons across models and evaluation rounds. Practically, it would make reported progress more trustworthy by reducing conflation between better reasoning, broader retrieval coverage, and more recent data exposure, while also moving benchmark design toward more auditable and realistic evaluation for dynamic knowledge environments.

9.3. Comprehensive Agentic Scenarios

Agentic reasoning evaluation opens a closely related frontier where safety, efficiency, and reliability must be co-measured rather than treated as separate leaderboards. In long horizon environments, a system can reach success while spending excessive resources, issuing risky actions, or relying on unstable recovery behavior. Future benchmarks should encode these tradeoffs directly in their reporting interfaces, allowing deployment teams to choose operating points with explicit risk awareness. This direction is especially urgent for real tool-based ecosystems, where small action errors can have irreversible downstream impacts [47,222,244,363].

A further opportunity emerges in long context and multimodal integration [221,535]. Many current suites still evaluate long context retrieval, multimodal grounding, and agentic planning in separate silos, while real deployments increasingly combine all three [77,216]. Future benchmark design should therefore move toward coupled tasks where systems must retrieve sparse evidence in long inputs, ground decisions in heterogeneous modalities, and execute coherent actions under bounded budgets. This coupling will likely reduce headline scores in the short term, but it will produce a more faithful picture of system readiness [44,45,79,244], a trend already visible in long-form or domain-specific settings and web or medical agent evaluations [192,194,195,196,205,227,228,235,236,237,238,536].

9.4. Embodied Scenarios

Embodied scenarios have become an important direction for next-generation reasoning benchmarks, as reasoning in the physical world is not only about producing plausible textual plans, but also about grounding decisions in perception, affordances, and action consequences. Recent embodied reasoning benchmarks have gradually moved from instruction following in simulated household environments to more realistic, long-horizon, and interaction-intensive settings [537,538,539]. Correspondingly, current methods mainly follow two technical routes. The first route is a modular planner–executor paradigm, where LLMs are used for high-level task decomposition, while external perception modules, value functions, or skill libraries are responsible for grounding and execution, as exemplified by SayCan [540]. The second route is end-to-end multimodal action modeling, which directly integrates visual observations, language, and robot trajectories into a unified model for action prediction, represented by PaLM-E and RT-2 [541,542]. Together, these lines of work suggest that embodied reasoning should be treated as a closed-loop process of observing, planning, acting, and revising, rather than as a static text-only reasoning problem.

Despite this progress, current embodied reasoning methods still face two major limitations. First, many existing settings remain overly dependent on simulation priors, fixed skill vocabularies, or narrow action interfaces, which means that strong benchmark performance does not necessarily indicate robust physical reasoning [539,543,544]. Second, current formulation protocols often emphasize final task success while under-evaluating process-level reasoning abilities, such as active exploration under partial observability, uncertainty awareness, task tracking, and recovery from execution errors [211,545]. Therefore, the key future direction is to design embodied reasoning benchmarks that explicitly evaluate closed-loop grounded reasoning: capabilities should be tested on whether they can update their beliefs from new observations, choose actions based on affordances and safety constraints, re-plan after failures, and coordinate with humans or agents in dynamic environments [211,544,545].

9.5. Benchmark Lifecycle Stewardship

Benchmark governance should move beyond a paper-centric release model toward full lifecycle stewardship. As reasoning continues to evolve in different forms and with shifting emphases, evaluation cannot rely on static benchmarks released once and then left unmanaged. Instead, benchmarks should be organized around a more controlled and comparable structure across three core dimensions: object, setting, and metrics. That is, the community should be explicit about what capability is being evaluated, under what conditions it is evaluated, and by what criteria performance is judged. Only with such structure can benchmark results remain interpretable and comparable across generations of tasks, models, and evaluation regimes.

Benchmark development should be treated as a lifecycle rather than a one-time artifact. The process should begin with careful design, including explicit construct definitions, scope decisions, contamination safeguards, and documentation of intended use [519,546]. It should then be followed by meta-evaluation: assessing not only model performance on the benchmark, but also the benchmark’s own validity, sensitivity, robustness, and patterns of actual use within the community [547,548]. These signals should, in turn, inform whether a benchmark should be incrementally updated, substantially rebuilt, or formally retired [549,550]. Versioned artifacts, transparent patch logs, deprecation policies, and community-readable schema mappings are therefore not auxiliary governance features, but core infrastructure for maintaining continuity across benchmark generations.

9.6. Benchmark Ecosystem

The broader value of such a lifecycle is to create an evaluative ecosystem that is fairer, more cumulative, and more comparable. Without it, the field risks mistaking benchmark churn for scientific progress, while allowing hidden shifts in task design, settings, or metrics to undermine comparison. With it, reasoning evaluation can develop in a more orderly way: new benchmarks and new results refine shared instruments, preserve continuity where possible, and introduce change in a traceable form when necessary. In this sense, lifecycle stewardship is not merely about benchmark maintenance; it is a condition for building a healthier evaluative culture and for supporting the disciplined growth of the community [271,310,343].

10. Related Work

Related studies can be broadly grouped into two categories: surveys on reasoning in large language models and surveys on benchmarks and evaluation.

The first line of work focuses on reasoning itself, including its conceptualization, elicitation, and training in large language models. Early survey efforts such as Huang and Chang [551] provide a systematic overview of reasoning in LLMs, with a particular emphasis on informal deductive reasoning. In recent surveys, Sun et al. [552] review reasoning in foundation models from a more comprehensive angle, summarizing tasks, methods, benchmarks, and future directions, while also discussing multimodal reasoning, agents, and alignment-related issues. Plaat et al. [27] concentrate on multi-step reasoning and organize the literature around a generate-evaluate-control framework. Other work further synthesizes the development of reasoning LLMs from the perspective of System 1 and System 2 reasoning, highlighting architectural choices, training strategies, and evaluation practices [7].

The second line of work focuses on benchmarks and evaluation.

A representative example is the survey by Chang et al. [553], which reviews LLM evaluation from the perspectives of what to evaluate, where to evaluate, and how to evaluate. Moving further in this direction, Ni et al. [22] systematically summarize a large collection of LLM benchmarks and categorize them into general, domain-specific, and target-specific benchmarks. At the same time, several studies critically examine the reliability and validity of benchmarks themselves. For instance, Xu et al. [554] survey benchmark data contamination and show that training-test leakage can seriously undermine the interpretability of benchmark scores. In the multimodal setting, Li et al. [444] review benchmarks for multimodal large language models.

Similarly, Yang et al. [555] provide a critical review of causal reasoning benchmarks and argue that many existing datasets remain vulnerable to shallow pattern matching or knowledge retrieval. Taken together, existing benchmark-oriented surveys either cover evaluation too broadly or focus on a specific reasoning subdomain. In contrast, our work centers specifically on reasoning benchmarks, with particular attention to capability coverage, task setting, and the corresponding evaluation.

11. Conclusions

In this work, we propose a comprehensive survey for reaoning benchmarks of large language models. This survey introduces a new perspective for better understanding the broad landscape of reasoning benchmarks, including the object, settings, evaluation as well as the extended scenarios. In addition, we extend our discussion to the current limitations of this area, practical guidelines for practitioners, and potential future directions for advancing reasoning benchmarks. To further support the community, we have also been maintaining an online survey repository that continuously tracks newly released benchmarks under our structured outline. We hope this effort can serve as a useful resource for researchers and practitioners, and contribute to the broader development of the community.