A Survey of LLM-Powered Question Answering Through the Lens of Query Processing

1. Introduction

Question answering (QA) has traditionally been studied as the problem of locating and organizing information relevant to a natural language question, where answers can be generated through retrieval, matching, and inference over a fixed corpus [1]. The rise of large language models (LLMs), however, has substantially expanded the scope of QA. Modern LLM-powered QA systems increasingly address complex questions over diverse information sources by integrating evidence, performing multi-step reasoning, invoking external tools and computational resources, and synthesizing verified conclusions from intermediate results [2,3,4,5]. In such settings, QA is no longer merely a single-step generation task, and inherently requires a coordinated, multi-stage analytical pipeline.

To illustrate this complexity, consider the question in Figure 1: “Which benchmark most accurately reflects the reasoning ability of large language models in real-world scenarios?” This semantically underspecified question cannot be reliably answered through direct retrieval or one-shot generation. Instead, answering it requires coordinated reasoning, retrieval, execution, and verification. To provide a structured view of this process, we organize it into a five-stage pipeline:

• Query Rewriting interprets the original question and transforms it into an explicit, grounded analytical intent. As shown in Figure 1, the pipeline begins by analyzing the essential intent of the question (e.g., ranking benchmarks) and translating vague concepts like “real-world scenarios” into concrete evaluation criteria (e.g., alignment and coverage) and reasoning constraints to guide subsequent planning.
• Logical Planning determines the required operations and their dependencies, independently of the specific tools or resources used to execute them. As illustrated in Figure 1, this stage constructs an abstract workflow that organizes the key operations, including defining the evaluation criteria, identifying candidate benchmarks, comparing their characteristics, and ranking the results.
• Physical Planning instantiates the logical workflow by binding each operation to concrete data sources, tools, or models under resource constraints. As illustrated in Figure 1, this stage may involve selecting sources such as Google Scholar and Hugging Face, as well as analytical tools such as Python scripts and LLM inference.
• Execution & Adaptation executes the instantiated workflow while adapting to runtime conditions and intermediate results. As shown in Figure 1, this stage performs retrieval in batches, runs generated Python code, and determines whether additional retrieval or re-execution is needed when the collected evidence is incomplete or inconsistent.
• Synthesis & Verification integrates collected results into a coherent answer and validates the answer against the supporting evidence and task constraints. As depicted in Figure 1, this stage resolves conflicting information across heterogeneous sources, traces the relevant evidence, performs fact-checking, and iteratively refines the draft into a final verified response.

The five-stage pipeline above is not merely a descriptive decomposition. It reflects a fundamental need to separate different types of decisions while coordinating them toward a common end-to-end objective. A similar principle has long underpinned database query processing. Database systems translate high-level declarative queries into executable plans through a systematic, multi-stage process that separates logical and physical optimization [6,7], applies cost-aware planning [8,9], and supports adaptive execution [10,11]. The established principle suggests that a pipeline-oriented perspective can provide a useful lens for understanding modern LLM-powered QA systems. However, this perspective has not yet been systematically reflected in existing surveys.

Broad QA surveys typically organize the literature around answer types, application scenarios, or generation paradigms [12,13,14], offering effective frameworks for understanding what QA systems achieve and where they are deployed. Specialized surveys examine particular components in greater depth, such as retrieval-augmented generation [15], reasoning and planning [16,17], or agent-based QA [18]. Despite these valuable perspectives, the literature still lacks an integrated account of how the stages of LLM-powered QA systems are connected, how design choices at each stage are motivated, and how these choices collectively determine end-to-end system behavior.

Complementing these existing perspectives, we organize recent research on LLM-powered QA through the five-stage pipeline – Query Rewriting, Logical Planning, Physical Planning, Execution & Adaptation, and Synthesis & Verification – and examine both stage-wise design objectives and cross-stage interactions. We focus on methods and systems operating over both textual and tabular data. These are two fundamental data modalities for answering questions and provide representative settings for studying the core challenges of processing unstructured and structured information. This scope keeps the analysis focused while retaining broad coverage of the architectural challenges shared across QA systems. Rather than cataloging techniques in isolation, we first examine how representative systems instantiate and coordinate the five stages. Building on this system-level view, we then organize the literature around the recurring design objectives at each stage, as summarized in Figure 2. This structured perspective clarifies not only what techniques have been developed, but also why they are needed within the broader pipeline and how they shape end-to-end system behavior.

Our Contributions. We make the following contributions:

• A Pipeline-Oriented Framework for reviewing LLM-powered QA. This survey presents the first systematic effort to organize the literature on LLM-powered QA, primarily published between 2022 and 2026, through a five-stage pipeline. This framework offers a unified, structured perspective for understanding the distinct roles of individual stages and their interactions within the end-to-end QA pipeline.
• A System-Level Analysis of LLM-powered QA Architectures. We map representative QA systems over both textual and tabular data onto the five-stage abstraction and analyze how their architectural designs have evolved. This analysis identifies recurring instantiation patterns and reveals cross-stage bottlenecks.
• A Stage-Wise Technical Taxonomy. Based on the system-level analysis, we organize the literature around the recurring design objectives within each pipeline stage and systematically review the corresponding methods, design choices, and stage-specific challenges. This taxonomy provides a fine-grained reference for understanding how individual techniques contribute to the broader QA pipeline.
• Cross-Stage Lessons and Future Directions. We synthesize the overarching challenges that cut across individual stages and identify five priorities for next-generation QA systems: 1) Evaluation must shift from end-to-end answer metrics toward stage-aware diagnostics that localize failures; 2) Pipeline Coordination calls for dynamic, bidirectional feedback loops that allow downstream signals to trigger upstream replanning; 3) Resource Management should be guided by cost-aware physical planning that jointly optimizes efficiency and quality; 4) Faithful Reasoning demands that intermediate outputs strictly reflect actual execution, making verification auditable rather than superficial; and 5) System Generality depends on modular compositionality through standardized stage interfaces, enabling reusable and adaptable architectures. These priorities collectively outline a principled research agenda for making LLM-powered QA systems more systematic, efficient, and trustworthy.

The remainder of this survey is organized as follows. Section 2 examines representative systems from a cross-stage perspective, analyzing their stage coverage, recurring pipeline patterns, and coordination limitations. This analysis shows that Query Rewriting and Logical Planning have received relatively extensive attention, whereas Physical Planning, Execution & Adaptation, and Synthesis & Verification remain less systematically explored and present broader design opportunities. Accordingly, Section 3, Section 4, Section 5, Section 6 and Section 7 examine the five stages in turn. For the more established stages, we concisely summarize the key objectives, representative methods, and remaining gaps. For the less explored stages, we provide a more detailed analysis of underexamined design choices and optimization opportunities. Section 8 then synthesizes the overarching challenges that span multiple stages, and Section 9 concludes the survey.

2. Pipeline Instantiation in LLM-Powered QA Systems

To understand modern LLM-powered QA architectures through the five-stage pipeline, Table 1 maps a range of representative systems onto this stage-wise abstraction. Based on this mapping, we first examine how stage coverage of the pipeline has evolved over time (Section 2.1). We then abstract diverse architectural designs into recurring patterns of pipeline instantiation (Section 2.2). Finally, we analyze the varying levels of development across stages and identify the critical cross-stage bottlenecks that limit end-to-end performance (Section 2.3).

2.1. System Evolution with Stage Coverage

As shown in Table 1, LLM-powered QA systems have gradually expanded from partial pipeline implementations toward broader stage coverage. This architectural expansion is not merely a result of chronological progression; rather, it reflects the growing complexity of user questions and the diversity of the underlying data modalities. We summarize this evolution through four representative architectural shifts.

• Early systems do not explicitly separate planning from execution. Early systems such as ReAct [3] and Reflexion [19] primarily support turn-by-turn interactions over textual data (e.g., simple conversational loops or raw document retrieval). Rather than explicitly interpreting the original question or constructing a workflow in advance, they rely on the LLM to determine the next action dynamically based on the current context, such as generating search terms, API parameters, or tool calls. This lightweight design supports flexible interaction, but without explicit intent interpretation and workflow planning, longer-horizon execution is more prone to context drift and hallucination.
• Hybrid data modalities drive explicit upstream planning. As QA systems increasingly operate over hybrid data, directly generating execution commands from the original question becomes less reliable. For example, operating over multi-table relational data and layout-heavy documents may require schema-aware operations and join planning for tables, as well as appropriate parsing and extraction workflows for documents. Systems such as SiriusBI [22] and ST-Raptor [23] therefore introduce explicit upstream planning. Query Rewriting clarifies and structures the user intent, while Logical Planning translates the clarified request into an abstract workflow before execution begins.
• Large-scale data processing necessitates the decoupling of physical planning from logical planning. When QA systems process large volumes of textual or tabular data, identifying the required operations is no longer sufficient. The system should also determine how to execute them efficiently. Operational cost, token consumption, and latency become important design considerations, motivating the separation of Physical Planning from Logical Planning. The former maps abstract logical operations to concrete execution strategies without changing the underlying intent. For example, PALIMPZEST [30] selects model configurations to balance quality and cost; QUEST [32] routes requests to semantic caches and vector indices to reduce latency; and DSPy [26] and DocETL [29] optimize prompt-based workflows to improve execution efficiency.
• Advanced analytical workloads encourage full-pipeline integration. Modern analytical workloads increasingly require long-horizon, multi-step reasoning and synthesis over large-scale, heterogeneous data sources. Recent systems such as Data Interpreter [34], AgenticData [35], and AgenticScholar [37] therefore coordinate a broader range of stages within a unified architecture. In these systems, Query Rewriting clarifies the original intent, Logical Planning constructs an abstract workflow, Physical Planning selects suitable execution strategies, Execution & Adaptation responds to intermediate results and runtime failures, and Synthesis & Verification validates the final conclusions. By integrating these stages, such systems provide stronger support for the efficiency, adaptability, and reliability required by complex analytical questions.

2.2. Common Patterns of Pipeline Instantiation

By reviewing these representative systems, we not only observe how different stages of the pipeline are instantiated, but also identify several recurring patterns in how these stages are organized and coordinated.

2.2.1. Pattern 1: Decompose-Then-Execute Paradigm

A common pattern is to decompose a complex question or processing task into smaller, more manageable units before execution. This pattern appears across systems operating over textual and tabular data, e.g., ST-Raptor [23] over tables and DocETL [29] over documents. It primarily connects Query Rewriting and Logical Planning: the former structures the original question, while the latter organizes the resulting units into an executable workflow.

2.2.2. Pattern 2: Iterative Feedback and Self-Refinement

Many systems move beyond one-pass generation by evaluating intermediate outputs and using feedback to revise subsequent steps. It is widely observed across the pipeline evolution, spanning early conversational systems like Reflexion [19] and recent analytic systems such as ST-Raptor [23] and DocETL [29]. It primarily links Execution & Adaptation directly with Synthesis & Verification: the former executes the planned operations to generate candidate answers or text segments, while the latter conducts validation routines – such as evaluating reflection logs against episodic memories [19] or checking data consistency against table schemas [23] – to provide feedback for successive refinement iterations.

2.2.3. Pattern 3: Multi-Agent Collaboration

This pattern replaces monolithic LLM interactions with distributed multi-agent coordination, dividing functional pipeline responsibilities among specialized autonomous components. This pattern is prevalent in systems that orchestrate complex multi-step workflows, ranging from early text-based systems like MetaGPT [20] to modern systems handling heterogeneous modalities, such as Insight Agents [24] and AgenticData [35]. It typically spans all five stages simultaneously it leverages structured communication protocols or localized state-machine graphs to route intermediate data, enabling specialized agent personas to independently manage and validate distinct stages of the end-to-end pipeline.

2.2.4. Pattern 4: Cost-Aware Query Optimization

This pattern explicitly decouples the logical plan from its physical implementation to enable cost-aware optimization. Systems such as DSPy [26], LOTUS [33], PALIMPZEST [30], and QUEST [32] represent the required processing logic using abstract modules or semantic operators, while allowing each operation to be executed through different physical implementations. Logical Planning specifies what operations are required by the query, while Physical Planning determines how to execute them efficiently under resource constraints by selecting suitable strategies, such as model scales, prompt strategies, memory mechanisms, or index structures, to balance cost, latency, and output quality.

2.3. Stage-Wise Development and Cross-Stage Bottlenecks

The preceding analysis of stage coverage and recurring instantiation patterns reveals two complementary observations. From a stage-wise perspective, different pipeline stages have received varying levels of research attention. From an end-to-end perspective, many remaining limitations arise not only within individual stages, but also from insufficient coordination across them.

Stage-wise development. Upstream stages – specifically Query Rewriting and Logical Planning – have largely converged toward standardized paradigms deeply inspired by traditional database query processing. Query Rewriting typically transforms ambiguous natural language into explicit analytical intents, while Logical Planning systematically maps these intents into well-defined intermediate representations. For instance, modern systems consistently compile queries into structured formalisms, such as relational and semantic operators in Unify [21] and AgenticData [35], Directed Acyclic Graphs (DAGs) in AgenticScholar [37], or tree-operation processes in ST-Raptor [23]. This reliance on established data structures makes both the rewriting and logical planning stages highly predictable.

By contrast, downstream stages – Physical Planning, Execution & Adaptation, and Synthesis & Verification – remain less systematically studied. LLM-based operations often introduce variable monetary costs, latency, and hallucination risks, making it challenging to determine suitable execution strategies. For example, Physical Planning still lacks a general cost model that can be applied across systems and workloads. Existing approaches typically optimize for system-specific objectives, such as budget-aware execution in Doctopus [31] or cost-aware operator ordering in QUEST [32]. Similarly, mechanisms for Execution & Adaptation often rely on workload-specific routing and recovery strategies. Synthesis & Verification has received comparatively less attention: as shown in Table 1, many systems do not include explicit verification mechanisms, limiting their ability to assess and improve end-to-end reliability.

Cross-stage bottlenecks. From an end-to-end perspective, limitations often arise at the interfaces between stages. Decisions made upstream may constrain downstream execution, while runtime observations and verification results are not always propagated back to earlier stages. We identify four recurring cross-stage bottlenecks:

• Decomposition-Execution Decoupling. Logical decomposition and planning are often performed before execution begins, with limited consideration of runtime factors such as resource constraints, tool availability, or data scale. As a result, an initially reasonable plan may become inefficient or infeasible during execution. More adaptive coordination is needed to revise logical plans based on downstream execution conditions.
• Localized Feedback Confinement. Existing refinement mechanisms often focus on revising local outputs or actions. For example, Self-Refine [38] iteratively improves generated responses through self-feedback, while CRITIC [39] incorporates external tools to validate and revise outputs. However, feedback signals are not always propagated back to earlier stages to trigger more fundamental changes, such as rewriting the original query or reconstructing the logical plan.
• Isolated Optimization Objectives. Many systems optimize individual components of the pipeline in isolation. For example, DSPy [26] optimizes LM pipelines through declarative modules and prompt-level configurations, while PALIMPZEST [30] explores cost-aware execution strategies for semantic data processing. Such component-level optimization is valuable, but pipeline-wide trade-offs among quality, latency, cost, and reliability remain insufficiently explored.
• Limited Uncertainty Propagation. Intermediate results produced by semantic operations may contain ambiguity or uncertainty, but downstream stages often consume them without explicitly modeling their reliability. As these intermediate results pass through multiple stages, small errors can accumulate and affect the final answer. This motivates end-to-end mechanisms that represent, propagate, and resolve uncertainty throughout the pipeline.

3. Query Rewriting

As discussed in Section 2, Query Rewriting has received substantial attention in modern LLM-powered QA systems. Its central role is to transform a raw natural language question into a disambiguated, structured, and knowledge-grounded representation that downstream planning and execution stages can reliably consume.

As illustrated in Figure 3, this stage involves three core objectives. Query Interpretation clarifies the user’s underlying intent. In our example, this involves resolving the vague notion of “real-world scenarios” into explicit evaluation criteria (e.g., alignment with practical tasks and coverage of diverse reasoning skills) and deciding whether the question presupposes a single best benchmark or a ranked list. Query Structuring then converts this interpreted intent into a machine-readable computational form, decomposing the question into ordered analytical steps, such as identifying candidate benchmarks, defining measurable proxies for alignment and coverage, retrieving evidence about each candidate, and ranking them by aggregated scores. Finally, Knowledge Grounding links abstract references in the structured query to concrete entities and schemas in external knowledge sources, e.g., mapping “benchmark” to actual named resources (e.g., MMLU [40], BIG-bench [41]), “reasoning ability” to documented metrics or leaderboard dimensions, and the intended data sources to specific venues such as Google Scholar or Hugging Face datasets. These objectives are closely related: inaccurate intent interpretation may lead to an inappropriate query structure, while incomplete grounding may make the resulting query difficult to execute reliably. We next survey representative methods for each objective and summarize the key insights.

3.1. Query Interpretation

Query interpretation addresses the inherent ambiguity and context dependency of natural language. The dominant paradigm, established by AmbigQA [42], is to enumerate multiple plausible interpretations rather than commit prematurely to a single reading, and to let downstream stages or users resolve the remaining uncertainty. This principle has been operationalized through clarification dialogues [43] and reinforcement-learning-based rewriting [44]. While LLMs are effective at local semantic reasoning – resolving anaphora or expanding under-specified references – they are not a substitute for higher-level control. Diagnostic studies [45] demonstrate that, when relied on too heavily, LLM-driven query expansion systematically favors popular interpretations and degrades recall on rare or highly polysemous queries; similarly, naive context retention can propagate stale information under topic shifts [46]. Hence, the key design tension is not whether to disambiguate, but when and at what cost: forced early commitment simplifies downstream processing at the risk of propagating errors, while preserving alternatives shifts the resolution burden to later stages.

Key Insight. A common strategy is to preserve multiple candidate interpretations when a query is ambiguous, rather than committing to one interpretation too early. However, current systems often lack explicit, context-aware policies for deciding when to retain multiple candidates, when to select one, and when to discard outdated context. As a result, ambiguity resolution remains fragile, and the trade-offs between accuracy, cost, and context usage are not systematically managed.

3.2. Query Structuring

Query structuring makes the computational logic of a question explicit, typically by decomposing it into a set of interdependent sub-queries. Early work such as the QDMR formalism [47] established that a complex question should be represented as an ordered sequence of atomic operations, where each step may explicitly reference the outputs of prior steps. This explicit dependency modeling is foundational because, without it, sub-questions risk being executed in arbitrary order and compounding errors. LLM-driven decomposition techniques, including least-to-most prompting [48] and decomposed prompting [49], have shown that solving simpler sub-problems first and composing their outputs can extend reasoning horizons, yet they inherit the same dependency challenge. A critical finding is the compositionality gap [50]: models often answer individual sub-questions correctly but fail to combine them into a coherent final answer. This reveals that generating plausible intermediate steps is not sufficient – those steps must be executable, correctly ordered, and composable.

Key Insight. It is increasingly recognized that LLM-driven decomposition should be paired with explicit dependency structures to avoid arbitrary execution orders and cascading errors. However, a compositionality gap remains: even if individual sub-tasks are answered correctly, the intermediate results may not combine into a correct final answer. Existing systems still lack verifiable constraints for checking, before execution, whether a generated plan is complete, whether its dependencies are valid, and whether its intermediate results can be reliably combined.

3.3. Knowledge Grounding

Knowledge grounding bridges the gap between the abstract query representation and the actual contents of external knowledge sources. A practical hurdle is that the surface forms used in the question (e.g., “real-world reasoning”) often differ substantially from the terminology found in databases, papers, or leaderboards. A common approach to mitigate this vocabulary gap is query enrichment: generating pseudo-documents [51] or hypothetical answers [52] that expand the query with related terms, effectively softening the mapping to downstream retrieval indices and entity descriptions; such generated content, however, carries a risk of hallucination that must be controlled. Beyond this enrichment step, structured grounding explicitly aligns query elements with knowledge base entries. For structured data, schema linking has matured from monolithic encoder-decoder models to modular pipelines that separate what to reference from how to query [53,54], with constrained decoding [55] and critical-element prioritization [56] significantly improving reliability. For textual corpora and knowledge bases, entity linking has similarly moved toward efficient bi-encoder architectures [57] and type-aware zero-shot methods [58]. A consistent observation across these settings is that reliability depends less on model scale than on how the grounding problem is formulated: decoupling selection from generation and prioritizing critical elements improves robustness more than switching to a larger LLM.

Key Insight. Reliable grounding depends more on structured representations and specialized linking mechanisms than on simply using larger models. Many systems therefore separate routine entity or schema linking from LLM-based semantic enrichment. However, a key challenge remains: existing systems lack principled mechanisms for coordinating these components, such as deciding when to rely on specialized linkers, when to invoke an LLM, and how to validate newly generated grounding results. As a result, hallucination risks remain insufficiently controlled across the pipeline.

4. Logical Planning

As discussed in Section 2, Logical Planning has become an increasingly explicit component of modern LLM-powered QA systems. Its central role is to commit to an explicit reasoning strategy: given a well-interpreted query from the rewriting stage, logical planning determines how the system intends to construct the answer. Overall, this stage concentrates on four core objectives.

As depicted in Figure 4, each objective corresponds to a distinct decision the system must make. Reasoning Planning determines the structural form of the reasoning process – for instance, whether benchmark comparison should follow a sequential chain, branch into parallel evaluation dimensions, or model inter-benchmark citation dependencies as a relational graph. Workflow Generation then transforms this topology into an executable coordination plan, partitioning the analysis into sub-tasks (e.g., defining evaluation criteria, scoring candidates, aggregating rankings) and deciding which are handled by LLM reasoning and which are delegated to external tools. Plan Representation defines the formal encoding of the workflow, choosing, for example, whether evaluation criteria remain as natural language instructions, become executable Python functions that query leaderboard APIs, or translate into structured SQL queries. Finally, Retrieval Planning schedules evidence acquisition: should all benchmark metadata be retrieved upfront, or should detailed evaluation results be fetched only after a candidate passes an initial relevance filter? These four objectives form a progressive refinement chain – reasoning planning defines the shape of reasoning, workflow generation assigns responsibilities, plan representation encodes the logic, and retrieval planning schedules evidence acquisition – jointly transforming an abstract analytical intent into an executable reasoning blueprint. Subsequently, we review prominent studies under each objective and critically analyze them to derive valuable insights.

4.1. Reasoning Planning

Reasoning planning has evolved from simple linear chains toward more expressive structures that accommodate the diversity of real-world analytical queries. Chain-of-thought prompting [2,59] demonstrated that exposing intermediate steps substantially improves reasoning quality, and subsequent work has focused on controlling the cost and robustness of linear trajectories through consistency checks [60], complexity-adaptive depth [61,62], and reinforcement learning [63]. When questions admit multiple plausible solution paths, tree-based approaches such as Tree of Thoughts [64] and RAP [65] explicitly search over candidate reasoning states, while graph-based methods [66,67] further support cyclic dependencies and knowledge-graph grounding.

Key Insight. A common strategy is to use linear reasoning chains for relatively simple factual questions and introduce tree- or graph-based search for more complex questions involving multiple candidate paths. However, such search structures require reliable evaluation signals to identify promising paths and prune low-quality ones. A key challenge is that LLMs are often unreliable at assessing their own intermediate outputs. As a result, tree- and graph-based reasoning may provide limited benefits unless they are supported by external verification mechanisms.

4.2. Workflow Generation

Workflow generation bridges the gap between abstract topology and executable orchestration. Hierarchical decomposition strategies [68,69] partition complex tasks into structured sub-goals and dispatch them to specialized models or tools. The separation between what remains inside the LLM and what is outsourced to deterministic modules is a recurring architectural motif: systems like MRKL [70] and Toolformer [71] route requests to external experts, while DSPy [26] abstracts entire pipelines as declarative computational graphs for systematic optimization. Planning strategy selection adds a further layer of adaptivity, with frameworks such as Self-RAG [72] and Adaptive-RAG [27] dynamically routing questions according to complexity estimates.

Key Insight. Finer-grained decomposition is not always beneficial, because coordinating a large number of subtasks may introduce substantial LLM costs and execution overhead. Existing systems therefore need to balance the benefits of decomposition against the cost of coordinating intermediate results, while assigning well-defined operations to deterministic modules whenever possible. However, this balance is often determined through manual heuristics. Systematic methods for estimating coordination costs in advance and automatically selecting an appropriate decomposition granularity remain underexplored.

4.3. Plan Representation

Plan representation defines the boundary between semantic reasoning and deterministic execution. Program-based representations delegate symbolic computation to Python interpreters or SQL engines, reducing numerical and logical errors [73,74,75], while structured query generation translates natural language into executable database queries [76,77]. Hybrid approaches [78,79] interleave natural-language reasoning with program execution to preserve flexibility where formalization is difficult.

Key Insight. A common strategy is to express well-defined operations in executable languages, such as SQL or Python, while reserving natural-language reasoning for steps that remain under-specified or require semantic judgment. However, determining which steps can be safely formalized often relies on manually designed heuristics. When a system falls back to LLM-based reasoning, semantic uncertainty is reintroduced, and the resulting outputs may be difficult to verify reliably.

4.4. Retrieval Planning

Retrieval planning schedules evidence acquisition to align with reasoning progress. Early work treated retrieval as a front-loaded preprocessing step [80], but subsequent research demonstrated that interleaving retrieval with reasoning [81] and adapting retrieval depth to runtime information gaps [82,83] consistently outperforms static strategies. Front-loaded retrieval often fetches irrelevant or prematurely aggregated evidence, whereas adaptive retrieval acquires information only when reasoning progress exposes genuine knowledge gaps.

Key Insight. A widely adopted practice is to interleave retrieval with reasoning, triggering evidence acquisition when an explicit knowledge gap is detected rather than following a fixed retrieval plan. However, the signals used to trigger retrieval remain unreliable. LLM confidence scores often correlate poorly with factual correctness [84], making it difficult to determine when additional evidence is genuinely needed. Without better-calibrated signals, a system may either miss necessary evidence or waste resources on unnecessary retrieval.

5. Physical Planning

Physical Planning bridges the gap between abstract logical plans and concrete execution strategies, determining how operations should be carried out under resource, tool, and infrastructure constraints. However, different aspects of the physical layer have received varying levels of attention: basic tool binding is widely adopted, whereas dynamic resource allocation and systematic memory management remain relatively underexplored. This imbalance can create cross-stage bottlenecks, as a well-formed logical plan may still lead to inefficient or unreliable execution when its physical implementation is poorly selected. These limitations give rise to three closely related challenges:

1. Intent vs. Physical Executability: Logical plans define what to retrieve but omit low-level physical configurations (index types, chunk sizes, retrieval depth, context ordering). Performance is acutely sensitive to these parameters, which remain statically configured in deployed systems and fail to generalize across query distributions.

2. Environment and Tool Heterogeneity: Systems must bind abstract operations to massive, evolving tool and model catalogs, necessitating robust routing across model pools that can withstand API drift, documentation changes, and execution failures.

3. Bounded State Management: Long-horizon QA tasks (e.g., multi-turn dialogues over long documents) operate under hard context window constraints, forcing strict trade-offs between information retention and compute costs that demand formal memory eviction criteria rather than heuristics.

5.1. From Challenges to Research Objectives

These challenges motivate four core objectives as illustrated in Figure 5. Within each, sub-tasks progress sequentially from structural setups to operational refinements. Across objectives, two critical dependencies govern execution: Memory Management’s budget constrains Retrieval Execution Planning, while Resource Allocation’s capacity bounds Tool Binding.

• Retrieval Execution Planning (addressing Challenge 1): Translates logical retrieval intents into concrete index access paths, hyperparameter configurations, and ordered contexts.
• Tool Binding (addressing Challenge 2): Grounds abstract tool descriptions into resilient API integrations that withstand specification drift.
• Resource Allocation (addressing Challenge 2): Optimizes infrastructure costs by routing queries across heterogeneous model pools and scheduling execution workloads.
• Memory Management (addressing Challenge 3): Governs state persistence, consolidation, and compression within hard context token limits.

5.2. Representative Methods Under Each Research Objective

This subsection systematically surveys the representative methods under each research objective, traces their evolutionary trajectories, and summarizes key insights.

5.2.1. Retrieval Execution Planning

Retrieval execution planning determines how a logical plan should access and organize evidence under resource constraints. It involves selecting suitable retrieval paths (Index & Access Path Selection), configuring extraction granularity (Parameter Configuration), and assembling the retrieved context so that the downstream LLM can process the evidence efficiently (Context Assembly).

•Index & Access Path Selection has evolved from balancing representation expressiveness and efficiency toward managing downstream LLM invocation overhead. Early dense and sparse paradigms, such as late interaction in ColBERT [85] and high-dimensional sparse mapping in SPLADE [86], decoupled semantic matching from heavy cross-encoders to optimize pure retrieval throughput. However, these methods typically treat retrieval as an isolated preprocessing phase, separate from downstream generation costs. To address this, modern architectures like QUEST [32] and DocDB [87] treat the index structure as part of the cost-management framework. By employing hierarchical data summaries and runtime-adaptive routing, they seek to defer or eliminate LLM execution during retrieval when structural evidence is sufficient, transforming the access path into a cost-aware pipeline gating mechanism.

•Parameter Configuration governs the query-time calibration of structural hyperparameters, such as chunk size, retrieval depth, and hybrid weights, to satisfy target quality-cost boundaries. Wang et al. [88] demonstrated that these low-level parameters are often highly interdependent; for instance, chunk sizing and hybrid ratios can significantly alter the semantic density of the context, meaning that lightweight cross-encoders approximate LLM reranking quality primarily when upstream parameters are properly conditioned. Going beyond empirical tuning, Doctopus [31] formalizes parameter selection as a budget-constrained portfolio optimization problem. Instead of applying uniform configurations across the entire system, it dynamically allocates distinct extraction and pruning strategies to individual text attributes based on their estimated information volatility, absorbing significant query variance before the generation stage.

•Context Assembly addresses the spatial-semantic organization of retrieved evidence to mitigate potential attention biases in LLMs. The lost-in-the-middle phenomenon [89] indicated that naive context concatenation can lead to attention degradation, as LLMs tend to underweight evidence located in the middle of long context windows. To enforce positional salience, UPR [90] utilizes a generative reranking paradigm, reordering passages based on query likelihood to place critical information near the window’s boundaries. Extending this to cost frontiers, Task Cascades [91] suggests that context composition is closely coupled with compute escalations; it utilizes low-cost surrogate models to filter and assemble preliminary contexts, escalating to full LLM context ingestion primarily when the proxy reports high semantic uncertainty.

Key Insight. These three subtasks are highly interdependent: they determine when LLM-based processing is needed and how much evidence should be passed to the model. Optimizing them independently may result in excessive context assembly. When upstream retrieval components cannot reliably assess whether sufficient evidence has been collected, the system may include more context than necessary to avoid missing relevant information. This increases token consumption and may distract the model with redundant evidence.

5.2.2. Tool Binding

Tool binding maps logical operations to appropriate external tools and services. It involves determining whether a tool call is needed and selecting the appropriate tool and invocation format (Tool Selection), and managing the live execution protocols and fault tolerance required in unpredictable web environments (External Integration).

• Tool Selection has transitioned from maximizing invocation accuracy toward incorporating execution restraint –specifically, training models to identify when external utilities may be redundant. While ToolLLM [92] pioneered large-scale API fine-tuning, its reliance on parametric memorization meant that models internalized static training-time documentation, making them sensitive to real-world interface alterations. Gorilla [93] counteracted this via Retriever-Aware Training, shifting the model’s role from memorizing APIs to dynamically parsing live, retrieved documentation at inference time. Furthermore, API-Bank [94] established that performance bottlenecks often lie in multi-step planning and error recovery rather than isolated tool invocation accuracy. To formalize this sequential decision process, ToolRL [95] frames tool selection as a reinforcement learning task with step-level rewards, helping cultivate self-correction and optimal non-invocation behaviors.
• External Integration elevates tool binding from closed execution sandboxes into live, interactive web services. WebGPT [96] established the foundational paradigm of LLMs behaving as interactive web agents, executing sequential browsing, text synthesis, and explicit citation logging. However, moving from internal tool calls to live external integration exposes the physical plan to environmental stochasticity, including network latencies, rate limits, and unannounced schema drift. Consequently, ensuring execution robustness often necessitates treating external integrations as dynamic and potentially untrusted dependencies that benefit from explicit exception handling, fallbacks, and multi-agent consensus protocols to protect the stability of the upstream logical plan.

Key Insight. The development of tool-using architectures indicates that physical robustness is difficult to achieve solely by expanding the capacity of the LLM. An alternative paradigm involves decoupling execution logic from tool specifications: the LLM serves as a coordination policy trained to handle control flow and restraint, while detailed API schemas are maintained in an externalized, non-parametric storage layer fetched dynamically at runtime. This separation can help protect the system from documentation drift without requiring frequent model retraining.

5.2.3. Resource Allocation

Resource allocation ensures logical plans execute within strict physical performance budgets by optimizing throughput, costs, and latency. Systems manage this across three architectural tiers: matching query complexity to appropriate model capacities (Model Routing), minimizing internal computation overhead for semantic operators (Inference Efficiency Optimization), and coordinating concurrent workloads to maximize aggregate batching efficiency (Compute Scheduling).

• Model Routing capitalizes on variations in query difficulty to reduce execution costs by mapping query complexity to corresponding model capacities. FrugalGPT [97] formalized this via sequential LLM cascades, routing queries to cheaper, specialized models first and stopping early if confidence metrics are satisfied, thereby approximating premium model performance at lower costs. To avoid the overhead of retraining routing classifiers whenever the available model pool shifts, UniRoute [98] introduced representation-based zero-shot routing, matching queries based on embedding space alignment with model capability profiles. Refining the routing telemetry, Self-REF [99] and RAGate [100] derive routing indicators directly from the model’s internal token-generation uncertainty, allowing the system to adaptively decide model scale and determine when retrieval-augmentation is necessary.
• Inference Efficiency Optimization targets the internal mechanics of individual model invocations and operator pipelines to compress per-token compute overhead. At the hardware-workload boundary, FlashAttention [101] and speculative decoding [102] minimize memory traffic bottlenecks and sequential generation latency via IO-aware kernel computation and small proxy draft models. Elevating this optimization to the relational abstraction layer, systems like Abacus [103] and LOTUS [33] introduce formal Cost-Based Optimizers (CBO) tailored for semantic operator pipelines. By separating logical execution flows from physical operator implementations, these frameworks compile optimized execution schedules, applying algebraic rewrites and cost-based selection to semantic joins and filters similarly to classical database query compilers.
• Compute Scheduling shifts the optimization focus from single-query latency to aggregate system throughput under concurrent, multi-tenant workloads. PagedAttention [104] resolved a key physical memory fragmentation bottleneck of serving infrastructure by introducing virtual memory management for LLM KV caches, enabling dynamic non-contiguous allocation across parallel requests. Moving beyond lower-level memory management, Halo [105] scales scheduling efficiency to cross-workflow compilation. It consolidates multiple independent query-plan directed acyclic graphs (DAGs) into a unified execution batch, executing shared semantic subexpressions simultaneously to maximize KV cache reuse and token throughput across separate requests.

Key Insight. Routing, inference optimization, and scheduling reduce execution costs from different perspectives, but their decisions may interact. For example, routing each query to its individually preferred model can distribute requests across a heterogeneous model pool, making batching and KV-cache reuse more difficult. Since efficient scheduling often benefits from processing similar requests together, system-level optimization must balance per-query routing decisions against overall batching efficiency.

5.2.4. Memory Management

Memory management transforms unbounded historical data into cost-effective context by balancing comprehensive preservation with strict token constraints. Systems coordinate this lifecycle by defining synchronization schedules and topologies (Memory Design), managing state migration across hierarchical storage tiers (State Persistence), and compressing retrieved states into information-dense formats prior to generation (Context Optimization).

• Memory Design dictates the synchronization schedule of state data, spanning a spectrum from write-heavy proactive structuring to read-heavy reactive indexing. MemoryBank [106] adopts a proactive write-time design, invoking the LLM upon receiving new inputs to summarize and index incoming experiences, which minimizes runtime latency via cheap, LLM-free retrieval during queries. Conversely, Generative Agents [107] utilizes an asynchronous offline approach, running periodic reflection loops where the LLM synthesizes high-level insights from raw memory logs during system idle time. HippoRAG [108] shifts the memory substrate itself to an offline graph topology, using the LLM to populate a schemaless knowledge graph that enables single-step, multi-hop contextual retrieval via personalized PageRank, bypassing iterative LLM reasoning at query time. Pushing toward continuous online synchronization, A-MEM [109] enforces real-time relational updates, where every new experience triggers an LLM inference step to trace and re-weight associative links across historical memory nodes.
• State Persistence manages coherent access across the boundaries of physical memory tiers, addressing the structural disconnect between historical state and finite context windows. MemGPT [110] handles this by replicating operating system memory architectures, providing the LLM with explicit control commands to read/write between in-context main memory and out-of-context archival storage; however, this can place noticeable cognitive and token overhead on the model during runtime loop control. To alleviate this, AgeMem [111] unifies long-term consolidation and short-term pruning into a single policy trained via reinforcement learning. By framing state migration as an optimization problem, it dynamically determines when short-term context should be consolidated into long-term archival stores or permanently evicted, facilitating state movement based on downstream performance metrics.
• Context Optimization acts as a compression filter, downscaling the persisted memory state to fit within physical token boundaries right before generation. RECOMP [112] trains specialized, task-aware compressors end-to-end to condense retrieved document clusters into compact summaries. Crucially, RECOMP incorporates a selective augmentation mechanism: when retrieved memory states are evaluated as irrelevant or noisy, the compressor outputs an empty string, executing an immediate physical prune. This mechanism is directly coupled with the Context Assembly phase of Retrieval Execution Planning; it helps ensure that token compression is optimized to protect the model’s positional attention focus against noise-induced degradation.

Key Insight. The architecture of a memory system largely depends on how computation costs are distributed across the state lifecycle. Proactive write-time or asynchronous offline processing can reduce runtime query latency, whereas reactive approaches minimize ingestion overhead at the expense of query-time computation. Beyond scheduling, memory management typically serves as an information-filtering mechanism; its performance often hinges on its ability to present relevant state data while minimizing redundant tokens that might otherwise degrade downstream generation quality.

6. Execution & Adaptation

Execution & Adaptation is the stage where physical plans are carried out in dynamic environments. It invokes tools, coordinates agents, and adjusts execution in response to partial, noisy, or changing external feedback. Although basic inference–action loops are widely adopted, more advanced mechanisms for multi-agent coordination and adaptive execution remain relatively fragmented. In particular, existing approaches often lack systematic frameworks for balancing execution quality against runtime costs. As a result, even well-designed plans may become ineffective when tools fail, intermediate results are incomplete, or runtime conditions change. We identify four closely related challenges:

1. Tight Coupling Between Reasoning and Action: LLM-powered execution interleaves reasoning and acting in a dynamic feedback loop, where each tool return can alter the subsequent reasoning trajectory in ways that cannot be pre-specified, and errors at any single step can cascade irreversibly.

2. Coordination Overhead in Multi-Agent Systems: Distributing execution across specialized agents multiplies LLM calls, introduces channels susceptible to semantic drift, and creates compounding error propagation at inter-agent boundaries.

3. Non-Stationarity of the Execution Environment: APIs return stale or inconsistent results, retrieved documents contradict one another, and tool invocations fail intermittently, invalidating prior assumptions mid-run.

4. Opaque Failure Attribution: Execution failures are heterogeneous in origin, yet pinpointing the root cause to trigger targeted recovery remains an introspective burden under tight time and resource constraints.

6.1. From Challenges to Research Objectives

Challenges 1 and 2 concern coherence and scalability during routine execution, whereas Challenges 3 and 4 address robustness when runtime conditions deviate from the original plan. Together, these challenges motivate three research objectives, as summarized in Figure 6. Within each, sub-tasks progress sequentially from structural decisions to operational refinements. Across objectives, two critical dependencies link execution quality to runtime monitoring: Adaptive Execution’s recovery signals redirect Workflow Execution’s live trajectory, while its failure diagnostics constrain the overhead decisions of Agent Coordination.

• Workflow Execution (addressing Challenge 1): Operationalizing plans through inference-action loops, external operation invocation, and incremental decomposition, while managing error propagation across interleaved steps.
• Agent Coordination (addressing Challenge 2): Organizing multiple LLM-powered agents, mediating their communication, synchronizing shared state, and reconciling conflicting conclusions while controlling coordination cost.
• Adaptive Execution (addressing Challenges 3 and 4): Monitoring intermediate outputs at runtime, detecting deviations from expected behavior, and triggering targeted plan revision or failure recovery.

6.2. Representative Methods Under Each Research Objective

For these objectives, we review the evolution of typical methods and further extract the core insights that underpin them.

6.2.1. Workflow Execution

Workflow execution operationalizes plans by balancing dynamic problem-solving with efficiency, ensuring premium LLM computation is allocated only where necessary. Systems achieve this by interleaving reasoning with environmental feedback (Inference-Action Loops), offloading deterministic tasks to specialized interpreters (External Operation Invocation), and decomposing goals into modular sub-tasks for localized recovery (Incremental Execution).

• Inference-Action Loops interleave LLM-generated reasoning traces with environment-facing actions, allowing dynamic adaptation rather than following a rigid pre-committed plan. ReAct [3] establishes the canonical form, but its cost is proportional to trajectory length since every step invokes the LLM. WebAgent [113] addresses this by routing routine decisions to a fine-tuned domain model and reserving the general LLM for program synthesis. ToolChain* [114] addresses a different limitation – greedy step-by-step execution’s susceptibility to local optima – by reformulating tool use as A* search over a decision tree. Together, these studies form a progression: ReAct establishes the paradigm; WebAgent reduces per-step cost through specialization; ToolChain* adds lookahead to avoid local optima.
• External Operation Invocation shifts the LLM’s role from acting to code authoring, delegating execution to deterministic components. The loop structure established by inference-action paradigms directly determines which operations are candidates for externalization. Chain of Code [115] delegates well-defined operations to a standard interpreter and invokes an LLM-powered emulator only for semantically complex sub-tasks. VOYAGER [116] addresses the orthogonal problem of cost across repeated executions by accumulating successfully acquired behaviors as an indexed skill library. These two studies target different cost sources – within-trace versus across-trace – and are complementary rather than competing.
• Incremental Execution decomposes complex queries into ordered subtasks, enabling localized error detection and partial substitution with non-LLM components. The granularity of external operations directly governs how finely this decomposition can be structured. Successive Prompting [117] establishes the core formulation with separate LLM calls for decomposition and answering, though the decomposition strategy remains hand-designed. Unify [21] reduces this dependency by matching queries to a library of semantic operators via embedding retrieval. Tool Zero [118] eliminates demonstration dependency entirely through pure reinforcement learning with a progressive reward schedule. The progression – from hand-designed prompts to programmatic operator matching to learned behavior – reflects a consistent move toward reducing reliance on human-specified decomposition strategies.

Key Insight. The core challenge is not choosing the right sequence of steps, but deciding how much reasoning to invest at each step and which component should handle it. Greedy inference-action loops fail not because their steps are wrong, but because they apply uniform LLM reasoning to both semantically complex decisions and routine operations. The research trajectory consistently moves toward narrowing the LLM’s scope to decisions where its flexibility is genuinely necessary, delegating the rest to deterministic or specialized components.

6.2.2. Agent Coordination

Agent coordination scales collective problem-solving while strictly controlling communication overhead. By adaptively managing structural complexity, systems first establish organizational topologies for role assignment and information flow (Coordination Structures), and subsequently implement the interaction protocols required for agents to exchange results and converge on shared solutions (Collaboration Mechanisms).

• Coordination Structures define the organizational topology governing role assignment and information flow. MetaGPT [20], AutoGen [119], and ChatDev [120] represent three points on the rigidity–flexibility spectrum. MetaGPT encodes standardized operating procedures with fixed roles and structured artifact handoffs. AutoGen adopts flexible conversation programming, allowing non-LLM components as first-class agents and configurable interaction topologies. ChatDev sits between these, relying on role-consistent behavior through prompting rather than pipeline engineering. The appropriate choice depends on task stability: rigid structures amortize well over homogeneous tasks; flexible topologies handle diversity at higher per-task overhead. Riedl [121] further shows that useful coordination can emerge from persona assignment and theory-of-mind prompting without explicit structure, raising the question of when the engineering cost of designed pipelines is justified.
• Collaboration Mechanisms concern the protocols through which agents exchange intermediate results and converge toward shared solutions – with topology serving as the structural substrate over which these protocols operate. Multi-agent debate [122] targets output quality through iterative critique, most effective when individual outputs are unreliable. CAMEL [123] targets design effort via role-based inception prompting, providing a lightweight baseline at the cost of a static conversation pattern. Liu et al. [124] target computational efficiency by caching prior outputs and pre-fetching predictable context queries. Multi-agent reinforcement learning [125] targets adaptability by enabling agents to jointly learn communication and task allocation policies.

Key Insight. The benefit of multi-agent coordination is real but conditional. Distributing work is most valuable when subtasks are genuinely heterogeneous; for homogeneous or well-structured tasks, coordination overhead can outweigh the gains. Similarly, iterative debate and critique improve quality precisely when individual agents disagree, but add little when the underlying model is already reliable. A practical strategy is to start with the simplest coordination structure the task demands, introducing additional agents or communication rounds only where single-agent performance is demonstrably insufficient.

6.2.3. Adaptive Execution

Adaptive execution ensures end-to-end reliability by transforming rigid trajectories into self-correcting processes. Bounded by initial diagnostic quality, systems manage anomalies through a sequential pipeline: generating objective signals to evaluate intermediate outputs (Execution Monitoring), revising high-level strategies when assumptions fail (Dynamic Replanning), and executing corrective protocols to navigate localized faults (Failure Handling).

• Execution Monitoring detects errors in intermediate outputs, establishing the diagnostic signal upon which replanning and recovery depend. LLM-AUGMENTER [126] externalizes verification by validating candidate responses against retrieved evidence. AgentPro [127] internalizes it through a trained Process Reward Model that applies rejection sampling to steer inference away from erroneous paths. Both share the insight that the LLM should not be its own judge, but differ in whether the monitoring signal comes from the external world or a separately trained model.
• Dynamic Replanning revises strategy mid-execution when monitoring signals indicate ineffectiveness. ExpeL [128] distills cross-task successes and failures into natural language insights that guide experience-informed revisions without additional inference-time cost. DEPS [129] delegates feasibility judgments to a lightweight trained selector, reserving the LLM for plan revision reasoning proper.
• Failure Handling detects, diagnoses, and recovers from execution-time errors, with the scope of recovery actions directly shaped by the specificity of the replanning strategy. Reflexion [19] generates natural language reflections on failed trajectories for use as context in subsequent attempts, turning failures into self-improvement signals without weight updates. CRITIC [39] grounds failure diagnosis in external tool feedback – search results or code execution outputs – providing a more reliable signal. FireAct [130] shifts failure handling to training time by fine-tuning on diverse agent trajectories, so inference-time recovery requires no additional LLM calls. The three represent a progression from inference-time self-reflection, to inference-time external grounding, to training-time internalization.

Key Insight. Monitoring, replanning, and failure handling each depend on a common prerequisite: a reliable signal for distinguishing expected from unexpected behavior. When that signal comes from the LLM itself, the result is circular and unreliable. The more effective pattern is to externalize the signal – through retrieved evidence, trained discriminators, or fine-tuning on diverse trajectories. Adaptive execution is thus not primarily a reasoning problem but a measurement problem: its effectiveness depends not only on the sophistication of the replanning logic, but also on the reliability of the monitoring signals that trigger adaptation.

7. Synthesis & Verification

Synthesis & Verification serves as the pipeline’s terminal interface – the only stage where output is directly exposed to the end user. It synthesizes accumulated evidence and reasoning into a coherent, grounded, and trustworthy response, uniquely reconciling narrative completeness with factual reliability. While answer synthesis is increasingly supported by retrieval-augmented generation, uncertainty modeling, verification, and iterative refinement remain less systematically developed. As a result, fluent responses may still contain unsupported claims or fail to communicate uncertainty reliably. We organize these limitations into four closely related challenges:

1. Evidence Aggregation and Coherence Maintenance: Fusing disparate, often contradictory, signals from heterogeneous sources while satisfying strict structural and attribution constraints.

2. Confidence Calibration and Uncertainty Communication: Accurately estimating and communicating the reliability of answers produced through multi-step reasoning, since fluent responses may still contain unsupported or incorrect claims.

3. The Verification Dilemma: Navigating shared biases between generators and verifiers, compounded by the difficulty of decomposing complex analytical answers into independently verifiable claims.

4. Refinement Convergence and Cross-Level Correction: Identifying and rectifying errors that originate in upstream stages, as output-level self-critique often fails to address systemic pipeline failures.

7.1. From Challenges to Research Objectives

These challenges can undermine system reliability: Challenge 1 affects response coherence, Challenge 2 limits transparent communication of uncertainty, and Challenges 3 and 4 directly affect factual correctness. Together, they motivate the four research objectives depicted in Figure 7:

• Answer Synthesis (addressing Challenge 1): Fusing heterogeneous evidence into coherent, multi-faceted answers while maintaining strict attribution and structural integrity.
• Confidence & Uncertainty Modeling (addressing Challenge 2): Estimating and calibrating model confidence to enable principled abstention and transparent epistemic communication.
• Verification & Validation (addressing Challenge 3): Detecting hallucinations and reasoning flaws across claim, step, and chain levels through internal consistency and external grounding.
• Iterative Refinement (addressing Challenge 4): Developing targeted correction mechanisms that address error sources at the appropriate abstraction level rather than defaulting to blanket regeneration.

7.2. Representative Methods Under Each Research Objective

In the following, we review representative studies under each objective and synthesize the key design principles they reveal.

7.2.1. Answer Synthesis

Answer synthesis transforms collected evidence into a coherent response while balancing readability with factual support. It typically involves three steps: integrating retrieved evidence into a unified representation (Evidence Aggregation), generating a fluent answer grounded in that evidence (Answer Generation), and ensuring that the final response satisfies structural, formatting, and citation requirements (Response Formatting).

• Evidence Aggregation combines disparate signals to ground generation. While foundational architectures like RAG [4] and Fusion-in-Decoder (FiD) [131] establish effective paradigms for fusing retrieved passages via cross-attention, they typically treat aggregation as a single-pass operation, which can lead to content conflation when processing massive source lists. To mitigate this, SeeKeR [132] decomposes aggregation into sequential modules (search, summarization, and generation), showing that explicit intermediate knowledge representations can reduce cross-document hallucination, albeit at the cost of additional inference steps.
• Answer Generation bridges these aggregated contexts with natural language, balancing factual groundedness and instruction adherence. Moving beyond standard autoregressive conditioning [133,134], models aligned via RLHF (e.g., InstructGPT [135]) demonstrate enhanced instruction-following for user-facing tasks. Furthermore, for highly quantitative queries, MatPlotAgent [136] highlights a code-augmented paradigm where the LLM generates verifiable executable scripts rather than direct text, externalizing computation to ensure reproducibility.
• Response Formatting ensures the generated output complies with citation norms and schema specifications. As Bohnet et al. [137] note, even advanced LLMs struggle to achieve perfect attribution natively. DVR [138] addresses this by decomposing complex formatting instructions into individual constraints verified by deterministic tools, replacing unreliable LLM self-assessment where formal rules apply.

Key Insight. A key challenge in answer synthesis is balancing fluency with traceability. End-to-end generation can efficiently produce coherent responses, but it may make it difficult to identify the evidence supporting each claim. Breaking synthesis into separate steps can improve traceability, although it introduces additional computational overhead. For explicit structural requirements, such as length limits or output schemas, deterministic validation tools are often more reliable than LLM self-assessment. Rule-based checks and generative models should therefore be used together.

7.2.2. Confidence & Uncertainty Modeling

Confidence and uncertainty modeling enables systems to estimate the reliability of their outputs and use these estimates to guide downstream decisions. It typically involves three steps: measuring uncertainty in intermediate or final outputs (Uncertainty Estimation), aligning confidence scores with empirical correctness (Confidence Calibration), and using calibrated confidence to determine whether the system should answer or abstain from answering (Abstention & Refusal).

• Uncertainty Estimation quantifies model confidence. While early methods successfully elicited natural-language confidence without logit access [139,140], they primarily measured uncertainty over token sequences. Kuhn et al. [141] address this by introducing semantic entropy, which clusters semantically equivalent outputs to measure uncertainty over meanings rather than surface forms, offering a more robust predictor of accuracy at scale.
• Confidence Calibration aligns internal estimates with empirical correctness. Zhao et al. [142] identify that in-context examples can skew output probabilities, proposing contextual calibration to normalize biases. However, Tian et al. [143] reveal that standard RLHF fine-tuning can degrade token-level calibration, complicating uncertainty estimation. For black-box models, APRICOT [144] provides a workaround by training auxiliary models to predict confidence externally.
• Abstention & Refusal translates confidence into action. As SQuAD 2.0 [145] established, detecting unanswerable queries is critical. R-Tuning [146] demonstrates that standard instruction tuning often trains models to guess rather than refuse. By incorporating explicit knowledge boundary identification during training, abstention can be cultivated as a transferable skill rather than a localized classifier.

Key Insight. Uncertainty estimation, calibration, and abstention are highly interdependent. An uncertainty signal that does not inform downstream pipeline behavior (e.g., triggering additional retrieval or refusal) remains merely diagnostic. Reliable uncertainty handling requires calibrated confidence signals to be integrated into the system’s control flow, allowing the system to adapt its behavior or abstain from answering when confidence is low.

7.2.3. Verification & Validation

Verification and validation balance computational overhead with evaluation depth by matching strategies to dominant error profiles. Systems deploy validation across an escalating hierarchy: running lightweight internal checks for consistency (Hallucination Detection), verifying atomic statements against external evidence (Claim-level Verification), evaluating intermediate logical soundness (Step-level Verification), and assessing the holistic quality of the response trajectory (Chain-level Evaluation).

• Hallucination Detection often serves as a lightweight first pass. SelfCheckGPT [147] and ITI [148] enable zero-resource detection by exploiting output consistency across stochastic samples or shifting internal attention head activations, avoiding external database overhead. ROWEN [149] further utilizes these internal signals to trigger external retrieval adaptively.
• Claim-level Verification grounds flagged content externally. Recognizing that response-level judgments obscure specific factual errors, FActScore [150] and Verify-and-Edit [151] decompose outputs into atomic claims, enabling targeted external retrieval that corrects factual discrepancies without rewriting the entire chain.
• Step-level Verification addresses intermediate reasoning logic. Lightman et al. [152] show that process supervision (human feedback per step) yields more reliable reward models than outcome supervision. To scale this, MATH-Shepherd [153] and DS-Agent [154] automate process supervision via Monte Carlo trajectory sampling or implicit execution feedback, bypassing expensive human annotation.
• Chain-level Evaluation assesses holistic properties. The LLM-as-a-Judge paradigm [155,156] can approximate human preferences for open-ended criteria, provided self-enhancement biases are mitigated. Salinas et al. [157] indicate that proper judge configuration and prompt design can yield evaluation quality comparable to scaling up the model itself.

Key Insight. Verification methods address largely orthogonal failure modes: consistency checking detects internal incoherence, retrieval grounding targets external contradictions, and process supervision identifies logical flaws. Consequently, selecting a verification strategy depends heavily on the dominant error profile of the task, and robust systems often combine these methods to achieve comprehensive validation.

7.2.4. Iterative Refinement

Iterative refinement balances correction capability, external grounding, and inference latency to elevate final output quality. Systems achieve this via three paradigms: leveraging internal capacity to critique drafts (Self-refinement & Critique), integrating tool feedback to resolve factual errors (Feedback-based Correction), and distilling refinement behaviors into model weights to eliminate runtime overhead (Preference Learning). Modern architectures increasingly prioritize tracing these errors to upstream origins over relying solely on surface-level rewriting.

• Self-refinement & Critique improves outputs intrinsically. Frameworks like SELF-REFINE [38], Reflexion [19], and Constitutional AI [158] demonstrate that LLMs can iteratively critique and refine their own outputs or align with principles using episodic memory or self-generated feedback. However, these methods are naturally bounded by the model’s parametric knowledge and often struggle to correct severe factual errors.
• Feedback-based Correction introduces external grounding. CRITIC [39] interfaces LLMs with external tools (code interpreters, search engines) to provide objective verification signals. To minimize the overhead of constant external calls, Lin et al. [159] utilize internal consistency metrics to trigger external refinement selectively.
• Preference Learning internalizes correction signals during training to eliminate inference-time overhead. Methods like DPO [160] and RLHF-V [161] optimize models directly from preference pairs, while SCRPO [162] distills inference-time self-refinement logic directly into model weights, often yielding better faithfulness without per-query latency penalties.

Key Insight. Intrinsic self-refinement is computationally accessible but bounded by internal knowledge. Tool-grounded correction extends this reach to external facts but incurs operational latency. Preference learning effectively amortizes these costs into the training phase. However, a persistent challenge remains that many synthesis-stage errors originate upstream (e.g., poor retrieval or flawed planning), suggesting that output-level rewriting should ideally be replaced by stage-aware error localization and targeted upstream re-execution.

8. Open Problems and Research Agenda

The cross-stage findings in Section 2 and stage-wise analysis in Section 3, Section 4, Section 5, Section 6 and Section 7 reveal that the most critical limitations of current LLM-powered QA systems are deeply rooted in structural problems. These overarching challenges manifest in how stages are evaluated, how they interact, how resources are allocated, how intermediate reasoning is verified, and how pipelines adapt to environmental shifts. We organize the discussion around five directions that address these key issues.

8.1. Stage-Aware Evaluation and Diagnostic Infrastructure

Current evaluation practices mostly measure final-answer quality, which obscures why or where a pipeline fails. Two systems with similar end metrics often rely on entirely different internal mechanisms, yet current benchmarks rarely compare them at the stage level. Developing a diagnostic infrastructure that isolates stage performance while respecting inter-stage dependencies represents a foundational research need.

Evaluating these pipelines effectively requires establishing stage-specific evaluation protocols and diagnostic datasets. Rather than prescribing a fixed set of metrics, future research can explore diverse evaluation dimensions tailored to each stage. For instance, potential benchmarks might incorporate annotated corpora to assess meaning preservation in query rewriting, labeled dependency graphs for logical planning, or fine-grained factual verification data for synthesis. Beyond static datasets, a promising approach is building controlled testbeds that inject specific stage-level faults. This allows researchers to systematically analyze how errors propagate through the pipeline and evaluate the system’s structural resilience.

8.2. Closed-Loop Feedback and Upstream Control

While individual stages like query rewriting and planning have advanced, the connections between them remain largely open-loop and unidirectional. Information typically flows forward from entry point to synthesis, leaving the system vulnerable to early-stage errors. For instance, even a well-designed query rewriter operates blind to downstream constraints; if later stages encounter empty retrieval results or execution failures, the system rarely possesses the feedback mechanism to return upstream and refine the initial query interpretation.

A key research priority is replacing these rigid, one-way interfaces with closed-loop communication pipelines. Downstream execution signals – such as high retrieval latency, low plan confidence, or verification contradictions – should propagate backward to trigger adaptive replanning or query refinement. Transitioning from fixed rule-based routing to learned coordination interfaces represents a critical step toward enabling pipelines to self-correct during runtime.

8.3. Cost-Aware Physical Planning and Global Optimization

QA pipelines incur substantial and unevenly distributed resource consumption: complex rewriting multiplies retrieval calls, elaborate planning branches out reasoning paths, and deep verification demands extra generation tokens. Currently, systems manage these expenditures through rigid budgets or heuristics, failing to treat efficiency and quality as a joint optimization problem.

A valuable direction is the development of formal cost models that estimate latency, token consumption, and monetary expenses for alternative pipeline configurations. These models can allow the physical planner to identify execution strategies that optimally balance efficiency and answer quality. Crucially, this resource management is expected to be dynamic: simple queries follow a lightweight, low-cost path, while genuinely ambiguous or complex questions automatically trigger more extensive processing. Managing resources adaptively based on real-time execution feedback represents a key step toward making production-scale QA economically viable.

8.4. Faithful Reasoning and Verifiable Execution

As pipelines become more complex, ensuring the reliability of intermediate states becomes as important as optimizing their efficiency. Current architectures often treat chain-of-thought traces, logical plans, and tool-call sequences as transparent records of actual computation. However, LLMs can generate these intermediate outputs post-hoc, creating a plausible and well-structured narrative that conceals underlying reasoning errors or factual shortcuts.

If verification modules only inspect these superficial narratives, their assessments lack grounding in the actual execution process. This challenge motivates the development of methods and metrics that evaluate the faithfulness of intermediate outputs. Ensuring that a system’s internal explanations strictly align with its actual data-processing steps is central to building trustworthy, auditable QA pipelines.

8.5. Modular Compositionality and System Generality

Most contemporary QA systems are tightly coupled and engineered for a fixed setting – a specific corpus, a set of tools, or a particular language model. When data sources evolve, APIs update, or tasks span new modalities, adapting the system typically requires substantial manual re-engineering, which prevents the accumulation of reusable software components.

Borrowing successful principles from database systems, a long-term research goal is achieving modular compositionality through standardized stage interfaces. By defining clear abstractions for operators like query rewriting, planning, and execution, researchers can design generic, swappable modules. This architectural decoupling is expected to significantly improve system generality, allowing QA pipelines to easily integrate new models and seamlessly scale to heterogeneous, multi-modal environments.

Taken together, these five directions offer a broader perspective. Building LLM-powered QA as an optimizable, engineered pipeline requires more than isolated component-level improvements. Instead, it calls for transparent stage-level evaluation, dynamic feedback-driven control, global resource optimization, faithful intermediate reasoning, and modular, adaptive system architectures. These capabilities provide a principled and practical roadmap for next-generation question answering systems.

9. Conclusion

In this survey, we have proposed a paradigm shift in understanding LLM-powered question answering: moving away from monolithic generation toward a structured, five-stage analytical pipeline inspired by the principles of database query processing. By decomposing the end-to-end pipeline into Query Rewriting, Logical Planning, Physical Planning, Execution & Adaptation, and Synthesis & Verification, we established a systematic framework for examining the field. This pipeline-oriented lens is not merely taxonomic; it renders comparable a vast body of research otherwise scattered across disparate application domains, data modalities, and model architectures, while making explicit the critical decision points embedded within modern QA design.

Transitioning these systems from heuristic implementations to engineered, optimizable architectures requires addressing fundamental structural limitations. As outlined in our research agenda, future progress hinges on five strategic pillars. First, the field must shift from coarse end-to-end metrics toward granular, stage-aware diagnostic evaluation. Second, rigid, unidirectional workflows must evolve into closed-loop, bidirectional pipelines capable of adaptive upstream refinement. Third, resource management should be treated as a global optimization problem guided by cost-aware physical planning. Fourth, ensuring system trustworthiness demands verifiable, faithful reasoning where internal explanations strictly align with actual execution steps. Finally, long-term adaptability requires embracing modular compositionality through standardized stage interfaces.

We hope this survey serves as a durable foundation for the continued evolution of LLM-powered QA. By reframing these systems through an engineered, pipeline-oriented lens, we aim to guide future research toward architectures that are not only more capable, but fundamentally more systematic, transparent, and robust.