Agent and Tool Search: Foundations, Techniques, and Open Challenges

1. Introduction

The rapid advancement of Artificial Intelligence (AI) has led to the emergence of AI agents and tools that can autonomously or semi-autonomously perform complex tasks across diverse application domains (Fang et al. 2025a; Gao et al. 2026; Li et al. 2025h; Luo et al. 2025; Qu et al. 2025b; Sun et al. 2025; Wang et al. 2024a; Yu et al. 2025; Zhang et al. 2025f). These systems are no longer limited to passive information processing, but instead actively perceive inputs, reason over intermediate states, call external tools, interact with environments, and execute actions to achieve user-specified goals. This shift is increasingly visible in coding agents, deep research agents, tool-augmented assistants, workflow automation systems, and multi-agent platforms. As a result, the digital ecosystem is undergoing a fundamental transition from information-centric services toward task-oriented, executable systems, leading to a growing proliferation of agents and tools with overlapping or complementary functionalities (Wu et al. 2026a,b,c).

This proliferation gives rise to a fundamental challenge: how to effectively identify, compare, and select the most suitable agent or tool for a given task. From the user perspective, individuals are increasingly faced with many available agents that may appear functionally similar, but differ in competence, reliability, cost, latency, safety, and domain fit. From the system perspective, this challenge also arises in multi-agent environments, where orchestrating agents or platforms must dynamically discover and select appropriate agents and tools as components for downstream task execution (Fourney et al. 2024; Hu et al. 2025b). The challenge is further amplified in open and evolving ecosystems, where agents and tools are developed and deployed by different organizations or service providers, often competing to offer similar functionality while differing in quality, availability, and trustworthiness. As a result, the candidate space becomes highly heterogeneous and difficult to organize, reinforcing the need for principled methods to discover, evaluate, and select agents under realistic constraints (Kim et al. 2026b; Wu et al. 2026a,b).

The need for agent search can also be understood through the evolution of AI systems. Early LLM-based systems were typically accessed as individual models or assistants, whereas contemporary agentic environments increasingly expose collections of tools, specialized agents, and multi-agent workflows that may be selected or composed for different tasks. As this progression moves from standalone large language models to tool-augmented agents, multi-agent systems, and open agent ecosystems, the candidate search space expands in scale, diversity, and specialization. This expansion makes discovery and selection more difficult, and motivates agent search as a distinct problem concerned with finding executable candidates that are suitable for task completion.

Agent and tool search is closely related to several existing research areas, but differs from them in important ways. While the above challenges resemble classical Information Retrieval (IR) problems, traditional retrieval settings primarily aim to identify and rank static information artifacts, such as documents or web pages, based on their relevance to a user’s query. In contrast, agent search operates over executable systems whose utility is determined by their ability to successfully accomplish tasks. Queries are therefore more naturally interpreted as task specifications, and the matching process involves assessing the alignment between task requirements and agent capabilities, rather than estimating semantic relevance alone. This shift also changes the notion of relevance itself, which becomes grounded in factors such as competence, reliability, and expected execution outcomes. Consequently, agent search extends retrieval from information matching to decision-making over executable systems, introducing new challenges in capability representation, task–agent alignment, and evaluation methodologies that account for downstream task performance (Wu et al. 2026a,b). Agent search is also related to research on multi-agent systems and orchestration, where the emphasis is on coordinating multiple agents to complete complex tasks (Fourney et al. 2024; Hu et al. 2025b). However, such approaches typically assume that candidate agents are already available and focus on how to organize their interactions, whereas agent search addresses the upstream problem of identifying and selecting appropriate agents in the first place.

Despite the growing body of work related to agent and tool discovery, retrieval, ranking, and evaluation, the literature remains fragmented across multiple research directions. Existing surveys have examined tool learning and tool use (Qu et al. 2025b), while broader surveys of AI agents provide general perspectives on agent architectures, capabilities, and applications (Luo et al. 2025; Wang et al. 2024a). Other surveys focus on individual components relevant to agent search, such as task planning (Huang et al. 2024b; Wei et al. 2025a), multi-agent coordination and workflows (Sun et al. 2025; Yu et al. 2025), and evaluation (Gu et al. 2024; You et al. 2026). While these surveys provide comprehensive perspectives on their respective areas, the problem of discovering, comparing, and selecting agents and tools has largely been studied through separate components rather than as an integrated search problem. Relevant contributions are consequently dispersed across information retrieval, multi-agent systems, tool learning, agent evaluation, and AI systems research, with different assumptions, problem formulations, and evaluation criteria. This fragmentation motivates a unified perspective that connects these components across the end-to-end process of agent and tool search.

In this survey, we present a systematic overview of agent and tool search, aiming to unify existing research under a common perspective. Our primary focus is on the discovery, representation, retrieval, and selection of agents and tools for accomplishing a given task, while also incorporating research from adjacent areas when it directly informs these processes. We begin by introducing the foundations of AI agents, tools, and executable capabilities (Section 2), and then formalize agent search as a retrieval and decision-making problem over executable systems (Section 3). Following the taxonomy developed in our conceptual framework, we organize the literature around task understanding (Section 4); agent and tool discovery, representation, and indexing (Section 5); retrieval, reranking, and selection (Section 6); and evaluation (Section 7). We further examine system-level dimensions that shape agent search, including safety, fairness, security, personalization, and transparency (Section 8), and discuss representative applications in healthcare, law, and deep research (Section 9). Finally, we synthesize the literature to identify key open challenges and promising directions for future research (Section 10).

2. Foundation of AI Agent Systems

2.1. AI Agents, Tools, and Executable Capability

AI agents are computational entities designed to autonomously or semi-autonomously accomplish tasks by perceiving inputs, reasoning over intermediate states, and executing actions in an environment (Luo et al. 2025; Wang et al. 2024a). Compared with traditional software components that perform fixed procedures, modern AI agents typically exhibit adaptive decision-making, planning capability, and the ability to interact with users, external tools, or other agents. Their effectiveness is therefore not solely determined by static descriptions or specifications, but emerges through execution and interaction during task solving. In this sense, AI agents can be viewed as executable problem solvers, whose capabilities are revealed dynamically as they process task requirements and generate outcomes.

In contrast, tools refer to callable functional components that implement specific operations, such as retrieval, data transformation, symbolic reasoning, or API invocation (Gou et al. 2024; Qin et al. 2024a,b; Qu et al. 2025b). Tools are generally designed with narrower functional scope and more clearly defined input–output behaviors than autonomous agents, and often lack independent goal formulation or long-horizon planning abilities. Nevertheless, tools play a critical role as building blocks within agentic systems, enabling agents to extend their operational capabilities by invoking specialized functions. From a system perspective, both agents and tools represent executable units that can be selected and composed to accomplish complex tasks, although they differ in levels of autonomy, reasoning complexity, and interaction patterns.

In practice, the boundary between agents and tools is increasingly blurred as AI systems evolve toward greater autonomy and compositionality. Some tools incorporate reasoning or limited planning capabilities, while many agents are constructed as orchestrations of multiple tool invocations (Qin et al. 2024a; Qu et al. 2025b). As a result, executable systems vary along multiple dimensions, including autonomy, functional scope, interaction capability, and decision complexity. To accommodate this heterogeneity, it is useful to adopt a unified abstraction that views both agents and tools as executable capability-bearing systems. Under this abstraction, the primary concern is not the internal architecture of a system, but the set of tasks it can successfully accomplish when invoked.

These executable systems typically expose their capabilities through various forms of representation prior to execution, including natural language descriptions, structured schemas, demonstrations, learned embeddings, or historical performance records (Lumer et al. 2025; Mulang et al. 2026; Shi et al. 2025d). These representations serve as proxies that allow users or higher-level controllers to estimate whether a system is suitable for a given task. However, such signals are often incomplete, noisy, or context-dependent, since true capability can only be fully observed through interaction and task execution. This inherent uncertainty makes it difficult to reliably assess and compare candidate systems, especially when the candidate space is large, heterogeneous, or dynamically evolving.

As the number and diversity of available executable systems continue to grow, determining which agents or tools to invoke becomes a non-trivial decision problem. In open ecosystems such as model hubs1, agent marketplaces2, and tool platforms3, users are increasingly required to identify suitable systems from extensive candidate pools. Similarly, within complex agentic workflows, controller components must dynamically select appropriate agents or tools to handle subtasks during execution (Fourney et al. 2024; Hu et al. 2025b). These challenges collectively highlight the need for principled mechanisms to discover and select executable systems capable of accomplishing a given task, motivating the study of agent search in large-scale AI environments (Wu et al. 2026b).

2.2. Motivation of Agent Search: User-Side and System-Side Demands

The need for agent search first becomes evident from the user perspective, as the rapid proliferation of AI agents and tools makes manual discovery increasingly difficult. Modern platforms and model hubs provide access to large collections of executable systems with diverse capabilities and interfaces (Ong et al. 2025; Patil et al. 2024; Qin et al. 2024b). While such diversity expands the range of tasks that can be addressed, it also introduces substantial uncertainty in determining which systems are suitable for a specific objective. Capability descriptions are often incomplete, inconsistent, or context-dependent, and actual performance may only be revealed through execution (Qu et al. 2025a; Wu et al. 2025a; Yuan et al. 2024c). Consequently, users are frequently required to rely on ad hoc exploration or trial-and-error invocation, resulting in increased decision cost and inefficiency (Balog et al. 2012; McDonald and Ackerman 1998). Under these conditions, selecting appropriate agents or tools naturally becomes a search problem that involves reasoning over uncertain capability signals and task requirements.

Beyond user-driven discovery, similar challenges arise from the system perspective, particularly within complex agentic workflows. Contemporary AI systems increasingly adopt modular architectures in which tasks are decomposed into subtasks handled by specialized agents or tools (Fourney et al. 2024; Hu et al. 2025b; Qin et al. 2024b). Controller or orchestration components must therefore dynamically determine which executable systems to invoke at different steps of task execution. This selection process is complicated by overlapping capabilities, evolving task contexts, and constraints such as latency, reliability, or resource consumption. As candidate spaces grow in size and diversity, naive selection strategies become insufficient, and overall system performance depends critically on the ability to identify suitable executors in a principled and scalable manner.

These challenges become even more pronounced in open and evolving agent ecosystems, where executable systems are developed, deployed, and updated in a decentralized manner (Fang et al. 2025a; Gao et al. 2026). In such environments, candidate spaces are not only large but also continuously changing, and interactions may occur among independently designed agents with heterogeneous assumptions and interfaces. Both users and agentic systems must therefore operate under heightened uncertainty when discovering and selecting appropriate capabilities. This transition from closed pipelines to open agent economies amplifies the complexity of selection decisions, making the ability to efficiently identify suitable executable systems a fundamental requirement. Motivated by these pressures, we formalize the problem of agent search in the next section.

3. A Conceptual Framework of Agent Search

In this section, we officially define the conceptual framework of agent search, covering the problem definition, system overview, system inputs and outputs.

3.1. Problem Definition

Agent search concerns the problem of identifying agents or tools capable of accomplishing a given task within an ecosystem of available systems. Unlike traditional information retrieval, where the search targets are static information objects such as documents, the targets in agent search are executable systems whose capabilities are revealed through reasoning, interaction, and task execution. Consequently, the objective is not merely to retrieve relevant descriptions, but to determine which agents or tools are most suitable for successfully completing a task.

At a high level, the agent search problem can be characterized by three fundamental elements: task specification, candidate agent space, and search outcome:

• Task Specification. The task specification describes the objective that needs to be accomplished. It defines the problem context that guides the search process, capturing the user’s goal or the system’s operational requirement.
• Candidate Agent Space. The candidate agent space refers to the set of available executable systems that may potentially accomplish the task. These candidates may include tools, agents, or other autonomous systems whose capabilities vary in scope and complexity.
• Search Outcome. The search outcome denotes the result produced by the search process, specifying which agents, tools, or composed solutions should be used or considered for addressing the task.

Figure 1 presents our conceptual framework of agent search, illustrating how task specifications, heterogeneous candidate spaces, capability representations, and search outcomes interact within an end-to-end search pipeline. We use this framework throughout the remainder of the survey to organize the discussion of different research directions. This formulation intentionally abstracts away from specific system implementations in order to capture the diverse settings in which agent search arises. In the following, we further elaborate how these elements are instantiated in practical systems, including the overall system workflow as well as the concrete forms of system inputs, candidate agents, and outputs.

3.2. Key Components

While Section 3.1 formulates agent search abstractly through task specification, candidate agent space, and search outcome, these components can each exhibit substantial variation across application settings. Such variation fundamentally shapes the difficulty, design, and behavior of agent search systems. To provide a more structured characterization of the problem space, we examine the dimensions along which task specifications, candidate agent spaces, and search outcomes may vary. This perspective complements the system-level abstraction introduced in Section 3.3 by clarifying how variation in these three components gives rise to distinct agent search scenarios.

3.2.1. Task Specification

The task specification in agent search may be characterized along several intrinsic dimensions.

Task Abstraction. One important dimension concerns the level at which a task objective is expressed. In some settings, tasks are specified at a high level of abstraction as user intents or goals, requiring substantial interpretation before suitable agents can be identified. In other settings, inputs may take the form of more concrete operational tasks, or even executable task specifications that can be more directly mapped to agent or tool operations. Differences in task abstraction influence how agent search systems perform task understanding, capability matching, and decomposition.

Task Observability. A second dimension concerns the extent to which requirements relevant for agent selection are explicitly available. Some tasks are fully observable, with clear objectives and constraints provided at search time. Others may be only partially observable, where important requirements are ambiguous, incomplete, or revealed only through clarification. In recommendation-oriented settings (Koren et al. 2022; Mao et al. 2025; Shi et al. 2025b; Zhang et al. 2024f), task needs may even be latent and must be inferred from contextual signals or user preferences. Variations in task observability introduce different forms of uncertainty into the search process and often affect whether interaction or inference is needed prior to agent selection.

Task Scope. A third dimension concerns the structural complexity of the task itself. Some tasks are atomic and may be addressed by a single specialized agent or tool, while others involve multiple interdependent subtasks requiring decomposition or coordinated execution. At a larger scope, tasks may resemble workflow-level objectives whose fulfillment may require compositions of agents rather than isolated selections. Such differences influence whether agent search targets individual agents, sets of agents, or potentially coordinated agent assemblies.

3.2.2. Candidate Agent Space

The candidate agent space in agent search can be viewed as an executable capability space, which may be characterized along several intrinsic dimensions.

Candidate Granularity. One important dimension concerns the functional scope at which executable systems are treated as retrieval targets. At one end, candidates may consist of fine-grained tools designed to perform specific operations, such as retrieval, computation, or API invocation. At a broader level, candidates may take the form of autonomous agents capable of reasoning, planning, and handling more complex objectives through interaction. At even higher levels of granularity, the search targets may include coordinated multi-agent systems or agent assemblies that function as composite executable units. Differences in candidate granularity influence what constitutes the search object itself, and consequently affect how capabilities are represented, matched, and selected.

Functional Overlap. A second dimension concerns the extent of functional overlap among candidates, namely whether multiple agents or tools provide redundant, substitutable, or partially overlapping functionality for a given task. In some settings, candidates exhibit low overlap and provide relatively specialized capabilities, making selection primarily a matter of identifying appropriate functionality. In other settings, particularly in open and rapidly evolving agent ecosystems, many candidates may potentially address the same task while differing in quality, reliability, efficiency, or cost. Such overlap increases the difficulty of agent search, as the problem extends beyond identifying feasible candidates to differentiating among competing alternatives according to multiple criteria.

Capability Observability. A third dimension concerns the degree to which a candidate’s competence can be assessed prior to execution. In highly observable settings, capabilities may be estimated through structured metadata, interpretable traces, demonstrations, or historical performance signals. In partially observable settings, such signals may be incomplete or noisy, making capability estimation uncertain. In more opaque settings, candidate competence may only be inferred indirectly through empirical interaction outcomes. Variations in capability observability affect how search systems reason about uncertainty and whether selection can rely primarily on prior representations or must incorporate exploratory interaction and feedback.

3.2.3. Search Outcome

The search outcome of agent search is a ranked list of candidate executable solutions, where each returned item represents one or more executable agents identified for a given task. The search outcome may be characterized by the compositional structure of the returned solutions.

Solution Composition. Solution composition refers to the structural organization of each executable solution returned by agent search. Depending on the complexity of the task, an executable solution may consist of a single agent or tool, a collection of complementary agents or tools, or an entire agentic workflow that explicitly specifies both the participating agents and their coordination strategy. Simple tasks can often be accomplished by a single executable unit, whereas complex tasks, such as multi-step tasks or long-horizon tasks, may require multiple agents or tools working collaboratively to complete different subtasks. Consequently, agent search may operate over executable solutions of varying composition, ranging from isolated agents to fully coordinated agentic systems.

3.3. A Pipeline View of Agent Search

Agent search systems operationalize the problem formulation introduced in Section 3.1 by transforming a task specification and a candidate agent space into a selection outcome. While concrete implementations may differ substantially, agent search can be conceptually understood through a pipeline view characterized by two orthogonal dimensions of system design: pipeline stages, which describe the major phases through which candidate spaces are prepared and task-conditioned selection is performed, and capability access levels, which describe what information about candidate agents is available to support those stages. Together, these dimensions provide a unified abstraction for organizing diverse agent search systems beyond particular architectures or implementations.

Pipeline Stages.

One important dimension concerns the major stages through which agent search is performed. A first stage involves offline capability preparation, where the candidate space is prepared for effective search before task-time. This may include constructing representations of agent capabilities (Qin et al. 2024a; Qu et al. 2025b), estimating competence (Wang et al. 2026; Yue et al. 2025b), organizing candidates into searchable structures (Du et al. 2024; Liu et al. 2025; Zheng et al. 2024), and maintaining these representations as candidate ecosystems evolve (Guo et al. 2024; Li et al. 2023b; Xu et al. 2024). The objective of this stage is to make potentially large, heterogeneous, and dynamic agent spaces accessible for efficient retrieval and selection. A second stage involves online task-conditioned selection, where the system produces task-appropriate agent selections at query time. This stage may involve task understanding (Li et al. 2025i; Lu et al. 2023; Song et al. 2023), capability matching (Qin et al. 2024b; Shi et al. 2025d; Zheng et al. 2024), candidate retrieval, and selection decisions over individual agents or composed solutions (Kalra et al. 2025; Kim and Diaz 2026). Depending on the setting, online selection may further involve iterative refinement through feedback obtained from interaction or execution (Li et al. 2025i; Xu et al. 2024; Yao et al. 2022). While conceptually distinct, these stages are closely coupled, as the effectiveness of online selection depends on how capabilities are modelled offline, while feedback from online usage may in turn improve offline preparation (Xu et al. 2024).

Capability Access Levels.

A second dimension concerns the extent of information available about candidate agents for supporting both offline preparation and online selection. At the most limited level, description-level access provides only declarative capability information, such as agent descriptions, metadata, interface specifications, or tool signatures (Qin et al. 2024a; Qu et al. 2025b). At a richer level, behavioral evidence access includes observations of agent behavior, such as input-output demonstrations, empirical evaluations, or benchmark performance signals (Guo et al. 2024; Shi et al. 2025d; Wu et al. 2026b), enabling capability estimation beyond static descriptions. Beyond behavioral evidence, trajectory-level access provides visibility into execution processes themselves, including reasoning traces, intermediate states, or tool-use trajectories (Li et al. 2025i; Xu et al. 2024; Yao et al. 2022), allowing search systems to reason about how candidates accomplish tasks rather than only what outcomes they produce. At the highest level, full candidate access assumes white-box access to agent internals, such as policies, source code, or underlying components (Hu et al. 2025c; Ocker et al. 2024; Yuan et al. 2024a), enabling deeper forms of modeling, analysis, or adaptation. These access levels are orthogonal to the pipeline stages, as different agent search systems may combine different stages of optimization with different levels of candidate access. This perspective highlights that system design in agent search is shaped not only by what decisions are made during search, but also by what information is available to support those decisions.

Figure 2. Evolution of research directions related to agent search from 2023 to 2026. Existing work has progressed from task understanding and planning capabilities, through representation and search mechanisms for tools and agents, toward deployment-oriented concerns such as evaluation, safety, robustness, and domain-specific applications. Together, these developments form the broader research landscape that underpins agent search systems.

Figure 2. Evolution of research directions related to agent search from 2023 to 2026. Existing work has progressed from task understanding and planning capabilities, through representation and search mechanisms for tools and agents, toward deployment-oriented concerns such as evaluation, safety, robustness, and domain-specific applications. Together, these developments form the broader research landscape that underpins agent search systems.

3.4. Agent Search Through the Lens of Retrieval and Distributed AI

Agent search is a new problem, but it is not an unprecedented one. Its defining question, i.e., given a task, identify the systems best able to accomplish it, recur in different forms across several established research traditions. In this subsection, we situate agent search within these traditions by following a single thread: how the object of search has progressively shifted from static content, to information-producing systems, to autonomous agents that act on the world. Tracing this progression clarifies both what agent search inherits from prior work and where it departs from it.

Document Retrieval.

The most familiar starting point is classical document retrieval, where a query is matched against a corpus of static artifacts and relevance is assessed from the topical correspondence between query and content (Reimers and Gurevych 2019; Robertson and Zaragoza 2009; Sparck Jones 1972). Agent search shares the high-level skeleton of this problem—identifying and ranking candidates in response to a query—but differs in what is being ranked. Rather than passive documents whose value lies in the information they contain, the candidates are executable systems whose value lies in the tasks they can accomplish. Queries therefore become task specifications rather than information needs, and relevance shifts from content match to competence: whether a candidate can successfully carry out the task, and how well it does so. This reframing carries retrieval from information matching into task-oriented decision making, and motivates the problems of capability representation, task–agent alignment, and execution-aware evaluation that structure the remainder of this survey.

Distributed Retrieval and Distributed AI.

Agent search draws closely related foundations from both distributed information retrieval (DIR) and distributed AI (DAI), which address a common question from complementary perspectives: given multiple distributed capability providers, how should a system identify the most appropriate one(s) for a task? In DIR, information is distributed across multiple collections or search services, and the central problem is deciding where to search before retrieving individual documents. Research on resource selection and ranking (Dai et al. 2017; Khramtsova et al. 2023), distributed and federated search (Callan et al. 1995; Callan 2002; Diaz et al. 2010), and meta-search (Chen et al. 2001; Glover et al. 1999) therefore studies how a query can be routed to promising resources and how their heterogeneous results can subsequently be aggregated (Arguello 2017; Cormack et al. 2009). Expert finding further strengthens this connection, since the competence of a candidate expert must be inferred from indirect evidence rather than from surface-level matching (Balog et al. 2012). DAI approaches the same problem from the perspective of autonomous systems rather than distributed collections, where heterogeneous agents advertise, discover, and coordinate their capabilities to accomplish delegated tasks (Weiss 1999). Representative paradigms include the blackboard architecture, where specialized knowledge sources advertise their competence and are opportunistically activated according to the current problem state (Nii 1986), and the Contract Net Protocol, where a manager announces subtasks and selects contractors through bidding based on their advertised capabilities (Smith 1980). Together, these ideas establish the conceptual foundations of modern agent search: DIR contributes the retrieval perspective of routing queries to the most appropriate capability providers, while DAI contributes the mechanisms for representing, advertising, and allocating capabilities among autonomous agents. Contemporary agent registries, marketplaces, and tool servers can therefore be viewed as LLM-era realizations that unify these two traditions, performing capability-aware retrieval over large, heterogeneous, and continually evolving agent ecosystems.

Agent Orchestration Systems.

Following DAI that supplies the ancestry of agent search, modern agent orchestration systems represent its immediate downstream sibling. Orchestration is concerned with how an available set of agents or tools should be coordinated to complete a complex task—task decomposition, dependency management, scheduling, and monitoring (Fourney et al. 2024; Hu et al. 2025b)—and typically assumes that the pool of candidate agents is already given. Agent search addresses the upstream question that orchestration presupposes: identifying and selecting the appropriate agents from a large, heterogeneous, and evolving space in the first place. The two are tightly coupled at both ends of the pipeline, since the agent search problem arises both at the user end, where users seek suitable agents for their needs, and at the system end, where an orchestrator must locate executable components for its subtasks. In this sense, orchestration can be viewed as a downstream instantiation of agent search, where the emphasis shifts from selecting agents to structuring their collaboration.

Figure 3. Hierarchical organization of the survey sections and associated works.

Figure 3. Hierarchical organization of the survey sections and associated works.

Across all of these traditions, a single distinction sets agent search apart: its retrieval targets act rather than merely inform. In document retrieval, DIR, and expert finding, the selected artifact, resource, or expert ultimately serves to surface static information, and its relevance can be estimated from evidence gathered offline. In agent search, by contrast, the selected system executes, incurs cost, and may alter external state, so its true competence is only partially observable before execution and its relevance must be grounded in expected task outcomes, reliability, and safety rather than topical similarity alone. Agent search thus inherits the routing and selection machinery of DIR, the advertisement and coordination abstractions of DAI, and the ranking foundations of classical retrieval, while extending all of them toward execution-grounded competence estimation over dynamic and continually evolving agent ecosystems.

4. Task Understanding

In the agent search process, understanding the user’s task is of critical importance, particularly in long-horizon scenarios, where decomposing complex objectives into step-by-step actionable sub-tasks is decisive for subsequent agent search performance (Huang et al. 2024b; Qiao et al. 2023; Wei et al. 2025a; Zhai et al. 2025). Beyond supporting task execution, task understanding serves as the bridge between user intent and downstream agent selection. A clear understanding of the task determines not only what capabilities should be searched for, but also what constraints, dependency structures, and interaction requirements must be satisfied. These requirements influence whether multiple specialized agents are needed, how retrieved agents should be coordinated, and whether additional information must be elicited from the user before retrieval can proceed. Consequently, techniques for task decomposition, planning, and clarification play a fundamental role in shaping the effectiveness of agent search systems.

Therefore, in this section, we provide a systematic review of existing task understanding techniques that are relevant to agent search, focusing on two progressive aspects: strategies for task decomposition (Section 4.1) and methods for enhancing planning capabilities based on these strategies (Section 4.2). Furthermore, we discuss proactive user intent clarification (Section 4.3), which plays a particularly important role in agent search due to the ambiguity and underspecification commonly observed in user requests.

4.1. Task Decomposition

Sequential vs. Structured Planning.

The emergence of Chain-of-Thought (CoT) (Kojima et al. 2022;Wei et al. 2022a,b) marked a pivotal milestone in enabling language models to progressively acquire the capability to decompose complex tasks. Building upon this foundation, ReAct (Yao et al. 2022) introduced agentic behavior to LLMs, where natural language–based function calling, tool usage, and action execution endowed models with the ability to interact with external environments. Its canonical thought–action–observation loop has since become a dominant interaction paradigm for modern agents. Methods exemplified by Reflexion (Shinn et al. 2023) further advanced this line of research by teaching LLMs to perform iterative reflection and refinement (Gou et al. 2024; Huang et al. 2023; Kumar et al. 2024a; Madaan et al. 2023; Paul et al. 2024; Renze and Guven 2024; Sun et al. 2023; Zelikman et al. 2022), enabling agents to revise their actions based on environmental feedback. Subsequent approaches further improve agent planning by enhancing the complexity (Fu et al. 2022; Press et al. 2023; Zhou et al. 2022), diversity (Zhang et al. 2022), and consistency (Li et al. 2023c; Wang et al. 2022; Weng et al. 2023) of reasoning and planning. However, this line of work remains constrained by sequential task decomposition strategies, which expose task requirements incrementally and may limit the system’s ability to reason about the full set of capability requirements, constraints, and dependency relationships before agent selection. These limitations become particularly pronounced in complex agent search scenarios involving multiple specialized agents. Accordingly, another line of research (e.g. ToT (Yao et al. 2023), GoT (Besta et al. 2024)) adopts structured planning approaches, introducing tree-based or graph-based structures to more effectively represent complex task decompositions (Shen et al. 2024; Wu et al. 2025c, 2024; Yang et al. 2025b). For example, some studies model agentic workflows as Directed Acyclic Graphs (DAGs) (Prasad et al. 2024; Qiao et al. 2024a; Schroeder et al. 2025; Yuan et al. 2024b; Zhang et al. 2025k), where subtasks without dependency relationships can be executed in parallel. These representations explicitly capture dependency structures among subtasks, providing richer signals for identifying, selecting, and coordinating multiple specialized agents. While certain approaches rely on human-designed fixed workflows (Hong et al. 2023; Qian et al. 2024a; Wang et al. 2023; Yang et al. 2025a), a growing body of work enables agents to automatically generate workflows and iteratively refine them during the planning process (Hong et al. 2025;Wang et al. 2025g; Zhang et al. 2025b, 2024c). In addition, some methods integrate tree search algorithms (e.g., MCTS) to explore the trajectory space for optimal solutions (Ding et al. 2025; Hao et al. 2023a; Jiang et al. 2025b; Koh et al. 2025; Ou et al. 2025; Zhang et al. 2024b; Zhuang et al. 2024); such approaches are also representative of test-time scaling strategies. These differences have important implications for agent search, as different decomposition strategies expose capability requirements and dependency structures at different levels of granularity.

Global vs. Iterative Planning.

With respect to when subtasks are generated during the planning process, we further categorize task decomposition strategies into global and iterative paradigms. For global task decomposition approaches (Wang et al. 2023;Wu et al. 2025c; Yang et al. 2025a; Yin et al. 2024a; Zhang et al. 2024c), the agent generates all subtasks in a single pass based on the user’s task and subsequently maps each subtask to executable actions in a step-by-step manner. A key limitation of this paradigm is its inability to dynamically revise the initial requirement specification when new information becomes available. Consequently, capability requirements identified during planning may become misaligned with the available agent ecosystem or evolving task constraints. Therefore, it is more suitable for tasks with relatively clear and well-defined workflows. For iterative approaches (Qiao et al. 2025b; Qin et al. 2024b; Sun et al. 2023; Yao et al. 2022; Zhang et al. 2025k; Zhou et al. 2022), the agent generates only one subtask at each step, dynamically determining the next subtask based on the current state and the historical context of previous steps. The flexibility of this paradigm has made it the dominant agent framework (e.g., ReAct). However, such methods are prone to trajectory collapse, as errors may accumulate progressively across planning steps, potentially leading to inaccurate requirement specifications and suboptimal agent selection decisions. Some approaches (Wang et al. 2025g; Zhang et al. 2025b, 2026c) attempt to combine global and iterative planning, where the agent first generates an initial global plan and then dynamically adjusts subsequent global steps based on feedback obtained during subtask execution. However, since the distinction between these approaches and iterative planning is relatively small, they have not become a dominant paradigm.

4.2. Planning Augmentation

Agentic Post-Training.

Accurately identifying capability requirements, constraints, and dependency structures relies heavily on the planning capability of the underlying system. Training therefore represents the most direct approach to improving requirement discovery and task understanding. The initial era of agentic training relied primarily on Supervised Fine-tuning (SFT) of models using trajectory data derived from human annotations or distilled from stronger models (Chen et al. 2023a, 2024d; Hu et al. 2025a; Qiao et al. 2024b; Yin et al. 2024a; Zeng et al. 2024; Zhu et al. 2025c). While such methods are straightforward and effective, they suffer from two significant limitations: first, the training trajectory distribution tends to be overly idealized, rendering the trained models prone to collapse when encountering unexpected situations; second, the annotation of trajectory data entails prohibitively high costs. Subsequent works have incorporated preference training (e.g., DPO (Rafailov et al. 2023), RPO (Pang et al. 2024), etc.) upon the SFT foundation (Hu et al. 2025b; Qiao et al. 2025a; Song et al. 2024), wherein negative samples are drawn from the post-SFT model and paired with gold trajectories to further train the agent model. Although such methods can yield performance improvements beyond the SFT baseline, they fundamentally fail to address the dual challenges of idealized training data and prohibitive costs inherent in SFT. Moreover, the stability of DPO-like algorithms on long-horizon trajectories remains questionable. The emergence of GRPO (Shao et al. 2024) has propelled the application of Reinforcement Learning (RL) in agentic training to its zenith. This approach, which relies solely on final rewards as supervisory signals without requiring intermediate annotations, effectively addresses the limitations inherent in prior training methodologies. Although zero-RL (Guo et al. 2025) has demonstrated efficacy in certain agent scenarios (Jin et al. 2025; Qian et al. 2025; Wei et al. 2025c), contemporary agent models remain incapable of handling most complex task environments, making it difficult to rollout high-quality trajectories for training. Consequently, the majority of works (Feng et al. 2025; Li et al. 2025i; Qiao et al. 2025b; Wu et al. 2025b; Zhang et al. 2025d) adopt SFT as a mid-training stage prior to RL to obtain a superior foundation model for subsequent fine-tuning. Some preliminary efforts have explored the integration of process reward mechanisms into RL training (Chen et al. 2025c; Fei et al. 2025a; Li et al. 2025i; Sullivan 2025). However, satisfactory process reward modeling remains elusive, limiting the ability of current systems to reliably learn the planning and requirement discovery behaviors needed for complex agent search scenarios.

Multi-Agent Framework.

Complex tasks often involve diverse capability requirements, constraints, and dependency structures that may be difficult for a single planning module to model effectively. Consequently, many agent systems distribute planning and decision-making across multiple specialized agents, giving rise to multi-agent frameworks. Contemporary multi-agent frameworks can be taxonomized into centralized and distributed systems based on the role of the planning agent. In centralized architectures, the planning agent assumes a central coordinating position, responsible for orchestrating the invocation of other functionally specialized agents (Hu et al. 2025b; Ruan et al. 2026; Su et al. 2025). This constitutes an agent-as-tool paradigm that separates requirement discovery from capability fulfillment, allowing the planning agent to focus on identifying, organizing, and coordinating specialized capabilities. Distributed frameworks more closely resemble sophisticated ensemble learning, wherein constituent agents maintain essentially equivalent status (Chen et al. 2023c, 2024a; Dang et al. 2025). Variations across different frameworks primarily reside in the mechanisms of inter-agent information transmission and fusion. Such mechanisms enable different agents to contribute complementary perspectives when modeling task requirements, reducing the reliance on a single planner’s understanding of the task. Another category of multi-agent research adopts human-orchestrated workflows, wherein distinct agents assume predefined task-specific roles (Hong et al. 2023; Li et al. 2023a, 2024c; Qian et al. 2024a;Wang et al. 2023; Yang et al. 2025a). As planning in such paradigms is performed by humans rather than agents, we do not elaborate on this approach herein. From the perspective of agent search, multi-agent frameworks can be viewed as systems that distribute requirement discovery and capability matching across multiple specialized entities. Consequently, they provide a natural foundation for studying how diverse capabilities can be identified, selected, and organized to satisfy complex user requirements.

Planning Offloading.

Planning offloading extends the reasoning-as-tool paradigm (Wang et al. 2025a) by treating planning itself as a specialized capability that can be delegated to external systems. Rather than assuming that planning must be performed by the agent itself, planning offloading enables agents to leverage more powerful or specialized planning modules when complex reasoning is required. For instance, in the small-large model collaboration scenario, smaller models may delegate complex planning components of a task to a stronger model (Chen et al. 2023b; Kon et al. 2026; Shao et al. 2025). The stronger model therefore acts as a cognitive tool that supplements capabilities unavailable to the smaller model, effectively expanding its problem-solving capacity without requiring those capabilities to be internally acquired. Indeed, prior to the emergence of LLMs, planning constituted a well-established domain within the machine learning community. Traditional planners such as PDDL (Gerevini 2020) have long been investigated in this context. Leveraging their extensive repository of world knowledge, LLMs can translate natural language specifications into formal planning languages, thereby delegating planning to dedicated planning systems (Dagan et al. 2023; Guan et al. 2023; Hirsch et al. 2024; Liu et al. 2023a). Such approaches separate requirement understanding from plan generation, allowing specialized planners to focus on solving well-defined planning problems.

Agentic Harness.

Complementing post-training, multi-agent collaboration, and planning offloading, an agentic harness is the model-external runtime layer that manages context, tools, state, control flow, and verification during task execution (Lin et al. 2026; Wang et al. 2025f; Yang et al. 2024a; Zhang et al. 2025c). Its components typically include system instructions, retrieved skills, memory, agent and tool interfaces, and policies for retry, stopping, reflection, and recovery. Harness engineering can therefore improve planning without modifying model weights by structuring how task requirements are represented and updated throughout an interaction. Recent studies further treat the harness itself as an optimization target, using execution traces, task outcomes, and accumulated experience to search or evolve its code and components (Huang et al. 2026; Lee et al. 2026c; Lin et al. 2026; Liu et al. 2026; Pan et al. 2026). These approaches span end-to-end code optimization, observability-driven component evolution, self-supervised refinement, and continual task-specific adaptation. In agent search, the harness converts evolving task states into capability queries and revises retrieval decisions using feedback from selected agents. This coupling makes observed agent capability dependent on the model–harness configuration rather than the base model alone (Kapoor et al. 2025; Lin et al. 2026; Yao et al. 2026; Zhang et al. 2026a). Current evidence, however, remains dominated by software-engineering settings, leaving cross-model and cross-domain transfer, computational cost, security, and selection bias insufficiently understood.

4.3. Proactive User Intent Clarification

The task decomposition and planning augmentation strategies surveyed above implicitly assume that the user’s task specification is sufficiently clear for the system to act upon. In practice, however, user-submitted task descriptions are frequently ambiguous or only partially specified, and an unclear specification will propagate errors into downstream requirement discovery, agent retrieval, and agent selection decisions. Proactive user intent clarification addresses this gap by engaging users through dialogue to elicit and resolve ambiguity before or during task understanding, thereby improving the quality of downstream requirement identification, agent retrieval, and orchestration. As dedicated literature on user intent clarification for agent search remains scarce, we draw upon adjacent fields including conversational recommender system (CRS) (Gao et al. 2021; Jannach et al. 2022), conversational search (Meng et al. 2023, 2025; Mo et al. 2025b), and tool learning (Qu et al. 2025b), and categorize existing methods by whether clarification operates over a closed-ended structured attribute space or an open-ended natural language space.

Closed-ended Attribute-based Clarification.

Closed-ended attribute-based clarification formulates user intent elicitation as sequentially asking users about predefined, structured attributes to progressively narrow down their requirements. Although no longer the dominant paradigm in the LLM era due to its dependence on predefined attribute spaces, it remains valuable for its efficiency and controllability. CRS formulates intent clarification as sequential decision-making over structured attributes, providing the closest methodological foundation for this approach. UNICORN (Deng et al. 2021) laid the foundation by unifying three previously separate CRS decisions (what attribute to ask about, when to ask, and what to recommend as the result) into a single graph-based deep Q-network policy. HutCRS (Qian et al. 2023) observed that asking users to give explicit yes/no feedback on every required attribute, regardless of their knowledge or interest, fails to capture their multi-faceted intent, and introduced a hierarchical interest tree that enables the system to first ask broad questions about interest aspects and then drill down to specific attributes through targeted follow-up questions via its Hierarchical-Interest Policy Learning module. SAPIENT (Du et al. 2025) advanced the attribute-asking strategy beyond the greedy action selection shared by prior methods, introducing non-myopic planning via Monte Carlo Tree Search (Kocsis and Szepesvári 2006) that simulates future attribute-answer trajectories to identify higher-reward clarification sequences rather than greedily selecting the next attribute. These CRS methods share a structural parallel with agent search: item attributes can correspond to agent capability dimensions. Agent capabilities, constraints, and interaction requirements, however, are open-ended and compositional, extending well beyond finite CRS attribute spaces. This suggests that future agent search systems may require structured capability taxonomies or hybrid clarification strategies that combine structured reasoning with natural language interaction.

Open-ended Language-based Clarification.

Rather than selecting from predefined attributes, open-ended methods generate clarifying questions in natural language, offering greater flexibility and expressiveness in capturing nuanced user needs, though at the cost of reduced controllability and a heavier reliance on the model’s capacity to decide when and what to ask. Related work in this area can be traced from foundational retrieval research to increasingly principled agent-oriented frameworks. Zamani et al. (Zamani et al. 2020) proposed the task of generating clarifying questions for open-domain information retrieval, training supervised and reinforcement learning models with weak supervision from query logs, and validating through large-scale online experiments. Tell Me More! (Qian et al. 2024b) extended this to LLM agent scenarios, training Mistral-Interact to proactively assess task vagueness and elicit implicit user intentions through explicit questioning, representing an early work on proactive agent questioning. AskToAct (Zhang et al. 2025g) further introduced self-correcting clarification with a selective masking mechanism for dynamic error detection and correction during multi-turn interaction, recovering the majority of critical unspecified intents. SAGE-Agent (Suri et al. 2025) advanced this direction by introducing structured uncertainty over parameter domains and quantifying each candidate question’s disambiguation value through an expected value of perfect information objective, achieving substantial coverage improvement with significantly fewer questions. On the evaluation side, Ambig-SWE (Vijayvargiya et al. 2025) validated the practical value of interactive clarification by constructing an underspecified SWE-Bench variant, demonstrating significant performance gains. Beyond clarifying a single task, a small but growing body of work begins to close the loop by using clarification to drive the agent search itself: iCARE (Wiratunga et al. 2026), for example, asks one or two targeted clarifying questions when an utterance is under-specified and routes the resulting clarified intent to the appropriate specialist agent, an early instance of clarification-driven agent routing. Compared with closed-ended methods, LLM-based clarification is naturally more compatible with agent search, as agent capabilities are typically described in natural language. However, agents are autonomous entities with internal state, dynamic behavioral characteristics, and reliability profiles that go beyond the static descriptions assumed in existing retrieval settings. Consequently, effective clarification may need to reason not only about what capabilities are required, but also about execution constraints, coordination preferences, and acceptable trade-offs among candidate agents and tools.

5. Agent Discovery, Representation and Indexing

Agent search fundamentally differs from traditional document retrieval because the retrieval target is an actionable entity rather than a passive information object. Consequently, an agent search system must not only identify relevant agents, but also represent their capabilities, constraints, behavioral characteristics, and execution requirements in a form that can be indexed and retrieved. The effectiveness of downstream retrieval and selection therefore depends critically on how agents and tools are discovered, represented, and maintained. Before retrieval and reranking algorithms can operate (Section 6), a system must (i) decide the unit of retrieval (single API endpoint, tool package, agent, or agent policy), (ii) define the representation surface of each unit, and (iii) construct and maintain an index over those representations. This framing is consistent with tool ecosystems that explicitly introduce a retrieval component over large tool pools (e.g., API-Bank, ToolLLM, and ToolRerank) (Li et al. 2023b; Qin et al. 2024b; Zheng et al. 2024).

We use representation to mean the information stored for each tool/agent that will later be used for matching and selection (e.g., names, natural-language documentation, structured schemas, usage examples, learned tool tokens) (Hao et al. 2023b; Li et al. 2023b; Liang et al. 2023). We use indexing to mean the data structure and preprocessing that makes retrieval feasible at scale (e.g., inverted indexes for lexical search, dense vector indexes for semantic search, hierarchical catalogs, or cache/lookup tables) (Du et al. 2024; Guo et al. 2024; Li et al. 2023b; Schick et al. 2023).

5.1. Units of Discovery

A fundamental design decision in agent search concerns the unit of retrieval itself. Different systems expose capabilities at different granularities, leading to different retrieval, orchestration, and composition behaviors: (1) API-level tools, where each callable function/API endpoint is a distinct item (e.g., API pools and plugin/function libraries) (Li et al. 2023b; Patil et al. 2024; Qin et al. 2024b); (2) toolkits / packages, where a set of APIs is retrieved as a group (e.g., domain-scoped packages or clustered toolsets) (Liang et al. 2023; Liu et al. 2025; Yuan et al. 2024a); and (3) agents, where the retrieved item is an agent policy/prompt/program or a capability/profile entry in a routing pool (Hu et al. 2025c; Wang et al. 2026; Yue et al. 2025b).

The unit choice interacts with context limits and orchestration overhead. For example, TaskMatrix.AI discusses an API selector over millions of APIs and also a package/module strategy to locate relevant APIs more efficiently (Liang et al. 2023). Conversely, API-Bank and ToolLLM emphasize selecting individual APIs from an API pool via a retriever (Li et al. 2023b; Qin et al. 2024b).

5.2. Tool Representation Surfaces

The representation surface determines which aspects of a tool or agent become visible to downstream retrieval models. Consequently, representation design directly defines the signals available for capability matching. Across the literature, tools are commonly represented by combinations of: identifier strings (tool/API names), natural-language descriptions, structured I/O specifications (parameter lists, JSON/OpenAPI schemas), code-level signatures (type hints, docstring signatures, JSDoc), and usage examples (demonstrations, snippets) (Esakkiraja et al. 2025; Li et al. 2023b; Liang et al. 2023; Ocker et al. 2024; Patil et al. 2024). Some systems additionally incorporate environment metadata (e.g., host URL, authentication requirements, registry/server info) (Fei et al. 2025b; Patil et al. 2024; Qin et al. 2024b) or even the tool implementation code itself when available (Esakkiraja et al. 2025; Yuan et al. 2024a). Table 1 summarizes these representation fields.

Meta-information and documentation.

A widely used baseline is to index API meta information (name, definition/description, and input/output parameters). API-Bank, for instance, introduces a dedicated API Search component that embeds query keywords and all API meta information, then retrieves by cosine similarity (Li et al. 2023b). TaskMatrix.AI standardizes API documentation into a unified schema consisting of API name, parameter list (including types/defaults), description, optional usage examples, and optional composition instructions (Liang et al. 2023). Gorilla similarly constructs JSON-form API documents (including api_name, arguments, example code, and environment requirements) and retrieves relevant API documentation to inject into the prompt (Patil et al. 2024).

Schemas and signatures.

Structured tool definitions can sharpen retrieval and invocation correctness by grounding parameter/return fields. Several systems explicitly store parameter/return fields and types (often in JSON-like forms) (Fei et al. 2025b; Li et al. 2023b; Qin et al. 2024b), while others rely on code-level signatures and docstrings/type hints. Tulip performs code introspection to extract function name, docstring, and parameter names/descriptions/types (Sphinx-style), then embeds the extracted record for vector-store search (Ocker et al. 2024). DeepCodeSeek similarly treats developer-facing API documentation as structured JSDoc summaries/signatures plus metadata, using this as an indexing substrate for multi-stage retrieval (Esakkiraja et al. 2025).

Examples and demonstrations.

Examples appear in two roles: (i) as part of the indexed tool description (to help matching and invocation), and (ii) as training data to learn better representations or selectors. TaskMatrix.AI explicitly allows usage examples in its unified API documentation schema (Liang et al. 2023). ToolkenGPT learns a dedicated embedding per tool token (“toolken”), trained on demonstrations so that tool selection can be triggered by token generation (Hao et al. 2023b). ToolBank also emphasizes usage-driven signals by constructing Tool2Vec embeddings derived from example user queries associated with each tool (Moon et al. 2024).

Environment and runtime metadata.

While early tool-retrieval work often treats tools as pure text, more realistic tool libraries increasingly encode environment constraints such as authentication, host URLs, server registries, and runtime requirements. For instance, Gorilla includes environment requirements in its JSON API documents (Patil et al. 2024); ToolLLM records per-tool host URLs in its API documentation fields (Qin et al. 2024b); and MCP-Zero represents MCP server metadata and tool parameters in a structured schema and builds a vector index over server and tool descriptions (Fei et al. 2025b).

5.3. Indexing and Retrieval Backends

Given a representation surface, systems operationalize search using a small set of recurring paradigms (Table 2). While indexing techniques largely originate from information retrieval, their role in agent search extends beyond relevance estimation because indexed representations must support capability discovery, constraint matching, and orchestration decisions.

Sparse lexical indexes.

BM25-style lexical retrieval is common as a baseline and remains attractive for its simplicity and robustness. Toolformer uses BM25 over a Wikipedia dump for its search tool component (Schick et al. 2023), and Gorilla explicitly evaluates BM25 document retrieval for API selection (Patil et al. 2024). Several tool-retrieval works also report BM25 as a baseline against dense or hybrid approaches (Chen et al. 2024b; Lin et al. 2025; Zheng et al. 2024).

Dense vector indexes.

Dense retrieval is pervasive when tool descriptions are long, heterogeneous, or semantically distant from user queries. API-Bank retrieves by embedding API meta information and query keywords (Li et al. 2023b); ToolLLM trains a Sentence-BERT-based dense bi-encoder API retriever (Qin et al. 2024b); ToolRerank uses a dual-encoder retriever for candidate generation (Zheng et al. 2024); and Tulip implements tool-library search as vector-store retrieval over introspected tool descriptors (Ocker et al. 2024).

Multi-stage selection (retrieve then refine/rerank).

Many practical systems follow multi-stage pipelines: a fast retriever proposes candidates, then a more expensive component refines, reranks, or validates. Re-Invoke enriches tool documents with synthetic queries and performs dense retrieval in embedding space (while also evaluating sparse baselines) (Chen et al. 2024b). ToolRerank applies cross-encoder reranking and hierarchy-aware reranking on top of dual-encoder retrieval (Zheng et al. 2024). ToolBank describes a two-stage retrieve-then-refine approach, including usage-driven tool embeddings (Tool2Vec) and a refinement classifier (ToolRefiner) (Moon et al. 2024). Tools-are-under-documented similarly highlights document expansion plus retriever+reranker stacks (Tool-Embed/Tool-Rank) for tool retrieval (Lu et al. 2025a).

Hierarchical catalogs and structured routing.

When tool libraries are very large, systems often introduce intermediate structure: domains, categories, modules, or clusters. AnyTool organizes the API collection into a hierarchy of categories, tools, and APIs, and employs a corresponding hierarchy of meta-, category-, and tool-level agents that progressively partition the search space, enabling scalable retrieval over more than 16,000 APIs despite LLM context-length constraints (Du et al. 2024). MCP-Zero also uses hierarchical vector routing by first matching servers and then ranking tools within matched servers (Fei et al. 2025b). ToolRerank explicitly exploits tool hierarchies to improve reranking (Zheng et al. 2024), and Tool-Planner retrieves/organizes at the level of clustered tool groups for planning (Liu et al. 2025).

Caching and lookup indexes.

Not all “retrieval” is document search. StableToolBench introduces a cache keyed by (category, tool, API name, arguments) to return stable responses, with fallback to simulated servers when the cache misses (Guo et al. 2024). API-Bank similarly hard-codes retrieved external-information results for test queries to ensure reproducibility (Li et al. 2023b). These approaches highlight that evaluation and production systems often combine semantic retrieval with explicit lookup mechanisms for determinism and cost control.

In-model tool identifiers (token indexing).

A distinct line of work treats tool selection as generating a discrete identifier inside the language model. ToolkenGPT learns a vocabulary of tool tokens and their embeddings, allowing the LM to select a tool by emitting the corresponding tool token, followed by argument completion (Hao et al. 2023b). Related “generation-as-retrieval” framing also appears in ToolGen, which unifies retrieval and calling via generation-based mechanisms (Wang et al. 2025c).

5.4. Agent Representations

Compared to tools, explicit agent representations are less standardized and often depend on the system’s routing design. We observe three recurring families: (1) policy/prompt representations, where an “agent” is represented by its instruction prompt, program, or workflow policy (Hu et al. 2025c); (2) capability/profile representations, where an agent is represented by a profile entry describing competence/cost/behavioral characteristics (e.g., an LLM profile pool for routing) (Yue et al. 2025b); and (3) routing metadata with evidence/provenance, where routing decisions are supported by structured registries/handbooks and stored traces (Wang et al. 2026).

For example, MasRouter constructs an LLM pool where each candidate has a name and free-text description (with additional attributes embedded in descriptions), and uses dense representations for routing (Yue et al. 2025b). SkillOrchestra formalizes a skill handbook (a graph-structured registry) and agent profiles with competence and cost signals, augmented with embedding-based local search (Wang et al. 2026). These systems illustrate that agent indexing remains substantially less standardized than tool indexing. Unlike tools, whose functionality is often explicitly specified through interfaces and schemas, agents possess behavioral, strategic, and executional characteristics that are considerably harder to represent. Developing expressive and searchable agent representations therefore remains a central challenge for future agent search systems.

5.5. Index Maintenance

Unlike traditional information retrieval, where the indexed corpus is relatively static after construction, agent search operates over continuously evolving ecosystems. Tool descriptions become stale, APIs are updated or deprecated, runtime environments change, and agents may continuously acquire new capabilities. Consequently, maintaining an accurate and up-to-date index becomes an integral component of the agent search pipeline rather than a one-time offline preprocessing step. Although many existing works largely assume static indexes, recent studies have begun exploring several strategies for mitigating representation drift: (i) document expansion to reduce under-specification in tool docs (Lu et al. 2025a); (ii) iterative feedback / self-reflection loops to improve retrievers over time (Du et al. 2024; Xu et al. 2024); and (iii) controlled evaluation environments and caching to ensure reproducibility (Guo et al. 2024; Li et al. 2023b).

Existing work largely treats index maintenance as an engineering issue, focusing on improving documentation quality, retrieval robustness, or evaluation reproducibility. However, from the perspective of agent search, maintenance should instead be viewed as a continuous representation problem. Agent capabilities, operational constraints, availability, and behavioral characteristics may all evolve over time, requiring indexed representations to be continuously updated rather than simply expanded with newly added tools. Developing principled mechanisms for capability tracking, version management, safety-aware indexing, freshness estimation, and dynamic representation updates therefore remains a central open challenge for agent search (Fei et al. 2025b; Li et al. 2023b; Patil et al. 2024).

6. Retrieval, Reranking, and Selection of Agents

After task understanding has transformed user requests into explicit capability requirements (Section 4), and suitable representations have been constructed for candidate agents and tools (Section 5), the next challenge is to identify which candidates should actually be selected. Unlike traditional document retrieval, agent retrieval is not solely concerned with topical relevance. Instead, retrieval models must estimate whether an agent possesses the required capabilities, satisfies operational constraints (e.g., cost, latency, safety, availability), and can effectively collaborate with other retrieved agents. Consequently, agent search extends classical retrieval into a capability matching and decision-making problem. In this section, we review existing retrieval, reranking, and selection techniques that support this process.

6.1. Agent and Tool Selection

Agent selection is the core decision-making stage of agent search. Given a set of capability requirements identified during task understanding and candidate representations constructed during indexing, the objective is to identify the agents or tools that best satisfy the requested capabilities and operational constraints. Existing approaches to agent/tool retrieval and selection can be broadly divided into representation-based methods that rely on similarity matching, reasoning-based methods that leverage large language models for compatibility estimation, and structure-aware methods that organize agents within hierarchical or graph-based search spaces.

Similarity Matching.

Early studies on tool and agent retrieval largely follow the classical information retrieval paradigm, modeling the problem as similarity matching between task descriptions and tool representations. These approaches typically employ either sparse retrieval, which relies on lexical overlap and probabilistic ranking models (Robertson and Zaragoza 2009; Sparck Jones 1972), or dense retrieval, which estimates semantic similarity using neural embeddings (Gao and Callan 2022; Hofstätter et al. 2021; Izacard et al. 2022; Reimers and Gurevych 2019; Xiong et al. 2021). Building on this paradigm, several recent works adopt general-purpose retrievers trained on query–document datasets to improve tool discovery and candidate generation (Guo et al. 2024; Li et al. 2023b; Patil et al. 2024; Qin et al. 2024b; Yuan et al. 2024a). Although highly scalable, similarity-based retrieval assumes that capability matching can be approximated by representation similarity. In agent search, however, successful task completion often depends on reasoning about complementary capabilities, constraints, and interactions among multiple agents, motivating richer retrieval paradigms (Qu et al. 2024). To better support tool-oriented search, subsequent studies construct large-scale tool retrieval benchmarks and adapt retrievers from general-domain relevance estimation to tool-specific capability matching (Shi et al. 2025d). More recent efforts further enhance similarity-based retrieval by introducing task decomposition strategies, which retrieve tools for sub-tasks sequentially (Tang et al. 2026), or by expanding tool representations through document augmentation and capability enrichment (Chen et al. 2024b; Lu et al. 2025a; Yuan et al. 2024c). These extensions aim to improve both recall and compositional coverage while preserving the scalability advantages of representation-based retrieval.

LLM-based Selection.

In contrast to similarity-based retrieval methods, another line of work leverages the reasoning and generalization capabilities of large language models (LLMs) to perform tool or agent selection. These approaches typically estimate task–tool compatibility through contextual reasoning rather than explicit similarity matching. Existing methods can be broadly divided into training-free and training-based paradigms. Training-free approaches (Lu et al. 2023; Shen et al. 2023; Song et al. 2023) primarily rely on prompting LLMs to select appropriate tools from a small candidate set, either by directly reasoning over tool descriptions or by incorporating auxiliary signals such as heuristic importance scores (Cai et al. 2024; Chen et al. 2024c; Hsieh et al. 2023; Huang et al. 2024a,c; Liu et al. 2024b; Paranjape et al. 2023). While flexible and easy to deploy, these methods often face scalability challenges due to the limited context window of LLMs and the well-documented degradation of performance in long-context settings (Dou et al. 2026; Liu et al. 2024a). To address these limitations, recent training-based approaches aim to internalize tool knowledge into the parametric space of the model (Hao et al. 2023b; Li et al. 2025i; Wang et al. 2025c). In these methods, tools are represented using dedicated tokens or structured action formats, and models are trained to generate appropriate tool calls as part of the decoding process. Consequently, LLM-based selection shifts the retrieval problem from explicit similarity estimation toward capability reasoning, allowing the model to jointly consider task requirements, constraints, and candidate interactions during selection.

Structure Retrieval.

As agent ecosystems continue to grow, exhaustively evaluating every candidate agent or tool becomes increasingly impractical. Structure-aware retrieval therefore introduces intermediate organizational structures that progressively narrow the search space before more expensive reasoning or reranking is performed. Depending on whether these structures are constructed independently of the input task or dynamically during retrieval, existing approaches can be broadly categorized into task-agnostic and task-specific structural selection methods. Task-agnostic approaches build structural organization over candidate tools prior to receiving a specific task, and subsequently exploit this structure during search. Typical designs include clustering-based organization (Liu et al. 2025; Yue et al. 2025a), hierarchical tree structures (Du et al. 2024; Ocker et al. 2024; Zheng et al. 2024), and various forms of capability or dependency graphs (Liu et al. 2024b; Qu et al. 2024). By precomputing relationships among tools, these methods aim to reduce search complexity and support structured candidate exploration at inference time. In contrast, task-specific structural approaches dynamically construct or expand the search structure conditioned on the input task, often leveraging test-time scaling strategies such as Chain-of-Thought reasoning (Wei et al. 2022b) or Tree-of-Thought exploration (Yao et al. 2023). These methods formulate tool selection as a structured search process, including depth-first tree exploration (Qin et al. 2024b), best-first (greedy) search (Koh et al. 2025), MCTS (Yang et al. 2026b; Zhou et al. 2023; Zhuang et al. 2024), and trajectory-conditioned search (Patel et al. 2026). Compared with task-agnostic organization, task-specific construction allows more flexible reasoning over task requirements but typically incurs higher computational cost during inference.

6.2. Iterative Tool Retrieval

6.2.1. Reason-Act-Reflection

Static retrieval assumes that all required agents can be identified before execution begins. In practice, however, newly retrieved agents often reveal previously unknown capabilities, missing dependencies, or unexpected execution outcomes. Consequently, many systems perform agent retrieval iteratively, allowing retrieval decisions to evolve together with task understanding. A large body of work follows the ReAct paradigm (Yao et al. 2022), which operates through iterative reason-act-reflection loops. They use LLMs as their decision-making core and perform multiple rounds of chain-of-thought (CoT) reasoning (Wei et al. 2022b) and tool invocations. Specifically, at each iteration, the agent reasons over the tool outputs from the previous interaction and plans the next action, namely whether to invoke another tool or produce the final answer; if the agent decides to call and execute another tool, the tool outputs will be sent back to the agent for reasoning at the next iteration. However, most work using the ReAct paradigm typically allows agents to access only a very limited number of tools. For example, agents in (Chen et al. 2025d; Meng et al. 2026) use only two tools, namely a document search tool and a full-document reader tool. However, relying on a limited toolset restricts agents’ generalizability across domains. To alleviate this limitation, Li et al. (2025i) propose DeepAgent, which at each iteration searches over a toolset, selects a tool to execute, and then reasons over the tool outputs. (Fei et al. 2025b) propose MCP-Zero, an MCP-based framework that resembles ReAct-style interaction. MCP-Zero generates structured requests that specify the agent’s requirements and performs hierarchical tool search; when the retrieved tools are insufficient, the agent refines the requests and continues the process. (Gaurav et al. 2025) propose Dynamic ReAct, which explores ReAct agents operating over large Model Context Protocol (MCP) tool sets.

6.2.2. Iterative Query Reformulation

While ReAct-style systems refine retrieval through interaction with the environment, another line of work improves retrieval by iteratively refining the search query itself. Rather than treating the initial task description as fixed, these methods progressively reformulate the query according to retrieval outcomes, enabling better alignment between user requirements and retrieved agents or tools. Xu et al. (2024) identify a misalignment between tool retrieval and downstream LLMs, where the two components operate largely independently. At each iteration, the tool-usage LLM assesses the retrieved tools and refines the query; the updated query is then used to retrieve a new set of tools for the next round. The number of iterations is a hyperparameter, and performance generally improves with more iterations (Xu et al. 2024).

6.3. Agent and Tool Recommendation

Capability matching alone does not fully determine which agent should be selected. When multiple agents satisfy the same functional requirements, user preferences become an additional optimization objective. A user may express they need “data analysis,” yet the query reveals nothing about which agent’s interaction style, response format, or domain expertise best matches their working habits. To bridge this gap, the system can mine accumulated behavioral histories to infer implicit preferences and personalize agent and tool recommendations accordingly. We survey relevant techniques from recommendation systems, where preference inference from behavioral data has been extensively studied, and organize the discussion along a progression from general recommendation techniques to agent and tool-specific applications.

General LLM-based recommenders infer user preferences from behavioral history; they differ mainly in which behavioral signal they exploit and how it is encoded. CLLM4Rec (Zhu et al. 2024) was the first to integrate user behavioral history into LLMs as dedicated user and item ID tokens, tightly coupling the ID-based collaborative paradigm with LLMs and learning collaborative filtering patterns (Koren et al. 2022) through mutually regularized pretraining. ECR (Zhang et al. 2024f) and PersonaX (Shi et al. 2025b) further refined behavioral preference modeling along complementary dimensions: ECR mined affective signals as finer-grained behavioral cues to improve preference capture, while PersonaX performed offline multi-persona profiling from long behavioral sequences to capture diverse interest facets. RPP (Mao et al. 2025) instead personalizes the prompt for each user, using multi-agent reinforcement learning to select prompt components from the user’s interaction history rather than applying a shared template.

A second line of work recasts the participants in recommendation, including users and items, as autonomous LLM agents. The recommended object is still a product, but this agentic reformulation is what directs agent recommendation to build on. AFL (Cai et al. 2025b) demonstrated that agentic feedback loops between recommendation agents and simulated user agents can leverage iterative interaction histories to enable co-evolutionary preference refinement, offering a mechanism directly informative for agent selection. AgentCF (Zhang et al. 2024e) modeled both users and items as autonomous LLM agents that collaboratively reflect and propagate preferences through inter actions.

Moving toward direct tool and agent recommendation, where the recommended object is itself a tool or agent, PTool (Huang et al. 2025a) addressed personalized tool invocation by modeling users’ implicit tool preferences from simulated behavioral histories when choosing among functionally similar tools, proposing a data synthesis framework and the first benchmark for this task. Extending personalization across sessions, PRefine (Yoon et al. 2026) represented user preferences as evolving hypotheses, extracting reusable constraints from a user’s cross-session interaction history to personalize tool calling, and introduced the MPT benchmark covering preference recall, induction, and transfer. Rec4Agentverse (Zhang et al. 2024d) further envisioned a full agent-platform recommendation paradigm with three evolutionary stages of user-agent-recommender collaboration, validated through preliminary case studies.

Despite targeting different recommendation scenarios, these methods consistently demonstrate that mining user behavioral histories is effective for personalized selection. While these techniques offer transferable methodological foundations for agent search, agent interactions encode far richer signals than traditional item interactions, including task success or failure, response quality, latency, and trust calibration, necessitating an extension from consumption-oriented preference profiles to multi-dimensional delegation experience profiles.

7. Evaluation of Agent Search

Evaluating agent and tool search systems presents challenges that differ fundamentally from those in classical information retrieval (IR). Traditional IR evaluation focuses on the relevance of retrieved documents to a user query, supported by well-established metrics, annotation protocols, and benchmark collections. In contrast, agent search retrieves executable systems whose value can only be fully assessed through task execution, making relevance dependent on capability, reliability, and downstream task success rather than textual similarity alone. This distinction motivates a two-level evaluation framework. Intrinsic evaluation assesses the quality of the search process itself—such as retrieval coverage, tool selection correctness, and invocation accuracy—by comparing intermediate outputs against annotated ground truth without requiring end-to-end execution. Extrinsic evaluation, by contrast, measures the utility of selected agents or tools through their performance on downstream tasks, focusing on whether deployment successfully achieves the user’s objective. These paradigms are complementary: intrinsic evaluation enables fine-grained diagnosis and scalable assessment, whereas extrinsic evaluation provides the most direct measure of real-world utility but is more costly, difficult to standardize, and dependent on reliable automated judging. We survey both approaches, reviewing their metrics, benchmarks, and evaluation frameworks, and highlighting the limitations that motivate further research in agent and tool search evaluation.

7.1. Intrinsic Evaluation

7.1.1. Automated Metrics

Automated intrinsic metrics evaluate the two online stages of the agent search pipeline: whether the system surfaces the right candidates (retrieval quality), and whether a selected candidate is invoked correctly (selection and invocation correctness). Both are assessed by comparing intermediate outputs against annotated ground truth without end-to-end execution; invocation correctness is treated as intrinsic because it is verified structurally against a reference call rather than by running the tool.

For the initial retrieval phase, traditional IR metrics are employed, including Recall@k (Manning 2008), MRR@k (Mean Reciprocal Rank) (Voorhees et al. 1999), and NDCG@k (Normalized Discounted Cumulative Gain) (Järvelin and Kekäläinen 2002), to assess how well a system surfaces relevant candidates from large tool or agent pools (Qin et al. 2024b; Shi et al. 2025d). Several benchmarks have been developed to evaluate this stage specifically. Shi et al. (2025d) introduced ToolRet, a heterogeneous benchmark comprising $\mathrm{7,600}$ diverse retrieval tasks paired with a corpus of over $\mathrm{43,000}$ tools. They found that even strong general-purpose IR models achieve substantially lower performance on tool retrieval than on conventional IR tasks, attributing this to low lexical overlap between natural language queries and formal tool documentation, as well as a fundamental domain shift from information-seeking to capability-matching retrieval. Qu et al. (2024) further showed that standard recall-oriented metrics can mask a distinct failure mode in retrieval evaluation: semantic matching tends to retrieve redundant, similar tools rather than a complete and diverse set, which is particularly problematic for multi-step tasks requiring the coordinated use of several tools. To address this, they introduced a new metric, completeness (COMP@K), which measures whether all ground-truth tools required for a query appear within the top-K retrieved results, relaxing the single-relevant-item assumption that Recall@k and MRR@k inherit from document retrieval.

Once candidates have been retrieved, the selection and invocation stage is evaluated using deterministic structural checks that verify output correctness against annotated ground truth. The Berkeley Function Calling Leaderboard (BFCL) (Patil et al. 2024, 2025) introduces Abstract Syntax Tree (AST) matching as a scalable proxy for execution that verifies tool names and parameter types against ground-truth specifications across serial, parallel, and multi-turn settings. To reduce false negatives, it applies type-aware normalization rules, including case-insensitive string comparison. ToolQA (Zhuang et al. 2023) applies analogous normalization principles more broadly, using deterministic rules covering time formats, price units, punctuation, and whitespace to enable reliable exact-match scoring of free-form answers.

Several benchmarks further decompose tool utilization into atomic sub-capabilities, providing finer-grained diagnostics beyond a single accuracy score. API-Bank (Li et al. 2023b) structures evaluation across three progressive levels, Call, Retrieve+Call, and Plan+Retrieve+Call, isolating whether failures originate in invocation correctness or multi-step planning. T-Eval (Chen et al. 2024c) decomposes tool utilization into six dimensions spanning planning, reasoning, retrieval, understanding, instruction following, and review, demonstrating that holistic outcome-oriented metrics can obscure systematic failures at intermediate stages. MetaTool (Huang et al. 2024c) targets the decision layer upstream of invocation, evaluating whether models correctly determine both when to invoke a tool and which tool to select, and revealing substantial accuracy degradation when candidates are semantically similar or when tool reliability is in question.

A limitation shared by many of these benchmarks is their reliance on single-trial evaluations, which can overestimate reliability; $\tau$-bench (Yao et al. 2025) addresses this with the pass⌃k metric, the probability that an agent succeeds on all k independent trials of the same task, revealing that consistency degrades substantially even for strong models. A further practical concern is reproducibility: many tool-use benchmarks depend on live API calls, making results sensitive to API deprecation and response drift, which StableToolBench (Guo et al. 2024) addresses by virtualising API execution through a caching system and API simulator. Furthermore, these benchmarks are mostly tool-centric, evaluating API or function selection and invocation rather than the retrieval and selection of agents, whose competence is behavioral and only partially observable before execution. Benchmarks aimed at agent search itself are only beginning to appear (Wu et al. 2026b), and agent-level intrinsic evaluation remains comparatively underdeveloped. Taken together, rigorous intrinsic evaluation therefore requires attention to retrieval quality, completeness, invocation correctness, sub-capability diagnostics, and the stability of the evaluation environment, while the extension of these protocols from tools to agents stays largely open.

7.1.2. LLM-as-a-Judge

Intrinsic evaluation of agent and tool search faces a fundamental coverage problem: annotating ground-truth tool sets for large, evolving registries is expensive and quickly becomes stale. LLM-as-a-judge offers a scalable alternative, replacing human annotators in assessing whether retrieved agents or tools match the capability requirements expressed in a query. A foundational effort in this space is the LLMJudge challenge (Rahmani et al. 2024c), introduced at the LLM4Eval workshop (Rahmani et al. 2025c) at SIGIR 2024 (Rahmani et al. 2024b), which benchmarks the capability of LLMs to generate relevance labels for the TREC Deep Learning track (Craswell et al. 2024) and systematically investigates prompt design, model choice, ensemble techniques, and biases in synthetic judgments (Rahmani et al. 2025b). These findings in traditional IR evaluation carry direct implications for agent search, where similar prompt design, model choice, and bias concerns apply when using LLMs to judge whether a retrieved agent’s capability profile genuinely matches a task specification. Subsequent theoretical and empirical analyses, including comprehensive surveys of the LLM-as-a-Judge paradigm (Gu et al. 2024; Li et al. 2024b), have established reliability assessment frameworks that highlight consistency, bias mitigation, and domain adaptation as key factors in building trustworthy evaluation systems (Wang et al. 2025h; Yang et al. 2026a; Zhang et al. 2025h).

Building on these foundations, a growing line of work investigates the applicability of LLM judges specifically in IR settings and relevance assessment. Early perspectives highlight both the promise and limitations of replacing human assessors with LLMs, emphasizing concerns around evaluation validity and reproducibility (Breuer et al. 2020; Faggioli et al. 2023). Complementary work on synthetic test collections and query generation further shows how LLM-based labeling pipelines can reshape evaluation distributions and introduce systematic biases (Rahmani et al. 2024a, 2025a; Zendel et al. 2025). At the same time, LLMs have demonstrated strong capability in modeling user preferences and relevance signals, suggesting their potential as scalable proxies for human judgments in ranking and retrieval scenarios (Chiang and Lee 2023; Thomas et al. 2024). These developments position LLM judges as a practical, though imperfect, evaluation mechanism for agent and tool search — particularly for assessing whether a retrieved candidate’s described capabilities align with the task intent, a judgment that is difficult to automate deterministically and for which human annotation at scale is infeasible.

To improve reliability, recent work has explored structured and rubric-based evaluation methods, where LLMs are guided by explicit criteria and multi-step reasoning. Approaches such as G-Eval and rubric-driven frameworks show improved alignment with human judgments by decomposing evaluation into interpretable dimensions (Chen et al. 2026b; Farzi and Dietz 2024; Liu et al. 2023b). Similarly, benchmark-driven efforts for ranking and evaluation, including MT-Bench and JustRank, demonstrate that pairwise comparison and system-level ranking can yield more stable and human-aligned judgments (Gera et al. 2025; Zheng et al. 2023). These structured evaluation paradigms are particularly important in agent and tool retrieval, where relevance is inherently multi-faceted and requires assessing functionality, context-awareness, and downstream utility rather than simple topical matching.

These limitations are particularly consequential in agent search evaluation, where positional bias and self-preference in LLM judges can systematically favour agents whose descriptions are verbose or stylistically similar to the query, rather than those that are genuinely capable. Studies show that LLM judges can be misled by adversarial or irrelevant inputs (Alaofi et al. 2024; Raina et al. 2024), exhibit systematic biases such as self-preference and verbosity bias (Liu et al. 2024c; Thakur et al. 2025), and fail in knowledge-intensive or reference-based evaluation settings (Lee et al. 2026a). To address these issues, recent approaches propose ensemble and meta-evaluation strategies, including jury-based systems and dynamic judge selection mechanisms that improve robustness and agreement with human evaluations (Gera et al. 2025; Li et al. 2025g). In parallel, human-centered and reportingfocused efforts emphasize transparency, interpretability, and human-in-the-loop evaluation (Ashktorab et al. 2025; Lee et al. 2025). Finally, extensions of LLM-based evaluation toward RAG and fact-level assessment further broaden their applicability, enabling fine-grained evaluation of retrieval-augmented and agentic systems (Petroni et al. 2021; Pradeep et al. 2025).

7.2. Extrinsic Evaluation

As opposed to directly evaluating retrieval performance, an alternative approach is to evaluate the performance of a tool-equipped model on downstream tasks using automated verification systems such as LLM-as-a-judge and the emerging Agent-as-a-judge paradigm.

7.2.1. LLM-as-a-Judge

As established in the preceding section, the dominant paradigm for extrinsic evaluation in agent and tool search relies on execution-based signals: pass rates on downstream tasks (Shi et al. 2025d), code execution against test suites (Yue et al. 2025b), or deterministic API-call matching (Li et al. 2023b; Patil et al. 2024). Few agent search papers surveyed employ LLM-as-a-Judge as a primary mechanism for measuring downstream task success.

Nevertheless, LLM-as-a-judge can still be used to assess whether a retrieved agent or tool actually resolves the user’s underlying intent. The paradigm is well-established in the broader NLP literature, where LLMs serving as automated evaluators have been shown to match or exceed inter-human agreement levels across pairwise, pointwise, and reference-guided evaluation modes (Gu et al. 2024; Liu et al. 2023b; Zheng et al. 2023). While LLM-as-a-judge enables rapid evaluation of agentic systems, they have several known weaknesses. For example, LLM-as-a-judge systems exhibit several biases: verbosity bias, intra-model family bias, and positional bias (Koo et al. 2024; Li et al. 2025a). They also lack explicit verification, memory capabilities and intermediate evaluation capabilities required for the evaluation of more complex tasks (You et al. 2026).

7.2.2. Agent-as-a-Judge

The emerging Agent-as-a-judge paradigm attempts to address the aforementioned issues in the LLM-as-a-judge paradigm and improve the evaluation performance of judges by providing them with planning, tool use, and multi-agent collaboration capabilities. These tools and capabilities mitigate biases exhibited by LLM-as-a-judge systems and improve correlation with human annotators in several domains (Gou et al. 2025; You et al. 2026; Zhuge et al. 2024).

Zhuge et al. (2024) pioneered research in this area by providing judge models with planning, retrieval, and memory capabilities when evaluating AI code generation tasks. This allowed judge models to navigate and inspect the complex codebases that they were evaluating, and substantially increased alignment with human annotators compared to LLM-as-a-judge. Despite the complexity of the annotation task – where intra-annotator human agreement was low – the introduction of agentic capabilities resulted in a marked increase in correlation with human judgements. An ablation study suggested that the introduction of a codebase graph and multimodal file inspection capabilities led to the greatest increases in judge accuracy in their domain. Mind2Web 2 (Gou et al. 2025) extends Agent-as-a-judge to long-horizon information-seeking tasks across 130 manually curated web search tasks, introducing automatically generated, task-specific rubrics structured as hierarchical trees of verifiable sub-goals. The judge agent is provided with tools to assist with fine-grained evaluation: for example, a “verify by URL” tool allows the agent to check a citation to identify whether a subcriterion holds (such as that a given retrieved result is from a certain website).

Personalized agent search, where the same task may warrant different agent selections depending on user history, expertise, or preferences, introduces a further evaluation challenge: judging whether a retrieved agent was appropriate for this user, not just for this task in the abstract (Liang et al. 2025; Shi et al. 2025b; You et al. 2026; Zhang et al. 2025e). The introduction of memory capabilities has enabled improvements in personalized evaluation by providing judges with user-specific context (Li et al. 2025f). For example, Singh et al. (2025a) propose FSPO, which improves user-specific reward modeling using few-shot preferences and demonstrate its performance on multiple domains: movie reviews, question answering, and education.

Taken together, the evaluation landscape for agent and tool search remains fragmented and, in many respects, immature. Intrinsic metrics adapted from classical IR, such as recall, MRR, and NDCG, measure retrieval performance but fail to capture the capability-matching nature of agent selection, where diversity, completeness, and functional adequacy are often as important as ranking quality. Invocation-correctness benchmarks provide greater specificity but depend on static ground-truth annotations that become increasingly difficult to maintain as candidate ecosystems expand and evolve. LLM-as-a-judge approaches offer scalable evaluation in both intrinsic and extrinsic settings, yet suffer from well-documented biases, including positional bias, verbosity bias, and self-preference, which can distort assessment outcomes. Agent-as-a-judge methods address some of these limitations by incorporating planning, tool-use, and memory capabilities, but remain an emerging area with limited application to agent search evaluation. More fundamentally, evaluation must account not only for whether a candidate was retrieved, but also whether it was capable, safe, reliable, and effective for a given task and user—dimensions that no existing metric or benchmark captures in a unified way. Developing principled evaluation frameworks that span pipeline stages, account for execution uncertainty, incorporate user-centric notions of utility, and remain reproducible as agent ecosystems evolve remains one of the most important open challenges.

8. System-Level Dimensions Influencing Agent and Tool Search

The previous sections examined the core agent search pipeline, including task understanding, capability representation, candidate retrieval and ranking, and evaluation. However, these components operate within broader system-level dimensions that significantly influence real-world performance. Far from being deployment-time concerns, these factors determine whether agent search systems are trustworthy, robust, fair, and usable in practice. In this section, we survey five such dimensions: safety, bias and fairness, security, personalization, and transparency. Safety concerns ensure that agent selection and execution remain aligned with intended goals and constraints. Bias and fairness examine disparities in how agents are discovered, ranked, and selected. Security addresses adversarial threats arising from open and dynamic agent ecosystems. Personalization explores adapting agent search to individual user preferences and objectives. Transparency and explainability focus on whether selection decisions can be understood, inspected, and audited. For each dimension, we distinguish challenges specific to agent and tool search from broader issues in agentic systems, situating agent search within the wider landscape of responsible AI system design.

8.1. Safety

Ensuring that AI systems behave reliably, harmlessly, and in alignment with the intended goals and ethical standards is a central challenge in both development and deployment settings. Safety is particularly critical in agentic AI systems, where models are deployed to perform complex, open-ended tasks involving iterative reasoning, tool use, and interaction with external environments. Safety in agent and tool search, therefore, carries distinct stakes: a retrieval decision that returns a capable but misaligned agent, or a composition that couples individually safe tools into an unsafe workflow, can propagate harm across every downstream step of execution. The risks are often hidden at search time and can materialize during execution, making pre-selection safety assessment a core requirement rather than a deployment-time concern.

8.1.1. Safety in Agent and Tool Search

Recent work highlights the unique challenges of agent search and tool pipelines and proposes methods to evaluate and mitigate them. Studies have shown that search agents are more likely to produce harmful outputs than base LLMs (Zhan et al. 2026), and propose a multi-objective reinforcement learning approach that couples a final-output safety/utility reward with a novel query-level shaping term that penalizes unsafe queries and rewards safe ones. Other works introduce an automated red-teaming framework for LLM search agents across diverse risk categories, including misinformation and adversarial prompts, and reveal the critical need for better defenses (Dong et al. 2025).

Another important direction is the safety of agent composition, which includes both the composition of multi-agent networks and of agent workflows. Real-world deployment requires selecting compositions that not only maximize task success but also minimize safety violations and risks. Recent work formalizes agentic workflows as directed acyclic graphs and introduces algorithms to select agent compositions that minimize low-probability, high-risk behaviors (Shabadi and Alur 2025). Other studies focus on investigating which topological properties of multi-agent networks contribute to safer behavior (Yu et al. 2024), finding that highly connected networks are more vulnerable to misinformation attacks.

8.1.2. Safety in Overall System

Existing alignment methods, including Reinforcement Learning from Human Feedback (RLHF) (Bai et al. 2022a) and Constitutional AI (Bai et al. 2022b), are directly applicable to agent systems and already underpin many safety layers in agent workflows (Dai et al. 2024). Recent work has also proposed agent-specific alignment frameworks, such as Agent-Constitution-based agents (Hua et al. 2024). However, applying these methods to agent search, where a retrieved agent may introduce risks that were not present in its isolated training context, remains largely unaddressed.

A particularly important safety dimension in agent and tool search is the gap between an agent’s stated output and its actual tool-calling behavior. Prior work shows that refusal messages alone are insufficient when an agent refuses a request in text but its tool calls still execute the forbidden action (Cartagena and Teixeira 2026). Proposed mitigations include constraining tool access through least-privilege or minimal permission scopes (Betser et al. 2026; Zhu et al. 2025a), as well as improving the helpfulness and harmlessness of LLMs in tool-use settings (Chen et al. 2024f).

A growing body of safety benchmarks provides evaluation baselines that can be applied to post-retrieval agent execution. AgentHarm measures the ability and propensity of LLM agents to complete harmful tasks, including fraud, cybercrime, and harassment (Andriushchenko et al. 2025), while OpenAgentSafety evaluates agent behavior across multi-turn, multi-user tasks using rule-based and LLM-as-judge assessments (Vijayvargiya et al. 2026). SG-Bench evaluates safety generalisation across tasks and prompts (Mou et al. 2024), while other benchmarks target interactive and embodied environments (Mullen et al. 2024; Yin et al. 2024b; Yuan et al. 2024d; Zhang et al. 2024g; Zhou et al. 2024). Application-specific evaluations have similarly been developed for desktop agents (OS-HARM) (Kuntz et al. 2025), web agents (SafeArena and CASA) (Qiu et al. 2025; Tur et al. 2025), mobile agents (MobileSafetyBench) (Lee et al. 2026b), and scientific agents (SafeScientist) (Zhu et al. 2025b).

Beyond evaluation, safeguarding approaches aim to monitor or constrain agent behavior. For example, GuardAgent uses an LLM agent to generate action plans and guardrail code based on prescribed safety requirements (Xiang et al. 2024), while BELLS provides tests for evaluating input-output safeguards across both established and emerging failure modes (Dorn et al. 2024).

In agent search settings where multi-agent workflows are assembled dynamically from retrieved components, interactions can introduce compounding safety risks that cannot be assessed by inspecting individual agents in isolation. This has motivated work on multi-agent safety, including automated attacks that probe vulnerabilities across roles and interactions (Tian et al. 2023), and multi-agent reinforcement learning frameworks that jointly optimise conversational and feedback agents for safety (Zhang et al. 2026b). Other work highlights the risk of alignment degradation in self-evolving agents through self-training, memory, tool evolution, and workflow optimisation (Shao et al. 2026). More generally, recent studies suggest that safety can depend on a small number of strategically important safety-critical components (Li and Kim 2026). For agent search, this suggests that robust safety may similarly require targeted constraints on high-influence selection and composition decisions rather than uniform enforcement at every retrieval stage.

8.2. Bias and Fairness

AI bias refers to the unintended and potentially harmful systematic disparities in model outcomes across groups (Barocas et al. 2023; Mehrabi et al. 2021). Bias is often studied with respect to protected attributes such as race, gender, geographic location, and other demographic characteristics (Mehrabi et al. 2021). Such biases often arise when models encode and reproduce patterns and imbalances in the training data, amplified by design choices and deployment contexts (Barocas et al. 2023; Hooker 2021). However, bias can also arise through systematic disparities associated with other features, including the sources of information retrieved or the tools selected during decision-making processes (La Malfa et al. 2024). Agents remain susceptible to issues of bias and fairness, challenges deeply rooted within the knowledge base and training of LLMs. Bias in agent and tool search could arise at every stage of the selection pipeline: query understanding may systematically misinterpret task intent for certain user groups, retrieval models may disproportionately surface agents from well-represented providers, and ranking functions may penalize tools with sparse or non-standard documentation regardless of capability.

8.2.1. Bias in Agent and Tool Search

Bias and fairness have long been recognized as challenges in traditional IR systems, where skewed data, ranking algorithms, and user interaction dynamics can systematically favor certain sources or perspectives. Prior IR research has addressed these issues through a range of mitigation strategies, typically categorized into pre-processing, in-processing, and post-processing techniques. These approaches provide a useful foundation for understanding and addressing bias in agent and tool search.

• Pre-processing techniques aim to mitigate biases within datasets before they are used for training models, often using different reweighting and sampling techniques (Kamiran and Calders 2012).
• In-processing techniques usually refer to techniques that penalize the model if it favours bias. For example, AdvBert (Rekabsaz et al. 2021), a BERT-based ranking model that uses adversarial training to simultaneously predict relevance and suppress protected attributes in content retrieved by IR systems. Some approaches modify the ranking process using policies to ensure fair exposure across different document categories (Jaenich et al. 2024), while others focus on fairness-aware learning-to-rank (LTR) frameworks using a policy-gradient method to enforce fairness constraints within a listwise LTR setting (Singh and Joachims 2019; Zehlike and Castillo 2020).
• Post-processing introduces fairness after the model or ranking output is generated (Naghiaei et al. 2022). For example, Yang and Stoyanovich (2017) proposed fairness measures for ranked outputs. Similarly, Zehlike et al. (2017) ensure group fairness in the retrieved documents by guaranteeing a minimum proportion of protected candidates.

Formal fairness frameworks for agent or tool search are largely missing, with existing agent frameworks prioritizing task completion and information relevance. There are some recent works addressing bias in knowledge retrieval. For example, works that point out that pulling from exterior information may introduce and exacerbate fairness issues (Wu et al. 2025d), and other works propose mitigation techniques such as a supervisor-based multi-agent system (Singh and Ngu 2025; Singh et al. 2025b). Bias has also been studied in LLM-as-a-judge evaluators for relevance judgements and different scoring methods (Li et al. 2025a; Rahmani et al. 2025a).

8.2.2. Bias in Overall System

There is emerging work addressing bias for different agent applications, decision scenarios, and multi-agent contexts. While demographic attributes and evaluating disparities between disadvantaged and privileged groups have been the focus in machine learning bias literature, there has been an important line of work adapting these to agent systems, which define protected attributes for multi-agent systems as characteristics that should not disadvantage an agent in terms of its expected rewards (La Malfa et al. 2024). Others highlight that current fairness methodologies are fundamentally unprepared and that bias must be monitored in more dynamic environments (Vakali and Dimitriadis 2025).

There are unique vulnerabilities with multi-agent systems, with prior work showing that complex architectures with more connectivity and recursive communication frequently exacerbate bias, and showing this is an emergent property of agent interactions (Cisneros-Velarde 2025; Coppolillo et al. 2025; Li et al. 2026a). There are also studies focused on fairness in the context of cooperative settings (Grupen et al. 2022). Multi-agent systems often involve competing interests, and allocation while maintaining efficiency also brings in new fairness problems (Trabelsi 2024). There are also new vulnerabilities introduced in memory banks (Gharat et al. 2026), with studies showing that memoryenhanced AI recruitment agents can propagate and amplify bias, with personalization mechanisms also introducing bias over time.

There is a growing body of work focused on tool selection bias that isolates the effects of tool features, metadata, and pre-training exposure (Blankenstein et al. 2026), and shows that manipulating tool descriptions can systematically bias agents toward selecting specific tools (Sneh et al. 2025). Another line of research focuses on the challenging task of measuring ‘implicit’ biases, often defined as systematic differences in model behaviour correlated with demographic attributes even when those attributes are not explicitly stated, and proposing strategies such as self-reflection and fine-tuning approaches (Borah and Mihalcea 2024). Related studies examine decision-making disparities among agents with socio-demographically informed personas (Cao et al. 2026; Li et al. 2025k; Vasista et al. 2025).

8.3. Security Risks

AI agent systems pose a wide range of security risks arising from unpredictable user inputs, complex and variable environments, and interactions with untrusted external entities (Deng et al. 2025). A growing body of work studies these risks through different attack models and defense strategies (Kim et al. 2026a). In agent and tool search, the dynamic and open-ended nature of retrieval creates additional attack surfaces: agents and tools are retrieved from continuously evolving registries, composed at runtime, and granted access to external environments. Consequently, adversarial content can enter the system through the retrieval process itself, making search both a functional component and a potential attack vector.

Security Risks in Agent and Tool Search.

Dynamic tool search introduces a fundamental trade-off between flexibility and security. Restricting agents to pre-authorized tools limits their ability to discover new workflows, whereas unconstrained retrieval increases exposure to adversarial content. In particular, reliance on RAG and persistent episodic memory creates opportunities for attackers to manipulate the information used by orchestrating agents. Prior work demonstrates attacks that poison long-term agent memory (Chen et al. 2024e) and corrupt RAG knowledge databases (Zou et al. 2025). In multi-agent systems, such vulnerabilities can further amplify privacy risks by enabling unauthorized information flows across agents. Traditional Data Loss Prevention (DLP) approaches, which assume fixed perimeters and human-defined data flows, may therefore be insufficient for autonomous systems, motivating approaches based on contextual integrity (Bagdasarian et al. 2024; Ghalebikesabi et al. 2024; Mireshghallah et al.).

These vulnerabilities can also enable confused-deputy attacks, where a privileged agent is manipulated into performing unauthorized actions or exfiltrating sensitive information. For example, poisoned tool metadata (Mo et al. 2026) or environmental injection on retrieved web pages (Liao et al.) can introduce malicious context that propagates across agent workflows. Even encrypted agent traffic can reveal task and tool fingerprints that expose user activity (Zhang et al. 2025j). More broadly, relying on secondary agent monitors introduces further vulnerabilities: agents can exploit reasoning or justification mechanisms to persuade monitors to approve policy-violating actions (Za et al. 2025), while the agent’s awareness of monitoring and the monitor’s scaffolding can substantially affect detection effectiveness (Kale et al. 2026; Korbak et al. 2025). These concerns are particularly relevant given evidence that chain-of-thought rationales can be unfaithful to the underlying computation (Lanham et al. 2023; Turpin et al. 2023) and that models may strategically simulate alignment under some deployment conditions (Greenblatt et al. 2024; Hubinger et al. 2024).

Security Risks in Overall Agent Systems.

Beyond search-specific threats, agent systems are vulnerable to external attacks such as indirect prompt injection (Greshake et al. 2023; Wang et al. 2025i; Zhan et al. 2024), tool poisoning and manipulation (Shi et al. 2025a), and control-flow hijacking in multi-agent systems (Jha et al. 2026). Attacks targeting inter-agent communication can compromise entire multi-agent workflows by manipulating messages passed between agents, motivating benchmarks such as A2ASECBENCH (Li et al. 2026b). Internal threats include memory poisoning (Chen et al. 2024e) and backdoor attacks that trigger malicious behavior under specific conditions (Wang et al. 2024c; Yang et al. 2021, 2024b), alongside broader risks of private data leakage, corruption, and unauthorized exfiltration (Wang et al. 2025d). Other attacks target agent reliability rather than explicitly harmful behavior, inducing repetitive or irrelevant actions (Zhang et al. 2025a).

Defenses against these threats range from probabilistic approaches, including alignment methods such as StruQ (Chen et al. 2025b), defensive system prompts (Yi et al. 2025), and classifiers (Jia et al. 2025), to deterministic enforcement mechanisms that constrain tool calls according to confidentiality and integrity policies (Costa et al. 2025). Autonomy metrics further quantify the fraction of consequential actions an agent can execute without human approval while preserving security (Kolluri et al. 2026). Safety monitors provide another line of defense (Korbak et al. 2025), but their susceptibility to manipulation and sensitivity to monitoring conditions highlight their limitations (Kale et al. 2026; Za et al. 2025). These findings motivate defenses that enforce security constraints independently of an agent’s stated reasoning or intentions.

Finally, security is also influenced by the backbone LLM underlying the agentic system. Prior work identifies risks that propagate from the backbone model to the agent level (Bazinska et al. 2026), while other studies show that models may refuse harmful instructions in isolation but comply when embedded in tool-using and autonomous workflows (Kumar et al. 2024b). This gap is particularly important for agent search, where the security properties of retrieved agents and tools may differ substantially from those of their underlying models and isolated components.

8.4. Personalization

Existing research in personalization around agent search is relatively limited. However, personalization is a long-standing field of study in information retrieval. For example, personalized web search is an area of substantial historical research; traditional web search personalization approaches typically apply user profiling, preference, and intention modeling to improve the personalization of search results for users (Gao et al. 2010; Hannak et al. 2013). Below, we discuss research in agentic search and user simulation that applies personalization to search problems using LLMs and relates closely to the agent search paradigm.

Personalization in RAG and agentic search.

Language models are increasingly used as part of retrieval systems such as retrieval-augmented generation (Lewis et al. 2020), agentic search (Chen et al. 2025d; Wei et al. 2025b; Wong et al. 2025) and deep research (Zhang et al. 2025f). These approaches employ explicit profile information, past history, or persona-based approaches to improve personalization (Li et al. 2025h) and a particular challenge is in the evaluation of personalized outputs (You et al. 2026). Liang et al. (2025) introduce Personalized Deep Research Bench, a novel benchmark which allows for the evaluation of personalization within the deep research setting. Their evaluation framework employs a personalization “meta-evaluator” that dynamically generates dimension-level importance weights for personalization criteria (e.g., goal alignment, content alignment), decomposes these into sub-criteria, and then prompts an LLM to score outputs accordingly. Mo et al. (2025a) motivate the need to balance personalization by noting that excessive personalization can introduce irrelevant profile facts (e.g., homework preferences surfacing in a visa query), and propose APCIR, a conversational IR framework that addresses this via adaptive query rewriting and profile-aware re-ranking. Outside the agentic search setting, various approaches have been proposed to align the evaluation of personalized search with the inferred user profile, such as custom, multi-dimensional rewards (Zhao et al. 2025).

8.4.1. User Simulation and Personalized Web Agents.

Emerging research has shown evidence that language models are capable of simulating user behavior and preferences. Early research in this area typically focuses on the "believability" of agent behavior: the extent to which LLM agents are able to plausibly reflect human behavior in various settings (Park et al. 2023). Various research efforts demonstrated that language models are capable of simulating individual user preferences by applying user simulation techniques to LLM-empowered recommendation generation systems (Bougie and Watanabe 2025; Huang et al. 2025b; Shi et al. 2025b; Wang et al. 2025b, 2024b; Zhang et al. 2024a, 2025e). These papers develop techniques to provide personalized recommendations based on past user behavior, with evaluations on datasets in movie, beauty, and gaming recommendations. A key challenge in this area is handling dataset contamination: the popular dataset MovieLens-1M movie recommendation (Harper and Konstan 2015) has shown some evidence of memorization within LLMs (Di Palma et al. 2025).

Beyond simply capturing user preference, language models have shown evidence of being able to predict user behavior. One such domain is the prediction of step-level user behavior in e-commerce flows, where language model agents attempt to predict which action a user is likely to take in completing a flow, such as clicking, scrolling, or searching (Wang et al. 2025k). Agents have been shown to be capable of simulating step-level user behavior when trained with supervised fine-tuning methods like LoRA (Lu et al. 2025b) and reinforcement learning approaches like GRPO (Wang et al. 2025l). Furthermore, by conditioning on personas during training, models can capture personalized, user-level behavioral differences (Zhang et al. 2025i). Lu et al. (2025c) find evidence that language model agents are capable of performing persona-specific usability testing; Cai et al. (2025a) show that web agents are capable of personalization and introduce a benchmark for this setting.

8.5. Transparency and Explainability

As agentic search systems scale from document retrieval to tool and agent retrieval, transparency must cover not only what output was produced, but also how candidate tools or agents were discovered, ranked, selected, parameterized, invoked, and evaluated. Drawing on explainable information retrieval and surveys of multi-component LLM-agent architectures, we argue that the most actionable explanations for agentic search are often not neuron- or attention-level accounts of a single model, but pipeline-level accounts linking user intent, registry metadata, ranking evidence, selection constraints, tool arguments, and observed execution outcomes (Anand et al. 2022; Wang et al. 2024a).

We use transparency to denote the inspectability of the search process, including candidate sets, metadata fields, ranking signals, selection constraints, tool schemas, and execution traces. We use explainability to denote stakeholder-facing accounts of why a particular tool or agent was preferred to alternatives. Prior work has examined explainable information retrieval (Anand et al. 2022), explainable recommendation (Zhang and Chen 2020), citation and RAG attribution (Gao et al. 2023;Wallat et al. 2025), and inspectable reasoning trajectories and API calls in tool-using language models (Schick et al. 2023; Yao et al. 2022). However, comparatively little work directly studies the full retrieve–rank–select–execute explanation problem for selecting tools or agents from a catalog or registry. We therefore treat the following techniques as transferable components rather than complete solutions.

8.5.1. Decision-Level Explainability in Tool and Agent Search.

Because tool and agent search can be viewed as a ranking and selection problem over structured or semi-structured descriptions, techniques from explainable information retrieval (ExIR) are directly relevant. Anand et al. (2022) provide a comprehensive survey of ExIR using a method-centric taxonomy that distinguishes, among other dimensions, local from global explanations and pointwise, pairwise, and listwise explanations. The survey also reviews method families, including feature attribution, free-text explanations, axiomatic analysis, and interpretable-by-design approaches. In agentic search, this perspective suggests that explanations should be scoped to the relevant stage. In particular, candidate-generation explanations should expose which registry fields, filters, or query rewrites produced the candidate set. Ranking explanations should indicate why one candidate was scored above another, while selection explanations should state why the system invoked one candidate rather than a close alternative.

Local surrogate and feature-attribution methods such as LIME (Ribeiro et al. 2016) and SHAP (Lundberg and Lee 2017) have been adapted to ranking models to produce token-, term-, or featurelevel accounts of ranking decisions. Singh and Anand (2019) developed EXS, an explainable search system that uses LIME to address questions such as why one result is ranked above another. Rank-LIME (Chowdhury et al. 2023) is a post-hoc attribution method for learning-to-rank models, and RankingSHAP (Heuss et al. 2025) is a listwise feature-attribution method for ranking models. Chowdhury et al. (2025) developed a feature-attribution framework for ranking problems and proposed a set of axioms that a valid feature-attribution algorithm should satisfy.

Contrastive explanations offer another useful approach because they address why one tool or agent was selected rather than another (Anand et al. 2022; Miller 2019). A useful contrastive explanation should identify the closest alternatives and the decisive differences among them. In agent and tool search, these methods can explain to users why a particular candidate was preferred over alternative agents or tools at a given turn.

Although these explainability methods can support explanation and diagnosis, they should not be treated as ground-truth accounts of a model’s decision process. Attribution results can be sensitive to methodological choices and may conflict across methods. For example, Fernando et al. (2019) found that DeepSHAP explanations for neural retrieval models are sensitive to the reference input and can differ considerably from explanations produced by LIME. More generally, an explanation’s plausibility to a human does not by itself mean that the explanation faithfully reflects the basis for the model’s decision (Jacovi and Goldberg 2020).

8.5.2. System-Level Transparency and Evaluation.

Beyond decision-level explanations, tool and agent search systems require system-level transparency for the entire decision-making pipeline. A transparent system trace should record the interpreted intent, query transformations, resulting tool calls, observations, errors, retries, and validation steps. These records should support stakeholder-appropriate explanations, such as concise rationales for users, diagnostic traces for developers, and reproducible audit records for governance.

Observable agent trajectories can provide part of this interface. ReAct (Yao et al. 2022) interleaves generated reasoning traces with actions and observations. Such traces can support debugging, but observability by itself does not establish faithfulness. Chain-of-thought explanations may omit influential input biases (Turpin et al. 2023), and interventions on reasoning traces reveal that models rely on them to varying degrees across tasks (Lanham et al. 2023). Moreover, Madsen et al. (2024) demonstrate that faithfulness depends on the explanation, model, and task, showing that self-explanations are insufficient for transparency. Free-form rationales should therefore be treated as summaries or hypotheses about decisions, as stronger explanations require structured traces and causal tests of retrieval, ranking, and selection inputs.

A similar distinction applies to provenance-based explanations. Citation correctness—whether a source supports a claim—does not imply citation faithfulness, defined as whether the model actually relied on that source when producing its response (Wallat et al. 2025). Citation support also remains an open challenge. For example, Gao et al. (2023) showed that even the best-performing evaluated systems lacked complete citation support for 50% of the ELI5 responses in their study. In the context of tool search, a tool may support an outcome without having influenced its selection. Overall, transparency should provide inspectable records of the complete agentic-search pipeline, while explainability should use those records to provide stakeholder-facing accounts of why particular decisions were made.

The five system-level dimensions examined in this section collectively shape whether agent search systems can be deployed responsibly and at scale. Each introduces challenges that extend beyond the core retrieval pipeline: safety must address emergent behaviors in multi-agent systems; fairness frameworks must adapt to dynamic and heterogeneous candidate spaces; security must defend against novel attack surfaces created by open agent ecosystems; personalization must balance user adaptation with the risks of bias amplification and feedback loops; and explainability methods must account for the multi-stage and executable nature of agent selection. Importantly, these dimensions are deeply interconnected: greater personalization may increase susceptibility to manipulation, greater transparency may expose exploitable patterns, and safety mechanisms themselves may be vulnerable to adversarial behavior. Despite growing attention to each area individually, integrated frameworks that address these concerns alongside the core search pipeline remain largely absent. Developing such frameworks where safety, fairness, security, personalization, and explainability are treated as first-class design requirements rather than post-hoc additions remains a key challenge for real-world deployment of agent search systems.

9. Applications

The technical pipeline of agent search—spanning task understanding, capability representation, retrieval, ranking, selection, and evaluation—gains its full significance when examined in the context of real-world deployment. Application domains not only demonstrate the practical importance of agent search as a foundational capability, but also expose domain-specific constraints, requirements, and failure modes that general formulations often overlook. In this section, we focus on three domains where agentic systems are particularly developed and where agent search plays a central role: healthcare, law, and deep research. These domains differ in risk, data modalities, regulation, and task structure, yet share a common need to dynamically identify, select, and compose appropriate agents and tools from heterogeneous spaces to solve complex objectives. In healthcare, modality diversity and the irreversibility of errors demand safety-aware, capability-matched selection of specialist agents. In law, requirements for precision, reproducibility, and legal grounding impose strict constraints on retrieval accuracy and evidence traceability. In deep research, open-ended problem solving relies on iterative decomposition and long-horizon tool and agent selection. Together, these domains highlight that agent search is not merely a system design problem, but a practical necessity wherever heterogeneous agents and tools must be composed to address complex real-world tasks.

9.1. Healthcare

Clinical workflows are inherently multi-step and multi-constraint: a clinician must retrieve patient history, reconcile it against current guidelines, reason under uncertainty, coordinate across specializes, and document decisions in auditable form. Single-turn LLMs lack the persistent state, tool-use capabilities, and self-verification mechanisms needed for this kind of longitudinal reasoning (Wang et al. 2025e). Agentic systems address this by adding planning, retrieval, iterative correction, structure escalation, and a specific medical problem-solving structure. When tasks require simultaneous expertise across specialties such as radiology, pharmacology, and patient communication, multi-agent architectures allow specialized components to collaborate and cross-check one another (Tang et al. 2024). Healthcare further imposes constraints that distinguish it from most deployment domains: consequences of errors are potentially irreversible, patient data is subject to strict regulation, and human professionals must remain meaningfully in the loop. These constraints make healthcare one of the most demanding and consequential testbeds for Agentic AI and, as we argue, one of the clearest illustrations of why agent and tool search is a foundational capability.

We organize healthcare agentic applications into five categories, each with distinct risk profiles and stakeholders derived from (Bedi et al. 2025):

• Clinical Decision Support: diagnosis assistance, risk stratification, differential diagnosis (Rose et al. 2025)
• Clinical Note Generation: discharge summaries, EHR entries, referral letters (Wang et al. 2025j)
• Patient Communication and Education: conversational agents for medication guidance and care navigation
• Medical Research Assistance: literature review, hypothesis generation, experimental support
• Administration and Workflow: scheduling, coding, referral routing (Lu et al. 2024)

Healthcare data spans free text, structured records, medical imaging, lab values, and wearable signals. On one hand, textual only task such as note generation or question answering are among the most mature applications, they are covered by benchmarks such as HealthBench (Arora et al. 2025), and MedAgentBench (Jiang et al. 2025a). On the other hand, Visual and multimodal tasks demand richer tool ecosystems. MMedAgent (Li et al. 2024a) addresses this directly, assembling a multimodal medical agent over six specialized tools spanning seven tasks across five modalities. CARE (Du et al. 2026) goes further, deploying a VLM coordinator that explicitly plans tool invocation across a segmentation model and a grounded reasoning module, then reviews evidence-answer consistency, a clear instance of agent search operating at inference time within a clinical pipeline. MMedAgent-RL (Xia et al. 2026) trains this selection behaviour end-to-end via reinforcement learning, optimising collaboration among vision-language agents across specialities. The Personal Health Agent (Heydari et al. 2025) extends the modality frontier further still, combining consumer device signals with medical records for longitudinal personalised guidance. Collectively, these systems illustrate that tool heterogeneity in healthcare creates precisely the conditions that motivate agent search: a large, diverse, partially overlapping candidate space where appropriate selection depends on task specification, input modality, and deployment context.

A key insight from the healthcare literature is that appropriate system architecture varies with task complexity and that getting this wrong has real costs. A simple complexity ladder guides architecture choice:

• Simple tasks (drug lookups, FAQ responses): a single LLM call suffices; deploying a multi-agent pipeline adds latency and failure surface without benefit
• Moderate tasks (structured note generation, guideline-based triage): a single orchestrated agent with retrieval and memory, as in MedAgents (Tang et al. 2024)
• Complex tasks (differential diagnosis, multi-disciplinary treatment planning, multimodal reasoning): full multi-agent pipelines with role specialization and cross-agent verification, as in MDAgents (Kim et al. 2024) and ColaCare (Wang et al. 2025j).

What emerges across this literature is a recurring design primitive: adaptive routing, the dynamic selection of which agent or tool handles a given task or subtask at runtime. MDAgents (Kim et al. 2024) instantiate this at the pipeline level, automatically selecting a collaboration structure from a solo agent to a full expert panel based on assessed task complexity. TriageAgent (Lu et al. 2024) instantiates it at the task level, using a heterogeneous multi-agent framework with self-confidence and early-stopping mechanisms to route clinical triage decisions. MMedAgent-RL (Xia et al. 2026) makes routing the explicit object of optimisation, training a dedicated triage doctor agent via RL whose sole function is to assign incoming cases to the appropriate specialist agent. These systems differ in framing and scope, but they converge on the same underlying problem: given a task specification and a candidate space of agents or tools, identify the most suitable configuration. This is, precisely, agent search operating locally within a clinical pipeline, and the fact that multiple independent systems have converged on it as a necessary component underscores its foundational status.

Three coordination patterns recur in the healthcare literature. Centralized orchestration uses a planner to delegate subtasks to specialist agents (Wang et al. 2025j) (Kim et al. 2025) (Du et al. 2026). Distributed deliberation assembles peer agents into structured debate to improve reasoning on ambiguous cases (Tang et al. 2024). Adaptive collaboration selects the coordination structure dynamically per query, of which adaptive routing is the most targeted instantiation (Kim et al. 2024), (Xia et al. 2026).

Common specialist roles across systems include: retrieval agents (guidelines, drug databases), patient-context agents (longitudinal summaries), safety checkers (contraindications), escalation agents (uncertainty triage), and compliance agents (privacy enforcement). The explicit instantiation of these roles rather than relying on a general-purpose LLM to handle all concerns implicitly is increasingly recognized as a deployment prerequisite in high-stakes settings, (Kim et al. 2025). But it also presupposes a mechanism for identifying which agent fills which role for a given task: the agent search problem, now made structurally visible.

The two principal opportunities of agentic healthcare systems are clinical workflow automation for reducing documentation burden through note drafting, triage routing, and administrative support (Qiu et al. 2024) and more trustworthy medical AI, through structural redundancy, retrieval grounding, and critic agents that surface inconsistencies before outputs reach clinicians (Kim et al. 2024). Both are contingent on correct agent selection: workflow automation fails if the wrong tool is invoked for a subtask; trustworthiness mechanisms fail if a safety-critical agent is omitted because the orchestrator lacked the means to identify and retrieve it. The risks of agentic healthcare systems such as cascading errors, prompt injection, automation bias, and accountability diffusion (Qiu et al. 2024) are similarly exacerbated when agent selection is poorly specified or opportunistic. Evaluation frameworks are beginning to capture this: (Schmidgall et al. 2025) benchmarks LLMs across nine medical specialties and seven languages in simulated clinical environments with multimodal data and tool use, while (Bedi et al. 2025) provides a clinician-validated taxonomy of 121 tasks and (Arora et al. 2025) evaluates 5,000 multi-turn conversations against physician-authored rubrics. Together they establish that healthcare agent evaluation must be task-specific, modality-aware, and sensitive to tool selection decisions, evaluation requirements that map directly onto the open challenges in agent search.

The personal/hospital agent vision is no longer purely speculative. Agent Hospital (Li et al. 2025d) demonstrates a full hospital simulacrum in which doctor agents evolve through simulated patient interactions, while the Personal Health Agent (Heydari et al. 2025) integrates wearable and medical record data for longitudinal personalized guidance. In such a system, inter-agent communication requires reliable capability discovery: a personal agent must know what a hospital agent can do, what inputs it accepts, and when to invoke it — precisely the agent search problem at system scale. Realizing this vision requires consent frameworks, minimum-necessary-disclosure protocols, escalation logic, and audit traceability, alongside benchmarks that test longitudinal consistency, red-flag escalation, and equity across populations — dimensions that remain largely unaddressed and that define the next frontier for both healthcare agentic AI and agent search research.

9.2. Law

The legal domain is an important and challenging setting for agentic search because the responses that an agent generates must be grounded in relevant legal authorities and structured so that a human reviewer can verify the claims. These stricter requirements mean that a legal agent must decide when its parametric knowledge is insufficient, account for jurisdictional and temporal constraints, understand the nuances of legal terminology, and expose the evidence chain behind its conclusion. Previous studies of legal LLMs have shown that models can hallucinate legal facts and authorities (Dahl et al. 2024) and produce unstable answers to difficult legal questions (Blair-Stanek and Van Durme 2025).

Much of the earlier legal NLP literature evaluates static legal understanding or retrieval, including legal language understanding, holding identification, contract review, legal reasoning, case retrieval, and statute retrieval (Chalkidis et al. 2022; Fei et al. 2024; Guha et al. 2023; Hendrycks et al. 2021; Zheng et al. 2021). More recent legal RAG benchmarks shift attention from only answering legal questions to evaluating whether systems retrieve the right legal evidence. LLegalBench-RAG evaluates the extraction of minimal, highly relevant text segments from legal documents (Pipitone and Alami 2024). Bar Exam QA and Housing Statute QA test the complexity of both legal retrieval and downstream legal question answering in realistic legal RAG settings (Zheng et al. 2025). LexRAG evaluates multi-turn legal consultation, where retrieval must use conversational context to identify relevant legal articles before generation (Li et al. 2025c). Together, these benchmarks motivate evaluating the retrieval process rather than only final answer accuracy.

Recent work more explicitly introduces agentic search and tool use into legal reasoning. Zhou et al. (2026) propose LRAS, formulating legal agentic search as a sequential decision process in which the model alternates between introspective reasoning, search, and answer generation. LRAS trains models to identify knowledge boundaries through Introspective Imitation Learning and to plan multi-step searches through Difficulty-aware Reinforcement Learning. Wang and Yuan (2025) introduce L-MARS, a multi-agent legal QA workflow that decomposes queries into subproblems, performs targeted retrieval over heterogeneous sources, uses a judge agent to verify sufficiency, jurisdiction, and temporal validity, and then synthesizes grounded answers. The same work introduces LegalSearchQA as an evaluation benchmark for legal agentic search. A parallel line of work applies agentic workflows to contract analysis: Watson et al. (2025) deploy modular agents and domain-specific tools for custody and fund-services contracts, while Raptopoulos et al. (2025) introduce PAKTON, an open-source multi-agent RAG framework for long legal agreements that emphasizes explainability, completeness, retrieval quality, and grounded justifications.

Given the shift from static QA toward interactive legal-agent environments, recent work has also changed how these agents are evaluated. Li et al. (2025b) provide LegalAgentBench, a benchmark for Chinese legal-domain agents with external tools, multi-hop reasoning and writing tasks, and progress-based process evaluation. Jia et al. (2026) extend this direction with J1-ENVS and J1-EVAL, where agents operate in dynamic legal settings such as knowledge questioning, consultation, complaint drafting, and defense drafting. Overall, legal agentic search should be treated as evidence-seeking under strict constraints. In particular, successful agents must not only know when and where to search, but also understand legal nuances in the retrieved documents and present traceable conclusions to human reviewers. The central challenge is therefore not simply to add retrieval to a legal LLM, but to build agents whose search trajectories are auditable and aligned with professional legal procedures.

9.3. Deep Research

Deep Research (DR) currently represents a highly prominent frontier agentic scenario. By positioning LLMs as autonomous agents within comprehensive research pipelines, these systems iteratively decompose intricate problems into tractable sub-tasks, dynamically retrieve and verify evidence through external search tools, and integrate validated findings into long-form answers (Shi et al. 2025c). From the perspective of task planning, the predominant open-source approaches currently adopt an iterative, sequential planning paradigm (Chen et al. 2025a; Jin et al. 2025; Li et al. 2025e,j,l; Team et al. 2025b;Wu et al. 2025b), whereas a limited body of work explores parallel architectures to accelerate the process (Chen et al. 2026a; Qin et al. 2025). Furthermore, large-scale trajectory synthesis and the SFT-then-RL training paradigm constitute the principal methodologies for enhancing planning capabilities (Li et al. 2025j; Tao et al. 2025; Team et al. 2025a; Wu et al. 2025b).

Given the highly constrained taxonomy of current search tools (essentially bifurcating into local text corpus retrieval and online search APIs), the DR community has devoted limited attention to tool selection and retrieval strategy optimization. The primary challenges at the tool level are twofold: 1) the misalignment between agent-generated queries and the query formulations required by conventional RAG retrievers (Chen et al. 2026c; Meng et al. 2026), which has garnered increasing scrutiny regarding their mutual adaptation; and 2) the opacity of web search APIs and the noise inherent in retrieved webpages. However, we argue that retrieval per se constitutes an extensively investigated domain within the IR community, where the choice among diverse retrievers and rerankers exerts substantial influence on retrieval efficacy (Meng et al. 2026). Moreover, the performance characteristics of local versus online retrieval diverge considerably across application scenarios. Consequently, we advocate for heightened scholarly attention toward systematic tool selection and retrieval configuration.

These three domains demonstrate that agent search is a foundational mechanism for complex agentic systems. In each case, selecting and composing tools and agents from large heterogeneous spaces is not optional but essential for correctness, safety, and effectiveness. However, each domain also exposes key limitations. Healthcare lacks robust safety- and modality-aware selection mechanisms; law lacks benchmarks capturing procedural and evidential constraints; and deep research lacks systematic treatment of retrieval and tool-selection strategies despite their impact on performance. Across domains, a consistent pattern emerges: as task requirements become more explicit and high-stakes, agent search becomes the primary bottleneck. These observations motivate the challenges discussed in the next section and highlight the need for closer integration between agent search research and domain practice.

10. Open Challenges in Agent and Tool Search

The preceding sections characterized agent and tool search as an end-to-end process spanning task understanding, candidate representation, retrieval, selection, execution, and evaluation. Despite rapid progress in each component, the field lacks a unified account of how executable candidates should be represented, compared, and selected in dynamic environments. This gap arises because agents and tools differ fundamentally from conventional retrieval objects: their value is conditional on execution, changes with the surrounding environment, and may depend on interactions with other selected components. This section outlines four interconnected challenges that must be addressed to establish agent and tool search as a rigorous research area.

10.1. The Retrieval Target and the Scope of Composition

A foundational direction concerns defining what constitutes the retrieval target in agent search, a question first introduced in Section 5.1. Existing systems operate at several granularities, including individual API endpoints, tool packages, agent profiles, policies, and complete workflows (Li et al. 2023b; Wang et al. 2026). However, unlike conventional retrieval settings where the retrieval unit is typically predefined, the appropriate unit in agent search may depend on the task itself. This issue becomes particularly important once composition is considered, as the desired outcome may range from a single executable component to a collection of agents or an entire workflow. We identify three challenges associated with defining this retrieval target.

Agent search may operate over executable entities at substantially different levels of granularity, ranging from individual APIs and tools to autonomous agents and complete agentic systems. Hierarchical routing schemes such as MCP-Zero’s server-to-tool matching and AnyTool’s meta-to-category-to-tool structure (Du et al. 2024; Fei et al. 2025b) provide mechanisms for navigating candidate spaces at multiple levels, but do not by themselves determine which level should constitute the final retrieval target. These alternatives are not interchangeable: retrieving a single tool emphasizes interface compatibility and invocation correctness, whereas retrieving an agent requires reasoning about behavioral competence, autonomy, and operational constraints.

The appropriate search outcome depends on the complexity and requirements of the task. A simple request may be served by one tool or specialized agent, whereas an open-ended or multi-step task may require multiple complementary agents with distinct roles, permissions, and dependencies. Agent search therefore cannot always be formulated as ranking independently relevant candidates; it must also determine how many executable components are required and how they should collectively address the task. The deep research systems reviewed in Section 9.3 illustrate the cost of leaving this question open: with tool choice largely reduced to a binary between local corpus retrieval and online search APIs, the field has given comparatively little systematic attention to which granularity of retrieval target, or which composition of tools, a given research trajectory actually requires.

Once multiple candidates are selected, their utility cannot be estimated independently. An agent that is highly capable in isolation may contribute little within a team because it duplicates another agent’s role, requires incompatible resources, or introduces excessive coordination overhead. Conversely, a moderately capable agent may substantially improve the overall solution by providing a missing or complementary capability. Search objectives must therefore account not only for individual candidate quality, but also for complementarity, redundancy, dependencies, compatibility, and composition feasibility. Developing formal retrieval objectives that capture these interactions and estimate the collective utility of composed solutions remains an important open challenge.

10.2. Capability Modeling Under Partial Observability and Uncertainty

A second research direction concerns how the capabilities of agents and tools can be reliably modeled before seearch, extending the representational surfaces surveyed in Section 5.4 from description toward capability estimation. Tools often expose structured interfaces, documentation, and input–output schemas that can support indexing and retrieval. Agents, in contrast, are behavioral entities whose performance depends on their underlying model, prompt or policy, accessible tools, interaction history, runtime environment, and resource constraints. Descriptive metadata can therefore indicate what an agent claims to do, but does not necessarily provide a reliable estimate of what the agent can accomplish on a particular task. We identify three challenges toward reliable capability modeling under such partial observability.

Existing representations capture useful signals, including natural-language descriptions, schemas, usage examples, historical traces, and empirical performance. However, these signals provide different and often incomplete views of candidate capability. Declarative descriptions indicate intended functionality but may not reflect actual performance, while behavioral evidence provides stronger empirical signals but is typically available only for a limited set of previously observed tasks. Capability modeling must therefore integrate heterogeneous evidence and generalize beyond observed interactions to estimate how a candidate is likely to perform on a new task.

Agent capability is not an intrinsic and deterministic property of a candidate. An agent may be appropriate for a task only under a particular repository state, data-access policy, model version, execution budget, or runtime environment. Future systems should therefore move beyond static capability labels toward calibrated capability models that quantify uncertainty over expected outcomes and retain the context and provenance under which supporting evidence was obtained. Such models must further account for operational properties often omitted from semantic representations, including availability, latency, reliability, permission requirements, safety constraints, and compatibility with other candidates. In healthcare deployments (Section 9.1), for instance, a capability claim is meaningful only relative to the modality, patient population, and clinical protocol under which it was demonstrated.

Capability evidence can become stale as tools are updated, APIs are deprecated, permissions change, underlying models are replaced, or agents acquire new capabilities. StableToolBench addresses one aspect of this problem for tool APIs by virtualizing execution against a cached simulator (Guo et al. 2024), while continual documentation adaptation begins to address evolving tool representations as toolsets change (Wu et al. 2026c). Nevertheless, maintaining accurate agent-level capability estimates remains largely unresolved. Future systems require mechanisms for tracking capability changes, updating representations from new observations, preserving the freshness and provenance of supporting evidence, and determining when existing capability estimates should be revised or discarded. Establishing such continuously evolving capability models while retaining efficient large-scale retrieval remains an important open challenge.

10.3. Similarity Matching Versus Constrained Utility Estimation

A third research direction concerns how agent search should rank and select candidates once potentially suitable agents or tools have been identified. Classical retrieval models primarily rank candidates according to query–document relevance, following the dominant paradigm reviewed in Section 6.1. Although semantic matching remains useful for candidate generation, it is insufficient as the final decision criterion for agent and tool search. A candidate may be semantically related to a task yet be unsuitable because it is unreliable, unavailable, unsafe, expensive, or incompatible with the execution environment. Conversely, a candidate with limited lexical or semantic similarity may provide a capability essential for successful task completion. Agent search therefore requires moving from similarity-based relevance estimation toward estimating the expected utility of executable candidates under task and operational constraints. We identify three challenges toward this objective.

Given a task, user requirements, an execution environment, and a set of available candidates, the system must estimate how much an agent, tool, or composition is expected to contribute to successful task completion. This requires reasoning beyond task–candidate similarity to account for whether the candidate possesses the required capabilities and can effectively contribute to the intended solution. The problem becomes particularly challenging for multi-step tasks, where successful execution may require a complete and diverse set of complementary capabilities rather than a ranked list of individually similar candidates (Qu et al. 2024).

The utility of a candidate may be non-stationary because its value depends on the current task state, intermediate observations, and agents or tools that have already been selected. A candidate that appears unnecessary initially may become essential after execution reveals a missing capability or unexpected constraint, while an earlier selection may become redundant as the task evolves. Selection may therefore need to be iterative, extending the iterative retrieval strategies surveyed in Section 6.2 from recovering missing tools to recovering missing capabilities discovered during execution. Agent search must consequently support updating utility estimates, retrieving additional candidates, and revising previous selections as new evidence becomes available.

Task success alone is insufficient to determine whether a candidate constitutes an appropriate selection. Search decisions may simultaneously depend on cost, latency, reliability, availability, user preferences, permissions, safety, and security, and these objectives may conflict with one another. In particular, safety and security cannot simply be treated as post-hoc filters: a composition that maximizes task performance may remain unacceptable if it requires excessive permissions or introduces harmful interactions (Shabadi and Alur 2025). This concern is especially important in safety-critical healthcare deployments discussed in Section 9.1 and the broader safety and security issues surveyed in Section 8.1. Developing principled multi-objective and constraint-aware ranking mechanisms that balance these competing factors while accounting for uncertainty in their estimates remains a central research challenge.

10.4. Evaluation in Dynamic and Open Candidate Ecosystems

A fourth research direction concerns how agent and tool search systems should be evaluated when both the retrieved objects and their downstream behaviors are executable, stochastic, and continuously evolving. As discussed in Section 7, intrinsic measures such as Recall@k, MRR, and NDCG assess whether appropriate candidates are surfaced, while extrinsic evaluation measures whether selected candidates successfully accomplish downstream tasks. Neither perspective alone is sufficient: retrieval correctness does not guarantee successful execution, while end-to-end success provides limited insight into whether failures arise from retrieval, selection, invocation, or execution. Evaluation must therefore account for both the quality of search decisions and the utility of their resulting executions. We identify three challenges toward developing such evaluation frameworks.

Unlike conventional retrieval settings with relatively stable relevance judgments, many agent search tasks admit multiple valid solutions. Different tools, agents, or workflows may successfully accomplish the same task, and their relative suitability may further depend on user preferences and operational constraints. Evaluation based on a fixed set of annotated candidates can therefore incorrectly penalize valid alternatives and becomes increasingly difficult to construct as candidate spaces grow. Existing work has begun to address related issues through metrics such as retrieval completeness (Qu et al. 2024), but developing evaluation protocols that recognize alternative candidates and compositions while still distinguishing their relative quality remains an open challenge.

Retrieving an appropriate candidate does not establish that it can be invoked correctly, composed effectively, or reliably complete the task. End-to-end evaluation provides a more direct measure of utility, but execution is often expensive and stochastic, making outcomes sensitive to individual trials and runtime conditions. Evaluation protocols must therefore assess not only task success, but also consistency across repeated executions, efficiency, safety, and compliance with task-specific constraints. Existing benchmarks demonstrate the importance of invocation accuracy and repeated execution, with $\tau$-bench introducing pass⌃k to measure whether an agent succeeds consistently across multiple trials (Yao et al. 2025), while recent agent-search benchmarks begin to examine execution-grounded evaluation at the agent level (Wu et al. 2026b). Nevertheless, comprehensive protocols that jointly connect retrieval decisions with downstream execution quality remain underdeveloped. This limitation is particularly visible in domains such as law (Section 9.2), where evaluation must capture procedural and evidential requirements rather than final-answer accuracy alone.

Agent and tool ecosystems are inherently dynamic: new candidates appear, existing tools are updated or deprecated, documentation changes, and observed performance may shift across versions and execution environments. Consequently, static benchmark collections can rapidly diverge from real deployment conditions, making both candidate annotations and measured performance stale. Future evaluation frameworks should incorporate versioned candidate registries, controlled execution environments, explicit freshness assumptions, and longitudinal measurements of retrieval and task performance. Such protocols are necessary to distinguish systems that perform well on a fixed snapshot from those that remain effective and reliable as the underlying candidate ecosystem evolves.

11. Conclusions

This survey presents agent and tool search as an emerging research problem concerned with identifying, comparing, and selecting executable systems for a given task. Unlike conventional information retrieval, where relevance is primarily defined over information artifacts, agent and tool search operates over candidates whose suitability depends on what they can actually accomplish. Their capabilities may be only partially observable from descriptions, their behavior may vary across tasks and execution environments, and their utility may depend on factors such as reliability, cost, safety, and compatibility with other selected components. Agent and tool search therefore extends retrieval from matching information needs to information artifacts toward selecting executable capabilities for task completion.

To provide a unified view of this problem, we developed a conceptual framework around task specification, candidate search space, and search outcome, and organized the existing literature along the end-to-end process of task understanding; agent and tool discovery, representation, and indexing; retrieval, reranking, and selection; and evaluation. We further considered the system-level dimensions that influence these decisions and examined how agent search arises in representative application domains. Across these areas, a common picture emerges: effective agent search requires moving beyond description-based similarity toward capability-aware and execution-grounded decision making, while accounting for uncertainty, constraints, composition, and the dynamic nature of agent and tool ecosystems.

Despite rapid progress, many of the foundations of agent and tool search remain open. In particular, future research must better determine what should be retrieved and composed, how candidate capabilities can be estimated under partial observability, how relevance and utility should be defined beyond semantic similarity, and how search systems should be evaluated as candidates and their capabilities evolve. Addressing these challenges will require closer interaction among information retrieval, multi-agent systems, tool learning, agent evaluation, and AI systems research. As agent ecosystems continue to grow in scale, diversity, and autonomy, the ability to reliably discover and select appropriate executable capabilities will become an increasingly important component of agentic AI systems. We hope this survey provides a common foundation for studying this emerging problem.