A Survey on AI Search with Large Language Models

1. Introduction

Searching for information is a fundamental daily necessity for humans. To meet the demand for rapid access to desired information, key web search technologies like PageRank  [1,2,3] have been developed to support information retrieval systems. These technologies power search engines such as Google, Bing, and Baidu, efficiently retrieving relevant web pages in response to user queries and providing convenient access to information on the internet. Advances in natural language processing (NLP) [4] and information retrieval (IR) [5] have further enhanced machines’ ability to accurately extract content from the vast array of websites available online. However, as user queries become increasingly complex and the demand for precise, contextually relevant, and up-to-date responses grows, traditional search technologies face challenges in fully comprehending intricate human intentions. Consequently, users often need to manually open, read, and synthesize information from multiple web pages to answer complex questions.

Recently, Large Language Models (LLMs) [6] have garnered significant attention in both academic and industrial domains. LLMs such as ChatGPT [7] and LLaMA [8] have demonstrated remarkable advancements in language understanding, reasoning, and information integration. However, LLMs face limitations in acquiring external knowledge and accessing the most current information. To address these challenges, researchers are integrating the impressive capabilities of LLMs with search engines and websites, aiming to enhance real-time evidence gathering and reflective reasoning. The complementary strengths of LLMs and search engines present an opportunity for synergy, where the reasoning abilities of LLMs are augmented by the vast web information accessible through search engines. This integration is revolutionizing the way we seek and synthesize web-based information, ushering in a new era of search technology known as Artificial Intelligence (AI) Search. In this survey, we provide an overview of recent advancements in the rapidly evolving field of AI Search. As depicted in Figure 1, we categorize the literature into five primary areas: (1) text-based AI search, (2) web browsing agents, (3) multimodal AI search, (4) benchmarks, and (5) software and products.

The classic Text-based AI Search operates through a Retrieval-Augmented Generation (RAG) framework [108]. In this workflow, RAG retrieves relevant passages from search engines based on the input query and integrates them into the context of a Large Language Model (LLM) for generating responses. This enables the LLM to utilize external knowledge when addressing questions. Another approach within text-based AI Search is the deep search method, which acquires external information by interacting with search engines as part of an end-to-end coherent reasoning process to tackle complex information retrieval challenges. Unlike predefined workflows, this method allows the model to autonomously determine when to employ search-related tools during its reasoning, enhancing flexibility and effectiveness. Web Browsing Agent accomplishes specific tasks on target websites through a sequence of actions, utilizing a thought-action-observation paradigm. For instance, if a web agent is tasked with calculating the driving time from Shanghai to Beijing using an open street map, it would perform this task by interacting with the website. Web agents are classified into two primary paths: Generalist Deep Browsing Web Agents, which perform complex web browsing tasks across multiple types of web pages, and Specialist Parsing Web Agents, which employ dedicated training procedures to focus specifically on action sequences or interface elements. Additionally, with the emergence of visual web-oriented benchmarks and the development of Multimodal Large Language Models, many agents now incorporate screenshots as sensory input to provide a more comprehensive understanding of the web environment. Unlike Multimodal AI Search, most current AI search methods are confined to text-only settings, overlooking the multimodal nature of user queries and the interleaved text-image information on websites. This limitation is particularly significant given the complexity and interleaved nature of modern websites. For example, capturing a photo of an antique at a museum without knowing its historical context could be addressed by a multimodal AI search engine, which would match the photograph with an interleaved table of images and text retrieved from the Internet, thereby providing the history and story behind it. Thus, a multimodal AI search engine is essential for advancing information retrieval and analysis.

Furthermore, this paper reviews the Benchmarks relevant to these methods. Evaluating the search capabilities of AI models, particularly LLM, is crucial for assessing their ability to effectively retrieve, filter, and reason over web-based information. This evaluation is essential for understanding the true web-browsing competence of LLMs and their potential to tackle real-world tasks that demand dynamic information retrieval. In recent years, significant efforts have been made to explore AI search from various perspectives. This paper concentrates on three key areas: text-based question-answering benchmarks, web agent benchmarks, and multimodal benchmarks. The Software and Products of AI Search, such as Perplexity [93], have the potential to change our daily lives. We introduce a wide array of state-of-the-art open-source and proprietary models, software, and mainstream AI search products, aiming to present a diverse and comprehensive overview of AI Search. Finally, we discuss the limitations of current AI search methods and explore promising future directions. To illustrate the evolution of AI search methods over time, Figure 2 presents a timeline of recent AI search technologies, related methods, and products.

2. Text-Based AI Search

AI search represents a transformative advancement in information retrieval systems, evolving from traditional search engines to sophisticated approaches incorporating RAG workflows and Deep Search capabilities, as shown in Figure 3. This section provides an overview of the key components and cutting-edge developments in modern text-based AI search technologies.

2.1. Traditional Search Engines

Traditional search engines form the foundation of modern search engines. They employ a variety of techniques to efficiently process user queries and return relevant results. Two key components of these systems are document retrieval and post-ranking, which work in tandem to provide users with the most pertinent information [109].

Document Retrieval

Document retrieval involves identifying relevant documents from a collection based on a user query. It is a crucial step in information retrieval, as it determines which documents are most relevant to the user’s query. Traditional document retrieval systems typically employ techniques such as inverted indexing, term frequency-inverse document frequency (TF-IDF), and BM25 models [110,111]. More advanced approaches incorporate semantic matching using dense vector representations and neural ranking models [112,113,114,115]. The retrieval process often involves query preprocessing, document indexing, similarity computation, and efficient search algorithms to handle large-scale document collections. Recently, some research has explored LLM-based generative retrieval [116,117,118,119], eliminating the need to build document indexes and directly generating document identifiers through LLMs.

Post-Ranking

Post-ranking refines the results of a search query after the initial retrieval stage. It is used to improve the quality of the search results by applying additional filters and reranking algorithms. Post-ranking systems typically employ learning-to-rank algorithms, neural reranking models, and LLM-based reranking models that combine multiple ranking signals [120,121,122]. This stage is crucial for improving search precision and user satisfaction by promoting the most relevant documents to top positions [120].

2.2. Retrieval-Augmented Generation with Pre-Defined Workflows

Retrieval-Augmented Generation (RAG) enhances generative models by integrating a retrieval mechanism, allowing the model to ground its responses in external, reliable knowledge [9,123]. Typically, a RAG system consists of a retriever and a generator, and the interaction between these components gives rise to four main RAG paradigms [108]:

Sequential RAG

Sequential RAG follows a linear ”retrieve-then-generate” workflow, where the retriever first fetches relevant documents and the generator produces the final response based on these documents [9,123,124,125,126,127]. Early works explored joint or separate training of the retriever and generator, while recent approaches often use a frozen generator and focus on optimizing the retriever [128,129,130]. Pre-retrieval modules, such as rewriters [126] and post-retrieval compressors [10,11,12,131,132,133] further improve efficiency and response quality.

Branching RAG

Branching RAG processes the input query through multiple parallel pipelines, each potentially involving its own retrieval and generation steps, and then merges the outputs for a comprehensive answer [13,14,15,16,134]. This approach enables finer-grained handling of complex queries, such as decomposing questions into sub-questions [13], augmenting queries with additional knowledge [16], or merging generated and retrieved content [14,15].

Conditional RAG

Conditional RAG introduces a decision-making module to adaptively determine whether retrieval is necessary for a given query, improving flexibility and robustness [17,18,19,20]. Methods include training classifiers to predict the need for retrieval [17,20], using model confidence to guide retrieval [18], or employing consistency checks across perturbed queries [19].

Loop RAG

Loop RAG features iterative and interactive retrieval-generation cycles, enabling deep reasoning and handling of complex queries [21,22,23,24,25,135,136]. These methods alternate between retrieval and generation [22,23], dynamically decide when to retrieve [24,25], or decompose and answer sub-questions with verification steps to reduce misinformation [135,136].

2.3. End-to-End Deep Search Within Reasoning Process

Unlike traditional RAG workflows, Deep Search methods integrate external knowledge acquisition by utilizing search engines within an end-to-end coherent reasoning process to address complex information retrieval challenges. This approach eliminates the need for predefined workflows, allowing the model to autonomously determine when to employ search-related tools during its reasoning process, thereby enhancing flexibility and effectiveness [137,138,139].

Training-Free Methods

These methods aim to augment the reasoning model’s search capabilities by crafting instructions that clarify the model’s task and the use of search tools. Initially, Search-o1 [26] introduced an agentic RAG mechanism enabling the reasoning model to autonomously retrieve external knowledge when faced with uncertain information during the primary reasoning process, thus addressing the knowledge gaps in long Chain-of-Thought (CoT) reasoning. They also introduced a Reason-in-Documents process, which thoroughly analyzes the content of retrieved documents after each search call, providing concise and useful information to the main reasoning chain. Experiments demonstrated significant performance improvements across mathematical, scientific, coding, and multi-hop question answering tasks.

Following this paradigm, a series of works such as WebThinker [27], WebDancer [28], ManuSearch [29], and HiRA [31] have proposed advanced frameworks. Typically, these methods incorporate browsing of collected webpage URLs to facilitate in-depth web exploration. Additionally, to enhance search efficiency, SearchAgent-X [30] proposed an efficient reasoning framework aimed at increasing system throughput and reducing latency through high-recall approximate retrieval, priority-aware scheduling, and non-stagnant retrieval mechanisms. Beyond directly answering users’ information-seeking questions, some works like WebThinker [27] have explored autonomously writing research reports while gathering information, offering users more comprehensive and cutting-edge knowledge.

Training-Based Methods

These methods design various training strategies to incentivize or enhance the LLM’s search capabilities within the reasoning process. These strategies encompass pre-training, supervised fine-tuning (SFT), and reinforcement learning (RL).

During pre-training, the MaskSearch [32] framework introduces a Retrieval-Augmented Mask Prediction (RAMP) task, which trains the model to use search tools to fill in masked text, thereby enhancing its retrieval and reasoning abilities.

For Supervised Fine-Tuning (SFT), several methods focus on synthesizing long chain-of-thought data that incorporates search actions [28,29,33,34,35,36,44,140]. Specifically, CoRAG [33] addresses the lack of intermediate retrieval steps in existing RAG datasets by automatically generating retrieval chains through rejection sampling. ReaRAG [34] avoids complex reinforcement learning by building a specialized dataset for fine-tuning via policy distillation. ExSearch [35] introduces an iterative self-incentivization framework based on the Generalized Expectation-Maximization (GEM) algorithm, enabling the model to learn from its own generated search trajectories. SimpleDeepSearcher [140] simulates user search behavior in a real-world web environment to synthesize multi-turn reasoning trajectories, which are then curated using a multi-criteria strategy. ManuSearch [29] leverages its multi-agent framework to decompose the deep search process and generate structured reasoning data. Lastly, WebCoT [36] synthesizes training data by reconstructing successful and failed trajectories, explicitly embedding reasoning skills like reflection, branching, and rollback into the chain of thought.

Furthermore, RL-based training has recently garnered significant attention. Some works leverage Direct Preference Optimization (DPO) [141]. For instance, DeepRAG [37] introduces a chain of calibration method to refine the model’s atomic decisions, thereby synthesizing preference data for training. WebThinker [27] constructs positive and negative pairs based on the model’s ability to correctly complete research tasks while efficiently using tools. By iteratively constructing data and training the model with DPO, it implements on-policy RL training, improving performance on complex reasoning and report generation tasks.

Another line of work has explored training strategies based on PPO [142], GRPO [143], REINFORCE++ [144], and others. Initially, Search-R1 [38], R1-Searcher [39], ReSearch [40], WebSailor [145], and WebShaper [45] used the accuracy of the generated answer as a rule-based reward to encourage the LLM to use Wikipedia-based search tools during reasoning, achieving significant performance improvements in multi-hop QA tasks. Subsequently, a series of studies have investigated various enhancement strategies. These include leveraging web search capabilities [28,42,44,47,146], refining retrieved information [42,43,147], enabling multi-tool usage [148], developing improved sampling techniques [149], designing advanced reward functions [150], combining outcome and process rewards [46,151], enhancing training efficiency [41], and implementing iterative SFT and RL training cycles [152]. To optimize search efficiency, methods such as SEM [153], $\beta$-GRPO [154], and s3 [155] have been proposed, which design training algorithms and reward functions for more efficient and accurate use of search tools.

3. Web Browsing Agent

The Web Agent is an AI-driven autonomous program designed to mimic human interactions within web browsers. Unlike AI search, the task of Web Agent is not limited to search. It is also specified in the network environment and even requires executing a series of specific interactive operations. This section explores the definition, classification, and advanced technologies related to Web Agent more broadly.

3.1. Agent

Agent [156] is defined as an autonomous intelligent entity with human-like decision-making abilities, capable of perceiving diverse environments, performing various tasks, and taking concrete actions. Traditional autonomous agents are typically based on rule-based approaches and tend to perform well only in specific tasks or closed environments. Therefore, agent based on large models(LM) has become a prominent research direction [157].

The core of LM-based autonomous agent lies in two aspects: architectural design and capability acquisition [158]. Architectural design [159,160,161,162] involves employing specialized network structures and modular components to enhance the agent’s comprehension and execution abilities. (e.g., AgentVerse’s unified framework [162]). Capability acquisition [47,163,164] focuses on optimizing task-specific performance through parameter adjustment or strategic prompt design, enabling agents to adapt to target domains.

Currently, agents can be categorized based on their architecture into single-agent and multi-agent systems [165,166,167]. Typically, a single-agent architecture can independently work without relying on other agents or user feedback, making it more efficient when addressing well-defined problems. In contrast, multi-agent architecture deploys specialized agents across distinct knowledge domains, enabling collaborative problem-solving that excels in complex, cross-domain tasks requiring iterative coordination.

As an extension of agent technology in vertical domains, Web Agents focus specifically on task generalization within more complex web environments [63]. According to their adaptation strategies, current Web Agents can be classified into two categories, as shown in Figure 4 and Figure 5. The first category is post-training-based Web Agents, which employ SFT and RL to adapt models, enhancing their understanding and execution of specific web tasks. The second category is prompt-based Web Agents, which rely on prompt engineering or context engineering to improve performance in complex web navigation tasks without directly modifying model parameters [168].

3.2. Post-Training-Based Web Agents

The open-ended nature of Web Agent applications limits the effectiveness of standard pre-trained models. To address this, techniques such as SFT and RL have become essential, enabling Web Agents to adapt to dynamic environments and learn from interactive feedback, thereby improving their performance on complex web tasks.

Reinforcement Learning

RL allows Web Agents to adapt to dynamic environments in real-time by exploring and learning from interactive feedback [28,47,169,170]. WebAgent-R1 is the first purely end-to-end RL-trained Web Agent [47]. It employs a multi-turn end-to-end RL framework, where the agent is trained through online interactions guided by rule-based outcome rewards. During training, it implements multi-group GRPO [171] method, utilizing multiple parallel interaction trajectories to enhance training efficacy.

The approach of WebAgent-R1 follows a standard RL process, while other methods adjust strategies within the framework. WebDancer [28] focuses on QA pair parsing, employing Decoupled Clip and Dynamic Sampling Policy Optimization [172] to internalize chains-of-thought (CoT) [173] from QA pairs as active behavioral components of the model. WebRL [169] adopts self-evolving online curriculum RL to dynamically generate new tasks and uses result-supervised reward models to provide feedback to the agent. WorkForceAgent-R1 [170] integrates behavior cloning (BC) [174] and GRPO, emphasizing the optimization of both single-step reasoning and planning abilities to help agents adapt to dynamic web environments.

Supervised Fine-Tuning

SFT uses labeled data to fine-tune a pre-trained model, transferring its general capabilities to specific tasks or domains [55,163,175,176]. For example, WebGUM [163] integrates the T5 [177] model with Vision Transformer (ViT) [178] for multimodal task processing. During SFT, ViT encodes images into tokens while T5 performs unified encoding of both text and tokens, enabling joint fine-tuning.

The focus of SFT is on the construction of task-aligned datasets to enhance tuning performance. Falcon UI [55] simulates realistic interactions by exclusively recording visible GUI [179] elements during browsing, constructing specialized datasets for improved GUI comprehension. WebExperT [175] refines existing datasets by annotating screenshot elements, combining rapid decisions with slow reasoning, and optimizes through self-reflection. WebCoT [176] generates “reasoning trajectories” through reflection, branching, and rollback mechanisms, subsequently converted into CoT data for SFT.

Joint Training

SFT can be combined with RL for complementary benefits. AutoWebGLM [48] processes information through HTML simplification and optical character recognition (OCR) [180]. Its training integrates SFT, RL, and rejection sampling fine-tuning (RFT) [48]. During SFT, curriculum learning (CL) [181] is used to enable the model to perform basic web navigation and improving the effectiveness of subsequent RL training.

3.3. Prompt-Based Web Agents

Although post-training methods are effective, prompt engineering [182] is preferable when a Web Agent must adapt quickly to dynamic environments or new tasks without the time or resources for SFT or RL. Both prompt engineering and its advanced form, context engineering [183], do not require complex model modifications; they guide model behavior by optimizing the input data.

A typical application of prompt engineering is the use of APIs [49,184,185]. For example, Microsoft’s CodeAct [49] adopts an “API-first” strategy: the agent first constructs an API documentation map of the website, then selects relevant APIs according to task requirements, and finally processes inputs via model APIs. Similarly, Infogent [184] establishes a modular feedback loop around LLM APIs: the Navigator collects raw data, the Extractor processes information via APIs, and the Aggregator evaluates results.

In theory, the more relevant information a model receives, the more accurate its decisions. Consequently, many Web Agents enrich prompts or context as much as possible [51,53,64]. WebVoyager [53] captures the web page’s accessibility tree, annotates interactive elements, and takes a screenshot; these components are combined as the final prompt to the GPT-4V [186] API to determine the next action. AgentCocam [51] incorporates diverse information into prompts by abstracting operations, merging elements, and selectively replaying interaction history, enabling an efficient LLM-API workflow.

In some cases, Web Agents may employ multiple models. PhishAgent [58], designed for phishing site detection, primarily relies on LLMs. When textual cues are insufficient, it activates a large multimodal model(LMM) to perform brand analysis, enhancing decision accuracy.

4. Multimodal AI Search

Current AI search methods are predominantly confined to text-only environments, often overlooking the multimodal nature of user queries and the intertwined text-image format of website content. This limitation is particularly significant when, for example, a user takes a photograph of an antique at the museum but lacks specific information about it. Therefore, the development of a multimodal AI search engine is essential for enhancing information retrieval and analysis.

4.1. Multimodal Large Language Models

Recently, Multimodal Large Language Models (MLLMs) or Large Multimodal Models (LMMs) [187] have demonstrated exceptional performance across a range of applications, including visual question answering (VQA), visual perception, understanding, and reasoning. Notable closed-source models include GPT-4V [186], GPT-4o [188], and Claude 3.5 Sonnet [189]. In the open-source domain, models such as BLIP [190,191], LLaVA [192,193], Qwen-VL [194], Gemini [195], InternVL [196], and EMU [197] have made significant advancements. The typical MLLM framework consists of three primary modules: a visual encoder responsible for processing visual inputs, a pre-trained language model that handles multimodal signals and performs reasoning, and a visual-language projector that serves as a bridge to align the two modalities. In recent years, substantial efforts have been dedicated to designing MLLM benchmarks [198], examining these models from various perspectives.

4.2. Multimodal Search

Inspired by Text-based AI Search, it is essential to develop a framework for MLLMs to function as multimodal AI search engines. MMSearch [59], illustrated in Figure 6, proposes a multimodal AI search engine pipeline named MMSEARCH-ENGINE, which enhances MLLMs with advanced search capabilities. MMSEARCH-ENGINE maximizes the use of MLLMs’ multimodal information comprehension abilities by incorporating both visual and textual website content as information sources during the searching process: requery, rerank, and summarization. MMSearch-R1 [60], also shown in Figure 6, represents an initial effort to equip MLLMs with active image search capabilities through an end-to-end reinforcement learning framework, assisted by image search tools. This method trains models not only to determine when to invoke the image search tool but also to effectively extract, synthesize, and utilize relevant information to support downstream reasoning. To improve the performance on knowledge-intensive VQA tasks, WebWatcher [199] use a multimodal Agent for Deep Research equipped with enhanced visual-language reasoning capabilities to achieve strong performance on several high-difficulty benchmarks.

4.3. Multimodal Web Agents

The rise of multimodal large models has driven many LLM-powered Web Agents toward multimodal capabilities [200]. By jointly encoding visual inputs and textual inputs like HTML/DOM [201], these agents enable integrated analysis and decision-making [61,62,63,64,202]. WebVoyager [202] first instantiates a web browser and utilizes both visual signals (screenshots) and textual signals (HTML elements) to perform actions. Its successor, OpenWebVoyager [63], adopts the more capable multimodal Idefics2 model and employs SFT, enabling direct understanding of image and text inputs and reducing dependence on prompt engineering. SEEACT [64] leverages a GPT-4V-like MLLM for integrated visual understanding, generating textual plans and performing website actions based on HTML elements. The optimization of multimodal web agents relies on high-quality multimodal datasets. Explorer [61] independently developed a web trajectory synthesis approach to construct the largest and most diverse trajectory dataset to date for GUI agent model training.

Beyond general domains, multimodal web agents also demonstrate high usability in specialized fields [58,65,203]. For instance, researchers at the University of Notre Dame developed a multimodal web agent [203] anonymously navigating dark web marketplaces, solve CAPTCHAs, and extract product information via the Tor browser.

5. Benchmarks

5.1. Text-Based QA Benchmark

As large language models (LLMs) evolve into tool-using agents, the ability to browse the web in real-time has become a critical yardstick for measuring their reasoning and retrieval competence. A variety of widely used English benchmarks have been proposed to assess retrieval capabilities, including TriviaQA, HotpotQA, FEVER, KILT, GAIA, etc. These datasets cover multi-hop reasoning, knowledge-intensive QA, and fact checking, typically relying on structured sources like Wikipedia and StackExchange.

Traditional Benchmarks Natural Questions (NQ)[66] is a large-scale QA dataset using real Google search queries and corresponding Wikipedia pages, requiring models to provide both long-form and short-form answers. TriviaQA [67] is a reading comprehension dataset characterized by complex, compositional questions with significant lexical variation from their evidence, often demanding multi-sentence reasoning. PopQA [68] is an entity-centric QA dataset designed to test factual knowledge recall across a long-tail distribution of entity popularity. HotpotQA [69] is a multi-hop QA dataset that requires reasoning across multiple documents and providing sentence-level supporting facts, making it a benchmark for explainable QA. 2WikiMultiHopQA [204] is a more challenging multi-hop QA dataset that integrates Wikipedia with Wikidata, using structured triples to explain complex reasoning paths. MuSiQue [70] is a multi-hop QA dataset emphasizing connected reasoning and includes unanswerable examples to challenge models that rely on shortcuts. FEVER [71] is a benchmark for fact verification, requiring systems to classify claims as SUPPORTED, REFUTED, or NOTENOUGHINFO against Wikipedia and provide sentence-level evidence. KILT [72] unifies 11 knowledge-intensive NLP tasks under a single Wikipedia snapshot, providing a standardized framework for evaluating both task performance and evidence retrieval. GAIA [73] evaluates general-purpose AI assistants with real-world questions that require a combination of reasoning, tool use, and multi-modality, revealing a large gap between AI and human performance. TREC Health Misinformation Track [205] provides datasets with binary “yes/no” health questions based on medical consensus to evaluate a system’s ability to combat health misinformation.

Modern Browsing Benchmarks While traditional benchmarks mentioned above have effectively measured an AI’s ability to retrieve straightforward information through basic queries (e.g., single-hop fact lookup), their simplicity has led to saturation—modern models now achieve near-perfect scores on these tasks. This progress reveals a critical gap: real-world information needs often require persistent navigation through complex data landscapes. These challenges mirror the evolutionary jump from arithmetic tests to mathematical proofs—where success depends less on recall and more on strategic problem-solving.

BrowseComp [74] is a benchmark dataset introduced to evaluate web-browsing AI agents. It contains 1,266 challenging questions requiring persistent navigation of the internet to find entangled information. Key features include: (1) High difficulty - questions are designed to be unsolvable by humans within 10 minutes; (2) Verifiability - short reference answers enable easy validation; (3) Diverse topics spanning sports, fiction, and academic publications; and (4) Core capability measurement focusing on persistence, factual reasoning, and creative search strategies. BrowseComp-ZH[75] benchmark is a high-difficulty Chinese web browsing evaluation dataset consisting of 289 multi-hop questions across 11 domains (e.g., Art, Film&TV, Medicine). Each question is reverse-engineered from verifiable factual answers and undergoes rigorous two-stage quality control to ensure retrieval difficulty and answer uniqueness. Figure 7 illustrates these two benchmarks and shows some complex and challenging queries. Mind2Web 2 [76] is also a modern benchmark with 130 realistic, high-quality, and long-horizon tasks that require real-time web browsing and extensive information synthesis.

5.2. Web Agent Benchmark

As integrated systems beyond standalone models, Web Agents require specialized evaluation standards. Web Agent Benchmarks address this need through standardized test tasks and assessment frameworks that quantify performance in navigation, operation, and reasoning within simulated web environments. Current benchmarks fall into two categories: general benchmarks [77,78,79,80,81,82,83,84,168].and specialized benchmarks [85,86,87,88,89,90].

General Benchmarks General benchmarks evaluate Web Agents’ open-ended browsing capabilities across diverse domains. Mind2Web [77] , shown in Figure 8, pioneering this approach through tasks spanning five domains (travel, shopping, services, entertainment, information) represented by task descriptions, action sequences, and webpage snapshots. WebArena [78] builds a reusable virtual web environment based on real-world site categories (e-commerce, forums, collaboration tools, CMS), minimizing real-world noise, while WebChoreArena [81] extends this framework with memory-intensive tasks, reduced ambiguity, and template-based generation, making it more challenging than WebArena. WebCanvas [83] employs a dynamic, real-time evaluation framework focused on key task steps (required task steps) to assess agent performance in live web environments. VisualWebArena [79] adds multimodal inputs(screenshot) to WebArena tasks to evaluate agents under visual challenges. REAL [80] provides a framework for multi-turn agent evaluation on simulated real-world websites, supporting both open-source and proprietary agents via black-box commands within a browser environment. BEARCUBS [82] contains 111 information-seeking queries requiring access to real web content, without using a simulated web environment.

Specialized Benchmarks For task-specific objectives, general benchmarks often prove inadequate for evaluating Web Agent performance in their domains, necessitating specialized benchmarks. DeepShop [90] targets e-commerce by generating queries across five popular shopping domains and assessing agents through product attribute analysis, matching, and ranking. SafeArena [85] focuses on malicious web agent use, offering 500 tasks with metrics including Task Completion Rate (TCR), Safety Score (NSS), and refusal rate. CASA [88] evaluates agents’ sensitivity to cultural and social norms by testing their responses to norm-violating requests. CVE Bench [86] is a benchmark based on critical vulnerabilities and exposures, featuring a sandbox that simulates web attacks to effectively assess agent vulnerabilities. Wasp [87] is another security benchmark designed to evaluate web agents’ robustness against prompt injection attacks in an end-to-end manner.

5.3. MM Search Benchmark

LLMs have made significant strides in understanding and reasoning about live textual content when integrated with search engines. Despite these advancements, a crucial question remains: has the understanding of other modalities, such as visual knowledge in live contexts, been similarly addressed? Are there benchmarks for multimodal search methods?

MMSearch [59] introduced a multimodal AI search engine benchmark to thoroughly assess the searching performance of MLLMs, marking the first evaluation dataset to measure MLLMs’ capabilities in multimodal searching. LIVEVQA [91], depicted in Figure 9, is an automatically collected benchmark dataset specifically designed to evaluate current AI systems on their ability to answer questions requiring live visual knowledge. However, existing benchmarks for this critical task face a significant shortage of suitable datasets and scientifically rigorous evaluation metrics. MRAMG-Bench [92] is a novel benchmark created to comprehensively evaluate the MRAMG task. It consists of six meticulously curated English datasets, sourced from three domains: Web, Academia, and Lifestyle, across seven distinct data sources. Recently, BrowseComp-VL [199] proposes a BrowseComp-style benchmark requiring complex visual and textual information retrieval.

VisualWebArena [79] is primarily designed for visual web agent tasks, incorporating both textual and visual content from real-world environments. It comprises 910 real-world tasks across three distinct web environments. A key feature of VisualWebArena is that all tasks require agents to process and interpret visual information, rather than relying solely on textual or HTML-based cues. For evaluation, the metrics follow WebArena’s framework but extend it by incorporating image verification alongside the original two assessment methods.

6. Softwares and Products

AI search ecosystem has rapidly diversified into general-purpose platforms, domain-specific tools, and integrated assistants, each leveraging large language models (LLMs), retrieval-augmented generation (RAG), and agentic workflows to redefine information retrieval. Below, we will introduce the key products driving this transformation.

Global General-Purpose AI Search Engines. A pioneer in deep research, ChatGPT Deep Research [7] integrates Bing’s real-time web search to provide concise, conversational responses, sparking a surge of interest among researchers in large language models. Perplexity Deep Research [93] combines GPT-4 and Claude 3 with real-time web crawling, providing source-attributed answers. Its Discover feature tracks trending topics, making it ideal for academic literature reviews and technical writing. You.com [206] prioritizes privacy and personalization, allowing model switching (e.g., GPT-4, Claude) mid-session. Its Smart mode offers free access, while Research mode supports deep investigations with citation exports. Gemini Deep Research [99] embeds multi-modal capabilities into Pixel phones and Wear OS, enabling real-time translation via camera and health data-driven recommendations, reinforcing its “hardware-software” synergy in high-end markets. Optimized for speed and cost-efficiency, Doubao [94] integrates seamlessly with Douyin for video-content searches. Yuanbao [95] redefines “search-as-service” by embedding within WeChat’s ecosystem. Its three-layer architecture—base model (trillion-parameter MoE), industry-specific tuning (e.g., medical diagnostics), and mini-program integration—enables seamless service execution (e.g., generating travel itineraries with bookings). This ecosystem approach has driven rapid adoption. Nano AI [96] is China’s first “super search agent” that autonomously plans tasks (e.g., travel itineraries, market reports) by integrating data from walled gardens. Its DeepSearch technology parses tables, formulas, and video comments, enabling cross-platform verification for reliable decision-making. Kimi [97] can process 200 K-context windows, ideal for academic paper analysis. Users highlight its semantic search for Chinese literature. DeepSeek Search [100] represents a paradigm shift in cost-efficient, open-source AI search. Quark DeepSearch [98] relies on the Qwen-QWQ [207] inference model. Unlike traditional search engines that rely on keyword matching, the model understands natural language and performs semantic analysis to more accurately grasp user intent. Other deep research products include MiroMind ODR [101] and Manus [102].

Domain-Specific AI Search Tools. MediSearch [103] provides evidence-based medical answers (e.g., drug interactions, treatment protocols), trusted by 74% of healthcare professionals for clinical decision support. Devv.ai [104] is a code-specific search engine offering real-time debugging snippets and GitHub integration. It supports Chinese queries but is limited to programming contexts. Consensus [105] accesses 200 M+ scientific papers, using NLP to extract hypotheses and methodologies. Researchers report 50% time savings in literature reviews.

Integrated AI Search Assistants WallesAI [106] is a browser-sidebar assistant that reads PDFs, videos, and webpages, enabling cross-document Q&A and content export. Bing Chat [107], deeply integrated into Edge’s ecosystem, delivers citation-backed answers through real-time web indexing and source attribution, establishing a unified search-browser experience.

7. Challenges and Future Research

Despite notable progress, this field still faces many unresolved challenges, and there is considerable room for improvement. We highlight several promising directions based on the reviewed progress:

•

Methods: More complex problems lead to a prolonged search process and additional actions, resulting in an extended search context. This extended context can limit the effectiveness of AI search methods and the ability of LLMs, causing search performance to degrade as the inference length increases.

•

Evaluations: There is a strong need for systematic and standardized evaluation frameworks in AI search. The datasets used for evaluation should be designed to closely resemble real-world scenarios, featuring complex, dynamic, and citation-supported answers.

•

Applications: The potential real-world applications of AI Search are significant. Beyond user scenarios, there are numerous applications across various industries. We hope to see the development of more AI search software and products to enhance the interaction between humans and machines.

8. Conclusions

Seeking and accessing information is a fundamental daily need for humans. In this survey, we provide a comprehensive overview of the latest research on AI Search based on LLMs. Our objective is to identify and highlight areas that require further research and suggest potential avenues for future studies. We begin by introducing the traditional information retrieval systems, LLMs, and AI Search based on LLMs. Subsequently, we classify existing studies into four categories: Text-based AI Search, Web Browsing Agent, Multimodal AI Search, and Benchmarks. We then highlight a range of current and significant software and products within the realm of AI search. Finally, we discuss the limitations of current AI search methods and explore promising future directions.