Fuzzing AI Systems: Foundations, Techniques, and Open Challenges

1. Introduction

AI-enabled systems have moved from experimental settings into everyday software and high-impact applications. Modern machine learning (ML) and deep learning (DL) components now support tasks such as diagnosis support, financial decision-making, autonomous navigation, cybersecurity analysis, code assistance, recommendation, and content generation. As these components become part of larger software systems, their reliability matters not only at the model level but also at the level of the surrounding software infrastructure that trains, executes, deploys, and integrates them.

An AI system can be broadly understood as a machine-based system that processes inputs and produces outputs, such as predictions, classifications, recommendations, or generated content, to support decisions or actions in real or virtual environments [1,2]. Such systems may process different forms of input, including text, images, video, sensor data, and source code. In this survey, we use the term AI system broadly to include traditional ML systems, DL models, large language models (LLMs), AI-enabled applications built on trained models, and the supporting software infrastructure. This infrastructure may include data pipelines, DL libraries and frameworks, compiler backends, runtime and inference components, training and deployment pipelines, and integrations with conventional software systems.

Despite rapid progress, ensuring the reliability, robustness, safety, and security of AI systems remains difficult. Unlike conventional software, AI systems depend heavily on data, learned behavior, complex model architectures, and interactions across multiple software and hardware components. As a result, failures may originate not only from the learned model, but also from the surrounding frameworks, compiler backends, runtime environments, deployment pipelines, and application-level integrations. Their behavior can also be difficult to predict, explain, and validate, particularly when inputs are high-dimensional, uncertain, or generated dynamically. Prior work has similarly highlighted recurring challenges in the development and maintenance of AI-based software, including data management, model reuse, system integration, and testing complexity [3]. These characteristics create a strong need for systematic testing techniques that can identify unexpected behavior, faults, vulnerabilities, and safety violations.

Fuzzing is an automated software testing technique that generates diverse or unexpected inputs to expose failures, crashes, vulnerabilities, and other abnormal behaviors in a target system [4]. A typical fuzzing process repeatedly generates or mutates test inputs, executes the target system, and monitors its behavior for abnormal outcomes. Because fuzzing can explore large input spaces without always requiring complete formal specifications, it has increasingly been adapted to AI systems. Existing studies have applied fuzzing to learned models, DL frameworks and libraries, compiler backends, autonomous systems, reinforcement-learning environments, LLM-based applications, and other AI-enabled systems.

However, the resulting body of work is fragmented. Existing studies differ substantially in their testing targets, fuzzing techniques, input-generation and mutation strategies, feedback signals, oracle mechanisms, and failure definitions. These differences make it difficult to form a unified view of how fuzzing is used across the AI software stack, how design choices vary across testing targets, and which challenges remain unresolved.

In this article, we present a systematic survey of fuzzing techniques for AI systems. Prior surveys have mainly examined ML testing, DL testing, adversarial robustness, general fuzzing, or fuzzing for specific AI targets. In contrast, our survey studies fuzzing across the AI software stack and organizes the literature according to the AI-system layer under test. We analyze 125 primary studies identified within the January 2015–February 2026 search window and examine the research landscape, fuzzing techniques, oracle-construction strategies, failure types, reported limitations, and open challenges. Our findings show rapid but uneven growth across testing targets, strong target dependence in fuzzing design, persistent difficulties in oracle construction, failure modes that extend beyond crashes and exceptions, and a continuing need for reproducible benchmarks and cross-layer evaluation. The core contributions of this survey are as follows:

• We systematically collect and curate 125 primary studies on fuzzing AI systems from major academic databases using predefined inclusion and exclusion criteria.
• We develop a unified taxonomy covering testing targets, system layers, AI paradigms, technique families, input-generation strategies, mutation strategies, oracle mechanisms, and failure types.
• We characterize research trends and examine how fuzzing techniques, feedback signals, oracle assumptions, and failure definitions vary across models, frameworks/libraries, compiler backends, and system-level AI applications.
• We analyze the oracle mechanisms and failure types used across different AI-system targets, including differential, crash/exception, specification-based, and inconsistency-based mechanisms.
• We synthesize recurring limitations, research gaps, and open challenges into five take-away messages that highlight priorities for future research on reliable, secure, and reproducible AI-system fuzzing.

Figure 1 summarizes the organization of this survey and its main analysis dimensions, including testing targets, input generation and mutation, fuzzing techniques, oracle mechanisms, failure types, and recurring challenge themes.

By providing a structured understanding of existing research on fuzzing AI systems and highlighting recurring gaps such as oracle design, semantic validity, realistic scenario generation, numerical inconsistency, reproducibility, and system-level evaluation, this survey can support researchers and practitioners working to improve the reliability, robustness, safety, and security of AI systems.

The remainder of this paper is organized as follows. Section 2 introduces the background on fuzzing and AI systems. Section 3 reviews related survey studies and positions this work within the existing literature. Section 4 describes the methodology used for this systematic survey. Section 5 presents the research questions and corresponding findings. Section 6 synthesizes the main findings into five take-away messages that integrate observed gaps, limitations, open challenges, and future implications. Section 7 discusses potential threats to the validity of this study. Finally, Section 8 concludes the paper and outlines future research directions.

2. Background

2.1. Overview of Fuzz Testing

Fuzz testing is an automated testing technique that repeatedly generates or mutates inputs, executes the system under test, observes its behavior, and reports failures such as crashes, exceptions, assertion violations, security vulnerabilities, or semantic inconsistencies [5]. A typical fuzzing loop starts from seed inputs or an input generator, applies generation or mutation operators, executes the target, and uses feedback such as coverage, exception types, or domain-specific signals to guide subsequent exploration [6].

Depending on the information obtained from the target, fuzzing may be black-box, grey-box, or white-box. Black-box fuzzing observes external behavior, grey-box fuzzing uses lightweight feedback such as coverage, and white-box fuzzing relies on deeper information, including symbolic constraints, source-level information, or internal execution states [6]. In AI systems, these principles are adapted to feedback such as model behavior, neuron activation, API execution, compiler paths, simulator states, and prompt responses [7,8,9].

2.2. Fuzzing in AI Systems

Fuzzing in AI systems refers to the automated generation or mutation of inputs, configurations, scenarios, prompts, or interaction traces to expose failures in AI-enabled systems. Unlike conventional fuzzing over byte-level program inputs, AI fuzzing must account for learned behavior, probabilistic outputs, numerical computation, and interactions across the AI software stack. Existing approaches target learned models [7,8,10], frameworks and compilers [9,11,12,13], integrated autonomous and decision-making systems [14,15,16], and security-sensitive ML, LLM, and agent-based applications [17,18,19].

The input space also varies with the target. Tests may involve images, text, tensors, API calls, computation graphs, driving scenarios, environment states, prompts, or multi-step interactions. Because many of these inputs are semantically constrained, effective fuzzing must preserve validity while exploring rare or failure-inducing behavior. For example, autonomous-driving and reinforcement-learning fuzzers manipulate scenarios, environments, or trajectories [14,15,16,20], whereas LLM and agent fuzzers operate over prompts, dialogue histories, tool requests, or interaction traces [18,19].

Failures may occur even when the program does not crash. Model-level testing can expose misclassification or robustness degradation [7,8,10]; framework and compiler testing can reveal exceptions, numerical inconsistencies, or semantic mismatches [9,11,12,13]; and system-level testing can expose unsafe behavior, policy violations, malware evasion, jailbreaks, or unsafe agent actions [15,16,17,18,19]. Accordingly, AI fuzzing may rely on crash signals, behavioral inconsistencies, robustness violations, safety rules, or security properties rather than a single expected output.

2.3. Boundary Between Fuzzing and Related Testing Techniques

To define the scope of this survey, we consider a technique to be fuzzing when it follows an iterative exploration loop that generates or mutates inputs, executes the target, observes the resulting behavior, and uses feedback or failure signals to guide further exploration [5]. Related techniques are excluded when they provide only an oracle, adequacy criterion, path constraint, attack method, or scenario-generation mechanism without such a fuzzing-style workflow.

2.3.1. Metamorphic Testing

Metamorphic testing is related to fuzzing because metamorphic relations can serve as test oracles when expected outputs are difficult to specify [21,22]. However, studies are excluded when their main contribution is limited to defining or evaluating metamorphic relations without an iterative fuzzing workflow. They are included only when those relations are used within an automated process that repeatedly generates or transforms inputs, executes the target, and evaluates the resulting behavior.

2.3.2. Mutation Testing

Mutation testing is related to fuzzing because both use the term “mutation.” However, they modify different artifacts. Fuzzing generally mutates inputs to expose failures in the original system, whereas mutation testing modifies programs, models, or tests to assess test-suite adequacy [5,23]. Therefore, mutation-testing studies are excluded when mutation is used only to evaluate testing effectiveness. They are included only when mutation supports iterative, fuzzing-style input exploration.

2.3.3. Concolic and Symbolic Testing

Concolic and symbolic testing are related to fuzzing because they generate inputs by exploring program paths and solving path constraints [24,25]. Studies are excluded when symbolic execution or constraint solving is the main testing mechanism and no fuzzing-style exploration loop is present. They are included only when these techniques are integrated into a workflow that also performs repeated input generation or mutation, execution, monitoring, feedback-guided exploration, and failure reporting.

2.3.4. Adversarial Testing

Adversarial testing is related to fuzzing because it generates inputs intended to expose model errors or robustness weaknesses [26]. Adversarial-attack studies are excluded when they are formulated only as optimization problems against a model and do not perform iterative fuzzing or failure discovery. They are included only when adversarial input generation is used as a mutation operator or search strategy within a broader fuzzing workflow.

2.3.5. Coverage-Based Autonomous Vehicle Testing

Coverage-based autonomous-vehicle testing is related to fuzzing because it often generates driving scenarios or simulator inputs and evaluates behavioral or coverage metrics [14,15,20]. Studies are excluded when they focus only on scenario generation, coverage measurement, or benchmark construction without feedback-guided mutation and failure discovery. They are included only when the method iteratively generates or mutates scenarios, executes the autonomous-driving system, observes coverage or safety feedback, and uses that feedback to guide subsequent tests.

2.4. Key Concepts in AI Fuzzing

AI fuzzing studies differ in the artifact they test, the system layer at which behavior is observed, the AI paradigm and software platform involved, the way inputs are generated or mutated, and the mechanisms used to detect and classify failures. Table 1 summarizes the concepts used throughout this survey.

2.5. Testing Targets in AI Systems

The testing target is the concrete artifact exercised by the fuzzer. Because different targets expose different failure surfaces, we organize AI fuzzing into four layers: model, framework/library, compiler/backend, and system level.

Table 1. Key concepts used to characterize AI-system fuzzing studies.

Concept	Definition and representative examples
Testing target	The concrete artifact exercised by the fuzzer, such as a trained model, framework or library API, compiler backend, or complete AI-enabled application. The target influences the input space, feedback signal, oracle, and reported failure type. We discuss these target layers in more detail in Section 2.5.
System level	The layer at which behavior or failure is observed: a learned model, an AI software component, or an integrated system. A test may exercise one layer while exposing a defect caused by another  [7,8,9,10,11,12,13,14,15,16,19].
AI paradigm	The type of AI under test. Examples include DL and computer-vision models, autonomous systems, reinforcement learning, NLP, LLMs, and agent-based systems  [7,8,10,14,15,16,18,19,20,27,28,29].
Target framework or platform	The software ecosystem in which fuzzing is performed. Examples include TensorFlow, PyTorch, Keras, and TensorFlow.js at the framework level; TVM, XLA, MLIR, TensorRT, and ONNXRuntime at the compiler or runtime level; and CARLA, OpenAI Gym, and Stable-Baselines3 at the system level  [9,11,12,13,15,30]. Datasets and model architectures describe experimental subjects but are not treated as frameworks or platforms.
Technique family	The main strategy used to explore the test space. Common families include coverage-guided, mutation-based, differential, constraint-guided, generation-based, learning-based, search- or heuristic-based, and LLM- or prompt-based approaches  [7,8,10,11,12,13,14,15,18,19,20].
Input-generation strategy	The process used to create initial test cases. Strategies may be seed-based, constraint-based, generative, model-guided, search-based, or LLM-generated, depending on the validity requirements of the target  [9,10,12,15,16,18,19,20,31,32].
Mutation strategy	The method used to transform an existing test into a new candidate. Examples include semantic-preserving input transformations, gradient-guided mutations, structure-aware API mutations, graph or intermediate-representation mutations, and scenario-, state-, or interaction-level mutations  [10,12,13,14,15,16,19,20,27,30,31,32,33,34].
Test oracle and oracle construction	The mechanism used to recognize a failure and the evidence used to construct that mechanism. Common oracles include crash/exception, differential, metamorphic or inconsistency-based, and specification- or rule-based checks. They may be constructed using multiple models or backends, reference implementations, documentation-derived constraints, simulator rules, or security properties  [7,9,10,11,12,13,14,15,19,20,31].
Failure type	The behavior ultimately exposed by the fuzzer. Examples include misclassification and robustness degradation; crashes and runtime errors; numerical or semantic inconsistencies; and safety or security violations  [7,8,9,10,11,12,13,15,16,17,18,19,27,28,32,34].

Table 1. Key concepts used to characterize AI-system fuzzing studies.

Concept	Definition and representative examples
Testing target	The concrete artifact exercised by the fuzzer, such as a trained model, framework or library API, compiler backend, or complete AI-enabled application. The target influences the input space, feedback signal, oracle, and reported failure type. We discuss these target layers in more detail in Section 2.5.
System level	The layer at which behavior or failure is observed: a learned model, an AI software component, or an integrated system. A test may exercise one layer while exposing a defect caused by another  [7,8,9,10,11,12,13,14,15,16,19].
AI paradigm	The type of AI under test. Examples include DL and computer-vision models, autonomous systems, reinforcement learning, NLP, LLMs, and agent-based systems  [7,8,10,14,15,16,18,19,20,27,28,29].
Target framework or platform	The software ecosystem in which fuzzing is performed. Examples include TensorFlow, PyTorch, Keras, and TensorFlow.js at the framework level; TVM, XLA, MLIR, TensorRT, and ONNXRuntime at the compiler or runtime level; and CARLA, OpenAI Gym, and Stable-Baselines3 at the system level  [9,11,12,13,15,30]. Datasets and model architectures describe experimental subjects but are not treated as frameworks or platforms.
Technique family	The main strategy used to explore the test space. Common families include coverage-guided, mutation-based, differential, constraint-guided, generation-based, learning-based, search- or heuristic-based, and LLM- or prompt-based approaches  [7,8,10,11,12,13,14,15,18,19,20].
Input-generation strategy	The process used to create initial test cases. Strategies may be seed-based, constraint-based, generative, model-guided, search-based, or LLM-generated, depending on the validity requirements of the target  [9,10,12,15,16,18,19,20,31,32].
Mutation strategy	The method used to transform an existing test into a new candidate. Examples include semantic-preserving input transformations, gradient-guided mutations, structure-aware API mutations, graph or intermediate-representation mutations, and scenario-, state-, or interaction-level mutations  [10,12,13,14,15,16,19,20,27,30,31,32,33,34].
Test oracle and oracle construction	The mechanism used to recognize a failure and the evidence used to construct that mechanism. Common oracles include crash/exception, differential, metamorphic or inconsistency-based, and specification- or rule-based checks. They may be constructed using multiple models or backends, reference implementations, documentation-derived constraints, simulator rules, or security properties  [7,9,10,11,12,13,14,15,19,20,31].
Failure type	The behavior ultimately exposed by the fuzzer. Examples include misclassification and robustness degradation; crashes and runtime errors; numerical or semantic inconsistencies; and safety or security violations  [7,8,9,10,11,12,13,15,16,17,18,19,27,28,32,34].

2.5.1. Model-level AI Fuzzing

Model-level AI fuzzing treats the trained model as the primary artifact under test. The fuzzer generates or mutates inputs and observes predictions, confidence scores, internal activations, coverage signals, or behavioral changes to expose failures such as misclassification, robustness degradation, prediction instability, or unsafe outputs. DeepXplore [7] searches for differential behavior across models, while TensorFuzz [8] and DeepHunter [10] use coverage-guided exploration and seed-based mutation. DLFuzz [27] and GradFuzz [33] further use gradient-related feedback to generate error-inducing inputs.

2.5.2. Framework- and Library-Level AI Fuzzing

Framework- and library-level AI fuzzing targets the software infrastructure that implements AI computation. Deep learning frameworks expose large API surfaces with constraints over tensor shapes, ranks, data types, devices, parameter values, and operator semantics. Fuzzers therefore generate valid but diverse API calls or operator invocations to reveal crashes, uncaught exceptions, inconsistent outputs, invalid tensor handling, or numerical discrepancies. CRADLE [11] uses cross-backend validation, FreeFuzz [12] derives tests from open-source framework usages, DocTer [32] uses documentation-guided constraints, and ACETest [31] automates constraint extraction for DL operators. TensorJSFuzz [35] and Predoo [36] also target framework- or operator-level behavior.

2.5.3. Compiler- and Backend-Level AI Fuzzing

Compiler- and backend-level AI fuzzing targets the transformation pipeline that converts high-level AI computations into optimized executable code. Bugs introduced during optimization, lowering, scheduling, or backend execution may cause crashes, wrong-code behavior, numerical inconsistencies, or semantic divergence even when the original model is valid. TzER [13] applies coverage-guided joint IR-pass mutation, while NNSmith [9] generates diverse and valid neural network models for compiler testing. HIRGEN [34] and MLIR-Smith [30] generate programs or intermediate representations, and NeuRI [37] improves the diversity of DNN generation for compiler testing.

2.5.4. System-Level AI Fuzzing

System-level AI fuzzing evaluates an integrated AI application rather than an isolated model, API, or compiler component. It explores failures that emerge from interactions among models, software components, simulators, environments, tools, and external inputs. Autonomous-driving fuzzers such as DeepTest [14], DeepRoad [20], AV-Fuzzer [28], and DriveFuzz [15] search for unsafe driving behaviors and scenario-level failures. Reinforcement-learning fuzzers such as MDPFuzz [16] and AgentFuzz [29] test sequential decision-making behavior. System-level fuzzing also extends to security-sensitive applications, including MalFuzz [17], LLM-Fuzzer [18], and AgentFuzz for LLM-based agents [19].

3. Related Work

Existing surveys related to our work fall into five broad categories: ML/DL testing, DL library and compiler testing, system-level AI testing, LLM evaluation and security, and AI-assisted fuzzing and testing. In contrast, our survey studies fuzzing in which AI systems are the testing targets across multiple layers of the AI software stack. Table 2 summarizes this positioning.

3.1. Surveys on Machine Learning and Deep Learning Testing

Zhang et al. [38] organize ML testing by testing properties, ML components, workflows, and application scenarios. Braiek and Khomh [39] review testing practices across the ML development pipeline, while Riccio et al. [40] map functional testing of ML-based systems in terms of testing problems, methods, evaluations, input generation, adequacy criteria, and test oracles. Huang et al. [41] examine DNN safety and trustworthiness through verification, testing, adversarial techniques, and interpretability, whereas Hu et al. [53] focus on low-labeling-cost DNN testing and test optimization.

3.2. Surveys on Deep Learning Library and Compiler Testing

Zhang et al. [42] survey testing methods for DL frameworks, compilers, and hardware libraries while excluding model and DL program testing. Li et al. [54] review DL compiler architectures, intermediate representations, frontend and backend optimizations, hardware-specific code generation, and deployment.

3.3. Surveys on System-Level AI Testing

Zhang et al. [43] review critical-scenario identification for automated-driving systems through scenario definition, derivation, and assessment. Zhong et al. [44] survey simulation-based ADS testing across simulators, systems under test, objectives, scenario parameters, and search methods. Tang et al. [55] provide a broader view spanning module- and system-level ADS testing and the simulation–reality gap.

3.4. Surveys on LLM Security and Foundation Model Testing

Zhao et al. [56] broadly review LLM development, adaptation, use, evaluation, resources, and open challenges, while Chang et al. [57] organize LLM evaluation around tasks, benchmarks, protocols, and emerging challenges. Yi et al. [46] review jailbreak attacks, defenses, and evaluation strategies, and Li and Fung [47] examine prompt manipulation, training-time attacks, malicious use, defenses, and agent-related risks. Asgari et al. [48] review metamorphic testing of deep code models, including transformations, datasets, metrics, and robustness challenges.

3.5. Surveys on AI-Assisted Fuzzing and Testing

A related but distinct line of work studies the use of AI, ML, DL, or LLMs to improve fuzzing and testing of conventional software. General fuzzing surveys establish the underlying concepts of input generation, mutation, coverage guidance, execution monitoring, crash analysis, and bug triaging [6,58]. Saavedra et al. [49] review ML applications in fuzzing for vulnerability assessment, while Wang et al. [50] analyze how ML is used across fuzzing stages, models, datasets, preprocessing methods, metrics, and performance. Zhang et al. [4] organize ML-based fuzzing around preprocessing, test generation, input selection, and result analysis, and Qiu et al. [51] focus on deep neural models for seed generation, selection, and mutation in greybox fuzzing. Wang et al. [45] review LLM use across software-testing activities, prompting strategies, and supporting techniques. Huang et al. [52] survey LLM-based fuzzing in both AI and non-AI software contexts and therefore partially cover direct AI-system fuzzing; however, their primary focus is the use of LLMs to support the fuzzing process.

3.6. Positioning of Our Survey

Prior surveys provide important but complementary perspectives. ML/DL testing surveys broadly examine learned models and testing workflows; DL library and compiler surveys focus on supporting infrastructure; automated-driving surveys study a specific application domain; and LLM surveys emphasize evaluation, robustness, attacks, and defenses. A separate body of work examines the opposite relationship between AI and fuzzing by using ML, DL, or LLMs to improve testing of conventional software. Consequently, knowledge about fuzzing AI systems remains distributed across targets, domains, and survey perspectives.

Our survey addresses this fragmentation through a dedicated, fuzzing-centered analysis of AI systems as testing targets. It provides a common taxonomy spanning models, frameworks/libraries, compiler backends, integrated AI applications, and LLM/agent-based systems; relates these targets to technique families, input-generation and mutation strategies, feedback, and oracle construction; and connects these design choices to the failures they expose.

This cross-layer perspective links where fuzzing is applied, how tests are generated, how failures are detected, and what failures are exposed across the AI software stack. It also reveals which design choices are shared across targets, which remain target-specific, and where recurring challenges require further research.

4. Methodology of Our Systematic Survey

We conducted a systematic literature review to analyze fuzzing techniques for AI systems. The methodology was guided by established systematic-review practices, including PRISMA, and followed a structured process for literature search, screening, study selection, metadata extraction, annotation, and validation. Additional implementation details, methodological artifacts, and supporting materials are available through the companion website.

4.1. Research Questions

The review was guided by the following research questions:

• RQ1. What is the research landscape of fuzzing for AI systems?

  −

  RQ1.1 How has research evolved over time?

  −

  RQ1.2 How are studies distributed across venues?

  −

  RQ1.3 What types of AI systems and components are targeted by fuzzing approaches?

• RQ2. What fuzzing techniques are used for testing AI systems?
• RQ3. What test oracles are used in AI fuzzing?
• RQ4. What types of failures are exposed by fuzzing AI systems?

4.2. Search Strategy and Data Sources

Following the guidelines of Nima et al. [59] and Dagher et al. [60], we collected studies on fuzzing AI systems within the January 2015–February 2026 search window. This period captures the emergence and growth of AI-system testing, including recent work on DL frameworks, AI compilers, LLM-based systems, and agent-based applications.

We searched five scholarly sources: ACM Digital Library, ScienceDirect, Scopus, Semantic Scholar, and Google Scholar. These sources provide broad coverage of software engineering, software testing, security, systems, and artificial-intelligence research. Records were retrieved using the search, export, or API mechanisms supported by each source and normalized into a common metadata structure. IEEE publications were also captured through indexing services such as Scopus and Google Scholar.

The source-specific retrieval procedures included ACM CSV export, the Elsevier Search API, the Semantic Scholar API, and metadata export through Publish or Perish for Google Scholar and Scopus. Detailed database-specific retrieval procedures, export settings, and the multi-source collection workflow are documented on the Data Collection page of the companion website.

4.3. Search Query Design

We designed the search strategy to maximize recall and reduce the risk of missing relevant studies. The search expressions combined fuzzing-related terms with AI-, model-, framework/library-, compiler/backend-, and deployment-related terms. Because terminology differs across research communities, the queries intentionally included related expressions such as mutation testing, coverage-guided testing, robustness testing, differential testing, and neural-network compiler testing.

The initial search therefore retrieved both fuzzing studies and related work from areas such as metamorphic testing, mutation testing, adversarial testing, symbolic or concolic testing, and coverage-based autonomous-vehicle testing. These studies were later evaluated using the inclusion and exclusion criteria, particularly the requirement that a study contain a recognizable fuzzing-style workflow involving input generation or mutation, target execution, monitoring or feedback, and failure reporting.

Database search alone may miss relevant studies because terminology in AI-system fuzzing is fragmented and some studies do not explicitly use the term “fuzzing” in their titles, abstracts, or indexed metadata. We therefore complemented database search with backward and forward snowballing.

Following prior survey work [61] and our knowledge of the topic, we identified key search phrases and constructed database-specific queries. For Semantic Scholar, Google Scholar, Scopus, and ScienceDirect, we used the following comprehensive query:

For the ACM Digital Library, we used a shorter query because of query-length and formatting limitations in its advanced search interface. The simplified query preserved the main fuzzing- and AI-related concepts while removing several platform-specific terms:

The complete search records, database-specific query adaptations, search dates, export settings, and supporting scripts are available on the Search Strategy page of the companion website.

4.4. Data Collection and Preprocessing

The initial search retrieved 3,905 records from the five scholarly sources: 1,242 from Semantic Scholar, 1,000 from ACM Digital Library, 844 from Google Scholar, 619 from ScienceDirect, and 200 from Scopus. Using multiple sources broadened the literature coverage and reduced the risk of missing studies that were not indexed consistently across databases.

Because metadata formats differed across sources, we standardized the records using common fields, including title, authors, publication year, venue, abstract, DOI, and URL. Title-based duplicate detection and DOI-based matching were then applied. After preprocessing and deduplication, 1,996 unique records remained for screening.

The source-distribution figure, detailed metadata-normalization procedure, preprocessing workflow, and corresponding scripts are available on the Data Preprocessing page of the companion website.

4.5. Study Selection

4.5.1. Inclusion and Exclusion Criteria

We defined inclusion and exclusion criteria to identify studies directly relevant to fuzzing AI systems.

A study was included if it satisfied the following criteria:

• It was published within the January 2015–February 2026 search window.
• It proposed, evaluated, or empirically analyzed a fuzzing technique targeting an AI system, including an ML/DL/LLM model, DL framework or library, AI compiler or backend, or system-level AI application.
• It involved a recognizable fuzzing-style workflow, such as automated input generation or mutation, target execution, monitoring or feedback, and failure reporting.
• It provided sufficient technical detail to support metadata extraction and classification.

A study was excluded if it satisfied any of the following criteria:

• It used AI, ML, DL, or LLMs to improve fuzzing of conventional software rather than fuzzing an AI system.
• It targeted an AI system but did not use a fuzzing-style testing approach.
• It was unrelated to AI-system testing.
• It focused only on adversarial attack optimization without a fuzzing-style workflow.
• It was a survey, review article, book chapter, short abstract, poster, tutorial, or other non-primary study.
• It was a short workshop paper that did not provide sufficient methodological or evaluation detail for systematic annotation.
• It was outside the selected publication period or was not written in English.
• It was a duplicate record or an earlier preprint version of an already included published study.

Using these criteria, we first screened titles and abstracts to remove clearly irrelevant records. We then applied multi-stage filtering, as described in Section 4.5.2, to refine the candidate set and identify the final primary studies for analysis.

4.5.2. Multi-Stage Filtering and Snowballing

We applied a multi-stage filtering process to refine the candidate set.

Stage 1: Automated Keyword Filtering.

An initial keyword-based filter removed clearly irrelevant records. The filter required the presence of AI-related concepts, such as machine learning, deep learning, neural networks, LLMs, transformers, foundation models, or model robustness, together with fuzzing-related concepts, such as fuzzing, fuzz testing, coverage guidance, or robustness testing. This stage reduced the dataset from 1,996 to 1,365 records.

Stage 2: Precision Tightening.

A stricter filtering step required clearer evidence of both an AI-system context and a fuzzing-related testing methodology. After this stage, 187 papers remained for manual title and abstract screening.

Stage 3: Manual Title and Abstract Screening.

We manually reviewed the 187 candidate papers using the inclusion and exclusion criteria in Section 4.5. For each paper, we examined whether the study targeted an AI system and contained a recognizable fuzzing-style workflow. This stage retained 45 papers.

Stage 4: Snowballing-Based Expansion.

To improve recall, we applied backward and forward snowballing to the studies retained after manual screening. Backward snowballing examined their reference lists, while forward snowballing examined citing studies identified through Google Scholar and Semantic Scholar. Automated deduplication and broad AI- and testing-related scope filters were used to reduce the candidate pool before manual review.

Snowballing produced an initial pool of 1,854 records, which was reduced to 254 candidates before manual review. After validation using the same inclusion and exclusion criteria, 167 studies were retained, including 30 from backward snowballing and 137 from forward snowballing. Combined with the 45 studies retained from database screening, this produced a set of 212 papers.

Detailed backward- and forward-snowballing procedures, filtering rules, keyword groups, intermediate counts, and automation scripts are available on the Snowballing page of the companion website.

4.6. Final Paper Selection

Following the multi-stage screening and snowballing process, 212 papers remained. We then performed an additional quality and eligibility check. Following supervisory guidance, papers from publishers or venues that raised quality concerns during screening, including KSI Research, MDPI, and Totem Publisher Inc., were re-examined individually. A paper was excluded only when it lacked sufficient relevance to fuzzing-based testing of AI systems, methodological clarity, evaluation detail, or eligibility as a primary study. This step removed four papers, leaving 208 candidates for full-text eligibility review.

The 208 papers were then assessed using the inclusion and exclusion criteria in Section 4.5. Full-text screening retained 127 eligible primary-study candidates. During the final consistency check, two duplicate study versions were found, where both a preprint and a later published version appeared in the corpus. For each duplicate pair, we retained the published or more complete version. The final dataset therefore contained 125 primary studies.

The complete collection and selection process is summarized in Figure 2. The 208 candidate papers also formed the input to the full-text eligibility review and LLM-assisted annotation process described in Section 4.8.

4.7. Metadata Extraction and Classification

For each included study, we retained bibliographic metadata and extracted technical attributes needed to answer the research questions. The bibliographic fields included the paper identifier, title, authors, publication year, venue, publisher, DOI, URL, and abstract.

The technical fields covered the testing target, system level, AI paradigm, target framework or platform, technique name and family, input-generation strategy, mutation strategy, oracle type and construction, failure type, reported limitations, threats to validity, future work, observed research gaps, main contributions, and additional notes.

To improve consistency, we used a predefined annotation schema with controlled labels for the main classification fields. The schema was developed from the research questions and refined during pilot annotation.

The complete annotation schema, field definitions, controlled labels, representative examples, and dataset data dictionary are available on the Annotation Schema page of the companion website.

4.8. LLM-Assisted Annotation and Manual Validation

We used LLM-assisted annotation followed by manual validation. Figure 3 summarizes the annotation process used to construct the final analysis-ready dataset.

Starting from the 208 candidate papers, we used the original PDF of each study for full-text eligibility assessment and technical annotation. The full text was extracted and organized into section-aware evidence for eligibility checking and metadata extraction.

We used GPT-5.4-mini and Claude Opus 4.6 to generate preliminary annotations independently. The models received the annotation schema, field definitions, allowed labels where applicable, and relevant evidence from each paper. They were instructed to classify the paper’s proposed technique rather than methods described only in background or related-work sections. The generated labels, summaries, and evidence snippets were treated as candidate annotations and were not accepted as final classifications.

Figure 2. Literature Collection Process for Fuzzing AI Systems.

Figure 2. Literature Collection Process for Fuzzing AI Systems.

A two-stage human validation process was then applied. First, an annotator checked the preliminary labels against the original paper, with particular attention to the abstract, methodology, evaluation, discussion, limitations, and conclusion. Unsupported, ambiguous, or overly broad labels were corrected using evidence from the paper. This review also supported the full-text eligibility decision.

Second, a checker reviewed the assigned labels and eligibility decisions, recorded agreement status, and documented disagreement notes and justifications. Cases involving disagreement between the LLM outputs, uncertainty in the assigned labels, or disagreement between the annotator and checker were re-examined using the original paper and resolved collaboratively.

Under this annotator-checker validation setup, the initial agreement was 144 out of 208 candidate studies (69.2%). We interpret this value as a diagnostic measure of annotation difficulty rather than as a formal inter-rater reliability score because the task combined full-text eligibility decisions with multi-label and target-dependent classification.

Disagreements mainly concerned the distinction between AI-system fuzzing and AI-assisted fuzzing, whether a study contained a recognizable fuzzing-style workflow, identification of the primary testing target, and overlapping technique, mutation, oracle, or failure labels. Each disagreement was checked against the original paper, documented with a justification, and resolved before the final dataset was produced.

After full-text eligibility decisions, disagreement resolution, and removal of duplicate study versions, 125 primary studies were retained for the final analysis.

Detailed information about PDF extraction, section-aware text preparation, prompt templates, field-level evidence selection, output schemas, and the human-validation procedure is available on the LLM-Assisted Annotation page of the companion website.

Figure 3. LLM-assisted annotation and manual-validation pipeline used to construct the final analysis-ready dataset. Starting from 208 candidate papers, the pipeline performs metadata and text extraction, section-aware evidence preparation, preliminary annotation using two LLMs, manual verification and eligibility review, and annotator–checker quality control. After exclusion of non-fuzzing, out-of-scope, or insufficiently detailed studies and duplicate-version removal, 125 primary studies were retained for the final analysis.

Figure 3. LLM-assisted annotation and manual-validation pipeline used to construct the final analysis-ready dataset. Starting from 208 candidate papers, the pipeline performs metadata and text extraction, section-aware evidence preparation, preliminary annotation using two LLMs, manual verification and eligibility review, and annotator–checker quality control. After exclusion of non-fuzzing, out-of-scope, or insufficiently detailed studies and duplicate-version removal, 125 primary studies were retained for the final analysis.

4.9. Annotation Quality Control

Quality control focused on fields that were frequently difficult to distinguish, including testing target versus system level, target framework/platform versus dataset or model architecture, input-generation strategy versus mutation strategy, and oracle type versus failure type.

Review-status and comment fields were maintained to document uncertain cases, annotation rationales, and changes made during validation. These auxiliary fields supported internal quality control and were not used as analysis variables.

All final labels were verified against the original papers. LLM-generated outputs were used only as preliminary suggestions. When limitations, threats to validity, or future directions were not explicitly reported but could reasonably be inferred from the study design, the annotation was marked with the prefix “Not explicitly stated. Possible inferred ...” to distinguish the interpretation from author-reported information.

The verified metadata was then organized into the structured dataset used for the analyses in the following sections. The complete quality-control protocol, validation checklist, documentation conventions, and examples of resolved annotation cases are available on the Annotation Quality Control page of the companion website.

5. Results Analysis

This section presents the findings from our analysis of the selected studies on fuzzing AI systems. We organize the results according to the research questions introduced in Section 4. For each RQ, we report the observed trends from the annotated corpus and summarize the main answer at the end of the subsection.

5.1. RQ1: Research Landscape of Fuzzing for AI Systems

RQ1 examines the research landscape of fuzzing for AI systems through publication trends, venue distribution, and the AI-system targets addressed by the selected studies.

5.1.1. RQ1.1: Publication Trends Over Time

Figure 4 shows the yearly distribution of the 125 selected studies identified within the January 2015–February 2026 search window. No selected study appears between 2015 and 2017, and the first studies in the corpus appear in 2018. Research activity remains limited from 2018 to 2020, with 4 studies in 2018, 4 in 2019, and 5 in 2020. These early studies primarily target trained models, with limited attention to frameworks/libraries and integrated AI systems.

The publication count increases to 10 studies in 2021 and rises further in 2022 and 2023, with 21 and 19 studies, respectively. The highest activity occurs in 2024 and 2025, with 27 and 29 studies. During this period, model-level work remains prominent, while framework/library, compiler/backend, and system-level fuzzing become more visible. The smaller count in 2026 reflects that the search window ends in February 2026 and should not be interpreted as a decline in research activity.

The study-level chronology further shows that later years cover a broader range of testing targets and technique families. The complete chronological overview is available on the Chronological Evolution page of the companion website.

5.1.2. RQ1.2: Venue Distribution

Figure 5 shows the most frequent publication venues in the final corpus. We combine QRS and QRS Companion because they belong to the same conference family. The complete venue distribution is available on the Venue Distribution page of the companion website.

ISSTA and ICSE are the most frequent venues, with 13 studies each. They are followed by ASE and arXiv with 10 studies each, and TSE and TOSEM with 8 studies each. The combined QRS/QRS Companion family contributes 7 studies, followed by FSE with 6 studies. IST and JSS each contribute 4 studies.

The presence of major conferences together with journals and transactions shows that AI-system fuzzing is supported by both rapid dissemination and longer-form archival research.

Overall, the venue distribution shows that AI-system fuzzing is primarily rooted in software engineering, testing, and reliability research while also spanning systems, security, programming languages, AI, and preprint venues.

5.1.3. RQ1.3: Targeted AI Systems and Components

We classify the selected studies into model, framework/library, compiler/backend, system-level, and mixed model-and-system-level targets. Figure 6 shows the resulting distribution.

Model targets form the largest group, with 59 studies (47.2%), followed by framework/library targets with 38 studies (30.4%). Compiler/backend targets account for 14 studies (11.2%), while system-level targets account for 13 studies (10.4%).

One study (0.8%) spans both model- and system-level targets, illustrating that some approaches cross the boundary between isolated model behavior and end-to-end system behavior [62].

The substantial representation of framework/library targets demonstrates that AI-system fuzzing extends beyond testing trained models. Failures in APIs, operators, tensor libraries, runtimes, or compiler components may affect multiple downstream models and applications.

Overall, model-level fuzzing remains dominant, but more than half of the remaining studies examine frameworks/libraries, compiler backends, system-level applications, or mixed targets. The resulting distribution reflects a broader view of AI-system reliability in which failures may originate from learned models, supporting software components, or their interactions.

5.2. RQ2: Fuzzing Techniques Used for Testing AI Systems

RQ2 examines the fuzzing techniques used across different AI-system targets. To separate high-level fuzzing mechanisms from implementation-level details, we analyze three related dimensions: technique family, input-generation strategy, and mutation strategy. Technique family captures the main fuzzing mechanism or guidance principle; input-generation strategy records how initial test cases are produced; and mutation strategy records how existing inputs are transformed. Because many studies combine more than one central mechanism, the technique-family categories are not mutually exclusive.

We use a normalized taxonomy to reduce fragmentation among overlapping descriptive labels while preserving the original labels in the annotated dataset. LLM-based and prompt-guided approaches are grouped under LLM/Prompt-guided; grammar-based, generative, and GAN/CGAN-based approaches are grouped under Generation-based; and guidance based on distance, robustness, history, vulnerability, prediction difference, MCTS, or explainability is grouped under Search/Heuristic-guided when it does not primarily represent structural coverage. Gradient-based optimization is treated as an implementation-level mutation or guidance mechanism when it is used under another primary fuzzing objective.

Figure 7 shows that mutation-based fuzzing is the most frequent family, appearing in 63 studies, followed by coverage-guided fuzzing in 52 studies. Learning-based techniques appear in 30 studies, followed by constraint-guided techniques in 17 studies, search/heuristic-guided techniques in 16 studies, differential testing in 15 studies, LLM/prompt-guided techniques in 14 studies, generation-based techniques in 7 studies, metamorphic testing in 3 studies, and debugging/fault-localization-oriented fuzzing in 1 study. The complete study-level mapping of technique families, input-generation strategies, and mutation strategies is available on the Technique Study Mapping page of the companion website.

5.2.1. Mutation-Based Fuzzing

Mutation-based fuzzing generates new tests by modifying existing inputs, programs, model structures, API calls, tensor programs, prompts, or system scenarios. The mutation operators are adapted to the target: model-level studies often transform data inputs, framework/library studies mutate APIs or tensor programs, and system-level studies modify scenarios or interaction traces. Representative examples include DeepMutation [63], DLJSFuzzer [64], and DriveFuzz [15]. Mutation is also frequently combined with additional guidance, as in DeepHunter [10], DLRegion [65], and DEEPWALK [66]. Its effectiveness remains target-dependent because overly simple mutations may produce invalid or uninformative tests, whereas semantic or structure-aware mutations require domain knowledge.

5.2.2. Coverage-Guided Fuzzing

Coverage-guided fuzzing uses execution feedback to retain and prioritize tests that explore previously unobserved behavior. Depending on the target, the feedback may include code, neuron, layer, state, trajectory, or other behavioral coverage. Model-level examples include DLFuzz [27], DeepHunter [10], DeepStellar [67], GradFuzz [33], and DeepCNP [68]. At the system level, MDPFuzz [16], DriveFuzz [15], and ReinSeed [69] use coverage-oriented feedback to explore reinforcement-learning or autonomous-driving behaviors. Although coverage provides an explicit exploration signal, increased coverage does not necessarily imply stronger fault detection, motivating more behaviorally meaningful AI-specific coverage criteria.

5.2.3. Constraint-Guided Fuzzing

Constraint-guided fuzzing generates or mutates tests under structural, syntactic, semantic, type, shape, API, graph, or intermediate-representation constraints. This family is especially important for framework/library and compiler/backend targets, where unconstrained generation may produce invalid tests that fail before reaching deeper execution behavior. NNsmith [9], NeuRI [37], and OATest [70] use constraint-aware generation for compiler/backend testing, while TensorJSFuzz [35], DeepConstr [71], DocTer [32], ConFL [72], and ACETest [31] apply related ideas to framework or library targets. Constraints improve input validity and support deeper exploration, but incomplete or overly restrictive constraints may exclude relevant behaviors or still permit invalid tests.

5.2.4. Differential Fuzzing

Differential fuzzing compares the behavior of multiple models, implementations, frameworks, versions, devices, backends, or reference systems on the same or equivalent input. It is particularly useful when exact expected outputs are unavailable but comparable execution paths exist. Representative examples include NNsmith [9], TensorJSFuzz [35], Muffin [73], FreeFuzz [12], DeepDiffer [74], and D3 [75]. TitanFuzz [?] illustrates a hybrid setting in which LLM-generated programs are checked differentially. Differential testing reduces dependence on complete specifications, but acceptable numerical variation, approximation, and nondeterminism must be handled carefully because not every disagreement indicates a defect.

5.2.5. Generation-Based, Grammar-Based, and Structure-Aware Fuzzing

Generation-based fuzzing synthesizes new test objects rather than relying only on mutations of existing seeds. Generated artifacts may include model inputs, tensor programs, model graphs, compiler intermediate representations, API sequences, prompts, or system scenarios. Grammar-based and structure-aware methods improve syntactic and structural validity by using templates, grammars, or learned generators. For example, SYNTHFUZZ [76] uses grammar-based construction, MLIR-Smith [30] generates compiler inputs, HIRGEN [34] combines generation with coverage guidance, and DeepMCC [77] uses CGAN-based generation under diversity-related constraints. The main challenge is balancing validity and diversity: unconstrained generators may produce invalid tests, whereas overly restrictive generation may limit exploration.

5.2.6. Learning-Based Fuzzing

Learning-based fuzzing uses machine learning, deep learning, reinforcement learning, historical information, or learned policies to guide generation, mutation, seed selection, prioritization, or exploration. The learned component helps adapt the fuzzing process when the input or state space is too large or structured for random exploration alone. ReinSeed [69] uses reinforcement learning to select fuzzing actions, SeqDivFuzz [78] uses learned diversity information, RGChaser [79] uses reinforcement-learning-guided testing, and GMFuzz [80] combines learned assessment with coverage, mutation, and search. Learning-based guidance can improve prioritization and exploration, but may introduce additional computational cost, instability, or bias toward patterns observed during training.

5.2.7. LLM/Prompt-Guided Fuzzing

LLM/prompt-guided fuzzing uses large language models as central components for test generation, mutation, repair, transfer, constraint inference, fuzz-driver construction, or prioritization. Typical workflows prompt an LLM to generate structured tests, validate the outputs, execute them on the target, and repair or regenerate invalid cases. TitanFuzz [?] uses LLM-driven program generation and mutation for DL-library testing, while FuzzGPT [81], DFUZZ [82], YANHUI [83], MirrorFuzz [84], TransFuzz [85], FD-FACTORY [86], and ALF [87] illustrate different LLM-supported fuzzing tasks. These approaches are promising for complex and structured input spaces, but prompt sensitivity, nondeterminism, cost, and invalid generations make validation and repair essential parts of the workflow.

5.2.8. Search/Heuristic-Guided and Feedback-Driven Fuzzing

Search/heuristic-guided fuzzing uses target-specific objectives rather than relying only on structural coverage. The guidance signal may reflect prediction difference, robustness, vulnerability likelihood, historical behavior, curiosity, explainability, or search-based scores. DRFuzz [88] uses prediction-difference feedback, GMFuzz [80] combines learned assessment with MCTS-based search, DeFinder [89] uses vulnerability-oriented guidance, Orion [90] uses historical behavior, and CureFuzz [91] uses curiosity-driven feedback. This family is flexible because the objective can be adapted to the target, but its effectiveness depends strongly on whether the selected heuristic correlates with meaningful failure discovery.

5.2.9. Metamorphic Fuzzing

Metamorphic fuzzing evaluates expected relations between related inputs and outputs when exact ground truth is unavailable. A typical workflow transforms a seed under a relation that should preserve or predictably change behavior, executes the original and transformed inputs, and checks whether the expected relation is violated. DeepRoad [20] uses metamorphic transformations for autonomous-driving testing, QATest [92] applies metamorphic ideas to question-answering systems, and $\pi$-fuzz [93] targets reinforcement-learning systems. Metamorphic fuzzing helps address the oracle problem, but the usefulness of the approach depends on defining relations that are neither too weak to miss failures nor too strict to report acceptable variation.

5.2.10. Debugging/Fault-Localization-Oriented Fuzzing

Debugging/fault-localization-oriented fuzzing extends failure exposure toward diagnosis by identifying suspicious model components, behaviors, or regions associated with failing tests. This category appears only once in the normalized distribution and is represented by MODE [94]. Although uncommon, it highlights the potential for AI-system fuzzing to support not only bug detection but also explanation, localization, and eventual repair.

5.2.11. Hybrid Nature of AI-System Fuzzing Techniques

Many AI-system fuzzers combine several technique families rather than relying on a single mechanism. DeepHunter [10] combines mutation with coverage guidance; NNsmith [9] combines constraint-guided generation with differential testing; TitanFuzz [?] combines LLM-based generation and mutation with differential checking; ReinSeed [69] combines learned guidance, coverage reward, and scenario mutation; and GMFuzz [80] combines learning, search, coverage, and mutation. These hybrid designs help address input validity, large or structured input spaces, complex target behavior, and the absence of simple test oracles. Technique-family labels should therefore be interpreted as complementary mechanisms rather than mutually exclusive categories.

Extended descriptions of the technique families, the common fuzzing-workflow figure, additional representative studies, and family-specific strengths and limitations are available on the Extended Technique Analysis page of the companion website.

5.3. RQ3: Test Oracles Used in AI Fuzzing

RQ3 examines how the selected studies determine whether a generated test exposes a failure. Because AI systems often lack a clear expected output for every input, we analyze two related dimensions: oracle type and oracle construction. Oracle type captures the high-level failure-detection principle, whereas oracle construction records how that principle is implemented or approximated in practice. Because some studies use multiple oracle types or construction mechanisms, the categories are not mutually exclusive. Figure 8 summarizes their distributions.

We classify oracle types into four categories. Specification-based oracles check whether behavior satisfies rules, constraints, metamorphic relations, safety requirements, or other domain-specific conditions. Examples include task- or scenario-level checks used by MDPFuzz [16] and DriveFuzz [15].

Differential oracles compare outputs or execution behavior across models, implementations, frameworks, versions, devices, backends, or reference implementations. For example, NNsmith [9] compares compiled-model outputs with reference execution, while FlashFuzz [95] compares CPU and GPU behavior.

Crash/exception oracles treat crashes, exceptions, assertion failures, abnormal termination, or invalid execution states as failures. DocTer [32], for example, detects severe execution failures caused by generated DL-API inputs.

Inconsistency-based oracles identify unexpected prediction changes, behavioral deviations, or numerical instability when exact ground truth is unavailable. DeepHunter [10] and DLRegion [65] use behavioral inconsistency as a failure signal.

As shown in Figure 8, specification-based oracles are the most common, appearing in 44 studies. Differential oracles appear in 35 studies, followed by crash/exception oracles in 26 studies and inconsistency-based oracles in 25 studies. Specification-based approaches reflect the importance of partial rules, constraints, metamorphic relations, and domain requirements. Differential and crash/exception oracles are more common in framework/library and compiler/backend testing, whereas inconsistency-based oracles are especially relevant to model-level behavior.

Oracle choice varies substantially by testing target. Framework/library and compiler/backend studies often use differential and crash/exception oracles because multiple implementations, devices, versions, or execution paths can be compared. For example, NNsmith [9] compares compiled outputs with reference execution, while TzER [13] checks for inconsistent results, performance degradation, crashes, and unexpected exceptions.

Model-level studies more often rely on specification-based or inconsistency-based checks, including prediction changes, label inconsistencies, robustness constraints, metamorphic relations, or numerical deviations. System-level studies commonly use domain-specific rules, such as safety requirements, trajectory validity, policy behavior, or scenario constraints [15,28,96].

For oracle construction, rules/constraints are the most common mechanism, appearing in 62 studies. This category includes manually defined specifications, validity constraints, metamorphic relations, safety rules, domain requirements, and threshold-based checks. Heuristic construction appears in 34 studies and captures approximate signals, suspicious behavior, thresholds, or tool-specific checks used when exact expected outputs are difficult to define.

Comparison-based construction appears less frequently: multiple models are used in 14 studies, reference implementations in 12 studies, multiple backends in 2 studies, and multiple frameworks/libraries in 2 studies. These mechanisms are particularly useful when an implementation or execution path can be evaluated against a trusted or comparable alternative.

Several studies combine multiple oracle types or construction mechanisms. For example, NNsmith [9] combines differential comparison with crash detection; HIRGEN [34] checks crashes and inconsistencies across transformed programs and hardware devices; TzER [13] combines result, performance, and runtime checks; and FlashFuzz [95] uses both output and exception divergence across devices. These combinations show that oracle design in AI fuzzing is frequently hybrid.

Overall, RQ3 shows that oracle construction remains a central challenge in AI fuzzing. Specification-based, differential, crash/exception, and inconsistency-based oracles support different targets and failure behaviors, but none provides a complete solution across the AI software stack. This motivates stronger semantic, numerical, metamorphic, safety-aware, and domain-specific oracles.

The complete study-level mapping of oracle types and oracle-construction mechanisms, together with extended examples of hybrid oracle design, is available on the Oracle Study Mapping page of the companion website.

5.4. RQ4: Failure Types Exposed by AI Fuzzing

RQ4 analyzes the types of failures reported by fuzzing techniques for AI systems. Because some studies report more than one failure type, the categories are not mutually exclusive. Figure 9 summarizes the distribution of reported failures.

We classify the reported failures into several categories. Misclassification includes incorrect predictions, changed labels, degraded task outputs, or unstable model behavior caused by fuzzed inputs. This category is common in model-level studies, such as DeepHunter [10] and CtrlFuzz [97].

Crash/runtime error includes crashes, exceptions, assertion failures, abnormal termination, and invalid execution states. These failures are more common in framework/library and compiler/backend testing, where generated API calls, tensor programs, model graphs, or intermediate representations may trigger execution failures [9,13,32].

Numerical inconsistency refers to divergent or unstable results across implementations, devices, optimization levels, execution paths, or backends. Such failures are reported by studies including HIRGEN [34], TzER [13], and YANHUI [83].

Security vulnerability captures security-relevant defects in AI frameworks, libraries, operators, or models. Examples include vulnerabilities reported by ConFL [72] and ACETest [31].

Safety violation refers to behavior that violates task-, policy-, or environment-level safety requirements. This category primarily appears in autonomous-driving, reinforcement-learning, and other system-level applications, including DriveFuzz [15] and the approach of Tappler et al. [98].

As shown in Figure 9, misclassification is the most frequently reported failure type, appearing in 49 studies. Crash/runtime errors appear in 33 studies, while numerical inconsistencies appear in 29 studies. The dominance of these categories reflects the strong representation of model-, framework/library-, and compiler/backend-level testing in the corpus.

Model-level studies primarily expose prediction errors, unstable decisions, or degraded task behavior. In contrast, framework/library and compiler/backend studies more often expose execution failures, invalid states, backend mismatches, or numerical divergence across implementations and optimization paths.

Security vulnerabilities appear in 8 studies and are mainly associated with framework/library, API, operator-level, and security-oriented model testing. Safety violations appear in 6 studies and are primarily reported in autonomous-driving, reinforcement-learning, and other system-level settings.

Less frequent categories include performance bugs, behavioral inconsistency, coverage deficiency, degraded code summarization, translation errors, and autonomous-driving-specific perception or steering failures. Although individually uncommon, these categories indicate that AI-system fuzzing increasingly considers task-specific and domain-specific failures beyond prediction correctness.

The reported failure type is closely related to the system layer under test. Model-level studies mainly report misclassification or behavioral inconsistency because they focus on model outputs. Framework/library and compiler/backend studies more often report crashes, runtime errors, numerical inconsistencies, performance bugs, and security vulnerabilities because they exercise executable software components, including APIs, tensor operations, compiler passes, backends, and runtime systems. System-level studies report broader outcomes, such as safety violations, policy failures, trajectory-level problems, perception failures, and other domain-specific behavioral errors.

Several studies receive multiple failure labels because the same fuzzing workflow may expose more than one type of defect. For example, HIRGEN [34] reports both crash/runtime errors and numerical inconsistencies; TzER [13] reports execution failures, numerical inconsistencies, and performance bugs; and Tappler et al. [98] combine task-performance and safety-related failures. These cases show that failure categories are often complementary rather than mutually exclusive.

Overall, RQ4 shows that the failure model in AI fuzzing extends beyond crashes and exceptions. Although misclassification remains the most common category, the corpus also reports runtime failures, numerical inconsistencies, security vulnerabilities, safety violations, performance bugs, and specialized domain-specific failures. This diversity shows that failure detection is inherently target-dependent and that different layers of the AI software stack require different failure definitions and detection mechanisms.

The complete study-level mapping of failure categories, together with extended examples of specialized and multi-label failures, is available on the Failure Study Mapping page of the companion website.

6. Discussion

The results point to a field that is expanding quickly but still lacks a stable common structure. Across RQ1–RQ4, the same pattern appears repeatedly: fuzzing for AI systems is no longer limited to trained models, but the methods, oracles, and failure definitions still depend heavily on the layer being tested. RQ1 shows growth across models, frameworks/libraries, compiler backends, and system-level applications. RQ2 shows that techniques are increasingly hybrid. RQ3 highlights oracle construction as a persistent bottleneck, and RQ4 shows that failures include numerical, security, safety, and domain-specific behavior, not only crashes or misclassifications.

Before presenting the broader take-away messages, we consolidate the main observations from RQ1–RQ4 and relate them to the corresponding take-away messages. The complete observation-to-take-away mapping is available on the Observation-to-Take-away Mapping page of the companion website.

The recurring challenge themes, including input validity, oracle construction, scalability, and generalization, are integrated into the five take-away messages below. The detailed challenge table and extended challenge-theme analysis are available on the Challenge Synthesis page of the companion website.

T1: AI-system fuzzing is growing rapidly, but the research landscape remains uneven. 

RQ1 shows that AI-system fuzzing has grown substantially since 2021 and has expanded beyond trained models toward frameworks/libraries, compiler backends, and integrated AI applications. Early studies largely emphasized model behavior, as illustrated by DLFuzz [27] and DeepHunter [10], whereas later work increasingly examines frameworks and libraries, such as FreeFuzz [12], compiler backends, such as NNsmith [9], and system-level applications, such as DriveFuzz [15].

Despite this growth, the target distribution remains uneven. Model-level fuzzing continues to dominate, while compiler/backend and system-level fuzzing account for smaller portions of the literature. This imbalance is important because failures may arise not only from learned model behavior but also from model conversion, optimization, runtime execution, supporting software infrastructure, and interactions among deployed components.

Future research should therefore place greater emphasis on underexplored system layers and on cross-layer fuzzing. A model that behaves correctly in isolation may fail after conversion, optimization, deployment, or integration, while a defect in a framework, compiler, or runtime may affect many downstream models and applications. Cross-layer approaches should capture these dependencies and examine whether failure behavior generalizes across targets, platforms, and deployment settings.

T2: Fuzzing techniques are becoming more diverse, but many remain target-specific. 

RQ2 shows that AI-system fuzzing now includes mutation-based, coverage-guided, learning-based, constraint-guided, search/heuristic-guided, differential, LLM/prompt-guided, generation-based, metamorphic, and debugging-oriented approaches. Many techniques are hybrid because no single mechanism fully addresses input validity, exploration, feedback, and failure detection. For example, DeepHunter [10] combines mutation with coverage guidance, NNsmith [9] combines constraint-guided generation with differential testing, and TitanFuzz [?] combines LLM-based generation and mutation with differential checking.

However, technique design remains strongly dependent on the target. Image transformations, API constraints, tensor-shape rules, compiler grammars, prompts, and system scenarios require different representations, validity conditions, feedback signals, and mutation operators. Consequently, a technique developed for one target may not transfer directly to another.

Input validity remains a central challenge. Invalid tests may be rejected before reaching deeper behavior, whereas overly restrictive constraints may reduce exploration and prevent unusual but failure-revealing inputs from being generated. Future work should therefore develop reusable but target-aware fuzzing abstractions that support common stages such as generation, mutation, feedback, oracle checking, and failure triaging while preserving target-specific semantics and constraints.

T3: Oracle construction is one of the most important challenges in AI-system fuzzing. 

RQ3 shows that AI-system fuzzing relies on specification-based, differential, crash/exception, and inconsistency-based oracles because complete expected outputs are often unavailable. Rules and constraints are the most common construction mechanisms, while comparisons across models, implementations, devices, frameworks, backends, or reference executions provide alternative forms of evidence. For example, MDPFuzz [16] uses task-related behavioral checks, NNsmith [9] compares compiled outputs with reference execution, and TzER [13] combines result comparison with runtime and performance checks.

Each oracle type captures only part of the possible failure space. Crash and exception checks are practical but may miss silent semantic, numerical, or behavioral errors. Differential comparison can reveal implementation inconsistencies, but acceptable numerical variation, nondeterminism, and platform-specific behavior may complicate interpretation. Metamorphic, heuristic, and specification-based approaches reduce dependence on complete ground truth, but their assumptions may be difficult to define and may not generalize across tasks or domains.

Future work should develop stronger semantic, numerical, metamorphic, security-aware, safety-aware, and domain-specific oracles. For LLM-based and agentic systems, oracle construction may also need to consider multi-step behavior, tool-use correctness, instruction adherence, policy compliance, and end-to-end task completion.

T4: The failure model for AI-system fuzzing should go beyond crashes and exceptions. 

RQ4 shows that AI fuzzing exposes a broader range of failures than traditional crash-oriented testing. Although misclassification remains the most frequently reported category, the literature also includes runtime failures, numerical inconsistencies, security vulnerabilities, safety violations, performance bugs, and specialized task- or domain-specific failures.

The observed failure type depends strongly on the target layer. Model-level studies commonly report incorrect predictions or unstable behavior, as illustrated by DeepHunter [10]. Framework/library and compiler/backend studies more often reveal crashes, exceptions, numerical divergence, and implementation defects, as shown by NNsmith [9] and HIRGEN [34]. System-level studies require broader behavioral definitions involving safety, policy quality, trajectories, perception, or task completion, as illustrated by DriveFuzz [15].

Future work should therefore develop clearer and more systematic taxonomies that connect the testing target, oracle type, oracle-construction mechanism, and observed failure. Such taxonomies would support richer failure reporting and triaging and help distinguish implementation defects, numerical errors, semantic failures, security vulnerabilities, safety violations, and domain-specific behavioral failures.

T5: Reproducibility and benchmarking are essential for future progress. 

The findings across RQ1–RQ4 reveal substantial variation in experimental settings. Studies differ in their target layers, datasets, models, frameworks, APIs, operators, compilers, backends, simulators, scenarios, generation methods, feedback signals, oracle definitions, and reported failure types. These differences make direct comparison difficult and limit the extent to which conclusions can be generalized across targets.

Evaluation settings are also frequently tied to specific infrastructures. For example, TzER [13] and MLIR-Smith [30] focus on particular compiler environments, DriveFuzz [15] depends on simulation-based system testing, and TitanFuzz [?] introduces LLM-generation and validation costs. Such differences affect scalability, reproducibility, and the comparability of reported results.

Future research should prioritize reusable benchmarks, open artifacts, shared seed corpora, documented target versions, standardized oracle definitions, consistent failure labels, and common reporting protocols.

For this area to mature, future evaluations need to make their experimental conditions easier to inspect and repeat. This includes reporting target versions, execution budgets, oracle thresholds, failure deduplication rules, and artifact availability. A failure count alone is not enough to compare fuzzers across AI-system layers. Researchers also need to know how many generated tests were valid, how much of the target was explored, how false positives were handled, and whether the same results can be reproduced under comparable settings. Without this infrastructure, useful individual tools will remain difficult to compare or build upon.

7. Threats to Validity

External Validity 

External validity concerns whether the findings generalize beyond the selected studies. Our final corpus contains 125 studies identified within the January 2015–February 2026 search window. Despite searching multiple scholarly sources and applying a systematic selection process, relevant studies may have been missed because of limitations in search queries, indexing coverage, terminology, publication access, search interfaces, export tools, or publication timing. In particular, studies may describe related work using terms such as robustness testing, adversarial testing, validation, differential testing, compiler testing, or test generation without explicitly using the term “fuzzing.” The use of multiple Boolean term groups may also have excluded studies whose titles, abstracts, or indexed metadata did not contain all required concepts. In addition, result-window and retrieval limits in sources such as Google Scholar and Scopus may have restricted access to some records.

To mitigate these threats, we used broad search expressions, multiple data sources, metadata deduplication, predefined inclusion and exclusion criteria, and backward and forward snowballing. Snowballing served as a complementary recall-enhancement step for studies potentially missed because of indexing differences, terminology variation, or retrieval limits. We also re-examined papers from venues or publishers that raised quality or eligibility concerns and excluded them only when they lacked sufficient relevance, methodological or evaluation detail, or eligibility as primary studies. Nevertheless, the final corpus remains dependent on the selected search strategy, indexing coverage, screening decisions, publication accessibility, quality criteria, and temporal scope. Studies published after February 2026 are outside the survey scope, and the 2026 publication count should therefore be interpreted as partial coverage.

Internal Validity 

Internal validity concerns possible errors in study selection, metadata extraction, and annotation. Many studies combine multiple testing targets, technique families, input-generation or mutation strategies, oracle mechanisms, and failure types, making classification difficult. Ambiguity may arise when distinguishing, for example, mutation-based from coverage-guided fuzzing, specification-based from inconsistency-based oracles, learning-based from LLM/prompt-guided approaches, or model-level from system-level targets. Differences between authors’ terminology and our normalized taxonomy may also introduce classification errors.

To reduce this threat, we used a predefined annotation schema, allowed multiple labels where appropriate, recorded notes for ambiguous cases, normalized overlapping labels, and manually verified the extracted metadata against the original papers. LLM outputs were used only to support evidence extraction, summarization, and organization; final eligibility decisions, labels, and interpretations were determined through human review and correction.

The initial annotator-checker agreement was 144 of 208 candidate studies (69.2%). We interpret this value as a diagnostic measure of annotation difficulty rather than a formal inter-rater reliability score because the process combined full-text eligibility assessment with multi-label, target-dependent classification and used checker validation followed by consensus resolution. We did not compute Cohen’s kappa because the workflow did not involve independent final coding, which is a limitation of the validation process. All disagreements were documented, rechecked against the original papers, and resolved before producing the final analysis-ready dataset. Despite these safeguards, minor annotation errors may remain and could slightly affect category counts or assignments to overlapping categories.

Construct Validity 

Construct validity concerns whether the taxonomy accurately represents the concepts analyzed in this survey. The taxonomy covers testing target, system level, AI paradigm, framework/platform, technique family, input-generation and mutation strategies, oracle type and construction, failure type, and reported study characteristics such as limitations, threats, future work, and observed gaps. Although these dimensions were designed to align with the research questions, many techniques combine several mechanisms and therefore do not fit into mutually exclusive categories. Similarly, a study may use multiple oracle types or report multiple failure types.

To reduce overlap, we treated relevant dimensions as non-mutually exclusive and separated high-level technique families from input-generation and mutation strategies. The corresponding results explicitly indicate when categories are multi-label. Nevertheless, the taxonomy remains an abstraction of complex techniques, and other researchers may organize some concepts differently.

Conclusion Validity 

Conclusion validity concerns whether the reported findings are supported by the annotated data. We report descriptive trends rather than causal claims and indicate when quantitative categories are not mutually exclusive. Qualitative findings, including recurring limitations, observed gaps, challenge themes, and future implications, are synthesized from patterns across the selected studies rather than interpreted as statistically causal relationships. The discussion integrates these themes into five take-away messages and relates them to the testing target because techniques, oracles, failure types, and challenges differ across model, framework/library, compiler/backend, and system-level studies. These choices help keep the conclusions consistent with the evidence extracted from the selected corpus.

8. Conclusion

This survey examined 125 studies on fuzzing AI systems identified within the January 2015–February 2026 search window. The main finding is that AI-system fuzzing has grown from model-centered robustness testing into a broader cross-layer testing practice. The reviewed studies now cover trained models, frameworks and libraries, compiler backends, and complete AI-enabled applications, although these targets are still unevenly represented.

Mutation-based and coverage-guided fuzzing remain the most common technique families, while learning-based, constraint-guided, differential, generation-based, search/heuristic-guided, metamorphic, and LLM/prompt-guided approaches support increasingly diverse testing goals. Many studies combine several of these mechanisms, indicating that AI-system fuzzing is becoming more hybrid and target-dependent. Different targets require different forms of input generation, mutation, feedback, validity checking, and execution guidance; therefore, a single universal fuzzing strategy is unlikely to be sufficient.

Oracle construction remains one of the central challenges in this area. Because complete expected outputs are often unavailable, existing studies rely on specification-based, differential, crash/exception, inconsistency-based, metamorphic, and hybrid oracles. These mechanisms are useful but often partial and target-specific. The reported failure types are similarly diverse. Model-level studies primarily expose misclassification and behavioral inconsistency, framework/library and compiler/backend studies frequently report runtime failures and numerical inconsistencies, and system-level studies identify broader safety-, policy-, perception-, and task-related failures.

Future research should address several recurring challenges. First, input generation should become more target-aware: model-level testing requires realistic and semantics-preserving inputs, framework/library and compiler/backend testing requires stronger API, tensor, graph, and intermediate-representation constraints, and system-level testing requires diverse and safety-relevant scenarios. Second, stronger semantic, numerical, metamorphic, security-aware, safety-aware, and domain-specific oracles are needed to detect subtle failures while accounting for acceptable nondeterminism and implementation variation. Third, feedback and adequacy criteria should better reflect failure-revealing behavior across models, software components, and integrated systems rather than relying only on conventional code coverage. Fourth, reusable benchmarks, open artifacts, shared datasets, documented experimental settings, and standardized reporting protocols are needed to improve reproducibility and fair comparison. Finally, fuzzing methods for LLM-based applications and autonomous agents should address multi-step behavior, tool use, context changes, policy constraints, and interaction-level failures.

Overall, effective fuzzing for AI systems requires more than adapting traditional fuzzing loops to new inputs. Future methods need to account for the validity constraints, feedback signals, oracle assumptions, and failure modes of each layer in the AI software stack. Progress will depend on scalable input generation, stronger semantic and domain-specific oracles, reproducible benchmarks, and evaluations that connect component-level defects with end-to-end AI-system behavior.