Safety Concerns in Autonomous Driving: A Taxonomy of Test and Evaluation Approaches

1. Introduction

Autonomous Driving Systems (ADS) have rapidly advanced in recent years, fueled by breakthroughs in artificial intelligence, sensor technologies, and vehicular connectivity [1,2,3]. These systems are envisioned to reduce traffic accidents, improve efficiency, and enhance mobility across diverse populations. Yet, before large-scale deployment, ensuring their safety and reliability remains a formidable challenge [4,5,6,7,8]. A major barrier lies in the need for rigorous testing strategies capable of validating system performance under the diversity, uncertainty, and risk inherent in real-world conditions [9,10,11,12].

Public perception underscores these concerns. According to AAA’s 2024 annual autonomous vehicle survey [13], 66% of U.S. drivers reported being afraid of fully self-driving vehicles, while only 9% expressed trust, with fears largely driven by highly publicized accidents and safety incidents. As illustrated in Figure 1, such skepticism has remained persistent, signaling that safety concerns are not only technical but also societal barriers to adoption.

Testing therefore plays a central role in the development, validation, and regulatory approval of autonomous vehicles (AVs) [14,15,16,17]. Unlike conventional vehicles, AVs must independently perceive, predict, and act in dynamic environments, where unpredictable human behaviors, ambiguous road infrastructure, and rare but safety-critical events—such as sudden occlusions, erratic maneuvers, or sensor degradation—may occur [18,19,20,21,22,23]. Traditional metrics based on mileage accumulation and disengagement rates are increasingly recognized as inadequate: real-world testing rarely captures statistically rare edge cases and often wastes resources on uneventful scenarios [24,25,26,27].

As a result, scenario-based testing has emerged as a more systematic and safety-oriented paradigm. This approach centers on constructing and executing targeted scenarios that represent both common driving contexts and rare but hazardous corner cases [28,29,30]. These include aggressive lane changes, pedestrian dart-outs, or multi-agent negotiation in dense traffic [31,32]. Scenario-based testing enables structured evaluation of AV capabilities under uncertainty, while simulation platforms and formal scenario description languages (e.g., ASAM OpenSCENARIO, Scenic) facilitate repeatable, scalable, and transferable validation [33,34,35]. Regulatory frameworks such as ISO 21448 (SOTIF) and UNECE WP.29 are increasingly incorporating scenario-driven safety validation, underscoring its role in certification.

Within this context, scenario generation has become a critical enabler. Automated generation methods aim to synthesize diverse, realistic, and safety-critical test cases, going beyond manually curated or replayed datasets. These synthetic scenarios are especially valuable for stress-testing ADS against rare but high-risk events—such as near-miss collisions, unprotected turns, or aggressive cut-ins—that are unlikely to be encountered through brute-force road testing [36,37,38,39]. Recent research has explored reinforcement learning, Bayesian optimization, and other advanced algorithms to produce edge-case scenarios at scale [16,40,41,42,43,44,45,46,47,48,49,50,51]. Such approaches enable closed-loop simulations that iteratively challenge AV decision-making under uncertainty [52,53,54,55].

This survey provides a comprehensive review of scenario generation techniques for AV safety testing. We classify existing methods into three main paradigms: rule-based, data-driven, and learning-based approaches. For each, we analyze their algorithmic foundations, scenario description languages, evaluation metrics (e.g., coverage, criticality, diversity), and supporting simulation tools. We then highlight research gaps, including generalization across domains, realism of simulated environments, and the modeling of complex multi-agent interactions. Finally, we discuss emerging directions, such as semantic generation with large language models (LLMs), interactive multi-agent testing, and standardized scenario libraries for benchmarking [56,57]. We aim for this survey to serve as a resource for researchers and practitioners striving to enhance the safety, reliability, and public trust of autonomous driving systems.

2. Autonomous Driving Testing

Ensuring the safety and reliability of Autonomous Driving Systems (ADS) requires rigorous and systematic testing pipelines. Due to the complexity and unpredictability of real-world environments, conventional automotive testing methodologies are insufficient to evaluate the operational safety of AVs across their intended domain. To address this limitation, researchers and regulators have established multi-tiered testing frameworks that combine simulation, controlled track experiments, and on-road trials, thereby enabling comprehensive assessment under diverse conditions.

Testing strategies are commonly divided into three major categories: simulation-based testing, closed-track testing, and on-road testing, each with distinct contributions to the validation process.

Simulation-based testing leverages virtual environments to evaluate AV perception, planning, and control modules across a wide range of conditions. Platforms such as CARLA [58] and LGSVL [59] allow scalable, low-cost testing of scenarios including sudden occlusions, erratic maneuvers, and adverse weather or lighting [60]. Simulation ensures coverage of rare or hazardous cases that are often infeasible in physical testing. However, challenges remain in closing the sim-to-real gap, particularly in sensor modeling, environmental fidelity, and behavioral realism [61,62].

Closed-track testing provides controlled, repeatable environments at facilities such as Mcity [63] and AstaZero [64]. It serves as a bridge between simulation and on-road validation, offering safe conditions to evaluate critical functionalities like automated emergency braking, lane changes, and V2X communication systems [65]. While track testing ensures physical realism and safety, it remains costly and limited in terms of environmental diversity and traffic complexity.

On-road testing represents the final and most realistic stage, exposing AVs to real traffic agents, unstructured environments, and regulatory contexts. Companies including Waymo, Cruise, and Baidu Apollo have reported millions of kilometers of road testing to validate their systems. Despite its necessity, on-road testing is constrained by the rarity of safety-critical events, limited statistical coverage, and significant liability risks [24,66]. For this reason, it is typically preceded by extensive simulation and track-based testing.

Recent efforts emphasize hybrid validation pipelines that integrate these three levels: simulation for exploratory testing, closed tracks for controlled validation, and on-road deployment for certification. Table ?? provides a comparative summary of their trade-offs.

In the context of AV safety validation, it is essential to distinguish between three core concepts foundational to testing and scenario generation [67]:

• Scene: A static snapshot of the driving environment at a particular time, including road layout, infrastructure, traffic participants, environmental conditions, and the states of agents (e.g., position, velocity, heading). Scenes capture the spatial and contextual setup without temporal evolution.
• Scenario: A temporal sequence of scenes that models interactions among agents and unfolding events (e.g., overtaking, merging, emergency braking). Scenarios provide the basis for behavior modeling and safety testing [68].
• Test Case: A parameterized instantiation of a scenario, specifying initial configurations (e.g., positions, velocities, traffic density) along with measurable evaluation criteria. Test cases enable structured verification outcomes such as pass/fail or quantitative safety margins.

Such distinctions help formalize testing logic, improve comparability across datasets and tools, and ensure consistent interpretation in regulatory frameworks.

To support reproducible and certifiable ADS testing, a number of international standards and research initiatives have been proposed. These frameworks define methodologies for scenario modeling, simulation integration, and safety validation.

PEGASUS Project (Project for the Establishment of Generally Accepted Quality Criteria, Tools and Methods as well as Scenarios and Situations) is a German-led initiative that formalizes scenario-based testing pipelines. It emphasizes data-driven scenario derivation, parameterization using statistical distributions, execution in simulation, and coverage-driven evaluation [69]. PEGASUS has become a reference model in Europe for integrating simulation, track, and road testing with quantifiable safety arguments.

ISO 21448, also known as SOTIF (Safety of the Intended Functionality), complements ISO 26262 by addressing safety risks arising from limitations of perception and decision-making in complex or uncertain scenarios. It focuses on hazard identification and mitigation for failures not caused by hardware/software faults but by incomplete environment modeling or AI reasoning [69].

ASAM (Association for Standardisation of Automation and Measuring Systems) has developed open standards that enable interoperability across testing toolchains:

• OpenSCENARIO: Defines dynamic scenario logic, including actors, maneuvers, triggers, and actions.
• OpenDRIVE: Provides detailed road network geometry, topology, and infrastructure models.
• OpenXOntology: Establishes a shared semantic vocabulary to ensure consistent tool integration.

Additional efforts, such as ISO 34501/34502, introduce common terminologies, scenario taxonomies, and auditability criteria to ensure traceability in AV certification processes. Collectively, these frameworks mark a global shift toward scenario-centric testing that emphasizes reproducibility, coverage, and proactive hazard exposure.

2.1. Rule-Based Scenario Generation

Rule-based scenario generation constructs test cases based on predefined rules, templates, or expert knowledge. These methods often rely on regulatory conditions, structured parameter combinations, or safety-critical templates, offering high interpretability and traceability.

A classical line of work focuses on parameterized templates constrained by domain knowledge. Klischat and Althoff [70] proposed an approach that minimizes the solution space of the ego vehicle’s motion planner using evolutionary algorithms such as Differential Evolution and Particle Swarm Optimization. Their framework effectively generates high-risk scenarios in multi-vehicle interactions and intersections, though computational efficiency decreases in high-dimensional settings due to collision constraints.

Althoff and Lutz [71] combined reachability analysis with optimization methods to synthesize collision-avoidance scenarios. Their method rapidly constructs scenarios within seconds, focusing on the safety envelope of the ego vehicle. However, it does not yet extend to multi-step trajectories or adversarial agent modeling.

To improve diversity and efficiency, Feng et al. [72] introduced the Adaptive Testing Scenario Library Generation (ATSLG) framework. Leveraging Bayesian Optimization with Gaussian Process Regression, ATSLG incrementally refines the scenario space, reducing required test iterations by one to two orders of magnitude. While effective in discovering critical scenarios, its scalability in high-dimensional feature spaces remains limited.

Gao et al. [73] proposed a combinatorial test generation method (CTBC) that integrates test matrices with a complexity-driven algorithm. By incorporating hierarchical influence modeling through AHP, their framework balances coverage and complexity. However, the accuracy of complexity estimation is constrained by underlying approximations.

More recently, Zhang et al. [74] extended rule-based methods by incorporating knowledge-enhanced scenario synthesis with large language models (LLMs). This approach aligns natural language intent with parameterized generation, enhancing flexibility while maintaining the structured backbone of rule-based design.

Overall, rule-based approaches remain widely adopted for their interpretability, repeatability, and suitability for regulatory testing. Nonetheless, they struggle to capture rare, emergent, or long-tail interactions and often lack adaptability in open-ended driving environments.

2.2. Data-Driven Scenario Generation

Data-driven scenario generation exploits large-scale naturalistic driving datasets to extract, learn, and synthesize scenarios that reflect realistic traffic patterns, driver behaviors, and rare safety-critical cases. Unlike rule-based methods that rely on expert-crafted templates, this approach leverages statistical analysis, machine learning, and data mining to uncover latent dynamics directly from empirical driving records. The availability of rich datasets such as NGSIM, Argoverse, and SHRP2 enables scalable creation of test cases that closely mirror real-world complexity.

A dominant line of work is trajectory-based learning. Zhang et al. [75] proposed DP-TrajGAN, a GAN-based framework augmented with differential privacy to synthesize realistic vehicle trajectories while preserving data confidentiality. Their method achieved strong fidelity and privacy trade-offs on NGSIM and Argoverse. Similarly, Krajewski et al. [76] combined GANs and VAEs to model diverse vehicle maneuvers, improving both scenario realism and simulation variability.

To generate more safety-critical conditions, researchers have explored adversarial perturbation. Wang et al. [77] introduced AdvSim, which perturbs trajectories and LiDAR signals to create adversarial test cases. Their results demonstrated the effectiveness of adversarial replay in uncovering system vulnerabilities.

Another important research thread focuses on criticality assessment and targeting. Westhofen et al. [78] provided a systematic review of criticality metrics and proposed a framework to evaluate their applicability for AV testing. Kang et al. [79] applied voxel-based 3D modeling and vision transformers to identify latent safety threats in LiDAR data, reporting an F1 score of 98.26% for risky zone detection.

Driving instability has also been employed as a data-driven indicator of crash risk. Arvin et al. [80] analyzed SHRP2 crash and near-crash events, demonstrating a strong correlation between pre-crash instability and severity. Such findings can guide the synthesis of high-risk scenarios for testing.

Rare and corner-case detection represents another active frontier. Bolte et al. [81] proposed a hybrid offline–online anomaly detection framework to identify low-frequency, high-impact events. This enables automatic harvesting of rare driving situations from continuous driving logs.

From a parameterization perspective, Muslim et al. [82] introduced a cut-out scenario generation method that abstracts highway traffic into interpretable parameter boundaries. This ensures both plausibility and controllable variability. Along similar lines, Huang et al. [83] presented the CaDRE framework, which integrates

3. Evaluation

A critical component of scenario-based testing for autonomous driving is not only the generation of test scenarios but also their systematic evaluation. Without rigorous assessment, generated scenarios may fail to meaningfully contribute to safety validation. To this end, researchers have proposed a range of quality criteria and quantitative metrics to judge whether scenarios are realistic, diverse, comprehensive, and safety-critical. These evaluation dimensions determine how effectively scenarios expose weaknesses in the system under test and ensure their value in regulatory and industrial validation pipelines.

Scenario evaluation is generally structured around four core dimensions:

• Realism: Assesses how closely generated scenarios reflect real-world driving conditions, including plausible agent behavior, physically consistent motion dynamics, and compliance with traffic rules. Realism is crucial for external validity and is typically verified through statistical comparison with naturalistic datasets or human expert annotation [84,85,86].
• Diversity: Measures the variability among generated scenarios in terms of maneuvers, traffic densities, environmental conditions, and multi-agent interactions. High diversity enhances the likelihood of uncovering unexpected system failures and ensures broader stress-testing of ADS [87,88,89,90,91].
• Coverage: Captures how thoroughly scenarios span the operational design domain (ODD) and functional safety requirements. Coverage can be quantified by semantic labels (e.g., highway merging, unprotected left turns) or by parameter-space sampling strategies, ensuring systematic exploration of safety-relevant conditions [43,45,90,92].
• Criticality: Evaluates the level of risk or difficulty posed by a scenario. Metrics include time-to-collision (TTC), minimum distance, required deceleration, and collision probability, all of which help identify scenarios most likely to trigger unsafe ADS responses [34,88,93,94,95,96,97,98].

To operationalize these metrics, effective evaluation requires advanced simulation environments that are realistic, flexible, and extensible. Such simulators allow controlled testing of perception, planning, and control modules at scale, while reducing the risks and costs of physical trials. In parallel, scenario description languages (e.g., OpenSCENARIO, Scenic) provide structured mechanisms to encode, manipulate, and standardize test cases. Together, high-fidelity simulators and formalized scenario languages form the backbone of scenario-based testing pipelines, ensuring that evaluation is both reproducible and transferable across different platforms and stakeholders.

4. Challenges and Research Gaps

Despite significant progress in scenario generation for autonomous driving testing, several challenges and open research questions remain. These issues span technical, practical, and regulatory domains, highlighting the need for more robust, scalable, and standardized solutions. This section outlines the most critical challenges faced by current approaches and identifies gaps that present opportunities for future work.

4.1. Limited Data Diversity and Generalization

Most data-driven and learning-based generation techniques rely on real-world driving datasets, which, while extensive, often exhibit distributional bias. They tend to overrepresent common urban driving patterns and underrepresent edge cases such as near-misses, rare weather conditions, or unusual traffic interactions. As a result, generated scenarios may lack diversity and fail to generalize across unseen domains or geographies. Bridging this gap requires improved domain adaptation methods, cross-city or cross-country datasets, and techniques for transferring knowledge between different driving contexts.

4.1.1. Reality Gap in Synthetic Scenarios

A persistent challenge is the so-called reality gap—the mismatch between synthetic, simulator-generated scenarios and real-world driving environments. Even high-fidelity simulators may fail to capture subtle behaviors of pedestrians, occlusions, sensor noise, or infrastructure imperfections. This gap can lead to overestimation of AV performance in simulated testing and underpreparedness in real-world deployment. Addressing this issue involves combining real-world and simulated data, applying domain randomization, and improving simulator realism both at the perception and decision-making levels.

4.1.2. Scalability of Scenario Space

The scenario space for AV testing is practically infinite, encompassing a large number of interacting variables—agent types, behaviors, road layouts, environmental conditions, and temporal sequences. Exhaustive exploration of this space is infeasible. Thus, existing generation methods often sample from constrained subspaces, risking incomplete validation. New scalable methods for scenario space abstraction, semantic scenario clustering, and combinatorial coverage optimization are needed to ensure high testing efficiency without sacrificing thoroughness.

4.1.3. Modeling Safety-Critical but Rare Events

Safety-critical scenarios—such as sudden pedestrian crossings, aggressive merges, or multi-agent collisions—are rare but essential for validating robustness. However, they are underrepresented in data and difficult to model without manual engineering or adversarial optimization. Existing methods either rely on hand-crafted triggers or heuristic optimization, both of which may fail to cover unknown failure modes. Learning methods that can autonomously discover and amplify rare, high-risk patterns are still in their infancy and represent a vital area for advancement.

4.1.4. Standardization and Regulatory Alignment

A major obstacle in the deployment of scenario-based testing is the lack of unified standards for scenario modeling, evaluation, and exchange. While efforts like ASAM OpenSCENARIO and ISO 21448 (SOTIF) provide foundational standards, their integration into automated generation pipelines remains limited. Furthermore, regulatory acceptance of synthetic and generated scenarios for AV certification is still evolving. Research is needed on formal scenario verification, traceability, and compliance to ensure generated scenarios meet safety assurance requirements in a legally verifiable way.

5. Emerging Trends and Future Directions

As autonomous driving technologies mature, the demands on scenario generation frameworks are becoming more sophisticated. To support next-generation AV testing, recent research efforts have begun to explore novel paradigms that go beyond conventional rule-based or data-centric approaches. This section highlights several emerging trends and outlines promising future directions that are expected to shape the field in the coming years.

5.1. Semantic and Language-Driven Scenario Generation

The integration of large language models (LLMs) such as GPT and Claude into the AV testing pipeline has opened a new frontier: semantic scenario generation. Instead of specifying low-level scene parameters manually or learning them from data, users can now describe high-level scenarios in natural language—e.g., “a pedestrian suddenly crosses at a dimly lit intersection during rain”—which are then automatically translated into structured scenario code (e.g., OpenSCENARIO or Scenic). This paradigm enables more intuitive, human-centered interaction and lowers the barrier for specifying complex or rare situations.

Several early-stage systems now link language models with simulation backends (e.g., CARLA + LLM), enabling real-time scenario synthesis and editing. Future work may focus on integrating commonsense reasoning, legal constraints, and safety specifications directly into the generation pipeline through natural language interfaces.

5.2. Multi-Modal and Multi-Agent Scene Synthesis

Another active trend is the development of multi-modal scenario synthesis that incorporates visual, spatial, and behavioral information from multiple sources—such as video, LiDAR, maps, and text—to construct comprehensive test scenes. Generative models are being trained to combine these modalities into coherent environments, which better reflect the sensor fusion-based perception systems in real AVs.

In parallel, there is increasing interest in multi-agent interaction modeling. Modern urban scenarios often involve complex interactions among multiple agents with varying intent (e.g., pedestrians, cyclists, autonomous and human-driven vehicles). Modeling these interactions realistically, and generating coordinated behavior trajectories, remains a significant challenge. Multi-agent reinforcement learning, game-theoretic approaches, and diffusion-based generative models are emerging tools for tackling this complexity.

5.3. Hybrid Data-Driven and Rule-Based Approaches

Recognizing the limitations of pure rule-based or data-driven methods, researchers are moving towards hybrid frameworks that combine both strengths. Rule-based constraints provide safety and structure, while data-driven models contribute realism and diversity.

In practice, this might involve using data-driven models to sample base scenes and agents, with rule-based logic applied to inject specific intent, constraints, or triggers. Alternatively, hybrid approaches may operate in a layered architecture—where a symbolic planner outlines scenario semantics, and a learned module fills in the low-level details. These combinations are particularly promising for balancing interpretability with expressive power.

5.4. Towards Standardized Scenario Repositories and Benchmarks

As the field grows, there is a growing need for open, standardized scenario repositories and benchmarking protocols. Currently, many datasets and scenarios are either proprietary or fragmented, making reproducibility and comparative evaluation difficult.

Initiatives such as the ASAM OpenX family (OpenSCENARIO, OpenDRIVE, OpenLABEL) and projects like GENEVA or SAFETAG aim to unify scenario description formats and provide comprehensive libraries of validated test cases. Benchmarking tools that evaluate scenario quality, failure exposure, and coverage are also under active development.

In the future, publicly maintained scenario banks—similar to ImageNet for vision or GLUE for NLP—may become the cornerstone for training, testing, and certifying autonomous driving systems under global safety standards.

6. Conclusions

Scenario generation has emerged as a central pillar in the validation and verification of autonomous driving systems. This survey has reviewed the landscape of scenario generation techniques from multiple perspectives, including rule-based, data-driven, and learning-based approaches. We have discussed key simulation platforms and scenario description languages that underpin automated testing pipelines, as well as the metrics used to evaluate scenario quality in terms of realism, diversity, criticality, and reproducibility.

From the reviewed literature and practices, several important takeaways can be drawn. Rule-based generation provides structure and standardization but struggles with diversity and scalability. Data-driven approaches benefit from realism grounded in real-world observations but are constrained by dataset limitations and rarity of critical events. Learning-based methods offer promising adaptability and automation, especially for generating adversarial or failure-triggering scenarios, but face challenges related to safety, interpretability, and validation.

Scenario quality evaluation remains a non-trivial task, requiring multidimensional metrics and feedback from simulation environments. Tools like Scenic and general coverage metrics are gaining traction in quantifying scenario space exploration. Moreover, the growing integration of scenario generation with closed-loop simulators enables dynamic, intelligent testing strategies that evolve alongside AV system development.

Looking forward, advancing scenario generation will be key to achieving safe and efficient autonomous driving. Future research should prioritize hybrid and semantic methods that balance structure with adaptability, develop standardized scenario libraries and benchmarking protocols, and close the realism gap between synthetic and real-world driving environments. Through collaborative efforts in research, tool development, and regulation, scenario-based testing will continue to evolve as a robust framework for ensuring safety in increasingly complex autonomous systems.