A Novel Approach for Modelling and Developing Virtual Sensors Utilized in the Simulation of an Autonomous Vehicle

1. Introduction

Sensors are electronic devices that generate electrical signals in response to various environmental stimuli [1]. Sensors' operating principles are determined by how information is recorded, and sensors can be classified as resistive, piezoelectric, capacitive, optical, magnetic, QTC (Quantum Tunnelling Composite), triboelectric effect, FET (Field-Effect Transistor), and so on [2]. Solid-state sensors are integrated devices that pre-process an analog or digital signal before delivering a sensory response that embedded systems can process.

A virtual sensor is a software emulation of a physical sensor that uses real data, mathematical models, fuzzy logic, neural networks, genetic algorithms, and ML (Machine Learning) and AI (Artificial Intelligence) models to estimate parameter values and anticipate scenarios [3]. ML algorithms comprise two types of mechanisms: interactive learning, in which the virtual sensor identifies relevant elements in data streams, and automatic teaching, in which the human factor proactively contributes relevant elements to training the learning process [4]. Virtual sensors are data-driven prediction models that identify the physical characteristics of the system into which they are integrated, providing a viable substitute for real sensors while also being a cost-effective solution [5,6,7,8,9]. Another significant advantage of virtual sensors is the total reduction in vehicle mass, which includes both the mass of the physical sensors that have been replaced by virtual sensors and the mass of the wiring, insulators, and connectors that connect the physical sensors to the embedded systems, resulting in fuel and/or energy savings, in addition to a reduction in the volume of polluting emissions.

Virtual sensors have been developed by converting real sensor models into virtual models using real-world behavioral models, so that the generated output variables are represented with only input parameters relevant to the process being studied. Virtual sensor models are algorithms or mathematical models that estimate physical quantities using input data obtained from real sensors, statistical analysis, or input data generated by AI [10]. Virtual sensors are developed using software applications and do not require any additional or specific hardware components to function. According to [11], a virtual sensor is an only software sensor (no hardware components) that can generate signals autonomously by integrating data/signals obtained synchronously or asynchronously from physical sensors or even other virtual sensors. Virtual sensor models developed by [12] and implemented in the software architecture of real vehicles include, for example, the tire wear virtual sensor and the brake wear virtual sensor, which monitor wear, lifespan, and potential anomalies during the operation of some of the vehicle's systems and components (headlight levelling, tire pressure, tire temperature, e-motor temperature, brake temperature, suspension displacement, and so on) [13].

Virtual sensors offer a wide range of applications in vehicle active safety systems, including ABS (Antilock Braking System), AWD (All-Wheel Drive), ACC (Adaptive Cruise Control), and SAS (Smart Airbag System). The integration of virtual sensor parameters into the previously specified systems contributes to the management of the vehicle's optimal operating state in order to improve functional performance and reduce fuel/energy consumption, which is correlated with a reduction in pollutant emissions [14].

Implementing virtual sensors on real vehicles improves the accuracy of monitored data while also expanding coverage to locations where physical sensors are unavailable. When integrated into an embedded system, virtual sensors perform pre-processing, error correction, merging, and optimization of input data sets. Essentially, virtual sensors utilize an algorithm or mathematical apparatus to process input data and produce high-complexity output data sets that match specified requirements, as demonstrated by Hu et al. [15].

Unlike physical sensors, which must be added to precise positions inside a vehicle's structural architecture in order to function properly, virtual sensors depend on data sets collected from the vehicle's embedded systems. Based on this information, virtual sensors calculate specified parameters without the need for extra hardware. The development of a virtual sensor necessitates the implementation of a functional algorithm for the system under consideration, which is based on a statistical model that reliably anticipates the essential parameters being studied [16]. Because virtual sensors are made up of software components, firmware upgrades can be accomplished remotely via the OTA (Over The Air) approach, eliminating the need for physical interventions to remove and install these sensors.

Virtual sensors may improve data accuracy and resolution by merging information from numerous sources (other sensors, electronic control units, actuator feedback) using advanced data fusion and processing algorithms [17,18]. These sensors might be very simple or extremely sophisticated, depending on the activities and consequences they simulate: stimulus, electrical requirements, ambient environment, operational restrictions, and functional safety [19]. However, the performance of virtual sensors could decrease with time because of changes in nonlinear dynamics and the complexity of physical processes in the environment, as well as nonlinear interactions between input and output variables [6,20]. Virtual sensors increase the accessibility of data from physical sensors, facilitating collaboration at the sensor, equipment, and organizational levels (allowing service providers to offer solutions based on the same hardware), allowing for more efficient use of the same hardware resources in interconnected systems, such as IoT (Internet of Things). Virtual sensors take data acquired by physical sensors and incorporate it into complicated software applications, where it is merged with other sources of information (databases) and processed by specialized algorithms to produce meaningful results [1,21].

Figure 1 illustrates three combinations of connected virtual and real sensors:

a)

Virtual sensors depend only on the data from physical sensors. ESC (Electronic Stability Control) uses physical sensors like gyroscopes, accelerometers, wheel speed sensors, and virtual sensors to estimate yaw/slip angle, allowing the vehicle to maintain control in low-grip conditions or dangerous turns;

b)

Virtual sensors depend entirely on information from other virtual sensors. In the case of FCW (Forward Collision Warning), AEB (Automatic Emergency Braking), virtual sensor is used to predict the trajectory of the vehicle and evaluate the distance to other vehicles;

c)

Virtual sensors depend on data from both physical and virtual sensors. This configuration can be found in the DMS (Driver Monitoring System), which uses physical sensors like a video camera and/or pressure sensors in the steering wheel and/or seat, and virtual sensors like those for estimating the driver's level of attention and detecting the intention to leave the lane.

Finally, there is a requirement for using a suitable combination of physical and virtual sensors, in addition to maintaining functional algorithms for virtual sensors up to date [22].

Tactile Mobility [23] is a platform for monitoring, processing, and storing data from specific types of physical sensors that are installed in smart and interconnected vehicles in over 70 locations globally. This platform utilizes the data to create virtual sensors that, based on recorded scenarios, generate output parameters designed to improve the safety and performance of these vehicles [23]. The Tactile Mobility platform's solution incorporates a software program into the vehicle's built-in command and control systems, improving the operating regime by delivering information on road traffic, road conditions, tire grip and condition, vehicle mass, and so on.

Another platform that enables the use of virtual sensors in the automobile sector is the Compredict Virtual Sensor Platform [24], which calibrates, verifies, and implements these sensors on a wide range of real vehicle models. Thus, the Compredict platform can generate virtual models based on Cloud-stored input data for the following virtual sensor categories: suspension travel, brake wear, brake temperature, wheel force transducer, vehicle mass, strain gauge, tire wear, tire pressure, tire temperature, LV (Low Voltage) battery health, HV (High Voltage) battery health, and battery anomaly.

The development of virtual sensors is accelerating; according to a market study conducted by Mordor Intelligence for the period 2025-2030 in [25], the virtual sensor market will be worth 1.37 billion USD in 2025 and 5.35 billion USD by 2030. The advancement of smart manufacturing technologies, specifically the digitization of industrial processes in addition to the digitalization and validation of real vehicle models, contributes to the development of the virtual sensor sector.

Virtual sensors are critical for modern vehicles in terms of improving autonomous driving capabilities, safety, and efficiency. Table 1. shows the sensors’ progression and their implementation in a vehicle's constructive architecture from level 1 to level 5 automation (according to SAE J3016^TM) [26,27]. The sensors used in the equipment of autonomous vehicles are constrained by their physical dimensions, mass, the necessity to be positioned in less accessible sections of the vehicle's structural architecture, and the cost of these sensors [28]. It is evident that the number of sensors increases as automation levels rise with ultrasonic sensors and Lidar 2D/3D indicating the most significant numerical increases.

2. Classification of the Virtual Sensor

2.1. Virtual Sensor Model

According to research teams, the key real sensors frequently utilized in the constructive architecture of autonomous vehicles, sensors that form the basis for defining the virtual sensors used in modeling the virtual vehicle model, are as follows [29]:

• Camera sensors generate synthetic data on the recognition and classification of objects in the area [30,31,32], in addition to the vehicle's positioning and orientation relative to close to objects and V2V (Vehicle-to-Vehicle) communication [33] based on the VLC (Visible Light Communication) principle [34]. The advantages of the camera sensor include the ability to provide data in real time, low latency in data acquisition and processing, adaptability to extreme lighting conditions (low lighting, bright lighting), accurate estimation of object position and orientation, and low production and implementation costs. The constraints of camera sensors include the need for direct view of surrounding objects, susceptibility to unexpected changes in lighting conditions, and the need for greater computer capacity due to the large quantities of data that are constantly generated;
• Radar sensor generates data based on the reflection duration of radio waves ToF (Time of Flight) when detecting nearby target vehicles [35,36] and uses ML methods to estimate the current and future positions of nearby vehicles [37], respectively using DL (Deep Learning) methods to avoid collisions [38]. The benefits of radar sensors include the capacity to provide the location of target vehicles in real time, flexibility to severe weather conditions (rain, snow, fog), and low manufacturing and installation costs. The constraints of radar sensors include the requirement for increased computer capacity due to the massive volumes of data generated on a continuous basis, as well as a reliance on extra hardware systems and software;
• Lidar sensors provide a system based on generating a point cloud through 2D and 3D laser scanning for real-time localization of static and dynamic objects in proximity [39,40] and applies the YOLO (You Only Look Once) picture segmentation technique [41]. The advantages of lidar sensors include the ability to localize static and moving objects in proximity precisely. The disadvantages of lidar sensors include the need for greater computer power due to the large quantity of data generated continuously, sensitivity to bad weather conditions (rain, snow, fog), and high manufacturing and implementation costs.

Sensor fusion is the process of combining sensor signals [42,43] using CNN (Convolutional Neural Network) neural networks, processing these signals with DL-type AI elements [44,45], detecting nearby objects in real time [46,47], and then making predictions about the evolution of these objects [48,49,50,51].

The virtual sensor models presented in this review were created, tested, and calibrated using the CarMaker simulation application from IPG Automotive, which is extensively used in the automotive industry for virtual vehicle model development at all stages. CarMaker is a platform that enables the development of any virtual test scenarios that are connected to other software applications [52].

Yeong et al. [53] classified physical and virtual sensors as smart or non-smart. Smart sensors are directly related to the IoT concept, and they are systems made up of interconnected devices that may collect and transport data remotely without the need for human involvement. A smart sensor is an IoT device that can condition and select incoming signals, process and interpret the generated data, and make decisions without the assistance of a separate processing unit [54].

Virtual sensors can be classed using the following criteria [55,56,57]:

• Sensor fidelity could be classified as high, medium, or low-income;
• Method for collecting information from the environment:

  a)

  A deterministic strategy based on the simulation application's mathematical apparatus and involves the usage of a vast volume of input parameters to represent the ideal behavior and response of the virtual sensor as accurately as possible;

  b)

  A statistical technique based on statistical distribution functions, which include the normal, binomial, Poisson, or exponential distribution;

  c)

  The electromagnetic field propagation approach simulates electromagnetic wave propagation using Maxwell's equations.

• The objective of using sensors is to develop a vehicle's operating mode based on observed metrics and to perform diagnostics using AI-based maintenance techniques to define the smart maintenance regime.

Virtual systems developed for simulation applications use additional virtual sensors that are intended to replace certain partial functionalities of the main real sensors to reduce the volume of input data, reduce computing power requirements, calibrate the main sensor, and provide an optimized output data stream [28].

CarMaker classifies virtual sensors into three types: Ideal Sensors, Hi-Fi (High Fidelity) Sensors, and RSI (Raw Signal Interface) Sensors (Figure 2). These virtual sensor models are intended to maximize the performance of the virtual vehicle model on which they are installed, as well as to assist the command-and-control system in developing and expanding the specific capabilities of each sensor to a higher class of sensors [58,59].

The virtual model developed in CarMaker incorporates the following command and control systems for advanced assistance functions ADAS (Advanced Driver Assistance Systems): ACC, EBA (Emergency Brake Assist), LDW (Lane Departure Warning), LKA (Lane Keeping Assist), PA (Park Assist), ILA (Intelligent Light Assist), and TSA (Traffic Sign Assist). All these embedded systems evaluate and interpret data about the motor and/or vehicle's operating mode by combining various virtual sensor models [60].

Table 2 illustrates the functional properties of the main virtual sensor models, in addition to the range of applications and resources used by the system via the CPU (Central Processing Unit) and GPU (Graphics Processing Unit) [26,61].

2.1.1. Ideal Sensors

The role of ideal sensors in the CarMaker simulation program is to collect information from the simulation environment and transmit it to an embedded system. Ideal sensors are virtual entities developed using software that are independent of technology (Figure 3) that equip the virtual vehicle model with the following: Slip Angle, Inertial, Object, Free Space, Traffic Sign, Line, Road and Object by Line Sensor. Physical impacts that occur in the real environment in the case of a model integrated into the HiL (Hardware-in-the-Loop) system have no effect on these ideal sensors that are integrated into the SiL (Software-in-the-Loop) model and don't generate information similar to a real sensor.

2.1.2. Hi-Fi Sensors

Hi-Fi sensors filter the information supplied to the embedded system and provide data on the physical impacts that occur in the real environment, particularly the detection and classification of static and dynamic objects in the area. The virtual vehicle model is equipped with the following Hi-Fi sensors (Figure 4): Camera, Global Navigation and Radar sensors. Hi-Fi sensors have a role in reducing the impacts of false positives and false negatives that can occur in object perception and identification due to scenarios where part of the objects overlap, or environmental conditions prevent exact identification [63].

2.1.3. RSI Sensors

RSI sensors provide raw data and function identically to real sensors. The system filters, extracts, and interprets the data sent by the RSI sensors. Processing the information assigned by the RSI sensors necessitates high computational power, particularly for graphics processing provided by the GPU. There are RSI sensor types that do a post-processing of input in order to reduce the computing load on the embedded system. In the CarMaker simulation program, the RSI sensors identify objects in traffic and proximity, as well as all 3D surfaces in the surrounding environment. The utility IPGMovie, which is integrated with CarMaker, provides raw information for all these images. The virtual vehicle model is equipped with the following RSI sensors (Figure 5): ultrasonic RSI and lidar RSI [63].

The use of RSI sensors in a virtual environment requires modeling the properties of the materials that compose the objects in their vicinity, namely relative electric permittivity for electromagnetic waves and scattering effects. The direction and intensity of the field of waves reflected off 3D surfaces are significantly influenced by the material's characteristic properties [64].

RSI sensors process 3D images and offer real-time output for embedded systems, namely images and videos for the IPGMovie and/or MovieNX simulated scenario rendering system in CarMaker (Figure 6) [65].

2.2. Virtual Vehicle Model

A virtual vehicle model is a prototype that precisely replicates the characteristics of the elements and systems of a real model using mathematical and physical models. After validating the model, the simulation method enables the virtual vehicle to run in any user-defined scenario in a short period of time and at a low cost [19,66].

A virtual vehicle model, also referred to as a DTw (Digital Twin), is a digital image of a physical vehicle. Renard et al. [67] define DTw as an entity made up of a real model in a real space, a virtual model in a virtual space, and the data links that connect the real and virtual models. According to [68], the notion of DTw has gained popularity since 2017, with applications in a variety of academic and industry domains. DTw systems, thanks to bidirectional communication (Figure 7), allow the virtual model to be updated when the real model's state changes, and vice versa [68,69,70,71].

Navya Autonom® Shuttle is an autonomous shuttle bus vehicle designed for public passenger transportation and based on the architecture of a fully electric vehicle. Navya Autonom® Shuttle was introduced by the French start-up Navya in October 2015, and the main technical requirements (navya.tech) were utilized for developing the virtual model in the CarMaker implementing virtual sensors (Figure 8). The results of the implemented scenarios were generated using the IPGControl application, which is a tool to generate diagrams based on computer simulation results [26,72,73,74,75].

2.3. Virtual Environmental Model

The virtual environmental model consists of a virtual road and a virtual environment. The virtual road in which the virtual model of the autonomous vehicle travels was defined by digitizing the real route in Lyon, France using geographical coordinates (latitude, longitude, and altitude) extracted from Google Earth (Figure 9) [26,76,77]. The digitized route was converted to SRTM (Shuttle Radar Topography Mission) coordinates [77] using the GPSPrune application [78]. The route with the altitude profile was loaded into CarMaker's IPGRoad utility [79], which defined the following parameters in addition to the geographical coordinates (latitude, longitude, altitude): dimensions (length, width), connection angle, curvature, inclination, speed limit, and friction coefficient.

The virtual environment for the computer simulations was created using the CarMaker application's Environment utility, which allowed the following atmospheric conditions to be defined: reference temperature, air density, air pressure, air humidity, cloud model, cloud intensity, fog, visibility, rain rate, wind velocity, and wind angle [61].

The autonomous driving system operating algorithm in CarMaker's Vehicle Control section defined virtual driver behavior by addressing different driving styles corresponding to a human driver's reaction speed and performance criteria under optimal energy consumption conditions [80].

Virtual traffic was defined using data collected on the flow of vehicles and pedestrians on the digitized route, traffic rules, peak hours and traffic congestion, and intelligent transportation systems (in the roundabout on the route, a traffic light is connected to autonomous vehicle via V2I (Vehicle-to-Infrastructure) technology.

3. Characteristics of the Virtual Sensor

3.1. Characteristics of Ideal Sensor

3.1.1. Slip Angle Sensor

The slip angle sensor monitors the lateral slip angle between the steering wheel angle and the vehicle's direction of motion. The slip angle sensor is located near the vehicle’s steering wheel [26,61,65].

The yaw angle is helpful in active safety systems because it controls cornering stability, prevents vehicle rollovers, and avoids lane departure. Controlling the yaw angle is required because a big yaw angle reduces the tires' capacity to create lateral forces and greatly impairs the effectiveness of the vehicle control system. Beside yaw angle, also yaw rate are required variables for vehicle stability management [81,82].

The Pacejka model [83], a lateral force model, describes the complexity of the interaction between the tire and the road surface during dynamic maneuvers that are specific to autonomous vehicles. The Pacejka model is a semi-empirical mathematical model that represents the behavior of forces and moments created by a tire in contact with the road surface. It additionally provides a nonlinear representation of lateral forces, accounting for both large slip angles and normal forces. The Pacejka model (Figure 10) is commonly used in vehicle simulation and control, particularly for the development of advanced support systems and autonomous vehicles. The Pacejka model includes the following broad shape [82,83,84]:

$$\mathrm{y}\left(\mathrm{x}\right)=\mathrm{D}·\mathrm{s}\mathrm{i}\mathrm{n}\left\{\mathrm{C}·\mathrm{a}\mathrm{r}\mathrm{c}\mathrm{t}\mathrm{a}\mathrm{n}\left[\left(1-\mathrm{E}\right)\mathrm{B}\mathrm{x}-\mathrm{E}·\mathrm{a}\mathrm{r}\mathrm{c}\mathrm{t}\mathrm{a}\mathrm{n}\left(\mathrm{B}\mathrm{x}\right)\right]\right\},$$

(1)

with:

$$\mathrm{Y}\left(\mathrm{X}\right)=\mathrm{y}\left(\mathrm{x}\right)+{\mathrm{S}}_{\mathrm{V}},$$

(2)

$$\mathrm{x}=\mathrm{X}+{\mathrm{S}}_{\mathrm{H}},$$

(3)

where y(x) represents F_x, F_y, M_z, B is the stiffness factor, C is the shape factor, D is the peak value, E is the curvature factor, S_H is the horizontal displacement, and S_V is the vertical displacement.

The Pacejka model generates a curve that passes through the origin (x=y=0), reaches a maximum value, and then tends to a horizontal asymptote. To create a more accurate representation of tire behavior, the Pacejka model allows the curve's position to be adjusted by inserting two translations, (S_H) and (S_V). These translations allow for the adjustment of potential asymmetries in the experimental data, resulting in a better fit between the model and reality. For specific coefficients B, C, D, and E, the curve exhibits anti-symmetry with respect to the origin. The coefficient D specifies the peak value, whereas the product B, C, D determines the curve's beginning slope. The coefficient C changes the operating range limits in the general formula, determining the shape of the curve. The stiffness factor is defined as the coefficient B, which is calculated using the inclination reported to the origin. The coefficient E is introduced to manage the curvature peak while also controlling its horizontal position. The shape factor C can be derived from the peak height and horizontal asymptote using the following formula [82,83,85]:

$$\mathrm{C}=1\pm \left(1-{\displaystyle \frac{2}{\mathsf{\pi }}}\mathrm{a}\mathrm{r}\mathrm{c}\mathrm{s}\mathrm{i}\mathrm{n}{\displaystyle \frac{{\mathrm{y}}_{\mathrm{a}}}{\mathrm{D}}}\right).$$

(4)

Curvature factor E is computed from B and C for the position x_m of the peak value using the following equation (if C > 1) [86]:

$$\mathrm{E}={\displaystyle \frac{\mathrm{B}{\mathrm{x}}_{\mathrm{m}}-\mathrm{t}\mathrm{a}\mathrm{n}\left[\frac{\mathsf{\pi }}{\left(2\mathrm{C}\right)}\right]}{\mathrm{B}{\mathrm{x}}_{\mathrm{m}}-\arctan \left(\mathrm{B}{\mathrm{x}}_{\mathrm{m}}\right)}}.$$

(5)

The specific force is expressed as follows in both the longitudinal and transverse directions [87]:

$$\mathsf{\Gamma }=\mathrm{D}·\mathrm{s}\mathrm{i}\mathrm{n}\left\{\mathrm{C}·\mathrm{a}\mathrm{r}\mathrm{c}\mathrm{t}\mathrm{g}\left[\mathrm{B}·\mathrm{s}-\mathrm{E}·\left(\mathrm{B}·\mathrm{s}-\mathrm{a}\mathrm{r}\mathrm{c}\mathrm{t}\mathrm{g}\left(\mathrm{B}·\mathrm{s}\right)\right)\right]\right\}.$$

(6)

(c) The lateral dynamics of motor vehicles are described using the three-degree-of-freedom mathematical model, as shown in Figure 11. [81,88].

The three-degree-of-freedom mathematical model accounts for pitch, yaw, and roll motions [88,89,90] expresses the mathematical model equation as follows:

$$\dot{{\mathrm{v}}_{\mathrm{x}}}=\mathsf{\gamma }{\mathrm{v}}_{\mathrm{y}}+{\displaystyle \frac{1}{\mathrm{m}}}\left[\left({\mathrm{F}}_{\mathrm{x}1}+{\mathrm{F}}_{\mathrm{x}2}\right)\mathrm{c}\mathrm{o}\mathrm{s}\mathsf{\delta }-\left({\mathrm{F}}_{\mathrm{y}1}+{\mathrm{F}}_{\mathrm{y}2}\right)\mathrm{s}\mathrm{i}\mathrm{n}\mathsf{\delta }+{\mathrm{F}}_{\mathrm{x}3}+{\mathrm{F}}_{\mathrm{x}4}\right],$$

(7)

$$\dot{{\mathrm{V}}_{\mathrm{y}}}=-\mathsf{\gamma }{\mathrm{v}}_{\mathrm{x}}+{\displaystyle \frac{1}{\mathrm{m}}}\left[\left({\mathrm{F}}_{\mathrm{x}1}+{\mathrm{F}}_{\mathrm{x}2}\right)\mathrm{s}\mathrm{i}\mathrm{n}\mathsf{\delta }+\left({\mathrm{F}}_{\mathrm{y}1}+{\mathrm{F}}_{\mathrm{y}2}\right)\mathrm{c}\mathrm{o}\mathrm{s}\mathsf{\delta }+{\mathrm{F}}_{\mathrm{y}3}+{\mathrm{F}}_{\mathrm{y}4}\right],$$

(8)

$$\dot{\mathsf{\gamma }}={\displaystyle \frac{1}{{\mathrm{I}}_{\mathrm{z}}}}\left[\left({\mathrm{F}}_{\mathrm{x}1}+{\mathrm{F}}_{\mathrm{x}2}\right){\mathrm{l}}_{\mathrm{f}}\mathrm{s}\mathrm{i}\mathrm{n}\mathsf{\delta }-\left({\mathrm{F}}_{\mathrm{y}3}+{\mathrm{F}}_{\mathrm{y}4}\right){\mathrm{l}}_{\mathrm{r}}+\left({\mathrm{F}}_{\mathrm{y}1}+{\mathrm{F}}_{\mathrm{y}2}\right){\mathrm{l}}_{\mathrm{f}}\mathrm{c}\mathrm{o}\mathrm{s}\mathsf{\delta }+\left({\mathrm{F}}_{\mathrm{y}1}-{\mathrm{F}}_{\mathrm{y}2}\right){\mathrm{b}}_{\mathrm{f}}\mathrm{s}\mathrm{i}\mathrm{n}\mathsf{\delta }-\left({\mathrm{F}}_{\mathrm{x}1}-{\mathrm{F}}_{\mathrm{x}2}\right){\mathrm{b}}_{\mathrm{f}}\mathrm{c}\mathrm{o}\mathrm{s}\mathsf{\delta }-\left({\mathrm{F}}_{\mathrm{x}3}-{\mathrm{F}}_{\mathrm{x}4}\right){\mathrm{b}}_{\mathrm{r}}\right].$$

(9)

The equations for a vehicle is as follows: v_x is the longitudinal speed, v_y is the lateral speed, γ is the yaw rate, m is the mass, δ is the steering angle of the front wheels, I_z is the moment of inertia, F_xj and F_yj (j=1,2,3,4) are the longitudinal and lateral forces of the tires, l_f and l_r are the distances from the vehicle’s center of gravity to the front and rear axles, and b_f and b_r are half of the tread of the front and rear axles, respectively.

To run the simulations in CarMaker, the virtual vehicle model dynamics library was utilized, which calculates the wheel slip angle using the force and moment equilibrium equations as the derivative of the virtual vehicle’s slip and yaw ratio [91,92]:

$$\dot{\mathsf{{\rm B}}}=-\mathsf{\gamma }+{\displaystyle \frac{2{\mathrm{C}}_{\mathrm{f}}}{\mathrm{m}{\mathrm{v}}_{\mathrm{x}}}}\left(\mathsf{\delta }-\mathsf{\beta }-{\displaystyle \frac{{\mathrm{l}}_{\mathrm{f}}\mathsf{\gamma }}{{\mathrm{v}}_{\mathrm{x}}}}\right)+{\displaystyle \frac{2{\mathrm{C}}_{\mathrm{r}}}{\mathrm{m}{\mathrm{v}}_{\mathrm{x}}}}\left(-\mathsf{\beta }+{\displaystyle \frac{{\mathrm{l}}_{\mathrm{r}}\mathsf{\gamma }}{{\mathrm{v}}_{\mathrm{x}}}}\right),$$

(10)

$$\dot{\mathsf{\gamma }}={\displaystyle \frac{2{\mathrm{C}}_{\mathrm{f}}{\mathrm{l}}_{\mathrm{f}}}{{\mathrm{I}}_{\mathrm{z}}}}\left(\mathsf{\delta }-\mathsf{\beta }-{\displaystyle \frac{{\mathrm{l}}_{\mathrm{f}}\mathsf{\gamma }}{{\mathrm{v}}_{\mathrm{x}}}}\right)+{\displaystyle \frac{2{\mathrm{C}}_{\mathrm{r}}{\mathrm{l}}_{\mathrm{r}}}{{\mathrm{I}}_{\mathrm{z}}}}\left(\mathsf{\beta }+{\displaystyle \frac{{\mathrm{l}}_{\mathrm{r}}\mathsf{\gamma }}{{\mathrm{v}}_{\mathrm{x}}}}\right).$$

(11)

For β vehicle slip angle and γ vehicle yaw rate, C_f vehicle front tire cornering stiffness, m vehicle mass and v_x longitudinal velocity. Vehicle parameters include δ front-steering angle, β body slip angle, l_f distance from center of gravity to front axle, and v_x longitudinal velocity, C_r vehicle rear tire cornering stiffness, l_r distance between the center of gravity and the rear axle, I_z moment of inertia for the vehicle’s yaw axis.

3.1.2. Inertial Sensor

The inertial sensor determines the vehicle’s position, speed, and acceleration. It is based on a three-axis accelerometer (x,y,z) that outputs information about the vehicle’s translational speed, translational acceleration, and rotational acceleration. The inertial sensor is located in the center of the vehicle [26,61,65].

Inertial sensors, coupled with the slip angle sensor, comprise the inertial positioning system IMU (Inertial Measurement Unit), which also incorporates a three-axis accelerometer. Inertial measurements include linear acceleration, angular velocity, and angular acceleration. The dynamic parameters (roll, pitch, and yaw rate) measured by the inertial sensor are incorporated in the following relationships [93]:

$$\left[\begin{array}{c}\begin{array}{c}\dot{{\mathrm{Q}}_0}\\ \dot{{\mathrm{q}}_1}\\ \dot{{\mathrm{q}}_2}\end{array}\\ \dot{{\mathrm{q}}_3}\end{array}\right]={\displaystyle \frac{1}{2}}·\left[\begin{array}{c}\begin{array}{c}0\\ \dot{\mathsf{\varphi }}\\ \dot{\mathsf{\theta }}\end{array}\\ \dot{\mathsf{\phi }}\end{array}\begin{array}{c}\begin{array}{c}\dot{-\mathsf{\varphi }}\\ 0\\ \dot{-\mathsf{\phi }}\end{array}\\ \dot{\mathsf{\theta }}\end{array}\begin{array}{c}\begin{array}{c}\dot{-\mathsf{\theta }}\\ \dot{\mathsf{\phi }}\\ 0\end{array}\\ \dot{-\mathsf{\varphi }}\end{array}\begin{array}{c}\begin{array}{c}\dot{-\mathsf{\phi }}\\ \dot{-\mathsf{\theta }}\\ \dot{\mathsf{\varphi }}\end{array}\\ 0\end{array}\right]·\left[\begin{array}{c}\begin{array}{c}{\mathrm{q}}_0\\ {\mathrm{q}}_1\\ {\mathrm{q}}_2\end{array}\\ {\mathrm{q}}_3\end{array}\right].$$

(12)

In three-dimensional rotation computations, the quaternion ${\left[q_0{q}_1{q}_2{q}_3\right]}^T=Q$ represents the ϕ roll, θ pitch, and ψ yaw rate. Based on the Taylor series results, the quaternion solution from instant k to moment k+1 is:

$$\mathrm{Q}\left(\mathrm{k}+1\right)=\left(\mathrm{I}\left(1-{\displaystyle \frac{\mathsf{\Delta }{\mathsf{\Theta }}^2}{8}}\right)+{\displaystyle \frac{\mathsf{\Delta }\mathsf{\Theta }}{2}}\right)·\mathrm{Q}\left(\mathrm{k}\right),$$

(13)

$$∆\mathsf{\Theta }={\int }_{\mathrm{k}}^{\mathrm{k}+1}\left[\begin{array}{c}\begin{array}{c}0\\ \dot{\mathsf{\varphi }}\\ \dot{\mathsf{\theta }}\end{array}\\ \dot{\mathsf{\phi }}\end{array}\begin{array}{c}\begin{array}{c}\dot{-\mathsf{\varphi }}\\ 0\\ \dot{-\mathsf{\phi }}\end{array}\\ \dot{\mathsf{\theta }}\end{array}\begin{array}{c}\begin{array}{c}\dot{-\mathsf{\theta }}\\ \dot{\mathsf{\phi }}\\ 0\end{array}\\ \dot{-\mathsf{\varphi }}\end{array}\begin{array}{c}\begin{array}{c}\dot{-\mathsf{\phi }}\\ \dot{-\mathsf{\theta }}\\ \dot{\mathsf{\varphi }}\end{array}\\ 0\end{array}\right]\mathrm{d}\mathrm{t}\approx \left[\begin{array}{c}\begin{array}{c}0\\ ∆{\mathsf{\Theta }}_{\mathsf{\varphi }}\\ ∆{\mathsf{\Theta }}_{\mathsf{\theta }}\end{array}\\ ∆{\mathsf{\Theta }}_{\mathsf{\phi }}\end{array}\begin{array}{c}\begin{array}{c}-∆{\mathsf{\Theta }}_{\mathsf{\varphi }}\\ 0\\ -∆{\mathsf{\Theta }}_{\mathsf{\phi }}\end{array}\\ ∆{\mathsf{\Theta }}_{\mathsf{\theta }}\end{array}\begin{array}{c}\begin{array}{c}-∆{\mathsf{\Theta }}_{\mathsf{\theta }}\\ ∆{\mathsf{\Theta }}_{\mathsf{\phi }}\\ 0\end{array}\\ -∆{\mathsf{\Theta }}_{\mathsf{\varphi }}\end{array}\begin{array}{c}\begin{array}{c}-∆{\mathsf{\Theta }}_{\mathsf{\phi }}\\ ∆{\mathsf{\Theta }}_{\mathsf{\theta }}\\ ∆{\mathsf{\Theta }}_{\mathsf{\varphi }}\end{array}\\ 0\end{array}\right].$$

(14)

For dt sampling time, ${\mathsf{\Delta }\mathsf{\theta }=\left[{\mathsf{\Delta }\mathsf{\theta }}_{\mathsf{\varphi }}{\mathsf{\Delta }\mathsf{\theta }}_{\mathsf{\theta }}{\mathsf{\Delta }\mathsf{\theta }}_{\mathsf{\phi }}\right]}^{\mathrm{T}}$ angular increments include Δθ_ϕ, Δθ_θ, Δθ_ψ roll, pitch, and yaw angles.

Inertial sensors IMU and VDM (Vehicle Dynamic Model) can be used to explain the performance and safety of autonomous vehicles by properly determining slip angle and altitude. VDM is composed of two basic parts: a delayed estimator and a predictor (Figure 12) [94]. The delayed estimator includes two types of estimators: those based on IMU data and those based on vehicle dynamic models. IMU estimators directly estimate variables like speed and attitude, but VDM estimators use mathematical models based on measurements from other sensors, like wheel speed sensors.

Under normal driving conditions, data from VDM estimators are utilized to correct errors that may develop in IMU estimates, using a Kalman filter to predict roll and pitch angles [94].

The vehicle dynamics models provide a more in-depth understanding of the vehicle’s overall behavior, which enhances estimate accuracy. Dynamic models may become less accurate under harsh driving situations, such as hard breaking or quick bends. In such cases, the IMU estimators are temporarily separated from the VDM estimators to prevent error propagation. To synchronize the input from the two estimators, a delay is added to the VDM-based estimate. To predict the system’s current state, the predictor uses delayed estimates as well as information about the vehicle’s controls. This enables a more precise estimate of the slip angle and attitude, even in dynamic conditions [95].

3.1.3. Object Sensor

Scanning the environment is an important stage in an autonomous vehicle since it offers information that allows it to perceive and understand its surroundings. This first stage is critical for obtaining a thorough and up-to-date image of the traffic situation, allowing the vehicle to make informed decisions and travel safely. To perform this comprehensive and multidimensional scanning, autonomous vehicles employ a complex network of sensors, each with particular expertise in providing information. The Object Sensor is a software component that simulates the operation of a real video camera in an autonomous vehicle. It detects objects in traffic and estimates their distance, with the nearest object considered the target. The data is utilized to make decisions in autonomous driving. Object Sensor employs image processing techniques and AI to identify and track objects such as vehicles, pedestrians, and bicycles, and the distance between them can be calculated using trigonometric calculations [96].

Bewley et al. in [97] describe a simple and efficient technique to real-time multi-object tracking. This method focuses on the quick connection of items observed in consecutive frames, highlighting the need for accurate detection for quality tracking. Using conventional methods such as the Kalman filter and the Hungarian algorithm, the method achieves accuracy comparable to complex systems. This method’s simplicity and efficiency make it ideal for real-time applications including pedestrian tracking in autonomous driving systems.

To predict the position of an object in the frame, a linear motion model with constant velocity is utilized, which is unaffected by other objects or camera movement.

$$\mathrm{X}={\left[\mathrm{u},\mathrm{v},\mathrm{s},\mathrm{r},\dot{\mathrm{u},}\dot{\mathrm{v}},\dot{\mathrm{s}}\right]}^{\mathrm{T}},$$

(15)

where u and v represent the horizontal and vertical location of the target center in pixels, and s and r indicate the scale (area) and aspect ratio of the target's bounding box, respectively [97,98].

The DPM (Deformable Part Model) algorithm, which detects surrounding vehicles and pedestrians, is an advanced object recognition method. DPM looks for and evaluates the characteristics of target objects in the images collected by the video camera, where objects are defined as a collection of parts organized in a deformable configuration. Each part represents the local attributes of an object's appearance, whereas links between pairs of parts defined the deformable configuration. The DPM algorithm learns to identify objects of interest from the background by comparing positive and negative examples. Therefore, the algorithm develops a collection of filters that respond to the object's specific properties, such as edges, corners, and textures. A filter is built using a rectangular template defined by a matrix of dimensional vectors d. The response of the filter F at a position (x,y) on a feature map G is the "local dot product" of the filter and a sub-window of the feature map at (x,y) [99,100].

$$\sum _{{\mathrm{x}}^{\mathrm{\text{'}}},{\mathrm{y}}^{\mathrm{\text{'}}}}\mathrm{F}\left[{\mathrm{x}}^{\mathrm{\text{'}}},{\mathrm{y}}^{\mathrm{\text{'}}}\right]·\mathrm{G}\left[{\mathrm{x}+\mathrm{x}}^{\mathrm{\text{'}}},{\mathrm{y}+\mathrm{y}}^{\mathrm{\text{'}}}\right].$$

(16)

A feature pyramid is used to specify an object's size and position in a picture. A pyramid is a series of feature maps with varying resolutions. In practice, pyramids are created by computing a conventional image pyramid, smoothing, and repeatedly down-sampling. A feature map is then generated from each point in the picture pyramid, as depicted in Figure 13 [101,102].

In the CarMaker virtual environment, the Object sensor is a virtual sensor that detects objects in traffic and calculates their distance. The most appropriate target object is determined by its proximity to the sensor [26,61]. The Object sensor transmits information to the ACC system, which is responsible for automatically adapting the acceleration in the vehicle's movement so that it maintains a consistent speed in comparison to the vehicles in front of it [66].

Figure 14 illustrates the structure of the ACC system, which includes the object sensor. A cluster of two sensors collects information about detected objects, one using an antenna for objects at a distance (long range antenna) and one using an antenna for objects in close proximity to the vehicle (short range antenna) and generates a list of intercepted objects. Raw sensor data is used to locate and track detected objects (object sensor detected objects list), with tracking algorithms performing data fusion to ensure control over the vehicle's cruising speed while maintaining a safe distance from the relevant detected objects [65].

The object list in the object sensor interface transmits data about the objects in the CarMaker application database that are detected by the sensor's detection field. These objects are identified based on the following characteristics: object ID (IDentifier), object dimensions, object orientation, distance to the object, and object speed, which correspond to the angle of incidence between the sensor beam and the object.

The algorithm for detecting the closest object is based on scanning all objects within range and selecting the relevant target objects for the object sensor by identifying the trajectory and movement lane, respectively by calculating the movement speed and distance to the object (Figure 15) [65].

In Figure 15 ds represents the projected distance to target vehicle, α angle of the target vehicle in sensor frame, ds_x component in X direction of projected distance, dsy component in Y direction of projected distance in sensor frame, r turning radius, y_off imaginary vehicle offset in sensor frame at the target position, l_off half of vehicle lane width, ay lateral acceleration of vehicle, and v the vehicle speed. Target selection algorithms can configure sensor response in two modes:

• Nearest object, this is the closest visible object that is considered a relevant target;
• Nearest object in the path, this is the closest object within an interval of an estimated vehicle trajectory.

The following relationships characterize components in object sensors:

$${\mathrm{d}\mathrm{s}}_{\mathrm{x}}=\mathrm{d}\mathrm{s}·\cos \left(\mathsf{\alpha }\right),$$

(17)

$${\mathrm{D}\mathrm{s}}_{\mathrm{y}}=\mathrm{d}\mathrm{s}·\sin \left(\mathsf{\alpha }\right).$$

(18)

The following relationships describe the vehicle's offset in the sensor's perception of the position of adjacent objects, and thus the limits of the vehicle's trajectory.

$${\mathrm{y}}_{\mathrm{o}\mathrm{f}\mathrm{f}}=\mathrm{r}-\mathrm{d}=(\mathrm{r}-\sqrt{{\mathrm{r}}^2-{\mathrm{d}\mathrm{s}}_{\mathrm{x}}^2)}·\mathrm{s}\mathrm{i}\mathrm{g}\mathrm{n}({\mathrm{a}}_{\mathrm{y}}),$$

(19)

$${\mathrm{L}}_{\mathrm{o}\mathrm{f}\mathrm{f}}={\displaystyle \frac{\mathrm{v}\mathrm{e}\mathrm{h}\mathrm{i}\mathrm{c}\mathrm{l}{\mathrm{e}}_{\mathrm{l}\mathrm{a}\mathrm{n}\mathrm{e}}}{2}}+\mathrm{l}\mathrm{a}\mathrm{n}{\mathrm{e}}_{\mathrm{o}\mathrm{f}\mathrm{f}\mathrm{s}\mathrm{e}\mathrm{t}},$$

(20)

$${(\mathrm{y}}_{\mathrm{o}\mathrm{f}\mathrm{f}}-{\mathrm{l}}_{\mathrm{o}\mathrm{f}\mathrm{f}}<{\mathrm{d}\mathrm{s}}_{\mathrm{y}})\cap {(\mathrm{d}\mathrm{s}}_{\mathrm{y}}<{\mathrm{y}}_{\mathrm{o}\mathrm{f}\mathrm{f}}+{\mathrm{l}}_{\mathrm{o}\mathrm{f}\mathrm{f}}).$$

(21)

The CarMaker Object Sensor module generates an item list for each configured sensor, with quantities for each traffic object in the sensor’s view. After scanning the environment, the list of objects that can be recognized around the virtual vehicle model will include the following markers [65]:

• Object ID a name or a code used to identify an object;
• Path direction (reference and closest point);
• Relative distance and velocity (between the reference and the nearest positions);
• Relative orientation in the axle x-y-z (the reference point);
• Sensor frame’s x-y-z distances (between the reference and the nearest point);
• Sensor frame’s x-y-z velocity (between the reference and the nearest point);
• Flag object has been identified (in the sensor viewing area);
• Flag object has been identified (in the observation area);
• Incidence angles between the sensor beam and the object being detected from proximity;
• Width, height, length of the object, and height above the ground.

3.1.4. Free Space Sensor

The free space sensor is a software component that detects free space around the vehicle and uses this information to plan routes and avoid obstacles. The sensor creates an accurate map of the surrounding environment by combining data from several sensors, including cameras, lidar and radar. The sensor data is analyzed to identify barriers and compute their distances. The free space sensor information is utilized to build safe and efficient trajectories, making it an important component in assuring the safety and efficiency of autonomous vehicles.

The free space sensor Is an extension of the object sensor, with the sensor beam separated into horizontal and vertical segments. Each segment determines the closest point of the observed objects in traffic, in addition to the vehicle’s angle with respect to these objects and their respective speeds. The sensor scans the environment and determines the free and occupied area in the vicinity, guiding the vehicle’s progress through it [26,62].

The free space sensor plus is an extension of the sensor that detects all around items using a separate computational approach based on 3D image analysis. Three-dimensional image analysis of objects in proximity uses two filtering methods: the closest point on the object surface (nearest) and the strongest point on the object surface (strongest). The nearest finds the point on the object surface (represented by a pixel in the generated image) that is closest to the sensor position. The strongest determines the point on the object’s surface with the least reflection angle relative to the incident vector for each pixel within the sensor’s detecting range [65].

Open space identification methods use either 2D models (camera pictures) or 3D models (point clouds obtained from lidar sensors or stereo cameras). 2D approaches segment the road using low-level cues including color and texture, but they may fail if the road textures are inconsistent. However, 3D algorithms may have difficulties recognizing modest height variations, such as those between the road and the sidewalk. The hybrid method combines the benefits of 2D, and 3D modeling to overcome the limits of each methodology and provide more robust open space identification [103,104,105]. Therefore, the use of 3D information obtained from the input 3D point cloud renders road plane recognition more efficient. The road plane is determined in a parametric space, which includes the plane distance from the center of the room as well as the angle between the plane normal and the room main axis. The plane in the parametric space is described by the following equation [104]:

$$\mathrm{Z}·\mathrm{s}\mathrm{i}\mathrm{n}\mathsf{\theta }-\mathrm{y}·\mathrm{c}\mathrm{o}\mathrm{s}\mathsf{\theta }=\mathrm{d}·\mathrm{c}\mathrm{o}\mathrm{s}\mathsf{\theta }.$$

(22)

This estimate suggests that the camera height and direction to the road remain constant (Figure 16). Encoders mounted on the vehicle wheel are used to correct camera and point cloud translations obtained by simultaneous localization and mapping to a metric space. The distance scale obtained from the encoders is utilized to adjust the camera translation scale, which then automatically scales the point cloud to metric space. This is required since it assists in parametrically altering the plane given a known initialization of d, dependent on the camera height [105].

3.1.5. Traffic Sign Sensor

The traffic sign sensor recognizes pre-selected signs within its defined range and sight region. The sensor determines if the detected traffic sign is directly facing the vehicle and ranks the detected signs in ascending order of distance from it before identifying and classifying them. The information supplied to embedded command and control systems about detected traffic signs aids in the comprehension and interpretation of the traffic rules and conduct specified by it [26,62]. The traffic sign sensor is an ideal camera, equipped with an algorithm for recognizing traffic signs and the colors of traffic lights within its field of vision, which uses the identifiers assigned to traffic signs and traffic lights in close proximity to locate, classify, and interpret their operation [106].

HD (High-Definition) maps can provide insight into the environment in which road traffic evolves. HD maps give precise information about the environment where static road traffic occurs, including details about roads and obstacles, across a radius of more than 200 meters, even in locations with no direct vision (in bends). This information, when combined with data from cameras and lidar sensors, allows for exact vehicle localization. Currently, creating HD maps requires a professional technique that includes specialized topography and mapping with a MMS (Mobile Mapping System). These maps are created by integrating road pictures with 3D data extracted from point clouds. Zhang et al. in [107] developed an architecture for real-time HD map production that includes an industrial camera, a cutting-edge GNSS (Global Navigation Satellite System)/IMU, and a high-performance computer platform on board the vehicle (Figure 17). The semantic SLAM (Simultaneous Localization And Mapping) technology, which is based on an enhanced BiSeNet (Bilateral Segmentation Network), is used to extract semantic data from point clouds, including information about the traffic situation.

SLAM is expensive and takes a long time to create HD maps. Furthermore, HD maps may contain inconsistencies between recorded road signs and real-time local modifications. In addition to supporting drivers, intelligent object identification systems can help with roadside maintenance, including road signs, lane markings, and guardrails. Road sign recognition systems, for example, can check for potential anomalies using autonomous vehicles, as human inspection of a complete road infrastructure is difficult.

As a result, the traffic sign recognition technique is an essential component of both autonomous driving systems and road management systems. Methods for recognizing road signs have centered on researching key aspects such as color and shape [108]. These feature-based approaches are particularly sensitive over long distances and in poor light. The usage of object detection models based on CNN has recently become popular in road sign recognition systems. DL based object identification algorithms, such as YOLO models, aid in the correct recognition of road signs in traffic. YOLO model-based studies for road sign identification have demonstrated great performance when using publicly available reference datasets [59,109,110]. Figure 18 shows a YOLO model-based arrangement for traffic sign identification [59].

3.1.6. Line Sensor

The line sensor detects road markings on the roadway in the direction of driving and organizes them into left and right lines, recording information about each. The recorded data includes the lateral distance to the given reference points, the type of lines, their width and height, and the color code. To detect roadway lines, the sensor generates a sequence of planes based on seven points (five vertical planes and three inclined horizontal planes) deployed along the travel direction. Road markings on the roadway are identified by recording the intersection points of the vertical and horizontal planes created by the sensor and the lines on the road surface [65]

The tread detection algorithm, illustrated in Figure 19, starts by capturing an image of the road. The image is processed in multiple phases. Initially, the image is transformed from RGB to grayscale, then noise is reduced using a symmetric 2D Gaussian filter. The image is then processed to improve contrast in order to recognize road markings. A Sobel operator can be used to detect edges. Finally, a binary image highlighting the tread markings is generated (Figure 20) [111].

If a combined laser scanning and video camera system is used for lane marking identification, the method is based on a top-hat transformation, which is preprocessed using the vertical-gradient Prewitt operator to generate a binary image. The binary image is then processed with a PPHT (Progressive Probabilistic Hough Transform) to detect lane markers. Figure 21 displays the lane marking detection algorithm [91].

A Top-Hat transformation is used to increase image contrast while reducing interference from non-tread marks. The Top-Hat transformation is a mathematical model that recovers small-sized bright objects and details from pictures using the following relationship:

$$\mathrm{h}=\mathrm{f}-(\mathrm{f}°\mathrm{b}),$$

(23)

where f is the source image, h is the final image after performing the Top-Hat transformation, and “◦” is an operator that is realized by the Top-Hat transformation and controlled by the choice of the structuring element (b). The size of the structural element b determines how many elements are extracted from the image [91].

The Prewitt vertical gradient operator uses the following mathematical model to detect vertical edges in an image [112]:

$$\mathrm{G}(\mathrm{x},\mathrm{y})=\left|\mathrm{I}\right(\mathrm{x}+1,\mathrm{y}-1)+\mathrm{I}(\mathrm{x}+1,\mathrm{y})+\mathrm{I}(\mathrm{x}+1,\mathrm{y}+1)–\mathrm{I}(\mathrm{x}-1,\mathrm{y}-1)–\mathrm{I}(\mathrm{x}-1,\mathrm{y})–\mathrm{I}(\mathrm{x}-1,\mathrm{y}+1\left)\right|,$$

(24)

where I(x,y) represents the pixel intensity at the coordinates (x,y).

PPHT reduces the amount of computing required to correctly detect the markings, utilizing a linear mathematical model:

$$\mathrm{y}=\mathrm{m}\mathrm{x}+\mathrm{b},$$

(25)

where m is the slope of the line and b is the intercept at the origin.

3.1.7. Road Sensor

The road sensor determines the following roadway attributes up to a specified distance: road curvature, road marker attributes (speed limits), longitudinal and lateral slope, and the distance and angle of deviation when driving. This data is sent to embedded command and control systems to perform the following functions: LK (Lane Keeping), LDW, AD (Autonomous Driving), SD (Sign Detection), EM (Energy Management), FC (Fuel Consumption), WLD (Wheel Lifting Detection), and PT (Powertrain) (Table 3) [65].

The road sensor is located in the middle of the vehicle’s front wheels. The sensor’s technique for detecting roadway features is based on projecting a point along the route reference line. The deviation represents the lateral offset of the projected point from the route reference line.

3.1.8. Object by Line Sensor

Object by line sensors detect and transmit information on traffic lanes and traffic objects passing through them by assigning POI (Point Of Interest) points to each of these objects.

The route’s number of lanes is divided into LaneScope sections, which include the road axis (LandScope Center), the left (LandScope Left), and the right (LandScope Right).

LaneScopes are used to structure information about objects and traffic lanes (Figure 22), with s_min, s_max, t_min, and t_max defining the offsets of the extremities of traffic objects along the route [65].

The LaneScope center is calculated using the POI position. The LaneScope center course is considered to be the basis for establishing the left and right LaneScope. The center lane course is determined by track lists generated by the computation method. The algorithm determines whether there is a lane where the POI is located. It then creates two lanes (successor/predecessor) beginning with this lane. If no acceptable successor/predecessor route is found, the relevant lane for the object by line sensor is terminated.

Lindenmaier et al. in [113] used a GNN (Global Nearest Neighbor) approach to assign detected objects to the reference object category in an interval d_c, respectively d_c,lat, resulting in a minimal global association distance. The Mahalanobis distance relation d_MH(xi, zj) between the reference object xi and the detection object zj defines the distance matrix D_ij∈R^+(N,M), which serves as the foundation of the GNN method.

$${\mathrm{D}}_{\mathrm{i}\mathrm{j}}={\mathrm{d}}_{\mathrm{M}\mathrm{H}}\left({\mathrm{x}}_{\mathrm{i}},{\mathrm{z}}_{\mathrm{j}}\right)={\left({\mathrm{x}}_{\mathrm{i}}^{\mathrm{p}\mathrm{o}\mathrm{s}}-{\mathrm{z}}_{\mathrm{j}}^{\mathrm{p}\mathrm{o}\mathrm{s}}\right)}^{\mathrm{T}}·{\mathrm{S}}^{-1}·\left({\mathrm{x}}_{\mathrm{i}}^{\mathrm{p}\mathrm{o}\mathrm{s}}-{\mathrm{z}}_{\mathrm{j}}^{\mathrm{p}\mathrm{o}\mathrm{s}}\right),$$

(26)

where N and M are the number of reference and detected objects, and $x_i^{pos}$ and $z_j^{pos}$ are the position vectors of the respective objects. The matrix of covariance S is calculated using the cutoff distance ratio $d_r=d_c/d_{c,lat}$ as follows [114]:

$$\mathrm{S}=\left[\begin{array}{cc}\cos \mathsf{\alpha }& -\sin \mathsf{\alpha }\\ \sin \mathsf{\alpha }& \cos \mathsf{\alpha }\end{array}\right]·\left[\begin{array}{cc}1& 0\\ 0& {\mathrm{d}}_{\mathrm{r}}^2\end{array}\right]{·\left[\begin{array}{cc}\cos \mathsf{\alpha }& -\sin \mathsf{\alpha }\\ \sin \mathsf{\alpha }& \cos \mathsf{\alpha }\end{array}\right]}^{\mathrm{T}},$$

(27)

where α represents the angle of the road path at the longitudinal distance of the detected object x_i.

3.2. Characteristics of Hi-Fi Sensor

3.2.1. Camera Sensor

The virtual vehicle model is equipped with a sensor camera that is positioned front-to-back to provide a circular image of its surroundings. The sensor camera’s objective is to constantly monitor the movement of the virtual vehicle model along the selected route’s traffic lane in order to detect and classify static and dynamic objects in the surroundings, as well as recognize traffic signs and traffic light colors. Monocular cameras take 2D images without determining the distances to the monitored objects, whereas stereo cameras may determine the distance by measuring the difference between two images taken from various perspectives [72,115].

Environmental conditions may reduce visibility and affect the identification of nearby objects. The influence of these elements is determined using the following relationship:

$${\mathsf{A}}_{\mathrm{E}\mathrm{n}\mathrm{v}}=\mathrm{m}\mathrm{a}\mathrm{x}\left(1.0-{\displaystyle \frac{\mathrm{R}\mathrm{a}\mathrm{i}\mathrm{n}\mathrm{R}\mathrm{a}\mathrm{t}\mathrm{e}}{{\mathrm{R}\mathrm{a}\mathrm{i}\mathrm{n}\mathrm{R}\mathrm{a}\mathrm{t}\mathrm{e}}_{\mathrm{m}\mathrm{a}\mathrm{x}}}},0\right)·\mathrm{m}\mathrm{i}\mathrm{n}\left({\displaystyle \frac{\mathrm{V}\mathrm{i}\mathrm{s}\mathrm{R}\mathrm{a}\mathrm{n}\mathrm{g}\mathrm{e}\mathrm{I}\mathrm{n}\mathrm{F}\mathrm{o}\mathrm{g}}{{\mathrm{V}\mathrm{i}\mathrm{s}\mathrm{R}\mathrm{a}\mathrm{n}\mathrm{g}\mathrm{e}}_{\mathrm{m}\mathrm{a}\mathrm{x}}}},1\right),$$

(28)

where RainRate is the rain’s intensity, and VisRangeInFog is the direct visibility under foggy conditions.

The camera sensor's maximum error in measuring the distance (dist_Err,max) to nearby objects will be computed using the following formula:

$${\mathrm{d}\mathrm{i}\mathrm{s}\mathrm{t}}_{\mathrm{E}\mathrm{r}\mathrm{r},\mathrm{m}\mathrm{a}\mathrm{x}}={\displaystyle \frac{{\mathrm{d}\mathrm{i}\mathrm{s}\mathrm{t}}^2}{\mathrm{f}·\mathrm{b}}}·{\mathrm{d}}_{\mathrm{E}\mathrm{r}\mathrm{r}},$$

(29)

where dist is the actual distance to the monitored object, f is the focal length, and b is the baseline. d_Err represents the disparity error.

The x and y coordinates of the image of the item acquired by the camera are determined using the following formulas [116,117]:

$$\mathrm{x}={\displaystyle \frac{\mathrm{h}·({\mathrm{x}}^{\mathrm{\text{'}}2}+{\mathrm{f}}^2-{\mathrm{y}}^{\mathrm{\text{'}}}·\mathrm{f}·\tan \mathsf{\alpha })·\sin \mathsf{\beta }}{\sqrt{{\mathrm{x}}^{\mathrm{\text{'}}2}+{\mathrm{f}}^2}·(\mathrm{f}·\tan \mathsf{\alpha }+{\mathrm{y}}^{\mathrm{\text{'}}})}},$$

(30)

$$\mathrm{Y}={\displaystyle \frac{\mathrm{h}·({\mathrm{x}}^{\mathrm{\text{'}}2}+{\mathrm{f}}^2-{\mathrm{y}}^{\mathrm{\text{'}}}·\mathrm{f}·\tan \mathsf{\alpha })·\cos \mathsf{\beta }}{\sqrt{{\mathrm{x}}^{\mathrm{\text{'}}2}+{\mathrm{f}}^2}·(\mathrm{f}·\tan \mathsf{\alpha }+{\mathrm{y}}^{\mathrm{\text{'}}})}},$$

(31)

where h is the camera's height from the ground, f is its focal length, and α is the angle between the camera's optical axis and the horizontal line to the target. Breaking down the relationships results [118]:

$${\mathrm{t}\mathrm{a}\mathrm{n}}^{-1}{\displaystyle \frac{\mathrm{f}·\tan \mathsf{\alpha }}{\sqrt{{\mathrm{x}}^{\mathrm{\text{'}}2}+{\mathrm{f}}^2}}}+{\mathrm{t}\mathrm{a}\mathrm{n}}^{-1}{\displaystyle \frac{{\mathrm{y}}^{\mathrm{\text{'}}}}{\sqrt{{\mathrm{x}}^{\mathrm{\text{'}}2}+{\mathrm{f}}^2}}}={\mathrm{t}\mathrm{a}\mathrm{n}}^{-1}{\displaystyle \frac{\frac{\mathrm{f}·\tan \mathsf{\alpha }}{\sqrt{{\mathrm{x}}^{\mathrm{\text{'}}2}+{\mathrm{f}}^2}}+\frac{{\mathrm{y}}^{\mathrm{\text{'}}}}{\sqrt{{\mathrm{x}}^{\mathrm{\text{'}}2}+{\mathrm{f}}^2}}}{1-\frac{\mathrm{f}·\tan \mathsf{\alpha }}{\sqrt{{\mathrm{x}}^{\mathrm{\text{'}}2}+{\mathrm{f}}^2}}·\frac{{\mathrm{y}}^{\mathrm{\text{'}}}}{\sqrt{{\mathrm{x}}^{\mathrm{\text{'}}2}+{\mathrm{f}}^2}}}}={\mathrm{t}\mathrm{a}\mathrm{n}}^{-1}{\displaystyle \frac{\sqrt{{\mathrm{x}}^{\mathrm{\text{'}}2}+{\mathrm{f}}^2}·(\mathrm{f}·\tan \mathsf{\alpha }+{\mathrm{y}}^{\mathrm{\text{'}}})}{{\mathrm{x}}^{\mathrm{\text{'}}2}+{\mathrm{f}}^2-{\mathrm{y}}^{\mathrm{\text{'}}}·\mathrm{f}·\tan \mathsf{\alpha }}}={\mathrm{t}\mathrm{a}\mathrm{n}}^{-1}{\displaystyle \frac{\mathrm{h}}{\mathrm{d}}},$$

(32)

$${\displaystyle \frac{\sqrt{{\mathrm{X}}^{\mathrm{\text{'}}2}+{\mathrm{f}}^2}·(\mathrm{f}·\tan \mathsf{\alpha }+{\mathrm{y}}^{\mathrm{\text{'}}})}{{\mathrm{x}}^{\mathrm{\text{'}}2}+{\mathrm{f}}^2-{\mathrm{y}}^{\mathrm{\text{'}}}·\mathrm{f}·\tan \mathsf{\alpha }}}={\displaystyle \frac{\mathrm{h}}{\mathrm{d}}},$$

(33)

$$\mathrm{d}={\displaystyle \frac{\mathrm{h}({\mathrm{x}}^{\mathrm{\text{'}}2}+{\mathrm{f}}^2-{\mathrm{y}}^{\mathrm{\text{'}}}·\mathrm{f}·\tan \mathsf{\alpha })}{\sqrt{{\mathrm{x}}^{\mathrm{\text{'}}2}+{\mathrm{f}}^2}·(\mathrm{f}·\tan \mathsf{\alpha }+{\mathrm{y}}^{\mathrm{\text{'}}})}}.$$

(34)

3.2.2. Global Navigation Sensor

The global navigation sensor locates a vehicle by using the positions of at least four geostationary GPS satellites. The sensor determines real-time positioning in geographic coordinates using information about the transmission time of the extrapolation signal emitted by the satellites and received by the vehicle’s receiver (x,y,z,t) [119].

CarMaker can represent any position of the virtual vehicle model in the global road architecture as a geographic point on the Earth’s surface. This uses the GCS (Geographic Coordinate System) coordinate system, which consists of latitude, longitude, and altitude.

The origin of the road frame on the Earth’s surface is determined using the FlatEarth projection method using GCS reference points (Figure 23). This projection method ignores the Earth curvature around the GCS reference point when calculating the relative position $\overrightarrow{\mathrm{R}\mathrm{e}\mathrm{f}\mathrm{P}\mathrm{o}}$ of point P in the road frame. The elevation value h at point P is calculated as follows [65]:

$$\mathrm{h}={\mathrm{h}}_{\mathrm{R}}+\overrightarrow{\mathrm{R}\mathrm{e}\mathrm{f}\mathrm{P}\mathrm{o}}{\left[\mathrm{z}\right]}_0.$$

(35)

The relative latitude Δϕ can be calculated as follows:

$$\mathsf{\Delta }\mathsf{\Phi }=\mathrm{a}\mathrm{s}\mathrm{i}\mathrm{n}\left({\displaystyle \frac{\mathsf{\Delta }\mathrm{y}}{{\mathrm{R}}_{\mathrm{N}}\left({\mathsf{\varphi }}_{\mathrm{R}}\right)+\mathrm{h}}}\right)\approx {\displaystyle \frac{\mathsf{\Delta }\mathrm{y}}{{\mathrm{R}}_{\mathrm{N}}\left({\mathsf{\varphi }}_{\mathrm{R}}\right)+\mathrm{h}}}={\displaystyle \frac{\overrightarrow{\mathrm{R}\mathrm{e}\mathrm{f}\mathrm{P}\mathrm{o}}{\left[\mathrm{y}\right]}_0}{{\mathrm{R}}_{\mathrm{N}}\left({\mathsf{\varphi }}_{\mathrm{R}}\right)+\mathrm{h}}},$$

(36)

where ${\mathrm{R}}_{\mathrm{N}}\left({\mathsf{\varphi }}_{\mathrm{R}}\right)$ is the radius of the Earth ellipsoid in the north, which is determined by the latitude of the GCS reference point. The latitude ϕ of point P is transformed into:

$$\mathsf{\Phi }={\mathsf{\varphi }}_{\mathrm{R}}+\mathsf{\Delta }\mathsf{\varphi }.$$

(37)

Similarly, the longitude at point P can be determined using the following formula:

$$\mathsf{\lambda }={\mathsf{\lambda }}_{\mathrm{R}}+{\displaystyle \frac{\overrightarrow{\mathrm{R}\mathrm{e}\mathrm{f}\mathrm{P}\mathrm{o}}{\left[\mathrm{z}\mathrm{x}\right]}_0}{\left({\mathrm{R}}_{\mathrm{E}}\left({\mathsf{\varphi }}_{\mathrm{R}}\right)·\cos \left({\mathsf{\phi }}_{\mathrm{R}}\right)\right)}},$$

(38)

where ${\mathrm{R}}_{\mathrm{E}}\left({\mathsf{\varphi }}_{\mathrm{R}}\right)$ is the radius of the Earth ellipsoid in the east, which is determined by the latitude of the GCS reference point. The factor cos(ϕ_R) considers decreasing radius with increasing latitude.

D-GNSS (Differential-Global Navigation Satellite System) with differential RTK (Real-Time Kinematic) correction reduces vehicle positioning errors by applying a differential correction relative to the coordinates of a reference base station, which then uses signals transmitted by satellites. The delivery of D-GNSS differential corrections with RTK from the reference base station to the vehicle takes place via mobile data connections [26].

CarMaker's virtual car model employs Cartesian coordinates (x, y, z), geocentric coordinates ECEF (Earth Centered Earth Fixed), and ellipsoidal coordinates (ϕ latitude, λ longitude, h elevation) for geostationary satellite positioning. GDOP (Geometric Dilution of Precision) refers to the accuracy of the computed position of the vehicle receiver in reference to the geostationary position of visible satellites.

3.2.3. Radar Sensor

The radar sensor detects static and dynamic objects on the virtual vehicle route using the SNR signal. The sensor detects things using cellular units and accounts for the effects of overlapping traffic items. The detected items are identified using particular RCS (Radar Cross Section) cross-section maps that take into account the angle of incidence and the signal reflected by the traffic objects. Depending on the signal-to-noise ratio, surrounding objects will be recognized, removed, or regarded as false negatives [26].

RCS is defined as the cross-section of a detected object that intercepts the most amount of power transmitted by the radar sensor. It is determined by the following parameters: the size and shape of the object, the antenna orientation angle, the frequency and polarization of the radar waves, the object's electromagnetic properties, and the surface structure. Radar sensor simulation is complex due to the dispersion effect of radar waves in virtual settings, and it is achieved using physically interpretable characteristics such as the distance to nearby objects, the movement speeds of traffic items, and their angular locations.

Elster et al. in [120] used the DVM (Double Validation Metric) methodology to validate radar sensor data virtually. DVM uses the reliability and repeatability of radar sensor readings to effectively quantify deviations between distributions for various types of detected objects. The measurement data (M1, M2) is preprocessed and filtered, and the number of points resulting from the measurements is determined for each data set using the EDF (Empirical cumulative Distribution Function) methodology.

The deviation of mean values d_bias for measurement data M1 is calculated and compared to measurement data M2. The shape deviation of the distribution function d_CAVM is calculated to highlight the difference in signal scattering between M1 and M2 (Figure 24) [121].

The radar equation defines the physical relationships connected to the features of the radar sensor, resulting in the received signal power ρ_s(r_s,υ_s,ϕ_s) [65,122,123].

$${\left|{\mathsf{\varrho }}_{\mathrm{s}\mathrm{P}}\left({\mathrm{r}}_{\mathrm{s}},{\mathsf{\upsilon }}_{\mathrm{s}},{\mathsf{\varphi }}_{\mathrm{s}}\right)\right|}^2={\displaystyle \frac{{\mathrm{P}}_{\mathrm{R}\mathrm{x}}}{{\mathrm{P}}_{\mathrm{T}\mathrm{x}}}}={\displaystyle \frac{{\mathsf{\lambda }}_{\mathrm{c}}^2·{\mathrm{G}}_{\mathrm{T}\mathrm{x}}\left({\mathsf{\upsilon }}_{\mathrm{s}},{\mathsf{\varphi }}_{\mathrm{s}}\right)·{\mathrm{G}}_{\mathrm{R}\mathrm{x}}\left({\mathsf{\upsilon }}_{\mathrm{s}},{\mathsf{\varphi }}_{\mathrm{s}}\right)·{\mathsf{\sigma }}_{\mathrm{s}}}{{\left(4\mathsf{\pi }\right)}^3·{\mathrm{r}}_{\mathrm{s}}^4}}·\mathrm{F}{\left({\mathrm{r}}_{\mathrm{s}},{\mathsf{\upsilon }}_{\mathrm{s}},{\mathsf{\varphi }}_{\mathrm{s}}\right)}^4,$$

(39)

where ρ_s is the impulse response for s point scatterer, positioned at a r_s distance at angular position, for υ_s elevation and ϕ_s azimuth, λ_c wavelength of the carrier frequency, G_Tx gain of the transmitter antenna, G_Rx gain of the receiver antenna, and σ_s reflection coefficient of the point scatterer.

A gain map is used to highlight antenna features, which is characterized by a unidirectional gain factor parameterized by azimuth and elevation in the scanning direction. The following equations describe field strength and antenna gain [124]:

$$\mathrm{f}\left(\mathsf{\theta },\mathsf{\varphi }\right)={\displaystyle \frac{\sin \left(\mathsf{\pi }\frac{\mathrm{a}}{\mathsf{\lambda }}\sin \mathsf{\theta }\cos \mathsf{\varphi }\right)}{\left(\mathsf{\pi }{\mathrm{v}}_{\mathrm{y}}\right)}}{\displaystyle \frac{\sin \left(\mathsf{\pi }\frac{\mathrm{b}}{\mathsf{\lambda }}\sin \mathsf{\theta }\sin \mathsf{\varphi }\right)}{\left(\mathsf{\pi }{\mathrm{v}}_{\mathrm{z}}\right)}},$$

(40)

The aperture dimensions include θ elevation, ϕ azimuth, a, and b major lobes.

The detection threshold can be determined by the least detectable value of the SNR_min and is calculated using the parameters minimal probability of detection P_Dmin and probability of false alarm P_FA [65]:

$${\mathrm{S}\mathrm{N}\mathrm{R}}_{\mathrm{m}\mathrm{i}\mathrm{n}}=2{\left({\mathrm{e}\mathrm{r}\mathrm{f}\mathrm{c}}^{-1}\left(2{\mathrm{P}}_{\mathrm{F}\mathrm{A}}\right)-{\mathrm{e}\mathrm{r}\mathrm{f}\mathrm{c}}^{-1}\left(2{\mathrm{P}}_{\mathrm{m}\mathrm{i}\mathrm{n}}\right)\right)}^2.$$

(41)

The strength S of the received signal is determined by the radar equation:

$$\mathrm{S}={\displaystyle \frac{\mathrm{P}{\mathrm{G}}^2{\mathsf{\lambda }}^2\mathrm{R}\mathrm{C}\mathrm{S}}{{\left(4\mathsf{\pi }\right)}^3{\mathrm{r}}^4}}·{\displaystyle \frac{1}{{\mathrm{L}}_{\mathrm{A}}{\mathrm{L}}_{\mathrm{a}\mathrm{t}\mathrm{m}}}},$$

(42)

where P is the transmitted power, G is the gain of the antenna, λ is the wavelength, r is the distance from the radar sensor to the object, L_A is the additional system losses and L_atm is the atmospheric losses.

The specific RCS cross-section maps for close objects are determined by radar sensor resolution, object size, direction of incidence, object occlusion, and object merging (Figure 25) [26].

The transmit gain map adjusts the power of the delivered signal to 3D objects. The transmit gain is calculated using linear interpolation of the parameterized gain map (Figure 26).

3.3. Characteristics of RSI Sensor

3.3.1. Lidar RSI Sensor

The lidar (light detection and ranging) sensor operates based on measuring the ToF for a beam of laser pointers with a wavelength of 905 nm that is emitted at objects in the area and computing the reception time of this beam reflected by these items. Lidars, including RSI sensors, generally use the ToF principle to determine distance. This involves generating a laser pulse and measuring its return time after hitting an object, as shown in Figure 27 [125].

The lidar 2D sensor captures information about nearby objects by sending a single laser beam onto a revolving mirror perpendicular to the axis of rotation. The lidar 3D sensor gathers information about nearby objects by shooting a beam of laser rays through a rotating mechanism, resulting in a point cloud for the contour of these objects and the capacity to build high-precision 3D maps.

Lidar RSI sensors use a rotating equipment to guide a laser beam across their range of vision. This scanning is often accomplished using rotating polygonal mirrors that direct the beam with each facet, resulting in a balance of accuracy and speed. Alternatively, revolving prisms can refract the beam with great precision and stability. A recent technique uses MEMS (Micro-ElectroMechanical Systems) technology, in which small mirrors direct the beam, allowing for more compact designs but potentially limiting the range and scan angle.

Lidar generates a 3D representation of the viewed reality by measuring the ToF at multiple points within their FoV (Field of View). The set of points is referred to as a point cloud. The n_th measurement point (p_n) in the lidar reference system {L} can be expressed as follows [126]:

$${{\left\{\mathrm{L}\right\}}_{\mathrm{p}}}_{\mathrm{n}}={\left[{{\left\{\mathrm{L}\right\}}_{\mathrm{x}}}_{\mathrm{n}},{{\left\{\mathrm{L}\right\}}_{\mathrm{y}}}_{\mathrm{n}},{{\left\{\mathrm{L}\right\}}_{\mathrm{z}}}_{\mathrm{n}}\right]}^{\mathrm{T}}={\displaystyle \frac{\mathrm{c}}{2}}{{\mathrm{t}}_{\mathrm{T}\mathrm{o}\mathrm{F},}}_{\mathrm{n}}·{{\left\{\mathrm{L}\right\}}_{\widehat{\mathrm{s}}}}_{\mathrm{n}},$$

(43)

where c is the speed of light in air, t_ToF,n is the ToF measure, and ŝ_n is the unitary vector indicating the scanning direction of the lidar in its reference system {L}. Equation (43) shows that accurate and exact point clouds require both the ToF measurement (t_ToF,n) and the scanning direction (ŝ_n). The scanning direction of the lidar, designated as ŝ, is expressed in the laser source’s reference system {I}. The mirror’s tilt angles (α and β) correspond to horizontal and vertical directions, respectively. This relationship is further developed in the following equations [126]:

$${\left\{\mathrm{I}\right\}}_{\widehat{\mathrm{s}}}={\left\{\mathrm{I}\right\}}_{\widehat{\mathrm{i}}}-\left(\mathsf{\gamma }(\mathsf{\alpha },\mathsf{\beta },\mathsf{\phi }\right)-\sqrt{{\left(\mathsf{\gamma }(\mathsf{\alpha },\mathsf{\beta },\mathsf{\phi }\right)}^2})·{\left\{\mathrm{I}\right\}}_{\widehat{\mathrm{n}}},$$

(44)

$${\left\{\mathrm{I}\right\}}_{\widehat{\mathrm{n}}}=\left[\begin{array}{c}\mathrm{s}\mathrm{i}\mathrm{n}\mathsf{\alpha }·\mathrm{c}\mathrm{o}\mathrm{s}\mathsf{\beta }\\ \mathrm{c}\mathrm{o}\mathrm{s}\mathsf{\phi }·\mathrm{s}\mathrm{i}\mathrm{n}\mathsf{\beta }+\mathrm{s}\mathrm{i}\mathrm{n}\mathsf{\phi }·\mathrm{c}\mathrm{o}\mathrm{s}\mathsf{\alpha }·\mathrm{c}\mathrm{o}\mathrm{s}\mathsf{\beta }\\ \mathrm{s}\mathrm{i}\mathrm{n}\mathsf{\phi }·\mathrm{s}\mathrm{i}\mathrm{n}\mathsf{\beta }-\mathrm{c}\mathrm{o}\mathrm{s}\mathsf{\phi }·\mathrm{c}\mathrm{o}\mathrm{s}\mathsf{\alpha }·\mathrm{c}\mathrm{o}\mathrm{s}\mathsf{\beta }\end{array}\right],$$

(45)

$$\mathsf{\gamma }\left(\mathsf{\alpha },\mathsf{\beta },\mathsf{\phi }\right)={\left\{\mathrm{I}\right\}}_{\widehat{\mathrm{n}3}}=\mathrm{s}\mathrm{i}\mathrm{n}\mathsf{\phi }·\mathrm{s}\mathrm{i}\mathrm{n}\mathsf{\beta }-\mathrm{c}\mathrm{o}\mathrm{s}\mathsf{\phi }·\mathrm{c}\mathrm{o}\mathrm{s}\mathsf{\alpha }·\mathrm{c}\mathrm{o}\mathrm{s}\mathsf{\beta }.$$

(46)

In the laser source reference system {I}, γ represents the third component of the normal vector $\widehat{n}$, whereas α and β represent the scanning tilt angles, direction, and mirror surface. Each point in the cloud is described by its (x, y, z) coordinates, which are computed from the measured distance, the horizontal angle (determined by the revolving mirror or prism), and the vertical angle (typically fixed but variable in multi-emitter systems).

For the CarMaker virtual vehicle model, lidar 2D and 3D sensors are defined as lidar RSI sensors that provide information about nearby objects and behave just like genuine sensors. The calibration of the lidar virtual sensor was performed out using simulation data from various surroundings that was correlated with the motion sensor [127].

Lidar 2D sensors are used in pairs of two units to reduce coverage gaps and ensure continuous visibility throughout the entire surface of a horizontal scanning plane (P2D) with a viewing angle of 180° and a field of view centered on the virtual vehicle model's median longitudinal plane. Lidar 3D sensors scan on 16 channels, following a planes (P3D) with successive inclinations along an increasing angular axis with respect to the upper horizontal scanning plane. They have a viewing angle of 30° and an opening of ± 15° with respect to the median longitudinal plane of the virtual vehicle model (Figure 28).

The interaction modes of the lidar RSI sensor are classified using the following criteria [65]:

• Diffuse reflected laser rays are distributed uniformly, regardless of the direction of the incident ray (Lambertian reflection), with the intensity of the reflected ray decreased by the inverse of the angle between the incident ray and the normal of the reflective surface;
• Retroreflective means that incident laser rays are reflected back in the direction of the emitter, with the intensity of the reflected ray being reduced based on the reflectance parameters and incident angle;
• Specular, the incident and reflected laser rays form identical angles with the normal of the reflective surface, and the incident and reflected laser rays are in the identical plane;
• Transmissive: the incident laser photons keep their course but are attenuated by transmission.

3.3.2. Ultrasonic RSI Sensor

CarMaker's ultrasonic RSI model is based on the signal chain of a real ultrasonic sensor. The process begins with a transmitter illuminating the environment and emitting sound waves (Figure 29). This method divides sound waves into a finite number of rays. This provides for a balance between simulation speed and physical accuracy. Environmental circumstances with physical effects on wave propagation are considered into account, and the Helmholtz equation is used to compute electrical scatter fields on object surfaces utilizing parameterizable material properties to predict reflections. Every suitable ray returns to the receiver [128].

For all detections, the sensor model returns the sound pressure level as well as the flight time. It accounts for dense packaging and associated effects by optionally sensing cross-echoes between sensors. Interfaces enable the replacement of individual steps with user-defined code, and clustering on available GPUs improves performance [128], as seen in Figure 30.

The ultrasonic RSI sensor uses mechanical acoustic pressure waves that are reflected by obstacles in the immediate vicinity of the virtual vehicle model using the ToF principle, and the distance to the respective obstacles is accurately calculated using the SPA (Sound Pressure Amplitude) (Figure 31). The sensor considers the impacts of overlapping objects in the area, the effects of physical propagation, and the classification as false positives or false negatives of observed objects. The propagation modes of acoustic pressure waves are classified using the following criteria [65]:

• Direct Echo: the acoustic pressure wave is reflected once by an object in close proximity and received by the emitting sensor;
• Indirect Echo: the acoustic pressure wave is reflected at least twice by objects or surfaces in the vicinity and received by the emitting sensor;
• Repeated Echo: the emitting sensor receives the acoustic pressure wave after it has been reflected by nearby objects or surfaces;
• Cross Echo: the reflected acoustic pressure wave is received by a sensor other than the originating sensor, resulting in a propagation mode known as cross echo;
• Road Clutter occurs when the acoustic pressure wave is reflected by bumps or irregularities in the roadway.

Ultrasonic RSI sensors are installed in the vehicle's front bumper, side panels, and rear bumper.

4. Virtual Sensor Parametrization

4.1. Ideal Sensor Parametrization

The Ideals sensors incorporated in the virtual vehicle model are as follows: Slip Angle, Inertial, Object, Free Space, Traffic Sign, Line, Road and Object by Line.

To highlight the major parameters generated by the indicated Ideal sensors, a series of simulations have been performed on the proposed route (see Figure 9), digitized, and implemented in the IPGRoad/CarMaker application.

For the slip angle sensor (Figure 32a), one parameter has been chosen: Sensor.SAngle.SL00.Ang (Figure 32b). The analysis was performed on an area of the route traveled, starting with a straight road section, then a left turn and another straight road section.

The graphic illustrates how the slip angle sensor's values change when the vehicle approaches a left curve (second 7) and returns to a straight road segment (second 25). According to the data displayed on the graph in the image, the maximum slip angle is around 0.25 degrees.

For the inertial sensor (Figure 33a), two parameters have been chosen: Sensor.Inertial.YRS.Vel_0.x (red curve) for the speed in the x direction and Sensor.Inertial.YRS.Acc_B.y (blue curve) for the lateral acceleration (Figure 33b).

The graphic shows that lateral acceleration increases from 0 m/s² when moving straight to 5.3 m/s² when turning. The vehicle's lateral acceleration reduces as it exits the turn, returning to near-zero m/s² levels. The sensor's observed lateral accelerations correspond with typical turns.

For the object sensor (Figure 34a), two parameters have been chosen: Sensor.Object.RadarL.relvTgt.NearPnt.ds.x (red curve) represents the distance (ds) along the specified x-axis, corresponding to the direction of the sensor system axes, from the most relevant target (relvTgt), respectively the detected object that is most important to track Sensor.Object.RadarL.relvTgt.NearPnt.ds.y (blue curve) represents the distance (ds) along a specific y-axis parallel to the sensor system axes and the detected object's nearest point (NearPnt) (Figure 34b). This value represents the shortest distance between the sensor and the relevant target (a point on the target), which could be a vehicle, a pedestrian etc.

The graphic shows the variations in x and y coordinates over time for the nearest point on an obstacle detected by the sensor. The coordinates corresponding to the z axis (height) have not been shown since they contain relatively minor fluctuations that have no significant effect on the position of the obstacle.

For the free space sensor (Figure 35a), two parameters have been chosen: Sensor.FSpace.Front.Segm.0.ds.x (red curve), and Sensor.FSpace.Front.Segm.198.ds.x (blue curve) that represents the segmentation areas, includes the closest point of a detected traffic obstacle. The segmentation area is divided into four quadrants. Figure 35b shows how the various parameters change along the x-axis, with each parameter representing a quadrant of the segmentation area. The graphs illustrate the change in expected parameter values across the segmentation region over the same time frame. Analyzing the graph which corresponds to the parameter Sensor.FSpace.Front.Segm.198.ds.x, one can see no more variations of the specified parameter after second 52, indicating that no obstacle was identified in the associated quadrant.

For the traffic sign sensor (Figure 36a), two parameters have been chosen: Sensor.TSign.FrontCam.SpeedLimit.0.ds.x (blue curve) indicates the distance between the detected indicator (Speed Limit sign) and the direction of travel (x), respectively Sensor.TSign.FrontCam.SpeedLimit.nSign (red curve) indicates the signal generated by the sensor when it detects a speed limitation sign (Figure 36b). When the distance between the vehicle (sensor) and the road sign is less than 30 meters, the signal reaches its maximum value.

For the line sensor (Figure 37a), one parameter has been chosen: Sensor.Line.Front.RLines.1.Type (Figure 37b) indicates the type of longitudinal road marking located on the right side of the road in the virtual vehicle model's direction of travel. The graph shows a variation in the signal of the parameter in the range of 1 to 2. The parameter's value 1 indicates the presence of a longitudinal road marking made up of a simple dashed line, while the parameter's value 2 indicates the presence of a longitudinal road marking made up of a single continuous line.

Changing the parameter value from 1 to 2 moves the virtual vehicle model from a road section with a simple dashed line (1) to a road section with a continuous line (2). If the parameter value is constant (1 or 2), the type of longitudinal road marking does not differ; if the values fluctuate frequently a road section with many changes in longitudinal road marking will result.

Two parameters have been chosen for the Road sensor (Figure 38a): Sensor.Road.AlongR.Path.tx (blue curve) represents the vehicle displacement along the x-axis, and Sensor.Road.AlongR.Path.DevDist (red curve) represents the deviation from the planned route (Figure 38b). Over a route length of around 8 m, the deviation ranges between 0 and 7*10^-9 m.

For the object by line sensor (Figure 39a), one parameter has been chosen: Sensor.ObjByLine.OBL00.LinesC.0.ObjF.0.sMax (Figure 39b), that highlights the variation in maximum distance between the virtual vehicle model and the POI over a 2.5-second period. Figure 22 shows that S_max decreases over time, indicating that the vehicle has approached the POI, where LinesC indicates the LaneScope Center section.

4.2. Hi-Fi Sensor Parametrization

The Hi-Fi sensors incorporated in the virtual vehicle model are as follows: (a) Camera, (b) Global Navigation, (c) Radar.

To highlight the major parameters generated by the indicated Ideal sensors, a series of simulations have been performed on the proposed route (see Figure 9), digitized, and implemented in the IPGRoad/CarMaker application.

For the camera sensor (Figure 40a), two parameters have been chosen: Sensor.Camera.CA00.Obj.0.nVisPixels (x-axis) indicates the number of visible pixels of the object recognized by the sensor and Sensor.Camera.CA00.Obj.0.Confidence (y-axis) indicates the confidence degree of object detection (Figure 40b). The graph illustrates the efficacy of object detection based on their size in the image. If an object has many visible pixels but a low confidence level, the recognition algorithm becomes unreliable. The maximum confidence level for recognized items is 1.

For the global navigation sensor (Figure 41a), one parameter has been chosen Sensor.GNav.Receiver.SatNoDirectView that highlights the number of satellites that are directly visible to the GNSS receiver in a specific time interval (Figure 41b).

The satellites' direct visibility influences measurement accuracy. The maximum number of visible satellites ranges from 4 to 10 and is influenced by tall buildings or underground passages that the vehicle passes through, and by extreme weather conditions.

For the radar sensor (Figure 42a), two parameters have been chosen: Sensor.Radar.RA00.Obj0.Dist (blue curve) represents the variation of the signal-to-noise ratio (SNR) as a function of the distance from the object and Sensor.Radar.RA00.Obj0.SNR represents the signal to noise ratio (Figure 42b). The graph illustrates the radar signal's detection efficiency versus background noise as a function of distance to the target object. A higher signal-to-noise ratio suggests better detection. As the object moves away, the signal-to-noise ratio decreases, resulting in poorer detection.

Another parameter evaluated for the radar sensor is Sensor.Radar.RA00.Obj0.RCS (red curve) represents a variation of the RCS parameter (target object's radar signature) in the case of detection (Figure 42c). The graph shows significant changes in RCS parameter values, ranging from -18dBm² to 21dBm². The analysis of the RCS parameter indicates that the radar sensor detects obstacles in the virtual vehicle model's travel environment, ranging from small (negative values for poor detection) to large (positive values for good detection).

4.3. RSI Sensor Parametrization

The RSI sensors incorporated in the virtual vehicle model are as follows: (a) Lidar RSI, (b) Ultrasonic RSI.

To highlight the major parameters generated by the indicated Ideal sensors, a series of simulations have been performed on the proposed route (Figure 9), digitized, and implemented in the IPGRoad/CarMaker application.

For the lidar RSI sensor (Figure 43a), one parameter was chosen: Sensor.LidarRSI.LIRS00.nScanPoints indicates the number of points detected in an interval of time (Figure 43b). The graph's parameter variations allow for an investigation of scanning conditions in particular areas of the virtual vehicle model's travel path. A high parameter value results in a congested area with many detected obstacles, whereas a low parameter value results in an area with fewer detected obstacles.

The ultrasonic RSI sensor (Figure 44a) has two parameters: Sensor.USonicRSI.Tx.USRS00.Rx.USRS00.nDetections (blue curve) and Sensor.USonicRSI.Tx.USRS01.Rx.USRS01.nDetections (red curve) represent the variation of obstacles detected by two ultrasonic sensors placed in different positions on the virtual vehicle model (Figure 44b). The graph illustrates how the ultrasonic sensors under consideration detect obstacles. If the variation curves for the two parameters overlap (blue/red), the considered ultrasonic sensors detect the same number of obstacles in a specific time interval, indicating the detection of a large object or a uniform surface.

If an ultrasonic sensor registers an increase in detections, it indicates that an obstacle has been recognized in the neighborhood; if the ultrasonic sensor detects nothing, it indicates that there may be blind zones around the virtual vehicle model.

5. Discussion

Replicating the behavior of real-world sensors in a virtual environment is a complex task. This involves capturing elements such as sensor noise, limitations in the field of view, uncertainty in physical parameters, and responses to varying environmental conditions. This can be achieved through advanced data-driven algorithms, AI-based simulation techniques, and innovative virtual sensor designs. Autonomous vehicles utilize a variety of sensors (cameras, radar, lidar, etc.), and combining data from multiple virtual and real sensors through sensor fusion techniques, each with its characteristics and potential errors, is a significant challenge [129]. Their performance can degrade due to changes in nonlinear dynamics, complex physical processes in the environment, and nonlinear interactions between input and output variables [130,131]. High-fidelity sensors can be computationally intensive to simulate real-world sensor data with high precision. This requires robust hardware and efficient algorithms to process the data in real-time [132]. Virtual sensors must operate in real time within the simulation environment, meaning they must generate and process data quickly enough to keep up with the simulation. This can be computationally demanding [133]. AI techniques enable the development of virtual sensors that can estimate physical parameters without traditional sensors, using data-driven models to emulate real-world conditions [10]. The integration of AI introduces potential uncertainties, necessitating validation to maintain trust in these systems [133].

Validating virtual sensor data against real-world data or physical sensor measurements can be difficult. Autonomous vehicles must handle unusual situations and realistically simulating these circumstances adds another layer of complexity. Virtual sensors in the overall software architecture of autonomous vehicles can be complicated, mainly if the architecture was not designed initially with virtual sensors in mind. The development and validation of virtual sensors can be time-consuming and costly, especially when dealing with complex sensor models or high-accuracy requirements.

6. Conclusions

Future studies could focus on improving the accuracy and robustness of virtual sensor models, particularly in complex and dynamic environments. This can address challenges related to nonlinear dynamics models with complex physical processes, research on environmental factors (such as weather conditions, temperature, and lighting), and detailed real-world behavioral models. Research into advanced AI and ML techniques for sensor fusion and data processing is also essential. Implementing these advancements is likely to result in more accurate and reliable data, thereby enhancing the overall performance of AV driving systems. Additionally, using DL architectures designed explicitly for particular tasks (like object detection or lane-keeping) would be highly beneficial. Efforts should also concentrate on developing robust data-driven system models capable of accurately identifying various physical characteristics. Such models can provide a viable and cost-effective alternative to traditional real sensors. Research should include testing virtual sensor performance in rare or unexpected situations, often referred to as edge cases. This requires testing and validating across a wide range of scenarios.

Validating and calibrating virtual sensors is also important and may involve establishing standardized testing procedures and metrics. Establishing industry standards for the development and validation would promote interoperability and facilitate the widespread adoption of this technology. It is also crucial to optimize algorithms and ensure that hardware can meet the computational demands of high-fidelity sensor interfaces, especially in complex scenarios. Exploring hybrid approaches that combine both virtual sensor data and real sensor data could leverage the strengths of each type, thus enhancing overall accuracy and reliability. Also, it is important to examine the necessary levels of simulation fidelity for various autonomous driving tasks to balance the need for accuracy with computational efficiency. Studying how virtual sensor data is presented to human operators in semi-autonomous systems can also be an important step, as it impacts their situational awareness and decision-making processes.