Optimized Right-Turn Pedestrian Collision Avoidance System Using Intersection LiDAR

1. Introduction

In today’s road traffic, intersections and their associated traffic systems are very complex. As intersections are a complex environment with many factors, vehicles and pedestrians in the intersection need to be more careful about their safety. However, as is often reported in the media, right-turning pedestrian accidents are one of the most common types of accidents in Korea. In response, the Korea Road Traffic Authority (KRTA) investigated right-turn accidents at all intersections over the past three years, and the results are shown in Figure 1 below.

In Figure 1, we can see the percentage of total accidents and right-turn accidents by vehicle type. As a result, we can note that the ratio of right-turn accidents to total pedestrian accidents is overwhelmingly higher for larger vehicles than for smaller passenger cars. Recognizing the seriousness of right-turn accidents, South Korea implemented the ‘Right Turn Pause’ law in January 2023. This law requires all vehicles to pause once when turning right to protect pedestrians. Since the law came into effect, all police forces in the country have been enforcing right-turn pauses, and local governments have been promoting the law, but even today, South Korea still suffers from right-turn accidents involving large vehicles, and the cause has been found to be the driver’s limited field of vision.

On the other hand, today’s autonomous vehicle technology has come a long way from the past [1,2,3]. It has been variously suggested that high-precision sensors and recognition systems can replace human object recognition intelligence [4,5,6], but it is still difficult to solve this problem technologically if the vehicle’s field of view is insufficient. For example, Figure 2 below is one of the common road environments in Korea. A vehicle planning to make a right turn is very likely to have insufficient obstacle information recognition due to the presence of equipment on the curbstone or street trees.

To solve this problem of insufficient perception information, many researchers have investigated various strategies. Cooperative perception is a method in which the sensor information of multiple agents is shared and complemented by each other to improve the cognitive performance of a single agent[7,8,9]. Agents can be not only vehicles, but also mobile robots and urban infrastructure systems, suggesting that sensors and systems designed for each environment can compensate for the limited field of view of a robot or vehicle. In particular, [10,11,12] propose various sensor fusion methods and 3D point cloud-based generation methods for collaborative perception among connected autonomous vehicles. These papers mainly propose solutions to the data occlusion phenomenon by fusing the features of point cloud data between different frames. However, these methods need further technical implementation and safety verification to be applied safely in real world, and it is not reliable that they can prevent right turn accidents when the vehicle is in a dynamic state. In [13,14], a cooperative recognition system is proposed using the infrastructure in urban areas. The implementation of the recognition system by attaching additional sensors to the infrastructure is one of the key components of the cooperative recognition system. This approach overcomes the limitations of existing recognition functions performed by individual vehicles or devices alone and enables more accurate and reliable recognition by cooperating with the infrastructure. However, most of the infrastructure facilities proposed in the literature are designed to target only a specific area, which limits their ability to cover the entire area within an intersection.

There are also studies related to right-turning pedestrian accidents in South Korea, such as [15]. In [15], the authors present criteria that can be legally reviewed to ensure that autonomous vehicles can safely make right turns. This study redesigned the right-turning process based on several case studies and presented an algorithm in the form of a flowchart. In [16], an algorithm is proposed to detect pedestrians by installing a LiDAR sensor in the middle of an intersection. This paper proposes a 128-channel LiDAR sensor for pedestrian detection, but 128-channel LiDAR is generally classified as a high-cost sensor, which is very uneconomical in terms of equipment. Therefore, this paper proposes a more practical right turn prevention system by compensating for these shortcomings, and verifies it in a virtual environment.

The primary contributions of this paper are as follows:

• Optimization of a right-turning pedestrian detection system: The use of 64-channel LiDAR is sufficient to satisfy the requirements of the proposed system and demonstrates that the use of 64-channel LiDAR is reasonable in terms of system efficiency and cost. These results show that the system can maximize the efficiency of the sensor configuration while maintaining high performance recognition capability.
• Cost Effectiveness: The study shows that pedestrian detection systems in intersection can complement the perceptual capabilities of existing autonomous vehicles and reduce the cost of additional sensors on the vehicle.
• Right turn scenario configuration and dedicated dataset: As there are not enough datasets for right turn scenarios compared to the existing studies, we constructed a dedicated dataset based on various right turn scenarios in the simulator separately.
• Robustness in Various Weather Conditions: For pedestrian detection, the robustness of the pedestrian detection system was enhanced by using only LiDAR sensors that are less sensitive to weather changes, as opposed to other sensors that are sensitive to weather changes.

2. System Architecture

The overall system proposed in this study is shown in Figure 3. A LiDAR sensor is placed near the middle of the intersection to observe pedestrians on the crosswalk. A bus is chosen as the experimental vehicle for this purpose because, unlike a regular passenger car, a vehicle with a high height has a limited field of view. It is assumed that this vehicle is capable of autonomous driving, so we will install a LiDAR on top of the bus vehicle to obtain vehicle-centric perception data at the same time. In order to solve the problem presented in this study, it is necessary to recognize pedestrian information from sensors at the intersection and transmit pedestrian information to the bus vehicle before an accident occurs. Therefore, we propose that a control tower is required to apply this system in the real world. However, in this experiment, we replace the control tower with the ROS platform server in a virtual simulator environment.

The pedestrian detection algorithm used in this paper is shown in Figure 4. First, we chose Complex-YOLO as the pedestrian detection deep learning model because unlike other deep learning algorithms, it is appropriate to use a one-stage detection model with fewer constraints on the inference speed [17]. Despite the primary classification of pedestrian classes from the deep learning model, further work needs to be done, and the reasons are as follows. Unlike vehicles and structures, pedestrians generate data from LiDAR that is regularly hard to identify (Figure 5), which can sometimes reduce the accuracy of Complex-YOLO. Therefore, filtering and clustering of regions of interest designated as dangerous areas are performed to determine the presence or absence of objects. Afterwards, the existing results are combined to increase the stability of pedestrian detection. To compare the performance of pedestrian detection using this approach, we used PV-RCNN as the pedestrian detection algorithm for onboard LiDAR [18]. It is important to note that PV-RCNN outperforms Complex-YOLO in terms of accuracy, although it is weaker in inference real-time, so it is used in this experiment in terms of accuracy comparison with Complex-YOLO.

3. Materials

3.1. Simulator Selection

We used a simulation environment for the purpose of this study. Today, there is a wide variety of simulator software for autonomous driving and road traffic simulation. However, we chose the CARLA simulator for the following reasons [19,20].

• Easy to access

The CARLA simulator is free and open software. Because it is open source, the simulator is continuously used by many users around the world and is well maintained for any functional errors or bugs. We found the CARLA simulator to be a good choice for this study, given its cost and available features.

• Integration with the ROS platform

The CARLA simulator is a simulator that ensures compatibility with multiple robot platforms. Particularly, it is very convenient to integrate with the Robot Operation System (ROS), and many users have conducted research on integrating ROS and CARLA [21,22,23]. ROS is a robot operating system and an open-source framework for robot software development [24]. It has the advantage of offering various tools and libraries needed to develop robot applications, and it is very well connected with the CARLA simulator. In this study, we chose ROS because we needed to simultaneously create and control multiple agents on a map to collect data. At the same time, we investigated several simulators aimed at autonomous vehicle research, and based on [25], we compared the ROS platform with CARLA and other simulators and determined that the CARLA simulator was the most suitable simulator for this experiment. The Figure 6 below shows the actual interworking of ROS and CARLA simulator, and it can be seen that the data of the LiDAR installed on the vehicle is output normally by the ROS-rviz tool.

3.2. Simulator Setting

The CARLA simulator supports agent assets that can be controlled by the user by default, and the properties contained in the assets are managed by a library of blueprints. In addition, all properties that can be controlled in maps also have blueprints, and information about the various features can be found in [26]. We used these to set up our experimental environment as follows.

①

Experiment Configuration

The CARLA simulator is a program based on the Unreal graphic engine. To run this simulator, a high-end PC environment is usually required. Therefore, to reduce the load on the PC consumption, we configured the simulation environment to be divided into two PCs as shown in Figure 7 above.

The role of each PC is as follows First, we set up the CARLA simulator server to run on PC1, which is equipped with a high-end GPU. We also set up PC1 to create and control pedestrian blueprints and weather blueprints to design scenarios over the maps loaded on the server. On the other hand, we set up a network environment for the client to run on a separate PC2, and we associated an onboard agent with the sensorset json file on the target vehicle and an intersection agent with the sensorset json file in the center of the intersection, respectively (Table 1 and Table 2).

In order to integrate CARLA with ROS, you must use a separately distributed package, and all data generated on CARLA can be accessed through the ROS interface.[27] We experimented by connecting each client to the CARLA server through the package, and when connecting to the server, you must specify the IP address of the host server and run the launch file.

②

Map

For this study, we chose the largest intersection in Town 10 as the experimental area to simulate accidents that occur within the urban area (Figure 8).

③

Onboard Sensor

For this study, we utilized a Mitsubishi-Fusorosa model bus vehicle asset (Figure 9) and specified a 32-channel LiDAR sensor to be attached to it. The data showed the following limitations of the onboard LiDAR sensor. First, the vehicle’s height is higher than that of a normal vehicle, which inevitably leads to blind spots, as shown in Figure 10. Second, the 32-channel LiDAR has a smaller resolution compared to other high-channel LiDARs, so the data sparsity is strongly affected by distance (Figure 11). As a result, we can see that the same problems with LiDAR that we encounter in the real world can also be applied in the simulator

④

Intersection Sensor

The perception sensors that can be utilized within the CARLA simulator are camera, LiDAR, and Radar sensors, and the output of each sensor can be seen as ROS message data. We installed a 64-channel LiDAR sensor at the intersection as a sensor for pedestrian detection and did not consider installing other sensors other than LiDAR for the following reasons. First, CARLA can handle extreme weather conditions, such as heavy fog or rain. For example, as shown in Figure 12a, we can see that the camera data does not produce a good image of the pedestrian in the fog, even though there are two pedestrians on the road, and the radar data also cannot output pedestrian data in the fog (Figure 12b). However, even in the fog, the LiDAR sensor can capture the person on the crosswalk unlike other sensors, and the ROS message data is obtained properly (Figure 12c). Second, unlike cameras and radar sensors, LiDAR sensors have a 360-degree field of view. The unique advantage of LiDAR sensors is that they can cover the entire area, not just one area within the range. Therefore, in addition to the proposed experimental area in this paper, the rest of the Mainstreet and the pedestrian crossing area can be observed simultaneously, which suggests that the LiDAR sensor is a very universal sensor compared to other sensors. Therefore, in this paper, LiDAR is selected as the main sensor for pedestrian identification for these reasons.

However, instead of the approach proposed in [16], we used a 64-channel LiDAR, which is lower than 128 channels. Table 3 below compares the specifications of 64-channel LiDAR and 128-channel LiDAR, and Figure 13 captures one frame of 128-channel LiDAR and one frame of 64-channel LiDAR. From the screen, we can see that the shape of the pedestrian on the crosswalk is displayed correctly.

⑤

Weather Condition

CARLA also supports blueprints for weather, so you can create any weather situation whatever you want except for snow conditions. Table 4 below shows representative parameters for each weather, and we created sunny_day, clear_night, fog_day, fog_night, rain_day, and rain_night, respectively, and used them in our experiments, and the results can be seen in Figure 14 below.

4. Experiment

4.1. Dataset

Since many existing datasets do not fit the purpose of this study, we designed a dedicated dataset for this experiment and prepared the dataset in the following order.

①

Pedestrian Setting

There are three types of pedestrian blueprints in the CARLA simulator: normal adult, overweight adult, and children (Figure 15), and CARLA supports each type of pedestrian with a different blueprint ID number, so you can create or control a pedestrian of that type at any time. This can be seen in Table 5 below.

②

Scenario Configuration

In this study, a right-turning accident scenario was designed to collect data. The target area is the Mainstreet area within town10, and various situations were created using pedestrian type, pedestrian speed, and number of pedestrians as detailed variables. The speed of the target vehicle was manually controlled at 15-25km/h as shown in Figure 16 below, and 18 scenarios were designed by applying various weather conditions during driving (Table 6). Through this scenario design, we were able to generate a dataset that more closely resembles and accurately reflects real-world right-turn accident situations.

• Data Time Synchronization

For our proposed method to prove correctly, the onboard agent and the intersection agent must be simulated at the same time and within the same period, which can be adjusted through a parameter named passive in CARLA. This parameter is specified as a bool type, and we were able to synchronize the time of the ROS Topic data through this parameter. In Figure 17 below, you can see how the time synchronization is achieved through the True/False of the passive parameter, and in this way we collected about 40 Gb of ROS bag files.

④

Data Extraction

To compare the pedestrian detection performance between the two agents, we extracted LiDAR bin files for onboard and intersection from the ROS bag files and label txt files containing pedestrian information. The ROS-ApproximateTimeSynchronizer package was utilized for extraction, and the label files stored label data in the order of x, y, z, dx, dy, dz, heading, and class type for 3D LiDAR detection training. As a result, the number of extracted data for each scenario is shown in Table 7 below.

4.2. Training Details

We trained the PV-RCNN algorithm on the onboard LiDAR data and the Complex-YOLO algorithm on the intersection LiDAR data from the extracted data. The GPU specification of the PC used as a training environment is NVIDIA GEFORCE-RTX-3090ti, and the parameter properties of each algorithm set for training are shown in Table 8 below. We utilized 70% of the total dataset as training data, and the training data was randomly mixed with scenario data of pedestrian type and weather type.

4.3. Clustering Details

We used Complex-YOLO for pedestrian detection and clustering at the same time. Since our experiment targets pedestrians on crosswalks, we first performed filtering by specifying a region of interest. By filtering the region of interest, the obstacles on the crosswalk are more visible. Then we applied the DBSCAN clustering method to cluster the objects on the crosswalk. The DBSCAN method is a density-based clustering algorithm that is robust to noise and performs well in clustering objects in the target area. You can see the sequential clustering process in Figure 18 below.

5. Experimental Result

We evaluated the performance of the algorithm proposed in this paper with about 30% of the data from our own dataset. At this time, we prepared evaluation datasets for each type of pedestrian according to the designed scenarios (Table 9) so that they can be evaluated in various environments, and the number of datasets prepared for each scenario can be found in Table 10 below..

5.1. Onboard Evaluation

5.1.1. Qualitative Evaluation

The PV-RCNN qualitative evaluation of Onboard’s data is shown below. As shown in Figure 19, it can be seen that pedestrians located within the blind spot of the bus are not detected properly. In addition, pedestrians located outside of a certain range are also poorly detected. From these results, we can conclude that large vehicles need to perform safe recognition functions for autonomous driving, and the larger the volume, the wider the blind spot tends to be. This shows not only the limitations of deep learning but also the need for additional sensors to compensate for blind spot, but we can conclude that installing additional sensors may not be economically reasonable.

5.1.2. Quantitative Evaluation

The PV-RCNN quantitative evaluation of Onboard’s data is shown below. As can be seen from Table 11, the results show low accuracy due to the failure to detect pedestrians within the blind spot and pedestrians beyond a certain distance. In particular, the number of LiDAR data that constituted pedestrians beyond a certain distance was too few to infer the model. (Figure 20) Therefore, it can be concluded that the 32-channel LiDAR in the simulator is not reliable for vehicle safety, and the lack of perception data in autonomous vehicles necessarily raises the need for external sensor involvement.

5.2. Intersection Evaluation

5.2.1. Qualitative Evaluation

The qualitative results of Complex-YOLO evaluated on our dataset were as follows. We found that sometimes two pedestrians were not detected even though they were on the crosswalk. The reason for this is that the number of LiDAR data corresponding to pedestrians is less than 128-channel LiDAR data, so BEV features are not fully reflected in the training. In particular, we found that child pedestrians, as shown in Figure 21, have significantly fewer points than adult pedestrians, so the general deep learning method sometimes fails to detect them. Figure 22 shows the examples of correctly detected and incorrect results when compared to the ground truth in each case (Table 9).

Figure 23 below is an example of the clustering results. As you can see in the result figures, after performing the clustering operation, we reconstructed the boxes in a 2D Bird Eye View (BEV) view, and we can clearly see that even objects that were not detected by Complex-YOLO were correctly detected.

Figure 24 below shows the final output of the fusion of Complex-YOLO and clustering results. The fusion of the bounding box from Complex-YOLO and the bounding box from clustering complements the results of Complex-YOLO. As a result, we can note that the current location of the pedestrian can be determined based on the Intersection LiDAR.

5.2.2. Quantitative Evaluation

The following is a quantitative evaluation of the Intersection LiDAR process. Table 12 shows the quantitative results of Complex-YOLO and our proposed model for each scenario. We can see that most of the results of Complex-YOLO detect pedestrians at a certain accuracy level, but when we fused the clustering results, we can see that the accuracy results are mostly above 0.9. This shows that our model is better at detecting pedestrians when fused with the clustering process, suggesting that LiDAR is better than other sensors at detecting pedestrians in a simulated environment. Furthermore, the performance is not significantly different from the previously proposed 128-channel LiDAR experiment, suggesting that the 64-channel LiDAR in our system is sufficient and can be replaced.(Table 13)

6. Conclusion And Discussion

Through this experiment, we found that the main cause of right-turning pedestrian accidents is the limited field of view, which is a problem that can also occur in large scale autonomous vehicles. Therefore, we proposed a solution for cooperative recognition with vehicles through simulation of pedestrian detection experiments using LiDAR installed at intersections, and confirmed that pedestrian detection performance using 64-channel LiDAR is excellent. However, this experiment was only verified in a virtual simulation, and there are limitations to implementing it in the real world, such as the following.

①

Sensitivity to weather: The LiDAR sensor used in this experiment is considered to be less sensitive to noise in different weather conditions because it produces idealized data in virtual environment. However, in the real world, LiDAR sensors are very sensitive to extreme external environments, making data collection difficult [28]. Therefore, in future research, we will install LiDAR in an urban environment in a real testbed (Figure 25) and collect and analyze LiDAR data directly as the weather changes.

②

Design of sensor-mounted equipment and wireless data transmission: For the system to work in the real world, a process is required to transmit pedestrian data from the sensors to the vehicle without loss. In this study, the data transmission was performed by replacing the ROS server, but in the real world, we plan to design a separate communicable hardware to further design the data transmission and reception process and verify the safety. In addition, the role of sensor-mounted equipment is very important in this system, and many studies have been conducted in this regard [29,30,31]. Based on this, we plan to implement this system by designing a device that satisfies the Korean road traffic law.