SaBi3d - A LiDAR Point Cloud Data Set of Car-to-Bicyle Overtaking Maneuvers

1. Summary

Cycling has multiple advantages over other modes of transport, such as reducing emissions in city centers [1] and promoting a healthy lifestyle [2]. Therefore the European Union, and some of its member states like Austria, have declared the goal of increasing the bicycle modal share [3,4]. However, cycling is one of the most dangerous modes of transport [5]. Besides the positive effects of cycling, it also faces a higher risk of fatalities compared to other transport modes, especially in case of inadequate cycling infrastructure [6].

Longitudinal overtaking maneuvers by motorized vehicles contribute most to a feeling of subjective risk, thus presenting a barrier to cycling participation [7], and are likely to lead to severe injury in the event of an accident [8]. Therefore, many studies have tried to capture characteristics of overtaking maneuvers in recent years. Most of these capture limited data such as the minimum lateral clearance and overtaking velocity at a single point [9,10,11], providing only a snapshot of an overtaking maneuver without capturing data representing the whole maneuver, which is a complex process. On the other hand, works relying on more detailed data, such as LiDAR point clouds, often require manual annotation, which is time-consuming and expensive [12,13]. A third approach is to use test tracks and equipment with extensive sensor coverage [14,15]. However, naturalistic studies are often preferable to get results representative for real world interactions. Therefore, it would be desirable to have a system that records point cloud data from an instrumented bicycle, automatically detecting vehicles in the data and extracting rich metrics. This would allow for a very detailed description of overtaking maneuvers in naturalistic environments without the need for further manual annotation and thereby remove the drawbacks mentioned above.

To this end, we present a new data set, the Salzburg Bicycle 3d (SaBi3d) data set, which is a first step towards bridging this gap. It consists of 59 overtaking maneuvers manually annotated in a total of 4811 point clouds collected by a LiDAR-equipped bicycle. This data set can be seamlessly substituted for the nuScenes [16] data set for object detection tasks without any additional adaptations.

2. Materials and Methods

As to the best of the authors knowledge no LiDAR data sets collected by a smart bicycle exist, this materials chapter includes a short review of the related research area. This way, necessary context for the understanding and usage of this data set is provided. Subsequently, the data collection using a LiDAR-equipped bicycle and the data preparation are discussed followed by a description of the metadata. Last, the creation of the object detection benchmark results is outlined.

2.1. Related Work

There are two research areas related to this work: First, cycling research that uses LiDAR, and more generally laser based distance measurements. LiDAR scanners only recently became small, light, and cheap enough to be mounted on bicycles, yielding a limited number of works in this area. The second area are LiDAR based traffic data sets collected by LiDARs mounted to a moving platform, a motorized vehicles for all regarded works. This is a wide area, therefore only the most relevant works are listed. Last in this section, a short introduction to the 3d object detection system used for the benchmark results and its surrounding research will be provided.

2.1.1. Lasers and LiDARs on Bicycles

There are multiple recent works on using single beam laser range finders on bicycles. Most often they are used for measuring lateral clearance during overtaking maneuvers [7,9,10,11,15]. Another idea is to control a single laser based distance sensor to search for and track single cars. This approach has been developed and extensively investigated in a series of works by Jeon Woonsung and Rajesh Rajamani [17,18,19,20,21]. This lead to a LiDAR based collision warning system on a bicycle [22,23], an approach also pursued by other authors [24]. One work first collected maps of the infrastructure, detecting moving objects by comparing newly collected data to these maps [25]. This approach however is hardly scalable for widespread use in collision warning systems.

LiDARs have also been used for analyzing cycling infrastructure. One work uses LiDARs on a bicycle to differentiate the cycling path from the surroundings [26]. Another uses LiDARs found in current smartphones and tablets for detecting infrastructure damages [27]. So far however, there has been no detailed analysis of traffic situations using LiDARs on bicycle, although such works have been announced as well [28].

2.1.2. Autonomous Driving Data Sets

Within the area of autonomous driving, the task of locating vehicles in LiDAR data has been worked on extensively in recent years. Therefore, research and especially data sets from this area can be used as a basis for similar research regarding LiDARs on bicycles. Commonly used data sets in this area are KITTI [29], nuScenes [16] and Waymo [30]. Most recent well-performing approaches are implemented for nuScenes, which is why this data set is chosen as a kind of blueprint for the SaBi3d data set. However, major shortcomings of this data set for the task at hand are that (1) the data is recorded from a car, and (2) the sequences are short (approximately 20 s), which makes it unlikely to capture complete over taking maneuvers.

nuScenes uses their own nuScenes Detection Score (NDS) as evaluation metric, which is composed of different Mean Average Precision (mAP) scores and True Positive scores for the different classes [16]. While in the nuScenes data set multiple classes are annotated, this will not be necessary for the SaBi3d data set since only car-to-bicycle overtaking maneuvers are of interest. Therefore, the mAP of the class ’car’ is taken as evaluation metric, which was also usual in earlier data sets [29,31].

2.1.3. 3D Object Detection

3d object detection algorithms are almost exclusively based on neural networks [32,33]. Object detection has originally been a computer vision task performed on image data [34], but has recently been adapted for point cloud data [35].

A 3d object detection algorithm consists of three components, i.e., (1) point cloud representation,(2) featureextraction, and (3) core object detection [33,35,36].

As a benchmark algorithm, VISTA [37] was chosen since it scored highest on the nuScenes leaderboard at the time. It uses a voxelized point cloud representation, various convolution operations [38,39,40], projection into two different views and cross-view transformers to bring the projections back together [41]. The core object detector can be varied, but the Object as Hotspots detector [42] yielded the best results [37]. Therefore, the same setup was used for the SaBi3d benchmark algorithm.

2.2. Data Collection

The data was collected using a Holoscene Edge bike by Boréal Bikes1. While this bicycle is equipped with multiple cameras, LiDAR sensors, IMUs, GNSS, and further sensors to measure brake activation, humidity and temperature, only the LiDAR sensors were used for this work. This is done to gain a more nuanced picture of the overtaking maneuver while preserving accurate depth information. The LiDAR sensors are Horizon LiDAR sensors by Livox [43]. Their scanning pattern is different from the scanning pattern used in previous data sets such as KITTI and nuScenes, which use rotating sensors. The Livox Horizon uses a scanning pattern reminding of a recumbent eight instead. Point clouds are sampled with a frequency of 5 Hz. The bicycle is equipped with five LiDAR sensors of this type. Since each of them has a horizontal field of view (FOV) of 81.7${ }^{\circ }$, a full 360${ }^{\circ }$ FOV can be achieved by positioning them appropriately.

The data was collected in the afternoons of weekdays in October 2022 in Salzburg, Austria. The data was taken from four different locations, which were selected in such a way that a variety of speed limits and types of bicycle infrastructure would be captured.

2.3. Data Preparation

After the data was recorded, the individual point clouds of each of the sensors were merged into one point cloud for each point in time [26]. The timestamps between each individual point cloud varied slightly but they were all mapped to the closest multiple of 200 ms. The point clouds were further transformed into the format of the nuScenes point clouds.

Using the open-source annotation tool SUSTechPOINTS2 [44], cars were annotated in four sequences, resulting in approximately 10,000 bounding boxes in 4,811 frames, which corresponds to approximately 17 min of recordings.

To allow for the evaluation of training results with this data set, the SaBi3d data set was split into a training split and validation split. The validation split was supposed to contain approximately 15-20% of the total data. One scene was chosen according to its length, resulting in a 82/18% training/validation split.

2.4. Metadata Generation

Furthermore, metadata in the format of nuScenes was generated. This allows for the seamless substitution of the nuScenes data set for the SaBi3d data set when using it for training 3d object detection algorithms. The nuScenes metadata is a database that consists of 13 tables which each have their primary key token and some are linked by foreign keys3. An overview of the tables in the original nuScenes data set and the changes made for the SaBi3d data set is displayed in Table 1. The tables are split into three sections. The first set of tables was not changed at all. The second set was manually adjusted since only minor adjustments were needed. The third set of tables was generated from the recorded point cloud data itself since it required iterating over the scenes, frames, or annotation files.

Some seemlingly unnecessary adjustments were made in order to guarantee the seamless substitution of the nuScenes and SaBi3d data set. This involved the inclusion of a solid black PNG as map data and the definition of an ego pose at zero for every frame. Leaving those out would not allow further processing for object detection through the nuScenes devkit4 which is used by many object detection algorithms.

In addition to the nuScenes-style annotation of cars in the individual point clouds, annotations for full vehicle trajectories were added as well. In particular, it was annotated whether a vehicle performed an overtaking maneuver and whether it constitutes oncoming traffic. This annotation is not needed for 3d object detection but will be useful for potential subsequent work on this data set focusing on the automatic detection of overtaking maneuvers.

2.5. 3D Object Detection Benchmark

Along with the data set itself, we provide 3d object detection benchmark results for comparison with future works using the data set. As a baseline, the VISTA [37] algorithm was chosen. It was hypothesized that the voxelized point cloud representation would minimize the effect of the different scanning pattern of the Livox Horizon Lidar sensors. Therefore, a model trained on the nuScenes data set should easily be transferable to the SaBi3d data.

3. Results

While reproducing the results of VISTA [37] on nuScenes [16], it was discovered that the detection performance of VISTA heavily relies on the class-balanced grouping and sampling (CBGS) preprocessing step [45]. Without the resampling step, poor results were achieved (Table 2). While the original algorithm yielded results between 85.0 and 85.5 mAP of a maximum of 100, the result without resampling was much lower, 53.6 mAP. It was in a similar range when training and evaluating only on the class ’car’, i.e., 58.1 mAP. This posed a problem since the SaBi3d data set only includes one single class, therefore resampling based on classes was not possible.

However, when evaluating the results on the SaBi3d evaluation split (Table 3), this proved not to be a problem. The evaluation on the model trained on nuScenes data with resampling yielded very poor results (0.3 mAP). When fine-tuning this model on only the class ’cars’ from the nuScenes data set, a slight improvement to 12.6 mAP was observed. However, when fine-tuning it on the SaBi3d training data, a mAP of 80.2 on the SaBi3d evaluation data was achieved. Noteably, training the model from uninitialized weights yielded results of 79.1 mAP, almost as high as the fine-tuned model. This shows that the expensive pre-training on the nuScenes data set is not necessary to achieve good results on the SaBi3d data set with current 3d object detection algorithms.

4. Discussion

To increase the modal share of cycling, desirable because of its environmental and health benefits, the characteristics of, objectively or perceived, dangerous overtaking maneuvers need to be understood. A deeper understanding will enable traffic planners to implement measures that will enhance safety. This topic has been studied using a variety of sensors and methods. LiDAR is an obvious choice for measuring the required accurate distance and speed of overtaking cars, however so far the costly manual annotation step presented a major obstacle to its usage. The presented SaBi3d data set is the first public, annotated data set of LiDAR point clouds of car-to-bicycle overtaking maneuvers collected by the bicycle itself, allowing for training and evaluation of a corresponding object detection system. To facilitate the use of our data set, we chose to structure the data almost identical to the popular nuScenes data set, allowing it to be used in various existing object detection systems. We also trained the, at the time of our evaluation, best system on the nuScenes 3d object detection benchmark on our data to provide strong baseline results. Any works using this data set can and should strive to achieve at least this level of object detection performance before pursuing any follow up applications.

Future work might include expanding this data set using different LiDAR setups for ease of usage by other research groups. Another consequent step could be adding a tracking component to the benchmark system and calculation of safety metrics to allow end-to-end assessments of overtaking maneuvers. And last, the data set and benchmark system could be expanded to cover other interesting maneuvers, e.g. at intersections.