Empirical Evaluation and Analysis of YOLO Models in Smart Transportation

1. Introduction

Smart transportation has recently emerged as the optimal solution to address the mobility needs of people [1,2,3]. By leveraging modern technology, such as wireless communication, cloud-fog-edge computing, and object detection, smart transportation aims to simplify processes, enhance safety, reduce costs, and improve overall efficiency in transportation systems [4,5,6]. For example, smart transportation can offer optimized route suggestions, easy parking reservations, accident prevention, traffic control, and support for autonomous driving [4,5,6]. Among modern technology building smart transportation, object detection techniques play the most important role, as identifying and tracking specific objects is crucial for various purposes, such as traffic monitoring [7], collision avoidance [8], pedestrian safety [7], parking assistance [7], infrastructure monitoring [7], and public transport security [9].

Specifically, several object detection techniques exist, such as pioneer detection (i.e., histogram of oriented gradients (HOG) [10], deformable part model (DPM) detector [11], etc.), two-stage detection (i.e., region-based convolutional neural network (R-CNN), spatial pyramid pooling network (SPP-net) [12], Fast R-CNN [13], RepPoints [14], etc.), transformer-based detection (i.e., detection transformer (DETR) [15], segmentation transformer (SETR) [16], transformer-based set prediction (TSP) [17], etc.), and one-stage detection (i.e., region proposal network (RPN) [18], single-shot detector (SSD) [19], You Only Look Once (YOLO) [20], RetinaNet [21], CornerNet [22], etc.). In smart transportation applications, real-time processing is essential to guarantee the safety, accuracy, and efficiency of operating vehicles. For example, in a self-driving car within a smart transportation system, a real-time object detector is required to quickly and accurately detect speed limit signs, pavement, pedestrian zone signs, and other vehicles on the street, allowing the car to take the appropriate subsequent actions in real-time. Therefore, one-stage detection is the most suitable for real-time smart transportation applications due to its fast detection speed and high accuracy [23,24]. Furthermore, among recently developed one-stage detection models, YOLO is considered the leading representative.

First launched in 2015, the YOLO [20] real-time object detection model has been developed through the years with multiple versions (i.e., YOLOv2 [25], YOLOv3 [26], YOLOv4 [27], YOLOv5 [28], YOLOv6 [29], YOLOv7 [30], and YOLOv8 [31]) as depicted in Figure 1. Specifically, YOLO takes a different approach compared with traditional object detection models. Instead of dividing the image into regions and applying classification and localization separately, YOLO performs both tasks in a single pass through its network [24]. It predicts bounding boxes and class probabilities directly on a grid-based output. By doing so, YOLO and its variants have significantly contributed to object detection. Moreover, YOLO models aim to improve themselves, becoming more efficient, accurate, and scalable for real-world applications [32,33,34]. However, YOLO models differ in their network backbone, neck, head, loss functions, and other features, making it crucial to select the optimal models for smart transportation applications by evaluating and analyzing their performance.

Previous studies have investigated object detection models. Nazir et al. [35] introduce an overview of YOLO variants, including YOLOv2, YOLOv3, YOLOv4, YOLOv5, YOLOv6, and YOLOv7. In addition, Sozzi et al. [36] introduce a comparative evaluation of the accuracy and speed of various versions of YOLO object detection algorithms, such as YOLOv3, YOLOv3-tiny, YOLOv4, YOLOv4-tiny, YOLOv5x, and YOLOv5s, for grape-yield estimation. Our study is inspired by these works regarding the performance comparison and architectural analysis of various YOLO models. In contrast, this article specifically focuses on the performance of YOLO models on transportation data that affects the autonomous driving situation in smart transportation. Notably, we evaluate YOLO performance across various datasets, object classes, and intersection-over-union (IoU) thresholds.

In this article, we conduct an empirical evaluation and analysis of multiple YOLO models, including YOLO version 3 to 8 to observe their performance and determine the optimal models for smart transportation, with a focus on road transportation. To achieve this, we first analyze and evaluate the performance of multiple YOLO models in terms of accuracy (i.e., average precision (AP)) according to road transportation data using multiple datasets (i.e., COCO and PASCAL VOC datasets). Second, we evaluate the performance of various YOLO models on multiple object classes within the datasets, specifically focusing on those relevant to road transportation. Third, multiple IoU thresholds are also considered in the performance measurement and analysis. By doing so, we make six observations related to the following three aspects:

• Datasets: The observations on datasets are obtained in the evaluation and analysis. First, the performance of YOLO models trained and evaluated on various datasets can be disparate. In particular, YOLO models perform better on PASCAL VOC than on COCO. Second, YOLOv5 and YOLOv8 are dominant models on both datasets. The reason for the performance disparity of YOLO models on the two datasets can be attributed to various factors, including differences in object classes, dataset sizes, object characteristics, architectural design, optimization techniques, and dataset biases. These factors affect model performance on various datasets, leading to variations in AP scores.
• IoU thresholds: The observation on IoU thresholds indicates that as the IoU threshold increases, the mean AP (mAP) of a YOLO model decreases. YOLOv5 and YOLOv8 perform better than other YOLO models across various IoU thresholds. The first observation is related to the essence of the IoU threshold, where a higher threshold demands a higher standard for accurate predictions. For the second observation, YOLOv5 and YOLOv8 outperform other models, because of their bag of freebies (BoF) and bag of specials (BoS) techniques, such as backbone, neck, head architectures, loss functions, anchor-based mechanisms, modules for enhancing the receptive field, data augmentation strategies, feature integration, and activation functions.
• Class of objects: Based on the observation of object classes, the performance of YOLO models can fluctuate or stabilize depending on the quality of the datasets and the characteristics of the objects within them. Similar to the above observations, YOLOv5 and YOLOv8 outperform most object classes in the two datasets. For the first observation related to the class of objects, we observe that differences in AP values of YOLO models across object classes come from certain factors, such as object class imbalance and object class-specific characteristics. For example, objects with smaller sizes or complex shapes can pose challenges for detection, resulting in lower AP values for those object classes. For the second observation, YOLOv5 and YOLOv8 still maintain their lead, because of their BoF and BoS techniques as analyzed in the observations related to IoU thresholds.

The rest of the article is organized as follows. Section 2 provides the background and motivation. Section 3 describes the evaluation and analysis of multiple YOLO versions in smart transportation. Section 4 discusses related work. Finally, we conclude our work in Section 5.

2. Background and Motivation

2.1. Object Detection in Road Transportation

2.1.1. Object Detection-based Applications

Object detection has become an essential element in contemporary road transportation systems, improving safety, efficiency, and decision-making processes. Four key applications of object detection in road transportation are: (1) intelligent traffic management [4], (2) advanced driver assistance systems, (3) autonomous vehicles [8], and (4) traffic-related object detection and recognition [7].

• Intelligent traffic management: Object detection is applied to monitor and analyze traffic conditions in real-time in traffic surveillance systems. For example, sensors in these systems, combined with object detection techniques, can identify and track vehicles, pedestrians, and other road users. The data gathered from detection and tracking is then employed for traffic flow management, congestion detection, and incident identification, helping to inform decisions and enhance traffic management strategies.
• Advanced driver assistance systems (ADAS): Object detection is a core technique in ADAS designed to improve driver safety and assist with driving tasks. By using sensors, object detection in ADAS can identify and track road objects, enabling features such as forward collision warnings, lane departure alerts, blind-spot detection, and adaptive cruise control.
• Autonomous vehicles: Object detection using multiple sensors, such as cameras, LiDAR (light detection and ranging), radar, and ultrasonic sensors, enables vehicles to perceive and comprehend their surroundings, facilitating autonomous driving without human intervention. The data collected by these sensors allow autonomous vehicles to identify and classify objects like vehicles, pedestrians, cyclists, traffic signs, and traffic lights. This information supports decision-making and ensures safe navigation, enabling vehicles to operate independently.
• Traffic sign detection and recognition: Object detection detects and recognizes traffic signs and road markings, such as traffic signs (speed limits, stop signs, yield signs, and lane markings). This information assists drivers in real-time compliance with traffic regulations and supports autonomous vehicles in understanding and obeying traffic signs.
• In summary, object detection techniques are vital in road transportation systems. They enhance safety and efficiency for drivers, passengers, and pedestrians. Additionally, these techniques enable smart transportation systems to perceive and adapt to dynamic road conditions.

2.1.2. One-Stage-Based Object Detection System

Object detection systems share common components, such as video/image from a camera, image frames generated from the video, object detectors, and object bounding boxes as output. In this article, we focus on evaluating and analyzing the performance of YOLO models (e.g., a one-stage object detector) in smart transportation (e.g., one of IoT applications). A one-stage detector (i.e., YOLO, SSD, etc.) is regarded as the object detector in an object detection system used in smart transportation, such as on autonomous vehicles, smart traffic lights. In Figure 2, we provide an overview of the one-stage object detection system in smart transportation. As illustrated in Figure 2, the one-stage object detection system maintains common components of a conventional object detection system. The video/image input is collected from cameras deployed in smart transportation. The output is bounding boxes of detected objects. The primary difference is which object detection mechanisms are utilized by the object detector. A one-stage object detection system employs one-stage object detection mechanisms as its object detectors. An architectural overview of one-stage object detectors is shown on the right side of the figure.

The architecture of one-stage object detectors in Figure 2 is referred from [37,38]. It typically includes a backbone network, usually pretrained models on ImageNet datasets [39], region-based analysis methods, such as region of interest (RoI) pooling, and a dense prediction with the object locations and classifications in bounding box outputs. For more details:

• Backbone network: acts as a feature generation network, which extracts features from an image. It inputs an image and generates a feature map while preserving spatial information of the image. Popular backbone networks used in object detection methods are Darknet [20], CSPDarknet [39], ResNet [40], VGG [41], EfficientNet [42], MobileNet [43], etc.
• RoI pooling: is a method focusing only on certain portions of the feature map generated from the backbone to output new fixed-length feature vectors. Using the feature map from the backbone network alone is ineffective for locating objects, so RoI pooling addresses this by focusing on a set number of Regions of Interest (RoIs), determined by a region proposal network (RPN), to produce a fixed-length feature vector for each RoI. Thus, RoI pooling gets two inputs, which are the feature map from the backbone and RoI proposals from an RPN.
• Classification and location: fixed-length feature vectors from RoI pooling are input to fully connected layers for classification and bounding box regression (e.g., location)

The YOLO models are one of the one-stage object detection algorithms (i.e., YOLO [20], SSD [19], RetinaNet [21], etc.). The following part provides an overview of the YOLO framework and its models, including key features used in different YOLO versions.

2.2. YOLO Models

Three essential components of the YOLO framework [27] are backbone, neck, and dense prediction (or head), as presented in Figure 3.

• Backbone: is a CNN which extracts features from input images, typically pre-trained on a classification dataset, usually ImageNet, as explained in Section 2.1.2. All backbone models are classification models. Well-known backbones used in object detection are VGG16 [41], ResNet50 [44], EfficientNet [42], CSPDarknet53 [45], etc.
• Neck: layers collected feature maps from different backbone stages. Simply, it works as a feature aggregator. It is composed of several bottom-up and top-down paths as depicted in the figure. Neck networks in object detection include spatial pyramid pooling (SPP) [46], feature pyramid network (FPN) [47], path aggregation network (PANet) [48], etc.
• Head/dense prediction: works as an object detector, which determines the object region but does not specify which object classes are presented in the region. It is equal to the RoI pooling of the one-stage object detection in Figure 2. Head networks can be classified into anchor-based or anchor-free detectors [27].

YOLO versions are developed based on the YOLO framework described above. They vary in the mechanisms used for the backbone, neck, and head. We present a summary of the primary mechanisms, such as backbone, neck, anchor box, activation function, loss function, and data augmentation, utilized in latest YOLO models (e.g., YOLOv3, YOLOv4, YOLOv5, YOLOv6, YOLOv7, YOLOv8) in Table 1 and Table 2.

2.3. Metrics for Evaluating Object Detection

Three popular performance metrics for object detection are frames per second (FPS), recall, and precision [33,49,50,51]. Two main approaches for object detection performance evaluations are speed and accuracy. The fps or frame rate is a performance metric focusing on the speed of the object detection procedure. This performance metric is critical in real-time applications based on object detection, such as autonomous driving, traffic monitoring, video surveillance, and medical detection in healthcare [52]. A typical fps for computer vision applications is about 15 to 30 (fps) [53]. For real-time object detection applications, a high speed of 10 (fps) or higher is required. For example, YOLO models are well-known for their accuracy and speed, on i7 CPU and the inference on 640-resolution images, YOLOv5 Nano, YOLOv6 Nano, and YOLOv6 Tiny perform more than 20 (fps) [54], satisfying the real-time requirement of object detection applications. However, this article focuses on comparing YOLO models in terms of accuracy. The following sections review two popular performance metrics related to accuracy: recall and precision.

Before considering the details of accuracy performance metrics, we discuss the same commonly used concepts. First, it is intersection over union (IoU), which is a principle in all state-of-the-art metrics [55]. It is defined as the intersection over a union of the detection bounding box and the ground truth bounding box.

Figure 4 visualizes the IoU with the detection and the ground truth bounding boxes. The formula for calculating IoU is given below:

$$IoU={\displaystyle \frac{\mathrm{Area}\mathrm{of}\mathrm{Overlap}}{\mathrm{Area}\mathrm{of}\mathrm{Union}}}$$

(1)

• According to IoU and its score threshold definition (e.g., $IoU\ge 0.55$ is considered as a good prediction), how many objects in an image are detected correctly will be determined.
• Other basic concepts used in accuracy metrics are the true positive (TP), false positive (FP), and false negative (FN), which are defined as follows:
  • True positive (TP): indicates the correct detection of the ground truth bounding box [49]. It means the IoU score of the prediction is greater than a predefined threshold
  • False positive (FP): indicates the incorrect detection of the ground truth bounding box [49]. It means the IoU score of the prediction is smaller than a predefined threshold
  • False negative (FN): indicates the undetected ground truth bounding box [49]
• A true negative (TN) indicates that the bounding boxes should not be detected in an image. This concept are not be considered in object detection, because the number of this kind of bounding boxes is massive in an image. Thus, metrics using the TN are not be used to assess object detection methods, such as the receiver operating characteristic (ROC) curves, FP rate (FPR), TP rate (TPR) [49].

Two basic metrics based on the above concepts are precision (P) and recall (R). The precision measures how accurate predictions are. In other words, this metric indicates the percentage of correct predictions.

$$P={\displaystyle \frac{TP}{TP+FP}}={\displaystyle \frac{TP}{\mathrm{all}\mathrm{detections}}}$$

(2)

Recall (R) measures how well the model determines all positives.

$$R={\displaystyle \frac{TP}{TP+FN}}={\displaystyle \frac{TP}{\mathrm{all}\mathrm{ground}\mathrm{truths}}}$$

(3)

F1-Score is the combination of both precision and recall. This metric is known as the harmonic mean of the two metrics. The formula for the F1-Score is represented as follows:

$$F1\_score=2.{\displaystyle \frac{P.R}{P+R}}$$

(4)

Instead of using only one metric (P or R) in the assessment process, an area under the curve (AUC) has been used to indicate both precision and recall. The higher AUC is, the higher both precision and recall are. However, AUC is usually a zigzag-like curve, which makes it hard to analyze the accurate measurement of AUC [49]. Average precision (AP) has been utilized popularly to overcome challenges with AUC. The following part provides an explanation of metrics related to AP.

• $AP@\alpha$ is an area under the precision-recall curve evaluated at $\alpha$ IoU threshold. It is defined as follows:

  $$AP@\alpha ={\int }_0^{1}p\left(r\right)dr$$

  (5)

  One of the most popular metrics developed from $AP@\alpha$ is the mean average precision (mAP). mAP is used to assess object detection methods over all classes in a particular dataset [49]. Basically, it is the average AP over all classes:

  $$mAP={\displaystyle \frac{1}{N}}\sum _{n=1}^{N}A{P}_n$$

  (6)

  where $A{P}_n$ denotes the AP of the $n^{th}$ class in the dataset, and N is the total number of classes in the dataset.
• Which metric should be used for measuring the performance of object detection is crucial. According to the steps in an object detection problem, mAP is suggested to use on a validation set. Otherwise, precision, recall, and F1-Score are adequate on the test set.
• In this article, we focus on analyzing and evaluating the performance of different YOLO versions, thus we consider mAP as the primary performance metric.

2.4. Datasets for Object Detection

Various datasets are used in object detection applications, such as ImageNet [56], COCO [57], PASCAL VOC [58], KITTI [59], and DOTA [60], etc. In this article, we focus on an empirical evaluation and analysis of YOLO models in smart transportation context, such as road transportation. Accordingly, we select COCO and PASCAL VOC as our experimental datasets. The two datasets are capable of providing not only a number of objects in road transportation but efficient notations as well with high-resolution images. Principle statistics of these datasets are provided next.

2.4.1. COCO (Common Objects in Context)

The COCO dataset is a large-scale image recognition dataset for object detection, segmentation, and captioning tasks. It contains over $330,000$ images. Each image is annotated with 80 object categories and 5 captions describing the scene. Object categories are also divided into two main categories, “things” and “stuff”. While the “things” contains object classes easily handled (i.e., animals, vehicles, household items, etc.), “stuff” includes environmental items (i.e., tree, sky, road, etc.) [61]. In particular, some prevalent object classes in 80 classes are person, bicycle, car, airplane, bus, train, truck, traffic light, cat, dog, sink, etc. Moreover, COCO offers various types of annotations, such as object detection, stuff image segmentation, panoptic segmentation, and keypoint. For more details, with object detection annotation, COCO provides annotations with bounding box coordinates for 80 object classes.

2.4.2. PASCAL VOC (Visual Object Classes)

PASCAL VOC provides standardized images for object detection and segmentation tasks. The latest update contains $11,530$ images with 20 object classes. Additionally, it provides $27,450$ annotated objects and $6,929$ segmentations [62]. These object classes are grouped into person, animal, vehicle, and indoor classes. In particular, some prevalent object classes are person, bird, horse, airplane, boat, bicycle, bus, chair, sofa, etc. PASCAL VOC also provides object detection annotation with bounding box coordinates for 20 object classes.

We conduct an accuracy comparison between models using average precision (AP).

Precision-recall curves are good for grasping the overall performance of an algorithm, but have limitations in quantitatively comparing performance between different algorithms. The alternative that came out is Average Precision. Average Precision is a single value representing the performance of the recognition algorithm and is calculated as the area under the graph line on the precision-recall curve. The higher the Average Precision, the better the overall performance of the algorithm. If you have multiple object classes, take the AP for each class, sum them all up, and divide by the number of object classes to evaluate the performance of your algorithm. This is called mAP (mean average precision). Ultimately, the AP of each model is measured by class, aggregated, and mAP is obtained for comparative analysis.

We used road transportation data as the dataset to validate the model. Our goal was to conduct a comparative analysis to determine the accuracy of object recognition in autonomous driving situations. We investigated which model structure and algorithm are most suitable for correctly recognizing objects encountered by a car. It is worth noting that even a YOLO model with a higher version or known for its high accuracy (mAP) may perform worse than previous models on certain datasets. Therefore, in the context of autonomous driving, particularly road transportation, it is crucial to identify the model with the highest recognition rate and accuracy among YOLO models. By doing so, we can find a suitable model for the specific transportation data and improve the overall performance of autonomous driving systems.

We divided the AP results into six dimensions for analysis. We calculated AP for IoU values greater than or equal to 0.50, greater than or equal to 0.75, and for each threshold between 0.5 and 0.95. Additionally, we calculated AP for three different object scales: AP(S) for small objects (area < 32² px), AP(M) for medium objects (32² < area < 96² px), and AP(L) for large objects (area > 96² px).

3. Evaluation and Result Analysis

In this article, the performance of YOLO models is extensively evaluated using various benchmark datasets (e.g., COCO and PASCAL VOC) and the evaluation metric setup (e.g., IoU thresholds). Specifically, we evaluate YOLO models on the COCO and PASCAL VOC datasets which cover a wide range of object categories on road transportation. The mAP evaluation metric is used to measure the accuracy of various YOLO models across different datasets, object classes, and IoU thresholds.

3.1. Experimental Setup

We use a server equipped with a 32-core Intel Xeon Gold CPU and 64 GB of DRAM, running with an 18.04.5 Ubuntu and 4.15.0 Linux kernel for training and evaluation. We use COCO2017 and PASCAL VOC as training and evaluation datasets. In this article, we focus on object detection in road transportation and only consider object classes related to road transportation in two datasets. The COCO2017 datasets have about 80 classes, but only road transportation-related object classes are used in the experiments, including ten classes: bicycle, car, motorcycle, bus, train, truck, traffic light, stop sign, parking meter, and person. The total number of images we use for the validation is 5000 and with each object class detection, the number of labels varies from 60 to 10777.

For the PASCAL VOC datasets, we use 11530 images with 20 object classes of road transportation for training. In validation, we use 3422 images with six object classes (e.g., bicycle, bus, car, motorbike, person, and train). The number of labels varies from 142 to 7835. A summary of the datasets’ usage in the experiments is presented in Table 3.

• We validate various YOLO models on COCO with pre-trained weights. Additionally, we train and validate various YOLO models on COCO and Pascal VOC with CPUs.

3.2. Evaluation Overview

We present the experiment results, comparing the performance of YOLO models in terms of the datasets (e.g., COCO and PASCAL VOC), accuracy (e.g., mAP scores at various IoU thresholds), and object classes. Figure 5a,b illustrate the AP results of YOLO models on COCO and PASCAL VOC datasets, respectively. Figure 6 and Figure 7 indicate how the IoU thresholds affect the AP of object classes in COCO and PASCAL VOC datasets, respectively. Table 4, Table 5, Table 6 and Table 7 show the AP at the IoU threshold 50 and the average AP (e.g., average AP at all IoU thresholds) of YOLO models on specific object classes in COCO and PASCAL VOC datasets, respectively.

3.3. Experimental Results and Observations

The next section examines the performance of YOLO from three perspectives: datasets, IoU thresholds, and object classes. Based on the performance analysis, we provide empirical observations on the performance of various YOLO models when applied to road transportation datasets (e.g., COCO and PASCAL VOC).

3.3.1. Dataset

Figure 5a,b present the AP of six YOLO models at four thresholds $Io{U}_{th}$ (e.g., IoU threshold $=\{50,75,85,95\}$) on the COCO and PASCAL VOC datasets, respectively. As depicted in the figures, YOLO models on the PASCAL VOC datasets can get higher mAP than on the COCO datasets across various IoU thresholds.

For example, as shown in Figure 5a, dominant YOLO models (e.g., YOLOv4, YOLOv5, YOLOv8), trained on COCO datasets, raise the AP in a range $(0.65,0.8)$ at $Io{U}_{th}=50$, $(0.6,0.68)$ at $Io{U}_{th}=75$, $(0.48,0.55)$ at $Io{U}_{th}=85$, and $(0.16,0.21)$ at $Io{U}_{th}=95$. Whereas, for these dominant YOLO models on PASCAL VOC datasets shown in Figure 5b, the AP range is greater with $(0.81,0.99)$ at $Io{U}_{th}=50$, $(0.71,0.8)$ at $Io{U}_{th}=75$, $(0.5,0.67)$ at $Io{U}_{th}=85$, and $(0.09,0.21)$ at $Io{U}_{th}=95$.

YOLO models perform better on the Pascal VOC dataset than on COCO because Pascal VOC contains fewer object classes and a less diverse range of object sizes compared to COCO. Additionally, the PASCAL VOC dataset is specifically designed for object detection tasks, whereas the COCO dataset covers a broader range of computer vision tasks, including object detection, instance segmentation, and more. Hence, images in PASCAL VOC can contain efficient and suitable objects for the training and validation processes of object detection tasks. Thus, according to the database variety, we make the following observation related to the YOLO performance.

• Observation 1: The performance of YOLO models trained or evaluated on different datasets can have disparate performance.

In addition, in Figure 5, we can observe that YOLOv5 and YOLOv8 are dominant models at various IoU thresholds. For example, on COCO datasets, YOLOv5 outperforms the others at $Io{U}_{th}=\{75,85,95\}$. Further, YOLOv8 follows YOLOv5. At $Io{U}_{th}=50$, YOLOv4 and YOLOv6 can achieve high performance, but at other IoU thresholds, they cannot maintain their performance like YOLOv5 and YOLOv8 can do. Therefore, we can not consider YOLOv4 and YOLOv6 as dominant YOLO models on the COCO datasets. On the PASCAL VOC datasets, YOLOv8 is better than the others at $Io{U}_{th}=\{75,85,95\}$. YOLOv5 goes after YOLOv8 at $Io{U}_{th}=\{85,95\}$. YOLOv4 can achieve high performance at $Io{U}_{th}=\{50,75\}$, but has the lowest performance at $Io{U}_{th}=95$. Therefore, we can not consider it a dominant YOLO model on the PASCAL VOC datasets.

• Furthermore, YOLOv5 outperforms other models because it uses the CSPDarknet53 backbone, PANet for effective multi-scale feature integration, the auto-anchor technique for bounding box adjustment, and improved loss function. Additionally, YOLOv8 demonstrates superior performance due to its use of the modified CSPDarknet53 backbone, PAN feature fusion, anchor-free detection, class loss optimization, complete IoU loss for precise bounding boxes, and distribution focal loss to enhance accuracy across diverse object classes. Therefore, we make the following observation related to the datasets as follows.
• Observation 2: The YOLOv5 model is dominant on the COCO dataset, and YOLOv8 is dominant on the PASCAL VOC dataset.

3.3.2. Intersection over Union (IoU) Thresholds

Next, we conduct experiments using various IoU thresholds to observe how YOLO models perform on the two datasets. In detail, four different IoU thresholds (e.g., 50, 75, 85, and 95) are set up in the experiment to measure the performance of the YOLO models using mAP. Figure 5a and Figure 5b present the mAP for six YOLO models on the COCO and PASCAL VOC datasets, respectively.

As shown in the two figures, as the IoU threshold increases (e.g., from 50 to 95), the mAP of all YOLO models tends to decrease. This finding is apparently according to (5) and (6), indicating that a higher IoU threshold results in a higher standard for good prediction. Thus, fewer predictions are considered good or correct. Hence, we make the first observation related to the IoU threshold as follows.

• Observation 3: The mAP of the YOLO models becomes lower with a higher IoU threshold.

Also in Figure 5, we can observe that YOLOv5 and YOLOv8 outperform the rest of the models for most of the IoU threshold scales. For example, on the COCO datasets, YOLOv5 and YOLOv8 obtain the highest mAP at three IoU thresholds (e.g., $75,85,95$). On PASCAL VOC datasets, YOLOv5 and YOLOv8 outperform the other models at two IoU thresholds (85 and 95).

This better performance can be attributed to various architectural improvements and optimizations introduced in YOLOv5 and YOLOv8, such as CSPDarknet53, PANet, Binary Cross Entropy, Logit Loss function, and CIoU loss. The first factor is the model architecture of YOLO models. The architectural design for each YOLO model plays a crucial role in performance at various IOU thresholds. Further, YOLOv3, YOLOv4, YOLOv5, YOLOv6, YOLOv7, and YOLOv8 have distinct network structures, layer configurations, and feature fusion mechanisms. These architectural variances affect the models’ ability to handle object detection tasks with varying IOU thresholds. For example, some models may prioritize precision at higher IOU thresholds, whereas others may focus on recall at lower IOU thresholds. Hence, we provide a second observation of the best YOLO models when varying IoU thresholds.

• Observation 4: The YOLOv5 and YOLOv8 models outperform other YOLO models across IoU thresholds.

3.3.3. Class of Objects

Figure 6 and Figure 7 illustrate the performance of various object classes in the two datasets. As depicted in Figure 6, the performance of YOLO models on object classes in the COCO datasets fluctuates, demonstrating the difference in performance between the traffic light and bicycle classes perform the worst for all YOLO models. For example, at $Io{U}_{th}=50$ and $Io{U}_{th}=75$, traffic light and bicycle classes get the lowest performance with all YOLO models. As for the bicycle class, it achieves around $0.45$ (e.g., with YOLOv7) and $0.22$ (e.g., with YOLOv3) at $Io{U}_{th}=50$ and $Io{U}_{th}=75$, respectively. As for the traffic light class, it even achieves even lower performance of around $0.4$ (e.g., with YOLOv7) and $0.19$ (e.g., with YOLOv3) at $Io{U}_{th}=50$ and $Io{U}_{th}=75$, respectively. Considering the same YOLO models at $Io{U}_{th}=50$ and $Io{U}_{th}=75$, the training object classes can achieve around $0.82$ and $0.7$, respectively, demonstrating a large performance gap between object classes on the same dataset.

In contrast, the performance of YOLO models on object classes in the Pascal VOC dataset remains more stable than in the COCO dataset, as presented in Figure 7 because PASCAL VOC contains more efficient and suitable objects than COCO. The fluctuating and stable performance of YOLO models on various object classes in the COCO and PASCAL VOC datasets respectively indicate that the data quality of object classes can influence the performance of YOLO models. Thus, the first observation related to the datasets is provided below.

• Observation 5: The performance of YOLO models on various object classes can fluctuate or stabilize according to the quality of the datasets.

Table 4 and Table 5 detail the YOLO performance at $Io{U}_{th}=50$ and the average YOLO performance at a range of IoU thresholds $Io{U}_{th}=[50-95]$ on the COCO datasets, respectively. The performance on the PASCAL VOC datasets is presented in Table 6 and Table 7. As shown in Table 4 and Table 6, at a low IoU threshold, $Io{U}_{th}=50$, YOLOv4 outperforms other YOLO models on both datasets. For example, on COCO datasets, shown in Table 4, YOLOv4 achieves the highest performance with $7/10$ object classes (e.g., bicycle, car, bus, truck, traffic light, and parking meter). On the PASCAL VOC datasets, in Table 6, YOLOv4 is dominant for $6/6$ object classes. However, at various IoU thresholds as displayed in Table 5 and Table 7, the dominant models are YOLOv5 and YOLOv8 on the COCO and PASCAL VOC datasets, respectively. For example, on COCO datasets as shown in Table 5, YOLOv5 outperforms with $8/10$ object classes. in contrast, on the PASCAL VOC datasets as shown in Table 7, YOLOv8 outperforms the other models for $5/6$ object classes. Thus, YOLOv5 and YOLOv8 outperform other models for most object classes in the two datasets, leading to the following observation.

• Observation 6: Both YOLOv5 and YOLOv8 outperform the other models on most of the object classes.
• The following section summarizes the overall performance results, demonstrating that YOLOv4, YOLOv5, and YOLOv8 outperform the other YOLO models on the two datasets.

3.4. Summary

In summary, the experimental results confirm the anticipated effects of datasets, object classes, and IoU thresholds on the performance of YOLO models. Specifically, first, the performance of YOLO models trained and validated on different datasets can perform disparately. Second, the higher the IoU threshold is, the lower YOLO performance achieves. Last, YOLO performance on various object classes stabilizes according to the dataset quality.

Moreover, the experimental results show that YOLOv5 and YOLOv8 consistently outperform others accross the IoU thresholds ranging from 50 to 95 on the COCO and PASCAL VOC datasets. Additionally, at a low IoU threshold (e.g., $Io{U}_{th}=50$), YOLOv4 exhibits the best performance on both datasets. These results suggest that YOLOv5 and YOLOv8 are promising choices for object detection tasks in the road transportation scenario, due to their strong performance across a suitable range of IoU thresholds (e.g., from 50 to 95).

4. Related Work

Several studies have focused on evaluating the performance of YOLO variants in object detection. For example, Liule et al. [63] experiment on three versions: YOLOv3, YOLOv4, and YOLOv5. They evaluate and analyze the performance of the three YOLO variants regarding accuracy and inference speed by using training and predicting on the PASCAL VOC dataset. The present study aligns with their research in evaluating and analyzing the performance of YOLO variants on Pascal VOC. In contrast, we evaluated and analyzed more YOLO variants, including the latest one (YOLOv8). Further, we considered their performance on various datasets (COCO and Pascal VOC) and IoU thresholds, whereas they [63] does not consider such diverse factors.

Nazir et al. [35] provide a review of the YOLO variants (YOLO version 1 to 5) on their performance. They also study the accuracy and inference speed of various YOLO models like [63] does but on various datasets (e.g., COCO, KITTI, and SUN-RGBD datasets). The present work aligns with this research regarding evaluating the performance of YOLO variants on various datasets. In contrast, the present study provides a performance evaluation of YOLO variants on various datasets and contributes an empirical analysis of the performance evaluation. Furthermore, the other difference is that this article considers YOLO performance on various IoU thresholds, whereas [35] does not.

Sozzi et al. [36] introduce a comparative evaluation of the accuracy and inference speed of various versions of YOLO object detection algorithms, such as YOLOv3, YOLOv3-tiny, YOLOv4, YOLOv4-tiny, YOLOv5x, and YOLOv5s, for grape-yield estimation. The current study aligns with this work regarding providing a performance evaluation of YOLO variants in object detection tasks. However, while [36] focuses on object detection tasks in smart farming, the current study focuses on smart transportation. Therefore, the datasets used in these experiments are different. Moreover, the current study considers more variants of YOLO than in their study [36], more datasets, and more IoU thresholds.

Finally, this study empirically evaluates and analyzes various YOLO variants considering multiple datasets, object classes in datasets, and IoU thresholds. However, it also considers the scenario of smart transportation (e.g., road transportation), which is novel and imperative among object detection studies in smart transportation.

5. Conclusions

Based on the evaluation and analysis of YOLO’s performance across different datasets, object classes, and IoU thresholds in the context of road transportation, it has been observed that YOLO’s performance can be inconsistent due to the varying quality and characteristics of the datasets and the specific object classes within each dataset. The performance of YOLO models trained and validated on the same datasets may not always align with the expectation that the latest YOLO version will achieve higher performance. This is because, while improvements in YOLO’s internal architecture enhance its capabilities as an object detector, the quality and characteristics of the datasets also significantly influence its performance. Specifically, the empirical evaluation shows that YOLOv5 and YOLOv8 outperform other YOLO models on two widely-used datasets, COCO and PASCAL VOC, which are commonly employed in smart transportation applications. Consequently, these models are recommended as suitable choices for object detection in road or smart transportation systems.