Object Detection Post-Processing Accelerator Based on Co-Design of Hardware and Software

1. Introduction

Object detection, a fundamental task in deep learning, has found widespread application in fields such as autonomous driving [1,2], agricultural monitoring [3,4], and geological exploration [5,6]. The complete training process for object detection includes feature extraction through the backbone [7,8], feature fusion via the neck [9,10], detection heads [11,12,13], post process [14], loss calculation [12,13,15,16], and back propagation. During inference, the step of loss calculation is omitted. The backbone, neck, and head are typically implemented using Convolutional Neural Networks (CNNs). Post process mainly selects valid bounding boxes. In the training phase, post process is responsible for assigning labels to the bounding boxes produced by the detection head. This step involves matching detected bounding boxes with ground truth targets and is commonly referred to as Label Assignment (LA). During the inference phase, post process involves choosing bounding boxes that represent true targets using Non Maximum Suppression (NMS).

As shown in Figure 1, Figure 1(a) illustrates the inference post process of NMS, which selects the most suitable bounding boxes to represent true targets based on the IoU threshold and confidence sorting. Figure 1(a) shows NMS excluding box1 and box3, and selecting box2 as the most appropriate bounding box to represent the true target. Figure 1(b) illustrates the training post process, namely Label Assignment (LA), with Faster-RCNN as an example. Figure 1(b) shows the Faster-RCNN post process, which eliminates box1 and box3, selects box2, and assigns the dog label to the ground truth (GT) target. In supervised learning, matching bounding boxes with true target boxes is crucial for improving model accuracy by optimizing classification losses and positional losses. Both NMS in inference and LA in training share the common goal of selecting the most appropriate bounding boxes to correspond with or represent true target boxes, highlighting their similarity in post-processing tasks. However, previous CNN hardware accelerators for object detection [17,18,19,20] have mainly focused on the feature processing stages, such as feature extraction through the backbone, feature fusion via the neck, and detection heads. Hardware acceleration for post process has received relatively little attention. This oversight stems from the differences in the algorithms of post process across various object detectors. For example, in the training phases of Faster-RCNN [21], YOLOv8s, and GFL [22], bounding boxes are assigned to true target boxes using Max IoU Assigner, Task Aligned Assigner [23], and Dynamic Soft Label Assigner, respectively. Compared to training post process, the pattern of inference post process is relatively singular. In inference post process, all three networks use NMS to select bounding boxes that can represent true targets. Therefore, despite the similarities between training and inference post processes, establishing a unified post processing workflow remains challenging.

In previous post-processing accelerators, NMS has been the primary focus due to its importance for edge inference tasks. However, as edge devices increasingly require more intelligent detection capabilities, they perform local fine-tuning based on real-time data collected through cameras [24,25], thus avoiding the need for cloud-based training and the complex design associated with data transmission between local and remote locations. This evolution underscores the critical need for hardware acceleration in training post process. Despite this, there has been limited research addressing hardware acceleration of training post process to date. This is because post process faces greater challenges in achieving standardization compared to feature process, and has a steeper learning curve, leading to fewer studies on accelerating post process. Our observations indicate that both training and inference post processes constitute a significant portion of the average total processing time for a single image. As shown in Figure 2, we have used an RTX 2080 Ti to measure the processing time for training and inference on images across three typical object detectors, using the COCO dataset. In these experiments, the average time dedicated to post process during training and inference accounted for 20.5% and 22.1% of the total processing time, respectively.

Post process involves a relatively small number of parameters, yet no unified approach has been developed for parameter statistics specific to post process. We propose two potential statistical approaches. Unlike feature process, where mainstream parameter statistics tools (e.g., ONNX tools) focus solely on the number of weights and biases, post process emphasizes bounding boxes, various thresholds, and empirical parameters. The number of bounding boxes, which depends on the size of the input image, includes elements such as confidence scores, categories, and coordinates. The number of thresholds and empirical parameters is significantly smaller than the number of bounding boxes. The first approach for post process includes all bounding box coordinates, categories, and confidence scores, along with thresholds and empirical parameters. During training, it also incorporates the coordinates and categories of true target boxes. The second approach focuses solely on thresholds and empirical parameters, including true target box coordinates and categories during training, which aligns with the parameter scope used in feature process. As illustrated in Figure 3, if post process accounts for the size of feature maps (denoted as Label Assignment with Picture and Non Maximum Suppression with Picture), the total parameters for feature process are 2.28 times to 4.92 times those for post process. Conversely, if post process disregards the feature map size (represented by Label Assignment and Non Maximum Suppression), the total parameters for feature process can be up to 787,000 times those for training post process and 21.4 million times those for inference post process. Given the significant disparity in parameter scale and execution time, accelerating post process becomes imperative. The primary bottleneck in post-processing acceleration is the repeated use of bounding boxes for filtering. Thus, reducing ineffective processes during filtering and maximizing data-level parallelism are critical for enhancing post process efficiency.

Reducing redundancy in post process primarily involves minimizing unnecessary IoU calculations and superfluous comparisons.

• Minimizing Redundant IoU Calculations: Traditional post-processing methods often involve redundant IoU calculations between redundant bounding boxes. To address this, we introduce priority of confidence threshold checking combined with classification (CTC-C) into NMS, effectively reducing the computation of ineffective IoUs. Given the operational similarities between inference post process and training post process, we also implement priority of confidence threshold checking combined with classification (CTC-C) in the training post process of Faster-RCNN. For YOLOv8s and GFL, we incorporate priority of center point position checking combined with classification (CPC-C) to further reduce redundant IoU calculations. We propose a unified algorithmic framework for these four common post-processing methods and present a design for a corresponding hardware accelerator.
• Reducing Redundant Comparisons: Traditional Top-K element selection methods often involve full sorting, which introduces unnecessary comparisons among non-essential elements. To overcome this, we introduce the hybrid bitonic sorter, which avoids redundant comparisons. We detail the algorithm and hardware design of the hybrid bitonic sorter, which efficiently performs IoU sorting for bounding boxes and rapidly identifies the Top-K elements, thereby optimizing the selection of the best bounding boxes during training post process.

Efforts to leverage parallelism in post process can be categorized into two main approaches: data-level parallelism computing with categories and parallel sorting.

• Data-level Parallelism Computing with Categories: By applying principles such as priority of CTC-C and CPC-C, we can determine the category of each bounding box prior to IoU calculation. This allows for simultaneous and independent execution of IoU computations and subsequent operations across different categories.
• Parallel Sorting: We incorporate bitonic sorting to facilitate parallel comparisons. Unlike sorting algorithms with limited parallelism, bitonic sorting offers significantly higher efficiency and speed.

The strategy of data-level parallelism outlined above is implemented in hardware through the design of multiple parallel computation units. The contributions of this paper are summarized as follows:

• Significance of Post Processing Software-Hardware Acceleration: This paper explores the critical importance of hardware acceleration in both training and inference post processing phases. We propose a co-design strategy that integrates both software and hardware to enhance the efficiency of post process.
• Unified Algorithm Framework and Hardware Design: We analyze the feasibility of unifying the algorithms for training and inference post processes and present a hardware designing framework based on a unified algorithmic workflow. The accelerator achieves a minimum speedup of 7.55 times for NMS acceleration compared to recent accelerators. When evaluated solely through IP simulation, our design demonstrates superior acceleration performance relative to the number of bounding boxes. Compared to the RTX 2080 Ti system, our solution provides a minimum speedup of 21.93 times for training post process and 19.89 times for inference post process.
• Reduction of Redundant IoU Calculations: We address the issue of redundant IoU calculations in training and inference post-processing algorithms. By introducing priority of CTC-C in NMS and Faster-RCNN training post process, as well as CPC-C in YOLOv8s and GFL training post process, we effectively reduce the computation of ineffective IoUs. This reduction improves algorithmic efficiency and supports effective software and hardware co-design. With the application of bounding box center point position checking (CPC) and confidence threshold checking (CTC), the number of bounding boxes is reduced to 12.9% and 11.9% of the original count, respectively. Classification further limits the number of maximum bounding boxes per class to 4.4%–19.1% of the original. These reductions validate the effectiveness of our approach, achieving a maximum speedup of 1.19 times and a minimum of 1.10 times.
• Design of Hybrid Bitonic Sorter: We introduce the hybrid bitonic sorter algorithm and its hardware implementation. The bitonic sorter enables highly parallel comparisons, and the Top-K algorithm within the hybrid sorter maintains this parallelism, reducing redundant comparisons typical in traditional Top-K element selection. This enhancement in sorting algorithm efficiency results in a more effective hardware solution, achieving high-performance co-design for both full sorting and Top-K sorting. Our design provides a maximum speedup of 1.25 times and a minimum of 1.05 times compared to recent sorters.

2. The Related Works

2.1. Object Detection and Post Process Algorithm

Traditional object detectors, characterized by their complex handcrafted features, generally lower accuracy, and difficulties in achieving end-to-end integration [26], are increasingly being supplanted by deep learning-based models. These modern approaches utilize network architectures with parameters scaling up to hundreds of megabytes [27,28], enabling extensive learning of object features without manual intervention. By leveraging deep networks, these models provide end-to-end object detection solutions that integrate image processing, feature learning, object classification, and localization in a unified framework.

Object detectors are classified into one-stage [29,30] and two-stage [21,31] detectors. One-stage detectors directly regress boxes in images, while two-stage detectors, although typically slower, first generate candidate regions and subsequently classify and refine these regions. Despite their slower processing speed, two-stage detectors often achieve higher detection accuracy. However, recent advancements, such as anchor-free methods[32,33,34] replacing anchor-based approaches[29,30], improved neck architectures that effectively fuse shallow and deep features[19,27], and adaptive post-processing algorithms that dynamically filter bounding boxes instead of relying on fixed empirical parameters, have led to a gradual increase in the detection accuracy of one-stage detectors. Due to their faster detection speeds, improved accuracy, and lighter models, one-stage detectors have become increasingly prevalent in applications such as intelligent transportation [35] and fire detection [36].

Post-processing algorithms in object detection can be divided into training post process and inference post process. The primary aim of training post process is to categorize samples into positive and negative classes, compute the loss between these samples and the ground truth boxes, and guide the model’s learning process based on supervisory signals. Label assignment is a key component of training post process. Label assignment methods are categorized into static and dynamic types, depending on whether thresholds for positive and negative samples vary. Static label assignment methods use fixed thresholds based on prior knowledge, such as distance and IoU, to differentiate between positive and negative samples. Examples of static methods include FCOS[37], the Max IoU Assigner in Faster-RCNN [21], and RFLA[38]. Conversely, dynamic label assignment methods adjust thresholds dynamically according to various strategies. These include ATSS[14], PAA[39], OTA[40], Dynamic Soft Label Assigner[22], SimOTA[41], and Task Align Assigner[23]. Among these, ATSS considers only dynamic IoU thresholds, while OTA, SimOTA, Dynamic Soft Label Assigner, and Task Align Assigner use joint scores of classification predictions and IoU to align the training and inference post processes. This approach results in more significant accuracy improvements compared to ATSS. Transitioning from the Max IoU Assigner to the Task Align Assigner reduces the reliance on empirical parameters and manual intervention, thereby enhancing the model’s generalizability. Despite these advancements, post process remains complex. Recent approaches have sought to simplify training post process. For instance, the Transformer-based DETR [15,42,43] end-to-end detector employs one-to-one Hungarian matching instead of one-to-many training post process. However, other Transformer-based visual detectors[44,45], such as those using the Swin Transformer[7], still utilize one-to-many training post process and achieve higher accuracy and faster convergence than DETR. Notably, although DETR claims to eliminate complex post process, NMS remains necessary during inference to filter bounding boxes. Consequently, it is expected that one-to-many training post process, which minimizes manual parameters, will continue to be the dominant approach in visual detectors for the foreseeable future.

In contrast to training post process, which has seen significant evolution, inference post process has developed more incrementally, with most advancements focusing on the IoU calculation paradigm [46,47,48,49]. Standard NMS remains the predominant technique, and thus this work employs standard NMS. NMS is a critical post process for eliminating redundant bounding boxes surrounding true targets. The standard NMS approach [50] ranks candidate bounding boxes by their confidence scores and iteratively removes those that overlap significantly with higher-IoU candidates. While standard NMS uses fixed empirical thresholds, it does not adapt to the unique characteristics of each input image, which can constrain the detector’s performance on testing datasets and lead to suboptimal detection results. To overcome this limitation, some research has redefined the selection of NMS thresholds as a convex optimization problem and applied various swarm intelligence optimization algorithms to identify the optimal solution to this problem [51]. Furthermore, adaptive NMS strategies that rely on specific complexity metrics have been developed for crowded scenarios, such as pedestrian and vehicle detection [52,53]. Despite these innovations, methods that aim to eliminate the need for NMS by improving network architectures (i.e., NMS-free mechanisms) [54,55,56] often exhibit lower performance in practical applications compared to detectors that utilize NMS.

In this paper, we focus on the latest advancements in label assignment algorithms, selecting both the Dynamic Soft Label Assigner and the Task Align Assigner. While both are derived from the paradigms established by OTA and SimOTA, they exhibit notable algorithmic differences. Additionally, we use the Max IoU Assigner from Faster-RCNN as a representative of static label assignment methods. For inference post process, we retain the standard NMS as our experimental approach.

2.2. The Hardware Accelerator in Inference Post-Processing Algorithms

Due to the extensive range of applications for object detection at the edge, hardware accelerators for inference post process garner significant attention. NMS is a critical step in this process. It involves selecting the bounding box with the highest confidence score, computing the IoU between this selected box and the remaining boxes, and then filtering out boxes that have an IoU above a specified threshold. This process of selecting the highest confidence box, computing IoU, and filtering continues iteratively until no boxes remain.

Due to the computational complexity of IoU calculation, requiring three multiplications and one division per comparison, with each bounding box needing to compute IoU with every other box, significant power consumption and hardware resource usage are incurred. To address this challenge, approximations of IoU can be employed to replace multiplications and divisions with additions and shifts, thereby reducing power consumption and hardware demands. For example, Shi et al. [57] approximate IoU by calculating the maximum values between coordinates, while Fang et al. [58] introduce additional boundary constraints to effectively eliminate the division step in IoU computation. However, these approximation methods often result in sacrifice with reduced model accuracy.

Due to the loss of model accuracy under IoU approximation, another approach is to enhance the NMS algorithm to minimize redundant calculations and leverage parallelism without loss of model accuracy. Shi et al. [57] improve NMS by employing a sliding window technique on a graph to cluster bounding boxes based on coordinate information. Their revised NMS algorithm decomposes the process into filtering within multiple bounding box clusters and uses position-based bitmap techniques to reduce ineffective data access. This approach lowers memory access and power consumption. Despite its effectiveness in evaluating similar bounding boxes through coordinate information, partitioning bounding box clusters remains challenging, particularly in dense scenarios such as face detection. Chen et al. [59] advance NMS by routing bounding boxes to three separate branches of the output head, each designed for a different ratio, and calculating the regression with maximum confidence for each branch. After merging these branches, the most suitable bounding box is selected. This method enhances parallelism up to three branches and decreases IoU calculations between bounding boxes across different branches. However, for both anchor-based and anchor-free methods, the varying number of anchors and points between different ratios makes it difficult to consistently maintain parallelism at three. Choi et al. [60] significantly improve NMS by introducing a fixed limit on the number of valid bounding boxes involved in each IoU calculation. While this modification reduces redundant calculations, the fixed limit on valid bounding boxes reduces the algorithm’s adaptability in detecting target boxes.

Furthermore, optimizing the NMS by maintaining its core principles while utilizing its threshold parameters to mitigate IoU redundancy represents another strategy for improving algorithm performance. Research [61,62,63,64] shows that applying confidence thresholds to filter out unsuitable bounding boxes can significantly reduce the number of boxes involved in IoU calculations. Anupreetham et al. [65] innovate by adapting the NMS approach for SSD detectors. Instead of the traditional method of waiting to process all bounding boxes and then selecting the one with the highest confidence, their approach employs a pipelined system where boxes are sequentially fed into the computation engine. They introduce three criteria for bounding box selection: whether the box belongs to the same class, whether its IoU exceeds a threshold, and whether its confidence is greater than that of the currently occupied box. This approach is distinguished by its incorporation of class information into the bounding box selection process, unlike the method proposed by Shi et al. [57], which focused solely on coordinate-based clustering and did not address the distribution of different classes within the same spatial region. However, despite [65] integrating class information into the selection process, Anupreetham et al.’s approach does not apply class information to the exclusion of IoU redundancy.

This work introduces an approach that prioritizes both class information and confidence thresholds checking to significantly reduce IoU calculations between bounding boxes of different classes before performing the IoU computation. Table 1 provides a detailed comparison of recent studies highlighting these differences.

2.3. The Hardware Accelerator in Sorter Algorithm

Due to the necessity of sorting by confidence scores and IoU in NMS, as well as the requirement for Top-K selection of alignment scores in both the Dynamic Soft Label Assigner and Task Aligned Assigner, efficient sorting techniques are critical. Fang et al. [58] address the storage challenges associated with sorting operations by using a comparator tree, which reduces the storage demands by efficiently managing sorted elements, given that NMS focuses only on the maximum confidence value in each iteration. Meanwhile, Sun et al. [64] employ parallel bitonic sorting to order IoU values between bounding boxes.

Bitonic sorting [66], in contrast to traditional bubble sorting, provides faster sorting through parallel comparisons. Bitonic sorting involves two phases: In the merge phase, input elements are organized into bitonic sequences that include both ascending and descending subsequences. At the end of this phase, two subsequences of elements each are created, corresponding to ascending and descending orders. The subsequent sort phase involves recursively subdividing these subsequences until each is reduced to a length of one.

Although this approach improves efficiency, hardware implementation incurs higher costs with increasing numbers of elements N due to the necessity of multiple comparators. To address this issue, Chen et al. [67] avoid further recursion after the merge phase by using dual insertion sorting, which reduces resource consumption. Zhao et al. [68] extend the bitonic sorting concept by introducing a smaller merge sort after the merge phase to facilitate Top-K extraction, specifically for K=2. Alternatively, Fang et al. [69] use insertion sorting to identify Top-K elements, but this method is constrained by the sorting scale and longer delays.

In line with the hybrid bitonic sorting methods proposed by [67,68], our work introduces a hybrid bitonic sorting that supports both full sorting and Top-K extraction. However, our method differs by incorporating multi-comparator trees during the merge phase to directly obtain Top-K elements, instead of the traditional approach that only identifies the maximum value. Additionally, we overcome the K=2 limitation present in previous methods, enabling greater parallelism due to the parallelizable nature of retrieving K elements. Table 2 presents the sorting schemes of recent sorting accelerators and provides a comparison.

3. The Unification of Post Process Algorithm and Optimization

3.1. Analysis of the Uniformity of Post-Processing Algorithms

In Faster-RCNN, the Max IoU Assigner begins by filtering out unsuitable bounding boxes based on a confidence threshold. It then calculates the IoU between the remaining bounding boxes and the ground truth boxes, selecting the bounding boxes that match the ground truth boxes according to predefined thresholds. Similarly, in Yolov8s, the Task Align Assigner and GFL’s Dynamic Soft Label Assigner first eliminate unsuitable bounding boxes based on whether their center points fall within the ground truth boxes. They then compute alignment scores for each bounding box by combining classification scores and IoUs. The Task Align Assigner assigns a fixed number of bounding boxes for each ground truth target, whereas the Dynamic Soft Label Assigner adaptively determines the number of bounding boxes by selecting and summing the top-K alignment scores for each ground truth target, making it more flexible.

For NMS, the process starts by selecting the bounding box with the highest confidence score from all candidate boxes. The IoU is then calculated between this highest-confidence box and the remaining boxes. Bounding boxes with an IoU exceeding a predefined threshold are removed. This process of selecting the highest-confidence box and recalculating IoU is repeated until no bounding boxes remain.

From a procedural standpoint, as illustrated in Figure 4, we categorize the workflow into four stages: Pre-Process, Alignment, Dynamic Sampling, and Sorter:

• Step 1. Pre-Process: This step involves extracting the bounding box with the highest confidence or verifying whether the bounding box center point is within the ground truth boxes. To be detailed, as shown in Figure 4, the light green background represents the Pre-Process stage, where the input is data related to bounding boxes, and the output is qualified bounding boxes that have undergone classification, confidence judgment, and center determination of the bounding box. The core element is the Pre-Process Engine.
• Step 2. Alignment: This step calculates the IoU between the bounding boxes and the ground truth. Both the Task Align Assigner and the Dynamic Soft Label Assigner additionally compute alignment scores by integrating classification values with IoU. As shown in Figure 4, the light pink background represents the Alignment stage, where the input includes the coordinate and category information of qualified bounding boxes, as well as the coordinate and category information of ground truth. Possible outputs include the IoU matrix and alignment score matrix constructed based on qualified bounding boxes and ground truth. The core elements are the IoU Engine, Score Engine, and Cost Engine, with the IoU Engine responsible for calculating the IoU matrix, the Score Engine responsible for calculating the classification score matrix, and the Cost Engine responsible for calculating the alignment score matrix.
• Step 3. Dynamic Sampling: This step determines the number of bounding boxes corresponding to each ground truth target, a requirement specific to the Dynamic Soft Label Assigner based on its alignment scores. In Figure 4, the light yellow background represents the Dynamic Sampling stage, where the input is the alignment score matrix, and the output is the maximum number of bounding boxes that each ground truth can be assigned to. The core element is the Top-K Sampling Engine, used only for Dynamic Soft Label Assigner.
• Step 4. Sorter: This final step involves sorting and selecting bounding boxes based on IoU matrix or alignment score matrix. In Figure 4, the light blue background represents the Sorter stage, where possible inputs include the IoU matrix, alignment score matrix, and the number of bounding boxes that each ground truth can be assigned to, with the output being the assignment of bounding boxes to each ground truth or true target.

Figure 5 illustrates the re-expression of the four types of algorithms within a unified workflow. The orange boxes highlight the enabled steps, while the light blue arrows indicate the flow of data.

• NMS: In the Pre-Process stage, NMS identifies the bounding box with the highest confidence score. During the Alignment stage, it calculates the IoU between the selected bounding box and the remaining ones. In the Sorter stage, IoUs are sorted, and bounding boxes are filtered based on predefined IoU thresholds. This iterative process continues until no bounding boxes remain.
• Max IoU Assigner: This algorithm filters bounding boxes based on confidence in the Pre-Process stage. It then computes the IoU matrix between bounding boxes and ground truth boxes in the Alignment stage. In the Sorter stage, IoUs are sorted and used to filter bounding boxes according to IoU thresholds, mapping each bounding box to its corresponding ground truth box.
• Task Align Assigner: In the Pre-Process stage, this algorithm filters bounding boxes based on confidence. The Alignment stage involves calculating IoU matrix and alignment score matrix. During the Sorter stage, alignment scores are sorted, and bounding boxes are selected based on a fixed number of boxes per ground truth target, mapping them accordingly.
• Dynamic Soft Label Assigner: This method filters bounding boxes based on confidence in the Pre-Process stage. It calculates IoU matrix and alignment score matrix, using both classification and IoU, during the Alignment stage. The number of allowable bounding boxes per ground truth target is determined based on alignment scores in Dynamic Sampling stage. In the Sorter stage, alignment scores are sorted, and bounding boxes are selected according to the number required for each ground truth target, effectively mapping them to the ground truth boxes.

Based on our analysis of post-processing algorithms, we propose a unified algorithmic workflow. By investigating the implementation details and characteristics of each algorithm, we examine their specific functionalities within this unified framework. This approach establishes a solid foundation for hardware design.

3.2. Redundancy Analysis of Post-Processing Algorithms

Both training and inference post processes involve extensive IoU calculations. The IoU calculation involves computing the areas of two bounding boxes, A and B, and their overlapping region. The IoU is then defined as the ratio of the overlapping area to the total area covered by both bounding boxes, as expressed by Equation 1. Here, Equation 2 calculates the area $S_A$ of A based on the width $w_A$ and height $h_A$ of A, Equation 3 calculates the area $S_B$ of B based on the width $w_B$ and height $h_B$ of B, and Equation 4 marks the overlapping area of A and B as $S_{\mathrm{cross}}$. Each IoU calculation requires three multiplications and one division. Consequently, optimizing the IoU calculation is crucial for efficiency. Unlike the methods proposed in [57,58], which simplify the IoU calculation to reduce the number of multiplications and divisions, our approach retains the standard IoU computation. This choice ensures the accuracy of alignment scores and maintains the algorithm’s generalizability in training post process.

$$\mathrm{IoU}={\displaystyle \frac{S_{\mathrm{cross}}}{S_A+S_B-S_{\mathrm{cross}}}}$$

(1)

$$S_A=w_A\times h_A$$

(2)

$$S_B=w_B\times h_B$$

(3)

$$S_{\mathrm{cross}}=S_A\cap S_B$$

(4)

In post-processing algorithms, IoU calculations are performed either between different bounding boxes or between bounding boxes and ground truth boxes. As depicted in Figure 6, the volume of such IoU calculations in post process can reach the order of millions. Given the previously discussed principles of post process, these calculations often involve redundancy. Consequently, eliminating redundant IoU calculations can significantly reduce the system’s overall power consumption and improve its acceleration efficiency.

In standard post-processing algorithms, a substantial number of IoU calculations are performed between bounding boxes of different classes or between bounding boxes and ground truth boxes of different classes. However, many of these IoU calculations do not aid in the selection process. To address this inefficiency, distinguishing bounding boxes by class through a prior classification step can be advantageous. Unlike the approach taken in [65], our method incorporates classification before performing IoU calculations. Specifically, this classification strategy is applied after confidence filtering or bounding box qualification but prior to IoU computation.

3.3. Redundancy Optimizations for Post-Processing Algorithms

Employing early classification prior to computing IoU can significantly reduce the number of redundant IoU calculations. For instance, in the inference phase, as depicted in Figure 7, the object detector’s classification head generates 8 bounding boxes, necessitating pairwise IoU calculations. This results in a total of 64 IoU computations. By incorporating early classification and limiting IoU calculations to bounding boxes of the same class, the number of required IoU computations is reduced to 22. Consequently, this approach decreases the total IoU calculations to 34.4% of the original amount. Similarly, during the training phase, if 8 bounding boxes are returned and need to be matched with 5 ground truth boxes, the number of IoU calculations would otherwise total 40. However, with early classification, this number is reduced to 13, which is 32.5% of the original number.

In contrast to [57,58], our approach, consistent with several other studies, retains confidence filtering in the NMS algorithm and preserves the bounding box center point position checking in the post-processing phase. Despite incorporating early classification, we have not modified the post-processing algorithms. Instead, we have reduced redundant calculations by leveraging early classification.

4. Accelerator Framework Design

4.1. Overall Framework Design of Accelerator

Based on the previous analysis of the uniformity and redundancy in the post-processing algorithms, we identify that post processes can be effectively structured into four key stages: Pre-Process, Alignment, Dynamic Sampling, and Sorter.

We propose a hardware accelerator architecture for post processes, as illustrated in Figure 8. The system consists of a CPU for managing read/write control registers, a DMA for transferring data to BRAM, BRAM which supplies data to the accelerator IP, the post-processing accelerator IP, and a bus system for control and data transfer. The post-processing accelerator IP is designed to implement the four stages: Pre-Process, Alignment, Dynamic Sampling, and Sorter. Control signals are utilized to identify whether the algorithm is in training or inference mode. In training mode, additional differentiation is made among the Max IoU Assigner, Task Align Assigner, and Dynamic Soft Label Assigner. Data pathways are selected through enabling mechanisms, as depicted in Figure 5.

In our design, the data width is set to 8 bits, primarily because the precision loss during post process is considerably less significant compared to that in feature processing network layers. Consequently, this choice does not markedly affect the final detection accuracy, with an observed average accuracy loss of only 0.02%. We process 8 ground truth boxes and 64 bounding boxes in a single pass. The input data comprises two coordinates for each GT, center point coordinates, width and height dimensions for each bounding box, and the category data and confidence scores for each box. This data is written to the corresponding BRAM write ports via DMA, while the read ports of BRAM supply data to the post-processing accelerator IP.

4.2. The Design of Hybrid Bitonic Sorter

The Box Confidence Filter, Top-K Engine, and Full Sorter within the post-processing accelerator (shown in Figure 8) all require sorting. In this paper, we propose an innovative hybrid sorting approach based on bitonic sorters. During the merge phase of bitonic sorting, the ascending and descending subsequences can be utilized to pinpoint the maximum values. Moreover, leveraging the known order relationships within these subsequences, a comparator tree can efficiently determine the Top-2 and Top-3 maximum values. This approach allows us to replace traditional sorting phases with a comparator tree, thus minimizing unnecessary sorting operations and facilitating faster retrieval of Top-K data. Our sorter design features a fundamental unit comprising a 32-input sorter, capable of managing up to 16 Top-K elements, as demonstrated in Figure 9.

4.3. The Design of Pre-Process Submodule

As illustrated in Figure 10, bounding boxes are derived from different ratio layers centered at various points (denoted as × on the left side of Figure 10). By sliding across image frames (i.e., moving × from left to right as shown), the number of bounding boxes that need to be compared or filtered can be significantly reduced. This method is effective because a bounding box in the bottom right corner of the image is unlikely to overlap with one in the top left corner. Moreover, this approach tends to group bounding boxes near the same center point into a limited number of categories, thereby facilitating the formation of clusters of bounding boxes within the same class. We utilize this sliding technique for bounding boxes around the input center points in our approach.

In the Pre-Process submodule, the data path varies according to the algorithm type, as depicted in Figure 10. The input to the Pre-Process submodule can come from bounding box-related data in both YOLOv8s training post process and GFL training post process, as well as from NMS and Faster-RCNN training post process. For the former, the data is processed in the Box Center Check to determine if the bounding box center point lies within the ground truth box. Here, x, w, y, and h represent the x-coordinate of the ground truth center point, the ground truth’s horizontal width, the y-coordinate of the ground truth center point, and the ground truth’s vertical height, respectively. $xl$, $xr$, $yb$, and $yu$ represent the left x-coordinate, right x-coordinate, bottom y-coordinate, and top y-coordinate of the ground truth, calculated based on x, w, y, and h. $xc$ and $yc$ represent the x-coordinate and y-coordinate of the bounding box center point, respectively. The output results are entered into the Trace Table’s inout column; for example, ➀, where 1 indicates that the bounding box center point is within the ground truth, and 0 indicates it is not. For the latter, the bounding box confidence is filtered by the Box Confidence Filter using a confidence threshold, and the results are filled into the filter column of the Trace Table; for example, ➀, where 1 indicates confidence above the threshold, and 0 indicates confidence below the threshold. Subsequently, all four algorithms proceed to Classification, where they are categorized using a classification tree. For example, both ➁ and ➁ send the probability of each category for qualified bounding boxes into Classification, and the results are written into the Trace Table, such as ➂ and ➂, which record the category judgment results in the corresponding class column of the Trace Table. With this, the Pre-Process for the post processes of YOLOv8s, GFL, and Faster-RCNN training is complete, and the Trace Table can be passed on to the Alignment submodule. NMS, however, still requires sending bounding boxes of the same category to Box Confidence Top1 to obtain the bounding box with the highest confidence for each category, such as box5 and box7 being the bounding boxes with the highest confidence in the first and third categories, respectively, and then passing the identifier and Trace Table to the Alignment submodule. The Top1 Sorter with confidence still employs the classification tree.

4.4. The Design of Alignment Submodule

As shown in Figure 11, the data flow within the Alignment submodule differs based on the algorithm used. Taking the training post processes of YOLOv8s and GFL as examples, in the IoU Engine, the GT coordinates for class 1 and class 3 serve as one input for IoU calculation, while the other input comes from the coordinates of bounding boxes of the same class, thereby obtaining the IoU between the GT and the bounding boxes. Subsequently, in the Score Engine, the category of the bounding box is input to obtain the cross-entropy category score. Due to the computational complexity of the logarithmic operations in the cross-entropy function, a lookup table is utilized within the module for this calculation. Following this, in the Cost Engine, the alignment score for GFL is derived by summing the classification score and the IoU. For YOLOv8s, the alignment score is obtained by multiplying the classification score and the IoU after applying a power function with an exponent of 1, which does not significantly affect the model’s accuracy. Finally, GFL sends the alignment score to the Dynamic Sampling submodule, while YOLOv8s sends the alignment score to the Sorter submodule. In contrast, for NMS and Faster-RCNN training post process, the Score Engine and Cost Engine are not required. NMS uses the coordinates of the highest confidence bounding boxes output by the Pre-Process submodule as one input for IoU calculation, with the other input being the coordinates of other bounding boxes of the same class. Faster-RCNN, on the other hand, uses the GT coordinates for class 1 and class 3 as one input for IoU calculation, with the other input coming from the coordinates of bounding boxes of the same class, thereby obtaining the IoU between the GT and the bounding boxes. The IoU obtained from NMS and Faster-RCNN post processes through the IoU Engine is directly sent to the Sorter submodule.

4.5. The Design of Dynamic Sampling and Sorter Submodules

As shown in Figure 12, the Dynamic Sampling submodule is exclusively used for training post process in GFL. The alignment scores between each bounding box and ground truth (i.e., the Cost Table) are obtained from the Cost Engine of the Alignment submodule. These alignment scores are then processed by the Top-K Engine and Sum Engine within the Dynamic Sampling module, as shown in Figure 8, yielding the results presented in the blue table of Figure 12. The Top-K Engine first selects the top K maximum cost values between the ground truth and the bounding boxes, and then the Sum Engine calculates the Sum values for each gt in the blue table in Figure 12. For example, for gt1, the top K maximum numbers are only 0.8 and 0.3, so they sum up to 1.1, and thus the Sum for gt1 is filled with 1.1. To determine the maximum number of bounding boxes, Num, that can be assigned to each gt, the decision is made based on whether the decimal part of the fixed-point number is zero—if it is zero, the integer part is taken, such as 1.0 being taken as 1; if it is not zero, the integer part is incremented by one, such as 1.1, where the integer part is incremented by one to get 2. This result is stored in the buffer of the Dynamic Sampling submodule and subsequently passed to the Sorter submodule.

For the Sorter submodule, the inputs vary among the four types of algorithms. For GFL, the number of boxes, Num, corresponding to each gt and the Cost Table are used as inputs, with boxes being assigned to the appropriate gt through the Top-K Engine; for YOLOv8s, the Cost Table is used as input, and boxes are assigned to the appropriate gt through the Top-K Engine; for Faster-RCNN and NMS, the IoU Table is used as input. For Faster-RCNN, boxes are assigned to the appropriate gt through the Top-K Engine, while for NMS, the most suitable boxes to represent the true targets are selected through the Full Sorter and Threshold Filter. Taking GFL as an example, gt1, gt2, and gt3 are allowed to be assigned up to 2, 1, and 2 boxes, respectively. Therefore, gt1 can initially be assigned to box1 and box5, gt2 can be assigned to box1 (box5, due to its large cost value and the fact that gt2 can be assigned to at most one box, is not assigned to gt2, which is referred to as The allocation failure using Top-K in training or using filter in NMS), and gt3 can be assigned to box4 and box7. Since the same box cannot be assigned to two gts, gt1 and gt2 compete for box1. According to the principle of the smallest cost, gt2 obtains box1, and gt1 loses its box1 (this is referred to as The allocation failure using Top-K in training after horizontal allocation). The same applies to YOLOv8s and Faster-RCNN. For NMS, the Full Sorter first completes the IoU sorting between each high-confidence box and other boxes within the same category, and then the Threshold Filter removes the bounding boxes above the IoU threshold.

5. Experiments and Results Analysis

5.1. Experimental Setup

We deploy the solution on the ZCU102 platform. The implementation uses the Xilinx Central DMA IP for data transfer and the Xilinx AXI BRAM Controller as the BRAM controller. The post-processing accelerator is encapsulated as an AXI Slave module and is connected via the AXI Interconnect. The CPU configures the control registers of the post-processing accelerator through the AXI Interconnect, including those for NMS, Faster-RCNN, Yolov8s, and GFL. While we have successfully implemented the complete post-processing accelerator pipeline and aimed to integrate the four distinct post-processing types, FPGA resource constraints limit the configurability of parameters for GFL and Yolov8s. As a result, we can only compute specific parameters for these detectors, as permitted by the algorithm. Despite these limitations on manually configured parameters for GFL and Yolov8s, our approach maintains software programmability. Overall, we have demonstrated the hardware accelerator’s programmability for common inference and training post-processing scenarios.

5.2. Redundancy Results Analysis

By employing the priority of Confidence Threshold Checking combined with Classification (CTC-C) and Center Point Position Checking combined with Classification (CPC-C), we effectively reduce redundant calculations. As shown in Figure 13, we assess the impact of classification on redundancy reduction using the COCO dataset. Applying CTC-C and CPC-C alone reduces the total number of bounding boxes to 12.9% and 11.9% of the original count (original), respectively. Furthermore, incorporating classification further reduces the maximum number of bounding boxes per category (the maximum reduction in the category) to between 4.4% and 19.1% of the original number (original). These results confirm the effectiveness of our methods.

5.3. Analysis of Results for Sorter Optimization

We conduct a comparative analysis of several sorting algorithms, focusing on sorting cycles and resource utilization. Figure 14 presents a comparison of resource consumption between our proposed sorter and other existing sorters. Notably, parallel bubble sort exhibits substantially higher resource requirements. In contrast, our proposed sorter maintains a moderate level of resource usage, demonstrating a more efficient use of resources.

Table 3 presents the cycle data, which detail the number of cycles needed for full sorting, finding the maximum value among 32 elements, and identifying the top 16 values among 32 elements. Although parallel bubble sort demonstrates shorter sorting cycles, it requires substantial hardware resources due to its complexity. In contrast, our proposed sorting solution uses only a modest amount of additional resources for Top-K element extraction compared to a basic bitonic sorter. When evaluating both resource consumption and cycle efficiency, our proposed sorting solution offers a well-balanced performance across all compared schemes.

5.4. Analysis of Results for Post-Processing Accelerator

We compare the performance, power consumption, and resource usage of various post-processing accelerator, as detailed in Table 4. Power consumption is measured as the total power, including both static and dynamic components. Our proposed hybrid bitonic sorter achieves performance improvements of up to 1.25 times and no less than 1.05 times compared to the solution presented in [67]. Relative to the original bitonic sorter, our implementation provides faster post processes for both inference and training, while incurring only a 0.05% increase in LUT usage and a 0.001% increase in FF usage.

Due to the differing degrees of redundancy among various solutions, we evaluate the effectiveness of redundancy reduction across different algorithms using our proposed accelerator scheme. As shown in Table 5, our solution provides a maximum speedup of 1.19 times and a minimum speedup of 1.10 times.

Table 6 presents a comparison between our approach and recent studies. Notably, aside from ASIC-based solutions, our work is one of the few that incorporates data retrieval from DDR and loading onto the accelerator while adhering to the official boundary box specifications. In accelerator that utilize DDR for data storage, our solution achieves a minimum speedup of 7.55 times in NMS. When considering IP simulation alone, our approach offers even greater acceleration benefits, particularly when factoring in the number of candidate boxes. In comparison to systems utilizing an RTX 2080 Ti, our solution provides a minimum speedup of 21.93 times in training post process and 19.89 times in inference post process.

6. Conclusions

This paper provides a comprehensive analysis of both training and inference post processes and proposes a unified approach for four common post-processing schemes. To address the issue of ineffective IoU calculations in these algorithms, we introduce priority of classification to further reduce ineffective IoU calculations for bounding boxes. We present a hybrid bitonic sorter algorithm and its hardware implementation, which utilizes the bitonic sorter for highly parallel comparisons. Lastly, we propose a unified hardware accelerator designed specifically for post-processing tasks.