Simple Evaluation of Association Quality in Tracking-by-Detection

1. Introduction

Tracking-by-detection is a standard paradigm in computer vision, radar technology, surveillance and autonomous driving. Tracking is a post-processing stage that follows detection. It is used to extend the applicability of automatic systems. Often, tracking and detection are considered complementary procedures within the same computer-vision pipeline. The quality of such pipelines is paramount. Consequently, several detection and tracking benchmark challenges exist to identify the most accurate algorithms. However, even in the tracking challenges, the organizers focus on an integral assessment of the entire computer-vision pipeline. This objective leads to cumbersome evaluation procedures in which both parts (detection and tracking) are effectively evaluated simultaneously.

Although detection and tracking are interconnected, they can be separated and optimized independently. This separation is often desirable and sometimes required in complex projects. Motivated by this need, we developed a lightweight evaluation technique dedicated specifically to tracking algorithms.

The proposed evaluation technique:

• focuses exclusively on tracking algorithms,
• relies on an instrumentation of the association procedure of the evaluated tracker,
• avoids an evaluation-scoped association procedure entirely,
• is equally simple to implement in 2D or 3D trackers,
• offers the standard binary classification of tracker output,
• enables experimentation with absent detections and clutter,
• serves as a complement to established evaluation methodologies.

In Section 2, we present the state-of-the-art tracking evaluation approaches in more detail. In Section 3, we review a prototypical tracker, analyze possible outcomes and develop a method to classify the result of the association procedure. In Section 4, we consider several toy scenarios to demonstrate the proposed evaluation procedure. To facilitate the practical adoption of this framework, we provide several software packages described in Section 5. Section 6 details a baseline use-case scenario of the proposed framework. Finally, we discuss potential directions for further work in Section 7 and conclude in Section 8.

2. Related Work

Several methodologies for evaluating tracking algorithms are established in the current literature. The most mathematically rigorous frameworks are derived from the optimal sub-pattern assignment (OSPA) family, notably the generalized (GOSPA) [1] and trajectory-generalized (T-GOSPA) [2] metrics. Although fully compatible with the demands of tracking challenges, these metrics have not been adopted by major mobility datasets.

Instead, the evaluation methodologies favored in autonomous mobility challenges originate from the CLassification of Events, Activities, and Relationships (CLEAR) project [3]. The application of CLEAR to autonomous driving yielded the ClearMOT metric [4], which was later revised into the Higher Order Tracking Accuracy (HOTA) framework [5], a practical solution for 2D tracking. Consequently, these CLEAR-derived metrics represent the established standard across widely used mobility datasets, including KITTI [6], nuScenes [7], and Waymo [8].

The goal of these evaluation procedures is to determine the best-performing computer-vision pipeline. For this purpose, each dataset is divided into training and evaluation splits. The training split contains raw sensor data (images and point clouds) along with annotations, while the evaluation split contains only raw, unannotated data. Contributors submit their tracking outputs to the challenge organizers. The organizers compare these outputs against the withheld ground truth and compute the quality metrics.

In the tracking challenges, the trackers are provided with raw detections, while the tracker output is ultimately compared with ground-truth annotations. In order to match annotations and tracked targets, an association procedure is employed. Given the probabilistic nature of this association, matching is evaluated at different parameter thresholds to select the maximal number of matches. Finally, various quality metrics are computed. These metrics include the binary classification of the tracker output as well as the deviations of the target characteristics.

This design has two key advantages: (i) developers do not need to disclose their algorithms, and (ii) the separation of annotations helps prevent a possible data-specific tuning. On the other hand, the evaluation must be performed on the entire computer-vision pipeline. The end-to-end evaluation objective increases the number of tunable parameters. Moreover, the evaluation becomes cumbersome. The association procedure employed during evaluation is similar to the tracker’s own association logic. This similarity introduces fundamental uncertainty and complicates the analysis of tracker shortcomings.

In many scenarios, particularly during the initial phases of development, end-to-end evaluation of tracking systems proves excessively complex. It is often more pragmatic to validate individual pipeline components prior to holistic testing. Furthermore, the stringent requirements of double-blind evaluation are typically unnecessary when the tracking algorithm remains fully accessible and subject to ongoing adjustment. Driven by these conditions, we introduce a streamlined evaluation methodology. Our solution circumvents probabilistic association in favor of directly analyzing the tracking algorithm, thereby facilitating immediate binary classification of the output. Notably, despite the inherent simplicity of this approach, we are unaware of any analogous proposals within the existing technical literature.

3. Methodology

A generic multiple-object tracker processes the detector output to generate a scene of moving targets (tracks).1 Targets include vehicles, pedestrians, houses, trees, and other obstacles. The objects are often represented by rectangles or cuboids located in space and possessing kinematic characteristics such as velocity and acceleration. The detector reports typically include the location and dimension of objects, but do not provide any identification. Detectors work on a timely schedule, providing time-stamped batches of measurements. The time stamps allow for the derivation of the kinematic characteristics of the targets.

A generic tracker corroborates a correspondence between the time-stamped measurements in the form of consistent, unique target IDs. The assignment of IDs occurs as a result of an association procedure. The association is a probabilistic procedure designed to establish a bipartite matching between registered targets and fresh measurements. In the simplest case, there is only one set of registered targets corresponding to the latest time stamp. The association procedure addresses several detector deficiencies. The deficiencies include slight imperfections such as regular measurement noise, as well as more serious discrete shortcomings such as absent detection and clutter.

The performance of the association procedure is crucial for the tracking quality, a fact well established and reflected in existing evaluation approaches [5]. We seek to evaluate the association quality as well. At a more technical level, we propose adding query-oriented instrumentation to the association procedure. As a result, the instrumentation allows for a binary classification of the bipartite matching. Hence, we term this approach Classification via Instrumented Association (ClavIA).

3.1. Association Outcomes and Their Classification

A typical tracker algorithm maintains an identified set of target descriptions. When provided with a current set of detections, the tracker propagates the targets according to their kinematic characteristics and starts a detection–target matching. There are several approaches and algorithms to realize such matching. Regardless of the specific approach, the tracker must provide a consistent identification of the targets immediately upon the arrival of the new detections. This requirement obligates any association algorithm to precisely link the detection reports and targets at every time step. After the correspondence of the detection reports and targets is established, an update step finalizes the tracking procedure until the next time stamp. The update step is important for the tracking quality as such. However, we should focus on the possible outcomes of the preceding association procedure to design the evaluation of its qualities.

Ideally, the association finds an accurate correspondence between each target and report. However, this is frequently unachievable because the number of detected objects varies across time steps. If there are more detection reports than targets, the association should mark new reports as unmatched and initialize corresponding targets after the update step. This occurs when new targets enter the field of view or when the detector malfunctions, producing spurious reports (clutter). Conversely, if there are fewer detection reports than targets, then the association should leave the appropriate targets unmatched while correctly associating the remaining targets. This arises when targets exit the field of view or when the detector misses objects. To allow a sufficiently complete classification of the association outcomes, we must distinguish between accurate (true positive and negative) and inaccurate (false positive and negative) associations, explicitly accounting for cases where an ideal outcome is impossible. For example, when an object leaves the field of view, the detector yields no reports; consequently, the tracker should keep the corresponding target dissociated (a true negative) until the target is terminated.

3.2. Instrumenting Association

The number of association outcomes is readily countable and classifiable, provided that the detections are appropriately identified, the tracker is suitably instrumented, and the classification procedure has sufficient information on the input and output of the tracker.

We propose to annotate true objects with unique non-negative IDs. To model detectors more realistically, we consider missing detections and clutter. We simulate missing detections by excluding some of the reports from the tracker input. This could be done randomly starting from an ideal ground truth. Beyond detector failures, the objects could leave the field of view. Both situations should ideally produce the same effect at the current step: an unmatched target. However, in the event of a detector failure, the detection reappears at the next time step and the tracker should correctly match this detection with the maintained target. Conversely, when an object exits the scene, the target should be deleted from the tracker output after a predefined number of steps. To ensure a correct classification, the detection IDs should not be reused when objects leave the tracking scene.

We represent clutter reports by assigning them a detection ID of $-1$. Multiple detection reports may be annotated as clutter all sharing the $-1$ detection ID. These clutter reports can be generated randomly. In this work, we consider only transient clutter. Transient clutter manifests as spurious reports that appear for a single time step and vanish in the subsequent one. Because our focus is on non-persistent clutter, it is sufficient to assign a single annotation ID of $-1$ to all clutter reports. Furthermore, even in scenarios involving persistent clutter, this inherent ambiguity would not lead to misclassification, as the annotation ID remains constant.

By design, all detection IDs are strictly greater than $-2$. These IDs are transmitted to the tracker via detection reports alongside the state measurements. The detection IDs are used to instantiate targets, at which point they become annotation IDs. Neither the detection nor the annotation IDs participate in the association procedure itself; however, the annotation IDs are essential for subsequent classification.

The instrumentation of the tracker includes processing the detection IDs. Upon the creation of a new target, it is assigned an annotation ID equal to the corresponding detection ID. This annotation ID is strictly preserved throughout the target’s lifetime. After each association, the target is assigned an update ID. If a target is associated, then its update ID is set to the ID of the associated detection. Ideally, the update ID is equal to the annotation ID, but this is not always guaranteed. If a target is not associated, then its update ID is assigned a special unmatched value. We denote the unmatched value by L to prevent misinterpretation as a standard update ID. By convention, we use $L=-9999$ to indicate unmatched targets. When a target is first instantiated in the tracker, the update ID is initialized to the annotation ID. The annotation and update IDs are instrumentation-only identifiers and do not interfere with the tracker’s native object identification mechanisms. We depict the data flow in Figure 1.

Once the targets are extended with annotation and update IDs, the classification procedure processes the output of the instrumented tracker on a target-by-target basis. Furthermore, since detections may be missed, the classifier must be informed about the supply of each annotation ID. This supply information is readily available because all detection IDs provided to the tracker are known. The supply condition is defined as $\mathrm{Supply}=\mathrm{Ann}\mathrm{ID}\in \left\{\mathrm{Det}\mathrm{IDs}\right\}$.

The classification procedure analyzes the annotation and update IDs stored in targets according to Table 1. The table summarizes the combinations of the supply condition, annotation IDs, and update IDs. Any conditions not explicitly mentioned in the table, as well as Case 5, are impossible by construction and will cause the classification procedure to abort. The cases listed in the table represent an exhaustive set of conditions necessary for the unequivocal classification of the association outcomes. Below, we discuss the cases in Table 1 individually.

The first case represents the only true-positive (TP) outcome. In this scenario, the annotation ID is non-negative, indicating a real object as opposed to clutter. The update ID is equal to the annotation ID, confirming an accurate association outcome. Finally, the annotation ID must be present in the set of current detection IDs to satisfy the supply condition.

The second case is one of the three false-negative (FN) outcomes. Indeed, although the annotation ID is supplied to the tracker and is non-negative (i.e. real object), the update ID differs from the annotation ID. This discrepancy indicates an incorrect association; specifically, the target is erroneously matched with a detection belonging to a different real object.

Cases 3 and 4 are similar to Case 2. However, unlike the second case, the association erroneously matches the target with a clutter detection (Case 3) or fails to provide any match (Case 4). Cases 2, 3, and 4 are all classified as false negatives (FN) because the association procedure produces a faulty result despite the availability of a correct match.

Cases 5 through 8 represent scenarios involving missing detections. In these instances, the corresponding annotation ID is not supplied to the tracker, despite representing a real target. Case 5 describes an outcome that is impossible by construction: the update ID equals the annotation ID even though that ID was absent from the supplied detections. This case is included in the table primarily to maintain the sequential order of conditions within the Update-ID column.

Cases 9 through 12 represent scenarios involving clutter detections, indicated by the clutter annotation IDs ($-1$). Since individual clutter detections are not differentiated, monitoring their supply is unnecessary. Case 9 occurs when a clutter detection is provided to the tracker, but the association procedure correctly avoids matching it with any tracked targets. Case 10 represents an erroneous association between a clutter detection and a real-object target. While Case 11 is functionally identical to Case 9, it is listed separately to maintain the sequential order of the Update-ID column. Finally, Case 12 occurs when a clutter detection remains unassociated due to the correct execution of the association procedure.

In this section, we have clearly defined all components of the confusion matrix, including the traditionally challenging identification of true-negative outcomes. Within the ClavIA framework, we overcome this limitation by defining TNs exhaustively and unequivocally, as detailed in Table 1. This rigorous classification scheme ensures that no tracking state remains unaccounted for within our evaluation framework.

3.3. Limitations of ClavIA

We emphasize that ClavIA is not designed to supersede existing evaluation frameworks, such as ClearMOT, HOTA, or T-GOSPA [2,4,5]. The fundamental limitation of this approach lies in its dependence on identified detections. Although easily acquired during developmental evaluation, these detections are inherently unavailable in production scenarios, rendering the requirement restrictive. Furthermore, ClavIA necessitates the use of specially instrumented trackers. Consequently, this methodology is unsuitable for competitive tracking challenges, which typically mandate blind evaluations with black-box tracker implementations. Nonetheless, the inherent benefits of ClavIA make it a practical tool for the broader research community.

4. Thought Scenarios of the Instrumented Tracking

Before discussing ClavIA’s implementation in Section 5, we analyze several toy examples to build a deeper intuition for our classification framework. Readers may skip this section without disrupting the paper’s flow.

We consider three simple tracking examples featuring identified detections and an instrumented tracker. Specifically, we examine toy scenarios involving two time steps and two objects reported at both steps. Furthermore, we assume an initially empty tracker (i.e., prior to the arrival of the first detection reports), thereby executing only a single association procedure per scenario. In addition to counting the classified targets, we compute standard binary quality metrics such as accuracy, precision, and recall [9].

4.1. Perfect Detection Sequence

In this example, the detection sequence contains neither clutter nor missing detections. Figure 2a illustrates the outcome at the initial time step for the majority of these scenarios. By construction, the positions of the detections and targets remain identical following the first time step. If all detections are correctly associated with their corresponding targets, all targets are classified as true positives, resulting in ideal accuracy, precision, and recall values of 1. This scenario is illustrated in Figure 2b. Because the prediction step in a tracker is seldom perfect, the detections and targets are depicted with a slight spatial offset for visual clarity.

If an association error occurs, it becomes evident at the second time step. For example, if a detection remains unassociated, the tracker generates a third target for that time step, resulting in five total targets across the scenario. The newly initialized target, spawned from the unassociated detection, is classified as a true-positive (TP) outcome. However, the unmatched target is assigned an unassociated update ID (represented by ’L’ in the figures) and categorized as a false negative (Case 4 in Table 1). In this scenario, the accuracy and recall both reduce to $4/5$, while the precision remains 1. This outcome is illustrated in Figure 2c.

If both detections fail to associate during the second time step, the tracker generates four targets, resulting in a total of six targets for the scenario. This outcome yields a reduced accuracy and recall of $4/6$, while maintaining a precision of 1. This scenario is illustrated in Figure 2d.

Another plausible outcome in two-object tracking occurs when the targets are erroneously associated with each other’s detections (an identity switch). In this scenario, the tracker produces two targets at the second time step, both of which are categorized as false negatives (Case 2 in Table 1). Consequently, the accuracy and recall both decrease to $1/2$, while precision remains 1. This scenario is illustrated in Figure 2e.

Finally, a scenario may occur where one target remains unassociated while the other is erroneously associated with the incorrect detection. In this instance, the tracker generates three targets at the second time step. This results in two false-negative associations, categorized under Cases 2 and 4 in Table 1, respectively. Consequently, the accuracy and recall decrease to $3/5$, while the precision remains 1. This scenario is illustrated in Figure 2f.

4.2. Sequences with Missing Detections

Consider a scenario where one of the real objects remains undetected at the initial time step but is subsequently reported at the second time step. The system state at the first time step is depicted in Figure 3a. Consequently, the tracker maintains only a single target upon the arrival of the second set of detections. If this target is correctly associated at the second time step, the procedure yields 3 true-positive outcomes out of 3 total targets. This ideal scenario is illustrated in Figure 3b.

It remains possible for the target to become unassociated. In this scenario, the classification yields one false-negative (FN) and two true-positive (TP) outcomes, resulting in an accuracy and recall of ${\displaystyle \frac{2}{3}}$ while maintaining a precision of 1. This association failure is illustrated in Figure 3c.

In another scenario, when two real-object reports arrive at the initial time step (see Figure 2a), one of the detections may remain undetected at the second time step. In this situation, the tracker yields a total of four correct classifications only if Case 8 (TN) from Table 1 is satisfied. This ideal outcome is illustrated in Figure 4a.

If the tracker erroneously associates the single detection, the outcome yields two true positives at the initial time step and two false negatives at the second time step, resulting in a reduced recall of $1/2$. This association failure is illustrated in Figure 4b.

Finally, if the tracker fails to associate either target, an additional true positive is generated at the second time step because the tracker is required to initialize a new target, resulting in a recall of $3/4$. This outcome is depicted in Figure 4c.

4.3. Detection Sequence with Clutter

Consider a scenario where, in addition to two real-object detections, a clutter detection appears at the initial time step and is absent at the second time step. Consequently, the tracker receives three detection reports at the first time step and two at the second. The system state at the initial time step is illustrated in Figure 5a.

Achieving perfect accuracy in this scenario requires satisfying Cases 9 and 12 from Table 1. At the initial time step, all targets yield correct classifications (a TP via Case 1 and a TN via Case 9). At the second time step, if both real-object reports are correctly matched and the clutter target remains unassociated, Case 12 dictates a correct true-negative (TN) classification. This yields 4 true positives and 2 true negatives, resulting in ideal values of 1 for accuracy, precision, and recall. This optimal outcome is illustrated in Figure 5b.

Any mismatched or unassociated targets will result in reduced quality metrics. For example, if the clutter target is erroneously associated with one of the two detection reports, Case 10 (FP) from Table 1 occurs. In this scenario, if the remaining detection report is correctly matched with its target (Case 1, TP), the remaining target inevitably remains unassociated (Case 4, FN). Consequently, this scenario encompasses all binary classification types among the reported targets: two TPs and one TN from the initial time step, alongside one TP, one FN, and one FP from the second time step. This classification yields an accuracy of ${\displaystyle \frac{4}{6}}$, with precision and recall both equal to ${\displaystyle \frac{3}{4}}$. This complex outcome is illustrated in Figure 5c.

If the clutter target is erroneously associated with one of the two detection reports, while a real-object target is incorrectly associated with the remaining detection report, no correct target associations occur at the second time step (yielding two FNs and one FP). This scenario results in an accuracy of ${\displaystyle \frac{3}{6}}$, a precision of ${\displaystyle \frac{2}{3}}$, and a recall of ${\displaystyle \frac{2}{4}}$. This worst-case outcome is illustrated in Figure 5d.

Two additional scenarios are depicted in Figure 5e and Figure 5f. In both instances, the clutter target yields a true-negative (TN) outcome, while the real-object targets experience association failures.

5. ClavIA in Open-Source Trackers

In practice, ClavIA implies modifications of the evaluated tracker. These modifications are strictly observational in nature; they are designed to probe the association result as non-invasively as possible, ensuring no interference with the underlying logic.

To demonstrate the instrumentation required by ClavIA, we surveyed several open-source trackers. The proposed framework is designed to be broadly applicable to both simple heuristic trackers, such as SORT [10], EagerMOT [11], and MCTrack [12], and to complex architectures like DeepMOT [13] and recent transformer-based models [14]. Ultimately, we selected AB3DMOT [15] for our experimental validation. This decision was driven by its architectural simplicity, its inclusion of a built-in ClearMOT evaluation module, and its established use with the widely adopted nuScenes dataset [7].

We integrated ClavIA into two open-source trackers: pure-ab-3d-mot [16] and kinematic-tracker [17]. Both trackers rely on Kalman filters and implement similar algorithms. The kinematic-tracker is designed to deliver a computationally efficient and generalized solution. In contrast, pure-ab-3d-mot was specifically adapted to demonstrate the instrumentation-related modifications within a well-established software solution. Consequently, we focus on pure-ab-3d-mot for the sake of clarity.

pure-ab-3d-mot is a simplified and refactored version of the standard 3D multiple-object tracking baseline (AB3DMOT). AB3DMOT is used as a reference tracker [15] for the nuScenes dataset [7]. The simplification involves the removal of the center-of-motion (COM) compensation module; this component does not directly affect target association, but significantly increases deployment complexity. Additionally, several structural refactoring changes were implemented within pure-ab-3d-mot. These modifications improve code clarity and maintainability without altering the underlying tracking results.

The ClavIA-specific modifications were introduced after the refactoring. We kept a protected branch without-clavia to facilitate a complete view of the instrumentation changes. Here, we briefly summarize the instrumentation changes. The Target class is extended to hold the attributes ann_id and upd_id. Within the Ab3dMot tracker class, new Target instances are created with their annotation IDs assigned in the Ab3dMot.birth() method. Furthermore, the update ID (upd_id) is updated within the Ab3dMot.update() method. In addition to these core extensions, minor tracker-specific adjustments were implemented to facilitate data transfer.

The scripts to run the tracker pure-ab-3d-mot and evaluate its output quality are consolidated within a separate package, eval-ab-3d-mot [18]. We designed the package eval-ab-3d-mot to analyze the tracker pure-ab-3d-mot according to both the ClearMOT and ClavIA methodologies on the KITTI dataset. The ClearMOT evaluation module is a refactored version from the original AB3DMOT repository [15]. Meanwhile, the ClavIA evaluation logic is encapsulated within a standalone, lightweight package, association-quality-clavia [19], to facilitate its integration into external projects.

6. Results

In order to demonstrate the utility of the proposed approach, we ran the pure-ab-3d-mot tracker on the training split of the KITTI tracking benchmark [6]. We utilized the package eval-ab-3d-mot [18] to conduct all computational experiments discussed in this work. The training subset consists of 21 scenes. In accordance with the KITTI challenge guidelines, objects within each scene are classified into three categories: car, cyclist and pedestrian. Several tracker parameters vary depending on the object category: the matching algorithm and threshold, the association metric, the minimum number of detections required for confirmation min_hits, and the maximum number of steps without a match max_age. Conversely, the Kalman-filter covariances remain constant across all categories. Keeping the covariances the same aligns well with our goals by reducing the dimensionality of the optimization space. Instead of optimizing these parameters, we intentionally apply minor perturbations to the measurement covariance in a stability study detailed in Section 6.3.

We compare ClearMOT and ClavIA evaluation methods. Although sharing a common objective, these methodologies are defined by distinct specifications and exhibit different limitations. Discrepancies make it impossible to reproduce the same numbers even in the best case scenario. The primary distinction between the two approaches lies in their input requirements. In ClearMOT, the detections remain unidentified as in real-world execution. In ClavIA, the detections must be identified. Creating anonymous detections from annotations is trivial, but doing the opposite is next to impossible.

To account for this discrepancy, we evaluate ClearMOT in two modes: first, using real detections as originally intended; and second, using ground-truth annotations with identity labels removed. The first mode enables a comparison of the methods under realistic operating conditions, without expecting strict quantitative agreement. The second configuration aims to provide a closer quantitative comparison by reducing differences in the input data. Despite the substantial methodological differences between ClearMOT and ClavIA, the qualitative conclusions obtained from both evaluation protocols remain consistent.

We opted not to compare our method against the more recent HOTA evaluation metric [5] due to practical considerations. While HOTA is largely an extension of ClearMOT, its official implementation is restricted to 2D tracking. Although ClavIA can evaluate both 2D and 3D tracking with equal ease, we chose to focus our experiments on the 3D domain for two primary reasons. Firstly, 3D tracking is arguably more important in the mobility context. Second, AB3DMOT is difficult to adapt for 2D tracking, and we want to avoid the discussion of yet another tracker.

6.1. Input by Design: Consuming Detections in ClearMOT

In the case of ClearMOT evaluator, the tracker is provided with the recommended R-CNN detections [20], as was ClearMOT designed to operate. In the case of ClavIA, the tracker processes the KITTI annotations.

The computed F1 scores are presented in Table 2. Despite the differences between ClearMOT and ClavIA, the F1 scores exhibit a similar trend: performance is highest for the car category and lowest for the pedestrian category. The scores are generally higher for ClavIA due to the higher quality of annotations compared to detections.

Reviewing the optimal parameters established in the AB3DMOT project [15], the matching algorithm is set to Hungarian for the car and cyclist categories, whereas a greedy algorithm is utilized for pedestrians. This inconsistency is unexpected and may contribute to the suboptimal performance observed in the pedestrian category. To evaluate the optimality of this configuration, we evaluated the tracker’s F1 scores across varying association thresholds. The results, obtained with instrumented association, are presented in Table 3. The data demonstrates that the optimal association threshold and matching algorithm are ($-0.5$, Hungarian), as opposed to the ($-0.4$, greedy) configuration recommended in the AB3DMOT project.

To further verify this result, we evaluated the tracker using ClearMOT with the optimal parameters derived from ClavIA ($-0.5$, Hungarian) and observed an improvement in the metrics (e.g., an F1 score of 0.8408 compared to 0.8292 with the recommended parameters). In contrast to ClavIA, ClearMOT performance peaks at a threshold of $-0.6$, yielding the highest F1 score of 0.8412 among the considered thresholds for the Hungarian algorithm. Other threshold values exhibit similar trends, as demonstrated in Table 4. Indeed, the use of greedy matching is ultimately justified, as the maximum overall F1 score is achieved using the greedy algorithm at a threshold of $-0.5$.

6.2. Ingesting Annotations in ClearMOT

In order to exclude the effect of detector quality, we performed a ClearMOT evaluation of the AB3DMOT tracker utilizing annotations. This comparison is closest to the ClavIA methodology because the tracker receives the same input. We present the ClearMOT F1 scores obtained using annotations in Table 5. As expected, the values are much closer to the corresponding ClavIA results (Table 3). Similar to ClavIA, the annotation-based ClearMOT scores demonstrate a slight advantage of the Hungarian matching algorithm over the greedy algorithm. Indeed, greedy matching may outperform the Hungarian approach when detection quality is poor.

Two primary differences remain. First, ClearMOT uses a probabilistic association procedure prone to numerical instability, whereas ClavIA establishes associations unequivocally using discrete IDs. Second, concerning target processing, ClavIA analyzes targets directly within the tracker, whereas ClearMOT uses the tracker’s output, which is withheld pending target confirmation.

In summary, while ClavIA is not intended as a direct replacement for ClearMOT, our computational experiments demonstrate that both frameworks yield qualitatively comparable performance patterns.

6.3. Sensitivity Analysis of F1 Scores

In this section, we analyze the stability of the previously reported F1 scores. To evaluate the sensitivity of the evaluation, we introduce minor perturbations to the measurement-noise covariance within the Kalman filters, generating slight variations in the target-state time series. These variations affect the scores, allowing us to assess the robustness of the tracking evaluation.

In the original AB-3D-MOT tracker, the measurement covariance is fixed to an identity matrix across all object categories [15]. We maintain a diagonal measurement covariance but scale it by factors of $0.9801$ and $1.0201$. These scaling factors correspond to a 1% decrease and increase in the standard deviation, respectively. For the remaining tracking parameters, we adopt the category-specific optimal values established in the original publication [15].

The resulting differences in F1 scores are summarized in Table 6. No variations are observed in the cyclist category, whereas for cars and pedestrians, the differences manifest only in the fourth or fifth decimal place. The small magnitude of these variations indicates the stability of both the ClavIA and ClearMOT evaluation methods.

7. Future Directions for Instrumented Association

In Section 6, we demonstrated the baseline application of ClavIA by supplying the instrumented tracker directly with annotations. However, utilizing annotations directly leaves several real-world scenarios untested. Specifically, this approach omits evaluations of clutter, missing detections, and varying detector-noise levels. To rigorously evaluate a tracker’s robustness against these detector shortcomings, one could systematically generate non-ideal, yet identified, synthetic detections.

We outline two possibilities for generating identified detections of controllable quality.

The first possibility involves starting with ground-truth annotations and introducing detector shortcomings to them. Indeed, random deviations can be easily added to the coordinates, sizes, and orientations of the annotated cuboids. Similarly, missing detections can be generated by randomly dropping specific annotations. Finally, it is straightforward to randomly generate cuboids within the field of view and assign them an annotation ID of $-1$.

A second approach to rigorously testing trackers is to generate ground-truth annotations using an analytical model and then randomly introduce simulated detector errors. Compared to real-world annotations, this method is advantageous because time derivatives are known precisely to any order. This is particularly useful for thoroughly assessing the precision of the tracker output [12] when derived variables, such as velocities and accelerations, are of interest. However, the primary challenge of this approach lies in generating analytical models that accurately reflect relevant end-use scenarios.

In addition to its role in evaluation, ClavIA instrumentation could be useful in other applications. Within the CAPACITY project [21], we used the ClavIA-enabled tracker [17] in a semi-automatic annotation tool. In this application, ClavIA verified the annotation quality by providing information on the established target-annotation matches.

8. Conclusion

We presented ClavIA, a method for evaluating tracker association quality. The proposed approach relies on a targeted modification of the tracking algorithm, enabling direct inspection of the association procedure’s results. Ultimately, this allows for a binary classification of the tracker output. The simplicity of the proposed method makes it widely applicable across a range of tracking applications, with the optimization of tracker parameters being the most straightforward use case. To demonstrate its practical utility, we instrumented the well-known AB3DMOT tracker—the baseline for the nuScenes dataset—and evaluated its performance on the KITTI dataset.