Experimental Evaluation of a MIMO Radar Performance for ADAS Application

1. Introduction

In detection systems for autonomous driving, radar is considered an essential element, along with cameras and, potentially, lidar [1,2,3,4,5,6,7,8]. The role of radar is relevant in providing an estimate of the distance to objects captured by the cameras, and also for detecting other vehicles or obstacles in adverse weather situations, such as heavy rain or fog. In these situations, cameras and lidar lose much or all their effectiveness. When it comes to providing depth to images, lidar-type sensors are even more effective than radar, but at the cost of a much higher price, greater size and mechanical complexity, and a significant computational cost. In this work we have tried to experimentally estimate the value of some parameters of interest related to radar performance, such as the effective false alarm rate, typical distance error, maximum errors and the number of received echoes that can be expected per unit of time.

We have used a Texas Instruments (TI) AWR1843 radar, which is a Multiple Input Multiple Output (MIMO) Frequency Modulated Continuous Wave (FMCW) radar designed to work in the 77 to 81 GHz band. It consists of three transmitting and four receiving antennas. The processing of the signals, after mixing with in-phase and quadrature signals carried out by the device itself, has been carried out off-line in a standard computer, after transferring the raw signals, using a DCA1000EVM card, also from TI. We have used a single transmitting antenna for the measurements, so in this case it has been operated as SIMO, single input multiple output. This mode of operation allows position information to be obtained in the horizontal plane; nevertheless, target height is not available. The signal processing and visualization program was developed in MATLAB. The radar has been operated with parameters optimized for short range, up to 25 meters away in our measurements. The scenarios include pedestrians and motorcycles, and static targets have been excluded from the analysis. Some other works may be found in the literature that use the same model or others commercial MIMO radars [9,10,11,12,13,14,15,16,17,18], always for short distances, although they do not provide enough measurements to have a complete idea of the performance of those models.

2. Materials and Methods

In each scenario, the targets that carry a certain speed (radial with respect to the radar) have been analyzed separately, to improve target separation and minimize background noise. That is, the speed estimation, obtained thanks to the Doppler shift of the signals, is carried out with the sole purpose of selecting particular targets, pedestrians [19,20] or motorcycles in our case. This way of proceeding is the natural one after performing a two-dimensional FFT of the received signals. It is the standard method of analysis, as shown in the bibliography provided by TI and in other references [21,22,23,24].

The emitted signal is made up of FMCW signals, where the frequency increases linearly over time in each chirp. The slope selected for the temporal frequency variation was $S=20MHz/\mu s$. The sampling frequency in all cases has been $f_s=12MHz$ and a number of samples were taken per chirp $N_{sc}=256$. Signals are processed in blocks, called frames, with $N_{cf}=128$ or $N_{cf}=256$ chirps per frame in this work. The chirps are separated by a certain interval, which can be modified [25].

When the $N_{sc}$ samples of the $N_{cf}$ chirps are received, that is, a frame, the samples are ordered by rows in a two-dimensional matrix $N_{sc}\times N_{cf}$. A 2D-FFT is performed and the range-doppler map (or range-velocity map) is obtained. From it you can select the columns of interest (targets with a certain speed) and the different targets present with that velocity. In our case we have analyzed the targets detected at all present speeds, except zero speed, that is, static targets with respect to the radar are ignored. This was done because the measurements were carried out in a parking lot with parked vehicles, which were of no interest to the study.

The selection of received echoes, both in speed and distance, has been carried out by applying a CA-CFAR (cell averaging constant false alarm rate) algorithm [26,27,28,29,30]. In the selection of speeds, an undemanding value has been set, with a probability of false alarm $PF{A}_{vel}=0.2$, or $0.3$ in order not to lose echoes, while in distance the results obtained for different speed values have been evaluated, with $PF{A}_{range}\in (0.1-0.001)$.

The parameters of interest were: the number of false targets in each scenario, which gives us a measurement of the final effective false alarm probability ($PF{A}_{eff}$), the error in estimating the distance of the targets, the maximum errors in distance and the number of useful echoes obtained per second.

The average error in distance, as well as the maximum errors, have been evaluated by measuring the error in each echo considered good by the algorithm. To do this, it is not enough to evaluate the error of the echoes with respect to the final trajectory traced by the target, but it is necessary to track the position of each echo and compare it with the estimated instantaneous position of the object. In our case we did not have a more accurate system capable of providing the instantaneous trajectory of the targets, so an estimate of the likely instantaneous trajectories has been built from the radar echoes themselves. The moment in which the echo appearing in the screen is assigned to a particular target is, sometimes, a critical one, because, in reality, you cannot be certain that this echo really comes from that target. This fact adds a point of uncertainty in the process, which, in any case, is inevitable. The recorded videos of the different scenes are used to visually validate the movement of the targets and to estimate the temporal duration of the measurements.

3. Results

3.1. Effective probablity of false alarm

In the first scenario there are three pedestrians moving in different directions and at different distances in front of the radar. All of them describe straight trajectories at approximately constant speeds. The captured scenes were also taken on video. To evaluate the error made in the processing of the different echoes, a most probable instantaneous trajectory is estimated with a constant speed, or constant in sections, for the target. The echoes are inspected one by one, selecting those that most likely come from the target under study. In this type of analysis there is always an intrinsic uncertainty since it is not possible to know if an echo comes from the target that is being specifically tracked, from a nearby target or if it is a false target generated by signal noise. Faced with this, the only practical procedure is to track with an appreciable number of echoes along the probable instantaneous trajectory, as we have tried to do in this study.

Figure 1 shows the trajectory of the three targets in this first scenario, with all the echoes selected by the analysis program. As stated, the selection of speeds was carried out with a value $PF{A}_{vel}=0.2$, while the result in the figure is the one corresponding to the value $PF{A}_{range}=0.05$. Radar is located at the origin of coordinates.

Figure shows the echoes that most likely belong to one of the targets, located along the path of each pedestrian. There are also false targets, some of them generated by noise peaks or coming from echoes affected by noise, so that their position has been incorrectly estimated. Some of the echoes would fall into the doubtful category, as those framed with a solid blue line, since the observer or a subsequent echo evaluation system could consider as valid, assigning them to a particular pedestrian. It is worth noting that to perform the task of classifying an echo as valid, doubtful or clearly erroneous, it is not enough to observe the final echo map shown in the figure. To make this classification it is necessary to track it in time during the course of the experiment, so that the position of the received echoes can be compared with the real or estimated instantaneous position of each real target.

The second scenario was composed by two pedestrians and a motorbike, describing straight paths and also the return of all three at their initial positions. As in the first case, the raw data obtained from the radar were analyzed again using $PF{A}_{vel}=0.2$, for a first selection of targets, in velocity, and with several values, increasingly small, for the parameter $PF{A}_{range}$. Figure 2 show the global set of echoes received from the radar during this experiment.

In the particular map of Figure 2 a larger number of echoes, valid and false, can be seen, in comparison with the first scene. This is because a less demanding PFA has been used, but also because the scene duration is longer.

Finally, the third scene is a motorbike describing rounds in front of the radar, with an approximate constant speed. The set of echoes received are shown in Figure 3.

We observed a concentration of false targets near the radar, in a position sixty degrees to the left, as is marked with a dashed contour in the figure. This phenomenon of concentration of echoes where there is not a real object did not happen in the other two experiments, and we do not know why they appear. It could be related with some kind of double reflection of the signal or, perhaps, with the reflection in the wheels, that can introduce an overlap with their own Doppler shift.

The first analysis carried out in the experiments was to evaluate the number of false targets detected, based on the PFA value used in the detection of targets for each of the selected speeds. That result is shown in Figure 4. It has been defined

$$PF{A}_{eff}=\frac{nºoffalseechoes}{nºofreceivedechoes}.$$

(1)

In this calculation, the class of doubtful echoes has been omitted, assigning them to the set of valid echoes or false echoes in each case.

The choice of a higher or a lower value for the PFA parameter depends on the necessity of having an enough number of echoes: so, in situations where the signal to noise ratio is low, even admitting an important number of false echoes, the choice could be $PF{A}_{range}~0.1$. The alternative is minimizing that number of false targets, when the signal to noise ratio is not bad. In this situation a value $PF{A}_{range}~0.01$ would be more appropriate. In our case, and having enough echoes received per second, avoiding, as much as possible, the number of false echoes seems the best strategy. The behavior of the echoes in the three scenarios is slightly different. In scenarios 1 and 2, the number of false targets is lower than in the other, and with a value of $PF{A}_{range}=0.01$ or $PF{A}_{range}=0.005$, the detection of targets is quite clean. In scenario 3, however, there are a significant larger number of false targets, and a value as low as $PF{A}_{range}={10}^{-3}$ should be used to sufficiently filter out false alarms. It must be considered, however, that an unnecessarily small value of this parameter would also cause a significant loss of true echoes. On the other hand, we have fixed most of time the other variable threshold, in the velocity axis, with $PF{A}_{vel}=0.2$, but it must be said that this value could be also relaxed.

3.2. Mean error in range

Each of the pedestrians in the first scene was also followed separately, calculating the average error of the position of the echoes that, with high probability, are coming from the target under study. As explained, the echoes that come from the target (or pedestrian) under test have to be selected manually, and comparing the position of those echoes with the corresponding points on the estimated instantaneous trajectory of each target.

In Figure 5 the obtained mean error in range is shown for the three pedestrians. The basic limit of the system spatial resolution is given by the known expression

$$∆r=\frac{c}{2S{T}_c},$$

(2)

S being the slope of the frequency temporal variation, c the speed of light and $T_c$ the duration of a chirp. In our case we have $S=20MHz/\mu s$ and $T_c=N_{pc}/f_s=21.33\mu s.$And it results $∆r=0.35m$.

As it can be seen in Figure 5 the mean error in the localization of the targets does not depend critically of the selected value for the $PF{A}_{range}$ , although is clearly better for low values. This behavior comes from the fact that the targets have been detected correctly most of the time. However, on some occasions, echoes affected by noise or false targets have been considered good. This can be seen in Figure 6, where the error in the localization of one of the pedestrians has been represented as a function of the distance to the radar, and for two cases.

There are two appreciable differences between the two cases shown in the figure. For the higher value of $PF{A}_{range}$ a quite large number of echoes has been accepted by the system/observer as coming from the target, including some of them with an important error (~2.5m). For the low value of that parameter, however, the number of valid echoes is much lower, as is the maximum error (<1m). It must be remembered, however, that the selection of the 'correct' echoes has been carried out visually, considering what the temporal processing of the signals was showing on the screen. This adds a point of arbitrariness that, on the other hand, cannot be avoided. Nor if the task is performed by an algorithm or an artificial intelligence application, because the system does not know previously the number, nature, trajectory or speed of the targets.

The evaluation of errors has been made also for the two paths of the motorbike in scenario 2, as shown in Figure 7. As the motorbike has an important horizontal dimension, the position mean error is higher than in the case of the pedestrians, although, on the other hand, in this particular experiment the radar never has a complete vision of that horizontal dimension of the vehicle, due to the angle of the paths with respect to the radar position (see Figure 2).

To end this section, in Figure 8 the range error measured during the evolution of the motorbike in scenario 3 is shown. Actually, the trajectory with which the position of the echoes is being compared has been deduced from the echo map itself, averaging the position of echoes as they appear in time, in order to obtain a smooth and continuous path. It could be seen that important errors are reported in some specific positions of the motorbike, which correspond to the moments when the motorcycle is traveling perpendicular to the line of sight of the radar (see Figure 3). As the length of the motorcycle is somewhat more of 2m, the echoes can come from different parts of the vehicle, while the errors are merely calculated with respect to the position of the hypothetical center of the vehicle. Of course, then, errors shown in figure are only a coarse approximation, and most of the echoes come certainly for one or another part of the target.

3.2. Number of echoes per unit of time

Finally, we have performed a count of the useful echoes available from some of the targets. The most significant point is the number of valid echoes diminishes strongly with a low value of the parameter $PF{A}_{vel}$, as can be seen comparing the two plots in Figure 9. If the requirements of the system are about 25 echoes per second (from each target), this parameter, $PF{A}_{vel}$, will have to be relaxed consequently. As it has been said before, a too low value of PFAs provokes not only the decrease of false echoes but also of the good ones.

As it can be noted in the figure, the number of good echoes from different targets follows a different behavior. That is due to the different trajectories of the pedestrians. The distance of pedestrian M with respect to the radar has a low variation along time, so that the number of received echoes from her suffers a variation only for the lowest value of PFA. Pedestrian A is progressively approaching the radar, and most of the echoes produced have a very good signal-to-noise ratio. As a consequence, the decreasing of the PFA value has little effect. On the contrary, pedestrian F is in a worst situation to be detected: at the beginning of his path the distance to the radar is larger than for the other two targets, and, at the final state, is near to go out of the field of view of the radar. So, the echoes produced by him are, generally speaking, of lower amplitude. Besides, the echoes of the farthest pedestrian are occasionally intercepted by the other two, when crossing the line of sight of the radar.

Also, worth noting is the fact that an appreciable number of frames sent by the radar have no response at all, or fewer responses than the number of real targets. This is an experimental verification observed in the three scenarios shown in this work.

In Figure 10 the time intervals to have into account in the configuration of the TI device are shown [25].

As it was explained above, each chirp was sampled with a frequency $f_s=12MHz$, up to 256 samples per chirp. So, we have the effective duration of the chirp $T_{acq}=256/12MHz=21.33\mu s.$ However the real period of the signal is not related with that value, as seen in the figure. The total period will be $T_c=T_{idle}+T_{ramp}$. In our experiments, we took $T_{idle}=100\mu s$ and $T_{ramp}=60\mu s$, as recommended values. (On the other hand, we set $T_{ADCst}=6\mu s$ and $T_{TXst}=0,$ which are the default values in the software pack mmWave Studio, provided by TI). With these values the expected number of frames per second is about 49 frames/s with 128 chirps per frame, and about 24 frames/s with 256 chirps per frame. This numbers include frames without response and frames provoking false echoes. It has to be said, however, that we are not sure if the radar takes some other period of time between frames, and the real expected number of frames emitted per second could be lower than these.

4. Discussion

The main idea of ​​the work was to evaluate the characteristics of the TI AWR1843 radar, in terms of the error committed in distance, the number of false echoes and the number of valid echoes per unit of time. Experiments were done with pedestrians and motorcycles, with the idea that they are the targets that could eventually go unnoticed by the radar.

In the analysis, a double adaptive threshold has been used, one on the speed axis and another on the distance axis, for each speed selected by the first. Both with the CA-CFAR method. The results show that it is a good strategy, and that it provides a level of intuitive control of the number of false echoes.

Initially the experiments were carried out setting the PFA in determining the threshold at speeds with a loose value of 0.2. It was later found that this value actually eliminates too many potentially valid echoes, and a higher value, $PF{A}_{vel}=0.3$, has been found to be more suitable for obtaining a higher echo rate. On the other hand, in our experiments, there is not appreciable difference by using $Ncf=256or128$, so $Ncf=128$ is a more economic value.

The maximum error of the radar in estimating the position of a target turns out to be a difficult parameter to quantify, since it depends critically on what criterion is used to assign an echo to a specific target.

The average error has turned out to be very acceptable, taking into account that interpolation techniques can still be included that, in this work, with the exception of zero-padding in the angle-FFT, have not been used.

Finally, we did not set out to estimate the maximum distance at which our radar can capture targets, although, in other measurements not presented in this work, pedestrians have been detected at about 35 meters and a car, in a single attempt, was perfectly detected at 80 meters.

5. Conclusions

We have carried out several experiments with a MIMO radar AWR1863 from Texas Instrument in realistic scenarios with pedestrian and motorcycles, in order to evaluate some critical points of its performance, as the number of false targets, the accuracy in the localization of the targets and the number of valid echoes received per second. These parameters are important for the usefulness of the radar itself and also for the use of the obtained data to make a fusion with the images captured by the cameras in a more complete ADAS system. We use a double CFAR filter, one of them in separating velocities and the other for separate targets in distance. We have used large values in the first of them, with $PF{A}_{velocity}=0.2-0.3$, and more restricted values for the second, with $PF{A}_{range}$ between 0.1 and 0.001. The results show that it exists a compromise in selecting the value for the two parameters, because a too small number diminish not only the false targets, but also the good ones. The accuracy in range is generally good, although a small value for $PF{A}_{range}$ helps to avoid disturbance in the measurements due to the false or poorly estimated targets. The mean error along the experiments is about 0.25 m for pedestrians and 0.5 m for the motorbike. The maximum error in the localization of targets cannot be estimated properly, because it is not possible to discriminate between false targets and positions erroneously calculated due to the present noise. Moreover, assigning a singular echo to a particular target depends on some chosen criterion, probably in a further stage in the complete detection algorithm, but always with a certain degree of arbitrariness.