Vision-Only Localization of Drones with Optimal Window Velocity Fusion

1. Introduction

The applications of unmanned aerial vehicles (UAVs) have expanded dramatically [1,2,3]. It is crucial to determine drones’ position to perform high-level tasks. Drones typically estimate their position through an integration of external sensors, such as global positioning system (GPS) and internal sensors such as inertial measurement units (IMUs) [4,5].

Accurate localization is very challenging where GPS signals are not available [5]. Without absolute position reference, drones must rely on alternative sensing methods to infer their positions, often requiring additional onboard sensing or external infrastructure [6,7]. One prominent solution to this problem is a visual-inertial odometry system that fuses data from an inertial measurement unit (IMU) and a camera [8,9,10].

Vision-based localization offers the advantages of low cost, light weight, passive sensing, and multi-functionality. One of vision-based approaches is template-based method which localize drones by matching the current image against a pre-stored template. Object motion can be tracked by matching a template between consecutive frames. The classical template matching is well-suited for estimating small translational displacements between frames when lighting is stable and there are no rotational or scale changes [11]. Normally, template selection has been scene or object dependent, thus incorrect template updates can lead to accumulated errors that degrade performance [12]. A robust illumination-invariant localization algorithm is proposed for UAV navigation [13]. Fourier-based image phase correlation was adopted to estimate absolute velocity estimation of UAV [14]. In [15], a multi-region scene matching-based method is proposed for automated navigation of UAV. Various matching techniques are surveyed for UAV navigation in [16]. Scene-independent template matching using optimal windows were proposed in [17].

This paper addresses vision-only localization of drones for short-range and high maneuvering flights. When the camera is tilted downward to the ground and located at a specific altitude, the pixel coordinates can be converted into real-world coordinates by a ray optics-based conversion function [18,19]. However, this image-to-position conversion generates non-uniform spacing in the real coordinate system. The optimal windows were proposed to overcome this non-uniform spacing in [17]. A piecewise linear regression (segment) model [20,21] determines the breakpoints of the optimal window. The piecewise linear regression model minimizes the total least-square errors over all linear segments that approximate the conversion function. In consequence, the optimal window divides a frame into several non-overlapping templates. Each template is matched between frames by minimizing the sum of normalized squared differences [22]. In [17], this technique combined with a state estimator achieves errors of several meters on very short-range flights.

In the paper, this optimal window template matching technique is significantly improved by the fusion of velocities. Two fusion rules are contrived considering the minimum detectable velocity as the velocity resolution: weighted averaging for the lateral velocity and winner-take-all decision for the longitudinal velocity. Additionally, a zero-order hold scheme was applied to reduce the high computational burden of template matching.

In the experiments, a multirotor drone (DJI Enterprise Advanced [23]) performed short-range flights (800 m to 2 km) with high maneuverings in rural and urban areas. In rural areas, four paths consists of a straight-forward-backward flight, a zigzag forward-backward flight (snake path), a squared path with three banked turns, and a free flight including both banked turns and zigzags. In urban areas, two paths are a straight outbound flight and a straight-forward-backward flight. No yaw-only rotation (flat turn) is included in the flight. The drone starts from a stable hovering state, accelerates, and flies as fast as possible while maintaining an altitude of 40 meters and tilting the camera at a 60-degree angle.

To evaluate performance, a ground-truth trajectory is generated from GPS signals. Since the ground-truth trajectory has a different coordinate system than the vision-only trajectory, a rigid body rotation is applied to the vision-only trajectory to align one trajectory with the other. The root mean square error (RMSE) and drift error are calculated at three intermediate points and one end point. It will be shown that the proposed method achieved an error range of a few meters to tens of meters after flight. Figure 1 shows a block diagram of the proposed method.

The contributions of this study are as follows: (1) The process of converting images into real-world coordinates has been improved. This allows for more precise analysis of velocity resolution. (2) Multiple velocities obtained from multiple template matching are fused based on velocity resolution. Two fusion rules are proposed for lateral and longitudinal velocities, respectively. (3) The robustness of the system was verified through high maneuvering flights. Furthermore, system performance was verified by comparing it with the ground-truth obtained from GPS signals.

The rest of this paper is organized as follows: the optimal window template matching are presented in Section 2. The velocity fusion rules and performance evaluation are described in Section 3. Section 4 demonstrates experimental results. Discussion and conclusions follow in Section 5 and Section 6, respectively.

2. Optimal Windows for Template Matching

This section describes how the optimal windows are derived from the improved image-to-position conversion.

2.1. Improved Image-to-Position Conversion

The image-to-position conversion [17,19] applies trigonometry to compute real-world coordinates from pixel coordinates when the camera’s angular field of view (AFOV), elevation, and tilt angles are known. It is assumed that the camera rotates only around the pitch axis and that the ground is flat. The improved conversion function is as follows

$$x_i\left(j\right)\approx \surd \mathrm{ }(h^2+y_j^2)·tan\left[\left(i-{\displaystyle \frac{W}{2}}+1\right){\displaystyle \frac{a_x}{W}}\right], i=0,\dots ,W-1,$$

(1)

$$y_j\approx h ·tan\left[{\theta }_T+\left({\displaystyle \frac{H}{2}}-j\right){\displaystyle \frac{a_y}{H}}\right] ,j=0,\dots ,H-1,$$

(2)

where W and H are the image sizes in horizontal and vertical directions, respectively, h is the altitude of the drone or the elevation of the camera, $a_x$ and $a_y$ are the AFOV in the horizontal and vertical directions, respectively, and ${\theta }_T$ is the tilt angle of the camera.

Figure 2a,b visualizes the coordinate conversion function in horizontal and vertical directions, respectively: W and H are set to 3840 and 2160 pixels, respectively; $a_x$ and $a_y$ are set to 68° and 42°, respectively; h is set to 40 m; ${\theta }_T$ is set to 60°. In Figure 2a, eight non-linear conversion functions in the horizontal direction are shown varying the vertical index j from 1 to 2101 by 300-pixel intervals. In [17], $\surd (h^2+y_j^2)$ was approximated by $\surd (h^2+y_{H/2}^2)$ which is represented by only one conversion function in the horizontal direction. Figure 2b shows that the nonlinearity increases rapidly in the vertical direction, resulting in non-uniform spacing in the real coordinate system.

2.2. Optimal Windows

The vertical conversion function in Figure 2b is approximated by a piecewise linear regression model. In this model, multiple linear segments approximate the nonlinear curve. Several linear segments minimize the sum of the least square errors of the separate linear regression as follows:

$${\widehat{s}}_1,\dots ,{\widehat{s}}_{N_w-1}={argmin}_{s_1,\dots ,s_{N_w-1}}\left[\sum _{n=0}^{N_w-1}\sum _{j=s_n}^{s_{n+1}-1}{{min}_{a_n,b_n}\left[y_j-\left(a_{n}j+b_n\right)\right]}^2\right],$$

(3)

where $s_1,\dots ,s_{N_w-1}$ are $N_w-1$ break points for $N_w$ segments, and $s_0$ and $s_{N_w}$ are equal to 0 and the image size in the vertical direction, respectively, and $a_n$ and $b_n$ are the coefficients of the n-th linear regression line [20,21]. It is noted that $N_w$ is equal to the number of windows. The number of windows can be determined heuristically; too many windows, equivalently too few sampling points (pixels) in one window, can lead to inaccurate matching, whereas too few windows cannot compensate for the uneven spacing of the nonlinear conversion function. In the experiments, the frame was cropped by 90 pixels near the edges to remove distortions that might occur during image capture, thus optimal windows tiled in an area of 3660 × 1800 pixels. The n-th window has the size of 3660 × $\left(s_{n+1}-s_n\right)$ pixels. Figure 3 shows three piecewise linear segment models when $N_w$=10, 12, 15. Figure 4 shows the corresponding optimal windows to Figure 3 in a sample frame. Total sum of squared errors which are minimized in Equation (3) are 55.3, 26.85, and 11.11.

The computational complexity of exhaustively searching all possible piecewise linear models grows exponentially with the number of segments. Dynamic programming [24,25] efficiently determines the optimal linear segments while significantly reducing computational burden. This approach first precomputes the least-squares fitting error for all candidate line segments. It then determines the optimal $n$segment solution when n = 2. During this step, all possible least partial sum of squared errors for two-segments are calculated at $j\in \left[\mathrm{0,2}\right],\left[\mathrm{0,3}\right],\dots ,\left[0,H-1\right]$. Then n is increased by one, the optimal solution for n segments are calculated using the least partial sum of squared errors obtained from the previous step. This process continues until n reaches $N_w$.

The minimum detectable velocities at pixel (i, j) are calculated as ${\displaystyle \frac{{|x}_{i+1}\left(j\right)-x_i\left(j\right)|}{T}}$ and ${\displaystyle \frac{{|y}_{j+1}-y_j|}{T}}$ in the horizontal and vertical directions, respectively. Figure 5 shows the minimum detectable velocity when T =1/30 sec and $N_w=12$. The j values in Figure 5a are the center of 12 optimal windows in the vertical direction in Figure 4b. The minimum detectable velocity increases rapidly in the vertical direction as shown in Figure 5b.

3. Velocity Fusion and Performance Evaluation

This section describes how to fuse the velocities of multiple optimal windows, and a zero-order hold scheme to reduce the matching counts. Performance evaluation using GPS signals is also presented at the end of the section.

3.1. Velocity Fusion

The minimum detectable velocity shown in Figure 5 represents the highest velocity resolution of each window. Assuming small window displacement between consecutive frames, the minimum detectable velocity can be considered the velocity resolution of that window. The weight on each window is calcualted based on its resolution as

$$w_{xn}={\displaystyle \frac{\frac{1}{{\sigma }_{xn}^2}}{{\sum }_{n=0}^{N_w-1}\frac{1}{{\sigma }_{xn}^2}}}={\displaystyle \frac{{\mathsf{\Delta }}_{xn}^2}{{\sum }_{n=0}^{N_w-1}{\mathsf{\Delta }}_{xn}^2}},$$

(4)

$$w_{yn}={\displaystyle \frac{\frac{1}{{\sigma }_{yn}^2}}{{\sum }_{n=0}^{N_w-1}\frac{1}{{\sigma }_{yn}^2}}}={\displaystyle \frac{{\mathsf{\Delta }}_{yn}^2}{{\sum }_{n=0}^{N_w-1}{\mathsf{\Delta }}_{yn}^2}},$$

(5)

where ${\mathit{\sigma }}_{\mathit{x}\mathit{n}}^2$ and ${\mathit{\sigma }}_{\mathit{y}\mathit{n}}^2$ are the variance of velocity of the n-th optimal window, in the horizontal and vertical directions, respectively, and ${\mathsf{\Delta }}_{\mathit{x}\mathit{n}}$ and ${\mathsf{\Delta }}_{\mathit{y}\mathit{n}}$ are the velocity resolution of the n-th window, in the horizontal and vertical directions, respectively. Assuming the uniform distribution, the variance is obtained from the resolution as ${\mathit{\sigma }}_{\mathit{x}\mathit{n}}^2={\displaystyle \frac{{\mathsf{\Delta }}_{\mathit{x}\mathit{n}}^2}{12}}$ and ${\mathit{\sigma }}_{\mathit{y}\mathit{n}}^2={\displaystyle \frac{{\mathsf{\Delta }}_{\mathit{y}\mathit{n}}^2}{12}}.$The lateral (sideways) velocities are fused by a weighted average as follows

$$v_x\left(k\right)={\sum }_{n=0}^{N_w-1}{w}_{xn}{v}_{xn}\left(k\right), k=0,\dots ,K-2,$$

(6)

where ${\mathit{v}}_{\mathit{x}\mathit{n}}\left(\mathit{k}\right)$ is the lateral velocity obtained from the n-th optimal window template matching between k and k+1 frames, and K is the number of frames. Equation (6) is the weighted least-squares estimator and the maximum likelihood estimator with zero-mean and uncorrelated Gaussian noise [26,27]. The resolution of the longitudinal velocities are drastically changed as shown in Figure 5b. Therefore, the winner-take-all strategy is adopted for the fusion of the longitudinal velocities as

$$v_y\left(k\right)={\sum }_{n=0}^{N_w-1}{w}_{yn}^{*}{v}_{yn}\left(k\right)=v_{y{N}_w-1}\left(k\right), k=0,\dots ,K-2,$$

(7)

$$w_{yn}^{*}=1\left({argmin}_{i=0,\dots ,N_w-1}{w}_{yi}\right),$$

(8)

where $v_{\mathrm{y}n}\left(k\right)$ is the longitudinal velocity obtained from the n-th optimal window template matching between k and k+1 frames. This winner-take-all decision rule is meaningful when the best resolution is significantly better than others [28].

3.2. Traejctory Generation

The vision-only trajectory is formed as

$${\mathit{x}}_v\left(k\right)=\left[\begin{array}{c}{x}_v\left(k\right)\\ y_v\left(k\right)\end{array}\right]={\mathit{x}}_v\left(k-1\right)+\mathit{v}\left(k-1\right)T, k=1,\dots ,K-1,$$

(9)

where $\mathit{v}\left(k\right)={\left[v_x\left(k\right) v_y\left(k\right)\right]}^t$, t denotes matrix transpose, and ${\mathit{x}}_v\left(0\right)$ is initialized to the zero vector. When the zero-order hold scheme is applied, the fused velocity $\mathit{v}\left(k\right)$ is replaced by ${\mathit{v}}_{zoh}\left(k\right)$as follows:

$${\mathit{v}}_{zoh}\left(k\right)=\left\{\begin{array}{c}\mathit{v}\left(1\right), 1\le k<M-1\\ \mathit{v}\left(M\right), M\le k<2M-1\\ .\\ .\\ .\end{array}\right.,$$

(10)

where $M-1$ is the number of frames before the next frame matching occurs; thus, the frame-matching speed becomes frame rate (frame capture speed) divided by M.

3.3. Perofrmance Evaluation

The ground turth trajectories are obtained from GPS signals (latitude and longitude). The vision-only trajectory has a different coordinate system than the GPS coordinate system except for a common origin. Therefore, it is assumed that the vision-only trajectory is a rigid body rotated to the ground-truth trajectory as

$${\mathit{x}}_g\left(k\right)=R\left({\theta }_l\right){\mathit{x}}_v\left(k\right), k=0,\dots ,l-1,$$

(11)

$$R\left(\theta \right)=\left[\begin{array}{cc}cos\theta & -sin\theta \\ sin\theta & cos\theta \end{array}\right],$$

(12)

where ${\mathit{x}}_g\left(k\right)$ is the gournd truth trajectory at frame k; the initial location${\mathit{x}}_g\left(0\right)$ is also set to the zero vector; ${\theta }_l$ is the counterclockwise angle from the vision-only trajectory to the ground-truth trajejctory when the frame number of the trajecoty is l. The optimal rotation angle that aligns two trajectories in the least sqaure sense is obtained as [29,30]:

$${\widehat{\theta }}_l= {argmin}_{\theta }\sum _{k=0}^{l-1}{‖{\mathit{x}}_g\left(k\right)-R\left(\theta \right){\mathit{x}}_v\left(k\right)‖}^2 ,$$

(13)

$$R\left({\widehat{\theta }}_l\right)={VU}^t,$$

(14)

where $UD{V}^t$ is a singular value decomposition of the cross-covariance matrix $X_v{X}_g^t$, $X_v=\left[{\mathit{x}}_v\right(0)\dots {\mathit{x}}_v(l-1\left)\right],$and $X_g=\left[{\mathit{x}}_g\right(0)\dots {\mathit{x}}_g(l-1\left)\right].$ It is noted that ${\widehat{\theta }}_l$ varies with the trajectory frame number l. The RMSE and drift error are defined, respectively, as

$$E_{RMSE}\left(l\right)=\sqrt{{\displaystyle \frac{1}{l}}{\displaystyle {\sum }_{k=0}^{l-1 }}{‖{\mathit{x}}_g\left(k\right)-R\left({\widehat{\theta }}_l\right){\mathit{x}}_v\left(k\right)‖}^2},$$

(15)

$${\mathit{E}}_{\mathit{D}}\left(\mathit{l}\right)=‖{\mathit{x}}_{\mathit{g}}\left(\mathit{l}\right)-\mathit{R}\left({\widehat{\mathit{\theta }}}_{\mathit{l}}\right){\mathit{x}}_{\mathit{v}}\left(\mathit{l}\right)‖.$$

(16)

The RMSE error evaluates the accuracy for the entire trajectory, whereas the drift error measures the accuracy at a point. In the experiments, RMSE and drift errors are caluated at three intermediate and one end points$.$

4. Results

4.1. Flight Paths

A multi-rotor drone (DJI Mavic Enterprise Advanced) [23] flew along six different paths in rural and urban areas. All videos were captured at 30 frame per second (FPS) with a frame size of 3840 × 2160 pixels. The altitude of the drone was maintained at 40 m, and the camera tilt angle was set to 60 degrees. The AFOV was assumed to be 68° and 42° in the horizontal and vertical directions, respectively. Figure 6 shows the ground-truth trajectories of six flight paths (Path 1 to 6) obtained from GPS signals. The starting point is marked 'O', three intermediate points or waypoints are marked 'A', 'B', 'C', and the end point is marked 'D'. Figures 6a-6d show four flight paths (Path 1 to 4) in rural areas, while Figure 6e and Figure 6f show two flight paths (Path 5 to 6) in urban areas. Rural areas are mostly flat and simple agricultural land. Urban areas, on the other hand, have complex and irregular terrain due to various artificial structures. Figure 6a and Figure 6b show a forward-backward straight path and a forward-backward zigzag path, respectively. Figure 6c is a squared path with three banked turns. Figure 6d is a free path with multiple banked turns and zigzags. Figure 6e and Figure 6f are an outbound straight path and a forward-backward straight path, respectively. No flat turn (yaw-only rotation) was included for all paths. In all flights, the drone starts from a stationary hovering state and accelerates to fly at its highest possible speed. For the forward–backward flight, an abrupt joystick reversal causes the drone to decelerate, stop, and move in the opposite direction.

Figure 7a–f show video frames when the drone passes points O, A, B, C, and D in six paths, respectively.

Table 1 shows the frame number and the ground-truth trajectory length of Paths 1 to 6. The shortest length is Path 5, at 810.99 m, and the longest length is Path 1, at 2066.7 m.

4.2. Vision-Only Rotated Trajectories

Figure 8 shows the vision-only trajectory aligned with the ground-truth trajectory. This alignment is necessary because the vision-only trajectory have a different coordinate system than the GPS trajectory. Alignment is achieved through rigid body rotation of the vision-only trajectory, as shown in Equations (11)-(14).

Figure 9 shows the ground-truth velocity obtained from the ground-truth trajectory and the vision-only rigid-body rotated velocities. The ground-truth velocity is calculated as the average velocity of the forward and backward velocities over one second. As shown in Figure 9, large errors often appears at high velocities. A few outliers were observed in the vision-only velocities. These isolated outliers can be removed using a robust statistical tool such as the Hampel filter [31], but the performance improvement was found to be minimal.

Table 2 shows the RMSE and drift error of six paths at Points A, B, C, D, and average. The RMSE error evaluates accuracy over the flight, while the drift error measures accuracy at each point. The average RMSE error ranges from 7.17 to 13.88 m, and the average drift error ranges from 9.44 to 20.25 m for six paths. The RMSE and drift errors at the end point range from 9.88 to 24.89 m and from 5.92 to 42.47 m, respectively. In terms of percentage, the worst RMSE and drift error at the end point were approximately 2.02% and 3.27%, respectively, on path 4.

4.3. Effects of Zero-Order Hold Scheme

Figure 10 shows the RMSE and drift errors for the end point and the average RMSE and drift errors when the zero-order hold scheme is applied. The frame matching rate varies from every frame (30 FPS) to one frame per second (1 FPS). The lower frame matching rate reduces the computational burden. Performance degraded in Paths 1, 2, and 5 as the frame matching rate decreased, but remained relatively stable in Paths 3, 4 and 6.

A total of 12 Supplementary Files are available online. Six MP4 videos for the flight and their corresponding GPS files (captions).

5. Discussion

The optimal window aims to ensure a uniform spacing in the real-world coordinate system. The number of optimal windows was set to 12, then the same number of velocities were obtained through template matching. These velocities were fused with two different fusion rules: weighted averaging for the lateral velocity and winner-take-all decision for the longitudinal velocity. Sideways flight produced more errors than longitudinal flight. Most errors appeared in the form of biases that were proportional to speed and in the opposite direction to the velocity. Several factors degraded the performance; the tilting caused by the drone’s thrust was ignored and the camera’s AFOV was set approximately. More precisely adjusting these approximations can improve performance. Additionally, the ground was assumed to be flat, which was more appropriate for high-altitude drones. This limitation could be alleviated through a practical post-processing solution [15]. Alternative solutions to overcome this limitation remain a topic for future research.

The zero-order hold scheme reduces computational complexity by a factor of several tens. However, slower frame matching can cause more errors.

6. Conclusions

This vision-only technique allows drones to estimate their location using only visual sensors. A novel frame-to-frame template-matching technique was significantly improved in the paper. The optimal window is derived from a piecewise linear regression model that linearizes the nonlinear image-to-real world conversion function. The optimal window template is scene-independent and requires no additional processes to update.

Multiple velocities are fused based on the minimum detectable velocity. The RMSE and drift error have been shown to range from several meters to a few tens of meters for around 800 m to 2 km length high maneuvering flights.

The technique supports a wide range of applications spanning commercial and industrial tasks as well as military missions. The flight range could be further extended, a topic of future research. Thermal imaging cameras [32] could be used in environments with no illumination, a possibility also being explored in future research.