Long Duration UAV Localization Cross Day and Night by Fusing Dual Vision Geo-Registration with Inertial Measurements

1. Introduction

Long-endurance UAVs are characterized by long duration, long flight distances, and high altitudes. Therefore, the long-endurance UAV requires a navigation system to maintain robust and accurate localization performance for long periods. The primary navigation method used on current long-endurance UAVs is GNSS-inertial integrated systems. However, GNSS is susceptible to interference, leading to task failure or catastrophes in challenging environments.

As an emerging navigation technology, visual navigation is fully autonomous and resistant to interference, making it an alternative method for the GNSS. The used visual navigation can be broadly divided into two categories: the Relative Visual Localization (RVL) methods and the Absolute Visual Localization (AVL) methods [1]. RVL uses image flows to estimate the relative motion, such as Visual Odometry (VO) [2,3,4] and Visual Simultaneous Localization and Mapping (VSLAM) [2,5,6]. While the AVL compares camera images with a reference map to determine the carrier's absolute position.

As most long-duration UAVs are equipped with navigation-level inertial systems for accurate dead reckoning, this work focuses on using AVL techniques to compensate the inertial drifts and achieve cross day-night accurate localization.

In AVL methods, template matching and feature matching are two common approaches for matching aerial images with reference maps [1]. To achieve better-matching results, obtaining a top-down view similar to the reference map is necessary. However, in UAV application scenarios, due to the horizontal attitude changes of the UAV, the camera axis is typically not perpendicular to the ground. Although mechanisms such as gimbals and stabilizers have been used to maintain the camera's perpendicularity to the ground [7,8,9,10], the limitations in UAV size make it difficult to accommodate these mechanical structures with multiple sensors in different positions on the UAV.

Long-range UAVs need to take long flights, so the vision geo-registration method needs to face the scene of night work. However, most of the traditional AVL researches focus on visible light image matching. In the nighttime, visible-light-image-based AVL methods usually cannot work well. Using thermal images instead is a realizable way to settle it. AVL methods also need reference maps, but few institutions devote themselves to building complete high-precision infrared radiation (IR) maps. Multimodal image matching (MMIM) is an interesting r3.2esearch point in image matching, which is studied wisely in the medical field [11]. Matching IR and visible images (IR-VIS) is one of the most important research directions in MMIM. Similar to the conventional visible image matching methods, the IR-VIS method also can be divided into template matching [12,13] and feature matching [14,15,16,17]. Learning-based methods [18,19] are also used in complex scenes that can’t extract descriptors directly. However, almost all MMIM methods only focus on street scenes and structured scene image registration.

We propose a long-term localization method by fusing the inertial navigation and cross-domain visual geo-registration component, as shown in Figure 1. This method includes three main components: inertial navigation, visible/thermal vision-based geo-localization localization, and inertial-visual integrated navigation. Compared to traditional matching localization algorithms. Our research aims at the day and night visual localization needs under long-term localization conditions. Therefore, we propose using deep learning features to describe and match images using graph neural networks. Besides, we analyze the potential errors in visual matching localization and propose compensation methods. Here are the main contributions of our work:

• To match visible and thermal camera images to a remote sensing RGB map, we investigate several visual features and propose outlier filtering methods to achieve cross day and night geo-registration performance.
• To obtain accurate localization results from geo-registration, we analyze the influence of horizontal attitude on the center point, and we propose the compensation method to correct the localization error to obtain more precise geo-localization.
• We conduct actual long-duration flight experiments with different situations. The experiments include visual registration using various features on visible light and thermal images, geo-localization with horizontal attitude compensation, and integrated navigation. These experiments prove the efficiency of our methods.

The paper is organized as follows. Section 2 presents some research related to our work. Section 3 introduces our proposal, including our integrated navigation framework, cross-domain visual registration method, how to get geo-localization from visual registration, and filter state updating. Section 4 describes our experiments, including system setup, dataset description, data evaluation criteria, and experiment results. Section 5 presents our conclusion.

2. Related Works

We focus on integrating dual vision geo-registration and inertial measurement unit (IMU) cross day and night. Therefore, we focus on two categories of works related to our work: visual navigation methods and IR-VIS methods.

2.1. Visual Navigation

Traditional visual navigation technologies can be divided into RVL and AVL [1]. VO and VSLAM are the two leading technologies under RVL, with VO often serving as a component of VSLAM. Classic pure visual relative localization algorithms include SVO [18], DSO [19], and ORB-SLAM [20], among others. To compensate for the integration errors of pure visual relative localization, many researchers have focused on fusing vision with other sensors, especially integrating the Inertial Measurement Unit (IMU) with vision systems. Representative works include VINS-mono [4] and ORB-SLAM3 [5]. RVL can estimate the relative position of the carrier, but without prior geographical coordinates, RVL cannot determine the carrier's absolute position in the geographical coordinate system. Moreover, VO has cumulative errors, and VSLAM requires loop closure to achieve high-precision localization and mapping, which limits the application of RVL technology in UAVs, especially for long-term and large-scale navigation requirements.

AVL aims to match camera images with reference maps to determine the carrier's position on the map. When the reference map can be aligned with the geographical coordinate system, the absolute position of the carrier in the geographical coordinate system can be determined. Image matching methods are mainly divided into template matching and feature matching. Template matching involves treating aerial images as part of the reference map, so the prerequisite is to transform the aerial images into images with the same scale and direction as the reference map before comparing them. The main parameters used include Normalized Cross-Correlation (NCC) [9,21], Mutual Information (MI) [22], and Normalized Information Distance (NID) [8]. Due to the high computational cost of template matching, an initial value is usually needed to narrow down the search space. At the same time, for better registration between aerial images and reference maps, mechanisms such as gimbals and stabilizers are commonly used to ensure the camera is always perpendicular to the ground [7,8].

Another AVL method is to match features between aerial images and reference maps. Conte [23] used edge detection to match aerial images with maps, but the results were unsatisfactory. M. Mantelli [10] proposed abBRIEF features based on BRIEF features [24] specifically for AVL, showing better performance than traditional BRIEF. J. Surber [25] and others used a self-built 3D map as a reference and matched using BRISK features [24], also utilizing weak GPS priors to reduce visual aliasing. M. Shan [26] and others constructed an HOG feature query library for the reference map. Since it includes a global search process, this method can determine the carrier's absolute position without providing location prior information. A. Nassar [27] proposed using SIFT features for image feature description and introduced Convolutional Neural Networks (CNN) for feature matching and vehicle localization in images.

In summary, while RVL has the disadvantage of cumulative errors, AVL can relatively accurately determine the carrier's position without relying on prior location information. However, most work directly uses the center point of the matched image in the reference map to approximate the carrier's position. Incorrect matching and UAV attitudes can significantly affect the results of visual matching localization. Moreover, most works focus on visible light images, which have poor matching performance at night.

2.2. IR-VIS Method

IR-VIS methods can be classified into template matching and feature matching. In template matching, Yu et al. [13] proposed a strategy to improve Normalized Mutual information(NMI) matching. They used a grayscale weighted window method to extract edge features, thus they reduced the joint entropy and local extrema of NMI. In [12], the author first transforms the image to an edge map and describes geometrical relations between rough and fine matching by affine and FFD. The work optimized matching by maximizing the overall similarity of MI between edge maps. As for feature matching, Hrkać, T et al. [14] detected Harris corners from IR and visible images, then used a simple similarity transformation to match them. However, this experiment is done in situations where images have stable corners. Ma et al. [15] built an inherent feature of the image by extracting the edge map of the image. They propose a Gaussian field criterion to achieve registration. This method shows good performance in registering IR and visible face images. [16] proposed a scale-invariant PIIFD for corner feature description. Besides, an affine matrix estimation method based on the Bayesian framework is proposed. In [17], the author first extracted the edge by morphological gradient method. They used a C_SIFT detector on the edge map to search distinct points and used BRIEF for description, finally making scale- and orientation-invariant matching come true.

With the development of deep learning, matching methods based on learning take an essential role in IR-VIS research. [28] proposed a two-phase GNN, which includes a domain transfer network and a geometrical transformer module. This method was used to obtain better-warped images across different modalities. Baruch et al. [29] bring a hybrid CNN framework to extract and match features jointly. The framework consists of a Siamese CNN and a dual non-weight-sharing CNN, which can capture multimodal image features.

We can see that although a lot of methods are proposed for use in IR-VIS, the majority of them focus on image registration of structured scenes like cities, buildings, and streets. Also, some researches focus on face images. Utilizing the IR-VIS method to achieve vision geo-registration has rarely been researched. Distinguishing from structured scenes and human faces, geo-registration problems focus on unstructured scenes represented by the natural environment. Especially for some situations like forest, desert, gobi, and ocean, few features present an excellent challenge for matching.

3. Method

This section presents our visual-inertial navigation system (VINS) framework, which consists of three components: inertial navigation, vision geo-registration, and integrated navigation. The vision geo-registration component matches the camera images (including the visible light and the thermal images) to the reference map and calculates the geolocation of the UAV. The INS component provides high-frequency navigation state and covariance (including attitude, velocity, and position) prediction. The integrated navigation component utilizes vision geo-registration results to compensate for the drift brought by INS prediction. The framework is shown in Figure 2.

3.1. Visual-Inertial System State Construction and Propagation

In this part, we present the filter used in the integrated navigation, including state construction, nominal state propagation, and covariance propagation. We use State Transformation Extend Kalman Filter (ST-EKF) [30,31] to estimate the navigation state.

3.1.1. Nominal State Propagation

In PINS, the nominal state vector $\mathit{x}$ can be defined as follows:

$$\mathit{x}={\left[\begin{array}{ccc}{\mathit{q}}_{1\times 4}& {\mathit{v}}_{1\times 3}& {\mathit{p}}_{1\times 3}\end{array}\right]}^T$$

(1)

where $\mathit{q}$, $\mathit{v}$ and $\mathit{p}$are attitude, velocity and position respectively. Note that, in nominal state propagation, the attitude is represented and updated by quaternion $\mathit{q}$, which is different from the filter state outlined below.

$\mathit{q}$, $\mathit{v}$ and $\mathit{p}$ need to be calculated by using IMU data. Unlike traditional VINS algorithms like VINS-mono and OpenVINS, which use numerical integration to update the state, we use a twin-sample algorithm [32] widely used in inertial navigation to propagate the state.

3.1.2. Filter State

From Section 3.1.1, we can define the error state $\delta x$ as:

$$\delta x={\left[\begin{array}{ccccc}\delta {\varphi }_{1\times 3}& \delta v_{1\times 3}& \delta r_{1\times 3}& b_{g_{1\times 3}}& b_{a_{1\times 3}}\end{array}\right]}^T$$

(2)

where $\delta \varphi$, $\delta v$ and $\delta r$ is attitude error, velocity error, and position error respectively.

Different from the nominal state, in the filter state, we use the Euler angle to represent attitude error.

Then, the linearized error model of INS at time $t$ can be described as follows:

$$\delta \mathit{x} \dot{}\left(t\right)=\mathit{F}\left(t\right)\delta \mathit{x}\left(t\right)+\mathit{G}\left(t\right)\mathit{u}\left(t\right)$$

(3)

where $\mathit{G}$ is the noise distribution matrix, and $\mathit{u}$ is the noise vector of the system. They are similar to those in traditional Extended Kalman Filter (EKF) of integrated navigation.

The conventional integrated navigation algorithm employing the EKF encounters challenges with inconsistent variance estimation [32]. By reformulating the velocity differential equation, the ST-EKF mitigates the impact of specific force on velocity computation and derives a novel filter system matrix, thereby enhancing the precision of state estimations. According to Reference [30] and [31], the system matrix $\mathit{F}$ can be described as follow:

$$\mathit{F}=\left[\begin{array}{ccccc}-\left({\mathit{\omega }}_{in}^n\wedge \right)+{\mathit{F}}_{ev}\left({ \tilde{\mathit{v}}}_{eb}^n\wedge \right)& {\mathit{F}}_{ev}& {\mathit{F}}_{er}& -{\mathit{C}}_b^n& 0_{3\times 3}\\ -\left[\begin{array}{l}\left({\mathit{g}}^n\wedge \right)+\\ \left({ \tilde{\mathit{v}}}_{eb}^n\wedge \right)\left({\mathit{\omega }}_{ie}^n\wedge \right)\end{array}\right]& -\left[\begin{array}{l}2\left({\mathit{\omega }}_{ie}^n\wedge \right)+\\ \left({\mathit{\omega }}_{en}^n\wedge \right)\end{array}\right]& \left({ \tilde{\mathit{v}}}_{eb}^n\wedge \right)\delta {\mathit{\Omega }}_r& \left({ \tilde{\mathit{v}}}_{eb}^n\wedge \right){\mathit{C}}_b^n& {\mathit{C}}_b^n\\ {\mathit{F}}_{rv}\left({ \tilde{\mathit{v}}}_{eb}^n\wedge \right)& {\mathit{F}}_{rv}& {\mathit{F}}_{rr}& 0_{3\times 3}& 0_{3\times 3}\\ 0_{6\times 3}& 0_{6\times 3}& 0_{6\times 3}& 0_{6\times 3}& 0_{6\times 3}\end{array}\right]$$

(4)

The definitions of the physical quantities in the aforementioned equation are detailed in Reference [30,31].

3.1.3. Covariance Propagation

From vision geo-registration that will be presented in Section 3.2 and INS prediction presented in Section 3.1.1, we get a 2D visual localization position $(lat{i}_V,long{i}_V)$ and INS prediction position $(lat{i}_{INS},long{i}_{INS})$. We define observation $\mathit{z}={\left[\begin{array}{cc}lat{i}_V& long{i}_V\end{array}\right]}^T$, then the observation equation can be described as:

$$\mathit{z}=h\left(\mathit{x}\right)=\left[\begin{array}{ccc}{0}_{2\times 6}& {\mathit{I}}_{2\times 2}& 0_{2\times 7}\end{array}\right]\mathit{x}$$

(5)

Then, the Jacobi matrix of the observation equation $\mathit{H}$ will be:

$$\mathit{H}={\displaystyle \frac{\partial h}{\partial \mathit{x}}}{\displaystyle \frac{\partial \mathit{x}}{\partial \delta \mathit{x}}}=\left[\begin{array}{ccc}{0}_{2\times 6}& {\mathit{I}}_{2\times 2}& 0_{2\times 7}\end{array}\right]$$

(6)

In covariance propagation, we need to predict the covariance matrix ${\mathit{P}}_{k|k-1}$ first:

$${\mathit{P}}_{k|k-1}={\mathit{\Phi }}_{k,k-1}{\mathit{P}}_{k-1}{\mathit{\Phi }}_{k,k-1}^T+{\mathit{Q}}_{k-1}$$

(7)

Where ${\mathit{\Phi }}_{k,k-1}$ is the discretized system matrix, and ${\mathit{Q}}_{k-1}$ is the noise distribution covariance matrix.

Then we calculate Kalman gain ${\mathit{K}}_k$:

$${\mathit{K}}_k={\mathit{P}}_{k|k-1}{\mathit{H}}^T{\left(\mathit{H}{\mathit{P}}_{k|k-1}{\mathit{H}}^T+{\mathit{R}}_k\right)}^{-1}$$

(8)

Where ${\mathit{R}}_k$ is the measurement noise covariance matrix of sensors.

Finally, we update the covariance matrix:

$${\mathit{P}}_{k|k}=\left(\mathit{I}-{\mathit{K}}_k\mathit{H}\right){\mathit{P}}_{k|k-1}{\left(\mathit{I}-{\mathit{K}}_k\mathit{H}\right)}^T+{\mathit{K}}_k{\mathit{R}}_k{\mathit{K}}_k^T$$

(9)

The diagonals of the updated covariance matrix contain elements related to the planimetric position $\left[\begin{array}{cc}{p}_{k-lati}& p_{k-longi}\end{array}\right]$, which will be used to construct the Gaussian elliptic constraint in integrated navigation.

3.2. Cross-Domain Visual Registration

Our work mainly aims at long-duration navigation, which means that the visual geo-localization needs to remain effective at nighttime. This section will present the method how to achieve cross-domain visual registration.

3.2.1. Features Extraction and Matching Method

To achieve day-night vision-based localization, the key is to develop visual features effective on visible light and thermal images. Although a series of hand-craft features and learned features have been used in RGB-image geo-registration [7,8,9,10], few works have shown the capabilities in thermal images.

To obtain a unified feature for cross-domain image matching, we have investigated in multiple hand-craft and learned features, including SIFT [33], XFeat [34], and SuperPoint [35]. Figure 3 shows some samples of the performance of these features on visible light and thermal images. SIFT is applicable to visible light images but demonstrates limited effectiveness on thermal images. XFeat also can be used comparably to SuperPoint in both visible and thermal images, but XFeat exhibits less matching points and matching rate than SuperPoint. The investigation finds that the SuperPoint has shown the best performance in both visible light and thermal matching. In the experimental part, we designed a comparative experiment to prove our conclusion.

3.2.2. Reference Visible Map Pre-Processing

For long-duration navigation, the task areas can be extensive, which brings in difficulties in real-time processing. Therefore, we need to pre-process the reference map before the task to improve the efficiency through the flight.

While the UAVs are equipped with both visible and thermal cameras, we propose only to use remote visible maps for reference, as high-resolution visible maps are easy to obtain.

To improve the algorithm efficiency, we divide the map into submaps and use the predicted location from INS to load the sub-maps. Figure 4 shows the method to build the reference map set.

Firstly, the remote visible light map along the planned flight path needs to be pre-downloaded. Then, we separate the whole map into sub-maps with the size of $w_{map}\times h_{map}$, which $w_{map}$ and $h_{map}$ are actual length of the sub-map in east and north respectively. When the drone flies to the maximum height in the flight, the sub-map should contain as much of the scenery of the camera image as possible. It means:

$$\min (w_{map},h_{map})\ge 2H_{\max }\tan {\displaystyle \frac{\theta }{2}}$$

(10)

in which $H_{\max }$ is the maximum height of UAVs, and $\theta$ is the field of view(FOV) of the camera. At the same time, when visual matching localization fails, pure inertial navigation will cause drift. Therefore, it is meaningful to consider appropriately increasing $w_{map}$ and $h_{map}$ to avoid map retrieval failure caused by localization divergence, which in turn leads to the complete failure of scene-matching localization. In addition, when the image matching area is close to the edge of the map, matching may fail due to the incomplete matching area. Therefore, when cropping the map, we first crop it to a size of $(w_{map}/2)\times (h_{map}/2)$and then stitch adjacent 2x2 map blocks into a sub-map. This ensures that there is a 50% overlap between adjacent sub-maps. Finally, we get the sub-map image set $\left\{m_{11},m_{12},\cdots ,m_{1q},\cdots m_{p1},\cdots m_{pq}\right\}$ numbered in row and column order.

Then, we build an index of the sub-map set by the latitude and longitude coordinates of the center. Finally, we extract features and descriptors for each sub-map and associate them with the index in Table 1.

3.2.3. Camera Image Pre-Processing

For long-duration tasks, the UAV can fly at different heights and with different headings, which leads to scale and view variations of the camera images. To reduce the visual aliasing between the camera images and the map, image rotation and scaling are applied before geo-registration. Furthermore, for thermal images, we apply gamma transformation to enhance the contrast, thereby highlighting features more and increasing the number of matching pairs.

• Rotation and Scaling

We perform rotation and scaling pre-processing on the images before feature point extraction, to align the geographic coordinate system represented by the images with the reference map (the reference map's default upward direction being north), and to make the pixel resolution close to that of the reference map. The scaling factor $img\_ratio$ can be given by:

$$\begin{array}{c}img\_rati{o}_x={\displaystyle \frac{H_r}{f_x{\rho }_m}}\\ img\_rati{o}_y={\displaystyle \frac{H_r}{f_y{\rho }_m}}\end{array}$$

(11)

Where $H_r$ is relative height, ${\rho }_m$is the pixel resolution of the sub-map, $f_x$and $f_y$ are camera intrinsics, which can be given by Zhang's camera calibration method [36].

• Thermal Image Gamma Transformation

Infrared imaging is based on the thermal radiation and temperature characteristic differences of ground scenes. However, in high-altitude ground imaging, the collected scenes are mainly large areas of buildings, forests, farmland, roads, etc., with high scene similarity and small thermal radiation differences, leading to infrared imaging having blurred contours, high noises, and low contrast, which is not conducive to feature extraction. Therefore, before extracting features from infrared images, preprocessing of the infrared images can be performed to extract the potential features of the images better. We choose Gamma transformation to adjust image contrast and brightness to enhance details:

$$s=c{r}^{\gamma },r\in [0,1]$$

(12)

In which $r$ is the normalized grayscale value of the infrared image, $c$ is the grayscale scaling factor, $\gamma$ is the gamma value, and $s$ is the normalized grayscale value after gamma transformation.

3.2.4. Camera-Map Registration

For visual registration, the key is how to match camera images and reference maps. This part will illustrate the whole process including sub-map querying, registration method and anomaly match checking.

Figure 5. Camera-map registration.

Figure 5. Camera-map registration.

• Sub-map querying

To achieve camera-map registration, the first step is to search out the corresponding sub-map. In section 3.2.2, the sub-maps are indexed by the 2D geo-position. Therefore, we can query the required sub-map features by position predicted by the INS prediction component.

• Camera-map registration method

SuperGlue [37] is a widely used matching method for SuperPoint features. However, the front end of SuperGlue contains multiple attentions and fully connected networks, which is not conductive in real-time processing. Therefore, we introduce SGMNet [38] to replace this part to improve the efficiency.

When camera image and sub-map features are sent into the SGMNet, the net first calculates two groups of matches ${\mathit{M}}_a$ and ${\mathit{M}}_b$ as the seed matches. Then the two seed matches pass through a multiple-layer seed graph network and get the best matches. Finally, the best matches are sent into the back end of SuperGlue to obtain the final matches.

• Anomaly match checking

Because of pre-processing, the matched points of the camera image and map should be connected by lines that are approximately parallel with similar lengths. For incorrect matches, the lines are usually not geometrically close, so it is possible to filter out incorrect matched point pairs by using consistency of the matched point line vectors, thereby removing the inconsistent parts from the set of matched point pairs. Assuming that after matching, there are $n$ pairs of matched points in the collection, where the feature point coordinates in the camera image are$\left[\begin{array}{cc}{u}_i^{img}& v_i^{img}\end{array}\right]$, and the corresponding feature point coordinates on the sub-map are$\left[\begin{array}{cc}{u}_i^{map}& v_i^{map}\end{array}\right]$. Then, the line vector for each set of matched point pairs is:

$$t_i=\left[\begin{array}{cc}{u}_i^{trans}& v_i^{trans}\end{array}\right]=\left[\begin{array}{cc}{u}_i^{map}-u_i^{img}& v_i^{map}-v_i^{img}\end{array}\right]$$

(13)

For each $t_i$, we can calculate its magnitude $dis{t}_i$ and slope$slop{e}_i$:

$$\begin{array}{c}dis{t}_i=\sqrt{{(u_i^{trans})}^2+{(v_i^{trans})}^2}\\ slop{e}_i={\displaystyle \frac{v_i^{trans}}{u_i^{trans}}}\end{array}$$

(14)

We define the distance consistency threshold $\sigma$ and direction consistency threshold $\omega$, and then we can get consistency set $si{m}_i$ for each $t_i$:

$$\begin{array}{c}si{m}_i=\left\{t_j|dis{t}_i-dis{t}_j\le \sigma ,\arctan (slop{e}_i)-\arctan (slop{e}_j)\le \omega \right\}\\ i,j=1,2,\cdots ,n\end{array}$$

(15)

At this point, if the number of filtered matches does not meet the threshold of a successful registration, it is considered a failed registration, thereby eliminating incorrectly matched images. If the number of matches meets the threshold, the obtained matches should have eliminated the vast majority of incorrect matches.

3.3. Geo-Localization from Visual Registration

After matching, we get a set of matching point pairs. In this part, we illustrate the method for obtaining location from visual registration to get the visual geo-localization result. Furthermore, we consider the influence of attitude in geo-localization and the design compensation method.

3.3.1. Geo-Localization

The matching relationship between images can be modeled as an affine transformation. Consequently, in our approach, the geo-location of any point on the reference map within the images taken from UAV can be determined by computing the affine transformation matrix $\mathit{H}$. We use homography matrix estimation based on Random Sample Consensus (RANSAC) to determine the affine matrix. If we assume that the center of the camera image $\left[\begin{array}{cc}{u}_c^{img}& v_c^{img}\end{array}\right]$ is the location of the UAV, we can transform the camera image into the reference map by using $\mathit{H}$. We will get the transformed center of the camera image $\left[\begin{array}{cc}{u}_{c-img}^{map}& v_{c-img}^{map}\end{array}\right]$, and the center of the reference map$\left[\begin{array}{cc}{u}_c^{map}& v_c^{map}\end{array}\right]$ represents the location in the real world $\left[\begin{array}{cc}lat{i}^{map}& long{i}^{map}\end{array}\right]$. Then the location of the UAV $\left[\begin{array}{cc}lati& longi\end{array}\right]$ can be given by:

$$\begin{array}{c}lati=lat{i}^{map}+{\rho }_m(u_{c-img}^{map}-u_c^{map})\\ longi=long{i}^{map}+{\rho }_m(v_{c-img}^{map}-v_c^{map})\end{array}$$

(16)

where ${\rho }_m$ is the pixel resolution of a map (length in the world coordinate system represented by a single pixel, with units in meters per pixel (m/pix).

3.3.2. Horizontal Attitude Error Compensation

In this part, we use the NED(north-east-down) local geodetic coordinate frame as the navigation coordinate frame(n-frame) and the FRD(front-right-down) frame as the carrier coordinate frame(b-frame). Then we can define rotation matrixes:

We can define a unit vector $e_n={\left[\begin{array}{ccc}0& 0& 1\end{array}\right]}^T$ in the n-system. Then we can transform the vector to the body frame as:

$$\begin{array}{c}{e}_b^n=C_x{C}_y{C}_z{e}_n\\ =\left(\begin{array}{ccc}\cos \theta \cos \phi & \cos \theta \sin \phi & -\sin \theta \\ \sin r\sin \theta \cos \phi & \sin r\sin \theta \sin \phi +\cos r\cos \phi & \sin r\cos \theta \\ \cos r\sin \theta \cos \phi +\sin r\sin \phi & \cos r\sin \theta \sin \phi -\sin r\cos \phi & \cos r\cos \theta \end{array}\right)e_n\\ ={\left(-\sin \theta ,\sin r\cos \theta ,\cos r\cos \theta \right)}^T\end{array}$$

(17)

Where $r$, $\theta$ and $\phi$ are roll, pitch, and yaw respectively.

Next, the transformed unit vector $e_b^n$ can be expressed in spherical coordinates ${\left[\begin{array}{ccc}1& \alpha & \beta \end{array}\right]}^T$like Figure 6, where

$$\begin{array}{c}\cos \alpha =\cos r\cos \theta \\ \cos \beta ={\displaystyle \frac{-\sin \theta }{\sqrt{{\sin }^2\theta +{\sin }^{2}r{\cos }^2\theta }}}\\ \sin \beta ={\displaystyle \frac{\sin \varphi \cos \theta }{\sqrt{{\sin }^2\theta +{\sin }^{2}r{\cos }^2\theta }}}\end{array}$$

(18)

Finally, the eastward error $\delta n$ and northward error $\delta e$ can be given by:

$$\begin{array}{c}\delta n=H_r{\displaystyle \frac{\cos \alpha }{\sin \alpha }}\cos \beta \\ \delta e=H_r{\displaystyle \frac{\cos \alpha }{\sin \alpha }}\sin \beta \end{array}$$

(19)

Where $H_r$ is the relative height of UAV.

3.4. Filter State Updation with GeoLocalization Observations

We use the above methods to ensure the correctness of geo-localization as much as possible. Despite this, geo-localization results will inevitably contain some wrong localization consequences. To eliminate these inaccuracy results, we introduce Gaussian elliptic constraint [39] to inspect the availability of visual geo-localization before ST-EKF.

Figure 7. Gaussian elliptic constraint. The Gaussian ellipse is constructed by ${\mathit{P}}_k$, and filter only accepts visual geo-localization position amidst the ellipse as the observation.

Figure 7. Gaussian elliptic constraint. The Gaussian ellipse is constructed by ${\mathit{P}}_k$, and filter only accepts visual geo-localization position amidst the ellipse as the observation.

In ST-EKF, the diagonal elements of the covariance matrix ${\mathit{P}}_k$ represent the error bounds of each state variable. Then we can use planar position-related elements $p_{k-lati}$ and $p_{k-longi}$ in ${\mathit{P}}_k$ to establish a Gaussian elliptic equation:

$${\displaystyle \frac{{\mathit{z}}_{1k}^2}{p_{k-lati}}}+{\displaystyle \frac{{\mathit{z}}_{2k}^2}{p_{k-longi}}}<\xi$$

(20)

If ${\mathit{z}}_k$ does not satisfy the Gaussian elliptic constraint, this observation will not be used in integrated navigation.

The pseudocode of state updating is as follows:

Table 2. Pseudocode of state updating.

Table 2. Pseudocode of state updating.

4. Experiment

To prove the effectiveness of the proposed method, we designed a set of experiments, including a comparison of feature matching between several different features, a vision geo-localization experiment and an integrated navigation experiment.

4.1. System Setup

The experiment location was in Jinmen, Hubei. We used a CH-4 UAV as the carrier, and the experiment system includes a Laser IMU working at 200Hz, a set of down-looking- cameras that consist of a visual light camera and an infrared camera, a barometric altimeter, and a satellite receiver that can output differential GPS data as the ground truth. The experiment was in a $15\mathrm{km}\times 65\mathrm{km}$area as Figure 8(a). Figure 8(b) shows the UAV and camera set. And Table 3 shows some parameters of cameras.

$1280\times 1024$$1280\times 1024$

4.2. Dataset Description and Evaluation Method

Two datasets were used to test the proposed method. Dataset 1 was collected in May. 22^nd, 2024, which includes daytime and nighttime RGB and thermal camera images. Dataset 2 was obtained in May. 9^th, 2024; this dataset only has visible light images. The details of the two datasets are shown in Table 4.

Table 4. Experiment datasets.

Dataset	Dataset 1	Dataset 2
Cruising Altitude(m)	2400	700
Time(s)	6892	5170
Length(km)	344.6	284.35
Daytime(s)	2504	5170
Frequency(Hz)	1	1
Acceptable Matching Threshold	50	50
Location Error Threshold(m)	100	100

Table 4. Experiment datasets.

Dataset	Dataset 1	Dataset 2
Cruising Altitude(m)	2400	700
Time(s)	6892	5170
Length(km)	344.6	284.35
Daytime(s)	2504	5170
Frequency(Hz)	1	1
Acceptable Matching Threshold	50	50
Location Error Threshold(m)	100	100

Based on the two datasets, three groups of experiments were conducted. The first experiment is feature matching. There are three features used in the experiment, SIFT , XFeat and SuperPoint. In the experiment, we choose 50 matches as the threshold of an acceptable visual registration and define the matching rate as the proportion of the accepted visual registrations in the dataset. In addition, both visible light and thermal images are registered with a visible light map.

The second experiment is the geo-localization experiment. We use SuperPoint features to match images and evaluate precision by root mean square error (RMSE) in experiments. We set 100 meters as the threshold of a valid geo-localization result. Like the definition of matching rate, we define valid matching rate as the proportion of the valid geo-localization in the dataset. Besides, we compare the accuracy and number of effective geo-localizations with or without attitude compensation to verify the effectiveness of attitude compensation.

The last experiment is the integrated navigation experiment. Since IMU data is only available in Dataset 2, we use a single Dataset 2 for the integrated navigation experiment. We prove the effectiveness of the proposed algorithm by comparing errors with or without proposed filtering methods.

4.3. Experimental Results

4.3.1. Registration Rate Results

This experiment presents the matching rate of three features in the two datasets, SIFT, XFeat, and SuperPoint. Figure 9 and Figure 10 show the performance of the three features in Dataset 1 and Dataset 2 respectively,

The matching rate of SIFT on both visible light and thermal images is far less than XFeat and SuperPoint. On visible light images, XFeat achieves a matching rate of 31.14%, close to 32.72% of SuperPoint. However, when applied to thermal images, XFeat exhibits inferior performance compared to SuperPoint. The performance of SuperPoint on infrared images is the best among the three features, with a matching rate of 68.67%. Considering the effectiveness of registration, the effective matching rate of SIFT on visible light images is only 2.9%, and only two images exhibit linear errors of less than 100 meters. Although XFeat maintained a satisfactory localization precision in visible light, its performance in thermal images remained suboptimal. SuperPoint has the best performance in localization, with an effective matching rate of 22.47% in visible light and 54.03% in thermal.

Additionally, from Figure 9, it can be concluded that while the geo-localization rate of visible images is superior to that of thermal images, thermal images can address the limitation of visible images being unusable at nighttime.

Figure 9. Matching rate and effective matching rate in Dataset 1. SuperPoint has the best performance of the three features in both registration and localization. Visible light images are easy to register and locate but thermal images can register and locate in nighttime.

Figure 9. Matching rate and effective matching rate in Dataset 1. SuperPoint has the best performance of the three features in both registration and localization. Visible light images are easy to register and locate but thermal images can register and locate in nighttime.

Figure 10. Performance of three features in Dataset 2.

Figure 10. Performance of three features in Dataset 2.

The results of the three features on Dataset 2 exhibit performance consistent with the aforementioned analysis. Note that, the matching rate in Dataset 2 is lower than that in Dataset 1. It is because that the cruising altitude of Dataset 2 is lower than that of Dataset 1. It means that images of Dataset 2 have a smaller FOV than Dataset 1. This indicates that images from Dataset 2 contain fewer distinctive features for extraction, consequently leading to a lower matching rate. Additionally, as a lightweight model, XFeat demonstrates greater degradation than SuperPoint in matching performance due to FOV contraction and the reduced availability of features. The sample images of XFeat in Section 3.2.4 also demonstrate similar conclusions.

4.3.2. Visual Geo-Localization Results

This experiment mainly presents the efficiency of attitude compensation. We use RMSE to describe the precision of geo-localization.

We evaluate the RMSE of geo-localization above in Figure 11. The figure shows that attitude compensation can improve the precision of geo-localization. In addition, after attitude compensation, the visible light geo-localization shows better precision than thermal. At last, by comparing the precision between Dataset 1 and Dataset 2, we can discover that different heights can influence the precision of geo-localization. The reason is that according to the equation (11), the pixel resolution of the map and relative height will affect the scaling factor. If the scaling factor is excessively small, it will be difficult to extract enough features used in registration, which leads to a worse matching rate. Therefore, for better registration performance, we try to control the scaling factor between 0.5 and 1. It means that we should confirm flight altitude and choose the map with the corresponding pixel resolution before the flight.

Figure 12 presents sample images of visible light and thermal which attitude compensation has effect on. We can easily see that both samples perform well in registration, but the geo-localization results show a large error. It illustrates that the horizontal attitude angle significantly impacts the visual geo-localization as the UAV approaches the turning point. In our experiment, the UAV consistently operates at a specified altitude. Therefore, the experimental data only clearly indicate that the roll angle $r$ significantly influences the geo-localization precision.

4.3.3. Visual-Inertial Integrated Navigation Results

We use Dataset 2 to present the effectiveness of our VINS method. Figure 13 shows the trajectory of integrated navigation.

The experiment lasted 5170 seconds. From section 4.3.1, we know that there are 997 correct matches. In integrated navigation, 1007 matches are used in ST-EKF, which proves that the proposed attitude compensation and Gaussian elliptic constraint can improve the availability of vision geo-registration.

We use GPS and INS integrated navigation as the ground truth, and compare VINS results with and without Gaussian elliptic constraint. We evaluate the precision using root mean square error (RMSE).

Table 5 shows the precision statistics of the experiment. When using integrated navigation without Gaussian elliptic constraint, the RMSE is 112.65 meters. But the RMSE of the proposed method is 42.38 meters. As demonstrated by Figure 13 and Table 5, the Gaussian elliptic constraint identified and excluded five outliers. These high-error points, if incorporated into the EKF, would significantly compromise the precision of integrated navigation solutions. This experimental result statistically corroborates the efficacy of our proposed Gaussian elliptic constraint methodology.

Figure 14 shows some sample matches eliminated by Gaussian elliptic constraint. Most eliminated matches occur in environments characterized by more repetitive and homogeneous terrain. Visual registration cannot hold robust in these environments, and registration results will be either that there are too few features to match, or that the matching result is to match the terrain similar to the real position, resulting in localization error like Figure 14. From the trajectory, we can see that the variety of mismatches has negative effect on ST-EKF, but these mismatches are difficult to eliminate by simply visual strategies. Gaussian elliptic constraint can eliminate them by using the covariance matrix of ST-EKF, and leave more accurate observations for the integrated navigation system.

5. Conclusions

In this paper, we establish a VINS framework to solve long-term navigation problems in GNSS-denied situations. The framework consists of vision registration, INS prediction, and ST-EKF, which uses vision geo-registration to redress the PINS error. We analyze the influence of attitude on vision geo-localization and propose a compensation method to increase the precision of visual geo-localization. The proposed method makes ST-EKF get more observations with fewer position errors in GNSS-denied situations. The designed experiment proves the effectiveness of our method.

From the experiment, we find that in mountain scenes or other scenes that have high repeatability, the vision geo-registration does not work well. So how to increase visual navigation in these situations may be the next direction of our research.