Monocular Vision Navigation Sensor for Autonomous Aerial Refueling

Autonomous aerial refueling technology is an effective solution to extend flight duration of unmanned aerial vehicles, and also a great challenge due to its high risk. For autonomous probeand-drogue refueling tasks, relative navigation to provide relative position between the receiver aircraft and the refueling drogue is the first and essential step, and vision-based method is the most frequently used. A new monocular vision navigation sensor with image processing strategy consisting of the drogue detection method and the tracking method is developed for autonomous aerial refueling in this paper. In the drogue detection method, thresholding and mathematical morphology method are adopted to eliminate image interference, and contours extraction method is applied to obtain all contours, which are then subsequently checked to achieve target contour of drogue. In the tracking method, a rectangle of interest (ROI) of current frame image is determined by positioning results of last frame, and then processed by the previous drogue detection method. Finally, the proposed image processing strategy in monocular vision navigation sensor is validated using real flight images, which are captured from an autonomous aerial refueling testbed using a micro six-rotor aircraft as receiver aircraft.


Introduction
Aerial refueling, defined as the operation of in-flight fuel transmission from the tanker aircraft to the receiver aircraft, provides an effective solution to expand the limited flying distance of the aircraft, and is widely used in military area [1][2][3].For unmanned aerial vehicles (UAV), autonomous aerial refueling (AAR) can greatly improve the UAV's performance through extending the UAV's flight duration [4].Two major hardware configurations used for aerial refueling are the boom-andreceptacle refueling system and the probe-and-drogue refueling system [5,6].In the boom-andreceptacle refueling system, a retractable boom with two attached aerodynamic control surfaces is extended from the tanker, and then the tanker steers the boom to a coupling on the receiver.In the probe-and-drogue refueling system shown as Figure 1(a), the tanker drags a refueling drogue with flexible hose, and then the receiver is maneuvered to bring the attached refueling probe into contact with the drogue [7].The major advantage of probe-and-drogue refueling system is that two or more aircrafts are allowed to refuel simultaneously.
Autonomous probe-and-drogue refueling relies on two key technologies: drogue detection and vision-based measurement.The first one is to robustly obtain drogue position in image, and the latter is to determine the relative position between the probe and the drogue.The attempt of this paper is to present a practical vision navigation sensor with a new image processing strategy to detect drogue image position for autonomous probe-and-drogue refueling system.Vision-based navigation method is considered the most feasible solution to the problem of autonomous navigation in deep-space missions, and becomes increasingly popular in autonomous aerial refueling tasks [8][9][10][11][12].The measurement precision is increased with proximity to the target in vision systems [13,14].Research on vision systems for probe-and-drogue refueling tasks can be divided into two categories: the active vision system and the passive vision system [15].Active vision systems employ obvious features, such as painted markers or LED beacons, to locate the drogue.VisNav is a commonly used active vision system, and has been used in several AAR studies.A set of LED beacons are mounted on the drogue, and the line of sight of beacon is detected by sensing diodes mounted on the receiver aircraft.Sequenced illumination of beacons and a communication link are employed to ensure the order of detected LED beacons.The employment of beacons is then advocated by Pollini, who used an inexpensive CCD webcam with an infra-red filter to detect LEDs placed on the drogue [16,17].In active vision systems, drogue target can be captured with relatively meager processing time and high reliability.The main disadvantage is that non-trivial modifications to the tanker are required to provide electrical power for beacons.
The passive vision system does not require additional hardware installed on the tanker aircraft.Martinez proposed a vision-based strategy only depending on the features of drogue itself for autonomous aerial refueling tasks [18].Direct methods [19] and hierarchical image registration [20] techniques are used for drogue tracking.And the drogue detection method is composed of two algorithms: the edge-image template matching algorithm using normalized cross correlation (NCC), and the image threshold segment algorithm.Yin developed drogue detection and tracking strategy based on the nearly circular and dark inner part of the drogue, and edge images are applied to extract image features [21].Gao proposed a drogue detection algorithm based on low rank and sparse decomposition with local multiple features, and the drogue image sequences are decomposed into the low rank background and sparse moving object [22].The challenge of passive vision systems is to improve the real-time performance and reliability.Therefore, simple and effective image processing algorithms with a high degree of robustness are indispensable for the success of autonomous aerial refueling navigation [23].
This paper focuses specifically on the proposal and evaluation of a practical monocular navigation sensor with a new image processing strategy for probe-and-drogue refueling system based on monocular vision in conjunction with real, full-scale drogue hardware.Direct methods are adopted and installation of specialized hardware (such as beacons) is not required.As shown in Figure 1(b), the inner refueling port is imaged as a nearly circular black block, and this is the main extraction feature used to detect and locate the drogue in the image.And image features are extracted from the binary image instead of edge image in [21], and less processing time is required.A high fidelity testbed based on a micro six-rotor aircraft for autonomous aerial refueling tasks is then developed to evaluate the positioning accuracy, real-time performance, and reliability of proposed image processing strategy.
This paper is organized as follows.Section 2 describes the image processing strategy for autonomous aerial refueling navigation, including the drogue tracking method and the drogue detection method.In Section 3, experiments based on the autonomous aerial refueling testbed are deducted and results are discussed in detail.Finally, Section 4 presents our conclusions and future work.

Strategy overview
The objective of vision navigation system for autonomous aerial refueling tasks is to determine the relative 3D position of the drogue with respect to the receiver aircraft.The first thing is to detect the drogue target in images, which are captured by the camera mounted on the receiver aircraft.Fast, reliable and accurate extracted image features of the drogue are required in order to achieve an accurate relative position between the drogue and the receiver aircraft.
The proposed strategy adopts the state machine with four states to manage image processing operation, and these four states include initial state, capture state, lock state and exit state, as shown in Figure 2(a).The state machine starts with initial state, in which a counter C is initialized as zero and added by one when the target in current frame image cannot be detected using the proposed drogue detection method.When the counter C equals Cmax, which means target cannot be found in continuous Cmax frames, then the initial state is transferred to the exit state, and refueling tasks will be abandoned.Once the target is detected, initial state will be transferred to the capture state.The drogue tracking method can be adopted in the capture state, and the drogue detection method will be used if failed to catch target in tracking procedure.If the target is detected or tracked successfully, the state will be transferred to lock state, and image positioning results can be output to solve visionbased relative position.If target cannot be found through detection and tracking methods, the capture state is then transferred to initial state, waiting for a new frame image to execute detection.As we can see in Figure 2(a), only the lock state can output reliable image position of drogue.The image position is then used to solve relative position between drogue and camera assuming that the camera is calibrated and that the dimension of the drogue is known.The drogue tracking method (see Section 2.2) is adopted in the capture state and the lock state.And the drogue detection method (see Section 2.3) is used only when the tracking method failed or the state machine worked at the initial state.

Drogue tracking method
As the receiver aircraft keeps stable flight state in aerial refueling procedure, it can be assumed that the refueling drogue in video frame streams shows slow change, and that drogue positions in current frame image and last frame image are very close.A priori information can be used to deduce image processing time and assure reliability in current frame image if the drogue is found in last frame image.
Drogue tracking is based on the image positioning result in last frame image.A rectangle of interest (ROI) is chosen and the drogue detection method developed in Section 2.3 is then applied to obtain target position.As shown in Figure 2(b), ( , , , , ) represent ellipse shape parameters of inner refueling port in last frame image, and the ROI in current frame image can be solved by where ( , ) are sizes of extended region in Figure 2(b).The proposed drogue detection method in this paper is composed of five steps, as shown in Figure 3. Firstly, we use binary images through thresholding origin image to reduce detection time.And mathematical morphology method is then applied to eliminate image interference, especially the umbrella skeleton of drogue as shown in Figure 1(b).After that, contours extraction method is used to obtain all the contours and judge them with four conditions including range condition, area condition, circularity condition and ellipse fitting condition.At last, target contour candidate with maximum logarithmic circular degree is determined as the target contour of drogue.Figure 3 depicts a sketch of the drogue detection procedure, and details of these five steps are provided in the following subsections.

Thresholding
Thresholding the image can split the image into a number of regions each having a high level of uniformity in color and brightness.As the inner refueling port in the captured drogue image shows a nearly circular black block, pixel points in the inner refueling port have low grayscale values, and can be easily extracted as a foreground region when using the inverse thresholding method.
Read the drogue grayscale image , and threshold it to a binary image by the following transformation： where is the thresholding grayscale value, ( , ) represents the image pixel coordinate, ( , ) and ( , ) are grayscale values of ( , ) pixel in and respectively.Figure 4

Mathematical morphology operation
As shown in Figure 4(b), the umbrella skeleton holds together with the inner refueling port, and extraction of the inner refueling port from the whole image is difficult.Morphological operation is able to eliminate pepper noise, fine hairs, and small protrusions.The nearby interference caused by the umbrella skeleton can be eliminated through morphological opening operation.
In mathematical morphology, a structuring element is always applied to a binary image, and the image is processed by treating as sets of points.A circular disk is adopted as the structuring element to do opening operation with as follows: where is the result binary image, ∘ is the opening operation symbol, ⊖ is an erosion operation, and ⊕ is a dilation operation.Figure 4(c) shows the binary image after opening operation with a circular disk whose size is 5, and it can be seen that the nearby interference of the inner refueling port is disappeared and that the whole structure of inner refueling port remains.

Contour extraction
The target region (inner refueling port) is separated from other regions in the binary image after mathematical morphology operation.Contours extraction method is always to obtain edges of all the separated regions, and then each extracted contour represents the target region.The operation of contours extraction is consisted of two steps: (1) Hollow out inner points of the region to preserve boundary points.(2) Track contour lines by means of the connectivity rules which describe the neighboring relationships between pixels.There are two common ways of defining connectivity: four-neighborhood where only immediate neighbors are analyzed for connectivity; or eightneighborhood where all the eight pixels surrounding a chosen pixel are analyzed for connectivity.Details of the contours extraction method can be found in [24].
Do contours extraction with to obtain sets of edge points of contours, denoted as where is the number of extracted contours, represents the set of edge points for k-th contour, is the number of edge points, and ( , ) represents the m-th edge point of .Results of contours extraction using eight-neighborhood connectivity are shown in Figure 4(d).Red lines represent edges of extracted contours, and the target region is extracted as a contour.

Judgments of target contour candidates
The primary task after contours extraction is to judge the contour representing the target region.Four conditions are applied to find target contour candidates, which are the range condition, the area condition, the circularity condition and the ellipse fitting condition.
For each contour , following judgments are done to find contours satisfying all conditions, and these contours are denoted as = = , , … , where ~ represent sequence numbers of contour candidates.

Range condition
Range condition eliminates the contour that has very little range or width and height are not roughly equal.The target contour candidate must satisfy the following constraint relationship where ( , ) is range thresholding values of x-axis and y-axis, is the range proportion thresholding value, and ( , ) refs to the pixel coordinate range of contour .

Area condition
Area condition eliminates the contour that has very small area.If the area of contour is denoted as , then the target contour must have area of larger than , which means > where ta is the area thresholding value.

Circularity condition
As the target region is nearly circular, the target circularity must satisfy = > where is the circularity of contour , is the circularity thresholding value, and is the perimeter of contour .

Ellipse fitting condition
The perspective projection of a circle forms an ellipse on the image plane.Once candidate edge points are found, an ellipse must be fitted to these data points.Apply least square ellipse fitting method to edge points of contour , and denote the result shape parameter as ( , , , , ), where( , ) is the center of ellipse, and ( , ) the major semi-axis and minor semi-axis, and the rotation angle.The fitting quadratic function can be described by [25] as where S represents the parameters = ( , , , , ).Then the ellipse fitting problem becomes an optimization problem with the objective function to be minimized as follows: = min ( , , ) Least square [26] can be applied to solve equation ( 9), and it should be pointed out that these fitting parameters must be checked according to the ellipse inequality constraint 4 > .The standard ellipse parameters ( , , , , ) can be obtained from the implicit coefficients , which is a well-known result from classical geometry.Then according to equation ( 6), the fitting result must satisfy Also a small fitting error is required, that means the fitting distance error must satisfy < and < where ( , ) are the mean value and STD (standard deviation) value of fitting distance errors { | = 1,2, … , }, and ( , ) are corresponding thresholding values.The fitting distance error can be calculated as follows where ( , ) is the normalized coordinate of ellipse point ( , ).

Target Determination
If the final number of target contour candidates is 0, then the target is not found in the current image.If it is equal to 1, the result is the final contour ellipse.If it is greater than 1, then choose the target contour candidate with maximum logarithmic circular degree as the target contour, and use its ellipse fitting result as the target feature location results.The logarithmic circular degree is defined as ( , ) = |( − )/( + )|, where ( , ) are the major semi-axis and minor semiaxis of fitting ellipse result for contour candidate Ck.Target contour candidates are shown in Figure 4(e) when using = = 8, = 0.65, = 100, = 0.65, = 1.0, = 0.6.It can be seen that only one target contour candidate is preserved, and then Figure 4(f) shows the final result of target detection.

Autonomous aerial refueling testbed
The drogue detection method presented in this paper was tested experimentally on an autonomous aerial refueling testbed (AART).As shown in Figure 5, AART consists of three parts: tanker, receiver and ground station.The tanker is a 2D mobile platform, on which the refueling drogue was mounted.To achieve a high fidelity, we use a full-scale drogue hardware which has the same size and color with the real drogue.The receiver is a micro quad-rotor aircraft (called Kopter-NRC), on which a camera was mounted to capture drogue images in vision system.The ground station is used to monitor and control system operation, thus experiments are operated in controllable and safe condition.
A Logitech C270 camera is mounted on Kopter-NRC as shown in Figure 5.This camera is capable of capturing images at 30Hz, and the image resolution is 1280×720.To reduce image processing time, we deal with images at resolution 640×480, and capture images at 25fps.The camera is connected to ODROID XU3 computer with Exynos 4412 ARM Cortex-A9 Quad Core 1.7GHz CPU, and the visual system run LUbuntu 14.04 as the operation system.The proposed image processing strategy is developed in C++, and the OpenCV [27] libraries are used for managing image data.To test the performance of proposed image processing strategy, four videos are captured with 640×480 resolution at 25fps.When capturing drogue images, the receiver aircraft gradually moves close to the drogue from a distance, and then backed away.Each video contains 800 images with different movements of the drogue, and the performance of proposed image processing strategy is analyzed using these four videos.In the first test, the precision and reliability of the proposed drogue detection method are evaluated.A second test is conducted in order to provide the time consuming of the drogue detection method.Finally, the real-time performance of developed image processing strategy containing drogue detection and drogue tracking methods is analyzed in the third test.

Test 1: Precision and Reliability of the Drogue Detection Method
In this test, all the images are processed with the drogue detection method in order to test the precision and reliability.Some results of drogue detection are shown in Figure 6, and red ellipses indicate the finally determined edges of inner refueling port.It shows that the red ellipse and the real drogue refueling port are basically coincident, which means the drogue detection method proposed in this paper can accurately extract the image features of drogue.
Drogue detection results for all images are checked, and eye observation is applied to judge if the drogue detection result correctly locates the real inner refueling port.Table 1 shows the success rate of drogue detection method for all 3200 images.We can see that targets in 4 frames are not found in video #1, and video #4 has 2 frames not found, and target is accurately located in all the frames of video #2 and #3.It is worth pointing out that the method proposed in this paper does not take the wrong target as a result.In conclusion, the method has high reliability and the overall detection success rate is 99.81%.
Distributions of positioning results are given to illustrate the effectiveness of the proposed drogue detection method.Figure 7 shows the image position trajectory of the center of inner refueling port which is successfully detected in each video.One can see that the imaged center of the inner refueling port is relatively concentrated in a small region and that any deviation point is not appeared.This is gentle consistent with the real flight maneuver.And it's obvious in video #4 that targets in the horizontal direction changes in a large scope, and the variation law is consistent with the real flight trajectory that the receiver aircraft has a large horizontal maneuver.
Figure 8 shows the imaged radius changes of drogue inner refueling port with the frame flow of videos.Radius of 0 in Figure 8 indicates the failure of target detection.We can see that several frames in video #4 and #1 has not been detected successfully, including video #1 has 4 consecutive frames (frame 545-548) not detected, and video #4 has 2 frames (frame 61 and 77) not detected.Eliminating the frames which are failed to detect, we can see that the imaged radius changes of refueling port increased firstly, and then decreased in a smooth level, which is consistent with the real flight maneuver.

Test 2: Time-Consuming of the Drogue Detection Method
Time-consuming of drogue detection for all the images are pictured in Figure 9.As the timeconsuming of contour extraction occupies a larger proportion in the drogue detection method, and it only depends on the image to be processed, some frames show a large time-consuming, and mutations are appeared in the change curve of time-consuming.Without consideration of mutations in Figure 9, we can see that the method takes the basic trend consistent with the drogue detection radius in Figure 8, that the time-consuming becomes shorter when radius gets smaller.
Statistical results of time-consuming in all frames (a total of 3200 frames) are as follows.Most frames are processed in about 20ms when using the drogue detection method, and the timeconsuming has an average of 34.3ms, a standard deviation of 32.2ms, a minimum of 14.4ms and a maximum of 328.1ms.2741 frames have time-consuming of less than 40ms, which mean the proposed drogue detection method can meet the real-time performance requirement in 84.81% case when image is captured at 25fps.As the target detection method is only applied when the system is in the initial state or failed in target tracking procedure, the time-consuming requirements of the drogue detection method can be appropriately relaxed, and 3053 frames (95.41% of the total) can satisfy the time-consuming requirement when 100ms is used as the boundary value of time-consuming.

Test 3: Real-time performance of proposed image processing strategy
As the detection time is relatively long, the tracking method is applied to reduce the image processing time-consuming.The detection and tracking methods are combined to achieve a high reliability and real-time performance.Using the proposed strategy to process previous four videos, the reliability and accuracy are almost the same with results only using the detection method, due to the detection method is always applied to determine the positioning result in the proposed strategy.The following is the analysis of the real-time performance of this strategy.
Processing time of four videos is shown in Figure 10, and one can see that a small number of frames spend a relatively long time when using detection method.And the tracking time is less than 40ms for all frames using tracking method, which shows that the use of tracking method can significantly improve the real-time performance.The statistical result of 3200 frames are as follows.
The average time-consuming is 13.6ms, the standard derivation is 8.4ms, the minimum tracking time is 10.5ms, and the maximum tracking time is 38.1ms.In conclusion, the image processing strategy developed in this paper can meet the requirements of precision, reliability and real-time performance.

Conclusions
Relative navigation technology is essential for autonomous probe-and-drogue refueling tasks, in which vision-based drogue detection and tracking method is the mostly used.In this paper, a new monocular vision navigation with image processing strategy consisting of the drogue detection method and the tracking method is developed for autonomous aerial refueling system.Experiments are deducted under an autonomous aerial refueling testbed which employs a micro six-rotor aircraft as receiver aircraft.And a full-scale drogue hardware is applied to provide high-fidelity images.Results show that the proposed strategy is able to meet requirements of the positioning accuracy, real-time performance, and reliability in the autonomous aerial refueling procedure.
Future work will focus on an end-to-end assessment of the performance of a complete vision navigation system for autonomous aerial refueling tasks, which will systematically incorporate the image processing method developed in this paper with vision-based pose estimation method and navigation filter algorithm.

Figure 1 .
Figure 1.Aerial refueling system.(a) Description of the probe-and-drogue refueling system; (b) Basic structure of a real aerial refueling drogue.

Figure 2 .
Figure 2. (a) State machine in the proposed strategy; (b) Schematic diagram of ellipse parameters.

Figure 1 (
Figure 1(b) shows the basic structure of drogue.It can be seen that the most obvious image feature is the inner refueling port imaged as a circular black block.When the receiver aircraft carries out close docking and locate in the rear side of a few meters with the tanker, the inner refueling port in the captured drogue image shows a nearly circular black block.In this paper, we will use the nearly circular black block as the main image feature to detect and locate drogue in the image.

Figure 3 .
Figure 3. Sketch of the drogue detection procedure.
(a) and Figure 4(b), respectively, show the origin gray image of drogue and the binary image with the thresholding grayscale value = 120 .It can be seen that partial background interferences are eliminated, and the structure of inner refueling port is still reserved.Preprints (www.preprints.org)| NOT PEER-REVIEWED | Posted: 16 May 2017 doi:10.20944/preprints201705.0117.v1

Figure 4 .
Figure 4. Results for each step in drogue detection method: (a) Origin gray image; (b) Image after thresholding; (c) Image after opening operation; (d) Contours extraction results; (e) Target contours candidates; (f) Target determination results.

Figure 7 .
Figure 7. Center of inner refueling port in imaged plane.

Figure 8 .
Figure 8. Variation curve of radius of inner refueling port in imaged plane.

Figure 10 .
Figure 10.Time consuming for drogue detection and tracking.

Table 1 .
Success rate for drogue detection