Research on the Relative Position Detection Method between Orchard Robots and Fruit Tree Rows

1. Introduction

With the rapid development of sensor technology and computer technology, robotic autonomous navigation technology has been applied in orchards on a large scale [1,2,3]. Autonomous navigation technology is the key technology to realize the intelligence of agricultural equipment in orchards, which is conducive to reducing the work intensity of operators, improving work efficiency and enhancing the quality of operations [4,5,6].

Autonomous navigation technology mainly perceives the dynamic environment around the vehicle through a variety of sensors to plan and navigate the path and complete the operations such as orchard weeding [7,8], furrowing and fertilizing [9,10,11], and picking [12,13].Commonly used navigation methods are classified as Global Navigation Satellite System (GNSS)/Global Positioning System (GPS) [14,15], Light Detection and Ranging (LIDAR) [16,17], vision sensors [18,19], and multi-sensor fusion navigation [20].Wei Shuang et al [21] proposed a pure tracking model based on GNSS autonomous navigation path search method for agricultural machines and pre-sighting point search, in straight line navigation, the root mean square error of navigation is 3.79, 4.28 and 5.39 cm when the speed is 0.8, 1.0 and 1.2m/s respectively. Luo Xiwen et al [22] developed an automatic navigation control system based on DGPS, and the maximum error of this system is less than 15 cm and the average error is less than 3 cm in straight line navigation when the tractor is at a speed of 0.8 m/s. However, when the fruit tree branches and leaves are luxuriant and the canopy is closed, and the environment between the rows of the tree is almost close to the semi-closed state, the satellite signals are blocked by the canopy, which leads to a certain limitation of the positioning and navigation method based on GNSS/GPS. positioning and navigation methods are subject to certain limitations. Liu Weihong et al [23] proposed an orchard inter-row navigation method based on 3D LiDAR, the method in the real pear orchard, mobile robot tracking tree rows at a speed of 1.35m/s, the maximum lateral deviation obtained is less than 22.1cm. 3D LiDAR is subject to weather conditions and expensive, greatly increasing the cost of inputs in orchard environments, and therefore difficult to be used universally. Han Zhenhao et al [24] proposed an orchard visual navigation path recognition method based on U-Net network, which uses a camera to collect images and semantically segment the road information in the image to calculate the navigation path, using this method to drive in the orchard road with a width of about 3.1m, the average distance error ratio is 1.4%, which is about 4.4cm, and the average processing time of a single frame of image is 0.154s.

In recent years, deep learning has made breakthrough progress in image classification research, and also driven the rapid development of target visual detection [25,26,27]. Deep learning-based target detection models, including two-stage target detection models and single-stage target detection models [28,29], two-stage target detection models have higher detection accuracy, but slower detection speed, therefore, most real-time detection tasks currently use single-stage target detection models. Typical single-stage detection models include the YOLO [30,31,32] family of models and the SSD [33] model. Xie Shuang et al. [34] used an improved YOLOv8 model for tea recognition, which combines deformable convolution, an attention mechanism, and improved spatial pyramid pooling, thus enhancing the model's ability to learn complex target invariance, reducing the interference of irrelevant factors, and achieving multi-feature fusion, which improves the detection accuracy. Wang et al. [35] constructed a model that fuses YOLO v5s and attention mechanism of convolutional neural network model YOLO_CBAM for the detection of spiny calyx lobelia weed. They devised a method for slicing high-resolution images. This method constructs the dataset by calculating the overlap rate to reduce the possibility of loss of details due to compression of high-resolution images during training, and the final accuracy is 92.72%. Tian et al. [36,37] successfully achieved detection of grape maturity and detection of weed identification and localization through the improved YOLOv4 algorithm. The experimental results show that this method has high accuracy.

This paper proposes a method based on binocular camera to determine the positional parameters of an orchard robot. When the orchard robot advances between rows, the binocular camera acquires images and transmits them to the improved YOLOv4 model for fruit tree trunk detection, obtains the camera coordinates of the trunks, and then obtains the ground projection coordinates of each trunk after coordinate conversion. The ground projection coordinates of different sides of the orchard robot are combined to take the midpoint and the least squares method is used to fit a straight line to the midpoint to obtain the navigation path, and the heading angle and lateral deviation values of the orchard robot are obtained through calculation.

2. Materials and Methods

In this paper, the proposed method for estimating the positional parameters between rows of orchard robots consists of four parts: fruit tree trunk detection, fruit tree trunk localization, navigation path calculation and calculation of positional parameters. The detection of fruit tree trunks is based on the improved YOLOv4 model to detect the trunks on both sides of the orchard robot; the positioning of fruit tree trunks is based on the parallax principle after the binocular camera acquires the image to obtain the camera coordinates of the fruit tree trunks; the calculation of the navigation path is based on the camera coordinates of the fruit tree trunks obtained from the previous step through the coordinate transformation to obtain the ground projection coordinates, the combination of projection coordinates of the different sides of the orchard robot to take the mid-point, and the mid-point is calculated using the least-squares method. The least squares method is used to fit the midpoint to a straight line; the attitude parameters are obtained by calculating the straight line relationship between the fitted straight line and the vehicle base coordinate system.

2.1. Hardware Composition

In this study, the electric-driven tracked orchard operation platform designed by the College of Engineering of Nanjing Agricultural University is used as the mobile carrier, which is driven by two 48V DC servo motors, and the motors are driven and controlled by one dual-channel servo motor driver; the upper computer is the New Creation Cloud Embedded Industrial Controller with an external display screen; STM32 is selected as the lower computer to control the servo motor driver of the mobile carrier; the vision sensor uses the ZED2i (polarized version) high-definition camera, the camera's focal length f is 1.8mm, the base distance b is 120mm, the camera captures the left and right images with a resolution of 1280×720. the overall scheme of the system is shown in Figure 1:

2.2. Technological Route

The technical flow of the machine vision based inter-row position estimation and navigation method for orchard robots at night is shown in Figure 2.

2.3. Improvements to the YOLOv4 Algorithm

2.3.1. YOLOv4 Algorithm

The YOLO family of algorithms is one of the advanced target detection algorithms in the world. The YOLOv4 algorithm uses the CSPDarknet53 as the backbone network for feature extraction, and uses the CBM (consisting of Convolution, Normalization, and Mish activation functions) module and CBL (consisting of Convolution, normalization, and LeakyReLU activation functions) module for feature extraction, and also joins the Spatial Pyramid Pooling Networks (SPPNet) and Path Aggregation Network (PANet) modules for feature extraction, while adding two network modules, Spatial Pyramid Pooling Networks (SPPNet) and Path Aggregation Network (PANet), so that the target detection accuracy of the YOLOv4 algorithm is better than that of the YOLOv3 algorithm. Meanwhile, YOLOv4 algorithm adds image enhancement during training, thus further expanding the dataset.

2.3.2. Improvement of the YOLOv4 Algorithm

In this paper, the YOLOv4 algorithm is used to detect fruit tree trunks. When the orchard robot travels between rows, the environment is relatively complex, and the detection accuracy of fruit tree trunks is an important prerequisite to ensure that the orchard robot can safely navigate, so it is necessary to carry out certain optimization of the YOLOv4 algorithm, so as to improve the detection accuracy of fruit tree trunks.

In this paper, we introduce an efficient channel attention mechanism ECA module, as shown in Figure 3, where k=5 is used as an example. the ECA module maintains the dimensionality of the channel, performs a global average pooling (GAP) operation on the channel, generates the channel weights using a Sigmoid activation function (σ) after one-dimensional convolution, and finally multiplies the channel weights with the original feature layer one by one to obtain a new feature layer. Where the convolution kernel scale k can be adaptively determined and is proportional to the channel dimension. Therefore, if the channel dimension C is known, then the convolutional kernel scale k can be obtained by calculating equation (1). The ECA module adds little computational effort to the algorithm and improves performance in all aspects.

$$k=\psi (C)={\left|\frac{{\log }_2(C)}{\mathrm{a}}+\frac{\mathrm{b}}{\mathrm{a}}\right|}_{\mathrm{odd}}$$

(1)

Where Ι*Ι_odd denotes the nearest odd number toΙ*Ι, a and b are function coefficients, in this paper a takes the value of 2 and b takes the value of 1.

In the YOLOv4 algorithm, the input dataset outputs three kinds of feature layers, 52×52×256, 26×26×512 and 13×13×1024, after feature extraction by the backbone feature extraction network CSPDarknet53, where the 52×52×256 and 26×26×512 are inputted into the PANet module after one convolution and the 13×13×1024 feature layer is inputted into PANet after channel stacking and three convolutions after SPP module. In this paper, the channel attention mechanism ECA module is added to these three feature layers before channel stacking and five convolution operations in PANet. In the PANet module, there are 2 times of up-sampling of the feature layer, this paper adds ECA module after up-sampling, and performs enhanced feature extraction operation on the feature layer after up-sampling, so this paper adds 5 ECA modules to the original YOLOv4 algorithm, and obtains the optimised new algorithm ECA5-YOLOv4 algorithm, and the network model of the ECA5-YOLOv4 algorithm is shown in Figure 4.

2.4. Fruit Tree Trunk Positioning

2.4.1. Fruit Tree Trunk Camera Coordinates

Fruit tree trunks are detected by improving the YOLOv4 model on the left and right images captured by the binocular camera. After detecting the fruit tree trunks, the fruit tree trunks are matched, and when the same fruit tree trunks in the left and right images are matched successfully, the pixel parallax D and image parallax d of the same fruit tree trunks in the left and right images are obtained; according to the principle of triangulation of the binocular camera, the camera coordinates of the fruit tree trunks are obtained (Xc,Yc,Zc). The method in this paper is to project the ground, so the Yc coordinates do not need to be solved to simplify the calculation process. The pixel coordinates of the geometric center of the fruit tree trunk in the left image are (u_L,v_L) and the image coordinates are (x_L,y_L); the pixel coordinates of the geometric center of the fruit tree trunk in the right image are (u_R,v_R) and the image coordinates are (x_R,y_R).

D=u_L − u_R

(2)

d=x_L − x_R

(3)

Binocular camera left and right camera optical center were expressed by O_L, O_R, and O_L as the origin, horizontally to the right for the axis X_L positive direction, vertically down for the axis Y_L positive direction, horizontally forward for the positive direction of the Z_L, the establishment of the camera coordinate system O_L-X_LY_LZ_L; f indicates the focal length of the binocular camera in mm; b indicates the baseline of the left and right cameras in mm; the structure of the binocular camera is shown in Figure 5:

According to the principle of conversion of pixel coordinates to image coordinates and the similarity triangle relationship between image coordinates and camera coordinates there:

$$X_c=\frac{Z_c\left(u_{\mathrm{L}}-u_0\right)}{f_x}$$

(4)

where ${\mathit{f}}_{\mathit{x}}$ denotes the pixel coordinate system u-axis scale factor, respectively; and u₀ denotes the amount of lateral translation of the image coordinate system origin in the pixel coordinate system.

According to the triangulation method there is:

$$Z_c=\frac{f\cdot b}{d}=\frac{f\cdot b}{x_{\mathrm{L}}-x_{\mathrm{R}}}$$

(5)

The horizontal coordinates of the geometric center of the fruit tree trunk in the pixel coordinate system u_L, u_R in the left and right images, respectively:

$$\{\begin{array}{c}{u}_{\mathrm{L}}=\frac{f\cdot x_{\mathrm{L}}}{f_x}+u_0\\ u_{\mathrm{R}}=\frac{f\cdot x_{\mathrm{R}}}{f_x}+u_0\end{array}$$

(6)

From Eqs. (5) and (6), the depth Zc is obtained as:

$$Z_c=\frac{f_x\cdot b}{u_{\mathrm{L}}-u_{\mathrm{R}}}$$

(7)

Substituting Eq. (7) into Eq. (4), there is:

$$X_c=\frac{b\cdot \left(u_L-u_0\right)}{u_L-u_R}$$

(8)

The coordinates (Xc,Zc) of the fruit tree trunk in the planar coordinate system X_LO_LZ_L are obtained from equations (7) and (8). Where, b, u₀,${\mathit{f}}_{\mathit{x}}$ are camera internal parameters, which can be obtained by camera calibration.

2.4.2. Fruit Tree Trunk Coordinate Conversion

Coordinate system and conversion relationship shown in Figure 6, this paper will be the orchard operation platform on the ground projection for the ground coordinate system origin O; binocular camera installed in the front of the vehicle in the center, binocular camera geometric center of the projection of the ground for the point F; $\stackrel{\rightharpoonup }{OF}$ for the ground coordinate system axis Z positive direction, horizontally to the right for the axis of the X-positive direction, the establishment of the ground coordinate system XOZ. Ground coordinate system can be obtained by the camera coordinate system in the planar coordinate system X_LO_LZ_L. The translation of X and Z direction is X₀ and Z₀ respectively. When the camera coordinates are converted to ground coordinates, the Y-axis direction is not to be calculated. Fruit tree trunk in the plane coordinate system X_LO_LZ_L coordinates for (Xc,Zc), in the ground coordinate system coordinates for (Xg,Zg), the conversion relationship as shown in equation (9).

$$\{\begin{array}{c}{X}_g=X_c-X_0\\ Z_g=Z_c+Z_0\end{array}$$

(9)

2.5. Calculation of Navigation Path and Attitude Parameters

When the orchard robot is travelling between rows, the binocular camera is used to locate on the tree trunks, and then the coordinate conversion is used to obtain the sitting mark of a plurality of fruit tree trunks in the ground coordinate system as G_i(X_gi,Z_gi). Among them, the coordinate point of G_i(X_gi,Z_gi)<0 is regarded as being located on the left side of the orchard operation vehicle, and at this time, the left-side sitting mark is G_Lj(X_glj,Z_glj); the coordinate point of X_gi≥0 is regarded as being located on the right side of the orchard operation vehicle, and at this time, the right-side sitting mark is G_Rk(X_grk,Z_grk).

$$\{\begin{array}{c}{\mathrm{G}}_{\mathrm{L}j}(X_{\mathrm{gl}j},Z_{\mathrm{gl}j}),X_{\mathrm{g}i}\text{<}0\\ {\mathrm{G}}_{\mathrm{R}k}(X_{\mathrm{gr}k},Z_{\mathrm{gr}k}),X_{\mathrm{g}i}\ge 0\end{array}$$

(10)

The combination of points on the left and the right is taken to be the midpoint, with a total of j × k midpoints, denoted C_j,k(X_j,k,Z_j,k).

$$\{\begin{array}{c}{X}_{j,k}=\frac{X_{\mathrm{gl}j}+X_{\mathrm{gr}k}}{2}\\ Z_{j,k}=\frac{Z_{\mathrm{gl}j}+Z_{\mathrm{gr}k}}{2}\end{array}$$

(11)

2.5.1. Navigation Path Calculation

The obtained j × k midpoints are fitted with a straight line using the least squares method, and the calculated straight line is the navigation path, noting that the fitted straight line is Z = a₁X + a₀. The calculation process is as follows:

$$\left[\begin{array}{cc}j\times k& {\displaystyle \sum _{j=1,k=1}^{j,k}{X}_{j,k}}\\ {\displaystyle \sum _{j=1,k=1}^{j,k}{X}_{j,k}}& {\displaystyle \sum _{j=1,k=1}^{j,k}{X}_{j,k}^2}\end{array}\right]·\left[\begin{array}{c}{\mathrm{a}}_0\\ {\mathrm{a}}_1\end{array}\right]=\left[\begin{array}{c}{\displaystyle \sum _{j=1,k=1}^{j,k}{Z}_{j,k}}\\ {\displaystyle \sum _{j=1,k=1}^{j,k}{X}_{j,k}{Z}_{j,k}}\end{array}\right]$$

(12)

Find a₀ and a₁ from equation (12):

$$\{\begin{array}{c}{\mathrm{a}}_0=\frac{{\displaystyle \sum _{j=1,k=1}^{j,k}{X}_{j,k}^2}{\displaystyle \sum _{j=1,k=1}^{j,k}{Z}_{j,k}}-{\displaystyle \sum _{j=1,k=1}^{j,k}{X}_{j,k}}{\displaystyle \sum _{j=1,k=1}^{j,k}{X}_{j,k}{Z}_{j,k}}}{(j\times k)·{\displaystyle \sum _{j=1,k=1}^{j,k}{X}_{j,k}^2}-{\left({\displaystyle \sum _{j=1,k=1}^{j,k}{X}_{j,k}}\right)}^2}\\ {\mathrm{a}}_1=\frac{{\displaystyle \sum _{j=1,k=1}^{j,k}{X}_{j,k}{Z}_{j,k}}-{\displaystyle \sum _{j=1,k=1}^{j,k}{X}_{j,k}}{\displaystyle \sum _{j=1,k=1}^{j,k}{Z}_{j,k}}}{{\displaystyle \sum _{j=1,k=1}^{j,k}{X}_{j,k}^2}-{\left({\displaystyle \sum _{j=1,k=1}^{j,k}{X}_{j,k}}\right)}^2}\end{array}$$

(13)

2.5.2. Calculation of Postural Parameters

The position relationship of the orchard operation platform when travelling between rows is shown in Figure 7, and the travelling direction of the vehicle is the same as the positive direction of the Z-axis, which can be expressed by X=0. Therefore the heading angle φ of the vehicle can be obtained by calculating the angle between X=0 and Z=a₁X+a₀; the lateral deviation λ of the vehicle can be obtained by calculating the perpendicular distance from the origin O to the fitted straight line Z=a₁X+a₀. The calculation formulas are shown in (14) and (15):

$$\phi ={90}^{\circ }-\left|\arctan a_1\right|$$

(14)

$$\lambda =\frac{\left|a_0\right|}{\sqrt{1+a_1^2}}$$

(15)

3. Results

3.1. Data Acquisition and Model Training

The experimental dataset collection site was located in the College of Engineering, Nanjing Agricultural University, and 700 fruit tree trunk images were collected. Data enhancement was performed on the images by adding noise and flipping to expand the dataset to 1500 sheets, including 1200 sheets for the training set and 300 sheets for the validation set. The fruit tree trunk part was labelled using LabelImg tool with the category information: tree.

In order to verify the higher accuracy of the ECA5-YOLOv4 algorithm, this paper replaces the five ECA modules in the model with the new attention mechanism modules SENet module and CBAM module. The original YOLOv4 algorithm, ECA5-YOLOv4 algorithm, SENet5-YOLOv4 algorithm and CBAM5-YOLOv4 algorithm were trained for 500 generations using the same dataset respectively. The loss functions of the training models are shown in Figure 8.

Training platform: Intel(R) Xeon(R) E5 2689 2.60GHz CPU, 32G RAM, NVIDIA GeForce GTX 1070 8G graphics card.

The evaluation metrics of training results in this experiment include Precision (P), Recall (R) and Frame rate. P and R are calculated as:

$$P=\frac{T_P}{T_P+F_P}\times 100\%$$

(16)

$$R=\frac{T_P}{T_P+F_N}\times 100\%$$

(17)

Where T_P denotes the number of correctly detected fruit tree trunks in the picture; F_P denotes the number of detection errors in the picture; and F_N denotes the number of missed targets in the picture

Table 1 shows the training results of the original YOLOv4 algorithm, the ECA5-YOLOv4 algorithm, the SENet5-YOLOv4 algorithm and the CBAM5-YOLOv4 algorithm.

From the result analysis, the ECA5-YOLOv4 algorithm, SENet5-YOLOv4 algorithm, and CBAM5-YOLOv4 algorithm have shown improvements in precision rate, recall rate, and frame rate compared to the original YOLOv4 algorithm. Among them, the ECA5-YOLOv4 algorithm has the highest precision rate and recall rate of 97.05% and 95.42% respectively, which are improvements of 5.92%, 2.8%, and 0.22% in precision rate, and 7.91%, 4.41%, 0.28% in recall rate when compared to the YOLOv4 algorithm, SENet5-YOLOv4 algorithm, and CBAM5-YOLOv4 algorithm respectively. The SENet5-YOLOv4 algorithm has the highest frame rate, which is 2.56 fps, 0.44 fps, and 0.53 fps higher than the YOLOv4 algorithm, ECA5-YOLOv4 algorithm, and CBAM5-YOLOv4 algorithm respectively.

3.2. Posture Parameter Determination Test

3.2.1. Binocular Camera Internal Reference Measurement

In this experiment, a ZED2i (polarized version) HD camera was used, the focal length f of the binocular camera was 1.8mm, the base distance b was 120mm, and the resolution of the camera grabbing the left and right images was 1280×720.The internal parameters of the camera were calibrated by using the Software Development Kit (SDK) that comes with the binocular camera. According to the need of fruit tree trunk localization, u0 and result were obtained as 645.25 pixel and 529.88 pixel, respectively.

3.2.2. Experimental Design and Evaluation Indicators

This experiment was carried out in the College of Engineering of Nanjing Agricultural University, and the orchard robot was driven into the rows of fruit trees and stopped at any position in the rows, at which time the real values of the heading angle and lateral deviation of the orchard robot were measured and recorded as φ_t, λ_t; the values measured by the method in this paper are estimated values and recorded as φ_e, λ_e. The difference between the real values of the heading angle and lateral deviation and the estimated values are the error values and recorded as E_φ, E_λ, which were taken as the evaluation indexes of the present experiment. evaluation index, calculated as follows:

$$E_{\phi }=\left|{\phi }_{\mathrm{t}}-{\phi }_{\mathrm{e}}\right|$$

(18)

$$E_{\lambda }=\left|{\lambda }_{\mathrm{t}}-{\lambda }_{\mathrm{e}}\right|$$

(19)

3.2.3. Experimental Results and Analysis

The fruit tree trunks are detected by different models and the positional parameters of the orchard robot are also calculated. The fruit tree trunk results are shown in Figure 9, and the calculation results are shown in Table 2. In Figure 9, the actual heading angle and lateral deviation of the orchard robot from position 1 to position 3 are 150.2° and 0.53 m, 158.5° and 0.48 m, and 160.4° and 0.52 m, respectively. From Table 2, it can be seen that the best estimation of the positional parameters of the orchard robot was made by the ECA5-YOLOv4 model, with the mean values of the errors of heading angle and lateral deviation being 0.57° and 0.02 m. The results were compared with those of the Original YOLOv4, SENet5-YOLOv4, and CBAM5-YOLOv4 models, the mean values of errors were reduced by 1.50° and 0.01m, 0.86° and 0.03m, and 0.60° and 0.03m, respectively.

4. Discussion

In this study, based on the YOLOv4 model, the ECA5-YOLOv4 model is obtained by introducing five ECA attention mechanism modules into the PANet module of the model, and the ECA5-YOLOv4 model can efficiently and accurately detect the trunks of fruit trees, which can provide a guarantee for obtaining the positional parameters of the orchard robot.

(1) Compared with the original YOLOv4, SENet5-YOLOv4, and CBAM5-YOLOv4 models, the accuracy of the ECA5-YOLOv4 model for fruit tree trunks improved by 5.92%, 2.8%, and 0.22%, respectively;

(2) The estimation method of inter-row position parameters of the orchard robot proposed in this paper obtains the mean values of the errors of heading angle and lateral deviation as 0.57° and 0.02m, with low errors, which can provide a theoretical basis for the orchard robot to navigate between the rows of fruit trees.