Methods and Software Tools for the Safety Movement of Agricultural Highly Automated Vehicle Based on Deep Learning Methods

1. Introduction

The agricultural sector is among the most profitable and significant industries. As the volume of agricultural production expands, there is growing interest in reducing costs and decreasing the number of workers engaged in labor-intensive or health-hazardous conditions [1]. The use of unmanned ground vehicles for tasks such as spraying, fertilizer spreading, harvesting, plowing, and others is considered one of the most promising methods for improving labor efficiency [2]. Their application in agricultural tasks offers a range of important advantages:

• Increased productivity in field operations through the use of highly automated vehicles capable of continuous operation without breaks and able to work at night;
• Reduction of labor costs by decreasing the number of drivers needed to operate agricultural machinery;
• Lowering the overhead costs of producing agricultural crops, leading to a reduction in the price of the final product and strengthening the company's market position;
• Minimization of harmful effects on machine operators during the application of agrochemicals.

For large-scale implementation of autonomous driving technologies in agriculture, it is important to consider economic efficiency, particularly the cost of equipping unmanned vehicles.

One of the key elements of an unmanned vehicle control system is its ability to perceive the environment. Lidars, which provide high-precision three-dimensional scanning of the surroundings, are one of the most reliable means for this task [3]. However, they are also among the most expensive components of the system, which can significantly increase the overall cost of an unmanned vehicle and make its operation economically impractical for farms, especially those with limited budgets [4].

Using cameras instead of lidars can significantly reduce the costs of control system components [5], thereby making unmanned vehicles more affordable for farmers. This, in turn, can accelerate the adoption and scaling of unmanned technologies in the agricultural sector, contributing to increased efficiency and sustainability of agriculture.

Currently, the main elements of unmanned agricultural vehicles are agronavigators integrated with steering devices. The most well-known solutions are from companies like Patchwork [6], Trimble [7], Ag Leader [8], SMAJAYU [9], solutions from John Deere, and others. Fundamentally, they have a system that, through precise tracking of equipment position, allows for controlled field navigation and accurate chemical application when integrated with executive devices. Additionally, some companies offer kits for retrofitting existing equipment for unmanned driving (Fieldin [10], Blue White Robotics [11], Braun Maschinenbau [12], GPX Solutions [13], ThornTek [14]). Most of them use lidars to prevent collisions with obstacles. Systems from Bear Flag Robotics [15], Sabanto [16], and Cognitive Agro Pilot [17] work with cameras without lidars.

One of the main challenges with control systems that use cameras instead of lidar is the lack of depth information in the images, making it difficult to estimate the distance to objects. From the perspective of constructing a sensory data processing system, combining data from multiple sources can improve the accuracy of the model. This approach is used for localization, for example, by combining GNSS data and the inertial system using Kalman filtering. This article proposes the combination of image analysis results by several deep learning models to obtain a local map of an unmanned vehicle used to implement a safety system for obstacle bypassing and emergency stopping. Such components are object detection and semantic segmentation of the image. To construct a local map that takes into account the distance to objects, a Birds-eye-View (BEV) projection is used. The advantage of this approach is that it can be combined with methods of analyzing spatial data obtained from lidar to improve the accuracy of obstacle distance recognition.

In BEV projection the geometry of objects is restored and the path is planned. The key advantage of an autonomous vehicle control system that uses only cameras and constructs a BEV projection, in addition to a frontal projection, lies in its enhanced perception of depth and distance to objects. This advantage encompasses several positive effects:

• The BEV projection provides a top view of the environment, allowing the system to more accurately determine the depth and distance to objects. This is crucial for safe maneuvering and collision avoidance.
• The BEV projection offers a wide view of the surroundings, enabling the control system to see more than what is possible with just a frontal camera. This improves the decision-making capabilities in complex traffic situations.
• With the depth and distance to objects taken into account, the system can more effectively plan paths and maneuver in complex conditions, such as at intersections or in heavy traffic.
• Utilizing only cameras with the capability of creating a BEV projection potentially reduces the need for additional sensors like LiDAR or radar, which can decrease the cost and complexity of the system.
• The BEV projection may be more adaptable to changes in the environment, such as variations in lighting or weather conditions, compared to other perception methods.

2. Obstacle detection and avoidance method

To implement a complex of obstacle avoidance systems integrated with linear and angular velocity control, it is necessary to implement four modules working together:

• Obstacle detection module and determination of their coordinates.
• Module for determining the space free for movement.
• Module for constructing a local map of the surrounding space of the unmanned vehicle.
• Module for controlling the linear and angular velocity of the unmanned vehicle.

The obstacle detection and coordinate determination module is responsible for detecting obstacles in the surrounding space of the unmanned vehicle. It uses sensor data (images from a depth camera) to detect and localize objects and obstacles using the YOLOv8 library [19]. The data is processed to determine the position and size of obstacles relative to the unmanned vehicle. The effectiveness of this module is critically important for the safety and reliability of the system.

Let's describe the mathematical formulation of the object detection task.

Let there be $I$ - a frame obtained from the camera, used for building models, $O=\left\{t_x,t_y,t_w,t_h,p_0,p_1,p_2,..,p_k\right\}$ – the resulting vector. It is necessary to select such a vector of parameters $a$ of the neural network model $A:F\to O\times N$, which delivers the minimum to the quality functional $J$ on the training sample $X^n$:

$$a^{\mathrm{*}}=\underset{a}{\mathrm{argmin}}J\left(a,X^n\right),$$

(1)

where $N$ – the number of objects in the image;

$n$ – the volume of the training sample;

$\left(t_x,t_y\right)$– the center of the bounding box in the image;

$\left(t_w,t_h\right)$ – the width and height of the bounding box;

$p_0$ – the probability of finding any object in this frame;

$\left(p_1,\dots ,p_k\right)$ – the probabilities of the object belonging to a certain class, $k=80$. The total number of recognizable classes corresponds to the number of classes recognized by the YOLOv8 model, trained on the COCO data set (80 classes).

The module for determining the free space for movement analyzes data about the surrounding space and identifies which areas are free of obstacles and safe for movement using semantic segmentation methods implemented in the fastseg library [20]. Let's present the mathematical formulation of the task for semantic segmentation of the input frame. Image semantic segmentation is the process of assigning a label to each pixel of an image in such a way that pixels with the same labels belong to the same class of objects. Mathematically, the process of semantic segmentation can be described as follows:

Let $I$ be the input frame of size $H\times W$, where $H$ is the height and $W$ is the width of the image. Each pixel in $I$ has RGB values or other color spaces.

The image $I$ is fed into a convolutional neural network (CNN). The CNN consists of a sequence of layers, including convolution layers (Conv), pooling layers (Pool), and fully connected layers (FC). Each layer transforms the input data as follows:

$$Conv:I^{\prime }=\sigma \left(W\mathrm{*}I+b\right)$$

(2)

$$Pool:I^{\prime \prime }=pool\left(I^{\prime }\right)$$

(3)

$$FC:O=W_{fc}^{\prime \prime }I+b_{fc}$$

(4)

where $\mathrm{*}$ is the convolution operation, $\mathsf{\sigma }$ is the activation function (e.g., ReLU), $W$ and $b$ are the weights and biases of the convolution layer, $W_{fc}$ and $b_{fc}$ are the weights and biases of the fully connected layer.

The output $O$ from the CNN is transformed into a pixel classification map $S$ of size $H\times W\times C$ , where $C$ is the number of classes. Each element $S_{i,j,k}$ represents the probability that the pixel at position $\left(i,j\right)$ belongs to class $k$.

The class of each pixel is determined as the class with the maximum probability:

$$P_{i,j}=\underset{k}{argmax}{S}_{i,j,k}$$

(5)

where $P_{i,j}$ is the class label for the pixel at position $\left(i,j\right)$.

Based on $P$, the output segmentation image is formed, where each pixel has a class label corresponding to the object it belongs to. A feature of the proposed method is that obstacle avoidance is based on a map with a "top-down" projection. When transforming the frontal projection into a top-down view, obstacles and objects become elongated and occupy pixels that are behind the object; such pixels should be marked as "free" or "unknown". To solve this problem, in the segmentation image, areas occupied by obstacle objects are replaced with free areas, so that later on the map, areas actually occupied by objects can be marked.

The mathematical description of the process of replacing pixels belonging to certain rectangular areas on the image can be presented as follows. Let $S$ be the semantic segmentation image of size $H\times W$, where each pixel $S_{i,j}$ has a class label $P_{i,j}$. Areas are defined by vectors $R_k=\left(x_k,y_k,w_k,h_k,c_k\right)$, where $x_k$ and $y_k$ are the coordinates of the middle point of the area$w_k$and $h_k$ are the width and height of the area respectively, and $c_k$ is the class of the object to be replaced. Let there be $N$ such vectors.

For each area $R_k$, the following steps are performed:

• The boundaries of the area are determined:

  $$x_{start}=x_k-\frac{w_k}{2};x_{end}=x_k+\frac{w_k}{2}$$

  (6)

  $$y_{start}=y_k-\frac{h_k}{2};y_{end}=y_k+\frac{h_k}{2}$$

  (7)
• For each pixel in this area, if the pixel belongs to class $c_k$, its value is replaced:

  $$S_{i,j}=100;for{x}_{start}\le i<x_{end},;y_{start}\le j<y_{end},;if{S}_{i,j}=c_k$$

  (8)

The result of this operation is a modified segmentation image $S^{\prime }$, where pixels in the specified areas belonging to the indicated class are replaced with the value 100.

This process can be implemented programmatically using loops to iterate through the areas and the corresponding pixels within each area. It is important to ensure that the indices and boundaries remain within the dimensions of the image.

The module for constructing a local map of the surrounding area and path planning creates a dynamic map of the surrounding area, including detected obstacles and safe routes using the "Inverse Perspective Mapping" method. This method is used in image processing and computer vision to transform images obtained from a perspective projection into images as if they were obtained from an orthographic (or planar) projection.

To transform the entire image into a Birds-eye-view (BEV) projection using a homography matrix [21], it is necessary to apply the transformation to each point of the image. In this case, the transformation can be described in the following general form:

• Let $S^{\prime }$ be the original image, and $S^{\prime \prime }$ be the transformed image.
• For each point $\left(x,y\right)$ in the image $S$, the homography matrix $H$ is applied to find the corresponding point $\left(x^{\prime \prime },y^{\prime \prime }\right)$ in the image $S^{\prime \prime }$:

  $$\left(\genfrac{}{}{0pt}{}{x^{\prime \prime }}{\begin{array}{c}{y}^{\prime \prime }\\ w^{\prime \prime }\end{array}}\right)=H·\left(\genfrac{}{}{0pt}{}{x}{\begin{array}{c}y\\ 1\end{array}}\right)$$

  (9)
• After applying the homography matrix, the coordinates are normalized to convert them into Euclidean coordinates:

  $$x_{\mathrm{norm}}=\frac{x^{\prime \prime }}{w^{\prime \prime }}$$

  (10)

  $$y_{\mathrm{norm}}=\frac{y^{\prime \prime }}{w^{\prime \prime }}$$

  (11)
• The pixel value at the new coordinates $\left(x_{\mathrm{norm}},y_{\mathrm{norm}}\right)$ is assigned to the corresponding pixel in the image $S^{\prime \prime }$. This process is repeated for each point of the original image $S^{\prime }$ to form the complete image $S^{\prime }’$ in the Birds-Eye-View projection.

To determine the position of the vehicle itself on the local map, it is assumed that the point with coordinates $\left(w/2,h\right)$ is the vehicle's position point $\left(x_{ego},y_{ego}\right)$. To determine the vehicle's coordinates on the local map $\left(x_{ego}’,y_{ego}’\right)$ using the homography matrix, the function $bev$ is used:

$$bev\left(x,y\right)=\left(\frac{H_{11}x+H_{12}y+H_{13}}{H_{31}x+H_{32}y+H_{33}},\frac{H_{21}{x}_{ego}+H_{22}y+H_{23}}{H_{31}x+H_{32}y+H_{33}}\right)$$

(12)

$$\left(x_{ego}’,y_{ego}’\right)=bev\left(x_{ego},y_{ego}\right)$$

(13)

In the next step, objects previously removed are marked on the local map so that their geometric characteristics correspond as closely as possible to their real values [22]:

• Let $S^{\prime \prime }$ be the Birds-Eye-View (BEV) projection image, where areas occupied by recognized obstacles have already been replaced with the value 100.
• Each recognized object is defined by a bounding rectangle with parameters: the rectangle's center $(x_{\mathrm{mid}},y_{\mathrm{mid}})$, width $w$, height $h$, and the object class $c$.
• For each bounding rectangle, the coordinates of the two lower points $\left(P_1,P_2\right)$ in the BEV projection are calculated:

  $$P_1=bev\left(x_{\mathrm{mid}}-\frac{w}{2},y_{\mathrm{mid}}+\frac{h}{2}\right)$$

  (14)

  $$P_2=bev\left(x_{\mathrm{mid}}+\frac{w}{2},y_{\mathrm{mid}}+\frac{h}{2}\right)$$

  (15)

• The area for replacement is calculated as follows:

  $$-\mathrm{Left}\mathrm{boundary}\text{: }{x}_{\mathrm{left}}=x_{\mathrm{mid}}-\frac{w}{2}$$

  (16)

  $$-\mathrm{Right}\mathrm{boundary}\text{: }{x}_{\mathrm{right}}=x_{\mathrm{mid}}+\frac{w}{2}$$

  (17)

  $$-\mathrm{Bottom}\mathrm{boundary}\text{: }{y}_{\mathrm{bottom}}=y_{\mathrm{mid}}+\frac{h}{2}$$

  (18)

  $$-\mathrm{Top}\mathrm{boundary}\text{: }{y}_{\mathrm{top}}=y_{\mathrm{bottom}}-2w$$

  (19)
• For each point $\left(x,y\right)$ in the specified area on the image $S^{\prime \prime }$, if it is within the boundaries of the area:

  $$x_{\mathrm{left}}\le x\le x_{\mathrm{right}}$$

  (20)

  $$y_{\mathrm{top}}\le y\le y_{\mathrm{bottom}}$$

  (21)

  the pixel value is replaced with the object class $c$.
• These steps are repeated for each recognized object from $O$ on the original image $I$.

The module for controlling the linear and angular velocity of the autonomous vehicle is responsible for executing control commands based on the output of the path planning module. It regulates the speed and direction of the vehicle's movement, ensuring adherence to the planned route and avoiding collisions. To achieve this, the module uses PID (Proportional-Integral-Derivative) control for precise angular velocity management.

Based on the obtained map, the module plans an optimal route for the autonomous vehicle, considering set goals and constraints. Route planning is carried out using the RRT* algorithm [23] on the local map, where areas free for movement are marked with the value 100, and all other values represent occupied cells. The target point is determined by matching the local map and the global map, considering the vehicle's position on the global map, defined by the tuple $\left(x_{global},y_{global},\theta \right)$.

Initially, the target direction of movement is determined using the first five points of the path $P_i=\left(x_i,y_i\right)for(i=\mathrm{1,2},\dots ,5)$, which form part of the vehicle's path. Vectors between these points are calculated: $\overrightarrow{V_i}=P_{i+1}-P_i$ for $i=1,2,3,4$, and the average direction of these vectors is determined.

$${\mathsf{\theta }}_{\mathrm{avg}}=\frac{1}{n-1}{\sum }_{i=1}^{n-1}\mathrm{atan}2\left(y_{i+1}-y_i,x_{i+1}-x_i\right)$$

(22)

This formula assumes that ${\mathsf{\theta }}_{\mathrm{avg}}$ will be the average value of angles between consecutive pairs of points on the path.

The PID controller is used to minimize the difference between the current orientation angle of the vehicle $\mathsf{\theta }$ and the target angle ${\mathsf{\theta }}_{\mathrm{avg}}$.

The error is defined as $e\left(t\right)={\theta }_{\mathrm{avg}}-\mathsf{\theta }$.

The proportional controller calculates the corrective action $u\left(t\right)$ as:

$$u\left(t\right)=K_p\cdot e\left(t\right)+K_i\cdot {\int }_0^{t}e\left(\mathsf{\tau }\right)d\mathsf{\tau }+K_d\cdot \frac{de\left(t\right)}{dt}$$

(23)

where $K_p,K_i,K_d$ are the coefficients of the proportional, integral, and differential components of the PID controller, respectively. These coefficients are determined experimentally.

This algorithm allows the autonomous vehicle to automatically adjust its course to follow the designated path while avoiding obstacles, using the direction information determined from the five path points and the PID controller for precise angle control. An emergency stop is executed if no path is found. The vehicle remains stationary until the objects obstructing the path are cleared.

4. Discussion

This article describes models and software that enable an unmanned vehicle to circumnavigate obstacles and stop urgently if a collision cannot be avoided. The main idea is to combine the results of processing an RGB camera frame by several methods (object detection, semantic segmentation, inverse perspective mapping), the combined results of which allow for greater accuracy in determining the distance to objects. The result of data processing is an annotated local map, which is used for path planning and determining the steering wheel angle, which determines the angular velocity using a PID controller. To smooth changes in angular velocity, the data are smoothed with a low-pass filter.

The implemented method is relevant in the tasks of designing unmanned ground vehicles, especially those retrofitted for autonomous driving, where the low cost of the equipment used is important, as only cameras are used as sensors.

Further development of methods is planned. It is important to consider information not only from the front camera but also from the side cameras. This can be done using 360-degree cameras. Then an important aspect of safety and route planning is building a full map using SLAM. Since depth information is inaccurate, the development of a SLAM algorithm using semantic information about the objects surrounding the car is promising.

Overall, the use of only cameras allows building a control system for an unmanned vehicle on cheap sensors. Such systems are less likely to be used in urban environments, where the complexity of the road scene is high, but can be useful in agriculture or in manufacturing, where traffic is not so complex.