3D Spatial Perception of Blueberry Fruits Based on Improved YOLOv11 Network

1. Introduction

The blueberry harvesting cycle is short, and fruit ripening is inconsistent. Manual picking is labor-intensive, inefficient, and costly [1]. Mechanical harvesting allows large-scale harvesting using methods such as shaking, or tapping. Although the harvesting efficiency is high, the fruit loss rate is significant, and cleaning and sorting of the fruits are required in the later stage [2]. Utilizing robots for automated and precise picking has become an effective solution to address these challenges. For picking robots, the priority is to perceive blueberry fruits in the surrounding environment, accurately determine their position in space, and provide data support for the robotic arm to grasp the fruits. Consequently, accurately perceiving blueberries is essential for robots to execute precise picking tasks.

With the help of some mature object detection frameworks [3,4], research on blueberry perception has been extensive. These researches primarily enhance the blueberry detection performance in the following ways:

(1) By introducing the attention mechanism, the model could learn more detailed and comprehensive representations of blueberry features.

The Convolutional Block Attention Module (CBAM) [5] was introduced to enhance the feature fusion capability of the blueberry detection network [6]. Feng et al. [7] introduced the SE attention module [8] into the YOLOv9 (the ninth version of the You Only Look Once) [9] network, whereas Zhai et al. [10] combined the SE module with BiFPN [11] in the YOLOv5 network to enhance the model’s attention to relevant blueberry features while reducing the influence of unnecessary information. Gai et al. [12] introduced the Multiplexed Coordinated Attention (MPCA) module in the last layer of the backbone to improve the feature extraction capability during network training. YOLO-BLBE [13] proposed the GhostNet [14] module embedded with the Coordinate Attention (CA) mechanism [15] to replace the backbone of YOLOv5s. Yang et al. [16] designed an NCBAM module in the YOLOv5 network, which improved the blueberry feature extraction capability of the backbone network. Liu et al. [17] replaced the C3 module with the Multi-Head Self-Attention (MHSA) module [18] before the SPPF module, enabling the network to learn more comprehensive feature representations and enhancing its understanding of complex spatial relationships and contextual information in blueberry images.

(2) The real-time performance of the detection network is improved by introducing a lightweight convolution module into the network.

Feng et al. [6] introduced the ShuffleNet [19] module, whereas Yang et al. [16] proposed the C3Ghost module, and Wang et al. [13] introduced the GhostNet [14] module in the YOLOv5 networks. Feng et al. [7] introduced the SCConv [20] module in the YOLOv9c network. These lightweight convolution modules significantly reduced the computational complexity of the detection network, which was very advantageous for deploying detection models on terminal devices with limited computing resources.

(3) Improving the detection capability of small-scale blueberry fruits.

A small object detection layer was introduced to improve the multi-scale recognition capability of blueberries [16]. Liu et al. [17] eliminated the maximum object detection layer from the backbone and head of the network to enhance the model’s detection capacity for small-scale blueberry fruits.

(4) Addressing occlusion issues.

TL-YOLOv8 [12] proposed the Multi-scale Separation and Occlusion-Aware Module (MultiSEAM) in the network. The multi-scale separation module captured the multi-scale features of blueberries by segmenting the input image at different scales, while the occlusion-aware module could infer the possible shape and position of the occluded part by analyzing the features of the surrounding area of the blueberry fruits. Feng et al. [7] introduced the MDPIoU loss function [21] to enhance the accuracy of the blueberry detection network in occluded environments.

Although the detection performance of the methods above is evaluated on private datasets, which cannot be compared horizontally, the results indicate that their detection accuracy can exceed 80%, and some can even exceed 95%, which can guide robotic arms in picking blueberry fruits. However, compared to other fruits, blueberry fruits have a unique characteristic: their skin is thin, which makes them prone to damage during picking. Particularly in the early stages of blueberry fruit ripening, the connection between the fruit stem and the fruit is quite strong, and grasping fruits directly can easily tear the skin, as shown in Figure 1. To avoid this problem, experienced pickers commonly hold the plane perpendicular to the fruit axis and rotate it to detach from the fruit stem. Therefore, when the robotic arm’s picking mechanism picks fruits according to the above pose and action, relying solely on the 3D spatial position of fruits without considering spatial pose is no longer sufficient to satisfy the needs of precise picking by the robotic arm.

Currently, research on the 3D pose perception of fruits is still limited. Li et al. [22] calculated the symmetry axis using the symmetry plane in the point cloud information of fruits and then evaluated the pose of the fruit in 3D space through the symmetry axis. Christopher Lehnert et al. [23] fitted a hyperelliptic model onto segmented sweet peppers and used nonlinear least squares optimization to estimate the 6DOF pose of the sweet peppers. Lin et al. [24] estimated the 3D pose of the pomegranate fruit by determining its center position and the nearest branch information. Zhang et al. [25] extracted the contours of clustered fruit stems through morphological segmentation, and then employed multidimensional vectors along with Gaussian mixture models to extract the poses of fruit stems. Yun et al. [26] proposed a method to determine the 3D pose of a pomelo by analyzing its boundary shape and applying projection geometry principles using binocular stereo-vision technology.

There is still no research on the 3D spatial pose perception of blueberry fruits. To this end, this paper proposes a blueberry detection, 3D spatial localization, and pose estimation method using visual perception, which can be deployed on an OAK depth camera. As shown in Figure 2, this method employs an improved lightweight YOLOv11 network to detect blueberry fruits and their calyx scars from the color image. Next, a 3D coordinate system is established based on the image plane, and the 3D pose of every blueberry fruit is calculated using the pixel coordinates of the fruit center and calyx scar center. Then, the depth estimation module in the OAK camera is used to gather the environmental depth information, which is combined with the sensing data from the three-axis gyroscope to calculate the 3D coordinates of each blueberry fruit relative to the camera. Finally, the 3D spatial position and pose information of blueberry fruits obtained through this method can be transmitted to the subsequent picking decision and control system.

The remainder of the work is organized as follows. In Section II, the blueberry and calyx scar detection method is described, including the blueberry and calyx scar dataset, and the improved lightweight YOLOv11 detection network. Section III introduces the 3D spatial localization and pose calculation method for blueberry fruits, based on the detection results of blueberries and calyx scars. The experimental results are given in Section VI, and the work of this paper is summarized in Section V.

2. Materials and Methods

2.1. Blueberry and Calyx Scar Detection Dataset

2.1.1. Collection and Partition of Blueberry Images

This paper primarily employs three ways to collect blueberry images. The first way is to take live-action shots at the Wolin Blueberry Base in Qingdao, Shandong, China, obtaining 642 original images, including different blueberry images under sunny, cloudy, backlit, and occluded conditions. The second way is to capture online images using web crawling technology, obtaining 1358 images. The third way involves data augmentation of 1500 images collected through the first two ways, generating 3000 images, including translation, flipping, rotation, brightness adjustment, and adding white noise. The partial augmentation is shown in Figure 3.

The dataset, which contains 5000 blueberry images collected through the above three ways, is divided into the training set, validation set, and testing set in an 8:1:1 ratio for detection network training and performance testing. It should be noted that data augmentation can lead to correlations between data. To avoid overfitting in the trained detection network, the 500 samples in the testing set are only the original images that have not undergone data augmentation in the first two collection ways.

2.1.2. Data Annotations

This paper uses a manual annotation software called LabelMe to complete the data labeling task. This software can generate a JSON file containing the image name, size, position, and specific annotation information for each image. During the annotation process, the blueberry fruits are roughly divided into three categories based on maturity: ripe blueberries, semi-ripe blueberries, and unripe blueberries, with colors corresponding to purple red, light red, and cyan, respectively.

The axis information of the blueberry fruit in the image can reflect its 3D pose, which can be determined by connecting the stem or calyx scar center to the fruit center. For clustered blueberries, it is more convenient to detect the calyx scar as their visibility is stronger than that of the fruit stem. Therefore, to obtain the fruit pose, the calyx scar centers of the blueberry fruits are also marked with dots, as shown in Figure 4. Finally, the annotation information of blueberry and calyx scar converted to YOLO format can be represented as:

$$Lable=[cls,c_x,c_y,w,h,k{p}_x,k{p}_y,vis],$$

(1)

where, cls represents the blueberry categories, and bb_R, bb_S, and bb_U represent the three states of blueberry in ripe, semi-ripe, and unripe, respectively; c_x and c_y represent the center coordinates of the bounding box, and w and h represent the length and width of the bounding box. kp_x and kp_y represent the coordinates of the calyx scar center; vis represents the visibility of the calyx scar center, ‘zero’ is visible, and ‘two’ is invisible. When the calyx scar center is invisible, the values of kp_x and kp_y are both ‘zero’.

Blueberries grow in clusters and have many individual fruits, making annotation time-consuming and laborious. To reduce workload, this paper divides the annotation into three stages. In the first stage, 500 images are labeled, and a preliminary model for blueberry and calyx scar detection is trained based on the YOLO network. In the second stage, the trained model is used to detect other unlabeled blueberry images, and the detection results are converted into JSON format as prelabeled information. In the last stage, the inaccurate prelabeled results are refined to generate accurate labeling information in LabelMe and finally converted into annotation files that are readable by YOLO.

2.2. Blueberry and Calyx Scar Detection Network

When utilizing a robot for blueberry picking, it is crucial to improve the picking efficiency. The YOLO series networks represent a seminal real-time detection framework in the object detection field, renowned for its exceptional detection accuracy coupled with minimal computational overhead, making it a ubiquitous choice for a wide array of embedded devices. The YOLO series networks currently have 11 versions and have spawned multiple variants based on them. YOLOv10 [27] and YOLOv11 [28] adopt the consistent dual assignments strategy, which significantly reduces computational overhead and meets the requirements for blueberry detection.

2.2.1. Basic Structure of the YOLOv11n Network

As shown in Figure 5, the YOLOv11n network consists of three parts: Backbone, Neck, and Head. Backbone gradually extracts high-order features of the target by stacking CBS, C3k2, SPPF, and C2PSA modules. Neck has a pyramid structure, including modules such as Concat, Upsample, C3k2, and CBS, which enhance the resolution of high-order features extracted by Backbone and output three feature layers with different scales. Head consists of two branches: one2many and one2one, used for NMS-Free training and inference. The structure of the Head varies depending on different visual tasks such as object detection, pose detection, and instance segmentation.

2.2.2. Improved Lightweight YOLOv11 Network for Blueberry and Calyx Scar Detection

Although the YOLOV11 is designed with different parameter scales to adapt to perception tasks of varying complexity, the parameters of the YOLOV11 network are still redundant for blueberry detection, due to the low diversity of fruit shape (i.e., circular or elliptical) and color (i.e., cyan, red, or purple). Therefore, considering the unique characteristics and requirements for blueberry detection, this paper designs a lightweight detection network based on the consistent dual assignment strategy and YOLOv11 framework, to maximize the accuracy and efficiency of blueberry detection. The specific network structure is shown in Figure 6.

Comparing Figure 5 and Figure 6, this paper has made improvements in the following aspects of blueberry and calyx scar detection:

• Proposing an improved depth-wise separable convolution (DSC) module to replace the CBS and C3K2 modules.

As shown in Figure 7, the DSC module decomposes the traditional convolution into two sequential processes: depth-wise convolution and point-wise convolution. Depth-wise convolution extracts spatial information from each feature channel independently, and point-wise convolution mixes features between channels, significantly reducing network computation while sacrificing some model performance.

Subsequent experimental results show that directly using the DSC module instead of the convolution module in YOLOv11 will significantly decrease the accuracy of blueberry and calyx scar detection. We have analyzed this phenomenon and found that the convolution process generally involves an increase or decrease in feature dimensionality. The order of depth-wise convolution and point-wise convolution will affect the ability of each layer to extract blueberry spatial features and the computational complexity of the network. According to the order of depth-wise convolution and point-wise convolution in Figure 7, the feature dimension increases from 4n to 8n, while the extracted spatial features are only 4n, which is insufficient to learn the spatial features of blueberry fruits well.

Therefore, this paper proposes an improved DSC that can adaptively change the order of depth-wise convolution and point-wise convolution based on whether the feature dimension is increased or decreased. When the feature dimension increases, the reverse DSC depicted in Figure 8 is employed to extract features, conversely, the DSC illustrated in Figure 7 is utilized, ensuring that the network is capable of learning more spatial features of blueberries.

• Removing the attention mechanism module C2PSA at the end of the Backbone network.

Our experiments find that the attention mechanism module C2PSA did not significantly improve the performance of blueberry detection, but instead produced inefficient computations. Therefore, it is removed from our detection network.

• Retaining only the relatively large scale outputs of features with 80 × 80 pixels and 40 × 40 pixels.

Given the comparatively small scale of the blueberry fruits in the image, the detection head only outputs detection results at the larger scales of 80 × 80 pixels and 40 × 40 pixels. This diminishes computational complexity and prompts the model to be more apt at learning the intricate features of blueberries, such as the details around the calyx scar.

2.3. 3D Spatial Localization and Pose Calculation of Blueberry Fruits

2.3.1. 3D Pose Calculation

During blueberry picking, the gripper mechanism must grasp the blueberry fruit in a plane roughly perpendicular to its axis and then rotate around this axis to detach the fruit from the fruit stem, preventing damage to the fruit's skin. Clarifying the pose of blueberry fruits in 3D space is a prerequisite for achieving the above operation.

Assuming the camera is installed at the end of the robotic arm, the ideal picking pose of the blueberry fruit in the image should be as shown in Figure 9. That is, the blueberry fruit is located at the center of the image, and the fruit axis is approximately perpendicular to the image plane. Therefore, we establish a 3D coordinate system relative to the camera using the image plane as the reference plane. This coordinate system takes the image center as the origin, the image plane as the XY-plane, and the direction perpendicular to the image plane as the Z-axis.

As shown in Figure 10, according to the detection results of blueberries and calyx scars, the angles between the fruit axis and the X-axis and Y-axis can be calculated by obtaining the pixel distance l between the fruit center and the calyx scar center, as well as their components Δx and Δy on the X-axis and Y-axis, respectively. Similarly, the angle between the fruit axis and the Z-axis can be calculated by taking the arcsine value of the pixel distance l and the fruit radius r. The calculation formula for blueberry fruits’ 3D pose is as follows:

$$Pos{e}_{3D}=\left\{\begin{array}{c}{\theta }_x=\mathit{arccos}({\displaystyle \frac{△x}{l}})\\ {\theta }_y=\mathit{arccos}({\displaystyle \frac{△y}{l}})\\ {\theta }_z=\mathit{arcsin}({\displaystyle \frac{l}{r}})\end{array}\right.$$

(2)

where, r is the radius of the fruit, due to the shape of blueberry fruits being mostly circular or elliptical in the image plane, half of the average length and width of the bounding box is taken as the approximate pixel radius of the blueberry fruit.

From Figure 10 and Equation (2), it can be seen that the smaller the pixel distance l, the smaller the angle between the fruit axis and the Z-axis, which is closer to the ideal picking pose of the blueberry fruit.

2.3.2. 3D Spatial Localization

To ensure the precise picking of blueberry fruits by the robotic arm, it is crucial to obtain accurate relative positional information between the gripper mechanism and the fruits. The OAK-D-Pro depth camera used in this paper contains a depth estimation module that can directly obtain environmental depth information. The depth estimation module has two ranging methods, namely structured light ranging and binocular depth ranging. Through algorithm fusion, the ranging error of the OAK camera within 4 meters is less than 2%, meeting the requirements for blueberry picking. By combining the depth information, the blueberry detection results, and the sensor data from BMI270 three-axis gyroscopes, the 3D coordinate of each fruit relative to the OAK camera can be ascertained.

In this way, by obtaining the 3D spatial position and pose of blueberry fruits, the robotic arm electronic control system can guide the gripper mechanism to reach the designated position and perform precise picking operations according to specific poses and actions.

3. Results and Discussion

The experiment in this paper is divided into three stages. The first stage is to construct and train a blueberry and calyx scar detection model based on the deep learning framework PyTorch, which can obtain the detection results of blueberry fruit and calyx scar center on the image plane in the form of rectangular boxes and key points, respectively. We compared the detection performance of models with different parameter scales and modules, and visualized some of the detection results. The second stage is to calculate the 3D pose information of blueberry fruits on the constructed 3D coordinate system based on the detection results of the first stage. We evaluated the errors of pose estimation in three directions and analyzed them. In the third stage, the trained detection model is converted into blob format and deployed to the OAK depth camera. It can detect blueberry fruits on terminal devices and obtain the 3D coordinates of blueberry fruits relative to the depth camera with the help of its internal depth estimation module. We analyzed whether the positioning error meets the requirements under the blueberry picking task.

3.1. Blueberry and Calyx Scar Detection Results

3.1.1. Experimental Result Statistics

As shown in Table 1, on the blueberry and calyx scar detection dataset, our method can achieve an accuracy of 99.2% mAP50 and 95.8% mAP50-90 for blueberry detection, as well as an accuracy of 87.0% mAP50 and 86.6% mAP50-90 for calyx scar detection. The time consumption of our model to detect a 640 × 640 pixels image on NVIDIA GeForce GTX 4090 and 13900KF CPU is 0.4 ms, making it suitable for picking robots that require fast and precise blueberry detection. Furthermore, Table 1 also displays the specific performance of blueberry detection at three maturity levels, which shows that the detection accuracy of blueberries at all maturity levels is very high.

In addition, we present some ablation experiments, which evaluate the detection performance of DSC, improved DSC (ours), and CBS modules within the YOLO framework across three parameter scales: N, S, and M. For detailed parameters, please refer to the description in YOLOv11.

Table 2 shows that all models can achieve over 93.5% mAP50-90 for blueberry detection, which we believe meets the demand for blueberry picking. For calyx scar detection, the mAP50-95s of all models are relatively low, only around 84.1% to 88.2%. Specifically, the models with the M scale don’t improve detection performance compared to models with the S scale, but increase the models’ time consumption significantly. Therefore, these models with the M scale are unsuitable for blueberry and calyx scar detection. At the N and S scales, models with DSC modules have the fastest detection speed, but their detection accuracies are only 84.1% and 85.2% mAP50-95. The detection accuracies of models with our improved DSC modules are slightly lower than models with the CBS modules, at 0.7 % and 0.3 % mAP50-95, respectively, but their detection speeds are faster. It can be seen that the improved DSC module proposed in this paper significantly improves detection speed while slightly sacrificing detection accuracy.

For the blueberry picking task, detection speed is more important than detection accuracy, because missed or obstructed blueberries will become detected blueberries as the viewing angle changes during the continuous picking process. Therefore, this paper selects the model with our improved DSC module at the N scale as the blueberry and calyx scar detection model.

3.1.2. Visualization of Detection Instances

To verify the practicality of our detection network, some blueberry images are selected from the testing set for the visual representation of detection results, as shown in Figure 11. As can be seen, our detection network can accurately identify blueberry fruits of different maturity levels, which is beneficial for picking only ripe blueberries and reducing fruit damage rates. At the same time, the point detection of the calyx scar is also basically accurate, which is beneficial for the 3D pose calculation of blueberry fruits.

3.2. 3D Pose Calculation Results of Blueberry Fruits

According to the calculation in section 2.3.1, the 3D poses of blueberry fruits are shown in Figure 12. It should be noted that blueberry fruits with invisible calyx scars will not display their 3D poses.

Based on the recorded detection results of blueberries and calyx scars in the testing set, compared with the ground truth, this paper calculated the average error of the predicted 3D pose estimation of blueberry fruits in three directions. The statistical results are shown in Table 3.

We find that the average error in the Z-axis direction is slightly higher than that in the X-axis and Y-axis directions. In fact, the picking robot is more concerned about the prediction error of the fruit axis in the Z-axis direction (i.e. perpendicular to the image plane), as it determines whether the fruit is in the ideal picking pose. Therefore, this paper conducted a detailed analysis of the errors on the Z-axis and calculated the errors at different angles of the fruit axis (every 5 degrees). The statistical results are shown in Figure 13.

It can be seen that the angle between the fruit axis and the Z-axis is smaller, the error of the prediction angle is also smaller, and conversely, the error is larger. This is primarily due to the fact that, during the calculation of the angle between the fruit axis and the Z-axis, an approximation of the blueberry’s radius is employed by utilizing half of the average length and width of the bounding box. This approximation error will amplify the calculation error of the inverse trigonometric function in Equation (2). However, as the robotic arm moves, the angle between the fruit axis and the Z-axis will become smaller, and the prediction error will also become smaller, eventually converging to a smaller angle range (approximately 10 degrees in Figure 13), which is sufficient to meet the automation and precision picking requirements of robotic arms.

3.3. 3D Localization Results of Blueberry Fruits

The blueberry and calyx scar detection model trained on the server is a pt file. Through the official tutorial provided by OAKChina, this paper converts the pt file into the blob format supported by the OAK depth camera. The OAK camera can directly locate the 3D coordinates of the detected blueberry fruits by using the internal depth estimation module. The schematic diagram of environmental depth estimation and blueberry fruit localization is shown in Figure 14.

We did not conduct experiments on the positioning accuracy of blueberry fruits but evaluated it based on the technical parameters of the OAK camera. According to the parameters provided by OAKchina, the ranging accuracy of the OAK depth camera is less than 2% within 4 meters. Therefore, for the blueberry picking, the theoretical positioning error within 0.5 meters can be controlled within 1 centimeter, and the actual error should be smaller, which can fully satisfy the positioning accuracy requirements for robotic arms. In addition, the model with our improved DSC module deployed on the OAK depth camera can achieve a detection speed of 13.4 FPS (Frames Per Second), while the model with the CBS module has a detection speed of only 9.3 FPS. Therefore, our detection model can better satisfy the requirements of terminal devices for detection efficiency.

4. Conclusions

To satisfy the demand for automated and precise blueberry picking operations, this paper proposes a blueberry fruit detection, 3D spatial localization, and pose calculation method using visual perception. An improved lightweight Yolov11 model proposed by this method can perceive blueberry fruits and calyx scars of different maturity levels in images, and calculate the 3D pose information of blueberry fruits relative to the camera. The detection model deployed to the OAK camera can use its internal depth estimation module and the three-axis gyroscope to obtain the 3D coordinates of blueberry fruits. The experimental results show that the blueberry perception method can achieve a detection accuracy of 95.2% mAP50-95 for blueberry fruits. The 3D positioning error of blueberry fruits can be controlled within 1 centimeter within a range of 0.5 meters, and the 3D pose estimation error can be controlled within 19.2%. This performance can fulfill the requirement for 3D spatial perception of blueberries during automated and precise picking operations by robotic arms. Therefore, the research in this paper is of great significance for improving the picking efficiency of blueberries, reducing picking costs and fruit damage rates, and has an extensive application prospect.