M18K: A Multi-Purpose Real-World Dataset for Mushroom Detection, 3D Pose Estimation, and Growth Monitoring

1. Introduction

The integration of Artificial Intelligence (AI)-driven automation into agricultural processes has gained significant momentum, offering opportunities for increased efficiency and sustainability in various farming practices [1,2,3,4]. Among these, the automatic harvesting of edible mushrooms in large farms holds substantial potential for streamlining operations and reducing labor-intensive tasks.

While computer vision and object detection technologies have seen significant advancements, a major challenge remains the lack of publicly available datasets specifically tailored for mushroom detection. This limitation impedes systematic evaluation and comparison of different mushroom detection algorithms. Synthetic mushroom datasets [1,5] have been introduced to aid model pre-training; however, they often fail to capture real-world complexities—such as varied lighting conditions, diverse growth stages, and cluttered backgrounds—necessitating fine-tuning on real images for robust performance. Additionally, in practical agricultural settings, accurate 3D orientations can facilitate automated harvesting by providing robotic arms with precise information about the mushroom’s position and tilt [6,7,8]. Furthermore, reliable 3D pose data are essential for tasks volumetric measurements where a mushroom’s overall shape and orientation must be understood in detail. To the best of authors’ knowledge, there are currently no other datasets of real mushrooms providing ground truth data for various modalities and purposes such as instance segmentation and object detection, yield estimation, and 3D pose estimation.

To address this gap, we present a comprehensive real-world mushroom dataset featuring 2,000 images dedicated to object detection, instance segmentation, and 3D pose estimation, plus an additional 3,838 images for yield estimation. The 2,000 images contain over 100,000 mushroom instances and the 3,838 images for yield estimation contain 8 different timelapse images of unique scenes covering the entire growth period. Our dataset’s growth monitoring and yield estimation subset is an augmented version of the data used in our previous work[9] .All images in the object detection subset are RGB-D pairs captured using an Intel RealSense D405 RGB-D camera, and the yield estimation subset’s images were captured by a TLC200 Pro timelapse camera. The dataset focuses on two commonly cultivated varieties—white button mushrooms and baby bella (cremini) mushrooms—and includes detailed 3D pose annotations, generated first by manual labeling of a small subset of 50 images, followed by pseudo-labeling the remaining 3D instances through a trained pose estimation model proposed in[10], and a final round of manual 3D label refinement. Each image captures various lighting conditions and growth stages, covering a broad range of orientations and backgrounds. Additionally, the second subset of 3,838 images includes repeated captures of the 8 different scenes for yield estimation, enabling measurement of growth rates over time and supporting a yield prediction model.

Figure 1 illustrates sample dataset images, corresponding ground truth instance segmentation masks, and depth images for both Baby Bella (BB) and White Button (WB) mushrooms. Figure 2 presents 3D point clouds of brown and white mushroom scenes, showing how our RGB-D data can be leveraged for 3D spatial and volumetric analyses.

Our main contributions are summarized as follows:

• We introduce a publicly available, real-world mushroom dataset of 2,000 RGB-D images for detection-related tasks, instance segmentation, and 3D pose estimation, as well as 3,838 images for growth monitoring.
• We provide comprehensive ground truth labels for object detection, instance segmentation, 3D pose estimation, and time-series data for yield forecasting.
• We establish a benchmark by evaluating state-of-the-art mushroom detection algorithms, facilitating reliable performance comparison.

In the subsequent sections of this paper, we describe the annotation methodology in detail, present descriptive statistics of the dataset, evaluate benchmark algorithms, and discuss potential applications of our dataset for automated harvesting, growth estimation, and other smart farming tasks.

2. Literature Review

Several articles and papers have been published regarding the application of image processing and machine learning algorithms for mushroom detection, quality assessment, disease detection, growth monitoring, and automatic harvesting. These works can be grouped into three main categories as follows.

2.1. Mushroom Detection and Localization

There is a growing body of literature dedicated to mushroom detection and localization, each focusing on different algorithms for mushroom detection and localization, using techniques from classical image processing to advanced deep learning and computer vision.

The YOLO (You Only Look Once) algorithm, a significant advancement introduced in 2016, is a cornerstone in this field. It simplified object detection into a single regression problem, directly predicting bounding boxes and class probabilities, thus enhancing efficiency in detection tasks. Several studies adapted the YOLO algorithm for mushroom detection, yielding notable results.

Wei et al. [20] developed Recursive-YOLOv5, significantly surpassing the original YOLOv5 with 98% accuracy, though at the cost of almost doubling the network’s parameters. Wang et al. [19] introduced Mushroom-YOLO, an improved version of YOLOv5, achieving a mean average precision of 99.24% by integrating Convolutional Block Attention Module (CBAM) and Bidirectional Feature Pyramid Network (BiFPN) into the architecture. Olpin et al. [15] used Region-Based Convolutional Networks for mushroom detection, finding that the Region-based Convolutional Neural Network(RCNN) model was more accurate (92.162%) than the Region-based Fully Convolutional Network (RFCN) (87.631%), albeit slower. Yang et al. [21] enhanced Agaricus Bisporus recognition using Mask RCNN, achieving an Average Precision (AP50) of 95.061%, though the operation was slow. Retsinas et al. [16] created a vision module for 3D pose estimation of mushrooms, resulting in high retrieval scores (up to 99.80% Mean Average Precision at 25% Intersection over Union) and accurate pose estimation (mean angle error as low as 8.70°). Lee et al. [12] automated the selective harvesting process, using the Faster R-CNN model for identification and a 3D point cloud for segmentation, achieving 70.93% accuracy in maturity identification. Baisa and Al-Diri [2] focused on robotic picking, detecting, and estimating the 3D pose of mushrooms with high precision (98.99%) and recall (99.29%), using RGB-D data. These studies demonstrate advancements in mushroom detection and localization, showcasing various algorithmic improvements. While they present significant progress, challenges such as computational efficiency, dataset quality, and the need for further accuracy and speed enhancements remain.

2.2. Mushroom Growth Monitoring And Quality Assessment

Lu et al. [23] developed a system using YOLOv3 for monitoring mushroom growth in greenhouses. The system could estimate mushroom cap size, count, growth rate, and predict harvest times. It demonstrated effectiveness in identifying growth patterns and provided real-time updates via a mobile interface. Lu and Liaw [24] introduced an image measurement algorithm utilizing YOLOv3, coupled with a Score-Punishment algorithm for accurate mushroom cap diameter measurement. The method excelled in accuracy over traditional methods but faced challenges with unclear mushroom contours or soil particles on caps. Nadim et al.’s image processing system [14] assessed mushroom quality based on color, area, weight, and volume, using data mining, neural networks, and fuzzy logic. It achieved a 95.6% correct detection rate but required image pre-processing to counteract quality issues from normal imaging conditions.

Moysiadis et al. [13] automated the monitoring and classification of oyster mushrooms in greenhouses. They used YOLOv5 for detection and classification, and Detectron2 for tracking growth, showing potential improvements in harvesting times but faced difficulties in detecting small mushrooms. Wang et al. [18] developed an automatic sorting system for white button mushrooms, using image processing to measure the pileus diameter. The system achieved high grading speed and accuracy, significantly improving over manual grading methods.

Benhaddou et al. [9] implemented a computer vision system to estimate mushroom yield and quality in a commercial farm setting, utilizing the Circular Hough Transform algorithm. They achieved detailed tracking of mushroom growth across 1960 frames over 20 days, demonstrating the system’s ability to monitor size distribution and development trends. This technological application offers a promising tool for optimizing cultivation and harvesting strategies by providing precise data on mushroom growth patterns and potential yield estimations.

2.3. Mushroom Disease Detection

Zahan et al. [22] employed deep learning models to classify mushroom diseases, with ResNet15 showing the best performance in accuracy and precision. The study emphasized the potential of deep learning in agricultural management. Vizhanyo and Felfoldi [17] used advanced image analysis to distinguish diseased spots on mushroom caps. Their method showed an 85% correct classification ratio, effectively differentiating between brown blotch and ginger blotch diseases.

Jareanpon et al. [11] developed an intelligent farm system for real-time detection of fungal diseases in Lentinus. The system integrated environment control, an imaging robot, and a deep learning-based prognosis system, achieving high precision in maintaining optimal conditions and disease detection.

In conclusion, the literature review section of this paper highlights a range of innovative approaches to mushroom detection, quality assessment, growth monitoring, and disease detection using image processing and machine learning algorithms. Table 1 lists these studies and their results. The studies vary in focus, ranging from detection and localization to quality and disease detection, each contributing valuable insights and advancements in the field. Notably, while these works demonstrate significant progress, there remains a gap in the standardization of methodologies and results comparison, compounded with challenge of data drift [25]. This gap is primarily due to the lack of a publicly available benchmark dataset. Based on the current literature [26], and to the best of the authors’ knowledge, there are no publicly available annotated edible mushroom datasets. A standardized dataset would provide a common ground for evaluating different algorithms and techniques, ensuring comparability and consistency across studies. The availability of such a dataset would benefit the field, allowing for more robust and reliable comparisons of methods and results, driving forward the development of more efficient and effective solutions for mushroom cultivation and processing in the agricultural sector.

3. Labeling Process

3.1. Detection and Segmentation

The images were initially labeled using the Segment Anything Model (SAM) automatic mask generation feature [27]. This feature provides several configurations and post-processing steps to ensure the best quality of automatically generated masks for specific purposes. For instance, a grid of $n\times n$ (with $n=32$ by default) positive points is placed on the original image and randomly cropped versions of it, and the generated masks are filtered by their confidence scores and stability within different confidence thresholds. The automatic mask generation algorithm considers a mask to be stable if thresholding the probability map at $(0.5-\delta )$ and $(0.5+\delta )$ results in similar masks. Additionally, masks can be filtered by their minimum and maximum area, and a certain maximum intersection over union (IOU) threshold can be set to avoid generation of overlapping masks. Nevertheless, these configurations do not guarantee the generation of perfect ground truth masks and can only be used as a starting point.

These generated masks were meticulously edited manually during several steps to guarantee the quality of the dataset ground truth masks. Initially, all the false positive masks were removed and the missing masks were added. Later, the intersection of all pairs of masks in the same image was calculated for all the images in the dataset, and the overlapping masks were removed manually.

3.2. 3D Pose Estimation

In addition to 2D instance segmentation, a subset of the images was labeled with 3D bounding boxes and rotation annotations. We began by manually labeling an initial subset of 50 images with precise 3D boxes; we then applied the pose-estimation pipeline proposed by [10] to generate pseudo-labels for the remaining images. A final manual refinement phase ensured the accuracy of these 3D labels, particularly in cases where overlapping mushrooms or complex poses challenged the automated predictions.

Each 3D annotation contains

• the object centre $(x,y,z)$ in pixel coordinates of the depth image,
• the bounding-box dimensions $(w,h,d)$ in pixels (width, height, depth), and
• the object orientation in Euler angles $(\varphi ,\theta ,\psi )$—roll, pitch, and yaw in degrees.

Following [28], we convert the Euler angles to the 6D Gram–Schmidt (GS) representation before feeding them to the network and again after inference. Given roll $\varphi$, pitch $\theta$, and yaw $\psi$ (in radians), we first form the rotation matrix

$$R=R_z\left(\psi \right)R_y\left(\theta \right)R_x\left(\varphi \right),$$

where

$$R_x\left(\varphi \right)=\left[\begin{array}{ccc}1& 0& 0\\ 0& cos\varphi & -sin\varphi \\ 0& sin\varphi & cos\varphi \end{array}\right],R_y\left(\theta \right)=\left[\begin{array}{ccc}cos\theta & 0& sin\theta \\ 0& 1& 0\\ -sin\theta & 0& cos\theta \end{array}\right],R_z\left(\psi \right)=\left[\begin{array}{ccc}cos\psi & -sin\psi & 0\\ sin\psi & cos\psi & 0\\ 0& 0& 1\end{array}\right]$$

The GS representation simply concatenates the first two columns of R:

$$\mathbf{g}={\left[R_{:,1}^{\top }{R}_{:,2}^{\top }\right]}^{\top }\in {\mathbb{R}}^6$$

At inference time, we recover the rotation by orthonormalising $\mathbf{g}$:

$${\widehat{\mathbf{r}}}_1={\displaystyle \frac{{\mathbf{g}}_{1:3}}{{\parallel {\mathbf{g}}_{1:3}\parallel }_2}},{\widehat{\mathbf{r}}}_2={\displaystyle \frac{{\mathbf{g}}_{4:6}-\left({\widehat{\mathbf{r}}}_1^{\top }{\mathbf{g}}_{4:6}\right){\widehat{\mathbf{r}}}_1}{{∥{\mathbf{g}}_{4:6}-\left({\widehat{\mathbf{r}}}_1^{\top }{\mathbf{g}}_{4:6}\right){\widehat{\mathbf{r}}}_1∥}_2}},{\widehat{\mathbf{r}}}_3={\widehat{\mathbf{r}}}_1\times {\widehat{\mathbf{r}}}_2,$$

yielding $\widehat{R}=\left[{\widehat{\mathbf{r}}}_1{\widehat{\mathbf{r}}}_2{\widehat{\mathbf{r}}}_3\right]\in \mathrm{SO}\left(3\right)$ and, if needed, the corresponding Euler angles.

This 6D representation avoids the discontinuities and singularities of Euler angles, leading to smoother optimisation during training and more stable pose predictions at inference.

3.3. Growth Monitoring

To facilitate the study of mushroom development over time, consecutive images of each mushroom were captured at multiple stages of growth. Each labeled instance retains a consistent ID across frames, allowing precise tracking and comparison of changes in shape and size throughout different growth phases. During annotation, we used manual inspections to refine the growth stage boundaries, ensuring accurate and reliable labeling for each mushroom across multiple developmental stages.

For the subset of images dedicated to yield estimation, we augmented and labeled the data from a similar work[9] and performed additional labeling to track mushroom growth over time. Specifically, the same scene was captured repeatedly, and each mushroom in the scene was assigned a consistent identifier across all time points to facilitate growth rate calculations. This subset of our dataset can be utilized to support the development of temporal models for yield forecasting and growth monitoring.

4. Data Description

Our dataset includes 2,000 RGB-D images for object detection, instance segmentation, and 3D pose estimation, plus an additional 3,838 RGB-D images for yield estimation and growth monitoring. In the main subset, we have both baby bella (BB) and white button (WB) mushroom images, whereas the yield estimation subset only consists of repeated captures of WB mushrooms over time. A sample of the main subset’s images, depth maps, and label masks is plotted in Fig. Figure 1. Both the RGB images and the depth images in the main subset have a resolution of 1280*720 pixels, and hence, no alignment will be required for using the RGB and depth images as a single RGB-D input. Sample 3D point cloud images of WB and BB mushrooms are plotted in Fig. Figure 2. The camera was placed at vertical distances of 27 cm (about 10.63 in) and 15 cm (about 5.91 in) above the cultivation beds during image acquisition of baby bella and white button mushrooms, respectively.

The total number of mushroom instances in the 2,000-image subset is approximately 70,000 for WB and 30,000 for BB mushrooms. The additional 3,838 images for yield estimation contain 8 different timelapse scenes observed repeatedly with time-steps of 15 minutes to track growth over time. These instances include a wide range of mushroom sizes for both categories, making our dataset suitable for training detection models for various tasks such as mushroom growth monitoring and automatic mushroom harvesting as well as 3D pose estimation. Histograms of mask areas and bounding box diagonal length of mushroom instances are plotted in Figure 4, and scatter plots of instance mask area versus bounding box diagonal length are shown in Figure 3. As illustrated in the scatter plots, some instances lie above the overall trend line, indicating mushrooms that are partially occluded by neighboring mushrooms. These occlusions lead to crescent-shaped masks rather than fully convex shapes. Both Figure 3 and Figure 4 show comparatively larger mask areas for white button mushrooms, explained by the shorter distance (15 cm) between the camera and the cultivation bed during WB image acquisition.

To study the separability of the two mushroom classes in different color spaces, we have plotted 3D scatter graphs (not shown here in detail) for RGB, HSV, and LAB in Figure 5, using the mean values of each instance’s mask in all color channels. The two clusters appear more intertwined in the RGB color space, whereas HSV and LAB yield more distinct separations of WB and BB mushrooms.

Furthermore, our main subset including the 2,000 images has been annotated with 3D bounding boxes and rotation information. These 3D labels extend the utility of the dataset for pose estimation tasks and can be used to study volumetric aspects of mushroom growth. For the additional yield-estimation subset, repeated captures of the same scene allow researchers to investigate the temporal evolution of mushroom size and count, thus enabling more accurate yield forecasts.

Figure 6 illustrates two scatter plots showing the spread of rotation and dimension values for a randomly selected subset of our 3D labels having 2500 instances.

The yield estimation subset of our dataset includes time-lapse images of 8 different scenes. For each scene, the images start at the stage where the mushrooms have not appeared yet, until the time when there are no mushrooms left in the scene, taking images every 15 minutes. These images were then labeled using a trained YOLO-V8 model and each mushroom instance was assigned a unique identifier throughout the entire growth period. Figure 7 illustrates a time-series plot where the x-axis represents time and the y-axis represents the mushroom size in millimeters; The average size for each of the 8 scenes is shown, each with a different color. It can be observed that the average time required for a mushroom to reach full maturity is approximately five days. Furthermore, the abrupt decline in the size curve observed on the fifth day can be attributed to the partial harvesting of larger mushrooms. This intervention exposed smaller mushrooms that were previously occluded, thereby altering the recorded size distribution. A sample of the labeled growth monitoring data from a single scene is illustrated in Figure 8.

5. Benchmarking and Results

In this section, we provide a detailed analysis of several object detection and instance segmentation algorithms tested on our mushroom dataset, along with the results of the pose-estimation pipeline and analysis of the yield-estimation data. The models were trained on a DGX-2 cluster with sixteen NVIDIA V100 GPUs (32 GB each), providing ample compute for large-scale experiments.

For both detection and 3D pose-estimation tasks, the data were split 70 %/10 %/20 % into train, validation, and unseen test sets, respectively; all metrics reported below refer exclusively to the test split to ensure unbiased generalisation estimates and guard against overfitting.

Every network (detection, segmentation, and pose) was optimised with AdamW, an initial learning rate of $1\times {10}^{-4}$, and a cosine-annealing scheduler that decays the learning rate to zero over the full training horizon.

• Detection / segmentation. Instead of explicit weight regularisation, we apply online data augmentation—random crops (up to 20 % of area), in-plane rotations of $\pm {15}^{\circ }$, and random translations of $\pm 10\%$ along both axes—to diversify the training distribution and reduce overfitting.
• 3D pose estimation. Following the protocol in [10], each mushroom instance’s point cloud is randomly down-sampled to 1,024 points at every iteration; this stochastic subsampling encourages robustness to point-density variation and further mitigates overfitting.

5.1. Object Detection and Instance Segmentation

All object detection and instance segmentation models underwent thorough evaluations, with their performance presented in Table 2. Our assessment utilized metrics such as F1 score, average precision (AP), and average recall (AR) for all models regardless of the task being detection or instance segmentation.

The data in Table 2 provides details of the performance of a variety of models with distinct backbone architectures, benchmarked on object detection and instance segmentation tasks. These models were trained on image resolutions of 640 * 360 and 1280 * 720. The MaskRCNN with a ResNet-50 backbone surpassed the other networks in instance segmentation, achieving a mean Average Precision (mAP) of 0.866 and an Average Recall (AR) of 0.896 when processing 1280-sized RGB images. When adapted for RGB-D input, which increased the network’s parameters over three times the original, there was no enhancement in performance. This lack of improvement may stem from the mandatory re-initialization of weights following the modifications to accommodate the added depth channel, which precluded the use of existing pre-trained weights that typically contribute to superior performance via fine-tuning on a specific dataset. Furthermore, the expanded parameter count makes the network less suitable for real-time use where hardware resources are constrained. Due to the lack of improvement in performance while adding the depth channel, other networks were not tested with RGB-D images. Furthermore, The RT-DETR Large model exhibited superior object detection performance on 1280-sized images, with a mAP of 0.907 and an AR of 0.923. In contrast, both RT-DETR variants performed poorly on 640-sized images, potentially due to the network not being as effective at capturing the necessary details at a reduced resolution given the small number of dataset images. Figure 9 provides a visual comparison of the prediction results between BB and WB images, generated by Mask R-CNN with a ResNet-50 backbone trained on RGB images of size 1280. Fig. Figure 9(a) and Fig. Figure 9(b) show the prediction results on RGB images for BB (Baby Bella) and WB (White Buttons) mushroom instances, respectively.

Overall, the results indicate that while all models performed well across various metrics, the choice of the backbone and the balance between the number of parameters and input channels play a significant role in the trade-off between accuracy and computational efficiency. These insights can inform the development and optimization of mushroom detection algorithms tailored for agricultural automation.

5.2. 3D Pose Estimation

To assure the usability of our 3D data, we trained and evaluated a 3D pose estimation pipeline on the bounding boxes and rotation annotations. Our 3D pose pipeline (based on our previously published approach in [10]) predicts roll, pitch, and yaw angles for each mushroom instance. To measure the accuracy of these predictions, we used a mean ${\theta }_{difference}^{\circ }$ metric, which averages the absolute degree differences between predicted and ground-truth angles across all mushrooms. Based on the findings of our previous study, we down-sampled each mushroom instance’s point cloud to include 1,024 points and used Geodesic loss function [37] along with a 6D Gram–Schmidt rotation representation [28] and achieved a mean ${\theta }_{difference}^{\circ }$ = $6.{61}^{\circ }$ demonstrating the usability of our data and 3D annotations in tasks such as pose estimation. Figure 10 provides a visual comparison of ground-truth versus predicted oriented bounding boxes on a random scene. The predictions closely match the ground-truth orientations, suggesting that the model successfully captures the poses of mushrooms in real-world scenes.

5.3. Yield Estimation

To facilitate the development of yield estimation models, we provide a curated subset of 3,838 images encompassing eight distinct scenes, spanning the full growth cycle of mushrooms—from initial emergence from the soil to full maturity and harvest. Figure 11 presents the average mushroom size in millimeters over time, where the solid line represents the mean size and the shaded region denotes the minimum and maximum size bounds. We can observe an abrupt drop in the sizes approximately at day five due to the partial harvesting of larger mushrooms, which enables the appearance of smaller, previously occluded mushrooms. Owing to the large volume of collected data, time series models can be effectively employed to accurately predict future mushroom growth and to estimate the optimal harvesting time.

6. Conclusion

In this work, we have presented a dedicated mushroom detection dataset tailored to train and evaluate object detection and instance segmentation methods for various purposes that require localization of mushrooms in images such as automated mushroom harvesting, mushroom quality assessment, and disease detection. Through comprehensive data collection and meticulous labeling processes, the dataset encompasses a wide spectrum of scenarios and challenges inherent in real-world mushroom growth environments including clusters of mushrooms with a variety of densities, images with different lighting conditions, occasional occlusion of mushroom instances by soil particles, and mushrooms growing at the edge of cultivation beds. The diverse nature of the dataset, including instances of both white button and baby bella mushrooms, allows for the robust training and benchmarking of detection and instance segmentation algorithms.

Beyond 2D annotations, our dataset also provides 3D bounding box and rotation information for the main subset of mushrooms, enabling pose estimation tasks. Additionally, we introduced a separate subset dedicated to yield estimation, capturing the same scenes over time to facilitate growth forecasting and harvesting schedule optimization.

Our evaluation of 9 different object detection and instance segmentation models highlights their respective strengths and weaknesses, shedding light on their suitability for mushroom detection and instance segmentation tasks. Furthermore, our exploration of color spaces provides insights into potential improvements in classification accuracy. By sharing this dataset and evaluation results, we hope to catalyze research and innovation in the domain of smart edible mushroom farming, fostering collaborations and accelerating progress in detection technologies. We anticipate that the integration of 3D pose estimation and yield forecasting components will further expand the dataset’s utility, supporting a broader range of applications in automated mushroom cultivation. The availability of this benchmark dataset, our code, and trained models contributes to the development of advanced mushroom detection techniques, pose estimation methods, and yield prediction models, paving the way for increased efficiency and sustainability in automated mushroom harvesting systems.