Enhancing Greenhouse Pollination with CNN-Based Micro-UAV for Real-Time Flower Detection

1. Introduction

Food security, a critical global concern, is intricately linked to the United Nations’ Sustainable Development Goals (SDGs), particularly SDG 2 (Zero Hunger). As the world population continues to grow and is projected to reach 9.7 billion by 2050, ensuring food security and maintaining food quality have become increasingly challenging [1,2]. Future forecasts suggest a potential strain on food systems, with climate change and resource scarcity further complicating these issues. In developing countries, agricultural industries play a pivotal role in addressing food security challenges while simultaneously contributing to national income [3,4]. These industries not only enhance local food production but also create employment opportunities and stimulate economic growth in the region. However, achieving sustainable food security requires a multifaceted approach that integrates innovative farming techniques, such as autonomous robots or agricultural drones, to enhance agricultural production, environmental conservation, and economic development [5,6].

Food security issues are closely intertwined with sustainable and smart agricultural practices. Sustainable agriculture focuses on producing food in an environment friendly, economically viable, and socially responsible manner. Smart agriculture, a subset of sustainable agriculture, leverages technological and data-driven approaches to optimize the farming process [7]. The elements of smart agriculture include: (a) precision farming techniques, (b) Internet of Things (IoT) sensors and devices, (c) data analytics and artificial intelligence (AI), (d) automated irrigation systems, (e) drone technology for crop monitoring, (f) climate-smart practices, and (g) soil and crop management tools [8,9]. Given the broad challenges of food security and smart agriculture, this study considers the transformative potential of drone technology as a breakthrough in some of the challenges. This study proposes a solution to agricultural problems using micro-unmanned aerial vehicle (UAV) technology with micro dimension which is less than 100 mm x 100 mm. This is to take advantage off navigating tight spaces, such as vertical farm. Previous drone-based pollination systems utilized larger dimensions [10,11]. The proposed method and solution lay the foundation for research on utilizing micro-UAVs as smart agricultural tools.

The novelty of this paper lies in the deployment of convolutional neural network (CNN) on a micro-sized, self-operating drone designed to function as a pollinator. The main tasks of drone is to find flowers and fly towards them from a specific starting point. Before the drone was used, it was trained to recognize flowers by using a CNN. During operation, the pollinator drone takes off and examines its surroundings. If it spots a flower nearby, it flies close to it. The goal of this study was to improve the navigation and flower detection capabilities of micro drones using CNN technology, potentially aiding pollination tasks in the future.

In Section II, we provide an overview of drone use in agriculture, the potential of drone pollinators, AI classifiers, and flower recognition systems, and discuss the problems. In Section III, we present the details of the experimental methods, which consist of a binary classification technique using a ResNet-18 CNN to detect flowers. We also elaborate on the integration of AI during real-time drone flights. In Section IV, we discuss the experimental results and effectiveness of drone AI detection versus distance. Finally, we conclude the paper and provide directions for future research in Section V.

2. Related Works

2.1. Agriculture Micro-UAV

Micro-UAVs, also called drones, offer numerous benefits for smart agriculture. Off-the-shelf agricultural drone solution packages are equipped with onboard monitoring sensors that can provide real-time data [12]. A typical onboard sensor package includes a camera that can be utilized for additional functionalities. Current solutions employing onboard cameras are used to monitor crop conditions, detect pest infestations, and perform other tasks [13]. Another promising area of research is the use of drones as agricultural pollinators. Bees are the primary pollinators of many crops. However, in greenhouses or enclosed agricultural settings, bee penetration is limited, posing challenges to pollination [14]. Given this limitation, pollinator drones are a potential solution. With the help of onboard cameras, pollinator drones can identify the location of flower buds for subsequent pollination [15]. Drones can simultaneously collect data on crop health and pollination efficacy while executing pollination tasks. Automated drone pollination has the potential to reduce labor costs associated with manual pollination methods. However, the use of drones as pollinators is still in the experimental phase and faces challenges such as battery life limitations, potential impacts on local ecosystems, and the need for further technological refinement.

2.2. Potential of Autonomous Pollinator Drone

Although bees naturally serve as pollinators, sustainable smart agriculture often employs closed environments, such as greenhouses or indoor spaces, such as vertical farms, which restrict bees’ access to natural pollination [16,17,18]. To address pollination challenges in controlled settings, the use of drones as pollinators has emerged as an innovative solution [15,19,20]. Despite their long and excellent track record as pollinators, traditional pollinators such as bees encounter difficulties in reaching enclosed agricultural areas [21,22]. This has led researchers to investigate alternative pollination methods for such crops. In recent years, various solutions have been proposed, including mobile robots with robotic hands and soft actuators with high potential as pollination mechanisms. However, a significant limitation is the initial investment cost, as mobile robots and soft actuators must be developed to meet specific environmental requirements [23,24,25,26]. In contrast, off-the-shelf micro-UAVs are highly programmable, making them well-suited for autonomous pollination tasks [27].

Recent reports have indicated that pollinator drones equipped with specialized attachments can replicate the pollination process. These drones, categorized under the vertical takeoff and landing (VTOL) configuration, offer precise control and ability to operate in confined spaces such as greenhouses and indoor farming environments [19]. This technology has the potential to sustain crop productivity. Empirical studies have confirmed the efficacy of drone pollination in various crops including strawberries, peppers, tomatoes, and kiwifruits [15,19,28,29]. The benefits of drone pollinators include their consistent operation regardless of weather conditions, programmability for optimal pollination timing, and potential to reduce labor costs [30,31]. Nonetheless, challenges persist, particularly the need for further refinement of drone sensor payloads, particularly navigation and imaging sensors, to enable autonomous navigation to flowers and match the efficiency of natural pollinators. Additionally, concerns regarding long-term ecological impact of deploying multiple drones simultaneously in greenhouse agricultural settings remain. Although short-term ecological impacts may enhance agricultural efficiency, long-term ecological consequences require further investigation [32,33]. One notable negative impact is the buzzing noise produced by drones, which may cause biological disturbances to local bird and insect populations [34]. As research in this domain advances, drone pollination technology continues to evolve, potentially serving as a viable complement to natural pollinators in ensuring food security and promoting sustainable agricultural practices.

Determining the appropriate size and suitability of pollinator drones for robotic pollination in greenhouse environments is crucial for optimizing their effectiveness and efficiency [15]. The drone must be micro-sized to navigate confined spaces while carrying essential pollination equipment. Micro drones can easily maneuver between plant rows and the surrounding structures [35]. However, accommodating a sufficient pollen payload along with necessary sensors or cameras presents significant challenges. The ideal size balances the agility and functionality, which is the primary reason why this study selected a micro-UAV configuration as the main drone platform.

2.3. Simultaneous Navigation and Flower Recognition

Flower detection and navigation are crucial components for the development of autonomous pollination systems using drones. Recent studies have focused on improving the accuracy and efficiency of these processes to enhance drone performance in agricultural settings [36]. AI techniques for computer vision, particularly deep-learning algorithms, have shown promising results in flower detection. CNNs have been successfully applied to identify and locate flowers in complex greenhouse environments, achieving high accuracy rates. These models can distinguish between different flower species and their growth stages, thereby enabling targeted pollination [37,38].

The enclosed nature of greenhouses limits the use of traditional global positioning systems (GPS), necessitating alternative navigation methods [39,40]. Researchers have explored various approaches to overcoming the limitations of GPS in enclosed spaces. Visual Simultaneous Localization and Mapping (VSLAM) techniques have been implemented to allow drones to create and update maps of their environments in real time [41]. Among these techniques, cameras and computer vision algorithms can help drones maintain their position and navigate through greenhouses [39,42]. Sensor fusion, which combines data from cameras, infrared sensors, and inertial measurement units, has improved the precision of drone movement around plants [43]. Some studies have investigated the use of artificial landmarks or QR codes within greenhouses to aid drone localization [44,45].

The importance of navigation in greenhouse settings cannot be overlooked. Accurate navigation ensures that drones systematically cover all plants that require pollination without missing areas or over-pollinating others. They also prevent collisions with greenhouse structures or plants that can damage both the drones and crops [46]. Infrared or ultrasonic sensors can be employed for obstacle avoidance and precise maneuvering around plants [47]. Efficient navigation contributes to energy conservation, allowing drones to operate for longer periods and cover larger areas. Additionally, precise navigation enables the collection of valuable data on plant health and growth patterns, contributing to overall greenhouse management and crop optimization [48]. The integration of flower detection and navigation systems has led to the development of more efficient pollination strategies, optimized flight paths, and reduced energy consumption.

3. Experimental Setup

This section provides an overview of the proposed drone pollination system, which comprises of two main parts. The first part involved the use of the ResNet-18 machine learning CNN algorithm for the binary classification of flowers, as illustrated in Figure 1. This algorithm, commonly used for image analysis, helps us categorize objects into two classes: `flower’ or `not flower’. We developed an AI-based flower identification system that allows drones to hover towards detected flowers without requiring human intervention. Building on this flower classification using the ResNet-18 algorithm, the second part of our study focused on autonomous navigation for real-time field test flower recognition. A drone platform with an onboard camera is the most crucial device. An indoor laboratory was chosen for the real-time field test drone flight and flower recognition evaluation.

3.1. AI Binary Classification for Flower Recognition

Before we conduct the real-time field test, there is an important step called AI binary classification for flower recognition using the ResNet-18 machine-learning CNN algorithm. As shown in Figure 1, this classification is a key part of the process for handling the raw visual data of the flowers. The process began with the collection of flower images, as shown in Figure 2. Subsequently, we pre-trained the model and performed classification. We used a pre-trained model, ResNet-18, which can be fine-tuned specifically for our dataset. This dataset was prepared for the computational model by splitting it into separate sets for training, validation, and testing. These categorized data subsets were fed into the ResNet-18 framework, which learns to recognize specific patterns and features of the flowers. Finally, the ResNet-18 framework performs a binary classification task to determine whether the input image shows a `flower’ or `not flower’. We compared the classification performance of a drone’s onboard camera and a standard webcam, focusing on the time it takes to recognize a flower, measured in seconds. The number of detections was set to 10, 20, 30, 40, 50, and 60.

3.2. Micro-UAV Specification

The system for the micro-UAV pollinator navigation platform involves careful selection of a lightweight, compact drone capable of stable flight within the confined and controlled environment of a greenhouse. The DJI Tello Robomaster TT was selected with key specifications including payload capacity to accommodate integrated sensors and cameras, extended battery life for prolonged operation, and precise maneuverability to navigate narrow aisles and avoid obstacles. The hardware setup prioritizes modularity and reliability, ensuring that UAV can be easily maintained and adapted to different greenhouse layouts and crop types. Table 1 summarizes the specifications of DJI Tello Robomaster TT used in this study. The onboard camera was central to the functionality of the drone for flower detection. A high-resolution 5 MP camera enabled the accurate identification and monitoring of flower positions. Sunflowers were used in this project because they have unique visual characteristics, such as bright yellow petals and distinct shapes, which make them distinguishable from the surrounding environment. An indoor laboratory was selected as the experimental field to mimic an indoor greenhouse scenario, as shown in Figure 3.

3.3. Integration of AI Detection Capability on the Micro-UAV

In the next part of our research, we incorporated a drone into the real-time field experimental setup, as depicted in Figure 3. In our laboratory tests, we arranged four identical flowers in a `cross’ configuration, maintaining specific distances between the flowers and drone, as illustrated in Figure 4. The drone was initially positioned at its center. We systematically adjusted the distance between the drone and the flowers for each subexperiment, conducting tests at intervals of 15.5 cm, 30.5 cm, 60.5 cm, 91.5 cm, and 116.5 cm, as shown in Figure 4. At each distance, we documented the efficacy of drone in flower detection. Upon takeoff readiness, the drone established a connection with the workstation via Wi-Fi. Following a vertical ascent to approximately 1.5 m above ground level, the drone commenced its search for flowers.

The flight path was pre-programmed as shown in Figure 5. Initially, the drone ascended vertically after it was successfully connected to the local Wi-Fi. Once it reached approximately 1.5 m above ground level, the drone was incrementally rotated 10° until a flower was detected in its video frame. The process of flower recognition during flight is illustrated in Figure 6. The drone advances forward when a flower is detected. In the event that no flower is identified in the video stream, the drone continues to rotate in 10° increments until the flower is located. If a flower is detected during rotation, the drone moves towards it. The drone lands when its battery level falls below 20%.

4. Results and Discussion

Two databases, designated as `flower’ and `non-flower’, were collected for this study. The training process was conducted over 64 epochs, with 14 iterations. As illustrated in Figure 7, the model achieved an accuracy of 1, indicating that it correctly classified all the instances in the training data without errors. This suggests that the model effectively learns the training data and accurately predicts the correct labels for each input.

The preliminary experiment aimed to assess the classification algorithm’s ability to recognize a flower, with the performance measured in seconds. A comparison was made between the drone’s onboard camera and the standard webcam. Table 2 presents the experimental data for detection time. Detection commenced at intervals of 10, 20, 30, 40, 50, and 60. Both the webcam and drone camera operated in the real-time mode. As the number of detections increased, the time required for both the webcam and the drone camera also increased. Notably, the webcam required more time for detection than the camera of the drone. This discrepancy can be attributed to several factors, primarily the hardware specifications. The DJI Tello Robomaster TT drone is equipped with specialized image-processing hardware and dedicated processors, including GPUs, optimized for real-time image-analysis tasks. These components facilitate parallel computations and expedite the detection process. In contrast, webcams typically rely on a computer’s CPU, which may have fewer computational resources and slower processing speeds compared to specialized hardware.

In the subsequent phase of this experiment, the drone was programmed to navigate in a `cross’-shaped pattern, as illustrated in Figure 8. The drone’s rotational movement was detected, and the graph indicated that the drone’s camera identified four flowers during its rotation to locate them. The first detection occurred between 32.2 seconds and 38.6 seconds, at angles ranging from 49° to 113.6°. The second detection took place from 46.5 seconds to 49.9 seconds, at an angle of -147.2° to -95°. The third detection was recorded between 103.7 seconds and 108.8 seconds, at an angle of 69.9° to 133.1°. The final detection occurred from 143.4 seconds to 149.1 seconds, at an angle of 82.5° to 145.2°. At certain points, the angle on the line graph dips in the negative direction, indicating a counterclockwise rotation and a descent or lowering of the drone.

According to Table 3, if the distance between the drone and the flower was too small (i.e., 15.5 cm), the drone failed to detect it. Conversely, if the distance is too large (i.e., 116.5 cm), the drone also struggles to locate the flower. This is because when the drone is too close, its sensors and cameras may not have a sufficiently wide field of view to capture the entire object or provide accurate measurements, making it difficult to effectively detect and track the flower. However, if the drone is too far, the flower’s size in the camera’s field of view diminishes, causing the drone’s computer vision algorithms to have difficulty accurately identifying the flower, especially if there are other objects or cluttered backgrounds. Additionally, lighting conditions can affect object detection; if the flower is poorly lit or the lighting is too harsh, the drone’s sensors and cameras may struggle to distinguish the flower from its surroundings.

5. Conclusion

This study effectively demonstrates the implementation of a micro-UAV equipped with a CNN for autonomous flower detection and navigation within a controlled greenhouse environment. The drone successfully identified sunflower targets at optimal distances with detection accuracy only when the drone is positioned excessively close to or distant from the flowers, thereby emphasizing the necessity of maintaining an effective operational range. The real-time detection capabilities of the drone’s specialized camera surpassed those of conventional webcams, thereby underscoring the benefits of dedicated onboard hardware. The `cross’-shaped flight patterns confirmed the drone’s ability to navigate and accurately identify multiple flowers. These findings highlight the potential of micro-UAVs as efficient pollination agents in enclosed agricultural settings and offer a promising solution to the challenges encountered by natural pollinators in greenhouses. Future research should focus on enhancing sensor capabilities, extending operational ranges, and assessing ecological impacts to advance the practical deployment of autonomous drone pollinators.