Application of Convolutional Neural Networks in Weed Detection and Identification: A Systematic Review.

1. Introduction

According to the United Nations, the world population is estimated to reach 9.7 billion inhabitants by 2050 [1]. Against this backdrop, facing the challenge of feeding this growing population with quality and sustainable products becomes an imperative task. Increasing crop productivity emerges as a measure to address this need. Thus, a strategy that contributes to improving productivity is properly managing weeds, given their direct impact on crop yields. Integrated weed management is essential to preserve agricultural productivity [2]. Plants considered weeds are fast-growing and actively compete for vital resources such as space, water, nutrients, and sunlight. This competition not only affects resource availability but also has a negative impact on crop yield and quality [3]. According to [4], damage due to weeds can represent up to 42% of agricultural production.

Currently, diverse weeding techniques are used, such as pre-and post-emergence herbicides, whose application not only generates environmental impacts but also affects the health of the workers who apply them [5]. Mechanical weeding applying mechanized or manually techniques, whose effectiveness in eliminating weeds is not always the desired one, depending on their stage of development [5]. Other weeding alternatives are still under development, or their feasibility has not been fully demonstrated. One is physical weeding using plastic covers [5] and microbiological weeding involving microorganisms [6]. Traditional weeding methods present environmental challenges or economic disadvantages, creating the need to explore innovative solutions based on new technologies to increase treatment efficiency. For instance, precision weeding uses image sensors and computational algorithms to apply herbicides only when weeds are identified [7].

Computational algorithms based on Deep Learning (DL) techniques [8], are used to improve the accuracy of weed detection. DL is an advanced branch of machine learning that uses multi-layered artificial neural networks to model and process more complex data, such as digital images. Within the processes of pattern recognition through digital images, the application of neural networks has evolved, creating new specific architectures for computer vision tasks such as Convolutional Neural Networks (CNN). CNNs efficiently detect spatial patterns in digital images by using convolution layers that apply filters to local regions of the input image [9]. These convolution layers allow the network to automatically learn hierarchical and complex features, such as edges, textures, and shapes, instead of relying on predefined features. The basic structure of a CNN model consists of three layers (Figure 1), a convolutional layer, a clustering layer, and a connection layer [10].

• The convolutional layer extracts feature from the image using mathematical filters; the features can be edges, corners, or alignment patterns, which give the output a feature map that serves as input to the next layer.
• The grouping layer reduces the resolution by reducing the dimension of the feature map in order to minimize the computational cost.
• The connection layer the image obtained from the previous layer is sent to the fully connected neural network layer, which contains the activation function used to recognize the final image.

The use of DL-based algorithms in agriculture has increased in recent years, and several works have been found that compile the applications of this algorithm [11]. One of the most interesting applications is weed detection using CNNs, which allow for rapid weed detection, localization, and recognition [12, 8]. These methods, supported by large-scale datasets, have demonstrated high robustness against biological variability and diverse imaging conditions [13], reaching the most accurate classification or detection [14] [15], which allows automating weeding or weeding processes accurately and efficiently [7]. However, weed detection faces several problems in practice, such as similarities in colors, textures, shapes, occlusion effects, and variations in illumination environments. To overcome these limitations both traditional and CNN-based machine vision offer effective solutions [16]. Advances in the development of DL algorithms that train CNN models faced some limitation specially when a new CNN model in particular needs to be trained, it required a large amount of data and additional computing equipment with high processing capabilities. Therefore, to reduce these limitations the transfer learning (TL) technique is adopted, this allows using pre-trained models, and they are applied to the new model with some modifications to solve the specific problem [17, 13]. TL aims to transfer the knowledge from the source domain to the target application by improving its learning performance [18]. One of the examples of using TL with CNN is AlexNet, which was trained on the ImageNet dataset [19]. In agriculture, the TL approach has been implemented in weed detection and classification helping to minimize the need for large-scale image data collection, and reduce the computational costs related to in the training hours of in a new CNN model [20, 21, 22]. In addition to the use of e TL in agriculture, some generative adversarial network (GAN) techniques have been applied to generate artificial images to augment the training set with TL [23]. The evolution of CNNs has been marked by significant advances in terms of architectures, training techniques, efficiency, and applications, starting from the need to implement fast solutions in image analysis; some of the most used CNN architectures in weed detection are mentioned below:

• AlexNet: Developed by [19] in 2012, won the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) competition, demonstrating the impact of CNNs in computer vision.
• GoogLeNet (Inception): Developed in 2014 by the Google Research team, introduced the concept of Inception modules with multiple filter sizes in parallel [24].
• VGG: Developed in 2014 by the Visual Graphics Group (VGG) at Oxford University, it is known for its simplicity and depth [25]. Its structure of pure convolutional layers and deep subsampling influenced the design of later architectures, which had different improved versions.
• ResNet: Developed by [26] in 2015, this architecture was highlighted using residual blocks, which allow the training of very deep networks by facilitating the information flow and mitigating the gradient vanishing problem.
• Fast R-CNN: In 2015 it was presented a significant improvement over its predecessor, the R-CNN (Region-Based Convolutional Neural Network) model. It was developed by the Microsoft Research group to address the speed and computational efficiency limitations associated with R-CNN, providing a faster and more practical solution for object detection in images [27].
• DenseNet: Proposed In 2017 by [28], it is notable for its densely connected structure, where each layer is directly connected to all subsequent layers. This dense connectivity can potentially improve information flow and mitigate the problem of gradient fading. It has influenced the design of subsequent architectures and continues to be a popular choice in research and practical implementation in computer vision tasks.
• MobileNet: Proposed in 2017 by [29], it is specially designed for implementations on mobile devices and uses lightweight and efficient operations to balance performance and resource consumption.
• YOLO (You Only Look Once): Developed in 2016 by [30], it is a fast and efficient object detection architecture as it approaches this task as a regression problem; instead of a separate classification for each region, this feature allows several versions from YOLOv1 to YOLOv8 in 2023. Starting with the fifth version released in 2020, known as YOLOv5, was built on PyTorch [31], maintaining the original YOLO approach of dividing the image into a grid and predicting bounding boxes with class probabilities for each cell. The overall architecture includes convolutional layers, attention layers, and other modern techniques; it is important to mention that this version was developed by the Ultralytics team, not by the original authors. In 2022, the YOLOv6 and YOLOv7 versions were developed, presenting improvements in their architecture and training scheme, and improving object detection accuracy without increasing the cost of inference, a concept known as "trainable feature bags" [32]. Finally, in 2023, YOLOv8 is presented; its improvements include new features, better performance, flexibility, and efficiency. Additionally, it includes improvements for detection, segmentation, pose estimation, tracking, and classification [33].

This study aims to review the latest research on the detection or identification of weed using CNN techniques, with the aim of providing a starting point for researchers interested in implementing this technique. Therefore, this review will be focus on analysing the existing type of architecture and the technology in weed detection.

2. Methods

In this study, a systematic review was carried out to identify and analyze scientific literature published on weed detection using CNN. The guidelines of the PRISMA statement [34] were followed in this review.

2.1. Research Question and Review Objectives

• Research question: What are the Deep Learning techniques based on Convolutional Neural Networks used for weed detection in agriculture?
• Main Objective: To identify the Deep Learning techniques that are employed in Convolutional Neural Networks for weed detection.
• Specific Objectives:

• To investigate the architectures of Convolutional Neural Networks employed in weed detection.
• To identify the technology used for weed identification in different forms of production.

2.2. Sources of Information

The scientific resources platform of the Spanish Foundation for Science and Technology (FECYT), which has access to the following databases: Web of Science and Scopus, was used for the systematic search.

2.3. Search for Keywords

A primarily search was carried out to establish the relevant words for the systematic search; for this, the Scopus database was used with the words "weeds detection deep learning," establishing the "search within all fields," (ALL (weeds AND detection AND deep AND learning), in this search 6096 results were obtained. With the filtering options in "Filter by keyword," we found the five most used keywords with their number of matches are: "Deep Learning" (1800), "Machine Learning" (988), "Remote Sensing" (704), "Crops" (647), and "Convolutional Neural Networks" (638). The initial search covered the topics of importance for this systematic review and took "Weed detection," "Deep Learning," and "Convolutional Neural Networks" as keywords for the search.

2.4. Inclusion and Exclusion Criteria

The following inclusion and exclusion criteria were used and implemented through the filters of each database:

• The search field is selected where the search is directed through titles, abstracts, and keywords, among others; this is specific to each database:

• In Scopus, "search within Article title, Abstract, Keywords" was established.
• In Web of Science, the search was established in "Topic"; this includes title, abstract, author keywords and keywords plus.

2.

The date range of the search is the last five years, from 2019 to 2023.

3.

Document type: "Document type: Article".

4.

Excluded are reviews, book chapters, narrative articles, conference or congress articles, unofficial notes or communications, and studies from other areas such as social, human, biological, chemical, legislative, social and economic impact.

5.

Language: "English Language".

2.5. Search Equations in Bibliographic Databases

The search equation is established by restrictively connecting all the results containing the keywords "weeds detection" AND "Deep learning" AND "Convolutional Neural Networks." With this, the search equation is established according to each platform:

• Scopus: TITLE-ABS-KEY ("weed detection" AND "deep learning" AND "Convolutional Neural Networks ")
• WOS: TS = ("weed detection " AND "deep learning" AND "Convolutional Neural Networks ")

2.6. Initial Search Results

• Initial records: Scopus 104 and WOS 40. Initial results obtained 144.
• Records eliminated by exclusion criteria: 65 results eliminated and 79 results remained, 61 from Scopus and 18 from WOS.

2.7. Duplicates and Screening

The free access tool Zotero 6.0.30 was used to eliminate duplicate results, and 14 duplicate results were eliminated, obtaining a total of 65 results.

2.8. Additional Records

16 articles obtained from reading book chapters and reviews were added to the results, obtaining 81 results.

2.9. Records Excluded

A total of 20 records were excluded because they do not meet the objective of this review or cannot be accessed through the scientific resources platform of the Spanish Foundation for Science and Technology (FECYT).

Finally, 61 articles that meet the established criteria were gathered and analyzed Figure 2 illustrates the process in a flow chart using the PRISMA methodology.

3. Results

3.1. Literature Analysis

A detailed analysis of the 61 bibliographic articles was conducted. The analysis indicated that in the last two years, there has been a massive growth in the number of articles published on weed detection, Figure 3. The increasing amount of research in this area is mainly due to the development of new and more efficient CNN architectures, the increase in the processing capacity of computers, as well as the reduction in the price of cameras and graphics processing units (GPU).

3.2. Technology for image acquisition

In the review of the selected articles, it was found that the authors used different types of technology for the acquisition of the images used in the training and validation of the CNNs, such as digital cameras, professional Reflex type, industrial high-speed and low-cost cameras such as those using Raspberry cards. UAVs with various types of cameras, both RGB and multispectral. Smartphones with high-resolution cameras were also used. In addition, it was found that some researchers did not acquire images and used free databases or previous works. Figure 4 shows the number of publications according to the technology used, where 49.2% used digital cameras, 29.5% used UAV as a mean of acquisition, 11.5% used smartphones, and 9.8% used already built datasets.

3.3. CNN architecture used

Table 1 summarizes relevant information extracted from the selected reviewed studies on CNN architectures such as author, publication year, CNN architecture used, technology and the species.

Figure 5 illustrates the frequency of use of the different CNNs for segmentation, detection, and classification of weeds. The YOLO family of algorithms with its multiple versions is the most applied technology, followed by the VGG, ResNet, and Faster R-CNN architectures and the previous one with various versions. Alexnet and MobileNet are also commonly used.

4. Discussion

The selected papers analysed in this review show the use of a range of technologies to capture images from multispectral cameras on UAVs, to low-cost system development, to smartphones available to everyone. Several articles integrated different technologies, as, for example, in [60], who used a Nikon 7000 camera to build the image dataset for training using YOLOv5. Additionally, they built a spraying system using Raspberry cameras to distinguish dicotyledonous from monocotyledonous weeds. In [68] study, they used an AUV, a multi-copter drone (Hylio Inc., Houston, TX, United States) equipped with a Fujifilm GFX100 (100 MP) camera, used YOLOv4 and Faster R-CNN architectures. In [37], they used two cameras Sony Cyber-Shot and Canon EOS Rebel T6 for image acquisition, used AlexNet, GoogLeNet, and VGGNet architectures to detect Perennial ryegrass, dandelion (Taraxacum officinale), ground ivy (Glechoma hederacea L.) and spotted spurge (Euphorbia maculata L.).

As the digital camera is the most used technology in this study, it can be concluded that it is preferred for its higher image quality and speed compared to that of a smartphone or an AUV camera, as for example[59] evaluated three digital cameras, a Canon T6 DSLR 7 camera, an LG G6 smartphone, and a Logitech c920 camera, the detection results were higher for the Canon T6 camera. In contrast, the Logitech c920 camera was not suitable for weed detection, demonstrating that SLR-type cameras are preferred for the development of mobile platforms, field carts, or robots due to their image quality and different adjustment options. In the work of [65], a semi-professional Nikon P250 camera was used to develop a prototype autonomous sprayer; similarly, in [36], a field robot was designed to detect weeds in high-density crops using a 20-Mpixel JAI camera, which is a high-speed industrial camera.

In this review the use of AUV cameras is largely based on the integrated cameras manufactured by DJI, in the Phanton 3 and 4, Matrice 600, Spark and Mavic versions. Therefore, they are limited to the performance and sensor configuration used by the brand. [72] used UAV Phantom 3 Professional imaging at three flight altitudes (10 m, 15 m, and 30 m), the VGG, ResNet, and DenseNet architectures, along with smaller ShuffleNet, MobileNet, EfficientNet, and MNASNet models were used to detect Rumex obtusifolius. In [51], they used a Mavic pro AUV integrated with a Parrot Sequoia camera, modified the CNN VGG-16 to detect weeds in sugar beet (Beta vulgaris subsp.), and used a Mavic pro AUV integrated with a Parrot Sequoia camera, modified the CNN VGG-16 to detect weeds in Sugar beet (Beta vulgaris subsp.).

The use of smartphones as imaging technology has increased in recent years, as can be seen in the work of [62], who used a Huawei Y7 Prime smartphone to take images in pea (Pisum sativum) to work with the Faster RCNN ResNet 50 model. Similarly, [70], a Xiaomi Mi 11 smartphone on bell pepper (Capsicum annum L.) to apply Alexnet, GoogLeNet, InceptionV3, and Xception. Researchers in [67] took advantage of existing databases and image repositories and used the DeepWeeds dataset to train SSD- MobileNet, SSD-InceptionV2, Faster R-CNN, CenterNet, EfficientDet, RetinaNet, and YOLOv4 models. [77] used the agri_data dataset available on Kaggle, on Falsethistle grass and walnut (Carya illinoinensis), to train VGGNet, VGG16, VGG19, and SVM models. In summary, Table 2 presents an analysis of the articles and groups them according to the technology used for the acquisition of the images and the most frequently used CNN architectures.

5. Conclusions

In this systematic review, following the guidelines of the PRISMA statement, 61 scientific articles on the detection of weeds using CNN were analysed, using the scientific resources platform of the Spanish Foundation for Science and Technology (FECYT), which has access to the following databases: Web of Science and Scopus. The review covers the last five years (2019-2023).

The CNN architectures were identified as the most applied for weed segmentation, detection, and classification followed by the YOLO family of algorithms with its different versions. The tVGG architecture and its versions are the third most used, followed by the ResNet, Faster R-CNNN. The Alexnet and MobileNet were the least used architectures.

The technology used to acquire the images used to train and validate the CNNs was identified. Fixed digital cameras, reflex type, or low-cost cameras which allow a wide configuration and higher image quality, are the main technology applied. Cameras integrated in UAVs are also used, despite their speed limitation and lower configurations compared to an SLR camera, which can be RGB or multispectral. Smartphones with high resolutions camera have also been used, with the drawbacks of low processing speed.

In this review, it was noted that some authors used free databases or databases from previous studies to avoid image acquisition and the difficulties associated with it.

As future work, it is expected that the review will be extended to focus on the search for the most appropriate CNN architectures for weed detection and classification. Finally, it is hoped that this review article will help researchers to create new technological developments that will improve weed detection.