Multitemporal Segmentation Techniques for Canapa Sativa L. Detection Through Unmanned Aerial Systems

1. Introduction

In the last decades many farmers affected by severe economic distress and extreme weather events started considering and cultivating alternative crops. Cannabis sativa L., one of the oldest crops known by humankind, thanks to the last decades' diffusion of low tetrahydrocannabinol (THC) concentration varieties, is experiencing a worldwide distribution [1,2]. Despite the potential use in numerous industries (energy, feed, health, etc.), knowledge and supply chains about Canapa Sativa L. are still limited [3,4,5]. High public interest in this crop is increasing with the need for genetic improvement [6,7] and performance optimization.

Remote sensing technologies introduction is a valuable solution to optimize cultivation techniques and input distribution and to perform more reliable and effective recognition procedures thanks to usable, near real-time information based on cultivars and cultivation environment knowledge [8]. Despite the numerous image thresholding techniques proposed in the literature, their performance analyses are relatively limited. Plant emergence and stand establishment are key indicators routinely assessed in Canapa improvement research [9,10]. Automatic Unmanned Aerial System (UAS) image processing for plant numbering in the field represents a valuable solution to manual plant counting performed at regular intervals. Image processing, widely used to solve various agriculture problems such as classification, object detection, object counting, and many more, can help extract crucial information from the image automatically. Canopy height models (CHM) derived from structure from motion or light detection and ranging (LiDAR) technologies represent a technique for canopy height, size, and biomass measurement also used to separate crop pixels from soil [11,12]. Likewise, Otsu thresholding, widely used in many computer vision applications and employable with vegetation indexes, is a single-intensity threshold method that separates pixels into the foreground and background classes [13]. Differently from Machine Learning (ML) techniques, computer vision does not require large labeled datasets, training, test, and validation phases to perform plant segmentation, resulting in an easier application and fast results obtainment.

Accurate plant information extraction from sensors and images is currently a bottleneck that depends on correct image segmentation [14]. Multispectral and hyperspectral images provide crucial information that can be used together with advanced ML tools to develop a direct method to discriminate cannabis from similar plants [15]. ML requires high quality data in a specific format and free from extraneous features to build models able to perform fast and automatic classification, recommendations, or prediction operations. Pre-processing techniques, essential to make the data suitable for model training, testing, and validating, are complex and time-consuming [16]. Between the multiple ML algorithms randomly selected or chosen after performance comparison, it is hard to choose the appropriate and suitable one to build and customize the learning model, delaying the deployment process. ML model building needs long training and multiple testing, generally time-consuming tasks. Moreover, the deployment of models is particularly challenging because of the complexity of the real-world scenario and the required deployment skills to employ the models using third-party software and hardware [17,18,19,20,21].

Relevant works report systems for plant detection and counting, showing different techniques and approaches. Sharma et al. used multi-spectral UAS images acquired at 15 m height combined with GIS software to estimate the number of plants of two different potato varieties [22]. The results showed a high Pearson correlation coefficient (r = 0.82) between image-based and manual counting. Another example is the implementation of computer vision using the Excess Green Index (EXG) and Otsu's method and transfer learning using convolutional neural networks to identify and count plants in a spinach field [23]. A similar experiment by Shirzadifar et al. [24] on Maize crop reported a detection algorithm using the EXG index and the k-means clustering technique, achieving an average accuracy of 46% and 91% in the field, respectively. Fully automatic plant counting in sugar beet, maize, and strawberry fields was performed by Barreto et al. [25] using UAS images and deep learning analysis. The number of crops per plot was automatically predicted in five locations at different growth stages, showing errors of less than 4.6 % for sugar beet and that the arrangement of plants in the field is a crucial factor influencing detection performance. The algorithm proposed by Fan et al. [26], based on deep neural networks for tobacco plant detection from UAS images, consisted of three stages. The first stage consisted of obtaining candidate regions of tobacco plants, containing either a tobacco plant or a non-tobacco plant, from UAS images. Second, the development of a deep convolutional neural network, trained to classify the candidate regions as either tobacco or non-tobacco plant regions. In the third step, the non-tobacco plant regions were removed.

Most of the proposed and analyzed works rely on machine learning systems for detecting and counting plants. Is it possible to obtain comparable performance using traditional segmentation techniques, or machine learning is the only approach suitable for a reliable detection? The work aimed to compare and evaluate the potential of two thresholding segmentation techniques for Cannabis sativa L. plants detection in two experimental fields in Sardinia (Italy) in a multi-temporal scale and evaluate the potential of such techniques to provide reliable information for farm management.

2. Materials and Methods

2.1. Study Site

Experiments took place in two study sites: Olbia (North-East Sardinia, Italy, Lat. 40° 93’ 37”; Long. 9° 44’ 88”, WGS84 EPSG 4326, at 37 m above sea level) and Guspini (Sud-West Sardinia, Italy, Lat. 39° 66’ 16”; Long. 8° 66’ 12”, WGS84 EPSG 4326, at 17 m above sea level). In Guspini, 2500 Canapa sativa L. (cv. Kompolti) plants were planted (1.8 m x 2.0 m) on a 10000 m2 surface (Figure 1a), where another scientific survey was already running to monitor the phenotypic plants’ response to different agronomic management combined with inoculation of rhizobia and the use of mulching cloths to limit the growth and development of weed on the rows. Another 1000 plants (2.00 m x 1.50 m) of the same cultivar were planted on a 3200 m² surface in Olbia (Figure 1b).

The experimental field in Guspini consisted of four plots (two of them were replications of the same treatment). Treatment 1 comprised mulching cloths and rhizobia inoculation, treatment 2 consisted of no mulching cloths and rhizobia inoculation, and treatment 3 comprised mulching cloths and no rhizobia inoculation.

Due to the stunted growth of plants in the Olbia study site in the last phenological phases (Figure 2b), the research team decided to execute a single flight over the second study site (Guspini) to test the proposed techniques on “normal” grown plants and compare the results with smaller plants of Olbia.

2.2. UAS Platform and Implemented Sensors

The UAS DJI Phantom 4 Pro (Figure 2a) equipped with RGB CMOS 1" sensor of 21 megapixels resolution, 8.8 mm/24 mm (35 mm format equivalent), Field of View (FOV) 84°, and f/2.8-f/11 autofocus 1 m-∞ was deployed to perform the remote photos acquisition. Two Mapir Survey 3 multispectral cameras were implemented on board of the UAS. One equipped with a Red Green NIR (RGN) filter, able to capture the reflected light of 660 nm (Red), 550 nm (Green), and 850 nm (Near Infra-Red), and the second with a Red Edge (RE) filter for 725 nm light detection. A GNSS Reach RS+ (Emlid) connected to an NTRIP correction system was used to record X-Y coordinates of six Ground Control Points (GCPs) in Olbia and 10 GCPs in Guspini for accurate orthomosaics orthorectification. UAS flights were planned through the DJI Pilot android app (flight details are reported in Table 1).

2.3. Surveying Dates

In Olbia site were performed multiple flights during crop’s development phases. The first flight (7 July 2021) aimed at obtaining the digital model of the reference ground elevation (Digital Terrain Model, DTM). The absence of plants helped simplify and optimize CHM calculation for the next surveying days. On this date, the UAS was equipped with only RGB cameras. The second flight was performed on 12 August 2021 (BBCH 25 [27]). This surveying was crucial for obtaining orthomosaics where the only vegetation in the field was the small Canapa plants without any weeds. Thanks to this specific approach, it was possible to apply the segmentation technique based on vegetation indexes. During this flight also multispectral cameras were implemented. The last two flights, executed on 30 September 2021 (BBCH 55) and 11 October 2021 (BBCH 67) respectively during the swelling of inflorescences and few days before harvesting, were intended to compare and verify the performance of the two segmentation techniques for plant detection. For each flight, except the first one with the bare soil, RGB and multispectral sensors were employed, capturing the reflected radiance in red, green, blue, near infra-red (NIR), and RE spectra.

The Guspini survey was performed only on 8 October 2021, a few days before harvesting. The purposes of this flight were the same as described for the other study site.

The flight conditions faced during the experiments were optimal: sunny days, clear sky, and strong wind absence at both sites.

2.4. Software Employed

Two software were used during the experiments: Agisoft Metashape (a structure from motion software) to process the UAS’s images and obtain the digital models, and QGIS to analyze the orthomosaics and perform the segmentation.

2.5. CHM Segmentation of Canapa sativa L.

CHM segmentation is based on the height difference between the soil surface plus the overlying Canapa plants (Digital Surface Model, DSM) and the soil (the DTM), performed using the raster calculator tool in QGIS.

Figure 3. A portion of the study area (a) and the yellow polygons obtained through Canopy Height Model (CHM) segmentation (b) after the manual removal of undesired polygons. The image highlights how the CHM segmentation technique helped to detect most of the plants.

Figure 3. A portion of the study area (a) and the yellow polygons obtained through Canopy Height Model (CHM) segmentation (b) after the manual removal of undesired polygons. The image highlights how the CHM segmentation technique helped to detect most of the plants.

The raster calculation tool was used to apply threshold height of 0 m to separate plant pixels from soil pixels. This value was chosen because of the best performances in detecting Canapa plants. Thanks to the conversion of the obtained raster into a vector layer, it was possible to obtain the detected plants number. The side effect was represented by the detection of other objects besides plants like stones, weeds, part of the soil, etc. Of the multiple threshold heights tried during the elaboration process, no one was able to detect only Canapa Sativa L. plants and no secondary elements. For this reason, once found the best threshold height based on the bigger number of plants and the lowest number of secondary objects, it was necessary to execute a manual cleaning procedure to remove all noise polygons from the study area (Figure 2).

For Guspini plants segmentation, the challenge derived from the presence of grown entangled plants. The segmentation provided several polygons consisting of several plants each, resulting in problems and errors in estimating the number of plants (Figure 4a). The problem was overcome by using the Voronoi tool in QGIS. Starting with the manually added plant positions, it was possible to create a Voronoi map with a polygon number equal to the exact number of plants (Figure 4b). The creation of the Voronoi map was possible only thanks to the manual positioning of all plants. The previous polygons obtained with the CHM segmentation technique were then clipped using the Voronoi map (Figure 4c).

The result shows the ability of this procedure to separate all merged plants in the same polygon and give a more reliable plant number. In the discussion section the problem related with the positioning of all Canapa sativa L. plants will be discussed to reduce the effort needed to place manually each plant and to automatize the process.

2.5. Otsu Thresholding Technique for Canapa sativa L. Segmentation

The Otsu thresholding technique derived by the reflectance distribution of vegetation indexes is based on the ability to differentiate between soil and plants' pixels. Starting with the analysis of the vegetation indexes’ histogram created on QGIS, it was selected using a python script. The 0.15 NDVI value was obtained as threshold (Figure 5).

Using the raster calculator tool, a new raster able to differentiate between plants and soil pixels was created. The polygon cleaning procedure is the same described for the CHM segmentation procedure. Due to the massive presence of weeds surrounding Canapa plants, it was possible to perform this technique only on the first surveying date in Olbia. The other dates were characterized by the presence of large zone constituted mainly by weeds (Figure 6).

2.6. Otsu Thresholding Technique for Canapa sativa L. Segmentation

The detection performances evaluation of the proposed supervised procedure started from the polygons derived by plants segmentation. The digital models used for the elaboration were 3 for Olbia and 1 for Guspini study sites. The possibility to use 3 digital models in Olbia was crucial to evaluate the performances of the missing plant detection procedure through the developing stages of the crop. The 8 October 2021 model obtained for Guspini had the objective of evaluating the procedure in a different site and comparing the results with the ones from Olbia in the days before harvesting with different volume growth of plants (small plants for Olbia and well grown plants in Guspini).

The first step for the performance evaluation was represented by the segmentation procedure described in the previous two chapters. The next phase regarded the creation of circular buffered plant positions derived from the buffered plant position inserted manually. The diameter of each buffered plant positions was 0.5 m (this value was selected based on the planting distance). This last procedure was essential to be completely sure of overlapping the small Canapa plant polygons created during the segmentation process and not perfectly positioned over the related plant position as shown in Figure 7.

Thanks to the tool “extract by location” in QGIS and the two overlaying layers, it was possible to automatically distinguish between detected Canapa plant (true positive) and the sum of missing and undetected plant (respectively true negative and false negative). This process is based on the overlapping between green plant polygons, extracted by the segmentation process, and the red circles buffered plant positions inserted manually (sure position of the plants after the plantation) to select only the plants still present in the field. If the buffered plant position corresponded to a segmented plant, this polygon was counted, otherwise, if no segmented plant was selected by the buffered plant position, it was counted as a missing/undetected plant. This process was performed for plants segmented by Otsu and CHM methods in both experimental study sites.

The last step regarded the missing plant detection process performance through the following definitions:

• True Positive (TP), polygons detected through the segmentation procedure that are effectively Canapa plants;
• True Negative (TN), Canapa plants that were not detected using segmentation techniques and effectively are not present into the field and considered as real missing plants;
• False Positive (FP), polygons recognized as plants that are not Canapa (e.g., weeds, rocks, etc.);
• False Negative (FN), real Canapa plants present in the field that were not detected by the segmentation process and classified as missing plants.

The detection accuracy was also calculated (equation 1) and reported in Table 4 in the results section:

$$accuracy=\frac{(TP+TN)}{(TP+TN+FP+FN)}$$

(1)

3. Results

3.1. Segmentation Results

Table 2 reports the two segmentation techniques results. Where the “Plants” indentation refers to the number of plants present in every single date of survey during the growing season and manually inserted in QGIS; “segmented polygons” refers to plant polygons generated after the segmentation procedure; the “plant polygons” are the plants obtained after the manual deletion of all the polygons representing extraneous objects (FP). The obtained results show how the value of segmented polygon is a counting estimation close to the real number of plants present in the field (Ground Truth). Nonetheless, a fair number of polygons does not include a plant, forcing the research team to execute a manual removing, decreasing the number of detected Canapa plants inside the field.

If only the real plants values are considered, the number of detected plants tends to increase during the growing season, reaching the 86.6% and 84.7% of existing plants respectively in Olbia and Guspini study sites. It can be noticed how the Otsu technique has been able to perform the segmentation only in the first date reaching a 95% of detected plants. This result is particularly interesting since it allows the obtainment of the position of each plant at the beginning of the growing season. Such information is crucial to speed up the detection process and the multispectral analysis in the following surveying dates. In the remaining dates it was not possible to get any reliable detection results due to the presence of weeds covering the surface between the Canapa plants.

On the contrary, the CHM technique showed better performances in the latter phenological phase of plant development. Table 2 helps have a better understanding of how the segmentation techniques perform in terms of created polygons. Despite the improving performances, it can be stated that without a manual supervision plants detection would be compromised by the presence of FP polygons.

3.2. Detection Performances

Table 3 reports the obtained results of the detection procedure. The TP, TN, and FN metrics (already defined in section 2.4.4) are also reported in percentage referring to the plants' number during the survey date. FP values are equal to zero in each date thanks to the manual removal executed by the operator. For this reason, they were not considered in plant detection performances (Table 3) since they were completely manually removed. This result helps define the segmentation techniques’ performance to detect Canapa plants. Otsu technique showed the highest performances in terms of correctly detected plants (TP) compared to the CHM during the first surveying date. The number of TP tends to decrease in the second date due to the presence of weeds in the field and the need to switch to the CHM technique, not the best choice when the size of the plants is still reduced.

Voronoi techniques described in section 2.5 helped gaining better results, moving from 1844 (84%) to 2039 (93.7%) correctly detected plants. During the growing season many plants died, decreasing the total number in the field. This reduction was detected thanks to the system described in section 2.6 and showed that the proposed techniques can detect missing plants (TN). In the first date FN values derived by CHM are higher than the one obtained though the Otsu technique, confirming the bigger reliability of Otsu at the beginning of the growing season. Again, Voronoi technique helped improving the detection performance by reducing the FN values in Guspini.

Table 4 reports the accuracy results based on the equation reported in section 2.6. Results confirm what stated for Table 3: Otsu is the best segmentation technique during the early season reaching 0.95 accuracy (considerably higher compared to the CHM); the accuracy reached high performance during the whole growing season; Voronoi technique sensibly improved the accuracy reaching a 0.94 value in Guspini.

5. Conclusions

The comparison between two segmentation techniques for the detection of Cannabis sativa L. plants in two experimental fields in Sardinia (Italy) showed better performance of the OTSU segmentation in the initial phase of cultivation to localize the plants and use it as a reference to evaluate detection performance. The CHM technique showed increasing performance in the Olbia study site but, due to the small size of the crop, it was not possible to achieve the same values obtained in Guspini, where the plants showed greater vegetative development. Although faster and more intuitive, the overall results described are not reliable unless carried out under the supervision of an operator. Although automatic procedures for plant segmentation and detection based on machine learning approaches imply a complex training, testing and validation process and specific programming skills, they represent the best option to obtain more reliable results and perform real-time operations. The main limitation is still related to the lack of extended labelled datasets, which are crucial for learning and testing phases. Future work will consider the comparison between supervised techniques and machine learning approaches and the development of the proposed technique as a labeling facilitation process.