Visible Image-Based Machine Learning for Identifying Abiotic Stress in Sugar Beet Crops

1. Introduction

Achieving global food security is an important 21st century challenge [1,2,3,4,5]. Population growth, climate variability, and mismanagement have reduced the availability of natural resources needed for growing crops [6,7]. Furthermore, biotic and abiotic stresses reduce crop yields, emphasising the importance of adopting precision agriculture practices [7,8,9]. Biotic stresses have biological sources, such as pathogens (viruses, bacteria and fungi), pests, and weeds [7,10,11]. Abiotic stressor examples are radiation, salinity, floods, water and nutrient deficiency and others [10,11,12].

Deficiencies of water and nutrients can significantly reduce agricultural productivity. Precise and accurate early detection of water and nutrient stress in plants can boost agricultural productivity while improving water use efficiency [13,14]. Such detection is a necessary first step for sustainable intensification, increasing crop yields without causing environmental degradation or converting additional non-agricultural land into farmland [15,16,17]. Then, corrective treatments, such as irrigation and fertiliser applications, should be applied to only the areas where the treatments would be beneficial, and in the amount to satisfy the actual need [12,16,18].

Because different stresses can cause similar symptoms, the determination of crop stresses using visible symptoms is often a complex manual task predominantly conducted by trained and experienced agronomists, crop scientists, and plant pathologists. In other words, manual stress detection is labor-intensive, time-consuming, and inconsistently reproducible due to differences in experience and subjective interpretation, and manual ratings[19]. Image processing (IP) and machine learning (ML) can be coupled to mitigate the weaknesses of manual methods. Recent advances in both disciplines have made remote sensing-based plant stress inference computationally tractable [20].

IP and ML techniques have been coupled in multiple agricultural applications [16,21,22,23,24,25]. They have been applied to identify [26,27,28], classify [29,30], quantify [19], and predict [31,32] crop stress. For instance, Raza et al. used visible and thermal image inputs with support vector machines (SVM) and Gaussian classifiers to detect water stress in spinach, achieving 97% accuracy [27]. Stas et al. compared Boosted Regression Trees (BRT) and SVM models using NDVI data from low-resolution SPOT VEGETATION imagery to predict wheat yield, finding BRT consistently superior based upon cross-validation error (RMSE) [33]. Naik et al. used high-quality RGB images of soybeans’ fractional vegetation ratios (FVRs) to develop a real-time classification framework for detecting iron deficiency chlorosis (IDC) [34]. They employed and compared Naïve Bayes (NB), Linear Discriminant Analysis (LDA), Quadratic Discriminant Analysis (QDA), SVM, K-Nearest Neighbour (KNN) and Gaussian mixture model, ultimately achieving 96% accuracy. When [35,36], used hyperspectral and multispectral images to evaluate salt tolerance and vegetation cover under water stress, Random Forest was the most reliable. Khanna et al., used a complex dataset consisting of 55-dimensional vectors of RGB, infrared, and hyperspectral images as well as canopy cover, crop height, reflectance and vegetation spectral indices, to detect crop stress [16].

Building on the work by Khanna et al. [16], this study focuses solely on developing, evaluating, and comparing Machine Learning Image Modules (MLIMs), each intended to identify stresses due to lack of water and nitrogen in growing sugar beet plants. At the beginning of the training and testing process, each MLIM consists of 54 pairs of one-of-six machine learning algorithms and one-of-nine input data sets derived from RGB images and canopy cover. Each such pair, referred to as a Machine Learning Image Submodule (MLIS), is intended to achieve accuracy in identifying the same particular stress. After training and testing, only the best one or more MLIS(s) are retained within each of the four MLIMs.

Unlike previous work that required specialized equipment to acquire, raw RGB images can be easily captured with common devices such as smartphones or SmartGlass. Use of the developed MLIMs would greatly simplify the process of detecting and classifying water and nitrogen stress in crops and determining nitrogen stress severity. Although demonstrated for sugar beets, application of the reported MLIM-development procedure to other crops would enable farmers worldwide to make more informed decisions, improving the economic impact of water use and enhancing agricultural productivity.

2. Materials and Methods

An overview of [16]'s experimental conditions and data is presented in below Section 2.1. Then, the proposed MLIM method of detecting abiotic stresses of different kinds and levels is illustrated. The MLIM method included two phases: preparing input data sets using image processing approaches; and developing and testing MLIMs for detecting and identifying four crop stress classifications.

2.1. Experiment Environment Conditions

The experiment was conducted at the ETH research station for plant sciences in Lindau Eschikon, Zurich, Switzerland (47.45°N, 8.68°E). Sugar beet plants of the variety "Samuela" were grown under controlled climate conditions in a greenhouse. In this experiment, six sugar beet plants were planted in rectangular plant cultivation boxes placed about 2 m below light sources.

The 150 images included 3 replications x 10 dates × 5 treatments: Control, Water stress (including sufficient and insufficient water), and Nitrogen stress (including Nitrogen stress, and more nitrogen stress). Table 1 and Appendix A Table A1 provide more information about the treatment used in this study.

2.2. Dataset and Pre-Processing

Visible images were captured using the Intel®Realsense ZR300 camera. The 150 raw images had 1920 × 1080 pixels. See [16] for more information on imaging setup. These raw images included information beyond the cropped area. Due to installation imprecision, the raw images were from -2 to 2.5 degrees away from perfect alignment. As part of pre-processing, the images were rotated to 0 degrees and cropped from the edges of the box. Subsequently, another crop was performed from the center of each image to ensure all were 600 x 400 pixels in size (Figure 1).

2.2. Proposed Method

Figure 2 illustrates aspects of the proposed crop stress detection framework and processing. In Phase 1 (Figure 2a), raw RGB images are pre-processed and subjected to multiple feature extraction techniques to generate nine different input sets. The set names refer to the employed extraction techniques, including: RGB, canopy cover, Excess Green (ExG), Excess Green-Blue (ExGB), Histogram of Oriented Gradients (HOG), Scale-Invariant Feature Transform (SIFT), Canny edges, RGB-based water stress (RGBWS), and Green index. Use of these diverse input sets enhances the robustness of the presented classification process.

For Phase 2, Figure 2b shows the four machine learning modules trained to identify, respectively: a) stress existence, b) water stress existence, c) nitrogen stress existence, d) the severity level of nitrogen stress, classified as sufficient, nitrogen stress, or more Nitrogen stress. Each of the four modules contains 54 Image Learning Machine Submodules (MLISs). Each MLIS is unique in accepting only one of the 9 input data sets and employing only one of the six tested Machine learning algorithms such as: MLP, SVM, RF, DT, LDA, and SGD.

Figure 2c depicts a close-up of the water stress existence module, when it is accepting merely the RGB data set as input to each of the six Machine learning algorithms separately. Each of the six MLAs will analyze the input and output the determination as to whether the plant had sufficient or insufficient water when the image was taken. That model output is then compared to the actual condition label for validation.

This two-phase pipeline enables scalable, accurate stress detection using only image data and machine learning. Each phase is explained in more detail in subsequent sections.

2.3.1. Phase 1: Generating Input Sets

Table 2 shows the nine input data sets that were created for this study by extraction from pre-processed RGB images. For the Fractional Canopy Cover (FCC) input of Row 2, we used the results from Haddadi’s 2022 study, which analyzed the same sugar beet image dataset used here [37]. Haddadi applied nine compound image segmentation methods to estimate fractional canopy cover under drought and nitrogen stress conditions. They showed that the combination of the Excess Green minus Excess Red vegetation index with manual thresholding achieved the highest accuracy (94.69%). Here, we used their output (Haddadi’s fractional canopy cover estimates), as part of our input data. Table 2, rows 3-9 show the other seven derived input data sets: Excess Green index (EGI); Excess Green minus excess Blue index (EGBI); Histogram of Oriented Gradients (HOG); Scale Invariant Feature Transform (SIFT); Canny Edge Detector (CED); RGB image without background pixels (RGBWB); and Green Band (Green).

2.3.2. Phase 2: Developing Machine Modules for Detection and Classification

The purpose of this phase was to formulate an Identify-Classify-Quantify crop stress Model that accepts RGB images or data derived from RGB images. The proposed model consists of four distinct modules, each designed to perform a different crop stress detection or classification task. The first module determines whether a crop is under stress or not, classifying it into one of two categories: under stress or not under stress. The second module classifies crops as having sufficient or insufficient water. The third module classifies a crop as being under nitrogen stress or not under nitrogen stress. The final module classifies a crop as being unstressed by nitrogen sufficiency, or nitrogen stress, or more nitrogen stress.

In all, each of the four MLIM modules incorporates 54 different Machine Learning Image Submodule (MLIS). Each MLIS employs one of the nine Table 2 input data sets and one of six Machine learning Algorithms (MLAs): Multi-Layer Perceptron artificial neural networks (MLP) [43], Support Vector Machines (SVM) [44], Random Forests (RF) [45], Decision Trees (DT) [46], Linear Discriminant Analysis (LDA) [47], and Stochastic Gradient Descent (SGD) [48].

To identify, classify, and quantify crop stress, the performance of each MLIS was evaluated. The same Seventy percent of the input data was used for training the MLIS to identify, classify, and quantify crop stress. Then, the same thirty percent of the data was used for testing the MLIS. All MLAs, except the MLP model, were developed using the scikit-learn library [49]. MLAs were fine-tuned by adjusting hyperparameters available in the same library. Details of the MLP structure and its hyperparameters’ tuning are provided below.

A multilayer perceptron (MLP) is an Artificial Neural Network (ANN) having multiple layers of interconnected nodes. A neuron in a hidden layer is connected to the neurons in the previous layer, and its output is passed to neurons in the next layer [50,51]. For the multiclass classification problem presented here, a single neuron per class exists in the output layer using the logistic activation function. Each employed MLP model has one input layer, one or more hidden layers and one output layer (Figure 3).

MLP development consisted of three parts: partitioning the total dataset into training and testing sub-sets; looping; and stopping callback. During training, all model weights were set in two ways: (1) Empirically by changing dense layers numbers, neurons of hidden layer and activation functions [52]; and (2) By using Keras tuner’s RandomSearch capability[53]. The optimiser was ADAM, and the loss function was categorical cross-entropy. The convergence of the model was evaluated based on a specific test accuracy value. Use of a loop with a stopping condition halted the training epochs when the highest test accuracy was achieved.

2.3.3. Training Process

The MLP was built using Keras and Tensorflow Python packages, and other ML models were designed by Scikit-learn. An Intel® Xeon® Silver 4210 processor with twenty 2.20 GHz cores, 125.5 GB of memory, and an NVIDIA GEFORCE RTX3090 graphics card was used to train the proposed model. After using 70 % of the input data to train an MLIS, the remaining 30 % of the data was used to test the MLIS and assess its accuracy.

2.4. Statistical Analysis

For each Machine Learning Image Module (MLIM), the performance of its associated Machine Learning Image Submodules (MLISs) was evaluated using statistical performance metrics and computational time. A confusion matrix was generated for each MLIS, and standard machine learning evaluation indicators, including accuracy, precision, recall, and F1-score, were calculated and compared to identify the most accurate MLIS for each module.

$$\mathrm{Accuracy}={\displaystyle \frac{\mathrm{TP}+\mathrm{TN}}{\mathrm{TP}+\mathrm{FN}+\mathrm{FP}+\mathrm{TN}}}$$

(5)

$$\mathrm{Precision}={\displaystyle \frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}}}$$

(6)

$$\mathrm{Recall}={\displaystyle \frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}}$$

(7)

$$\mathrm{F}1\mathrm{Score}=2\text{×}{\displaystyle \frac{\mathrm{Precision}\text{×}\mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}}}$$

(8)

In this study, ‘positive’ refers to non-stress conditions, such as sufficient water or the absence of nitrogen stress, while ‘negative’ refers to stressed conditions, such as insufficient water or nitrogen stress. Here, TP represents True Positives, where the model correctly classifies a non-stress condition; FP represents False Positives, where the model incorrectly classifies a stressed condition as non-stress; FN represents False Negatives, where a non-stress condition is misclassified as stressed; and TN represents True Negatives, where the model correctly classifies a stressed condition.

2.5. Equilibrating Conflicting Objectives

Concerning each MLIS (combination of ML and input data set), two critical outputs of this study are the accuracy of the computed results and the employed computer execution time. Both for each MLIM alone and when comparing all MLISs against each other, a user’s objective of achieving the highest accuracy can conflict with their objective of requiring the least execution time. While achieving the highest accuracy or the shortest execution time might be important for some situations, under other circumstances, it might be preferable to reach an intermediate state that balances both objectives. Bargaining theory includes methods useful for finding the equilibrium point and the trade-off value between such conflicting objectives [54,55,56].

The trade-off between two objectives is calculated by examining how the pareto optimal value of one objective shifts in response to a change in the pareto optimal value of the other objectives, within the set of all feasible pareto optimal solutions defined in two-dimensional real space (S ⊆ℜ², where ℜ² refers to two dimensions in real number space). Assuming a maximization objective (in which the largest objective function value is the best), Equation (9) is used to normalize the optimal accuracy and time-execution values. A normalized value of one, represents the best objective function value, and a zero value represents the worst.

$${\mathrm{y}}_{\mathrm{n},\mathrm{i}}={\displaystyle \frac{{\mathrm{y}}_{\mathrm{i}}-{\mathrm{y}}_{\min ,\mathrm{i}}}{{\mathrm{y}}_{\max ,\mathrm{i}}-{\mathrm{y}}_{\min ,\mathrm{i}}}}$$

(9)

where, $y_{n,i}$ is the normalized ith objective function value, $y_i$ is the ith objective function value, $y_{min,i}$is the ith minimum objective function value, and $y_{min,i}$ is the ith maximum objective function value.

Before applying any conflict resolution method, MLISs dominated by others are removed from the dataset. For example, if two MLISs have the same accuracy but one has a lower execution cost, the combination with the higher cost is considered dominated and removed. In this work, we utilized the Nash method to illustrate how decision-makers can assign proportional priorities to maximizing accuracy versus minimizing execution time to choose between different MLISs.

Nash’s solution, an economics-based bargaining game theory method, is applied here [57,58]. This method aims to find an optimal point upon the Pareto frontier that concurrently is most distant from the point of disagreement where both objectives achieve their worst values. Therefore, a pair of (f^*₁, f^*₂) is a Nash bargaining solution if it solves the following optimization problem:

$$\mathrm{Maximize}{({\mathrm{f}}_1-{\mathrm{d}}_1)}^{{\mathrm{w}}_1}.{({\mathrm{f}}_2-{\mathrm{d}}_2)}^{{\mathrm{w}}_2}$$

(10)

where f₁ and f₂ are the values of the first and second objective functions, respectively; ${\mathrm{w}}_1$and ${\mathrm{w}}_2$ are the weights of importance of the first and second objective functions, respectively; and ${\mathrm{d}}_1\mathrm{a}\mathrm{n}\mathrm{d}{\mathrm{d}}_2$ are the values of the least achievements of the first and second objective functions, respectively. This conflict is defined in the S set (S⊆R²), in which S is the set of feasible bargaining objective function value pairs.

We calculated the average performance for each combination of machine learning model and input set (MLIS) across all four modules. By adjusting the weights assigned to the objectives (high accuracy and low computational cost), we could: a) determine which input sets were most cost-effective (Meaning they balance accuracy with execution time cost) for each machine learning algorithm; and b) determine which algorithms were most cost-effective for each input set.

3. Results and Discussion

The identification of water stress, as well as the identification and severity classification of nitrogen stress, were each addressed using the same set of machine learning algorithms and input sets. Each of the 54 unique combinations of an algorithm and an input set is referred to as a Machine Learning Image Submodule (MLIS). The following sections report the results for each step of phase 2. Phase 2 develops MLIMs for detecting stress existence, the type of stress, and the severity of Nitrogen stress.

3.1. Stress Existence Module

To evaluate the existence of stress, each of the six ML algorithms was run separately with each of the nine different types of input sets. Then, the obtained results were organized and summarized based on test accuracy (%) into a radar chart (Figure 4). According to the radar chart, the further away from the center, the closer to 100%‏ ‏test accuracy, and the closer to the center, the closer to zero accuracy.‏ ‏

One hundred percent accuracy in detecting a stress existence was achieved by the MLP and SGD algorithms using RGB input. With the same input, the RF, DT, and LDA algorithms were the least accurate, below 86%. Using RGBCC input, MLP achieved 100% accuracy, while SVM and DT had the lowest accuracy at 84.44%. Overall, the MLP algorithm often performed the best with all inputs, while DT and LDA frequently were least accurate.

Figure 5 shows the confusion matrix of the most accurate MLISs for detecting stress existence, MLP, and SGD with RGB input. This shows that all predicted classification labels of non-stress and stressed matched the actual true labels of the tested boxed crops. The seven true positives, 0 false positives, 0 false negatives, and 38 true negatives corresponds to correctly classifying all seven non-stress and 38 stressed boxed crops in the test data.

The statistical indices in Table 3 confirm that these models also had 100% precision, recall, and F1-score for both stress and non-stress categories, indicating perfect detection. Notably, this study investigates stress presence or absence, which was not covered in [16].

3.2. Detecting the Type of Stress and Its Severity

3.2.1. Water Stress Module

Water stress detection module was less accurate than stress existence detection module, and the resulting hexagonal Figure 6a had larger radii. The most accurate MLISs had: RGB input with the MLP model (95.56%); EGI with MLP (84.44%); and RGBCC with RF (86.67%). The least accurate MLISs were: CED with LDA (55.56%), SIFT with LDA (66.67%), and RGBWB with DT (64.44%). Figure 7a displays the confusion matrix for the most accurate water stress detection MLIS (MLP with RGB input). This model correctly classified 18 out of 19 instances of insufficient water (stressed condition) and 25 out of 26 instances of sufficient water (non-stressed condition). The model misclassified one sample in each group.

Table 4 summarizes the statistical parameters of the most accurate MLIS for water stress module, obtained using RGB image input. That 95.56% test accuracy was lower than the 97.62%, achieved by Khanna et al. [16], using Subspace KNN and RUSBoostedTrees algorithms with a 55-dimensional input set, including height, canopy cover, reflectance, and hyperspectral bands. Note the relatively comparable accuracy of the much simpler MLP-RGB-based MLIM approach presented here.

3.2.2. Nitrogen Stress

Two types of classifiers were designed for detecting nitrogen stress. The first tried to identify the severity of nitrogen stress at three levels (sufficient, Nitrogen stress, and more nitrogen stress), respectively, resulting from low, medium and high nitrogen inputs, as was previously examined by Khanna et al. [16]. The second classifier type merely tried to detect nitrogen stress existence.

For three-level nitrogen stress detection, Figure 6b shows that the highest accuracy was achieved by RGB with MLP (86.67%), while the weakest accuracy was by RGBCC with MLP and RF (71.11%) (Figure 6b). Also, for the three levels, Figure 7b shows that RGB with MLP was the most accurate classifier, correctly identifying all non-stressed cases (7 out of 7), 15 out of 19 more nitrogen stress cases, and 17 out of 19 nitrogen stress cases. Note that the RGB with MLP model’s 86.67% accuracy was 5.72% higher than the highest accuracy obtained by Khanna et al. [16], 80.95% for the same three-level problem (using an SVM algorithm), with a complex 55-dimensional input set.

For two-level nitrogen stress detection, RGBCC with RF achieved the highest accuracy at 92.59%, followed by RGB with RF and DT, SIFT with RF, and Green with SVM and RF, all with 88.89%. HOG with DT was the least accurate, with an accuracy of 55.56% (Figure 6c). This new two-level detection machine improved nitrogen stress detection accuracy by 5.92% compared to the three-level detection model. For two-level nitrogen stress detection, RGBCC with RF was the most accurate classifier, correctly identifying 20 of 20 stressed samples and 5 of 7 non-stressed samples. This model categorized nitrogen levels into two groups: sufficient (high nitrogen input) and insufficient (medium and low nitrogen input) (Figure 7c).

Several studies indicate that detecting plant stress, such as water and nutrient deficiencies, is more accurate with ANN-based models such as Multi-Layer Perceptron (MLP) than with simpler machine learning models such as Support Vector Machines (SVM), Random Forests (RF), and Decision Trees (DT). This is due to the complexity of neural networks, their iterative training through epochs, and their layered structure. However, results can vary depending on study conditions [49,50,51,52].

3.3. Execution Time and Accuracy Comparison

Figure 8 shows the average computation time used by the MLISs for the above classifications. MLISs employing the MLP algorithm were most accurate but also required the most computation time. SGD was the fastest algorithm. Most algorithms, other than MLP, executed within about one second. Despite slight accuracy differences, faster algorithms such as SGD can be substitute for MLP for some stress treatments.

When considering just one MLA, different input sets required different execution times. The green band input required the least execution time, followed by HOG input. SIFT input required the longest execution times, followed closely by RGB.

Evaluation of MLIS test results shows a clear conflict between processing execution time and the accuracy of the results. To address that conflict using the Nash method, weights were applied to prioritize the two objectives involving accuracy and time. For example, the weight pairs of 50%:50%, 30%:70% and 70%:30% respectively represented: equal weight for both objectives; 30 % weight for accuracy and 70% for speed; and 70% for accuracy and 30% for speed.

First, the goal was to determine the best input set for each MLA. Three optimizations were run using the above weights for accuracy and cost. Table 5 shows the resulting most cost-effective input sets.

Second, for the goal of determining the best MLAs for each input set, three more optimizations were performed, using the same weights as above. Then, for each input set, the most cost-effective machines were determined (Table 6). As shown in Table 4, when considering both time and accuracy equally, the combination of MLISs that stood up were: SIFT for MLP; EGBI for SVM; EGBI for RF; EGI for DT; RG for LDA; and Green for SGD.

If time was prioritized, the results were similar, except that RGB for SVM became the most cost-effective MAL. This indicates that the time factor had little impact on determining the most cost-effective model. When accuracy was prioritized, the best combinations were SIFT for MLP, Green for SVM, Green for RF, RGB for DT, Green for LDA, and SIFT for SGD. Notably, MLP consistently performed well regardless of the weight crossing all three scenarios.

The most cost-effective machines were independent of time and accuracy weights for each machine learning algorithm. The combinations of RGB and SGD, RGBCC and RF, EGI and SVM, EGBI and SVM, HOG and RF, SIFT and MLP, CED and RF, RGBWB and SGD and Green and SGD were the most economical.

4. Conclusions

This study demonstrated the effectiveness of using RGB-based Machine Learning Image Modules (MLIMs) for detecting and classifying stress in crops, focusing on both water and nitrogen stress. Developed and tested MLIMs, each employed one of nine input feature sets and one of six machine learning algorithms for analyzing RGB images to detect: stress; stress due to water insufficiency; stress due to nitrogen insufficiency; and stress due to two levels of nitrogen insufficiency. Achieving perfect accuracy for detecting general stress due to water or nitrogen insufficiency, the MLP algorithm with RGB or SIFT input outperformed more complex methods that require additional data, such as canopy cover and hyperspectral bands. The MLP algorithm with RBG input achieved accuracy within two percent of that achieved by previously published models having much more complex and costly input, such as the 55-dimensional dataset of [16]. This underscores the potential of RGB images as a cost-effective, readily accessible data source. An MLIM that uses RGB input to an MLP algorithm can provide growers with a practical tool to assess crop stress and optimize water and nitrogen use with minimal technological overhead.

In terms of nitrogen and water stress detection, other machine learning algorithms such as RF, SVM, and SGD also performed well, particularly when considering both accuracy and computational time. The MLIMs that best balanced accuracy and execution time, included MLP with SIFT, SVM with Green, and RF with EGBI. Although MLP offered the highest accuracy, its computational burden was significantly higher. Depending on the priority (time vs. accuracy), other algorithms such as SGD and SVM might be preferable for some real-time applications. Overall, this research highlights the feasibility of using simple RGB images combined with machine learning techniques to offer robust and efficient crop stress monitoring solutions, contributing to improved agricultural productivity and sustainability.

SRH, MH, RCP, MS