Detection of Aspergillus Flavus in Figs by Means of Hyperspectral Images and Deep Learning Algorithms

1. Introduction

Plant and crop diseases are an economic cost to the agricultural and food sectors. Mycotoxins are substances that are naturally produced by certain types of fungi under particular conditions of humidity and temperature. The presence of mycotoxins in food poses a serious problem for the health of human and animals. The International Agency for Research on Cancer (IARC) included a group of aflatoxins into the group 1 carcinogenic substances [1]. The common via of mycotoxin exposure is the ingestion of contaminated food. Among the risks to human health, risks to the immune system, kidney disease and a higher probability to some types of liver cancer has been indentified [2,3,4]. The fungus Aspergillus flavus, which occurs at temperatures between 12º and 27º and 85% humidity, multiplies in foods such as maize, peanuts, rice, nuts, figs, etc., and although its presence is typical of tropical climates, it also proliferates under some irrigation conditions. In this study, we focus on the proliferation of Aspergillus flavus fungi in fig tree fruits, especially fresh figs.

The cultivation of the fig tree (Ficus carica L.) has its origins in the Carian region of Asia and has spread to other areas such as the Mediterranean region, Africa and the Americas. Today, Spain is the sixth largest producer in the world, accounting of $3.5\%$. Figs grown in Spain reached $21,719$ ha in 2023 ($17,381$ dry and $4,339$ irrigated). The region of Extremadura with $12,771$ ha ($10,306$ dry and $2,465$ irrigated) represents the largest producer in Spain [5]. In terms of production [6], Extremadura accounted for $55.5\%$ ($24,296$ tonnes) of national production, which reached $43,800$ tonnes in 2022 (data of 2023 are not available yet). The increase in productivity is linked to the adoption of innovative techniques such as fertilization, pruning, soil treatment and irrigation [7,8]. Irrigation techniques pose a trend that can be observed in the gradual increase in the number of hectares under cultivation for irrigated figs ($2,465$ vs ) and the decrease for dried figs ($10,306$ vs ).

Arid and semi-arid areas have perfect conditions for fig trees cultivation, but its shallow roots and prolonged droughts result in a lack of water and mineral requirements to ensure the quality of the fruit [9]. In order to alleviate the tree stress, increase productivity and improve the fruit quality, supplemental irrigation arose as a good alternative for different crops [10], included fig trees [11,12]. However, changes in humidity facilitate the spread of Aspergillus flavus mycotoxin and scientists needs to carry out additional research to analyse and prevent infected figs from entering the human food chain.

The Scientific and Technological Research Center of Extremadura (CICYTEX-“Finca La Orden-Valdesequera” located in Guadajira (Badajoz)) is referent in agri-food research, including fig tree cultivation, on which different research works have been carried, analysing the proliferation of the Aflatoxin before and after harverst. Just to mention a few, researchers in [13] addressed the effect of the temperature during the fig productive process, controlling the hygienic and sanitary quality of the figs. In [14], authors evaluated the application of two treatments at the early post-harvest stages of dried figs to control mycotoxin. Authors in [15] analysed the suitability of two yeast strains (H. opuntiae L479 and H. uvarum L793) to act as biopreservatives, to inhibit A. flavus growth and to reduce aflatoxin accumulation.

Even though the detection of the aflatoxin Aspergillus has been widely addressed, the most commonly used methods are intrusive and they involve destruction of the sample. On the other hand, dried figs have been the main focus of the application of innovative methods using AI techniques. However, fresh figs are perishable, they have a limited shelf life and are more sensitive to microbial growth [16], which alters the quality of the product and becomes a serious risk to the human health. The detection of the aflatoxin Aspegillus Flavus in Fresh figs is the goal of this study.

Over the past few decades, the use of hyperspectral imaging (HSI) techniques has been on the rise as an analytical tool. Its performance identifying chemical composition faster than traditional methods and non-destructively meets the needs of the agriculture sector, among others [17]. Furthermore, the superficial distribution of the aflatoxin is an advantage for the analysis by HSI. Thus, last decades its application has attracted the researcher’s interest. However, data provided by HSI poses high dimensionality, which is challenge for classification algorithms. In this light, Deep Learning (DL) has become an innovative and not intrusive method to analyse and interpret HSI data [18,19].

The aim of this study is to develop an accurate method for the early detection of aflatoxin in fresh figs. Our proposal is based on the analysis of hyperspectral images at a pixel level via non-intrusive methods such as deep learning. Two different phases can be identified in this research work. The first one is a parameter tuning phase that is responsible for finding the best parameter values for the deep learning algorithms to be applied in the next phase; the second one corresponds to the comparison of two different deep learning approaches: (i) A Full Connected Neural Network (FCNN) that works with the hyperspectral signatures of each image pixel, and (ii) A Convolutional Neural Network (CNN) that is fed up with images obtained after applying the wavelet transform to the spectral signature of each image pixel.

The rest of the paper is organized as follows: Section 2 analyses different research work in the literature. Section 3 presents the methodology applied to perform the analysis. Section 4 presents the results obtained. Finally, the conclusion and future work are presented in Section 5.

2. Background

Precision agriculture represents a key instrument to increase productivity, improve crop quality and address problems due to climate change. The development of IoT systems through the use of remotely controlled sensors, imaging with unmanned vehicles, and the ability to analyse the collected data using AI techniques has made precision agriculture a reality and a dynamic area of research. A wide range of research in this field is reported in recent literature reviews [20,21,22,23].

In this light, HSI has become a popular technique applied in a wide range of areas to identify internal and external features of the objects under study. Just for name a few, mineral mapping [24], atmospheric characterization [25], biological and medical issues [26], food and agriculture [27,28], among others. HSI can capture fine spectral features through consecutive channels and many narrow spectral bands [29], the larger the number of bands, the more detailed the spectral characteristics. However, not all bands include relevant information and enhance the accuracy, so removing redundant wavebands is useful to facilitate the analysis [30]. Detailed information facilitates the detection of subtle changes and improves the accuracy, which is a crucial factor for the early detection of plant diseases [31].

HSI offers significant advantages in terms of monitoring tasks. The interest in using it in the agri-food sector has been driven by the need to improve product quality and provide automated, non-destructive methods at all stages to identify potential issues early, including plant diseases, and provide a solution [32]. In addition, HSI generates huge amounts of complex data that needs to be analysed. This is a resource-intensive and time-consuming task for which machine learning techniques, especially deep learning, are well suited to [33,34,35]. During the last few decades, the use of Machine Learning in all fields of science has skyrocketed. ML makes use of a wide range of algorithms that allow computer systems to learn from data and to identify patterns in order to infer outcomes. Authors in [36,37] presented comprehensive reviews focused on food sciences and safety applying Neural Networks, and general ML techniques, respectively.

With regard to the aim of our study, the identification of Aspergillus flavus is challenging due to the variety of the morphological features, even genetically [3,4]. As previously mentioned, the aflatoxins produced by A. Flavus, in particular aflatoxin $B_1$, are very harmful to animals and humans. Their detection is essential to prevent contaminated food from entering the food chain [38].

Siedliska et al [39] applied HSI with the assistance of two different cameras, VNIR (400 nm-1000 nm) and SWIR (1000 nm-2500 nm) on strawberries inoculated with the fungal pathogens Botrytis cinerea and Collatotrichum acutatum (900 samples for each different type) and healthy controls (900 samples). The images were taken every 24 hours after inoculation (for each set of strawberries). Based on the second derivative of the original spectra, they selected 19 wavelengths. They carried out two experiments using the FCNN, RF and SVM models, obtaining better results with the FCNN network model.

In [40] authors presented an extensive literature review, where studies published used HSI aimed to identify fungal and mycotoxin-contaminated individual grain kernels applying different classifiers. Authors in [41] addressed the detection of aflatoxin in peanuts using a deep convolution neural network (CNN) and they achieved an accuracy of $95\%$. In a later work [42] they also included corn with a dimensional CNN, where the accurary was $96.2\%$ and $92.1\%$ for peanut and corn, respectively. Similarly, [43] recently addressed the identification of mycotoxins produced by Aspergillus flavus in peanuts and corn using six ML models. With an accuracy of $100\%$ in distinguishing between healthy and infected samples, the authors pointed out that better data processing and further analysis are needed.

Kim et al [44] presented a method based on hyperspectral fluorescence, VNIR, or SWIR imaging combined with machine learning algorithms to detect contamination of aflatoxin in ground maize. Higher accuracy and generalizasion is reached by the SWIR with SVM model.

Contamination in figs has been usually addressed in dried figs. Just to mention a few recent ones, in [45], authors analysed UV light images using a deep CNN approach to detect and classified aflatoxin in dried figs. They carried out a fine-tuning process for pre-trained models, and obtained a validation accuray of $97.50\%$. The same authors [46] proposed a real-time method applying pre-trained models for deep feature extraction and classification. Images were acquired using light sources of different wavelengths. The best performance was achieved by SVM with accuracy of $100\%$ and $92.3\%$ for contaminated and non-contaminated samples, respectively.

Our study focuses on fresh figs, which will also be consumed, thus entering the food chain and posing a risk to human health. On the other hand, detecting before drying means eliminating the risk at an earlier stage.

3. Materials and Methods

This section describes the methodology applied to carried out this research work. Figure 1 represents the general methodology scheme, where the main steps are defined.

This study manages images of figs collected from a fig tree plantation of calabacita variety at the "Finca La Orden-Valdesequera" located at 38º51’N, 6º40’W, altitude 184 m Guadajira, Spain, where CICYTEX has its headquarter. This area is characterized by a mediterranean climate, with dry and hot summers, where the average annual temperature between 2000 and 2022 was 16.28ºC, rainfall 447.08 mm and reference evapotranspiration 1296.15 mm. The soil texture was loamy loam and the farm was provided by a irrigation system. The following section details the procedure to take hyperspectral images.

3.1. Image Acquisition

The images were taken with a SPECIM hyperspectral camera, in particular the model FX10 VNIR, whose main technical characteristics are: spatial resolution of 1024 pixels, spectral range from 400 nm to 1000 nm (VIS and part of the NIR), 448 spectral bands and a spectral leap of $1.339$.

We worked with a set of 320 figs harvested from the plantation previously mentioned. Hyperspectral images were taken for 2 weeks, each week using 160 figs harvested at different stages of maturity. Each week, the 160 figs were divided into four subsets of 40 figs. The first corresponds to the healthy controls (class 0); the following three sets of 40 figs were inoculated with a concentration of 103 UFC/ml (class 1), 105 UFC/ml (class 2) and 107 UFC/ml (class 3), respectively. Hyperspectral images were taken post-innoculation every 24 hours, for five days on figs that were harvested during the current week, according to protocol with next describe. Each day, one toxin-inoculating fig for each class was extracted for chemical analysis and testing for the presence of the toxin. Thus, the number of figs for each class, except the healthy controls, decreased daily by one unit (from 40 the first day to 36 the last). The inoculation process with the A. Flavus was done by immersion that has a similar surface distribution to natural aflatoxin contamination. Each class consits of 380 hyperspectral images, giving a total of 1900 hyperspectral images.

Once acquired, the images were pre-processed with black/white correction to remove noise due to particle variation and to compensate for poor light distribution. This black/white reference is provided by the camera during hyperspectral imaging.

As the inoculation procedure was carried out by immersion, the region of interest (ROI) was not the entire surface of the hyperspectral image, but rather the lower part of each fig image.

Given that our study works at the pixel level, it was necessary to delimit the study areas from which the pixels were to be extracted. In this light, pixel masking was carried out to identify the infected regions and extract the pixels only for these regions. This task was performed with the Labelme software 1, a free and open source image annotation tool developed in Python, and widely applied to label and generate masks for different types of images. Figure 2 shows an example of mask generation and labeled for a set of figs.

For all the images, pixels in the ROI are extracted, this process uses the coordinates of the pixels to be studied. Each pixel was extracted with its 448 spectral bands representing its features, and a new feature was included indicating the class to which it belongs. Additional information has also been included to make it easier to identify the pixel. Figure 3 shows the shape of a fig pixel and its hyperspectral signature.

As a result, our dataset consists of $1,330,300$ hyperspectral pixels with 450 features (448 spectral bands, class label and pixel identification).

3.2. Data Analysis

Deep learning (DL) has become an innovative and not intrusive method for image classification. However, before using any AI technique is necessary to analyse data we are working with. This section carried a brief statistics analysis before addressing the main part of this work. Considering a spectral band as a predictive numerical variable, all spectral bands obtained values between 0 and 1, as expected. Three bands stand out from the rest, taking into account the standard deviation: Bands closed to ultraviolet (400nm-415nm); bands between the blue and the orange colours (526.17nm-623nm), and bands closed to the red and infrared channels (700nm-1000nm). The distribution in the first few spectral bands appears to be symmetrical, and there is a slight difference between the mean value and the median value. From 500 nm the distribution is slightly asymmetric towards high values.

Next, the distribution of each spectral band by class was analysed. Thus, the dataset was separated into each of the categories and we calculated the 10th, 25th, 50th (median), 75th, and 90th percentiles along with the maximum and minimum values. The aim was to find differences in the distribution of the bands in the different classes. We can conclude that maximum values are really similar for all classes and differences are present between $571nm$ and $665nm$. Regarding minimum values, we appreciate differences in the previously mentioned range of bands (400nm-415nm, 526.17nm-623nm, and 700nm-1000nm). Figure 4 shows a comparison of these distributions.

With regard to the number of instances per classes, classes are balanced with $313,500$, $327,500$, $346,400$ and $342,900$ pixels belonging to classes 0, 1, 2 and 3, respectively.

The dataset was randomly divided into two different dataset, 80% for the training set ($1,064,240$) and 20% for the test set ($266,060$). 20% of the training set was used for training validation. In addition, 20% of the instances get for training was randomly chosen for training validation ($212,848$) to avoid the over-fitting issue. Before run the different algorithms, we analysed if data were balanced both in the training and test sets. Figure 5 graphically shows the balanced distribution of instances per class in training and test sets. Next section details the DL techniques and perspectives used to address this problem using a comparative point of view.

3.3. Deep Learning Techniques

This comparative analysis has been carried out with two types of DL models, which have demonstrated its performance in a variety of reseach works. As DL models are really sensitive to the hiperparameters values, a hyperparameter search was done to find the optimal ones for each type of model, which will be detailed in Section 3.4.1. The DL models chosen for this study were:

3.3.1. FCNN

This model has an input layer of 448 neurons (spectral signature pixels). The number of hidden layers and the number of neurons in each are determined by the hyper-parameter search. The output layer has 4 neurons. Softmax is defined as the activation function to select the class to be predicted.

3.3.2. CNN-FCNN Pipeline

This model consists of an input layer of 448, although in this case it is a 2D image (90 px height and 448 px. height), after the application of the continuous wavelet transform to obtain its frequency information. Wavelet transform facilitates image compression due to its ability to localize time and frequency. The number of intermediate convolutional layers, hidden layers and their number of neurons, the activation function, and the pooling technique to be applied are determined using a hyperparameter search. The output layer has 4 neurons and the activation function used is the softmax function.

A Wavelet analysis can provide information in the frequency domain of the hyperspectral pixel, revealing details that are missed in the time domain. Regarding the wavelet transform, the selection of the mother wavelet is crucial to obtain a smooth continuous wavelet amplitude and phase information. Since complex Morlet wavelet (CWT) is a non-orthogonal complex exponential wavelet tapered by Gaussian, it meets both requirements [47]. CWT, represented by eq. 1 2, is an effective method for image processing, and it is chosen as the mother wavelet.

$$\Psi \left(t\right)=exp{\displaystyle \frac{-t^{2}2}{cos}}5t$$

(1)

Figure 6 shows the architecture models for both approaches.

3.4. Sustainability Analysis

In our view, there is a need for researchers to understand the environmental impact of their AI models. AI requires a huge amount of computing power to train the models, and it can consume large amounts of energy and generate large amounts of carbon dioxide emissions. In this work, we take into account the impact of the solutions proposed during the training phase. With that said, CodeCarbon and Green Algorithms are the only Python tools that estimate energy consumption and emissions for memory, CPU and GPU separately.

CodeCarbon tracks carbon dioxide ($C{O}_2$) emissions from machine learning experiments, measuring the carbon footprint in kilograms $C{O}_{2eq}$ per kilowatt-hour consumed. This tool helps estimate and minimise the environmental impact of AI models by providing accurate data on energy consumption and greenhouse gas emissions [48]. Recent studies [49] confirm that CodeCarbon provides values close to those provided by a wattmeter. In this way, both the energy consumed by memory, CPU and GPU during the training phase of the models and the $C{O}_2$ emissions were evaluated. Results will be drawn in Section 4.

3.4.1. Hyper-Parameter Search

Hyperparameter search was carried out for both network architectures to identify those values that reached the best performance. An accuracy value of $0.7$ was defined as the threshold to address the next level of hyperparameter search within a particular network setting. Table 1 shows the hyperparameters and values tested for the FCNN and CNN approaches. We used the Random Search method over 100 networks or combination of hyperparameter for the FCNN approach and 40 for the CNN. Differences are because of the excessive training time for CNN.

For both architecture networks, the Adam algorithm was used as the optimisation function and the categorical cross-entropy as the loss function.

4. Experimental Results

This section presents and analyses the results obtained for each model individually and comparatively. Furthemore, a comparative study about the training time, energy consumption and carbon footprint during the training phase is shown.

The implementation of FCNN and CNN in this research was done with the Tensorflow library (version $2.17$) and Python (version $3.9$) as the programming language. Table 2 shows the technical information about the computer systems used.

The results are evaluated using a set a well-known metrics. The choice of evaluation metrics used in this study is not arbitrary. It responds to the need to capture different aspects of the performance of the classification models.

• Accuracy provides the total percentage of correct predictions and it is useful in scenarios where the data are balanced, as our data are.
• Precision measures the proportion of instances correctly classified as positive out of those that are predicted to be positive. Precision helps to minimize the number of false positives.
• Recall quantifies the number of positive class predictions with respect to all positive instances in the dataset.
• F-Measure provides a single score that balances both precision and recall in one single metric.

In addition, the confusion matrix is provided to help understand how the models perform and the nature of the errors (false positives or false negatives), and gives us a general overview to improve the models.

4.1. FCNN Results

As a result of the hyperparameter search, the obtained network architecture consists of 8 layers, 1 input layer, 6 hidden layers, and 1 output layer. Regarding the hidden layers, the dropout ratio was applied between them, increasing with the network architecture. In addition, ReLU has been chosen as the activating function. Figure 7 shows the FCNN settings and Table 3 details additional information about the architecture structure.

Regarding the hidden layers, the dropout ratio was applied between them, increasing with the network architecture. In addition, ReLU has been chosen as the activating function. This model has been training Results reached with this approach are shown in Table 4, which represents the confusion matrix, where columns represent the predicted class and rows the real class. To clarify the results, Table 5 shows the values for all different metrics used to evaluate the model presented.

The analysis of the results shows a good performance of the FCNN model. On average, $83\%$ of instances are correctly classified. The next step is the analysis of each metric individually. Considering that classes 1, 2 and 3 correspond to different levels of aflatoxin inoculated, the precision metric is high and there is a low incidence of false positives. Class 0 represents healthy controls and the precision value is lower, and consequently, the number of false positive results is higher than in the other classes. Due to the low presence of false negatives in the corresponding predictions, class 0 has a higher recall value than the other classes. The F1-measure is the harmonic mean between Precision and Recall. Classes 1,2,3, which represent the inoculated instances, obtain similarly good results, and only class 0 obtains a low value due to its low precision value. We can conclude that our model based on the spectral signature of individual pixels can identify cases inoculated with aflatoxin with high values for each metric, but performs worse for healthy controls. To complete this analysis, we measure the goodness of the classification results generating the ROC curve for each class and computing the AUC value for each ROC curve. Figure 8 plots how well the classification model behaves for each different classes. Even though further research needs to be done, the FCNN approach obtained high AUC values.

In addition, we analysed the environmental cost of this solution during the training phase. Table 6 presents the training time in hours, the $C{O}_2$ emissions in Carbon dioxide equivalent kilograms ($C{O}_{2eq}Kg$) and the energy consumption (KWh)for RAM, CPU and GPU. The emission rate, which is measured in $kgC{O}_{2eq}$ per hour, provides a more detailed view of the impact on the environment per unit of time. Finally, energy consumption in kilowatt-hours (kWh) measures the amount of energy consumed during the training process. These metrics are important for the assessment of the environmental and energy costs associated with the development and training of artificial intelligence models.

Although this experiment does not release a large amount of CO2 into the atmosphere, the measurements are an important part of improving the models and reducing their environmental impact.

4.2. CNN Model

The purpose of this experiment was the processing of images after wavelet transform. It was started with the network architecture shown in Figure 9. This was the one that gave the best result in the hyperparameter search. This architecture consists of an input layer with a number of neurons equal to the size of the image (90 px high; 448 px wide), a combination of convolutional and sequential layers (detailed in Table 7), and an output layer using Softmax as the activation function.

This experiment was launched on the Nvidia A100-PCIE-40GB GPU, because of the computational requirements. Results obtained in the test phase are detailed in the confusion matrix in Table 8, where the instances correctly predicted (main diagonal) and uncorrectly predicted for each class are shown. Based on these results Precision, Recall y F1-measure metrics are computed in Table 9 to analyse the performance of the model when predict each class individually.

On average, $77\%$ of the instances are classified correctly and it is a relatively good performance. Next, we analyse each metric individually.

The best result for Precision metric is obtained by class 3, $82\%$ of the instances were correctly identified. Class 3 correspond with the highest inoculated concentration. Classes 0, 1 and 2 reached $74\%$, $76\%$ and $77\%$, respectively. This results showed that higher concentration of the aflatoxin are better identified with a lower number of false positives. Regarding the Recall metric, all classes obtained a value $>=74\%$, indicating that the model is able to identify relevant instances (true positives and false negatives). The higher value ($81\%$) is obtained by the class 0 (healthy controls). The harmonic mean between Precision and Recall, represented by F1-measure are similar, all values are between [$0.75$ - $0.79$]. This metric gives us information about model reliability, although we need to explore this issue further, the results are promising.

As it was done before for the FCNN model, the ROC curve is generated for each class and the AUC values are calculated. Figure 10 shows the results obtained, where the AUC values reveal a high discriminative power of $0.95$, $0.95$, $0.93$ and $0.95$ for classes 0, 1, 2 and 3, respectively.

As for the FCNN model, we have evaluated the environmental impact of this proposal during the training phase. Table 10 presents information previouly described, the training time in hours, the $C{O}_2$ emissions in Carbon dioxide equivalent kilograms ($C{O}_{2eq}Kg$) and the energy consumption (KWh) for RAM, CPU and GPU, and the emission rate, which is measured in $kgC{O}_{2eq}$ per hour.

This experiment releases a significant amount of CO2 into the atmosphere and consumes a lot of energy. In addition, the emissions and energy consumption are much higher than those of the FCNN, whose metrics are higher. We must pay attention to this information for further planned research to reduce environmental impact.

5. Conclusions

In conclusion, this study demonstrates the feasibility of early detection of Aspergillus flavus toxin in fresh figs using hyperspectral imaging combined with deep learning techniques, without the use of invasive techniques. The FCNN model achieved an accuracy of $83\%$, while the CNN model performed slightly lower, with $77\%$ accuracy. These results demonstrate the effectiveness of pixel-level analysis in identifying the presence of the fungus, both across different days and different contamination concentrations. This evidence supports the viability of the non-invasive techniques applied in improving food safety.

Although both approaches demonstrated relative success, it was observed that the FCNN network exhibited superior performance in classifying samples infected with different aflatoxin concentrations, while the CNN encountered difficulties in distinguishing between the different classes. Moreover, the detection of healthy figs presented challenges in both models, with a higher number of false positives. This suggests the need for improved algorithms to reduce errors and improve the classification of uncontaminated samples. The models presented in this paper have considerable scope for improvement, with the ones presented in this work representing a robust and highly promising starting point.

The necessity of assessing the environmental impact of AI models is underscored. During the training process, the CNN network exhibited considerably higher energy consumption and CO2 emissions than the FCNN network. This highlights the imperative to strike a balance between technical performance and environmental sustainability in future studies. The work suggests further research into optimising models to enhance their accuracy and reduce their ecological impact.