Plant Identification Using Convolution Neural Network and Vision Transformer-Based Models

1. Introduction

Plants are one of the most essential life forms on Earth and play a significant role in maintaining healthy ecosystems. In order to obtain information about the uses of any plant, users must first identify the plant, by matching the physical characteristics to a specific name (either scientific latin name or common name). Knowing one or more discriminating features of an unknown plant (e.g., shape, color, petal, sepal length) helps in identifying the candidate species. Identification of plants using key features is difficult for people who don’t have specific knowledge of plants and even for specialists such as botanist, agroforestry managers, and scientist to identify plants correctly at different hierarchical levels [1]. Variation of key characters among species and even within species, are some of the challenges for identifying plants manually. Hence, automated species identification can be used for the identification of plants [2]. Automated plant classification is an important research area in computer vision. It is a fine-grained classification task concerned with the identification of plants at various hierarchical levels, such as family, genus, or species level [3]. A user can take a picture of the plant using a camera or mobile device and then analyze it with a plant identification model to identify the plant or a list of possible candidate plants at various hierarchical levels. The identification problem faces several challenges due to inter-class similarities among plant families. Another problem is huge intra-class variations in color, background, occlusion, shape, and illumination within the same plant class such as family, genus, or species. Several studies have been conducted to address the plant classification problem using deep learning-based algorithms and have been able to accomplish significant success in classifying plants [2,4,5]. Compared to traditional machine learning algorithms where features were manually selected and extracted, deep learning-based algorithms automatically detect increasingly higher-level features from data [6]. Several works have shed light on plant identification using deep neural networks, which significantly improved the accuracy of large-scale plant classification tasks. Various Convolutional Neural Network (CNN) models have been proposed and implemented for plant identification tasks and achieved better performance compared to other artificial neural networks (ANN) and CNN-based models suggested in prior research [7,8]. Transformer-based architecture has become de facto in natural language processing (NLP) tasks, its application in computer vision attains significantly good performance compared to state-of-the-art convolution neural networks. Vision transformer models have achieved better performance than other CNN-based models for fine-grained image classification tasks [9]. Training both the CNN and Vision transformer models to contain millions of parameters requires large amounts of data to properly constrain the optimization. The requirement for extensive computational resources for training these models motivates to use of transfer learning with pre-trained networks [10,11,12]. While the transformer models have become the de facto standard in NLP applications, their applications in computer vision tasks remain limited but have been rising recently. Vision transformer (ViT) attain excellent results compared to state-of-the-art convolutional networks while requiring substantially fewer computational resources to train [13]. ViT is a model used in the field of computer vision that employs a transformer-like architecture over patches of the image. It works like the transformers used in the field of natural language processing (NLP). Over the years, deep CNNs have been the state-of-the-art networks for image classification but ViT has shown great potential in achieving competitive performance for complex image classification tasks [13]. Internally, transformers learn by calculating the relationship between pairs of input tokens (words in the case of a string), termed attention in NLP tasks. In computer vision, an Image is split into various fixed-size patches. These image patches are used the same way as tokens and ViT calculates the relationship among pixels between various patches. Each of the image patches is then linearly embedded and patch embeddings are finally augmented with one-dimensional position embeddings. Positional information is introduced into the input using position embeddings, which is learned during training. An extra learnable “classification token” is added at the start of the sequence to the patch embedding. The resulting sequence of the embedding vector is fed to the encoder part of the Transformer architecture. A classification head attached to the encoder output gets the value of learnable class embedding to perform the classification based on its state.

This paper has three main contributions. First, a ViT with a Custom balanced loss function is used for handling class imbalance and improving the model performance. Second, the proposed combination of augmentation techniques enhances the quality of data and improves the model performance. Lastly, for analyzing the distribution of images captured from a near or far distance within classes, CNN based classification model is implemented. Near/Far image distribution helped in visualizing the data imbalance issue and enhancing the quality of the training dataset. It also helped in analyzing the performance of the proposed models on both types of distribution. Finally, it also helped in balancing the data when there was a significant difference in the distribution of near and far images by adding more images and using augmentation to include more diversity within classes. All these components have significantly improved the plants classification performance at the genus and species level which can be extended to classification at the cultivar level. Several augmentation techniques were used in combination for this study to enhance the model performance. The research proposed by Hiary et al. [23] shows the importance of image augmentation for improving the model performance. The authors used fine-tuned VGG-16 model to classify flower species for Oxford-17, Oxford-102 and the dataset consists of 612 flower images from 102 categories [24]. Generally, a large amount of diverse training data is required because the small-size dataset may easily overfit the training model. Data augmentation can address this problem and helps in improving the size and quality of training data. Zhong et al., [25] introduced a novel technique for augmentation called Random Erasing, a new augmentation technique to improve the quality and size of training data. Random Erasing selects a rectangular region randomly in an image and erases random pixel values. This improves the robustness of the models and has better generalization capabilities.

CNN (ResNet50 and ResNet-RS-420) and transformer (ViT) based models were used for this study. Images are acquired using digital cameras, mobile phones or other equipment by the Royal Horticultural Society (RHS), UK. Images are then pre-processed, and augmentation techniques [14] are applied to enhance the size and quality of training data. After that, the area of interest was segmented, and the features were extracted. Based on the extracted features, plants are classified at the genus or species level. An automated flower classification task is a difficult task because of the considerable number of similarities among various flower species and due to intra-class variation. More differences in the background, viewpoint, occlusion, flower image scale, indoor-outdoor lighting conditions, climate and season are some of the problems which make the classification of flowers more difficult [15]. In this research, 113 different plant genera and 53 species were considered. It is very difficult to differentiate these plants from a certain distance, specifically for the human eye. Automated image classification using a deep learning-based approach provides performance above the functions of the human eye and produces accurate results [16,17,18]. Many techniques have been proposed for the classification of plants. Şekeroğlu et al. [4] proposed a leaf classification system using a neural network to identify 27 different types of leaves and achieved a recognition rate of 97.2%. Deep learning-based Convolution neural networks (CNN) are quite popular and achieved signification success in image classification based-task in recent years [19,20,21]. CNN models are widely used for plant classification problems and achieved significantly better performance compared to other machine learning and ANN-based networks [7,22]. Deep learning (DL) based methods are widely used for plant recognition tasks with large image datasets. Heredia et al. [8] used a PlantNet database consisting of 250K images belonging to more than 1,500 plant species. The authors used the ResNet50 model and achieved significant improvement in model performance compared to widespread classification models on test data composed of thousands of different species [8]. CNN-based methods have also been used in health care such as medical image classification, and tumor detection [16,17]. The recently proposed transformer-based approach appears to be a major step toward plant identification tasks. Using the self-attention paradigm, ViT models can achieve better results for image classification tasks compared to CNN-based models, such as AlexNet, EfficientNet and ResNet without applying any convolution approaches. The research proposed by Conde et al. [9] on four popular fine-grained benchmarks: CUB-200-2011, Stanford Cars, Stanford Dogs, and FGVC7 Plant Pathology have used a multi-stage ViT framework and achieved better performance compared to CNN-based models. Given the huge amount of training data and computational resources, ViT has shown better performance compared to CNN models in image classification tasks [13]. Based on the literature review, we have used CNN and ViT-based models for the proposed research because these models have shown better performance for classifying plants and flowers in the past.

2. Methods

This research consisted of four main steps, which are image pre-processing, augmentation, feature extraction and classification. First, the plant images were acquired by the Royal Horticultural Society (RHS) and combined with two open-source datasets, PlantCLEF2015 and iNaturalist. The images were pre-processed using augmentation techniques to improve the quality of the training data. The processed image features were extracted using CNN or ViT models. Finally, the extracted features were trained, and the plant classification was performed using CNN or ViT. We also used image augmentation and sampling techniques to balance the distribution of the near and far captured images to improve the model performance.

2.1. Data Collection

PlantCLEF dataset focuses on 1,000 different herb, tree and fern species centered in France and neighboring countries. It contains 113,205 pictures belonging to 1,000 species. PlantCLEF dataset has information about the plants at different hierarchical levels (family, genus, or species). iNaturalist dataset contains around 1 million images for 4,271 plant species. Figure 1 illustrates the distribution of the RHS dataset for 113 genera and 53 species. Figure 2 and Figure 3 illustrate the distribution of PlantCLEF and iNaturalist images used for classifying plants at genus and species level respectively. After combining the three datasets, we selected 113 genera and 53 species representing 88K images and 16K images, respectively. Figure 4 illustrates the distribution of genera and species of the combined dataset. The datasets were highly imbalanced, with some species containing a far greater number of images than others.

2.2. Image pre-processing

As a data preprocessing method, Image augmentation plays an important role in improving the model performance for deep learning-based networks. We used a variety of augmentation techniques in combination with each other to add more information and increase the dataset diversity. Several augmentation techniques, such as Resizing, Color jitter, Gaussian blur, greyscale, Random perspective, Random Rotation, Random Cropping, Sharpness and Grey Scale as shown in Figure 5, are used in combination. Images are resized to 384x384 pixels, and the center is cropped to 224x224 pixels after doing empirical testing with various combination of pixel values. Several combinations of different augmentation methods are tried with various hyperparameter values, and then the best values are chosen by empirical testing to improve the model performance. Augmentation methods help in adding more diversity to the dataset. Image Augmentation also helped in handling class imbalance where there are a smaller number of images.

A CNN-based model was implemented for classifying near or far-captured images labelled manually which helps in seeing the distribution of near and far-captured images for each class. The model was trained on a subset of the PlantCLEF dataset containing approximately 12,000 images. Images of plants were classified as either near-captured or far-captured images. The images were normalized to make the computation efficient. To improve the model performance, we implemented a transfer learning-based pre-trained VGG16 model for classifying near or far-captured images.

Figure 6 illustrates the near/far image distribution for the proposed dataset consisting of 113 genera after combining the RHS, PlantCLEF and iNaturalist datasets. Near/Far image distribution for 53 species after combining all three datasets is illustrated in Figure 7.

2.3. Convolution Neural Network

Custom pre-trained InceptionV3, ResNet50 and ResNet420 were used for this study. Figure 8 shows the CNN-based plant classification system for this research. We used cross validation technique to check the model performance on validation dataset. We tried multiple experiments with different CNN-based networks (InceptionV3, ResNet50, and ResNet420) using images belonging to 39 genera, where each class had at least 100 images. In experiment 1, we used 10,128 images belonging to 39 genera. Pre-trained InceptionV3 network with a cross-entropy loss function is used for identifying plants. The model was fine-tuned by adding custom layers using the TensorFlow framework to make it more suitable for the plant's classification task. In experiment 2, the pre-trained ResNet50 model on the ImageNet database with cross-entropy loss function is used. Augmentation techniques have been applied to the proposed dataset and used for feature extraction using CNN followed by a classification task to identify plants at the genus or species level. PyTorch framework is used for model implementation and training. In experiment 3, we combined our original dataset with open-source PlantCLEF and iNaturalist datasets to improve the data size and quality. During the first two experiments, we did not cover much variation within classes to cover the overall information. Therefore, transfer learning based fine-tuned ResNet50 model is used for classifying 334 genera with at least 100 images in each class. Approximately 220K images for 334 genera after augmentation were used for training the model. The fine-tuned ResNet50 model with PyTorch framework on the augmented dataset is used for model building and training purposes. In experiment 4, We used around 88K images belonging to 113 genera with an average image count per class of 778. Augmentation techniques, such as Resizing, Color jitter, Gaussian blur, greyscale, Random perspective, Random Rotation, Random Cropping, Sharpness and Grey Scale are used for improving the quality of the training dataset and handling the class imbalance. After augmentation, the Image count has been increased to 300K images. The best hyperparameter values and a combination of augmentation techniques are chosen after empirical testing of the model. Custom SoftMax balanced loss function with the ability to handle image imbalance issues is used for model training [27]. Pre-trained ResNet420 model trained on the ImageNet database with custom SoftMax balanced loss function using PyTorch framework is fine-tuned and used for model building and training. Finally, Species classification is done in experiment 5, we have used around 16k images for 53 classes before augmentation. After applying selected augmentation techniques, the image count has been increased to 150K images. Fine-tuned ResNet420 model with a custom balanced loss function is used for species classification.

2.4. Vision Transformer

The ViT models were implemented using the PyTorch framework and can classify plants at different hierarchical levels, such as genus or species level. In experiment 6, we used around 88K images for 113 genera with an average image count per class of 778. Augmentation techniques, such as Resizing, Color jitter, Gaussian blur, greyscale, Random perspective, Random Rotation, Random Cropping, Sharpness and Grey Scale were used for improving the quality of the training dataset. After augmentation, the Image count has been increased to 300K images. Custom SoftMax balanced loss function with the ability to handle image imbalance issues is used for ViT model training [27]. Pre-trained ViT model trained on the ImageNet database with SoftMax balanced loss function is used for model building. The model is then fine-tuned and trained using the last 4 blocks of the ViT model and two linear layers which have been added at the end of the ViT model. In experiment 7, around 16k images belonging to 53 classes have been picked up for species classification. After applying the augmentation techniques mentioned in Section 2.2, the image count has been increased to 150K. Species classification has been done using the same ViT model architecture used for genera classification with custom SoftMax balanced loss function and is fine-tuned for species classification. Figure 9 shows the ViT architecture for the plant classification task used for this study.

3. Result

3.1. Image pre-processing

The CNN based model for classifying images as near or far achieved a validation accuracy of 90.34% and test accuracy of 83% and overall model performance increased significantly after handling the near/far distribution, as shown in Table 1.

3.2. Experiments using CNNs

In experiment 1, the fine-tuned InceptionV3 network with cross-entropy loss function achieved validation and testing accuracy of 76% and 35% respectively. In experiment 2, the pre-trained ResNet50 model on the ImageNet database with a cross-entropy loss function achieved the validation and test accuracy of 58% and 25% respectively. In experiment 3, the fine-tuned ResNet50 model achieved validation and testing accuracy of 68% and 23% respectively. In experiment 4 the fine-tuned ResNet420 model achieved validation and testing accuracy of 83% and 71% respectively. Lastly, in experiment 5, the fine-tuned pre-trained ResNet420 model with a custom balanced loss function achieved validation and testing accuracy of 94% and 84% respectively. A comparison of different experiments and techniques is illustrated in Table 2.

3.3. Experiments using ViT

In experiment 6, the fine-tuned ViT model trained on ImageNet database with custom SoftMax balanced loss function achieved the validation and testing accuracy of 86% and 83% respectively. Finally, in experiment 7, the fine-tuned ViT model with SoftMax balanced loss function achieved validation and testing accuracy of 96% and 92.5% respectively. The model experiment summary table, as illustrated in Table 3, shows the performance of different experiments using ViT performed in this research. The top 3 predicted results metric was also used for measuring model performance, where images were classified correctly if it were present in the top 3 predicted results. From the comparison of results illustrated in Figure 10 and Figure 11, ViT models outperformed CNN based models.

4. Discussion

The first part of this research was evaluated by employing fine-tuned CNN models, such as InceptionV3, ResNet50, and ResNet-RS-420 for feature extraction followed by a classification task. Next, ViT models are used for classifying plants at the genera and species level. By doing the comparison between Table 2 and Table 3, ViT models performed better compared to CNN-based models for the plant’s classification task. Based on the experiment results, it is seen that the correct set of augmentation alone or in combination plays an important role in improving the size and quality of training data, which helps in improving the classification performance. In terms of performance, the testing accuracy of ViT is 83% and 92.5% for classifying plants at genus and species level respectively. Proposed ViT models, fine-tuned for plant classification tasks can be an efficient automated plant classification since they can achieve significantly better performance than fine-tuned CNN-based models. Classifying near or far captured images mentioned in section 2.3 helped in seeing the distribution of near and far captured images for each class. During testing with unseen images, plant images which were misclassified, most of them were images captured from a far distance. The reason behind the misclassification of more images captured from far distance compared to near-captured images is the significant difference in image count for near and far-captured images for each class on which the CNN model is trained. Another reason for the misclassification of far-captured images is the similarity and less variance among different plant classes. Additionally, fine-tuning it with more trainable layers might improve the model performance but it increases the training time and is computationally more expensive. Feedback approach can be used for the model retraining, as shown in Figure 12. Plant images can be captured from the mobile camera using the web/mobile app and sent to the plant classification model API. User can provide feedback in the form of comments or ratings, based on the top predicted results returned by plants classification API. Top predicted results with respective probabilities and API request Id can be stored in databases such as AWS DynamoDB, SQL, or any other DB schema. Based on the returned response from API, user can provide feedback in the form of comments or ratings which can be used for model retraining and maintenance. API response along with user feedback is passed to feedback API where responses are stored and analyzed. Retrain the model if needed based on the feedback received from feedback API.

5. Conclusions

In this work, we conclude that ViT models perform better than CNN-based models and extract more information about the features when classifying plants. We also highlight that augmentation plays an important role in enhancing the data quality and making the network more robust and generalizable. A Custom CNN-based model has been implemented for classifying near or far-captured images which helped in inferring that far-captured images are misclassified more compared to near ones. The experiments using the custom SoftMax balanced loss function suggested that the proposed loss function performed better than the most adopted cross-entropy loss for plant classification tasks. In future work, we would like to include more far-captured images, so that we have an equal distribution of near and far-captured images and improve the near/far classification model with more manually annotated images. We can improve the model performance by handling the misclassification for far-captured plant images and making it more generalizable. Proposed models and techniques for handling data imbalance can be extended for classifying plants at the cultivar level, fruit grading, plant/crop disease classification, quality assessment, flower classification and more. The proposed research can also be used and scaled with location attributes, such as device location, and country. By considering more features and narrowing down the plants based on location attributes, a more robust model with improved performance can be implemented. Based on user input provided on different plant categories such as gardening, indoor and outdoor plants, category-specific models can be implemented to improve the model performance. Tree-based approaches, such as top-down or bottom-up can be implemented for different hierarchical levels, such as family, genus and species based on the use case requirements. Like the near/far distribution classification model used in this study, analysis of leaf, stem, or flower distribution can also complement the proposed research to improve the model performance.