A Review: Tree Species Classification Based on Remote Sensing Data and Classic Deep Learning-based Methods

1. Introduction and Review Approach

1.1. Significance of Tree Species Information

Timely and accurate information on the spatial distribution of tree species (TS) has an immeasurable value. In forest management, information on TS classification is required for forest inventory[1], biodiversity assessment and monitoring[2], invasive species monitoring[3], and forest sustainable management. In ecology, large-scale spatial information on tree species improves the understanding of the ecology of tree species [4,5]. In the environmental field, tree species information facilitates the estimation of wildlife habitat [6]and forest insect abundance [7].

In recent decades, remote sensing technology has made great progress in spatial and spectral resolution, and a variety of remote sensing data has made it possible to classify TS. The interest of practitioners in remote sensing-derived tree species information is reflected in the survey conducted by Felbermeier et al [8]. They analyzed 347 questionnaires sent to professionals working in the forestry sector. Two-thirds of the interviewees reported deficiencies in forest information, and 90% of them expected improvements through the application of remote sensing. When asked which parameters should be addressed by remote sensing applications, tree species were ranked first out of 63 parameters [9]. Additionally, in the sustainable management of urban trees, remote sensing methods have been an effective alternative to field surveys [10].

1.2. Objectives

Generally, TS classification is done at the species level of the plant classification system, i.e., varieties, species, genera, and families. This review of tree species classification includes species groups, dominant species, stands, and individual trees, but does not include forest type. The main objectives of this review on TS classification are as follows:

Analysis of the literature on the classification of tree species by remote sensing in the past 25 years and quantifying general trends.

Provide a detailed overview of the classic deep learning-based methods for classifying tree species.

Identification of research gaps in TS classification and description of future trends in TS classification using remote sensing data

1.3. Review Approach

In this review, we browsed the literature from Web of Science (WOS) Core Collection and Google Scholar databases for the last 35 years, in the period of January 1988- June 2023, using the following keywords: remote sensing OR LiDAR OR UAV OR tree species OR clsssifi* OR map* OR indenti* OR discriminat* OR detect*. Given the fact that there were fewer qualified papers published in 1988-1998, this review is focused on papers published after 1998, especially after 2013. The studies that were found to satisfy the above conditional search were then further filtered based on the following criteria:

• TS classification objects must be group tree species OR main tree species OR dominant tree species OR stand tree species OR individual tree.

• The research must report on the corresponding specific remote sensing data.
• The research must report the tree species classification methods.
• The research must report the assessment of the classification result.
• A total of 300 papers met the criteria for review in this study.

The remainder of the study is organized as follows: Section 2 analyzes the literature on TS classification by remote sensing in the past 25 years and the quantification of general trends. Section 3 provides a detailed overview of classic deep learning-based methods for tree species classification. Limitations and future work in TS classification are discussed in Section 4. Section 5 conclusions are drawn.

2. Trends in Tree Species Classification

2.1. Remote Sensing Data for TS Classification

Green plants are sensitive to solar radiation as a variety of phytochromes absorb visible light and their internal cellular structures are capable of multiple scattering of near-infrared light, while intracellular water and biochemical constituents absorb short-wave infrared [11]. The spectra of plant foliage and canopy are related by internal leaf structure, the thickness of cell walls, and biochemical components such as photosynthetic pigments [12,13,14,15,16,17], canopy structure, plant phenology, and background signals related to bare soil [15,18,19]. Different tree species have different characteristics such as internal structure, biochemical composition, and phenology, and thus absorb and reflect solar radiation with different intensities, so passive optical remote sensing measures the spectral response of the tree canopy to provide useful information for TS classification [19,20]. Active remote sensing Light Detection and Ranging (LiDAR) sensors can calculate parameters such as tree height, forest density, and leaf area index at the single wood and stand level from the recorded intensities[21,22], which, are mainly determined by the structural morphology of the tree foliage, so there are differences in the parameters of different tree species, and the active LiDAR and passive remote sensing data complement each other in the classification of tree species. Active remote sensing synthetic aperture radar (SAR) sensor is capable of observing the ground 24 hours a day, regardless of weather conditions, and can provide unique tree species classification information.

Typical moderate spatial resolution satellite multispectral image (MSI)sensors include ASTER, Landsat-5 TM, Landsat-7 ETM+, Landsat-8 OLI/TIRS, Sentinel-2A MSI, and SPOT-1-5 [23,24,25,26,27,28,29,30]. Individual TS classification cannot be achieved using single sensor data at such resolutions, and moderate-resolution images achieve TS classification by coupling with other sensor data. Moderate-resolution images have an auxiliary role in the improvement of individual TS classification accuracy [31].

Since 2000, data from Very high spatial resolution (VHR) commercial satellite sensors have shown the potential to create digital base maps [32], and images acquired by VHR sensors enable the successful classification of tree species [33,34,35,36,37,38,39,40]. Typical VHR satellite sensors include GeoEye-1, Gaofen-2, IKONOS, Quickbird, Plé-aides, RapidEye and WorldView-2/3/4 (WV2/3/4).

The most common hyperspectral image (HSI) sensors used for TS classification include AVIRIS, CASI, HYDICE, and HyMAP airborne sensors and Hyperion and CHRIS satellite sensors. The researchers used the subtle spectral information of HSI to successfully classify tree species. Satellite HSI sensor data, due to the small number of HSI sensors in operation, only a few HSI sensor data are currently available to classify tree species [41], of which the airborne HSI data are more useful and important for the classification of tree species[42,43,44,45,46,47,48,49].

Data acquired by LiDAR sensors measure the reflected energy from the target surface and record features of the reflected spectrum, such as amplitude, frequency, and phase [50] which allows for the extraction of tree structural parameters and vertical structural properties. LiDAR-derived height information allows for the classification of tree species [51,52,53,54,55,56], but TS classification accuracy is limited. LiDAR data are usually combined with optical remote sensing data [57,58,59,60,61,62], and it has been shown that the combination of LiDAR data and optical remote sensing data for TS classification can significantly improve the accuracy [63,64], because optical remote sensing data provide rich spectral, spatial, and textural information of tree species, and LiDAR data provide a vertical profile and structural information, which form a useful complement to each other and can more comprehensively describe the characteristics of tree species and achieve better classification results.

Unmanned Aerial Vehicles (UAV) remote sensing systems are flexible and can fly freely in unrestricted areas for data collection. They are also inexpensive, cloud-free, and can acquire data with high spatial, spectral, and temporal resolution [65,66]. UAV data can be used to accurately classify tree species and even individual trees [67,68,69,70,71,72,73,74,75,76], as different data provide different quantitative features for TS classification. UAV remote sensing systems are limited in the application of large-scale TS classification due to short flight endurance and unstable conditions such as high winds.

SAR is primarily used to classify forest types because forest information by SAR relates mainly to canopy structure and water content [77].

Ancillary data are mainly topographic and meteorological information obtained, such as elevation, slope, slope direction, temperature, and precipitation [78,79,80].

2.2. Literature Trends in Remote Sensing Data

Multimodal remote sensing data is currently considered mainstream due to its availability at low cost, and the emergence of literature on tree classification. In this paper, we statistically analyze the literature on tree species classification from a data-driven perspective.

Detailed statistical analyses were performed on the 300 papers. The number of studies focusing on TS classification has constantly increased over the past 25 years (in Figure 1), which indicates that TS classification is a hot topic in current research. This growth is closely linked to the development of remote sensing technology and its corresponding computer science and technology. Remote sensing technology has advanced significantly since 2010, particularly for UAV, hyperspectral data, and LiDAR data. Advances in computer science and technology have brought about the evolution of classifiers for TS classification. The employment of high-resolution spatial and spectral remote sensing data, along with sophisticated classification algorithms, has resulted in further facilitation of TS classification research.

Figure 1 shows that there is no advantage in tree classification between unimodal and multimodal remote sensing data. It is not possible to conclude that multimodal remote sensing data tree classification is the mainstream direction.

A total of 208 papers were analyzed on the use of unimodal remote sensing data for tree species classification over the last 25 years. Among them, four papers were published from 01/01/1998 to 01/01/2003, three of which used colored infrared aerial photographs and one used MSI. Additionally, a total of 92 papers were analyzed on the use of multimodal remote sensing data for tree species classification over the last 20 years. Multimodal data tree species classification literature before 2003 was not found. Figure 2 shows the number of papers per five-year interval for various modal data.

Focusing on unimodal remote sensing data in Figure 2a, the most used unimodal data is HIS. In the 02/01/2018 and 01/01/2023 time span, the main unimodal data used for tree species classification were HIS (total of 28 cases), and RGB (24 cases). Since 2018, deep learning algorithms have dominated digital image processing. These algorithms rely on RGB images as input data, and the proliferation of drones has made it easier to obtain small-scale RGB images of tree species. Consequently, a vast amount of tree species classification literature on CNN and RGB has emerged within this timeframe. In the 02/01/2013 and 01/01/2018 time span, the main unimodal data were LiDAR (total of 16 cases) and HSI (total of 11 cases). In the 02/01/2008 and 01/01/2013 time span, the main unimodal data were LiDAR (7 cases), HIS (4 cases), and VHR(5 cases). In the 02/01/2003 and 01/01/2008 time span, the main unimodal data were HSI (4 cases) and MSI (5 cases). So, the main unimodal data for tree species classification were HSI, LiDAR, RGB, and VHR.

Focusing on multimodal remote sensing data in Figure 2b, regardless of the time frame, the most used multimodal data is HIS & LiDAR. in the 02/01/2018 and 01/01/2023 time span, the main multimodal data used for tree species classification were HIS & LiDAR (total 26 cases), VHR & LiDAR (total 10 cases), and MSI & HSI (total 6 cases). In the 02/01/2013 and 01/01/2018 time span, the main multimodal data were HIS & LiDAR (total 9 cases). In the 02/01/2008 and 01/01/2013 time span, the main multimodal data were HIS & LiDAR (total 5 cases).

2.3. Methods for TS Classification

2.3.1. Classification Methods of Unimodal Remote Sensing Data

208 papers with unimodal data were statistically analyzed using the classification method that produced the best results (Figure 3). In addition to the main classifier in Figure 3, the remaining 30 papers used other classifiers such as logistic regression classifiers, fractal geometry and quantitative structure models, and so on.

Focusing on unimodal remote sensing data and its’ most efficient classifier in Figure 3, in the 02/01/2018 and 01/01/2023 time span, the main classifiers were CNN, SVM, and RF. In the 02/01/2013 and 01/01/2018 time span, the main classifiers were RF, and SVM, In the 02/01/2008 and 01/01/2013 time span, the main classifiers used for optimal classification results were LDA. In the 02/01/2003 and 01/01/2008 time span, the main classifiers used for optimal classification results were LDA and MLC. With the development of remote sensing and computer science and technology, the main classification methods of unimodal remote sensing were CNN, RF, and SVM.

2.3.2. Classification Methods of Multimodal Remote Sensing Data

92 papers with multimodal data were statistically analyzed using the classification method that produced the best results. The literature on tree species classification using multimodal data prior to the year 2003 was not retrieved. Two papers, one using HIS & LiDAR data and nearest neighbor rules classifier and another using MSI & SAR data and Bayes rule classifier were published from 02/01/2003 to 01/01/2008. The statistical analysis results of the multimodal data and the main classifiers from 02/01/2008 to 01/01/2023 are shown in Figure 4. In addition to the main classifier in Figure 4, the remaining 12 papers used other classifiers such as spectrum angle mapper classifiers, nearest neighbor classifiers, and so on.

Focusing on multimodal remote sensing data and its’ most efficient classifier in Figure 4, in the 02/01/2018 and 01/01/2023 time span, the main classifiers were CNN, SVM, and RF. In the 02/01/2008 and 01/01/2018 time span, the main classifiers were SVM and RF. In all, the most used classification method is SVM, followed by RF and CNN.

2.4. Literature Trends in TS Classification Methods

Figure 3 and Figure 4 present the trends of the TS classification methods. Currently, there are two main types of methods, traditional machine learning methods and classic deep learning-based methods. The former includes RF, SVM, LDA, and MLC, among others. The latter refers specifically to CNNs and excludes methods such as transformers. Figure 5 depicts the tree classification process of these two methods.

Scales and categories for TS classification were determined based on a study of the natural variability and tree species composition of the study area. Remote sensing data for TS classification were collected and acquired based on data accessibility. The data underwent preprocessing, including atmospheric and terrain correction, de-clouding, and image inpainting. The tree samples were then labeled based on the complete ground inventory data, as the tree species categories in the remotely sensed data were not easily identifiable through visual interpretation alone.

The classification process for tree species using traditional machine learning methods involves preprocessing, extracting and selecting features, classifying the species using classifiers, and evaluating the results. The process of classic deep learning-based methods involves preprocessing, and cutting the remote sensing image data with labels into a patch cube, dividing the patch cube into a training set and a test set, using the data from the training set to train the CNN, storing the parameters of the best classified CNN model, and finally testing the network with the data from the test set to obtain the final TS classification result. The main difference between these two methods is whether or not the feature extraction is automated.

In Figure 6a, the study areas are categorized by size. Red dots represent areas larger than 10,000 hectares, green rectangles represent areas between 1,000 and 10,000 hectares, blue triangles represent areas between 100 and 1,000 hectares, and magenta hexagrams represent areas smaller than 100 hectares. In Figure 6b, the size of study areas for traditional machine learning methods is represented by green bubbles and classic deep learning-based methods are represented by orange bubbles. Meanwhile, a larger bubble indicates a larger study area.

Figure 6 only statistically analyzes the literature that presents the area and latitude/longitude range of the study area. In cases where multiple papers cover the same study area, only one area is counted. Figure 6a indicates that tree classification studies are typically conducted on a larger scale. Additionally, Figure 6b demonstrates that traditional machine learning methods are utilized for tree classification in large study areas, while classic deep learning-based methods are employed for tree classification in small study areas. It is worth noting that there are larger orange bubbles in Figure 6b, the classic deep learning-based methods are also beginning to be used for tree classification in large study areas. A multi-sensor, multi-label dataset for tree species classification using CNN was created by Steve Ahlswede et al [81]. using aerial images, Sentinel-1 images, and Sentinel-2 images. The study area, which covers approximately 47,710 km², is located in the federal state of Lower Saxony, Germany. Xueliang Wang and Honge Ren [82] used HSI from the HJ-1A satellite and MSI from the Sentinel-2 satellite to classify tree species using CNN. The study was conducted in the Tahe Forestry Bureau, which is situated in the center of the Daxing’an Mountains in the northwest of China's Heilongjiang Province. The total area covered by the study was 14,420 km².

Fassnacht et al. [9] and Ruiliang Pu [31] provide a detailed overview of traditional machine learning methods for TS classification. This thesis only covers classic deep learning-based methods for TS classification.

3. Literature Review on Classic deep Learning-Based Methods

In 2015, the article "Deep Learning" was published in Nature, LeCun Y and others predicted that the future application of deep learning in images, video, audio, and other aspects will break through [83], and deep learning algorithms based on big data platform is the direction of the future development of artificial intelligence, and also the mainstream of modern artificial intelligence research. Image recognition is one of the important research contents of deep learning, and research on classic deep learning (convolutional neural network, CNN) applied to TS classification in remote sensing images will be more and more in 2018 and beyond.

The classic deep learning structure consists of a convolutional layer, a pooling layer, and a fully connected layer, and the activation function (commonly used as Relu) is typically after the convolutional layer. The basic structure is shown in Figure 7, the convolutional layer and the pooling layer are used to extract TS features, and the fully connected layer is used to classify TS. The whole classification process is an automatic process, which is an end-to-end process, only the corresponding data of the input layer needs to be given, and features are automatically extracted from the network. It is more time-consuming and labor-intensive to create the samples (labels).

A total of 71 research papers were filtered. These papers focused on classic deep learning- based methods for tree species classification. Seventy papers were published between 02/01/2018 and 01/01/2023, highlighting the rapid development of deep learning-based methods for TS classification, which is a relatively new field.

3.1. Patch Size

After preprocessing the remote sensing data, the images must be divided into patches of size m*n. These patches will form the CNN dataset, also known as the patch cube. The size of the patches is the size of the CNN input image and is crucial for the classification object, such as individual TS, dominant TS, and the design of the classification CNN network structure. Out of all the literature, 62 papers provided a description of patch size, while 9 papers did not mention the size of the input image (patch size) for the CNN. Table 1 shows some patch sizes used for various data.

In Table 1, The bold patch size indicates a higher frequency of occurrence in the literature, and “3ⅹ3~15ⅹ15” means that the patch size was tested from 3ⅹ3 to 15ⅹ15 in Step 2.

After analyzing and summarizing the literature on patch size, we have concluded that there are five ways to introduce patch size. (1) The patch size is the same as the input image size of the CNN model itself in some papers, such as 224ⅹ224, and 227ⅹ227. (2) The optimal patch size for classification results is given in some papers, while other patch sizes used for comparison are not mentioned. (3) some papers used multiple patch sizes without final analytical interpretation. (4) In three research papers, a variety of patch sizes have been tried and some analytical interpretations have been made. Long Chen et al. [47] used prototypical networks and HSI data to classify TS. They experimented with 3ⅹ3~31ⅹ31 patch sizes and found that under the same conditions, a patch size of 17ⅹ17 resulted in significantly improved classification accuracy. This suggests that the spatial and channel feature extraction of prototypical networks for this patch size meets the requirements for high-precision classification. As patch size increases, more noise may be introduced, potentially causing fluctuations in classification accuracy. Janne Mäyrä et al.[58] utilized a 3D CNN network and HSI data to classify individual TS. The patch sizes used were 9ⅹ9, 13ⅹ13, 17ⅹ17, and 21ⅹ21 (square patches with diameters of 4, 6, 8, and 10 m, respectively). The results showed that the 9ⅹ9 patch size had the highest producer accuracy of 0.84. The classification accuracy was not significantly affected by the different patch sizes throughout the experiment. The paper demonstrates that larger image patches contain multiple trees of different species. Ying Sun et al. [84] utilized RGB to classify individual TS, and demonstrated the best performance for TS classification. RGB & CHM and RGB & VHR were also utilized but did not perform as well. The patch sizes used were 32ⅹ32, 48ⅹ48, and 64ⅹ64, with ResNet50 performing best in the 64ⅹ64 patch size. The study concluded that larger patch sizes are more effective in deep learning-based methods, while smaller patch sizes lead to higher overall accuracy in traditional machine learning methods. When using traditional machine learning methods, larger patch sizes may mix other information and influence the features of the tree species if the mean feature value was used. (5) Six research papers provided information on patch size and its corresponding spatial resolution (Table 2).

3.2. Reference Data

Classic deep learning-based methods belong to the supervised modeling approach. These datasets require remote sensing images and reference data, also known as annotations, samples, or labels. The most commonly used method for acquiring reference data was through field plot surveys (79%). A small percentage of the literature used forest management inventory data (7%), while 6% relied on visual interpretation. The remaining studies used public datasets with their own data labels.

3.3. TS Classification Scales

Determining the appropriate scale for TS classification depends on the needs of the application. The scale can range from dominant tree species to individual trees, and it is important to consider the required data resolution and labeling process. Based on classic deep learning-based methods, 51% of the research papers achieved the individual tree classification, 25% were focused on tree species classification, 19% on the dominant TS of main TS classification, and the remaining papers used public datasets. The classification of individual trees relies mainly on LiDAR and RGB data. In most cases, the crowns of individual trees are segmented and used for classification. In some literature, each image contains only one tree, and therefore, individual tree classification is performed directly.

3.4. CNN Architectures and Application

3.4.1. CNN from the Functional Perspective

Depending on the specific use, classic deep learning-based methods can be categorized into four main groups. The main function, representative networks, labeling structure, resulting output, and usage of these four groups in TS classification are shown in Table 3.

CNNs can be classified into four categories: classic CNN, object detection, semantic segmentation, and instance segmentation based on their functions. Classic CNN, object detection, and instance segmentation have the function of classification, so classic CNN, object detection, and instance segmentation can be used to classify TS, while object detection and instance segmentation, which have additional functions such as localization, require more complex and time-consuming labeling. Before TS classification, semantic segmentation is performed to obtain the tree’s canopy.

The statistical analysis results of the 71 papers categorized by the function of CNNs are presented in Figure 8.

Figure 8 shows that ResNet is the most commonly used classic CNN network for TS classification. In TS classification, the input image size to the network is relatively small, which limits the use of some classic networks. Self-designed shallower CNN networks are more common due to the feature extraction problems that arise in the convolutional and pooling layers when using classic CNN networks. The PointNet network is often preferred when segmenting individual trees for TS classification using LiDAR point cloud data (Figure 8). The infrequent use of object detection and instance segmentation in TS classification is shown in Figure 8, labels could be complex, the bounding box is that they often do not explicitly define crown/canopy boundaries, and crown/canopy is not rectangular. Natural canopies often have smooth transitions or overlapping crowns, which limits the object detection and instance segmentation that can be applied in TS classification. But for certain tree species with non-overlapping crowns, it's important to try object detection and instance segmentation so that the tree transformation can be easily tracked afterward.

3.4.2. CNN from the Functional Perspective

Depending on the input remote sensing data, classic deep learning-based methods can be categorized into 1DCNN, 2DCNN, and 3DCNN, where the number refers to the dimensions of the convolutional kernel. The multi-temporal remote sensing data use Recurrent Neural Network (RNN) or 1DCNN. Other input data can use 1DCNN, 2DCNN, or 3DCNN. Among them, 2DCNN is the most commonly used in the TS classification (75%). 3DCNN was used most often for LiDAR and HSI data (19%), and the model was self-designed.

Yanbiao Xi et al. [110] utilized 1DCNN in conjunction with spectral and crown texture features of HIS to classify 7 TS, resulting in an overall accuracy of 85.04%. The features were used to generate a vector as the input layer for the 1DCNN network, which comprised two convolutional layers, one max pooling layer, and fully connected layers. Research papers on 3DCNN tree classification have been concentrated between 2019 and 2021. Haiyan Guan et al. [111] and Maohua Liu et al. [56] utilized a self-designed 3DCNN and LiDAR data to achieve TS classification with an overall accuracy of 96.4% and 92.5%, respectively. Somayeh Nezami et al. [112] achieved TS classification using a self-designed 3DCNN and RGB & HIS data, with a producer accuracy of 99.6%. Bin Zhang et al. [113] and Janne Mäyrä et al. [58] employed a self-designed 3DCNN and HSI data to classify the TS, achieving classification accuracy of 93.14% and 87%, respectively. The two studies utilized the canopy height model derived from LiDAR data to match ground reference data to aerial imagery.

3.4.3. CNN from the Functional Perspective

Multimodal data fusion is generally categorized into three types: input-stack fusion, feature-stack fusion, and decision-level fusion. In the literature on TS classification based on classic deep learning-based methods, only two types of fusion were found: input-stack fusion (6 papers) and feature-stack fusion (3 papers).

Bingjie Liu et al. [114] utilized PointMLP to extract features from LiDAR point cloud data and 2DCNN to extract features from UAV RGB images for tree species classification with 98.52% accuracy. Xueliang Wang et al. [82] proposed the double-branch multi-source fusion (DBMF) method. One branch utilized the CNN network to abstract the spatial features for the MSI, while the other branch employed the bidirectional long short-term memory (Bi-LSTM) to abstract the spectral features for the HSI. The resulting features were concatenated to classify TS, achieving an overall accuracy of 92%. Ira Harmon et al.[115], Sean Hartling et al.[116], and Hui Li et al.[117] all utilized input-stack fusion used HCM & DEM & RGB/HSI data, VHR & LiDAR data, and MSI & LiDAR data to classify TS, respectively.

3.5. CNN Architectures and Application

Figure 9 shows the statistical results of the CNN operational framework for TS classification. There are lots of frameworks for deep learning. Based on the development of deep learning in artificial intelligence, the most popular framework in the past few years was Google's TensorFlow, but in the past two years, the most popular framework has been Facebook's PyTorch. Our statistical results from the literature are consistent with the development of deep learning artificial intelligence. Currently, the use of the PyTorch framework is gradually increasing and replacing TensorFlow.

3.6. CNN Model Assessment in TS Classification

The TS classification results were evaluated based on the overall accuracy (OA), precision, recall, and F1 score, kappa [58,86,91,105,109,118,119,120], these metrics are defined as follows:

$$precision=UA=\frac{TP}{TP+FP}$$

(1)

$$Recall=PA=\frac{TP}{TP+FN}$$

(2)

$$F1=2\ast \frac{precision\ast Recall}{precision+Recall}=2\ast \frac{UA.PA}{UA+PA}$$

(3)

$$OA=\frac{TP+TN}{TP+TN+FP+FN}=\frac{TP+TN}{N}$$

(4)

$$kappa=\frac{OA-P_e}{1-P_e}$$

(5)

The variables used in this study include TP (true positives), FP (false positives), FN (false negatives), TN (true negatives), $P_e$ (expected proportion of agreement), and N (total sample size). TP represents the number of positive samples correctly predicted by the CNN model, while FP represents the number of negative samples incorrectly predicted as positive. FN represents the number of positive samples incorrectly predicted as negative, while TN represents the number of negative samples correctly predicted by the model. $P_e$ is calculated by dividing the sum of the product of the actual sample size and the predicted sample size by the square of the total number of samples. The user's accuracy (UA), producer's accuracy (PA), and Dice similarity coefficient are equivalent to precision, Recall, and F1, respectively. UA measures the proportion of relevant positive predictions, while PA measures the proportion of correctly classified positive results. F1 is the weighted average of UA and PA, with a best value of 1 and a worst value of 0. The kappa coefficient and OA measure the overall consistency between the identification result and the reference data. The OA value ranges from 0 to 1, while the kappa coefficient value ranges from -1 to 1 and is typically greater than 0. Equations (1)-(5) compute their values based on the confusion matrix, ranging from 0 to 1. A value of 1 indicates the highest similarity between the predicted TS and reference data, while 0 indicates no similarity.

In the sematic and instance segmentation, the intersection over union (IoU, also known as Jaccard Index) is used to evaluate the performance of the different trained models on independent testing datasets [109].

$$IoU=\frac{area(B_{actual}\cap B_{predicted})}{area(B_{actual}\cup B_{predicted})}\ast 100\%$$

(6)

The crown polygons from the test set ($B_{actual}$) and the predicted crown polygons from the segment CNN algorithm ($B_{predicted}$) are compared using the intersection and union operations to determine their common and combined areas, respectively.

4. Discussion of Current Limitations

There are still many factors that limit the accuracy and precision of TS classification, such as spatial resolution, temporal resolution, spectral resolution, phenological transformation information, forest environments, shadows, cloud shading, small canopies, spectral shapes of different canopies, and canopy overlap, and so on [121]. At the same time, the current TS classification used in different study areas has some limitations because the algorithms themselves have prerequisites for using them, and the data may not always be able to meet the prerequisites of the algorithms perfectly. Thus, TS classification based on remote sensing is full of challenges, and there are many aspects worth our efforts in the future.

4.1. Data Fusion

Spatially sharpened data fusion method

Pan-sharpening is a pixel-level fusion that can be applied to both single-sensor data and multi-sensor images. Single-sensor fusion produces a high-resolution multispectral image by fusing a pan-band and a low-resolution multispectral image. Such a sharpened image has a nominal pan high resolution but its MS property may be slightly different from the original MS property. Compared to the direct use of low-resolution MS images, sharpened images improve the quality of individual canopy object segmentation for optimal canopy object segmentation. Spatial sharpening of multi-sensor images is performed using different optical sensor data, one high-resolution and one low-resolution multispectral data. The low-resolution multispectral band image is then resampled to a higher resolution to make the images from both sensors the same size, and finally, the spatially sharpened data are obtained by using a spatial sharpening algorithm. No studies have been found in the literature reporting the direct use of spatial sharpened methods to improve TS classification, but spatial sharpened improves the spatial resolution and maintains the spectral properties of MS [122], and spatial sharpened fusion methods should be useful in practice for TS classification[31].

Feature-level data fusion method

Feature-level fusion is the fusion of features extracted from different sensors, which is a simple overlapping of multi-source features to increase the number of features for TS classification, not true fusion.

Spatiotemporal Data Fusion method

At present, the spatiotemporal fusion algorithm of remote sensing data has matured. The spatiotemporal fused time series data have a high temporal and spatial resolution, which can respond well to the information of phenological changes of tree species. However, so far, spatiotemporal fusion has not been directly applied to the research of TS classification.

Multi-source remote sensing data fusion has developed rapidly in recent years. The fusion of homogeneous remote sensing data has achieved the fusion of multi-sensor data in the three dimensions of time, space, and spectra, and obtained high-quality data with multi-temporal phase, high spectral, and high spatial and temporal resolution. The fusion of heterogeneous remote sensing data is mainly the fusion of optical remote sensing data and active remote sensing data, of which the fusion of SAR images and optical remote sensing data is more sophisticated. In the future, to effectively improve the accuracy and precision of TS classification, the real fusion algorithm of multimode remote sensing data will be applied to TS classification, or the existing multimode remote sensing data fusion algorithm will be improved to study TS classification.

4.2. Phenology Information

The multi-temporal remote sensing data have great significance in improving the classification accuracy of tree species [123,124,125,126,127], as current research has shown that time series images can correspond to the phonological and seasonal characteristics of tree species, and the characteristics change differently for different tree species throughout the year [128]. However, the seasonal division of the series is currently not uniform; some literature divides the multi-temporal data into 4 seasons (Spring, Summer, Fall, and Winter), some in 2 seasons (Wet/Dry, Dry/Rainy, Growing/Non-Growing, and so on), and some in multiple seasons (Early Spring, Late Spring, and Midsummer). How to extract the valuable features from the multi-temporal data and how to use the features has not yet been completed, and it has not yet been concluded which combination of two-season, one-season, or multi-season time series images is most effective for the classification accuracy of tree species. For example, Li et al [123] used two seasons of WV2 (September 14, 2012) and WV3 (October 18, 2014) data to classify five tree species in Beijing, China, and Karlson et al [129] used two seasons of WV2 images to classify five dominant species in West Africa and found that TS classification using two seasons of time series data was better than that using single or multiple seasons. Ferreira et al [130] investigated whether using wet and dry seasons separately or in combination improved tree classification accuracy, and the study concluded that the two-season combination did not have higher tree classification accuracy than the single-season. Pu et al [124] tested combinations of two, three, four and five-season multi-temporal images for a study on TS classification ability, and they found that combining more than two combinations of seasonal images did not produce better TS classification results than combinations of two seasonal images (dry-wet season images).

Most current studies select the best seasonal time-series images and do not fully utilize images from all seasons when classifying tree species. In the literature, R. Pu and S. Landry [131] proposed a seasonal trajectory difference index, which integrates the possible contributions of all seasons as a feature for classifying tree species and can help improve the accuracy and precision of classifying tree species. How to take full advantage of the phonological and seasonal information, and combine the algorithms in the fields of time series processing and digital signal processing to create a characteristic or method with an explicit physical meaning of phenological variation, used to improve the accuracy of TS classification.

4.3. Data Label

The remote sensing data label is directly related to the usefulness of the final TS classification product. Currently, data is labeled through plot surveys, and it is generally difficult for someone familiar enough with the study area to accurately identify all the tree species through remote sensing data, so visual interpretation is essentially not used. For large-scale TS classification, the plot survey is very difficult and takes several years or more, resulting in a temporal mismatch between remote sensing data and labels. In some primary forests or parts of forests with complex terrain and difficult transportation, it is impossible to conduct field surveys, so it is even more difficult to solve the labeling problem.

Weakly supervised learning labels at the image level or sparsely at the pixel level, reducing the cost of manual labeling. Even when results from other data studies are available and ground truth data is scarce, the method can help solve the labeling problem [132,133]. Semi-supervised learning lies between supervised and weakly supervised learning. It has been shown to train robust CNNs with a small number of high-quality labels [134]. In the future, weakly supervised learning or semi-supervised learning can be used to support data labeling when large data sets are constructed.

4.4. Patch Size

When utilizing remote sensing for tree species classification, it is important to consider the optimal ground sampling density and spatial unit. Specifically, it is necessary to determine the spatial unit for obtaining tree species information and the optimal ground sampling density for deriving such information using a given sensor [9]. The size detailed statistics and analysis reveal that out of 71 papers on classic deep learning-based methods of tree species classification, only three provide rough explanations of different patch sizes, and only six offer information on patch size and its corresponding spatial resolution.

Fromm et al.[135] have shown that the accuracy of detection of tree seedlings can vary by up to 20% based on the resolution of the UAV image (0.3-6.3 cm). Similarly, when the pixel size was reduced from 40 cm to 60 cm, Neupane et al.[136] found a 17% decrease in detection accuracy for banana trees in plantations. So, when selecting the patch size for the TS classification, it is important to consider the spatial unit and optimal ground sampling density. This could contribute to the criteria for selecting the patch size. This is a very interesting problem, and it's worth trying to solve in the future.

4.5. CNN Model Optimization Approaches

Due to the increasing performance of computer hardware, CNN models now have billions of parameters. This gives them incredible fitting ability but also makes them highly susceptible to overfitting for a particular dataset. To improve the generalization ability of CNN models and alleviate the overfitting problem, the following strategies can be employed: To improve the dataset, consider using data augmentation procedures. To improve the CNN model, add regularization techniques to the weights. To improve the training process, consider stopping the model early. To improve the model structure, add the Dropout layer and the normalization layer.

When a CNN model experiences overfitting, the first network structure typically considered for use is Dropout. Dropout reduces the coupling between nodes during training by replacing some nodes with masks to achieve regularity. There are also other improved versions of Dropout, such as Spatial Dropout, DropBlock, and Max-pooling Dropout. Normalization is a rapidly growing set of algorithms in deep learning. Batch normalization (BN) is a classical normalization algorithm that is frequently used to mitigate overfitting in deep learning. It normalizes features from the same channel of different samples and effectively solves the exploding gradient problem, making the CNN network model more stable.

5. Conclusions

In this study, a total of 300 publications related to the study of classifying and mapping TS using images from different remote sensing sensors were analyzed. In this regard, a review of the unimodal and multimodal remote sensor data and the classic deep learning-based methods for TS classification was carried out. After carrying out the review and summary, some conclusions with remarks and recommendations will be summarized in the following paragraphs.

• From the number of publications, tree species classification has become a hot topic in current research. From the unimodal and multimodal remote sensor data utilization, it is not possible to conclude that multimodal remote sensing data tree classification is the mainstream direction. The main unimodal data for TS classification were HSI, LiDAR, RGB, VHR, the most used multimodal data is HIS & LiDAR.

• According to the literature analysis of TS classification methods, the most commonly used classifiers for remote sensing data, whether unimodal or multimodal, are CNN, RF, and SVM. Therefore, this article summarizes the process of remote sensing TS classification and condenses the two major current TS classification methods: traditional machine learning methods and classic deep learning-based methods.
• traditional machine learning methods are utilized for tree classification in large study areas, while classic deep learning-based methods are employed for tree classification in small study areas. The classic deep learning-based methods are beginning to be used for tree classification in large study areas.
• The classic deep learning-based methods for TS classification are reviewed in detail in terms of patch size, reference data, TS classification scales, CNN architectures and applications, CNN operational framework, and CNN model assessment.
• five limitations discussed and suggested to overcome in the future. (a) data fusion. spatial-temporal fusion algorithm and real fusion algorithm of multimodal remote sensing data should apply to TS classification or the existing multimodal remote sensing data fusion algorithm should be improved to study TS classification. (b) phenology information. A feature or method was created with an explicit physical meaning of phenological variation used to improve the accuracy of TS classification (c)data label. Label production is very labor-intensive, and field surveys for TS classification labeling are time-consuming and laborious. It is recommended to combine field surveys with weakly supervised and semi-supervised learning for labeling. (d)patch size. When utilizing remote sensing data for tree species classification, it is important to consider the optimal ground sampling density and spatial unit. Specifically, it is necessary to determine the spatial unit for obtaining tree species information and the optimal ground sampling density for deriving such information using a given sensor The patch size has not been studied enough, the patch size may depend on the spatial resolution of the classification target, the distribution and size of the forest stand area, or other factors, which is an interesting problem to study. (e)CNN model optimization. To improve the generalization ability of CNN models and alleviate the overfitting problem, some strategies were given.
• Vision transformers (ViTs) have been trending in image classification tasks due to their promising performance when compared to convolutional neural networks (CNNs). Multimodal deep learning can fuse different modalities of remote sensing data to achieve richer information representation and more accurate TS classification. We believe that the Transformer & multimodal-based methods will be applied to TS classification shortly. The methods will comprehensively improve the effect of TS classification and create a new situation of TS classification by mining and fusing the data information of each modality.