LLFWCNN: A Light-Weighted Low Frequency Wave-Based Network for Hyperspectral Image Classification

1. Introduction

Hyperspectral image (HSI) classification is a common but very important task in hyperspectral image processing. It is widely used in vegetation cover monitoring, ground object classification, mineral exploration, military reconnaissance, atmospheric environment research, etc. [1,2,3,4,5,6]. Hyperspectral classification (HSIC) is to identify each pixel on the hyperspectrum as the correct category, and the spectral and spatial information of each pixel provides effective information for hyperspectral classification. For example, Hecker etc. used spectral angle for pixel matching to evaluate the effect of spectral information in HSIC [7]. Demir and Camps-Valls proposed classification model using the of hyperspectrum on statistics and kernel method [8,9]. while another main approach combined the spectral information and spatial information of hyperspectrum for research [10,11,12,13,14]. In recent years, Convolutional Neural networks (CNN) have made remarkable achievements in the field of image processing. Researchers have introduced convolutional neural networks into hyperspectral classification. Network models such as 1D-CNN [15,16], 2D-CNN [17,18,19,20], 3D-CNN [21,22,23] and hybrid CNN [24,25,26] have been proposed successively, all of which have achieved good classification results.

In recent years, many researchers have introduced wavelet decomposition into hyperspectral classification tasks and proposed many novel classification methods [27,28,29,30]. For example, in 3DGPC-HDM framework [28], a three-dimensional Gabor wavelet (Figure 1) was introduced to extract the joint spectral-spatial features of HSI, and a phase coding and Hamming-distance matching framework based on the three-dimensional Gabor wavelet was proposed. Cao etc. [30] proposed an enhanced 3DDWT (E-3DDWT, Figure 2) method which discarded partial of the sub cubes after wavelet decomposition to extract features and alleviate the noise simultaneously. SpectralNet [31] carried out four-layer wavelet decomposition of HSI blocks, extracted further features by CNN and then classified (Figure 3). It had achieved very good results on the three classical HSI datasets, however, the number of parameters in the model was relatively large, about 6-7M.

However, there are two problems for most of the above wavelet-based models. One is the heavy training parameters and the other is that there is no distinction between the classification effectiveness of low frequency and high frequency features after wavelet decomposition, both low and high frequency components are used in the classification model.

In this paper, a light-weighted low frequency wavelet-based convolutional neural network model LLFWCNN is proposed. There are two main contributions of this paper:

• The proposed LLFWCNN using a combination of wavelet decomposition, feature reshape and a well-designed CNN to obtains comparable or even better results and fewer parameters than other state-of-the-art models, especially in small samples.
• It proves that the influence of low frequency and high frequency after wavelet decomposition on HSIC is different. The low frequency features include much more advantageous information for HSIC than high frequency features. Low frequency component is enough.

2. Proposed Model

2.1. Overall Architecture

The overall architecture of LLFWCNN is shown as Figure 4.

The LLFWCNN contains the following four stages: dimension reduction, wavelet decomposition, feature extraction and classification, the first three of which are described in detail below.

2.2. Dimension Reduction and Wavelet Decomposition

2.2.1. Dimension Reduction

In order to effectively reduce the computational complexity of HSI data, we introduce Principal Component Analysis (PCA) method as preprocessing to lower the dimensionality of the original HSI. PCA is one of the most widely used data dimensionality reduction algorithms, which preserve the most important features of the original high dimensional data and map the high dimensional data to the low dimensional space.

The original input HSI is $X_{image}\in R^{H\times W\times B}$，where H, W and B represent the length, width and spectral dimension of HSI respectively. After PCA dimension reduction, the input X_image is transformed to $X_{image}^{\text{'}}\in R^{H\times W\times p}$ (p<<B).

2.2.2. Wavelet Decomposition

Wavelet decomposition is carried out on the resized HSI. Wavelet decomposition is a classical method for signal processing and analysis. Figure 5 illustrates the process of how two-dimensional discrete wavelet decomposition works on images.

The image is decomposed into four sub-bands by two-dimensional wavelet decomposition, in which the LL sub-band is the low frequency component, and the HL, LH and HH sub-bands are the high frequency components in the horizontal, vertical and diagonal directions, respectively. The low frequency component with a low resolution is an approximation to the original image and describes the main part of the image. While the high frequency component embodies the details of the image.

The procedure after dimension reduction is divided into three steps as follow:

1) Firstly, extract the hyperspectral input cube $X_{cube}^0\in R^{S\times S\times p}$ pixel by pixel from the pre-processed input HSI $X_{image}^{\text{'}}$, where S represents the cube size.

2) Secondly, a two-dimensional wavelet decomposition is performed on a single image $I_m\in R^{S\times S\times 1}$（m=1, 2, …, p）extracted from $X_{cube}^0$ along the spectral dimension, forming the result denoted as $I_m^{\text{'}}$ （m=1, 2, …, p）with the size $S/2\times S/2\times 4$. p images from p bands lead to p features $I_m^{\text{'}}$（m=1, 2, …, p）.

3) Thirdly, the p features $I_m^{\text{'}}$ （m=1, 2, …, p）are stacked in the order of "low frequency + high frequency", and then obtain the wavelet decomposition result $X_{cube}^1\in R^{S/2\times S/2\times 4p}$ (Figure 6). The low frequency part of $X_{cube}^1$ is denoted as $X_{low}^1$, and the high frequency part is denoted as $X_{high}^1$.

4) The low frequency part $X_{low}^1$ is selected and decomposed until the Lth layer in the above way to get the result $X_{cube}^L\in R^{S/2^L\times S/2^L\times 4p}$of the L-layer wavelet decomposition. The complete procedure is shown in Figure 7.

2.2.3. Feature Reshape

Feature reshape is a very important step in the LLFWCNN. Except the results obtained in the last wavelet decomposition layer, the results of each layer were reshaped to the size of the features of the last layer.

The commonly used reshape mode for a single image is “reshape by row”. For example, if we want to reshape a 4×4×1 feature into size of 2×2×4, the reshape mode is shown in Figure 8(a). In this mode, each row vector of size 1×4×1 is rearranged to a vector of size 1×1×4 (left part of Figure 8(a)). And then the four new vectors were stacked in the order (right part of Figure 8(a)) to obtain the feature cube of size 2×2×4.

However, this mode is not suitable when the feature is a result of wavelet decomposition. Because there is inner relationship among the pixels in a square area not in a row. Therefore, another reshape mode was used. For the above example, pixels in a specific 2×2 square area are considered to be intrinsically related. Figure 8(b) shows how the reshaping is executed.

After the features of multi-channels were reshaped, they were stacked in the way as in Figure 9. Hence, in this stage, each $X_{cube}^i\in R^{S/2^i\times S/2^i\times 4p}$ (i≠L) was finally reshaped to $X_{cube}^{i\text{'}}\in R^{S/2^L\times S/2^L\times 4^{L-i}\times p}$, that helps to reduce the parameters in the next CNN stage.

2.3. Convolutional Neural Network RM-CNN

Following the wavelet decomposition, there is a specially designed convolutional neural network RM-CNN. The detailed structure is shown in Figure 10.

RM-CNN consists of two different types of modules: Rblock and Mblock. There is one Rblock at the front and end of the network each. It mainly realizes feature extraction and pooling. The spatial size of the output feature is reduced to half of the original input feature when passing through Rblock. Compared with Rblock, Mblock does not have a pooling layer, but adopts skip connection. It brings two benefits, one of which is to maintain the spatial size of the output feature and the other is to keep the original input feature information from being lost by adding input information. K is a hyper-parameter denoting the number of Mblock. The inner structures of Rblock and Mblock are shown in Figure 11(a) and (b).

In Figure 11(a), n is the number of convolutional kernels in convolutional layer. It can be the same or not the same as b in Rblock, while in Mblock it must be the same as b.

Through the combined use of Reshape and RM-CNN, we achieved the goal of reducing the number of parameters. Because the spatial size of the last features after wavelet decomposition is relatively small, and was designed to be 4×4 or 6×6. After spatial dimension reduction twice by RMCNN, the output size could be restricted to 1×1 or 2×2, so that the number of parameters in the fully connected layer would be effectively reduced. For example, there is a feature cube of size 16×16×12. The number of convolutional kernels in Rblock and Mblock are 32, and kernel size is 3×3.

The parameter differences between reshaping and non-reshaping are listed in Table 1, and the parts with the same parameters are omitted.

It is clear that the number of parameters after reshaping is only one-tenth of that without reshaping.

3. Experiment

3.1. Datasets

We did experiments on three classical experimental datasets: Indian Pines, Pavia University and Salinas. Indian Pines and Salinas were acquired by the Airborne Visible Infrared Imaging Spectrometer (AVIRIS) sensor over the Indian Pines test site in North-western Indiana and Salinas Valley. And Pavia University was captured by the Reflective Optics System Imaging Spectrometer (ROSIS-03). AVIRIS provides 224 contiguous spectral bands, covering wavelengths from 0.4 to 2.5μm and with a spatial resolution of 20m/pixel, while ROSIS-03 delivers 115 bands with a spectral coverage ranging from 0.43 to 0.86μm and with a spatial resolution of 1.3m/pixel. And we used a corrected version of them by removing some water absorption bands and a blank strip. Indian Pines consists 145×145 pixels, Salinas consists 512×217 pixels and Paiva University consists of 610×340 pixels. Both Indian Pines and Salinas consist of 16 ground-truth classes while Pavia University contains 9 different classes.

3.2. Settings

3.2.1. Hyper-parameter Settings

There are six hyper-parameters of LLFWCNN, which are input spatial size, PCA value, depth of wavelet decomposition, number of Rblock and Mblock and nodes in fully connected layer. Different configurations for different datasets are listed in Table 2.

3.2.2. Training parameter Settings

Parameters related to training are shown in Table 3. The stochastic gradient descent (SGD) method was chosen for optimization. Batch size and learning rate were the same for all datasets that were 8 and 0.02, but the epoch was different. Training proportion was set to 10% and 30% to illustrate performance of all the six compared models. Table 4, Table 5 and Table 6 give the detailed split of training and testing on the three datasets.

The running environment is: CPU i7-10700, GPU Nvidia GeForce RTX 2070, 8G video memory.

3.3. Results and Analysis of the Experiment

Experiments on each dataset were carried out 5 to 10 times to ensure the accuracy and reliability. Three commonly used measurement indicators: Overall Accuracy (OA), Average Accuracy (AA) and Kappa Coefficient (Kappa) were adopted. And their corresponding data listed in all the following tables are multiplied by 100.

3.3.1. Comparisons of classification results

We compared the classification results of LLFWCNN and SpectralNet [31], 2D CNN [32], 3D CNN [32], M3D CNN [33], FuSENET [34].

Table 7 and Table 8 list the results under proportion 1 and 2. Optimal results are shown in bold, and sub-optimal results are shown in italics.

From the data of Table 2 and Table 3 we found that both LLFWCNN and SpectralNet can perform well, outperforming other models. Then, we further reduced the proportion of training samples to 5% and 1% to evaluate the classification effects of the two models under even smaller samples. The results are given in Table 9.

3.3.2. Comparison of the parameters

We compared the number of parameters of LLFWCNN with that of SpectralNet. And the results are shown in Table 10.

Based on the data in Table 7, Table 8, Table 9 and Table 10, it can be seen that with only about 1/40 of the parameters of SpectralNet, LLFWCNN obtains the same classification results as SpectralNet, and even better classification results under small sample (training sample proportion 5%, 1%).

3.3.3. Ablation experiment

Ablation experiments were done to evaluate the performance of the low frequency component and high frequency component. In the next tables, $X_{low}^i$ represented using only low frequency, $X_{high}^i$ represented using only high frequency and $X_{low}^i{+X}_{high}^i$ represented using both. The training sample proportions were set to 10%, 5%, and 1% for all the three datasets. The classification indicators of the three datasets were shown in Table 11, Table 12, Table 13, Table 14, Table 15, Table 16, Table 17, Table 18 and Table 19. And Figure 11(a)-(i) show the variation of the overall accuracy at the ratio of 10%, 5% and 1%, respectively.

Figure 11. Variation of the Overall Accuracy on the three datasets.

Figure 11. Variation of the Overall Accuracy on the three datasets.

As can be seen from the experiment results in Table 11, Table 12, Table 13, Table 14, Table 15, Table 16, Table 17, Table 18 and Table 19 and Figure 11, the effects of low frequency and high frequency components after wavelet decomposition on HSIC are indeed significantly different.

1) The classification result of low frequency features is better or much better than that of high frequency features;

2) For datasets itself with large sample such as Pavia University and Salinas, the smaller the proportion of training samples, the better the effect of using low frequency component $X_{low}^i$ for classification. While for dataset with a small sample e.g. Indian Pines, the advantage of using low frequency component $X_{low}^i$ for classification is more obvious;

3) For the different components from the same layer after wavelet decomposition, the classification effect of the mixed features of low frequency and high frequency is between that of the low frequency and high frequency;

4) In most cases, the higher the decomposition level, the better the classification results of the corresponding low frequency or high frequency features, especially when the sample size is small (such as 1%). The best classification results of the three datasets are obtained by the low frequency component at the highest wavelet decomposition level.

3.3.4. Parameters of different frequency

At the same time, a comparison of the number of parameters used in the classification for low frequency and high frequency of different layers is shown in Figure 12(a)-(c).

Two conclusions can be drawn from Figure 12:

1) When using the features of different layers to classify, the number of parameters producing in the model is different. The deeper the layer number, the smaller the number of parameters;

2) When low frequency component and high frequency component participate in classification, the number of parameters producing in the model is also different, which is smaller for low frequency than that of high frequency.

3.3.5. Influence of other hyper-parameters

In paragraph 3.2, we listed the settings for most of the hyper-parameters. In fact, during the experiments, the best value of the hyper-parameters was not easy to determine. Most of the time, we took all aspects of the factors in consideration, and finally determined a most appropriate value.

For example, considering the requirement of wavelet decomposition, the input spatial size was first set to 16, 32, 64 which were multiples of 2. And then some available values were also tried, e.g. 24, 48. At the same time, the bigger the input size, the bigger the depth of the wavelet decomposition. Otherwise, the input for RMCNN would be bigger and produce more parameters.

Take Indian Pines for example. The low frequency features after the last wavelet decomposition were feed to the model and training ratio was 1%. Figure 13 is a comprehensive illustration for all the influential factors to be taken into account of.

At the same level (L=2) of wavelet decomposition, the overall accuracy raised when it turned from 16 to 64. And the number of parameters and training time increased as well. The increase of OA was relatively small, but the increase of parameter number and training time were very large. A similar situation occurred when it came to a high level (L=3). Because of the small size of Indian Pines itself, the larger input such as 128×128 was not used.

We could see that the three optimal OA results were the results of the same input 64×64 when L was 2, 3, 4 respectively. When the number of parameters, the training time and the training ratio were all taken into account, setting the depth of wavelet decomposition be 4 was better.

For Pavia University and Salinas, the same experiments were carried out, and the final hyper-parameters were set as shown in Table 2.

4. Conclusion

In this paper, a LLFWCNN network model is proposed to classify HSI after two-dimensional wavelet decomposition. The following conclusions can be drawn from results of the experiments:

1. LLFWCNN uses fewer parameters to obtain a comparable classification effect compared with other excellent models, especially in the case of small samples.

2. Results of the experiment show that spatial information from the low frequency and high frequency after wavelet decomposition have different effects on HSIC. Spatial information of the low frequency contains more favorable information for classification, which is relatively more useful.

3. The classification results of the LLFWCNN model prove that the low frequency features after wavelet decomposition already have enough effective information. It can be considered to discard the high frequency components after wavelet decomposition or to employ different processing methods to improve the efficiency of the use of high frequency components in terms of the different effects of the low frequency and high frequency components on HSIC.

The conclusions above provide effective experimental support for the follow up study on how to make full use of wavelet decomposition in HSIC in the case of small samples.