Deep Learning for On-board Processing of Imaging Spectroscopy: A Survey

1. Introduction

Imaging spectroscopy in the visible to short-wave infrared (VSWIR) portion of the electromagnetic spectrum is a powerful Earth observation tool that evolved tremendously in the last 40 years (for a review see Rast and Painter [1]). A broad range of research fields and operational applications benefit from the unique capability of imaging spectroscopy sensors to accurately measure the spectral signature of Earth surface from remote sensing platforms, such as but not limited to monitoring of industrial activities, agriculture, ocean colour, as well as pre- and post-monitoring of natural hazards. Nowadays, several hyperspectral sensors are producing an almost continuous stream of data from airborne and spaceborne platforms i.e., AVIRIS-NG [2], EnMAP [3], PRISMA [4], EMITS [5] and future to-be-launched CHIME [6] and SBG [7]. Nearly all hyperspectral spaceborne sensors capture data with a bandwidth of $\approx 10$ nm and a spatial resolution of $\approx 30$ m. When combined with a relatively large swath (30km–150km), and repeated acquisition schemes, the produced data will need huge storage and computational power to be processed. Because of this and the increasing demand for rapid information and insights from Earth observation sensors, there is an urgent need for near real-time information extraction which is hardware friendly and can be embedded into airborne andor space-borne sensors. While multi-spectral sensors capture information in few spectral bands, HS sensors are capable of recording hundreds of spectral bands for each pixel. Therefore, an HS image can be considered as a multi-dimensional data cube which d(dimension)>150. Hence, the spectral signature [8], or fingerprint, of each pixel can be obtained. This signature can be used to extract information on the underlying surface and its properties in a quantitative way (e.g. quantitative retrieval of geop-physical properties) or for image classification (Figure 1).

For many applications i.e., Security, natural hazards, chemical leak detection, etc., is necessary to do the pixel-wise classification of Imaging spectroscopy. Pixel-wised classification is also known as image segmentation or semantic segmentation [10]. Hereafter throughout the document image segmentation and classification have been used interchangeably. It has been a recent trend to develop algorithms that can process data (near) real-time, and extract required information to prevent huge down-linking of data and further storage/processing costs [11]. For this goal, traditional machine learning techniques that require manual feature extraction are not a suitable candidate thus deep learning has found its place within the hyperspectral community [12,13]. Moreover, deep learning techniques can design features that are rarely possible by humans analyses [14]. Deep learning algorithms for on-board processing of HS data can be focused on data volume reduction [15,16], feature extraction [17], and target detection from raw data [18].

One should consider the limited memory and power supply on board as well as the quality of acquired data from the satellite to successfully deploy deep learning algorithms. The segmentation algorithms for HS imagery are often referred as supervised segmentation and using mainly spectral information that results in super-pixels [19] or homogeneous regions. In contrary, in computer vision and image processing community refer to both supervised and unsupervised methods and image classification normally is referred to assigning a label to every pixel in the whole imagery [18].

In early studies, Imaging spectroscopy segmentation was performed using a K-nearest neighbor classifier [20], support vector machines (SVMs) [21] and Gaussian un-mixing models [22]. Moreover, sparse signal representation methods have been used to classify noisy data with help of a learned dictionary [15]. These methods were extensively used before the emergence of deep learning techniques.

The objective of this paper is to evaluate various deep learning techniques in terms of network architecture, reliability, and the ability to handle noisy data. These factors play a crucial role in the implementation of deep learning for on-board applications. Additionally, the study will assess the capability of networks to be trained with limited training samples. The outcome of this analysis will inform the decision on which network architecture and configurations are optimal for onboard Imaging Spectroscopy segmentation.

2. Deep Learning for Imaging Spectroscopy Segmentation/Classification

We start with Convolutional Neural Networks (CNN) in different approaches (spectral, spatial, spectral–spatial). Other significant architectures we consider are Autoencoders, Deep Belief Networks, Generative Adversarial Networks, and Recurrent Neural networks. These architectures are flexible and adaptable to onboard Imaging spectroscopy processing as well. Discussion about challenges and new trends to handle them will be followed later in this section.

2.1. Spectral and Spatial Dimensions in Imaging Spectroscopy Processing

Hyperspectral data can be processed using different viewpoints. In early studies, pixel-wise processing was preferred using deep learning methods. This is done by extracting the spectral signature from each pixel and then comparing it to a known object’s spectral signature. We require some prior knowledge about the desired target in this approach. an example of such a study can be found in [23]. To reduce correlated information in spectral signature and remove redundant data, we can perform dimensionality reduction methods i.e.PCA [24], ICA [25], and autoencoders[26].

Dimensionality reduction is usually applied in addition to extracting features from the whole spectral span or on defined 2-dimensional patches (both spectral and spatial dimension dimensions). Extracting features in spectral-spatial dimensions requires extracting information from raw hyperspectral data cubes without applying prior knowledge and/or dimension reduction. This is heavy in computation thus there is a preference to work on sub-volumes instead of the whole data cube.

2.2. Convolutional Neural Networks

Artificial Neural Networks (ANNs) stemmed from biological neural systems. They contain an input layer, one or more hidden layer(s) and an output layer [27]. Historically, the development of neural networks has been based on the mathematical modeling of neurons in biological systems. Neurons are defined as the basic computational units in brains. Input is given to the neuron from the dendrite, the output is sent out via axon and the transmission is done through the synapse. A comparison of a biological neuron and the mathematical model in the neural network is provided in Figure 2.

A network with multiple hidden layers is called a deep neural network [29]. A simple drawing of a deep neural network is pictured in Figure 3.

For extracting information from images, Convolution Neural Networks (CNNs) have been introduced [31]. This type of network has been extensively used so far for different imagery analyses [32]. In CNN, the input image is constrained by its architecture. Normally, the neurons are arranged in three dimensions width $\left(w\right)$, height $\left(h\right)$, and depth $\left(d\right)$. Depth is the input depth, in the case of hyperspectral imagery, depth is the number of bands. By proceeding deeper into the network, it refers to the number of features of the input layer. In each layer, the neurons are connected to a selected number of neurons from the previous layer. This is to decrease the number of weights that needs to be defined [32].

CNN’s have also been combined with machine learning methods i.e.SVM to extract features and increase robustness toward over-fitting [33]. In this study, a target pixel and the spectral information of its neighbors are organized into a spectral–spatial multi-feature cube without extra modification of the CNN to classify land cover. Another example in [34] is a 2-channel deep CNN that has been used to do land cover classification combining spectral-spatial features. A hierarchical framework has been used for this purpose in [35]. Similarly, in [36] a method is proposed in which spatial and spectral features are extracted through CNNs from Imaging spectroscopy and Lidar. A pixel-wise classification using a 2-channel CNN and multi-source feature extraction was done in [37]. In [38] a framework for Imaging spectroscopy classification has been proposed that uses a fully-convolutional network to predict spatial features starting from multiscale local information and to fuse them with spectral features through a weighted method. This approach later performs classification using SVM.

2.2.1. Spectral Dimensional CNN

one-dimensional CNN (1D-CNN) is used to perform pixel-wise classification for Imaging spectroscopy processing. These networks apply to the spectral or spatial dimension. These networks are affected by noise easily thus making it challenging to use them for remote sensing, in general [39]. One solution is to use averaged spectrum from a group of neighboring pixels. This method best suits small-scale analyses such as crop segmentation [40]. Another solution is to perform PCA analyses before running CNN however, in the case of near real-time image processing, there is no room for heavy pre-processing tasks such as PCA. A different solution described in [41] uses a multi-scale CNN that applies to a pyramid of data that contains spatial features in multiple-scale. For small training samples, a band selection before CNN analysis has been proposed [42].

2.2.2. Spectral-Spatial Dimensions CNN

Working with both spectral and spatial features generally leads to better results in Imaging spectroscopy processing. In [43] a dual-stream channel CNN has been used that gets spectral features from the approach of [39], spatial features using the approach of [44] and a softmax regression classifier to combine those features. In [45] a combination of L2 norm and sparse constraint have been used with a similar combination of spectral-spatial features. In other studies, AlexNet [46] have been employed to do spatial-spectral analyses i.e. Densenet and architecrures like VGG-16 [38,47]. In [48] few-shot learning approach [49] has been used to learn a metric space that causes the samples of the same class to be close to each other to deal with the problem of a few training samples. Another way to improve accuracy while having a shortage of training information has been proposed in [50]. In this approach, the redundant information in the hidden layer is explored to find connections and improve the training process. Other examples of using spatial-spectral features together and improving the learning process can be found in [51,52,53]. They have used a variety of methods based on the super-pixel reconstruction of different features to improve the accuracy of segmentation and classification. The sensor-based feature learning is another method proposed by [54] in which five layers of spectral-spatial features were reconstructed according to sensor specifications. Another improvement to sensor-based training was explained in [55] which uses a novel architecture that actively processes input features into meaningful response maps for classification. All of the mentioned studies have used complex multi-step procedures that make them not suitable for (near)real-time processing of Imaging spectroscopy, however, the better performance of using multi-scale and multi-feature approaches has been proved according to mentioned studies in this section.

2.3. Auto-Encoders

To deal with the issue of limited training samples when processing Imaging spectroscopy, auto-encoders in different variations have been tested. For the first time in [56] PCA in spectral dimension was combined with auto-encoder in the other two dimensions to improve feature extraction for classification. In [57] and [58] stacked auto-encoders were employed in combination with PCA to flatten spectral dimension and followed by SVM and multi-layer perceptron (MLP) to perform classification. In [59] stacked auto-encoder was optimized for anomaly detection in Imaging spectroscopy. A combination of auto-encoders and CNN have been also tested in multi-scale approaches to extract features [60]. Another important point of using a stacked auto-encoder is the capability of handling noisy input. An example described in [61] used a stacked auto-encoder to generate feature maps from noisy input and then used super-pixel segmentation and majority voting. Another study used a pre-trained network by stacked encoders combined with logistic regression on noisy input to do supervised classification [62]. A framework based on stacked auto-encoders has been proposed to perform unsupervised classification on noisy input [63], this later was improved to an end-to-end classification pipeline for Imaging spectroscopys [64].

2.4. Deep Belief Networks, Generative Adversarial Networks, Recurrent Neural Networks

Deep belief networks (DBNs) have the capability of dimension reduction which makes them a good candidate to extract features. In [65] a DBN was combined with logistic regression to perform feature extraction. A combination of one- and two-layer DBN was combined with PCA. To perform (near)real-time anomaly detection DBN has been tested that also delivered promising results in extracting local objects [66]. A combination of DBN and wavelet transform has been also proposed by [67]. In [68,69] unsupervised classification was performed using DBNs and in the later study, an end-to-end classification framework based on DBNs and spectral angle distance metric were proposed. In Generative Adversarial Networks (GANs) two competing neural networks are used as generator and discriminator [70]. These networks have been used to perform classification when dealing with small training samples [71]. In similar cases, GANs have been employed to perform the final phase of Imaging spectroscopy classification using the discriminator agent [72,73,74]. Recurrent Neural Networks (RNNs) are mainly used to process time series. In the case of this, they are considered as sequences of a video series (each spectral band as a sequence), and RNNs are used to find similarities between time frames [75,76]. A combination of RNN to explore the spectral domain and LSTM(Long Short Term Memory) for exploring spatial features was proposed in [77]. RNNs have also been used to process mixed pixels in spectral dimension affected by noise [78].

2.5. Unsupervised and Semi-Supervised Approaches

Based on the fact that usually we face the problem of having limited training samples at hand semi-supervised and unsupervised approaches are getting more popular in the domain of Imaging spectroscopys. Examples can be found in [79,80] that use semi-supervised and layer-wise classification to process large-scale Imaging spectroscopys. Another example of performing pixel-wise classification can be found in [81] using an unsupervised method with CNN. First, inaccurate training samples were used and the classification was improved with a small set of accurately labeled training samples. In [82] to handle the limited training sample problem, a convolution-deconvolution network was used for unsupervised spectral-spatial feature learning. The convolution network was used to reduce dimensionality and deconvolution was used to reconstruct input data respectively. Another possibility that has been explored to deal with a few training samples is improving the training procedure as explained in [83] where unlabeled data is used in combination with a few labeled samples and RNN to classify Imaging spectroscopys. Another approach that was tested is using ResNet to learn spectral-spatial features from unlabeled data which also showed promising results [84].

2.6. Challenges in Imaging Spectroscopy Processing and New Trends for Handling Them

2.6.1. Limited Training Sets

The issue of having limited training samples remains a constant problem so far in the world of Imaging spectroscopy processing. New approaches have been explored in the direction of using semi-supervised techniques [85], self-supervising approaches [86] and domain adoption [87] which explores the discriminative input information to feed the neural network. Another approach is active transfer learning which used the most discriminative features from unlabeled input training samples [88].

2.6.2. Handling Noisy Data

To reconstruct high-quality input data for classification, some approaches are getting noticed. One study has explored super-resolutions in combination with transfer learning to reduce noise and improve the quality of the input training samples [89]. Other studies have used CNN with sparse signal reconstruction [90] and Laplacian pyramid network (LPN) [91] for enhancing input data. Another method explored presented in [92] uses structure tensors with a deep convolutional neural network to improve the quality and reduce noise.

2.7. Increase Speed and Accuracy

A new trend in the field of computer vision is using CapsuleNets (CapsNet) [93] which uses a set of nested neural layers. These networks increase the scalability of the model while increasing the speed of computation. Examples can be found in [94,95,96]. It was shown that by using spectral-spatial Capsnet the model converged quickly while avoiding over-fitting [97].

2.8. Hardware Accelerators

To increase the performance of HS data processing different hardware has been tested, such as computing clusters [98], GPUs and FPGAs (Field Programmable Gate Arrays) [99]. Recent advances in FPGAs have made them a suitable candidate to perform on-board image processing in both airborne and spaceborne platforms [100].

FPGA is a hardware unit consisting of an array of logic blocks, RAMs, hardcopy IPs, I/O pads, routing channels, etc [101]. It can be customized to perform different functions to be performed at different times and levels. A previous generation of similar technology was called ASICs [102]. FPGAs are more flexible and easier to program. It has shown lower power consumption and improved performance compared to on-board processing of hyperspectral imagery [103]. A few studies recently have performed different functions related to onboard HIS processing including data compression and image segmentation [104].

Older versions were using FPGAs for end-member extractions [103], another one used Xilinx Virtex-5 FPGA for automatic target detection [105] and Xilinx FPGA was used to perform end-member extraction for multiple targets [106]. Spectral signature un-mixing has been also tested on FPGA and graphical processing units (GPUs). Results have been competitive in terms of accuracy.

One study has used FPGA to demonstrate onboard processing capability to detect chemical plumes [107]. This study has been a pilot phase for developing an AI unit for the upcoming hyperspectral satellite to be launched by NASA JPL. A main drawback of using FPGA is the difficulty of its configuration and programming. For solving this OpenCL package from Intel and VITIS library (previously VHDL) from Xilinx have been developed [108]. Therefore, there have been limited studies on implementing deep learning for FPGA. Thus, our future step is going to be implementing deep learning on FPGA on the proposed hardware architecture for future CHIME missions [109].

3. Summary and Discussion

We explored the most recent trends in using deep learning for hyperspectral imagery. Almost all of the reviewed studies referred to limited training samples as a main limiting factor to employing deep learning widely in the HS image processing field. Another mentioned limiting factor is the lack of computation infrastructure and hardware in remote sensing-related studies. According to the review, there are many studies on using deep learning for land cover classification however there is still a gap in the studies on target and anomaly detection as well as data fusion and spectral unmixing. Segmentation of Imaging spectroscopy using deep learning is still a path less walked. Network architectures such as UNet, ResNet, and VNet are proven to be good choices to start with, although the application-based scenarios still need more work to be defined. Regarding the classification of Imaging spectroscopys, deep learning has shown to be effective, however since a lot of computational resources are required to perform deep learning and satisfactory results can be obtained from traditional classification approaches i.e.SVM, there is still reluctance in many users to employ deep learning. To handle the problem of limited training sets and noisy input data using GANs can be a good option to produce an augmented dataset and reduce noise in training samples. reinforcement learning can also be a good candidate that is worth further exploration. Since there is a trend to process Imaging spectroscopy onboard (using hardware accelerators) for both remote sensing and non-remote sensing applications, a summary of the most common methods according to their suitability for on-board implementation is provided in Table 1.

According to Table 1, conventional methods are easy to implement but need many training samples and traditional processing and updating procedures. Therefore, using CNN-based methods has found its place within the hyperspectral users’ community. Several versions of neural networks have been tested and in total deep CNN and 3D-kernel-CNN networks have shown very good results. Since we are focusing on optimizing network structure for onboard processing, GhostNets might be a good option as well. However, the accuracy might not be optimal. Other challenges when aiming for on-board processing of Imaging spectroscopy are noisy data and no atmospheric correction available at level zero data as a well limited training set. Therefore, we should focus on testing different network structures on simulated and real data similar to the upcoming CHIME mission shortly [114]. Overall, on-board processing of HS imagery is the new area of study that will open many new possibilities in the remote sensing domain.

4. Conclusion

Particularly in industries that profit from the computer-assisted interpretation of both visible and unseen (to the human sight) occurrences, the depth of information present in Imaging spectroscopy data is unquestionably attractive. However, cost-benefit analyses of industrial and professional Imaging spectroscopy technologies make it necessary for enabling elements to be present to activate their deployment potential. Machine learning technologies are expanding quickly in scope these days, and with the introduction of Deep Learning, they are changing the field of digital data analysis. By using a multidisciplinary approach and making our work accessible to practitioners, machine learning scientists, and domain experts, we attempted to examine what is currently occurring with the convergence of Imaging spectroscopy and deep learning technologies in this study. One of the key problems that developed as a barrier to high-quality scientific production is the publicly available datasets, even though pixel- and spectral-based analysis jobs may count on an order of thousands of training samples for Imaging spectroscopy volume. More generally, the quantity and caliber of data collected across the spectrum of disciplines continue to be a major obstacle to the creation of solid, efficient, and comprehensive Imaging spectroscopy-DL solutions. The provision of high-quality Imaging spectroscopy datasets can instead be encouraged by the investigation of various DL techniques for the RS field. Additionally, the ability to approach difficult visual tasks via DL solutions can be beneficial for other application domains where the penetration of Imaging spectroscopy technology is still far behind.