Combining KAN with CNN: KonvNeXt's Performance in Remote Sensing and Patent Insights

1. Introduction

The fast growth of satellite technology has resulted in a surge of high-resolution remote sensing (RS) images [1,2]. Integrating data from many satellites into time series has increased the identification of particular occurrences and trends, increasing our understanding of global climate patterns and ecological changes [3,4]. Satellites such as Landsat, Sentinel, MODIS, and Gaofen are now among the most significant in Earth observation, with uses ranging from monitoring deforestation to tracking atmospheric conditions. To monitor vegetation, crop yields, and forest disturbances, for instance, developing sun-synchronous polar orbiting satellites has been essential [5].[6]. Deep learning models used in those cases have shown significant results in a variety of applications, including environmental monitoring, item detection, and land cover categorization [7,8]. Advanced image processing and analysis were made possible by the early active application of deep learning techniques in the RS area.

To gain a better knowledge of the usage of deep learning (DL) and machine learning (ML) techniques in remote sensing, we examined patent titles that contained terminology of this kind. As Figure 1 shows, from 23 in 2015 to 278 in 2023, there was a significant increase in the quantity of these patents, which increased steadily between 2015 and 2023. This tendency suggests that while these technologies will continue to be employed in remote sensing, more advanced deep learning technologies will be necessary to manage more complex and vast remote sensing data and applications [9].

However, applying deep learning to high-resolution datasets raises major obstacles. The vast number of parameters required for deep neural networks, particularly for high-resolution images, can result in higher processing costs and memory needs, rendering the training process inefficient and potentially infeasible with standard hardware resources. Furthermore, over-parameterization can lead to longer training durations and perhaps overfitting, in which the model memorizes the training data rather than generalizing from it, limiting its usefulness in real-world applications [10]. To address these difficulties, novel ways for successful training are needed, and while reducing the number of parameters the model should retain its performance. Model pruning and quantization are representative solutions and have been widely used, but each method has its own set of benefits and downsides [11,12]. Therefore, in this paper, we introduce a novel network, which is the Kolmogorov-Arnold Network (KAN). The KAN's characteristic could provide a more efficient learning framework [13]. The main contributions of this paper can be summarized below.

By replacing standard Multi-Layer Perceptrons (MLPs) with KAN, we hope to improve the efficiency and performance of remote sensing applications. Our strategy includes using and comparing different pre-trained Convolutional Neural Networks (CNNs) and Vision Transformer (ViT) models to determine the best KAN pairings, resulting in optimum performance.

Based on the above result, we presented and evaluated our suggested model, KonvNeXt, on four different remote sensing datasets to compare its performance to existing results. This experiment allowed us to scrutinize KonvNeXt's performance in remote sensing fields.

In addition, we applied Explainable AI (XAI) approaches to our model for its interpretability. This process is significant for understanding the decision-making processes of deep learning models, leading to transparency in AI-driven remote sensing applications

2. Related Works

First, we introduce the related research that applied deep learning technologies to the remote sensing datasets. Han and Basalamah showed the effectiveness of a multi-branch system combining pre-trained models for local and global feature extraction in satellite image classification and achieving superior accuracy across three datasets [14]. By maximizing the performance of small neural networks, Chen et al. used knowledge distillation to increase accuracy on a variety of remote sensing datasets [15]. Broni-Bediako et al. used automated neural architecture search to develop efficient CNNs for scene classification, which outperformed more traditional models with fewer parameters, such as EuroSAT and BigEarthNet [16]. Temenos et al. also applied an interpretable deep learning framework to land use classification using SHAP for both high accuracy and enhanced interpretability [17]. Yadav et al. enhanced classification on the EuroSAT dataset by using pre-trained CNNs and advanced preprocessing techniques, and GoogleNet showed the best performance [18]. These studies highlight advancements in model accuracy, efficiency, and interpretability for remote sensing image classification.

Then, we introduce the research that utilized KAN for classification or regression. In satellite traffic forecasting, Vaca-Rubio et al. showed that KANs performed better with fewer parameters than traditional MLPs while yielding greater accuracy. The substantial influence of KAN-specific factors was demonstrated by that research, which highlighted their potential for adaptive forecasting models [19]. Bozorgasl and Chen addressed diverse data distribution across clients by developing Wavelet Kolmogorov-Arnold Networks (Wav-KAN) for hybrid learning. Extensive trials on datasets including MNIST, CIFAR10, and CelebA validated their improvements, which were substantially increased by their incorporation of wavelet-based activation functions [20]. Using KANs, Abueidda and Pantidis created DeepONets to develop efficient substitutes for mechanics problems. Their approach required fewer learnable parameters than traditional MLP-based models, and it was validated in computational solid mechanics. This demonstrates notable computational speedups and efficiency, making it suitable for high-dimensional and complex engineering applications [21].

While previous research has utilized pre-trained models, knowledge distillation, NAS, and interpretability frameworks to enhance performance on remote sensing datasets, our technique focuses on determining the possibility of applying KAN to the RS dataset. Our suggested methodology makes sense with the current trends of KAN, which provide a new way for optimization in remote sensing applications. Our integration of KAN with ConvNeXt indicates its capacity to handle remote sensing datasets effectively, unlike other approaches that rely on complex multi-branch frameworks or preprocessing procedures, offering a viable alternative for future study in this sector.

3. Materials and Methods

3.1. Dataset Description and processing

Four different datasets, including EuroSAT, Optimal-31, AID, and Merced were used in this experiment. The EuroSAT dataset proposed by Helber et al. is used to categorize land cover and land use. This collection has 27,000 photographs from 34 different places in Europe, divided into ten categories: annual crop, forest, herbaceous vegetation, highway, industrial, pasture, permanent crop, residential, river, and sea lake [22]. The OPTIMAL-31 dataset consists of 1860 aerial images sourced from Wang et al. This dataset includes 31 different land use types, with each type represented by 60 images of 256×256 pixels, boasting a spatial resolution of 0.3 meters [23]. AID is a large-scale aerial image dataset sourced from Google Earth, 600 × 600 pixels, containing 10,000 images across 30 scene types such as airport, forest, and residential areas. Despite post-processing, these images are proven comparable to real optical aerial images for land use and cover mapping [24]. Merced consists of 21 land use classes, with 100 images per class, each measuring 256x256 pixels. The images were manually extracted from the USGS National Map Urban Area Imagery collection, featuring a pixel resolution of 1 foot [25]. Figure 2, Figure 3, Figure 4 and Figure 5 describe examples from each dataset.

The dataset has been divided into three sections for experimentation: approximately 70% of the data is allocated for training, 15% for validation, and 15% for testing. As part of the preparation of the dataset, images are resized to 224 × 224 pixels and normalized using the mean and standard deviation values of ImageNet (mean = [0.485, 0.456, 0.406] and std = [0.229, 0.224, 0.225]). The transformations applied to the data include normalization, conversion to tensors, random horizontal and vertical flips, and random resizing of the cropped images.

3.2. ConvNeXt

Based on conventional ConvNet methods, the ConvNeXt algorithm explicitly modernizes the ResNet model by integrating several Vision Transformers (VIT) design features. As Figure 6 describes, the main structural changes include converting the stem cell to a patchify layer, which divides the input picture into smaller, non-overlapping patches and treats each as a sequence token for additional processing. The algorithm adopts an inverted bottleneck block, used by MobileNetV2, to expand the number of channels before reducing them, capturing spatial and channel-wise features with reduced computational cost. Depthwise convolutions with larger network widths process each input channel separately, which could reduce computational complexity, while pointwise convolutions (1x1 convolution) combine information across channels. Moreover, ConvNeXt applies big kernel sizes for convolutions, Gaussian Error Linear Unit (GELU) activation functions, and LayerNorm which is also the main strategy of the vision transformer. With these modifications, the ConvNeXt could achieve better accuracy than VIT models. Furthermore, by adjusting the hidden dimension sizes, there are several versions such as ConvNeXt-tiny, and ConvNeXt-large [26,27].

3.3. Kolmogorov Arnold Network

Inspired by the Kolmogorov-Arnold representation theorem, Kolmogorov-Arnold Networks (KANs) is a sophisticated kind of neural network with learnable activation functions on edges as opposed to fixed activations on nodes in conventional Multi-Layer Perceptrons (MLPs). B-splines, piecewise polynomial functions defined by control points and knots, parameterize these activation functions. Spline-parameterized functions ${\phi }_{q,p}$ change each input feature $x_p$, aggregate the results into intermediate values for each q, and then pass the values through functions ${\Phi }_q$ . The total of these converted values is the final output f(x) which enables the network to effectively and flexibly capture complex data patterns. The activation functions in KANs are a combination of a Basis Function and a Spline, with the Basis Function often being the Sigmoid Linear Unit (SiLU), defined as $silu\left(x\right)=\frac{x}{(1+e^{-x})}$. The spline component $spline\left(x\right)={\sum }_i{c}_i{B}_i\left(x\right)$uses B-spline basis functions Bi(x) and coefficients ci, which are learned during training and Figure 7 depicts the plot. These coefficients determine the final shape of the activation functions, replacing the need for traditional linear transformation parameters W and b in MLPs.

There are also multiple neurons in the KAN which is similar to that of the MLP. Each hidden layer contains nodes that aggregate the inputs from the previous layer. The output of each node is determined by the spline-parameterized functions of the incoming edges. As data flows through the network, each input to a node is transformed by the B-spline functions on the edges. The transformed inputs are then aggregated (usually by summation) at the node. This aggregation forms the input to the next layer. The training process in KANs involves adjusting the control points and knots of the B-splines to minimize the loss function, and there are three key steps. The first is to compute the output by passing inputs through the network and applying spline transformations at each edge. Backpropagation computes gradients of the loss concerning the control points and knots. These gradients are utilized to update the parameters using optimization algorithms like gradient descent. KANs exhibit various advantages over MLPs, including better accuracy and interpretability with fewer parameters. They achieve this by having smaller architectures that can perform comparably or better than larger MLPs in tasks such as data fitting and partial differential equation (PDE) solving. Since KANs can be depicted, they help discover mathematical and physical laws in scientific applications. Moreover, KANs can aid in preventing catastrophic forgetting, a neural network problem when the learning of new information causes the loss of previously learned information. One notable drawback of KANs over MLPs is their slower training pace, which optimization can address [13,28].

The formula for the Kolmogorov-Arnold Network (KAN) is given by:

$$f\left(x\right)={\sum }_{q=1}^{2n+1}{\Phi }_q\left({\sum }_{p=1}^n{\phi }_{q,p}\left(x_p\right)\right)$$

(1)

The function f (x) in a KAN, where ${\phi }_{q,p}\left(x_p\right)$ are spline functions and Φq are transformations.

$$\phi \left(x\right)=w\left(b\right(x)+spline(x\left)\right)$$

(2)

φ(x) denotes the activation function, w is the weight, b(x) is the basis function, and spline(x) is the spline function.

$$b\left(x\right)=silu\left(x\right)=\frac{x}{(1+e^{-x})}$$

(3)

b(x) is the basis function, implemented as silu(x) (Sigmoid Linear Unit).

3.4. ConvNeXt Kolmogorov-Arnold Networks: KonvNeXt

The proposed approach combines KAN and the ConvNeXt architecture to improve a pre-trained ConvNeXt model's learning capacity. To enhance feature-extracting performance, we adopted the models that were trained with the Imagenet datasets. The model adds two KANLinear layers instead of the conventional MLP classifier. Unlike typical neural networks, which use fixed activation functions on the nodes, KANLinear layers use learnable activation functions on the edges. More stability and flexibility are provided by the use of B-splines to describe these activation functions. By substituting these spline functions for linear weights in the KANLinear layer, the model is better able to execute the model efficiently because the KAN layer does not consume much memory compared to the original layers. This approach is not only applicable to ConvNeXt, which means it could replace neural net layers in various deep learning models.

4. Results

First, we applied various CNN-based models that were pre-trained on the ImageNet dataset to figure out the most optimal one on the EuroSAT dataset. Typically, these pre-trained networks utilize MLP layers for classification or regression tasks. However, since our research aimed to demonstrate the potential of replacing MLP with KAN, we substituted the MLP with KAN in these models. We evaluated several models, including VGG16, MobileNetV2, EfficientNet, ConvNeXt, ResNet101, and ViT [29,30,31,26,32,33]. The observed accuracies for these models were 88%, 75%, 67%, 94%, 75%, and 92%, respectively, as shown in Figure 8. To demonstrate the efficiency of KAN, we initially configured each KAN layer with 256 nodes and subsequently reduced the nodes to 32 for comparison. The results revealed identical performance between both configurations. Both setups achieved an accuracy of 94% in the first epoch, which increased to 96% in the second epoch, maintaining this accuracy in the subsequent epochs. This experiment indicates that KAN layers can achieve high accuracy with fewer training epochs, even when the number of nodes is significantly reduced and this proved the efficiency of the KAN layer. From the proposed approach of integrating ConvNeXt with KAN, we figured out interesting results in terms of classification accuracy per epoch. The model achieved an accuracy of 94% in the first epoch, which increased to 96% in the second epoch. This accuracy of 96% was maintained consistently through the fifth epoch. These results suggest that even with one or two epochs, sufficient training of KAN-based networks can be achieved. Despite the drawback that KAN is ten times slower than traditional MLPs, the reduced number of necessary training epochs effectively overcomes this downside [34].

Then, with our KonvNeXt model, we tested three more popular datasets in remote sensing fields, including Optimal-31, AID, and Merced. We chose these datasets since there already exists a paper that utilized VIT for remote sensing datasets [35]. As Table 1 describes, with 25 epochs, our proposed model achieved 90.59% for Optimal-31, 94.1% for AID, and 98.1% for Merced. Since Bazi et al.’s VIT model yielded 92.76% for Optimal-31, 91.76% for AID, and 92.76% for Merced datasets, this result implied that our proposed approach resulted in better performance than the VIT models. This result is meaningful since the proposed model did not demand long training time. For instance, it took 107.14 seconds for the Optimal-31 dataset, 545.91 seconds for the AID dataset, and Merced for 107.63 seconds. Since Bazi et al.’s model took an average of 30 minutes to train those datasets, these results proved that our approach not only achieved high performance but also a fast speed which could overcome the original downsides of the KAN models [35].

In another experiment, we compared the performance of our proposed model to that of the original ConvNeXt models. The ConvNeXt achieved accuracy rates of 84.68%, 94.6%, and 97.8% on the three datasets as depicted in Table 2. These results indicate that substituting the neural network's linear layers with the KAN layers can be effective in terms of performance and memory efficiency. Interestingly, by using only two layers for comparison, the speed gap between the models was small. This result suggests that the ConvNeXt model with the KAN layer is efficient enough for these remote sensing datasets, even with fewer linear layers.

Since our proposed approach utilizes ConvNeXt to capture features from images, XAI methods can be applied to our model. For instance, we applied Occlusion Sensitivity in the above experiments. Occlusion sensitivity is a technique used in explainable AI to identify which regions of an image most influence a model's predictions by systematically occluding parts of the image and observing changes in the output. The heatmap overlay in the right image highlights these influential regions, with red/yellow areas indicating high sensitivity and blue areas indicating low sensitivity, and each results are shown in Figure 9, Figure 10 and Figure 11. The results demonstrated that the proposed model classified the target with reasonable decision-making. Furthermore, this result also shows the potential for applying this model to diverse datasets, where interpreting deep learning is crucial, such as in weather prediction or medical diagnosis [36].

5. Discussion

This paper introduces the application of Kernel Attention Networks (KAN) to remote sensing datasets by integrating them with the ConvNeXt algorithm. Given that KAN is in its early stages of development and has primarily been tested on MNIST, CIFAR-10, CIFAR-100, and ImageNet datasets, its suitability for specific tasks was previously uncertain. Therefore our work is pioneering in this field, since we evaluated the efficacy of our proposed model, KonvNeXt, by testing it on four different representative remote sensing classification datasets. The experiments demonstrated that KonvNeXt has significant potential for application across various datasets. Additionally, our approach may offer more applicability because many existing deep learning models utilize neural network layers that demand substantial GPU memory based on their parameters. In this context, the KAN layer could serve as a substitute, extending beyond computer vision tasks to include natural language processing (NLP) tasks, even in large language model (LLM) tasks. For future work, we will explore models that consist solely of KAN layers, following the current trend, without merging them with any CNN models, and apply them to remote sensing datasets to compare the performance of KAN to the existing CNN models [37,38].