Multiband Image Analysis Using Ensemble Neural Networks

1. Introduction

Image classification, a historically challenging task, has seen dramatic advancements with the rise of deep learning (DL) and improvements in computational hardware. Among DL models, Convolutional Neural Networks (CNNs) and vision transformers have consistently demonstrated superior performance in various image classification challenges [1, 2]. However, neural networks often produce results influenced by inherent randomness, leading to inconsistent outcomes [3]. Ensemble Learning (EL), which combines outputs from multiple models, has proven effective in mitigating these inconsistencies, yielding improved accuracy and reliability.

Multiband images, characterized by their representation of data across multiple spectral or feature dimensions, are increasingly prevalent in various fields of study and application. Unlike traditional grayscale or RGB images, multiband images provide a richer and more detailed view of the underlying data, capturing diverse information that spans beyond the visible spectrum or typical feature space. The added complexity of multiband images opens avenues for advanced analysis, including precise classification tasks in domains such as remote sensing, medical imaging, industrial inspection, and scientific research [4].

The classification of multiband images poses unique challenges and opportunities. The high dimensionality of the data often encapsulates critical features necessary for distinguishing classes, yet it also introduces noise, redundancy, and computational complexity. To address these challenges, researchers have developed a range of techniques for feature extraction, dimensionality reduction, and classification that aim to balance accuracy and efficiency. Recent advancements in computational methodologies, including machine learning and DL, have shown significant promise in harnessing the potential of multiband datasets, enabling more robust and adaptive classification frameworks [5].

This paper explores a practical application of EL on two classification tasks. The first task addresses the problem discussed in [6], which involves training a CNN for foraminifera classification—a topic of significant interest in both industrial and research contexts [7]. Planktic foraminifera species serve as paleo-environmental bioindicators, with their radiocarbon measurements providing insights into parameters such as global ice volume, temperature, salinity, pH, and nutrient levels in ancient marine environments. Traditionally, foraminifera classification is carried out by large groups of human experts, typically ranging from 500 to 1000 individuals. This process is repetitive, labor-intensive, and time-consuming. Since the early 1990s, several efforts have aimed to automate this task [8]. Despite notable advancements, most methods still required substantial human oversight. However, the development of the neural network SYRACO2 in 2004 marked a significant breakthrough by reliably automating the identification of single-celled organisms [9]. In 2017, CNNs demonstrated substantial success in diatom identification [10], suggesting that further advancements in CNN technology could effectively alleviate the demanding task of foraminifera classification. In [6], the authors utilized a combination of ResNet50 and Vgg16, employing colorization techniques based on intensity percentiles across sixteen grayscale channels. Several novel colorization methods are introduced in [11] that diverge from the current state-of-the-art approaches [12] yet deliver impressive ensemble results. The findings in [11] demonstrated that training diverse CNN models on differently colorized images enhances system performance compared to other leading methods. Additionally, in both of these studies and in [6] the proposed systems surpassed human experts in classification accuracy.

The other classification problem is related to the classification of land use and land cover (LULC), a crucial task for understanding environmental changes, urbanization patterns, and resource management. Accurate LULC mapping relies on robust algorithms capable of interpreting complex datasets generated by modern remote sensing technologies. With the advent of high-resolution multiband satellite imagery, such as those provided by the Sentinel-2 mission, researchers have gained access to rich datasets that capture diverse spectral characteristics of the Earth’s surface, enabling enhanced LULC classification [13]. DL has emerged as a breakthrough approach in remote sensing, offering unparalleled performance in tasks such as object detection, segmentation, and classification [14, 15]. By leveraging hierarchical feature extraction, DL models outperform traditional machine learning techniques in handling high-dimensional and heterogeneous satellite datasets. However, the development of effective DL models necessitates standardized benchmarks for comparative evaluation, reproducibility, and optimization [16]. The EuroSAT dataset, a labeled multiband satellite image dataset derived from Sentinel-2 data, has become a prominent benchmark in this field. It provides labeled samples across ten classes, such as urban, agricultural, and natural landscapes, representing diverse LULC categories. The dataset availability in multiple spectral bands, including red, green, blue (RGB), and near-infrared (NIR), facilitates the exploration of spectral and spatial features for LULC classification tasks [13]. As mentioned above, in this paper, we use the EuroSAT dataset based on Sentinel-2, which is widely used in the literature. A ranking of the performance of a large set of different methods using this dataset to validate their performance is available at https://paperswithcode.com/sota/image-classification-on-eurosat (accessed on 12/17/24).

In this paper, we show how to create an ensemble of neural networks, creating three-channel images using different methods that can be used to tune a pretrained neural networks on RGB images. We use three architectures well-known in the literature: ResNet50; MobileNetV2, and DenseNet201. Each ensemble is obtained by combining (with the sum rule) 20 networks; thus, 20 different three-channel images are created from the initial multiband images. The proposed system obtains SOTA in both datasets used in this work.

The contributions in this paper are as follows:

• Presentation of a simple method for randomly creating three-channel images starting from multiband images, useful for CNNs pretrained on very large RGB image datasets, such as ImageNet;
• An example of EL based on DL architectures and the proposed approach for generating three-channel images from multiband images, where each network is trained on a given set of generated images (e.g., each network is trained on images created in the same way, where at least one channel is from one of the three RGB channels);
• This approach, though computationally more expensive than other approaches, is simple, consisting of only a few lines of code;
• This approach obtains SOTA with no change in hyperparameters between datasets.

1.1. Related Work

Mitra et al. [6] introduced a framework based on DL in 2019 that achieved moderate success over the performance of human experts (85% accuracy over 83%, respectively). Their approach, however, placed limited emphasis on the preprocessing stage, particularly the colorization process, potentially overlooking an opportunity to enhance performance further.

It is worth noting that the application of multi-grayscale channel colorization to RGB is not confined to foraminifera classification; it extends to other fields as well. For example, in remote sensing [17, 18], multispectral images often represented in grayscale can benefit from this technique, as can medical imaging, where grayscale formats like CT scans and MRI images [19] are prevalent. These colorization techniques can be employed on diverse classification challenges, including clinical diagnosis and image-guided interventions [20]. Studies have demonstrated that effective image fusion and colorization of multiband representations can enhance the informational content of images [21]. This success indicates that improved preprocessing methods could enable DL frameworks to achieve superior performance, as shown in prior research [22]. The use of RGB colorization in these domains has already proven beneficial, improving classification accuracy and result interpretability. For instance, in medical imaging, converting grayscale images to RGB can make subtle features more discernible, potentially improving diagnostic accuracy [23]. Similarly, in remote sensing, RGB colorization can highlight specific elements, such as vegetation or water bodies, enhancing CNN performance [24]. Therefore, the proposed methods are likely to increase classification accuracy across various domains that rely on grayscale imagery. They may also perform well in image fusion tasks involving grayscale images from multispectral analyses [17, 18] or polarized/filtered light sources used to capture objects outside the visible spectrum.

Additionally, ensemble learning, which combines different neural networks utilizing diverse image fusion techniques, has been shown to significantly outperform individual methods [25, 26]. This suggests that integrating multiple colorization approaches into ensembles could further boost the key performance indicators for the classification tasks under consideration.

2. Materials and Methods

2.1. Foramnifera Dataset

The foraminifera dataset [27], see Figure 1 for selected samples utilized in this study, is available at https://doi.pangaea.de/10.1594/PANGAEA.897873 (accessed 12/17/2024).

It consists of 1,437 samples categorized as follows:

• 178 images of G. bulloides;
• 182 images of G. ruber;
• 150 images of G. sacculifer;
• 174 images of N. incompta;
• 152 images of N. pachyderma;
• 151 images of N. dutertrei;
• 450 images labeled as “rest of the world,” representing other species of planktic foraminifera.

The original images were captured using a reflected light binocular microscope, with a light source positioned at 22.5° intervals. The equipment employed was an AmScope SE305R-PZ binocular microscope at 30×magnification [6]. For each foraminifera sample, sixteen grayscale images were taken under varying illumination angles. Image resolution varies across samples but is generally around 450×450 pixels. As the testing protocol, a four-fold cross-validation is used.

2.2. EuroSAT Dataset

Sentinel-2, a part of the European Union’s Earth observation program, Copernicus, provides high-resolution satellite images that are both openly accessible and freely available to the global research community. These satellite images capture detailed information across thirteen distinct spectral bands (see Table 1 for details), making them a valuable resource for diverse Earth observation applications.

This dataset [13] encompasses a comprehensive collection of 27,000 labeled images of size 64×64 pixels images. The images are geo-referenced and meticulously categorized into ten distinct classes representing various land use and land cover types. Each class contains somewhere between 2,000 and 3,000 images. Samples are shown in Figure 2. The combination of multispectral data and precise geospatial referencing provides an unprecedented opportunity for researchers to develop and evaluate robust classification models. The ten classes are the following: industrial buildings, residential buildings, annual crop, permanent crop, river, sea & lake, herbaceous vegetation, highway, and pasturem forest.

For the testing protocol, we split the data into 80% for the training and 20% for testing as is standard practice. This dataset is available at https://github.com/phelber/eurosat(accessed (accessed at 12/17/2024). It is split into training/test sets available at https://huggingface.co/datasets/torchgeo/eurosat/tree/main (accessed 12/17/2024).

It is important to note that the maximum value in each band is not 255, as in standard color images. For this reason, we normalize each band so that the maximum value becomes 255. The normalization parameters are extracted from the training set. Handling outliers is very simple all values greater than one-tenth of the maximum value in the training set, of a given band, will take the value 255. Let us define ‘rec’ as the maximum value, in the training set of a given band, therefore normalization is done as follows:

rec=rec/10;

Image=Image./(rec/255);

Image=uint8(Image); %values are changed to 8-bit unsigned integers, so values greater than 255 become 255.

2.3. CNN Ensemble Learning (EL)

The concept of EL is grounded in a straightforward principle: combining multiple models can lead to improved and more reliable results. Ensembles are most effective when the individual models exhibit significant diversity [28]. In this study, we construct ensembles by employing different techniques to represent input images as three-channel images. These processed images are used to train multiple networks, and their predictions are combined using the sum rule.

Convolutional Neural Networks (CNNs) were first introduced in the 1980s by French researcher Yann LeCun [29] for handwritten number classification and demonstrated strong performance throughout the 1990s [30, 31]. Over the past decade, advancements in big data and GPU computing have significantly enhanced CNN performance, establishing them as the state-of-the-art approach in computer vision and image recognition. CNNs process three-dimensional tensors, often used to represent images of specific dimensions, where the width, height, and number of channels (e.g., RGB color values, alpha transparency, and depth) define the input.

In this work, we utilize ResNet50 (Res) [32], DenseNet201 (DN) [33], and MobileNetV2 (MV2) [34] architectures, all pre-trained on ImageNet. These models are fine-tuned further over 20 epochs, with a learning rate of 0.001 and a batch size of 30, using the SGD optimization approach. A key advantage of using pre-trained networks is the application of transfer learning. This approach enables the model to leverage the knowledge acquired during prior training on one dataset and apply it to a new dataset [35]. This benefit is particularly valuable in DL models, which process vast arrays of weights and features. Transfer learning reduces both the training time and the quantity of data needed, making it ideal for relatively small datasets. In this study, all layers of the pre-trained networks are fine-tuned, with none kept frozen.

In this study, the fusion method employed is the sum rule [26], defined as:

$$sum=\mathrm{}\sum _{i=1}^N\mathrm{v}\mathrm{};\mathrm{}out=argmax\mathrm{}\left\{{sum}_j\right\},\mathrm{}j=\mathrm{}1\dots n$$

Here, N represents the number of models, and n denotes the size of each confidence vector v. The sum rule is considered one of the most effective fusion methods, as it avoids potentially harmful operations such as multiplication by zero.

2.4. 3-Channel Image Creation

Since we utilize pretrained networks that require RGB images as the input, the initial step in our pipeline involves converting the dataset into CNN-compatible inputs.

The GraySet method achieves this by transforming each of the sixteen grayscale images within a pattern into RGB images. This transformation is accomplished by replicating the grayscale values across all three color channels, resulting in sixteen RGB images for each training and testing pattern. Specifically, for a given pattern comprising sixteen grayscale images (denoted as ${\mathrm{I}}_1$ ... ${\mathrm{I}}_{16}$) , it is encoded into an RGB image following this rule:

• $\mathit{R}\left(\mathit{x}\right)={\mathbf{I}}_1,\mathit{G}\left(\mathit{x}\right)={\mathbf{I}}_1,\mathit{B}\left(\mathit{x}\right)={\mathbf{I}}_1$ becomes the first RGB image
• $\mathit{R}\left(\mathit{x}\right)={\mathbf{I}}_2,\mathit{G}\left(\mathit{x}\right)={\mathbf{I}}_2,\mathit{B}\left(\mathit{x}\right)={\mathbf{I}}_2$ becomes the second RGB image,…
• $\mathit{R}\left(\mathit{x}\right)={\mathbf{I}}_{16},\mathit{G}\left(\mathit{x}\right)={\mathbf{I}}_{16},\mathit{B}\left(\mathit{x}\right)={\mathbf{I}}_{16}$ becomes the last RGB image.

This transformation is applied consistently across both training and testing patterns. For each test pattern, the method generates sixteen RGB images, leading to sixteen scores produced by the trained CNN. These scores are then combined using the average rule to produce the final result.

In the Random method, the three channels are obtained simply by randomly extracting three channels from the multiband images.

In the RandomOneRGB method, in the case of the EuroSat dataset, the performance of the RGB channels is higher than the other channels, so the following procedure is used: two channels are randomly drawn among the thirteen bands, and the other channel is randomly drawn among the three R, G, and B bands.

We are generating images to work well with an ensemble. For this reason there are no contraints imposed on the random draw. Thus, it is possible for a network in our ensemble to be trained on multiband images transformed into RRR, that is on three red channels in RGB.

3. Conclusions

Although the experiments presented in this paper are focused on only two datasets (representing two very different problems), the impact of ensemble DL is clearly evident. Though classification using multiple CNNs proved to be time-intensive, our efforts directed toward optimizing accuracy were met with considerable success. As we show, our ensembling method for multiband images significantly improves classification performance compared to SOTA—such a level of performance positions the model as a valuable tool for the tasks at hand. The substantial improvement in key performance indicators, with respect to stand-alone networks, justifies the computational trade-offs, highlighting the effectiveness of a robust ensemble method over single neural networks for image classification tasks.

Future developments of this work will include modifications to the input layers of the pre-trained networks. For example, enabling variable-resolution images or adapting the input layers to handle multichannel inputs could significantly enhance flexibility. However, this would necessitate retraining specific layers to integrate effectively with the pre-trained backbone.