Comparison of Different Methods for Building Ensembles of Convolutional Neural Networks

1. Introduction

Artificial neural networks (ANNs), which were initially developed in the 1950s, have had a checkered history, at times appreciated for their unique computational capabilities and at other times disparaged for being no better than statistical methods. Opinions shifted about a decade ago with deep neural networks, whose performance swiftly overshadowed that of other learners across various scientific, medical, and engineering domains. The prowess of deep learners is especially exemplified by the remarkable achievements of Convolutional Neural Networks (CNNs), one of the most renowned and robust deep learning architectures.

CNNs have consistently outperformed other classifiers in numerous applications, particularly in image recognition competitions where they frequently emerge as winners [1]. Not only do CNNs surpass traditional classifiers, but they also often outperform the recognition abilities of human beings. In the medical field, for instance, CNNs have been shown to perform better than human experts in detecting skin cancer [2,3], skin lesions on the face and scalp, and esophageal cancer (e.g., [4]). These successes, unsurprisingly, have led to a surge in imaging research investigating CNNs and other deep learners. For example, in the medical field, deep learners have become state-of-the-art at diagnosing diabetic retinopathy [5], Alzheimer’s disease [6], skin detection [7], gastrointestinal ulcers, and a host of different cancers, as evidenced in recent reviews and studies.

CNNs, however, have limitations. It is widely recognized that they require many samples to avoid overfitting [8]. Acquiring image collections numbering in the hundreds of thousands for proper CNN training is an enormous enterprise [9]. In certain medical domains, it is prohibitively labor-intensive and costly [10]. Several well-established techniques have been developed to address the issue of overfitting with limited data, the two most common being transfer learning using pre-trained CNNs and data augmentation [11,12]. The literature is abundant with studies investigating both methods, and it has been observed that combining the two yields better results (e.g., [13]).

In addition to transfer learning and data augmentation, another powerful technique for enhancing the performance of deep learners generally, as well as on small sample sizes, is to construct ensembles of pre-trained CNNs [14]. Many robust CNN ensembles have been reported in recent years, showcasing their effectiveness in various applications. For instance, in [15], a deep CNN ensemble was developed for classifying ER (Estrogen Receptor) status from DCE-MRI (Dynamic Contrast-Enhanced Magnetic Resonance Imaging) breast volumes. In [16], the authors focused on diabetic muscular edema diagnosis and employed a hierarchical ensemble approach. In [17], a CNN ensemble was designed for whole-brain segmentation, while in [18] an ensemble approach for small lesion detection was proposed. In each of these cases, the ensemble was shown to perform better than standalone CNNs.

The success of ensemble learning involves combining the outputs of multiple classifiers [19] that introduce some degree of diversity, which can be achieved through various means [20]. One common approach is to train each CNN on different subsets of the available data [21]. Other approaches involve combining different types of CNN architectures [22], varying network depth [23,24], and introducing different activation functions [19]. Related work using these different approaches is addressed in Section 2.

This research paper focuses on image classification using ensembles of CNNs built with one network, ResNet50 [25], chosen because of its balanced tradeoff between performance and training time. The goal of this work is to perform an exhaustive evaluation across many data sets of the performance of combining three ensembling methods: 1) replacing ReLU layers with twenty different activation functions, 2) applying various data augmentation techniques, and 3) utilizing several methods to perturb network weights. Results of our experiments demonstrate that combining ensembles built with these methods produces superior results.

The contributions of this study include:

• An in-depth comparison and evaluation of three methods for building CNN ensembles, both standalone and in combination, verified across five different data sets;
• The introduction of several new methods for perturbing network weights;
• Free access to all resources, including the MATLAB source code, used in our experiments.

The remainder of this paper is organized as follows. Section 2 provides a review of the literature on building ensembles for CNNs. Full details about our ensemble are provided in Section 3. Section 4 presents and discusses the results of our experiments. Section 5 provides some final remarks and outlines research opportunities for the future.

2. Related Work

The related work in this field explores various strategies for creating ensembles of CNNs, with a focus on achieving high performance and maximizing the independence of predictions. Already addressed in the introduction are approaches based on training networks with different architectures and activation functions and using diverse training sets and data augmentation approaches for the same network architecture. In addition, ensembles can be generated by combining multiple pre-trained CNNs, employing various training algorithms, and applying distinct rules for combining networks.

The most intuitive approach to form a diverse ensemble involves training different models on the entire dataset. It can be challenging, however, to find models with similar performance, and employing a large number of models may only yield marginal improvements over the best model’s performance. Conversely, the predictions of high-performing models might exhibit strong correlations, leading to an ensemble that performs on par with a single model.

Most researchers taking this intuitive approach primarily fine-tune or train well-known architectures from scratch, average the results, and then demonstrate through experiments that the ensemble outperforms individual stand-alone networks. For instance, in [26], Kassani et al. employed an ensemble of VGG19 [27], MobileNet [28], and DenseNet [29] to classify histopathological biopsies, showing that the ensemble consistently achieved better performance than each individual network across four different datasets. Similarly, Qummar et al. [5] proposed an ensemble comprising ResNet50 [30], Inception v3 [31], Xception [32], DenseNet121, and DenseNet169 [29] to detect diabetic retinopathy.

In their study, Liu et al. [33] constructed an ensemble comprising three distinct CNNs proposed in their paper and averaged their results. Their ensemble achieved higher accuracy than the best individual model on the FER2013 dataset [34]. Similarly, Kumar et al. [35] introduced an ensemble of pretrained AlexNet and GoogleNet [36] models from ImageNet, which were then fine-tuned on the ImageCLEF 2016 collection dataset [37]. They utilized the features extracted from the last fully connected layers of these networks to train an SVM, an approach that outperformed CNN baselines and remained competitive with state-of-the-art methods at that time. Pandey et al. [38] proposed FoodNet, an ensemble composed of finetuned AlexNet, GoogleNet, and ResNet50 models designed for food image recognition. The output features from these models were concatenated and passed through a fully connected layer and softmax classifier.

Utilizing diverse training sets to train a classifier proves to be an effective approach in generating semi-independent classifiers. This can be achieved through various methods, with one classic technique being bagging [39,40,41]. Bagging involves creating m training sets of size n from a larger training set by randomly selecting samples with uniform probability and with replacement. Subsequently, the same model is trained on each of these training sets. Examples of this approach to building ensembles include the work of Kim et al. [42], who proposed a bagging-based approach to train three distinct CNNs for vehicle type classification. Similarly, Dong et al. [43] applied bagging and CNNs to improve short-term load forecasting in smart grid, resulting in a significant reduction of the mean absolute percentage error (MAPE) from 33.47 to 28.51. As another example, Guo et al. [44] employed eight different datasets to train eight distinct networks for object detection. These datasets were formed by combining existing datasets in various ways. Remarkably, this straightforward approach led to substantial performance gains compared to individual models and brought the ensembles’s performance close to the state-of-the-art on competitive datasets like COCO 2012.

The training algorithms employed in CNNs follow stochastic trajectories and operate on stochastic data batches. Consequently, training the same network multiple times may lead to different outcomes at the end of the process. To enhance the diversity among final models, they can be trained using distinct training algorithms. For instance, the authors in [45] constructed an ensemble for facial expression recognition using soft-label perturbation, where different losses were propagated for different samples. Similarly, Antiov et al. [46] utilized different network initializations to train multiple networks for gender predictions from face images.

Another approach to building ensembles is to adopt the same architecture but vary the activation functions. This can be done in a set of CNNs or within different layers of a single CNN [47]. One way to implement the latter approach is to select a random activation function from a pool for each layer in the original network [48]. The diversity introduced in this way makes activation functions an excellent candidate for generating ensembles of deep learners, as noted in [49], and is the tactic adopted in this work.

Finally, ensembles can vary in the selection of rules for merging results. A straightforward approach is majority voting, where the predominant output selected by the majority of the networks is taken [50,51,52,53]. Another common technique frequently cited in the literature is to average the softmax outputs of the networks [54,55,56]. More intricate methods have been proposed, such as that proposed in Lumini et al. [57], which calculates a learned weighted average of the softmax output.

3. Methods

The ensembles in this study are constructed using sets of ResNet50 architectures pre-trained on the ImageNet dataset, famous for its large scale. Since winning the ILSVRC 2015 contest, ResNet50 has gained in popularity and is well understood. It is particularly known for its skip connections, which allow the input of a block to be added to its output. This technique promotes gradient propagation and facilitates the flow of lower-level information to higher-level layers.

The training process in this work involves training each network with a batch size (BS) of 30 and a learning rate (LR) of 0.001 for 20 epochs. It is worth noting that the last fully connected layer has a learning rate 20 times larger than the rest of the layers. Ensemble decisions are combined using the average rule. This means that the softmax probabilities generated by each network in the ensemble for a given sample are averaged, resulting in a new score that is used for classification.

In the remainder of this section, we describe the methods we use to create ensembles of ResNet50.

3.1. Activation Functions

In this study, a set of more than twenty activation functions is investigated in building CNN ensembles. These activation functions include such well-known ones as ReLU, Leaky ReLU, ELU, SELU, PReLU, APLU, SReLU, MeLU, Splash, Mish, PDELU, Swish, Soft Learnable, etc. (the complete details/list of these activation functions can be found in [47]).

The main advantage of complex activation functions with learnable parameters is their ability to capture abstract features through nonlinear transformations, a characteristic commonly observed in shallow networks [58]. However, a potential drawback lies in their complexity: multiple learnable parameters require large data sets for training.

The stochastic approach mentioned in [47] is used to alter the activation functions in ResNet50. This method involves randomly replacing all activations within a network with a new activation function selected from a pool of potential candidates. The random selection process is repeated multiple times to generate a set of networks that are fused together in the ensemble. The performance of a pool of candidate activation functions varies depending on the specific CNN architecture. What this means is that some activation functions will perform poorly with ResNet50, while others will perform well. The result is significant variance among the ensemble members.

In the experimental section, the stochastic method of combining CNNs is referred to as "SE." It is important to note that the proposed ensemble approach does not pose a risk of overfitting. The replacement of activation functions is performed randomly without any ad hoc selection of specific datasets. Overfitting could potentially occur if the activation functions were chosen based on ad hoc data sets, but this is not the case in the proposed ensemble method.

3.2. Data Augmentation

During the training process, sets of networks are trained using different data augmentation techniques [11]. The following data augmentation methods have been utilized:

• APP1: This augmentation generates three new images based on a given image. It randomly reflects the image vertically and horizontally, resulting in two new images. The third transformation involves linearly scaling the original image along both axes with two factors randomly selected from a uniform distribution ranging from 1 to 2.
• APP2: Building upon APP1, this augmentation generates six new images. It includes the transformations of APP1 and adds three additional manipulations. First, image rotation is applied with a random angle extracted from the range of -10 to 10 degrees. Second, translation is performed by shifting the image along both axes with values randomly sampled from the interval of 0 to 5 pixels. Last, shear transformation is applied, with vertical and horizontal angles randomly selected from the range of 0 to 30 degrees.
• APP3: This augmentation replicates APP2 but excludes the shear and the scale transformations, resulting in four new images.
• APP4: This augmentation approach generates three new images by applying a transform based on Principal Component Analysis (PCA). The PCA coefficients extracted from a given image are subjected to three perturbations that generate three new images. For the first image, each element of the feature vector has a 50% probability of being randomly set to zero. For the second, noise is added to each component based on the standard deviation of the projected image. For the third, five images from the same class as the original image are selected, and their PCA vectors are computed. With a 5% probability, components from the original PCA vector are swapped with corresponding components from the other five PCA vectors. The three perturbed PCA vectors are then transformed back using the inverse PCA transform to produce the augmented images.
• APP5: Similar to APP4, this augmentation generates three new images using the perturbation method described above. However, instead of using PCA, the Discrete Cosine Transform (DCT) is applied. It should be noted that the DC coefficient is never changed during this transformation.
• APP6: This augmentation is designed specifically for color images. It creates three new images by color shifting and by altering contrast and sharpness. Contrast alteration is achieved by linearly scaling the original image’s contrast between the lowest value (a) and the highest value (b) allowed for the augmented image. Any pixel in the original image outside this range is mapped to 0 if it is lower than a or 255 if it is greater than b. Sharpness is modified by blurring the original image with a Gaussian filter (variance = 1) and subtracting the blurred image from the original. Color shifting is performed by applying integer shifts to the three RGB filters, and each shift is added to one of the three channels in the original image.

These data augmentation techniques aim to increase the diversity of the training data, helping the networks learn robust features and improve their performance on the classification task.

3.3. Parameter Ensembling via Perturbation (PEP)

Another approach for creating ensembles is PEP [59]. In this method, only a single network is trained, but the ensemble is formed by introducing perturbations to the weights of the final network using additive Gaussian random noise. The researchers who designed this method have demonstrated that by appropriately adjusting the amount of noise the performance of the original network can be surpassed. The determination of the optimal noise level can be accomplished, as in [59], through experimentation on a validation set.

In our evaluation, we have tested several variants of PEP, which include the original version and the following new ones proposed here:

• Dout: similar to drop-out: 2% of the weights zeroed out;
• DCTa: each set of weights is projected onto a Discrete Cosine Transform (DCT) space, with (3.33%) randomly chosen DCT coefficients set to zero (the DC component is never zeroed out), after which the inverse DCT is applied.
• DCTb: each set of weights is projected onto a Discrete Cosine Transform (DCT) space where a small amount of random noise is injected (the DC component is never perturbed), after which the inverse DCT is applied.
• PEPa: method similar to the original version, but where a small amount of random noise is injected.
• PEPb: the same idea as PEPa, but noise is injected in a different manner.

We apply these methods as follows. First, we train the network for 20 epochs to obtain netA. Next, we apply weight perturbation on netA, then train the network again for a single epoch. The resulting network is netP. Perturbation is performed five times, and each time the perturbation is applied to netA. In this way, we obtain five netPs (netP(1), netP(2), ..., netP(5)). The final output is given by the average rule between the output of netA and the five netPs.

Below is the pseudo-code of the proposed approaches.

Listing 1. Dout: similar to dropout filter

Perturbation = rand(size(Weights));     % Weights are the weights of the given net     % Perturbation is a tensor of the same size as the     % set of weights of the net, randomly initialized to [0,1]     Weights = Weights.*Perturbation;     % some weights are zeroed out

Listing 2. DCTa: DCT based perturbation approach

for each layer             for each channel                 IMG = Weights(layer,channel);                 % weights of a given channel-layer are stored                 dctProj = dct2(IMG);  % DCT projection                 dctProj_reset = dctProj;                 % reset some random dct coefficients                 dctProj_reset("random indexes") = 0;                 % DC component is never zeroed out                 dctProj_reset(1,1) = dctProj(1,1);                 Weights(layer,channel) = idct2(dctProj_reset); % retroprojection             end         end

Listing 3. DCTb: DCT based perturbation approach

for each layer         for each channel             IMG = Weights(layer,channel);             % weights of a given channel-layer are stored             dctProj = dct2(IMG);  % DCT projection             % standard deviation of the values of the weights             noise = std(dctProj) / 4;             % random noise             dctProjNew = dctProj + (rand-0.5) .* noise;             % rand is random between 0 and 1             dctProjNew(1,1) = dctProj(1,1); % DC component is never zeroed out             Weights(layer, channel) = idct2(dctProjNew); % retroprojection         end     end

Listing 4. PEPa: Method 3 is similar to DCT1

sigma = 0.002;     Weights = Weights + rand(size(Weights)) .* sigma;     % Weights are the weights of the given net

Listing 5. PEPb: Method 3 is similar to DCT1

sigma = 0.2;     Weights = Weights .* (1 + rand(size(Weights)) .* sigma);     % Weights are the weights of the given net

4. Experimental Results

In this section, we detail the experimental analysis of the ensemble methods.

4.1. Datasets

In our study, we utilized the following data sets to evaluate the performance of our approach:

• HE (2D HeLa data set [60]): This data set has 862 fluorescence microscopy images of HeLa cells stained with different fluorescent dyes specific to various organelles. The data set is well-balanced and divided into ten classes representing different organelles, including DNA (Nuclei), ER (Endoplasmic reticulum), Giantin (cis/medial Golgi), GPP130 (cis Golgi), Lamp2 (Lysosomes), Nucleolin (Nucleoli), Actin, TfR (Endosomes), Mitochondria, and Tubulin. A 5-fold cross-validation is applied.
• MA (C. elegans Muscle Age data set [61]): This data set focuses on classifying the age of C. elegans nematodes. It has 257 images of C. elegans muscles collected at four different ages, representing distinct classes based on age. A 5-fold cross-validation is applied.
• BG (Breast Grading Carcinoma [62]): This data set, obtained from Zenodo (record: 834910#.Wp1bQ-jOWUl), has 300 annotated histological images of breast tissues from patients diagnosed with invasive ductal carcinoma. The data set is categorized into three classes representing different grades (1-3) of carcinoma. A 5-fold cross-validation is applied.
• LAR (Laryngeal data-set [63]): Obtained from Zenodo (record: 1003200#.WdeQcnBx0nQ), has 1320 images of laryngeal tissues. It includes both healthy and early-stage cancerous tissues, representing a total of four tissue classes. This data set is split into three folds by the original authors.
• POR (portrait dataset) data set [64] focuses specifically on portrait images of humans. It is designed to evaluate segmentation performance in the context of portrait photography, considering factors such as facial features, skin tones, and background elements. This dataset includes 1447 images for training and 289 images for validation. POR can be accessed at https://github.com/HYOJINPARK/ExtPortraitSeg.

By utilizing these diverse data sets, we aimed to evaluate the performance of our ensembling approaches across various imaging tasks and scenarios (mostly medical), providing a comprehensive analysis of the different methods’ effectiveness on small data sets.

4.2. Results

In the first test reported in Table 1, we compare the performance of RE and SE both with APP3 data augmentation (DA) and without a data augmentation step (noDA). We do this by varying the number of classifiers that build the ensemble. In this table, SE(x) means that we combine by sum rule ’x’ networks coupled with the stochastic approach for replacing the activation function layers, and RE(x) means that we combine by sum rule ’x’ standard ResNet50, where each network is simply re-trained on the training set.

From the results reported in Table 1, we can draw the following conclusions:

• Data augmentation is useful both for RE and SE.
• Both RE and SE increase performance only slightly from x = 14 to x = 30.
• SE outperforms RE.

In Table 2, we compare the different data augmentation approaches. In this case, there is no clear winner. In each data set, the rank position of the single data augmentation method varies.

Examining Table 3, no clear winner is evident, even when we compare the different PEP-based approaches. However, in BG and LAR the new proposed DCTa outperforms both PEPa and PEPb. The most interesting result is that ALL (the fusion of the five PEP-based approaches) outperforms PEPa(5), a set of five PEPa methods (each obtained by retraining the networks). PEPa(5) always performs worse than ALL. Interestingly, ALL almost always improves RE(5)-noDA, implying that PEP-based methods are indeed an effective way to build network ensembles. The results of this experiment show that it is useful to apply different perturbation approaches to obtain a set of networks. It should be noted that for the sake of computation time we did not use DA in this test.

In Table 4 and Table 5, we compare different approaches for building an ensemble of ResNet50s. The methods are those explained above. We also report in this table the performance of:

• StocDA_PEP(18), which was created as follows: for each DA method, we train three networks for a total of eighteen. Each network is then coupled with one of the five PEP variants (randomly chosen).
• StocDA(18), which was created as follows: For each DA method, we train three SE networks for a total of eighteen.

In Table 4 the accuracy obtained by the compared methods is reported, instead, in Table 5 the error under the ROC curve (EUC) is reported, it is defined as (100-"Area under the ROC curve)%. In addition, we use three other datasets in the following tables for a more robust statistical comparison:

• PEST, [65], It is a dataset of 563 pest images, 10 classes, commonly found on plants. We use the split training-test sets suggested by the original authors.
• InfLAR, [66], it is a dataset of 720 images, four classes, extracted from laryngoscopic videos. We use the split training-test sets (three different folds) suggested by the original authors.
• TRIZ, [67], it is a dataset of 574 gastric lesion type images, four classes; as suggested by the original authors we apply a 10-fold cross-validation.

From Table 4 and Table 5, we can draw the following conclusions:

• In data sets with gray-level images, StocDA_PEP(18) and StocDA(18) are less useful (probably because some data augmentation methods are suitable for color images), whereas in LAR, POR, PEST, InfLAR, and TRIZ they fare better;
• There is a noticeable difference in performance between each ensemble compared to the original single ResNet50.
• considering the well-known Wilcoxon-signed rank test, considering both the performance indicators (i.e. accuracy and EUC): RE(18)-DA outperforms RE(1)-DA with a p-value of 0.01; SE(18)-DA outperforms RE(18)-DA with a p-value of 0.05; StocDA(18) obtains similar performance with respect to SE(18)-DA (due to the gray level images datasets); StocDA_PEP(18) outperforms StocDA(18) with a p-value of 0.05.

Finally, in Table 6, we report tests on computation time.

The hardware improvements, as clearly expected, reduce the inference time; there are several applications where it is not a problem to classify 100 images in just a few seconds.

5. Conclusion

The aim of this study was to evaluate the effectiveness of advanced ensemble deep learning techniques. Various methods for creating ensembles, specifically focused on CNNs, were investigated. The main intention of this work was on comparing the performance of standalone ensembles and various combinations of the following ensembling techniques: replacing ReLU layers with different activation functions, employing various data augmentation techniques, and utilizing several methods to perturb the network weights. To assess the performance of these ensembles, the well-known ResNet50 model was employed due to its balanced tradeoff between performance and training time. The evaluation was carried out on five challenging image datasets encompassing diverse tasks.

The experimental results demonstrated that the proposed ensemble of CNNs outperformed standalone methods for building CNN ensembles. However, additional research is necessary to investigate the performance benefits of this approach on a broader range of data sets, such as Computer Tomography (CT), Magnetic Resonance Imaging (MRI), and image/tumor segmentation. Experiments using different network topologies also need to be performed.

Conducting such investigations poses challenges, however, due to the substantial computational resources required for CNN analysis. Nevertheless, these studies are vital for improving the accuracy of deep learning systems in image and data classification tasks.