Enhanced Superpixel Guided ResNet Framework with Optimized Deep Weighted Averaging Based Feature Fusion for Lung Cancer Detection in Histopathological Images

1. Introduction

Cancer is a complex set of diseases marked by uncontrolled growth and spread [1], unlike benign tumors, which remain localized, whereas malignant tumors invade and damage nearby tissues. Lung cancer is the leading type in men and the third in women, closely linked to smoking, and is the primary contributor to cancer associated mortality globally [2]. The WHO projects cancer will become the top global cause of death by 2020 [3], with lung cancer alone causing around 1.80 million deaths. Projections indicate that by 2035, lung cancer might contribute up to 60% of all cancer-related fatalities [4]. Early-stage cancers that are operable have a 5-year survival rate of approximately 34%, but for inoperable cases, the rate drops to under 10%. Lung cancer, which is predominantly classified into non-small cell lung carcinoma (NSCLC) and small cell lung carcinoma (SCLC) [5], shows varying characteristics. NSCLC, making up about 85% of cases, includes adenocarcinoma (ADC), squamous cell carcinoma (SCC), and large cell carcinoma (LCC). The remaining 15% are SCLC cases.

Histopathological examination identifies lung cancer subtypes through biopsy reports [6], crucial for accurate diagnosis and effective treatment planning [7]. Computer-Aided Diagnosis (CAD) systems support pathologists by providing automated assessments to prevent misclassification [8]. Advances in artificial intelligence (AI) have enhanced both the precision and effectiveness of histopathological slide analysis. This study centers on categorizing lung cancer biopsy images into two distinct categories, adenocarcinoma and benign using deep learning frameworks.

1.1. Contribution of the Work

The Major Contribution of this research work can be summarized as follows:

1. Histopathological images are preprocessed using an adaptive fuzzy filter and segmented using the Modified SLIC algorithm.

2. The segmented images are passed through deep learning models such as ResNet-50, ResNet-101, and ResNet-152 for feature extraction, followed by a proposed deep weighted averaging feature fusion technique to generate RN-X features.

3. The extracted features from the ResNet models and RN-X are input to a Selective Feature Pooling Layer, which leverages PSO and RDO optimization algorithms for feature selection.

4. Finally, the Classification Layer implements the classifiers such as SVM, DT, RF, KNN, SDC, BLDC, and MLP, evaluated using K-fold cross-validation with K values of 2, 4, 5, 8, and 10.

This study is organized as follows: Section 2 provides a review of recent research on lung cancer detection. Section 3 presents the methodology. Section 4 details the proposed deep weighted averaging feature fusion technique and discusses the Selective Feature Pooling Layer, incorporating PSO and RDO methods, along with the classification layer. Section 5 focuses on result comparisons. Finally, Section 6 highlights key findings and suggests directions for future research.

2. Review of Lung Cancer Detection

Over recent decades, various approaches have been proposed for automated detection, segmentation, and classification in histopathological images using machine learning (ML) and deep learning (DL). Anthimopoulos M et al. [9] developed a CNN with five convolutional layers using Leaky ReLU activation, average pooling, and three fully connected layers. Lizuka et al. [10] combined Inception v3 and an RNN to classify stomach and colon biopsies, incorporating regularization and augmentation for robustness. Wang et al. [11] used a CNN for lung cancer pathology, achieving 90.1% accuracy with Softmax activation. Gessert N et al. [12] explored multiresolution EfficientNet for skin sore classification. Liu Y et al. [13] applied wavelet-based denoising to address noise in histopathological images, achieving 94.37% accuracy on the BreakHis dataset.

Zhou Y et al. [14] designed a hierarchical model using SVM and SURF features, achieving 91.14% accuracy, but performance at 400X magnification needs improvement. Wang et al. [15] introduced FE-BkCapsNet, combining CNNs with CapsNet, yielding up to 94.52% accuracy. Aresta et al. [16] used DenseNet121, achieving 87% accuracy on the BACH 2018 dataset. Spanhol et al. [17] combined CNN predictions, achieving 84% accuracy on BreaKHis at 200X magnification, while Filipczuk et al. [18] focused on nuclei segmentation and trained several classifiers using 25 shape and texture features.

Nada Mobarak et al. [19] created CoroNet, a CNN based on Xception, achieving 88.67% accuracy for breast cancer detection on the CBIS-DDSM dataset. Teresa et al. [20] applied CNN models on the Bioimaging2015 dataset, segmenting images into 512×512-pixel patches, achieving up to 83.3% accuracy. Ahsan Rafiq et al. [21] proposed a three-CNN model for breast cancer classification, achieving 90.10% accuracy. Hameed et al. [22] fine-tuned VGG models and used an ensemble approach, outperforming individual models. Wang P et al. [23] achieved 96.19% accuracy using wavelet transforms and SVM with a genetic algorithm for feature selection.

3. Materials and Methods

This section offers an in-depth overview of the resources and methodologies employed in the classification of lung and colon cancers. The methodological framework of this study is illustrated in Figure 1.

3.1. Dataset Used

The LC25000 dataset, introduced in 2020 [24], contains 25,000 color images of five tissue types, expanded through augmentation from an original 1,250 cancerous images. Images were resized to 768 × 768 pixels and verified for HIPAA compliance. This study focuses on 5,000 benign and 5,000 adenocarcinoma lung cancer images. Adenocarcinoma originates in glandular cells and often spreads to the alveoli. Benign tissues, while non-cancerous, typically require surgical removal and biopsy to confirm their nature.

3.2. Image Preprocessing

Histopathological image analysis is crucial for assessing tumor characteristics, clinical staging, and predicting patient survival [25]. However, these images face challenges such as Complex Geometric Patterns and Textures, Critical Textural Features, Image Dimension and Resolution Variations, and Colour and Noise Issues. This study demonstrates that applying an adaptive fuzzy filter to resized images (224 × 224) enhances clarity by reducing noise and artifacts, resulting in more accurate diagnoses. The filtered images are then used for segmentation of the region of interest (ROI).

3.3. Modified SLIC Algorithm-Based Segmentation

A super pixel groups adjacent pixels that exhibit similar color, luminance, and texture properties to segment an image [26]. The SLIC algorithm allocates M initial seed points uniformly throughout the image. For an image with N pixels segmented into M super pixels, each super pixel contains N/M pixels. The separation between neighboring seed points is $S=\sqrt{N/M}$. The feature vector of the centroid is $C_i={\left[l_i,a_i,b_i,x_i,y_i\right]}^T$, combining the CIELAB color values $\left[l_i,a_i,b_i\right]$ and the pixel position $\left(x_i,y_i\right)$. To enhance segmentation, the SLIC algorithm adjusts each centroid to the point with the minimal gradient in a 3x3 neighborhood. After initialization, it iteratively clusters pixels by assigning them to the nearest center and computing distances within a 2S x 2S neighborhood of each center. In SLIC algorithm, the measure of proximity between a candidate pixel and the centroid of a cluster is expressed as,

$$d_{ss}\left(i,j\right)=\sqrt{{\left(x_i-x_j\right)}^2+{\left(y_i-y_j\right)}^2}$$

------(1)

$$d_{cs}\left(i,j\right)=\sqrt{{\left(l_i-l_j\right)}^2{\left(a_i-a_j\right)}^2+{\left(b_i-b_j\right)}^2}$$

------(2)

$$d_{ts}\left(i,j\right)=\sqrt{d_{ss}^2+\alpha d_{cs}^2}$$

------(3)

Here, $i$ denotes the centroid label, and $j$ denotes the pixel index in the 2S×2S neighborhood. $d_{ss}$ represents spatial similarity, $d_{cs}$ represents color similarity, and $d_{ts}$ is the total similarity with a lower $d_{ts}$ signifying higher similarity. The parameter $\alpha =s/m$, where $s$ denotes neighborhood size and $m$ indicates compact factor balancing $d_{ss}$ and $d_{cs}$, typically ranges from 1 to 40. This paper introduces a modified SLIC algorithm that simplifies calculations using a 3-dimensional feature vector consisting of spatial co-ordinates $\left(x,y\right)$ and grayscale feature ($gs)$. The distance between a candidate pixel and the cluster centroid is revised as follows:

$$d_{gs}\left(i,j\right)=\sqrt{{\left(g_i-g_j\right)}^2}$$

------(4)

$$d_{ts}^{\prime }\left(i,j\right)=\sqrt{d_s^2+\alpha d_{gs}^2}$$

------(5)

Where $d_{gs}$ denotes pixel similarity in grayscale values, $d_{ts}^{\prime }$ represents the overall similarity between the cluster centroid and the pixel co-ordinates. The algorithm of the modified SLIC Super pixel Segmentation as follows:

The microscopic color cell image is initially transformed into a grayscale format. It is then randomly split into $K$ segments. Given the grayscale probability distribution $p_0,p_1,...,p_{n-1}$, and multiple thresholds $t_1,t_2,...,t_k$ (where $t_1<t_2<...<t_k$), the entropy for these segments can be expressed as:

$$\begin{gathered} \phi \left(t_1,t_2,\dots ,t_k\right)=\mathit{log}\left({\sum }_{i=0}^{t_1}{p}_i\right)+\mathit{log}\left({\sum }_{i=t_1+1}^{t_2}{p}_i\right)+\dots +\mathit{log}\left({\sum }_{i=t_k+1}^n{p}_i\right) \\ -{\displaystyle \frac{{\sum }_{i=0}^{t_1}{p}_i\mathit{log}{p}_i}{{\sum }_{i=0}^{t_1}{p}_i}}-{\displaystyle \frac{{\sum }_{i=t_1+1}^{t_2}{p}_i\mathit{log}{p}_i}{{\sum }_{i=t_1+1}^{t_1}{p}_i}}-...-{\displaystyle \frac{{\sum }_{i=s_k+1}^n{p}_i\mathit{log}{p}_i}{{\sum }_{i=t_k+1}^{t_1}{p}_i}} \end{gathered}$$

------(6)

The multiple thresholds $t_1,t_2,...,t_k$ for ideal classification for each segment adhere to the principle of maximum entropy, as follows:

$$\left(t_1,t_2,...,t_k\right)=\mathit{arg}\left\{\underset{t_1,t_2,...,t_k}{max}\left[\phi \left(t_1,t_2,...,t_k\right)\right]\right\}$$

------(7)

These thresholds are determined using a conditional iteration algorithm.

Using the optimal thresholds, the grayscale image is divided into $K+1$ intervals: $\left[X_i,X_{i+1}\right]$, $X_i\in \left\{t_1,t_2,...,t_k\right\}$, $i\in \left\{\mathrm{1,2},...,k\right\}$. Transform each interval $\left[X_i,X_{i+1}\right]$ to $\left[Y_i,Y_{i+1}\right]$ with a contrast-enhancing function $f\left(x\right)\circ f\left(x\right)$. The function is convex in $\left[X_i,X_m\right]$ and concave in $\left[X_m,X_{i+1}\right]$, with turning point $\left[X_m,Y_m\right]$, where $Y_m=\left(Y_i+Y_{i+1}\right)/2$. $X_m$ is determined by using the least squares principle:

$$X_m={\displaystyle \frac{{\int }_{X_i}^{X_i}xp\left(x\right)dx}{{\int }_{X_i}^{X_i}p\left(x\right)dx}}$$

------(8)

To simplify image processing, grayscale transformation is modeled by the function:

$$f\left(x\right)=a{x}^r+b,x\ge 1,r\ge 1$$

------(9)

Here, $a=\left(Y_{i+1}-Y_i\right)/\left(X_{i+1}^r-X_i^r\right),b=Y_i-a{X}_i^r$. Varying $r$, generates different transformation curves. A higher $r$, improves gray equalization in the interval $\left[X_i,X_{i+1}\right]$. By choosing appropriate $Y_i, i\in \left\{\mathrm{1,2},...,n\right\}$ and $r$ values regional balance and contrast can be enhanced, leading to a more evenly adjusted and contrasted image.

Initialize clustering centers $C_i$ using grid super pixels with side length $S=\sqrt{N/M}$, and assign labels.

Move each center $C_i$ to the location with the minimum gradient within its 3 x 3 neighborhood.

Calculate the similarity distance $d^{\prime }$ from each pixel $j$ to $C_i$ within a radius S which matches the circular shape of the cell image.

Set $dist\left(i\right)=\infty$. If $d^{\prime }\left(i\right)<dist\left(i\right)$ and is within range, update $d^{\prime }\left(i\right)$ to $dist\left(i\right)$, and assign the label $i$ to pixel $j$.

Repeat steps 4 to 6 until clustering converges. Recalculate each cluster's mean grayscale and spatial features to update centers.

Merge isolated small super pixels using an adjacent merging strategy for improved fit and coherence.

Figure 2 shows the image progression: from the original to the filtered image, followed by Original SLIC Superpixel Segmentation, Modified SLIC Superpixel Segmentation, and finally, the Modified SLIC result for the Adenocarcinoma Class (ACA).

4. Deep Feature Extraction

Deep learning networks are powerful but face challenges like saturation, accuracy degradation, and vanishing or exploding gradients. Architectures like ResNet-50 (RN-50), ResNet-101 (RN-101), and ResNet-152 (RN-152) address these issues using residual learning and identity mapping [27]. It uses shortcut connections that helps mitigate the vanishing gradient problem and prevents overfitting [28]. The mapping function as shown in Figure 3 is expressed as:

$$W\left(x\right)=F\left(x\right)+x$$

------(10)

In Table 1, A, B, C, and D represent the number of blocks in the first, second, third, and fourth stages of the ResNet versions, respectively.

The ResNet architecture configurations consist of different stages that are stacked across various ResNet versions, resulting in a 1D feature vector with 2048 elements for each image as shown in Figure 4.

4.1. Proposed DWAFF Technique for ResNet-X Features

This study proposes a Deep Weighted Averaging based Feature Fusion (DWAFF) technique. In this method, ResNet variants are evaluated, and weights are assigned to their feature vectors based on performance. By prioritizing contributions from each architecture, weights (ranging from 0 to 1) are adjusted in increments of 0.1 through trial and error. The final fused feature set for each image is computed using the weighted sum of the features as follows:

$$ResNet\_{\mathrm{X}}_{\mathrm{feature}}=w_1\times ResNet\text{\_}152_{\mathrm{feature}}+w_2\times ResNet\text{\_}101_{\mathrm{feature} }+w_3\times ResNet\text{\_}50_{\mathrm{feature} }$$

(11)

The optimal weight combination for feature fusion was determined using K-fold Cross Validation on the dataset for each architecture - RN-50, RN-101, and RN-152, across various K values of 2, 4, 5, 8, and 10, as summarized in Table 7. Among these, ResNet 152 demonstrated highest accuracy followed by ResNet 101 and ResNet 50. The weight values for the architecture were chosen through a trail-and-error method, constrained to lie between 0 and 1, with their sum equal to 1. The best performing combination was identified as 0.45 for RN-152 ($w_1$), 0.35 for RN-101 ($w_2$), and 0.20 for RN-50 ($w_3$). These weights were subsequently applied to fuse features using Equation 11. Additionally, the mean value of the normal class is added to the normal features, and the mean value of the abnormal class is added to the abnormal features, enhancing class separation, and improving classification. The equation for generating DWAFF-based RN-X features is given by:

For Normal cases,

$$ResNet\_{\mathrm{X}}_{\mathrm{feature} \left(i,j\right)}={\displaystyle \frac{\left(w_1\times ResNet\text{\_}152_{\mathrm{feature} \left(i,j\right)}+w_2\times ResNet\text{\_}101_{\mathrm{feature} \left(i,j\right)}+w_3\times ResNet\text{\_}50_{\mathrm{feature} \left(i,j\right)}\right)}{3}}+mean\_normal$$

(12)

For Abnormal cases,

$$ResNet\_{\mathrm{X}}_{\mathrm{feature} \left(i,j\right)}={\displaystyle \frac{\left(w_1\times ResNet\text{\_}152_{\mathrm{feature} \left(i,j\right)}+w_2\times ResNet\text{\_}101_{\mathrm{feature} \left(i,j\right)}{w}_3\times ResNet\text{\_}50_{\mathrm{feature} \left(i,j\right)}\right)}{3}}+mean\_abnormal$$

(13)

In this context, $i \in 0 to 2047$, denotes the features extracted per image, $j$ represents the image index, $j \in 0 to 4999$for the normal class and $5000 to 9999$for the abnormal class, $mean\_normal$ is the average of mean values from all three ResNet variants for normal images, while $mean\_abnormal$ represents the same abnormal images. The algorithm for the proposed DWAFF technique for ResNet X features is shown below:

Algorithm 1

Step 01: Extract Features

Extract feature vectors for each image from ResNet-50, ResNet-101, and ResNet-152.

Store the feature vectors: ResNet-50_feature [i, j], ResNet-101_feature [i, j], and ResNet-152_feature [i, j], where $i\in \left[\mathrm{0,2047}\right]$ and $j\in \left[\mathrm{0,9999}\right]$.

Step 02: Perform K-Fold Cross Validation

For each K value (2, 4, 5, 8, 10):

Split the dataset into K folds.

Train the classifiers on the dataset set split.

Evaluate performance of the classifiers using performance metrics.

Step 03: Set Initial Weight Range

Initialize a range of possible weights $w_1$, $w_2,$and $w_3$ based on the trial-and-error method, such that their sum must be equal to 1 according to the results obtained in Table 7 after K-fold Cross Validation.

Step 04: Identify Optimal Weights

For each weight combination, calculate the average performance across the K-folds.

Select the weight combination that achieves the highest average performance.

Optimal weights are identified as, 0.45 for ResNet-152 ($w_1$), 0.35 for ResNet-101 ($w_2$), and 0.20 for ResNet-50 ($w_3$).

Step 05: Compute Mean Values

Compute $mean\_normal$ and $mean\_abnormal$, across all three ResNet variants.

Step 06: Fuse Features for Final Feature Set

For Normal cases ($j\in \left[\mathrm{0,4999}\right]$): Fuse features of normal cases, Using Equation 12.

For Abnormal cases ($j\in \left[\mathrm{5000,9999}\right]$): Fuse features of abnormal cases, Using Equation 13.

Step 07: Output Final Fused Features

The final fused feature set for both normal and abnormal cases, which is a ResNet-X features are used for subsequent classification tasks.

4.2. Statistical Analysis

To enhance cancer classification accuracy with a reduced number of features, statistical measures play a crucial role in further analysis. The extracted features from ResNet-50, ResNet-101, ResNet-152, and the fused features from ResNet-X are analyzed by calculating statistical metrics such as Mean, Variance, Skewness, Kurtosis, Pearson Correlation Coefficient (PCC), and Canonical Correlation Analysis (CCA). These measures help assess how effectively the features capture lung cancer characteristics in both cancerous and non-cancerous data.

Table 2 presents the statistical parameters for the ResNet-50, ResNet-101, ResNet-152, and DWAFF-ResNet-X architectures for Normal (N) and Abnormal (ACA) cases. DWAFF-ResNet-X shows the best average performance with the highest mean values for both N (0.453891) and ACA (0.453709), outperforming the ResNet models, whose performance improves with depth. In terms of variance, DWAFF-ResNet-X has the lowest values (N: 0.380702, ACA: 0.444597), indicating more consistent performance compared to the higher variances in ResNet models, especially in ACA. For skewness, DWAFF-ResNet-X (N: 3.767961, ACA: 4.486885) shows a more symmetrical performance distribution, whereas ResNet models have higher skewness, indicating more inconsistent results. Kurtosis is also lower for DWAFF-ResNet-X (N: 21.14865, ACA: 33.4781), reflecting fewer extreme outliers than ResNet models, which have higher kurtosis values. PCC is highest in DWAFF-ResNet-X (N: 0.938638, ACA: 0.944338), indicating stronger alignment between predictions and outcomes compared to ResNet models. CCA also improves with model depth, with DWAFF-ResNet-X showing the highest CCA for ACA (0.8816). The Dice Coefficient values indicate the performance of the models in segmentation tasks. ResNet-50 shows moderate performance, with slightly better accuracy for normal cases. ResNet-101 outperforms ResNet-50, especially for normal cases. ResNet-152 demonstrates significant improvement, achieving higher accuracy for both normal and abnormal cases. DWAFF-ResNet-X delivers the best performance, with the highest Dice Coefficients for both normal and abnormal cases, making it the most effective model.

Figure 5 shows a scatterplot matrix of features extracted from ResNet-50, ResNet-101, ResNet-152, and ResNet-X. Normal and abnormal features are represented as RN-50-N/RN-50-ACA, RN-101-N/RN-101-ACA, RN-152-N/RN-152-ACA, and RN-X-N/RN-X-ACA, respectively. The scatterplots reveal nonlinear relationships between normal and abnormal features, with dense clusters near one axis and sparser distributions elsewhere. Significant overlaps between the classes are evident, particularly in ResNet-50 and ResNet-101, complicating classification. However, ResNet-152 and ResNet-X show better class separation, with abnormal features appearing more dispersed and normal features more clustered. This improved separation suggests that ResNet-X and ResNet-152 features may support more accurate classification. While overlaps persist across all models, careful feature selection could enhance classification performance, particularly with nonlinear classifiers.

The violin plot shown in Figure 6 illustrates the data distributions of features extracted by ResNet models (ResNet-50, ResNet-101, ResNet-152, and ResNet-X) for normal (N) and abnormal (ACA) classes. ResNet-50 shows minimal variation and significant overlap, indicating poor class differentiation. ResNet-101 captures more variation but still has considerable overlap, limiting separability. ResNet-152 demonstrates distinct distributions with reduced overlap, suggesting better feature separability. ResNet-X exhibits the widest distributions and more pronounced separation, making it the most promising for classification tasks. Feature differentiation improves progressively from ResNet-50 to ResNet-X.

4.3. Selective Feature Pooling Layer

The Selective Feature Pooling Layer is designed to condense the features of histopathological images into compact feature vectors, enhancing classifier performance and promoting high generalization capability. In lung cancer diagnosis [29], these techniques enhance accuracy by using Bio-inspired Optimization algorithms like PSO and RDO.

4.3.1. Particle Swarm Optimization (PSO)

The Particle Swarm Optimization (PSO), first proposed by Kennedy and Eberhart in 1995, mimics bird flock behavior to optimize problems. It begins by initializing particles and defining essential parameters [30]. The algorithm for the PSO with particle position and velocity updates are given below.

Algorithm 2: PSO

Initialization:

- Maximum iteration count: $k_{max}$- Inertia weight range: ($w_{min}$, $w_{max}$)

- Acceleration coefficients: c1, c2

- Set the position of each particle randomly: $p_i^k=\left(p_{i1}^k,p_{i2}^k,.....,p_{ix}^k\right)$ ------ (14)

- Set the velocity of each particle randomly: $q_i^k=\left(q_{i1}^k,q_{i2}^k,.....,q_{iy}^k\right)$ ------ (15)

- Initialize best position for an individual particle as $pbes{t}_i=p_i^k$and $gbest$ = the best of all $pbes{t}_i$.

for k = 0 to k_max-1 do:

    for i = 1 to n do:

        Calculate the inertia weight: $w_i={\displaystyle \frac{w_{max}-w_{min}}{k_{max}}}\times k$ ------ (16)

        Update the velocity: $q_i^{k+1}=w_i{q}_i^k+c_1{r}_1\left(pbes{t}_i-p_i^k\right)+c_2{r}_2\left(gbes{t}_i-p_i^k\right)$ ------ (17)

        Update the position: $p_i^{k+1}=p_i^k+q_i^{k+1}$ ------ (14)

        Update $pbes{t}_i$ if the new position surpasses the previous $pbes{t}_i$        Update $gbest$ if the new $pbes{t}_i$ surpasses the current $gbest$Output the final $gbest$ as the optimal solution.

In this study, the following parameter values are selected through an iterative process of experimentation and refinement: Inertia weight (wi) - between 0.45 and 0.9, Maximum number of iterations - between 100 and 1000, Random values (r1 and r2) - set to 0.85, Cognitive component (c1) and social component (c2) - between 1.0 and 2.0.

4.3.2. Red Deer Optimization (RDO)

Red Deer Optimization (RDA), introduced in 2016 [31], emulates the courtship rituals of Scottish Red Deer. The algorithm starts with an initial population of "red deer" (RDs). The best RDs, called "RD males," are split into "commanders" and "stags" based on their initial performance. Commanders and stags compete for harems, with successful stags potentially becoming commanders. Commanders pair with hinds in their harems and others, while stags mate with nearby hinds. This process blends exploration and exploitation, generating new solutions and allowing weaker solutions to evolve. In terms of dimensionality reduction, RDA uses this evolutionary process to refine and optimize the solution space by iteratively improving and filtering candidate solutions. The process for RDO follows here:

Algorithm 3: RDO

Initial Population:

-Define the solution space with dimensions:

$$Value=f\left(RedDeer\right)=f\left(S_1,S_2,S_3,....,S_{N_{var}}\left(\right)\right)$$

------(18)

Here, $S_{N_{var}}$ represents the array size, set to 50. Each component $S_i$ corresponds to a vector of values for each of the 50 images, as defined by the equation below.

$$S_i=\left[{\theta }_1,{\theta }_2,{\theta }_3,\dots ,{\theta }_{50}\right]=for i=\mathrm{1,2},3,....,S_{N_{var}}$$

------(19)

-Initialize a random population of red deer (RDs).

Roaring Stage:

-For each male RD:

--Calculate the new position based on fitness function (FF) value using.

$$Mal{e}_{new}=\left\{\begin{array}{c}Mal{e}_{old}+a_1\times \left(\left(\left(UL-LL\right)\ast a_2\right)+LL\right),if{a}_3\ge 0.5\\ Mal{e}_{old}-a_1\times \left(\left(\left(UL-LL\right)\ast a_2\right)+LL\right),if{a}_3<0.5\end{array}\right\}$$

------(20)

Here, UL and LL represent the maximum and minimum boundaries of the search region, respectively. The factors a1, a2 and a3 are randomly selected from a uniform distribution between zero and one.

--Update the RD position and evaluate its fitness.

--Promote successful RDs to commander status if they show improved fitness.

Competition Stage:

-Each commander competes with random stags:

--Compute new positions:

$$New1={\displaystyle \frac{\left(Com+Stag\right)}{2}}+b_1\times \left(\left(\left(UL-LL\right)\ast b_2\right)+LL\right)$$

------(21)

$$New2={\displaystyle \frac{\left(Com+Stag\right)}{2}}-b_1\times \left(\left(\left(UL-LL\right)\ast b_2\right)+LL\right)$$

------(22)

--Select the position with the best fitness function (FF) to update the commander status.

Harem Creation Phase:

-For harems with:

--A commander and several hinds based on the commander's fitness.

--Calculate the number of hinds as:

$$N.hare{m}_n=round\left\{P_n\cdot N_{hind}\right\}$$

------(23)

--Stags do not participate in harems.

Mating Phase:

-Commander Mating Within Harems: Each commander mates with a proportion ($\alpha$) of its hinds

-Commander Expansion Beyond Harems: Commanders mate with a percentage ($\beta$) of hinds from other harems. The parameter ($\beta$) ranges from 0 to 1.

-Stag Mating: Stags mate with the closest hind.

Offspring Creation:

-Generate new offspring using:

$$offspring={\displaystyle \frac{\left(Com+Hind\right)}{2}}+\left(UL-LL\right)\ast c$$

------(24)

Where $offspring$ is the new offspring RD, c is randomly chosen between 0 and 1. For -Stage mating, replace Com with Stag.

Next Generation Solution:

-Retain a percentage of the best male RDs.

-Select hinds and offspring for the next generation using fitness-based methods.

Stopping Criterion:

-RDO's stopping criteria include:

1. Fixed number of iterations. 2. Achievement of a quality threshold. 3. Exceeding a time limit.

The parameters of the RDO algorithm are described in the following Table 3.

4.4. Entropy Based on Statistical Analysis

In biomedical applications, entropy has emerged as a widely used approach. When applied to feature selection, entropy-based techniques assess the relevance and significance of selected features by quantifying the amount of information each feature contributes to predicting the target variable. In this study, the selected features from the normal and abnormal classes are evaluated using Approximate Entropy, Shannon Entropy, and Fuzzy Entropy.

4.4.1. Approximate Entropy

Approximate Entropy is a statistical method for measuring the regularity and unpredictability of variations in time-series data [32]. It calculates the difference between the natural logarithms of repeating patterns of length n and n+1, using the formula:

$$ApproximateEntropy\left(AE\right)=\mathit{ln}\left({\displaystyle \frac{b_n\left(r\right)}{b_{n+1}\left(r\right)}}\right)$$

------(25)

Here, $n$ is the input feature length and $b_n\left(r\right)$ is the mean of all $b_{in}\left(r\right)$ ranges. $b_{in}\left(r\right)$ is given as,

$$b_{in}\left(r\right)={\displaystyle \frac{m_{in}\left(r\right)}{M-n+1}}$$

------(26)

In the input vector $V_m$ of length $\left[V_m\right(1),V_m(2),...,V_m(M-n+1\left)\right]$, $b_{in}\left(r\right)$ represents the number of features. A higher approximate entropy value indicates that the input feature vectors are more complex and less predictable.

4.4.2. Shannon Entropy

The Shannon Entropy of a random variable S containing values $s_1,s_2,....,s_m$ is determined by,

$$ShannonEntropy\left(SE\right)=-{\sum }_{n=1}^{m}p\left(s_m\right).\mathit{log}p(s_m)$$

------(27)

Here, $p\left(s_m\right)$ represent the $s_m$ probability function. If the entropy score is high, it means that the outcome is hard to predict because it is uncertain.

4.4.3. Fuzzy Entropy

Fuzzy Entropy, a statistical method used to quantify the uniformity of input feature vectors [33]. It is defined by the following formula:

$$FuzzyEntropy\left(FE\right)=\mathit{ln}\left({\displaystyle \frac{{\varphi }^n}{{\varphi }^{n+1}}}\right)$$

------(28)

Here, ${\varphi }^n={\displaystyle \frac{1}{M-n}}{\sum }_{p=1}^{M-n}{\sum }_{q=1,q\ne p}^{M-n}{\displaystyle \frac{F_{pq}^n}{M-n-1}}$, and $F_{pq}^n$ represents the membership value of the fuzzy set and M is the total number of data points.

Table 4 compares the feature selection methods PSO and RDO in terms of entropy based statistical parameters such as Approximate Entropy, Shannon Entropy, and Fuzzy Entropy. Approximate Entropy measures the regularity and predictability of time-series data. PSO, with lower Approximate Entropy values (1.2385 for N and 1.7816 for ACA) compared to RDO (2.0123 for N and 2.4893 for ACA), indicates more regularity and less complexity. RDO, with higher values, suggests greater variability and less predictability, possibly capturing more nuanced features. Shannon Entropy reflects uncertainty or information content. RDO's higher values (5.0821 for N and 5.8982 for ACA) show greater complexity and feature diversity compared to PSO's lower values (3.8523 for N and 4.9891 for ACA), which suggest more structured data but less feature variety. Fuzzy Entropy, which measures complexity in a fuzzy system, is higher for RDO (0.7283 for N and 0.9182 for ACA), indicating more ambiguity in feature relationships. PSO's lower values (0.4862 for N and 0.5231 for ACA) suggest clearer, better-defined relationships with less uncertainty.

4.5. Classification Layer

Classifiers are essential for categorizing data, aiming for high accuracy and minimal errors while balancing computational complexity. This study utilized the following classifiers in the classification layer part of the ResNet architectures:

4.5.1. Support Vector Machine (SVM)

SVM is a set of supervised learning methods utilized for categorization, prediction, and anomaly detection, due to its scalability and high performance [34]. Linear SVMs use a maximum-margin hyperplane (either hard or soft margin), while non-linear SVMs apply kernel functions for classification. The Hyperplane is determined by,

$$Minimize,{\displaystyle \frac{1}{2}}{‖w‖}^2+\mathrm{M}{\sum }_{j=1}^n{\mu }_j$$

------(29)

Subject to $z_j{\left(w^N{x}_j+f\right)}^3\ge 1-{\mu }_j,{\mu }_j\ge 0.$ Here $w$ is the vector that is perpendicular to the hyperplane, $x_j$ is a data point, $f\in R$ is a scalar, and ${\mu }_j$ are slack variables penalizing misclassifications. The decision function is $w^N{x}_j+f$. Various kernels such as linear, polynomial, RBF, and sigmoid are used in SVMs. This study uses the SVM-RBF kernel to enhance classification accuracy.

4.5.2. Decision Tree (DT)

A Decision Tree (DT) is a flexible algorithm for categorization and regression, using a tree structure with decision nodes based on features and leaf nodes for outcomes [34]. Starting at the root, the tree is traversed to make predictions. Nodes split data by feature and threshold, while leaf nodes provide final predictions. Key metrics for node impurity include:

$$InformationGain(I,F)=Entropy\left(I\right)-{\sum }_{v\in values\left(F\right)}{\displaystyle \frac{\left|I_v\right|}{I}}\ast Entropy$$

------(30)

where $F$ represents the feature, $I$ denotes the collection of instances at the node, and $I_v$ is the subset of instances with feature $F$ has the value $v$. Gini Impurity is given as follows:

$$Gini\left(p\right)=1-{\sum }_{k=1}^C(p_k{)}^2$$

------(31)

where $p_k$ is the frequency of class k and $C$ is the number of classes. The objective is to find the feature and threshold that minimize impurity, with the optimal split S given by:

$$S\left(I\right)={argmax}_{x,t} impurity\left(I\right)-{\sum }_{v\in values\left(F\right)}{\displaystyle \frac{\left|I_v\right|}{I}}\ast impurity\left(I_v\right)$$

------(32)

4.5.3. Random Forest (RF)

The Random Forest algorithm excels in image classification due to its accuracy and robustness [35]. It uses multiple independent decision trees, with key parameters including the number of trees and features considered by each tree. The final prediction is made by combining the decision from all trees, with the formula:

$$f={\mathit{argmax}}_k{\sum }_{j=1}^D{p}_j=l$$

------(33)

where $f$ is the final prediction, D is the number of trees, $p_j$ is the prediction from the $j^{th}$ tree, and $l$ is the class label.

4.5.4. K-Nearest Neighbor (KNN)

The KNN algorithm determines the category of a data point by comparing its distance to the K closest points in the training data and assigns it to the class that appears most frequently among these neighboring points. It requires no separate training phase and uses the entire dataset for classification. In Weighted KNN, neighbors are weighted inversely to their distance from the query point [36]. The distance between two points $Z_1=(z_{11},z_{12},.....,z_{1n})$ and $Z_2=(z_{21},z_{22},.....,z_{2n})$ , is calculated as:

$$dist(Z_1,Z_2)=\sqrt{{\sum }_{i=1}^n(z_{1i}-z_{2i}{)}^2}$$

------(34)

In weighted KNN, $w_i$ is given by $w_i={\displaystyle \frac{1}{d\left(z_1-z_2\right)+\in }}$, with a small constant $\in$ added to avoid division by zero. The classification of a query point is determined by the most common class among its K closest neighbors,

$$\widehat{b}={\mathit{argmax}}_c{\sum }_{i\in S_{K\left(z\right)}}{w}_i\cdot I\left(b_i=c\right)$$

------(35)

Here $S_{K\left(z\right)}$ represent the set of K nearest neighbors, $b_i$ is the class label of neighbor $b_i$ and $I\left(\cdot \right)$ is the indicator function. This study uses k=5 with mixed Euclidean distance to improve classification accuracy by weighing closer neighbors more heavily.

4.5.5. Softmax Discriminant Classifier (SDC)

The SDC identifies and verifies the class of a test sample [37], by measuring its distance to training samples within each class. Given a training set, $S=[S_1,S_2,....,S_q]\in {\mathfrak{R}}^{a\times b}$, where $S_q=[S_1^q,S_2^q,.....,S_{b_q}^q]\in {\mathfrak{R}}^{a\times b_q}$ contains $b_q$ samples from class q, with ${\sum }_{j=1}^k{m}_j=m$. The samples used for testing are presumed to be $S\in {\mathfrak{R}}^{a\times 1}$, then SDC is defined as,

$$h\left(y\right)=\mathit{arg}\underset{j}{max}{S}_w^j$$

------(36)

$$h\left(y\right)=\mathit{arg}\underset{j}{max}\mathit{log}\left({\sum }_{n=1}^{b_i}\mathit{exp}\left(-\lambda \left|\right|v-v_n^j|{|}_2\right)\right)$$

------(37)

where $S_w^j$ measures the distinction between the test sample and class j. A penalty cost λ > 0 is applied. If v and vn are similar, and y belongs to class i, $\left|\right|v-v_n^j|{|}_2$ is close to zero, making $S_w^j$ approach its maximum value.

4.5.6. Multi-Layer Perceptron (MLP)

Multilayer Perceptrons (MLPs) are used for function approximation tasks like regression [38]. The MLP structure consists of an input layer with n nodes, an intermediate layer, and an output layer. Input-output pairs are denoted as $\left(a_m,b_m\right),for m=\mathrm{1,2},...,p$, where $a_m=(a_{m1},a_{m2},....,a_{mq})$ is the input vector and $b_m$ is the target output, the output $z_{mp}$ of the k-th hidden node is computed as:

$$z_{mp}=f_s\left({\sum }_{j=1}^n{w}_{jk}{a}_{mj}+{\theta }_k\right)$$

------(38)

The final output $z_m$ is given by,

$$z_m=f_s\left({\sum }_{p=1}^j{w}_p{z}_{mp}+\theta \right)$$

------(39)

Where $j$ represent the number of hidden units, θ denote the bias at the output layer and $w_p$ be the weight connecting the k-th hidden unit to the output layer. This configuration results in (n+2) j+1 connections. The cost function for training the MLP is:

$$F={\displaystyle \frac{1}{2}}{\sum }_{k=1}^z{\left(q_m-z_m\right)}^2$$

------(40)

In this study, a 3-layer architecture was utilized, recognized for its effectiveness in approximating continuous functions [39].

4.5.7. BLDC

The Bayesian Linear Discriminant Classifier (BLDC) employs the Fisher linear discriminant alongside Bayes decision rule to reduce the probability of classification errors [40], effectively regularizing high-dimensional signals and enhancing computational efficiency. In Bayesian regression, the target a is defined as:

$$a=q^{T}s+n$$

------(41)

where q is the weight vector and n is white gaussian noise. The weighted likelihood function is:

$$p\left({\displaystyle \frac{D}{\beta ,q}}\right)={\left({\displaystyle \frac{\beta }{2\pi }}\right)}^{{\displaystyle \frac{M}{2}}}\mathit{exp}\left({\displaystyle \frac{-\beta }{2}}{‖V^{C}q-a‖}^2\right)$$

------(42)

Here, $a$ is the target value, $V$ is the matrix of training feature vectors, D combines $V$ and $a$. $\beta$ is the inverse noise variance, and $C$ is the number of training samples. The prior distribution is given by:

$$p\left(q\right|\alpha )={\left({\displaystyle \frac{\alpha }{2\pi }}\right)}^{{\displaystyle \frac{N}{2}}}{\left({\displaystyle \frac{\epsilon }{2\pi }}\right)}^{{\displaystyle \frac{1}{2}}}\mathit{exp}\left({\displaystyle \frac{-q^C{R}^{\prime }\left(\alpha \right)q}{2}}\right)$$

------(43)

where N is the feature size, $R^{\prime }\left(\alpha \right)$ denotes the (N+1) dimensional regularization diagonal matrix, represented as follows:

$$R^{\prime }\left(\alpha \right)=\left[\begin{array}{ccc}\alpha & \cdots & 0\\ ⋮& \ddots & ⋮\\ 0& \cdots & \epsilon \end{array}\right]$$

------(44)

The posterior distribution of s is:

$$p\left({\displaystyle \frac{q}{\beta ,\alpha ,D}}\right)={\displaystyle \frac{p\left(\frac{D}{\beta ,s}\right)p\left(\frac{s}{\alpha }\right)}{\int p\left(\frac{D}{\beta ,s}\right)p\left(\frac{s}{\alpha }\right)}}$$

------(45)

This posterior distribution is Gaussian with covariance matrix H and mean vector U:

$$U=\beta {\left(\beta V{V}^C+R^{\prime }\left(\alpha \right)\right)}^{-1}Va$$

------(46)

$$H={\left(\beta V{V}^C+R^{\prime }\left(\alpha \right)\right)}^{-1}$$

------(47)

For predictive variation of $\widehat{q}$ , the distribution on the regression target is:

$$p\left({\displaystyle \frac{\widehat{a}}{\beta ,\alpha ,\widehat{q},D}}\right)=\int p\left({\displaystyle \frac{\widehat{a}}{\beta ,\alpha ,\widehat{q},s}}\right)p\left({\displaystyle \frac{s}{\beta ,\alpha ,D}}\right)ds$$

------(48)

This Predictive distribution is also gaussian with mean and variance as follows:

$$\mu =v^C\widehat{q}$$

------(49)

$${\delta }^2=\left({\displaystyle \frac{1}{\beta }}\right)+{\widehat{q}}^{C}H\widehat{q}$$

------(50)

5. Results and Discussion

This study uses deep learning-based feature extraction and feature fusion techniques along with feature selection using PSO and RDO for categorizing histopathological images of lung cancer on a Windows 11 workstation with an AMD Ryzen 7 5700G processor and integrated Radeon Graphics, running MATLAB 2018a.

5.1. Training and Testing of the Classifiers

In this study, K-fold cross-validation is used for classification. For instance, with K=10, the dataset is divided into 10 equal segments, where each segment is used once as the test set and the remaining nine as the training set. Performance metrics are average across iterations. Different K values such as 2, 4, 5, 8, and 10 were evaluated. The training data is partitioned into smaller batches, and classifiers such as SVM, DT, RF, KNN, and BLDC are trained iteratively on these smaller batches, while MLP and SDC are trained directly over epochs. After each epoch, performance is evaluated on both training and validation sets, and accuracy is recorded. Finally, the training and validation accuracies are plotted to visualize performance over epochs. Training stops after a maximum of 15 epochs or when accuracy levels suggest potential overfitting. Testing ends once all batches are processed. Higher accuracy and lower error rates indicate better classifier performance. Table 5 lists the parameters selected for various classifiers, chosen through trial and error, with a maximum of 15 epochs to prevent overfitting.

5.2. Standard Benchmark Metrics of the Classifiers

In this study, several transfer learning models are assessed using a confusion matrix. The evaluation process involves using 90% of the input features for training and setting aside 10% for testing. In the context of lung cancer detection, the clinical scenarios related to the confusion matrix are defined as follows: True Positive (TP): Correct identifying a patient as Benign. True Negative (TN): Correct identifying of a patient as Adenocarcinoma. False Positive (FP): Incorrectly classifying an Adenocarcinoma patient as Benign. False Negative (FN): Misclassifying a Benign patient as Adenocarcinoma. The performance of the classifiers is evaluated using metrics such as Accuracy, Error Rate, F1 Score, Matthews Correlation Coefficient (MCC), Jaccard Index, G-Mean, and Kappa. The mathematical formulations for these metrics are detailed in Table 6.

5.3. Performance Analysis of the Classifiers in Terms of Accuracy for Different K Values

In this study, the performance of seven classifiers—SVM, KNN, Random Forest, Decision Tree, Softmax Discriminant, MLP, and BLDC—was evaluated for cancer image classification across K values of 2, 4, 5, 8, and 10, as shown in Table 7.

In the first scenario, without segmentation, the ResNet-X based feature fusion technique combined with an MLP model achieved an accuracy of 58.610% at K=2 and 63.150% at K=4. The ResNet-50 based feature extraction with the MLP model reached its highest accuracy of 65.230% at K=5, while the ResNet-152 based feature extraction with the MLP model attained a peak accuracy of 68.783% at K=8. Additionally, the ResNet-X based feature fusion technique combined with the MLP model achieved a top accuracy of 69.610% at K=10. In the second scenario, with segmentation, ResNet-152 based feature extraction combined with the MLP model achieved 66.920% accuracy at K=2 and 74.380% at K=5. The ResNet-101 based feature extraction with the MLP reached 72.910% at K=4, while the ResNet-50 based feature extraction with the MLP peaked at 83.730% at K=8. Additionally, the ResNet-X based feature fusion technique combined with the MLP attained a top accuracy of 86.460% at K=10.

In the third scenario, with segmentation and Feature selection, applying PSO for feature selection resulted in a ResNet-50 based feature extraction with the MLP achieving 72.250% accuracy at K=2. The ResNet-152 based feature extraction with the MLP attained 76.432% at K=4, 79.490% at K=5, and 93.508% at K=8, while the ResNet-X based feature fusion technique combined with the MLP reached 96.490% at K=10. Using RDO for feature selection, the ResNet-50 based feature extraction with the MLP achieved 77.980% accuracy at K=2. The ResNet-152 based feature extraction with the MLP reached 87.240% at K=5, while the ResNet-X based feature fusion technique combined with the MLP recorded 82.810% at K=4, 94.531% at K=8, and 98.680% at K=10. Across all scenarios, ResNet-X based feature fusion technique combined with the MLP consistently achieved the highest accuracy, demonstrating its effective deep-weighted averaging feature fusion capabilities.

Table 7. Performance analysis of the Classifiers for all three cases in terms of Accuracy.

DL model with Classifiers	Without Segmentation and FS					With Segmentation only					With Segmentation and PSO FS					With Segmentation and RDO FS
	K = 2	K = 4	K = 5	K = 8	K = 10	K = 2	K = 4	K = 5	K = 8	K = 10	K = 2	K = 4	K = 5	K = 8	K = 10	K = 2	K = 4	K = 5	K = 8	K = 10
ResNet-50-SVM	53.700	59.324	62.990	62.500	57.200	65.820	65.100	72.270	73.703	74.350	65.820	72.983	71.633	82.872	90.110	73.230	77.082	83.140	85.803	89.840
ResNet-50-DT	54.500	56.860	60.160	57.063	67.060	60.830	64.780	71.520	70.603	73.910	61.380	71.270	73.173	83.600	91.290	72.250	78.922	78.382	85.681	90.630
ResNet-50-RF	56.070	59.164	62.630	59.914	68.230	62.960	66.213	68.260	72.790	79.420	62.960	74.150	74.743	86.920	93.110	75.830	79.040	82.422	87.141	91.140
ResNet-50-KNN	53.690	59.650	62.370	66.540	65.540	61.530	68.033	66.473	71.100	72.790	61.530	73.570	65.430	87.500	89.840	73.590	75.950	78.642	84.120	93.350
ResNet-50-SDC	55.190	60.450	64.070	68.445	59.790	59.730	69.733	70.573	77.862	80.730	71.100	72.790	75.260	91.012	92.750	77.760	76.302	82.812	85.810	93.750
ResNet-50-BLDC	51.420	56.510	57.430	57.950	58.150	63.590	63.404	64.490	74.673	77.090	65.950	65.230	65.883	78.840	87.100	73.170	73.700	77.220	83.332	90.880
ResNet-50-MLP	53.710	61.704	65.230	62.440	58.700	66.850	71.110	72.363	83.730	81.250	72.250	74.773	77.082	92.181	94.010	77.980	79.752	85.160	91.731	95.310
ResNet-101-SVM	54.900	61.270	61.220	67.580	64.330	63.220	64.524	66.103	77.990	69.760	63.590	70.893	68.163	86.713	89.960	76.590	80.800	75.130	89.451	90.760
ResNet-101-DT	56.620	61.820	62.760	61.440	60.940	60.830	67.210	64.980	73.960	76.110	61.340	71.550	64.230	80.210	90.620	73.373	76.942	76.170	82.950	91.670
ResNet-101-RF	57.310	62.350	63.970	63.670	65.130	62.000	69.010	66.000	78.640	74.020	62.000	72.500	65.890	82.292	91.670	74.090	77.932	78.902	89.711	93.350
ResNet-101-KNN	54.242	60.960	56.330	60.430	67.969	64.010	70.380	67.380	73.050	80.980	64.750	65.310	73.153	86.761	87.760	74.590	80.270	81.572	83.592	90.230
ResNet-101-SDC	58.450	62.990	63.770	66.570	60.680	64.520	71.910	68.360	80.790	82.550	68.700	71.383	77.342	89.321	93.490	75.010	78.390	83.752	90.760	97.130
ResNet-101-BLDC	53.700	56.720	56.249	56.774	54.450	59.680	63.530	67.970	75.000	81.310	60.830	63.570	68.040	79.560	90.360	69.240	73.113	75.520	82.620	89.320
ResNet-101-MLP	58.610	63.150	64.490	66.780	60.530	66.420	72.910	72.581	82.940	84.120	70.590	74.610	78.130	92.441	94.530	77.830	76.432	84.640	92.906	96.350
ResNet152-SVM	56.090	61.080	60.414	66.123	61.360	63.580	66.410	71.750	77.150	74.480	66.420	68.050	69.013	84.900	93.220	74.290	76.240	81.250	82.560	92.370
ResNet152-DT	53.840	62.870	61.704	62.714	55.970	64.660	68.820	66.410	75.062	73.370	64.520	70.960	71.873	88.541	90.110	70.890	80.080	81.672	82.352	92.570
ResNet152-RF	57.950	63.090	62.990	68.160	63.670	65.840	70.060	68.490	71.873	80.070	68.100	71.350	76.822	89.190	92.180	71.093	81.900	82.820	85.970	93.030
ResNet152-KNN	54.940	62.000	57.120	60.804	66.270	61.380	65.853	61.704	70.320	81.510	63.580	69.530	73.953	79.820	91.460	72.340	81.380	84.250	86.983	92.310
ResNet152-SDC	56.190	60.690	63.386	65.500	67.550	65.950	69.760	73.940	74.480	82.090	68.100	75.690	73.960	92.190	93.750	73.563	82.170	85.160	92.748	95.960
ResNet152-BLDC	55.930	55.810	59.444	58.840	59.340	59.830	64.000	66.670	72.620	75.770	63.220	63.900	70.920	79.750	93.230	69.273	78.970	76.885	82.690	86.710
ResNet152-MLP	56.250	61.460	64.157	68.783	57.880	66.920	72.010	74.380	78.902	85.420	70.480	76.432	79.490	93.508	94.520	77.100	82.550	87.240	93.435	97.660
ResNet-X-SVM	57.410	56.120	57.670	67.300	56.780	62.100	66.703	68.230	70.703	73.300	66.280	66.490	70.813	82.812	91.670	77.410	80.110	77.862	89.650	91.640
ResNet-X-DT	54.900	59.830	61.460	63.480	60.020	61.080	71.650	66.970	72.530	76.690	64.010	73.390	66.670	85.680	88.600	69.730	76.170	78.252	84.770	92.180
ResNet-X-RF	56.120	62.100	63.150	64.390	64.800	64.660	72.460	71.093	77.410	77.580	66.920	74.800	70.633	87.761	92.190	70.543	79.920	84.892	86.391	94.010
ResNet-X-KNN	56.860	55.514	58.760	60.610	54.850	61.340	67.650	67.153	77.990	78.650	68.690	70.303	68.670	88.801	91.660	71.240	77.790	78.772	89.871	93.220
ResNet-X-SDC	58.450	60.960	62.760	67.690	68.930	64.750	65.450	73.180	80.210	83.850	70.080	75.130	70.633	90.881	93.490	72.530	82.712	82.032	92.451	97.860
ResNet-X-BLDC	54.090	54.900	57.310	59.414	61.060	56.860	63.780	68.813	71.230	72.550	60.830	64.800	64.157	87.760	92.180	69.010	76.822	78.780	82.030	88.540
ResNet-X--MLP	58.570	61.050	64.980	68.033	69.610	66.280	68.650	74.020	81.772	86.460	71.000	76.240	71.100	93.231	96.490	74.090	82.810	84.633	94.531	98.680

Table 7. Performance analysis of the Classifiers for all three cases in terms of Accuracy.

DL model with Classifiers	Without Segmentation and FS					With Segmentation only					With Segmentation and PSO FS					With Segmentation and RDO FS
	K = 2	K = 4	K = 5	K = 8	K = 10	K = 2	K = 4	K = 5	K = 8	K = 10	K = 2	K = 4	K = 5	K = 8	K = 10	K = 2	K = 4	K = 5	K = 8	K = 10
ResNet-50-SVM	53.700	59.324	62.990	62.500	57.200	65.820	65.100	72.270	73.703	74.350	65.820	72.983	71.633	82.872	90.110	73.230	77.082	83.140	85.803	89.840
ResNet-50-DT	54.500	56.860	60.160	57.063	67.060	60.830	64.780	71.520	70.603	73.910	61.380	71.270	73.173	83.600	91.290	72.250	78.922	78.382	85.681	90.630
ResNet-50-RF	56.070	59.164	62.630	59.914	68.230	62.960	66.213	68.260	72.790	79.420	62.960	74.150	74.743	86.920	93.110	75.830	79.040	82.422	87.141	91.140
ResNet-50-KNN	53.690	59.650	62.370	66.540	65.540	61.530	68.033	66.473	71.100	72.790	61.530	73.570	65.430	87.500	89.840	73.590	75.950	78.642	84.120	93.350
ResNet-50-SDC	55.190	60.450	64.070	68.445	59.790	59.730	69.733	70.573	77.862	80.730	71.100	72.790	75.260	91.012	92.750	77.760	76.302	82.812	85.810	93.750
ResNet-50-BLDC	51.420	56.510	57.430	57.950	58.150	63.590	63.404	64.490	74.673	77.090	65.950	65.230	65.883	78.840	87.100	73.170	73.700	77.220	83.332	90.880
ResNet-50-MLP	53.710	61.704	65.230	62.440	58.700	66.850	71.110	72.363	83.730	81.250	72.250	74.773	77.082	92.181	94.010	77.980	79.752	85.160	91.731	95.310
ResNet-101-SVM	54.900	61.270	61.220	67.580	64.330	63.220	64.524	66.103	77.990	69.760	63.590	70.893	68.163	86.713	89.960	76.590	80.800	75.130	89.451	90.760
ResNet-101-DT	56.620	61.820	62.760	61.440	60.940	60.830	67.210	64.980	73.960	76.110	61.340	71.550	64.230	80.210	90.620	73.373	76.942	76.170	82.950	91.670
ResNet-101-RF	57.310	62.350	63.970	63.670	65.130	62.000	69.010	66.000	78.640	74.020	62.000	72.500	65.890	82.292	91.670	74.090	77.932	78.902	89.711	93.350
ResNet-101-KNN	54.242	60.960	56.330	60.430	67.969	64.010	70.380	67.380	73.050	80.980	64.750	65.310	73.153	86.761	87.760	74.590	80.270	81.572	83.592	90.230
ResNet-101-SDC	58.450	62.990	63.770	66.570	60.680	64.520	71.910	68.360	80.790	82.550	68.700	71.383	77.342	89.321	93.490	75.010	78.390	83.752	90.760	97.130
ResNet-101-BLDC	53.700	56.720	56.249	56.774	54.450	59.680	63.530	67.970	75.000	81.310	60.830	63.570	68.040	79.560	90.360	69.240	73.113	75.520	82.620	89.320
ResNet-101-MLP	58.610	63.150	64.490	66.780	60.530	66.420	72.910	72.581	82.940	84.120	70.590	74.610	78.130	92.441	94.530	77.830	76.432	84.640	92.906	96.350
ResNet152-SVM	56.090	61.080	60.414	66.123	61.360	63.580	66.410	71.750	77.150	74.480	66.420	68.050	69.013	84.900	93.220	74.290	76.240	81.250	82.560	92.370
ResNet152-DT	53.840	62.870	61.704	62.714	55.970	64.660	68.820	66.410	75.062	73.370	64.520	70.960	71.873	88.541	90.110	70.890	80.080	81.672	82.352	92.570
ResNet152-RF	57.950	63.090	62.990	68.160	63.670	65.840	70.060	68.490	71.873	80.070	68.100	71.350	76.822	89.190	92.180	71.093	81.900	82.820	85.970	93.030
ResNet152-KNN	54.940	62.000	57.120	60.804	66.270	61.380	65.853	61.704	70.320	81.510	63.580	69.530	73.953	79.820	91.460	72.340	81.380	84.250	86.983	92.310
ResNet152-SDC	56.190	60.690	63.386	65.500	67.550	65.950	69.760	73.940	74.480	82.090	68.100	75.690	73.960	92.190	93.750	73.563	82.170	85.160	92.748	95.960
ResNet152-BLDC	55.930	55.810	59.444	58.840	59.340	59.830	64.000	66.670	72.620	75.770	63.220	63.900	70.920	79.750	93.230	69.273	78.970	76.885	82.690	86.710
ResNet152-MLP	56.250	61.460	64.157	68.783	57.880	66.920	72.010	74.380	78.902	85.420	70.480	76.432	79.490	93.508	94.520	77.100	82.550	87.240	93.435	97.660
ResNet-X-SVM	57.410	56.120	57.670	67.300	56.780	62.100	66.703	68.230	70.703	73.300	66.280	66.490	70.813	82.812	91.670	77.410	80.110	77.862	89.650	91.640
ResNet-X-DT	54.900	59.830	61.460	63.480	60.020	61.080	71.650	66.970	72.530	76.690	64.010	73.390	66.670	85.680	88.600	69.730	76.170	78.252	84.770	92.180
ResNet-X-RF	56.120	62.100	63.150	64.390	64.800	64.660	72.460	71.093	77.410	77.580	66.920	74.800	70.633	87.761	92.190	70.543	79.920	84.892	86.391	94.010
ResNet-X-KNN	56.860	55.514	58.760	60.610	54.850	61.340	67.650	67.153	77.990	78.650	68.690	70.303	68.670	88.801	91.660	71.240	77.790	78.772	89.871	93.220
ResNet-X-SDC	58.450	60.960	62.760	67.690	68.930	64.750	65.450	73.180	80.210	83.850	70.080	75.130	70.633	90.881	93.490	72.530	82.712	82.032	92.451	97.860
ResNet-X-BLDC	54.090	54.900	57.310	59.414	61.060	56.860	63.780	68.813	71.230	72.550	60.830	64.800	64.157	87.760	92.180	69.010	76.822	78.780	82.030	88.540
ResNet-X--MLP	58.570	61.050	64.980	68.033	69.610	66.280	68.650	74.020	81.772	86.460	71.000	76.240	71.100	93.231	96.490	74.090	82.810	84.633	94.531	98.680

5.4. Performance Analysis of Classifiers for K = 10

Table 8 presents the classifier performance at K=10. Without segmentation, the ResNet-X based feature fusion technique combined with MLP classifiers achieved the highest performance, with an accuracy of 69.610%, an F1 score of 73.184%, a Jaccard index of 57.709%, a G-Mean of 68.321%, and an error rate of 30.390%. In contrast, the ResNet-101 based feature extraction with BLDC classifiers had the lowest accuracy at 54.450%, an F1 score of 54.345%, a Jaccard index of 37.310%, a G-Mean of 54.449%, and a high error rate of 45.550%.

With segmentation, the ResNet-X based feature fusion technique with MLP classifiers performed best, achieving 86.460% accuracy, an F1 score of 86.317%, a Jaccard index of 75.928%, a G-Mean of 86.453%, and an error rate of 13.540%. On the other hand, the ResNet-101 based feature extraction combined with SVM classifiers had the lowest performance, with 69.760% accuracy, an F1 score of 66.622%, a Jaccard index of 49.950%, a G-Mean of 69.123%, and an error rate of 30.240%.

Table 9 presents the classifier performance at K=10 for segmentation and feature selection. Using PSO feature selection, the ResNet-X based feature fusion technique combined with MLP classifiers achieved the highest accuracy at 96.490%, with an F1 score of 96.476%, a Jaccard index of 93.192%, a G-Mean of 96.489%, and an error rate of 3.510%. In contrast, the ResNet-50 based feature extraction with BLDC classifiers had the lowest accuracy at 87.100%, an F1 score of 86.483%, a Jaccard index of 76.186%, a G-Mean of 86.980%, and an error rate of 12.900%. Using RDO feature selection, the ResNet-X based feature fusion technique with MLP classifiers again achieved the highest performance, with 98.680% accuracy, an F1 score of 98.669%, a Jaccard index of 97.347%, a G-Mean of 98.677%, and an error rate of 1.320%. The lowest accuracy was recorded by the ResNet-152 based feature extraction with BLDC classifiers, which had an accuracy of 86.710%, an F1 score of 85.863%, a Jaccard index of 75.228%, a G-Mean of 86.502%, and an error rate of 13.290%.

Figure 7 shows that while the training loss decreases and accuracy improves, validation loss and accuracy fluctuate, indicating overfitting. Epoch 15 represents the peak of validation performance before overfitting worsens. Early stopping could improve generalization. The model learns well from epochs 1 to 5 but begins to overfit between epochs 6 and 10, with continued overfitting from epochs 11 to 16, excelling on training data but underperforming on validation data which shows the system is not stable for classification purposes.

Figure 8 shows that the model using segmentation and RDO for feature selection experiences a rapid drop in training loss that stabilizes, reflecting effective learning. However, validation loss decreases initially but then rises, indicating overfitting. Training accuracy quickly approaches 100%, while validation accuracy fluctuates and slightly declines, supporting the overfitting observation. Unlike PSO, RDO's higher randomness in the search process helps avoid premature convergence, potentially improving generalization which makes the system more stable for classification purposes.

Figure 9 and Figure 10 show radar plots that evaluate the performance of classifiers using ResNet-based deep feature extraction and optimization techniques in the selective feature pooling layer for feature selection, with K = 10 in K-fold Cross Validation. The analysis compares input images with segmentation, without segmentation, and with segmentation combined with PSO and RDO-based feature selection. The results indicate that ResNet-X-MLP achieves the highest accuracy of 69.610% without segmentation and 86.460% with segmentation alone. When feature selection is applied, ResNet-X-MLP achieves 96.490% with PSO and 98.680% with RDO. RDO shows more consistent performance across epochs, while PSO demonstrates instability as shown in Figure 7 and Figure 8, making ResNet-X-MLP with RDO the more stable choice for classification.

Figure 11 shows the Jaccard Index and F1 Score performance for K = 10 across the classifiers. It reveals a strong positive linear relationship in scenarios with no segmentation, with segmentation alone, and with both segmentation and feature selection using PSO and RDO. The R² value of 0.993 indicates an almost perfect linear correlation. The regression line $y=0.933\cdot x+1.72\times 10^{-16}$ shows that the F1 Score deviation increases slightly less than the Jaccard Index deviation, with a near-zero intercept suggesting minimal deviation from the mean. Table 10 details the previous work carried out in lung cancer detection on various datasets.

5.5. Major Outcomes and Limitations

This research may face limitations due to the specific histopathological images used, which might not be generalizable to other image types or healthcare settings. Issues such as reliance on intensity values from segmented images, outliers, and data overlap could impact classification accuracy. Despite these challenges, the study’s approach, which combines various feature extraction methods, shows promise for identifying cancerous cells in histopathological images. A significant outcome is the creation of a comprehensive lung cancer screening database, which could enhance early detection and improve patient outcomes. Overall, this research provides valuable insights into early lung cancer detection and paves the way for further exploration.

5.6. Computational Complexity

This study evaluates the computational complexity of ResNet-50, ResNet-101, and ResNet-152 based feature extraction, as well as Deep Weighted Averaging Based Feature Fusion features (DWAFF), in combination with feature selection techniques like PSO and RDO across various classifiers, using Big O notation.

In k-fold cross-validation, training on one-fold has a time complexity of $O\left(k\times T\right)$, as the model is trained k times. Complexity grows with input size n, where $O\left(1\right)$ signifies minimal complexity, while $O\left(\log n\right)$ denotes logarithmic growth. Table 11 details the computational complexity and execution times for pretrained transfer learning architectures with various classifiers and feature extraction methods. DWAFF based ResNet-X fused features with an MLP classifier, using RDO feature selection, has the highest complexity at $O \left({4n}^{10}\log n\right)$ and the longest execution time of 480 seconds, due to the extensive training across multiple layers.

6. Conclusions

Lung cancer represents a major worldwide health issue, leading significantly to illness and mortality. Although treatment advancements have been made, early detection and prevention remain vital for addressing this serious public health issue. This study implements the Deep Weighted Averaging-Based Feature Fusion (DWAFF) technique on ResNet-50, ResNet-101, and ResNet-152 architectures for deep feature extraction. Additionally, a selective feature pooling layer is applied after feature extraction to reduce the feature set, which is then fed into seven classifiers for the effective classification of Adenocarcinoma and Benign images from the LC25000 dataset. Performance is measured using standard benchmark metrics, demonstrating strong results in classifying complex lung cancer images. Training and testing were conducted with K-fold Cross Validation. The DWAFF based ResNet-X fused features combined with MLP classifiers for the RDO feature selection method achieved the highest performance, with an accuracy of 98.68%, an F1 score of 98.67%, a Jaccard index of 97.37%, and a G-Mean value of 98.68% at K=10. Future research will focus on extending this approach to multiclass classification and other cancers, such as colon cancer, and exploring the incorporation of RNN models like LSTM and BiLSTM to further improve classification accuracy and support ongoing clinical monitoring.