Enhancement of Classifier Performance with Adam and RanAdam Hyper Parameter Tuning for Lung Cancer detection from Microarray Data - In Pursuit of Precision

1. Introduction

Cancer is a major threat and health concern worldwide. It is a medical condition characterized by the unregulated growth of abnormal cells. Different types of cancers occur in virtually any tissue or organ in the body as mentioned in Egeblad et al. [1]. Among the different cancer types, lung cancer is one of the leading causes of cancer-related deaths worldwide, as reported by Dela et al. [2]. It is considered the most dangerous type of cancer due to several factors, such as late diagnosis, rapid spread, limited treatment options, poor survival rate, etc., as mentioned by Schabath et al. [3]. Lung cancer begins in the cells of the lungs and is primarily caused by smoking, as indicated by Alaoui et al. [4]. Additional risk factors for lung cancer include a familial background of the disease and prior chest radiation therapy, exposure to secondhand smoke and occupational exposure to certain hazardous substances like asbestos, arsenic, diesel exhaust, and chromium as pointed out in Mustafa et al. [5]. The survival rates of lung cancer are highly dependent on the prognosis of cancer at early stages. Next, we discuss about some of the related research literature associated with lung cancer prognosis.

1.1. Review of Previous Work

As suggested in Dela al. [2], the early detection and identification of lung cancer tissues will increase the survival rate. Diagnosing lung cancer involves a combination of medical history assessment, physical examination, clinical techniques such as Chest X-rays, Computer Tomography (CT) scans, Sputum Cytology, Bronchoscopy, Positron Emission Tomography (PET) Scans, etc., effectively detect cancer tissue presence in the human body. CT scans produce detailed cross-sectional images of the lungs by utilizing X-ray images captured from various angles. CT scans can provide more precise information about lung nodules or tumour's size, shape, and location, Causey et al. [6]. Sputum cytology involves examining a sample of mucus coughed up from the lungs under a microscope. It is mainly used to detect lung cancer in individuals with coughing, chest pain, or shortness of breath, Mukae et.al. [7]. The sensitivity and specificity of these clinical procedures can be used to explain their major limitations. Chest X-rays have relatively low sensitivity, mainly for detecting cancer cells in the early detection stages, as indicated by Konstantina Kourou et al. [8]. Also, high radiation exposure often makes CT cumbersome. Sputum Cytology also faces issues like low sensitivity, particularly in the early stages of cancer, and dependency on the presence of cancer cells in the collected sputum. Bronchoscopy collects small lesions from peripheral lung areas and may contain potential false negatives, Leong et al. [9]. Like the Bronchoscopy technique, PET Scans analyze small lesions to distinguish benign and malignant abnormalities but have limited spatial resolution, Visser et al. [10].

Furthermore, invasive procedures such as Bronchoscopy and Sputum Cytology carry the potential risk of severe complications, including bleeding, pain, and infection, and it is only possible to detect malignant cells. So, there are inherent risks invested in collecting tissue samples for the above methods as mentioned in Rivera et al. [11]. Hence, these methods are suggested when an oncologist witnesses significant and solid observations in the early stages of lung cancer.

For the above concern, as suggested in Lubitz et al. [12], the MicroArray Gene Expression (MAGE) data analysis is often preferred due to minimally invasive methods such as fine needle biopsies and blood tests for sample collection. The microarray method comforts and lowers the overall risk profile, making molecular analysis a safer alternative for obtaining diagnostic information, Dhaun et al. [13]. The MAGE data analysis provides a comprehensive molecular profile of the tumour, allowing for a detailed understanding of the genetic alterations associated with lung cancer. This way, the microarray data analysis is unique compared to bronchoscopy and sputum cytology methods, which may only detect malignant cells. The microarray method can unveil specific genetic compositions and its mutations, MAGE patterns, and molecular signatures indicating lung cancer's possibility. Thus microarray method aids accurate diagnosis and envisages personalized treatment strategies based on unique genetic characteristics.

MAGE data is typically a high-dimensional dataset containing measurements of thousands of gene expression levels, as discussed in Nguyen et al. [14]. Data analysis is difficult due to the large number of features, which makes it cumbersome to visualize the relationships between the genes. This problem is often regarded as the curse of dimensionality, as mentioned in Saheed et al. [15]. In [15], the authors have suggested dimensionality reduction (DimRe) as an effective tool to improve the classification performance of the Machine Learning (ML) classification model for MAGE data. The DimRe process aims to decrease the number of features in a dataset while retaining the crucial information. The DimRe methods facilitate the identification of patterns and relationships within the data at subspace, ultimately enhancing the effectiveness of ML algorithms. Further, Feature Selection (FS), as performed by Jager et al. [16], refines the features obtained after DimRe to improve the classification performance.

In De Souza et al. [17], Principal Component Analysis (PCA) was suggested as one of the methods for DimRe in lung cancer MAGE data sets. However, PCA does not capture the inherent nonlinear relationships in the MAGE data. A t-distributed Stochastic Neighbor Embedding (t-SNE) is proposed as a DimRe method by Rafique et al. [18]. However, t-SNE is sensitive to the choice of hyperparameters, and different runs may yield different results. Inamura et al. [19] utilized Non-negative Matrix Factorization (NMF) as a DimRe for MAGE data. The NMF is very sensitive to initialization conditions, leading to potential results variability, especially when applied to microarray data. Sparse Principal Component Analysis (Sparse PCA) was proposed by Hsu et al. [20] for the processing of lung cancer microarray datasets. The major challenge in Sparse PCA that impacts the overall results is selecting an appropriate sparsity parameter. Mollaee et al. [21] utilized Independent Component Analysis (ICA) to reduce MAGE data's dimensionality. However, the ICA assumes statistical independence in the data, which may only hold well in complex biological datasets like the MAGE data. For DimRe, Chen et al. [22] proposed LASSO (Least Absolute Shrinkage and Selection Operator) for MAGE analysis of Adenocarcinoma and lung squamous cell carcinoma medical conditions. The LASSO requires careful tuning of regularization parameters, and selecting an optimal parameter might be data-dependent. Using Genetic Algorithm (GA) FS and manifold learning technique is implemented by Wang et al. [23] for cancer classification using microarray data. This Isomap (Isometric Mapping) technique is computationally expensive and sensitive to noise in the data. So, absolute noise removal is essential for properly working Isomap in MAGE data analysis. The methods like Locally Linear Embedding (LLE) are investigated by Lee et al. [24] for DimRe in MAGE data. LLE is sensitive to the choice of neighbours, and results may vary with different parameter settings. The Fast Fourier Transform (FFT) technique is utilized for the DimRe of DNA methylation data by Raweh et al. [25]. FFT-aided classification reported accurate results with reduced training time. FFT allows for fast computation of frequency components and reduces the training time. Also, the Frequency Domain Interpretation enhances the classification accuracy by revealing hidden periodic and cyclic patterns in the data, providing insights not easily captured by methods like PCA or t-SNE. Otoom et al. [26] utilized Mixture Model (MM) analysis for DimRe of breast cancer microarray data. The MM method's DimRe reported enhanced classification performance for the ML classifiers. The MM is a probabilistic framework that allows a more nuanced understanding of uncertainty and variability in the microarray dataset. The MM also shows cluster interpretability that can naturally represent clusters within the data, interpreting distinct subgroups of MAGE profiles easier. For the above advantages in this research, we adopt FFT and MM as DimRe methods for the lung cancer microarray data.

After DimRe, the reduced data containing distinct and relevant features are subjected to classification. Orsenigo et al. [27] utilized nonlinear manifold techniques for various cancer microarray data classification. The nonlinear manifold technique reported 81% classification accuracy for the lung cancer microarray data. Independent component analysis with naïve Bayes classification attained 83% accuracy for lung microarray datasets, as reported by Fan et al. [28]. Chen et al. [29] used a combination of particle swarm optimization and C4.5 decision tree classification for cancer classification from MAGE data reported 87% accuracy. Díaz et al. [30] achieved a minimum Out-of-bag (OOB) error rate between 0.1% and 0.2% for Random Forest based classification with exhaustive evaluation (large tree size). Support Vector Machine (SVM) and Radial Basis Function (RBF) classification is applied by Azzawi et al. [31] for the cancer classification with microarray data, reporting 90% classification accuracy. Given the above research, we choose Nonlinear Regression (NR), Naive Bayesian (NB), Random Forest (RF), Decision Tree (DT) and SVM (RBF) as the classification methods for lung cancer classification from microarray data. However, these classification methods need further enhancement to improve overall classification performance.

One of the significant techniques to improve a classifier's overall classification outcome is optimizing the parameters associated with the classifier methodology. As sermonized in Kotsiantis et al. [32], the parameters are internal coefficients or weights learned during the training phase of the classifier. The parameters adapt the model's decision boundaries to represent what it has learned from the input data. Traditionally, a fixed learning rate for parameter updates is chosen in classifiers. However, as mentioned by Ioannou et al. [33], adjusting this rate can impact convergence speed and prevent overfitting or under fitting. Therefore, a hyperparameter tuning method can control how the classifier model learns from the data. It can optimize the learning rates, regularization strength, kernel parameters, etc., thereby significantly boosting the classification accuracy and overall performance of the classifier. Hyperparameter tuning helps to balance the classifier model in terms of complexity and flexibility. In this way, the hyperparameter tuning can improve the memory of training data, preventing overfitting and unveiling unseen data and patterns, preventing under fitting.

The grid search is used for parameter tuning by Alrefai et al. [34] to improve the classification performance of microarray data. The Bayesian optimization is employed by Quitadadmo et al. [35] for microarray data. It is more computationally efficient than Grid Search but may not always outperform Random Search. A momentum back propagation is implemented as parameter tuning for cancer detection from microarray data in Wisesty et al. [36] reported 94% accuracy in lung cancer classification. Rakshitha et al. [37] used RMSprop (Root Mean Square Propagation) as a tuning technique for classifying and predicting ovary cysts and reported 89% accuracy. Adaptive Moment Estimation (Adam) combines ideas from both momentum optimization and RMSprop. This optimizer can adjust learning rates based on gradients, offering faster convergence for MAGE data. Sena et al. [38] utilized Adam for ECG classification using Convolutional Neural Networks (CNNs). Random Adaptive Moment Estimation (RanAdam) is an extension of the Adam optimization algorithm with the addition of randomization. So, RanAdam anticipates further improvement in the tuning capability compared to the Adam hyperparameter tuning method. So, based on all the above observations in the literature, this research considers both Adam and RanAdam hyperparameter tuning methods for improving the classification performance of the ML classifiers.

2. Materials and Methods

The well-known Gordon MAGE dataset [39] that contains malignant pleural Mesothelioma and Adenocarcinoma is used for this research. The dataset contains gene expressions useful in lung cancer classification and aid in cancer prognosis at a much earlier stage.

The overall methodology adopted in this research work consists of three approaches. In all three approaches, DimRe is performed as a first step. The DimRe converts higher dimensional MAGE data into lesser dimensional data, retaining the unseen patterns and significant information. The first approach classifies the data after DimRe using ML classifiers into Adeno and Meso classes. The evaluation of the classifier's performance involves various performance metrics. The second approach utilizes FS methods after reducing dimensionality to remove redundant or noisy information. These relevant features are subjected to classification, and their performance is evaluated. In the third approach, after performing FS, the Adam and RanAdam hyperparameter tuning is incorporated into the classifiers to optimize the overall performance of the classifiers. The above approaches employed in this research are abridged in Figure 1.

2.1. Details about the Dataset

The Gordon dataset [39] comprises two distinct classes: Adenocarcinoma and Mesothelioma. The dataset consists of 181 tissue samples, with 150 samples of Adenocarcinoma and 31 of malignant Mesothelioma. 12533 characterize each tissue sample. The total number of rows in the matrix is 12534, including the last row for class labels: ADCA for Adenocarcinoma and MPM for malignant Mesothelioma samples. The MAGE dataset is built on original surgical specimens from patients with microarray experiments. The method is independent of the platform employed for data acquisition and does not need integration of the method of transcription to translation for selected genes. These reasons make MAGE ratios a useful method for training and evaluating algorithms for lung cancer classification.

2.2. Dimensionality Reduction (DimRe)

As previously stated, the microarray data is inherent with the curse of dimensionality, leading to significant Computational Complexity (CC) and reduced model generalization. DimRe techniques become imperative here, as they mitigate the risk of overfitting, enhance interpretability, and facilitate more efficient analysis by extracting meaningful patterns from the high-dimensional data. The volume of data is reduced while preserving essential information. So, the method improves accuracy and scalability in analyzing MAGE data. The methodology for DimRe using the Mixture Model and the Fast Fourier Transform algorithm is discussed in the next section.

2.2.1. Mixture Model for DimRe

In the MM methodology, each gene's expression pattern is considered a univariate distribution, as indicated by Liu et al. [40]. So, one or two Gaussian distributions will be fitted for each distribution and then unequally sub-distributed to fractions and variances. Maximization of Bayesian Information Criteria is the base for selecting the Gaussian fitted distribution. It is expressed as follows:

$${\mathrm{B}}_{\mathrm{A},\mathrm{N}}=\mathrm{x}\log \left(\mathrm{n}\right) -2\mathrm{l}\mathrm{o}\mathrm{g}\left({\mathrm{H}}_{\mathrm{A},\mathrm{N}}\left(\mathrm{y}|\mathsf{\theta }\right)\right)$$

(1)

$$B=y^2+x ln\left(n\right)$$

(2)

Here, model A contains N components with a maximum likelihood function of${\mathrm{H}}_{\mathrm{M},\mathrm{N}}\left(\mathrm{y}|\mathsf{\theta }\right)$, where $\mathsf{\theta }$ the maximizing parameter of the model with respect to mean and variance, x is the total of estimated parameters with sample size ‘n’. Since MM is a probabilistic framework, it can model nonlinear relationships and identify latent structures within the data. Therefore, it provides a more nuanced representation of the underlying biological processes of the microarray data. However, MM requires more computational resources and is often sensitive to the choice of model parameters. Next, we discuss the Fast Fourier Transform (FFT), which is computationally efficient compared to the MM.

2.2.2. Fast Fourier Transform for DimRe

Fast Fourier Transform (FFT) is a frequency-domain technique that is computationally efficient and can be used as a DimRe technique. FFT algorithm is a technique for calculating the discrete Fourier transform of a sequence for a time domain signal. The data is transformed from the time domain to the frequency domain by FFT, allowing for the detection and extraction of prominent periodic patterns naturally present in MAGE profiles. In this way, the relationships inherent in MAGE data are uncovered, and the subtle biological nuances could be unveiled. As given in Cheong Hee Park et al. [41], it is possible to simplify FFT by separately considering odd and even terms and also considering the periodic terms from the DFT expression given as:

$$\mathrm{X}(\mathrm{c}) = {\sum }_{n=0}^{N-1}x\left(n\right){W_N}^{cn}$$

(3)

Where

$$W_N={\mathrm{e}}^{-\mathrm{j}2\mathsf{\pi }/\mathrm{N}}$$

(4)

Here N stands for number of FFT points, n=1, 2, 3…N, c=1, 2, 3…N.

Similarly the Inverse DFT (IDFT) is given by:

$$\mathrm{x}(\mathrm{n}) = \frac{1}{N}{\sum }_{c=0}^{C-1}X\left(c\right){W_N}^{-cn}$$

(5)

Next is a statistical analysis of the feature-extracted data to understand the changes in the dataset after the adopted DimRe methods.

2.2.3. Impact Analysis of DimRe Methods through Statistics

The Pearson Correlation Matrix (PCM) can be used to analyze microarray gene expression data after dimensionality reduction, as performed by Kim et al. [42]. PCM provides insights into the relationships between gene expression profiles by calculating pairwise correlations between genes across samples. Thus, PCM can uncover key gene clusters or functionally related modules, facilitating the discovery of cancer biomarkers. In this work, we evaluate the dataset with PCM after DimRe using the correlation coefficient p. The PCM measures the linear relationship between two data, ranging from -1 to 1. Here, p=1 represents a positive linear relationship, p=-1 represents a negative linear relationship, and if p=0, there is no relationship. Figure 2 explores the Correlation of the FFT DimRe Method for Adeno and Meso cancer classes, respectively. For the Adeno class, correlation values lie between 0.63 and 1.00. So, there is a strong positive linear relationship within the data of the Adeno class. The positive relationship implies that the data in this class move together in a positive direction, and an increase in one variable is associated with an increase in the other. So, the data is more internally consistent and cohesive.

For the meso class, correlation values lie between -0.08 and 0.27. Correlation values closer to zero suggest a weak or no relationship; hence, it is internally consistent. In essence, for the Adeno case, consistent patterns and relationships are available, and meso has inconsistent and diverse patterns and relationships. Figure 3 examines the Correlation of the MM DimRe Method for Adeno and Meso cancer classes, respectively. For the Adeno class, correlation values lie between 0.30 and 0.93. So, once again, there is a positive linear relationship within the data of the Adeno class. For the meso class, correlation values lie between -0.10 and 0.14. So, the data presents a weak positive correlation and a negative correlation. Once again, Meso data is internally consistent. Overall, in the case of both FFT and MM DimRe techniques, weaker correlations pose a challenge for the classification model because it may need to rely on nonlinear relationships or interactions between features to distinguish instances of the class accurately. Also, the high correlations in the Adeno class pose a risk of overfitting. So, further data processing techniques like the FS method must be employed to mitigate the impact of these highly consistent and inconsistent correlated features to improve the classifier model's generalization performance.

After DimRe, we assess whether the selected features provide meaningful information about the underlying patterns in MAGE data. Statistical analysis helps validate the effectiveness of DimRe techniques by examining the significance of the extracted features about the target variable or the problem at hand. This section analyzes whether the outcomes of MAGE data after DimRe is related to statistical parameters such as mean, variance, skewness, T-test, kurtosis, CCA, p-value, and Pearson correlation coefficient (PCC). Table 1 represents the statistical features analysis for MAGE data after DimRe. As mentioned in Table 1, the FFT-based method displays higher mean values and variance among the classes.

The MM method depicts low mean and variance parameters, indicating a class part of variables within the cancer classes. All three types of DimRe methods give positive skewness values and flat kurtosis values. PCC shows a good correlation in intra-class outputs. The T-test and p-value reveal no significant nature after DimRe of the MAGE data. The canonical correlation coefficient indicates the strength and direction of the linear association between the canonical variables. The value of 0.3852 and 0.3371 suggests a moderate positive linear association between the two sets of variables after FFT and MM DimRe. So, there is some degree of association between the variables in the first and second classes, suggesting shared patterns or information between the two classes. Moderate CCA values after feature extraction positively affect classification by providing relevant information for distinguishing between classes.

In the previous discussion, the correlation plot delivered the correlation of data within each cancer class across various subjects in the database. However, it is important to visualize the two cancer classes combined to visualize the distribution of the overall dataset. The violin plot is a data visualization combining aspects of a box plot and a kernel density plot, providing insights into a dataset's distribution and probability density. The comparison of Adeno and Meso cancer classes FFT and MM methods are performed using the violin plot in Figure 4. The width of each violin represents the frequency of data points at different values. The range of the violin represents the comprehensive view of the data distribution. In Figure 4(a), the Adeno data is distributed from 0 to 17500 and Meso data from 0 to 20000. In Figure 4(b), the Meso data is distributed from -75 to 180 and Meso data from -210 to 830. All the observations from Table 1 are reflected in Figure 4. Overall, DimRe reveals unseen patterns and creates complex relations in the data distribution between the two cancer classes. In essence, the DimRe enhances the pre-classification step. But still, techniques like FS are essential and must be put forth to avoid over fitting and under fitting issues during classification. Following this, in the next section, the FS technique is discussed.

2.3. Feature Selection (FS) Techniques

FS is a crucial step in classifying lung cancer data from MAGE data, as it discovers relevant genes that contribute significantly to the classification task while removing irrelevant and redundant features. Several FS techniques exist, including filter, wrapper, and embedded methods prevailing in literature. Filter Methods like Correlation-based involving information gain and mutual information techniques are performed over MAGE data in Almugren et al. [43], which delivered 85% to 90% accuracy on different datasets. The filter techniques may not capture complex interactions among features that contribute to the classification of MAGE data. Wrapper Methods like Recursive Feature Elimination (RFE) and forward Selection with backward elimination are used as FS in lung cancer MAGE data classification by Cai et al. [44] (accuracy - 86.54%) and Alhenawi et.al [45] (accuracy - 94%), respectively. Wrapper Methods are computationally expensive and prone to overfitting, as they optimize based on the specific classifier's performance on the training data. LASSO FS for Tumour classification using MAGE data tested in Kang et al. [46] (accuracy - 96%). A random forest-based FS is performed by Dagnew et al. [47] (accuracy - 94%) for cancer classification from MAGE data. LASSO and random forest-based FS sometimes fail to figure out the intricacies of MAGE data due to the diverse and nonlinear nature of MAGE data. However, Cui et al. [48] proposed dragonfly FS for MAGE data that delivered 97% accuracy on lung cancer datasets. Based on the above reports in the literature, this research employs the meta-heuristic DragonFly (DF) Optimization technique for FS on MAGE data after DimRe.

The DF is an optimization technique influenced by dragonflies' static and dynamic behaviour. In the research by Majdi Mafarja et al. [49],the binary version of the DF algorithm approach is employed to solve FS problems. The static swarming is for feeding, and dynamic swarming is for migrating. Dragonflies make small groups for feeding and fly over a small area to hunt their prey. But for migration, they will form large groups and fly in one direction over a long distance. In static swarming, the movement is not in a single direction but follows a back-and-forth movement. These are the exploration and exploitation phases of a meta-heuristic algorithm. As represented by Chnoor M. Rahman et al. [50] Separation. Alignment, Cohesion, Attraction to food and distraction from the enemy are the important features of the DF algorithm.

Separation is the mechanism for avoiding collision with neighbours.

$${\mathrm{S}}_{\mathrm{i}}=-\sum _{\mathrm{j}=1}^{\mathrm{N}}\mathrm{X}-{\mathrm{X}}_{\mathrm{j}}$$

(6)

Here S_i is the i^th individual’s separation motion, X is the position of the current individual, X_j is the position of j^th dragonfly and N is the total number of dragonflies in the swarm.

Alignment is the matching velocity with the neighbours.

$$A_i=\frac{\sum _{j=1}^N{V}_j}{N}$$

(7)

Here A_i is the i^th individual’s alignment motion, V is the velocity of j^th dragonfly in the neighbourhood. Cohesion represents the tendency of a neighbouring group towards the centre.

$$C_i=\frac{\sum _{j=1}^N{X}_j}{N}-X$$

(8)

Here C_i is the ith individual’s cohesion, X is the position of the current individual, X_j is the position of j^th dragonfly, and N is the total number of dragonflies in the swarm.

Attraction to food can be calculated as

$$F_i=X^{+}-X$$

(9)

F_i is the attraction to food for i^th individual, X is the position of the current individual, and X⁺ is the position of the source of food.

Distraction from enemies is as follows

$$E_i=X^{-}+X$$

(10)

E_i is the i^th individual's distraction motion from the enemy, X is the position of the current individual and X^- is the position of enemy. DF algorithm uses two vectors in an optimization problem, step vector and position vector. Step vector is as follows:

$${∆X}_{t+1}=\left(s{S}_i+a{A}_i+{cC}_i+f{F}_i+{eE}_i\right)+w∆X_t$$

(11)

where w is the inertia weight, t is the iteration number, s indicates the separation weight, S_i gives the separation of the i^th individual, a represents the alignment weight, A_i shows the alignment of i^th individual, c is the cohesion weight, C_i indicates the cohesion of the i^th individual, f is the food factor, F_i gives the food source of the i^th individual, e represents the enemy factor, E_i is the position of enemy of the i^th individual, after calculating step vector, position vector can be calculated as follows:

$$X_{t+1}=X_t+{∆X}_{t+1}$$

(12)

Here t is the current iteration. Figure 5 depicts the impact of MM and FFT DimRe Methods with DF FS for Adeno and Meso carcinoma Cancer Classes through the Normal Probability Plot. In ideal probability plot cases, a straight diagonal line suggests normality, aiding in identifying outliers and assessing the quality of data preprocessing. However, there are departures from linearity indicates non-normality and the presence of subpopulations. There are distinct clusters or deviations from linearity, indicating nonlinearity and divergence within the dataset, representing the underlying subtypes and biological variations between cancer classes.

2.4. Classification

The prime objective of the research is lung cancer classification from MAGE data. As discussed previously, we use five classification algorithms from various observations in reported research: NR, NB, DT, RF and SVM (RBF). The presence of distinct clusters and subpopulations makes these classifiers perform better for the lung cancer MAGE data. MAGE data often exhibits nonlinear relationships, where the expression levels of genes may interact in complex ways to determine the class label. NR classifiers can better capture complex decision boundaries in the data. They can flexibly model complex relationships, potentially improving classification accuracy. Almugren et al. [43] have utilized NB as one of the classification techniques for cancer classification from MAGE data. The naive Bayesian Classifier is based on the probabilistic principle, specifically Bayes’ theorem. NB will calculate the probability based on feature values, and then the class label will be allocated with the highest probability. NB is better suitable for huge datasets due to the computational efficiency of the classifier. Decision trees, especially when deep and complex, can model these nonlinear relationships effectively.

Peng. et al. [51] classified different cancer types from MAGE data, including lung, breast, and colon tumours. In MAGE data, where the number of genes can be very high, the ability of DT to automatically select relevant features can be advantageous. Mohapatra P et al. [52] used the random forest to classify medical data, which consists of eight datasets for different cancer types like breast cancer, prostate cancer, colon tumour and leukaemia. Random Forests, which are ensembles of decision trees, can further enhance the performance of decision tree models. They reduce overfitting, increase accuracy, and estimate feature importance. Random Forests are popular for MAGE data classification because they handle noise and variability in MAGE data. Huynh et al. [53] analyzed SVM as a classification technique to classify MAGE data. SVM classifier deals with the curse of DimRe by obtaining a hyper-plane in high dimensional feature space. In most cases, SVM produces sparse solutions, which will reduce computational burden and thereby improve accuracy.

2.4.1. Nonlinear Regression

NR is a statistical method that works on linear/nonlinear data. One of the powerful tools for analyzing the data is linear regression. But in real-time scenarios, researchers have to deal with the mathematical models whose results are related to nonlinear predictor variables, as mentioned by Martín, C.A et al. [30]. The Euclidean distance is primarily considered from the target of MAGE data with input data using the following equation as represented in Wenseng et al. [54],

$$\sum d={\left|\left|T_i-X_i\right|\right|}^2$$

(13)

$T_i$ indicates the data target, and $X_i$ represents the input data with index $i$. A cuboid expression representing a 3D space is used to project ‘d’ in the following way.

The projection to the 3D space is expressed by the following cuboid equation:

Minimize: a=n₁ × d +n₂² × d² +n₃³ × d³

(14)

$$\mathrm{Subject} \mathrm{to}\text{:} \left\{\begin{array}{c}{n}_{1 }>n_{2 }>n_{3 }>0, n_{i }[0,1]\mathrm{f}\mathrm{o}\mathrm{r}\mathrm{i}=1,2\mathrm{a}\mathrm{n}\mathrm{d}3\\ {\left(\frac{n_{1 }-n_2}{2}\right)}^2<0.5\\ n_{2 }=\frac{n_{1 }}{10},n_{3 }=\frac{n_{2 }}{10}\end{array}\right\}$$

(15)

Later, f = min (a) is calculated and threshold function ‘s’ is chosen for the NR with b₀ as the sum of the squares of average deviation.

s = f + b₀

(16)

The computation of the b₀ is performed using the least squares method.

2.4.2. Naive Bayesian Classifier

NB is based on Bayesian statistical principles. It is a simple and efficient classifier for MAGE data, as described by A. Kelemen et al. [55]. The equations and expressions for Naïve Bayesian Classifier are as follows: Let y_i be the class label for i^th training instance.

$$P\left(y\right)=\raisebox{1ex}{$n$}\!\left/ \!\raisebox{-1ex}{$N$}\right.$$

(15)

Where P(y) is the prior probability of class y, n is the number of instances of class y and N is the total number of training instances. The likelihood of class y can be calculated as

$$P\left(x|y\right)=P\left(x_1,y\right)\ast P\left(x_2,y\right)\ast \dots \dots ..\ast P(x_n,y)$$

(16)

The posterior probability of each class given the observed features x is found using Bayes theorem:

$$P\left(y|x\right)=(P\left(x|y\right)\ast P\left(y\right))/P\left(x\right)$$

(17)

By evaluating the posterior probability of each class, it is possible to make predictions for a new feature vector x.

2.4.3. Decision Tree Classifier

DT can be used for both classification and regression problems. It is a tree-like structure which uses decision nodes for making decisions, and leaf nodes represent the output of those decisions. It will start from the root node and traverse to the leaf node to make new predictions. The class label was stored at the leaf node. A feature and a threshold are used to split the data into two subsets at each node. In each node, the class labels will be in a mixed way. Hastie et al. [56] stated that information gain and Gini impurity are the common impurity measures for measuring it.

The Information gain can be calculated as follows:

$$Information gain\left(S,X\right)=Entropy\left(S\right)-\sum _{u\in values\left(X\right)}\frac{{|S}_u|}{S}\ast Entropy(S_u)$$

(18)

Here X is the feature, S is the set of instances in a node, and S_u is the subset of instances with the value u of feature X. Gini impurity in given as follows:

$$Gini\left(p\right)=1-\sum _{j=1}^N{\left(p_j\right)}^2$$

(19)

Where p represents the proportion of instances of class j in the node and N is the total number of classes. The goal is to select the feature and threshold to reduce maximum impurity, called best split. The best split can be represented by

$$Bestsplit\left(S\right)={argmax}_{a,t } Impurity\left(S\right)-\sum _{u\in values\left(X\right)}\frac{{|S}_u|}{S}\ast Impurity (S_u)$$

(20)

Here X is the feature, S is the set of instances in a node, S_u is the subset of instances with the value u of feature X.

2.4.4. Random Forest

Random Forest classifier is an effective technique for both regression and classification problems. It includes a set of decision trees. It takes predictions from each decision tree, and the final prediction will be based on the majority votes of prediction. It helps to improve the accuracy of prediction for the dataset. Random forest works based on the technique of bootstrap sampling. There is a decision tree for each bootstrap sample. As discussed in previous cases, decision trees select the best split using information gain or Gini impurity criteria. At each node in the decision tree, a random subset of features is chosen to decide the split. Each tree in the Random Forest will independently make predictions, and the final prediction is taken by aggregating the votes of all the trees. The expression for prediction using the majority vote in Random Forest described by James et al. [57] is

$$x={argmax}_a\sum _{n=1}^N{x}_i=a$$

(21)

Where x is the final prediction, N is the number of decision trees in the forest, x_i is the prediction of the i^th tree, and ‘a’ is the class label.

2.4.5. SVM (RBF)

The support vector classifier with the Radial Basis Function is a powerful classifier for handling nonlinear decision boundaries, as explained by El Kafrawy et al. [58]. The RBF kernel maps the input data into a higher dimensional feature space. Vapnik [59] pioneered the SVM concept, which emphasizes finding a decision boundary that maximally separates the data points of different classes by maintaining a maximum margin, thereby enhancing the classifier's generalization capability. The margin, representing the distance between the decision boundary and the closest support vector, is crucial in determining the classifier's robustness and ability to classify unseen data accurately. The margin is given by:

$$K\left(a_i,a_j\right)=e^{(-\gamma {\left|\left|a_i-a_{j }\right|\right|}^{2)}}$$

(22)

Here a_i and a_j are FS vectors of i^th and j^th occurrences, the width of the radial basis function is controlled by the parameter$\gamma$. The objective function of SVM RBF can be represented as:

$$\begin{gathered} {minimize}_v \frac{1}{2}{\displaystyle \sum _{i=1}^N}{\displaystyle \sum _{j=1}^N}{v}_i{ v}_j{x}_i{x}_j K\left(a_i{a}_j\right)-{\displaystyle \sum _{i=1}^N}{v}_i \\ \mathrm{With} {\sum }_{i=1}^N{v}_i{x}_i=0 \\ 0\le v_i\le R for i=\mathrm{1,2},\dots ...N \end{gathered}$$

(23)

Here, R indicates the regularization parameter, x_i is the class label of the i^th instance, v is the dual vector, a_i is the i^th instance, and N is the total of instances. The prediction for a new instance is made by computing the value of the decision function. The decision function is given by:

$$g\left(x\right)= sign ({\displaystyle \sum _{i=1}^N}{v}_i{x}_i K(a_i,a)+b)$$

(24)

Here $K\left(a_i, a\right)$ is the radial basis kernel function, a is the new instance, and b is the bias.

2.5. Training and Testing

As MAGE dataset is limited, the research used the K-fold cross-validation method for training. Xiong et al. [60] explained that the dataset will be distributed into k–equal-sized subsets in the K-fold cross-validation technique. Each fold should have an almost equal distribution of classes. Then, perform k iterations where each iteration uses a different fold as the validation set, and the remaining folds are used for training. In each iteration, the model will be trained using the training set, and the results will be assessed by means of the validation set. The process will be repeated until all folds are taken from the validation set. Once the k-fold cross-validation process has been completed, it is possible to retrain the full dataset, and new predictions can be made on unseen data. In this paper, 10-fold cross validation is performed. There are 1253 features per patient in this work after DimRe. Mean Square Error (MSE) is utilized for supervising the training methodology.

$$MSE=\frac{1}{N}{\sum }_{j=1}^N{\left(O_j-T_j\right)}^2$$

(25)

Here T_j is the target value at model j and O_j is the observed value at time j.

Table 2 shows the training and testing MSE of the classifiers with and without FS Methods for both MM and FFT DimRe Methods. The training MSE always lies between 10^-07 to 10^-09, while the testing MSE changes from 10^-05 to 10^-08. The maximum number of iterations for the training process is 2000. Naïve Bayesian classifier without FS method settled at minimum training and Testing MSE of 1.56 × 10^-09 and 2.93 × 10^-07, respectively. SVM (RBF) Classifier with Mixture Model DimRe Method and DF FS scores minimum training and Testing MSE of 1.96× 10^-09 and 5.18 × 10^-07correspondingly. As in the case of the FFT DimRe Method and With DF FS, the Nonlinear Regression classifier attained minimum Training and Testing MSE of 2.54 × 10^-09 and 6.24 × 10^-08, respectively. The parameters and their values selected for classification are furnished in Table 3.

3. Results and Discussion

In this research work, diverse ML algorithms are assessed with the help of a confusion matrix as given in Table 4, uses 90% of input features for training and 10% for testing.

For lung cancer detection, based on the confusion matrix shown in Table 4, the clinical situations are defined as:

True Positive (TP): A patient is accurately recognized with an Adeno cancer.

True Negative (TN): A patient is accurately recognized with Meso cancer.

False Positive (FP): A patient is wrongly recognized with Adeno cancer when they have Meso cancer.

False Negative (FN): A patient is wrongly recognized with Meso cancer when they have Adeno Cancer.

Next is an analysis of the different parameter metrics such as Accuracy, F1 score, MCC, Error Rate, Youden Index, and Kappa can be used for analyzing the performance. The equations corresponding to the different performance metrics used for evaluating the classifier performance are represented in Table 5. The Accuracy metric is used to evaluate the overall correctness of the classifier's predictions, which is crucial for ensuring reliable identification of gene expression patterns associated with lung cancer. The F1 score balances precision and recall between the imbalanced Adeno and Meso class distributions. The F1 score is important in this research as it aids in accurately identifying genes relevant to disease classification for the imbalanced dataset. Matthews Correlation Coefficient (MCC) provides a balanced measure of classifier performance by evaluating models in datasets with varying class distributions. The proportion of misclassified instances is marked with Error Rate. Error Rate offers insights into the classifier's performance in accurately distinguishing between different gene expression profiles, which is essential for minimizing false discoveries in microarray data analysis. The Youden Index is used to quantify the classifier's ability to identify true positives while minimizing false positives correctly. The Kappa metric measures the agreement between observed and predicted classifications, showing the repeatability of the produced classification results.

Table 6 depicts the performance of the classifiers based on metrics such as Accuracy, Error Rate, F1 Score, MCC, Kappa and YI for Mixture Model and FFT DimRe Techniques without FS. From Table 6, it is explored that the Naïve Bayesian Classifier with FFT DimRe Technique performing with high accuracy of 88.950 %, an F1 Score of 93.464 % and with a low error rate of 11.050 %. The Decision Tree Classifier with FFT DimRe technique performing with a low accuracy of 54.144 %, an F1 Score of 66.122 % and with a high error rate of 39.779%.

Table 7 depicts the performance analysis of the classifiers for the Mixture Model and FFT DimRe Techniques with DF FS. It is clear from Table 7 that the decision tree Classifier achieved a high accuracy of 91.160%, an F1 Score of 94.558%, and a low error rate of 8.840% for the mixture model DimRe method. The Random forest classifier is placed at the lower edge with a low accuracy of 53.039 %, a high Error Rate of 46.961% and an F1 Score of 65.021%. The comparison with Table 6 and Table 7 reveals that the accuracy of the NB classifier is reduced from 76.243% to 68.508%. The reduction in accuracy is because NB assumes independence between features, and DF FS has removed certain independent features that are not directly correlated with class labels. Conversely, with the application of DF FS, the performance of SVM (RBF) is improved from 59.669% to 91.160%. This improvement in classification accuracy is because DF FS reduced the dimensionality of MAGE data by selecting subsets with informative features. With these fewer features, the SVM RBF classifier can more effectively model the MAGE data and avoid overfitting. Moreover, DF FS has retained the most discriminative features that helped the SVM RBF classifier to establish clearer decision boundaries that separates data in the higher-dimensional feature space.

The above uncertainty of classifier performance observed with DF FS is improved by employing Adam and RanAdam hyperparameter tuning methods. Adam and RanAdam are adaptive optimization algorithms that can efficiently adjust each parameter's learning rate associated with the classifier during the training phase. This adaptiveness helps in navigating the parameter space more effectively. Therefore, an accelerated convergence leads to better solutions within a few iterations.

3.1. Hyper Parameter Tuning

The objective of hyperparameter tuning is to optimize the hyperparameters of ML models to improve the performance as described by Daud Muhajir et al. [61]. Hyper-parameters are parameters not learned from the data but are set before training the model. They can control various aspects of the training process. There are different approaches to determine the best values, such as the Adaptive moment estimation method (Adam), Stochastic gradient method, Relative Randomness Function (RRF), Random Weights (RW) hyperparameter updating and Grid Search (GS) method as indicated by Elgeldawi et al. [62]. RanAdam is a new hyper-parameter tuning method used in this work to improve the accuracy of lung cancer classification.

3.1.1. Adam Hyper Parameter Tuning

Adam is one of the commonly used optimization algorithms used in training. It is more effective in handling non-convex optimization problems as mentioned by Sena et al. [38]. The key parameters used by Adam for tuning are learning rate, β₁, β₂, $\in$, and decay rates. $L_r$ (Learning rate) controls the step size during parameter updates in the Adam algorithm. β₁ and β₂ were used to control the exponential moving averages of the gradient and its square, respectively. $\in$ is a small constant added to the denominator in the Adam update rule to prevent division by zero. Adam often has optional learning rate decay mechanisms. It is possible to prevent overfitting by combining Adam with L2 regularization. The strength of regularization can be controlled by tuning the weight decay coefficient. Another parameter which is having an impact on the convergence of Adam is batch size. Larger batch sizes can provide more accurate gradient estimates, while smaller ones can introduce more noise, which might require a smaller learning rate. The number of training epochs can also be considered a hyper-parameter, as stated by Kaur S et al. [64]. Finding the optimal number of epochs for each specific task is necessary. After defining the hyper-parameter space, we must select a tuning strategy like grid search, random search, Bayesian optimization, etc. In this work, accuracy is chosen as the performance metric to optimize classifier parameters. For each hyper-parameter set in our tuning strategy, classifier model is trained using Adam on the training data and validate it on the validation set as indicated by Masud et al. [67]. Table 8 indicates the optimal and initial values of hyper-parameter tuning with Adam for different classifiers. The hyper-parameters are updated according to the following Equation.

$$w_{t+1}=w_t-\frac{L_r}{\in +\sqrt{\widehat{S_t}}}\ast \widehat{V_t}$$

(26)

$$\widehat{V_t}=\frac{v_t}{1-{\beta }_1^t}$$

(27)

$$\widehat{S_t}=\frac{s_t}{1-{\beta }_2^t}$$

(28)

$$v_t={\beta }_1\ast v_{t-1}+\left(1-{\beta }_1\right)\ast \frac{\partial L}{\partial w_t}$$

(29)

$$s_t={\beta }_2\ast s_{t-1}+\left(1-{\beta }_2\right)\ast {\left(\frac{\partial L}{\partial w_t}\right)}^2$$

(30)

In the above equations, $w_t$ & $w_{t+1}$denotes to past and new hyper-parameters; $\frac{\partial L_r}{\partial w_t}$refers to the loss function which has to be minimized according to hyper-parameter$w.$

$$\frac{\partial L}{\partial w_{tr}}=\frac{{ER}_{tr}}{w_{in}}, if tr=1$$

(31)

$$\frac{\partial L}{\partial w_{tr}}=\frac{{ER}_{tr}-{ER}_{tr-1}}{w_{tr}-w_{tr-1}} , if tr>1$$

(32)

Here Error Rate is indicated by $ER$ with $tr$ as the present iteration and $tr-1$ as the previous iteration. Algorithm 1 illustrates the execution of classifier with Adam method.

3.1.2. RanAdam Hyper Parameter Tuning

RanAdam is a hyperparameter optimization technique that efficiently searches for the best hyperparameters for ML models. Randomized Search is particularly useful when the hyperparameter search space is large, and the computational resources are limited. The RanAdam method is introduced to improve the classification performance of Adam further. The procedure can be divided into Adam and Controlled Randomization (CR). The Adam part of the algorithm is the same as performed previously and is used without any changes in the RanAdam method. The CR procedure in RanAdam is responsible for improving performance over the Adam method. Algorithm 2 represents the way of implementing RanAdam. The ideal values for hyper-parameters with high precision can be identified using the nested CR procedure inside the Adam algorithm. In other words, the CR will explore optimal and highly precise hyper-parameters neighbouring the values Adam's method gives in each iteration. The controlled randomization approach uses randomization with two control parameters, such as solution considering rate and solution adjusting rate. The optimal and initial values of hyperparameters β_1, β_2,$\in$, $L_r$ , $w_t$ , $w_{t+1}$, ${v_t, s}_t$ is considered to be the same as that of Adam method. Algorithm 2 illustrates the execution of classifier with RanAdam method.

The RanAdam method employed in this research uses the following values: bandwidth=0.0095, maximum number of iterations =100 or ConvCrit MSE, whichever is met first. Solution considering rate = 0.6, Solution adjusting rate=0.9, Rand 1, Rand 2, Rand 3 ∈ (0, 1) and Rand 4 ∈ (0, 0.1). Next is the analysis of training and testing accuracy with Adam hyper parameter tuning for MM and DDT DimRe techniques with DF FS.

Table 9 shows the Training and Testing Accuracy Analysis of Classifiers with Adam Hyperparameter tuning for the Mixture Model and FFT DimRe Technique with DF FS. Random Forest classifier shows the highest test accuracy of 91.95 %, and SVM (RBF) shows 93.79% training accuracy for the FFT DimRe Method and With DF FS. For the Mixture Model DimRe Method and dragonfly FS, SVM (RBF) shows the highest accuracy for both training and testing at 98.66% and 96.47 %.

Table 10 shows the Training and Testing Accuracy Analysis of Classifiers with RanAdam Hyperparameter tuning for the Mixture Model and FFT DimRe Technique with DF FS. SVM (RBF) classifier shows the top test accuracy of 98.86 % and 99.41% of training accuracy as well for the FFT DimRe Method and With DF FS. For the Mixture Model DimRe Method and dragonfly FS, the Naïve Bayesian classifier shows the highest accuracy for training and testing at 93.22% and 95.87 %.

Table 11 depicts the performance analysis of the Classifiers with Adam Hyperparameter tuning for the Mixture Model and FFT DimRe Techniques with DF FS. It is identified from Table 10 that SVM (RBF) achieved a high accuracy of 94.475% and an F1 Score of 96.667% with an Error rate of 5.525% for the mixture model DimRe method. The Random forest classifier is placed at the higher edge with an accuracy of 88.950%, an Error Rate of 11.050% and an F1 Score of 93.243% for the FFT DimRe method.

Table 12 shows the performance analysis of the Classifiers with RanAdam Hyperparameter tuning for the Mixture Model and FFT DimRe Techniques with DF FS. It is identified from Table 11 that SVM (RBF) achieved a high accuracy of 98.343% and an F1 Score of 98.997% with a low Error rate of 1.657% for the FFT DimRe method. The Random forest and Naïve Bayesian classifiers are placed at the higher edge with the same accuracy of 91.160%, with an Error Rate of 8.840% for the FFT DimRe method. F1 Score is 94.667% for the Naïve Bayesian classifier and 94.702% for the Random Forest classifier.

The Radar plot is depicted in Figure 6, which compares the classification methodologies researched in this paper. The analysis uses ten selected subsets of the main MAGE data based on high variability. Four classification methods are compared: Classification without DF, Classification with DF, Classification with DF & Adam, and Classification with DF & RanAdam. The angular axis (X) represents the various classifiers, and the radial axis (Y) represents the ten selected data sets. The distance of each data point from the centre on its corresponding axis indicates the classifier's accuracy. The classification technique with maximum accuracy is the data point farthest from the centre on the X and Y axis. The Radar plot indicates that the classification with DF & RanAdam is the best performer. Also, in the Radar plot, there are large differences in distances between data points on the same axis. This spread of data points indicates significant performance variations, suggesting that some methods are more sensitive to data and parameter changes.

Finally, Table 13 shows the improvement in the Accuracy of Classifiers with Adam and RanAdam Hyperparameter tuning for the Mixture Model and FFT DimRe Technique with DF FS. Random forest classifier has the highest improvement in accuracy of 41.81 % with the RanAdam Method. SVM (RBF) classifier has the lowest accuracy improvement of 3.509 % with Adam hyper-parameter tuning.

3.2. Computational Complexity (CC)

The classifiers are studied by evaluating the CC. The CC is identified according to input O (n) size. CC is less if it equals O (1). The CC will increase as the number of inputs, ‘n’ increases.

Table 14 depicts the CC for all the classifiers among different DimRe methods with and without FS techniques. It is reported from Table 14 that the Naïve Bayesian classifier are at the level of low CC. SVM (Linear) and Naïve Bayesian classifiers attained moderate complexity for the EHO and Cuckoo search FS methods across the three DimRe techniques. The Least Square Linear regression DimRe method with EHO FS leads to high CC overhead of the classifiers. The Random Forest classifiers across the DimRe methods with and without FS techniques induced high CC but the achieved Accuracy of the classifier is at the lower edge. SVM (Linear) and SVM (RBF) Classifiers perform well with moderate CC.

The Table 15 displays the comparison of research work reported in this paper with the previous works on the lung cancer detection from Micro array gene using binary classifiers.

4. Limitations

The conclusions of this research may be restricted to the specific population of Adeno and Meso cancer classes and may not apply to other populations. The techniques proposed in this work depend on MAGE data, which may involve complex and expensive procedures that are not practicable for routine clinical trials. The presence of outliers in the data has a big role in the accuracy and reliability of the classification results in this work. An outcome of this study is establishing a comprehensive database for mass screening and sequencing cancer genomes. By incorporating MAGE data and adopting the proposed classification techniques, this database allows the identification of patterns and trends in cancer genomes. Early-stage detection and prediction are paramount to improving cancer patients' survival rates.

5. Conclusion and Future Work

The early detection of lung cancer has a very important role in improving the treatments, thereby increasing the survival rate. The MAGE data analysis of lung cancer is an effective technique for early detection. This research combines ML techniques with MAGE data analysis to enhance lung cancer data classification. FFT and MM are used as DimRe techniques and DF is employed as FS technique. The classification was done by five classifiers with hyperparameter tuning were compared and the performance was evaluated. The result shows that, the SVM (RBF) classifier with the FFT DimRe method and DF FS achieved the highest accuracy of 98.86% with RanAdam hyperparameter tuning. The future work planned for this research is to employ LASSO as a method for dimensionality reduction and use ML classifiers, CNN classifiers, DNN classifiers, and LSTM methods for lung cancer classification from MAGE data.