Cancer Classification in Low- and High-Dimensional Biomedical Data Using Machine Learning Models

1. Introduction

Cancer remains a major global health challenge [1]. This is because of its biological complexity and differences among the types of tissues, the stages of the disease, and other features related to genetics and epigenetics [2]. To enhance diagnosis, outcome prediction, and treatment planning, accurate and reliable cancer classification is essential to improve decision-making in biomedical research [3]. A classification method is even more valuable for practical purposes if it is based on noninvasively or less invasively obtained features instead of invasive biopsies. Data-driven analytical approaches for cancer classification are an important step in that direction.

Machine learning has become a powerful and effective tool for cancer classification. It can learn and identify patterns in biological data. In cancer research, we can use machine learning as a tool to classify tissue types, find the differences between disease stages, and support clinical outcomes [3,4]. These methods learn from the data and use the learned patterns to classify new observations. This implies that the structure of the data contributes heavily to the classifier’s performance, as well as its reliability and stability.

In low-dimensional settings, the number of features is relatively small or not too large compared to the number of observations. In this case, machine learning algorithms may have enough information from the samples to learn the patterns in the data. However, in high-dimensional datasets, such as omics datasets, often the number of features is relatively large compared to the number of samples. This results in the well-known ‘too many features but small sample sizes’ problem. This data structure can lead to unreliable and unstable model estimation and model performance [5,6].

In this paper, the DNA methylation dataset is an example of high-dimensional omics data. DNA methylation is a well-studied epigenetic modification involving the addition of a methyl group to the fifth carbon of cytosine, forming 5-methylcytosine, primarily at CpG sites in DNA without altering the DNA sequence [7].

Abnormal patterns in DNA methylation, including hypomethylation, which refers to reduced methylation levels, and hypermethylation, which refers to increased methylation levels, particularly in CpG-rich regions such as gene promoters, have been associated with cancer development and progression [8].

The nature and structure of the DNA methylation data can make it difficult to analyze, especially when the available sample size is small [9,10,11]. Therefore, Principal Component Analysis, a dimension reduction method, was used to reduce the number of features while retaining the relevant information regarding the directions of maximum variability in the dataset before applying classification models [12,13].

In this study, machine learning algorithms were applied to both the low-dimensional breast tissue dataset and the high-dimensional DNA methylation dataset to evaluate how data structure affects model performance. Logistic Regression, Support Vector Machine, Random Forest, and K-Nearest Neighbors were applied and compared using accuracy, precision, recall, and F1-score. As indicated earlier, Principal Component Analysis was applied to the DNA methylation dataset before classification to reduce the number of features while retaining the information regarding dominant directions of variation in the data.

The goal of this study is to evaluate how machine learning algorithms perform under different data conditions and to examine if the classifier’s performance remains reliable when there are a limited number of samples but many features. The focus is not only on model accuracy but on the stability and the reliability of the model performance across different data structures. This is relevant because, especially in high-dimensional biomedical datasets with small sample sizes, model parameter estimates and performance indicators of learning algorithms tend to become unstable and difficult to generalize.

2. Materials and Methods

2.1. Study Design

This study involves two datasets: The first dataset is the Breast Tissue dataset, which represents the low-dimensional problem because the number of predictors in this dataset is relatively small compared to the number of observations. The second dataset was the DNA methylation dataset from osteosarcoma samples, which represents a high-dimensional scenario because the number of features is much larger than the number of samples.

This study was designed as a comparative machine learning analysis to evaluate model performance in both the low-dimensional and high-dimensional settings.

Logistic Regression, Support Vector Machine, Random Forest, and K-Nearest Neighbors were compared in the low- and high-dimensional scenarios. In the high-dimensional dataset, Principal Component Analysis was applied to reduce the number of features before classification.

2.2. Datasets

Among the two datasets used in this study, the first is the Breast Tissue dataset. It consists of 106 observations and 9 numerical predictor variables. The predictor variables include I0 (impedivity at zero frequency, measured in ohms), PA500 (phase angle at 500 kHz), HFS (high-frequency slope of the phase angle), DA (impedance distance between the spectral ends), Area (area under the impedance spectrum), A/DA (area normalized by DA), Max IP (maximum of the impedance spectrum), DR (distance between I0 and the real part of the maximum-frequency point), and P (length of the spectral curve). The response variable is the breast tissue class, making the dataset a multiclass classification problem. The tissue classes include adi (adipose), car (carcinoma), con (connective), fad (fibro-adenoma), gla (glandular), and mas (mastopathy). Since the number of predictors is small compared to the number of observations, this dataset represents a low-dimensional classification problem.

The second dataset is the DNA methylation dataset from osteosarcoma samples. It contains DNA methylation difference scores (‘diffscore’) measured across CpG sites using the Illumina Infinium HumanMethylation450 platform. Each CpG site is a feature, while each osteosarcoma patient is a sample observation. The methylation difference score represents the difference between methylation levels in tumor samples and the average methylation levels in control samples. Positive values indicate hypermethylation, while negative values indicate hypomethylation. See Appendix A for the details of this ‘diffscore’ computation.

The ‘control’ tissue sample that each of the osteosarcoma patient sample was compared with in order to compute the difference score (‘diffscore’) values is actually an average of six different normal tissue samples — adult skeletal muscle, fetal liver, fetal brain, adult adipose tissue, normal adult bone, normal fetal bone.

Clearly, this DNA methylation dataset poses a high-dimensional classification problem because the number of CpG features is much larger than the number of samples. In this study, the disease stage was used as the response variable, where 0 stands for early stage and 1 for advanced stage. As mentioned earlier, Principal Component Analysis was applied to reduce the number of features before applying the machine learning algorithms.

Table 1 summarizes the main characteristics of the two datasets used in this study.

2.3. Data Preprocessing

Data preprocessing was done to prepare both datasets before applying the machine learning models. For the Breast Tissue dataset, the data was first imported and checked to understand the structure of the variables. The column names were cleaned to make them suitable for analysis in R. R is an open-source programming language and software environment used for statistical computing, graphics, and data analysis. Any case or identification column was removed because it does not add information to the classification problem.

The response variable, which is the breast tissue class, was converted into a factor because the problem is a classification problem. The predictor variables were also converted into a numerical format. The dataset was checked for missing values, and missing values in the predictor variables were replaced using the median value of each variable. In some of the classification models, predictors with near-zero variance were removed because they do not contribute much information to the model.

The Breast Tissue dataset was divided into training and testing datasets using an 80/20 split. The training dataset was used to train models, while the testing dataset was used to evaluate how the models perform on unseen data. A seed value of 42 was used to make the results reproducible. Since the predictor variables were measured on different scales, the variables were standardized by centering and scaling. This was important because models such as Support Vector Machine and K-Nearest Neighbors are affected by differences in the scale of the variables.

For the DNA methylation dataset, the data was first imported together with the supplementary sample information. The dataset was checked for missing values before the analysis. The methylation difference score columns were selected for the analysis because they represent the difference between methylation levels in tumor samples and control samples.

The DNA methylation dataset was originally arranged in a way that CpG sites were rows and samples were columns. Since machine learning models require observations as rows and features as columns, the dataset was transposed so that the osteosarcoma samples became the rows and the CpG sites became columns. Constant or almost zero-variance CpG features were removed because they do not provide useful variation for the analysis.

The methylation data was standardized before applying Principal Component Analysis. Principal Component Analysis was then used to reduce the very large number of features into a smaller number of principal components while retaining the dominant directions of variation in the data. The first eight principal components were retained and used as the predictor variables for classification. The Stage variable was added as the response variable, where 0 represents early stage and 1 represents advanced stage.

After preprocessing, Logistic Regression, Support Vector Machine, Random Forest, and K-Nearest Neighbors were applied to the datasets. For the DNA methylation dataset, different train-test splits were also used to examine how stable the model’s performance was under different data partitions.

2.4. Principal Component Analysis

Principal Component Analysis was applied to the DNA methylation dataset because the dataset is high-dimensional. The number of CpG features was much larger than the number of samples, and this can make the machine learning models unstable. Principal Component Analysis is a dimension reduction method that combines the original variables into a smaller number of mutually perpendicular linear combinations called principal components. These principal components retain most of the variation in the original data [12,13].

The data was also standardized before the analysis so that the features can contribute equally to the principal components. The principal components were selected based on the proportion of variance explained by them and the cumulative variance proportion they explain together. In this study, the first eight principal components were retained because they explained approximately 81.6% of the total variance in the DNA methylation dataset. These eight principal components were used as the predictor variables for the machine learning models. This significant reduction facilitated the implementation of Logistic Regression, Support Vector Machine, Random Forest, and K-Nearest Neighbors to this dataset for disease stage classification.

2.5. Machine Learning Models

The four classifiers Logistic Regression, Support Vector Machine, Random Forest, and K-Nearest Neighbors were selected because they are commonly used for classification problems and represent different ways of learning patterns from data [13,14].

Logistic Regression estimates the relationship between predictor variables and a categorical outcome by modeling the log-odds of class membership as a linear combination of the predictors. The logistic function transforms this linear predictor into a probability between 0 and 1. In binary classification, the predicted probability is compared with a classification threshold to assign an observation to one of two classes. Multinomial Logistic Regression extends the same principle to outcomes with more than two categories by estimating the log-odds of each category relative to a reference category [13,15]. In the Breast Tissue dataset, multinomial Logistic Regression was applied because the response variable had six tissue classes. In the DNA methylation dataset, binary Logistic Regression was used because disease stage was coded as 0 for early stage and 1 for advanced stage.

Support Vector Machine constructs a decision boundary, called a hyperplane, that separates the classes while maximizing the margin between the boundary and the nearest training observations from each class. These nearest observations are called support vectors because they largely determine the position of the decision boundary. When the classes cannot be separated adequately by a straight boundary, a kernel function can represent the data in a higher-dimensional feature space where separation may be easier. Common kernels include linear, radial basis function, polynomial, and sigmoid kernels. The regularization parameter controls the balance between obtaining a wider margin and allowing classification errors [16,17]. For the Breast Tissue dataset, linear, radial, polynomial, and sigmoid kernels were considered. For the DNA methylation dataset, a linear Support Vector Machine was applied using the retained principal components as predictors.

Random Forest is an ensemble method that builds many decision trees rather than relying on a single tree. Each tree is trained using a bootstrap sample drawn from the training data, and only a randomly selected subset of predictors is considered at each possible split. This additional randomization reduces correlation among the trees. Each tree recursively divides the predictor space into increasingly homogeneous groups, and the final class is determined by majority voting across the trees. Combining many partly independent trees usually reduces prediction variance and makes the model less sensitive to the instability of an individual decision tree. Random Forest can also capture nonlinear relationships and interactions among predictors and can provide measures of predictor importance [18,19]. In this study, Random Forest was applied to both the Breast Tissue dataset and the DNA methylation dataset.

K-Nearest Neighbors is a nonparametric, instance-based method that does not estimate an explicit mathematical model during training. To classify a new observation, the algorithm calculates its distance from the observations in the training dataset, identifies the K closest observations, and assigns the class most common among those neighbors. The choice of K affects the flexibility of the classifier: a small value can make the model sensitive to noise and individual observations, whereas a larger value produces a smoother decision boundary but may overlook local class structure. Because the method is based on distances, predictor scaling is important, and its performance can deteriorate when irrelevant variables or very high-dimensional feature spaces make distances less informative [20]. The predictor variables were therefore standardized before K-Nearest Neighbors was applied.

For the Breast Tissue dataset, the models were applied directly after preprocessing because the dataset was low-dimensional. For the DNA methylation dataset, the models were applied after Principal Component Analysis because the original number of CpG features was much larger than the number of samples. The same set of models was used so that their performance could be compared under low-dimensional and high-dimensional data conditions.

2.6. Model Evaluation

The performance of the machine learning models was evaluated using accuracy, precision, recall, and F1-score. These metrics were used to compare how well each model classified the observations in both the low-dimensional and high-dimensional datasets.

Accuracy was used to measure the overall proportion of correctly classified observations. Precision was used to measure how many of the predicted class labels were correctly classified. Recall was used to measure how many of the actual class observations were correctly identified by the model. The F1-score was also used because it gives a balance between precision and recall.

For the Breast Tissue dataset, the data were divided into training and testing datasets using an 80/20 split. The training dataset was used to train the models, while the testing dataset was used to evaluate model performance on unseen data.

For the DNA methylation dataset, model evaluation was done using different train-test splits because the sample size was small. The splits used were 80/20, 70/30, and 60/40. This was done to examine whether the model performance remained stable when the data partition changed. This was important because in a high-dimensional dataset with small sample size, model performance can change depending on which samples are included in the training and testing datasets.

The evaluation was therefore based not only on the performance values, but also on how stable the models were across different data conditions.

3. Results

3.1. Breast Tissue Classification Results

Figure 1 shows the 3D PCA plot of the Breast Tissue dataset using the first three principal components. The plot was used to visualize the structure of the dataset and to see how the tissue classes are distributed in the reduced feature space. From the plot, some tissue classes show visible separation, while some classes overlap. This means that some classes may be easier for the machine learning models to classify, while overlapping classes may be more difficult to classify correctly.

Principal Component Analysis was used here only for visualization of the Breast Tissue dataset. The machine learning models for the Breast Tissue dataset were trained using the original predictor variables after preprocessing. On the other hand, as has been pointed out earlier, PCA was used as a dimension reduction step for the DNA methylation dataset because the number of CpG features was much larger than the number of samples.

Table 2 presents the comparative performance of the machine learning models applied to the Breast Tissue dataset. Random Forest and SVM with linear kernel achieved the highest accuracy of 0.7895. The SVM linear model also recorded the highest precision of 0.8056 and the highest F1-score of 0.7841, showing strong overall classification performance. Random Forest had the highest recall of 0.7917, indicating that it identified more of the actual class observations correctly. Logistic Regression showed moderate performance, while K-NN with k = 5 had the lowest performance across all the evaluation metrics.

3.2. DNA Methylation Principal Component Analysis (PCA) Results

Table 3 shows the variance explained by the principal components from the DNA methylation dataset. In this study, the first eight principal components were used for the main machine learning analysis because they retained more than 80% of the total variation in the data. This reduced the large number of DNA methylation features into a smaller number of linear combinations while still retaining the dominant directions of variation in the dataset. Additional analyses using fewer and more principal components were also considered to examine how the number of retained components affected model performance.

Although PC1 to PC14 together explained all the variation in the dataset, all the components were not used for the main classification analysis because the number of samples was small. Using too many principal components with a small sample size may increase model instability and reduce the reliability of the performance estimates.

Figure 2 shows the scree plot of the principal components from the DNA methylation dataset. The plot shows that the first few principal components explained more variation than the later components. The curve decreases gradually as the number of principal components increases. This means that the variation in the DNA methylation dataset was spread across several principal components.

The scree plot supports the use of the first eight principal components for the main machine learning analysis because PC1 to PC8 explained about 81.60% of the total variation in the data. After the eighth principal component, the additional components contributed smaller amounts of variation.

3.3. DNA Methylation Classification Results

So, the first eight principal components were used as the predictor variables for disease stage classification. Logistic Regression, Support Vector Machine, Random Forest, and K-Nearest Neighbors were applied to the DNA methylation dataset using PC1 to PC8. Since the sample size was small, different train-test splits were used to check how the model performance changes.

Table 4 shows the model performance across the 80/20, 70/30, and 60/40 train-test splits. From the table, the model performance changes across different splits. In the 80/20 split, all the models had the same accuracy of 0.3333 and F1-score of 0.5000. In the 70/30 split, Logistic Regression had the highest performance with accuracy, precision, recall, and F1-score all equal to 1.0000, while the other models performed lower. In the 60/40 split, Logistic Regression also performed better than the other models, but the performance was not the same as the 70/30 split.

This shows that the model performance in the DNA methylation dataset was not stable across the different train-test splits. The results suggest that the small sample size affected the reliability of the machine learning models, even after Principal Component Analysis was used to reduce the dimension of the data.

3.4. Sensitivity Analysis Using Different Numbers of Principal Components

Table 5 shows the sensitivity analysis using different numbers of principal components. The results show that when the number of principal components changed, the model performance also changed. When PC1 to PC5 were used, the model performance changed across the train-test splits, and some models had zero precision, recall, and F1-score. When PC1 to PC8 were used, Logistic Regression gave high performance in some splits, but the overall result still showed instability. When PC1 to PC11 were used, more variation was retained in the data, but the model performance was still unstable across the splits.

This shows that increasing the number of principal components did not automatically improve the model performance. The result also shows that the DNA methylation classification was sensitive to the number of principal components used and how the data was divided into training and testing datasets.

3.5. Summary of Results

Overall, the results show that the machine learning models performed differently in the Breast Tissue dataset and the DNA methylation dataset. In the Breast Tissue dataset, Random Forest and Support Vector Machine performed better than the other models. Logistic Regression showed moderate performance, while K-Nearest Neighbors had the lowest performance.

In the DNA methylation dataset, the model performance was not stable across the different train-test splits. Even though Principal Component Analysis was used to reduce the dimension of the data, the performance still changed depending on how the data was divided into training and testing datasets.

The sensitivity analysis also showed that when the number of principal components changed, the model performance also changed. This means that the DNA methylation classification was affected by both the number of principal components used and the train-test split. These results show that the machine learning models performed more reliably in the low-dimensional Breast Tissue dataset than in the high-dimensional DNA methylation dataset with small sample size.

4. Discussion

4.1. Interpretation of Breast Tissue Results

The Breast Tissue dataset was the low-dimensional dataset used in this study. The number of predictors was small compared to the number of observations, and this made it easier for the machine learning models to learn the pattern in the data. From the results, Random Forest and Support Vector Machine performed better than Logistic Regression and K-Nearest Neighbors.

Random Forest performed well because it can learn complex patterns in the data. It combines many decision trees, and this helps the model to improve classification performance. This may be why Random Forest had high accuracy and recall in the Breast Tissue dataset.

Support Vector Machine with the linear kernel also performed well. This shows that some of the tissue classes were separated well using the predictor variables. The SVM model had the highest precision and F1-score, which means it performed well in classifying the tissue classes.

Logistic Regression showed moderate performance. This may be because Logistic Regression uses a linear decision boundary, and this may not fully capture the pattern among all the tissue classes. However, it still performed better than K-Nearest Neighbors.

K-Nearest Neighbors had the lowest performance. This may be because K-Nearest Neighbors depends on distance between observations. If some tissue classes overlap or have similar values, the model may find it difficult to classify them correctly. This may explain why K-Nearest Neighbors did not perform as well as Random Forest and Support Vector Machine.

Overall, the Breast Tissue results show that the models performed better in the low-dimensional dataset. The results also show that the choice of model is important because the models did not perform the same on the same dataset.

4.2. Interpretation of DNA Methylation Results

The DNA methylation dataset was the high-dimensional dataset used in this study. The number of CpG features was much larger than the number of samples, and this made the analysis more difficult. Even though Principal Component Analysis was used to reduce the dimension of the data, the model performance was still not stable across the different train-test splits.

From the results, the models did not perform the same when the train-test split changed. In the 80/20 split, all the models had low performance. In the 70/30 split, Logistic Regression gave very high performance, but this result should be interpreted carefully because the sample size was small. In the 60/40 split, Logistic Regression still performed better than the other models, but the performance was not the same as the 70/30 split.

This shows that the DNA methylation results were affected by how the data was divided into training and testing datasets. Since the sample size was small, changing the samples in the training and testing sets changed the model performance. This means that the high performance in some splits may not represent true model reliability.

Random Forest, Support Vector Machine, and K-Nearest Neighbors did not show consistent performance across the splits. This may be because the number of samples was too small for the models to learn stable patterns from the data. The high-dimensional structure of the DNA methylation data also made the classification problem more difficult.

Overall, the DNA methylation results show that machine learning performance can become unstable when the dataset has many features but only few samples. This means that the results from high-dimensional biomedical data should be interpreted carefully, especially when the sample size is limited.

4.3. Implications of the Study

The results of this study show that the structure of the data is important when applying machine learning models for cancer classification. In the Breast Tissue dataset, the models performed better because the number of predictors was small compared to the number of observations. This means that when the dataset has enough observations compared to the number of features, the models may learn the pattern in the data better.

For the DNA methylation dataset, the results showed that high-dimensional data with small sample size can make model performance unstable. Even after Principal Component Analysis was used to reduce the dimension of the data, the performance still changed across the train-test splits. The performance also changed when the number of principal components changed. This shows that high accuracy in one split does not always mean that the model is reliable.

This is important in cancer classification because machine learning models can support diagnosis, disease stage classification, and treatment planning. If the model’s performance is not stable, then the result may be difficult to trust for decision-making. This is more important in omics data, where the number of features is very large and the number of samples may be small.

The study also shows that model evaluation should not depend only on accuracy. Precision, recall, F1-score, and stability across different splits should also be considered. This helps to know if the model is learning useful patterns or if the result is affected by the small sample size and how the data was divided.

Overall, this study shows that machine learning can support cancer classification, but the results must be interpreted carefully. For high-dimensional biomedical data, enough sample size, proper validation, and careful feature reduction are important to improve the reliability of model performance.

4.4. Limitations

This study has some limitations. The first limitation is the small sample size, especially in the DNA methylation dataset. The DNA methylation dataset had about 480,000 CpG features but only 15 samples. Because of this, the number of features was much larger than the number of samples, and this made it difficult for the machine learning models to learn stable patterns from the data.

Another limitation is that the model performance was evaluated using train-test splits. Even though different train-test splits were used, the small sample size still made the results sensitive to how the data was divided. This means that some high-performance values may be due to the specific samples in the training and testing datasets.

Also, Principal Component Analysis was used to reduce the dimension of the DNA methylation data. Even though PCA helped to reduce the number of features, it may also remove some information from the original data. The number of principal components used also affected the model performance, which shows that the results can change depending on the dimension reduction choice.

Another limitation is that external validation was not used. The models were tested only on the available datasets. Without an independent external dataset, it is difficult to confirm whether the models will perform the same way on new biomedical data.

Overall, these limitations show that the results should be interpreted carefully, especially for the DNA methylation dataset. Future studies should use larger sample sizes, external validation datasets, and more robust validation methods to improve the reliability of model performance.

5. Conclusions

This study compared machine learning performance in a low-dimensional Breast Tissue dataset and a high-dimensional DNA methylation dataset. Logistic Regression, Support Vector Machine, Random Forest, and K-Nearest Neighbors were applied to both datasets to examine how the structure of the data affects model performance.

The results showed that the models performed more reliably in the Breast Tissue dataset. Random Forest and Support Vector Machine had the highest performance in the main analysis, while Logistic Regression showed moderate performance and K-Nearest Neighbors had the lowest performance. This shows that machine learning models can perform better when the number of predictors is small compared to the number of observations.

For the DNA methylation dataset, the results were not stable across the different train-test splits. The dataset had about 480,000 CpG features but only 15 samples, which made the classification problem more difficult. Even after Principal Component Analysis was used to reduce the dimension of the data, the model performance still changed across the splits and across the number of principal components used.

Overall, this study shows that machine learning can support cancer classification, but the reliability of the results depends on the structure of the data. In high-dimensional biomedical data with small sample size, high performance in one split should be interpreted carefully. Future studies should use larger sample sizes, external validation datasets, and stronger validation methods to improve the reliability of machine learning models in cancer classification.