Bootstrap Approach for Principal Component Analysis Method; Cases studies

1. Introduction

Data visualizing are receiving more attention since they are easier to understand and summing up data. Dealing with big data poses a challenge with "curse of dimension" and computation capacity among other issues [1]. Dimension reduction methods project the data dimensionality into a lower-dimensional space while retaining as much information as possible. These methods reduce the data storage and computation time, besides, overcome the complexity of the model. In recent decades, scientists have used various methods of machine learning (ML) to detect patterns and determine the intrinsic dimensions of big data to generate actionable insights, this could be handled by a range of dimension reduction methods. The set of techniques available for dimensionality reduction can be divided into techniques for supervised and unsupervised learning applications. Unsupervised methods aim to detect similarities and differences inside a dataset.

Principal component analysis (PCA) is one of the most popular dimension reduction methods [2], it is also known as unsupervised method. The main purpose of PCA is to reduce the dimensionality of a data set consisting of a large number of interrelated variables while retaining as much as possible of the variation present in the data set. PCA was briefly mentioned by Fisher and Mackenzie [3] as being more suitable than analysis of variance for the modelling of response data.

Fisher and Mackenzie [3] also outlined the NIPALS algorithm, later rediscovered by Wold [4]. Hotelling [5] further developed PCA to its present stage. Since then, the utility of PCA has been rediscovered in many diverse scientific fields, resulting in, amongst other things, an abundance of redundant terminology. Rao [6] paper is remarkable for the large number of new ideas concerning uses, interpretations and extensions of PCA that it introduced. Various fields apply PCA in applications such as: face recognition, handprint recognition, and mobile robotics. More applications are discussed in [7].

The bootstrap method was introduced by Efron [8]. It is a resampling technique for estimating the distribution of statistics based on independent observations, and then developed to work with other statistical inference. It is used for assigning the measures of accuracy of the sample estimate, especially the standard error. The bootstrap sample is obtained by randomly sampling n times with replacement from the original sample. In this paper, the bootstrap method is used to analyze the stability of PCA results and to estimate the number of PCA in efficient way. The method is used to estimate the number of PCA that needed to classify the data set. For a good introduction to the bootstrap method, see[9]. The bootstrap method can be applied to with a lot of applications, such as tests, regression, and confidence intervals. And used as a useful statistical tool with clustering and mixture models [10,11]. Binhimd and Coolen [12] introduced a new bootstrap method based on nonparametric predictive inference. It is called NPI-B. They compared between (NPI-B) and Efron's classical bootstrap method (Ef-B), then applied (NPI-B) for reproducibility of some statistical tests. Binhimd and Almalki [13] discussed the comparison between Efron's bootstrap and smoothed Efron's bootstrap using different method. Some literatures discussed the work of bootstrapping principal component analysis [14, 15,16].

Practically, one needs to decide the number of components that need to be retained, which means measuring the stability of components or variables. In terms of statistical modelling, stability could be demanded that a data set drawn from the same underlying distribution should give rise to more or less the same parameter. It means that a meaningful valid parameter (component) should not disappear easily if the data set is changed in a non-essential way [17]. Some of stability rules depend on computationally intensive, such as cross validation, bootstrap or jackknife. In the literatures, bootstrap used to estimate the confidence region for differentiable function of the covariance matrix of the original data, this is done after implementing PCA on the considered data. Another stability criterion, one based on a risk function which is known a distance between orthogonal projectors. The aim of this paper to study the stability of PCA method in such a way to improve the process of existing method and the effectiveness of the discussed techniques is demonstrated through a simulation study and real data sets.

2. Materials and Methods

This section provides a brief overview of the algorithms and procedures that will be used in the paper.

2.1. Principal Component Analysis

Principal component analysis (PCA) is a dimension reduction method that projects the data with high dimensional space into a lower-dimensional sub-space. The new data can then be more easily visualized and analyzed, while retaining as much as possible of the data’s variation. It has applications in various fields, such as face recognition and image compression, it is a common technique for finding patterns in high-dimensional data.

Hence, let $X={(x_1,\dots ,x_D)}^T$ be a random vector that has a probability density function $f\left(x\right)$ from a multivariate Gaussian distribution with mean and variance denoted by $\mu$ and $\Sigma$, respectively. Assume that a sample of size $N$ is drawn from the random vector $X$, yielding data $Z=\left(x_1,\dots ,x_N\right)\in R^D$, which is N independent and identically distributed (iid) units. The matrix Z has the following structure:

$$Z=\left[\begin{array}{ccccccc}{x}_{11}& x_{12}& x_{13}& \cdots & x_{1j}& \cdots & x_{1D}\\ x_{21}& x_{22}& x_{23}& \cdots & x_{2j}& \cdots & x_{2D}\\ x_{31}& x_{32}& x_{33}& \cdots & x_{3j}& \cdots & x_{3D}\\ ⋮& ⋮& ⋮& ⋮& ⋮& ⋮& ⋮\\ x_{N1}& x_{N2}& x_{N3}& \cdots & x_{Nj}& \cdots & x_{ND}\end{array}\right]$$

(1)

The principal component analysis (PCA) method reduces a large number of associated variables into a manageable number of main components, which are independent linear combinations of those variables.

The main idea of PCA is to illustrate the variation that appears in a dataset that has correlated variable, $X^T=\left(x_1,\dots ,x_D\right)$ , then the new dataset of uncorrelated variables$Y^T=\left(y_1,\dots ,y_D\right)$, where each of $y_{is}$ a linear combination of X. The variables in the new data are the principal components, where, the first principal component of the observation, $y_1$, is the linear combination:

$$y_1=a_{11}{x}_1+a_{12}{x}_2+\cdots +a_{1D}{x}_D$$

(2)

Whose sample variance is greatest among all such linear combinations. Because the variance of $y_1$ can be increased without limit simply by increasing the coefficients $a_1=(a_{11},a_{12},\cdots ,a_{1D})$, a restriction must be placed on these coefficients. As we shall see later, a sensible constraint is to require that the written

$a_1^{⊺}{a}_1=1$. The sample variance of $y_1$ that is a linear function of the $x$ variables is given by $a_1^{⊺}{Sa}_1$, where $S$ is the $D\times D$ sample covariance matrix of the $x$ variables. To maximise a function of several variables subject to one or more constraints, the method of $Lagrangemultipliers$ is used.

The second principal component, $y_2$, is defined to be the linear combination:

$$y_2=a_{21}{x}_1+a_{22}{x}_2+\cdots +a_{2D}{x}_D$$

(3)

i.e., $y_2=a_{2}x$, where $a_2=(a_{21},a_{22},\cdots ,a_{2D})$ and x =$(x_1,x_2,\cdots ,x_D)$ that has the greatest variance subject to the following two conditions:

$$\begin{gathered} a_2^{⊺}{a}_2=1, \\ {a}_2^{⊺}{a}_1=0. \end{gathered}$$

(4)

Usually, the total sum of eigenvalues that are obtained from a covariance matrix is equal to the total sum of variance of each variable.

2.2. Bootstrap

The bootstrap's primary tenet is that, in some circumstances, it is preferable to draw conclusions about a population parameter only from the available data, without making any assumptions about underlying distributions. Compared to traditional inferences, bootstrap approaches can often be considerably more correct. This method was developed to work with different statistical inferences, such as confidence intervals, hypothesis tests, and regression, it is good technique if the sample size is small. The basic method here depends on drawing B random samples of size n with replacement from the original sample, and then calculating the statistic of interest to estimate the sampling distribution and work with the required statistical inference. The issue with the method is that each time we gather a sample, we can receive somewhat different findings. The standard deviation of a point estimate for repeated population samplings theoretically may be quite high, which could bias the estimate.

2.3. Principal Components Analysis with bootstrap confidence interval

The extent to which the estimator is likely to diverge from the parameter must be determined in order to build a confidence interval for the parameter. The main premise behind the bootstrap is to assess how much the estimate swings when it is calculated using resampled versions of the original data.

2.4. Approaches using bootstrap

The bootstrap sample is drawn at random from the original, size N sample with replacement. Construct the bootstrap confidence interval:

• The maximum eigenvalue ${\lambda }_i$ can be found by repeatedly resampling dataset of (B) times. Using the resampled dataset I's covariance matrices.
• The difference between the true and bootstrap eigenvalues (d) should be calculated.
• A 0.025 percentile and a b=0.975 percentile of d are needed to calculate the confidence interval 95%. The appropriate confidence interval is ($\lambda$true-b, $\lambda$true+a).

3. Results

This section explains the implementation of bootstrap method in PCA on the selected data, Protein and Chemical data sets. The section offering a discussion across the following subsections: Section 3.1 presents the results of Protein data; Section 3.2 presents the results of Chemical data.

3.1. Protein data

Protein consumption data consists of 25 rows of observations and 11 variables, the data is collected from 25 European countries for 9 kinds (groups) of foods Gabriel (1981). As first step, the PCA is implemented on this data to insight the data structure and conduct its variability, result is shown in Figure 1. It presents that 91% of the total variance is explained by five components, with PC1 explaining 44.5% of the total variance, and PC2 explaining 18.2% of the total variance. PC3 further explains 12.5% of the total variance, PC4 and PC5 explaining 10.6% and 5.2%, respectively, of the total variance. This offers a perspective on these components’ comparative relationships to the dataset, with just five PCs explaining 91.43% of the total variance. To evaluate the relationship between the multiple variables and the relevant PCs, Table 1 illustrates the squared factor loading for each variable, the total percentage of the variance that can be explained by the two components is around 62.7%. Consider the variables Red_Meat, White_Meat, Eggs, Milk, Fish, Cereal, Starch, Nuts and Fruits_Vegetables, we shall represent them by ${\mathbf{X}}_1,{\mathbf{X}}_2,{\mathbf{X}}_3,{\mathbf{X}}_4,{\mathbf{X}}_5,{\mathbf{X}}_6,{\mathbf{X}}_7,{\mathbf{X}}_8,{\mathbf{X}}_9$, respectively. As illustration in Eq. (5) and Eq. (6), one can infer that highest contributing variables in PC1 are Cereal and Nuts while Eggs provides the highest negative level of contribution. In PC2, the highest contributing variables are Fish and Fruits_Vegetables while Eggs provides a low negative level of contribution.

$$\mathbf{P}\mathbf{C}1=-0.303{\mathbf{X}}_1-0.311{\mathbf{X}}_2-0.427{\mathbf{X}}_3-0.378{\mathbf{X}}_4-0.136{\mathbf{X}}_5+{0.438\mathbf{X}}_6-0.297{\mathbf{X}}_7+{0.420\mathbf{X}}_8+0.110{\mathbf{X}}_9$$

(5)

$$\mathbf{P}\mathbf{C}2={-0.056\mathbf{X}}_1-0.237{\mathbf{X}}_2-0.035{\mathbf{X}}_3-0.185{\mathbf{X}}_4+{0.647\mathbf{X}}_5-0.233{\mathbf{X}}_6+{0.353\mathbf{X}}_7+{0.143\mathbf{X}}_8+0.536{\mathbf{X}}_9$$

(6)

$$\mathbf{P}\mathbf{C}3=-0.298{\mathbf{X}}_1+0.624{\mathbf{X}}_2+0.182{\mathbf{X}}_3-0.386{\mathbf{X}}_4-0.321{\mathbf{X}}_5+{0.096\mathbf{X}}_6+0.243{\mathbf{X}}_7-{0.054\mathbf{X}}_8+0.407{\mathbf{X}}_9$$

(7)

$$\mathbf{P}\mathbf{C}4={-0.646\mathbf{X}}_1+0.037{\mathbf{X}}_2-0.313{\mathbf{X}}_3+0.003{\mathbf{X}}_4+{0.216\mathbf{X}}_5+0.006{\mathbf{X}}_6+{0.337\mathbf{X}}_7-{0.330\mathbf{X}}_8-0.462{\mathbf{X}}_9$$

(8)

$$\mathbf{P}\mathbf{C}5=0.322{\mathbf{X}}_1-0.300{\mathbf{X}}_2+0.079{\mathbf{X}}_3-0.200{\mathbf{X}}_4-0.290{\mathbf{X}}_5+{0.238\mathbf{X}}_6+0.736{\mathbf{X}}_7+{0.151\mathbf{X}}_8-0.234{\mathbf{X}}_9$$

(9)

Hence, the bootstrap on PCA is implemented on the data. The type of bootstrap is nonparametric method with 1000 Bootstrap samples that were drawn. Then, projection of the original matrix is done onto the bootstrap principal components, the results are discussed as follows.

Figure 2 and Figure 3 shows the observations as points in the plane formed by two principal components (synthetic variables), with the original variables appearing as vectors. The figures thus represent the interactions between variables, as well as highlighting the consistency of variable representation and the association between variables and measurements. Some factors are positively connected, such as Cereal, Nuts and Fruit and vegetables, with a high correlation between Fish, Eggs and Cereal, as represented in Figure 4.

In addition, Table 2 displays a matrix containing the proportions of accounted variance (columns) for each bootstrap sample (columns). The bootstrap mean in our observed samples for PC1 is equal to 46.95. The 95% confidence interval around the estimate of the variance difference, between test and control population variances, is $\left[38.577,56.741\right]$. While the 95% confidence interval around the estimate of the mean difference, between Test and control population means, is $\left[38.041,55.867\right]$. Now, with PC2, the bootstrap mean equals 19.77 and the 95% confidence interval of the mean difference and variance difference are $\left[13.63,25.91\right]$ and $\left[14.51,26.41\right]$, respectively.

3.2. Chemical data

Chemical data is an ecological data collected by Department of Ecology at the University of Leon, (Spain), and available in R package. The data consists of 324 observations and 16 variables. As first step, the PCA is implemented and the result is shown in Figure 5 It presents that 92.7% of the total variance is explained by five components, with PC1 explaining 51.1% of the total variance, and PC2 explaining 18.4% of the total variance. PC3 further explains 13% of the total variance, PC4 and PC5 explaining 6.2% and 4%, respectively, of the total variance. This offers a perspective on these components’ comparative relationships to the dataset, with just five PCs explaining 92.7% of the total variance. To evaluate the relationship between the multiple variables and the relevant PCs, Table 3 illustrates the squared factor loading for each variable, the total percentage of the variance that can be explained by the two components is around 69.5%. Consider the variables pH, ALKALINITYmeql, CO2free, NNH4mgl, NNO3mgl, SRPmglP, TPmgl, TSSmgl, CONDUCTIVITYmScm, TSPmglP and Chlorophyllamgl as represented by ${\mathbf{X}}_1,{\mathbf{X}}_2,{\mathbf{X}}_3,{\mathbf{X}}_4,{\mathbf{X}}_5,{\mathbf{X}}_6,{\mathbf{X}}_7,{\mathbf{X}}_8,{\mathbf{X}}_9$, ${\mathbf{X}}_{10},{\mathbf{X}}_{11}$ respectively. As illustration in Eq. (10) and Eq. (11), one can infer that highest contributing variables in PC1 are SRPmglP and TSPmglP while TSSmgl provides the highest negative level of contribution. In PC2, the highest contributing variables are pH and Chlorophyllamgl while SRPmglP provides a low negative level of contribution.

$$\mathbf{P}\mathbf{C}1=0.094{\mathbf{X}}_1+0.395{\mathbf{X}}_2-0.085{\mathbf{X}}_3+0.314{\mathbf{X}}_4+0.276{\mathbf{X}}_5+{0.403\mathbf{X}}_6+0.378{\mathbf{X}}_7{-0.106\mathbf{X}}_8+0.402{\mathbf{X}}_9+0.403{\mathbf{X}}_{10}+{0.114\mathbf{X}}_{11}$$

(10)

$$\mathbf{P}\mathbf{C}2={0.573\mathbf{X}}_1-0.044{\mathbf{X}}_2-0.496{\mathbf{X}}_3-0.202{\mathbf{X}}_4{-0.110\mathbf{X}}_5-0.029{\mathbf{X}}_6+{0.112\mathbf{X}}_7+{0.351\mathbf{X}}_8-0.086{\mathbf{X}}_9+0.005{\mathbf{X}}_{10}+{0.476\mathbf{X}}_{11}$$

(11)

$$\mathbf{P}\mathbf{C}3=0.365{\mathbf{X}}_1-0.077{\mathbf{X}}_2-0.486{\mathbf{X}}_3-0.012{\mathbf{X}}_4+0.258{\mathbf{X}}_5-0.052{\mathbf{X}}_6-0.205{\mathbf{X}}_7{-0.566\mathbf{X}}_8+0.016{\mathbf{X}}_9-0.077{\mathbf{X}}_{10}-0.432{\mathbf{X}}_{11}$$

(12)

$$\mathbf{P}\mathbf{C}4={0.082\mathbf{X}}_1-0.199{\mathbf{X}}_2+0.216{\mathbf{X}}_3-0.475{\mathbf{X}}_4{+0.726\mathbf{X}}_5+0.062{\mathbf{X}}_6+{0.050\mathbf{X}}_7+{0.334\mathbf{X}}_8+0.069{\mathbf{X}}_9+0.053{\mathbf{X}}_{10}-0.167{\mathbf{X}}_{11}$$

(13)

$$\mathbf{P}\mathbf{C}5=0.024{\mathbf{X}}_1+0.127{\mathbf{X}}_2+0.013{\mathbf{X}}_3+0.504{\mathbf{X}}_4+0.414{\mathbf{X}}_5-{0.363\mathbf{X}}_6-0.364{\mathbf{X}}_7{+0.160\mathbf{X}}_8+0.222{\mathbf{X}}_9-0.362{\mathbf{X}}_{10}+{0.295\mathbf{X}}_{11}$$

(14)

As we did earlier in the protein data, the bootstrap on PCA is implemented on the data the results are discussed as follows.

Figure 6 and Figure 7 shows the observations as points in the plane formed by two principal components (synthetic variables), with the original variables appearing as vectors. The figures thus represent the interactions between variables, as well as highlighting the consistency of variable representation and the association between variables and measurements. Some factors are positively connected, such as

NNH4mgl, NNO3mgl, SRPmglP, TPmgl, CONDUCTIVITYmScm and TSPmglP, with a high correlation in between, as represented in Figure 8.

In addition, Table 4 displays a matrix containing the proportions of accounted variance (columns) for each bootstrap sample (columns). The bootstrap mean in our observed samples for PC1 is equal to 51.32. The 95% confidence interval around the estimate of the variance difference, between test and control population variances, is $\left[49.330,53.395\right]$. While the 95% confidence interval around the estimate of the mean difference, between Test and control population means, is $\left[49.308,53.330\right]$. Now, with PC2, the bootstrap mean equals 18.465 and the 95% confidence interval of the mean difference and variance difference are $\left[16.997,19.932\right]$ and $\left[17.150,20.068\right]$, respectively.

4. Discussion

This paper has covered Principal component analysis and bootstrap method and their stability in data analytics employing the correlation matrix have all been explored. The visualization of PCA illustrates its expectations in the part of parameter uncertainties. As shown by results of Protein and chemical datasets the using of confidence limits based on the bootstrap method provides meaningful statements and answers that to be preferred over all matches situations.