Parallel Guiding Sparse Auto-Encoder with Wasserstein Regularization for Efficient Classification

1. Introduction

Efficient feature learning algorithms have become an essential element and the hot research topic in the era of big data [1]. Massive and high-dimensional data has generally a lot of redundancies and noise. It brings us huge challenges to extract important features, process them, and identify meaningful pattern. Many traditional methods are also fine in some cases. For example, many methods are based on linear relationships. It is intuitive and easily understandable. However, there are also non-linear data structures. Therefore, the machine learning-based model can be a solution to explain the non-linear structure. Autoencoder [2,3,4,5] is a representative machine learning technique for finding manifolds that represent large-scale data and is employed vastly in the field of feature extraction and dimensionality reduction. It consists of two parts: an encoder and a decoder. The encoder transforms the input into a latent vector that contains the main information. The decoder reconstructs the latent vector to new features same as the input form. It learns to minimize the reconstruction error. This method has become one of the useful feature engineering methods.

Feature engineering is a process of extracting important information from raw data and can be generally divided into two types: feature selection [6,7,8] and feature extraction [9,10]. Feature selection can be explained as a method of finding a subset of features with higher feature importance for classification. Therefore, it is possible to effectively remove less relevant features. These methods generally have a high generalization performance and computational efficiency. For example, there is a regularization [11,12], information gain [13], chi-square test [14], etc. However, this method has the disadvantage of removing potential non-noise factors of the original data. On the other hand, feature extraction methods are usually based on the reconstruction of features and projection from high-dimensional data into low-dimensional subspace. This kind of algorithm normally preserves the variance of features, so it has high information retention. However, since these algorithms consider almost all features in the transformation process, the dimensionality reduction effect is usually depleted. Principal component analysis (PCA) [15,16], canonical correlation analysis (CCA) [17], linear discriminant analysis (LDA) [18,19], and autoencoder using machine learning models for feature extraction are representative methods.

However, autoencoder has overfitting problems and highly complex structural issues. Improved structures have been proposed in recent years to solve these problems. Peicheng et al. [20] applied a stacked autoencoder (SAE) to hyperspectral image classification to generate high-quality hyperspectral image data to maximize the performance of the task. Meidi et al. [21] applied a stacked autoencoder to fault diagnosis of rolling bearings and obtained promising results. They increase the size of the network to get a better result. However, the autoencoder-based simple method is still a field of feature extraction, so it is difficult to modify and remove features with less relevance, which is the advantage of feature selection. To improve this shortcoming, new techniques applying regularization have been studied. Especially, stacked sparse autoencoder [22,23,24] is one of the representative deep autoencoder methods which have feature selection ideas. Stacked sparse autoencoder (SSAE) improved the autoencoder by a combination of two advanced autoencoders: a sparse autoencoder and stacked autoencoder. A sparse autoencoder adds sparse constraints to the autoencoder. It restricts the useless node to reduce the dimension efficiently. A stacked autoencoder improves the training performance by stacking autoencoders. It mediates to deeply learn the features in comparison to the simple learning process of traditional autoencoders. Anush et al. [25] proposed the structure of group sparse autoencoder (GAE) with regularization applied to solve the overfitting, which is a chronic problem of autoencoders.

Although SSAE’s regularization-based methods achieved successful results in various applications, due to its structural issues, it is difficult to apply the dynamic and efficient feature selection method. In many cases, there are vanishing gradient problems and over-generalizing information (e.g., on sensitive applications). Zhilei et al. [26] proposed a label and sparse regularization autoencoder (LSRAE). The main idea of LSRAE is to learn by dividing the output layer of the autoencoder into two parts, the actual label and output data, to extract features that involve the relationship between features and targets, and to prevent overfitting via sparse regularization. Most of the regularization-based autoencoders optimize the loss function without considering the structure. However, LSRAE has structural changes and the advantages of supervised learning. This is an important point in the feature engineering field. Hongteng et al. [27] proposed a relational regularized autoencoder (RAE) that flexibly learns a structured prior distribution using Fused Gromov-Wasserstein (FGW) [28] distance. The main idea is to limit the difference by regularizing the structural distance between the latent prior and the corresponding posterior using a metric that determines the distribution similarity. Therefore, to obtain high-quality and low-dimensional features, the structure of SSAE needs to be improved and reformed.

In this paper, we propose a parallel guiding sparse autoencoder (PGSAE) which is a deep advanced autoencoder model. PGSAE is a solution to the above-mentioned problems. The proposed PGSAE focuses on two parts: advanced structure and regularization. It is a structure that can effectively utilize feature importance analysis, which is the advantage of feature selection, in the autoencoder. There are two guiding layers before and after the reconstruction layer (output layer of the autoencoder) to adjust the leverage of feature importance. Guiding layers are for regularization of the autoencoder and limit their distributive difference by using Wasserstein distance to improve the performance of regularization. Therefore, it is a new manifold learning method that can improve the quality of deep features and obtain low-dimensional features.

2. Background and Related works

In this section, we describe the autoencoder based models.

2.1. Sparse Autoencoder (SAE)

An autoencoder consists of two parts: an encoder and a decoder. In the encoder step, autoencoder transforms the original feature into latent representation. It is model-interpretable information. In the decoder step, autoencoder reconstructs the latent representation into new meaningful features.

An encoding function $f$ maps the input data sample $x\in {ℝ}^m$ to a low-dimensional latent feature space ${ℝ}^n$. It is formulated as:

$$h=f\left(x\right)=\varphi \left(W_1^{T}x+b_1\right)$$

(1)

where $W_1$ is the weight matrix between the input layer and hidden layer, $\varphi$ is the activation function, $b_1$ is a bias and $h$ is a latent vector of the hidden layer. A decoding function $g$ maps $h$ to reconstructed feature $x^{\prime }$. It can be expressed as:

$$x^{\prime }=g\left(x\right)=\varphi \left(W_2^{T}h+b_2\right)$$

(2)

where $W_2$ is the weight matrix between the hidden layer and output layer. The purpose of the autoencoder is to find a structure that minimizes the reconstruction error, and the objective function is defined as follows:

$$\underset{\theta }{\min }{J}_{AE}\left(\theta \right)=\frac{1}{2N}{\displaystyle \sum }_{i=1}^N\parallel x_i-x_i^{\text{'}}\parallel {}^2$$

(3)

where $N$ is a amount of input samples. SAE’s structure is based on autoencoder. However, in basic autoencoder, when reconstructing input, the distribution of reconstructed features has a lot of overlap due to problems such as overfitting [29] and simply copy input and may not represent key information well. To solve these problems, we add a sparsity constraint as a penalty term that can suppress the output of neurons when the number of neurons in the hidden layer is large and formulated as follows:

$$\underset{\theta }{\min }{J}_{SAE}\left(\theta \right)=\frac{1}{N}{\displaystyle \sum }_{i=1}^N\parallel x_i-x_i^{\text{'}}\parallel {}^2+\epsilon {\displaystyle \sum }_{j=1}^{s}KL(\rho \parallel {\widehat{\rho }}_j)$$

(4)

where ${\widehat{\rho }}_j$ is the activation probability, $\rho$ is the sparse parameter. KL divergence is briefly a measure of how differences between two probability distributions [30].

2.2. Label and sparse regularization autoencoder (LSRAE)

Zhilei et al. proposed LSRAE, a structure that combines supervised learning methods with the autoencoder, which is unsupervised learning, to improve the performance of classification tasks. They thought that the overfitting and the irregular information problem of classification, which are chronic problems of autoencoders, occur because the relation between the target data and the input data is not considered at all in the reconstruction process. Therefore, they solved the problem by introducing the two main methods: First is adding a label regularization term to the objective function to reduce the difference between the actual label and the desired label and second is using the extreme learning machine [31] for fast optimization. LSRAE with label regularization term $J_{label}$ added is based on sparse autoencoder and formulated as follows:

$$J_{label}=\frac{1}{2n}{\displaystyle \sum }_{i=1}^n{\parallel L-T\parallel }^2$$

(5)

where $L$ is the actual label as the output of autoencoder, $T$ is the desired label and $n$ is the number of input samples. Figure 1 shows a way to utilize the extreme learning machine (ELM) on autoencoder. Blue lines and red lines connect the hidden layer and the output layer and the label layer, respectively. ELM is utilized to obtain the output layer. It improves the classification accuracies. And they also use the sparse constraint to restrict useless nodes. The simple objective function of LSRAE is calculated as follows:

$$J_{sparse}={\displaystyle \sum }_{j=1}^{s}KL(\rho \parallel {\widehat{\rho }}_j)$$

(6)

$$J_{LSRAE}=J_{AE}+\mathsf{\lambda }{J}_{label}+\mathsf{\beta }{J}_{sparse}$$

(7)

where $J_{AE}$ is the loss function of the basic autoencoder.

3. Parallel guiding sparse autoencoder

Autoencoder has traditionally been used as one of the dimensionality reduction methods such as PCA and LDA. However, when overfitting occurs in the learning process, the following two major problems occur. First, even if unrelated data is used as input, it transforms the data to similar distribution with a training set. Second, while learning the information of features with low importance, a noise can be recognized as important information which affects classification accuracy. To solve the problem, we propose a parallel guiding sparse autoencoder (PGSAE) that learns the manifold of selected features and guides it to the rest of the features.

As shown in Figure 2, PGSAE has a structure with two added guiding layers to the basic autoencoder so that the two layers are parallel. From left to right, it is composed in the order of input layer, hidden layer (with guiding layer), reconstruction layer, guiding layer (output layer). The whole process is explained in small five processes. In the first process of PGSAE, $H$, $G_1$ are output of hidden units and output of first guiding units, respectively. The step to calculate $H$ and $G_1$ can be represented as follows:

$$H=\varphi \left(W_{1}X+b_1\right)$$

(8)

$$G_1=\varphi \left(W_1{}^{\text{'}}X+b_1{}^{\text{'}}\right)$$

(9)

where $X=\left\{x_1, x_2, \dots , x_n\right\}$ and $X_g=\left\{x_{n+1}, x_{n+2}, \dots , x_m\right\}$ are input data, $\varphi$ is activation function, $W_1, W_1{}^{\text{'}}$ and $b_1, b_1{}^{\text{'}}$ are the weights and biases between the first and second layers, respectively. $X$ denotes a guided feature and $X_g$ denotes a guiding feature. The learning process of PGSAE can be divided into two parts: (a) to learn the guiding feature as a target from the guided feature to involve the relation between the two features, and (b) to reconstruct the guided feature. A pretrained model from guided feature $X$ to guiding feature $X_g$ is required to involve the relation between the features. The corresponding part of PGSAE maps the guided feature of the input layer $X$ to the guiding feature of the first guiding layer $G_1$ and is pretrained partially. The cost function $J_{fg}$ of partial pretrained model is formulated as follows:

$$\begin{array}{ll}{J}_{fg}& =\frac{1}{2n}\parallel X_g-G_1\parallel {}^2 \\ & =\frac{1}{2n}\parallel X_g-\varphi {\left(W_1{}^{\text{'}}X+b_1{}^{\text{'}}\right)\parallel }^2\end{array}$$

(10)

The cost function used for pretraining is also used as a regularization term for the objective function in Eq. (18).

In the second process, as the number of units in the hidden layer increases, there may be too much sparse information. As a sparse constraint, the average activity of the hidden unit is calculated. If it is an important unit, it is activated, and if not, it is deactivated. The sparsity criterion for determining the importance of a unit uses the Kullback-Leibler (KL) divergence. KL divergence is the relative entropy to measure the similarity between a sparsity parameter $\rho$ and the average activation probability ${\widehat{\rho }}_j$. $s$ is the number of the hidden units. The average activation ${\widehat{\rho }}_j$ is represented as

$${\widehat{\rho }}_j=\frac{1}{n}{\displaystyle \sum }_{i=1}^n\varphi \left(x_i\right)$$

(11)

And $J_{sparse}$(KL divergence) is represented as

$$\begin{array}{ll}{J}_{sparse}& ={\displaystyle {\displaystyle \sum }_{j=1}^s}KL(\rho ||{\widehat{\rho }}_j) \\ & ={\displaystyle {\displaystyle \sum }_{j=1}^s}\rho \log \frac{\rho }{{\widehat{\rho }}_j}+\left(1-\rho \right)\log \frac{1-\rho }{1-{\widehat{\rho }}_j}\end{array}$$

(12)

In the third process, there is a reconstruction loss $J_{AE}$ and the loss can be calculated by the output vector of the reconstruction layer $X^{\prime }$. The output of the reconstruction layer is calculated as follows:

$$\begin{array}{ll}{X}^{\prime }& =\varphi \left(W_{2}H+b_2\right)+\varphi \left(W_2{}^{\text{'}}G+b_2{}^{\text{'}}\right) \\ & =\varphi \left(W_2\varphi \left(W_{1}X+b_1\right)+b_2\right)+\varphi \left(W_2{}^{\text{'}}\varphi \left(W_1{}^{\text{'}}X+b_1{}^{\text{'}}\right)+b_2{}^{\text{'}}\right)\end{array}$$

(13)

where $\varphi$ is activation function and $W_2, W_2{}^{\text{'}}$ and $b_2, b_2{}^{\text{'}}$ are the weights and biases between the hidden (first guiding) layer and reconstruction layers, respectively. The term $J_{AE}$ is a typical loss used in the existing autoencoder and is the main loss of PGSAE. In general, the mean squared deviation is used for linear cases, and cross-entropy is used for non-linear cases. In this section, we compute using mean squared deviation. $J_{AE}$ is computed as follows:

$$\begin{array}{ll}{J}_{AE}& =\frac{1}{2n}\parallel X-{X^{\prime }\parallel }^2\\ & =\frac{1}{2n}\parallel X-\varphi \left(W_2\varphi \left(W_{1}X+b_1\right)+b_2\right)+\varphi {\left(W_2{}^{\text{'}}\varphi \left(W_1{}^{\text{'}}X+b_1{}^{\text{'}}\right)+b_2{}^{\text{'}}\right)\parallel }^2\end{array}$$

(14)

In the fourth process, PGSAE has a second guiding part. Its operation step is almost the same to get the first guiding layer and two guide layers operate complementarily. The second guiding regularization term $J_{sg}$ is obtained by $W_3$ and calculated as follows:

$$\begin{array}{ll}{G}_2& =\varphi \left(W_3{X}^{\prime }+b_3\right) \\ & =\varphi \left(W_3\left(\varphi \left(W_2\varphi \left(W_{1}X+b_1\right)+b_2\right)+\varphi \left(W_2{}^{\text{'}}\varphi \left(W_1{}^{\text{'}}X+b_1{}^{\text{'}}\right)+b_2{}^{\text{'}}\right)\right)+b_3\right)\end{array}$$

(15)

$$\begin{array}{ll}{J}_{sg}& =\frac{1}{2n}\parallel X_g-G_2\parallel {}^2 \\ & =\frac{1}{2n}\parallel X_g-\varphi {\left(W_3\left(\varphi \left(W_2\varphi \left(W_{1}X+b_1\right)+b_2\right)+\varphi \left(W_2{}^{\text{'}}\varphi \left(W_1{}^{\text{'}}X+b_1{}^{\text{'}}\right)+b_2{}^{\text{'}}\right)\right)+b_3\right)\parallel }^2\end{array}$$

(16)

The guiding term $J_{guide}$ is computed by two guiding terms Eq. (3), (9), and formulated as follows:

$$J_{guide}=J_{fg}+J_{sg}$$

(17)

According to the structure of Figure 2. described above, the objective function of PGSAE using the above three equations (5), (7), (10), and Wasserstein regularization term $J_{was}$ Eq. (12) is defined as:

$$\begin{array}{ll}\underset{w\in W }{\min }{J}_{PGSAE}& =\underset{w\in W }{\min }{J}_{AE}+\epsilon J_{sparse}+\lambda J_{guide}+\mu J_{was }\\ & =\underset{w\in W }{\min }\frac{1}{2n}\parallel X-{X^{\prime }\parallel }^2+\epsilon {\displaystyle {\displaystyle \sum }_{j=1}^s}KL(\rho \parallel {\widehat{\rho }}_j)+\lambda \frac{1}{2n}\parallel X_g-G_2\parallel {}^2\\ & +\frac{1}{2n}\parallel X_g-G_1\parallel {}^2+\mu \underset{r\in \prod \left(P_{{\theta }_1}, P_{{\theta }_2}\right)}{\inf }{\mathrm{E}}_{\left(X, H\right)~\gamma }\left[d\left(G_1, G_2\right)\right]\end{array}$$

(18)

where $W$ is the weight vector $\left\{W_1, W_2, W_3, W_1{}^{\text{'}}, W_2{}^{\text{'}}\right\}$, three coefficients$\epsilon$, $\lambda$ and $\mu$ are to balance the $J_{PGSAE}$, $P_{{\theta }_1}$ is the pretrained prior distribution which we assume it as real data distribution and $P_{{\theta }_2}$ is the latent variable distribution. The Wasserstein regularization term $J_{was}$ is explained in the next part and shown as:

$$J_{was}=\underset{r\in \prod \left(P_{{\theta }_1}, P_{{\theta }_2}\right)}{\inf }{\mathrm{E}}_{\left(X, H\right)~\gamma }\left[d\left(G_1, G_2\right)\right]$$

(19)

where $J_{was}$ Eq. (12) is the lowest estimate of the expected value of $\prod \left(P_{{\theta }_1}, P_{{\theta }_2}\right)$ among all joint probability distributions $d\left(G_1, G_2\right)$. However, the infimum in Eq. (12) is too intractable and complex. Kantorovich-Rubinstein duality suggests us the solution to replace the problem with a low complexity duality problem. It can be formulated as:

$$\underset{w\in W }{\min }{J}_{PGSAE}=\underset{w\in W }{\min }{J}_{AE}+\epsilon J_{sparse}+\lambda J_{guide}+\mu \underset{f\in L}{\sup }{\mathrm{E}}_{x~P_{{\theta }_1}}\left[f\left(x\right)\right]-{\mathrm{E}}_{x~P_{{\theta }_2}}\left[f\left(x\right)\right]$$

(20)

where $L$ means all ranges that satisfy 1-Lipschitz function and $f$ is a function about ${\theta }_1$ that satisfies 1-Lipschitz function. By adding a weight clipping parameter to the $f$ function to fit the duality condition, it can be expressed as follows:

$$\underset{w\in W }{\min }{J}_{PGSAE}=\underset{w\in W }{\min }{J}_{AE}+\epsilon J_{sparse}+\lambda J_{guide}+\mu \underset{w\in W}{\max }{\mathrm{E}}_{x~P_{{\theta }_1}}\left[f_w\left(x\right)\right]-{\mathrm{E}}_{x~P_{{\theta }_2}}\left[f_w\left(g_{{\theta }_2}\left(x\right)\right)\right]$$

(21)

PGSAE assumes that we already have good performance $f_w$ throught pretraining. Ultimately our remaining goal is to find the optimum to update $W_1$, $W_2$, $W_2{}^{\text{'}}$ through the Wasserstein regularization. That is, it should find optimal $g_{{\theta }_2}$. Furthermore, we could consider and compute differentiating the changed $J_{was}$ through the back-propagation method in Eq. (14). via getting $\underset{w\in W}{\max }{\mathrm{E}}_{x~P_{{\theta }_1}}\left[f_w\left(x\right)\right]$. According to the above methods, using back-propagation of the entire objective function is possible by gradient descent algorithm. In the partial derivative formula to optimize $J_{PGSAE}$, bias is omitted to simplify the expression.

The weights $W$ is formulated in the order as follows:

$$\begin{array}{ll}\frac{\partial J_{PGSAE}}{\partial W_1}& =\frac{\partial J_{AE}}{\partial W_1}+\epsilon \frac{\partial J_{sparse}}{\partial W_1}+\lambda \frac{\partial J_{guide}}{\partial W_1}+\mu \frac{\partial J_{was}}{\partial W_1}\\ & =\frac{\partial J_{AE}}{\partial W_1}+\epsilon \frac{\partial J_{sparse}}{\partial W_1}+\lambda \frac{\partial J_{sg}}{\partial W_1}+\mu \frac{\partial J_{was}}{\partial W_1}\end{array}$$

(22)

Since $W_1$ is not affected by $J_{fg}$, when differentiation is performed, it is removed from $J_{guide}$ and only $J_{sg}$ remains. $W_2$ is computed as follows:

$$\begin{array}{ll}\frac{\partial J_{PGSAE}}{\partial W_2}& =\frac{\partial J_{AE}}{\partial W_2}+\epsilon \frac{\partial J_{sparse}}{\partial W_2}+\lambda \frac{\partial J_{guide}}{\partial W_2}+\mu \frac{\partial J_{was}}{\partial W_2}\\ & =\frac{\partial J_{AE}}{\partial W_2}+\lambda \frac{\partial J_{sg}}{\partial W_2}+\mu \frac{\partial J_{was}}{\partial W_2} \end{array}$$

(23)

Since $W_2$ is the weight between the first guiding layer and the reconstruction layer, it is not affected by $J_{sparse}$. Therefore, $J_{sparse}$ is removed. $W_1{}^{\text{'}}$ is computed as follows:

$$\begin{array}{ll}\frac{\partial J_{PGSAE}}{\partial W_1{}^{\text{'}}}& =\frac{\partial J_{PGAE}}{\partial W_3} \\ & =\left[\frac{\partial J_{AE}}{\partial W_1{}^{\text{'}}}+\epsilon \frac{\partial J_{sparse}}{\partial W_1{}^{\text{'}}}+\lambda \frac{\partial J_{guide}}{\partial W_1{}^{\text{'}}}+\mu \frac{\partial J_{was}}{\partial W_1{}^{\text{'}}}\right]\\ & +\left[\frac{\partial J_{AE}}{\partial W_3}+\epsilon \frac{\partial J_{sparse}}{\partial W_3}+\lambda \frac{\partial J_{guide}}{\partial W_3}+\mu \frac{\partial J_{was}}{\partial W_3}\right]\\ & =\lambda \frac{\partial J_{guide}}{\partial W_1{}^{\text{'}}}+\mu \frac{\partial J_{was}}{\partial W_3} \end{array}$$

(24)

Since $W_1{}^{\text{'}}$ and $W_3$ must have the adjacent gradient value and are updated in parallel, the sum of partial derivatives with respect to $W_1{}^{\text{'}}$ and partial derivatives with respect to $W_3$ is used as gradients and updated simultaneously. $W_2{}^{\text{'}}$ is computed as follows:

$$\begin{array}{ll}\frac{\partial J_{PGSAE}}{\partial W_2{}^{\text{'}}}& =\frac{\partial J_{AE}}{\partial W_2{}^{\text{'}}}+\epsilon \frac{\partial J_{sparse}}{\partial W_2{}^{\text{'}}}+\lambda \frac{\partial J_{guide}}{\partial W_2{}^{\text{'}}}+\mu \frac{\partial J_{was}}{\partial W_2{}^{\text{'}}}\\ & =\frac{\partial J_{AE}}{\partial W_2{}^{\text{'}}}+\lambda \frac{\partial J_{guide}}{\partial W_2{}^{\text{'}}}+\mu \frac{\partial J_{was}}{\partial W_2{}^{\text{'}}} \end{array}$$

(25)

Through the above methods, the main steps of PGSAE algorithm which is the guiding feature learning method are summarized as follows:

4. Experimental result of PGSAE

4.1. Datasets and experimental setups

In this paper, experiments are conducted using six public datasets with various dimensionality: Heart Disease (Heart) dataset, Wine quality (Wine) dataset, U.S. Postal Service (USPS) dataset, Fashion-MNIST dataset, Yale face (Yale) dataset and Columbia Object Image Library (COIL-100) dataset. The features of the dataset are divided into low-dimensional features, medium-dimensional features, and high-dimensional features. Brief information of the dataset is summarized in Table 1. We carry out the experiments on a personal server with the following configurations: two CPUs (AMD 7742 64-core), eight GPUs (NVIDIA DGX A100), memory (1 TB), and a NVMe Storage (15 TB).

The proposed PGSAE has three regularization terms: a parallel guiding term ($J_{guide}$), a sparse regularization term ($J_{sparse}$), and Wasserstein regularization term ($J_{was})$ in Eq. (18). The hyperparameters $\epsilon , \lambda$, and $\mu$ for the regularization term are defined, and we need to find the optimal coefficient suitable for different datasets. In particular, the parallel guiding coefficient $\lambda$ and Wasserstein regularization parameter $\mu$ consider the leverage of the guiding feature. Thus, the classification performance depends on the sensitively tuned hyperparameters. The structure of PGSAE that contains guiding layers and the coefficients in this experiment are summarized in Table 2.

4.2. Authentication of main innovation

The main innovation of this study is to solve the overfitting problem of the autoencoder model, which is one of the dimensionality reduction algorithms, by applying a new regularization technique based on Wasserstein distance and guide layers. Overfitting of autoencoder is mainly divided into two categories: Learning the distribution of the identity function and transforming unrelated data like training data [32,33,34,35]. To solve these two overfitting categories, experiments were conducted in two phases. First, we designed an experiment to prove that PGSAE is not learning the distribution of identity functions. Second, the experiment was conducted to test whether the deep features extracted from PGSAE could potentially express main information rather than simply transforming the input data into the distribution of the training data to reduce the reconstruction error. In this section, an ablation study was applied to prove the performance of the basic structure of PGSAE and the effects of sparse regularizer and Wasserstein regularizer. Table 3 shows the results of using SVM as a classifier. The symbols in Table 3 are explained as follows. RF: raw features; DF: deep features trained by PGSAE without Wasserstein regularization and sparse regularization; DF & WR: deep features with Wasserstein regularization; DF & WR & SR: deep features with Wasserstein regularization and sparse regularization. As shown in Table 1 and Table 3, in most datasets, our methods have low-dimensional features compared to conventional RF, but the benchmark results show no significant performance degradation. DF & WR (DW) and DF & WR & SR (DWS) showed excellent results as follows: on the Heart, DWS shows 0.17% higher performance; on the Wine, DWS shows 0.59% lower performance; on the USPS, DW shows 1.42% lower performance; on the Fashion-MNIST, DWS shows 1.73% lower performance; on the Yale, DWS shows 3.21% lower performance; on the COIL-100, DWS shows 0.86% lower performance. In most cases, the performance slightly decreases, but the training efficiency increases a lot as follows: on the Heart, the efficiency increases about 31% from 13 features to 9; on the Wine, the efficiency increases about 33% from 12 to 8; on the USPS, the efficiency increases about 29% from 256 to 180; on the Fashion-MNIST, the efficiency increases about 29% from 784 to 549; on the Yale, the efficiency increases about 30% increase from 1024 to 717; on the COIL-100, the efficiency increases about 30% increase from 1024 to 717. Most of the results show a high learning efficiency of about 30%. Therefore, this strategy has been proven to be effective and can improve the performance and stability of the existing autoencoder model.

4.3. Comparison of the benchmark datasets

The efficiency of the proposed PGSAE is proven in Section 4.2, and in this section, we experiment to prove that PGSAE shows better performance than existing dimensionality reduction algorithms and does not learn the distribution of identity function. When the autoencoder model learns the distribution of the identity function to minimize the reconstruction error, it is difficult to expect improvement in classification because there are few transformations of the feature. Therefore, if the classification accuracy is improved compared to the result of using only the raw features by the hybrid feature of the deep features extracted from PGSAE and the raw features, it shows that our PGSAE is not overfitted without learning the distribution of the identity function. In Table 4, Hybrid PGSAE showed improvement in accuracy of 0.75%, 1.09%, 0.45%, 0.48%, 2.02%, and 0.57% for 6 datasets, respectively. This means that PGSAE does not learn the distribution of identity functions. In addition, Table 4 illustrates how much performance improvement of PGSAE was achieved compared to the existing feature extraction method through comparison experiments with representative dimensionality reduction algorithms. For comparison, the proposed PGSAE is compared with representative autoencoder models and dimensionality reduction methods as follows: Kernel PCA, LDA, stacked autoencoder (SAE), stacked sparse autoencoder (SSAE), uniform manifold approximation and projection (UMAP) and t-Stochastic neighbor embedding (t-SNE) [36,37,38]. As shown in Table 4, the proposed method in this study showed superior achievement in most cases compared to other methods on the six data sets. The PGSAE on the datasets Heart, USPS, Fashion-MNIST, and COIL-100 shows the best outcome compared to other dimensionality reduction techniques, and on the datasets Wine and Yale shows slightly lower performance at 0.62% and 0.13%, respectively, compared to the best algorithms. In addition, the accuracy of Hybrid PGSAE on the dataset Heart is 0.75% and 0.58% higher than that of RF and the best algorithm, respectively. In addition, the performance of Hybrid PGSAE on the Heart dataset is higher than RF and the best algorithm by 0.75% and 0.58%, respectively. Similarly, performance increase shows 1.09% and 1.06% on the Wine, 0.45% and 1.87% on the USPS, 0.48% and 2.21% on the Fashion-MNIST, 2.02% and 5.23% on the Yale, and 0.57% and 1.43% on the COIL-100, respectively to RF and the best algorithm. This means that the proposed method solved the overfitting issue of the autoencoder and displayed superior results.

4.4. Prevention of transforming irrelevant data as like training data

In Section 4.3, we proved that PGSAE does not learn the distribution of identity functions, but basically, the autoencoder performance relies on reconstruction quality, which is the most important component in the encoding-decoding process. The reconstruction error can still be an insufficient indicator due to other types of overfittings, and experiments to supplement are necessary. In this section, we compare the reconstruction error with the actual image simultaneously and show that the proposed PGSAE solves the overfitting that transforms the data unrelated to the training set into the distribution of the training set. In Figure 3, the experimental setup of the PGSAE on the datasets Fashion-MNIST and COIL-100 is illustrated in Table 2. In addition, Gaussian noise was used as an extra test set to confirm the transformation of the data distribution. In Figure 3, the PGSAE shows better restoration ability than the basic autoencoder model for detailed information. However, in Table 5, the reconstruction error of the PGSAE on the Fashion-MNIST and COIL-100 datasets is higher than the basic autoencoder. It means that the overfitting occurred because the basic autoencoder, which has lower reconstruction error, shows lower performance than the PGSAE. In Figure 3, the result of using Gaussian noise as an input value shows that most of the noise information is vaguely expressed in the case of a basic autoencoder. When there are many similar data in the training set, the reconstruction error can be reduced by transforming the information in this way, but the main information can be lost. PGSAE shows that this overfitting is improved through the guiding layer, Wasserstein regularization, and sparse regularization.

4.5. Effect of the coefficients and classifiers

4.5.1. Coefficients Optimization on the six datasets

In this experiment, two optimization techniques are used to find the optimum of the three coefficients $\epsilon , \lambda$ and $\mu$ to be defined in PGSAE. The first one is a grid search algorithm that searches all values ​​within a specific range and the second is a Bayesian optimization algorithm that probabilistically finds the optimal parameter composed of a surrogate model (Gaussian process) and an acquisition function [39,40,41]. Figure 4 shows the experimental results for coefficients $\epsilon , \lambda$ and $\mu$ using grid search, and the ranges are $\left\{0.005\times \mathrm{e}|\mathrm{e}=1, 2, 3, \dots , 10\right\}, \{0.01\times \mathrm{e}|\mathrm{e}=1, 2, 3, \dots , 10\}$ and $\{0.001\times \mathrm{e}|\mathrm{e}=1, 2, 3, \dots , 10\}$. Since $\lambda$ is relatively more influential than $\epsilon$ and $\mu$, we reduce the range of the sensitive coefficient by the results of grid search and additionally test the coefficient using Bayesian optimization algorithm to get a suitable value efficiently. The six datasets are split into two subsets: one accounting for about one-fifth of all samples as test data and the rest as the training data. We ran tests five times with randomly shuffled samples. As a classifier, SVM is chosen to optimize all coefficients. Figure 4 (a) shows the results of the parameter experiment on the Heart dataset. As the Wasserstein regularization parameter increases, the classification accuracy decreases. It shows superior outputs for values between about 0.003 and 0.004. The $\lambda$ is the most important coefficient for extracting core features, and since the performance varies greatly depending on the data, the optimal value should be obtained through experiments. If it is too low, it causes a huge loss of the guiding feature and low accuracy of classification. Or if $\lambda$ is too high, the weights are updated to improve the performance of guiding layers only. It brings a high reconstruction error of the PGSAE. The $\mu$ is the role to prevent overfitting. If $\mu$ is too low, the first guiding and second guiding layers are not interactive. In this case, the imbalanced guiding layers are updated only to reduce the reconstruction error thus overfitting occurs easily. If $\mu$ is too high, the first guiding layer and the second guiding layer become almost the same. It is too stiff to get optimal and leads to high reconstruction error. The $\epsilon$ is to filter sparse nodes out. It may increase or decrease the performance depending on the data and hidden unit. In Figure 4 (c), PGSAE showed better results when sparse regularization is not applied. Therefore, the parameter optimization process to get useful features is imperative in PGSAE. Figure 4 (b), (c), (d), (e), and (f) are the optimization result of Wine, USPS, Fashion-MNIST, Yale and COIL-100, respectively.

4.5.2. Effects of classifier

The classifier has a significant impact on the performance of PGSAE. We conducted an experiment using the following three representative classifiers: support vector machine (SVM), K-nearest neighbor (KNN), and random forest [42]. The experimental results are shown in Table 6. As a result, KNN shows poor performance in most cases in the proposed method. The random forest achieves higher accuracies than KNN in most cases and it is the highest accuracy on the Yale. However, in this experiment, it does not show optimal performance because all hyperparameters are set as default. Since SVM can be applied most stably to the proposed algorithm and shows excellent accuracies in most cases, it is chosen as a final classifier for PGSAE. Since the proposed PGSAE is in a field of dimensionality reduction and feature extraction domain, further experiments on different classifiers have not been carried out. However, if a more suitable and optimized classifier is used in the future work, performance can be improved, and it can be extended to other tasks.

5. Conclusion and future work

In this paper, we introduced a new method called PGSAE that works with parallel guiding layers. It is a dimensionality reduction method that combines the ideas of feature selection and feature extraction. There are two key points. The first one is using guiding layers to flexibly update the weights. It offers the advantages of feature selection and manages the effects of each feature to reduce the classification performance loss. The second one is a constraint method by Wasserstein distance and KL divergence. Wasserstein distance prevents overfitting in guiding layers and KL divergence restricts the useless nodes of the network. These two main ideas interactively perform to extract the main information of the origin feature and reduce the dimensionality. In the experiment, we compared PGSAE with other traditional dimensionality reduction methods and showed the performance differences according to classifiers on the Pima, heart, wine, and musk datasets. The proposed method outperforms other methods on average and shows the highest classification accuracies due to the coefficients of $J_{guide}$ and $J_{was}$ are efficiently optimized through the experimental tests.

Feature work includes improving PGSAE in two ways. The first is through optimization methods. PGSAE is a sensitive model. Therefore, the optimization process is necessary. Thus, the next step is to generalize this model to utilize in other applications. The second one is by applying visualization using convolutional layers to ensure compatibility for image data and high dimensional data representation. Because dimensionality reduction methods are used not only for efficiency and performance but also for visualization to show a representative data structure.