ASISO: A Robust and Stable Data Synthesis Method to Optimize Samples

1. Introduction

    Synthetic data presents an effective solution to the challenges of inadequate or low-quality sample, particularly in the era of big data. Through the use of data synthesis methods to generate synthetic data, it provides a cost-effective and efficient alternative to collecting and labeling large amounts of real-world data. Furthermore, these methods can address privacy concerns associated with real-world data, making it safer to share and analyze [1]. Recently, there has been a surge in the use of synthetic data in machine learning, with various synthetic methods being developed [2,3].

Representative data synthesis methods can be classified into three categories. The first category entails techniques such as interpolation, extrapolation, and other methods [4] that generate additional data points representative of the underlying distribution. These methods improve the quality of the dataset and model generalization. For image data, deep learning-based methods such as VAE and GAN can also be used to generate new data [5,6]. These methods are useful for improving the quality of the dataset and model generalization. The second category is to address the issue of dataset imbalance, such as SMOTE [7], and some of its enhancements [8,9,10]. These methods can synthesize minority class samples to balance the dataset and improve the model’s performance. Finally, data sharing and research require sensitive data privacy protection. Synthetic data can help protect sensitive information while still enabling data sharing and research. Normally, we can add random noise to protect original data, generating synthetic data for sharing and research purposes like differential privacy methods [11]. Overall, these methods help optimize the representation and use of data in various applications.

For some actual tasks, original data often contains a multitude of features and unknown noise [12]. Since synthetic method involves generating new data based on existing data, the quality of the new points depends on the quality and quantity of the original data. If the quality of the original data is poor or the quantity is insufficient, then the synthetic data may have limitations in terms of its quality [13]. However, there exists a lack of effective synthetic methods for datasets with a restricted size and complex noise to expand the size of data set.

Based on the purpose of improving the quality and quantity of the dataset, and motivated by piecewise linear interpolation and spline interpolation, we propose a robust and stable data synthesis method named Adaptive Subspace Interpolation for Sample Optimization (ASISO), which aims to adaptively adjust the sample size and structure of the original dataset containing unknown noise. The idea is to divide the original feature space into several subspaces with an equal number of samples, and then perform linear interpolation for the samples in the adjacent subspaces. This method achieves sample optimization in two aspects. First, it can adaptively adjust the size of the dataset, and the expanded data typically contains minimal error with actual. Second, it adjusts the structure of the samples, which can significantly reduce the proportion of samples with large errors, thereby minimizing the impact of noise for model generalization.

The rest of this paper is organized as follows. Section 2 reviews the research of existing interpolation. Section 3 details the concept of the proposed ASISO method and provides proof for the effectiveness of this method. Experimental results are presented and analyzed in Section 4. Finally, we conclude this paper in Section 5.

2. Related Work

     Traditional interpolation methods are based on the function values of known data points for extrapolation and prediction. For a given data set, there are generally two cases. In the case of two known data points and interpolation in between, interpolation can be selected based on distance, such as nearest-neighbor interpolation [14]. If it is assumed that the unknown point between these two data points is consistent with the straight line between them, a linear function can be used to fit and interpolate, such as linear interpolation, piecewise linear interpolation [15,16]. For interpolation among multiple given points, a commonly used method is to predict the value of the unknown point by constructing a high-degree polynomial based on the given points, such as Lagrange interpolation, Newton interpolation [17,18]. However, with an increase in the number of data points and the degree of the polynomial, there is a risk of overfitting and numerical instability (Runge’s phenomenon) [19]. Another solution is to construct a global smooth function by fitting a low-degree polynomial in a local region. In comparison to high-degree polynomial interpolation methods, this method has better smoothness and numerical stability, such as spline interpolation [20]. Nevertheless, as it is necessary to fit multiple local low-degree polynomials, this method can lead to relatively high computational complexity.

When expanding the sample size using interpolation methods, the selection of node positions and quantities has a significant impact on the accuracy and stability of interpolation results. Typically, equidistant nodes [21] are used for interpolation position selection, where nodes are equally spaced within the interpolation interval. Chebyshev nodes [22] are selected within the interpolation interval to satisfy certain conditions for better fitting of the function. In addition, the choice of the number of interpolations can lead to instability of the data and insufficient validation accuracy. One way to determine the appropriate number of interpolations is by comparing the fitting degree of the model trained under different interpolation numbers with the original data, but this may lead to overfitting and poor generalization ability of the model.

3. Proposed Method

     In this section, we will explain the proposed ASISO method in detail. ASISO introduces two algorithms: K-Space and K-Match. To clearly illustrate the proposed method, we will first provide an overview of ASISO and then introduce the specific algorithms involved.

3.1. Overview

The ASISO is mainly based on the linear interpolation to increase the size and improve the quality of the original dataset. The idea is to divide the original feature space into several subspaces with an equal number of samples, and then perform linear interpolation for the samples in the adjacent subspaces. This method requires two hyperparameters (k and $\eta$) in advance. Interpretation of parameter k is the number of samples existing in each feature subspace, while $\eta$ is the number of equidistant nodes interpolated per unit distance in the linear interpolation of the samples. This proposed method is illustrated in Figure 1.

The data set $D={\{{\mathit{x}}_i,y_i\}}_{i=1}^n$ has been given and is assumed to be contaminated with unknown noise, where ${\mathit{x}}_i\in \mathcal{X}=R^p$ and $y_i\in \mathcal{Y}=R$. Assuming ${\dot{y}}_i=f\left({\dot{\mathit{x}}}_i\right)$ where ${\dot{\mathit{x}}}_i$ is the actual value of ${\mathit{x}}_i$, ${\dot{y}}_i$ is the actual value of $y_i$ and $f(·)$ is a continuous function, represents the relationship in reality between ${\mathit{x}}_i$ and y. Consider the model:

$${\dot{y}}_i+{ϵ}_{i,y}=f({\dot{\mathit{x}}}_i+{ϵ}_{i,x})+{ϵ}_i,$$

(1)

where ${ϵ}_{i,y}$ is the noise in ${\dot{y}}_i$, ${ϵ}_{i,x}$ is the noise in ${\dot{\mathit{x}}}_i$, and ${ϵ}_i$ represents the error term. The expression (1) can be rewritten as:

$$y_i=f\left({\mathit{x}}_i\right)+{ϵ}_i.$$

(2)

Let ${\mathit{x}}_0=\{x_0^1,\dots ,x_0^p\}$, we have $x_0^j=inf{\left\{x_i^j\right\}}_{i=1}^n$ for $\forall j=1,\dots ,p$, and call ${\mathit{x}}_0$ the sample minimum point.

Given the hyperparameter k, we provide an unsupervised clustering method called K-Space. As it is shown in Figure 1(b), the space can be partitioned into $n/k$ subspaces, each containing k samples. i.e., $\mathcal{X}={\cup }_{s=1}^{n/k}{\mathcal{X}}_s$, ${\mathcal{X}}_i\cap {\mathcal{X}}_j=\varnothing ,i,j=1,2,\dots ,\frac{n}{k}$ and $i\ne j$. The datasets corresponding to different subspaces $D={\cup }_{s=1}^{n/k}{D}_s,D_s={\left\{({\mathit{x}}_i^s,y_i^s)\right\}}_{i=1}^k$, ${\mathit{x}}_i^s\in {\mathcal{X}}_s$. For two adjacent subspaces, since $f(·)$ is a continuous function, we assume that it can be approximated as a linear function $g(·)$, and then (2) can be transformed into:

$$y_i=g\left({\mathit{x}}_i\right)+{ϵ}_i+{ϵ}_i^{\prime },$$

(3)

where ${ϵ}_i^{\prime }$ is the linear fitting error term. When the distance between two adjacent subspaces approaches zero and the measurements of subspaces tend to zero, obtain ${ϵ}_i^{\prime }\to 0$. Next, we will perform sample interpolation between adjacent subspaces.

We need to calculate the centers of each cluster, as follow:

$${\overline{\mathit{x}}}^s={\displaystyle \frac{1}{k}}\sum _{{\mathit{x}}_i^s\in D_s}{\mathit{x}}_i^s.$$

(4)

To make ${ϵ}_i^{\prime }\to 0$, we need to ensure that the interpolation is performed between clusters that are close in distance as much as possible. Among ${\left\{D_s\right\}}_{s=1}^{n/k}$, we define $D_{\left(1\right)}$ whose cluster center has the minimum distance to the sample minimum point ${\mathit{x}}_0$, and define $D_{\left(d\right)}$ whose cluster center has the minimum distance to the center of $D_{(d-1)},D_{(d-1)}\ne D_{\left(1\right)},\dots ,D_{(d-1)}$ and $d>1$.

$$D_{\left(1\right)}=\underset{D_s\in D}{argmin} dist({\mathit{x}}_0,{\overline{\mathit{x}}}^s),$$

(5)

$$D_{\left(d\right)}=\underset{\{D_s\in D\},D_s\ne D_{\left(1\right)},\dots ,D_{(d-1)}}{argmin}dist({\overline{\mathit{x}}}_0^{(d-1)},{\overline{\mathit{x}}}^s),$$

(6)

where ${\overline{\mathit{x}}}^{(d-1)}$ is the center of $D_{(d-1)}$. Perform interpolation in ${\left\{D_{\left(d\right)}\right\}}_{d=1}^{n/k}$ sequentially according to the order of d values, and interpolate only between adjacent subspaces (i.e., interpolate between $D_{\left(1\right)}$ and $D_{\left(2\right)}$, between $D_{\left(2\right)}$ and $D_{\left(3\right)}$, and so on).

When performing linear interpolation between adjacent subspaces, we should pair the k samples from the first subspace with an equal number of samples from the second subspace. The interpolation rules between adjacent subspaces are as follows:

1. Linear interpolation can only be performed between two samples belonging to different adjacent subspace sets.

2. Interpolation must be performed for each sample.

3. Participation of each sample point is restricted to a single interpolation instance.

The number of matching schemes is $k!$. As it is shown in Figure 1(c), we provide a matching method called K-Match. Suppose ${ϵ}_i^{\prime }\to 0$, this method can select a good-performing matching scheme ${\{({\mathit{x}}_i^{\left(d\right)},y_i^{\left(d\right)}),({\mathit{x}}_i^{(d+1)},y_i^{(d+1)})\}}_{i=1}^k$ from $k!$.

Assuming $\mathit{x}$ and y are continuous variables. Given another hyperparameter $\eta$, the number of samples inserted using linear interpolation method between $D_{\left(d\right)}$ and $D_{(d+1)}$ is ${\displaystyle \sum _{i=1}^k}\lfloor \eta ·dist({\mathit{x}}_i^{\left(d\right)},{\mathit{x}}_i^{(d+1)})\rfloor$. Taking $({\mathit{x}}_i^{\left(d\right)},y_i^{\left(d\right)})\in D_{\left(d\right)}$ and $({\mathit{x}}_i^{(d+1)},y_i^{(d+1)})\in D_{(d+1)}$ as example, $\left\{\right({\mathit{x}}_{(d,d+1)}^{(m,i)},$$y_{(d,d+1)}^{(m,i)}{\left)\right\}}_{m=1}^{\lfloor \eta ·dist({\mathit{x}}_i^{\left(d\right)},{\mathit{x}}_i^{(d+1)})\rfloor }$ is the set of inserted samples, the linear interpolation formula is defined as:

$${\mathit{x}}_{(d,d+1)}^{(m,i)}={\mathit{x}}_i^{\left(d\right)}+m·{\displaystyle \frac{{\mathit{x}}_i^{(d+1)}-{\mathit{x}}_i^{\left(d\right)}}{\lfloor \eta ·dist({\mathit{x}}_i^{\left(d\right)},{\mathit{x}}_i^{(d+1)})\rfloor +1}},$$

(7)

$$y_{(d,d+1)}^{(m,i)}=y_i^{\left(d\right)}+m·{\displaystyle \frac{y_i^{(d+1)}-y_i^{\left(d\right)}}{\lfloor \eta ·dist(y_i^{\left(d\right)},y_i^{(d+1)})\rfloor +1}}.$$

(8)

After ASISO processing, the original dataset will be optimized. The main steps of the ASISO algorithm are summarized in Algorithm 1.

Algorithm 1: ASISO

Input: Data set $D=\{{({\mathit{x}}_i,y_i\}}_{i=1}^n$; hyperparameters k and $\eta$ Output: Optimized data set $D^{\prime }$

The assumptions of ASISO are as follows:

1. $f(·)$ is a continuous function.

2. the linear fitting error ${ϵ}_i^{\prime }\to 0$.

3. $\mathit{x}$ and y are continuous variables.

3.2. K-Space

     The implementation of ASISO requires an unsupervised clustering method to partition the feature space into multiple subspaces, each containing k samples. Based on this, we propose the K-Space clustering method. The clustering method has the following performance:

1.Each subspace contains an equal number of samples, i.e., $D={\cup }_{s=1}^{n/k}{D}_s,D_s={\left\{({\mathit{x}}_i^s,y_i^s)\right\}}_{i=1}^k$;

2.Each sample belongs to only one subset, i.e., $D_i\cap D_j=\varnothing ,i\ne j$.

Maintaining continuity and similarity between adjacent subspaces is essential for synthesizing data via multiple linear interpolations in ASISO. Our objective is to minimize the linear fitting error ${ϵ}_i^{\prime }$, which helps to satisfy ASISO assumption 2 as much as possible.

To determine the sample set $D_s$ for subspace ${\mathcal{X}}_s$, it is necessary to determine the first sample ${\mathit{x}}_1^s$ in $D_s$.

$${\mathit{x}}_1^s=\underset{\mathit{x}:\mathit{x}\in D,\mathit{x}\notin D_1,\dots ,D_{s-1}}{argmin}dist(\mathit{x},{\overline{\mathit{x}}}^{s-1}),$$

(9)

where $s=1,\dots ,\frac{n}{k}$, ${\overline{\mathit{x}}}^{s-1}$ is the cluster center of $D_{s-1}$, ${\overline{\mathit{x}}}^0={\mathit{x}}_0$. We define $D_s=\left\{x_1^s\right\}$ and determine ${\mathit{x}}_d^s$ as follows:

$${\mathit{x}}_d^s=\underset{\mathit{x}:\mathit{x}\in D,\mathit{x}\notin D_1,\dots ,D_s}{argmin}dist(\mathit{x},{\overline{\mathit{x}}}^s),$$

(10)

where $d=2,\dots ,k$. Obtain ${\mathit{x}}_d^s$ and update $D_s\leftarrow D_s\cup \left\{{\mathit{x}}_d^s\right\}$.

The main steps of the K-Space algorithm are summarized in Algorithm 2.

Algorithm 2: K-Space

Input: Data set $D={\left\{({\mathit{x}}_i,y_i)\right\}}_{i=1}^n$; hyperparameter k Output: ${\left\{D_s\right\}}_{s=1}^{n/k}$

3.3. K-Match

     We can calculate the total error of the matching scheme to measure the quality of the scheme, for the sake of simplicity, let $\mathcal{X}=R$, as follow:

$$\sum _{i=1}^{k}S(x_i^{\left(d\right)},x_i^{(d+1)})=\sum _{i=1}^k{\int }_{x_i^{\left(d\right)}}^{x_i^{(d+1)}}|f\left(x\right)-L_i\left(x\right)|dx,$$

(11)

where $L_i\left(x\right)$ is the linear expression passing through the points $(x_i^{\left(d\right)},y_i^{\left(d\right)})$ and $(x_i^{(d+1)},y_i^{(d+1)})$.

Theorem 1.

Let ${\mathcal{X}}_{\left(d\right)}$ and ${\mathcal{X}}_{(d+1)}$ be two adjacent subspaces, the datasets corresponding to different subspaces are $D_{\left(d\right)},D_{(d+1)}$, and $(x_i^{\left(d\right)},y_i^{\left(d\right)})\in D_{\left(d\right)},(x_i^{(d+1)},y_i^{(d+1)})\in D_{(d+1)}$. Consider the model $y_i=f\left(x_i\right)+{ϵ}_i$, let ${ϵ}_i^{\left(d\right)}=y_i^{\left(d\right)}-f\left(x_i^{\left(d\right)}\right)$. For $\forall i=1,2,\dots ,k$, suppose that ${ϵ}_i^{\prime }\to 0$, then $E\left({\displaystyle \frac{S(x_i^{\left(d\right)},x_i^{(d+1)})}{|x_i^{\left(d\right)}-x_i^{(d+1)}|}}\right)<E\left({\displaystyle \frac{|{ϵ}_i^{\left(d\right)}|+|{ϵ}_i^{(d+1)}|}{2}}\right)$.

Proof of Theorem 1.

Since ${ϵ}_i^{\prime }\to 0$, and according to (3), the model can be transformed into:

$$y_i=g\left(x_i\right)+{ϵ}_i,$$

where $g(·)$ is a linear function. According to (11), it follows that:

$$S(x_i^{\left(d\right)},x_i^{(d+1)})={\int }_{x_i^{\left(d\right)}}^{x_i^{(d+1)}}|g\left(x\right)-L_i\left(x\right)|dx.$$

When ${ϵ}_i^{\left(d\right)}·{ϵ}_i^{(d+1)}<0$, let $(x^{\prime },y^{\prime })$ be the intersection point between $y=L_i\left(x\right)$ and $y=g\left(x\right)$. We can simplify $S(x_i^{\left(d\right)},x_i^{(d+1)})$ using basic geometric area calculations, and according to Law of Iterated Expectations (LIE):

$$\begin{array}{cc}\hfill E\left({\displaystyle \frac{S(x_i^{\left(d\right)},x_i^{(d+1)})}{|x_i^{\left(d\right)}-x_i^{(d+1)}|}}\right)& ={\displaystyle \frac{E\left(S(x_i^{\left(d\right)},x_i^{(d+1)})|{ϵ}_i^{\left(d\right)}·{ϵ}_i^{(d+1)}\ge 0\right)P({ϵ}_i^{\left(d\right)}·{ϵ}_i^{(d+1)}\ge 0)}{|x_i^{\left(d\right)}-x_i^{(d+1)}|}}\hfill \\ \hfill & +{\displaystyle \frac{E\left(S(x_i^{\left(d\right)},x_i^{(d+1)})|{ϵ}_i^{\left(d\right)}·{ϵ}_i^{(d+1)}<0\right)P({ϵ}_i^{\left(d\right)}·{ϵ}_i^{(d+1)}<0)}{|x_i^{\left(d\right)}-x_i^{(d+1)}|}}\hfill \\ \hfill & ={\displaystyle \frac{E\left(|{ϵ}_i^{\left(d\right)}|+|{ϵ}_i^{(d+1)}|\right)P({ϵ}_i^{\left(d\right)}·{ϵ}_i^{(d+1)}\ge 0)}{2}}\hfill \\ \hfill & +{\displaystyle \frac{\left(h_1·E|{ϵ}_i^{\left(d\right)}|+h_2·E\left|{ϵ}_i^{(d+1)}\right|\right)P({ϵ}_i^{\left(d\right)}·{ϵ}_i^{(d+1)}<0)}{2}},\hfill \end{array}$$

where $h_1={\displaystyle \frac{|x^{\prime }-x_i^{\left(d\right)}|}{|x_i^{\left(d\right)}-x_i^{(d+1)}|}},h_2={\displaystyle \frac{|x^{\prime }-x_i^{(d+1)}|}{|x_i^{\left(d\right)}-x_i^{(d+1)}|}}$. Since $P({ϵ}_i^{\left(d\right)}·{ϵ}_i^{(d+1)}\ge 0)+P({ϵ}_i^{\left(d\right)}·{ϵ}_i^{(d+1)}<0)=1$, and $h_1+h_2=1$, it follows that $E\left({\displaystyle \frac{S(x_i^{\left(d\right)},x_i^{(d+1)})}{|x_i^{\left(d\right)}-x_i^{(d+1)}|}}\right)<E\left({\displaystyle \frac{|{ϵ}_i^{\left(d\right)}|+|{ϵ}_i^{(d+1)}|}{2}}\right)$.

   □

If our approach is to randomly select a matching scheme, the validity of this method can be proved by Theorem 1. However, randomly selecting a matching scheme does not guarantee the uniqueness of the results, and it also does not guarantee that we will necessarily select the good-performing matching scheme. We found that for $x_i^{\left(d\right)}$ and $x_i^{(d+1)}$, if ${ϵ}_i^{\left(d\right)}·{ϵ}_i^{(d+1)}<0$, there will be a better interpolation effect.

Theorem 2.

Let $y_i^{\left(d\right)}=f\left(x_i^{\left(d\right)}\right)+{ϵ}_i^{\left(d\right)},y_i^{(d+1)}=f\left(x_i^{(d+1)}\right)+{ϵ}_i^{(d+1)}$. Suppose that ${ϵ}_i^{\prime }\to 0$, then we have $E\left(S(x_i^{\left(d\right)},x_i^{(d+1)})|{ϵ}_i^{\left(d\right)}·{ϵ}_i^{(d+1)}<0\right)<E\left(S(x_i^{\left(d\right)},x_i^{(d+1)})|{ϵ}_i^{\left(d\right)}·{ϵ}_i^{(d+1)}\ge 0\right)$.

Proof of Theorem 2.

Since ${ϵ}_i^{\prime }\to 0$, based on the proof of theorem 1, we have:

$$\begin{array}{cc}\hfill & E\left(S(x_i^{\left(d\right)},x_i^{(d+1)})|{ϵ}_i^{\left(d\right)}{ϵ}_i^{(d+1)}\ge 0\right)\hfill \\ \hfill & ={\displaystyle \frac{|x_i^{\left(d\right)}-x_i^{(d+1)}|·E|{ϵ}_i^{\left(d\right)}|+|x_i^{\left(d\right)}-x_i^{(d+1)}|·E|{ϵ}_i^{(d+1)}|}{2}},\hfill \end{array}$$

$$\begin{array}{cc}\hfill & E\left(S(x_i^{\left(d\right)},x_i^{(d+1)})|{ϵ}_i^{\left(d\right)}{ϵ}_i^{(d+1)}<0\right)\hfill \\ \hfill & ={\displaystyle \frac{|x^{\prime }-x_i^{\left(d\right)}|·E|{ϵ}_i^{\left(d\right)}|+|x^{\prime }-x_i^{(d+1)}|·E|{ϵ}_i^{(d+1)}|}{2}}.\hfill \end{array}$$

It follows that

$$\begin{array}{c}\hfill E\left(S(x_i^{\left(d\right)},x_i^{(d+1)})|{ϵ}_i^{\left(d\right)}{ϵ}_i^{(d+1)}<0\right)<E\left(S(x_i^{\left(d\right)},x_i^{(d+1)})|{ϵ}_i^{\left(d\right)}{ϵ}_i^{(d+1)}\ge 0\right).\end{array}$$

   □

According to Theorem 2, we can match samples with opposite signs of ${ϵ}_i$ to achieve a good data synthesis effect. Therefore, the core idea of K-match is to make a judgment on the positive or negative sign of ${ϵ}_i$ for each sample, and then interpolate the samples with opposite signs as much as possible.

In K-Match, we need to choose an appropriate linear regression method to fit the dataset $D_{\left(d\right)}\cup D_{(d+1)}$ based on the performance of the noise. For example, Lasso regression, Locally Weighted Linear Regression (LWLR) [23], and other methods can be used [24,25]. In our experiments, we use OLS or SVR method to fit and obtain $\widehat{g}(·)$. Specially, the kernel function is Linear in SVR. According to (3), and suppose the linear fitting error ${ϵ}_i^{\prime }\to 0$, for dataset $D_{\left(d\right)}\cup D_{(d+1)}$, we have:

$${ϵ}_i=y_i-\widehat{g}\left({\mathit{x}}_i\right).$$

(12)

Then, we sort the samples in dataset $D_{\left(d\right)}$ in ascending order according to the value of ${ϵ}_i$, and obtain ${\left\{({\mathit{x}}_i^{\left(d\right)},y_i^{(d+1)})\right\}}_{i=1}^k$; we sort the samples in dataset $D_{(d+1)}$ in descending order and obtain ${\left\{({\mathit{x}}_i^{(d+1)},y_i^{(d+1)})\right\}}_{i=1}^k$. As it is shown in Figure 1(d), combine the sorted datasets $D_{\left(d\right)}$ and $D_{(d+1)}$ into the matching scheme ${\{({\mathit{x}}_i^{\left(d\right)},y_i^{\left(d\right)}),({\mathit{x}}_i^{(d+1)},y_i^{(d+1)})\}}_{i=1}^k$.

Algorithm 3: K-Match

Input: Subset $D_{\left(d\right)}={\left\{({\mathit{x}}_i^{\left(d\right)},y_i^{\left(d\right)})\right\}}_{i=1}^k,D_{(d+1)}=\{{({\mathit{x}}_i^{(d+1)},y_i^{(d+1)}\}}_{i=1}^k$ Output: Matching scheme ${\{({\mathit{x}}_i^{\left(d\right)},y_i^{\left(d\right)}),({\mathit{x}}_i^{(d+1)},y_i^{(d+1)})\}}_{i=1}^k$ 1 Fitting the dataset $D_{\left(d\right)}\cup D_{(d+1)}$ and obtain $\widehat{g}(·)$ 2 Obtain $\left\{{ϵ}_i\right\}$ using (10) 3 Sorting the samples in $D_{\left(d\right)}$ and $D_{(d+1)}$ according to the value of ${ϵ}_i$, obtain $D_{\left(d\right)}^{\prime }$ and $D_{(d+1)}^{\prime }$ 4 Combine $D_{\left(d\right)}^{\prime }$ and $D_{(d+1)}^{\prime }$ into ${\{({\mathit{x}}_i^{\left(d\right)},y_i^{\left(d\right)}),({\mathit{x}}_i^{(d+1)},y_i^{(d+1)})\}}_{i=1}^k$

3.4. Supplements

The proposed method can effectively expand the size of dataset and adjust the dataset structure, reducing the proportion of samples that deviate significantly from the actual distribution, thereby improving the model generalization. Refer to Figure 2.

The supplements about ASISO are given below:

1. The choice of the hyperparameter k is crucial as different datasets require different values of k. Conversely, hyperparameter $\eta$ tends to exhibit better performance as its value increases, which will be illustrated in the following experimental results.

2. It is necessary to normalize the data if there is a significant difference in the dimensional scale between the features of the data. It avoids the issue of generating an excessive number of samples.

3. In most cases, $n/k$ is not an integer, and for the excess samples, we usually have two solutions of handling them. The first one is to use the LOF algorithm [26] to filter out the excess samples that will not participate in the ASISO, as it is shown in Figure 2(c). Another solution is to treat the excess samples as a dataset $D^{\prime }$ of a subspace, $D^{\prime }={\left\{({\mathit{x}}_i,y_i)\right\}}_{i=1}^{nmodk}$.When interpolating between $D^{\prime }$ and other subspaces $D^{\prime \prime }$, choose an appropriate linear regression method to fit the dataset $D^{\prime }\cup D^{\prime \prime }$, and obtain $\widehat{g}(·)$. Then, use the same method to sort $D^{\prime }$ and $D^{\prime \prime }$. Only $nmodk$ interpolations are performed, with each sample in $D^{\prime }$ being interpolated, while for $D^{\prime \prime }$, only $nmodk$ samples are interpolated. Moreover, interpolate the samples with opposite signs of ${ϵ}_i$ as much as possible, as it is shown in Figure 3.

4. Experiments

For Artificial data, as $f(·)$ can be known, we investigated the hyperparameter selection, and optimization performance after processing by ASISO. For benchmark data, we studied the prediction performance using this method.

4.1. Artificial Data Sets

Some verification indicators, the proportion of samples with ${ϵ}_i$ greater than $\alpha$, and the mean square error (MSE), can be selected to test the optimization effect of the ASISO on the original sample after processing, as follows:

$$p\left(\alpha \right)={\displaystyle \frac{1}{n}}\sum _{i=1}^{n}I\left(\right|f\left({\mathit{x}}_i\right)-y_i|>\alpha ),$$

(13)

$$MSE={\displaystyle \frac{1}{n}}\sum _{i=1}^n{(f\left({\mathit{x}}_i\right)-y_i)}^2.$$

(14)

For artificial data sets, ${\left\{{\mathit{x}}_i\right\}}_{i=1}^n$ are generated by $N_p(\mathbf{0},\mathit{E})$. Let ${\mathit{W}}_1\in {\mathbb{R}}_{(p\times p_1)}$ and ${\mathit{W}}_2\in {\mathbb{R}}_{(p_1\times 1)}$, where all elements of both ${\mathit{W}}_1$ and ${\mathit{W}}_2$ are independently and identically distributed as $N(0,1)$. Consider $y_i=f\left({\mathit{x}}_i\right)+{ϵ}_i=tanh\left({\mathit{x}}_i^{\prime }{\mathit{W}}_1\right){\mathit{W}}_2+{ϵ}_i$.

$$tanh\left(x\right)={\displaystyle \frac{e^x-e^{-x}}{e^x+e^{-x}}}.$$

For a given data set, we typically cannot ascertain the distribution of noise. Hence, we construct datasets that contain unknown noises [12]. The unknown noises are simulated by mixture noises that contain uniform noises and Gaussian noises. The generation of ${ϵ}_i$ referenced to [27]. We have fixed the random effects of the generated datasets; experiments are repeated 100 times in each artificial dataset. Table 1 shows 6 artificial datasets.

4.1.1. Hyperparameter Selection

First, we investigate the selection of parameter k. To ensure that the number of feature subspaces is sufficient. We set the hyperparameter $\eta =1$, let $k=1,2,\dots ,\lfloor \frac{n}{2}\rfloor$. Calculate the changes in MSE of the datasets at different values of k, obtain $k^{\prime }=\underset{k:k=1,2,\dots ,\lfloor \frac{n}{2}\rfloor ,\eta =1}{argmin}MSE$. The change trend of the MSE index before and after the ASISO processing is shown in Figure 4. Then, let $\eta =1,2,\dots ,30,k=k^{\prime }$, calculate the changes in MSE of the datasets (Figure 5), obtain ${\eta }^{\prime }=\underset{\eta :\eta =1,2,\dots ,30,k=k^{\prime }}{argmin}MSE$. At last, let $k=k^{\prime },\eta =1$ and ${\eta }^{\prime }$, calculate the changes in $p\left(\alpha \right)$ of the artificial datasets (Figure 6).

As can be seen from Figure 4, ASISO demonstrates good optimization performance for datasets with varying sample sizes or feature dimensions. Experimental results also show that the performance of ASISO does not decline dramatically as the content of noise with large variances increases. Hence, it can deal with unknown noise better and has good robustness. In addition, it can be shown that ASISO exhibits good optimization performance for all datasets from experimental results, it is also a stable data synthesis method. As can be seen from Figure 5, the hyperparameter $\eta$ has a generally monotonically decreasing relationship with MSE, and the larger the value of $\eta$, the more significant the effect. From Figure 6, it can be observed that ASISO can adaptively adjust the sample structure, which reduces the proportion of samples with large errors.

4.1.2. Comparison of Optimization Performance

To verify the optimization performance of ASISO, it is compared three methods: piecewise linear interpolation, linear extrapolation, and nearest neighbor interpolation. Specifically, let $k=k^{\prime },\eta ={\eta }^{\prime }$ in ASISO. Based on the given dataset, we use the samples generated by ASISO as interpolation points to calculate the output values in linear extrapolation and nearest neighbor interpolation. Moreover, let $k=1$ and $\eta ={\eta }^{\prime }$ in ASISO, we can regard it as piecewise linear interpolation. The experimental results show that ASISO has the smallest MSE among all methods for each dataset (Figure 7).

4.2. Benchmark Data Sets

To verify the prediction performance of ASISO, 4 benchmark data sets are used [28,29]. We partitioned the dataset into training and testing sets using a 7:3 ratio and normalize the data using the min-max normalization method. We preprocessed the training data using ASISO, train multiple machine learning models on the training set, and evaluated the prediction performance of the models on the testing set. The evaluation metric we used was mean absolute error (MAE).

$$MAE={\displaystyle \frac{1}{n}}\sum _{i=1}^n|y_i-{\widehat{y}}_i|.$$

(15)

In addition, we removed features that cannot be directly used, such as “Datetime” for bike sharing demand, “Month” for forest fires, and so on. We choose K-Nearest Neighbor (KNN), Random Forest (RF), Gradient Boosting Decision Tree (GBDT), Multilayer Perceptron (MLP) and Support Vector Regression (SVR) as the machine learning prediction models. Specifically, the kernel function is Radial Basis Function (RBF) in SVR. Moreover, set the number of hidden layers to 3 for the MLP and use different numbers of neurons based on the input dimensionality and sample size.

The experimental results of the five models are shown in Table 2. It is evident that ASISO performs well on each benchmark dataset. This method is highly applicable to all five models, and in most cases, it can improve the predictive performance. It is worth mentioning that the dataset contains many categorical features, which are not continuous variables. Moreover, for sparse samples (Facebook Metrics and Forest Fires), it is difficult to guarantee that the linear fitting error ${ϵ}_i^{\prime }\to 0$ when interpolating between subspaces. This indicates that even if there are violations of the ASISO assumptions in practical applications, this method may still achieve good optimization results. This further demonstrates that it is robust and stable.

5. Conclusions

In this paper, we propose a data synthesis method, ASISO, which can adaptively adjust the size of the dataset, and the expanded data typically contain minimal error with actual. Moreover, it can adjust the structure of the samples, which can significantly reduce the proportion of samples with large errors. The experimental results on artificial data sets demonstrate that ASISO can optimize the samples, and compared to other methods, the generated data using this method has smaller errors. It can deal with unknown noise better and has good robustness. The results on benchmark data sets show the proposed method is applicable to many machine models, and in most cases, it can improve the model generalization.