Adversarial Variational Autoencoders to Extend and Improve Generative Model

1. Introduction

Variational Autoencoders (VAE) and Generative Adversarial Network (GAN) are two popular approaches for developing deep generative model with support of deep neural network (DNN) where high capacity of DNN contributes significantly to successes of GAN and VAE. There are some researches which combined VAE and GAN. Larsen et al. (Larsen, Sønderby, Larochelle, & Winther, 2016) proposed a traditional combination of VAE and GAN by considering decoder of VAE as generator of GAN (Larsen, Sønderby, Larochelle, & Winther, 2016, p. 1558). They constructed target optimization function as sum of likelihood function of VAE and target function of GAN (Larsen, Sønderby, Larochelle, & Winther, 2016, p. 1560). This research is similar to their research (Larsen, Sønderby, Larochelle, & Winther, 2016, p. 1561) except that the construction optimization functions in two researches are slightly different where the one in this research does not include target function of GAN according to traditional approach of GAN. However uncorrelated variables will be removed after gradients are determined. Moreover, because encoded data z is basically randomized in this research, I do not make a new random z’ to be included into target function of GAN. This research also mentions skillful techniques of derivatives in backpropagation algorithm.

Mescheder et al. (Mescheder, Nowozin, & Geiger, 2017) transformed gain function of VAE including Kullback-Leibler divergence into gain function of GAN via a so-called real-valued discrimination network (Mescheder, Nowozin, & Geiger, 2017, p. 2394) related to Nash equilibrium equation and sigmoid function and then, they trained the transformed VAE by stochastic gradient descent method. Actually, they estimated three parameters (Mescheder, Nowozin, & Geiger, 2017, p. 2395) like this research, but their method focused on mathematical transformation while I focus on skillful techniques in implementation. In other words, Mescheder et al. (Mescheder, Nowozin, & Geiger, 2017) tried to fuse VAE into GAN whereas I combine them by mutual and balancing way but both of us try to make unification of VAE and GAN. Rosca et al. (Rosca, Lakshminarayanan, Warde-Farley, & Mohamed, 2017, p. 4) used a density ratio trick to convert Kullback-Leibler divergence of VAE into the mathematical form log(x / (1–x)) which is similar to GAN target function log(x) + log(1–x). Actually, they made a fusion of VAE and GAN like Mescheder et al. did. The essence of their methods is based on convergence of Nash equilibrium equation. Ahmad et al. combined VAE and GAN separately as featured experimental research. Firstly, they trained VAE and swapped encoder-decoder network to decoder-encoder network so that output of VAE becomes some useful information which in turn becomes input of GAN instead that GAN uses random information as input as usual (Ahmad, Sun, You, Palade, & Mao, 2022, p. 6). Miolane et al. (Miolane, Poitevin, & Li, 2020) combined VAE and GAN by summing target functions of VAE and GAN weighted with regular hyperparameters (Miolane, Poitevin, & Li, 2020, p. 974). Later, they first trained VAE and then sent output of VAE to input of GAN (Miolane, Poitevin, & Li, 2020, p. 975).

In general, this research focuses on incorporating GAN into VAE by skillful techniques related to both stochastic gradient descent and software engineering architecture, which neither focuses on purely mathematical fusion nor focuses on experimental tasks. In practice, many complex mathematical problems can be solved effectively by some skillful techniques of computer programming. Moreover, the proposed model called Adversarial Variational Autoencoders (AVA) aims to extend functions of VAE and GAN as a general architecture for generative model. For instance, AVA will provide encoding function that GAN does not concern and provide discrimination function that VAE needs to distinguish fake data from realistic data. The corporation of VAE and GAN in AVA is strengthened by regular and balance mechanism, which obviously, is natural and like fusion mechanism. In some cases, it is better than fusion mechanism because both built-in VAE and GAN inside AVA can uphold their own strong features. Therefore, experiment in this research is not too serious with large data when I only compare AVA and VAE within small dataset, which aims to prove the proposed method mentioned in the next section.

2. Methodology

In this research I propose a method as well as a generative model which incorporate Generative Adversarial Network (GAN) into Variational Autoencoders (VAE) for extending and improving deep generative model because GAN does not concern how to code original data and VAE lacks mechanisms to assess quality of generated data with note that data coding is necessary to some essential applications such as image impression and recognition whereas audit quality can improve accuracy of generated data. As a convention, let vector variable x = (x₁, x₂,…, x_m)^T and vector variable z = (z₁, z₂,…, z_n)^T be original data and encoded data whose dimensions are m and n (m > n), respectively. A generative model is represented by a function f(x | Θ) = z, f(x | Θ) ≈ z, or f(x | Θ) → z where f(x | Θ) is implemented by a deep neural network (DNN) whose weights are Θ, which converts the original data x to the encoded data z and is called encoder in VAE. A decoder in VAE which converts expectedly the encoded data z back to the original data is represented by a function g(z | Φ) = x’ where g(z | Φ) is also implemented by a DNN whose weights are Φ with expectation that the decoded data x’ is approximated to the original data x as x’ ≈ x. The essence of VAE is to minimize the following loss function for estimating the encoded parameter Θ and the decoded parameter Φ.

$$l_{\mathrm{V}\mathrm{A}\mathrm{E}}\left(\mathsf{\Theta },\mathsf{\Phi }\right)=\frac{1}{2}{‖\mathit{x}-{\mathit{x}}^{\prime }‖}^2+\mathrm{K}\mathrm{L}\left(\mu \left(\mathit{x}\right),\mathsf{\Sigma }\left(\mathit{x}\right)|N\left(0,I\right)\right)$$

(1)

Such that:

$$\begin{gathered} {\mathsf{\Theta }}^{*}=\underset{\mathsf{\Phi }}{\mathrm{argmin}}\mathrm{K}\mathrm{L}\left(\mu \left(\mathit{x}\right),\mathsf{\Sigma }\left(\mathit{x}\right)|N\left(0,I\right)\right) \\ {\mathsf{\Phi }}^{*}=\underset{\mathsf{\Theta }}{\mathrm{argmin}}\frac{1}{2}{‖\mathit{x}-{\mathit{x}}^{\prime }‖}^2 \end{gathered}$$

Note that ||x – x’|| is Euclidean distance between x and x’ whereas KL(μ(x), Σ(x) | N(0, I)) is Kullback-Leibler divergence between Gaussian distribution of x whose mean vector and covariance matrix are μ(x) and Σ(x) and standard Gaussian distribution N(0, I) whose mean vector and covariance matrix are 0 and identity matrix I.

GAN does not concern the encoder f(x | Θ) = z but it focuses on optimizing the decoder g(z | Φ) = x’ by introducing a so-called discriminator which is a discrimination function d(x | Ψ): x → [0, 1] from concerned data x or x’ to range [0, 1] in which d(x | Ψ) can distinguish fake data from real data. In other words, the larger result the discriminator d(x’ | Ψ) derives, the more realistic the generated data x’ is. Obviously, d(x | Ψ) is implemented by a DNN whose weights are Ψ with note that this DNN has only one output neuron denoted d₀. The essence of GAN is to optimize mutually the following target function for estimating the decoder parameter Φ and the discriminator parameter Ψ.

$$b_{\mathrm{G}\mathrm{A}\mathrm{N}}\left(\mathsf{\Phi },\mathsf{\Psi }\right)=\log \left(d\left(\mathit{x}|\mathsf{\Psi }\right)\right)+\log \left(1-d\left(g\left(\mathit{z}|\mathsf{\Phi }\right)|\mathsf{\Psi }\right)\right)$$

(2)

Such that Φ and Ψ are optimized mutually as follows:

$$\begin{gathered} {\mathsf{\Phi }}^{*}=\underset{\mathsf{\Phi }}{\mathrm{argmin}}{b}_{\mathrm{G}\mathrm{A}\mathrm{N}}\left(\mathsf{\Phi },{\mathsf{\Psi }}^{*}\right) \\ {\mathsf{\Psi }}^{*}=\underset{\mathsf{\Psi }}{\mathrm{argmax}}{b}_{\mathrm{G}\mathrm{A}\mathrm{N}}\left({\mathsf{\Phi }}^{*},\mathsf{\Psi }\right) \end{gathered}$$

The proposed generative model in this research is called Adversarial Variational Autoencoders (AVA) because it combines VAE and GAN by fusing mechanism in which loss function and balance function are optimized parallelly. The AVA loss function implies loss information in encoder f(x | Θ), decoder g(z | Φ), discriminator d(x | Ψ) as follows:

$$l_{\mathrm{A}\mathrm{V}\mathrm{A}}\left(\mathsf{\Theta },\mathsf{\Phi },\mathsf{\Psi }\right)=\frac{1}{2}{‖\mathit{x}-{\mathit{x}}^{\prime }‖}^2+\mathrm{K}\mathrm{L}\left(\mu \left(\mathit{x}\right),\mathsf{\Sigma }\left(\mathit{x}\right)|N\left(0,I\right)\right)+\log \left(1-d\left(g\left(\mathit{z}|\mathsf{\Phi }\right)|\mathsf{\Psi }\right)\right)$$

(3)

The balance function of AVA is to supervise the decoding mechanism, which is the GAN target function as follows:

$$b_{\mathrm{A}\mathrm{V}\mathrm{A}}\left(\mathsf{\Phi },\mathsf{\Psi }\right)=b_{\mathrm{G}\mathrm{A}\mathrm{N}}\left(\mathsf{\Phi },\mathsf{\Psi }\right)=\log \left(d\left(\mathit{x}|\mathsf{\Psi }\right)\right)+\log \left(1-d\left(g\left(\mathit{z}|\mathsf{\Phi }\right)|\mathsf{\Psi }\right)\right)$$

(4)

The key point of AVA is that the discriminator function occurs in both loss function and balance function via the expression log(1 – d(g(z | Φ) | Ψ)), which means that the capacity of how to distinguish fake data from realistic data by discriminator function affects the decoder DNN. As a result, the three parameters Θ, Φ, and Ψ are optimized mutually according to both loss function and balance function as follows:

$$\begin{gathered} {\mathsf{\Theta }}^{*}=\underset{\mathsf{\Theta }}{\mathrm{argmin}}{l}_{\mathrm{A}\mathrm{V}\mathrm{A}}\left(\mathsf{\Theta },{\mathsf{\Phi }}^{*},{\mathsf{\Psi }}^{*}\right) \\ {\mathsf{\Phi }}^{*}=\underset{\mathsf{\Phi }}{\mathrm{argmin}}{l}_{\mathrm{A}\mathrm{V}\mathrm{A}}\left({\mathsf{\Theta }}^{*},\mathsf{\Phi },{\mathsf{\Psi }}^{*}\right) \\ {\mathsf{\Psi }}^{*}=\underset{\mathsf{\Psi }}{\mathrm{argmax}}{b}_{\mathrm{A}\mathrm{V}\mathrm{A}}\left({\mathsf{\Phi }}^{*},\mathsf{\Psi }\right) \end{gathered}$$

Because the encoder parameter Θ is independent from both the decoder parameter Φ and the discriminator parameter Ψ, its estimate is specified as follows:

$${\mathsf{\Theta }}^{*}=\underset{\mathsf{\Phi }}{\mathrm{argmin}}\left(\mathrm{K}\mathrm{L}\left(\mu \left(\mathit{x}\right),\mathsf{\Sigma }\left(\mathit{x}\right)|N\left(0,I\right)\right)\right)$$

Because the decoder parameter Φ is independent from the encoder parameter Θ, its estimate is specified as follows:

$${\mathsf{\Phi }}^{*}=\underset{\mathsf{\Theta }}{\mathrm{argmin}}\left(\frac{1}{2}{‖\mathit{x}-{\mathit{x}}^{\prime }‖}^2+\log \left(1-d\left(g\left(\mathit{z}|{\mathsf{\Phi }}^{*}\right)|{\mathsf{\Psi }}^{*}\right)\right)\right)$$

Note that the Euclidean distance ||x – x’|| is only dependent on Θ. Because the discriminator tries to increase credible degree of realistic data and decrease credible degree of fate data, its parameter Ψ has following estimate:

$${\mathsf{\Psi }}^{*}=\underset{\mathsf{\Psi }}{\mathrm{argmax}}\left(\log \left(d\left(\mathit{x}|\mathsf{\Psi }\right)\right)+\log \left(1-d\left(g\left(\mathit{z}|{\mathsf{\Phi }}^{*}\right)|\mathsf{\Psi }\right)\right)\right)$$

By applying stochastic gradient descent (SDG) algorithm into backpropagation algorithm, these estimates are determined based on gradients of loss function and balance function as follows:

$$\begin{gathered} \mathsf{\Theta }=\mathsf{\Theta }-\gamma {\nabla }_{\mathsf{\Theta }}\left(\mathrm{K}\mathrm{L}\left(\mu \left(\mathit{x}\right),\mathsf{\Sigma }\left(\mathit{x}\right)|N\left(0,I\right)\right)\right) \\ \mathsf{\Phi }=\mathsf{\Phi }-\gamma {\nabla }_{\mathsf{\Phi }}\left(\frac{1}{2}{‖\mathit{x}-{\mathit{x}}^{\prime }‖}^2+\log \left(1-d\left(g\left(\mathit{z}|{\mathsf{\Phi }}^{*}\right)|{\mathsf{\Psi }}^{*}\right)\right)\right) \\ \mathsf{\Psi }=\mathsf{\Psi }+\gamma {\nabla }_{\mathsf{\Psi }}\left(\log \left(d\left(\mathit{x}|\mathsf{\Psi }\right)\right)+\log \left(1-d\left(g\left(\mathit{z}|{\mathsf{\Phi }}^{*}\right)|\mathsf{\Psi }\right)\right)\right) \end{gathered}$$

where γ (0 < γ ≤ 1) is learning rate. Let a_f(.), a_g(.), and a_d(.) be activation functions of encoder DNN, decoder DNN, and discriminator DNN, respectively and so, let a_f’(.), a_g’(.), and a_d’(.) be derivatives of these activation functions, respectively. The encoder gradient regarding Θ is (Doersch, 2016, p. 9):

$${\nabla }_{\mathsf{\Theta }}\left(\mathrm{K}\mathrm{L}\left(\mu \left(\mathit{x}\right),\mathsf{\Sigma }\left(\mathit{x}\right)|N\left(0,I\right)\right)\right)=\left(\mu \left(\mathit{x}\right)-\frac{1}{2}{\left(\mathsf{\Sigma }\left(\mathit{x}\right)\right)}^{-1}+\frac{1}{2}I\right)a_f^{\prime }\left(\mathit{x}\right)$$

The decoder gradient regarding Φ is:

$${\nabla }_{\mathsf{\Phi }}\left(\frac{1}{2}{‖\mathit{x}-{\mathit{x}}^{\prime }‖}^2+\log \left(1-d\left(g\left(\mathit{z}|{\mathsf{\Phi }}^{*}\right)|{\mathsf{\Psi }}^{*}\right)\right)\right)=-\left(‖\mathit{x}-{\mathit{x}}^{\prime }‖+\frac{a_d^{\prime }\left(d\left({\mathit{x}}^{\prime }|{\mathsf{\Psi }}^{*}\right)\right)}{1-d\left({\mathit{x}}^{\prime }|{\mathsf{\Psi }}^{*}\right)}\right)a_g^{\prime }\left({\mathit{x}}^{\prime }\right)$$

where,

$$g\left(\mathit{z}|{\mathsf{\Phi }}^{*}\right)={\mathit{x}}^{\prime }$$

The discriminator gradient regarding Ψ is:

$${\nabla }_{\mathsf{\Psi }}\left(\log \left(d\left(\mathit{x}|\mathsf{\Psi }\right)\right)+\log \left(1-d\left({\mathit{x}}^{\prime }|\mathsf{\Psi }\right)\right)\right)=\frac{a_d^{\prime }\left(d\left(\mathit{x}|\mathsf{\Psi }\right)\right)}{d\left(\mathit{x}|\mathsf{\Psi }\right)}-\frac{a_d^{\prime }\left(d\left({\mathit{x}}^{\prime }|\mathsf{\Psi }\right)\right)}{1-d\left({\mathit{x}}^{\prime }|\mathsf{\Psi }\right)}$$

As a result, SGD algorithm incorporated into backpropagation algorithm for solving AVA is totally determined as follows:

$$\mathsf{\Theta }=\mathsf{\Theta }-\gamma \left(\mu \left(\mathit{x}\right)-\frac{1}{2}{\left(\mathsf{\Sigma }\left(\mathit{x}\right)\right)}^{-1}+\frac{1}{2}I\right)a_f^{\prime }\left(\mathit{x}\right)$$

(5)

$$\mathsf{\Phi }\left[i\right]=\mathsf{\Phi }\left[i\right]+\gamma \left(\left(\mathit{x}\left[i\right]-{\mathit{x}}^{\prime }\left[i\right]\right)+\frac{a_d^{\prime }\left(d\left({\mathit{x}}^{\prime }|{\mathsf{\Psi }}^{*}\right)\right)}{1-d\left({\mathit{x}}^{\prime }|{\mathsf{\Psi }}^{*}\right)}\right)a_g^{\prime }\left({\mathit{x}}^{\prime }\left[i\right]\right)$$

(6)

$$\mathsf{\Psi }=\mathsf{\Psi }+\gamma \left(\frac{a_d^{\prime }\left(d\left(\mathit{x}|\mathsf{\Psi }\right)\right)}{d\left(\mathit{x}|\mathsf{\Psi }\right)}-\frac{a_d^{\prime }\left(d\left({\mathit{x}}^{\prime }|\mathsf{\Psi }\right)\right)}{1-d\left({\mathit{x}}^{\prime }|\mathsf{\Psi }\right)}\right)$$

(7)

where notation [i] denotes the i^th element in vector. Please pay attention to the derivatives a_f’(.), a_g’(.), and a_d’(.) because they are helpful techniques to consolidate AVA. The reason of two different occurrences of derivatives a_d’(d(x’ | Ψ^*)) and a_g’(x’) in decoder gradient regarding Φ is nontrivial because the unique output neuron of discriminator DNN is considered as effect of the output layer of all output neurons in decoder DNN.

Figure 1. Causality effect relationship between decoder DNN and discriminator DNN.

Figure 1. Causality effect relationship between decoder DNN and discriminator DNN.

When weights are assumed to be 1, error of causal decoder neuron is error of discriminator neuron multiplied with derivative at the decoder neuron and moreover, the error of discriminator neuron, in turn, is product of its minus bias –d’(.) and its derivative a’_d(.).

$$\begin{gathered} \mathrm{e}\mathrm{r}\mathrm{r}\mathrm{o}\mathrm{r}\left({\mathit{x}}^{\prime }\left[i\right]\right)=1*\mathrm{e}\mathrm{r}\mathrm{r}\mathrm{o}\mathrm{r}\left(d_0\right)a_g^{\prime }\left({\mathit{x}}^{\prime }\left[i\right]\right) \\ \mathrm{e}\mathrm{r}\mathrm{r}\mathrm{o}\mathrm{r}\left(d_0\right)=-d^{\prime }\left(d_0\right)a_d^{\prime }\left(d_0\right) \end{gathered}$$

It is necessary to describe AVA architecture because skillful techniques cannot be applied into AVA without clear and solid architecture. The key point to incorporate GAN into VAE is that the error $\frac{a_d^{\prime }\left(d\left({\mathit{x}}^{\prime }|{\mathsf{\Psi }}^{*}\right)\right)}{1-d\left({\mathit{x}}^{\prime }|{\mathsf{\Psi }}^{*}\right)}$ of generated data is included in both decoder and discriminator, besides decoded data x’ which is output of decoder DNN becomes input of discriminator DNN.

$$\begin{gathered} \mathsf{\Phi }\left[i\right]=\mathsf{\Phi }\left[i\right]+\gamma \left(\left(\mathit{x}\left[i\right]-{\mathit{x}}^{\prime }\left[i\right]\right)+\frac{a_d^{\prime }\left(d\left({\mathit{x}}^{\prime }|{\mathsf{\Psi }}^{*}\right)\right)}{1-d\left({\mathit{x}}^{\prime }|{\mathsf{\Psi }}^{*}\right)}\right)a_g^{\prime }\left({\mathit{x}}^{\prime }\left[i\right]\right) \\ \mathsf{\Psi }=\mathsf{\Psi }+\gamma \left(\frac{a_d^{\prime }\left(d\left(\mathit{x}|\mathsf{\Psi }\right)\right)}{d\left(\mathit{x}|\mathsf{\Psi }\right)}-\frac{a_d^{\prime }\left(d\left({\mathit{x}}^{\prime }|\mathsf{\Psi }\right)\right)}{1-d\left({\mathit{x}}^{\prime }|\mathsf{\Psi }\right)}\right) \end{gathered}$$

Figure 2 shows the AVA architecture.

AVA architecture follows an important aspect of VAE where the encoder f(x | Θ) does not produce directly decoded data z as f(x | Θ) = z. It actually produces mean vector μ(x) and covariance matrix Σ(x) belonging to x instead. In this research, μ(x) and Σ(x) are flatten into an array of neurons output layer of the encoder f(x | Θ).

$$f\left(\mathit{x}|\mathsf{\Theta }\right)=\left(\begin{array}{c}\mu \left(\mathit{x}\right)\\ \mathsf{\Sigma }\left(\mathit{x}\right)\end{array}\right)\to \mathit{z}$$

The actual decoded data z is calculated randomly from μ(x) and Σ(x) along with a random vector r.

$$\mathit{z}=\mu \left(\mathit{x}\right)+{\left(\mathsf{\Sigma }\left(\mathit{x}\right)\right)}^{\frac{1}{2}}\mathit{r}$$

(8)

where r follows standard Gaussian distribution with mean vector 0 and identity covariance matrix I and each element of (Σ(x))^1/2 is squared root of the corresponding element of Σ(x). This is an excellent invention in traditional literature which made the calculation of Kullback-Leibler divergence much easier without loss of information.

The balance function b_AVA(Φ, Ψ) aims to balance decoding task and discrimination task without partiality but it can lean forward decoding task for improving accuracy of decoder by including the error of original data x and decoded data x’ into balance function as follows:

$$b_{\mathrm{A}\mathrm{V}\mathrm{A}}\left(\mathsf{\Phi },\mathsf{\Psi }\right)=b_{\mathrm{G}\mathrm{A}\mathrm{N}}\left(\mathsf{\Phi },\mathsf{\Psi }\right)-\frac{1}{2}{‖\mathit{x}-{\mathit{x}}^{\prime }‖}^2=\log \left(d\left(\mathit{x}|\mathsf{\Psi }\right)\right)+\log \left(1-d\left(g\left(\mathit{z}|\mathsf{\Phi }\right)|\mathsf{\Psi }\right)\right)-\frac{1}{2}{‖\mathit{x}-{\mathit{x}}^{\prime }‖}^2$$

(9)

As a result, the estimate of discriminator parameter Ψ is:

$$\mathsf{\Psi }=\mathsf{\Psi }+\gamma \left(\frac{a_d^{\prime }\left(d\left(\mathit{x}|\mathsf{\Psi }\right)\right)}{d\left(\mathit{x}|\mathsf{\Psi }\right)}-\frac{a_d^{\prime }\left(d\left({\mathit{x}}^{\prime }|\mathsf{\Psi }\right)\right)}{1-d\left({\mathit{x}}^{\prime }|\mathsf{\Psi }\right)}+a_d^{\prime }\left(d_0\right)\sum _i\left(\mathit{x}\left[i\right]-{\mathit{x}}^{\prime }\left[i\right]\right)a_g^{\prime }\left({\mathit{x}}^{\prime }\left[i\right]\right)\right)$$

(10)

In a reverse causality effect relationship in which the unique output neuron of discriminator DNN is cause of all output neurons of decoder DNN as shown in Figure 3.

Figure 3. Reverse causality effect relationship between discriminator DNN and decoder DNN.

Figure 3. Reverse causality effect relationship between discriminator DNN and decoder DNN.

Suppose bias of each decoder output neuron is bias[i], error of the discriminator output neuron error[i] is sum of weighted biases which is in turn multiplied with derivative at the discriminator output neuron with note that every weighted bias is also multiplied with derivative at every decoder output neuron. Suppose all weights are 1, we have:

$$\begin{gathered} \mathrm{e}\mathrm{r}\mathrm{r}\mathrm{o}\mathrm{r}\left[i\right]=a_d^{\prime }\left(d_0\right)\sum _i\mathrm{b}\mathrm{i}\mathrm{a}\mathrm{s}\left[i\right]a_g^{\prime }\left({\mathit{x}}^{\prime }\left[i\right]\right) \\ \mathrm{b}\mathrm{i}\mathrm{a}\mathrm{s}\left[i\right]=\mathit{x}\left[i\right]-{\mathit{x}}^{\prime }\left[i\right] \end{gathered}$$

Because the balance function b_AVA(Φ, Ψ) aims to improve the decoder g(z | Φ), it is possible to improve the encoder f(x | Θ) by similar technique with note that output of encoder is mean vector μ(x) and covariance matrix Σ(x). In this research, I propose another balance function B_AVA(Θ, Λ) to assess reliability of the mean vector μ(x) because μ(x) is most important to randomize z and μ(x) is linear. Let D(μ(x) | Λ) be discrimination function for encoder DNN from μ(x) to range [0, 1] in which D(μ(x) | Λ) can distinguish fake mean μ(x) from real mean μ(x’). Obviously, D(μ(x) | Λ) is implemented by a so-called encoding discriminator DNN whose weights are Λ with note that this DNN has only one output neuron denoted D₀. The balance function B_AVA(Θ, Λ) is specified as follows:

$$B_{\mathrm{A}\mathrm{V}\mathrm{A}}\left(\mathsf{\Theta },\mathsf{\Lambda }\right)=\log \left(D\left(\mu \left(\mathit{x}\right)|\mathsf{\Lambda }\right)\right)+\log \left(1-D\left(\mu \left({\mathit{x}}^{\prime }\right)|\mathsf{\Lambda }\right)\right)$$

(11)

Note,

$$g\left(\mathit{z}|\mathsf{\Phi }\right)={\mathit{x}}^{\prime }$$

AVA loss function is modified with regard to the balance function B_AVA(Θ, Λ) as follows:

$$l_{\mathrm{A}\mathrm{V}\mathrm{A}}\left(\mathsf{\Theta },\mathsf{\Phi },\mathsf{\Psi },\mathsf{\Lambda }\right)=\frac{1}{2}{‖\mathit{x}-{\mathit{x}}^{\prime }‖}^2+\mathrm{K}\mathrm{L}\left(\mu \left(\mathit{x}\right),\mathsf{\Sigma }\left(\mathit{x}\right)|N\left(0,I\right)\right)+\log \left(1-d\left({\mathit{x}}^{\prime }|\mathsf{\Psi }\right)\right)+\log \left(1-D\left(\mu \left({\mathit{x}}^{\prime }\right)|\mathsf{\Lambda }\right)\right)$$

(12)

By similar way of applying SGD algorithm, it is easy to estimate the encoding discriminator parameter Λ as follows:

$$\mathsf{\Lambda }=\mathsf{\Lambda }+\gamma \left(\frac{a_D^{\prime }\left(D\left(\mu \left(\mathit{x}\right)|\mathsf{\Lambda }\right)\right)}{D\left(\mu \left(\mathit{x}\right)|\mathsf{\Lambda }\right)}-\frac{a_D^{\prime }\left(D\left(\mu \left({\mathit{x}}^{\prime }\right)|\mathsf{\Lambda }\right)\right)}{1-D\left(\mu \left({\mathit{x}}^{\prime }\right)|\mathsf{\Lambda }\right)}\right)$$

(13)

where a_D(.) and a’_D(.) are activation function of the discriminator D(μ(x) | Λ) and its derivative, respectively.

The encoder parameter Θ is consisted of two separated parts Θ_μ and Θ_Σ because the output of encoder f(x | Θ) is consisted of mean vector μ(x) and covariance matrix Σ(x).

$$\mathsf{\Theta }=\left(\begin{array}{c}{\mathsf{\Theta }}_{\mu }\\ {\mathsf{\Theta }}_{\mathsf{\Sigma }}\end{array}\right)$$

where,

$$\begin{gathered} {\mathsf{\Theta }}_{\mu }={\mathsf{\Theta }}_{\mu }-\gamma \mu \left(\mathit{x}\right)a_f^{\prime }\left(\mathit{x}\right) \\ {\mathsf{\Theta }}_{\mathsf{\Sigma }}={\mathsf{\Theta }}_{\mathsf{\Sigma }}-\gamma \left(-\frac{1}{2}{\left(\mathsf{\Sigma }\left(\mathit{x}\right)\right)}^{-1}+\frac{1}{2}I\right)a_f^{\prime }\left(\mathit{x}\right) \end{gathered}$$

When the balance function B_AVA(Θ, Λ) is included in AVA loss function, the part Θ_μ is recalculated whereas the part Θ_Σ is kept intact as follows:

$${\mathsf{\Theta }}_{\mu }={\mathsf{\Theta }}_{\mu }-\gamma \left(\mu \left(\mathit{x}\right)-\frac{a_D^{\prime }\left(D\left({\mathit{x}}^{\prime }|\mathsf{\Lambda }\right)\right)}{1-D\left({\mathit{x}}^{\prime }|\mathsf{\Lambda }\right)}\right)a_f^{\prime }\left(\mathit{x}\right)$$

(14)

Figure 4 shows AVA architecture with support of assessing encoder.

Similarly, the balance function B_AVA(Φ, Λ) can lean forward encoding task for improving accuracy of encoder f(x | Θ) by concerning the error of original mean μ(x) and decoded data mean μ(x’) as follows:

$$B_{\mathrm{A}\mathrm{V}\mathrm{A}}\left(\mathsf{\Phi },\mathsf{\Lambda }\right)=\log \left(D\left(\mu \left(\mathit{x}\right)|\mathsf{\Lambda }\right)\right)+\log \left(1-D\left(\mu \left({\mathit{x}}^{\prime }\right)|\mathsf{\Lambda }\right)\right)-\frac{1}{2}{‖\mu \left(\mathit{x}\right)-\mu \left({\mathit{x}}^{\prime }\right)‖}^2$$

(15)

Without repeating explanations, the estimate of discriminator parameter Λ is modified as follows:

$$\mathsf{\Lambda }=\mathsf{\Lambda }+\gamma \left(\frac{a_D^{\prime }\left(D\left(\mu \left(\mathit{x}\right)|\mathsf{\Lambda }\right)\right)}{D\left(\mu \left(\mathit{x}\right)|\mathsf{\Lambda }\right)}-\frac{a_D^{\prime }\left(D\left(\mu \left({\mathit{x}}^{\prime }\right)|\mathsf{\Lambda }\right)\right)}{1-D\left(\mu \left({\mathit{x}}^{\prime }\right)|\mathsf{\Lambda }\right)}+a_D^{\prime }\left(D_0\right)\sum _i\left(\mu \left(\mathit{x}\right)\left[i\right]-\mu \left({\mathit{x}}^{\prime }\right)\left[i\right]\right)a_g^{\prime }\left(\mu \left({\mathit{x}}^{\prime }\right)\left[i\right]\right)\right)$$

(16)

These variants of AVA are summarized, and their tests are described in the next section.

3. Experimental results and discussions

In this experiment, AVA is tested with VAE but there are 5 versions of AVA such as AVA1, AVA2, AVA3, AVA4, and AVA5. Recall that AVA1 is normal version of AVA whose parameters are listed as follows:

$$\begin{gathered} \mathsf{\Theta }=\mathsf{\Theta }-\gamma \left(\mu \left(\mathit{x}\right)-\frac{1}{2}{\left(\mathsf{\Sigma }\left(\mathit{x}\right)\right)}^{-1}+\frac{1}{2}I\right)a_f^{\prime }\left(\mathit{x}\right) \\ \mathsf{\Phi }\left[i\right]=\mathsf{\Phi }\left[i\right]+\gamma \left(\left(\mathit{x}\left[i\right]-{\mathit{x}}^{\prime }\left[i\right]\right)+\frac{a_d^{\prime }\left(d\left({\mathit{x}}^{\prime }|{\mathsf{\Psi }}^{*}\right)\right)}{1-d\left({\mathit{x}}^{\prime }|{\mathsf{\Psi }}^{*}\right)}\right)a_g^{\prime }\left({\mathit{x}}^{\prime }\left[i\right]\right) \\ \mathsf{\Psi }=\mathsf{\Psi }+\gamma \left(\frac{a_d^{\prime }\left(d\left(\mathit{x}|\mathsf{\Psi }\right)\right)}{d\left(\mathit{x}|\mathsf{\Psi }\right)}-\frac{a_d^{\prime }\left(d\left({\mathit{x}}^{\prime }|\mathsf{\Psi }\right)\right)}{1-d\left({\mathit{x}}^{\prime }|\mathsf{\Psi }\right)}\right) \end{gathered}$$

AVA2 leans forward improving accuracy of decoder DNN by modifying discriminator parameter Ψ as follows:

$$\begin{gathered} \mathsf{\Theta }=\mathsf{\Theta }-\gamma \left(\mu \left(\mathit{x}\right)-\frac{1}{2}{\left(\mathsf{\Sigma }\left(\mathit{x}\right)\right)}^{-1}+\frac{1}{2}I\right)a_f^{\prime }\left(\mathit{x}\right) \\ \mathsf{\Phi }\left[i\right]=\mathsf{\Phi }\left[i\right]+\gamma \left(\left(\mathit{x}\left[i\right]-{\mathit{x}}^{\prime }\left[i\right]\right)+\frac{a_d^{\prime }\left(d\left({\mathit{x}}^{\prime }|{\mathsf{\Psi }}^{*}\right)\right)}{1-d\left({\mathit{x}}^{\prime }|{\mathsf{\Psi }}^{*}\right)}\right)a_g^{\prime }\left({\mathit{x}}^{\prime }\left[i\right]\right) \\ \mathsf{\Psi }=\mathsf{\Psi }+\gamma \left(\frac{a_d^{\prime }\left(d\left(\mathit{x}|\mathsf{\Psi }\right)\right)}{d\left(\mathit{x}|\mathsf{\Psi }\right)}-\frac{a_d^{\prime }\left(d\left({\mathit{x}}^{\prime }|\mathsf{\Psi }\right)\right)}{1-d\left({\mathit{x}}^{\prime }|\mathsf{\Psi }\right)}+a_d^{\prime }\left(d_0\right)\sum _i\left(\mathit{x}\left[i\right]-{\mathit{x}}^{\prime }\left[i\right]\right)a_g^{\prime }\left({\mathit{x}}^{\prime }\left[i\right]\right)\right) \end{gathered}$$

AVA3 supports the balance function B_AVA(Θ, Λ) for assessing reliability of encoder f(x | Θ). Its parameters are listed as follows:

$$\begin{gathered} {\mathsf{\Theta }}_{\mu }={\mathsf{\Theta }}_{\mu }-\gamma \left(\mu \left(\mathit{x}\right)-\frac{a_D^{\prime }\left(D\left({\mathit{x}}^{\prime }|\mathsf{\Lambda }\right)\right)}{1-D\left({\mathit{x}}^{\prime }|\mathsf{\Lambda }\right)}\right)a_f^{\prime }\left(\mathit{x}\right) \\ {\mathsf{\Theta }}_{\mathsf{\Sigma }}={\mathsf{\Theta }}_{\mathsf{\Sigma }}-\gamma \left(-\frac{1}{2}{\left(\mathsf{\Sigma }\left(\mathit{x}\right)\right)}^{-1}+\frac{1}{2}I\right)a_f^{\prime }\left(\mathit{x}\right) \\ \mathsf{\Phi }\left[i\right]=\mathsf{\Phi }\left[i\right]+\gamma \left(\left(\mathit{x}\left[i\right]-{\mathit{x}}^{\prime }\left[i\right]\right)+\frac{a_d^{\prime }\left(d\left({\mathit{x}}^{\prime }|{\mathsf{\Psi }}^{*}\right)\right)}{1-d\left({\mathit{x}}^{\prime }|{\mathsf{\Psi }}^{*}\right)}\right)a_g^{\prime }\left({\mathit{x}}^{\prime }\left[i\right]\right) \\ \mathsf{\Lambda }=\mathsf{\Lambda }+\gamma \left(\frac{a_D^{\prime }\left(D\left(\mu \left(\mathit{x}\right)|\mathsf{\Lambda }\right)\right)}{D\left(\mu \left(\mathit{x}\right)|\mathsf{\Lambda }\right)}-\frac{a_D^{\prime }\left(D\left(\mu \left({\mathit{x}}^{\prime }\right)|\mathsf{\Lambda }\right)\right)}{1-D\left(\mu \left({\mathit{x}}^{\prime }\right)|\mathsf{\Lambda }\right)}\right) \end{gathered}$$

AVA4 is a variant of AVA3 along with leaning forward improving accuracy of encoder f(x | Θ) like AVA2. Its parameters are listed as follows:

$$\begin{gathered} {\mathsf{\Theta }}_{\mu }={\mathsf{\Theta }}_{\mu }-\gamma \left(\mu \left(\mathit{x}\right)-\frac{a_D^{\prime }\left(D\left({\mathit{x}}^{\prime }|\mathsf{\Lambda }\right)\right)}{1-D\left({\mathit{x}}^{\prime }|\mathsf{\Lambda }\right)}\right)a_f^{\prime }\left(\mathit{x}\right) \\ {\mathsf{\Theta }}_{\mathsf{\Sigma }}={\mathsf{\Theta }}_{\mathsf{\Sigma }}-\gamma \left(-\frac{1}{2}{\left(\mathsf{\Sigma }\left(\mathit{x}\right)\right)}^{-1}+\frac{1}{2}I\right)a_f^{\prime }\left(\mathit{x}\right) \\ \mathsf{\Phi }\left[i\right]=\mathsf{\Phi }\left[i\right]+\gamma \left(\left(\mathit{x}\left[i\right]-{\mathit{x}}^{\prime }\left[i\right]\right)+\frac{a_d^{\prime }\left(d\left({\mathit{x}}^{\prime }|{\mathsf{\Psi }}^{*}\right)\right)}{1-d\left({\mathit{x}}^{\prime }|{\mathsf{\Psi }}^{*}\right)}\right)a_g^{\prime }\left({\mathit{x}}^{\prime }\left[i\right]\right) \\ \mathsf{\Lambda }=\mathsf{\Lambda }+\gamma \left(\frac{a_D^{\prime }\left(D\left(\mu \left(\mathit{x}\right)|\mathsf{\Lambda }\right)\right)}{D\left(\mu \left(\mathit{x}\right)|\mathsf{\Lambda }\right)}-\frac{a_D^{\prime }\left(D\left(\mu \left({\mathit{x}}^{\prime }\right)|\mathsf{\Lambda }\right)\right)}{1-D\left(\mu \left({\mathit{x}}^{\prime }\right)|\mathsf{\Lambda }\right)}+a_D^{\prime }\left(D_0\right)\sum _i\left(\mu \left(\mathit{x}\right)\left[i\right]-\mu \left({\mathit{x}}^{\prime }\right)\left[i\right]\right)a_g^{\prime }\left(\mu \left({\mathit{x}}^{\prime }\right)\left[i\right]\right)\right) \end{gathered}$$

AVA5 is the last one which supports all functions such as decoder supervising, leaning decoder, encoder supervising, and learning encoder.

$$\begin{gathered} {\mathsf{\Theta }}_{\mu }={\mathsf{\Theta }}_{\mu }-\gamma \left(\mu \left(\mathit{x}\right)-\frac{a_D^{\prime }\left(D\left({\mathit{x}}^{\prime }|\mathsf{\Lambda }\right)\right)}{1-D\left({\mathit{x}}^{\prime }|\mathsf{\Lambda }\right)}\right)a_f^{\prime }\left(\mathit{x}\right) \\ {\mathsf{\Theta }}_{\mathsf{\Sigma }}={\mathsf{\Theta }}_{\mathsf{\Sigma }}-\gamma \left(-\frac{1}{2}{\left(\mathsf{\Sigma }\left(\mathit{x}\right)\right)}^{-1}+\frac{1}{2}I\right)a_f^{\prime }\left(\mathit{x}\right) \\ \mathsf{\Phi }\left[i\right]=\mathsf{\Phi }\left[i\right]+\gamma \left(\left(\mathit{x}\left[i\right]-{\mathit{x}}^{\prime }\left[i\right]\right)+\frac{a_d^{\prime }\left(d\left({\mathit{x}}^{\prime }|{\mathsf{\Psi }}^{*}\right)\right)}{1-d\left({\mathit{x}}^{\prime }|{\mathsf{\Psi }}^{*}\right)}\right)a_g^{\prime }\left({\mathit{x}}^{\prime }\left[i\right]\right) \\ \mathsf{\Lambda }=\mathsf{\Lambda }+\gamma \left(\frac{a_D^{\prime }\left(D\left(\mu \left(\mathit{x}\right)|\mathsf{\Lambda }\right)\right)}{D\left(\mu \left(\mathit{x}\right)|\mathsf{\Lambda }\right)}-\frac{a_D^{\prime }\left(D\left(\mu \left({\mathit{x}}^{\prime }\right)|\mathsf{\Lambda }\right)\right)}{1-D\left(\mu \left({\mathit{x}}^{\prime }\right)|\mathsf{\Lambda }\right)}+a_d^{\prime }\left(d_0\right)\sum _i\left(\mathit{x}\left[i\right]-{\mathit{x}}^{\prime }\left[i\right]\right)a_g^{\prime }\left({\mathit{x}}^{\prime }\left[i\right]\right)+a_D^{\prime }\left(D_0\right)\sum _i\left(\mu \left(\mathit{x}\right)\left[i\right]-\mu \left({\mathit{x}}^{\prime }\right)\left[i\right]\right)a_g^{\prime }\left(\mu \left({\mathit{x}}^{\prime }\right)\left[i\right]\right)\right) \end{gathered}$$

The experiment is performed on a laptop with CPU AMD64 4 processors, 4GB RAM, Windows 10, and Java 15. The dataset is a set of ten 180x250 images, but convolution layers built in AVA zoom out 3 times smaller due to lack of memory. The four AVA variants will be evaluated by root mean square error (RMSE) with 19 learning rates γ = 1, 0.9,…, 0.1, 0.09, 0.001 because stochastic gradient descent (SGD) algorithm is affected by learning rate and the accuracy of AVA varies a little bit within a learning rate because of randomizing encoded data z in VAE algorithm. Given an AVA was trained by 10 images in the dataset, let imageBest be the best image generated by such AVA, which is compared with the i^th image denoted images[i] in dataset and then, RMSE of such AVA and the i^th image is calculated as follows:

$$\mathrm{R}\mathrm{M}\mathrm{S}\mathrm{E}\left[i\right]=\frac{1}{n_i}\sum _j\frac{1}{2}‖\mathrm{i}\mathrm{m}\mathrm{a}\mathrm{g}\mathrm{e}\mathrm{B}\mathrm{e}\mathrm{s}\mathrm{t}\left[j\right]-\mathrm{i}\mathrm{m}\mathrm{a}\mathrm{g}\mathrm{e}\left[i\right]\left[j\right]‖$$

The overall RMSE of such AVA is average RMSE over N=10 test images as follows:

$$\mathrm{R}\mathrm{M}\mathrm{S}\mathrm{E}=\frac{1}{N}\sum _i\mathrm{R}\mathrm{M}\mathrm{S}\mathrm{E}\left[i\right]$$

This means:

$$\mathrm{R}\mathrm{M}\mathrm{S}\mathrm{E}=\frac{1}{N}\sum _i\frac{1}{n_i}\sum _j\frac{1}{2}‖\mathrm{i}\mathrm{m}\mathrm{a}\mathrm{g}\mathrm{e}\mathrm{B}\mathrm{e}\mathrm{s}\mathrm{t}\left[j\right]-\mathrm{i}\mathrm{m}\mathrm{a}\mathrm{g}\mathrm{e}\left[i\right]\left[j\right]‖$$

(17)

Where N is the number of images, N=10 and n_i is the number of pixels of the i^th image. Obviously, image[i][j] (imageBest[j]) is the j^th pixel of the i^th image (the best image). The notation ||.|| denotes norm of pixel. For example, norm of RGB pixel is $\sqrt{r^2+g^2+b^2}$ where r, g, and b are red color, green color, and blue color of such pixel. The smaller the RMSE is, the better the AVA is. Table 1 shows RMSE values of AVA1, AVA2, AVA3, AVA4, AVA5, and VAE with 10 learning rates γ = 1, 0.9, 0.8,…, 0.1.

Table 1 shows RMSE values of AVA1, AVA2, AVA3, AVA4, AVA5, and VAE with 9 learning rates γ = 0.09, 0.08,…, 0.01.

From experimental results shown in Table 1 and Table 2, RMSE means of AVA1, AVA2, AVA3, AVA4, AVA5, are VAE over all learning rates are 0.1787, 0.1786, 0.1783, 0.1783, 0.1783, and 0.2454, respectively. Because AVA3, AVA4, and AVA5 result the same best RMSE (0.1783), it is asserted that AVA with any discriminators in regardless of decoder discriminator (AVA5) or encoder discriminator (AVA3, AVA4, AVA5) but it seems that encoder discriminator is better than traditional decoder discriminator, for instance, Larsen et al. (Larsen, Sønderby, Larochelle, & Winther, 2016) focused on decoder discriminator. Therefore, we check again RMSE standard deviations of AVA1, AVA2, AVA3, AVA4, AVA5, and VAE which are 0.0028, 0.0017, 0.0019, 0.0011, 0.0012, and 0.0054, respectively. AVA4 results out the best RMSE standard deviation (0.0011), which implies undoubtful that encoder discriminator is as good as decoder discriminator at least because it is necessary to test AVA3 with larger dataset and moreover, AVA4 with smallest RMSE standard deviation leans forward encoder mechanism and so, AVA4 is not as fair as AVA1 without leaning forward decoder mechanism. Note that AVA1 and AVA3 are fairest because they do not lean any encoder/decoder mechanism. We check again RMSE minimum values of AVA1, AVA2, AVA3, AVA4, AVA5, and VAE which are 0.1769, 0.177, 0.1765, 0.177, 0.1768, and 0.2366, respectively. The fair AVA3 gains the best minimum value (0.1765), which asserts again that AVA with encoder discriminator is as good as decoder discriminator at least with note that biases among these minimum values is too small to conclude a conclusion of preeminence of encoder discriminator. Table 3 and Figure 5 show RMSE means, RMSE minimum values, and RMSE standard deviations of AVA1, AVA2, AVA3, AVA4, AVA5, and VAE.

It is concluded from Figure 5 that the corporation of GAN and VAE which produces AVA in this research results out better encoding and decoding performance of deep generative model when RMSE means, standard deviation, and minimum value of all AVA are smaller than the ones of VAE. Moreover, AVA5 which is full of functions including decoder discriminator, decoder leaning, encoder discrimination, and encoder leaning but, it does not produce the best result as expected although is a very good AVA, especially, with regard to mean (0.1783), minimum value (0.1768), and standard deviation (0.0012). The reason may be that it is necessary to test AVA5 with large data. Alternately, in some complex systems, many constraints can annul mutually or maybe they restrict mutually instead of reaching the best result or reach a perfect balance. However, AVA5 in this research is stabler than other ones because encoder performance and decoder performance are proportional together, which means that improvement of encoder is to improve decoder and vice versa.

4. Conclusions

It is undoubtful that AVA is better than traditional VAE due to the built-in discriminator function of GAN that assesses reliability of data. I think that VAE and GAN are solid models in both theory and practice when their mathematical foundation cannot be changed or transformed but it is still possible to improve them by modifications or combinations as well as applying them into specific applications where their strong points are brought into play. In applications related to raster data like image, VAE has a drawback of consuming much memory because probabilistic distribution represents entire image whereas some other deep generative models focus on representing product of many conditional probabilistic distributions for pixels. However, this pixel approach for modeling pixels by recurrent neural network does not consume less memory but it is significantly useful to fill in or recover smaller damaged areas in a bigger image. In the future trend, I try to apply the pixel approach into AVA, for instance, AVA processes a big images block by block and then, every block is modeled by conditional probability distribution with recurrent neural network as well as long short-term memory network.