Survey of Optimization Algorithms in Modern Neural Networks

1. Introduction

The question of rising the accuracy of neural networks remains actual. There are many approaches applied for solving this problem: data augmentation [1], improving the mathematical model of neurons [2], adding complement neural network [3] and so on. Indeed, all this approaches resolve the problem of increasing the accuracy of neural networks particularly. But there exist universal methods for improving the quality of various types of networks, and one of them is the optimization of the loss function. The problem of minimization of such functions, which contain local extremes, is a descent to the neighborhood of the global minimum, which remains important in machine learning. First attempts to achieve the minimal value were realized by using stochastic gradient descent (SGD) [4]. Such approach is still widely used in modern neural networks and belongs to set of first order optimization methods. Later, there was stochastic gradient descent with momentum and Nesterov condition (SGDM Nesterov) [5], which increased the rate of convergence and gave the high accuracy for less number of iterations.

According to the structure of SGD, there were proposed AdaGrad [6], RMSProp [7], Adadelta [8] and Adam [9], which allow to attain the minimum of functions that contain little number of local extremes. But in case of Rastrigin, Rosenbrouck and Ackley functions from [10] these approaches are not able to reach the global minimum. It is explained by achieving the minimum taking into account only directions of gradients. Even presence of exponential moving averages in Adam do not suffice for minimization such test functions. The same technique is used in modifications of Adam and SGD. Adam-type algorithms, which updates the exponential moving averages of the gradient, were build to improve the process of minimization for multi-extreme functions. The advantage of these approaches is that the initialization of biases can be easily counteracted, resulting in bias-corrected estimates. However, they are able to minimize the loss function, but not always in global minimum. Using Adam-type algorithms, it is possible to receive required accuracy without big time consumption. Therefore, for attaining higher accuracy one needs to apply second order optimization algorithms.

The main goal of second order optimization algorithms [11] is the achieving global minimum for short number of iterations, because, usually, they are slower compared with first order optimization algorithms. It can be explained by the fact, that second order algorithms take into account not only directions of gradients, but the convexity (curvature) of objective functions by Hessian matrix. Computations of inverted Hessian matrix make the second optimization more complex. This approach is called Newton optimization method [12]. In machine learning, where number of neurons can attain over one hundred, Newton optimization is an ineffective tool, but the approximation of inverted Hessian matrix makes possible to minimize the loss function for required time consumption. Such technique is called quasi-Newton optimization method [13]. Quasi-Newton optimization algorithms analyze the loss function from functional point of view, which allows to avoid local minimums and rapidly converge in global minimum. For increasing the quality of minimization, it is possible to observe the objective function from geometric and probabilistic points of view.

First attempt to geometrically minimize smooth functions was applied in [14], where authors engaged the properties of Rimannian geometry. Such optimization used the gradient flow for receiving the global minimum. Later, there was an idea to engage information geometry in [15], which is an intersection of Riemannian geometry and probability theory. Such approach is divided in two branches, which utilize Fisher-Rao [16] and Bregman [17] metrics, respectively. The optimization with Fisher-Rao metrics (Fisher information matrix) is called a natural gradient descent. In case of utilizing Bregman metrics, one receives the mirror descent. These optimization algorithms proved their abilities in image recognintion [18], time series prediction [19]. The main application they find in physics-informed neural networks [20], which are devoted to solve partial differential and integro-differential equations.

In this paper the main goal is the classification of optimization algorithms, which are used in machine learning, identification their properties, which allow to increase the rate of convergence for certain neural network, and providing ways of developing minimization methods in further researches. After gradient-based optimization methods, we will demonstrate alternative approaches in minimization of loss function: gradient-free and bilevel optimization.

All provided above optimization algorithms are presented in the Figure 1, which constructs the contests of this paper.

The remainder of the paper is organized as follows. Section 2 is devoted to first order optimization algorithms, which are described in 3 subsections: SGD-type, Adam-type and PNM-type optimization algorithms. Section 3 presents second order optimization techniques, such as Newton and Quasi-newton algorithms. Section 4 consists of representation of information-geometric methods, which are divided on Fisher-Rao and Bregman metrics. Such Part of this paper contains the most progressive approaches in modern theory of optimization methods. Section 5 represents applications of all optimization algorithms, introduced in Section 2, Section 3 and Section 4. In Section 6, we report the conclusions of the provided optimization algorithms and suggestions for their improving and further exploitation in neural networks.

2. First Order Optimization Algorithms

2.1. SGD-Type Algorithms

How can we see from Figure 1, the earliest first order optimization algorithm is SGD [4], which can be described by the iterative formula

$${\theta }_{t+1}={\theta }_t-{\alpha }_t\nabla f\left({\theta }_t\right),$$

(1)

where ${\theta }_t$ denotes weight, $f\left({\theta }_t\right)$ is a loss function with its gradient $\nabla f\left({\theta }_t\right)$ and ${\alpha }_t$ is a learning rate. Later such approach was modified to SGDM with Nestreov condition [5] presented by

$${\theta }_{t+1}={\theta }_t-{\alpha }_t(\nabla f\left({\theta }_t\right)+\mu b_{t+1}),$$

(2)

where ${\theta }_0$ is an initial point, $b^{(k+1)}=\mu b^{\left(k\right)}+(1-\tau )(\nabla f\left({\theta }^{\left(k\right)}\right)+\lambda {\theta }^{\left(k\right)})$, where $\tau$ is a damping parameter and $\mu$ is a momentum. The algorithm of SGDM with Nesterov condition is constructed in [22]. This optimization method is still used in approximation theory and machine learning. Various back propagation methods ([23,24,25,26]) are based on calculating of local partial derivatives, which rectify the value of weights of neural networks using (2). But such approach can be modified to others more effective versions, which converge to minimum faster. Therefore, because of relatively small rate of convergence of SGDM with Nesterov condition and its step-size updating, There was introduced Adaptive gradient method (AdaGrad) in [6].

AdaGrad differs from SGDM with Nesterov condition by the adaptive step-size. This advantage allows to increase learning rate and, simultaneously, reduce time consumption. AdaGrad is described by the following formula

$${\theta }_{t+1}={\theta }_t-\frac{{\alpha }_t}{\sqrt{G_{t+1}+ϵ}}\nabla f\left({\theta }_t\right),$$

where the sum of gradients is presented as

$$G_{t+1}=G_t+\nabla f\left({\theta }_{t-1}\right).$$

As was noted above, AdaGrad updates the stepsize ${\alpha }_t$ on the given information of all previous gradients observed along the way. But its disadvantage is similar, like in SGD, where minimization is based on directions of the gradients and step-size regulation, which does not guarantee the convergence in the neighborhood of global minimum. Afterwards, Adagrad was equipped with the gradient norm information, which improves the convergence rate. Such modification is called AdaGrad-Norm [6], presented as follows:

$${\theta }_{t+1}={\theta }_t-\frac{\alpha }{1+(t-1)\eta }\frac{g_t}{\sqrt{G_t}+ϵ}.$$

(3)

AdaGrad and AdaGrad-Norm adapt the step-size more accurately, compared with SGD and SGDM with Nesterov condition, what allows to increase the learning rate for growing the probability of reaching the global minimum. But accumulation of previous gradients does not resolve the problem of convergence in the neighborhood of global minimum. This approach, like SGD, descends towards negative directions of gradients. Such disadvantage caused researchers to come up with idea of adapting of the step-size using mean moments, what implied the root mean square propagation (RMSProp) optimization algorithm.

The RMSprop optimizer [7] partially uses the same technique as SGDM and limits the oscillations in the upright direction, what allows to increase the learning rate. It makes possible to take fast substantial step-sizes in the horizontal direction converging $v_t$. Such approach is described as

$$v_t=\gamma v_{t-1}+(1-\gamma )g_t^2,$$

$${\theta }_{t+1}={\theta }_t-{\alpha }_t\frac{g_t}{\sqrt{v_t}+ϵ},$$

(4)

where $\gamma \in (0,1)$ is a moment. RMSProp is still actual algorithm in many neural networks, like SGDM with Nesterov condition, because it contains prerequisites of exponential moving averages, which later will have a lot of modifications, that give an opportunity to avoid some local minimums. Moreover, according to techniques in AdaGrad and RMSProp, there was introduce AdaDelta [8].

AdaDelta is a modification, based on techniques in SGD, RMSProp and AdaGrad, that separate dynamic learning rate per-dimension, requires no manual setting of a learning rate and takes minimal computation over gradient descent. Additionaly, it is insensitive to hyperparemeters and robust to blow-up gradient, noise and architecture choice. Unlike Adagrad such method reduces aggressive, monotonically decreasing learning rate. Adadelta restricts the window of accumulated past gradients to a fixed size, instead of accumulating all past squared gradients. The running average $E{\left[g^2\right]}_t$ depends only on the previous average and current gradient. Initial accumulation variables $E{\left[g^2\right]}_0$, $E{[\Delta {\theta }^2]}_0$ are equal to 0. AdaDelta algorithms is described as follows. Accumulate gradient:

$$E{\left[g^2\right]}_t=\rho E{\left[g^2\right]}_{t-1}+(1-\rho )g_t^2,$$

where $\rho$ is a decay rate paremeter. Compute update:

$$\Delta {\theta }_t=-\frac{RMS{[\Delta \theta ]}_{t-1}}{RMS{\left[g\right]}_t}{g}_t,$$

where

$$RMS{\left[g\right]}_t=\sqrt{E{\left[g^2\right]}_t+ϵ}.$$

Accumulate updates:

$$E{[\Delta {\theta }^2]}_t=\rho E{[\Delta {\theta }^2]}_{t-1}+(1-\rho )\Delta {\theta }_t^2,$$

$${\theta }_{t+1}={\theta }_t+\Delta {\theta }_t.$$

(5)

The stochastic gradient descent was modified in other variations, which, like AdaGrad, AdaDelta and RMSProp, increase the test accuracy and accelerate the implementation of algorithm. Such modifications can improve the process of recognition, prediction, generation and making decision, what develops the theory of neural networks. But in deep convolutional neural networks modification (2)-(5) does not achieve the higher accuracy compared with SGDM with Nesterov condition, how can be seen in [27], where only some modified algorithms presented in next subsection give higher test accuracy of recognition. Therefore, one needs algorithms, which take into account gradient directions and values weight. First very important modification of SGDM is Nesterov Accelerated Gradient (NAG), presented in [28].

Nesterov’s accelerated gradient descent is widely used in practice for training deep networks and other supervised learning models, because it often provides significant improvements over SGDM with Nesterov condition. Rigorously speaking, “fast gradient” methods have provable ammendments over gradient descent only for the deterministic case, where the gradients are exact. In the stochastic case, "fast gradients" partially mimic their exact gradient counterparts, resulting in some practical gain. Such approach is described as

$$v_t=\mu v_{t-1}+\alpha f(\theta -\mu v_{t-1}),$$

$${\theta }_t={\theta }_{t-1}+v_t.$$

(6)

As said before, there is an efficient technique of utilizing Lebesgue norm on weight of neurons for improving resulting test accuracy. Such approach is called $L^2$ regularization. It is well described and tested on SGDM with Nesterov condition in [29].

Proposed $L^2$ regularization in [30] acts similarly with the usual weight decay, which is used in SGD. Indeed, both approaches evaluate weights closer to zero with the same rate. The $L^2$ regularized SGD (SGDW) is a modification of usual SGD, which differs by recovering original weight decay using decoupling weight decay from the optimization steps with respect to the loss function. However, for adaptive gradient algorithms the main difference is a presence of adapted sums of gradients of the loss function and the regularizer, whereas with decoupled weight decay, only gradients of the loss function are adapted. With $L^2$ regularization both types of gradients are normalized by their typical summed magnitudes, and therefore weights with large typical gradient magnitude are regularized by a smaller relative amount than other weights. The introduced SGDW method is described by the following iterative formula

$$g_t=\nabla f\left({\theta }_t\right)+\lambda {\theta }_{t-1},$$

where $\lambda$ is a weight decay parameter ($L_2$ regularization factor),

$$m_t={\beta }_1{m}_{t-1}+{\eta }_t\alpha g_t,$$

$${\theta }_t={\theta }_{t-1}-m_t-{\eta }_t\lambda {\theta }_{t-1},$$

(7)

where ${\eta }_t$ is a schedule multiplier. Unfortunately, such SGDW does not significantly improve usual SGD, especially in deep neural networks, auto-encoders and graph-neural networks. Because the architecture of these neural networks confuses the $L^2$ regularization and make the process of minimization too difficult for SGDW. But there are two techniques, which can improve the quality of minimization of the loss function using projection [31] and hyper-parameter methods [32].

Before introducing the projection technique for SGD, it is necessary to recall a batch normalization [33]. In practise, normalization techniques, such as batch normalization, play an important role in for modern deep learning. They allow weights converge more rapid with better generalization performances. The normalization-induced scale invariance among the weights gives advantages to SGD, such as the effective step-size automatic regularization and stabilizing the training procedure. In practise, one can notice that the including momentum in SGD-type optimizers reduces much step-sizes for scale-invariant weights. Such phenomenon is not yet studied and causes unwanted side effects in the process of minimization. This is a crucial issue because the vast majority of modern deep neural networks consist of SGD- and Adam-type optimizer, which contain momentum, and scale-invariant parameters. Therefore, there was proposed SGD with projection (SGDP), which removes the radial component, or the norm-increasing direction, at each optimizer step. Because of the scale invariance, this modification only alters the effective step-sizes without changing the effective update directions, thus satisfying the original convergence properties of GD optimizers.

SGDP can be presented as following iterative formulas:

$$\begin{array}{c}\hfill p_t=\mu p_{t-1}+\nabla f\left({\theta }_{t-1}\right),\\ \hfill {\theta }_t={\theta }_{t-1}-\alpha p_t.\end{array}$$

(8)

This approach, compared with SGDW, increases the rate of convergence, but it is not enough to significantly advance the process of minimization of the loss function in global minimum. what can be seen in experiments of minimizing Rastrigin function in (https://github.com /jettify/pytorch-optimizer). But the problem of avoiding the local minimums was solved by introducing the hyper-parameter methods, which is called quasi-hyperbolic momentum algorithm (QHM).

Momentum-based acceleration of SGD is widely used in deep learning. There is presented QHM as an extremely simple alteration of momentum SGD, averaging a plain SGD step with a momentum step, what presented in [34]. This approach introduces the immediate discount factor $\nu$, encapsulating plain SGD ($\nu =0$) and momentum ($\nu =1$). A self-evident interpretation of QHM is a $\nu$-weighted average of the momentum update step and the plain SGD update step. The expressive power of QHM intuitively comes from decoupling the momentum buffer’s discount factor $\beta$ from the current gradient’s contribution to the update rule $1-\nu \beta$. In contrast, momentum tightly couples the discount factor $\beta$ and the current gradient’s contribution $1-\beta$.

Let us present QHM as the following iterative process:

$$g_t=\beta g_{t-1}+(1-\beta ){\widehat{g}}_t,$$

$${\theta }_t={\theta }_{t-1}-\alpha \left[(1-\nu )·{\widehat{g}}_t+\nu g_t\right].$$

(9)

Such method lets surmount the local minimums. This approach has no disadvantages, compared with demonstrated above algorithms. But, like others, it QHM does not suffice for achieving the highest accuracy in deep neural networks, because it still takes into account the direction of gradients and amendments with parameters and gradient normalization.

Before drawing conclusions, it should be noted that for any smooth objective function $f:{\mathbb{R}}^n\to \mathbb{R}$ provided optimization methods in above have almost the same regret bound [35,36]

$$R\left(T\right)=\sum _{t=1}^T{f}_t\left(x_t\right)-\underset{x\in {\mathbb{R}}^n}{min}\sum _{t=1}^T{f}_t\left(x\right),$$

(10)

which measures the convergence rate and belongs to set of $O\left(\sqrt{T}\right)$. Presented above algorithms can be improved by performing their regret bounds to $O(logT)$, which was done in [37] and. Such algorithms are called strongly convex.

SGD-type algorithms are used in convolutional, recurrent, autoencoder, graph neural networks. But their huge disadvantage is insufficient information about the properties of loss function to reach the global minimum. SGD is led by the direction of gradient, which is not enough for increasing the test accuracy. In modern neural networks more preferred approaches are Adam and its modifications (Adam-type algorithms), which contain exponential moving averages of gradient and squared gradient, that significantly improves the quality of training neural networks. These methods improve the optimization by estimation of moments, which gives more information of about the global minimum. Moreover, such improvement allows to increase the accuracy of image recognition, time series prediction and object classification.

2.2. Adam-Type Algorithms

The adaptive moment estimate (Adam) [39] algorithm is a continuation of developing gradient based optimization, which significantly increases the accuracy in neural networks. Taking into account directions of gradients and means of moments, minimization of loss function with higher frequency converges in the neighborhood of global minimum. The iterative formula is presented as

$$m_t={\beta }_1{m}_{t-1}+(1-{\beta }_1)g_t,$$

$$v_t={\beta }_2{v}_{t-1}+(1-{\beta }_2)g_t^2,$$

$${\widehat{m}}_t=m_t/(1-{\beta }_1^t),{\widehat{v}}_t=v_t/(1-{\beta }_2^t),$$

$${\theta }_t={\theta }_{t-1}-\alpha {\widehat{m}}_t/(\sqrt{{\widehat{v}}_t}+ϵ).$$

(11)

Note that ${\beta }_1,{\beta }_2$ are called moments, $m_t$ and $v_t$ are exponential moving averages of $g_t$ and $g_t^2$, respectively.

According to the adaptive moment estimate algorithm (11), there were provided modified versions, which distinct by step-size adaptation and manipulation with exponential moving averages. Like SGDW, there was provided a modification in [40], which is called Adam algorithm with $L^2$ regularization. In the case of usual Adam, weights, that tend to have large gradients of f, are not regularized as much as they would with decoupled weight decay, since the gradient of the regularizer is scaled. AdamW with the same exponential moving averages can be described as the following iteration formula:

$${\theta }_t={\theta }_{t-1}-\eta +t\left[\alpha ·{\widehat{m}}_t/(\sqrt{{\widehat{v}}_t}+ϵ)+\lambda {\theta }_t\right],$$

(12)

where ${\eta }_t$ is a schedule multipliers and $\lambda$ is a $L^2$ regularization factor. There is proved fact, that such modification with decoupled weight decay yields substantially better generalization performance than the common implementation of Adam with $L^2$ regularization, in [41]. Another approach for modification of Adam algorithms is projected Adam, which is called AdamP [42].

AdamP, like SGDP in (8), is based on projection of the sum of gradient and momentum vectors onto the tangent space of weights. Such approach allows to accelerate effective step sizes for scale-invariant weights. This technique with its applications is described in [43] and has the following representation:

$$g_t=\nabla f\left({\theta }_{t-1}\right)+\lambda {\theta }_{t-1},$$

$$m_t={\beta }_1{m}_{t-1}+(1-{\beta }_1)g_t,$$

$$v_t={\beta }_2{v}_{t-1}+(1-{\beta }_2)g_t^2,$$

$$p_t=\frac{m_t}{\sqrt{v_t}+ϵ}.$$

Then, according to cosine condition we obtain

$$ifcos\left(g_i\right)<\delta /\sqrt{dim\left(\theta \right)}:q_i=p_i-(\frac{\theta }{{\parallel \theta \parallel }_2}·p_i)\frac{\theta }{{\parallel \theta \parallel }_2}$$

$$else:q_i=p_i$$

Afterwards, one obtain

$${\theta }_t={\theta }_{t-1}-{\eta }_t\left(\frac{\alpha m_t}{\sqrt{v_t}+ϵ}+\lambda {\theta }_{t-1}\right).$$

(13)

According to notes in [44], it implies that momentum-based optimizers induce an excessive growth of the scale-invariant weight norms, which prematurely decay the effective optimization steps, leading to sub-optimal performances. The resulting AdamP, like SGDP in Section 2.1, successfully suppresses the weight norm growth and train a model at an unobstructed speed. Another approach to accelerate the convergence rate is the applying the quasi-hyperbolic momentum, like it was made for QHM in (9).

As for QHM, there was proposed QHAdam [45], which replaces both of Adam’s moment estimators with quasi-hyperbolic terms. This approach is described as

$$g_{t+1}={\beta }_1{g}_t+(1-{\beta }_1)\nabla f\left({\theta }_t\right),g_{t+1}^{\prime }={(1-{\beta }_1^{t+1})}^{-1}{g}_{t+1},$$

$$s_{t+1}={\beta }_2{s}_t+(1-{\beta }_2){(\nabla f\left({\theta }_t\right))}^2,s_{t+1}^{\prime }={(1-{\beta }_2^{t+1})}^{-1}{s}_{t+1},$$

$${\theta }_{t+1}={\theta }_t-\alpha \left[\frac{(1-{\nu }_1)\nabla f\left({\theta }_t\right)+{\nu }_1{g}_{t+1}^{\prime }}{\sqrt{(1-{\nu }_2{(\nabla f\left({\theta }_t\right))}^2)+{\nu }_2{s}_{t+1}^{\prime }}+ϵ}\right]$$

(14)

As was noted in previous subsection, there exist Nesterov trick NAG, which accelerates usual SGD in [46]. The same modification propagates to Adam and transform it to NAdam [47]. This technique with higher frecuency converge to the neighborhood of global minimum for less number of iteration, compared with Adam and its previous modifications, and constructed more simple than AdamW, AdamP and QHAdam. But for cases of Rastrigin functions and Rosenbrock functions, (11)-(13) do not converge in neighborhood of global minimum. As was said in Section 2.1, $L^2$ regularization, projection and quasi-hyperbolic parameters techniques influence to step-size, taking into account gradient directions. In cases, when included moment estimation, such modifications can accelerate the process of convergence, but not necessary in the neighborhood of global minimum. But there are two techniques, which make the process of minimization ’smoother’. Such approaches are called Nesterov-accelerated (NAdam) and Rectified (RAdam) adaptive moment estimation.

Let us present the iterative formula of NAdam:

$${\theta }_{t+1}={\theta }_t-\frac{\eta }{\sqrt{{\widehat{v}}_t}+ϵ}\left({\beta }_1{\widehat{m}}_t+\frac{(1-{\beta }_1)g_t}{1-{\beta }_1^t}\right)$$

(15)

This method is a continuation of NAG due to the added exponential moving averages. Compared with Adam and its previous modifications, NAdam increases the accuracy of converging in deep convolutional neural networks and, simultaneously, make the minimization process faster by ’smoothing’ the descent. But such technique is ineffective in physics-informed neural networks, because of the smoothing descent trajectory, which for partial differential equation solution gives more deviations. Such disadvantage was resolved by rectified Adam (RAdam).

Proposed in [48], RAdam differs from other optimization method by introducing a term to rectify the variance of the adaptive language modeling and learning rate. This modification proved its ability to receive higher test accuracy. Such optimization method has the following iterative formula:

$${\rho }_t={\rho }_{\infty }-2t{\beta }_2^t/(1-{\beta }_2^t)$$

$${\rho }_{\infty }=\frac{2}{1-{\beta }_2}-1$$

If the variance is tractable (${\rho }_t>4$) then the adaptive learning rate is computed as:

$$l_t=\sqrt{(1-{\beta }_2^t)/v_t},$$

the variance rectification term is calculated as:

$$r_t=\sqrt{\frac{({\rho }_t-4)({\rho }_t-2){\rho }_{\infty }}{({\rho }_{\infty }-4)({\rho }_{\infty }-2){\rho }_t}}$$

and we update parameters with adaptive momentum:

$${\theta }_t={\theta }_{t-1}-{\alpha }_t{r}_t{\widehat{m}}_t{l}_t.$$

(16)

If the variance is not tractable we update instead with:

$${\theta }_t={\theta }_{t-1}-{\alpha }_t{\widehat{m}}_t.$$

(17)

Such method overtake NAdam and other algorithms, especilly in deep neural networks [49], such as AlexNet [50], ResNet [51], InceptionNet [52], GoogleNet [53] and Res-Next [54]. But RAdam adapts the learning rate too complex and, like previous analogues, can not converge in the neighborhood of global minimum of Rastrigin function. Moreover, there exist other simple ways to adjust the learning rate and make the process of minimization faster. One of them is difference gradient approach, which is called DiffGrad [55].

The difference gradient approach is based on moment estimate technique and, instead of complex manipulations with learning rate and weights, calculate only additional coefficient, which is called DiffGrad friction coefficient (DFC). The main distinction of DiffGrad is that such approach is based on the change in short-term gradients and controls the learning rate based on the need of dynamic adjustment of learning rate. This means that diffGrad follows the norm that the parameter update should be smaller in low gradient changing regions. The friction coefficient diffGrad (DFC) is designed to control the learning rate using information about the short-term behavior of the gradient. The DFC is represented by ${\xi }_t$ and defined as

$${\xi }_t=\frac{1}{1+exp-|\Delta g_t|},$$

(18)

where $\Delta g_t$ is the change between previous and current gradients, given as

$$\Delta g_t=g_{t-1}-g_t.$$

In the proposed DiffGrad optimization method, the steps up to the computation of bias-corrected 1-st order moment $m_t$ and bias-corrected 2-nd order moment $v_t$ are the same as those of Adam optimization [38]. The DiffGrad optimization method updates ${\theta }_{t+1}$, using the following update rule:

$${\theta }_{t+1}={\theta }_t-\frac{{\alpha }_t{\xi }_t{\widehat{m}}_t}{\sqrt{{\widehat{v}}_t}+ϵ}.$$

(19)

DiffGrad generate a high learning rate if the gradient change is more (i.e., the optimization is far from the optimum solution), and a low learning rate if the gradient changes minimally (i.e., the optimization is near to the optimum solution). Moreover, such technique lets to avoid some local minimums, which can be seen in minimization of Rastrigin and Rosenbrock functions [56]. This approach is suitable for deep convolutional neural networks due to moment estimation, analysing past and current gradient. Also, DiffGrad adjusts the learning rate very accurately for avoiding overshooting of the global minimum and reducing oscillation around it. Such algorithms is tested with the ResNet50 model for an image categorization task over the CIFAR10 and CIFAR100 datasets in [57]. According to result of recognition, it became clear that DiffGrad works better, compared with SGDM (2), AdaDelta (5) and Adam (11). But this approach does not guarantee the high accuracy in other neural networks, especially in quantum, spiking, complex valued and physics-informed neural networks. Such disadvantage is explained by the lack of analyzing the curvature of minimizing loss function. For that reason, there was introduced progressive optimization algorithm in [58], which is called Yogi.

The Yogi algorithms, like Adam, relies on scaling gradients down by the square root of exponential moving averages of past squared gradients and controls the increase in effective learning rate, leading to even better performance with similar theoretical guarantees on convergence in [59]. It allows to solve the problem of convergence failure in simple convex optimization settings, which Adam-type algorithms, like AdamW, AdamP, QHAdam, NAdam and RAdam, can not handle. The difference between $v_t$ and $v_{t-1}$ and its magnitude depend on $v_{t-1}$ and $g_t^2$. When $v_{t-1}$ is much larger than $g_t^2$, Yogi, like Adam, increase the effective learning rate, but such procedure is more controllable. This approach has the following describtion:

$$m_t={\beta }_1{m}_{t-1}+(1-{\beta }_1)g_t,$$

$$v_t=v_{t-1}-(1-{\beta }_2)sign(v_{t-1}-g_t^2)g_t^2,$$

$${\widehat{m}}_t=m_t/(1-{\beta }_1^t),{\widehat{v}}_t=v_t/(1-{\beta }_2^t),$$

$${\theta }_t={\theta }_{t-1}-\gamma {\widehat{m}}_t/(\sqrt{{\widehat{v}}_t}+ϵ).$$

(20)

This method shows better results in deep convolutional neural networks, compared with DiffGrad and other previous Adam-type algorithms. But authors from [60,61] defined new optimization methods for deep learning such as AdaBelief and AdaBound.

The main feature of AdaBelief is adapting the learning rate, according to the "belief" in the current gradient direction. There is a difference between AdaBelief and Adam in parameters $v_t$ and $s_t$, which are defined as exponential moving averages of $g_t^2$ and ${(g_t-m_t)}^2$, respectively. According to $s_t$ as the prediction of the gradient at the next time step, if the observed gradient greatly deviates from the prediction, then one distrust the current observation and take a small step; if the observed gradient is close to the prediction, therefore one trust it and take a large step. This allows to achieve the high test accuracy in convolutional neural networks, that was presented in [61] on ImageNet and CIFAR10.

$$m_t={\beta }_1{m}_{t-1}+(1-{\beta }_1)g_t,$$

$$s_t={\beta }_2{s}_{t-1}+(1-{\beta }_2){(g_t-m_t)}^2+ϵ,$$

$$\widehat{m_t}=\frac{m_t}{1-{\beta }_1^t},\widehat{s_t}=\frac{s_t}{1-{\beta }_2^t},$$

$${\theta }_t={\theta }_{t-1}-\gamma {\widehat{m}}_i/(\sqrt{{\widehat{s}}_i}+ϵ).$$

(21)

It should be noticed that such approach has a modified version with fast gradient sign method (FGSM) presented in [62,63].

The AdaBound method gives an opportunity to restrict the learning rate between upper and lower continuous functions, which are called clips. Such technique reduces the probability of vanishing and blow-up gradient. This approach is defined as

$$m_t={\beta }_1{m}_{t-1}+(1-{\beta }_1)g_t,$$

$$v_t={\beta }_2{v}_{t-1}+(1-{\beta }_2)g_t^{2}and{V}_t=diag\left(v_t\right),$$

$${\widehat{\eta }}_t=Clip(\alpha /\sqrt{V_t},{\eta }_l\left(t\right),{\eta }_u\left(t\right))and{\eta }_t={\widehat{\eta }}_t/\sqrt{t},$$

$${\theta }_{t+1}={\theta }_t-{\eta }_t{\widehat{m}}_t/(\sqrt{{\widehat{v}}_i}+ϵ).$$

(22)

This approach is more complex, compared with DiffGrad, Yogi and AdaBelief, but it is capable to converge in the neighborhood of global minimum. There were made experiments of minimization of Rastrigin and Rosenbrock functions (https://github.com/jettify/pytorch-optimizer), where AdaBound achieved the highest accuracy, converging in the neighborhood of global minimum. But such approach is too complex for optimization and there exists much simple method, which is called AdamInject [64]. It reduces time consumption, preserving convergence rate.

AdamInject is one of the most recent approaches in first order optimization algorithms, which, unlike the AdaBelief algorithm, modifies $m_t$, which is the exponential moving average of $g_t$, into $s_t$. Such parameter is equipped with the difference between the previous parameters ${\theta }_{t-1}$ and ${\theta }_{t-2}$. AdamInject has the following description.

If $t=1$:

$$s_t={\beta }_1{s}_{t-1}+(1-{\beta }_1)g_t.$$

Else:

$$\Delta \theta ={\theta }_{t-2}-{\theta }_{t-1}$$

$$s_t={\beta }_1{s}_{t-1}+(1-{\beta }_1)(g_t+\Delta g_t^2)/k.$$

Afterwards, one make usual calculations

$$v_t={\beta }_2{v}_{t-1}+(1-{\beta }_2)g_t^2$$

$$\widehat{s_t}=\frac{s_t}{1-{\beta }_1^t},\widehat{v_t}=\frac{v_t}{1-{\beta }_2^t}$$

$${\theta }_t={\theta }_{t-1}-\gamma \widehat{s_t}/(\sqrt{\widehat{v_t}}+ϵ)$$

This algorithms was tested on Rastrigin and Rosenbrock functions. Afterwards, there were trained on CIFAR10 VGG 16, ResNet, ResNext, SENet and DenseNet equipped with AdamInject, which presented the best results, compared with known analogues.

Introduced Adam-type optimization algorithms are used in deep convolutional neural networks, like Res-Net, Res-Next, InceptionNet, GoogleNet and so on. Also they find an application in recurrent and spiking neural networks due to their described above advantages, which SGD-type algorithms does not have. But in quantum, complex-valued and quaternion-valued neural networks such approaches loses in accuracy to usual SGDM with Nesterov condition. This problem caused researches in [65] come up with extending the number of moment from 2 to 3. This approach is called positive-negative momentum (PNM)

2.3. Positive-Negative Momentum

Developing SGD- and Adam-types optimization algorithms can not be infinite, there have to another approaches and techniques for extending class of first-order methods. This issue made the researches to consider the methods, which allow more than two exponential moving averages $m_t$ and $v_t$. Because step-size regularization, according to moment estimation and introduced modification in Section 2.1 and Section 2.2, has its limits, what can be seen in convolutional neural networks, such as ResNet18, GoogLeNet and DenseNet. In this cases there have to additional exponential moving average, that let descend to neighborhood of global minimum.

In the paper [66] introduced conventional momentum method, also called Heavy Ball (HB), which is described in [67]. Later there was proposed positive-negative momentum (PNM) optimization algorithms. In this approach the main feature is a positive-negative averaging, which is analogue of exponential moving average in Adam. This averaging is described as follows:

$$m_t=\sum _{k=0}^t{\beta }_3{\beta }_1^{t-k}{g}_k.$$

Inspired by this simple idea, there were proposed the combining the positive-negative averaging with the conventional momentum method in [66]. The positive-negative average is described as

$$m_t=(1+{\beta }_0)m_t^{\left(odd\right)}+{\beta }_0{m}_t^{even}=(1+{\beta }_0)\sum _{k=1,3,...,t}{\beta }_3{\beta }_0^{t-k}{g}_k+{\beta }_0\sum _{k=0,2,...,t}{\beta }_3{\beta }_0^{t-k}{g}_k.$$

The stochastic gradient descent equipped with the conventional momentum estimates, which adjust the learning rate and value of the gradient, is written in the following formula

$$m_t={\beta }_1^2{m}_{t-2}+(1-{\beta }_1^2)g_t,$$

$${\theta }_{t+1}={\theta }_t-\frac{\eta }{\sqrt{{(1+{\beta }_0)}^2+{\beta }_0^2}}[(1+{\beta }_0)m_t-{\beta }_0{m}_{t-1}].$$

(23)

This approach is an analogue of SGD-type algorithm, which differs by presence of positive-negative average $m_t$ and step-size adaptation. There is also proposed Adam-type analogue of PNM algorithm, which is called AdaPNM, described as

$$m_t={\beta }_1^2{m}_{t-2}+(1-{\beta }_1^2)g_t,$$

$${\widehat{m}}_t=\frac{(1+{\beta }_0)m_t-{\beta }_0{m}_{t-1}}{1-{\beta }_1^t},$$

$$v_t={\beta }_2{v}_{t-1}+(1-{\beta }_2)g_t^2,$$

$$v_{max}=max(v_t,v_{max}),{\widehat{v}}_t=\frac{v_{max}}{1-{\beta }_2^t},$$

$${\theta }_{t+1}={\theta }_t-\frac{\eta }{\sqrt{{(1+{\beta }_0)}^2+{\beta }_0^2}}{\widehat{m}}_t.$$

(24)

The advantage of PNM and AdaPNM can be seen in [68], where were tested deep neural networks, such as ResNet, VGG, DenseNet and GoogleNet on image bases CIFAR10 and CIFAR100. These approaches gives higher test accuracy, compared with advances Adam-type optimization algorithms, like Yogi and AdaBound. If one equips AdaPNM and PNM with techniques contained in known analogues with two exponential moving averages, then it can improve the quality of optimization process. Besides these algorithms, there exists the extension of NAG and Nadam, which is called adaptive Nesterov momentum Algorithm (Adan).

In [69] proposed the Adaptive Nesterov momentum algorithm is devoted to effectively accelerate the training of deep neural networks. Adan first reformulates the vanilla Nesterov acceleration to develop a new Nesterov momentum estimation method, which reduces extra computations and memory overhead of computing gradient at the extrapolation point. Adan adopts the first and second order moments of the gradient in adaptive gradient algorithms for convergence acceleration. This approach has the following form:

$$m_t=(1-{\beta }_0)m_{t-1}+{\beta }_0{g}_t,$$

$$v_t=(1-{\beta }_1)v_{t-1}+{\beta }_1(g_t-g_{t-1}),$$

$$n_t=(1-{\beta }_2)n_{t-1}+{\beta }_2{[g_t+(1-{\beta }_1)(g_t-g_{t-1})]}^2,$$

$${\eta }_t=\eta /\left(\sqrt{n_t+ϵ}\right),$$

$${\theta }_{t+1}={(1+\lambda \eta )}^{-1}[{\theta }_t-{\eta }_t(m_t+(1-{\beta }_1))v_t].$$

(25)

This method is the same generalization of SGD, Adam and NAdam. There is presented the advantage of this approach over AdaBelief, which shew third result of test accuracy after usual SGD. Regardless the modifications of Adam-type algorithms, usual SGD can give even better results, what claims to search other methods, that approve their modifications.

First order optimization methods are suitable for problem of image recognition, time series prediction and object classification. They do not consume much time of execution and power, what makes them actual in modern neural networks. But first order optimization algorithms, except PNM, AdaPNM and Adan, can not significantly increase the accuracy in neural networks with complex architecture, such as graph, complex-valued and quantum neural networks. In the solving differential equations they can not overtake the results of Adam, because physics-informed neural networks contain automatic differentiation, which work after multilayer perceptron.. Therefore, one needs second order optimization algorithms, which are able to significantly improve the process of minimization of the loss function.

3. Second Order Optimization Algorithms

First order optimization algorithms are not the most effective for receiving the global minimum of objective smooth function. Such approaches of minimization generally take into account the directions of gradient in every iterations. The customization and performing of class of first order optimization algorithms can only increase the accuracy and avoid some local minimums. Therefore, for improving the quality of minimization there are provided second order optimization algorithms. They reach the minimum taking into account not only directions of gradients, but the convexity (curvature) of objective function, which is measured with the Hessian. Such approach is called as Newton method.

3.1. Newton Algorithms

The main idea of the Newton optimization method [70] is based on gradient descent, containing the calculation of inverse Hessian of smooth objective function. Such approach increases the accuracy of minimization of functions with multiple number of local minimums. Newton optimization algorithm is described as

$${\theta }_{t+1}={\theta }_t-Hess{\left({\theta }_t\right)}^{-1}\nabla f\left({\theta }_t\right),$$

(26)

where $t=0,...,M>0$ and

$$Hess\left({\theta }_t\right)={\nabla }^{2}f\left({\theta }_t\right).$$

(27)

This iterative formula is received from second order Taylor expansion.

In [71] were introduced Newton-Minimum-Residual (Newton-MR) optimization method, which calculates the Hessian matrix by Moore-Penrose inverse [72] operator ${[·]}^{†}$ such as

$${\theta }_{t+1}={\theta }_t+p_t={\theta }_t+{\left[{\nabla }^{2}f\left({\theta }_t\right)\right]}^{†}\nabla f\left({\theta }_t\right).$$

(28)

According to Newton and Newton-MR methods, there were suggested OverSketch Newton Fast convex optimization. Such approach is described in [73] with proper algorithms, which compute the update direction, taking into account the case of strongly-convexity. But there are other modifications of Newton’s optimization methods, that differs by their simplicity in realization and implementation in neural networks.

There are Krylov methods, such as conjugate gradients (CG) [74], the minimal residual method (MINRES) [75], and the generalized minimal residual method (GMRES) [76]. GMRES applies to indefinite matrices, MINRES applies to symmetric indefinite matrices. Besides GD, MINRES and GMRES, there exists a generic stochastic inexact Newton-Krylov method, described in [77]. Such approach can be implemented and applied in physics-informed neural networks, because of suitable approximations for time-dependent equations with divergence operator $(\nabla ·)$ . But for vast majority of neural networks CG is the most preferred.

One of the most dispersed Newton’s optimization method is conjugate gradient. This approach comprises a class of unconstrained optimization algorithms, characterized by low memory requirements and strong local and global convergence properties. Such optimization algorithms is described as the following iterative formula:

$${\theta }_{t+1}={\theta }_t+{\eta }_t{d}_t,$$

$$d_{t+1}=-g_{t+1}+{\beta }_t{d}_t,d_0=-g_0.$$

The main part of this formula is ${\beta }_t$, which is called CG update parameter, which are presented in Table 1.

Despite the increasing the accuracy of minimization, Newton method optimization is slower comparing with the first order optimization. This disadvantage significantly impacts on time consumption, that slows the training of deep neural networks. For accelerating minimization process, there are developed many approximations of Hessian matrix. Second order optimization algorithms, which contain the approximation of Hessian matrix, are called quasi-Newton.

3.2. Quasi-Newton Algorithms

The class of quasi-Newton optimization algorithms shows approximately the same accuracy as Newton optimization algorithms, but ability to converge faster makes them useful in machine learning. Ones of the first quasi-newton optimization algorithms are BFGS [87], and L-BFGS [88].

BFGS method and its limited memory version, at the t-th iteration, is presented as the following iterative formula:

$${\theta }_{t+1}={\theta }_t-{\alpha }_t{H}_t\nabla f\left({\theta }_t\right),$$

(29)

where ${\alpha }_t$ is the step length, $\nabla f\left({\theta }_t\right)$ is the gradient and $H_t$ is the inverse BFGS Hessian approximation, that is updated at every iteration by means of the formula

$$H_{t+1}=V_k^T{H}_t{V}_t+{\rho }_t{s}_t{s}_t^T,$$

(30)

$${\rho }_t=\frac{1}{y_t^T{s}_t},V_t=I-{\rho }_t{y}_t{s}_t^T,$$

(31)

where the curvature pairs $(s_t,y_t)$ are defined as

$$s_t={\theta }_t-{\theta }_{t-1},y_t=\nabla f\left({\theta }_t\right)-\nabla f\left({\theta }_{t-1}\right).$$

(32)

The curvature pairs $(s_t,y_t)$ are constructed sequentially at every iteration, and inverse Hessian approximation at the t-th iteration $H_t$ depends on iterate and gradient information from past iterations.

The inverse BFGS Hessian approximations have to satisfy secant and curvature conditions:

$$H_{t+1}{y}_t=s_t,s_t^T{y}_t>0,$$

(33)

Consequently, as a result, as long as the initial inverse Hessian approximation is positive definite, then all subsequent inverse BFGS Hessian approximations are also positive definite. Note, the inverse Hessian approximation $H_{k+1}$ differs from the approximation $H_k$ by a rank-2 matrix.

In the limited memory version, the matrix $H_k$ is defined at each iteration, as the result of applying m BFGS updates to a multiple of the identity matrix, using the set of m most recent curvature pairs $\{s_t,y_t\}$ kept in storage. As a result, one does not need store four dense inverse Hessian matrices approximation, rather one can store two $m\times d$ matrices and compute the matrix-vector product via the two-loop recursion [89]. After the step has been computed, the oldest pair $(s_t,y_t)$ is discarded and the new curvature pair is stored.

Analogically, there were proposed symmetric rank-one (SR-1) [90] formula and its limited memory version in [91]. At the t-th iteration, the SR1 method computes a new iterate by the formula

$${\theta }_{k+1}={\theta }_k+p_k,$$

(34)

where $p_k$ is the minimizer of the following equation

$$\underset{p}{min}{m}_k\left(p\right)=f\left({\theta }_k\right)+\nabla f{\left({\theta }_k\right)}^{T}p+\frac{1}{2}{p}^T{B}_{k}p,$$

(35)

$${\parallel p\parallel }_2\le {\Delta }_k,$$

where ${\Delta }_k$ is the trust region and $B_k$ is the SR1 Hessian approximation computed as

$$B_{k+1}=B_k+\frac{(y_k-B_k{s}_k){(y_k-B_k{s}_k)}^T}{{(y_k-B_k{s}_k)}^T{s}_k}.$$

(36)

Similar to L-BFGS, in the limited memory version of SR1 the matrix $B_k$ is defined at each iteration as the result of applying m SR1 updates to a multiple of the identity matrix, using a set of m correction pairs $\{s_i,y_i\}$ kept in storage.

Provided BFGS and SR-1 with their low memory versions preserve the rate of convergence and reduce the computational complexity. But this methods remain ineffective in deep neural networks, which makes calculations of matrices of extremely large dimension. Therefore, one needs to come with simplified versions of quasi-Newton methods. One of the most used quasi-Newton optimization method in neural networks is Apollo [92]. Apollo is a quasi-Newton method for nonconvex stochastic optimization, which dynamically incorporates the curvature of the loss function by approximating the Hessian via a diagonal matrix. Importantly, the update and storage of the diagonal approximation of Hessian is as efficient as adaptive first-order optimization methods with linear complexity for both time and memory. To handle nonconvexity, there is replaced the Hessian with its rectified absolute value, which is guaranteed to be positive definite. Experiments on deep neural networks, which recognize images from Cifar10 and ImageNet, show that Apollo achieves significant improvements over other stochastic optimization methods, including SGD and variants of Adam, in terms of both convergence speed and generalization performance. Such approach is described as:

$$g_{t+1}=\nabla f\left({\theta }_t\right),$$

$$m_{t+1}=\frac{\beta (1-{\beta }^t)}{1-{\beta }^{t+1}}{m}_t+\frac{1-\beta }{1-{\beta }^{t+1}}{g}_{t+1},$$

$$\alpha =\frac{d_t^T(m_{t+1}-m_t)+d_t^T{B}_t{d}_t}{(\parallel d_t{{\parallel }_4+ϵ)}^4},$$

$$B_{t+1}=B_t-\alpha ·diag\left(d_t^2\right),$$

$$D_{t+1}=rectify(B_{t+1},1),$$

$$d_{t+1}=D_{t+1}^{-1}{m}_{t+1},$$

$${\theta }_{t+1}={\theta }_t-{\eta }_{t+1}{d}_{t+1},$$

(37)

Apollo is a method, which is devoted to accelerate the process of minimization. But there is a problem of losing accuracy for achieving lower time consumption. Because of this disadvantage, authors in [93] proposed another often used quasi-Newton optimization method in neural networks, which is called AdaHessian.

AdaHessian is a second order stochastic optimization algorithm which dynamically incorporates the curvature of the loss function via adaptive estimates of the Hessian. Second order algorithms are among the most powerful optimization algorithms with superior convergence properties as compared to first order methods such as SGD and Adam. The main disadvantage of traditional second order methods is their heavier per-iteration computation and poor accuracy as compared to first order methods. But AdaHessian includes a fast Hutchinson based method to approximate the curvature matrix with low computational overhead, root-mean-square exponential moving average to smooth out variations of the Hessian diagonal across different iterations and a block diagonal averaging to reduce the variance of Hessian diagonal elements. This approximation of Hessian matrix makes the process of learning faster, preserving the reta of convergence.

Before presenting AdaHessian algorithm, it needs to note that gradient $g_t=\nabla f\left({\theta }_t\right)$ and Hessian $H_t={\nabla }^{2}f\left({\theta }_t\right)$. Let present the following matrix diagonalization of Hessian matrix as

$$D_t=diag\left(H\right)=\mathbb{E}[{\theta }_t\odot \left(H_t{\theta }_t\right)],$$

where ⊙ is a componentwise multiplication of vector. Perform a simple spatial averaging on the Hessian diagonal as follows:

$$D^{\left(s\right)}[ib+j]=\frac{1}{b}\sum _{k=1}^{b}D[ib+k],$$

for $1\le j\le b$, $0\le i\le \frac{d}{b}-1$. Hessian diagonal with momentum:

$${\overline{D}}_t=\sqrt{\frac{(1-{\beta }_2){\sum }_{i=1}^t{\beta }_2^{t-i}{D}_i^{\left(s\right)}{D}_i^{\left(s\right)}}{1-{\beta }_2^t}},$$

$$m_t=\frac{(1-{\beta }_1){\sum }_{i=1}^t{\beta }_1^{t-i}{g}_i}{1-{\beta }_1^t},$$

$$v_t={\left({\overline{D}}_t\right)}^k,$$

$${\theta }_t={\theta }_{t-1}-\eta m_t/v_t.$$

(38)

Second order optimization algorithms have higher rate of convergence, especially in the neighborhood of global minimum. Unfortunately, they are not suitable in deep convolutional, recurrent and spiking neural networks, because of their high complexity. Therefore, one needs methods that satisfy the high rate of convergence and low time consumption. In 2012 there were proposed developing second order optimization algorithms using the smooth manifolds in [94]. But in [95] authors came up with idea of using probability distribution manifolds, instead of smooth. Such technique appeared on the intersection of geometry, probability, statistics and optimization.

4. Information-Geometric Optimization Methods

Second order optimization methods converge faster to the neighborhood of global minimum, compared with first order methods. But their time consumption is not fast enough for deep neural networks. Therefore, it is necessary to make quasi-Newton optimization algorithms faster. The most recent way is applying the information geometry. In this section we present a concise and modern view of the basic structures of information geometry, and report some applications in machine learning (statistical mixture clustering).

Initially, idea of using means of geometry in problems of optimization takes its beginning sicne 2010 in [96]. Hessian matrix, presented in second order optimization algorithms, does not contain full information about surface of the loss function, moreover, it makes process of descent too complex from computational point of view. But taking into account the geometric structure of the surface and corresponding spaces (Euclidean, hyperbolic, parabolic) allows to define the shortest way for descent and can reduce unnecessary computations. In [97] authors for defining shortest way applied gradient flow, that improved the quality of searching global minimum. This method applied smooth manifolds and generalized definition of gradient. But later, researches came up with idea to use probability distribution manifolds, which prove their advantage over smooth manifolds in problem of optimization. The intersection of probability theory and statistics with Riemannian geometry produces information geometry.

Analogically to information theory, considering the communication of messages over noisy transmission channels, one defines information sciences as the fields of studying connection between data and families of models, i.e. information sciences create methods to transform information from data to models. Such transformation is made by means of probability, statistics and geometry, that allows to include it in the theory of machine learning.

There is another definition of information geometry given by F. Nielson in [98] as the geometry of decision making. It includes model fitting (inference) which can be interpreted as a decision problem, namely, deciding which model parameter to choose from a family of parametric models. This framework was advocated by Abraham Wald [99,100] who considered all statistical problems as statistical decision problems. Dissimilarities (also loosely called distances among others) play a crucial role not only for measuring the goodness-of-fit of data to model (likelihood in statistics, classifier loss functions in machine learning, objective functions in mathematical programming or operations research, etc.) but also for measuring the discrepancy (or deviance) between models.

In this section we show two distinguish optimization algorithms, based on means of information geometry – natural gradient descent [101] and mirror descent [102]. These methods differ from any other optimization algorithms by their ability to measure distances in non-Euclidean domains using Kullback-Leibler and Bregman divergences, resperctively.

4.1. Natural Gradient Descent

Initially, there was proposed gradient descent with gradient flow in [103]. This approach is a generalization of second-order optimization method on Riemannian manifolds, which, particularly, can be Euclidead or non-Euclidean. Let $({\mathcal{M}}^n,g)$ be a Riemannian manifold, where ${\mathcal{M}}^n$ is a topological space, which can be expressed in local coordinate system of an atlas $\mathcal{A}={\left\{(U_i,x_i)\right\}}_i$ of charts $(U_i,x_i)$, and for tangent bundle $T{\mathcal{M}}^n$ Riemannian metric $g:T{\mathcal{M}}^n\otimes T{\mathcal{M}}^n\to \mathbb{R}$.

The Riemannian gradient glow dynamics $\theta \left(t\right)$ for the optimization problem ${min}_{\theta \in {\mathcal{M}}^n}f\left(\theta \right)$, obtained by seeking an infinitesimal change in $\theta \left(t\right)$ that would lead to the best improvement in objective value, while controlling the length of the change in terms of the manifold geometry, that is,

$$\theta (t+dt)=argmi{n}_{\theta }f\left(\theta \right)dt+\frac{1}{2}d{\theta }^{T}g(\theta ,\theta +d\theta )d\theta .$$

(39)

For $dt$, utilizing $d\theta \left(t\right)=\theta (t+dt)-\theta \left(t\right)$, one can substitute $f\left(\theta \right(t\left)\right)+\langle d\theta ,\nabla f\left(\theta \right(t\left)\right)\rangle$ instead of $f\left(\theta \right)$ and receive

$$d\theta \left(t\right)=argmi{n}_{d\theta }\langle d\theta ,\nabla f\left(\theta \left(t\right)\right)\rangle dt+\frac{1}{2}d{\theta }^{T}g(\theta ,\theta +d\theta )d\theta .$$

Solving for $d\theta$, one get:

$$\frac{d\theta \left(t\right)}{dt}=-g{(\theta ,\theta +d\theta )}^{-1}\nabla f\left(\theta \left(t\right)\right).$$

(40)

For better understanding, there are provided examples of Riemannian metric is the standard Euclidean manifold has the corresponding metric $g=I$, that reduces the gradient flow to usual gradient descent. But there is another very important example – probability distribution manifold with K-L divergence metric ([104,105]), that put the beginning of quantum neural networks.

There is noted the advantage of using probability distribution manifolds in [106], which can be equipped not necessary with KL-divergence, but Bergman, Jensen and others. There are presented extension of Rimannian manifolds with Levi-Civita connection $({\mathcal{M}}^n,g,{\nabla }_{LC})$ to conjugate connection manifolds $({\mathcal{M}}^n,g,\nabla ,{\nabla }^{*})$, where ${\nabla }_{LC}=\frac{\nabla +{\nabla }^{*}}{2}$. Conjugate connection manifolds are the private case of divergence manifolds, denoted as $({\mathcal{M}}^n,D^g,{\nabla }^D),{\nabla }^{*D}$. In such manifolds D can be Kullback-Leibler or Bregman divergences. There are two ways to imply natural gradient descent using direct K-L divergence, like in [95], and more fundamental way, presented in [98]. Let $f\in C^{\infty }\left({\mathcal{M}}^n\right)$ be a smooth loss function, ${exp}_{\theta }:T_{\theta }{\mathcal{M}}^n\to {\mathcal{M}}^n$ be the Riemannian exponential map for updating the sequence of points ${\theta }_t$ on the manifold as follows:

$${\theta }_{t+1}={exp}_{{\theta }_t}(-{\alpha }_t{\nabla }_{M}f\left({\theta }_t\right)),$$

(41)

where

$${\nabla }_{M}f\left(\theta \right)={\nabla }_v\left(f\left({exp}_{\theta }\left(v\right)\right)\right){|}_{v=0}=\underset{h\to 0}{lim}\frac{f(\theta +hv)-f\left(\theta \right)}{h}.$$

(42)

Returning to the formula of gradient flow, we receive:

$${\theta }_{t+1}=R_{{\theta }_t}(-{\alpha }_t{\nabla }_{\theta }f\left({\theta }_t\right)).$$

Using the retraction $R_{\theta }\left(v\right)=\theta +v$ which corresponds to a first-order Taylor approximation of the exponential map, one implies the natural gradient descent:

$${\theta }_{t+1}={\theta }_t-{\alpha }_t{g}^{-1}(\theta ,\theta +d\theta ){\nabla }_{\theta }f\left({\theta }_t\right).$$

(43)

Here $g(\theta ,\theta +d\theta )=F\left({\theta }_t\right)$ is called Fisher information matrix the natural gradient is defined as follows:

$${\nabla }^{NG}f\left(\theta \right)=F^{-1}\left(\theta \right){\nabla }_{\theta }f\left(\theta \right).$$

(44)

Afterwards, we come to natural gradient descent:

$${\theta }_{t+1}={\theta }_t-{\alpha }_t{\nabla }_{\theta }^{NG}f\left({\theta }_t\right).$$

(45)

Natural gradient descent differs from the first- and second-order optimization algorithms presented in Section 2 and Section 3, respectively, by ability to converge in global minimum for time consumption, suitable for deep learning. As was said, such approach created new branch in the theory of artificial intelligence – quantum machine learning. Application of neural networks for training quantum circuits required new structure of mathematical model of neurons, appropriate for quantum computations. Afterwards, vanilla natural gradient descent was replaced with quantum in [107]. Such networks are part of developing quantum information theory and quantum computers, that makes the process of training significantly faster. But in actual researches, vanilla natural gradient descent with Dirichlet distributions is already tested on Rastrigin and Rosebrock functions in [108] and exploited in convolutional, recurrent neural networks in [109]. The important thing in this method is selecting appropriate probability distribution, which can increase the rate of convergence and even accelerate learning process. There are presented probability distributions in Table 2, which can improve the quality of minimization of the loss functions.

As can be seen, this is second order optimization algorithm. But selecting appropriate probability distribution, like Gauss and Dirichelt, we can reduce the variable $\theta$ in Fisher infromation matrix, what makes possible to avoid its calculation in every iteration. Such approach is realized in [108,109,110,111]. The natural gradient descent, based on Fisher-Rao metrics, can replace second order optimization algorithms, due to their rate of convergence and time consumption. But there is another approach, that uses Bregman metrics, which is called mirror descent.

4.2. Mirror Descent

Bregman metrics, as an alternative for Fisher-Rao metrics, is based on dual Hessian manifolds. This approach takes into account not only gradient direction, curvature of the minimizing loss function, but the duality. Dual spaces are defined as the set of maps from the topological spaces to their underlying field, which is usually presented as $\mathbb{R}$. This feature impacted on the name of such method. This technique is a reason continuation of second order optimization algorithms towards the information geometry. Moreover, this method contains a potential in the theory of quantum and complex-valued neural networks. Also it can be used in the physics-informed neural networks, where dual averaging procedure can increase the accuracy of final solutions.

Recall that the gradient descent can be extended by proximity function $\Phi (·,·)$ as follows:

$${\theta }_{t+1}=argmi{n}_{\theta }\left\{\langle \theta ,\nabla f\left({\theta }_t\right)\rangle +\frac{1}{{\alpha }_t}\Phi (\theta ,{\theta }_t)\right\}.$$

(46)

If $\Phi (\theta ,{\theta }_t)=\frac{1}{2}{\parallel \theta -{\theta }_t\parallel }_2^2$ then we obtain usual gradient descent. For Bregman divergence the proximity function $\Phi (\theta ,{\theta }_t)=D_{\psi }(\theta ,{\theta }_t)$ and we get mirror descent

$${\theta }_{t+1}=argmi{n}_{\theta }\left\{\langle \theta ,\nabla f\left({\theta }_t\right)\rangle +\frac{1}{{\alpha }_t}{D}_{\psi }(\theta ,{\theta }_t)\right\},$$

(47)

where

$$D_{\psi }(\theta ,{\theta }_t)=\psi \left(\theta \right)-\psi \left({\theta }_t\right)-\langle \theta -{\theta }_t,\nabla \psi \left({\theta }_t\right)\rangle .$$

(48)

According to presented above divergence manifolds, it is possible to imply the equivalence between natural gradient descent and mirror descent. Also, this fact was proven in [98], but without geometric tools. The mirror descent can be interpreted as a natural gradient descent as follows: Bregman mirror descent on the Hessian manifold $(M,g={\nabla }^{2}F\left(\theta \right))$ is equivalent to natural gradient descent on the dual Hessian manifold $(M,g^{*}={\nabla }^{2}F\left(\eta \right))$, where F is a Bregman generator, $\eta =\nabla F\left(\theta \right)$ and $\theta =\nabla F^{*}\left(\eta \right)$.

Indeed, the mirror descent rule yields the following natural gradient update rule:

$$N{G}^{*}:{\xi }_{t+1}={\xi }_t-\eta {\left(g_{\xi }^{*}\right)}^{-1}\left({\xi }_t\right){\nabla }_{\xi }{f}_{\theta }\left(\theta \left({\xi }_t\right)\right),$$

(49)

$$N{G}^{*}:{\xi }_{t+1}={\xi }_t-\eta {\left(g_{\xi }^{*}\right)}^{-1}\left({\xi }_t\right){\nabla }_{\xi }{f}_{\xi }\left({\xi }_t\right)$$

where $g_{\xi }^{*}\left(\xi \right)={\nabla }^2{F}^{*}\left(\xi \right)={\left({\nabla }_{\theta }^{2}F\left(\theta \right)\right)}^{-1}$ and $\theta \left(\xi \right)=\nabla F^{*}$.

The method is called mirror descent because of performing the gradient step in the dual space, which plays the role of ’mirror’. It means that mirror descent search the global minimum, according to duality of probability distribution manifold, which is equivalent to Fisher infomration matrix for natural gradient descent.

Besides the usual mirror descent, there is proposed stochastic mirror descent (SMD) by Nemirovski and Yudin in [113]. This method presented high accuracy on ResNet18 recognising images from Cifar10. For a strictly convex differentiable function $\psi (·)$, called the potential (proximity) function, stochastic mirror descent is presented by the following iterative formula

$$\nabla \psi \left({\theta }_{t+1}\right)=\nabla \psi \left({\theta }_t\right)-\eta \nabla f\left({\theta }_t\right),$$

(50)

which is equivalent to the following expression

$${\theta }_{t+1}=arg\underset{\theta }{min}\left\{\eta {\theta }_t^T\nabla f\left({\theta }_t\right)+D_{\psi }(\theta ,{\theta }_t)\right\},$$

(51)

where Bregman divergence can be presented as

$$D_{\psi }(\theta ,{\theta }_t)=\psi \left(\theta \right)-\psi \left({\theta }_t\right)-\nabla \psi {\left({\theta }_t\right)}^T(\theta -{\theta }_t)$$

(52)

is the Bregman divergence with respect to the potential function $\psi$. Note that $D_{\psi }$ is nonnegative, convex in its first argument, and that, due to strict convexity, $D_{\psi }(\theta ,{\theta }^{\prime })=0$ if and only if $\theta ={\theta }^{\prime }$.

Different choices of the potential function $\psi$ yield different optimization algorithms, which will potentially have different implicit biases. A few examples follow in Table 3.

Potentially, this method can improve the process of minimization of the loss function in convolutional, graph and recurrent neural networks with huge architectures. Also mirror descent can be equipped with the adaptive moment estimation and other modifications proper for first-order optimization methods.

5. Application of Optimization Methods in Modern Neural Networks

Introduced optimization methods find their applications in various architectures of artificial neural networks. All presented above optimization algorithms show good results in some determined problems, which can be solved by means of machine learning.

First order optimization methods are reliable in problems of image recognition, time series prediction, voice detection and text analysis. In such cases one needs deep convolutional and recurrent neural networks, which can be equipped with meta-data [114]. Provided architectures train neural networks too long even with GPU, therefore one needs first optimization methods, which are classified in SGD- and Adam-type algorithms. Such approaches do not consume much time and power, that simplify the process of training. For convolutional neural networks, such as AlexNet, GoogleNet, ResNet, SqueezeNet [115] and VGG [116] it suffice to use SGD-type algorithms for achieving high test accuracy. In case of DenseNet [117], Xception [118], ShuffleNet [119] and GhostNet [120] it is necessary to use advanced Adam-type algorithms, such as DiffGrad, Yogi, AdaBelief, AdaBound, AdamInject and AdaPNM. The more complex deep neural network applied in problem of recognition, the more advanced first order optimization methods it requires. The same proposition holds for recurrent neural networks, where for SVR [121], XGBoost [122], LSTM [123] and GRU [124] is enought to apply SGD-type and Adam-type algorithms, such as Adam, RAdam, NAdam and QHAdam. Recurrent neural networks, containing CNN-LSTM [125], CNN-GRU [126], TCN-LSTM [127] layers, need advanced Adam-type algorithms.

Second-order optimization methods have a higher rate of convergence in the neighborhood of global minimum, but their time consumption grows. Even quasi-Newton methods, which are dedicated to reduce time consumption, preserving the rate of convergence, still consume more time resources compared with first-order optimization algorithms. But they can be used in some convolutional neural networks, such as AlexNet, Res-Net, VGG and SqueezeNet. For these architectures time and power consumption is not critical. The most appropriate algorithms for this architectures are Apollo and AdaHessian. In recurrent neural networks second-order optimization algorithms show better results, compared with first-order, but they rise time of training. But second order methods are good in physics-informed neural networks (PINN). In the problem of finding solution of some partial differential equation with initial and boundary conditions one needs to extensively analyse the loss function. In this case L-BFGS, SR1, Apollo and AdaHessian can reach higher accuracy compared with first-order optimization methods. Simple PINN, DeepONet [128], MFNN [129] are not large like CNN, what allows to use second-order optimization methods, which gives solution with the high accuracy. For Riemannian neural networks, which rise the accuracy by determining the geodesic, one prefers to apply Apollo and AdaHessian, because with gradient directions they analyze the curvature of loss function. But for cases of Riemannian convolutional neural networks [130] they can not reduce time and power consumption, and applying first-oder optimization algorithm one loses the test accuracy. Therefore, it is necessary to engage algorithms based on information geometry.

Optimization methods based on information geometry have advantages in speed and accuracy. They can be used in convolutional, recurrent, physics-informed and Rimannian neural networks. Moreover, according to their principle of work there are provided quantum neural networks, which correlate with complex-valued neural network. Natural gradient descent and mirror descent are able to achieve the same accuracy and consume low time resources, as second- and first-order optimization methods, respectively. Such possibilities allows to include such algorithms in convolutional, recurrent, graph and auto-encoder neural networks. Moreover, natural gradient and mirror descent extended the application of neural network in quantum computations, where vanilla natural gradient descent is modified into its quantum analogue. For physics-informed neural networks opportunity to converge in the neighborhood of global minimum, consuming time and power similar to first-order methods, can improve the process of solving partial differential and integral equations. This allows to compete with traditional finite difference, element and volume methods.

The summarizing of the above areas of applications is contained in Table 4.

Note, that CNN, RNN, GNN, PINN, SNN, CVNN, QNN are convolutional, recurrent, graph, physics-informed, spiking, complex-valued and quantum neural networks, respectively. These neural networks proved their ability to solve vast majority of problems, related to recognition, prediction, generation, processing, detecting and so on. All of them belong to set of neural networks with gradient based architectures. Recall, that machine learning is the theory, which studies self-learning algorithms, which also can be classified by means of learning. Gradient based neural networks present one of classes in machine learning. Meanwhile, there are many other ways train, which are advanced gradient and gradient free learning methods.

6. Challenges and Potential Research

Despite the advanced optimization methods, there exist problems, which concern the fundamental theory of machine learning. All algorithms presented above are utilizable for neural networks with gradient backpropagation, which is not the unique way to rectify weights. There are presented potential methods in [173], which let reduce gradient calculation, i.e. make the gradient-free error backpropagation process. Therefore, for minimizing loss function one needs to engage alternative optimization methods. On example Alternating Direction Method of Multipliers in [174] or ensemble neural network in [175], where gradient is reduced. For such models one need to use the following gradient free optimization methods in Table 5.

Gradient free optimization methods attract the interest of many researches, because of their computational simplicity, absence of unwanted differentiation operators and convergence rate, that is not less compared with usual gradient descent. Such approach makes mathematicians to develop alternative mathematical models of neural networks, which show their effectiveness over gradient based neural networks in [193]. But it does not say, that one needs to refuse from the gradient-based optimization algorithms. They have worthy continuation, which is an engaging fractional derivatives for computing the gradient.

Fractional calculus, which is a reason continuation of classical calculus, put significant influence in theories of partial differential and integral equations, approximations, signal processing and optimizations. Attempts to generalize differential operators implied many various properties and helpful propositions, which concern the theory of machine learning. It can be seen by extending gradient based optimization methods by fractional derivatives. Let in Table 6 summarize all known fractional derivatives [194], which can improve the process of minimization of loss functions in neural networks.

Fractional optimization methods have proved their efficiency in [195], but there is a question: that chain rule for fractional derivatives makes the modeling of error propagation more complex? No, because there are implied generalized chain and Leibniz rules in [196], which do not differ much from usual versions. Therefore, it is possible to generalize first order optimization methods from SGD- to Adam-type algorithms.

Simultaneously, there is the problem of developing neural networks, containing bilevel optimization [197]. For neural networks equipped with meta-learning, utilization of provided optimization methods does not suffice. There have to be used bilevel optimization algorithms, such as BSA [198], TTSA [199], HOAG [200], AID-FP [201], AID-CG [202] and stocBiO [203]. Such approaches provide the theoretical guarantee in hyperparameter optimization and meta-learning. Potentionaly, these methods can be improved by means of information geometry or techniques from advanced first order optimization algorithms. Moreover, these algorithms can be useful in physics-informed neural networks.

Another problem in physics-informed neural networks is their extension on delay differential equations. The problem of solving such kind of equations with various type of delay is not solved yet, whereas there is found a model for solving fractional differential equations in [204]. In this case one rises the question about using different activation functions, what, consequently, impact on process of optimization of the loss function. Therefore, one can use optimizers based on information geometry. Afterwards, they can be modified in advanced algorithms, which can guarantee the higher accuracy for minimal number of iterations. Especially, information-geometric methods are actual for complex-valued neural networks.

Complex-valued neural networks shew their efficiency in engineering areas such as antenna design, radar imaging, optical/lightwave information processing and quantum devices such as superconductive devices. All these areas of applications contain the mathematical models with rotational point, wave functions and integral transforms such as Fourier, Laplace and Hilbert. This kind of models suggest the future realization of intrinsically non-von Neumann computers including pattern-information representing devices. Conventional quantum computation is strictly limited in its treatable problems. Contrarily, CVNN-based quantum computation can deal with more general problems, which leads to wider applications of quantum computer science. Therefore, one needs to imply optimization methods, based on quantum and tensor computations. On example techniques, used in shampoo [205], can perform the optimization on tensor level.

Quantum neural networks start to attract attention of many researches, who research and develop advanced methods of machine learning. With appearance of quantum computers it was necessary to expand the theory of neural networks on quantum devices, which let to analyze quantum processes, and tensor-network circuits. For problems of image recognition, time series prediction and moving object detection one needs to use quanvolutional neural networks [206], which engage quantum natural gradient. This optimization algorithms presents a quantum analogue of vanilla natural gradient descent, but it uses the Fubini-Study tensor instead of the Fisher information matrix. Unlike the usual Fisher and Hessian matrices, such tensor considers quantum computations with wave functions, bra- and ket-vectors. Such method, can be equipped with exponential moving and positive-negative averages, that can increase the test accuracy for quantum neural networks. There are proposition of equipping the quantum neural networks with spiking neurons in recent researches, what will causes other tasks for developing optimization methods with memory.

Recall that neural networks of third generation are based on weights with memory. Such approach is more close model to biological neuron networks, what makes the process of training more accurate. Potentially, for such models one can use corresponding memory-based optimization algorithms, such as mixed stochastic gradient descent, denoted MEGA I and MEGA II in [207]. The ability of remembering past experiences while being trained on a continuum of tasks makes possible to significantly rise the test accuracy of spiking neural networks. This kind of neural networks experienced the extension from multi-layer perceptrons to various advanced networks, such as Feedback SNN, attice map SNN, deep convolutional SNN, spike timing dependent plasticity and many other networks. But one of the most recent architectures are graph networks, which without spiking neurons can successfully train the models.

Graph neural networks, which structure reminds the simple graph, are utilized in many tasks related to medicine and biology. As can be seen in neuroscience resource, biological neural networks do not present the linear, sequential and organized in determined order model, but construct another more progressive connections, which significantly impacted in mathematical model of neurons. If it is possible to create the model with corresponding structure, then one receives the network without unwanted layers and computations. But such model rise the problem of error propagation, which definitely differs from classical error backpropagation. Afterwards, there were proposed alternative error rectification methods, which are called neighbor aggregation and information update in [208]. For this methods are suitable to apply presented above first order optimization methods, which are based on information geometry, on example mirror descent. Such algorithms can be modified with new proximal functions, and extended version of Bregman divergence in further researches. Neverless, there are exists other divergence formulas, which can produce new information-geometric optimization methods. But, for modifying the structure of neural networks, one may use wavelet decompositions, which potentially can process input data more accurately.

Neural networks, based on wavelet, are dispersed in the problems of signal processing. Moreover, such tool is already utilized in numerical solving partial differential equations, which seems as an important advance of physics-informed neural networks. Wavelet decomposition was used in convolution, LSTM and GRU layers, what made the data processing more accurate, especially in cases of scaling input information. Moreover, there is presented model of graph-wavelet neural networks, which gave the best results of node classification in [209], compared with spectral CNN, ChebyNet, GCN and MoNet. For wavelet neural networks one can use algorithms presented in talbe 4, but there are presented two new approaches, which are called whale and butterfly optimization methods, in [210]. Also for such kind of network researches often utilize particle swarm method, which is gradient free. Therefore, there is a possibility in comparing gradient free optimization methods between each other and developing new approaches.

In summary, one needs to say, that developing of optimization algorithms correlates with evolution of architectures of neural network and challenges, posed before researches. Also such relation works in backward, what can be seen in case of quantum natural gradient descent and quantum neural networks. Note that improving neural network can be infinite on the problems of image recognition, therefore one needs to consider the problem of moving detection. In case of time series predictions, it should to consider the stochastic process and Brownian motion, which comes from statistical physics and thermodynamics. Such kind of challenges have a big influence to the theory not just neural networks, but the theory of artificial intelligence.

7. Conclusions

In conclusion of this review, we can say, that fundamental developing of the theory of neural networks allows to simplify the work of human in more scales. The developing of optimization methods in machine learning allowed to achieve not only higher test accuracy for short time, but to imply necessary features and disadvantages of existing models, what through the time made researches to provide advanced architectures. First order optimization algorithms, presented in section 2, were improving and changing simultaneously with growing of neural networks in sizes and qualities. There were described every SGD-, Adam- and PNM-type algorithms, their approaches in optimization process, improving techniques and evolution. In section 3, we provided second order optimization algorithms, which are divided on Newton and quasi-Newton methods. There were described their applications in neural networks, simplifying their computation complexity by means of approximation theory, which produced other useful modifications and variations. Such approximation led to changing the Hessian by gradient flow tensor, and then by Fisher information matrix. This modification brought us to information geometry, that allowed to provide natural gradient and mirror descents, which equivalence was introduced in section 4. Afterwards, we summarized in Table 4 all type of neural networks, suitable for introduced optimization methods. There are provided modern networks, which already have used introduced optimization algorithms and those, that could have been used for increasing test accuracy. In section 6, we demonstrated ways of further developing and applications of optimization methods. There are provided gradient-free, fractional order and bilevel optimization algorithms, their existing versions and potentials for rising test accuracy for various type of neural networks. In the end, this survey can demonstrate to readers all type of existing optimization methods, their modifications, applications, which can help researches to comprehend the state of modern theory of optimization and machine learning. Such information allows to create advanced fundamental modifications and extend the area of applications for modern neural networks of all types.