Optimization Methods for Solving Deep Learning Problems: A Case Study of Adaptive Learning Rate Optimizers

1. Introduction

Optimization has been an important part of neural network (NN) research for an extended period [1], as its contribution to efficient model training and assurance of convergence to an optimal solution can not be overemphasized. It is noticeable that there are some traditional methods, such as Stochastic Gradient Descent (SDG) are popular in use, but adaptive learning rate methods (such as Adam, RMSProp, and AdaGrad) have gained recognition because of their ability to adjust their learning rate in the process of training, which helps in faster convergence, managing noisy gradient better, and lower the training error.

This project will concentrate on adaptive learning rate optimizers, evaluating their adaptability to various deep learning tasks as well as their strengths and limitations. The performance of adaptive optimizers will be contrasted with traditional methods such as SGD through a series of experiments, with an emphasis on their impact on the overall accuracy of the model, generalization, and training speed.

1.1. Problem Description

In this section, we describe an optimization problem for a supervised learning problem. Suppose that we have a data point $x_i\in {\mathbb{R}}^{d_x},y_i\in {\mathbb{R}}^{d_y},i=1,\dots ,n$, where n is the number of training samples, $x^i$ is the feature vector of the i th sample (an object, image, etc.), $y^i$ is the corresponding label. Our goal is to predict $y_i$ based on the information of predictors in $x_i$, which means that we are interested in learning the underlying mapping that maps each $x_i$ to $y_i$. To approximate the mapping, we use a neural network $f_{\theta }:{\mathbb{R}}^{d_x}\to {\mathbb{R}}^{d_y}$. The general form of the neural network for an L-layer network, the output $f_{\theta }\left(x\right)$ is defined as

$$\begin{array}{cc}\hfill f_{\theta }\left(x\right)& =\mathrm{Output}\mathrm{Layer}\left(Hidden{Layer}_L\left(\dots \left(Hidden{Layer}_1(InputLayer\left(X\right))\right)\right)\right)\hfill \\ \hfill & =\left\{\begin{array}{cc}{\beta }_0+\sum _{k=1}^{m_L}{\beta }_k{A}_k^{\left(L\right)}\hfill & \left(\mathrm{Regression}\right)\hfill \\ \mathrm{Softmax}\mathrm{or}\mathrm{Sigmoid}\left({\beta }_0\sum _{k=1}^{m_L}{\beta }_k{A}_k^{\left(L\right)}\right)\hfill & \left(\mathrm{Classification}\right)\hfill \end{array}\right.\hfill \\ \hfill & =\sum _{k=1}^{m_L}{\beta }_k{A}_k^{\left(L\right)}\left[\mathrm{for}\mathrm{simplicity}\right]\hfill \end{array}$$

(1)

where

$$A_k^{\left(l\right)}=g\left(w_{k0}^{\left(l\right)}+\sum _{j=1}^{m_{l-1}}{w}_{kj}^{\left(l\right)}{A}_j^{(l-1)}\right)$$

(2)

for hidden Layer $l=1,\dots ,L$, and $A^{\left(0\right)}=X$ for input Layer, $g\left(z\right)$ is a nonlinear activation function that is declared in advance. $\theta$ is a collection of all parameters $\left({\beta }^{\prime }\mathrm{s}\mathrm{and}{w}^{\prime }\mathrm{s}\right)$ need to be estimated from data (see Figure 1).

Now, to predict the ${\widehat{y}}_i=f_{\theta }\left(x_i\right)$ that is close to the true output $y_i$, we consider the minimization problem.

$$\underset{\theta }{min}F\left(\theta \right)\triangleq {\displaystyle \frac{1}{n}}\sum _{i=1}^n\ell \left(y_i,f_{\theta }\left(x_i\right)\right),$$

(3)

where $\ell (·,·)$ is a loss function. For regression problems, $\ell (y,z)$ is often chosen to be the quadratic loss function $\ell (y,z)={\parallel y-z\parallel }^2$. For binary classification problem, a popular choice of ℓ is $\ell (y,z)=$$log(1+exp(-yz\left)\right)$

2. Literature Review

In this section, we review some related papers on adaptive learning rate optimizers. We proceed by presenting some basic concepts of those adaptive learning rate methods, where their update rules are provided.

Ruo-Yu Sun discusses the generic optimization methods used in training neural networks, such as stochastic gradient descent and adaptive gradient methods in [1] in recognition of their capability to improve the training of NN. Adaptive learning rate methods such as Adam (Adaptive Moment Estimation), RMSProp (Root Mean Square Propagation), and Adagrad (Adaptive Gradient Methods) modify the learning rate based on the historical gradients, bringing more effective navigation of the optimization settings. He further notes that there is still a gap in practical performance and theoretical understanding of these methods emphasizing that “bringing theory closer to practice is still a huge challenge for both theoretical and empirical researchers”. In a nutshell, adaptive learning rate methods are essential methods to mitigate issues related to gradient explosion/vanishing and improve convergence rate.

Moreover, Diederik P. Kingma and Lei Ba Jimmy [2] introduce the Adam (Adaptive Moment Estimation) method as an adaptive learning rate algorithm for first-order gradient-based optimization of stochastic objective functions, based on adaptive estimates of lower-order moments that effectively combine the strengths of AdaGrad and RMSProp, which make it well-suited for sparse gradients and non-stationary objectives. Their empirical results demonstrate that Adam outperforms traditional optimization methods like SGD and other adaptive methods in consideration of convergence rate and robustness in different machine learning tasks (see Figure 2). Its advantages include low memory requirements and intuitive interpretation of hyperparameters that require little tuning. However, it is noted that it can exhibit instability if the learning rate is not properly adjusted.

As noted earlier, optimization is crucial for deep learning training. The traditional optimizer has been SGD coupled momentum because of its simplicity, low computational cost, and good generalizability. But it needs fine-tuning of a constant learning rate, which can cause slow convergence or suboptimal solutions in some cases. Adaptive learning rate optimizers like Adam, RMSProp, and AdaGrad modify the learning rate accordingly based on the gradient history, which often leads to faster convergence, especially when you have a large dataset or in tasks with sparse gradients. We first discuss the gradient descent method (GD) briefly for smooth transitioning of understanding since some of these adaptive learning rate optimizers are based on it

Gradient Descent (GD): A GD method is an iterative process, that starts from an arbitrary point on the loss function and moves down its slope in steps until it reaches the minimum point of the loss function. Its update rule is given as

$${\theta }_{t+1}={\theta }_t-{\alpha }_t\nabla F\left({\theta }_t\right),$$

(4)

where ${\alpha }_t$ is the learning rate and $\nabla F\left({\theta }_t\right)$ is the corresponding gradient of $F\left({\theta }_t\right)$. This method converges at a linear rate; its solution is globally optimal for the convex objective function, while its downside is high computational cost as it uses the whole data for gradient computation.

Stochastic Gradient Descent (SGD): SGD was introduced to address the issue of the high computational cost of GD for large-scale data, as it (GD) passes the whole data at once in each iteration, but instead, SGD randomly samples one data point $x_i$ or a mini-batch from the training dataset for each update. Let us rewrite equ( 3) as

$$\underset{\theta }{min}F\left(\theta \right)\triangleq {\displaystyle \frac{1}{B}}\sum _{i=1}^B{F}_i\left(\theta \right),$$

(5)

where $F_i\left(\theta \right)$ represents the sum of training loss for a mini-batch of training samples and B is the total number of mini-batches. Its update’s rule is given as

$${\theta }_{t+1}={\theta }_t-{\alpha }_t\nabla F_i\left({\theta }_t\right).$$

(6)

This method converges at a sublinear rate, which saves computation costs, while its downside is that the solution may be stuck at the saddle point in some cases.

SGD with Momentum: Shiliang Sun et al [3] note that the concept of momentum is derived from the mechanics of physics, which simulates the inertia of objects, and the idea of applying momentum in SGD is to preserve the influence of the previous update direction on the next iteration to a certain degree. When SGD randomly picks i at t-th iteration, its update rule is given as

$$\left\{\begin{array}{cc}\hfill g_t& =\nabla F_i\left({\theta }_t\right)\hfill \\ \hfill m_t& =\beta m_{t-1}+(1-\beta )g_t,\hfill \\ \hfill {\theta }_{t+1}& ={\theta }_t-{\alpha }_t{m}_t.\hfill \end{array}\right.$$

(7)

Diederik P. Kingma and Lei Ba Jimmy [2] call this method the stochastic versions of the heavy-ball method and accelerated gradient method, which are also known as “momentum methods" in deep learning.

Adaptive Gradient Methods (AdaGrad): AdaGrad is an improvement of SGD [4], it modifies the learning rate dynamically using historical gradients from previous steps. Its update rule is given as

$$\left\{\begin{array}{cc}\hfill g_t& =\nabla F_i\left({\theta }_t\right)\hfill \\ \hfill v_t& =\sum _{j=1}^{\mathrm{T}}{g}_j\circ g_j,\hfill \\ \hfill {\theta }_{t+1}& ={\theta }_t-{\alpha }_t{v}_t^{-1/2}\circ g_t,t=0,1,2,\dots ,\hfill \end{array}\right.$$

(8)

where ∘ represents the element-wise product. The method is suitable for dealing with sparse gradient problems but is not suitable for dealing with non-convex problems [3].

Root Mean Square Propagation (RMSProp): RMSProp was introduced in order to correct the drawback in AdaGrad and Rprop (Resilient Propagation) []. This leads to a new definition of $v_t$. So, at the t-th iteration of RMSProp i is randomly selected and the update rule is given as

$$\left\{\begin{array}{cc}\hfill g_t& =\nabla F_i\left({\theta }_t\right)\hfill \\ \hfill v_t& =\beta v_{t-1}+(1-\beta )g_t\circ g_t,\hfill \\ \hfill {\theta }_{t+1}& ={\theta }_t-{\alpha }_t{v}_t^{-1/2}\circ g_t,t=0,1,2,\dots ,\hfill \end{array}\right.$$

(9)

This method improves the drawback in the late stage of AdaGarad and is suitable for non-stationary and non-convex optimization problems. But the downside of it is that the update process may be repeated around the local minimum in the late stage.

Adaptive Moment Estimation (Adam): Adam [2] combines RMSProp and momentum methods, It makes use of both first- and second-order momentum estimation of the gradient to modify the step size dynamically of each parameter. Its update rule is given as

$$\left\{\begin{array}{cc}\hfill g_t& =\nabla F_i\left({\theta }_t\right),\hfill \\ \hfill m_t& ={\beta }_1{m}_{t-1}+\left(1-{\beta }_1\right)g_t,\hfill \\ \hfill v_t& ={\beta }_2{v}_{t-1}+\left(1-{\beta }_2\right)g_t\circ g_t,\hfill \\ \hfill {\theta }_{t+1}& ={\theta }_t-{\alpha }_t{v}_t^{-1/2}\circ m_t,t=0,1,2,\dots ,\hfill \end{array}\right.$$

(10)

where ${\beta }_1$ and ${\beta }_2$ are exponential decay rates. The Adam method is relatively stable in gradient processes and suitable for most non-convex problems with high-dimensional datasets.

3. Methodology/Algorithm Description

In this section, we first describe our methodology, then we proceed to describe some of the algorithms of adaptive learning rate method discussed in Section 2

In our study, we employ an experimental methodology to evaluate the performance of various adaptive optimizers (Adam, RMSProp, and AdaGrad) in comparison to traditional optimizers (SDG) and a non-tuning parameter machine learning method (LDA: linear discriminant analysis) on deep learning tasks. The analysis will concentrate on the efficiency of resources, generalization, and the speed of convergence. We evaluate the performance of optimizers by utilizing handwritten zip-code (see Figure 3), which have training data of $7291\times 256$ dimensions and test data of $2007\times 256$ dimensions. Our tools include R programming (version 2023.09.1 Build494), Keras(Version: 2.13.0), Tensorflow, and convolutional neural networks (CNN) as frameworks for model implementation and experimentation using HP Pavilion x360 Convertible (processor: 8×intel@ corei5-8250U CPU @1.60GHZ, memory: 8.00 GIG of RAM) on Windows 11.

The following are the description of the algorithms

Algorithm 1 Stochastic Gradient Descent (SGD)

Require:  Objective function $F\left(\theta \right)$, initial parameters ${\theta }_0$, learning rate ${\alpha }_t$, number of iterations T 1: Initialize ${\theta }_0$ 2: for$t=1$ to T (And $\theta$ not converge) do 3:     Randomly select $i\in \{1,2,\dots ,N\}$ 4:     Compute gradient $g_t=\nabla F_i\left({\theta }_t\right)$ 5:     Update parameters ${\theta }_{t+1}={\theta }_t-{\alpha }_t·g_t$ 6: end for 7: return${\theta }_t$

Algorithm 2 AdaGrad General Algorithm

Require:  Step size $\eta$, stochastic objective function $f\left(\theta \right)$, initial parameter vector ${\theta }_1$ 1: for$t=1$ to T (And $\theta$ not converge) do 2:     Evaluate $f_t\left({\theta }_t\right)$ 3:     Get and save $g_t$ 4:     $v_t\leftarrow {\sum }_{\tau =1}^t{g}_{\tau }{g}_{\tau }^{\top }$ 5:     ${\theta }_{t+1}\leftarrow {\theta }_t-\eta v_t^{-1/2}{g}_t$ 6: end for 7: return${\theta }_t$

Algorithm 3 RMSProp Algorithm

Require:  Step size $\eta$, decay rate $\beta$, stochastic objective function $f\left(\theta \right)$, initial parameter vector ${\theta }_1$, small constant $ϵ>0$ 1: Initialize $v_0=0$ 2: for$t=1$ to T (And $\theta$ not converge) do 3:     Evaluate $f_t\left({\theta }_t\right)$ 4:     Compute gradient $g_t=\nabla f_t\left({\theta }_t\right)$ 5:     Update the moving average of squared gradients: $$v_t\leftarrow \beta v_{t-1}+(1-\beta )g_t\circ g_t$$ 6:     Update parameters: $${\theta }_{t+1}\leftarrow {\theta }_t-{\displaystyle \frac{\eta }{\sqrt{v_t}+ϵ}}\circ g_t$$ 7: end for 8: return${\theta }_t$

Algorithm 4 Adam: Stochastic Optimization Algorithm

Require:  Step size $\alpha$, exponential decay rates ${\beta }_1,{\beta }_2\in [0,1)$, stochastic objective function $f\left(\theta \right)$ with parameters $\theta$, initial parameter vector ${\theta }_0$ 1: Initialize $m_0=0$ (First moment vector) 2: Initialize $v_0=0$ (Second moment vector) 3: Initialize $t=0$ (Timestep) 4: while${\theta }_t$ not converged do 5:     $t\leftarrow t+1$ 6:     $g_t\leftarrow {\nabla }_{\theta }{f}_t\left({\theta }_{t-1}\right)$▹ Compute gradients at timestep t 7:     $m_t\leftarrow {\beta }_1·m_{t-1}+(1-{\beta }_1)·g_t$▹ Update biased first moment estimate 8:     $v_t\leftarrow {\beta }_2·v_{t-1}+(1-{\beta }_2)·g_t^2$▹ Update biased second raw moment estimate 9:     ${\widehat{m}}_t\leftarrow m_t/(1-{\beta }_1^t)$▹ Bias-corrected first moment estimate 10:     ${\widehat{v}}_t\leftarrow v_t/(1-{\beta }_2^t)$▹ Bias-corrected second raw moment estimate 11:     ${\theta }_t\leftarrow {\theta }_{t-1}-\alpha ·{\widehat{m}}_t/(\sqrt{{\widehat{v}}_t}+ϵ)$▹ Update parameters 12: end while 13: return${\theta }_t$▹ Resulting parameters

4. Numerical Experiment

In this section, we present and analyze our numerical experiment results following the methodology described in Section 3. Generally, we set our number of epochs to 30 in training CNN in R Programming Language using the Keras library. We will not give much detail about our CNN setup since our focus is on the optimization methods. We tune the important hyperparameters of each discussed optimizer, respectively. Each digit of the handwritten zip code is $16\times 16$ grayscale image. This data was collected by the neural network group at AT&T research labs and they noted that the test set is notoriously "difficult", that any trained model with $2.5\%$ an error rate is excellent [5].

4.1. Experiment on Hyperparameter Tuning

In this subsection, we present results on tuning the combination of some relevant hyperparameters for each discussed optimizer.

Figure 4 shows the test and training accuracy of AdaGrad optimizer by tuning its hyperparameters (learning rate $\alpha$ and batch size) using CNN to classify handwritten zip-code. The test (green) and train (red) accuracy are plotted as a function of learning rate ($\alpha$) for each batch size. It is noticeable in each batch size plot there is a little discrepancy between train and test accuracy, indicating that there is no overfitting going on, that is, the model performs well on unseen data, and our best-tuned hyperparameters for learning rate and batch size are $\alpha =0.05$ and 64 respectively with test accuracy of $96.76\%$.

Figure 5, Figure 6, and Figure 7 account for the test and train accuracy of RMSProp, Adam, and SGD optimizers, respectively; see Table 1 for their respective best-tuned hyperparameters and accuracy. All the optimizers perform well in terms of generalizability and accuracy, but the Adam method performs best.

4.2. Experiment on Best-Tunned and Default Hyperparameters

Continuing our study, we train our model using our best-tuned and default hyperparameters to compare their accuracy and performance. We plot the test (green) and training (red) accuracy as a function of epochs for each method, and we can see that all the models are robust as their test and train accuracy are closed together for each epoch (see Figure 8 and Figure 9). Table 1 shows the best-tuned (black text) and default (green text) hyperparameters, train and test accuracy, test error, test loss function, and running time (seconds). Considering the test accuracy, all the methods perform better with best-tuned hyperparameters than with default hyperparameters except for the RMSProp which is slightly lower (see Table 1). Generally, the Adam methods give the best performance out of all the considered methods, its test error is $2.99\%$ which is closer to the mentioned excellent error of $2.5\%$ earlier. The last row in the table shows the result of our considered non-tuning machine learning model (LDA). It is very fast in terms of speed but the accuracy is lower than deep learning models.

5. Concluding Remarks

In our study, we are able to describe each considered adaptive learning rate optimizer, stating their weakness, strengths, and suitability cases. We proceed to explore their hyperparameter effects (such as learning rate, decay rate, etc.). We conclude by analyzing their performance on accuracy, error, convergence speed, and generalization. Our empirical results show that Adam has the best performance in terms of test accuracy, speed, and generalization, which is suitable for optimizing non-stationary and non-convex problems.

There are a number of promising avenues for further investigation into adaptive learning rate optimizers in deep learning that have not been explored in this work. Firstly, consider the theoretical guarantee of convergence of those methods instead of just the empirical approach alone. Secondly, consider a more dense grid for hyperparameter tuning. Lastly, consider tuning-free methods and compare them with those adaptive learning rate methods.