Evolutionary Reinforcement Learning of Neural Network Controller for Acrobot Task – Part1: Evolution Strategy

1. INTRODUCTION

Neural networks can be effectively trained using gradient-based methods for supervised learning tasks, where labeled training data are readily available. In such cases, the errors between the neural network outputs and their corresponding target values can be observed, and these errors are utilized to perform backpropagation through the network. As a result, the node connection weights and biases are appropriately adjusted. However, when it comes to reinforcement learning tasks, where labeled training data are not provided, neural networks demand the utilization of gradient-free training algorithms. In these scenarios, traditional gradient-based methods are not applicable due to the absence of labeled data. Instead, alternative approaches that do not rely on gradients must be employed to train the neural networks effectively. Evolutionary algorithms [1,2,3,4,5] find applicability in the reinforcement learning of neural networks due to their independence from gradient-based methods. Another representative reinforcement learning technique is Q-learning [6,7,8]. Q-learning requires obtaining the reward, denoted as $\mathrm{r}\left(\mathrm{t}\right)$, associated with action $\mathrm{a}\left(\mathrm{t}\right)$ taken at state $\mathrm{s}\left(\mathrm{t}\right)$ to determine the subsequent action, $\mathrm{a}(\mathrm{t}+1)$, at the given time step $\mathrm{t}$. In contrast, evolutionary algorithms do not require $\mathrm{r}\left(\mathrm{t}\right)$ at every step; instead, they evaluate the rewards after an episode is completed. Consequently, evolutionary algorithms alleviate the need for designing specific rewards for each state-action pair, providing a significant advantage in certain scenarios.

Evolution Strategy [9,10], Genetic Algorithm [11,12,13,14], and Differential Evolution [15,16,17] stand as representative evolutionary algorithms. To achieve successful training of neural networks using evolutionary algorithms, careful considerations must be made regarding: i) selecting appropriate algorithms due to the availability of various algorithmic variations, and ii) designing hyperparameters as they significantly impact performance. The author previously reported experimental evaluations on Evolution Strategy for reinforcement learning of neural networks, utilizing the pendulum control task [18]. In this study, the Acrobot control task is adopted as another task.

2. ACROBOT CONTROL TASK

As a task that requires reinforcement learning to solve, this work employs Acrobot control task provided at OpenAI Gym. Figure 1 shows a screenshot of the system. The webpage for this system describes as follows1; The system consists of two links connected linearly to form a chain, with one end of the chain fixed. The joint between the two links is actuated. The goal is to apply torques on the actuated joint to swing the free end of the linear chain above a given height while starting from the initial state of hanging downwards.

Let p_y denote the height of the free end of the linear chain, where the minimum (maximum) value of p_y is 0.0 (1.0) as shown in Figure 2. The goal of the task is originally to achieve p_y$\ge$0.5, and an episode is finished when the goal is achieved or the time step reaches to a preset limit. In this work, the goal is changed so that the free end of the linear chain is kept as high as possible (i.e., let the value of p_y as greater as possible) throughout an episode, where an episode consists of 200 time steps. Besides, the author changed the system so that (i) the control task starts with the state shown in Figure 2a where p_y=0.0, and (ii) the applicable torque to the actuated joint is continuous within [-1.0, 1.0] while the torque is originally discrete (either of -1, 0 or 1).

In each step, the controller observes the current state and then determines the action. An observation obtains cos(θ₁), sin(θ₁), cos(θ₂), sin(θ₂), and the angular velocity of θ₁ and θ₂, where θ₁ is the angle of the first joint and θ₂ is relative to the angle of the first link¹. The ranges are -1.0$\le$cos(θ₁), sin(θ₁), cos(θ₂), sin(θ₂)$\le$1.0, $-4\pi \le$ angular velocity of θ₁$\le 4\pi$, and $-9\pi \le$ angular velocity of θ₂$\le 9\pi$ respectively.

In this work, the author defines the fitness of a neural network controller as shown in eq. (1).

$$\mathrm{F}\mathrm{i}\mathrm{t}\mathrm{n}\mathrm{e}\mathrm{s}\mathrm{s}=\frac{1}{200}{\sum }_{\mathrm{t}=1}^{200}{\mathrm{p}}_{\mathrm{y}}\left(\mathrm{t}\right),$$

(1)

In eq. (1), p_y(t) denotes the height p_y at each time step t. The fitness score is larger as p_y(t) is larger for more time steps. Thus, a controller fits better as it can keep the free end of the linear chain as higher as possible.

3. NEURAL NETWORKS

In this study, the author employs a three-layered feedforward neural network known as a multilayer perceptron (MLP [19,20]) as the controller. The topology of the MLP is illustrated in Figure 3, while eqs. (2)-(6) present the feedforward calculations.

Input layer:

$${\mathrm{o}\mathrm{u}\mathrm{t}}_{\mathrm{i}}^{\left(1\right)}={\mathrm{x}}_{\mathrm{i}},\mathrm{}\mathrm{i}=\mathrm{1,2},\dots ,\mathrm{N}$$

(2)

Hidden layer:

$${\mathrm{i}\mathrm{n}}_{\mathrm{j}}^{\left(2\right)}={\mathsf{\theta }}_{\mathrm{j}}^{\left(2\right)}+{\sum }_{\mathrm{i}}{\mathrm{w}}_{\mathrm{i},\mathrm{j}}^{\left(2\right)}{\mathrm{o}\mathrm{u}\mathrm{t}}_{\mathrm{i}}^{\left(1\right)},\mathrm{}\mathrm{j}=\mathrm{1,2},\dots ,\mathrm{M}$$

(3)

$${\mathrm{o}\mathrm{u}\mathrm{t}}_{\mathrm{j}}^{\left(2\right)}=\mathrm{h}\left({\mathrm{i}\mathrm{n}}_{\mathrm{j}}^{\left(2\right)}\right)\mathrm{},\mathrm{}\mathrm{j}=\mathrm{1,2},\dots ,\mathrm{M}$$

(4)

Output layer:

$${\mathrm{i}\mathrm{n}}_{\mathrm{k}}^{\left(3\right)}={\mathsf{\theta }}_{\mathrm{k}}^{\left(3\right)}+{\sum }_{\mathrm{j}}{\mathrm{w}}_{\mathrm{j},\mathrm{k}}^{\left(3\right)}{\mathrm{o}\mathrm{u}\mathrm{t}}_{\mathrm{j}}^{\left(2\right)},\mathrm{}\mathrm{k}=\mathrm{1,2},\dots ,\mathrm{L}$$

(5)

$${\mathrm{o}\mathrm{u}\mathrm{t}}_{\mathrm{k}}^{\left(3\right)}=\mathrm{h}\left({\mathrm{i}\mathrm{n}}_{\mathrm{k}}^{\left(3\right)}\right),\mathrm{}\mathrm{k}=\mathrm{1,2},\dots ,\mathrm{L}$$

(6)

The activation function denoted as h() is the hyperbolic tangent function whose shape is illustrated in Figure 4. This activation function is widely used in neural networks due to its ability to produce a smooth non-linear output that ranges from -1.0 to 1.0.

$$\mathrm{h}\left(\mathrm{x}\right)=\frac{{\mathrm{e}}^{\mathrm{x}}-{\mathrm{e}}^{-\mathrm{x}}}{{\mathrm{e}}^{\mathrm{x}}+{\mathrm{e}}^{-\mathrm{x}}}$$

(7)

$$-1.0<\mathrm{h}\left(\mathrm{x}\right)<1.0$$

(8)

In this study, the MLP serves as the policy function: action(t) = F(observation(t)). The input layer consists of six units, each corresponding to the values obtained by an observation. To ensure the input value falls within the range [-1.0, 1.0], the angular velocity of θ₁ (θ₂) is divided by $4\pi$ ($9\pi$). The output layer comprises one unit, and its output value is directly applied as the torque to the joint.

4. TRAINING OF NEURAL NETWORKS BY EVOLUTION STRATEGY

A three-layered perceptron, as depicted in Figure 3, includes M+L unit biases and NM+ML connection weights, resulting in a total of M+L+NM+ML parameters. Let D represent the quantity M+L+NM+ML. For this study, the author sets N=6 and L=1, leading to D=8M+1. The training of this perceptron is essentially an optimization of the D-dimensional real vector. Let $\mathbf{x}=({\mathrm{x}}_1,{\mathrm{x}}_2,...,{\mathrm{x}}_{\mathrm{D}})$ denote the D-dimensional vector, where each ${\mathrm{x}}_{\mathrm{i}}$ corresponds to one of the D parameters in the perceptron. By applying the value of each element in $\mathbf{x}$ to its corresponding connection weight or unit bias, the feedforward calculation in eqs. (2)-(6) can be processed.

The process of training neural networks using evolutionary algorithms is known as neuroevolution [21,22]. Neuroevolution has found applications in various domains, including games [23,24,25,26]. For instance, Togelius et al. [26] utilized neuroevolution for simulated car racing. In this study, the D-dimensional vector $\mathbf{x}$ is optimized using Evolution Strategy [9,10]. ES treats $\mathbf{x}$ as a chromosome (a genotype vector) and applies evolutionary operators to manipulate it. The fitness of $\mathbf{x}$ is evaluated based on eq. (1).

Figure 5 illustrates the ES process. Step 1 initializes vectors ${\mathbf{y}}^1$, ${\mathbf{y}}^2$, …, ${\mathbf{y}}^{\mathsf{\lambda }}$ randomly within a predefined range, denoted as ${\left[min,max\right]}^{\mathrm{D}}$, where $\lambda$ represents the number of offsprings. A larger value of $\lambda$ encourages more explorative search. In Step 2, the values in each vector ${\mathbf{y}}^{\mathrm{c}}$ (c=1, 2, ..., $\lambda$) are fed into the MLP, which subsequently controls the Acrobot system for a single episode consisting of 200 time steps. The fitness of ${\mathbf{y}}^{\mathrm{c}}$ is evaluated based on the outcome of the episode. In Step 3, the evolutionary training loop concludes upon meeting a preset condition. A straightforward example of such a condition is reaching the limit number of fitness evaluations. Proceeding to Step 4, from the $\mathsf{\mu }$ vectors in the current parent population ${\mathbf{z}}^1$, ${\mathbf{z}}^2$, …, ${\mathbf{z}}^{\mathsf{\mu }}$ and the $\mathsf{\lambda }$ vectors in the current offspring population ${\mathbf{y}}^1$, ${\mathbf{y}}^2$, …, ${\mathbf{y}}^{\mathsf{\lambda }}$, only the vectors with the top $\mathsf{\mu }$ fitness scores are selected to survive as parents in the next reproduction step, while the remaining vectors are discarded. Here, $\mathsf{\mu }$ represents the number of parents. A smaller value of $\mathsf{\mu }$ encourages more exploitative search. Note that the parent population is empty during the first occurrence of Step 4. Consequently, among the $\mathsf{\lambda }$ vectors in the current offspring population ${\mathbf{y}}^1$, ${\mathbf{y}}^2$, …, ${\mathbf{y}}^{\mathsf{\lambda }}$, only the vectors with the top $\mathsf{\mu }$ fitness scores survive as parents. In Step 5, new $\mathsf{\lambda }$ offspring vectors are generated by applying the reproduction operator to the parent vectors ${\mathbf{z}}^1$, ${\mathbf{z}}^2$, …, ${\mathbf{z}}^{\mathsf{\mu }}$ selected in the previous Step 4. These newly produced offspring vectors form the updated offspring population ${\mathbf{y}}^1$, ${\mathbf{y}}^2$, …, ${\mathbf{y}}^{\mathsf{\lambda }}$. The process of reproduction is described in Figure 6.

$${\mathrm{y}}_{\mathrm{d}}^{\mathrm{c}}\leftarrow {\mathrm{y}}_{\mathrm{d}}^{\mathrm{c}}+\mathrm{s}\ast \mathrm{r}\mathrm{a}\mathrm{n}\mathrm{d}$$

(9)

$$\mathrm{i}\mathrm{f} {\mathrm{y}}_{\mathrm{d}}^{\mathrm{c}}<min \mathrm{t}\mathrm{h}\mathrm{e}\mathrm{n} {\mathrm{y}}_{\mathrm{d}}^{\mathrm{c}}\leftarrow min$$

(10)

$$\mathrm{i}\mathrm{f} \mathit{max}<{\mathrm{y}}_{\mathrm{d}}^{\mathrm{c}}\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{n} {\mathrm{y}}_{\mathrm{d}}^{\mathrm{c}}\leftarrow max$$

(11)

5. EXPERIMENT

The number of hidden units significantly impacts the MLP’s ability to model nonlinear functions. Utilizing evolutionary algorithms for optimizing a smaller MLP is advantageous, as it involves a reduced number of variables in the genotype vector $\mathbf{x}$ so that the search space is smaller. However, reducing the number of hidden units may hinder the MLP’s effectiveness in controlling the Acrobot system. On the other hand, a larger MLP will demonstrate better control performance, but optimizing it becomes more challenging due to the larger search space. Additionally, implementing a larger MLP demands more computational resources. Hence, finding a balance between these trade-offs is crucial to determine the optimal number of hidden units for the specific task. In this study, the author investigates four different configurations of hidden units: 4, 8, 16, and 32, to explore their impact on the system’s performance.

The ES hyperparameter values are determined through empirical analyses, as presented in Table 1. The number of generations is configured as either 500 or 100, corresponding to the population sizes (the number of offsprings $\lambda$) of 10 and 50, respectively. Consequently, the total number of fitness evaluations remains constant at 5,000, which is equivalent to the product of the number of generation and the population size.

Selecting an appropriate search domain is vital since the values in the genotype vector $\mathbf{x}$ serve as connection weights or unit biases in the neural network. The range should be chosen judiciously, neither excessively large nor small. For this experiment, the author sets $min$ and $max$ in eqs. (10) and (11) as -10.0 and 10.0 respectively.

An MLP with 4, 8, 16, or 32 hidden units underwent independent training 11 times. Table 2 presents the best, worst, average, and median fitness scores of the trained MLPs across the 11 trials. Each of the two hyperparameter configurations (a) and (b) in Table 1 was applied.

Upon comparing the scores in Table 2 between configurations (a) and (b), it is evident that the values obtained using configuration (b) are higher than those obtained using configuration (a). This result indicates that configuration (b) outperforms configuration (a). The Wilcoxon signed rank test confirmed that this difference is statistically significant (p<.01). Hence, in this study, increasing the population size, rather than the number of generations, allowed ES to discover superior solutions. In ES, augmenting the population size promotes global exploration during the initial stages, while increasing the number of generations enhances local exploitation during the later stages. Based on the findings of this experiment, it is apparent that early-stage global exploration is more critical for this learning task. This aligns with the author’s prior study [18], which employed a pendulum task instead of the Acrobot task.

Next, upon comparing the fitness scores obtained using configuration (b) among the four variations of M (the number of hidden units), it is observed that the scores with M=8 are better than those of M=4, 16 or M=32. The Wilcoxon rank sum test confirmed that the difference between M=8 and either of M=4, M=16 or M=32 is statistically significant (p<.01, p<.05 and p<.05 respectively). This observation suggests two conclusions: (i) 4 hidden units are not sufficient for this task, and (ii) 16 or more hidden units are excessive, resulting in an excessively large search space for ES with configuration (b).

Figure 7 presents learning curves of the best, median, and worst runs among the 11 trials where the configuration is (b). Note that the horizontal axis of these graphs is in a logarithmic scale. The random solution for the first evaluation have fitness values almost exclusively at 0.0. The graphs in Figure 7 show increases in fitness from 0.1 to 0.2 by approximately the first 100 evaluations, which corresponds to 2% of the total 5,000 evaluations. Afterward, over the remaining 4,900 evaluations, the fitness gradually rises to around 0.4 to 0.45. For all the four M variations, even in the worst trials the final fitness scores are not significantly worse than the corresponding best trials. This indicates that ES could robustly optimize the MLP so that the variance was small within the 11 trials.

Figure 8a illustrates the actions by the MLP and the heights p_y(t) (see eq. (1)) in the 200 steps prior to training, while Figure 8b displays the corresponding actions and heights after training. In this scenario, the MLP employed 8 hidden units, and the configuration (b) was utilized. Figure 8a reveals that the MLP before training consistently provides a torque of -1.0 regardless of the system’s state, resulting in the chain end hardly rising from the bottom. On the other hand, as seen in Figure 8b, the MLP after training outputs alternating torques of -1.0 and 1.0, occasionally providing intermediate torque values as well. From the graph of p_y(t), it is evident that the chain gradually swings, causing the height of the end to rise, reaching 0.9850 at 100 steps. The best score of p_y(t) was 0.9985 at 183 steps, which is 99.85% of the maximum height. However, the MLP does not maintain the chain end in the upright position, and the value of p_y(t) oscillates between around 1.0 and 0.1. In this system, the torque is applied to the joint of the chain. Therefore, it is relatively easy to raise the height of the joint, but maintaining the chain’s free end at the upright position is difficult, as observed from the results. Supplementary videos are provided which demonstrate the motions of the chain2,3.

6. Conclusion

The author conducted an empirical application of Evolution Strategy to the reinforcement learning of a neural network controller for the Acrobot task. The experimental results demonstrated that ES successfully trained the multilayer perceptron to achieve a remarkable height of 99.85% concerning the maximum height. However, the trained MLP failed to maintain the chain end in an upright position throughout an episode.

In this study, it was observed that employing 8 hidden units in the neural network yielded better results with statistical significance compared to using 4, 16, or 32 hidden units. This suggests that selecting an appropriate number of hidden units is crucial, as an excessive number may not contribute to improved performance. Furthermore, the findings indicate that a larger population size in ES led to a more extensive exploration of potential solutions over a greater number of generations. This result aligns with the previous study by the author. However, to establish the generality of this observation, further investigations are necessary to verify if this holds true for evolutionary algorithms other than ES. Besides, the author intends to conduct additional evaluations to enhance evolutionary algorithms by applying them to diverse reinforcement learning tasks beyond the Acrobot task.