Discovering a Single Neural Network Controller for Multiple Tasks with Evolutionary Algorithms

1. Introduction

Multi-Objective Optimization (MOO) [1] is a fascinating research field in which the ultimate goal consists of discovering solutions fulfilling multiple, often conflicting, objectives simultaneously [1]. Since the task is extremely complex or even impossible, compromise solutions are taken into account, which are referred to as Pareto-optimal [2]. By definition, a Pareto-optimal solution is “a set of ’non-inferior’ solutions in the objective space defining a boundary beyond which none of the objectives can be improved without sacrificing at least one of the other objectives” [3]. An illustration of the concept of Pareto optimality is provided in Figure 1.

Several techniques have been proposed to solve MOO problems, such as Evolutionary Algorithms (EAs) [4,5,6], Genetic Algorithms (GAs) [7,8,9] or swarm intelligence approaches [10,11,12]. For example, the authors in [13] applied GAs to discover Pareto-optimal solutions for a Supply Chain Network (SCN) design problem. Their analysis indicates the feasibility of the approach in the context of a Turkish company producing plastic products. Applications of GAs to MOO of test functions can be found in [14,15,16]. In [17,18], the OpenAI Evolutionary Strategy (OpenAI-ES) [19] has been used to cope with a MOO problem requiring a swarm of AntBullet robots [20] to simultaneously locomote and aggregate: the authors show that robots develop a good locomotion capability, while aggregation is obtained very rarely. This underscores the difficulty of optimizing both objectives at the same time.

Specific algorithms have been developed to address MOO with regard to test function optimization, such as Multi Objective Genetic Algorithm (MOGA) [21], Niched Pareto Genetic Algorithm (NPGA) [22], Nondominated Sorting Genetic Algorithm II (NSGA-II) [23] and Vector Evaluated Genetic Algorithm (VEGA) [24]. In particular, NSGA-II [23] enables to discover optimized solutions for test functions defined in [24,25] by exploiting non-dominated sorting, diversity-preservation and elitism combined with typical mutation and crossover operators from GAs. VEGA [24] represents a variant of a classic GA tailored for MOO scenarios. Specifically, the solution evaluation produces a vector of fitness values, one for each objective, and selection is made by choosing non-dominated solutions according to Pareto optimality. To this end, the whole population is split into sub-populations (one for each objective) from which the best individuals are selected. Finally, recombination and mutation of selected solutions allow to generate the new population and ensure diversity.

An interesting field related to MOO involves the discovery of effective controllers to manage different problems belonging to the same class. For example, the authors in [26] propose an approach called Shared Modular Policy (SMP) in which a global policy is used to control different modular neural networks. In particular, each module is responsible for the control of a single actuator based on current sensory readings and shared information in the form of current reward function. Experiments performed on robot locomotors characterized by diverse morphologies (e.g., Halfcheetah, Hopper, Humanoid, Walker2D) demonstrate the validity of the approach and a good generalization capability on new morphologies. Another worthwhile example can be found in [27], in which the authors propose an approach to discover a single controller for a class of dynamical systems. Specifically, through in-context learning [28], the authors demonstrate that an effective controller for a specific numerical problem can be immediately applied to a different problem within the same class, with no need for fine-tuning.

This work is halfway between MOO and the evolution of a single controller for multiple problems. In more detail, we present a comparative study of four EAs — Generational Genetic Algorithm (GGA) [29], OpenAI-ES [19], Stochastic Steady State [30] and Stochastic Steady State with Hill-Climbing (SSSHC) [31] — that must discover a single neural network controller capable to minimize simultaneously four benchmark problems: (i) 4-bit parity, (ii) double-pole balancing, (iii) grid navigation and (iv) test function optimization. Although using methods specifically tailored for MOO scenarios like NSGA-II or VEGA could be rather intuitive, we decided instead to put emphasis on the usage of general and relatively simple EAs [32] to assess how effective they are at sampling the search space and discovering optimized solutions for the MOO scenario. Moreover, the considered algorithms proved successful with respect to the benchmark problems addressed in isolation [30,31,33,34,35,36,37]. Our results clearly demonstrate that OpenAI-ES and SSSHC outperform GGA and SSS thanks to their propensity to reduce the size of weights, which is pivotal in the considered scenario. Furthermore, detailed analysis of the performance on the individual benchmark problems reveal that all the algorithms focus mainly on the minimization of test functions, which allows to quickly optimize performance. Interestingly, SSSHC achieves better performance than OpenAI-ES with respect to grid navigation and test function optimization, while the opposite is true for 4-bit parity and double-pole balancing. This implies that relatively simple strategies can perform similarly to more sophisticated algorithms.

The key contributions of this research are listed as follows:

• a new MOO scenario is introduced, which entails benchmark problems like 4-bit parity, double-pole balancing, grid navigation and test function optimization;
• a comparison of some state-of-the-art EAs on the defined MOO scenario is proposed;
• OpenAI-ES and SSSHC emerge as more effective than GGA and SSS at dealing with the novel MOO scenario;
• SSSHC outperforms OpenAI-ES with respect to grid navigation and test function optimization, while the opposite is true for 4-bit parity and double-pole balancing;
• OpenAI-ES has an intrinsic propensity to reduce the size of weights/parameters, which is paramount to discover effective solutions in the considered domain;
• the definition of the MOO scenario highlights the existence of a non-negligible relationship between problems and algorithms.

The paper starts with a thorough description of the benchmark problems, the considered EAs, the neural network controller and the experimental settings in Section 2. Then, a detailed illustration of the outcomes is provided in Section 3, which are further discussed in Section 4. Finally, Section 5 reports the main findings and potential future research ideas.

2. Materials and Methods

This section contains a description of the experimental problems that have been considered, as well as the evolutionary algorithms, the neural network controller, the simulator and the parameter settings used to perform the experiments.

2.1. Problems

2.1.1. 4-Bit Parity

The first problem is the 4-bit parity task, which consists in calculating the number of 1-bits in the input string and returning a value checking if the sum of 1-bits is even (output is 1) or odd (output is 0). The problem is illustrated in Figure 2 and is a commonly used task in the evolutionary computation literature [38,39,40,41,42].

Specifically, our evaluation is made by considering all the possible 4-bit input strings (i.e., $2^4=16$) and verifying the number of correct parity values returned. The fitness function is defined as in Eq. 1.

$$F_{parity}=\sum _{s=1}^{N_{strings}}|O_s-\widehat{O_s}|$$

(1)

In Eq. 1, the symbol $O_s$ is the expected parity output for the input string s, $\widehat{O_s}$ indicates the output returned by the controller and $N_{strings}$ represents the number of considered 4-bit input strings ($N_{strings}=16$).

2.1.2. Double-Pole Balancing

The second problem is the double-pole balancing task [37], a widespread benchmark to assess an algorithm performance as demonstrated by the large body of literature in the field [30,43,44,45,46,47]. The task involves the presence of two poles placed on the top of a wheeled mobile cart, which has to move properly to avoid poles falling (see Figure 3). The description of the parameters characterizing the problem is reported in Table 1.

The objective function of the problem can be formulated as in Eq. 2.

$$F_{dpole}={\displaystyle \frac{1}{N_{trials}}}\sum _{i=1}^{N_{trials}}{N}_{steps}-n_{steps}\left(i\right)$$

(2)

Symbol $N_{steps}$ denotes the length of a trial ($N_{steps}=1000$), $n_{steps}\left(i\right)$ in Eq. 2 represents the number of steps the cart keeps both poles upright during the trial i, whereas $N_{trials}$ indicates the number of trials. An episode is prematurely ended in two cases:

• the cart goes out of the track ($\left|x\right|>2.4m$);
• the pole angles are falling ($\left|{\theta }_j\right|>{36}^{\circ }$, with $j=1,2$).

In this work, we focus on the “Fixed Initial States condition” version of the problem [30,31], in which the evaluation of possible solutions is averaged over 8 trials ($N_{trials}=8$). In more detail, at each trial the state variables (x: position of the cart, ${\theta }_j$: angle of the pole j, with $j=1,2$) are initialized according to Table 2.

Further details on the system dynamics can be found in [30,31,37].

2.1.3. Grid Navigation

The third problem is a grid navigation task in which an agent has to reach the central location (target) of a square grid starting from one of the four corners (see Figure 4). The grid has size $501\times 501$. Similarly to the afore-described problems, grid navigation represents a largely used benchmark to evaluate genetic and evolutionary algorithms [48,49,50,51].

The objective/fitness function considers the distance between the target location and the final agent location, as expressed in Eq. 3.

$$F_{grid}={\displaystyle \frac{1}{N_{trials}}}\sum _{i=1}^{N_{trials}}{d}_{ta}\left(i\right)$$

(3)

In Eq. 3, the symbol $d_{ta}\left(i\right)$ indicates the final agent-target distance at the trial i and $N_{trials}$ counts the number of trials. Since there are four possible starting locations (i.e., the corners of the grid), we set $N_{trials}=4$. The target-agent distance $d_{ta}\left(i\right)$ is calculated by considering the minimum number of cells the agent has to visit in order to arrive at the target, and is defined in Eq. 4.

$$d_{ta}\left(i\right)=|t_x\left(i\right)-a_x\left(i\right)|+|t_y\left(i\right)-a_y\left(i\right)|$$

(4)

The symbols $(t_x\left(i\right),t_y\left(i\right))$ and $(a_x\left(i\right),a_y\left(i\right))$ denote, respectively, the location of the target and the agent during the trial i.

The agent can move one cell either horizontally (i.e., left and right) or vertically (i.e., up and down) within the grid. A trial prematurely stops if the action of the agent causes its exit from the grid.

2.1.4. Test Function Optimization

The last problem is test function optimization, which constitutes a common benchmark for evaluating optimization methods [23,31,34,52,53,54,55]. In particular, we consider five different test functions: Ackley [56], Griewank [57], Rastrigin [58], Rosenbrock [59] and Sphere. Eqs. 5 - 9 provide the definition of the considered functions, in which the input sequence is identified with the symbol v and its length with the symbol n.

$$F_{ackley}=-20exp\left(-0.2\sqrt{{\displaystyle \frac{1}{n}}\sum _{i=1}^n{v_i}^2}\right)-exp\left({\displaystyle \frac{1}{n}}\sum _{i=1}^{n}cos\left(2\pi v_i\right)\right)+20+exp\left(1\right)$$

(5)

$$F_{griewank}=1+{\displaystyle \frac{1}{4000}}\sum _{i=1}^n{v_i}^2-\prod _{i=1}^{n}cos\left({\displaystyle \frac{v_i}{\sqrt{i}}}\right)$$

(6)

$$F_{rastrigin}=\sum _{i=1}^n({v_i}^2-10cos\left(2\pi v_i\right)+10)$$

(7)

$$F_{rosenbrock}=\sum _{i=1}^{n-1}(100{(v_{i+1}-{v_i}^2)}^2+{(v_i-1)}^2)$$

(8)

$$F_{sphere}=\sum _{i=1}^n{v_i}^2$$

(9)

The fitness function for test function optimization is defined as the sum of the single functions according to Eq. 10.

$$F_{funct}=F_{ackley}+F_{griewank}+F_{rastrigin}+F_{rosenbrock}+F_{sphere}$$

(10)

2.2. Performance

To assess the overall performance of the discovered solutions with respect to the considered MOO scenario, we adopt the fitness function described in Eqs. 11 - 12:

$$minF$$

(11)

$$F=w_{parity}{F}_{parity}+w_{dpole}{F}_{dpole}+w_{grid}{F}_{grid}+w_{funct}{F}_{funct}$$

(12)

As concerns Eq. 12, we use coefficients equal to 1.0 for each objective (i.e., $w_{parity}=w_{dpole}=w_{grid}=w_{funct}=1.0$). Despite the simplicity of the approach, our design choice alleviates from the burden of choosing tailored values, which requires expertise and may lead to different evolutionary paths.

2.3. Evolutionary Algorithms

The EAs employed in this work are described in the next sections. A comprehensive schematic is provided in Figure 5. Further details about the different algorithms can be found in [19,29,30,31].

We emphasize that the term “solution” represents a set of floating-point values (also called “genes”) directly encoding the parameters (i.e., biases and connection weights) of the controller that must be evolved.

2.3.1. GGA

The Generational Genetic Algorithm (GGA) [29] is a pioneering method to evolve solutions capable of adapting to dynamic environments [60,61]. Moreover, it has been largely used in collective domains like Swarm Robotics [62,63,64,65,66] or Multi-Agent Systems (MASs) [67,68,69,70,71,72], as well as for problems involving single agents [73,74,75,76].

The algorithm works according to the schematic provided in Figure 5-(a): a population of candidate solutions is evolved until a termination criterion is not met. Each solution undergoes an evaluation process returning a fitness value rating its effectiveness. After the entire population has been evaluated, the best $N_r$ solutions are selected for reproduction, each one generating $N_o$ offspring through mutation and asexual crossover operators. Moreover, selected solutions are retained in the population through elitism. This iterative process aims to enhance the efficacy of solutions during evolution.

2.3.2. OpenAI-ES

The OpenAI Evolutionary Strategy (OpenAI-ES) [19] is a relatively novel technique that achieved competitive results in problems like Atari games [19], classic control [33,35], competitive co-evolution [77], robot locomotion [19,33,78] and swarm robotics [79,80,81]. Differently from a traditional EA, OpenAI-ES evolves a single solution termed “centroid”. In particular, at each iteration of the evolutionary process, $N_{samples}$ samples are extracted from a Gaussian distribution and a set of solutions are derived according to Eq. 13.

$$s\to \left\{\begin{array}{c}{z}^{+}=c+\sigma \times s\hfill \\ z^{-}=c-\sigma \times s\hfill \end{array}\right.$$

(13)

In Eq. 13, the symbol s represents the generic sample, c denotes the centroid, $z^{+}$ and $z^{-}$ indicate the solutions that undergo evaluation, while $\sigma$ is the mutation rate.

Once the pool of solutions has been evaluated, a fitness ranking is used to estimate the gradient of the expected fitness function. Finally, the centroid is updated through Adam [82], a widespread optimizer keeping historical information to drive the search process towards solutions that are more likely to be effective. A schematic of OpenAI-ES is shown in Figure 5-(b). Differently from existing works [19,33,78,79,80], here OpenAI-ES does not exploit weight decay [83], since reducing the size of weights represents a clear advantage in the considered MOO scenario.

2.3.3. SSS

The Stochastic Steady State (SSS) [30] is an EA that found applications in problems like double-pole balancing [30,31,34,84], swarm robotics [84,85], car racing games [84] and Pybullet robot locomotors [86]. The algorithm differs from GGA with respect to the reproduction and selection process: at each iteration, each solution in the population (parent) generates one offspring through mutation and asexual crossover. Then, both parents and offspring are evaluated and the best $PopSize$ solutions are retained and form the next population. This iterative process allows SSS to improve performance during evolution. Figure 5-(c) provides a schematic of the operation of SSS.

2.3.4. SSSHC

The Stochastic Steady State with Hill-Climbing (SSSHC) [31] is a memetic algorithm [87,88,89] combining SSS with a Hill-Climbing algorithm [90] seeking to further refine the solution performance. SSSHC proves successful with regard to problems such as 5-bit parity, double-pole balancing and test function optimization [31,34]. Specifically, for the experiments reported here, we used the variant introduced in [34].

The operation of SSSHC follows the one of SSS, but a refinement phase is added after selection: each selected solution undergoes $N_{refs}$ iterations in which a single gene mutation is applied and the modified solution is retained only if its performance improves. A description of SSSHC is illustrated in Figure 5-(d).

2.4. Controller

As a single controller for the considered problems, we used a recurrent neural network [91,92] formed by 4 inputs, an internal layer of 10 neurons, and 1 output. Bias is applied to internal and output neurons, which activate based on the tanh function. The information the input and output neurons encode depending on the specific problem is listed here below (see also Table 3).

• 4-bit parity: the inputs represent the bits of the string that has to be checked, while the output is the parity value associated to the string;
• double-pole balancing: the inputs encode normalized values for the cart position (x) and pole angles (${\theta }_1$ and ${\theta }_2$), and an alert signal that activates if either the cart approaches track edges ($\left|x\right|>2m$), or the poles are almost falling ($\left|{\theta }_j\right|>{30}^{\circ }$, with $j=1,2$). The output represents the force that will be exerted to the cart;
• grid navigation: the inputs are the normalized locations of the agent ($(a_x,a_y)$) and the target ($(t_x,t_y)$), whereas the output indicates the motion direction (i.e., left, up, right or bottom).

Regarding test function optimization, the connection weights of the neural network constitute the input vector v used to compute the functions in Eq. 5 - 9. Therefore, not only does the controller have to optimize simultaneously four problems exhibiting different properties, but it also has to address them in potentially conflicting ways. In fact, the latter problem requires to minimize the size of connection weights, which could prevent the discovery of effective solutions for the other problems.

2.5. Simulator and Experimental Settings

The Framework for Autonomous Robotics Simulation and Analysis (FARSA) [93] tool has been employed for the experiments reported in this work, since it found applications in similar experimental studies [30,31,34]. Moreover, it has been successfully used also in robotic scenarios [76,85,94,95].

The considered EAs have been compared by running 30 replications of the experiments. Evolution last ${10}^9$ evaluation steps for each algorithm in order to provide a fair comparison. Other parameter settings are reported in Table 4.

3. Results

The performance comparison is shown in Table 5, Figure 6 and Figure 7. Clearly, OpenAI-ES and SSSHC strongly outperform GGA and SSS (Kruskal-Wallis H test, $p<{10}^{-6}$, see Table 6), which indicates their capability to discover more effective solutions for the considered MOO scenario. As can be seen in Figure 6, OpenAI-ES and SSSHC quickly reduce the fitness in the first ${10}^8$ evaluation steps and then almost stabilize after $2\times {10}^8$ evaluation steps. This implies that both algorithms manage to identify the direction in the search space corresponding to an optimized performance. Conversely, the fitness curves of GGA and SSS show a slower drop, which prevents them from further optimization.

Notably, OpenAI-ES and SSSHC have similar performance (Mann-Whitney U test, $p>0.05$, see also Figure 7 and Table 6). Therefore, a relatively simple strategy like SSSHC is not inferior to a more sophisticated algorithm like OpenAI-ES, which exploits historical information to channel the search in the space of possible solutions. By looking at the outcomes reported in Figure 7, OpenAI-ES manages to find solutions with performance below 1000 (see bottom outliers in Figure 7), while SSSHC does not reach similar performance levels.

We corroborate our results by delving into the fitness achieved by the different algorithms with regard to the specific problems. As highlighted in Figure 8 and Table 7, OpenAI-ES achieves the best results in the 4-bit parity and double-pole balancing problems, while SSSHC bests other algorithms with respect to grid navigation and test function optimization. In particular, OpenAI-ES and SSSHC succeed in the minimization of test functions, which allows to quickly reduce fitness (see also Figure 6). Instead, GGA and SSS fail in finding similar solutions. Furthermore, SSSHC is notably superior to the other algorithms in the grid navigation problem (Kruskal-Wallis H test, $p<{10}^{-6}$). Interestingly, OpenAI-ES is the only algorithm that manages to discover improved solutions in the double-pole balancing problem (Kruskal-Wallis H test, $p<{10}^{-6}$, see also Figure 8 and Table 7).

Lastly, we investigate the performance of the different algorithms with respect to Ackley, Griewank, Rastrigin, Rosenbrock and Sphere functions, as illustrated in Figure 9 and Table 8. Notably, OpenAI-ES is more effective than other algorithms in optimizing the Rosenbrock function (Kruskal-Wallis H test, $p<{10}^{-6}$), whereas SSSHC bests other algorithms with respect to the remaining four test functions (Kruskal-Wallis H test, $p<{10}^{-6}$). GGA and SSS achieve poor performance in this context, particularly concerning the Rosenbrock function (see Table 8).

Overall, the reported analysis clearly reveals a superior performance of OpenAI-ES and SSSHC over GGA and SSS. This is related to their capability to decrease weights, as shown in Table 9. In particular, the former algorithms evolve controllers with a limited weight size, which is paramount for the test function optimization. Nevertheless, this tendency prevents the algorithms from finding effective solutions with respect to the other problems, particularly double-pole balancing. This is especially true for SSSHC (see Figure 8 and Table 7). OpenAI-ES and SSS have similar weight sizes, but their performance is notably different. This can be explained by considering that OpenAI-ES exploits historical information to effectively sample the search space, while SSS does not adopt similar mechanisms.

4. Discussion

The presented results show that two algorithms emerge as more suitable options for the considered MOO scenario. A noteworthy aspect regards the different level of complexity between OpenAI-ES and SSSHC. In fact, the former method exploits mirrored sampling [96] and historical information to identify the search direction in the space of solutions. Conversely, SSSHC is a memetic algorithm that seeks to refine selected solutions through single gene mutations. Despite the simplicity, SSSHC works well in this scenario. However, the approach presents benefits and drawbacks: on one hand, modifying one gene at a time eliminates the risk of disruptive effects, since only adaptive mutations are retained. On the other hand, the process does not exclude failures in the attempt of improving solution performance and is costly in terms of number of evaluation steps.

OpenAI-ES has an intrinsic propensity to reduce the size of weights even when weight decay is not applied, as in the case of our experiments. This turns out to be crucial in the considered domain, as well as in other domains like robot locomotion [33] and swarm robotics [80]. Conversely, SSSHC exploits single gene mutations performed during refinement to progressively reduce the weight size.

GGA and SSS do not manage to achieve performance comparable to the other algorithms. Specifically, they fail in discovering effective solutions for test function optimization, particularly with respect to the Rosenbrock function. This can be explained by considering that both GGA and SSS seek to enhance performance through mutation and crossover operators, and do not have explicit techniques to reduce weights. As a consequence, they might require longer evolutionary processes in order to find more effective solutions.

Another relevant insight concerns the types of solutions discovered by the considered EAs. In particular, we observe that all the algorithms mainly focus on test function optimization, which allows to quickly enhance performance, and almost ignore the remaining problems. This is strongly related to the remarkable different magnitudes of the fitness values in the worst case (see Table 10): because test function optimization is three order of magnitude bigger than double-pole balancing and grid navigation, the fitness function defined in Eq. 12 is minimized when the component $F_{funct}$ is strongly reduced. As a consequence, the algorithms evolve solutions mostly addressing test function optimization and do not manage to improve performance with respect to the other objectives.

Overall, our results underscore a non-negligible relationship between problem and algorithm. In fact, although OpenAI-ES, SSS and SSSHC proved effective at optimizing double-pole balancing in previous studies [30,31,33,35], the definition of the performance measure for the considered MOO scenario guides EAs to different evolutionary paths, in which one of the objectives dominate the others.

5. Conclusions

Multi-Objective Optimization (MOO) concerns the identification of solutions capable of optimizing multiple conflicting objectives at the same time. Because the discovery of a global optimum is not trivial, if not impossible, Pareto-optimal solutions are typically taken into account. Evolutionary Algorithms (EAs) have demonstrated a great ability to deal with MOO scenarios, since they provide effective solutions without any prior knowledge about the considered domains. A partially related research field lies in the search for a single controller able to address different problems within the same class. The main advantage is the possibility of transferring acquired knowledge on a given problem to others, without further fine-tuning. The approach is similar to transfer learning [97], which proved valuable in Deep Learning (DL) applications [98,99,100].

This work delves into the definition of a novel MOO scenario and the analysis of how some state-of-the-art works address it. Specifically, we compared GGA, OpenAI-ES, SSS and SSSHC with respect to the evolution of a single neural network controller able to simultaneously optimize four benchmark problems: (i) 4-bit parity, (ii) double-pole balancing, (iii) grid navigation, and (iv) test function optimization. The definition of the scenario makes the overall problem challenging, since the multiple objectives conflict with each other. Furthermore, the controller is used differently depending on the specific problem: for example, the connection weights determine the action to be executed in the double-pole balancing or grid navigation problems, while they represent the input vector for the test functions to be optimized. Outcomes indicate that OpenAI-ES and SSSHC best the other algorithms, since they are more effective at sampling the search space. In particular, OpenAI-ES exploits symmetric sampling and historical information, whereas SSSHC benefits from single-gene mutations. Interestingly, a relatively trivial algorithm like SSSHC is not inferior to a modern and sophisticated method like OpenAI-ES in this context. Instead, SSSHC is better than OpenAI-ES with respect to grid navigation and test function optimization, while the opposite is true for 4-bit parity and double-pole balancing. In addition, OpenAI-ES and SSSHC manage to reduce weight size more efficiently than GGA and SSS, a worthwhile feature in the considered domain.

As future research directions, we plan to incorporate algorithms specifically tailored for MOO domains, such as NSGA-II and VEGA, for future comparisons. This will provide a comprehensive analysis of the most suitable methods for MOO. Furthermore, modifications of the problem formulation in Eq. 12 will be object of future studies, particularly through either the usage of different coefficient values for the different objectives, or the normalization of the fitness functions defined in Eqs. 1, 2, 3 and 10. In addition, further experiments in which parameter settings are systematically varied will be performed, aiming to validate our analysis more thoroughly. Finally, future works could investigate different scenarios (e.g., robotics, classic control) in order to extend and, possibly, generalize the considerations reported here.