Constructing the Bounds for Neural Network Training Using Grammatical Evolution

1. Introduction

Artificial neural networks (ANNs) are widespread machine learning models, which base their dynamics on a series of parameters which are also called computational units or weights in the relevant literature [1,2]. Neural networks are widely used in a variety of scientific applications, such as problems from physics [3,4,5], solutions of differential equations [6,7], agriculture problems [8,9], chemistry problems [10,11,12], problems that appear in the area of economic sciences [13,14,15], problems related to medicine [16,17] etc. Commonly, the neural networks are expressed as a function $N(\overrightarrow{x},\overrightarrow{w})$, with the assumption that the vector $\overrightarrow{x}$ stands for the input vector to the neural network and the vector $\overrightarrow{w}$ is the parameter vector that should be estimated. The first vector is called pattern in the bibliography and the second one weight vector. Techniques that adjust the parameter vector $\overrightarrow{w}$ minimize the so-called training error, defined as:

$$E\left(N\left(\overrightarrow{x},\overrightarrow{w}\right)\right)=\sum _{i=1}^M{\left(N\left({\overrightarrow{x}}_i,\overrightarrow{w}\right)-y_i\right)}^2$$

(1)

In Equation (1) the set $\left(\overrightarrow{x_i},y_i\right),i=1,...,M$ is denoted as the train set of the neural network. The $y_i$ are considered the target outputs for the $\overrightarrow{x_i}$ points . As proposed in [18], neural networks can be expressed as the following summation:

$$N\left(\overrightarrow{x},\overrightarrow{w}\right)=\sum _{i=1}^H{w}_{(d+2)i-(d+1)}\sigma \left(\sum _{j=1}^d{x}_j{w}_{(d+2)i-(d+1)+j}+w_{(d+2)i}\right)$$

(2)

The number of processing units is denoted by H and the value d stands for the dimension of input vector $\overrightarrow{x}$. The function$\sigma \left(x\right)$ denotes the well - known sigmoid function expressed as:

$$\sigma \left(x\right)=\frac{1}{1+exp(-x)}$$

(3)

From the above equations one can calculated the total number of parameters in the neural network as:

$$n=\left(d+2\right)H$$

(4)

The training of neural networks has been performed by a variety of numerical methods in recent bibliography, such as: the Back Propagation method [19,20], the RPROP method [21,22,23], the Adam optimizer [24,25] etc. Furthermore, more advanced optimization methods have been incorporated to optimize the parameter vector of neural networks by minimizing Equation (1) such as Quasi Newton methods [26,27], the Tabu Search method [28], Simulated Annealing [29,30], Genetic Algorithms [31,32], Particle Swarm Optimization [33,34], Differential Evolution [35,36], Ant Colony Optimization [37] etc.

Also, recently a series of hybrid optimization methods have been proposed to tackle the minimization of the training error, such as a method that combines PSO and genetic algorithms [38,39], incorporation of a particle swarm optimization algorithm and gravitational search algorithm [40], a combination of a genetic algorithm and controlled gradient method [41] etc.

In addition, an important topic for artificial neural networks such as parameter initialization has been studied by several researchers in recent years. In this area appeared papers such as initialization with decision trees [42], incorporation of the Cauchy’s inequality [43], discriminant learning [44] etc. Also, another interesting topic in the area of neural networks is weight decaying, where some regularization is applied to the parameters of the neural network to avoid the overfitting problem. For this topic, several research papers have been suggested, such as a method that incorporates positive correlation [45], the SarProp algorithm [46], usage of pruning techniques [47] etc.

This paper proposes a two-stage method for efficient training of the parameters of artificial neural networks. In the first stage, a value interval is constructed for the parameters of the neural network through Grammatical Evolution [48]. In the first stage of this work, after an initial estimate of the interval for the parameters of the neural network is made, a series of rules for partitioning and expanding the initial value interval are applied with the usage of Grammatical Evolution, until a promising value interval is found. In the second stage of the method, a genetic algorithm is applied to train the parameters of the artificial neural network. Training is performed within the optimal interval of values found in the first stage of the method.

The rest of this article is organized as follows: in Section 2 the steps of the proposed method are outlined in detail, in Section 3 the datasets used in the conducted experiments as well the experimental results are presented and finally in Section 4 some conclusions are presented.

2. Method description

In this section a detailed description of the Grammatical Evolution technique is provided, accompanied by the proposed language that creates intervals for the parameters of the neural networks. Subsequently, the first phase of the proposed technique is described and finally, the second phase with the application of a Genetic Algorithm to the outcome of the first phase, is thoroughly analyzed.

2.1. Grammatical evolution

Grammatical evolution is an evolutionary process, where the chromosomes represent production rules of some provided BNF (Backus–Naur form) grammar [49] and this evolutionary process can evolve programs in the underlying language. Grammatical Evolution has been used successfully in many real world problems, such as function approximation[50,51], solution of trigonometric equations [52], automatic music composition [53], neural network construction [54,55], creating numeric constraints [56], video games [57,58], estimation of energy demand [59], combinatorial optimization [60], cryptography [61], evolution of decision trees [62], automatic design of analog electronic circuits [63] etc.Any BNF grammar can be described as the set $G=\left(N,T,S,P\right)$ where

• N is the set of the non-terminal symbols. Every symbol in N has a series of production rules, used to produce terminal symbols.
• Tis the set of terminal symbols.
• S denotes the start symbol of the grammar, with $S\in N$.
• P is the set of production rules, to create terminal symbols non - terminal symbols. These rules are in the form $A\to a$ or $A\to aB,A,B\in N,a\in T$.

The algorithm initiates from the symbol S and iteratively produces terminal symbols by replacing non-terminal symbols with the right hand of the selected production rule. Every production rule is is selected using the following steps:

• Get the next element V from the current chromosome.
• Select the next production rule according to: Rule = V mod $N_R$, where $N_R$ is the total number of production rules for the current non – terminal symbol.

The grammar used in the current work is shown in Figure 1.

The symbols in <> represent the non-terminal symbols of the grammar. The numbers in parentheses in the right part of each rule, indicate production rule sequence numbers. In the proposed language described by the grammar of the scheme, three distinct operations can be applied to parameters of the artificial neural network either on the left end of the parameter’s value range (described by <lcommand>) or on the right end of the value range (described by the <rcommand> symbol). These commands are

• NOTHING. This command means that no action takes place.
• EXPAND. With this command, the corresponding end of the value field is extended by 50% of the width of the field.
• DIVIDE. With this command, the corresponding end of the value field is shrunk by 50% of the width of the field.

As a complete example consider a neural network with $H=2$ hidden nodes and the dimension of the objective problem is set to $d=2$. Hence, the total number of parameters in the neural network is $n=(d+2)H=8$. Also, for reasons of simplicity, we consider that the initial range of values for all parameters of the artificial neural network is the interval [-2,2] and is denotes as a the set of intervals

$$I_N=\left\{\left[-2,2\right],\left[-2,2\right],\left[-2,2\right],\left[-2,2\right],\left[-2,2\right],\left[-2,2\right],\left[-2,2\right],\left[-2,2\right]\right\}$$

(5)

and consider also the chromosome

$$x=\left[9,8,6,4,15,9,16,23,7\right]$$

The steps to produce the final program

$$p_{test}=(x7,EXPAND,NOTHING),(x1,DIVIDE,EXPAND)$$

are shown in Table 1.

After the application of the program $p_{test}$ to the original interval $I_N$ of Equation (5), the new set has as follows:

$$I_N=\left\{\left[-4,2\right],\left[-2,2\right],\left[-2,2\right],\left[-2,2\right],\left[-2,2\right],\left[-2,2\right],\left[0,4\right],\left[-2,2\right]\right\}$$

(6)

2.2. The first phase of the proposed method

Before starting the process of creating partitioning rules and expanding the value interval of the neural network weights, an initial estimate of this interval should be made. In the present work, only a few steps of a genetic algorithm are applied to make an initial estimate of the value interval. At the end of the genetic algorithm, the chromosome with the best value is the

$$IX=\left({IX}_1,{IX}_2,\dots ,{IX}_n\right)$$

(7)

So, the initial interval set for the parameters of the neural network is calculated as

$$I_N=\left\{\left[-{IX}_1,{IX}_1\right],\left\{-{IX}_2,{IX}_2\right\},\dots ,\left[-{IX}_n,{IX}_n\right]\right\}$$

(8)

The first phase of the proposed technique will accept the set $I_N$ as input and seek to compute a new set by reducing the training error of the artificial neural network. The first step is mainly a genetic algorithm and the corresponding steps are as follows:

• Set $N_c$ as the number of chromosomes for the Grammatical Evolution.
• Set as H the number of weights for the neural network.
• Set $N_g$ the maximum number of allowed generations.
• Set as $p_s$ the selection rate , with $p_s\le 1$.
• Set as $p_m$ the mutation rate, with $p_m\le 1$.
• Set $N_s$ as the number of randomly created neural networks, that will be used in the fitness calculation.
• Initialize randomly the $N_c$ chromosomes. Every chromosome is a set of integer numbers used to produce valid programs through Grammatical Evolution and the associated grammar of Figure 1.
• Set $f^{*}=\left[\infty ,\infty \right]$, the best discovered fitness. For this algorithm, we consider the fitness function $f_g$ of any given chromosome g as an interval $f_g=\left[f_{g,\mathbf{low}},f_{g,\mathbf{upper}}\right]$
• Set iter=0.
• For$i=1,\dots ,N_c$ do

  (a)

  Createfor the chromosome $c_i$ the corresponding program $p_i$ using the grammar of Figure 1.

  (b)

  Applythe program $p_i$ to $I_N$ in order to produce the bounds $\left[\overrightarrow{L_{p_i}},\overrightarrow{R_{p_i}}\right]$.

  (c)

  Set$E_{min}=\infty ,E_{max}=-\infty$

  (d)

  For$j=1,\dots ,N_S$do

  • Create randomly $g_j\in$$\left[\overrightarrow{L_{p_i}},\overrightarrow{R_{p_i}}\right]$ as a set for the parameters of neural network.
  • Calculate the associated training error $E_{g_j}={\sum }_{k=1}^M{\left(N\left({\overrightarrow{x}}_k,\overrightarrow{g_j}\right)-y_k\right)}^2$
  • If $E_{g_j}\le E_{min}$ then $E_{min}=E_{g_j}$
  • If $E_{g_j}\ge E_{max}$ then $E_{max}=E_{g_j}$

  (e)

  EndFor

  (f)

  Set $f_i=\left[E_{min},E_{max}\right]$ as the fitness value for the chromosome $c_i$.

• EndFor
• Apply the selection procedure. Firstly, the chromosomes are sorted with correspondence to their fitness values. Since fitness is considered an interval, a fitness comparison function is required. For this reason the operator $L^{*}\left(f_a,f_b\right)$ to compare two fitness values $f_a=\left[a_1,a_2\right]$ and $f_b=\left[b_1,b_2\right]$ as follows:

  $$\begin{array}{ccc}\hfill L^{*}\left(f_a,f_b\right)& =& \left\{\begin{array}{cc}TRUE,\hfill & a_1<b_1,OR\left(a_1=b_{1}AND{a}_2<b_2\right)\hfill \\ FALSE,\hfill & OTHERWISE\hfill \end{array}\right.\hfill \end{array}$$

  (9)
  In practice this means that the fitness value $f_a$ is considered smaller than $f_b$ if $L^{*}\left(f_a,f_b\right)=TRUE$. The first $\left(1-p_s\right)\times N_c$ chromosomes with the lowest fitness values are copied to the next generation. The remaining chromosomes are substituted by chromosomes produced by the crossover procedure. During the selection process, for every new offspring, two chromosomes are selected as parents from the population using well - known procedure of tournament selection.
• Apply the crossover procedure. For each pair $(z,w)$ of parents, two new chromosomes $\tilde{z}$ and $\tilde{w}$ are created using the one - point crossover, graphically shown in Figure 2.
• Apply the mutation procedure. For each element of every chromosome alter the corresponding element with probability $p_m$.
• Set iter=iter+1
• If $iter\le N_g$goto step 10.

2.3. The second phase of the proposed method

• Initialization Step

  (a)

  Set $N_c$ as the number of chromosomes that participate in the genetic algorithm.

  (b)

  Set $N_g$ the maximum number of allowed iterations.

  (c)

  Set H the number of weights for the neural network.

  (d)

  Get the best interval S from the previous step of Subsection 2.2.

  (e)

  Initialize using uniform distribution the $N_C$ chromosomes in S.

  (f)

  Set as $p_s$ the selection rate, with $p_s\le 1$.

  (g)

  Set as $p_m$ the mutation rate, with $p_m\le 1$.

  (h)

  Set iter=0.

• Fitness calculation Step

  (a)

  For$i=1,\dots ,N_g$do

  • Calculate the fitness $f_i$ of chromosome $g_i$ as $f_i={\sum }_{j=1}^M{\left(N\left({\overrightarrow{x}}_j,\overrightarrow{g_i}\right)-y_j\right)}^2$

  (b)

  EndFor

• Genetic operations step

  (a)

  Selection procedure. Initially, the chromosomes are sorted according to their fitness values. The first $\left(1-p_s\right)\times N_c$ chromosomes with the lowest fitness values are copied to the next generation. The remaining chromosomes are substituted by chromosomes produced by the crossover procedure. During the selection process, for every new offspring, two chromosomes are selected as parents from the population using well - known procedure of tournament selection.

  (b)

  Crossover procedure: For each pair $(z,w)$ of selected parents, two chromosomes $\tilde{z}$ and $\tilde{w}$ are constructed using the following equations:

  $$\begin{array}{ccc}\hfill \widetilde{z_i}& =& a_i{z}_i+\left(1-a_i\right)w_i\hfill \\ \hfill \widetilde{w_i}& =& a_i{w}_i+\left(1-a_i\right)z_i\hfill \end{array}$$

  (10)

  Where $a_i$ is random number with the property $a_i\in [-0.5,1.5]$ [64].

  (c)

  Mutation procedure: For each element of every chromosome alter the corresponding element with probability $p_m$.

• Termination Check Step

  (a)

  Set $iter=iter+1$

  (b)

  If $iter\le N_g$goto step 2. else apply a local search procedure to the best chromosome of the population. In the current work the BFGS variant of Powell [65] was used.

3. Experiments

The effectiveness of the proposed method was evaluated on a series of classification and regression datasets, used in the relevant literature. These datasets can be downloaded from the following online databases:

• UCI dataset repository, https://archive.ics.uci.edu/ml/index.php [66]
• Keel repository, https://sci2s.ugr.es/keel/datasets.php [67].
• The Statlib URL ftp://lib.stat.cmu.edu/datasets/index.html.

3.1. Experimental datasets

The classification datasets used in this paper are the following:

• Appendictis a medical purpose dataset, suggested in [68].
• Australian dataset [69], a dataset related to credit card transactions.
• Balance dataset [70], related to psychological states.
• Cleveland dataset, which is a medical dataset [71,72].
• Dermatology dataset [73], a medical dataset related to erythemato-squamous diseases.
• Heart dataset [74], a medical dataset related to heart diseases.
• Hayes roth dataset [75].
• HouseVotes dataset [76], related to votes in the U.S. House of Representatives Congressmen.
• Ionosphere dataset, used for classification of radar returns from the ionosphere [77,78].
• Liverdisorder dataset [79], a medical dataset related to liver disorders.
• Mammographic dataset [80], used to identify breast tumors.
• Parkinsons dataset, a medical dataset related to Parkinson’s disease (PD) [81].
• Pima dataset [82], used to detect the presence of diabetes.
• Popfailures dataset [83], a dataset related to climate measurements.
• Regions2 dataset, medical dataset related to hepatitis C [84].
• Saheart dataset [85], a medical dataset related to heart diseases.
• Segment dataset [86], an image processing dataset.
• Wdbc dataset [87], a medical dataset related to breast tumors.
• Wine dataset, used to detect the origin of wines [88,89].
• Eeg datasets, a medical dataset related to EEG measurements [90]. There are 3 different cases from this dataset used here denoted as Z_F_S, ZO_NF_S, ZONF_S.
• Zoo dataset [91], used to classify animals.

The description of the used regression datasets has as follows:

• Abalone dataset [93], used to to predict the age of abalone from physical measurements.
• Airfoildataset, derived from NASA [94].
• Baseball dataset, used to estimate the salary of baseball players.
• BK dataset [95], used to predict the points scored in a basketball game.
• BL dataset, an electrical engineering dataset.
• Concrete dataset [96].
• Dee dataset, used to predict the price of electricity.
• Diabetes dataset, a medical dataset.
• Housing dataset, provided in [97].
• FA dataset, used to predict the fit body fat.
• MB dataset, available from from Smoothing Methods in Statistics [98].
• MORTGAGE dataset, related to Economic data from USA.
• PYdataset, (Pyrimidines problem) [99].
• Quake dataset, used to predict the strength of a earthquakes.
• Treasure dataset, related to Economic data from USA.
• Wankara dataset, a dataset related to weather.

3.2. Experimental results

The proposed method was coded in ANSI C++ and the optimization methods were obtained from the freely available from the OPTIMUS computing environment, downloaded from https://github.com/itsoulos/OPTIMUS/(accessed on 9 September 2023). The results were validated using the 10 - fold validation technique in all datasets. The experiments were executed 30 times for each dataset using a different seed for the random generator each time. The average classification error is reported for the case of classification datasets and the average mean test error for the regression datasets. The machine used in the experiments was an AMD Ryzen 5950X with 128GB of RAM, running the Debian Linux operating system. The experimental settings are listed in the Table 2. The experimental results for the classification datasets are shown in Table 3, while for the regression datasets the results are shown in Table 4. The following applies to the results tables:

• A genetic algorithm where the parameters have the values of Table 2 used to train a neural network with H hidden nodes. The results in the experimental tables are denoted by the label GENETIC.
• The Adam optimization method is used to train a neural network with H hidden nodes. The column ADAM denotes the results for this method.
• The RPROP method is used to train a neural network with H hidden nodes. The corresponding results are denoted by RPROP in the relevant tables.
• The NEAT method (NeuroEvolution of Augmenting Topologies ) [100], where the maximum number of allowed generations is the same as in the case of the Genetic algorithm.
• The proposed method (denoted as PROPOSED) was used with the experimental settings and are shown in Table 2.
• An extra line was also added to the experimental tables under the title AVERAGE. This line represents the average classification or regression error for all datasets.

In general, the proposed technique outperforms the rest of the techniques in the experiments. In fact, in many cases, the test error is reduced by more than 70%. The numerical superiority of the proposed technique over the others in the performed experiments is presented graphically and in the Figure 3 and Figure 4, where Wilcoxon signed-rank tests were performed on the executed experiments. However, since the proposed technique consists exclusively of a series of genetic algorithms, it can be significantly accelerated by the use of parallel computing techniques, since one of the main characteristics of genetic algorithms is their ease of parallelization [101,102]. Programming techniques that can be used to parallelize the method could be the MPI programming interface [103] or the OpenMP programming library [104].

4. Conclusions

This paper proposed a two-step technique to efficiently find a reliable interval of values for the parameters of artificial neural networks. In the first stage with partitioning techniques and using Grammatical Evolution, a range of parameter values was searched and, in the second stage, a genetic algorithm trained the artificial neural network within the optimal range of values of the first stage. The proposed technique is highly general and has been successfully applied to both data fitting and classification problems. In both cases, the experimental results demonstrate its superiority over other techniques appearing in the relevant literature on the training of artificial neural networks. In fact, in many cases in the performed experiments, the proposed method outperforms others that were tested in percentages that exceed 70%. In the future, a series of actions can be carried out to improve the results and speed up the proposed method, such as:

• There is a need for more efficient techniques for initializing the value space for artificial neural network parameters. In the present work, the optimal result from the execution of a limited number of steps by a genetic algorithm was used as an initial estimate of the value interval.
• In the present work, the same techniques as in any whole-chromosome genetic algorithm were used to perform the crossover and mutation operations. Research could be done at this point to find more focused crossover and mutation techniques for this particular problem.
• The present technique consists of two phases, in each of which a problem-adapted genetic algorithm is executed. This means that significant computational time is required to complete the algorithm. However, since genetic algorithms are inherently parallelizable, modern parallel programming techniques could be used here.