Hyperparameter Optimization of CNN Using DBO for Speaker Recognition

1. Introduction

Recognition is widely used in fields such as healthcare, security, and intelligent transportation. Image recognition is a mature method that uses light waves. Similarly, sound waves are also a significant approach to identify the biometrics, e.g., speaker recognition is standing out for its convenience and reliability. A variety of voice feature extraction methods and recognition models have been investigated. The main feature extraction methods include the Linear Prediction Coding [1], the Linear Prediction Cepstral Coefficients (LPCC) [2,3] and the Mel-Frequency Cepstral Coefficients (MFCC) [4,5]. Based on the above methods, a large number of mature recognition models and applications have been proposed. Nakagawa et al., proposed a text-independent speaker recognition method based on Hidden Markov models (HMM) and Gaussian mixture model (GMM), and evaluated the robustness of the model affected by speech style [6]. Matsui et al., studied a speaker recognition method that was robust to background noise. This method combined the speaker and noise source into a noisy speaker HMM with a specific signal-to-noise ratio (SNR), and used this likelihood value to obtain recognition results [7]. Limkar et al., proposed to compare the recognition rates of multiple combination models using vector quantization and dynamic time warping, and the results showed that LPCC and MFCC had better performance [8]. Keogh et al., introduced a new technology for precise indexing, which was a model based on template matching. Using the idea of dynamic programming, they solved the problem of different pronunciation lengths, however, it could only be used in isolated word speech recognition. The effect of other application scenarios was not ideal [9]. Zheng et al., proposed the GMM Universal Background Model, which utilized the speaker's trained speech and a Bayesian adaptive form to adjust the parameters of UBM. This model reduced dependence on text and accurately identified the speaker [10]. Campbell et al., proposed the method of using GMM hypervectors in support vector machine (SVM) classifiers, and using two new SVM kernels based on distance measurement between GMM models. Experiments had shown that these SVM kernels produced excellent classification accuracy in NIST speaker recognition and evaluation tasks [11]. However, the performance of the above models is extremely vulnerable to environmental noise.

In recent years, CNN has been widely applied in the field of image recognition and object detection due to its excellent feature extraction ability, thus it can be also used in the field of speaker recognition. Achar et al., proposed a hybrid recognition method based on CNN and MFCC, which achieved a recognition accuracy of 87.5% [12]. Liu et al., proposed a new model to improve the recognition accuracy of short speech speaker recognition systems by addressing the issue of GMM being unable to accurately recognize short speech speakers, and reduce the recognition error rate from 4.9% to 2.5% [13]. Joonet et al., created the VoxCeleb2 dataset and used a deep CNN to classify it with 92.67% accuracy [14]. Jagiasi et al., described a text-independent CNN model for speaker recognition, and achieved recognition rates of 75% to 85% [15]. Wang et al., proposed a voiceprint recognition model based on Mel time-spectrum convolutional neural network for identifying faults in transformers during operation. This method constructed a CNN model by feature extraction preprocessing and Mel filter. This model could recognize the voiceprint of transformers by four different operating faults [16]. However, the hyperparameter settings of CNNs had a large impact on model performance, and the work was usually based on trial-and-error or experience. Population-based intelligent algorithms such as genetic algorithm (GA) and particle swarm optimization (PSO) were used to optimize the hyperparameters of CNN [17,18,19,20]. Yoo et al., proposed a method of optimizing CNN with GA and tested image recognition on the MNIST dataset, and achieved an accuracy rate of 99.4% [17]. Ishaq et al., used GA to adjust the hyperparameters of CNN, and achieved an accuracy rate of 95.5% in the emotion recognition test, which had a great advantage over other methods [18]. Chen et al., proposed a PSO-optimized adaptive CNN(PSO-CNN) to analyze the spectrogram during the working process of the bearing to determine whether the bearing was damaged. This method had a recognition accuracy rate of 99.9% for the four damage situations [19]. Bhuvansehwari et al., used the dragonfly optimizer based on information gain and the CNN classifier optimized by particle PSO based on depth clustering to identify network attack aircraft types. The optimization algorithm could reduce clustering losses and network losses [20]. The above studies showed that these optimization methods were effective for improving the performance of CNN. After decades of scientific and technological development, the above-mentioned intelligent algorithms have shown limited effects in current complex and huge engineering problems.

Researchers have proposed many heuristic search algorithms Inspired by nature that imitate various biological habits. These algorithm models have two core survival strategies of biological populations, role division and behavioral differentiation. Role division refers to the use of fitness to divide the population into different groups, while behavioral difference means that individuals in different groups have different action strategies. In population, the role played by each individual is not static. During the foraging process, the fitness of individuals will change, which will change their role. But no matter how the population is divided, the ultimate goal of all individuals is the same, that is to find the most suitable habitat for the survival and reproduction of the population. In particular, the action strategies of individuals in bionic algorithms are more reasonable and effective, because the objects they imitate are multiplying and thriving in real life. In recent years, the Whale Algorithm [21], Grey Wolf Optimization [22], Sparrow Search Algorithm (SSA) [13] and Jumping Spider Optimization Algorithm [24] were heuristic algorithms proposed based on the above ideas which were with efficient search capabilities. They have been applied in many fields.

In this study, the DBO is used to optimize the hyperparameters of the CNN to identify speakers. DBO is the latest heuristic algorithm proposed by Xue et al., with excellent exploration and high local optimal avoidance ability [25]. This algorithm simulates the stratum distribution and foraging habits in the dung beetle group, and uses different foraging strategies to find the optimal solution within a certain range, which is the first time this algorithm has been used to optimize CNNs. In addition, a comparison between the DBO with the other algorithms is proposed, and the experimental results demonstrate the superiority of DBO in optimizing CNN hyperparameters. This paper is structured as follows: Sec. II shows the processing flow of audio data. Sec. III introduces CNN model architecture and DBO calculation process. Sec. IV shows the result and discussion about optimized CNN. Sec. V presents conclusions and future work.

2. Preliminary Work

2.1. Speaker Recognition in CNN

The process of using CNN for speaker recognition includes two parts, the generation of spectrogram and the calculation of CNN, as shown in Figure 1. Firstly, convert the audio signal into spectrum diagram data, and then import the obtained images into CNN for feature extraction and classification. Finally, accurately identify the speaker.

2.2. Generation of Spectrogram

Since two-dimensional data is used in CNN, it is necessary to expand the dimension of audio data. The spectrogram is an image that displays the audio frequency spectrum, which denotes the variation of the frequency and amplitude of a speech signal over time. In the spectrogram, the energy of sound is displayed in the form of texture, which is called voiceprint. Voiceprint contains a lot of speaker characteristics, and everyone's voiceprint is different. The essence of speaker recognition is to extract and classify voiceprints.

As shown in Figure 2, the production process of the spectrogram is divided into five steps: i) pre-emphasis is used to enhance the high-frequency content of the signal to compensate for the loss during the acquisition process; ii) due to the short-term stability of the voice signal, the audio signal is decomposed into some small fragments, and then the Hanning window is added to these fragments; iii) the amplitude-frequency characteristics of the above speech segment sequence are obtained after the fast Fourier transform (FFT) processing, and then the modulus is taken to obtain the transformed linear spectrum; iv) a linear spectrum is converted into a spectrogram by the logarithmic transformation; v) color mapping is used to add details and features to the image and resize the image for CNN recognition.

2.3. CNN Architecture

As shown in Figure 3, CNN can be functionally divided into two parts, feature extraction and classification. The feature extraction is composed of two-dimensional convolutional layers and pooling layers stacked in a certain order. The number of filters in each convolutional layer corresponds to the number of features currently expected to be extracted from the image, and the pooling layer abstracts it to a higher level while reducing the size of the image. The classification is the combination of the full connection layer and the activation function, which maps the extracted features to the sample space, and nonlinear calculations are utilized to improve the expressiveness of the network by the activation function.

When building a CNN, there are numerous hyperparameters that need to be determined. The hyperparameter types that this study focuses on are as follows:

• The number of filters per convolution layer: this parameter determines the abstraction ability of the network and the number of features to be extracted eventually;
• The number of neurons in the fully connected layer: too few neurons may result in failure to train a model that meets the requirements, while numerous neurons may lead to overfitting;
• Learning rate: if the learning rate is too low, it is easy for the model to fall into a local optimum, and if it is too high, it is easy to miss the global optimum and fail to complete the training.

It is not simple to build and train CNN with good performance, which requires a lot of time and cost. Therefore, there is an urgent need for a method that can automatically build CNN in various tasks, which can help people determine various kinds of super parameters.

3. Proposed Approaches

3.1. Resnet

CNNs significantly outperform other traditional classifier models in the field of image recognition. However, as the depth of the model increases, problems such as gradient disappearance and gradient explosion will degrade the model and make the performance of the model worse. The fundamental reason is that the parameters of the lower convolution kernel are difficult to be effectively adjusted in the later stage of training. In order to solve the above problems, some scholars proposed a deep residual network (ResNet). ResNet directly transmits low-level features to higher levels by introducing residual blocks, which makes it more capable of feature extraction and representation. This connection mode will not be affected by the depth of the network.

As shown in Figure 4, Resnet contains many residual blocks. There are two parts of each block: short cut and convolution calculation. Short cut maps the image of the previous layer directly to the next layer. The convolution calculation includes several convolution kernels, batch normalization (BN) and activation functions. BN aims to normalize the same batch of data into standard normal distribution, which can make the model converge quickly.

The calculation process of the whole residual can be described as:

$${\mathrm{y}}_{\mathrm{i}}=\mathrm{F}({\mathrm{x}}_{\mathrm{i}},{\mathrm{W}}_{\mathrm{i}})+{\mathrm{H}(\mathrm{x}}_{\mathrm{i}})$$

(1)

where F represents the convolution calculation of the ${\mathrm{i}}^{\mathrm{th}}$ block, ${\mathrm{W}}_{\mathrm{i}}$ represents weight and offset parameters, H represents the short cut, and ${\mathrm{x}}_{\mathrm{i}}$, ${\mathrm{y}}_{\mathrm{i}}$ are the input and output, respectively.

A large number of different models of ResNet are developed by stacking different number of residual blocks, ResNet18/43 and ResNet50/101/152. The optimization method proposed in this study is applied to ResNet18.

3.2. DBO

DBO that was inspired by the biological behavior process of dung beetles, is a swarm intelligent optimization algorithm with strong optimization ability and fast convergence speed. In a beetle population, differences in food abundance at foraging locations among individuals lead to differences in fitness. By using the fitness ranking as a division criterion, these individuals were divided into four different roles in descending order of fitness: ball-rolling beetles, brood balls, small beetles, and thief beetles. Rolling ball beetles are individuals with high fitness. Their goal is to move the food ball to a place suitable for breeding. When the female beetle finds the food ball of the rolling ball beetle, it will move it for a short distance and lay eggs on it, which is called the brood ball. After hatching, the baby beetle will look for food around the brood ball. Thief beetles will snatch the food balls of other beetles. Each role corresponds to a specific adjustment strategy of position. After each foraging, the fitness ranking of each individual redefines their roles in the next foraging. Based on fitness ranking, the mechanism of role division helps individuals optimize strategies of foraging behavior according to their fitness level, so as to find a suitable location for the survival and reproduction of the population. The location update formula of ball-rolling beetles is given as:

$${\mathrm{x}}_{\mathrm{i},\mathrm{j}}(\mathrm{t}+1)=\left\{\begin{array}{c}{\mathrm{x}}_{\mathrm{i},\mathrm{j}}\left(\mathrm{t}\right)+\mathsf{\alpha }\times \mathrm{k}\times {\mathrm{x}}_{\mathrm{i},\mathrm{j}}(\mathrm{t}-1)+\mathrm{b}\times ∆\mathrm{x},\mathrm{R}\text{<}0.9\\ {\mathrm{x}}_{\mathrm{i},\mathrm{j}}\left(\mathrm{t}\right)+\mathrm{t}\mathrm{a}\mathrm{n}\left(\mathsf{\theta }\right)\times \left|{\mathrm{x}}_{\mathrm{i},\mathrm{j}}\left(\mathrm{t}\right)-{\mathrm{x}}_{\mathrm{i},\mathrm{j}}(\mathrm{t}-1)\right|,\mathrm{R}\text{≥}0.9\end{array}\right.$$

(2)

$$∆\mathrm{x}=\left|{\mathrm{x}}_{\mathrm{i},\mathrm{j}}\left(\mathrm{t}\right)-{\mathrm{X}}^{\mathrm{w}}\right|$$

(3)

where ${\mathrm{x}}_{\mathrm{i},\mathrm{j}}\left(\mathrm{t}\right)$ represents the position of the ${\mathrm{j}}^{\mathrm{th}}$ dimension of the ${\mathrm{i}}^{\mathrm{th}}$ beetle in the ${\mathrm{t}}^{\mathrm{th}}$ iteration, $\mathsf{\alpha }$ is a natural coefficient which is assigned -1 or 1, $\mathrm{k}\text{∈}(0,0.2]$ denotes a constant value which indicates the deflection coefficient, b is a constant from 0 to 1, $\mathsf{\theta }\text{∈}[0,\mathsf{\pi }]$ is the deflection angle, R is a random number belonging to (0,1), when $\mathrm{R}\text{≥}0.9$, beetle has encountered obstacles and needs to adjust its direction, $∆\mathrm{x}$ indicates the changes of light intensity ,${\mathrm{X}}^{\mathrm{w}}$ represents the current global worst position.

The position update formula of brood balls can be expressed as:

$$\left\{\begin{array}{c}{\mathrm{Lb}}^{*}=\mathrm{m}\mathrm{a}\mathrm{x}({\mathrm{X}}^{*}\times (1-\mathrm{R}),\mathrm{L}\mathrm{b})\\ {\mathrm{Ub}}^{*}=\mathrm{m}\mathrm{i}\mathrm{n}({\mathrm{X}}^{*}\times (1+\mathrm{R}),\mathrm{U}\mathrm{b})\\ {\mathrm{x}}_{\mathrm{i},\mathrm{j}}(\mathrm{t}+1)={\mathrm{X}}^{*}+{\mathrm{b}}_1\times \left({\mathrm{x}}_{\mathrm{i},\mathrm{j}}\right(\mathrm{t})-{\mathrm{Lb}}^{*})+{\mathrm{b}}_2\times \left({\mathrm{x}}_{\mathrm{i},\mathrm{j}}\right(\mathrm{t})-{\mathrm{Ub}}^{*})\end{array}\right.$$

(4)

where ${\mathrm{X}}^{*}$ denotes the current local best position, Lb and Ub represent the upper and lower bounds of the search area respectively, ${\mathrm{Lb}}^{*}$ and ${\mathrm{Ub}}^{*}$ represent the upper and lower bounds of the spawning area, respectively. ${\mathrm{b}}_1$ and ${\mathrm{b}}_2$ represent two independent random D-dimensional vectors belongs to (0,1), D is the dimension of the optimization problem.

The position update formula of small beetles can be expressed as:

$$\left\{\begin{array}{c}{\mathrm{Lb}}^{\mathrm{b}}=\mathrm{m}\mathrm{a}\mathrm{x}({\mathrm{X}}^{\mathrm{b}}\times (1-\mathrm{R}),\mathrm{L}\mathrm{b})\\ {\mathrm{Ub}}^{\mathrm{b}}=\mathrm{m}\mathrm{i}\mathrm{n}({\mathrm{X}}^{\mathrm{b}}\times (1+\mathrm{R}),\mathrm{U}\mathrm{b})\\ {\mathrm{x}}_{\mathrm{i},\mathrm{j}}(\mathrm{t}+1)={\mathrm{x}}_{\mathrm{i},\mathrm{j}}\left(\mathrm{t}\right)+{\mathrm{C}}_1\times \left({\mathrm{x}}_{\mathrm{i},\mathrm{j}}\right(\mathrm{t})-{\mathrm{Lb}}^{\mathrm{b}})+{\mathrm{C}}_2\times \left({\mathrm{x}}_{\mathrm{i},\mathrm{j}}\right(\mathrm{t})-{\mathrm{Ub}}^{\mathrm{b}})\end{array}\right.$$

(5)

where ${\mathrm{X}}^{\mathrm{b}}$ denotes the global best position, ${\mathrm{Lb}}^{\mathrm{b}}$ and ${\mathrm{Ub}}^{\mathrm{b}}$ represent the upper and lower bounds of the optimal foraging area respectively. ${\mathrm{C}}_1$ belongs to (0,1), which follows normally distributed, ${\mathrm{C}}_2$ represent a random D-dimensional vector.

The location update formula of thief beetles is given as:

$${\mathrm{x}}_{\mathrm{i},\mathrm{j}}(\mathrm{t}+1)={\mathrm{X}}^{\mathrm{b}}+\mathrm{S}\times \mathrm{g}\times (\left|{\mathrm{x}}_{\mathrm{i},\mathrm{j}}\left(\mathrm{t}\right)-{\mathrm{X}}^{*}\right|+\left|{\mathrm{x}}_{\mathrm{i},\mathrm{j}}\left(\mathrm{t}\right)-{\mathrm{X}}^{\mathrm{b}}\right|)$$

(6)

where S represent a constant value, g is a random D-dimensional vector that follows normally distributed.

3.3. Optimization Process

ResNet18 has five residual blocks. We need to determine the number of output channels or convolution kernel for the five residual blocks. In order to improve the expression ability of the network, an additional hidden layer is added to the full connection layer, thus the number of neurons with two hidden layers needs to be determined. Finally, we need to determine the learning rate in the training process. All hyperparameters and their ranges that need to be optimized for this model are shown in Table 1. The dimension that beetles need to search is 8. The calculation method of beetle fitness is described as, decoding the position coordinates into hyperparameters, and then using these hyperparameters to establish ResNet18. After a training, the test set is led into testing, and the obtained accuracy is the fitness of the beetle.

The process of speaker recognition using DBO-CNN is shown in Figure 5, which can be divided into five steps:

• Step I, divide the data set into a training set and a test set in a ratio of about 8:2;
• Step II, initialize the population and divide it into different beetle roles according to the fitness ranking. Among them, ball-rolling beetles accounts for 6/30, brood balls accounts for 6/30, small beetles accounts for 7/30 and thief beetles accounts for 11/30;
• Step III, the beetles search for the optimal hyperparameter group according to their own strategies of position adjustment;
• Step IV, build a convolutional neural network for speaker recognition by using the optimal hyperparameters;
• Step V, evaluate the model on the test set after training.

4. Experiments and Results

4.1. Dataset

The data set used in this study is the Chinese Mandarin open source voice database from Hill Shell. The database contains voice data covering voice control, unmanned driving, industrial production and other fields. In order to increase the richness of the database, the database is built by recruiting 400 people from different accent regions in China to participate in the recording.

The voice data of 50 people with 300 to 400 voice messages for each person are randomly selected. These voices are cut into 2.5s long segments, and each segment contains 40000 data points. When using FFT to make spectrogram, set the length of Hanning window to 256, and the moving width of each frame to 128, which is half of the window length. Finally, the obtained spectrograms are normalized to 128 × 128. The data sets contain 17265 images. The number of images is 14605 for training sets and 3200 for test sets.

4.2. Hyperparameters Optimization

DBO provides an efficient guide for determining the hyperparameters of CNN, however it also has some population-related constants to be set. In this study, the constants are used as the same with the author of DBO: k = 0.1, b = 0.3, S = 0.5. In order to evaluate the performance of DBO-CNN, we compared it with PSO-CNN, Sparrow Algorithm Optimized CNN (SSA-CNN), and CNN built by experience. In the three optimization processes, the population and search times are set to 50 and 30. After iteration, the hyperparameters found by the three algorithms are shown in Table 2, and the hyperparameters chosen empirically are also listed.

4.3. Model Training

The CNNs using the above four sets of hyperparameters are constructed, and the produced spectrograms are imported into them for training. The loss function is SoftMaxCrossEntropy. The number of iterations is set to 50.

As shown in Figure 6, the loss of traditional CNN does not decrease below 0.01 after 50 iterations, which is the worst in the model. DBO-CNN performs the best by reducing losses to below 0.01 after only 5 iterations, PSO-CNN spends 25 times, and SSA-CNN spends 21 times, which indicates that DBO significantly improves the training speed of the model. The evaluation results of the four models on the test set are shown in Figure 7. After the optimization of the intelligent algorithm, the accuracy of the model has been significantly improved. Among them, DBO-CNN has the greatest improvement with the accuracy rate of 98.34%, followed by SSA-CNN with the accuracy rate of 97.71%, and finally PSO-CNN with the accuracy rate of 95.62%. It is demonstrated that DBO is indeed superior to traditional intelligent algorithms such as PSO.

4.4. Noise Resistance Test

Due to environmental noise interference or performance limitations of recording equipment, the effects of the audio are always unsatisfactory. In order to test the anti-interference ability of the model in this environment, some white noise is added to the previous test set data. We tested the performance of the model by different signal-to-noise ratios. As shown in Figure 8, as the proportion of noise increases, the recognition accuracy of all models decreases. The anti-noise ability of the two newer meta heuristic algorithms is basically the same, and the accuracy is still around 80% in a signal-to-noise ratio environment of 30 dB. Even in a harsh environment of 20 dB, the accuracy of both algorithms is above 50%, which is acceptable. However, the accuracy of particle algorithms and traditional CNN in noisy environments has significantly decreased, and their anti-noise performance is significantly weaker than the new meta heuristic algorithm.

5. Conclusions and Future Work

A DBO-CNN algorithm is proposed for speaker recognition in this study, and compared with the other three algorithms on a dataset of 50 people. After optimization, the performance of all models has improved. Compared to other algorithms, DBO-CNN has better optimization ability. It not only has the fastest convergence speed, but also has the highest accuracy, reaching 98.34%. In terms of anti-interference, DBO-CNN and SSA-CNN have similar performance, and their recognition rates are both above 50% in harsh environments of 20 dB. Based on the results, it demonstrates that speaker recognition has been greatly improved by the DBO-CNN compared with other optimization algorithms, and also illustrates that DBO has the enormous potential in optimizing CNN.

The future work is to expand the scale of the dataset and deploy the model on embedded devices, and a synthetic approach to measure the performance of the indexes will be used to speaker recognition [26]. As well as, swarm intelligence is a promising and challenging science subbranch. In 2021, the Nobel Physics Award is partially grant to the work about the swarm behavior and intelligence. Meanwhile, new developments from multiple fields are deepening broadening the cognition on swarm intelligence. For swarm intelligent method based on bionic computing, some recent biological achievements such as adaptive mutability [27], epigenetics [28] are sure to improve the performance of speaker recognition. These swarm optimization algorithms will be attempted for further research.