CSA-SA-CRTNN: A Dual-Stream Adaptive Convolutional Cyclic Hybrid Network Combining Attention Mechanisms for EEG Emotion Recognition

1. Introduction

Emotion is a complex state influenced by various feelings, behaviors, and thoughts when the human brain is stimulated[1]. It plays a crucial role in shaping our mental states and guiding behavioral decisions. The advancement of emotion recognition systems has become a prominent focus across disciplines such as psychology, artificial intelligence, computer vision, consumer behavior analysis, and medical treatment[2].

Understanding human emotions is a complex task that involves analyzing both non-physiological and physiological signals. Facial expressions, voice tone, and text messages are some of the non-physiological signals we use to decipher someone’s emotions[3,4]. On the other hand, physiological signals like brain activity, muscle movements, and skin responses provide a deeper insight into our emotional state[5]. People can mask their true emotions, making it tricky to accurately predict how their emotions solely based on non-physiological signals they emit. In contrast, physiological signals like EEG offer a more reliable glimpse into someone’s emotions. EEG stands out as a star performer in capturing emotions in real-time with high precision and at a reasonable cost. It has become a go-to method for reading emotions without invading someone’s privacy[6,7,8].

In the early stages of EEG emotion recognition research, scholars focused on traditional EEG features and used algorithms like Support Vector Machines(SVM) and K-nearest neighbors(KNN)[9,10]. However, manual feature selection in machine learning is time-consuming and laborious, and the accuracy is not ideal. With the advancement of technology, deep learning has become a new solution for EEG emotion recognition, demonstrating superior performance in identifying human emotions[11,12,13]. Various models including convolutional neural networks(CNN)[14], capsule networks(CapsNet)[15], and long short-term memory networks(LSTM) [16]have been developed by researchers to comprehend human emotions through EEG signals. For instance, Huang et al. [17] devised an S-EEGNet model achieving an accuracy of 89.91% and 88.31% on the DEAP dataset. In a separate study, Zheng and colleagues[18] introduced a CNNFF architecture with an average classification accuracy of 93.61% for valence and 94.04% for arousal respectively. Although the above studies have achieved good results, they do not provide further results. Zheng et al. [19] trained a DBN to analyze differential entropy features across multiple EEG channels, comparing deep and shallow models using the SEED dataset with an average accuracy of 86.08%, 83.99%, 82.70%, and 72.60% in DBN, SVM, LR, and KNN, respectively. Cui and his team [20] developed a new model called RACNN for classifying emotions in EEG data, achieving over 95% identification accuracy on both valence and arousal classification tasks using DEAP and DREAMER datasets. However, these methods only use a single network model, and the recognition accuracy needs to be improved. To further improve the accuracy of EEG emotion recognition, researchers have been experimenting with hybrid network models that combine techniques like CNN, RNN, LSTM, GCNN, and attention mechanisms in innovative ways. Zong et al. [21]introduced the FCAN-XGBoost algorithm which blends FCAN and XGBoost algorithms for processing DEAP and dream data sets with accuracies of 95.26% and 94.05%, respectively. Chakravarthi et al. [22] developed an automated system using a combination of CNN and LSTM based on the ResNet-152 algorithm achieving an impressive accuracy of 98%.

The aforementioned methods have yielded a series of satisfactory results. However, the sheer volume of irrelevant information present in EEG signals means that analyzing them requires a lot of computational power[23]. Cutting through this noise to uncover the true emotional cues is a challenge that researchers are striving to overcome. Also, the current methods for extracting features often result in a significant loss of important information [22], presenting a pressing issue that needs to be addressed promptly.

To address the aforementioned issues, a two-flow adaptive convolutional cyclic mixing network (CSA-SA-CRTNN) combined with an attention mechanism is proposed for emotion recognition in EEG. The model initially utilizes the CSAM module to assign weights to EEG channels and then inputs the EEG signals into the two-stream adaptive convolutional circulation network to extract local spatiotemporal features. Subsequently, the local features extracted from the two-stream network are spliced and inputted into the time-series Convolutional network combined with a multi-head self-attention mechanism (MHSA-TCN) to extract global information. Finally, the feature vectors are fed into the fully connected layer, and a softmax classifier is employed for emotion classification.

In summary, the contributions of this paper are as follows:

A novel approach has been developed for attending to various brain signals, referred to as the channel-wise attention module (CSAM). This module function selectively highlights the most pertinent aspects of the signal while disregarding extraneous information that is inconsequential about emotions. This facilitates more efficient cognitive processing, conserving time and energy resources while ensuring focused attention on critical informational components.

A multi-head self-attention mechanism (MHSA) has been developed and integrated with CRNN and TCN to effectively capture both local and global key features in emotional data. This innovative technique significantly enhances our model’s capacity for accurate emotion recognition, while also reducing computational complexity.

An adaptive convolutional recurrent network (SA-CRNN) is proposed for extracting local EEG information. This network can adaptively match appropriate convolutional stride and pooling parameters for the provided convolutional kernel size, effectively addressing the issue of information loss and improving the final recognition performance.

A dual-stream adaptive convolutional recurrent hybrid network with an attention mechanism (CSA-SA-CRTNN) is designed for emotion recognition.

2. Related Work

This section introduces the relevant knowledge and research in the field of EEG emotion recognition from three aspects: emotion model, feature fusion and mixing model, and attention mechanism.

2.1. Emotional Model

Researchers studying emotion recognition use two models to describe emotions: the Discrete Model[24,25] and the Dimensional Model[26,27]. The Discrete Model sees emotions as distinct points and believes that emotions are made up of basic emotions. Although the Discrete Model is straightforward to understand, it falls short in describing the complexity of human emotions. The Dimensional Model tries to understand human emotions by mapping them in a two- or three-dimensional space. The Valence-Arousal Model is the most classic dimensional model, which represents different emotions as points on a two-dimensional plane of Valence and Arousal. Other emotions are made up of different combinations of Valence and Arousal[28]. The Dimensional Emotional Model is continuous and can express emotions across a wide range.

2.2. Feature Fusion and Hybrid Models

Recent studies[29,30] have revealed that combining multiple EEG features leads to superior results in detecting emotions compared to using just one feature. In the field of emotion recognition research using deep learning methods, feature fusion is often closely associated with the development of hybrid models[31,32,33]. For example, Feng et al.[34] proposed a new model called ST-GCLSTM, which combines SGCN and attention-enhanced bidirectional LSTM. This model can obtain the temporal patterns of brain signals and capture the different intensities of these signals at different times, achieving a recognition accuracy of 95.52%. Li et al.[35] proposed a BLSTM network model based on multimodal attention, which can learn the optimal temporal features of EEG signals with a recognition accuracy of 81.3%. Overall, hybrid network models incorporating CNN, RNN, LSTM, GCNN, and attention mechanisms tend to yield good results in emotion recognition tasks.

2.3. Attention Mechanism

Humans have the remarkable ability to focus on specific elements of a scene while filtering out other distractions – a phenomenon known as the attention mechanism. In recent years, the integration of neural networks with attention mechanisms has revolutionized the field of emotion recognition, particularly in EEG-based applications[34,36]. By selectively filtering and enhancing crucial components of the input data, these attention mechanisms have significantly boosted the accuracy of emotion recognition systems. Researchers like Zhang et al. [36]have developed innovative deep-learning models that leverage attention mechanisms to extract key features from EEG signals, yielding impressive classification results in emotion recognition tasks, and the classification accuracy reaches 92.47%. Similarly, Lew et al. [37]have explored the use of attention mechanisms in conjunction with adversarial networks to capture complex spatiotemporal relationships within EEG data, ultimately mitigating domain shift issues, and the classification accuracy reaches 98.15%.

3. Methodology

The proposed CSA-SA-CRTNN model’s framework and process are displayed in Figure 1, which consists of four modules: CSAM, SA-CRNN, MHSA-CRNN, and MHSA-TCN. The CSAM assigns weights to EEG channels, while the dual-stream convolutional recurrent framework composed of SA-CRNN and MHSA-CRNN extracts local EEG features. The MHSA-TCN module extracts global EEG features, and the feature vectors are fed into the softmax classifier for emotion classification. In the following chapters, we will explain the model’s framework and process in detail.

3.1. CSAM

Inspired by[38], a new channel attention mechanism (CSAM) has been introduced. It receives preprocessed EEG signals, allocates attention to channel dimensions, and enhances the effect of attention mechanisms on improving model performance. The proposed CSAM module can be found in Figure 2.

The module comprises a global pooling layer, linear layer, convolution layer, and activation layer. It allocates weights to the channel dimension, enhancing important information and minimizing unimportant information, thereby improving the ability to recognize and classify emotions while reducing computing costs. Table 1 shows the specific parameter settings for this module.

3.2. MHSA-CRNN

MHSA-CRNN comprises a Convolutional Recurrent Neural Network (CRNN) and a Multi-Head Self-Attention (MHSA) mechanism. It is capable of effectively extracting critical spatio-temporal features at a local level.

3.2.1. CRNN

CRNN is the basic framework of the MHSA-CRNN module. To reduce the computational complexity of the model, a three-layer CNN and a two-layer LSTM are used here.

This module first uses CNN to extract spatial dimension information and then adopts LSTM to extract the temporal dependency of the signal, as shown in equation (1):

$$v_{st}=LSTM\left(Conv\left(x\right)\right)$$

(1)

Here, $x$ is the input signal, and$v_{st}$is the feature vector output through the CRNN network.

CNN consists of different layers, including a convolutional layer, an activation layer using exponential linear units (ELUs), and a maximum pooling layer. In the activation after convolution, the ELU function is preferred over the commonly used ReLU function. The kernel size and step size of the convolution operation are set to (32,40) and 1, respectively, using the same Settings as in the literature [38]. The pooling layer uses a kernel size of (1,75) with a stride length of 10. Next comes the LSTM model, which consists of two stacked layers. Each layer of the LSTM receives hidden states as input from the previous layer to learn more advanced feature representations in the data. While adding more layers can improve a model’s performance, it also increases training time and computational requirements. The input dimension of the LSTM unit is set to 80 and the hidden layer dimension to 64.

3.2.2. Multi-Headed Self-Attention (MHSA)

To better understand contextual information and further improve model performance, we developed a novel multi-head self-attention mechanism, MHSA, and integrated it into CRNN to automatically capture dependencies among sequences and enhance the representation ability of the network. The structure of MHSA is shown in Figure 3.

The MHSA model utilizes multiple sets of query, key, and value mappings to concurrently compute several attention representations. These representations are then linearly stacked to produce the final multi-head attention representation. This approach allows the model to focus on different aspects of the input sequence, thereby enhancing the model’s ability to generalize.

In this paper, the Head of MHSA is equal to 8, and the overall calculation process is as follows:

$$SA\left(Q,K,V\right)=softmax\left({\displaystyle \frac{{QK}^T}{\sqrt{d_k}}}\right)V$$

(2)

$$h_i=SA(Q{W}_i^Q,K{W}_i^K,V{W}_i^V)$$

(3)

$$MHSA\left(Q,K,V\right)=Concat(h_1,\dots \dots ，h_8)W^O$$

(4)

Here, $SA\left(Q,K,V\right)$represents the self-attention output, $d_k$ is the scaling factor, $W_i^Q,$ $W_i^K,W_i^V$ are the query, key and value transformation matrix of the$i\_th$ head respectively, $h_i$represents the$i\_th$ head attention output, $MHSA\left(Q,K,V\right)$is the final output of MHSA, and $W^O$ represents the transformation matrix of the output.

The self-attention mechanism acts as the central processing unit of the Multi-Head Attention (MHSA) algorithm, ensuring seamless operation. Initially, input features undergo linear transformations to yield query (Q), key (K), and value (V) features. These transformed features are then partitioned into three groups: Q, K, and V. The attention scores are computed by multiplying the query and key together and incorporating a scaling factor (Scale). Subsequently, these scores are adjusted based on an input mask to filter out irrelevant information. Following this, the attention scores undergo a Softmax function for normalization purposes. Ultimately, the output is generated by combining values with attention scores. This output then undergoes further transformation to revert to its original form, along with some Dropout processing for additional fine-tuning.

3.3. SA-CRNN

The structure of the Self-Adaptive Convolutional Recurrent Neural Network (SA-CRNN) is similar to the CRNN described in Section 3.2.1, with the difference being that its corresponding convolutional stride, pooling kernel size, and pooling stride can be adaptively adjusted based on the given convolutional kernel. This structural design aims to fully extract EEG emotional information and avoid the loss of feature information due to fixed parameter settings. The calculation methods for the convolutional stride and pooling layer parameters of the convolution operation in this module are as follows:

$$s_c={\displaystyle \frac{k_c[-1]}{10}}$$

(5)

$$k_p=(1,{\displaystyle \frac{k_c\left[-1\right]}{2}})$$

(6)

$$s_p={\displaystyle \frac{k_c[-1]}{10}}$$

(7)

Where $s_c$ is the convolutional stride, $k_c$[-1] is the element of the last dimension of the convolutional kernel, $k_p$ is the size of the pooling kernel, and $s_p$ is the pooling stride. In this module, the input dimension of the LSTM unit is 720, and the hidden layer dimension is 64.

3.4. MHSA-TCN

A common way to understand time-dependent problems is to use recurrent neural networks (RNNS). However, RNNS also has its disadvantages, such as difficulty in parallelizing calculations, low processing efficiency, and the risk of gradient explosion and gradient disappearance. To address these challenges, Bai S and their team [39] introduced temporal convolutional networks (TCN). TCN uses one-dimensional convolution, extended convolution, causal convolution, and residual convolution to maximize the ability of convolution to analyze time series. This approach is superior to LSTM (a type of RNN) in many different tasks.

TCN is a technology that enables efficient parallel computing and is well-suited for processing large amounts of data. Its benefits include the ability to extract features of different scales by stacking multiple convolutional layers, accurately capturing local dependencies in sequence data by increasing the receptive field of the convolutional kernel and avoiding issues with gradient vanishing and explosion, etc. The main reason why TCN is so useful is that it significantly increases the receptive field[40]. The calculation method of the receptive field for the$i\_th$ layer is as follows:

$$F_i=\left(k-1\right)\times (\sum _{i=0}^{L-1}{d}_i)+k$$

(8)

Where $F_i$is the receptive field of the $i\_th$ layer, $k$ is the size of the convolutional kernel, $L$ is the number of convolutional layers in each residual block, and $d_i$ is the dilation factor of the $i\_th$ layer.

To make the network work better, we’ve added a complex feature called multi-head self-attention mechanism (MHSA) that we talked about in Section 3.2.2 to TCN. Look at Figure 4 to see how MHSA-TCN is structured.

This network takes the local features it already got from MHSA-CRNN and SA-CRNN to find global EEG features. In this experiment, TCN consists of four hidden layers, with each hidden layer having 25 channels, the kernel size =2, and dilation factors of d= [1,2,4,8].

4. Experiments

4.1. Datasets

The DEAP dataset [41], also known as the Database for Emotion Analysis using Physiological Signals, is a comprehensive collection of data utilized by researchers to gain a deeper understanding of emotions. This dataset was compiled by researchers at the Queen Mary University of London and encompasses various physiological signals from the body and brain of 32 individuals, consisting of an equal number of male and female participants with an average age of 26 years. Additionally, videos capturing the facial expressions of 22 participants were recorded to observe their reactions. The research involved having the participants watch short music videos and evaluate their emotional responses. Parameters such as arousal level, valence, liking or disliking, dominance, and familiarity were measured using the Self-Assessment Manikin (SAM) scale. Notably, each participant viewed 40 different videos in real-time to ensure the accuracy of data collection. The ratings were based on a two-dimensional arousal-valence model rated on a scale from 1 to 9. The DEAP dataset includes 32 channels of EEG signals and 8 channels of peripheral physiological signals, all recorded at a rate of 512 Hz.

For this experiment, we decided to focus solely on the EEG signals for emotion recognition, ditching the peripheral data. The experimental data within the DEAP dataset is split into 32 separate files, each representing an individual experimental subject. Within each participant’s file, you can find two arrays of data, which, after some basic preprocessing are shown in Table 2.

4.2. Data Preprocessing and Emotional Label Processing

Data preprocessing is a necessary process for EEG-based emotion recognition. In this paper, we first downsampled the signals to 128Hz. Then, the 63-second data was segmented into 1-second windows, and the arithmetic mean of the first three 1-second baseline segments was calculated. Subsequently, the mean baseline was subtracted from each of the remaining 60 segments to remove the baseline. The calculation process is as follows:

$$s_{baseline}={\displaystyle \frac{\sum _{i=1}^3{x}_i}{3}}$$

(9)

$$s=s_{original}-s_{baseline}$$

(10)

Where $s_{baseline}$ represents the average baseline signal, $s_{original}$ is the original EEG signal, and $s$ represents the signal after baseline removal. Finally, the 60 1-second segments after baseline removal are rearranged in chronological order into 3-second segments as input for the model.

Similar to the literature [42], this experiment adopts two schemes based on the valence dimension and arousal dimension for the processing of emotional labels. The first uses 5 as the threshold, arousal and valence SAM scores 1-5 are classified as low scores, 6-9 as high scores, low (LA) and high (HA) in the arousal dimension, and low (LV) and high (HV) in the valence dimension. In the second scheme, 4 and 6 were used as thresholds. Arousal and valence SAM scores of 1-4 were classified as low scores, 5-6 as medium scores, 7-9 as high scores, low (LA), medium arousal (MA), and high (HA) in arousal dimension, and low (LV), medium valence (MV), and high (HV) in valence dimension. The specific division of the two schemes is shown in Table 3.

4.3. Experimental Setup

The DEAP public dataset was utilized to train and validate the model on a powerful GeForce RTX 2080 Ti GPU using the Pytorch framework in the research. The model was optimized using Adam as the optimizer, with a learning rate of 0.001, a batch size of 128, and a cross-entropy loss function. The experiment ran for 30 epochs to fine-tune the model. The data was split into a training set and a test set in an 8:2 ratio to ensure robust evaluation. To further evaluate our model’s performance, a rigorous 10-fold cross-validation approach was employed.

5. Results and Discussion

5.1. Convergence of the Model

Figure 5 illustrates the fluctuating pattern of accuracy in binary and triple classification across the DEAP dataset during training. As the process unfolds, the accuracy experiences a steady increase before eventually stabilizing at a consistent level, highlighting the model’s strong convergence after a certain number of iterations. Notably, the data suggests that the model reaches its peak performance at around 30 iterations, demonstrating remarkable accuracy while utilizing minimal resources. Consequently, for this experiment, an epoch of 30 is considered ideal for optimal results.

5.2. Overall Performance

In Figure 5, it is evident that the CSA-SA-CRTNN model is reliable and Effective when used on the DEAP dataset. Moving on to Figure 6, we can observe the accuracy of the model on individual subjects within the DEAP dataset, especially when the epoch is finely tuned to the ideal 30.

In the binary experiment, the average accuracy for the valence dimension reached 99.15%, with the exceptions of s05 and s22, while for the arousal dimension, it reached 99.26%, except for s02 and s22. In the ternary classification experiment, the average accuracy of the valence dimension was 98.05% (except for s05 and s22), and in the arousal dimension, it was 97.69% (except for s02 and s22). It is noteworthy that subject No. 22 exhibited consistent characteristics in both two-category and three-category experiments, with significantly lower recognition accuracy compared to other subjects - achieving 90.27% and 93.75% in two-category valence and arousal, as well as 87.40% in three-category valence; whereas arousal achieved only 80.75% in three-category. This could be attributed to the possibility that the participants were distracted during the experiment or due to their subjective biases in scoring.

The confusion matrix is shown in Figure 7. The confusion matrix shows the excellent classification ability of CSA-SA-CRTNN for each emotion category.

5.3. Comparison with Related Research

To further evaluate the effectiveness of the model, we compared CSA-SA-CRTNN with previous studies on EEG-based emotional recognition using the DEAP dataset. Our comparison methods are outlined below:

• Tsception (Yi et al，2021)[43]: Tsception is a cutting-edge Deep learning model that can analyze EEG signals in a way that mimics the complexities of the human brain. It uses a dynamic temporal layer that can adapt to different time scales and frequencies present in EEG signals. Additionally, Tsception has an asymmetric spatial layer that takes into account the asymmetrical neural activations associated with emotional responses. This layer can pick up on subtle differences in how the brain processes information, allowing it to create a more accurate representation of the data it receives. Finally, Tsception has a fusion layer that brings together all the information.
• DCRNN (Li et al，2022)[44]: DCRNN sifts through EEG signals, using a deep sparse autoencoder network (DSAE). Its mission is to cut through the noise and reconstruct the underlying features of EEG signals. By teaming up a convolutional neural network (CNN) with long short-term memory (LSTM), DCRNN digs deep into the connections between different parts of the brain and integrates contextual information from EEG signal frames.
• 4D-CRNN(Shen et al,2020) [45]: The 4D-CRNN is a groundbreaking technique that takes complex features from various sources and transforms them into an intricate, four-dimensional framework. It operates by leveraging the powers of both the convolutional neural network (CNN) and long short-term memory (LSTM) network. The CNN delves into the frequencies and spatial intricacies of each slice within the 4D input, while the LSTM uncovers the temporal relationships hidden within the CNN’s findings. The fusion of these two powerful networks results in a model that can truly understand and make sense of the intricate connections within the data.
• BiSMSM(Li et al，2023) [46]: BiSMSM is a complex framework that delves into the intricacies of time and space. It consists of two streams, one focusing on spatial aspects and the other on temporal aspects. Designed to decipher information from various viewpoints including time, space, locality, and globality, this framework is made up of interconnected modules. The spatial and temporal streams mirror each other in structure, featuring a module centered around a multilayer perceptron (MLP). This module is key in unraveling both intra- and inter-channel insights from specific regions. Additionally, a self-attention mechanism module is in place to extract the global signal correlations.
• AP-CapsNet(Liu et al，2023) [47]: The AP-CapsNet method combines coordinated attention to help understand where things are about each other in the input data, and then transforms this information into a complex space to identify emotions. To achieve this, a pre-trained model is utilized to extract features, and a double-layer capsule network is built for in-depth analysis.
• ATDD-LSTM(Du et al，2021)[48]: The ATDD-LSTM model functions as a mechanism to focus on specific segments of information produced by the LSTM to gain a deeper understanding of emotions. Additionally, it incorporates a domain discriminator to ensure consistency of information across diverse contexts.
• ICaps-ResLSTM(Fan et al，2024)[49]: ICaps-ResLSTM can understand the different patterns and locations within EEG data using capsule networks. And get this - it also has a ResLSTM module that helps it learn even more detailed and complex features by connecting different modules in time and space. This means it can pick up on even the most subtle differences in EEG data, making it good at distinguishing between different brain activities.
• CADD-DCCNN(Li et al，2024)[50]: To gain a deeper understanding of the emotions captured in EEG signals, utilizing DE features obtained through STFT. Each DE feature channel provides a unique perspective and employs an attention mechanism to extract key emotional data across the EEG timeline. CADD-DCCNN explores intricate interconnections among diverse timeframes, enhancing the comprehension of nonlinear relationships. Additionally, CADD-DCCNN incorporates a domain discriminator to ensure consistency in data representation and bridge gaps among different sources.

Table 4 presents a comparison of the outcomes yielded by these approaches. As shown in the table, the accuracy of CSA-SA-CRTNN on the DEAP dataset is 99.26% and 99.15%, respectively. Compared with existing methods, our model has achieved more competitive results.

5.4. Ablation Study

The proposed model consists of four modules: the CSAM module, the SA-CRNN module, the MHSA-CRNN module, and the MHSA-TCN module. To evaluate the contribution of each module in the model, we conducted four types of ablation experiments on the DEAP dataset.

Specifically, four ablation models were constructed: SA-CRTNN, CSA-CRTNN, CA-SA-CRTNN, and CSA-SA-CRNN. Among them, SA-CRTNN is the model that abandons the CSAM module. CSA-CRTNN is the model without the SA-CRNN module. CA-SA-CRTNN is a model excluding the MHSA-CRNN module. CSA-SA-CRNN is a model that removes the MHSA-TCN module, and the detailed composition of each model is shown in Table 5.

Each model was tested on the DEAP dataset for binary and ternary classification experiments, and the results are shown in Table 6. Currently, our proposed model achieves the best accuracy, with 99.26% and 99.15% for binary classification and 97.69% and 98.05% for ternary classification, respectively. Compared to the proposed model, SA-CRTNN has the largest decrease in accuracy, with 13.01% and 14.15% for binary classification and 13.95% and 19.30% for ternary classification, respectively. This indicates that the proposed channel attention module CSAM plays a crucial role in the model. Without CSAM, the model may focus on some useless information, leading to a decrease in accuracy. CSA-CRTNN has the second largest gap in accuracy compared to the proposed model, with 6.76% and 4.16% for binary classification and 3.94% and 11.80% for ternary classification, respectively. This suggests that the proposed SA-CRNN module can effectively address the issue of information omission during feature extraction. CSA-SA-CRNN ranks third in terms of accuracy gap with the proposed model, with 4.47% and 6.24% for binary classification and larger gaps of 10.61% and 9.51% for ternary classification, respectively. This underscores the necessity of global feature extraction. CA-SA-CRTNN has the smallest decrease in accuracy, with 2.11% and 2.90% for binary classification and 1.15% and 3.06% for ternary classification, this indicates that MHSA-CRNN has a certain contribution in the model.

5.5. CSAM Compared with Other Channel Attention Mechanism

To validate the efficacy of the channel attention mechanism proposed in this paper, we conducted a comparative analysis with the commonly used mechanisms SENet and ECANet. The results are depicted in Figure 8.

SENet [51]gives each feature channel in a feature map a weight value. The network can concentrate on certain characteristics since the squeeze and excitation procedures automatically determine the significance of each channel. This method aids in enhancing the network’s performance across a range of jobs.

The ECANet neural network architecture[52] eliminates the requirement for fully connected layers by using 1x1 convolutional layers after the global average pooling layer. This method helps to prevent size reduction. In addition, the design achieves cross channel information exchange by using dimensional convolution. The size of the convolutional kernel is adaptive, which means that layers with many channels can interact more across channels.

Based on the experimental results, it has been found that the channel attention mechanism, CSAM, proposed in this paper, outperforms SENet and ECANet. This suggests that in our model, CSAM can more accurately identify the important channels of EEG signals in comparison to other commonly used channel attention mechanisms. Consequently, this leads to an improvement in the recognition accuracy of the model.

5.6. Evaluation of Model Efficiency

The primary objective of this paper is to enhance model accuracy while simultaneously reducing computational costs. To illustrate the capability of CSA-SA-CRTNN in reducing the computational burden of the model, we have presented the parameters of the proposed model in Table 7 and compared them with those of four other models in Table 8. The findings indicate that our model exhibits higher efficiency, with model parameters totaling 2.90M. This outcome suggests that the model can achieve superior accuracy while consuming fewer resources.

6. Conclusion

In this work, we introduce CSA-SA-CRTNN, a novel model for emotion recognition. To increase accuracy, our model is made to extract pertinent data from several viewpoints, such as temporal, Spatial, local, and global. The CSAM channel attention module is introduced to improve the classification performance by assigning varying weights to distinct channels. Additionally, we introduce SA-CRNN, an adaptive convolutional recurrent module that tackles the problem of missing information during feature extraction. To extract local and global information, we also integrate CRNN and TCN with the multi-head self-attention mechanism, MHSA. We conducted binary and ternary classification experiments on the DEAP dataset, achieving 99.26% and 99.15% accuracy for arousal and valence in binary classification, and 97.69% and 98.05% in ternary classification, demonstrating the effectiveness of our model. Furthermore, our model is efficient, achieving the highest accuracy with lower computational costs than other models. In future work, we plan to explore the performance of our model on different datasets and subjects to evaluate its effectiveness and generalization further.