CognEmoSense: A Continual Learning and Context-Aware EEG Emotion Recognition System Using Transformer-Augmented Brain-State Modeling

1. Introduction

Emotions play a pivotal role in shaping human cognition, decision-making, and behavior. In recent years, the ability to automatically recognize emotional states has garnered significant attention in domains such as affective computing, adaptive human-computer interaction (HCI), neurofeedback therapy, and mental health monitoring. Among various physiological signals used for emotion recognition, electroencephalography (EEG) stands out due to its direct measurement of brain activity, high temporal resolution, and non-invasiveness. EEG-based emotion recognition systems offer a promising avenue for real-time, personalized affective state decoding. However, building reliable and generalizable models remains a complex challenge due to signal noise, inter-subject variability, and dynamic emotional baselines that shift over time.

Traditional EEG-based emotion recognition frameworks often rely on pre-processed features such as power spectral density, Hjorth parameters, or statistical moments extracted from fixed channels. These features are typically fed into shallow classifiers such as Support Vector Machines (SVM), k-Nearest Neighbors (kNN), or simple feedforward neural networks. While such models achieve reasonable performance on benchmark datasets (e.g., DEAP, DREAMER), they are limited by several key shortcomings: (1) they lack temporal awareness and cannot capture long-term dependencies in EEG dynamics, (2) they generalize poorly across users, and (3) they do not incorporate contextual or behavioral information that may significantly modulate emotional responses.

Recent advances in deep learning, especially convolutional and recurrent neural networks (CNNs, LSTMs), have enabled end-to-end learning from raw EEG signals. These models have shown promise in extracting hierarchical representations and temporal patterns, yet they still fall short in generalization, personalization, and adaptability. In particular, their fixed architectures cannot learn incrementally or adapt to new users without retraining, making them unsuitable for real-world, evolving use cases such as long-term mental health tracking or stress monitoring. Furthermore, few existing approaches consider the broader context—such as environmental conditions, time of day, or recent activity—which are often crucial in interpreting affective signals correctly.

To address these limitations, we introduce CognEmoSense, a novel EEG-based emotion recognition system that integrates three core innovations: (1) a temporal transformer encoder that learns long-range dependencies in EEG sequences more effectively than LSTM architectures, (2) a context embedding layer that fuses environmental, behavioral, and cognitive state metadata with EEG inputs, and (3) a continual learning module based on Elastic Weight Consolidation (EWC) to enable lifelong learning without catastrophic forgetting. The proposed system models emotion as a temporally distributed phenomenon modulated by both intrinsic neural patterns and external context, enabling personalized and adaptive classification.

CognEmoSense processes raw or minimally filtered EEG data using fixed-length windows (e.g., 1–3 seconds) and classifies each window into valence-arousal quadrants or discrete emotion labels (e.g., joy, fear, sadness). Unlike static models, it updates its internal representations over time, learns user-specific traits, and adapts to new emotion patterns as the user interacts with the system. Our architecture also supports few-shot adaptation, meaning it can personalize to a new individual with just a few seconds of calibration data—addressing a critical limitation in general-purpose models.

The contributions of this paper are threefold:

• We propose a transformer-based neural architecture tailored for raw EEG time series, enhanced with contextual embeddings and continual learning.
• We conduct a comprehensive evaluation using an augmented DEAP dataset, demonstrating superior performance over classical and deep learning baselines in accuracy, robustness, and generalization.
• We validate the real-world applicability of CognEmoSense in scenarios including stress detection, mood tracking, and emotion-aware user interfaces.

The remainder of this paper is structured as follows. Section II reviews related work in EEG-based emotion recognition. Section III presents the CognEmoSense system architecture and training methodology. Section IV describes the experimental setup, datasets, and evaluation metrics. Section V reports quantitative results and comparative analysis. Section VI discusses the broader implications, limitations, and future directions. Finally, Section VII concludes the paper.

2. Related Work

Emotion recognition from EEG signals has been an active area of research for over two decades, intersecting neuroscience, signal processing, machine learning, and human-computer interaction. Early studies primarily focused on the extraction of handcrafted features such as band power, entropy, and statistical descriptors from EEG signals, followed by traditional classifiers like Support Vector Machines (SVM), k-Nearest Neighbors (kNN), Linear Discriminant Analysis (LDA), and Decision Trees. While these approaches established baseline performance levels, they often relied on manually tuned pipelines and struggled with generalization across subjects and contexts.

A foundational benchmark in this domain is the DEAP dataset, introduced by Koelstra et al. (2012), which contains EEG recordings from participants exposed to emotional stimuli such as music videos. Following this, datasets like DREAMER, SEED, and MAHNOB-HCI expanded the availability of labeled EEG data for affective computing. Most studies using these datasets report valence-arousal-based emotion classification accuracy ranging between 65–80%, depending on feature engineering and classifier type.

The rise of deep learning has significantly altered the research landscape. Several studies have employed Convolutional Neural Networks (CNNs) to automatically learn spatial features from EEG topographies. For example, Bashivan et al. transformed EEG into multi-spectral image representations to be processed by CNNs. Others have used Recurrent Neural Networks (RNNs), particularly Long Short-Term Memory (LSTM) networks, to model the temporal dynamics of EEG sequences. These models outperform traditional methods on static datasets but often fail to generalize across unseen users or real-world noisy settings.

Some hybrid models, such as CNN-LSTM combinations, have shown promise in capturing both spatial and temporal EEG patterns. However, these architectures are still constrained by fixed-length memory and often ignore contextual factors such as environment, fatigue level, or user history. In addition, deep models are typically trained in batch mode and lack mechanisms for online or continual learning, making them brittle in evolving, real-time applications.

The issue of personalization has been explored through techniques like subject-wise cross-validation, domain adaptation, and transfer learning, but none provide satisfactory generalization without large-scale fine-tuning. Some few-shot learning techniques attempt to mitigate this by requiring minimal calibration data, yet they are not robust to long-term usage or concept drift.

Regarding contextual awareness, very few models integrate non-EEG signals such as environment metadata, user activity logs, or time-series of behavioral events. Yet, psychology literature repeatedly shows that emotional responses are modulated by situational context and cognitive state. The omission of such information limits the interpretability and adaptability of emotion recognition systems.

Another under-explored direction is the use of Transformer architectures in EEG analysis. While transformers have revolutionized natural language processing and speech recognition due to their capability to model long-range dependencies, their application in EEG time series is still emerging. A handful of studies have tested EEG-Transformers for motor imagery or seizure detection but rarely for emotion recognition. Moreover, these studies typically lack integration with continual learning strategies, which are essential for lifelong, adaptive systems.

In summary, current emotion recognition models based on EEG signals suffer from at least three critical gaps:

(1)

lack of adaptive personalization,

(2)

absence of contextual embedding, and

(3)

insufficient exploration of transformer-based temporal modeling.

CognEmoSense addresses these limitations by combining transformer-based EEG sequence encoding, contextual augmentation, and Elastic Weight Consolidation (EWC) for continual learning. This unique integration not only boosts classification performance but also ensures robustness, adaptability, and practical deployment readiness—qualities rarely achieved simultaneously in prior work.

3. Methodology

The CognEmoSense architecture is designed to achieve real-time, adaptive, and context-aware emotion recognition from raw EEG signals. This section details the proposed system’s data pipeline, neural architecture, context integration, and continual learning framework. The methodology consists of four primary components: EEG signal preprocessing, temporal transformer-based encoder, context embedding module, and Elastic Weight Consolidation (EWC)-based continual learning mechanism.

A. EEG Signal Acquisition and Preprocessing

We use raw EEG data sampled at 128 Hz, with 32 channels conforming to the 10-20 international electrode placement system. The input signals are segmented into overlapping windows of 3 seconds (384 samples per channel), with a stride of 1 second to provide temporal continuity. Basic preprocessing steps include notch filtering at 50/60 Hz to eliminate line noise and bandpass filtering between 1–45 Hz to isolate cognitive-relevant frequency bands. Unlike traditional approaches, we do not extract handcrafted features; instead, we feed minimally preprocessed raw time-series directly into the model. Let the EEG input be denoted by a tensor $X \in =R^{CxT}$, where C = 32 is the number of channels, and T = 384 is the number of time points in each window. These are transformed into a 2D input representation via channel-wise positional encoding and normalized with z-score standardization.

B. Temporal Transformer Encoder

The backbone of CognEmoSense is a Temporal Transformer that models time-dependent patterns in EEG windows using self-attention. Transformers, unlike RNNs or LSTMs, allow for non-sequential attention over all time steps, enabling the model to focus on temporally significant events irrespective of position.

The input EEG window is first passed through a learnable linear projection:

$$\mathrm{Z}=\mathrm{ReLU}\left({\mathrm{W}}_{1}t X+b_1\right),\mathrm{Z} \in R^{dxT}$$

Then, it is combined with a learnable temporal positional encoding:

$${\mathrm{Z}}_{PE}=\mathrm{Z}+\mathrm{PE}\left(\mathrm{t}\right),\mathrm{PE}\left(\mathrm{t}\right) \in R^{dxT}$$

This embedded sequence is passed into L layers of a standard transformer encoder with multi-head attention:

$$H^{\left(l\right)}=TransformerEncoder\left(H^{\left(l-1\right)}\right),l=1,\mathrm{...},L$$

The output is pooled using attention-based pooling to produce a fixed-length vector

${\mathrm{h}}_{\mathit{EEG}} \in R^d$representing the EEG-based emotional embedding.

C. Contextual Embedding Layer

One of CognEmoSense’s novel contributions is the integration of context vectors into the emotion classification process. Context features include:

• Time of day (morning, afternoon, evening)
• Environment noise level
• Previous emotional state history
• Physiological stress markers (optional)

Each categorical or continuous context element is embedded using dedicated dense layers and combined via concatenation:

$${\mathrm{h}}_{\mathit{context}}=Concat\left(\mathrm{Embed}\left(f_1\right) ,\mathrm{...},\mathrm{Embed}\left(f_1\right)\right) \in R^{d_c}$$

The final joint representation is then formed:

$${\mathrm{h}}_{\mathit{joint}}= LayerNorm\left(\left[h_{EEG}; h_{\mathit{context}}\right]\right),l=1,\mathrm{...},L$$

This fused vector is passed to a two-layer feedforward classification head with softmax output to predict valence-arousal categories or discrete emotion classes (joy, sadness, fear, etc.).

D. Continual Learning with Elastic Weight Consolidation (EWC)

Traditional models suffer from catastrophic forgetting when trained on new user data. To combat this, we adopt EWC, which adds a penalty to the loss function based on how important a model weight is to prior tasks. For a given parameter θ, the EWC-augmented loss function is:

$$L\left(\mathsf{\theta }\right)=L_{task}\left(\mathsf{\theta }\right)+{\displaystyle \frac{\lambda }{2}}{\displaystyle \sum }_2 F_i{\left({\mathsf{\theta }}_i-{\mathsf{\theta }}_i^{*}\right)}^2$$

Where:

• L is the standard cross-entropy loss,
• ${\mathsf{\theta }}_i^{*}$ are the optimal parameters from previous task(s),
• $F_i$are elements of the Fisher information matrix estimating importance,
• λ is a regularization coefficient controlling forgetting vs. adaptation.

This allows CognEmoSense to retain performance on past user profiles while adapting to new subjects with minimal retraining.

E. Training and Optimization

The entire model is trained end-to-end using Adam optimizer with learning rate scheduling and dropout regularization (0.3). We use batch size of 64 and early stopping based on validation loss. For personalization, a few-shot adaptation phase is added post-training, where 10–30 seconds of new user EEG data is used to fine-tune only the final classification layer.