eXCube2: Explainable Brain-Inspired Spiking Neural Network Framework for Emotion Recognition from Audio-, Visual- and Multimodal Audio-Visual Data

1. Introduction: Towards Brain-Inspired Biomimetic Systems for Audio-, Visual- and Audio-Visual Pattern Recognition

1.1. Problem Definition

Current technologies for speech recognition and face recognition have advanced significantly in recent years, driven by modern statistical and neural network methods [1,2,3,4,5,6,7,8,9,10,11]. However, voice and face data can be used to address many other challenging AI problems [12,13,14,15,16]. An open problem is the development of AI systems that use voice and vision data to recognize and explain human brain states, such as emotional states and brain diseases. Current voice and computer vision technologies need to be further developed, and new approaches must be created to make AI systems closer to human perception, human expression, and human understanding, and perhaps even to human consciousness [14]. One way to target this goal is to develop brain-inspired AI systems (BIAI).

Current brain-inspired systems are mostly based on spiking neural networks (SNNs) [9,17]. An example is the brain-inspired SNN architecture NeuCube, introduced in [18].

The aim of the proposed here novel eXCube2 SNN framework is to recognize emotional states from audio, visual, and multimodal audio-visual data. While based on the NeuCube architecture, the eXCube2 framework is a novel one that introduces new original methods for the problem in hand.

1.2. Why Use Brain-Inspired SNN and the NeuCube Architecture for Audio-Visual Data?

Spiking neural networks (SNNs) are biologically inspired artificial neural networks in which information is represented as binary events (spikes), similar to action potentials in the brain, and learning is also inspired by principles observed in the brain. SNNs are also universal computational mechanisms [17]. Learning in SNNs refers to changes in the connection weights in the network. Many learning paradigms, such as Spike-Timing-Dependent Plasticity (STDP), are inspired by the Hebbian learning principle. In STDP, synaptic weights are adjusted based on the temporal order of the incoming spike (pre-synaptic) and the output spike (post-synaptic). This synaptic weight adjustment determines synaptic potentiation, known as long-term potentiation (LTP), when the synaptic weight increases (positive change). On the other hand, synaptic depression, known as long-term depression (LTD), occurs when the synaptic weight decreases (negative change). If a pre-synaptic spike arrives before (after) a post-synaptic spike, the synaptic link between the two neurons is potentiated (depressed). Thus, learning in the network depends on spike times, which leads to changes in synaptic strength.

STDP is defined mathematically in Equation (1):

$$W \left(t_{pre}-t_{post}\right)=\left\{\begin{array}{c}{ \mathrm{A}}^{+} e^{(t_{pre}-t_{post})/{\tau }_{+}}, {\mathrm{if} \mathrm{t}}_{\mathrm{pre}}{ \text{<} \mathrm{t}}_{\mathrm{post}} \\ -{\mathrm{A}}^{-} e^{(t_{post}-t_{pre})/{\tau }_{-}} {\mathrm{if} \mathrm{t}}_{\mathrm{pre}}{ \text{>} \mathrm{t}}_{\mathrm{post}}\end{array}\right.,$$

(1)

where, $W \left(t_{pre}-t_{post}\right)$is the change in weight as a function of the difference between the pre- and post-synaptic spike times, τ+ and τ− are the LTP and LTD time constants, respectively, and ${\mathrm{A}}^{+}$and ${\mathrm{A}}^{-}$are the maximum adjustment to synaptic weight when $t_{pre}-t_{post}$ approaches zero.

Overall, an SNN trained with the STDP rule can capture spatio- and spectro-temporal patterns from data, where input neurons are spatially distributed, and connection weights learn temporal associations between them.

Izhikevich [19] has shown that similar activation patterns (called ‘polychronous waves’) can be generated in an SNN reservoir with recurrent connections to represent short-term memory. This is a further extension of the ‘synfire chain’ theory by Abeles [20]. The above principles are utilized in [21,22,23,24] for the creation of spatio-temporal associative memories in SNN, which is a brain-inspired principle in audio-visual perception [25].

The eXCube2 architecture is based on the NeuCube SNN brain-inspired architecture (Figure 1) [18,26].

The list below describes the functionality of the NeuCube architecture [18]:

• Temporal inputs (features) are converted into spike trains.
• Inputs are mapped spatially into a 3D SNNcube, that consists of spiking neurons spatially organized in a topological 3D map. For modelling cognitive brain-related data, the SNNcube is built using a brain template, such as Talairach or MNI, etc. (e.g., [27,28,29,30]).
• An output classifier/regressor SNN is connected to neurons from the SNNcube, e.g., deSNN [31].
• The SNNcube structure is initialized as a small world connectivity 3D structure of spiking neurons.
• Unsupervised learning is performed in the SNNcube using STDP.
• Supervised learning is performed in the output SNN module, e.g. deSNN for classification.
• The learned connectivity patterns in the SNNcube can be interpreted as deep knowledge, representing deep spatio-temporal patterns in the data. Learned connectivity patterns in the deSNN output module can be interpreted for rule extraction related to outputs [32,33].
• The model is further trained and adapted on new data, where new connections are evolved in the SNNcube and new output neurons are evolved in the deSNN classifier to capture new patterns and new classes different from those previously used.

1.3. Experimental Data

For the initial development and testing of the eXCube2, we use part of the Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS): a dynamic, multimodal set of facial and vocal expressions in North American English [34,35]. The dataset is available at : https://zenodo.org/records/1188976, along with: https://zenodo.org/records/3255102. Examples of the data are available at: https://www.youtube.com/ watch? v=cxMK2J0P7J0.

The Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS) contains 7356 files (total size: 24.8 GB). The dataset includes 24 professional actors (12 female and 12 male) vocalizing two lexically matched statements in a neutral North American accent. Speech includes calm, happy, sad, angry, fearful, surprised, and disgust expressions, and song contains calm, happy, sad, angry, and fearful emotions. Each expression is produced at two levels of emotional intensity (normal, strong), with an additional neutral expression. All conditions are available in three modality formats: audio-only (16-bit, 48 kHz, .wav format), audio-video (720p H.264, AAC 48 kHz, .mp4 format), and video-only (no sound). The RAVDESS was developed by Dr Steven R. Livingstone [35].

For the experimental study, the following labelling of the data has been used:

• Class 0 = Low emotional arousal: neutral, calm, sad;
• Class 1 = High emotional arousal: happy, angry, fearful, disgust, surprised.

2. Methods: A general eXCube2 Framework and Models for Emotion Recognition Based on Audio-, Visual- and Multimodal Audiovisual Data

2.1. The General eXCube2 Framework

The problem of detecting a brain state of emotion using audio, visual, or both modalities is represented here as a classification problem (Figure 2).

The eXCube2 architecture applies brain-inspired tonotopic mapping of audio signals and topographic (retinotopic) mapping of images into the 3D SNNcube, and the learned or recalled patterns in the SNNcube are then classified (Figure 3).

2.2. Audio Feature Extraction and Feature Encoding

Different features can be extracted from raw speech data and used for different applications. In the context of brain state recognition, this paper suggests the use of mel-spectrogram features, after considering and comparing them with three other possible feature types, as shown in Table 1. Each feature is mapped into the 3D SNN as an input neuron.

The audio features are mapped into the SNNcube as input neurons to both the left and right areas of the SNNcube, which correspond to the left and right auditory cortex according to the selected brain template. Each of the above features can be used in the development and implementation of an eXCube2 model for specific applications. Mel_spectrogram features, as used in the current implementation of eXCube2, are shown in Figure 4. For comparison, an example of 24 linear_fft feature extraction, encoding, and mapping is given in Appendix A.1.

The extracted audio features are then encoded into spikes using different possible encoding schemes. Figure 5 illustrates the encoding of the data from Figure 4 (top) using the Step-Forward method [17] (middle), as well as the reconstruction of the original signals from the spike trains (bottom), with the reconstruction error quantified by the normalized MSE.

2.3. Tonotopic Mapping of Features into a 3D SNNcube

We employ a tonotopic mapping of the audio features, motivated by the tonotopic organization of the human auditory cortex, where neurons are spatially arranged according to their preferred frequency. In this organization, low frequencies and high frequencies are mapped to distinct but adjacent cortical regions, forming a continuous frequency gradient across the primary auditory cortex (A1).

Following this principle, the extracted features are mapped into a pre-structured eXCube2 SNN using MNI brain template coordinates and a tonotopic assignment of spatial locations to the selected features. Figure 6 illustrates the mapping of the 40×2=80 mel-spectrogram features into the SNNcube, and the corresponding algorithm is presented in Table 2.

For a comparative analysis, the tonotopic mapping of the 24 linear_fft features into the SNNcube are provided in Appendix A.2.

2.4. Feature Extraction and Topographic Mapping of Visual data

Different visual features can be extracted from the video data and mapped topographically into the visual cortex regions of the SNNcube according to the MNI template. In this study, face features are extracted as 52 facial blendshapes from the RAVDESS video files using MediaPipe Face Landmarker [34,35]. These features are then mapped to the FFA/STS brain regions for SNN processing.

The following 52 features are extracted:

• Brow (browDownLeft/Right, browInnerUp, browOuterUpLeft/Right), 5 features.
• Eye (eyeBlinkLeft/Right, eyeSquintLeft/Right, eyeWideLeft/Right), 8 features.
• Cheek (cheekPuff, cheekSquintLeft/Right), 3 features.
• Nose (noseSneerLeft/Right), 2 features.
• Jaw (jawOpen, jawForward, jawLeft/Right), 4 features.
• Mouth (mouthSmileLeft/Right, mouthFrownLeft/Right, etc), 28 features.
• Tongue (tongueOut), 1 feature
• Neutral (neutral (always class 0), 1 feature.

The 52 visual features are mapped to the FFA/STS areas of the SNNcube in the right hemisphere only, since face processing is predominantly right-hemisphere dominant (in contrast to audio processing, which is bilateral). This topographic mapping is motivated by well-established neuroscience findings:

• The Occipital Face Area (OFA) is responsible for early face detection (right hemisphere).
• The Fusiform Face Area (FFA) is responsible for face identity (right occipital, coordinates around (40, -55, -15).
• The Superior Temporal Sulcus (STS) is responsible for dynamic facial expressions and gaze (approximately (50, -45, 10)).

It is also known that if right hemisphere is damaged as in prosopagnosia, the subject cannot recognize faces. If left hemisphere is damaged, there is no face recognition deficit. Right hemisphere specializes in holistic/configural processing.

Blendshapes features are mapped topographically within FFA/STS region of the SNNcube as follows:

• Dorsal area (top, higher Z) encodes brow features (browDown, browInnerUp), eye features (eyeBlink, eyeSquint), and nose features (noseSneer);
• Ventral area (bottom, lower Z) encodes mouth-related features (mouthSmile, jawOpen).

The spatial mapping uses the following coordinate ranges: X dimension 28–60 mm (right hemisphere, lateral), Y dimension −62 to −42 mm (posterior temporal), and Z dimension −11 to 16 mm (ventral to mid-level).

2.5. Mapping Multimodal Audio-Visual Features into an eXCube2 Model

The audio and visual features described in the previous sections are combined to form a multimodal audio-visual eXCube2 system. The extraction of audio and visual features is synchronized at 10 ms. A snapshot of the resulting feature activity for an exemplar multimodal data sample is shown in Figure 7.

2.6. Training of an SNNcube for eXCube2 Models on Audio-, Visual- and Audio-Visual Data

Separate eXCube2 models are constructed for audio-only, visual-only, and multimodal audio-visual data using the features described above. For unsupervised training of the SNNcube, Spike-Timing-Dependent Plasticity (STDP) is employed (see [17]), with the training and testing parameters summarized in Table 3. Further experimental details and parameter settings for the training and testing procedures are provided in Appendix A.3.

After training each SNNcube model, state vectors are extracted from the reservoir and used to train a classifier in a supervised mode.

2.7. State Vector Extraction from a Trained SNNcube and Their Classification

Different approaches can be used to extract state vectors from a trained SNNcube.

(a)

Spike Count: This method sums the total number of spikes per neuron across all timesteps for each sample:

$$s_i={\displaystyle \sum }_{t=1}^T x_i\left(t\right),$$

(2)

where $x_i\left(t\right)$ denotes the spike activity of neuron i at time t. In this case, temporal information is aggregated into a single value per neuron, and the state vector is represented by the spike counts of all neurons.

• (b) DeSNN weight-based state vectors: Alternatively, state vectors can be extracted from the connectivity weights of a DeSNN classifier (see [31]). These weights are determined by the first spike time and the total number of spikes:

$$\begin{gathered} {\omega }_i=\alpha * m^{t_i^{first}}+d_{up}* n_i^{total}-d_{down}* \left(T-n_i^{total}\right), \\ \mathrm{where}\text{:} \alpha =5.0, \mathrm{m}=0.8, d_{up}=0.8, d_{down=0.01} \end{gathered}$$

The extracted state vectors are used to train a classifier to recognize two emotion classes, corresponding to arousal and calm. In practice, simple spike counting performs comparably to DeSNN connection weighting methods because the reservoir has already transformed temporal information into spatial patterns. Through recurrent dynamics and STDP learning, different neurons become selective to different temporal motifs, and their firing patterns implicitly encode the temporal evolution of the input. Moreover, STDP strengthens connections between neurons that fire in consistent sequences, thereby embedding temporal structure into the network connectivity.

Spike counting on reservoir neurons captures discriminative information, because each neuron's firing reflects integrated temporal patterns across the network through learned connectivity. The reservoir performs temporal feature extraction. Once the state vectors are extracted from the trained SNNcube, different classification methods have been applied and compared to classify these vectors into the two output classes as described in Table 4.

In the current implementation of the eXCube2 framework, the Learned Prototype classifier is used, as its clustering capability is well suited to the experimental data. Other classifiers can be employed for different applications while still using the same eXCube2 framework.

3. Experimental Results

3.1. Classification Results on the Experimental Data

Table 5 compares the classification performance obtained using multimodal data in the eXCube2 model with the performance achieved using only audio data or only visual data. The eXCube2 model can operate on the integrated multimodal input as well as on each modality separately. Using different methods for state vector extraction and classification yields comparable results, with accuracies consistently in the range of around 80%.

• I = Input neurons only (spike-encoded features)

• R = Reservoir neurons only

• I+R = Combined input + Reservoir neurons

• SC = Spike count (sum of spikes over time)

• split = Separate positive/negative spike counts (doubles feature dimensionality)

• STS = Superior Temporal Sulcus brain region reservoir neurons only (audio-visual integration)

• A1 = Primary Auditory Cortex brain region reservoir neurons only

• py = DeSNN Python implementation (from neucubepy library)

• hybrid = DeSNN Hybrid (combines MATLAB-style with order encoding).

Using the introduced “don’t know” output with a confidence threshold in the range 0.55–0.65 improves the effective classification accuracy up to 88% by rejecting low-confidence samples, as shown in Table 6.

The classification results of the three models in Table 6 are illustrated in Figure 8, Figure 9 and Figure 10 respectively.

3.2. Explainability of the eXCube2 Models

The eXCube2 models provide informative representations at multiple levels of their structure and dynamics. First, the spatial mapping of features is itself interpretable, as it follows a brain template and incorporates established neuroscience knowledge (e.g., Figure 6 and Figure 7, and A.1).

More importantly, the models capture dynamic interactions between features. This is illustrated in Figure 11, which shows spike-time associations between 24 linear_fft features (see Appendix A.2 for their mapping into the SNNcube). The associations are visualized as arcs between features represented as nodes (L denotes the left hemisphere; R denotes the right hemisphere). The thicker the arc, the more frequently spikes at one node are followed by spikes at the connected node in the next time step (10 ms), indicating stronger temporal coupling. This provides a spike-based representation of information exchange within the model.

4. Conclusions, Discussions and Future Work

The paper presents a novel SNN-based framework for capturing and classifying patterns from audio-, visual or combined audio-visual data. The framework consists of modules for feature extraction, spike encoding, mapping features into a 3D SNN pre-structured according to a brain template, training the SNNcube, extracting state vectors from the SNNcube, training a classifier on the state vectors, recalling and adapting the system to new data, as well as visualisation and dynamic explanation of the processing. Each module is grounded in brain-inspired information processing principles.

At each stage of the framework, multiple methods can be employed, some of which were explored in this work through a case study on emotion recognition using speech, facial images, and video. An integrated eXCube2 system has been developed for audio-visual data, combining three models for audio-only, visual-only, and multimodal processing. The system was evaluated on benchmark datasets for each modality and their integration, achieving accuracies of up to around 90% when confidence-based rejection is applied.

The eXCube2 framework has potential applications in a wide range of domains using one or both modalities, including recognition of basic emotional states; brain state diagnosis (e.g., Parkinson’s disease, Alzheimer’s disease, ADHD, dementia) [16,36]; monitoring response to treatment over time [14]; evaluation of user satisfaction in online services [37]; human–robot interaction [38]; chatbots [39]; interactive games, and related applications.

The three models are implemented as Python software tools, enabling the design and experimentation of eXCube2 models across sound, image, and video data. The use of an SNNcube together with evolving classifiers supports scalability to larger datasets, adaptability to new data, and explainability of the underlying dynamic processes. Additional software has been developed to dynamically visualize the activity of eXCube2 during recall on new data.

Importantly, the SNN-based implementation of Brain-Inspired AI (BIAI) makes the framework suitable for deployment on neuromorphic hardware platforms, enabling reduced power consumption, smaller device size, and improved efficiency, while preserving adaptability and explainability [40,41,42,43,44]. Ultimately, the use of shared brain templates for both biological and artificial systems may help bring human–machine symbiosis closer in the future.

Future work will focus on: (1) evaluating additional classifiers for the extracted state vectors (e.g., [45,46]); (2) using brain data (e.g., EEG, fMRI) to pre-train the system and then fine-tune it for audio, visual, or audiovisual applications [32,47]; (3) further extending the concept of evolving spatio-temporal associative memory based on brain principles [23,48]; and (4) implementing the models on contemporary hardware platforms for real-world applications [17,43,49].

6. Patents

The authors declare that there are no patents associated with this work.

A.1. Encoding and Mapping of the 24 linear_fft Features

Figure A.1. Example of linear_fft features showing spike encoding and signal reconstruction.

Figure A.1. Example of linear_fft features showing spike encoding and signal reconstruction.

Figure A.2. Mapping of linear_fft features into the SNNcube.

Figure A.2. Mapping of linear_fft features into the SNNcube.

A.2 Experimental Details of the Tonotopic Mapping of Audio Features into the SNNcube

• The following setup was used for the tonotopic mapping of audio features:
• The HCP-MMP1 atlas (Glasser et al., 2016) on MNI152 template was used.
• A hybrid downsampling scheme was applied: 2.5 mm resolution for the auditory cortex and 8.1 mm for the rest of the brain, resulting in 4176 neurons in total (840 auditory + 3336 other neurons).
• Input neurons were placed only in A1 (primary auditory cortex) in both hemispheres.
• A PCA-based gradient was used to distribute neurons evenly along the tonotopic axis (high → low → high frequencies).
• A direct 1:1 mapping was applied: sample column i → neuron i

The HCP-MMP1 atlas provides precise anatomical boundaries for auditory regions:

• A1 (Core): Primary auditory cortex with sharp frequency tuning and direct sensory input.
• Belt: Surrounding A1, integrates frequency channels and supports phonemes/timbre processing
• Parabelt: Higher-level auditory processing, including speech and music categories.

Features were mapped specifically to the A1 region of the SNNcube for the fol-lowing reasons:

• Biological plausibility: Mimics how the real auditory cortex receives input.
• Spatial learning: Enables the SNN to learn spatial relationships between frequency bands.
• Interpretability: Neuron activations correspond to known brain regions.
• Emergent organization: As shown in TopoAudio (29), spatially constrained networks can develop brain-like organization without explicit supervision.

With this mapping, the following behavior is observed:

• Only A1 (core) receives direct input, not belt or parabelt regions.
• A1 neurons are frequency-selective, similar to mel spectrogram or fft bands.
• Belt regions receive processed output from A1 via learned SNN connections.
• Parabelt regions receive output from belt regions.
• This mapping matches the biological processing hierarchy.

From the HCP-MMP1 atlas on MNI152 template, the hybrid downsampling is applied as follows:

• Auditory cortex: 2.5 mm voxel size (high resolution for input neurons).
• Other regions: 8.1 mm voxel size (standard NeuCube resolution).
• Result: 840 auditory neurons + 3336 other neurons = 4176 total neurons.

A.3. Experimental Details of the Training/Testing Parameters of the eXCube2 Models

The following parameters were used for the three experiments: audio-only, visual-only, and multimodal audio-visual:

Initial dataset (before oversampling):

• Training: 360 samples (6 actors × 60 recordings)
• Testing: 120 samples (2 actors × 60 recordings)

Class imbalance in training data:

• Low arousal (neutral, calm, sad): 120 samples
• High arousal (happy, angry, fearful, disgust, surprised): 240 samples
• Ratio: 1:2 (imbalanced)

After oversampling (training only):

• Low arousal: 240 samples (duplicated from 120)
• High arousal: 240 samples (unchanged)
• Total training set: 480 samples (balanced 1:1)

Test set (unchanged distribution):

• 40 low arousal + 80 high arousal = 120 samples

Signal duration:

• 234–501 timesteps after silence trimming
• Shortest: 234 timesteps × 10 ms = 2.34 s
• Longest: 501 timesteps × 10 ms = 5.01 s

Feature extraction parameters:

• 10 ms hop size, 25 ms window length (standard in speech processing)

Full dataset split:

• Training: 1520 samples (760 class 0, 760 class 1), 19 actors (3–12, 15–23)
• Testing: 1820 samples (860 class 0, 960 class 1), 5 actors (1, 2, 13, 14, 24)