Brain Inspired Spatio-Temporal Associative Memories for Neuroimaging Data: EEG and fMRI

1. Introduction

Memory is referred to as the brain’s ability to recall experiences or information that is encountered or learnt previously. If this information is recalled using only partial inputs, we refer to it as Associative Memory (AM) [1,2]. There are three main types of memory in the brain, namely sensory memory, short-term or working memory, and long-term memory, which function in different ways, but each of these types are manifested through brain activities in space (areas of the brain) and time (spiking sequences), stored as connection weights and recalled always with only partial input information in time and space. AM in the brain is always spatio-temporal.

Humans can learn and understand many categories and objects from spatio-temporal stimuli by finding spatial and temporal association between them. Inspired by this human brain capability, AM has been introduced to the machine learning field to memorize information and retrieve it from partial or noisy data. For example, neural network models for associative pattern recognition was proposed by J.Hopfield [3] and B.Kosko [4]. In 2019 Haga and Fukai [5] introduced a memory system for neural networks based on an attractor network, which is a group of connected nodes that display patterns of activity and tend towards certain states. They applied the concept of excitatory and inhibitory nodes to their proposed network to mimic the role of hippocampus in balancing network to form new association. The work above related to vector-based data (e.g., static images) and not to spatio-temporal data. None of them related specifically to neuroimaging (NI) data.

The idea of using a brain-inspired and brain-structured spiking neural network (SNN) as a spatio-temporal associative memory (STAM) was first introduced in [6] as part of the NeuCube SNN architecture, but the main concepts and definitions of STAM were introduced in [7], where a NeuCube model, trained on complete spatio-temporal data, creating spatio-temporal patterns in its connections, is recalled when only partial spatio- or/and temporal information is provided as inputs.

In this paper we introduce for the first time STAM on NI data, such as EEG and fMRI. The paper is organized in the following way. Section 2 presents the background concept of STAM from [7]. Section 3 presents a STAM-EEG classification model, while section 4 presents a STAM-fMRI classification model. Section 5 is a conclusion and further discussions towards using the STAM approach across bioengineering applications, including multimodal neuroimaging data.

2. SNN and the STAM-SNN concept from [7]

2.1. Spiking neural networks (SNN)

Spiking neural networks (SNN) are biologically inspired ANN where information is represented as binary events (spikes), similar to the event potentials in the brain, and learning is also inspired by principles in the brain. SNN are also universal computational mechanisms [8,9]. Learning in SNN relates to changes of connection weights between two spatially located spiking neurons over time (Figure 1), so that both “time” and “space” are learned in the spatially distributed connections.

A well-known unsupervised learning paradigm inspired by the Hebbian learning principle is spike-time dependent plasticity (STDP) [8], in which the synaptic weights are adjusted based on the temporal order of the incoming spike (pre-synaptic) and the output spike(post-synaptic). STDP is expressed in (Equation 1), where ${\tau }_{+}$and ${\tau }_{-}$ are time parameters and $A_{+}$and $A_{-}$refer to the temporal synaptic adjustment.

$$W\left(t_{pre}-t_{post}\right)=\left\{\begin{array}{c}{A}_{+}exp\left(\frac{t_{pre}-t_{post}}{{\tau }_{+}}\right) if{t}_{pre}<t_{post},\\ A_{-}\mathit{exp}\left(-\frac{t_{pre}-t_{post}}{{\tau }_{-}}\right)if{t}_{pre}>t_{post},\end{array}\right.$$

(1)

Many computational models and architectures have been developed with the use of SNN (see for a review [9]). One of them, NeuCube [6,10] has been used in the proposed for the development of the STAM framework [7].

2.2. STAM on the NeuCube Framework [7]

The NeuCube architecture is depicted in Figure 2 [6]. It consists of the following functional modules:

• Input data encoding module.
• 3D SNN reservoir module (SNNcube).
• Output function (classification) module, such as deSNN [11].
• Gene regulatory network (GRN) module (optional).
• Parameter optimization module (optional).

Spatio-temporal input data is first encoded into spike sequences and then spatio-temporal patterns of these sequences are learned in a SNNcube that is structured according to a brain temple, such as Talairach [12], MNI [13], personal MRI [14]. Connections are created and strengthened in the SNNcube through DTDP learning. Once data is learned, the SNNcube retains the connections as a long-term memory.

The main trust of the proposed STAM concept in [7] is that, since a SNNcube learns functional pathways of spiking activities represented as structural pathways of connections, when only a small initial part of input data is entered the SNN will ‘synfire’ and ‘chain-fire’ learned connection pathways [15] to reproduce learned functional pathways as polichronisation of neuronal clusters [16]. Some studies defined the state of a SNN as a dynamic chaotic attractor [17] that can be reached with partial input information. In [18,19] polychronous neuronal groups are studied that are activated from partial inputs.

In a STAM model based on NeuCube [7], for every spatio-temporal input sequence Pi, a state Si of the SNNcube is defined as a sequence of activated neurons Si1, Si2, …, SiN over time, that is learned in a deSNN classifier [11], forming an N -element vector Wi of connection weights of an output neuron Oi assigned through supervised learning to the input sequence Pi. When a new input sequence Pnew is presented, either as a full sequence in time and space (number of input variables) or as a partial one, the new SNNcube state Snew is learned as a new output neuron Onew, which weigh vector Wnew is compared with the weight vectors of the existing output neurons for classification tasks or for time series prediction tasks using k-nearest neighbour method. The Pnew is classified based on the output classes (for classification tasks) or the assigned real output values (for prediction tasks) of the closest, according to the Euclidean distance, output weight vectors.

In principle, STAM is a system that is trained for classification or prediction on all available spatio-temporal variables and recalled only on part of the spatial or/and temporal components of the data, without compromising the model accuracy. To validate a STAM model, several types of accuracy tests are introduced in [7]:

• Temporal association accuracy: validating the full model on partial temporal data of the same variables and same data.
• Spatial association accuracy: validation of the full model on full temporal data and on a subset of variables, using the same data set.
• Temporal generalization accuracy: validation of the full model on partial temporal data of the same variables or a subset of them, but on a new data set.
• Spatial generalization accuracy: validation of the full model on full or partial temporal data and on a subset of variables, using a new data set.

The above accuracy tests are applied differently for classification and for prediction tasks [7]. The STAM concept is used in the next subsection to develop specific STAM-EEG (section 3) and STAM-fMRI (section 4).

3. STAM-EEG for classification

3.1. The proposed STAM-EEG classification method

The proposed STAM-EEG classification method, consists of the following steps:

• Defining the spatial and the temporal components of the EEG data for the classification task, e.g., EEG channels and EEG time series data.
• Designing a SNNcube that is structured according to a brain template suitable for the EEG data (e.g., Talairach, or MNI, etc.).
• Defining the mapping in the input EEG channels into the SNNcube 3D structure (see Figure 3a as an example of mapping 14 EEG channels in a Talairach structured SNNcube).
• Encode data and train a NeuCube model to classify a complete spatio-temporal EEG data, having K EEG channels measured over time T.
• Analyse the model through cluster analysis, spiking activity and the EEG channel spiking proportional diagram (Figs. 3b, c,d,e).
• Recall the STAM-EEG model on the same data and same variables but measured over time T1 < T to calculate the classification temporal association accuracy.
• Recall the STAM-EEG model on K1<K EEG channels to evaluate the classification spatial association accuracy.
• Recall the model on the same variables, measured over time T or T1 < T on a new data to calculate the classification temporal generalization accuracy.
• Recall the NeuCube model on K1<K EEG channels to evaluate the classification spatial generalization accuracy using a new EEG dataset.
• Evaluate the K1 EEG channels as potential classification brain biomarkers according to the problem at hand.

3.2. Experimental results

The EEG data consist of 60 recordings of 14 EEG channels of a subject who is moving a wrist: up (class 1), straight (class 2) and down (class 3). The data includes 20 samples for each class, each sample measured as 128 time points used to discretize 1,000ms signal.

The main question to address here is: What will be the classification temporal association accuracy when a trained NeuCube STAM-EEG model on all 60 samples and 14 variables is recalled on a shorter sub-section of time of the same variables and same data? The parameter settings of the STAM-EEG NeuCube model are shown Table 1.

To validate the performance of a NeuCube STAM-EEG model in the realm of associative memory tasks, we initially trained the full model with a complete set of 60 samples (Figure 3a) and analyzed the connectivity and the spiking activity of the model (Figure 3b-e).

To evaluate the classification temporal association accuracy of the fully trained NeuCube STAM-EEG model from Figs.4a-e, the model was recalled with the same EEG data but on a smaller percentage of time. The validation results are shown in Table 2, along with a new association measurement introduced here, called retained temporal association (RTA) as calculated using Equation. 2 below:

$$RTA=\frac{Ar}{Af}$$

(2)

where: Af is the classification accuracy of the full model, validated on the same training data, and Ar is the accuracy of the model validated on the same data, but using shorter time series.

3.3. Why STAM-EEG are needed?

The proposed in this section method STAM-EEG, based on the NeuCube SNN, is illustrated on a simple EEG problem, but its applicability is wide across any studies involving EEG data. A large STAM-EEG system can be developed for a particular problem, involving millions of EEG samples and hundreds of EEG channels, integrating EEG data from different sources and studies. This model can be validated for its temporal and spatial association and generalisation accuracy on a particular sub-set of EEG channels, measured at shorter times. If the validation accuracies are acceptable, then the model can be successfully used on the new EEG data.

In this case, studies that resulted in smaller, but specific data sets, can benefit from the use of larger models on the same problem.

4. STAM-fMRI for classification

4.1. The proposed STAM-fMRI classification method

The proposed STAM-fMRI method consists of the following steps:

• Defining the spatial and the temporal components of the fMRI data for the classification task, e.g., fMRI voxels and the time series measurement.
• Designing a SNNcube that is structured according to a brain template suitable for the fMRI data. This could be a direct mapping of the fMRI voxel coordinates or transforming the voxel coordinates from the fMRI image to another template, such as Talairach, MNI, etc. [20] (Figure 4a).
• Selecting voxel features K from the full set of voxels (Figure 4b) and defining their mapping as input neurons in the 3D SNNcube. (Figure 4c).
• Encode data and train a NeuCube model to classify a complete spatio-temporal fMRI data, having K voxel inputs measured over time T.
• Analyse the model through connectivity and spiking activity analysis around the input voxels (Table 3).
• Recall the STAM-fMRI model on the same data and same variables but measured over time T1 < T to calculate the classification temporal association accuracy.
• Recall the STAM-fMRI on K1<K EEG channels to evaluate the classification spatial association accuracy.
• Recall the model on the same variables, measured over time T or T1 < T on a new data to calculate the classification temporal generalization accuracy.
• Recall the NeuCube model on K1<K voxel variables to evaluate the classification spatial generalization accuracy using a new fMRI dataset.
• Evaluate the K1 fMRI voxel variables as potential classification brain biomarkers (section 4.4).

4.2. STAM-fMRI for classification on experimental fMRI data

The fMRI data set was originally collected by Marcel Just and his colleagues at Carnegie Mellon University’s Center for Cognitive Brain Imaging (CCBI) [21]. The fMRI recorded 5,062 voxels from the whole brain volume while a subject was performing a cognitive reading task. In this task, there were two categories of sentences (affirmative and negative) each was remined on the screen for 8 seconds correspond to 16 measured brain images in fMRI. There were a total number of 40 sentences.

A NeuCube STAM-fMRI model is developed for classification of fMRI samples into two classes (class 1: affirmative sentence and class 2: negative sentence). Signal to noise ratio (SNR) feature selection method was applied to the fMRI data to select vital fMRI voxels with a high power of discrimination between the defined classes. As shown in Figure 4a, we selected 20 top important voxels that had SNR values higher than 0.4 threshold. These 20 fMRI voxels from 40 fMRI samples (related to reading 40 sentences) were used as input variables to train STAM-fMRI model for classification. To this end, we defined two sets of experiments as below:

• Experiment 1 (Section 4.2): training and testing the SNN model using the whole space and time information of fMRI dat. The results are shown in Figure 4a-c.
• Experiment 2 (Section 4.3): training the SNN model using the whole space (voxels) and time information of fMRI data but testing the model using a smaller temporal length of fMRI data. The results are shown in Figure 5.
• Experiment 3 (Section 4.4): training the SNN model using the whole space (voxels) and time information of fMRI data but testing the model using a smaller portion of the spatial information (a smaller number of fMRI variables/voxels). The results are shown in Figure 6.

The fMRI data dimension is defined by the maximum value of x, y, and z coordinates of voxels that forms a volume size of 51×56×8. In this dimension, 5062 voxels are captured. Figure 4a shows that 5,062 voxel coordinates are mapped into a 3-dimensional SNNcube. T top-informative voxels selected via SNR are shown in Figure 4b, which are used as input variables to train the SNNcube model. Table 1 shows the brain region of interest (RoI) associated with these top-20 selected fMRI voxels.

Informative voxels are selected using an SNR feature selection from the fMRI data. The voxels were selected due to their SNR values being higher than a threshold= 0.4. The number of of the selected voxels is 20 representing different areas of interest: LT (3), LOPER (3), LIPL (1), LDLPFC (6), RT (2), CALC (1), LSGA (1), RDLPFC (1), RSGA (1), RIT (1), wth the following full names of the area: left temporal lobe (LT); left opercularis (LOPER); left inferior parietal lobule (LIPL); left dorsolateral prefrontal cortex (LDLPFC) and right dorsolateral prefrontal cortex (RDLPFC); calcarine sulcus (CALC); right supramarginal gyrus (RSGA).

In this classification experiment, we employed the same number of samples, temporal and number of variables for both training and testing the SNN model for classification of fMRI samples into affirmative vs negative classes. The model achieved 100% association accuracy. This is shown in Figure 4c.

The trained model is analyzed in terms of connectivity around each of the input voxels, so that the most connected voxels point to most important areas of the brain involved. This is shown in Table 3.

Figure 5a presents three snapshots of deep learning of eight-second fMRI data in a SNNcube when a subject is reading a negative sentence (time in seconds). Figure 5b captures the internal structural pattern, represented as spatio-temporal connectivity in the SNN model trained with eight-second fMRI data streams. The corresponding functional pattern is illustrated in Figure 5c as a sequence of spiking activity of clusters of neurons in a trained SNNcube. The internal functional dimensionality of the SNN model shows that while the subject was reading a negative sentence, the activated cognitive functions were initiated from the Spatial Visual Processing function. Then it was followed by the Executive functions, including decision making and working memory. From there, the Logical and Emotional Attention functions were involved. Finally, the Emotional Memory formation and Perception functions are evoked.

area	LT	LOPER	LIPL	LOPER	LDLPFC	LOPER	LT	LDLPFC	RT	CALC
Neg	1.4	0.92	1.87	1.03	2.08	1.12	1.48	0.44	0.2	0.89
Aff	0.9	0.56	1.01	0.87	1.03	0.65	0.89	0.23	0.1	0.43
area	LSGA	LDLPFC	LT	LDLPFC	RT	LDLPFC	LDLPFC	RDLPFC	RSGA	RIT	Avg
Neg	1.84	1.03	1.9	0.45	1.1	1.26	0.56	0.19	0.43	1.4	1.7
Aff	1.04	0.68	1.1	0.17	0.8	0.24	0.22	0.11	0.32	0.9	0.6

4.3. STAM-fMRI recalled on partial temporal fMRI data

In this experiment, we used the whole spatial and temporal information from the fMRI data to train the SNN model, however, for testing the model, we used a smaller temporal length of data (from 70% to 50% of fMRI timepoints). This is illustrated in Figure 6.

4.4. STAM-fMRI, recalled on partial spatial fMRI data

This section is related to training the SNNcube with all the 40 samples of fMRI data (20 voxels recorded for 8 seconds), while testing the models by recalling it on smaller number of voxel data to evaluate the classification spatial association accuracy. This is illustrated in Figure 7.

4.5. Potential marker discovery from the STAM-fMRI

The trained STAM-fMRI model can be analysed in terms identifying the most activated brain regions related to reading affirmative and negative sentences. Figure 8 demonstrate the distribution of the average connection weights around the input voxels located in the left and right hemispheres of the trained SNN models related to different sentences.

4.6. STAM for longitudinal MRI neuroimaging

STAM systems can be developed for longitudinal MRI data (STAM-longMRI), such as the one used in [21], where 6 years MRI data has been modelled to predict dementia and AD in 2 and 4 years ahead from a large cohort of data. A STAM-longMRI system can be trained on full length of longitudinal MRI data and used to be recalled on a shorter time to predict future events. MRI images are used to preselect regents of interest (RoI), which are consequently measured as longitudinal times series. The RoI are used as input variables/features and are mapped into a SNNcube structured to a brain template according to their 3D coordinates in this template. The measurements are linearly interpolated to make more time points that still preserve the trend of changes in the data over time. This is detected in a SNNcube connectivity through learning and used to create a STAM-longMRI and to test its association and generalisation accuracy over time and over a smaller number of features.

5. Discussions, conclusions, and directions for further research

The potential applications of the STAM-EEG and STAM-fMRI become evident in various fields, including post-stroke recovery prediction, early diagnosis, and prognosis of mild cognitive impairment (MCI) and Alzheimer's disease (AD), as well as depression and other mental health conditions. These applications can leverage neuroimaging techniques such as EEG and fMRI to analyze spatio-temporal patterns of brain activity and make accurate predictions or classifications.

One notable application is in post-stroke recovery prediction. By training the STAM model on neuroimaging data collected from stroke patients, it can learn the spatio-temporal patterns associated with successful recovery. Subsequently, the model can be recalled using only partial neuroimaging variables or timepoints to predict the recovery trajectory of new stroke patients. This capability can assist clinicians in personalized treatment planning and rehabilitation strategies [23,24].

Another application lies in the early diagnosis and prognosis of MCI and Alzheimer's disease. By training the STAM model on longitudinal neuroimaging data, such as EEG and fMRI recordings, from individuals with and without MCI/AD, it can learn the complex spatio-temporal patterns indicative of disease progression. The model can then be utilized to classify new individuals based on their neuroimaging data, enabling early detection and intervention for improved patient outcomes [25,26].

Depression is another mental health condition that can benefit from the STAM systems. By training the model on neuroimaging data, such as resting-state fMRI, from individuals with depression, it can capture the spatio-temporal associations related to the disorder. This trained model can subsequently be used to classify new individuals as either depressed or non-depressed based on their neuroimaging data, aiding in early diagnosis and treatment planning [27].

Furthermore, the STAM systems hold potential for applications in neurodevelopmental disorders, such as autism spectrum disorder (ASD). By training the model on EEG data, it can identify distinctive spatio-temporal patterns associated with ASD, contributing to early diagnosis and intervention [28]. Similarly, the framework can be applied to investigate brain disorders related to aging, such as Parkinson's disease or age-related cognitive decline [29].

By incorporating multimodal spatio-temporal data, including clinical, genetic, cognitive, and demographic information, during the training phase, a STAM model can enable comprehensive analyses. This integration of multiple modalities aims to enhance the model's ability to make accurate predictions or classifications, even when only a subset of the modalities is available for recall. Such a capability can provide valuable insights for personalized medicine, treatment planning, and patient management [30].

One challenge in STAM system design is how it can effectively associate different data modalities during learning, enabling successful recall even when only one modality is available. For instance, can a STAM model learn brain data from synesthetic subjects who experience auditory sensations when they see colors? Addressing this challenge requires leveraging prior knowledge about brain structural and functional pathways, as well as stimuli data and corresponding spatio-temporal data from subjects. Current understanding of structural connectivity and functional pathways during perception can be utilized to initialize the connectivity of the SNN Cube before training [31,32,33].

Another open question pertains to how sound, image, and brain response data (e.g., EEG, fMRI) can be inputted as associated spatio-temporal patterns into dedicated groups of neurons. This concept aligns with the principles employed in neuroprosthetics, where stimulus signals are delivered to specific brain regions to compensate for damage, effectively "skipping" damaged areas [34], [35]. Experiments conducted using the STAM-SNN framework have the potential to provide insights and ideas for the development of new types of neuroprosthetics that leverage spatio-temporal associations in neural activity.

Looking ahead, a future challenge in utilizing NeuCube as a STAM is incorporating other multimodal spatio-temporal data in addition to NI data, including clinical, genetic, cognitive, demographic, and other modalities, during the training phase. Subsequently, the model should still be able to achieve accurate classifications or predictions even when only a subset of the modalities is available for recall. This challenge necessitates the exploration of novel methodologies that can effectively handle multimodal spatio-temporal data and extract meaningful patterns from diverse sources.

The STAM approached from [7] was used here for NI data, but it can be used for multisensory data streams, where spatial or temporal similarity information can be converted into spatial location of neurons in a SNNcube [36,37] and then a STAM is developed based on the methods above for early prediction of events, such as stroke [38], psychosis [39], air pollution [40,41].

STAM, based on SNN, can be implemented in neuromorphic microchips, consuming much less energy and being implantable for on-line adaptive learning and control [42-45] with wider applications [46-51].