NEeuroMOorphic Neuro-Response Decoding System for Adaptive and Personalised Neuro-Prosthetics Control

1. Introduction

Recent advancements in Brain-Computer Interface (BCI) technology significantly improved their reliability [1,2] providing revolutionary benefits to patients with neurological conditions such as stroke, spinal cord injuries, and neuro-degenerative disorders [3,4]. In particular, the BCIs applications to human prosthetics control demonstrated remarkable progress in the last decade [5,6,7,8,9]. A pioneering work using ElectroCorticoGram (ECoG) mesurements from the motor cortex in the brain to decode movement intentions of tetraplegic patients has been reported in [10,11,12]. The performance of the developed Motor Control Decoder (MCD) allows the patients to use the neuro-prosthesis for several weeks without recalibration. However, an unsupervised recalibration/adaptation would be a great added value for home use of such a BCI controlled device.

For the aim of auto-adaptation of MCD feedback of how it performs on given task is beneficial. Different methods for assessing a person’s performance and for delivering feedback are used in different scenarios [6,8,9,13,14]. For the aim of MCD training the patient was asked to imagine desired movement and recorded ECoG signals are marked accordingly. However, since there is no guarantee that the patient is satisfied by the effector actions, the need for classification of brain signals as satisfactory/unsatisfactory raised. For this aim in [13] each movement of prosthetic device provoked by a patient’s brain signals and decoded by a well trained MCD was labeled as satisfactory or not satisfactory in dependence on resulting error of the controller (MCD). Only satisfactory brain signals were used to train the MCD. Then another model for prediction of the neural response quality, called Neural Response Decoder (NRD), was trained. A further aim of NRD was to allow for auto-adaptation of the MCD in on-line mode. However, since the initial MCD, supposed to be well trained, might not be perfect such a labeling of ECoG data as satisfactory/unsatisfactory might not be always exact.

In our previous work [15] a novel neuromorphic framework of a BMI system for prosthetics control via decoding ECoG brain signals was described. It includes a three-dimensional spike timing neural network (3D-SNN) for brain signals features extraction and an on-line trainable recurrent reservoir structure (Echo state network (ESN)) for Motor Control Decoding (MCD). The decoder auto-adapts to the incoming brain signals via Spike Timing Dependent Plasticity (STDP) of connections (synapses) in the 3D-SNN structure. In further work [16] a novel technique for features extraction from raw ECoG data was proposed that improved significantly accuracy of the MCD from [15].

Based on our previous work here we upgrade our neuromorphic MCD with a NRD that should be able to predict whether the decoder actions and/or ECoG data from the patient is correct (satisfactory) or not. The main aim is to design an actor-critic structure able to adapt via reinforcement learning MCD (actor) based on NRD (critic) predictions. It would allow for continuous adaptation of the decoder to changes in brain signals in real time.

2. Materials and Methods

2.1. Experimental Data

The developed MCD-NRD structure was trained on the data base (DB) called RUNNER. Figure 1 shows the experimental set-up.

During the data collection experiment the patient was seated in front of a computer screen where a human avatar was represented from a third person perspective. The avatar could either stand still or walk forward at a fixed speed. The RUNNER DB was collected from a patient with traumatic sensorimotor tetraplegia caused by a complete C4–C5 spinal cord injury and having implanted two chronic wireless ECoG WIMAGINE implants [17] composed of 64 planar electrodes [13,14]. The patient was involved in the “BCI and Tetraplegia” clinical trial at CEA/Clinatec (NCT02550522), which focuses on the recording and decoding of motor intentions with different effectors. Here, the patient controlled the avatar using leg motor imagery decoded by a trained MCD. Totally 9 sessions, of on average 11 min of recording, were acquired over 99 days. For blind-test of the algorithms, the database has been split in two parts: the first 7 sessions were used to train the decoder while the last 2 were left for testing the trained model accuracy.

The $Stat{e}_{desired}$, i.e. the prescribed on the screen avatar state that should be predicted by MCD, has two possible values: $idle$ or $walk$. The satisfaction of the decoded brain signals and achieved new state of the avatar (denoted briefly as $SATIS$) was marked according to the procedure described in [13,14] as follows: after a lag of time when the instruction on the screen was changed if the avatar is in the $Stat{e}_{desired}$ as a result of proper decoding of patient’s ECoG signals by the MCD, it is considered as satisfactory ($SATIS=1$) while if it is not - the ECoG signal was marked as non satisfactory ($SATIS=0$).

However, the way of labeling does not relay on patient’s opinion. Since the avatar states were decoded by a pre-trained MCD module, the decoding error could result either from wrong patients’ brain signals due to fatigue and distraction or from MCD incorrectness. Hence such a labeling could serve to predict an error in desired movement decoding with unknown origin.

Nevertheless, the experimental data base is a good starting point to train initial NRD that could be further refined using feedback from the patient.

2.2. Neuromorphic Framework for MCD and NRD

In our previous work [15] we described a neuromorphic structure called MCD. It was trained to predict desired by patient avatar actions from ECoG signals. Here we upgrade it with a NRD structure whose aim is to predict correctness (satisfaction) of the actions decoded by the MCD. The basic idea comes from the reinforcement learning theory [18] that is considered as a biologically plausible way our brain learns from experience via interacting with the environment and receiving a feedback about the resulting outcome. The feedback signal is called $reinforcement$ and could be very simple, e.g. a binary label $good=1$ or $bad=0$.

Figure 2 presents the proposed reinforcement learning framework. In order to transfer the terminology from the experimental set-up to the terms in control theory and reinforcement learning, we refer to the patient as an $object$ under control whose state ($objec{t}_{state}$) is assessed by the extracted features from the signals measured by the ECoG implants; the MCD will be refereed further as the $actor$ (controller) that has to be optimized to generate $Action$ resulting in satisfaction ($SATIS=1$) rather than to non satisfaction ($SATIS=0$); the satisfaction itself is considered as binary $reinforcementsignal$ to be predicted by the $critic$ element (in our scheme the NRD). Table 1 summarizes the correspondence between experimental and RL terms.

Thus the MCD role is to generate an $Action$ based on the perceived object’s state ($ECoGfeatures$) while the NRD should predict the outcome ($SATIS$) from the $Action$ and perceived object’s state. In situation, when the movement desired by the patient is not known, a perfectly trained NRD should be able support MCD in generation of proper motion of end effectors of the exoskeleton supporting the patient. However, since the labels in RUNNER DB described above were obtained without feedback from the patient, we can’t be sure whether a wrong avatar movement is due to fatigue or distraction of the patient or because the MCD was not perfectly trained. If in the experimental set-up a feedback from the patient ($SATI{S}_{labelfrompatient}$) is included, the RL framework will allow for simultaneous training of MCD and NRD.

The overall neuromorphic structure upgrades the MCD described in details in [15] as shown on Figure 3. It consists of the following basic modules:

• Filtering module that transforms the raw ECoG signals to input signal for 3D-SNN using Morlet wavelet transformation for multiple fundamental frequencies and their combination into a features matrix of same size as the original one [16]
• 3D recurrent SNN architecture called 3D SNN cube, spatially structured and adaptable to an individual 3D brain template, for feature extraction from processed ECoG signals. It adapts continuously to the incoming input in unsupervised mode via STDP rule.
• Two recurrent Echo State Network (ESN) structures for decoding of the desired movement (MCD) and satisfaction (NRD) from extracted features (spiking frequencies of the selected neurons in the 3D-SNN module). It can be trained on-line in supervised mode via recursive least squares (RLS) or in unsupervised regime via reinforcement learning (RL) rules.

The novel approach for ECoG signals’ features extraction is described in details in [16]. It consists of combination of 15 wavelet transformations of every portion of 59 records from the 64 ECoG electrodes’ signals for approximately 100 ms using Morlet wavelets with 15 different fundamental frequencies (from 10 to 150 Hz with step of 10 Hz). The obtained filtered ECoG signals on Figure 3 are a matrix of the same size as the original one, i.e. $59\times 64$. It is fed to the 3D SNN cube as generating currents for the time period of approximately 100 ms to each one of the 64 neurons in the structure. The firing rates during the stimulation period of the 3D SNN cube are the ECoG features from 3D SNN fed to both MCD and NRD modules.

The top part of Figure 4 shows the 3D SNN cube structure. It consists of 64 spiking neurons (simulated by the Leaky Integrate and Fire (LIF) model) positioned at the ECoG electrode positions in the 3D space. The synaptic connections between each pair of neurons were randomly generated with weights proportional to their distance in the 3D space. The positive connections are marked in red while the negative once - in blue. All synapses are plastic, i.e. their strength (weight) adapts continuously to the input signals via the associative learning rule called Spike Timing Dependent Plasticity (STDP). The auto-adaptation in the 3D SNN structure reflects the spatio-temporal dependence of ECoG signals thus accounting for both mutual positions of the ECoG electrodes and changes of the recorded signals with time. Bottom part on Figure 4 shows the initial and two snapshots (after training and after testing) of the connections weights in the 3D-SNN structure. The color of each dot corresponds to the magnitude of the connection weight between each pair of neurons, numbered from 1 to 64 on both x and y axes. We observe that the initial weights (most left plot) converge very fast to connectivity corresponding to the model input signals and continue with small adaptation changes during training (middle plot) and testing (right plot) phases.

Both actor and critic modules are fast trainable recurrent neural network structures called Echo state networks (ESN) [19]. They consist of a randomly connected pool of neurons with hyperbolic tangent nonlinear activation function and a linear readout with weights trainable via least squares method.

The MCD is trained to predict the desired movement ($Action=idle/walk$) of the prosthetic device (in current experiment the avatar on the screen) while the NRD - to predict whether the brain signal from the ECoG generated desired ($SATIS=1$) or incorrect ($SATIS=0$) movements. The output of NRD is considered as prediction of reward/punish signal in form of satisfaction/no satisfaction. This signal should allow for adjustment of the actor behavior (MCD predictions) so as to decrease its error.

2.3. Software Implementation

All modules are written in Python, version 3.8.9. The 3D-SNN is based on NEST simulator library, version 3.3 [20] while the rest of code exploits numpy, SciPy and other Python libraries for mathematical calculations. The software works in pseudo-online mode reading from $D{B}_{edf}$ files the next portion of 59 records from ECoG electrodes, desired decoder outputs ($Actio{n}_{desiredfromDB}$ and $SATI{S}_{labelfromDB}$) from $D{B}_{beh}$ file and generating as output $Actio{n}_{predicted}$ as well as $SATI{S}_{predicted}$ for current time period. During training the model parameters are adjusted in supervised mode while during testing the model output is kept in csv files.

The adjustable in supervised mode parameters of the model are the output connection weights of the ESN modules. They are tuned incrementally with every new input/output training data pair via recursive least squares (RLS) method. The 3D-SNN connection weights auto-adapt via the STDP rule continuously. The 3D SNN cube state is the membrane potentials of all the neurons in the structure. All model parameters are kept in csv files.

3. Methodology

We investigated four training approaches and performed three testing experiments described further.

3.1. Training Approaches

The training algorithm is shown as Algorithm 1. $TA1$, $TA2$, $TA3$ and $TA4$ denote the four training approaches.

Algorithm 1 Pseudo-code of training algorithm

Initialization Initialize $ES{N}_{MCD}$ and $ES{N}_{NRD}$ modules parameters Compose 3D-SNN module using ECoG positions Initialize the cube connection weights based on neurons’ distances      while $newdata$ do 2:    $ECoGsignalsfortimeperiod{t}_{ECoG}\leftarrow ReadfromD{B}_{edf}file$    $Actio{n}_{desiredfromDB}andSATI{S}_{labelfromDB}\leftarrow ReadfromD{B}_{beh}file$ 4:    $filteredECoGsignals\leftarrow filter\left(ECoGsignals\right)$    $3D-SNNinput\leftarrow filteredECoGsignals$ 6:    for $t_{ECoG}$ do      $Simulate3D-SNN$ 8:    end for    $ECoGfeatures\leftarrow spikingfrequenciesof3D-SNN$ 10:   if MCD training then      $ES{N}_{MCD}\leftarrow ECoGfeatures$ 12:     $Actio{n}_{predicted}\leftarrow ES{N}_{MCD}output$      if TA1 or TA3 then 14:        $MC{D}_{error}=Actio{n}_{predicted}-Actio{n}_{desiredfromDB}$      else if TA2 or TA4 then 16:         if $SATI{S}_{labelfromDB}=1$ then           $MC{D}_{error}=Actio{n}_{predicted}-Actio{n}_{desiredfromDB}$ 18:         else if $SATI{S}_{labelfromDB}=0$ then           $MC{D}_{error}=Actio{n}_{predicted}-revertedActio{n}_{desiredfromDB}$ 20:        end if      end if 22:      $newES{N}_{MCD}parameters\leftarrow RLS\left(MC{D}_{error}\right)$   else if NRD training then 24:      if TA1 then         $ES{N}_{NRD}\leftarrow [ECoGfeatures,Actio{n}_{desiredfromDB}]$ 26:      else if TA2 then         if $SATI{S}_{labelfromDB}=1$ then 28:           $ES{N}_{NRD}\leftarrow [ECoGfeatures,Actio{n}_{desiredfromDB}]$         else if $SATI{S}_{labelfromDB}=0$ then 30:           $ES{N}_{NRD}\leftarrow [ECoGfeatures,revertedActio{n}_{desiredfromDB}]$         end if 32:      else if TA3 then         $ES{N}_{NRD}\leftarrow [ECoGfeatures,Actio{n}_{predicted}]$ 34:      else if TA4 then         if $SATI{S}_{labelfromDB}=1$ then 36:          $ES{N}_{NRD}\leftarrow [ECoGfeatures,Actio{n}_{predicted}]$         else if $SATI{S}_{labelfromDB}=0$ then 38:          $ES{N}_{NRD}\leftarrow [ECoGfeatures,revertedActio{n}_{predicted}]$         end if 40:     end if     $SATI{S}_{predicted}\leftarrow ES{N}_{NRD}output$ 42:     $NR{D}_{error}=SATI{S}_{predicted}-SATI{S}_{labelfromDB}$     $newES{N}_{NRD}parameters\leftarrow RLS\left(NR{D}_{error}\right)$ 44:   end if end while 46: $Keepmodelparameters$

Since in case of non satisfaction it is supposed that MCD should generate the opposite state, first we’ve performed the following two training approaches:

• $Firsttrainingapproach$ denoted further as TA1 (Figure 5): Use desired state of MCD from the DB denoted as $Actio{n}_{desiredfromDB}$ as target for MCD and input to NRD no matter whether the training example is labeled as satisfaction or non satisfaction.
• $Secondtrainingapproach$ denoted further as TA2 (Figure 6): Use swapped desired state of MCD denoted as $revertedActio{n}_{desiredfromDB}$ (if $idle$ revert to $walk$ and vise versa) as target fro MCD and input for NRD if the training example is labeled as non satisfaction in the DB ($SATI{S}_{labelfromDB}=0$).

Since in real time the decoder will not be aware of exact target movement of the test subject, it is important to be able to rely on predictions from MCD to train and test NRD. In order to start training of NRD with better trained MCD we performed also the next two training approaches:

• $Step1$: Use the $Firsttrainingapproach$ (TA1) to train initial model of both MCD and NRD using only first training session from the DB.
• $Step2$: For the rest of the training sessions use the $Thirdtrainingapproach$ denoted as TA3 (Figure 7) or $Fourthtrainingapproach$ denoted as TA4 (Figure 8).

The $Third$ (TA3) and $Fourth$ (TA4) training approaches exploit the idea from the previous once: to use the $Actio{n}_{desiredfromDB}$ or to invert it (reverted $Actio{n}_{desiredfromDB}$) based on the value of the $SATI{S}_{labelfromDB}$ for training of the MCD.

All training approaches relay on model target output data from DB so they could be implemented in off-line mode. In real time, if the target output was available, the RLS algorithm will allow to adjust the model parameters in on-line mode too.

3.2. Testing Experiments

The testing algorithm is shown as Algorithm 2. $TE1$ and $TE2$ denote the first and second testing experiment, described in next section.

Algorithm 2 Pseudo-code of testing algorithm

Initialization Set $ES{N}_{MCD}$ and $ES{N}_{NRD}$ modules parameters to the trained once Compose 3D-SNN module using ECoG positions Set 3D SNN state to the achieved after training Set cube connection weights to the achieved after training values while $newdata$ do 2:    $ECoGsignalsfortimeperiod{t}_{ECoG}\leftarrow ReadfromD{B}_{edf}file$    $Actio{n}_{desiredfromDB}\leftarrow ReadfromD{B}_{beh}file$ 4:    $filteredECoGsignals\leftarrow filter\left(ECoGsignals\right)$    $3D-SNNinput\leftarrow filteredECoGsignals$ 6:    for $t_{ECoG}$ do      $Simulate3D-SNN$ 8:    end for    $ECoGfeatures\leftarrow spikingfrequenciesof3D-SNN$ 10:   $ES{N}_{MCD}\leftarrow ECoGfeatures$   $Actio{n}_{predicted}\leftarrow ES{N}_{MCD}output$ 12:   if TE1 then     $ES{N}_{NRD}\leftarrow [ECoGfeatures,Actio{n}_{desiredfromDB}]$ 14:   else if TE2 then     $ES{N}_{NRD}\leftarrow [ECoGfeatures,Actio{n}_{predicted}]$ 16:   end if   $SATI{S}_{predicted}\leftarrow ES{N}_{NRD}output$ 18: end while $Keep3DSNNstateandcinnectionweights$

The three testing experiments are summarized as follows:

• $Firstexperiment$ denoted further as TE1 (Figure 9): feed the trained NRD with desired action from DB ($Actio{n}_{desiredfromDB}$) rather than from the trained MCD prediction. In this way we skip MCD imitating knowledge about instructions on the screen. However, in on-line mode the NRD must know the target action that is not always possible.
• $Secondexperiment$ denoted further as TE2 (Figure 10): feed the trained MCD prediction ($Actio{n}_{predicted}$) to NRD that are not always correct but will be available in real situation. In this way the decoder works in fully on-line mode.
• $Thirdexperiment$ denoted further as TE3: Testing of both models trained via the Third (TA3) and Fourth (TA4) training approaches was done as in the Second experiment TE2, i.e. in on-line mode.

4. Results

For the training and testing of both MCD and NRD modules we use the fully labeled data from training sessions of RUNNER DB, i.e. sessions from 1 to 7 for training and the rest of sessions (8 and 9) - for testing.

Table 2 and Table 3 show the NRD testing accuracy from the $Firstexperiment$ and the $Secondexperiment$ for both models trained using $First$ and $Second$ training approach respectively. For the $Firstexperiment$ we observe that when the NRD was trained using information about desired avatar movement from the instructions on the screen, it could be better trained to predict $SATIS$ label. In case the desired actions were replaced by MCD predictions as in the $Secondtrainingapproach$ however, the NRD accuracy drops significantly that is expected since the MCD is not perfectly trained.

$Secondexperiment$ results show that even if the model was trained using exact information about target movement as in the $Firsttrainingapproach$, testing with trained MCD predictions yields decreased accuracy in comparison with $Firstexperiment$ . Again $Secondtrainingapproach$ results in lower model accuracy.

Testing of both models trained via the $Third$ (TA3) and $Fourth$ (TA4) training approaches was done as in the $Secondexperiment$TE2 (Figure 10). Table 4 shows testing accuracy of the NRD from the third experiment. It is above the accuracy achieved from the $Secondexperiment$ and below the accuracy from the $Firstexperimet$. The $Fourthtrainingapproach$ yielded model with better accuracy that is expected result.

Finally, we tested whether predictions from the trained NRD could be applied to improve the MCD accuracy as it is shown by the dashed arrow from NRD to MCD on Figure 10. The results from the $First$ and $Second$ experiment are the same since the MCD was trained in the same way in both cases. They are shown in Table 5. Table 6 shows the MCD accuracy from the $Thirdexperiment$.

We did not observe significant difference of MCD accuracy in $First$/$Second$ and $Thirdexperiment$. However, in all experiments using training approaches TA2 ($Second$) and TA4 ($Fourth$) with reverting of $Actio{n}_{desiredfromDB}$ in case of $SATI{S}_{labelfromDB}=0$ yielded better accuracy of MCD in comparison with the $First$ (TA1) and $Third$ (TA3) training approaches (without reverting of $Actio{n}_{desiredfromDB}$). This proves the hypothesis that even if NRD is not perfectly trained, its predictions can be applied to improve MCD accuracy during real time work of the decoder.

5. Discussion

The results about NRD accuracy reported here demonstrate that if the NRD is given the target action to be performed by MCD it is able to distinguish correct from incorrect decoding of brain signals with much higher accuracy in comparison with the case when it is given only the brain signals. However, if we start training both NRD and MCD using knowledge about target actions and then let the NRD use pretrained MCD predictions, the achieved NRD accuracy is significantly higher.

The testing results demonstrated also that even non-perfectly trained NRD could improve the MCD predictions in a pseudo on-line mode if the MCD was trained with reverting of $Actio{n}_{desiredfromDB}$ in case of $SATI{S}_{labelfromDB}=0$.

In order to be able to exploit the NRD predictions for auto-adaptation of MCD, the proposed RL framework needs to be applied in real time with the patient in the loop as in Figure 2. This will allow communication with the patient and observation of changes in his/her brain signals, provoked by adjustment of both MCD and NRD. Further work in this direction using true reinforcement learning algorithm will allow to fully utilize the NRD for MCD performance optimization in on-line manner.

6. Conclusions

The proposed neuromorphic NRD system inspired by reinforcement learning theory proves to be a good idea to be explored further. Even though we did not have the opportunity to perform real time experiments with patient, the collected experimental data base allowed to test several training and testing approaches. We demonstrated that if training of NRD starts with information about correct target behavior of MCD, this will be a good starting point for further reinforcement training of both MCD and NRD parts of the decoder.

Further experiments in real time with patient in loop would allow for even better results. We believe that the patient must be asked about his satisfaction in order to achieve better auto-adaptation of his brain signals decoder during its exploitation in real life situations. Reinforcement learning would allow to continuously re-adjust the decoder in real time.