Reinforcement Learning Using Spike-Timing-Dependent Plasticity for Image Recognition

1. Introduction

Image recognition has become ubiquitous, powering applications such as self-driving cars, medical image analysis, and facial recognition systems. Deep learning approaches [1] have achieved remarkable success, but their reliance on extensive computations requires powerful hardware, leading to high energy consumption and hindering deployment in devices with limited resources. Furthermore, deep learning models often lack biological plausibility, limiting their potential to understand neural computation and develop neuromorphic systems that mimic brain-like processing. Spiking neural networks (SNNs) [2] offer a bioinspired alternative to deep learning for image recognition. SNN neurons communicate through discrete spikes, mimicking the information processing of biological neurons. This spiking behavior promises lower power consumption and potentially better reflects the brain’s processing capabilities. Spike Timing-Dependent Plasticity (STDP) [3] is a learning rule that allows SNNs to learn by dynamically adjusting synaptic strengths based on the timing of pre- and postsynaptic neuron spiking. This learning mechanism allows SNNs to extract features from the data, which makes them potentially well suited for image recognition tasks.

Various STDP learning rules and SNN architectures have been explored [4]. Unsupervised learning, leveraging STDP, allows SNNs to autonomously acquire selectivity to recurring input patterns without external guidance. Supervised learning in SNNs using backpropagation is hindered by the nondifferentiable nature of spiking neurons and the “weight transport” problem [5]. While significant progress has been made in addressing these challenges, opportunities remain to develop more bioplausible solutions that better capture the complexities of biological systems. Beyond backpropagation, other supervised learning approaches include optimizing the probability of desired output spikes or implementing competitive dynamics among output neurons that represent distinct data classes. In terms of architectural design, researchers have proposed various SNN models, including deep-fully-connected SNNs, spiking convolutional neural networks, and spiking deep belief networks. These models have demonstrated performance comparable to traditional deep neural networks on a range of tasks.

This paper presents a supervised learning approach for SNNs that integrates gradient-free optimization with STDP for image classification on the MNIST dataset [6]. To enhance class selectivity, each hidden-layer neuron is assigned to a unique group that corresponds to a specific digit class. Synaptic weights are updated only when the target class aligns with the neuron’s group, promoting specialization. A unique feature of the proposed approach is the absence of inhibitory synaptic connections between neurons during the training stage. This allows neurons to fully explore the input space, fostering the development of class preferences. Lateral inhibition, introduced only during validation and testing, enables competitive dynamics among neurons, leading to a clean classification based on firing rates. The network architecture is scalable, supporting the construction of deep SNN models for complex tasks. The experimental results on MNIST demonstrate the efficacy of the proposed method in achieving good classification accuracy. The SNN simulations are performed using the Brian2 simulator v2.7.1 [7] on HPC resources provided by the Purdue Rosen Center for Advanced Computing [8]. The software package and optimized hyperparameters for this simulation are provided on GitHub1.

The remaining sections of this paper are organized as follows. Section 2 presents the neuron and synapse models, as well as the network architecture. Section 3 details the implementation of supervision in the STDP learning rule. Section 4 describes the hyperparameter tuning process. Section 5 evaluates the performance of the network on the MNIST dataset. Section 6 concludes the paper.

2. Network Design

2.1. Modeling Neurons and Synapses

The leaky integrate-and-fire model is chosen as a computational framework to simulate neuronal dynamics. Neurons are simplified as electrical circuits, incorporating capacitance and resistance, while essential features are captured without undue complexity. The membrane potential (V) evolves over time according to the following differential equation:

$${\tau }_m{\displaystyle \frac{dV}{dt}}=(E_{\mathrm{rest}}-V)+g_{\mathrm{e}}(E_{\mathrm{exc}}-V)+g_{\mathrm{i}}(E_{\mathrm{inh}}-V)$$

(1)

where $E_{\mathrm{rest}}$ represents the resting membrane potential, $E_{\mathrm{exc}}$ and $E_{\mathrm{inh}}$ denote the equilibrium potentials of the excitatory and inhibitory synapses, respectively, $g_{\mathrm{e}}$ and $g_{\mathrm{i}}$ correspond to the conductances of the excitatory and inhibitory synapses, and ${\tau }_m$ signifies the time constant of neurons. When a neuron’s membrane potential exceeds its threshold, it generates an action potential and subsequently resets its potential to $V_{\mathrm{reset}}$. The threshold, initially set at $V_{\mathrm{thres}}$, is dynamically adjusted based on the neuron’s firing rate. During the subsequent refractory period, neurons can not spike again. Synaptic strengths are modulated by changes in conductance. When a presynaptic spike occurs at the synapse, synapses increase their conductance by synaptic weight $\mathit{w}$. However, when the neuron is not active, the conductance decays exponentially. This dynamic can be mathematically described as

$${\tau }_{g_e}{\displaystyle \frac{d{g}_e}{dt}}=-g_e,{\tau }_{g_i}{\displaystyle \frac{d{g}_i}{dt}}=-g_i$$

(2)

where ${\tau }_{g_e}$ and ${\tau }_{g_i}$ represent the time constant of the excitatory and inhibitory conductance of a postsynaptic neuron, respectively.

Table 1 presents the fixed parameters of the neuron and synapse model. These parameters remain constant for neurons and synapses in different layers of the network. Other parameters, such as ${\tau }_m$, ${\tau }_{g_e}$, ${\tau }_{g_i}$ are determined through an optimization process as described in Section 4.

2.2. Adaptive Threshold

Balanced neuronal activation is essential for optimal network performance. To prevent individual neurons from unduly dominating network responses, an adaptive membrane threshold mechanism, inspired by the work in [9,10], is implemented. The adaptive threshold ($V_t$) is initialized at $V_{thres}$ and increases by a small value upon each neuronal firing. The small incremental change is calculated as $\Delta V_t{\Theta }_{vt}$, where ${\Theta }_{vt}$ prevents the threshold of a neuron from reaching too high and stops firing.

$${\Theta }_{vt}=\left(0.5-0.5tanh\left(\left({\displaystyle \frac{-2(V_t-V_{thres}(V_{tshift}-0.5))}{V_{thres}}}+1\right)/V_{tscale}\right)\right)$$

(3)

Here $\Delta V_t$ is the maximum potential increase upon each neuronal firing, $V_{tshift}$ represents the point at which ${\Theta }_{vt}$ start to decrease significantly from one, and $V_{tscale}$ regulates the rate of decrease. This formulation prevents excessive threshold elevation, thus maintaining neuronal responsiveness. When a neuron is not active, $V_t$ decreases exponentially as a function of time with a time constant ${\tau }_{adpt}$, as described by:

$${\tau }_{adpt}{\displaystyle \frac{d{V}_t}{dt}}=(V_{thres}-V_t)$$

(4)

2.3. Network Architecture

The network architecture is a hierarchical feedforward structure composed of an input layer and hidden layers, as illustrated in Figure 1. The input layer comprises 784 neurons, each representing a pixel of a MNIST image. These neurons encode image information through Poisson-distributed spike trains, with firing rates directly proportional to pixel intensities. Each hidden layer is divided into ten groups of neurons, with each group dedicated to recognizing a specific class of digit. A hidden layer unit, as depicted in Figure 1a, consists of ten neurons, one for each digit group. All neurons within a unit share identical spatial coordinates (X, Y) positioned uniformly above the input layer. For this study, a fully connected architecture is employed, where the first hidden layer is fully connected to the input layer, and subsequent hidden layers maintain full connectivity to both the input layer and all preceding layers. The spatial coordinates assigned to neurons offer potential for future enhancements, such as localized connectivity between layers. To enhance digit selectivity and inter-digit competition during testing and validation, inhibitory connections are established between neurons belonging to different digit groups. The firing of a neuron in one group suppresses the activity of neurons in other groups, promoting focused digit representation. To facilitate the development of digit preferences by all neurons during training, these inhibitory connections are absent in the training phase. The image classification is determined by the group exhibiting the highest firing rate in the last hidden layer.

3. STDP with Supervised Learning

Excitatory synaptic weights are adjusted using the triplet STDP learning rule [11], known for its superior performance as demonstrated in [12]. According to this learning rule, a presynaptic spike at time $t^{pre}$ introduces a weight change:

$$w\left(t\right)\to w\left(t\right)-o_1\left(t\right)[A_2^{-}+A_3^{-}{r}_2(t-\epsilon )]\mathrm{if}t=t^{pre}$$

(5)

Conversely, a postsynaptic spike at time $t^{post}$ results in a weight change:

$$w\left(t\right)\to w\left(t\right)+r_1\left(t\right)[A_2^{+}+A_3^{+}{o}_2(t-\epsilon )]\mathrm{if}t=t^{post}$$

(6)

Here, $r_1$ and $r_2$ are presynaptic event detectors, while $o_1$ and $o_2$ are postsynaptic detectors, each with biological counterparts. $A_2^{+}$ and $A_2^{-}$ quantify the weight changes for pre-post pair or a post-pre pair, respectively. $A_3^{+}$ and $A_3^{-}$ represent the amplitude of the triplet term for potentiation and depression. For simplicity, the minimal model ($A_2^{+}$ = 0, $A_3^{-}$ = 0), which is intended to fit the visual cortex data, is used for this study.

The supervison is incorporated by modifying eq.(6) as follows:

$$w\left(t\right)\to w\left(t\right)+r_1\left(t\right)A_3^{+}{o}_2(t-\epsilon ){\Theta }_w(1-\beta )$$

(7)

where $\beta$ = 0 if the postsynaptic neuron group index matches the stimulus label, otherwise $\beta$ = 1. The variable ${\Theta }_w$ prevents premature weight saturation, and is defined as:

$${\Theta }_w=\left(0.5-0.5tanh\left(\left({\displaystyle \frac{2(w+w_{max}(w_{shift}-0.5))}{w_{max}}}-1\right)/w_{scale}\right)\right)$$

(8)

with $w_{shift}$ determining the oneset of a significant decrease in ${\Theta }_w$ start from one, and $w_{scale}$ regulates the rate of decrease.

To promote equitable participation of neurons in network dynamics [13], the average weights of a postsynaptic neuron are normalized to $\lambda w_{max}$ with $w_{max}$ being the maximum possible weight. This normalization is achieved by scaling each synaptic weight according to the following formula:

$$w\to w·{\displaystyle \frac{\lambda w_{max}}{w_{\mathrm{tot}}/N_{\mathrm{pre}}}}$$

(9)

where $w_{tot}$ is the sum of all synaptic weights of a postsynaptic neuron and $N_{pre}$ is the number of presynaptic neurons connected to it. The hyperparameter $\lambda$ ($0\le \lambda \le 1$) controls the degree of normalization of the weight. This procedure allows synaptic weight to have ample space to grow with a large sample of stimuli and is performed after processing each stimulus.

4. Hyperparameter Optimization

Hyperparameter optimization was conducted using the Hyperopt algorithm [14] within the Ray framework [15]. Each stimulus was presented to the network for 0.5 seconds. The input layer converted images into Poisson spike trains, starting with a mean firing rate proportional to 25% of the original MNIST image intensity. To ensure sufficient spiking activity, the firing rate was adaptively increased by increments of 25% of the intensity until the spike count in each hidden layer reached a minimum of five, or the full intensity was attained. Preliminary experiments indicates that image recognition accuracy is relatively insensitive to this intensity modulation. Table 2 outlines the hyperparameter search ranges for different network configurations. A base network consists of a single hidden layer with one hidden layer unit, while a 2-hidden-layer network comprises two hidden layers.

Hyperparameter optimization was performed on a subset of 10,000 MNIST images for a single epoch and validated with a subset of 1000 MNIST images. For the optimization of a multilayer network, all hyperparameters except for ${\lambda }_{e\to e}$, $w_{e\to e}\left(max\right)$ and $ma{x}_{delay}$, were randomly selected from sets validated to have good performance during base network optimization. The parameters ${\lambda }_{e\to e}$ and $w_{e\to e}\left(max\right)$ can be influenced by the size of the preceding hidden layers, while the $ma{x}_{delay}$ between hidden layers may depend on the $ma{x}_{delay}$ between hidden layer and input layer. In this study, the sizes of all hidden layers in a multilayer network are fixed at 10 neurons and the optimized ${\lambda }_{e\to e}$, $w_{e\to e}\left(max\right)$ and $ma{x}_{delay}$ are thus tailored for this structure. Multiple hyperparameter sets yielded comparable validation performance under these conditions and Table 3 presents one set of optimal hyperparameters identified in this study. It is acknowledged that hyperparameter with better performance may be attainable with access to greater computational resources.

5. Results and Discussion

The performance of two different network training methods was evaluated using ${10}^4$ testing samples. The direct approach involves training the network on a fixed number of training samples and then testing it as is. The disadvantage of this method is that the training time is proportional to the number of excitatory neurons in the hidden layers. As the number of neurons increases, the training process can become very slow. To overcome this limitation, we leverage the fact that the smallest network in this model, referred to as the base network, consists of only 10 neurons, each responsible for identifying a single MNIST label, with no lateral excitatory connections between them. During training, a larger network with, for example, one hidden layer containing 250 hidden neurons, can be decomposed into 25 base networks, each processing a different subset of training samples via a single computing process (worker). During testing, the 25 base networks are combined, applying inhibitory connections within each base network among neurons belonging to different groups. Each base network reads in its own respective training synaptic weights and neuron threshold. This approach significantly reduces training time, as each worker process only trains a smaller network. Furthermore, each small network uses a unique set of hyperparameters that have been validated to have good performance during the optimization stage, resulting in a combined network with a diverse set of hyperparameters. This approach is termed “parallel training with diversity”.

Figure 2 presents testing accuracy results for networks trained using parallel training with diversity and direct training approaches. The data points of different colors represent the results with different network size and training approaches. The spike counts from all neurons within each neuron group in the last hidden layer are summed, and the group with the maximum spike counts is treated as the prediction of the network. For some stimuli, more than one neuron groups have the same maximum spike counts. In this case, if one of these groups matches the input label, it is counted as a correct identification, albeit with ambiguity. The left panel of the figure illustrates the dependence of the overall testing accuracy on size of the training sample per worker, including those identified with ambiguity. The right panel displays the fraction of correctly identified stimuli with ambiguity, i.e., test accuracy with ambiguity. The result labeled “250 (I)” comes from a network with one hidden layer containing 250 excitatory neurons, trained using parallel training with diversity. The result labeled “250 (II)” is from a network with two hidden layers, each containing 250 excitatory neurons, also trained using parallel training with diversity. In both cases, 25 workers are used during the training stage. The other results are from networks of various sizes trained using the direct training approach. Using the direct approach, the overall accuracy goes from $\sim 26\%$ to $\sim 89\%$ as the size of the training sample grows from 10 to 5$\times {10}^4$. Increasing the number of excitatory neurons in the hidden layer from 10 to 250 does not improve the overall test accuracy but reduce ambiguity by more than 10 folds. For smaller training samples, parallel training with diversity significantly outperforms the direct approach. As the training set size increases, their accuracies converge. However, the test accuracy with ambiguity using parallel training with diversity is significantly lower with the same network configuration. Compared to the single hidden layer network (250(I)), adding a second hidden layer with the specified configuration (250(II)) does not enhance the performance. This suggests that the information processed from the first hidden layer is too weak to influence the decision-making of the second layer. Other configurations, such as increasing the number of neurons in the first hidden layer, introducing lateral connections, or localized feedforward synaptic connections, may improve the network performance, as demonstrated for artificial neuron networks.

Figure 3 illustrates the spike counts from each neuron group in response to a specific label in the test sample, obtained through parallel training with diversity when the number of training samples per worker is 10. The network comprises of 250 neurons in the single hidden layer, with 25 workers (denoted as 250(I) in Figure 2). In this setup, each training stimulus is exposed to the network only once on average. Due to the limited number of training samples, the synaptic weights have not fully specialized, leading to similar responses to a given stimulus across all neuron groups. Notably, even with such small differences, the network achieves a $\sim 40\%$ testing accuracy with low ambiguity. In contrast, Figure 4 displays the spike count distribution with a significantly larger training sample size per worker ($3\times {10}^4$). With this extensive training set, the synaptic weights of different neuron groups have become more specialized. Consequently, the spike rate of the correct neuron group is substantially higher than that of other groups, indicating improved discrimination.

Figure 5 illustrates the testing accuracy as a function of input label for various numbers of excitatory neurons in a hidden layer. The left panel displays the overall accuracy, while the right panel shows the accuracy with ambiguity. The number of training samples per worker is $3\times {10}^4$. The color coding is consistent with Figure 2. Notably, the number of neurons in the hidden layer has a minimal impact on the overall test accuracy but significantly affects the test accuracy with ambiguity. For instance, the test accuracy with ambiguity for a network with 250 neurons trained directly is more than 10 times lower than that of a network with 10 neurons. With 250 neurons, the network trained through parallel training with diversity shows an accuracy with ambiguity approximately 2 times lower. The highest overall accuracy is achieved for input label 0, which also has the lowest ambiguity. In contrast, input label 5 yields the lowest overall testing accuracy. The ambiguity increases from label 0 to label 5, then decreases at label 6, and subsequently increases again until label 9.

Figure 6 displays the distribution of incorrect predictions for various input labels using a 1-hidden layer network with 250 neurons (10 neurons per worker) trained through parallel training with diversity. Each worker received $3\times {10}^4$ training stimuli, and the test set consisted of ${10}^4$ samples. Several notable patterns emerge. For instance, the network most frequently confuses labels 4 and 9, as well as labels 3 and 5. Additionally, label 1 is often mispredicted as label 3 or 8, Label 6 is frequently misclassified as label 0 and 5, while label 7 is often mispredicted as label 9 and 2.

6. Conclusions

This study introduces a supervised learning approach for spiking neural networks that leverages spike-timing-dependent plasticity within a supervised framework for image recognition tasks, thereby eliminating the reliance on traditional backpropagation. The MNIST dataset effectively demonstrates the strengths of this approach. Using only 10 hidden neurons, the model obtains an 89% accuracy rate with approximately 10% ambiguity. When the model includes a larger number of hidden neurons and utilizes a diverse set of hyperparameters among them, it attains around 40% learning accuracy with just 10 training stimuli, where each category is presented only once during training (one-shot learning). As the training sample size increases, the accuracy improves to 87%, maintaining negligible ambiguity. Additionally, a parallel training and testing method is proposed, leveraging small networks per computing process, which significantly reduces the training time and increases the testing accuracy with small training samples. Future refinements, such as increasing the number of neurons in the first hidden layer, introducing excitatory lateral connections, or confining the region in feedforward synaptic connections, may further improve the network performance.