A Comparison of Two Connectivity Reduction Methods for Spiking Neural Networks with Memristive Plasticity

1. Introduction

Neural network-based intelligent systems are widely employed in a wide range of tasks, from natural language processing to computer vision and signal processing. In edge computing, however, the use of deep learning methods still poses a variety of challenges, including latency and power consumption constraints, both during training and inference.

Neuromorphic computing devices, in which the information is encoded and processed in the form of binary events called spikes, offer a prospective solution to these problems. Modern neuroprocessors, e.g. TrueNorth [1], Loihi [2], or Altai 1, were shown to achieve power consumption on the order of milliwatts [3]. Thus, these devices offer a powerful inference interface, which can be used to deploy spiking neural networks (SNNs) for inference.

In turn, memristor-based training poses its own unique set of challenges and limitations. The most prominent one arises from the hardware implementation of synapses, where each memristor can have only a limited amount of synapses [4,5,6], imposing limitations on the number of weights that a given network may have. In this regard, sparsely connected spiking networks, where the connectivity can be reduced depending on the hardware specifications, are a plausible solution.

In this study, we compare two methods of reducing connectivity in memristive spiking neural networks: a bagging ensemble of spiking neural networks and a probabilistic sparse SNN. Using a three-layer SNN with inhibitory and excitatory synapses, we solve the handwritten and spoken digits classification tasks and compare the outcomes for the proposed connectivity reduction types and three plasticity models. The main contributions of this work are:

–

We design a probabilistic sparse connectivity approach to creating a two-layer spiking neural network, implement a bagging ensemble of two-layer SNNs, and compare these two methods;

–

We propose an efficiency index that facilitates the comparison among different methods of connectivity reduction, and apply it to the SNNs used in the study;

–

We demonstrate that both connectivity reduction methods achieve competitive results on handwritten and spoken digits classification tasks and can be used with the memristive plasticity models.

The rest of the study is structured as follows: In Section 2 we provide a brief overview of the existing connectivity reduction methods for SNNs. In Section 3 we describe the datasets we use, the plasticity models and the base spiking neural structure, and the sparsity methods we utilize for comparison. In Section 4 we provide the accuracy estimations for the proposed approaches and discuss the obtained results in Section 5. Finally, we draw the conclusions in Section 6.

2. Literature Review

Connectivity reduction concerning spiking and artificial neural networks has been studied in several existing works. For example, the study [7] proposes a probabilistic approach to connectivity reduction in SNNs based on the spatial structure of the network’s layers. It is shown that a spiking neural network pruned according to this method achieves 98% accuracy on the EMNIST [8] dataset. The authors of [9] propose a joined connectivity and weight learning approach inspired by synapse elimination and synaptogenesis in biological neurons. The gradient in this work is redefined as an additional synaptic parameter, facilitating better adaptation of the network topology. A multilayer convolutional SNN trained with error backpropagation and pruned according to the designed method demonstrates an accuracy loss of 3.5% on MNIST [10] and 0.73% on CIFAR-10 [11] datasets. In [12] a two-stage pruning method for on-chip SNNs is developed. The pruning is first performed during training based on the weight update history and spike timing, and then after training via weight thresholding. By training a deep SNN with time-to-first-spike coding using the proposed approach the authors decrease latency by a factor of 2 and reduce the network connectivity by 92% without accuracy loss. Another example can be found in [13], the authors use a method of zeroing weights above a given threshold and achieve a 70% reduction in connectivity. In this paper the network consists of a mixture of formal and spiking convolutional layers, and the resulting sparse hybrid network achieves more than 71% accuracy on the IVS 3cls [14] dataset. Finally, in [15] a sparse SNN topology is proposed, where the connectivity reduction is performed via a combination of pruning and quantization based on the power law weight-dependent plasticity model. After training, the three-layer fully connected SNN designed in the study achieves a classification accuracy of 92% on the MNIST dataset. Another approach to connectivity reduction present in the literature is based on designing locally-connected SNNs, the weights in which are created in a sparse fashion according to a certain rule. In [16], for example, a routing scheme using a hybrid of short-range direct connectivity and address event representation network is developed. Without providing any benchmark results, the authors focus on the details of mapping a given SNN to the proposed architecture and show that it yields up to a 90% reduction in connectivity. In [17], sparsity in a formal multilayer convolutional network is achieved by limiting the number of connections associated with each neuron by a factor of 2. The proposed approach is shown to achieve high accuracy on such classical datasets as DVS-Gesture [18] (98%) MNIST (99%), CIFAR-10 (94%), and N-MNIST [19] (99%). The authors of [20] propose locally connected SNN layers, in which subsets of neurons compete via inhibitory interactions to learn features from different parts of the input space. A three-layer locally connected spiking neural network is used in the study to solve the MNIST and EMNIST classification challenges, on which it achieves 95% and 70% accuracy scores, respectively. Thus, currently employed methods of reducing the connectivity in spiking neural networks are mostly encompassed by pruning, quantization, and local connectivity. However, ensemble learning, where multiple smaller networks are used together to form a stronger classifier can be also viewed as a single sparse network. In this work, we explore this path to connectivity reduction and compare it to a probabilistic locally connected SNN topology proposed in the work [21] and investigated in our previous research with different types of plasticity models[22,23,24].

3. Materials and Methods

3.1. Datasets and Preprocessing

For training and evaluation of the proposed methods, we use two benchmark datasets: scikit-learn Digits [25] and the Free Spoken Digits Dataset [26]. The former consists of 1797 $8\times 8$ images of handwritten digits, while the latter contains 3000 audio recordings of the spoken numbers from 0 to 9 in English.

Thus, both datasets have 10 classes, resulting in 180 samples per class for Digits and 300 samples per class for FSDD; additionally, the samples in FSDD vary by speaker: 6 speakers in total, 50 recordings of each digits per speaker with different intonnations.

The raw data was preprocessed as follows:

(1)

Feature engineering: for Digits, the original features in the form of pixel intensity were used without changes; for FSDD, features were extracted by splitting the signal into frames, extracting 30 Mel Frequency Cepstral Coefficients [27] (MFCC) and then averaging across frames.

(2)

Normalization: Depending on the type of plasticity, the input vectors were normalized either by reducing to zero mean and one standard deviation (Standard Scaling) or by L2 normalization.

(3)

Gaussian Receptive Fields (GRF): This step is necessary to significantly increase the selectivity of the network, as a consequence, increase the number of weights between the input and output layers of the spike network. At this stage, the normalized feature vectors were divided into M equal intervals for each feature. At each interval $j=1,...,M$, a Gaussian peak was constructed with center ${\mu }_j$ and standard deviation $\sigma$ (see Eq. 1, Figure 1). The value of each component $x_i$ of the input vector was replaced by a set of values $G_j\left(x\right)$ characterizing the proximity of $x_i$ to the center of the j-th receptive field. Thus, the dimension of the input vector increased M times.

(4)

Spike encoding: To convert the normalized and GRF-processed input vectors into spike sequences, we used a frequency-based approach. With this encoding method, each input neuron (spike generator) emits spikes at frequency $\nu$ during the entire sample time $t_e$, where $\nu ={\nu }_{max}·k$. Here ${\nu }_{max}$ is the maximum frequency of spike emission, and k is the value of the input vector component. After time $t_e$ has passed, the generators do not emit spikes for $t_p=50$ ms to allow the neuron potentials to return to their original values.

$$G_j\left(x_i\right)=exp\left(\frac{{(x_i-{\mu }_j)}^2}{{\sigma }^2}\right)$$

(1)

Figure 1. An example of Gaussian receptive fields with number of fields equal to 5. The input feature $x_i$ is intersected with overlapping Gaussians to produce a vectorized feature representation $G_j\left(x_i\right),j\in [0,5)\cap \mathbb{N}$.

Figure 1. An example of Gaussian receptive fields with number of fields equal to 5. The input feature $x_i$ is intersected with overlapping Gaussians to produce a vectorized feature representation $G_j\left(x_i\right),j\in [0,5)\cap \mathbb{N}$.

3.2. Synaptic Plasticity Models

In this work, we consider two memristive plasticity models: nanocomposite (NC) [28] and poly-p-xylylene (PPX) [29]. These models were proposed to approximate the real-world dependence of synaptic conductance change $\Delta w$ on the value of the conductance w and on the time difference $\Delta t$ between presynaptic and postsynaptic spikes and are defined in Eq. 2 and Eq. 3.

$$\Delta w(\Delta t)=\left\{\begin{array}{c}{A}^{+}·w·\left[1+tanh\left(-\frac{\Delta t-{\mu }_{+}}{{\tau }_{+}}\right)\right]\mathrm{if}\Delta t>0;\hfill \\ A^{-}·w·\left[1+tanh\left(\frac{\Delta t-{\mu }_{-}}{{\tau }_{-}}\right)\right]\mathrm{if}\Delta t<0;\hfill \end{array}\right.$$

(2)

In 2, $A^{+}=0.074$, $A^{-}=-0.047$, ${\mu }^{+}=26.7$ ms, ${\mu }^{-}=-22.3$ ms, ${\tau }^{+}=9.3$ ms, ${\tau }^{-}=10.8$ ms.

$$\Delta w(\Delta t)=\left\{\begin{array}{c}\frac{|\Delta t|}{\tau }{\alpha }^{+}{e}^{-{\beta }^{+}\left(\frac{w_{max}-w}{w_{max}-w_{min}}\right)}{e}^{-{\gamma }^{+}{\left(\frac{\Delta t}{\tau }\right)}^2}\mathrm{if}\Delta t>0;\\ \frac{|\Delta t|}{\tau }{\alpha }^{-}{e}^{-{\beta }^{-}\left(\frac{w-w_{min}}{w_{max}-w_{min}}\right)}{e}^{-{\gamma }^{-}{\left(\frac{\Delta t}{\tau }\right)}^2}\mathrm{if}\Delta t<0.\end{array}\right.$$

(3)

Here $\tau =10$ ms, ${\alpha }^{+}=0.316$, ${\alpha }^{-}=0.011$, ${\beta }^{+}=2.213$, ${\beta }^{-}=-5.969$, ${\gamma }^{+}=0.032$, ${\gamma }^{-}=0.146$, $w_{max}=1$, $w_{min}=0$.

Additionally, we consider a classical additive Spike Timing-Dependent Plasticity (STDP) [30] model to study the impact of sparcity on the memristor-based network relative to the simpler synapse models.

3.3. Spiking Classification Models

Within the framework of the frequency approach to encoding input data, we considered a hybrid architecture consisting of a three-layer Winner-Takes-All (WTA) network [21] serving as a feature extraction module in combination with a formal classifier.

The WTA network is based on three layers (see Figure 2). The input layer consists of spike generators that convert input vectors into spike sequences according to the algorithm described above. The size of the input layer corresponds to the size of the input vector after preprocessing steps. The generated spike sequences are transmitted to the layer of Leaky Integrate-and-Fire (LIF) neurons with an adaptive threshold (excitatory layer). This layer is connected to the input via trainable weights with one of the previously described plasticities according to the “all-to-all” rule. The number of neurons in the excitatory layer can be optimized depending on the complexity of the problem being solved. In turn, the excitatory layer is connected to the third layer of non-adaptive LIF neurons of the same size, which is called the inhibitory layer. Connections from the excitatory to the inhibitory layer are not trainable and have a fixed positive weight $w_{syn,exc}>0$. In this case, each neuron in the excitatory layer is connected to a single neuron (partner) in the inhibitory layer. The connections directed from the inhibitory layer to the excitatory layer are called inhibitory connections. These connections are static and have weight $w_{syn,inh}<0$. Each neuron in the inhibitory layer is connected to all neurons in the excitatory layer except its partner. Finally, generators in the input layer are also connected to inhibitory neurons by randomly distributed static links with weight $w_{syn,gen}>0$. In all our experiments, the number of such connections is equal to 10% of the number of connections between the input and excitatory layers.

The spiking neural network was implemented using the NEST simulator[31].

We chose logistic regression (LGR) optimized for multi-class problems using the one-versus-all (OVR) scheme as the formal classifier [32].

In this work, we considered two methods for reducing connectivity in the WTA network: an ensemble of several classifiers trained using the bagging technique and sparse connectivity between layers.

3.3.1. Classification Ensemble

The Bagging method was chosen as an ensemble creation technique, in which several identical classifiers are trained on subsets of input data, after which their predictions are aggregated by voting. This method has several advantages compared to using a single larger network, in particular, it reduces the total number of connections within the network and increases the classification speed due to parallelization. In addition, it allows you to break unwanted correlations in the training data set, resulting in improved architecture stability.

Connectivity within an ensemble is controlled using the following parameters:

–

n_estimators: defines the number of models within the ensemble.

–

max_features: determines the proportion of input features that are passed to the input of each of the models in the ensemble.

In addition, the bagging architecture allows you to regulate the number of examples on which each network is trained using the max_samples parameter.

The ensemble was implemented using the BaggingClassifier method of the Scikit-Learn [25] library. In all experiments, based on the preliminary empirical observations the parameters $max\_features=1.0$ and $max\_samples=0.7$ were fixed.

3.3.2. Sparse Connectivity

Another way to reduce the connectivity of a spike network is to set a rule that allows you to regulate the number of connections and their organization. To this end, we formally place the neurons of the excitatory and inhibitory layers on two-dimensional square grids of size 1 mm × 1 mm (the dimension was chosen for the convenience of further presentation), oriented mirror-image relative to each other. Neurons on the grids may be arranged irregularly. Initialization of sparse connections occurs according to the following algorithm:

(1)

The presynaptic neuron projects onto the plane of the postsynaptic neurons.

(2)

The projection of the presynaptic neuron becomes the center of a circular neighborhood, all postsynaptic neurons within which will be connected to this presynaptic neuron with some probability.

This process is shown schematically in Figure 3.

Thus, connectivity within the network is regulated using two parameters:

–

Probability P of connection formation between pre- and postsynaptic neurons.

–

The radius of the circular neighborhood is R. This parameter is defined only for connections between the inhibitory and excitatory layers since neurons in the input layer do not have a spatial structure.

In this work, we regulate the connectivity between the input and excitatory layers using the parameter $P_{gen\_exc}=0.4$, as well as between the inhibitory and excitatory layers using the parameters ${ }_{Pinh\_exc}=0.4$ and $R_{inh\_exc}=0.9$. The connections between the excitatory and inhibitory, as well as the input and inhibitory layers remain unchanged.

4. Experiments and Results

Experiments on the Digits dataset were conducted using hold-out cross-validation; 20% of training examples were used for testing. On FSDD, a fixed testing data set was used. For all experiments, such parameters as the number of neurons in the networks, the number of receptive fields, and the number of networks in the ensemble were optimized for each plasticity and for each data set by maximizing the training classification accuracy.

In all experiments, the time for submitting one example to the WTA network was 350 ms, followed by a relaxation period of 50 ms; learning took place over one epoch.

As a baseline, we conducted an experiment using the classical WTA network. The parameters for the base WTA network are presented in Table A1.

Additionally, for this and subsequent experiments, we present the number of connections within the network with a breakdown by type of pre- and postsynaptic neurons (see Table 1).

After applying the selected methods for reducing the number of connections in the WTA network, the parameters of the resulting models, which demonstrate the best accuracy, are presented in the Table 2.

The results of experiments of applying different types of reduction methods on the F1-score metric are presented in Table 3.

5. Discussion

Since the number of neurons and connections differs for each plasticity and for each dataset, to evaluate the effectiveness of the connectivity reduction methods, we introduce the connectivity index $\kappa$ defined in Eq. 4:

$$\kappa =\frac{N_{sparse}}{N_{base}}$$

(4)

Here $N_{sparse}$ and $N_{base}$ are the total number of connections in the sparse network and the equivalent fully connected WTA network, respectively. Based on this definition, the efficiency of the connectivity reduction method can be assessed by calculating the ratio of the classification accuracy to the connectivity index (see Eq. 5, where the efficiency is represented by the index $\eta$).

$$\eta =\frac{\text{F1\_score}}{\kappa }$$

(5)

The values of connectivity and efficiency indices for different data sets, plasticities and network types are presented in Table 4.

From the table above it follows that in our experiments, the relative efficiency of ensembles of spike networks in comparison with spike networks with sparse connectivity is higher: on average across plasticities and datasets, the efficiency of bagging is 3.0, while the efficiency of the sparse WTA network is on average equal to 1.58. In addition, the average connectivity index $\kappa$ is lower for the Bagging method - 0.34 versus 0.6 for the sparse WTA network.

Despite that, both methods can be effectively used to reduce connectivity depending on the specifics of the problem and the hardware requirements. In the context of on-chip memristive spiking neural networks, the bagging ensemble technique is preferable if there is a strong limitation on the overall network connectivity, especially with regard to plastic synapses. On the other hand, the necessity of orchestrating multiple networks may become a concern. The probabilistic sparse WTA network is simpler in design and can be applied in situations where the connectivity limitations mainly affect inhibitory static synapses.

6. Conclusions

In this work, we have compared two approaches to connectivity reduction in memristive spiking neural networks: the bagging ensemble technique and probabilistic sparse connectivity. Using a three-layer WTA network, we have demonstrated that both methods achieve competitive performance on the handwritten digits and spoken digits classification tasks. On the Digits dataset, the bagging ensemble yields the F1-score of 0.88, 0.92, and 0.91 for the STDP, NC, and PPX plasticity rules, respectively, while the sparse WTA network achieves 0.90, 0.89 and 0.89.

On FSDD the F1-score values lie within the 0.94 – 0.96 range for the ensemble of WTA networks, and within 0.80 – 0.83 interval for the sparse WTA network.

Additionally, by studying the ratio between the proposed connectivity index and the F1-score we show that the bagging ensemble achieves higher efficiency relative to the overall number of connections, while the sparse WTA network effectively reduces the number of inhibitory connections while being significantly simpler in design.

In our future research, we plan to expand the scope of the classification problems that can be solved using the proposed methods, as well as work on hardware implementations of the designed networks.