Studies on integrating Spiking Neural Networks (SNNs) with the Transformer architecture holds promise for enabling models to achieve ultra-low energy consumption while possessing the performance of the Transformer architecture. Currently, studies on ANN-to-SNN conversion of integrating Spiking Neural Networks (SNNs) with the Transformer architecture mainly focuses on simple activation functions in MLPs, and has not yet addressed the mismatch between the Softmax activation function in the self-attention mechanism and the computation rules of SNNs. Consequently, the ANN-to-SNN conversion efforts have consistently failed to make the Transformer architecture directly applicable to SNNs. To address this challenge, we propose the Spiking-Softmax method, which integrates Spiking Exponential Neuron (SI-exp) and Spiking Collaboration Normalized Neuron (SI-norm). The Spiking-Softmax method accurately simulates the Softmax activation function with only 12 time steps. Building upon this, we propose the Spike Integrated Transformer conversion (SIT-conversion) method, which enables the conversion of the Transformer architecture to SNNs. The SNNs generated by the SIT-conversion of Transformer models of various sizes achieve accuracy nearly identical to their ANNs counterparts, achieving nearly lossless and ultra-low-latency ANN-to-SNN conversion. This work represents the first implementation of simulating the Softmax activation function and fully converting the Transformer architecture into SNNs through spike firing.

Standard ANNs lack flexibility when handling corrupted input due to their fixed structure. Spiking Neural Networks (SNNs) can utilize biological temporal coding features, such as noise-induced stochastic resonance and dynamical synapses to increase a model’s performance when its parameters are not optimized for a given input. Using the analog XOR task as a simplified Convolutional Neural Network analog, this paper demonstrates two key results: (1) SNNs solve the problem linearly inseparable in ANN with over 10% improvement in the optimal setting, and (2) in SNNs, the synaptic noise and dynamical synapses compensate for non-optimal parameters, achieving near-optimal results.

The event camera generates an event stream based on changes in brightness, retaining only the characteristics of moving objects, addresses the high power consumption associated with using high-frame-rate cameras for high-speed eye tracking tasks. However, the asynchronous incremental nature of event camera output has not been fully utilized, and there are also issues related to missing event datasets. Combining the temporal information encoding and state-preserving properties of spiking neural network (SNN) with event camera, a near-range eye tracking algorithm is proposed as well as a novel event-based dataset for validation and evaluation. According to experimental results, the proposed solution outperforms artificial neural network (ANN) algorithms, while computational time remains only 12.5\% of that of traditional SNN algorithms. Furthermore, the proposed algorithm allows for self-adjustment of time resolution, with a maximum achievable resolution of 0.081ms, enhancing tracking stability while maintaining accuracy.

Spiking neural networks (SNNs) promise to perform tasks currently done by classical artificial neural networks (ANNs) faster, in smaller footprints and using less energy. Neuromorphic processors are set out to revolutionize computing at a large scale, but the move to edge-computing applications calls for finely-tuned custom implementations to keep pushing towards more efficient systems. To that end, we have examined the architectural design space for executing spiking neuron models on FPGA platforms, focusing on achieving ultra-low area and power consumption. This work presents an efficient clock-driven spiking neuron architecture used for the implementation of both fully-connected cores and 2D convolutional cores, which rely on deep pipelines for synaptic processing and distributed memory for weight and neuron states. With them, we have developed an accelerator for an SNN version of the LeNet-5 network trained on the MNIST dataset. At around 5.5 slices/neuron and only 348 mW, it is able to use 33% less area and 4 times as less power per neuron as current state-of-the-art implementations while keeping low simulation step times.

In the pursuit of energy-efficient spiking neural network (SNN) hardware, synaptic devices lev-eraging emerging memory technologies hold significant promise. This study investigates the ap-plication of the recently proposed HfO2/SiO2-based interface dipole modulation (IDM) memory for synaptic spike timing-dependent plasticity (STDP) learning. Firstly, through pulse measure-ments of IDM metal-oxide-semiconductor (MOS) capacitors, we demonstrate that IDM exhibits an inherently nonlinear and near-symmetric response. Secondly, we discuss the drain current re-sponse of a field-effect transistor (FET) incorporating a multi-stack IDM structure, revealing its nonlinear and asymmetric pulse response, and suggest that the degree of the asymmetry depends on the modulation current ratio. Thirdly, to emulate synaptic STDP behavior, we implement dou-ble-pulse-controlled drain current modulation of IDMFET using a simple bipolar rectangular pulse. Additionally, we propose a double-pulse-controlled synaptic depression that is valuable for optimizing STDP-based unsupervised learning. Integrating the pulse response characteristics of IDMFETs into a two-layer SNN system for synaptic weight updates, we assess training and classification performance on handwritten digits. Our results demonstrate that ID-MFET-based synaptic devices can achieve classification accuracy comparable to previously re-ported simulation-based results.

Recent neuroscience research results show that the nerve information in the brain is not only encoded by the spatial information. Spiking neural network based on pulse frequency coding plays a very important role in dealing with the problem of brain signal, especially complicated space-time information. In this paper, an unsupervised learning algorithm for bilayer feedforward spiking neural networks based on spike-timing dependent plasticity (STDP) competitiveness is proposed and applied to SAR image classification on MSTAR for the first time. The SNN learns autonomously from the input value without any labeled signal and the overall classification accuracy of SAR targets reached 80.8%. The experimental results show that the algorithm adopts the synaptic neurons and network structure with stronger biological rationality, and has the ability to classify targets on SAR image. Meanwhile, the feature map extraction ability of neurons is visualized by the generative property of SNN, which is a beneficial attempt to apply the brain-like neural network into SAR image interpretation.

In this article, we propose a circuit to imitate the behavior of a Reward-Modulated Spike-Timing-Dependent Plasticity synapse. When two neurons in adjacent layers produce spikes, each spike modifies the thickness of the common synapse. As a result, the synapse’s ability to conduct impulses is controlled, leading to an unsupervised learning rule. By introducing a reward signal, reinforcement learning is enabled by redirecting the growth and shrinkage of synapses based on signal feedback from the environment. The proposed synapse manages the convolution of the emitted spike signals to promote either the strengthening or weakening of the synapse, which is represented as the resistance value of a memristor device. As memristors have a conductance range that may differ from the available current input range of typical CMOS neuron designs, the synapse circuit can be adjusted to regulate the spike’s amplitude current to comply with the neuron. The circuit described in this work allows for the implementation of fully interconnected layers of neuron analog circuits. This is achieved by having each synapse reconform the spike signal, thus removing the burden of providing enough power from the neurons to each memristor. The synapse circuit was tested using a CMOS analog neuron described in the literature. Additionally, the article provides insight into how to properly describe the hysteresis behavior of the memristor in Verilog-A code. The testing and learning capabilities of the synapse circuit are demonstrated in simulation using the Skywater-130nm process. The article’s main goal is to provide the basic building blocks for Deep Neural Neural Networks relying on spiking neurons and memristors as the basic processing elements to handle spike generation, propagation, and synaptic plasticity.

This research investigates the implementation of complex exponential-based neurons in FPGA, which can pave the way for implementing bio-inspired spiking neural networks to compensate for the existing computational constraints in conventional artificial neural networks. The increasing use of extensive neural networks and the complexity of models in handling big data lead to higher power consumption and delays. Hence, finding solutions to reduce computational complexity is crucial for addressing power consumption challenges. The complex exponential form effectively encodes oscillating features like frequency, amplitude, and phase shift, streamlining the demanding calculations typical of conventional artificial neurons through levering simple phase addition of complex exponential functions. The article implements such a two-neuron and a multi-neuron neural model using Xilinx system generator and Vivado design suite, employing 8-bit, 16-bit, and 32-bit fixed-point data format representations. The study evaluates the accuracy of the proposed neuron model across different FPGA implementations while also providing a detailed analysis of operating frequency, power consumption, and resource usage for the hardware implementations. BRAM-based Vivado designs outperformed Simulink regarding speed, power, and resource efficiency. Specifically, the Vivado BRAM-based approach supported up to 128 neurons, showcasing optimal LUT and FF resource utilization. Such outcomes accommodate choosing the optimal design procedure for implementing spiking neural networks on FPGAs.

Recent advancement of deep learning has been elevated the multifaceted nature in various applications of this field. Artificial neural networks are now turning into a genuinely old procedure in the vast area of computer science; the principal thoughts and models are more than fifty years of age. However, in this modern computing era, 3rd generation intelligent models are introduced by scientists. In the biological neuron, actual film channels control the progression of particles over the layer by opening and shutting in light of voltage changes because of inborn current flows and remotely led to signals. A comprehensive 3rd generation, Spiking Neural Network (SNN) is diminishing the distance between deep learning, machine learning, and neuroscience in a biologically-inspired manner. It also connects neuroscience and machine learning to establish high-level efficient computing. Spiking Neural Networks initiate utilizing spikes, which are discrete functions that happen at focuses as expected, as opposed to constant values. This paper is a review of the biological-inspired spiking neural network and its applications in different areas. The author aims to present a brief introduction to SNN, which incorporates the mathematical structure, applications, and implementation of SNN. This paper also represents an overview of machine learning, deep learning, and reinforcement learning. This review paper can help advanced artificial intelligence researchers to get a compact brief intuition of spiking neural networks.

The Spiking Neural Network (SNN) has attracted more and more attention recently. It adopts binary spike signals to transmit information. Benefitting from the information passing paradigm of SNNs, the multiplications of activations and weights can be replaced by additions, which are more energy-efficient. However, its ``Hard Reset" mechanism for the firing activity would ignore the difference among membrane potentials when the membrane potential is above the firing threshold, causing information loss. Meanwhile, quantifying the membrane potential to 0/1 spikes at the firing instants will inevitably introduce the quantization error thus bringing about information loss too. To address these problems, we propose to use the ``Soft Reset" mechanism for the supervised training-based SNNs, which will drive the membrane potential to a dynamic reset potential according to its magnitude, and Membrane Potential Rectifier (MPR) to reduce the quantization error via redistributing the membrane potential to a range close to the spikes. Results show that the SNNs with the ``Soft Reset" mechanism and MPR outperform their vanilla counterparts on both static and dynamic datasets.

In this paper, we demonstrate that fixed-weight layers, generated from random distribution or logistic functions, can effectively extract significant features from input data, resulting in high accuracy on a variety of tasks, including Fisher’s iris, Wisconsin Breast Cancer, and MNIST datasets. We have observed that logistic functions yield high accuracy with less dispersion in results. We have also assessed the precision of our approach under conditions of minimizing spikes number generated in the network, it is a practically useful for reducing energy consumption in spiking neural networks. Our findings reveal that the proposed method demonstrates a highest accuracy on Fisher’s iris and MNIST datasets with logistic regression. Furthermore, they surpass the accuracy of the conventional (non-spiking) approach using logistic regression in the case of Wisconsin Breast Cancer. We have also investigated the impact of non-stochastic spike generation on accuracy.

Working memory (WM) is a brain system for short-term storage and manipulation of information and plays an important role in complex cognitive tasks. In the synaptic theory of WM memorized elements are stored in the form of short-term potentiated connections in a sample population of neurons. In this paper, we show that such populations can be formed due to the mechanisms of spike-timing-dependent plasticity (STDP) – the phase dependence associated with the ratio of the pulse times of the interacting neurons. We propose a WM model considering two types of plasticity: short-term plasticity and STDP. We have shown formation of neuronal clusters encoding items in the WM model, that can be formed by external stimulation of a group of neurons due to the mechanisms of STDP and hold and reactivated by short-term plasticity mechanisms. The dynamic formation of neuronal clusters instead of pre-formed clusters gives additional flexibility to the model.

Accurate and robust deep learning models for facial expression recognition are challenging to achieve, given the diversity of human faces and variations in images, including different facial poses and lighting conditions. In this work, we proposed a clock-driven convolutional Spiking Neural Network (SNN) and STDP fine-tune architecture, meticulously calibrated its hyperparameters, and experimented with various optimization methods. The best model resulted was trained and evaluated on the Fer2013 and FER+ database, obtaining an accuracy of 61.87% and 79.97% without requiring auxiliary training data or face registration. To our best knowledge, the proposed SNN achieved comparable accuracy to CNNs of similar depth and possessed the advantages of low energy consumption and high computational efficiency. The computational efficiency of the proposed SNN is approximately three times that of CNNs. Along with this, we introduced the very recent cumulative spike guided encoder visualization technique and revealed the strong encoding capability of the proposed SNN.

This work introduces a neuromorphic sensor based on force-sensing resistors (FSR) and spiking neurons for robotic systems. The proposed sensor integrates the FSR in the schematic of spiking neuron in order to make the sensor to generate spikes with frequency that depends on the applied force. The performance of the proposed sensor is evaluated for the control of a SMA-actuated ro-botic finger by monitoring the force during steady state when the finger pushes on a tweezers. For comparison purposes, we performed similar evaluation when the SNN receives input from a widely used compression load cell (CLC). The results show that the proposed FSR based neuro-morphic sensor has very good sensitivity to low forces and the function between the spiking rate and the applied force is continuous with good variation range. However, when compared to the CLC, the proposed sensor has lower linearity of the response, but it benefits from two orders of magnitude lower power consumption. These aspects imply that the neuromorphic sensor based on FSR are useful for bioinspired control of humanoid robotics representing a low power and low cost alternative the widely used sensors.

Accurate and robust deep learning models for facial expression recognition are challenging to achieve, given the diversity of human faces and variations in images, including different facial poses and lighting conditions. In this work, we proposed a clock-driven convolutional Spiking Neural Network (SNN) and STDP fine-tune architecture, meticulously calibrated its hyperpa-rameters, and experimented with various optimization methods. The best model resulted was trained and evaluated on the Fer2013 and FER+ database, obtaining an accuracy of 61.87% and 79.97% without requiring auxiliary training data or face registration. To our best knowledge, the proposed SNN achieved comparable accuracy to CNNs of similar depth and possessed advantages of low energy consumption and high computational efficiency. The computational efficiency of the proposed SNN is approximately three times that of CNNs. Along with this, we introduced the very recent cumulative spike guided encoder visualization technique and revealed the strong encoding capability of the proposed SNN.

Human activity recognition (HAR) using wearable sensors is gaining popularity in the healthcare domain, enabling continuous monitoring and timely interventions. The conventional HAR systems are centralised and need data sharing with a server for processing and model training. However, this data sharing leads to various challenges, such as privacy concerns, increased communication and storage costs, inefficient computational algorithms, data scarcity due to unwillingness to share, and difficulty achieving high accuracy and personalised models simultaneously. To overcome these challenges, this paper introduces a privacy and energy-aware federated learning (FL) framework for model training without data sharing. The proposed scheme incorporates spiking neural networks (SNNs), which are particularly suited for resource-constrained devices due to their event-driven information processing. The key idea is to synergise the strengths of SNNs and long short-term memory (LSTM) to create a hybrid model for robust HAR. The proposed hybrid spiking LSTM (S-LSTM) model uses the LSTM as an input layer to capture temporal dependencies efficiently, followed by spiking layers that replace the typical activation using leaky integrate-and-fire neurons. We employ the combination of surrogate gradient learning and backpropagation through time (BPTT) to update the weights of SNN layers in a supervised manner. The performance of the proposed model is evaluated using two publicly available datasets for both indoor and outdoor scenarios. The results show that the proposed S-LSTM performs much better than spike convolutional neural networks (S-CNN) and simple CNN, achieving the accuracy of 97% and 87% for indoor and outdoor datasets, respectively. Personalisation, also known as fine-tuning the global model using local data, is crucial in HAR. It allows the model to adapt to individual behaviour and preferences, making predictions more accurate. The results also show that personalisation improves the performance of participants up to 9% on average. Additionally, analysis with 50% random client participation reduces communication costs by 50% with minimal impact on accuracy.

Why do neurons communicate through spikes? By definition, spikes are all-or-none neural events which occur at continuous times. In other words, spikes are on one side binary, existing or not without further details, and on the other can occur at any asynchronous time, without the need for a centralized clock. This stands in stark contrast to the analog representation of values and the discretized timing classically used in digital processing and at the base of modern-day neural networks. As neural systems almost systematically use this so-called event-based representation in the living world, a better understanding of this phenomenon remains a fundamental challenge in neurobiology in order to better interpret the profusion of recorded data. With the growing need for intelligent embedded systems, it also emerges as a new computing paradigm to enable the efficient operation of a new class of sensors and event-based computers, called neuromorphic, which could enable significant gains in computation time and energy consumption — a major societal issue in the era of the digital economy and global warming. In this review paper, we provide evidence from biology, theory and engineering that the precise timing of spikes plays a crucial role in our understanding of the efficiency of neural networks.

Memristive spiking neural networks are a promising emerging technology in the field of deep learning. These models can be implemented using neuromorphic hardware and therefore offer low power consumption and low latency, crucial to edge-computing applications. These systems, however, pose several challenges, including on-chip training and connectivity reduction. The latter is essential to ease the manufacturing complexity of the system. On-chip training can be realized using synaptic plasticity enabled by memristors. In this work, we compare two methods of connectivity reduction applicable to memristive spiking networks: an ensemble-based approach and a probabilistic sparse connectivity approach. We evaluate both of these methods in conjunction with a three-layer spiking neural network on the handwritten and spoken digits classification tasks using two memristive plasticity models and a classical time-dependent plasticity rule. On the handwritten digits recognition task, both methods achieve the F1-score of 0.89–0.93, and yield the 0.80–0.96 F1-score on the spoken digits recognition task.

Situational awareness detection and characterization of mental states have a vital role in medicine and many other fields. An electroencephalogram (EEG) is one of the most effective tools for identifying and analyzing cognitive stress. Yet, the measurement, interpretation, and classification of EEG sensors is a challenging task. This study introduces a novel machine learning-based approach to assist in evaluating situational awareness detection using EEG signals and spiking neural networks (SNNs) based on a unique spike-continuous-time-neuron (SCTN). The implemented biologically inspired SNN architecture is used for effective EEG feature extraction by applying time-frequency analysis techniques and allows adept detection and analysis of the various frequency components embedded in the different EEG sub-bands. The EEG signal undergoes encoding into spikes and is then fed into an SNN model which is well-suited to the serial sequence order of the EEG data. We utilize the SCTN-based resonator for EEG feature extraction in the frequency domain which demonstrates a high correlation with the classical FFT features. A new SCTN-based 2-D neural network is introduced for efficient EEG feature mapping aiming to achieve a spatial representation of each EEG sub-band. To validate and evaluate the performance of the proposed approach a common publicly available EEG dataset is used. Experimental results show that by using the extracted EEG frequencies features and the SCTN-based SNN classifier the mental state can be accurately classified with an average accuracy of 96.8% for the common EEG dataset. Our proposed method outperforms existing machine learning-based methods and demonstrates the advantages of using SNN for situational awareness detection and mental state classifications.

Humans learn from a lot of information sources to make decisions. Once this information is learned in the brain, spatio-temporal associations are made, connecting all these sources (variables) in space and time represented as brain connectivity. In reality, to make a decision, we usually have only part of the infor-mation, either as a limited number of variables, or limited time to make the decision, or both. The brain functions as spatio-temporal associative memory. Inspired by the ability of the human brain, a brain-inspired spatio-temporal associative memory was proposed earlier, that utilizes the NeuCube brain-inspired spiking neural network framework. Here we apply the STAM framework to develop STAM for neuroimaging data, on the cases of EEG and fMRI, resulting in STAM-EEG and STAM-fMRI. The paper shows that once a NeuCube STAM classification model is trained on a complete spatio-temporal EEG or fMRI data, it can be recalled using only part of the time series, or/and only part of the used variables. We evaluate accordingly the temporal association accuracy and spatial association accuracy. This is a pilot study that opens the field for the development of multimodal classification systems on other multimodal neuroimaging data, such as the also shown longitudinal MRI data, trained on complete data, but recalled on partial data collected across different settings, in different labs and clinics, that may vary in terms of variables, time of data collection, and other parameters. The proposed methods will allow also for brain diagnostic/prognostic marker discovery using spatio-temporal neuroimaging data.

Neuromorphic computing, a brain inspired non-Von Neumann computing system, addresses the challenges posed by the Moore’s law memory wall phenomenon. It has the capability to increasingly enhance performance while maintaining power efficiency. Neuromorphic chip architecture requirements vary depending on the application and optimizing it for large-scale applications remains to be a challenge. Neuromorphic chips are programmed using spiking neural networks which provide them with important properties such as parallelism, asynchronism, and on-device learning. Widely used spiking neuron models include the Hodgkin-Huxley Model, Izhikevich model, integrate-and-fire model and spike response model. Hardware implementation platforms of the chip follow three approaches: analog, digital, or a combination of both. Each platform can be implemented using various memory topologies which interconnects with the learning mechanism. Current neuromorphic computing systems typically use the unsupervised learning spike timing-dependent plasticity algorithms. However, algorithms such as voltage-dependent synaptic plasticity have the potential to enhance performance. This review summarizes the potential neuromorphic chip architecture specifications and highlights which applications they are suitable for.

Tefluthrin (Tef) is categorized as a type-I pyrethroid insecticide, telmisartan (Tel) functions as an angiotensin II receptor blocker, and KB-R7943 was identified as an inhibitor of Na+-Ca2+ exchange process. However, the influence of these compounds on the amplitude and gating properties of voltage-gated Na+ current (INa) in neurons associated with pain signaling remains unclear. In cultured dorsal root ganglion (DRG) neurons, whole-cell current recordings revealed that Tef or Tel increased the peak amplitude of INa, concomitant with an elevation in the time constant of INa inactivation, particularly in the slow component. Conversely, exposure to KB-R7943 resulted in a depression in INa, coupled with a decrease in the slow component of the inactivation time constant of INa. Theoretical simulations and bifurcation analyses were performed on a modeled interneuron in the spinal dorsal horn. The occurrence of INa inactivation accentuated the subthreshold oscillations (SO) in the membrane potential. With an increase in applied current, SO became more pronounced, accompanied by the emergence of high-frequency spiking (HF) with a frequency of approximately 150 Hz. Moreover, an elevation in INa conductance further intensified both SO and HF. Consequently, through experimental and in silico studies, this work reflects that Tef, Tel, or KB-R7943 significantly impacts the magnitude and gating properties of INa in neurons associated with pain signaling. The alterations in INa magnitude and gating in these neurons suggest a close relationship with pain transmission.

Electroencephalography (EEG) signals are the primary source for discriminating the preictal from the interictal stage, enabling early warnings before the seizure onset. The epileptic seizure prediction models face significant challenges due to data scarcity, diversity, and privacy. This paper proposes a three-tier architecture for epileptic seizure prediction associated with the Federated Learning (FL) model that enhances the capability by utilizing the significant amount of seizure patterns from globally distributed patients with data privacy. The determination of the preictal state is influenced by global and local model-assisted decision-making by modeling the two-level edge layer. Integrating the Spiking Encoder (SE) with Graph Convolutional Neural Network (Spiking-GCNN) works as the local model trained using the bi-timescale approach. Each local model utilizes the aggregated seizure knowledge from the different medical centers through FL and determines the preictal probability in the coarse-grained personalization. The Adaptive Neuro-Fuzzy Inference System (ANFIS) is utilized in fine-grained personalization to recognize epileptic seizure patients by examining the outcomes of the FL model, heart rate variability features, and patient-specific clinical features. The proposed seizure prediction is evaluated using benchmark datasets by comparing them with the existing works to demonstrate the potential results.

In recent years, advancements in brain science and neuroscience have significantly influenced the field of computer science, particularly in the domain of reinforcement learning (RL). Drawing insights from neurobiology and neuropsychology, researchers have leveraged these findings to develop novel mechanisms for understanding intelligent decision-making processes in the brain. Concurrently, the emergence of quantum computing has opened new frontiers in artificial intelligence, leading to the development of quantum machine learning (QML). This study introduces a novel model that integrates Quantum Spiking Neural Networks (QSNN) and Quantum Long Short-Term Memory (QLSTM) architectures, inspired by the complex workings of the human brain. Specifically designed for reinforcement learning tasks in energy-efficient environments, our approach progresses through two distinct stages mirroring sensory and memory systems. In the initial stage, analogous to the brain’s hypothalamus, low-level information is extracted to emulate sensory data processing patterns. Subsequently, resembling the hippocampus, this information is processed at a higher level, capturing and memorizing correlated patterns. We conduct a comparative analysis of our model against existing quantum models such as Quantum Neural Networks and their classical counterparts, elucidating its unique contributions. Through empirical results, we aim to underscore the effectiveness of our approach in optimizing energy consumption in practical scenarios.

The rate of technical innovation, system interconnection, and advanced communications undoubtedly boost distributed energy networks' efficiency. However, when an additional attack surface is made available, the possibility of an increase in attacks is an unavoidable result. The energy ecosystem's significant variety draws attackers with various goals, making any critical infrastructure a threat, regardless of scale. Outdated technology and other antiquated countermeasures that worked years ago cannot address the complexity of current threats. As a result, robust artificial intelligence cyber-defense solutions are more important than ever. Based on the above challenge, this paper proposes an ensemble transfer learning spiking immune system for adaptive smart grid protection. It is an innovative Artificial Immune System (AIS) that uses a swarm of Evolving Izhikevich Neural Networks (EINN) in an Ensemble architecture, which optimally integrates Transfer Learning methodologies. The effectiveness of the proposed innovative system is demonstrated experimentally in multiple complex scenarios that optimally simulate the modern energy environment. In this way, the proposed system fully automates the strategic security planning of energy networks with computational intelligence methods. It allows the complete control of the digital strategies of the potential infrastructure that frames it, thus contributing to the timely and valid decision-making during cyber-attacks.

The ongoing revolution in Deep Learning is redefining the nature of computing that is driven by the increasing amount of pattern classification and cognitive tasks. Specialized digital hardware for deep learning still holds its predominance due to the flexibility offered by the software implementation and maturity of algorithms. However, it is being increasingly desired that cognitive computing occurs at the edge, i.e. on hand-held devices that are energy constrained, which is energy prohibitive when employing digital von Neumann architectures. Recent explorations in digital neuromorphic hardware have shown promise, but offer low neurosynaptic density needed for scaling to applications such as intelligent cognitive assistants (ICA). Large-scale integration of nanoscale emerging memory devices with Complementary Metal Oxide Semiconductor (CMOS) mixed-signal integrated circuits can herald a new generation of Neuromorphic computers that will transcend the von Neumann bottleneck for cognitive computing tasks. Such hybrid Neuromorphic System-on-a-chip (NeuSoC) architectures promise machine learning capability at chip-scale form factor, and several orders of magnitude improvement in energy efficiency. Practical demonstration of such architectures has been limited as performance of emerging memory devices falls short of the expected behavior from the idealized memristor-based analog synapses, or weights, and novel machine learning algorithms are needed to take advantage of the device behavior. In this work, we review the challenges involved and present a pathway to realize ultra-low-power mixed-signal NeuSoC, from device arrays and circuits to spike-based deep learning algorithms, with ‘brain-like’ energy-efficiency.

Creating self-learning algorithms, developing deep neural networks and improving other methods that "learn" for various areas of human activity is the main goal of the theory of machine learning. It helps to replace the human with a machine, aiming to increase the quality of production. The theory of artificial neural networks, which already have replaced the humans in problems of detection of moving objects, recognition of images or sounds, time series prediction, big data analysis and numerical methods remains the most dispersed branch of the theory of machine learning. Certainly, for each area of human activity it is necessary to select appropriate neural network architectures, methods of data processing and some novel tools from applied mathematics. But the universal problem for all these neural networks with specific data is the achieving the highest accuracy in short time. Such problem can be resolved by increasing sizes of architectures and improving data preprocessing, where the accuracy rises with the training time. But there is a possibility to increase the accuracy without time growing, applying optimization methods. In this survey we demonstrate existing optimization algorithms of all types, which can be used in neural networks. There are presented modifications of basic optimization algorithms, such as stochastic gradient descent, adaptive moment estimation, Newton and quasi-Newton optimization methods. But the most recent optimization algorithms are related to information geometry, for Fisher-Rao and Bregman metrics. This approach in optimization extended the theory of classic neural networks to quantum and complex-valued neural networks, due to geometric and probabilistic tools. There are provided applications of all introduced optimization algorithms, what delighted many kinds of neural networks, which can be improved by including any advanced approaches in minimization of the loss function. Afterwards, we demonstrated ways of developing optimization algorithms in further researches, engaging neural networks with progressive architectures. Classical gradient based optimizers can be replaced by fractional order, bilevel and, even, gradient free optimization methods. There is a possibility to add such analogues in graph, spiking, complex-valued, quantum and wavelet neural networks. Besides the usual problems of image recognition, time series prediction, object detection, there are many are other tasks for modern theory of machine learning, such as solving problem of quantum computations, partial differential and integro-differential equations, stochastic processes and Brownian motion, making decisions and computer algebra.