A Memristor Neural Network Based on Simple Logarithmic-Sigmoidal Transfer Function with MOS Transistors

1. Introduction

Artificial neural networks have inspiration from electro-chemical communication between neural cells in human brain and other biological neural systems [1,2]. Their applications have revealed power efficiency and promising utilization in very low-energy and mobile electronic schemes and devices [3,4]. Neuromorphic integrated chips have the ability to perform complex computations quickly and effectively, consuming minimal electric power [5,6,7]. In the past, Metal Oxide Semiconductor (MOS) transistor-based neural nets and optical neural networks were a leading approach for realizing analog neural nets, utilizing light, instead of electrical pulses to conduct neural computing, image and video processing, and other applicable tasks [8,9,10]. Some of the state-of-the-art improvements in this field are the neural nets based on memristors [11,12,13,14], which are generally used in the synaptic bonds for storing the synaptic weights [15,16,17].

Memristors predicted by Leon Chua [18] are electronic passive components with two electrodes, that have the ability to store electric charges, passed through their structure [19,20]. By applying electric pulses – voltage or current with definite amplitudes and durations, the resistance of memristors (also known as a memristance) could be altered. The memristors could work as tunable electronic resistors [21,22]. The first material memristor based on titanium dioxide (TiO₂), is created in Hewlett-Packard scientific and research labs by Stanley Williams and his collaborators [21,23,24]. Different metal oxides, such as HfO₂, Ta₂O₅, Nb₂O₅, and many others substances are utilized for realizations of memristor elements [25,26]. Some valuable properties of memristors are their low energy usage, very good switching and memory properties, high switching speed, nano-sizes and a good compatibility to the present-day Complementary MOS (CMOS) integrated chips and circuits [27,28]. Memristors are potentially applicable in memory matrices, reconfigurable analog and digital circuits, neural nets, and many others [29,30]. They are related to storage and computing in a single electronic circuit element [31].

Several different categories of memristor-based synapses, artificial neurons and neural networks are accessible in the related scientific works [32,33]. Some electronic synapses based on single memristor elements [7] and MOS transistors ensure only positive weights, which is a weakness of such realizations. More complicated schematics employ multiple memristors for each synapse [9], arranged in anti-parallel and bridge circuits. Such configurations enable realization of positive, zero, and negative synaptic weights [9,12]. An arrangement containing five memristors and electronic switches forms a π-type memristor synapse [13]. Certain synapses within this category utilize operational amplifiers (op-amps) and metal-oxide-semiconductor (MOS) transistors as differential amplifiers [14]. In contrast to the advantage of realization of positive and negative weights, the main drawback lies in the higher number of memristors per single synapse. For synapse realization, usually three or more memristors are used, and several op-amps are included [15]. A possibility to realize each synaptic weight with just a single memristor is available, which could significantly reduce neural circuit complexity [34,35].

For analysis and simulation of memristor-based neurons and neural networks with a large number of memristors, fast-operating and simple SPICE models of memristors and MOS transistors are desirable, for simplification the analyses and for decreasing the simulation time [36,37]. For this aim, a modified, enhanced and simplified SPICE model of metal-oxide memristors with activation thresholds is utilized [35]. Along with the software products of the Simulation Program with Integrated Circuits Emphasis (SPICE) family, the used here LTSPICE simulator is a preferred one, owing to its user-friendly interface, free license, an absence of artificial restrictions on the maximal number of the considered electronic elements and their connections, and very good convergence characteristics [33]. Owing to its benefits LTSPICE product is applied for the electronic simulations in the present work. For comparison and confirmation of the derived results, MATLAB, Simulink and Neural Network Toolbox [32] are also utilized.

After a comprehensive reference investigation, it was established that complete circuits of memristor-based synaptic schemes, neurons, transfer functions and neural nets, and related programming codes and results, together with comparisons are not represented in the available scientific papers. It is also known that some issues with realization of negative synaptic weights are reported [6]. This was the main motivation for proposing the implementation of a memristor-based neural network in LTSPICE, as the activation function was realized with a voltage-controlled voltage source in LTSPICE simulator [34]. Subsequently a single neuron with activation function implemented with memristors and MOS transistors is also realized and proposed to the readers [35].

The goal of this paper is realization of a whole electronic implementation of a memristor neural network in LTSPICE environment. To achieve this purpose the following tasks are placed: realization of a new electronic circuit based on memristor and MOS transistors, realizing a logarithmic-sigmoidal transfer function; realization of a simple adder based on memristors and operational amplifiers with minimal number of elements; synthesis and analysis of a simple multi-layer neural network, utilizing memristors for the synaptic weights of positive and negative signs, and to employ a minimal number of electronic components in LTSPICE environment [33]. An artificial neuron with two distinct inputs for the positive and for the negative synaptic weights is implemented by employing memristor elements and operational amplifiers. Single metal-oxide memristors are used to implement the synapses of the considered neural network. For the analyses and simulations, a simple, fast-operating and accurate LTSPICE memristor model with activation threshold is applied [35]. In this paper, the opportunity to implement memristor-based neural network circuits and to make analyses and comparisons between different realizations of neural nets in SPICE simulators is proposed to the readers. Future work on the subject is predicted, considering synthesis and analysis of more complex multi-layer memristor-based neural networks, as well as their practical implementation and comparison with other proposed realizations.

The paper is organized as follows. Section 2 presents memristors, their modeling, tuning and a brief comparison between memristor models. The LTSPICE realization and analysis of the applied memristor model is considered in Section 3. The considered simple and fast-operating LTSPICE memristor neuron is discussed in Section 4. The applied activation function based on memristors and MOS transistors is presented in Section 5. Section 6 presents the offered simple neural network with memristors and MOS transistors. Section 7 presents the derived results in MATLAB, SIMULINK and LTSPICE and their comparison. The conclusion shown in Section 8 summarizes the results and proposes future work on the topic.

2. Memristors - Modeling, Tuning, Comparison, and the Applied Memristor Model

For improvement and understanding the main conception of the work, according to the structure, operation, modeling, tuning and comparison of metal-oxide memristors [36,37], a short overview of the basic aspects of memristors is first presented.

2.1. A short depiction of memristor elements and their operation

The memristors are passive one-port electronic elements with memory and switching properties. In the scientific reports, a lot of memristors based on different materials, as metal-oxides, polymeric, ferroelectric, spintronic and others are described [17,18,19,20,21,22,23,24]. Metal-oxide memristors have a central position in the family of memristors, owing to their stable characteristics and parameters, very good switching and memorizing properties [24]. A part of the memristor nanostructure is doped by oxygen vacancies, applying electroforming processes [18]. The memory and switching effects in memristors depend on their ability to undergo alterations of their resistance, proportional to their state variable, when voltage or current signals are applied [19]. The state variable of a memristor x indicates the ratio between the lengths of the doped layer and the whole memristor structure. The memristor has two limiting values of its resistance, usually denoted as R_ON (the ON-resistance state) and R_OFF (the OFF-resistance state).

The behavior of memristors in electric field is quite different, in comparison to those of the classical electronic passive elements – the resistor, inductor and capacitor. Owing to its specifics modeling of memristors is mandatory for its analysis by electronic simulators and applications in electronic schemes and devices [19,26].

2.2. The Process of Modeling of Memristors [19,24,30]

Each mathematical model of a memristor is built on two key equations [26]. The first one offers the i-v relation, and the second one connects the time derivative of state variable x and the memristor current i (or the memristor voltage v). A large collection of repeatedly used metal-oxide memristor models, enclosing these of Williams-Strukov [18], Joglekar [22] and Biolek [23] is represented by the next general set of equations (1) [26]:

$$\left|\begin{array}{l}i=v·M\hspace{0.17em}{\left(x\right)}^{-1}\\ \frac{dx}{dt}=k·f\left(x\right)·i\end{array}\right.$$

(1)

where i and v are the memristor current and voltage, M is the so-called memristance (the state-dependent resistance of the memristor), x is the state variable of the memristive element, k is a constant which is dependent on the main physical parameters of the element – the ionic drift mobility µ, the ON-state resistance and the whole length D. The expression f(x) is a window function, utilized for limitation of the state variable in the interval from 0 to 1 and for introducing the boundary effects at a hard-switching mode [26]. Strukov-Williams memristor model [18] is described by the next set of equations (2):

$$\left|\begin{array}{l}i=v·{\left[R_{ON}x+R_{OFF}\left(1-x\right)\right]}^{-1}\\ \frac{dx}{dt}=k·f_{SW}\left(x\right)·i\\ f_{SW}\left(x\right)=\hspace{0.17em}-x\left(x-1\right)\hspace{0.17em}\end{array}\right.$$

(2)

where R_ON and R_OFF are minimal and maximal values of the memristance, and f_SW(x) is a simple and low-order polynomial window function proposed by Williams and Strukov [18]. This classical memristor model is a very simple one, and owing to the reduced number of the elementary mathematical operations it has a very high operating speed []. Owing to the linear dependence between the time derivative of the state variable dx/dt and the memristor current i, this model could not express the different behavior of the memristors for voltages, lower or higher than one volt [21,26]. The window function is related to the so called terminal state problems [20,31]. This memristor model has a comparatively low accuracy in simulation of complex and non-symmetrical experimental current-voltage characteristics of metal-oxide memristors. Joglekar memristor model is one of the widely-used metal-oxide memristor models [22] and it is presented by the following system (3):

$$\left|\begin{array}{l}i=v·{\left[R_{ON}x+R_{OFF}\left(1-x\right)\right]}^{-1}\\ \frac{dx}{dt}=k·f_J\left(x\right)·i\\ f_J\left(x\right)=1-{\left(2x-1\right)}^{2p}\end{array}\right.$$

(3)

where f_J (x) is a polynomial window, proposed by Joglekar and Wolf [22]. This traditional model for metal-oxide memristors is a simple one, and it has a comparatively high functioning speed [24,31]. Due to the linear relation between the time derivative of state variable x and the current, this model fails to capture the distinct behavior of memristors under voltages lower or higher than one volt. Terminal state problems are related to the applied window function [22,24]. The used polynomial window function is adjustable one and ensures different slope, according to the parameter p and the memristor state variable.

Biolek model for metal-oxide memristors, which is one of the mainly-utilized standard memristor models, is presented by the next system (4) [23]:

$$\left|\begin{array}{l}i=v·{\left[R_{ON}x+R_{OFF}\left(1-x\right)\right]}^{-1}\\ \frac{dx}{dt}=k·f_B\left(i,x\right)·i\\ f_B\left(i,x\right)=1-{\left(x-stp\left(-i\right)\right)}^{2p}\\ stp\left(i\right)=\left\{\begin{array}{l}0,\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}i<0\\ 1,\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}i\ge 0\end{array}\right.\end{array}\right.$$

(4)

where f_B (x,i) is a window function, proposed by Biolek, and p is a positive integer [23]. This classic model is comparatively simple and fast-operating one [24,30]. The window function correctly represents the boundary effects and is not related to terminal state issues [24]. This classical memristor model has a comparatively good accuracy in modeling of experimental current-voltage relationships.

Lehtonen-Laiho memristor model is a classic one, frequently applied for modeling of metal-oxide memristors, and is expressed by the next set of equations (5) [21]:

$$\left|\begin{array}{l}i=\hspace{0.17em}\left[\exp \left(\gamma \hspace{0.17em}v\right)-1\right]·\chi +\beta ·x^n·\sinh \left(\alpha \hspace{0.17em}v\right)\\ \frac{dx}{dt}=a·f_B\left(i,x\right)·v^m\end{array}\right.$$

(5)

where m and n are integer coefficients, and γ, α, β, and χ are coefficients for tuning the memristor model [21]. This model is a relatively complex one and has a high operating rate []. Due to the highly nonlinear relation between time derivative of state variable x and the memristor voltage, this standard model correctly expresses the different behavior of metal-oxide memristors for signals, higher or lower than 1 V [21]. Usually, in this memristor model, the classical Biolek window is applied [24,31]. This model has a high accuracy.

The applied in the present analyses modified memristor model, denoted by A14mod [35], is based on Joglekar memristor model and Lehtonen-Laiho model using the Hann window function sin²(πx) [24,35]. The model has an activation threshold v_thr, applied in the state differential equation, using the standard Heaviside step function [23]. This modified memristor model is presented by the next set of equations (6) [35]:

$$\left|\begin{array}{l}i=v·{\left[R_{ON}x+R_{OFF}\left(1-x\right)\right]}^{-1}\\ \frac{dx}{dt}=k·{\sin }^2\left(\pi \hspace{0.17em}x\right)·v^m·stp\left(\left|v\right|-v_{thr}\right)\end{array}\right.$$

(6)

where k is a constant dependent on the physical parameters of the memristor nanostructure and m is an odd and integer exponent [35]. This modified memristor model has a simple mathematical structure, the applied Hann window function is not related to terminal state problems. If the voltage is lower than the activation threshold v_thr, then the time derivative of the state variable is zero and the memristor behaves as a simple and linear resistor with a constant conductance [24,26]. This mode is used for the operation of metal-oxide memristors in adjusted neural networks after finishing the training processes and establishing constant values of the synaptic weights [24]. The modified memristor model A14 mod is a simple one, with a high operating speed and a good accuracy, and it is very appropriate for application in neural networks.

2.3. Fine adjustment and parameter estimation of the applied memristor model [24,31,32]

In this work, we analyse the considered modified model of a memristor based on metal oxides [35] and it is fully allocated by equation (6). This memristor model incorporates multiple parameters that allow its precise and fine-tuning process. In [35], this modified memristor model is tuned according to experimental data of metal-oxide memristors [20]. In this work, it is adjusted in accordance with current-voltage dependencies, derived by experimental data of Knowm self-directed channel memristors [36]. A technique for altering the model’s parameters till reaching the complete minimum of the Root Mean Square (RMS) error between the simulated and the experimental current of the memristor is applied [32]. Many scientists utilize simulated annealing and gradient descent algorithms for reaching the optimal values of the parameters of the memristor models [24,32]. The applied model is simple and applicable for simulations of memristor-based circuits and devices.

MATLAB-SIMULINK environment is utilized to extract the parameters from the considered memristor model [32]. The optimization approach for tuning the metal oxide memristor model involves modifying the coefficients and looking for the global minimum of the root mean square (RMS) error between the experimental and simulated current-voltage (i-v) characteristics. At each iteration step one of the coefficients of the model is altering by a small growth [32]. The RMS error between simulated and experimental i-v relations is computed. The other parameters of the model are also changing. After finalizing the tuning procedure, a graphical evaluation of the obtained current-voltage relationship and its nearness to the experimental one is also realized, paying attention on the form of the derived i-v pinched hysteresis loop and especially on the intervals of switching the resistance of the memristor element. The respective time diagrams of simulated and experimental memristor currents are also compared. An important condition for finalizing the adjustment process is minimization of RMS error [32]. Extra simulations are conducted for obtaining the optimal model’s coefficients, applying smaller increments for their alterations [32]. The precise values of memristor model’s parameters are also obtained, using the least squares method in MATLAB, applying Optimization Toolbox [32]. The parameter estimation process could be summarized in several main steps:

• Initialization - choosing the initial values of model’s coefficients, using their values, presented in the original references [36], and the parameters in the modified memristor model (6), after their evaluation in Simulink and MATLAB environment [32];
• Determination the value of root mean square error (RMSE) for stopping the estimation process, the corresponding deviations and the number of maximum iteration steps;
• Starting simulation and computing the current of the memristor element according to the applied modified memristor model;
• Evaluation of the RMS error (the cost function) between the calculated and experimental currents of the considered memristor;
• Alteration of model’s parameters according to a gradient descent of RMS error;
• When the determined RMS error is reached, or the maximal number of iteration steps are realized, the simulation process and evaluation of the derived model’s parameters are finished. The voltage signals in MATLAB-Simulink memristor model are previously sampled, the time step is with a value of 10 µs. The experimental memristor current is represented by i_mes. The output of the Simulink memristor model is related to the simulated current of the memory element i_calc. The cost function S_cost is an algebraic sum of the squared differences between the experimental and calculated currents – eq. (7) [24,32]:

$$S_{\cos t}\left(i_{mes},i_{calc}\right)={\displaystyle \sum _{k=1}^N{\left[i_{mes}\left(k\right)-i_{calc}\left(k\right)\right]}^2}$$

(7)

where N = 1000 is the total samples of the considered signals, and k is the number of the current and considered sample. The criterion for finalizing the parameter estimation process is minimizing the cost function S_cost [24]. The obtained optimal values of the model’s parameters are used for creation of the respective LTSPICE memristor library model, which is considered in the next section. The obtained values of the coefficients ensure that the RMS error is about 15 %. The derived time diagrams of simulated and experimental memristor current after the parameter extraction are presented in Figure 1a for their visual observation and comparison. The normalized values of trajectories of coefficients during the estimation process are presented in Figure 1b to express their alteration in the time domain. The initial values of parameters are chosen to be equal to the presented in [36]. After finishing the tuning process of the considered memristor model, the obtained optimal vales of the parameters are: k = 3.12.10⁶; x₀ = 0.0103; R_on = 3.37 ohm; R_off = 57.033 kΩ; m = 5; v_thr = 0.1583 V. After the parameter estimation process the modified memristor model is analyzed in Simulink-MATLAB and the derived results are presented in Figure 2 for their visual comparison. The obtained time diagrams of the applied voltage signal, the corresponding experimental and simulated memristor current and state variable after the parameter extraction process are shown in Figure 2a for their visual expression and comparison. The next Figure 2b represents the corresponding experimental and simulated current-voltage characteristics for their comparison and visual observation. For a short comparison of the considered modified metal-oxide memristor model to some of the standard existing and frequently utilized memristors, several main criteria, as the simulation time, the operation rate, accuracy and convergence [24,31] are considered in the next Table 1. After the conducted comparison between the memristor models, it could be stated that the applied modified memristor model A14mod has a little bit lower accuracy than Lehtonen-Laiho model, but it has lower complexity, good switching properties and high operating frequency [35]. According to the standard Joglekar and Biolek memristor models, the modified model A14mod has better characteristics and properties with a little bit higher complexity and simulation time.

The type of the applied memristor model is important for training the networks and adjustment of the synaptic weights of memristor-based neural networks by short voltage pulses. In the next section the generation and analysis of LTSPICE memristor model, corresponding to the considered modified memristor model, its main characteristics and properties are discussed.

3. LTSPICE Library Model [35] and analysis of the applied Memristor model

The main idea, utilized for construction of the LTSPICE [33] memristor library model, is to build a simple and fast-operating equivalent circuit, corresponding to (6). Before realization of the current-voltage characteristics, the derivation of the state variable x is needed. The memristor state variable is obtained by integration of the state differential equation with respect to time. This process is ensured by a capacitor; its current is set to be proportional to the time derivative of the state variable x [26,31]. The current of the capacitor is generated by a voltage-controlled current source, according to the left-hand side of the state equation in (6). The initial voltage of the integrating capacitor is set to be equal to the starting value of the memristor state variable. The voltage across the integrating capacitor is directly proportional to the state variable in each moment in the time domain. To avoid rapid changes of the voltage across the capacitor, which could lead to convergence issues, a high-valued resistor could be attached in parallel to the capacitor [31,36]. Additional voltage-controlled current source is generating the memristor current, according to the right-hand side of the first equation in (6). The initial row of the LTSPICE code displays the memristor model identifier, indicated as A14mod. The memristor terminals are denoted as top electrode (te) and bottom electrode (be). The optional terminal Y is utilized for measuring the memristor state variable. The parameters of the model - R_ON, R_OFF, k, m, mm, m0 and v_thr are presented in the next row. The coefficient m0 represents the initial value of the memristance. The integrating capacitor is connected between the terminal Y and the ground electrode. Its capacitance is 1000 mF. The initial voltage of integrating element C₁, denoted as “.IC”, is expressed by the first equation of (6) [35]. It is inversely proportional to the starting value of the memristance. The fifth row represents the additional resistor R_ad with a value of 100 GΩ, connected in parallel to C₁ and utilized for a partial avoidance of convergence problems. The dependent source G₂ presented in the sixth row expresses the time derivative of the state variable x.

The controlled source G₁ expresses the current of the memristor element. In row 8, a differentiable and flat step-like function is presented. It is utilized for realization of the state equation, containing the memristor activation threshold v_thr. The code, presented below is finished with the command “ends” [26,33,35].

1

.subckt A14mod te be Y

2

.params ron=100 roff=25e3 m0=300 k=10e4 m=5 vthr=0.2 mm=1e-24

3

C1 Y 0 {1}

4

.IC V(Y)={(m0-roff)/(ron-roff)}

5

R1 Y 0 100G

6

G2 0 Y value={((k*pow(V(te,be),m))*((pow(sin(pi*V(Y)),2)))*

  (stpp((vthr-abs(V(te,be))),mm)))}

7

G1 te be value={V(te,be)*((1/(ron*(V(Y))+roff*(1-V(Y)))))}

8

.func stpp(x,p)={0.5*(1+(x/sqrt(pow(x,2)+p)))}

9

.ends A14mod

The parameter mm determines the sharpness of the step-like function stpp. For a quick and easier tuning of the initial value of the memristance m₀, the state variable of the memristor in the initial moment is expressed as a function of m₀ in the fourth row. After selecting the memristor model A14mod in LTSPICE and placing on the plotting field, right-click on the memristor, in the SPICE row of the appeared window, write m₀=300, then place a tick in the right by double left mouse button click, to make m₀ visible and confirm by clicking OK. The considered LTSPICE code is included in a unified memristor model library, freely available for use and download at https://github.com/mladenovvaleri/Advanced-Memristor-Modeling-in-LTSpise [31]. This library contains many LTSPICE models of memristors and memristor-based electronic circuits, available for use and comparisons. The considered modified memristor model is analyzed with pulse and sine signals at a constant amplitude of 1.1 V and for different frequencies. In Figure 3a, the testing circuit for memristor analysis is presented for its visual observation. In the present case, the additional electrode Y is not included for simplification of the electronic circuitry. The initial value of the memristance is set to be 5 kOhms. After obtaining the i-v relationships of the memristor model for sinusoidal signals with amplitude of 1.1 V and different frequencies (100 kHz, 1 MHz and 10 MHz) in LTSPICE environment, they are presented in Figure 3b–d for their visual comparison and for confirmation of the correct operation of the modified memristor model. It is visible that, when the signal’s frequency increases, then the surface of the i-v loop decreases in accordance with the basic memristors’ properties and fingerprints [24,30].

Memristors, participating as synaptic weights, are tuned during the neural network training by positive or negative voltage pulses with a level, which absolute value is higher than the activation threshold v_thr [35]. An example of synaptic weight adjustment is presented in Figure 4. A simple electronic circuit, containing a pulse voltage source and a memristor, represented by the modified model A14mod, is shown in Figure 4a for a visual expression of its structure and operation. The voltage source generates a sequence of rectangular and positive pulses with a duration of 1 ns and a duty cycle of 50 %. The amplitude of the voltage is 1.1 V. Figure 4b represents the time diagrams of the pulse voltage signal and the corresponding change of the memristance M in LTSPICE. In this case the initial value of the memristance is M₁ = 10 kOhm, which corresponds to a synaptic weight of w₁ = 1. The final value of M₂ = 9 kOhm corresponds to a synaptic weight of w₂=1.1111. In this case, the change of memristance with ∆M = 1 kOhm corresponds to alteration of the synaptic weight with ∆w = 0.1111. The needed time interval for this memristance change, together with the pauses between the impulses, is about 28 ns. The effective duration of the voltage pulses for this time interval is about 14 ns. Figure 4c represents the results, derived by a simulation in MATLAB, which have a very good coincidence to Figure 4b and confirms the tuning process.

This result will be also mathematically confirmed by integration of the state differential equation of system (6), modified and represented here by (8):

$$\frac{dx}{dt}=k·{\sin }^2\left(\pi \hspace{0.17em}x\right)·v^m$$

(8)

In eq. (8), because the used voltage pulses are higher than the activation threshold v_thr, in the right hand side, the step function “stp” is missed. Equation (8) is transformed to (9), where the state variable x and the time variable t are separated one to another:

$$\frac{dx}{{\sin }^2\left(\pi \hspace{0.17em}x\right)}=k·v^{m}dt$$

(9)

According to the current-voltage relationship in system (6), the state variable x is expressed as a function of the memristance M by the next eq. (10):

$$x=\frac{M-R_{OFF}}{R_{ON}-R_{OFF}}$$

(10)

Equation (9) is integrated between x₁ = 0.8251, corresponding to M₁ = 10 kOhm, and x₂ = 0.8427, related to M₂ = 9 kOhm, using (10), and the following eq. (11) is obtained:

$${\displaystyle \underset{x_1}{\overset{x_2}{\int }}\frac{1}{{\sin }^2\left(\pi \hspace{0.17em}x\right)}\hspace{0.17em}}dx=k·v^m{\displaystyle \underset{0}{\overset{T_{\max }}{\int }}dt\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\Rightarrow \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}-\left[\frac{cotg\left(\pi x_2\right)}{\pi }-\frac{cotg\left(\pi x_1\right)}{\pi }\right]}=k·v^m·T_{\max }$$

(11)

Then, the effective duration of the applied pulses (a single pulse or a sequence of short impulses) is obtained as follows – formula (12):

$$T_{\max }=\frac{-\left[\frac{cotg\left(\pi x_2\right)}{\pi }-\frac{cotg\left(\pi x_1\right)}{\pi }\right]}{k·v^m}=\frac{-\left[\frac{cotg\left(\pi \mathrm{.0.8427}\right)}{\pi }-\frac{cotg\left(\pi \mathrm{.0.8251}\right)}{\pi }\right]}{{3.10}^6·{1.1}^5}=14.53\hspace{0.17em}ns$$

(12)

The obtained time for synaptic weight adjustment corresponds to the derived simulation result and confirms the proper tuning the memristance by pulses. Another simulation is conducted with negative voltage pulses and the same circuit shown in Figure 4a is used. The respective time diagram of the applied negative pulses and the corresponding memristance alteration in LTSPICE is presented in Figure 5a. Figure 5b represents the results, derived from analysis in MATLAB, which is in a good agreement with Figure 5b and confirms the correctness of the adjustment process. The analysis starts with initial value of the memristance, equal to M₁ = 5 kOhms, related to a synaptic weight of w₁ = 2 and a state variable of x₁ = 0.9129. The final value of the memristance M₂ is 6 kOhms, corresponding to a synaptic weight with a value of w₂ = 1.6667 and a state variable x₂ = 0.8954. The change of the memristance with 1 kOhm in this case corresponds to alteration of synaptic weight with ∆w = 1.6667-2= - 0.3333. The effective duration of the pulse is about 79ns/2 = 39.5 ns.

The analytical confirmation of this result is as follows – formula (13):

$$T_{\max }=\frac{-\left[\frac{cotg\left(\pi x_2\right)}{\pi }-\frac{cotg\left(\pi x_1\right)}{\pi }\right]}{k·v^m}=\frac{-\left[\frac{cotg\left(\pi \mathrm{.0.8954}\right)}{\pi }-\frac{cotg\left(\pi \mathrm{.0.9129}\right)}{\pi }\right]}{-{3.10}^6·{1.1}^5}=41.5\hspace{0.17em}ns$$

(13)

For the future practical implementation of memristor-based synapses, the tuning process will be related to a measurement of the achieved memristance, until reaching the needed resistance and stopping the pulse sequence. According to the requirement for a quick change of the memristance and the corresponding synaptic weight, the applied modified memristor model A14mod ensures a rapid alteration of the state variable and the memristance at applied voltage pulses. Its simple structure and low simulation time ensures the generation and analysis of complex neural networks with a high number of memristors in LTSPICE environment. Additional comparison of the applied memristor models, according to simulation time is conducted for two cases – decreasing and increasing of memristance, according to the analyses shown in Figure 4 and Figure 5. The results presented in Table 2 express the faster operation of the applied modified model, according to Lehtonen-Laiho model. The simulations were conducted on a computer with 8 GB RAM, i5 Intel processor, and Microsoft Windows 10 Pro operating system. According to Joglekar and Biolek models, the modified model A14mod has comparable simulation time and better switching properties.

4. Memristor-based Transfer Function

The considered realization of a logarithmic-sigmoidal activation function is presented in Figure 6a for its visual expression and for further discussion. The considered transfer (activation) function is realized by two MOS transistors M₁ and M₂, connected in anti-parallel, the memristor U₁ and a DC voltage source V₂ [35]. The MOS transistors are of N-type and in the present case, they are chosen to be Si7540DP_N, using the standard LTSPICE library [33]. The transistors are connected as diodes and their gate and source terminals are directly attached one to another. The anti-parallel connection of the MOS transistors ensures a symmetrical transfer function. The memristor U₁ is connected in series to the MOS transistors. Its voltage drop is the difference between the input and the output signals. The voltage source V₂ ensures shifting of the output signal [35]. The dependence between the output and input voltage signals is very similar to a logarithmic-sigmoidal function, presented by the following expression (14):

$$v_{out}=\frac{1}{1+\exp \left(-k·v_{in}\right)}$$

(14)

For determination of the optimal value of the coefficient k, several different experiments and comparisons are conducted with the use of the dependent voltage source B₁ shown in Figure 6b, which voltage represents a logarithmic-sigmoidal activation function. It is established that for k = 5 and m₀ = 150 Ohms, the best proximity between the transfer functions is realized. The expression of the voltage of the behavioral source B₁ is v_tf = 1/(1+exp(-5*v_{in_tf})).

The n-channel MOS transistor Si7540DP_N used in this paper has a simpler construction, lower drain-source voltage, lower drain current, and higher speed compared to the Si4864DY transistor [35]. The activation functions depicted in Figure 7 correspond to the circuits illustrated in Figure 6. These graphics are presented for their visual expression and comparison of their proximity. The transfer function of the MOS transistor-based schematic is represented by the black curve, while the theoretical activation function is depicted by the blue curve. After several experiments with different values of the coefficient k, a very good proximity between the considered transfer functions is established when the coefficient in the theoretical expression is k = 5. In the next section, an artificial neuron, based on memristors is discussed.

5. Memristor-based Neuron [35]

Figure 8 presents a block diagram of a neuron and illustrates the structure and operation of a neuron based on memristors for further explanations and discussion [2,3]. The input signals of the neuron are represented as x₁, x₂, … and x_N. The signals are directed to synapses based on memristors, with corresponding weights denoted as w₁, w₂, ..., and w_N. A single memristor is employed for each synapse, and its resistance determines the corresponding synaptic weight. The change of the respective synaptic weight is realized by alteration of the corresponding memristance with externally applied voltage or current pulses. The signals obtained after the memristor synapses are directed to an adder, implemented by the use of operational amplifiers and memristors, operating as linear and constant resistors [34,35]. The signal derived after the adder is represented as y_in. This signal is directed to a log-sigmoidal transfer function, constructed by metal-oxide memristors and MOS transistors [35]. The output signal of the artificial neuron is indicated by y.

The functioning of the memristor-based neuron under consideration is related to a feed-forward and back-error propagation learning algorithm [3]. Various activation functions could be utilized after the adder [3,7]. Commonly utilized in the neural networks are smooth and differentiable transfer functions, such as logarithmic-sigmoidal and tangent-sigmoidal ones [7]. The relay activation function, also known as the standard Heaviside function, is also utilized in artificial neural nets [3]. A gain factor can also be utilized to apply a linear activation function in artificial neurons. Figure 9 illustrates a principal schematic of the considered artificial neuron based on memristors providing a description of its structure and operation. The input signals x₁ – x₆ are normalized in the interval between -0.1 and +0.1 volts, in order to avoid alteration of the resistances of the synaptic weights during the application of the adjusted neural network. The number of the memristors, realizing the synaptic weights could vary, according to the needed inputs of the artificial neuron. The synaptic weights are presented by the metal-oxide memristors M₁ – M₆. The memristors M₁, M₂ and M₃ are related to negative weights, while the other memristors - M₄, M₅ and M₆ are placed for realization of the positive synaptic weights. The considered synapses are connected to the inverting inputs of the op-amps OA₁ and OA_2, respectively. The memristors M₇ and M₉ are applied for realization of voltage feedbacks of the respective op-amps. The output signal of the op-amp OA₂ is directed to the inverting input of op-amp OA₁ via the memristor element M₈. The non-inverting inputs of the operational amplifiers are connected to the ground [34,35]. The feedback memristors M₇, M₈ and M₉ are adjusted to a memristance of 10 kOhms.

Using Kirchhoff’s laws, the output signal of op-amp 1, correspondent to the signal y_in, is derived as follows – formula (15):

$$y\_in=v_{out\_OA1}=-\frac{M_7}{M_1}{x}_1-\frac{M_7}{M_2}{x}_2-\frac{M_7}{M_3}{x}_3-\frac{M_7}{M_8}\left(-\frac{M_9}{M_4}{x}_4-\frac{M_9}{M_5}{x}_5-\frac{M_9}{M_6}{x}_6\right)$$

(15)

Having in mind that M₇ = M₈ = M₉ = 10 kΩ, the next expression (16) is derived after algebraic processing of formula (15):

$$y\_in=v_{out\_OA1}=-\frac{M_7}{M_1}{x}_1-\frac{M_7}{M_2}{x}_2-\frac{M_7}{M_3}{x}_3+\frac{M_7}{M_4}{x}_4+\frac{M_7}{M_5}{x}_5+\frac{M_7}{M_6}{x}_6$$

(16)

where the coefficients in front of the signals x₁ – x₆ are the corresponding synaptic weights - formula (17) [34,35]:

$$\begin{array}{l}{w}_1=-\frac{M_7}{M_1};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{w}_2=-\frac{M_7}{M_2};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{w}_3=-\frac{M_7}{M_3};\hspace{0.17em}\hspace{0.17em}\\ w_4=\frac{M_7}{M_4};\hspace{0.17em}\hspace{0.17em}{w}_5=\frac{M_7}{M_5};\hspace{0.17em}\hspace{0.17em}{w}_6=\frac{M_7}{M_6}\end{array}$$

(17)

Observably, synaptic weights w₁, w₂, and w₃ are with negative values, whereas the weights from w_4, w₅ and w₆ are positive.

Figure 10 illustrates the implementation of the considered memristor neuron in LTSPICE environment, providing further details and discussions. It is associated with the electronic circuit presented in Figure 11. The presented pulse voltage sources V₁, V₂, V₇, V₈, V₉ and V₃₇ are corresponding to the signals x₁ – x₆ and they generate the input signals – v_in1, v_in2, v_in3, v_in4, v_in5 and v_bias. The adder has two inputs – for negative and for positive synaptic weights, which are denoted by minus and plus, respectively [35]. The output electrode of the adder is denoted by “summ”. The structure of the log-sigmoidal transfer function corresponds to Figure 6a. A detailed realization of the adder in LTSPICE simulator is presented in Figure 11 for its description and further explanations. The op-amps U₁ and U₂ are power supplied by two DC voltage sources of 3 Volts. Sometimes, the voltage across the feedback memristors could exceed the activation threshold of the applied metal-oxide memristors [35]. The feedback voltage depends on the output signal y_in, which is proportional to the weighted sum of the input signals. To avoid the respective voltages across the feedback memristors to exceed the memristor activation threshold, the voltage feedback elements M₇, M₈ and M₉ are substituted by memristor-based blocks, enclosing two memristors connected in a series connection [35].

The exact number of memristor elements n in a block, realizing a feedback element could be calculated, using the absolute value of the output signal v_summ=y_in (in Figure 9), divided by the activation threshold v_thr, and applying the truncation function “ceil”, which returns the smallest integer, not less than its argument – formula (18):

$$n=ceil\left(\frac{\left|v_{summ}\right|}{v_{thr}}\right)$$

(18)

The signal after the adder v_summ is connected to a memristor-based voltage divider and a buffer amplifier, presented in Figure 12. The transfer coefficient of the voltage divider is 0.1. This divider is needed for ensuring the normal operation of the memristor-based synapses of the next layer of the neural network [34,35], which voltage should not exceed the activation threshold v_thr.

The element U₄₆ has a constant resistance of 1777.78 Ohms and is realized by only one memristor, while the number of the other memristors in the voltage divider is determined by the next formula (19), to ensure that the voltage drop across the memristors does not exceed the activation threshold v_thr:

$$n=ceil\left(\frac{\left|v_{tfprim4}\right|}{v_{thr}}\right)$$

(19)

Sometimes, the normal operation of the memristors in the voltage divider could be violated, owing to the connected synapses of the next layer of the neural network. To ensure their normal functioning without exceeding the activation threshold and to avoid functioning in a soft-switching mode, a buffer amplifier is connected after the voltage divider. It is based on non-inverting operational amplifier.

6. Memristor-Based NN with TF, realized with MOS transistors

The block diagram of the memristor-based neural network under analysis is presented in Figure 13 for its visual expression and further comments on their structure and operation. It contains five input nodes for applying the input signals x₁ – x₅. The hidden layer of the neural network contains four neurons – N₁, N₂, N₃ and N₄. The bias signals are denoted by b₁, b₂, b₃ and b₄. The synaptic weights are denoted by w_ij, where i = 1 - 5 is the number of the neuron, and j is the number of the input signal. The adders are used for summing the weighted input signals. The output signals of the adders are denoted by s₁, s₂, s₃ and s₄. These signals are applied to the logarithmic-sigmoidal activation functions tf₁, tf₂, tf₃ and tf₄, respectively. The described elements form the hidden layer of the considered neural network. The signals tf₁, tf₂, tf₃ and tf₄ are applied to the output layer of the neural network, which contain two neurons – N_out1 and N_out2. The synaptic weights are denoted by v_kp, where k is the number of the neuron, and p is the number of the respective input signal. The bias signals of the adders are denoted by b_out1 and b_out2. The output signals of the adders are denoted by s_1out and s_2out. The applied transfer functions tf_out1 and tf_out2 are of type “purelin” []. They are realized with buffer amplifiers, using op-amps. The output signals of the neural network are denoted by y₁ and y₂.

The operation of the presented memristor-based neural network is based on its training in MATLAB environment, using feed-forward and back-error propagation algorithm, for adjustment of the synaptic weights [3,32]. The obtained values of the weights are used for tuning the memristances and to create the neural network in LTSPICE environment, which will be discussed in the next section.

The input signals are sequences of rectangular voltage pulses. The pulse duration is 0.8 ms. The duty cycle is 50 %. The levels of the input and the bias voltage signals are shown in Table 3 for their visual evaluation. The bias signal is set to 100 mV and does not exceed the memristor activation threshold v_thr.

The synaptic weights for the hidden layer of the neural network are presented in Table 4 for their evaluation and for further discussion.

The corresponding resistances of the single-memristor synapses are denoted by M₁ – M₅ and are presented in Table 5. Their values are obtained using formula (11). Having in mind that the bias signal is v_bias = 0.1 V, instead of 1 V, the value of the memristance M_bias is obtained by multiplication with 10.

The synaptic weights of the synapses in the output layer are presented in Table 6.

The memristances, corresponding to the synaptic weights for the output layer are represented in Table 7.

A principal schematic of the considered memristor-based neural network is illustrated in Figure 14. The input signals are denoted by x₁ – x₅, the bias signal is denoted by b. This schematic corresponds to the block diagram, presented in Figure 13. The electronic realization of the considered neural network in LTSPICE simulator, corresponding to the block diagram of Figure 13 and to the principal schematic presented in Figure 14, is given in Figure 15. It is presented for a visual observation and realization by the readers in similar SPICE simulators. The circuits is also available at: https://github.com/mladenovvaleri/Advanced-Memristor-Modeling-in-LTSpise [31]. Figure 16а illustrates the time diagrams of the input signals and the bias signal, giving their visual representation. The signals after the adders are presented in Figure 16b. Figure 16c represents the signals after the logarithmic-sigmoidal transfer functions and Figure 16d illustrates the output signals of the neural network in LTSPICE environment. The considered neural network is also realized in SIMULINK-MATLAB environment [32]. The SIMULINK schematic of the network is presented in Figure 17 for its visualization and further comments. The input signals are realized by pulse generators. They are forwarded to MATLAB workspace by the standard blocks “To workspace” for further analysis. The synaptic weights are corresponding to gain amplifiers, which zoomed diagram is given the right down corner of the schematic. The adders are realized by standard summing blocks. The logarithmic-sigmoidal and the linear transfer function is realized by “MATLAB function” blocks. The time step is 0.001 ms. The simulation time is 6 ms. The time diagrams, obtained by the function “plot” after finishing the simulation on MATLAB-SIMULINK environment are presented in Figure 18. The input signals x₁ – x₅ are represented in Figure 18a. The signals after the adders and after the transfer functions of the hidden layer are illustrated in Figure 18b. Figure 18c presents the output signals of the neural network.

The considered neural network is also simulated in MATLAB, using the neural network toolbox [32], for conducting a comparison of the results, derived in SIMULINK and in LTSPICE [33]. Table 8 compares the signals’ levels, derived in LTSPICE and MATLAB -Simulink [32]. Identical results are derived from the conducted analyses and simulations. This comparison confirms the capability of the considered memristor-based neural network and activation function to properly operate with pulse input signals with levels lower than the activation threshold of the applied metal-oxide memristors.

4. Discussion

Comparing the simulation results of the considered neural network, realized in LTSPICE, MATLAB and SIMULINK software, a very good matching of the signals after the adders, the transfer functions and the outputs is established. The maximal error between the output signal’s levels is less than 3 %. The synapses of the neural network are implemented with single memristors and operational amplifiers for summation, utilizing their inverting inputs. The suggested synapses implement both positive and negative weights. The logarithmic-sigmoidal activation function utilized in the research relies on a combination of a metal-oxide memristor and two MOS transistors, connected in anti-parallel. The analyses demonstrate a strong correlation between the obtained results, confirming the accurate functionality of the suggested memristor-based neural network.

5. Conclusions

The main purpose of this paper - realization of a complete electronic implementation of a memristor-based neural network in LTSPICE environment has been achieved. The related tasks are resolved – a modified logarithmic-sigmoidal transfer function, based on memristors and MOS transistors is realized; a simple adder, founded on memristors and operational amplifiers is realized with minimal number of electronic elements; a simple multi-layer artificial neural network, using memristors for the synaptic weights of positive and negative signs, employing minimal number of electronic components in LTSPICE environment is generated and analyzed. The analyses and simulations are performed, using MATLAB, SIMULINK and LTSPICE. We have applied A14mod memristor model, which is better than the existing ones (in terms of faster operation and simulation), which model helps for faster simulation when the synaptic weights are tuning, during the training of the neural network. The main advantage of the considered memristor-based artificial neural network is its simple realization with the use of very low-power and nano-sized metal-oxide memristor elements, with a very good compatibility to the present-day CMOS high density integrated electronic chips and circuits. This work introduces an opportunity for readers to explore implementations of neural network circuits, based on memristors. It encourages the analysis and comparison of various realizations of artificial neural networks, using SPICE simulators, together with practical measurements. The future work in this field will be related to exploration of synthesis and analysis of more complex multi-layer memristor-based neural networks for artificial intelligence. Additionally, it aims to explore into practical implementation of such neural networks and compare them with other similar proposed implementations.