Speed Optimization Control of Permanent Magnet Synchronous Motor Based on TD3

1. Introduction

PMSM has gained significant traction in industrial automation and electric vehicle applications due to their exceptional efficiency, high power density, and superior dynamic performance. The PMSM drive system represents a typical dual time-scale system. Among the most efficient control designs for PMSM is the cascade structure, which incorporates a rapid inner loop responsible for armature current control while employing a slower outer loop intended to adjust angular velocity by generating appropriate current reference signals. However, achieving the desired control performance for PMSM applications presents challenges. Including issues such as parameter variations, external disturbances, and system nonlinearity. In reference [1], it is pointed out that the permanent magnetic flux linkage can vary by 20% of its nominal value, while the stator resistance can vary by 200% of its nominal value." In practical applications, researchers often aim to employ a stable and highly accurate control strategy that can effectively achieve rapid response and robustness in the presence of uncertain parameters. However, conventional proportional-integral (PI) controllers prove inadequate for tracking the outer loop velocity when motor parameter uncertainties arise. Consequently, researchers have explored diverse advanced control techniques to address the velocity servo problem, encompassing model predictive control (MPC), robust control, adaptive control, fuzzy control, disturbance rejection control, sliding mode control, prediction-based model-free control as well as deep learning and reinforcement learning-based controls. The paper [2] proposes a robust model predictive current control method based on nonlinear extended state observation to enhance the control performance of PMSM in the presence of parameter variations. The paper [3] presents a robust adaptive model predictive speed control method based on recurrent neural network to tackle the speed control problem of permanent magnet synchronous motors under parameter mismatch conditions.

The paper [4] presents a universal control framework that utilizes an observer to estimate both the system state and disturbances, while establishing a predictive current controller based on an enhanced system model. The paper [5] introduces a motor-parameter-free model predictive voltage control strategy for permanent magnet synchronous motor drive systems. Its fundamental concept aims at reducing dependence on motor parameters in PMSM control, thus enhancing the robustness of this control strategy. The literature [6,7,8,9,10] employs the MPC algorithm to mitigate torque fluctuations in PMSM drive systems induced by interturn faults (ITF). It simplifies the complexity of control methods for PMSM drives considering ITF by incorporating an adaptive compensation current approach.

In the field of robust adaptive control, the compensator based on the extended state observer proposed in reference [11] effectively addresses the issue of excessively high switching gains required for disturbance rejection. Reference [12] introduced a system transformation method that effectively converts a PMSM system with current constraints into an unconstrained system, thereby streamlining the controller design process. Reference [13] proposed an observer scheme utilizing neural networks along with a sensorless robust optimal control approach to address the speed and current tracking challenges in partially unknown PMSM systems under disturbances and saturation voltages. The paper [14] integrates the adaptive integral sliding mode method and employs a self-regulation approach to adjust the amplitude of the sliding mode function as well as compensate for load disturbances, thereby enhancing the dynamic performance of the system. The paper [15,16,17,18,19] utilizes deep reinforcement learning(DRL) to solve the control problem of PMSM. By introducing artificial intelligence algorithms into the traditional parameter optimization process, a DRL model is constructed that can automatically optimize and adjust parameters in different application scenarios, thereby achieving optimal control effects in various environments.

In the domain of prediction-based control strategies, reference [20] introduces an advanced model-free active disturbance rejection dead-zone predictive current control approach specifically designed for permanent magnet synchronous motors. This approach is based on a data-driven methodology, aiming to address the issue of parameter mismatch in dead-zone prediction current control and improve the performance of PMSM control systems. The paper [21] introduces a model-free predictive current control drive system that incorporates an extended Kalman filter to address performance degradation in model predictive control due to variations in motor parameters. The paper [22] introduces a model-free predictive current control strategy for the drive system of PMSM in electric vehicles. This innovative approach effectively mitigates performance limitations caused by inaccurate inertia estimation through real-time dynamic adjustment of inertia parameters. The paper [23] presents a novel speed control strategy that combines an adaptive speed controller with a radial basis function neural network for precise speed regulation of PMSM. This approach effectively mitigates the impact of parameter uncertainties and load variations on system performance. The paper [24] utilizes a linear-nonlinear switching active disturbance rejection control strategy to design speed and current controllers for PMSM in servo systems, aiming to improve the disturbance rejection performance of the PMSM speed and current controllers. The paper [25] presents an optimal tracking control strategy for PMSM systems characterized by partially unknown dynamics, voltage saturation, and varying speed and current. By integrating an advanced feedforward control input, the conventional velocity and current tracking challenges are redefined as novel optimal control problems within a cascaded framework. Experimental results demonstrate that both tracking, and approximation errors are uniformly bounded.

In the vector control system of a three-phase PMSM, the conventional PI regulator is widely adopted as the speed controller due to its simplicity and robustness. However, the PMSM exhibits nonlinear dynamics and strong coupling among multiple variables. In the presence of external disturbances or variations in internal parameters, the traditional PI control method struggles to meet stringent control requirements. To improve the dynamic performance of the PMSM speed regulation system, it is crucial to implement a control strategy that remains insensitive to external disturbances and parameter changes while ensuring rapid response and high accuracy. Furthermore, achieving high-performance control for a PMSM requires precise rotor position and speed information within the magnetic field-oriented vector control framework. The use of mechanical sensors for this purpose, however, increases system cost, size, and weight, and imposes strict operating environment constraints. Sensorless control technology addresses these challenges by monitoring electrical signals within the motor windings and employing advanced algorithms to accurately estimate rotor position and speed, thereby enhancing the robustness and reliability of the PMSM vector control system. This paper introduces an energy-optimized speed control algorithm based on the TD3 for PMSM drive systems. The main contributions are summarized as follows.

●The TD3-based optimal control reduces the design difficulty of the speed tracking controller for nonlinear PMSMs.

●Adding energy consumption optimization control to the traditional control objective of steady, accurate and fast effectively improves the efficiency of the motor.

●The better generalization of the algorithm enables the motor to exhibit better control performance under different operating conditions.

2. Description of the Control Problem

2.1. Control Cbject Model

The motion equation of PMSM can be described as follows:

$$J{\displaystyle \frac{d{\omega }_m}{dt}}=T_e-T_L-B{\omega }_m$$

(1)

$$T_e={\displaystyle \frac{3}{2}}{p}_n{i}_q\left[i_d\left(L_d-L_q\right)+{\phi }_f\right]$$

(2)

Among them, ${\omega }_m$, $J$, $B$ and $T_L$ represent the mechanical angular velocity, moment of inertia, damping coefficient, and load torque of the motor respectively. To facilitate controller design in a synchronous rotating coordinate system $d-q$ for PMSM models are commonly established. The stator voltage equation can be expressed as equation (3).

$$\left\{\begin{array}{c}{u}_d=R{i}_d+{\displaystyle \frac{d}{dt}}{\phi }_d-{\omega }_e{\phi }_q\\ u_q=R{i}_q+{\displaystyle \frac{d}{dt}}{\phi }_q+{\omega }_e{\phi }_d\end{array}\right.$$

(3)

The stator magnetic flux equation is given by equation (4).

$$\left\{\begin{array}{l}{\phi }_d=L_d{i}_d+{\phi }_f\hfill \\ {\phi }_q=L_q{i}_q\hfill \end{array}\right.$$

(4)

The substitution of equation (4) into equation (3) yields the stator voltage equation as (5).

$$\left\{\begin{array}{l}{u}_d=R{i}_d+L_d{\displaystyle \frac{d}{dt}}{i}_d-{\omega }_e{L}_q{i}_q\hfill \\ u_q=R{i}_q+L_q{\displaystyle \frac{d}{dt}}{i}_q+{\omega }_e\left(L_d{i}_d+{\phi }_f\right)\hfill \end{array}\right.$$

(5)

Among them: $u_d,u_q$ are the components of the stator voltage on the $d-q$ axis, while $i_d,i_q$ are the components of the stator current on the $d-q$ axis, $R$ represents the stator resistance. ${\phi }_d,{\phi }_q$ are the X-axis components of the stator magnetic flux. ${\omega }_e$ represents the angular velocity of electricity, $L_d,L_q$ are the inductance components of the $d-q$ axis, ${\phi }_f$ represents the magnetic flux of a permanent magnet. In addition, it is important to pay attention to the relationship between variables in equation (6) when constructing a PMSM simulation model.

$$\left\{\begin{array}{c}{\omega }_e=p_n{\omega }_m\\ N_r={\displaystyle \frac{30}{\pi }}{\omega }_m\\ {\theta }_e={\displaystyle \int {\omega }_e}dt\end{array}\right.$$

(6)

Where:${\omega }_m$ represents the mechanical angular velocity of the motor ($rad/s$), and $N_r$ represents the motor speed ($r/\min$). The common methods of traditional vector control include $i_d=0$ control and maximum torque current ratio control，The control method where the $dq$-axis inductance of surface-mounted PMSM is equal, i.e., $L_d=L_q$ and $i_d=0$, is mainly applicable to three-phase surface-mounted PMSMs. In this case, equation (2) can be rewritten as equation (7).

$$T_e={\displaystyle \frac{3}{2}}{p}_n{i}_q{\phi }_f$$

(7)

For surface-mounted three-phase PMSM, $i_d=0$ control and maximum torque current ratio control are equivalent.

2.2. Speed Loop Control

Assuming the motor starts with no load, i.e., $T_L=0$, when adopting control strategy $i_d=0$, the active damping is defined as equation (8).

$$i_q={i^{\prime }}_q-B_a{\omega }_m$$

(8)

The combination of equation (1) and equation (2) yields equation (9), thereby establishing a comprehensive relationship.

$${\displaystyle \frac{d{\omega }_m}{dt}}={\displaystyle \frac{1.5p_n{\phi }_f}{J}}\left({i^{\prime }}_q-B_a{\omega }_m\right)-{\displaystyle \frac{B}{J}}{\omega }_m$$

(9)

Reposition the poles of equation (9) at the desired closed-loop bandwidth $\beta$, and apply a Laplace transform to derive the transfer function representing motor speed in relation to shaft current as equation (10).

$${\omega }_m\left(s\right)={\displaystyle \frac{1.5p_n{\phi }_f/J}{s+\beta }}{i^{\prime }}_q\left(s\right)$$

(10)

The $\beta$ represents the desired bandwidth of the speed control loop. The coefficient $B_a$ of active damping, as derived from the comparison between equation (9) and equation (10), is presented in equation (11).

$$B_a={\displaystyle \frac{\beta J-B}{1.5p_n{\phi }_f}}$$

(11)

The speed loop controller can be mathematically represented as equation (12) when employing a conventional PI controller.

$${i_q}^{*}=\left(K_p+{\displaystyle \frac{K_i}{s}}\right)\left({{\omega }_m}^{*}-{\omega }_m\right)-B_a{\omega }_m$$

(12)

Where $K_p={\displaystyle \frac{\beta J}{1.5p_n{\phi }_f}},K_i=\beta K_p$.

2.3. Current Loop Control

The current equation for the $d-q$-axis, which corresponds to equation (13), can be obtained by rewriting equation (5).

$$\left\{\begin{array}{l}{\displaystyle \frac{d}{dt}}{i}_d=-{\displaystyle \frac{R}{L_d}}{i}_d+{\displaystyle \frac{L_q}{L_d}}{\omega }_e{i}_q+{\displaystyle \frac{1}{L_d}}{u}_d\hfill \\ {\displaystyle \frac{d}{dt}}{i}_q=-{\displaystyle \frac{R}{L_d}}{i}_d-{\displaystyle \frac{1}{Lq}}{\omega }_e\left(L_d{i}_d+{\phi }_f\right)+{\displaystyle \frac{1}{L_q}}{u}_q\hfill \end{array}\right.$$

(13)

The complete decoupling of $i_d$ and $i_q$ yields equation (14).

$$\left\{\begin{array}{l}{u}_{d0}=u_d+{\omega }_e{L}_q{i}_q\hfill \\ u_{q0}=u_q-{\omega }_e\left(L_d{i}_d+{\phi }_f\right)\hfill \end{array}\right.$$

(14)

The substitution of equation (5) into equation (14) results in the derivation of equation (15).

$$\left\{\begin{array}{l}{u}_{d0}=R{i}_d+L_d{\displaystyle \frac{d}{dt}}{i}_d\hfill \\ u_{q0}=R{i}_q+L_q{\displaystyle \frac{d}{dt}}{i}_q\hfill \end{array}\right.$$

(15)

where $u_{d0},u_{q0}$ are the $d$-axis and $q$-axis voltages after current decoupling, respectively. Using the conventional PI controller and combining with the feed-forward decoupling control strategy, the $d-q$-axis voltage can be obtained as Eq. (16).

$$\left\{\begin{array}{l}{u_d}^{*}=\left(K_{pd}+{\displaystyle \frac{K_{id}}{s}}\right)({i_d}^{*}-i_d)-{\omega }_e{L}_q{i}_q\hfill \\ {u_q}^{*}=\left(K_{pd}+{\displaystyle \frac{K_{iq}}{s}}\right)({i_q}^{*}-i_q)+{\omega }_e(L_d{i}_d+{\psi }_f)\hfill \end{array}\right.$$

(16)

where $K_{pd},K_{pq}$ is the proportional gain of the PI regulator and $K_{id},K_{iq}$ is the integral gain of the PI regulator.

The block diagram in Figure 1 illustrates the three-phase PMSM vector control employing the method with $i_d=0$. It is evident from the figure that three-phase PMSM vector control primarily comprises of three components, namely a speed loop PI controller, a current loop PI controller, and the Space Vector Pulse Width Modulation (SVPWM) algorithm.

3. TD3 OF PMSM

The TD3 algorithm proposes three key enhancement techniques based on the -deep deterministic policy gradient(DDPG) algorithm, as delineated below. (1)Double network refers to the utilization of two Critic networks, wherein the smaller one is employed for computing the target value in order to mitigate potential issues related to overestimation.(2)Target policy smoothing regularization: Incorporate perturbations to the action in the subsequent state while calculating the target value, aiming to enhance the precision of value evaluation.(3) Following multiple updates of the Critic network, subsequent updates are made to the Actor network to enhance training stability. Figure 2 shows the structure of TD3 algorithm.

The update process of the TD3 algorithm is not significantly different from that of the DDPG algorithm, except for the distinction in how the target values are computed. The Actor network updates by maximizing the cumulative expected return through deterministic policy gradient, while both Critic1 and Critic2 networks update by minimizing the error between the evaluation value and the target value using mean squared error. All target networks are updated using an exponential moving average (EMA) soft update method. During the training phase, a batch of data is sampled from the Replay Buffer with a specific batch size. Assuming one sample is denoted as $\left(s,a,r,s^{\prime },done\right)$, the update process for all networks follows. The updating procedure for Critic1 and Critic2 networks entails employing the Target Actor network to calculate the action $a^{\prime }={\mu }^{\prime }\left(s^{\prime }|{\theta }^{{\mu }^{\prime }}\right)$ in state $s^{\prime }$. Subsequently, target policy smoothing regularization is implemented, and noise $\epsilon$ is incorporated into the target action $a^{\prime }$.

$$\begin{array}{c}{a}^{\prime }=a^{\prime }+\epsilon \\ \epsilon ~clip\left(N\left(0,\sigma \right),-c,c\right)\end{array}$$

(17)

Building upon the concept of dual networks, equation (18) is employed for the computation of the target value $y$.

$$y=r+\gamma {\min }_{i=1,2}{Q_i}^{\prime }\left(s^{\prime },a^{\prime }|{\theta }_i^{Q^{\prime }}\right)$$

(18)

The gradient descent algorithm is ultimately employed to minimize the error between the evaluation value and the target value, thereby facilitating parameter updates in both Critic1 and Actor network update process.

$$y=r+\gamma {\min }_{i=1,2}{Q_i}^{\prime }\left(s^{\prime },a^{\prime }|{\theta }_i^{Q^{\prime }}\right)$$

(19)

After updating the Critic1 and Critic2 networks for d steps, initiate an update of the Actor network) Employ the Actor network to compute action $a_{new}$ for state $s$. Where $a_{new}=\mu \left(s|{\theta }^{\mu }\right)$.

It is crucial to emphasize that there is no necessity to introduce noise after computing the action, as our objective is for the Actor network to update towards the direction of maximum value. Adding noise would be inconsequential in this context. Subsequently, we employ either Critic1 or Critic2 network to assess the state-action pair$\left(s,a_{new}\right)$, assuming utilization of Critic1 network $q_{new}=Q_1\left(s,a_{new}|{\theta }^{Q_1}\right)$. The $q_{new}$ optimization is ultimately achieved by employing a gradient ascent algorithm to facilitate updates in the Actor network.

The target network is updated using a soft update method, wherein a learning rate $\tau$ is introduced to calculate the weighted average between the old parameters of the target network and their corresponding new parameters. Subsequently, these averaged parameters are assigned to the target network (equation 20).

$$\begin{array}{c}{\theta }^{{Q_i}^{\prime }}=\tau {\theta }^{Q_i}+\left(1-\tau \right){\theta }^{{Q_i}^{\prime }},\left(i=1,2\right)\\ {\theta }^{{\mu }^{\prime }}=\tau {\theta }^{\mu }+\left(1-\tau \right){\theta }^{{\mu }^{\prime }}\end{array}$$

(20)

where $\tau \in \left(0,1\right)$, The value of t is usually set to 0.005. The reward at each time step is:

$$r_t=({\alpha }_1\ast i{d_{err}}^2+{\alpha }_2\ast i{q_{err}}^2+\beta \ast {\displaystyle \sum _j{{u^j}_{t-1}}^2})$$

(21)

Here, ${\alpha }_1,{\alpha }_2,\beta$ are the coefficients of each term，$i{d}_{err}$ is the d-axis current error, $i{q}_{err}$ is the q-axis current error, ${u^j}_{t-1}$ are the actions from the previous time step.

4. Results and Analysis

The parameters of the motor utilized in the simulation presented in this paper are detailed in Table 1.

In this article, simulation experiments and data analysis employ per unit (pu) values. Per unit values are a dimensionless indicator commonly utilized in power systems to represent the ratio between actual values and reference values. This enables relative comparisons among different systems, electrical quantities, or engineering parameters, facilitating quantitative analysis and research. The reference values used in this article can be found in Table 2.

To evaluate the effectiveness and robustness of the algorithm, simulation experiments were conducted under two representative PMSM operating conditions.

Working condition 1. PMSM load is constant, the starting moment is given 0.5pu, the reference speed trajectory is a sinusoidal signal, $Spee{d}_{ref}=0.3\ast \sin (t)+0.5$, no-load startup, the experimental results are shown in Figure 3.

From Figure 3(a), it is evident that the TD3 algorithm of this design achieves faster tracking of the upper reference trajectory speed with minimal overshoot compared to the other controllers. The traditional PI controller exhibits significant overshoot, whereas the LADRC controller requires a longer time to reach steady state. From Figure 3(b), it can be observed that the TD3-based controller stabilizes the q-axis current closer to the reference value more rapidly and with less fluctuation. In contrast, the PI controller shows substantial overshoot, and the LADRC controller experiences greater fluctuations. From Figure 3(c), it is clear that the TD3 algorithm, as a reinforcement learning approach, demonstrates the smallest speed tracking error and superior stability.

Working condition 2. The PMSM speed is suddenly increased, the load is constant, the given speed is 0.5pu at the starting moment, and it is suddenly changed to 0.8pu at 1 second, and the no-load is started, the experimental results are shown in Figure 4.

From Figure 4(a), it is evident that the controller using the LADRC algorithm exhibits the smallest overshoot but requires the longest time to reach steady state. In contrast, the RL algorithm achieves the fastest return to steady state with significantly less overshoot compared to the PI algorithm. From Figure 4(b), it can be observed that the torque fluctuation caused by a sudden change in rotational speed is also minimized. Furthermore, as shown in Figure 4(c), the proposed algorithm demonstrates the smallest overall rotational speed tracking error. Table 3 summarizes the performance comparison of the three algorithms over a 1-second period.

From Table 3, it is evident that the RL algorithm demonstrates superior adaptability to sudden speed changes. It can rapidly return to a steady state, with the overshoot remaining within an acceptable range.

5. Conclusions

The PMSM drive system is a typical dual time-scale system characterized by control challenges such as parameter variations, external disturbances, and nonlinearities. In this paper, we propose a TD3-based optimal control method aimed at minimizing the energy consumption of PMSM drive systems. We conduct simulation experiments under two typical operating conditions to validate the effectiveness and robustness of the proposed algorithm. The experimental results demonstrate that the TD3 algorithm surpasses both the traditional PI controller and the linear active disturbance rejection control (LADRC) algorithm in terms of reference trajectory tracking accuracy, q-axis current regulation, and speed tracking error minimization. This study offers a novel approach to PMSM drive system control, significantly enhancing motor efficiency and system robustness through the integration of deep reinforcement learning.