Deep Deterministic Policy Gradient (DDPG) Agent-Based Sliding Mode Control for Quadrotor Attitudes

1. Introduction

As a kind of unmanned flight platform, quadrotor UAV has the advantages of simple structure, lightweight fuselage and low cost, and is widely used in numerous tasks, such as aerial photography, agricultural plant protection, rescue and relief, remote sensing mapping and reconnaissance [1,2,3,4]. A wide range of application scenarios also put forward strict requirements for its flight control capability, especially the attitude control during the flight of UAV [4,5,6]. However, the lightweight fuselage of quadrotor leads to its poor ability to resist external disturbances, which reduces the accuracy of attitude control.

There have been many researches on attitude control methods of quadrotor. Some linear control methods such as proportional integral derivative (PID) control [7,8,9] and linear quadratic regulator [10] have been widely used in engineering practice, due to their advantages of simple structure and easy implementation. The PID and LQ methods were applied in the attitude angles control for a micro quadrotor, and the control laws were validated by autonomous flight experiments in the presence of external disturbances [11]. A robust PID control methodology was proposed for a quadrotor UAV regulation, which could reduce the power consumption and perform well in the disturbances of parameter uncertainties and aerodynamic interferences [12]. 12 PID coefficients of a quadrotor controller were optimized by 4 classical evolutionary algorithms, respectively, and the simulation results indicated that the coefficients optimized from the differential evolution algorithm (DE) could minimize the energy consumption, compared with other algorithms [7]. While linear or coefficient-optimized linear controllers may be suitable for some of the above scenarios, it is often found that the nonlinear effects of the quadrotor dynamics are non-negligible [13], and that the linear control methodologies are incapable due to the reliance on approximately linearized dynamical models. Various control approaches have been used in the quadrotors considering the nonlinear dynamics model. One of these approaches is nonlinear dynamic inversion (NDI), which can theoretically eliminate the nonlinearities of the control system [14], but this control method is on much dependence of the model accuracies [15]. The incremental nonlinear dynamic inversion (INDI) methodology was used to improve the robustness against the model inaccuracies, which could achieve the stable attitude control even the change in pitch angle was up to [16]. Adaptive control algorithm has also been widely used in quadrotor systems [17,18]. Two adaptive control laws were designed for the attitude stabilization of a quadrotor, to deal with the problem of parametric uncertainty and external disturbance [18]. A robust adaptive control strategy was developed for the attitude tracking of foldable quadrotors, which were modeled as switched systems [19].

Due to the advantages of fast response and strong robustness, the sliding mode control (SMC) methodology has been widely applied in the attitude tracking of quadrotors [20,21]. However, the problem of control input chattering is apparent, as the traditional reaching law designed in SMC. A fuzzy logic system was designed to schedule the control gains of the sign function adaptively, which could effectively suppress the control signal chattering [22].A novel discrete-time sliding mode control (DSMC) reaching law was proposed based on theoretical analysis, which could reduce the chattering significantly [23]. An adaptive fast nonsingular terminal sliding mode (AFNTSM) controller was introduced to achieve the attitude regulation and suppress chattering phenomenon, and the effectiveness of this controller was verified by experiments [24]. A fractional-order sliding mode surface was designed to adjust the parameters of SMC adaptively, in the fault tolerant control for a quadrotor model with mismatched disturbances [25].

The above researches have great reference significance. However, the control signal chattering still needs further improvement and attention, when the SMC method applied in attitude regulation with external disturbances. With the development of artificial intelligence technology, more and more reinforcement learning algorithms have been applied to traditional control methodologies [26,27].Inspired by these studies, a deep deterministic policy gradient (DDPG) [28] agent was introduced to the SMC in this paper. The parameters associated with the sign function can be regulated adaptively by the trained DDPG agent, and the phenomenon of control input chattering can be suppressed in the attitude control, in the presence of external disturbances.

The main contributions of our work are outlined as follows:

(1) A sliding mode controller is designed for attitude control for quadrotor UAV, in the presence of external disturbances.

(2) A reinforcement learning agent based on DDPG is trained to adjust the switching control gain adaptively in the traditional SMC method.

(3) The proposed DDPG-SMC approach can suppress the chattering phenomenon in the attitude control using traditional SMC method.

The remainder of this paper is organized as follows. Section 2 introduces the attitude dynamics modeling for quadrotor UAV. In section 3, the traditional SMC and proposed DDPG-SMC are designed for solving the attitude control problems. In Section 4, the robustness and effectiveness of the proposed control approach are validated through simulation results, followed by key conclusions in Section 5.

2. Attitude Dynamics Modeling for Quadrotor UAV

The quadrotor is considered as a rigid body, which attitude motion can be described by two coordinate frames, an inertial reference frame (frame I) $O_i{x}_i{y}_i{z}_i$ and a body reference frame (frame B) $O_b{x}_b{y}_b{z}_b$, as shown in Figure 1. The attitude motion of the quadrotor can be achieved by the rotation of each propeller. The attitude angles can be described as $\mathit{\eta }={[\varphi ,\theta ,\psi ]}^T$ in the frame B, where $\varphi ,\theta ,\psi$ are the roll angle (rotation around the x-axis), pitch angle (rotation around the y-axis) and yaw angle (rotation around the z-axis), respectively. The attitude angular velocities are expressed as $\zeta ={[p,q,r]}^T$, where $p,q,r$ are the angular velocities in the roll, pitch and yaw directions, respectively.

According to the relationship between the angular velocities and the attitude rate, the attitude kinematics equation of the quadrotor can be expressed as follows [29]:

$$\dot{\mathit{\eta }}=\mathbf{\Phi }\left(\mathit{\eta }\right)\mathit{\zeta }$$

(1)

where

$$\mathbf{\Phi }(\mathit{\eta })=\left[\begin{array}{ccc}1& \tan \theta \sin \varphi & \tan \theta \cos \varphi \\ 0& \cos \varphi & -\sin \varphi \\ 0& \sec \theta \sin \varphi & \sec \theta \cos \varphi \end{array}\right]$$

(2)

The attitude dynamics equation of the quadrotor can be written as follows [30]:

$$\mathit{J}\dot{\mathit{\zeta }}+\mathit{\zeta }\times \left(\mathit{J}\mathit{\zeta }\right)=\mathit{\tau }$$

(3)

where $\mathit{J}=\mathrm{diag}\left(J_x,\hspace{0.33em}{J}_y,\hspace{0.33em}{J}_z\right)$, $J_x$, $J_y$ and $J_z$ are the moments of inertia along $O_b{x}_b$, $O_b{y}_b$ and $O_b{z}_b$ axes, respectively; $\mathit{\tau }={\left[L,\hspace{0.17em}M,\hspace{0.17em}N\right]}^T$ denotes the control inputs, $L$, $M$ and $N$ are the control torques in the roll, pitch and yaw directions, respectively. The control inputs $\tau$ developed by the four propellers can be defined as follows [31,32]:

$$\mathit{\tau }=\left[\begin{array}{c}L\\ M\\ N\end{array}\right]=\left[\begin{array}{c}l\left(M_2-M_4\right)\\ l\left(M_3-M_1\right)\\ \frac{d}{b}\left(M_1-M_2+M_3-M_4\right)\end{array}\right]$$

(4)

where the parameter $b$ denotes the lift coefficient, the parameter $d$ represents the drag coefficient, the parameter $l$ is the length between the quadrotor’s center of mass and the rotation axis of any propeller, and $M_i\left(i=1,2,3,4\right)$ represents the thrust force provided by the ith propeller.

When the external disturbances are taken into account, the attitude dynamics equation (3) can be rewritten as

$$\mathit{J}\dot{\mathit{\zeta }}=-\mathit{\zeta }\times \left(\mathit{J}\mathit{\zeta }\right)+\mathit{\tau }+{\mathit{\tau }}_d,$$

(5)

where ${\mathit{\tau }}_d$ denotes the external disturbances.

3. Control Design for Attitude Control

In view of the attitude control in the presence of external disturbances, a sliding mode controller including its sliding mode surface and reaching law is selected for the quadrotor dynamic system, and the stability of the designed SMC system is supported by Lyapunov stability theorem. Then a reinforcement learning agent based on DDPG is trained and applied to the above SMC method, without affecting the stability of the system.

3.1. SMC Design

In this section, a sliding mode controller is designed for the attitude regulation of the quadrotor. The control objective can be described as: the actual attitude $\mathit{\eta }={\left[\varphi ,\theta ,\psi \right]}^T$ need to be regulated to the desired attitude ${\mathit{\eta }}_d={\left[{\varphi }_d,{\theta }_d,{\psi }_d\right]}^T$ asymptotically, i.e., $\underset{t\to \infty }{\lim }‖\mathit{\eta }-{\mathit{\eta }}_d‖=0$.

In the controller designed process, the sliding mode surface is first selected, then the control law is chosen to compute the control signal, in the end, the stability proof of the designed SMC system is supported by Lyapunov stability theorem. The control scheme of SMC for attitude tracking is depicted in Figure 2.

The specific design of the sliding mode controller can be expressed as follows.

SMC Algorithm

Input:  (1) Desired attitude angles ${\mathit{\eta }}_d$  (2) Actual attitude angles $\mathit{\eta }$  (3) Model parameters of the quadrotor Output: The control signals for attitude dynamics model Step 1: Design of the control signal  (a) Define the sliding mode surface s  (b) Select the reaching law $\dot{\mathit{s}}$  (c) Compute the control signal $\mathit{\tau }$ Step 2: Proof of the stability of closed-loop system  (a) Select a Lyapunov candidate function $V$  (b) Calculate the first-order derivative of $V$  (c) Analyze the sign of the above derivative of $V$  (d) Conclude the convergence of the attitude motion Step 3: Termination  If the attitude control errors meet the requirements, conduct the algorithm termination and output the control signal $\mathit{\tau }$. Otherwise, go to step 1 until the convergence of control errors.

Step 1 (a):

The control error can be defined as

$$\mathit{e}={\mathit{\eta }}_d-\mathit{\eta }$$

(6)

Then sliding mode surface can be derived as

$$\mathit{s}=\dot{\mathit{e}}+\mathit{ce},$$

(7)

where $\mathit{c}=\mathrm{diag}\left(c_1,c_2,c_3\right)$, and $c_1,\hspace{0.33em}{c}_2,\hspace{0.33em}{c}_3$ are selected positive numbers.

The derivative of s can be expressed as

$$\dot{\mathit{s}}=\ddot{\mathit{e}}+\mathit{c}\dot{\mathit{e}}$$

(8)

Substitute Equation (1)-(3), (6) into (8), then we can obtain

$$\dot{\mathit{s}}=\mathit{c}\dot{\mathit{e}} +\ddot{\mathit{e}}=\mathit{c}\dot{\mathit{e}}+{\ddot{\mathit{\eta }}}_d-\dot{\mathbf{\Phi }}(\mathit{\eta })\mathit{\zeta }-\mathbf{\Phi }(\mathit{\eta }){\mathit{J}}^{-1}\left(-\mathit{\zeta }\times (\mathit{J}\mathit{\zeta })+\mathit{\tau }+{\mathit{\tau }}_d\right),$$

(9)

where we can define

$$\mathit{d}={\mathit{\tau }}_d$$

(10)

Equation (9) can be rewritten as

$$\dot{\mathit{s}}=\mathit{c}\dot{\mathit{e}}+{\ddot{\mathit{\eta }}}_d-\dot{\mathbf{\Phi }}(\mathit{\eta })\mathit{\zeta }-\mathbf{\Phi }(\mathit{\eta }){\mathit{J}}^{-1}\left(-\mathit{\zeta }\times (\mathit{J}\mathit{\zeta })+\mathit{\tau }+\mathit{d}\right)$$

(11)

Assumption 1: The external disturbance $\mathit{d}$ is assumed to be bounded and satisfies

$$‖\mathit{d}‖\le D,$$

(12)

in which $D$ is a positive finite variable .

Step 1 (b):

The reaching law of the sliding mode surface can be selected as follows [33]:

$$\dot{\mathit{s}}=-\mathit{\lambda }\mathit{s}-\mathit{k}\mathrm{sign}(\mathit{s}),$$

(13)

in which $\mathit{\lambda }$ and $\mathit{k}$ are both diagonally positive definite matrices, with $\mathit{\lambda }=\mathrm{diag}({\lambda }_1,{\lambda }_2,{\lambda }_3)$, and ${\lambda }_1$, ${\lambda }_2$, ${\lambda }_3$ are selected as positive numbers, as same as $\mathit{k}=\mathrm{diag}(k_1,k_2,k_3)$, and $k_i$($i=1,2,3$) is also a selected positive number, and $\mathrm{sign}(\cdot )$ represents the sign function.

Step 1 (c):

Based on the calculations of angular velocity $\mathit{\zeta }$ and transformation matrix $\mathbf{\Phi }\left(\mathit{\eta }\right)$, as well as the derivation of Equation (9) and (13), the control signal for attitude dynamics model can be designed as follows

$$\mathit{\tau }=\mathit{\zeta }\times (\mathit{J}\mathit{\zeta })+\mathit{J}{\mathbf{\Phi }}^{-1}(\mathit{\eta })\left(\mathit{\lambda }\mathit{s}+\mathit{k}\mathrm{sign}(\mathit{s})+\mathit{c}\dot{\mathit{e}}+{\ddot{\mathit{\eta }}}_d-\dot{\mathbf{\Phi }}(\mathit{\eta })\mathit{\zeta }\right)$$

(14)

Step 2:

The stability of the closed-loop system is proven as follows.

Theorem 1. 

Considering the attitude dynamics system described as Equation (5), with the sliding mode surface selected as Equation (7), if the exponential reaching law is chosen as Equation (13), and the control signals for attitude dynamics model is designed as Equation (14), then the designed SMC system is stable and the actual attitude can converge to the desired attitude in finite time.

Proof of Theorem 1. 

We can select a Lyapunov candidate function as

$$V=\frac{1}{2}{\mathit{s}}^T\mathit{s}$$

(15)

Based on Equation (11), take the derivative of Equation (15) with respect to time, we can obtain that

$$\dot{V}={\mathit{s}}^T\dot{\mathit{s}}={\mathit{s}}^T\left\{\mathit{c}\dot{\mathit{e}}+{\ddot{\mathit{\eta }}}_d-\dot{\mathbf{\Phi }}(\mathit{\eta })\mathit{\zeta }-\mathbf{\Phi }(\mathit{\eta }){\mathit{J}}^{-1}\left(-\mathit{\zeta }\times (\mathit{J}\mathit{\zeta })+\mathit{\tau }+\mathit{d}\right)\right\}$$

(16)

Then substitute (14) into (16), we have

$$\begin{array}{l}\dot{V}={\mathit{s}}^T\left\{\begin{array}{l}\mathit{c}\dot{\mathit{e}}+{\ddot{\mathit{\eta }}}_d-\dot{\mathbf{\Phi }}(\mathit{\eta })\mathit{\zeta }-\mathbf{\Phi }(\mathit{\eta }){\mathit{J}}^{-1}\\ \left(\begin{array}{l}-\mathit{\zeta }\times (\mathit{J}\mathit{\zeta })+\mathit{\zeta }\times (J\mathit{\zeta })+J{\mathbf{\Phi }}^{-1}(\mathit{\eta })\\ \left(\mathit{\lambda }\mathit{s}+\mathit{k}\mathrm{sign}(\mathit{s})+\mathit{c}\dot{\mathit{e}}+{\ddot{\mathit{\eta }}}_d-\dot{\mathbf{\Phi }}(\mathit{\eta })\mathit{\zeta }+\mathit{d}\right)\end{array}\right)\end{array}\right\}\\ \hspace{0.33em}\hspace{0.33em}\hspace{0.33em}={\mathit{s}}^T\left(-\mathit{\lambda }\mathit{s}-\mathit{k}\mathrm{sign}(\mathit{s})-\mathit{d}\right)\\ \hspace{0.33em}\hspace{0.33em}\hspace{0.33em}=-\mathit{\lambda }{\mathit{s}}^T\mathit{s}-\mathit{k}‖\mathit{s}‖-{\mathit{s}}^T\mathit{d}\\ \hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\le -‖\mathit{\lambda }‖{‖\mathit{s}‖}^2-‖\mathit{k}‖‖\mathit{s}‖-D‖\mathit{s}‖\le 0\end{array}$$

(17)

We can assume that the sliding mode surface $\mathit{s}=0$ and obtain the following equation:

$$\underset{t\to \infty }{\lim }\mathit{s}=\underset{t\to \infty }{\lim }\left(\mathit{e}+\mathit{c}\dot{\mathit{e}}\right)=\underset{t\to \infty }{\lim }\left\{\left({\mathit{\eta }}_d-\mathit{\eta }\right)+\mathbf{c}\left({\dot{\mathit{\eta }}}_d-\dot{\mathit{\eta }}\right)\right\}$$

(18)

Based on the selection of diagonal positive definite matrix $c$, we can obtain the following expression:

$$\underset{t\to \infty }{\lim }‖{\mathit{\eta }}_d-\mathit{\eta }‖=0,\hspace{1em}\underset{t\to \infty }{\lim }‖{\dot{\mathit{\eta }}}_d-\dot{\mathit{\eta }}‖=0,$$

(19)

Remark 1. 

From (19), the designed control law (14) can guarantee the stability of the closed-loop system based on Lyapunov stability theorem. The attitude tracking error will converge to zero asymptotically, if the sliding mode surface is equal to zero. Consequently, the stability prove of the designed SMC system has been completed.

3.2. DDPG-SMC Design

The above derivations have proven that the control error can converge to zero asymptotically in the designed SMC for nonlinear system (5). However, high-frequency chattering of the control signal will appear near the sliding surface, due to the selected reaching law (13) with a sign function, and chattering intensity is determined by the parameter related to the sign function, i.e. control gain $\mathit{k}$.

Inspired by the combination of reinforcement learning algorithms and traditional control methodologies, a reinforcement learning agent based on DDPG is trained to adjust the switching control gain adaptively. During the training process, the input signals of agent are the actual and desired attitude angles, and the output action is the time-varying control gain. Then the trained agent is applied as a parameter regulator for the designed SMC, and the block diagram of designed DDPG-SMC is shown in Figure 3.

The architecture of the DDPG-based parameter regulator is shown in Figure 4.

DDPG is an algorithm to solve continuous action problem in the framework of Actor-Critic (AC) [34], where the policy network parameters are continuously optimized, so that its output action a can get higher and higher scores in the value network. In the designed DDPG-SMC approach this paper, the DDPG agent need to be trained beforehand. The described system in Figure 3 is served as training environment, and the training data are derived from multiple flight simulations.

The basic principle of DDPG algorithm can be introduced as follows.

DDPG Algorithm

Input: Experience replay buffer $D$, Initial critic networks’ Q-function parameters ${\theta }^Q$, actor networks’ policy parameters ${\theta }^{\pi }$, target networks $Q^{\prime }$ and ${\pi }^{\prime }$. Initialize the target network parameters: ${\theta }^{Q^{\prime }}\leftarrow {\theta }^Q,\hspace{0.33em}{\theta }^{{\pi }^{\prime }}\leftarrow {\theta }^{\pi }$. for episode =1 to M do  Initialize stochastic process $N$ to add exploration to the action.  Observe initial state $s_1$.  for time step =1 to T do   Select action $a_t=\pi \left(\left.s_t\right|{\theta }^{\pi }\right)+N_t$.   Perform action $a_t$ and transfer to next state $s_{t+1}$, then acquire the reward value $r_t$ and the termination signal $d_t$.   Store the state transition data $\left(s_t,\hspace{0.33em}{a}_t,\hspace{0.33em}{r}_t,\hspace{0.33em}{d}_t,\hspace{0.33em}{s}_{t+1}\right)$ in experience replay buffer $D$.   Calculate the target function:    $y\left(r_t,\hspace{0.33em}{s}_{t+1},\hspace{0.33em}{d}_t\right)=r_t+\gamma \left(1-d_t\right)Q^{\prime }\left(s_{t+1},\hspace{0.33em}{\pi }^{\prime }\left(s_{t+1}\left|{\theta }^{{\pi }^{\prime }}\right.\right)\left|{\theta }^{Q^{\prime }}\right.\right)$   Update the critic network by the minimized loss function:    $L=\frac{1}{B}{\displaystyle \sum _{\left(s_t,\hspace{0.33em}{a}_t,\hspace{0.33em}{r}_t,\hspace{0.33em}{d}_t,\hspace{0.33em}{s}_{t+1}\right)\in B}{\left(y\left(r_t,\hspace{0.33em}{s}_{t+1},\hspace{0.33em}{d}_t\right)-Q\left(s_t,\hspace{0.33em}{a}_t\left|{\theta }^Q\right.\right)\right)}^2}$   Update the actor network by policy gradient method:     ${\nabla }_{{\theta }^{\pi }}J\approx {\nabla }_{{\theta }^{\pi }}\frac{1}{\left|B\right|}{\displaystyle \sum _{s\in B}{Q}_{{\theta }^{\pi }}\left(s,\hspace{0.33em}{\mu }_{{\theta }^Q}(s)\right)}$   Update target networks:      ${\theta }^{Q^{\prime }}\leftarrow \rho {\theta }^{Q^{\prime }}+\left(1-\rho \right){\theta }^Q$      ${\theta }^{{\pi }^{\prime }}\leftarrow \rho {\theta }^{{\pi }^{\prime }}+\left(1-\rho \right){\theta }^{\pi }$  end for end for

The design of DDPG-based parameter regulator consists of two processes: training and validation. In the training process, the flight simulation of the quadrotor is carried out, and all the state and control data of the quadrotor are collected, that is, the accumulation of experience. Then, according to the accumulated experience data, the neural network parameters are optimized and updated with gradient calculation, stochastic gradient descent method and other methods. After multiple episodes of iterative training, the policy in the policy function converged to the optimal one. The validation process is used to validate the feasibility and generalization of the trained agent's optimal policy.

To train the DDPG agent to adjust the switching control gain $\mathit{k}$, the training episodes were set as 200, with the simulation time of each episode being 10 s and the time step being 0.02 s. Initial and desired attitude angles during the training were selected as ${\mathit{\eta }}_0={\left[0.1\mathrm{rad},0.2\mathrm{rad},-0.1\mathrm{rad}\right]}^T$ and ${\mathit{\eta }}_d={\left[0.2\mathrm{rad},0.1\mathrm{rad},0.1\mathrm{rad}\right]}^T$, respectively. The input signals of the agent were chosen as the actual and desired attitude angles ($\mathit{\eta }$,${\mathit{\eta }}_d$), and the output action was the time-varying control gain $\mathit{k}$.

The cumulative reward after each episode of training was recorded and output, and the reward at each step could be calculated using the following equation:

$$r\left(t\right)=-{\varpi }_{1}t\cdot \left|\mathit{\eta }\left(t\right)-{\mathit{\eta }}_d\left(t\right)\right|,$$

(20)

where ${\varpi }_1$ were selected as $\left[\begin{array}{ccc}1& 1& 1\end{array}\right]$.

The training process would be stopped when the average reward of cumulative training was less than -1 or the number of training episodes reached 200. The final training result is shown in Figure 5, it can be seen that the reward value converges to the maximum at the 170th training episodes. It indicates that the agent has completed training, and can be introduced as a parameter regulator in the above sliding mode controller.

In order to verify the generalization of the trained agent's optimal policy, it is necessary to test the control performance for the UAV model under different flight conditions. Specifically, it is necessary to evaluate the improvement of the control performance, through adjusting control parameters adaptively under different flight conditions. The relevant numerical simulation results are shown in Section 4.3.

Remark 2. 

By using the designed parameter regulator based on trained DDPG agent, the switching control gain related to reaching law, can be adjusted adaptively according to the attitude control error.

Therefore, the control signal in DDPG-SMC can be represented as

$$\mathit{\tau }=\mathit{\zeta }\times (J\mathit{\zeta })+J{\mathbf{\Phi }}^{-1}(\mathit{\eta })\left(\mathit{\lambda }\mathit{s}+\hat{\mathit{k}}\mathrm{sign}(\mathit{s})+\mathit{c}\dot{\mathit{e}}+{\ddot{\mathit{\eta }}}_d-\dot{\mathbf{\Phi }}(\mathit{\eta })\mathit{\zeta }\right),$$

(21)

in which $\hat{\mathit{k}}$ is the time-varying switching control gain related to reaching law.

4. Simulation Results

The robustness and effectiveness of the proposed control approach can be verified via flight simulation, and the model parameters of the quadrotor are depicted in Table 1.

The basic simulation conditions are described as follows. Initial attitude angles and angular velocities of the quadrotor are set as: ${\mathit{\eta }}_0={\left[0.1\mathrm{rad},0.2\mathrm{rad},-0.1\mathrm{rad}\right]}^T$ and ${\mathit{\zeta }}_0={\left[0\mathrm{rad}/s,0\mathrm{rad}/s,0\mathrm{rad}/s\right]}^T$, respectively. A desired attitude angles are selected as: ${\mathit{\eta }}_d={\left[0\mathrm{rad},0.1\mathrm{rad},0\mathrm{rad}\right]}^T$. The external disturbances are assumed to act on the system in the form of torques: ${\mathit{\tau }}_d=0.005\times {\left[\begin{array}{ccc}\sin \left(\pi /100t\right)\hspace{0.17em}& \cos \left(\pi /100t\right)& \sin \left(\pi /100t\right)\end{array}\right]}^T\mathrm{N}\cdot \mathrm{m}$. Three control approaches including SMC, AFGS-SMC proposed in reference [22] and DDPG-SMC designed this paper, are used in the flight simulation, respectively.

4.1. Simulation Results of SMC

The relevant control parameters of sign function in SMC are designed as: $\mathit{k}=\mathrm{diag}(0.2,0.2,0.2)$ and $\mathit{\lambda }=\mathrm{diag}(1.5,1.5,1.5)$. The numerical simulation results are depicted in Figure 6.

As shown in Figure 6 (a), the dashed and real lines represent the desired and actual attitude angles, respectively. The convergence time of attitude angles in three directions (roll, pitch and yaw) is 1.8 s, 2.1 s and 1.8 s, respectively, from the initial value to desired value. It indicates that the quadrotor attitude can be regulated into the desired attitude using the SMC algorithm, in the presence of disturbances.

The time histories of angular velocities and control inputs are depicted in Figure 6 (b) and (c), respectively. It can be seen that attitude angular velocities in three directions approach 0 rad/s during time periods of 2.2 s, 2.4 s and 2.2 s, respectively. Then the angular velocities oscillate slightly around 0 rad/s to keep the quadrotor system balanced.

However, the chattering of the control inputs is more severe when the control system is stabilized, and the control input signal in roll direction oscillates in the range from -0.012 N·m to 0.014 N·m, the control input signal in pitch direction oscillates in the range from -0.008 N·m to 0.016 N·m, and the control input signal in yaw direction oscillates in the range from -0.018 N·m to 0.024 N·m. Since the control input signals denote the torques produced by the quadrotor propellers, chattering at high frequencies is absolutely not allowed for the actuators of the quadrotor.

Figure 6 (d) and (e) represent the time evolutions of control gains related to reaching law, and sliding mode surfaces, respectively. It can be seen that sliding mode surfaces are converge to zero asymptotically, and the control gains are constant in entire simulation process, these constant gains result in the chattering phenomenon of control signals.

4.2. Simulation Results of AFGS-SMC

The chattering phenomenon is caused by the high frequency switching around the sliding mode surface, due to the term ksign(s) in the SMC. The adaptive fuzzy gain-scheduling sliding mode control (AFGS-SMC) method in the reference [22] is proposed to suppress the chattering phenomenon. Simulation results of this method are depicted in Figure 7.

As presented in Figure 7 (a), the dashed and real lines represent the desired and actual attitude angles, respectively. The convergence time of attitude angles in three directions (roll, pitch and yaw) is 2.2 s, 2.0 s and 2.2 s, respectively, from the initial value to desired value. It demonstrates that the quadrotor attitude can be regulated into the desired attitude using the AFGS-SMC algorithm, in the presence of disturbances.

The time evolutions of the quadrotor’s angular velocities and control inputs are depicted in Figure 7 (b) and (c), respectively. It can be seen that attitude angular velocities in three directions approach 0 rad/s during time periods of 3.1 s, 2.8 s and 3.1 s, respectively, and the oscillation is much less. Different from the results in SMC, the chattering phenomenon of the control input is significantly reduced.

Figure 7 (d) and (e) represent the time evolutions of control gains related to reaching law, and sliding mode surfaces, respectively. It can be seen that sliding mode surfaces are converge to zero asymptotically, and the control gains are adjusted by the associated fuzzy rules adaptively in AFGS-SMC, which can reduce the chattering phenomenon of control signals.

4.3. Simulation Results of DDPG-SMC

Similar to the above AFGS-SMC method, the control gains $\hat{\mathit{k}}=\mathrm{diag}(k_1,k_2,k_3)$ of DDPG-SMC are time-varying, and they can be scheduled adaptively through the DDPG-based parameter regulator. Simulation results of DDPG-SMC are depicted in Figure 8.

As depicted in Figure 8 (a), the dashed and real lines represent the desired and actual attitude angles, respectively. The convergence time of attitude angles in three directions (roll, pitch and yaw) is 2.0 s, 1.9 s and 2.0 s, respectively, from the initial value to desired value. It demonstrates that the quadrotor attitude can be regulated into the desired attitude using the designed DDPG-SMC algorithm, in the presence of disturbances.

The time evolutions of the quadrotor’s angular velocities and control inputs are presented in Figure 8 (b) and (c), respectively. It can be seen that attitude angular velocities in three directions approach 0 rad/s during time periods of 2.1 s, 2.4 s and 2.6 s, respectively, and the oscillation is much less. Compared with the results in AFGS-SMC, the proposed DDPG-SMC method can reduce the chattering phenomenon significantly, with less convergence time.

Figure 8 (d) and (e) represent the time evolutions of control gains related to reaching law, and sliding mode surfaces, respectively. It can be seen that sliding mode surfaces are converge to zero asymptotically, and the control gains related to reaching law are adjusted adaptively, via trained DDPG agent, which can reduce the chattering phenomenon of control signals.

Remark 3. 

1) The traditional SMC, referenced AFGS-SMC and designed DDPG-SMC methods all perform effectively and robustly in attitude control, with the presence of external continuous disturbances. 2) The disadvantage of traditional SMC is that high-frequency chattering phenomenon exists in the control input signals. 3) The control gains related to reaching law in DDPG-SMC, can be adjusted adaptively via the trained reinforcement learning agent, where the chattering phenomenon is reduces effectively.

5. Conclusions

In view of the chattering phenomenon in the traditional SMC for quadrotor attitudes, a novel reinforcement learning-based DDPG-SMC approach is proposed. The attitude dynamics model of the studied quadrotor is derived and the attitude control problem is described by formulas firstly. Then a traditional sliding mode controller is designed for the nonlinear dynamic system, and the stability of the closed-loop system is supported by Lyapunov stability theorem. A reinforcement learning agent based on DDPG is trained to adjust the switching control gain in the traditional SMC adaptively, and this trained agent is applied to SMC as a parameter regulator to design the DDPG-SMC approach. The simulation results indicate that the proposed DDPG-SMC approach has excellent robustness and effectiveness in attitude control for quadrotors. Compared with traditional SMC method, the proposed approach can suppress the chattering phenomenon effectively, in the presence of external disturbances. The research in this paper can provide a method reference for solving the chattering problem of SMC, when the control system is affected by external disturbances.