Intelligent Flight Tracking for Un-Crewed Aerial Vehicle with Aerodynamic Blade Element Momentum Modeling

1. Introduction

Unmanned aerial vehicles (UAVs), particularly quadrotors, have become an important research topic because they combine vertical take-off and landing, high maneuverability, and relatively low operating cost[1,2,3,4,5]. They had been widely used in variety of real-world applications which include environmental exploration, search and rescue missions and aerial mappings [2,6,7,8]. The development of a flight controller generally begins with the formulation of a linear or nonlinear mathematical model of the UAV [9]. This model provides the basis for analyzing the vehicle dynamics and designing a closed-loop feedback controller. A wide range of control methods has therefore been investigated, including proportional-integral-derivative (PID) control, sliding-mode control (SMC), $H_{\mathrm{\infty }}$ control, backstepping, linear-quadratic regulation (LQR), and adaptive or learning-based methods [10,11,12,13,14,15,16,17].

These techniques have been applied to attitude stabilization, position regulation, trajectory tracking, and obstacle-avoidance tasks [11,12,13,14,17,18]. Conventional PID controllers remain attractive because of their simple structure and low computational cost, whereas nonlinear and robust methods are often preferred when stronger guarantees are required in the presence of disturbances and parameter variations.

Accurate modeling of the aerodynamic forces and moments is also important for assessing controller performance. Blade-element momentum theory (BEMT) is one of the most commonly used approaches for estimating rotor and propeller aerodynamic loads [19,20]. Earlier quadrotor studies combined aerodynamic modeling with flight-dynamics analysis and experimental control validation[21,22] . More detailed formulations have also incorporated variations in rotor aerodynamics and motor operating conditions to improve thrust prediction and control performance [23]. These studies demonstrate that aerodynamic effects can become significant during aggressive maneuvers and should not be neglected when a high-fidelity model is required.

Because quadrotor dynamics are nonlinear and strongly coupled, stability analysis is an essential part of controller design. Lyapunov-based methods are frequently used to establish stability properties and to evaluate the behavior of nonlinear controllers. Several studies have developed Lyapunov-based sliding-mode and robust adaptive controllers for quadrotor trajectory tracking and attitude regulation[24,25,26,27]. Other work has investigated the relationship between Lyapunov stability conditions and PID or adaptive backstepping control laws [28,29,30,31]. Comparative studies have further examined the performance of PD, PID, and sliding-mode controllers for multirotor systems under different operating conditions [32]. Related research on vehicles carrying liquid payloads has shown that fluid motion can introduce additional disturbances and coupling effects that must be considered in the control design [9,33].

In parallel with these model-based approaches, reinforcement learning (RL) and adaptive dynamic programming have attracted increasing attention for nonlinear flight-control problems [34]. RL-based attitude controllers have been developed for fixed-wing UAVs and quadrotors, with reported improvements in adaptation and tracking performance under uncertain or changing conditions [35]. Deep-RL methods have also been applied to autonomous UAV tracking and landing, demonstrating their potential for handling complex maneuvering tasks [36]. Nevertheless, learning-based controllers may require substantial training data and careful safety constraints before deployment on physical platforms. Combining a physically derived nonlinear model with a stability-oriented controller and an RL-based parameter adaptation mechanism can therefore provide a useful compromise between interpretability, robustness, and adaptability.

In this work, the nonlinear dynamics of an X-configuration quadrotor are formulated using the Euler–Lagrange method. The rotor thrust and torque coefficients are estimated using BEMT. The main contribution is a Lyapunov-based altitude and yaw control strategy in which reinforcement learning is used to obtain locally optimized parameters $\left(kd, \lambda , ∆\right)$ for the sliding-mode controller. The resulting controller is designed to stabilize the six-degree-of-freedom quadrotor system while reducing altitude and yaw tracking errors, as illustrated in Figure 1.

2. Mathematical Modeling

This section contains an overview of the quadrotor kinematics and dynamics. The general drone control mechanism included in the suggested methodology is shown in Figure 1. RL optimized SMC controllers, together with blade aerodynamics, actuator kinematics, and nonlinear quadrotor dynamics, are employed for optimal feedback control of quadrotor altitude and position. Description of geometric parameters and coordinate reference frames is included here which is necessary for understanding the quadrotor dynamics. Specifications of DJI Tello Drone[37] are used for this study. The mass m is 80 gm whereas, diagonal length is 0.119 m.

Figure 2 covers two different reference frames required to represent quadrotor system. $\left\{I\right\}$ denotes Earth-fixed or Inertial coordinate system which is considered as primary frame of reference and $\left\{B\right\}$ denotes body frame which defines the location of points relative to the quadrotor body.

In order to make the equations simple, following assumptions are considered true:

• We assumed quadrotor as symmetric rigid body with constant mass distribution.
• The center of gravity (CG) of the drone and its geometric center coincide.
• Thrust vector is assumed to be perpendicular to the drone frame.
• Gravity direction is positive opposite to the thrust vector.
• Direction of motion in clockwise (CW) is considered positive, while counter clockwise (CCW) is considered negative.
• Propellers with odd indices indicate CW rotation, while propellers with even indices indicate CCW rotation as seen in Figure 2.

The location of a quadrotor in inertial frame of reference is given by vector $\mathsf{\Xi }= {\left[x y z\right]}^T \in {\mathbb{R}}^3$ whereas, $\mathsf{\Lambda } = {\left[\varphi \theta \psi \right]}^T \in {\mathbb{R}}^3$ represents orientation of vector Euler angles. Roll, Pitch and Yaw angles are denoted by $\varphi$, $\theta$ and $\psi$, respectively. State vector for such system $x \in {\mathbb{R}}^6$ is written as:

$$\mathrm{x}=\left[\begin{array}{c}\Xi \\ \Lambda \end{array}\right]$$

(1)

2.1. Kinematics

The linear velocity v and angular velocity ω can be defined in inertial frame as:

$$v=R_B^I{v}_B$$

(2)

$$\omega =W_B^I{\omega }_B$$

(3)

where $\dot{\mathsf{\Xi }}= v \in {\mathbb{R}}^3$ , $\dot{\mathsf{\Lambda }}=\mathsf{\omega }\in {\mathrm{R}}^3$ , $v_B\in R^3$ denotes the linear velocity vector of body frame and $\omega B\in R^3$ denotes a vector of the quadrotor body rates. $R_B^I$ is orthogonal rotational and $W_B^I$ is angular transformation matrix from body frame to inertial frame defined[38].

2.2. Dynamics

In essence, Lagrangian mechanics is an optimization process that depends on the difference between the system’s kinetic and potential energy. The Lagrangian $\mathcal{L}(\mathbf{x},\dot{\mathbf{x}})$ is defined as the difference between kinetic and potential energies:

$$\mathcal{L}(\mathbf{x},\dot{\mathbf{x}})={\displaystyle \frac{1}{2}}m{v}^{T}v+{\displaystyle \frac{1}{2}}{\omega }^{T}J\omega -mg\left[\begin{array}{c}0\\ 0\\ z\end{array}\right]$$

(4)

where term, $J=W_I^{B^T}{J}_i{W}_I^B$ is the Jacobian Matrix, $J_i$ is termed as inertia tensor, and $g$ denotes gravitational constant.

The moment of inertias of DJI Tello Drone are estimated experimentally using the bifilar pendulum method [39], since this method has been found more robust to external disturbances when compared to conventional pendulum methods.

$$J_{i_{\times \times }}={\displaystyle \frac{T_{avg}^2{l}_1{l}_{2}mg}{4{\pi }^2{L}_{avg}}}$$

(5)

In the above equation subscript $\times =1, 2, 3$ represents an element of the inertia tensor, $l_1$ and $l_2$ denotes the perpendicular distance between both strings and the axis of rotation, whereas, $L_{avg}$ is the average length of the strings, and $T_{avg}$ is average time period for $j$ oscillations in $k$ experiments. The Euler-Lagrange expression can be written as:

$$\begin{array}{c}{\displaystyle \frac{d}{dt}}\left({\displaystyle \frac{\partial \mathcal{L}(\mathbf{x},\dot{\mathbf{x}})}{\partial \dot{\mathbf{x}}}}\right)-{\displaystyle \frac{\partial \mathcal{L}(\mathbf{x},\dot{\mathbf{x}})}{\partial \mathbf{x}}}=\left[\begin{array}{c}{R}_B^{I}f\\ \tau \end{array}\right]\end{array}$$

(6)

where $f$ is the net thrust and $\tau$ is the torque acting on the body of the quadrotor. Eq. 6 can be simplified and solved into the generalized form of robot dynamics:

$$D(\mathbf{x})\ddot{\mathbf{x}}+C(\mathbf{x},\dot{\mathbf{x}})\dot{\mathbf{x}}+G(\mathbf{x})=\left[\begin{array}{c}{R}_B^{I}f\\ \tau \end{array}\right]$$

(7)

where $D\left(x\right) \in I{R}^6$ denotes the inertial matrix which is dependent on the geometric properties of the quadrotor. $C\left(\mathbf{x},\dot{\mathbf{x}}\right)\dot{\mathbf{x}}\in {\mathrm{I}\mathrm{R}}^6$ is known called the Coriolis matrix, which is composed of gyroscope and centripetal terms, and $G\left(\mathbf{x}\right) \in I{R}^6$ is gravitational vector, derived from potential energy equation. These terms can be expressed in array form as:

$$D(\mathbf{x})=\left[\begin{array}{cc}{\overline{D}}_{3\times 3}& 0_{3\times 3}\\ 0_{3\times 3}& J_{3\times 3}\end{array}\right];C(\mathbf{x},\dot{\mathbf{x}})=\left[\begin{array}{cc}{0}_{3\times 3}& 0_{3\times 3}\\ 0_{3\times 3}& {\overline{C}}_{3\times 3}\end{array}\right]$$

$$G(\mathbf{x})=\left[\begin{array}{c}0\\ 0\\ mg\\ 0\\ 0\\ 0\end{array}\right]$$

where $\overline{D}$ is a 3 $\times$ 3 diagonal mass matrix, and $\overline{C}={\displaystyle \frac{dJ}{dt}}\dot{\mathsf{\Lambda }}-{\displaystyle \frac{1}{2}}{\displaystyle \frac{\dot{{\partial (\mathsf{\Lambda }}^{\mathrm{T}}J\dot{\mathsf{\Lambda }})}}{\partial v}}$.

The right-hand side of Eq. 6 represents the control effectiveness matrix for the $\times$ configuration model, and can also expressed in its matrix form:

$$\left[\begin{array}{c}{f}_x\\ f_y\\ f_z\\ {\tau }_x\\ {\tau }_y\\ {\tau }_z\end{array}\right]=\left[\begin{array}{c}{f}_t\left(c_{\varphi }{s}_{\theta }{c}_{\psi }+s_{\varphi }{s}_{\psi }\right)\\ f_t\left(c_{\varphi }{s}_{\theta }{s}_{\psi }-s_{\varphi }{c}_{\psi }\right)\\ f_t\left(c_{\varphi }{c}_{\theta }\right)\\ -d_1{k}_f{s}_{{\vartheta }_1}{\overline{\omega }}_{m1}^2-d_2{k}_f{s}_{{\vartheta }_2}{\overline{\omega }}_{m2}^2-d_3{k}_f{s}_{{\vartheta }_3}{\overline{\omega }}_{m3}^2-\\ d_4{k}_f{s}_{{\vartheta }_4}{\overline{\omega }}_{m4}^2\\ d_1{k}_f{c}_{{\vartheta }_1}{\overline{\omega }}_{m1}^2+d_2{k}_f{c}_{{\vartheta }_2}{\overline{\omega }}_{m2}^2+d_3{k}_f{c}_{{\vartheta }_3}{\overline{\omega }}_{m3}^2+\\ d_4{k}_f{c}_{{\vartheta }_4}{\overline{\omega }}_{m4}^2\\ k_{\tau }{\delta }_1{\overline{\omega }}_{m1}^2+k_{\tau }{\delta }_2{\overline{\omega }}_{m2}^2+k_{\tau }{\delta }_3{\overline{\omega }}_{m3}^2+\\ k_{\tau }{\delta }_4{\overline{\omega }}_{m4}^2\end{array}\right]$$

(8)

${\vartheta }_n$ is defined as the angle between ${\mathit{X}}_B$ and the $nth$ arm in CW direction, $d_n$ is the distance from CG of quadrotor to center of the $nth$ motor, rotational speed of nth motor is denoted by ${\overline{\omega }}_{mn}$ while ${\delta }_n$ = ${\left(-1\right)}^n$, $n = 1, 2, 3, 4$. Total thrust generated is $f_t$, aerodynamic thrust coefficient is $k_f$ , aerodynamic torque coefficient is denoted by $k_{\tau }$ , while $\cos x$ is denoted by $c_x$ and $\sin x$ is denoted by $s_x$. Eq. 7 is rearranged and separated into linear $A \in R^3$ and rotational $\overline{\alpha } \in R^3$ acceleration components:

$$\left[\begin{array}{c}\overline{A}\\ \overline{\alpha }\end{array}\right]=\left[\begin{array}{c}{\overline{D}}^{-1}\left(G\left(\mathbf{x}\right)-R_B^I\mathbf{f}\right)\\ J^{-1}(\tau -\overline{C}(x,\dot{\mathbf{x}}\left)\dot{\mathsf{\Lambda }}\right)\end{array}\right]$$

(9)

2.3. Propeller Aerodynamics

Understanding the different aerodynamic impacts of the propellers and their interaction with the quadrotor’s rigid body motion is essential. Thrust $k_f$ and torque $k_{\tau }$ coefficients are considered critical parameters for BEMT in order to evaluate the performance of quadrotor propeller based on its mechanical and geometric parameters. [40]considered uniform inflow model for hover and edgewise flight. We assume the following to further simplify the equations:

• Thin air foil theory suggests, lift curve passes through the origin and lift slope is $2\pi$

• Axial velocity component is imperceptible when compared to tangential velocity.
• Viscosity effects are ignored.
• In-plane velocities are negligible.
• Inflow angle $\phi$ is small.

According to thin airfoil theory, the effective angle of attack ${\alpha }_{eff}$ and sectional lift coefficient $c_l$ is defined as[41]:

$${\alpha }_{\mathrm{eff} }=\beta +{\alpha }_0-\phi$$

(10)

$$c_l=c_{l_{\alpha }}{\alpha }_{\mathrm{eff} }$$

(11)

where $\beta$ is the sectional pitch angle, ${\alpha }_0$ is the sectional zerolift angle of attack, and the inflow angle $\phi =\lambda \left(r\right)/r$, where $r$ is the radial distance of the center of the hub to the blade element. Inflow coefficient $\lambda \left(r\right)$ can be calculated by solving the following equations iteratively [38]:

$${\lambda }_{i+1}=\mu t_{\alpha }+{\displaystyle \frac{C_T}{2\sqrt{{\mu }^2+{\lambda }_i^2}}}$$

(12)

$$d{C}_T={\displaystyle \frac{1}{2}}\sigma c_l{r}^{3}dr$$

(13)

$$d{C}_P={\displaystyle \frac{1}{2}}\sigma c_l{r}^3\lambda \left(r\right)dr$$

(14)

where $\mu$ is called the advance ratio, $\sigma$ is known as the solidity ratio, $C_T$ is the rotor thrust coefficient, and $C_P$ is the rotor power coefficient. Blade tip vortices are considered as dominant feature of rotor wakes. Empirical models are often used to cater for non-uniform inflow effects. [42]suggested that the subsequent inflow at the rotor disc may be characterized by a longitudinally varying, non-uniform first-harmonic model of the form:

$${\lambda }_i={\lambda }_0\left(1+k_{x}r{c}_{\mathsf{\Psi }}+k_{y}r{s}_{\mathsf{\Psi }}\right)$$

(15)

$k_x$ and $k_y$ are the factors that contribute to flow changes along the length and width of the wake. ${\lambda }_0$ is the induced flow ratio from the uniform momentum theory, and $\Psi$ is the blade azimuth angle[43]. Experimental models for $kx$ and $ky$ are derived from[44]. Thrust and torque coefficients can be estimated by using converged values of $C_T$ and $C_P$ can be estimated as:

$$k_f=C_T\rho \pi R^4$$

(16)

$$k_{\tau }=C_P\rho \pi R^5$$

(17)

where $\rho$ denotes the air density and $R$ is the radius of blade. Using linearization around the hover point in equation (9) leads to following equations for altitude and yaw of quadrotor.

$$\ddot{z}=-g+{\displaystyle \frac{cos\left(\varphi \right)cos\left(\theta \right)}{m}} U1$$

(18)

3. Stability And Control

This section introduces sliding mode technique-based control law for stabilizing the altitude and yaw of a quad-rotor aircraft to a desired reference during a flight mission. The control involves coordination of all four rotors to achieve the desired altitude and yaw control.

3.1. Sliding Mode Control

SMC technique is an approach of variable structure control (VSC). The objective of SMC deals with constraining the error vector to reach a desired state through two parts[45]. The first step is creating a control law that, during the reaching phase, directs the error vector in the direction of a sliding surface. This part is characterized by the control being switched on different sides of the sliding surface. The error vector is confined to the sliding surface and the system tracks the dynamics imposed by the sliding surface equations in the second part which is referred as equivalent control[46]. To stabilize the quadrotor, four control equations are used to maintain the system’s altitude and orientation close to the desired reference value, despite external disturbances. The signals $U1, U2, U3, and U4$ are utilized to regulate the altitude, roll, pitch, and yaw of the quadrotor, respectively.

$$s=\dot{e}+\lambda e$$

(19)

Equation (18) represents the sliding surface, where $e$ is the system error, $\dot{e}$ is the derivative of the error, and $\lambda$ is a positive constant. The value of $\lambda$ can be selected based on the desired performance criteria. The control input signals are designed such that the error vector approaches and remains on the sliding surface, ensuring the desired system behavior. The control law, $U\left(t\right),$ is a mathematical expression defined as:

$$U\left(t\right)=u_{eq}\left(t\right)+u_D\left(t\right)$$

(20)

where $u_{eq}\left(t\right)$ represents the continuous part of the control law and $u_D\left(t\right)$ represents the discontinuous part. The continuous part represents a smooth, continuous control input to the system while the discontinuous part represents a sudden change in the control input that leads to a discontinuity in the system’s behavior.

$$u_D\left(t\right)=k_D\Theta \left(s\right(t\left)\right)$$

(21)

Equation (20) can be rewritten as,

$$u_D\left(t\right)=k_{D}sign\left(s\right(t\left)\right)$$

(22)

where, $k_D$ is tuning parameter responsible for reaching phase and $\Theta \left(s\right(t\left)\right)$ is nonlinear function of $s\left(t\right)$. We can reduce the chattering problem if we rewrite equation (21) as,

$$u_D\left(t\right)=k_D{\displaystyle \frac{s\left(t\right)}{\left|s\right(t\left)\right|}}+\delta$$

(23)

$\delta$ is a tuning parameter used to reduce the chattering effect. The equivalent control procedure is utilized to determine $u_{eq}\left(t\right)$. The sliding surface for quadrotor error can be written by using (18) as,

$$s=( {\dot{z}}_d-\dot{z})+\lambda (z_d-z)$$

(24)

Using sliding condition $\dot{s}=0$

$$\dot{s}=\ddot{e}+\lambda \dot{e}=0$$

(25)

Replacing the values $\dot{e}$ and $\ddot{e}$ we get equation (26) below,

$$s=({\ddot{z}}_d-\ddot{z})+\lambda ({\dot{z}}_d-\dot{z})$$

(26)

Substituting values from equation (18), sliding condition is expressed as,

$$s=\left(\ddot{z_d}+g-{\displaystyle \frac{\cos \left(\varphi \right)\cos \left(\psi \right)}{m}} U_l\right)+\lambda ({\dot{z}}_d-\dot{z})$$

(27)

Using equation (20) and (21), setting $\dot{s}=0$, we get expression,

$$u_{eq\varphi }=\left[\lambda \left({\dot{\varphi }}_d-\dot{\varphi }\right)+{\ddot{\varphi }}_d-\dot{\theta }\dot{\psi }\left({\displaystyle \frac{I_y-I_z}{I_x}}\right)+{\displaystyle \frac{J_r}{l_x}}\dot{\theta }\cdot \mathsf{\Omega }\right]{\displaystyle \frac{I_x}{l}}$$

(28)

$$u_{eq\theta }=\left[\lambda \left({\dot{\theta }}_d-\dot{\theta }\right)+{\ddot{\theta }}_d-\dot{\varphi }\dot{\psi }\left({\displaystyle \frac{I_z-I_x}{I_y}}\right)-{\displaystyle \frac{J_r}{I_y}}\dot{\theta }\cdot \mathsf{\Omega }\right]{\displaystyle \frac{I_y}{l}}$$

(29)

$$u_{eq\psi }=\left[\lambda \left({\dot{\psi }}_d-\dot{\psi }\right)+{\ddot{\psi }}_d-\dot{\theta }\dot{\varphi }\left({\displaystyle \frac{I_x-I_y}{I_z}}\right)\right]I_Z$$

(30)

Algorithm 1 shows the steps involved in 6-DOF quadrotor control system.

3.2. Stability Analysis

For a given control system, the Lyapunov function is a scalar function $V$ that satisfies two conditions: positive-definiteness and monotonicity. Positive-definiteness means that the function is positive for all non-zero values of the state, while monotonicity means that the function decreases over time along the solutions of the system. The control of the altitude reference and attitude of a quadrotor is achieved through the use of the Lyapunov stability-based method, which directly focuses on attitude control of the quadrotor. The equilibrium point is defined as $z = 0$, and $E$ represents a compact region near $f\left(0\right)$ in $R^n$. The continuous Lyapunov function $V:E\to R^{+}$ is used to meet the requirements.

$$V\left(0\right)=0, h\ne 0$$

(31)

$$\dot{V}\left(h\right)\le 0$$

(32)

Consider $X=({\varphi }_d, {\varphi }_d, {\theta }_d,{\dot{\theta }}_d, {\dot{\psi }}_d, {\dot{\psi }}_d),$ where ${\varphi }_d, {\theta }_d,$ and ${\psi }_d$ are the desired attitude angles for the quadrotor. As the angular velocities will be zero at the stabilization point, their time derivatives will also be zero. The function $V\left(h\right)$ for yaw is described as a positive definite Lyapunov function at the desired position.

$$V\left(h\right)=\left[{\left(\psi -{\psi }_{des}\right)}^2+{\dot{\psi }}^2\right]$$

(33)

The equation for derivative is written as,

$$\dot{V}\left(h\right)=(\psi -{\psi }_{des})\dot{\psi }+\dot{\psi }\ddot{\psi }$$

(34)

The equation of motion at equilibrium point for yaw can be written as,

$$\dot{V}\left(h\right)=\left(\psi -{\psi }_{des}\right)\dot{\psi }+\dot{\psi }{\displaystyle \frac{l}{l_z}}U4$$

(35)

Control Input equation leads to,

$$U_4=I_z(\psi -{\psi }_{des})+f_3\dot{\psi }$$

(36)

Equation of motion for yaw can be rewritten as,

$$\dot{V}\left(h\right)=-\dot{{\psi }^2}{\displaystyle \frac{l}{l_Z}} f_3$$

(37)

The Lyapunov theorem provides simple stability for the equilibrium. Asymptotic stability is achieved through the application of LaSalle’s theorem, as the maximum invariance rotation subsystem that is controlled within the set $S=h\in R^6 :{\dot{\left.V\right|}}_{h=0}$ is confined only to the equilibrium point. The Lyapunov stability function and its derivative with respect to the angles and altitude are described as,

$$V\left(h\right)={\displaystyle \frac{1}{2}} [{\left(Z-Z_d\right)}^2+{\dot{z}}^2 ]$$

(38)

Taking the derivative of above equation, we get

$$\dot{V}\left(h\right)=(z-Z_d)\dot{z}+\dot{z}(g-(cos\theta cos\phi )U1/m)$$

(39)

Control input is given to system for stability,

$$U_1=-{\displaystyle \frac{m}{cos\theta cos\phi }}(Z_d-z-g)-f_z\dot{z}$$

(40)

Replacing (40) in (39) leads to,

$$\dot{V}\left(z\right)=-{\dot{z}}^2{\displaystyle \frac{f_z}{m}} \left(cos\theta cos\phi \right)$$

(41)

The constant $f_z$ described by equation (41) is a positive constant that is only negative semi-definite.

3.3. Optimal Gain Tuning using Reinforcement Learning

Deep deterministic policy gradient (DDPG) algorithm is chosen as the RL component of the controller due to its ability to obtain a deterministic policy and its sample efficiency compared to other actor-critic methods as shown in Figure 3. In the machine learning branch of reinforcement learning, an agent learns to interact with its surroundings in a way that maximizes a cumulative reward signal over time[47]. Learning a policy that maximizes the predicted cumulative reward is the goal. The agent’s behavior is described by a policy that associates states with actions. Value functions, which calculate the predicted cumulative reward of performing a certain action in a given state, are used by RL algorithms[48]. The Q-learning technique is a popular method that iteratively updates estimates of the anticipated reward for every state-action pair in order to learn the best action-value function[49]. Using deep neural networks as function approximators, deep reinforcement learning (DRL) is a well-liked method for RL that estimates value functions and policies in high dimensional, continuous state and action spaces[50].

Natural language processing, gaming, robotics, and many more fields have seen the effective use of DRL[51]. Without requiring human feature engineering, DRL can learn directly from high dimensional sensory inputs like 1-D audio signals or 2-D visuals, which is one of its main benefits.

This makes it ideal for applications where the input space can be rather complicated and extensive, such visual object recognition [52]. The target networks slowly monitor the learned networks in order to improve their convergence stability during the learning process by maximizing the agent’s cumulative reward:

$${\displaystyle \sum _{t=1}^{\mathrm{\infty }}}{\mathsf{\gamma }}_{\mathrm{d}}^{\mathrm{t}-1} r_t$$

(42)

where ${\gamma }_d\in [0, 1]$ is called the discount factor, and for the scope of this study, $r_t$ is intuitively defined as:

$$\begin{array}{c}{r}_1=\left\{\begin{array}{cc}100,& \mathrm{if} e_t^2\le 0.5\\ 0,& \mathrm{else}\end{array}\right.\\ r_2=\left\{\begin{array}{cc}-50,& \mathrm{if}{\left({\mathbf{s}}_t-{\mathbf{s}}_{des}\right)}^2>0.25{\mathbf{s}}_{des}^2\\ 0,& \mathrm{else}\end{array}\right.\\ r_3=-10e_t^2\\ r_4=-0.5u_{t-1}^2\\ r_5=-1\\ r_t={\displaystyle \sum _{n=1}^5}{r}_n \end{array}$$

(43)

$r_n$ rewards the agent when the squared error term is 0.5 or less. $r_2$ penalizes the agent if it exceeds 25% of the desired step value or goes in the negative direction during training (given positive step input). $r_3$ penalizes the agent exponentially with respect to error. $r_4$ and $r_5$ punishes the agent with increasing control effort and simulation time, respectively. Using this reward function, the optimal gain auto-tuning process for a 6-DOF quadrotor is outlined in Algorithm 2.

4. Simulation and Results

The MATLAB/Simulink platform serves as the foundation for the control strategy experiment in this section. Two different trajectories were employed to demonstrate the efficacy of results. This study’s goal is to assess the validity and efficacy of the proposed control strategy. The comparative experiments were conducted between two control techniques: manually tuned (MT)-SMC and RL-SMC (auto-tuned) for a dynamic model of a quadrotor. Auto-tuned controller refers to those optimal sliding mode gains which are obtained using RL.

Figure 4 shows the individual trajectory tracking comparison in X, Y, and Z directions between MT-SMC and RL-SMC of quadrotor. Here, quadrotor followed the trajectory 1 which involved rather simple maneuvers with step commands.

Error comparison for all directions between actual and desired trajectories (trajectory 1 case) can be seen in Figure 5. It can be noted that RL tuned SMC minimizes the error terms, since, the manually tuned SMC carries the inherent property of chattering.

Figure 6 shows the individual trajectory tracking comparison in X, Y, and Z directions for MT-SMC and RL-SMC of quadrotor. In this case, a relatively complex and longer trajectory (trajectory 2) has been tested with the both manual and auto-tuned controllers.

Error comparison between actual and desired trajectories for all directions can be seen in Figure 7. It can be inferred from the results that the tracking error for manually tuned SMC is relatively higher that RL tuned SMC with optimal gains.

Moreover, during this activity, it is observed that as nonlinearity increases, the number of episodes needed to attain convergence increases, with Z-Agent achieving convergence in 35 episodes, ψ-Agent in 79 episodes, ϕ-Agent in 145 episodes, and X-Agent in 338 episodes. A tabular comparison of the gains obtained from the two methods, for all directions in both test trajectories, is shown in Table 1.

Similarly, Table 2 describes the performance comparison of both the controllers in terms of RMSE and maximum error for all the employed test cases.

5. Discussion

This work implements a systematic auto-tuning framework for a 6-degree-of-freedom nonlinear quadrotor using a sliding mode controller, though the approach extends to other mechanical systems. Blade aerodynamics are modeled via blade element momentum theory, while the vehicle's dynamic equations are derived using the Euler-Lagrange formalism. A closed-loop cascaded architecture governs trajectory tracking, with SMC serving as the regulation law. Lyapunov's direct method confirms closed-loop stability, with adaptations incorporated to align the SMC design for robotic manipulators. In contrast to fixed-gain baselines, we embed a deep reinforcement learning agent specifically, DDPG, within the control loop to automatically search for and update the optimal controller gains online. This agent continuously adapts the gains using real-time state feedback, which enhances robustness under highly nonlinear and uncertain operating conditions. For experimental validation, we choose the DJI Tello drone as our physical platform, determining its mass, dimensions, and moments of inertia through bifilar suspension tests, while the thrust and torque coefficients are obtained numerically from BEMT computations. The identified parameters show close agreement with reported literature values, confirming consistency of our identification procedure. The DRL-optimized gains satisfy all prescribed stability bounds.