Adaptive Actor-Critic Optimal Tracking Control for a Class of High-Order Nonlinear Systems with Partially Unknown Dynamics

1. Introduction

In recent decades, optimal tracking control for nonlinear systems has remained one of the most significant research topics in control theory, with extensive applications in industrial manufacturing [1], aerospace [2], and robotics [3,4]. However, the majority of existing controller designs fail to incorporate formal optimization frameworks for performance cost functions, despite energy/time efficiency being critical in practical engineering. This study addresses this gap by developing an optimal tracking control law for partially unknown continuous-time nonlinear systems. Using an actor-critic reinforcement learning strategy derived from the HJB equation, the proposed method enables system states to achieve asymptotic tracking of desired reference trajectory with guaranteed convergence.

Nonlinear tracking control problems can be effectively addressed through nonlinear control techniques including ‌feedback linearization[5,6], backstepping approach [7,8,9,10], and‌ sliding mode control(SMC) [3,11,12,13], ‌among others. Feedback linearization, a mature methodology in nonlinear control theory, transforms nonlinear systems into partially or fully linear equivalents through coordinated nonlinear state feedback and coordinate transformations (including dynamic compensation). This enables the application of established linear control design methods. Unlike feedback linearization, the backstepping approach provides an alternative framework for nonlinear systems unbounded by linear constraints. This methodology synthesizes feedback control laws through ‌an iterative recursive procedure. Its principal advantage lies in ‌preserving beneficial nonlinearities‌ while guaranteeing ‌asymptotic stability‌ for regulation and tracking tasks. However, these methods are mainly applicable to strictly feedback nonlinear systems. To enhance transient response performance, SMC has garnered significant research attention. As a robust control strategy, SMC’s distinctive feature lies in its ‌adaptive control structure switching‌ based on state-dependent sliding surfaces. Its multiple variants have been developed, including terminal SMC [11], integral SMC [3,12], and hierarchical SMC [13]. Recent years have witnessed increasingly in-depth research on adaptive optimal control integrated with sliding mode surface (SMS). Zhang et al. addressed the problem of SMS-based adaptive optimal control for a class of switched continuous-time nonlinear systems with average dwell time by employing an actor-critic RL strategy [14]. Zhao et al. investigated the SMS-based approximate optimal control problem in the context of nonlinear multi-player Stackelberg-Nash games [15]. Furthermore, Zhang et al. studied the adaptive optimal control scheme based on a hierarchical SMS for a class of switched continuous-time nonlinear systems subject to unknown disturbances, utilizing an actor-critic NNs architecture [16]. However, all of these are aimed at the problem of optimal regulation.

Designing optimal control laws for nonlinear systems typically requires solving HJB equation. However, the inherent nonlinearity of HJB equation makes analytical solutions challenging to derive via conventional methods. To address this, RL algorithms have been integrated into the optimal control framework, enabling feasible solutions. As a machine learning paradigm, RL allows agents to learn optimal policies through environmental interactions [17,20]. This RL-based function approximation approach has successfully enabled adaptive optimal control, emerging as a prominent methodology for complex nonlinear control in recent decades.

Current research extensively explores optimal tracking for nonlinear systems using adaptive dynamic programming (ADP) [18,19]. This method combines dynamic programming (DP) and RL. Notably, Modares et al. proposed a novel integral RL formulation for continuous-time nonlinear systems [20]. ‌This approach adapts methodologies originally developed for optimal regulation problems to address optimal tracking control. Recent advancements in RL-based tracking control have demonstrated significant progress across diverse nonlinear systems. Wen et al. developed an optimized tracking control framework employing neural network-based actor-critic reinforcement learning for a class of nonlinear dynamic systems [21]. Wang et al. subsequently proposed a novel actor-critic RL scheme to achieve optimal tracking control for unmanned surface vehicles, explicitly addressing complex unknowns including dead-zone input nonlinearities, uncertain system dynamics, and external disturbances [22]. Further innovations include fuzzy integral RL-based fault-tolerant control algorithm, which integrates RL techniques with fuzzy-augmented models to handle partially unknown systems with actuator faults [23]. Specialized applications have been demonstrated, including robust online tracking control for space manipulators and nearly optimal trajectory tracking of autonomous surface vehicles [24,25]. While the backstepping technique has been widely applied in optimal tracking control of strict-feedback nonlinear systems [26,27,28,29], the same problem for high-order canonical nonlinear systems remains largely unexplored.

Motivated by the above discussions, this paper addresses a class of canonical form high-order nonlinear system by proposing a novel adaptive optimal tracking control scheme. The main contributions are in three aspects. First, an adaptive optimal tracking control scheme is proposed via a SMS for high-order canonical nonlinear systems. By constructing a cost function specifically related to the SMS, the original control problem is equivalently transformed into one that seeks an optimal control strategy. The designed SMS constrains the states of the error dynamic system, thereby forcing the tracking error to converge to zero with predefined dynamics. Second‌, in most of the existing literature, such as [19,20,21,22], the requirement of persistent excitation is necessary for training the RL optimal control with adaptive parameters. The proposed optimization method can avoid this requirement. Third, an adaptive optimal tracking law is developed using an‌ actor-critic NNs RL framework. Compared with the [30] method, our method achieves higher tracking accuracy and computational efficiency with fewer conditional constraint.

The rest of this paper is organized as follows: Section 2 describes the optimal tracking control problem of a class of canonical form high-order nonlinear system. In Section 3, we present a SMS-based adaptive optimal tracking control using actor-critic RL. Based on Lyapunov theory, it is proved that the error signals are semi-globally uniformly ultimately bounded (SGUUB), and the tracking error can be steered into a small neighborhood of zero in Section 4. Simulation are given to demonstrate the effectiveness of the proposed method in Section 5. Finally, the conclusion summarized in Section 6.

2. Problem Description

A canonical form nonlinear system with relative degree n is given by [31]

$$\begin{array}{cc}\hfill & {\dot{x}}_i\left(t\right)=x_{i+1}\left(t\right),i=1,\dots ,n-1\hfill \\ \hfill & {\dot{x}}_n\left(t\right)=f\left(\overline{x}\right)+gu,\hfill \end{array}$$

(1)

where $\overline{x}\left(t\right)={\left[x_1\left(t\right),x_2\left(t\right),\dots ,x_n\left(t\right)\right]}^T\in {\mathbb{R}}^n$ is the system state vector, $u\in \mathbb{R}$ is the control input, $f\left(\overline{x}\right)\in \mathbb{R}$ is the unknown and bounded nonlinear dynamic function, and $g\in \mathbb{R}$ is the known control gain constant.

Assumption A1. 

The system (1) is stabilizable, meaning there exists an admissible control policy $u\in \Psi \left(\mathsf{\Omega }\right)$ that ensures global stability of the closed-loop system.

Assumption A2. 

The reference trajectory $y_d\left(t\right)$ and its i-th order derivatives $y_d^{\left(i\right)}\left(t\right)(i=1,...,n)$ are assumed to be bounded.

Definition 1 

([32]). For a nonlinear system $\dot{\zeta }=f(\zeta ,t)$, the solution $\zeta \left(t\right)\in {\mathbb{R}}^n$ is called SGUUB if for any initial state $\zeta \left(t_0\right)\in \mathsf{\Omega }$, where Ω is a compact set, there exist constants $\sigma >0$ and $T(\sigma ,\zeta \left(t_0\right))>0$ such that $\parallel \zeta \left(t\right)\parallel ⩽\sigma$ for all $t\ge t_0+T(\sigma ,\zeta \left(t_0\right))$.

The control objective of this paper is to design an optimized tracking controller for partially unknown nonlinear dynamic system (1) using a NN-based actor-critic RL strategy. The proposed control law not only ensures all closed-loop error signals are SGUUB, but also guarantees that the output state $x_1\left(t\right)$ and its derivative signals rapidly track the reference trajectory $y_d\left(t\right)$ and its corresponding derivatives, respectively.

3. Optimal Tracking Control Design

3.1. Tracking HJB Equation

The tracking errors are defined as $z_1\left(t\right)=x_1\left(t\right)-y_d\left(t\right),z_2\left(t\right)=x_2\left(t\right)-{\dot{y}}_d\left(t\right),\dots ,z_n\left(t\right)=x_n\left(t\right)-y_d^{(n-1)}\left(t\right)$. Based on the dynamic model (1) and the above tracking error definitions, the error dynamical system is derived as follows:

$$\begin{array}{cc}\hfill & {\dot{z}}_i\left(t\right)=z_{i+1}\left(t\right),i=1,\dots ,n-1\hfill \\ \hfill & {\dot{z}}_n\left(t\right)=f(\overline{z},{\overline{y}}_d)+gu-y_d^{\left(n\right)}\left(t\right)\hfill \end{array}$$

(2)

where $\overline{z}={[z_1\left(t\right),z_2\left(t\right),\dots ,z_n\left(t\right)]}^T\in {\mathbb{R}}^n$, and ${\overline{y}}_d={[y_d\left(t\right),{\dot{y}}_d\left(t\right),\dots ,y_d^{(n-1)}\left(t\right)]}^T\in {\mathbb{R}}^n$.

Based on the tracking errors, the SMS [33] is constructed as

$$s\left(t\right)=K_0^T\overline{z}\left(t\right)=k_1{z}_1\left(t\right)+\dots +k_{n-1}{z}_{n-1}\left(t\right)+z_n\left(t\right),$$

(3)

where $s\left(t\right)\in \mathbb{R}$, $K_0={[k_1,\dots ,k_{n-1},1]}^T\in {\mathbb{R}}^n$, and $k_j>0(j=1,\dots ,n-1)$ are design parameters.

Taking the time derivative of $s\left(t\right)$ along the error dynamical system (2) yields

$$\begin{array}{cc}\hfill \dot{s}\left(t\right)& =K_1^T\overline{z}\left(t\right)+f(\overline{z},{\overline{y}}_d)+gu-y_d^{\left(n\right)}\left(t\right)=k_1{z}_2\left(t\right)+\dots +k_{n-1}{z}_n\left(t\right)\hfill \\ \hfill & +f(\overline{z},{\overline{y}}_d)+gu-y_d^{\left(n\right)}\left(t\right),\hfill \end{array}$$

(4)

where $K_1={[0,k_1,\dots ,k_{n-1}]}^T\in {\mathbb{R}}^n$.

The design parameters $k_j$ are chosen such that the polynomial ${\rho }^{n-1}+k_{n-1}{\rho }^{n-2}+\dots +k_2\rho +k_1$ is Hurwitz, i.e., all its roots lie in the open left-half plane. Note that when the tracking errors $z_1\left(t\right),\dots ,z_n\left(t\right)$ are constrained to the SMS $s\left(t\right)=0$, they will converge to a small neighborhood of zero.

Define an infinite-horizon cost function with respect to the SMS s as follow

$$V\left(s\right)={\int }_t^{\infty }r(s\left(\tau \right),u\left(s\right))d\tau ,$$

(5)

where $r(s,u)=s^2+u^2$ denotes the utility function.

Definition 2. 

For the error dynamical system (2), the control policy $u\left(s\right)$ is defined as admissible with respect to $V\left(s\right)$ on the compact set Ω, denoted by $u\left(s\right)\in \Psi \left(\mathsf{\Omega }\right)$, if $u\left(s\right)$ is continuous, $u\left(0\right)=0$, $u\left(s\right)$ can stabilize (2) on Ω and $V\left(s\right)$ is finite for $\forall s\in \mathsf{\Omega }$.

The optimal control objective for the error dynamical system (2) is to find an admissible control law that minimizes the cost function (5) over an infinite horizon, thus achieving the desired control task. Accordingly, the optimal cost function is defined as

$$\begin{array}{cc}\hfill V^{*}\left(s\right)& =\underset{u\in \Psi \left(\mathsf{\Omega }\right)}{min}\left({\int }_t^{\infty }r(s\left(\tau \right),u\left(s\right))d\tau \right)={\int }_t^{\infty }r(s\left(\tau \right),u^{*}\left(s\right))d\tau ,\hfill \end{array}$$

(6)

where $V^{*}\left(s\right)$ and $u^{*}\left(s\right)$ denote the optimal cost function and optimal control law, respectively, and $\Psi \left(\mathsf{\Omega }\right)$ represents the set of admissible controls over the domain $\mathsf{\Omega }$.

By calculating the time derivative of the cost function (6) along the sliding mode dynamic (4) and applying the optimality condition, the following HJB equation is derived:

$$\begin{array}{cc}\hfill H(s,u^{*},\nabla V^{*}\left(s\right))& =r(s,u^{*})+\nabla V^{*}\left(s\right)\dot{s}\hfill \\ & =s^2+u^{*2}+\nabla V^{*}\left(s\right)(K_1^T\overline{z}\left(t\right)+f(\overline{z},{\overline{y}}_d)+g{u}^{*}-y_d^{\left(n\right)}\left(t\right))\hfill \\ & =0\hfill \end{array}$$

(7)

where $\nabla V^{*}\left(s\right)=\partial V^{*}\left(s\right)/\partial s$ denotes the partial derivative of $V^{*}\left(s\right)$ with respect to s.

Assuming the solution of (7) exists and is unique, the optimal control $u^{*}$ can be obtained by applying the stationarity condition ${\displaystyle \frac{\partial H(s,u^{*},\nabla V^{*}\left(s\right)}{\partial u^{*}}}=0$ as

$$u^{*}=-{\displaystyle \frac{g}{2}}\nabla V^{*}\left(s\right).$$

(8)

Substituting (8) into (7) yields

$$\begin{array}{c}\hfill s^2-{\displaystyle \frac{g^2}{4}}{\left(\nabla V^{*}\left(s\right)\right)}^2+\nabla V^{*}\left(s\right)\left(K_1^T\overline{z}\left(t\right)+f(\overline{z},{\overline{y}}_d)-{y_d}^{\left(n\right)}\left(t\right)\right)=0.\end{array}$$

(9)

The partial derivative $\nabla V^{*}\left(s\right)$ can be obtained by solving the HJB equation (9). The optimal control protocol can then be derived by combining the result with (8). However, solving (9) analytically is difficult due to the inherent system nonlinearity. Furthermore, the lack of complete knowledge about the system dynamics $f(\overline{z},{\overline{y}}_d)$ further complicates the solution of (9). To overcome these challenges, this paper employs a reinforcement learning method with an actor-critic architecture.

3.2. NNs Approximation in Actor-Critic RL

To achieve the optimal tracking control, the term $\nabla V^{*}\left(s\right)$ in (9) is decomposed as

$$\nabla V^{*}\left(s\right)=2{\gamma }_{u}s\left(t\right)+{\displaystyle \frac{2}{g^2}}{K}_1^T\overline{z}\left(t\right)+V^0(s,\overline{z}),$$

(10)

where ${\gamma }_u>0$ is the designed parameter and $V^0(s,\overline{z})=-2{\gamma }_{u}s\left(t\right)-{\displaystyle \frac{2}{g^2}}{K}_1^T\overline{z}\left(t\right)+\nabla V^{*}\left(s\right)$ is the unknown function in this dual architecture.

Remark 1. 

In (10), the terms $2{\gamma }_{u}s\left(t\right)$ and ${\displaystyle \frac{2}{g^2}}{K}_1^T\overline{z}\left(t\right)$ are designed to ensure the boundedness and stability of the system signals.

According to the Weierstrass approximation theorem [34], the continuous function $V^0(s,\overline{z})$ can be approximated by weighted sum of basis functions. Given the relationship $s\left(t\right)=K^T\overline{z}\left(t\right)$ between s and $\overline{z}$, the approximation takes the form as

$$V^0(s,\overline{z})=W^{*T}\mathsf{\Phi }\left(\overline{z}\right)+\epsilon \left(\overline{z}\right),$$

(11)

where $W^{*}=[w_1,w_2,\dots ,w_N]\in {\mathbb{R}}^N$ is the ideal weight vector and bounded by a positive constant $\overline{W}$ such that $\parallel W^{*}\parallel \le \overline{W}$. Moreover, $\mathsf{\Phi }=[{\varphi }_1,{\varphi }_2,\dots ,{\varphi }_N]\in {\mathbb{R}}^N$ and $\epsilon \left(\overline{z}\right)\in \mathbb{R}$ are, respectively, activation function vector and the approximation error. With the number of neurons in the hidden layer $N\to \infty$, the NN approximation error $\epsilon \left(\overline{z}\right)\to 0$.

Putting (11) into (10) yields

$$\nabla V^{*}\left(s\right)=2{\gamma }_{u}s\left(t\right)+{\displaystyle \frac{2}{g^2}}{K}_1^T\overline{z}\left(t\right)+W^{*T}\mathsf{\Phi }\left(\overline{z}\right)+\epsilon \left(\overline{z}\right),$$

(12)

since the ideal weight vector $W^{*}$ is unknown, (12) is approximated by critic NN as

$$\nabla {\widehat{V}}^{*}\left(s\right)=2{\gamma }_{u}s\left(t\right)+{\displaystyle \frac{2}{g^2}}{K}_1^T\overline{z}\left(t\right)+{\widehat{W}}_c^T\mathsf{\Phi }\left(\overline{z}\right),$$

(13)

where ${\widehat{W}}_c$ denotes the current estimate of $W^{*}$. The weight estimation error is then defined as ${\tilde{W}}_c={\widehat{W}}_c-W^{*}$.

According to (8) and (13), the optimal control to the system is approximated by an additional actor NN as

$${\widehat{u}}^{*}=-g{\gamma }_{u}s\left(t\right)-{\displaystyle \frac{1}{g}}{K}_1^T\overline{z}\left(t\right)-{\displaystyle \frac{g}{2}}{\widehat{W}}_a^T\mathsf{\Phi }\left(\overline{z}\right),$$

(14)

where ${\widehat{W}}_a$ is the current estimate of $W^{*}$. The actor NN weight estimation error is defined as ${\tilde{W}}_a={\widehat{W}}_a-W^{*}$.

The critic and actor adaptive training laws are designed as

$$\begin{array}{c}\hfill {\dot{\widehat{W}}}_c\left(t\right)=-{\gamma }_c\left(\mathsf{\Phi }\left(\overline{z}\right){\mathsf{\Phi }}^T\left(\overline{z}\right)+\sigma I_m\right){\widehat{W}}_c\left(t\right),\end{array}$$

(15)

$${\dot{\widehat{W}}}_a\left(t\right)=-\left(\mathsf{\Phi }\left(\overline{z}\right){\mathsf{\Phi }}^T\left(\overline{z}\right)+\sigma I_m\right)\left[{\gamma }_a\left({\widehat{W}}_a\left(t\right)-{\widehat{W}}_c\left(t\right)\right)-{\gamma }_c{\widehat{W}}_c\left(t\right)\right],$$

(16)

where ${\gamma }_c>0$ and ${\gamma }_a>0$ are the learning rates for the critic and actor networks, respectively. Here, $\sigma >0$ is a regularization parameter, and $I_m$ denotes the $m\times m$ identity matrix.

Inserting both (13) and (14) into (7), the approximated HJB equation yields

$$\begin{array}{cc}\hfill \widehat{H}(s,{\widehat{u}}^{*},\nabla {\widehat{V}}^{*}\left(s\right))& =s^2+g^2{\left(-{\gamma }_{u}s\left(t\right)-{\displaystyle \frac{1}{g^2}}{K}_1^T\overline{z}\left(t\right)-{\displaystyle \frac{1}{2}}{\widehat{W}}_a^T\mathsf{\Phi }\left(\overline{z}\right)\right)}^2\hfill \\ & +\left(2{\gamma }_{u}s\left(t\right)+{\displaystyle \frac{2}{g^2}}{K}_1^T\overline{z}\left(t\right)+{\widehat{W}}_c^T\left(t\right)\mathsf{\Phi }\left(\overline{z}\right)\right)\hfill \\ & \times \left(-{\gamma }_u{g}^{2}s\left(t\right)-{\displaystyle \frac{g^2}{2}}{\widehat{W}}_a^T\mathsf{\Phi }\left(\overline{z}\right)+f(\overline{z},{\overline{y}}_d)-y_d^{\left(n\right)}\left(t\right)\right).\hfill \end{array}$$

(17)

The Bellman residual error E is introduced, with its expression as

$$\begin{array}{cc}\hfill E& =\widehat{H}(s,{\widehat{u}}^{*},\nabla {\widehat{V}}^{*}\left(s\right))-H(s,u^{*},\nabla V^{*}\left(s\right))\hfill \\ \hfill & =\widehat{H}(s,{\widehat{u}}^{*},\nabla {\widehat{V}}^{*}\left(s\right))\hfill \end{array}$$

(18)

According to the previous analysis and the optimal control theory, the approximate optimized control ${\widehat{u}}^{*}$ should ensure $E\to 0$. If the condition $E=0$ is met and has a unique solution, it is equivalent to

$$\begin{array}{c}\hfill {\displaystyle \frac{\partial \widehat{H}(s,{\widehat{u}}^{*},\nabla {\widehat{V}}^{*}\left(s\right))}{\partial {\widehat{W}}_a}}={\displaystyle \frac{g^2}{2}}\mathsf{\Phi }\left(\overline{z}\right){\mathsf{\Phi }}^T\left(\overline{z}\right)\left({\widehat{W}}_a\left(t\right)-{\widehat{W}}_c\left(t\right)\right)=0.\end{array}$$

(19)

To make (19) hold, construct the following function $\mathsf{\Gamma }\left(t\right)$ such that

$$\begin{array}{c}\hfill \mathsf{\Gamma }\left(t\right)={\left({\widehat{W}}_a\left(t\right)-{\widehat{W}}_c\left(t\right)\right)}^T\left({\widehat{W}}_a\left(t\right)-{\widehat{W}}_c\left(t\right)\right).\end{array}$$

(20)

Obviously, $\mathsf{\Gamma }\left(t\right)=0$ ensures that (19) holds. When the weight vectors of both networks are trained to satisfy ${\widehat{W}}_a\left(t\right)={\widehat{W}}_c\left(t\right)$, equations (13) and (14) fulfill the relation (8), and the control action approaches the optimal solution.

Theorem 1. 

Under the adaptive laws given by (15) and (16), the function $\mathsf{\Gamma }\left(t\right)$ converges to zero asymptotically.

Proof. 

The partial derivatives of $\mathsf{\Gamma }\left(t\right)$ with respect to the weight estimates are

$${\displaystyle \frac{\partial \mathsf{\Gamma }\left(t\right)}{\partial {\widehat{W}}_a\left(t\right)}}=-{\displaystyle \frac{\partial \mathsf{\Gamma }\left(t\right)}{\partial {\widehat{W}}_c\left(t\right)}}=2\left({\widehat{W}}_a\left(t\right)-{\widehat{W}}_c\left(t\right)\right).$$

(21)

The time derivative of $\mathsf{\Gamma }\left(t\right)$ along the trajectories of the system is

$$\begin{array}{cc}\hfill \dot{\mathsf{\Gamma }}\left(t\right)& ={\displaystyle \frac{\partial \mathsf{\Gamma }\left(t\right)}{\partial {\widehat{W}}_a\left(t\right)}}{\dot{\widehat{W}}}_a\left(t\right)+{\displaystyle \frac{\partial \mathsf{\Gamma }\left(t\right)}{\partial {\widehat{W}}_c\left(t\right)}}{\dot{\widehat{W}}}_c\left(t\right)={\displaystyle \frac{\partial \mathsf{\Gamma }\left(t\right)}{\partial {\widehat{W}}_a\left(t\right)}}\left({\dot{\widehat{W}}}_a\left(t\right)-{\dot{\widehat{W}}}_c\left(t\right)\right).\hfill \end{array}$$

(22)

Substituting the adaptive laws (15) and (16) into (22) yields

$$\begin{array}{cc}\hfill \dot{\mathsf{\Gamma }}\left(t\right)& =2{\left({\widehat{W}}_a-{\widehat{W}}_c\right)}^T\left[-{\gamma }_a\left(\mathsf{\Phi }\left(\overline{z}\right){\mathsf{\Phi }}^T\left(\overline{z}\right)+\sigma I_m\right)\left({\widehat{W}}_a-{\widehat{W}}_c\right)\right]\hfill \\ \hfill & =-2{\gamma }_a{\left({\widehat{W}}_a-{\widehat{W}}_c\right)}^T\left(\mathsf{\Phi }\left(\overline{z}\right){\mathsf{\Phi }}^T\left(\overline{z}\right)+\sigma I_m\right)\left({\widehat{W}}_a-{\widehat{W}}_c\right).\hfill \end{array}$$

(23)

Noting that $\mathsf{\Gamma }\left(t\right)=\parallel {\widehat{W}}_a-{\widehat{W}}_c{\parallel }^2$, the expression in (23) can be rewritten as a function of $\mathsf{\Gamma }\left(t\right)$ itself. Since the matrix $\left(\mathsf{\Phi }\left(\overline{z}\right){\mathsf{\Phi }}^T\left(\overline{z}\right)+\sigma I_m\right)$ is positive definite, it follows that

$$\begin{array}{cc}\hfill \dot{\mathsf{\Gamma }}\left(t\right)& =-2{\gamma }_a{\left({\widehat{W}}_a-{\widehat{W}}_c\right)}^T\left(\mathsf{\Phi }\left(\overline{z}\right){\mathsf{\Phi }}^T\left(\overline{z}\right)+\sigma I_m\right)\left({\widehat{W}}_a-{\widehat{W}}_c\right)\hfill \\ \hfill & \le -2{\gamma }_a{\lambda }_{min}{\parallel {\widehat{W}}_a-{\widehat{W}}_c\parallel }^2=-2{\gamma }_a{\lambda }_{min}\mathsf{\Gamma }\left(t\right),\hfill \end{array}$$

(24)

where ${\lambda }_{min}>0$ is the minimum eigenvalue of the positive definite matrix $\left(\mathsf{\Phi }\left(\overline{z}\right){\mathsf{\Phi }}^T\left(\overline{z}\right)+\sigma I_m\right)$.

This inequality, $\dot{\mathsf{\Gamma }}\left(t\right)\le -2{\gamma }_a{\lambda }_{min}\mathsf{\Gamma }\left(t\right)$, implies that $\mathsf{\Gamma }\left(t\right)$ converges to zero exponentially. □

Remark 2. 

The critic NN in (14) may be used to determine the actor without using another NN for the actor. However, to obtain the form of (19) and to facilitate the derivation of weight update laws and subsequent stability analysis, separate NNs are necessary for the actor and critic.

Remark 3. 

As demonstrated in this subsection, the proposed RL approach not only features a simple structure but also eliminates the need for explicit knowledge of the system dynamic $f(\overline{z},{\overline{y}}_d)$ when solving the optimal control problem.

4. Main Results

Lemma 1 

([35]). $X\left(t\right)\in \mathbb{R}$ is a positive continuous function and has the bounded initial value $X\left(0\right)$. If the inequality $\dot{X}\left(t\right)⩽-\alpha X\left(t\right)+\beta$ holds, where $\alpha >0$ and $\beta >0$ are constants, then

$$X\left(t\right)⩽e^{-\alpha t}X\left(0\right)+{\displaystyle \frac{\beta }{\alpha }}(1-e^{-\alpha t}).$$

(25)

Theorem 2. 

Consider the partially unknown nonlinear system (1) under Assumptions 1 and 2, with a bounded initial state. The optimal control law is derived from the actor-critic RL framework, where the critic and actor are updated according to (13) and (14), using the learning rules given in (15) and (16), respectively. If the parameters satisfy ${\gamma }_u>{\displaystyle \frac{1}{2}}+{\displaystyle \frac{1}{g^2}}$, ${\gamma }_c>{\displaystyle \frac{{\gamma }_a}{2}}$ and ${\gamma }_a>1$, then all estimation errors are SGUUB. Moreover, the tracking errors converge to an arbitrarily small neighborhood of zero.

Proof. 

Choose the Lyapunov candidate function as

$$L\left(t\right)={\displaystyle \frac{1}{2}}{s}^2+{\displaystyle \frac{1}{2}}{\tilde{W}}_c^T\left(t\right){\tilde{W}}_c\left(t\right)+{\displaystyle \frac{1}{2}}{\tilde{W}}_a^T\left(t\right){\tilde{W}}_a\left(t\right).$$

(26)

Calculating the time derivative (26) along (4), (15) and (16) yields

$$\begin{array}{cc}\hfill \dot{L}\left(t\right)& =s\left(-g^2{\gamma }_{u}s-{\displaystyle \frac{g^2}{2}}{\widehat{W}}_a^T\mathsf{\Phi }\left(\overline{z}\right)+f(\overline{z},{\overline{y}}_d)-y_d^{\left(n\right)}\left(t\right)\right)-{\gamma }_c{\tilde{W}}_c^T\left(t\right)(\mathsf{\Phi }\left(\overline{z}\right){\mathsf{\Phi }}^T\left(\overline{z}\right)\hfill \\ & +\sigma I_m){\widehat{W}}_c\left(t\right)-{\gamma }_a{\tilde{W}}_a^T\left(t\right)\left(\mathsf{\Phi }\left(\overline{z}\right){\mathsf{\Phi }}^T\left(\overline{z}\right)+\sigma I_m\right)\left({\widehat{W}}_a\left(t\right)-{\widehat{W}}_c\left(t\right)\right)\hfill \\ & +{\gamma }_c{\tilde{W}}_a^T\left(t\right)\left(\mathsf{\Phi }\left(\overline{z}\right){\mathsf{\Phi }}^T\left(\overline{z}\right)+\sigma I_m\right){\widehat{W}}_c\left(t\right).\hfill \end{array}$$

(27)

It follows from ${\tilde{W}}_c={\widehat{W}}_c-W^{*}$ and ${\tilde{W}}_a={\widehat{W}}_a-W^{*}$ that

$${\widehat{W}}_a^T\mathsf{\Phi }\left(\overline{z}\right)={\tilde{W}}_a^T\mathsf{\Phi }\left(\overline{z}\right)+{W^{*}}^T\mathsf{\Phi }\left(\overline{z}\right)$$

(28)

and

$${\widehat{W}}_a\left(t\right)-{\widehat{W}}_c\left(t\right)={\widehat{W}}_a\left(t\right)-W^{*}+W^{*}-{\widehat{W}}_c\left(t\right)={\tilde{W}}_a-{\tilde{W}}_c.$$

(29)

Using (28) and (29), (27) can be rewritten as

$$\begin{array}{cc}\hfill \dot{L}\left(t\right)& =-g^2{\gamma }_u{s}^2-{\displaystyle \frac{g^2}{2}}s{\tilde{W}}_a^T\mathsf{\Phi }\left(\overline{z}\right)-{\displaystyle \frac{g^2}{2}}s{W^{*}}^T\mathsf{\Phi }\left(\overline{z}\right)+sf(\overline{z},{\overline{y}}_d)-s{y}_d^{\left(n\right)}\left(t\right)\hfill \\ & -{\gamma }_c{\tilde{W}}_c^T\left(t\right)\left(\mathsf{\Phi }\left(\overline{z}\right){\mathsf{\Phi }}^T\left(\overline{z}\right)+\sigma I_m\right){\widehat{W}}_c\left(t\right)-{\gamma }_a{\tilde{W}}_a^T\left(t\right)\left(\mathsf{\Phi }\left(\overline{z}\right){\mathsf{\Phi }}^T\left(\overline{z}\right)+\sigma I_m\right)\hfill \\ & \times \left({\tilde{W}}_a\left(t\right)-{\tilde{W}}_c\left(t\right)\right)+{\gamma }_c{\tilde{W}}_a^T\left(t\right)\left(\mathsf{\Phi }\left(\overline{z}\right){\mathsf{\Phi }}^T\left(\overline{z}\right)+\sigma I_m\right){\widehat{W}}_c\left(t\right).\hfill \end{array}$$

(30)

From Cauchy-Schwartz and Young’s inequalities, we obtain

$$\begin{array}{cc}\hfill & sf(\overline{z},{\overline{y}}_d)\le {\displaystyle \frac{1}{2}}{s}^2+{\displaystyle \frac{1}{2}}{f}^2(\overline{z},{\overline{y}}_d),\hfill \\ & -s{y}_d^{\left(n\right)}\left(t\right)\le {\displaystyle \frac{1}{2}}{s}^2+{\displaystyle \frac{1}{2}}{\left(y_d^{\left(n\right)}\left(t\right)\right)}^2,\hfill \\ & -s{\tilde{W}}_a^T\mathsf{\Phi }\left(\overline{z}\right)\le {\displaystyle \frac{1}{2}}{s}^2+{\displaystyle \frac{1}{2}}{\tilde{W}}_a^T\left(t\right)\left(\mathsf{\Phi }\left(\overline{z}\right){\mathsf{\Phi }}^T\left(\overline{z}\right)+\sigma I_m\right){\tilde{W}}_a\left(t\right),\hfill \\ & -s{W^{*}}^T\mathsf{\Phi }\left(\overline{z}\right)\le {\displaystyle \frac{1}{2}}{s}^2+{\displaystyle \frac{1}{2}}{W^{*}}^T\left(\mathsf{\Phi }\left(\overline{z}\right){\mathsf{\Phi }}^T\left(\overline{z}\right)+\sigma I_m\right)W^{*}.\hfill \end{array}$$

(31)

Under these inequalities, (30) reduces to

$$\begin{array}{cc}\hfill \dot{L}\left(t\right)& \le -\left[g^2\left({\gamma }_u-{\displaystyle \frac{1}{2}}\right)-1\right]s^2+{\displaystyle \frac{1}{2}}{f}^2(\overline{z},{\overline{y}}_d)+{\displaystyle \frac{1}{2}}{\left(y_d^{\left(n\right)}\left(t\right)\right)}^2-{\gamma }_c{\tilde{W}}_c^T\left(t\right)\left(\mathsf{\Phi }\left(\overline{z}\right){\mathsf{\Phi }}^T\left(\overline{z}\right)+\sigma I_m\right)\hfill \\ & \times {\widehat{W}}_c\left(t\right)-\left({\gamma }_a-{\displaystyle \frac{g^2}{4}}\right){\tilde{W}}_a^T\left(t\right)\left(\mathsf{\Phi }\left(\overline{z}\right){\mathsf{\Phi }}^T\left(\overline{z}\right)+\sigma I_m\right){\tilde{W}}_a\left(t\right)+{\gamma }_a{\tilde{W}}_a^T\left(t\right)(\mathsf{\Phi }\left(\overline{z}\right){\mathsf{\Phi }}^T\left(\overline{z}\right)\hfill \\ & +\sigma I_m){\tilde{W}}_c\left(t\right)+{\gamma }_c{\tilde{W}}_a^T\left(t\right)\left(\mathsf{\Phi }\left(\overline{z}\right){\mathsf{\Phi }}^T\left(\overline{z}\right)+\sigma I_m\right){\widehat{W}}_c\left(t\right)+{\displaystyle \frac{g^2}{4}}{W^{*}}^T\left(\mathsf{\Phi }\left(\overline{z}\right){\mathsf{\Phi }}^T\left(\overline{z}\right)+\sigma I_m\right)W^{*}.\hfill \end{array}$$

(32)

From the definition ${\tilde{W}}_c={\widehat{W}}_c-W^{*}$, the following equation can be derived

$$\begin{array}{cc}\hfill & {\tilde{W}}_c^T\left(t\right)\left(\mathsf{\Phi }\left(\overline{z}\right){\mathsf{\Phi }}^T\left(\overline{z}\right)+\sigma I_m\right){\widehat{W}}_c\left(t\right)\hfill \\ & ={\tilde{W}}_c^T\left(t\right)\left(\mathsf{\Phi }\left(\overline{z}\right){\mathsf{\Phi }}^T\left(\overline{z}\right)+\sigma I_m\right){\tilde{W}}_c\left(t\right)+{\widehat{W}}_c^T\left(t\right)(\mathsf{\Phi }\left(\overline{z}\right)\hfill \\ & \times {\mathsf{\Phi }}^T\left(\overline{z}\right)+\sigma I_m){\widehat{W}}_c\left(t\right)-W^{*T}\left(t\right)\left(\mathsf{\Phi }\left(\overline{z}\right){\mathsf{\Phi }}^T\left(\overline{z}\right)+\sigma I_m\right)W^{*}.\hfill \end{array}$$

(33)

Moreover, the following inequality can be derived according to Young’s inequality,

$$\begin{array}{cc}\hfill {\tilde{W}}_a^T\left(t\right)\left(\mathsf{\Phi }\left(\overline{z}\right){\mathsf{\Phi }}^T\left(\overline{z}\right)+\sigma I_m\right){\tilde{W}}_c& \le {\displaystyle \frac{1}{2}}{\tilde{W}}_a^T\left(t\right)\left(\mathsf{\Phi }\left(\overline{z}\right){\mathsf{\Phi }}^T\left(\overline{z}\right)+\sigma I_m\right){\tilde{W}}_a\left(t\right)\hfill \\ & +{\displaystyle \frac{1}{2}}{\tilde{W}}_c^T\left(t\right)\left(\mathsf{\Phi }\left(\overline{z}\right){\mathsf{\Phi }}^T\left(\overline{z}\right)+\sigma I_m\right){\tilde{W}}_c\left(t\right),\hfill \\ \hfill {\tilde{W}}_a^T\left(t\right)\left(\mathsf{\Phi }\left(\overline{z}\right){\mathsf{\Phi }}^T\left(\overline{z}\right)+\sigma I_m\right){\widehat{W}}_c& \le {\displaystyle \frac{1}{2}}{\tilde{W}}_a^T\left(t\right)\left(\mathsf{\Phi }\left(\overline{z}\right){\mathsf{\Phi }}^T\left(\overline{z}\right)+\sigma I_m\right){\tilde{W}}_a\left(t\right)\hfill \\ & +{\displaystyle \frac{1}{2}}{\widehat{W}}_c^T\left(t\right)\left(\mathsf{\Phi }\left(\overline{z}\right){\mathsf{\Phi }}^T\left(\overline{z}\right)+\sigma I_m\right){\widehat{W}}_c\left(t\right).\hfill \end{array}$$

(34)

According to the above equation (33) and inequality (34), the following inequality can be derived from (32)

$$\begin{array}{cc}\hfill \dot{L}\left(t\right)& \le -\left[g^2\left({\gamma }_u-{\displaystyle \frac{1}{2}}\right)-1\right]s^2-\left({\gamma }_c-{\displaystyle \frac{{\gamma }_a}{2}}\right){\tilde{W}}_c^T\left(t\right)\left(\mathsf{\Phi }\left(\overline{z}\right){\mathsf{\Phi }}^T\left(\overline{z}\right)+\sigma I_m\right){\tilde{W}}_c\left(t\right)\hfill \\ & -{\displaystyle \frac{1}{2}}\left({\gamma }_a-{\gamma }_c-{\displaystyle \frac{g^2}{2}}\right){\tilde{W}}_a^T\left(t\right)\left(\mathsf{\Phi }\left(\overline{z}\right){\mathsf{\Phi }}^T\left(\overline{z}\right)+\sigma I_m\right){\tilde{W}}_a\left(t\right)-{\displaystyle \frac{{\gamma }_c}{2}}{\widehat{W}}_c^T\left(t\right)(\mathsf{\Phi }\left(\overline{z}\right){\mathsf{\Phi }}^T\left(\overline{z}\right)\hfill \\ & +\sigma I_m){\widehat{W}}_c\left(t\right)+\left({\gamma }_c+{\displaystyle \frac{g^2}{4}}\right){W^{*}}^T\left(\mathsf{\Phi }\left(\overline{z}\right){\mathsf{\Phi }}^T\left(\overline{z}\right)+\sigma I_m\right)W^{*}+{\displaystyle \frac{1}{2}}{f}^2(\overline{z},{\overline{y}}_d)+{\displaystyle \frac{1}{2}}{\left(y_d^{\left(n\right)}\left(t\right)\right)}^2.\hfill \end{array}$$

(35)

This inequality (35) can be simplified to

$$\begin{array}{cc}\hfill \dot{L}\left(t\right)& \le -\left[g^2\left({\gamma }_u-{\displaystyle \frac{1}{2}}\right)-1\right]s^2-\left({\gamma }_c-{\displaystyle \frac{{\gamma }_a}{2}}\right){\tilde{W}}_c^T\left(t\right)\left(\mathsf{\Phi }\left(\overline{z}\right){\mathsf{\Phi }}^T\left(\overline{z}\right)+\sigma I_m\right){\tilde{W}}_c\left(t\right)\hfill \\ & -{\displaystyle \frac{1}{2}}\left({\gamma }_a-{\gamma }_c-{\displaystyle \frac{g^2}{2}}\right){\tilde{W}}_a^T\left(t\right)\left(\mathsf{\Phi }\left(\overline{z}\right){\mathsf{\Phi }}^T\left(\overline{z}\right)+\sigma I_m\right){\tilde{W}}_a\left(t\right)+C\left(t\right),\hfill \end{array}$$

(36)

where $C\left(t\right)=({\gamma }_c+{\displaystyle \frac{g^2}{4}}){W^{*}}^T\left(\mathsf{\Phi }\left(\overline{z}\right){\mathsf{\Phi }}^T\left(\overline{z}\right)+\sigma I_m\right)W^{*}+{\displaystyle \frac{1}{2}}{f}^2(\overline{z},{\overline{y}}_d)+{\displaystyle \frac{1}{2}}{\left(y_d^{\left(n\right)}\left(t\right)\right)}^2$ can be bounded by a constant $\beta$, i.e., $\left|C\left(t\right)\right|\le \beta$.

Let $\alpha =min\left\{g^2\left({\gamma }_u-{\displaystyle \frac{1}{2}}\right)-1,({\gamma }_c-{\displaystyle \frac{{\gamma }_a}{2}})\sigma ,{\displaystyle \frac{1}{2}}({\gamma }_a-{\gamma }_c-{\displaystyle \frac{g^2}{2}})\sigma \right\}$, then (36) can be rewritten as

$$\dot{L}\left(t\right)\le -\alpha L\left(t\right)+\beta$$

(37)

Applying Lemma 1, we rigorously obtain the following inequality:

$$L\left(t\right)\le e^{-\alpha t}L\left(0\right)+{\displaystyle \frac{\beta }{\alpha }}\left(1-e^{-\alpha t}\right).$$

(38)

The above inequality implies that the signals $\overline{z}\left(t\right)$, ${\tilde{W}}_c\left(t\right)$, and ${\tilde{W}}_a\left(t\right)$ are SGUUB. Moreover, by selecting the design parameters to make $\alpha$ sufficiently large, the tracking errors $z_1\left(t\right),\dots ,z_n\left(t\right)$ can be made to converge to an arbitrarily small neighborhood of zero. This completes the proof. □

5. Simulation Experiment

In this section, the effectiveness and advantages of the proposed adaptive optimal tracking control method are demonstrated through two simulation examples.

Example 1: Consider a second-order nonlinear dynamic model as presented in [30]:

$$\begin{array}{cc}\hfill & {\dot{x}}_1\left(t\right)=x_2\left(t\right),\hfill \\ & {\dot{x}}_2\left(t\right)={\left(1-x_1\left(t\right)\right)}^2-x_2\left(t\right)+gu,\hfill \end{array}$$

(39)

where $x_1\left(t\right)$ and $x_2\left(t\right)$ are the system states, u is the control input, and $g=1$ is the control gain, with initial conditions $x_1\left(0\right)=2$ and $x_2\left(0\right)=3$.

Based on the reference trajectory $y_d=5sin\left(t\right)$, the tracking errors can be obtained

$$\begin{array}{cc}\hfill & z_1\left(t\right)=x_1\left(t\right)-5sin\left(t\right),\hfill \\ & z_2\left(t\right)=x_2\left(t\right)-5cos\left(t\right).\hfill \end{array}$$

(40)

The error dynamics derived in (39) are

$$\begin{array}{cc}\hfill & {\dot{z}}_1\left(t\right)=z_2\left(t\right),\hfill \\ & {\dot{z}}_2\left(t\right)={\left(1-z_1\left(t\right)-5sin\left(t\right)\right)}^2-z_2\left(t\right)-5cos\left(t\right)+u+5sin\left(t\right).\hfill \end{array}$$

(41)

Following from (3), the SMS is designed as

$$s\left(t\right)=2z_1\left(t\right)+z_2\left(t\right).$$

(42)

The critic and actor network weight vectors are defined as ${\widehat{W}}_c={\left[{\widehat{W}}_{c,1},{\widehat{W}}_{c,2},{\widehat{W}}_{c,3}\right]}^T$ and ${\widehat{W}}_a={\left[{\widehat{W}}_{a,1},{\widehat{W}}_{a,2},{\widehat{W}}_{a,3}\right]}^T$, respectively. The activation function vector is given by $\mathsf{\Phi }\left(\overline{z}\right)={\left[z_1^2,z_1{z}_2,z_2^2\right]}^T$. The learning rates and regularization parameters for the training laws of both the critic and actor (15) and (16) are set to ${\gamma }_c=3$, ${\gamma }_a=5$, and $\sigma =1$. The weight vectors are initialized to ${\widehat{W}}_c\left(0\right)={\left[-1,0,1\right]}^T$ and ${\widehat{W}}_a\left(0\right)={\left[0,1,0\right]}^T$.

The simulation results are presented in Figure 1–fg4. Specifically, Figure 1 shows that the states $x_1\left(t\right)$ and $x_2\left(t\right)$ closely follow the reference trajectories $y_d$ and its derivative ${\dot{y}}_d$, respectively. The corresponding tracking errors $z_1$ and $z_2$, shown in Figure 2, converge to zero. Furthermore, the neural network weights for the actor and critic remain bounded, as illustrated in Figure 3. Finally, the evolution of the utility function is depicted in Figure 4. Compared with the simulation results in [30], the method proposed in this paper achieves superior tracking performance.

Remark 4. 

The purpose of presenting Example 1 is to compare the performance of our proposed algorithm with that of Algorithm in [30]. To further verify its adaptability, we select a higher-order problem as Example 2.

Example 2: This example considers a third-order system. The system dynamics are described as follows:

$$\begin{array}{c}{\dot{x}}_1\left(t\right)=x_2\left(t\right),\hfill \\ {\dot{x}}_2\left(t\right)=x_3\left(t\right),\hfill \\ {\dot{x}}_3\left(t\right)=x_1\left(t\right)x_2\left(t\right)+e^{-2x_3\left(t\right)}+gu.\hfill \end{array}$$

(43)

The control gain is $g=2$, and the initial states are $x_1\left(0\right)=4$, $x_2\left(0\right)=0$, $x_3\left(0\right)=1.5$.

Given the reference trajectory $y_d=2(sin\left(t\right)+cos\left(0.5t\right))$, the tracking errors can be calculated

$$\begin{array}{cc}\hfill & z_1\left(t\right)=x_1\left(t\right)-2(sin\left(t\right)+cos\left(0.5t\right)),\hfill \\ & z_2\left(t\right)=x_2\left(t\right)-2(cos\left(t\right)-0.5sin\left(0.5t\right)),\hfill \\ & z_3\left(t\right)=x_3\left(t\right)+2(sin\left(t\right)+0.25cos\left(0.5t\right))\hfill \end{array}$$

(44)

The error dynamics derived in (43) are

$$\begin{array}{cc}\hfill {\dot{z}}_1\left(t\right)& =z_2\left(t\right),\hfill \\ \hfill {\dot{z}}_2\left(t\right)& =z_3\left(t\right),\hfill \\ \hfill {\dot{z}}_3\left(t\right)& =\left[z_1\left(t\right)+2(sin\left(t\right)+cos\left(0.5t\right))\right]\left[z_2\left(t\right)+2(cos\left(t\right)-0.5sin\left(0.5t\right)\right]\hfill \\ \hfill & +exp\left[-2\left(z_3\left(t\right)-2(sin\left(t\right)+0.25cos\left(0.5t\right))\right)\right]+gu+2(cos\left(t\right)-0.125sin\left(0.5t\right)).\hfill \end{array}$$

(45)

Based on (3), the SMS is designed as

$$s\left(t\right)=z_1\left(t\right)+2z_2\left(t\right)+z_3\left(t\right).$$

(46)

The activation function vector is given by $\mathsf{\Phi }\left(\overline{z}\right)={\left[z_1^2,z_1{z}_2,z_1{z}_3,z_2^2,z_2{z}_3,z_3^2\right]}^T$. The weight vectors are initialized to ${\widehat{W}}_c\left(0\right)={\left[-1,0,1,2,0,1\right]}^T$ and ${\widehat{W}}_a\left(0\right)={\left[0,40,0,20,0,10\right]}^T$. The parameters are set to ${\gamma }_u=80$, ${\gamma }_c=3$, ${\gamma }_a=5$, and $\sigma =1$.

The simulation results of Example 2 are presented in Figure 5, Figure 6, Figure 7 and Figure 8. As shown in Figure 5, the states $x_1\left(t\right)$, $x_2\left(t\right)$ and $x_3\left(t\right)$ closely follow the reference trajectories $y_d$ and its first- and second-order derivatives ${\dot{y}}_d$ and ${\ddot{y}}_d$, respectively. The corresponding tracking errors $z_1$, $z_2$, and $z_3$ depicted in Figure 6, converge to a small neighborhood of zero. Furthermore, Figure 7 demonstrates that the neural network weights for both the actor and critic remain bounded. Finally, the evolution of the utility function is shown in Figure 8. These results demonstrate that the proposed high-order tracking control scheme for canonical nonlinear systems maintains the boundedness of the error signal while ensuring accurate tracking of each system state to its corresponding reference signal.

6. Conclusions

This paper proposes a SMS-based adaptive optimal tracking control scheme, utilizing an NN-based RL approach, for a class of high-order nonlinear systems with partial unknowns. A cost function defined in terms of the SMS is constructed to derive the optimal control law. Accordingly, the HJB equation is addressed within an actor-critic NN framework. Leveraging the inherent relationship between the SMS and the tracking errors significantly simplifies the computational process. Under the premise of ensuring the stability of the closed-loop system, the update law of the actor-critic network is indirectly derived under the condition that a simple function is zero, without the need for persistence excitation condition. Furthermore, the proposed design requires no prior knowledge of the internal system dynamics. Meanwhile, the proposed method offers a significant advantage over [30] by eliminating the need for a Riccati-like equation. Comparative simulation results demonstrate the superior tracking performance of the proposed scheme over existing methods. Finally, future work will extend this framework to tackle the inverse optimal tracking problem, addressing the challenge of manual cost function specification.