An Implementation based on Mask-D3QN of Quadruped Robot Motion for Steam Generator

1. Introduction

With the rapid development of nuclear energy, research on robots in the nuclear power field has become increasingly prevalent [1]. Within pressurized water reactors of nuclear power plants, the heat transfer tubes within the Steam Generators (SG) serve as critical components for heat exchange, which requires regular inspection and maintenance. How-ever, the specifications of tubes within steam generators vary, as do the distribution of the tube plates. There are many kinds of inspection robots in service, such as ZR-100 [2], PEGASYS [3], HIT-Crawler [4], etc. However, each ro-bot can only be applied to a specific type of steam genera-tor, featuring limited motion capabilities. Consequently, inspection tasks for different steam generators necessitate the corresponding robots, followed by re-installment and recalibration, leading to suboptimal inspection efficiency.

To enhance the adaptability of robots to various types and specifications of tube plates, enabling them to move and inspect tube holes on the plate more flexibly, safely, and efficiently, we have developed a tube plate quadruped inspection robot named HIT_Spibot [5]. Concurrently, the development of a motion planner is required to ensure its rapid and reliable movement along specified paths on the tube plate.

Research on the locomotion of legged robots primarily focuses on ensuring stability, as these robots rely on contact forces such as friction to traverse on the ground. Ensuring stability is particularly crucial to prevent falls or overturning [6]. However, the toes of HIT_Spibot are firmly attached to the tube plate, allowing the robot to remain stable on the plate as long as three feet are fixed. Due to the unique character of the plate and robot structure, the objective of motion planning for tube plate quadruped inspection robots is to achieve flexible and efficient motion planning while ensuring safety. It is noteworthy that, to date, there has been no scholarly investigation into the motion planning of such robots engaged in complex planar movements.

Therefore, this paper proposes a hierarchical planning method for the tube plate quadruped inspection robot. Utilizing an environment map constructed based on grid-based methods, the method designs the robot base motion planner, the robot base turning planner, and the gait motion planner, to enable the robot to move on the tube plate efficiently and safely. The task analysis and algorithm implementation framework are illustrated in Figure 1.

This paper aims to address three key issues: How to enhance planning efficiency using hierarchical planning? How to plan the motion of the robot's base and foot-points? How to utilize the results of hierarchical planning to realize robot motion? In this work, we make three significant contributions:

(1) We devise a base motion planner, formulating the base trajectory planning problem as a constrained nonlinear optimization problem. By employing the Lagrangian method, we attain the desired optimal solution to facilitate motion planning for the robot's base.

(2) We establish a base turning planner based on the TOPSIS method, enabling the robot to turn to an optimal initial pose, facilitating movement along the specified direction.

(3) We propose a gait motion planner based on the MD3QN algorithm, designing a mask during the action selection and designing a reward function, thereby enabling autonomous learning of the robot foot-points. By integrating the outcomes of the above two planners, the robot can move on the tube plate flexibly and efficiently along the desired trajectory. Experimental validation demonstrates the effectiveness and advancement of the MD3QN algorithm.

The chapter arrangement of the paper is as follows. Section 2 presents the related work, and Section 3 introduces the structure of the robot, analyzing its motion characteristics and kinematics. Section 4 outlines the proposed algorithm. Section 5 provides experimental validation, and finally, the conclusion is presented.

2. Related Work

Early scholars typically employed model-based approaches for the motion planning of quadruped robots. The leg motion problem was transformed into a mathematical optimization problem, generating motion plans for quadruped robots facing complex terrain and tasks [7]. While this mathematical modeling approach is simple and feasible, it necessitates the establishment of sufficiently accurate models to compute a single trajectory optimization formula for leg motion, thereby determining the robot's gait sequence and foot placement [8].

Although model-based methods can plan robot motion for various environments [9], each environment requires a separate analysis for its motion conditions, leading to substantial computation. In addition, traditional model-based planning methods in specific environments require exhaustive analysis of the robot's motion characteristics, operating conditions, and potential future scenarios. Inadequate analysis may result in unsolvable motion planning.

With the gradual advancement of computer technology, scholars have discovered a more generalized and automated approach to achieve motion planning for quadruped robots. This involves specifying a high-level task, wherein algorithms determine the robot's motion while considering the physical constraints of both foot and base movements. Given that the problem is properly formulated, ideally, the algorithm can generate motion for any high-level task and address motion planning issues at a generic level [7]. This is the planning method based on reinforcement learning (RL), where the control of robot motion strategies can be automatically generated through network learning. Moreover, the same learning method enables robots to learn optimal strategies across various environments [10].

On the other hand, as research on deep learning continues to deepen, deep reinforcement learning (DRL), combining deep learning with reinforcement learning, has been produced [11]. Researchers have found that applying DRL algorithms to motion planning for quadruped robots can prompt robots to adapt actively to the environments [12]. The essence of DRL algorithms lies in the agent learning which actions to take in a given environment to maximize numerical reward signals [13]. Based on the different actions output by robot planning, DRL algorithms can be categorized into methods based on value functions to ad-dress discrete action spaces, and methods based on policy gradients to address continuous action spaces [14].

Learning problems in continuous action spaces are more complex than those in discrete action spaces [15]. As the heat transfer tubes are distributed on specified tube plates, the motion planning can be transformed into planning discrete foot-points that meet task requirements. Compared to planning continuous joint angles, this method is more precise and straightforward, and it can also reduce unnecessary search space.

The earliest proposed DRL algorithm for solving discrete action space problems is the Deep Q-Network (DQN) algo-rithm [16]. This algorithm utilizes convolutional neural networks to approximate the state-action value function and reduces data correlation through the experience replay mechanism to improve learning efficiency. Simultaneously, it employs target Q-networks for training, providing relia-ble target values during temporal difference backups [17]. The application of the DQN algorithm to legged robots has also yielded significant results. Researchers constructed a discrete gait model for a hexapod robot and designed a method based on the DQN algorithm, planning a free gait with high stability margin and a strong adaptability[18].

However, traditional DQN algorithms suffer from issues such as sparse rewards, low sample utilization, and overestimation. Therefore, many variants of DQN have been pro-posed. Among them, Double DQN primarily addresses the problem of overestimation by using different value functions to select actions and evaluate them [16]. On the other hand, Dueling DQN decomposes the output of the Q-values in the neural network into two parts: state value V and the difference A between action value and state value [19].

In task-oriented DRL, convergence difficulties inherent in RL itself render learning from scratch impractical for complex tasks. Therefore, hierarchically processing the required network structure and learning process is an intuitive choice. Jain et al. [20] proposed a hierarchical RL scheme enabling the Ghost robot to navigate along specific trajectories. This method employs a high-level policy net-work to control a low-level policy network, enabling the former to focus on determining the robot's motion direction without learning the motor control commands. [21] introduced a hierarchical control method that employs hierarchical reinforcement learning to decompose complex tasks and yielded significant results in practice.

In summary, inspired by the advantages of hierarchical RL, we propose a hierarchical motion planning method based on the MD3QN algorithm. This method comprises three layers: the base motion planner, the base turning planner, and the gait motion planner. Specifically, the gait motion planner, based on the MD3QN algorithm, is utilized to plan the robot foot-points, thereby achieving efficient, safe, and flexible robot motion.

3. Preliminaries

3.1. Robot Structure

The structure of the tube plate quadruped inspection robot HIT_Spibot is illustrated in Figure 2 (a). The robot consists of a base module, four leg modules, and a tool module, enabling moving on the tube plate through combined motion. The tool module carries eddy current tools to inspect the heat transfer tubes. The design of the toe modules ensures that when three or four toes are fixed on the tube plate, the robot can be stably and reliably fixed to the tube plate. (The subsequent description and illustration hide the tool module.)

Through analysis of the robot's structure, we can observe significant differences between the HIT_Spibot and traditional quadruped robots:

Stability Considerations: Traditional quadruped robots require sufficient stability margin during motion, commonly employing control methods such as Zero Moment Point [22] and Central Pattern Generators [23]. However, due to the structural specificity of the HIT_Spibot, as long as the toe modules are fixed to the tube plate, the robot can maintain stability without tipping over or falling. Nevertheless, considering the lifespan of components, it is advisable for the robot to minimize the stress on the toe modules during motion. This implies that the robot's center of mass should be positioned as close as possible to the given base trajectory, preferably within the support polygon.

Foot-point Selection: Traditional quadruped robots need to consider terrain conditions, their own state, and the impact of accessibility on stability and speed. In contrast, for HIT_Spibot, any position satisfying the kinematic conditions can serve as a feasible foot-point.

3.2. Motion Analysis

The motion of HIT_Spibot can be categorized into the robot foot motion and base motion. Based on the robot's configuration, we establish the coordinate system as illustrated in Figure 3 and present the Denavit-Hartenberg parameters in Table 1.

The robot’s foot motion is as follows. Three legs act as supporting feet, while the remaining one serves as the swinging foot, forming a 2R-type serial mechanical arm with the robot base, as depicted in Figure 4(a). Assuming the position of the robot base is denoted as $B(r_B, c_B)$ , where $r_B$ and $c_B$ represent the row and column coordinates of the robot base, respectively. The forward kinematics simplified calculation formula for each foot $F_i\left(r_i, c_i\right)$ is as follows, where ${\mathsf{\alpha }}_i=45+90\text{∙}(i-1), i=1,2,3,4$, denotes the angle of each foot's rotation axis relative to the tube plate.

$$\left\{\begin{array}{c}\begin{array}{c}{r}_i=r_B+\sqrt{b^2+w^2} \cos ({\mathsf{\alpha }}_i+{\theta }_{1i})+{ a}_1 \cos ({\mathsf{\alpha }}_i+{\theta }_{1i}+{\theta }_{2i}) \\ +{ a}_2 \cos ({\mathsf{\alpha }}_i+{\theta }_{1i}+{\theta }_{2i}+{\theta }_{3i})\end{array}\\ \begin{array}{c}{c}_i=c_B+\sqrt{b^2+w^2} \sin ({\mathsf{\alpha }}_i+{\theta }_{1i})+a_1 \sin ({\mathsf{\alpha }}_i+{\theta }_{1i}+{\theta }_{2i}) \\ +a_2 \sin ({\mathsf{\alpha }}_i+{\theta }_{1i}+{\theta }_{2i}+{\theta }_{3i})\end{array}\end{array}\right.$$

(1)

The robot base motion is driven by the contact forces between the feet and the tube plate. In the case of three or four supporting feet, the tube plate along with the fixed feet and robot base, form a 3-degree-of-freedom (DOF) parallel robot, as depicted in Figure 4 (b) and (c). Assuming the positions of the three supporting feet relative to the base are denoted as $F_1\left(r_1, c_1\right)$, $F_2\left(r_2, c_2\right)$, and $F_3\left(r_3, c_3\right)$, the simplified inverse kinematics formulas for the rotation angles ${\theta }_{2i}$ and ${\theta }_{3i}$ of each foot, are given as follows.

$$\left\{\begin{array}{c}{\theta }_{2i}=\left(arctan{\displaystyle \frac{r_i}{c_i}}or\text{±}{\displaystyle \frac{\pi }{2}}\right)\text{±}arccos{\displaystyle \frac{{r_i}^2+{c_i}^2+{a_1}^2-{a_2}^2}{2a_1\sqrt{{r_i}^2+{c_i}^2}}}\\ {\theta }_{3i}=arccos{\displaystyle \frac{{r_i}^2+{c_i}^2-{a_1}^2-{ a_2}^2}{2a_1{a}_2}}\end{array}\right.$$

(2)

4. Method

Given the tube plate and the robot's motion path, we con-sider the solved regular robot workspace as prior knowledge. Based on this, we plan the robot's motion, including base motion, base turning, and gait motion.

4.1. Enveloping Method for the Workspace Solution

Due to the irregular workspace of the robot's base and feet, exploration and computation are significant challenges. Therefore, this study considers envelope analysis to solve the regular workspace.

Given the robot base pose, the actual foot workspace is illustrated in Figure 5 (a), which is intersected by two lines, presenting a ring-shaped area as depicted in the orange-shaded region in Figure 5 (b). On the other hand, given the robot feet position, the actual base workspace is depicted in Figure 5 (c), which is in the region formed by the intersection of four or three circular rings, as shown in the blue-shaded region in Figure 5 (b).

We adopt minimum bounding rectangles to represent the workspaces of the robot base and feet. The foot workspace’s bounding area is necessarily a truncated ring. By considering the four endpoints of the truncated ring and the tan-gents of each circle, the foot workspace bounding rectangle can be determined. As for the base workspace, its bounding area is necessarily formed by the intersection points of three or four circles. By considering these intersection points and the tangents of each circle, the base workspace bounding rectangle can be determined, as illustrated in Figure 6.

4.2. Robot Base Motion Planner

Base motion planning involves determining feasible base trajectories when the foot swings. The points along the base trajectory should lie within the base workspace and align as closely as possible with the given path. Therefore, this section aims to plan the local destinations for each base motion under three supporting feet and one swinging foot, modeling this process as a nonlinear optimization problem to obtain the optimal desired base position. For constrained optimization problems with inequality constraints, various solving methods can be employed. In this section, the Lagrange multiplier method [24].

Assuming the starting point of the base is $P_s(x_s,y_s)$, the destination of the motion direction is $P_e\left(x_e, y_e\right)$, and the coordinates of the two corner points of the current supporting feet’s bounding rectangle are $R_1\left(a, c\right)$ and $R_2\left(b, d\right)$, the optimal desired base position $P_b\left(x, y\right)$ is formulated as follows:

$$\left\{\begin{array}{c}minf\left(x, y\right)={‖\overrightarrow{P_s{P}_b}‖}_2+{‖\overrightarrow{P_e{P}_b}‖}_2\\ subjectto\begin{array}{c}{g}_1(x, y)=-x+\min (a, b)\text{≤}0\\ g_2(x, y)=x - \max (a, b)\text{≤}0\\ \begin{array}{c}{g}_3(x, y)=-y+\min (c, d)\text{≤}0\\ g_4(x, y)=y - \max (c, d)\text{≤}0\end{array}\end{array}\end{array}\right.$$

(3)

where ${‖A‖}_2$ denotes the L2 norm of A, that is, ${‖\overrightarrow{P_s{P}_b}‖}_2={\displaystyle \frac{\left|x-x_1\right|}{\sqrt{{\left(x-x_1\right)}^2+{(y-y_1)}^2}}}+{\displaystyle \frac{\left|y-y_1\right|}{\sqrt{{\left(x-x_1\right)}^2+{(y-y_1)}^2}}}$. Introducing Lagrange multipliers $\mu \text{{}{\mu }_i, i=1, \dots , 4\text{}}$, the corresponding Lagrangian function is formulated as follows:

$$L(x, y, \mu )=f(x, y)+\sum _{j=1}^n{\mu }_j{g}_j(x, y)$$

(4)

By introducing the Karush-Kuhn-Tucker conditions as follows, the solution set for the optimal ideal base position can be obtained. The evaluation criterion is based on the distance from the global motion destination, where a closer distance indicates a more optimal base position, leading to the derivation of the optimal ideal base position $P_b\left(x^{*}, y^{*}\right)$.

$$\begin{array}{c}\begin{array}{c}{\text{∇}}_{x, y}L(x, y, \mu )=0\\ g_j(x, y) \text{≤} 0, j=1,2,3,4\end{array}\\ {\mu }_j\text{≥}0\\ {\mu }_j{g}_j(x,y)=0\end{array}$$

(5)

Near the position $P_b\left(x^{*}, y^{*}\right)$, conducting a localized search to obtain an optimal feasible base position, satisfying the inverse kinematic feasible solutions. Through the process from the current base position to the optimal feasible base position, a series of base positions can be obtained by solving the inverse kinematics along the motion direction, thereby acquiring the local base motion trajectory.

4.3. Robot Base Turning Planner

By analyzing the robot's motion characteristics and working environment, we can determine the desirable initial posture of the robot based on the following conditions: one of the robot's orientations aligns as closely as possible with the given direction; the four feet are evenly distributed along this orientation to facilitate subsequent motion; and this posture ensures sufficient feasible space for the swinging foot to place at the start of motion, thereby enhancing the likelihood of movement. These conditions can be translated into evaluation criteria: Cost Index 1—the angle between the robot's orientation and the given direction; Cost index 2—the deviation of the foot angle relative to the base from the standard 45° angle; and Benefit index 3—the size of the feasible set of foot-points for the next swinging foot. Therefore, the planning of the base turning planner can establish an evaluation system as depicted in Figure 7.

The objective of the base turning planning can be formulated as a Multi-Criteria Decision Making (MCDM) problem. The most commonly used method for solving MCDM problems is the TOPSIS method, which is a multi-criteria approach that determines the optimal solution from a finite set of alternatives by minimizing the distance to the ideal point and maximizing the distance to the worst point [25]. The TOPSIS method is mathematically simple and offers flexible selection [26]. In this section, the base turning planning is based on the TOPSIS algorithm, and the process is outlined in Algorithm 1. Here, n represents the number of landing points to be evaluated, i denotes the landing point index, and the termination condition minimizes the angle between the orientation and the given direction while ensuring uniform distribution of the four feet.

Algorithm 1 Turning Planning Based on TOPSIS

Input: target base angle ${\mathsf{\alpha }}_T$ and current state s

Output: Motion sequences

Compute feasible foot-point $I_p,p=1,2,\text{⋯},n$

for $p$in range(n) do

Compute base direction angle $x_{p1}$, foot to base angle $x_{p2}$ and next foot-point number $x_{p3}$

end

while not stopping condition(s) do

Obtain TOPSIS score:

Step 1: Positivize Data

$x_{\mathrm{ij}}$ on Cost index index:$\widehat{x_{\mathrm{PQ}}}=\max \text{{}{x}_p\text{}}-x_{\mathrm{pq}},q=1,2$

$x_{\mathrm{ij}}$ on Benefit index index:$\widehat{x_{\mathrm{PQ}}}=x_{\mathrm{pq}},q=3$

Step 2: Normalize Data

$z_{\mathrm{pq}}=\widehat{x_{\mathrm{PQ}}}/\sqrt{{\sum }_{p=1}^n{x_{\mathrm{pq}}}^2}$

Step 3: Determine the positive and negative ideal solutions

$z_{\mathrm{bt}\text{\_}p}=[\max \text{{}{x}_{p1}\text{}},\max \text{{}{x}_{p2}\text{}},\max \text{{}{x}_{p3}\text{}}],p=1,2,\text{⋯},n$     $z_{\mathrm{wt}\text{\_}p}=[\min \text{{}{x}_{p1}\text{}},\min \text{{}{x}_{p2}\text{}},\min \text{{}{x}_{p3}\text{}}],p=1,2,\text{⋯},n$

Step 4: Calculate the TOPSIS score   $S_p={\sum }_{q=1}^3{(z_{\mathrm{wtq}\text{\_}p}-z_{\mathrm{pq}})}^2/({\sum }_{q=1}^3{(z_{\mathrm{wtq}\text{\_}p}-z_{\mathrm{pq}})}^2+{\sum }_{q=1}^3{(z_{\mathrm{btq}\text{\_}p}-z_{\mathrm{pq}})}^2)$

Obtain foot-point ${I_p}^{*}=\max \text{{}{S}_p\text{}}$

Update state s

4.4. Robot Gait Motion Planner

Taking the robot posture output from the base turning planner as input to the gait motion planner, we start learning the robot's motion along the given path from existing strategies, aiming to reduce the difficulty of DRL learning [18]. The network model trained by the D3QN algorithm obtains foot-points, then updates the status, repeating this process until reaching the destination. In this section, the robot's free gait motion is to plan the foot-points, ensuring that the robot can reach the destination as quickly as possible and minimizing the force on the toe modules.

To enhance the flexibility of robot motion, this paper se-lects the foot-point sequence of the left front foot (lf), right front foot (rf), right back foot (rb), and left back foot (lb) along the motion direction, as illustrated in Figure 8.

The learning-based approach is that the agent learns by multiple interacting with the environment, encouraging positive behaviors and punishing negative ones. We address the sequential decision-making problem of interaction between an agent and its environment with the objective of maximizing the cumulative discounted reward. This problem is modeled as a discrete-time Markov Decision Process, consisting of a tuple $⟨\mathcal{S},\mathcal{A},\mathcal{ }\mathcal{P},\mathcal{R},\gamma ⟩$, where $\mathcal{S}$ represents the states of the robot, $\mathcal{A}$ represents actions, $\mathcal{P}$ represents the state transition distribution, $\mathcal{R}$ represents the reward function, and $\gamma$ represents the discount factor.

During the training process, the state $\mathcal{S}$ serves as the input to the robot's motion planning policy network, while the action $\mathcal{A}$ serves as the output of the policy network. The robot interacts with the environment based on the selected actions from the policy network, leading to transitions in the robot's state. Simultaneously, the environment provides feedback through a reward function $\mathcal{R}$. The policy network then selects the next action based on the updated state and reward function. The continuous updating of the policy network is the training process for the motion planner using RL algorithms.

Action Space $\mathcal{A}$: For most quadruped robots employing RL, the action space typically consists of joint positions [27]. However, since each foot places on a specific hole on the plate, in order to reduce unnecessary exploration at the decision-making level, this paper defines the action space as a one-dimensional vector of discrete foot-point scores, as shown as follows. Each action corresponds to a foot-point $F_k\left(r_{F_k}, c_{F_k}\right)$, which is an offset relative to the current foot position $F_c\left(r_{F_c}, c_{F_c}\right)$, where $k=1, 2, \text{⋯},{ k}_{\max }$. Here, ${ k}_{\max }$ represents the number of foot-points that satisfy all possible inverse kinematics conditions and is also the dimension of the action space.

$${\mathit{a}}_C=[{\mathrm{Score}}_1,{\mathrm{Score}}_2, \text{⋯}, {\mathrm{Score}}_{k_{\max }}]$$

(6)

State Space$\mathcal{S}$: The state space typically includes information such as the robot's position and joint positions [21]. To simplify the learning process, this paper considers prior information on the accessibility of actions offline when designing the state space, and adds the target position to guide learning. The state space $s_C$ is defined as follows, containing the pose of the robot $\mathit{P}=[r_B,c_B,{\theta }_1,{\theta }_{21}, {\theta }_{31}, {\theta }_{22}, {\theta }_{32}, {\theta }_{23},{\theta }_{33}, {\theta }_{24}, {\theta }_{34}]$, the global motion destination$T\left(r_T, c_T\right)$, and the judgment flag $f=[f_1, f_2, \text{⋯}, f_{k_{\max }}]$ indicating whether the foothold point corresponding to the action space can be landed on. Here, $f_k=1$ indicates that the foot-point is feasible, and $f_k=0$ indicates that the foot-point is not feasible.

$$s_C=[\mathit{P},{ \mathit{a}}_C, \mathit{f}, T]$$

(7)

Reward Function $\mathcal{R}$: The reward function is set based on specific task objectives [27]. The reward function $R_C$ consists of three parts and is designed as follows. One part is the regular penalty, where each step incurs a penalty $m_0$ to ensure that the robot does not move excessively or remain stationary; another part is the reward/penalty for forward/backward movement ${\mathit{R}}_{\mathit{foot}}=[R_{\mathrm{foot}\text{\_}1},{ R}_{\mathrm{foot}\text{\_}2}, \text{⋯},{ R}_{\mathrm{foot}{\text{\_}k}_{\max }}]$; the third part is the stability reward/penalty ${\mathit{R}}_{\mathit{stable}}=[R_{\mathrm{stable}\text{\_}1}, R_{\mathrm{stable}\text{\_}2}, \text{⋯},{ R}_{\mathrm{stable}{\text{\_}k}_{\max }}]$. Besides, $l_{\mathrm{foot}\text{\_}k}$ represents the distance traveled by the foot along the motion direction, where a positive value indicates proximity to the target, a negative value indicates moving away from the target, and zero indicates stationary movement along the direction of motion; $l_{\mathrm{foot}\text{\_}\max }$ represents the maximum distance the foot can move within the action space; $l_{c\text{\_}\mathrm{to}\text{\_}\sup \text{\_}k}$ represents the distance from the centroid of the stable triangle formed by the current supporting feet to the base; $l_{c\text{\_}\mathrm{to}\text{\_}\mathrm{line}\text{\_}k}$ represents the distance from the centroid of the stable triangle formed by the current supporting feet to the given motion direction. $m_1\text{~}{m}_8$ are parameters related to the reward function.

$$\left\{\begin{array}{c}{\mathit{R}}_C=-m_0+{\mathit{R}}_{\mathrm{foot} }+{ m}_1{\mathit{R}}_{\mathit{stable}}\\ R_{\mathrm{foot}\text{\_}k}=\left\{\begin{array}{c}{m}_2\text{×}{(l_{\mathrm{foot}\text{\_}k}/l_{\mathrm{foot}\text{\_}\max })}^{m_3} -m_4,l_{\mathrm{foot}\text{\_}k}\text{≥}0\\ m_{5 }-{m_6}^{l_{\mathrm{foot}\text{\_}k }/ l_{\mathrm{foot}\text{\_}\max }},else\end{array}\right.\\ R_{\mathrm{stable}\text{\_}k}=\left\{\begin{array}{c}0\\ -m_7\text{×}{(\max \text{{}{l}_{c\text{\_}\mathrm{to}\text{\_}\sup \text{\_}k},{ l}_{c\text{\_}\mathrm{to}\text{\_}\mathrm{line}\text{\_}k}\text{}})}^{m_8}\end{array}\right.\end{array}\right.$$

(8)

Termination Conditions: Termination conditions are crucial for initializing the state during the training process and for early termination of erroneous actions, thereby avoiding the wastage of computational resources [21]. This paper designs the following termination conditions:

Success Termination Condition: When the robot's base position matches the target base position, it indicates that the robot has successfully reached the destination.

Failure Termination Condition: When the robot's base position oscillates repeatedly, or when the current swing foot position oscillates back and forth, it indicates failure of the current exploration.

Action Selection: Action selection essentially addresses the proportion problem between exploration and exploitation, i.e., a trade-off between exploration and exploitation [28]. A simple and feasible method is the $\mathsf{ϵ}$-greedy method [29]. At each time step, the agent takes a random action with a fixed probability $0 \text{<} \mathsf{ϵ} \text{<} 1$, instead of greedily selecting the optimal action learned about the Q function [30]. In this paper, during greedy selection, a mask layer is added behind the output layer of the neural network, which is a list indicating whether the foot-point is feasible, to ensure that the highest scoring action is reachable, thereby avoiding a large number of unnecessary failure termination conditions. Similarly, during random selection, actions are chosen only from the list of feasible foot-points ${\mathcal{A}}_f\left(s\right)$. Action selection is depicted as follows, where $0 \text{≤} \mathsf{\xi } \text{≤} 1$ is a uniformly random number sampled at each time step.

$$\pi (s)=\left\{\begin{array}{c}randomactionfromfeasible{A}_f\left(s\right),\mathsf{\xi }\text{<}\mathsf{ϵ}\\ arg{\max }_{a\text{∈}A(s)}Q(s,a)f\left(s\right),otherwise\end{array}\right.$$

(9)

Gait Planner: The gait planning strategy parameters are transformed into a neural network, consisting of two hidden layers with a size of 64 each and an activation function. In each iteration, based on the current state, the neural net-work outputs the Q-values for each action through a linear output layer, and then the action selection method is used to choose the action for this state.

D3QN algorithm：

The D3QN algorithm is a variant algorithm that combines aspects of both Double DQN and Dueling DQN. It aims to eliminate the maximization bias during network updates, address the issue of overestimation, and expedite algorithm convergence. Compared to the traditional DQN algorithm, D3QN primarily optimizes in two aspects. Firstly, it adopts the same optimized TD error target as the Double DQN algorithm and utilizes two networks for updates: an evaluation network (parameter $w_e$)) is employed to determine the next action, while a target network (parameter $w_t$) calculates the value of the state at time t + 1, thereby mitigating the problem of overestimation. The target for D3QN updates is expressed as follows, where $\mathsf{\gamma }$ represents the discount factor, $r_{t+ 1}$ denotes the reward function at time t + 1, and $Q(s_t, a)$ denotes the state-action value function with respect to state $s_t$ and action $a_t$.

$$y_t=\left\{\begin{array}{c}{r}_{t+1},ifnextstateisfinalstate\\ r_{t+1}+\mathsf{\gamma }{Q}_{w_t}(s_{t+1},arg\underset{a}{ \max }{Q}_{w_e}(s_{t+1},a)),else\end{array}\right.$$

(10)

Secondly, the D3QN algorithm decomposes the state-action value function into two components, consistent with the Dueling DQN algorithm, to model the state value function and the advantage function separately, thereby better-handling states with smaller associations with actions. The newly defined state-action value function is represented as follows, where the maximization operation is replaced with the mean operation for enhanced stability of the algorithm. Here, $V\left(s_t\right)$ denotes the state value function with respect to the state $s_t$, $A_{w_A}(s_t,a_t)$ represents the advantage function with respect to the state $s_t$ and action $a_t$, and mean denotes the mean operation.

$$Q(s_t,a_t)=V(s_t)+A(s_t,a_t)-\underset{a}{\mathrm{mean}}A(s_t,a_t)$$

(11)

Building upon the foundation of the D3QN algorithm, this paper establishes four separate MD3QN network models for each leg that incorporates a mask during action selection. The gait planning algorithm for quadrupedal robots based on the MD3QN algorithm is depicted in Algorithm 2.

Algorithm 2 Gait Planning Based on MD3QN

Input:$\mathrm{learning} \mathrm{rate} \mathit{lr}, \mathrm{batch} \mathrm{size} \mathit{bs}, \mathrm{discounting} \mathrm{factor} \gamma$, attenuation parameter $ϵ$, attenuation rate ${ϵ}_{dec}$, minimum attenuation value ${ϵ}_{min}$, target network update frequency freq, Soft update parameter $\tau$, maximum number of per episode $n_t$, maximum number episodes $n_s$, maximum number of experience pool $n_e$

Output: eval network that can output gaits according to state s

Initialize the experience pool M, and the parameters $w_t$ and $w_e$

for episode $e\in \{\mathrm{1,2},\cdots ,n_s\}$ do

Reset and obtain the initialization state $s_t$

for time step $t\in \{\mathrm{1,2},\cdots ,n_t\}$ do

for foot $i\in \{\mathrm{1,2},\mathrm{3,4}\}$ do

Choose action $a_{ti}={\pi }_i\left(s_{ti}\right|s_{ti}=s_t)$

Compute reward $r_{ti}$

Obtain the new state $s_{t\{i+1\}}$ and done flag $done$

Store experience {$s_t,a_{ti},r_{ti},s_{t\{i+1\}},done\}$ into M

if len(M) $\ge$bs then

Sample batch data from the experience pool M

Obtain the target values $y_t$

Do a gradient descent step with loss ${‖y_t-Q_{w_e}(s_{ti},a_{ti})‖}^2$ update $w_e$

end if

if$t \% freq=0$

Update $w_t$:$w_t\leftarrow \tau w_e+(1-\tau )w_t$

end if

$s_t\leftarrow s_{t\{i+1\}}$

end for

end for

end for

5. Experiment

5.1. Experimental Setup

The experiments were conducted on two types of tube plates, as illustrated in Figure 9. The algorithm's hyperparameters are shown in Table 2, utilizing the PyTorch framework with Adam optimizer. The training iterations $n_s$, reward function parameters $m_0$ and $m_2$ are correlated with the robot's maximum step length $l_{foot\_max}$, as outlined in Table 3.

Our independently developed tube plate quadruped in-spection robot HIT_Spibot are depicted in Figure 10, and the parameters are outlined in Table 4. The experimental tube plate is arranged in a square distribution format, with dimensions of 43 rows by 73 columns.

To validate the feasibility and effectiveness of the proposed algorithm, three experiments are designed as follows:

Ablation experiments: Randomly selected combinations of environments and tasks are used to train four mutually influencing networks separately with MD3QN and D3QN algorithms. In each experimental group, task action space, state space, $n_s$, $m_0$, $m_2$, and $l_{\mathit{foot}\text{\_}\mathit{max}}$ are set according to the dimensions of the robot and the tube plate.

Performance experiments: Obstacles are randomly added to the above environments, and the starting points and destinations of the paths are changed along the specified path direction. The trained models from the previous experiments are utilized for analysis of the adaptability of the proposed algorithm to different environments and tasks.

Practical Experiment: Obstacles are added to an actual square small tube plate, two motion paths are set, and the trained model is used to test and analyze the performance of the proposed algorithm when applied to a real robot.

5.2. Ablation Experiment

To evaluate the proposed method, we apply the MD3QN algorithm and the D3QN algorithm to gait motion planning. Trainings are conducted under the same environmental and hyperparameter conditions to compare algorithm performance. We introduce obstacles in both square and triangular plate environments, setting up 5 tasks randomly to obtain 10 different environments and tasks. Three metrics are employed to measure the planning performance, including Reward (total score of the planning results, higher is better), Step (total number of steps in the planning results, lower is better), and Stable Reward (stability score of the planning results, with stability score always non-positive, higher is better).

The final training results for the 10 environment-task combinations are presented in Table 5, where 'S' denotes the square plate and 'T' denotes the triangular plate, followed by the task number. It can be observed that, across different environments and tasks, compared to the D3QN algorithm, the final planning results obtained using the MD3QN algo-rithm exhibit higher total scores, with an average increase of 6.27%; fewer steps, with an average reduction of 39.04%; and better stability, with an average increase of 58.02%. The comparative experimental results validate the effec-tiveness of the MD3QN algorithm.

Figure 11 illustrates the training results for Square-1 and Triangle-1. Each curve represents the mean and variance of training scores over a specified number of rounds using 5 random seeds for an environment-task combination. Due to the higher exploration of ineffective routes by the D3QN algorithm in the early exploration phase, the likelihood of finding the optimal route within a limited number of rounds is lower, resulting in poor convergence and lower final scores. Under the same exploration conditions, the MD3QN algorithm can learn a control strategy with higher stable scores, as well as better convergence effectiveness.

5.3. Performance Experiment

Using the models trained in Section 4.4, five sets of experiments are conducted by randomly adding obstacles to the environment. We use four metrics to assess the performance of the planning: Average Success Rate (ASR, the probability of successfully reaching the goal without encountering obstacles, higher is better), Average Episode Reward (AER, the average total planning score of the five-model experiments, higher is better), Average Episode Step (AES, the average total step difference of the five-model experiments, lower is better), and Average Episode Stable Reward (AESR, the average stable score of the five-model experiments, constantly non-positive, higher is better). Figure 12 illustrates one random obstacle environment and task among the ten sets of models for S1-T5.

The experimental results for square and triangular boards are presented in Table 6. Across various environments, compared to the D3QN algorithm model, the MD3QN algorithm model achieves a success rate of 100%, higher total scores, fewer total steps, and higher stability scores in all environments. In other words, its performance surpasses that of the D3QN model in all aspects. The MD3QN algorithm model demonstrates superior adaptability to paths of different lengths, different start points and destinations, and randomly distributed obstructed tube holes. For both types of tube plates, when the robot navigates through narrow areas near the edges of SG (e.g., T3-E5, S2-E5), or when the robot's walking task paths were longer (e.g., S4-E3, T5-E3), the planning models learned by MD3QN could successfully avoid obstructed holes. The performance of the MD3QN algorithm across different environments exhibits excellence and stability.

5.4. Practical Experiment

The motion of the robot is planned using Python and the underlying robot controller is written in C++ to jointly control the actual motion of the robot on the tube plate. On the square-distributed tube plate, 542 obstacles are randomly added. On each defined path, the robot's motion is planned first using the base turning planner, and then the base motion planner and the gait motion planner. The robot's movement is depicted in Figure 13.

The HIT_Spibot successfully executes the commands of the planning algorithm proposed in this paper. Following the motion sequence, the robot smoothly and steadily moves each leg and foot to the planned positions, then the toe modules secure to the tube plate, releasing the next swing-ing foot to continue the motion. Ultimately, the robot walks along the given path, reaching the target safely and successfully.

6. Conclusions

This paper proposes a motion planning method based on the MD3QN algorithm, which is primarily divided into base motion planner, base turning planner, and gait motion planner. Firstly, we utilize envelope methods to calculate the workspace of the robot's foot and base, and employ the Lagrange multiplier method to solve the base trajectory. Secondly, based on the designed base turning criteria, we establish an evaluation system using the TOPSIS method to measure the performance of the base turning planning. Thirdly, we model the gait motion planning of the robot as a Markov decision process and utilize the designed MD3QN algorithm to generate the optimal gait motion strategy. The MD3QN algorithm reduces the exploration of unreachable spaces by introducing a Mask layer and de-signing a discrete action space, thereby enhancing the convergence speed of the policy. It achieves the maximum reward value after training. The MD3QN algorithm can adapt to tube plates with different distributions and randomly obstructed tube holes, enabling the robot to smoothly traverse the given path from the initial point to the destination. Our proposed method shows promising applications in the field of nuclear power. Future research will focus on robot motion planning based on operational tasks to achieve the goal of autonomous completion of inspection tasks by the robot. This section is not mandatory but can be added to the manuscript if the discussion is unusually long or complex.