Assembly Measurement Path Planning for Mobile Robots Using an Improved Deep Reinforcement Learning

1. Introduction

The introduction should briefly place the study in a broad context and highlight why it is important. With the rapid advancement of Industry 4.0 technologies in the realm of intelligent manufacturing, assembly measurement path planning (AMPP) for complex products has increasingly evolved into a process characterized by digitalization, intelligence, and integrated smart control [1]. The accuracy requirements for AMPP associated with complex products are exceptionally rigorous, presenting a broad global measurement range that complicates precise positioning. Traditionally, the AMPP has predominantly relied on manual processes [2]. However, this conventional approach is characterized by relatively low efficiency. During the AMPP process, numerous measurements, including point application, calibration, and station transfer, must be executed manually [3]. This reliance on manual procedures ultimately results in extended assembly cycles and challenges in ensuring timely product delivery [4,5]. The AMPP is evolving towards a comprehensive measurement approach that integrates multiple technologies and couples various sensors. Mobile robots offer significant advantages, including substantial payload capacity and high precision in positioning accuracy [6,7]. When combined with the exceptional adaptability and flexibility of human operators, this synergy empowers robots to execute automatic assembly and measurement tasks effectively. This advancement enables flexible, high-precision assembly and measurement processes for complex products. Furthermore, the design of high-precision measurement trajectories is a crucial element in automating the assembly operations for large-scale, complex product sites, as it significantly contributes to ensuring overall assembly accuracy. Path planning for mobile robots is typically categorized into global path planning and local path planning [8]. The methods employed in global path planning primarily encompass graph search-based [9], random sampling-based [10], heuristic algorithm [11], and artificial intelligence [12]. Notably, the success of global planning methods heavily relies on the availability of prior information regarding the environment [13]. Due to the inherent uncertainty that robots encounter in complex environments, an increasing number of researchers are focusing on local path planning method. Local path planning involves the exploration of mobile robot pathways by gathering environmental information through sensors, assessing rewards and penalties, and subsequently determining the next action [14]. This process leads to the formulation of a relatively optimal collision-free path. Importantly, local path planning operates independently of prior knowledge about the environment and demonstrates excellent adaptability as well as real-time performance in response to dynamic conditions [15]. Planning algorithms encompass both obstacle avoidance planning algorithms and deep reinforcement learning [16,17]. In contrast to obstacle avoidance planning algorithms, deep reinforcement learning derives an optimal action strategy from the initial state to the target state through interactive iterative learning within the environment. As a result, deep reinforcement learning demonstrates superior adaptability to complex environments and exhibits outstanding performance in path planning for mobile robots [18].

Deep reinforcement learning (DRL), recognized as a pivotal technology in intelligent manufacturing, has garnered increasing attention from researchers regarding its application to the challenges of robot path planning [19,20]. This includes methodologies such as the Deep Q Network (DQN) algorithm, which is grounded in value functions [21], and the Deterministic Policy Gradient (DPG) algorithm [22]. Deep learning enables agents to perceive their environment, while reinforcement learning equips them with the capability to devise optimal strategies for addressing complex challenges. Consequently, deep reinforcement learning facilitates dynamic planning in uncertain environments by identifying pathways that optimize reward values to effectively reach target objectives. As a prominent deep reinforcement learning algorithm, DQN has been successfully applied across various domains, including energy management, workshop scheduling, and robotic obstacle avoidance [23]. It exhibits considerable efficacy in addressing path planning challenges within uncertain environments. Smart et al [24] were pioneers in applying Q-learning to the problem of robot navigation, developing a straightforward reward function to assist robots in identifying viable solutions. Low et al [25] introduced an enhanced DQN designed to tackle path planning challenges, thereby improving the convergence speed of Q-learning. However, this approach encounters difficulties in effectively adapting to complex working environments during the path planning process. Cheng et al [26] proposed a streamlined deep reinforcement learning algorithm for obstacle avoidance, leveraging a robust DQN architecture that effectively addresses both obstacle avoidance and path planning challenges faced by unmanned vessels. Lou et al [27] advanced a method that integrates DQN with prior knowledge through the design of a fuzzy logic controller, which significantly aids robots in mitigating unproductive exploration and consequently reduces training time. To solve the sequential decision-making problem of ship collision avoidance, Yang et al [28] improved DQN by sampling priority and adopting non-uniform sampling techniques, subsequently validating their proposed algorithm within a simulation environment. Nakamura et al [29] utilized DQN to obtain the control strategy for path planning on narrow roads, facilitating the robot's ability to reach its target with minimal speed variations and fewer turns in constrained environments. To achieve collection-free path planning in complex environments, Gu et al [30] applied M-DQN to address the mobile robots' path planning challenges. By decomposing the network architecture and incorporating an artificial potential field as a reward function, they attained an enhanced learning strategy. Hu et al [31] proposed a multi-agent deep deterministic policy gradient method to solve the anti-collision problem of AGVs in container terminal transportation, utilizing the Gumbel-softmax technique to discretize scenarios generated by the node network. Li et al [32] proposed a deep learning algorithm combining Angle search and DQN to achieve precise control over the search direction of mobile robots, thereby enhancing the efficiency of path planning. Wu et al [33] proposed a path planning method for multi-robot systems based on distributed multi-agent deep reinforcement learning, allowing for local optimal path planning that take into account the behaviors of other robots. To address issues related to low solution efficiency and blind exploration in robot path planning, Li et al [34] proposed an adaptive Q-learning algorithm based on virtual target guidance. This method incorporates a dual memory mechanism and an adaptive greed factor, effectively improving the convergence speed of the algorithm.

To enable robots to make adaptive decisions in complex environments, DRL brings new potential to robot path planning. Hua et al [35] present an advanced hierarchical motion planning strategy that synergistically combines DRL with collaborative design of motion optimization, aiming to achieve efficient and seamless navigation in unfamiliar environments. Zhao et al [36] introduced an advanced hierarchical path planning framework that intricately integrates global path planning with local dynamic obstacle avoidance, leveraging the power of deep reinforcement learning alongside the Grey Wolf optimization algorithm. This innovative approach aims to facilitate efficient and secure navigation for robots operating within unpredictable environments. Dawood et al [37] employed multi-agent reinforcement learning (MARL) to address the intricacies of behavior-based cooperative navigation for mobile robots, successfully achieving a paradigm of safe and efficient collaborative movement. Jonnarth et al [38] explored the potential of continuous spatial reinforcement learning in the realm of coverage path planning, empowering agents to adeptly adapt to distinct environmental characteristics. Jiao et al [39] introduced a deep reinforcement learning algorithm featuring automatic entropy adjustment tailored for shearing, aimed at enhancing the efficacy of strategy acquisition in mobile robots by effectively mitigating the evaluation of entropy.

DRL algorithms have achieved remarkable advancements in the realm of path planning, but it still harbors certain shortcomings. The intricacies of reward mechanism design are significantly reliant on expert prior knowledge and necessitate a delicate balance among various objective criteria, including path length, safety considerations, and energy efficiency [40,41]. Furthermore, the opaque nature of the DRL algorithm engenders a lack of interpretability in its decision-making processes, rendering it challenging to furnish formal guarantees regarding security [42]. Throughout the training process, the convergence performance remains unstable, and the agent is prone to collisions with obstacles. When the next action taken by the agent relies solely on observed environmental data, the DRL path planning algorithm demonstrates insufficient generalization capabilities [43]. To address these issues regarding slow convergence rates and inadequate generalization capabilities in DRL, we introduce N-step priority strategy and double Q-learning techniques. This work aims to propose an N-step priority double DQN learning algorithm (NDDQN), wherein the learning strategy leverages not only the current state of the agent but also its target location. By selecting appropriate actions grounded in this comprehensive understanding, we guide the robot in its pursuit of optimal path.

In comparison with the existing studies, this paper makes the following contributions:

A NDDQN algorithm with N-step priority experience replay and double-Q learning is proposed. This method effectively addresses the challenges of sluggish convergence and the overestimation bias commonly encountered in DRL for mobile robot path planning. By mitigating maximum operational deviations through a double DQN, expediting the acquisition of experiences via prioritized sampling, and employing N-step updates to adeptly rewards forward, this method ensures not only an accelerated convergence rate but also enhanced stability in the learning process.

We have adeptly refined the reward function and experience replay strategy, significantly alleviating the challenge of sluggish learning rates encountered in DRL algorithms. The estimation of rewards is conducted utilizing the double DQN, which proficiently mitigates overestimation issues and enhances both the accuracy and stability of Q value estimations. Moreover, prioritized experience replay amplifies the weight priority assigned to stored experiences, thereby facilitating a more efficient exploitation of data resources.

We meticulously crafted an abstract discrete grid environment alongside a highly sophisticated continuous dynamic simulation framework to address the challenges posed by mobile robot model adaptation in both dynamic and static settings. Our approach has facilitated remarkably efficient adaptive path planning for mobile robots, which has been rigorously validated with respect to its generalization capabilities and robustness.

The rest of this paper is organised as follows: Section 2 is problem description and modeling. Section 3 introduces the improved deep reinforcement learning algorithm. Section 4 presents the experimental verification and analysis. Section 5 summarizes the content of the study and outlines areas for future research.

2. Problem Description and Modeling

2.1. Problem Description

The assembly measurement process for complex products is conducted by a mobile measurement robot equipped with a scanner, which automatically scans the workpiece along a predetermined trajectory. Taking the assembly measurement process of an aircraft tube section as an example, the simplified schematic model is illustrated in Figure 1. The measurement robot utilizes the scanner to perform localized scanning operations, ensuring that the scanning path runs parallel to the axis of the large-diameter cylindrical product. Upon completing the scanning process for one measuring unit and transitioning to the next, the robot executes a horizontal movement to facilitate this transfer operation.

The measurement task encompasses five distinct measurement units: S1 and S2 are conical surfaces, while S3 is a cylindrical surface. Additionally, S4 and S5 consist of symmetrically arranged circular planes. The measurement area encompasses overlapping regions that enhance image stitching capabilities, thereby improving overall scanning accuracy. The measurement robot conducts shape scanning of each component along a predetermined trajectory path and subsequently acquires point cloud data. The measurement robot and the calibration target are positioned within the visible range of the tracker, which plans its movement trajectory based on an optimized scanning path. The primary function of the tracker is to ascertain the spatial position of the measuring robot, thereby facilitating an indirect determination of the scanner's spatial position. Scanning data is subsequently transferred into a global coordinate system via the laser tracker's measurement coordinate system, effectively achieving unification within the spatial measurement domain. The object under examination undergoes a comprehensive scanning process. It is crucial for the tracker to maintain proximity to this object in order to ensure precise measurements.

2.2. Dynamic Modeling for Mobile Robots

The mobile measurement robot is primarily utilized for the precise scanning of assembly dimensions in AMPP. This system comprises omnidirectional mobile automated guided vehicles (AGVs), industrial robotic arms, distance sensors, visual guidance systems, and end support devices. Equipped with five ultrasonic distance sensors, the mobile measurement robot can effectively determine its directional orientation. The angle between each direction measured by these ultrasonic sensors is set at 45 degrees. The kinematic structure of the robot, along with its integrated sensors, is depicted in Figure 2. In this illustration, XOY represents the global coordinate system, while X'OY' denotes the local coordinate system of the robot. The term $W_p$ refers to the distance between the left and right wheels. When a robot performs a turn, the point around which it rotates is referred to as the instantaneous curvature center $R_c$. $\theta$ represents the angle between the center of the robot and the global coordinate system. The position of the robot can be expressed using global coordinates $(R_x,R_y)$ and its directional orientation.

The mobile robot's movement is facilitated through a differential drive system. Consider that the left front wheel and the left rear wheel of the robot operate at an identical linear velocity, denoted as $v_l$, while the right front wheel and the right rear wheel maintain a consistent linear velocity represented as $v_r$. The forward linear velocity $v$ and the turning angular velocity $\omega$ of the robot are as shown in Formula (1).

$$\left\{\begin{array}{c}v={\displaystyle \frac{v_r+v_l}{2}}\\ \omega ={\displaystyle \frac{v_r-v_l}{2}}\end{array}\right.$$

(1)

When the robot turns, the instantaneous curvature radius $R_r$between the robot center and the curvature center is calculated as Formula (2):

$$R_r={\displaystyle \frac{\omega \left(v_r-v_l\right)}{2\left(v_r+v_l\right)}}$$

(2)

Generally, the motion data directly read by mobile robots is based on the local coordinate system. The motion data in the global coordinate system is obtained through Euler transformation, and the formula is using Formula (3).

$$\left[\begin{array}{c}{R}_x\\ R_y\\ \theta \end{array}\right]=\left[\begin{array}{ccc}\mathrm{c}\mathrm{o}\mathrm{s}\left(\mathsf{\omega }dt\right)& -\mathrm{s}\mathrm{i}\mathrm{n}\left(\omega dt\right)& 0\\ \mathrm{s}\mathrm{i}\mathrm{n}\left(\mathsf{\omega }dt\right)& \mathrm{c}\mathrm{o}\mathrm{s}\left(\omega dt\right)& 0\\ 0& 0& 1\end{array}\right]\left[\begin{array}{c}{R}_{cx}\\ R_{cy}\\ \theta \end{array}\right]$$

(3)

In light of the premise of uniform motion, the trajectory generation and assessment for the mobile robot will be directly derived from the sequence of actions undertaken, thereby obviating the need for explicit integration of left and right wheel velocities in computational processes. This comprehensive model is meticulously crafted to furnish a robust theoretical foundation for evaluative indicators while simultaneously establishing interfaces that anticipate future investigative endeavors.

3. Improved Deep Reinforcement Learning Algorithm

The path planning model for the mobile robot presented above constitutes a complex nonlinear optimization problem. The uncertainty associated with the number of inflection points in the robot's trajectory introduces variability in the resultant path. Deep reinforcement learning algorithms facilitate path optimization for mobile robots operating within uncertain environments by leveraging neural network learning capabilities. Consequently, this study addresses the path planning model relevant to robotic assembly measurement through enhancements in deep learning algorithms.

Figure 3 illustrates the framework predicated on the NDDQN algorithm. We have devised a path planning model for mobile robots utilizing NDDQN, framing the path planning conundrum as a Markov decision process. Subsequently, the mobile robot acquires environmental data through an array of sensors and meticulously calculates both the direction and distance to obstacles as well as target objects. The mobile robot selects action values from the replay buffer by employing prioritized experience replay while simultaneously referencing the reward function. These action values and states are then processed through layers of neural networks, culminating in the output of Q values aimed at minimizing loss functions, which are subsequently fed back into the neural network for parameter updates. Upon reaching its designated endpoint, it assesses whether convergence has been achieved. If convergence is confirmed, the program concludes; otherwise, it persists in generating new target endpoints and engaging with its environment until termination occurs. Ultimately, this interaction enables the mobile robot to gather training data and refine its learning based on sampled information.

3.1. Markov Process

The path planning problem for mobile robot installation and testing can be regarded as a Markov process. In each time step, the robot agent will choose an action to change the environmental state, which is represented as a five-tuple $\left(\mathit{S},\mathit{A},p,r,\gamma \right)$. The environment returns the next state $s_{t+1}$ and reward $r$ of the robot agent, and the robot agent selects the next action $a_{t+1}$ based on the reward. Under the time step $t$, the state of the environment is $s_t$, the agent selects action $a_t$, and the environment provides the next state $s_{t+1}$, the next reward $r_{n+1}$ and the discount factor $\gamma$, where the discount factor is used for the discount reward, and $\gamma \in \left[\mathrm{0,1}\right]$. The set of all states is the state space $S=\left\{s_1,s_2,\dots ,s_n\right\}$, and the set of all actions is the action space $\mathit{A}=\left\{a_1,a_2,\dots ,a_n\right\}$. The probability distribution of each state behavior is strategy $\pi$. The robot agent selects action $a$ through strategy $\pi$, and $\pi \left(a|s\right)$ represents the probability of choosing action $\mathrm{a}$ in state $s$. The transfer function $p\left(s^{\text{'}}|s,a\right)$ represents the probability of obtaining the next state $s^{\text{'}}$ under state $s$ and behavior $a$, and $r\left(s,a\right)$ represents the reward for choosing behavior $\mathrm{a}$ under state $s$, as Formula (4).

$$r\left(s,a\right)=\mathbb{E}\left[r_{t+1}\left|s_t\right.,a_t\right]={\sum }_{r}r{\sum }_{s^{\text{'}}}p\left(s^{\text{'}}|s,a\right)$$

(4)

The cumulative rewards of the robot agent are as follows.

$$G_t={\sum }_{k=0}^{\infty }{r}_t^k{r}_{t+k+1}$$

(5)

where the discount factor of step k is $r_t^k={\prod }_{i=1}^k{\gamma }_{t+i}$, and the goal of reinforcement learning is to maximize the cumulative reward $G_t$.

The value estimation of reinforcement learning is defined as a value function, which is divided into state value functions and action value functions. The state value function represents the expected reward of the strategy in the state, and the action value function represents the expected reward of the strategy taking actions in the state, as shown in Formulas (6) and (7).

$$v_{\pi }\left(s\right)\approx \mathbb{E}\left[G_t\left|S_t\right.=s\right]={\mathbb{E}}_{\pi }\left[{\sum }_{k=0}^{\infty }{r}_t^k{r}_{t+k+1}\left|s_t\right.\right]=\sum \pi \left(a|s\right)\sum p\left(s^{\text{'}}|s,a\right)\left[r_t+{\gamma }_t{v}_{\pi }\left(s^{\text{'}}\right)\right]$$

(6)

$$q_{\pi }\left(s,a\right)\approx \mathbb{E}\left[G_t\left|S_t\right.=s,A_t=a\right]={\mathbb{E}}_{\pi }\left[{\sum }_{k=0}^{\infty }{r}_t^k{r}_{t+k+1}\left|s_t\right.,a_t\right]$$

(7)

The relationship between $v_{\pi }$ and $q_{\pi }$ can be expressed as $v_{\pi }=\underset{a\in A}{\mathit{max}}{q}_{\pi }\left(s,a\right)$. The optimal strategy ${\pi }^{*}\left(s\right)$ selects the action $a$ with the maximum action value for each state $s$, as shown in Formula (8).

$${\pi }^{*}\left(s\right)=\arg \underset{a\in A}{\max }{v}_{\pi }\left(s^{\text{'}}|s,a\right)=\arg \underset{a\in A}{\max }{q}_{\pi }\left(s,a\right)$$

(8)

The optimal state value and the optimal action value can be calculated as

$$v^{*}\left(s\right)=\underset{a}{\max }{\sum }_{s^{\text{'}}}p\left(s^{\text{'}}|s,a\right)\left[r+{\gamma }_t{v}^{*}\left(s^{\text{'}}\right)\right]$$

(9)

$$q^{*}\left(s,a\right)=\max \mathbb{E}\left[r_{t+1}+{\gamma }_t{v}^{*}\left(s^{\text{'}}\right)\left|s_t\right.,a_t\right]$$

(10)

Therefore, the goal of reinforcement learning can be transformed into solving the optimal state value $v^{*}\left(s\right)$ and action value $q^{*}\left(s,a\right)$.

3.2. Double-Q Learning

When the state space or action space is excessively large, accurately estimating the value of each action becomes a challenging task. Consequently, the incorporation of deep neural networks to approximate the action value function has given rise to what is known as deep reinforcement learning. In the context of deep Q-networks, the optimization objective is to progressively minimize the discrepancy between the estimated return value and the predicted action value. This involves updating network weight $w$ through loss function optimization, as illustrated in Formula (11).

$$L\left(w\right)={\mathbb{E}}_{\pi }\left[{\left(r+\gamma \underset{a\text{'}}{\max }\left(s^{\text{'}},a^{\text{'}};w\right)-q_{\pi }\left(s,a;w\right)\right)}^2\right]$$

(11)

In the process of enhancing DQN, the introduction of a target network serves to improve stability. In DQN's update mechanism, the neural network utilized for estimating rewards is identical to that used for action value estimation. Consequently, it leads to fluctuations in the estimated values as parameters vary. The updating of DQN's action values can easily result in an overestimation bias. To address this issue of overestimation, Double Q-learning employs a strategy where actions are selected using the evaluation network while utilizing the target network for reward estimation. Accordingly, the loss function is represented as shown in Formula (12).

$$L\left(w\right)={\mathbb{E}}_{\pi }\left[{\left(r+\gamma \underset{a^{\text{'}}}{\max }{q}_{w_t}\left(s^{\text{'}},\mathrm{arg\; max}{q}_w\left(s^{\text{'}},a^{\text{'}}\right)\right)-q_w\left(s,a\right)\right)}^2\right]$$

(12)

where $q_{w_t}$ is the target network.

3.3. Priority Experience Reply

Experience replay serves as a storage mechanism that retains actions, behaviors, and rewards within a discrete time step. During the training process, samples are drawn from the stored experiences based on specific criteria, after which the network is updated utilizing these sampled experiences. This paper employs prioritized experience replay (PER) to enhance the weight priority of the stored experiences. In DQN, the temporal difference (TD) error represents the discrepancy between the target value and the current value. A larger TD error significantly influences backpropagation. Conversely, a smaller TD error results in reduced influence of that sample on gradient descent updates. Refining this concept further, prioritized experience replay can expedite the convergence speed of the network. This work introduces SumTree to store samples in priority experience replay [44]. All empirical samples are stored at the bottom leaf nodes, where the leaf nodes preserve the sample priorities. The top of the SumTree is the sum of the experiences of all samples. Then, the loss function of the sample priorities becomes formula (13).

$$L\left(w\right)={\mathbb{E}}_{\pi }\left[w_j{\left(r+\gamma \underset{\mathit{a\text{'}}}{\max }{q}_{w_t}\left(s\text{'},\mathrm{arg\; max}{q}_w\left(s\text{'},a\text{'}\right)\right)-q_w\left(s,a\right)\right)}^2\right]$$

(13)

where ${\mathrm{w}}_{\mathrm{j}}$ is the priority weight of the jth sample. After performing gradient update on the Q network, the TD error needs to be recalculated and the SumTree updated.

3.4. Priority N-step Update Strategy

The above-mentioned method is a one-step update method, which only uses the information of the current time step and the next time step, that is

$$G_t=r_{t+1}+\gamma v_t\left(s_{t+1}\right)$$

(14)

To accelerate the learning process, an N-step update strategy is adopted and multi-step information is used for updates. The return value of step N is expressed as

$$G_t^{\left(n\right)}=r_{t+1}+\gamma r_{t+2}+\dots +{\gamma }^{n-1}{r}_{t+n}+{\gamma }^n{v}_{t+n-1}\left(s_{t+n}\right)$$

(15)

Then, the loss function is as formula (16).

$$L\left(w\right)={\mathbb{E}}_{\pi }\left[w_j{\left(r_t^n+{\gamma }_t^n\underset{a^{\text{'}}}{\max }{q}_{w_t}\left(s_{t+n},\mathrm{arg\; max}{q}_w\left(s_{t+n},a^{\text{'}}\right)\right)-q_w\left(s,a\right)\right)}^2\right]$$

(16)

By adopting the above-mentioned NDDQN algorithm, taking the current position, movement information and current speed of the mobile robot as the input states, the movement direction of the mobile robot is regulated by this algorithm.

4. Experimental Verification and Analysis

4.1. Exoerimental Environment

To verify the effectiveness of the proposed NDDQN algorithm, comparative experiments were conducted in two training environments, namely the virtual Grid Map environment and the Gazebo simulation environment. The Grid Map experiment builds a GUI environment model on the built-in standard library of PyCharm. The experiment adopted a 1m×1m grid map, where black squares represent obstacles, blank areas are passable zones, the lower left corner is the starting point, the upper right corner is the target point, and the black lines throughout the map represent the grid. The developed grid environments are 10×10, 20×20 and 30×30 respectively. The robot agent is located on the map by the coordinates of the two diagonal points.

The Gazebo simulation environment similar to the real-world environment is: an environment with static obstacles (E1); An environment containing both static and dynamic obstacles (E2). The Gazebo robot serves as a mobile measurement platform for testing in a simulated environment. The environment E1 has 13 fixed rectangular obstacles, with white objects serving as static obstacles. The layout of environment E2 is more complex. Two blue cylinders are randomly placed in the environment as moving obstacles. Six target points (P1 to P6) are placed in the unobstructed area within the environment.

The total number of training rounds of the simulation experiment is set to 1000 episodes. When the agent reaches the target or collides, the current round terminates. The training is completed when the total reward of the agent reaches a stable and satisfactory level. The computer environment is 2.1GHz Intel Core i7-12700, 128GB of memory, and the operating system is windows11. Various path planning algorithms are implemented using Python 3.12 and invoke the built-in interfaces. TensorFlow is used to define neural network architectures, enabling models to gain GPU-accelerated advantages during the training process. In all the simulation experiments, the same robot model parameters were set. The mobile robot is trained by different algorithms, and then experiments are conducted using the empirical dataset generated from the training to verify the performance of the proposed algorithms. Table 1 presents the network hyperparameters of the NDDQN algorithm, detailing both network and training parameters. The fully connected layers of the network are managed substantial amounts of data, with the first layer containing 256 units, the second with 256 units, and the third with 128 units. This study employs a maximum epoch limit of 1000 across all training algorithms and establishes a maximum exploration step threshold of 800 for continuous learning as well as algorithm feasibility evaluation. The mobile robot learning rate is fixed at 0.005. To facilitate efficient training and enhance the learning rate of the NDDQN algorithm, we set the minimum batch learning size to 64 and the discount factor to 0.95. The proposed algorithm network updates occur at a 0.001 update rate between the current and target networks. The optimal distance that a mobile robot should maintain while in motion is set at 0.5 meters. Comparative simulation experiments were conducted respectively for the NDDQN algorithm, the proximal policy optimization (PPO) algorithm [45] and the DQN algorithm [24] in different scenarios, and the convergence was analyzed.

4.2. Deep Learning Parameter Settings

The settings of deep reinforcement learning include states, goals, behaviors and reward functions. The status is determined by the information obtained by the ultrasonic sensor. The target includes the distance $d$ from the current position of the robot to the target point, as well as the relative direction $\theta \text{'}$ of the target point relative to the robot. The calculation formula is as follows.

$$d=\sqrt{{\left(R_x-T_x\right)}^2+{\left(R_y-T_y\right)}^2}$$

(17)

$${\theta }^{\text{'}}=\left\{\begin{array}{c}\mathrm{arc\; tan}{\displaystyle \frac{T_x-R_x}{T_y-R_y}}-\theta \left(T_x-R_x\right)\ge 0\\ \mathrm{arc\; tan}{\displaystyle \frac{T_x-R_x}{T_y-R_y}}+\pi -\theta \left(T_x-R_x\right)<0\end{array}\right.$$

(18)

where $\left(R_x,R_y\right)$ and $\left(T_x,T_y\right)$ are the coordinates of the robot and the target point relative to the global coordinate system.

For the behavior of mobile robots, it is manifested as five actions: forward movement, left turn, right turn, quick left turn and quick right turn. The forward linear velocity $v$and the turning angular velocity $\omega$ corresponding to the action are shown in Table 2. Each action lasts for 1 second. After the agent completes the current action, it will re-read the status and target information and proceed to the next action.

When setting the reward function, the reward depends on the distance of the obstacle, the collision with the obstacle, and the comprehensive score of reaching the target point. The reward function includes a negative reward $r_1$ for collision, a reward $r_2$ for reaching the target point, and a positive reward $r_3$ for maintaining a safe distance. To prevent the agent from colliding with the surrounding environment during movement, when the robot agent collides, it will receive a negative reward of $r_1=-100$. To encourage agents to move towards the target point, the closer the agent is to the target, the higher the reward will be. When the agent successfully reaches the target position, a positive reward of $r_2=100$ will be directly given. When the agent approaches the collision obstacle, the states of the robot agent are divided into the safe distance state $S$ and the non-safe distance state $NS$. To reduce the possibility of colliding with obstacles, the definition of the safety distance reward is as follows.

$$r_3=\left\{\begin{array}{c}5,S\to S\\ -5S\to NS\\ 10,NS\to S\end{array}\right.$$

(19)

Therefore, when the agent reaches the target point, the comprehensive reward function is as formula (20).

$$r=r_1+r_2+r_3$$

(20)

4.3. Simulation Results Analysis in Grid Map

To verify the effectiveness of the proposed NDDQN algorithm in solving the path planning problem, simulation experiments were conducted using grid obstacles of different scales to verify the effectiveness of the proposed algorithm in solving the path planning problem. To achieve the simulation scenarios of small-scale, medium-scale and large-scale examples, grid simulation examples of three scenarios, namely 10×10, 20×20 and 30×30, are constructed. In this paper, four parameters, namely the total substitution value of the path, the maximum completion time, the total length of the path and the total number of turns of the path, are adopted as the evaluation indicators for solving the quality of the path. After 800 generations of training, the convergence of the models trained by NDDQN, DQN and PPO was compared. The comparison of the average rewards of the three algorithms is shown in Figure 4, Figure 5 and Figure 6. In small-scale simulation examples, the convergence speeds of the three algorithms do not differ much and basically achieve convergence after 100 generations of training. In the medium-scale simulation results, NDDQN has a relatively fast convergence speed and a high reward value. The three algorithms reach convergence around the 130th generation. In the large-scale simulation results, the iteration speed of NDDQN is significantly accelerated. It can avoid obstacles and reach the target point more quickly, and the number of training generations required to reach the target is smaller than that of the other two algorithms. With the increase of environmental complexity, the number of convergence generations required by the agent has a continuous increasing trend, and the final reward value is also increasing.

To compare the stability of the models trained by the three methods, 50 tests were conducted on the trained models under the same conditions. The average results are shown in Table 3. It can be seen from the training results of the three scenarios that the arrival time, total path length, turning times and average curvature of NDDQN are significantly lower than those of the other two algorithms, and the total reward values are all higher than those of DQN and PPO algorithms. The powerful learning ability of NDDQN is more prominent in complex scenarios, reaching the destination with the fewest turns and the highest reward value. The paths selected by the three algorithms when successfully reaching the target point are shown in Figure 7, Figure 8 and Figure 9. Among them, the blue line represents the NDDQN planning path, the yellow line represents the DQN planning path, and the red line represents the PPO planning path. It can be seen from the simulation paths that the paths for NDDQN and DQN to reach the target point are similar, and the number of turns required to reach the target is relatively small. Although the path planned by the DQN algorithm is similar to that of the NDDQN algorithm, the path length is longer, so the time is longer. The path planning obstacle avoidance ability of the PPO algorithm is more prominent. However, during the exploration process, there are more turns, resulting in a larger total path length and an increase in the time to reach the destination accordingly. To sum up, the experimental simulation results demonstrate the effectiveness of the NDDQN algorithm, which can help mobile robots obtain more effective obstacle avoidance experience in the task of measurement path planning.

4.4. Simulation Results Analysis in Gazebo

In the E1 and E2 test environments, there are six selected navigation points (P) and two moving obstacle points (D). The robot's task is to independently explore the environment, traverse from position P1 to P6, and then return to the initial position S without colliding with obstacles. To verify the effectiveness of the proposed NDDQN algorithm, we conducted path planning simulations in two Gazebo simulation environments. In scenario E1, the robot achieved path exploration from position P1 to P6 and returned to the target position S, as shown in Figure 10. In scenario E2, the robot completed the path exploration from position P1 to P6, avoided two dynamic obstacles, and successfully returned to the target position S, as shown in Figure 11.

In the static environment, the robot meticulously devises a smooth and nearly optimal trajectory from S to P6 and back, exemplifying the algorithm's remarkable efficiency in navigating obstacle-free pathways. As illustrated in Figure 10, the robot's trajectory is direct and streamlined, epitomizing a goal-oriented path characterized by minimal redundancy. In stark contrast, the dynamic environment presents two mobile obstacles, as depicted in Figure 11. The generated path clearly reveals noticeable detours and recalibration points where the robot adeptly anticipated and circumvented collisions prior to advancing towards its target and returning to S. These detours are far from arbitrary. They represent strategic avoidance maneuvers executed with precision to uphold a safe distance from the projected trajectories of dynamic obstacles. This vividly underscores the algorithm's robustness and real-time responsiveness when confronted with dynamic impediments. The comparative analysis conclusively demonstrates that while the algorithm excels at identifying optimal paths within static contexts, its paramount strength lies in its predictive avoidance capabilities amidst dynamic scenarios.

The ability of the cumulative reward response algorithm path planning for robot agents. To more intuitively reflect the characteristics of each agent reward, Figure 12, Figure 13 and Figure 14 present the total rewards of the three algorithms in the E1 environment. As shown in Figure 14, the NDDQN algorithm is significantly superior to other algorithms. The reward converging to 2600 is significantly higher than 1000 for PPO and 1700 for DQN. Similarly, in the more complex E2 environment, the performance gap is even greater. The superior performance of NDDQN in both environments proves that it can learn the optimal strategy more effectively. The simplicity of PPO limits its performance. Although DQN performs well, it is still surpassed by the efficient learning ability of NDDQN.

Table 4 compares the success rates and convergence times of the PPO, DQN, and NDDQN algorithms used for path planning in two scenarios. For the E1 scenario, the success rate of the NDDQN algorithm is 34.4% higher than that of the DQN algorithm and 58.5% higher than that of PPO. Table 4 simultaneously compares the convergence speeds of the three algorithms. The shortest time required for NDDQN to obtain the final target point is 114 minutes, while PPO and DQN take 248 and 153 minutes respectively. For the E2 scenario, the introduction of dynamic obstacles leads to a decline in the performance of all algorithms. The success rate of NDDQN was 60.2% and 39.4% higher than that of PPO and DQN, respectively. NDDQN converges the fastest and completes the path task in 142 minutes. This performance outperforms the DQN algorithm, which takes 193 minutes to converge, and is faster than PPO, which takes 286 minutes. These simulation results prove that NDDQN performs superior in both dynamic and static environments, demonstrating a faster convergence speed and learning efficiency.

4.5. Ablation Study

To validate the efficacy of each integrated strategy employed in this study and its contribution to overall performance, we conducted a series of ablation experiments focusing on double Q-learning (DQL), priority experience replay (PER), and N-step returns. The objective of these experiments is to quantitatively assess the impact of omitting any given strategy on the algorithm's performance, thereby elucidating its essentiality. The experimentation was carried out within the Gazebo environment, allowing us to compare five distinct configurations: 1) Our model: NDDQN; 2) -N-step: excluding n-step returns while relying solely on single-step TD errors; 3) -PER: eliminating PER in favor of uniform random sampling; 4) -DQL: removing DQL to revert to the target value computation method utilized by traditional DQN; and 5) Benchmark DQN: employing the standard DQN algorithm as a baseline for performance comparison. All models were configured with an identical set of hyperparameters, ensuring consistency across tests. Each configuration underwent independent execution for ten iterations to garner statistically significant results. In assessing asymptotic performance, we elected to utilize average final returns. Sample efficiency and learning speed were measured by the number of training steps required to achieve predetermined performance thresholds. Finally, stability and robustness were quantified through evaluating the standard deviation across multiple return runs.

The experimental results are shown in Table 5. All strategies are key contributors to the overall performance of the proposed algorithm, but their main aspects of influence are different. The N-step model has the most significant damage to the convergence speed of the algorithm. The number of training steps required to reach the threshold is approximately 45% higher than that of the NDDQN model, indicating that the multi-step reward mechanism plays a core role in accelerating temporal credit allocation and policy optimization. The -PER shows the most severe degradation in terms of final performance and stability, with the lowest average return and the largest standard deviation. This highlights the irreplaceability of PER in enhancing sample efficiency and avoiding training stagnation by prioritizing the replay of high-learning-value experiences. Although -DQL can achieve a final performance close to that of NDDQN model, its stability significantly decreases, verifying the crucial role of -DQL in alleviating Q value overestimation and ensuring smooth convergence in the learning process. Ultimately, the NDDQN model significantly outperformed all ablation variants and benchmark models in all evaluation dimensions. This not only demonstrated the effectiveness of each component but also indicated a positive synergy among them, collectively forming an efficient, stable, and robust reinforcement learning agent.

5. Conclusions

This study introduces a path planning algorithm founded on deep reinforcement learning, specifically designed to address the complexities associated with mobile robot assembly and measurement path planning within the intricate processes of aerospace product assembly and measurement. Firstly, the path planning problem pertaining to the installation and testing phases of the mobile robot is defined through model simplification, followed by dynamic modeling of the mobile robot. Secondly, to enhance the convergence speed of deep learning algorithms, Double DQN is employed for evaluating network selection actions. The implementation of reward estimation using a target network effectively mitigates issues related to overestimation. To further expedite network convergence, SumTree is introduced for storing samples within prioritized experience replay. Additionally, an N-step update strategy is incorporated to strengthen the robot's obstacle avoidance capabilities of the robot while enhancing its learning rate by updating using multi-step information. Finally, grid-based obstacle avoidance simulation experiments are conducted across three grid map environments, 10×10, 20×20, and 30×30. These experiments aim to compare and analyze the path planning performance of various algorithms. We conducted simulations of robot path planning capabilities under the influence of both static and dynamic obstacles within the Gazebo simulation environment. The results indicate that, compared to others path planning algorithms, the proposed NDDQN algorithm exhibits superior solution efficiency and stability in path planning tasks. Importantly, its path optimization capability becomes particularly pronounced in complex environments. Future research will focus on tackling complex path planning problems involving multiple robots and numerous target points while analyzing the generalization capability of robotic collision avoidance against diverse types of obstacles to further enhance adaptive path planning abilities. To ultimately realize the goal of deployment, future endeavors must encompass the meticulous performance profiling of representative robotic computing platforms. Furthermore, it is imperative to devise innovative optimization strategies aimed at mitigating computational costs.