PSN-PPO: A Policy-Splitting Network-Based Deep Reinforcement Learning Approach for Autonomous Navigation of Unmanned Surface Vehicles

Deep reinforcement learning (DRL) offers a promising approach to autonomous navigation and collision avoidance for unmanned surface vehicles (USVs) by learning decision-making policies through interaction with maritime environments. However, existing methods often couple goal-directed navigation and obstacle avoidance within a single policy network, without explicitly distinguishing their contributions or regulating the strength of avoidance intervention. This implicit coupling may induce premature deviations from the intended course or inadequate avoidance maneuvers, reducing navigation efficiency and increasing collision risk. To address this limitation, we propose proximal policy optimization with a policy-splitting network (PSN-PPO), which restructures the PPO actor into jointly trained navigation and avoidance branches. The navigation branch generates a nominal navigation output, while the avoidance branch provides a bounded correction. A learnable risk gate modulates this correction based on encounter risk assessed over a finite prediction horizon. In the absence of obstacles or when the predicted risk falls below the activation threshold, the policy mean is determined solely by the navigation branch. A closed-loop simulation framework incorporating a three-degree-of-freedom vessel motion model is used to compare PSN-PPO with standard PPO across five scenarios: obstacle-free navigation, static-obstacle avoidance, head-on encounters, overtaking, and crossing. Both methods are trained independently in each scenario. Simulation results show that PSN-PPO improves navigation efficiency and collision-avoidance performance relative to standard PPO in the evaluated scenarios.