PSN-PPO: A Policy-Splitting Network-Based Deep Reinforcement Learning Approach for Autonomous Navigation of Unmanned Surface Vehicles

1. Introduction

The growing demand for autonomous marine observation, inspection, and transportation is driving the development of unmanned surface vehicles (USVs) from remotely operated platforms toward intelligent systems capable of environmental perception, path planning, and autonomous maneuvering [1]. Autonomous navigation requires USVs to pursue a destination while avoiding obstacles and respecting vessel motion constraints. Coordinating these objectives is particularly challenging in dynamic encounters, where avoidance maneuvers modify the vessel’s trajectory and thereby alter subsequent encounter conditions. Goal-directed navigation and collision avoidance are thus intrinsically coupled within a sequential decision-making process.

Conventional approaches to safe navigation commonly combine global path planning with local obstacle avoidance. Graph-search methods, such as A*, identify feasible routes from environmental maps [2]; artificial potential field methods derive local navigation directions from attractive and repulsive potentials associated with the destination and surrounding obstacles [3]; and the dynamic window approach evaluates candidate maneuvers within a dynamically admissible velocity space [4]. To coordinate global routing and local avoidance, researchers have combined improved A* with artificial potential fields [5] and nonuniform Theta* with the dynamic window approach [6]. Hybrid planning frameworks have also been extended to accommodate ocean currents [7] and dynamic obstacles [8]. Although these approaches offer explicit planning mechanisms, their performance depends on the environmental representation, trajectory evaluation criteria, and accuracy of target-vessel motion prediction [9]. These dependencies have motivated interest in deep reinforcement learning (DRL), which enables navigation policies to be learned through interaction with the environment.

In DRL-based USV navigation, an agent learns a control policy from environmental interaction and task-specific rewards, with observations typically comprising the own-vessel state, goal information, and surrounding obstacle states. Woo and Kim investigated DRL-based USV collision avoidance using collision risk information [10]. Beyond collision avoidance alone, subsequent studies have examined the joint learning of navigation and avoidance behaviors. Meyer et al. employed proximal policy optimization (PPO) to integrate path following and static obstacle avoidance within a unified learning task [11] and also investigated rule-compliant collision avoidance in dynamic vessel encounters [12]. Related work has addressed multivessel encounter decision-making [13], spatiotemporal information modeling [14], and risk-based coordination of path following and collision avoidance [15], broadening the application of DRL to complex navigation tasks.

To improve policy responses to changing encounter conditions, researchers have incorporated encounter geometry, motion constraints, and collision risk assessment into DRL-based navigation. Xue et al. integrated reciprocal velocity obstacles with PPO to exploit relative motion information for collision avoidance among multiple USVs [16]. Wu et al. combined PPO with the dynamic window approach to incorporate dynamically feasible vessel motions into collision-avoidance path planning [17]. Xie et al. augmented the policy state with parameters associated with collision avoidance and course recovery to distinguish between these phases during multivessel encounters [18]. Xu et al. developed a hybrid collision risk assessment model incorporating the distance and time to the closest point of approach and ship safety domains, together with a reward formulation accounting for route efficiency, navigational safety, and collision-avoidance rules [19]. These studies have improved information representation, local planning, risk assessment, and reward design. However, such improvements do not, by themselves, explicitly separate the contributions of goal-directed navigation and collision avoidance within the actor network. When both functions share a unified output mapping, their interaction is learned implicitly through joint optimization, without explicit structural constraints on the information used by each function, the magnitude of avoidance corrections, or the conditions under which these corrections are activated. This leaves scope for an actor architecture that explicitly separates nominal navigation from risk-dependent avoidance.

To address this gap, we propose proximal policy optimization with a policy-splitting network (PSN-PPO) for autonomous USV navigation. The policy-splitting network (PSN) restructures the PPO actor into two jointly trained branches. The navigation branch generates a nominal navigation output from the own-vessel state and goal information, while the avoidance branch produces a bounded correction from the complete observation vector. A learnable risk gate modulates this correction according to encounter risk assessed using the minimum predicted separation distance over a finite horizon. In the absence of obstacles or when the predicted risk falls below the activation threshold, the policy mean is determined solely by the navigation branch. The two branches and the gating parameter are jointly optimized using the standard PPO update procedure. We analyze the architecture’s structural properties, including gating monotonicity, reduction to the navigation branch in obstacle-free conditions, and bounded correction magnitude. We then evaluate PSN-PPO against standard PPO in a closed-loop simulation framework across five scenarios: obstacle-free navigation, static-obstacle avoidance, head-on encounters, overtaking, and crossing. Both methods are trained independently in each scenario.

The main contributions of this study are as follows:

1.

We introduce a policy-splitting actor architecture that assigns goal-directed navigation and obstacle avoidance to two branches with distinct information inputs. The branches generate a nominal navigation output and a bounded avoidance correction, explicitly separating their contributions to policy generation.

2.

We develop a learnable risk gate that modulates the avoidance correction according to encounter risk assessed over a finite prediction horizon and integrate the resulting architecture into PPO for joint optimization. We analyze gating monotonicity, reduction to the navigation branch in obstacle-free conditions, and the bound on the avoidance correction.

3.

We evaluate PSN-PPO against standard PPO using a closed-loop simulation framework incorporating a three-degree-of-freedom vessel motion model. Independent training and evaluation across five representative scenarios assess navigation efficiency, vessel–obstacle separation, and task completion performance.

The remainder of this paper is organized as follows. Section 2 presents the vessel motion model and collision risk assessment framework. Section 3 describes the PSN-PPO architecture and its implementation, including the observation space, action space, and reward function. Section 4 presents the experimental setup and simulation results and analyzes navigation and collision-avoidance performance. Section 5 concludes the paper and outlines future research directions.

2. Three-Degree-of-Freedom Vessel Dynamics and Collision Risk Assessment

This section presents the three-degree-of-freedom (3-DOF) vessel dynamics model and collision risk assessment framework used for autonomous USV navigation. The dynamics model describes vessel motion under surge thrust and yaw moment inputs, accounting for inertia, coupling among motion components, and hydrodynamic damping, and defines the state transitions in the simulation environment. Representative vessel encounter situations and the associated collision-avoidance responsibilities are then outlined to inform scenario design and the analysis of navigational behavior. Finally, ship-domain constraints are combined with relative-motion geometry to quantify encounter risk over a finite prediction horizon, providing the input to the risk gate introduced in the following section.

2.1. Three-Degree-of-Freedom Vessel Dynamics

Vessel motion generally comprises six degrees of freedom. For the calm-water navigation scenarios considered here, a horizontal-plane model retaining surge, sway, and yaw is adopted to represent the maneuvering dynamics relevant to navigation and collision avoidance [20]. The vessel is controlled through surge thrust and yaw moment. Its motion is described using an earth-fixed reference frame and a body-fixed frame attached to the vessel. The pose vector $\mathit{\eta }$ comprises the vessel’s position and heading in the earth-fixed frame, while the generalized velocity vector $\mathit{\nu }$ comprises the body-fixed surge and sway velocities, u and v, and the yaw rate $\omega$.

$$\mathit{\eta }=\left[\begin{array}{c}x\\ y\\ \psi \end{array}\right],\mathit{\nu }=\left[\begin{array}{c}u\\ v\\ \omega \end{array}\right]$$

(1)

The heading angle $\psi$ is measured counterclockwise from the positive x-axis of the earth-fixed frame. Based on the coordinate transformation between the earth-fixed and body-fixed frames, the vessel kinematics are expressed as

$$\dot{\mathit{\eta }}=\mathbf{J}\left(\psi \right)\mathit{\nu },\mathbf{J}\left(\psi \right)=\left[\begin{array}{ccc}\cos \psi & -\sin \psi & 0\\ \sin \psi & \cos \psi & 0\\ 0& 0& 1\end{array}\right]$$

(2)

Expanding Equation (2) yields the time evolution of the vessel’s position and heading:

$$\dot{x}=u\cos \psi -v\sin \psi ,\dot{y}=u\sin \psi +v\cos \psi ,\dot{\psi }=\omega$$

(3)

Building upon the kinematic relationships, the vessel dynamics are formulated to account for inertia, motion coupling, and hydrodynamic damping, thereby characterizing the velocity response to the applied control inputs:

$$\mathbf{M}\dot{\mathit{\nu }}+\mathbf{C}\left(\mathit{\nu }\right)\mathit{\nu }+\mathbf{D}\left(\mathit{\nu }\right)\mathit{\nu }=\mathit{\tau },\mathit{\tau }=\left[\begin{array}{c}F\\ 0\\ N\end{array}\right]$$

(4)

where $\mathbf{M}$ is the inertia matrix, $\mathbf{C}\left(\mathit{\nu }\right)$ is the Coriolis and centripetal matrix, and $\mathbf{D}\left(\mathit{\nu }\right)$ is the hydrodynamic damping matrix. The control inputs F and N denote the surge thrust and yaw moment, respectively. Although sway is not directly actuated, it remains coupled with surge and yaw through the vessel dynamics.

The model defines the vessel state transitions in the simulation environment. At each control interval, the policy actions are mapped to surge thrust and yaw moment, and the vessel model is integrated to update the velocities, position, and heading. These updated states are used to construct the agent’s observations, closing the action–feedback loop. The vessel’s position and velocity also enter the relative-motion calculations between the own vessel and target vessels for collision risk assessment. Figure 1 illustrates the vessel dynamics model.

2.2. Collision Risk Assessment

2.2.1. International Regulations for Preventing Collisions at Sea (COLREGs)

Maritime collision avoidance requires adequate vessel separation as well as appropriate responses to encounter situations and the associated navigational responsibilities [21]. The International Regulations for Preventing Collisions at Sea (COLREGs) prescribe responsibilities and maneuvering requirements for vessels in different encounter situations [22]. This study considers representative encounters between power-driven vessels in sight of one another, with emphasis on head-on, crossing, and overtaking situations. The relevant requirements provide a reference for scenario design and the interpretation of navigational behavior. Figure 2 illustrates the encounter situations summarized below.

Head-on encounter. Under Rule 14, when two power-driven vessels meet on reciprocal or nearly reciprocal courses with a risk of collision, both vessels must turn to starboard to achieve a port-to-port passage.

Crossing encounter (give-way vessel). Under Rules 15 and 16, when two power-driven vessels cross with a risk of collision, the vessel with the other on its starboard side must give way. As far as possible, it must act early and decisively to achieve adequate clearance and avoid passing ahead of the other vessel when circumstances permit.

Crossing encounter (stand-on vessel). Under Rule 17, the stand-on vessel must initially maintain its course and speed. It may take avoiding action on its own when the give-way vessel is evidently failing to act appropriately. When action by the give-way vessel alone can no longer prevent collision, the stand-on vessel must take whatever action would best assist in avoiding it. When acting under Rule 17(a)(ii), a power-driven stand-on vessel should, where circumstances permit, avoid a port turn for another power-driven vessel on its port side, as required by Rule 17(c).

Overtaking encounter. Under Rule 13, a vessel coming up with another from more than $22.5^{\circ }$ abaft that vessel’s beam is considered to be overtaking. The overtaking vessel must remain clear of the overtaken vessel until it has fully passed and is clear. This obligation persists despite subsequent changes in relative bearing. In the scenarios considered here, the responsibility therefore depends on whether the own vessel is overtaking or being overtaken.

These rules provide the behavioral context for the encounter scenarios. The risk model developed below quantifies geometric collision risk from relative vessel motion and uses this measure to modulate the avoidance correction. It does not explicitly encode maneuvering direction, give-way obligations, or the conditions for stand-on intervention. Accordingly, the experiments assess navigation and collision-avoidance performance in the prescribed scenarios; collision-free trajectories alone are not treated as evidence of full COLREGs compliance.

2.2.2. Ship Collision Risk Assessment Model

A ship domain defines a region around the own vessel that other vessels should remain outside, providing a geometric criterion for assessing inter-vessel separation [23]. Early studies by Fujii and Tanaka on vessel separation and by Goodwin on the statistical characterization of ship domains laid the foundation for their use in navigational safety assessment [24,25]. Here, the ship domain is represented by a circular safety region centered on the own vessel, with a fixed radius of $d_{\mathrm{safe}}=110\mathrm{m}$. This simplified representation applies the same separation threshold in all directions. In the reward formulation introduced later, incursions into this region incur a safety penalty. The adopted ship domain is illustrated in Figure 2.

The ship domain specifies a spatial safety boundary, whereas predicting whether a target vessel will enter this region requires consideration of relative vessel motion. Figure 3 illustrates the closest point of approach (CPA) geometry between the own ship (OS) and the target ship (TS). Let ${\mathit{p}}_t^s$ and ${\mathit{p}}_t^o$ denote their respective positions at time t, and let ${\mathit{v}}_t^s$ and ${\mathit{v}}_t^o$ denote the corresponding velocities, all expressed in the same earth-fixed frame. The relative position, relative velocity, and instantaneous inter-vessel distance are defined as follows:

$${\mathit{q}}_t={\mathit{p}}_t^o-{\mathit{p}}_t^s,{\mathit{w}}_t={\mathit{v}}_t^o-{\mathit{v}}_t^s,D_t={∥{\mathit{q}}_t∥}_2$$

(5)

Under a short-horizon constant-velocity extrapolation, the relative position after a time interval $\mathit{\tau }$ can be written as

$${\widehat{\mathit{q}}}_t\left(\mathit{\tau }\right)={\mathit{q}}_t+\mathit{\tau }{\mathit{w}}_t$$

(6)

The closest point of approach (CPA) in Figure 3 lies on the relative-motion trajectory of the target ship. When the relative velocity is nonzero, the time to CPA, denoted by $T_t^{\mathrm{CPA}}$, and the corresponding closest distance, denoted by $D_t^{\mathrm{CPA}}$, are given by

$$T_t^{\mathrm{CPA}}=-{\displaystyle \frac{{\mathit{q}}_t^{\top }{\mathit{w}}_t}{{∥{\mathit{w}}_t∥}_2^2}},D_t^{\mathrm{CPA}}={∥{\mathit{q}}_t+T_t^{\mathrm{CPA}}{\mathit{w}}_t∥}_2$$

(7)

The CPA may already lie in the past, or it may fall beyond the time interval relevant to the current collision-avoidance decision. To characterize near-term collision risk, the prediction time is constrained to the interval $[0,T_H]$, yielding the finite-horizon closest distance

$${\widehat{T}}_t=clip\left(-{\displaystyle \frac{{\mathit{q}}_t^{\top }{\mathit{w}}_t}{\max \left({∥{\mathit{w}}_t∥}_2^2,{\epsilon }_v\right)}},0,T_H\right),D_t^H={∥{\mathit{q}}_t+{\widehat{T}}_t{\mathit{w}}_t∥}_2$$

(8)

where $T_H=30\mathrm{s}$ is the risk prediction horizon, and ${\epsilon }_v={10}^{-8}$ is introduced to avoid numerical division by zero. For static obstacles, the obstacle velocity is set to zero. The above prediction is updated at each control interval according to the latest navigation state.

Using the safety distance of the circular ship domain as a reference, the range for activating avoidance intervention is further expanded to $d_{\mathrm{act}}=2d_{\mathrm{safe}}=220\mathrm{m}$. A normalized risk metric is then defined as

$$q_t^{\mathrm{risk}}=clip\left({\displaystyle \frac{d_{\mathrm{act}}-D_t^H}{d_{\mathrm{act}}-d_{\min }}},0,1\right)$$

(9)

where $d_{\min }=80\mathrm{m}$ denotes the risk-saturation distance. As the predicted closest distance decreases from $d_{\mathrm{act}}$ to $d_{\min }$, the risk metric increases from zero to one. In this way, both the spatial safety boundary of the ship domain and the short-term encounter trend are jointly incorporated into the collision-avoidance decision process. In the next section, this risk metric is used as the input to the PSN gating mechanism. A schematic illustration of the relative motion and collision risk assessment process is shown in Figure 3, and the corresponding vessel and risk-assessment parameters are listed in Table 1.

3. Method

This section presents PSN-PPO for autonomous USV navigation. The proposed policy-splitting network (PSN) restructures the PPO actor to generate a nominal navigation output and a bounded avoidance correction, explicitly separating their contributions to policy generation. A learnable risk gate modulates the correction according to predicted encounter risk. We first introduce the reinforcement learning formulation and then describe the PSN architecture and analyze its structural properties, including the gate’s range and monotonicity, reduction to the navigation branch in obstacle-free conditions, and bounded correction magnitude. We subsequently present the joint optimization of the two branches and the gating parameter within PPO, followed by the observation space, action space, and reward design.

3.1. Deep Reinforcement Learning

Autonomous USV navigation is formulated as a sequential decision-making problem with underlying environment dynamics represented by a Markov decision process (MDP), $\mathcal{M}=(\mathcal{S},\mathcal{A},P,R,\gamma )$ [26]. Here, $\mathcal{S}$ and $\mathcal{A}$ are the state and action spaces, respectively, P is the state transition kernel, R is the reward function, and $\gamma$ is the discount factor. At each time step t, the agent receives an observation ${\mathit{o}}_t=h\left({\mathit{s}}_t\right)$ derived from the environment state ${\mathit{s}}_t$ and samples an action from the stochastic policy ${\pi }_{\mathit{\theta }}(·\mid {\mathit{o}}_t)$. The policy parameters $\mathit{\theta }$ are optimized to maximize the expected discounted return:

$$J\left(\mathit{\theta }\right)={\mathbb{E}}_{{\pi }_{\mathit{\theta }}}\left[{\displaystyle \sum _{t=0}^{T-1}}{\gamma }^t{r}_t\right],$$

(10)

where $r_t$ is the reward at time step t and T is the episode length. The expectation is taken over trajectories induced by the policy and the environment dynamics.

For continuous control, an actor–critic architecture combines a parameterized policy with a learned value function [27]. The actor defines the action distribution ${\pi }_{\mathit{\theta }}(·\mid {\mathit{o}}_t)$, while the critic estimates the expected discounted return from its current input. This value estimate serves as a baseline for advantage estimation and subsequent policy updates.

In this study, proximal policy optimization (PPO) is used to train the policy. PPO alternates between collecting trajectories with a sampling policy and optimizing a clipped surrogate objective that discourages excessively large policy updates [28]. The objective to be maximized is

$$L_{\mathrm{PPO}}\left(\mathit{\theta }\right)={\mathbb{E}}_t\left[\min \left({\rho }_t\left(\mathit{\theta }\right){\widehat{A}}_t,clip\left({\rho }_t\left(\mathit{\theta }\right),1-ϵ,1+ϵ\right){\widehat{A}}_t\right)\right],$$

(11)

where ${\rho }_t\left(\mathit{\theta }\right)={\pi }_{\mathit{\theta }}({\mathit{b}}_t\mid {\mathit{o}}_t)/{\pi }_{{\mathit{\theta }}_{\mathrm{old}}}({\mathit{b}}_t\mid {\mathit{o}}_t)$ is the policy probability density ratio evaluated at the sampled action ${\mathit{b}}_t$, ${\widehat{A}}_t$ is the estimated advantage, and $ϵ$ is the clipping parameter. Here, ${\mathit{\theta }}_{\mathrm{old}}$ denotes the parameters of the policy used to collect the trajectories and remains fixed during the associated optimization epochs. The expectation ${\mathbb{E}}_t$ is estimated over the collected samples.

The policy likelihoods are evaluated at the unclipped action ${\mathit{b}}_t$, whereas the action passed to the environment is ${\mathit{a}}_t=clip({\mathit{b}}_t,-1,1)$, with clipping applied componentwise. Thus, the likelihood ratio is computed for the sampled policy action before the deterministic clipping operation. PSN-PPO retains this optimization procedure while restructuring the actor to distinguish nominal navigation outputs from risk-dependent avoidance corrections.

3.2. Policy-Splitting Network-Based Proximal Policy Optimization

Goal-directed navigation is required throughout a USV navigation task, whereas collision-avoidance corrections depend on the presence of obstacles and the associated encounter risk. To reflect these distinct information requirements and activation conditions, PSN-PPO restructures the actor into a navigation branch and an avoidance branch. A risk-dependent gate modulates the avoidance correction, explicitly separating its contribution from the nominal navigation output.

The observation vector is partitioned into navigation-related and obstacle-related components:

$${\mathit{o}}_t=\left[{\mathit{o}}_t^{\mathrm{nav}};{\mathit{o}}_t^{\mathrm{obs}}\right],{\mathit{o}}_t^{\mathrm{nav}}\in {\mathbb{R}}^{12},{\mathit{o}}_t^{\mathrm{obs}}\in {\mathbb{R}}^7,$$

(12)

where ${\mathit{o}}_t^{\mathrm{nav}}$ contains the own-vessel motion states and goal-relative position information, while ${\mathit{o}}_t^{\mathrm{obs}}$ contains obstacle-relative positions and velocities, obstacle types, and a presence indicator. The navigation branch uses only ${\mathit{o}}_t^{\mathrm{nav}}$ to generate a nominal output, whereas the avoidance branch uses the complete observation vector to generate a bounded correction:

$${\mathit{z}}_t^{\mathrm{nav}}=f_{{\mathit{\theta }}_n}\left({\mathit{o}}_t^{\mathrm{nav}}\right),\Delta {\mathit{z}}_t=\kappa \tanh \left(f_{{\mathit{\theta }}_a}\left({\mathit{o}}_t\right)\right),$$

(13)

where ${\mathit{\theta }}_n$ and ${\mathit{\theta }}_a$ are the parameters of the navigation and avoidance branches, respectively. The coefficient $\kappa =1.0$ bounds the absolute value of each correction component, and tanh is applied elementwise. Both branches are trained jointly under the same task objective, without separate supervision or branch-specific losses.

The normalized encounter risk $q_t^{\mathrm{risk}}\in [0,1]$, defined in Equation (9), is first transformed using a cubic smoothstep function. A learnable exponent then adjusts the gate response at intermediate risk levels:

$$\begin{array}{cc}\hfill H_t& ={\left(q_t^{\mathrm{risk}}\right)}^2\left(3-2q_t^{\mathrm{risk}}\right),{\alpha }_g=0.5+1.5sigmoid\left(\beta \right),g_t=m_t{H}_t^{{\alpha }_g},\hfill \end{array}$$

(14)

where $m_t\in \{0,1\}$ indicates obstacle presence and $\beta$ is a learnable scalar parameter. For finite $\beta$, the exponent satisfies $0.5<{\alpha }_g<2$, and initialization with $\beta =-\ln 2$ gives ${\alpha }_g=1$. Adjusting the exponent changes the gate response at intermediate risk levels while preserving its endpoint values. Specifically, $g_t=0$ when no obstacle is present or when the minimum predicted separation is at least $d_{\mathrm{act}}$. When an obstacle is present and the minimum predicted separation is at most $d_{\min }$, $g_t=1$.

The nominal output and the gated correction are combined to form the policy mean:

$${\mathit{z}}_t={\mathit{z}}_t^{\mathrm{nav}}+g_t\Delta {\mathit{z}}_t.$$

(15)

Since $0\le g_t\le 1$, the correction satisfies

$${∥g_t\Delta {\mathit{z}}_t∥}_{\infty }\le g_t\kappa \le \kappa .$$

Thus, the architecture explicitly bounds the avoidance-induced change in the policy mean. When the gate is inactive, the mean reduces to the navigation-branch output. These properties hold for fixed network parameters and concern the policy’s algebraic structure. They do not establish closed-loop stability, collision-free operation, or invariant performance in unseen environments. The correction bound applies to the policy mean; realized control actions and vessel trajectories also depend on action sampling, action clipping, vessel dynamics, and encounter conditions.

The policy is Gaussian, with mean ${\mathit{z}}_t$ and a diagonal covariance matrix. Its standard deviation is parameterized by two state-independent, trainable log-standard deviations, ${\ell }_i=\log {\sigma }_i$, initialized to ${\ell }_i=-1$ for $i\in \{1,2\}$, giving ${\sigma }_i=e^{-1}\approx 0.3679$. During training, an intermediate action is sampled and then clipped componentwise to the normalized action range:

$${\mathit{b}}_t\sim \mathcal{N}\left({\mathit{z}}_t,diag\left({\sigma }_{\mathrm{PPO}}^2\right)\right),{\mathit{a}}_t=clip\left({\mathit{b}}_t,-1,1\right).$$

(16)

The two components of ${\mathit{a}}_t$ are mapped to surge thrust and yaw moment for execution in the vessel dynamics model. During PPO updates, Gaussian log-probabilities and policy probability density ratios are evaluated at the stored, unclipped samples ${\mathit{b}}_t$. This ensures that likelihood evaluation corresponds to the sampled actions before deterministic clipping. For the trajectory evaluations reported in Section 4, actions are instead obtained deterministically as ${\mathit{a}}_t=clip({\mathit{z}}_t,-1,1)$, without sampling from the Gaussian distribution.

PSN-PPO retains the trajectory collection, advantage estimation, and clipped policy optimization procedures of standard PPO. The critic estimates the expected return from the complete observation vector, while the architectural modification is confined to the actor’s generation of the policy mean.

The gate is monotonic in the normalized encounter risk. For fixed $\beta$, with an obstacle present ($m_t=1$) and $0<q_t^{\mathrm{risk}}<1$, differentiating Equation (14) gives

$${\displaystyle \frac{\partial g_t}{\partial q_t^{\mathrm{risk}}}}=6{\alpha }_g{H}_t^{{\alpha }_g-1}{q}_t^{\mathrm{risk}}\left(1-q_t^{\mathrm{risk}}\right)\ge 0.$$

(17)

Because the normalized risk is nondecreasing as the minimum predicted separation decreases, the gate value is likewise nondecreasing with increasing encounter risk. Higher gate values permit a larger correction bound, $g_t\kappa$, although the realized correction also depends on the avoidance-branch output. The exponent ${\alpha }_g$ adjusts the response profile without changing the endpoint values.

The gate also modulates gradient propagation to the avoidance branch. For a fixed sampled observation, the gate is independent of the branch parameters, and differentiation of Equation (15) yields

$${\displaystyle \frac{\partial {\mathit{z}}_t}{\partial {\mathit{\theta }}_n}}={\displaystyle \frac{\partial {\mathit{z}}_t^{\mathrm{nav}}}{\partial {\mathit{\theta }}_n}},{\displaystyle \frac{\partial {\mathit{z}}_t}{\partial {\mathit{\theta }}_a}}=g_t{\displaystyle \frac{\partial \Delta {\mathit{z}}_t}{\partial {\mathit{\theta }}_a}}.$$

(18)

The navigation branch therefore has an ungated gradient pathway to the policy mean, whereas the corresponding pathway for the avoidance branch is scaled by $g_t$. When $g_t=0$, the current sample contributes no policy-loss gradient to the avoidance branch. The gating parameter is learned through its effect on the policy mean at intermediate risk levels. The two branches and the gate can therefore be optimized jointly using the PPO objective, without additional branch-specific loss functions.

Both methods are implemented using Stable-Baselines3 2.7.0. The standard PPO actor maps the 19-dimensional observation to a two-dimensional Gaussian mean through two fully connected hidden layers of 64 units each. PSN-PPO uses navigation and avoidance branches with 12- and 19-dimensional inputs, respectively, each containing two 64-unit hidden layers and a two-dimensional output. Both methods use a separate critic with architecture $19\to 64\to 64\to 1$, without sharing trainable layers with the actor. All hidden layers use Tanh activations. The standard PPO mean, the PSN navigation output, and the critic output are linear, while the avoidance output is bounded by the scaled Tanh transformation in Equation (13). No additional activation is applied to the combined PSN policy mean.

Including the two log-standard-deviation parameters and, for PSN-PPO, the scalar gate parameter, the actors contain 5,572 and 10,695 trainable parameters, respectively. Each critic contains 5,505 parameters. The two methods therefore share the critic architecture and optimization settings, but differ in actor architecture and parameter count. Their optimization and policy settings are summarized in Table 2. Advantages are normalized before policy updates, and each update uses at most ten optimization epochs. An update is terminated early if the approximate KL divergence exceeds $1.5$ times the target value of $0.02$. Gradient-norm clipping is applied jointly to the actor and critic parameters. Figure 4 and Algorithm 1 present the PSN-PPO architecture and training procedure, respectively.

3.3. Observation and Action Space Design

3.3.1. Observation Space Design

To provide the policy with sufficient information on vessel motion, goal-directed navigation, and obstacle proximity, a 19-dimensional observation vector is constructed. In the present experiments, each scenario contains at most one static obstacle or one dynamic target vessel. Accordingly, the observation space consists of 12 navigation-related features and 7 obstacle-related features, expressed as

$$\begin{array}{cc}\hfill {\mathit{o}}_t=[& {\overline{\mathit{p}}}_t^{\top },{\overline{\mathit{\nu }}}_t^{\top },\sin {\psi }_t,\cos {\psi }_t,{\left({\overline{\mathbf{e}}}_{G,t}^b\right)}^{\top },{\overline{d}}_t^G,\hfill \\ \hfill & \sin e_t^{\psi },\cos e_t^{\psi },{\left({\overline{\mathit{q}}}_t^b\right)}^{\top },{\overline{D}}_t,{\left({\overline{\mathit{w}}}_t^b\right)}^{\top },c_t/2,m_t{]}^{\top }\in {[-1,1]}^{19}\hfill \end{array}$$

(19)

where ${\mathit{p}}_t$, ${\mathit{\nu }}_t$, and ${\psi }_t$ denote the own-vessel position, body-fixed velocity vector, and heading angle, respectively. The terms ${\mathbf{e}}_{G,t}^b$, $d_t^G$, and $e_t^{\psi }$ represent the relative goal position expressed in the body-fixed frame, the distance to the goal, and the heading error relative to the goal direction, respectively. Similarly, ${\mathit{q}}_t^b$, $D_t$, and ${\mathit{w}}_t^b$ denote the relative position, separation distance, and relative velocity of the obstacle. The variable $c_t$ identifies the obstacle type, taking values of 1 and 2 for static obstacles and dynamic target vessels, respectively, while $m_t$ is the obstacle-presence indicator.

The overbar denotes variables normalized and clipped according to the map dimensions or the corresponding velocity scales. The first 12 components constitute the navigation observation, whereas the remaining 7 components represent obstacle-related information. In obstacle-free scenarios, the normalized obstacle-distance component is set to unity, while all other obstacle-related components are set to zero. This representation preserves a consistent observation dimension across different navigation scenarios.

3.3.2. Action Space Design

Consistent with the three-degree-of-freedom vessel dynamics, a two-dimensional continuous action space is adopted to regulate the surge thrust and yaw moment. The normalized actions generated by the policy are mapped to the corresponding control inputs in the body-fixed frame as

$${\mathit{a}}_t=\left[\begin{array}{c}{a}_t^F\\ a_t^N\end{array}\right]\in {[-1,1]}^2,{\mathit{\tau }}_t=\left[\begin{array}{c}{F}_t\\ 0\\ N_t\end{array}\right]=\left[\begin{array}{c}{F}_{\max }{a}_t^F\\ 0\\ N_{\max }{a}_t^N\end{array}\right]$$

(20)

The resulting control inputs drive changes in the vessel’s translational velocities and yaw rate through the dynamics model, thereby modifying its trajectory to accomplish goal-directed navigation and obstacle avoidance.

3.4. Reward Function Design

To balance goal-directed progress, navigational safety, and maneuvering effort, a composite reward function is formulated by integrating goal-directed progress, navigation and maneuvering penalties, collision-avoidance penalties, and terminal rewards:

$$R_t=R_t^{\mathrm{goal}}+R_t^{\mathrm{nav}}+R_t^{\mathrm{avoid}}+R_t^{\mathrm{terminal}}$$

(21)

(1) Goal-directed progress reward.

The goal-directed progress reward measures the reduction in the distance to the destination between consecutive time steps, encouraging the USV to continuously approach its target:

$$R_t^{\mathrm{goal}}=w_p{\displaystyle \frac{d_t^G-d_{t+1}^G}{L}}$$

(22)

where $d_t^G$ and $d_{t+1}^G$ denote the distances to the goal before and after action execution, respectively, L is the distance normalization scale, and $w_p$ is the corresponding reward weight. A reduction in the goal distance produces a positive reward, whereas movement away from the destination incurs a penalty.

(2) Navigation and maneuvering penalties.

The navigation and maneuvering component jointly accounts for heading deviation, traveled distance, control effort, action variations, and elapsed navigation time. These penalties are introduced to discourage unnecessary detours and excessive maneuvering:

$$\begin{array}{cc}\hfill R_t^{\mathrm{nav}}=& -w_h\left(1-\cos e_{t+1}^{\psi }\right)\Delta t-w_{\ell }{\displaystyle \frac{{∥{\mathit{p}}_{t+1}-{\mathit{p}}_t∥}_2}{L}}\hfill \\ \hfill & -w_u{∥{\mathit{a}}_t∥}_2^2\Delta t-w_{\Delta }{∥{\mathit{a}}_t-{\mathit{a}}_{t-1}∥}_2^2-w_T\Delta t\hfill \end{array}$$

(23)

where $e_{t+1}^{\psi }$ represents the heading error relative to the goal direction, ${\mathit{a}}_t$ denotes the normalized action executed by the environment, and $\Delta t$ is the control interval. The weighting coefficients regulate the relative contributions of heading deviation, traveled distance, control effort, action variation, and navigation time to the overall penalty.

(3) Collision-avoidance penalties.

The collision-avoidance component incorporates both the instantaneous safety separation and the predicted encounter risk, allowing the severity and temporal proximity of potentially hazardous encounters to jointly influence the learning process:

$$R_t^{\mathrm{avoid}}=-m_{t}C\left(D_t^{\mathrm{step}}\right)\Delta t+R_t^{\mathrm{risk}}$$

(24)

where the separation-dependent penalty function is defined as

$$C\left(D\right)=w_{s}clip\left({\displaystyle \frac{d_{\mathrm{safe}}-D}{d_{\mathrm{safe}}-d_{\min }}},0,1\right)+w_d{\left[1-{\displaystyle \frac{D}{d_{\min }}}\right]}_{+}$$

(25)

The predictive risk penalty is further formulated as

$$R_t^{\mathrm{risk}}=\left\{\begin{array}{cc}-w_r{\left[{\displaystyle 1-\frac{D_{t+1}^{\mathrm{CPA}}}{d_{\mathrm{safe}}}}\right]}_{+}^2\exp \left({\displaystyle -\frac{T_{t+1}^{\mathrm{CPA}}}{T_H}}\right)\Delta t,\hfill & {\chi }_{t+1}=1,\hfill \\ 0,\hfill & {\chi }_{t+1}=0.\hfill \end{array}\right.$$

(26)

where ${\left[z\right]}_{+}=\max (0,z)$ denotes the positive-part operator, and $D_t^{\mathrm{step}}$ represents the minimum center-to-center separation predicted under linear relative motion within the current control interval. The parameters $d_{\mathrm{safe}}$ and $d_{\min }$ determine the safety-penalty range and the high-risk proximity region, respectively. The indicator ${\chi }_{t+1}$ is set to unity when an obstacle is present, the relative motion satisfies the prediction conditions, and the time to closest point of approach satisfies $0<T_{t+1}^{\mathrm{CPA}}\le T_H$. Otherwise, ${\chi }_{t+1}=0$.

(4) Terminal rewards and penalties.

The terminal reward assigns a positive reward to successful goal arrival and imposes penalties for collision, boundary violation, and episode timeout:

$$R_t^{\mathrm{terminal}}=\left\{\begin{array}{cc}-P_{\mathrm{col}},\hfill & \mathrm{collision},\hfill \\ -P_{\mathrm{out}},\hfill & \mathrm{out}\mathrm{of}\mathrm{bounds},\hfill \\ R_G,\hfill & \mathrm{safe}\mathrm{arrival},\hfill \\ -P_{\mathrm{timeout}},\hfill & \mathrm{timeout},\hfill \\ 0,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(27)

where $R_G$ denotes the reward for successful goal arrival, while $P_{\mathrm{col}}$, $P_{\mathrm{out}}$, and $P_{\mathrm{timeout}}$ are positive penalty magnitudes associated with the corresponding failure conditions.

Together, the observation space, action space, and reward function establish the closed-loop training framework of PSN-PPO. At each control interval, the actor generates an action distribution from the current observation, and the sampled action is transformed into surge thrust and yaw moment to drive the vessel dynamics. The resulting state transitions provide updated observations and reward feedback, while the critic estimates the state value to support advantage estimation and policy optimization. Through joint optimization under a shared task return, the navigation branch, avoidance branch, and risk-gating mechanism learn to coordinate goal-directed progress, collision avoidance, and maneuvering effort.

4. Results and Discussion

PSN-PPO is evaluated against standard PPO in a Python-based vessel navigation simulator across five scenarios: obstacle-free navigation, static-obstacle avoidance, head-on encounters, overtaking, and crossing. Training reward curves, vessel trajectories, and the associated motion and control responses are analyzed to assess learning behavior, navigation efficiency, and collision-avoidance performance under different task conditions.

4.1. Simulation Environment

The simulation environment comprises a $1000\mathrm{m}\times 1000\mathrm{m}$ navigable area, with the own vessel navigating from the lower-left region toward a designated goal in the upper-right region. Vessel motion follows the horizontal-plane three-degree-of-freedom dynamics model, with surge thrust and yaw moment as control inputs. The vessel dynamics, observation and action space definitions, and reward formulation are identical across scenarios and shared by both methods.

All experiments are conducted on a workstation equipped with an AMD Ryzen 9 5950X 16-core processor operating at 3.40 GHz, an NVIDIA GeForce RTX 3090 GPU with 24 GB of video memory, and 64 GB of system RAM.

Figure 5(a–e) shows representative configurations of the five scenarios. The obstacle-free scenario evaluates goal-directed navigation, while the static-obstacle scenario examines the ability to adjust trajectories around fixed obstacles. The head-on, overtaking, and crossing scenarios evaluate avoidance behavior under different relative-motion geometries. During training, static-obstacle positions are randomized between the starting point and the goal. In the dynamic scenarios, target vessels are initialized at randomized positions and subsequently maintain their prescribed headings and constant speeds. This randomization varies the obstacle configurations and encounter geometries encountered during training.

Each method is independently trained in each scenario using five distinct random seeds, without continued training or parameter transfer between scenarios. Each training run uses a single environment and terminates after 350 completed episodes, with at most 1500 steps per episode. The resulting number of environment interactions varies with episode length. The rollout being collected when the episode limit is reached is not used for a further optimization update, and the final saved policy corresponds to the most recent completed update.

The five independent runs are used to assess the learning behavior and variability of both methods across different random initializations. For the representative trajectory comparisons in Section 4.3, the final checkpoints obtained with seed 1 are retained, rather than selecting checkpoints based on the best training performance. The comparisons assess performance within each scenario following separate training. Cross-scenario policy transfer and generalization to unseen encounter situations are outside the scope of this evaluation.

4.2. Algorithm Performance Analysis

The learning performance of PSN-PPO is compared with that of standard PPO across five navigation scenarios. Both methods adopt the optimization settings in Table 2 and the episode-based training protocol described in Section 4.1. To evaluate the influence of random initialization and stochastic training, five independent runs with different random seeds are conducted for each method in each scenario. Figure 6 presents the averaged learning curves across the five runs, with shaded regions indicating inter-seed variability. Comparisons are based on the first 350 completed episodes rather than an equal number of environment interactions.

In the obstacle-free scenario, both methods exhibit rapid reward growth and reach similarly high return levels within approximately 50 episodes, indicating that the proposed architecture preserves effective goal-directed navigation when avoidance intervention is unnecessary. In the static-obstacle scenario, PSN-PPO exhibits progressive reward improvement and achieves a performance level comparable to standard PPO toward the end of training. More pronounced advantages emerge in dynamic encounter scenarios. In head-on encounters, PSN-PPO achieves faster initial reward improvement and maintains higher mean returns over substantial portions of the early and intermediate training stages. In the overtaking scenario, PSN-PPO reaches high-return behavior earlier than standard PPO, demonstrating improved learning efficiency in coordinating obstacle avoidance and goal-directed progress. In crossing encounters, PSN-PPO maintains higher mean returns throughout much of the intermediate and later training stages, indicating more effective policy optimization under complex encounter conditions.

Overall, the five-seed comparisons demonstrate that PSN-PPO preserves the basic navigation capability of standard PPO while exhibiting more favorable learning dynamics in several collision-avoidance scenarios. The improvements are particularly reflected in faster reward growth and higher mean returns during substantial portions of training. Although inter-seed variability remains evident in challenging scenarios and the performance advantage is not uniform throughout training, the results support the effectiveness of the proposed policy-splitting architecture in improving the learning efficiency of autonomous USV navigation.

4.3. Experimental Results

This section compares the navigation and collision-avoidance behavior of the trained policies using representative test trajectories and the corresponding time histories of vessel states and control inputs. Unlike the five-seed learning comparisons in Section 4.2, the trajectory evaluations are conducted using the final checkpoints obtained with training seed 1. In each scenario, both methods are evaluated for one episode under the same fixed initial conditions, with scenario randomization disabled and the evaluation seed set to 202042. Actions are obtained deterministically from the Gaussian mean and clipped componentwise to $[-1,1]$, without stochastic sampling. The five scenarios therefore yield ten individual evaluation episodes across the two methods.

In the obstacle-free scenario, lateral deviation is measured relative to the straight line connecting the start and goal positions. In the obstacle scenarios, proximity is quantified by obstacle clearance, with zero clearance marking the physical contact boundary. This metric differs geometrically from the center-to-center separation used in the risk model: positive clearance indicates physical separation but does not establish that the 110 m safety threshold is maintained. The numerical results reported below refer to these individual fixed-scenario evaluations and characterize behavior under their specific initial conditions.

4.3.1. Obstacle-Free Scenario

Figure 7 compares the navigation performance of the two methods in the obstacle-free scenario. Both policies reach the goal, but the PSN-PPO trajectory remains closer to the straight line connecting the start and goal positions. The maximum lateral deviation decreases from $41.2\mathrm{m}$ with standard PPO to $30.8\mathrm{m}$ with PSN-PPO, a reduction of $25.2\%$. The time to reach the goal decreases from $207.0\mathrm{s}$ to $189.2\mathrm{s}$, corresponding to an $8.6\%$ reduction.

The surge-velocity profiles are consistent with the difference in travel time. PSN-PPO maintains a relatively steady cruising speed over most of the route, whereas standard PPO decelerates earlier during the final approach and continues to adjust its heading. PSN-PPO exhibits localized variations in yaw moment and sway velocity near the goal, but its heading deviation is smaller during the main transit phase.

Throughout this test, the avoidance gate remains inactive, and the PSN-PPO policy mean is determined solely by the navigation branch. The results show that this branch supports effective goal-directed navigation, with smaller lateral deviation and shorter travel time in the illustrated case. These improvements cannot be attributed to active avoidance intervention, since the gated correction is zero throughout the trajectory.

4.3.2. Static-Obstacle Scenario

Figure 8 presents a test case with a circular obstacle of radius $80\mathrm{m}$ centered at $(500,500)\mathrm{m}$. As the vessel approaches the obstacle, PSN-PPO executes an avoidance maneuver over a relatively brief interval, then gradually realigns its heading toward the goal after passing the obstacle. Standard PPO, by comparison, turns over a longer portion of the approach and takes longer to restore a goal-directed heading after clearing the obstacle. Both methods reach the goal without collision. However, the minimum obstacle clearance increases from $13.8\mathrm{m}$ with standard PPO to $37.5\mathrm{m}$ with PSN-PPO, an increase of $23.7\mathrm{m}$ relative to the physical contact boundary.

The larger minimum clearance is achieved alongside a $2.7\%$ reduction in travel distance and a $10.0\%$ reduction in travel time relative to standard PPO. The sway-velocity, heading-angle, and yaw-moment histories indicate that PSN-PPO applies larger yaw moments primarily during avoidance and subsequent heading recovery, while maintaining a higher cruising speed after obstacle passage. Standard PPO exhibits a more gradual steering response and passes the obstacle with a smaller minimum clearance. Its prolonged heading recovery and lower speed after obstacle passage are consistent with the longer route and travel time. Together, these results indicate more effective coordination of local avoidance and goal-directed navigation by PSN-PPO in the illustrated test case.

4.3.3. Classical Vessel Encounter Scenarios

Head-on encounter.

Figure 9 presents a head-on encounter in which the target vessel starts at $(700,700)\mathrm{m}$ and travels at a constant speed of $2\mathrm{m}/\mathrm{s}$ on a course reciprocal to the own vessel’s nominal course. Standard PPO deviates from the nominal route early in the episode and negotiates the encounter through gradual heading changes. PSN-PPO initially remains closer to the nominal route and makes a more pronounced starboard turn as the vessels approach. Between $50\mathrm{s}$ and $70\mathrm{s}$, its heading changes by approximately $20.5^{\circ }$ to starboard, compared with $2.2^{\circ }$ for standard PPO over the same interval. After the vessels pass, PSN-PPO gradually realigns its heading toward the goal, producing more clearly distinguishable navigation, avoidance, and heading-recovery phases.

Both policies maneuver to starboard and reach the goal. PSN-PPO increases the minimum inter-vessel clearance from $58.4\mathrm{m}$ to $93.0\mathrm{m}$, a gain of $59.3\%$, while requiring an additional $4.0\mathrm{s}$ to complete the task. In this test case, the larger clearance is therefore accompanied by a modest increase in travel time. The avoidance maneuver is also more concentrated in time, followed by a distinct return to goal-directed navigation. This pattern is consistent with the intended role of risk-dependent local corrections, although the trajectory comparison alone cannot disentangle the contributions of the dual-branch architecture and the risk gate.

Overtaking encounter.

Figure 10 presents an overtaking encounter in which the target vessel starts at $(300,300)\mathrm{m}$ and travels at a constant speed of $0.8\mathrm{m}/\mathrm{s}$ on the own vessel’s initial heading. With the target vessel initially positioned ahead at relatively close range, PSN-PPO makes an early heading adjustment to establish lateral separation before continuing toward the goal. Standard PPO also overtakes the target vessel but repeatedly decelerates and continues turning near the goal, resulting in a prolonged final approach.

PSN-PPO reduces the time to reach the goal from $265.6\mathrm{s}$ to $197.8\mathrm{s}$, a reduction of $25.5\%$, and shortens the travel distance by $20.3\%$ relative to standard PPO. The cumulative heading change decreases from $663.4^{\circ }$ to $148.9^{\circ }$, a reduction of $77.6\%$, while the minimum inter-vessel clearance increases from $77.0\mathrm{m}$ to $118.3\mathrm{m}$. These results show that, in the illustrated case, PSN-PPO achieves a larger passing clearance together with a shorter route, faster task completion, and less turning near the goal. This behavior is consistent with a more efficient return to goal-directed navigation after overtaking.

Crossing encounter.

The crossing scenario considers the own vessel in the give-way role and the target vessel in the stand-on role. The target vessel is initially located to starboard of the own vessel at $(700,260)\mathrm{m}$ and travels at a constant speed of $3.5\mathrm{m}/\mathrm{s}$ on a prescribed heading. Its path crosses the own vessel’s nominal route, requiring avoidance maneuvers while maintaining progress toward the goal.

Figure 11 compares the resulting trajectories and motion responses. Standard PPO departs substantially from the nominal route early in the episode and maintains the altered course for an extended period, leading to increasing lateral deviation. At $30\mathrm{s}$, its lateral deviation from the nominal route reaches $50.1\mathrm{m}$, compared with $6.0\mathrm{m}$ for PSN-PPO. PSN-PPO initially remains closer to the nominal route, adjusts its heading further as it approaches the encounter region, and resumes goal-directed navigation after passing the target vessel.

The standard PPO episode terminates when the own vessel exits the navigable area, with $266.8\mathrm{m}$ remaining to the goal. PSN-PPO reaches the goal at $250.4\mathrm{s}$ following further heading adjustments. Standard PPO achieves a larger minimum inter-vessel clearance of $140.4\mathrm{m}$, compared with $48.3\mathrm{m}$ for PSN-PPO, but this greater separation is accompanied by sustained route deviation and failure to complete the task. PSN-PPO maintains positive clearance and reaches the goal, demonstrating a more effective balance between collision avoidance and goal-directed progress in this test case. Nevertheless, deceleration and continued turning near the goal prolong the final approach, indicating scope for improving terminal maneuvering.

4.4. Summary of Results

The five-scenario evaluation shows that PSN-PPO retains goal-directed navigation capability while exhibiting different avoidance responses under distinct encounter conditions. In the representative test cases, it reduces early route deviations and repeated turning near the goal relative to standard PPO. In the static-obstacle and overtaking scenarios, shorter routes and travel times are accompanied by larger minimum clearances. In the head-on scenario, PSN-PPO achieves a larger minimum clearance with a modest increase in travel time. In the crossing scenario, it reaches the goal with positive clearance, whereas standard PPO maintains greater separation but leaves the navigable area before completing the task.

These results indicate improved coordination between navigation and avoidance under the evaluated conditions, rather than a uniform advantage in every metric. The five-seed training comparisons further demonstrate that PSN-PPO preserves effective learning in obstacle-free navigation while exhibiting faster reward improvement and higher mean returns over substantial training intervals in several dynamic encounter scenarios. Although inter-seed variability remains evident, the repeated training experiments provide broader evidence of the learning advantages associated with the proposed architecture. The representative trajectory evaluations further illustrate its potential to coordinate navigation efficiency and collision avoidance, while generalization across scenarios and explicit COLREGs compliance remain to be investigated.

5. Conclusions and Future Work

This study proposed PSN-PPO for autonomous USV navigation. The method restructures the PPO actor into two jointly trained branches that generate a nominal navigation output and a bounded avoidance correction. A learnable gate modulates the correction using encounter risk assessed over a finite prediction horizon. This architecture explicitly separates the two contributions to the policy mean while retaining the standard PPO update procedure. Its structural properties include a bounded, monotonic risk gate, a bounded correction, and reduction to the navigation-branch mean when the gate is inactive.

Comparisons with standard PPO were conducted in a closed-loop simulation framework incorporating three-degree-of-freedom vessel dynamics, with independent training across five navigation scenarios using five random seeds. The training results demonstrate that PSN-PPO preserves effective goal-directed navigation while achieving faster reward improvement and higher mean returns over substantial training intervals in several collision-avoidance scenarios. Representative trajectory evaluations further show improvements in navigation efficiency and obstacle clearance in the static-obstacle and overtaking scenarios, increased clearance with a modest time penalty in the head-on scenario, and successful goal arrival in the crossing case where standard PPO terminated at the navigation boundary. These findings support the potential of explicit actor decomposition and risk-dependent correction to improve the coordination of navigation and collision avoidance.

The present study is limited to separately trained scenarios, with deterministic trajectory evaluations conducted under fixed initial conditions and dynamic target vessels maintaining prescribed headings and constant speeds. Future work will investigate component ablations and parameter-matched comparisons to distinguish the contributions of the branch structure, risk gate, and network capacity. Additional evaluations across randomized encounter configurations and mixed navigation scenarios will assess policy generalization and task completion reliability, while explicit COLREGs compliance tests will examine rule-consistent behavior. Further extensions will incorporate multiple vessels, maneuvering targets, and environmental disturbances, followed by real-world USV validation under perception uncertainty and practical maneuvering constraints.