Reinforcement Learning-Based Tension Control for Marine Winches Under JONSWAP Irregular Wave Disturbances

1. Introduction

Marine winches constitute a fundamental part of offshore handling and lifting systems. They are used whenever cables, loads, subsea devices, or other suspended equipment must be deployed, recovered, positioned, or maintained from a vessel that is itself moving in a dynamic marine environment. The controlled movement of the winch drum determines the amount of cable deployed and therefore strongly influences the mechanical state of the suspended system. In a stationary industrial setting, tension regulation can often be treated as a relatively well-defined feedback-control problem. In offshore applications, however, the supporting vessel is continually affected by environmental excitation, and the resulting motion is transmitted through the cable to the load. The controller must therefore regulate tension while simultaneously compensating for a disturbance that is dynamic, irregular, and only partially predictable.

Wave-induced heave motion is particularly important because it acts directly along the vertical direction in which many offshore lifting operations are performed. A vessel moving upward or downward changes the effective geometry and velocity of the cable system. The resulting relative motion between the winch, rope, and suspended load can generate significant changes in tension even when the commanded winch operation is unchanged. Under irregular waves, the tension signal may contain multiple frequency components and may exhibit transient peaks, oscillations, and periods of reduced tension. Such behavior is undesirable because the mechanical system must repeatedly absorb changing loads.

The consequences of poor tension regulation extend beyond tracking error alone. Large tension peaks may increase structural loading and accelerate fatigue or mechanical wear, whereas insufficient tension can permit the rope to become slack. Slack conditions are particularly problematic because the subsequent restoration of tension can create impact-like transients. The source manuscript cites prior work showing alternating tension and slack conditions under irregular excitation and notes that repeated impacts following slack events can contribute to rope failure. These considerations establish a strong motivation for active heave compensation and robust control strategies capable of maintaining stable tension over a range of environmental and operating conditions.

Conventional PI and PID controllers remain widely attractive for this class of engineering problems. Their principal advantages are interpretability, modest computational requirements, straightforward implementation, and the possibility of tuning the controller using established control-engineering procedures. A PI controller can provide accurate tracking around a nominal operating point by responding to the instantaneous error and its accumulated history. Nevertheless, a fixed-gain PI controller is fundamentally reactive. If the sea state, payload condition, rope behavior, or other plant characteristics differ from those assumed during tuning, the controller may no longer provide the same quality of response. Feed-forward and model-based active heave compensation can reduce this limitation, but their performance depends on the quality and timing of disturbance measurements or predictions and on the adequacy of the underlying model.

Reinforcement learning provides a different perspective. Rather than explicitly prescribing the control law, an RL agent learns a policy through interaction with an environment. At each decision step, the agent observes a representation of the system state, selects an action, and receives a reward describing the quality of that action. Repeated interaction allows the policy to improve. In model-free RL, the learned policy does not require a complete analytical description of the plant dynamics, which is attractive for nonlinear systems in which accurate modeling can be difficult.

The present work focuses on PPO because it is a widely used policy-gradient method for continuous control. PPO constrains policy updates using a clipped surrogate objective, limiting excessively large changes to the policy during training. The approach is combined with LSTM recurrent memory because wave disturbances are sequential processes. A single instantaneous measurement may not contain sufficient information to characterize the current phase and recent evolution of a disturbance. A recurrent network can maintain an internal state and thereby exploit temporal dependencies in the observation sequence.

The paper uses a JONSWAP spectrum to represent irregular wave excitation. The JONSWAP formulation is appropriate for fetch-limited sea states and permits the generation of stochastic wave realizations by combining frequency components with randomized phases. In the simulation, the dominant peak frequency is randomized between episodes. The PPO-LSTM agent also receives sine and cosine representations of the dominant wave phase, giving it an implicit representation of wave-state information while avoiding a discontinuity associated with directly representing an angular phase variable.

The study is designed around three central contributions. First, it establishes a Unity ML-Agents simulation environment that couples a nonlinear drum-winch model with a JONSWAP disturbance generator. Second, it trains a PPO-LSTM controller using a graduated curriculum over an 8–20 kN setpoint range. Third, it compares the learned controller with a manually tuned PI controller over randomized episodes and evaluates not only average error but also error dispersion, high-percentile error, and slack incidence. The source manuscript describes this as the first application, within its stated scope, of PPO with LSTM recurrent memory to marine winch tension control in a JONSWAP simulation environment.

1.1. Background on Marine Winch Tension Control

The control objective in a marine winch can be expressed conceptually as maintaining the measured rope tension near a desired reference despite disturbances introduced by vessel motion and other dynamic effects. Let $\mathrm{T}$ denote the measured tension and ${\mathrm{T}}_{\mathrm{s}\mathrm{e}\mathrm{t}\mathrm{p}\mathrm{o}\mathrm{i}\mathrm{n}\mathrm{t}}$ the desired tension. The instantaneous tracking error is e_T = T_setpoint − T. A controller must choose a motor torque that causes the drum to modify rope motion in a direction that reduces this error. The problem becomes more difficult when the rope is coupled to an external moving boundary, because the same drum action can have different effects depending on the current vessel motion, rope length, and dynamic state.

Active heave compensation is therefore closely related to the broader problem of tension control. A compensation system can use measurements or predictions of vessel motion to counteract disturbances before they propagate into the load. However, the quality of such compensation depends on sensors, estimation, communication, actuation, and model accuracy. The RL formulation examined here instead attempts to learn a policy that incorporates dynamic information directly into its decision process.

1.2. Motivation for Recurrent Reinforcement Learning

The key motivation for LSTM memory is temporal dependence. If wave-induced tension variations were independent from one sampling instant to the next, historical information would have limited value. In realistic irregular-wave signals, however, consecutive observations are correlated. The current phase of a dominant component and its recent evolution can provide information about the likely near-term behavior of disturbance. The source implementation explicitly supplies $\sin \left({\mathsf{\phi }}_{\mathrm{w}\mathrm{a}\mathrm{v}\mathrm{e}}\right)$ and $\cos \left({\phi }_{wave}\right)$ and also uses recurrent hidden state. This combination allows the learned policy to condition its torque command on both instantaneous measurements and a compact representation of recent history.

This distinction is important when interpreting the expected advantage of the proposed controller. The principal hypothesis is not necessarily that RL will eliminate mean error. Instead, the hypothesis is that temporal memory and learned adaptation may produce a smoother response, reduce variability and limiting the magnitude of difficult excursions. The reported results support this interpretation: mean-error improvement is modest, whereas standard-deviation reduction is considerably larger.

1.3. Research Questions and Scope

The study can be understood through four practical questions. First, can PPO-LSTM achieve average tension-tracking performance comparable with a tuned PI controller? Second, can recurrent memory reduce the variability of tracking error under randomized JONSWAP excitation? Third, does the learned policy retain useful performance across different tension setpoints rather than specializing in one operating point? Fourth, what limitations become visible when the controller is trained through a curriculum or fine-tuned for a narrower operating range? These questions define the scope of simulation study.

The work remains a simulation study. No physical winch test bench or hardware-in-the-loop validation is reported in the source manuscript. Consequently, the numerical results should be interpreted as evidence about the behavior of the proposed policy within the stated simulation model rather than as proof of direct readiness for deployment on offshore hardware.

1.4. Structure of the Paper

· Section 1 (Introduction) presents the motivation and background of reinforcement learning (RL)-based tension control for marine winches operating under irregular wave disturbances. The section introduces the challenges associated with wave-induced vessel motion, rope-tension fluctuations, and the limitations of conventional fixed-gain PI control. It also presents the objective of research, the proposed PPO-LSTM control strategy, the use of JONSWAP irregular-wave disturbances, and the main contributions of the study.

· Section 2 (Marine Winch System and Dynamic Model) describes the simulated marine winch plant and its main physical components, including the torque-controlled drum winch, motor actuator, drum rotational dynamics, rope, and seabed-contact or submerged-payload condition. The section formulates the rope-tension dynamics using a first-order differential equation incorporating environmental stiffness, rope velocity, and tension relaxation. The relationships between drum angular velocity, vessel motion, rope velocity, motor torque, and tension are also presented, together with actuator saturation and physical operating constraints.

· Section 3 (JONSWAP Wave Disturbance and Simulation Environment) presents the modelling of irregular marine-wave excitation using the JONSWAP spectrum. The section describes the generation of stochastic wave realizations, wave-frequency and phase variations, and the coupling between wave-induced vessel motion and rope dynamics. The Unity-based simulation environment and its integration with the reinforcement learning framework are also described, including the observation, action, episode, and fault-reset mechanisms used during training and evaluation.

· Section 4 (PPO-LSTM Reinforcement Learning Control Methodology) presents the proposed reinforcement learning controller based on Proximal Policy Optimization (PPO) with Long Short-Term Memory (LSTM) recurrent dynamics. The section defines the observation vector, including tension, tension reference, tracking error, drum angular velocity, rope length, motor torque, wave-phase information, and tension rate. The continuous torque-control action, LSTM temporal representation, reward formulation, actuator constraints, and PPO training procedure are presented. The curriculum-learning strategy and randomized tension-reference range of $8\text{–}20\mathrm{k}\mathrm{N}$are also detailed.

· Section 5 (Simulation Results and Comparative Evaluation) presents the performance evaluation of the proposed PPO-LSTM controller against the conventional PI controller under randomized JONSWAP irregular-wave disturbances. The section evaluates the controllers using mean absolute tension error, standard deviation of tension error, ${95}^{\mathrm{t}\mathrm{h}}$-percentile error, and slack-event rate. The results are analyzed over the complete $8\text{–}20\mathrm{k}\mathrm{N}$operating range and within the $8\text{–}12\mathrm{k}\mathrm{N}$, $12\text{–}15\mathrm{k}\mathrm{N}$, and $15\text{–}20\mathrm{k}\mathrm{N}$sub-ranges. Attention is given to tracking consistency, high-tension operation, statistical significance, and the effects of the training strategy.

· Section 6 (Discussion) analyses the principal findings and their implications for reinforcement learning-based marine winch control. The section discusses the comparable mean tracking accuracy of PPO-LSTM and PI control, the reduction in tension-error variability achieved by the recurrent RL controller, and the improved consistency observed at higher tension levels. It further examines the limitations associated with curriculum bias, low-tension performance, catastrophic forgetting during high-setpoint fine-tuning, and slack events. The section also discusses the implications for controller robustness, safety, generalization, and potential future deployment.

· Section 7 (Conclusions) summarizes the main findings of the study and evaluates the effectiveness of PPO-LSTM tension control under JONSWAP irregular-wave disturbances. The section highlights the principal performance improvements relative to PI control, particularly the reduction in tension-error variability, while acknowledging the absence of a statistically significant improvement in mean tracking error. Finally, the section outlines future research directions, including improved setpoint randomization, domain randomization, incorporation of wave-forecast information, continual-learning methods to mitigate catastrophic forgetting, and experimental validation using a physical marine-winch test platform.

2. Materials and Methods

2.1. Simulation Environment

The simulation environment was implemented in Unity 2022.3 LTS using the Unity ML-Agents Toolkit as is shown in Figure 1. The training configuration uses the ML-Agents Python package together with PyTorch as the deep-learning backend. Unity provides a configurable environment in which the winch plant, disturbance generation, observations, actions, episode logic, and fault-reset mechanisms can be executed repeatedly. This arrangement is particularly useful for RL because a large number of interactions can be generated without exposing physical equipment to unsafe exploratory actions.

Each training or evaluation episode has a maximum duration of 2000 decision steps. With a decision period of $60\mathrm{m}\mathrm{s}$, this corresponds to approximately 120 seconds of simulated time. At the beginning of an episode, a new JONSWAP wave realization is generated and the tension setpoint is selected from the configured operating range. Consequently, the controller is exposed to different disturbance realizations and, during randomized training, different target tensions.

The simulation architecture can be divided into four functional blocks. The first block is the disturbance generator, which produces a stochastic wave realization and associated vessel-heave input. The second is the winch plant, which converts motor torque and relative rope motion into tension dynamics. The third is the observation interface, which extracts and normalizes plant and wave-state variables. The fourth is the PPO-LSTM policy, which converts the observation sequence into a continuous torque command. This command is scaled to the physical motor-torque limits before being applied to the simulated plant.

2.2. Winch Plant Model

The simulated plant represents a torque-controlled drum winch coupled through a rope to a seabed-contact or submerged-payload condition. The rope tension is represented by a first-order differential equation:

$${\displaystyle \frac{\mathrm{d}\mathrm{T}}{\mathrm{d}\mathrm{t}}}={\mathrm{k}}_{\mathrm{e}\mathrm{n}\mathrm{v}}·{\mathrm{v}}_{\mathrm{r}\mathrm{o}\mathrm{p}\mathrm{e}}-{\mathsf{\alpha }}_{\mathrm{r}\mathrm{e}\mathrm{l}\mathrm{a}\mathrm{x}}·\mathrm{T}$$

(1)

Here, $k_{env}$ represents environmental stiffness, $v_{rope}$ is the effective rope velocity, and $alph{a}_{relax}$ represents a tension-relaxation coefficient. The source model uses ${\mathrm{k}}_{\mathrm{e}\mathrm{n}\mathrm{v}}=150{\displaystyle \frac{\mathrm{k}\mathrm{N}}{\mathrm{m}}}$ and ${\mathsf{\alpha }}_{\mathrm{r}\mathrm{e}\mathrm{l}\mathrm{a}\mathrm{x}}=0.12{\mathrm{s}}^{-1}$. Rope velocity is derived from drum angular velocity together with wave-induced vessel motion.

This formulation captures three important effects at a reduced-order level. First, changes in relative rope velocity produce changes in tension through the environmental stiffness term. Second, tension is not assumed to persist indefinitely because the relaxation term causes a gradual decay. Third, the drum’s rotational state and external vessel motion jointly determines the effective rope velocity. The model therefore contains dynamic coupling between actuator input, drum motion, environmental excitation, and measured tension.

The motor is represented as a first-order torque actuator with saturation limits, while drum inertia is included through a rotational dynamics equation. Actuator saturation is important because it prevents the simulated policy from generating unrealistically large torque commands. The environment also contains a fault-reset mechanism that reinitializes an episode if tension exceeds physical limits. This mechanism prevents unstable early exploratory behavior from causing persistent numerical divergence during training.

2.3. JONSWAP Wave Disturbance Model

Ocean wave disturbances were modeled using the JONSWAP (Joint North Sea Wave Observation Project) directional wave spectrum [13], which is standard for North Sea fetch-limited sea states. The JONSWAP spectrum, $\mathrm{S}\left(\mathrm{f}\right),$ is given by:

$$\mathrm{S}\left(\mathrm{f}\right)=\left(\mathsf{\alpha }{\displaystyle \frac{{\mathrm{g}}^2}{{\left(2\mathsf{\pi }\right)}^4}}{\mathrm{f}}^5\right)·\exp \left[-\left({\displaystyle \frac{5}{4}}\right){\left({\displaystyle \frac{{\mathrm{f}}_{\mathrm{p}}}{\mathrm{f}}}\right)}^4\right]·{\mathsf{\gamma }}^{\exp \left[-{\displaystyle \frac{{\left(\mathrm{f}-{\mathrm{f}}_{\mathrm{p}}\right)}^2}{2{\mathsf{\sigma }}^2{\mathrm{f}}_{\mathrm{p}}^2}}\right]}$$

(2)

The peak enhancement factor is $\gamma =3.3$. The spectral-width parameter is $\sigma =0.07$ for frequencies at or below the peak and $\sigma =0.09$ above the peak. The normalization constant $\alpha$ is selected according to the simulation formulation, as is shown in Figure 2.

The evaluation uses a significant wave-force level of $H_s=4.0kN$, representing the moderate sea-state condition specified in the source. The spectrum is discretized into $N=40$ component frequencies between $\left[0.05,0.35\right]Hz$. Each component receives an independent random phase offset uniformly distributed over $\left[0,2\pi \right)$. The peak frequency $f_p$ is randomized between $\left[0.05,0.20\right]Hz$ at the episode level. This procedure creates different time-domain wave realizations while preserving the overall statistical structure of the selected spectrum.

The use of multiple frequency components is important because a single sinusoidal disturbance would not adequately represent the irregularity that motivates the study. By varying the phase realization and peak frequency between episodes, the simulation exposes the controller to different temporal patterns. The agent additionally receives the sine and cosine of the dominant wave phase. This representation avoids the numerical discontinuity that would occur if phase were provided directly at the $\left[0,2\pi \right)$ boundary.

for three representative peak frequencies $\mathrm{f}\mathrm{p}\in \left\{0.10,0.13,0.16\right\}\mathrm{H}\mathrm{z}$ with peak enhancement factor $\mathsf{\gamma }=3.3$. (b) Sample time-domain heave displacement realization $\mathrm{f}\mathrm{p}=0.13\mathrm{H}\mathrm{z},\mathrm{N}=40$ frequency components used as disturbance input to the winch plant model.

2.4. PPO-LSTM Agent Architecture

The controller uses PPO with LSTM recurrent memory in both actor and critic networks. The architecture contains three fully connected hidden layers of 256 units each, followed by an LSTM layer with $256$hidden units. The LSTM sequence length is $256$ steps. Because the decision interval is $60\mathrm{m}\mathrm{s}$, the sequence length corresponds to approximately 15.36 seconds of simulated history when interpreted continuously. This provides the recurrent controller with a substantial temporal window for learning relationships between wave evolution and tension response.

PPO is based on repeated policy improvement using trajectories generated by the current policy. The clipped objective restricts the ratio between the updated and previous policy probabilities to a specified interval. In the present training configuration, the clipping parameter is $0.20$ during the first phase and $0.15$ during the second phase. The lower value in the second phase reduces the permitted policy change as training moves from a specialized initial behavior toward the broader randomized operating range.

Generalized Advantage Estimation is used with lambda = 0.95, while the discount factor is gamma = 0.995. These settings determine how future rewards and multi-step temporal information contribute to the policy update. The entropy coefficient is reduced from 0.005 in Phase 1 to 0.003 in Phase 2. The learning rate is also reduced from 3.0 x 10^-4 to 1.0 x 10^-4. Batch size and buffer size are both maintained at 2048 and 20480, respectively, while observation normalization is enabled in both phases. Table 1 summarizes the key training hyperparameters.

2.5. Observation and Action Space

The agent receives nine scalar observations at each decision step. The first is the normalized current tension, followed by the normalized setpoint and normalized tracking error. The fourth observation is drum angular velocity, which provides information about actuator and mechanical motion. The fifth is deployed rope length, reflecting the changing configuration of the winch system. The sixth is the current motor torque. The seventh and eighth are the sine and cosine of the dominant wave phase. The ninth is the normalized tension rate of change. Therefore, the agent receives a nine-dimensional continuous observation vector at each decision step, as summarized in Table 2. The single continuous action output represents a normalized torque command in the range $\left[-1,+1\right],$ which is scaled to the motor torque limits before application to the plant.

2.6. Reward Function

The reward is designed to prioritize tension tracking while explicitly discouraging near-slack conditions.

$$\mathrm{r}=-\left|{\mathrm{e}}_{\mathrm{T}}\right|-{\mathrm{w}}_{\mathrm{s}\mathrm{l}\mathrm{a}\mathrm{c}\mathrm{k}}\max \left(0,{\mathrm{T}}_{\mathrm{s}\mathrm{l}\mathrm{a}\mathrm{c}\mathrm{k}}-\mathrm{T}\right),$$

(3)

where: ${\mathrm{e}}_{\mathrm{T}}={\mathrm{T}}_{\mathrm{s}\mathrm{e}\mathrm{t}\mathrm{p}\mathrm{o}\mathrm{i}\mathrm{n}\mathrm{t}}-\mathrm{T}$, ${\mathrm{T}}_{\mathrm{s}\mathrm{l}\mathrm{a}\mathrm{c}\mathrm{k}}=0.5\mathrm{k}\mathrm{N}$, and ${\mathrm{w}}_{\mathrm{s}\mathrm{l}\mathrm{a}\mathrm{c}\mathrm{k}}=8.0$. The first term imposes a continuous penalty proportional to absolute tracking error. The second term is inactive when tension is safely above the slack threshold and becomes increasingly negative as tension approaches zero.

This reward design encodes an engineering preference that cannot be expressed adequately by mean error alone. Two controllers may produce similar average tracking error while differing substantially in the frequency and depth of low-tension events. By assigning an additional penalty to low tension, the RL agent is encouraged to avoid trajectories that approach rope slack. The evaluation nevertheless uses a separate slack metric, defined as the proportion of episodes in which minimum tension falls below $1.0\mathrm{k}\mathrm{N}$, allowing the safety-related outcome to be assessed independently of the training reward.

2.7. Curriculum Training Strategy

Training was conducted in two phases using a graduated curriculum approach, as is shown in Figure 3. In Phase 1 (8 million steps), the setpoint was fixed at $8\mathrm{k}\mathrm{N}$ to allow the agent to develop basic tension-tracking behavior in low-tension conditions. In Phase 2 (5 million additional steps, initialized from Phase 1 weights), the setpoint was randomized uniformly from $\left[8,20\right]\mathrm{k}\mathrm{N}$ at each episode reset, with reduced learning rate $1.0\times {10}^{-4}$ and clip epsilon $0.15$ to prevent catastrophic forgetting of the Phase 1 policy. The Phase 2 model $\mathrm{v}{15}_{\mathrm{f}\mathrm{t}}$ was used for all reported evaluations.

An additional Phase 3 fine-tune targeting the $15-20\mathrm{k}\mathrm{N}$ subrange (4 million steps, $\mathrm{L}\mathrm{R}=5\times {10}^{-5}$) was also investigated but resulted in catastrophic forgetting at lower setpoints and was not adopted as the final model. This finding is discussed in Section 4.

covers the full $8-20\mathrm{k}\mathrm{N}$ setpoint range. Phase 2 begins at $8\mathrm{M}$ steps with reduced learning rate ($LR=1\times {10}^{-4}$). Smoothed curve: 20-point moving average.

2.8. PI Controller Reference

A discrete-time PI controller is implemented as the reference baseline. It uses the same simulated plant and the same JONSWAP disturbance conditions as the RL agent, ensuring that the comparison is performed within the same environmental framework. The PI gains are manually tuned to minimize mean tension error across the full $8-20\mathrm{k}\mathrm{N}$ operating range.

The PI controller receives only the current tension and the desired setpoint. It does not receive the explicit wave-phase information available to the RL controller. This configuration represents a conventional feedback baseline without active-heave feed-forward compensation. The comparison should therefore be interpreted as a comparison between a tuned fixed-gain feedback controller and a recurrent learning-based controller with access to additional state information.

2.9. Evaluation Protocol and Statistical Analysis

Both the PI controller and RL agent were evaluated to be over 300 independent episodes ($\mathrm{P}\mathrm{I}:300$ MaxSteps episodes; $\mathrm{R}\mathrm{L}:330$ MaxSteps episodes) with setpoints randomized uniformly from $\left[8,20\right]\mathrm{k}\mathrm{N}$ and independent JONSWAP realizations at each episode (see Figure 4).

Four principal performance metrics are considered. MeanErr measures average absolute tension-tracking error. StdErr describes the dispersion of the tracking error and is used as an indicator of control consistency. P95 represents the 95th percentile of error and therefore characterizes a difficult upper-tail portion of the observed error distribution. Slack rate is the proportion of episodes whose minimum recorded tension is below $1.0\mathrm{k}\mathrm{N}$. The results are also divided into low ($8-12\mathrm{k}\mathrm{N}$), medium ($12-15\mathrm{k}\mathrm{N}$), and high ($15-20\mathrm{k}\mathrm{N}$) setpoint ranges.

Statistical significance is assessed with the two-sided Mann–Whitney U test. This non-parametric test is appropriate for comparing two independent samples without assuming normally distributed error data. A significance threshold of $\mathsf{\alpha }=0.05$ is applied. The rank-biserial correlation coefficient is reported as an effect-size measure. This combination allows the study to distinguish between numerical improvements and improvements that meet the selected statistical significance criterion.

3. Results

3.1. Overall Performance Comparison

Across the combined $8-20\mathrm{k}\mathrm{N}$ operating range, the RL agent achieves a mean tension error of $1.724\mathrm{k}\mathrm{N}$, while the PI controller achieves $1.780\mathrm{k}\mathrm{N}$. The corresponding improvement is $3.1\mathrm{\%}$. However, the Mann–Whitney U test gives $\mathrm{p}=0.134$, so the overall mean-error difference does not reach the 0.05 significance threshold. The appropriate interpretation is therefore that the RL controller demonstrates comparable average performance rather than a statistically established improvement in mean error, as is shown in Table 3.

* Statistically significant at $p<0.05$ (two-sided Mann-Whitney U test).

3.2. Low Setpoint Range$:8-12kN$

In the $8-12\mathrm{k}\mathrm{N}$ range, the PI controller performs better in mean tracking error. The PI mean error is $1.295kN$ compared with $1.351\mathrm{k}\mathrm{N}$ for the RL controller, corresponding to a 4.3% regression for RL. The difference is statistically significant at $p=0.020$. This is the clearest operating region in which the learned controller underperforms the baseline.

It can attribute this result to curriculum bias. The initial training phase uses a fixed $8kN$ setpoint for 8 million steps. Although the second phase introduces randomized setpoints, the learned policy begins its broader training from a strongly specialized initial condition. The observed regression suggests that this initial specialization may persist and affect generalization over the remainder of the low-tension interval.

The slack data provides an additional indication of the low-setpoint limitation. The RL slack rate is $4.8\%$ ($6\mathrm{o}\mathrm{f}124\mathrm{e}\mathrm{p}\mathrm{i}\mathrm{s}\mathrm{o}\mathrm{d}\mathrm{e}\mathrm{s}$), compared with $2.1\mathrm{\%}$ ($2\mathrm{o}\mathrm{f}95$) for PI. Mean minimum tension is also lower for RL in this range, 3.89 kN versus $4.16kN$. These results suggest that the low-setpoint weakness is not restricted to average tracking error; it is also visible in the occurrence of low-tension events.

3.3. Medium Setpoint Range: $12-15kN$

In the $12-15\mathrm{k}\mathrm{N}$ range, the two controllers exhibit essentially equivalent mean tracking performance. The PI mean error is $1.736kN$ and the RL mean error is $1.726kN$, an improvement of only 0.6%. The p-value is 0.763, indicating no statistical evidence of a difference in mean error.

Despite the near-identical means, the RL controller has substantially lower error variability. The standard deviation decreases from $0.198kN$ for PI to $0.146kN$ for RL, representing a $26.3\%$ reduction. The 95th-percentile error also decreases from 2.032 to 1.946 kN, an improvement of $4.2\%$. Thus, the principal benefit in this operating region is not better central tendency but tighter concentration of the error distribution.

3.4. High Setpoint Range: 15–20 kN

At higher tension setpoints, the RL controller provides its clearest mean-performance advantage. Mean error decreases from $2.180kN$ for PI to $2.121kN$ for RL, corresponding to a 2.7% improvement. The p-value of 0.065 is close to but above the selected $0.05$ threshold, so the difference is not conventionally statistically significant under the stated test criterion.

The variance result is much stronger. Error standard deviation falls from 0.283 kN for PI to $0.172kN$ for RL, a 39.2% reduction. The P95 error decreases from 2.566 to 2.433 kN, representing a 5.2% improvement. Both results indicate that the RL controller produces a more consistent response at the higher tension levels that are particularly relevant to demanding offshore lifting conditions.

Neither controller records a slack event in the $15-20k$N range. Thus, the advantage of the RL controller at high setpoints is not primarily a reduction in slack occurrence; rather, it is reflected in reduced tracking-error variability and improved upper-tail performance.

3.5. Variance Reduction and Extreme-Condition Robustness

The strongest aggregate finding is the reduction in error variability. Across the complete $8-20\mathrm{k}$N range, standard deviation decreases from $0.439kN$ for PI to $0.363kN$ for RL, corresponding to a $17.3\%$ reduction. In the medium and high ranges, the reductions are $26.3\%$ and 39.2%, respectively. The low range shows a smaller $9.4\%$ reduction in standard deviation despite the worse mean error, as is shown in Table 4.

The overall P95 reduction is 6.4%. Because P95 focuses on the upper tail of the error distribution, the result indicates that the RL controller is less likely to produce very large tracking deviations under the most challenging realizations represented in the evaluation set. The improvement is particularly visible in the $12-15$ and $15-20\mathrm{k}\mathrm{N}$ ranges.

3.6. Slack Prevention

Table 5 summarizes the slack prevention performance, defined as episodes in which the minimum recorded tension fell below $1.0\mathrm{k}\mathrm{N}$. At the $15-20\mathrm{k}\mathrm{N}$ range, both controllers achieved zero slack events. At $12-15\mathrm{k}\mathrm{N}$, both controllers had equivalent slack rates $(~1\mathrm{\%}$). At $8-12\mathrm{k}\mathrm{N}$, the RL agent exhibited a slightly higher slack rate ($4.8\mathrm{\%}\mathrm{v}\mathrm{s}.2.1\mathrm{\%}$), consistent with the overall regression at this sub-range. The overall slack rates were 2.1% (RL) vs. $1.0\mathrm{\%}(\mathrm{P}\mathrm{I}$), a modest difference that may warrant attention in safety-critical deployments.

4. Discussion

4.1. Interpretation of the Main Result

The results support a specific interpretation of the proposed PPO-LSTM controller. The principal advantage is not a dramatic improvement in average tracking accuracy. The overall mean-error reduction is only 3.1% and is not statistically significant. Instead, the most consistent advantage is a reduction in the dispersion of the error distribution. This distinction is important because a controller that produces a slightly better average error but substantially fewer large deviations may be attractive in an offshore mechanical system.

The high-setpoint results reinforce this interpretation. In the $15-20kN$ range, mean error improves by 2.7%, but standard deviation falls by 39.2%. The P95 error also decreases by 5.2%. The difference between these metrics indicates that the learned controller is changing the shape of the error distribution more strongly than its center. In engineering terms, the controller appears to produce more repeatable behavior even when its average tracking accuracy remains close to the baseline.

4.2. Possible Role of LSTM Temporal Memory

The recurrent architecture provides a plausible explanation for the observed reduction in variability. The agent does not operate solely on a single instantaneous measurement. Its LSTM maintains an internal state that summarizes information from previous observations. Because the wave disturbance is temporally correlated, recent observations can contain useful information about the phase and trend of future tension changes. The explicit sine and cosine phase inputs further expose a representation of the dominant wave state.

The evidence in the source manuscript does not isolate the contribution of LSTM from the contribution of PPO, wave-phase observations, or other architectural choices. Therefore, the improvement should not be interpreted as a proven causal effect of LSTM memory alone. A dedicated ablation study comparing PPO without recurrence, PPO-LSTM without wave-phase inputs, and PPO-LSTM with the complete observation vector would be needed to quantify the individual contribution of each element.

4.3. Comparison with Fixed-Gain PI Control

The PI controller has an important advantage: its behavior is relatively transparent and can be understood through its feedback structure and tuned gains. It does not require a large training process, a recurrent neural network, or a simulation environment. In the low setpoint region it also achieves better mean error and lower slack incidence. These characteristics should not be overlooked when evaluating an RL controller for safety-critical applications.

The RL controller, however, has a different information structure. It receives wave-phase information and multiple plant-state variables and can form a nonlinear mapping from the observation history to torque. This flexibility can be beneficial when the relationship between disturbance, plant state, and required control action is complex. The observed reduction in error variability suggests that the policy is using information unavailable to the PI baseline in a useful way.

A fair engineering comparison should therefore distinguish between controller complexity and control performance. The present results show that PPO-LSTM can reach PI-comparable mean performance while providing lower variability. They do not demonstrate that RL is universally superior to PI. Instead, they show that a recurrent learning-based policy can be competitive within the defined simulation environment and operating range.

4.4. Curriculum Bias at Low Setpoints

The low-range regression is one of the most informative findings in the study. Training begins at a single setpoint of $8kN$ for 8 million steps. This strategy makes the initial learning problem easier, but it also creates a strong distributional bias. When the policy is later exposed to the 8–20 kN range, it must adapt from a policy that has already been shaped around one operating point.

The statistical result at $8-12kN$, where RL is 4.3% worse than PI with p = 0.020, indicates that the bias is not merely theoretical. It has a measurable effect on generalization. The most direct proposed remedy is to randomize setpoints from the beginning of training rather than maintaining a single initial target. Another possibility would be to use a shorter initial specialization phase or a curriculum in which the setpoint range expands progressively without remaining fixed for such a large fraction of total training.

4.5. Catastrophic Forgetting During Fine-Tuning

The Phase 3 experiment demonstrates a second learning-related limitation. The controller was fine-tuned for 4 million steps on the $15-20kN$ range using a learning rate of $5\times {10}^{-5}$. Although the objective was to improve a region in which the RL policy was already competitive, the specialization caused substantial degradation at $8-15kN$. This is a classic form of catastrophic forgetting: adaptation to a new distribution modifies parameters in a way that damages previously acquired behavior.

The finding has direct relevance to practical deployment. If a controller is retrained after new operating conditions are encountered, improving one operating region should not compromise established performance elsewhere. The source manuscript proposes elastic weight consolidation (EWC) and experience replay from earlier sub-ranges as possible remedies. Both approaches are consistent with the broader requirement that future RL training should preserve knowledge acquired across the operating envelope.

4.6. Simulation-to-Reality Gap

The most important deployment limitation is the difference between the simulated plant and a physical winch. The source model includes nonlinear dynamics, tension relaxation, actuator saturation, drum inertia, and wave-induced disturbance, but it does not capture every source of physical uncertainty. The manuscript specifically identifies rope-elasticity variation, sheave friction, and sensor noise as examples of omitted or simplified effects.

A policy can therefore learn behaviors that are highly effective in the simulated environment but less effective when exposed to unmodeled physical phenomena. This is the well-known simulation-to-reality gap. Domain randomization is one proposed strategy: rather than training on a single fixed model, parameters such as stiffness, friction, actuator characteristics, and measurement noise can be randomized within plausible ranges. The policy then learns behavior that is less dependent on one exact simulation configuration.

Physical validation is ultimately necessary before any safety-critical deployment. The source manuscript proposes validation on a scaled winch test bench. Such experiments would allow the controller to be evaluated under measured sensor noise, actuator delay, mechanical backlash, rope compliance, and other effects that are difficult to represent completely in a reduced-order simulation.

4.7. Safety Considerations

The increased slack rate in the low operating range deserves explicit consideration. Across all episodes, RL produces 7 slack events out of 330 compared with 3 out of 300 for PI. Although the absolute event counts are small, the difference matters because slack can have consequences beyond the instantaneous control metric. A controller intended for offshore lifting should therefore not be evaluated solely on mean error or even standard deviation. Minimum tension, peak tension, actuator saturation, rate of tension change, and recovery behavior following severe disturbances should also be monitored.

A practical RL controller would also require supervisory safety mechanisms independent of the learned policy. The simulation already contains a fault-reset mechanism when tension exceeds physical limits. Physical implementation could similarly employ hard torque limits, tension limits, emergency shutdown logic, and rule-based fallback control. Such mechanisms are especially important because neural policies can produce unexpected actions outside the training distribution.

4.8. Reproducibility and Benchmarking

The use of Unity ML-Agents provides a reproducible computational framework for repeated simulation-based experiments. The source manuscript specifies the principal plant parameters, wave-generation settings, observation variables, reward function, training phases, and major PPO-LSTM hyperparameters. These details establish a useful baseline for subsequent research.

Future benchmarking should extend this baseline through controlled ablations and repeated random seeds. For example, the statistical robustness of the reported differences could be evaluated across multiple independent training runs rather than a single final policy. Additional comparisons could include PI controllers with feed-forward compensation, model-predictive control, non-recurrent PPO, alternative recurrent architectures, and robust control methods. Such comparisons would clarify whether the observed performance is specific to the proposed architecture or represents a broader advantage of learning-based control.

4.9. Limitations of the Present Study

Several limitations should be considered when interpreting the results. First, the investigation is entirely simulation-based. Second, the JONSWAP evaluation uses a fixed significant wave-force level of 4.0 kN, even though peak frequency and phase realizations are randomized. Third, the PI baseline is a fixed-gain controller without wave feed-forward information, so the comparison does not represent every possible conventional AHC implementation. Fourth, the RL and PI evaluation sample counts differ, with 330 RL episodes and 300 PI episodes.

Fifth, the study does not report a formal ablation analysis that isolates the contribution of LSTM memory, phase information, reward shaping, or curriculum design. Sixth, the results are reported for the specified $8-20kN$ range and therefore should not be extrapolated beyond that range without additional testing. Finally, the simulation does not include all physical uncertainties that may affect hardware performance. These limitations do not invalidate the reported simulation findings, but they define the boundary within which those findings should be interpreted.

5. Conclusions

This expanded study presents a PPO-LSTM reinforcement-learning controller for marine drum-winch tension regulation under irregular JONSWAP wave disturbances. The controller is trained in a Unity ML-Agents environment containing a nonlinear winch model, stochastic wave excitation, actuator limits, a nine-dimensional observation vector, and a reward function that combines tracking accuracy with a penalty against near-slack conditions. A graduated curriculum is used to develop the policy from an initial 8 kN operating point toward randomized setpoints over 8–20 kN.

The main result is that the PPO-LSTM controller achieves mean performance comparable to the manually tuned PI baseline. Over the complete range, mean tension error is 1.724 kN for RL and 1.780 kN for PI, a 3.1% improvement that is not statistically significant at alpha = 0.05. Therefore, the study does not establish a statistically significant improvement in average tracking accuracy.

The stronger result concerns consistency. Overall error standard deviation decreases from 0.439 to 0.363 kN, corresponding to a 17.3% reduction. The reduction reaches 26.3% in the 12–15 kN range and 39.2% in the 15–20 kN range. The 95th-percentile error is reduced by 6.4% overall, with the largest improvements occurring in the medium and high setpoint ranges. These findings indicate that the RL policy can produce a more consistent tension response under the randomized wave conditions used for evaluation.

The study also demonstrates important limitations. At 8–12 kN, RL performs 4.3% worse than PI in mean error, with p = 0.020, and has a higher slack rate. The most plausible explanation identified in the source manuscript is bias introduced by the initial fixed-8 kN curriculum. Furthermore, targeted fine-tuning on 15–20 kN causes catastrophic forgetting at lower setpoints, illustrating the difficulty of improving one region of the operating envelope without degrading another.

The results support PPO-LSTM as a promising candidate for further investigation rather than as a completed replacement for conventional marine winch controllers. Its most notable advantage is enhanced consistency, particularly at higher tension setpoints, while its principal challenges include low-setpoint generalization, safe learning, and transfer from simulation to physical hardware. Future research should investigate randomized setpoint training from the beginning, domain randomization, explicit wave forecasts or Motion Reference Unit measurements, continual-learning methods such as elastic weight consolidation, and physical validation on a scaled winch test bench.

Overall, the work demonstrates the feasibility of using recurrent reinforcement learning to address marine winch tension regulation under irregular-wave excitation. The Unity ML-Agents environment and the reported PPO-LSTM benchmark provide a foundation for systematic comparison of learning-based and conventional control methods. A successful transition toward practical offshore deployment will require stronger validation, broader disturbance distributions, safety supervision, and careful treatment of uncertainty. Within the limits of the present simulation study, however, the results indicate that recurrent RL can match conventional PI control in average performance while offering a meaningful reduction in tension-error variability.