Risk-Gated Hierarchical Option Policies for Budgeted Web Navigation with Irreversible-Action Failure

1. Introduction

Web navigation has become an important benchmark for evaluating decision-making capability in interactive digital environments, particularly for reinforcement learning (RL) agents and large language model (LLM)-based systems. Unlike static question answering, web navigation requires sequential reasoning, dynamic interpretation of page content, and adaptive interaction with partially observable user interfaces. Recent studies demonstrate that language-guided agents can complete structured tasks such as form filling, information retrieval, and multi-step transactions with steadily improving accuracy [1,2]. Progress in foundation models has further strengthened the alignment between natural language instructions and executable UI actions, enabling more reliable grounding from textual commands to interface elements [3,4]. At the same time, decision models that explicitly integrate multi-cost constraints and failure-risk control into web-agent learning have recently been proposed, highlighting the growing recognition that task success alone is insufficient for reliable deployment in real-world online services [5]. Despite these advances, long-horizon navigation remains challenging because errors accumulate over time, rewards are delayed, and irreversible actions may terminate tasks before corrective strategies can be applied [6,7].

Hierarchical reinforcement learning (HRL) provides a structured approach to long-horizon decision problems. The options framework decomposes complex tasks into temporally extended sub-policies, allowing abstraction over primitive actions and improving exploration efficiency [8,9]. Manager–worker architectures further enhance stability and generalization in large-scale decision settings by separating high-level planning from low-level execution [10,11]. In web environments, hierarchical control has been applied to partition navigation into semantic subgoals such as search, filter, compare, and checkout, which reduces redundant interactions and improves sample efficiency [12,13]. However, most existing hierarchical approaches prioritize reward maximization and do not explicitly address the asymmetric risks introduced by irreversible UI operations. As a result, high-level abstractions may accelerate task completion while simultaneously amplifying the impact of unsafe subgoal activation. Safety-aware RL has developed rapidly under constrained or budgeted formulations. Constrained Markov decision processes and Lagrangian optimization enable policies to satisfy cumulative cost limits, including latency, monetary expenditure, and request quotas [14,15]. Budgeted RL incorporates cost signals directly into value estimation, allowing agents to adapt their behavior according to remaining resources [16,17]. Risk-sensitive approaches introduce measures such as value-at-risk and hazard estimation to mitigate catastrophic outcomes [18,19]. Recent web-navigation benchmarks have begun reporting cost-related metrics, including request budgets and response delays, thereby encouraging more comprehensive evaluation criteria [20]. Nevertheless, most safe RL methods manage risk either at the primitive-action level or across entire trajectories. Few studies embed safety control directly within structured sub-policies, where critical semantic decisions are made. Irreversible actions, such as submitting forms, deleting content, or confirming purchases, create failure-absorbing states from which recovery is impossible. Empirical analyses indicate that a small subset of such actions accounts for a disproportionate share of navigation failures [21,22]. Flat policies operating at the action level often fail to anticipate delayed consequences associated with these operations, particularly under tight latency or request constraints. In addition, uncertainty calibration in sequential decision systems remains insufficiently explored. Poorly calibrated risk estimates can lead either to overly conservative behavior that reduces efficiency or to unsafe execution of high-risk actions [23]. Many existing experimental studies are limited to a small number of websites or short tasks, restricting conclusions about scalability and robustness under diverse, real-world conditions.These limitations suggest the need for a hierarchical and budget-aware framework that explicitly models the risk of entering failure-absorbing states and integrates this information into subgoal selection. Embedding risk estimation at the option level allows safety considerations to influence structured behaviors rather than only primitive actions. Such integration is particularly important in web environments where semantic decisions—such as committing a purchase or confirming deletion—carry long-term consequences. A mechanism that dynamically adjusts subgoal activation according to remaining multi-cost budgets can align safety sensitivity with resource availability, thereby reducing both catastrophic failures and inefficient conservatism.

The objective of this study is to enhance safe web navigation under irreversible-action risk and constrained operational budgets. A Risk-Gated Option Critic architecture is developed to combine hierarchical option policies with constrained value updates and calibrated hazard prediction. Each option is equipped with a learned hazard estimator that predicts short-horizon failure probability, and a risk gate is introduced at the option boundary to suppress high-risk subgoal activation when the predicted hazard exceeds a budget-dependent threshold. This design embeds safety control within hierarchical abstraction, balances task performance with multi-cost compliance, and reduces the accumulation of irreversible errors over long horizons. Large-scale evaluation across diverse websites and multi-step tasks assesses irreversible-action failure rate, budget adherence, efficiency, and risk calibration. By unifying hierarchical decision structure with budget-aware risk control, the proposed framework aims to provide a principled and practically deployable solution for robust web navigation in complex, real-world digital environments.

2. Materials and Methods

2.1. Sample and Study Environment Description

The dataset included 1,200 multi-step web navigation tasks collected from 48 publicly accessible commercial and service websites, such as e-commerce platforms, booking systems, and information portals. Each task required 8–20 sequential UI actions and contained at least one irreversible operation, including form submission, account update, or payment confirmation. The selected websites differed in layout structure, dynamic content loading, and authentication requirements, ensuring environmental diversity. All experiments were conducted under fixed browser conditions with identical viewport size, standardized network latency (80–120 ms), and consistent user-agent settings to reduce environmental variation. The state representation included structured DOM features, text embeddings, historical actions, and remaining multi-cost budgets (request limits, latency allowance, and fee constraints). Tasks were divided into training (70%), validation (15%), and test (15%) subsets without overlapping page instances to assess generalization ability.

2.2. Experimental Design and Control Settings

A controlled comparative design was used to evaluate the proposed Risk-Gated hierarchical option framework. The experimental group adopted a two-level architecture composed of a high-level manager for subgoal selection and low-level option policies for UI execution, together with a hazard estimation module and budget-based gating mechanism. Three control groups were defined. The first control used a flat actor–critic policy that operated directly on primitive actions without hierarchical structure. The second control adopted a standard Option-Critic architecture without hazard prediction or risk gating. The third control applied constrained reinforcement learning with Lagrangian penalties but without option-level safety control. All models shared the same action space, reward definition, and computational settings. This design enabled assessment of the independent contribution of hierarchical abstraction and risk gating to safety and efficiency.

2.3. Measurement Methods and Quality Control

Model performance was evaluated using four indicators: irreversible-action failure rate, average number of steps to successful completion, budget adherence ratio, and calibration error of hazard prediction. Irreversible failure was defined as transition into a predefined absorbing state caused by unsafe UI operations. Budget adherence was calculated as the proportion of tasks completed without exceeding any cost constraint. Calibration quality of hazard estimation was measured using Expected Calibration Error (ECE) across probability intervals. Each configuration was trained and evaluated over five independent runs with different random seeds. Statistical comparison was conducted using paired t-tests at a significance level of 0.05. Complete action logs, cost trajectories, and hazard scores were stored for verification. Experiments were executed on identical hardware to avoid variation due to computational differences.

2.4. Data Processing and Model Formulation

Interaction logs were converted into state–option–cost transition records. Continuous cost variables were scaled to the interval [0, 1] to stabilize optimization. The constrained value update of the manager policy followed a multi-cost Bellman equation:

$$\mathrm{Q}(\mathrm{s},\mathrm{o})=\mathrm{r}(\mathrm{s},\mathrm{o})-\sum _{\mathrm{i}=1}^{\mathrm{m}}{\mathsf{\lambda }}_{\mathrm{i}}{\mathrm{c}}_{\mathrm{i}}(\mathrm{s},\mathrm{o})+\mathsf{\gamma }{\mathrm{E}}_{\mathrm{s}\text{’}}\left[\underset{\mathrm{o}\text{’}}{\max }\mathrm{Q}(\mathrm{s}\text{’},\mathrm{o}\text{’})\right],$$

where $\mathrm{r}(\mathrm{s},\mathrm{o})$ denotes task reward, ${\mathrm{c}}_{\mathrm{i}}(\mathrm{s},\mathrm{o})$represents the i-th cost term, ${\mathsf{\lambda }}_{\mathrm{i}}$ is its penalty coefficient, and $\mathsf{\gamma }$ is the discount factor.

The hazard estimator computed the probability of reaching a failure state within k steps:

$$\mathrm{h}(\mathrm{s},\mathrm{o})=\mathrm{P}\left({\mathrm{s}}_{\mathrm{t}+\mathrm{k}}\text{∈}\mathrm{F}\text{∣}{\mathrm{s}}_{\mathrm{t}}=\mathrm{s},{\mathrm{o}}_{\mathrm{t}}=\mathrm{o}\right),$$

where F represents the set of irreversible failure states. Option activation was blocked when $\mathrm{h}(\mathrm{s},\mathrm{o})$ exceeded a threshold $\mathsf{\tau }\left(\mathrm{B}\right)$, which was determined by the remaining budget B. Parameters were optimized using stochastic gradient descent with entropy regularization to maintain policy diversity.

2.5. Implementation and Reproducibility

All models were implemented in PyTorch and trained using parallel web environments. The manager and option policies employed transformer-based encoders to process DOM features and textual context. The hazard estimator shared base representations with the option network but used a separate prediction layer. Hyperparameters, including learning rate, discount factor, and penalty weights, were selected based on validation performance. Early stopping was applied according to irreversible-action failure rate to avoid overfitting. Source code, environment settings, and anonymized task templates were archived to support reproducibility and independent validation.