A Survey on Hint-Based RLVR: Overcoming Zero-Advantage Failures with External Textual Signals

1. Introduction

Reinforcement Learning with Verifiable Rewards (RLVR) [1] has emerged as a central paradigm for post-training large language models (LLMs) [2], with group-relative advantage methods such as Group Relative Policy Optimization (GRPO) [3] achieving strong results on mathematical reasoning [4], code generation [5], and agentic reasoning [6]. Despite their simplicity and effectiveness, these methods rely on a stringent condition: sampled rollouts within each training query must exhibit sufficient reward variation to define a meaningful update direction. Otherwise, when all rollouts in a group receive identical rewards, the advantage collapses to zero and the policy-gradient signal vanishes [7,8]. Consequently, zero-advantage groups incur both computational and data waste: they consume rollout budget without producing gradients, while filtering them squanders valuable verifiable training questions, especially hard questions that are most valuable for learning.

Recent work has approached this bottleneck from two directions. One refines reward signals within sampled trajectories, converting sparse outcome-level rewards into finer-grained feedback [9,10]; such process-reward methods have been systematically surveyed elsewhere [11]. The other line intervenes in rollout-group construction itself to obtain higher-quality responses that restore learnable contrasts. Within this line, methods divide further by how such rollouts are obtained. The first searches more aggressively within the current policy’s own distribution, using methods such as resampling [12] or tree-structured search [13]. The second introduces external information beyond the model’s own distribution, such as reference trajectories, abstract knowledge, or accumulated experience. The former can only sharpen the policy’s existing distribution, whereas the latter expands it, directly addressing a well-recognized ceiling of on-policy RLVR [14]. Therefore, recent RLVR research has increasingly turned to externally guided branch, which is the focus of our survey.

In this work, we use Hint as a unifying term for these external textual signals that reshape rollout-group construction. We systematically survey hint-based RL methods, examining how such hints are constructed and exploited to restore learnable contrasts on otherwise zero-advantage groups. We organize these methods into two principal branches. Sample-level hints are tailored to a single training query, providing precise per-problem guidance that does not transfer across questions. We further divide them into two types: (1) Trajectory-based guidance [15,16] supplies complete or partial reference solutions; it provides direct signals on hard problems but constrains exploration. (2) Scaffold-based guidance [17,18] instead supplies higher-level abstractions, constraints, or feedback; it preserves exploratory autonomy, but is harder to construct and may be insufficient to push through the hardest problems. Task-level hints, by contrast, are stored in shared bases and retrieved across questions, turning accumulated experience into reusable priors. Among them, static bases [19] are fixed before training, offering stability and predictability, whereas evolving bases [20] are updated during training to adapt to the changing policy.

In summary, our contributions are as follows:

• We present the first survey of hint-based RL, consolidating works that introduce external textual signals into rollout-group construction to overcome zero-advantage failure.
• We propose a taxonomy of hints across two granularity levels, covering sample-level (trajectory-based and scaffold-based) and task-level (static and evolving experience bases).
• We further review domain-specific instantiations and discuss the boundaries, open problems, and future directions of hint-based RL.

2. Preliminary

We take GRPO as the primary focus of analysis, since it and its variants serve as the base algorithm for many recent hint-based RL methods. Our discussion also applies to policy-gradient methods whose updates depend on reward contrast among sampled responses.

Given a prompt x, the current policy ${\pi }_{\theta }$ samples G rollouts ${\left\{y_i\right\}}_{i=1}^G$, each scored by a verifier as $r_i=R(x,y_i)$. GRPO uses the group-relative advantage $A_i=(r_i-{\mu }_G)/({\sigma }_G+ϵ)$, where ${\mu }_G$ and ${\sigma }_G$ are the within-group mean and standard deviation of rewards, and $ϵ$ is a small constant. The advantage $A_i$ then weights each rollout’s token-level contribution to the clipped policy-gradient update. When all rewards in a group are identical, ${\sigma }_G=0$ and all advantages collapse to zero, causing the policy-gradient signal to vanish. This zero-advantage failure typically manifests as all-failure groups on hard samples or all-success groups on saturated ones. In both cases the sample yields no effective update.

We define Hint-based RL as a class of post-training methods that guide RL training with hints, explicit textual signals carrying task-relevant information beyond what ordinary rollouts under the current policy can reliably produce. By supplying this, hints make otherwise uninformative samples more likely to yield useful updates. Figure A1 presents our taxonomy. Sample-level hints provide guidance for individual training samples, while task-level hints maintain reusable guidance across samples through persistent experience repositories. The following sections elaborate these two branches.

3. Sample-Level Hints

Sample-level hints are the foundation of Hint-based RL. Each hint is exclusive to a single training sample and not shared across samples. Figure 1 illustrates how sample-level hints are constructed and utilized. Hints can originate from offline human annotations, verifiable text, or text selected and processed from the policy or teacher model during training. Their utilization falls into four categories, spanning all stages of the rollout. We further divide hints by content refinement into trajectories and scaffolds, and organize the taxonomy around the basic utilization and construction mechanisms.

3.1. Trajectory-Based Hints

Trajectory-based hints take the form of policy-like responses, typically full solution trajectories or their prefixes. They expose concrete reasoning paths that the current policy may fail to sample reliably, making informative states more reachable and giving low-variance samples new rollout variations. We organize them by where the trajectory signal enters the RL training process.

3.1.1. Trajectory Injection

Trajectory Injection places a trajectory segment in the input context while the policy still generates a full response. We distinguish these methods by the source of the injected segment.

Reference Prefix

These methods use early steps from verified reference trajectories to reduce the effective difficulty of hard training queries. The main design choice is how to calibrate the prefix length so that the augmented query remains challenging enough to yield a useful learning signal. QuestA [21], POPE [8], CCL [22], and SEELE [23] calibrate offline through pass-rate filtering, shortest-useful-prefix search, threshold-based prefix growth, and an accuracy-conditioned hint model targeting half-correct groups, respectively. GHPO [24] and BREAD [25] calibrate online, triggering reference-prefix resampling when the original rollout group has no or too few successful responses; GHPO uses pass-rate-proportional checkpoints, while BREAD locates episode-level truncation points.

Self-Replay Prefix

These methods construct prefixes from the policy’s own current or historical rollouts, keeping the hint close to the policy distribution. Most methods reuse successful trajectories to bring difficult samples back into a learnable range. HiPO [26] draws successful prefixes from the current batch to replace zero-variance groups with hinted groups, PROS [27] stores historical correct rollouts and truncates them at uncertain positions to expand the training pool, and RPO [28] maintains per-sample historical responses as a refreshable prefix cache. Failure-Prefix [29] takes the opposite direction, extracting rare incorrect prefixes from saturated samples to make over-easy queries informative again.

Hybrid Prefix

These methods lie between the previous two sources, using online trajectories from another policy solving the same query. For example, CORE [30] lets multiple policies sample independently, extracts answer-stripped reasoning context from a successful policy, and provides it to failed policies for resampling.

3.1.2. Trajectory Continuation

Trajectory Continuation uses a trajectory prefix as a fixed generation context and trains the policy to complete the suffix. The main design variable is how much prefix to reveal during training.

Scheduled Prefix Decay

These methods decay the given prefix under a preset curriculum, moving from strong continuation support toward prefix-free RL. UFT [31] and Prefix-RFT [16] use cosine decay, with Prefix-RFT keeping most rollouts hint-free. EvoCoT [32] uses a discrete step curriculum, removing trailing reasoning steps and refreshing the reference with the updated policy.

Adaptive Prefix Control

These methods set prefix length from online estimates of policy competence. G²RPO-A [33] uses recent reward trends to adjust the next prefix length, while ADHint [34] estimates query difficulty from naive rollouts, assigns longer prefixes to harder queries, and reweights advantages by posterior difficulty. Other methods adapt by rollout success under candidate prefixes. Hint-GRPO [35] increases the hint ratio until a correct group appears, BHA [36] maintains bucket-level hint ratios over verified teacher trajectories, and TRAPO [37] lengthens the expert prefix within a group when early hint-free rollouts have low average reward.

Intra-group Prefix Mixing

This branch contrasts different prefix strengths within the same rollout group. StepHint [38] segments a reference trajectory into reasoning steps, mixes continuations from different step boundaries with hint-free rollouts, and clips negative advantages on failed continuations to protect correct prefixes.

3.1.3. Trajectory Replacement

Trajectory Replacement modifies the rollout group with trajectory-level samples before the RL update. It changes the candidate set by inserting reliable trajectories or repairing failed ones.

Unconditional Augmentation

These methods add a verified reference trajectory to every rollout group as a positive anchor. ANCHOR [7] and LUFFY [15] prepare such trajectories offline for each training sample and mix them with on-policy rollouts during training.

Triggered Substitution

These methods substitute trajectories only when the original group is uninformative. AMPO [39], S-GRPO [40], and HAPO [41] use external reference trajectories under different triggers, replacing failed or low-reward rollouts in all-fail or low-confidence groups. HiPO [26] instead derives the replacement from the current policy, using successful self-hinted groups to replace zero-advantage groups in the batch.

Reconstructive Repair

These methods repair failed trajectories by keeping the valid prefix and regenerating the suffix. SCOPE [42] uses a PRM to locate the first erroneous step, preserves the preceding on-policy prefix, and lets a teacher generate the corrective suffix.

3.1.4. Trajectory Optimization

Trajectory Optimization converts trajectory hints into auxiliary update signals. ExPO [43] generates a self-explanation trajectory conditioned on the ground-truth answer and uses it as a low-weight SFT target during GRPO training. MENTOR [44] samples rollouts from a mixture of the policy and an expert model, lets the expert intervene more heavily at high-uncertainty token positions, and assigns positive advantage only when these rollouts outperform the on-policy baseline.

3.2. Scaffold-Based Hints

Scaffold-based hints are high-level textual guidance expressed as abstractions, constraints, or feedback. Compared with trajectory-based hints, they are better suited for steering how the policy approaches a problem while keeping multiple valid reasoning paths open.

3.2.1. Scaffold Injection

Scaffold Injection places a scaffold in the input prompt as an additional condition for rollout sampling, guiding the policy toward responses that better satisfy the intended target. We divide it into four types according to the guidance form.

Answer-Level Scaffold

These methods use the ground-truth answer as input guidance. By conditioning generation on the target answer, they turn open-ended problem solving into answer-conditioned reasoning, making it easier for the policy to sample reasoning paths that support the correct outcome. RAVR [45] and CoVRL [46] apply this guidance to each query, whereas HDPO [47] triggers it only for all-fail rollout groups and retains the correct answer-conditioned trajectories.

Solution-Blueprint Scaffold

These methods inject a blueprint to clarify the key idea needed for solving the problem. Some methods precompute the blueprint offline and inject it during training when needed. Guide-GRPO [48] uses teacher-generated high-level guidance for all-fail groups, AVATAR [49] uses precomputed strategy hints when exploration on a hard sample stagnates, and PieceHint [50] selects high-value reasoning pieces from reference trajectories and gradually withdraws them during training. Other methods learn or dynamically generate the blueprint. RLAD [51] and A2D [52] train scaffold generators to produce abstractions and sub-questions, while SAGE_scaf1 [53] uses the policy model as a self-hinter and increases the hint level until a correct response is sampled.

Knowledge-Level Scaffold

These methods introduce problem-specific meta-knowledge needed for solving the problem, such as relevant theorems, minimal concepts, or techniques. Both KnowRL [18] and NuRL [54] construct such scaffolds by distilling reference trajectories into compact, non-leaking knowledge cues. KnowRL further filters candidate knowledge points and selects a minimal compatible subset through offline evaluation, while NuRL obtains the cue by compressing an answer-conditioned reference trajectory into abstract knowledge guidance.

Format-Level Scaffold

These methods inject formatting specifications before rollout sampling. MeRF [55] appends a natural-language reward specification to the prompt, while RuscaRL [56] converts high-quality responses into checklist-style rubrics, varies the proportion of injected rubric across rollouts, and gradually reduces the scaffold during training.

3.2.2. Scaffold Replacement

Scaffold Replacement uses scaffolds to generate replacement responses for failed trajectories, offering more flexible and diverse forms of intervention than trajectory replacement.

Pedagogical Guidance

This branch can be viewed as a replacement-oriented extension of scaffold injection. HINT [57] uses teacher-generated heuristic hints, Scaf-GRPO [17] uses hierarchical hints ranging from knowledge to solution steps, and KEPO [58] uses answer-conditioned teacher reasoning hints. The hinted trajectories then replace failed rollouts or enter the rollout buffer. HiLL [59] further trains a hinter to generate concise pedagogical hints from the current failed rollout and reference solution, using the selected hint to replace the original all-fail group with a hinted rollout group.

Critique-Driven Refinement

These methods treat the initial rollout as a diagnostic target. The scaffold specifies what went wrong or what remains missing, then conditions a regeneration step whose output replaces the failed response or the erroneous suffix. LTE [60] instantiates the lightest form, converting wrong final answers from an all-fail group into negative hints that condition the regeneration. RGR-GRPO [61] and GOLF [62] use richer feedback, with the former refining the current best rollout according to unmet rubric criteria and the latter aggregating failed attempts with external critiques into group-level feedback. Beyond reshaping feedback, R³L [63] makes the repair local, diagnosing the error point and regenerating only the suffix from there. ECHO [64] further learns the critic itself, using the reward gain of refined trajectories to train both the policy and the feedback generator.

Action-Level Guidance

This branch moves replacement to the finer-grained action level. InfoFlow [65] uses pathfinding hints as search queries, replacing a stalled search action with a more informative query so that the agent can continue the rollout from a better retrieved state.

3.2.3. Scaffold Optimization

Scaffold Optimization keeps the policy scaffold-free at inference, converting behavior generated under scaffold conditions into auxiliary training signals for the unguided policy.

Auxiliary Supervision

These methods convert scaffold-derived signals into auxiliary targets. MEL [66] learns verified meta-experience from success–failure contrasts, while ThinkTuning [67] injects teacher reflection tokens into selected student rollouts and updates them with advantage-aware shaping. RLTF [68] uses text feedback for critique prediction and feedback-conditioned self-distillation. KEPO [58] distills only positive-reward trajectories.

Distribution Alignment

These methods use scaffold-guided generation to define a distributional target for the unguided policy. HDPO [47] builds an answer-conditioned teacher distribution and transfers it to the question-only policy through Jensen–Shannon divergence. RAVR [45] and CoVRL [46] align the question-only distribution with answer-guided reference behavior through KL regularization.

4. Task-Level Hints

Task-level hints extend sample-specific hints into a cross-sample hint base, enabling experience reuse across samples during training. As such hints are inherently cross-sample, they are also termed Experience. As shown in Figure 2, their distinctive feature over sample-level hints lies in the differentiated construction mechanisms of the experience base. We branch primarily by whether the base is updated during training, and further subdivide by utilization method and content.

4.1. Static Experience Base

Static experience base methods construct the experience base before target RL training and keep it fixed during policy optimization. During training, they retrieve relevant experience by task, domain, or similarity and use it as an additional condition for the current training sample.

Demonstration-Based Experience

These methods use cross-sample problems and solutions as few-shot hints. CBRL [69] retrieves relevant demonstrations by task tags and anneals the injection probability to zero during training. ICPO [70] uses demonstrations to induce an expert-conditioned trajectory from the current policy; when the trajectory passes the verifier, it randomly replaces one ordinary trajectory in the rollout group.

Skill-based Experience

These methods use reusable skills or strategy templates as hints. TemplateRL [71] searches successful paths with MCTS on seed problems, removes problem-specific solution content, keeps the action sequence, and retrieves nearby templates by problem complexity. SKILL0 [19] selects skill files by the performance gap on matched validation tasks with and without the skill, and gradually reduces the skill context until the policy can solve tasks without it.

4.2. Evolving Experience Base

Evolving experience base methods add, modify, remove, or reconstruct experience entries during RL training, allowing hints to evolve with policy optimization. Compared with static experience bases, they adapt to the changing capability boundary of the policy and preserve the usefulness of experience hints.

4.2.1. Accumulative Experience Base

Accumulative experience base methods continuously add new experience from training interactions into the experience base, ensuring that newly discovered hints can be retained and reused.

Reflection Experience

These methods write reflections over the agent’s own interaction rollouts into the experience base, so later rollouts can reuse improvement signals distilled from prior trial and error. EvolveR [72] and RetroAgent [73] both accumulate reflective experience from rollout outcomes. The former distills successful and failed trajectories into guiding and cautionary principles, while the latter stores retrospective lessons and retrieves them by relevance and historical utility. ERL [74] focuses on failure-driven correction, adding a reflection to the experience base only when it leads to a successful retry. MAGE [75] keeps reflection within the same meta-rollout, using previous interaction history, rewards, and reflections as the experience context for the next rollout.

Strategy Experience

These methods store decision strategies formed during training. SGE [76] stores strategy sequences from rollouts, indexed by task and outcome, later using successful strategies as references and failed strategies to trigger critique and revision. CRMWeaver [77] distills advanced-model execution logs into workflow guidelines and retrieves them for similar business queries. IntPro [78] stores intent explanations from user history, allowing the agent to retrieve historical intent patterns under uncertainty. MetaClaw [79] generates skill instructions from failed trajectories, immediately saves them into a long-term experience base, and uses them in later tasks.

4.2.2. Curated Experience Base

Curated experience base methods go beyond appending new entries; they merge, revise, or replace existing ones to ensure experience reliability.

Skill Evolution

These methods treat skills as experience entries and maintain reusable action procedures. SkillRL [20] and ARISE [80] generate structured skills from training rollouts. SkillRL updates the SkillBank by analyzing failures in underperforming task categories, while ARISE lets the policy handle skill generation, selection, and use in one loop. COS-PLAY [81] and SAGE_exp [82] make skills executable. COS-PLAY turns interaction segments into skill protocols with effect contracts, while SAGE_exp lets the agent create, call, repair, and save function skills. K²-Agent [83] combines task-level know-what rules with low-level know-how action hints, revising high-level rules after execution failures.

Experience Revision

These methods revise experience entries or their retrieved form using new rollouts, failures, or usage feedback. BEPA [84] first converts expert trajectories into policy-compatible successful trajectory caches, then refreshes each task cache with new successes from the current policy during RL. DGO [85] and Comp.RL [86] both update experience from fresh interaction trajectories. DGO extracts strategies and pitfall warnings from correct and incorrect trajectories and renews the experience base after each RL round, while Comp.RL lets an extractor decide whether to add, update, or merge entries from the current trajectory, outcome, and previously used experience. INSPO [87], AgentEvolver [88], and PEARL [89] revise higher-level strategy text. INSPO rewrites the instruction population from failed trajectories, AgentEvolver reranks and rewrites retrieved experience into a task-specific hint, and PEARL explicitly reads and updates user preference strategies during calendar decisions.

4.2.3. Optimized Experience Base

Optimized experience base methods bring the utility of experience into training, so that experience selection, weighting, or generation modules are shaped by policy feedback.

Experience-Derived Utility

These methods turn retrieved experience into update signals while still using experience to guide rollout generation or teacher-side scoring. D2Skill [90] and SLEA-RL [91] both inject retrieved experience during rollout sampling and then estimate its utility from the resulting performance. D2Skill updates skill utility and forms hindsight reward from the success gap between skill and baseline groups. SLEA-RL retrieves strategy or warning experience by observation clusters and combines step-level experience with local advantage. Skill-SD [92] uses per-task skills only on the teacher side, turning skill-conditioned teacher scores on student on-policy tokens into a self-distillation loss.

Trainable Experience Module

These methods train a module that produces or selects textual experience before it guides generation. UMEM [93] trains a Mem-Optimizer to extract and manage generalizable memory entries using semantic-neighborhood utility. Trainable Graph Memory [94] organizes historical trajectories into a weighted graph of query, transition, and meta-cognition nodes, and trains the weights so useful meta-cognition strategies are more likely to be retrieved and prepended to the policy prompt.

5. Domain-Specific Hints

In domain-specific settings, hint-based RL converts domain-native signals into training-time guidance. We organize representative work along three domain families. In technical domains, hints are drawn from executable or formally checkable feedback. Kevin [95] leverages compilation and runtime feedback for CUDA kernel refinement, SGS [96] synthesizes related theorems around unsolved targets with Lean providing verification, and C2F-Thinker [97] steers multimodal sentiment reasoning through hierarchical coarse-to-fine cues. In interactive agent domains, hints are drawn from environment states and action-validity signals. COS-PLAY [81] abstracts game interaction segments into skill protocols, WebGen-Agent [98] turns execution, visual, and GUI-agent feedback into step-level guidance, and UI-S1 [99] patches failed rollouts with expert GUI actions. In business and personal-service domains, hints are drawn from behavioral norms, workflow rules, and user-specific preferences. TaoSR-AGRL [100] treats e-commerce relevance labels as guided replay signals, CRMWeaver [77] retrieves business workflow guidelines, VeriRole [101] derives verifiable role-awareness signals, and PEARL [89] maintains preference strategies for time-management decisions.

6. Discussion

6.1. Scope and Boundaries

Several lines of work are formally similar to hint-based RL but fall outside our scope. Trajectory augmentation methods expand sampling at predefined branching points via random perturbation or tree search to enhance exploration [102,103], raising the pass@k of the current policy without introducing information beyond its own distribution. Input rewriting methods modify the query itself, for instance by adjusting problem difficulty [86,104,105] or converting open-ended questions into multiple-choice ones [106], which also mitigates zero-advantage failure. Such methods reshape the training query distribution, whereas hints inject external information while preserving the original query. Finally, some methods introduce hint-like mechanisms such as reference trajectory segments [107], critics [108], and finer evaluation criteria [109]; others use teacher [110] or self-generated [111] text for on-policy distillation [112]. Their shared purpose is to assign more accurate rewards to existing rollouts rather than to expand the reachable trajectory set, which places them under fine-grained reward methods.

Table 1. Count-based cross-analysis of hint construction sources and utilization mechanisms. Abbreviations on the left correspond to the four utilization types.

	Offline			Online
	Human	Base	Teacher	Old	Current	Teacher	Total
Inj.	3	1	16	6	10	2	38
Cont.	1	1	8	1	1	0	12
Repl.	1	1	9	1	9	2	23
Opt.	3	2	3	3	3	6	20
Total	8	5	36	11	23	10	93

Table 1. Count-based cross-analysis of hint construction sources and utilization mechanisms. Abbreviations on the left correspond to the four utilization types.

	Offline			Online
	Human	Base	Teacher	Old	Current	Teacher	Total
Inj.	3	1	16	6	10	2	38
Cont.	1	1	8	1	1	0	12
Repl.	1	1	9	1	9	2	23
Opt.	3	2	3	3	3	6	20
Total	8	5	36	11	23	10	93

6.2. Cross-Level Analysis of Hints

Section 3 and Section 4 organize methods by content refinement and experience base type, respectively. Here we provide a cross-analysis of construction sources and utilization mechanisms.

Task-level hints extend sample-specific hints into cross-sample experience bases, with their unique contribution in base construction and maintenance. On the utilization side, however, task-level methods inherit the same four mechanisms. Table A4 shows task-level methods distributing across the same four rows, with injection predominant and continuation least used. Task-level methods also show notably lower density in replacement, as abstracted experience entries are better suited for injection than trajectory substitution. The two levels thus share a utilization framework but diverge in construction: sample-level methods focus on per-hint generation quality, task-level methods on cross-sample accumulation and management.

Task-level hints extend sample-specific hints into cross-sample experience bases, with their unique contribution in base construction and maintenance. On the utilization side, however, task-level methods inherit the same four mechanisms. Table A4 shows task-level methods distributing across the same four rows, with injection predominant and continuation least used. Task-level methods also show notably lower density in replacement, likely because experience entries are typically abstracted and better suited as contextual conditions than as trajectory substitutes. The two levels thus share a utilization framework but diverge in construction: sample-level methods focus on per-hint generation quality, task-level methods on cross-sample accumulation and management.

This divergence entails different trade-offs. Sample-level hints offer the most direct intervention for zero-advantage groups but are single-use, leaving construction costs unamortized. Task-level hints amortize costs through reuse and support continual adaptation and self-evolution, yet risk reinforcing erroneous or stale entries. Static bases are stable but cannot adapt to policy changes; evolving bases are flexible but harder to control. Further category-level comparisons are in Appendix A.3.

6.3. Challenges and Future Directions

Despite the promise of hint-based RL in mitigating zero-advantage failure, several open problems and promising directions remain. (1) Regarding the quality of hints, controllable construction of high-quality hints remains a core challenge. An ideal hint should elicit responses that remain close to the policy’s distribution while introducing novel knowledge or reasoning patterns that the policy can effectively absorb. Meanwhile, the train-inference mismatch caused by hints being available only during training has yet to be systematically studied, since policies may degrade once the hint is removed. We provide an extended discussion of this issue in Appendix A.4. (2) Regarding the use of hints, hint-based RL offers a viable path toward model self-evolution, and debiasing mechanisms for task-level experience bases are crucial for long-term training stability, since uncurated entries bias future retrieval. Learnable hinters also stand out as a promising direction, where multiple models may serve as hinters for one another and be jointly optimized during training. (3) Finally, when hints are biased or experience bases are contaminated with inadvertent errors or deliberately injected malicious entries, the policy may drift unpredictably or even violate safety constraints. Trustworthy hint-based RL incorporating trustworthiness estimation and safety mechanisms thus stands as an important direction for future research.

7. Conclusions

This survey consolidates recent hint-based RL methods that address the zero-advantage failure through explicit textual guidance. We organize them into sample-level hints, intervening within a single sample via trajectory-based or scaffold-based guidance, and task-level hints, maintaining reusable experience bases across samples. We further outline open problems and future directions.

Appendix A.1. Taxonomy and Representative Methods

Figure A1 provides an expanded view of the taxonomy defined in Section 2 and lists representative methods under each branch. Sample-level methods are divided into trajectory-based and scaffold-based hints according to whether the explicit textual signal is a response trajectory or a semantic scaffold. Task-level methods are divided into static and evolving experience bases according to whether the shared hint repository is updated during RL training. Domain-specific methods are shown as a separate application-oriented branch because domain-native signals are especially important when specialized knowledge, executable feedback, environment states, or user preferences must be converted into training-time guidance. These methods can still be analyzed through the sample-level or task-level granularity, and the separate branch highlights important deployment settings without treating them as a disjoint mechanism family.

Figure A1. Taxonomy of hint-based reinforcement learning methods.

Figure A1. Taxonomy of hint-based reinforcement learning methods.

Appendix A.2. Comparison Tables and Field Definitions

Table A1, Table A2, and Table A3 summarize the reporting fields used for method-level comparison. The fields are defined as follows.

• Hint Content. This field names the explicit textual signal used by a method, such as a reference prefix, repaired trajectory, critique, workflow, skill, memory entry, executable feedback, or preference signal. It corresponds to the hint object that directly affects rollout generation, rollout repair, or the RL update.
• Source. This field records where the hint comes from. On-policy indicates that the hint is generated by the trained model itself. Off-policy indicates that the hint is produced with an external model, teacher model, external annotated trajectory, or external reference solution. Hybrid corresponds to Off&On-policy, where hint construction explicitly depends on both the trained model and an external model or teacher. External verifiers, rule-based scorers, and Math-Verify style correctness checkers are treated as reward or validation mechanisms. They do not change the hint source by themselves.
• Trigger. This field records when the hint intervenes in RL training. We use Always, Stage-wise Curriculum, and Specific Rules to distinguish persistent intervention, curriculum-dependent intervention, and rule-triggered intervention.
• Inference. This field records whether hints are still used after training. It supports the deployment discussion in Appendix A.4, where training-time hints may be removed, retained, or partially retained at inference.
• Objective. This field records the concrete optimization objective or training recipe reported by each method. It may include GRPO-family objectives, DAPO, REINFORCE, or method-specific variants, as well as auxiliary terms such as SFT, Kullback-Leibler divergence (KL), negative log-likelihood (NLL), Jensen-Shannon divergence (JSD), or self-distillation losses (SDL). Note that objectives labeled as GRPO in the table may also refer to GRPO variants adapted to specific methods.
• Retrieval / Use. This field appears in the task-level table and records how stored experience is selected or applied, such as random sampling, similarity retrieval, category retrieval, top-K retrieval, or population sampling.
• Experience Base Operation. This field in the task-level table records whether the experience repository is fixed or updated through operations including addition, update, deletion, merging, pruning, or reconstruction.

Table A1. Detailed comparison of sample-level hint methods in hint-based reinforcement learning.

Method	Hint Content	Source	Trigger	Inference	Objective
LLM
StepHint [38]	step-level reference prefixes	Off-policy	Always	No	GRPO
ANCHOR [7]	verified reference trajectory anchor	Off-policy	Always	No	GRPO
LUFFY [15]	off-policy expert reasoning trace	Off-policy	Always	No	GRPO
RAVR [45]	ground-truth answer	Off-policy	Always	No	GRPO+KL
CoVRL [46]	ground-truth answer	Off-policy	Always	No	GRPO+KL+NLL
MeRF [55]	natural-language reward specification	Off-policy	Always	No	GRPO
RAVR [45]	answer-guided reference distribution	Off-policy	Always	No	GRPO+KL
CoVRL [46]	answer-guided mixed trajectory distribution	Off-policy	Always	No	GRPO+KL+NLL
POPE [8]	short useful reference prefix	Off-policy	Rules	No	GRPO
SEELE [23]	reference prefix with learned length control	Off-policy	Rules	No	GRPO+SFT
BREAD [25]	episode-level expert prefix	Off-policy	Rules	No	GRPO
G2RPO-A [33]	trajectory prefix adjusted by reward trend	Off-policy	Rules	No	GRPO
TRAPO [37]	expert prefix selected within the rollout group	Off-policy	Rules	No	GRPO+SFT
AMPO [39]	selected teacher reference trajectories	Off-policy	Rules	No	GRPO
HAPO [41]	reference trajectory replacing a low-reward rollout	Off-policy	Rules	No	GRPO+SFT
HDPO [47]	ground-truth answer	Off-policy	Rules	No	GRPO+JSD
Guide-GRPO [48]	teacher-generated high-level guidance	Off-policy	Rules	No	GRPO
KnowRL [18]	minimal sufficient knowledge points	Off-policy	Rules	Opt.	GRPO
HINT [57]	teacher-generated heuristic hints	Off-policy	Rules	No	GRPO
RGR-GRPO [61]	unmet rubric criteria	Off-policy	Rules	No	GRPO
HDPO [47]	answer-conditioned teacher distribution	Off-policy	Rules	No	GRPO+JSD
UFT [31]	reference prefix with scheduled decay	Off-policy	Curriculum	No	GRPO+KL
PieceHint [50]	high-value reasoning pieces	Off-policy	Curriculum	No	GRPO
RuscaRL [56]	checklist-style rubric criteria	Off-policy	Curriculum	No	GRPO
ThinkTuning [67]	teacher reflection tokens	Off-policy	Curriculum	No	GRPO
QuestA [21]	partial reference solution prefix	Off-policy	Rules+Curriculum	No	GRPO
LLM
CCL [22]	threshold-calibrated reference prefix	Off-policy	Rules+Curriculum	No	GRPO
GHPO [24]	stagewise reference prefix	Off-policy	Rules+Curriculum	No	GRPO
Prefix-RFT [16]	offline demonstration prefix	Off-policy	Rules+Curriculum	No	Dr.GRPO
Scaf-GRPO [17]	hierarchical knowledge, planning, and solution hints	Off-policy	Rules+Curriculum	No	GRPO
RPO [28]	truncated historical response prefix	On-policy	Always	No	GRPO/DAPO
HiPO [26]	successful self-prefix from current batch	On-policy	Rules	No	GRPO
PROS [27]	uncertainty-truncated historical correct prefix	On-policy	Rules	No	GRPO
HiPO [26]	successful self-hinted trajectory group	On-policy	Rules	No	GRPO
LTE [60]	wrong-answer negative hints	On-policy	Rules	No	GRPO
Failure-Prefix [29]	rare failure prefix from saturated samples	On-policy	Rules+Curriculum	No	GRPO
ExPO [43]	answer-conditioned self-explanation trajectory	Hybrid	Always	No	GRPO+SFT
RLAD [51]	reasoning abstraction	Hybrid	Always	Yes	DAPO+RFT+SFT
ECHO [64]	diagnostic feedback from a trainable critic	Hybrid	Always	No	GRPO
RLTF [68]	natural-language critique feedback	Hybrid	Always	Opt.	GRPO+CE+AWR
CORE [30]	answer-stripped prefix from a successful peer policy	Hybrid	Rules	No	GRPO
SCOPE [42]	teacher-corrected suffix after a valid prefix	Hybrid	Rules	No	GRPO+NLL
A2D [52]	decomposer-generated sub-questions	Hybrid	Rules	No	GRPO+NLL
SAGEscaf [53]	self-generated plan or decomposition	Hybrid	Rules	No	GRPO
HiLL [59]	hinter-generated pedagogical hints	Hybrid	Rules	No	GRPO
GOLF [62]	group-level feedback from critiques	Hybrid	Rules	Opt.	GRPO
MEL [66]	verified meta-experience target	Hybrid	Rules	No	GRPO+NLL
EvoCoT [32]	CoT prefix constructed from final answer	Hybrid	Curriculum	No	GRPO
BHA [36]	distribution-aligned teacher prefix	Hybrid	Rules+Curriculum	No	DAPO
MENTOR [44]	mixed-policy rollout with expert token intervention	Hybrid	Rules+Curriculum	No	GRPO
NuRL [54]	self-generated abstract knowledge cue	Hybrid	Rules+Curriculum	No	GRPO
VLLM
ADHint [34]	reference prefix selected by query difficulty	Off-policy	Rules	No	GRPO
Hint-GRPO [35]	stepwise reference prefix	Off-policy	Rules	No	GRPO
S-GRPO [40]	single verified reference trajectory	Off-policy	Rules	No	GRPO
AVATAR [49]	precomputed strategy hints	Off-policy	Rules	No	GRPO+SFT
KEPO [58]	answer-conditioned teacher reasoning hints	Off-policy	Rules	No	GRPO+KL+SFT
Agent
InfoFlow [65]	pathfinding hint used as a search query	Off-policy	Rules	No	SFT+GRPO+KL
R3L [63]	reflection diagnosis with corrective guidance	Hybrid	Rules	No	GRPO

Table A1. Detailed comparison of sample-level hint methods in hint-based reinforcement learning.

Method	Hint Content	Source	Trigger	Inference	Objective
LLM
StepHint [38]	step-level reference prefixes	Off-policy	Always	No	GRPO
ANCHOR [7]	verified reference trajectory anchor	Off-policy	Always	No	GRPO
LUFFY [15]	off-policy expert reasoning trace	Off-policy	Always	No	GRPO
RAVR [45]	ground-truth answer	Off-policy	Always	No	GRPO+KL
CoVRL [46]	ground-truth answer	Off-policy	Always	No	GRPO+KL+NLL
MeRF [55]	natural-language reward specification	Off-policy	Always	No	GRPO
RAVR [45]	answer-guided reference distribution	Off-policy	Always	No	GRPO+KL
CoVRL [46]	answer-guided mixed trajectory distribution	Off-policy	Always	No	GRPO+KL+NLL
POPE [8]	short useful reference prefix	Off-policy	Rules	No	GRPO
SEELE [23]	reference prefix with learned length control	Off-policy	Rules	No	GRPO+SFT
BREAD [25]	episode-level expert prefix	Off-policy	Rules	No	GRPO
G2RPO-A [33]	trajectory prefix adjusted by reward trend	Off-policy	Rules	No	GRPO
TRAPO [37]	expert prefix selected within the rollout group	Off-policy	Rules	No	GRPO+SFT
AMPO [39]	selected teacher reference trajectories	Off-policy	Rules	No	GRPO
HAPO [41]	reference trajectory replacing a low-reward rollout	Off-policy	Rules	No	GRPO+SFT
HDPO [47]	ground-truth answer	Off-policy	Rules	No	GRPO+JSD
Guide-GRPO [48]	teacher-generated high-level guidance	Off-policy	Rules	No	GRPO
KnowRL [18]	minimal sufficient knowledge points	Off-policy	Rules	Opt.	GRPO
HINT [57]	teacher-generated heuristic hints	Off-policy	Rules	No	GRPO
RGR-GRPO [61]	unmet rubric criteria	Off-policy	Rules	No	GRPO
HDPO [47]	answer-conditioned teacher distribution	Off-policy	Rules	No	GRPO+JSD
UFT [31]	reference prefix with scheduled decay	Off-policy	Curriculum	No	GRPO+KL
PieceHint [50]	high-value reasoning pieces	Off-policy	Curriculum	No	GRPO
RuscaRL [56]	checklist-style rubric criteria	Off-policy	Curriculum	No	GRPO
ThinkTuning [67]	teacher reflection tokens	Off-policy	Curriculum	No	GRPO
QuestA [21]	partial reference solution prefix	Off-policy	Rules+Curriculum	No	GRPO
LLM
CCL [22]	threshold-calibrated reference prefix	Off-policy	Rules+Curriculum	No	GRPO
GHPO [24]	stagewise reference prefix	Off-policy	Rules+Curriculum	No	GRPO
Prefix-RFT [16]	offline demonstration prefix	Off-policy	Rules+Curriculum	No	Dr.GRPO
Scaf-GRPO [17]	hierarchical knowledge, planning, and solution hints	Off-policy	Rules+Curriculum	No	GRPO
RPO [28]	truncated historical response prefix	On-policy	Always	No	GRPO/DAPO
HiPO [26]	successful self-prefix from current batch	On-policy	Rules	No	GRPO
PROS [27]	uncertainty-truncated historical correct prefix	On-policy	Rules	No	GRPO
HiPO [26]	successful self-hinted trajectory group	On-policy	Rules	No	GRPO
LTE [60]	wrong-answer negative hints	On-policy	Rules	No	GRPO
Failure-Prefix [29]	rare failure prefix from saturated samples	On-policy	Rules+Curriculum	No	GRPO
ExPO [43]	answer-conditioned self-explanation trajectory	Hybrid	Always	No	GRPO+SFT
RLAD [51]	reasoning abstraction	Hybrid	Always	Yes	DAPO+RFT+SFT
ECHO [64]	diagnostic feedback from a trainable critic	Hybrid	Always	No	GRPO
RLTF [68]	natural-language critique feedback	Hybrid	Always	Opt.	GRPO+CE+AWR
CORE [30]	answer-stripped prefix from a successful peer policy	Hybrid	Rules	No	GRPO
SCOPE [42]	teacher-corrected suffix after a valid prefix	Hybrid	Rules	No	GRPO+NLL
A2D [52]	decomposer-generated sub-questions	Hybrid	Rules	No	GRPO+NLL
SAGEscaf [53]	self-generated plan or decomposition	Hybrid	Rules	No	GRPO
HiLL [59]	hinter-generated pedagogical hints	Hybrid	Rules	No	GRPO
GOLF [62]	group-level feedback from critiques	Hybrid	Rules	Opt.	GRPO
MEL [66]	verified meta-experience target	Hybrid	Rules	No	GRPO+NLL
EvoCoT [32]	CoT prefix constructed from final answer	Hybrid	Curriculum	No	GRPO
BHA [36]	distribution-aligned teacher prefix	Hybrid	Rules+Curriculum	No	DAPO
MENTOR [44]	mixed-policy rollout with expert token intervention	Hybrid	Rules+Curriculum	No	GRPO
NuRL [54]	self-generated abstract knowledge cue	Hybrid	Rules+Curriculum	No	GRPO
VLLM
ADHint [34]	reference prefix selected by query difficulty	Off-policy	Rules	No	GRPO
Hint-GRPO [35]	stepwise reference prefix	Off-policy	Rules	No	GRPO
S-GRPO [40]	single verified reference trajectory	Off-policy	Rules	No	GRPO
AVATAR [49]	precomputed strategy hints	Off-policy	Rules	No	GRPO+SFT
KEPO [58]	answer-conditioned teacher reasoning hints	Off-policy	Rules	No	GRPO+KL+SFT
Agent
InfoFlow [65]	pathfinding hint used as a search query	Off-policy	Rules	No	SFT+GRPO+KL
R3L [63]	reflection diagnosis with corrective guidance	Hybrid	Rules	No	GRPO

Table A2. Detailed comparison of task-level hint methods in hint-based reinforcement learning.

Method	Hint Content	Source	Retrieval / Use	Experience Base Operation	Trigger	Inference	Objective
LLM
CBRL [69]	Cross-sample solved demonstrations	Off-policy	Label sampling	Fixed	Curriculum	No	GRPO
TemplateRL [71]	Problem-solving templates	Hybrid	PCC matching	Fixed	Always	Yes	Dr.GRPO
ICPO [70]	Expert problem-solution demonstrations	Hybrid	Random sampling	Fixed	Rules	No	GRPO
DGO [85]	Correct and wrong reasoning experience	Hybrid	Experience retrieval	Add/Update/Delete	Curriculum	Yes	GRPO+SFT
Agent
SkillRL [20]	Task-level skills	Off-policy	Category retrieval	Add/Update	Always	Yes	GRPO+SFT
CRMWeaver [77]	Workflow guidelines	Off-policy	Top-1 similarity	Add/Update	Rules	Yes	DAPO+SFT
SKILL0 [19]	Procedural skills	Off-policy	Skill selection	Fixed	Rules+Curriculum	No	GRPO
INSPO [87]	Evolved instructions	Off-policy	Population sampling	Add/Delete/Reconstruct	Rules+Curriculum	Yes	GRPO
COS-PLAY [81]	Long-horizon game skills	On-policy	Skill retrieval	Add/Update/Merge/Delete	Always	Yes	GRPO+SFT
SAGEexp [82]	Executable function skills	On-policy	Skill retrieval	Add/Update	Always	Yes	GRPO+SFT
UMEM [93]	Editable memory-style experience entries	On-policy	Top-K retrieval	Add/Update	Rules	Yes	GRPO
EvolveR [72]	Guiding and cautionary reflections	On-policy	Explicit search	Add/Update/Merge/Prune	Rules	Yes	GRPO+SFT
RetroAgent [73]	Retrospective lessons	On-policy	SimUtil-UCB	Add/Update	Rules	Yes	GRPO
ERL [74]	Failure-correction reflections	On-policy	Threshold retry	Add	Rules	No	GRPO+SFT
SGE [76]	Successful and failed strategies	On-policy	FIFO sampling	Add/Delete	Rules	No	GRPO
SLEA-RL [91]	Step-level strategies and warnings	On-policy	Cluster retrieval	Add/Delete	Rules	Yes	GRPO
MAGE [75]	Interaction history and reflections	On-policy	Meta-context	Temporary aggregation	Curriculum	Yes	GiGPO
PEARL [89]	Time-management strategies	On-policy	StrategyHub	Add/Update/Delete	Rules+Curriculum	Yes	GRPO
Comp.RL [86]	Reusable trajectory-derived experience	Hybrid	Search-and-ask	Add/Update/Merge/Delete	Always	Yes	Split-GRPO
D2Skill [90]	Task and step skills	Hybrid	Utility comparison	Add/Update/Prune	Always	Yes	GRPO
Skill-SD [92]	Teacher-side task skills	Hybrid	Teacher scoring	Add/Update	Always	No	GRPO+SDL
IntPro [78]	User intent patterns with explanations	Hybrid	Tool retrieval	Add/Update	Rules	Yes	GRPO+SFT
MetaClaw [79]	Failure-driven skill instructions	Hybrid	Embedding search	Add	Rules	Yes	GRPO
BEPA [84]	Per-task successful trajectories	Hybrid	Cache lookup	Add/Update	Rules	No	GRPO
AgentEvolver [88]	Natural-language experience entries	Hybrid	Vector retrieval	Add/Update	Rules	Yes	GRPO
Trainable G.Mem. [94]	Graph-structured meta-cognition	Hybrid	Graph retrieval	Add/Merge/Update	Rules	Yes	REINFORCE
K2-Agent [83]	Mobile-control rules and demonstrations	Hybrid	Type retrieval	Add/Update/Merge	Curriculum	Yes	C-GRPO
ARISE [80]	Seed and generated skills	Hybrid	Manager selection	Add/Update/Delete	Rules+Curriculum	Yes	GRPO

Table A2. Detailed comparison of task-level hint methods in hint-based reinforcement learning.

Method	Hint Content	Source	Retrieval / Use	Experience Base Operation	Trigger	Inference	Objective
LLM
CBRL [69]	Cross-sample solved demonstrations	Off-policy	Label sampling	Fixed	Curriculum	No	GRPO
TemplateRL [71]	Problem-solving templates	Hybrid	PCC matching	Fixed	Always	Yes	Dr.GRPO
ICPO [70]	Expert problem-solution demonstrations	Hybrid	Random sampling	Fixed	Rules	No	GRPO
DGO [85]	Correct and wrong reasoning experience	Hybrid	Experience retrieval	Add/Update/Delete	Curriculum	Yes	GRPO+SFT
Agent
SkillRL [20]	Task-level skills	Off-policy	Category retrieval	Add/Update	Always	Yes	GRPO+SFT
CRMWeaver [77]	Workflow guidelines	Off-policy	Top-1 similarity	Add/Update	Rules	Yes	DAPO+SFT
SKILL0 [19]	Procedural skills	Off-policy	Skill selection	Fixed	Rules+Curriculum	No	GRPO
INSPO [87]	Evolved instructions	Off-policy	Population sampling	Add/Delete/Reconstruct	Rules+Curriculum	Yes	GRPO
COS-PLAY [81]	Long-horizon game skills	On-policy	Skill retrieval	Add/Update/Merge/Delete	Always	Yes	GRPO+SFT
SAGEexp [82]	Executable function skills	On-policy	Skill retrieval	Add/Update	Always	Yes	GRPO+SFT
UMEM [93]	Editable memory-style experience entries	On-policy	Top-K retrieval	Add/Update	Rules	Yes	GRPO
EvolveR [72]	Guiding and cautionary reflections	On-policy	Explicit search	Add/Update/Merge/Prune	Rules	Yes	GRPO+SFT
RetroAgent [73]	Retrospective lessons	On-policy	SimUtil-UCB	Add/Update	Rules	Yes	GRPO
ERL [74]	Failure-correction reflections	On-policy	Threshold retry	Add	Rules	No	GRPO+SFT
SGE [76]	Successful and failed strategies	On-policy	FIFO sampling	Add/Delete	Rules	No	GRPO
SLEA-RL [91]	Step-level strategies and warnings	On-policy	Cluster retrieval	Add/Delete	Rules	Yes	GRPO
MAGE [75]	Interaction history and reflections	On-policy	Meta-context	Temporary aggregation	Curriculum	Yes	GiGPO
PEARL [89]	Time-management strategies	On-policy	StrategyHub	Add/Update/Delete	Rules+Curriculum	Yes	GRPO
Comp.RL [86]	Reusable trajectory-derived experience	Hybrid	Search-and-ask	Add/Update/Merge/Delete	Always	Yes	Split-GRPO
D2Skill [90]	Task and step skills	Hybrid	Utility comparison	Add/Update/Prune	Always	Yes	GRPO
Skill-SD [92]	Teacher-side task skills	Hybrid	Teacher scoring	Add/Update	Always	No	GRPO+SDL
IntPro [78]	User intent patterns with explanations	Hybrid	Tool retrieval	Add/Update	Rules	Yes	GRPO+SFT
MetaClaw [79]	Failure-driven skill instructions	Hybrid	Embedding search	Add	Rules	Yes	GRPO
BEPA [84]	Per-task successful trajectories	Hybrid	Cache lookup	Add/Update	Rules	No	GRPO
AgentEvolver [88]	Natural-language experience entries	Hybrid	Vector retrieval	Add/Update	Rules	Yes	GRPO
Trainable G.Mem. [94]	Graph-structured meta-cognition	Hybrid	Graph retrieval	Add/Merge/Update	Rules	Yes	REINFORCE
K2-Agent [83]	Mobile-control rules and demonstrations	Hybrid	Type retrieval	Add/Update/Merge	Curriculum	Yes	C-GRPO
ARISE [80]	Seed and generated skills	Hybrid	Manager selection	Add/Update/Delete	Rules+Curriculum	Yes	GRPO

Table A3. Detailed comparison of domain-specific hint methods in hint-based reinforcement learning.

Method	Hint Content	Source	Trigger	Inference	Objective
LLM
TaoSR-AGRL [100]	Dimension-level business labels	Off-policy	Rules	No	GRPO
Kevin [95]	Execution feedback and prior kernel attempts	On-policy	Always	Yes	GRPO
SGS [96]	Synthetic theorem or subproblem	On-policy	Rules	No	REINFORCE
VeriRole [101]	Role evidence snippets	Hybrid	Always	Yes	GRPO
VLLM
C2F-Thinker [97]	Ground-truth polarity label	Off-policy	Rules	No	GRPO+SFT
Agent
WebGen-Agent [98]	Execution, visual, and GUI feedback	Off-policy	Rules	Yes	Step-GRPO+SFT
UI-S1 [99]	Expert GUI action patch	Off-policy	Rules	No	Semi-online GRPO+SFT
CRMWeaver [77]	Workflow guideline	Off-policy	Rules	Yes	DAPO+SFT
COS-PLAY [81]	Skill protocol	On-policy	Always	Yes	GRPO+SFT
PEARL [89]	Preference strategy	On-policy	Rules+Curriculum	Yes	GRPO

Table A3. Detailed comparison of domain-specific hint methods in hint-based reinforcement learning.

Method	Hint Content	Source	Trigger	Inference	Objective
LLM
TaoSR-AGRL [100]	Dimension-level business labels	Off-policy	Rules	No	GRPO
Kevin [95]	Execution feedback and prior kernel attempts	On-policy	Always	Yes	GRPO
SGS [96]	Synthetic theorem or subproblem	On-policy	Rules	No	REINFORCE
VeriRole [101]	Role evidence snippets	Hybrid	Always	Yes	GRPO
VLLM
C2F-Thinker [97]	Ground-truth polarity label	Off-policy	Rules	No	GRPO+SFT
Agent
WebGen-Agent [98]	Execution, visual, and GUI feedback	Off-policy	Rules	Yes	Step-GRPO+SFT
UI-S1 [99]	Expert GUI action patch	Off-policy	Rules	No	Semi-online GRPO+SFT
CRMWeaver [77]	Workflow guideline	Off-policy	Rules	Yes	DAPO+SFT
COS-PLAY [81]	Skill protocol	On-policy	Always	Yes	GRPO+SFT
PEARL [89]	Preference strategy	On-policy	Rules+Curriculum	Yes	GRPO

Appendix A.3. Category-Level Comparison of Hints

Section 6.2 provides a cross-analysis of construction sources and utilization mechanisms across sample-level and task-level hints. Here we further compare the major subcategories in Figure A1 and Table A1 and Table A2. For sample-level methods, trajectory-based hints and scaffold-based hints differ mainly in how explicitly they constrain the rollout. Trajectory-based hints expose concrete solution states, such as reference prefixes, completed trajectories, repaired suffixes, or mixed expert-policy rollouts. They are therefore effective for cold-start samples where the current policy rarely reaches a successful path, because they reduce the search depth or insert a reliable contrastive anchor. Their strength also brings risk. If the trajectory is too long, too off-policy, or too close to the answer, the policy may learn to imitate the hint context and fail to recover the underlying reasoning ability. This makes trajectory source, revealed amount, and trigger condition central to trajectory-based methods.

Scaffold-based hints provide less explicit guidance, such as abstractions, constraints, knowledge cues, rubrics, or critiques. They are useful when multiple valid reasoning paths exist or when the goal is to steer the policy’s strategy without prescribing a concrete path. Compared with trajectory-based hints, scaffolds preserve more generation freedom and are less tied to one reference solution. However, their effectiveness is harder to verify. A scaffold can be too vague to change the rollout distribution or too noisy to provide reliable improvement. As a result, scaffold-based methods often control the construction or use of scaffolds through compact and non-leaking cues, verifier-guided replacement or refinement, and auxiliary supervision or distribution alignment.

Within task-level methods, the contrast between static experience base and evolving experience base concerns how persistent guidance is managed. A static experience base is constructed before target RL training and remains fixed, making it relatively stable, auditable, and easy to reproduce. This is useful when the experience source is trusted and the task distribution is stable. Its limitation is that the base cannot absorb newly discovered strategies or remove entries that become stale as the policy changes. An evolving experience base updates entries during training through addition, revision, merging, deletion, pruning, or utility estimation. This allows the experience base to track the policy’s changing capability boundary and is better aligned with self-evolving agents. This flexibility also makes control more difficult. Erroneous entries, retrieval bias, duplicated strategies, or contaminated memories can be reinforced over time unless the system includes debiasing, validation, and maintenance mechanisms.

Table A4. Cross-analysis of hint construction sources and utilization mechanisms across sample-level and task-level methods. Sample-level methods: trajectory-based, scaffold-based. Task-level methods: static experience base, evolving experience base. Methods spanning two construction sources appear in both columns. “–” denotes unexplored combinations. ^† Method spans two construction sources and appears in both corresponding columns.

	Offline Construction			Online Construction
Utilization	Human	Base Policy	Teacher	Old Policy	Current Policy	Teacher
Hint Injection	GHPO MeRF CBRL†	NuRL	QuestA, POPE CCL, SEELE BREAD Guide-GRPO PieceHint, AVATAR KnowRL, RuscaRL RLAD† SKILL0, CBRL† CRMWeaver, IntPro SkillRL	PROS, RPO Failure-Prefix† EvolveR RetroAgent, SGE	CORE Failure-Prefix† SAGEscaf RLAD† MAGE, ARISE COS-PLAY SAGEexp Comp.RL, PEARL	MetaClaw INSPO
Hint Continuation	K2-Agent†	EvoCoT†	UFT, Prefix-RFT G2RPO-A, ADHint Hint-GRPO, BHA TRAPO, StepHint	EvoCoT†	K2-Agent†	–
Hint Replacement	ICPO	BEPA†	ANCHOR, LUFFY AMPO, S-GRPO HAPO HINT, Scaf-GRPO InfoFlow RGR-GRPO†	BEPA†	HiPO, SCOPE† LTE, R3L ECHO, HiLL GOLF† RGR-GRPO† ERL	SCOPE† GOLF†
Hint Optimization	RAVR, CoVRL HDPO	ExPO A2D†	MENTOR DGPO A2D†	DGO† SLEA-RL Trainable G.Mem.	MEL DGO† UMEM	KEPO ThinkTuning RLTF AgentEvolver D2Skill, Skill-SD

Table A4. Cross-analysis of hint construction sources and utilization mechanisms across sample-level and task-level methods. Sample-level methods: trajectory-based, scaffold-based. Task-level methods: static experience base, evolving experience base. Methods spanning two construction sources appear in both columns. “–” denotes unexplored combinations. ^† Method spans two construction sources and appears in both corresponding columns.

	Offline Construction			Online Construction
Utilization	Human	Base Policy	Teacher	Old Policy	Current Policy	Teacher
Hint Injection	GHPO MeRF CBRL†	NuRL	QuestA, POPE CCL, SEELE BREAD Guide-GRPO PieceHint, AVATAR KnowRL, RuscaRL RLAD† SKILL0, CBRL† CRMWeaver, IntPro SkillRL	PROS, RPO Failure-Prefix† EvolveR RetroAgent, SGE	CORE Failure-Prefix† SAGEscaf RLAD† MAGE, ARISE COS-PLAY SAGEexp Comp.RL, PEARL	MetaClaw INSPO
Hint Continuation	K2-Agent†	EvoCoT†	UFT, Prefix-RFT G2RPO-A, ADHint Hint-GRPO, BHA TRAPO, StepHint	EvoCoT†	K2-Agent†	–
Hint Replacement	ICPO	BEPA†	ANCHOR, LUFFY AMPO, S-GRPO HAPO HINT, Scaf-GRPO InfoFlow RGR-GRPO†	BEPA†	HiPO, SCOPE† LTE, R3L ECHO, HiLL GOLF† RGR-GRPO† ERL	SCOPE† GOLF†
Hint Optimization	RAVR, CoVRL HDPO	ExPO A2D†	MENTOR DGPO A2D†	DGO† SLEA-RL Trainable G.Mem.	MEL DGO† UMEM	KEPO ThinkTuning RLTF AgentEvolver D2Skill, Skill-SD

Appendix A.4. Hint Availability at Inference

The Inference column in Table A1, Table A2, and Table A3 records whether a method requires hints after training. This distinction is important because many hint-based RL methods use hints only during training as textual signals. In the broader coding underlying these tables, 86 out of 116 entries do not use hints at inference, while 30 do. The pattern is uneven across categories. Sample-level trajectory and scaffold methods are dominated by inference-free designs, whereas task-level experience methods more often keep retrieved skills, memories, workflows, or preferences available during deployment.

For sample-level methods, removing the hint at inference creates a potential mismatch between training and inference. Reference prefixes, answer-conditioned scaffolds, teacher critiques, and replacement trajectories can make hard samples learnable during RL, but the deployed policy must solve the original query without these guided conditions. If the policy only learns to complete from hinted contexts, removing the hint can shift the input distribution back to the hard prompt, causing the model to skip early reasoning steps, rely on answer-conditioned shortcuts, or fail to reproduce the capability learned under guidance.

Existing methods mitigate this mismatch in several ways. One line gradually reduces hint exposure through prefix decay, backward annealing, hint dropout, random withholding, or curriculum schedules [16,19,30,31,36,50,56,69]. Another line controls hint strength so that the weakest useful hint is used only when the original group is uninformative, reducing long-term dependence on guided contexts [8,17,25,26,35,37]. A third line transfers behavior from hinted generation to the unguided policy through distribution alignment, distillation, or losses computed under unscaffolded conditions [45,46,47,56,92]. Many task-level experience methods intentionally keep retrieved skills, memories, or preference strategies at inference [20,71,77,88,89,90]. In these cases, the mismatch is reduced, but deployment depends on reliable retrieval, bounded context cost, and safeguards against stale or malicious experience. A more systematic evaluation of hint reliance, for example by comparing hinted, hint-free, and partially ablated inference at the same checkpoint, remains an important direction.