Training–Rollout Mismatch in Low-Precision LLM Reinforcement Learning: A Survey

1. Introduction and Scope

1.1. Why Low-Precision Rollouts Complicate Policy Optimization

Response generation can account for a substantial share of language-model RL cost, particularly with long rollouts. Low precision can reduce computation and memory costs [4,12], but discrepancies between rollout and learner execution can compromise the policy updates that use this experience.

The sampler generates responses, and the learner recomputes token probabilities for updates. Even with synchronized parameters, differences in kernels, numerical representations, routing, or cached states can yield unequal probabilities for the same token and history [1,12]. Quantization can enlarge this training–inference mismatch (TIM), changing both the sampled experience and its influence through importance weights and clipping. Each update then changes the policy, its numerical behavior, and the histories sampled next.

Recent methods intervene at different points in this loop. Jet-RL coordinates precision across execution paths, FlashRL reweights behavior samples, and TRIAGE controls contributions using mismatch direction [1,3,12]. Their mechanisms differ in whether they change the computations that generate probabilities or the learning rules applied to sampled experience. This distinction also determines how they can be combined and which errors remain.

Figure 2 traces representative developments in rollout correction, quantization-assisted exploration, FP8 execution, FP4 alignment, and cache-aware mitigation.

We organize these studies by intervention target. Numerical design (A) acts on the computations that produce mismatch through alignment or fidelity control. Learning under residual mismatch (B) acts on its effects through estimator correction or update control (Figure 4). Papers can contribute to both families; Appendix A maps their components.

1.2. Scope and Related Work

The survey examines quantized rollouts with higher-precision learners or low precision in both learner and sampler. Weight, activation, and key–value (KV) cache quantization affect different numerical interfaces; parameter staleness can compound their mismatch.

Direct low-precision methods and systems form the core, supported by estimation theory and general training–inference mismatch studies. A study can serve several roles: MiniRL develops the formulation and evaluates FP8 rollout [14]. Quantization-assisted exploration provides a boundary case. Appendix B details source selection and versions.

Broader surveys cover low-precision training and reasoning-model RL [33,35]. Other rollout surveys organize pipelines by lifecycle [34] or efficiency mechanisms and bottlenecks [36]. Here, numerical execution links alignment, correction, and cache design through their effects on estimation and policy updates [3,6,10,12,17].

1.3. Perspective and Contributions

Distinguishing learner, behavior, proximal, and reference policies separates execution discrepancy from lag and clarifies correction ratios and update constraints.

The taxonomy separates computation alignment, fidelity and update visibility, gradient estimation, and update control. Precision, W/A/KV target, architecture, and collection protocol define settings; granularity and information requirements distinguish components within each class.

The evidence synthesis compares measurement practices, policy quality, and total cost, identifying gaps in long-horizon stability and quality–cost evaluation.

2. Background and Problem Formulation

2.1. Learner, Behavior, Proximal, and Reference Policies

Let x be a prompt, $y_{1:T}$ a response, and $h_t=(x,y_{<t})$ its prefix. A policy depends on both parameters and numerical execution. Write $p_{\theta }(a\mid h)$ for the current learner and $p_k(a\mid h)$ for the same computation at sampling snapshot ${\theta }_k$.

The behavior policy $q_k(a\mid h)$ samples tokens through the rollout engine and its numerical and decoding rules, including temperature and vocabulary truncation. Raw-model log-probabilities need not describe this sampling distribution.

The proximal policy $p_{\mathrm{prox}}$ anchors the PPO and GRPO update ratio, commonly using a frozen pre-update learner [30,31]. A separate fixed checkpoint can supply the regularization reference $p_{\mathrm{ref}}$. Mismatch-aware formulations distinguish the proximal policy from behavior [1,14].

A running GRPO update.

For one prompt, the sampler generates G responses with rewards $R_i$. In RL with verifiable rewards, a correctness checker can replace a learned reward model. A common group-relative advantage is $A_i=(R_i-\overline{R})/({\sigma }_R+{ϵ}_A)$, where $\overline{R}$ and ${\sigma }_R$ are the group reward mean and standard deviation, and ${ϵ}_A>0$ stabilizes normalization. Binary rewards $(1,1,0,0)$ give advantages approximately $(1,1,-1,-1)$. The advantage remains fixed during optimization and is usually shared by valid tokens of a response; some variants omit standard-deviation normalization [31].

At the learning stage, a representative clipped objective on this fixed group is

$$\begin{array}{cc}\hfill \widehat{J}\left(\theta \right)& ={\displaystyle \frac{1}{G}}{\displaystyle \sum _{i=1}^G}{\displaystyle \frac{1}{T_i}}{\displaystyle \sum _{t=1}^{T_i}}\left[\ell (r_{i,t},A_i)-\beta {\mathcal{K}}_{i,t}\left(\theta \right)\right],\hfill \\ \hfill \ell (r,A)& =\min \left\{rA,clip(r,1-ϵ,1+ϵ)A\right\}.\hfill \end{array}$$

(1)

Here $T_i$ counts valid response tokens, $r_{i,t}=p_{\theta }(y_{i,t}\mid h_{i,t})/p_{\mathrm{prox}}(y_{i,t}\mid h_{i,t})$, $ϵ$ sets the clipping interval, and $\beta$ controls optional reference regularization. The conditional KL is ${\mathcal{K}}_{i,t}=D_{\mathrm{KL}}(p_{\theta }(·\mid h_{i,t})\parallel p_{\mathrm{ref}}(·\mid h_{i,t}))$; implementations can use sampled estimators. Maximizing the surrogate encourages positive-advantage actions and discourages negative-advantage actions. Clipping limits incentives for large changes relative to the proximal policy; reference KL provides a separate anchor.

Weights are synchronized and converted for the next rollout. Matched execution and decoding can give $q=p_{\mathrm{prox}}$; quantization can break this equality before any update. At $p_{\theta }=p_{\mathrm{prox}}$, the proximal ratio is one, but $p_{\mathrm{prox}}/q$ can differ from one. Behavior correction and the update ratio therefore have separate roles (Section 5.1): numerical design changes computation, estimator correction adjusts gradient estimation, and update control constrains learning.

MIPU targets improvement in the inference policy obtained after conversion and synchronization, since it supplies the next experience [23].

Table 1. Policies and quantities used throughout the survey. Execution paths are part of policy identity.

Symbol	Meaning
$p_{\theta }$	Current learner policy under its specified execution path.
$p_k$	Learner-side policy evaluated at sampling snapshot ${\theta }_k$.
$q_k$	Actual behavior policy that sampled the token.
$p_{\mathrm{prox}}$	Policy used to define the proximal update ratio.
$p_{\mathrm{ref}}$	Policy used for KL regularization; not the behavior denominator.
${\Delta }_k$	Same-snapshot log-probability difference, $\log p_k-\log q_k$.
$d_{\theta }$	Current learner–behavior gap, $\log p_{\theta }-\log q_k$.
$A_t$	Advantage or other policy-gradient learning signal.
$s_{\theta }(a,h)$	Score vector, ${\nabla }_{\theta }\log p_{\theta }(a\mid h)$.
$c_t,{\overline{c}}_t$	Behavior-to-proximal correction and its truncated or masked form.
$r_t\left(\theta \right)$	Proximal update ratio, $p_{\theta }/p_{\mathrm{prox}}$.
$W_{\theta }(y\mid x)$	Complete-response likelihood ratio, $P_{\theta }/Q$.
$N,sg$	Valid-token count and stop-gradient operator in Equation (10).

Table 1. Policies and quantities used throughout the survey. Execution paths are part of policy identity.

Symbol	Meaning
$p_{\theta }$	Current learner policy under its specified execution path.
$p_k$	Learner-side policy evaluated at sampling snapshot ${\theta }_k$.
$q_k$	Actual behavior policy that sampled the token.
$p_{\mathrm{prox}}$	Policy used to define the proximal update ratio.
$p_{\mathrm{ref}}$	Policy used for KL regularization; not the behavior denominator.
${\Delta }_k$	Same-snapshot log-probability difference, $\log p_k-\log q_k$.
$d_{\theta }$	Current learner–behavior gap, $\log p_{\theta }-\log q_k$.
$A_t$	Advantage or other policy-gradient learning signal.
$s_{\theta }(a,h)$	Score vector, ${\nabla }_{\theta }\log p_{\theta }(a\mid h)$.
$c_t,{\overline{c}}_t$	Behavior-to-proximal correction and its truncated or masked form.
$r_t\left(\theta \right)$	Proximal update ratio, $p_{\theta }/p_{\mathrm{prox}}$.
$W_{\theta }(y\mid x)$	Complete-response likelihood ratio, $P_{\theta }/Q$.
$N,sg$	Valid-token count and stop-gradient operator in Equation (10).

2.2. Execution Mismatch Versus Policy Staleness

At a fixed prefix and token with positive relevant probabilities,

$${\displaystyle \frac{p_{\theta }(a\mid h)}{q_k(a\mid h)}}=\underset{\mathrm{parameter}\mathrm{change}/\mathrm{lag}}{\underbrace{\left({\displaystyle \frac{p_{\theta }(a\mid h)}{p_k(a\mid h)}}\right)}}\times \underset{\begin{array}{c}\mathrm{same}-\mathrm{snapshot}\\ \mathrm{execution}/\mathrm{decoding}\end{array}}{\underbrace{\left({\displaystyle \frac{p_k(a\mid h)}{q_k(a\mid h)}}\right)}}.$$

(2)

The first factor captures parameter change since collection, including asynchronous lag; no independence assumption is required. The second compares execution at the sampling snapshot, isolating numerical differences under matched decoding and including decoding effects otherwise. Implementations need not retain $p_k$. Under partial rollout, $k=k\left(t\right)$ can vary across tokens. Retained KV entries encode earlier computations, so $q_k$ abbreviates behavior dependent on the parameter versions and numerical paths that produced this state.

Figure 3 separates these snapshot comparisons from the two policy roles used by the loss. A proximal anchor and a KL reference can be chosen independently of the actual sampler.

At the sampling snapshot, the first factor is one despite possible execution mismatch. Matched execution and decoding set the second factor to one without removing lag. Numerical alignment and staleness correction address these distinct factors.

Importance sampling requires positive behavior probability wherever the target has support. Vocabulary truncation can violate this condition despite a defined denominator for every observed token. At fixed h,

$${\mathbb{E}}_{a\sim q_k(·\mid h)}\left[{\displaystyle \frac{p_{\theta }(a\mid h)}{q_k(a\mid h)}}f(a,h)\right]={\mathbb{E}}_{a\sim p_{\theta }(·\mid h)}\left[f(a,h)\right]$$

(3)

holds under support and integrability assumptions. It corrects the action distribution at a fixed prefix while leaving the distribution over prefixes unchanged.

The sampler can therefore visit histories with different frequencies from the target policy. Figure 12 illustrates this distinction, and Appendix C gives the full-response identities. In group-relative RL, changing the response distribution also changes the groups used to compute advantages [10].

2.3. A Component-Level Taxonomy of Interventions

Components are classified by their intervention target: the computation or learning rule they change. Mismatch sources explain why probabilities differ; experimental settings specify precision, adaptation, and collection. Neither determines the method class.

A: numerical design changes the computations that produce mismatch. A1 aligns learner and sampler paths; A2 preserves fidelity or parameter-update visibility after quantization. Either can act on the learner, sampler, or both. B: learning under residual mismatch changes how collected samples influence learning. B1 corrects gradient estimation; B2 constrains, selects, or accepts updates. These interventions can preserve learning without reducing the policy gap (Table 2).

Figure 4. Component taxonomy of the focal studies. Papers can occupy several primary classes through distinct components. Scheduling, supporting connections, and the QeRL exploration boundary are shown separately. Numerical regimes are independent comparison axes; Appendix A maps the discussions.

Figure 4. Component taxonomy of the focal studies. Papers can occupy several primary classes through distinct components. Scheduling, supporting connections, and the QeRL exploration boundary are shown separately. Numerical regimes are independent comparison axes; Appendix A maps the discussions.

Classification follows purpose rather than loss form. B1 corrects a specified expectation or gradient; B2 constrains, selects, or accepts updates. Both can weight a loss, and truncation can bias a B1 estimator. DPPO and DRPO change trust-region geometry (B2), while Score Centering corrects a conditional score expectation (B1) [10,22,29].

QaRL combines aligned QAT (A1) with TBPO (B2); QuRL combines update-aware quantization (A2) with adaptive clipping (B2); QUADS combines weight alignment (A1) with activation compensation (A2) [5,6,8]. Adopted corrections such as TIS are separate from these contributions. Scheduling complements both families (S; Section 5.3).

2.4. Numerical Regimes and Contribution Roles

Quantization maps values to a discrete set through a format, scale, and rounding rule. Weight quantization changes parameters shared within a sampling version; activation quantization acts on values recomputed each forward pass; KV quantization changes attention state retained across tokens. Their errors therefore arise at different stages and persist for different durations.

Post-training quantization converts a learned model without adaptation through the target quantizer; quantization-aware training (QAT) models quantization during learning. Fake quantization quantizes and dequantizes operands while matrix multiplication can remain at higher precision. A straight-through estimator supplies surrogate rounding derivatives, as in QUADS [8]. Native kernels perform low-precision arithmetic directly. QAT specifies adaptation, separately from execution speed and backward, master-weight, and optimizer precision.

QuRL adapts to quantized rollout without an explicit learner-side QAT path: the sampler is repeatedly quantized while a higher-precision learner trains on its experience [5]. Low-rank or full-parameter updates, native or simulated arithmetic, architecture, and collection protocol specify the setting independently of the intervention target.

3. How Execution Mismatch Affects Learning

3.1. Sources and Propagation of Execution Discrepancy

Scales, rounding, accumulation, kernels, batch shape, and cached state can alter probabilities at the same nominal precision. FP16-based analysis examines precision selection [16]; R3 isolates routing differences that can occur without quantization [15].

Reduction order can change even when precision is fixed. TBIK coordinates reduction trees across tensor-parallel sizes, providing a matched-execution reference [28]. Under low precision, HiFloat4’s ablations link degraded learning to outlier-driven underflow in rollout activations [24].

Shared-prefix probes fix histories; free-running comparisons also capture trajectory changes.

Quantization can hide an update within one cell or amplify a small change across a boundary. QuRL seeks to make updates visible to the sampler; TRACE reduces inconsistent rounding [5,13]. Both effects depend on the values’ positions relative to quantization boundaries.

Backward arithmetic can change updates despite forward agreement. Jet-RL specifies backward precision; Miles offers a dequantized backward path to protect gradients [3,25]. Both policy gaps and derivative computation therefore matter to low-precision failures.

Figure 5 shows how discrepancy changes sample influence, induces score drift, and alters update selection. An aggregate gap does not distinguish these mechanisms.

3.2. Distorted Estimation and Update Decisions

Discrepancy affects relative sample influence, conditional score drift, and update decisions, depending on the policy, learning signal, and objective.

Relative influence.

Writing $p=q+e$ gives $p/q=1+e/q$: the same absolute perturbation has greater relative influence when q is small. Tail and confidence-conditioned statistics expose structure hidden by a mean absolute error; CIS uses confidence in its correction [20].

DPPO distinguishes relative ratios from moved probability mass [29]. A large ratio on a rare token can move little mass, while a moderate ratio on a common token can move much more. Ratio magnitude alone does not specify an appropriate update constraint.

Conditional score drift.

Estimation can also acquire a systematic drift. For a normalized differentiable learner distribution, ${\mathbb{E}}_{a\sim p_{\theta }}\left[s_{\theta }(a,h)\right]=0$. Under behavior sampling, the corresponding expectation need not vanish. At fixed h,

$${\mathbb{E}}_q\left[A{s}_{\theta }\right]={Cov}_q(A,s_{\theta })+{\mathbb{E}}_q\left[A\right]{\mathbb{E}}_q\left[s_{\theta }\right].$$

(4)

At a fixed prefix, the expectations include the sampled action and any random continuation used to form A. Centering advantages within a prompt group need not center their conditional mean at each prefix. Score Centering removes this first-moment term without changing behavior [10]. The covariance remains under q, and prefix-distribution error is uncorrected.

Update decisions.

Mismatch can change which branch of a clipped surrogate is active and remove a token’s gradient contribution. The full-pipeline FP8 study identifies excessive clipping of negative-advantage tokens [11]. This decision error is distinct from high variance in an importance weight.

Away from the boundaries, the clipped surrogate with fixed A and $0<ϵ<1$ satisfies

$${\displaystyle \frac{\partial \ell }{\partial r}}=\left\{\begin{array}{cc}A\mathbf{1}\{r<1+ϵ\},\hfill & A>0,\hfill \\ A\mathbf{1}\{r>1-ϵ\},\hfill & A<0,\hfill \\ 0,\hfill & A=0.\hfill \end{array}\right.$$

(5)

The active branch depends on the advantage sign and the ratio being clipped. Clipping the combined learner–behavior ratio suppresses a negative-advantage token at $r=0.7$ when $ϵ=0.2$, even at the sampling snapshot. Section 5.1 separates this combined ratio from a decoupled surrogate; Figure 13 in Appendix F illustrates the branches.

3.3. Feedback Across Optimization Steps

Updates move values relative to quantization boundaries, changing scales and sampler update visibility [4,5]. The changed sampler visits different histories, altering rewards, group advantages, and subsequent data [10]. Fixed-batch probes characterize local effects without following the feedback through subsequent sampling and optimization.

The direction of an update matters as well as the gap’s magnitude. Locally, reducing a learner probability already below the behavior probability can enlarge the log gap. ACRL and TRIAGE use directional information to control updates [9,12]. In a shared-parameter network, one token’s gradient also changes other probabilities, so this local sign argument alone cannot guarantee stability.

Beyond Precision responds to this feedback with response-length-triggered learning-rate decay [18]. Its effectiveness under aggressive FP4 execution remains to be established.

3.4. Diagnosing and Measuring Mismatch

Mismatch diagnostics address computational disagreement, estimator reliability, update decisions, and learning outcomes (Table 3). Their interpretation depends on the policy pair, conditioning, and aggregation; Appendix E records these choices across the core studies.

Policy comparison and conditioning.

A controlled numerical probe fixes parameters and prefixes to compare execution paths; an online diagnostic compares the current learner with recorded behavior and can include policy lag (Equation (2)). Figure 6 contrasts them. Raw, decoded, and recomputed probabilities remain distinct; VeXact supplies a matched-execution reference [19].

KV4-RL computes full-vocabulary $D_{\mathrm{KL}}(p_{\mathrm{BF}16}\parallel q_{\mathrm{quant}})$ at identical BF16-generated prefixes from the same checkpoint, before temperature or truncation [17]. The probe isolates conditional KV distortion and excludes trajectory selection. Conditioning also differs within TRIAGE: its 30B characterization replays sampled MoE routes, whereas its 4B run measures the ordinary learner–sampler gap. Their raw magnitudes are therefore not directly comparable [12].

Probability gaps and KL answer different questions.

For a specified pair of normalized conditional policies $p,q$, let $d_t=\log p(a_t\mid h_t)-\log q(a_t\mid h_t)$ on sampled actions $a_t\sim q(·\mid h_t)$. Common summaries include

$$G_{\mathrm{abs}}={\displaystyle \frac{1}{N}}\sum _t\left|d_t\right|,{\widehat{K}}_1=-{\displaystyle \frac{1}{N}}\sum _t{d}_t,{\widehat{K}}_3={\displaystyle \frac{1}{N}}\sum _t\left(e^{d_t}-d_t-1\right).$$

(6)

Here N counts valid measured tokens, and $G_{\mathrm{abs}}$ discards sign. Under common support and finite expectations, both KL estimators target conditional $D_{\mathrm{KL}}(q\parallel p)$ averaged over observed prefixes. Finite-sample ${\widehat{K}}_1$ can be negative; every ${\widehat{K}}_3$ summand is nonnegative. Neither estimate should be conflated with reverse-direction $D_{\mathrm{KL}}(p\parallel q)$, full-vocabulary evaluation, or KL to a regularization reference.

TRIAGE and AIS report mean absolute log gaps; TRACE reports maxima and minima; QUADS tracks a maximum gap; QuRL reports behavior-to-proximal KL and a maximum ratio; MIPU uses sampled-token K3 [5,7,8,12,13,23]. Extremes can expose errors hidden by a mean, but also depend on population size and composition. Their magnitudes are comparable only under matched definitions, populations, and reductions.

Estimator and update diagnostics.

Normalized empirical effective sample size (ESS) summarizes weight concentration:

$${\mathrm{ESS}}_{\mathrm{norm}}\left(w\right)={\displaystyle \frac{{\left({\sum }_t{w}_t\right)}^2}{N{\sum }_t{w}_t^2}}.$$

(7)

P3O computes ESS from raw current-to-behavior ratios; AIS uses truncated weights [7,21]. The values can therefore differ on identical data. Uniform nonzero weights give normalized ESS one regardless of their common value, so high ESS establishes neither calibration nor gradient fidelity. Score Centering addresses the conditional score mean instead (Section 5.1) [10].

The full-pipeline FP8 study uses BF16 shadow passes to identify tokens whose active clipping branch changes under FP8, separating the two advantage signs [11]. This counterfactual over-clipping is not an ordinary clipping fraction. TRIAGE measures directional mass and local concentration from signed gaps, then tests the indicator against realized updates [12]. These diagnostics assess how discrepancy changes an update; legitimate learning can still increase disagreement.

Measurement population and aggregation.

Token and response averages weight data differently; masking can lower a gap by selection, and capping can raise ESS by removing its tail. Appendix E details the corresponding populations, reductions, and probability conventions.

Training duration and delayed instability.

Longer training tests persistence across updates; longer responses test trajectory length. Late-stage and best-checkpoint measurements need optimizer steps, rollout and learner tokens, and elapsed cost as context. Paired probes isolate execution error from lag; online diagnostics and held-out quality assess learning.

Time to failure requires a defined event and persistence window, such as sustained quality loss supported by update diagnostics. Independent runs measure variation in onset; failure-free runs establish only an observation limit. Retained contributions distinguish learning from suppression.

The distinction between execution error and policy movement determines how an online gap is interpreted. Its evolution can reflect both numerical disagreement and the intended departure of the learner from the collection policy.

Box 1. Quantization and exploration QeRL combines low-precision execution, low-rank adaptation, and adaptive quantization noise, making exploration an explicit part of the training design [2]. It occupies a boundary of this survey: the numerical perturbation changes the search process as well as agreement between computations. The reported benefit concerns structured quantization noise under low-rank adaptation and depends on the induced sampling distribution and the learner’s adaptation capacity.

The following sections examine remedies in the computations that produce disagreement (Section 4) and in the learning rules that use the resulting experience (Section 5).

4. Numerical Design and Execution Alignment

Numerical design acts on the learner, sampler, or their interface. Alignment brings corresponding computations closer; fidelity control limits information loss under low precision. Two approximate paths can agree without either approaching a higher-precision reference.

4.1. Aligning Learner and Rollout Computation

Alignment couples operator precision, quantized representations, or realized rounding outcomes. The coupled object determines the required coordination and residual discrepancy.

Jet-RL shares precision and quantization granularity across training and rollout [3]. Linear-layer GEMMs use FP8 inputs in forward and backward passes, with BF16 inter-operator gradients.

QaRL combines rollout-aligned learner QAT (A1) with TBPO (B2) [6]. QAT matches the quantized representation; TBPO controls sequence-level updates on problematic low-bit responses.

QUADS distinguishes synchronized weights from activations recomputed from inputs that can differ across engines [8]. Nearby activations can map to different codewords even under the same quantizer. It fake-quantizes learner weights, retains BF16 learner activations, and compensates selected rollout activation errors. Its asymmetric-versus-symmetric ablation shows that quantizing the same operands on both sides need not improve agreement. Section 4.2 examines the compensation.

Representation alignment leaves rounding dependent on each path’s inputs. TRACE couples realized rounding decisions [13]: with exact guidance and fixed rollout scales, it chooses the learner value’s neighboring codeword nearest the rollout codeword. Figure 7 shows how this can reduce cross-path discrepancy while slightly increasing learner quantization error.

TRACE approximates this reference by caching one low code bit and scales from the latter half of the layers by default, paired by trajectory, token position, and quantization site. Reconstruction uses the learner sign and the E2M1 magnitude nearest the scale-normalized learner activation among those consistent with the retained bits. Adjacent learner candidates are compared with this reference. The record narrows a local choice rather than recovering arbitrary rollout values independently, relying on paired activations remaining locally adjacent.

TRACE’s main rollout quantizes expert W/A and softmax-layer KV to NVFP4, retaining BF16 elsewhere. The learner uses guided fake quantization at matching sites, with BF16 matrix multiplication, attention, and backward computation. Routing replay and token-level correction address residual differences; unpaired sites and policy lag remain sources of error [13].

Alignment must survive conversion and synchronization. Miles coordinates checkpoint conversion, learner forward, rollout kernels, and live export through shared quantization rules; its MXFP8 and NVFP4 recipes include precision exceptions and are designated Beta [25]. QATFactory’s QARL uses BF16 quantization simulation in training, native NVFP4 rollout, and low-bit weight export [26].

The alignment target determines the required correspondence among tokens, modules, and parameter versions. Routing replay matches MoE choices [15]; TBIK matches reduction order [28]. Other operations require separate alignment or correction.

4.2. Controlling Quantization Error and Preserving Parameter Updates

A2 preserves information within a numerical path by retaining activation or state fidelity and making parameter changes visible after quantization.

Preserving update visibility.

Cross-path agreement and update visibility are separate numerical objectives (Figure 7 and Figure 8). Alignment alone does not ensure that the sampler can express the changes made by training. For an illustrative deterministic quantizer Q, an update can satisfy

$$Q(w+\delta w)=Q\left(w\right),$$

(8)

even when $\delta w\ne 0$. Visibility depends on update magnitude, distance to a quantization boundary, and scale behavior. QuRL’s Update-Aware Quantization applies one-time invariant rescaling before RL [5]. The identity $WX=(W/s)\left(sX\right)$ preserves the initial real-valued layer output while changing the magnitudes seen by quantization and optimization. Rescaling seeks to improve update size relative to quantization noise, but also changes gradients and potentially later optimization. UAQ targets sampler update visibility; ACR controls learner clipping (Section 5.2).

Figure 8 shows a smaller update crossing a boundary while a larger one stays within its cell. Changing the scale moves these boundaries.

Preserving activation and state fidelity.

QUADS complements learner-side weight alignment with Residual Activation Compensation, restoring selected high-error channels around native W4A4 rollout computation [8]. Compensation changes the sampler’s activation approximation.

HiFloat4 / Rollout-ResQ corrects rollout activation underflow with sparse residuals [24]. Its learner–sampler precision comparisons isolate an error that raising learner precision alone does not resolve. These FP4 experiments simulate the arithmetic and do not measure native-FP4 acceleration.

FP8-RL maintains numerical fidelity as the policy changes through blockwise W8A8 rollout, scale recalibration for FP8 attention and KV execution, and dynamic weight synchronization [4]. It integrates these components with adopted TIS/MIS correction. Its v2 experiments also explore end-to-end FP8, so learner precision varies across the paper’s experimental settings.

Miles retains selected tensors in BF16 and offers an optional dequantized backward path that trades training throughput for gradient stability without changing the quantized forward values [25]. These choices specify the fidelity of its forward and derivative computations separately.

KV4-RL uses INT4 KV with bounded INT8 protection for attention-sink and recent positions [17]. TIS remains fixed across cache-precision comparisons, isolating the benefit of changing the sampler computation. In colocated, quota-based partial collection, generation and optimization alternate; responses can span collection boundaries.

Update visibility preserves parameter changes; compensation, recalibration, and cache protection preserve information used in rollout generation [4,5,17,24]. Their runtime and coordination costs differ from quantizer matching (Table 4).

4.3. Residual Mismatch and Limits of Alignment

Alignment compares learner and sampler computation at a specified interface, snapshot, and history. Fidelity measures approximation to a higher-precision reference. Shared quantizers can still disagree because their inputs or surrounding kernels differ.

A numerically faithful sampler can still lag behind the learner. Agreement at observed prefixes also leaves differences in how often those prefixes are reached. Learning-side correction and control address the resulting effects on estimation and updates.

Precision flow requires coordinated operators, QAT requires matching quantizers, and guided rounding requires stored records paired with learner states [3,6,8,13]. Fidelity control uses calibration, compensation, or selective precision [4,5,17] (Table 4).

5. Learning Under Residual Mismatch

Learning-side methods act on collected experience: B1 corrects gradient estimation, while B2 controls how samples influence policy updates. Their information requirements and effects on bias, variance, and update selection distinguish the mechanisms within each family.

5.1. Correcting Gradient Estimation

Multiplicative correction and truncation.

Equation (3) gives the conditional change of measure. A decoupled surrogate separates behavior correction from the proximal update ratio. For a frozen proximal policy $p_{\mathrm{prox}}$ and recorded behavior q, define

$$c_t={\displaystyle \frac{p_{\mathrm{prox}}(a_t\mid h_t)}{q(a_t\mid h_t)}},r_t\left(\theta \right)={\displaystyle \frac{p_{\theta }(a_t\mid h_t)}{p_{\mathrm{prox}}(a_t\mid h_t)}}.$$

(9)

A representative decoupled surrogate is

$$\widehat{L}\left(\theta \right)={\displaystyle \frac{1}{N}}\sum _{t}sg\left({\overline{c}}_t\right)\min \left[r_t\left(\theta \right)A_t,clip(r_t\left(\theta \right),1-ϵ,1+ϵ)A_t\right],$$

(10)

where ${\overline{c}}_t$ is a possibly truncated correction, advantages remain fixed, N counts valid tokens, and sg stops gradients through the correction weight. Before truncation and clipping, $c_t{r}_t=p_{\theta }/q$. Bounding $c_t$ limits behavior correction; clipping $r_t$ limits incentives for change from the proximal policy. Upper truncation uses ${\overline{c}}_t=\min (c_t,C)$; masking discards selected ratios. Response averaging and length normalization change relative weights and therefore the estimator.

FlashRL uses actual rollout probabilities for truncated importance sampling (TIS) [1]. MiniRL distinguishes behavior and proximal policies [14]. FP8-RL adopts token-level TIS and masked importance sampling (MIS), while KV4-RL retains TIS alongside cache protection [4,17]. Truncation bounds influence but introduces bias.

Adapting the amount and location of correction.

Adaptive importance sampling (AIS) adjusts correction to its necessity and reliability [7]. Its multiplier is $1+\alpha (\overline{w}-1)$: $\alpha =0$ gives the uncorrected estimator, and $\alpha =1$ gives truncated weight $\overline{w}$. A mean absolute log gap measures the need for correction; capped-weight ESS and advantage-weight variance amplification assess reliability. AIS can reduce correction under large mismatch when weights are concentrated or noisy.

Calibrated importance sampling (CIS) adjusts the truncation cap according to confidence. It represents cross-engine perturbation as a log-odds displacement, which can produce very different ratios for low- and high-probability actions [20]. Thus, AIS adapts the mixture strength, whereas CIS adapts the cap. CIS evaluates BF16 cross-engine mismatch; transfer to FP4 remains untested.

Additive score correction.

Score Centering corrects the score vector $s_{\theta }(a,h)={\nabla }_{\theta }\log p_{\theta }(a\mid h)$, rather than a reward or sampled-token importance weight [10]. Its mean is zero under the learner distribution but can be nonzero under behavior sampling, producing the extra term in Equation (4). Prompt-group reward centering need not remove this prefix-conditional term.

Score Centering replaces the score at a prefix with

$${\tilde{s}}_{\theta }(a,h)=s_{\theta }(a,h)-{\mathbb{E}}_{b\sim q(·\mid h)}\left[s_{\theta }(b,h)\right].$$

(11)

With an exact expectation, the mean under q vanishes and the drift term in Equation (4) is removed [10]. The covariance remains under q, so the result does not recover the full on-policy gradient. The implementation reconstructs behavior using sampler top-k log-probabilities ($k=128$ by default), modeling the omitted tail with the learner distribution [10]. Correction coefficients are detached; fidelity depends on this reconstruction.

Figure 14 in Appendix F illustrates centering with both policies fixed. Estimator correction can therefore leave numerical disagreement unchanged. Under importance weighting, the center must correspond to the weighted score; applying an unweighted center to an arbitrary weighted loss need not cancel the intended term [10].

Baselines and score centering interact differently at token and sequence level [27]. For the sequence-level regularized importance estimators studied there, the group-mean baseline already supplies the centering benefit. At token level, that baseline generally leaves bias under regularized ratios. Centering must match the estimand, weights, and reduction.

TIS, AIS, and CIS require sampled-action probabilities; Score Centering reconstructs a behavior law. Truncation and masking trade tail control for bias.

5.2. Controlling the Influence of Policy Updates

Update controls differ in their decision rule, control signal, and application unit (Table 5). Signal and application granularity can differ: P3O uses batch statistics and ACRL uses response statistics to guide token contributions [9,21].

Restoring or reshaping update constraints.

QuRL’s Adaptive Clipping Range (ACR) uses proximal and behavior probabilities to reshape a region already distorted at collection [5]. UAQ (Section 4.2) makes the resulting weight changes visible to the quantized sampler. The two components address learning an update and expressing it through quantization.

QaRL’s TBPO constrains whole responses through negative-sample treatment and a sequence-level trust band [6]. Full-pipeline FP8 instead calibrates clipping against BF16: it matches the lower quantile on negative-advantage tokens, then adjusts the upper boundary to rebalance signed contributions [11]. Periodic reference passes amortize calibration of the update boundary.

Clipping statistics require the advantage sign, reduction, and comparison policy to be specified. An aggregate fraction cannot reveal which useful contributions a boundary removes.

Adapting control to batch reliability.

P3O uses normalized effective sample size (ESS) of the batch’s policy ratios to adapt a score-weight cap and off-policy regularization [21]. Its FP8-rollout experiment studies numerical mismatch directly, while broader comparisons show dependence on the model and collection setting. AIS adapts an estimator mixture; P3O adjusts update constraints.

Choosing the constraint geometry.

DPPO introduces a divergence-based trust region [29]. DRPO retains Binary-TV geometry but replaces hard masking with a smooth regularizer that supplies corrective gradients outside the boundary [22]. DRPO evaluates this update geometry in FP8-rollout-only and end-to-end FP8 MoE settings.

Directional control.

ACRL steers disagreement through advantage-guided token reweighting controlled by response-level discrepancy from a reference [9].

TRIAGE combines advantage sign with the learner–sampler log gap [12]. In our notation, $A_i{d}_{\theta ,i,t}$ identifies the sign of a simplified token’s own mismatch-energy contribution, where $d_{\theta }=\log p_{\theta }-\log q_k$. Cross-token coupling and optimizer history can alter the realized motion. Both $A_i>0,d_{\theta }>0$ and $A_i<0,d_{\theta }<0$ are locally amplifying; TRIAGE selectively gates the latter because increasing a reward-positive response’s probability can be useful even when its gap grows.

The gate first diagnoses 64-position segments of negative-advantage responses from their mean gap and deep negative tail. Within an eligible risky segment, only negative-gap tokens receive an attenuated raw weight $b_{i,t}$. The weighted response loss is normalized by the sum of its weights. Relative to ordinary response averaging, its effective multiplier is therefore $b_{i,t}/{\overline{b}}_i$, not simply $b_{i,t}$. Normalization reallocates contributions: for illustrative raw weights $(0.3,1,0.3,1)$, the effective multipliers are approximately $(0.46,1.54,0.46,1.54)$. If every weight is equal, the normalization cancels it. This controls relative influence, not the total gradient norm.

Repair targets positive-advantage responses with sufficiently negative mean gaps in 128-position segments. A one-sided pseudo-Huber penalty raises learner log probabilities. Its derivative bounds the additive coefficient, not the full gradient or every token’s gap. All valid tokens in an active segment share that coefficient, including positive-gap tokens. Gating and repair thus address different signed contributions [12]. The tested recipe uses native W4A4 forward on both endpoints and BF16 KV. Applying it to KV4 remains an untested transfer.

Accepting whole updates in the inference policy.

MIPU constructs candidates relative to the sampler and assesses each after synchronization and numerical conversion [23]. Its inference-gap proxy supports acceptance or rollback in the evaluated FP8 runs; a general monotonic-improvement guarantee requires stronger conditions.

Token and segment rules localize effects; batch aggregation pools evidence; whole-policy acceptance assesses the synchronized result. Granularity thus trades localization against statistical reliability, while clipping and regularization determine enforcement.

5.3. Adapting the Optimization Schedule

Beyond Precision adapts learning rates to evolving optimization and mismatch [18], complementing numerical alignment and sample-level control. Direct aggressive-quantization evidence remains limited. Reduced update sizes can delay divergence and learning alike.

5.4. Compatibility, Bias, and Failure Modes

Complementary methods address different residual errors. Weight alignment can leave activation error, while a more faithful sampler can still generate off-policy data. QaRL, QuRL, and QUADS test combinations addressing such residual effects [5,6,8].

Estimator compatibility depends on weighting, centering, masking, and reduction. Token- and sequence-level regularized estimators can require different centers [27]. Figure 9 distinguishes this estimation target from the contributions selected by update control.

Score Centering removes a conditional score mean; TRIAGE selects contributions using advantage and log-gap signs. Their joint effect remains untested [10,12].

Correction uses each token’s generation policy; weighted scores need matching centers [10]. Masks can overlap, while guidance and reference passes add costs (Table 6).

6. Evidence Synthesis and System Trade-Offs

Published comparisons relate numerical agreement, policy quality, and cost within specific execution settings, evaluators, and training budgets.

6.1. Experimental Regimes

A higher-precision learner and quantized sampler differ in representation; quantizing both sides can still leave different kernels, activations, and updates. KV quantization changes retained state, while W/A quantization changes other computations. Low-rank and full-parameter adaptation also differ in their capacity to absorb these errors.

AIS evaluates diffusion-based LLaDA alongside autoregressive Qwen models [7]. The Qwen experiments follow the token-level formulation used here; the denoising results require likelihoods defined for their generation process.

KV4-RL evaluates learned parameters through a common BF16 path [17]; TRACE evaluates the FP4 policy [13]. The same checkpoint can therefore receive different quality assessments.

Routing, cache costs, and version mixing under partial collection affect mismatch and cost.

Miles contributes production-scale agentic evidence, while QATFactory isolates a smaller NVFP4 QARL workload [25,26]. Native arithmetic, quantization simulation, and BF16 computation on quantized operands remain distinct; HiFloat4’s FP4 experiments are simulated [24].

Table 7 spans shared FP8 execution, aligned QAT, simulated FP4, native FP4, estimator correction, and cache-focused rollout. Appendix B summarizes the wider setting axes (Table 12).

6.2. Findings on Stability and Learning Quality

Component comparisons connect changes in numerical execution or gradient estimation to learning outcomes. Figure 10 maps the TRACE and TRIAGE baselines; Table 9 compares their findings with Score Centering and KV4-RL. Appendix E details the corresponding measurements.

Numerical agreement and learning quality.

Policy entropy, mean reward, and probability gaps summarize the token distribution, sampled performance, and learner–sampler disagreement (Table 8). Their interpretation depends on the policy pairs and reductions in Section 3.4.

TIS prevents aggregate-gap divergence in TRIAGE’s 30B experiment but trails the full method in late reward and downstream scores [12]. The full method adds direction-dependent gating and repair, whose effects extend beyond aggregate gap control.

AIS uses a gap to adapt correction, KV4-RL uses paired KL to isolate cache distortion, and TRIAGE uses signed gaps to select contributions [7,12,17].

Persistence over training.

As the policy and numerical state evolve, early stabilization can give way to renewed disagreement or quality loss. QuRL’s disagreement after initial stability motivates joint attention to clipping and update visibility [5]. Adaptive correction, directional control, and centering address different feedback mechanisms [7,9,10]; P3O’s Qwen3-8B FP8 experiment shows preserved later performance over its evaluated horizon [21].

TRACE reports growing discrepancy and later quality loss for conventional FP4 QAT in its joint W/A/KV setting [13]. TRIAGE does not evaluate substantially longer training [12].

Alignment depends on the shared numerical interface.

QUADS’ asymmetric weight QAT recovers much of the lost performance, with further gains from rollout activation compensation; quantizing more learner operands can worsen agreement [8]. HiFloat4’s simulated ablations also locate residual error in rollout activations [24]. The relevant alignment target can therefore be operator precision, a quantized representation, or a realized rounding decision [3,6,13]. Calibration and conversion also belong to that interface [4,26].

Learning effects depend on update decisions.

The full-pipeline FP8 study links excessive lower-side clipping of negative-advantage tokens to pathological training dynamics; TRIAGE selects interventions using local, directional information [11,12]. DRPO preserves corrective gradients outside its trust region; MIPU assesses a whole candidate after conversion to the inference policy [22,23]. These controls act at different locations and stages, which aggregate clipping fractions or mean gaps cannot distinguish. TRIAGE’s routing-conditioned MoE analysis also finds that a population-level directional trend need not predict every token’s motion. Checkpoint selection further affects the comparison: the full-pipeline FP8 study chooses its main GRPO endpoints by best average performance over 500 steps [11], whereas TRIAGE uses the final scheduled step for stable arms. Similar endpoint scores can summarize different learning trajectories.

Table 9. Within-study evidence with mechanistic interpretations. The controlled setting, checkpoint rule, and evaluation protocol bound each conclusion.

Study	Controlled comparison	Reported result	Evidence and scope
TRACE [13] 35B MoE	Joint NVFP4 W/A/KV rollout; TIS+SC uses cap 2 and top-128 reconstruction.	Four-task mean: TRACE 75.3, TIS+SC 74.1, BF16 74.9.	Recipe comparison in the joint W/A/KV regime. SC uses its best mean checkpoint through 400 steps; task-wise gains are mixed and aggregate uncertainty is unreported.
TRIAGE [12] 30B MoE	BF16, native NVFP4, TIS, and full gating/repair recipe. Stable arms use step 1700.	Five-task mean: TRIAGE 70.96, TIS 63.18, BF16 72.41.	Stabilization comparison beyond TIS in native W4A4. SC/TIS+SC, TRACE, and KV4 are outside this comparison.
Score Centering [10] 30B MoE	Correction rules share the objective; quantization and lag are tested separately.	With FP4 KV, SC and TIS remain stable; stronger INT8/INT4 stress separates them further.	Correction-rule comparison under a shared objective. Curves report training accuracy; shared-objective controls do not reproduce every baseline recipe.
KV4-RL [17] 8B dense	Cache precision varies with TIS retained; all learned policies are evaluated in BF16.	At 400 updates, six-task mean: 52.8 versus BF16’s 52.4; cumulative full-iteration speedup 1.27×.	Cache-precision control with TIS retained. Common BF16 evaluation assesses learned parameters; evaluation-seed variation differs from training-run variation.

Table 9. Within-study evidence with mechanistic interpretations. The controlled setting, checkpoint rule, and evaluation protocol bound each conclusion.

Study	Controlled comparison	Reported result	Evidence and scope
TRACE [13] 35B MoE	Joint NVFP4 W/A/KV rollout; TIS+SC uses cap 2 and top-128 reconstruction.	Four-task mean: TRACE 75.3, TIS+SC 74.1, BF16 74.9.	Recipe comparison in the joint W/A/KV regime. SC uses its best mean checkpoint through 400 steps; task-wise gains are mixed and aggregate uncertainty is unreported.
TRIAGE [12] 30B MoE	BF16, native NVFP4, TIS, and full gating/repair recipe. Stable arms use step 1700.	Five-task mean: TRIAGE 70.96, TIS 63.18, BF16 72.41.	Stabilization comparison beyond TIS in native W4A4. SC/TIS+SC, TRACE, and KV4 are outside this comparison.
Score Centering [10] 30B MoE	Correction rules share the objective; quantization and lag are tested separately.	With FP4 KV, SC and TIS remain stable; stronger INT8/INT4 stress separates them further.	Correction-rule comparison under a shared objective. Curves report training accuracy; shared-objective controls do not reproduce every baseline recipe.
KV4-RL [17] 8B dense	Cache precision varies with TIS retained; all learned policies are evaluated in BF16.	At 400 updates, six-task mean: 52.8 versus BF16’s 52.4; cumulative full-iteration speedup 1.27×.	Cache-precision control with TIS retained. Common BF16 evaluation assesses learned parameters; evaluation-seed variation differs from training-run variation.

TRACE tests guided alignment against QAT and TIS+SC in joint low-precision execution; TRIAGE tests recovery beyond TIS in native W4A4. These different baseline sets and precision regimes preclude a direct TRACE–TRIAGE ranking [12,13].

Residual errors motivate component combinations.

Ablations identify which residual error makes a second component useful. QUADS separates weight alignment from activation compensation, QuRL combines update visibility with clipping control, and KV4-RL changes cache fidelity while retaining TIS [5,8,17]. These results support the tested combinations; extending them requires identifying the residual error in the new setting.

6.3. Efficiency Gains and Implementation Costs

Under serialized execution, total iteration time is

$$T_{\mathrm{iter}}=T_{\mathrm{rollout}}+T_{\mathrm{learner}}+T_{\mathrm{sync}}+T_{\mathrm{quant}/\mathrm{calib}}+T_{\mathrm{other}},$$

(12)

with synchronization and quantization/calibration accounted for separately. Accelerating one phase increases the relative cost of the others. Overlapped execution instead depends on the critical path, for which the additive model in Equation (12) does not apply.

For fixed workload and remaining phases, rollout fraction f, rollout speedup s, and added work $h{T}_0$ give $T/T_0=(1-f)+f/s+h$ (Figure 11). Overlap, response-length changes, or altered concurrency require a different model.

Mitigation adds scale estimation and conversion in FP8-RL [4], learner passes for reference calibration, storage and transport for guided rounding, and computation in selectively protected regions. Table 10 connects these costs to information requirements.

Information requirements constrain practical scope. P3O needs batch ratio statistics; MIPU adds candidate assessment and possible rollback [21,23]; Miles and QATFactory require coordinated conversion and inference-ready weight transfer [25,26]. TBIK’s reduction coordination can cost kernel performance [28]. Rollout-ResQ targets efficient sparse correction, but its simulations do not establish native-FP4 speedup [24].

The useful quantization target depends on the workload as well as the format. QUADS targets native W4A4 expert computation, whereas KV4-RL studies how sustained concurrency and long contexts increase the value of reducing KV processing [8,17]. Their settings motivate different numerical targets; a bit-width comparison alone does not identify the dominant cost.

Response length and memory-dependent concurrency also change total work. PagedAttention illustrates how cache management and scheduling improve serving efficiency [32].

KV4-RL reports $1.80\times$ cumulative rollout speedup and $1.27\times$ cumulative full-iteration speedup over BF16 rollout across 400 Qwen3-8B-Base iterations [17]. The difference reflects the cost of the remaining phases. Its common-BF16 evaluation separately compares learned-policy quality, with both results tied to the evaluated collection workload.

Three complementary comparisons relate quality to cost [17,36]: quality at a fixed budget, cost to a target quality, and extended-budget trajectories. Comparisons specify the evaluator, include extra passes and response-length changes, and account for failures to reach the target. Under overlap, guidance traffic can also affect the critical path.

6.4. Evidence Limits and Coverage Gaps

Controlled execution settings provide the clearest mechanism evidence. Mathematical reasoning dominates the collection; interactive tasks and hardware transfer have fewer controlled comparisons. Native arithmetic, simulation, and BF16 computation on quantized operands must be distinguished.

Comparisons depend on horizon, budget, checkpoint selection, and uncertainty units (Section 3.4). Best-checkpoint scores can hide later loss. Repeated decoding measures evaluation variation; independent RL runs measure training variation. Smoothing estimates neither.

Separate W4A4 and KV4 studies do not establish their joint behavior; asynchronous demonstrations do not isolate every precision–lag interaction. Scheduling needs direct low-bit validation, while diffusion needs generation-specific likelihoods before the autoregressive analysis can transfer.

These differences in execution and evaluation limit cross-study comparisons. A controlled comparison can isolate an intervention’s effect by matching the numerical setting, base objective, and resource budget, then measuring the resulting policy quality and additional cost.

7. Open Problems

Sustained learning, mitigation costs, method composition, and transfer remain open problems.

7.1. Predicting Harmful Mismatch and Sustaining Learning

Predictive diagnostics should detect lost learning before held-out quality deteriorates. Ratio tails, conditional score drift, clipping errors, and segment concentration provide mechanism-specific signals. Useful diagnostics must distinguish harmful mismatch from ordinary policy movement and changes caused by sample selection. Paired probes isolate execution, fixed-batch audits isolate update effects, and continued training tests the timing and severity of deterioration.

Extended runs distinguish continued improvement, delayed deterioration, and stagnation as policies and numerical states evolve. Recalibration, guidance allocation, and update control trade persistent gains against adaptation cost.

ACRL identifies a specific durability question: its reference discrepancy is fixed at initialization, and substantially longer training may require a moving baseline [9]. Score Centering notes that its severe-mismatch, short-sequence experiments proxy longer training under milder mismatch [10]. Whether these stress tests predict long-run failures remains unresolved.

7.2. Composing Numerical and Learning Interventions

Weight alignment can leave activation error; estimator correction may reduce the need to suppress updates. Score Centering leaves a behavior-weighted reward–score covariance, motivating importance weighting under severe staleness [10]. Guided rounding with centering and cache protection with directional control remain untested (Table 6). Precision, guidance, and correction can share a budget, with ablations distinguishing error reduction, retained contributions, cost, and retuning.

7.3. Reducing Information and Control Costs

TRACE records low code bits and scales for selected layers [13]. Compression or selection by layer, token, or training phase could retain alignment benefits with lower acquisition, storage, transfer, reconstruction, and synchronization costs.

TRIAGE combines segment tests, token gates, normalization, and repair. Adapting control to mismatch magnitude, locality, and persistence [12] could retain gains with less tuning and overhead.

QaRL proposes token-level approximations to its sequence objective; MIPU identifies cheaper candidate assessment as an open choice [6,23].

7.4. Modeling Stateful and Asynchronous Behavior

TRIAGE leaves asynchronous validation open; KV4-RL identifies disaggregated systems, other attention architectures, and multi-turn workloads as extensions [12,17]. These settings add dependencies on parameter versions, cache history, and external state.

Responses can cross synchronization boundaries, while retained KV entries encode earlier computations. Correction and controlled comparison therefore need sufficient token, version, and cache provenance (Equation (2)) without retaining the full execution history.

Early perturbations change later prefixes beyond conditional token correction. Precision allocation should therefore follow learning benefit across layers, positions, and training phases. Multi-turn workloads extend this dependence to tool observations and state transitions.

7.5. Generalization and Quality–Cost Evaluation

Transfer depends on the availability of behavior information and numerical interfaces. AIS leaves quantization-driven exploration open [7], where exploration, smoothing, and weighting require separate ablations. Component comparisons can share execution settings and objectives; complete-system comparisons retain each method’s essential design under matched resources. Quality at a fixed budget and cost to a target quality capture the resulting trade-offs (Section 6).

8. Conclusion

Low-precision RL couples rollout computation, gradient estimation, and policy updates. Separating computation alignment, fidelity and update visibility, gradient correction, and update control explains both combined methods and their residual errors.

Assessing an intervention requires linking its effects on estimation and updates to policy quality and total cost. Lower-cost mitigation, reliable long-horizon diagnostics, and comparisons across matched execution settings remain central research directions.