Efficient On-Policy Distillation for Language Models: A Survey of Costs, Trade-Offs, and Evaluation

1. Introduction

Knowledge distillation makes a teacher’s behavior available to a student through training signals [1]. In autoregressive language models, the prefixes used during training are consequential: a student deployed on its own outputs can visit states absent from teacher demonstrations. Imitation-learning approaches address this mismatch by obtaining supervision on learner-induced states [2,3]. Generalized knowledge distillation (GKD) and MiniLLM develop this idea for language models, with different treatments of divergences and trajectory gradients [4,5]. OPD has subsequently become part of practical language-model post-training, including the Qwen3 pipeline [6].

The computational appeal is conditional. Dense teacher feedback can make each trajectory informative, but acquiring that feedback requires generation, scoring, optimization, and coordination. A change that reduces one component can increase another. Prefix truncation changes the visited states; replay changes their freshness; reward shaping changes both convergence and response length; asynchronous execution changes utilization and policy lag. Even the word efficient is ambiguous: an update can fit in less memory while taking longer, and a compressed model can require enough additional inference samples to erode its serving advantage.

We organize the literature around a practical question: What is the least resource expenditure needed to achieve a specified capability, with a specified deployment budget? Figure 1 separates six tracks while retaining their dependencies. Our synthesis has three deliverables. First, a cost-accounting framework connects method choices to the resources they consume. Second, a source-versioned ledger makes reported savings inspectable through model pairs, hardware, horizons, timing boundaries, and quality conditions. Third, an evaluation protocol distinguishes fixed-workload measurements, learning curves, and deployment amortization. The contribution is this operational synthesis, not a new distillation algorithm or an empirical ranking of incompatible results.

Relation to existing surveys.

The April survey of Song and Zheng [7] already discusses FLOPs, memory, and compute–quality trade-offs in v1 (April 1). Its v5 (October 21) expands training dynamics and efficiency, including horizon control, stale replay, sparse supervision, and preparation costs. We compare against that latest version, not only its original release. Zhang [8] organize OPD by feedback-to-update formulations and explicitly discuss estimator and evidence boundaries. Zan et al. [9] offer another broad taxonomy with rollout-cost discussion. These contributions establish much of the conceptual groundwork. Our narrower emphasis is a structured, quantitative audit and a reusable experimental comparison procedure across the six resource tracks. Table 1 identifies the organizing differences without claiming that earlier surveys omit efficiency.

Scope and evidence.

We conduct a targeted scoping review, using keyword search and citation tracing through October 25, 2026. Inclusion requires relevance to teacher-guided student trajectories, their resource cost, or a necessary methodological/system foundation. We prioritize primary papers, proceedings, and official implementation documents, and pin versions for the central comparisons. We distinguish direct OPD results from enabling systems and adjacent offline distillation. The companion catalog records these roles; Appendix A describes selection and its limitations. We make neither an exhaustive-recall claim nor a meta-analytic estimate of a universal speedup.

2. OPD Under a Resource Budget

2.1. Separate States, Targets, and Updates

Let $p_{\theta }$ be the student, $q_{\varphi }$ the teacher, $x\sim {\mu }_k$ a prompt at round k, and $h=(x,y_{<t})$ a prefix. The behavior policy $b_k$ induces a normalized prefix distribution $d_{b_k}$ under a declared time/horizon convention. A common fixed-state surrogate is

$${\mathcal{L}}_k\left(\theta \right)={\mathbb{E}}_{x\sim {\mu }_k,h\sim d_{b_k}(·\mid x)}\left[w_k\left(h\right)D(p_{\theta },q_{\varphi };h)\right].$$

(1)

States, weights, and teacher targets are fixed during differentiation, including detached shared-weight self-teachers. The per-state divergences are

$$\begin{array}{cc}\hfill D_{\mathrm{F}}(p,q;h)& =\sum _{a}q(a\mid h)\log {\displaystyle \frac{q(a\mid h)}{p(a\mid h)}},\hfill \end{array}$$

(2)

$$\begin{array}{cc}\hfill D_{\mathrm{R}}(p,q;h)& =\sum _{a}p(a\mid h)\log {\displaystyle \frac{p(a\mid h)}{q(a\mid h)}}.\hfill \end{array}$$

(3)

Current-student states do not force reverse KL, and forward KL does not force teacher-generated trajectories. Full-vocabulary losses, selected-logit approximations, and sampled-token policy-gradient losses further differ in information access and computation. Figure A1 and Appendix B disentangle these choices. GKD’s fixed-sampling treatment and MiniLLM’s trajectory-level formulation should not be treated as one exact gradient estimator [4,5].

We use strictly current for newly sampled $b_k=p_{{\theta }_k}$, recent for a lagged student, and frozen for an initial-student pool. These descriptions are more informative than a binary on/off-policy label. Teacher scores can be refreshed on old states without refreshing the state distribution. Likewise, teacher-guided continuation can change the behavior policy even when the prefix began with the student.

2.2. Define the Efficiency Denominator

For an additive resource r, such as accelerator-hours or expenditure, write

$$\begin{array}{cc}\hfill {\mathcal{C}}_{\mathrm{train}}^{\left(r\right)}=& {\mathcal{C}}_{\mathrm{prep}}^{\left(r\right)}+{\displaystyle \sum _{k=1}^K}({\mathcal{C}}_{\mathrm{gen},k}^{\left(r\right)}+{\mathcal{C}}_{\mathrm{score},k}^{\left(r\right)}\hfill \\ \hfill & +{\mathcal{C}}_{\mathrm{update},k}^{\left(r\right)}+{\mathcal{C}}_{\mathrm{coord},k}^{\left(r\right)}).\hfill \end{array}$$

(4)

Preparation includes prompt synthesis, warm-up, cached trajectory collection, and any teacher or questioner adaptation attributed to the run. Evaluation and hyperparameter-search costs should be reported separately and included whenever the claimed boundary is the full experiment. FLOPs, tokens, money, and accelerator-hours are different resources, not interchangeable measurements.

Wall-clock time follows the critical path, not generally Eq. 4. Report per-device peak memory; an aggregate peak is the maximum simultaneously resident memory across devices. Neither is a sum of phase peaks. We report a resource vector. For a prespecified resource r, quality metric Q, and serving budget $B_{\mathrm{serve}}$, define

$$C_A^{\left(r\right)}(q^{*};B_{\mathrm{serve}})=\underset{k:Q(A_k;B_{\mathrm{serve}})\ge q^{*}}{\inf }{\mathcal{C}}_{A,\le k}^{\left(r\right)},$$

(5)

with $+\infty$ for an unreached target. Report other resource coordinates at that checkpoint; elapsed time must be measured through the critical path. Estimation requires fixed validation and checkpoint-selection rules. Better quality per example need not mean better quality per accelerator-hour.

3. Six Tracks of Efficient OPD

3.1. Data: Information per Acquired Task

Data efficiency concerns the capability obtained from a limited task pool, annotation budget, or synthesis budget. These denominators differ from generated completion tokens. Reusing one prompt with many student continuations can reduce unique-prompt requirements while leaving rollout expenditure high. Earlier active-distillation work provides useful acquisition ideas, but results for classification or offline explanations do not directly establish OPD savings [10,11,12].

Data-Free OPD constructs prompts from teachers and studies whether external datasets are necessary for capability integration [13]. Here “data-free” concerns the external task corpus; teacher pretraining, generation, and student optimization remain resources. Its latest version uses format/length filtering and deduplication, so it should not be described as literally unfiltered synthesis. Ask-to-Distill trains a questioner with a frozen teacher–student distributional proxy for learning value, without retraining the student for each candidate [14]. This adds questioner-training and proxy-estimation costs.

These works motivate a distinction between difficulty and transfer value. A task can be hard because the student lacks prerequisites, because the teacher is unreliable, or because a useful skill is within reach. Difficulty alone does not identify which explanation holds. Moreover, approximately token-matched prompt banks can still yield different transfer [15]. Acquisition comparisons must control the model pair, downstream optimization, generated tokens, and evaluation distribution. Charge generator preparation once or amortize it over declared reuses. Diversity and decontamination also matter: familiar benchmark-shaped questions can inflate apparent sample efficiency without establishing wider transfer.

3.2. Rollout: Depth, Freshness, and Reuse

Generating student trajectories incurs sequential decoding and, for agents, environment interaction. Prefix-OPD uses short reasoning prefixes; ESR stops generation early to concentrate on informative initial supervision [16,17]. These methods avoid producing suffixes, unlike loss masks applied after a full trajectory has already been generated. Their benefit must be evaluated against the state coverage removed by truncation. STRIDE further studies adaptive stopping and prefix restart [18].

Freshness offers another control axis. Semi-OPD trains from a fixed pool generated by the initial student, whereas AsyncOPD overlaps generation and learning with delayed student policies [19,20]. The former trades tracking for reuse; the latter trades synchronization for bounded or measured lag. DistiLLM and DistiLLM-2 also motivate treating reuse and collection cadence as explicit method variables [21,22]. Neither cached prefixes nor importance weighting automatically recover missing future states.

Speculative generation requires identifying the distribution being preserved. Exact speculative decoding targets a designated model distribution [23]; accepting tokens according to teacher preferences can instead change the behavior distribution. Flash-OPD couples verification and distillation to reduce redundant work, but verification itself processes tokens and can increase teacher-scoring cost [24]. Draft-OPD concerns training draft models for downstream speculative inference; its deployment gain is not an OPD-training speedup [25].

For multi-turn agents, tokens are an incomplete workload measure. ReOPD reuses prefixes to access later interaction states, while turn-aware approaches control the depth and allocation of supervision [26,27,28]. Comparisons should count tool calls, environment resets, prefix acquisition, and long-horizon success in addition to language-model tokens.

3.3. Teacher: Useful Feedback per Query

Teacher cost depends on both the number of evaluated positions and the interface. A locally hosted teacher can score a student sequence in a batched forward pass; generating a new teacher solution adds sequential decoding. Full distributions, top-k distributions, selected-token probabilities, and text-only feedback expose different information. Restricting the returned logits may reduce transfer and retained activations without reducing the teacher’s Transformer computation. A top-k objective must specify how omitted probability mass is treated.

Teacher selection adds a quality–cost trade-off. MOPD combines specialist teachers; Open-MOPD, ROAD-MOPD, TRUST-MOPD, and MOPD-Router examine capability imbalance, cross-teacher constraints, reliability, or routing [29,30,31,32,33]. Their efficiency cannot be summarized by the number of teachers stored. One must count how many teachers score each trajectory, whether routing needs additional inference, and how much work is reused. Consensus supervision can also require more teacher passes [34].

Reliability-aware allocation aims to avoid spending updates on misleading feedback. Teacher gating and sparse probing connect feedback acquisition to estimated usefulness [35,36]. SCOUT goes further by adapting the teacher on student-induced prefixes, improving compatibility at the cost of teacher training and verifier access [37]. These methods belong on a cost–quality frontier even when a single update becomes more expensive. The operative question is whether their additional supervision reduces the total expenditure to reach the desired quality.

3.4. Optimization: Progress per Unit of Work

Optimization efficiency has two components: the cost of an update and the number of updates needed. Their product, together with preparation and rollout, determines the training budget. REOPOLD relaxes imitation through reward control and selective learning; PowerOPD replaces the unbounded log-ratio signal with bounded probability-power differences [38,39]. Both motivate examining gradient variance, response length, and convergence jointly rather than treating fewer steps as a sufficient cost claim.

Divergence design changes what knowledge is transferred. EOPD adds forward-KL supervision at high-entropy teacher positions, while hybrid policy distillation combines divergence and sampling choices [40,41]. Evidence on factual knowledge versus compositional skills further cautions against equating lower imitation loss with uniformly improved capability [42]. Efficiency claims should therefore preserve the intended quality vector, including inference diversity when relevant.

G-OPD/ExOPD alter reference and reward weighting, and OPD² extracts a signal from the difference between a tuned teacher and its base model [43,44]. Additional reference-model forwards must enter the cost ledger. Dr. OPD optimizes supervision weights through a performance-oriented objective, introducing another allocation mechanism whose extra computation must be counted [45].

SparseOPD separates dense correction construction from sparse differentiation [46]. Its results illustrate why memory and speed require separate reporting. Parameter-space analyses find sparse or concentrated OPD changes, but masks identified after a full training run do not by themselves supply a cheap prospective training procedure [47]. LoRA and QLoRA reduce trainable state and storage under specific adaptation recipes; they do not remove rollout or teacher costs [48,49]. Simple-OPD also makes warm-up a first-class part of the recipe rather than an uncharged prerequisite [50].

3.5. Systems: Utilization and Data Movement

OPD alternates execution modes with different bottlenecks: autoregressive decoding, teacher scoring, and backpropagation. vLLM and SGLang improve serving and prefix management; FlashAttention reduces attention memory traffic [51,52,53]. HybridFlow/verl and OpenRLHF coordinate distributed post-training components [54,55]. These are enabling systems, not independent demonstrations of OPD learning efficiency.

OPD-specific systems such as Lightning-OPD and KDFlow address execution and resource organization [56,57]. The evaluated objective matters: an off-policy forward-KL benchmark cannot be relabeled as a measured on-policy speedup merely because the system also supports OPD. Asynchronous execution can shorten elapsed time while using more devices or increasing policy lag. Figure A2 illustrates this distinction; Appendix C gives the accounting assumptions.

Sharding, fused kernels, and vocabulary-memory optimizations can remove implementation overhead without changing the task distribution [58,59,60]. Nevertheless, a cross-entropy optimization is not automatically a complete full-vocabulary KL implementation. Teacher scoring also needs target probabilities at particular prefixes, so KV-cache availability alone does not establish reusable logits. Reports should pin inference-engine versions, scoring interfaces, and synchronization behavior.

3.6. Deployment: Amortized Cost of Capability

The training objective is often motivated by a cheaper deployed model, but reduced parameter count is only a proxy. Serving depends on active parameters, context length, KV storage, generated tokens, sample count, retries, and hardware. Test-time scaling studies show that OPD improvements at a small sampling budget need not persist at larger budgets [61]. Thus deployment should be evaluated at the intended inference budget, not only with one favorable accuracy metric.

For additive resource units, lifetime cost is

$${\mathcal{C}}_{\mathrm{life}}\left(N\right)={\mathcal{C}}_{\mathrm{train}}+N\mathbb{E}\left[{\mathcal{C}}_{\mathrm{serve}}\right],$$

(6)

where N is the number of deployed requests and serving includes the complete decision procedure. Comparing two quality-matched systems yields a break-even deployment volume (Appendix C). Reusing a trained questioner, prefix pool, or adapted teacher raises an analogous amortization question. A preparation-heavy recipe can be appropriate for repeated deployments while remaining expensive for a single small experiment.

4. What the Efficiency Evidence Establishes

We audit reported quantities with their experimental boundaries. The catalog contains 88 sources; the ledger contains 28 records, some retaining several explicitly distinguished comparisons from one paper. These counts describe the included evidence, not independent experiments. Table 2 and Figure 2 give complementary views; Appendix E retains source locators and further configurations. Derived ratios are arithmetic summaries, not independent reproductions.

4.1. Three Levels of an Efficiency Claim

A reported saving can be valid while answering a narrower question than the reader assumes. We therefore separate three tests.

Resource identification.

Establish the unit, baseline, workload, and measurement boundary. Extra backward allocation, whole-step peak memory, and complete-run accelerator-hours describe different effects. A single-update profile can establish a resource difference for its configuration without establishing a learning benefit. Measured quantities and analytical estimates remain separate.

Quality identification.

Check whether cost and quality describe the same recipe: model checkpoints, adaptation regime, sequence limits, and evaluation budgets must correspond. Joining a full-parameter memory profile to a separate adapter experiment would manufacture a cost–quality point. Even a matched final cost and accuracy cannot locate the first crossing of a quality target without intermediate evaluations.

Mechanism identification.

A complete recipe may improve cost and quality while changing its rollout horizon, objective, and execution schedule together. Such recipe-level evidence is useful; attributing the gain to one component requires a control that isolates it. Conversely, a faster kernel on fixed trajectories need not make the learning procedure reach a target sooner. These three tests distinguish what was saved, what capability was obtained, and which intervention explains the change.

4.2. Lessons from Reported Comparisons

Changing the workload changes the claim.

Semi-OPD’s largest cited 11.4× GPU-hour reduction uses a 16K cap for OPD and a 4K cap for Semi-OPD; its matched-16K example supports a different, 7.6× comparison. Both are informative, but only the latter holds that cap fixed. Likewise, generator-token efficiency and downstream student-token efficiency must be separated. Ask-to-Distill explicitly notes that its downstream token curves do not establish lower end-to-end cost [14].

A saved tensor is not a saved training loop.

SparseOPD’s 70.5% reduction is extra backward allocation above the pre-step baseline. In the same 8K profiling configuration, complete-update time increases by 14.5%. The memory study uses full-parameter training and one measured update per configuration, separately from its LoRA accuracy experiments. This is useful memory evidence with a narrower scope than a full training-run result. Figure 2 visualizes such distinctions.

Faster generation can move the bottleneck.

Flash-OPD reduces overall step time despite increased teacher time in the displayed pair. MOPD-Router provides a related denominator lesson: total training cost and time per generated token respond differently when methods change output length [33]. These observations justify recording generated, scored, retained, and learner-used tokens separately. Discarded drafts and verification passes are work even when they do not appear in the final training sequence.

Measurement boundaries determine interpretation.

Prefix-OPD’s timing footnote describes hours on an eight-A100 virtual machine, despite the surrounding “GPU hours” terminology [16]. ESR’s reported phase timings exclude model-loading overhead [17]. Such details are not reasons to dismiss the results; they identify the quantities that can be compared. A missing field remains “not reported”, never zero. We reserve measured, modeled, and proposed evidence categories so analytical token-cost estimates do not become empirical wall-clock claims.

5. Evaluation Protocol and Research Agenda

5.1. Three Complementary Comparisons

Fixed workload.

First replay the same prompts and trajectories to isolate scoring, update, memory, and communication costs. Hold hardware, precision, batch size, padding, optimizer, and token counts fixed. This diagnoses implementation efficiency but intentionally excludes differences in the trajectories each learning algorithm would generate. Time the complete update as well as its stages.

Fixed budget and target quality.

Next compare actual learning loops under declared accelerator-hour, teacher-token, or wall-clock constraints, evaluating each budget regime separately. Include frozen-student replay, full current-student OPD, teacher-response SFT, and a simple short-prefix baseline where interfaces permit. Plot quality against consumed resources, not only steps. Record unreached targets, use a fixed development set for checkpoint selection, and report independent test scores. Repeated seeds and timing variability distinguish consistent effects from one favorable run. Appendix F supplies a concrete comparison template and minimum reporting fields.

Matched deployment.

Finally compare deployed quality under identical decoding, sample, context, tool, and retry budgets. Report pass@1 alongside suitable multi-sample or task-completion metrics and response lengths. The resulting serving costs determine whether training investment is amortized. This third comparison is necessary even when model architecture is unchanged, because post-training can change generation length and diversity.

5.2. Open Problems That Survive the Accounting

Joint allocation rather than isolated knobs.

How should a fixed budget be split between acquiring more questions, generating deeper states, querying more teacher information, and taking additional updates? A useful study measures marginal held-out improvement per additional resource and tests interactions, rather than multiplying standalone speedups. Factorial ablations can distinguish complementary components from repeated removal of the same bottleneck.

State value and refresh policy.

The central question is not whether shorter or fresher is always better. It is when a newly visited state adds more useful learning than another update on an existing one. Compare refresh and restart rules under matched generation budgets, stratified by model compatibility and task horizon. Evaluate both local imitation and terminal success; low teacher–student disagreement alone need not certify a good task trajectory.

The value of supervision.

Data acquisition and teacher selection need an objective tied to the student after learning. Difficulty, teacher confidence, and disagreement are candidate proxies, not universal utility measures. Compare proxies against realized validation gains while charging their estimation cost. A trained questioner or adapted teacher should be tested across multiple students to determine when preparation amortizes.

Coverage under aggressive savings.

Efficiency constraints can alter which skills, languages, and modalities remain represented. Video-OPD, Video-MOPD, and student-side multimodal studies show that OPD extends beyond text reasoning [62,63,64]. Their accounting must include media encoding and modality-dependent context. Agentic tasks add environment latency and irreversible trajectory decisions. Budgeted evaluations should stratify long-horizon success, rare capabilities, factual transfer, and relevant safety behavior rather than rely exclusively on one aggregate reasoning score.

Hardware-aware optimization without objective drift.

The opportunity is to co-design kernels and supervision while tracking how approximation changes learning. Report exact-gradient checks at fixed prefixes, end-to-end resource use, and resulting quality for the same implementation. Sparse vocabulary selection, mixed precision, and teacher caching are especially useful test cases. A method that moves the cost–quality frontier on one device or tokenizer should be evaluated on another before claiming broad portability.

6. Conclusion

Efficient OPD is a problem of allocating resources to useful student learning. Its methods change task acquisition, visited states, supervision, updates, execution, and deployment costs in coupled ways. A six-track organization becomes useful when it is paired with explicit measurement boundaries and quality conditions. The versioned ledger and evaluation protocol make those conditions inspectable, while the research agenda focuses on improvements that remain valuable after preparation, discarded work, and deployment are counted.

Appendix A.1. Search and Inclusion Procedure

The review was assembled through primary-source searches, backward citation tracing from established OPD papers and surveys, and targeted follow-up queries for efficiency mechanisms. Queries combined “on-policy distillation” with “survey”, “efficient”, “prefix”, “early stopping”, “asynchronous”, “offline student rollouts”, “data-free”, “synthetic”, “multi-teacher”, “sparse”, and “speculative”. Once a method name was identified, its paper or official implementation was inspected rather than relying on an aggregator’s summary. Searches also covered the spelling variants “on policy” and common method acronyms.

We include (i) direct teacher-guided learning on student-induced states, including clearly labeled replay or mixed-policy variants; (ii) closely related KD/IL foundations needed to define the objective; and (iii) systems or deployment work needed to interpret resource claims. We exclude papers from the quantitative audit when a numerical statement cannot be connected to a primary source and identifiable workload. Such a source may still support a qualitative discussion. We do not infer that an unreported cost is absent, and we do not count every citation in a survey as an independently reviewed paper.

The supplied literature_catalog.csv identifies sources and their author-assigned roles and tracks. CODEBOOK.md defines the annotations, record types, and distinct states for mismatched configurations, unresolved links, and comparisons outside a paired training-cost claim. The claim-level evidence_ledger.csv records multiple comparisons from the same paper separately; model pairs or sequence caps must not be merged into one method-level value. The files contain source URLs and section/table locators. Full-text-derived measurements, abstract-level contextual references, and mutable software documents have different evidential roles. The catalog is a reproducible inclusion set, not an estimate of the total size of the field.

Versions and publication status.

We use the latest inspected version available by the cutoff for rapidly changing preprints, but prefer a verified proceedings record when discussing the historical published contribution. For example, the ICLR 2024 MiniLLM publication retains its proceedings title even though its later arXiv version changes the title. A mutable documentation page is identified by its access date, October 25, 2026, rather than treated as a permanent specification. No source dated after the cutoff is intended to support a claim.

Appendix A.2. Exact Relation to Earlier Surveys

For the April survey, the relevant v1 locators are Section 4.3.1, 7.2, and 7.3; for v5 they are Section 8.1–8.6 and 10.4 [7]. Thus a cost-centered title is insufficient evidence of a new contribution. The formula-driven survey is especially relevant to the estimator distinctions in Appendix B; its proposed GAE-OPD and counterfactual-routing ideas are not treated here as evaluated methods [8]. The author-uploaded Echoes of the Oracle manuscript is a preprint, and its Section 5.5 discusses rollout cost [9].

Adjacent reviews cover general KD, LLM distillation, self-distillation, and the relation between on-policy and off-policy post-training [67,68,69,70]. We retain their distinctions but use resource accounting as the organizing index. The extra deliverable is a claim-level ledger and an explicit test of what each comparison identifies, not a claim that these concepts were previously absent.

Appendix A.3. Applying the Three Audit Tests

Section 4.1 defines resource, quality, and mechanism identification. The ledger records these tests through the metric and phase fields, configuration correspondence, controls, and caveats. A valid complete-recipe improvement can identify resource and quality effects without isolating the mechanism. Conversely, a fast microbenchmark may identify a resource effect without establishing a learning benefit. Endpoint accuracy and endpoint cost do not reconstruct the learning curve in Eq. 5; the ledger does not imply a retrospective time-to-quality result that was never measured.

Appendix B.1. Historical Distinctions

Teacher-response training in sequence-level KD minimizes a token negative log-likelihood on teacher-generated responses [71]. Scheduled sampling modifies the source of previous tokens, but that alone does not supply teacher feedback on every student-induced state [72]. ImitKD explicitly connects autoregressive KD to imitation learning [3]. General f-divergence treatments clarify why different distributional objectives need not transfer the same behavior [73]. TAID adjusts the target along an interpolated path, an adjacent efficiency idea whose changing target should not be confused with fresh rollout collection [74].

Figure A1. Three independent design choices in OPD. (a) State source determines visited prefixes, while KL direction determines action weighting at each fixed prefix. Method placements illustrate formulations rather than uniquely classify method families; “possible” denotes a mathematically definable design. (b) Feedback may expose a full distribution, selected logits, or a sampled action. Probability glyphs are schematic.

Figure A1. Three independent design choices in OPD. (a) State source determines visited prefixes, while KL direction determines action weighting at each fixed prefix. Method placements illustrate formulations rather than uniquely classify method families; “possible” denotes a mathematically definable design. (b) Feedback may expose a full distribution, selected logits, or a sampled action. Probability glyphs are schematic.

Appendix B.2. The Local Reverse-KL Gradient

Fix a prefix h and teacher probabilities $q(a\mid h)$. For student logits $z_j$, differentiation gives

$${\displaystyle \frac{\partial D_{\mathrm{R}}}{\partial z_j}}=p_j\left[\log {\displaystyle \frac{p_j}{q_j}}-D_{\mathrm{R}}(p,q;h)\right].$$

(A1)

Equivalently, the negative gradient can be estimated by sampling $a\sim p_{\theta }(·\mid h)$ and multiplying its score function by $\log q(a\mid h)-\log p_{\theta }(a\mid h)$, with the reward detached. The constant term cancels because ${\mathbb{E}}_{a\sim p}\nabla \log p\left(a\right)=0$. This is a statement about a fixed prefix. It does not differentiate how changing the policy changes later prefixes.

In forward KL, the corresponding local gradient is $-{\mathbb{E}}_{a\sim q}{\nabla }_{\theta }\log p_{\theta }(a\mid h)$. The teacher can therefore provide an action distribution at a student-generated prefix. This separates the source of the action-weighting measure from the source of the prefix. Full-vocabulary summation avoids Monte Carlo noise over actions at that prefix, but retains the cost of producing and processing both distributions.

Appendix B.3. The Full Trajectory Reverse-KL Gradient

For a fixed prompt, let $P_{\theta }\left(y\right)={\prod }_t{p}_{\theta }(y_t\mid h_t)$ and $Q\left(y\right)={\prod }_{t}q(y_t\mid h_t)$, with a frozen teacher and a common finite horizon including EOS/absorbing termination. Define ${\ell }_t=\log [p_{\theta }(y_t\mid h_t)/q(y_t\mid h_t)]$ and $g_t={\nabla }_{\theta }\log p_{\theta }(y_t\mid h_t)$. Under the usual support and differentiation assumptions,

$$\begin{array}{cc}\hfill J\left(\theta \right)& ={\mathbb{E}}_{y\sim P_{\theta }}\sum _t{\ell }_t,\hfill \end{array}$$

(A2)

$$\begin{array}{cc}\hfill \nabla J\left(\theta \right)& ={\mathbb{E}}_{y\sim P_{\theta }}\sum _t{g}_t{\displaystyle \sum _{u=t}^T}{\ell }_u.\hfill \end{array}$$

(A3)

To see this, differentiating the sampling distribution gives $\mathbb{E}\left[\left({\sum }_u{\ell }_u\right)\left({\sum }_t{g}_t\right)\right]$; differentiating the explicit log-probability contributes $\mathbb{E}\left[{\sum }_t{g}_t\right]=0$. Past costs are measurable before the current action and disappear by the conditional score-function identity. Future costs remain because the current action changes their prefixes.

Keeping only $g_t{\ell }_t$ generally removes those future-state terms. It can be an intentional local surrogate, but should not be described as the exact full-trajectory gradient without further assumptions. Likewise, return discounting, token masks, trajectory-length normalization, advantage baselines, and PPO-style clipping alter the estimator or objective in different ways [75]. A baseline independent of the sampled action can reduce variance without bias under the relevant sampling measure; arbitrary clipping need not preserve that property. This distinction follows the methodological boundary emphasized by GKD, MiniLLM, and the formula-driven survey.

Appendix B.4. Stale Actions Versus Stale States

Suppose a cached trajectory was generated by b, while the desired current policy is p. At a fixed prefix, an importance ratio $p(a\mid h)/b(a\mid h)$ can correct an action expectation when support and numerical conditions hold. It does not replace the outer prefix distribution $d_b$ by $d_p$. Full prefix-occupancy correction would require additional trajectory information and ratios, whose variance can be large. If the behavior sampler truncates support through top-p or top-k sampling, some target actions may have zero behavior probability and exact correction is unavailable from those samples.

For normalized prefix distributions with a shared prompt and horizon convention, and a bounded loss $0\le \ell \left(h\right)\le L$, a diagnostic is

$$\left|{\mathbb{E}}_{d_p}\ell -{\mathbb{E}}_{d_b}\ell \right|\le L\mathrm{TV}(d_p,d_b),$$

(A4)

where $\mathrm{TV}$ is half the ${\ell }_1$ distance. Unbounded KL does not satisfy this assumption automatically. The inequality motivates measuring drift under an explicit bounded proxy; it is not a convergence or task-accuracy guarantee for stale OPD.

Appendix B.5. What top-k compression discards

Let S be the retained vocabulary and $\overline{S}$ its complement. Write $q_{\overline{S}}={\sum }_{a\in \overline{S}}{q}_a$, and similarly for p. Forward KL decomposes exactly as

$$\begin{array}{cc}\hfill D_{\mathrm{F}}(p,q)=& \sum _{a\in S}{q}_a\log {\displaystyle \frac{q_a}{p_a}}+q_{\overline{S}}\log {\displaystyle \frac{q_{\overline{S}}}{p_{\overline{S}}}}\hfill \\ \hfill & +q_{\overline{S}}{D}_{\mathrm{KL}}(q(·\mid \overline{S})\parallel p(·\mid \overline{S})).\hfill \end{array}$$

(A5)

A tail bucket preserves total tail mass but omits the last, conditional-divergence term. Renormalizing only the retained entries makes yet another objective. Thus “top-k KL” requires a normalization and tail convention. Current verl documentation makes related interface distinctions [76]. Vocabulary compression can reduce the feedback payload without eliminating the teacher forward pass.

Appendix C.1. Resource Units and Critical Paths

Accelerator-hours should specify whether they count reserved device time or active kernel time. When stages share devices, their allocation intervals must not be counted twice. Time on an eight-device node and summed device-hours differ by a factor of eight only when that allocation is maintained for the complete measured interval. FLOP estimates should specify forward/backward conventions, attention, output heads, active MoE parameters, and discarded work; neither FLOPs nor tokens capture hardware utilization alone.

For three serial stages with durations $G,S,U$ per batch, K batches take $K(G+S+U)$, excluding overhead. With distinct stage resources, stationary durations, sufficient buffering, and no additional dependencies, an ideal pipeline takes

$$T_{\mathrm{pipe}}=G+S+U+(K-1)\max (G,S,U).$$

(A6)

This is a conditional scheduling identity, not an OPD speed prediction. Actor refresh, finite memory, shared devices, variable sequence lengths, environment latency, and transfers can violate its assumptions. Device-time still counts all stage allocations. AReaL and slime provide relevant asynchronous post-training infrastructure, but their original results cannot be substituted for an OPD-specific comparison [77,78].

Figure A2. Elapsed time and stage work need not change together. For three batches with equal stage duration $\tau$, serial execution takes $9\tau$, while an ideal pipeline takes $5\tau$; summed stage work remains $9\tau$. The illustration assumes dedicated stage resources, sufficient buffers, and no transfer or refresh stalls. Reserved device-time can differ, and overlapping batches may use lagged actors. These quantities are schematic, not measured speedups.

Figure A2. Elapsed time and stage work need not change together. For three batches with equal stage duration $\tau$, serial execution takes $9\tau$, while an ideal pipeline takes $5\tau$; summed stage work remains $9\tau$. The illustration assumes dedicated stage resources, sufficient buffers, and no transfer or refresh stalls. Reserved device-time can differ, and overlapping batches may use lagged actors. These quantities are schematic, not measured speedups.

Appendix C.2. Avoided Work Versus Selected Supervision

Suppose a selector is applied after a full trajectory has been generated and scored. If generation and teacher work remain unchanged, its additive saving is

$$\mathsf{\Delta }C=(U_{\mathrm{full}}-U_{\mathrm{selected}})-C_{\mathrm{selector}}.$$

(A7)

A positive resource saving requires avoided update work to exceed selection overhead. A mask that merely zeros losses but executes the same dense backward graph may not satisfy this condition. Conversely, a method can be worthwhile when it reduces peak memory enough to make a configuration feasible, even if $\mathsf{\Delta }C<0$. Resource feasibility and cost minimization are separate objectives.

For a fixed serial workload whose fraction f is accelerated by s, the ideal total speedup is ${[1-f+f/s]}^{-1}$. This elementary bound explains why rollout-token reductions need not translate proportionally to runtime. It does not apply unchanged when the method changes output length, optimizer steps, or resource concurrency. Flash-OPD’s component timings in the ledger are an example of such a changed workload and shifted cost, rather than an isolated kernel acceleration.

Appendix C.3. Amortization at a fixed quality requirement

Let recipes A and B have preparation costs $P_A,P_B$ and compatible per-use costs $c_A,c_B$, measured in the same additive units. If $P_A>P_B$ but $c_A<c_B$, then A becomes less expensive after

$$R>{\displaystyle \frac{P_A-P_B}{c_B-c_A}}.$$

(A8)

This assumes both recipes meet the declared quality requirement on all counted uses. A questioner whose prompts cease to be useful for a different student, or a prefix pool invalidated by a changed environment, cannot be credited with arbitrary future reuse. Report an amortization curve over plausible R rather than choosing the most favorable deployment volume.

For student deployment against an alternative system, substitute training costs for P and expected complete serving costs for c in Eq. A8. Include extra samples or retries. When the distilled model has a higher up-front cost, no positive deployment break-even exists unless its quality-matched per-request cost is lower. Startup and maintenance can be added explicitly; request latency remains a separate, non-additive service constraint.

Table A1. Analytical composition audit. These are accounting conditions for hypothetical combinations, not experimentally established compound methods.

Combination	Potentially shared saving	New or remaining cost	Necessary check before claiming composition
Truncation + token masking	Both can remove supervision from late positions	Mask selection; the retained prefix still needs generation and scoring	Measure the mask’s incremental saving after truncation; do not multiply standalone ratios.
Replay + score caching	Historical tokens and fixed-teacher scores can be reused together	Storage, reads, changed teacher versions, missing new states	Pin teacher, tokenizer, precision, and prompt context; refreshing scores does not refresh occupancy.
Sparse logits + asynchronous execution	Lower retained memory may allow a better resource partition	Sparse planning, queues, teacher forwards, lag	Re-profile the complete schedule; shrinking a noncritical stage may not improve elapsed time.
Self-distillation + multiple teachers	Some model weights may be shared across roles	Privileged contexts, reference solutions, multiple forward passes	Count calls and context tokens; shared weights do not imply one inference pass.
Short rollout + speculative verification	Fewer decoded positions	Drafts, rejected tokens, repeated prefix prefill	Count processed tokens, not only retained tokens; identify the preserved behavior distribution.

Table A1. Analytical composition audit. These are accounting conditions for hypothetical combinations, not experimentally established compound methods.

Combination	Potentially shared saving	New or remaining cost	Necessary check before claiming composition
Truncation + token masking	Both can remove supervision from late positions	Mask selection; the retained prefix still needs generation and scoring	Measure the mask’s incremental saving after truncation; do not multiply standalone ratios.
Replay + score caching	Historical tokens and fixed-teacher scores can be reused together	Storage, reads, changed teacher versions, missing new states	Pin teacher, tokenizer, precision, and prompt context; refreshing scores does not refresh occupancy.
Sparse logits + asynchronous execution	Lower retained memory may allow a better resource partition	Sparse planning, queues, teacher forwards, lag	Re-profile the complete schedule; shrinking a noncritical stage may not improve elapsed time.
Self-distillation + multiple teachers	Some model weights may be shared across roles	Privileged contexts, reference solutions, multiple forward passes	Count calls and context tokens; shared weights do not imply one inference pass.
Short rollout + speculative verification	Fewer decoded positions	Drafts, rejected tokens, repeated prefix prefill	Count processed tokens, not only retained tokens; identify the preserved behavior distribution.

Appendix C.4. Three Conditional Decision Examples

Dense supervision is memory-limited.

If teacher feedback is useful but full-vocabulary backward storage exceeds the device limit, compare sparse differentiation, chunked losses, sharding, and adapter training on a fixed batch. Check the whole-step resident peak and resulting throughput. A slower feasible method can be preferable to an infeasible baseline; report that as feasibility rather than a speedup.

Rollout dominates elapsed time.

First profile generation and environment waits. Compare a simple prefix cap, cached initial-student replay, and asynchronous collection, then inspect which task depths or model pairs lose quality. Use matched rollout budgets to evaluate an adaptive controller. If scoring becomes critical after shortening, its former small share cannot justify excluding it from the revised profile.

Teacher/API expenditure is limited.

Determine whether the service exposes selected probabilities, top-k outputs, or text only. Compare reducing queried positions, sharing a fixed-teacher cache, and paying for richer feedback on selected tasks. Include all proposal, probe, and verification calls. A method requiring inaccessible logits is not a feasible candidate even if its locally hosted benchmark is strong.

Appendix D.1. Access and Modality Boundaries

An OPD objective on student tokens does not inherently require ground-truth answers. However, prompt utility estimation, teacher gating, teacher adaptation, and outcome-based selection can introduce reference answers, executable tests, environment rewards, or judges. These dependencies belong in the task specification. “No external task data”, “no answer labels”, “no teacher logits”, and “no additional training” are four different claims.

For multi-agent debate, teacher construction can itself contain several interacting models; MAD-OPD is a relevant example [82]. The resulting teacher answer should not be accounted as one free supervision event. For video and speech, media encoding and long input contexts add work absent from text-only completion counts.

Speech supplies concrete preparation examples. X-OPD transfers text-teacher capabilities to a speech student using paired spoken and textual inputs. Its preparation includes rewriting, speech synthesis, and ASR-based filtering, with a frozen audio tower and adapter [83]. Ark-ASR applies OPD after supervised ASR training and reuses an audio corpus; its comparison with a much larger encoder-pretraining corpus does not isolate data savings caused by OPD [84]. Audio duration, media preprocessing, and encoder work belong alongside generated-token accounting.

Data-source effects remain conditional.

With 48 prompts and a Qwen3-4B student and a 30B Instruct teacher, Beyond Prompt Count reports 44.84 for token-matched GSM8K at 16.14 million supervised tokens versus 51.02 for DeepMath at 16.63 million. The macro-average covers mathematics, GPQA, HumanEval+, and LiveCodeBench [15]. These are within-study source comparisons, not a contradiction inferred by comparing raw scores with another paper. Further work on data selection and generalization reinforces the need to specify what transfer is being measured [85,86].

Appendix F.1. Specify the Comparison Before Running

Choose a teacher–student pair, a capability vector, and the resource that is genuinely limiting. Include the teacher’s architecture, tokenizer, accessible outputs, precision, and preparation. For MoE models, report both total and active parameters, plus the expert-parallel execution scheme. For self-teachers, specify the privileged context and where it comes from. Neither matching parameter count nor sharing base weights guarantees identical execution cost.

For broad claims, include closely related and mismatched teacher–student pairs. Cover short-answer tasks, extended reasoning or code, and long-horizon interaction when agentic applicability is claimed. These are proposed evaluation strata, not a newly assembled benchmark.

Baselines.

Use a common initialization and include teacher-response SFT, current-student full-rollout OPD, frozen initial-student replay, and a simple fixed-prefix OPD baseline where feasible. If the proposed method changes the loss, retain a control with its sampling policy but the original loss. If it changes sampling, retain a control with its loss and the original sampling. Conventional offline distillation such as the distilled models in DeepSeek-R1 is context for the SFT baseline, not evidence that every such baseline has the same preparation cost [87].

Budget controls.

Two experimental questions require different controls. A fixed-token or fixed-trajectory replay profile isolates systems effects; a fixed-resource learning run evaluates the actual recipe. Use both instead of presenting one as the other. When varying rollout length, report prompt and response tokens separately. When varying refresh rate, report the number of distinct states, reuse multiplicity, behavior-policy versions, and age in updates or seconds.

Table A8. Minimum reporting template. This is a proposed protocol grounded in the audited failure modes, not a claim that a new experimental benchmark has been run.

Reporting group	Minimum fields	Why the fields are necessary
Models and access	Exact checkpoints; tokenizer; total/active parameters; precision; trainable modules; teacher API	Determines representational compatibility and feasible supervision.
Preparation	SFT, teacher training, questioner training, pool creation, filters, probes	Prevents offline work from disappearing from the denominator.
State distribution	Source policy; refresh cadence; lag distribution; prefix provenance; reset/continuation behavior	Distinguishes current, recent, frozen, and mixed-policy training.
Objective	KL direction; temperature; top-k/tail convention; score detachment; returns; clipping; normalization	Determines what information and gradient are being approximated.
Token accounting	Prompt, generated, scored, selected, discarded, replayed, and learner-used tokens	Generated output length is not the total processed workload.
Systems	Device type/count; placement; parallelism; inference engine and version; warm-up; transfer/swap cost	Makes timing reproducible and identifies changed allocations.
Resources	Complete-run time; allocated GPU-hours; peak resident/extra memory; teacher bill; FLOP assumptions	Separates throughput, work, feasibility, and monetary cost.
Quality and selection	Development/test split; checkpoint rule; seeds; metrics; inference and tool budgets	Prevents favorable checkpoint or decoding choices from changing the target.
Uncertainty and failures	Variation across runs; OOM; unreached targets; stale-data divergence; missing values	Avoids silently dropping infeasible or failed configurations.

Table A8. Minimum reporting template. This is a proposed protocol grounded in the audited failure modes, not a claim that a new experimental benchmark has been run.

Reporting group	Minimum fields	Why the fields are necessary
Models and access	Exact checkpoints; tokenizer; total/active parameters; precision; trainable modules; teacher API	Determines representational compatibility and feasible supervision.
Preparation	SFT, teacher training, questioner training, pool creation, filters, probes	Prevents offline work from disappearing from the denominator.
State distribution	Source policy; refresh cadence; lag distribution; prefix provenance; reset/continuation behavior	Distinguishes current, recent, frozen, and mixed-policy training.
Objective	KL direction; temperature; top-k/tail convention; score detachment; returns; clipping; normalization	Determines what information and gradient are being approximated.
Token accounting	Prompt, generated, scored, selected, discarded, replayed, and learner-used tokens	Generated output length is not the total processed workload.
Systems	Device type/count; placement; parallelism; inference engine and version; warm-up; transfer/swap cost	Makes timing reproducible and identifies changed allocations.
Resources	Complete-run time; allocated GPU-hours; peak resident/extra memory; teacher bill; FLOP assumptions	Separates throughput, work, feasibility, and monetary cost.
Quality and selection	Development/test split; checkpoint rule; seeds; metrics; inference and tool budgets	Prevents favorable checkpoint or decoding choices from changing the target.
Uncertainty and failures	Variation across runs; OOM; unreached targets; stale-data divergence; missing values	Avoids silently dropping infeasible or failed configurations.

Appendix F.2. Measure the Full Execution Boundary

Warm up compilation and the serving engine, then explicitly report whether the warm-up is included in run totals. Synchronize devices for stage-level kernel timing without turning every production step into an artificially synchronized pipeline. Record end-to-end wall-clock duration independently. Report the maximum simultaneously allocated memory, persistent model state, and extra stage allocations as separate quantities. Do not add forward and backward peaks into a purported concurrent peak.

In API experiments, count input, cached-input, output, and retry tokens according to the actual interface, retaining prices and access dates if reporting money. Include unsuccessful calls and rejected synthetic questions. For locally hosted teachers, include every student, teacher, reference, probe, and verification forward. Weight transfers and model swapping are part of the complete recipe even when excluded from a kernel microbenchmark.

Teacher-output interfaces.

The inspected vLLM teacher-scoring documentation supports per-position candidate token probabilities through a prefill and GPU gather, but that path skips prefix-cache reads because those cache hits do not provide the requested logits [88]. This example illustrates why a scoring integration must be tested against its exact backend version. It does not imply that every KV-caching technique is unusable for every scoring task.

Appendix F.3. Estimate Quality Without Hidden Reuse

Fix the validation subset, evaluation frequency, and checkpoint-selection criterion before training. Evaluate selected checkpoints once on a distinct test set. If an existing paper selects checkpoints on a named benchmark, scores on that benchmark should not be treated as held-out generalization estimates. For exact-match metrics, record parsing and truncation rules. For code or agents, record tests, timeouts, environment versions, and whether execution retries are counted.

Plot quality against consumed resources across repeated seeds. Fixed evaluation intervals locate target crossing only between checkpoints; report that resolution. Retain unreached targets. Final cost and accuracy cannot determine a cost-to-quality ratio without the intervening learning curve.

Appendix F.4. A practical Factorial Pilot

A compact study can begin with three rollout regimes (current full, current prefix, frozen replay), two divergences (forward and reverse), and two feedback resolutions (full vocabulary and a fully specified sparse approximation). This creates twelve recipe cells before seeds, assuming all interfaces are available. Profile a fixed batch first to remove clearly infeasible configurations; then allocate the same pilot accelerator-hour budget to each remaining cell. Record preparation separately, and do not discard a failed learning run because its final score is low.

Use the pilot to choose one main comparison axis rather than conducting an unrestricted search for the strongest score. For an adaptive method, compare against a small grid of fixed schedules using the same development budget. Report tuning cost separately from the selected run. The final study should include a test of the proposed mechanism: for example, equalizing state depth when evaluating freshness, or equalizing teacher calls when evaluating routing quality.