TDLM: Temporal Self-Conditioning Diffusion Language Model

1. Introduction

Masked diffusion language models offer a route to parallel text generation: a bidirectional denoiser predicts multiple missing tokens, and a decoding policy decides which predictions to commit. LLaDA, iLLaDA, and Dream demonstrate the capabilities of diffusion language modeling at billion-parameter scale [13,14,19]. Nemotron-Labs-Diffusion and DiffusionGemma further develop efficient parallel generation while retaining support for autoregressive decoding [4,6]. The practical efficiency of diffusion generation depends on how many predictions can be committed reliably at each step. Committing too few requires repeated expensive forward passes; committing uncertain predictions can damage the entire answer. Confidence-based decoding exposes this tension directly through a threshold controlling parallelism [17].

Consider a token position that remains masked across several denoising steps. Its discrete input has not changed, yet the model has repeatedly computed a rich representation of what could belong there. As other tokens become visible, these representations evolve. A conventional denoiser carries committed tokens into the next step, but provides no explicit route for an unresolved position to consult its earlier representations. The computation spent on these intermediate estimates has no persistent representation available for direct reuse.

This suggests a question about the use of inference computation: can a denoiser reuse its earlier internal estimates to reach reliable predictions sooner? Historical representations do not provide new external evidence beyond the request and evolving response. Their potential value lies in preserving computations that the model has already performed under earlier contexts. If these computations help resolve uncertainty, more tokens may be committed per evaluation without a corresponding loss in answer quality. If they merely increase confidence, apparent acceleration may instead conceal additional errors. Both possibilities must be tested.

We introduce TDLM, a temporal self-conditioning extension to a pretrained diffusion language model. At selected Transformer layers, each unresolved position attends to its own representations from earlier denoising steps. This preserves the alignment between a pending prediction and the history used to refine it. Narrow temporal projections keep the adaptation small, while a learned zero-value sink allows the model to suppress unhelpful history. The backbone remains frozen. During blockwise generation, unresolved positions continue to accumulate history across block boundaries.

The training procedure is simple: the frozen backbone generates its own trajectories, and the added modules learn to predict the tokens revealed between consecutively sampled decoding steps using cross-entropy. This reuses the backbone’s existing capabilities and the memory accumulated during denoising, while updating only a small set of parameters. It requires no new manually annotated reference answers. Strictly causal temporal attention permits the corresponding partially masked inputs to be evaluated in parallel during training.

Our experiments on LLaDA and Dream study generation efficiency, answer quality, and the role of the available history. Figure 1 illustrates the resulting quality–latency trade-off. We examine whether temporal reuse makes additional computation useful before a token is committed, and how restricting that history changes the model’s predictions.

Our contributions are:

• A lightweight, position-aligned temporal attention mechanism that makes earlier denoising representations available to unresolved predictions in a frozen language model.
• A simple cross-entropy self-distillation objective for earlier prediction, with causal parallel evaluation of sampled trajectory inputs.
• An empirical study of the quality–latency trade-off and the effects of restricting the history available to unresolved predictions.

2. Method

TDLM lets each masked token position reuse hidden representations from earlier denoising steps. Spatial attention exchanges information across positions in the current input; temporal attention retrieves earlier representations of the same position. The backbone remains frozen, and only the added temporal modules are trained.

2.1. Conditioning on Denoising History

Let c be a prompt and N the response length. Each decoding step evaluates the denoiser and reveals one or more tokens: their masks are replaced with predicted tokens, which are then kept fixed. We refer to this operation as commitment.

We call a partially revealed response sequence a state and write it as $S_i$. Its associated global decoding step $s_i$ is the number of updates already completed. Thus, i indexes the selected sequences, $s_i$ records their times along the decoding trajectory, and p denotes a token position. During training we select a subset of teacher decoding steps; during inference we evaluate every successive input. The initial sequence is $S_0=mas{k}^N$ at $s_0=0$.

For a masked position p, TDLM extends the information available to predict its token $y_p$:

$$p_{\theta }(y_p\mid c,S_i)⟶p_{\theta ,\varphi }(y_p\mid c,S_i,{\mathcal{H}}_{<i}),$$

(1)

where $\theta$ denotes the frozen backbone parameters, $\varphi$ the temporal parameters, and ${\mathcal{H}}_{<i}$ the hidden representations retained from earlier evaluated inputs $S_j$, $j<i$. The input at position p can remain a mask while its internal representation changes with the surrounding context. Temporal memory makes those earlier representations available to the next prediction.

2.2. Temporal Attention at Masked Token Positions

At a selected layer ℓ, let $h_{\ell ,p}\left(S_i\right)\in {\mathbb{R}}^d$ be the hidden representation at position p, after spatial attention and before the feed-forward sublayer. Define $m_{i,p}=\mathbf{1}[{\left(S_i\right)}_p=mask]$ and let $r_i={\sum }_p(1-m_{i,p})$ count the revealed response tokens in $S_i$. A decoding step can reveal several tokens, so $r_i$ need not equal the step number $s_i$.

For a currently masked position, every earlier input also masks that position because revealed tokens are never remasked. Its layer-specific history ${\mathcal{H}}_{\ell ,p}^{<i}$ contains earlier representations $h_{\ell ,p}\left(S_j\right)$ and their progress coordinates $r_j$, with $j<i$. The current representation queries this history through a residual update:

$${\tilde{h}}_{\ell ,p}\left(S_i\right)=h_{\ell ,p}\left(S_i\right)+m_{i,p}{TempAttn}_{\ell }\left(h_{\ell ,p}\left(S_i\right),{\mathcal{H}}_{\ell ,p}^{<i};r_i\right).$$

(2)

TempAttn uses standard multi-head attention with narrow projections, retrieving only earlier representations of the same token position at the same layer. Temporal rotary embeddings [16] use revealed-token counts to encode decoding progress. A learned attention sink with zero value gives the module an option to ignore history. Appendix B.1 gives the projection, normalization, and multi-head formulas.

Figure 2. Reusing computation before commitment. Each column is a denoising evaluation, read from bottom to top. The bottom shows the partially revealed input; the top shows token predictions before commitment. Decoding Steps 1–3 use inputs $S_0,S_1,S_2$ at $s_0=0,s_1=1,s_2=2$, respectively. The green position remains masked while its predictions can change. Green arrows retrieve earlier hidden representations at that position and layer. Yellow modules are trainable, and the backbone is frozen.

Figure 2. Reusing computation before commitment. Each column is a denoising evaluation, read from bottom to top. The bottom shows the partially revealed input; the top shows token predictions before commitment. Decoding Steps 1–3 use inputs $S_0,S_1,S_2$ at $s_0=0,s_1=1,s_2=2$, respectively. The green position remains masked while its predictions can change. Green arrows retrieve earlier hidden representations at that position and layer. Yellow modules are trainable, and the backbone is frozen.

The original feed-forward sublayer processes the corrected representation $\tilde{h}$. Keys and values are formed from h before the same layer’s temporal correction, a property used by parallel trajectory training below. Only masked token positions receive a correction. Empty history produces zero correction, and a zero-initialized output projection preserves the pretrained denoiser at initialization. The precise statements and proofs are in Appendix B.

2.3. Simple Trajectory Self-Distillation

For each prompt, the frozen backbone generates a teacher response in T decoding steps. We record the final tokens $y_p$ and the step $a_p$ at which each token is revealed. These records let us reconstruct the input at any selected step $s_i$: tokens revealed by that step are visible, and later tokens remain masked,

$${\left(S_i\right)}_p=\left\{\begin{array}{cc}{y}_p,\hfill & a_p\le s_i,\hfill \\ mask,\hfill & a_p>s_i.\hfill \end{array}\right.$$

(3)

Hidden representations are computed during training from these reconstructed inputs.

We always include $s_0=0$, sample $K-1$ distinct steps uniformly from $\{1,\dots ,T-1\}$, and sort them. We also set $s_K=T$ to mark the end of the trajectory:

$$0=s_0<s_1<\dots <s_{K-1}<s_K=T.$$

The gap $n_i=s_{i+1}-s_i$ is the number of decoding steps between two selected sequences. Every decoding step reveals at least one token, so the number of masks strictly decreases from $S_i$ to $S_{i+1}$. We evaluate $S_0,\dots ,S_{K-1}$; the fully revealed $S_K$ only marks the end of the last supervision interval and requires no forward pass.

For an input $S_i$, the training targets are positions that are still masked in $S_i$ but revealed in the next selected sequence $S_{i+1}$. For example, if $s_i=2$ and $s_{i+1}=4$, we train the model to predict the tokens the teacher reveals at steps 3 and 4. Positions revealed later remain masked and contribute temporal history, but receive no loss on this input. Formally, the target positions are

$${\mathcal{T}}_i=\{p:{\left(S_i\right)}_p=mask,{\left(S_{i+1}\right)}_p\ne mask\},1\le i<K.$$

(4)

The initial input $S_0$ seeds temporal memory without supervision because its temporal correction is zero.

Let ${\pi }_{i,p}$ be TDLM’s predicted token distribution at position p, conditioned on c, $S_i$, and ${\mathcal{H}}_{<i}$. We optimize only $\varphi$, using cross-entropy averaged over all supervised token positions across the sampled inputs:

$${\mathcal{L}}_{\mathrm{CE}}\left(\varphi \right)=\underset{\begin{array}{c}1\le i<K\\ p\in {\mathcal{T}}_i\end{array}}{mean}CE({\pi }_{i,p},y_p).$$

(5)

Each supervised token has equal weight. This objective teaches the model to predict the next interval’s tokens using the current input and its accumulated history. Figure 3 illustrates the inputs and targets; Appendix A gives training details.

Parallel evaluation along a fixed trajectory. Spatial attention processes each reconstructed input independently. At a temporal layer, the causal mask $A_{p,i,j}=\mathbf{1}[j<i]m_{j,p}$ lets the representation for $S_i$ retrieve from earlier inputs $S_j$ where position p was masked. Equation (2) restricts the correction to currently masked positions. Since each layer’s keys and values precede its temporal correction, all sampled inputs can be evaluated together layer by layer. With identical history selection and dropout disabled, this is equivalent in exact arithmetic to processing the same inputs sequentially (Appendix B). Training computes no representations for unsampled steps; inference instead accumulates history from each actual forward pass.

2.4. From Retrieval to Fewer Evaluations

At inference, let ${\mathcal{U}}_b$ be the set of masked token positions in the active block. For the current input $S_i$, let ${\widehat{y}}_p$ be the greedy prediction at position p and ${\gamma }_p={\pi }_{i,p}\left({\widehat{y}}_p\right)$ its confidence. The decoder reveals tokens with confidence above a threshold $\tau$, or the most confident token otherwise:

$$C=\left\{\begin{array}{cc}\{p\in {\mathcal{U}}_b:{\gamma }_p>\tau \},\hfill & \mathrm{if}\mathrm{nonempty},\hfill \\ \left\{\arg \underset{p\in {\mathcal{U}}_b}{\max }{\gamma }_p\right\},\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(6)

TDLM and our Fast-dLLM policy baseline share this commitment rule. Original DLM reveals one token per evaluation. Decoding settings are specified together in Section 3.1.

Each forward pass retrieves earlier temporal keys and values, then appends the current entries. All masked response positions accumulate history, including those outside the active block. Revealing a token invalidates that position’s history; changing blocks does not reset the histories of other positions. The full spatial sequence is recomputed at every step.

With $L_t$ temporal layers and total temporal width $d_t=M{d}_h$, the module parameter count is $L_t(4d{d}_t+1)$. Retrieving from H earlier evaluations costs $O\left(NH{d}_t\right)$ per temporal layer, in addition to the projections. Full-history memory requires $O\left(L_{t}NH{d}_t\right)$ storage, and unrestricted attention work grows quadratically with the number of denoising steps. TDLM is therefore useful when it saves enough backbone evaluations to offset this overhead. We measure both evaluation counts and wall time, together with answer quality, to assess that trade-off.

3. Experiments

We evaluate the quality–latency trade-off of TDLM on two backbones and study the contribution of its temporal history.

3.1. Experimental Setup

Models and training. We augment frozen LLaDA-8B-Instruct and Dream-v0-Instruct-7B backbones. LLaDA uses sixteen dispersed temporal layers with four heads of width 64; Dream uses fourteen middle layers with four heads of width 84. For each backbone, we train only the temporal modules on 3,500 self-generated trajectories for 750 optimizer updates using the simple CE objective in Eq. (5). These results therefore require a modest adaptation dataset and a straightforward self-distillation procedure. Appendix A gives the model and training details. Our MetaState baseline uses reference responses from the Tulu-3 SFT mixture as supervision [18], whereas TDLM learns from responses generated by its own frozen backbone.

Tasks and scoring. We evaluate mathematical reasoning on MATH-500 and GSM8K, and code generation on MBPP sanitized [1,3,7,9]. We use the complete test sets of 500, 1,319, and 257 problems, respectively. Math answers are checked with Math-Verify, and code solutions are executed against the dataset tests. We report accuracy or pass@1 from one greedy generation per problem; unparseable answers count as incorrect.

Decoding and measurement. All methods in the main comparison use a generation budget of 256 tokens and block size 32. Confidence-based decoding uses $\tau =0.9$, following released evaluation settings [11,17]; d3LLM uses its default entropy threshold $H=0.5$ [15]. On Dream, TDLM uses $\tau =0.95$ and d3LLM uses $H=0.4$. Original DLM commits one token per evaluation. All methods use compiled model forwards with batch size one on GH200 GPUs. We report mean denoiser evaluations (NFE) and generation seconds, excluding compilation and warmup. Speedup is measured against the original model of the same backbone on the same task. Further evaluation details are in Appendix C.

3.2. Generation Efficiency and Answer Quality

Table 1 compares both backbones across the three tasks. TDLM achieves up to 7.15× faster generation than original DLM in the main comparison, and up to 1.85× over Fast-dLLM. Its largest score improvement over Fast-dLLM is 2.72 percentage points on LLaDA MBPP. Across the six backbone–task combinations, TDLM ranks first, second, and third in quality in one, four, and one cases, respectively; the corresponding latency counts are three, two, and one, using the table’s tie convention. Quality changes remain task-dependent.

Figure 1 compares generation quality and latency across decoding thresholds to explore the empirical Pareto frontier. TDLM pushes the Pareto frontier, offering higher accuracy at fast speeds. Appendix F extends the LLaDA evaluation to 128- and 384-token generation budgets, with speedups over original DLM reaching 8.54× at the longer budget.

3.3. What Does Temporal History Contribute?

We keep the LLaDA checkpoint fixed and vary the context available to the temporal modules: representations from the current evaluation only, the previous one, three or five evaluations, or the complete history. Table 2 reports these inference-time restrictions. The current-only condition retains the learned Q/K/sink gate.

Extending history from one evaluation to the complete trajectory raises observed accuracy from 77.62% to 79.08%, while reducing NFE from 44.06 to 40.08 and mean time from 0.637 to 0.585 seconds. Current-only inference achieves 76.48% accuracy with 42.64 evaluations. These measurements show how the available history changes predictions and commitment behavior for a fixed checkpoint. Implementation details and architecture comparisons are given in Appendices Appendix E and Appendix D.

4. Related work

Parallel decoding and diffusion acceleration. Fast-dLLM [17] combines confidence-based parallel decoding with KV-cache reuse. dKV-Cache [12] uses delayed and conditioned caching to reuse key and value states during denoising. Training-based methods include CDLM [11], which combines consistency training with blockwise causal attention, and d3LLM [15], which uses pseudo-trajectory distillation and entropy-based multi-block decoding. TDLM studies position-aligned temporal retrieval as a way to make earlier denoising computations useful for later predictions.

Self-conditioning and persistent representations. Analog Bits [2] conditions diffusion predictions on earlier model predictions. Recurrent Interface Networks [10] introduce latent self-conditioning across reverse diffusion steps for image and video generation. In language models, RCD [8] recycles discarded-token representations as residual context for subsequent denoising. MetaState [18] equips a frozen discrete diffusion model with fixed-size recurrent working memory, while ALoDLM [5] uses token-adaptive latent recurrence. TDLM retrieves an explicit history of same-position representations at multiple layers of a frozen backbone, with simple trajectory self-distillation and causal parallel evaluation of sampled trajectory inputs.

5. Discussion and Future Work

Temporal memory provides a way to use earlier denoising computations when predicting unresolved tokens. The generation and history-restriction experiments show how this changes the balance between answer quality and the number of evaluations. Further analysis of token-level correctness and confidence, together with a capacity-matched adapter using only the current hidden representation, would help characterize when history is most useful.

Self-distillation inherits the quality limitations of the teacher-generated trajectories. With a frozen backbone and teacher tokens drawn from its own outputs, supervision offers limited evidence for correcting systematic teacher errors. Teacher capability therefore provides a practical reference for the quality available through this training signal. Future work can explore higher-quality training data and full-model fine-tuning, allowing the backbone and temporal attention to adapt jointly and make better use of the accumulated memory.

Sparse teacher trajectories and the student’s own multi-token commitments can expose the model to different histories. Longer responses, repeated measurements, and combinations with spatial KV caching and early stopping are additional directions for evaluation.

6. Conclusion

TDLM augments a frozen diffusion language model with lightweight attention over the history of unresolved positions. A simple trajectory self-distillation objective trains earlier predictions, while causal temporal attention permits parallel evaluation of sampled trajectory inputs. The available experiments show that fewer backbone evaluations can outweigh the added temporal computation at useful quality levels. Reusing denoising history thus offers a practical direction for improving generation efficiency through the model’s existing capabilities.

Appendix B.1. Temporal Attention Operator

We use the notation of Section 2: $S_i$ is the selected input at decoding step $s_i$, p is a token position, and $h_{\ell ,p}\left(S_i\right)$ is its hidden representation at layer ℓ. Each temporal module has M heads of width $d_h$, with total width $d_t=M{d}_h$. Head a uses bias-free projections $W_{\ell ,a}^Q,W_{\ell ,a}^K,W_{\ell ,a}^V\in {\mathbb{R}}^{d_h\times d}$:

$$\begin{array}{cc}\hfill q_{\ell ,p,a}\left(S_i\right)& =R\left(r_i\right)W_{\ell ,a}^Q{h}_{\ell ,p}\left(S_i\right),k_{\ell ,p,a}\left(S_i\right)=R\left(r_i\right)W_{\ell ,a}^K{h}_{\ell ,p}\left(S_i\right),\hfill \\ \hfill v_{\ell ,p,a}\left(S_i\right)& =W_{\ell ,a}^V{h}_{\ell ,p}\left(S_i\right).\hfill \end{array}$$

(7)

$R\left(r\right)$ applies temporal RoPE within each head. Its rotation angles are $r{\omega }_u$, with ${\omega }_u={10000}^{-2u/d_h}$ for $u=0,\dots ,d_h/2-1$. The relative positional term depends on $r_j-r_i$, while the hidden representations and available histories can still differ across decoding schedules.

Let ${\mathcal{J}}_p\left(i\right)$ contain the indices $j<i$ of earlier evaluated inputs retained as history for masked position p. The attention score and normalizer are

$$e_{\ell ,p,a}^{i,j}={\displaystyle \frac{q_{\ell ,p,a}{\left(S_i\right)}^{\top }{k}_{\ell ,p,a}\left(S_j\right)}{\sqrt{d_h}}},Z_{\ell ,p,a}^{\left(i\right)}=\exp \left(b_{\ell }\right)+\sum _{j\in {\mathcal{J}}_p\left(i\right)}\exp \left(e_{\ell ,p,a}^{i,j}\right).$$

(8)

Here $b_{\ell }$ is a learned sink logit shared by the heads; its value vector is zero. Each head retrieves

$$z_{\ell ,p,a}\left(S_i\right)=\sum _{j\in {\mathcal{J}}_p\left(i\right)}{\displaystyle \frac{\exp \left(e_{\ell ,p,a}^{i,j}\right)}{Z_{\ell ,p,a}^{\left(i\right)}}}{v}_{\ell ,p,a}\left(S_j\right).$$

(9)

The operator in Eq. (2) is

$${TempAttn}_{\ell }\left(h_{\ell ,p}\left(S_i\right),{\mathcal{H}}_{\ell ,p}^{<i};r_i\right)=W_{\ell }^O{concat}_{a=1}^M{z}_{\ell ,p,a}\left(S_i\right),W_{\ell }^O\in {\mathbb{R}}^{d\times d_t}.$$

(10)

All keys and values use the residual representation before this layer’s temporal correction. The resulting correction is $\Delta h_{\ell ,p}\left(S_i\right)={\tilde{h}}_{\ell ,p}\left(S_i\right)-h_{\ell ,p}\left(S_i\right)$. This explicit form gives the same update as the compact operator in the main text.

Appendix B.2. Sink, Cold Start, and Initialization

The sink probability in head a is

$${\pi }_{⌀,a}={\displaystyle \frac{\exp \left(b_{\ell }\right)}{Z_{\ell ,p,a}^{\left(i\right)}}}.$$

Its history weights sum to $1-{\pi }_{⌀,a}$. If $\parallel v_{\ell ,p,a}\left(S_j\right){\parallel }_2\le V$ for every head and available history, then

$$\parallel \Delta h_{\ell ,p}\left(S_i\right){\parallel }_2\le m_{i,p}{\parallel W_{\ell }^O\parallel }_{2}V\sqrt{{\sum }_{a=1}^M{(1-{\pi }_{⌀,a})}^2}.$$

(11)

This follows directly from the triangle inequality and the nonnegative attention weights. The sink therefore gates the magnitude of the retrieved history without a separate gating network. This is a local bound on the module output, not a guarantee of whole-model stability or accuracy.

Proposition 1

(Cold start and initialization). For history-only temporal attention, the first forward pass of each request has zero temporal correction at every layer. If each $W_{\ell }^O$ is initialized to zero, the entire initialized TDLM denoiser equals the backbone on every input along any fixed trajectory, in exact arithmetic.

Proof. 

On the first pass, ${\mathcal{J}}_p\left(0\right)$ is empty, so the only attention entry is the zero-value sink. At initialization, multiplication by $W_{\ell }^O=0$ removes the correction even when history is available. Induction over backbone layers gives identical representations and output distributions.    □

Appendix B.3. Parallel and sequential evaluation

Proposition 2

(Parallel evaluation on a fixed trajectory). With dropout disabled and identical history selection, evaluating the sampled inputs together using A gives the same outputs as evaluating those inputs sequentially with temporal caches, in exact arithmetic.

Proof. 

The input embeddings agree for each $S_i$. Suppose the representations agree at the input to a layer. Spatial attention processes each sequence independently, so the post-attention representations $h_{\ell ,p}\left(S_i\right)$ also agree. Keys and values are computed before the temporal correction at this layer. The parallel mask therefore selects exactly the keys and values that sequential evaluation would have cached from earlier sampled inputs. The temporal update and the unchanged feed-forward sublayer then give the same layer outputs. Induction over layers establishes the claim.    □

This equivalence concerns a fixed set of sampled inputs. No hidden representations are computed or inserted into memory for unsampled decoding steps. Training histories and those accumulated during generation can consequently differ. Although backbone weights are frozen, gradients still propagate through later backbone computations to train the temporal modules.

Appendix D.1. Different Temporal-Layer Allocations

Table 4 favors sixteen dispersed layers, used for the main LLaDA results, among the measured allocations. They achieve 79.08% accuracy versus 77.86% for eight dispersed layers, at similar NFE and latency. Sixteen middle layers achieve nearly identical accuracy (79.00%), but require 45.72 NFE and 0.683 seconds, compared with 40.08 NFE and 0.585 seconds for the dispersed allocation. The thirty-two-layer configuration has no task-level measurements. Its training OOM status was supplied by the authors; a corresponding log is not included in the evidence snapshot.

Table 4. Different temporal-layer allocations on GSM8K. Every temporal module uses four heads of width 64; measured models share the 750-update training recipe.

Temporal layers	Accuracy (%)	NFE	Avg. sec
8, dispersed	77.86	40.15	0.571
16, middle	79.00	45.72	0.683
16, dispersed (main)	79.08	40.08	0.585
32, all	Training OOM

Table 4. Different temporal-layer allocations on GSM8K. Every temporal module uses four heads of width 64; measured models share the 750-update training recipe.

Temporal layers	Accuracy (%)	NFE	Avg. sec
8, dispersed	77.86	40.15	0.571
16, middle	79.00	45.72	0.683
16, dispersed (main)	79.08	40.08	0.585
32, all	Training OOM

Appendix D.2. Temporal Width and Head Configurations

In Table 5, the single-head runs use 500 updates, include 192-token training trajectories, and differ in length-sampling weights and lookahead scheduling. The multi-head runs use 750 updates and 128/256-token trajectories. Thus, the full table does not isolate width from training-recipe differences. The two width-256 rows do share training settings: four heads of width 64 achieve 79.00% accuracy versus 78.47% for two heads of width 128, with similar NFE and latency. These single-run differences do not establish statistical significance.

Table 5. Temporal-width and head configurations on GSM8K. All models augment the middle sixteen backbone layers (zero-based indices 8–23). Total width is the product of head count and head width. Training-recipe differences are described in the text.

Heads × head width	Total width	Updates	Accuracy (%)	NFE	Avg. sec
$1\times 64$	64	500	76.35	47.34	0.682
$1\times 128$	128	500	77.71	47.66	0.701
$2\times 128$	256	750	78.47	45.84	0.688
$4\times 64$	256	750	79.00	45.72	0.683

Table 5. Temporal-width and head configurations on GSM8K. All models augment the middle sixteen backbone layers (zero-based indices 8–23). Total width is the product of head count and head width. Training-recipe differences are described in the text.

Heads × head width	Total width	Updates	Accuracy (%)	NFE	Avg. sec
$1\times 64$	64	500	76.35	47.34	0.682
$1\times 128$	128	500	77.71	47.66	0.701
$2\times 128$	256	750	78.47	45.84	0.688
$4\times 64$	256	750	79.00	45.72	0.683