Towards NVIDIA 1:4 Semi-Structured 75% Sparsity via Cannistraci–Hebb N:M Dynamic Sparse-to-Sparse Training

1. Introduction

Semi-structured sparsity has become a central hardware-oriented sparsity format for neural network acceleration [1]. Its most established instance is 2:4 sparsity, where two nonzero weights are retained in every group of four [2]; this format is natively supported by NVIDIA sparse Tensor Cores and has become the de facto reference point for semi-structured sparse training and pruning. A natural next algorithmic question is whether the same local N:M principle can be pushed to a more restrictive regime, such as 1:4 sparsity, where only one weight remains active in every group of four [3,4]. This setting is attractive because it preserves the group-of-four structure familiar from 2:4 sparsity while increasing the sparsity ratio to 75%. The main difficulty, however, is no longer only how to exploit a sparse pattern at inference time, but whether pre-training can preserve model quality under such an extreme legal constraint.

Although our title is hardware motivated, the technical claim of this paper is algorithmic. Current NVIDIA GPUs provide native sparse Tensor Core support for 2:4 execution, but not for native 1:4 execution. We therefore do not claim measured native 1:4 speedup on current NVIDIA hardware. Throughout the paper, “1:4 NVIDIA sparsity” refers to a hardware-forward N:M target that keeps the local group-of-four structure of the existing NVIDIA semi-structured sparsity interface, rather than to an already deployed NVIDIA execution mode. We want to clarify that many efficient-AI techniques first emerge as algorithmic evidence, and then hardware manufacturers provide support for them, such as 8-bit Training [5,6,7,8]. Our primary contribution is to study whether sparse pre-training algorithms can make this more aggressive 1:4 target viable in model quality. For hardware evaluation, we report measured speedups only for native 2:4 execution on NVIDIA hardware, and we separately provide a clearly labeled CSR sparse-GEMM surrogate as a projection for potential future 1:4 acceleration.

In this work, we show that moving from 2:4 to 1:4 sparsity is not a simple increase in sparsity ratio, but a regime shift in sparse pre-training. The 1:4 constraint changes the optimization problem in three coupled ways. First, proxy-gradient dense-to-sparse methods [3,9] become increasingly mismatched with the actually executed sparse model, because most weights in each local group are inactive in the forward pass while still receiving surrogate gradients. Second, low-rank residual branches [10] become more valuable, because the sparse backbone loses most local degrees of freedom and therefore needs an explicit compensation path for missing function directions. Third, matrix-geometry optimizers such as Muon-style [11] Newton–Schulz updates are promising but cannot be directly transplanted into dynamic sparse training, because the active support changes after each topology refresh. Appendix A provides the supporting sparsity-regime studies.

The first regime-shift signal concerns topology learning. Existing semi-structured sparse training methods such as STE-based [12] training typically maintain dense latent weights and use Straight-Through Estimator surrogate gradients to update weights that are masked out in the forward computation. This mismatch is moderate in 2:4 sparsity, where half of the weights in each local group remain active, but it is amplified in 1:4 sparsity, where three quarters of the weights are inactive during the forward pass [13]. This motivates true sparse-to-sparse training: only active weights are optimized, and the sparse topology evolves through legal remove-regrow operations. As shown in Table A1, the gap between proxy-gradient training and sparse-to-sparse topology learning widens sharply as the legal support becomes more restrictive, supporting the view that 1:4 requires a different training paradigm.

The second signal concerns residual capacity. In standard dense-model adaptation, LoRA is usually introduced as a lightweight task-adaptation module on top of an already expressive pretrained model [14]. In semi-structured sparse pre-training, its role is different: the auxiliary low-rank branch acts as a train-from-scratch residual compensation path for function directions suppressed by the sparse backbone. Table A2 shows that the gain from a 1M-param LoRA branch grows as the sparsity pattern becomes more restrictive. This does not by itself make low-rank compensation the main contribution of our method, but it motivates including a small residual branch when studying the extreme 1:4 regime.

The third and most important signal concerns optimization. Recent matrix-geometry optimizers [15] can substantially improve dense neural network training by shaping updates at the matrix level rather than treating coordinates independently [16]. Our sparsity-level optimizer sweep in Table A3 shows that such geometry is also useful for sparse pre-training. However, dynamic N:M sparse training introduces a topology mismatch that dense optimizers do not face: after each refresh, removed weights leave the executed model, newly activated weights enter with little optimizer history, and a dense matrix update may place mass on inactive coordinates. This identifies the central technical bottleneck of 1:4 sparse-to-sparse pre-training: matrix geometry must be made compatible with a changing legal sparse support.

Motivated by this diagnosis, we propose CHTsNM, a true sparse-to-sparse training framework centered on Topology-Aware Newton–Schulz optimization (TANS). TANS is the main technical contribution of this work. It adapts Newton–Schulz-style matrix updates to dynamic semi-structured sparse topology through three design choices: active-mask projection, which prevents the optimizer from becoming a hidden dense optimizer; active-support RMS matching, which preserves the trusted magnitude of the sparse preconditioned base update; and refresh-aware ramping, which gradually restores matrix geometry after topology changes. In this sense, TANS is not a direct application of Muon or a generic dense matrix optimizer to sparse weights, but a topology-aware interface between sparse preconditioning and matrix-level update geometry.

CHTsNM includes two lightweight supporting mechanisms around TANS. First, Contextually Modulated LoRA (CoMoLoRA) treats the auxiliary branch as an input-adaptive residual compensator for missing sparse-backbone directions. It starts from standard LoRA and modulates the LoRA rank space with input context, so the residual branch can emphasize or suppress compensation directions for different tokens. Second, Motif Pattern Revisitation (MPR) extends CHTs24-style sparse-to-sparse topology learning from block-wise motifs to row-wise N:M masks and discourages ineffective same-pattern revisits during row-wise refresh. We emphasize that CoMoLoRA and MPR are not positioned as equally central innovations to TANS. They are supporting mechanisms designed to test two consequences of the 1:4 regime shift: the increased value of residual compensation and the need for effective row-wise legal-pattern exploration.

We evaluate CHTsNM on LLaMA [17] pre-training benchmarks, focusing on 1:4 sparsity with additional 2:4 comparisons to characterize the sparsity-regime transition. We compare CHTsNM against STE [18], SR-STE [3], CAST [19], and CHTs24 [13] as the main method-level baselines. Each baseline is evaluated under its intended sparse-training rule, while CHTsNM uses TANS because topology-aware optimization is the proposed method rather than an external tuning advantage. To make this optimizer contribution explicit, we also provide an optimizer-controlled diagnostic in which the CHTs24 topology engine is fixed and only the update rule is changed. This separates the question of whether CHTsNM outperforms prior sparse-training recipes from the question of which optimizer design choices account for the gain.

The empirical results support the TANS-centered interpretation. CHTsNM substantially outperforms STE, SR-STE, CAST, and CHTs24 under matched sparsity settings, and its no-branch variant already improves strongly over prior sparse baselines, showing that the gain is not merely due to auxiliary parameters. Ablations show that TANS contributes the dominant improvement, while CoMoLoRA and MPR provide smaller but complementary gains in the regimes where they are designed to help. Together, these results suggest that 1:4 sparse pre-training can be very close to dense-level quality when dynamic sparse topology is paired with topology-aware matrix optimization, and when residual compensation and row-wise pattern exploration are added as targeted supporting mechanisms.

Our contributions are summarized as follows:

• We provide a systematic study of the transition from 2:4 to 1:4 semi-structured sparse pre-training, identifying 1:4 sparsity as a regime shift rather than a linear increase in sparsity ratio. We also explicitly separate this algorithmic study from any claim of currently available native 1:4 execution on NVIDIA GPUs.
• We propose Topology-Aware Newton–Schulz optimization (TANS), a refresh-aware matrix optimizer for dynamic N:M sparse-to-sparse training. TANS makes matrix-geometry updates compatible with changing sparse supports through active-mask projection, active-support scale matching, and refresh-aware ramping.
• We instantiate TANS inside CHTsNM, a practical sparse-to-sparse training framework for extreme N:M sparsity. CHTsNM includes two supporting mechanisms: CoMoLoRA for input-adaptive low-rank residual compensation, and MPR for row-wise legal-pattern exploration.
• We provide method-level, component-level, and optimizer-controlled experiments showing that TANS is the dominant source of improvement, while CoMoLoRA and MPR provide complementary gains. We further separate measured native 2:4 speedups from a clearly labeled CSR surrogate analysis for potential future 1:4 acceleration.

2. Related Work

N:M semi-structured sparsity and hardware scope.

N:M semi-structured sparsity restricts each group of M weights to contain at most N nonzeros [1,2,3]. Its most established hardware instance is 2:4 sparsity, which is supported by NVIDIA sparse Tensor Cores. In this work, 1:4 sparsity is used as a hardware-forward algorithmic target: it preserves the group-of-four structure while increasing sparsity to 75%. We do not assume native 1:4 execution on current NVIDIA GPUs; instead, we study whether sparse pre-training algorithms can make this more restrictive N:M target viable in model quality. We evaluate both row-wise and block-wise layouts, since row-wise masks naturally align with forward sparse multiplication, whereas block-wise or transposable N:M masks better preserve sparse structure across forward and backward computations [18,20].

Dense-to-sparse versus sparse-to-sparse N:M training.

Most N:M sparse training methods were developed around the 2:4 regime. STE-based training, SR-STE [3], S-STE [9], and continuous projection methods such as CAST [19] maintain dense or dense-latent parameters and use surrogate gradients or gradual projection to obtain legal sparse weights. This dense-to-sparse design becomes increasingly mismatched with the executed sparse model as the legal support becomes more restrictive. True sparse-to-sparse training instead optimizes only active weights and evolves topology through legal remove-regrow transitions. CHTs24 [13] is a representative sparse-to-sparse method for semi-structured topology learning. CHTsNM builds on this principle, extends it to the 1:4 regime and row-wise N:M masks, and introduces topology-aware matrix optimization as the central missing component.

Sparse residual compensation.

Low-rank branches are an effective way to recover capacity lost by sparse backbones. CHTsL [21] studies CHT-style sparse backbones with LoRA-like branches, while SLTrain [22] represents a broader sparse-plus-low-rank pre-training paradigm. For semi-structured sparsity, HASSLE-Free [23], SLOPE [24], and ARMOR [25] show that low-rank or factorized auxiliary capacity can improve the quality–efficiency trade-off of sparse models. Our CoMoLoRA component follows this residual-compensation view, but adds input-conditioned rank-space modulation inspired by contextual modulation methods such as Transponder [26]. We position CoMoLoRA as a lightweight supporting mechanism for 1:4 sparse pre-training, rather than as the main source of improvement.

Matrix-geometry optimization under changing sparse support.

Most semi-structured sparse training methods rely on Adam-style coordinate-wise optimizers; for example, CAST [19] proposes AdamS. In parallel, matrix-geometry optimizers updates [11] improve dense training by shaping updates at the matrix level. These optimizers, however, assume a stable dense parameterization. Dynamic N:M sparse training changes the active support after each topology refresh, so newly activated weights have limited optimizer history and unconstrained matrix updates may place mass on inactive coordinates. TANS bridges this gap by using Newton–Schulz geometry only as a masked, scale-calibrated, and refresh-aware residual correction on the active support.

A more detailed discussion is provided in Appendix B.

3. Method

3.1. Overview of CHTsNM

CHTsNM is designed for true sparse-to-sparse pre-training under legal $N:M$ semi-structured masks. For each target linear layer, we write the executed sparse weight as $W_t$. Unlike dense-to-sparse methods based on STE-style surrogate gradients, inactive weights are not used in the forward pass and are not updated. The mask evolves only through legal topology refreshes.

The framework has three components, ordered by their role in the training pipeline. First, Topology-Aware Newton–Schulz (TANS) is the main optimization component: it makes matrix-geometry updates compatible with changing sparse supports. Second, Contextually Modulated LoRA (CoMoLoRA) adds a small input-adaptive residual branch to compensate for function directions removed by the sparse backbone. Third, Motif Pattern Revisitation (MPR) improves the effectiveness of row-wise topology refreshes by discouraging ineffective same-pattern revisits. In a target linear layer with CoMoLoRA enabled, the output is

$$y=x{W}_t^{\top }+\Delta y_{\mathrm{CoMoLoRA}}\left(x\right),$$

where TANS optimizes the sparse backbone $W_t$, Adam optimizes the dense low-rank branch, and MPR modifies only the row-wise mask-refresh candidate set.

3.2. TANS: Topology-Aware Newton–Schulz Optimization

Modern matrix optimizers, such as Muon-style Newton–Schulz updates, suggest that matrix-level update geometry can improve neural network training. However, dynamic semi-structured sparse training introduces a topology mismatch that does not exist in dense optimization. In CHTsNM, the sparse backbone is optimized under a changing active support: after each topology refresh, removed weights leave the executed model, while newly activated weights enter the forward computation with little optimizer history. Therefore, a dense matrix-geometry update can be stale with the current legal sparse topology. TANS addresses this issue by using Newton–Schulz geometry as a topology-aware residual correction on top of a masked preconditioned base update.

For a sparse weight matrix $W_t=M_t\odot {\Theta }_t$, TANS first restricts optimization to the active support by using the masked gradient $G_t^M=M_t\odot G_t$. To stabilize optimization under the highly constrained 1:4 backbone, we compute a preconditioned gradient and a momentum-smoothed base direction:

$$\begin{array}{cc}\hfill V_t& ={\beta }_2{V}_{t-1}+(1-{\beta }_2){\left(G_t^M\right)}^2,{\widehat{G}}_t={\displaystyle \frac{G_t^M}{\sqrt{V_t}+ϵ}},\hfill \\ \hfill P_t& ={\beta }_1{P}_{t-1}+(1-{\beta }_1){\widehat{G}}_t,U_t^{\mathrm{base}}=M_t\odot \left[(1-{\beta }_1){\widehat{G}}_t+{\beta }_1{P}_t\right].\hfill \end{array}$$

This base update is not intended as a new optimizer family by itself. Instead, it provides a sparse-compatible direction whose magnitude is reliable on the actually executed support.

TANS then computes a Newton–Schulz matrix-geometry direction from the base update. We normalize $U_t^{\mathrm{base}}$ and apply $K=5$ polynomial Newton–Schulz iterations:

$$\begin{array}{cc}\hfill X_0& ={\displaystyle \frac{U_t^{\mathrm{base}}}{\parallel U_t^{\mathrm{base}}{\parallel }_F+ϵ}},\hfill \\ \hfill A_k& =X_k{X}_k^{\top },X_{k+1}=a_k{X}_k+\left(b_k{A}_k+c_k{A}_k^2\right)X_k,k=0,\dots ,K-1.\hfill \end{array}$$

The resulting matrix $O_t=X_K$ gives a matrix-shaped correction direction. For rectangular matrices, we transpose the update when needed so that the iteration is applied on the lower-dimensional side, and transpose it back afterward. The coefficient values are listed in Appendix C.

The topology-aware part of TANS consists of two legality and stability steps. First, since Newton–Schulz is a matrix operation, its output may contain nonzero values on inactive coordinates. TANS projects the correction back to the current active support $O_t^M$, which prevents the optimizer from becoming a hidden dense optimizer and preserves true sparse training. Second, TANS matches the correction scale using only active entries:

$$\begin{array}{cc}\hfill O_t^M=M_t\odot O_t,s_t& ={\displaystyle \frac{{RMS}_{M_t}\left(U_t^{\mathrm{base}}\right)}{{RMS}_{M_t}\left(O_t^M\right)+ϵ}},{\tilde{O}}_t=s_t{O}_t^M.\hfill \end{array}$$

Here ${RMS}_{M_t}(·)$ denotes the RMS computed over the active support of $M_t$. Active-support RMS matching lets the Newton–Schulz transform control update geometry while retaining the trusted magnitude of the sparse preconditioned base update.

Finally, TANS makes the geometry correction aware of topology age. Let ${\tau }_t$ be the number of optimizer steps since the last topology refresh of the current sparse matrix. We use a refresh ramp ${\gamma }_t$ and form the final sparse-backbone update direction as $U_t^{\mathrm{TANS}}$.

$$\begin{array}{c}\hfill {\gamma }_t={\gamma }_{\mathrm{NS}}min\left(1,{\displaystyle \frac{{\tau }_t}{B_{\mathrm{ramp}}}}\right),U_t^{\mathrm{TANS}}=U_t^{\mathrm{base}}+{\gamma }_t\left({\tilde{O}}_t-U_t^{\mathrm{base}}\right)\end{array}$$

Immediately after a topology refresh, TANS stays close to the stable preconditioned base optimizer. As the active support becomes older and more reliable, the Newton–Schulz geometry correction is gradually restored. This refresh-age modulation is the key distinction between TANS and directly applying a dense Muon-style optimizer to a dynamically sparse backbone.

The parameter update uses AdamW-style decoupled weight decay and a fan-ratio learning-rate scale ${\eta }_{\mathrm{mat}}=\eta \sqrt{d_{\mathrm{out}}/d_{\mathrm{in}}}$. TANS first applies multiplicative weight decay and then takes the sparse update step:

$${\Theta }_{t+1}\leftarrow (1-\eta {\lambda }_{\mathrm{wd}}){\Theta }_t-{\eta }_{\mathrm{mat}}{U}_t^{\mathrm{TANS}}.$$

We apply TANS only to the semi-structured sparse backbone. CoMoLoRA parameters are optimized with Adam because the low-rank branch has a stable dense parameterization and does not undergo topology refresh. Thus, TANS is designed not as a generic optimizer for all parameters, but as a topology-aware interface between sparse preconditioning and matrix-geometry optimization under changing 1:4-active supports.

3.3. CoMoLoRA: Contextual Low-Rank Residual Compensation

Under 1:4 sparsity, the sparse backbone removes most local degrees of freedom, so a low-rank branch acts as a train-from-scratch residual compensation path rather than a standard adaptation module. CoMoLoRA makes this compensation input-adaptive by modulating the LoRA rank space:

$$m\left(x\right)=1+\gamma tanh\left(G\left(x\right)\right),\Delta y_{\mathrm{CoMoLoRA}}\left(x\right)=\left(m\left(x\right)\odot x{A}^{\top }\right)B^{\top }.$$

The last layer of G is initialized to zero, so $m\left(x\right)=1$ at initialization and CoMoLoRA starts exactly as standard LoRA. During training, it gradually learns input-dependent residual compensation for the sparse backbone. Appendix D shows the details and motivation of CoMoLoRA.

3.4. From Block-Wise CHTs24 to Row-Wise CHTsNM via Motif Pattern Revisitation

CHTs24[13] does not define a row-wise $N:M$ topology operator. CHTsNM supports row-wise semi-structured masks by partitioning each row into local groups of four weights and constraining each group to choose a legal row-wise pattern ${\mathcal{R}}_N$. This extension preserves true sparse-to-sparse training: inactive weights are not used in the forward pass and are not updated. For a row group g with regrowth score $S_g\in {\mathbb{R}}^4$, the standard row-wise projection selects the highest-scoring legal pattern from ${\mathcal{R}}_N$. However, in row-wise 1:4 there are only four legal patterns, so a refresh step may repeatedly select the previous pattern and produce little effective topology change. MPR addresses this issue by suppressing same-pattern revisitation with probability ${\rho }_{\mathrm{mpr}}$, and full details are provided in Appendix E.

$$p_g^{\mathrm{new}}=arg\underset{p\in {\mathcal{R}}_{N,g}^{\mathrm{MPR}}}{max}\langle p,S_g\rangle ,{\mathcal{R}}_{N,g}^{\mathrm{MPR}}=\left\{\begin{array}{cc}{\mathcal{R}}_N\setminus \left\{p_g^{\mathrm{old}}\right\},\hfill & \mathrm{with}\mathrm{probability}{\rho }_{\mathrm{mpr}},\hfill \\ {\mathcal{R}}_N,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

4. Experiments

4.1. Main Results: Narrowing the 1:4 Sparse–Dense Gap

Table 1 compares CHTsNM with STE, SR-STE, CAST, and CHTs24 under matched sparsity settings. Under 1:4 sparsity, CHTsNM with a 1% auxiliary branch reduces the PPL of the strongest CHTs24 baseline by $21.6\%$ on LLaMA-60M and $18.2\%$ on LLaMA-130M, averaged over row-wise and block-wise layouts. The resulting 1:4 models are only $2.1\%$ and $6.3\%$ above the dense references on LLaMA-60M and LLaMA-130M, respectively. At the 2:4 sparsity, CHTsNM still outperforms dense on the LLaMA-350M model.

The no-branch variant of CHTsNM already outperforms all sparse baselines across both 1:4 and 2:4 regimes, indicating that the gain is not merely due to adding auxiliary parameters. The 1% CoMoLoRA branch further improves the 1:4 results, where the sparse backbone has fewer local degrees of freedom. Appendix F visualizes these results, and Appendix G reports consistent zero-shot improvements on WikiText [27], LAMBADA [28], SciQ [29], and ARC-Easy [30].

These results support the central empirical claim: 1:4 sparsity can approach dense-quality pre-training when sparse topology learning, residual compensation, and topology-aware optimization are redesigned jointly. Section 3.2 further studies the optimization component, and Appendix H provides baseline and training details.

4.2. Ablation: TANS Is the Primary Driver

Table 2 isolates the two main components of CHTsNM across sparsity regimes and mask layouts. The largest improvement comes from TANS. Averaged over row-wise and block-wise results, adding TANS reduces PPL by $17.9\%$ under 1:4 sparsity and $16.7\%$ under 2:4 sparsity. CoMoLoRA provides a smaller but complementary gain: by itself, it reduces PPL by $7.3\%$ under 1:4 and $3.5\%$ under 2:4. When combined with TANS, CoMoLoRA gives an additional $4.3\%$ reduction under 1:4, compared with $0.8\%$ under 2:4. This supports two conclusions: topology-aware optimization is the main bottleneck, and residual low-rank compensation becomes more valuable as the sparsity pattern becomes more restrictive. Appendix F visualizes this ablation.

4.3. Why Dense Matrix Geometry Needs Topology Awareness

We next analyze whether matrix-geometry optimization can be directly transferred to dynamic 1:4 sparse pre-training. Table 3 compares increasingly sparse-compatible update rules under the same CHTs24 backbone. Geometry alone is not sufficient: the geometry-only update performs worse than AdamW. In contrast, a sparse preconditioned base update gives a large gain, and Newton–Schulz geometry becomes beneficial only when used as a residual correction on top of this sparse base. TANS further improves the correction by matching its active-support RMS scale and ramping it after topology refreshes, yielding the best operating point.

Appendix I.1 shows that RMS matching is not a brittle one-point trick; rather, it becomes essential for stability when ${\gamma }_{\mathrm{ortho}}$ is high. Removing RMS matching degrades PPL to $46.72$, whereas either RMS-matching variant recovers it to $36.93$. Appendix I.2 shows that refresh ramping is consistently useful under dynamic sparse topology. Moreover, explicit optimizer-state repair after refresh is complementary to ramping rather than a replacement for it, as detailed in Appendix I.3.

Overall, Appendix I shows that the gain does not come from a single fragile coefficient choice, but from a coherent response to dynamic sparse topology: keeping matrix-geometry updates numerically aligned, softening the post-refresh transition, and repairing optimizer history where the support actually changes.

This mechanism ablation separates the sparse base update, matrix-geometry correction, and scale control under one sparse setting. The optimizer benchmark in Table A9 in Appendix J evaluates whether TANS can be replaced by existing optimizers; TANS achieves the lowest final PPL for dense, CHTs 2:4, and CHTs 1:4 training, with the largest gap at 1:4 compared with AdamW and Muon.

4.4. Auxiliary Branch and Contextually Modulated LoRA

Design space.

Under 1:4 sparsity, the sparse backbone loses many local degrees of freedom, so an auxiliary branch should act as residual compensation rather than only as a standard adaptation module. We compare whether the branch should add new low-rank directions [24], align with the sparse backbone [21], or multiplicatively modulate sparse features [26].

Let $y_s=W_{s}x$ be the sparse-backbone output. We write the auxiliary-branch family as

$$y=p\left(x\right)\odot y_s+{\mathbb{1}}_{\Delta }{\Delta }_q\left(x\right),{\Delta }_q\left(x\right)={\displaystyle \frac{\alpha }{r}}B\left(q\left(x\right)\odot Ax\right),$$

where $p\left(x\right)$ controls product modulation on the sparse output, $q\left(x\right)$ controls rank-space modulation inside the low-rank branch, and ${\mathbb{1}}_{\Delta }$ indicates whether an additive residual branch is used. This form covers additive low-rank compensation, alignment-regularized compensation, product-only contextual gating, and CoMoLoRA. We denote these variants as Add, Add+Align, Prod-Gate, and CoMo, respectively; full formulas are provided in Appendix K.

Results.

Table 4 shows that Add+Align is strongest at the 1% budget, where the rank-space gate has limited modulation capacity. From 2% to 5%, CoMoLoRA consistently obtains the best PPL, indicating that additive residual compensation and contextual rank-space modulation are complementary. The small gap between Add and Add+Align suggests that, under 1:4 sparsity, the auxiliary branch should primarily recover residual directions missing from the sparse backbone rather than directly mimic the sparse output.

4.5. Motif Pattern Revisitation

Finally, we evaluate MPR as a row-wise topology-refresh refinement. On row-wise 1:4, all tested MPR values $\{0.25,0.5,0.75,1\}$ reduce PPL by more than 1% relative to disabling MPR, with the best setting $0.5$ reducing PPL by $1.64\%$. On row-wise 2:4, the best tested setting $0.75$ gives a smaller $0.27\%$ PPL reduction.

This difference is expected: under row-wise 1:4, changing the motif relocates the only active position in each $1\times 4$ group, so overly aggressive forcing can be disruptive; under row-wise 2:4, a motif change can preserve one active position and tolerate stronger revisit suppression. MPR is also not equivalent to increasing the DST refresh ratio $\zeta$: changing $\zeta$ controls how many groups are refreshed, whereas MPR controls whether a selected group actually changes its legal pattern. Detailed MPR statistics and $\zeta$ controls are provided in Appendix L.

5. Inference Speedup

Since the ablation shows that the LoRA branch is less important under 2:4 sparsity, we isolate the sparse backbone and measure its inference acceleration on current NVIDIA hardware. Table 5 reports the speedup of trained CHTsNM sparse-backbone checkpoints. The speedup increases with model scale and reaches up to $1.66\times$ on LLaMA-1B.

Current NVIDIA hardware does not support 1:4 execution. We therefore analyze 1:4 acceleration with a matched CSR sparse-GEMM surrogate. The CSR analysis gives an average $1.83\times$ relative runtime improvement from 2:4 to 1:4, which maps the best measured 2:4 result to a first-order target speedup of approximately $3.12\times$. Full throughput tables, hardware details, and the extrapolation protocol are provided in Appendix M.

6. Conclusion

We studied 1:4 semi-structured sparse pre-training as a more restrictive regime beyond standard 2:4 sparsity. Our results suggest that this transition changes the training problem: proxy-gradient dense-to-sparse methods become increasingly mismatched with the executed sparse model, low-rank residual capacity becomes more important, and matrix-level optimizers must respect changing sparse supports. Motivated by these observations, CHTsNM keeps the executed backbone sparse-to-sparse and jointly adapts topology refresh, residual compensation, and optimization for the 1:4 regime.

Empirically, TANS is the main contributor, making Newton–Schulz-style matrix geometry compatible with dynamic sparse topology through masked updates, active-support scale matching, and refresh-aware ramping. CoMoLoRA further improves input-adaptive residual compensation, especially under 1:4 sparsity, while MPR improves row-wise legal-pattern exploration. Together, these components substantially narrow the PPL gap between 1:4 models and dense references across LLaMA pre-training experiments. Since current hardware only provides native 2:4 sparse Tensor Core support, our 1:4 results should be interpreted as an algorithmic step toward future high-sparsity N:M hardware rather than a measured 1:4 deployment claim. We list the limitation of our research in Appendix N

Appendix A.1. Sparse-to-Sparse Training Matters Much More at 1:4

Our first study compares a sparse-to-sparse backbone (CHTs24) against the dense-to-sparse baseline (STE). At light sparsity the difference is visible but modest, while at 1:4 it becomes large. The key point is not only that CHTs24 is better, but that the gap widens sharply as the feasible support shrinks.

Table A1. Sparse training regime shift from STE to CHTs24. The PPL gap widens strongly as we move from 3:4 to 2:4 and especially to 1:4 sparsity.

Regime	Sparsity	CHTs24 PPL	STE PPL	STE − CHTs24
block-wise 3:4	25%	32.41	34.02	1.60
block-wise 2:4	50%	34.46	38.45	3.99
block-wise 1:4	75%	42.85	53.91	11.06

Table A1. Sparse training regime shift from STE to CHTs24. The PPL gap widens strongly as we move from 3:4 to 2:4 and especially to 1:4 sparsity.

Regime	Sparsity	CHTs24 PPL	STE PPL	STE − CHTs24
block-wise 3:4	25%	32.41	34.02	1.60
block-wise 2:4	50%	34.46	38.45	3.99
block-wise 1:4	75%	42.85	53.91	11.06

The most relevant transition for the main paper is the jump from 2:4 to 1:4: the PPL gap increases from $3.99$ to $11.06$. This supports the view that the 1:4 setting requires a sparse-to-sparse training method rather than a purely straight-through sparse proxy.

Appendix A.2. The Value of a LoRA Residual Branch Grows with Sparsity

Our second study examines a fixed 1M-param LoRA residual branch on top of static sparse backbones. Here the regime-shift pattern is especially clear: the denser settings already benefit slightly from LoRA, but the gain grows rapidly as sparsity becomes more severe. This is consistent with the interpretation that, under 1:4 sparsity, the low-rank branch is increasingly acting as a residual compensation path for missing local degrees of freedom.

Table A2. Value of a fixed small LoRA branch as sparsity increases. The benefit grows from a minor dense improvement to a much larger gain at 1:4.

Regime	Sparsity	Anchor PPL	+LoRA PPL	PPL Gain	Gain vs Dense Gain
Dense	0%	32.70	32.38	0.32	$1.0\times$
Block-Wise 3:4	25%	34.53	33.70	0.84	$2.46\times$
Block-Wise 2:4	50%	37.19	35.66	1.54	$4.24\times$
Block-Wise 1:4	75%	42.81	39.19	3.62	$8.89\times$

Table A2. Value of a fixed small LoRA branch as sparsity increases. The benefit grows from a minor dense improvement to a much larger gain at 1:4.

Regime	Sparsity	Anchor PPL	+LoRA PPL	PPL Gain	Gain vs Dense Gain
Dense	0%	32.70	32.38	0.32	$1.0\times$
Block-Wise 3:4	25%	34.53	33.70	0.84	$2.46\times$
Block-Wise 2:4	50%	37.19	35.66	1.54	$4.24\times$
Block-Wise 1:4	75%	42.81	39.19	3.62	$8.89\times$

The most important comparison again is 2:4 versus 1:4. The LoRA branch improves PPL by $1.54$ points at 2:4, but by $3.62$ points at 1:4. This suggests that the 1:4 regime increasingly rewards methods that restore missing capacity through an explicit residual path rather than relying only on the sparse backbone.

Appendix A.3. Dense-Oriented Optimizer Geometry Does Not Remove the 1:4 Transition

Our third study compares Muon against an AdamW anchor on the same static sparse backbones. In absolute sparse PPL, Muon remains better than AdamW at all three sparsity levels. However, relative to each optimizer’s own dense baseline, Muon still suffers large degradation as sparsity tightens. This makes the optimizer picture more nuanced than the topology picture: better dense geometry helps, but it does not neutralize the 1:4 transition.

Table A3. Muon versus AdamW across sparsity regimes. “$\Delta$dense” denotes the PPL increase relative to the dense model trained by the same optimizer. Muon keeps a better absolute sparse PPL, but both optimizers degrade substantially by 1:4.

Regime	Sparsity	Muon PPL	AdamW PPL	Muon ${\mathbf{\Delta }}_{\mathit{d}\mathit{e}\mathit{n}\mathit{s}\mathit{e}}$	AdamW ${\mathbf{\Delta }}_{\mathit{d}\mathit{e}\mathit{n}\mathit{s}\mathit{e}}$
Block-Wise 3:4	25%	31.55	34.33	1.73	1.81
Block-Wise 2:4	50%	34.18	37.19	4.36	4.68
Block-Wise 1:4	75%	39.34	42.74	9.52	10.22

Table A3. Muon versus AdamW across sparsity regimes. “$\Delta$dense” denotes the PPL increase relative to the dense model trained by the same optimizer. Muon keeps a better absolute sparse PPL, but both optimizers degrade substantially by 1:4.

Regime	Sparsity	Muon PPL	AdamW PPL	Muon ${\mathbf{\Delta }}_{\mathit{d}\mathit{e}\mathit{n}\mathit{s}\mathit{e}}$	AdamW ${\mathbf{\Delta }}_{\mathit{d}\mathit{e}\mathit{n}\mathit{s}\mathit{e}}$
Block-Wise 3:4	25%	31.55	34.33	1.73	1.81
Block-Wise 2:4	50%	34.18	37.19	4.36	4.68
Block-Wise 1:4	75%	39.34	42.74	9.52	10.22

The takeaway is that the transition from 2:4 to 1:4 is not solved by optimizer choice alone. Even when Muon remains the stronger absolute optimizer, the sparse penalty still grows sharply at 1:4, indicating that the core difficulty is structural rather than purely geometric.

Summary. Taken together, these three studies motivate our broader claim that the jump from 2:4 to 1:4 is a genuine training-regime transition. At 1:4, sparse-to-sparse training method becomes much more important, optimizer improvements alone do not remove the sparse penalty, and dense residual compensation becomes substantially more valuable.

Appendix D.1. Sparse Pretraining Changes the Role of LoRA

In standard dense adaptation, LoRA is often interpreted as a lightweight task-adaptation module. Our setting is different. Under legal semi-structured sparsity such as $1:4$, the executed backbone weight W has already lost most local degrees of freedom before training converges. As a result, the auxiliary low-rank branch is better understood as a train-from-scratch residual compensation path rather than as a conventional adapter.

Let $W^{★}$ denote the dense weight that would ideally be learned by a dense model under the same training budget, and let W denote the executed sparse weight. The missing output component is

$$r\left(x\right)=x{(W^{★}-W)}^{\top }.$$

Standard LoRA approximates this missing component by a fixed low-rank residual

$$\Delta y_{\mathrm{LoRA}}\left(x\right)=x{A}^{\top }{B}^{\top }.$$

This is already useful, because it supplies a dense compensation path that is not constrained by the sparse backbone mask. However, it is still a global low-rank compensator: all tokens share the same rank-space geometry and the same output directions.

Appendix D.2. A Least-Squares View of Residual Fitting

The residual-compensation interpretation can be made explicit through least squares. Suppose we have training activations ${\left\{x_i\right\}}_{i=1}^N$ and corresponding target residuals

$$r_i=r\left(x_i\right)=x_i{(W^{★}-W)}^{\top }.$$

Then standard LoRA can be viewed as solving the constrained regression problem

$$\underset{A,B}{min}{\displaystyle \sum _{i=1}^N}{∥r_i-x_i{A}^{\top }{B}^{\top }∥}_2^2.$$

Equivalently, if we define the rank hidden state

$$z_i=x_i{A}^{\top }\in {\mathbb{R}}^r,$$

then LoRA predicts

$$\Delta y_{\mathrm{LoRA}}\left(x_i\right)=z_i{B}^{\top }={\displaystyle \sum _{k=1}^r}{z}_{ik}{b}_k^{\top },$$

where $b_k$ is the k-th column of B. In this view, LoRA learns:

• a shared projection A that extracts rank-space coefficients, and
• a shared dictionary ${\left\{b_k\right\}}_{k=1}^r$ of output-space compensation directions.

This least-squares formulation also clarifies the limitation of standard LoRA in our sparse-pretraining regime. The best low-rank fit is forced to use a single global factorization to approximate a residual map whose structure may vary substantially across inputs. When the sparse backbone drops different useful coordinates for different tokens, the residual $r\left(x\right)$ is not only low-rank-like; it is also input-varying in strength and composition.

Appendix D.3. Why Fixed Low-Rank Compensation Is Not Enough

Under strong semi-structured sparsity, two tokens may require different compensation even when they activate the same low-rank subspace:

• one token may mainly need recovery along a small subset of rank directions;
• another token may require the same directions but with different amplitudes;
• a third token may need some directions to be suppressed because the sparse backbone already handled that local pattern adequately.

Standard LoRA cannot express this distinction cleanly. Its coefficients $z_i=x_i{A}^{\top }$ are determined entirely by the shared projection A. Thus, the model must use the same rank-space response rule for all inputs. In effect, basis learning and coefficient selection are entangled inside one static factorization.

This motivates CoMoLoRA: keep the low-rank residual path, but add an explicit context-dependent control variable that decides how much of each learned rank direction should be used for the current token.

Appendix D.4. CoMoLoRA as Conditional Least-Squares Compensation

CoMoLoRA introduces an input-dependent modulation vector

$$m\left(x\right)=1+\gamma tanh\left(G\right(x\left)\right)$$

and applies it in the LoRA rank space:

$$\Delta y_{\mathrm{CoMoLoRA}}\left(x\right)=\left(m\left(x\right)\odot x{A}^{\top }\right)B^{\top }.$$

If we again write $z_i=x_i{A}^{\top }$, then CoMoLoRA predicts

$$\Delta y_{\mathrm{CoMoLoRA}}\left(x_i\right)={\displaystyle \sum _{k=1}^r}{m}_k\left(x_i\right)z_{ik}{b}_k^{\top }.$$

This admits a useful conditional least-squares interpretation. For fixed A and B, define the input-dependent design matrix

$$U_i=\left[\begin{array}{cccc}{z}_{i1}{b}_1^{\top }& z_{i2}{b}_2^{\top }& \dots & z_{ir}{b}_r^{\top }\end{array}\right]\in {\mathbb{R}}^{d_{\mathrm{out}}\times r}.$$

Then the CoMoLoRA prediction is

$$\Delta y_{\mathrm{CoMoLoRA}}\left(x_i\right)=U_{i}m\left(x_i\right),$$

and the ideal modulation for sample i would solve

$$\underset{m_i\in {\mathbb{R}}^r}{min}{∥r_i-U_i{m}_i∥}_2^2.$$

CoMoLoRA does not solve this least-squares problem independently for each token at inference time. Instead, it amortizes it by learning a shared predictor $G\left(x\right)$ that maps the input x directly to a good rank-space modulation vector $m\left(x\right)$. In this sense, CoMoLoRA can be viewed as learning an amortized conditional least-squares fit of the sparse residual.

Appendix D.5. Concrete Parameterization of G(x)

The abstract notation $G\left(x\right)$ above denotes a lightweight controller that maps the layer input to an r-dimensional rank-space modulation signal. In our implementation, $G\left(x\right)$ is a small two-layer MLP:

$$G\left(x\right)=W_2\varphi (W_{1}x+b_1)+b_2,$$

where

$$W_1\in {\mathbb{R}}^{h\times d_{\mathrm{in}}},W_2\in {\mathbb{R}}^{r\times h},$$

h is the controller hidden size, and $\varphi$ is a pointwise nonlinearity. In the current codebase, we use $\varphi =\mathrm{SiLU}$.

Thus the implemented CoMoLoRA branch can be written as

$$z=x{A}^{\top },G\left(x\right)=W_2\mathrm{SiLU}(W_{1}x+b_1)+b_2,$$

$$m\left(x\right)=1+\gamma tanh\left(G\left(x\right)\right),\Delta y_{\mathrm{CoMoLoRA}}\left(x\right)=\left(m\left(x\right)\odot z\right)B^{\top }.$$

The gate is therefore conditioned on the original layer input x, not on the rank hidden state z itself. This choice keeps the compensation basis $(A,B)$ and the context predictor G functionally separated.

Initialization. The final linear layer of G is initialized to zero, so that at initialization

$$G\left(x\right)=0,m\left(x\right)=1,$$

and the branch starts exactly as standard LoRA:

$$\Delta y_{\mathrm{CoMoLoRA}}\left(x\right)=x{A}^{\top }{B}^{\top }.$$

This makes CoMoLoRA a strict refinement of pure LoRA rather than a different branch from step 0.

Canonical setting in the main paper. For the main sparse-hybrid experiments, the approximately $1\%$ auxiliary branch uses the “pure internal CoMoLoRA” setting

$$r=7,h=2,\gamma =0.5.$$

Here r controls the LoRA rank, h controls the hidden size of the controller MLP G, and $\gamma$ controls the strength of the residual modulation around the identity gate $m\left(x\right)=1$.

Appendix D.6. Why Modulate in Rank Space

The rank-space placement is important. If one modulates only after the B projection, the gate acts on output channels rather than on compensation directions. That loses the decomposition that makes LoRA attractive in the first place. By contrast, rank-space modulation preserves the interpretation

$$“\mathrm{shared}\mathrm{low}-\mathrm{rank}\mathrm{directions}”+“\mathrm{input}-\mathrm{dependent}\mathrm{usage}\mathrm{of}\mathrm{those}\mathrm{directions}”.$$

This keeps the method parameter-efficient:

• A and B still define a shared low-rank residual basis;
• $G\left(x\right)$ only needs to output an r-dimensional control vector;
• modulation happens before the expensive expansion back to $d_{\mathrm{out}}$.

The resulting separation of roles is clean:

• A: extract candidate residual features from the input;
• B: decode those features into output-space compensation directions;
• $m\left(x\right)$: decide which rank directions are emphasized or suppressed for the current token.

Appendix D.7. Why the Residual Gate Matters

CoMoLoRA uses a residual gate

$$m\left(x\right)=1+\gamma tanh\left(G\right(x\left)\right)$$

rather than a purely suppressive gate such as $\sigma \left(G\right(x\left)\right)$. This has two benefits.

First, it preserves standard LoRA as the initialization point. Because the last layer of G is initialized to zero, we have

$$G\left(x\right)=0⟹m\left(x\right)=1⟹\Delta y_{\mathrm{CoMoLoRA}}\left(x\right)=x{A}^{\top }{B}^{\top }.$$

Thus CoMoLoRA starts exactly as pure LoRA and only later learns to deviate from it.

Second, the residual form allows both amplification and attenuation of each rank direction. This is important in sparse pretraining, where some tokens need stronger repair than pure LoRA would provide, while others need less.

Appendix D.8. Summary

The motivation for CoMoLoRA is therefore not merely “LoRA plus a gate”. Rather, it is the observation that in train-from-scratch sparse pretraining, the low-rank branch is fitting the residual left by the sparse backbone. Standard LoRA performs a global low-rank least-squares fit to that residual.

In short:

• sparse pretraining turns LoRA into a residual compensator;
• residual compensation can be written as a least-squares fitting problem;
• the residual is input-dependent under strong semi-structured sparsity;
• CoMoLoRA addresses this by making the low-rank coefficients input-adaptive;

Appendix E.1. From Block-Wise Motifs to Row-Wise Legal Patterns

CHTs24 was originally formulated for block-wise semi-structured sparse topology learning. Each local block selects a legal motif from a predefined motif library, and topology updates are performed as remove-regrow transitions between legal block motifs. This gives a true sparse-to-sparse training procedure, but the original formulation does not directly define a row-wise N:M topology operator.

CHTsNM extends the same sparse-to-sparse training principle to row-wise semi-structured masks. For a weight matrix W, each row is partitioned into contiguous groups of four weights. For each row group g, the mask pattern must belong to

$${\mathcal{R}}_N=\left\{p\in {\{0,1\}}^4:{\parallel p\parallel }_0=N\right\}.$$

The cases $N=2$ and $N=1$ correspond to row-wise 2:4 and row-wise 1:4 sparsity, respectively. Throughout training, CHTsNM keeps the executed model strictly legal: inactive weights are absent from the forward computation and are not optimized through dense-latent surrogate gradients. Topology changes are performed only through legal remove-regrow refreshes.

Appendix E.2. Standard Row-Wise Legal Projection

Let $S_g\in {\mathbb{R}}^4$ denote the regrowth score assigned to the four coordinates of row group g. The standard row-wise legal projection selects

$$p_g^{★}=arg\underset{p\in {\mathcal{R}}_N}{max}\langle p,S_g\rangle ,M_g\leftarrow p_g^{★}.$$

This operation is the row-wise counterpart of the block-wise motif selection used in CHTs24.

However, row-wise 1:4 has a very small legal pattern space. Each group contains only four legal patterns and only one active coordinate. If the regrowth score remains dominated by the currently active coordinate, the projection may return to the previous pattern:

$$p_g^{★}=p_g^{\mathrm{old}}.$$

In this case, the group is nominally refreshed by the DST schedule but its actual mask does not change. This reduces the effective refresh rate and weakens legal-pattern exploration.

Appendix E.3. Motif Pattern Revisitation

Motif Pattern Revisitation (MPR) is designed to make row-wise refreshes more effective. Its goal is not to refresh more groups, but to increase the probability that a selected group actually changes its legal pattern.

Let $p_g^{\mathrm{old}}\in {\mathcal{R}}_N$ be the pattern of group g before refresh. For each refreshed group, MPR samples

$$z_g\sim \mathrm{Bernoulli}\left({\rho }_{\mathrm{mpr}}\right),$$

where ${\rho }_{\mathrm{mpr}}$ is the revisit-suppression probability. When $z_g=1$, the previous pattern is removed from the candidate set; when $z_g=0$, CHTsNM uses the standard legal projection:

$${\mathcal{R}}_{N,g}^{\mathrm{MPR}}\left(z_g\right)=\left\{p\in {\mathcal{R}}_N|z_g=0\mathrm{or}p\ne p_g^{\mathrm{old}}\right\}.$$

The new pattern is then selected by

$$p_g^{\mathrm{new}}=arg\underset{p\in {\mathcal{R}}_{N,g}^{\mathrm{MPR}}\left(z_g\right)}{max}\langle p,S_g\rangle .$$

The parameter ${\rho }_{\mathrm{mpr}}$ controls an exploration–stability trade-off. When ${\rho }_{\mathrm{mpr}}=0$, MPR reduces to standard score-based legal projection. When ${\rho }_{\mathrm{mpr}}=1$, every refreshed group is forced to switch to a different legal pattern. Intermediate values preserve stable motifs while reducing ineffective same-pattern refreshes.

Appendix E.4. Difference from Increasing the DST Refresh Ratio

MPR is not equivalent to increasing the DST refresh ratio. Let $\zeta$ denote the fraction of row groups selected for refresh. Changing $\zeta$ changes how many groups are considered for update, whereas MPR changes the candidate set within each selected group. Thus, MPR affects

$$Pr\left(p_g^{\mathrm{new}}\ne p_g^{\mathrm{old}}\mid g\mathrm{is}\mathrm{selected}\mathrm{for}\mathrm{refresh}\right),$$

rather than the number of selected groups. This distinction matters because a larger refresh ratio can still produce many ineffective updates if the projection repeatedly selects the old pattern.

Appendix H.1. STE

The STE[18] baseline applies a legal semi-structured mask in the forward pass, but passes the identity gradient to the underlying dense weight. If $m\left(W\right)$ is the legal mask induced by the current weight W, then the forward uses

$$\tilde{W}=W\odot m\left(W\right),y=\tilde{W}x,$$

while the backward treats the layer as if the effective parameter were W itself. Equivalently, the autograd path is

$$W_{\mathrm{STE}}={\left(\tilde{W}-W\right)}^{\mathrm{stopgrad}}+W,$$

so the optimizer receives the task gradient with no extra sparse regularization term.

Project settings. For the canonical main runs, STE is used as a direct sparse-layer replacement baseline with:

• no DST scheduler,
• Adam,
• row-wise or block-wise legal masking depending on the experiment regime.

Appendix H.2. SR-STE

SR-STE[3] keeps the same straight-through forward path as STE, but adds a regularization gradient on pruned entries. Let $m\left(W\right)$ be the legal mask and let $\overline{m}=1-m$ denote the pruned support. The task gradient is augmented by

$${\nabla }_W{\mathcal{L}}_{\mathrm{SRSTE}}={\nabla }_W{\mathcal{L}}_{\mathrm{task}}+{\lambda }_{w}W\odot \overline{m}.$$

In other words, active entries still receive the ordinary task gradient, while pruned entries are softly pushed toward zero.

Project settings. We use:

• lambda_w$={10}^{-4}$,
• no DST scheduler,
• the same legal row-wise or block-wise mask family as the corresponding STE run.

Appendix H.3. CAST

CAST[19] was originally proposed as a sparse training recipe with additional post-training machinery. For fairness in our pretraining comparison, we remove the teacher-distillation / KD parts and keep only the sparse-optimization core:

• the same pretraining data and token budget as the other baselines,
• the same target linear layers,
• no teacher model, no KD loss, and no extra post-training stage beyond the final sparse export.

Training-time behavior. CAST is not a sparse-forward method like STE or SR-STE. During training it keeps a dense forward weight, while sparsity only enters through two components:

• AdamS, which applies selective ${\ell }_1$ decay to currently pruned coordinates,
• weight scaling, which learns per-group scaling factors to compensate magnitude shrinkage.

In our implementation, the dense backbone parameters outside CAST layers are optimized normally, while CASTlinear.weight uses AdamS through a hybrid optimizer.

Final projection. At export time, CAST performs a final legal sparse projection and folds the learned scaling factors back into the weight. This is the point where the model becomes actually sparse.

Project settings. Our CAST implementation differs from plain CAST in the finalization rule: the final legal mask is computed from the scaled weight, and the finalized sparse weight is obtained by projecting that scaled weight directly.

In addition, the canonical CAST runs in this repo use:

• scaling groups$=2$,
• mask update freq$=10$,
• lambda_w$=2\times {10}^{-4}$,
• fp32 storage for the scaler,
• fp32 multiply for weight * scaler, then cast back for the forward path.

Why CAST is only evaluated in row-wise mode. CAST is a dense-to-sparse projection method whose final sparse structure is induced only at export time from a dense weight. Therefore, in the training process, CAST is unable to enjoy the semi-structured sparse acceleration. However, block-wise sparsity only helps to accelerate the training process. It makes no sense for CAST to test in block-wise sparsity. Accordingly, the canonical comparisons only evaluate CAST on row-wise regimes, and we do not report a block-wise CAST baseline.

Appendix H.4. CHTs24

CHTs24[13] is our legality-preserving DST backbone baseline. Unlike STE, SR-STE, or CAST, it explicitly trains a sparse backbone throughout training and updates the legal support with a structured DST scheduler.

Core idea. CHTs24 alternates between:

• structured removal on the current support, and
• legality-preserving regrowth inside the same row-wise or block-wise semi-structured regime.

This keeps the model sparse during training rather than only at export time.

Canonical settings. The main experiments use:

• adaptive zeta [32]=True,
• zeta=0.1,
• DST remove method=weight magnitude with soft sampling,
• DST regrow method=CH2-L3n-soft [33],
• mask update interval=100 steps,
• Adam for the original CHTs24 baseline.

Appendix H.5. CHTsNM (∼1% Auxiliary Branch)

This is the main CHTsNM method. It starts from the same CHTs24 backbone, keeps TANS on the sparse backbone, and adds a small CoMoLoRA auxiliary branch (Optional).

Optimizer split. The defining implementation choice is the optimizer split:

• the sparse backbone is optimized by TANS,
• the CoMoLoRA branch is optimized by Adam.

Auxiliary-branch scale. The canonical main setting uses an approximately 1% auxiliary branch budget. Operationally, this is the “pure internal CoMoLoRA” branch added on top of a CHTs24 sparse backbone.

• Parameter budget= 1%,
• CoMoLoRA LoRA rank=7,
• Gated MLP hidden size=2,
• gamma=0.5.

Row-wise MPR variants. For row-wise follow-ups, the same MPR settings are used as above:

• row-wise 1:4: MPR forcing probability $0.5$,
• row-wise 2:4: MPR forcing probability $0.75$.

TANS. The pure backbone TANS follow-up keeps the same CHTs24 topology engine but swaps the optimizer to TANS. The canonical settings are:

• lr$={10}^{-3}$,
• eps$={10}^{-5}$,
• beta1$=0.9$,
• beta2$=0.999$,
• weight decay$=0$,
• tans gamma ortho$=0.5$,
• refresh blend steps$=8$.

Appendix I.1. RMS-Matching Robustness Grid

Table A6 reports the RMS-matching grid . Two points stand out. First, the exact RMS-matching variant is a second-order design choice at moderate matrix-correction strength. Second, RMS matching becomes critical as the correction strength grows: at ${\gamma }_{\mathrm{ortho}}=1.0$, the no-RMS variant collapses to $46.72$ PPL, while either RMS-matching variant recovers the result to $36.93$.

Table A6. PPL of the TANS RMS-matching robustness grid under a fixed matrix-geometry path. The main pattern is not that one RMS variant always wins, but that RMS matching becomes increasingly important as the correction strength grows.

${\mathit{\gamma }}_{\mathbf{ortho}}$	none	pre-mask global	post-mask active
0.25	34.63	34.95	35.14
0.50	34.61	34.27	34.51
0.75	35.39	34.57	34.77
1.00	46.72	36.93	36.93

Table A6. PPL of the TANS RMS-matching robustness grid under a fixed matrix-geometry path. The main pattern is not that one RMS variant always wins, but that RMS matching becomes increasingly important as the correction strength grows.

${\mathit{\gamma }}_{\mathbf{ortho}}$	none	pre-mask global	post-mask active
0.25	34.63	34.95	35.14
0.50	34.61	34.27	34.51
0.75	35.39	34.57	34.77
1.00	46.72	36.93	36.93

The grid therefore supports a robustness claim rather than a narrow one-point optimum. TANS does not require an exact RMS-matching choice to work well, but strong matrix-geometry updates do require some form of RMS calibration to stay numerically aligned with the active sparse backbone.

Appendix I.2. Refresh-Ramp Analysis

Table A7 isolates the effect of the refresh ramp under the default topology-refresh cadence. The no-ramp baseline reaches $34.53$ PPL. All tested nonzero ramps are competitive, and the best tested setting is the longest one in this sweep, ramp=32, which reaches $34.12$ PPL. Even the shortest nonzero ramp, ramp=2, already improves over the no-ramp baseline.

Table A7. Refresh-ramp sweep under the default dynamic-topology setting. Any nonzero ramp is at least competitive with the no-ramp baseline, and the best tested value is 32 steps.

Refresh ramp	Mean PPL	$\mathbf{\Delta }$ vs ramp 0
0	34.53	0.00
2	34.17	$-0.36$
4	34.44	$-0.09$
8	34.51	$-0.02$
16	34.36	$-0.17$
32	34.12	$-0.41$

Table A7. Refresh-ramp sweep under the default dynamic-topology setting. Any nonzero ramp is at least competitive with the no-ramp baseline, and the best tested value is 32 steps.

Refresh ramp	Mean PPL	$\mathbf{\Delta }$ vs ramp 0
0	34.53	0.00
2	34.17	$-0.36$
4	34.44	$-0.09$
8	34.51	$-0.02$
16	34.36	$-0.17$
32	34.12	$-0.41$

The main conclusion is not that the exact best ramp value is universal, but that post-refresh softness matters. This supports the interpretation that the topology refresh introduces a genuine optimizer-transition problem, and that a gradual restoration of full matrix correction is preferable to an abrupt switch.

Appendix I.3. Optimizer-State Reset Versus Refresh Ramp

Table A8 compares no reset, changed-entry-only reset, and full-state reset, each with and without a refresh ramp. The best tested configuration is changed-entries reset + ramp=8, which reaches $34.27$ PPL. This is better than the corresponding no-reset setting ($34.51$) and also better than full-state reset with the same ramp ($34.38$).

Table A8. Optimizer-state reset versus refresh ramp. A localized reset on only the changed entries works best when combined with ramping, suggesting that the optimizer mismatch is concentrated on refreshed coordinates rather than on the whole matrix state.

State-reset mode	Ramp 0 PPL	Ramp 8 PPL
none	34.53	34.51
changed entries only	34.50	34.27
all state entries	34.45	34.38

Table A8. Optimizer-state reset versus refresh ramp. A localized reset on only the changed entries works best when combined with ramping, suggesting that the optimizer mismatch is concentrated on refreshed coordinates rather than on the whole matrix state.

State-reset mode	Ramp 0 PPL	Ramp 8 PPL
none	34.53	34.51
changed entries only	34.50	34.27
all state entries	34.45	34.38

This result is mechanistically useful. It suggests that the refresh mismatch is not well modeled as a global optimizer restart problem. Instead, the damage is localized to coordinates whose sparse support status changes, so a targeted state repair is more appropriate than resetting the entire matrix state.

Appendix I.4. Summary

The extra benefit of TANS is not exhausted by the main optimizer-family ablation. Rather, the mechanism suite shows that the method remains effective across a nontrivial RMS-matching grid, benefits from explicit post-refresh softness, and is further improved by localized repair of optimizer state after topology changes. Together, these results support the view that TANS succeeds because it makes matrix-geometry optimization compatible with a changing legal sparse support, not because it depends on one fragile hyperparameter setting.

Appendix M.1. Hardware and Software Setup

All measurements were collected on a single NVIDIA RTX 5080 (Blackwell) under PyTorch 2.10.0 with CUDA 12.8 and TensorRT 10.15. Both engines are compiled in BF16; the only difference is the sparse-tactic flag enabling the 2:4 sparse Tensor Core path. Each (model, shape) cell reports the mean throughput from 100 timed iterations after 20 warmup iterations, executed back-to-back on the same process and device with no other GPU workloads running. Run-to-run variance for the throughput numbers is below one percent and is therefore not shown in the table.

Appendix M.2. Measured 2:4 Speedup Protocol

The main-paper speedup result compares two BF16 inference engines built from the same trained sparse-backbone checkpoint. The dense engine treats all weights as ordinary dense tensors, while the sparse engine enables the NVIDIA 2:4 sparse Tensor Core path. This keeps model weights, precision, batch size, and sequence length fixed, so the speedup isolates the effect of legal 2:4 sparse execution.

Post-training sparsification is not applied to every matrix. The transformer projection weights keep their trained 2:4 structure, while dense components such as the embedding and output head remain dense in both engines. This evaluates a realistic sparse-transformer deployment rather than an upper bound where every linear layer is made sparse after training.

Table A11. Full measured 2:4 inference throughput table. Throughput is reported as dense → sparse queries per second.

Model	Batch 8, Seq. 128		Batch 8, Seq. 256
	Throughput	Speedup	Throughput	Speedup
LLaMA-60M	$6103\to 7149$	$1.17\times$	$3184\to 3830$	$1.20\times$
LLaMA-130M	$2761\to 3435$	$1.24\times$	$1491\to 1956$	$1.31\times$
LLaMA-250M	$1337\to 1918$	$1.43\times$	$771\to 1064$	$1.38\times$
LLaMA-350M	$971\to 1380$	$1.42\times$	$527\to 785$	$1.49\times$
LLaMA-1B	$298\to 497$	$1.66\times$	$154\to 254$	$1.65\times$

Table A11. Full measured 2:4 inference throughput table. Throughput is reported as dense → sparse queries per second.

Model	Batch 8, Seq. 128		Batch 8, Seq. 256
	Throughput	Speedup	Throughput	Speedup
LLaMA-60M	$6103\to 7149$	$1.17\times$	$3184\to 3830$	$1.20\times$
LLaMA-130M	$2761\to 3435$	$1.24\times$	$1491\to 1956$	$1.31\times$
LLaMA-250M	$1337\to 1918$	$1.43\times$	$771\to 1064$	$1.38\times$
LLaMA-350M	$971\to 1380$	$1.42\times$	$527\to 785$	$1.49\times$
LLaMA-1B	$298\to 497$	$1.66\times$	$154\to 254$	$1.65\times$

Appendix M.3. Surrogate Extrapolation from 2:4 to 1:4

The 1:4 setting is the main target of this paper, but current NVIDIA sparse Tensor Cores do not expose a native 1:4 execution mode. The 1:4 result is therefore reported as a hardware-target extrapolation rather than a measured current-hardware result.

Let $t_{2:4}^{\mathrm{CSR}}$ and $t_{1:4}^{\mathrm{CSR}}$ denote measured CSR runtimes for matched sparse GEMM shapes under 2:4 and 1:4 masks. CSR is used only as a relative sparsity surrogate, not as an absolute performance model. The ratio

$$R_{\mathrm{CSR}}={\displaystyle \frac{t_{2:4}^{\mathrm{CSR}}}{t_{1:4}^{\mathrm{CSR}}}}$$

measures the relative gain from halving the number of active values under the same sparse-matrix backend. Across the model shapes used in this surrogate study, $R_{\mathrm{CSR}}$ ranges from $1.74\times$ to $1.88\times$ with mean $1.83\times$ (Figure A3). The hardware-target extrapolation is then

$$S_{1:4}^{\mathrm{target}}=S_{2:4}^{\mathrm{measured}}·R_{\mathrm{CSR}}.$$

This treats $R_{\mathrm{CSR}}$ as a first-order FLOP-scaling indicator: halving the number of active values approximately halves the arithmetic and operand traffic that a sparse-aware kernel must perform. It does not capture second-order effects such as instruction issue, scheduler behavior, or the exact memory hierarchy of a hypothetical native 1:4 Tensor Core, so the resulting target speedup should be read as a target consistent with FLOP scaling rather than as an architectural prediction.

Figure A3. Stability of the CSR 2:4-to-1:4 runtime ratio across model shapes used in the surrogate study. The dashed line marks the mean $1.83\times$.

Figure A3. Stability of the CSR 2:4-to-1:4 runtime ratio across model shapes used in the surrogate study. The dashed line marks the mean $1.83\times$.

Table A12. Surrogate extrapolation for potential native 1:4 Tensor Core inference. The 2:4 speedup column is the per-shape average from Table A11.

Model	CSR ratio ${\mathit{R}}_{\mathbf{CSR}}$	Measured 2:4 speedup	Target 1:4 speedup
LLaMA-130M	$1.74\times$	$1.28\times$	$2.22\times$
LLaMA-250M	$1.85\times$	$1.41\times$	$2.60\times$
LLaMA-350M	$1.84\times$	$1.46\times$	$2.68\times$
LLaMA-1B	$1.88\times$	$1.66\times$	$3.12\times$

Table A12. Surrogate extrapolation for potential native 1:4 Tensor Core inference. The 2:4 speedup column is the per-shape average from Table A11.

Model	CSR ratio ${\mathit{R}}_{\mathbf{CSR}}$	Measured 2:4 speedup	Target 1:4 speedup
LLaMA-130M	$1.74\times$	$1.28\times$	$2.22\times$
LLaMA-250M	$1.85\times$	$1.41\times$	$2.60\times$
LLaMA-350M	$1.84\times$	$1.46\times$	$2.68\times$
LLaMA-1B	$1.88\times$	$1.66\times$	$3.12\times$

Appendix M.4. Hardware-Target 1:4 Plot

Figure A4 shows the target 1:4 acceleration.

Figure A4. Hardware-target 1:4 extrapolation. Blue bars are the per-shape averaged measured 2:4 speedups from the NVIDIA sparse Tensor Core path. Green bars multiply each measured 2:4 speedup by that model’s own $R_{\mathrm{CSR}}$ (shown under the model name), not by the cross-model mean. Coverage is restricted to the model sizes for which a matched CSR surrogate measurement was available.

Figure A4. Hardware-target 1:4 extrapolation. Blue bars are the per-shape averaged measured 2:4 speedups from the NVIDIA sparse Tensor Core path. Green bars multiply each measured 2:4 speedup by that model’s own $R_{\mathrm{CSR}}$ (shown under the model name), not by the cross-model mean. Coverage is restricted to the model sizes for which a matched CSR surrogate measurement was available.

Appendix N.1. Training-Side Speed and Energy

Quality results aside, we also measure the wall-clock and energy cost of one training step under the same optimizer/sparsity combinations. Each configuration runs 1000 steps of LLaMA pre-training at sequence length 256 in bf16 on a single NVIDIA RTX 5080, with concurrent pynvml power sampling for the energy and average-power numbers; see Appendix M.1 for the broader hardware and software setup.

At LLaMA-60M the per-step optimizer cost is large relative to forward and backward, so the ranking is dominated by optimizer overhead alone. We therefore report 350M, where the model contributes most of the per-step cost and the comparison better reflects realistic training; the 60M numbers behave the same way qualitatively.

Table A13 shows that TANS is slower than dense AdamW, since the orthogonalized update is substantially more expensive than an AdamW step. Against Muon the picture reverses: at every sparsity level, TANS is faster and lower-energy than the matched Muon configuration, roughly $10\%$ faster and about 15 to $20\%$ less energy, and the dense TANS run is itself already faster and lower-energy than dense Muon. Mean power is also lower for TANS than Muon in each measured comparison: $172.8$ vs. $189.6$ W for dense, $177.4$ vs. $192.5$ W for CHTs 2:4, and $171.4$ vs. $188.8$ W for CHTs 1:4. The gap is concentrated almost entirely in the optimizer step; forward and backward times differ by only a few percent across configurations.

Table A13. LLaMA-350M training efficiency over 1000 steps, batch size 2, sequence length 256, bf16, RTX 5080. Energy is normalized as joules per token.

Setting	Step time	Opt. step	Throughput	J / token
Dense AdamW	248.19 ms	16.89 ms	2,025 tok/s	0.0893
Dense Muon	354.82 ms	124.46 ms	1,419 tok/s	0.1336
Dense TANS	315.43 ms	84.53 ms	1,595 tok/s	0.1083
CHTs 2:4 + TANS	328.26 ms	96.22 ms	1,533 tok/s	0.1157
CHTs 2:4 + Muon	366.93 ms	133.58 ms	1,373 tok/s	0.1402
CHTs 1:4 + TANS	331.84 ms	98.43 ms	1,517 tok/s	0.1130
CHTs 1:4 + Muon	367.18 ms	133.56 ms	1,372 tok/s	0.1376

Table A13. LLaMA-350M training efficiency over 1000 steps, batch size 2, sequence length 256, bf16, RTX 5080. Energy is normalized as joules per token.

Setting	Step time	Opt. step	Throughput	J / token
Dense AdamW	248.19 ms	16.89 ms	2,025 tok/s	0.0893
Dense Muon	354.82 ms	124.46 ms	1,419 tok/s	0.1336
Dense TANS	315.43 ms	84.53 ms	1,595 tok/s	0.1083
CHTs 2:4 + TANS	328.26 ms	96.22 ms	1,533 tok/s	0.1157
CHTs 2:4 + Muon	366.93 ms	133.58 ms	1,373 tok/s	0.1402
CHTs 1:4 + TANS	331.84 ms	98.43 ms	1,517 tok/s	0.1130
CHTs 1:4 + Muon	367.18 ms	133.56 ms	1,372 tok/s	0.1376