Progressive Weight Refinement: Exact 8-Bit Decoding When the Model Does Not Fit

Auto-regressive decoding is limited by weight movement, not arithmetic. If an accurate 8-bit copy of the model does not fit in GPU memory, the usual choices are a coarser resident copy, which is fast but changes the model, or keeping part of the accurate copy in CPU memory, which preserves the model but is slow. Progressive Weight Refinement (PWR) splits each 8-bit round-to-nearest weight into a 4-bit high plane and a 4-bit low plane. The high planes stay on the GPU and are themselves a 4-bit model which drafts tokens. The low planes stay in pinned CPU memory and are streamed once per verification pass, overlapping the transfer with drafting, so several accepted tokens share one transfer. Because the draft is half of the target’s own bits, it costs no extra memory, and each verify moves only half of the 8-bit weights. The output matches the 8-bit target: greedy identity on 102,399 tokens from Qwen3-8B and 5,111 tokens from Qwen3-32B, and, for sampling, no deviation detected on 8B at a test resolution of about 0.02 total variation. On a 24 GB RTX 3090 Ti, where Qwen3-32B at 8-bit does not fit, PWR decodes at 8.99 tok/s, 1.80× llama.cpp’s automatic layer offload and 1.65x an upper bound on any whole-layer split. A three-term cost model built from per-layer transfer time and per-step draft time predicts measured round times to within 5% when transfer dominates. PWR is slower than resident 8-bit decode when the model fits and 4.6x slower than resident 4-bit on 32B, which costs 6–9% perplexity. It is a memory-placement method, not a general decoder speedup.