DepthSketch-KV: Query-Conditioned Cross-Layer Residual Sketches for Long-Context KV Cache Compression

Long-context language model inference is increasingly limited by the memory footprint and bandwidth cost of KV caches, especially when retrieval-heavy prompts require preserving information across many layers. We introduce DepthSketch-KV, a cache compression method that exploits cross-layer redundancy by storing a shared low-rank residual sketch for adjacent transformer layers together with small per-layer key and value corrections. During decoding, query-conditioned depth gates estimate attention drift and choose whether each layer should access its local full-rank cache, the shared compressed cache, or a mixed reconstruction. A calibration-free error controller converts logit drift into per-sequence cache budgets, avoiding task-specific tuning while adapting to heterogeneous prompts. Across LongBench, Needle-in-a-Haystack, NarrativeQA, Qasper, HotpotQA, GovReport, MultiNews, HumanEval, and MBPP, DepthSketch-KV consistently reduces KV cache memory at 32k and 64k contexts while preserving retrieval, QA, summarization, and code completion accuracy. Compared with MiniCache, H2O, StreamingLLM, SnapKV, PyramidKV, KIVI, and FlexGen, it improves the accuracy-memory tradeoff, increases decoding throughput, and reduces time to first token with only small changes in perplexity.