Submitted:
11 November 2026
Posted:
13 November 2026
You are already at the latest version
Abstract
AI agents call a language model, run a tool, and call the model again with the whole historyattached. Keeping an agent’s KV cache in GPU memory while it waits on a tool avoids recomputingthat history, but holds memory other requests could use. Prior systems (InferCept,Pensieve, Continuum, TokenCake and others) model this trade-off and evaluate it against earlierversions of vLLM. We ask how it plays out against the current prefix caches of vLLM 0.33.0 andSGLang 0.5.23. We built a serving engine in Rust (about 1,700 lines) with per-session KV retention,cross-session prefix sharing and session-level admission, and replayed 24 real coding-agentsessions (791 turns) against all three engines with the same GPU, model and KV cache size, tworuns per configuration. With a fixed 0.5-second tool wait, close to the median gap measuredfor coding agents, our engine finishes the workload 1.31x faster than the best baseline (1.45xfaster than vLLM) and returns the first token about 5x sooner. With a 5-second wait the gain is1.13x (1.24x over vLLM). With a 60-second wait, all engines finish within 2% of each other andour engine is slower to the first token, consistent with prior reports that retention stops payingoff when agents are away long. Three pinning policies we tried, each with a pass/fail criterionwritten down before seeing its results, did not beat the baselines at 60 seconds.
Keywords:
LLM inference
; LLM serving
; AI agents
; KV cache
; prefix caching
; session pinning
; cache eviction
; agent workloads
; tool calls
; vLLM
; SGLang
; Rust
Copyright: This open access article is published under a Creative Commons CC BY 4.0 license, which permit the free download, distribution, and reuse, provided that the author and preprint are cited in any reuse.