ReLLM-OS: Chip Virtualization of Hardware Large Language Models (LLMs) over a Reusable Transformer Datapath

Large language model (LLM) accelerators are commonly optimized for fixed model structures and isolated inference workloads, whereas emerging artificial-intelligence operating systems (AI OSs) primarily virtualize agents, memory, and tools in software. This work investigates whether LLM inference can instead be exposed as a secure and reusable hardware service. We introduce HCDP-LLM, an AI-OS-managed hierarchical common datapath that composes reusable tensor, reduction, nonlinear, quantization, memory, and key–value (KV)-cache modules through model capsules and a Moore finite-state controller. The architecture is formalized as a finite-precision multi-tape computational transducer, enabling deterministic checkpointing, migration, isolation, and Turing-machine-based bit-time, space, and I/O analysis. The derived bounds distinguish quadratic prompt-attention work, context-linear per-token attention, and context-linear KV-cache storage, while exposing the different compute- and bandwidth-dominated regimes of prefill and decoding. The formal model further establishes that any fixed physical realization is necessarily finite-state, whereas a scalable implementation family with extensible read/write memory can support universal computation. These findings indicate that Hardware LLMs are feasible not as permanently hardwired models, but as runtime-composable, weight-programmable inference fabrics. HCDP-LLM therefore provides a pragmatic foundation for secure multi-model and multi-agent acceleration without full hardware resynthesis.