Where the Blocks Live: Tier-Aware KV-Cache Routing for vLLM at Scale

Prefix-cache reuse is what makes long-context and multi-turn LLM serving economical: keeping KV blocks alive across requests turns prefills into HBM reads. Once GPU pressure forces eviction, the cache moves to CPU or storage — and if the scheduler cannot see which tier holds which block on which engine, offloaded blocks become invisible to routing, requests land on cold engines, and tail TTFT erupts.

llm-d, a CNCF open-source cloud-native distributed inference framework, closes the loop: emit KVEvents as blocks to move between tiers, and keep a near-real-time map of which engine holds each block on which tier — GPU, CPU, or shared storage. The router's scorer ranks engines by longest prefix match weighted by tier, so a CPU hit beats a cold engine, and a storage backend with GPU-direct DMA lets new engines read cache on first request.

This session walks through the design end-to-end: the OffloadingConnector, the KVEvents protocol, the tier-weighted scorer, and a benchmark showing low TTFT  after tier-aware routing. We close honestly: CPU offloading is the simpler default; storage offloading is the choice when prefix state needs to be shared across pods or outlive a single pod.