A Shared KV Cache for vLLM: Reuse, Failover, and Elastic Scaling

In production vLLM, every replica owns its own KV cache by default. Replicas with overlapping prefixes each build their own copy, duplicating prefill across the fleet. When a replica fails, its in-flight requests recompute from the first token. When the autoscaler removes a replica, its warm cache goes with it; new replicas spin up cold.

The shift is to make the KV cache a property of the deployment, not the replica. A shared pool, external to the replicas and reachable by every vLLM instance, holds the cache for the fleet. Overlapping prefixes resolve to the same blocks; a failed replica's KV remains for siblings to claim; elastic events preserve fleet warmth.

This session presents the architecture of a shared KV-cache pool for vLLM: how a single pool serves the fleet, how a surviving replica claims a failed peer's blocks, and how Kubernetes scaling events preserve cache across elastic transitions. We close on where shared pools pay off, and where a per-replica pool still wins. Authors: Mengmei Ye and Chen Wang, IBM Research.