Beyond the Execution Layer: Integrating Custom Accelerators with vLLM

What does it take to fit an accelerator with different execution constraints into vLLM without forking the serving stack? This poster presents our experience building the IBM Spyre out-of-tree plugin and examines where vLLM's extension points suffice and where device behavior requires changes beyond the execution layer.

We trace the serving path from hardware-agnostic model definitions in IBM's Foundation Model Stack through torch.compile with a Spyre backend that captures the FX graph and lowers it to the accelerator's native representation. We show how we adapted performance-critical vLLM features, including continuous batching with paged attention, chunked prefill, and prefix caching, to Spyre's execution model. We illustrate how device constraints propagate into scheduling through a plugin feature that pauses and resumes decoding sequences while retaining their on-device KV-cache and preserving serving semantics.

This poster presents architecture diagrams and scheduler timelines showing control and data flow across the plugin. They reveal where hardware assumptions surface across compilation, execution, and scheduling, and which vLLM interfaces custom accelerators need.