One Model Definition, Many Accelerators: Scaling vLLM Across Hardware Without Forks

Open‑source inference projects like vLLM sit at a critical crossroads. On one hand, there is intense pressure to deliver state‑of‑the‑art performance for a small set of mission‑critical “head” models on rapidly scaling GPU infrastructure. On the other, a long tail of models and a diverse hardware ecosystem depend on continued support.

Balancing these demands is challenging. The abstractions that once enabled portability across models and hardware are being reworked—or removed—to unlock peak performance for frontier workloads. At the same time, users of tail models and alternative accelerators still expect support without slowing innovation.

In this talk, we present hardware‑agnostic model definitions for vLLM: a design approach that decouples model logic from hardware execution paths, enabling the same model definition to run efficiently across accelerators without forking or per‑platform maintenance.

Our approach relies on three pillars: (1) compatibility with torch.compile, (2) well‑defined extensibility hooks, and (3) strict isolation from hardware‑specific paths. We will show how this design supports Intel Gaudi/HPU and IBM Spyre without hardware‑specific modeling code.