Bringing GenAI awareness to workload management

With the rise of Large Language Models, enterprises today, use fine-tuning to adopt models for their specific usecases. Fine tuning jobs are compute intensive, and hence, while deploying such jobs, a scheduler must efficiently allocate the resources of accelerators such as GPUs. However, cluster orchestrators such as Kubernetes do not consider accelerators as first class citizens, and hence, are not equipped to ensure optimized resource allocations for fine tuning jobs. To address this issue, in this paper, we propose KAM, a Generative AI (GenAI)-aware workload manager. KAM provides users with the flexibility to choose and scale the amount of GPU allocations for a job with the aim of reducing its makespan and improve GPU utilization. It is light weight and works alongside existing schedulers. Experimental results show that KAM reduces job wait time by 3.5× and improves makespan by 1.6× on real workload traces, thereby making it a valuable addition to enterprise GenAI deployments.