Parameter-Efficient Adaptation of Speech-Aware LLMs for Timestamp Prediction

Word-level timestamps are essential for many speech applications. Existing methods either degrade transcription quality when interleaving timestamps or require separate alignment models that do not integrate into speech LLMs. We propose a parameter efficient framework for speech recognition with timestamps (SRWT) within a single speech LLM. Our approach generates transcriptions in the first step and regenerates them with timestamps in the second, preserving transcription quality. We introduce modular low rank adaptation (LoRA) variants that train only on SRWT data without degrading base model capabilities, enabling isolated adaptation as speech LLMs scale to more tasks. We propose novel activated LoRA (aLoRA) for SRWT, enabling KV-cache reuse thus allowing improved performance through base model usage while preserving modularity. Experiments show a 27ms alignment error in English (35% relative improvement over the best baseline) and 21ms in multilingual data, while maintaining WER identical to the base model. Proposed methods also exhibit strong zero-shot transfer, achieving 37ms error on multilingual data when trained only on English.