Understanding Small Scale Fine-tuning of LLM-as-a-Judge: A Case Study in Cybersecurity

Automatic evaluation of open-ended cybersecurity question answering is challenging due to specialized terminology and the impracticality of expert annotation at scale. We construct a 6,000-pair cybersecurity QA benchmark and fine-tune Llama-3.2-3B-Instruct using supervised fine-tuning followed by GRPO-based reinforcement learning with three ranking reward formulations. Our Composite Ranking Reward achieves the best overall performance, with an Exact Match of 0.599 and Pairwise Agreement of 0.855, substantially outperforming the base model and demonstrating the viability of compact, domain-adapted LLM-as-a-Judge models for cybersecurity evaluation.