Load Fiction: How Your Load Generator is Misleading You

Load Generators (LGs) are widely used to stress systems, measure performance, and identify bottlenecks. “Open-loop” LGs, used to schedule the sending of requests to the “system under test” (SUT) independent of the SUT’s response time, have become standard. Our main finding is that many open-loop LGs “silently” fail to deliver the intended workload. We call these failures silent because aggregate metrics reported by LGs, such as send rate or total request count, can make a load test appear correct. This paper identifies various ways in which LGs unintentionally fail to remain open-loop, examines the impact of these failures, and introduces a framework for detecting and classifying them. Our proposed XLG Inspector represents an observability layer for diagnosing LG behavior using in-flight request counts and request-level delay signatures. We integrate XLG Inspector with the popular LG called Vegeta. Empirical evaluation with systematic fault injections demonstrates that XLG Inspector identifies bottlenecks that prevent intended workloads from reaching the SUT.