Quick Acquisition of Accurate Performance Models for New Configuration Space Areas

Performance modeling is a viable alternative to the often impractical task of exhaustively benchmarking an application’s performance. However, this is complicated by configuration spaces with numerous categorical parameters, a problem that is acute when these spaces expand frequently due to rapid application development. Such expansions necessitate the creation of new performance models, but the time spent acquiring training data creates a significant bottleneck, a problem not adequately addressed by prior work. To address this, we introduce Iterative Modeling, a sample-efficient process for building a predictive model. It systematically acquires data until model performance converges, thereby reducing acquisition time. Using a new benchmark based on Large Language Model fine-tuning we evaluate a wide-range of state-of-the-art model building strategies within this iterative framework to determine which is the most effective. As a key contribution, we introduce Selective Knowledge Transfer (SKT), a strategy designed specifically for this problem. SKT leverages prior knowledge from the original configuration space by incorporating it as a feature in the new regression problem, creating a dataset on which standard tabular machine learning algorithms can perform more effectively. We show that in the context of Iterative Modeling, SKT achieves the same or better predictive accuracy (mean absolute percentage error) with up to 14x less data than the state-of-the-art. In scenarios where data acquisition is the bottleneck, this translates to a 14x reduction in time. Furthermore, across fixed sampling or time budgets, SKT consistently delivers the most accurate surrogate models, establishing it as a competitive strategy for rapidly and accurately generating models for expanding configuration spaces.