Hamster: A Large-Scale Study and Characterization of Developer-Written Tests

Automated test generation (ATG), which aims to reduce the cost of manual test suite development, has been investigated for decades and has produced countless techniques based on a variety of approaches: symbolic analysis, search-based, random and adaptive-random, learning-based, and, most recently, large-language-models-based approaches. However, despite this large body of research, there is still a gap in our understanding of the characteristics of developer-written tests and, consequently, in our assessment of how well ATG techniques and tools can generate realistic and representative tests. To bridge this gap, we conducted an extensive empirical study of developer-written tests for Java applications, covering 1.7 million test cases. Our study is the first of its kind in studying aspects of developer-written tests that are mostly neglected in the existing literature, such as test scope, complexity of test fixtures and assertions, types of inputs, and use of mocking. Based on the characterization, we then compare existing tests with those generated by two state-of-the-art ATG tools. Our results highlight that a vast majority of developer-written tests exhibit characteristics and complexity that are beyond the capabilities of current ATG tools. Finally, based on the insights gained from the study, we identify promising research directions that can help bridge the gap between current tool capabilities and more effective tool support for developer testing practices. We hope that this work can set the stage for new advances in the field and bring ATG tools closer to generating the types of tests developers write.