Using a model-based test generator to test for standard conformance

In this paper we describe two experiments in the verification of software standard conformance. In our experiments, we use a model-based test generator to create a test suite for parts of the POSIX™ standard and another test suite for the specification of Java™ exception handling. We demonstrate that models derived from specifications produce better test suites than the suites specified by standards. In particular, our test suites achieved higher levels of code coverage with complete test requirements coverage. Moreover, the test suite for the Java study found code defects that were not exposed by other benchmark test suites. The effort involved in producing these models and test suites was comparable to the effort involved in developing a test suite by more conventional methods. We avoid the state space explosion problem by modeling only the external behavior of a specific feature of the standard, without modeling the details of any particular implementation.