A family of coverage criteria based on patterns to the test of metaprograms

Abstract

Although there are several techniques for the automatic generation of test data based on grammars, few studies have been proposed to improve the test data generated by applying semantic restrictions. In this sense, we intend to contribute in this direction for the particular case of testing metaprograms, programs that have other programs as input. Currently, the natural alternative to testing this kind of program is using the grammar-based testing. That testing technique can be applied relatively easily, but with high costs, related to the generation and execution of the test set, and low effectiveness. Many researches and tools dedicated to the development of metaprograms make heavy use of pattern matching for their implementation and specification. In this case, the patterns offer an interesting source of information for creating tests that are syntactically valid and also satisfy semantic constraints. Given the limitation of grammar-based testing and pattern information on the metaprograms, we identified an opportunity to contribute to the improvement of the testing process for these programs. Therefore, the goal of this work is to evaluate the use of pattern information for the testing of pattern-based metaprograms and thus contribute to their testing process. In order to systematize the software testing process, a family of coverage criteria based on patterns is proposed to test pattern-based metaprograms efficiently and systematically. Four pattern-based coverage criteria are proposed, they are based on classical input space partitioning combination criteria. Furthermore, a hierarchical relationship between the criteria is presented. Therefore, different levels of rigor can be required by choosing the appropriate criterion. The validation of these contributions is made using a case study and empirical validation. The case study presents a reference instantiation for the test design process applied to a type checker implemented as metaprogram based on patterns. The type checker is tested using a test set generated by the pattern-based coverage criteria, and the quality of this set is evaluated using the mutation technique. The results obtained are compared with those produced by a test set generated by the grammar-based criteria. The experimental studies indicate the effectiveness of the application of these patternbased criteria and a gain of cost-return in relation to the grammar-based coverage criteria.