Automated testing on the analysis of variability-intensive artifacts: An exploratory study with SAT Solvers

Abstract

The automated detection of faults on variability analysis tools is a challenging task often infeasible due to the combinatorial com plexity of the analyses. In previous works, we successfully automated the generation of test data for feature model analysis tools using metamor phic testing. The positive results obtained have encouraged us to explore the applicability of this technique for the efficient detection of faults in other variability-intensive domains. In this paper, we present an auto mated test data generator for SAT solvers that enables the generation of random propositional formulas (inputs) and their solutions (expected output). In order to show the feasibility of our approach, we introduced 100 artificial faults (i.e. mutants) in an open source SAT solver and com pared the ability of our generator and three related benchmarks to detect them. Our results are promising and encourage us to generalize the tech nique, which could be potentially applicable to any tool dealing with variability such as Eclipse repositories or Maven dependencies analyzers.