The Drupal Framework: a Case Study to Evaluate Variability Testing Techniques

Abstract

Variability testing techniques search for effective but manageable test suites that lead to the rapid detection of faults in systems with high variability. Evaluating the effectiveness of these techniques in real settings is a must but challenging due to the lack of variability-intensive systems with available code, automated tests and fault reports. in this paper, we propose using the Drupal framework as a case study to evaluate variability testing techniques. First, we represent the framework variability as a feature model. Then, we report on extensive data extracted from the Drupal git repository and the Drupal issue tracking system. Among other results, we identified 378 faults in single features and 11 faults triggered by the interaction between two of the features of Drupal v7.23, reported during a one-year period. These data may give a new insight into the distribution of faults in variability-intensive systems and the fault propensity of features. To show the feasibility of our work, we used the case study to evaluate the effectiveness of a historybased test case prioritization criterion. Results suggest that this technique could contribute to accelerate the detection of faults of test suites based on combinatorial testing.