Replication of Studies in Empirical Software Engineering: A Systematic Mapping Study, From 2013 to 2018

Abstract

Context: In any discipline, replications of empirical studies are necessary to consolidate the acquired knowledge. In Software Engineering, replications have been reported since the 1990s, although their number is still small. The difficulty in publishing, the lack of guidelines, and the unavailability of replication packages are pointed out by the community as some of the main causes. Objective: Understanding the current state of replications in Software Engineering studies by evaluating current trends and evolution during the last 6 years. Method: A Systematic Mapping Study including articles published in the 2013-2018 period that report at least one replication of an empirical study in Software Engineering. Results: 137 studies were selected and analysed, identifying: i) forums; ii) authors, co-authorships and institutions; iii) most cited studies; iv) research topics addressed; v) empirical methods used; vi) temporal distribution of publications; and vii) distribution of studies according to research topics and empirical methods. Conclusions: According to our results, the most relevant forums are the Empirical Software Engineering and Information and Software Technology journals, and the Empirical Software Engineering and Measurement conference. We observed that, as in previous reviews by other researchers, most of the studies were carried out by European institutions, especially Italian, Spanish, and German researchers and institutions. The studies attracting more citations were published mainly in journals and in the International Conference on Software Engineering. Testing, requirements, and software construction were the most frequent topics of replication studies, whereas the usual empirical method was the controlled experiment. On the other hand, we identified research gaps in areas such as software engineering process, software configuration management, and software engineering economics. When analysed together with previous reviews, there is a clear increasing trend in the number of published replications in the 2013-2018 period.