Performance Metamorphic Testing: Motivation and Challenges
|Author||Segura Rueda, Sergio
Troya Castilla, Javier
Durán Toro, Amador
Ruiz Cortés, Antonio
|Department||Universidad de Sevilla. Departamento de Lenguajes y Sistemas Informáticos|
|Abstract||Performance testing is a challenging task mainly due
to the lack of test oracles, that is, mechanisms to decide whether
the performance of a program under a certain workload is either
acceptable or poor due to a performance ...
Performance testing is a challenging task mainly due to the lack of test oracles, that is, mechanisms to decide whether the performance of a program under a certain workload is either acceptable or poor due to a performance bug. Metamorphic testing enables the generation of test cases in the absence of an oracle by exploiting the relations (so–called metamorphic relations) between the inputs and outputs of multiple executions of the program under test. In the last two decades, metamorphic testing has been successfully used to detect functional faults in a variety of domains, ranging from web services to simulators. However, the applicability of metamorphic testing to detect performance bugs is a topic that remains unexplored. In this vision paper, we introduce Performance Metamorphic Relations (PMRs) as expected relations between the performance measurements of multiple executions of the program under test. We hypothesize that these relations can be turned into assertions for the automated detection of performance bugs removing the need for complex benchmarks and domain experts guidance. As a further benefit, PMRs can be turned into fitness functions to guide search-based techniques on the generation of test data that violate the relations, revealing bugs. This novel idea is motivated with examples and an overview of some of the challenges in this promising topic.
|Citation||Segura Rueda, S., Troya Castilla, J., Durán Toro, A. y Ruiz Cortés, A. (2017). Performance Metamorphic Testing: Motivation and Challenges. En ICSE-NIER '17: IEEE/ACM 39th International Conference on Software Engineering: New Ideas and Emerging Technologies Results Track (7-10), Buenos Aires, Argentina: ACM Digital Library.|