Which generated test failures are fault revealing? prioritizing failures based on inferred precondition violations using PAF

Abstract

Automated unit testing tools, such as Randoop, have been developed to produce failing tests as means of finding faults. However, these tools often produce false alarms, so are not widely used in practice. The main reason for a false alarm is that the generated failing test violates an implicit precondition of the method under test, such as a field should not be null at the entry of the method. This condition is not explicitly programmed or documented but implicitly assumed by developers. To address this limitation, we propose a technique called Paf to cluster generated test failures due to the same cause and reorder them based on their likelihood of violating an implicit precondition of the method under test. From various test executions, Paf observes their dataflows to the variables whose values are used when the program fails. Based on the dataflow similarity and where these values are originated, Paf clusters failures and determines their likelihood of being fault revealing. We integrated Paf into Randoop. Our empirical results on open-source projects show that Paf effectively clusters fault revealing tests arising from the same fault and successfully prioritizes the fault-revealing ones.