An Open-Source Software Metric Tool for Defect Prediction, Its Case Study and Lessons We Learned

Abstract

The number of research papers on defect prediction has sharply increased for the last decade or so. One of the main driving forces behind it has been the publicly available datasets for defect prediction such as the PROMISE repository. These publicly available datasets make it possible for numerous researchers to conduct various experiments on defect prediction without having to collect data themselves. However, there are potential problems that have been ignored. First, there is a potential risk that the knowledge accumulated in the research community is, over time, likely to overfit to the datasets that are repeatedly used in numerous studies. Second, as software development practices commonly employed in the field evolve over time, these changes may potentially affect the relation between defect-proneness and software metrics, which would not be reflected in the existing datasets. In fact, these potential risks can be addressed to a significant degree, if new datasets can be prepared easily. As a step toward that goal, we introduce an open-source software metric tool, SMD (Software Metric tool for Defect prediction) that can generate code metrics and process metrics for a given Java software project in a Git repository. In our case study where we compare existing datasets with the datasets re-generated from the same software projects using our tool, we found that the two datasets are not identical with each other, despite the fact that the metric values we obtained conform to the definitions of their corresponding metrics. We learned that there are subtle factors to consider when generating and using metrics for defect prediction.