Visible to the public Will My Tests Tell Me if I Break This Code?

TitleWill My Tests Tell Me if I Break This Code?
Publication TypeConference Paper
Year of Publication2016
AuthorsNiedermayr, Rainer, Juergens, Elmar, Wagner, Stefan
Conference NameProceedings of the International Workshop on Continuous Software Evolution and Delivery
Date PublishedMay 2016
Conference LocationNew York, NY, USA
ISBN Number978-1-4503-4157-8
Keywordscode coverage, Measurement, Metrics, metrics testing, Mutation testing, pubcrawl, regression testing, test suite effectiveness

Automated tests play an important role in software evolution because they can rapidly detect faults introduced during changes. In practice, code-coverage metrics are often used as criteria to evaluate the effectiveness of test suites with focus on regression faults. However, code coverage only expresses which portion of a system has been executed by tests, but not how effective the tests actually are in detecting regression faults. Our goal was to evaluate the validity of code coverage as a measure for test effectiveness. To do so, we conducted an empirical study in which we applied an extreme mutation testing approach to analyze the tests of open-source projects written in Java. We assessed the ratio of pseudo-tested methods (those tested in a way such that faults would not be detected) to all covered methods and judged their impact on the software project. The results show that the ratio of pseudo-tested methods is acceptable for unit tests but not for system tests (that execute large portions of the whole system). Therefore, we conclude that the coverage metric is only a valid effectiveness indicator for unit tests.

Citation Keyniedermayr_will_2016