Visible to the public An Analysis of Utility for API Recommendation: Do the Matched Results Have the Same Efforts?

TitleAn Analysis of Utility for API Recommendation: Do the Matched Results Have the Same Efforts?
Publication TypeConference Paper
Year of Publication2020
AuthorsLi, H., Xie, R., Kong, X., Wang, L., Li, B.
Conference Name2020 IEEE 20th International Conference on Software Quality, Reliability and Security (QRS)
Date PublishedDec. 2020
ISBN Number978-1-7281-8913-0
KeywordsAPI, API Recommendation, APIs, application programming interface, compositionality, Conferences, correctness, Current measurement, evaluation, Programming, pubcrawl, Reliability engineering, resilience, Resiliency, security, software quality, software reliability, term frequency, utility

The current evaluation of API recommendation systems mainly focuses on correctness, which is calculated through matching results with ground-truth APIs. However, this measurement may be affected if there exist more than one APIs in a result. In practice, some APIs are used to implement basic functionalities (e.g., print and log generation). These APIs can be invoked everywhere, and they may contribute less than functionally related APIs to the given requirements in recommendation. To study the impacts of correct-but-useless APIs, we use utility to measure them. Our study is conducted on more than 5,000 matched results generated by two specification-based API recommendation techniques. The results show that the matched APIs are heavily overlapped, 10% APIs compose more than 80% matched results. The selected 10% APIs are all correct, but few of them are used to implement the required functionality. We further propose a heuristic approach to measure the utility and conduct an online evaluation with 15 developers. Their reports confirm that the matched results with higher utility score usually have more efforts on programming than the lower ones.

Citation Keyli_analysis_2020