Visible to the public Characterizing Scientific Reporting in Security Literature: An Analysis of ACM CCS and IEEE S&P PapersConflict Detection Enabled

TitleCharacterizing Scientific Reporting in Security Literature: An Analysis of ACM CCS and IEEE S&P Papers
Publication TypeConference Paper
Year of Publication2017
AuthorsBurcham, Morgan, Al-Zyoud, Mahran, Carver, Jeffrey C., Alsaleh, Mohammed, Du, Hongying, Gilani, Fida, Jiang, Jun, Rahman, Akond, Kafalı, Özgür, Al-Shaer, Ehab, Williams, Laurie
Conference NameProceedings of the Hot Topics in Science of Security: Symposium and Bootcamp
Date Published04/05/2017
Conference LocationHanover, MD, USA
ISBN Number978-1-4503-5274-1
Keywordsliterature review, science of security

Scientific advancement is fueled by solid fundamental research, followed by replication, meta-analysis, and theory building. To support such advancement, researchers and government agencies have been working towards a "science of security". As in other sciences, security science requires high-quality fundamental research addressing important problems and reporting approaches that capture the information necessary for replication, meta-analysis, and theory building. The goal of this paper is to aid security researchers in establishing a baseline of the state of scientific reporting in security through an analysis of indicators of scientific research as reported in top security conferences, specifically the 2015 ACM CCS and 2016 IEEE S&P proceedings. To conduct this analysis, we employed a series of rubrics to analyze the completeness of information reported in papers relative to the type of evaluation used (e.g. empirical study, proof, discussion). Our findings indicated some important information is often missing from papers, including explicit documentation of research objectives and the threats to validity. Our findings show a relatively small number of replications reported in the literature. We hope that this initial analysis will serve as a baseline against which we can measure the advancement of the science of security.

Citation KeyBurcham:2017:CSR:3055305.3055307
Refereed DesignationRefereed

Other available formats:

p13-Burcham.pdfPDF document718.98 KBDownloadPreview