Visible to the public Recognition of Human Computer Operations Based on Keystroke Sensing by Smartphone Microphone

TitleRecognition of Human Computer Operations Based on Keystroke Sensing by Smartphone Microphone
Publication TypeJournal Article
Year of Publication2018
AuthorsYu, Z., Du, H., Xiao, D., Wang, Z., Han, Q., Guo, B.
JournalIEEE Internet of Things Journal
Date PublishedApril 2018
Keywordsacoustic features, Acoustic Fingerprints, Acoustics, activity recognition, classification model, composability, computer keyboard, Context-aware services, determined keystrokes, feature extraction, Feature fusion, fingerprint identification, fingerprint identification techniques, Games, Human Behavior, human computer interaction, human computer operation, human computer operation recognition, Keyboards, keystroke sensing, microphones, pubcrawl, Resiliency, semantic features, Semantics, Sensors, smart phones, smartphone microphone, smartphone sensing, ubiquitous computing, word recognition procedure

Human computer operations such as writing documents and playing games have become popular in our daily lives. These activities (especially if identified in a non-intrusive manner) can be used to facilitate context-aware services. In this paper, we propose to recognize human computer operations through keystroke sensing with a smartphone. Specifically, we first utilize the microphone embedded in a smartphone to sense the input audio from a computer keyboard. We then identify keystrokes using fingerprint identification techniques. The determined keystrokes are then corrected with a word recognition procedure, which utilizes the relations of adjacent letters in a word. Finally, by fusing both semantic and acoustic features, a classification model is constructed to recognize four typical human computer operations: 1) chatting; 2) coding; 3) writing documents; and 4) playing games. We recruited 15 volunteers to complete these operations, and evaluated the proposed approach from multiple aspects in realistic environments. Experimental results validated the effectiveness of our approach.

Citation Keyyu_recognition_2018