Sonar sensing on unmodified smartwatch

Abstract

Smartwatches are used by millions of people for applications in health, finance, and communication. However, their diminutive screen size limits the expressivity and security encompassing touchscreen interaction. To address this issue, this thesis explored two novel sonar-based approaches, each for general and user-specific perspectives. Our first exploration targets the realm of general interaction, specifically finger identification. Despite the recognized potential of finger identification for enhancing smartwatch expressivity, its implementation remains challenging, often relying on external devices (e.g., worn magnets) or explicit instructions. Addressing these limitations, this paper explores a novel approach to natural and unencumbered finger identification on an unmodified smartwatch: sonar. To do this, we adapt an existing finger-tracking smartphone sonar implementation rather than extract finger motion, we process raw sonar fingerprints representing the complete sonar scene recorded during a touch. We capture data from 16 participants operating a smartwatch and use their sonar fingerprints to train a deep learning recognizer that identifies taps by the thumb, index, and middle fingers with an accuracy of up to 93.7%, sufficient to support meaningful application development. We then pivot to a user-specific angle, specifically user authentication. While various user authentication technologies have been extensively explored in smartphone use scenarios, the applicability of these approaches to smartwatches is typically limited due to the small watch form factor. To improve authentication on smartwatches, we propose SonarAuth, a novel user authentication system for unmodified commercial smartwatches using behavioral biometrics derived from motion, touch, and around-device motions. We collected data from 24 participants from single touch to the watch screen with the thumb, index, and middle fingers. Using a multi-modal deep learning classifier, we achieved a promising mean Equal Error Rate(EER) of 6.41% for user authentication based on a single thumb tap. We note that our system is usable and has good potential to be combined with other authentication modalities. Through this holistic investigation, the thesis highlights the transformative capability of sonar sensing in unmodified smartwatches, forging a path for more intuitive and secure wearable interactions in the real world.