Improving Hand Gesture Recognition via Infrared Tomography of the Wrist over Multiple Wearing Sessions

Abstract

Smartwatches now enable a wide range of end user applications. However, despite their increasingly sophisticated capabilities, the bandwidth of the user input that they support remains strongly limited by the small size of their touch screens. Numerous techniques have been proposed to improve this situation by integrating novel sensing systems and input modalities. A prominent approach here has been to detect gestures made by the hand wearing the watch with sensors capable of imaging either the associated distortions to the surface of the wrist, or changes to the wrist’s internal structures. While performance of such systems is promising (e.g., gesture accuracy of up to 93.3%), most studies currently examine performance during single studies and sessions. As such they fail to take account of the variability in measurement of wrist shapes and/or structures that might result from minor changes in sensor placement each time a device is donned. To explore the impact of this type of natural variability, we conducted a study using a watch strap prototype implementing infrared tomography to image the surface of the wrist during hand gesture production. While recognition performance during a single session of wearing this device was high (92.1%), it dropped substantially when the device was removed and re-worn between training and testing (to 22.9%). To alleviate this problem, we explore whether calibration processes that seek to maximize the consistency of sensor placements can yield improved performance. A study studies achieves this via IMU-based measurement of sensor placement similarity between sessions and shows greatly improved inter-session performance (up to 86.7%). Based on this result, we suggest that IMU based calibration of sensor placement can improve the real-world performance of gesture input systems based on wrist imaging techniques. © 2023, The Author(s), under exclusive license to Springer Nature Switzerland AG.