Efficient Sensor Fusion for Gesture Recognition on Resource-Constrained Devices

arXiv:2605.13462 · cs.LG · Submitted 2026-05-13 · Read on arXiv

cs.LG

Submitted: 2026-05-13

Updated: 2026-05-13

Comments: The article is already accepted for IEEE Sensors Applications Symposium (IEEE SAS) 2026

Journal ref: 2026 IEEE Sensors Applications Symposium (SAS)

DOI: 10.1109/SAS70013.2026.11744138

License: http://creativecommons.org/licenses/by-nc-nd/4.0/

The gist: Gesture recognition is a cornerstone of Human-Computer Interaction (HCI) for smart eyewear, enabling natural and device-free control in augmented reality environments.

Terminology

Abstract

Gesture recognition is a cornerstone of Human-Computer Interaction (HCI) for smart eyewear, enabling natural and device-free control in augmented reality environments. Traditional vision-based approaches face significant challenges regarding power consumption, computational latency, and user privacy. This paper proposes a lightweight, privacy-preserving gesture recognition system based on the fusion of low-resolution Time-of-Flight (ToF) and Infrared (IR) thermal sensors. We used an 8 times 8 multizone ToF sensor (VL53L8CH) and an 8 times 8 IR array (AMG8833) to capture complementary depth and thermal cues. A compact Convolutional Neural Network (CNN) with a specialized grouped-convolution architecture is designed to fuse these modalities efficiently on a microcontroller (MCU). Experimental results on a custom dataset of 7 static gestures, validated via k-fold cross-validation, demonstrate that the proposed fusion strategy significantly outperforms single-sensor baselines with an accuracy of 92.3% and a macro F1-score of 0.93. Finally, on-device benchmarks on STM32F4 and STM32H7 MCUs confirm the system's suitability for resource-constrained wearables, requiring only 6,343 parameters and achieving millisecond-level inference latency with a total system power of 50 mW.

Sources

Related papers