Deep Multimodal Wearable Sensor Fusion for Detection of Body-Focused Repetitive Behaviors

The paper introduces a multimodal deep learning framework for detecting and classifying body-focused repetitive behaviors (BFRBs), such as hair pulling and skin picking, using wrist-worn sensor data. The framework integrates inertial measurement units (IMU), thermopile (THM) sensors, and time-of-flight (ToF) sensors from the Helios wrist-worn device, capturing kinematic, thermal, and proximity information. The architecture combines a convolutional neural network with a gated recurrent unit (CNN-GRU), alongside modality-specific autoencoders and a late-fusion classifier, to exploit temporal and spatial dynamics.

The model achieved an F1 score of 0.985 and an area under the receiver operating characteristic curve (ROC-AUC) of 0.997 for binary detection, distinguishing BFRBs from other activities. For a nine-class scheme that distinguished each individual behavior from a single grouped Non-Target class, it achieved a macro-averaged F1 score of 0.700 with an ROC-AUC of 0.963, improving over single-modality baselines. Post-hoc interpretability based on Shapley additive explanations (SHAP) showed that the time-of-flight and inertial modalities dominated discriminative power by capturing spatial proximity and dynamic movement, while hierarchical clustering indicated that misclassifications were driven primarily by the anatomical region of the gesture.

The study concludes that multimodal sensor fusion enables accurate, objective, and continuous behavioral monitoring, establishing a foundation for real-time, wearable-assisted mental health diagnostics and personalized interventions in biomedical research and clinical care.

Improvements to AI systems:

1. Multimodal sensor fusion architecture: Integrate IMU, thermopile, and time-of-flight data using a CNN-GRU backbone with modality-specific autoencoders and late fusion. This enables the AI to jointly model kinematic, thermal, and proximity signals, capturing both fine-grained temporal patterns and spatial context that single-modality systems miss.

2. Hierarchical classification with anatomical awareness: Implement a two-tier classification scheme—first binary detection (target vs. non-target), then fine-grained behavior classification. Use hierarchical clustering of latent features to create an auxiliary loss that penalizes misclassifications based on anatomical region, improving robustness for gestures with similar motion but different body locations.

3. Interpretable decision-making via SHAP-based feature attribution: Add a post-hoc explainability layer that computes Shapley values for each sensor modality and time window. This allows the AI to dynamically weight modalities (e.g., prioritizing ToF for proximity and IMU for movement) and provide real-time, human-readable reasons for each detection, enabling clinicians to verify and trust the system.

4. Continuous learning from wearable data streams: Design the model to operate on sliding windows of sensor data with online inference, using the autoencoders to detect distribution shift and trigger retraining on new user-specific patterns. This supports personalized calibration without forgetting general behavior.

What the improved AI system can do:

• Detect and classify BFRBs (hair pulling, skin picking, etc.) with 98.5% F1 for binary detection and 70% macro-F1 for nine-class fine-grained classification, in real time from a wrist-worn device.

• Distinguish between behaviors occurring on different body parts (e.g., face vs. arm) even when motion patterns are similar, by fusing proximity and thermal cues.

• Provide clinicians with per-detection explanations (e.g., high hand-to-head proximity and rapid wrist rotation) to support remote monitoring and personalized intervention timing.

• Adapt to individual users over days, improving accuracy for unique gesture styles while maintaining generalizability across populations.

• Operate continuously on battery-constrained devices, enabling passive, objective tracking of BFRB episodes for mental health research and just-in-time behavioral therapy.