EgoEMG: A Multimodal Egocentric Dataset with Bilateral EMG and Vision for Hand Pose Estimation

summary

The gist

Surface electromyography (sEMG) records muscle activity during hand movement and can be decoded to recover detailed hand articulation The paper presents EgoEMG…

In short

EgoEMG introduces a multimodal dataset combining synchronized bilateral EMG and various vision modalities for hand pose estimation. The study evaluated different methods—EMG-to-pose, vision-to-pose, and fusion—showing that combining EMG with vision consistently improves accuracy, especially when visual evidence is occluded or ambiguous. This suggests physiological signals offer unique cues beyond what cameras can see.

Key concepts

EgoEMG Dataset
This is a new dataset that synchronizes multiple data sources: 16 channels of bilateral wristband EMG, IMU data, wide-angle RGB video, external RGB-D video, and motion capture data. It includes labels for 60 different hand gestures performed by 41 participants over more than ten hours.
Bimanual Hand Pose
This refers to estimating the precise 3D position and orientation of both hands simultaneously during movement. The target is a detailed vector describing every joint angle in both hands, including finger angles and wrist articulation angles.
Modality Complementarity
This analyzes how different data types (like EMG and vision) work together. The research found that fusion consistently outperformed vision-only methods across many gestures, indicating that EMG provides valuable information when visual data is unreliable or when subtle hand articulations are involved.
Hand Self-Occlusion
This occurs when a hand movement causes parts of the hand to block the view from the camera. The study found that EMG signals remain useful even under high occlusion, proving that muscle activity provides pose cues independent of what is visible in an image.

Terminology used across episodes

This episode discusses

The paper

EgoEMG: A Multimodal Egocentric Dataset with Bilateral EMG and Vision for Hand Pose Estimation · Read on arXiv

Ziheng Xi, Jiayi Yu, Yitao Wang, Yanbo Duan, Jianjiang Feng, Jie Zhou

Department of Automation, Tsinghua University

Hand pose estimation from egocentric video is challenging under adverse visual conditions, such as occlusion. Wrist-worn surface electromyography (sEMG) measures muscle activity without requiring visual access to the hand, offering a wearable complement to egocentric cameras. Evaluating their combined use requires synchronized physiological and visual recordings with continuous pose annotations, which existing EMG-based hand pose datasets do not jointly provide. We introduce EgoEMG, a multimodal dataset pairing bilateral wrist sEMG with egocentric video and motion-capture-derived hand poses, supplemented by external multi-view RGB-D recordings. EgoEMG contains 25.9 hours of recordings from 68 participants, covering 60 bare-hand gesture classes and interactions with 10 everyday objects. We establish EMG-only, vision-only, and fusion benchmarks with shared 22-dimensional pose targets and evaluations on unseen users, gestures, and objects. For EMG-only estimation, EMGFormer achieves the lowest overall MAE among the compared methods on EgoEMG and competitive results on EMG2Pose. Our residual fusion model combines ViT and EMGFormer features. Adding EMG reduces error for both single-frame and temporal vision models, with MAE decreasing from 7.39 to 7.06 for ViT-L. The largest absolute reduction occurs when evaluating interactions with an unseen object. Generalization to unseen users remains challenging. Project page: https://zhenqis123.github.io/egoemg-site/.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "EgoEMG: A Multimodal Egocentric Dataset with Bilateral EMG and Vision for Hand Pose Estimation".

Jane: Here's the gist of the paper. Surface electromyography (sEMG) records muscle activity during hand movement and can be decoded to recover detailed hand articulation The paper presents EgoEMG, a multimodal egocentric dataset for bimanual hand pose estimation This dataset synchronizes bilateral EMG and vision modalities with pose labels, establishing a foundation for future research on multimodal hand pose estimation with EMG and vision.

Tom: Dataset Composition EgoEMG includes synchronized bilateral wristband EMG with 16 total channels (8 per wrist) sampled at 2 kHz, 120 Hz IMU, egocentric wide-angle RGB video, external RGB-D video, and mocap-derived hand motion with wrist articulation angles.

Jane: The dataset covers 41 participants performing 60 gesture classes, including 30 single-hand gestures and 30 bimanual gestures, totaling more than 10 hours of recording.

Tom: Pose Target and Evaluation The prediction target is a 22-DoF joint-angle vector per hand: 20 finger angles and 2 wrist articulation angles.

Paper summary: Tom: So looking at "EgoEMG: A Multimodal Egocentric Dataset with Bilateral EMG and Vision for Hand Pose Estimation", the authors are setting a new benchmark by providing this synchronized bilateral EMG and vision data.

Jane: They’re showing that you can actually have muscle activity and visual input working together to reconstruct detailed hand articulation, which is what the paper is all about.

Lu: The main implication here is that when visual evidence gets tricky—like when your hand is self-occluded or you are moving fast—the muscle signals provide cues that don't depend on what the camera sees.

Meng: They found that the fusion benefit is systematic, improving over vision only in fifty-four of fifty-nine analyzed gesture classes, which means it’s not just a minor improvement across the board.

Tom: The largest gain they point out happens on single hand gestures where the difference in pose can be about zero point five degrees because EMG helps capture those subtle within hand articulations or asymmetric crosshand coordination.

Jane: That tells us that when you're looking at small movements, like a slight twist in a finger or how two hands are coordinating, the muscle data is really helpful.

Lu: They also found that the largest fusion benefit happens at moderate to high occlusion, where the difference between vision-only and fused results is about zero point eight degrees near occlusion, which is quite significant when you can’t see clearly.

Meng: The authors themselves flag that cross user generalization remains a central challenge for EMG based pose estimation, which means it’s still hard to get the muscle signal model to work reliably across different people.

Tom: So for someone listening who doesn't read the math, this paper is about how you can use physiological signals to give your computer a better picture of your hand when the camera isn't giving you a clear view.

Conclusion: Tom: So we’ve been talking about EgoEMG, which is this new dataset that brings together bilateral electromyography and vision for hand pose estimation.

Jane: Yeah, they're setting a new benchmark here with this multimodal egocentric data set including forty-one participants performing sixty gesture classes.

Lu: What’s interesting is the mix of data they put in there: synchronized bilateral wristband electromyography with sixteen channels, along with one hundred twenty hertz inertial measurement units, and both egocentric wide-angle rgb video and external rgb-d video.

Meng: It’s a lot of input for any model to process when you’re trying to figure out exactly where your hand is in three dimensions.

Lalam: The target pose they are aiming for is a twenty two degrees of freedom vector per hand, which means predicting twenty finger angles and two wrist articulation angles.

Tom: So the researchers tested three different ways of using this data: just electromyography to predict the pose, just vision to predict the pose, and then fusing them together.

Jane: And they found that fusion actually works better than vision only in fifty four out of fifty nine gesture classes, which is a systematic gain.

Lu: The largest gain happens on single hand gestures where the difference in pose can be about zero point five degrees because the electromyography helps capture those subtle within hand articulations or asymmetric crosshand coordination.

Meng: But they also found that when the camera sees things poorly, like at moderate to high occlusion, the fusion benefit is about zero point eight degrees near occlusion, which is substantial when you can’t see clearly.

Lalam: The authors themselves admit that cross user generalization remains a central challenge for electromyography based pose estimation.

Tom: It means that even with all this data, getting a model to work reliably across different people is still the main hurdle they’re facing right now.

Jane: So this paper gives us a solid testbed for figuring out exactly when those physiological signals add information beyond just what we see visually, especially when visual evidence becomes unreliable under fast motion or self-occlusion.

Lu: The implication here is that if you want robust hand pose estimation, you might need to look at how these different modalities work together under those tricky conditions.

More episodes

← Home