EgoEMG: A Multimodal Egocentric Dataset with Bilateral EMG and Vision for Hand Pose Estimation
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "EgoEMG: A Multimodal Egocentric Dataset with Bilateral EMG and Vision for Hand Pose Estimation".
Jane: Here's the gist of the paper. Surface electromyography (sEMG) records muscle activity during hand movement and can be decoded to recover detailed hand articulation The paper presents EgoEMG, a multimodal egocentric dataset for bimanual hand pose estimation This dataset synchronizes bilateral EMG and vision modalities with pose labels, establishing a foundation for future research on multimodal hand pose estimation with EMG and vision.
Tom: Dataset Composition EgoEMG includes synchronized bilateral wristband EMG with 16 total channels (8 per wrist) sampled at 2 kHz, 120 Hz IMU, egocentric wide-angle RGB video, external RGB-D video, and mocap-derived hand motion with wrist articulation angles.
Jane: The dataset covers 41 participants performing 60 gesture classes, including 30 single-hand gestures and 30 bimanual gestures, totaling more than 10 hours of recording.
Tom: Pose Target and Evaluation The prediction target is a 22-DoF joint-angle vector per hand: 20 finger angles and 2 wrist articulation angles.
Paper summary: Tom: So looking at "EgoEMG: A Multimodal Egocentric Dataset with Bilateral EMG and Vision for Hand Pose Estimation", the authors are setting a new benchmark by providing this synchronized bilateral EMG and vision data.
Jane: They’re showing that you can actually have muscle activity and visual input working together to reconstruct detailed hand articulation, which is what the paper is all about.
Lu: The main implication here is that when visual evidence gets tricky—like when your hand is self-occluded or you are moving fast—the muscle signals provide cues that don't depend on what the camera sees.
Meng: They found that the fusion benefit is systematic, improving over vision only in fifty-four of fifty-nine analyzed gesture classes, which means it’s not just a minor improvement across the board.
Tom: The largest gain they point out happens on single hand gestures where the difference in pose can be about zero point five degrees because EMG helps capture those subtle within hand articulations or asymmetric crosshand coordination.
Jane: That tells us that when you're looking at small movements, like a slight twist in a finger or how two hands are coordinating, the muscle data is really helpful.
Lu: They also found that the largest fusion benefit happens at moderate to high occlusion, where the difference between vision-only and fused results is about zero point eight degrees near occlusion, which is quite significant when you can’t see clearly.
Meng: The authors themselves flag that cross user generalization remains a central challenge for EMG based pose estimation, which means it’s still hard to get the muscle signal model to work reliably across different people.
Tom: So for someone listening who doesn't read the math, this paper is about how you can use physiological signals to give your computer a better picture of your hand when the camera isn't giving you a clear view.
Conclusion: Tom: So we’ve been talking about EgoEMG, which is this new dataset that brings together bilateral electromyography and vision for hand pose estimation.
Jane: Yeah, they're setting a new benchmark here with this multimodal egocentric data set including forty-one participants performing sixty gesture classes.
Lu: What’s interesting is the mix of data they put in there: synchronized bilateral wristband electromyography with sixteen channels, along with one hundred twenty hertz inertial measurement units, and both egocentric wide-angle rgb video and external rgb-d video.
Meng: It’s a lot of input for any model to process when you’re trying to figure out exactly where your hand is in three dimensions.
Lalam: The target pose they are aiming for is a twenty two degrees of freedom vector per hand, which means predicting twenty finger angles and two wrist articulation angles.
Tom: So the researchers tested three different ways of using this data: just electromyography to predict the pose, just vision to predict the pose, and then fusing them together.
Jane: And they found that fusion actually works better than vision only in fifty four out of fifty nine gesture classes, which is a systematic gain.
Lu: The largest gain happens on single hand gestures where the difference in pose can be about zero point five degrees because the electromyography helps capture those subtle within hand articulations or asymmetric crosshand coordination.
Meng: But they also found that when the camera sees things poorly, like at moderate to high occlusion, the fusion benefit is about zero point eight degrees near occlusion, which is substantial when you can’t see clearly.
Lalam: The authors themselves admit that cross user generalization remains a central challenge for electromyography based pose estimation.
Tom: It means that even with all this data, getting a model to work reliably across different people is still the main hurdle they’re facing right now.
Jane: So this paper gives us a solid testbed for figuring out exactly when those physiological signals add information beyond just what we see visually, especially when visual evidence becomes unreliable under fast motion or self-occlusion.
Lu: The implication here is that if you want robust hand pose estimation, you might need to look at how these different modalities work together under those tricky conditions.
Ziheng Xi, Jiayi Yu, Yitao Wang, Yanbo Duan, Jianjiang Feng, Jie Zhou
Department of Automation, Tsinghua University
cs.CV
Submitted: 2026-05-07
Updated: 2026-11-16
Comments: 21 pages, 13 figures, 20 tables, including supplementary material
Code: https://github.com/zhenqis123/EgoEMG
License: http://creativecommons.org/licenses/by-nc-sa/4.0/
Importance score: 98/100
The gist: Surface electromyography (sEMG) records muscle activity during hand movement and can be decoded to recover detailed hand articulation The paper presents EgoEMG…
Key concepts
- EgoEMG Dataset
- This is a new dataset that synchronizes multiple data sources: 16 channels of bilateral wristband EMG, IMU data, wide-angle RGB video, external RGB-D video, and motion capture data. It includes labels for 60 different hand gestures performed by 41 participants over more than ten hours.
- Bimanual Hand Pose
- This refers to estimating the precise 3D position and orientation of both hands simultaneously during movement. The target is a detailed vector describing every joint angle in both hands, including finger angles and wrist articulation angles.
- Modality Complementarity
- This analyzes how different data types (like EMG and vision) work together. The research found that fusion consistently outperformed vision-only methods across many gestures, indicating that EMG provides valuable information when visual data is unreliable or when subtle hand articulations are involved.
- Hand Self-Occlusion
- This occurs when a hand movement causes parts of the hand to block the view from the camera. The study found that EMG signals remain useful even under high occlusion, proving that muscle activity provides pose cues independent of what is visible in an image.
Terminology
Summary
Surface electromyography (sEMG) records muscle activity during hand movement and can be decoded to recover detailed hand articulation The paper presents EgoEMG, a multimodal egocentric dataset for bimanual hand pose estimation This dataset synchronizes bilateral EMG and vision modalities with pose labels, establishing a foundation for future research on multimodal hand pose estimation with EMG and vision. p. 0 cn
Abstract
Surface electromyography (sEMG) records muscle activity during hand movement
and can be decoded to recover detailed hand articulation. EMG and egocentric
vision are complementary for hand sensing: EMG captures fine-grained finger
articulation even under occlusion and poor lighting, while vision provides global
hand configuration.Where the paper says it
Dataset Composition
EgoEMG includes synchronized bilateral wristband EMG with 16 total channels (8 per wrist) sampled at 2 kHz, 120 Hz IMU, egocentric wide-angle RGB video, external RGB-D video, and mocap-derived hand motion with wrist articulation angles. The dataset covers 41 participants performing 60 gesture classes, including 30 single-hand gestures and 30 bimanual gestures, totaling more than 10 hours of recording. p. 0 EgoEMG includes bilateral wristband EMG with 16 total channels
(8 per wrist) sampled at 2 kHz, 120 Hz IMU, egocentric wide-angle RGB video,
external RGB-D video, and mocap-derived hand motion with wrist articulation
angles. The dataset covers 41 participants performing 60 gesture classes, including
30 single-hand gestures and 30 bimanual gestures, totaling more than 10 hours of
recording.Where the paper says it
Pose Target and Evaluation
The prediction target is a 22-DoF joint-angle vector per hand: 20 finger angles and 2 wrist articulation angles. The three tasks evaluated are EMG-to-pose, vision-to-pose, and EMG+vision fusion under a shared joint-angle prediction target and common generalization split axes (cross-gesture, cross-user, and combined). p. 0 The dataset covers 41 participants performing 60 gesture classes, including
30 single-hand gestures and 30 bimanual gestures, totaling more than 10 hours of
recording. We also introduce a benchmark with three tasks—EMG-to-pose, visionto-pose, and EMG+vision fusion—under a shared joint-angle prediction target
and common generalization split axes (cross-gesture, cross-user, and combined).Where the paper says it
Model Baselines and Performance
EMGFormer variants outperform the prior TDS+LSTM baseline by up to 22% on EMG-to-pose, with EMGFormer-S achieving the best efficiency trade-off. Vision backbones like ResNet and ViT regressors outperform generic ViT regressors at lower parameter counts, while the hand-specialized WiLoR baseline achieves the strongest standalone visual accuracy. p. 1 On EMG-to-pose, EMGFormer variants outperform the prior TDS+LSTM baseline by
up to 22%, with EMGFormer-S achieving the best efficiency trade-off (42% fewer parameters) and
EMGFormer-L the best absolute accuracy. On vision-to-pose, ResNet backbones outperform generic
ViT regressors at lower parameter counts, while the hand-specialized WiLoR baseline achieves
the strongest standalone visual accuracy. On vision-to-pose, ResNet backbones outperform generic
ViT regressors at lower parameter counts, while the hand-specialized WiLoR baseline achieves
the strongest standalone visual accuracy. A residual EMG+vision fusion architecture consistently
improves over matched lightweight generic vision-only baselines, suggesting EMG provides complementary pose information beyond single-frame visual features; whether this complementarity extends
to stronger specialized vision models remains an open question for future work.Where the paper says it
Where the paper says it
Modality Complementarity Analysis
One pattern emerging is that the fusion gain is systematic, as fusion improves over vision-only on 54 of 59 analyzed gesture classes. The largest family-level gain occurs on single-hand gestures (∆ = +0. p. 0 EgoEMG includes bilateral wristband EMG with 16 total channels
(8 per wrist) sampled at 2 kHz, 120 Hz IMU, egocentric wide-angle RGB video,
external RGB-D video, and mocap-derived hand motion with wrist articulation
angles. The dataset covers 41 participants performing 60 gesture classes, including
30 single-hand gestures and 30 bimanual gestures, totaling more than 10 hours of
recording. Figure 3(a) shows the distribution
across 41 participants, demonstrating substantial inter-subject hand-shape variation. The 60 gesture
classes include 30 single-hand gestures and 30 bimanual interaction gestures, covering both symmetric
coordination and asymmetric manipulation.Where the paper says it
Where the paper says it
Fusion Benefit vs. p. 7 Scaling reduces error within both
generic backbone families, while cross-user and both splits remain harder than the gesture split across
all vision-only models. The relative underperformance of generic ViT regressors may reflect the scale
of the available supervised training data, as transformer-based vision models often benefit from larger
datasets.Where the paper says it
The largest fusion benefit occurs at moderateto-high occlusion (∆ ≈ 0.8◦ near occlusion ≈ 0. p. 0 cn
Abstract
Surface electromyography (sEMG) records muscle activity during hand movement
and can be decoded to recover detailed hand articulation. EMG and egocentric
vision are complementary for hand sensing: EMG captures fine-grained finger
articulation even under occlusion and poor lighting, while vision provides global
hand configuration. Unless
otherwise specified, all errors are reported on the held-out test split. EMG evaluation uses the full
predicted trajectory over each EMG window, whereas the vision-only and fusion models are evaluated
on the window center frame, where visual evidence is available; cross-modality comparisons should
therefore be interpreted with this difference in supervision granularity in mind. 6) shows a positive correlation between
hand self-occlusion and fusion gain. Figure 7 illustrates this complementarity: when visual evidence
is ambiguous, EMG retains pose cues independent of image appearance, and the fused model recovers
articulation closer to the ground truth.Where the paper says it
Where the paper says it
Where the paper says it
Conclusion and Future Work
EgoEMG provides a multimodal benchmark for hand pose estimation with synchronized bilateral EMG, IMU, egocentric RGB, external RGB-D, and hand-pose annotations including wrist articulation. Cross-user generalization remains the central challenge for EMG-based pose estimation. p. 8 Our experiments highlight three findings. First, cross-user generalization remains the central challenge
for EMG-based pose estimation: errors increase more on unseen participants than on unseen gestures. On vision-to-pose, ResNet backbones outperform generic
ViT regressors at lower parameter counts, while the hand-specialized WiLoR baseline achieves
the strongest standalone visual accuracy. A residual EMG+vision fusion architecture consistently
improves over matched lightweight generic vision-only baselines, suggesting EMG provides complementary pose information beyond single-frame visual features; whether this complementarity extends
to stronger specialized vision models remains an open question for future work. expressive architectures, including cross-attention, uncertainty-aware fusion, and explicit temporal
alignment, may better exploit the complementary strengths of EMG and vision. More broadly, EgoEMG provides a testbed for studying when physiological signals add information
beyond visual observations, especially under unreliable visual evidence such as self-occlusion and
fast motion.Where the paper says it
Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Multimodal Hand Pose Estimation: The improved system can perform hand pose estimation by fusing surface electromyography (EMG) signals with egocentric vision to recover detailed finger articulation even under occlusion and poor lighting.
-
Bimanual Gesture Recognition: The system can accurately recognize 60 gesture classes, including 30 single-hand gestures and 30 bimanual gestures, leveraging the synchronized bilateral EMG data for capturing natural coordination patterns.
-
Cross-Modal Robustness: The system demonstrates improved performance when fused with vision-only models, as fusion
consistently improves over matched lightweight generic vision-only baselines,
suggesting it is particularly effective when visual evidence is ambiguous due to self-occlusion or fast motion. -
Pose Regression via EMG: The system can predict 22 degrees of freedom (DoF) joint angles from raw EMG windows using the EMGFormer architecture, which utilizes a TDS-style temporal convolutional frontend and a Transformer decoder to model long-range temporal dependencies in muscle activity.
-
Shape and Pose Reconstruction: By utilizing the markers2mano pipeline, the system can reconstruct 3D hand meshes from optical motion capture data into MANO parameters, achieving a mean marker-to-mesh alignment error of 4.3 mm on EgoEMG data.
-
Modality Complementarity Analysis: The system can be used to characterize where EMG provides complementary information beyond vision by showing that
fusion improves over vision-only on 54 of 59 analyzed gesture classes,
indicating EMG is most useful for subtle within-hand articulation and asymmetric crosshand coordination.
Abstract
Hand pose estimation from egocentric video is challenging under adverse visual conditions, such as occlusion. Wrist-worn surface electromyography (sEMG) measures muscle activity without requiring visual access to the hand, offering a wearable complement to egocentric cameras. Evaluating their combined use requires synchronized physiological and visual recordings with continuous pose annotations, which existing EMG-based hand pose datasets do not jointly provide. We introduce EgoEMG, a multimodal dataset pairing bilateral wrist sEMG with egocentric video and motion-capture-derived hand poses, supplemented by external multi-view RGB-D recordings. EgoEMG contains 25.9 hours of recordings from 68 participants, covering 60 bare-hand gesture classes and interactions with 10 everyday objects. We establish EMG-only, vision-only, and fusion benchmarks with shared 22-dimensional pose targets and evaluations on unseen users, gestures, and objects. For EMG-only estimation, EMGFormer achieves the lowest overall MAE among the compared methods on EgoEMG and competitive results on EMG2Pose. Our residual fusion model combines ViT and EMGFormer features. Adding EMG reduces error for both single-frame and temporal vision models, with MAE decreasing from 7.39 to 7.06 for ViT-L. The largest absolute reduction occurs when evaluating interactions with an unseen object. Generalization to unseen users remains challenging. Project page: https://zhenqis123.github.io/egoemg-site/.
Sources
- CPEP: Contrastive Pose-EMG Pre-training Enhances Gesture Generalization on EMG Signals
- TinyMyo: a Tiny Foundation Model for Flexible EMG Signal Processing at the Edge
- UmeTrack: Unified multi-view end-to-end hand tracking for VR
- GigaHands: A Massive Annotated Dataset of Bimanual Hand Activities
- Masked Autoencoder with Swin Transformer Network for Mitigating Electrode Shift in HD-EMG-based Gesture Recognition
- Self-Supervised Learning via VICReg Enables Training of EMG Pattern Recognition Using Continuous Data with Unclear Labels
- EMGBench: Benchmarking Out-of-Distribution Generalization and Adaptation for Electromyography
- DexEMG: Towards Dexterous Teleoperation System via EMG2Pose Generalization
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs