EgoEMG: A Multimodal Egocentric Dataset with Bilateral EMG and Vision for Hand Pose Estimation

arXiv:2605.05712 · cs.CV · Submitted 2026-05-07 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "EgoEMG: A Multimodal Egocentric Dataset with Bilateral EMG and Vision for Hand Pose Estimation".

Jane: Here's the gist of the paper. Surface electromyography (sEMG) records muscle activity during hand movement and can be decoded to recover detailed hand articulation The paper presents EgoEMG, a multimodal egocentric dataset for bimanual hand pose estimation This dataset synchronizes bilateral EMG and vision modalities with pose labels, establishing a foundation for future research on multimodal hand pose estimation with EMG and vision.

Tom: Dataset Composition EgoEMG includes synchronized bilateral wristband EMG with 16 total channels (8 per wrist) sampled at 2 kHz, 120 Hz IMU, egocentric wide-angle RGB video, external RGB-D video, and mocap-derived hand motion with wrist articulation angles.

Jane: The dataset covers 41 participants performing 60 gesture classes, including 30 single-hand gestures and 30 bimanual gestures, totaling more than 10 hours of recording.

Tom: Pose Target and Evaluation The prediction target is a 22-DoF joint-angle vector per hand: 20 finger angles and 2 wrist articulation angles.

Paper summary: Tom: So looking at "EgoEMG: A Multimodal Egocentric Dataset with Bilateral EMG and Vision for Hand Pose Estimation", the authors are setting a new benchmark by providing this synchronized bilateral EMG and vision data.

Jane: They’re showing that you can actually have muscle activity and visual input working together to reconstruct detailed hand articulation, which is what the paper is all about.

Lu: The main implication here is that when visual evidence gets tricky—like when your hand is self-occluded or you are moving fast—the muscle signals provide cues that don't depend on what the camera sees.

Meng: They found that the fusion benefit is systematic, improving over vision only in fifty-four of fifty-nine analyzed gesture classes, which means it’s not just a minor improvement across the board.

Tom: The largest gain they point out happens on single hand gestures where the difference in pose can be about zero point five degrees because EMG helps capture those subtle within hand articulations or asymmetric crosshand coordination.

Jane: That tells us that when you're looking at small movements, like a slight twist in a finger or how two hands are coordinating, the muscle data is really helpful.

Lu: They also found that the largest fusion benefit happens at moderate to high occlusion, where the difference between vision-only and fused results is about zero point eight degrees near occlusion, which is quite significant when you can’t see clearly.

Meng: The authors themselves flag that cross user generalization remains a central challenge for EMG based pose estimation, which means it’s still hard to get the muscle signal model to work reliably across different people.

Tom: So for someone listening who doesn't read the math, this paper is about how you can use physiological signals to give your computer a better picture of your hand when the camera isn't giving you a clear view.

Conclusion: Tom: So we’ve been talking about EgoEMG, which is this new dataset that brings together bilateral electromyography and vision for hand pose estimation.

Jane: Yeah, they're setting a new benchmark here with this multimodal egocentric data set including forty-one participants performing sixty gesture classes.

Lu: What’s interesting is the mix of data they put in there: synchronized bilateral wristband electromyography with sixteen channels, along with one hundred twenty hertz inertial measurement units, and both egocentric wide-angle rgb video and external rgb-d video.

Meng: It’s a lot of input for any model to process when you’re trying to figure out exactly where your hand is in three dimensions.

Lalam: The target pose they are aiming for is a twenty two degrees of freedom vector per hand, which means predicting twenty finger angles and two wrist articulation angles.

Tom: So the researchers tested three different ways of using this data: just electromyography to predict the pose, just vision to predict the pose, and then fusing them together.

Jane: And they found that fusion actually works better than vision only in fifty four out of fifty nine gesture classes, which is a systematic gain.

Lu: The largest gain happens on single hand gestures where the difference in pose can be about zero point five degrees because the electromyography helps capture those subtle within hand articulations or asymmetric crosshand coordination.

Meng: But they also found that when the camera sees things poorly, like at moderate to high occlusion, the fusion benefit is about zero point eight degrees near occlusion, which is substantial when you can’t see clearly.

Lalam: The authors themselves admit that cross user generalization remains a central challenge for electromyography based pose estimation.

Tom: It means that even with all this data, getting a model to work reliably across different people is still the main hurdle they’re facing right now.

Jane: So this paper gives us a solid testbed for figuring out exactly when those physiological signals add information beyond just what we see visually, especially when visual evidence becomes unreliable under fast motion or self-occlusion.

Lu: The implication here is that if you want robust hand pose estimation, you might need to look at how these different modalities work together under those tricky conditions.

Ziheng Xi, Jiayi Yu, Yitao Wang, Yanbo Duan, Jianjiang Feng, Jie Zhou

Department of Automation, Tsinghua University

cs.CV

Submitted: 2026-05-07

Updated: 2026-11-16

Comments: 21 pages, 13 figures, 20 tables, including supplementary material

Code: https://github.com/zhenqis123/EgoEMG

License: http://creativecommons.org/licenses/by-nc-sa/4.0/

Importance score: 98/100

The gist: Surface electromyography (sEMG) records muscle activity during hand movement and can be decoded to recover detailed hand articulation The paper presents EgoEMG…

Key concepts

EgoEMG Dataset
This is a new dataset that synchronizes multiple data sources: 16 channels of bilateral wristband EMG, IMU data, wide-angle RGB video, external RGB-D video, and motion capture data. It includes labels for 60 different hand gestures performed by 41 participants over more than ten hours.
Bimanual Hand Pose
This refers to estimating the precise 3D position and orientation of both hands simultaneously during movement. The target is a detailed vector describing every joint angle in both hands, including finger angles and wrist articulation angles.
Modality Complementarity
This analyzes how different data types (like EMG and vision) work together. The research found that fusion consistently outperformed vision-only methods across many gestures, indicating that EMG provides valuable information when visual data is unreliable or when subtle hand articulations are involved.
Hand Self-Occlusion
This occurs when a hand movement causes parts of the hand to block the view from the camera. The study found that EMG signals remain useful even under high occlusion, proving that muscle activity provides pose cues independent of what is visible in an image.

Terminology

Summary

Surface electromyography (sEMG) records muscle activity during hand movement and can be decoded to recover detailed hand articulation The paper presents EgoEMG, a multimodal egocentric dataset for bimanual hand pose estimation This dataset synchronizes bilateral EMG and vision modalities with pose labels, establishing a foundation for future research on multimodal hand pose estimation with EMG and vision. p. 0

Where the paper says it

cn Abstract Surface electromyography (sEMG) records muscle activity during hand movement and can be decoded to recover detailed hand articulation. EMG and egocentric vision are complementary for hand sensing: EMG captures fine-grained finger articulation even under occlusion and poor lighting, while vision provides global hand configuration.

Dataset Composition

EgoEMG includes synchronized bilateral wristband EMG with 16 total channels (8 per wrist) sampled at 2 kHz, 120 Hz IMU, egocentric wide-angle RGB video, external RGB-D video, and mocap-derived hand motion with wrist articulation angles. The dataset covers 41 participants performing 60 gesture classes, including 30 single-hand gestures and 30 bimanual gestures, totaling more than 10 hours of recording. p. 0

Where the paper says it

EgoEMG includes bilateral wristband EMG with 16 total channels (8 per wrist) sampled at 2 kHz, 120 Hz IMU, egocentric wide-angle RGB video, external RGB-D video, and mocap-derived hand motion with wrist articulation angles. The dataset covers 41 participants performing 60 gesture classes, including 30 single-hand gestures and 30 bimanual gestures, totaling more than 10 hours of recording.

Pose Target and Evaluation

The prediction target is a 22-DoF joint-angle vector per hand: 20 finger angles and 2 wrist articulation angles. The three tasks evaluated are EMG-to-pose, vision-to-pose, and EMG+vision fusion under a shared joint-angle prediction target and common generalization split axes (cross-gesture, cross-user, and combined). p. 0

Where the paper says it

The dataset covers 41 participants performing 60 gesture classes, including 30 single-hand gestures and 30 bimanual gestures, totaling more than 10 hours of recording. We also introduce a benchmark with three tasks—EMG-to-pose, visionto-pose, and EMG+vision fusion—under a shared joint-angle prediction target and common generalization split axes (cross-gesture, cross-user, and combined).

Model Baselines and Performance

EMGFormer variants outperform the prior TDS+LSTM baseline by up to 22% on EMG-to-pose, with EMGFormer-S achieving the best efficiency trade-off. Vision backbones like ResNet and ViT regressors outperform generic ViT regressors at lower parameter counts, while the hand-specialized WiLoR baseline achieves the strongest standalone visual accuracy. p. 1

Where the paper says it

On EMG-to-pose, EMGFormer variants outperform the prior TDS+LSTM baseline by up to 22%, with EMGFormer-S achieving the best efficiency trade-off (42% fewer parameters) and EMGFormer-L the best absolute accuracy. On vision-to-pose, ResNet backbones outperform generic ViT regressors at lower parameter counts, while the hand-specialized WiLoR baseline achieves the strongest standalone visual accuracy.

A residual EMG+vision fusion architecture consistently improves over matched lightweight generic vision-only baselines, suggesting EMG provides complementary pose information beyond single-frame visual features. p. 1
Where the paper says it

On vision-to-pose, ResNet backbones outperform generic ViT regressors at lower parameter counts, while the hand-specialized WiLoR baseline achieves the strongest standalone visual accuracy. A residual EMG+vision fusion architecture consistently improves over matched lightweight generic vision-only baselines, suggesting EMG provides complementary pose information beyond single-frame visual features; whether this complementarity extends to stronger specialized vision models remains an open question for future work.

Modality Complementarity Analysis

One pattern emerging is that the fusion gain is systematic, as fusion improves over vision-only on 54 of 59 analyzed gesture classes. The largest family-level gain occurs on single-hand gestures (∆ = +0. p. 0

Where the paper says it

EgoEMG includes bilateral wristband EMG with 16 total channels (8 per wrist) sampled at 2 kHz, 120 Hz IMU, egocentric wide-angle RGB video, external RGB-D video, and mocap-derived hand motion with wrist articulation angles. The dataset covers 41 participants performing 60 gesture classes, including 30 single-hand gestures and 30 bimanual gestures, totaling more than 10 hours of recording.

50◦), suggesting that EMG is most useful when the prediction depends on subtle within-hand articulation or asymmetric crosshand coordination. p. 3
Where the paper says it

Figure 3(a) shows the distribution across 41 participants, demonstrating substantial inter-subject hand-shape variation. The 60 gesture classes include 30 single-hand gestures and 30 bimanual interaction gestures, covering both symmetric coordination and asymmetric manipulation.

Fusion Benefit vs. p. 7

Where the paper says it

Scaling reduces error within both generic backbone families, while cross-user and both splits remain harder than the gesture split across all vision-only models. The relative underperformance of generic ViT regressors may reflect the scale of the available supervised training data, as transformer-based vision models often benefit from larger datasets.

Hand Self-Occlusion

The largest fusion benefit occurs at moderateto-high occlusion (∆ ≈ 0.8◦ near occlusion ≈ 0. p. 0

Where the paper says it

cn Abstract Surface electromyography (sEMG) records muscle activity during hand movement and can be decoded to recover detailed hand articulation. EMG and egocentric vision are complementary for hand sensing: EMG captures fine-grained finger articulation even under occlusion and poor lighting, while vision provides global hand configuration.

72), where visual evidence is substantially degraded. p. 4
Where the paper says it

Unless otherwise specified, all errors are reported on the held-out test split. EMG evaluation uses the full predicted trajectory over each EMG window, whereas the vision-only and fusion models are evaluated on the window center frame, where visual evidence is available; cross-modality comparisons should therefore be interpreted with this difference in supervision granularity in mind.

This indicates that EMG retains pose cues independent of image appearance when visual evidence is ambiguous. p. 8
Where the paper says it

6) shows a positive correlation between hand self-occlusion and fusion gain. Figure 7 illustrates this complementarity: when visual evidence is ambiguous, EMG retains pose cues independent of image appearance, and the fused model recovers articulation closer to the ground truth.

Conclusion and Future Work

EgoEMG provides a multimodal benchmark for hand pose estimation with synchronized bilateral EMG, IMU, egocentric RGB, external RGB-D, and hand-pose annotations including wrist articulation. Cross-user generalization remains the central challenge for EMG-based pose estimation. p. 8

Where the paper says it

Our experiments highlight three findings. First, cross-user generalization remains the central challenge for EMG-based pose estimation: errors increase more on unseen participants than on unseen gestures.

Future work should test whether EMG also improves specialized vision models such as WiLoR. p. 1
Where the paper says it

On vision-to-pose, ResNet backbones outperform generic ViT regressors at lower parameter counts, while the hand-specialized WiLoR baseline achieves the strongest standalone visual accuracy. A residual EMG+vision fusion architecture consistently improves over matched lightweight generic vision-only baselines, suggesting EMG provides complementary pose information beyond single-frame visual features; whether this complementarity extends to stronger specialized vision models remains an open question for future work.

The paper provides a testbed for studying when physiological signals add information beyond visual observations, especially under unreliable visual evidence such as self-occlusion and fast motion. p. 9
Where the paper says it

expressive architectures, including cross-attention, uncertainty-aware fusion, and explicit temporal alignment, may better exploit the complementary strengths of EMG and vision. More broadly, EgoEMG provides a testbed for studying when physiological signals add information beyond visual observations, especially under unreliable visual evidence such as self-occlusion and fast motion.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Multimodal Hand Pose Estimation: The improved system can perform hand pose estimation by fusing surface electromyography (EMG) signals with egocentric vision to recover detailed finger articulation even under occlusion and poor lighting.

  2. Bimanual Gesture Recognition: The system can accurately recognize 60 gesture classes, including 30 single-hand gestures and 30 bimanual gestures, leveraging the synchronized bilateral EMG data for capturing natural coordination patterns.

  3. Cross-Modal Robustness: The system demonstrates improved performance when fused with vision-only models, as fusion consistently improves over matched lightweight generic vision-only baselines, suggesting it is particularly effective when visual evidence is ambiguous due to self-occlusion or fast motion.

  4. Pose Regression via EMG: The system can predict 22 degrees of freedom (DoF) joint angles from raw EMG windows using the EMGFormer architecture, which utilizes a TDS-style temporal convolutional frontend and a Transformer decoder to model long-range temporal dependencies in muscle activity.

  5. Shape and Pose Reconstruction: By utilizing the markers2mano pipeline, the system can reconstruct 3D hand meshes from optical motion capture data into MANO parameters, achieving a mean marker-to-mesh alignment error of 4.3 mm on EgoEMG data.

  6. Modality Complementarity Analysis: The system can be used to characterize where EMG provides complementary information beyond vision by showing that fusion improves over vision-only on 54 of 59 analyzed gesture classes, indicating EMG is most useful for subtle within-hand articulation and asymmetric crosshand coordination.

Abstract

Hand pose estimation from egocentric video is challenging under adverse visual conditions, such as occlusion. Wrist-worn surface electromyography (sEMG) measures muscle activity without requiring visual access to the hand, offering a wearable complement to egocentric cameras. Evaluating their combined use requires synchronized physiological and visual recordings with continuous pose annotations, which existing EMG-based hand pose datasets do not jointly provide. We introduce EgoEMG, a multimodal dataset pairing bilateral wrist sEMG with egocentric video and motion-capture-derived hand poses, supplemented by external multi-view RGB-D recordings. EgoEMG contains 25.9 hours of recordings from 68 participants, covering 60 bare-hand gesture classes and interactions with 10 everyday objects. We establish EMG-only, vision-only, and fusion benchmarks with shared 22-dimensional pose targets and evaluations on unseen users, gestures, and objects. For EMG-only estimation, EMGFormer achieves the lowest overall MAE among the compared methods on EgoEMG and competitive results on EMG2Pose. Our residual fusion model combines ViT and EMGFormer features. Adding EMG reduces error for both single-frame and temporal vision models, with MAE decreasing from 7.39 to 7.06 for ViT-L. The largest absolute reduction occurs when evaluating interactions with an unseen object. Generalization to unseen users remains challenging. Project page: https://zhenqis123.github.io/egoemg-site/.

Sources

Related papers