MyoMechanix: Biomechanically-Grounded Compositional Skilled Activity Understanding and Coaching
cs.CV, cs.AI, cs.ET, cs.HC, cs.LG
Submitted: 2026-08-26
Updated: 2026-08-26
License: http://creativecommons.org/licenses/by-nc-sa/4.0/
The gist: Existing action quality assessment (AQA) datasets and methods rely primarily on visual inputs such as RGB and pose, overlooking physiological dynamics such as muscle mechanics and often modeling
Terminology
Abstract
Existing action quality assessment (AQA) datasets and methods rely primarily on visual inputs such as RGB and pose, overlooking physiological dynamics such as muscle mechanics and often modeling actions as monolithic patterns. These limitations hinder fine-grained, biomechanically grounded feedback. We introduce MyoMechanix, a multimodal ecosystem for weight-loaded actions that aligns motion with muscle activity. Expert-annotated, it contains 7,500+ samples of 20 actions from 38 subjects, with synchronized multiview RGB video, 3D pose, sEMG, and additional physiological signals, forming the largest multimodal AQA benchmark to date. We further construct the Fitness Knowledge Graph (FKG), which organizes expert annotations into structured relationships among actions, phases, key steps, errors, and corrective feedback, enabling compositional scoring and interpretable assessment. Building on these representations, we develop CUBIST (Compositional Ontological Reasoning Engine), which performs decomposition-analysis-recomposition for fine-grained error attribution and feedback generation. We also establish MyoMechanix-AQA, MyoMechanix-VideoQA, and a novel MyoMechanix-Video2EMG task. Experiments show that multimodal sensing and structured representations improve performance, interpretability, and error attribution, with CUBIST achieving state-of-the-art results; VideoQA enhances language-grounded action understanding; and Video2EMG suggests video-based alternatives to costly EMG sensing. MyoMechanix advances skilled activity understanding toward biomechanically grounded, multimodal, and compositional reasoning for Physical AI applications in fitness, rehabilitation, healthcare, and machine learning. Project page: https://haoyin116.github.io/MyoMechanix/
Sources
- FitAQA: A Benchmark of Fitness Action Quality Assessment for Multimodal Large Language Models
- Learning to Visually Connect Actions and their Effects
- Uncertainty-Driven Action Quality Assessment
- SkillSight: Efficient First-Person Skill Assessment with Gaze
- Video Action Differencing
- RICA2: Rubric-Informed, Calibrated Assessment of Actions
- TVQA+: Spatio-Temporal Grounding for Video Question Answering
- Explainable Action Form Assessment by Exploiting Multimodal Chain-of-Thoughts Reasoning
- MiniCPM-V: A GPT-4V Level MLLM on Your Phone
- InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models
- Qwen2.5-VL Technical Report
- LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Related papers
- Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
- AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
- Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
- MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
- TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
- A Survey on Efficient Vision-Language-Action Models