MyoSem: Aligning Electromyography with Natural-Language Action Semantics

arXiv:2606.00174 · cs.CV, cs.AI · Submitted 2026-05-29 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom, and with me are Jane, Lu, senior AI researcher at Tsinghua, Meng, lead engineer at a mysterious AI startup and Lalam, the in-house Large Language Model.

Jane: Today's paper: "MyoSem: Aligning Electromyography with Natural-Language Action Semantics".

Tom: The gist The framework MyoSem presents an EMG–action semantic alignment framework that maps low-level EMG signals into a shared semantic space constructed from multi-view action descriptions,

Jane: First, who's behind it and why it matters.

Paper summary: Tom: So we’re diving deeper into MyoSem: Aligning Electromyography with Natural-Language Action Semantics. We’ve established that the main thesis is creating an EMG–action semantic alignment framework that maps low-level EMG signals into a shared semantic space constructed from multi-view action descriptions.

Jane: They claim this framework combines three key things to achieve this: multi-view action-semantic construction, activation-aware EMG encoding, and then semantic query alignment for bidirectional retrieval between the EMG signals and text descriptions.

Lu: The paper is focused on solving an EMG–action semantics grounding problem—given the raw class-level action descriptions, they want to learn a shared space that supports both retrieval and evaluation on new users and held-out classes whose descriptions are only available at inference time.

Meng: This addresses the text challenge where EMG datasets often just provide short labels that can’t express rich action information, but more importantly, it solves the problem where different expressions of the same action might get far apart in semantic space.

Lalam: They tackle that by defining four text views for each action class—T0 being the original short label, T1 being a pose-sequence-derived process description, T2 canonical action description, and T3 natural language paraphrases—and they encode all of these with a shared MacBERT text encoder.

Tom: That’s an interesting setup because they are aligning the canonical descriptions with the process descriptions while also anchoring everything to the short labels and ensuring consistency across paraphrases. They compute the class-level action-semantic representation Pc using a specific formula involving Vc and X v e(v)c!

Jane: Then they tackle the EMG challenge by training an activation-aware EMG encoder that preserves local activation intensity, energy changes, and contraction dynamics by injecting descriptors like rmst for intensity, energyt for the envelope, and vart for contraction state.

Lu: Those specific descriptors are injected into the patch tokens through gated residual connections before feeding them into a Transformer encoder to give temporal context to the sequence H equals ht.

Meng: So they’re making sure that when we look at an EMG window, the model isn't just looking at a raw signal; it’s looking at how hard the muscles are contracting and how that intensity is changing over time.

Lalam: And finally, for alignment, they propose semantic query alignment where a global representation zglobal is pooled and projected from the EMG token sequence Hemg to get zglobal.

Tom: Then a set of learnable semantic queries Q attends to those EMG tokens to produce zquery using fq(Q˜m). The final EMG representation zemg is then a weighted combination of that global evidence and the query-based evidence.

Jane: This whole structure is designed for bidirectional retrieval, where the EMG segment x maps to a normalized representation zemg, and you use that zemg as a query against the action-semantic representations Pc to find semantic similarity s(x, c) equals zemg(x)⊤Pc.

Lu: The evaluation tests this across seven protocols on datasets including NinaPro DB2 and EMG2Pose, covering everything from random splits to able-bodied-to-amputee transfer.

Meng: And the key finding is that MyoSem achieves the strongest overall bidirectional retrieval performance among all the methods they evaluated.

Lalam: The ablation studies show how much each piece—the semantic prototypes, activation-aware encoding, and those semantic queries—shapes the learned space, which really shows how each component contributes to the final result.

Tom: So it’s moving EMG hand action understanding from that old fixed-label recognition approach toward this new language-queryable semantic retrieval system. This is a significant step forward for how we handle these physiological signals.

Conclusion: Jane: Wrapping up on MyoSem: Aligning Electromyography with Natural-Language Action Semantics, the authors essentially provide a practical framework for description-based EMG hand action understanding that opens the door to future language-mediated interaction systems.

Tom: The framework addresses those core challenges through three main components: the multi-view action-semantic text space, the activation-aware EMG encoder, and semantic query alignment.

Jane: It’s about moving past just classifying actions into a retrieval system where you can search using descriptions rather than just matching fixed labels. This allows for evaluation on new users or new action classes whose descriptions are available only at inference time.

Lu: The implication is that we are laying groundwork for future language-mediated interaction systems built on physiological signals like EMG, which could become much more intuitive because they aren't limited by the initial labeling of every single possible gesture.

Meng: From an engineering standpoint, this means that prosthetic or wearable systems can be much more robust because they don't have to rely on a perfectly labeled dataset for every new interaction.

Lalam: It sets the stage for a future where we have language-grounded understanding of these non-text perceptual signals, which is something really exciting when you consider how it could improve culture and interaction.

Tom: Overall, MyoSem gives us a practical way to handle this complex alignment problem between muscle signal dynamics and natural language action semantics. This work lays the groundwork for systems that can truly understand and interact with hand gestures based on what we say about them.

South China University of Technology

cs.CV, cs.AI

Submitted: 2026-05-29

Updated: 2026-10-09

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 77/100

The gist: The gist The framework MyoSem presents an EMG–action semantic alignment framework that maps low-level EMG signals into a shared semantic space constructed from multi-view action descriptions,

Key concepts

Multi-View Action-Semantic Text Space
This concept involves taking one hand action and describing it from several different perspectives, such as original labels, pose sequences, canonical descriptions, and paraphrases. These varied text views are then processed by a shared language model to create rich semantic representations for each action class.
Activation-Aware EMG Encoder
This component processes the raw EMG signal by injecting descriptors that capture local muscle activation intensity (rmst), energy changes (energyt), and contraction state variations (vart). These specific features are woven into the EMG data tokens, ensuring the representation preserves important dynamics related to how a muscle is actually contracting.
Semantic Query Alignment
This technique uses a small set of learnable 'queries' to actively read and extract evidence from the processed EMG signal. By combining this query-based evidence with a global summary of the signal, the model creates an EMG representation that is specifically tuned to match action descriptions during retrieval.

Terminology

Summary

The gist The framework MyoSem presents an EMG–action semantic alignment framework that maps low-level EMG signals into a shared semantic space constructed from multi-view action descriptions, enabling bidirectional retrieval between EMG signals and text descriptions.

Introduction

Electromyography (EMG) directly reflects human muscle activation and is a key sensing modality for gesture recognition, prosthetic control, and wearable interaction Most existing EMG methods commonly formulate hand action understanding as classification over fixed labels, making it difficult to support querying, retrieval, and generalization based on action descriptions. MyoSem combines multi-view action-semantic construction, activation-aware EMG encoding, and semantic query alignment.

Research Questions

The study addresses three research questions to guide the framework RQ1 is How can we construct an action semantic space that groups different natural-language descriptions of the same action while preserving finegrained differences between actions? RQ2 is How can we learn stable and generalizable EMG representations that preserve activation, energy, and contraction dynamics relevant to action semantics? RQ3 is How can we align EMG representations to the action semantic space and extract actionrelevant local evidence from EMG sequences?

Method

MyoSem is a three-stage framework for aligning EMG signals with description-based action semantics Stage 1 organizes short action labels into multi-view natural-language descriptions and constructs class-level action-semantic representations. Stage 2 trains an activation-aware EMG encoder to preserve local activation intensity, energy changes, and contraction dynamics. Stage 3 performs semantic query alignment, aligning EMG representations with these action-semantic representations.

**- **

Task Formulation

The task is formulated as bidirectional retrieval between EMG signals and action-semantic representations, rather than as closed-set classification Given an EMG segment x ∈ R T ×C, the model maps it to a normalized representation zemg ∈ R d. Each action class c is associated with an action-semantic representation Pc ∈ R d, and all class-level representations form P ∈ R C×d. At inference time, EMG-to-text retrieval uses zemg as the query and retrieves from the actionsemantic representation set for cosine similarity s(x, c) = zemg(x)⊤Pc (1).

**- **

Multi-View Action-Semantic Text Space

For each action class, four text views are defined: T0 is the original short label, T1 is a pose-sequence-derived process description, T2 is a canonical action description, and T3 is natural-language paraphrases. All text views used for training are written in Chinese and encoded by a shared MacBERT text encoder. The full objective aligns process descriptions with canonical descriptions, adding short-label anchoring and paraphrase consistency. After training, the class-level action-semantic representation is computed as Pc = norm 1Vc X v e(v)c! (20).

**- **

Activation-Aware EMG Encoder

The encoder injects activation-aware descriptors into EMG patch tokens to preserve local activation intensity, energy changes, and contraction dynamics Given an EMG window, three local descriptors are computed: rmst describes local activation intensity, energyt captures the energy envelope, and vart reflects changes in contraction state. These descriptors are injected into the patch token through gated residual connections. The token sequence H = ht is then fed into a Transformer encoder for temporal contextualization

**- **

Semantic Query Alignment

The model proposes semantic query alignment, which uses a small set of learnable queries to read action-relevant evidence from EMG tokens. Given the EMG token sequence Hemg ∈ R L×d, a global representation zglobal is obtained through pooling and projection. M learnable semantic queries Q ∈ R M×d attend to the EMG token sequence to produce zquery = norm 1X M m=1 fq(Q˜m)! (23). The final EMG representation combines global and query-based evidence as zemg = norm(1 − α)zglobal + αzquery (12).

Experiments

The experiments are formulated as bidirectional retrieval between EMG representations and classlevel action-semantic representations rather than closed-set classification. Seven protocols are evaluated, including NinaPro-random, NinaPro-cross-user, NinaPro-cross-class, E2Prandom, E2P-cross-user, E2P-cross-class, and DB2→DB3. The retrieval score is s(x, c) = z⊤ x Pc (16).

**- **

Results and Analysis

MyoSem achieves the best or near-best performance in most protocols, indicating that it preserves semantic retrieval ability not only under random splits, but also for new users, held-out action classes or amputee-user transfer. Table 1 shows EMG-to-text retrieval results with uncertainty. MyoSem outperforms all generic text encoder baselines under all seven protocols, showing that textderived action-semantic representations can also serve as action-description query entries for retrieving EMG samples. Figure 3 shows that the Full MyoSem model achieves Top-1 73.9% | margin +0.07 in the NinaPro cross-modal class similarity metric.

Conclusion

MyoSem provides a practical framework for description-based EMG hand action understanding and lays groundwork for future language-mediated interaction systems built on physiological signals such as EMG. The framework addresses three central challenges through a multiview action-semantic text space, an activationaware EMG encoder, and semantic query alignment.

Limitations

The study has several boundaries First, although MyoSem moves EMG-based hand action understanding from closed-set classification toward action-semantic alignment, our experiments are still mainly organized around dataset-defined hand action categories. Second, the multi-view text descriptions are constructed from dataset labels, pose or stage information, and human-designed normalization rules. Third, although we evaluate cross-user, crossclass, and DB2→DB3 transfer to amputee users, real EMG applications are also affected by device type, electrode placement, skin condition. Finally, MyoSem studies the alignment between EMG and language-described action semantics rather than serving as a complete general-purpose EMG foundation model.

Improvements for AI systems

  1. Bold header: Action-Semantic Grounding Framework

This framework transforms EMG hand action understanding from fixed-label recognition toward queryable bidirectional semantic retrieval, enabling systems to support querying, retrieval, and generalization based on action descriptions rather than just classification over predefined labels.

  1. Bold header: Multi-View Semantic Text Space Construction

The system constructs a rich text space by expanding short labels into process descriptions, canonical descriptions, and natural-language paraphrases, ensuring that different expressions of the same action are organized as retrievable semantic targets.

  1. Bold header: Activation-Aware EMG Encoding

The system introduces an encoder that preserves local activation intensity, energy changes, and contraction dynamics, specifically by injecting descriptors like rmst (local activation intensity), energyt (energy envelope), and vart (changes in contraction state) into the patch tokens.

  1. Bold header: Semantic Query Alignment Mechanism

The system employs a mechanism where semantic queries read action-relevant evidence from EMG token sequences, resulting in an EMG representation that combines global context with local cues, as shown by the final equation: zemg = norm(1 − α)zglobal + αzquery.

  1. Bold header: Bidirectional Retrieval Capabilities

The improved system supports bidirectional retrieval, meaning it can perform both EMG-to-text retrieval (using an EMG segment to retrieve descriptions) and text-to-EMG retrieval (using a class-level description to retrieve candidate EMG samples).

Sources

Related papers