Semantic Motion Anchors: Bridging Motion and Meaning in Co-Speech Gestures
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "Semantic Motion Anchors".
Jane: Here's the gist of the paper. Learning a shared representation between spoken text and gesture is central to co-speech gesture retrieval, synthesis, and understanding Direct contrastive alignment between transcripts and continuous motion embeddings often overemphasizes low-level kinematics and misses the symbolic content of semantic gestures We propose semantic motion anchors, natural-language abstractions of gesture motion capturing physical form and communicative intent Our method improves text-to-gesture R@1 by 8.2% over a direct text-motion baseline and outperforms prior retrieval approaches on text to gesture and gesture to text retrieval directions Semantic motion anchor supervision helps retrieve gestures that are semantically meaningful for the spoken query, rather than defaulting to generic motion patterns.
Tom: Gestures are a core channel of human communication, and learning a shared space between language and gestures that meaningfully aligns raw motion sequences with spoken language is a key requirement for these tasks Approaches that directly map raw motion sequences with spoken text often fail to capture higher-level semantics, instead learning averaged representations dominated by frequent beat gestures Semantic gestures are sparse and lie in the long tail of natural human motion distributions, making them underrepresented despite their importance for conveying communicative intent Current retrieval strategies are often based on heuristic or rule-based matching, which remain limited in effectively modeling semantic gesture alignment.
Jane: The core challenge lies in how gestures and spoken language are represented and mapped, as existing methods conflate similarity in physical motion with similarity in semantic meaning.
Tom: How the training signal is built The approach consists of three main components to generate semantic motion anchors First, a two-stream RVQ-VAE compresses continuous 3D gesture sequences into discrete motion tokens, and deterministically maps each primitive to a structured natural-language fragment describing observable spatial and kinematic properties Second, an LLM composes these token-level descriptions with the speech transcript into semantic motion anchors for each gesture Third, these anchors are used as auxiliary supervision in contrastive text-gesture motion retrieval training.
Title and authors: Jane: So, what exactly are these semantic motion anchors? How do they change the game compared to just looking at raw motion data against speech transcripts?
Lu: The method works by first taking continuous three-dimensional gesture sequences and compressing them into discrete tokens using a two stream RVQ VAE <ref:2605.30608#pg1>. Then, each of those tokens gets deterministically mapped to a structured natural-language fragment describing its spatial and kinematic properties <ref:2605.30608#pg1>.
Tom: And that’s where the LLM comes in—it composes those token-level descriptions with the speech transcript to create these semantic motion anchors for each gesture <ref:2605.30608#pg1>. So it’s like giving the movement a verbal label based on what it looks like and what it might mean in context.
Meng: So they are essentially grounding the motion in language before you even try to match it to a sentence, which makes sense for filtering out all those generic beat gestures we talked about <ref:2605.30608#pg3>.
Lalam: I see the structure of their objective function as a combination of terms: they have the motion-to-text alignment, plus a physical motion term and a semantic intent term, plus a bounding box term <ref:2605.30608#pg1>. It routes all these abstractions through this shared anchor projector to connect them to the right modality.
Tom: Right, so they’re not just comparing movement vectors anymore; they are comparing descriptions of movement and description of meaning against each other <ref:2605.30608#pg1>.
The paper's summary: Jane: So, if we boil it down to what the paper is actually trying to achieve, what’s the main payoff they are claiming here? What did they measure that showed this worked?
Lu: They claim that using these semantic motion anchors as auxiliary supervision helps them retrieve gestures that have actual semantic meaning for the spoken query, instead of just defaulting to generic motion patterns <ref:2605.30608#pg1>. This moves retrieval away from just matching physical shape to matching communicative intent.
Tom: And the numbers back up this idea pretty strongly. They improved their text-to-gesture R at one from thirty-nine point one to forty-two point three which is an eight point two percent relative improvement over a direct text-motion baseline <ref:2605.30608#pg1>.
Meng: An eight point two percent jump sounds solid for a retrieval system, but that depends on how you measure the success of the retrieved gesture in a real app, right? Does that R at one actually mean anything to a user?
Jane: It does. And they showed in a downstream retrieval augmented gesture generation study that users significantly preferred the gestures retrieved by this approach over those from RAG-Gesture Mughal et al. <ref:2605.30608#pg1>. They saw about seventy-two point two percent preference for their results compared to twenty-seven point eight percent <ref:2605.30608#pg1>.
Lalam: The largest numerical gains they report are in categories like Quantification, Temporal reference, Uncertainty, and Emotion when you look at the semantic label match rate <ref:2605.30608#pg1>. That tells us the semantic grounding is hitting things beyond just basic motion recognition.
The paper's improvements: Tom: Beyond just that retrieval metric, what else did they show about how this method works under the hood? What’s the real mechanism behind why it’s better than just using a standard text-to-motion mapping?
Lu: They introduced this four term objective function: the motion-to-text alignment, plus a physical motion term scaled by lambda p, plus a linguistic term for semantic intent scaled by lambda s, and finally a bounding box term for the bridge between them <ref:2605.30608#pg1>. This structure routes each abstraction to its corresponding modality through that shared anchor projector.
Jane: So they are explicitly training the model to handle both the physical form and the communicative intent separately, using those two different supervision signals <ref:2605.30608#pg2>. That’s a neat way to decompose the problem into what looks like how it is and what it means.
Meng: If they are separating those signals, does that mean if you only focus on the physical form, you can still get some decent motion alignment even without the semantic part? Or is that just noise?
Lu: The paper shows that this separation allows them to supervise motion with descriptions of how the gesture looks, while supervising speech with descriptions of what the gesture means <ref:2605.30608#pg2>. They argue this leads to more robust representations because they are learning distinct, complementary pieces of information <ref:2605.30608#pg2>.
Tom: And that leads to their cross-dataset generalization study. They tested whether a model trained only on the BEAT two dataset could transfer to the unseen TED dataset <ref:2605.30608#pg1>.
Conclusion: Jane: So, wrapping up, what is the big picture implication here for how we build these co-speech systems? What does this mean for someone who just listens to the show?
Lu: The main implication is that abstracting gestures into relevant physical or communicative properties provides a more transferable retrieval interface across different datasets <ref:2605.30608#pg1>. It suggests that if you can capture those properties, the system has a better chance of working on new data, not just the one it was trained on.
Tom: And I agree with that part because they showed it improves over the text-contrastive baseline when moving from BEAT two to TED <ref:2605.30608#pg1>. The numbers show that this abstraction is actually helping them generalize beyond just the initial training set <ref:2605.30608#pg1>.
Meng: From a deployment standpoint, the fact that they can use these anchors to ground generation in a way that users prefer means we might be able to deploy systems where they generate gestures that actually convey the right message, not just random motion sequences <ref:2605.30608#pg1>.
Lalam: I just want to add something about what they admit is missing. They state that their semantic motion anchors only capture a subset of gesture-relevant attributes, meaning fine-grained properties like specific gesture phases and subtle finger articulation aren't fully modeled <ref:2605.30608#pg1>.
Tom: That’s fair. It tells us the method is good at capturing high-level intent and form, but it won't capture every single tiny detail of human motion <ref:2605.30608#pg1>.
Jane: So, to sum up, Semantic Motion Anchors offer a way to use natural language abstractions to bridge the gap between what a person does physically and what they are trying to communicate verbally in co-speech gestures <ref:2605.30608#pg1>. That’s where we’ll leave it for today.
Saarland University of Applied Sciences Saarland University Institute for Informatics Department of Computer Science University of British Columbia Vector Institute Zuse School ELIZA
cs.CL
Submitted: 2026-05-28
Updated: 2026-11-09
Comments: NeuRIPS 2026
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 92/100
The gist: Learning a shared representation between spoken text and gesture is central to co-speech gesture retrieval, synthesis…
Key concepts
- Semantic Motion Anchors
- These are natural-language descriptions that capture both the physical form (how it looks) and communicative intent (what it means) of a gesture. They act as a bridge between raw 3D motion data and textual language, allowing the model to understand gestures beyond just their appearance.
- Two-Stream RVQ-VAE
- This component compresses continuous 3D gesture sequences into discrete motion tokens. It deterministically maps these tokens to structured natural-language fragments describing the observable spatial and kinematic properties of the gesture, essentially turning complex motion into understandable descriptions.
- Contrastive Text-Gesture Retrieval
- This is the training process where the model learns to align spoken text with continuous motion embeddings. The semantic motion anchors are used as auxiliary supervision in this contrastive learning, teaching the system to prioritize gestures that have a meaningful semantic relationship with the query.
Terminology
Summary
Learning a shared representation between spoken text and gesture is central to co-speech gesture retrieval, synthesis, and understanding Direct contrastive alignment between transcripts and continuous motion embeddings often overemphasizes low-level kinematics and misses the symbolic content of semantic gestures We propose semantic motion anchors, natural-language abstractions of gesture motion capturing physical form and communicative intent Our method improves text-to-gesture R@1 by 8. p. 0 de
Abstract
Learning a shared representation between spoken text and gesture is central to
co-speech gesture retrieval, synthesis, and understanding, but remains challenging
for semantically meaningful gestures whose communicative intent is not captured
by motion alone. Direct contrastive alignment between transcripts and continuous
motion embeddings often overemphasizes low-level kinematics and misses the
symbolic content of semantic gestures. 2% over a direct text-motion baseline and outperforms prior retrieval approaches on text to gesture and gesture to text retrieval
directions. Beyond aggregate retrieval metrics, semantic motion anchor supervision
helps retrieve gestures that are semantically meaningful for the spoken query, rather
than defaulting to generic motion patterns.Where the paper says it
Where the paper says it
Introduction and Problem Statement
Gestures are a core channel of human communication, and learning a shared space between language and gestures that meaningfully aligns raw motion sequences with spoken language is a key requirement for these tasks Approaches that directly map raw motion sequences with spoken text often fail to capture higher-level semantics, instead learning averaged representations dominated by frequent beat gestures Semantic gestures are sparse and lie in the long tail of natural human motion distributions, making them underrepresented despite their importance for conveying communicative intent Current retrieval strategies are often based on heuristic or rule-based matching, which remain limited in effectively modeling semantic gesture alignment. The core challenge lies in how gestures and spoken language are represented and mapped, as existing methods conflate similarity in physical motion with similarity in semantic meaning. p. 1 A central challenge underlying these approaches lies in how gestures and spoken language are
represented and mapped. Most existing methods learn motion embeddings under reconstructionbased objectives, which emphasize low-level kinematic features, but these are often not directly
aligned with communicative intent, which is highly relevant for semantic gestures.Where the paper says it
How the training signal is built
The approach consists of three main components to generate semantic motion anchors First, a two-stream RVQ-VAE compresses continuous 3D gesture sequences into discrete motion tokens, and deterministically maps each primitive to a structured natural-language fragment describing observable spatial and kinematic properties Second, an LLM composes these token-level descriptions with the speech transcript into semantic motion anchors for each gesture Third, these anchors are used as auxiliary supervision in contrastive text-gesture motion retrieval training. The full objective combines four contrastive terms: L = Ltm(zt, zm) + λp Lphys(zp, zm) + λs Lint(zs, zt) + λb Lbr(zp, zs) This structure routes each abstraction to its corresponding modality through a shared anchor projector. p. 4 Each L term denotes the symmetric InfoNCE loss with learnable temperature. The full objective combines four contrastive terms:
L = Ltm(zt, zm) + λp Lphys(zp, zm) + λs Lint(zs, zt) + λb Lbr(zp, zs), (2)
with auxiliary weights λp, λs, λb.Where the paper says it
Where the gains come from
The method improves text-to-gesture R@1 from 39.1 to 42.3, a +3.2 point absolute gain, corresponding to an 8. p. 1 , 2024], our method improves text-to-gesture R@1 from 39. 2
point absolute gain, corresponding to an 8. 2% relative improvement over the direct text-gesture
motion baseline. In a downstream retrieval-augmented gesture generation study, users significantly
preferred gestures retrieved by our approach over those from RAG-Gesture Mughal et al. , 2023, Liu et al. , 2025, Hegde et al. Unlike
Recall@1, this metric rewards semantic alignment:
retrieving a different gesture instance that expresses
the same communicative intent is counted as correct. From Table 3 we see that the largest numerical gains
appear in Quantification, Temporal reference, Uncertainty, and Emotion, which are categories with
distinctive gestural form where intent-conditioning
provides the clearest signal and rarer in the data. On BEAT2, our method
improves text-to-gesture R@1 by 8. 2% over a direct text-motion baseline and outperforms prior retrieval approaches on text to gesture and gesture to text retrieval
directions.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Cross-Dataset Generalization
We evaluate whether a retrieval model trained only on BEAT2 transfers to the unseen TED dataset Ram et al. [2025] (N=778) In the TED-to-BEAT2 setting, replacing the BEAT2 motion gallery with gesture-anchor proxies improves over the text-contrastive baseline This suggests that abstracting gestures into relevant physical or communicative properties provides a more transferable retrieval interface across datasets. p. 7 The motion-embedding gallery tests whether the learned BEAT2 motion
space transfers directly. The anchor-proxy gallery tests whether abstracting gestures into relevant
physical or communicative properties provides a more transferable retrieval interface.Where the paper says it
Limitations and Future Work
Our semantic motion anchors capture only a subset of gesture-relevant attributes, meaning fine-grained properties such as gesture phases and subtle finger articulation are not fully modeled Anchor generation introduces computational overhead and the use of closed-source LLM, which is a one-time cost used only offline to create training anchors The method may not generalize equally across cultures, languages, or demographic groups since gesture conventions vary significantly across these dimensions.
Conclusion
We introduced semantic motion anchors for text-gesture retrieval, natural-language abstractions that represent gesture motion through physical form and communicative intent Our framework converts 3D motion into upper-body tokens, verbalizes them into structured physical descriptions, and grounds them in the transcript to generate semantic motion anchors used as auxiliary supervision Applied to retrieval-augmented co-speech gesture generation, users significantly preferred gestures retrieved by our approach over those retrieved by RAG-Gesture suggesting using semantic motion anchors produce better gestures that match communicative intent.
References
Tenglong Ao, Zeyi Zhang, and Libin Liu. Gesturediffuclip: Gesture diffusion model with clip latents Vasileios Choutas, Georgios Pavlakos, Timo Bolkart, Dimitrios Tzionas, and Michael J. p. 9 ACM Transactions on Graphics (TOG), 42(4):1–18, 2023. Vasileios Choutas, Georgios Pavlakos, Timo Bolkart, Dimitrios Tzionas, and Michael J. URL https://expose. Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang, Wei Ji, Xingyu Li, and Li Cheng. In Proceedings of the IEEE/CVF conference on
computer vision and pattern recognition, pages 5152–5161, 2022. Sindhu B Hegde, KR Prajwal, Taein Kwon, and Andrew Zisserman. In Proceedings of the IEEE/CVF International Conference on Computer
Vision, pages 9977–9987, 2025. Biao Jiang, Xin Chen, Wen Liu, Jingyi Yu, Gang Yu, and Tao Chen. Biao Jiang, Xin Chen, Wen Liu, Jingyi Yu, Gang Yu, and Tao Chen. Motiongpt: Human motion as a
foreign language. Adam Kendon. Gesture: Visible action as utterance. Haiyang Liu, Zihao Zhu, Giorgio Becherini, Yichen Peng, Mingyang Su, You Zhou, Xuefei Zhe,
Naoya Iwamoto, Bo Zheng, and Michael J Black. Emage: Towards unified holistic co-speech
gesture generation via expressive masked audio gesture modeling. Haiyang Liu, Zihao Zhu, Giorgio Becherini, Yichen Peng, Mingyang Su, You Zhou, Xuefei Zhe,
Naoya Iwamoto, Bo Zheng, and Michael J Black. Emage: Towards unified holistic co-speech
gesture generation via expressive masked audio gesture modeling. In Proceedings of the 33rd ACM International Conference
on Multimedia, pages 11091–11100, 2025. Xian Liu, Qianyi Wu, Hang Zhou, Yinghao Xu, Rui Qian, Xinyi Lin, Xiaowei Zhou, Wayne
Wu, Bo Dai, and Bolei Zhou. Xian Liu, Qianyi Wu, Hang Zhou, Yinghao Xu, Rui Qian, Xinyi Lin, Xiaowei Zhou, Wayne
Wu, Bo Dai, and Bolei Zhou. Learning hierarchical cross-modal association for co-speech
gesture generation. M Hamza Mughal, Rishabh Dabral, Merel CJ Scholman, Vera Demberg, and Christian Theobalt. Retrieving semantics from the deep: an rag solution for gesture synthesis. M Hamza Mughal, Rishabh Dabral, Merel CJ Scholman, Vera Demberg, and Christian Theobalt. Retrieving semantics from the deep: an rag solution for gesture synthesis. Vasileios Choutas, Georgios Pavlakos, Timo Bolkart, Dimitrios Tzionas, and Michael J. Monocular expressive body regression through body-driven attention. Vasileios Choutas, Georgios Pavlakos, Timo Bolkart, Dimitrios Tzionas, and Michael J. Monocular expressive body regression through body-driven attention. Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang, Wei Ji, Xingyu Li, and Li Cheng. Generating
diverse and natural 3d human motions from text. Semantic Motion Anchors: Bridging Motion and
Meaning in Co-Speech Gestures
Varsha Suresh1,*, Mohammad Mahdi Abootorabi3,4,5,†, *,
Mohamed Salman1 M. Hamza Mughal2
,
Christian Theobalt1,2
, Ashwin Ram1
, Jürgen Steimle1
, Vera Demberg1,2
1Saarland University, 2MPI for Informatics, Saarland Informatics Campus,
3University of British Columbia, 4Vector Institute, 5Zuse School ELIZA
{vsuresh,vera}@lst. We generate each anchor in three steps: motion tokenization, token
verbalization, and transcript-grounded reasoning. Motion Tokenization: Given a gesture sequence Xi
, we first compress the continuous 3D motion
sequence into a sequence of discrete motion tokens using a two-stream RVQ-VAE [Van Den Oord
et al. de
{mmughal,theobalt}@mpi-inf. de
Abstract
Learning a shared representation between spoken text and gesture is central to
co-speech gesture retrieval, synthesis, and understanding, but remains challenging
for semantically meaningful gestures whose communicative intent is not captured
by motion alone. As the method is
trained primarily on BEAT2 and TED, it may not generalize equally across cultures, languages, or
demographic groups, since gesture conventions vary significantly across these dimensions. References
Tenglong Ao, Zeyi Zhang, and Libin Liu. To address this, semantics-aware approaches
introduce language-motion alignment, semantic planning, or retrieval-augmented generation to
produce gestures that better match discourse meaning [Ao et al. , 2023, Zhi et al. Haiyang Liu, Zihao Zhu, Giorgio Becherini, Yichen Peng, Mingyang Su, You Zhou, Xuefei Zhe,
Naoya Iwamoto, Bo Zheng, and Michael J Black. Emage: Towards unified holistic co-speech
gesture generation via expressive masked audio gesture modeling.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Table 1
Gesture → Text Text → Gesture Method R@1 R@5 R@10 MRR R@1 R@5 R@10 MRR
Table 2
Anchor ablation on BEAT2. † indicates significant improvement over using paired t-test; p < 0. p. 6 85†
Table 2: Anchor ablation on BEAT2. † indicates significant improvement over using
paired t-test; p < 0.Where the paper says it
Table 3
Semantic label match rate (%) on BEAT2 test set
Table 4
Qualitative retrieval examples
Table 5
TED-to-TED since motion-embed. † Here, the motion description is passed via the transcript encoder
Table 11
Full anchor-content ablation on BEAT2. Significance markers indicate paired t-tests against the specified baseline
Table 12
Proxy-based cross-dataset generalization on TED.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Semantic motion anchors improve text-to-gesture retrieval by providing auxiliary supervision, which
helps retrieve gestures that are semantically meaningful for the spoken query, rather than defaulting to generic motion patterns.
This allows the model to learnrelevant details required for retrieval, i.e., gesture form and function,
making the mapping between spoken language and motionmore semantically grounded.
-
The framework enables modality-matched supervision by decomposing each semantic anchor into a physical-form component and a communicative intent component, allowing for separate contrastive learning on both modalities:
Motion is supervised by descriptions of how the gesture looks, while the transcript is supervised by descriptions of what the gesture means.
-
The system can perform more accurate retrieval in downstream generation tasks because
users significantly preferred gestures retrieved by our approach over a retrieval-augmented generation baseline,
demonstrating thatsemantically grounded retrieval translates to gestures that better convey communicative intent in downstream generation.
Abstract
Learning a shared representation between spoken text and gesture is central to co-speech gesture retrieval, synthesis, and understanding, but remains challenging for semantically meaningful gestures whose communicative intent is not captured by motion alone. Direct contrastive alignment between transcripts and continuous motion embeddings often overemphasizes low-level kinematics and misses the symbolic content of semantic gestures. We propose semantic motion anchors, natural-language abstractions of gesture motion capturing physical form and communicative intent. Our method discretizes 3D gestures into body-hand motion primitives, verbalizes them into structured descriptions, and grounds them in the transcript to provide auxiliary contrastive supervision. On BEAT2, our method improves text-to-gesture R@1 by 8.2% over a direct text-motion baseline and outperforms prior retrieval approaches on text to gesture and gesture to text retrieval directions. Beyond aggregate retrieval metrics, semantic motion anchor supervision helps retrieve gestures that are semantically meaningful for the spoken query, rather than defaulting to generic motion patterns. A downstream retrieval-augmented gesture generation study showed that users significantly preferred gestures retrieved by our approach over a retrieval-augmented generation baseline, demonstrating that semantically grounded retrieval translates to gestures that better convey communicative intent in downstream generation.
Sources
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data