Motion-Omni: End-to-End Joint Speech and Full-Body Motion for Spoken Dialogue

arXiv:2609.04250 · cs.SD, cs.CV, eess.AS · Submitted 2026-08-28 · Read on arXiv

cs.SD, cs.CV, eess.AS

Submitted: 2026-08-28

Updated: 2026-09-25

Comments: 30 pages, 6 figures, 12 tables. Updated figures, presentation, and author notes. Project page: https://step-out.github.io/Motion-Omni-Page/ Code: https://github.com/step-out/Motion-Omni Data: https://huggingface.co/datasets/ChengqianMa/Motion-Omni

Code: https://github.com/project-numina/aimo-progress-prize

Project page: https://step-out.github.io/Motion-Omni-Page

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Benchmark — 9× in this paper · explained in 2 episodes
• Alignment — 8× in this paper · explained in 1 episode
• Fidelity — 6× in this paper · explained in 2 episodes
• Adaptation — 5× in this paper · explained in 1 episode
• Model-Agnostic — 5× in this paper · explained in 2 episodes
• Calibration — 4× in this paper · explained in 7 episodes
• Pseudo-labeling — 4× in this paper · explained in 3 episodes
• Semantic Alignment — 4× in this paper · explained in 3 episodes
• Transformer — 4× in this paper · explained in 2 episodes
• Word Error Rate — 4× in this paper
• Ablation — 3× in this paper · explained in 1 episode
• Reconstruction Error — 3× in this paper · explained in 2 episodes
• Consistency — 2× in this paper · explained in 3 episodes
• Decode — 2× in this paper · explained in 2 episodes
• Embeddings — 2× in this paper · explained in 2 episodes
• Fine-Tuning — 2× in this paper · explained in 5 episodes
• Flow Matching — 2× in this paper · explained in 8 episodes
• Language of Motion — 2× in this paper
• Pseudo-labels — 2× in this paper · explained in 2 episodes
• Token-as-Query Gated Fusion — 2× in this paper
• Aggregation — 1× in this paper · explained in 1 episode
• Annotator Agreement — 1× in this paper · explained in 1 episode
• Automatic Speech Recognition — 1× in this paper
• Automatic Speech Recognition (ASR) — 1× in this paper · explained in 2 episodes
• B EAT C ORRELATION S CORE — 1× in this paper
• Benchmarking — 1× in this paper · explained in 3 episodes
• Core — 1× in this paper · explained in 2 episodes
• Diffusion Model — 1× in this paper · explained in 8 episodes
• Diffusion Models — 1× in this paper · explained in 6 episodes
• Distillation — 1× in this paper · explained in 2 episodes
• Feature Extraction — 1× in this paper · explained in 2 episodes
• Foundation Model — 1× in this paper · explained in 3 episodes
• Generative AI — 1× in this paper · explained in 2 episodes
• Gradient Flow — 1× in this paper · explained in 2 episodes
• Transfer Learning — 1× in this paper · explained in 10 episodes
• Unified Framework — 1× in this paper · explained in 5 episodes

Sources

• Seamless Interaction: Dyadic Audiovisual Motion Modeling and Large-Scale Dataset
• Seed-TTS: A Family of High-Quality Versatile Speech Generation Models
• GLM-TTS Technical Report
• Moshi: a speech-text foundation model for real-time dialogue
• CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokens
• CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models
• CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-training
• VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interaction
• OpenThoughts: Data Recipes for Reasoning Models
• FireRedTTS: A Foundation Text-To-Speech Framework for Industry-Level Generative Speech Applications
• Kimi-Audio Technical Report
• Tulu 3: Pushing Frontiers in Open Language Model Post-Training
• Fake it to make it: Using synthetic data to remedy the data shortage in joint multimodal speech-and-gesture synthesis
• MIBURI: Towards Expressive Interactive Gesture Synthesis
• GPT-4o System Card
• Dialogue Act Modeling for Automatic Tagging and Recognition of Conversational Speech
• Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
• Qwen3-TTS Technical Report
• Mini-Omni2: Towards Open-source GPT-4o with Vision, Speech and Duplex Capabilities
• Qwen2.5-Omni Technical Report

Related papers

• Few-Shot Open-Set Audio Classification via Transductive Prototype Refinement and Class Logit Enhancement
• Spectral Masking and Interpolation Attack (SMIA): A Black-box Adversarial Attack against Voice Authentication and Anti-Spoofing Systems
• Remix the Timbre: Diffusion-Based Style Transfer Across Polyphonic Stems
• AVMeme Exam: A Multimodal Multilingual Multicultural Benchmark for LLMs' Contextual and Cultural Knowledge and Thinking
• SoundWeaver: Compositional Warm-Starting for Text-to-Audio Diffusion Serving
• WASIL: In-the-Wild Arabic Spoken Interactions with LLMs