Embodied Learning of Reward for Musculoskeletal Control with Vision Language Models
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.
Rosa: I'm Rosa.
Dev: And I'm Dev, here with Taro.
Rosa: We're looking at the paper: "Embodied Learning of Reward for Musculoskeletal Control with Vision Language Models".
Dev: Here's the gist of the paper. Discovering effective reward functions remains a fundamental challenge in motor control of highdimensional musculoskeletal systems <ref:2512.23077#pg2>.
Rosa: MoVLR is introduced as a framework that leverages vision-language models (VLMs) to bridge the gap between goal specification and movement control <ref:2512.23077#pg2>.
Dev: Framework Overview MoVLR is a framework for automatic reward learning in high-dimensional musculoskeletal systems that integrates descriptive and dynamical feedback <ref:2512.23077#pg2>.
Rosa: It extracts musculoskeletal dynamics through policy optimization and rollout, rendering the resulting behaviors as movement videos <ref:2512.23077#pg2>.
Title and authors: Rosa: So, to get into the specifics of Embodied Learning of Reward for Musculoskeletal Control with Vision Language Models, we have to look at who wrote this. The authors are Saraswati Soedarmadji Chen Xuying Yunyue Wei Chen Zhang Yisong Yue and Yanan Sui.
Dev: That’s a pretty big team spanning Tsinghua University and Caltech, which suggests they’re tackling a problem that requires both deep control knowledge and strong AI capabilities.
Taro: I wonder how their different backgrounds—control engineering versus autonomy research versus vision modeling—actually helped them structure this specific workflow.
Rosa: They seem to have clearly divided the roles; you’ve got the control guys handling the policy optimization, and then the VLM side taking on that perceptual evaluation role.
Dev: The paper itself sets up a finite horizon Markov decision process framework for this reward learning problem, meaning they are treating it like a standard control problem where we're just trying to find that optimal single-step reward r star.
Taro: That’s interesting because the paper says the objective is to find one step reward r star that maximizes the global reward function R, which implies they're aiming for a single, effective learning signal.
Rosa: And they frame it as needing two main things: first, an embodied understanding of human movement to extract that implicit biomechanical knowledge from motion descriptions.
Dev: And second, an effective reward design part where the multimodal feedback—the video and the VLM assessment—is used to create interpretable and executable reward terms for that policy optimization.
Taro: It’s like they’re building a system that learns physics by watching itself move through a loop of action and perception.
Rosa: So, this paper is about taking those complex human movements, which are often implicit, and making them explicit so an AI can learn to replicate them effectively.
Dev: The authors clearly argue that existing methods often rely on episodic statistics or simple success signals, which isn't enough for the temporal dynamics involved in musculoskeletal control.
The paper's summary: Rosa: Moving on to what the paper actually summarizes, they introduce MoVLR as a framework specifically designed to bridge that gap between a high-level goal and the low-level movement control policy.
Dev: MoVLR works by taking a control policy, running it for some time to generate movement videos zeta (i), and then feeding those videos into a vision language model.
Taro: The VLM doesn't just give a number; it evaluates the motion against what the current best reward definition r star is supposed to look like, producing structured biomechanical feedback.
Rosa: That feedback is then used by a language model to refine that reward formulation in the next iteration, effectively doing a local search over the reward space.
Dev: This whole loop means they are iteratively distilling those implicit dynamical regularities—the underlying physics of movement—into explicit reward terms for policy optimization.
Taro: It sounds like they are essentially using language as a tool to reason about physical dynamics in a way that is grounded in what the vision model observes.
Rosa: The paper shows that this iterative refinement process leads to reward terms that are biomechanically consistent, which is the key finding they want to highlight.
Dev: They show that by using this multimodal feedback mechanism, you get interpretable and dynamically consistent reward representations for policy optimization.
The paper's improvements: Rosa: The paper highlights several improvements over previous approaches. One major one is the shift from relying on just episodic statistics to using continuous, structured feedback from the VLM.
Dev: They also emphasize that they are focusing on high-dimensional musculoskeletal systems, which means they’re dealing with redundancy and complexity that traditional torque-driven systems don't handle well.
Taro: And another point is how they handle the failure modes; you see them tested in settings like locomotion environments spanning flat, rough, and sloped terrain.
Rosa: The evaluation shows that MoVLR consistently achieves higher task performance in those locomotion environments compared to reward functions engineered by humans.
Dev: In manipulation tasks, they report achieving the lowest average position and orientation errors when compared against all other baselines they tested.
Taro: What’s interesting is how the evolution of those weighted reward terms across the refinement stages reveals how that feedback-driven process actually reorganizes the internal optimization landscape toward more natural behavior.
Conclusion: Rosa: So, to wrap up on Embodied Learning of Reward for Musculoskeletal Control with Vision Language Models, this framework provides a systematic way to discover reward functions without relying solely on manual engineering.
Dev: It’s about using vision language models as a perceptual bridge so that linguistic goals get grounded in physical dynamics through iterative refinement.
Taro: This approach offers a scalable path toward control for complex behaviors and morphologies because it generalizes across different movement tasks and system designs.
Rosa: The paper suggests that the iterative refinement mirrors the structure of human motor learning, moving from coarse stability constraints to fine-grained joint coordination.
Dev: It’s a principled way to get biologically plausible and interpretable rewards that guide musculoskeletal agents toward stable, natural motion.
Taro: I just think the VLM acting as that perceptual bridge really grounds the linguistic goal in physical dynamics, which is crucial for getting that consistency across different morphologies.
Rosa: That’s right. It offers a scalable path toward control for complex behaviors and morphologies by learning those implicit musculoskeletal dynamics automatically.
Tsinghua University · California Institute of Technology
cs.RO
Submitted: 2025-12-28
Updated: 2026-10-29
Comments: Accepted by L4DC 2026
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 91/100
The gist: Discovering effective reward functions remains a fundamental challenge in motor control of highdimensional musculoskeletal systems <ref:2512.23077#pg2>.
Key concepts
- Vision-Language Models (VLMs)
- VLMs are AI models that can understand both images (like movement videos) and text descriptions simultaneously. In this framework, the VLM observes the physical motion generated by the control policy and translates it into structured, language-based feedback that helps guide reward learning.
- Iterative Reward Refinement
- This is a learning loop where an initial reward function is tested against movement data. The VLM provides feedback on how well the movement matches desired dynamics, and a language model uses this feedback to synthesize better, more specific reward terms for the next attempt. This process refines rewards until they accurately reflect natural biomechanics.
- Musculoskeletal Dynamics
- This refers to the complex physical interactions and movements of a body's muscles, bones, and joints. The framework aims to extract these hidden dynamics from observed motion videos so that the AI can learn control policies that result in stable, biologically plausible movements across different environments.
- Finite Horizon MDP
- This is a mathematical model used to define the control problem as a sequence of decisions over time, where each decision affects future states. The reward function is defined as a combination of terms derived from the learned dynamics, allowing the system to optimize its actions toward long-term goals.
Terminology
Summary
Discovering effective reward functions remains a fundamental challenge in motor control of highdimensional musculoskeletal systems <ref:2512. p. 0 EDU
Yanan Sui1 YSUI@TSINGHUA. CN
1Tsinghua University, 2California Institute of Technology
Abstract
Discovering effective reward functions remains a fundamental challenge in motor control of highdimensional musculoskeletal systems. While humans can describe movement goals explicitly such
as “walking forward with an upright posture,” the underlying control strategies that realize these
goals are largely implicit, making it difficult to directly design rewards from high-level goals
and natural language descriptions. We introduce Motion from Vision-Language Representation
(MoVLR), a framework that leverages vision-language models (VLMs) to bridge the gap between
goal specification and movement control.Where the paper says it
Where the paper says it
Framework Overview
MoVLR is a framework for automatic reward learning in high-dimensional musculoskeletal systems that integrates descriptive and dynamical feedback <ref:2512.23077#pg2>. It extracts musculoskeletal dynamics through policy optimization and rollout, rendering the resulting behaviors as movement videos <ref:2512. p. 1 In this paper, we introduce Motion from Vision Language Representation (MoVLR), a framework for automatic reward learning in high-dimensional musculoskeletal systems that integrates
descriptive and dynamical feedback. As shown in Figure 1, MoVLR extracts musculoskeletal dynamics through policy optimization and rollout, rendering the resulting behaviors as movement
videos. As shown in Figure 1, MoVLR extracts musculoskeletal dynamics through policy optimization and rollout, rendering the resulting behaviors as movement
videos. A vision-language model evaluates these motions to produce structured biomechanical
feedback, which is then used by a language model to refine the reward formulation in subsequent
iterations. A vision-language model evaluates these motions to produce structured biomechanical
feedback, which is then used by a language model to refine the reward formulation in subsequent
iterations. Through this iterative process, implicit dynamical regularities are progressively distilled
into explicit reward terms.Where the paper says it
Where the paper says it
Where the paper says it
Control and Reward Learning
The paper formalizes the control problem as a finite horizon Markov decision process (MDP) where the reward is a linear combination of reward terms <ref:2512.23077#pg5>. The objective of reward learning is to find a single-step reward r∗ that maximizes the global reward function R <ref:2512. p. 3 Reward learning for musculoskeletal control
While the above control problem provides single-step reward definition, the control performances
are usually evaluated over full horizon. The objective of reward learning is to find single step reward
r
∗
that maximize the global reward function R:
r
∗ = argmaxrR(ζ), ζ = (s0, a0, · · · , sT −1, aT −1), ut = π
∗
r
(st), st+1 = f(st,ut), (5)
where π
∗
r
is the optimal policy derived by maximizing r. ” The
single-step reward r consists of multiple reward terms and parameters as code pieces which need to
be compatible with the policy optimization framework. Effective reward learning requires two key
components: (1) Embodied understanding of human movement which extracts implicit biomechanical knowledge from motion descriptions, and (2) Effective reward design which integrates
multimodal feedback to produce interpretable and executable reward terms for policy optimization.Where the paper says it
Where the paper says it
Multimodal Feedback Mechanism
The VLM acts as a semantic observer that produces interpretable, language-based evaluations rather than scalar scores <ref:2512.23077#pg5>. The VLM compares the rendered control dynamics ζ(i) against the dynamics generated under the current best reward definition <ref:2512. p. 5 Motion-language representation
We use the VLM as a semantic observer that produces interpretable, language-based evaluations
rather than scalar scores. The VLM compares the rendered control dynamics ζ
(i)
against the dynamics generated under the current best reward definition. The VLM also produces structured textual
feedback F of r
∗ which qualitatively evaluates the motion relative to R. This feedback serves as
reflective input to the LLM, guiding subsequent iterations of reward synthesis. , joint angles, actuator lengths,
and contact forces) that can parameterize executable reward terms. Given the contextual information and VLM-based evaluations F, the LLM performs a local
search over the current best reward function r
∗
to synthesize new reward terms.Where the paper says it
Where the paper says it
Where the paper says it
Experimental Validation
MoVLR was evaluated across diverse musculoskeletal systems and tasks, including locomotion environments spanning flat, rough, and sloped terrain <ref:2512.23077#pg9>. Experimental results show that MoVLR consistently achieves higher task performance in locomotion environments compared to human-engineered reward functions <ref:2512. p. 9 Through this method, we demonstrate that VLMs can successfully translate high-level motion descriptions into detailed control objectives, improving musculoskeletal performance across diverse environments and tasks. Experimental results across locomotion and manipulation tasks show that MoVLR consistently
outperforms both human-designed and language-based baselines. These results indicate that multimodal reward refinement produces
robust and transferable control objectives across biomechanical structures. As shown in Figure 4 (b), MoVLR achieves the lowest average position and orientation errors
compared to all baselines in the manipulation tasks, indicating more precise and stable object interactions. Evolution of weighted reward terms across refinement stages. The progression of residual reward weights across refinement stages reveals how the feedback-driven process reorganizes the
internal optimization landscape toward biomechanically consistent behavior.Where the paper says it
Where the paper says it
Where the paper says it
Generalization and Modularity
Ablation studies show that MoVLR generalizes across movement tasks, environments and system morphologies, producing interpretable reward terms that represent implicit musculoskeletal dynamics <ref:2512.23077#pg10>. The framework demonstrates robustness to actuator failure in the injured-body setting <ref:2512. p. 8 In the injured-body setting, transfer performance slightly improves (5. 8 m),
indicating robustness to actuator failure. The unified variant shows substantially degraded performance, often producing invalid or incomplete reward code and failing to improve control behavior
over iterations. These observations suggest that current vision–language models do not yet possess the compositional or programmatic reasoning required to perform both tasks simultaneously,
highlighting the importance of maintaining distinct feedback and synthesis stages. To assess whether this modularity is necessary, we implement a unified configuration in which a single multimodal model performs both
feedback interpretation and code synthesis. The unified variant shows substantially degraded performance, often producing invalid or incomplete reward code and failing to improve control behavior
over iterations.Where the paper says it
Where the paper says it
Where the paper says it
Conclusion
In this work, MoVLR is introduced as an automatic workflow that leverages vision-language models to bridge explicit language descriptions with implicit motor control required for high-dimensional musculoskeletal systems <ref:2512.23077#pg2>. By integrating multimodal feedback into the learning loop, MoVLR finds biomechanically grounded reward functions that are iteratively refined to guide the musculoskeletal agent toward stable, natural motion <ref:2512. p. 9 Conclusion and Discussion
In this work, we introduce MoVLR , an automatic workflow that leverages vision-language models
to bridge explicit language descriptions with implicit motor control required for high-dimensional
musculoskeletal systems. By integrating multimodal feedback into the learning loop, MoVLR finds
biomechanically grounded reward functions that are iteratively refined to guide the musculoskeletal
agent toward stable, natural motion. Acting as a perceptual bridge, the VLM grounds
linguistic goals in physical dynamics, producing interpretable and dynamically consistent reward
representations. This work offers a scalable and principled path toward biologically plausible, interpretable, and generalizable control for complex behaviors and morphologies. Acting as a perceptual bridge, the VLM grounds
linguistic goals in physical dynamics, producing interpretable and dynamically consistent reward
representations. This work offers a scalable and principled path toward biologically plausible, interpretable, and generalizable control for complex behaviors and morphologies. Acting as a perceptual bridge, the VLM grounds
linguistic goals in physical dynamics, producing interpretable and dynamically consistent reward
representations. This work offers a scalable and principled path toward biologically plausible, interpretable, and generalizable control for complex behaviors and morphologies. Acting as a perceptual bridge, the VLM grounds
linguistic goals in physical dynamics, producing interpretable and dynamically consistent reward
representations. This work offers a scalable and principled path toward biologically plausible, interpretable, and generalizable control for complex behaviors and morphologies. Acting as a perceptual bridge, the VLM grounds
linguistic goals in physical dynamics, producing interpretable and dynamically consistent reward
representations. This work offers a scalable and principled path toward biologically plausible, interpretable, and generalizable control for complex behaviors and morphologies. Acting as a perceptual bridge, the VLM grounds
linguistic goals in physical dynamics, producing interpretable and dynamically consistent reward
representations. This work offers a scalable and principled path toward biologically plausible, interpretable, and generalizable control for complex behaviors and morphologies. Acting as a perceptual bridge, the VLM grounds
linguistic goals in physical dynamics, producing interpretable and dynamically consistent reward
representations. This work offers a scalable and principled path toward biologically plausible, interpretable, and generalizable control for complex behaviors and morphologies. Acting as a perceptual bridge, the VLM grounds
linguistic goals in physical dynamics, producing interpretable and dynamically consistent reward
representations. This work offers a scalable and principled path toward biologically plausible, interpretable, and generalizable control for complex behaviors and morphologies. Acting as a perceptual bridge, the VLM grounds
linguistic goals in physical dynamics, producing interpretable and dynamically consistent reward
representations. This work offers a scalable and principled path toward biologically plausible, interpretable, and generalizable control for complex behaviors and morphologies. Experimental results across locomotion and manipulation tasks show that MoVLR consistently
outperforms both human-designed and language-based baselines. The iterative refinement of rewards mirrors the hierarchical structure of human motor learning, progressing from coarse stability
constraints to fine-grained joint coordination. Experimental results across locomotion and manipulation tasks show that MoVLR consistently
outperforms both human-designed and language-based baselines. The iterative refinement of rewards mirrors the hierarchical structure of human motor learning, progressing from coarse stability
constraints to fine-grained joint coordination. Experimental results across locomotion and manipulation tasks show that MoVLR consistently
outperforms both human-designed and language-based baselines. The iterative refinement of rewards mirrors the hierarchical structure of human motor learning, progressing from coarse stability
constraints to fine-grained joint coordination. Experimental results across locomotion and manipulation tasks show that MoVLR consistently
outperforms both human-designed and language-based baselines. The iterative refinement of rewards mirrors the hierarchical structure of human motor learning, progressing from coarse stability
constraints to fine-grained joint coordination. Experimental results across locomotion and manipulation tasks show that MoVLR consistently
outperforms both human-designed and language-based baselines. The iterative refinement of rewards mirrors the hierarchical structure of human motor learning, progressing from coarse stability
constraints to fine-grained joint coordination. Experimental results across locomotion and manipulation tasks show that MoVLR consistently
outperforms both human-designed and language-based baselines. The iterative refinement of rewards mirrors the hierarchical structure of human motor learning, progressing from coarse stability
constraints to fine-grained joint coordination. Experimental results across locomotion and manipulation tasks show that MoVLR consistently
outperforms both human-designed and language-based baselines. The iterative refinement of rewards mirrors the hierarchical structure of human motor learning, progressing from coarse stability
constraints to fine-grained joint coordination. Experimental results across locomotion and manipulation tasks show that MoVLR consistently
outperforms both human-designed and language-based baselines. The iterative refinement of rewards mirrors the hierarchical structure of human motor learning, progressing from coarse stability
constraints to fine-grained joint coordination. Beyond performance gains, MoVLR highlights a fundamental connection between explicit language intent and implicit reward emergence. Acting as a perceptual bridge, the VLM grounds
linguistic goals in physical dynamics, producing interpretable and dynamically consistent reward
representations. Beyond performance gains, MoVLR highlights a fundamental connection between explicit language intent and implicit reward emergence. Acting as a perceptual bridge, the VLM grounds
linguistic goals in physical dynamics, producing interpretable and dynamically consistent reward
representations. Beyond performance gains, MoVLR highlights a fundamental connection between explicit language intent and implicit reward emergence. Acting as a perceptual bridge, the VLM grounds
linguistic goals in physical dynamics, producing interpretable and dynamically consistent reward
representations. Beyond performance gains, MoVLR highlights a fundamental connection between explicit language intent and implicit reward emergence. Acting as a perceptual bridge, the VLM grounds
linguistic goals in physical dynamics, producing interpretable and dynamically consistent reward
representations. Beyond performance gains, MoVLR highlights a fundamental connection between explicit language intent and implicit reward emergence. Acting as a perceptual bridge, the VLM grounds
linguistic goals in physical dynamics, producing interpretable and dynamically consistent reward
representations. Beyond performance gains, MoVLR highlights a fundamental connection between explicit language intent and implicit reward emergence. Acting as a perceptual bridge, the VLM grounds
linguistic goals in physical dynamics, producing interpretable and dynamically consistent reward
representations. Beyond performance gains, MoVLR highlights a fundamental connection between explicit language intent and implicit reward emergence. Acting as a perceptual bridge, the VLM grounds
linguistic goals in physical dynamics, producing interpretable and dynamically consistent reward
representations. Beyond performance gains, MoVLR highlights a fundamental connection between explicit language intent and implicit reward emergence. Acting as a perceptual bridge, the VLM grounds
linguistic goals in physical dynamics, producing interpretable and dynamically consistent reward
representations.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Embodied Learning of Reward for Musculoskeletal Control: The system can discover
explicit reward terms that represent implicit musculoskeletal dynamics
by iteratively exploring reward space throughiterative interaction between control optimization and VLM feedback.
-
Bridging Perception and Dynamics: The improved AI can translate abstract motion descriptions into structured guidance for embodied learning, grounding
temporal dynamics in semantically meaningful descriptors
to bridge perceptual feedback with domain-informed motion representations. -
Multi-modal Reward Refinement: The system can refine reward functions by incorporating both
video observations of policy-executed trajectories
and VLM assessments, enabling it to produce reward terms that capturenonlinear couplings between balance, posture, and coordination.
-
Scalable Reward Discovery: MoVLR provides a framework for
automatic reward learning in high-dimensional musculoskeletal systems
that generalizes acrossmovement tasks, environments and system morphologies,
producing interpretable reward terms.
Abstract
Discovering effective reward functions remains a fundamental challenge in motor control of high-dimensional musculoskeletal systems. While humans can describe movement goals explicitly such as "walking forward with an upright posture," the underlying control strategies that realize these goals are largely implicit, making it difficult to directly design rewards from high-level goals and natural language descriptions. We introduce Motion from Vision-Language Representation (MoVLR), a framework that leverages vision-language models (VLMs) to bridge the gap between goal specification and movement control. Rather than relying on handcrafted rewards, MoVLR iteratively explores the reward space through iterative interaction between control optimization and VLM feedback, aligning control policies with physically coordinated behaviors. Our approach transforms language and vision-based assessments into structured guidance for embodied learning, enabling the discovery and refinement of reward functions for high-dimensional musculoskeletal locomotion and manipulation. These results suggest that VLMs can effectively ground abstract motion descriptions in the implicit principles governing physiological motor control.
Sources
- PaLM-E: An Embodied Multimodal Language Model
- Gemini: A Family of Highly Capable Multimodal Models
- Qwen2.5 Technical Report
Related papers
- Rethinking Actions: Continuity-Aware Priors for World-Model Planning
- MOSAIC: Modular Obstacle and Spatial-Aware Framework for Manipulation In Cluttered Scenes
- SAGAS: Semantic-Aware Graph-Assisted Stitching for Offline Temporal Logic Planning
- DexSeed: Expanding Human Hand-Object Interactions into Robot Demonstrations for Dexterous Policy Learning
- LaST0.5: A Generalist World Action Model with Interaction-Centric Latent Reasoning
- CIALLO: Learning Shared Intent Representations for Cross-Embodiment Manipulation