CLAP: Direct VLM-to-VLA Adaptation via Language-Action Grounding

summary

The gist

Vision-language-action models (VLAs) inherit semantic capabilities from pretrained VLMs, yet large-scale post-training on robot data and architectural modifications can reshape the backbone so…

In short

Vision-language-action models struggle because predicting raw numeric actions loses the semantic knowledge from their language pretraining. CLAP solves this by prepending a natural language action description to numeric tokens, conditioning the action prediction on this plan. This reformulation allows the model to leverage its language understanding for precise control without changing the core architecture.

Key concepts

Output-distribution mismatch
This occurs when models are trained to predict simple numeric sequences (like action tokens) instead of natural language. This forces the model's output distribution away from what it learned during pretraining, degrading its ability to perform complex control tasks.
CLAP (Causal Language-Action Prediction)
CLAP is a method that fixes the mismatch by generating both a text description and the action tokens autoregressively. The language description acts as a condition, guiding the model to predict precise numeric actions based on the preceding natural language plan.
Language-conditioned action generation
Instead of treating action prediction as an independent task, CLAP treats it as generating a sequence conditioned on a language description. This means the model uses its learned linguistic understanding to inform and constrain the specific numeric actions it outputs, improving performance.
Output representation change
The core innovation is changing *how* the output is structured—by combining language and action tokens into one autoregressive sequence. This allows for causal attention to link the descriptive language directly to the subsequent action predictions, achieving better grounding.

Terminology used across episodes

This episode discusses

The paper

CLAP: Direct VLM-to-VLA Adaptation via Language-Action Grounding · Read on arXiv

Yuri Ishitoya, Jeremy Siburian, Masashi Hamaya, Kuniaki Saito, Cristian C. Beltran-Hernandez, Mai Nishimura

Ochanomizu University · University of Tokyo · ÉMRON SINIC X Corp.

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: We're looking at the paper: "CLAP: Direct VLM-to-VLA Adaptation via Language-Action Grounding".

Dev: Here's the gist of the paper. Vision-language-action models (VLAs) inherit semantic capabilities from pretrained VLMs, yet large-scale post-training on robot data and architectural modifications can reshape the backbone so extensively that it becomes difficult to isolate what the VLM contributes to control The core obstacle is output-distribution mismatch: predicting actions as bare numeric token sequences moves generation away from the VLM’s pretrained language distribution, degrading the capabilities we seek to preserve The Proposed Solution CLAP (Causal Language-Action Prediction) is proposed to address this mismatch by prepending each numeric action sequence with a natural-language action description, causally conditioning precise action-token prediction on a language-action plan without modifying the backbone architecture This approach reformulates action prediction as language-conditioned action generation by generating both in a single autoregressive sequence: before predicting discrete numeric action tokens, the model first generates a natural-language action description Because the full sequence is generated autoregressively, causal attention conditions the numeric action tokens on the preceding language description, making it a conditioning intermediate rather than an auxiliary prediction CLAP requires no action expert, vocabulary modification, architectural change, or manual annotation while retaining executable numeric action tokens for precise control Learning Efficiency and Performance Gains The evaluation of CLAP across compact Qwen3.5 backbones at 0.8B, 2B, and 4B parameters demonstrates significant learning efficiency improvements On LIBERO, 2B CLAP achieves 90.8% success with a single fine-tuning epoch, outperforming VLA-0 by +14.9 percentage points under identical conditions The gains are broad across task suites: 2B CLAP improves Spatial from 77.8% to 93.0%, Object from 86.6% to 97.4%, Goal from 77.0% to 90.8%, and Long from 62.4% to 82.0 These gains come from changing the output representation rather than adding data, model capacity, or architectural components Robustness and Generalization Analysis CLAP improves robustness under instruction, object, and spatial perturbations on LIBERO-PRO Furthermore, CLAP consistently improves over VLA-0 across model scales with average OOD gains of +5.5, +11.1, and +10.9 points for unmasked CLAP at 0.8B, 2B, and 4B The most striking improvement appears on the Spatial suite with novel visual instances for 4B models, where unmasked CLAP improves over VLA-0 by +42.6 pt Inference and Deployment Considerations Inference latency is dominated by token generation overhead rather than model size at the 0.8B and 2B scales, with CLAP adding approximately 1.0 s per query (+32%) compared to VLA-0 GPU memory consumption remains identical, confirming that the prefix introduces no architectural overhead Real-World Validation On a UR5e manipulator, CLAP 2B achieves the highest success rate (60%) in real-robot pick-and-place tasks under both in-domain and out-of-domain conditions The results suggest that the larger backbone is more robust to visual distractors, as the 2B model maintains its in-domain success rate (60%) while the 0.8B model drops substantially to 10% under OOD conditions The main contribution is CLAP: a minimal VLM-to-VLA recipe that changes only the output representation, reaching ≈ 90% success rate on LIBERO within 6.

Paper summary: Rosa: So we’ve talked about what CLAP is and how they structured their proposal, focusing on why this direct VLM-to-VLA adaptation matters for understanding model transfer across scales. The paper explains that the core issue they are tackling is the output distribution mismatch that happens when you move from predicting bare numeric action tokens to a language-action plan.

Dev: They argue that predicting actions as bare numeric sequences moves generation away from the VLM’s original language distribution, which hurts those semantic capabilities we want to preserve during fine-tuning on robot data >

Taro: The authors claim that their solution, CLAP, addresses this mismatch by prepending a natural language action description to each numeric action sequence without changing the underlying architecture >

Rosa: They call it Causal Language-Action Prediction because it reformulates the task as generating a whole autoregressive sequence where the numeric tokens are conditioned on that preceding language description >

Dev: The key claim is that this approach keeps every prediction step much closer to what the VLM was originally trained to do, which means we’re preserving those useful representations >

Taro: They argue this works because it grounds the action prediction in natural language, which helps with intermediate reasoning and semantic structure during the control process >

Rosa: Basically, they show that this technique requires no separate action expert or any major architectural modifications to make it work >

Dev: That minimal change is what makes it a transparent path to understanding how those VLM capabilities transfer across different model scales when you move from massive pretraining down to deployment >

Conclusion: Rosa: So looking at the paper, "CLAP: Direct VLM-to-VLA Adaptation via Language-Action Grounding," the main contribution seems to be this minimal recipe for adapting VLMs directly into deployable agents. It’s about fixing that distribution mismatch by just changing the output format >

Dev: And they achieved that by proving it works without needing a separate action expert or any major architectural redesign, which is a significant finding if you’re looking at transparent paths to understanding model transfer >

Taro: The authors suggest this approach provides cleaner settings for studying exactly what happens when you fine-tuning compact VLAs, which are often more sensitive to distribution shift issues than larger models >

Rosa: So in simple terms, the implication is that we can potentially get better alignment between vision language understanding and robot action with less plumbing involved >

Dev: It suggests that for a lot of VLA work, the focus should be on how you represent the action output, rather than building entirely new control mechanisms >

More episodes

← Home