CLAP: Direct VLM-to-VLA Adaptation via Language-Action Grounding
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.
Rosa: We're looking at the paper: "CLAP: Direct VLM-to-VLA Adaptation via Language-Action Grounding".
Dev: Here's the gist of the paper. Vision-language-action models (VLAs) inherit semantic capabilities from pretrained VLMs, yet large-scale post-training on robot data and architectural modifications can reshape the backbone so extensively that it becomes difficult to isolate what the VLM contributes to control The core obstacle is output-distribution mismatch: predicting actions as bare numeric token sequences moves generation away from the VLM’s pretrained language distribution, degrading the capabilities we seek to preserve The Proposed Solution CLAP (Causal Language-Action Prediction) is proposed to address this mismatch by prepending each numeric action sequence with a natural-language action description, causally conditioning precise action-token prediction on a language-action plan without modifying the backbone architecture This approach reformulates action prediction as language-conditioned action generation by generating both in a single autoregressive sequence: before predicting discrete numeric action tokens, the model first generates a natural-language action description Because the full sequence is generated autoregressively, causal attention conditions the numeric action tokens on the preceding language description, making it a conditioning intermediate rather than an auxiliary prediction CLAP requires no action expert, vocabulary modification, architectural change, or manual annotation while retaining executable numeric action tokens for precise control Learning Efficiency and Performance Gains The evaluation of CLAP across compact Qwen3.5 backbones at 0.8B, 2B, and 4B parameters demonstrates significant learning efficiency improvements On LIBERO, 2B CLAP achieves 90.8% success with a single fine-tuning epoch, outperforming VLA-0 by +14.9 percentage points under identical conditions The gains are broad across task suites: 2B CLAP improves Spatial from 77.8% to 93.0%, Object from 86.6% to 97.4%, Goal from 77.0% to 90.8%, and Long from 62.4% to 82.0 These gains come from changing the output representation rather than adding data, model capacity, or architectural components Robustness and Generalization Analysis CLAP improves robustness under instruction, object, and spatial perturbations on LIBERO-PRO Furthermore, CLAP consistently improves over VLA-0 across model scales with average OOD gains of +5.5, +11.1, and +10.9 points for unmasked CLAP at 0.8B, 2B, and 4B The most striking improvement appears on the Spatial suite with novel visual instances for 4B models, where unmasked CLAP improves over VLA-0 by +42.6 pt Inference and Deployment Considerations Inference latency is dominated by token generation overhead rather than model size at the 0.8B and 2B scales, with CLAP adding approximately 1.0 s per query (+32%) compared to VLA-0 GPU memory consumption remains identical, confirming that the prefix introduces no architectural overhead Real-World Validation On a UR5e manipulator, CLAP 2B achieves the highest success rate (60%) in real-robot pick-and-place tasks under both in-domain and out-of-domain conditions The results suggest that the larger backbone is more robust to visual distractors, as the 2B model maintains its in-domain success rate (60%) while the 0.8B model drops substantially to 10% under OOD conditions The main contribution is CLAP: a minimal VLM-to-VLA recipe that changes only the output representation, reaching ≈ 90% success rate on LIBERO within 6.
Paper summary: Rosa: So we’ve talked about what CLAP is and how they structured their proposal, focusing on why this direct VLM-to-VLA adaptation matters for understanding model transfer across scales. The paper explains that the core issue they are tackling is the output distribution mismatch that happens when you move from predicting bare numeric action tokens to a language-action plan.
Dev: They argue that predicting actions as bare numeric sequences moves generation away from the VLM’s original language distribution, which hurts those semantic capabilities we want to preserve during fine-tuning on robot data >
Taro: The authors claim that their solution, CLAP, addresses this mismatch by prepending a natural language action description to each numeric action sequence without changing the underlying architecture >
Rosa: They call it Causal Language-Action Prediction because it reformulates the task as generating a whole autoregressive sequence where the numeric tokens are conditioned on that preceding language description >
Dev: The key claim is that this approach keeps every prediction step much closer to what the VLM was originally trained to do, which means we’re preserving those useful representations >
Taro: They argue this works because it grounds the action prediction in natural language, which helps with intermediate reasoning and semantic structure during the control process >
Rosa: Basically, they show that this technique requires no separate action expert or any major architectural modifications to make it work >
Dev: That minimal change is what makes it a transparent path to understanding how those VLM capabilities transfer across different model scales when you move from massive pretraining down to deployment >
Conclusion: Rosa: So looking at the paper, "CLAP: Direct VLM-to-VLA Adaptation via Language-Action Grounding," the main contribution seems to be this minimal recipe for adapting VLMs directly into deployable agents. It’s about fixing that distribution mismatch by just changing the output format >
Dev: And they achieved that by proving it works without needing a separate action expert or any major architectural redesign, which is a significant finding if you’re looking at transparent paths to understanding model transfer >
Taro: The authors suggest this approach provides cleaner settings for studying exactly what happens when you fine-tuning compact VLAs, which are often more sensitive to distribution shift issues than larger models >
Rosa: So in simple terms, the implication is that we can potentially get better alignment between vision language understanding and robot action with less plumbing involved >
Dev: It suggests that for a lot of VLA work, the focus should be on how you represent the action output, rather than building entirely new control mechanisms >
Yuri Ishitoya, Jeremy Siburian, Masashi Hamaya, Kuniaki Saito, Cristian C. Beltran-Hernandez, Mai Nishimura
Ochanomizu University · University of Tokyo · ÉMRON SINIC X Corp.
cs.RO, cs.AI
Submitted: 2026-07-09
Updated: 2026-11-03
Comments: Accepted to CoRL 2026. Project page: https://omron-sinicx.github.io/clap
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Importance score: 93/100
The gist: Vision-language-action models (VLAs) inherit semantic capabilities from pretrained VLMs, yet large-scale post-training on robot data and architectural modifications can reshape the backbone so…
Key concepts
- Output-distribution mismatch
- This occurs when models are trained to predict simple numeric sequences (like action tokens) instead of natural language. This forces the model's output distribution away from what it learned during pretraining, degrading its ability to perform complex control tasks.
- CLAP (Causal Language-Action Prediction)
- CLAP is a method that fixes the mismatch by generating both a text description and the action tokens autoregressively. The language description acts as a condition, guiding the model to predict precise numeric actions based on the preceding natural language plan.
- Language-conditioned action generation
- Instead of treating action prediction as an independent task, CLAP treats it as generating a sequence conditioned on a language description. This means the model uses its learned linguistic understanding to inform and constrain the specific numeric actions it outputs, improving performance.
- Output representation change
- The core innovation is changing *how* the output is structured—by combining language and action tokens into one autoregressive sequence. This allows for causal attention to link the descriptive language directly to the subsequent action predictions, achieving better grounding.
Terminology
Summary
Vision-language-action models (VLAs) inherit semantic capabilities from pretrained VLMs, yet large-scale post-training on robot data and architectural modifications can reshape the backbone so extensively that it becomes difficult to isolate what the VLM contributes to control The core obstacle is output-distribution mismatch: predicting actions as bare numeric token sequences moves generation away from the VLM’s pretrained language distribution, degrading the capabilities we seek to preserve
The Proposed Solution
CLAP (Causal Language-Action Prediction) is proposed to address this mismatch by prepending each numeric action sequence with a natural-language action description, causally conditioning precise action-token prediction on a language-action plan without modifying the backbone architecture This approach reformulates action prediction as language-conditioned action generation by generating both in a single autoregressive sequence: before predicting discrete numeric action tokens, the model first generates a natural-language action description Because the full sequence is generated autoregressively, causal attention conditions the numeric action tokens on the preceding language description, making it a conditioning intermediate rather than an auxiliary prediction CLAP requires no action expert, vocabulary modification, architectural change, or manual annotation while retaining executable numeric action tokens for precise control
Learning Efficiency and Performance Gains
The evaluation of CLAP across compact Qwen3.5 backbones at 0. p. 18B, 2B, and 4B parameters demonstrates significant learning efficiency improvements On LIBERO, 2B CLAP achieves 90. p. 18% success with a single fine-tuning epoch, outperforming VLA-0 by +14. p. 19 percentage points under identical conditions The gains are broad across task suites: 2B CLAP improves Spatial from 77. p. 58% to 93.0%, Object from 86. p. 06% to 97.4%, Goal from 77. p. 30% to 90.8%, and Long from 62. p. 44% to 82.0 These gains come from changing the output representation rather than adding data, model capacity, or architectural components
Robustness and Generalization Analysis
CLAP improves robustness under instruction, object, and spatial perturbations on LIBERO-PRO Furthermore, CLAP consistently improves over VLA-0 across model scales with average OOD gains of +5.5, +11.1, and +10.9 points for unmasked CLAP at 0. p. 68B, 2B, and 4B The most striking improvement appears on the Spatial suite with novel visual instances for 4B models, where unmasked CLAP improves over VLA-0 by +42. p. 66 pt
Inference and Deployment Considerations
Inference latency is dominated by token generation overhead rather than model size at the 0.8B and 2B scales, with CLAP adding approximately 1. p. 20 s per query (+32%) compared to VLA-0 GPU memory consumption remains identical, confirming that the prefix introduces no architectural overhead
Real-World Validation
On a UR5e manipulator, CLAP 2B achieves the highest success rate (60%) in real-robot pick-and-place tasks under both in-domain and out-of-domain conditions The results suggest that the larger backbone is more robust to visual distractors, as the 2B model maintains its in-domain success rate (60%) while the 0.8B model drops substantially to 10% under OOD conditions
Conclusion and Future Directions
The main contribution is CLAP: a minimal VLM-to-VLA recipe that changes only the output representation, reaching ≈ 90% success rate on LIBERO within 6.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Improve VLM-to-VLA adaptation by using CLAP to prepends each numeric action sequence with a natural-language action description, causally conditioning precise action-token prediction on a language-action plan without modifying the backbone architecture. This allows for
a more transparent path to understanding how VLM capabilities transfer across model scales.
-
Enable single-epoch fine-tuning of compact VLAs to achieve high performance, as demonstrated by CLAP reaching 90.8% on LIBERO with a single epoch and outperforming VLA-0 by +14.9 percentage points at the 2B parameter scale.
-
Enhance robustness in VLA systems under instruction, object, and spatial perturbations on benchmarks like LIBERO-PRO because CLAP improves robustness under these conditions compared to standard VLA fine-tuning.
-
Enable controlled analysis of VLM-to-VLA capability transfer across compact scales (0.8B, 2B, and 4B) by applying the same architecture-preserving recipe across different model sizes without adding action experts or vocabulary modifications.
-
Increase out-of-distribution robustness on LIBERO-PRO by using CLAP with action masking as an optional augmentation, which
substantially enhances OOD robustness at 4B scale
and suggests thatmasking can reduce reliance on action-token shortcuts that become brittle under distribution shift.
-
Enable better task planning for compact VLAs by avoiding reliance on free-form reasoning at inference time, as CLAP's structured language-action template provides a
compact intermediate representation before numeric action prediction, keeping smaller variants competitive.
Sources
- GR00T N1: An Open Foundation Model for Generalist Humanoid Robots
- LAP: Language-Action Pre-Training Enables Zero-shot Cross-Embodiment Transfer
- SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Robotics
- VLA-0: Building State-of-the-Art VLAs with Zero Modification
- Actions as Language: Fine-Tuning VLMs into VLAs Without Catastrophic Forgetting
- PaliGemma: A versatile 3B VLM for transfer
- RT-1: Robotics Transformer for Real-World Control at Scale
- Octo: An Open-Source Generalist Robot Policy
- FAST: Efficient Action Tokenization for Vision-Language-Action Models
- PaLM-E: An Embodied Multimodal Language Model
- Enhancing Generalization in Vision-Language-Action Models by Preserving Pretrained Representations
- LIBERO-PRO: Towards Robust and Fair Evaluation of Vision-Language-Action Models Beyond Memorization
- Don't Blink: Evidence Collapse during Multimodal Reasoning
Related papers
- Rethinking Actions: Continuity-Aware Priors for World-Model Planning
- MOSAIC: Modular Obstacle and Spatial-Aware Framework for Manipulation In Cluttered Scenes
- SAGAS: Semantic-Aware Graph-Assisted Stitching for Offline Temporal Logic Planning
- DexSeed: Expanding Human Hand-Object Interactions into Robot Demonstrations for Dexterous Policy Learning
- LaST0.5: A Generalist World Action Model with Interaction-Centric Latent Reasoning
- CIALLO: Learning Shared Intent Representations for Cross-Embodiment Manipulation