Should the Action Expert Train the VLM? Action-Token Supervision and Gradient Routing in Mixture-of-Transformer VLAs
cs.RO
Submitted: 2026-11-03
Updated: 2026-11-03
Code: https://github.com/NVlabs/alpasim
Project page: https://mot-vla-gradient-routing.github.io/ABSTRACT
Terminology
Sources
- PaliGemma: A versatile 3B VLM for transfer
- $\pi_0$: A Vision-Language-Action Flow Model for General Robot Control
- RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control
- Microsoft COCO Captions: Data Collection and Evaluation Server
- Diffusion Policy: Visuomotor Policy Learning via Action Diffusion
- MolmoAct2: Action Reasoning Models for Real-world Deployment
- Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering
- Enhancing Generalization in Vision-Language-Action Models by Preserving Pretrained Representations
- Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipes
- Actions as Language: Fine-Tuning VLMs into VLAs Without Catastrophic Forgetting
- EMMA: End-to-End Multimodal Model for Autonomous Driving
- ${\pi}_{0.7}$: a Steerable Generalist Robotic Foundation Model with Emergent Capabilities
- PolaRiS: Scalable Real-to-Sim Evaluations for Generalist Robot Policies
- DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset
- Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success
- Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
- Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution
- Scaling Verification Can Be More Effective than Scaling Policy Learning for Vision-Language-Action Alignment
- CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulation
- UniDriveVLA: Unifying Understanding, Perception, and Action Planning for Autonomous Driving
Related papers
- Rethinking Actions: Continuity-Aware Priors for World-Model Planning
- MOSAIC: Modular Obstacle and Spatial-Aware Framework for Manipulation In Cluttered Scenes
- SAGAS: Semantic-Aware Graph-Assisted Stitching for Offline Temporal Logic Planning
- DexSeed: Expanding Human Hand-Object Interactions into Robot Demonstrations for Dexterous Policy Learning
- LaST0.5: A Generalist World Action Model with Interaction-Centric Latent Reasoning
- CIALLO: Learning Shared Intent Representations for Cross-Embodiment Manipulation