EmbodiedMidtrain: Bridging the Gap between Vision-Language Models and Vision-Language-Action Models via Mid-training
cs.CV, cs.AI, cs.CL
Submitted: 2026-04-21
Updated: 2026-10-15
License: http://creativecommons.org/licenses/by/4.0/
The gist: Vision-Language-Action Models (VLAs) inherit their visual and linguistic capabilities from Vision-Language Models (VLMs), yet most VLAs are built from off-the-shelf VLMs that are not adapted to the
Terminology
Abstract
Vision-Language-Action Models (VLAs) inherit their visual and linguistic capabilities from Vision-Language Models (VLMs), yet most VLAs are built from off-the-shelf VLMs that are not adapted to the embodied domain, limiting their downstream performance. In this work, we propose EmbodiedMidtrain to bridge the gap between VLMs and VLAs. We first characterize the data distribution gap between them, showing that VLA data occupy compact regions that are largely separated from the broader VLM distribution, while the degree of alignment varies substantially both across and within VLM data sources. Then, we build a mid-training data engine that leverages a lightweight learnable proximity estimator to select the most VLA-aligned candidates from a large VLM pool, and produce EmbodiedMidtrain-1M, a VLM dataset that we further use to mid-train the VLM before downstream VLA fine-tuning. Experiments on both simulated robot manipulation benchmarks and real-world tasks show consistent VLA performance gains over off-the-shelf checkpoints across VLM backbones and VLA action heads. These gains extend to downstream tasks not used to guide data selection. Further analysis reveals that mid-training provides a stronger initialization for VLA fine-tuning, with gains emerging from the earliest steps and widening throughout training. Moreover, the data engine captures both dataset-level and sample-level alignment signals, favoring spatial reasoning over text-centric tasks while preserving the diversity of the VLM data. We will release all code, data and models for future research.
Sources
- Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond
- Qwen3-VL Technical Report
- Qwen2.5-VL Technical Report
- PaliGemma: A versatile 3B VLM for transfer
- $\pi_0$: A Vision-Language-Action Flow Model for General Robot Control
- SpatialBot: Precise Spatial Understanding with Vision Language Models
- Robo2VLM: Visual Question Answering from Large-Scale In-the-Wild Robot Manipulation Datasets
- InternVLA-M1: A Spatially Guided Vision-Language-Action Framework for Generalist Robot Policy
- Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling
- BLINK: Multimodal Large Language Models Can See but Not Perceive
- Gemini Robotics 1.5: Pushing the Frontier of Generalist Robots with Advanced Embodied Reasoning, Thinking, and Motion Transfer
- The Llama 3 Herd of Models
- MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies
- RoboBrain: A Unified Brain Model for Robotic Manipulation from Abstract to Concrete
- OpenVLA: An Open-Source Vision-Language-Action Model
- Contrastive Representation Regularization for Vision-Language-Action Models
- Eagle 2: Building Post-Training Data Strategies from Scratch for Frontier Vision-Language Models
- LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning
- GR00T N1: An Open Foundation Model for Generalist Humanoid Robots
- 2 OLMo 2 Furious
Related papers
- Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
- AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
- Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
- MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
- TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
- A Survey on Efficient Vision-Language-Action Models