FineVLA: Fine-Grained Instruction Alignment for Steerable Vision-Language-Action Policies
cs.RO, cs.AI
Submitted: 2026-05-26
Updated: 2026-09-02
Comments: 26 pages, 7 figures, 25 tables
Code: https://github.com/NVIDIA/Isaac-GR00T
Project page: https://finevla.xlang.ai
License: http://creativecommons.org/licenses/by-sa/4.0/
The gist: Vision-Language-Action (VLA) models are increasingly expected to not only complete robot tasks, but also follow human instructions about how those tasks should be executed.
Terminology
Abstract
Vision-Language-Action (VLA) models are increasingly expected to not only complete robot tasks, but also follow human instructions about how those tasks should be executed. However, existing robot datasets usually pair trajectories with coarse goal-level language, leaving execution-critical details such as active arm, approach direction, and contact region unspecified. This limits steerable policy learning and robotic video understanding. We introduce FineVLA, an open framework for action-aligned fine-grained VLA supervision. The framework includes: (1) a data construction tool that unifies 972,247 trajectories across 85K tasks from 10 open-source robot datasets and builds FineVLA-Data, a human-verified dataset of 47,159 fine-grained trajectories; (2) a held-out benchmark with 500 videos, 11,631 atomic facts, and 1,030 VQA questions; (3) a robotics-specialized VLM annotator for scalable fine-grained annotation; and (4) a steerable VLA policy trained with controlled mixtures of fine-grained and raw goal-level instructions. Our experiments yield three findings. First, fine-grained supervision does not sacrifice goal-level success: FG-only improves over Raw-only by +1.4 to +8.1 success-rate points across settings. Second, fine-grained and raw instructions are complementary, following a consistent inverted-U trend peaking at FG:Raw = 1:2 to 1:1. The best mixed setting reaches 86.8%/82.5% in RoboTwin simulation and 62.7/100 in real-world dual-arm manipulation (vs. 49.9 Raw-only). Third, fine-grained supervision improves steerable control: the largest real-world gains appear on pose (+23), color (+18), and approach direction (+18)--factors where goal-level instructions provide no guidance. Overall, fine-grained language should augment goal-level instructions: specifying how to execute alongside what to achieve. Project page: https://finevla.xlang.ai/
Sources
- ${\pi}_{0.7}$: a Steerable Generalist Robotic Foundation Model with Emergent Capabilities
- A Pragmatic VLA Foundation Model
- Open X-Embodiment: Robotic Learning Datasets and RT-X Models
- RDT-1B: a Diffusion Foundation Model for Bimanual Manipulation
- RoboVQA: Multimodal Long-Horizon Reasoning for Robotics
- HanDyVQA: A Video QA Benchmark for Fine-Grained Hand-Object Interaction Dynamics
- BridgeData V2: A Dataset for Robot Learning at Scale
- BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning
- RT-1: Robotics Transformer for Real-World Control at Scale
- Galaxea Open-World Dataset and G0 Dual-System VLA Model
- RoboMIND 2.0: A Multimodal, Bimanual Mobile Manipulation Dataset for Generalizable Embodied Intelligence
- RoboCOIN: An Open-Sourced Bimanual Robotic Data Collection for Integrated Manipulation
- RH20T: A Comprehensive Robotic Dataset for Learning Diverse Skills in One-Shot
- DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset
- StarVLA: A Lego-like Codebase for Vision-Language-Action Model Developing
- RoboTwin: Dual-Arm Robot Benchmark with Generative Digital Twins (early version)
- RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control
- OpenVLA: An Open-Source Vision-Language-Action Model
- $\pi_0$: A Vision-Language-Action Flow Model for General Robot Control
- Octo: An Open-Source Generalist Robot Policy
Related papers
- FMT x: An Efficient and Asymptotically Optimal Extension of the Fast Marching Tree for Dynamic Replanning
- MPCFormer: A physics-informed data-driven approach for explainable socially-aware autonomous driving
- RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies
- ARC: A Reasoning Recipe for Robot Foundation Models
- HRDexDB: A 4D Dexterous Grasping Dataset Across Human and Multiple Robot Embodiments
- APT: Action Expert Pretraining Improves Instruction Generalization of Vision-Language-Action Policies