R cubed: Training Robots to Reason in Natural Language via Reinforcement Learning
cs.RO, cs.AI, cs.CL, cs.LG
Submitted: 2026-08-26
Updated: 2026-08-26
Comments: 42 pages, 23 figures
Project page: https://robotic-reasoner.github.io
License: http://creativecommons.org/licenses/by/4.0/
The gist: Reasoning in language allows foundation models to spend more test-time compute on hard problems, such as those requiring decomposition, constraint tracking, and prediction of future consequences.
Terminology
Abstract
Reasoning in language allows foundation models to spend more test-time compute on hard problems, such as those requiring decomposition, constraint tracking, and prediction of future consequences. Whether this mechanism can improve robotic manipulation remains unclear, where long-horizon tasks require tracking partial progress, reasoning about object relations, recovering from mistakes, and steering noisy low-level policies. In this paper, we study whether VLMs can be trained to reason directly in natural language to guide low-level manipulation policies. We introduce R cubed, a simple post-training recipe that turns off-the-shelf VLMs into robotic reasoners: it first mid-trains a VLM on expert-generated reasoning traces to initialize the desired reasoning style, then improves the reasoner with single-step rubric-based RL from offline action data. Unlike prior robotic reasoning methods that mostly use structured traces as auxiliary supervision, R cubed trains free-form language reasoning to produce test-time guidance for action. We instantiate R cubed on Language Table and simulated bimanual grocery packing, two controlled testbeds for studying robotic reasoning and long-horizon manipulation. R cubed improves exploration and generalization across unseen tasks and significantly outperforms instruction-only imitation learning baselines on both benchmarks. Our analyses suggest that free-form language reasoning can function as a test-time compute mechanism for steering low-level policies. Our project page is available at https://robotic-reasoner.github.io/.
Sources
- Do As I Can, Not As I Say: Grounding Language in Robotic Affordances
- GR00T N1: An Open Foundation Model for Generalist Humanoid Robots
- RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control
- Benchmarking in Manipulation Research: The YCB Object and Model Set and Benchmarking Protocols
- GR-3 Technical Report
- Training Strategies for Efficient Embodied Reasoning
- See, Think, Confirm: Interactive Prompting Between Vision and Language Models for Knowledge-based Visual Reasoning
- Self-Correcting Code Generation Using Small Language Models
- Action-Free Reasoning for Policy Generalization
- RACER: Rich Language-Guided Failure Recovery Policies for Imitation Learning
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- MUSE: Benchmarking Assembly Structure and Interface Geometry in Multi-Component Text-to-CAD Generation
- PaLM-E: An Embodied Multimodal Language Model
- Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control
- Step-CoT: Stepwise Visual Chain-of-Thought for Medical Visual Question Answering
- MolmoAct2: Action Reasoning Models for Real-world Deployment
- ProcVLM: Learning Procedure-Grounded Progress Rewards for Robotic Manipulation
- SteerVLA: Steering Vision-Language-Action Models in Long-Tail Driving Scenarios
- Integrated Task and Motion Planning
- RaC: Robot Learning for Long-Horizon Tasks by Scaling Recovery and Correction
Related papers
- FMT x: An Efficient and Asymptotically Optimal Extension of the Fast Marching Tree for Dynamic Replanning
- MPCFormer: A physics-informed data-driven approach for explainable socially-aware autonomous driving
- RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies
- ARC: A Reasoning Recipe for Robot Foundation Models
- HRDexDB: A 4D Dexterous Grasping Dataset Across Human and Multiple Robot Embodiments
- APT: Action Expert Pretraining Improves Instruction Generalization of Vision-Language-Action Policies