Recursive Video In-Context Learning for Agentic Robot
cs.RO, cs.AI, cs.CL, cs.MA
Submitted: 2026-10-05
Updated: 2026-10-05
License: http://creativecommons.org/licenses/by/4.0/
The gist: LLM agents that orchestrate frozen vision-language-action (VLA) policies improve across episodes through text memory, which records what the agent did but not how the task is done.
Terminology
Abstract
LLM agents that orchestrate frozen vision-language-action (VLA) policies improve across episodes through text memory, which records what the agent did but not how the task is done. A demonstration video shows it, but fits poorly into an agent's context. The full video slows every turn, fixed keyframes lose the contact detail that decides whether a grasp holds, and what the agent needs shifts from the task's structure while planning to the frames around each contact. We introduce Recursive Video In-Context Learning (RV-ICL), a training-free method that turns a demonstration into a hierarchy the agent navigates rather than a prompt it receives. The hierarchy is built from the sub-events of the demonstration, such as grasps and releases. Its levels grow finer, from keyframes of the whole task to phases, moments and short clips, and are exposed through read-only tools. The agent reads the coarse levels before planning. During execution it re-enters the hierarchy whenever a step needs more detail and loads only the clip of its current sub-goal. One demonstration per task is enough. Built on RPent, RV-ICL raises success from 92.6% to 96.5% on LIBERO-PRO and from 86.7% to 95.8% on LIBERO-Plus.
Sources
- Keypoint Action Tokens Enable In-Context Imitation Learning in Robotics
- LIBERO-Plus: In-depth Robustness Analysis of Vision-Language-Action Models
- In-Context Imitation Learning via Next-Token Prediction
- CaP-X: A Framework for Benchmarking and Improving Coding Agents for Robot Manipulation
- NORA: A Small Open-Sourced Generalist Vision Language Action Model for Embodied Tasks
- Vid2Robot: End-to-end Video-conditioned Policy Learning with Cross-Attention Transformers
- OpenVLA: An Open-Source Vision-Language-Action Model
- MolmoAct: Action Reasoning Models that can Reason in Space
- Robo-Harness K1: Harnessing Robot-Use Agents via Perception Augmentation
- RAPID: Robot Agentic Programming from Demonstrations
- MemGPT: Towards LLMs as Operating Systems
- $\pi_{0.5}$: a Vision-Language-Action Model with Open-World Generalization
- AtomVLA: Scalable Post-Training for Robotic Manipulation via Predictive Latent World Models
- Voyager: An Open-Ended Embodied Agent with Large Language Models
- Recursively Summarizing Books with Human Feedback
- StellaVLA: In-Context Structured Demonstration for Generalizable Vision-Language-Action Models
- RLinf: Flexible and Efficient Large-scale Reinforcement Learning via Macro-to-Micro Flow Transformation
- Recursive Language Models
- Playful Agentic Robot Learning
- An Unexpected Robot Policy: Early Evaluations of GPT-6 Astra on RoboDojo and Beyond
Related papers
- FMT x: An Efficient and Asymptotically Optimal Extension of the Fast Marching Tree for Dynamic Replanning
- MPCFormer: A physics-informed data-driven approach for explainable socially-aware autonomous driving
- RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies
- ARC: A Reasoning Recipe for Robot Foundation Models
- HRDexDB: A 4D Dexterous Grasping Dataset Across Human and Multiple Robot Embodiments
- APT: Action Expert Pretraining Improves Instruction Generalization of Vision-Language-Action Policies