2AM: Grounding Agent-Side Memory as Guidance for Steerable Action Models in Long-Horizon Manipulation
summary
The gist
Long-horizon robot manipulation requires memory, but not necessarily inside the action policy <ref:2609.11308#pg2>.
In short
The paper introduces 2AM, a system where an Agent holds task memory and an Action Model executes motion. The Agent translates history into subtask language and provides 2D hints (grasp, place, move) to guide the model. This setup shows that task memory can reside with the Agent side. Grounded steering significantly improves performance by helping the Action Model execute intended actions.
Key concepts
- 2AM
- A design where a multimodal Agent stores all task memory and generates steering commands. It uses this command to guide an episodically stateless Action Model, which is responsible only for executing physical motion based on the Agent's guidance.
- Steering Contract
- A compositional interface that turns remembered intent into actionable 2D guidance. The Agent creates a command (ck) containing language, grasp hints, place hints, and move hints. These hints are designed to bind the physical intention across different time scales of the task.
- Action Model
- A single RGB-based model that executes task-relevant motion. It is episodically stateless and relies entirely on the Agent's steering command for its behavior. This model handles low-level details like grasp closure, collision avoidance, and transport geometry.
- Grounded Steering
- The process of providing the Action Model with specific 2D hints (grasp, place, move) derived from the Agent's memory. This grounding helps the model understand its required physical actions more precisely than relying solely on high-level language instructions.
Terminology used across episodes
This episode discusses
- 2AM: Grounding Agent-Side Memory as Guidance for Steerable Action Models in Long-Horizon Manipulation · Paper Radio
- OpenVLA: An Open-Source Vision-Language-Action Model
- pi 0: A Vision-Language-Action Flow Model for General Robot Control
- pi 0.5: a Vision-Language-Action Model with Open-World Generalization
- AI Research Agents for Machine Learning: Search, Exploration, and Generalization in MLE-bench
- AIRA 2: Overcoming Bottlenecks in AI Research Agents
- MemoryVLA: Perceptual-Cognitive Memory in Vision-Language-Action Models for Robotic Manipulation
- RMBench: Memory-Dependent Robotic Manipulation Benchmark with Insights into Policy Design · Paper Radio
- RoboMME: Benchmarking and Understanding Memory for Robotic Generalist Policies
- MemER: Scaling Up Memory for Robot Control via Experience Retrieval
- Hi Robot: Open-Ended Instruction Following with Hierarchical Vision-Language-Action Models
- Agentic Robot: A Brain-Inspired Framework for Vision-Language-Action Models in Embodied Agents
- Goal2Skill: Long-Horizon Manipulation with Adaptive Planning and Reflection · Paper Radio
- Harness VLA: Steering Frozen VLAs into Reliable Manipulation Primitives via Memory-Guided Agents · Paper Radio
- Steerable Vision-Language-Action Policies for Embodied Reasoning and Hierarchical Control
- Rethinking Progression of Memory State in Robotic Manipulation: An Object-Centric Perspective
- SlotVLA: Towards Modeling of Object-Relation Representations in Robotic Manipulation · Paper Radio
- Do As I Can, Not As I Say: Grounding Language in Robotic Affordances
- Code as Policies: Language Model Programs for Embodied Control
- VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models
- ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulation
The paper
2AM: Grounding Agent-Side Memory as Guidance for Steerable Action Models in Long-Horizon Manipulation · Read on arXiv
Yutong Hu, Fengjiao Chen, Xuezhi Cao, Renaud Detry
KU Leuven · Meituan Inc.
Transcript
Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.
Rosa: We're looking at the paper: "2AM: Grounding Agent-Side Memory as Guidance for Steerable Action Models in Long-Horizon Manipulation".
Dev: Here's the gist of the paper. Long-horizon robot manipulation requires memory, but not necessarily inside the action policy <ref:2609.11308#pg2>.
Rosa: The Agent compiles interaction history into subtask language and optional 2D grasp, place, and move hints that bind its physical intention at different time scales <ref:2609.11308#pg3>.
Dev: Core Design Philosophy The paper introduces 2AM, a design where the multimodal Agent holds task memory and an episodically stateless Action Model executes task-relevant motion <ref:2609.11308#pg3>.
Rosa: This setup isolates attribution by making the Agent the sole holder of task memory and a single RGB-based, episodically stateless Action Model the sole executor of task-relevant motion <ref:2609.11308#pg3>.
Paper summary: Dev: So to wrap up this discussion on 2AM: Grounding Agent-Side Memory as Guidance for Steerable Action Models in Long-Horizon Manipulation, the authors argue that separating memory from the policy is a viable path forward <ref:2609.11308#pg1,2AM: Grounding Agent-Side Memory as Guidance for Steerable Action Models in>.
Rosa: They show that by making the Agent the sole holder of task memory and giving it subtask language and those grasp, place, and move hints, you create a single executor for motion—the Action Model.
Taro: I think what this means for future work is focusing on that steering contract. If we can get better at defining what those cues mean across different time scales, that’s where the real progress is going to be.
Dev: It confirms the idea that the Action Model's capability depends not only on what the policy has learned but on how precisely and clearly the Agent can steer it through that interface.
Rosa: That’s the big picture: task memory can stay in that higher-level Agent, and its usefulness comes from how well it translates into actionable guidance for the lower-level model.
Taro: So we’re looking at a system where you have an agent reasoning about the whole sequence, and a model that just needs to follow those precise, grounded hints for each local step.
Dev: It suggests that instead of trying to fit everything into one monolithic policy, maybe splitting it up like this—memory on one side, execution on another—is a more robust structure for long-horizon problems.
Conclusion: Rosa: So we've looked at how this two-agent system, called 2AM, works to solve long-horizon robot tasks by keeping the memory on one side and the action on the other <ref:2609.11308#pg1>.
Dev: Yeah it’s interesting because it separates what you’re remembering from what you’re actually doing. The paper says they put all that task history into a language and some 2D hints like where to grasp or place things later <ref:2609.11308#pg1>.
Taro: It means the robot agent is just thinking about the whole sequence, and the action model is just responsible for the immediate physical movements based on those hints.
Rosa: Exactly. And they show that this setup lets you isolate where things go wrong, whether it’s a bad memory or a bad execution plan.
Dev: They train it under a lot of noise like dropping data out or adding jitter so the system can handle when the agent makes slightly messy hints.
Taro: It’s about teaching the action model to be robust even when the Agent isn't perfectly precise with its instructions.
Rosa: And on this specific test, without using any extra depth cameras or a complex planner, 2AM still hits seventy-six point three percent success on their benchmark <ref:2609.11308#pg1>.
Dev: That’s a huge jump from what they reported before, which was barely over fifteen percent for the strongest baseline they looked at.
Taro: They also showed that just adding those grounded hints—the grasp and place information—can boost relaxed success by forty-three point five eight percent on their language test alone.
Rosa: So the conclusion is that this Agent-side memory can actually guide action much better than a traditional policy would.
Dev: But they also made it clear that even with all these hints, getting the robot to stop exactly where it needs to is still a challenge for now.
Taro: They suggest the next big thing would be unifying this memory-steering idea across different real-world testing setups, like different arenas.
Rosa: It sounds like the next step is seeing if this memory and steering contract works when you move it from a controlled lab setting to something more open.
More episodes
- 2608.13438-ContactGuard: Pre-Contact Execution Monitoring with Action-Conditioned Latent World Models
- 2610.34593-Future Prediction as Training-Only Video Supervision for Pretrained VLA Policies
- 2610.35385-EmbodiedLab: Beyond Manipulation, Benchmarking Embodied Agents That Act to Know
- 2610.36378-Data-Driven Hamiltonian for Neural Optimal Control
- 2610.37275-A Heuristic for Scheduling Autonomous Vehicles at Intersections
- 2610.37288-Dynamic Coordination of Autonomous Vehicles at Intersections
- 2610.37632-Real-Time Coordination of Autonomous Vehicles at Intersections
- 2611.13779-Softness is something you also need! Why compliance helps control: information metrics of a passive walker with compliant ankle
- 2611.13411-Residual-Corrected Physics-Informed Neural Network KKL Observers with Input-to-State Stability Guarantees
- 2611.14488-Is Your Robot's Memory Just a Clock? Temporal Shortcuts in Memory-Augmented VLA Policies