PREM: Prediction-Regulated Memory for Robot Manipulation
cs.RO, cs.CV
Submitted: 2026-10-31
Updated: 2026-10-31
Comments: 23 pages, 7 figures
License: http://creativecommons.org/licenses/by/4.0/
The gist: A robot's next action can depend on recent motion or earlier visual evidence absent from its current observation.
Terminology
Abstract
A robot's next action can depend on recent motion or earlier visual evidence absent from its current observation. A recurrent visual-memory architecture can retain this history in a visual state, but its usefulness depends on what the state represents and how it is updated with new observations. We introduce Prediction-Regulated Memory (PREM), a framework that uses action-conditioned future prediction to guide both memory learning and updating in vision-language-action policies. During training, predicting future visual features provides temporal supervision for memory learning. During execution, PREM compares current visual features with the prediction made at the preceding decision and uses the spatially normalized prediction discrepancy to regulate updates to the visual state at each spatial location. The updated visual state supplies temporal context for action generation alongside current observations. Built on π 0.5, PREM improves average success rate from 9.63% to 20.74% on DOMINO dynamic manipulation tasks and from 14.4% to 83.8% on the RMBench M(1) memory-dependent setting. Further evaluations on LIBERO and LIBERO-Pro show improvements in general manipulation and robustness to spatial perturbations. PREM incurs a mean inference overhead of 7.7 ms on a single NVIDIA H800 GPU.
Sources
- V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning
- Revisiting Feature Prediction for Learning Visual Representations from Video
- GR00T N1: An Open Foundation Model for Generalist Humanoid Robots
- $\pi_0$: A Vision-Language-Action Flow Model for General Robot Control
- RT-1: Robotics Transformer for Real-World Control at Scale
- RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control
- RMBench: Memory-Dependent Robotic Manipulation Benchmark with Insights into Policy Design
- PaLM-E: An Embodied Multimodal Language Model
- SAM2Act: Integrating Visual Foundation Model with A Memory Architecture for Robotic Manipulation
- ChainVLA: Chaining Vision-Language-Action Queries through a Unified Execution State for Long-Horizon Manipulation
- What Are We Actually Benchmarking in Robot Manipulation?
- OpenVLA: An Open-Source Vision-Language-Action Model
- Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success
- FAST: Efficient Action Tokenization for Vision-Language-Action Models
- $\pi_{0.5}$: a Vision-Language-Action Model with Open-World Generalization
- SpatialVLA: Exploring Spatial Representations for Visual-Language-Action Model
- MemoryVLA++: Temporal Modeling via Memory and Imagination in Vision-Language-Action Models
- Static In, Dynamic Out: Counterfactual Action Augmentation for Moving Object Manipulation
- Octo: An Open-Source Generalist Robot Policy
- A Pragmatic VLA Foundation Model
Related papers
- Rethinking Actions: Continuity-Aware Priors for World-Model Planning
- MOSAIC: Modular Obstacle and Spatial-Aware Framework for Manipulation In Cluttered Scenes
- SAGAS: Semantic-Aware Graph-Assisted Stitching for Offline Temporal Logic Planning
- DexSeed: Expanding Human Hand-Object Interactions into Robot Demonstrations for Dexterous Policy Learning
- LaST0.5: A Generalist World Action Model with Interaction-Centric Latent Reasoning
- CIALLO: Learning Shared Intent Representations for Cross-Embodiment Manipulation