Benchmarking and Enhancing Skill-Level Memory for Partially Observable Robotic Manipulation
cs.RO
Submitted: 2026-09-30
Updated: 2026-09-30
Comments: Project page: https://nanamma.github.io/HIDE-SEEK/
License: http://creativecommons.org/licenses/by/4.0/
The gist: Recent advances in robot learning have enabled manipulation policies to perform increasingly diverse tasks and generalize across environments.
Terminology
Abstract
Recent advances in robot learning have enabled manipulation policies to perform increasingly diverse tasks and generalize across environments. However, reliable execution often depends on hidden task states that cannot be determined from current observations alone, making interaction history essential. We introduce HIDE, a benchmark for evaluating manipulation memory under partial observability. HIDE comprises 15 tasks covering repetition counting, historical-state recall, and execution-progress tracking, with randomized initial configurations and decision points where similar observations require different actions depending on prior events. We further propose SEEK, a framework combining three complementary memory mechanisms to retain historical evidence and track execution state. Evaluations reveal substantial limitations in existing policies on HIDE, while memory augmentation improves task success in both simulation and real-world experiments. Individual mechanisms benefit some tasks but can degrade others; their combination achieves the highest average success rate on HIDE among the evaluated configurations. These findings highlight the importance of maintaining internal representations of hidden task states and matching memory design to task-specific information requirements.
Sources
- $\pi_0$: A Vision-Language-Action Flow Model for General Robot Control
- History-Aware Visuomotor Policy Learning via Point Tracking
- PolarNet: 3D Point Clouds for Language-Guided Robotic Manipulation
- RMBench: Memory-Dependent Robotic Manipulation Benchmark with Insights into Policy Design
- Memory, Benchmark & Robots: A Benchmark for Solving Complex Tasks with Reinforcement Learning
- $\mu$VLA: On Recurrent Memory for Partially Observable Manipulation in VLA Models
- RoboMME: Benchmarking and Understanding Memory for Robotic Generalist Policies
- Act3D: 3D Feature Field Transformers for Multi-Task Robotic Manipulation
- RVT-2: Learning Precise Manipulation from Few Demonstrations
- Instruction-driven history-aware policies for robotic manipulations
- Coarse-to-Fine Q-attention: Efficient Learning for Visual Robotic Manipulation via Discretisation
- BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning
- ContextVLA: Vision-Language-Action Model with Amortized Multi-Frame Context
- 3D Diffuser Actor: Policy Diffusion with 3D Scene Representations
- OpenVLA: An Open-Source Vision-Language-Action Model
- Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success
- RoboMemArena: A Comprehensive and Challenging Robotic Memory Benchmark
- GR00T N1: An Open Foundation Model for Generalist Humanoid Robots
- $\pi_{0.5}$: a Vision-Language-Action Model with Open-World Generalization
- THE COLOSSEUM: A Benchmark for Evaluating Generalization for Robotic Manipulation
Related papers
- FMT x: An Efficient and Asymptotically Optimal Extension of the Fast Marching Tree for Dynamic Replanning
- MPCFormer: A physics-informed data-driven approach for explainable socially-aware autonomous driving
- RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies
- ARC: A Reasoning Recipe for Robot Foundation Models
- HRDexDB: A 4D Dexterous Grasping Dataset Across Human and Multiple Robot Embodiments
- APT: Action Expert Pretraining Improves Instruction Generalization of Vision-Language-Action Policies