RMBench: Memory-Dependent Robotic Manipulation Benchmark with Insights into Policy Design

The gist: This paper introduces RMBench, a new simulation benchmark for memory-dependent robotic manipulation, and Mem-0, a modular memory-oriented policy designed to systematically evaluate and improve policy design concerning long-term memory.

Task Memory Complexity (TMC)

Task Memory Complexity is defined as the smallest integer m ≥ 0 such that there exists an optimal policy π∗ whose action at time t depends only on the memory state M(m)t, formally stated as ∃ π∗ s.t. π∗(at ht) = π∗(at M(m)t), ∀t

RMBench Benchmark

RMBench is a robotic manipulation benchmark built on RoboTwin 2.0 platform, consisting of 9 dual-arm manipulation tasks spanning different levels of task memory complexity The tasks are grouped into two categories: five M(1) tasks, and four M(n) tasks The M(1) tasks consist of Observe and Pick Up, Rearrange Blocks, Put Back Block, Swap Blocks and Swap T <ref:2603.01229#pg5> The M(n) tasks include Blocks Ranking Try, Press Button, Cover Blocks, and Battery Try <ref:2603.01229#pg5>

Mem-0 Policy Architecture

Mem-0 is a novel memory-oriented robotic policy designed with modular components that can be easily replaced or reconfigured <ref:2603.01229#pg3> It adopts a dual system architecture with a task-phase classifier that explicitly distinguishes different stages of a task, allowing structured memory usage across long horizons <ref:2603.01229#pg3> The Planning Module performs subtask-level reasoning using a vision language model conditioned on visual observations and structured memory, predicting the next subtask as st = Vplan(o0, g, Mt−1) <ref:2603.01229#pg5> This design reduces the number of planning invocations to O(N), where N ≪ T, because it performs subtask reasoning only upon subtask termination <ref:2603.01229#pg5>

Memory Components and Ablation Insights

The Execution Module incorporates an anchor memory module and a sliding memory window module to handle M(1)-type tasks without additional subtask decomposition <ref:2603.01229#pg6> Analysis on Anchor Memory Performance shows that removing the anchor memory leads to a substantial reduction in success rates across most tasks, indicating that it is crucial for a policy to explicitly identify and retain task-critical information throughout execution <ref:2603.01229#pg9> The Sliding Memory Window primarily captures short-term historical motion trends, but its removal still degrades performance across most tasks, with success rates falling below those of the vanilla model <ref:2603.01229#pg9> Interestingly, on the Swap T task, the w/o Sliding setting outperforms the vanilla model <ref:2603.01229#pg9>

Classifier Limitations and Future Directions

The Subtask End Classifier serves to connect the Planning and Execution Modules by triggering high-level reasoning only when necessary, reducing inference cost <ref:2603.01229#pg7> Failures in M(n) tasks often stem from poor classifier performance, such as failing to accurately detect the completion of an Uncover xxx subtask <ref:2603.01229#pg16> The Press Button task shows that the visual tokens generated by the VLM backbone lack the granularity to encapsulate fine-grained information, creating a fundamental bottleneck for the downstream Classifier To address these observations, one potential avenue for enhancement involves exploring more richer representation fusion mechanisms to improve the utilization of both Anchor and Sliding Memory <ref:2603.01229#pg15> Moreover, incorporating proprioceptive or tactile feedback could provide the Classifier with critical non-visual information The efficacy of the current design has been validated across various M(n) tasks, notably yielding substantial performance gains in Cover Blocks compared to the baseline

Real-World Performance

Mem-0 outperforms both baseline policies in real-world experiments, with success rates showing substantial improvements on average of 38.4% on M(1) tasks and 21.2% on M(n) tasks relative to the baselines <ref:2603.01229#pg10> Most failures of Mem-0 arise from imprecise block manipulation rather than high-level task planning <ref:2603.01229#pg10> The results show that explicit memory modeling yields significant improvements on the majority of memory-dependent tasks in RMBench <ref:2603.01229#pg7>

Conclusion

In this paper, we present RMBench (benchmark) and Mem-0 (policy) to systematically evaluate memory in robotic manipulation, revealing the memory limitations of existing policies and how different architectural choices affect memory performance <ref:2603.01229#pg2> As for future work, promising directions include improved memory representation and fusion, more robust subtask termination criteria, and the integration of pretraining to enhance semantic understanding and generalization <ref:2603.01229#pg7> The paper provides preliminary insights into the principled integration of memory mechanisms for effective memory-dependent robotic manipulation. >

--- Page 1 ---

RMBench: Memory-Dependent Robotic Manipulation Benchmark with Insights into Policy Design

with Insights into Policy Design <ref:2603.01229#pg2> Tianxing Chen, Yuran Wang, Mingleyang Li, Yan Qin, Hao Shi, Zixuan Li, Yifan Hu, Yingsheng Zhang, Kaixuan Wang, Yue Chen, Hongcheng Wang <ref:2603.01229#pg2>

Abstract

Robotic manipulation policies have made rapid progress in recent years <ref:2603.01229#pg3>, yet most existing approaches give limited consideration to memory capabilities <ref:2603.01229#pg3>. Consequently, they struggle to solve tasks that require reasoning over historical observations and maintaining task-relevant information over time, which are common requirements in real-world manipulation scenarios <ref:2603.01229#pg3>. Although several memory-aware policies have been proposed, systematic evaluation of memory-dependent manipulation remains underexplored <ref:2603.01229#pg3>, and the relationship between architectural design choices and memory performance is still not well understood <ref:2603.01229#pg3>. To address this gap, we introduce RMBench, a simulation benchmark comprising 9 manipulation tasks that span multiple levels of memory complexity, enabling systematic evaluation of policy memory capabilities <ref:2603.01229#pg4>. We further propose Mem-0, a modular manipulation policy with explicit memory components designed to support controlled ablation studies <ref:2603.01229#pg4>. Through extensive simulation and real-world experiments, we identify memory related limitations in existing policies and provide empirical insights into how architectural design choices influence memory performance <ref:2603.01229#pg4>.

1. Introduction

Recent progress in robotic manipulation has demonstrated strong capabilities across a wide range of tasks <ref:2603.01229#pg2>. Modern policies such as Pi0.6 (Intelligence et al., 2025a) and 1MMLab@HKU 2 PKU 3 PsiBot 4 HKUST (GZ) 5 THU 6 SZU 7 SJTU <ref:2603.01229#pg2> RDT2 (Team, 2025) achieve impressive performance in flexible object manipulation and fine-grained skills, including complex activities like coffee making <ref:2603.01229#pg2>. Nevertheless, most existing robotic policies are primarily designed for short-horizon tasks and fine-grained manipulation <ref:2603.01229#pg2>. These approaches typically rely on a fixed-length window of recent observations, implicitly assuming that the underlying decision process is approximately Markovian <ref:2603.01229#pg2>.

In contrast, memory-dependent tasks are ubiquitous in realworld robotic applications <ref:2603.01229#pg2>. Such tasks are inherently nonMarkovian, as past observations and actions may influence future decisions over extended temporal horizons <ref:2603.01229#pg2>. Examples include remembering the location of previously placed objects or reasoning over multiple attempts after forgetting a password <ref:2603.01229#pg2> These tasks are both challenging and practically important, as they require robots to retain, retrieve, and utilize information beyond short-term sensory inputs <ref:2603.01229#pg2>.

Motivated by this challenge, several recent works have begun to explore memory-aware robotic policies <ref:2603.01229#pg3>. Approaches such as MemoryVLA (Shi et al., 2025), MemER (Sridhar et al., 2025), CronusVLA (Li et al., 2025), and SAM2Act (Fang et al., 2025) incorporate explicit memory mechanisms to address long-horizon and memorydependent decision making <ref:2603.01229#pg3>. Despite these efforts, the field currently lacks a systematically designed experimental platform for evaluating and analyzing robotic policies under long-term memory requirements <ref:2603.01229#pg3>. In particular, there is limited understanding of the underlying mechanisms that make memory strategies effective for robotic manipulation <ref:2603.01229#pg3>.

Existing benchmarks only partially bridge this gap <ref:2603.01229#pg3>. MemoryBench (Fang et al.

1. Improve memory-dependent robotic policies by implementing Task Memory Complexity (TMC) as a principled metric for classifying tasks, which allows for systematic design of benchmarks to better characterize how much task-relevant information must be retained over time.

2. Develop a policy like Mem-0 with a dual-system architecture featuring an explicit key memory window in the Planning Module, enabling the model to perform subtask reasoning based on aggregated past subtask outcomes, as described by the equation for key memory: Mt−1 = si, oend i t−1 i=1.

3. Enhance execution stability by incorporating a robust Subtask End Classifier that enforces a temporal consistency criterion—requiring L = 8 consecutive timesteps of termination signals—to prevent premature subtask transitions, addressing the failure mode where the classifier fails to accurately perceive the ongoing task progress.

4. Improve handling of M(n) tasks by refining memory fusion mechanisms, specifically by exploring more richer representation fusion mechanisms to improve the utilization of both Anchor and Sliding Memory, aiming to mitigate performance degradation observed when omitting these components.

5. Increase robustness in button-pressing operations by integrating proprioceptive or tactile feedback into the classifier's input, as suggested by the analysis that incorporating proprioceptive or tactile feedback could provide the Classifier with critical non-visual information.