Learning What to Recall: Adaptive Multi-Cue Episodic Memory for World Models
cs.LG, cs.AI, cs.CV
Submitted: 2026-09-28
Updated: 2026-09-28
Comments: Preprint, Project Page: https://1202kbs.github.io/FAR-Project-Page/
Project page: https://1202kbs.github.io/FAR-Project-Page
License: http://creativecommons.org/licenses/by/4.0/
The gist: World models predict future observations from current experience and actions, yet prediction can depend on observations seen far in the past.
Terminology
Abstract
World models predict future observations from current experience and actions, yet prediction can depend on observations seen far in the past. Episodic memory preserves past observations for later recall; however, as memory accumulates, it raises a fundamental question: which memories are useful for the current prediction, and which available retrieval cues should be trusted to find them? This is challenging because fixed criteria based on recency, pose overlap, or visual similarity can be unreliable across environments and queries. We propose Future-Aware Recall (FAR), a framework that learns episodic recall from future-aware predictive supervision and adaptive multi-cue scoring. During training, FAR measures predictive utility by the conditional log-likelihood of the realized future given recalled context, approximated by negative diffusion prediction loss, and uses it to train a retriever that remains future-blind at inference. The retriever learns cue-specific relevance and automatically determines which available retrieval cues, such as time, pose, vision, and audio, to trust for each query when selecting memories. Across three complementary settings, FAR outperforms hand-designed recall even with the same retrieval cues, automatically adapts which available cues to trust, and recalls the right history as the world changes. Together, these results establish FAR as a flexible, principled approach to episodic memory access in world models.
Sources
- MilliVid: Hierarchical Latents for Long-Range Consistency in Video Generation
- LongLive-RAG: A General Retrieval-Augmented Framework for Long Video Generation
- AI2-THOR: An Interactive 3D Environment for Visual AI
- The Principles of Diffusion Models
- I3DM: Implicit 3D-aware Memory Retrieval and Injection for Consistent Video Scene Generation
- LoopNav: Benchmarking Spatial Consistency in World Models
- Compression and Retrieval: Implicit Memory Retrieval for Video World Models
- Matrix-Game 3.0: Real-Time and Streaming Interactive World Model with Long-Horizon Memory
- SANA-WM: Efficient Minute-Scale World Modeling with Hybrid Linear Diffusion Transformer
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks