Memory Decoder at Scale: A Pretrained, Parametric Long-Term Memory
Rubin Wei, Jiaqi Cao, Jiarui Wang, Junming Zhang, Qipeng Guo, Bowen Zhou, Zhouhan Lin
cs.CL
Submitted: 2026-07-30
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Terminology
Sources
- Titans: Learning to Memorize at Test Time
- Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
- Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context
- LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens
- TwinVoice: A Multi-dimensional Benchmark Towards Digital Twins via LLM Persona Simulation
- The Pile: An 800GB Dataset of Diverse Text for Language Modeling
- Biology-Instructions: A Dataset and Benchmark for Multi-Omics Sequence Understanding Capability of Large Language Models
- Measuring Massive Multitask Language Understanding
- Generalization through Memorization: Nearest Neighbor Language Models
- LogiQA: A Challenge Dataset for Machine Reading Comprehension with Logical Reasoning
- Decoupled Weight Decay Regularization
- 2 OLMo 2 Furious
- Olmo 3
- GPQA: A Graduate-Level Google-Proof Q&A Benchmark
- Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
- OpenAI GPT-5 System Card
- Qwen3.5-Omni Technical Report
- MLP Memory: A Retriever-Pretrained Memory for Large Language Models
- DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
- Qwen3 Technical Report
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering