MegaMem: A Retrieval Solution for Ultra-Large Context Windows

arXiv:2608.22137 · cs.AI · Submitted 2026-08-22 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "MegaMem: A Retrieval Solution for Ultra-Large Context Windows".

Jane: The gist MegaMem introduces a source-resolved dual-view retrieval system that separates semantic access from generation evidence to support ultra-large persistent memory while keeping the generation context bounded.

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So we're looking at this paper, 'MegaMem: A Retrieval Solution for Ultra-Large Context Windows'. The title itself tells you exactly what it's trying to tackle, which is scaling up retrieval systems that have massive amounts of data. It’s not just about fitting bigger models; it’s about managing how we search through those huge stores effectively.

Jane: Right, and the authors are Xinyuan Song, Bowen Zhu, Hasibul Haque, Liang Zhao at Emory University and the Causal Dynamics Lab at Emory University. They've got a solid team behind this work.

Tom: What they are doing is introducing a source-resolved dual-view retrieval system. That sounds complicated, but the core idea is that you separate what you can search—the persistent context—from what you actually use to generate an answer, which is the evidence context.

Lu: Exactly, Tom. The paper suggests that instead of just searching through everything and throwing a lot of noise at the model to get an answer, we need a way to keep the search broad while keeping the final input small and focused on what’s actually relevant to the question.

Meng: I wonder how much extra storage this system needs for all these dual views. If we're talking about hundreds of millions or even a billion tokens, storing both detailed evidence and those distilled memories sounds like it could be quite demanding on resources.

Lalam: From my perspective, if the system can keep the search space vast without overwhelming the generation context, that’s what matters most for deploying these things in production environments.

The paper's summary: Tom: So, to summarize what MegaMem proposes here is this source-resolved dual-view retrieval system. It takes the massive persistent context and breaks it down into two different types of memory structures.

Jane: Think of it like this: you have the original documents with all their detailed evidence, and then you create these more compact, typed memories for each piece of that evidence.

Lu: That’s right. Then you index both views separately: one detailed index for the raw chunks and another distilled index for those compact memories. This lets them search in two different ways simultaneously.

Tom: The retrieval part is where it gets interesting because they build multiple query routes, including a canonical rewrite and some terminology diverse expansions, searching both of those indices at the same time.

Meng: Searching multiple routes sounds computationally expensive. How do you manage the complexity when you're trying to keep things fast for online retrieval?

Jane: They solve that by resolving all those distilled hits back to their original detailed chunks before they even start aggregating candidates, which keeps things grounded in reality.

Lalam: And then they use weighted reciprocal rank fusion and a cross-encoder reranking step to combine the results from all those routes into one final ranked list.

The paper's improvements: Tom: The main improvement they highlight is this separation of semantic access from generation evidence. That’s the big idea that lets them scale the searchable context up significantly while keeping the evidence context bounded.

Jane: They show that this approach is effective even when scaling persistent memory up to two hundred fifty million tokens under a fixed budget for what they consider the evidence budget.

Lu: The analysis shows that performance doesn't just drop off sharply as you go from, say, twenty million tokens to two hundred fifty million tokens. It degrades gradually instead of collapsing. That’s a key finding about how robust this architecture is when scaling up the context size.

Meng: From an engineering standpoint, that gradual degradation is better than a total failure when we’re trying to keep things running reliably at scale. What about the cost of building this dual index structure initially?

Tom: They estimate the offline construction cost for two hundred fifty million tokens to be about five point six days, which is manageable for an initial setup, but they emphasize that this is done once before you start querying.

Jane: And they also have a post-answer attribution step. This identifies which of those loaded sources actually supported the final answer and reduces the reported source set without changing the quality of the answer itself.

Conclusion: Tom: So, wrapping this up for MegaMem: it successfully addresses that problem we talked about—scaling persistent memory while keeping the evidence context bounded. It shows a practical path toward handling memories ranging from hundreds of millions to one billion tokens accurately.

Jane: The architecture design study found that combining those distilled retrieval keys with the source-resolved detailed evidence is more effective than just retrieving directly from the original corpus. It’s about that careful selection process they built in.

Lu: I think what’s really important is how they separated the roles: query transformation broadens recall, reranking selects the specific evidence used for generation, and post-answer attribution cleans up the final source set. That separation is the key architectural change here.

Meng: For deployment, it seems like online retrieval stays under one point one second at one hundred million tokens, and end-to-end answering stays under three point three seconds. Those are achievable numbers we need to see when we build production systems.

Lalam: I agree with the performance metrics. They achieve an overall score of eighty-two point three in their architecture design study, which shows that this dual-view approach is the most effective way to combine compressed retrieval keys with source-resolved detailed evidence for large context windows <ref:2608.22137#pg1>.

Emory University

cs.AI

Submitted: 2026-08-22

Updated: 2026-10-15

Code: https://github.com/xfab-xinyuansong/MegaMem

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 89/100

The gist: The gist MegaMem introduces a source-resolved dual-view retrieval system that separates semantic access from generation evidence to support ultra-large persistent memory while keeping the generation

Key concepts

Dual-View Memory Construction

This involves segmenting large documents into detailed evidence pieces and creating up to three compact, typed memories for each piece. These pieces are then indexed in two separate ways: one index holds embeddings of the original detailed chunks, and another stores embeddings of the distilled summaries.

Distilled Index

The distilled index stores embeddings derived from compact, typed memories rather than the full source text. This allows for faster searching across a much larger set of information while maintaining semantic relevance to the original content. These summaries are used as a proxy for the detailed evidence during retrieval.

Source-Resolved Retrieval

Every retrieved hit from the distilled index is mapped back to its specific, immutable source ID in the detailed index before any final ranking occurs. This ensures that when evidence is selected for answering, it can be precisely traced back to its original location in the massive corpus.

Bounded-Evidence Generation

The system uses a fixed budget of detailed evidence chunks to support answer generation, even if the initial search returned many candidates. After selecting the highest-ranked detailed evidence within this budget, post-answer attribution identifies exactly which loaded sources provided that final answer.

Summary

Improvements for AI systems

Abstract

Sources

• LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks
• Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory
• From Local to Global: A Graph RAG Approach to Query-Focused Summarization
• SPLADE v2: Sparse Lexical and Expansion Model for Information Retrieval
• Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
• LightRAG: Simple and Fast Retrieval-Augmented Generation
• RULER: What's the Real Context Size of Your Long-Context Language Models?
• FinanceBench: A New Benchmark for Financial Question Answering
• LongRAG: Enhancing Retrieval-Augmented Generation with Long-context LLMs
• Long Context RAG Performance of Large Language Models
• Fine-Tuning LLaMA for Multi-Stage Text Retrieval
• Generation-Augmented Retrieval for Open-domain Question Answering
• Passage Re-ranking with BERT
• Document Ranking with a Pretrained Sequence-to-Sequence Model
• MemGPT: Towards LLMs as Operating Systems
• EnterpriseRAG-Bench: A RAG Benchmark for Company Internal Knowledge
• Augmenting Language Models with Long-Term Memory
• In Defense of RAG in the Era of Long-Context Language Models
• $\infty$Bench: Extending Long Context Evaluation Beyond 100K Tokens
• MemoryBank: Enhancing Large Language Models with Long-Term Memory

Related papers

• MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
• Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
• Agents' Last Exam
• The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
• MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
• Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems