NEAR: Neighborhood-Aware Residuals for Memory Retrieval
summary
The gist
Memory retrieval aims to identify evidence from stored records to answer a query.Such shared contents would make records without the required evidence more likely to receive high similarity scores…
In short
Memory retrieval often struggles when similar records compete for attention, potentially ranking incorrect evidence higher than correct information. NEAR is a training-free framework that constructs local references from neighboring memories to create residual representations that complement original embeddings. This allows for better query-memory matching through a single approximate nearest-neighbor search.
Key concepts
- Memory Retrieval Challenge
- Standard retrieval ranks memories based only on similarity to the query, which can cause records without the right evidence to outrank correct ones when similar records exist. NEAR addresses this by using neighborhood information to provide better matching signals.
- Neighborhood-Aware Residuals (NEAR)
- NEAR is a training-free method that builds local references for each memory by looking at its nearest neighbors. It then removes the embedding component along this reference direction to create residual representations that enhance the original embeddings for matching.
- Query-Reference Table
- A small, precomputed table is built during offline indexing. This table stores query references derived from spherical K-means clustering of memory states. At retrieval time, this table allows NEAR to use a single approximate nearest-neighbor search over a fixed index.
Terminology used across episodes
This episode discusses
- NEAR: Neighborhood-Aware Residuals for Memory Retrieval · Paper Radio
- Llama-Embed-Nemotron-8B: A Universal Text Embedding Model for Multilingual and Cross-Lingual Tasks
- REALTALK: A 21-Day Real-World Dataset for Long-Term Conversation
- PRQ-KMeans: Projection Residual Quantization for Semantic ID Tokenization · Paper Radio
- ViDoRe Benchmark V2: Raising the Bar for Visual Retrieval
- NV-Retriever: Improving text embedding models with effective hard-negative mining
- EvoEmbedding: Evolvable Representations for Long-Context Retrieval and Agentic Memory
- EmbeddingGemma: Powerful and Lightweight Text Representations
- Qwen3 Technical Report
- Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models
- LMEB: Long-horizon Memory Embedding Benchmark · Paper Radio
The paper
NEAR: Neighborhood-Aware Residuals for Memory Retrieval · Read on arXiv
Southeast University · MemoraX AI Institute of Technology, Leiden University, University of Sydney
Transcript
Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "NEAR: Neighborhood-Aware Residuals for Memory Retrieval".
Jane: Here's the gist of the paper. Memory retrieval aims to identify evidence from stored records to answer a query.Such shared contents would make records without the required evidence more likely to receive high similarity scores and outrank the correct evidence.In this paper, we use the information shared by these records to construct local references for query–memory matching.We propose NEighborhood-Aware Residuals (NEAR), a training-free framework that uses these references to construct residual representations that complement the original embeddings.Experiments on eight long-context retrieval benchmarks show consistent improvements across nine embedding models.Downstream evaluations show that NEAR improves question-answering performance.
Tom: Pruning redundant reasoning tokens Memory retrieval enables intelligent assistants to find evidence in past conversations and stored documents to answer current queries.A common approach is embedding-based retrieval, which encodes queries and memories separately and ranks memories by vector similarity.When ranking relies only on query–memory similarity, such overlap would make records that do not provide the required evidence more likely to outrank correct evidence.We argue that they also provide a local neighborhood that can serve as a reference for each memory.We turn this potential source of confusion into useful matching information by constructing memory-specific local references from the relationships among similar memories.In this paper, we propose NEighborhood-Aware Residuals (NEAR), a training-free framework that uses these local references for query–memory matching.For each memory, NEAR builds a reference from its nearest neighbors and removes the embedding component along this direction.A small precomputed table provides references for new queries without an initial search over the memory bank.NEAR combines original and residual representations so that the final matching score is a single inner product.Offline indexing enables retrieval through a single approximate nearest-neighbor (ANN) search over the fixed memory index.
Jane: How the training signal is built For each memory, NEAR builds a reference from its nearest neighbors and removes the embedding component along this direction.For subsequent layers, we repeat this procedure using the normalized residuals h(l)i = unit(r(l)i).The matching representation either retains the residual length or is normalized: z(l)i = (r(l)i, η = 0, unit(r(l)i), η = 1).At each layer, NEAR precomputes a small table that maps queries to local reference directions.Spherical K-means (Dhillon & Modha, 2001) clusters the input memory states, with cluster centers serving as anchors a(l)h.Similarity-based softmax weights combine their stored directions into a query reference: p(l)h(q) = explbraceβ(h(l−1)q)⊤a(l)h Pj∈H(l)q explbraceβ(h(l−1)q)⊤a j.
Tom: The same projection and normalization rules yield the query matching representation z(l)q and next-layer state h(l)q.
Paper summary: Tom: So we're diving into this paper called "NEAR: Neighborhood-Aware Residuals for Memory Retrieval." What's the main idea here?
Jane: The core problem they’re tackling is that when you retrieve information from stored records, if those records share topics or events, the system might mistakenly give a high score to a record that doesn't actually have the right evidence because it just looks similar.
Lu: So they use this shared content to build local references for query memory matching. They propose something called Neighborhood-Aware Residuals, which is a training-free framework designed to complement the original embeddings with these new references.
Meng: Training-free sounds good on paper, but from my side, what does that actually mean for deployment? Does it add significant overhead to the inference pipeline?
Tom: Well, they claim this approach uses the information shared by these records to construct local references for query memory matching. They propose this framework to build residual representations that complement the original embeddings.
Jane: So how do they use these references practically? It sounds like a clever way to handle those confusing overlaps we talked about earlier.
Lu: For each memory, NEAR builds a reference from its nearest neighbors and removes the embedding component along this direction. This creates these local references for each memory.
Meng: Building references from neighbors sounds computationally expensive, especially if you have millions of records in that memory bank. What's the trade-off there?
Tom: They address that by having a small precomputed table that provides these references for new queries without an initial search over the entire memory bank. It allows offline indexing and retrieval through a single approximate nearest neighbor search.
Jane: So instead of searching everything from scratch every time, you have this fixed index and these query references ready to go? That sounds efficient for real-world use.
Lu: Exactly, they construct a compact query-reference table that contains at most sixteen entries and they build it from the memory embeddings using spherical K means. This table then combines the original and residual views into a single inner product for that search.
Meng: Sixteen entries is small enough, but how robust is this precomputed table when dealing with new types of queries? Does it still work well across different contexts?
Tom: They selected the fusion weight and residual construction on development data, and those same configurations were reused for held-out top ten and top five evaluations. They say NEAR maintains gains over Base across multiple residual constructions and fusion weights, not just one specific setup.
Paper summary: Jane: So they're saying this isn't just a lucky configuration; the general idea of neighborhood awareness is what makes it work across different settings?
Lu: The paper shows that unnormalized residuals perform better on long context retrieval, while normalized residuals achieve higher ranking at moderate and large weights on long memory evaluation. It depends on the task.
Meng: I see a dependence there. So for a production system, you’d need to figure out which residual construction or normalization method is best for the specific type of memory retrieval you're doing?
Tom: They found that uncompressed NEAR reaches mean recall at ten and ndcg at ten of seventy-one point eight two and fifty-nine point two one compared to fifty-seven point nine one and forty five point eight eight for Base. This is a significant lift in retrieval quality for the same storage footprint, although it uses more storage than the FP32 Base SQ4 model, with a total storage ratio of two point zero two relative to that model.
Jane: So they're saying that even when you're looking at these numbers, you still get a tangible improvement in how well the system finds relevant evidence. What about performance on actual question answering?
Lu: Downstream evaluations show that NEAR improves question answering accuracy by nine point six percentage points on LoCoMo with Qwen3-star. This means the retrieval gains translate directly into better answers to questions about conversation histories.
Meng: Nine point six percentage points is a solid lift, but how much does that real-world accuracy improvement depend on the specific model you're using for the final question answering step?
Tom: They tested it across nine embedding models and showed consistent improvements. The results show that NEAR improves both accuracy and F1 across all nine backbones using original conversation histories of LoCoMo and LongMemEval. For Qwen-4B, accuracy increases from sixty-two point seven to seventy-two point three, while F1 rises from thirty point nine to thirty five point five <ref:2610.13314#pg1>.
Jane: That's a pretty strong lift for the Qwen models, especially looking at the F1 score improvement. But they did mention some trade-offs in those final evaluations?
Lu: They noted that on LongMemEval, NEAR improves accuracy on seven of the nine backbones, including Qwen-4B from sixty-nine point three to seventy-two point one and KaLM-12B from seventy five point eight to seventy seven point four <ref:2610.13314#pg2>. But they also saw some small decreases for Llama 3B and EmbeddingGemma <ref:2610.13314#pg1>.
Meng: Small decreases are worrying for production. If you're deploying this, you need to know exactly where the performance dips, especially on smaller models like Llama 3B <ref:2610.13314#pg1>. What does that tell us about the generality of this neighborhood-aware approach?
Paper summary: Tom: The authors are pretty clear here: they studied the difficulty of retrieving correct evidence from memory records with shared topics or entities. Their key observation was that these similar records weren't just competing candidates; they also provided a local neighborhood for each memory.
Jane: So the paper is really focused on solving that confusion caused by records sharing content, turning that potential noise into useful matching information through these local references.
Lu: That’s the crux of it, they turned this potential source of confusion into useful matching information by constructing memory-specific local references. They also built a compact query-reference table and a fixed memory index, enabling new queries to retrieve memories through a single approximate nearest neighbor search without preliminary candidate retrieval.
Meng: So for deployment, that means the system can handle new queries with just one ANN search over this fixed index? That’s much better than running a full retrieval process every time.
Tom: Right, that's exactly what they show—NEAR obtains query references from a precomputed table and combines residual representations with the original embeddings in a fixed index for a single approximate nearest neighbor search.
Jane: So, to summarize this paper "NEAR: Neighborhood-Aware Residuals for Memory Retrieval," it’s a training-free method that uses local neighborhood information to create residual representations that boost query memory matching and ultimately improve downstream question answering performance by nine point six percentage points on LoCoMo with Qwen3-star.
Lu: And they found that the gains extend beyond just the top ten results, improving recall at every tested budget across both datasets.
Tom: So to wrap up, NEAR is this training-free method that constructs memory-specific local references from shared contents and uses residual representations to complement original embeddings for query memory matching.
Jane: It’s a training-free framework that uses these local references to construct residual representations that complement the original embeddings for query memory matching.
Lu: They found that the evidence outscores Base's top ten in the first residual layer, which is a key finding showing how much power these new representations are adding.
Meng: The system can use one ANN search over a fixed index with a small table of sixteen entries to find memories for any query.
Tom: This allows NEAR to use one ANN search over a fixed index, combining the original and residual matching components into a single inner product representation during retrieval.
Jane: The selection of the fusion weight and residual construction are selected on development data, but those specific configurations are reused for held-out top ten and top five evaluations across multiple residual constructions, which shows robustness.
Conclusion: Tom: So we’re wrapping up on NEAR: Neighborhood-Aware Residuals for Memory Retrieval, looking at what this whole paper is really saying about finding evidence in old conversations and documents.
Jane: It basically shows how you can use the overlap between similar records to create better matching information for a query. It’s all about turning similarity into a useful reference point.
Lu: The authors propose this training-free framework, NEAR, which builds local references from memory neighbors and uses those to complement the original embeddings for query memory matching.
Meng: So they've got this system that doesn't need retraining because it figures out these connections on the fly using some precomputed tables. That’s something I can actually think about deploying at scale.
Tom: It is pretty clean—they use a fixed index and just one approximate nearest neighbor search to find memories for any new query. That cuts down on the time needed to search through everything from scratch every single time you ask a question.
Jane: The implication here is that we don't have to re-rank everything every time we want an answer; we just look at these specific, relevant neighborhood references.
Lu: They found that this method actually helps downstream question answering accuracy by nine point six percentage points on the LoCoMo benchmark with the Qwen three star model. That’s a direct hit on how well the AI can actually use that retrieved evidence to form a final answer.
Meng: Nine point six is a solid lift, but what about those models that didn't see such big gains? They mentioned some small decreases for Llama three b and Embedding Gemma on the LongMemEval test. That’s important for me to know where the limits are.
Tom: Yeah, it means the quality isn't perfectly uniform across every single model out there, which is expected when you introduce these new ways of handling similarity.
Jane: So what we see here is that this neighborhood awareness solves a real headache in retrieval by making sure similar things don't accidentally cancel each other out during the search.
Lu: It’s a way to manage those confusing overlaps between records that share topics or events, turning them from potential confusion into actual matching information.
Tom: So to sum it up, NEAR gives us this training-free tool that uses local neighborhood data to build residual representations for query memory matching.
Jane: And the big win is that these improvements translate directly into better answers when the AI tries to answer a question based on those retrieved memories.
Lu: The work suggests that we can get more out of existing embeddings by adding these learned, context-specific local references without needing to retrain the whole system from scratch.
Meng: So for anyone building something like this, it means you get better retrieval performance with less training overhead, provided you can handle that fixed index setup.
Tom: Next time we talk about this, we’ll look at how they tuned the fusion weight and residual construction to see what settings actually gave them those best results across different tasks.
More episodes
- 2505.16254-Human-AI Writing as Distributed Cognition: Reassessing Collaborative Writing Theories in the Age of LLMs
- 2506.10913-Choreographic Quick Changes: First-Class Location (Set) Polymorphism
- 2507.14812-A Black-Box Approach for Exogenous Replenishment in Online Resource Allocation
- 2507.15448-Frames over finite fields with respect to the Galois inner product and their connections to self-orthogonal and LCD codes
- 2508.21572-NewsReX: An Open-Source Multi-Framework for Neural News Recommendation
- 2509.17028-Impact of Packetization on Network Calculus Analysis
- 2510.17342-AoA Services in 5G Networks: A Framework for Real-World Implementation and Systematic Testing
- 2511.04088-Efficient and rate-optimal list-decoding in the presence of minimal feedback
- 2511.17264-Notes on Stack Machines and Quantum Stack Machines
- 2601.15318-On the closest balanced game