NEAR: Neighborhood-Aware Residuals for Memory Retrieval
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "NEAR: Neighborhood-Aware Residuals for Memory Retrieval".
Jane: Here's the gist of the paper. Memory retrieval aims to identify evidence from stored records to answer a query.Such shared contents would make records without the required evidence more likely to receive high similarity scores and outrank the correct evidence.In this paper, we use the information shared by these records to construct local references for query–memory matching.We propose NEighborhood-Aware Residuals (NEAR), a training-free framework that uses these references to construct residual representations that complement the original embeddings.Experiments on eight long-context retrieval benchmarks show consistent improvements across nine embedding models.Downstream evaluations show that NEAR improves question-answering performance.
Tom: Pruning redundant reasoning tokens Memory retrieval enables intelligent assistants to find evidence in past conversations and stored documents to answer current queries.A common approach is embedding-based retrieval, which encodes queries and memories separately and ranks memories by vector similarity.When ranking relies only on query–memory similarity, such overlap would make records that do not provide the required evidence more likely to outrank correct evidence.We argue that they also provide a local neighborhood that can serve as a reference for each memory.We turn this potential source of confusion into useful matching information by constructing memory-specific local references from the relationships among similar memories.In this paper, we propose NEighborhood-Aware Residuals (NEAR), a training-free framework that uses these local references for query–memory matching.For each memory, NEAR builds a reference from its nearest neighbors and removes the embedding component along this direction.A small precomputed table provides references for new queries without an initial search over the memory bank.NEAR combines original and residual representations so that the final matching score is a single inner product.Offline indexing enables retrieval through a single approximate nearest-neighbor (ANN) search over the fixed memory index.
Jane: How the training signal is built For each memory, NEAR builds a reference from its nearest neighbors and removes the embedding component along this direction.For subsequent layers, we repeat this procedure using the normalized residuals h(l)i = unit(r(l)i).The matching representation either retains the residual length or is normalized: z(l)i = (r(l)i, η = 0, unit(r(l)i), η = 1).At each layer, NEAR precomputes a small table that maps queries to local reference directions.Spherical K-means (Dhillon & Modha, 2001) clusters the input memory states, with cluster centers serving as anchors a(l)h.Similarity-based softmax weights combine their stored directions into a query reference: p(l)h(q) = explbraceβ(h(l−1)q)⊤a(l)h Pj∈H(l)q explbraceβ(h(l−1)q)⊤a j.
Tom: The same projection and normalization rules yield the query matching representation z(l)q and next-layer state h(l)q.
Paper summary: Tom: So we're diving into this paper called "NEAR: Neighborhood-Aware Residuals for Memory Retrieval." What's the main idea here?
Jane: The core problem they’re tackling is that when you retrieve information from stored records, if those records share topics or events, the system might mistakenly give a high score to a record that doesn't actually have the right evidence because it just looks similar.
Lu: So they use this shared content to build local references for query memory matching. They propose something called Neighborhood-Aware Residuals, which is a training-free framework designed to complement the original embeddings with these new references.
Meng: Training-free sounds good on paper, but from my side, what does that actually mean for deployment? Does it add significant overhead to the inference pipeline?
Tom: Well, they claim this approach uses the information shared by these records to construct local references for query memory matching. They propose this framework to build residual representations that complement the original embeddings.
Jane: So how do they use these references practically? It sounds like a clever way to handle those confusing overlaps we talked about earlier.
Lu: For each memory, NEAR builds a reference from its nearest neighbors and removes the embedding component along this direction. This creates these local references for each memory.
Meng: Building references from neighbors sounds computationally expensive, especially if you have millions of records in that memory bank. What's the trade-off there?
Tom: They address that by having a small precomputed table that provides these references for new queries without an initial search over the entire memory bank. It allows offline indexing and retrieval through a single approximate nearest neighbor search.
Jane: So instead of searching everything from scratch every time, you have this fixed index and these query references ready to go? That sounds efficient for real-world use.
Lu: Exactly, they construct a compact query-reference table that contains at most sixteen entries and they build it from the memory embeddings using spherical K means. This table then combines the original and residual views into a single inner product for that search.
Meng: Sixteen entries is small enough, but how robust is this precomputed table when dealing with new types of queries? Does it still work well across different contexts?
Tom: They selected the fusion weight and residual construction on development data, and those same configurations were reused for held-out top ten and top five evaluations. They say NEAR maintains gains over Base across multiple residual constructions and fusion weights, not just one specific setup.
Paper summary: Jane: So they're saying this isn't just a lucky configuration; the general idea of neighborhood awareness is what makes it work across different settings?
Lu: The paper shows that unnormalized residuals perform better on long context retrieval, while normalized residuals achieve higher ranking at moderate and large weights on long memory evaluation. It depends on the task.
Meng: I see a dependence there. So for a production system, you’d need to figure out which residual construction or normalization method is best for the specific type of memory retrieval you're doing?
Tom: They found that uncompressed NEAR reaches mean recall at ten and ndcg at ten of seventy-one point eight two and fifty-nine point two one compared to fifty-seven point nine one and forty five point eight eight for Base. This is a significant lift in retrieval quality for the same storage footprint, although it uses more storage than the FP32 Base SQ4 model, with a total storage ratio of two point zero two relative to that model.
Jane: So they're saying that even when you're looking at these numbers, you still get a tangible improvement in how well the system finds relevant evidence. What about performance on actual question answering?
Lu: Downstream evaluations show that NEAR improves question answering accuracy by nine point six percentage points on LoCoMo with Qwen3-star. This means the retrieval gains translate directly into better answers to questions about conversation histories.
Meng: Nine point six percentage points is a solid lift, but how much does that real-world accuracy improvement depend on the specific model you're using for the final question answering step?
Tom: They tested it across nine embedding models and showed consistent improvements. The results show that NEAR improves both accuracy and F1 across all nine backbones using original conversation histories of LoCoMo and LongMemEval. For Qwen-4B, accuracy increases from sixty-two point seven to seventy-two point three, while F1 rises from thirty point nine to thirty five point five <ref:2610.13314#pg1>.
Jane: That's a pretty strong lift for the Qwen models, especially looking at the F1 score improvement. But they did mention some trade-offs in those final evaluations?
Lu: They noted that on LongMemEval, NEAR improves accuracy on seven of the nine backbones, including Qwen-4B from sixty-nine point three to seventy-two point one and KaLM-12B from seventy five point eight to seventy seven point four <ref:2610.13314#pg2>. But they also saw some small decreases for Llama 3B and EmbeddingGemma <ref:2610.13314#pg1>.
Meng: Small decreases are worrying for production. If you're deploying this, you need to know exactly where the performance dips, especially on smaller models like Llama 3B <ref:2610.13314#pg1>. What does that tell us about the generality of this neighborhood-aware approach?
Paper summary: Tom: The authors are pretty clear here: they studied the difficulty of retrieving correct evidence from memory records with shared topics or entities. Their key observation was that these similar records weren't just competing candidates; they also provided a local neighborhood for each memory.
Jane: So the paper is really focused on solving that confusion caused by records sharing content, turning that potential noise into useful matching information through these local references.
Lu: That’s the crux of it, they turned this potential source of confusion into useful matching information by constructing memory-specific local references. They also built a compact query-reference table and a fixed memory index, enabling new queries to retrieve memories through a single approximate nearest neighbor search without preliminary candidate retrieval.
Meng: So for deployment, that means the system can handle new queries with just one ANN search over this fixed index? That’s much better than running a full retrieval process every time.
Tom: Right, that's exactly what they show—NEAR obtains query references from a precomputed table and combines residual representations with the original embeddings in a fixed index for a single approximate nearest neighbor search.
Jane: So, to summarize this paper "NEAR: Neighborhood-Aware Residuals for Memory Retrieval," it’s a training-free method that uses local neighborhood information to create residual representations that boost query memory matching and ultimately improve downstream question answering performance by nine point six percentage points on LoCoMo with Qwen3-star.
Lu: And they found that the gains extend beyond just the top ten results, improving recall at every tested budget across both datasets.
Tom: So to wrap up, NEAR is this training-free method that constructs memory-specific local references from shared contents and uses residual representations to complement original embeddings for query memory matching.
Jane: It’s a training-free framework that uses these local references to construct residual representations that complement the original embeddings for query memory matching.
Lu: They found that the evidence outscores Base's top ten in the first residual layer, which is a key finding showing how much power these new representations are adding.
Meng: The system can use one ANN search over a fixed index with a small table of sixteen entries to find memories for any query.
Tom: This allows NEAR to use one ANN search over a fixed index, combining the original and residual matching components into a single inner product representation during retrieval.
Jane: The selection of the fusion weight and residual construction are selected on development data, but those specific configurations are reused for held-out top ten and top five evaluations across multiple residual constructions, which shows robustness.
Conclusion: Tom: So we’re wrapping up on NEAR: Neighborhood-Aware Residuals for Memory Retrieval, looking at what this whole paper is really saying about finding evidence in old conversations and documents.
Jane: It basically shows how you can use the overlap between similar records to create better matching information for a query. It’s all about turning similarity into a useful reference point.
Lu: The authors propose this training-free framework, NEAR, which builds local references from memory neighbors and uses those to complement the original embeddings for query memory matching.
Meng: So they've got this system that doesn't need retraining because it figures out these connections on the fly using some precomputed tables. That’s something I can actually think about deploying at scale.
Tom: It is pretty clean—they use a fixed index and just one approximate nearest neighbor search to find memories for any new query. That cuts down on the time needed to search through everything from scratch every single time you ask a question.
Jane: The implication here is that we don't have to re-rank everything every time we want an answer; we just look at these specific, relevant neighborhood references.
Lu: They found that this method actually helps downstream question answering accuracy by nine point six percentage points on the LoCoMo benchmark with the Qwen three star model. That’s a direct hit on how well the AI can actually use that retrieved evidence to form a final answer.
Meng: Nine point six is a solid lift, but what about those models that didn't see such big gains? They mentioned some small decreases for Llama three b and Embedding Gemma on the LongMemEval test. That’s important for me to know where the limits are.
Tom: Yeah, it means the quality isn't perfectly uniform across every single model out there, which is expected when you introduce these new ways of handling similarity.
Jane: So what we see here is that this neighborhood awareness solves a real headache in retrieval by making sure similar things don't accidentally cancel each other out during the search.
Lu: It’s a way to manage those confusing overlaps between records that share topics or events, turning them from potential confusion into actual matching information.
Tom: So to sum it up, NEAR gives us this training-free tool that uses local neighborhood data to build residual representations for query memory matching.
Jane: And the big win is that these improvements translate directly into better answers when the AI tries to answer a question based on those retrieved memories.
Lu: The work suggests that we can get more out of existing embeddings by adding these learned, context-specific local references without needing to retrain the whole system from scratch.
Meng: So for anyone building something like this, it means you get better retrieval performance with less training overhead, provided you can handle that fixed index setup.
Tom: Next time we talk about this, we’ll look at how they tuned the fusion weight and residual construction to see what settings actually gave them those best results across different tasks.
Southeast University · MemoraX AI Institute of Technology, Leiden University, University of Sydney
cs.IR
Submitted: 2026-10-09
Updated: 2026-11-07
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Importance score: 91/100
The gist: Memory retrieval aims to identify evidence from stored records to answer a query.Such shared contents would make records without the required evidence more likely to receive high similarity scores…
Key concepts
- Memory Retrieval Challenge
- Standard retrieval ranks memories based only on similarity to the query, which can cause records without the right evidence to outrank correct ones when similar records exist. NEAR addresses this by using neighborhood information to provide better matching signals.
- Neighborhood-Aware Residuals (NEAR)
- NEAR is a training-free method that builds local references for each memory by looking at its nearest neighbors. It then removes the embedding component along this reference direction to create residual representations that enhance the original embeddings for matching.
- Query-Reference Table
- A small, precomputed table is built during offline indexing. This table stores query references derived from spherical K-means clustering of memory states. At retrieval time, this table allows NEAR to use a single approximate nearest-neighbor search over a fixed index.
Terminology
Summary
Memory retrieval aims to identify evidence from stored records to answer a query. p. 0Such shared contents would make records without the required evidence more likely to receive high similarity scores and outrank the correct evidence. p. 0In this paper, we use the information shared by these records to construct local references for query–memory matching. p. 0We propose NEighborhood-Aware Residuals (NEAR), a training-free framework that uses these references to construct residual representations that complement the original embeddings. p. 0Experiments on eight long-context retrieval benchmarks show consistent improvements across nine embedding models. p. 0Downstream evaluations show that NEAR improves question-answering performance. p. 0
Pruning redundant reasoning tokens
Memory retrieval enables intelligent assistants to find evidence in past conversations and stored documents to answer current queries.A common approach is embedding-based retrieval, which encodes queries and memories separately and ranks memories by vector similarity. p. 0When ranking relies only on query–memory similarity, such overlap would make records that do not provide the required evidence more likely to outrank correct evidence. p. 0We argue that they also provide a local neighborhood that can serve as a reference for each memory. p. 0We turn this potential source of confusion into useful matching information by constructing memory-specific local references from the relationships among similar memories. p. 0In this paper, we propose NEighborhood-Aware Residuals (NEAR), a training-free framework that uses these local references for query–memory matching. p. 0For each memory, NEAR builds a reference from its nearest neighbors and removes the embedding component along this direction. p. 0A small precomputed table provides references for new queries without an initial search over the memory bank. p. 0NEAR combines original and residual representations so that the final matching score is a single inner product. p. 0Offline indexing enables retrieval through a single approximate nearest-neighbor (ANN) search over the fixed memory index. p. 0
How the training signal is built
For each memory, NEAR builds a reference from its nearest neighbors and removes the embedding component along this direction.For subsequent layers, we repeat this procedure using the normalized residuals h(l)i = unit(r(l)i). p. 3The matching representation either retains the residual length or is normalized: z(l)i = (r(l)i, η = 0, unit(r(l)i), η = 1). p. 3At each layer, NEAR precomputes a small table that maps queries to local reference directions. p. 3Spherical K-means (Dhillon & Modha, 2001) clusters the input memory states, with cluster centers serving as anchors a(l)h. p. 3Similarity-based softmax weights combine their stored directions into a query reference: p(l)h(q) = explbraceβ(h(l−1)q)⊤a(l)h Pj∈H(l)q explbraceβ(h(l−1)q)⊤a j. p. 3 The same projection and normalization rules yield the query matching representation z(l)q and next-layer state h(l)q. p. 3
Where the gains come from
NEAR constructs residual representations that complement the original embeddings for query–memory matching.The evidence outscores Base’s top ten in the first residual layer. p. 6 p. 6Fusing all residual scores with original matching moves it from rank 75 to rank 1. p. 6Local references define the projection directions. p. 6The evidence outscores Base’s top ten in the first residual layer.
We propose NEAR, a training-free method that constructs memory-specific local references from shared contents and uses the resulting residual representations to complement the original embeddings for query–memory matching.We construct a compact query-reference table and a fixed memory index, enabling new queries to retrieve memories through a single ANN search without preliminary candidate retrieval. p. 1
We demonstrate improvements in mean Recall@10 and NDCG@10 across eight retrieval benchmarks and nine embedding models.NEAR improves mean Recall@10 and NDCG@10 over the original embeddings and the evaluated same-backbone baselines. p. 0Downstream evaluations further show that NEAR improves question-answering accuracy by 9. p. 06 percentage points on LoCoMo with Qwen3-star. p. 0NEAR obtains query references from a precomputed table and combines residual representations with the original embeddings in a fixed index for a single approximate nearest-neighbor search. p. 8
We evaluate NEAR on eight long-context retrieval benchmarks using nine frozen embedding models.Across all nine models, NEAR improves mean Recall@10 and NDCG@10 over Base for all tested methods. p. 7NEAR achieves the best same-backbone scores on both metrics for ESG-Reports with Qwen-4B and Qwen-8B, and the highest same-backbone NDCG@10 for PeerQA at all three Qwen sizes. p. 4
NEAR improves both Accuracy and F1 across all nine backbones using original conversation histories of LoCoMo and LongMemEval.For Qwen-4B, Accuracy increases from 62. p. 47 to 72.3, while F1 rises from 30. p. 49 to 35.5.The retrieval gains can translate into better answers to questions about conversation histories. p. 5On LongMemEval, NEAR improves Accuracy on seven of the nine backbones, including Qwen-4B from 69. p. 53 to 72.1 and KaLM-12B from 75. p. 48 to 77.4; Llama-3B and EmbeddingGemma show small decreases. p. 5
We study the effects of fusion weight, residual normalization, and depth using Qwen3-Embedding-4B on the complete query sets of LoCoMo and LongMemEval.As shown in Figure 3, all displayed configurations outperform Base in both Recall@10 and NDCG@10 on both tasks throughout the tested weight range. p. 7The tasks favor different residual constructions: unnormalized residuals perform better on LoCoMo, whereas normalized residuals achieve higher NDCG@10 at moderate and large weights on LongMemEval. p. 7
Uncompressed NEAR reaches mean Recall@10 and NDCG@10 of 71.82 and 59.21, compared with 57. p. 8 p. 8 p. 891 and 45.88 for Base. p. 1 p. 1 p. 1NEAR therefore trades additional retrieval computation for improved quality rather than reducing query latency. p. 8Uncompressed NEAR has a median total-storage ratio of 2. p. 8 p. 802 relative to FP32 Base. p. 8 p. 8SQ4 reduces NEAR’s total assets by a median 82. p. 8 p. 87%, while mean Recall@10 changes from 71. p. 8 p. 882 to 71.97 and NDCG@10 from 59. p. 0 p. 021 to 59.16.
NEAR obtains query references from a precomputed table and combines residual representations with the original embeddings in a fixed index for a single approximate nearest-neighbor search.This allows NEAR to use one ANN search over a fixed index. p. 2 p. 2 p. 2The query-reference table contains at most 16 entries and is constructed from the memory embeddings using spherical K-means. p. 3 p. 3 p. 3At retrieval time, the original and residual matching components are combined into a single inner-product representation, allowing NEAR to use one ANN search over a fixed index. p. 2 p. 2
The fusion weight and residual construction are selected on development data.The selected configurations are reused for held-out Top-10 and Top-5 evaluation. p. 4NEAR maintains gains over Base across multiple residual constructions and fusion weights, rather than only at a particular configuration. p. 7
We report Top-5 results using the configurations selected for Top-10 evaluation.NEAR improves Recall@K and NDCG@K over Base at every tested budget on both datasets. p. 7The retrieval gains therefore extend beyond the Top-10 setting used in the main comparison. p. 7 p. 7
NEAR's median query latency with SQ4 is 0.590ms, close to its uncompressed latency, so the measured benefit is primarily in storage. p. 8Storage and quantization. p. 7 Uncompressed NEAR has a median total-storage ratio of 2.02 relative to FP32 Base.SQ4 reduces NEAR’s total assets by a median 82.7%, while mean Recall@10 changes from 71.82 to 71.97 and NDCG@10 from 59.21 to 59.16.The retrieval gains therefore extend beyond the Top-10 setting used in the main comparison.
We compare retrieval quality, storage, and query latency using Qwen-0.6B/4B on LoCoMo and LongMemEval. p. 3 p. 3Offline construction starts from cached embeddings. p. 8NEAR (SQ4) reaches mean Recall@10 and NDCG@10 of 75. p. 8 p. 823 and 59.21, compared with 57.91 and 45.88 for Base.Median query latency increases from 0. p. 8 p. 8242ms to 0. p. 8 p. 8579ms, an additional 0. p. 8 p. 8337ms, or approximately 2. p. 8 p. 839 times the Base latency. p. 8 p. 8
We study the difficulty of retrieving correct evidence from memory records with shared topics, entities, or events.Our key observation was that these similar records were not only competing candidates; they also provided a local neighborhood that could serve as a reference for each memory. p. 8 p. 8In this paper, we proposed NEAR, a training-free framework that turned this potential source of confusion into useful matching information by constructing memory-specific local references. p. 8 p. 8
NEAR obtained query references from a precomputed table and combined residual representations with the original embeddings in a fixed index for a single approximate nearest-neighbor search.This allows NEAR to use one ANN search over a fixed index.The query-reference table contains at most 16 entries and is constructed from the memory embeddings using spherical K-means.At retrieval time, the original and residual matching components are combined into a single inner-product representation, allowing NEAR to use one ANN search over a fixed index.
We evaluate 5,873 queries across eight benchmarks.Following LMEB, we report capped Recall@10 and NDCG@10 with linear relevance gains. p. 4We fix the neighborhood width to 50, the reference-table size to 16, the number of routing entries to four, and the inverse temperature to 10. p. 3Depth, residual normalization, and total residual weight are selected on development data. p. 5
NEAR obtains query references from a precomputed table and combines residual representations with the original embeddings in a fixed index for a single approximate nearest-neighbor search.This allows NEAR to use one ANN search over a fixed index.The query-reference table contains at most 16 entries and is constructed from the memory embeddings using spherical K-means.
We compare retrieval quality, storage, and query latency using Qwen-0.6B/4B on LoCoMo and LongMemEval.NEAR (SQ4) reaches mean Recall@10 and NDCG@10 of 75.23 and 59.21, compared with 57.91 and 45.88 for Base.Median query latency increases from 0.242ms to 0.579ms, an additional 0.337ms, or approximately 2.39 times the Base latency.
We study the difficulty of retrieving correct evidence from memory records with shared topics, entities, or events.Our key observation was that these similar records were not only competing candidates; they also provided a local neighborhood that could serve as a reference for each memory.In this paper, we proposed NEAR, a training-free framework that turned this potential source of confusion into useful matching information by constructing memory-specific local references.</ref:2610. p. 0
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Neighborhood-Aware Residuals for Memory Retrieval (NEAR) can be used to construct residual representations that complement original embeddings, leading to a single approximate nearest-neighbor search over a fixed memory index instead of preliminary candidate retrieval.
-
The system can leverage memory-specific local references derived from nearest neighbors and a precomputed table for queries, allowing new queries to obtain their reference without an initial search over the entire memory bank.
-
NEAR improves mean Recall at 10 and NDCG at 10 across eight long-context retrieval benchmarks and nine embedding models, with downstream evaluations showing improvements in question answering accuracy by up to 9.6 percentage points on LoCoMo with Qwen3-Embedding-4B.
-
The system can be optimized for storage using four-bit scalar quantization (SQ4) while retaining retrieval quality, achieving a median total-storage ratio of 2.02 relative to FP32 Base and reducing assets by a median 82.7%.
-
The system can perform efficient query processing by combining original and residual representations into a static memory key and query vector for a single inner product search, enabling retrieval with query latency around 0.590ms with SQ4 quantization on LongMemEval.
Sources
- Llama-Embed-Nemotron-8B: A Universal Text Embedding Model for Multilingual and Cross-Lingual Tasks
- REALTALK: A 21-Day Real-World Dataset for Long-Term Conversation
- PRQ-KMeans: Projection Residual Quantization for Semantic ID Tokenization
- ViDoRe Benchmark V2: Raising the Bar for Visual Retrieval
- NV-Retriever: Improving text embedding models with effective hard-negative mining
- EvoEmbedding: Evolvable Representations for Long-Context Retrieval and Agentic Memory
- EmbeddingGemma: Powerful and Lightweight Text Representations
- Qwen3 Technical Report
- Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models
- LMEB: Long-horizon Memory Embedding Benchmark
Related papers
- Revisiting Text Ranking in Deep Research
- GateSID: One Gate to Balance Semantic and Collaborative Signals across the Item Lifecycle
- SAM-D2Q: Aligning Multimodal Doc2Query with Search Demand and Conversion for E-commerce
- GINGER: Spicing Up Graph-Based Vector Search
- Subtraction Gets You More: Gap-Aware Retrieval for Multimodal Multi-Hop QA
- One-Step Retrieval Framework for Real-Time Sponsored Search Ads Using Hierarchical Text Representations