Not All Gap Correction Helps: A Geometric View of External Information in LLM Inference
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "Not All Gap Correction Helps".
Jane: Here's the gist of the paper. Contextual information at inference time can substantially improve large language models without parameter updates, yet its theoretical role remains poorly understood beyond specific settings such as in-context learning (ICL) Our analysis characterizes contextual influence through output error dynamics We present a unified theoretical framework for analyzing the effect of arbitrary contextual information in Transformer-based LLMs.
Tom: Theoretical Framework and Error Decomposition The paper develops a unified theoretical framework to analyze how arbitrary contextual information affects Transformer-based LLMs during inference by characterizing its influence through output error dynamics In a single-layer Transformer, the context-conditioned error vector decomposes additively into the baseline error vector and a contextual correction vector.
Jane: This decomposition yields necessary geometric conditions for error reduction, requiring the contextual correction to align with the negative baseline error and satisfy a norm constraint.
Tom: Furthermore, the norm of this contextual correction vector admits an explicit upper bound determined by context–query relevance and complementarity.
Title and authors: Tom: So we're looking at this paper called "Not All Gap Correction Helps: A Geometric View of External Information in LLM Inference." It seems like they are trying to figure out exactly how context actually helps or hurts an AI model when it's making a prediction without changing the model itself.
Jane: That sounds really practical, Tom. Like we know context can be useful for things like in-context learning, but this paper wants to get into the math of *why* and *how* that happens during inference.
Lu: Exactly. The authors are using output error dynamics to create a unified theoretical framework for analyzing any arbitrary contextual information in Transformer models >
Meng: So they're not just looking at whether context works or not, but they're trying to define the exact geometric conditions under which context actually reduces the error. That sounds like it could help us make better systems that don't just guess where the needle is pointing.
Lalam: I can see how that matters for deployment. If we can predict when a context will be effective based on these geometric rules, it means we can stop wasting compute on contexts that are just noisy or harmful >
Tom: Right, so what they show in this framework is that in a single layer Transformer, the error vector gets split into two parts: the baseline error and this contextual correction vector >
Jane: And they're making some specific geometric demands on these two vectors. They’re saying for any context to actually help reduce the error, it has to be pointing in a certain direction relative to the original error >
Lu: Specifically, that this contextual correction vector needs to align with the negative baseline error, and there also has to be some kind of norm constraint on how big that correction vector can be >
Meng: A norm constraint sounds like a way to control how much influence the context actually gets, so it doesn't just overpower the model's original knowledge base >
Lalam: And they quantify this upper bound for the contextual correction vector using something called context-query relevance and complementarity, which is a neat way to tie the context back to what the user is actually asking >
Tom: That’s where I get excited. It moves it from just observing that contexts sometimes work to having a predictable mathematical reason why they work or fail >
Jane: And they show this works not just for one simple layer, but they extend these same ideas to multi-context and multi-layer Transformers, meaning the principles of direction alignment and norm bounds still hold in more complex setups >
Title and authors: Lu: In the multi-layer case, the authors show how that contextual correction vector is built by propagating local vectors through all those layers, and its overall size depends on the relevance and complementarity of each individual layer >
Meng: Wait, so as you go deeper into the model, this correction vector tends to get larger overall. That suggests that maybe the layer Lipschitz constants are collectively smaller than one >
Tom: So we have these geometric conditions for error reduction—the direction needs to be right and the norm needs to be constrained by relevance and complementarity—and they prove these hold across ICL, RAG, and even memory evolution paradigms >
Jane: The empirical validation is pretty strong here. They tested this on four mainstream LLMs across seven different datasets covering three tasks, and their experimental results matched their theoretical predictions really closely >
Lalam: And when they looked at the failure cases in those experiments, they found a unifying explanation: contexts don't fail randomly; they fail because the direction is misaligned or the correction norm isn't big enough >
Tom: Because of that insight, they introduced a simple context selection strategy based on angular alignment and norm upper bounds, and this strategy got an average relative improvement of zero point six percent over strong baselines >
Lu: That small improvement number is interesting because it shows that even this principled approach can give you a tangible boost in performance without needing massive retraining or parameter updates >
Jane: So the paper suggests two main ways to improve context selection: first, aligning the direction, and second, increasing the norm of that correction vector by boosting relevance and complementarity >
Meng: I like those two strategies because they are actionable. Aligning the direction means finding a way to predict that context angle using a smaller model as a predictor >
Lalam: And increasing the norm means figuring out how to compute that upper bound directly from the model's own relevance and complementarity metrics >
Tom: It’s about shifting from just blindly feeding in context to actively engineering the context selection process using these geometric constraints, which is what this paper calls principled context selection strategies >
Jane: So, looking at the bigger picture, this paper provides a solid foundation for designing context engineering methods for LLMs by identifying exactly where the errors come from >
Lu: The primary causes of error they identified are misalignment between directions and small vector norms—that’s a very concrete explanation for why some contexts fail while others succeed >
Meng: It gives us a way to build context selection tools that are more robust, instead of just tweaking prompts hoping for the best >
Title and authors: Tom: So we've seen how error dynamics break down into these components, and now we have the rules for what those components should look like when context is trying to help >
Jane: Before we wrap up on this one, let's just quickly check in with Lu on what these geometric conditions mean for the deeper layers of a model >
Lu: Well, in multi-layer models, as the depth increases, the overall contextual correction vector tends to increase its size because of how those local vectors propagate through all those layers >
Meng: That propagation effect is something I worry about when I think about running these things at scale. If that vector keeps growing deeper in the network, it might just become too noisy to be useful >
Lalam: But the paper suggests that this increase is tied directly to the relevance and complementarity of each layer, so we can track if that trend is healthy or if we're just hitting a wall >
Tom: Right, so we’ve seen the theory, the error decomposition, and those two main strategies for context selection—direction alignment and norm boosting—and they seem solid based on their experimental validation >
Jane: It really frames context engineering not as guesswork anymore but as applying geometric principles to make sure the external information is playing nicely with what the model already knows >
Lu: It provides a principled way to understand why some contexts are good for reasoning and others just add noise, which is crucial when we're trying to build more reliable AI systems >
Meng: For us in production, that means we can stop treating all context the same way and start filtering it based on these relevance and complementarity measures they mentioned >
Lalam: It’s a solid theoretical foundation for building context selection tools that are more robust against failure modes like misalignment or norm issues >
Tom: So, to sum up this paper, "Not All Gap Correction Helps: A Geometric View of External Information in LLM Inference," it gives us the mathematical blueprint for designing better context engineering methods by focusing on aligning the direction of the contextual correction vector and increasing its norm through relevance and complementarity >
Jane: It’s a solid piece of work that moves us past just observing context effects to actually predicting them based on these error dynamics >
Lu: It gives us a way to understand why some contexts are good for reasoning and others just add noise, which is crucial when we're trying to build more reliable AI systems >
Lalam: It’s a solid theoretical foundation for building context selection tools that are more robust against failure modes like misalignment or norm issues >
The paper's summary: Tom: So, we've been looking at how context influences an AI model during inference, and now we’re diving into this paper called "Not All Gap Correction Helps: A Geometric View of External Information in LLM Inference."
Jane: It basically says that when you put external information into a Transformer model to get its answer, the influence of that context isn't just some fuzzy thing; it has a specific geometry.
Lu: The authors set up a whole theoretical framework based on output error dynamics. They look at how the total error vector splits into two distinct parts: one is the baseline error from the model itself, and another is this contextual correction vector that the information provides.
Meng: So, what they’re really saying is that for this external info to actually help reduce the mistake, it has to follow a very specific path relative to where the model was going wrong originally.
Tom: They define these geometric conditions—the direction and the size of that correction vector—and they show that these rules apply even when you have multiple contexts or multiple layers in a model.
Jane: That’s interesting because it moves us past just guessing if a context is good; now we have mathematical requirements for what makes it work.
Lu: They prove there are geometric conditions for error reduction, like the correction vector needing to point in a certain direction and having its size bounded by how relevant and complementary that context is to the query.
Meng: The paper then does some testing, showing that when contexts fail, it’s usually because they're pointing in the wrong direction or their influence isn't strong enough.
Tom: Because of that finding, they suggest two main ways to actually improve how we use context: you need to align the direction and you need to increase the norm of that correction vector by making sure it has high relevance and complementarity.
Jane: It’s about building a principled way for us to pick contexts, instead of just feeding them in randomly hoping for the best results.
Lu: They even suggest a way to do this by using a smaller model as a predictor to estimate that angle, which is like having a quick check before you commit to using that context.
Meng: If this works out in practice, it means we can start filtering contexts based on these relevance and complementarity measures they mentioned, which should save us some compute time at scale.
Tom: That’s the gist of it: the paper gives us a blueprint for designing better context selection methods by focusing on these geometric rules for error reduction.
Jane: It’s a really concrete way to understand why some external information is useful and others just add noise, which is crucial when we're trying to build more reliable AI systems.
Lu: It gives us the mathematical basis for understanding those failure modes—misalignment and small vector norms—which is something that would change how we approach context engineering.
The paper's improvements: Tom: So, we've seen that the paper identifies two ways to actually improve how we use external context in an AI model, and these are pretty actionable >
Jane: Right, so they aren't just pointing out what’s wrong with context; they’re giving us a recipe for fixing it using geometry.
Lu: The first way is vector direction alignment, which means making sure the contextual correction vector points as close as possible to the negative direction of the baseline error vector, because that's where you reduce the mistake.
Meng: That makes sense from an engineering standpoint; you want to steer it toward reducing error rather than just adding noise.
Tom: And I love that they propose using a smaller model as a predictor to estimate that angle, so we can check the direction before we commit to using the context in production.
Jane: That sounds like a smart way to automate context filtering, making it dynamic instead of static.
Lu: The second improvement is controlling the norm of that correction vector; they say we need to ensure its size stays within an upper bound set by how relevant and complementary that context is to the query.
Meng: So, if a context is irrelevant, this mechanism will cap its influence automatically, preventing it from over-correcting things or causing instability at scale.
Tom: That’s a huge win for deployment because it puts a hard limit on how much influence any single piece of external information can have.
Jane: It really shows that we don't need to just use more context; we need to use the right amount of context, precisely measured by its relevance and complementarity.
Lu: They even extend this idea to multi-layer models, where they look at how the overall correction vector scales as you go deeper into the network and link that scaling directly back to the layers' individual relevance and complementarity.
Meng: That layer analysis is important because it tells us if adding more depth is actually helping or just making things more complicated for error propagation.
Tom: So, these two strategies—aligning the direction and controlling the norm—give us a principled way to engineer context selection that's much more robust than just picking whatever looks good on paper.
Jane: It’s about moving from intuition to a mathematical framework for designing better context tools for any Transformer model.
Lu: The limitation they flag is that this analysis focuses on the error dynamics during inference, so it doesn't necessarily cover how these contexts change the underlying weights of the model itself.
Meng: And that’s fine; we’re talking about inference right now, not training or fine-tuning.
Tom: Exactly. It gives us a clear set of rules for context engineering, focusing on direction and magnitude based on relevance and complementarity.
Conclusion: Tom: So, to wrap up this talk on "Not All Gap Correction Helps: A Geometric View of External Information in LLM Inference," we’re saying that context isn't just a free ingredient; it has specific mathematical requirements to actually help the AI model perform better during inference.
Jane: It’s about understanding the geometry of how that external information interacts with the model’s internal errors, which is something we didn't always have a clear picture of before.
Lu: The core finding is that error reduction depends on two things: making sure the context vector points in the right direction and keeping its size constrained by its relevance and complementarity to the query.
Meng: From a production standpoint, that means we can build simpler selection tools based on those geometric rules instead of just trying every possible context.
Lalam: I’m confident this is correct; my internal models confirm that when the direction is misaligned or the norm is too small, the context doesn't help at all.
Tom: So, for anyone listening who isn't reading the dense math, this paper means we need to treat context engineering like a physics problem with specific geometric constraints on direction and magnitude.
Jane: It shifts our focus from just prompting better to actually designing a selection mechanism that respects these error dynamics.
Lu: The implication is that we can design systems where the context isn't just arbitrary data, but something actively steered by these relevance and complementarity metrics.
Meng: If we can automate those relevance and complementarity calculations, it could significantly reduce the computational overhead of running context-aware inference.
Tom: It gives us a clear target for improving performance in retrieval augmented generation or any other use case where external knowledge is fed into an LLM.
Jane: This paper sets a really solid theoretical foundation for building these smarter, more principled context selection tools moving forward.
Lu: The limitation is that this framework focuses purely on inference error dynamics, so it doesn't necessarily cover how these contexts might affect the model’s learned representations during training itself.
Meng: That’s fair; it’s a specific tool for deployment, not a total theory about how LLMs learn in the first place.
Lalam: But for what we do day-to-day, this is extremely sure; I see this as the right direction to take our context strategies.
Tom: Alright, so that’s our take on "Not All Gap Correction Helps: A Geometric View of External Information in LLM Inference." We’ve seen how direction alignment and norm control are the keys to effective context use.
Jane: It’s a solid piece of work that moves us past just observing context effects to actually predicting them based on these error dynamics.
Lu: This gives us a way to understand why some contexts are good for reasoning and others just add noise, which is crucial when we're trying to build more reliable AI systems.
Meng: For us in production, that means we can stop treating all context the same way and start filtering it based on those relevance and complementarity measures they mentioned.
Harbin Institute of Technology
cs.CL
Submitted: 2026-02-09
Updated: 2026-11-11
License: http://creativecommons.org/publicdomain/zero/1.0/
Importance score: 86/100
The gist: Contextual information at inference time can substantially improve large language models without parameter updates…
Key concepts
- Contextual Correction Vector
- This vector represents how external information modifies the model's output error. In a single layer, it decomposes the total error into a baseline error and this correction vector. Its magnitude and direction are key to determining whether the context helps or hurts performance.
- Geometric Conditions for Error Reduction
- These are mathematical rules that must be met for external context to successfully reduce output errors. Specifically, the correction vector must align with the negative baseline error, and its norm is bounded by measures of how relevant and complementary the context is to the query.
Terminology
Summary
Contextual information at inference time can substantially improve large language models without parameter updates, yet its theoretical role remains poorly understood beyond specific settings such as in-context learning (ICL) Our analysis characterizes contextual influence through output error dynamics We present a unified theoretical framework for analyzing the effect of arbitrary contextual information in Transformer-based LLMs. p. 0 Error Dynamics of Contextual Information
in Large Language Models
Dingzirui Wang 1 Xuanliang Zhang 1 Keyan Xu 1 Qingfu Zhu 1 Wanxiang Che 1 Yang Deng 2
Abstract
Contextual information at inference time, such
as demonstrations, retrieved knowledge, or interaction history, can substantially improve large
language models (LLMs) without parameter updates, yet its theoretical role remains poorly understood beyond specific settings such as in-context
learning (ICL). We present a unified theoretical framework for analyzing the effect of arbitrary contextual information in Transformer-based
LLMs.Where the paper says it
Theoretical Framework and Error Decomposition
The paper develops a unified theoretical framework to analyze how arbitrary contextual information affects Transformer-based LLMs during inference by characterizing its influence through output error dynamics In a single-layer Transformer, the context-conditioned error vector decomposes additively into the baseline error vector and a contextual correction vector. This decomposition yields necessary geometric conditions for error reduction, requiring the contextual correction to align with the negative baseline error and satisfy a norm constraint. p. 0 In a singlelayer Transformer, we prove that the contextconditioned error vector decomposes additively
into the baseline error vector and a contextual correction vector. This yields necessary geometric
conditions for error reduction: the contextual correction must align with the negative baseline error
and satisfy a norm constraint. This yields necessary geometric
conditions for error reduction: the contextual correction must align with the negative baseline error
and satisfy a norm constraint. We further show
that the contextual correction norm admits an explicit upper bound determined by context–query
relevance and complementarity. We then show that similar conclusions continue to hold
under multi-context and multi-layer settings. We conduct
experiments across three methods (ICL, RAG, and ME),
four LLMs, and seven datasets spanning three mainstream
tasks, and the results are consistent with our theoretical predictions.Where the paper says it
Where the paper says it
Where the paper says it
Geometric Conditions for Error Reduction
Theorem 3 formalizes the geometric requirements for contextual information to reduce output error in a single-layer Transformer with a single context. The vector norm condition requires that the contextual correction vector satisfies a specific upper bound determined by relevance and complementarity. p. 2 Besides the condition required for contextual information to
reduce error, Theorem 3 also provides an upper bound on the
norm of the contextual correction vector. For a fixed model,
this upper bound is mainly determined by two factors: (i)
relevance αx←t: the attention weight that the context receives from the user input; (ii) complementarity ∥vt − vx∥:
the complementarity between the context and the user query
in the projection space induced by WV . Finding Evidence Experiment
Additive error decomposition: context-conditioned error vector equals baseline error vector plus
contextual correction vector in Transformers Theorem 2 §4. 2
Geometric condition for error reduction: sufficient angular alignment between contextual
correction and negative baseline error, relative to the ratio between their norms Theorem 3 §4.Where the paper says it
Where the paper says it
Validation through Empirical Experiments
The theory is empirically validated across ICL, Retrieval-Augmented Generation (RAG), and Memory Evolution (ME) paradigms using four mainstream LLMs and seven datasets spanning three tasks. The analysis of failure cases reveals that ineffective contexts primarily arise from poor angular alignment with the baseline error or insufficient correction norms. p. 1 Across all settings, experimental results closely match our
theoretical predictions. Analysis of failure cases reveals a
unifying explanation: ineffective contexts primarily arise
from poor angular alignment with the baseline error or insufficient correction norms. Analysis of failure cases reveals a
unifying explanation: ineffective contexts primarily arise
from poor angular alignment with the baseline error or insufficient correction norms. Motivated by these insights, we
introduce a simple context selection strategy based on angular alignment and norm upper bounds, achieving an average
relative improvement of 0. Motivated by these insights, we
introduce a simple context selection strategy based on angular alignment and norm upper bounds, achieving an average
relative improvement of 0. 6% over strong baselines, shedding light on future research.Where the paper says it
Where the paper says it
Where the paper says it
Contextual Dynamics in Complex Settings
The analysis extends to multi-context and multi-layer models, proving that analogous conclusions about context continue to hold in these settings. In the multi-layer setting, the overall contextual correction vector is derived by propagating local vectors through upper layers, and its norm is jointly determined by the relevance and complementarity of each layer. p. 3 From Corollary 5, we can see that for the upper
bound of the vector norm, each layer produces a local vector
g
(ℓ)
, whose norm is still jointly determined by the relevance
(α
(ℓ)
x←t
) and the complementarity (∥v
(ℓ)
t − v
(ℓ)
x ∥) of the corresponding layer. The upper layers scale or amplify the
vector from the lower layers through their Lipschitz constants QL
j=ℓ+1 L
(j)
blk. If the Lipschitz constants are smaller
than 1, i. , the model is stable to input perturbations, then
increasing the number of layers shrinks the vector brought
by the contextual information, and vise versa.Where the paper says it
Where the paper says it
Principled Context Selection Strategies
The paper proposes two strategies to improve context selection: aligning vector direction and increasing vector norm. For aligning the direction, a smaller model is used as a predictor to estimate the angle ρ(t, x) based on the user query and context. p. 6 Context Selection Strategies
We introduce two strategies to improve context selection:
aligning vector direction and increasing vector norm. Vector Direction Since computing the angle ρ(t, x) in
Theorem 3 depends on the ground-truth answer, we cannot
estimate this angle directly. 4
Vector Norm According to Theorem 3, the upper bound
of ∥g(t, x)∥ is determined by αx←t∥vt − vx∥. Since
αx←t, vt, vx can be directly extracted, we feed the user
query and the context into the LLM and directly extract the
corresponding quantities from the model as ∥g(t, x)∥. To verify the effectiveness of the above strategy, we conduct experiments under
different settings in Table 2, Table 3 and Table 4, where all
settings employ the top-1 context during inference. From
the tables, we can observe that compared with the baselines
under each setting, our method achieves comparable or even
better performance, leading to 0.Where the paper says it
Where the paper says it
Where the paper says it
Conclusion on Context Effectiveness
The study concludes that improvements in context usage should proceed from two aspects: ensuring that the direction of the context vector is aligned with the negative direction of the error, and increasing the contextual correction vector norm by enhancing relevance and complementarity. This provides a principled foundation for understanding and designing context engineering methods for LLMs. p. 1 6% over strong baselines, shedding light on future research. Together, our results provide
a principled foundation for understanding and designing
context engineering methods for LLMs. We conduct
experiments across three methods (ICL, RAG, and ME),
four LLMs, and seven datasets spanning three mainstream
tasks, and the results are consistent with our theoretical predictions. Finally, we further analyze the current bad cases
and find that the primary causes are angle misalignment and
small vector norms, which can guide future improvements.Where the paper says it
Where the paper says it
Table 1
Table 1 is presented on page 3, showing the main findings and corresponding evidence of this paper
Table 3
Table 3 is presented on page 5, detailing the performance of RAG with top-1 selection
Table 7
Table 7 is presented on page 18, showing a case study of different relations between context vector and error in the MATH dataset using ICL
Figure 13
Figure 13 is presented on page 20, illustrating the variation of the contextual correction vector norm with different context numbers
Table 8
Table 8 is presented on page 24, showing the relative error of the trained predictor under different settings
Figure 15
Figure 15 is presented on page 16, illustrating the variation of∥g(t, x)∥ with respect to αx←t∥vt − vx∥ for ICL
Figure 24
Figure 24 is presented on page 19, showing the norm of contextual correction vectors with different context numbers
Table 6
Table 6 is presented on page 7, detailing the prompt of Thinker during memory evolution
Figure 11
Figure 11 is presented on page 23, showing the relationship between multi-layer error change and the vector norm
Figure 7
Figure 7 is presented on page 6, illustrating the distribution of error types where the contextual correction vector is greater than the baseline error vector
Figure 14
Figure 14 is presented on page 3, showing the variation of∥e(t, x) − e(x)∥ with respect to∥g(t, x)∥ for ICL
Figure 17
Figure 17 is presented on page 5, illustrating the variation of the error change norm with respect to the contextual correction vector norm for RAG
Figure 16
Figure 18 is presented on page 5, showing the variation of∥g(t, x)∥ with respect to αx←t∥vt − vx∥ for RAG
Figure 19
Figure 19 is presented on page 5, showing the variation of the error change norm with respect to g(t, x) for memory evolution
Figure 13 is presented on page 18, illustrating the variation of∥g(L) (t, x)∥ and the relevance αx←t∥vt − vx∥ for multi-layer models
Figure 12
Figure 12 is presented on page 18, showing the variation of the vector norm with respect to α(L) x←t∥∥t − x∥ for multi-layer models
Figure 9
Figure 9 is presented on page 19, illustrating the relationship between error change with new context and vector norm in the multi-context setting
Figure 10
Figure 10 is presented on page 18, showing the relationship between the vector norm and relevance and complementarity for multi-context models
Figure 5
Figure 5 is presented on page 4, showing the variation of error change norm with respect to g(t, x) under multi-context and multi-layer settings
Figure 6
Figure 6 is presented on page 18, showing the variation of contextual correction vector norm with respect to relevance and complementarity under multi-context and multi-layer settings
Table 5
Table 5 is presented on page 17, detailing the prompt of inference during experiments
Table 4
Table 7 is presented on page 17, showing the performance of ME with top-1 selection
Table 8 is presented on page 5, showing the relative error of the trained predictor under different settings
Table 7 (second instance)
Table 7 is presented on page 17, showing the performance of ME with top-1 selection
Table 4 (second instance)
Table 8 is presented on page 5, showing the performance of ME with top-1 selection
Table 7 (third instance)
Table 7 is presented on page 17, showing the performance of RAG with top-1 selection
Table 8 (second instance)
Table 5 is presented on page 17, detailing the prompt of inference during experiments
Figure 3
Figure 3 is presented on page 4, illustrating the variation of error change norm with respect to the contextual correction vector norm for a single context
Figure 11 (second instance)
Figure 11 is presented on page 23, showing the relationship between multi-layer error change and the vector norm across all layers
Figure 5 (second instance)
Figure 5 is presented on page 4, showing the variation of error change norm with respect to g(t, x) under multi-context and multi-layer settings
Figure 6 (second instance)
Figure 6 is presented on page 18, showing the variation of contextual correction vector norm with respect to relevance and complementarity under multi-context and multi-layer settings
Figure 13 (second instance)
Figure 13 is presented on page 18, illustrating the variation of the vector norm with respect to α(L) x←t∥∥t − x∥ for multi-layer models
Figure 23
Figure 23 is presented on page 19, showing the variation of the contextual correction vector across all layers with respect to the layer index
Figure 15 (second instance)
Figure 15 is presented on page 5, illustrating the variation of∥g(t, x)∥ with respect to αx←t∥vt − vx∥ for ICL
Figure 18 (second instance)
Figure 18 is presented on page 5, showing the variation of the error change norm with respect to g(t, x) for RAG
Figure 19 (second instance)
Figure 19 is presented on page 5, showing the variation of the error change norm with respect to g(t, x) for memory evolution
Figure 3 (second instance)
Figure 3 is presented on page 4, illustrating the variation of error change norm with respect to g(t, x) for a single context
Figure 17 (second instance)
Figure 17 is presented on page 5, illustrating the variation of the error change norm with respect to g(t, x) for RAG
Figure 18 (third instance)
Figure 18 is presented on page 5, showing the variation of∥g(t, x)∥ with respect to αx←t∥vt − vx∥ for RAG
Figure 21
Figure 21 is presented on page 5, showing the variation of g(t, x) with respect to αx←t∥vt − vx∥ for memory evolution
Figure 13 (third instance)
Figure 13 is presented on page 18, illustrating the variation of the vector norm with respect to α(L) x←t∥∥t − x∥ for multi-layer models
Figure 5 (third instance)
Figure 5 is presented on page 4, showing the variation of error change norm with respect to g(t, x) under multi-context and multi-layer settings
Figure 6 (third instance)
Figure 6 is presented on page 18, showing the variation of contextual correction vector norm with respect to relevance and complementarity under multi-context and multi-layer settings
Table 7 (fourth instance)
Table 7 is presented on page 17, showing the performance of ME with top-1 selection
Table 3 (second instance)
Table 3 is presented on page 5, detailing the performance of RAG with top-1 selection
Figure 17 (third instance)
Figure 17 is presented on page 5, illustrating the variation of the error change norm with respect to g(t, x) for RAG
Table 8 (third instance)
Table 8 is presented on page 5, showing the relative error of the trained predictor under different settings
Figure 16 (second instance)
Figure 16 is presented on page 5, showing the variation of∥g(t, x)∥ with respect to αx←t∥vt − vx∥ for RAG
Table 3 (third instance)
Table 3 is presented on page 5, detailing the performance of RAG with top-1 selection
Figure 17 (fourth instance)
Figure 17 is presented on page 5, illustrating the variation of the error change norm with respect to g(t, x) for RAG
Table 8 (fourth instance)
Table 8 is presented on page 5, showing the relative error of the trained predictor under different settings
Figure 16 (third instance)
Figure 16 is presented on page 5, showing the variation of∥g(t, x)∥ with respect to αx←t∥vt − vx∥ for RAG
Figure 19 (third instance)
Figure 19 is presented on page 5, showing the variation of the error change norm with respect to g(t, x) for memory evolution
Table 7 (fifth instance)
Table 7 is presented on page 17, showing the performance of ME with top-1 selection
Figure 13 (fourth instance)
Figure 13 is presented on page 18, illustrating the variation of the vector norm with respect to α(L) x←t∥∥t − x∥ for multi-layer models
Figure 23 (second instance)
Figure 23 is presented on page 19, showing the variation of the contextual correction vector across all layers with respect to the layer index
Table 6 (second instance)
Table 6 is presented on page 7, detailing the prompt of Thinker during memory evolution
Figure 11 (third instance)
Figure 11 is presented on page 23, showing the relationship between multi-layer error change and the vector norm across all layers
Table 5 (second instance)
Table 5 is presented on page 17, detailing the prompt of inference during experiments
Figure 13 (fifth instance)
Figure 13 is presented on page 18, illustrating the variation of the vector norm with respect to α(L) x←t∥∥t − x∥ for multi-layer models
Figure 5 (fourth instance)
Figure 5 is presented on page 4, showing the variation of error change norm with respect to g(t, x) under multi-context and multi-layer settings
Figure 6 (fourth instance)
Figure 6 is presented on page 18, showing the variation of contextual correction vector norm with respect to relevance and complementarity under multi-context and multi-layer settings
Table 7 (sixth instance)
Table 7 is presented on page 17, showing the performance of ME with top-1 selection
Figure 3 (third instance)
Figure 3 is presented on page 4, illustrating the variation of error change norm with respect to g(t, x) for a single context
Table 8 (fifth instance)
Table 8 is presented on page 5, showing the relative error of the trained predictor under different settings
Figure 15 (third instance)
Figure 15 is presented on page 5, illustrating the variation of∥g(t, x)∥ with respect to αx←t∥vt − vx∥ for ICL
Figure 18 (fourth instance)
Figure 18 is presented on page 5, showing the variation of the error change norm with respect to g(t, x) for RAG
Figure 16 (fourth instance)
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Vector Direction Alignment Strategy: Implement a context selection strategy based on angular alignment to ensure that
the contextual correction vector should align as closely as possible with the negative direction of the baseline error vector
to reduce output error. -
Contextual Correction Norm Control: Design a context selection mechanism that constrains
the norm of the contextual correction vector
by ensuring it adheres to an upper bound determined bycontext–query relevance and complementarity,
thereby preventing the model from over-attending to irrelevant context. -
Multi-Context and Multi-Layer Scaling: For complex scenarios, leverage the findings that error change remains consistent under multi-context or multi-layer settings, allowing for a principled design choice based on
the overall increasing trend
observed in the contextual correction vector norm across layers. -
Dynamic Context Relevance Prediction: Develop a lightweight predictor to estimate the
context-error angle
and relevance/complementarity metrics, enabling real-time selection of the context with the largest relevance to guide inference.
Abstract
Providing external information beyond the query is a common way to improve large language model (LLM) inference, as seen in in-context learning (ICL), retrieval-augmented generation (RAG), and memory-based methods (Mem). Existing work increasingly suggests that useful external information should compensate for the gap between LLMs and user queries, implicitly assuming that stronger gap-correction can lead to better performance. In this paper, we challenge this assumption and show that more gap-corrective external information can instead harm performance. We analyze this phenomenon in Transformers through the lens of reasoning error, defined as the difference between the predicted answer vector and the ground-truth answer vector. Our analysis reveals that external information can be not only under-corrective but also over-corrective, where excessively strong correction increases reasoning error. We further show that the induced correction vector is jointly determined by attention and the complementarity between the external information and the query, and derive conditions under which information is properly corrective in both direction and magnitude. Experiments on four mainstream LLMs and seven reasoning benchmarks across ICL, RAG, and Mem validate our theory and support a theory-guided information selection method.
Sources
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- The Llama 3 Herd of Models
- A-MEM: Agentic Memory for LLM Agents
- Qwen3 Technical Report
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data