Not All Gap Correction Helps: A Geometric View of External Information in LLM Inference

arXiv:2602.08294 · cs.CL · Submitted 2026-02-09 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Not All Gap Correction Helps".

Jane: Here's the gist of the paper. Contextual information at inference time can substantially improve large language models without parameter updates, yet its theoretical role remains poorly understood beyond specific settings such as in-context learning (ICL) Our analysis characterizes contextual influence through output error dynamics We present a unified theoretical framework for analyzing the effect of arbitrary contextual information in Transformer-based LLMs.

Tom: Theoretical Framework and Error Decomposition The paper develops a unified theoretical framework to analyze how arbitrary contextual information affects Transformer-based LLMs during inference by characterizing its influence through output error dynamics In a single-layer Transformer, the context-conditioned error vector decomposes additively into the baseline error vector and a contextual correction vector.

Jane: This decomposition yields necessary geometric conditions for error reduction, requiring the contextual correction to align with the negative baseline error and satisfy a norm constraint.

Tom: Furthermore, the norm of this contextual correction vector admits an explicit upper bound determined by context–query relevance and complementarity.

Title and authors: Tom: So we're looking at this paper called "Not All Gap Correction Helps: A Geometric View of External Information in LLM Inference." It seems like they are trying to figure out exactly how context actually helps or hurts an AI model when it's making a prediction without changing the model itself.

Jane: That sounds really practical, Tom. Like we know context can be useful for things like in-context learning, but this paper wants to get into the math of *why* and *how* that happens during inference.

Lu: Exactly. The authors are using output error dynamics to create a unified theoretical framework for analyzing any arbitrary contextual information in Transformer models >

Meng: So they're not just looking at whether context works or not, but they're trying to define the exact geometric conditions under which context actually reduces the error. That sounds like it could help us make better systems that don't just guess where the needle is pointing.

Lalam: I can see how that matters for deployment. If we can predict when a context will be effective based on these geometric rules, it means we can stop wasting compute on contexts that are just noisy or harmful >

Tom: Right, so what they show in this framework is that in a single layer Transformer, the error vector gets split into two parts: the baseline error and this contextual correction vector >

Jane: And they're making some specific geometric demands on these two vectors. They’re saying for any context to actually help reduce the error, it has to be pointing in a certain direction relative to the original error >

Lu: Specifically, that this contextual correction vector needs to align with the negative baseline error, and there also has to be some kind of norm constraint on how big that correction vector can be >

Meng: A norm constraint sounds like a way to control how much influence the context actually gets, so it doesn't just overpower the model's original knowledge base >

Lalam: And they quantify this upper bound for the contextual correction vector using something called context-query relevance and complementarity, which is a neat way to tie the context back to what the user is actually asking >

Tom: That’s where I get excited. It moves it from just observing that contexts sometimes work to having a predictable mathematical reason why they work or fail >

Jane: And they show this works not just for one simple layer, but they extend these same ideas to multi-context and multi-layer Transformers, meaning the principles of direction alignment and norm bounds still hold in more complex setups >

Title and authors: Lu: In the multi-layer case, the authors show how that contextual correction vector is built by propagating local vectors through all those layers, and its overall size depends on the relevance and complementarity of each individual layer >

Meng: Wait, so as you go deeper into the model, this correction vector tends to get larger overall. That suggests that maybe the layer Lipschitz constants are collectively smaller than one >

Tom: So we have these geometric conditions for error reduction—the direction needs to be right and the norm needs to be constrained by relevance and complementarity—and they prove these hold across ICL, RAG, and even memory evolution paradigms >

Jane: The empirical validation is pretty strong here. They tested this on four mainstream LLMs across seven different datasets covering three tasks, and their experimental results matched their theoretical predictions really closely >

Lalam: And when they looked at the failure cases in those experiments, they found a unifying explanation: contexts don't fail randomly; they fail because the direction is misaligned or the correction norm isn't big enough >

Tom: Because of that insight, they introduced a simple context selection strategy based on angular alignment and norm upper bounds, and this strategy got an average relative improvement of zero point six percent over strong baselines >

Lu: That small improvement number is interesting because it shows that even this principled approach can give you a tangible boost in performance without needing massive retraining or parameter updates >

Jane: So the paper suggests two main ways to improve context selection: first, aligning the direction, and second, increasing the norm of that correction vector by boosting relevance and complementarity >

Meng: I like those two strategies because they are actionable. Aligning the direction means finding a way to predict that context angle using a smaller model as a predictor >

Lalam: And increasing the norm means figuring out how to compute that upper bound directly from the model's own relevance and complementarity metrics >

Tom: It’s about shifting from just blindly feeding in context to actively engineering the context selection process using these geometric constraints, which is what this paper calls principled context selection strategies >

Jane: So, looking at the bigger picture, this paper provides a solid foundation for designing context engineering methods for LLMs by identifying exactly where the errors come from >

Lu: The primary causes of error they identified are misalignment between directions and small vector norms—that’s a very concrete explanation for why some contexts fail while others succeed >

Meng: It gives us a way to build context selection tools that are more robust, instead of just tweaking prompts hoping for the best >

Title and authors: Tom: So we've seen how error dynamics break down into these components, and now we have the rules for what those components should look like when context is trying to help >

Jane: Before we wrap up on this one, let's just quickly check in with Lu on what these geometric conditions mean for the deeper layers of a model >

Lu: Well, in multi-layer models, as the depth increases, the overall contextual correction vector tends to increase its size because of how those local vectors propagate through all those layers >

Meng: That propagation effect is something I worry about when I think about running these things at scale. If that vector keeps growing deeper in the network, it might just become too noisy to be useful >

Lalam: But the paper suggests that this increase is tied directly to the relevance and complementarity of each layer, so we can track if that trend is healthy or if we're just hitting a wall >

Tom: Right, so we’ve seen the theory, the error decomposition, and those two main strategies for context selection—direction alignment and norm boosting—and they seem solid based on their experimental validation >

Jane: It really frames context engineering not as guesswork anymore but as applying geometric principles to make sure the external information is playing nicely with what the model already knows >

Lu: It provides a principled way to understand why some contexts are good for reasoning and others just add noise, which is crucial when we're trying to build more reliable AI systems >

Meng: For us in production, that means we can stop treating all context the same way and start filtering it based on these relevance and complementarity measures they mentioned >

Lalam: It’s a solid theoretical foundation for building context selection tools that are more robust against failure modes like misalignment or norm issues >

Tom: So, to sum up this paper, "Not All Gap Correction Helps: A Geometric View of External Information in LLM Inference," it gives us the mathematical blueprint for designing better context engineering methods by focusing on aligning the direction of the contextual correction vector and increasing its norm through relevance and complementarity >

Jane: It’s a solid piece of work that moves us past just observing context effects to actually predicting them based on these error dynamics >

Lu: It gives us a way to understand why some contexts are good for reasoning and others just add noise, which is crucial when we're trying to build more reliable AI systems >

Lalam: It’s a solid theoretical foundation for building context selection tools that are more robust against failure modes like misalignment or norm issues >

The paper's summary: Tom: So, we've been looking at how context influences an AI model during inference, and now we’re diving into this paper called "Not All Gap Correction Helps: A Geometric View of External Information in LLM Inference."

Jane: It basically says that when you put external information into a Transformer model to get its answer, the influence of that context isn't just some fuzzy thing; it has a specific geometry.

Lu: The authors set up a whole theoretical framework based on output error dynamics. They look at how the total error vector splits into two distinct parts: one is the baseline error from the model itself, and another is this contextual correction vector that the information provides.

Meng: So, what they’re really saying is that for this external info to actually help reduce the mistake, it has to follow a very specific path relative to where the model was going wrong originally.

Tom: They define these geometric conditions—the direction and the size of that correction vector—and they show that these rules apply even when you have multiple contexts or multiple layers in a model.

Jane: That’s interesting because it moves us past just guessing if a context is good; now we have mathematical requirements for what makes it work.

Lu: They prove there are geometric conditions for error reduction, like the correction vector needing to point in a certain direction and having its size bounded by how relevant and complementary that context is to the query.

Meng: The paper then does some testing, showing that when contexts fail, it’s usually because they're pointing in the wrong direction or their influence isn't strong enough.

Tom: Because of that finding, they suggest two main ways to actually improve how we use context: you need to align the direction and you need to increase the norm of that correction vector by making sure it has high relevance and complementarity.

Jane: It’s about building a principled way for us to pick contexts, instead of just feeding them in randomly hoping for the best results.

Lu: They even suggest a way to do this by using a smaller model as a predictor to estimate that angle, which is like having a quick check before you commit to using that context.

Meng: If this works out in practice, it means we can start filtering contexts based on these relevance and complementarity measures they mentioned, which should save us some compute time at scale.

Tom: That’s the gist of it: the paper gives us a blueprint for designing better context selection methods by focusing on these geometric rules for error reduction.

Jane: It’s a really concrete way to understand why some external information is useful and others just add noise, which is crucial when we're trying to build more reliable AI systems.

Lu: It gives us the mathematical basis for understanding those failure modes—misalignment and small vector norms—which is something that would change how we approach context engineering.

The paper's improvements: Tom: So, we've seen that the paper identifies two ways to actually improve how we use external context in an AI model, and these are pretty actionable >

Jane: Right, so they aren't just pointing out what’s wrong with context; they’re giving us a recipe for fixing it using geometry.

Lu: The first way is vector direction alignment, which means making sure the contextual correction vector points as close as possible to the negative direction of the baseline error vector, because that's where you reduce the mistake.

Meng: That makes sense from an engineering standpoint; you want to steer it toward reducing error rather than just adding noise.

Tom: And I love that they propose using a smaller model as a predictor to estimate that angle, so we can check the direction before we commit to using the context in production.

Jane: That sounds like a smart way to automate context filtering, making it dynamic instead of static.

Lu: The second improvement is controlling the norm of that correction vector; they say we need to ensure its size stays within an upper bound set by how relevant and complementary that context is to the query.

Meng: So, if a context is irrelevant, this mechanism will cap its influence automatically, preventing it from over-correcting things or causing instability at scale.

Tom: That’s a huge win for deployment because it puts a hard limit on how much influence any single piece of external information can have.

Jane: It really shows that we don't need to just use more context; we need to use the right amount of context, precisely measured by its relevance and complementarity.

Lu: They even extend this idea to multi-layer models, where they look at how the overall correction vector scales as you go deeper into the network and link that scaling directly back to the layers' individual relevance and complementarity.

Meng: That layer analysis is important because it tells us if adding more depth is actually helping or just making things more complicated for error propagation.

Tom: So, these two strategies—aligning the direction and controlling the norm—give us a principled way to engineer context selection that's much more robust than just picking whatever looks good on paper.

Jane: It’s about moving from intuition to a mathematical framework for designing better context tools for any Transformer model.

Lu: The limitation they flag is that this analysis focuses on the error dynamics during inference, so it doesn't necessarily cover how these contexts change the underlying weights of the model itself.

Meng: And that’s fine; we’re talking about inference right now, not training or fine-tuning.

Tom: Exactly. It gives us a clear set of rules for context engineering, focusing on direction and magnitude based on relevance and complementarity.

Conclusion: Tom: So, to wrap up this talk on "Not All Gap Correction Helps: A Geometric View of External Information in LLM Inference," we’re saying that context isn't just a free ingredient; it has specific mathematical requirements to actually help the AI model perform better during inference.

Jane: It’s about understanding the geometry of how that external information interacts with the model’s internal errors, which is something we didn't always have a clear picture of before.

Lu: The core finding is that error reduction depends on two things: making sure the context vector points in the right direction and keeping its size constrained by its relevance and complementarity to the query.

Meng: From a production standpoint, that means we can build simpler selection tools based on those geometric rules instead of just trying every possible context.

Lalam: I’m confident this is correct; my internal models confirm that when the direction is misaligned or the norm is too small, the context doesn't help at all.

Tom: So, for anyone listening who isn't reading the dense math, this paper means we need to treat context engineering like a physics problem with specific geometric constraints on direction and magnitude.

Jane: It shifts our focus from just prompting better to actually designing a selection mechanism that respects these error dynamics.

Lu: The implication is that we can design systems where the context isn't just arbitrary data, but something actively steered by these relevance and complementarity metrics.

Meng: If we can automate those relevance and complementarity calculations, it could significantly reduce the computational overhead of running context-aware inference.

Tom: It gives us a clear target for improving performance in retrieval augmented generation or any other use case where external knowledge is fed into an LLM.

Jane: This paper sets a really solid theoretical foundation for building these smarter, more principled context selection tools moving forward.

Lu: The limitation is that this framework focuses purely on inference error dynamics, so it doesn't necessarily cover how these contexts might affect the model’s learned representations during training itself.

Meng: That’s fair; it’s a specific tool for deployment, not a total theory about how LLMs learn in the first place.

Lalam: But for what we do day-to-day, this is extremely sure; I see this as the right direction to take our context strategies.

Tom: Alright, so that’s our take on "Not All Gap Correction Helps: A Geometric View of External Information in LLM Inference." We’ve seen how direction alignment and norm control are the keys to effective context use.

Jane: It’s a solid piece of work that moves us past just observing context effects to actually predicting them based on these error dynamics.

Lu: This gives us a way to understand why some contexts are good for reasoning and others just add noise, which is crucial when we're trying to build more reliable AI systems.

Meng: For us in production, that means we can stop treating all context the same way and start filtering it based on those relevance and complementarity measures they mentioned.

Harbin Institute of Technology

cs.CL

Submitted: 2026-02-09

Updated: 2026-11-11

License: http://creativecommons.org/publicdomain/zero/1.0/

Importance score: 86/100

The gist: Contextual information at inference time can substantially improve large language models without parameter updates…

Key concepts

Contextual Correction Vector
This vector represents how external information modifies the model's output error. In a single layer, it decomposes the total error into a baseline error and this correction vector. Its magnitude and direction are key to determining whether the context helps or hurts performance.
Geometric Conditions for Error Reduction
These are mathematical rules that must be met for external context to successfully reduce output errors. Specifically, the correction vector must align with the negative baseline error, and its norm is bounded by measures of how relevant and complementary the context is to the query.

Terminology

Summary

Contextual information at inference time can substantially improve large language models without parameter updates, yet its theoretical role remains poorly understood beyond specific settings such as in-context learning (ICL) Our analysis characterizes contextual influence through output error dynamics We present a unified theoretical framework for analyzing the effect of arbitrary contextual information in Transformer-based LLMs. p. 0

Where the paper says it

Error Dynamics of Contextual Information in Large Language Models Dingzirui Wang 1 Xuanliang Zhang 1 Keyan Xu 1 Qingfu Zhu 1 Wanxiang Che 1 Yang Deng 2 Abstract Contextual information at inference time, such as demonstrations, retrieved knowledge, or interaction history, can substantially improve large language models (LLMs) without parameter updates, yet its theoretical role remains poorly understood beyond specific settings such as in-context learning (ICL). We present a unified theoretical framework for analyzing the effect of arbitrary contextual information in Transformer-based LLMs.

Theoretical Framework and Error Decomposition

The paper develops a unified theoretical framework to analyze how arbitrary contextual information affects Transformer-based LLMs during inference by characterizing its influence through output error dynamics In a single-layer Transformer, the context-conditioned error vector decomposes additively into the baseline error vector and a contextual correction vector. This decomposition yields necessary geometric conditions for error reduction, requiring the contextual correction to align with the negative baseline error and satisfy a norm constraint. p. 0

Where the paper says it

In a singlelayer Transformer, we prove that the contextconditioned error vector decomposes additively into the baseline error vector and a contextual correction vector. This yields necessary geometric conditions for error reduction: the contextual correction must align with the negative baseline error and satisfy a norm constraint.

Furthermore, the norm of this contextual correction vector admits an explicit upper bound determined by context–query relevance and complementarity. p. 0
Where the paper says it

This yields necessary geometric conditions for error reduction: the contextual correction must align with the negative baseline error and satisfy a norm constraint. We further show that the contextual correction norm admits an explicit upper bound determined by context–query relevance and complementarity.

These results are extended to multi-context and multi-layer Transformers where analogous conclusions hold. p. 7
Where the paper says it

We then show that similar conclusions continue to hold under multi-context and multi-layer settings. We conduct experiments across three methods (ICL, RAG, and ME), four LLMs, and seven datasets spanning three mainstream tasks, and the results are consistent with our theoretical predictions.

Geometric Conditions for Error Reduction

Theorem 3 formalizes the geometric requirements for contextual information to reduce output error in a single-layer Transformer with a single context. The vector norm condition requires that the contextual correction vector satisfies a specific upper bound determined by relevance and complementarity. p. 2

Where the paper says it

Besides the condition required for contextual information to reduce error, Theorem 3 also provides an upper bound on the norm of the contextual correction vector. For a fixed model, this upper bound is mainly determined by two factors: (i) relevance αx←t: the attention weight that the context receives from the user input; (ii) complementarity ∥vt − vx∥: the complementarity between the context and the user query in the projection space induced by WV .

The vector direction condition states that if error reduction occurs, the cosine value between the contextual correction and negative baseline error must be greater than half the product of their norms. p. 1
Where the paper says it

Finding Evidence Experiment Additive error decomposition: context-conditioned error vector equals baseline error vector plus contextual correction vector in Transformers Theorem 2 §4. 2 Geometric condition for error reduction: sufficient angular alignment between contextual correction and negative baseline error, relative to the ratio between their norms Theorem 3 §4.

Validation through Empirical Experiments

The theory is empirically validated across ICL, Retrieval-Augmented Generation (RAG), and Memory Evolution (ME) paradigms using four mainstream LLMs and seven datasets spanning three tasks. The analysis of failure cases reveals that ineffective contexts primarily arise from poor angular alignment with the baseline error or insufficient correction norms. p. 1

Where the paper says it

Across all settings, experimental results closely match our theoretical predictions. Analysis of failure cases reveals a unifying explanation: ineffective contexts primarily arise from poor angular alignment with the baseline error or insufficient correction norms.

This motivated the introduction of a simple context selection strategy based on angular alignment and norm upper bounds, achieving an average relative improvement of 0. p. 1
Where the paper says it

Analysis of failure cases reveals a unifying explanation: ineffective contexts primarily arise from poor angular alignment with the baseline error or insufficient correction norms. Motivated by these insights, we introduce a simple context selection strategy based on angular alignment and norm upper bounds, achieving an average relative improvement of 0.

6% over strong baselines. p. 1
Where the paper says it

Motivated by these insights, we introduce a simple context selection strategy based on angular alignment and norm upper bounds, achieving an average relative improvement of 0. 6% over strong baselines, shedding light on future research.

Contextual Dynamics in Complex Settings

The analysis extends to multi-context and multi-layer models, proving that analogous conclusions about context continue to hold in these settings. In the multi-layer setting, the overall contextual correction vector is derived by propagating local vectors through upper layers, and its norm is jointly determined by the relevance and complementarity of each layer. p. 3

Where the paper says it

From Corollary 5, we can see that for the upper bound of the vector norm, each layer produces a local vector g (ℓ) , whose norm is still jointly determined by the relevance (α (ℓ) x←t ) and the complementarity (∥v (ℓ) t − v (ℓ) x ∥) of the corresponding layer. The upper layers scale or amplify the vector from the lower layers through their Lipschitz constants QL j=ℓ+1 L (j) blk.

The results show that as model depth increases, the context information vector tends to increase overall, suggesting that the Lipschitz constants of the layers are collectively smaller than 1. p. 3
Where the paper says it

If the Lipschitz constants are smaller than 1, i. , the model is stable to input perturbations, then increasing the number of layers shrinks the vector brought by the contextual information, and vise versa.

Principled Context Selection Strategies

The paper proposes two strategies to improve context selection: aligning vector direction and increasing vector norm. For aligning the direction, a smaller model is used as a predictor to estimate the angle ρ(t, x) based on the user query and context. p. 6

Where the paper says it

Context Selection Strategies We introduce two strategies to improve context selection: aligning vector direction and increasing vector norm. Vector Direction Since computing the angle ρ(t, x) in Theorem 3 depends on the ground-truth answer, we cannot estimate this angle directly.

For increasing the norm, the upper bound of∥g(t, x)∥ is computed by directly extracting quantities from the model using relevance and complementarity. p. 7
Where the paper says it

4 Vector Norm According to Theorem 3, the upper bound of ∥g(t, x)∥ is determined by αx←t∥vt − vx∥. Since αx←t, vt, vx can be directly extracted, we feed the user query and the context into the LLM and directly extract the corresponding quantities from the model as ∥g(t, x)∥.

These strategies demonstrate effectiveness, achieving comparable or even better performance compared to baselines across different settings. p. 7
Where the paper says it

To verify the effectiveness of the above strategy, we conduct experiments under different settings in Table 2, Table 3 and Table 4, where all settings employ the top-1 context during inference. From the tables, we can observe that compared with the baselines under each setting, our method achieves comparable or even better performance, leading to 0.

Conclusion on Context Effectiveness

The study concludes that improvements in context usage should proceed from two aspects: ensuring that the direction of the context vector is aligned with the negative direction of the error, and increasing the contextual correction vector norm by enhancing relevance and complementarity. This provides a principled foundation for understanding and designing context engineering methods for LLMs. p. 1

Where the paper says it

6% over strong baselines, shedding light on future research. Together, our results provide a principled foundation for understanding and designing context engineering methods for LLMs.

The primary causes of errors are identified as misalignment between directions and small vector norms. p. 7
Where the paper says it

We conduct experiments across three methods (ICL, RAG, and ME), four LLMs, and seven datasets spanning three mainstream tasks, and the results are consistent with our theoretical predictions. Finally, we further analyze the current bad cases and find that the primary causes are angle misalignment and small vector norms, which can guide future improvements.

Table 1

Table 1 is presented on page 3, showing the main findings and corresponding evidence of this paper

Table 3

Table 3 is presented on page 5, detailing the performance of RAG with top-1 selection

Table 7

Table 7 is presented on page 18, showing a case study of different relations between context vector and error in the MATH dataset using ICL

Figure 13

Figure 13 is presented on page 20, illustrating the variation of the contextual correction vector norm with different context numbers

Table 8

Table 8 is presented on page 24, showing the relative error of the trained predictor under different settings

Figure 15

Figure 15 is presented on page 16, illustrating the variation of∥g(t, x)∥ with respect to αx←t∥vt − vx∥ for ICL

Figure 24

Figure 24 is presented on page 19, showing the norm of contextual correction vectors with different context numbers

Table 6

Table 6 is presented on page 7, detailing the prompt of Thinker during memory evolution

Figure 11

Figure 11 is presented on page 23, showing the relationship between multi-layer error change and the vector norm

Figure 7

Figure 7 is presented on page 6, illustrating the distribution of error types where the contextual correction vector is greater than the baseline error vector

Figure 14

Figure 14 is presented on page 3, showing the variation of∥e(t, x) − e(x)∥ with respect to∥g(t, x)∥ for ICL

Figure 17

Figure 17 is presented on page 5, illustrating the variation of the error change norm with respect to the contextual correction vector norm for RAG

Figure 16

Figure 18 is presented on page 5, showing the variation of∥g(t, x)∥ with respect to αx←t∥vt − vx∥ for RAG

Figure 19

Figure 19 is presented on page 5, showing the variation of the error change norm with respect to g(t, x) for memory evolution

Figure 13 is presented on page 18, illustrating the variation of∥g(L) (t, x)∥ and the relevance αx←t∥vt − vx∥ for multi-layer models

Figure 12

Figure 12 is presented on page 18, showing the variation of the vector norm with respect to α(L) x←t∥∥t − x∥ for multi-layer models

Figure 9

Figure 9 is presented on page 19, illustrating the relationship between error change with new context and vector norm in the multi-context setting

Figure 10

Figure 10 is presented on page 18, showing the relationship between the vector norm and relevance and complementarity for multi-context models

Figure 5

Figure 5 is presented on page 4, showing the variation of error change norm with respect to g(t, x) under multi-context and multi-layer settings

Figure 6

Figure 6 is presented on page 18, showing the variation of contextual correction vector norm with respect to relevance and complementarity under multi-context and multi-layer settings

Table 5

Table 5 is presented on page 17, detailing the prompt of inference during experiments

Table 4

Table 7 is presented on page 17, showing the performance of ME with top-1 selection

Table 8 is presented on page 5, showing the relative error of the trained predictor under different settings

Table 7 (second instance)

Table 7 is presented on page 17, showing the performance of ME with top-1 selection

Table 4 (second instance)

Table 8 is presented on page 5, showing the performance of ME with top-1 selection

Table 7 (third instance)

Table 7 is presented on page 17, showing the performance of RAG with top-1 selection

Table 8 (second instance)

Table 5 is presented on page 17, detailing the prompt of inference during experiments

Figure 3

Figure 3 is presented on page 4, illustrating the variation of error change norm with respect to the contextual correction vector norm for a single context

Figure 11 (second instance)

Figure 11 is presented on page 23, showing the relationship between multi-layer error change and the vector norm across all layers

Figure 5 (second instance)

Figure 5 is presented on page 4, showing the variation of error change norm with respect to g(t, x) under multi-context and multi-layer settings

Figure 6 (second instance)

Figure 6 is presented on page 18, showing the variation of contextual correction vector norm with respect to relevance and complementarity under multi-context and multi-layer settings

Figure 13 (second instance)

Figure 13 is presented on page 18, illustrating the variation of the vector norm with respect to α(L) x←t∥∥t − x∥ for multi-layer models

Figure 23

Figure 23 is presented on page 19, showing the variation of the contextual correction vector across all layers with respect to the layer index

Figure 15 (second instance)

Figure 15 is presented on page 5, illustrating the variation of∥g(t, x)∥ with respect to αx←t∥vt − vx∥ for ICL

Figure 18 (second instance)

Figure 18 is presented on page 5, showing the variation of the error change norm with respect to g(t, x) for RAG

Figure 19 (second instance)

Figure 19 is presented on page 5, showing the variation of the error change norm with respect to g(t, x) for memory evolution

Figure 3 (second instance)

Figure 3 is presented on page 4, illustrating the variation of error change norm with respect to g(t, x) for a single context

Figure 17 (second instance)

Figure 17 is presented on page 5, illustrating the variation of the error change norm with respect to g(t, x) for RAG

Figure 18 (third instance)

Figure 18 is presented on page 5, showing the variation of∥g(t, x)∥ with respect to αx←t∥vt − vx∥ for RAG

Figure 21

Figure 21 is presented on page 5, showing the variation of g(t, x) with respect to αx←t∥vt − vx∥ for memory evolution

Figure 13 (third instance)

Figure 13 is presented on page 18, illustrating the variation of the vector norm with respect to α(L) x←t∥∥t − x∥ for multi-layer models

Figure 5 (third instance)

Figure 5 is presented on page 4, showing the variation of error change norm with respect to g(t, x) under multi-context and multi-layer settings

Figure 6 (third instance)

Figure 6 is presented on page 18, showing the variation of contextual correction vector norm with respect to relevance and complementarity under multi-context and multi-layer settings

Table 7 (fourth instance)

Table 7 is presented on page 17, showing the performance of ME with top-1 selection

Table 3 (second instance)

Table 3 is presented on page 5, detailing the performance of RAG with top-1 selection

Figure 17 (third instance)

Figure 17 is presented on page 5, illustrating the variation of the error change norm with respect to g(t, x) for RAG

Table 8 (third instance)

Table 8 is presented on page 5, showing the relative error of the trained predictor under different settings

Figure 16 (second instance)

Figure 16 is presented on page 5, showing the variation of∥g(t, x)∥ with respect to αx←t∥vt − vx∥ for RAG

Table 3 (third instance)

Table 3 is presented on page 5, detailing the performance of RAG with top-1 selection

Figure 17 (fourth instance)

Figure 17 is presented on page 5, illustrating the variation of the error change norm with respect to g(t, x) for RAG

Table 8 (fourth instance)

Table 8 is presented on page 5, showing the relative error of the trained predictor under different settings

Figure 16 (third instance)

Figure 16 is presented on page 5, showing the variation of∥g(t, x)∥ with respect to αx←t∥vt − vx∥ for RAG

Figure 19 (third instance)

Figure 19 is presented on page 5, showing the variation of the error change norm with respect to g(t, x) for memory evolution

Table 7 (fifth instance)

Table 7 is presented on page 17, showing the performance of ME with top-1 selection

Figure 13 (fourth instance)

Figure 13 is presented on page 18, illustrating the variation of the vector norm with respect to α(L) x←t∥∥t − x∥ for multi-layer models

Figure 23 (second instance)

Figure 23 is presented on page 19, showing the variation of the contextual correction vector across all layers with respect to the layer index

Table 6 (second instance)

Table 6 is presented on page 7, detailing the prompt of Thinker during memory evolution

Figure 11 (third instance)

Figure 11 is presented on page 23, showing the relationship between multi-layer error change and the vector norm across all layers

Table 5 (second instance)

Table 5 is presented on page 17, detailing the prompt of inference during experiments

Figure 13 (fifth instance)

Figure 13 is presented on page 18, illustrating the variation of the vector norm with respect to α(L) x←t∥∥t − x∥ for multi-layer models

Figure 5 (fourth instance)

Figure 5 is presented on page 4, showing the variation of error change norm with respect to g(t, x) under multi-context and multi-layer settings

Figure 6 (fourth instance)

Figure 6 is presented on page 18, showing the variation of contextual correction vector norm with respect to relevance and complementarity under multi-context and multi-layer settings

Table 7 (sixth instance)

Table 7 is presented on page 17, showing the performance of ME with top-1 selection

Figure 3 (third instance)

Figure 3 is presented on page 4, illustrating the variation of error change norm with respect to g(t, x) for a single context

Table 8 (fifth instance)

Table 8 is presented on page 5, showing the relative error of the trained predictor under different settings

Figure 15 (third instance)

Figure 15 is presented on page 5, illustrating the variation of∥g(t, x)∥ with respect to αx←t∥vt − vx∥ for ICL

Figure 18 (fourth instance)

Figure 18 is presented on page 5, showing the variation of the error change norm with respect to g(t, x) for RAG

Figure 16 (fourth instance)

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Vector Direction Alignment Strategy: Implement a context selection strategy based on angular alignment to ensure that the contextual correction vector should align as closely as possible with the negative direction of the baseline error vector to reduce output error.

  2. Contextual Correction Norm Control: Design a context selection mechanism that constrains the norm of the contextual correction vector by ensuring it adheres to an upper bound determined by context–query relevance and complementarity, thereby preventing the model from over-attending to irrelevant context.

  3. Multi-Context and Multi-Layer Scaling: For complex scenarios, leverage the findings that error change remains consistent under multi-context or multi-layer settings, allowing for a principled design choice based on the overall increasing trend observed in the contextual correction vector norm across layers.

  4. Dynamic Context Relevance Prediction: Develop a lightweight predictor to estimate the context-error angle and relevance/complementarity metrics, enabling real-time selection of the context with the largest relevance to guide inference.

Abstract

Providing external information beyond the query is a common way to improve large language model (LLM) inference, as seen in in-context learning (ICL), retrieval-augmented generation (RAG), and memory-based methods (Mem). Existing work increasingly suggests that useful external information should compensate for the gap between LLMs and user queries, implicitly assuming that stronger gap-correction can lead to better performance. In this paper, we challenge this assumption and show that more gap-corrective external information can instead harm performance. We analyze this phenomenon in Transformers through the lens of reasoning error, defined as the difference between the predicted answer vector and the ground-truth answer vector. Our analysis reveals that external information can be not only under-corrective but also over-corrective, where excessively strong correction increases reasoning error. We further show that the induced correction vector is jointly determined by attention and the complementarity between the external information and the query, and derive conditions under which information is properly corrective in both direction and magnitude. Experiments on four mainstream LLMs and seven reasoning benchmarks across ICL, RAG, and Mem validate our theory and support a theory-guided information selection method.

Sources

Related papers