Context Injection and Long-Horizon Reliability: Separating Duplicate Recognition, Local Processing, and Counter Tracking

arXiv:2610.20633 · cs.CL · Submitted 2026-09-30 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: Today's paper: "Context Injection and Long-Horizon Reliability".

Jane: The gist The counter follows the value present in the model’s own history, and an explicit external statement of the state restores it,

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So we're talking about this paper called "Context Injection and Long-Horizon Reliability: Separating Duplicate Recognition, Local Processing, and Counter Tracking." It looks like they're trying to get a better look at why language models fail over really long tasks.

Jane: Right. The core idea seems to be breaking down a long task into smaller pieces so we can see exactly what’s going wrong step by step. It’s not just looking at the final result, but tracking three specific things that happen every single time the model answers something new in this ledger task.

Lu: Exactly. They build this bench to score recognizing a record already seen, processing the current record locally, and writing a counter that increments at every step >

Meng: So they're treating these components separately instead of just giving one overall score for success or failure across the whole four hundred steps <ref:2610.20633#pg1>. That feels like it would help us understand *why* something breaks.

Lalam: I see what they're doing with the scoring, breaking it down into R1 for local processing, R2 for duplicate recognition, and a dedicated counter tracking mechanism >

Tom: And on Qwen3-8B, they found two things hold up <ref:2610.20633#pg1>. First, the duplicate recognition errors actually start to rise later in the episode as you get further out to step two hundred <ref:2610.20633#pg1>.

Jane: That's interesting because we usually expect errors to be constant if the model is struggling with something consistently. But here it seems like it’s driven by a combination of long-distance recall issues and false alarms on new records >

Lu: That rise isn't just about distance from the start; it’s combining those two specific kinds of errors, which is a detail they picked up >

Meng: And then there's this second finding where they injected irrelevant content before each step. They saw that that block completely ruins the counter tracking in every single seed they tested >

Lalam: That suggests that context injection isn't always just about adding information; sometimes it can actively disrupt critical state tracking, which is a big operational concern for production systems >

Tom: It means we have to be careful about what we inject into the context window, because if it’s not relevant, it can actively break something essential >

Title and authors: Jane: It points towards a need for more specific ways to handle state information than just stuffing more text in there. What did they suggest as an improvement for this setup?

Lu: They argue that carried state, which is what matters in these long tasks, should be stated explicitly and often instead of left entirely to the model's own history >

Tom: So the suggestion is to move away from relying solely on the model's internal memory for things like a counter, and instead force it to state those critical values out loud more frequently >

Jane: Because they showed that a corrupted value just keeps persisting without being corrected in every seed, which is a real problem for reliability >

Meng: So the paper is saying we need an external mechanism or an oracle to supply that critical state information so the model doesn't get lost in its own history >

Lalam: They show that an exact external reminder, repeated before each message, manages to re-align the counter within one or two steps, even when things are disrupted >

Tom: That's a concrete result. It means if we want something tracked perfectly over many steps, we probably can't just rely on implicit history tracking alone >

Jane: And the improvement test showed that under a disruption that usually breaks the counter, they observed zero counter errors when an exact statement of the state preceded each message >

Lu: That implies that if you have a trusted external source supplying the state, you can mitigate those long-horizon failures effectively, even against noise >

Meng: But there's still a limitation they point out. They use one synthetic task family in French with a trivially incremented counter, so they say this result can't be generalized to arithmetic states >

Lalam: That’s a fair caveat. The counter needs to be something more complex than just simple addition for this kind of proof to hold up >

Tom: And they also note that the frequency needed for that state information repair wasn't established yet, so we don't know exactly how often we need to inject it >

Jane: So, if you take away the explicit state injection, what does that mean for how we design these long-horizon systems in general?

Title and authors: Lu: It means that simply increasing context length or relying on internal history isn't enough; we have to engineer a better way to explicitly manage critical variables >

Meng: From a deployment standpoint, if we can rely on an external source for the counter, it simplifies our infrastructure because the model doesn't have to perfectly maintain that arithmetic state internally >

Lalam: It suggests that for truly reliable long-horizon work, we might need a hybrid approach where the model handles local processing and recall, but a separate system manages the critical state updates >

Tom: So we’re moving from hoping the internal memory holds up to actually enforcing the tracking externally with specific reminders or data feeds >

Jane: It sounds like this paper gives us a blueprint for how to stop these long-horizon failures by being more explicit about what we need the model to remember at every stage.

Lu: Precisely, it's about separating those three components so we can target the fix precisely, rather than just tweaking one broad setting >

Meng: It’s a good way to think about scaling reliability; instead of trying to make the whole system more robust, you focus on making sure these specific state variables don't get corrupted >

Lalam: And it shows that even with context injection causing disruption, if we use an exact external reminder as a repair mechanism, we can recover the counter tracking very well >

Tom: So to wrap up this segment on "Context Injection and Long-Horizon Reliability: Separating Duplicate Recognition, Local Processing, and Counter Tracking," the main point is that explicit state management through external reminders is a way to keep counters accurate over long sequences >

Jane: It’s about recognizing that what matters in these long tasks isn't just the raw text input but the explicit handling of critical values like a counter >

Lu: And it shows us exactly where the current methods fall short, pointing directly at carried state as the biggest weakness we need to address >

Meng: We’ll have to work on making sure that whenever we deploy these agents, there’s a clear path for injecting this kind of externally supplied state information if things go sideways >

Lalam: And it's a solid direction for future work because it gives us a specific mechanism—the exact external reminder—to test against when we build our next agent framework >

The paper's summary: Tom: So, to recap, this paper is basically setting up a scoreboard for how good an AI is at handling really long tasks by looking at three distinct skills: spotting things already seen again, processing new stuff right here in the moment, and keeping a running tally or a counter >

Jane: Right. And it breaks down the failure modes. They find that when you look at these components separately, you see that the counter tracking is actually quite stable for a long time—like eight thousand steps—but the error rate for spotting duplicates gets worse as you go further out >

Tom: That’s what I’m excited about because we usually get one big score, and that score can hide problems. This study shows that the rise in duplicate recognition errors isn't just because of the distance; it’s actually a mix of those long-distance recall issues and false alarms on new records >

Jane: And then they look at what actually breaks things when you add context. They test injecting irrelevant content before every step, and that shows that even if the text looks normal, that block completely ruins the counter tracking in every single seed they tested >

Tom: So it tells us that context injection isn't always helpful; sometimes it actively causes corruption on things like counters >

Jane: And what this means for us is that we can’t just rely on the model remembering everything internally over hundreds of steps. The paper argues that carried state, the stuff that really matters, needs to be stated out loud more often >

Tom: I agree. They show a specific fix—using an exact external reminder repeated before each message—that re-aligns the counter within one or two steps even when things are noisy >

Jane: That’s a practical way to think about it. It suggests that for long tasks, if you want something tracked perfectly, you need an explicit mechanism from outside the model's internal memory >

Tom: Exactly. It points toward needing an oracle or a trusted external source to supply those critical state values instead of just hoping the model maintains them correctly >

Jane: But there are some caveats they mention. They say this specific counter experiment was on a very simple task, like just incrementing a number in French, so you can't take this result and apply it to complex arithmetic states yet >

Tom: True. It’s not a universal fix for all kinds of math; it’s more about the *method* of state management rather than the specific task itself >

Jane: So the paper gives us a blueprint for how we need to design these systems—focusing on making critical variables explicit through external input, rather than just relying on internal history >

Tom: And that’s where I see a lot of room for work. We need better ways to engineer that state injection so it happens at the right frequency without clogging up the context window with useless information >

Jane: Exactly. So, moving forward, we should be looking at how to build that external verification layer so it becomes a standard part of our agent frameworks >

The paper's improvements: Tom: So we’re talking about how to actually fix this, what the paper suggests we do next regarding those counter failures >

Jane: Right. The big suggestion is to stop relying on implicit memory and start using explicit state mechanisms, like that external reminder I mentioned before >

Tom: Exactly. They show that an exact external statement of the state works really well—it re-aligns the counter in just one or two steps when things go wrong >

Jane: That implies we should design systems where they have a trusted source supplying those critical values, maybe an oracle, instead of trying to let the model track every single number internally >

Tom: I like that. It shifts the burden from perfect internal memory to reliable external data feeds which is something I can actually work with on production systems >

Jane: It moves us away from just hoping the model keeps its place and toward a more engineered way of managing arithmetic or sequential states >

Tom: And they also point out that we need to figure out the right frequency for this state information to repair things; how often do we need to inject that reminder >

Jane: That’s a good question. Because if you inject it too often, you might just clutter the context window with useless instructions >

Tom: Yeah, that’s the trade-off. We need an optimal injection strategy—not too sparse, not too dense—to get that repair mechanism working without killing performance >

Jane: And remember they also flagged a limitation: this whole counter thing is currently only proven on a very simple French task with just basic incrementing, so we can't assume this works for complex math yet >

Tom: True. It’s an ablation study on a tiny problem, not a proof for arbitrary arithmetic; that’s the caveat >

Jane: So what this means for the world is that when we build long-horizon AI agents, we have to start thinking about their state management as an engineering problem where you explicitly design how critical variables get updated and verified >

Tom: Exactly. It's not just about training a bigger model; it's about designing a better operational loop for state maintenance >

Jane: And that’s where the next thing we should look at is how to integrate this kind of explicit external state tracking with the context Weaver framework, which organizes memory into these reasoning graphs >

Conclusion: Tom: So, to wrap up this discussion on "Context Injection and Long-Horizon Reliability: Separating Duplicate Recognition, Local Processing, and Counter Tracking," the main point is that we can't just rely on the model's internal history for critical tracking >

Jane: Right. The paper shows that carried state needs to be stated out loud explicitly, often, rather than left to the model’s own memory >

Tom: It gives us a concrete test: if you inject an exact external reminder repeated at every step, you can repair that state within one or two steps when things get disrupted >

Jane: That’s the practical implication—we need a way to enforce state information from an oracle, not just let the AI guess it internally >

Lu: I think this pushes us toward building hybrid systems where the model handles its local processing but a separate system manages those critical values like counters >

Meng: From my side, it means when we deploy models for long tasks, we have to budget for that external state check—it adds a layer of complexity, but it reduces catastrophic failure risk on production runs >

Lalam: I’m pretty sure this paper proves that a properly supplied external state can overcome those long-horizon corruption issues I’ve seen in my testing >

Tom: So we're moving from hoping the internal memory holds up to actually enforcing the tracking externally with specific data feeds >

Jane: It shows that for truly reliable long-horizon work, we need to engineer a better way to manage those critical variables explicitly rather than just relying on implicit history >

Lu: And that’s a good direction because it gives us a mechanism—that exact external reminder—to test against when we build our next agent framework >

Meng: We’ll have to figure out the necessary frequency for that state information repair; how often do we need to check in on the counter >

Lalam: I'm sure this is something we can quantify with more experiments, but it shows that explicit state management is definitely the way forward >

Baracoda AI Labs · NEOMA Business School

cs.CL

Submitted: 2026-09-30

Updated: 2026-09-30

Comments: 17 pages, 5 figures. Code, data and reproduction script: https://github.com/Baracoda-ai-labs/context-injection-long-horizon (tag v1.0-arxiv)

Code: https://github.com/Baracoda-ai-labs/context-injection-long-horizon

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 89/100

The gist: The gist The counter follows the value present in the model’s own history, and an explicit external statement of the state restores it, while context additions strongly disrupt counter tracking

Key concepts

Duplicate Recognition (R2): This measures the model's ability to correctly identify a record it has already seen earlier in the long sequence. Errors in this component tend to increase as the task gets longer, suggesting long-distance recall issues are a factor.
Carried State Tracking (Counter): The counter represents a value that must be rewritten at every step based on previous output. This tracking is highly sensitive to context; irrelevant content inserted before each step causes the counter to fail significantly, even though it doesn't affect how the model processes the current record locally.
External Reminder Repair: Providing an exact, compact statement of the required state externally before each message can fix a corrupted counter value. This external signal proved more effective at re-aligning the counter than simply repeating neutral text or relying on internal history alone.

Summary

The gist The counter follows the value present in the model’s own history, and an explicit external statement of the state restores it, while context additions strongly disrupt counter tracking while their effect on local processing is much smaller, and an aggregate or exclusive error classification can hide this difference.

How it works

The paper builds a bench to score three step-level components of a 400-step ledger task: recognizing a record already seen, processing the current record locally, and writing a counter incremented at every step <ref:2610.20633#pg6> The protocol scores every step component by component from the logged expected and observed lines <ref:2610.20633#pg6> The counter is observable on every readable line, including a DOUBLON answer, which still begins with the counter <ref:2610.20633#pg6>

The three components are scored separately to read them as a hazard, defined as the probability of an error at step k given no earlier error of the same component <ref:2610.20633#pg6> For duplicate recognition (R2), this means answering DOUBLON exactly when the current code was seen earlier in the episode <ref:2610.20633#pg6> For carried state, it means operationally, a value the model must rewrite at every step from its own previous output — here, a record counter <ref:2610.20633#pg6>

Main Results (Qwen3-8B)

The main results show that duplicate-recognition errors rise with the horizon — from 0.0019 to 0.011 per step at risk across step 200 — while the counter makes no error in 8,000 steps and local processing is descriptively stable This rise is carried by long-distance recall errors and by false alarms on new records, not by a loss at constant distance Injected context disrupts the counter far more than local processing A block of irrelevant content before each step makes the counter fail in every seed; its effect on local processing is much smaller and not distinguishable from none

Secondary Results and Follow-ups

Three pre-registered follow-ups probe the counter's value, showing that a corrupted value persists without correction in every seed An exact external reminder, repeated before each message, re-aligns it within one to two steps Under the irrelevant block, no counter error was observed when an exact compact reminder of the state, supplied externally by an oracle, preceded each message (0 of 20 seeds with an error)

Discussion and Conclusions

The paper concludes that carried state which matters should be stated explicitly and often rather than left to the model’s own history Experiment R establishes the behaviour: the counter depends on the value present in the response history — an injected wrong value is propagated by every seed, and a reminder repeated at every step re-aligns it within one to two steps The repair test shows that under a disruption that otherwise breaks the counter, no counter error was observed when an exact statement of the state preceded each message, while the absence of degradation of the other components was not established by the sealed criterion

Limitations

The limitations include using one synthetic task family in French with a trivially incremented counter, which means the counter result must not be generalised to arithmetic state The model used is Qwen3 via ollama, greedy, non-thinking; thinking mode may change the picture Furthermore, Experiment R establishes a behaviour, not a mechanism The necessary frequency for the state information to repair the counter was not established

Reproducibility

Code, all logged episodes, pre-registrations and seals are available at https://github.com/Baracoda-ai-labs/context-injection (MIT for code, CC BY 4.0 for data) This paper, its PDF and its paper/numbers.json correspond to the tag v1.0-arxiv of that repository To reproduce it: git clone the repository, git checkout v1.0-arxiv, then bash reproduce.sh, which regenerates every number, table and figure from the logs without calling a model It also runs verify seals.py, which checks the seals, and harnais/verifier papier.py, which re-derives the key numbers with independent code All design decisions and every sealed pre-registration were reviewed by the author

Seals

Every confirmatory campaign has a pre-registration whose SHA-256 hash is recorded (eleven seals, all verified by the package; the ninth covers the follow-up experiments of §5.3–5.4 and the last two experiment S (§5.5), each committed to the repository before any of their episodes was run) A hash proves that a document has not changed since it was hashed; it does not prove when The Phase 0 hazard test is therefore described as developed on preliminary data, evaluated on the remaining seeds and replicated on M2 (§5.1), and not as pre-registered The sealed prediction was refuted: the size map predicted no counter error at 14B and 32B; the 32B made 2

Audit

The bench’s original evaluator gave each line one exclusive label (R3, format, R2, counter, R1, in that order) After an external review showed that this could hide lower-ranked errors, every step was re-scored component by component

Decision Rule

The decision rule committed before the re-scoring run was: the counter result holds if, for the irrelevant block D on both machines, every test claim p survives reference correction survives per-claim correction The upper bound of the 95% seed-bootstrap interval of the R1 error-rate ratio D/bare is below 2; the label "lower bound > 1 if the lower bound exceeds 1; not conclusive" otherwise The sealed verdict is repairs

Replaced Values

Under the exclusive labels, the R1 hazard of the injected arms was biased downward: on M2, 0.0043 (bare), 0.0022 (D) and 0.0076 (B), against 0.015, 53 / 7,569 — — —————— The R2 is identical under both scorings; counter hazards differ by at most a few steps at risk The first counter error of a seed moves between the two scorings in 1 D seed and 1 A seed on M1, 0 D and 2 A seeds on M2, and in no bare or B seed The sealed R2 test is unchanged by the re-scoring (p = 1.5 × 10−4 on M1, 3.5 × 10−5 on M2), since only R3 and format outranked R2

References

Rauno Arike, Elizabeth Donoway, Henning Bartsch, and Marius Hobbhahn. Technical report: Evaluating goal drift in language model agents. In AAAI/ACM Conference on AI, Ethics, and Society (AIES), 2025 Mohsen Arjmandi. The LLM proposes, the executive disposes: A self-verifying agent instrument that dissociates commitment drift from binding drift in long-horizon agents, 2026 Yufeng Du, Minyang Tian, Srikanth Ronanki, Subendhu Rongali, Sravan Bodapati, Aram Galstyan, Azton Wells, Roy Schwartz, Eliu A. Huerta, and Hao Peng. Context length alone hurts LLM performance despite perfect retrieval Kelly Hong, Anton Troynikov, and Jeff Huber. Context rot: How increasing input tokens impacts LLM performance Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg.

Improvements for AI systems

Improve long-horizon reliability by explicitly stating carried state to prevent corruption, as a corrupted value persists without correction in every seed and an exact external reminder, repeated before each message, re-aligns it within one to two steps. The improved system should be designed to use an explicit state mechanism from an oracle when tracking critical values.
Implement a component-wise scoring framework for task execution instead of relying on aggregate success metrics, as an aggregate or exclusive error classification can hide the difference we report. This allows practitioners to see that the counter makes no error in 8,000 steps and local processing is descriptively stable while other components degrade.
Enhance duplicate recognition robustness by understanding the interplay between long-distance recall errors and false alarms, noting that The rise combines long-distance recall errors and rising false alarms. The system should be tuned to mitigate both these error sources simultaneously, rather than focusing on a single type of failure.
Optimize context injection strategy by prioritizing state preservation over irrelevant content, as the paper shows that A block with the exact form of the helpful one but irrelevant content was not conclusive and that the counter fails in 20/20 seeds on each machine when irrelevant content is injected.
Design a verification layer for carried state by requiring an explicit external source, as The state came from an oracle and the effective condition repeated it at every step; the necessary frequency was not established. The improved system must rely on a trusted, externally supplied state information rather than attempting to track it purely internally.

Abstract

Long-horizon failures of language models are usually measured as end-task success or as a horizon length. We instead score, step by step, three components of a 400-step ledger task in which every step is easy in isolation and the model's own answers are re-injected into its context: recognising a record already seen 5 to 300 steps earlier, processing the current record locally, and writing a counter incremented at every step. On Qwen3-8B (non-thinking, 20 seeds, replicated on a second machine and backend), with every step scored component by component, two results hold. First, duplicate-recognition errors rise late in the episode -- from 0.0019 to 0.011 per step at risk across step 200 -- while the counter makes no error in 8,000 steps and local processing is descriptively stable; the rise combines long-distance recall errors and rising false alarms. Second, a block of irrelevant content inserted before each step makes the counter fail in 20 of 20 seeds on each machine, against none without it, while its effect on local processing is much smaller and not distinguishable from none. Three pre-registered follow-ups probe the counter. Its value depends on the value present in the response history: a corrupted value persists without correction in every seed, and an exact external reminder, repeated before each message, re-aligns it within one to two steps. Under the irrelevant block, no counter error was observed when an exact compact reminder of the state, supplied externally by an oracle, preceded each message (0 of 20 seeds with an error, against 15 without it), and this reminder outperformed a length-matched neutral text; whether a single reminder suffices was not conclusive, and the absence of damage to the other components was not established. An aggregate or exclusive error classification can hide the difference we report.