MARCH: Scaling Recurrent Memory with Content-Routed State Anchors

MARCH: Scaling Recurrent Memory with Content-Routed State Anchors

Abstract

Transformers owe much of their strong long-context retrieval capability to a token-level memory that grows with context length. This flexibility, however, incurs a quadratic computation complexity during training and a key–value cache that grows linearly during autoregressive inference. Recurrent alternatives offer efficient decoding by compressing the entire history into a fixed-size state, but often underperform on recall-intensive tasks since earlier associations usually get overwritten by subsequent updates, and only the most recent contextual information is retained. In this paper, we introduce Memory-Anchor Routing across Context History (MARCH), a network architecture that effectively scales state-space models beyond a fixed-size dimension, while maintaining computational efficiency over long-sequences. MARCH periodically caches cumulative recurrent-state checkpoints as state anchors and associates each anchor with a compact, content-conditioned anchor key. This lets MARCH maintain a memory bank, which can grow as context length increases, providing a controllable trade-off between historical resolution and memory cost. At each token, MARCH produces an anchor query to attend all causally available state anchors, and the output is calculated as an attention-style aggregation over all historical anchors along the current state. We show that after standard pretraining, MARCH consistently outperforms multiple linear attention variants across commonsense reasoning, LongBench, and in-context retrieval. These results demonstrate that content-routed state caching substantially strengthens recurrent long-range memory while preserving its native computation path.

1. Introduction

Large language models (LLMs) have demonstrated remarkable capabilities across a wide range of language understanding and generation tasks. However, many real-world applications—including long-document understanding, multi-turn interaction, and in-context learning—require models to integrate information distributed across extended sequences. Supporting such contexts involves more than simply increasing the number of input tokens: models must retain relevant information across long intervening spans and reliably retrieve it when needed. Effective long-context modeling therefore hinges on a model’s ability to manage memory—determining what information to preserve, how to represent it, and when to retrieve it. A useful perspective is to view a sequence model as a memory system with two basic operations: writing, which incorporates each new input into memory, and reading, which retrieves information relevant to the current input. Under this view, standard self-attention maintains a growing token-level memory in its key–value cache: it writes by appending each new key–value pair without compressing the existing cache, and reads by matching the current query against all stored keys and combining their associated values. This uncompressed, token-level memory provides a direct path to every preceding token, enabling accurate recall of fine-grained details and distant dependencies. Its flexibility, however, entails quadratic computation during training and a key–value cache that grows linearly with sequence length during autoregressive inference, making self-attention increasingly costly as context windows expand.

Linear attention and modern recurrent sequence models make the opposite trade-off. They compress the causal prefix into a fixed-size, matrix-valued recurrent state, update this state with each new input, and read from it by applying the current query. This design enables constant-memory recurrent decoding, but the same compressed write that makes it efficient also limits its long-range memory. At each step, information from the new token is written into a state already shared by the entire history. Much recent work has therefore focused on improving the write operation. Selective state-space models such as Mamba introduce input-dependent state transitions and forgetting, whereas DeltaNet and Gated DeltaNet use data-dependent delta-rule updates to revise existing associations before incorporating new information. These mechanisms improve state tracking and mitigate indiscriminate accumulation, but they do not eliminate the underlying fixed-state bottleneck: the entire history must still share one evolving state, and only its latest version remains available for reading. Indeed, although recent linear recurrent models can match or surpass softmax attention in short-context settings, their performance often degrades as the evaluation context grows. Once an earlier association has been weakened by forgetting or modified by subsequent writes, the model has no direct path to its earlier representation and cannot recover it from the latest state alone.

Recent works have relaxed the fixed-state bottleneck along two broad directions. One approach increases the memory capacity available at each step through partitioned sparse states, large memories with sparse reads and writes, or routed mixtures of independent states. A second line preserves a temporally structured collection of compressed states through logarithmic hierarchies, adaptive state construction and merging, or recurrent-state caching. Collectively, these approaches demonstrate that expanding memory capacity or temporal coverage can improve long-range recall. However, with the exception of certain instances in [Behrouz et al., 2026], all of the existing works still maintain a finite or upper-constrained state space dimension, though the dimensionality of which is increased. Moreover, as multiple states become available, the primary bottleneck shifts from memory construction to memory retrieval: the model must determine which state retains the information most relevant to the current token. In particular, how to construct context-dependent representations for historical checkpoints that facilitate effective and efficient query-dependent retrieval remains underexplored.

In this work, we introduce Memory-Anchor Routing across Context History (MARCH), a memory-augmented recurrent architecture that enables selective retrieval from earlier versions of recurrent memory. Without modifying the underlying recurrence, MARCH periodically preserves cumulative states as state anchors, giving later tokens access to earlier versions of the evolving memory. Each anchor is associated with a compact learned descriptor, allowing the model to route each token to relevant historical states when additional context is needed and combine their contents with the current-state readout. MARCH thereby complements efficient recurrent processing with selective access to preserved historical memory. The resulting mechanism remains causal and is trained end to end with the standard language-modeling objective.

Main contributions:

• Routable memory-state bank. We introduce MARCH, which expands fixed-state recurrence into a growing bank of historical memory states which increases its capacity as context grows, alleviating the single-state memory bottleneck without modifying the underlying recurrence.

• Content-conditioned historical retrieval. MARCH brings attention-style content routing to recurrent memory by applying a standard softmax over compact keys for a temporally sparse set of state anchors rather than token-level key–value pairs. A learned null route allows the model to suppress the historical branch when the current recurrent state is sufficient, while residual fusion preserves the original recurrent path and supports end-to-end training.

• Extensive empirical validation. We demonstrate consistent improvements over strong recurrent baselines across commonsense reasoning, LongBench, in-context retrieval, and NIAH evaluations, including robust extrapolation beyond the training context length.

2. Preliminaries

Full and Linear Attention as Memory. Let xt ∈ Rd denote the hidden representation at position t. The corresponding query, key, and value vectors are obtained through learned linear projections: qt = Wqxt, kt = Wkxt, vt = Wvxt, where Wq, Wk ∈ Rdk×d and Wv ∈ Rdv×d, such that qt, kt ∈ Rdk and vt ∈ Rdv. Following the memory-system perspective adopted in prior work, we view a causal sequence mixer as an online memory system. Let Mt denote the memory state after processing the first t tokens, with M0 denoting its initial state. At each position, the current key–value pair is first written into memory, after which the updated memory is queried using the current query: Mt = Write(Mt−1; kt, vt), ot = Read(Mt; qt), where ot ∈ Rdv denotes the memory readout at position t. For causal softmax attention, the memory explicitly retains all projected key–value pairs observed up to position t: Mt = (K≤t, V≤t), where K≤t ∈ Rt×dk and V≤t ∈ Rt×dv stack the keys and values row-wise, respectively. Writing appends (kt, vt) to these matrices, whereas reading performs content-based retrieval: ot = V⊤≤t softmax(K≤tqt/√dk). This explicit storage keeps individual tokens directly retrievable and enables fine-grained retrieval from the entire causal prefix. However, processing a sequence of length T requires O(T2) query–key interactions, while autoregressive decoding maintains a key–value cache of size O(T(dk+dv)) per attention head.

Linear attention instantiates the memory state Mt as a fixed-size matrix St ∈ Rdv×dk. Its write and read operations are given by St = St−1 + vtkt⊤, ot = Stqt. Each write therefore adds a rank-one key–value association to the shared matrix, while each read retrieves a query-dependent superposition of the stored values. This enables constant-memory recurrent decoding, but introduces interference as the compressed history grows.

Gated DeltaNet (GDN). To mitigate the interference caused by the additive write rule of linear attention, GDN retains the same state-based read operation but introduces input-dependent retention and a targeted delta-rule write: St = alphatSt−1 + betat(vt − alphatSt−1kt)kt⊤, ot = Stqt, where alphat ∈ (0,1) is an input-dependent retention gate, and betat ∈ [0,1] modulates the strength of the targeted delta update. Despite its more adaptive state dynamics, GDN still compresses the entire causal history into a single fixed-size recurrent state. Because all associations share this evolving state, information weakened or overwritten by subsequent updates has no direct retrieval path, limiting reliable long-context recall.

Scaling recurrent memory. Recent work has sought to relax the fixed-state bottleneck along two broad directions. Capacity-expansion methods enlarge the current recurrent state, whereas temporal-expansion methods retain multiple versions of the state along its trajectory: M cap t:= S̃t = [St(1) · · · St(P)] ∈ Rdv×Dmem, Dmem = Σp=1..P dp; M temp t:= St,1,..., St,Mt, St,m ∈ Rdv×dk, 1 ≤ taut,1 < · · · < taut,Mt ≤ t. In the capacity formulation, P is the fixed number of state partitions and Dmem is their total memory dimension. Such methods increase Dmem while using sparse access to keep computation tractable. In the temporal formulation, Mt is the number of retained state representations, each associated with a temporal boundary taut,m. These methods preserve states from distinct temporal regions or earlier stages of the recurrent trajectory. MARCH follows the latter direction by retaining cumulative snapshots of a continuously evolving recurrent state.

3. Method

Figure 2 illustrates MARCH, a content-routed recurrent memory framework that enables selective retrieval from earlier versions of an evolving recurrent state. Rather than routing over predefined state indices or temporal scales, MARCH matches each query against individual historical states based on their contents. As tokens are processed, MARCH periodically checkpoints the cumulative recurrent state, producing a bank of state anchors. Each checkpoint is paired with an occurrence of a shared learned anchor token, whose hidden representation yields a compact routing key. For each text token, a routing query scores all causally visible anchors alongside a learned null option, allowing the model to use historical memory only when useful. The resulting routing probabilities define a weighted combination of the visible anchor states, which is read using the token’s standard recurrent query. This historical readout is added to the current-state readout, preserving the native recurrent path while introducing a content-dependent route to earlier memory. Together, state anchoring and content-routed retrieval turn the otherwise transient state trajectory into a persistent source of long-range memory.

3.1. Continuous Recurrent-State Anchoring

Anchor placement. Let T = [t1,..., tL] be a sequence of L text tokens. An anchoring policy specifies an ordered set of text boundaries B = bm m=1..M, where 0 = b0 < b1 < · · · < bM ≤ L. We insert an anchor position after each boundary: T̃ = ∥m=1..M [tbm−1+1,..., tbm] ∥ [xim] ∥ [tbM+1,..., tL], where ∥ denotes sequence concatenation, and xim is the m-th occurrence of a shared learned anchor embedding xi. Text and anchor positions serve different computational roles. Text positions apply the base recurrent update, allowing the matrix-valued state to evolve continuously across anchor boundaries. Immediately after processing tbm, MARCH checkpoints the resulting cumulative state to form the m-th state anchor. The following anchor position xim does not modify the recurrent state; instead, its hidden representation provides the routing metadata associated with that checkpoint. Thus, each anchor boundary produces two coupled objects: a snapshot of the recurrent memory and a compact representation through which that snapshot can later be retrieved.

Cumulative recurrent-state checkpointing. At layer l, MARCH leaves the underlying recurrent update unchanged and carries the state St(l) ∈ Rdv×dk continuously across anchor boundaries. At each boundary bm, it snapshots the current state: A(m,l) = Sbm(l) ∈ Rdv×dk, m = 1,..., M. Because the recurrence is not reset between anchor boundaries, A(m,l) encodes the cumulative prefix up to position bm, rather than only the segment since the preceding boundary. We therefore refer to it as a state anchor. The ordered bank A(1,l),..., A(M,l) traces the temporal evolution of a single recurrent memory, preserving earlier versions before subsequent decay and delta updates attenuate or modify their contents.

Content-conditioned anchor metadata. Let um(l) denote the normalized input representation of anchor position xim at layer l. The anchor position reads only its aligned state checkpoint: qm(l) = Wq(l)um(l), om(l) = A(m,l)qm(l). The same input representation is projected into a compact routing key: kappam(l) = Wk(l)um(l) ∈ Rdr. The aligned readout is incorporated into the anchor position through the standard output projection and residual pathway. Consequently, um(l+1) depends on A(m,l), and the routing key produced at layer l+1 becomes conditioned on the content retained by the aligned state anchor. Thus, although all anchor positions share the same learned input embedding, they acquire distinct, state-dependent representations after the first layer. This cross-layer construction makes routing explicitly dependent on what each state anchor contains, rather than only on its temporal index.

3.2. Content-Routed Historical Reading

Content-based routing. For a text token at posit…

Based on the paper, here are the specific improvements you can make to AI systems and what the improved system can do:

Improvement 1: Content-Routed Historical State Retrieval

• What to implement: Add a memory bank of periodic recurrent-state checkpoints (state anchors), each paired with a learned, content-conditioned routing key. At each token, compute a routing query that scores all causally visible anchors plus a learned null option, then aggregate the top or all anchors’ states weighted by softmax probabilities, and add this historical readout to the current-state readout.

• What the improved AI system can do: Retrieve earlier versions of its compressed memory that would otherwise be overwritten or forgotten. This enables accurate recall of distant facts, multi-hop reasoning across long contexts, and fine-grained retrieval from earlier parts of a sequence—without quadratic attention cost or a linearly growing KV cache. It can also bypass historical memory when the current state is sufficient, preserving efficiency.

Improvement 2: Scalable, Content-Adaptive Memory Capacity

• What to implement: Replace the fixed-size recurrent state with a growing bank of state anchors that increases in number as context length grows. Use periodic checkpointing (e.g., every 512 tokens) and allow the bank to expand without bound, while keeping per-token computation sublinear via sparse routing (e.g., Top-4 anchors).

• What the improved AI system can do: Maintain long-context performance beyond its training length (e.g., extrapolate from 16K to 32K tokens) without new parameters or retraining. It can handle arbitrarily long documents, multi-turn dialogues, and in-context learning tasks where earlier information must be retrieved after long intervening spans, while keeping memory and compute costs controllable via checkpoint density and routing sparsity.

Improvement 3: Hardware-Efficient Fusion of Routing and State Aggregation

• What to implement: Use a two-stage producer–reader computation with chunkwise recurrent updates and I/O-aware tiling. Fuse routing-score computation, online softmax, and weighted state accumulation into a streaming reduction, avoiding materialization of dense routing matrices or per-anchor readout tensors.

• What the improved AI system can do: Achieve higher training throughput than FlashAttention-2 at sequence lengths of 64K and above, and lower core runtime from 32K onward. This makes long-context training and inference practical on existing GPUs, enabling deployment in resource-constrained settings without sacrificing retrieval quality.

Improvement 4: Adaptive Anchoring and Hierarchical Memory Organization

• What to implement: Extend the fixed-interval checkpointing to adaptive anchoring based on state novelty or update magnitude (e.g., checkpoint more frequently when the state changes rapidly, consolidate redundant anchors in stable regions). Optionally, organize the anchor bank hierarchically (e.g., Fenwick tree) to retain O(log T) anchors as context grows.

• What the improved AI system can do: Optimize memory usage dynamically, reducing redundancy and improving temporal resolution where it matters most. This yields a flexible accuracy–memory trade-off at inference, allowing the system to adapt its memory footprint to the task’s demands (e.g., denser anchors for retrieval-heavy tasks, sparser for general reasoning) without retraining.

Improvement 5: Null-Route Suppression and Residual Fusion

• What to implement: Include a learned null option in the router that can suppress the historical branch entirely, and add the historical readout as a residual to the current-state readout, preserving the original recurrent path.

• What the improved AI system can do: Automatically decide when historical memory is unnecessary, avoiding noise or interference from irrelevant past states. This improves robustness on short-context tasks and prevents performance degradation, while maintaining end-to-end trainability with standard language-modeling objectives.

Improvement 6: Multi-Scale and Specialized Memory Partitions

• What to implement: Combine state anchoring with multiple memory partitions (e.g., short-term, episodic, and consolidated knowledge), each with distinct write, retention, and forgetting mechanisms, and use a hierarchical router to select both the partition and the stored state for each query.

• What the improved AI system can do: Handle heterogeneous information types (e.g., recent dialogue, salient events, long-term facts) with specialized memory, improving performance on tasks requiring both immediate context and distant knowledge. This enables more human-like memory consolidation and supports continual learning or test-time training by updating only relevant partitions.