Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention

summary

The gist

Linear attention replaces the unbounded cache of softmax attention with a fixed-size recurrent state, reducing sequence mixing to linear time and decoding to constant memory The Gated DeltaNet-2…

In short

Gated DeltaNet-2 introduces a new recurrent attention layer that separates memory editing into distinct channel-wise erase and write operations. This decoupling allows the model to selectively remove stale associations while inserting only necessary new values, improving targeted memory control and performance across various tasks.

Key concepts

Gated Delta Rule-2
This is the core innovation where erasing old content and writing new content are handled by separate channel-wise gates ($b_t$ and $w_t$). This allows the model to clear broad context, remove specific stale associations, and insert only persistent value channels independently.
Decoupling Memory Operations
The key idea is separating the roles of erasing (removing old content) and writing (inserting new content). This is achieved by using independent projections for the erase gate ($b_t$) and write gate ($w_t$), enabling targeted editing without scrambling existing associations.
WY Form
The design absorbs cumulative channel-wise decay into rank-one erase factors, resulting in a compact WY form. This structure maintains the efficient chunkwise training pattern used by previous delta-rule kernels, ensuring fast weight updates during training.
Asymmetric Gate Contribution
Ablations show that the erase gate ($b_t$) contributes most to performance gains. Keeping channel structure only in $b_t$ recovers most of the full model's capability on language modeling and retrieval, highlighting its critical role in memory management.

Terminology used across episodes

This episode discusses

The paper

Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention · Read on arXiv

Ali Hatamizadeh, Yejin Choi, Jan Kautz

NVIDIA

Linear attention replaces the unbounded cache of softmax attention with a fixed-size recurrent state, reducing sequence mixing to linear time and decoding to constant memory. The hard part is not just what to forget, but how to edit this compressed memory without scrambling existing associations. Delta-rule models subtract the current read before writing a new value, and Kimi Delta Attention (KDA) sharpens forgetting with channel-wise decay. But the active edit still uses a single scalar gate to control two different things: how much old content to erase on the key side and how much new content to commit on the value side. We introduce Gated DeltaNet-2, which generalizes both Gated DeltaNet and KDA by inheriting adaptive forgetting and channel-wise decay while addressing their shared limitation, the scalar tie between erasing and writing. Gated Delta Rule-2 separates these roles with a channel-wise erase gate b t and a channel-wise write gate w t, reducing to KDA when both gates collapse to the same scalar and to Gated DeltaNet when the decay also collapses. We derive a fast-weight update view, a chunkwise WY algorithm with channel-wise decay absorbed into asymmetric erase factors, and a gate-aware backward pass that preserves efficient parallel training. At 1.3B parameters trained on 100B FineWeb-Edu tokens, Gated DeltaNet-2 achieves the strongest overall results among Mamba-2, Gated DeltaNet, KDA, and Mamba-3 variants across language modeling, commonsense reasoning, and retrieval. Its advantage is most pronounced on long-context RULER needle-in-a-haystack benchmarks, where it improves the evaluated multi-key retrieval setting and remains strong in both recurrent and hybrid settings. Code is available at https://github.com/NVlabs/GatedDeltaNet-2.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention".

Jane: Here's the gist of the paper. Linear attention replaces the unbounded cache of softmax attention with a fixed-size recurrent state, reducing sequence mixing to linear time and decoding to constant memory The Gated DeltaNet-2 architecture decouples the erase and write operations in the delta rule by introducing channel-wise erase and write gates, which generalizes previous models by inheriting adaptive forgetting and channel-wise decay This decoupling allows for targeted editing of compressed memory without scrambling existing associations Decoupling Memory Operations The core innovation is Gated Delta Rule-2, which separates the roles of erasing old content and writing new content through a channel-wise erase gate b t and a channel-wise write gate w t The model can clear broad context through decay, remove selected stale associations through erase, and insert only the value channels that should persist through write This separation is achieved by treating both decisions as independent projections of the token representation, b t = sigma(Wb x t) and w t = sigma(Ww x t) When the erase and write gates are tied to the same scalar, Gated DeltaNet-2 recovers KDA Preserving Efficient Training Paths The design preserves the efficient training path by absorbing cumulative channel-wise decay into the rank-one erase factors, resulting in a compact WY form with the same high-level chunkwise structure used by efficient delta-rule kernels The fast-weight update view interprets Gated Delta Rule-2 as an online update of a fast-weight memory state This leads to the minimization of the local online problem S t = arg min S L t(S), where L t(S) is defined by a residual comparing the gated write target z t against the content read from S¯ t along e t Performance and Empirical Results Empirically, Gated DeltaNet-2 improves the recurrent attention frontier, showing clearest gains on long-context retrieval tasks On RULER needle-in-a-haystack benchmarks, it remains strong as context length grows and is especially effective on the evaluated multi-key case where a fixed-size state must separate competing associations The model achieves the strongest overall results among Mamba-2, Gated DeltaNet, KDA, and Mamba-3 variants across language modeling, commonsense reasoning, and retrieval at 1.3B parameters trained on 100B FineWeb-Edu tokens Gate Structure Ablations Ablations show that both gates contribute to the performance gain, with the erase gate accounting for most of it Keeping channel structure only in b t recovers most of the full model on language modeling and retrieval, while keeping it only in w t recovers less The asymmetry is clear, as keeping channel structure only in b t recovers most of the gain on language modeling and retrieval Throughput and Practicality Gated DeltaNet-2 preserves the near-flat scaling profile of recurrent mixers as sequence length grows, dropping only mildly from 38.0 to 36.1 Kt/s Relative to KDA, the small gap reflects the added channel-wise erase and write gates The layer produces the two gates with independent projections of the token representation, b t = sigma(Wb x t), w t = sigma(Ww x t) This gate-aware accumulation is the main mathematical change required for training Gated Delta Rule-2 The layer produces RMS-normalized, multiplied by a separate SiLU output gate, and projected back to the model dimension Chunkwise Derivation Details The chunkwise derivation shows that the channel-wise decay is absorbed into the two factors of each rank-one erase, while the update remains a product of matrices of the form I - k¯ r e¯ r T The end-of-chunk state is then S

n+1: = Diag(g C)S

n: + K T tail(U - YS

n: ) The output at token r is O = Q g S 0 + A q k (U - YS 0), which is Eq.

Tom: 24 in the main text The backward pass involves a gate-aware accumulation where dA += dU Z T and dA += dY E¯ T This is the main Gated DeltaNet-2 specific kernel Numerical Stability and Verification The decay gate in Eq.

Jane: 86 is computed in explicit fp32 before entering the kernels because the local cumulative sum G r = sum over i <= r g i is a path-length-dependent quantity The triangular solve for A = (I + T)-1 is the most precision-sensitive part of the chunk computation In fp64 reference tests, gradients for Q, K, V, B, W, the log-decay, and the initial state agree to machine precision The model is trained on 100B tokens from FineWeb-Edu

26: The training sequence length is 4K tokens We introduced Gated DeltaNet-2, a delta-rule recurrent attention layer that decouples the active memory edit into channel-wise erase and write decisions This removes the scalar delta gate tie in Gated DeltaNet and KDA, recovers both as special cases, and preserves efficient chunkwise training through a WY form with gate-aware kernels Under matched 1.3B training, Gated DeltaNet-2 improves the recurrent and hybrid frontier across language modeling, commonsense reasoning, synthetic retrieval, and real-world recall The erase gate accounting for most of the gain in ablations suggests its critical role The key-side erase gate b t selectively protects or revises key channels, while the value-side write gate w t controls which value channels enter the state This decoupled recurrent update preserves longer-range associations more effectively than a scalar delta gate The hybrid model retains linear sequence scaling and a bounded attention cache, following the recurrent attention hybrid design pattern

24, 25: References

1: Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret.

Tom: Transformers are rnns: Fast autoregressive transformers with linear attention.

Title and authors: Tom: So we're looking at this paper, Gated DeltaNet-two: Decoupling Erase and Write in Linear Attention, and it tackles something really specific about how these recurrent attention models handle memory editing <ref:2605.22791#pg1,Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention>.

Jane: Right. Basically, the main thing they’re focusing on is that the existing models had this problem where they used just one scalar gate to control both erasing old information from the key side and writing new information to the value side.

Lu: Exactly. The authors introduced two separate gates, a channel-wise erase gate b t and a channel-wise write gate w t, which are projections of the token representation themselves.

Meng: So instead of one scalar controlling both operations, you have these two independent decisions happening at the channel level? That sounds like it would make targeted editing much more precise.

Lalam: It does. The paper says this decoupling allows them to clear broad context through decay, remove selected stale associations through erase, and insert only the value channels that should persist through write.

Tom: That’s what they call separating the roles of erasing old content and writing new content, which is a big deal because it means you can edit memory without completely scrambling everything else.

Jane: And they show that when both these gates collapse back into the same scalar, it just recovers models we already know, like KDA or Gated DeltaNet.

Lu: That’s important because it shows they aren't inventing something entirely new, but rather generalizing previous work by inheriting things like adaptive forgetting and channel-wise decay while fixing that specific tie.

Meng: How does this change the training side? We all know that introducing complexity can slow down training significantly.

Tom: They actually did a lot of work to keep the efficient training path intact, and they do it by absorbing the cumulative channel-wise decay into those rank-one erase factors.

Jane: So they end up with this compact WY form with the same high-level chunkwise structure that you see in other efficient delta rule kernels.

Lu: That structure is key because it keeps things fast during training, and they interpret Gated Delta Rule-two as an online update of a fast-weight memory state <ref:2605.22791#pg1>.

Meng: What does that mean for deployment? Does this add overhead when we're running these models at scale?

Title and authors: Tom: It preserves the near flat scaling profile of recurrent mixers as the sequence length grows, dropping only mildly from thirty-eight point zero to thirty-six point one teraflops per second.

Jane: That's a small dip, and they say that gap is just due to those extra channel-wise erase and write gates they introduced.

Lu: Empirically, Gated DeltaNet-two shows clearest gains on long context retrieval tasks, especially on the RULER needle-in-a-haystack benchmarks where a fixed size state has to separate competing associations <ref:2605.22791#pg1>.

Meng: So it’s better at keeping track of things when the context gets really long? What about real world recall, not just benchmarks?

Lalam: The paper shows that in real world recall, Gated DeltaNet-two gives the strongest overall retrieval profile in both recurrent and hybrid settings <ref:2605.22791#pg1>.

Tom: It also sees gains across language modeling and commonsense reasoning, which is good because it shows it’s not just for one niche task.

Jane: And when we look at the gate structure ablation studies, they found that both gates contribute to the performance boost, but the erase gate actually accounts for most of that gain.

Lu: They also noted that keeping only the channel structure in the erase gate b t recovers most of the full model on language modeling and retrieval tasks.

Meng: That tells me we should probably focus our testing on those long context retrieval scenarios if we want to see a real payoff in production environments.

Tom: And they confirmed that when both gates are tied to the same scalar, it recovers KDA, which is good because it means this new structure is more flexible than just a simple scalar gate.

Jane: So what’s the big picture implication here? It seems like the core pressure point for fixed-state recurrence was that one scalar had to do too much work controlling two different kinds of memory operations simultaneously.

Lu: Precisely. The Gated DeltaNet-two paper shows that decoupling those operations—erasing and writing—allows for targeted editing without scrambling existing associations, which is a big win <ref:2605.22791#pg1>.

Meng: So, if we were building a system for something like long document summarization or complex retrieval, this could be the way to go because we can selectively remove noise while keeping the relevant data intact.

Title and authors: Tom: Yeah, and it maintains that linear sequence scaling profile they wanted from the start, which is crucial if you want to keep things efficient as the context window grows.

Jane: It’s a neat design because it keeps that bounded attention cache while still allowing for long context handling through the recurrent state.

Lu: The chunkwise derivation details show that this channel-wise decay is absorbed into the two factors of each rank one erase, and the update remains a product of matrices of the form I minus k bar r e bar t.

Meng: That sounds mathematically sound, but how much effort is it to implement these new projections for b t and w t in our existing pipeline?

Tom: The layer produces these two gates with independent projections of the token representation, b t equals sigma of W b x t and w t equals sigma of W w x t.

Jane: That means the layer itself has to produce a separate output path for each gate, which adds some complexity to the architecture itself.

Lu: But it’s worth that gate-aware accumulation is the main mathematical change required for training Gated Delta Rule-two <ref:2605.22791#pg1>. It’s how they handle things on the back end.

Meng: Okay, so we need to assess if that added complexity in production is worth the retrieval improvements they're seeing on those long context tasks.

Tom: They did show a clear advantage on those RULER needle-in-a-haystack tasks, and it holds up as the context length grows.

Jane: It’s interesting how effective it is especially on that evaluated multi key case where you have a fixed size state trying to separate many competing associations.

Lu: That separation capability is what makes this design work well in those challenging scenarios where the memory state has to be very specific.

Meng: So, if we compare this against Mamba-two or Gated DeltaNet, what’s the practical difference when we look at inference speed <ref:2605.22791#pg1>?

Tom: It's near flat scaling relative to sequence length, which is good for throughput, and it only drops mildly from thirty-eight point zero to thirty-six point one teraflops per second.

Jane: That mild drop is what they attribute to those extra gate operations they added for the erase and write functions.

Lu: The numerical stability is also being handled carefully; the decay gate in equation eighty six is computed in explicit fp32 before entering the kernels because it’s a path length dependent quantity.

Title and authors: Meng: So, we have these performance gains on retrieval and language modeling, but we need to be sure about the actual runtime cost under heavy load.

Tom: The training was done on one hundred billion tokens from FineWeb-Edu, which is a massive dataset for this kind of work.

Jane: It’s impressive that they achieved the strongest overall results among Mamba-two Gated DeltaNet, KDA and Mamba-three variants at one point three billion parameters trained on that scale.

Lu: The key thing to remember is the asymmetry in the ablations; keeping channel structure only in b t recovers most of the gain on language modeling and retrieval.

Meng: So for us, if we are prioritizing general language modeling, focusing on that erase gate seems like a smarter place to start looking at implementation details.

Tom: It really does point to its critical role there. The key side erase gate b t selectively protects or revises key channels while the value side write gate w t controls which value channels enter the state.

Jane: That separation of control along different channel patterns is what allows it to preserve longer-range associations more effectively than a single scalar delta gate could do.

Lu: It’s about controlling old content and new values along different channel patterns, which is essential for handling interference among many compressed associations in long contexts.

Meng: So, the paper on Gated DeltaNet-two really shows that we can move beyond simple scalar gates to get fine-grained control over memory operations in these recurrent models <ref:2605.22791#pg1>.

Tom: It confirms the idea that decoupling erase and write is a viable way to solve the pressure point of fixed state recurrence.

Jane: We've got a paper here called Gated DeltaNet-two: Decoupling Erase and Write in Linear Attention, which really shows how separating those two actions can improve performance across retrieval, reasoning, and language modeling <ref:2605.22791#pg1,Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention>.

Lu: It’s a solid paper because it keeps the efficient training path intact through that compact WY form with gate aware kernels.

Meng: For me, the key is assessing how much complexity we need to add versus the retrieval gains we see on those needle-in-a-haystack tasks.

Tom: It seems like a very promising direction for improving how we handle compressed memory in these attention architectures.

The paper's summary: Tom: So, we've been looking at Gated DeltaNet-two and now let's talk about what they actually did in their summary of the paper.

Jane: Okay, so basically, the main point is this model uses two separate gates: an erase gate and a write gate.

Tom: Right. The core idea is decoupling the erase operation—removing old info—from the write operation—adding new info to the memory state.

Lu: It’s not just that they have two gates; it's *how* they use them, you see. They treat these decisions as independent projections of the token representation itself.

Jane: So, instead of one single number controlling both erasing and writing, you've got these two separate ways the model decides what to do at the channel level.

Tom: Exactly. And they say this lets them target editing specifically by saying, "remove these stale associations through erase, and insert *only* the value channels that should persist through write."

Lu: That is a huge deal because it means you can clear broad context using decay, but you can also surgically remove specific bad associations without messing up the rest of the memory.

Jane: But they also showed that if you force those two gates to act like one scalar again, it just turns back into models we already know, like KDA or Gated DeltaNet.

Tom: So it proves that this decoupling isn't inventing something entirely new; it's just a more flexible way to do what we already did.

Lu: And they kept the training side efficient by absorbing the cumulative decay into those rank one erase factors, which keeps them using a compact structure called WY form.

Jane: So, for someone just listening, it means this model can handle long contexts better while still keeping that fast update speed during training.

Tom: It does. And empirically, they found that this approach gives the best overall results among several strong variants when trained on a huge dataset of one hundred billion tokens.

Lu: The ablation studies pointed out that the erase gate contributes most of the performance gain, and keeping just the channel structure in the erase gate actually recovered most of it for language modeling tasks.

Jane: So, it seems like controlling what you keep versus what you throw away is a more powerful lever than just having one global control knob.

Tom: It really is. The erase gate selects which key channels to protect or revise, while the write gate controls which value channels get to enter the state.

Lu: That separation of control along different channel patterns is what lets it preserve longer-range associations much more effectively than a simple scalar gate could manage.

Jane: So, for someone building an AI system, this suggests you can get fine-grained control over memory editing without introducing massive computational overhead that ruins the training speed.

Tom: It’s promising because it keeps that linear scaling profile they wanted while giving you much smarter control over the compressed memory state.

Lu: We still need to see how this translates when we push it to real traffic, but for now, this looks like a solid way to handle the limitations of fixed-size recurrent states.

The paper's improvements: Tom: So, we’re looking at what Gated DeltaNet-two suggests as improvements for their model architecture itself, not just how they trained it.

Jane: Okay, the authors are really highlighting how this decoupling makes the model more flexible than previous versions.

Tom: They show that by having independent projections for both the erase gate and the write gate, they can control old content and new values along different channel patterns at once.

Lu: That’s what matters because it means you have a much finer way to manage interference among all those compressed associations in long contexts.

Jane: It’s like having two different pens for editing a document instead of just one dull pencil that does everything poorly.

Tom: And they confirm that this asymmetry is pretty important, showing that keeping the channel structure only in the erase gate gives them most of the performance boost on language modeling and retrieval tasks.

Lu: That tells us where to focus if we want to get the biggest immediate win, which is definitely in language modeling and retrieval.

Jane: But they also show that if you tie those gates together with a single scalar, it recovers KDA, proving this new decoupled way isn't just an extra option; it’s a more capable general framework.

Tom: It means the design is robust enough to handle simpler cases while still offering that deep control when you need it most.

Lu: And they also showed how they preserved those efficient training paths by absorbing the cumulative decay into those rank one erase factors, keeping things compact in that WY form.

Jane: So, even with this new complexity for better memory control, they managed to keep the training process relatively smooth and fast.

Tom: They did. The layer still produces these two gates with independent projections of the token representation—the erase gate projects from model dimension to eight times the key dimension—and a separate write gate that projects from model dimension to twice its own dimensions.

Lu: That’s a big mathematical change, producing those two distinct control signals independently, which is what makes it work.

Jane: So, the implication here is that we can build recurrent attention systems that are both efficient and incredibly precise about what they remember and what they forget.

Tom: It really does. And when you combine this with their strong empirical results on those long context retrieval benchmarks, it suggests a model that’s much better at handling complex, long-range reasoning.

Lu: It confirms the ambition: moving from simple scalar control to channel-wise control is a valid path for improving memory dynamics in these architectures.

Conclusion: Tom: So we’re wrapping up on Gated DeltaNet-two by summarizing what it means for the field and saying our goodbyes to this paper.

Jane: Basically, we’ve seen that this model takes a fixed-size recurrent state and makes editing memory much smarter by separating the erase and write operations.

Tom: Right. The main implication is that we can have models with bounded attention caches but still manage very long context retrieval tasks effectively.

Lu: It changes how we think about memory management in these sequence models because it moves us past simple scalar gates to a more nuanced control over the state transitions.

Jane: That’s what I like to hear; moving toward more specific control instead of just one big switch.

Tom: And the numbers back up this idea; they showed it performs at one point three billion parameters trained on one hundred billion tokens, beating several other strong variants in language modeling and retrieval.

Meng: From my side, I’m still looking at the operational cost, but if it means we can deploy a model that handles long sequences better without needing a massive cache for every single input, that’s valuable.

Lu: It's about finding the right balance between performance gains and the computational overhead of those extra gates.

Jane: I think it shows we can push the limits of what these recurrent architectures can do with context length while staying relatively efficient in training.

Tom: So, Gated DeltaNet-two confirms that decoupling erase and write is a viable way to solve the pressure point of fixed state recurrence.

Lu: It’s a solid direction for future work because it shows that channel-wise control is the way to go for this type of memory structure.

Jane: We're going to keep watching how these kinds of decoupled operations evolve as we build bigger and better sequence models.

Tom: Next week, we're diving into something completely different, so stick around.

More episodes

← Home