ReWY: Faster Backpropagation for Delta-Rule Attention - From 7 to 3 Matrix Multiplications
cs.LG, cs.DC
Submitted: 2026-11-01
Updated: 2026-11-03
Code: https://github.com/flashserve/ReWY
Terminology
Sources
- FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
- Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention
- Kimi Linear: An Expressive, Efficient Attention Architecture
- A Contract-Grade Verifier for LLM-Generated GPU Kernels, and a Native Blackwell Backward for the Gated-Linear-Recurrence Family
- Gated Delta Networks: Improving Mamba2 with Delta Rule
- Parallelizing Linear Transformers with the Delta Rule over Sequence Length
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis