RangeSWA: Training-Free Length Extrapolation for Continuous Diffusion Language Models
cs.CL
Submitted: 2026-10-30
Updated: 2026-10-30
License: http://creativecommons.org/licenses/by-sa/4.0/
The gist: Continuous diffusion language models show promising potential in text generation tasks, but are prone to problems such as text repetition, semantic drift…
Terminology
Abstract
Continuous diffusion language models show promising potential in text generation tasks, but are prone to problems such as text repetition, semantic drift, and generation quality degradation when the generation length exceeds the training length. To address these challenges, we propose RangeSWA, a training-free length extrapolation method. The method combines local sliding-window attention, remote block summaries, and dynamic content retrieval to preserve local details and support long-range context access. In addition, generation degradation caused by length extrapolation is mitigated through positional remapping, conditioning attention correction, and denoising-stage-adaptive window scheduling. We conducted experiments on multiple continuous diffusion language models, and the results show that RangeSWA can effectively reduce generative perplexity and text repetition rates in long-text generation.
Sources
- Longformer: The Long-Document Transformer
- Extending Context Window of Large Language Models via Positional Interpolation
- LangFlow: Continuous Diffusion Rivals Discrete in Language Modeling
- Hacking Generative Perplexity: Why Unconditional Text Evaluation Needs New Metrics
- DiffuSeq: Sequence to Sequence Text Generation with Diffusion Models
- Continuous Latent Diffusion Language Model
- LM-Infinite: Zero-Shot Extreme Length Generalization for Large Language Models
- ELF: Embedded Language Flows
- LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuning
- Sliding-window beats linear attention
- YaRN: Efficient Context Window Extension of Large Language Models
- Simple and Effective Masked Diffusion Language Models
- RoFormer: Enhanced Transformer with Rotary Position Embedding
- Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference
- Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention
- A Dominant Self-Conditioning Direction Drives Repetition in Unconditional Continuous Diffusion Language Models
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data