MDLMPE: Distribution Aware Positional Encoding for Masked Diffusion Language Models
Tong Ling, Hang Lei, Feng Xiao, Changhui Sun, Jiahang Xie, Hao Liu, Lu Liu, Yanlong Du
cs.CL, cs.AI
Submitted: 2026-08-04
Project page: http://skylion007.github.io/OpenWebTextCorpus
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Terminology
Sources
- SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model
- Program Synthesis with Large Language Models
- LLaDA2.0: Scaling Up Diffusion Language Models to 100B
- One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling
- DFlash: Block Diffusion for Flash Speculative Decoding
- Evaluating Large Language Models Trained on Code
- Extending Context Window of Large Language Models via Positional Interpolation
- Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
- Training Verifiers to Solve Math Word Problems
- LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens
- UltraLLaDA: Scaling the Context Length to 128K for Diffusion Large Language Models
- Tulu 3: Pushing Frontiers in Open Language Model Post-Training
- LongLLaDA: Unlocking Long Context Capabilities in Diffusion LLMs
- Pointer Sentinel Mixture Models
- Seed Diffusion: A Large-Scale Diffusion Language Model with High-Speed Inference
- LLaMA: Open and Efficient Foundation Language Models
- MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark
- Dream 7B: Diffusion Large Language Models
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering