Hierarchical Continuous Diffusion Language Models
cs.CL, cs.AI, cs.LG
Submitted: 2026-10-01
Updated: 2026-10-01
Project page: https://hc-dlm.github.io
License: http://creativecommons.org/licenses/by/4.0/
The gist: Discrete diffusion language models offer a compelling alternative to autoregressive generation for tasks demanding bidirectional reasoning and global constraint satisfaction.
Terminology
Abstract
Discrete diffusion language models offer a compelling alternative to autoregressive generation for tasks demanding bidirectional reasoning and global constraint satisfaction. Yet they share a structural bottleneck: when decoding in parallel, each token is sampled independently from its marginal, severing the statistical dependencies among the tokens decoded together. Continuous diffusion language models avoid this by denoising a shared continuous state, but their denoiser sees only that state, so nothing ties it to a valid token configuration until it is finally decoded. To address this, we propose Hierarchical Continuous Diffusion Language Models (HC-DLM), which couple discrete token generation with a continuous latent trajectory in a single, principled denoising process, whose training objective is derived from a variational bound on the token likelihood. In contrast to recent methods that attach continuous context to a self-contained discrete chain, HC-DLM makes the latent the only persistent generative state: tokens are read out from it at every step and feed back as a scaffold for the next latent update. On structured reasoning (Sudoku), mathematical planning (Countdown) and language modeling (LM1B), HC-DLM improves over discrete and continuous diffusion baselines at matched model size, in puzzle accuracy on Sudoku and Countdown and in generative perplexity on LM1B. Project page: https://hc-dlm.github.io/.
Sources
- One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling
- DiT-Air: Revisiting the Efficiency of Diffusion Model Architecture Design in Text to Image Generation
- LangFlow: Continuous Diffusion Rivals Discrete in Language Modeling
- Continuous Latent Diffusion Language Model
- Locally Coherent Parallel Decoding in Diffusion Language Models
- ELF: Embedded Language Flows
- DAPD: Dependency-Aware Parallel Decoding via Attention for Diffusion LLMs
- DiLaDiff: Distilled Latent-Augmented Diffusion for Language Modeling
- Breaking the Factorization Barrier in Diffusion Language Models
- Understanding Diffusion Models: A Unified Perspective
- How to Train Your Latent Diffusion Language Model Jointly With the Latent Space
- CANDI: Hybrid Discrete-Continuous Diffusion Models
- Dependency-Guided Parallel Decoding in Discrete Diffusion Language Models
- Latent-Augmented Discrete Diffusion Models
- CoDAR: Continuous Diffusion Language Models are More Powerful Than You Think
- Qwen3 Technical Report
- Variational Masked Diffusion Models
- A Reparameterized Discrete Diffusion Model for Text Generation
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering