LangFlow: Continuous Diffusion Rivals Discrete in Language Modeling
cs.CL, cs.LG
Submitted: 2026-04-13
Updated: 2026-11-05
Comments: NeurIPS 2026
Code: https://github.com/nealchen2003/LangFlow
License: http://creativecommons.org/licenses/by/4.0/
The gist: Continuous diffusion has served as a foundation for high-fidelity, controllable, and few-step generation across continuous data modalities such as images, videos, and molecular structures.
Terminology
Abstract
Continuous diffusion has served as a foundation for high-fidelity, controllable, and few-step generation across continuous data modalities such as images, videos, and molecular structures. However, in language modeling, prior continuous diffusion language models (DLMs) lag behind discrete counterparts. Existing categorical and simplex-based approaches operate over extremely large and sparse language spaces, while prior embedding-space approaches avoid this sparsity but lack a well-selected design space. In this work, we close this gap with LangFlow by connecting embedding-space DLMs to Flow Matching, alongside three key innovations: (1) we derive a novel ODE-based NLL bound for principled evaluation of continuous flow-based language models; (2) we propose an information-uniform principle for setting the noise schedule, which motivates a learnable noise scheduler based on a Gumbel distribution; and (3) we revise prior training protocols by incorporating self-conditioning, which improves both likelihood and sample quality for embedding-space DLMs and behaves differently from its use in discrete diffusion. Putting everything together, LangFlow is competitive with top discrete DLMs on generative quality, reaching a generative perplexity of 92.3 on LM1B and 36.5 on OpenWebText. It also outperforms discrete counterparts on reasoning tasks like Sudoku. As the first continuous DLM shown to rival discrete diffusion in generative quality, LangFlow provides clear evidence that continuous diffusion is a promising paradigm for language modeling. Code: https://github.com/nealchen2003/LangFlow.
Sources
- Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models
- Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets
- One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling
- Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioning
- $\alpha$-Flow: A Unified Framework for Continuous-State Discrete Flow Matching Models
- SDAR: A Synergistic Diffusion-AutoRegression Paradigm for Scalable Sequence Generation
- Continuous diffusion for categorical data
- DiffuSeq: Sequence to Sequence Text Generation with Diffusion Models
- Scaling Diffusion Language Models via Adaptation from Autoregressive Models
- Exponential Family Variational Flow Matching for Tabular Data Generation
- Continuous Diffusion Model for Language Modeling
- Flow Map Language Models: One-step Language Modeling via Continuous Denoising
- Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution
- Large Language Diffusion Models
- Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data
- Diffuse Everything: Multimodal Diffusion Models on Arbitrary State Spaces
- The Diffusion Duality
- Esoteric Language Models: A Family of Any-Order Diffusion LLMs
- Simple Guidance Mechanisms for Discrete Diffusion Models
- CoDAR: Continuous Diffusion Language Models are More Powerful Than You Think
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data