ELF: Embedded Language Flows
cs.CL, cs.AI, cs.LG
Submitted: 2026-05-11
Updated: 2026-11-05
Comments: NeurIPS 2026. arXiv v3: add GSM8k results
Code: https://github.com/lillian039/ELF
License: http://creativecommons.org/licenses/by/4.0/
The gist: Diffusion and flow-based models have become the de facto approaches for generating continuous data, e.g., in domains such as images and videos.
Terminology
Abstract
Diffusion and flow-based models have become the de facto approaches for generating continuous data, e.g., in domains such as images and videos. Their success has attracted growing interest in applying them to language modeling. Unlike their image-domain counterparts, today's leading diffusion language models (DLMs) primarily operate over discrete tokens. In this paper, we show that continuous DLMs can be made effective with minimal adaptation to the discrete domain. We propose Embedded Language Flows (ELF), a class of diffusion models in continuous embedding space based on continuous-time Flow Matching. Unlike existing DLMs, ELF predominantly stays within the continuous embedding space until the final time step, where it maps to discrete tokens using a shared-weight network. This formulation makes it straightforward to adapt established techniques from image-domain diffusion models, e.g., classifier-free guidance (CFG). Experiments show that ELF substantially outperforms leading discrete and continuous DLMs, achieving better generation quality with fewer sampling steps. These results suggest that ELF offers a promising path toward effective continuous DLMs.
Sources
- CoBit: Language Modeling with Bitstream Diffusion
- FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Space
- LangFlow: Continuous Diffusion Rivals Discrete in Language Modeling
- Continuous diffusion for categorical data
- Improved Mean Flows: On the Challenges of Fastforward Generative Models
- Continuous Latent Diffusion Language Model
- Flow Map Language Models: One-step Language Modeling via Continuous Denoising
- Omni-Diffusion: Unified Multimodal Understanding and Generation with Masked Discrete Diffusion
- Back to Basics: Let Denoising Generative Models Denoise
- A Survey on Diffusion Language Models
- How to Train Your Latent Diffusion Language Model Jointly With the Latent Space
- Discrete Flow Maps
- CANDI: Hybrid Discrete-Continuous Diffusion Models
- Categorical Flow Maps
- Scaling Beyond Masked Diffusion Language Models
- Why Gaussian Diffusion Models Fail on Discrete Data and How to Prevent It?
- GLU Variants Improve Transformer
- CoDAR: Continuous Diffusion Language Models are More Powerful Than You Think
- Seed Diffusion: A Large-Scale Diffusion Language Model with High-Speed Inference
- Self-conditioned Embedding Diffusion for Text Generation
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data