How to Train Your Latent Diffusion Language Model Jointly With the Latent Space
cs.CL
Submitted: 2026-05-08
Updated: 2026-11-08
Project page: http://skylion007.github.io/OpenWebTextCorpus
License: http://creativecommons.org/licenses/by/4.0/
The gist: Latent diffusion models offer an attractive alternative to discrete diffusion for non-autoregressive text generation by operating on continuous text representations and denoising entire sequences in…
Terminology
Abstract
Latent diffusion models offer an attractive alternative to discrete diffusion for non-autoregressive text generation by operating on continuous text representations and denoising entire sequences in parallel. The major challenge in latent diffusion modeling is constructing a suitable latent space. In this work, we present the Latent Diffusion Language Model (LDLM), in which the latent encoder, diffusion model, and decoder are trained jointly. LDLM builds its latent space by reshaping the representations of a pre-trained language model with a trainable encoder, yielding latents that are easy to both denoise and decode into tokens. We show that naive joint training produces a low-quality diffusion model, and propose a simple training recipe consisting of an MSE decoder loss, diffusion-to-encoder warmup, adaptive timestep sampling, and decoder-input noise. Ablations show that each component substantially impacts generation performance. On OpenWebText and LM1B, LDLM achieves better generation performance than existing discrete and continuous diffusion language models while being 2 - 13 times faster, indicating that jointly learning the latent space is a key step toward making latent diffusion competitive for text generation.
Sources
- One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling
- AlignTok: Aligning Visual Foundation Encoders to Tokenizers for Diffusion Models
- The Diffusion Duality, Chapter II: $\Psi$-Samplers
- Boosting Generative Image Modeling via Joint Image-Feature Synthesis
- Flow Map Language Models: One-step Language Modeling via Continuous Denoising
- Guided Star-Shaped Masked Diffusion
- Discrete Flow Maps
- Categorical Flow Maps
- Score-Based Generative Modeling through Stochastic Differential Equations
- Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding
- Beyond Autoregression: Discrete Diffusion for Complex Reasoning and Planning
- SeqDiffuSeq: Text Diffusion with Encoder-Decoder Transformers
- Diffusion Transformers with Representation Autoencoders
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data