CoBit: Language Modeling with Bitstream Diffusion
cs.CL
Submitted: 2026-05-07
Updated: 2026-11-06
Code: https://github.com/GBATZOLIS/BitstreamDiffusion
License: http://creativecommons.org/licenses/by/4.0/
The gist: We introduce CoBit, a continuous diffusion language model that generates text by jointly denoising fixed-width binary token codes.
Terminology
Abstract
We introduce CoBit, a continuous diffusion language model that generates text by jointly denoising fixed-width binary token codes. Grouping each code's bits into a single Transformer patch preserves sequence length while replacing vocabulary-wide predictions with 2 V bit logits. A matched-filter residual parameterization supplies the analytic isolated-bit posterior, leaving the network to learn contextual corrections. Stochastic sampling on an entropy-rate grid provides the largest empirical gain; the grid allocates both solver resolution and effective Langevin correction to noise levels where information is resolved. At 256 denoiser evaluations, CoBit-S (130M parameters) achieves generative perplexity 59.76 on LM1B and 27.06 on OpenWebText, at unigram entropies 4.31 and 5.26, respectively. Scaling to 462M parameters improves OpenWebText generative perplexity to 19.48 at entropy 5.40. Beyond open-ended text generation, CoBit achieves 65.9% exact-match accuracy on hard Sudoku. On GSM8K, score-temperature scaling with an untempered final integration step raises CoBit-S accuracy from 28.1% to 35.3% without additional denoiser evaluations, placing it within the reported range of temperature-tuned discrete diffusion baselines. Matched-trunk profiling also shows lower peak memory and higher throughput than a dense token-output baseline.
Sources
- LangFlow: Continuous Diffusion Rivals Discrete in Language Modeling
- Continuous diffusion for categorical data
- ELF: Embedded Language Flows
- Continuous Diffusion Model for Language Modeling
- Flow Map Language Models: One-step Language Modeling via Continuous Denoising
- CANDI: Hybrid Discrete-Continuous Diffusion Models
- Generative Frontiers: Why Evaluation Matters for Diffusion Language Models
- Noise Scheduling as Information-Guided Allocation in Diffusion Training
- Categorical Flow Maps
- Simple and Effective Masked Diffusion Language Models
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data