On the Design Space of Continuous Diffusion Language Models

arXiv:2610.13834 · cs.LG · Submitted 2026-10-09 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "On the Design Space of Continuous Diffusion Language Models".

Jane: The gist: Self-conditioning increases the density of highly likely inference regions during training, diminishing the train–test mismatch,

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So, looking at the full picture of "On the Design Space of Continuous Diffusion Language Models," it seems Oscar Davis and his team are pointing toward a cleaner way to design these models.

Jane: They focus on self-conditioning and embedding geometry as the two main levers, showing that by understanding those things, we can find designs that are both cheaper and more effective for language models.

Meng: I think what this means for us in the production space is that if we can get a model to leverage these geometric insights, we might be able to achieve better performance without massively increasing the training compute budget.

Lu: The paper shows concrete ways, like DoubleCat and SemiCat, that allow us to recover performance gains by introducing mechanisms that mimic self-conditioning or by using better embedded spaces.

Lalam: Ultimately, this research suggests that for continuous diffusion language models, we have a clearer picture of the design space—we know where the levers are and how they interact.

Jane: It’s about taking these ideas about conditioning and geometry and turning them into actual model structures that perform better on benchmarks like TinyGSM.

Tom: So, "On the Design Space of Continuous Diffusion Language Models" is showing us a path to making continuous diffusion models more predictable during training, which leads to better performance at scale.

Conclusion: Tom: So we've been looking at how these continuous diffusion language models can actually be designed, focusing on self-conditioning and embedding geometry.

Jane: Right, so what did this paper actually find? What's the main point they are making about that design space?

Lu: They’re showing that self-conditioning makes the model better at figuring out the right state during training by exposing it to inference states. It fixes a mismatch between how it trains and how it runs.

Meng: So, if we can get that alignment right, does it actually make deploying these models cheaper or more stable?

Lalam: DoubleCat is a simple two-level flow that recovers a lot of the benefit from self-conditioning without needing extra training. It's about imitating that conditioning with just a little bit of extra work.

Jane: That sounds pretty practical, but what about the bigger picture? What does this mean for how we think about continuous diffusion models in general?

Tom: It means we have a clearer map now. We know which design choices, like self-conditioning and geometry, are the most important levers for improving these systems.

Jane: And the results they show with things like SemiCat reaching discrete diffusion levels on benchmarks at scale are pretty compelling for showing what's possible.

Lu: They also look into hybrid methods and schedules that try to get better control over how the diffusion process evolves across the sequence of tokens. It’s a lot of fine-tuning happening there.

Tom: So, it’s not just about one trick; it’s about understanding the whole design space—from how we condition the model to how we shape its underlying mathematical space.

Jane: And that opens up so many new avenues for us to explore if we want to build models that are both powerful and efficient.

Lalam: It really seems like the next step is figuring out how these specific geometric ideas interact with those continuous tools they've developed.

Oscar Davis, *, Daan Roos, *, Alice Bizeul, *, Amitis Shidani, *, Victor Turrisi

Apple University of Oxford University of Amsterdam

cs.LG

Submitted: 2026-10-09

Updated: 2026-10-09

Project page: http://skylion007.github.io/OpenWebTextCorpus

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 60/100

The gist: The gist: Self-conditioning increases the density of highly likely inference regions during training, diminishing the train–test mismatch, and DoubleCat recovers a large part of those benefits by

Key concepts

Self-conditioning: This technique exposes the language model to states similar to inference time during training. This helps correct a mismatch between how the model is trained and how it performs when actually used, leading to better results.
DoubleCat: A simple two-level interpolant that recovers most of the performance boost gained from self-conditioning. It achieves this without needing an additional forward pass through the main model, making it computationally cheap.
Token Identifiability: This refers to a problem where the model struggles to correctly identify specific tokens because of a mismatch in norms between the noise and the token's embedding. Self-conditioning helps resolve this by working on embeddings supported on a convex hull.
One-hot flows: These are diffusion models that operate effectively in an embedded space where noise is defined by the geometry of the embeddings. Training with cross-entropy drives these embeddings toward a specific geometry that matches their performance.

Summary

The gist: Self-conditioning increases the density of highly likely inference regions during training, diminishing the train–test mismatch, and DoubleCat recovers a large part of those benefits by introducing a double-level flow, imitating self-conditioning

How it works

Continuous diffusion language models (CDLMs) aim to bring continuous flow modelling to language, including accelerated sampling and controllable generation The work focuses on two levers, namely self-conditioning and embedding geometry, suggesting cheaper and more effective designs Self-conditioning corrects a train/inference mismatch by exposing the model to inference-like states that are otherwise rare during training This gain can be recovered by a simple two-level interpolant, called DoubleCat, which requires no additional forward pass of the model and which more than triples the accuracy on TinyGSM from 4.42% to 16.00%

Self-conditioning and Token Identifiability

Self-conditioning aids in token identifiability because given the noisy state, the model cannot easily identify the expected endpoint of the trajectory due to a norm mismatch between the noise and the token’s embedding Self-conditioning is supported on the convex hull formed by embeddings, and replacing that signal with a better representation of the state recovers a substantial part of the performance gain at no additional training cost To address token identifiability in DoubleCat, discrete stochastic reveals akin to CANDI are introduced, yielding SemiCat which boosts performance up to 32.90%, reaching the discrete diffusion baseline of about 33.74% without self-conditioning

Embedded Spaces and Geometry

One-hot flows effectively operate on an embedded space with an anisotropic noise defined by the embeddings’ geometry, and cross-entropy drives embeddings towards a geometry whose decoding error rate closely matches the one-hot one Naive embedded training underperforms, as cross-entropy ignores the geometry of the embeddings and spreads tokens apart on the sphere approximately recovering the one-hot geometry in terms of their DER Plaid learns its log-SNR endpoints, and variants all pin them

Hybrid Methods and Schedules

The paper explores various corruption processes, including hybrid methods like CANDI, which continuously evolves each state of the sequence until some randomly sampled unmasking time τl per position Asynchronous schedules are also explored, where local times tl are derived from a global time tg such that tl = F(tg, l), which is monotonic in l and approximately mean-preserving across positions The entropic schedule is proposed to advance linearly in S, aiming for S(γt) = (1 − t) log V rather than linearizing directly the schedule in the probability of argmax Ilt!= argmax Xl1

Results and Scaling

SemiCat reaches 49.94% without any self-conditioning on TinyGSM, and at the 1.7B scale, SemiCat outperforms discrete diffusion models on four out of six likelihood-based benchmarks At scale, SemiCat outperforms discrete diffusion on four out of six of our likelihood tasks and surpasses one-hot flows on unconditional generation The results show that SemiCat outperforms the one-hot flow baseline on every MCQA task and outperforms the discrete diffusion baselines on ARC-E by 8.4 points SemiCat also dominates the Gen. PPL/Entropy Pareto frontier of one-hot flows on OpenWebText at all NFEs

Conclusion

Self-conditioning holds a more identifiable representation of the state, on the convex hull formed by the embeddings, which aids in token identifiability DoubleCat recovers a large part of those benefits by introducing a double-level flow, imitating self-conditioning SemiCat can attain discretizediffusion-level performance on our benchmarks, and at scale, SemiCat outperforms discrete diffusion on four out of six of our likelihood tasks and surpasses one-hot flows on unconditional generation The work unveils important avenues left to explore to understand continuous diffusion for discrete data

--- Page 5 ---

How it works

Self-conditioning corrects a train/inference mismatch by exposing the model to inference-like states that are otherwise rare during training This gain can be recovered by a simple two-level interpolant, called DoubleCat, which requires no additional forward pass of the model and which more than triples the accuracy on TinyGSM from 4.42% to 16.00% Self-conditioning also aids in recognising tokens dominated by large-norm Gaussians—an issue known as token identifiability

Token Identifiability

Self-conditioning helps in token identifiability because the model cannot easily identify the expected endpoint of the trajectory due to the norm mismatch between the noise and the token’s embedding, whereas self-conditioning is supported on the convex hull formed by embeddings By introducing discrete stochastic reveals akin to CANDI, SemiCat further boosts performance up to 32.90%, reaching the discrete diffusion baseline of about 33.74% without self-conditioning

Embedded Spaces

One-hot flows are secretly embedded in an embedded space with an anisotropic noise defined by the embeddings’ geometry, on which we can directly train Cross-entropy training spreads tokens apart, inducing the same DER as a one-hot flow Plaid learns its log-SNR endpoints, and the variants all pin them

Scaling SemiCat

We train a larger SemiCat model following the protocol of Davis et al. (2026); Sahoo et al. (2026) At scale, SemiCat outperforms discrete diffusion on four out of six likelihood tasks and surpasses one-hot flows on unconditional generation

Conclusion

Self-conditioning increases the density of highly likely inference regions during training, diminishing the train–test mismatch DoubleCat recovers a large part of those benefits by introducing a double-level flow, imitating self-conditioning SemiCat can attain discretizediffusion-level performance on our benchmarks, and at scale, SemiCat outperforms discrete diffusion on four out of six of our likelihood tasks and surpasses one-hot flows on unconditional generation The work leaves to future work accelerating these methods with the continuous toolbox, and understanding how that toolbox interacts with our design choices

--- Page 9 ---

How it works

Improvements for AI systems

The system can achieve higher accuracy on TinyGSM by implementing DoubleCat, which more than triples the accuracy on TinyGSM compared to a base one-hot flow, effectively recovering benefits from self-conditioning by introducing a double-level flow.
The system can better identify tokens in noisy states by employing SemiCat, which achieves discrete diffusion-level performance on TinyGSM, surpassing discrete diffusion models on four out of six likelihood evaluations and matching them on a fifth, and further outperforms one-hot flows in unconditional generation quality.
The system can be adapted to embedded spaces where embeddings are trained jointly rather than naively with cross-entropy, as cross-entropy ignores the geometry of the embeddings, and spreads tokens apart on the sphere approximately recovering the one-hot geometry in the DER sense.
The system's performance can be enhanced by incorporating self-conditioning into embedded methods using a cheaper proxy at train time; specifically, replacing true self-conditioning with the Bayesian posterior expectation helps substantially at a lower cost while attaining better results.
The system can be optimized for token identifiability by introducing revealing mechanisms akin to CANDI, leading to SemiCat, which mitigates name errors by allowing the model to cleanly condition on revealed inputs.
For continuous modalities, the system can benefit from methods that address the discrepancy between inference and training trajectories, such as DoubleCat, which correct[s] a train/inference mismatch by expos[ing] the model to inference-like states that are otherwise rare during training.
The system can utilize an entropic schedule for training times, which is proposed to advance linearly in the expected remaining entropy to match the performance of mixture schedules, suggesting that making uniform progress in the DER probability Pe might be the wrong criterion.
The system can employ asynchronous schedules where local times tl are derived from a global time tg, allowing for a distinct time for each position to be revealed, which is hypothesized to improve accuracy over fixed schedules.

Abstract

Continuous diffusion language models (CDLMs) promise to bring the benefits of continuous flow modelling to language, including accelerated sampling and controllable generation. The field has grown rapidly, and the capabilities of CDLMs have begun to rival those of discrete diffusion. Yet, unlike for continuous modalities such as images or videos, our grasp of the levers that matter when training CDLMs remains limited. In this work, we focus on two of these levers, namely self-conditioning and embedding geometry, and shed light on some of the mechanisms behind them, which in turn suggest cheaper and more effective designs. We begin by showing that self-conditioning corrects a train/inference mismatch by exposing the model to inference-like states that are otherwise rare during training. These gains can be recovered without an extra forward pass by a two-level interpolant, coined DoubleCat, which improves TinyGSM accuracy more than threefold. Self-conditioning also aids in recognising tokens dominated by large-norm Gaussians an issue known as token identifiability. To address it directly, we introduce discrete reveals to DoubleCat, yielding SemiCat, which achieves discrete diffusion-level performance on TinyGSM, reaching 49.94% with standard training hyperparameters and a tuned sampler. At the 1.7 B scale, SemiCat outperforms discrete diffusion on four of six likelihood-based evaluations, and surpasses one-hot flows on unconditional generation. On embedded spaces, we find that cross-entropy drives embeddings towards a geometry whose decoding error rate closely matches the one-hot one. We briefly explore a range of further options, such as hybrid corruption processes and noise schedules, including asynchronous ones. We believe this exploration can help the community better understand and advance the design space of CDLMs.

Listen

Transcript

Key concepts

Summary

Improvements for AI systems

Abstract

Related papers