Where Should Diffusion Enter a Language Model? Geometry-Guided Hidden-State Replacement

arXiv:2605.14368 · cs.CL, cs.AI · Submitted 2026-05-14 · Read on arXiv

cs.CL, cs.AI

Submitted: 2026-05-14

Updated: 2026-09-27

License: http://creativecommons.org/licenses/by/4.0/

The gist: Continuous diffusion language models require choosing a representation space in which denoising operates, yet it remains unclear which representations are most compatible with diffusion.

Terminology

Abstract

Continuous diffusion language models require choosing a representation space in which denoising operates, yet it remains unclear which representations are most compatible with diffusion. We study a basic design question: where inside a pretrained language model should continuous diffusion operate? We formulate this as a hidden-state interface-selection problem and hypothesize that diffusion-friendly interfaces can be identified from representation geometry. We operationalize this hypothesis using three training-free geometric proxies: local compactness, global stiffness, and effective rank. Across two 8B-scale backbones, the resulting geometry score predicts fixed-budget diffusion bridgeability beyond the dominant effect of layer depth. We then instantiate DiHAL, a Locate-and-Replace framework that replaces the transformer prefix below a selected interface with conditional diffusion while retaining the pretrained suffix and LM head. Under full training, geometry-selected interfaces remain close to validation-loss oracles and outperform worst-layer controls, while the diffusion bridge outperforms a parameter-matched deterministic replacement under matched diagnostic conditions. These results suggest that the representation space in which diffusion operates should itself be treated as a first-class design variable for continuous diffusion in language models.

Sources

Related papers