Sticky Jump Diffusions: A Unifying View of Masked, Continuous, and Hybrid Diffusion
Pascal Jutras-Dubé, Patrick Pynadath, Jeremy Lu, Yuan Gao, Ruqi Zhang
cs.LG, stat.ML
Submitted: 2026-07-12
Code: https://github.com/PascalJD/sticky-jump-diffusions
License: http://creativecommons.org/licenses/by/4.0/
The gist: We introduce Sticky Jump Diffusions (SJDs), continuous-time Markov processes on R d whose discrete anchors are token embeddings.
Terminology
Abstract
We introduce Sticky Jump Diffusions (SJDs), continuous-time Markov processes on R d whose discrete anchors are token embeddings. In forward time, anchors release their mass at a hazard rate and the released mass diffuses in the continuous ambient space; time reversal couples a score-driven SDE with a sticky jump kernel whose rate and destination are fixed by flux balance with the forward law. We estimate the score and the per-anchor reverse hazards from a single denoising classifier via Denoising Hazard Matching, the hazard analogue of denoising score matching, with simulation-free cross-entropy training. SJD recovers masked diffusion, continuous diffusion, and hybrid diffusion as limits. Its reversal explains features that each family treats as given: the mask of masked diffusion carries no evidence about the source token because the unsticking kernel of every anchor collapses to the same absorbing point; the terminal projection of continuous diffusion is required due to the absence of atoms in its forward marginal, without which flux balance yields no reverse jumps; and the update rules of hybrid diffusion (commit rate, destination, and drift) all follow from flux balance rather than from separate design. Beyond these limits, the unsticking kernel becomes a design space: a cross-position blending corrupts each position toward a blend of its neighbors' clean values or embeddings, turning dependency structure such as spatial locality or a constraint graph into an inductive bias of the corruption itself, and improves over the identity-kernel hybrid on CIFAR-10, Text8, and Sudoku.
Sources
- LangFlow: Continuous Diffusion Rivals Discrete in Language Modeling
- Continuous diffusion for categorical data
- ELF: Embedded Language Flows
- Flow Map Language Models: One-step Language Modeling via Continuous Denoising
- Continuous Diffusion Scales Competitively with Discrete Diffusion for Language
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks