A Mechanistic Study of Algorithmic Generalization in Diffusion Models on Modular Arithmetic
Diffusion models trained for flow-matching objectives can perform modular addition by using periodic representations composed via trigonometric identities. A key finding is a phase transition during sampling: the model switches from algorithmic reasoning to visual denoising at a specific timestep, marked by a sudden spike in Fractional Variance Explained entropy.