Attention Sinks in Diffusion Transformers: A Causal Analysis
Researchers analyzed attention sinks—tokens receiving disproportionate attention—in text-to-image diffusion models like Stable Diffusion 3 and SDXL. They dynamically identified these sinks per timestep and suppressed them using training-free interventions. Findings show that removing these sinks does not harm semantic alignment but causes large, sink-specific perceptual shifts, suggesting they carry trajectory information rather than being strictly necessary for basic image quality.