A Phase Transition between Positional and Semantic Learning in a Solvable Model of Dot-Product Attention
This episode discusses the paper on phase transitions in dot-product attention, which analyzes how transformers learn attention. The hosts explain that models switch between paying attention based on a word's position (positional) and paying attention based on its meaning (semantic). This shift occurs sharply—a 'phase transition'—when the model receives enough training data to exploit semantic content.