Kinetic theory for Transformers and the lost-in-the-middle phenomenon
The study models Transformer causal self-attention as a non-exchangeable interacting particle system to rigorously explain empirical 'lost-in-the-middle' phenomena in token retrieval. Using advanced mathematical tools like cumulant expansions and Glauber calculus, the authors prove a mean-field limit result and show that for random prompts, the retrieval accuracy profile is U-shaped with a unique minimum in the middle.