Learning to Extrapolate in Depth with Looped Transformers
The study investigates how far a looped transformer, trained at depth K, can be safely extrapolated to greater depths for graph connectivity tasks. It finds that training converges to a single effective parameter balancing graph propagation and structure fitting. This creates a finite extrapolation window; too much looping causes false connections, providing explicit bounds on solvable graph hardness.