When Does Depth Matter For In-Context Learning? Adaptive Inference in Deep Transformers
The theory proposes that transformers can implement adaptive inference using internal 'function vectors' as compressed state variables. These vectors allow the model to choose which statistics to measure next at different layers, enabling a richer class of in-context learning algorithms than standard methods. Depth provides a significant advantage when dealing with complex data structures like tree priors.