Distinct mechanisms underlying in-context learning in transformers

arXiv:2604.12151 · cs.LG, cond-mat.dis-nn, cond-mat.stat-mech · Submitted 2026-04-14 · Read on arXiv

cs.LG, cond-mat.dis-nn, cond-mat.stat-mech

Submitted: 2026-04-14

Updated: 2026-11-17

Comments: 56 pages, 28 figures

Code: https://github.com/cole-gibson/context-adaptive-transformer

License: http://creativecommons.org/licenses/by/4.0/

The gist: Modern distributed networks, notably transformers, acquire a remarkable ability (termed `in-context learning') to adapt their computation to input statistics…

Terminology

Abstract

Modern distributed networks, notably transformers, acquire a remarkable ability (termed `in-context learning') to adapt their computation to input statistics, such that a fixed network can be applied to data from a broad range of systems. Here, we provide a complete mechanistic characterization of this behavior in transformers trained on a finite set S of discrete Markov chains. The transformer displays four algorithmic phases, characterized by whether the network memorizes and generalizes, and whether it uses 1-point or 2-point statistics. We show that the four phases are implemented by multi-layer subcircuits that exemplify two qualitatively distinct mechanisms for implementing context-adaptive computations. Minimal models isolate the key features of both motifs. Memorization and generalization phases are delineated by two boundaries that depend on data diversity, K = S. The first (K 1) is set by a kinetic competition between subcircuits and the second (K 2) is set by a representational bottleneck. A symmetry-constrained theory of a transformer's training dynamics explains the sharp transition from 1-point to 2-point generalization and identifies key features of the loss landscape that allow the network to generalize. Put together, we show that transformers develop distinct subcircuits to implement in-context learning and identify conditions that favor certain mechanisms over others.

Sources

Related papers