Distinct mechanisms underlying in-context learning in transformers
cs.LG, cond-mat.dis-nn, cond-mat.stat-mech
Submitted: 2026-04-14
Updated: 2026-11-17
Comments: 56 pages, 28 figures
Code: https://github.com/cole-gibson/context-adaptive-transformer
License: http://creativecommons.org/licenses/by/4.0/
The gist: Modern distributed networks, notably transformers, acquire a remarkable ability (termed `in-context learning') to adapt their computation to input statistics…
Terminology
Abstract
Modern distributed networks, notably transformers, acquire a remarkable ability (termed `in-context learning') to adapt their computation to input statistics, such that a fixed network can be applied to data from a broad range of systems. Here, we provide a complete mechanistic characterization of this behavior in transformers trained on a finite set S of discrete Markov chains. The transformer displays four algorithmic phases, characterized by whether the network memorizes and generalizes, and whether it uses 1-point or 2-point statistics. We show that the four phases are implemented by multi-layer subcircuits that exemplify two qualitatively distinct mechanisms for implementing context-adaptive computations. Minimal models isolate the key features of both motifs. Memorization and generalization phases are delineated by two boundaries that depend on data diversity, K = S. The first (K 1) is set by a kinetic competition between subcircuits and the second (K 2) is set by a representational bottleneck. A symmetry-constrained theory of a transformer's training dynamics explains the sharp transition from 1-point to 2-point generalization and identifies key features of the loss landscape that allow the network to generalize. Put together, we show that transformers develop distinct subcircuits to implement in-context learning and identify conditions that favor certain mechanisms over others.
Sources
- Attention Is All You Need
- In-context Learning and Induction Heads
- General-Purpose In-Context Learning by Meta-Learning Transformers
- Uncovering mesa-optimization algorithms in Transformers
- Birth of a Transformer: A Memory Viewpoint
- Meta-learning of Sequential Strategies
- Trained Transformers Learn Linear Models In-Context
- TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second
- Zero-shot forecasting of chaotic systems
- Transformers for dynamical systems learn transfer operators in-context
- In-Context Imitation Learning via Next-Token Prediction
- In-context Reinforcement Learning with Algorithm Distillation
- A Survey of In-Context Reinforcement Learning
- On the Opportunities and Risks of Foundation Models
- How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression?
- Differential learning kinetics govern the transition from memorization to generalization during in-context learning
- Competition Dynamics Shape Algorithmic Phases of In-Context Learning
- In-Context Learning Strategies Emerge Rationally
- Asymptotic theory of in-context learning by linear attention
- When can in-context learning generalize out of task distribution?
Related papers
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis
- (Mis)generalization of Helpful-Only Fine-tuning