Sequential Correlations Change In-Context Learning: Effective Context Length and Architectural Mismatch
Mary Letey, Yue M. Lu, Cengiz Pehlevan, Jacob Zavatone-Veth
stat.ML, cs.LG, stat.CO
Submitted: 2026-07-04
Code: https://github.com/Pehlevan-Group/sequential-correlations-in-context-regression
License: http://creativecommons.org/licenses/by/4.0/
The gist: Modern sequence models have a striking capacity for in-context learning (ICL); they can perform new tasks based only on examples given in the prompt.
Terminology
Abstract
Modern sequence models have a striking capacity for in-context learning (ICL); they can perform new tasks based only on examples given in the prompt. Understanding how this ability emerges requires theory that captures important properties of natural data. Linear regression has served as a useful sandbox for ICL theory, but existing work has largely focused on prompts with independent examples. In this work, we extend this setting to sequentially correlated data, a basic feature of real sequences. We present a solvable model based on linear attention and test our predictions on realistic transformer architectures. We identify two distinct effects: First, when the query token is independent of the context, within-context correlations induce an effective context length: correlated prompts behave like shorter i.i.d. prompts. Second, when the query is also correlated with its context, test error is reduced, particularly for softmax attention when compared to linear attention. These results suggest that correlated prompts alter not only the effective sample size of in-context learning, but also which attention architectures are best matched to the task.
Sources
- GPT-4 Technical Report
- In-Context Language Learning: Architectures and Algorithms
- Multi-layer Cross-attention is Provably Optimal for Multi-modal In-context Learning
- Theory of Scaling Laws for In-Context Regression: Depth, Width, Context and Time
- Sparks of Artificial General Intelligence: Early experiments with GPT-4
- PaLM: Scaling Language Modeling with Pathways
- In-Context Learning of Linear Dynamical Systems with Transformers: Approximation Bounds and Depth-Separation
- In-Context Operator Learning on the Space of Probability Measures
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- A Survey on In-context Learning
- The Evolution of Statistical Induction Heads: In-Context Learning Markov Chains
- Transformers Learn to Achieve Second-Order Convergence Rates for In-Context Linear Regression
- When can in-context learning generalize out of task distribution?
- Test-Time Training Provably Improves Transformers as In-context Learners
- What Do Language Models Learn in Context? The Structured Task Hypothesis
- Probing In-Context Learning: Impact of Task Complexity and Model Architecture on Generalization and Efficiency
- One Step of Gradient Descent is Provably the Optimal In-Context Learner with One Layer of Linear Self-Attention
- Pretrained transformer efficiently learns low-dimensional target functions in-context
- Competition Dynamics Shape Algorithmic Phases of In-Context Learning
- (How) Can Transformers Predict Pseudo-Random Numbers?
Related papers
- Behavior of prediction performance metrics with rare events
- Optimal Estimation of Generic Dynamics by Path-Dependent Neural Jump ODEs
- A Posterior-Dynamics Framework for Imaging Inverse Problems with Pretrained Diffusion Priors
- Scalable Posterior Uncertainty for Flexible Density-Based Clustering
- A Transport Geometry of Benign Overfitting
- Efficient quadratic entropy with distance sketches