Transformers Provably Implement In-Context Reinforcement Learning with Policy Improvement
stat.ML, cs.AI, cs.LG
Submitted: 2026-05-07
Updated: 2026-11-07
Comments: 38 pages, 7 figures, 6 tables
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Terminology
Sources
- Language Models are Few-Shot Learners
- What Can Transformers Learn In-Context? A Case Study of Simple Function Classes
- A Survey on In-context Learning
- GPT-4 Technical Report
- Transformers learn in-context by gradient descent
- One Step of Gradient Descent is Provably the Optimal In-Context Learner with One Layer of Linear Self-Attention
- Trained Transformers Learn Linear Models In-Context
- Transformers as Decision Makers: Provable In-Context Reinforcement Learning via Supervised Pretraining
- Large Language Models can Implement Policy Iteration
- In-Context Convergence of Transformers
- Training Dynamics of Multi-Head Softmax Attention for In-Context Learning: Emergence, Convergence, and Optimality
- Scan and Snap: Understanding Training Dynamics and Token Composition in 1-layer Transformer
Related papers
- Uniform-in-Time Weak Propagation-of-Chaos in Shallow Neural Networks
- Diffusion Models Observe Only Gradients: A Geometric Perspective on Score Matching Errors
- Quantifying Error Propagation and Model Collapse in Diffusion Models
- Behavior of prediction performance metrics with rare events
- Statistical Inference and Learning for Shapley Additive Explanations (SHAP)
- Control Variate Design for Score-Function Gradient Estimators in Importance-Weighted Variational Inference