One for All: A Non-Linear Transformer can Enable Cross-Domain Generalization for In-Context Reinforcement Learning
cs.LG, cs.AI
Submitted: 2026-05-10
Updated: 2026-11-07
Terminology
Sources
- Transformers learn to implement preconditioned gradient descent for in-context learning
- What learning algorithm is in-context learning? Investigations with linear models
- Transformers Implement Functional Gradient Descent to Learn Non-Linear Functions In Context
- Rethinking Attention with Performers
- Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizers
- XCiT: Cross-Covariance Image Transformers
- A Theory of Emergent In-Context Learning as Implicit Structure Induction
- In-Context Decision Transformer: Reinforcement Learning via Hierarchical Chain-of-Thought
- Supervised Pretraining Can Learn In-Context Reinforcement Learning
- Emergent Agentic Transformer from Chain of Hindsight Experience
- One Step of Gradient Descent is Provably the Optimal In-Context Learner with One Layer of Linear Self-Attention
- Improving Transformers with Probabilistic Attention Keys
- Cross-Episodic Curriculum for Transformer Agents
- Transformers learn in-context by gradient descent
- Transformers Can Learn Temporal Difference Methods for In-Context Reinforcement Learning
- An Explanation of In-context Learning as Implicit Bayesian Inference
- Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning
- Trained Transformers Learn Linear Models In-Context
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis