In-context learning of solution to simple linear regression task using transformer with masked linear self-attention
cs.LG
Submitted: 2026-11-02
Updated: 2026-11-02
Terminology
Sources
- Transformers learn to implement preconditioned gradient descent for in-context learning
- What learning algorithm is in-context learning? Investigations with linear models
- Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selection
- Language Models are Few-Shot Learners
- Adam: A Method for Stochastic Optimization
- What Can Transformers Learn In-Context? A Case Study of Simple Function Classes
- In-context learning of closed form solution to simple linear regression task using transformer with linear self-attention
- Deep Residual Learning for Image Recognition
- One Step of Gradient Descent is Provably the Optimal In-Context Learner with One Layer of Linear Self-Attention
- Transformers learn in-context by gradient descent
- Attention Is All You Need
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis