Task Vector Geometry Underlies Dual Modes of Task Inference in Transformers
cs.LG, cs.CL, stat.ML
Submitted: 2026-05-05
Updated: 2026-11-07
Code: https://github.com/ezyhdxm/mini-ICL
Terminology
Sources
- Dynamics of Transient Structure in In-Context Linear Regression Transformers
- When Models Manipulate Manifolds: The Geometry of a Counting Task
- In-context Learning and Induction Heads
- Qwen2.5 Technical Report
- Transformers learn factored representations
- Uncovering hidden geometry in Transformers via disentangling position and context
- Larger language models do in-context learning differently
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis