UpLoop: Training Looped Transformers via Upcycling at No Extra Pretraining Compute
cs.LG, cs.CL
Submitted: 2026-10-30
Updated: 2026-10-30
Terminology
Sources
- PIQA: Reasoning about Physical Commonsense in Natural Language
- Net2Net: Accelerating Learning via Knowledge Transfer
- BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions
- Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
- One LR Doesn't Fit All: Heavy-Tail Guided Layerwise Learning Rates for LLMs
- Training Compute-Optimal Large Language Models
- Do Depth-Grown Models Overcome the Curse of Depth? An In-Depth Analysis
- From Growing to Looping: A Unified View of Iterative Computation in LLMs
- Landscape-Aware Growing: The Power of a Little LAG
- Asymmetric Looped Transformers: Fixed Global Representations for an Iterative Decoder
- Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering
- Parcae: Scaling Laws For Stable Looped Language Models
- SocialIQA: Commonsense Reasoning about Social Interactions
- How Much Is One Recurrence Worth? Iso-Depth Scaling Laws for Looped Language Models
- 2 OLMo 2 Furious
- SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers
- HellaSwag: Can a Machine Really Finish Your Sentence?
- Scaling Latent Reasoning via Looped Language Models
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis