An Analytical Model of Compute-limited Multistage Training Pipelines
cs.LG
Submitted: 2026-11-09
Updated: 2026-11-09
Code: https://github.com/nishp99/multistage_pipeline
Terminology
Sources
- GPT-4 Technical Report
- Front-Loading Reasoning: The Synergy between Pretraining and Post-Training Data
- Scaling and renormalization in high-dimensional regression
- Representation Learning: A Review and New Perspectives
- On the Opportunities and Risks of Foundation Models
- Training Verifiers to Solve Math Word Problems
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- 2 OLMo 2 Furious
- Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey
- Training Compute-Optimal Large Language Models
- OpenAI o1 System Card
- Scaling Laws for Neural Language Models
- An analytic theory of generalization dynamics and transfer learning in deep linear networks
- Why Do Animals Need Shaping? A Theory of Task Composition and Curriculum Learning
- Maslow's Hammer for Catastrophic Forgetting: Node Re-Use vs Node Activation
- DeepSeek-V3 Technical Report
- ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models
- AceReason-Nemotron 1.1: Advancing Math and Code Reasoning through SFT and RL Synergy
- Layer-Wise Evolution of Representations in Fine-Tuned Transformers: Insights from Sparse AutoEncoders
- Spurious Rewards: Rethinking Training Signals in RLVR
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis