RAPID: Increasing Efficiency of Internal Distillation via Randomized Methods
cs.LG
Submitted: 2026-11-02
Updated: 2026-11-05
Terminology
Sources
- SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model
- Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
- Toy Models of Superposition
- GLM-5: from Vibe Coding to Agentic Engineering
- The False Promise of Imitating Proprietary LLMs
- OpenThoughts: Data Recipes for Reasoning Models
- Language Models Represent Space and Time
- Distilling the Knowledge in a Neural Network
- TinyBERT: Distilling BERT for Natural Language Understanding
- Mitigating Negative Transfer in Multi-Task Learning with Exponential Moving Average Loss Weighting Strategies
- Dynamic Knowledge Distillation for Pre-trained Language Models
- Randomized algorithms for matrices and data
- Randomized Numerical Linear Algebra: Foundations & Algorithms
- The Linear Representation Hypothesis and the Geometry of Large Language Models
- Qwen2.5 Technical Report
- GPQA: A Graduate-Level Google-Proof Q&A Benchmark
- DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
- Patient Knowledge Distillation for BERT Model Compression
- Gemma 2: Improving Open Language Models at a Practical Size
- Gemma 3 Technical Report
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis