Perturbing Inputs Improves Pretraining Distillation
cs.CL, cs.AI
Submitted: 2026-11-08
Updated: 2026-11-08
Terminology
Sources
- On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes
- SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model
- What Is a Repeated Token Worth? The Scaling Geometry of Multi-Epoch Pretraining
- Demystifying Training-Time Augmentation for Data-Constrained Language Model Pretraining
- Gemma 2: Improving Open Language Models at a Practical Size
- Distilling the Knowledge in a Neural Network
- MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies
- NEFTune: Noisy Embeddings Improve Instruction Finetuning
- MixKD: Towards Efficient Distillation of Large-scale Language Models
- Muse Glimmer 30B: An On-Device Agentic Model
- Diffusion Language Models are Super Data Learners
- The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale
- Efficient Pre-Training with Token Superposition
- Pre-training Distillation for Large Language Models: A Design Space Exploration
- Diffusion Beats Autoregressive in Data-Constrained Settings
- Qwen2.5 Technical Report
- Knowledge Transfer with Jacobian Matching
- Data-Constrained Language Model Pretraining: Improved Regularization and Scaling Laws
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data