Advancing Model Refinement: Muon-Optimized Distillation and Quantization for LLM Deployment
cs.LG, cs.AI
Submitted: 2026-01-14
Updated: 2026-11-08
Code: https://github.com/tatsu-lab/stanford_alpaca
Terminology
Sources
- Optuna: A Next-generation Hyperparameter Optimization Framework
- Towards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMs
- Muon-Accelerated Attention Distillation for Real-Time Edge Synthesis via Optimized Latent Diffusion
- Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
- Subliminal Learning: Language models transmit behavioral traits via hidden signals in data
- QLoRA: Efficient Finetuning of Quantized LLMs
- Fine-Tuning and Deploying Large Language Models Over Edges: Issues and Approaches
- SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot
- GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
- Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
- Distilling the Knowledge in a Neural Network
- Language model compression with weighted low-rank factorization
- LoRA: Low-Rank Adaptation of Large Language Models
- Adam: A Method for Stochastic Optimization
- AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
- Muon is Scalable for LLM Training
- Outlier-Safe Pre-Training for Robust 4-Bit Quantization of Large Language Models
- WinoGrande: An Adversarial Winograd Schema Challenge at Scale
- On Accelerating Edge AI: Optimizing Resource-Constrained Environments
- SocialIQA: Commonsense Reasoning about Social Interactions
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis