Knowledge Offloading: Decomposing LLMs into Sparse Backbones and Memory Modules
cs.LG
Submitted: 2026-05-27
Updated: 2026-11-05
Terminology
Sources
- SliceGPT: Compress Large Language Models by Deleting Rows and Columns
- Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
- Training Verifiers to Solve Math Word Problems
- Cartridges: Lightweight and general-purpose long context representations via self-study
- Measuring Massive Multitask Language Understanding
- LoRA: Low-Rank Adaptation of Large Language Models
- FASP: Fast and Accurate Structured Pruning of Large Language Models
- Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Models
- Prefix-Tuning: Optimizing Continuous Prompts for Generation
- Learning Sparse Neural Networks through $L_0$ Regularization
- LLM-Pruner: On the Structural Pruning of Large Language Models
- Pretraining with hierarchical memories: separating long-tail and common knowledge
- Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them
- LLaMA: Open and Efficient Foundation Language Models
- Qwen2 Technical Report
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis