How Small Can We Go? Calibrating Activation-Cache Compression for SAE Training
cs.LG, cs.CL
Submitted: 2026-11-05
Updated: 2026-11-05
Code: https://github.com/decoderesearch/SAELens
Project page: http://skylion007.github.io/OpenWebTextCorpus
Terminology
Sources
- Llama Scope: Extracting Millions of Features from Llama-3.1-8B with Sparse Autoencoders
- Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2
- Perplexity Can Miss SAE Feature Damage Under Quantization
- Sparse Autoencoders Trained on the Same Data Learn Different Features
- Unstable Features, Reproducible Subspaces: Understanding Seed Dependence in Sparse Autoencoders
- Sparse but Wrong: Incorrect L0 Leads to Incorrect Features in Sparse Autoencoders
- Interpretability as Compression: Reconsidering SAE Explanations of Neural Activations with MDL-SAEs
- Nondeterminism and Instability in Neural Network Optimization
- The Proxy Presumption: From Semantic Embeddings to Valid Social Measures
- Validity Threats for Foundation Model Research
- LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale
- SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
- KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache
- Exact Recovery of Sparsely-Used Dictionaries
- New Algorithms for Learning Incoherent and Overcomplete Dictionaries
- Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling
- The Pile: An 800GB Dataset of Diverse Text for Language Modeling
- Character-level Convolutional Networks for Text Classification
- Scaling and evaluating sparse autoencoders
- Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoders
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis