Massive Activations Confound Sparse Autoencoders
cs.LG, cs.AI
Submitted: 2026-11-10
Updated: 2026-11-10
Code: https://github.com/nathandelisle/massive-saes
Project page: https://skylion007.github.io/OpenWebTextCorpus
Terminology
Sources
- Do Sparse Autoencoders Capture Concept Manifolds?
- BatchTopK Sparse Autoencoders
- Are Sparse Autoencoder Benchmarks Reliable?
- Sparse Autoencoders Find Highly Interpretable Features in Language Models
- Qwen-Scope: Turning Sparse Features into Development Tools for Large Language Models
- Unstable Features, Reproducible Subspaces: Understanding Seed Dependence in Sparse Autoencoders
- Llama Scope: Extracting Millions of Features from Llama-3.1-8B with Sparse Autoencoders
- Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2
- DINOv2: Learning Robust Visual Features without Supervision
- Sparse Autoencoders Trained on the Same Data Learn Different Features
- Improving Dictionary Learning with Gated Sparse Autoencoders
- Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoders
- Data Whitening Improves Sparse Autoencoder Learning
- The Spike, the Sparse and the Sink: Anatomy of Massive Activations and Attention Sinks
- The Super Weight in Large Language Models
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis