Understanding AI from the Inside: The State of Mechanistic Interpretability for AI Safety
cs.CL, cs.AI, cs.LG
Submitted: 2026-11-05
Updated: 2026-11-09
Project page: https://science-of-training.github.io/assets/proposal.pdf
Terminology
Sources
- Towards Combinatorial Interpretability of Neural Computation
- Evaluating Explanations: An Explanatory Virtues Framework for Mechanistic Interpretability -- The Strange Science Part I.ii
- ProofNet: Autoformalizing and Formally Proving Undergraduate-Level Mathematics
- Layer Normalization
- Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
- Constitutional AI: Harmlessness from AI Feedback
- Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation
- Multimodal Model Diffing for Feature Discovery and Control
- Eliciting Latent Predictions from Transformers with the Tuned Lens
- Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
- International AI Safety Report
- Interpretability in Parameter Space: Minimizing Mechanistic Description Length with Attribution-based Parameter Decomposition
- Pretraining Curricula Enable Selective Fine-tuning
- Stochastic Parameter Decomposition
- Evaluating Open-Source Sparse Autoencoders on Disentangling Factual Knowledge in GPT-2 Small
- Evaluating Large Language Models Trained on Code
- Persona Vectors: Monitoring and Controlling Character Traits in Language Models
- Reasoning Models Don't Always Say What They Think
- Training Verifiers to Solve Math Word Problems
- Eliciting Secret Knowledge from Language Models
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data