The Model Organism Lottery: Model Organism Interpretability Strongly Depends on Training Methodology
cs.LG
Submitted: 2026-07-01
Updated: 2026-10-29
Comments: 11 pages, 6 figures, references and appendices
Code: https://github.com/model-organisms-for-real/model-organism-lottery
License: http://creativecommons.org/licenses/by/4.0/
The gist: Model organisms (MOs) - language models trained to exhibit undesired or unnatural behaviours - are frequently used as testbeds for evaluating white-box interpretability techniques.
Terminology
Abstract
Model organisms (MOs) - language models trained to exhibit undesired or unnatural behaviours - are frequently used as testbeds for evaluating white-box interpretability techniques. Current MOs are typically constructed via post-hoc supervised fine-tuning (SFT). Prior research has shown that interpretability methods can easily identify hidden behaviours in these MOs. However, recent work suggests that such post-hoc training methods may make interpretability unrealistically easy. We investigate this claim by constructing a suite of 61 OLMo2-1B - and gemma-3-1b-it-based MOs trained with seven different techniques, including standard post-hoc SFT, post-hoc DPO, and more realistic integration of MO data into the OLMo post-training DPO phase. We also use these to train 96 further variants via knowledge distillation. We use these MO variants to benchmark activation oracles, activation steering, the logit lens, and the Jacobian lens. Our findings show that (i) MO interpretability depends strongly on training objective, target behaviour, model architecture, and training data generation pipeline; (ii) substantial variance remains even after controlling for differences in the strength of target behaviour expression; and (iii) both our more realistic integrated training and knowledge distillation often yield less interpretable MOs than standard post-hoc methods. Our results cast substantial doubt on the validity of current MOs as interpretability proxies and suggest actionable steps towards more robust benchmarks.
Sources
- Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
- Sparse Autoencoders Find Highly Interpretable Features in Language Models
- Towards eliciting latent knowledge from LLMs with mechanistic interpretability
- Gemma 3 Technical Report
- Activation Oracles: Training and Evaluating LLMs as General-Purpose Activation Explainers
- Narrow Finetuning Leaves Clearly Readable Traces in Activation Differences
- 2 OLMo 2 Furious
- AI Deception: A Survey of Examples, Risks, and Potential Solutions
- Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- Steering Language Models With Activation Engineering
- AI Sandbagging: Language Models can Strategically Underperform on Evaluations
- HelpSteer3-Preference: Open Human-Annotated Preference Data across Diverse Tasks and Languages
- WizardLM: Empowering large pre-trained language models to follow complex instructions
Related papers
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis
- (Mis)generalization of Helpful-Only Fine-tuning