Diff Mining: Logit Differences Reveal Finetuning Objectives
cs.LG, cs.AI, cs.CL
Submitted: 2026-08-26
Updated: 2026-08-26
Comments: 37 pages, 7 figures. ICLR 2026 Workshop: Principled Design for Trustworthy AI. Code available at https://github.com/science-of-finetuning/diffing-toolkit
Code: https://github.com/safety-research/petri
License: http://creativecommons.org/licenses/by/4.0/
The gist: Finetuning has become the gold standard for refining existing behaviors and inducing new ones in language models, yet it often remains unclear exactly which behaviors emerge during this process.
Terminology
Abstract
Finetuning has become the gold standard for refining existing behaviors and inducing new ones in language models, yet it often remains unclear exactly which behaviors emerge during this process. As models grow ever more capable, understanding finetuning better becomes increasingly important, particularly since unwanted behaviors may arise during finetuning. In this paper, we introduce Diff Mining, a simple yet effective framework for identifying what a finetuned model has learned by comparing its logits to those of its base model. Diff Mining effectively surfaces salient tokens that are amplified in the finetuned model, serving as a fingerprint of its training -- even on text unrelated to the finetuning domain. Unlike many existing model diffing methods which require model internals, Diff Mining only needs access to output logits and scales to large models. The framework consists of two modular stages: (i) extracting per-context logit differences between the finetuned and base models on a reference corpus, and (ii) aggregating the resulting signals to construct an interpretable token set representing the finetune. For aggregation, we explore both a simple Top-K frequency method and a Non-negative Matrix Factorization (NMF)-based approach for disentangling multiple finetuning objectives into distinct token clusters. Empirically, Diff Mining succeeds across diverse settings: on finetune domain detection, it significantly outperforms state-of-the-art model diffing methods both in identifying relevant tokens and in downstream performance when an interpretability agent is given access to the extracted token set; on models with injected biases, it identifies more than one third of the biases without targeted probing. Overall, our framework shows promise in developing auditing tools to detect finetuning objectives.
Sources
- Weird Generalization and Inductive Backdoors: New Ways to Corrupt LLMs
- MEDITRON-70B: Scaling Medical Pretraining for Large Language Models
- On Domain-Adaptive Post-Training for Multimodal Large Language Models
- Training Verifiers to Solve Math Word Problems
- Towards eliciting latent knowledge from LLMs with mechanistic interpretability
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- Alignment faking in large language models
- Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training
- Open Character Training: Shaping the Persona of AI Assistants through Constitutional AI
- Auditing language models for hidden objectives
- Frontier Models are Capable of In-context Scheming
- Overcoming Sparsity Artifacts in Crosscoders to Interpret Chat-Tuning
- OpenAI o1 System Card
- Towards Understanding Sycophancy in Language Models
- Believe It or Not: How Deeply do LLMs Believe Implanted Facts?
- Auditing Games for Sandbagging
- Persona Features Control Emergent Misalignment
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks