Pre-training interventions, ex post facto: Grafting model beliefs across checkpoints
cs.LG, cs.AI, cs.CL
Submitted: 2026-09-30
Updated: 2026-09-30
Comments: 78 pages. Code: https://github.com/peternutter/grafting-beliefs
Code: https://github.com/peternutter/grafting-beliefs
License: http://creativecommons.org/licenses/by/4.0/
The gist: Pre-training interventions are critical to alignment research, since beliefs formed during pre-training shape how a model generalizes from later training.
Terminology
Abstract
Pre-training interventions are critical to alignment research, since beliefs formed during pre-training shape how a model generalizes from later training. One recently popular technique for such interventions is synthetic document fine-tuning (SDF), which aims to alter what the model believes. Ideally, synthetic documents would be mixed into pre- or mid-training, but every change to a pre-training corpus must be followed by a full post-training run before its effect can be measured, making iteration slow and expensive. Common practice instead applies SDF to an already post-trained model. This is known to leave artifacts and degrade capabilities, and, as we show, it makes the model treat fabricated entities unrelated to the documents as real, a failure we call reality drift. We propose grafting: train the SDF adapter on the pre-trained checkpoint, then add the learned weight update to the post-trained model, which approximates the faithful approach while reusing the existing post-training. We demonstrate this by installing false facts, training misaligned model organisms and applying a constitutional mid-training intervention, across model families up to 284B parameters. Grafting installs the target belief as strongly as SDF on the post-trained model while reducing both reality drift and the loss of preference coherence by more than half on average, and it stays closer to a faithful mid-training run. Because grafting requires no post-training, the same adapter can be applied to any later checkpoint, enabling researchers to iterate quickly on pre-training interventions at the cost of a single fine-tuning run.
Sources
- Persona Cartography: Charting Language Model Personality Traits in Weight Space
- Constitutional Midtraining: Content Presence Drives Alignment Gains
- Training Verifiers to Solve Math Word Problems
- "Did you lie?" Evaluating Lie Detectors across Model Scale and Belief-Verified Model Organisms
- DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
- Alignment faking in large language models
- Editing Models with Task Arithmetic
- Model Spec Midtraining: Improving How Alignment Training Generalizes
- Safety Pretraining: Toward the Next Generation of Safe AI
- Auditing language models for hidden objectives
- Negation Neglect: When models fail to learn negations in training
- Utility Engineering: Analyzing and Controlling Emergent Value Systems in AIs
- Qwen3 Technical Report
- GPQA: A Graduate-Level Google-Proof Q&A Benchmark
- GIFT: Guided Fine-Tuning and Transfer for Enhancing Instruction-Tuned Language Models
- AuditBench: Evaluating Alignment Auditing Techniques on Models with Hidden Behaviors
- Believe It or Not: How Deeply do LLMs Believe Implanted Facts?
- When Role-playing, Do Models Believe What They Say?
- The Model Organism Lottery: Model Organism Interpretability Strongly Depends on Training Methodology
- Olmo 3
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks