Beyond Safe Data: Pretraining-Stage Alignment with Regular Safety Reflection
cs.AI, cs.LG
Submitted: 2026-06-17
Updated: 2026-11-02
Code: https://github.com/Lightning-AI/litgpt
License: http://creativecommons.org/licenses/by/4.0/
The gist: To achieve deeper safety alignment for large language models (LLMs), recent efforts have studied how to push safety interventions earlier into the pretraining stage…
Terminology
Abstract
To achieve deeper safety alignment for large language models (LLMs), recent efforts have studied how to push safety interventions earlier into the pretraining stage, primarily by filtering unsafe data or rewriting it into safer forms. We argue that pretraining-stage alignment should go beyond making the data safe: LLMs may generalize unsafe behaviors from seemingly benign pretraining data. To this end, we propose Safety Reflection Pretraining, a pretraining-stage alignment method which regularly inserts short safety reflections between sentences in pretraining corpora. This integrates self-monitoring directly into language modeling, establishing a foundational capability that is subsequently reinforced by compatible post-training. Our experiments with 1.7B models pretrained on FineWeb-Edu show that Safety Reflection Pretraining improves safety classification accuracy and substantially reduces the success rates of inference-stage and fine-tuning attacks. Complementary to our real-world experiments, we also introduce a fully controlled synthetic environment, MedSafetyWorld, with a clear definition of safety and a reasoning structure under which we find that models can easily generalize unsafe behaviors from safe data. Ablations in MedSafetyWorld further demonstrate a clear advantage of Safety Reflection Pretraining over data filtering and rewriting in preventing models from acting on unsafe behaviors. Taken together, our findings suggest that pretraining alignment should not only make the training data safe, but also shape the behaviors that models are likely to acquire from safe data.
Sources
- From Refusal Tokens to Refusal Control: Discovering and Steering Category-Specific Refusal Directions
- Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
- Constitutional AI: Harmlessness from AI Feedback
- Jailbreaking Black Box Large Language Models in Twenty Queries
- Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
- Training Verifiers to Solve Math Word Problems
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- Phantom Transfer: Data Poisoning can Survive Data-Level Defences
- TinyStories: How Small Can Language Models Be and Still Speak Coherent English?
- Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training
- Generative Data Refinement: Just Ask for Better Data
- Data-efficient pre-training by scaling synthetic megadocs
- What Matters For Safety Alignment?
- Robustifying Safety-Aligned Large Language Models through Clean Data Curation
- The Llama 3 Herd of Models
- Qwen3 Technical Report
- Shaping capabilities with token-level data filtering
- Reasoning to Learn from Latent Thoughts
- When Should We Introduce Safety Interventions During Pretraining?
- Backtracking for Safety
Related papers
- Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Embeddings for Preferences, Not Semantics
- Can We Trust Item Response Theory for AI Evaluation?
- Diagnosing Task Insensitivity in Language Agents