Rethinking Token Reweighting for SFT: Suppress, Reverse, and Extrapolate Learned Features
cs.CL
Submitted: 2026-09-27
Updated: 2026-09-27
Comments: Preprint, Under Review
License: http://creativecommons.org/licenses/by/4.0/
The gist: Supervised fine-tuning (SFT) learns most aggressively from tokens that the model deems least likely.
Terminology
Abstract
Supervised fine-tuning (SFT) learns most aggressively from tokens that the model deems least likely. This helps acquire new behaviors, but also amplifies noisy or conflicting supervision and can overwrite useful pretrained knowledge. Through a unified policy-loss view, we revisit existing token-reweighting methods and show that they assign nonnegative coefficients to demonstrated tokens. Consequently, they can suppress or amplify supervised updates, but cannot reverse harmful features once learned. Moreover, larger training weights do not amount to feature extrapolation, since they change the optimization trajectory rather than scale a fixed SFT direction. We argue that reversal and extrapolation require a stable reference frame defined by a fixed SFT delta. Motivated by this, we propose SCALE (Selective Control of Adaptation via Local Entropy), an entropy-guided adaptation-strength-control method that freezes the pretrained model and the SFT delta and learns bounded token- and module-specific gates by minimizing predictive entropy alone. These gates suppress, reverse, or extrapolate frozen SFT features according to their alignment with entropy reduction. Across Qwen2.5-Math-1.5B, Qwen2.5-Math-7B, and Qwen3-4B-Base, SCALE achieves mathematical-reasoning averages of 37.84, 43.60, and 36.57, exceeding the strongest corresponding baselines while remaining competitive on general-retention benchmarks. It also attains the best average code-generation performance across HumanEval, HumanEval+, and MBPP for all three models. These results suggest that effective SFT correction can benefit from controlling how already learned residuals are used, rather than only modifying how they are learned.
Sources
- Program Synthesis with Large Language Models
- Evaluating Large Language Models Trained on Code
- Self-training Avoids Using Spurious Features Under Domain Shift
- SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training
- Entropy-Adaptive Fine-Tuning: Resolving Confident Conflicts to Mitigate Forgetting
- Measuring Mathematical Problem Solving With the MATH Dataset
- RL Fine-Tuning Heals OOD Forgetting in SFT
- Entropy-KL Divergence-based Token Masking: A Novel Approach for Selective Fine-tuning of Large Language Models
- ProFit: Leveraging High-Value Signals in SFT via Probability-Guided Token Selection
- Token Cleaning: Fine-Grained Data Selection for LLM Supervised Fine-Tuning
- Efficient and Effective Weight-Ensembling Mixture of Experts for Multi-Task Model Merging
- Forgetting: A New Mechanism Towards Better Large Language Model Fine-tuning
- Magicoder: Empowering Code Generation with OSS-Instruct
- On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectification
- DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
- Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement
- Qwen3 Technical Report
- AdaMerging: Adaptive Model Merging for Multi-Task Learning
- MoE$^2$-LoRA: When MoE Models Meet MoE-style Low-Rank Adaptation
- GLM-5: from Vibe Coding to Agentic Engineering
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering