Where Do Distilled Students Store Thinking?
cs.CL, cs.LG
Submitted: 2026-11-03
Updated: 2026-11-03
Comments: 19 pages, 4 figures. Code: https://github.com/kimtaeyeong1229/cot-distill-lora-subspace
Code: https://github.com/kimtaeyeong1229/cot-distill-lora-subspace
License: http://creativecommons.org/licenses/by/4.0/
The gist: Small students degrade on large reasoning models' (LRMs') long solutions, so recent work shortens them by interpolating an LRM with a short-answer model at ratio λ.
Abstract
Small students degrade on large reasoning models' (LRMs') long solutions, so recent work shortens them by interpolating an LRM with a short-answer model at ratio λ. We ask what student weights retain when this teacher stops thinking, comparing LoRA adapters of three students from three teacher pairs at each λ. In each pair, the adapters' update subspace shifts most where the teacher stops thinking and solutions shorten most, locating the change without attributing it. We identify a shared adapter subspace tied to thinking-like behavior (reflective markers, monologue format and long solutions). Shared by adapters before that point, it keeps about half (post hoc) while one pair's teacher halves its solutions but keeps its think tag, and largely empties after thinking stops. Removing it largely turns this behavior off for every pair, student and benchmark. Only in that pair and one other student does it leave clearly fewer markers than removing the update's top components; elsewhere, same-energy random directions often nearly match it. Grafted into a short-answer student at twice its weight, it turns early markers (a post hoc count) on, like the whole update, but the student rarely finishes. Our work identifies a weight subspace tied to distilled thinking-like behavior.
Sources
- Towards Understanding Distilled Reasoning Models: A Representational Approach
- Scale or Reason? A Compute-Equivalent Analysis of Reasoning Distillation
- The Molecular Structure of Thought: Mapping the Topology of Long Chain-of-Thought Reasoning
- Training Verifiers to Solve Math Word Problems
- When One Adapter Speaks for Many: Discovering Low-Rank Redundancy in Continual Fine-Tuning
- Mind the Cap: Output-Budget Regimes Change the Measured Multilingual Reasoning Gap
- The Llama 3 Herd of Models
- Distilling the Knowledge in a Neural Network
- Transcoder Adapters for Reasoning-Model Diffing
- When the Strongest Teacher Is Not the Best Teacher: Student-Centric Answer Selection
- The Halt Vector: Internalizing a Causal Steering Intervention for Efficient Reasoning
- Cross-Architecture Model Diffing with Crosscoders: Unsupervised Discovery of Differences Between LLMs
- Kimi k1.5: Scaling Reinforcement Learning with LLMs
- Assembly of Experts: Linear-time construction of the Chimera LLM variants with emergent and adaptable behaviors
- The Thinking Spectrum: An Empirical Study of Tunable Reasoning in LLMs through Model Merging
- MI-Distillation: Selecting from Model-Interpolated Instruct-Reasoning Data Spectrum for Chain-of-Thought Distillation
- Gold-Switch: Training-Free Superposition of Slow- and Fast- Thinking LLMs
- S cubed: Spectral Null-Space Swap Makes Reasoning Models Efficient
- Reasoning Models Can Be Effective Without Thinking
- Qwen2.5 Technical Report
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data