How Post-Training Influences Verbalized Algorithmic Computation in Language Models
cs.CL, cs.LG
Submitted: 2026-11-10
Updated: 2026-11-10
License: http://creativecommons.org/licenses/by/4.0/
The gist: Post-training improves language-model reasoning, but how it changes the use of internal information and verbalized chain of thought (CoT) remains unclear.
Terminology
Abstract
Post-training improves language-model reasoning, but how it changes the use of internal information and verbalized chain of thought (CoT) remains unclear. We study successive SFT, DPO, and RLVR checkpoints of Olmo-3-7B-Instruct on multi-digit addition, using carry values as known intermediate state. Probes recover carry information from internal activations with similar accuracy in SFT and RLVR, although manipulating this information does not reliably control model outputs. In contrast, the usefulness of verbalized reasoning differs: CoTs produced by RLVR yield higher accuracy across receiving checkpoints than those produced by SFT or DPO. We also show that carry-related statements receive higher importance scores after RLVR than matched random controls. DPO shows similar sensitivity to carry-related statements but without comparable accuracy gains. These results suggest that post-training changes the usefulness of written reasoning and sensitivity to its content in meaningful ways, although greater sensitivity does not necessarily coincide with better problem performance.
Sources
- Mechanistic Indicators of Understanding in Large Language Models
- AI-rithmetic
- Thought Anchors: Which LLM Reasoning Steps Matter?
- Reasoning Theater: Disentangling Model Beliefs from Chain-of-Thought
- Reasoning Models Don't Always Say What They Think
- How to use and interpret activation patching
- Can LLMs subtract numbers?
- Olmo 3
- Beyond the Commitment Boundary: Probing Epiphenomenal Chain-of-Thought in Large Reasoning Models
- Steering Language Models With Activation Engineering
- Structure Development in List-Sorting Transformers
- Base Models Know How to Reason, Thinking Models Learn When
- Manifold Steering Reveals the Shared Geometry of Neural Network Representation and Behavior
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data