Probing the Origins of Reasoning Performance: Representational Quality for Mathematical Problem-Solving in RL vs. SFT Fine-Tuned Models
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "Probing the Origins of Reasoning Performance".
Jane: Here's the gist of the paper. Large reasoning models trained via reinforcement learning (RL) have increasingly been shown to outperform their supervised fine-tuned (SFT) counterparts on mathematical reasoning tasks; yet the mechanistic basis for this advantage remains unclear The authors investigate what internal representational differences enable RL models’ superior performance Probing Representation Clarity Linear probes trained on layer-wise hidden states reveal that RL models tend to achieve higher accuracy in predicting answer correctness compared to SFT models, indicating more linearly separable and structured representations This suggests that RL training fundamentally restructures how models represent and process reasoning problems Specifically, reasoning-capable models exhibit remarkably high probe accuracy from the very first layer, with test accuracies of 70% at layer 0 compared to 65% for Instruct models The authors hypothesize that the improved mathematical reasoning ability of reasoning-capable models stems from developing clearer internal representations of correctness—i.e., representations that are more linearly separable and consistently structured across samples Architectural Restructuring Mean ablation studies show that RL models develop a hierarchical architecture where deeper layers become progressively more critical, whereas SFT models distribute importance uniformly across layers The authors found that RL-trained models show earlier engagement and progressive concentration in deeper layers (r=0.47), whereas instruction-tuned models distribute reasoning uniformly (r=-0.11) This architectural difference, combined with earlier emerging and stronger answer representations in RL models, provides initial mechanistic insight into performance differences Token Allocation Dynamics The analysis of token-count variability under repeated sampling across problems suggests that token allocation may depend more on the overall training pipeline than on RL versus SFT alone While higher variability is observed in some RL-tuned models, strong consistency is seen in others, suggesting that token allocation may depend more on the overall training pipeline than on RL versus SFT alone This variability reveals the spread of plausible onpolicy reasoning, highlighting which models exhibit stable policies versus those that are under-determined The findings demonstrate that RL-trained models develop superior representations that emerge earlier in the network; specifically, in the DeepSeek model family, RL-trained models develop representations that are more linearly separable and emerge earlier while Olmo models show a similar pattern for representation quality although with less pronounced differences in emergence timing The authors suggest designing reward structures that explicitly incentivize adaptive token allocation could better exploit RL’s potential Further analysis should extend probing to intermediate reasoning steps to reveal how multi-step solutions are constructed and validated Designing methods to detect and measure representation quality during training could enable real-time assessment of model reliability in high-stakes domains Pruning redundant reasoning tokens The authors analyze token-count variability under repeated sampling across problems to assess adaptive compute allocation They observe higher variability in some RL-tuned models than in their SFT counterparts, suggesting that token allocation may depend more on the overall training pipeline than on RL versus SFT alone This token-allocation variability reveals the spread of plausible onpolicy reasoning, highlighting which models exhibit stable policies versus those that are under-determined How the training signal is built The authors present two converging lines of evidence: First, linear probes trained on layer-wise hidden states reveal that RL models tend to achieve higher accuracy in predicting answer correctness compared to SFT models, indicating more linearly separable and structured representations Second, mean ablation studies show that RL models develop a hierarchical architecture where deeper layers become progressively more critical, whereas SFT models distribute importance uniformly across layers Together, these findings demonstrate that RL training fundamentally restructures how models represent and process reasoning problems Where the gains come from The authors conclude that RL-trained models develop superior representations that emerge earlier in the network; specifically, in the DeepSeek model family, RL-trained models develop representations that are more linearly separable and emerge earlier while Olmo models show a similar pattern for representation quality although with less pronounced differences in emergence timing This suggests that RL training fundamentally restructures how models represent and process reasoning problems Token Variability in Mathematical Problem-Solving The authors investigate whether the representational differences observed in their previous findings manifest in downstream behaviours such as token variability between RL and SFT models They evaluate on 50 problems from GSM8K-Platinum, generating 50 independent responses per problem per model The authors use the token coefficient of variation (CV) to measure variability across the 50 responses per problem as CV = σtokens/µtokens, where σtokens and µtokens are the standard deviation and mean of output token counts respectively DeepSeek-Math-RL exhibits high variability at capability boundaries (CV > 0.8), meaning identical problems elicit vastly different response lengths—precisely the calibration failure predicted by informationtheoretic analysis This divergence suggests that the relationship between training methodology and token allocation consistency is model-dependent, likely influenced by the specific training pipeline and reward structure rather than RL versus SFT alone Pruning redundant reasoning tokens The authors analyze token-count variability under repeated sampling across problems to assess adaptive compute allocation While higher variability is observed in some RL-tuned models, strong consistency is seen in others, suggesting that token allocation may depend more on the overall training pipeline than on RL versus SFT alone This variability reveals the spread of plausible onpolicy reasoning, highlighting which models exhibit stable policies versus those that are under-determined Pruning redundant reasoning tokens
Title and authors: Tom: So this paper is titled "Probing the Origins of Reasoning Performance: Representational Quality for Mathematical Problem-Solving in RL vs. SFT Fine-Tuned Models". It's looking at why reinforcement learning models are doing so much better on math problems than their supervised fine-tuned versions.
Jane: That sounds like it gets right to the heart of the mystery, doesn't it? The title says they’re probing the origins of that performance difference by looking at representation quality.
Lu: Exactly. They're not just saying RL is better; they want to show *how* RL fundamentally changes what the model understands internally when it comes to math problems.
Meng: So, they are investigating what specific internal differences allow these reinforcement learning models to perform better on mathematical reasoning tasks than the supervised fine-tuned ones.
Lalam: It’s about figuring out if it’s just a training trick or if there’s something structurally different happening inside the model's layers.
The paper's summary: Tom: Okay, so what they found is two main things that suggest this structural difference exists. First, they used linear probes on hidden states and found that reinforcement learning models tend to have higher accuracy when predicting if an answer is correct compared to the supervised fine-tuned models.
Jane: So, what does that mean for how we look at these models? If the linear probes are more accurate, it suggests that the representations inside those RL models are more linearly separable and structured than in SFT models.
Lu: That's what they hypothesize. They believe this means that the improved mathematical reasoning ability of these RL-trained models comes from developing clearer internal representations of correctness, which are more linearly separable and consistently structured across different samples.
Meng: So, they’re suggesting that the way RL training restructures how the model represents and processes problems is what gives it an edge.
Lalam: It seems like they’re showing that the way reinforcement learning shapes the internal structure of a model is key to its superior mathematical performance compared to instruction tuning.
The paper's improvements: Tom: Beyond just saying RL models are better, they show some architectural things too. They did mean ablation studies and found that RL models develop a hierarchical architecture where deeper layers become more critical as you go down.
Jane: That contrasts with supervised fine-tuned models, which tend to distribute importance more uniformly across all the layers. So, the importance shifts from being spread out to becoming concentrated in deeper layers for RL models.
Lu: Yes, they found that RL-trained models show earlier engagement and progressive concentration in deeper layers, with a ratio of zero point four seven compared to negative zero point eleven for instruction-tuned models. That’s a measurable architectural difference.
Meng: So, the model is not just getting smarter overall; its structure changes how it uses its capacity during reasoning tasks.
Lalam: And this architectural shift, combined with those earlier and stronger answer representations they found in the RL models, gives us some initial mechanistic insight into why they perform better initially.
Conclusion: Tom: So to wrap up, the main finding is that reinforcement learning models develop superior representations that emerge earlier in the network. Specifically, in the DeepSeek model family, RL-trained models develop representations that are more linearly separable and emerge earlier while Olmo models show a similar pattern for representation quality although with less pronounced differences in emergence timing.
Jane: It seems like they’re saying that this means RL training fundamentally restructures how these large reasoning models represent and process mathematical problems. This suggests the gains aren't just about brute force training, but about building a better internal map of correctness from the start.
Lu: And they did touch on how we can improve things by suggesting designing reward structures that explicitly incentivize adaptive token allocation. They think that could better exploit reinforcement learning’s potential for controlling output length and reasoning steps.
Meng: If you look at the token variability analysis, there was some higher variability in some RL-tuned models than in SFT counterparts, which hints that token allocation might depend more on the overall training pipeline than just RL versus SFT alone.
Lalam: And they also pointed out that for models like DeepSeek-Math-RL, there's high variability at capability boundaries where identical problems elicit vastly different response lengths, which is what we call calibration failure.
Tom: So the implication is that future work should focus on analyzing intermediate reasoning steps to see exactly how multi-step solutions are constructed and validated. It’s a path toward building models that are more reliable in high-stakes domains.
Jane: It sounds like a deep dive into the mechanics of why some models just seem to nail math problems better than others, and that's what we were hoping for from this paper on Probing the Origins of Reasoning Performance.
Antyabha Rahman, Akshaj Gurugubelli, Omar Ankit, Kevin Zhu, Aishwarya Balwani
University of New South Wales · Algoverse AI Research · University of Waterloo · St. Jude Children’s Research Hospital
cs.AI, cs.CL
Submitted: 2026-07-28
Updated: 2026-11-10
Comments: Second Workshop on XAI4Science, AAAI 2026
Project page: https://oankit.github.io/-rl-sft-reasoning/Introduction
License: http://creativecommons.org/licenses/by-nc-nd/4.0/
Importance score: 92/100
The gist: Large reasoning models trained via reinforcement learning (RL) have increasingly been shown to outperform their supervised fine-tuned (SFT) counterparts on mathematical reasoning tasks…
Key concepts
- Linear Probes
- These are tests run on the hidden states of a neural network to see how well they can predict if an answer is correct. The findings show that RL models' probes are more accurate, indicating their internal representations of correctness are more structured and easier to distinguish from incorrect answers.
- Architectural Restructuring
- This refers to how the importance of different layers in the model changes during training. RL models develop a hierarchical structure where deeper layers become increasingly vital for reasoning, unlike SFT models which distribute importance more evenly across all layers.
- Token Allocation Dynamics
- This examines how much 'space' (tokens) a model uses to generate an answer when solving problems. The paper finds that token allocation variability is influenced by the overall training pipeline rather than RL versus SFT alone, showing how stable or unpredictable the model's reasoning policy is.
Terminology
Summary
Large reasoning models trained via reinforcement learning (RL) have increasingly been shown to outperform their supervised fine-tuned (SFT) counterparts on mathematical reasoning tasks; yet the mechanistic basis for this advantage remains unclear The authors investigate what internal representational differences enable RL models’ superior performance
Probing Representation Clarity
Linear probes trained on layer-wise hidden states reveal that RL models tend to achieve higher accuracy in predicting answer correctness compared to SFT models, indicating more linearly separable and structured representations This suggests that RL training fundamentally restructures how models represent and process reasoning problems Specifically, reasoning-capable models exhibit remarkably high probe accuracy from the very first layer, with test accuracies of 70% at layer 0 compared to 65% for Instruct models The authors hypothesize that the improved mathematical reasoning ability of reasoning-capable models stems from developing clearer internal representations of correctness—i.e., representations that are more linearly separable and consistently structured across samples
Architectural Restructuring
Mean ablation studies show that RL models develop a hierarchical architecture where deeper layers become progressively more critical, whereas SFT models distribute importance uniformly across layers The authors found that RL-trained models show earlier engagement and progressive concentration in deeper layers (r=0.47), whereas instruction-tuned models distribute reasoning uniformly (r=-0. p. 0 Our analysis reveals that training methodology fundamentally reshapes computational architecture: RL-trained models show earlier engagement and progressive concentration in deeper layers (r=0. 47), whereas
instruction-tuned models distribute reasoning uniformly (r=-
0.Where the paper says it
Token Allocation Dynamics
The analysis of token-count variability under repeated sampling across problems suggests that token allocation may depend more on the overall training pipeline than on RL versus SFT alone While higher variability is observed in some RL-tuned models, strong consistency is seen in others, suggesting that token allocation may depend more on the overall training pipeline than on RL versus SFT alone This variability reveals the spread of plausible onpolicy reasoning, highlighting which models exhibit stable policies versus those that are under-determined
Conclusion and Future Directions
The findings demonstrate that RL-trained models develop superior representations that emerge earlier in the network; specifically, in the DeepSeek model family, RL-trained models develop representations that are more linearly separable and emerge earlier while Olmo models show a similar pattern for representation quality although with less pronounced differences in emergence timing The authors suggest designing reward structures that explicitly incentivize adaptive token allocation could better exploit RL’s potential Further analysis should extend probing to intermediate reasoning steps to reveal how multi-step solutions are constructed and validated Designing methods to detect and measure representation quality during training could enable real-time assessment of model reliability in high-stakes domains
Pruning redundant reasoning tokens
The authors analyze token-count variability under repeated sampling across problems to assess adaptive compute allocation They observe higher variability in some RL-tuned models than in their SFT counterparts, suggesting that token allocation may depend more on the overall training pipeline than on RL versus SFT alone This token-allocation variability reveals the spread of plausible onpolicy reasoning, highlighting which models exhibit stable policies versus those that are under-determined
How the training signal is built
The authors present two converging lines of evidence: First, linear probes trained on layer-wise hidden states reveal that RL models tend to achieve higher accuracy in predicting answer correctness compared to SFT models, indicating more linearly separable and structured representations Second, mean ablation studies show that RL models develop a hierarchical architecture where deeper layers become progressively more critical, whereas SFT models distribute importance uniformly across layers Together, these findings demonstrate that RL training fundamentally restructures how models represent and process reasoning problems
Where the gains come from
The authors conclude that RL-trained models develop superior representations that emerge earlier in the network; specifically, in the DeepSeek model family, RL-trained models develop representations that are more linearly separable and emerge earlier while Olmo models show a similar pattern for representation quality although with less pronounced differences in emergence timing This suggests that RL training fundamentally restructures how models represent and process reasoning problems
Token Variability in Mathematical Problem-Solving
The authors investigate whether the representational differences observed in their previous findings manifest in downstream behaviours such as token variability between RL and SFT models They evaluate on 50 problems from GSM8K-Platinum, generating 50 independent responses per problem per model The authors use the token coefficient of variation (CV) to measure variability across the 50 responses per problem as CV = σtokens/µtokens, where σtokens and µtokens are the standard deviation and mean of output token counts respectively DeepSeek-Math-RL exhibits high variability at capability boundaries (CV > 0.8), meaning identical problems elicit vastly different response lengths—precisely the calibration failure predicted by informationtheoretic analysis This divergence suggests that the relationship between training methodology and token allocation consistency is model-dependent, likely influenced by the specific training pipeline and reward structure rather than RL versus SFT alone
The authors analyze token-count variability under repeated sampling across problems to assess adaptive compute allocation While higher variability is observed in some RL-tuned models, strong consistency is seen in others, suggesting that token allocation may depend more on the overall training pipeline than on RL versus SFT alone This variability reveals the spread of plausible onpolicy reasoning, highlighting which models exhibit stable policies versus those that are under-determined
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Representational Quality Improvement by RL Training: RL trained models develop
clearer
internal representations becauseproblems with more linearly separable internal representations should exhibit higher discriminability between correct and incorrect answers.
This enables a mechanism for better mathematical reasoning by encoding correctness asrobust, linearly separable features.
-
Architectural Optimization via Layer Criticality: AI systems can be improved by designing architectures that mimic the RL-trained model's strategy, which shows
higher early-layer impact combined with progressive deepening,
indicating a hierarchical reasoning architecture. This means concentrating reasoning in deeper layers like 9-18 and 22-26 for superior mathematical performance. -
Adaptive Token Allocation Strategy: The system can be improved by designing reward structures that
explicitly incentivize adaptive token allocation could better exploit RL’s potential.
This allows the model to move beyond fixed output lengths and potentially reducecompression inefficiency at capability boundaries
seen in models like DeepSeek-Math-RL.
Abstract
Large language models trained with reinforcement learning (RL) to reason step by step have been increasingly shown to outperform their supervised fine-tuned (SFT) counterparts on mathematical reasoning tasks. Yet the mechanistic basis for this improvement remains unclear. We therefore ask, what internal representational differences enable RL-trained reasoning language models to reason better? Our work presents two primary lines of evidence: First, layer-wise linear probes tend to predict answer correctness more accurately in RL-trained reasoning language models than in instruction-tuned models on synthetic math problems, although output length alone matches the DeepSeek-Math-RL probe on two templates. Second, mean ablation studies suggest that accuracy drop rises with layer depth in DeepSeek-Math-RL but not in DeepSeek-Math-Instruct. Finally, we analyse token-count variability across accuracy bins for RL-trained reasoning language models and instruction-tuned models to measure whether token counts vary more on harder problems. Token counts tend to vary more on lower-accuracy problems, although in DeepSeek-Math-Instruct only with truncated responses included. At high accuracy, we observe lower variability in DeepSeek-Math-RL than in DeepSeek-Math-Instruct, and we observe higher variability in Olmo-3-Think than in Olmo-3-Instruct.
Sources
- Understanding intermediate layers using linear classifier probes
- How does Chain of Thought Think? Mechanistic Interpretability of Chain-of-Thought Reasoning with Sparse Autoencoding
- Training Verifiers to Solve Math Word Problems
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model
- On the Origins of Linear Representations in Large Language Models
- How Well do LLMs Compress Their Own Chain-of-Thought? A Token Complexity Approach
- Locating and Editing Factual Associations in GPT
- GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models
- Olmo 3
- OpenAI o1 System Card
- The Linear Representation Hypothesis and the Geometry of Large Language Models
- Probing for Arithmetic Errors in Language Models
- Do Large Language Model Benchmarks Test Reliability?
- Demystifying Long Chain-of-Thought Reasoning in LLMs
- Reasoning Models Know When They're Right: Probing Hidden States for Self-Verification
- Towards Best Practices of Activation Patching in Language Models: Metrics and Methods
- A Closer Look at How Fine-tuning Changes BERT
- Language Models Encode the Value of Numbers Linearly
Related papers
- Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Embeddings for Preferences, Not Semantics
- Can We Trust Item Response Theory for AI Evaluation?
- Diagnosing Task Insensitivity in Language Agents