Math Reasoning in LLMs is Organized by Approach, Not Topic
cs.AI
Submitted: 2026-09-22
Updated: 2026-09-24
License: http://creativecommons.org/licenses/by/4.0/
The gist: Mathematical reasoning benchmarks are typically organized by topic, but language models may organize their internal computation by reusable reasoning approach instead.
Terminology
Abstract
Mathematical reasoning benchmarks are typically organized by topic, but language models may organize their internal computation by reusable reasoning approach instead. In this paper, we investigate whether open math-capable LLMs organize internally by topical sub-skill or by reasoning approach, and we present evidence that the approach is the key. We introduce a generation-replay protocol: a model first generates a solution, after which we replay the exact prompt-plus-generation trajectory and extract activation-importance signatures over the reasoning tokens. We cluster these signatures without supervision across eight models and five mathematical reasoning sources, then evaluate the recovered structure with structural, semantic, and intervention tests. Across all 40 model-source cells, the recovered clusters outperform matched-size random baselines. Two independent frontier-LLM judges find approach-level coherence in 77-82% of real clusters versus 6-11% in within-source controls, and topic-pure clusters usually receive labels finer than the topic itself. In approach-controlled prompting, changing the requested reasoning approach shifts cluster assignment in seven of eight model conditions, whereas paraphrases largely preserve it. These results indicate that math-capable LLMs organize internal mathematical computation by reasoning approach rather than benchmark topic. The implication is that topic-stratified benchmarks and topic-balanced training corpora can still miss the axis that matters: even deliberately topic-balanced corpora may remain imbalanced over reasoning approaches.
Sources
- Discovering Latent Knowledge in Language Models Without Supervision
- Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks
- Training Verifiers to Solve Math Word Problems
- Sparse Autoencoders Find Highly Interpretable Features in Language Models
- Measuring Mathematical Problem Solving With the MATH Dataset
- Measuring Faithfulness in Chain-of-Thought Reasoning
- GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models
- Show Your Work: Scratchpads for Intermediate Computation with Language Models
- Analysing Mathematical Reasoning Abilities of Neural Models
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
- Base Models Know How to Reason, Thinking Models Learn When
- Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small
- Self-Consistency Improves Chain of Thought Reasoning in Language Models
- Least-to-Most Prompting Enables Complex Reasoning in Large Language Models
- Representation Engineering: A Top-Down Approach to AI Transparency
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems