The Scaling Properties of Implicit Deductive Reasoning in Transformers
cs.AI, cs.CC, cs.LO, cs.SC
Submitted: 2026-05-05
Updated: 2026-09-25
Comments: Accepted TMLR
License: http://creativecommons.org/licenses/by/4.0/
The gist: We investigate the scaling properties of implicit deductive reasoning over Horn clauses in depth-bounded Transformers.
Terminology
Abstract
We investigate the scaling properties of implicit deductive reasoning over Horn clauses in depth-bounded Transformers. By discouraging the reliance on statistical shortcuts via counterfactual data augmentation, and promoting the learning of shared reasoning primitives across direct and CoT modes, we find that in sufficiently deep models with a bidirectional prefix mask, implicit reasoning approaches explicit CoT performance across graph topologies and problem widths, though CoT remains necessary for depth extrapolation. These findings represent a step toward achieving better compositional reasoning in Transformers. The code and models to reproduce this work are available at: https://github.com/envomp/Implicit-Deductive-Reasoning-in-Transformers
Sources
- Grokking in the Wild: Data Augmentation for Real-World Multi-Hop Reasoning with Transformers
- Understanding intermediate layers using linear classifier probes
- Lower Bounds for Sparse Recovery
- Eliciting Latent Predictions from Transformers with the Tuned Lens
- Decoding by Linear Programming
- Premise Order Matters in Reasoning with Large Language Models
- Faithful Reasoning Using Large Language Models
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- How Many Features Can a Language Model Store Under the Linear Representation Hypothesis?
- Looped Transformers as Programmable Computers
- ProtoReasoning: Prototypes as the Foundation for Generalizable Reasoning in LLMs
- SGD on Neural Networks Learns Functions of Increasing Complexity
- Learning the Difference that Makes a Difference with Counterfactually-Augmented Data
- When Does LeJEPA Learn a World Model?
- Locating and Editing Factual Associations in GPT
- The Illusion of State in State-Space Models
- The Llama 3 Herd of Models
- Magistral
- WT5?! Training Text-to-Text Models to Explain their Predictions
- How Transformers Learn Causal Structure with Gradient Descent
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems