Reproducing and Stress-Testing Two Approaches to LLM Reasoning Reliability: Test-Time Probability Aggregation and Logic-Representation Editing
Minhan Cho, Jimin Kweon
cs.AI, cs.CL, cs.LG
Submitted: 2026-08-09
Updated: 2026-08-13
Comments: 16 pages, 3 figures, 9 tables. Code, data, and experiment logs: https://github.com/rabqatab/llm-reasoning-reliability-reproduction
Code: https://github.com/rabqatab/llm-reasoning-reliability-reproduction
License: http://creativecommons.org/licenses/by/4.0/
The gist: We independently reproduce two recent methods for making large language model (LLM) reasoning more reliable, and stress-test them across domains and models (RPC across four new task domains with
Terminology
Abstract
We independently reproduce two recent methods for making large language model (LLM) reasoning more reliable, and stress-test them across domains and models (RPC across four new task domains with Qwen3-8B, LCF across four 7-8B models). The first, RPC, aggregates token probabilities and self-consistency at inference; the second, LCF, trains projectors that split hidden states into "content" and "logic" and edits the logic part toward a valid region. Validating such reliability claims matters because the original evaluations are run by each method's own authors and were never independently reproduced or stress-tested across models and domains, and LCF shipped no public code. We re-run RPC's published-path aggregation and re-implement LCF's projector, contrastive, and intervention pipeline, then extend both to text-to-SQL, legal extraction, fallacy identification, and precedent grading, and probe LCF's representation directly. RPC reproduces the original grid exactly on the authors' released reasoning paths; on four new domains its edge over self-consistency is never significant (ties or small mixed differences, paired p >= 0.28), and on BIRD, the one domain where we vary the budget, the edge grows with K as predicted but its largest gap (+2.5 accuracy at K=32, p=0.16) reverses to-0.25 when we enlarge the sample to n=200. LCF's logic-validity direction is real but weak (0.82 separability at the single best sub-layer versus 0.95 for a semantic-attribute control); its one positive effect (Qwen3 Prob) is not significant (p=0.56), while it significantly reduces Prob on two of the other three models.
Sources
- Training Verifiers to Solve Math Word Problems
- Language models show human-like content effects on reasoning tasks
- Mistral 7B
- Language Models (Mostly) Know What They Know
- Reasoning or Fluency? Dissecting Probabilistic Confidence in Best-of-N Selection
- A Looming Replication Crisis in Evaluating Behavior in Language Models? Evidence and Solutions
- Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificity
- Decodable but Not Corrected by Fixed Residual-Stream Linear Steering: Evidence from Medical LLM Failure Regimes
- Qwen3 Technical Report
- Llama 2: Open Foundation and Fine-Tuned Chat Models
- Steering Language Models With Activation Engineering
- Representation Engineering: A Top-Down Approach to AI Transparency
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems