SymDiag: Explainable Diagnosis for LLM Reasoning via Neuro-Symbolic Verification
Wenyao Cui, Huaping Zhang, Yongyi Huang, Qiuchi Li, Jian Xu, Cheng-Lin Liu, Chunxiao Gao, Juan Wang, Baohua Zhang
cs.AI
Submitted: 2026-08-09
Updated: 2026-08-11
License: http://creativecommons.org/licenses/by/4.0/
The gist: Large language models (LLMs) increasingly serve as data-driven reasoners, yet their chains-of-thought (CoT) can be unfaithful even when final answers are correct.
Terminology
Abstract
Large language models (LLMs) increasingly serve as data-driven reasoners, yet their chains-of-thought (CoT) can be unfaithful even when final answers are correct. Most existing ``verification'' signals are not diagnostic: answer matching observes only the outcome, LLM-as-judge provides subjective and non-verifiable critiques, and scalar rewards (e.g., PRMs/RMs) offer little insight into where a multi-step derivation fails.We propose SymDiag, a neuro-symbolic framework that reframes reasoning verification as structured failure diagnosis. SymDiag translates natural-language CoT into symbolic constraints and performs step-level satisfiability/entailment checks to (i) localize failing steps and (ii) produce verifiable diagnostic evidence, including counterexamples, inconsistency witnesses, and missing-premise indicators. A central challenge is that apparent ``logic violations'' can be caused either by genuine reasoning defects or by neural-to-symbolic translation noise. SymDiag therefore incorporates a Self-Auditor that disentangles TranslationError from ReasoningError via dual symbolic encodings consistency checks, enabling robust diagnosis under partial observability. Across diverse mathematical, logical, scientific, and general reasoning benchmarks, SymDiag improves detection of unfaithful reasoning and provides substantially more effective feedback for multi-round reasoning repair than outcome-only verification and LLM-based judging, offering a principled foundation for trustworthy and scalable reasoning diagnosis.
Sources
- gpt-oss-120b & gpt-oss-20b Model Card
- Logically Consistent Language Models via Neuro-Symbolic Integration
- A Comparative Study of Neurosymbolic AI Approaches to Interpretable Logical Reasoning
- Training Verifiers to Solve Math Word Problems
- ReTool: Reinforcement Learning for Strategic Tool Use in LLMs
- The Llama 3 Herd of Models
- Learning an Efficient Optimizer via Hybrid-Policy Sub-Trajectory Balance
- Learning to Learn Weight Generation via Local Consistency Diffusion
- Towards Learning to Reason: Comparing LLMs with Neuro-Symbolic on Arithmetic Relations in Abstract Reasoning
- HybridProver: Augmenting Theorem Proving with LLM-Driven Proof Synthesis and Refinement
- RAM: Recover Any 3D Human Motion in-the-Wild
- Multi-step Inference over Unstructured Data
- NeSTR: A Neuro-Symbolic Abductive Framework for Temporal Reasoning in Large Language Models
- Graph Canvas for Controllable 3D Scene Generation
- Eidoku: A Neuro-Symbolic Verification Gate for LLM Reasoning via Structural Constraint Satisfaction
- VIRO: Robust and Efficient Neuro-Symbolic Reasoning with Verification for Referring Expression Comprehension
- Reliable Proof Generation with LLMs via Analogical Retrieval and Symbolic Verification: A Case Study in Euclidean Geometry
- Solving math word problems with process- and outcome-based feedback
- Enhancing Large Language Models through Neuro-Symbolic Integration and Ontological Reasoning
- HelpSteer3-Preference: Open Human-Annotated Preference Data across Diverse Tasks and Languages
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems