Stratified Consistency Distillation for Natural Language Formalization
cs.CL, cs.AI
Submitted: 2026-08-31
Updated: 2026-10-02
License: http://creativecommons.org/licenses/by/4.0/
The gist: Neurosymbolic reasoning has shown promising success in addressing complex reasoning tasks by combining large language models (LLMs) and symbolic solvers.
Terminology
Abstract
Neurosymbolic reasoning has shown promising success in addressing complex reasoning tasks by combining large language models (LLMs) and symbolic solvers. While this approach shows promise, a fundamental challenge remains: improving the accuracy of translations from natural language to logical formulas. Current methods predominantly rely on prompt engineering, which is difficult to scale across different domains and input formats. Drawing inspiration from the success of fine-tuning in other model adaptation and alignment applications, we propose a fine-tuning-based Stratified Consistency Distillation approach: (1) We generate K logical translations per input using a frontier LLM and cluster them by semantic equivalence (2) Based on the entropy level, we apply majority voting (low entropy), LLM-as-a-Judge (medium entropy), or unification/abstention (high entropy), and (3) fine-tune a smaller model using the selected pseudo-labels. Our experiments show significant and consistent improvements in both Pass@K and our novel Equivalent Logical Similarity metrics, demonstrating the potential of advancing logical translation through consistency distillation.
Sources
- Training Verifiers to Solve Math Word Problems
- Complexity-Based Prompting for Multi-Step Reasoning
- Divide and Translate: Compositional First-Order Logic Translation and Verification for Complex Logical Reasoning
- Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
- Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations
- Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models
- Fine-Tuning Language Models from Human Preferences
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering