Collaborative Reasoning Distillation via Cross-Feedback and Coherent Curation
cs.LG, cs.AI, cs.CL
Submitted: 2026-10-07
Updated: 2026-10-07
Comments: Accepted at NeurIPS 2026
License: http://creativecommons.org/licenses/by/4.0/
The gist: Reasoning capabilities are critical for advancing Large Language Models, yet current approaches either require massive computational budgets or struggle to effectively distill reasoning to smaller
Terminology
Abstract
Reasoning capabilities are critical for advancing Large Language Models, yet current approaches either require massive computational budgets or struggle to effectively distill reasoning to smaller models. Standard distillation methods rely on outcome-based rewards, failing to distinguish between sound reasoning and lucky guesses. We propose Collaborative Reasoning Distillation (CRD), a framework that enhances reasoning in compact models through three innovations: (1) interactive cross-feedback where teachers iteratively critique each other's reasoning, (2) fine-grained step-wise quality assessment capturing logical validity independent of final answers, and (3) coherence-aware step stitching that synthesizes complementary strengths. Students are trained via Reasoning Quality Optimization (RQO) with budget constraints. Our model, CRD-4B, achieves 97.3% on MATH-500 and 70.3% on AIME'25, surpassing baselines while using only 50K training examples, up to 12 times smaller than the datasets of comparable models.
Sources
- Demystifying Errors in LLM Reasoning Traces: An Empirical Study of Code Execution Simulation
- EXAONE Deep: Reasoning Enhanced Language Models
- xVerify: Efficient Answer Verifier for Reasoning Model Evaluations
- MAGDi: Structured Distillation of Multi-Agent Interaction Graphs Improves Reasoning in Smaller Language Models
- Evaluating Large Language Models Trained on Code
- Training Verifiers to Solve Math Word Problems
- The rising costs of training frontier AI models
- Process Reinforcement through Implicit Rewards
- MiCoTA: Bridging the Learnability Gap with Intermediate CoT and Teacher Assistants
- Specializing Smaller Language Models towards Multi-Step Reasoning
- A Comparison of DeepSeek and Other LLMs
- A Survey on LLM-as-a-Judge
- Train Long, Think Short: Curriculum Learning for Efficient Reasoning
- Measuring Mathematical Problem Solving With the MATH Dataset
- Distilling the Knowledge in a Neural Network
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code
- MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs
- Explanations from Large Language Models Make Small Reasoners Better
- Learning from Committee: Reasoning Distillation from a Mixture of Teachers with Peer-Review
- Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks