PROOF-Gen: From Optimized Data to Better Distillation
cs.AI, cs.LG
Submitted: 2026-08-24
Updated: 2026-08-24
License: http://creativecommons.org/licenses/by/4.0/
The gist: Supervised fine-tuning on teacher-generated trajectories is the standard first stage for distilling tool-calling capabilities into deployable models.
Terminology
Abstract
Supervised fine-tuning on teacher-generated trajectories is the standard first stage for distilling tool-calling capabilities into deployable models. Post-training pipelines that drive shipped tool-calling agents re-run this stage on a daily or weekly cadence, paying the frontier-teacher cost each cycle, yet the mechanism is generate-and-filter (keep the teacher's passing trajectories, discard the rest) and each cycle leaves behind the same hard scenarios because failures supply no signal. On τ2-bench, 57% of teacher trials fail, two-thirds of them near-misses (most tool calls correct, undone by one decisive error). We introduce PROOF-Gen (Per-scenario Reflective Optimization to Overcome FailedGeneration), which recovers golden trajectories from these failures via per-scenario prompt optimization. For each failed task, a reflector analyzes the execution trace and evaluation feedback, then writes corrective guidance that steers the teacher to a passing trajectory. The guidance is stripped before training, so the student learns from clean demonstrations with no task-specific scaffold. On τ2-bench, per-scenario optimization recovers 93% of failed scenarios. Fine-tuned on the combined data, Qwen3-4B-Instruct-2507 improves from Pass 1=0.132 to 0.529 and Gemma 4 E4B-it gains +7.2pp on BFCL v4 multi-turn. In a deployed pipeline, the method lifts trajectory quality by +6.3pp goal completion and transfers to a deployed on-device model (+1.5pp goal completion; +1.7 to +5.0pp across response-quality metrics), with positive transfer in every locale (non-English average +1.48pp).
Sources
- GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning
- RL for Reasoning by Adaptively Revealing Rationales
- $\tau^2$-Bench: Evaluating Conversational Agents in a Dual-Control Environment
- FireAct: Toward Language Agent Fine-tuning
- Reinforced Self-Training (ReST) for Language Modeling
- DRP: Distilled Reasoning Pruning with Skill-aware Step Decomposition for Efficient Large Reasoning Models
- NaturalThoughts: Selecting and Distilling Reasoning Traces for General Reasoning Tasks
- AgentArk: Distilling Multi-Agent Intelligence into a Single LLM Agent
- Qwen3 Technical Report
- Agent-R: Training Language Model Agents to Reflect via Iterative Self-Training
- Scaling Relationship on Learning Mathematical Reasoning with Large Language Models
- Does RL Expand the Capability Boundary of LLM Agents? A PASS@(k,T) Analysis
- Agentic Context Engineering: Evolving Contexts for Self-Improving Language Models
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems