Function-Level Execution Feedback for Code Preference Optimization
cs.AI, cs.SE
Submitted: 2026-08-23
Updated: 2026-08-23
Comments: 20 pages, 8 figures, 14 tables. Accepted to Findings of the Association for Computational Linguistics: EMNLP 2026
Code: https://github.com/inechnech/STEP-KTODER
License: http://creativecommons.org/licenses/by/4.0/
The gist: Process supervision has improved mathematical reasoning, where intermediate steps are naturally expressed as chains of thought.
Terminology
Abstract
Process supervision has improved mathematical reasoning, where intermediate steps are naturally expressed as chains of thought. In code generation, however, process supervision remains underexplored because there is no standard notion of a step. Supervision can target lines, reasoning traces, or program states, making it unclear what to label and optimize. We propose STEP-KTODER, a framework for code preference optimization that defines steps as module-level functions in decomposed multi-function programs and assigns binary correctness labels via automatically generated unit tests. Our method provides a code-specific instantiation of stepwise KTO, combining function-level process supervision with outcome-level feedback on the full program. We evaluate on HumanEval(+), MBPP(+), BigCodeBench, and LiveCodeBench, showing that STEP-KTODER improves over outcome-only KTO and DPO. Further analysis shows that execution-based labels are essential: LLM-as-a-judge annotations systematically over-predict function failures, corrupt positive step labels, and degrade downstream preference optimization. Code is available at: https://github.com/inechnech/STEP-KTODER.
Sources
- Program Synthesis with Large Language Models
- Efficient Training of Language Models to Fill in the Middle
- Evaluating Large Language Models Trained on Code
- Training Verifiers to Solve Math Word Problems
- Process Supervision-Guided Policy Optimization for Code Generation
- DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence
- Qwen2.5-Coder Technical Report
- TACO: Topics in Algorithmic COde generation dataset
- Improve Mathematical Reasoning in Language Models by Automated Process Supervision
- MoT: Modularization-of-Thought Prompting for Effective Code Generation
- Solving math word problems with process- and outcome-based feedback
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems