Bidirectional Preference Synthesis: Learning Prompt-Conditioned Preferences from Boundary Failures
cs.AI, cs.CL
Submitted: 2026-10-03
Updated: 2026-10-03
Comments: 18 pages, 3 figures
License: http://creativecommons.org/licenses/by/4.0/
The gist: Correction-based offline preference pipelines commonly treat model failures only as rejected responses under the original prompt.
Abstract
Correction-based offline preference pipelines commonly treat model failures only as rejected responses under the original prompt. This supervision is incomplete for boundary failures: responses that violate the given instruction yet coherently satisfy a nearby intent or constraint setting. We introduce Bidirectional Preference Synthesis (BPS), a data-construction method for standard Direct Preference Optimization (DPO) that makes this missing prompt dependence explicit. For each validated boundary failure, BPS keeps the conventional forward pair under the original prompt and adds a reverse pair under a synthesized achieved prompt, so the same response is rejected where it is wrong and chosen where it is right, without changing the DPO objective, training a reward model, or requiring online sampling. On Qwen3-4B-Instruct-2507, BPS preserves original-side pairwise ranking while raising achieved-side ranking accuracy from 6.8% to 62.3% on held-out crossed anchors, with a similar shift under a Kimi-K2.6 cross-teacher probe. A blind human audit supports the intended reverse preference direction, and downstream evaluations show the clearest separation from Forward-DPO in multilingual multi-turn instruction following, with consistent capability-retention patterns on agentic, tool-use, and code checks.
Sources
- Program Synthesis with Large Language Models
- $\tau^2$-Bench: Evaluating Conversational Agents in a Dual-Control Environment
- Evaluating Large Language Models Trained on Code
- Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators
- KTO: Model Alignment as Prospect Theoretic Optimization
- Multi-IF: Benchmarking LLMs on Multi-Turn and Multilingual Instructions Following
- Measuring Mathematical Problem Solving With the MATH Dataset
- RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback
- Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive
- Generalizing Verifiable Instruction Following
- ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs
- Qwen2.5 Technical Report
- GPQA: A Graduate-Level Google-Proof Q&A Benchmark
- Llama 2: Open Foundation and Fine-Tuned Chat Models
- Finetuned Language Models Are Zero-Shot Learners
- Qwen3 Technical Report
- $\tau$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains
- RRHF: Rank Responses to Align Language Models with Human Feedback without tears
- SLiC-HF: Sequence Likelihood Calibration with Human Feedback
- Instruction-Following Evaluation for Large Language Models
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems