Learning When to Trust via Selective Context Preference Optimization

The paper addresses a critical failure mode in large language models: their susceptibility to misleading contextual signals. The authors observe that a model that solves a problem cleanly will often abandon its correct answer the moment the prompt carries a plausible but wrong suggestion. They note that such cues steer predictions without being faithfully reflected in the model's stated reasoning, citing work on unfaithful chain-of-thought explanations and preference-based alignment that can make matters worse by rewarding agreement with the user over truth.

The authors critique the instinctive remedy of training models to resist misleading context, arguing that "resistance hides a failure mode. A model that simply ignores all external context looks robust whenever evaluation only contrasts clean prompts with prompts carrying wrong hints. A genuinely useful reasoning model, however, must do three things at once: reject a misleading signal without sacrificing clean reasoning, benefit from context that is actually correct, and stay unmoved by irrelevant context." They frame the problem as selective trust — models should be judged on selective trust, not on resistance alone.

The authors introduce MIST (Misleading Signal Testbed), a human-annotated benchmark that renders each reasoning item under four matched conditions (clean, misleading, correct-context, and irrelevant-context). Specifically:

1. Clean: the original task with no added context. It must be answerable on its own.

2. Misleading: the same task plus natural context that explicitly or strongly implies a plausible wrong answer.

3. Correct-context: matched context with the same style, source type, and approximate confidence level as the misleading context, but pointing to the gold answer.

4. Irrelevant: matched context with similar length and source framing but no answer-bearing information.

The benchmark contains 1,000 source items: 800 adapted from existing question-answering (QA), math, and reasoning benchmarks, and 200 written by annotators as longer, deliberately challenging scenarios, yielding 4,000 matched condition rows. The adapted subset draws on StrategyQA, CSQA, ARC, GSM8K, MMLU, OBQA, MATH, AQuA, and SVAMP. Because the question, answer space, and gold answer are held fixed across all four versions, any change in accuracy is attributable to the added signal rather than to task difficulty.

The benchmark is built by STEM-trained annotators through four stages: source pooling, manual screening, human annotation, and review before freezing. Misleading contexts are balanced across nine source channels (including answer keys, retrieved documents, lecture notes, TA notes, forum posts, search snippets, calculator notes, user beliefs, and other) and wrong answers span seven plausibility classes (near option, misread condition, arithmetic slip, off-by-one, common misconception, overgeneralization, entity confusion).

The authors argue that "Aggregate accuracy alone, however, blurs a model's underlying ability with its susceptibility to signals: a model can post a high score simply because it is strong on the clean questions, even as misleading context quietly overturns answers it would otherwise get right." They therefore propose SC2W (signal-induced correct-to-wrong), a paired metric counting how often a misleading signal flips a clean-correct answer to wrong, defined as:

SC2W = P(I[a clean = 1 ∧ a mis = 0]) / P(I[a clean = 1])

It conditions on the items a model already solves when clean and measures how often a misleading signal flips them to wrong, separating signal-induced failures from questions the model cannot answer.

The authors propose SCOPE (Selective Context Preference Optimization), a signal-counterfactual preference framework that mines the failures that matter: items a base model answers correctly on its own but gets wrong once a misleading signal is added. For each mined item, we form a single counterfactual response pair, i.e., a correct full-completion response r+ whose reasoning and final answer agree, and the base model's signal-following wrong full-completion response r−, paired with the four matched context prompts.

The key design principle is balance: rather than train on the misleading pairs alone, which merely teaches blanket distrust, we weight them equally against the clean, correct-context, and irrelevant-context controls. The unified objective is:

LSCOPE = λm Lmis + λc Lclean + λp Lnonadv

where Lnonadv = ρLcor + (1−ρ)Lirr, and the headline configuration uses (λm, λc, λp, ρ) = (0.25, 0.25, 0.50, 0.50), which assigns 25% sampling mass to each of Dmis, Dclean, Dcor, and Dirr. Crucially, the novelty lies entirely in the matched, balanced preference data rather than the optimizer — the method uses standard full-completion DPO with the loss unchanged.

The authors emphasize leakage controls: gold labels are used for scoring and filtering, but hidden gold explanations are not inserted into training prompts or retained in chosen completions. The chosen response r+ is either that base model's clean-condition correct response when available; otherwise fall back to a correct diagnostic response generated by the same frozen base checkpoint, with filtering to remove responses mentioning privileged-note provenance or answer-key/gold-solution wording.

Across 23 frontier and open-weight models, a single misleading signal costs 17.1 points on average. The benchmark study shows Susceptibility is widespread: GPT-5.5 has nonzero SC2W, and strong open-weight models stay vulnerable even at high clean accuracy. Proprietary models show substantial SC2W values (e.g., GPT-5.5 at 10.5, Claude Opus 4.7 at 24.6, Gemini 3.1 Pro at 12.9), while open-weight models are worse (e.g., Qwen2.5-1.5B at 51.7, SmolLM2-1.7B at 48.3, Qwen3-4B at 35.0).

On the two trainable bases, our method cuts SC2W from 35.0 to 16.3 on Qwen3-4B and from 31.5 to 20.6 on Llama-3.2-3B while preserving or improving the clean, correct-context, and irrelevant-context controls. The method comparison shows that several defenses raise misleading-context accuracy, but our method attains the best Overall accuracy and the strongest control preservation on Qwen3-4B. On Llama-3.2-3B, Standard-DPO nudges misleading-context accuracy up yet collapses correct-context accuracy, and OPSD is negative overall. The authors conclude: Our approach alone is the stable cross-family mitigation.

SCOPE repairs 188 of the 331 Qwen3-4B failures triggered by misleading context, with 182 net repairs overall and achieves an 11.1× gain-to-cost ratio (+200 gains vs. −18 costs). It improves misleading-context accuracy while preserving performance on clean, correct-context, and irrelevant-context conditions, and held-out SC2W is lower at the later reported checkpoints.

"Unmatched or random pairs substantially degrade misleading accuracy, overall accuracy, and SC2W, showing that matched signal-counterfactual pairing is load-bearing. Removing the clean, correct-context, or irrelevant-context pairs weakens control preservation and overall balance: reduced-control variants can raise raw resistance but sacrifice controls."

The learned behavior transfers to three external benchmarks (GSM-IC, GSM-Plus, and Sharma-style sycophancy examples): trained only on the matched pool, our method is best or tied on all four higher-is-better metrics for both families. The authors conclude: "Standard-DPO attains the lowest Sharma Bias but loses accuracy on several tasks, whereas our method improves or preserves external correctness while keeping bias-following comparable to the base models. It transfers by learning when to rely on context, not by ignoring it."

A reference-assisted human scoring-audit confirms that automatic win ratios align with automatic win ratios across all six main metrics, with five have Spearman correlations of 0.97 or higher, and SC2W remains positive (ρ = 0.821).

The authors conclude: "Robustness to misleading context is the wrong target: under single-condition evaluation, a model that distrusts every signal looks identical to one that judges signals on their merits, yet only the latter is useful... By changing only what enters a standard DPO objective, namely matched and balanced signal-counterfactual pairs, SCOPE roughly halves SC2W on two model families while preserving clean, correct-context, and irrelevant-context accuracy and transferring zero-shot to external benchmarks. Reliable reasoning models should be trained and judged on selective trust, not resistance alone."

The stated limitations: "Our benchmark is a controlled, text-only diagnostic: its rates measure susceptibility, not deployment prevalence, and our two-family mitigation leaves broader architectures and scales untested. Robustness under matched prompts is not evidence of chain-of-thought faithfulness, and as most items are adapted from public benchmarks, contamination cannot be fully excluded, though conditioning SC2W on clean-correct behavior mitigates it."

Based on the paper, I can improve AI systems in the following specific ways:

Improvement: Replace standard preference optimization (which often rewards agreement with user-provided hints) with a signal-counterfactual preference framework. The training data must include, for each reasoning question, four matched context conditions:

• clean (no context)

• misleading (plausible wrong hint)

• correct (hint pointing to the right answer)

• irrelevant (no answer-bearing info)

What the improved AI system can do: It learns to reject a misleading hint without ignoring useful context. It will:

• Stick with its correct answer when a prompt contains a convincing but wrong suggestion.

• Follow a correct hint and improve or maintain performance when context is actually right.

• Remain unaffected by irrelevant context.

• Avoid the failure mode of simply distrusting all external signals, which would collapse correct-context performance.

What the improved AI system can demonstrate: A model can no longer appear robust simply by being strong on clean questions. The improved system is explicitly optimized and evaluated on whether misleading context changes correct answers into wrong ones, separating signal-induced failure from general task difficulty.

Language models increasingly condition their answers on external signals, and a single misleading one can turn a correct answer wrong. The obvious remedy, training models to resist such signals, hides a failure mode: a model that ignores all context looks robust yet is useless when the context is worth trusting. We recast the problem as selective trust and introduce MIST, a human-annotated benchmark that renders each reasoning item under four matched conditions (clean, misleading, correct-context, and irrelevant-context), together with SC2W, a paired metric counting how often a misleading signal flips a clean-correct answer to wrong. Across a comprehensive benchmark study, we observe that such a susceptibility is universal. We then propose SCOPE, which mines clean-correct/misleading-wrong failures and optimizes a standard Direct Preference Optimization (DPO) objective over matched preference pairs balanced equally across all four conditions, rather than over misleading items alone. Our approach substantially reduces SC2W on popular open-sourced models while preserving accuracy when the added context is clean, correct, or irrelevant. With this work, we argue that models should be judged on selective trust, not on resistance alone.