FutureBench: A Renewable Framework for Evaluating Knowledge-Context Conflict in LLMs

arXiv:2611.23601 · cs.CL, cs.AI · Submitted 2026-11-17 · Read on arXiv

cs.CL, cs.AI

Submitted: 2026-11-17

Updated: 2026-11-17

License: http://creativecommons.org/licenses/by/4.0/

The gist: Asking a model to check retrieved text can also tell it which source to trust.

Abstract

Asking a model to check retrieved text can also tell it which source to trust. We introduce FutureBench, a renewable framework for evaluating knowledge-context conflict using timestamped questions, controlled passage triplets, and no-context probes. Across 3,629 questions and six models, Verify (a prompt combining checking with passage priority) raises false-answer adoption in all five models tested on pre-cutoff events, mainly by displacing answers that were correct under Baseline, which allows both passage evidence and model knowledge. On post-cutoff events, only o4-mini increases adoption (+13.7 percentage points). A four-prompt comparison on 902 post-cutoff questions finds that passage priority without checking raises adoption, while moving to Verify lowers it in all five models. Adversarial-Aware, which warns of misleading passages and prioritizes model knowledge, restores 82.5-98.9% of displaced pre-cutoff answers with under one point of change in true-passage accuracy. FutureBench supports repeated evaluation on fixed cohorts and extension to new events as models change.

Related papers