Know Before You Fetch: When Does Confidence-Based Retrieval-Budget Allocation Pay Off?

arXiv:2606.29959 · cs.IR, cs.CL · Submitted 2026-06-29 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.

Tom: We're looking at the paper: "Know Before You Fetch".

Paper summary: Tom: So we're looking at this paper, "Know Before You Fetch: When Does Confidence-Based Retrieval-Budget Allocation Pay Off?". The main idea here is that instead of just picking a fixed number of passages for every query, which wastes time if the answer is already in the model's memory and can even confuse things if the retrieved stuff isn't relevant, they are proposing adaptive RAG.

Jane: So it's moving away from that fixed retrieval amount to a system where the model decides whether to answer directly, grab a small context of one passage, get a full context of five passages, or just abstain entirely. The core contribution is this probability interface instead of just using some raw uncertainty signal as it is now <ref:2606.29959#pg6>.

Lu: It's about taking signals like sequence log-probability and prefix-logit uncertainty and calibrating them into actual probabilities of correctness for the closed-book answer, which they do by fitting a logistic calibrator to a scalar signal, like length-normalized sequence log-probability <ref:2606.29959#pg4>.

Meng: So they use these calibrated probabilities to decide on three budget levels: zero retrieval, compact retrieval with one passage, or full retrieval with five passages <ref:2606.29959#pg3>. That sounds like a lot of decision-making for every single query.

Tom: Exactly. They have graded allocation policies where they use two thresholds to define how much context to get based on that probability of correctness, meaning if the probability is high enough, they skip retrieval entirely <ref:2606.29959#pg3>.

Jane: And they look at performance across three datasets: TriviaQA, Natural Questions, and MS MARCO <ref:2606.29959#pg1>. The key empirical finding is that this calibration substantially improves the quality of the probability predictions while keeping the ranking of queries similar <ref:2606.29959#pg2>.

Lalam: I see how it relates to what we're doing with TARG prefix entropy, because they apply this graded calibrated allocation to TARG entropy and that improves the passage-budget and full-context frontier <ref:2606.29959#pg4>.

Tom: It's interesting because they also looked at cost and deployment trade-offs, revealing that while gating isn't always faster—it can actually increase latency by about twenty-seven percent on Qwen3 8B—it can save about eight percent on Qwen3 32B <ref:2606.29959#pg5>.

Paper summary: Jane: That means the cost of a binary gate is calculated as the cost of a closed-book answer plus five times the cost of retrieving one passage plus five times the cost of retrieving five passages <ref:2606.29959#pg5>. So it's only faster when that probability of getting zero retrieval is higher than that total retrieval time and token cost <ref:2606.29959#pg5>.

Lu: They found that graded allocation improves the full-context and passage-budget frontiers but it doesn't improve the retrieval call frontiers, which prevents you from thinking a one passage retrieval call is free <ref:2606.29959#pg5>.

Meng: So what does this mean for us in terms of deployment? If we want to save latency, we need to figure out the right break-even point between the decision overhead and the time spent actually fetching documents <ref:2606.29959#pg5>.

Tom: The authors suggest that calibrated probabilities give us a reusable interface for deciding between no retrieval, compact retrieval, full retrieval, or even abstention <ref:2606.29959#pg5>. It also helps clarify the difference between a simple retrieval call and managing the actual context budget <ref:2606.29959#pg5>.

Jane: So they conclude that this method is strongest when closed-book accuracy is meaningful, meaning the model actually needs to retrieve information, and when retrieval has enough room to rescue low-confidence queries <ref:2606.29959#pg4>.

Lalam: I think the main lesson here is that a single uncertainty signal isn't everything; you need calibrated probabilities that work well across different query types <ref:2606.29959#pg3>. It complements signals like TARG, but it gives us a concrete way to manage budget allocation <ref:2606.29959#pg4>.

Tom: So for someone just listening, the big picture is that this approach helps us decide when to spend the computational effort on fetching documents versus when we can trust our model's internal knowledge <ref:2606.29959#pg1>. It shows that adaptive retrieval can actually save latency under certain conditions <ref:2606.29959#pg5>.

Jane: The authors also mention some practical limitations, like calibration drifting when the model shifts to a new domain, so deployment requires ongoing monitoring <ref:2606.29959#pg4>. And they are mostly testing on short-answer QA where correctness can be judged using aliases <ref:2606.29959#pg5>.

Lu: And another thing is that the paper doesn't reproduce the full Self-RAG or Adaptive-RAG pipelines, so we should only read their comparisons as evidence for decision signals and not as a complete pipeline blueprint <ref:2606.29959#pg5>.

Paper summary: Meng: I agree with the point about deployment; if you're running this at scale, you have to model that latency increase on Qwen3 8B versus Qwen3 32B <ref:2606.29959#pg5>. We need those real traffic numbers, not just batch-one measurements <ref:2606.29959#pg5>.

Tom: So the final idea is that this calibrated probability interface gives us a way to choose between no retrieval, one passage, or five passages based on a calculated probability of getting the right answer <ref:2606.29959#pg5>. It makes the decision process explicit <ref:2606.29959#pg4>.

Jane: It seems like the title really sums it up: "Know Before You Fetch: When Does Confidence-Based Retrieval-Budget Allocation Pay Off?" it’s about making that decision consciously and not just letting a fixed system do the work <ref:2606.29959#pg1>.

Lalam: I'm pretty sure this framework is useful because it gives us a concrete way to manage the trade-off between retrieval calls and context tokens, which is something we need to track closely in production systems <ref:2606.29959#pg3>.

Tom: It’s about using these calibrated probabilities to make decisions about retrieval budget and abstention <ref:2606.29959#pg5>. It's a reusable interface that works with strong training-free signals like TARG <ref:2606.29959#pg4>.

Jane: So, if you're listening and you want to know what this means for the world, it’s that we might finally have a solid framework for deciding when a large language model should spend its resources on searching for information versus when it can just rely on what it already knows <ref:2606.29959#pg1>.

Lu: It changes how we think about RAG because instead of just blindly retrieving, we are learning to adapt the amount of evidence needed based on the model's own confidence in its answer <ref:2606.29959#pg3>.

Meng: From my side, it means we can start thinking about latency and token costs not just as fixed numbers but as variables that change based on how confident the model is before it even starts searching <ref:2606.29959#pg5>.

Tom: So, we're looking at calibrated retrieval-budget allocation to decide between no retrieval, compact retrieval, full retrieval, and abstention <ref:2606.29959#pg5>. It’s a concrete decision interface that I think is really important for shipping models <ref:2606.29959#pg4>.

Jane: And we'll keep watching how this holds up when the domain shifts, because the authors themselves admit calibration can drift under domain shift, which is something we need to watch out for in deployment <ref:2606.29959#pg4>.

Conclusion: Tom: So we're wrapping up this look at calibrated retrieval-budget allocation to decide between no retrieval, compact retrieval, full retrieval, or abstention.

Jane: It really boils down to a decision interface for when the AI should spend its resources searching for information versus when it can just rely on what it already knows.

Tom: The title itself says "Know Before You Fetch," which implies making that choice consciously rather than just letting a fixed system do the work.

Jane: Yeah, and it’s about using those calibrated probabilities to actually make that choice explicit, connecting the model's confidence directly to its retrieval strategy.

Lu: The core idea is that instead of just picking a number for how many passages to fetch, you calibrate signals like sequence log-probability into a real probability of getting the correct answer.

Meng: So it’s moving away from blindly fetching five documents and instead saying, "Based on how confident I am right now, I should probably only look at one passage."

Tom: Exactly. It shows that adaptive retrieval can actually save latency under certain conditions, though we saw it could actually increase latency on some smaller models.

Jane: The authors point out that this method is strongest when the model genuinely needs to retrieve information and when retrieval has enough headroom to rescue low-confidence queries.

Lu: And they also flag a big limitation—calibration can drift if the AI moves to a new domain, so you gotta keep watching it in production.

Meng: That’s what I worry about most; if the calibration drifts, we could start skipping retrieval for questions that actually need up-to-date evidence.

Tom: It’s a reusable interface that works with training-free signals like TARG, clarifying the difference between a simple retrieval call and managing the actual context budget.

Jane: So it gives us a concrete way to manage the trade-off between those retrieval calls and the cost of fetching tokens.

Lu: And it helps us expose those cost regimes where adaptive retrieval actually saves time, which is important for deployment planning.

Meng: Right, so for deployment, we need to figure out that break-even point between the decision overhead and the actual time spent retrieving documents.

Tom: This framework provides a way to decide between no retrieval, one passage, five passages, or abstention based on a calculated probability of getting the right answer.

Jane: It’s about making that decision conscious instead of relying on just one uncertainty signal to guide everything.

Lu: The authors conclude that this calibrated interface is valuable because it complements strong training-free signals while clarifying what we're actually measuring.

Meng: So, the big lesson is that we need to look at confidence not just as a score, but as a decision point for resource allocation.

Zhe Dong, Fang Qin, Manish Shah, Yicheng Wang

University of Maine at Presque Isle · Stanford University

cs.IR, cs.CL

Submitted: 2026-06-29

Updated: 2026-11-18

Comments: 25 pages, 6 figures, 13 tables

Code: https://github.com/dongzhe1/know-before-you-fetch

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 94/100

The gist: Retrieval-augmented generation typically retrieves a fixed number of passages for every query; this is wasteful when the reader already knows the answer…

Key concepts

Calibration of Uncertainty Signals
This involves mapping raw uncertainty signals, like sequence log-probability, into a meaningful probability of correctness. It uses a logistic calibrator to estimate P(closed-book answer correct) based on the signal's value. This turns an abstract signal into a usable metric for decision-making.
Graded Budget Allocation Policies
Instead of fixed retrieval amounts, this method uses policies based on the calibrated probability of correctness (pi). These policies define three context budgets: k=0 (no retrieval), k=1 (compact), or k=5 (full). Thresholds are used to decide which budget to use based on pi.
Probability Interface
The core contribution is using calibrated probabilities as a reusable interface for choosing between four options: no retrieval, compact retrieval, full retrieval, or abstention. This interface complements other signals like TARG and helps clarify the difference between context budget and actual retrieval calls.

Terminology

Summary

Retrieval-augmented generation typically retrieves a fixed number of passages for every query; this is wasteful when the reader already knows the answer, and it can be harmful when irrelevant or partially relevant passages distract the reader <ref:2606. p. 029959#pg2> The contribution is a probability interface rather than a new raw uncertainty signal <ref:2606. p. 029959#pg6>.

Calibration of Uncertainty Signals

The method formulates adaptive RAG as calibrated retrieval-budget allocation by calibrating sequence log-probability and prefix-logit uncertainty signals into probabilities of correctness <ref:2606.29959#pg4>. This calibration is achieved by fitting a logistic calibrator to a scalar signal, such as length-normalized sequence log-probability, to map it to P(closed-book answer correct) using the formula pi = Pr(y(0)i = 1 si) = σ(αsi + β) <ref:2606. p. 129959#pg5>. Two calibration protocols are employed: diagnostic out-of-fold estimates on the evaluation pool, and train/validation/test splits for deployment experiments <ref:2606. p. 229959#pg5>.

Graded Budget Allocation Policies

The paper introduces policies for context selection based on the calibrated probability of correctness, allowing for three context budgets: k = 0 (closed-book answer), k = 1 (compact retrieval), or k = 5 (full retrieval) <ref:2606.29959#pg3>. A graded policy uses two thresholds, a policy skips retrieval when pi ≥ τ and otherwise retrieves k = 5, and a graded policy uses two thresholds to define ai as 0, 1, or 5 based on pi. p. 2 The key accounting correction is that compact retrieval reduces passage budget and full-context usage but is still a retrieval call <ref:2606. p. 329959#pg5>.

Performance Across Datasets

The central empirical conclusions show that calibration substantially improves probability quality while preserving ranking, which is crucial because the paper’s claim is about probabilistic decisions, not merely AUROC <ref:2606.29959#pg2>. Graded allocation improves full-context and passage-budget frontiers but not retrieval-call frontiers, preventing the misleading claim that k = 1 retrieval is free <ref:2606. p. 129959#pg5>. The framework is complementary to TARG prefix entropy, applying graded calibrated allocation to TARG entropy improves its passage-budget and full-context frontier <ref:2606. p. 129959#pg4>.

Cost and Deployment Trade-offs

A realistic two-stage cost model reveals that gating is not universally faster; it increases latency by about 27% on Qwen3-8B but saves about 8% on Qwen3-32B <ref:2606.29959#pg5>. The cost of a binary gate is Cbinary = cCB + p5(cret,5 + cOB,5), so it is faster only when Pr(k = 0) > cCB (ret,5 + cOB,5) <ref:2606. p. 329959#pg5>. The measured cost regimes show that for Qwen3-8B, always-RAG costs 107. p. 65 ms per query, while binary gating costs 136. p. 61 ms and graded gating costs 133. p. 63 ms <ref:2606. p. 029959#pg5>.

Conclusion on Adaptive RAG

The main lesson is that calibrated probabilities provide a reusable interface for deciding among no retrieval, compact retrieval, full retrieval, and abstention <ref:2606.29959#pg5>. This interface complements strong training-free signals such as TARG, clarifies the distinction between retrieval calls and context budget, and exposes the cost regimes in which adaptive retrieval truly saves latency <ref:2606. p. 029959#pg5>. The method is strongest when closed-book accuracy is nontrivial, retrieval has enough headroom to rescue low-confidence queries, and retrieval can also harm high-confidence queries <ref:2606. p. 829959#pg4>.

Extended Method Details

A.1 Prompting and decoding involves asking the reader to answer the question directly in a short phrase for closed-book prompts, while open-book prompts prepend retrieved passages and ask for the final answer <ref:2606. p. 2 p. 229959#pg5>. Main generations use greedy decoding with a short answer length cap, and sequence log-probability is length-normalized over generated answer tokens to avoid mechanically favoring short outputs <ref:2606. p. 129959#pg5>.

Additional Analyses and Limitations

Figure 7 reproduces the PopQA motivation showing that parametric recall is higher for popular entities, but retrieval can still rescue tail entities <ref:2606.29959#pg4>. The method is strongest when three conditions hold: closed-book accuracy is nontrivial, retrieval has enough headroom to rescue low-confidence queries, and retrieval can also harm high-confidence queries <ref:2606. p. 829959#pg4>. Most experiments are on short-answer QA where correctness can be judged with aliases <ref:2606. p. 929959#pg5>. Calibration can drift under domain shift, so deployment requires monitoring. p. 9

Ethical Considerations

The main risk is over-trusting calibrated confidence outside the domain on which it was validated. Miscalibration could lead to skipped retrieval for questions that require up-to-date evidence, or unequal abstention rates across user groups and topics. p. 9 The experiments use public datasets and do not introduce new human-subject data. p. 1

Conclusion

We presented calibrated retrieval-budget allocation for RAG. The main lesson is not that a single uncertainty signal wins everywhere, but that calibrated probabilities provide a reusable interface for deciding among no retrieval, compact retrieval, full retrieval, and abstention. p. 0 This interface complements strong training-free signals such as TARG, clarifies the distinction between retrieval calls and context budget, and exposes the cost regimes in which adaptive retrieval truly saves latency. p. 0 The operational skip rate therefore grows from almost zero to 54% on TriviaQA. p. 5

A.1 Prompting and decoding involves asking the reader to answer the question directly in a short phrase for closed-book prompts, while open-book prompts prepend retrieved passages and ask for the final answer <ref:2606.29959#pg5>. Main generations use greedy decoding with a short answer length cap, and sequence log-probability is length-normalized over generated answer tokens to avoid mechanically favoring short outputs. p. 1

Answer Evaluation and Label Audit

Correctness uses normalized string matching and alias sets when available, and manual audit identified a small number of semantic-equivalence and yes/no errors, especially in MS MARCO. Corrected tables are included as sensitivity artifacts, but corrections do not change qualitative conclusions; the largest numerical change is MS MARCO AUROC, which increases by about two percentage points after fixing obvious label errors. p. 4

Retriever Quality and Corpus Scale

Table 6 shows that absolute RAG accuracy depends strongly on retriever quality, and the full-Wikipedia condition is much harsher than the per-query pool, especially for NQ. This suggests that confidence remains predictive of reader knowledge, but no gating method can compensate for a very noisy retriever if the goal is maximum answer accuracy. p. 7 The Wiki index shows lower retrieval success rates compared to the BGE-large pool. p. 3

Decision-Signal Baselines

Figure 6 compares against Self-RAG trigger and Adaptive-RAG-inspired decision signals under a fixed reader, isolating the retrieve/skip decision. Confidence-based decisions are stronger under this matched protocol, but this is not a full Self-RAG or Adaptive-RAG pipeline reproduction. p. 8

How to Deploy

A practical deployment should first decide its dominant cost: retrieval calls, context tokens, latency, or monetary token price. It should then calibrate on a small validation set, select thresholds on the relevant budget axis, and verify the break-even condition in Equation 7. p. 9 For high-stakes domains, retrieval may be mandatory even for high-confidence queries. p. 9

Limitations

Most experiments are on short-answer QA where correctness can be judged with aliases, and long-form answers require evidence attribution and human evaluation. The measured latency numbers are batch-1 and hardware-specific; production batching, asynchronous retrieval, and KV-cache reuse may shift the break-even point. p. 9 We do not reproduce full Self-RAG or Adaptive-RAG pipelines, and our comparison to those methods should be read as decision-signal evidence only. p. 9 Finally, calibration can drift under domain shift, so deployment requires monitoring. p. 9

References

Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-RAG: Learning to retrieve, generate, and critique through self-reflection. p. 2

Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. p. 2 Weinberger. On calibration of modern neural networks. p. 2

Soyeong Jeong, Jinheon Baek, Sukmin Cho, Sung Ju Hwang, and Jong C. p. 1 Park. Adaptive-RAG: Learning to adapt retrieval-augmented large language models through question complexity. p. 1

Zhengbao Jiang, Frank F. p. 1 Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig. Active retrieval augmented generation. p. 1

Mandar Joshi, Eunsol Choi, Daniel S. p. 1 Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. p. 3

Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston.

Vladimir Karpukhin et al. p. 1 Dense passage retrieval for open-domain question answering. p. 1

Patrick Lewis et al. p. 1 Retrieval-augmented generation for knowledge-intensive NLP tasks. p. 0

Llama Team, AI @ Meta. p. 3 The Llama 3 herd of models. p. 1

Alex Mallen et al. p. 1 When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. p. 1

Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. p. 1 MS MARCO: A human generated machine reading comprehension dataset. p. 4

Qwen Team. Qwen3 technical report. p. 0

Qwen Team. Qwen3.5: Towards native multimodal agents, February 2026. p. 0

Yufeng Wang, Lu Wei, and Haibin Ling. p. 0 p. 0 Retrieval as a decision: Training-free adaptive gating for efficient RAG. p. 1 p. 1

Shitao Xiao, Zheng Liu, Peitian Zhang, Niklas Muennighoff, Defu Lian, and Jian-Yun Nie. p. 3 C-Pack: Packed resources for general chinese embeddings.

Yufeng Wang, Lu Wei, and Haibin Ling. Retrieval as a decision: Training-free adaptive gating for efficient RAG.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Calibrated Retrieval-Budget Allocation allows for graded context selection by mapping sequence log-probability to probabilities of correctness, enabling a decision policy that selects between closed-book (k=0), compact retrieval (k=1), full retrieval (k=5), or abstention based on calibrated confidence.

  2. The system can implement selective RAG by using the confidence of the chosen answer—either the calibrated closed-book probability or a separate calibration of the open-book answer’s log-probability—to yield a risk-coverage curve, allowing it to remove its likely errors early.

  3. The AI system can dynamically adjust retrieval based on cost regimes, as gating is only faster when the skip rate clears a measurable break-even threshold, optimizing the trade-off between latency and accuracy for specific model sizes.

Sources

Related papers