Iterate Until Retrieved: Factual Nugget Optimization for Discoverable Continual Corrections in Agentic RAG
summary
The gist
The gist Iterative Nugget Optimization (INO) is an index-time optimization method that uses production agentic RAG as a test harness to iteratively refine factual nuggets from user feedback until
In short
Iterative Nugget Optimization (INO) is an index-time method that refines factual corrections from user feedback into 'factual nuggets.' It uses a production agentic RAG system as a test harness to iteratively improve these nuggets until they are discoverable and optimized for future queries. This process ensures high usage of corrected information.
Key concepts
- Iterative Nugget Optimization (INO)
- INO is an optimization loop that refines factual nuggets by testing them against a production RAG system. It starts with a correction, tests if the nugget is used, and if not, uses an LLM to revise the nugget's wording until it becomes discoverable for future queries.
- Factual Nugget
- A factual nugget is a concise piece of information derived from user feedback that needs to be indexed. INO treats its creation as an optimization problem, aiming to make this nugget highly usable and retrievable by the RAG system.
- Agentic RAG System
- This is the production system used as a test harness for INO. It simulates how real users interact with the indexed information. The method uses this system to determine if a nugget is actually retrieved and used, guiding the refinement process.
Terminology used across episodes
This episode discusses
- Iterate Until Retrieved: Factual Nugget Optimization for Discoverable Continual Corrections in Agentic RAG · Paper Radio
- jina-embeddings-v5-text: Task-Targeted Embedding Distillation
- Feedback Adaptation for Retrieval-Augmented Generation
- RAG4Tickets: AI-Powered Ticket Resolution via Retrieval-Augmented Generation on JIRA and GitHub Data · Paper Radio
- KARL: Knowledge Agents via Reinforcement Learning
- M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation
- Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory
- Llama2Vec: Unsupervised Adaptation of Large Language Models for Dense Retrieval
- Document Expansion by Query Prediction
- Optimizing and Evaluating Enterprise Retrieval-Augmented Generation (RAG): A Content Design Perspective
- A-MEM: Agentic Memory for LLM Agents
The paper
Iterate Until Retrieved: Factual Nugget Optimization for Discoverable Continual Corrections in Agentic RAG · Read on arXiv
Mosaic AI · The Hebrew University of Jerusalem
Agentic retrieval-augmented generation (RAG) systems in business-to-business settings may often receive free-form response feedback. Rather than generic feedback signals such as style, preference, or overall response quality, we focus on actionable factual corrections. We identify these instances and convert them into compact knowledge-base entries, which we call factual nuggets. We introduce Iterative Nugget Optimization (INO), an index-time optimization method that uses the production agentic RAG as a test harness: it creates an initial nugget, probes it with the triggering query and paraphrases, reflects over failed retrieval and answer traces, and revises the nugget until it is discoverable. We evaluate INO with two production B2B knowledge-assistance agents across multiple companies that use our system: a product support agent that answers questions over company-specific knowledge bases, and a support ticket agent that assists support engineers. INO consistently improves results over baselines in terms of discoverability and usage of factual corrections, in automated and human evaluations. A controlled experiment in production shows that the nuggets improve user satisfaction with responses.
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "Iterate Until Retrieved".
Jane: The gist Iterative Nugget Optimization (INO) is an index-time optimization method that uses production agentic RAG as a test harness to iteratively refine factual nuggets from user feedback until they…
Tom: First, who's behind it and why it matters.
Title and authors: Tom: So we’re diving deeper into the mechanics of this Iterative Nugget Optimization for Discoverable Continual Corrections in Agentic RAG. The authors are Moshe Hazoom, Gal Patel, and Alon Talmor from Mosaic AI and The Hebrew University of Jerusalem >
Jane: Those are the folks who did the heavy lifting on this research. They’re looking at how agentic retrieval-augmented generation systems in business settings handle free-form feedback >
Lu: Their core contribution is using the production agentic RAG system itself as a test harness to iteratively refine these factual nuggets until they become discoverable and optimized for future queries >
Meng: So they're not just doing a one-time indexing job; they're building a loop that tests the nugget, probes it with queries, and reflects over failures >
Tom: Right. They start with an initial nugget based on the feedback, then they generate some test queries—a triggering query and some paraphrases—and if that nugget doesn't get used for those or probe queries, an LLM reflector steps in to revise the representation >
Jane: It’s like a quality control loop specifically for knowledge that needs to be reusable and standalone >
The paper's summary: Tom: So the summary boils down to this: they take user feedback, filter it down to actionable factual corrections, structure those into nuggets, and then run them through the production RAG stack in a loop until they become discoverable for relevant future traffic >
Jane: The key is that the nugget has to be encoded conservatively so it doesn't introduce wrong knowledge but still have enough context to be reusable and standalone >
Lu: They explicitly point out that a straightforward approach of just inheriting the wording and context from the original interaction often ends up being too narrow because future users ask from different roles or use different company vocabulary >
Meng: So they are trying to solve the problem where a correction is only useful for one specific way of asking a question, but they want it to be useful for many ways >
Tom: Precisely. The goal is to avoid the nugget becoming so broad that it gets retrieved for completely unrelated traffic, which is a big risk in large knowledge bases >
Jane: And they show how this iterative process works by creating candidate nuggets and running them through the same agentic RAG system that will serve future users >
The paper's improvements: Tom: Now let’s talk about what they actually suggest improving. They compared five different strategies for nugget construction to see which one optimizes discoverability and usage of factual corrections best >
Jane: They looked at things like a standard nugget, a trigger-query anchor, synthetic query anchors, and combinations of those >
Lu: One interesting strategy is the "Synthetic query anchors" where an LLM generates five short future-user queries that the nugget should answer and appends them to the nugget >
Meng: That seems ambitious because generating five queries for every correction adds overhead. What makes one better than another in terms of what actually helps retrieval?
Tom: The results showed that all these strategies consistently improved results across retrieval, citation, and answer feedback compliance compared to the standard nugget baseline >
Jane: But they found that using "Trigger plus synthetic anchors" seems particularly strong because it combines a paraphrase of the original query with four other queries >
Conclusion: Tom: So to wrap up on this paper, Iterative Nugget Optimization for Discoverable Continual Corrections in Agentic RAG shows that you can optimize the artifact itself rather than just relying on its initial creation >
Jane: It proves that by putting the nugget construction into an index-time optimization loop, we can ensure these factual corrections are actually discoverable and usable for future, different types of queries >
Lu: The implication is that we need a more dynamic way to think about how knowledge is indexed in RAG systems so it adapts to evolving user language without manual intervention >
Meng: From a deployment standpoint, the cost they report for this optimization process is about zero point three one per correction and under two minutes per nugget in setup which makes it relatively efficient for running at scale >
Tom: So we’ve seen how they use the production agentic RAG system as a test harness to make sure these nuggets are discoverable, and it’s a practical method that can be run offline with a small cost per correction >
Jane: It gives us a solid blueprint for handling the inevitable stream of factual corrections in complex B2B settings where knowledge bases are constantly being updated >
More episodes
- 2609.19969-DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression
- 2309.01196-A Visual Interpretation-Based Self-Improved Classification System Using Virtual Adversarial Training
- 2504.20829-GaussTrap: Stealthy Backdoor Attacks on 3D Gaussian Splatting for Targeted Scene Misperception
- 2506.00060-Comparative analysis of privacy-preserving open-source LLMs regarding extraction of diagnostic information from clinical CMR imaging reports
- 2506.10974-AutoMind: Adaptive Knowledgeable Agent for Automated Data Science
- 2507.14056-Dynamic gain neuromodulation attenuates stability gaps during continual joint training
- 2508.03411-SlotMatch: Distilling Object-Centric Representations for Unsupervised Video Segmentation
- 2509.09009-Open-sci-ref-0.02: open reference baselines for reproducible language model and dataset comparison
- 2512.14554-VLegal-Bench: Cognitively Grounded Benchmark for Vietnamese Legal Reasoning of Large Language Models
- 2601.10960-Steering Language Models Before They Speak: Logit-Level Interventions