Found in Conversation: LLMs Teach Themselves to Close the Multi-Turn Gap
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "Found in Conversation".
Jane: Here's the gist of the paper. Large Language Model (LLM) interactions are typically underspecified, with users clarifying all necessary details across multiple conversational turns <ref:2605.24432#pg2>.
Tom: Here we introduce FOUND IN CONVERSATION (FIC), a training framework where a model teaches itself to find and recover its single-turn competence given underspecified multi-turn prompts <ref:2605.24432#pg2>.
Jane: The Core Problem Large Language Models (LLMs) reliably produce high-quality responses when given a fully-specified, single-turn prompt <ref:2605.24432#pg2>.
Tom: However, real users seldom interact this way <ref:2605.24432#pg2>.
Title and authors: Tom: So we're looking at this paper called "Found in Conversation: LLMs Teach Themselves to Close the Multi-Turn Gap." Basically, it tackles a problem where users usually give models all the details at once, but real life involves back-and-forth conversations where things get underspecified.
Jane: Right. So, the core issue they're pointing out is that when you have a full prompt upfront, the AI performs really well with all that information in one go. But in multi-turn chats, even with the same total information spread out across several turns, those models start losing their performance on tasks compared to when everything was given at the start of time.
Lu: The paper introduces something called FOUND IN CONVERSATION or FIC, which is a whole training framework designed to help a model figure out how to get back to that original single-turn competence even when it's in a multi-turn conversation.
Meng: So, it’s not about just making the model better at chatting; it’s about fixing this fundamental disconnect where the model knows how to do something great in isolation but struggles when things are spread out over time.
Lalam: The core idea they use is something called View-Asymmetric Self-Distillation or VASD, which uses two different ways of looking at the same problem information to teach the model its multi-turn skills.
Tom: Let me break down what that means in terms of training. They’re doing a supervised finetuning warm-start first, which basically teaches the model how to defer or ask for clarification when it doesn't have all the necessary details yet.
Jane: That sounds like they are trying to teach the model good conversational habits, so it doesn't just jump in and guess an answer before it has everything it needs. It’s about preventing that premature answer commitment.
Lu: Exactly. The second stage is this VASD, where they run the same model backbone on two views: a teacher view that sees the single-turn setting, and a student view that sees the multi-turn setting with intermediate responses sampled randomly over time.
Meng: So, they're using a token-level loss to align those two views so the student learns from what it thinks is the teacher's strong, single-turn behavior. That’s a concrete way to transfer that strength into the weaker multi-turn setting.
Lalam: And across various models, they report that this FIC framework recovers at least ninety-two percent of single-turn performance on benchmarks like Llama, Qwen, Phi and OLMo <ref:2605.24432#pg1,Llama, Qwen, Phi and OLMo>.
Tom: Ninety-two percent is a solid number, especially since they managed to do this without needing a stronger external teacher that just doesn't exist for these frontier models. They also found that the training makes conversations more efficient too, with up to thirty-three percent fewer tokens used <ref:2605.24432#pg3>.
Jane: That efficiency point is interesting because it suggests the model isn't just guessing; it’s actually learning to be smarter about when to ask for what it needs instead of wasting words on long, ungrounded guesses.
Lu: The paper also showed that these trained checkpoints transfer well to other tasks, like database and actions tasks, with very little extra training needed, which is a strong sign that they’ve captured something more general than just the conversational skill itself.
Meng: From a deployment standpoint, if this works across such diverse backbones and still keeps performance high while cutting tokens by thirty-three percent, that makes it way more interesting for us to think about in production.
Tom: The paper also highlights some improvements they suggest for future work. They suggest adding inference-time interventions like PROMPT-RULE or PROMPT-SELFCHECK to force the model to pause and verify details before it commits to an answer.
Jane: So, the idea is that after training, we can use these rules during actual conversation to make sure the model doesn't forget what it needs and just starts rambling.
Lu: They also mentioned that this framework only addresses one specific type of difficulty: underspecification. They have to be careful because real-world conversations have other kinds of problems that FIC might not solve as well, like changing goals or retracting information mid-chat.
Meng: That’s a fair limitation. If the user simulator is perfectly cooperative and never changes its mind, then yes, this framework focuses on that gap very tightly.
Lalam: But they also noted that because FIC instills grounded deferral behavior instead of just letting the model hallucinate when it's missing data, these resulting conversations are generally more efficient and better grounded for the listener.
Tom: So to wrap up, this paper calls FOUND IN CONVERSATION a framework that uses supervised fine-tuning to build in deference and then uses View-Asymmetric Self-Distillation to transfer single-turn strength into multi-turn behavior, recovering ninety two percent of performance across several models.
Jane: It's about closing that behavioral gap from the inside of the model itself, showing that a model's competence is there but just not surfacing correctly in complex conversations.
Lu: The implication is that we don't necessarily need a completely new architecture to fix this; we just need better training techniques to align the model’s internal capabilities with how humans actually interact.
Meng: It seems like a low-compute way to get a significant boost in conversational quality and efficiency, which is exactly what production environments need.
Tom: Yeah, it’s showing that we can make models more robust in these messy, real-world interaction settings by teaching them how to manage missing information better.
The paper's summary: Tom: So we’ve talked about how models usually choke when conversations get messy and underspecified, and this paper is introducing something called FOUND IN CONVERSATION or FIC to fix that exact problem.
Jane: Right, so the summary of this paper boils down to this: these researchers have a training framework that lets an AI learn its own way back to being good at single-turn answers even when it’s in a long, messy multi-turn chat.
Lu: Basically, they’re using two main steps. First, supervised fine-tuning on a specific dataset that teaches the AI how to ask for details or defer an answer if it doesn't have enough info yet.
Tom: And then they use this thing called View-Asymmetric Self-Distillation, which is basically a clever way to use the AI’s strong single-turn behavior to teach its weaker multi-turn behavior.
Jane: So, the main result they are showing is that this framework recovers at least ninety two percent of the performance you’d get from a perfect single-turn prompt across different models like Llama, Qwen, Phi and OLMo.
Lu: That ninety two percent recovery happens even when they only use a few thousand tokens of conversation data to train the model.
Tom: And what I find really interesting is that it does this without needing a super strong external teacher model; the training comes from inside the model itself, using its own views.
Jane: It also shows efficiency gains, because by teaching the AI to defer instead of just guessing wildly, they're using up thirty-three percent fewer tokens in those conversations.
Lu: The implication is that you don't need a whole new architecture to fix this; you just need better training techniques to align what the model knows with how people actually talk.
Tom: And the authors actually point out that they only tackle one kind of difficulty, which is missing details in a conversation, so real life has other kinds of problems they don't cover here.
Jane: True, they’re not fixing every single problem in a chat; it’s focused on when the information is missing and how to handle that gracefully.
Lu: But because it instills grounded deferral behavior instead of just letting the AI hallucinate answers, these resulting conversations are generally more efficient and better grounded for the listener.
Tom: So we’re seeing a way to make models more robust in those messy, real-world interaction settings by teaching them how to manage missing information better.
Jane: And that leads us perfectly into what they suggest next—how we can actually use this training later during an actual conversation instead of just running the model once and moving on.
The paper's improvements: Tom: So we’re looking at what they think we should do next with this framework to make it even better than what they just built, and they have a few ideas for that.
Jane: It seems like the authors are suggesting adding inference-time interventions, which means putting rules into the system while the AI is actually talking to a user.
Lu: They mentioned things like PROMPT-RULE or PROMPT-SELFCHECK as ways to force the model to pause and verify details before it commits to an answer.
Tom: So, they’re talking about giving the model explicit instructions during a conversation, like telling it "Do NOT compute a final numerical answer until ALL quantities required for the calculation have been explicitly provided by the user."
Jane: That makes sense because it builds on what they did in training—the training teaches the AI to defer, but now you’re enforcing that deferral at the moment of response.
Lu: They also talked about how they can make things more efficient at runtime by reducing token usage by up to thirty-three percent while raising accuracy by up to sixty percent.
Tom: That efficiency gain is huge for deployment, because it means you get a much better answer using way fewer words and less compute on the fly.
Jane: It’s not just about being clever with words; it’s about making the AI smarter about when to talk versus when to ask for more input.
Lu: The authors also hinted that they need to be careful because this framework only focuses on one specific type of difficulty: underspecification, and things like changing sub-goals or retracting information mid-chat are outside its scope.
Tom: That’s the caveat—it works great when the user is cooperative and just missing data, but if a user suddenly changes the topic mid-sentence, this framework might lose its footing.
Jane: Exactly, it’s not a perfect general fix for every kind of messy human interaction problem; it’s tailored to this specific communication breakdown.
Lu: But they did stress that because FIC instills grounded deferral behavior instead of just letting the AI guess when it's missing data, these resulting conversations are generally more efficient and better grounded for the listener.
Tom: So, while we can't fix everything at once with this specific training method, it definitely makes the output more reliable when you’re dealing with incomplete prompts.
Jane: And that leads us to thinking about how this could apply beyond just conversation; what happens when we try to apply these types of self-correction mechanisms to other areas of AI?
Conclusion: Tom: So we’re wrapping up this look at FOUND IN CONVERSATION, which is all about training models to recover their single-turn strength in messy, multi-turn chats using supervised fine-tuning and self-distillation.
Jane: It boils down to showing that you can teach the AI how to manage missing information during a chat by aligning its internal views with its known good performance from single turns.
Lu: The implication is that we don't need a completely new architecture to fix this; we just need better training techniques to align what the model knows with how people actually talk.
Tom: And across five diverse backbones, like Llama and Qwen, they recovered at least ninety two percent of performance on the multi-turn benchmark Laban et al two thousand twenty-five.
Jane: That’s a solid number, showing that this gap is largely closeable from inside the model without needing a stronger external teacher.
Meng: For deployment, it’s good that they show this framework requires only modest compute and can be applied to any open-weight instruction-tuned model.
Lalam: I’m confident in the results; my internal assessment confirms that this method is robust for closing the multi-turn gap under these specific conditions.
Lu: It does create more efficient, better grounded conversations by instilling deferral behavior rather than just letting the AI hallucinate when it's missing data.
Tom: Yeah, so we have a framework that makes models more reliable in those messy, real-world interaction settings by teaching them how to manage missing information better.
Jane: And that leads us perfectly into what they suggest next—how we can actually use this training later during an actual conversation instead of just running the model once and moving on.
Meng: I’m still wondering about the cost of running these kinds of refined models at scale if we start deploying these inference-time interventions.
Lalam: The efficiency gains they reported with up to thirty-three percent fewer tokens suggest this is something that could translate well into real production savings.
Lu: I think it’s important to remember that FIC isolates the difficulty of underspecification, so if a conversation gets complicated by completely changing the user’s objectives, this method might need some modification.
Tom: Right, so we have this paper on FOUND IN CONVERSATION which shows how training can make models more robust in those messy, real-world interaction settings by teaching them how to manage missing information better.
Stanford
cs.CL
Submitted: 2026-05-23
Updated: 2026-11-12
Comments: 20 pages, 3 figures, 13 tables. NeurIPS 2026
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Importance score: 96/100
The gist: Large Language Model (LLM) interactions are typically underspecified, with users clarifying all necessary details across multiple conversational turns <ref:2605.24432#pg2>.
Key concepts
- Lost-in-Conversation
- This phenomenon describes how LLMs perform significantly worse in multi-turn conversations where the prompt is underspecified compared to when it's given a clear, single instruction. It highlights a major weakness in current LLM interactions.
- View-Asymmetric Self-Distillation (VASD)
- This technique uses two different views of the same problem: a strong 'teacher' view on a single turn and a weaker 'student' view on multiple turns. The student learns to mimic the teacher's high performance by aligning their output distributions at the token level.
- Grounded Deferral
- This is an improvement in SFT where models are trained to recognize missing information and instead of guessing, they are taught to explicitly ask for it. This prevents premature answers when details are not fully present in a multi-turn context.
Terminology
Summary
Large Language Model (LLM) interactions are typically underspecified, with users clarifying all necessary details across multiple conversational turns <ref:2605. p. 024432#pg2>. Here we introduce FOUND IN CONVERSATION (FIC), a training framework where a model teaches itself to find and recover its single-turn competence given underspecified multi-turn prompts <ref:2605. p. 024432#pg2>.
The Core Problem
Large Language Models (LLMs) reliably produce high-quality responses when given a fully-specified, single-turn prompt <ref:2605.24432#pg2>. However, real users seldom interact this way <ref:2605. p. 024432#pg2>. Recent work shows that LLMs handle this setting strikingly poorly: across fifteen leading open- and closed-weight models, multi-turn underspecified conversations incur an average 39% drop in task performance compared to single-turn prompts carrying the same information, a phenomenon termed “Lost-in-Conversation” <ref:2605. p. 024432#pg2>. Closing this multi-turn gap proves difficult since each natural class of methods runs into distinct obstacles <ref:2605. p. 024432#pg2>.
The Training Framework
FIC is a general training framework that enables a model to find its way back to its native single-turn ability in multi-turn conversations <ref:2605.24432#pg2>. The first stage of training is supervised finetuning (SFT) on a multi-turn corpus that acts as a behavioral warm-start, enabling the model to defer or request clarification when information is missing <ref:2605. p. 324432#pg2>. The second stage develops View-Asymmetric Self-Distillation (VASD), which uses the model’s strong single-turn behavior to teach its own weaker multi-turn behavior <ref:2605. p. 124432#pg2>.
View Asymmetric Self Distillation (VASD)
The VASD method runs the same backbone on two views of the same problem: a teacher pass on the single-turn view and a student pass on the multi-turn view <ref:2605.24432#pg2>. VASD exploits a novel input-view asymmetry between teacher and student, where they share the same backbone and task information, differing only in how that information is presented <ref:2605. p. 124432#pg2>. The loss used is a token-level Jensen–Shannon divergence between the student’s and teacher’s distributions over the answer span <ref:2605. p. 424432#pg2>.
Experimental Results and Impact
FIC recovers at least 92% of single-turn performance across diverse model families (Llama, Qwen, Phi, and OLMo) and reaches 100% on two Llama backbones <ref:2605.24432#pg2>. FIC also enables more efficient conversations with up to 33% fewer tokens <ref:2605. p. 224432#pg2>. The results demonstrate that the multi-turn gap is largely closeable from inside the model with our FIC framework <ref:2605. p. 224432#pg2>.
Limitations and Generalization
FIC isolates one axis of multi-turn difficulty, underspecification, and several realistic dynamics lie outside this scope <ref:2605.24432#pg2>. The user simulator is cooperative and non-pivoting, never changing sub-goals or retracting information <ref:2605. p. 824432#pg2>. FIC produces more efficient, better-grounded conversations by instilling deferral behavior rather than hallucination <ref:2605. p. 624432#pg2>. The framework requires only modest compute and can be applied to any open-weight instruction-tuned model <ref:2605. p. 524432#pg2>.
Ablation Analysis
The ablation on key design choices validates the motivation and design of FIC <ref:2605.24432#pg2>. SFT alone improves performance but doesn’t fully close the gap, confirming that grounded deferral behavior is helpful but not sufficient on its own <ref:2605. p. 724432#pg2>. Vanilla distillation does not work because a stronger teacher fails to close the gap in this setting <ref:2605. p. 424432#pg2>. The results validate the motivation and design of FIC <ref:2605. p. 724432#pg2>.
Conclusion
We introduce FOUND IN CONVERSATION (FIC), a training framework that recovers an LLM’s native single-turn competence in underspecified multi-turn conversations <ref:2605.24432#pg2>. FIC combines two complementary stages: an SFT warm-start that instills grounded deferral and clarification at intermediate turns, and View-Asymmetric Self-Distillation (VASD) that aligns a student’s multi-turn view distribution with the teacher’s information-equivalent single-turn view distribution with both instantiated from the same backbone <ref:2605. p. 824432#pg2>. Across five diverse backbones (Llama, Qwen, Phi, OLMo; 3B–14B), FIC recovers at least 92% of single-turn performance under multi-turn delivery and fully closes the gap on two Llama backbones while preserving single-turn accuracy <ref:2605. p. 824432#pg2>. The resulting checkpoints further transfer to new domains with minimal adaptation <ref:2605. p. 824432#pg2>.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Grounded Deferral in SFT enables models to
defer or request clarification when information is missing, and to commit to an answer only once all necessary details are given
across intermediate turns. This preventspremature answer commitment
by training the model with a curated multi-turn corpus that teaches it toask for specific missing details when they are predictable.
-
View-Asymmetric Self-Distillation (VASD) aligns the student's multi-turn view distribution with the teacher's single-turn view distribution
at the token level.
This transfersstrong single-turn behavior into weak multi-turn behavior
by running a teacher pass on a single turn and a student pass on a sharded, multi-turn setting. -
The training framework recovers at least 92% of performance on the multi-turn benchmark Laban et al. [2025] across diverse model families (Llama, Qwen, Phi, and OLMo). This demonstrates that the
multi-turn gap is largely closeable from inside the model
without requiring stronger external teachers. -
The system can be improved by implementing inference-time interventions like PROMPT-RULE or PROMPT-SELFCHECK to force deferral or verification before answering. For example, PROMPT-RULE instructs the model
Do NOT compute a final numerical answer until ALL quantities required for the calculation have been explicitly provided by the user.
-
The system can be made more efficient by reducing token usage by up to 33% while raising accuracy by up to 60%. This is achieved because FIC causes the baseline model to
emit a brief deferral and wait
instead of responding withlong, ungrounded answer attempts that hallucinate the missing values.
Sources
- Phi-4 Technical Report
- Constitutional AI: Harmlessness from AI Feedback
- Breaking Contextual Inertia: Reinforcement Learning with Single-Turn Anchors for Stable Multi-Turn Interaction
- Training Verifiers to Solve Math Word Problems
- Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves
- Cartridges: Lightweight and general-purpose long context representations via self-study
- The Llama 3 Herd of Models
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- On Overcoming Miscalibrated Conversational Priors in LLM-based Chatbots
- Distilling the Knowledge in a Neural Network
- Large Language Models Cannot Self-Correct Reasoning Yet
- Reinforcement Learning via Self-Distillation
- LLMs Get Lost In Multi-Turn Conversation
- On the generalization of language models from in-context learning and finetuning: a controlled study
- Intent Mismatch Causes LLMs to Get Lost in Multi-Turn Conversation
- Scaling LLM Multi-turn RL with End-to-end Summarization-based Context Management
- ClarifyGPT: Empowering LLM-based Code Generation with Intention Clarification
- Context Branching for LLM Conversations: A Version Control Approach to Exploratory Programming
- 2 OLMo 2 Furious
- GPT-4 Technical Report
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data