Which Questions Improve Learning the Most? Utility Estimation of Questions with LM-based Simulations
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "Which Questions Improve Learning the Most? Utility Estimation of Questions with LM-based Simulations".
Jane: Here's the gist of the paper. Asking good questions is critical for comprehension and learning, yet evaluating and generating such questions remains a challenging problem The paper introduces QUEST, a framework that uses language models to simulate learners and directly quantify the utility of a question – its contribution to exam performance Experiments show that questions generated by QUEST-trained models improve simulated test scores by over 20% compared to strong baselines that are fine-tuned using indirect metrics or leverage prompting methods Utility is only weakly correlated with salience and similarity to exam questions, suggesting that it captures unique signal that benefits downstream performance QUEST offers a new outcome-driven paradigm for question evaluation and generation – one that moves beyond question-answer content toward measurable improvements in learning outcomes.
Tom: Pruning redundant reasoning tokens The core of the QUEST framework involves simulating a learner who asks questions and receives answers while studying a textbook chapter, then uses them to take an end-of-chapter exam This simulation allows for the estimation of each question's utility by its direct effect on exam performance, rather than inferred indirectly based on the underlying content.
Jane: The framework consists of four components: (1) a question generator (Mq) that generates questions from a document D, (2) an answer generator (Ma) that independently produces answers to form QA pairs QA = (q1, a1),...,(qn, an), and (3) a learner simulator composed of a learner model Ml and an evaluator Me that assesses the effectiveness of QA by having Ml attempt an exam E using only the QA pairs.
Tom: Finally, the utility estimator runs the learner simulator under different QA subsets to estimate each pair’s marginal contribution to the learner’s exam performance.
Paper summary: Jane: So, to unpack that a bit more, the authors define utility for each question qi as its marginal contribution to the exam score by looking at two different gains. First, they look at the single one gain, which is how much the score changes when you have just that specific question and its answer compared to having no questions at all.
Lu: And then there's the all-but-one gain, which measures how much performance improves when you include that specific question and its answer alongside every other question in the set of questions. The final utility is the average of those two gains, so it's this way they capture both local and global context for each question qi >
Tom: And to actually use this to train their system, they use rejection sampling, which means they only keep the QA pairs that have a utility score greater than some threshold theta. This filters out the low-utility stuff and keeps only the questions that actually make a difference in performance according to their simulation.
Meng: But wait, if you're filtering based on simulated performance, how do they handle the fact that they don't have access to the original textbook text when simulating those exams? That sounds like a huge limitation for real-world application.
Lalam: They do simulate exam performance taken with different subsets of QA pairs without having access to the original text during that test phase, which is how they isolate the marginal benefit of each question >
Jane: Right, so the simulation is done on the QA pairs themselves, not by feeding them back into a model that reads the textbook chapter every time. This allows them to focus purely on what those specific question and answer pairings contribute to exam performance without getting bogged down in reading everything >
Tom: And their experiment shows that this direct utility measure beats traditional indirect metrics like salience or expected information gain, because those metrics just don't have a meaningful correlation with actual exam performance, with a spearman rho less than zero point one >
Lu: That really validates the idea that utility captures something unique about the question itself that isn't just surface-level similarity or how much information it contains in an abstract sense >
Meng: So, if we translate this for deployment, it means we need a way to reliably measure this simulation gain without needing access to massive, proprietary datasets just to get those utility scores right >
Conclusion: Tom: Looking at the whole paper, the authors introduce QUEST as a framework that uses language models to simulate learners and directly quantify question utility by measuring its contribution to exam performance. The main result is that training question generators with this method leads to higher-utility questions, and this translates directly into an average improvement of over twenty percent in simulated exam scores across five different academic subjects >
Jane: The implication here for us as researchers is that we can move beyond just creating questions that are grammatically correct or semantically similar to the text. We're talking about a system where the generation process is actively optimized by simulating how well those questions actually help a learner pass an exam, which feels like a more honest way to build educational tools >
Lu: The authors show that increasing the utility threshold in their rejection sampling process leads to generated questions that enable higher exam scores, confirming that stricter filtering selects more educationally valuable questions, even though it does reduce the number of training examples available for fine-tuning >
Meng: From a deployment standpoint, while the gpt-4o mini variant performed best across all subjects, they found that using a larger model like gpt-4o leads to mixed results because different models might use different reasoning styles, which suggests we need to be careful about which model we optimize for in production >
Lalam: The paper confirms that this utility estimation approach provides a unique signal for downstream performance, showing that it's not just noise and is actually correlated with success on exams >
Dong-Ho Lee, Hyundong Cho, Jonathan May, Jay Pujara
Information Sciences Institute, University of Southern California
cs.CL
Submitted: 2025-02-24
Updated: 2026-11-09
Comments: EMNLP 2026 Findings
Code: https://github.com/philschatz/textbooks
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 90/100
The gist: Asking good questions is critical for comprehension and learning, yet evaluating and generating such questions remains a challenging problem The paper introduces QUEST…
Key concepts
- Utility ($u_i$)
- This measures how much a specific question contributes to the final exam score. It is calculated based on two gains: the single-one gain (the direct impact of one question) and the all-but-one gain (the impact of all questions except that one). High utility means a question is highly valuable for learning.
- Learner Simulator
- This component simulates a student studying material and taking an exam using only the generated Question-Answer (QA) pairs. It uses a learner model to attempt the exam, allowing researchers to see how well the QA pairs help in actual performance.
- Rejection Sampling
- This is a training strategy used for the question generator ($M_q$). Instead of keeping all generated questions, this method only keeps QA pairs that have high utility (utility $\ge \theta$). This ensures the model is trained primarily on questions that are proven to be educationally valuable.
- Utility vs. Salience
- The study found that traditional metrics like salience (how noticeable a question is) and similarity to exam questions do not correlate with actual learning utility. This means focusing only on surface-level features is ineffective; utility captures a deeper, unique signal beneficial for performance.
Terminology
Summary
Asking good questions is critical for comprehension and learning, yet evaluating and generating such questions remains a challenging problem The paper introduces QUEST, a framework that uses language models to simulate learners and directly quantify the utility of a question – its contribution to exam performance Experiments show that questions generated by QUEST-trained models improve simulated test scores by over 20% compared to strong baselines that are fine-tuned using indirect metrics or leverage prompting methods Utility is only weakly correlated with salience and similarity to exam questions, suggesting that it captures unique signal that benefits downstream performance QUEST offers a new outcome-driven paradigm for question evaluation and generation – one that moves beyond question-answer content toward measurable improvements in learning outcomes. p. 0 Experiments show that questions generated by QUEST-trained models improve simulated test scores by over 20% compared to strong baselines
that are fine-tuned using indirect metrics or leverage prompting methods. Furthermore, utility is only weakly correlated
with salience and similarity to exam questions, suggesting
that it captures unique signal that benefits downstream performance.Where the paper says it
Pruning redundant reasoning tokens
The core of the QUEST framework involves simulating a learner who asks questions and receives answers while studying a textbook chapter, then uses them to take an end-of-chapter exam This simulation allows for the estimation of each question's utility by its direct effect on exam performance, rather than inferred indirectly based on the underlying content. The framework consists of four components: (1) a question generator (Mq) that generates questions from a document D, (2) an answer generator (Ma) that independently produces answers to form QA pairs QA =
(q1, a1),...,(qn, an), and (3) a learner simulator composed of a learner model Ml and an evaluator Me that assesses the effectiveness of QA by having Ml attempt an exam E using only the QA pairs. p. 1 , qn}, grounded in
the document’s content; (2) answer generator (Ma) independently produces an answer ai for each question qi using general parametric knowledge, forming the set of QA
pairs QA = {(q1, a1), . ,(qn, an)}; (3) learner simulator, composed of a learner model Ml and an evaluator Me,
assesses the effectiveness of QA by having Ml attempt an
exam E using only the QA pairs, and Me compute the resulting exam score; (4) utility estimator runs the learner
simulator under different QA subsets to estimate each pair’s
marginal contribution to the learner’s exam performance. , qn}, grounded in
the document’s content; (2) answer generator (Ma) independently produces an answer ai for each question qi using general parametric knowledge, forming the set of QA
pairs QA = {(q1, a1), . ,(qn, an)}; (3) learner simulator, composed of a learner model Ml and an evaluator Me,
assesses the effectiveness of QA by having Ml attempt an
exam E using only the QA pairs, and Me compute the resulting exam score; (4) utility estimator runs the learner
simulator under different QA subsets to estimate each pair’s
marginal contribution to the learner’s exam performance.Where the paper says it
Where the paper says it
How the training signal is built
The authors define utility ui of each question qi as its marginal contribution to the exam score, formulated as ui = (s(qi,ai) − s∅) + (sQA − sQA(qi,ai)). This utility is computed as the average of two gains: the single-one gain and the all-but-one gain. p. 2 To do this, we estimate the utility ui of each QA pair
(qi
, ai) based on its marginal contribution to exam performance. The utility estimator runs the learner simulator under two perturbations: (1) the single-one gain, defined as the
exam score when only (qi
, ai) is provided: s{(qi,ai)} − s∅,
Chapter C
Extract
sections
Exam E
Document D
Parse questions
Classify
Pair D – E
(section – question)
Bloom’s Taxonomy
Figure 3: TEXTBOOK-EXAM Curation Pipeline. We treat this as a background prior and compute utility relative to this baseline. The final utility is computed as the average of the two gains:
ui =
(s{(qi,ai)} − s∅) + (sQA − sQA\{(qi,ai)})
2
We retain only the QA pairs with ui ≥ θ, and update Mq
using these high-utility examples via a rejection sampling
strategy (Bai et al. The final utility is computed as the average of the two gains:
ui =
(s{(qi,ai)} − s∅) + (sQA − sQA\{(qi,ai)})
2
We retain only the QA pairs with ui ≥ θ, and update Mq
using these high-utility examples via a rejection sampling
strategy (Bai et al. 3 TEXTBOOK-EXAM: A Dataset for
Outcome-Based Question Evaluation
To evaluate the effectiveness of QUEST in diverse learning
settings, we construct TEXTBOOK-EXAM, a dataset composed of documents D and corresponding exam questions
E, derived from real-world textbooks. The final utility is computed as the average of the two gains:
ui =
(s{(qi,ai)} − s∅) + (sQA − sQA\{(qi,ai)})
2
We retain only the QA pairs with ui ≥ θ, and update Mq
using these high-utility examples via a rejection sampling
strategy (Bai et al. 3 TEXTBOOK-EXAM: A Dataset for
Outcome-Based Question Evaluation
To evaluate the effectiveness of QUEST in diverse learning
settings, we construct TEXTBOOK-EXAM, a dataset composed of documents D and corresponding exam questions
E, derived from real-world textbooks.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the gains come from
The experiments demonstrate that QUEST-trained question generators produce higher-utility questions, leading to an average improvement of >20% in simulated exam scores across five academic subjects. Furthermore, indirect metrics such as salience and expected information gain have no meaningful correlation with utility (Spearman ρ < 0. p. 1 Using TEXTBOOK-EXAM,
we experiment with various question generation strategies
and quality metrics. Our experiments show that: (1) QUEST-trained question
generators produce higher-utility questions, leading to an
average improvement of >20% in simulated exam scores
across five academic subjects; (2) Indirect metrics such as
salience and expected information gain have no meaningful
correlation with utility (Spearman ρ < 0. Our experiments show that: (1) QUEST-trained question
generators produce higher-utility questions, leading to an
average improvement of >20% in simulated exam scores
across five academic subjects; (2) Indirect metrics such as
salience and expected information gain have no meaningful
correlation with utility (Spearman ρ < 0. 1), and optimizing
for these metrics does not improve exam performance; (3)
Utility is also only weakly correlated with exam question
similarity. However, they do not explicitly optimize
for learning outcomes, limiting their effectiveness; (2) SFT
achieves only marginal improvements over prompting, suggesting that learning the surface style of exam questions
does not translate into meaningful gains in learner performance; and (3) QUEST (QUEST) achieves the highest
exam scores across all subjects, with an average gain of
20% over prompting and SFT methods. Its subject-specific
variant consistently outperforms the cross-subject version,
highlighting the importance of domain-specific optimization. In fact, models fine-tuned with exam questions
from the training set lead to questions with lower utility. These results suggest that high-utility questions are not effective because they resemble the format or content of exam
questions and rather that utility provides unique signal that
benefits downstream performance.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Qualitative characteristics of high-utility questions
Analysis of high-utility questions reveals that they are not simple paraphrases of exam questions but may introduce novel concepts that enhance learning beyond surface-level similarity. The correlation between utility and semantic similarity is 0. p. 5 We then
assess the correlation between utility and the most similar
exam question based on these measures. The correlation between utility and semantic similarity is 0. The correlation between utility and semantic similarity is 0. 001),
indicating a weak positive relationship, while the correlation
with ROUGE is nearly zero at 0. Using Bloom’s taxonomy as a cognitive depth
scale (Likert 1-6), the correlation between utility and cognitive depth is 0. 001), indicating a weak positive
relationship. 4 Rejection Sampling Analysis
We examine the effect of the utility threshold on rejection
sampling results by varying it for chemistry as a representative example. Figure 5 illustrates that increasing the utility threshold leads to generated questions that enable higher
exam scores, confirming that stricter filtering selects more
educationally valuable questions.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Effects of design choices
The study examines the effect of varying the utility threshold on rejection sampling results, illustrating a trade-off between quality and quantity. Increasing the utility threshold leads to generated questions that enable higher exam scores, confirming that stricter filtering selects more educationally valuable questions. p. 5 4 Rejection Sampling Analysis
We examine the effect of the utility threshold on rejection
sampling results by varying it for chemistry as a representative example. Figure 5 illustrates that increasing the utility threshold leads to generated questions that enable higher
exam scores, confirming that stricter filtering selects more
educationally valuable questions. Figure 5 illustrates that increasing the utility threshold leads to generated questions that enable higher
exam scores, confirming that stricter filtering selects more
educationally valuable questions. However, higher thresholds also reduce the number of training examples, potentially limiting model learning. For the Ma and Ml
variants, we reuse the same questions (and answers) from
the zero-shot baseline and re-run the simulation only. Our main findings are the following: (1) Question Generator: Upgrading to a larger model (gpt-4o) yields
moderate improvements by 5. Our main findings are the following: (1) Question Generator: Upgrading to a larger model (gpt-4o) yields
moderate improvements by 5. 7%, but still underperforms
compared to our optimized QUEST model trained with
gpt-4o-mini by 12. However, higher thresholds also reduce the number of training examples, potentially limiting model learning. This trade-off highlights the
importance of balancing quality and quantity.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Model variants analysis
The performance of different model variants across subjects is detailed in Table 5, showing that the QUEST gpt-4o-mini variant achieves the highest exam scores across all subjects. The reader simulator using a larger model (gpt-4o) leads to mixed results, with performance gains in some subjects but a sharp decline in Chemistry. p. 5 However, it remains 11% behind
the optimized gpt-4o-mini. (3) Reader Simulator: Using a larger model (gpt-4o) as the reader simulator leads to
mixed results, with performance gains in some subjects but a sharp decline in Chemistry. This suggests that larger models
may apply different reasoning styles, leading to inconsistent
evaluations. This suggests that larger models
may apply different reasoning styles, leading to inconsistent
evaluations. Our framework, optimized for gpt-4o-mini,
performs more reliably in this role.Where the paper says it
Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Outcome-based question generation allows AI systems to produce questions that directly improve simulated exam performance by estimating
utility ui of each question is defined as its marginal contribution to the exam score.
-
The system can filter question generators using rejection sampling based on high utility, leading to models that "produce higher-utility questions, leading to an average improvement of >20% in simulated exam scores across five academic subjects."
-
AI systems can be trained specifically on high-utility questions derived from the QUEST framework, resulting in question generation that
consistently outperforms existing prompting and finetuning baselines.
-
The system can distinguish between superficial question quality and actual learning impact because "indirect metrics such as salience and expected information gain have no meaningful correlation with utility (Spearman ρ < 0.1)."
-
Models can be fine-tuned using high-utility questions, but the results show that
models fine-tuned with exam questions from the training set lead to questions with lower utility,
suggesting a focus on outcome over mere format matching. -
The system can optimize question generation by selecting examples based on an empirical threshold, as increasing the utility threshold
leads to generated questions that enable higher exam scores.
-
AI systems can be robustly tested across different model sizes and components, with findings showing that
optimized QUEST model trained with gpt-4o-mini performs more reliably in this role
compared to larger models in certain modules.
Abstract
Asking questions is fundamental for deeper understanding, yet evaluating and generating questions that improve downstream outcomes remains a challenging problem. To address this gap, we introduce QUEST (Question Utility Estimation with Simulated Tests), a framework that offers a new outcome-driven paradigm for question evaluation and generation, instead of indirect metrics that are based on the underlying content of the question and its answer. QUEST uses language models to simulate learners that ask questions while studying a textbook chapter and quantifies the utility of a question as its direct contribution to understanding as measured by the learner's exam performance. To analyze the effectiveness of QUEST, we curate TEXTBOOK-EXAM, a benchmark that aligns textbook sections with exam questions across five academic disciplines. Using QUEST, we filter for high-utility questions and fine-tune question generators via rejection sampling and show that questions generated by QUEST-trained models improve simulated exam scores by 18.5% compared to the next strongest baseline. Furthermore, we demonstrate the practical real-world value of QUEST by showing that simulation-computed utility of a question is moderately positively correlated with its usefulness as perceived by human evaluators.
Sources
- Constitutional AI: Harmlessness from AI Feedback
- Evaluating and Optimizing Educational Content with Large Language Model Judgments
- AgentReview: Exploring Peer Review Dynamics with LLM Agents
- REALTALK: A 21-Day Real-World Dataset for Long-Term Conversation
- Agent Hospital: A Simulacrum of Hospital with Evolvable Medical Agents
- NewsInterview: a Dataset and a Playground to Evaluate LLMs' Ground Gap via Informational Interviews
- AI Simulations Grounded in Rich Self-Reports Increase Accuracy and Reduce Distortion
- AgentClinic: a multimodal agent benchmark to evaluate AI in simulated clinical environments
- Which questions should I answer? Salience Prediction of Inquisitive Questions
- Simulating Classroom Education with LLM-Empowered Agents
- Self-Discover: Large Language Models Self-Compose Reasoning Structures
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data