Which Questions Improve Learning the Most? Utility Estimation of Questions with LM-based Simulations

arXiv:2502.17383 · cs.CL · Submitted 2025-02-24 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Which Questions Improve Learning the Most? Utility Estimation of Questions with LM-based Simulations".

Jane: Here's the gist of the paper. Asking good questions is critical for comprehension and learning, yet evaluating and generating such questions remains a challenging problem The paper introduces QUEST, a framework that uses language models to simulate learners and directly quantify the utility of a question – its contribution to exam performance Experiments show that questions generated by QUEST-trained models improve simulated test scores by over 20% compared to strong baselines that are fine-tuned using indirect metrics or leverage prompting methods Utility is only weakly correlated with salience and similarity to exam questions, suggesting that it captures unique signal that benefits downstream performance QUEST offers a new outcome-driven paradigm for question evaluation and generation – one that moves beyond question-answer content toward measurable improvements in learning outcomes.

Tom: Pruning redundant reasoning tokens The core of the QUEST framework involves simulating a learner who asks questions and receives answers while studying a textbook chapter, then uses them to take an end-of-chapter exam This simulation allows for the estimation of each question's utility by its direct effect on exam performance, rather than inferred indirectly based on the underlying content.

Jane: The framework consists of four components: (1) a question generator (Mq) that generates questions from a document D, (2) an answer generator (Ma) that independently produces answers to form QA pairs QA = (q1, a1),...,(qn, an), and (3) a learner simulator composed of a learner model Ml and an evaluator Me that assesses the effectiveness of QA by having Ml attempt an exam E using only the QA pairs.

Tom: Finally, the utility estimator runs the learner simulator under different QA subsets to estimate each pair’s marginal contribution to the learner’s exam performance.

Paper summary: Jane: So, to unpack that a bit more, the authors define utility for each question qi as its marginal contribution to the exam score by looking at two different gains. First, they look at the single one gain, which is how much the score changes when you have just that specific question and its answer compared to having no questions at all.

Lu: And then there's the all-but-one gain, which measures how much performance improves when you include that specific question and its answer alongside every other question in the set of questions. The final utility is the average of those two gains, so it's this way they capture both local and global context for each question qi >

Tom: And to actually use this to train their system, they use rejection sampling, which means they only keep the QA pairs that have a utility score greater than some threshold theta. This filters out the low-utility stuff and keeps only the questions that actually make a difference in performance according to their simulation.

Meng: But wait, if you're filtering based on simulated performance, how do they handle the fact that they don't have access to the original textbook text when simulating those exams? That sounds like a huge limitation for real-world application.

Lalam: They do simulate exam performance taken with different subsets of QA pairs without having access to the original text during that test phase, which is how they isolate the marginal benefit of each question >

Jane: Right, so the simulation is done on the QA pairs themselves, not by feeding them back into a model that reads the textbook chapter every time. This allows them to focus purely on what those specific question and answer pairings contribute to exam performance without getting bogged down in reading everything >

Tom: And their experiment shows that this direct utility measure beats traditional indirect metrics like salience or expected information gain, because those metrics just don't have a meaningful correlation with actual exam performance, with a spearman rho less than zero point one >

Lu: That really validates the idea that utility captures something unique about the question itself that isn't just surface-level similarity or how much information it contains in an abstract sense >

Meng: So, if we translate this for deployment, it means we need a way to reliably measure this simulation gain without needing access to massive, proprietary datasets just to get those utility scores right >

Conclusion: Tom: Looking at the whole paper, the authors introduce QUEST as a framework that uses language models to simulate learners and directly quantify question utility by measuring its contribution to exam performance. The main result is that training question generators with this method leads to higher-utility questions, and this translates directly into an average improvement of over twenty percent in simulated exam scores across five different academic subjects >

Jane: The implication here for us as researchers is that we can move beyond just creating questions that are grammatically correct or semantically similar to the text. We're talking about a system where the generation process is actively optimized by simulating how well those questions actually help a learner pass an exam, which feels like a more honest way to build educational tools >

Lu: The authors show that increasing the utility threshold in their rejection sampling process leads to generated questions that enable higher exam scores, confirming that stricter filtering selects more educationally valuable questions, even though it does reduce the number of training examples available for fine-tuning >

Meng: From a deployment standpoint, while the gpt-4o mini variant performed best across all subjects, they found that using a larger model like gpt-4o leads to mixed results because different models might use different reasoning styles, which suggests we need to be careful about which model we optimize for in production >

Lalam: The paper confirms that this utility estimation approach provides a unique signal for downstream performance, showing that it's not just noise and is actually correlated with success on exams >

Dong-Ho Lee, Hyundong Cho, Jonathan May, Jay Pujara

Information Sciences Institute, University of Southern California

cs.CL

Submitted: 2025-02-24

Updated: 2026-11-09

Comments: EMNLP 2026 Findings

Code: https://github.com/philschatz/textbooks

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 90/100

The gist: Asking good questions is critical for comprehension and learning, yet evaluating and generating such questions remains a challenging problem The paper introduces QUEST…

Key concepts

Utility ($u_i$)
This measures how much a specific question contributes to the final exam score. It is calculated based on two gains: the single-one gain (the direct impact of one question) and the all-but-one gain (the impact of all questions except that one). High utility means a question is highly valuable for learning.
Learner Simulator
This component simulates a student studying material and taking an exam using only the generated Question-Answer (QA) pairs. It uses a learner model to attempt the exam, allowing researchers to see how well the QA pairs help in actual performance.
Rejection Sampling
This is a training strategy used for the question generator ($M_q$). Instead of keeping all generated questions, this method only keeps QA pairs that have high utility (utility $\ge \theta$). This ensures the model is trained primarily on questions that are proven to be educationally valuable.
Utility vs. Salience
The study found that traditional metrics like salience (how noticeable a question is) and similarity to exam questions do not correlate with actual learning utility. This means focusing only on surface-level features is ineffective; utility captures a deeper, unique signal beneficial for performance.

Terminology

Summary

Asking good questions is critical for comprehension and learning, yet evaluating and generating such questions remains a challenging problem The paper introduces QUEST, a framework that uses language models to simulate learners and directly quantify the utility of a question – its contribution to exam performance Experiments show that questions generated by QUEST-trained models improve simulated test scores by over 20% compared to strong baselines that are fine-tuned using indirect metrics or leverage prompting methods Utility is only weakly correlated with salience and similarity to exam questions, suggesting that it captures unique signal that benefits downstream performance QUEST offers a new outcome-driven paradigm for question evaluation and generation – one that moves beyond question-answer content toward measurable improvements in learning outcomes. p. 0

Where the paper says it

Experiments show that questions generated by QUEST-trained models improve simulated test scores by over 20% compared to strong baselines that are fine-tuned using indirect metrics or leverage prompting methods. Furthermore, utility is only weakly correlated with salience and similarity to exam questions, suggesting that it captures unique signal that benefits downstream performance.

Pruning redundant reasoning tokens

The core of the QUEST framework involves simulating a learner who asks questions and receives answers while studying a textbook chapter, then uses them to take an end-of-chapter exam This simulation allows for the estimation of each question's utility by its direct effect on exam performance, rather than inferred indirectly based on the underlying content. The framework consists of four components: (1) a question generator (Mq) that generates questions from a document D, (2) an answer generator (Ma) that independently produces answers to form QA pairs QA =

(q1, a1),...,(qn, an), and (3) a learner simulator composed of a learner model Ml and an evaluator Me that assesses the effectiveness of QA by having Ml attempt an exam E using only the QA pairs. p. 1

Where the paper says it

, qn}, grounded in the document’s content; (2) answer generator (Ma) independently produces an answer ai for each question qi using general parametric knowledge, forming the set of QA pairs QA = {(q1, a1), . ,(qn, an)}; (3) learner simulator, composed of a learner model Ml and an evaluator Me, assesses the effectiveness of QA by having Ml attempt an exam E using only the QA pairs, and Me compute the resulting exam score; (4) utility estimator runs the learner simulator under different QA subsets to estimate each pair’s marginal contribution to the learner’s exam performance.

Finally, the utility estimator runs the learner simulator under different QA subsets to estimate each pair’s marginal contribution to the learner’s exam performance. p. 1
Where the paper says it

, qn}, grounded in the document’s content; (2) answer generator (Ma) independently produces an answer ai for each question qi using general parametric knowledge, forming the set of QA pairs QA = {(q1, a1), . ,(qn, an)}; (3) learner simulator, composed of a learner model Ml and an evaluator Me, assesses the effectiveness of QA by having Ml attempt an exam E using only the QA pairs, and Me compute the resulting exam score; (4) utility estimator runs the learner simulator under different QA subsets to estimate each pair’s marginal contribution to the learner’s exam performance.

How the training signal is built

The authors define utility ui of each question qi as its marginal contribution to the exam score, formulated as ui = (s(qi,ai) − s∅) + (sQA − sQA(qi,ai)). This utility is computed as the average of two gains: the single-one gain and the all-but-one gain. p. 2

Where the paper says it

To do this, we estimate the utility ui of each QA pair (qi , ai) based on its marginal contribution to exam performance. The utility estimator runs the learner simulator under two perturbations: (1) the single-one gain, defined as the exam score when only (qi , ai) is provided: s{(qi,ai)} − s∅, Chapter C Extract sections Exam E Document D Parse questions Classify Pair D – E (section – question) Bloom’s Taxonomy Figure 3: TEXTBOOK-EXAM Curation Pipeline.

The final utility is computed as ui = (s(qi,ai) − s∅) + (sQA − sQA(qi,ai)). p. 2
Where the paper says it

We treat this as a background prior and compute utility relative to this baseline. The final utility is computed as the average of the two gains: ui = (s{(qi,ai)} − s∅) + (sQA − sQA\{(qi,ai)}) 2 We retain only the QA pairs with ui ≥ θ, and update Mq using these high-utility examples via a rejection sampling strategy (Bai et al.

To improve the question generator Mq, they use rejection sampling to retain only QA pairs with ui ≥ θ. p. 2
Where the paper says it

The final utility is computed as the average of the two gains: ui = (s{(qi,ai)} − s∅) + (sQA − sQA\{(qi,ai)}) 2 We retain only the QA pairs with ui ≥ θ, and update Mq using these high-utility examples via a rejection sampling strategy (Bai et al. 3 TEXTBOOK-EXAM: A Dataset for Outcome-Based Question Evaluation To evaluate the effectiveness of QUEST in diverse learning settings, we construct TEXTBOOK-EXAM, a dataset composed of documents D and corresponding exam questions E, derived from real-world textbooks.

This process ensures that high-utility questions are selected for training via a rejection sampling strategy. p. 2
Where the paper says it

The final utility is computed as the average of the two gains: ui = (s{(qi,ai)} − s∅) + (sQA − sQA\{(qi,ai)}) 2 We retain only the QA pairs with ui ≥ θ, and update Mq using these high-utility examples via a rejection sampling strategy (Bai et al. 3 TEXTBOOK-EXAM: A Dataset for Outcome-Based Question Evaluation To evaluate the effectiveness of QUEST in diverse learning settings, we construct TEXTBOOK-EXAM, a dataset composed of documents D and corresponding exam questions E, derived from real-world textbooks.

Where the gains come from

The experiments demonstrate that QUEST-trained question generators produce higher-utility questions, leading to an average improvement of >20% in simulated exam scores across five academic subjects. Furthermore, indirect metrics such as salience and expected information gain have no meaningful correlation with utility (Spearman ρ < 0. p. 1

Where the paper says it

Using TEXTBOOK-EXAM, we experiment with various question generation strategies and quality metrics. Our experiments show that: (1) QUEST-trained question generators produce higher-utility questions, leading to an average improvement of >20% in simulated exam scores across five academic subjects; (2) Indirect metrics such as salience and expected information gain have no meaningful correlation with utility (Spearman ρ < 0.

1), and optimizing for these metrics does not improve exam performance. p. 1
Where the paper says it

Our experiments show that: (1) QUEST-trained question generators produce higher-utility questions, leading to an average improvement of >20% in simulated exam scores across five academic subjects; (2) Indirect metrics such as salience and expected information gain have no meaningful correlation with utility (Spearman ρ < 0. 1), and optimizing for these metrics does not improve exam performance; (3) Utility is also only weakly correlated with exam question similarity.

Utility-trained models consistently yield the highest exam scores across all subjects, with an average gain of 20% over prompting and SFT methods. p. 4
Where the paper says it

However, they do not explicitly optimize for learning outcomes, limiting their effectiveness; (2) SFT achieves only marginal improvements over prompting, suggesting that learning the surface style of exam questions does not translate into meaningful gains in learner performance; and (3) QUEST (QUEST) achieves the highest exam scores across all subjects, with an average gain of 20% over prompting and SFT methods. Its subject-specific variant consistently outperforms the cross-subject version, highlighting the importance of domain-specific optimization.

This suggests that utility provides unique signal that benefits downstream performance. p. 1
Where the paper says it

In fact, models fine-tuned with exam questions from the training set lead to questions with lower utility. These results suggest that high-utility questions are not effective because they resemble the format or content of exam questions and rather that utility provides unique signal that benefits downstream performance.

Qualitative characteristics of high-utility questions

Analysis of high-utility questions reveals that they are not simple paraphrases of exam questions but may introduce novel concepts that enhance learning beyond surface-level similarity. The correlation between utility and semantic similarity is 0. p. 5

Where the paper says it

We then assess the correlation between utility and the most similar exam question based on these measures. The correlation between utility and semantic similarity is 0.

25 (p < 0.001), indicating a weak positive relationship, while the correlation with ROUGE is nearly zero at 0. p. 5
Where the paper says it

The correlation between utility and semantic similarity is 0. 001), indicating a weak positive relationship, while the correlation with ROUGE is nearly zero at 0.

04 (p < 0.01). Bloom’s taxonomy also shows a weak positive relationship with utility, with the correlation being 0. p. 5
Where the paper says it

Using Bloom’s taxonomy as a cognitive depth scale (Likert 1-6), the correlation between utility and cognitive depth is 0. 001), indicating a weak positive relationship.

12 (p < 0.001). The authors observe that increasing the utility threshold leads to generated questions that enable higher exam scores, confirming that stricter filtering selects more educationally valuable questions. p. 5
Where the paper says it

4 Rejection Sampling Analysis We examine the effect of the utility threshold on rejection sampling results by varying it for chemistry as a representative example. Figure 5 illustrates that increasing the utility threshold leads to generated questions that enable higher exam scores, confirming that stricter filtering selects more educationally valuable questions.

Effects of design choices

The study examines the effect of varying the utility threshold on rejection sampling results, illustrating a trade-off between quality and quantity. Increasing the utility threshold leads to generated questions that enable higher exam scores, confirming that stricter filtering selects more educationally valuable questions. p. 5

Where the paper says it

4 Rejection Sampling Analysis We examine the effect of the utility threshold on rejection sampling results by varying it for chemistry as a representative example. Figure 5 illustrates that increasing the utility threshold leads to generated questions that enable higher exam scores, confirming that stricter filtering selects more educationally valuable questions.

The authors note that higher thresholds also reduce the number of training examples, potentially limiting model learning. p. 5
Where the paper says it

Figure 5 illustrates that increasing the utility threshold leads to generated questions that enable higher exam scores, confirming that stricter filtering selects more educationally valuable questions. However, higher thresholds also reduce the number of training examples, potentially limiting model learning.

Model size variations show that while upgrading to a larger model yields moderate improvements by 5. p. 5
Where the paper says it

For the Ma and Ml variants, we reuse the same questions (and answers) from the zero-shot baseline and re-run the simulation only. Our main findings are the following: (1) Question Generator: Upgrading to a larger model (gpt-4o) yields moderate improvements by 5.

7%, the optimized QUEST model trained with gpt-4o-mini outperforms it by 12. p. 5
Where the paper says it

Our main findings are the following: (1) Question Generator: Upgrading to a larger model (gpt-4o) yields moderate improvements by 5. 7%, but still underperforms compared to our optimized QUEST model trained with gpt-4o-mini by 12.

3%. This highlights the importance of balancing quality and quantity. p. 5
Where the paper says it

However, higher thresholds also reduce the number of training examples, potentially limiting model learning. This trade-off highlights the importance of balancing quality and quantity.

Model variants analysis

The performance of different model variants across subjects is detailed in Table 5, showing that the QUEST gpt-4o-mini variant achieves the highest exam scores across all subjects. The reader simulator using a larger model (gpt-4o) leads to mixed results, with performance gains in some subjects but a sharp decline in Chemistry. p. 5

Where the paper says it

However, it remains 11% behind the optimized gpt-4o-mini. (3) Reader Simulator: Using a larger model (gpt-4o) as the reader simulator leads to mixed results, with performance gains in some subjects but a

This suggests that larger models may apply different reasoning styles, leading to inconsistent evaluations. p. 6
Where the paper says it

sharp decline in Chemistry. This suggests that larger models may apply different reasoning styles, leading to inconsistent evaluations.

The framework, optimized for gpt-4o-mini, performs more reliably in this role. p. 6
Where the paper says it

This suggests that larger models may apply different reasoning styles, leading to inconsistent evaluations. Our framework, optimized for gpt-4o-mini, performs more reliably in this role.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Outcome-based question generation allows AI systems to produce questions that directly improve simulated exam performance by estimating utility ui of each question is defined as its marginal contribution to the exam score.

  2. The system can filter question generators using rejection sampling based on high utility, leading to models that "produce higher-utility questions, leading to an average improvement of >20% in simulated exam scores across five academic subjects."

  3. AI systems can be trained specifically on high-utility questions derived from the QUEST framework, resulting in question generation that consistently outperforms existing prompting and finetuning baselines.

  4. The system can distinguish between superficial question quality and actual learning impact because "indirect metrics such as salience and expected information gain have no meaningful correlation with utility (Spearman ρ < 0.1)."

  5. Models can be fine-tuned using high-utility questions, but the results show that models fine-tuned with exam questions from the training set lead to questions with lower utility, suggesting a focus on outcome over mere format matching.

  6. The system can optimize question generation by selecting examples based on an empirical threshold, as increasing the utility threshold leads to generated questions that enable higher exam scores.

  7. AI systems can be robustly tested across different model sizes and components, with findings showing that optimized QUEST model trained with gpt-4o-mini performs more reliably in this role compared to larger models in certain modules.

Abstract

Asking questions is fundamental for deeper understanding, yet evaluating and generating questions that improve downstream outcomes remains a challenging problem. To address this gap, we introduce QUEST (Question Utility Estimation with Simulated Tests), a framework that offers a new outcome-driven paradigm for question evaluation and generation, instead of indirect metrics that are based on the underlying content of the question and its answer. QUEST uses language models to simulate learners that ask questions while studying a textbook chapter and quantifies the utility of a question as its direct contribution to understanding as measured by the learner's exam performance. To analyze the effectiveness of QUEST, we curate TEXTBOOK-EXAM, a benchmark that aligns textbook sections with exam questions across five academic disciplines. Using QUEST, we filter for high-utility questions and fine-tune question generators via rejection sampling and show that questions generated by QUEST-trained models improve simulated exam scores by 18.5% compared to the next strongest baseline. Furthermore, we demonstrate the practical real-world value of QUEST by showing that simulation-computed utility of a question is moderately positively correlated with its usefulness as perceived by human evaluators.

Sources

Related papers