Which Questions Improve Learning the Most? Utility Estimation of Questions with LM-based Simulations
summary
The gist
Asking good questions is critical for comprehension and learning, yet evaluating and generating such questions remains a challenging problem The paper introduces QUEST…
In short
QUEST is a framework using language models to simulate learners and directly measure a question's utility by its impact on exam performance. It found that questions trained with QUEST improve simulated test scores by over 20%, proving utility is a unique signal for learning, unlike simple content similarity metrics.
Key concepts
- Utility ($u_i$)
- This measures how much a specific question contributes to the final exam score. It is calculated based on two gains: the single-one gain (the direct impact of one question) and the all-but-one gain (the impact of all questions except that one). High utility means a question is highly valuable for learning.
- Learner Simulator
- This component simulates a student studying material and taking an exam using only the generated Question-Answer (QA) pairs. It uses a learner model to attempt the exam, allowing researchers to see how well the QA pairs help in actual performance.
- Rejection Sampling
- This is a training strategy used for the question generator ($M_q$). Instead of keeping all generated questions, this method only keeps QA pairs that have high utility (utility $\ge \theta$). This ensures the model is trained primarily on questions that are proven to be educationally valuable.
- Utility vs. Salience
- The study found that traditional metrics like salience (how noticeable a question is) and similarity to exam questions do not correlate with actual learning utility. This means focusing only on surface-level features is ineffective; utility captures a deeper, unique signal beneficial for performance.
Terminology used across episodes
This episode discusses
- Which Questions Improve Learning the Most? Utility Estimation of Questions with LM-based Simulations · Paper Radio
- Constitutional AI: Harmlessness from AI Feedback
- Evaluating and Optimizing Educational Content with Large Language Model Judgments
- AgentReview: Exploring Peer Review Dynamics with LLM Agents
- REALTALK: A 21-Day Real-World Dataset for Long-Term Conversation
- Agent Hospital: A Simulacrum of Hospital with Evolvable Medical Agents
- NewsInterview: a Dataset and a Playground to Evaluate LLMs' Ground Gap via Informational Interviews
- AI Simulations Grounded in Rich Self-Reports Increase Accuracy and Reduce Distortion · Paper Radio
- AgentClinic: a multimodal agent benchmark to evaluate AI in simulated clinical environments
- Which questions should I answer? Salience Prediction of Inquisitive Questions
- Simulating Classroom Education with LLM-Empowered Agents
- Self-Discover: Large Language Models Self-Compose Reasoning Structures
The paper
Which Questions Improve Learning the Most? Utility Estimation of Questions with LM-based Simulations · Read on arXiv
Dong-Ho Lee, Hyundong Cho, Jonathan May, Jay Pujara
Information Sciences Institute, University of Southern California
Asking questions is fundamental for deeper understanding, yet evaluating and generating questions that improve downstream outcomes remains a challenging problem. To address this gap, we introduce QUEST (Question Utility Estimation with Simulated Tests), a framework that offers a new outcome-driven paradigm for question evaluation and generation, instead of indirect metrics that are based on the underlying content of the question and its answer. QUEST uses language models to simulate learners that ask questions while studying a textbook chapter and quantifies the utility of a question as its direct contribution to understanding as measured by the learner's exam performance. To analyze the effectiveness of QUEST, we curate TEXTBOOK-EXAM, a benchmark that aligns textbook sections with exam questions across five academic disciplines. Using QUEST, we filter for high-utility questions and fine-tune question generators via rejection sampling and show that questions generated by QUEST-trained models improve simulated exam scores by 18.5% compared to the next strongest baseline. Furthermore, we demonstrate the practical real-world value of QUEST by showing that simulation-computed utility of a question is moderately positively correlated with its usefulness as perceived by human evaluators.
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "Which Questions Improve Learning the Most? Utility Estimation of Questions with LM-based Simulations".
Jane: Here's the gist of the paper. Asking good questions is critical for comprehension and learning, yet evaluating and generating such questions remains a challenging problem The paper introduces QUEST, a framework that uses language models to simulate learners and directly quantify the utility of a question – its contribution to exam performance Experiments show that questions generated by QUEST-trained models improve simulated test scores by over 20% compared to strong baselines that are fine-tuned using indirect metrics or leverage prompting methods Utility is only weakly correlated with salience and similarity to exam questions, suggesting that it captures unique signal that benefits downstream performance QUEST offers a new outcome-driven paradigm for question evaluation and generation – one that moves beyond question-answer content toward measurable improvements in learning outcomes.
Tom: Pruning redundant reasoning tokens The core of the QUEST framework involves simulating a learner who asks questions and receives answers while studying a textbook chapter, then uses them to take an end-of-chapter exam This simulation allows for the estimation of each question's utility by its direct effect on exam performance, rather than inferred indirectly based on the underlying content.
Jane: The framework consists of four components: (1) a question generator (Mq) that generates questions from a document D, (2) an answer generator (Ma) that independently produces answers to form QA pairs QA = (q1, a1),...,(qn, an), and (3) a learner simulator composed of a learner model Ml and an evaluator Me that assesses the effectiveness of QA by having Ml attempt an exam E using only the QA pairs.
Tom: Finally, the utility estimator runs the learner simulator under different QA subsets to estimate each pair’s marginal contribution to the learner’s exam performance.
Paper summary: Jane: So, to unpack that a bit more, the authors define utility for each question qi as its marginal contribution to the exam score by looking at two different gains. First, they look at the single one gain, which is how much the score changes when you have just that specific question and its answer compared to having no questions at all.
Lu: And then there's the all-but-one gain, which measures how much performance improves when you include that specific question and its answer alongside every other question in the set of questions. The final utility is the average of those two gains, so it's this way they capture both local and global context for each question qi >
Tom: And to actually use this to train their system, they use rejection sampling, which means they only keep the QA pairs that have a utility score greater than some threshold theta. This filters out the low-utility stuff and keeps only the questions that actually make a difference in performance according to their simulation.
Meng: But wait, if you're filtering based on simulated performance, how do they handle the fact that they don't have access to the original textbook text when simulating those exams? That sounds like a huge limitation for real-world application.
Lalam: They do simulate exam performance taken with different subsets of QA pairs without having access to the original text during that test phase, which is how they isolate the marginal benefit of each question >
Jane: Right, so the simulation is done on the QA pairs themselves, not by feeding them back into a model that reads the textbook chapter every time. This allows them to focus purely on what those specific question and answer pairings contribute to exam performance without getting bogged down in reading everything >
Tom: And their experiment shows that this direct utility measure beats traditional indirect metrics like salience or expected information gain, because those metrics just don't have a meaningful correlation with actual exam performance, with a spearman rho less than zero point one >
Lu: That really validates the idea that utility captures something unique about the question itself that isn't just surface-level similarity or how much information it contains in an abstract sense >
Meng: So, if we translate this for deployment, it means we need a way to reliably measure this simulation gain without needing access to massive, proprietary datasets just to get those utility scores right >
Conclusion: Tom: Looking at the whole paper, the authors introduce QUEST as a framework that uses language models to simulate learners and directly quantify question utility by measuring its contribution to exam performance. The main result is that training question generators with this method leads to higher-utility questions, and this translates directly into an average improvement of over twenty percent in simulated exam scores across five different academic subjects >
Jane: The implication here for us as researchers is that we can move beyond just creating questions that are grammatically correct or semantically similar to the text. We're talking about a system where the generation process is actively optimized by simulating how well those questions actually help a learner pass an exam, which feels like a more honest way to build educational tools >
Lu: The authors show that increasing the utility threshold in their rejection sampling process leads to generated questions that enable higher exam scores, confirming that stricter filtering selects more educationally valuable questions, even though it does reduce the number of training examples available for fine-tuning >
Meng: From a deployment standpoint, while the gpt-4o mini variant performed best across all subjects, they found that using a larger model like gpt-4o leads to mixed results because different models might use different reasoning styles, which suggests we need to be careful about which model we optimize for in production >
Lalam: The paper confirms that this utility estimation approach provides a unique signal for downstream performance, showing that it's not just noise and is actually correlated with success on exams >
More episodes
- 2603.28113-The multiplicative Lipschitz bound as an inductive bias
- 2603.28595-Optimistic Actor-Critic with Parametric Policies for Linear Markov Decision Processes
- 2603.29765-Training-Free Dynamic Upcycling of Expert Language Models
- 2605.00778-Observable Performance Does Not Fully Reflect Adaptive System Organization: A Multi-Level Analysis of Gait Dynamics Under Occlusal Constraint
- 2605.06906-TraXion: Rethinking Pre-training Frameworks for Mobility and Beyond
- 2605.09165-Sparse Layers are Critical to Scaling Looped Language Models
- 2605.09439-Inverse Design for Conditional Distribution Matching
- 2605.10313-Signature Approach for Contextual Bandits with Nonlinear and Path-dependent Rewards: Regret Bound and Statistical Inference
- 2605.10878-Minimum Neural Weight Norm = Kolmogorov Complexity
- 2605.11617-MIST: Reliable Streaming Decision Trees for Online Class-Incremental Learning via McDiarmid Bound