Adaptive Testing for LLM Evaluation: A Psychometric Alternative to Static Benchmarks

arXiv:2511.04689 · cs.CL, cs.AI · Submitted 2025-10-26 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Adaptive Testing for LLM Evaluation".

Jane: The gist: ATLAS, an adaptive testing framework based on Item Response Theory (IRT) that estimates model ability using Fisher information–guided item selection,

Tom: First, who's behind it and why it matters.

Title and authors: Tom: Okay, so we’re looking at "Adaptive Testing for LLM Evaluation: A Psychometric Alternative to Static Benchmarks" today, and the title itself tells you a lot about what they’re doing. Jane, can you give us the simple breakdown of why this name matters?

Jane: Well, it suggests that instead of just running a fixed test and getting an average score for every item set, we should be estimating the actual intelligence—the ability—of the model itself. It moves away from just looking at raw accuracy over a set of questions.

Lu: Exactly. The paper points out that current methods treat all items as equally informative even when they are totally different in how hard they are or how well they discriminate between different types of models >

Tom: So, the core idea is using Item Response Theory, or IRT, to estimate this latent ability theta for each model based on its response pattern >

Jane: Right. And once you have those ability estimates, you can compare models even if their final raw accuracy scores are exactly the same; it reveals substantial rank reordering across different subsets of questions and related domains >

Meng: So, if we have two models that both score eighty-five percent on a benchmark, ATLAS helps us figure out which one is actually better because they might have different underlying abilities >

Tom: That’s the main shift here. It’s about moving from just counting right answers to actually estimating the intelligence of an AI model using this psychometric approach >

The paper's summary: Jane: Moving into what the paper actually proposes, they outline a four-stage process for building this ATLAS framework, starting with problem formulation and then moving into data construction using psychometric filtering to clean up the item bank >

Lu: They do this by first selecting high-quality items and then calibrating that entire item bank using IRT models, specifically a three-parameter logistic model to estimate item difficulty and discrimination parameters >

Tom: That calibration step is crucial because it lets them figure out exactly how hard each question is and how much that question separates a good model from a bad one >

Jane: And the real magic happens in the testing phase, where they use Fisher information guided selection to choose which questions to ask next, prioritizing items that give the most new information about the model's current estimated ability >

Meng: So, instead of asking every single question in a fixed set, ATLAS picks only a few highly informative ones based on what the model is doing at that moment >

Tom: They claim this adaptive testing can reduce the number of required items by up to ninety percent while still maintaining measurement precision >

The paper's improvements: Jane: The authors highlight three specific ways this framework improves upon existing methods. First, it significantly cuts down on the computational cost because it requires ninety percent fewer items compared to what’s usually needed for these benchmarks >

Lu: Second, they say it overcomes those ceiling effects you see with accuracy-based metrics and preserves discrimination across the ability spectrum, meaning we don't lose track of how well a model handles questions at different levels of difficulty >

Tom: And that’s important because static evaluations just give you one number, but this method shows that ability-based evaluation reveals substantial rank reordering and produces more stable model rankings across benchmark subsets and related domains >

Meng: From a deployment standpoint, if we have to run an expensive evaluation, this method makes it much cheaper by requiring significantly fewer items >

Lalam: I’m confident in what the paper says. They reconstruct accuracy from those ability estimates and found that reconstructed accuracies closely match raw accuracies across all five benchmarks >

Jane: So the big implication is that we can get a clearer picture of LLM performance without needing hundreds of items for every single test, provided you use this adaptive testing framework >

Conclusion: Tom: So we just looked at "Adaptive Testing for LLM Evaluation: A Psychometric Alternative to Static Benchmarks," and essentially, they are showing us how to stop relying just on counting right answers and start estimating the actual intelligence of an AI model using Item Response Theory >

Jane: Exactly. It’s about moving from static scoring to dynamic ability estimation using this psychometric approach, which lets you see differences even when two models have the same overall accuracy >

Lu: The methodology involves a four-stage process: problem formulation, data construction with psychometric filtering, item bank calibration using IRT models, and finally adaptive testing with randomesque selection >

Meng: And the big win there is that this adaptive testing framework reduces the number of required items by up to ninety percent while maintaining measurement precision >

Tom: That’s a big reduction in evaluation cost. But Jane, what about the quality of those items they're using? It sounds like if you feed bad questions into this system, you get bad results >

Jane: They address that with rigorous data construction and item filtering—things like low-variance removal and discrimination filtering—to make sure the item bank is high quality before calibration begins >

Lu: The calibration stage uses a three-parameter logistic model to estimate item difficulty, discrimination, and guessing parameters, but they do it in parallel subsets first to manage the computational complexity >

Tom: So the core of it is Fisher information guided selection during testing. The AI picks questions that give the most new information about the model's ability at any given moment >

Meng: It’s smart, yes, but for production, we need to know if this is just a neat trick or something that holds up when you deploy it on real traffic where things break easily >

Jane: Well, the results show that ability-based rankings are more stable across different benchmark forms and more consistent across related benchmarks than raw accuracy rankings >

Tom: And they confirm their method works because the reconstructed accuracies closely match the raw percent correct, which means those latent ability estimates faithfully preserve the global performance structure >

Lu: They also use a common-person linking strategy during calibration to align different item subsets onto one unified scale, which helps prevent scale drift across a large item bank >

Tom: This work on "Adaptive Testing for LLM Evaluation: A Psychometric Alternative to Static Benchmarks" shows us how to build more reliable and cost-effective ways to measure model intelligence >

Lu: It sets a new standard for how we think about evaluating these systems, moving beyond just checking the final score on a fixed test set >

Meng: I'm curious what comes next. If this framework works, can it scale up to evaluate millions of parameters without needing an entire item bank for every single one?

Jane: That’s the big question for future work. We need to figure out how robust this whole psychometric structure is when dealing with massive, real-world datasets >

University of Notre Dame

cs.CL, cs.AI

Submitted: 2025-10-26

Updated: 2026-07-13

Comments: ICML 2026 Spotlight

Journal ref: Proceedings of the 43rd International Conference on Machine Learning (ICML 2026), PMLR 306, 70246-70269, 2026

Code: https://github.com/Peiyu-Georgia-Li/ATLAS

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 86/100

The gist: The gist: ATLAS, an adaptive testing framework based on Item Response Theory (IRT) that estimates model ability using Fisher information–guided item selection, reduces evaluation item requirements

Key concepts

Item Response Theory (IRT)
A statistical framework used in psychometrics to model how well a test item measures a latent trait, like an LLM's ability. It helps estimate item characteristics such as difficulty and discrimination, which are crucial for accurate testing.
Fisher Information-Guided Item Selection
The core adaptive mechanism of ATLAS. Instead of using all items equally, this method selects the next question based on which item provides the most new information about a model's ability ($ heta$). This ensures that testing focuses on the most informative questions first.
Latent Ability ($ heta$)
The underlying, unobservable skill level of an LLM. ATLAS estimates this ability for each model by observing how they respond to a set of items. This estimated $ heta$ is used to compare models even if their raw accuracy scores are identical.
Fisher Information Score (IES)
A metric used by ATLAS to evaluate the efficiency of the testing process. A lower IES indicates that the framework is more efficient at estimating model ability with fewer items, demonstrating its success in reducing evaluation costs.

Terminology

Summary

The gist: ATLAS, an adaptive testing framework based on Item Response Theory (IRT) that estimates model ability using Fisher information–guided item selection, reduces evaluation item requirements by up to 90% while maintaining measurement precision.

Problem Formulation and Setup

The paper frames LLM evaluation as a psychometric measurement problem where the objective is to estimate the latent ability θl of each model based on its response pattern while calibrating item-level parameters like difficulty, discrimination, and guessing parameters<ref:2511.04689#pg4>. This approach enables fine-grained model comparison even when models achieve identical accuracy, as θl accounts for the varying informativeness of different items<ref:2511.04689#pg4>. The framework is structured in four stages: problem formulation (Section 3.1), data construction with psychometric filtering (Section 3.2), item bank calibration using IRT models (Section 3.3), and adaptive testing with randomesque selection (Section 3.4)<ref:2511.04689#pg5>.

Data Construction with Psychometric Filtering

The data construction phase involves rigorous filtering to ensure quality for IRT calibration<ref:2511.04689#pg4>. This includes model selection, where models with complete responses across all items are retained, and a split into training and testing sets using stratified random sampling<ref:2511.04689#pg4>. Item filtering applies two complementary filters: low-variance removal (items with response standard deviation 95% are discarded) and discrimination filtering, where items with point-biserial correlation rpb(i).

Scalable IRT Calibration

The calibration stage estimates item parameters using the three-parameter logistic (3PL) IRT model, which characterizes item difficulty, discrimination, and guessing parameters<ref:2511.04689#pg4>. To manage computational complexity when fitting the full 3PL model to the entire item pool at once, a partition-based calibration procedure is adopted where items are divided into K non-overlapping subsets Ik (each with Ik ≥ 100 items), and each subset is calibrated independently<ref:2511.04689#pg5>. These independently calibrated scales are then linked onto a unified scale using mean–sigma transformations, leveraging the model population as common persons to serve as linking anchors<ref:2511.04689#pg4>.

Adaptive Testing with Information Selection

The core of ATLAS is the dynamic selection process, which aims to select items with maximum Fisher information for each model’s current estimated ability<ref:2511.04689#pg4>. The algorithm selects items based on Fisher information Ii(ˆθt−1) for unadministered items and then randomly samples from the top-5 most informative candidates<ref:2511.04689#pg4>. The process terminates when precision thresholds are reached, specifically when SE(ˆθt) ≤ τ, after a minimum number of items has been administered<ref:2511.04689#pg4>. This adaptive structure addresses the limitation of static benchmarks by prioritizing highly discriminative items rather than treating all questions as equally informative<ref:2511.04689#pg4>.

Performance and Reliability Analysis

ATLAS was evaluated across five major benchmarks, including WinoGrande, TruthfulQA, HellaSwag, GSM8K, and ARC<ref:2511.04689#pg5>. The framework achieved the lowest Information Efficiency Score (IES) across all benchmarks<ref:2511.04689#pg5>. Ability-based rankings were shown to be more stable across benchmark forms and more consistent across related benchmarks compared to raw accuracy-based rankings<ref:2511.04689#pg8>. Furthermore, the reconstructed accuracies closely matched raw accuracies, with MAE consistently below 5% across all benchmarks, confirming that θ faithfully preserves global performance structure<ref:2511.04689#pg8>.

Conclusion and Implications

The work concludes by emphasizing several psychometric principles for future benchmark design: item quality is critical for reliable evaluation, requiring discrimination screening<ref:2511.04689#pg6>. Model fit diagnostics should be a standard requirement whenever IRT is used for item calibration or reduction<ref:2511.04689#pg6>. Principled linking strategies are crucial for scalable benchmarking to prevent scale drift<ref:2511.04689#pg6>. ATLAS successfully reframes LLM evaluation by moving beyond static, accuracy-based benchmarking toward dynamic ability estimation<ref:2511.04689#pg10>. The framework demonstrates that adaptive item selection enables high-fidelity ability estimates while dramatically reducing evaluation cost<ref:2511.04689#pg4>.

Impact Statement

This work aims to advance the methodology of large language model evaluation by introducing a psychometrically grounded, adaptive testing framework<ref:2511.04689#pg11>. By improving the reliability, efficiency, and interpretability of model assessment, our approach may support more informed benchmarking and comparison of LLMs<ref:2511.04689#pg11>.

References

Allen M J and Yen W M Introduction to measurement theory. Waveland Press, 2001<ref:2511.04689#pg24>.

Baker F B and Kim S H Item response theory: Parameter estimation techniques. CRC press, 2004 doi: https://doi.org/10.1201/9781482276725<ref:2511.04689#pg4>.

Birnbaum A Some latent trait models and their use in inferring an examinee’s ability. Statistical theories of mental test scores, 1968<ref:2511.04689#pg4>.

Bock R D and Mislevy R J Adaptive eap estimation of ability in a microcomputer environment. Applied psychological measurement, 6(4):431–444, 1982 doi: https://doi.org/10.1177/014662168200600405<ref:2511.04689#pg4>.

Brennan R L Test equating, Scaling, and Linking-Methods and practices. Springer-verlag New York Incorporated, 2014<ref:2511.04689#pg4>.

Chalmers R P mirt: A multidimensional item response theory package for the r environment. Journal of statistical Software, 48:1–29, 2012 doi: 10.18637/jss.v048.i06<ref:2511.04689#pg4>.

Chang H H Psychometrics behind computerized adaptive testing. Psychometrika, 80(1):1–20, 2015<ref:2511.04689#pg4>.

Cheng Y and Chang H H The maximum priority index method for severely constrained item selection in computerized adaptive testing. British journal of mathematical and statistical psychology, 62(2):369–383, 2009 doi: https://doi.org/10.1348/000711008X304376<ref:2511.04689#pg6>.

Christensen K B Makransky G and Horton M Critical values for yen’s q 3: Identification of local dependence in the rasch model using residual correlations. Applied psychological measurement, 41(3):178–194, 2017<ref:2511.04689#pg4>.

Clark P Cowhey I Etzioni O Khot T Sabharwal A Schoenick C and Tafjord O Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018<ref:2511.04689#pg4>.

Cobbe K Kosaraju V Bavarian M Chen M Jun H Kaiser L Plappert M Tworek J Hilton J Nakano R Hesse C and Schulman J Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021<ref:2511.04689#pg4>.

Gema A P Leang J O J Hong G Devoto A Mancino A C M Saxena R He X Zhao Y Du X Madani M R G et al. Are we done with mmlu? In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics:

Improvements for AI systems

  1. Use ATLAS to reduce evaluation cost by up to 90% while maintaining measurement precision, as it reduces the number of required items by up to 90% while maintaining measurement precision. This allows for scalable and precision-targeted evaluation beyond fixed-length adaptive designs and attains the lowest Information Efficiency Score (IES) across all benchmarks.

  2. Employ Fisher information–guided item selection to prioritize highly discriminative questions, which addresses the limitation that static evaluations treat poorly discriminative items as equally informative as high-quality questions, thereby ensuring that ability θ preserves the global performance structure while providing finer discrimination within accuracy-equivalent models.

  3. Implement a psychometric framework to distinguish models with identical raw accuracy by estimating latent ability θ, which reveals substantial rank reordering and produces more stable model rankings across benchmark subsets and related domains. This allows for comparisons where models with identical accuracies receive meaningfully different ability estimates.

  4. Utilize the Performance-IRT (p-IRT) estimator to reconstruct accuracy from estimated abilities, as it shows that reconstructed accuracies closely match raw full-bank percent correct, confirming that latent ability estimates faithfully preserve global performance as measured by accuracy.

  5. Incorporate common-person linking during calibration to align independently calibrated item subsets onto a unified scale, ensuring the overall performance patterns match and maintaining coherence across a large item bank through the principle of common-person linking.

Sources

Related papers