Adaptive Testing for LLM Evaluation: A Psychometric Alternative to Static Benchmarks
ATLAS is an adaptive testing framework for evaluating Large Language Models (LLMs) that uses Item Response Theory (IRT). It selects items dynamically based on a model's estimated ability, drastically reducing item requirements by up to 90% while maintaining measurement precision. This allows for finer comparisons between models and reveals how ability ($ heta$) preserves global performance structure.