Anticipatory Evaluation of Language Models: Predicting LLM Benchmark Scores from Descriptions
Researchers investigated if models can predict LLM performance scores just from text descriptions of tasks and setups, without running experiments. They created EVALCAST, a resource of description-performance pairs. Experiments showed that reasoning-enabled LLMs significantly outperform simpler methods at predicting absolute performance, suggesting this forecasting method is feasible for prioritizing benchmarks.