Few Trials, Many Metrics: Beta Mixtures for Low-Budget AI Evaluation
cs.AI
Submitted: 2026-10-31
Updated: 2026-10-31
Terminology
Sources
- Large Language Monkeys: Scaling Inference Compute with Repeated Sampling
- Evaluating Large Language Models Trained on Code
- Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference
- BayesAME: Bayesian Active Model Evaluation
- Beyond Benchmarks: MathArena as an Evaluation Platform for Mathematics with LLMs
- Test-Time Scaling in Reasoning LLMs: Inference Regimes, Evaluation, and Reproducibility
- Best-of-N Jailbreaking
- Efficient Prediction of Pass@k Scaling in Large Language Models
- A Simple Model of Inference Scaling Laws
- SuperGPQA: Scaling LLM Evaluation across 285 Graduate Disciplines
- How Do Large Language Monkeys Get Their Power (Laws)?
- $\tau$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains
Related papers
- Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Embeddings for Preferences, Not Semantics
- Can We Trust Item Response Theory for AI Evaluation?
- Diagnosing Task Insensitivity in Language Agents