Risk-Aware Adaptive Evaluation: Finding High-Impact Failures Under Limited Budgets
cs.AI, cs.LG, cs.MA
Submitted: 2026-09-30
Updated: 2026-09-30
Code: https://github.com/priyanathmaji/risk-aware-eval
Sources
- Analysis of Thompson Sampling for the multi-armed bandit problem
- How Many Tasks Are Enough for Agent Benchmark Decisions? A Replay Analysis of Public LLM Agent Benchmarks
- MCP-Bench: Benchmarking Tool-Using LLM Agents with Complex Real-World Tasks via MCP Servers
- Stochasticity in Agentic Evaluations: Quantifying Inconsistency with Intraclass Correlation
- On Randomness in Agentic Evals
- On the Reliability of Computer Use Agents
- Beyond Task Completion: Revealing Corrupt Success in LLM Agents through Procedure-Aware Evaluation
- Deep Bayesian Bandits Showdown: An Empirical Comparison of Bayesian Deep Networks for Thompson Sampling
- Neural Thompson Sampling
- Thompson Sampling in Partially Observable Contextual Bandits
- An adaptive importance sampling algorithm for risk-averse optimization
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems