Beyond Fixed Benchmarks and Worst-Case Attacks: Dynamic Boundary Evaluation for Language Models

arXiv:2605.06213 · cs.AI · Submitted 2026-05-07 · Read on arXiv

cs.AI

Submitted: 2026-05-07

Updated: 2026-10-08

Terminology

• Benchmark — 2× in this paper · explained in 2 episodes
• Dynamic Boundary Evaluation — 2× in this paper
• Large Language Models (LLMs) — 1× in this paper · explained in 6 episodes
• Sycophancy — 1× in this paper · explained in 4 episodes

Sources

• Jailbreaking Black Box Large Language Models in Twenty Queries
• Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
• Adversarial D'ej\`a Vu: Jailbreak Dictionary Learning for Stronger Generalization to Unseen Attacks
• h4rm3l: A language for Composable Jailbreak Attack Synthesis
• SuperGPQA: Scaling LLM Evaluation across 285 Graduate Disciplines
• AILuminate: Introducing v1.0 of the AI Risk and Reliability Benchmark from MLCommons
• Growing Pains: Extensible and Efficient LLM Benchmarking Via Fixed Parameter Calibration
• Fluid Language Model Benchmarking
• Measuring Sycophancy of Language Models in Multi-turn Dialogues
• WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Models
• Adaptive Testing for LLM Evaluation: A Psychometric Alternative to Static Benchmarks
• HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal
• GPQA: A Graduate-Level Google-Proof Q&A Benchmark
• A StrongREJECT for Empty Jailbreaks
• Reliable and Efficient Amortized Model-based Evaluation
• AIR-Bench 2024: A Safety Benchmark Based on Risk Categories from Regulations and Policies
• FalseReject: A Resource for Improving Contextual Safety and Mitigating Over-Refusals in LLMs via Structured Reasoning
• Lost in Benchmarks? Rethinking Large Language Model Benchmarking with Item Response Theory
• Instruction-Following Evaluation for Large Language Models
• Universal and Transferable Adversarial Attacks on Aligned Language Models

Related papers

• MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
• Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
• Agents' Last Exam
• The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
• MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
• Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems