Safety is Contextual, LLM-Judges Are Not: Navigating the Rigid Priors of Evaluators
cs.AI
Submitted: 2026-06-05
Updated: 2026-10-04
Terminology
Sources
- Command A: An Enterprise-Ready Large Language Model
- AILuminate: Introducing v1.0 of the AI Risk and Reliability Benchmark from MLCommons
- Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations
- CultureGuard: Towards Culturally-Aware Dataset and Guard Model for Multilingual Safety Applications
- The Scales of Justitia: A Comprehensive Survey on Safety Evaluation of LLMs
- LinguaSafe: A Comprehensive Multilingual Safety Benchmark for Large Language Models
- A Coin Flip for Safety: LLM Judges Fail to Reliably Measure Adversarial Robustness
- MM-Eval: A Multilingual Meta-Evaluation Benchmark for LLM-as-a-Judge and Reward Models
- Beyond Accuracy: Policy Invariance as a Reliability Test for LLM Safety Judges
- Instruction-Following Evaluation for Large Language Models
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems