To Jev or Not to Jev: When Can System-1 Decision Models Replace LLM Judges?
cs.AI
Submitted: 2026-10-31
Updated: 2026-10-31
Code: https://github.com/jaredpalmer/kev
Terminology
Sources
- FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance
- TICKing All the Boxes: Generated Checklists Improve LLM Evaluation and Generation
- Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators
- How to Evaluate Reward Models for RLHF
- A Survey on LLM-as-a-Judge
- On Calibration of Modern Neural Networks
- Trust or Escalate: LLM Judges with Provable Guarantees for Human Agreement
- Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models
- RewardBench: Evaluating Reward Models for Language Modeling
- CheckEval: A reliable LLM-as-a-Judge framework for evaluating text generation using checklists
- JEV-as-a-Judge: Accept When Confident, Escalate When Unsure
- G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment
- RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style
- AgentRewardBench: Evaluating Automatic Evaluations of Web Agent Trajectories
- RewardBench 2: Advancing Reward Model Evaluation
- s1: Simple test-time scaling
- OffsetBias: Leveraging Debiased Data for Tuning Evaluators
- Is LLM-as-a-Judge Robust? Investigating Universal Adversarial Attacks on Zero-shot LLM Assessment
- Optimization-based Prompt Injection Attack to LLM-as-a-Judge
- JudgeBench: A Benchmark for Evaluating LLM-based Judges
Related papers
- Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Embeddings for Preferences, Not Semantics
- Can We Trust Item Response Theory for AI Evaluation?
- Diagnosing Task Insensitivity in Language Agents