Follow the Policy, Fail the Benchmark: Auditing and Revising Policy-Based Decision Benchmarks
cs.CL, cs.SE
Submitted: 2026-11-04
Updated: 2026-11-04
Code: https://github.com/SugarRuy/BAR-Policy-Decision
Project page: https://sharc-data.github.io/data/sharc1-official.zip
Terminology
Sources
- Policy-as-logic for robust reasoning over rules
- DeonticBench: A Benchmark for Reasoning over Rules
- HLE-Verified: A Systematic Verification and Structured Revision of Humanity's Last Exam
- The Benchmark Lottery
- ELT-Bench-Verified: Benchmark Quality Issues Underestimate AI Agent Capabilities
- UniGuard: Multi-View Guardrail Learning for Cross-Interface Safety Generalization
- TeleTune: Evolving Agent Skills From Offline Telemetry
- Is Bash All You Need? An Empirical Study of Tool Interfaces for Enterprise Digital Worker Agents
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data