PReMISE: Policy Rubrics as Measurement Specifications for LLM Judges
PREMISE is a system that discovers high-quality policy rubrics from human preference data and validates them using LLM judges. It moves beyond simple scoring by simultaneously measuring applicability, source specificity, and dimensionality. The framework includes discovery, rigorous auditing across four axes (adequacy, reliability, fit, robustness), and repair operations to improve judge accuracy and reduce adversarial exploitation.