Benchmarking Code LLM-as-a-Judge via Constrained Perturbation and Calibration
The AXIOM framework reframes code evaluation as measuring refinement effort to create balanced benchmarks for LLM-as-a-Judge metrics. It uses rule-guided perturbation to synthesize programs with controlled quality and a multisource calibration stage to streamline human annotation. This allows researchers to rigorously test existing metrics against a diverse, high-quality dataset.