Benchmarking Code LLM-as-a-Judge via Constrained Perturbation and Calibration

arXiv:2512.20159 · cs.SE, cs.AI · Submitted 2025-12-23 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Benchmarking Code LLM-as-a-Judge via Constrained Perturbation and Calibration".

Jane: The gist The proposed AXIOM framework reframes code evaluation as measuring refinement effort, enabling the creation of large-scale, balanced benchmarks to rigorously test existing LLM-as-a-Judge metrics.

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So, we're looking at this paper today, "Benchmarking Code LLM-as-a-Judge via Constrained Perturbation and Calibration." It sounds like they're trying to fix some big problems with how we evaluate code written by large language models.

Jane: Yeah, the title itself tells you a lot about what they’re doing here. They're using some kind of constrained perturbation and calibration to test those LLM-as-a-Judge metrics, which have been around for a while.

Lu: What this paper is setting up is a way to get past the limitations of existing code evaluation benchmarks that are often small and unbalanced, or have unreliable scores. They're aiming for something much bigger and more reliable for meta-evaluating those LLM-as-a-judge metrics themselves, which is a pretty ambitious goal.

Meng: I wonder if this means they're building a benchmark that actually reflects what real software engineering looks like, instead of just surface-level similarity.

Lalam: It seems like the core idea is to move away from simple scoring and toward measuring the actual refinement effort needed to get code ready for deployment, which is a much more concrete way to look at code quality.

The paper's summary: Tom: So, AXIOM, that's what they call their framework. The main idea is reframing code evaluation not as judging the final output directly, but as measuring the hypothetical effort needed to repair programs for deployment.

Jane: That’s a big shift. Instead of just saying "this code is good," they want to assign scores based on how much work it would take to polish it up, which gives them a really fine-grained, interpretable score.

Lu: They achieve this through a two-stage process. First is rule-guided perturbation where they prompt LLMs to modify existing high-quality programs using sequences of predefined rules to control the target score precisely.

Meng: So they're not just asking the LLM to generate new code, but making it change something that already exists, and controlling the outcome with these rules. That sounds like it would be much more controllable than letting an LLM go wild on its own.

Lalam: And then there’s the second stage, multisource quality calibration where they use a mix of LLMs, static analysis reports, and execution reports to determine the final score for each perturbed program.

The paper's improvements: Tom: The paper outlines several improvements over what we have now. They propose using rule-guided perturbation to synthesize programs with predetermined target scores, which helps them create a large-scale multilingual benchmark with one thousand nine hundred sixty-two problems in a balanced score distribution <ref:2512.20159#pg1>.

Jane: That sounds like they are solving the problem of having small, unbalanced benchmarks by systematically generating programs across different quality levels. They use exemplar augmentation to get more production-ready programs and deteriorative perturbation to make realistic flaws.

Lu: They also have this multisource quality calibration step where human annotators don't do complex logic reasoning anymore. They just review the difference between the program and its parent, using all that context to calibrate the final score into a ground truth score.

Meng: That’s smart because it frees up the human annotators from having to act like deep logic experts, which is a major hurdle in these kinds of studies. They are simplifying their task to focusing on the diff and using reports from multiple sources.

Lalam: They also mention representative candidate selection, where they use metrics like normalized token edit distance and a greedy approach to pick the most diverse set of programs for final calibration, making sure the benchmark isn't just focused on one type of code.

Conclusion: Tom: So, to wrap up, AXIOM is this two-stage framework that uses perturbation to create a large-scale benchmark and then uses multisource calibration to get reliable scores for LLM-as-a-Judge metrics.

Jane: The main implication is that it shows we can synthesize code evaluation data in a way that's grounded in measurable effort, rather than just subjective judgment. It helps us meta-evaluate those judges themselves.

Lu: It’s about creating a foundation for testing those LLM-as-a-judge metrics rigorously across a much wider range of scenarios and score distributions.

Meng: From an engineering standpoint, the focus on controlling the scores via perturbation sounds like it addresses the inconsistency issues we see when we try to run these metrics at scale on real traffic.

Lalam: I think this work provides a way to systematically generate data that’s balanced and reliable enough for serious research into how good those LLM-as-a-judge metrics really are.

Harbin Institute of Technology Institute of Technology Shenzhen University Monash University Malaysia Zhejiang University

cs.SE, cs.AI

Submitted: 2025-12-23

Updated: 2026-10-15

Code: https://github.com/containers/bubblewrap8https:

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 86/100

The gist: The gist The proposed AXIOM framework reframes code evaluation as measuring refinement effort, enabling the creation of large-scale, balanced benchmarks to rigorously test existing LLM-as-a-Judge

Key concepts

Rule-Guided Perturbation: This stage uses LLMs and predefined rules to systematically modify existing code. It creates programs with specific target quality scores (e.g., 1 to 5 points) by applying sequences of controlled changes, ranging from minor flaws to complete context disruption.
Multisource Quality Calibration: Instead of complex logic, human annotators review the differences between a program and its original high-quality version. They use multiple data sources—like static analysis reports and LLM quality assessments—to calibrate the final score, simplifying subjective judgment.
Refinement Effort as Score: AXIOM changes how scores are viewed: instead of judging the final code directly, the score represents how much effort (refinement) is needed to get there. This metric is more objective and helps control program quality distribution across a benchmark.

Terminology

Benchmark — 4× in this paper · explained in 2 episodes
Calibration — 2× in this paper · explained in 7 episodes
LLM-as-a-Judge — 2× in this paper · explained in 3 episodes
Benchmarking — 1× in this paper · explained in 4 episodes
Large Language Models (LLMs) — 1× in this paper · explained in 8 episodes

Summary

The gist The proposed AXIOM framework reframes code evaluation as measuring refinement effort, enabling the creation of large-scale, balanced benchmarks to rigorously test existing LLM-as-a-Judge metrics.

AXIOM Framework Overview

AXIOM is a novel perturbation-based framework designed to automate the creation of code evaluation benchmarks at scale by reframing program scores as the refinement effort needed for deployment This framework consists of two main stages: (1) Rule-guided perturbation, which prompts LLMs to apply sequences of predefined perturbation rules to existing high-quality programs to modify their functionality and code quality, enabling us to precisely control each program’s target score (2) Multisource quality calibration, which first selects a subset of semantically distinct programs to ensure benchmark diversity, and then performs a streamlined manual annotation procedure with multisource context from LLMs, static analysis, and execution report. Based on this framework, the authors curate AXIOMBench, a large-scale multilingual benchmark with 1962 code evaluation problems in a balanced score distribution and reliable annotated scores.

Rule-Guided Perturbation

The first stage of AXIOM utilizes rule-guided perturbation to synthesize programs with predetermined target scores. This process involves three specific steps:

Exemplar augmentation, which generates more production-ready (5-point) programs besides the reference programs in the original code generation benchmark.
Deteriorative perturbation, which synthesizes 1- to 4-point programs from 5-point ones. This step uses a sequence of up to Nmax = 5 perturbation steps to create more complex and realistic flaws than a single change.
Context disruption, which produces worthless 0-point programs by pairing a program from one requirement with another semantically unrelated requirement.

The perturbation step transforms program p into p' with perturbation rule u, where u consists of a natural language instruction on how to perform the code transformation, and a score ceiling c(u) ∈ [1, 5], indicating that the perturbed program’s score s(p') is capped at c(u). The final program p' results in a score equal to min(s(p), c(u1), …, cn).

Multisource Quality Calibration

The second stage, multisource quality calibration, focuses on determining the final score s for each perturbed program p. The core principle is to decouple the assessment of functionality and code quality. Human annotators are freed from complex logic reasoning and instead focus on a simpler judgment: reviewing the diff between a program p and its 5-point pre-perturbation parent, determining p’s code quality with multisource information, and calibrating s(p) into the final score s.

This stage includes two sub-steps:

Representative Candidate Selection, which selects a balanced and diverse subset of candidate programs for final calibration. This is achieved by quantifying the dissimilarity between all programs with normalized token edit distance and employing a greedy approach described in Algorithm 2 to select candidates.
Multisource-Informed Score Calibration, which calibrates the target score s(p) into a final, human-verified groundtruth score s. The annotator is provided with a multisource diagnosis report including the diff comparison between p and its 5-point parent, the target score s(p), the sequence of perturbation rules, a static analysis report, and an LLM-generated code quality report.

Benchmark and Study Design

AXIOMBench is curated using four recent and challenging code generation benchmarks: BigCodeBench [34], LiveCodeBench [8], APPS [7], and Aider-Polyglot5. Requirements are sampled randomly from these benchmarks, and reference programs serve as the initial, 5-point programs after being verified on test cases. During perturbation, a cost-efficient LLM is sampled for each step to generate M = 2 programs per target score per requirement with temperature T = 0.3 and output length limit to 8192 tokens. For the final context disruption step, OpenAI’s text-embedding-3-large is used as the embedding model. In the multisource quality calibration stage, lexers from Pygments are used to tokenize programs before computing distances and selecting up to m = 90 programs per benchmark per target score. To validate objectivity, a preliminary annotation study with 10% randomly selected data was conducted with three experts, achieving a Krippendorff’s α of 96.40%.

Study Results

The experiments investigate the performance of existing LLM-as-a-judge metrics on AXIOMBench. Finding 1 reveals a trade-off between LLM capability and metric complexity, noting that LLMs with stronger evaluation capabilities achieve peak performance with simple metrics with a single inference pass. Finding 2 highlights a gap between ranking ability and human scoring criteria, as simpler metrics yielding higher agreement are superior to complex ones like CodeJudge and CodeVisionary. Finding 4 demonstrates that current LLM-as-a-judge metrics exhibit a pronounced tendency to overestimate the scope of changes, frequently misclassifying simple localized corrections as substantial architectural changes.

Furthermore, the study found that no tested metric can reliably perform refinement effort estimation, with extremely low MCC below 10.0, which is barely better than random guessing. Finding 5 shows that the procedural complexity of an LLM-as-a-judge metric directly impacts its scoring consistency, where complex agentic metrics suffer from enlarged scoring variance due to accumulated uncertainty.

In conclusion, AXIOM provides a framework to synthesize code evaluation data to meta-evaluate existing LLM-as-a-judge metrics for code. The paper successfully addresses issues of previous benchmarks by proposing a two-stage procedure where the first guides LLMs to perform program perturbation and the second calibrates scores using a multisource approach with reduced human effort.

How it works

Rule-Guided Perturbation

The first stage of AXIOM utilizes rule-guided perturbation to synthesize programs with predetermined target scores This process involves three specific steps:

Exemplar augmentation, which generates more production-ready (5-point) programs besides the reference programs in the original code generation benchmark.
Deteriorative perturbation, which synthesizes 1- to 4-point programs from 5-point ones This step uses a sequence of up to Nmax = 5 perturbation steps to create more complex and realistic flaws than a single change.
Context disruption, which produces worthless 0-point programs by pairing a program from one requirement with another semantically unrelated requirement.

Multisource Quality Calibration

The second stage, multisource quality calibration, focuses on determining the final score s for each perturbed program p The core principle is to decouple the assessment of functionality and code quality Human annotators are freed from complex logic reasoning and instead focus on a simpler judgment: reviewing the diff between a program p and its 5-point pre-perturbation parent, determining p’s code quality with multisource information, and calibrating s(p) into the final score s.

This stage includes two sub-steps:

Representative Candidate Selection, which selects a balanced and diverse subset of candidate programs for final calibration This is achieved by quantifying the dissimilarity between all programs with normalized token edit distance and employing a greedy approach described in Algorithm 2 to select candidates.

Improvements for AI systems

textbf Rule-guided perturbation for precise score control: The framework reframes program scores as the refinement effort needed for deployment, allowing researchers to precisely control each program’s target score to achieve balanced score distributions by using Rule-guided perturbation, which prompts LLMs to apply sequences of predefined perturbation rules.
textbf Multisource quality calibration for reliable annotation: This stage addresses the subjectivity of human scoring by narrowing the annotator's task to reviewing the diff between a program p and its 5-point pre-perturbation parent, determining p’s code quality with multisource information, and calibrating s(p) into the final score s.
textbf Creation of AXIOMBench for balanced evaluation: AXIOM synthesizes a large-scale multilingual benchmark with 1962 code evaluation problems in a balanced score distribution by generating programs via Exemplar augmentation, Deteriorative perturbation, and Context disruption, ensuring the benchmark is diverse and balanced.
textbf Enhanced metric performance analysis: Experiments reveal that current LLM-as-a-judge metrics, while powerful at relative ranking, suffer from issues such as hallucinating flaws in functionality and code quality, guiding the development of more robust evaluation strategies.
textbf Refinement effort estimation (RQ2): The framework allows for the study of refinement effort by isolating refactor (positive) and tweak (negative) classes within a program's score, though current metrics are shown to have an extremely low MCC below 10.0 for this task.

Abstract

Code from LLM-assisted development may contain defects like API hallucinations, motivating the development of code evaluation metrics to assess such code before deployment. As conventional rule-based metrics merely score code based on similarity to reference programs, and execution-based metrics require expensive manual preparation for tests, researchers have developed LLM-as-a-judge metrics for both behavioral and nonbehavioral code evaluation, and curated multiple benchmarks to validate their effectiveness. However, such benchmarks suffer from critical limitations: Some feature coarse-grained binary labels of passing/failing unit tests, unable to distinguish between fundamental and minor flaws; others utilize fine-grained criteria with manually annotated scores as ground-truth, but such criteria are often vaguely defined, introducing subjectivity and unreliability in human scores. Furthermore, they often directly prompt LLMs to generate code for evaluation, which may lead to unbalanced code quality distributions, failing to test the metrics' discriminative capability across varying quality levels. To curate a diverse benchmark with a balanced program quality distribution and streamline manual annotation, we propose AXIOM, a novel perturbation-based framework. It reframes program scores as the refinement effort needed for deployment, with two stages: (1) Quality-constrained semantic perturbation, prompting LLMs to apply sequences of predefined perturbation rules to existing programs, precisely controlling each program's target quality to achieve a balanced distribution. (2) Hybrid quality calibration, selecting semantically distinct programs for benchmark diversity, and performing streamlined manual annotation with multisource context. With AXIOM, we curate AXIOMBENCH, a large-scale multilingual benchmark with 1,962 code evaluation problems in a more balanced score distribution...