LLMs are Brittle to Simple Code Transformations: Introducing CETBench - A Benchmark for Code-Equivalence Checking

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Hi, I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "LLMs are Brittle to Simple Code Transformations".

Jane: Here's the gist of the paper. The gist The CETBench dataset is constructed via transformations over programs to benchmark LLMs for code-equivalence checking, revealing that very simple code transformations can result in a significant drop in performance of SOTA LLMs for the task of code-equivalence checking How it works The CETBench dataset is constructed by taking a pair of programs from a repository and applying a random series of pre-defined code transformations to generate (non-)equivalent pairs.

Tom: This process involves starting with a repository of programs written to solve various programming tasks, constructing pairs that may solve the same or different tasks, and then subjecting each pair to a sequence of transformations such as variable renaming or swap of if-else blocks.

Jane: The perturbation pipeline in Algorithm 1 formalizes this procedure for generating perturbed code pairs via controlled semantic modifications.

Tom: The algorithm randomly decides whether the full sequence should preserve the original semantic relationship between the program pair or introduce a semantic break.

Title and authors: Jane: Well, this paper introduces CETBench as a novel benchmark constructed by applying random series of pre-defined code transformations over a repository of programs to check for code equivalence. The goal is to see if LLMs can accurately determine if two programs are functionally equivalent or not after these changes.

Tom: So it’s taking existing pairs, mixing in pairs that solve the same task versus pairs that solve different tasks, and then running those pairs through a sequence of transformations like variable renaming or swapping blocks.

Lu: They note that some of these transformations are semantic-preserving, or sp, while others are not preserving meaning at all, or np. This process generates the set of perturbed pairs we call CETBench.

Meng: The construction involves starting with a repository from competitive programming problems and creating pairs that might solve the same task or different tasks initially before applying these controlled semantic modifications to create the final dataset of perturbed pairs.

Tom: So it’s not just a static test; it’s a dynamic process where they are actively creating new, potentially broken, code examples for the LLMs to judge.

Jane: The resulting dataset is structured with a prefix that describes the initial relationship between the pair and a flag that indicates the net semantic effect of all those transformations.

Lu: They perform a detailed analysis of relative performance across different pair categories both before and after introducing these perturbations, which is what really sets this work apart from previous benchmarks like GPTCloneBench or EqBench.

Meng: It’s important to note that CETBench applies composed sequences of up to seven diverse transformations, often across distinct but algorithmically similar programs.

Tom: That depth of transformation sequence seems key; it’s not just one small change, it’s a whole chain of modifications that can confuse the model.

Jane: It really shows the limitations of older benchmarks because they were limited in scale and diversity, like EqBench which only had one hundred forty-seven equivalent and one hundred twenty-five non-equivalent C/Java pairs <ref:2506.04019#pg3,147 equivalent and 125 non-equivalent C/Java pairs>.

Lu: This work is pushing the evaluation toward models like GPT-4o, which are frontier models we're currently seeing used for testing these kinds of complex reasoning tasks <ref:2506.04019#pg1>.

Meng: It’s good to see them extend beyond just single-step transformations, which is where many simpler benchmarks fall short.

Tom: So the summary is that CETBench is a new way to stress test LLMs on code equivalence by systematically breaking those equivalences with controlled semantic modifications.

The paper's summary: Jane: The paper suggests a simple fine-tuning approach as a remedy for the performance drop they observed on these transformed pairs. They found that fine-tuning using a combination of both the perturbed and unperturbed datasets yields better results than just using one dataset alone.

Tom: So, they found that if you train the model on both normal code and this messy, perturbed code together, it helps boost its performance on both types of test sets.

Lu: They also suggest that for models like Mistral or Qwen that are weaker, explicitly asking them for a chain of thought helps them significantly in both the original pairs and the perturbed pairs when you prompt them without that extra guidance.

Meng: It sounds like the solution isn't just about better data, but about using specific prompting techniques to guide weaker models through this complex reasoning task.

Jane: They also suggest analyzing how sensitive different transformation types are; they found that logic-altering perturbations like boolean inversion or if-condition flipping have a slightly higher delta in performance drop compared to simple renaming.

Tom: So, the paper is giving us a way to tune our prompting strategy based on what specific type of code manipulation is causing the most trouble for a given model.

Lu: They also point toward model specialization, suggesting that different models might perform better under certain perturbation scenarios; for example, Deepseek-Coder showed high performance in some of those challenging scenarios.

Meng: If we can identify which models specialize well in handling specific types of code changes, it gives us a clearer path for deploying them where they’ll actually be useful in production.

Tom: So the authors are suggesting we move from a one-size-fits-all approach to more targeted fine-tuning and prompting based on the specific failure mode.

Jane: It seems like they are acknowledging that we don't need a single magic fix, but rather a combination of better data mixing and smarter ways to prompt the models.

The paper's improvements: Tom: Overall, they conclude that LLMs are definitely brittle to simple code transformations, especially when we look at how they handle code equivalence checking. The key finding is that their performance drops significantly on these perturbed datasets for almost all SOTA LLMs.

Jane: They emphasize that while models *can* work with perturbations, they absolutely require explicit supervision and training to effectively deal with this task well.

Lu: They wrap up by saying future work should include experimenting with additional datasets and figuring out how we can get models to provide explanations for their equivalence judgments.

Meng: I think getting those explanations is the next big thing because it moves us from just knowing *if* the answer is right to understanding *why* it was right or wrong.

Tom: So, they are looking toward better interpretability as a way to push beyond statistical pattern matching for this task of code equivalence checking.

Jane: It’s a solid piece of work because it clearly defines the challenge and points us toward needing more than just bigger models; we need better training strategies and better reasoning capabilities.

Lu: The CETBench benchmark gives us a concrete way to measure this brittleness, which is valuable for directing future research in representation learning for code tasks.

Meng: For practical deployment, this means we have a clearer idea of the kinds of adversarial inputs that could cause our deployed models to fail during equivalence checks in real traffic.

Tom: So it’s a good warning sign: simple changes are surprisingly hard for LLMs to handle semantically, and we need more targeted methods to make them reliable for this kind of judging.

Conclusion: Tom: So we've been looking at CETBench, which is this new benchmark designed to see how brittle Large Language Models are when checking if two pieces of code are equivalent after some simple changes have been made.

Jane: Right, so the core idea is they take pairs of programs—maybe two that solve the same problem—and they apply a random series of transformations, like renaming variables or swapping if-else blocks, to see if the AI can still tell them apart.

Lu: They constructed this dataset by extending existing work, taking stuff from CodeContests and creating these perturbed pairs labeled with prefixes that describe the initial relationship and flags that show what happened after the transformations.

Meng: What I find interesting is how they generate these pairs; they decide randomly whether to keep the original meaning or introduce a break in meaning at each step of up to Kmax perturbations. It’s a controlled way to break things.

Lalam: Lalam here, from my side, I see that the analysis shows a significant drop in performance across SOTA LLMs for code-equivalence checking when they look at these perturbed datasets.

Tom: That confirms what we suspected, Lu—the simple transformations cause a big hit to their accuracy on this task. But what about the fix? What’s the solution they propose?

Jane: The remedy they suggest is a fine-tuning approach, but it's not just fine-tuning on the perturbed data alone; they found that combining pairs from both the original and the perturbed datasets gives better results than training only on one set.

Lu: They show that LLMs do have some capability with these perturbations, but they absolutely need that explicit supervision to handle this task effectively.

Meng: From a production standpoint, if we want to deploy models for code checking, this tells us we can’t just rely on out-of-the-box performance; we gotta supervise the training process with this kind of mixed data.

Tom: And there's more than just fine-tuning. They also found that for models like Mistral and Qwen, explicitly asking them to follow a chain of thought gives them a significant boost in both the original and the perturbed pairs compared to just prompting without that extra guidance.

Jane: That makes sense; when the logic gets twisted by those transformations, they need that structured reasoning help to keep track of what they're supposed to be checking.

Lu: They also pinpoint specific types of transformations where models struggle more—like boolean inversion or if-condition flipping—showing where their understanding breaks down most severely.

Meng: So, we can start thinking about tuning our prompting based on the type of code change we expect to see in real-world scenarios, rather than just hoping the model figures it out.

Tom: It sounds like CETBench isn't just a test; it’s a roadmap showing us exactly what kind of data and training strategy we need to make LLMs actually reliable for these kinds of critical code checks.

Jane: It really highlights that even simple changes in code can have a big impact on how much an AI understands the underlying meaning.

Lu: So, while they point out limitations, like only testing Python and using a binary equivalence scheme, the main message is about how to supervise these models effectively.

Meng: I'm still cautious because this evaluation is restricted to one language; we need more breadth before we trust these results for general use.

Lalam: Lalam confirms that the paper demonstrates that with targeted fine-tuning and better prompting, the performance gap on those transformed pairs shrinks considerably for most models tested.

Tom: So, to wrap up, CETBench is a crucial benchmark showing us that code equivalence checking requires more than just raw pattern matching from LLMs; it needs careful supervision and tailored training data.

Jane: That’s what this paper suggests about how we should approach making these models more robust for real-world tasks.

Lu: The authors conclude that future work should focus on getting better explanations from the models themselves so we can understand *why* they think two programs are equivalent or not.

Meng: I agree, moving toward explainability is a necessary step before we can really deploy these tools in high-stakes environments.

Tom: So, CETBench gives us a clear idea of the brittle spots in current LLM code reasoning, and now we have some direction for how to improve them through better training and prompting.

Jane: Next time we talk about benchmarks, I want to look at something that shows how models handle visual reasoning under stress.