LLMs are Brittle to Simple Code Transformations: Introducing CETBench - A Benchmark for Code-Equivalence Checking
CETBench is a new benchmark testing Large Language Models (LLMs) for code-equivalence checking after applying simple code transformations. The study found that even very small, random changes to programs can cause a significant drop in LLM performance for determining if two pieces of code mean the same thing. This suggests current LLMs rely too much on statistical patterns rather than deep semantic understanding of logic.