Don't Judge Code by Its Cover: Exploring Biases in LLM Judges for Code Evaluation
The paper 'Don't Judge Code by Its Cover' investigates systematic biases in LLM code evaluation. Researchers found that LLMs penalize code based on its superficial appearance, regardless of functional correctness. This bias is tied to training data patterns. The conclusion is that relying solely on LLM grading is insufficient; a robust validation strategy requires incorporating test-case generation.