Impossible Code, Barely Changed Score: A Replay Study of Judge-Based Evaluation
cs.SE, cs.AI
Submitted: 2026-11-01
Updated: 2026-11-01
Code: https://github.com/curiousorbitnotes/evorobust-replay-audit
Project page: https://curiousorbitnotes.github.io/evorobust-replay-audit
Terminology
Sources
- Program Synthesis with Large Language Models
- Evaluating Large Language Models Trained on Code
- LLMs Gaming Verifiers: RLVR can Lead to Reward Hacking
- Tulu 3: Pushing Frontiers in Open Language Model Post-Training
- AlphaEvolve: A coding agent for scientific and algorithmic discovery
- Qwen2.5 Technical Report
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- Benchmarking Code LLM-as-a-Judge via Constrained Perturbation and Calibration
- Position: The Hidden Costs and Measurement Gaps of Reinforcement Learning with Verifiable Rewards
Related papers
- ArkEval: Benchmarking and Evaluating Automated CodeRepair for ArkTS
- TestPrism: Rethinking Test Evaluation Beyond a Single Reference
- Mining Workflow Graphs for Black-Box Boundary Testing of Conversational LLM Agents
- Falsification-Based Verification of LLM-Generated Optimization Models: Sound Test Batteries and Their Detection Limits
- GitSkills: A Dataset of Agent Skills on GitHub
- For Coding Agents, the Repository Matters