CodeArena: A Collective Evaluation Platform for LLM Code Generation
cs.SE
Submitted: 2025-03-03
Updated: 2026-10-29
Project page: https://codearena.online
License: http://creativecommons.org/licenses/by/4.0/
The gist: Large Language Models (LLMs) have reshaped code generation by synergizing their exceptional comprehension of natural language and programming syntax…
Terminology
Abstract
Large Language Models (LLMs) have reshaped code generation by synergizing their exceptional comprehension of natural language and programming syntax, thereby substantially boosting developer productivity. These advancements have prompted numerous efforts to quantitatively evaluate their coding capabilities. However, persistent challenges, such as benchmark leakage, data dissipation, and limited system accessibility, continue to impede a timely and accurate assessment. To address these limitations, we introduce CodeArena, an online evaluation framework tailored for LLM code generation. The key innovation is a collective evaluation mechanism, which dynamically recalibrates individual model scores based on the holistic performance of all participating models, mitigating score biases caused by widespread benchmark leakage. In addition, CodeArena ensures open access to all submitted solutions and test cases and provides automation-friendly APIs to streamline the code evaluation workflow. Our main contributions are: (1) a collective evaluation system for unbiased assessment, (2) a public repository of solutions and test cases, and (3) automation-ready APIs for seamless integration.
Sources
- GPT-4 Technical Report
- CyberSecEval 2: A Wide-Ranging Cybersecurity Evaluation Suite for Large Language Models
- Purple Llama CyberSecEval: A Secure Coding Benchmark for Language Models
- Evaluating Large Language Models Trained on Code
- Mercury: A Code Efficiency Benchmark for Code Large Language Models
- EffiBench: Benchmarking the Efficiency of Automatically Generated Code
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code
- Mistral 7B
- StarCoder 2 and The Stack v2: The Next Generation
- Code Llama: Open Foundation Models for Code
- Gemini: A Family of Highly Capable Multimodal Models
- LLaMA: Open and Efficient Foundation Language Models
- DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence
- BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions
Related papers
- ArkEval: Benchmarking and Evaluating Automated CodeRepair for ArkTS
- TestPrism: Rethinking Test Evaluation Beyond a Single Reference
- Mining Workflow Graphs for Black-Box Boundary Testing of Conversational LLM Agents
- Falsification-Based Verification of LLM-Generated Optimization Models: Sound Test Batteries and Their Detection Limits
- GitSkills: A Dataset of Agent Skills on GitHub
- For Coding Agents, the Repository Matters