Beyond Synthetic Benchmarks: Evaluating LLM Performance on Real-World Class-Level Code Generation
cs.SE, cs.AI, cs.LG
Submitted: 2025-10-30
Updated: 2026-11-15
Code: https://github.com/mrsumitbd/RealClassEvalReplication
Terminology
Sources
- gpt-oss-120b & gpt-oss-20b Model Card
- Program Synthesis with Large Language Models
- RepairAgent: An Autonomous, LLM-Based Agent for Program Repair
- Evaluating Large Language Models Trained on Code
- ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-level Code Generation
- UniXcoder: Unified Cross-Modal Pre-training for Code Representation
- GraphCodeBERT: Pre-training Code Representations with Data Flow
- Measuring Coding Challenge Competence With APPS
- Qwen2.5-Coder Technical Report
- CodeSearchNet Challenge: Evaluating the State of Semantic Code Search
- A Survey on Large Language Models for Code Generation
- GraphCodeAgent: Dual Graph-Guided LLM Agent for Retrieval-Augmented Repo-Level Code Generation
- StarCoder: may the source be with you!
- DeepSeek-V3 Technical Report
- CoDet-M4: Detecting Machine-Generated Code in Multi-Lingual, Multi-Generator and Multi-Domain Settings
- Automatic Detection of LLM-Generated Code: A Comparative Case Study of Contemporary Models Across Function and Class Granularities
- A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applications
- Automated Prompt Engineering for Cost-Effective Code Generation Using Evolutionary Algorithm
- Retrieval-Augmented Code Generation: A Survey with Focus on Repository-Level Approaches
- CodeT5+: Open Code Large Language Models for Code Understanding and Generation
Related papers
- ArkEval: Benchmarking and Evaluating Automated CodeRepair for ArkTS
- TestPrism: Rethinking Test Evaluation Beyond a Single Reference
- Mining Workflow Graphs for Black-Box Boundary Testing of Conversational LLM Agents
- Falsification-Based Verification of LLM-Generated Optimization Models: Sound Test Batteries and Their Detection Limits
- GitSkills: A Dataset of Agent Skills on GitHub
- For Coding Agents, the Repository Matters