From Conversation to Evaluation: Benchmarking LLMs on Development Knowledge via SimpleDevQA
cs.SE
Submitted: 2025-12-09
Updated: 2026-11-13
DOI: 10.18653/v1/2026.findings-acl.1877
Code: https://github.com/openai/tiktoken
Project page: https://evalplus.github.io/leaderboard.html
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Terminology
Sources
- Program Synthesis with Large Language Models
- The TechQA Dataset
- Humans or LLMs as the Judge? A Study on Judgement Biases
- CoReQA: Uncovering Potentials of Language Models in Code Repository Question Answering
- Evaluating Large Language Models Trained on Code
- FacTool: Factuality Detection in Generative AI -- A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarios
- Quasar: Datasets for Question Answering by Search and Reading
- The Llama 3 Herd of Models
- A Survey on LLM-as-a-Judge
- Deliberative Alignment: Reasoning Enables Safer Language Models
- DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence
- SWE-Factory: Your Automated Factory for Issue Resolution Training Data and Evaluation Benchmarks
- From Code to Courtroom: LLMs as the New Software Judges
- Chinese SimpleQA: A Chinese Factuality Evaluation for Large Language Models
- Measuring Massive Multitask Language Understanding
- CodeRepoQA: A Large-scale Benchmark for Software Engineering Question Answering
- Qwen2.5-Coder Technical Report
- SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
- Binary Code Summarization: Benchmarking ChatGPT/GPT-4 and Other Large Language Models
- TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
Related papers
- ArkEval: Benchmarking and Evaluating Automated CodeRepair for ArkTS
- TestPrism: Rethinking Test Evaluation Beyond a Single Reference
- Mining Workflow Graphs for Black-Box Boundary Testing of Conversational LLM Agents
- Falsification-Based Verification of LLM-Generated Optimization Models: Sound Test Batteries and Their Detection Limits
- GitSkills: A Dataset of Agent Skills on GitHub
- For Coding Agents, the Repository Matters