ClarifyCodeBench: Evaluating LLMs on Clarifying Ambiguous Requirements for Code Generation

arXiv:2607.00711 · cs.SE, cs.AI · Submitted 2026-07-01 · Read on arXiv

cs.SE, cs.AI

Submitted: 2026-07-01

Updated: 2026-08-31

Comments: Code: https://github.com/fangz-cs/ClarifyCodeBench

Code: https://github.com/fangz-cs/ClarifyCodeBench

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Benchmark — 2× in this paper · explained in 2 episodes
• Decoupling — 1× in this paper · explained in 2 episodes

Sources

• Conversational AI as a Coding Assistant: Understanding Programmers' Interactions with and Expectations from Large Language Models for Coding
• Program Synthesis with Large Language Models
• Evaluating Large Language Models Trained on Code
• Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
• DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
• ConvCodeWorld: Benchmarking Conversational Code Generation in Reproducible Feedback Environments
• Measuring Coding Challenge Competence With APPS
• GPT-4o System Card
• LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code
• LLMs Get Lost In Multi-Turn Conversation
• When Prompts Go Wrong: Evaluating Code Model Robustness to Ambiguous, Contradictory, and Incomplete Task Descriptions
• DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models
• OpenAI GPT-5 System Card
• Ambig-SWE: Interactive Agents to Overcome Underspecificity in Software Engineering
• CodeIF-Bench: Evaluating Instruction-Following Capabilities of Large Language Models in Interactive Code Generation
• CodeFlowBench: A Multi-turn, Iterative Benchmark for Complex Code Generation
• Qwen3 Technical Report

Related papers

• Falsification-Based Verification of LLM-Generated Optimization Models: Sound Test Batteries and Their Detection Limits
• GitSkills: A Dataset of Agent Skills on GitHub
• SABER: Benchmarking Operational Safety of LLM Coding Agents in Stateful Project Workspaces
• PackMonitor: Enabling Zero Package Hallucinations Through Decoding-Time Monitoring
• IntentCoding: Amplifying User Intent in Code Generation
• Incentives and Outcomes in Bug Bounties