DualGauge: Automated Joint Security-Functionality Benchmarking of Specification-Only Code Generation by LLMs and Coding Agents
cs.SE, cs.AI, cs.CR
Submitted: 2025-11-24
Updated: 2026-10-28
Comments: Accepted to Findings of the Association for Computational Linguistics: EMNLP 2026
Code: https://github.com/SRestLabUB/DualGauge
Project page: https://srestlabub.github.io/DualGauge
License: http://creativecommons.org/licenses/by/4.0/
The gist: Large language models (LLMs) and LLM-based coding agents are now used to generate code from natural-language specifications…
Terminology
Abstract
Large language models (LLMs) and LLM-based coding agents are now used to generate code from natural-language specifications, yet ensuring such code is both functionally correct and secure remains a challenge. We present DualGauge, the first fully automated framework for jointly evaluating correctness and security of specification-only code generation, supported by DualGauge-Bench, a language-agnostic benchmark of 307 coding tasks each paired with functional and security tests derived from the same specification. Evaluating 10 representative LLMs across Python, C++, and JavaScript, we find that functional correctness substantially overestimates reliable code generation: even the strongest model remains below 15% joint security-functionality success in every language. Common model-side factors--scale, extended thinking, quantization, instruction tuning, and code specialization--do not reliably improve joint performance, suggesting secure-and-correct code generation does not simply emerge from stronger coding capability. Evaluation of 3 leading coding agents (Codex, OpenHands, and Claude Code) shows that iterative scaffolding is not a reliable remedy on specification-only tasks: results are strongly scaffold-dependent, with two of the three agents failing to outperform a comparable direct model. A qualitative audit reveals that these failures concentrate at the output contract boundary and in guards that exist but are insufficient--patterns that only joint benchmarking reliably exposes.
Sources
- SetupBench: Assessing Software Engineering Agents' Ability to Bootstrap Development Environments
- Program Synthesis with Large Language Models
- SecureVibeBench: Benchmarking Secure Vibe Coding of AI Agents via Reconstructing Vulnerability-Introducing Scenarios
- Evaluating Large Language Models Trained on Code
- EnvBench: A Benchmark for Automated Environment Setup
- Constrained Decoding for Secure Code Generation
- Unlocking Reproducibility: Automating re-Build Process for Open-Source Software
- Measuring Coding Challenge Competence With APPS
- The Rise of AI Teammates in Software Engineering (SE) 3.0: How Autonomous Coding Agents Are Reshaping Software Engineering
- SafeGenBench: A Benchmark Framework for Security Vulnerability Detection in LLM-Generated Code
- The Impact of LLM-Assistants on Software Developer Productivity: A Systematic Review and Mapping Study
- BaxBench: Can LLMs Generate Correct and Secure Backends?
- AI Agentic Programming: A Survey of Techniques, Challenges, and Opportunities
- Enhancing Large Language Models for Secure Code Generation: A Dataset-driven Study on Vulnerability Mitigation
- SeCodePLT: A Unified Platform for Evaluating the Security of Code GenAI
- Can Language Models Go Beyond Coding? Assessing the Capability of Language Models to Build Real-World Systems
- Is Vibe Coding Safe? Benchmarking Vulnerability of Agent-Generated Code in Real-World Tasks
Related papers
- ArkEval: Benchmarking and Evaluating Automated CodeRepair for ArkTS
- TestPrism: Rethinking Test Evaluation Beyond a Single Reference
- Mining Workflow Graphs for Black-Box Boundary Testing of Conversational LLM Agents
- Falsification-Based Verification of LLM-Generated Optimization Models: Sound Test Batteries and Their Detection Limits
- GitSkills: A Dataset of Agent Skills on GitHub
- For Coding Agents, the Repository Matters