When Talk Isn't Code: Comparing LLM Agents That Simulate and Develop Software
cs.SE, cs.CR
Submitted: 2026-10-03
Updated: 2026-10-03
Sources
- RAS-Eval: A Comprehensive Benchmark for Security Evaluation of LLM Agents in Real-World Environments
- An Empirical Evaluation of LLM-Generated Code Security Across Prompting Methods
- How Well Can LLM Agents Simulate End-User Security and Privacy Attitudes and Behaviors?
- SecQA: A Concise Question-Answering Dataset for Evaluating Large Language Models in Computer Security
- What Does an Agentic Software Engineering Benchmark Measure? Profiling Task Demands and Agent Behaviour Beyond What Category Labels Reveal
- Adversarial Stress Testing of Role-Playing Language Agents using Multi-Agent Evaluation
- The Illusion of Secure LLM Code: Closing the Security Gap via Iterative Reprompting
Related papers
- Falsification-Based Verification of LLM-Generated Optimization Models: Sound Test Batteries and Their Detection Limits
- GitSkills: A Dataset of Agent Skills on GitHub
- SABER: Benchmarking Operational Safety of LLM Coding Agents in Stateful Project Workspaces
- PackMonitor: Enabling Zero Package Hallucinations Through Decoding-Time Monitoring
- IntentCoding: Amplifying User Intent in Code Generation
- Incentives and Outcomes in Bug Bounties