PRWeaver: Evaluating LLM-Based Code Auditors against Long-Horizon Malicious Pull Requests

arXiv:2608.02693 · cs.SE, cs.CR · Submitted 2026-08-03 · Read on arXiv

Yuekun Wang, Mingfei Cheng, Xiaofei Xie

cs.SE, cs.CR

Submitted: 2026-08-03

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Benchmark — 29× in this paper · explained in 2 episodes
• Trace — 5× in this paper · explained in 2 episodes
• Ablation — 4× in this paper · explained in 1 episode
• Benchmarking — 2× in this paper · explained in 4 episodes
• Fine-Tuning — 2× in this paper · explained in 5 episodes
• Evaluation Protocol — 1× in this paper · explained in 2 episodes
• Failure Modes — 1× in this paper · explained in 3 episodes
• Grounding — 1× in this paper · explained in 3 episodes
• Judge Reliability — 1× in this paper · explained in 2 episodes
• Localization — 1× in this paper · explained in 2 episodes
• Papers — 1× in this paper · explained in 3 episodes
• Parameter-Efficient Fine-Tuning — 1× in this paper · explained in 3 episodes
• Provenance — 1× in this paper · explained in 5 episodes
• Robustness — 1× in this paper · explained in 6 episodes
• Scaffolding — 1× in this paper · explained in 3 episodes

Sources

• Benchmarking LLMs for Fine-Grained Code Review with Enriched Context in Practice
• SEVRA-BENCH: Social Engineering of Vulnerabilities in Review Agents
• The Impact of AI on Developer Productivity: Evidence from GitHub Copilot
• SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation
• Code Review Agent Benchmark

Related papers

• Falsification-Based Verification of LLM-Generated Optimization Models: Sound Test Batteries and Their Detection Limits
• GitSkills: A Dataset of Agent Skills on GitHub
• For Coding Agents, the Repository Matters
• SABER: Benchmarking Operational Safety of LLM Coding Agents in Stateful Project Workspaces
• PackMonitor: Enabling Zero Package Hallucinations Through Decoding-Time Monitoring
• IntentCoding: Amplifying User Intent in Code Generation