SIExBench: A Benchmark for Automated Repair of Sensitive Information Exposure
cs.SE, cs.CR
Submitted: 2026-11-07
Updated: 2026-11-07
Code: https://github.com/SWE-agent/mini-swe-agent
License: http://creativecommons.org/licenses/by/4.0/
The gist: Modern software systems routinely process credentials, personal records, and other sensitive information.
Terminology
Abstract
Modern software systems routinely process credentials, personal records, and other sensitive information. A sensitive information exposure (SIEx) defect makes such information accessible to an unauthorized actor and can cause serious and often irreversible disclosure. However, prior research has focused largely on detecting such exposures rather than repairing the underlying defects, and existing repository-level issue-resolution benchmarks do not specifically target this defect class. We present SIExBench, to our knowledge the first benchmark dedicated to automated repair of SIEx defects. SIExBench contains 161 execution-verified instances from 64 widely used Python and Java repositories, each with an issue report, buggy repository snapshot, developer patch, executable exposure and regression tests, and defect characterization. We evaluate six repair systems with three LLMs, yielding 18 configurations, and systematically analyze their overall effectiveness, performance across defect characteristics, and failure mechanisms. The strongest configuration resolves only 39.1% of the instances, while 40.4% remain unresolved by all 18 configurations. More importantly, 62.1% of unsuccessful attempts produce a valid patch that still leaves sensitive information exposed, making incomplete exposure removal a central challenge. Systems often generate plausible patches and modify relevant code without fully eliminating the exposure, indicating that effective repair must ensure that all relevant paths to sensitive-information exposure are closed. Overall, SIExBench provides a foundation for systematically evaluating and advancing automated repair of sensitive information exposure defects.
Sources
- Collaborative Agents for Automated Program Repair in Ruby
- RepairAgent: An Autonomous, LLM-Based Agent for Program Repair
- Prometheus: Towards Long-Horizon Codebase Navigation for Repository-Level Problem Solving
- DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models
- DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression
- SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?
- Trae Agent: An LLM-based Agent for Software Engineering with Test-time Scaling
- DeepSWE: Measuring Frontier Coding Agents on Original, Long-Horizon Engineering Tasks
- SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
- SIExVulTS: Sensitive Information Exposure Vulnerability Detection System using Transformer Models and Static Analysis
- SEC-bench: Automated Benchmarking of LLM Agents on Real-World Software Security Tasks
- SoK: Towards Effective Automated Vulnerability Repair
- MarsCode Agent: AI-native Automated Bug Fixing
- Knowledge-Enhanced Program Repair for Data Science Code
- Argus: A Multi-Agent Sensitive Information Leakage Detection Framework Based on Hierarchical Reference Relationships
- Defects4C: Benchmarking Large Language Model Repair Capability with C/C++ Bugs
- OpenHands: An Open Platform for AI Software Developers as Generalist Agents
- PATCHEVAL: A New Benchmark for Evaluating LLMs on Patching Real-World Vulnerabilities
- Live-SWE-agent: Can Software Engineering Agents Self-Evolve on the Fly?
- DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
Related papers
- ArkEval: Benchmarking and Evaluating Automated CodeRepair for ArkTS
- TestPrism: Rethinking Test Evaluation Beyond a Single Reference
- Mining Workflow Graphs for Black-Box Boundary Testing of Conversational LLM Agents
- Falsification-Based Verification of LLM-Generated Optimization Models: Sound Test Batteries and Their Detection Limits
- GitSkills: A Dataset of Agent Skills on GitHub
- For Coding Agents, the Repository Matters