A.S.E 2.0: Understanding How Repository Context Impacts the Security of LLM-Generated Code
summary
The gist
Large language model (LLM) coding assistants and agents construct a repository context before generating code <ref:2611.06294#pg2>.
In short
Researchers created CASE-Bench, a benchmark testing how LLM coding assistants use repository context to generate secure code. They found that focusing on functionality improves general correctness but only slightly boosts security, while omitting safeguards significantly increases vulnerabilities. This suggests context should prioritize security measures over mere size or function.
Key concepts
- CASE-Bench
- A benchmark created by security experts to test LLM assistants' ability to generate secure code based on repository context. It involves 145 tasks derived from real software vulnerabilities across 51 projects, allowing for reproducible testing against both vulnerable and patched versions.
- FuncPass@1 vs. SecPass@1
- These metrics measure performance in two ways: FuncPass@1 measures how often the generated code works correctly (functionality), while SecPass@1 measures how often the generated code is secure (security). The study found that context methods that boost functionality also only slightly improve security.
- Context Budgeting
- The size or method of repository context provided to an LLM matters for security. Findings indicate that context budgets should be designed around covering necessary safeguards rather than simply increasing the token limit, as larger contexts often introduce more insecure examples.
Terminology used across episodes
This episode discusses
- A.S.E 2.0: Understanding How Repository Context Impacts the Security of LLM-Generated Code · Paper Radio
- Purple Llama CyberSecEval: A Secure Coding Benchmark for Language Models
- SecureVibeBench: Benchmarking Secure Vibe Coding of AI Agents via Reconstructing Vulnerability-Introducing Scenarios
- SecCodeBench-V2 Technical Report
- Evaluating Large Language Models Trained on Code
- Constrained Decoding for Secure Code Generation
- What to Retrieve for Effective Retrieval-Augmented Code Generation? An Empirical Study and Beyond
- SEC-bench Pro: Can Language Models Solve Long-Horizon Software Security Tasks?
- SafeGenBench: A Benchmark Framework for Security Vulnerability Detection in LLM-Generated Code
- A.S.E: A Repository-Level Benchmark for Evaluating Security in AI-Generated Code
- Exploring the Security Threats of Knowledge Base Poisoning in Retrieval-Augmented Code Generation
- ARVO: Atlas of Reproducible Vulnerabilities for Open-Source Software
- DualGauge: Automated Joint Security-Functionality Benchmarking of Specification-Only Code Generation by LLMs and Coding Agents · Paper Radio
- SecRepoBench: Benchmarking Code Agents for Secure Code Completion in Real-World Repositories
- RESCUE: Retrieval Augmented Secure Code Generation
- Prompt-based Code Completion via Multi-Retrieval Augmented Generation
- Toward Secure Code Generation: Bridging Correctness and Security via Task-Adaptive Vulnerability Modeling and Execution-Based Benchmarking · Paper Radio
- RealSec-bench: A Benchmark for Evaluating Secure Code Generation in Real-World Repositories
- Top Leaderboard Ranking = Top Coding Proficiency, Always? EvoEval: Evolving Coding Benchmarks via LLM
- Is Vibe Coding Safe? Benchmarking Vulnerability of Agent-Generated Code in Real-World Tasks · Paper Radio
The paper
A.S.E 2.0: Understanding How Repository Context Impacts the Security of LLM-Generated Code · Read on arXiv
Jiongchi Yu, Keke Lian, Junqing Wang, Zhikang Fan, Zhaoxuan Li, Yongheng Huang, Ziming Zhao, Dong Zhang
Nanyang Technological University · Tencent University of Technology Singapore and Tencent China, Zhejiang University, Institute of Information Engineering CAS
Large language model (LLM) coding assistants and agents construct a repository context before generating code. For security-sensitive code, this context decides whether the model sees the project's safeguards (e.g., authorization checks) or insecure code to imitate. However, context construction is designed almost exclusively for functional correctness, and secure-code benchmarks treat the context as fixed, so how repository context affects the security of generated code remains unclear. We construct CASE-Bench, a benchmark of 145 repository-level tasks from real CVEs in 51 projects. Each task runs in Docker, has functional tests and a proof-of-concept exploit validated in both directions, and is annotated with the repository code that enforces or undermines its security. Using CASE-Bench, we compare 13 context construction methods and six controls with six LLMs, edit single context elements, and analyze coding-agent trajectories. We find that (1) functionality-oriented context raises the functional pass rate by up to 8.2 percentage points (pp) but the secure pass rate by at most 3.7 pp, (2) withholding a safeguard lowers the secure pass rate by up to 8.1 pp, while one inserted insecure exemplar raises correct-but-exploitable code by 6.3 pp, and (3) coding agents observe the relevant safeguard in only 27-36% of trajectories. These findings inform security-aware context construction and secure-code evaluation.
Transcript
Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.
Nadia: We're looking at the paper: "A.S.E 2.0: Understanding How Repository Context Impacts the Security of LLM-Generated Code".
Elias: Here's the gist of the paper. Large language model (LLM) coding assistants and agents construct a repository context before generating code <ref:2611.06294#pg2>.
Nadia: The study introduces CASE-Bench, a benchmark of 145 repository-level tasks derived from real CVEs in 51 projects to investigate how repository context affects the security of generated code <ref:2611.06294#pg3>.
Elias: Benchmark Construction and Evaluation CASE-Bench was constructed by ten security experts across five steps to create an executable benchmark where tasks are validated in both directions and annotated with security-relevant repository code <ref:2611.06294#pg4>.
Nadia: The tasks cover 28 CWEs in seven languages and span different difficulty levels, ensuring reproducibility via Docker environments and validation against both vulnerable and patched versions <ref:2611.06294#pg4>.
Title and authors: Nadia: We’ve talked about the setup of CASE-Bench and the general idea that context construction is usually for functionality, but this paper shows how that focus hurts security when it comes to LLM generated code <ref:two thousand six hundred eleven point zero six two nine four#pg1.
Elias: Basically, they constructed this benchmark with one hundred forty-five tasks pulled from real vulnerabilities across fifty open-source projects <ref:two thousand six hundred eleven point zero six two nine four#pg3. The whole point is to see if the context an AI agent builds before generating code is actually seeing the project's safeguards or just insecure code it can imitate <ref:two thousand six hundred eleven point zero six two nine four#pg3.
Priya: It’s interesting because they explicitly state that security relevant context isn't measurable, which is a big limitation when you’re trying to prove how well the context is working <ref:two thousand six hundred eleven point zero six two nine four#pg2. They are trying to solve that problem by making the tasks reproducible in Docker environments and validating them against both vulnerable and patched versions <ref:two thousand six hundred eleven point zero six two nine four#pg4.
Nadia: So they have this whole experimental setup where they hold the task, prompt, and oracle fixed while only varying the context itself to see how it affects the outcome <ref:two thousand six hundred eleven point zero six two nine four#pg2. They are looking at how context construction directly influences whether the model generates secure or insecure code <ref:two thousand six hundred eleven point zero six two nine four#pg7.
Elias: The results show a clear trade-off; functionality oriented context boosts the functional pass rate by up to eight point two percentage points, but it only boosts the secure pass rate by about three point seven percentage points <ref:two thousand six hundred eleven point zero six two nine four#pg9.
Priya: So functionally correct code isn't automatically secure if you’re not careful about what you feed the model during that context-building process <ref:two thousand six hundred eleven point zero six two nine four#pg9. And they also showed that removing a safeguard from the context drops the secure pass rate by up to eight point one percentage points <ref:two thousand six hundred eleven point zero six two nine four#pg9.
Nadia: And they found that when you insert an insecure example, even if the safeguard is there, it can increase correct but exploitable code by six point three percentage points <ref:two thousand six hundred eleven point zero six two nine four#pg9. That’s a really important finding about how context construction can introduce new risks <ref:two thousand six hundred eleven point zero six two nine four#pg9.
Elias: So the summary is that we have this benchmark testing different context methods against different LLMs to quantify exactly how much functional correctness and security correctness we get for every single approach <ref:two thousand six hundred eleven point zero six two nine four#pg7.
Priya: And they’ve also found that the performance of these context construction methods is highly dependent on the security oracle you are using, because static analysis and proof of concept tools disagree on about thirty six point eight percent of functionally correct generations <ref:two thousand six hundred eleven point zero six two nine four#pg13.
Nadia: So the paper is essentially saying that we need to get better at designing context that actively seeks out and incorporates the necessary security mechanisms, not just chasing functional completeness <ref:two thousand six hundred eleven point zero six two nine four#pg9.
Elias: That’s a clear direction for how we should think about building these assistants moving forward <ref:two thousand six hundred eleven point zero six two nine four#pg13.
Priya: It also suggests that context budgets need to be dictated by what safeguards you are trying to cover, not just how big the context window is <ref:two thousand six hundred eleven point zero six two nine four#pg9.
The paper's summary: Nadia: Now let’s talk about what the authors suggest we actually do with this information, because it’s not just about reporting numbers <ref:two thousand six hundred eleven point zero six two nine four#pg13. They are proposing concrete improvements for how we build these systems <ref:two thousand six hundred eleven point zero six two nine four#pg13.
Elias: The main idea is that context budgets should follow safeguard coverage instead of just being based on the overall size of the context window <ref:two thousand six hundred eleven point zero six two nine four#pg9. This means we need to think about what security features are needed for a specific task before we start building the context <ref:two thousand six hundred eleven point zero six two nine four#pg9.
Priya: They suggest that context constructors could actually identify what type of safeguard a task requires before they go out and retrieve any examples from the repository <ref:two thousand six hundred eleven point zero six two nine four#pg14. That sounds like a smart way to stop wasting context space on irrelevant things <ref:two thousand six hundred eleven point zero six two nine four#pg13.
Nadia: They also look at retrieval based approaches and say that while they can expose safeguards, those retrieved examples absolutely must be checked for the required safeguards before they are used as templates <ref:two thousand six hundred eleven point zero six two nine four#pg14. It’s not enough just to retrieve something relevant <ref:two thousand six hundred eleven point zero six two nine four#pg9.
Elias: They are pushing back on the idea of simply relying on similarity search, suggesting that you need explicit checks to ensure you're getting project policies and not just general API definitions <ref:two thousand six hundred eleven point zero six two nine four#pg3.
Priya: It seems like they are arguing against a purely functionality oriented context approach, which we’ve already seen hurts security pass rates, so we need to be more intentional about what we retrieve <ref:two thousand six hundred eleven point zero six two nine four#pg9.
Nadia: And if you look at the agent behavior, the paper implies that agents are mostly searching for functionality because that's what they prioritize in their queries <ref:two thousand six hundred eleven point zero six two nine four#pg10. We need to design them to actively search for security concepts instead <ref:two thousand six hundred eleven point zero six two nine four#pg9.
Elias: It’s about shifting the focus from what the agent *thinks* it needs—functionality—to what the task actually *requires* in terms of its security posture <ref:two thousand six hundred eleven point zero six two nine four#pg13.
Priya: This whole discussion points toward a necessary change in how we design context systems, moving them from just being code assemblers to being security-aware assistants <ref:two thousand six hundred eleven point zero six two nine four#pg9.
The paper's improvements: Nadia: So wrapping this up on "A.S.E two point zero: Understanding How Repository Context Impacts the Security of LLM-Generated Code," the main implication is that context construction must prioritize covering necessary safeguards over simply maximizing context size <ref:two thousand six hundred eleven point zero six two nine four#pg9.
Elias: And they are pushing for a smarter system where the context construction process can actually identify what kind of safeguard a task needs before it goes out to retrieve any examples <ref:two thousand six hundred eleven point zero six two nine four#pg14. That’s the big direction for future work in this area <ref:two thousand six hundred eleven point zero six two nine four#pg9.
Priya: From a measurement standpoint, they showed that context budgets should be tied to safeguard coverage because larger contexts just expose more insecure examples, which we already saw <ref:two thousand six hundred eleven point zero six two nine four#pg9. They also stressed that checking retrieved examples for the right safeguards is non-negotiable <ref:two thousand six hundred eleven point zero six two nine four#pg14.
Nadia: And they reminded us that because static analysis and proof of concept tools disagree on thirty six point eight percent of functionally correct generations, we have to be careful about which oracle we trust <ref:two thousand six hundred eleven point zero six two nine four#pg13. That uncertainty stays with us <ref:two thousand six hundred eleven point zero six two nine four#pg13.
Elias: So the paper is essentially giving us a blueprint for context construction that treats security as a primary constraint, not an afterthought <ref:two thousand six hundred eleven point zero six two nine four#pg7. It’s about making sure the context actually helps with security outcomes, and not just functional ones <ref:two thousand six hundred eleven point zero six two nine four#pg9.
Priya: I think the real lesson here is that designing these systems means designing them to be security aware from the start, ensuring that we’re looking for policies and checks rather than just boilerplate code <ref:two thousand six hundred eleven point zero six two nine four#pg3.
Nadia: Exactly. We’ve heard about this paper, "A.S.E two point zero: Understanding How Repository Context Impacts the Security of LLM-Generated Code," and it shows that how we construct repository context has real consequences for the security of generated code <ref:two thousand six hundred eleven point zero six two nine four#pg1.
Elias: It’s a lot about moving beyond just making things work to making sure they work securely <ref:two thousand six hundred eleven point zero six two nine four#pg7.
Priya: Yeah, it’s about being more intentional about what we feed the models in those contexts <ref:two thousand six hundred eleven point zero six two nine four#pg9.
Conclusion: Nadia: So what we’re hearing today is that when you build context for an AI coding assistant, focusing too much on getting the code to run right actually makes it less secure <ref:two thousand six hundred eleven point zero six two nine four#pg9.
Elias: Right, they show how functionality-oriented context construction can boost the functional pass rate by up to eight point two percentage points but only lift the secure pass rate by about three point seven percentage points <ref:two thousand six hundred eleven point zero six two nine four#pg9. It’s a clear trade-off.
Priya: And they highlight that withholding a safeguard from that context lowers the secure pass rate by up to eight point one percentage points, so it shows how much you really need to include those security mechanisms <ref:two thousand six hundred eleven point zero six two nine four#pg9.
Nadia: Plus, inserting an insecure example into the context can actually raise correct but exploitable code by six point three percentage points even when the safeguard is already there <ref:two thousand six hundred eleven point zero six two nine four#pg9. That’s a serious risk when you’re using retrieval methods to build those contexts <ref:two thousand six hundred eleven point zero six two nine four#pg3.
Elias: And they point out that coding agents are mostly searching for functionality because that's what they prioritize in their queries, and they only observe security-relevant context in about twenty seven to thirty six percent of the time <ref:two thousand six hundred eleven point zero six two nine four#pg9.
Priya: It’s also important that this whole study depends on the security oracle you use because static analysis and proof of concept tools disagree on about thirty six point eight percent of functionally correct generations <ref:two thousand six hundred eleven point zero six two nine four#pg13. So we can't just take the numbers at face value without knowing what tool you’re using to check the security <ref:two thousand six hundred eleven point zero six two nine four#pg13.
Nadia: So, the practical implication is that context budgets should follow safeguard coverage instead of just being based on how big the context window is <ref:two thousand six hundred eleven point zero six two nine four#pg9. It means we need to design our retrievers to actually look for what security concepts a task needs before they pull anything in <ref:two thousand six hundred eleven point zero six two nine four#pg14.
Elias: Exactly, and they suggest that retrieval based approaches can expose those safeguards, but you absolutely have to check those retrieved examples for the required safeguards before you use them as templates <ref:two thousand six hundred eleven point zero six two nine four#pg14. It’s not automatic <ref:two thousand six hundred eleven point zero six two nine four#pg3.
Priya: So the takeaway is that we need to be intentional about what goes into the context, moving away from just chasing functional completeness <ref:two thousand six hundred eleven point zero six two nine four#pg9.
Nadia: And they’re calling this paper "A.S.E two point zero: Understanding How Repository Context Impacts the Security of LLM-Generated Code" for a reason, because it lays out how we need to rethink context construction for security <ref:two thousand six hundred eleven point zero six two nine four#pg1.
Elias: It’s a lot about making sure that when we build these assistants, they are actively seeking out and incorporating the necessary security mechanisms instead of just chasing functional completeness <ref:two thousand six hundred eleven point zero six two nine four#pg9.
Priya: Speaking of intentional design, I wonder how this impacts the next generation of agentic tools we see coming up <ref:two thousand six hundred eleven point zero six two nine four#pg7.
More episodes
- 2608.27496-ROPE: Routed Origin Policy Enforcement against Indirect Prompt Injection
- Daily Summary for 2026-11-19
- 2607.07085-Is Randomness Necessary for Adaptive Data Analysis?
- The day in conversation for 2026-11-19
- Daily Summary for 2026-11-19
- 2609.10601-Threshold Choice, Not Sample Size, Bounds Trustless Verification of Nondeterministic Compound AI Workflows
- 2610.38786-Adversarial Process Disturbances at a Deployable Alarm Budget: Detector Evaluation on a Simulator and a Real Plant
- 2511.08660-Binary and Multiclass Cyberattack Classification on GeNIS Dataset
- 2604.27302-Static Attribution of Android Residential Proxy Malware Using Graph Kernels
- Daily Summary for 2026-11-18