A.S.E 2.0: Understanding How Repository Context Impacts the Security of LLM-Generated Code

arXiv:2611.06294 · cs.CR, cs.AI, cs.SE · Submitted 2026-11-04 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.

Nadia: We're looking at the paper: "A.S.E 2.0: Understanding How Repository Context Impacts the Security of LLM-Generated Code".

Elias: Here's the gist of the paper. Large language model (LLM) coding assistants and agents construct a repository context before generating code <ref:2611.06294#pg2>.

Nadia: The study introduces CASE-Bench, a benchmark of 145 repository-level tasks derived from real CVEs in 51 projects to investigate how repository context affects the security of generated code <ref:2611.06294#pg3>.

Elias: Benchmark Construction and Evaluation CASE-Bench was constructed by ten security experts across five steps to create an executable benchmark where tasks are validated in both directions and annotated with security-relevant repository code <ref:2611.06294#pg4>.

Nadia: The tasks cover 28 CWEs in seven languages and span different difficulty levels, ensuring reproducibility via Docker environments and validation against both vulnerable and patched versions <ref:2611.06294#pg4>.

Title and authors: Nadia: We’ve talked about the setup of CASE-Bench and the general idea that context construction is usually for functionality, but this paper shows how that focus hurts security when it comes to LLM generated code <ref:two thousand six hundred eleven point zero six two nine four#pg1.

Elias: Basically, they constructed this benchmark with one hundred forty-five tasks pulled from real vulnerabilities across fifty open-source projects <ref:two thousand six hundred eleven point zero six two nine four#pg3. The whole point is to see if the context an AI agent builds before generating code is actually seeing the project's safeguards or just insecure code it can imitate <ref:two thousand six hundred eleven point zero six two nine four#pg3.

Priya: It’s interesting because they explicitly state that security relevant context isn't measurable, which is a big limitation when you’re trying to prove how well the context is working <ref:two thousand six hundred eleven point zero six two nine four#pg2. They are trying to solve that problem by making the tasks reproducible in Docker environments and validating them against both vulnerable and patched versions <ref:two thousand six hundred eleven point zero six two nine four#pg4.

Nadia: So they have this whole experimental setup where they hold the task, prompt, and oracle fixed while only varying the context itself to see how it affects the outcome <ref:two thousand six hundred eleven point zero six two nine four#pg2. They are looking at how context construction directly influences whether the model generates secure or insecure code <ref:two thousand six hundred eleven point zero six two nine four#pg7.

Elias: The results show a clear trade-off; functionality oriented context boosts the functional pass rate by up to eight point two percentage points, but it only boosts the secure pass rate by about three point seven percentage points <ref:two thousand six hundred eleven point zero six two nine four#pg9.

Priya: So functionally correct code isn't automatically secure if you’re not careful about what you feed the model during that context-building process <ref:two thousand six hundred eleven point zero six two nine four#pg9. And they also showed that removing a safeguard from the context drops the secure pass rate by up to eight point one percentage points <ref:two thousand six hundred eleven point zero six two nine four#pg9.

Nadia: And they found that when you insert an insecure example, even if the safeguard is there, it can increase correct but exploitable code by six point three percentage points <ref:two thousand six hundred eleven point zero six two nine four#pg9. That’s a really important finding about how context construction can introduce new risks <ref:two thousand six hundred eleven point zero six two nine four#pg9.

Elias: So the summary is that we have this benchmark testing different context methods against different LLMs to quantify exactly how much functional correctness and security correctness we get for every single approach <ref:two thousand six hundred eleven point zero six two nine four#pg7.

Priya: And they’ve also found that the performance of these context construction methods is highly dependent on the security oracle you are using, because static analysis and proof of concept tools disagree on about thirty six point eight percent of functionally correct generations <ref:two thousand six hundred eleven point zero six two nine four#pg13.

Nadia: So the paper is essentially saying that we need to get better at designing context that actively seeks out and incorporates the necessary security mechanisms, not just chasing functional completeness <ref:two thousand six hundred eleven point zero six two nine four#pg9.

Elias: That’s a clear direction for how we should think about building these assistants moving forward <ref:two thousand six hundred eleven point zero six two nine four#pg13.

Priya: It also suggests that context budgets need to be dictated by what safeguards you are trying to cover, not just how big the context window is <ref:two thousand six hundred eleven point zero six two nine four#pg9.

The paper's summary: Nadia: Now let’s talk about what the authors suggest we actually do with this information, because it’s not just about reporting numbers <ref:two thousand six hundred eleven point zero six two nine four#pg13. They are proposing concrete improvements for how we build these systems <ref:two thousand six hundred eleven point zero six two nine four#pg13.

Elias: The main idea is that context budgets should follow safeguard coverage instead of just being based on the overall size of the context window <ref:two thousand six hundred eleven point zero six two nine four#pg9. This means we need to think about what security features are needed for a specific task before we start building the context <ref:two thousand six hundred eleven point zero six two nine four#pg9.

Priya: They suggest that context constructors could actually identify what type of safeguard a task requires before they go out and retrieve any examples from the repository <ref:two thousand six hundred eleven point zero six two nine four#pg14. That sounds like a smart way to stop wasting context space on irrelevant things <ref:two thousand six hundred eleven point zero six two nine four#pg13.

Nadia: They also look at retrieval based approaches and say that while they can expose safeguards, those retrieved examples absolutely must be checked for the required safeguards before they are used as templates <ref:two thousand six hundred eleven point zero six two nine four#pg14. It’s not enough just to retrieve something relevant <ref:two thousand six hundred eleven point zero six two nine four#pg9.

Elias: They are pushing back on the idea of simply relying on similarity search, suggesting that you need explicit checks to ensure you're getting project policies and not just general API definitions <ref:two thousand six hundred eleven point zero six two nine four#pg3.

Priya: It seems like they are arguing against a purely functionality oriented context approach, which we’ve already seen hurts security pass rates, so we need to be more intentional about what we retrieve <ref:two thousand six hundred eleven point zero six two nine four#pg9.

Nadia: And if you look at the agent behavior, the paper implies that agents are mostly searching for functionality because that's what they prioritize in their queries <ref:two thousand six hundred eleven point zero six two nine four#pg10. We need to design them to actively search for security concepts instead <ref:two thousand six hundred eleven point zero six two nine four#pg9.

Elias: It’s about shifting the focus from what the agent *thinks* it needs—functionality—to what the task actually *requires* in terms of its security posture <ref:two thousand six hundred eleven point zero six two nine four#pg13.

Priya: This whole discussion points toward a necessary change in how we design context systems, moving them from just being code assemblers to being security-aware assistants <ref:two thousand six hundred eleven point zero six two nine four#pg9.

The paper's improvements: Nadia: So wrapping this up on "A.S.E two point zero: Understanding How Repository Context Impacts the Security of LLM-Generated Code," the main implication is that context construction must prioritize covering necessary safeguards over simply maximizing context size <ref:two thousand six hundred eleven point zero six two nine four#pg9.

Elias: And they are pushing for a smarter system where the context construction process can actually identify what kind of safeguard a task needs before it goes out to retrieve any examples <ref:two thousand six hundred eleven point zero six two nine four#pg14. That’s the big direction for future work in this area <ref:two thousand six hundred eleven point zero six two nine four#pg9.

Priya: From a measurement standpoint, they showed that context budgets should be tied to safeguard coverage because larger contexts just expose more insecure examples, which we already saw <ref:two thousand six hundred eleven point zero six two nine four#pg9. They also stressed that checking retrieved examples for the right safeguards is non-negotiable <ref:two thousand six hundred eleven point zero six two nine four#pg14.

Nadia: And they reminded us that because static analysis and proof of concept tools disagree on thirty six point eight percent of functionally correct generations, we have to be careful about which oracle we trust <ref:two thousand six hundred eleven point zero six two nine four#pg13. That uncertainty stays with us <ref:two thousand six hundred eleven point zero six two nine four#pg13.

Elias: So the paper is essentially giving us a blueprint for context construction that treats security as a primary constraint, not an afterthought <ref:two thousand six hundred eleven point zero six two nine four#pg7. It’s about making sure the context actually helps with security outcomes, and not just functional ones <ref:two thousand six hundred eleven point zero six two nine four#pg9.

Priya: I think the real lesson here is that designing these systems means designing them to be security aware from the start, ensuring that we’re looking for policies and checks rather than just boilerplate code <ref:two thousand six hundred eleven point zero six two nine four#pg3.

Nadia: Exactly. We’ve heard about this paper, "A.S.E two point zero: Understanding How Repository Context Impacts the Security of LLM-Generated Code," and it shows that how we construct repository context has real consequences for the security of generated code <ref:two thousand six hundred eleven point zero six two nine four#pg1.

Elias: It’s a lot about moving beyond just making things work to making sure they work securely <ref:two thousand six hundred eleven point zero six two nine four#pg7.

Priya: Yeah, it’s about being more intentional about what we feed the models in those contexts <ref:two thousand six hundred eleven point zero six two nine four#pg9.

Conclusion: Nadia: So what we’re hearing today is that when you build context for an AI coding assistant, focusing too much on getting the code to run right actually makes it less secure <ref:two thousand six hundred eleven point zero six two nine four#pg9.

Elias: Right, they show how functionality-oriented context construction can boost the functional pass rate by up to eight point two percentage points but only lift the secure pass rate by about three point seven percentage points <ref:two thousand six hundred eleven point zero six two nine four#pg9. It’s a clear trade-off.

Priya: And they highlight that withholding a safeguard from that context lowers the secure pass rate by up to eight point one percentage points, so it shows how much you really need to include those security mechanisms <ref:two thousand six hundred eleven point zero six two nine four#pg9.

Nadia: Plus, inserting an insecure example into the context can actually raise correct but exploitable code by six point three percentage points even when the safeguard is already there <ref:two thousand six hundred eleven point zero six two nine four#pg9. That’s a serious risk when you’re using retrieval methods to build those contexts <ref:two thousand six hundred eleven point zero six two nine four#pg3.

Elias: And they point out that coding agents are mostly searching for functionality because that's what they prioritize in their queries, and they only observe security-relevant context in about twenty seven to thirty six percent of the time <ref:two thousand six hundred eleven point zero six two nine four#pg9.

Priya: It’s also important that this whole study depends on the security oracle you use because static analysis and proof of concept tools disagree on about thirty six point eight percent of functionally correct generations <ref:two thousand six hundred eleven point zero six two nine four#pg13. So we can't just take the numbers at face value without knowing what tool you’re using to check the security <ref:two thousand six hundred eleven point zero six two nine four#pg13.

Nadia: So, the practical implication is that context budgets should follow safeguard coverage instead of just being based on how big the context window is <ref:two thousand six hundred eleven point zero six two nine four#pg9. It means we need to design our retrievers to actually look for what security concepts a task needs before they pull anything in <ref:two thousand six hundred eleven point zero six two nine four#pg14.

Elias: Exactly, and they suggest that retrieval based approaches can expose those safeguards, but you absolutely have to check those retrieved examples for the required safeguards before you use them as templates <ref:two thousand six hundred eleven point zero six two nine four#pg14. It’s not automatic <ref:two thousand six hundred eleven point zero six two nine four#pg3.

Priya: So the takeaway is that we need to be intentional about what goes into the context, moving away from just chasing functional completeness <ref:two thousand six hundred eleven point zero six two nine four#pg9.

Nadia: And they’re calling this paper "A.S.E two point zero: Understanding How Repository Context Impacts the Security of LLM-Generated Code" for a reason, because it lays out how we need to rethink context construction for security <ref:two thousand six hundred eleven point zero six two nine four#pg1.

Elias: It’s a lot about making sure that when we build these assistants, they are actively seeking out and incorporating the necessary security mechanisms instead of just chasing functional completeness <ref:two thousand six hundred eleven point zero six two nine four#pg9.

Priya: Speaking of intentional design, I wonder how this impacts the next generation of agentic tools we see coming up <ref:two thousand six hundred eleven point zero six two nine four#pg7.

Jiongchi Yu, Keke Lian, Junqing Wang, Zhikang Fan, Zhaoxuan Li, Yongheng Huang, Ziming Zhao, Dong Zhang

Nanyang Technological University · Tencent University of Technology Singapore and Tencent China, Zhejiang University, Institute of Information Engineering CAS

cs.CR, cs.AI, cs.SE

Submitted: 2026-11-04

Updated: 2026-11-04

Comments: 15 pages

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 100/100

The gist: Large language model (LLM) coding assistants and agents construct a repository context before generating code <ref:2611.06294#pg2>.

Key concepts

CASE-Bench
A benchmark created by security experts to test LLM assistants' ability to generate secure code based on repository context. It involves 145 tasks derived from real software vulnerabilities across 51 projects, allowing for reproducible testing against both vulnerable and patched versions.
FuncPass@1 vs. SecPass@1
These metrics measure performance in two ways: FuncPass@1 measures how often the generated code works correctly (functionality), while SecPass@1 measures how often the generated code is secure (security). The study found that context methods that boost functionality also only slightly improve security.
Context Budgeting
The size or method of repository context provided to an LLM matters for security. Findings indicate that context budgets should be designed around covering necessary safeguards rather than simply increasing the token limit, as larger contexts often introduce more insecure examples.

Summary

Large language model (LLM) coding assistants and agents construct a repository context before generating code <ref:2611. p. 0

Where the paper says it

cn Zhejiang University Hangzhou, China Dong Zhang† zalezhang@tencent. com Tencent China Abstract Large language model (LLM) coding assistants and agents construct a repository context before generating code.

06294#pg2>. The study introduces CASE-Bench, a benchmark of 145 repository-level tasks derived from real CVEs in 51 projects to investigate how repository context affects the security of generated code <ref:2611. p. 0
Where the paper says it

However, context construction is designed almost exclusively for functional correctness, and secure-code benchmarks treat the context as fixed, so how repository context affects the security of generated code remains unclear. We construct CASE-Bench, a benchmark of 145 repository-level tasks from real CVEs in 51 projects.

06294#pg3>.

Benchmark Construction and Evaluation

CASE-Bench was constructed by ten security experts across five steps to create an executable benchmark where tasks are validated in both directions and annotated with security-relevant repository code <ref:2611.06294#pg4>. The tasks cover 28 CWEs in seven languages and span different difficulty levels, ensuring reproducibility via Docker environments and validation against both vulnerable and patched versions <ref:2611. p. 1

Where the paper says it

The tasks are diverse, covering 28 CWEs in seven languages, and they span different difficulty levels, from generic checks to project-specific policies. They are also of high quality: every task is reproducible in a Docker environment, and its functional tests and proof-of-concept (PoC) exploit are validated against both the vulnerable and the patched version (❷, ❸).

06294#pg4>. Evaluation involves three stages: Build, Functional tests, and Exploit, where security is determined by an observable effect on the PoC <ref:2611. p. 2
Where the paper says it

, benign inputs along the PoC path), so deny-all code fails them; correctness remains relative to these tests. Exploit: the PoC is replayed (timeout 2 min), and exploitation is decided by an observable effect, e.

06294#pg5>.

Context Construction Methods and Security Outcomes

The research compares 13 context construction methods against six LLMs under a fixed 8K-token budget to measure functional pass rates (FuncPass@1) and secure pass rates (SecPass@1) <ref:2611.06294#pg7>. Findings indicate that functionality-oriented context raises the functional pass rate by up to 8. p. 0

Where the paper says it

Using CASE-Bench, we compare 13 context construction methods and six controls with six LLMs, edit single context elements, and analyze coding-agent trajectories. We find that ❶ functionality-oriented context raises the functional pass rate by up to 8.

2 percentage points but the secure pass rate by at most 3. p. 0
Where the paper says it

We find that ❶ functionality-oriented context raises the functional pass rate by up to 8. 2 percentage points (pp) but the secure pass rate by at most 3.

7 pp <ref:2611. p. 0
Where the paper says it

❷ Security oracles are static or untargeted. Many benchmarks judge security arXiv:2611.

06294#pg9>. Conversely, withholding a safeguard lowers the secure pass rate by up to 8. p. 0
Where the paper says it

2 percentage points (pp) but the secure pass rate by at most 3. 7 pp, ❷ withholding a safeguard lowers the secure pass rate by up to 8.

1 pp, while one inserted insecure exemplar raises correct-but-exploitable code by 6. p. 0
Where the paper says it

7 pp, ❷ withholding a safeguard lowers the secure pass rate by up to 8. 1 pp, while one inserted insecure exemplar raises correct-but-exploitable code by 6.

3 pp <ref:2611. p. 0
Where the paper says it

❷ Security oracles are static or untargeted. Many benchmarks judge security arXiv:2611.

06294#pg9>.

Agent Behavior and Context Observation

Coding agents are observed to gather security-relevant context in only 27-36% of their trajectories <ref:2611.06294#pg9>. Agents primarily search for functionality, with 71% of queries naming functional entities, and the relevant safeguard is observed in only 27-36% of their trajectories <ref:2611. p. 9

Where the paper says it

Although agents observe 16-26× as many repository tokens as an 8K context, they observe the relevant safeguard in only 27-36% of trajectories, yet an insecure exemplar in 43-51% of the trajectories on tasks that contain one. Their searches target functionality: 71% of queries name functional entities (e.

06294#pg10>. Trajectories that observed and used the safeguard reach 34. p. 9
Where the paper says it

Observation and outcome. Trajectories that observed and used the safeguard reach 34.

5% SecPass@1, whereas those that never observed it reach only 12. p. 9
Where the paper says it

Trajectories that observed and used the safeguard reach 34. 5% SecPass@1, versus 12.

7% SecPass@1 <ref:2611. p. 0
Where the paper says it

❷ Security oracles are static or untargeted. Many benchmarks judge security arXiv:2611.

06294#pg10>.

Key Findings and Implications

The study reveals several critical findings regarding context effects:

functionality-oriented context raises the functional pass rate by up to 8.2 percentage points (pp) but the secure pass rate by at most 3. p. 0

Where the paper says it

We find that ❶ functionality-oriented context raises the functional pass rate by up to 8. 2 percentage points (pp) but the secure pass rate by at most 3.

7 pp, with no significant difference from the target file alone

withholding a safeguard lowers the secure pass rate by up to 8.1 pp

one inserted insecure exemplar raises correct-but-exploitable code by 6.3 pp even with the safeguard present

These results inform guidelines for security-aware context construction, suggesting that context budgets should follow safeguard coverage rather than size <ref:2611.06294#pg13>. The paper concludes that retrieval-based approaches can expose safeguards, but retrieved examples must be checked for the required safeguards before they serve as templates <ref:2611.06294#pg14>.

Measurement Robustness and Oracle Dependence

The conclusions regarding context methods depend on the security oracle, as static analysis and the PoC oracle disagree on 36.8% of functionally correct generations <ref:2611. p. 5

Where the paper says it

Oracle comparison. We cross-tabulate static and exploit verdicts for all functionally correct RQ1 generations, calibrate the static queries on the known versions of each task, and compare method rankings on the same 145 tasks.

06294#pg13>. Context budgets should follow safeguard coverage rather than size because larger contexts also expose more insecure exemplars <ref:2611. p. 6
Where the paper says it

Beyond 8K, its fix-API recall still grows, but so does the share of contexts with an insecure exemplar (from 41% to 90%), while security density drops from . The decline thus coincides with more exemplars rather than with lower recall.

06294#pg9>. The study suggests that context constructors could identify the type of safeguard a task needs before retrieval <ref:2611.06294#pg14>. The annotations of CASE-Bench make such retrievers measurable since safeguard coverage can be computed for any context before generation <ref:2611. p. 3
Where the paper says it

Conference’17, July 2017, Washington, DC, USA Yu et al. Construction ❶ Collecting CVEs 144 CVEs from 51 projects ❷ Masking Regions 145 regions + requirements ❸ Building Environments 145 images, tests, PoCs ★ executable ❹ Validating Oracles vulnerable and patched versions ★ cross-validated ❺ Annotating Context safeguards and insecure siblings ★ measurable CASE-Bench 145 tasks 28 CWEs 7 languages ★ no leakage Visible: requirement · masked file · repository Hidden: fix · tests · PoC · annotations Evaluation Constructing Context 11 functionality-oriented 2 security-aware 6 control conditions same 8K-token budget ★ context as variable Generating Code GPT-5.

06294#pg9>.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Context construction should prioritize security-aware retrieval methods over purely functionality-oriented ones because functionality-oriented context raises the functional pass rate by up to 8.2 percentage points (pp) but the secure pass rate by at most 3.7 pp.

  2. Implement safeguards explicitly in retrieved contexts rather than relying on similarity search, as similarity retrieval thus occasionally surfaces a project API (L2) but almost never a project policy (L3).

  3. Introduce an explicit check for retrieved examples before they are used as templates to prevent the introduction of exploitable code, noting that an inserted insecure exemplar raises correct-but-exploitable code by 6.3 pp even with the safeguard present.

  4. Design agents to actively search for security concepts rather than focusing solely on functionality, since coding agents mostly search for functionality and trajectories that observed an insecure exemplar produce more correct-but-exploitable code (61.6% vs. 48.4%; Figure 7c).

  5. Evaluate context explicitly as an experimental variable in security benchmarks, ensuring that secure-code benchmarks treat the context as fixed is avoided by treating it as a controlled variable to measure its impact on security outcomes.

Abstract

Large language model (LLM) coding assistants and agents construct a repository context before generating code. For security-sensitive code, this context decides whether the model sees the project's safeguards (e.g., authorization checks) or insecure code to imitate. However, context construction is designed almost exclusively for functional correctness, and secure-code benchmarks treat the context as fixed, so how repository context affects the security of generated code remains unclear. We construct CASE-Bench, a benchmark of 145 repository-level tasks from real CVEs in 51 projects. Each task runs in Docker, has functional tests and a proof-of-concept exploit validated in both directions, and is annotated with the repository code that enforces or undermines its security. Using CASE-Bench, we compare 13 context construction methods and six controls with six LLMs, edit single context elements, and analyze coding-agent trajectories. We find that (1) functionality-oriented context raises the functional pass rate by up to 8.2 percentage points (pp) but the secure pass rate by at most 3.7 pp, (2) withholding a safeguard lowers the secure pass rate by up to 8.1 pp, while one inserted insecure exemplar raises correct-but-exploitable code by 6.3 pp, and (3) coding agents observe the relevant safeguard in only 27-36% of trajectories. These findings inform security-aware context construction and secure-code evaluation.

Sources

Related papers