A.S.E 2.0: Understanding How Repository Context Impacts the Security of LLM-Generated Code
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.
Nadia: We're looking at the paper: "A.S.E 2.0: Understanding How Repository Context Impacts the Security of LLM-Generated Code".
Elias: Here's the gist of the paper. Large language model (LLM) coding assistants and agents construct a repository context before generating code <ref:2611.06294#pg2>.
Nadia: The study introduces CASE-Bench, a benchmark of 145 repository-level tasks derived from real CVEs in 51 projects to investigate how repository context affects the security of generated code <ref:2611.06294#pg3>.
Elias: Benchmark Construction and Evaluation CASE-Bench was constructed by ten security experts across five steps to create an executable benchmark where tasks are validated in both directions and annotated with security-relevant repository code <ref:2611.06294#pg4>.
Nadia: The tasks cover 28 CWEs in seven languages and span different difficulty levels, ensuring reproducibility via Docker environments and validation against both vulnerable and patched versions <ref:2611.06294#pg4>.
Title and authors: Nadia: We’ve talked about the setup of CASE-Bench and the general idea that context construction is usually for functionality, but this paper shows how that focus hurts security when it comes to LLM generated code <ref:two thousand six hundred eleven point zero six two nine four#pg1.
Elias: Basically, they constructed this benchmark with one hundred forty-five tasks pulled from real vulnerabilities across fifty open-source projects <ref:two thousand six hundred eleven point zero six two nine four#pg3. The whole point is to see if the context an AI agent builds before generating code is actually seeing the project's safeguards or just insecure code it can imitate <ref:two thousand six hundred eleven point zero six two nine four#pg3.
Priya: It’s interesting because they explicitly state that security relevant context isn't measurable, which is a big limitation when you’re trying to prove how well the context is working <ref:two thousand six hundred eleven point zero six two nine four#pg2. They are trying to solve that problem by making the tasks reproducible in Docker environments and validating them against both vulnerable and patched versions <ref:two thousand six hundred eleven point zero six two nine four#pg4.
Nadia: So they have this whole experimental setup where they hold the task, prompt, and oracle fixed while only varying the context itself to see how it affects the outcome <ref:two thousand six hundred eleven point zero six two nine four#pg2. They are looking at how context construction directly influences whether the model generates secure or insecure code <ref:two thousand six hundred eleven point zero six two nine four#pg7.
Elias: The results show a clear trade-off; functionality oriented context boosts the functional pass rate by up to eight point two percentage points, but it only boosts the secure pass rate by about three point seven percentage points <ref:two thousand six hundred eleven point zero six two nine four#pg9.
Priya: So functionally correct code isn't automatically secure if you’re not careful about what you feed the model during that context-building process <ref:two thousand six hundred eleven point zero six two nine four#pg9. And they also showed that removing a safeguard from the context drops the secure pass rate by up to eight point one percentage points <ref:two thousand six hundred eleven point zero six two nine four#pg9.
Nadia: And they found that when you insert an insecure example, even if the safeguard is there, it can increase correct but exploitable code by six point three percentage points <ref:two thousand six hundred eleven point zero six two nine four#pg9. That’s a really important finding about how context construction can introduce new risks <ref:two thousand six hundred eleven point zero six two nine four#pg9.
Elias: So the summary is that we have this benchmark testing different context methods against different LLMs to quantify exactly how much functional correctness and security correctness we get for every single approach <ref:two thousand six hundred eleven point zero six two nine four#pg7.
Priya: And they’ve also found that the performance of these context construction methods is highly dependent on the security oracle you are using, because static analysis and proof of concept tools disagree on about thirty six point eight percent of functionally correct generations <ref:two thousand six hundred eleven point zero six two nine four#pg13.
Nadia: So the paper is essentially saying that we need to get better at designing context that actively seeks out and incorporates the necessary security mechanisms, not just chasing functional completeness <ref:two thousand six hundred eleven point zero six two nine four#pg9.
Elias: That’s a clear direction for how we should think about building these assistants moving forward <ref:two thousand six hundred eleven point zero six two nine four#pg13.
Priya: It also suggests that context budgets need to be dictated by what safeguards you are trying to cover, not just how big the context window is <ref:two thousand six hundred eleven point zero six two nine four#pg9.
The paper's summary: Nadia: Now let’s talk about what the authors suggest we actually do with this information, because it’s not just about reporting numbers <ref:two thousand six hundred eleven point zero six two nine four#pg13. They are proposing concrete improvements for how we build these systems <ref:two thousand six hundred eleven point zero six two nine four#pg13.
Elias: The main idea is that context budgets should follow safeguard coverage instead of just being based on the overall size of the context window <ref:two thousand six hundred eleven point zero six two nine four#pg9. This means we need to think about what security features are needed for a specific task before we start building the context <ref:two thousand six hundred eleven point zero six two nine four#pg9.
Priya: They suggest that context constructors could actually identify what type of safeguard a task requires before they go out and retrieve any examples from the repository <ref:two thousand six hundred eleven point zero six two nine four#pg14. That sounds like a smart way to stop wasting context space on irrelevant things <ref:two thousand six hundred eleven point zero six two nine four#pg13.
Nadia: They also look at retrieval based approaches and say that while they can expose safeguards, those retrieved examples absolutely must be checked for the required safeguards before they are used as templates <ref:two thousand six hundred eleven point zero six two nine four#pg14. It’s not enough just to retrieve something relevant <ref:two thousand six hundred eleven point zero six two nine four#pg9.
Elias: They are pushing back on the idea of simply relying on similarity search, suggesting that you need explicit checks to ensure you're getting project policies and not just general API definitions <ref:two thousand six hundred eleven point zero six two nine four#pg3.
Priya: It seems like they are arguing against a purely functionality oriented context approach, which we’ve already seen hurts security pass rates, so we need to be more intentional about what we retrieve <ref:two thousand six hundred eleven point zero six two nine four#pg9.
Nadia: And if you look at the agent behavior, the paper implies that agents are mostly searching for functionality because that's what they prioritize in their queries <ref:two thousand six hundred eleven point zero six two nine four#pg10. We need to design them to actively search for security concepts instead <ref:two thousand six hundred eleven point zero six two nine four#pg9.
Elias: It’s about shifting the focus from what the agent *thinks* it needs—functionality—to what the task actually *requires* in terms of its security posture <ref:two thousand six hundred eleven point zero six two nine four#pg13.
Priya: This whole discussion points toward a necessary change in how we design context systems, moving them from just being code assemblers to being security-aware assistants <ref:two thousand six hundred eleven point zero six two nine four#pg9.
The paper's improvements: Nadia: So wrapping this up on "A.S.E two point zero: Understanding How Repository Context Impacts the Security of LLM-Generated Code," the main implication is that context construction must prioritize covering necessary safeguards over simply maximizing context size <ref:two thousand six hundred eleven point zero six two nine four#pg9.
Elias: And they are pushing for a smarter system where the context construction process can actually identify what kind of safeguard a task needs before it goes out to retrieve any examples <ref:two thousand six hundred eleven point zero six two nine four#pg14. That’s the big direction for future work in this area <ref:two thousand six hundred eleven point zero six two nine four#pg9.
Priya: From a measurement standpoint, they showed that context budgets should be tied to safeguard coverage because larger contexts just expose more insecure examples, which we already saw <ref:two thousand six hundred eleven point zero six two nine four#pg9. They also stressed that checking retrieved examples for the right safeguards is non-negotiable <ref:two thousand six hundred eleven point zero six two nine four#pg14.
Nadia: And they reminded us that because static analysis and proof of concept tools disagree on thirty six point eight percent of functionally correct generations, we have to be careful about which oracle we trust <ref:two thousand six hundred eleven point zero six two nine four#pg13. That uncertainty stays with us <ref:two thousand six hundred eleven point zero six two nine four#pg13.
Elias: So the paper is essentially giving us a blueprint for context construction that treats security as a primary constraint, not an afterthought <ref:two thousand six hundred eleven point zero six two nine four#pg7. It’s about making sure the context actually helps with security outcomes, and not just functional ones <ref:two thousand six hundred eleven point zero six two nine four#pg9.
Priya: I think the real lesson here is that designing these systems means designing them to be security aware from the start, ensuring that we’re looking for policies and checks rather than just boilerplate code <ref:two thousand six hundred eleven point zero six two nine four#pg3.
Nadia: Exactly. We’ve heard about this paper, "A.S.E two point zero: Understanding How Repository Context Impacts the Security of LLM-Generated Code," and it shows that how we construct repository context has real consequences for the security of generated code <ref:two thousand six hundred eleven point zero six two nine four#pg1.
Elias: It’s a lot about moving beyond just making things work to making sure they work securely <ref:two thousand six hundred eleven point zero six two nine four#pg7.
Priya: Yeah, it’s about being more intentional about what we feed the models in those contexts <ref:two thousand six hundred eleven point zero six two nine four#pg9.
Conclusion: Nadia: So what we’re hearing today is that when you build context for an AI coding assistant, focusing too much on getting the code to run right actually makes it less secure <ref:two thousand six hundred eleven point zero six two nine four#pg9.
Elias: Right, they show how functionality-oriented context construction can boost the functional pass rate by up to eight point two percentage points but only lift the secure pass rate by about three point seven percentage points <ref:two thousand six hundred eleven point zero six two nine four#pg9. It’s a clear trade-off.
Priya: And they highlight that withholding a safeguard from that context lowers the secure pass rate by up to eight point one percentage points, so it shows how much you really need to include those security mechanisms <ref:two thousand six hundred eleven point zero six two nine four#pg9.
Nadia: Plus, inserting an insecure example into the context can actually raise correct but exploitable code by six point three percentage points even when the safeguard is already there <ref:two thousand six hundred eleven point zero six two nine four#pg9. That’s a serious risk when you’re using retrieval methods to build those contexts <ref:two thousand six hundred eleven point zero six two nine four#pg3.
Elias: And they point out that coding agents are mostly searching for functionality because that's what they prioritize in their queries, and they only observe security-relevant context in about twenty seven to thirty six percent of the time <ref:two thousand six hundred eleven point zero six two nine four#pg9.
Priya: It’s also important that this whole study depends on the security oracle you use because static analysis and proof of concept tools disagree on about thirty six point eight percent of functionally correct generations <ref:two thousand six hundred eleven point zero six two nine four#pg13. So we can't just take the numbers at face value without knowing what tool you’re using to check the security <ref:two thousand six hundred eleven point zero six two nine four#pg13.
Nadia: So, the practical implication is that context budgets should follow safeguard coverage instead of just being based on how big the context window is <ref:two thousand six hundred eleven point zero six two nine four#pg9. It means we need to design our retrievers to actually look for what security concepts a task needs before they pull anything in <ref:two thousand six hundred eleven point zero six two nine four#pg14.
Elias: Exactly, and they suggest that retrieval based approaches can expose those safeguards, but you absolutely have to check those retrieved examples for the required safeguards before you use them as templates <ref:two thousand six hundred eleven point zero six two nine four#pg14. It’s not automatic <ref:two thousand six hundred eleven point zero six two nine four#pg3.
Priya: So the takeaway is that we need to be intentional about what goes into the context, moving away from just chasing functional completeness <ref:two thousand six hundred eleven point zero six two nine four#pg9.
Nadia: And they’re calling this paper "A.S.E two point zero: Understanding How Repository Context Impacts the Security of LLM-Generated Code" for a reason, because it lays out how we need to rethink context construction for security <ref:two thousand six hundred eleven point zero six two nine four#pg1.
Elias: It’s a lot about making sure that when we build these assistants, they are actively seeking out and incorporating the necessary security mechanisms instead of just chasing functional completeness <ref:two thousand six hundred eleven point zero six two nine four#pg9.
Priya: Speaking of intentional design, I wonder how this impacts the next generation of agentic tools we see coming up <ref:two thousand six hundred eleven point zero six two nine four#pg7.
Jiongchi Yu, Keke Lian, Junqing Wang, Zhikang Fan, Zhaoxuan Li, Yongheng Huang, Ziming Zhao, Dong Zhang
Nanyang Technological University · Tencent University of Technology Singapore and Tencent China, Zhejiang University, Institute of Information Engineering CAS
cs.CR, cs.AI, cs.SE
Submitted: 2026-11-04
Updated: 2026-11-04
Comments: 15 pages
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 100/100
The gist: Large language model (LLM) coding assistants and agents construct a repository context before generating code <ref:2611.06294#pg2>.
Key concepts
- CASE-Bench
- A benchmark created by security experts to test LLM assistants' ability to generate secure code based on repository context. It involves 145 tasks derived from real software vulnerabilities across 51 projects, allowing for reproducible testing against both vulnerable and patched versions.
- FuncPass@1 vs. SecPass@1
- These metrics measure performance in two ways: FuncPass@1 measures how often the generated code works correctly (functionality), while SecPass@1 measures how often the generated code is secure (security). The study found that context methods that boost functionality also only slightly improve security.
- Context Budgeting
- The size or method of repository context provided to an LLM matters for security. Findings indicate that context budgets should be designed around covering necessary safeguards rather than simply increasing the token limit, as larger contexts often introduce more insecure examples.
Summary
Large language model (LLM) coding assistants and agents construct a repository context before generating code <ref:2611. p. 0 cn
Zhejiang University
Hangzhou, China
Dong Zhang†
zalezhang@tencent. com
Tencent
China
Abstract
Large language model (LLM) coding assistants and agents construct
a repository context before generating code. However, context construction is designed almost exclusively for functional correctness, and secure-code benchmarks treat the context
as fixed, so how repository context affects the security of generated
code remains unclear. We construct CASE-Bench, a benchmark of
145 repository-level tasks from real CVEs in 51 projects.Where the paper says it
Where the paper says it
Benchmark Construction and Evaluation
CASE-Bench was constructed by ten security experts across five steps to create an executable benchmark where tasks are validated in both directions and annotated with security-relevant repository code <ref:2611.06294#pg4>. The tasks cover 28 CWEs in seven languages and span different difficulty levels, ensuring reproducibility via Docker environments and validation against both vulnerable and patched versions <ref:2611. p. 1 The tasks are diverse, covering 28
CWEs in seven languages, and they span different difficulty levels,
from generic checks to project-specific policies. They are also of
high quality: every task is reproducible in a Docker environment,
and its functional tests and proof-of-concept (PoC) exploit are validated against both the vulnerable and the patched version (❷, ❸). , benign inputs along the PoC path), so deny-all
code fails them; correctness remains relative to these tests. Exploit:
the PoC is replayed (timeout 2 min), and exploitation is decided
by an observable effect, e.Where the paper says it
Where the paper says it
Context Construction Methods and Security Outcomes
The research compares 13 context construction methods against six LLMs under a fixed 8K-token budget to measure functional pass rates (FuncPass@1) and secure pass rates (SecPass@1) <ref:2611.06294#pg7>. Findings indicate that functionality-oriented context raises the functional pass rate by up to 8. p. 0 Using CASE-Bench,
we compare 13 context construction methods and six controls with
six LLMs, edit single context elements, and analyze coding-agent
trajectories. We find that ❶ functionality-oriented context raises
the functional pass rate by up to 8. We find that ❶ functionality-oriented context raises
the functional pass rate by up to 8. 2 percentage points (pp) but the
secure pass rate by at most 3. ❷ Security
oracles are static or untargeted. Many benchmarks judge security
arXiv:2611. 2 percentage points (pp) but the
secure pass rate by at most 3. 7 pp, ❷ withholding a safeguard lowers
the secure pass rate by up to 8. 7 pp, ❷ withholding a safeguard lowers
the secure pass rate by up to 8. 1 pp, while one inserted insecure exemplar raises correct-but-exploitable code by 6. ❷ Security
oracles are static or untargeted. Many benchmarks judge security
arXiv:2611.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Agent Behavior and Context Observation
Coding agents are observed to gather security-relevant context in only 27-36% of their trajectories <ref:2611.06294#pg9>. Agents primarily search for functionality, with 71% of queries naming functional entities, and the relevant safeguard is observed in only 27-36% of their trajectories <ref:2611. p. 9 Although agents observe 16-26× as many
repository tokens as an 8K context, they observe the relevant safeguard in only 27-36% of trajectories, yet an insecure exemplar in
43-51% of the trajectories on tasks that contain one. Their searches
target functionality: 71% of queries name functional entities (e. Observation and outcome. Trajectories that observed and used
the safeguard reach 34. Trajectories that observed and used
the safeguard reach 34. 5% SecPass@1, versus 12. ❷ Security
oracles are static or untargeted. Many benchmarks judge security
arXiv:2611.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Key Findings and Implications
The study reveals several critical findings regarding context effects:
We find that ❶ functionality-oriented context raises
the functional pass rate by up to 8. 2 percentage points (pp) but the
secure pass rate by at most 3.functionality-oriented context raises the functional pass rate by up to 8.2 percentage points (pp) but the secure pass rate by at most 3. p. 0
Where the paper says it
withholding a safeguard lowers the secure pass rate by up to 8.1 pp
one inserted insecure exemplar raises correct-but-exploitable code by 6.3 pp even with the safeguard present
These results inform guidelines for security-aware context construction, suggesting that context budgets should follow safeguard coverage rather than size <ref:2611.06294#pg13>. The paper concludes that retrieval-based approaches can expose safeguards, but retrieved examples must be checked for the required safeguards before they serve as templates <ref:2611.06294#pg14>.
Measurement Robustness and Oracle Dependence
The conclusions regarding context methods depend on the security oracle, as static analysis and the PoC oracle disagree on 36.8% of functionally correct generations <ref:2611. p. 5 Oracle comparison. We cross-tabulate static and exploit verdicts for all functionally correct RQ1 generations, calibrate the static
queries on the known versions of each task, and compare method
rankings on the same 145 tasks. Beyond 8K, its fix-API recall still grows, but so does the share
of contexts with an insecure exemplar (from 41% to 90%), while
security density drops from . The decline thus coincides
with more exemplars rather than with lower recall. Conference’17, July 2017, Washington, DC, USA Yu et al. Construction
❶
Collecting
CVEs
144 CVEs from
51 projects
❷
Masking
Regions
145 regions +
requirements
❸
Building
Environments
145 images,
tests, PoCs
★ executable
❹
Validating
Oracles
vulnerable and
patched versions
★ cross-validated
❺
Annotating
Context
safeguards and
insecure siblings
★ measurable
CASE-Bench
145 tasks
28 CWEs
7 languages
★ no leakage Visible: requirement · masked file · repository Hidden: fix · tests · PoC · annotations
Evaluation
Constructing
Context
11 functionality-oriented
2 security-aware
6 control conditions
same 8K-token budget
★ context as variable
Generating
Code
GPT-5.Where the paper says it
Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Context construction should prioritize security-aware retrieval methods over purely functionality-oriented ones because
functionality-oriented context raises the functional pass rate by up to 8.2 percentage points (pp) but the secure pass rate by at most 3.7 pp.
-
Implement safeguards explicitly in retrieved contexts rather than relying on similarity search, as
similarity retrieval thus occasionally surfaces a project API (L2) but almost never a project policy (L3).
-
Introduce an explicit check for retrieved examples before they are used as templates to prevent the introduction of exploitable code, noting that
an inserted insecure exemplar raises correct-but-exploitable code by 6.3 pp even with the safeguard present.
-
Design agents to actively search for security concepts rather than focusing solely on functionality, since
coding agents mostly search for functionality
andtrajectories that observed an insecure exemplar produce more correct-but-exploitable code (61.6% vs. 48.4%; Figure 7c).
-
Evaluate context explicitly as an experimental variable in security benchmarks, ensuring that
secure-code benchmarks treat the context as fixed
is avoided by treating it as a controlled variable to measure its impact on security outcomes.
Abstract
Large language model (LLM) coding assistants and agents construct a repository context before generating code. For security-sensitive code, this context decides whether the model sees the project's safeguards (e.g., authorization checks) or insecure code to imitate. However, context construction is designed almost exclusively for functional correctness, and secure-code benchmarks treat the context as fixed, so how repository context affects the security of generated code remains unclear. We construct CASE-Bench, a benchmark of 145 repository-level tasks from real CVEs in 51 projects. Each task runs in Docker, has functional tests and a proof-of-concept exploit validated in both directions, and is annotated with the repository code that enforces or undermines its security. Using CASE-Bench, we compare 13 context construction methods and six controls with six LLMs, edit single context elements, and analyze coding-agent trajectories. We find that (1) functionality-oriented context raises the functional pass rate by up to 8.2 percentage points (pp) but the secure pass rate by at most 3.7 pp, (2) withholding a safeguard lowers the secure pass rate by up to 8.1 pp, while one inserted insecure exemplar raises correct-but-exploitable code by 6.3 pp, and (3) coding agents observe the relevant safeguard in only 27-36% of trajectories. These findings inform security-aware context construction and secure-code evaluation.
Sources
- Purple Llama CyberSecEval: A Secure Coding Benchmark for Language Models
- SecureVibeBench: Benchmarking Secure Vibe Coding of AI Agents via Reconstructing Vulnerability-Introducing Scenarios
- SecCodeBench-V2 Technical Report
- Evaluating Large Language Models Trained on Code
- Constrained Decoding for Secure Code Generation
- What to Retrieve for Effective Retrieval-Augmented Code Generation? An Empirical Study and Beyond
- SEC-bench Pro: Can Language Models Solve Long-Horizon Software Security Tasks?
- SafeGenBench: A Benchmark Framework for Security Vulnerability Detection in LLM-Generated Code
- A.S.E: A Repository-Level Benchmark for Evaluating Security in AI-Generated Code
- Exploring the Security Threats of Knowledge Base Poisoning in Retrieval-Augmented Code Generation
- ARVO: Atlas of Reproducible Vulnerabilities for Open-Source Software
- DualGauge: Automated Joint Security-Functionality Benchmarking of Specification-Only Code Generation by LLMs and Coding Agents
- SecRepoBench: Benchmarking Code Agents for Secure Code Completion in Real-World Repositories
- RESCUE: Retrieval Augmented Secure Code Generation
- Prompt-based Code Completion via Multi-Retrieval Augmented Generation
- Toward Secure Code Generation: Bridging Correctness and Security via Task-Adaptive Vulnerability Modeling and Execution-Based Benchmarking
- RealSec-bench: A Benchmark for Evaluating Secure Code Generation in Real-World Repositories
- Top Leaderboard Ranking = Top Coding Proficiency, Always? EvoEval: Evolving Coding Benchmarks via LLM
- Is Vibe Coding Safe? Benchmarking Vulnerability of Agent-Generated Code in Real-World Tasks
Related papers
- From Strategy Search to Policy Learning: MCTS-Guided Adversarial Co-Evolution for LLM Safety Alignment
- Privacy Leakage from Recurrent State in LLM Inference
- Incrementally Verifiable Oracle Aided Computation
- Privilege and confidentiality in generative AI workflows
- BadMoE: Backdooring Mixture-of-Experts LLMs via Routing-Aware Triggers and Task-Underutilized Experts
- CoVault: Secure, Scalable Analytics of Personal Data