Words Speak Louder Than Code: Investigating Cognitive Heuristics in LLM-Based Code Vulnerability Detection

arXiv:2606.30587 · cs.CR, cs.AI · Submitted 2026-06-29 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.

Nadia: Today's paper: "Words Speak Louder Than Code".

Elias: The gist The first systematic investigation of cognitive heuristics in LLM-based code vulnerability detection reveals that all evaluated models are susceptible to these heuristics,

Nadia: First, who's behind it and why it matters.

Title and authors: Elias: First, the title itself, "Words Speak Louder Than Code," suggests that the context around the code—the words—can override what's actually in the code itself.

Nadia: It implies that sometimes a bad author tag or a scary task description can trick an LLM into making a wrong security call.

Elias: The authors are Asif Shahriar, Hongyu Cai, Hadjer Benkraouda, Gang Wang, and Z. Berkay Celik from BRAC University and Purdue University at the University of Illinois Urbana-Champaign.

Priya: It’s interesting that they’re focusing on how these context signals—like an author's reputation or a task description—can flip the model's security verdict.

Nadia: And what they want us to get from this is understanding if we can actually use these known cognitive biases to trick the AI into missing vulnerabilities or flagging harmless code incorrectly.

The paper's summary: Elias: The paper introduces a framework where they fix the actual source code and only change things around it, like commit messages or author names, to trigger these three specific heuristics.

Nadia: They evaluate eight different LLMs across three programming languages using this setup to see how susceptible they are to these biases.

Priya: What I’m hearing is that they found that all the models they tested were susceptible to these heuristics, and there was a consistent pattern of bias across them.

Elias: The numbers show the framing effect had the highest average susceptibility at thirty-three point two percent, followed by anchoring at twenty-three point five percent, and then the halo effect at eighteen point four percent <ref:2606.30587#pg1>.

Nadia: And they found that vulnerabilities that require deep reasoning, like tracking program state, were way more susceptible to these cognitive tricks than just things you can spot by looking for a simple pattern match.

The paper's improvements: Nadia: The authors point out that cognitive biases don't actually make the model better at finding bugs; instead, they just make it more or less willing to flag something as vulnerable, shifting both its recall and false positive rate by a similar amount.

Elias: That means the bias isn't improving accuracy in any direction; it’s just changing its willingness to say yes or no to a vulnerability verdict.

Priya: And they show that the direction of influence isn't always the same for every model; some models go against what you expect, especially with author attribution and prior analysis results.

Nadia: The paper demonstrates a practical exploit by showing a black-box attack that can suppress up to ninety-seven percent of previously detected vulnerabilities in a real CI/CD environment just by faking commit metadata and scan reports.

Conclusion: Elias: So, the main conclusion is that cognitive heuristics are a consistent and exploitable property of LLM-based vulnerability detection; an attacker can strategically place context to deceive the model.

Nadia: It shows that prompt-based defenses aren't sufficient against these attacks because they don't eliminate the suppression effect on most models.

Priya: The paper also makes it clear that we need to focus our testing on those deep logic checks, like tracking program state, because those classes are consistently more susceptible to these heuristics than surface-level ones.

Elias: Ultimately, the implication is that security analysis needs to account for this possibility that context can systematically bias the model's objective verdict.

Nadia: What we’ve heard today about "Words Speak Louder Than Code" is that even advanced AI security tools have these human cognitive weaknesses, and those weaknesses are measurable and exploitable in a real workflow.

BRAC University · Purdue University · University of Illinois Urbana-Champaign

cs.CR, cs.AI

Submitted: 2026-06-29

Updated: 2026-10-15

Comments: Accepted at IEEE Symposium on Security and Privacy (IEEE S&P) 2027

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 90/100

The gist: The gist The first systematic investigation of cognitive heuristics in LLM-based code vulnerability detection reveals that all evaluated models are susceptible to these heuristics, and cognitive

Key concepts

Halo Effect: This is a cognitive shortcut where a positive impression of an entity in one area influences the evaluation of unrelated attributes. In this study, it means an LLM might trust code from a high-prestige author (like a principal engineer) more than code from a junior developer, regardless of the actual security flaws.
Framing Effect: This refers to how the way information is presented—the 'frame'—affects choices. For vulnerability detection, this means whether asking an LLM to identify 'violations' (negative framing) or verify 'compliance' (positive framing) changes its assessment of vulnerable code.
Anchoring Effect: This heuristic describes how an initial piece of information disproportionately influences subsequent judgments. In this context, it means that a prior analysis result, such as a static analyzer's verdict, can anchor the LLM's current assessment of new code.
Recall Gap (∆R) and Utility Index (UI): These are custom metrics used to measure how much a cognitive heuristic shifts recall (finding true vulnerabilities) versus false positive rate. A positive UI means the heuristic is useful, but the study found that for most heuristics, this utility index is negative, meaning biases don't improve detection.

Terminology

Large Language Models (LLMs) — 1× in this paper · explained in 8 episodes

Summary

The gist The first systematic investigation of cognitive heuristics in LLM-based code vulnerability detection reveals that all evaluated models are susceptible to these heuristics, and cognitive susceptibility is a consistent and exploitable property of LLM-based vulnerability detection.

How it works

The researchers introduced a controlled framework that holds the code fixed and only varies the surrounding context to trigger three cognitive heuristics: the halo effect through author attribution, the framing effect through task objectives and consequences, and the anchoring effect through prior analysis results. This framework allows for a systematic exploration of how non-code contextual metadata can impact a model’s security verdict by triggering its inherent cognitive heuristics.

The study evaluated eight LLMs across three programming languages and performed both quantitative and code-level analyses to assess the models' susceptibility to these biases. The researchers tested two prompt variants for each heuristic, with one polarity being reassuring (e.g., a high-prestige author) and the other being alarming (e.g., a low-prestige author).

Key Findings on Susceptibility and Utility

The findings demonstrated that all evaluated models are susceptible to these heuristics, with cross-model average susceptibility being highest for framing at 33.2%, followed by anchoring at 23.5% and halo at 18.4%. Code-level analysis revealed that vulnerabilities that require semantic reasoning for detection are more susceptible to cognitive heuristics than those identifiable through pattern matching. Furthermore, cognitive biases do not improve a model’s detection capability; instead, they make a model more or less willing to flag code as vulnerable, affecting both recall and FPR by a near-equal magnitude while precision stays flat within a narrow range.

The Nature of Cognitive Influence

The direction of cognitive influence is not uniform; most models respond in the expected direction (reassuring signals suppress detection, alarming signals raise it), but some models reverse this on halo and anchoring. Framing induces the expected response in all eight models, and framing is consistently shown to be the strongest effect among all three heuristics under study.

Adversarial Exploitation and Practical Impact

The researchers demonstrated a proof-of-concept black-box cognitive attack that can suppress up to 97% of previously detected vulnerabilities in a realistic CI/CD threat model by forging commit metadata and a fabricated prior scan report. This attack shows that the mere presence of non-code contexts can trigger the cognitive heuristics latent in LLMs and systematically bias their objective security verdicts. Prompt-based defenses were shown to be insufficient against these cognitive attacks, as they fail to eliminate the suppression effect on most models.

Vulnerability Class Dependence

Code-level analysis indicated that vulnerability classes requiring semantic reasoning, such as tracking program state or exceptional control flow, are consistently more susceptible to cognitive heuristics (1.5 × −2×) than those with surface-level signatures. For instance, CWEs like Divide By Zero and Use After Free showed high susceptibility under the framing effect in PRIMEVUL (Table 7).

Conclusion on Cognitive Susceptibility

Cognitive heuristics are a consistent and exploitable property of LLM-based vulnerability detection, meaning that an attacker can exploit these biases to deceive models through strategic placement of benign-looking context. The results indicate that the model does not become more or less capable under any cognitive setting; it simply becomes more or less willing to say VULNERABLE. The cost of running experiments prevents the researchers from conducting repeated runs and reporting confidence intervals, but the study highlights a critical issue for security analysis.

The paper concludes that cognitive susceptibility is a consistent and exploitable property of LLM-based vulnerability detection. The findings indicate that cognitive susceptibility is a consistent and exploitable property of LLMbased vulnerability detection.

Improvements for AI systems

System instruction modification to enforce context isolation: Implement a system prompt that explicitly instructs models to analyze code based solely on code logic and control flow and treat all surrounding contexts, including PR titles, descriptions, commit messages, author attribution, branch names, and code comments as authored by the same party submitting the code. This mitigates cognitive bias by instructing the model to ignore non-code context.
Adversarial attack detection via semantic reasoning focus: Prioritize vulnerability detection for vulnerability classes that require semantic reasoning because they are consistently more susceptible to cognitive heuristics (1.5 × −2×) than those with surfacelevel signatures. This suggests focusing on deep logic checks over simple pattern matching when context is present.
Contextual signal neutralization: Implement a defense mechanism that specifically targets the halo effect by detecting and neutralizing positive author attribution signals, as high-prestige author attribution (e.g., a principal security engineer) can work as a reassuring signal to the model and lower its vigilance.
Framing-based detection enhancement: Utilize framing to improve detection for specific vulnerability types by employing violation framing in task directives, which is shown to raise recall above neutral, as it makes models find vulnerabilities they would otherwise miss.
Anchoring strength tuning: Adjust the anchoring mechanism based on its strength; when using a soft anchor (the outcome of a pre-merge check), the system should be tuned to leverage this because it shows that soft anchors in AV2 are more effective on Mistral and Gemini compared to hard anchors.
Recall/FPR lockstep monitoring: Monitor the relationship between recall and false positive rate by observing that recall and false positive rate shift in near-perfect lockstep across all models, effects and variants, indicating that cognitive manipulation primarily changes the model's propensity to flag a code as vulnerable.
Utility-based heuristic selection: Select heuristics based on their Utility Index, prioritizing those where recall improves over baseline more than FPR increases, as only "four models see a useful shift (HUI > 0) and All positive utilities come from open-source models."

Abstract

Researchers and practitioners increasingly apply Large Language Models (LLMs) for automated vulnerability detection. Recent work has shown that LLMs are susceptible to the same cognitive heuristics that bias human judgment. Yet, no work has investigated whether these heuristics affect a model's assessment of code vulnerabilities. In this paper, we present the first systematic exploration of cognitive heuristics in LLM-driven code vulnerability detection. We introduce a controlled framework that holds the code fixed and only varies the surrounding context to trigger three cognitive heuristics: the halo effect through author attribution, the framing effect through task objectives and consequences, and the anchoring effect through prior analysis results. Within this framework, we evaluate eight LLMs across three programming languages. Our results demonstrate that all evaluated models are susceptible to these heuristics. Cross-model average susceptibility is highest for framing at 33.2%, followed by anchoring at 23.5% and halo at 18.4%. Vulnerabilities that require semantic reasoning for detection are more susceptible to cognitive heuristics than those identifiable through pattern matching. Furthermore, models often change their verdict from safe to vulnerable based on the cognitive condition, without accurately identifying the actual vulnerability. To highlight the practical impact, we demonstrate a proof-of-concept black-box cognitive attack that can suppress up to 97% of previously detected vulnerabilities. Together, these findings show that cognitive bias is a consistent and exploitable property of LLM-based vulnerability detection.