Preserve the Vulnerability, Break the Alert: Testing the Robustness of CodeQL Taint Tracking with Natural Rewrites
summary
The gist
Static application security testing (SAST) tools such as CodeQL serve as security gates in continuous integration and code review, and a gate is also read in reverse.
In short
Researchers tested if CodeQL's JavaScript taint tracking is robust by applying natural code rewrites that preserve vulnerabilities. They found that three such rewrites can silence over a quarter of CodeQL's detected flows, but these silences often occur because the changes cross narrow modeling gaps within CodeQL's existing logic, not because the vulnerability was fixed.
Key concepts
- Taint Tracking
- This is a security analysis technique used by tools like CodeQL to trace how untrusted user input (the 'source') flows through an application's code to a dangerous operation (the 'sink'). The tool marks this flow as potentially vulnerable if the data reaches the sink without proper sanitization.
- Vulnerability-Preserving Rewrites
- These are small, plausible code changes made to a function that keep the original security flaw intact but alter its structure slightly. The goal is to see if an analyzer like CodeQL can still detect the issue after these realistic maintenance-style modifications.
- Modeling Boundary Gap
- This refers to a specific point in the code's logic where CodeQL's analysis stops or fails to track data correctly. The study found that most 'disappearing alerts' happen when a change crosses one of these gaps, meaning the alert disappears because the analyzer failed to model that specific transformation correctly.
- Metamorphic Relation
- This is a mathematical way of defining robustness in this context. It means an analyzer is robust if it still reports a flow even after the code has been transformed into a plausible maintenance change. If the alert disappears, the relation fails.
Terminology used across episodes
This episode discusses
- Preserve the Vulnerability, Break the Alert: Testing the Robustness of CodeQL Taint Tracking with Natural Rewrites · Paper Radio
- DeepCode AI Fix: Fixing Security Vulnerabilities with Large Language Models
- Evaluating Large Language Models Trained on Code
- Multi-Agent Taint Specification Extraction for Vulnerability Detection
- Obfuscating Code Vulnerabilities against Static Analysis in JavaScript Code
The paper
Preserve the Vulnerability, Break the Alert: Testing the Robustness of CodeQL Taint Tracking with Natural Rewrites · Read on arXiv
ZHEZHEN CAO, WEIYUAN DING, BOWEN XU
North Carolina State University
Transcript
Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.
Nadia: We're looking at the paper: "Preserve the Vulnerability, Break the Alert".
Elias: Here's the gist of the paper. Static application security testing (SAST) tools such as CodeQL serve as security gates in continuous integration and code review, and a gate is also read in reverse.
Nadia: This paper tests the robustness of CodeQL’s JavaScript taint tracking with natural, vulnerability-preserving rewrites to determine if disappearing alerts indicate a fix or merely a change crossing an analyzer's modeling boundary.
Elias: Pruning redundant reasoning tokens The core investigation examines whether the inference that a previously reported alert is gone only holds if the analyzer is robust to changes that keep the vulnerability, testing this robustness for CodeQL’s JavaScript taint tracking with natural, vulnerability-preserving rewrites.
Nadia: The study uses a large language model to rewrite one function along a reported flow as a plausible maintenance change and verifies by manual audit that the flaw survives and the alert does not.
Paper summary: Nadia: So this paper is called "Preserve the Vulnerability, Break the Alert: Testing the Robustness of CodeQL Taint Tracking with Natural Rewrites." It looks at how we trust security scanners when an alert suddenly disappears after a change.
Elias: Exactly. The main idea is that if an alert goes away, it doesn't automatically mean the vulnerability is fixed; it might just be a change that avoids the scanner's focus because it crossed some invisible line in the scanner’s logic.
Priya: So they are testing if CodeQL’s JavaScript taint tracking is actually robust against these kinds of changes, specifically using natural, vulnerability-preserving rewrites to see if disappearing alerts mean a fix or just a change crossing an analyzer boundary.
Nadia: Right. They use a large language model to rewrite one function along a reported flow as if it were just routine maintenance code and then they check by hand if the flaw still exists and if the alert is gone.
Elias: That sounds like they’re trying to see how often ordinary changes cross those hidden boundaries that an analyzer models. The core investigation is whether the idea that a previous alert is gone only holds if the analyzer can handle changes that keep the vulnerability, which they test by checking robustness for CodeQL’s JavaScript taint tracking with natural, vulnerability-preserving rewrites.
Priya: And what they found is that three natural rewrites per flow silence about twenty six point two percent of the flows CodeQL detects, at a cost of about fifty one cents per silenced flow. But when you refactor functions without an alert in mind, that generator doesn't silence any flows at all.
Nadia: That’s interesting because it shows that these specific kinds of rewrites are effective for silencing detections where they are applicable. The authors found that the silencing rewrites trace to seven gaps at the library modeling boundary, and sixty four of those seventy rewrites cross a gap inside an API that CodeQL already models.
Elias: That means the issue isn't necessarily just CodeQL leaving things unmodeled, but there are specific narrow gaps even within APIs that CodeQL already tracks. The largest gap they found was related to the WHATWG URL constructor, which carries taint to the query components of a URL but not to its pathname, and that alone accounted for forty three of those seventy cases.
Priya: So what this means for us in terms of practical security testing is that when we see an alert disappear, we can’t just assume the code is safe; we have to consider if that change hit one of these specific modeling gaps. This boundary isn't just for CodeQL, because seventy four percent of those silencing rewrites also silence an independently built taint analyzer, and fifty three percent escape three analyzers combined.
Nadia: That points to a bigger problem here for how we evaluate taint analyzers across the board. If you’re building a security tool, you have to worry about these kinds of gaps where one analysis might see something another misses entirely. So, what we hold fixed in this study is preserving the vulnerability and the function’s externally visible interface while making sure the change looks like routine maintenance instead of some kind of evasion artifact.
Elias: That focus on preservation is key because it makes these rewrites plausible maintenance changes rather than something designed to bypass security checks. The paper suggests that disappearing alerts after an ordinary-looking change made with the alert in view are weak evidence that the flaw is gone.
Priya: I think what this study really changes for people who just listen to this is that these specific gaps, like the one in URL constructors, are concrete repair targets for those who build security scanners. Robustness to these gaps should be considered alongside detection rate when we evaluate taint analyzers.
Nadia: So the takeaway is that instead of just looking at what CodeQL misses outside its models, we need to look closely at those narrow gaps inside the models too, especially where constructs like URL constructors behave differently regarding taint. This paper tests this robustness for CodeQL’s JavaScript taint tracking with natural, vulnerability-preserving rewrites.
Elias: It really shows that the strength of a tool isn't just in its detection rate but in how robust it is to the kinds of subtle, ordinary changes that developers make when they are fixing things. This paper tests this robustness for CodeQL’s JavaScript taint tracking with natural, vulnerability-preserving rewrites.
Priya: It makes you think about how much we rely on a single tool's perfect understanding of the code structure before we can trust its results completely in a real development cycle.
Conclusion: Nadia: So, after looking at all that testing, we're talking about a paper called "Preserve the Vulnerability, Break the Alert: Testing the Robustness of CodeQL Taint Tracking with Natural Rewrites."
Elias: Yeah, it's about how reliable security scanners are when you change code in a way that looks normal. They’re checking if an alert disappearing actually means the bug is fixed or if it just slipped past the scanner because of some modeling trick.
Priya: So they found that three natural rewrites for every flow silenced about twenty six point two percent of the flows CodeQL detected, and those silencing rewrites pointed to seven specific gaps in how libraries are modeled.
Nadia: It seems like this isn't just about whether CodeQL sees the vulnerability at all, but how it understands the code structure itself when things change.
Elias: Exactly. The paper argues that these small gaps inside what the scanner already models, like a specific way a URL constructor works, are where you find trouble.
Priya: So for anyone using these tools, a disappearing alert after an ordinary-looking change isn't proof the flaw is gone; it might just be hitting one of those modeling gaps.
Nadia: It means that for developers and security tools to be trustworthy, we have to test their robustness against these kinds of subtle maintenance changes.
Elias: The authors point out that this boundary isn't just CodeQL specific, because those same rewrites also bypass other tools you might be using.
Priya: So the big picture here is that we need to focus on fixing those specific gaps in how code is understood, not just adding more detectors on top of everything.
Nadia: It makes you think about how much faith we put in the analysis when things look routine but behave unexpectedly. Next time, we'll talk about what this means for the actual work of building better security tools.
More episodes
- 2608.27496-ROPE: Routed Origin Policy Enforcement against Indirect Prompt Injection
- Daily Summary for 2026-11-19
- 2607.07085-Is Randomness Necessary for Adaptive Data Analysis?
- The day in conversation for 2026-11-19
- Daily Summary for 2026-11-19
- 2609.10601-Threshold Choice, Not Sample Size, Bounds Trustless Verification of Nondeterministic Compound AI Workflows
- 2610.38786-Adversarial Process Disturbances at a Deployable Alarm Budget: Detector Evaluation on a Simulator and a Real Plant
- 2511.08660-Binary and Multiclass Cyberattack Classification on GeNIS Dataset
- 2604.27302-Static Attribution of Android Residential Proxy Malware Using Graph Kernels
- Daily Summary for 2026-11-18