Preserve the Vulnerability, Break the Alert: Testing the Robustness of CodeQL Taint Tracking with Natural Rewrites

arXiv:2611.17123 · cs.CR, cs.SE · Submitted 2026-11-12 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.

Nadia: We're looking at the paper: "Preserve the Vulnerability, Break the Alert".

Elias: Here's the gist of the paper. Static application security testing (SAST) tools such as CodeQL serve as security gates in continuous integration and code review, and a gate is also read in reverse.

Nadia: This paper tests the robustness of CodeQL’s JavaScript taint tracking with natural, vulnerability-preserving rewrites to determine if disappearing alerts indicate a fix or merely a change crossing an analyzer's modeling boundary.

Elias: Pruning redundant reasoning tokens The core investigation examines whether the inference that a previously reported alert is gone only holds if the analyzer is robust to changes that keep the vulnerability, testing this robustness for CodeQL’s JavaScript taint tracking with natural, vulnerability-preserving rewrites.

Nadia: The study uses a large language model to rewrite one function along a reported flow as a plausible maintenance change and verifies by manual audit that the flaw survives and the alert does not.

Paper summary: Nadia: So this paper is called "Preserve the Vulnerability, Break the Alert: Testing the Robustness of CodeQL Taint Tracking with Natural Rewrites." It looks at how we trust security scanners when an alert suddenly disappears after a change.

Elias: Exactly. The main idea is that if an alert goes away, it doesn't automatically mean the vulnerability is fixed; it might just be a change that avoids the scanner's focus because it crossed some invisible line in the scanner’s logic.

Priya: So they are testing if CodeQL’s JavaScript taint tracking is actually robust against these kinds of changes, specifically using natural, vulnerability-preserving rewrites to see if disappearing alerts mean a fix or just a change crossing an analyzer boundary.

Nadia: Right. They use a large language model to rewrite one function along a reported flow as if it were just routine maintenance code and then they check by hand if the flaw still exists and if the alert is gone.

Elias: That sounds like they’re trying to see how often ordinary changes cross those hidden boundaries that an analyzer models. The core investigation is whether the idea that a previous alert is gone only holds if the analyzer can handle changes that keep the vulnerability, which they test by checking robustness for CodeQL’s JavaScript taint tracking with natural, vulnerability-preserving rewrites.

Priya: And what they found is that three natural rewrites per flow silence about twenty six point two percent of the flows CodeQL detects, at a cost of about fifty one cents per silenced flow. But when you refactor functions without an alert in mind, that generator doesn't silence any flows at all.

Nadia: That’s interesting because it shows that these specific kinds of rewrites are effective for silencing detections where they are applicable. The authors found that the silencing rewrites trace to seven gaps at the library modeling boundary, and sixty four of those seventy rewrites cross a gap inside an API that CodeQL already models.

Elias: That means the issue isn't necessarily just CodeQL leaving things unmodeled, but there are specific narrow gaps even within APIs that CodeQL already tracks. The largest gap they found was related to the WHATWG URL constructor, which carries taint to the query components of a URL but not to its pathname, and that alone accounted for forty three of those seventy cases.

Priya: So what this means for us in terms of practical security testing is that when we see an alert disappear, we can’t just assume the code is safe; we have to consider if that change hit one of these specific modeling gaps. This boundary isn't just for CodeQL, because seventy four percent of those silencing rewrites also silence an independently built taint analyzer, and fifty three percent escape three analyzers combined.

Nadia: That points to a bigger problem here for how we evaluate taint analyzers across the board. If you’re building a security tool, you have to worry about these kinds of gaps where one analysis might see something another misses entirely. So, what we hold fixed in this study is preserving the vulnerability and the function’s externally visible interface while making sure the change looks like routine maintenance instead of some kind of evasion artifact.

Elias: That focus on preservation is key because it makes these rewrites plausible maintenance changes rather than something designed to bypass security checks. The paper suggests that disappearing alerts after an ordinary-looking change made with the alert in view are weak evidence that the flaw is gone.

Priya: I think what this study really changes for people who just listen to this is that these specific gaps, like the one in URL constructors, are concrete repair targets for those who build security scanners. Robustness to these gaps should be considered alongside detection rate when we evaluate taint analyzers.

Nadia: So the takeaway is that instead of just looking at what CodeQL misses outside its models, we need to look closely at those narrow gaps inside the models too, especially where constructs like URL constructors behave differently regarding taint. This paper tests this robustness for CodeQL’s JavaScript taint tracking with natural, vulnerability-preserving rewrites.

Elias: It really shows that the strength of a tool isn't just in its detection rate but in how robust it is to the kinds of subtle, ordinary changes that developers make when they are fixing things. This paper tests this robustness for CodeQL’s JavaScript taint tracking with natural, vulnerability-preserving rewrites.

Priya: It makes you think about how much we rely on a single tool's perfect understanding of the code structure before we can trust its results completely in a real development cycle.

Conclusion: Nadia: So, after looking at all that testing, we're talking about a paper called "Preserve the Vulnerability, Break the Alert: Testing the Robustness of CodeQL Taint Tracking with Natural Rewrites."

Elias: Yeah, it's about how reliable security scanners are when you change code in a way that looks normal. They’re checking if an alert disappearing actually means the bug is fixed or if it just slipped past the scanner because of some modeling trick.

Priya: So they found that three natural rewrites for every flow silenced about twenty six point two percent of the flows CodeQL detected, and those silencing rewrites pointed to seven specific gaps in how libraries are modeled.

Nadia: It seems like this isn't just about whether CodeQL sees the vulnerability at all, but how it understands the code structure itself when things change.

Elias: Exactly. The paper argues that these small gaps inside what the scanner already models, like a specific way a URL constructor works, are where you find trouble.

Priya: So for anyone using these tools, a disappearing alert after an ordinary-looking change isn't proof the flaw is gone; it might just be hitting one of those modeling gaps.

Nadia: It means that for developers and security tools to be trustworthy, we have to test their robustness against these kinds of subtle maintenance changes.

Elias: The authors point out that this boundary isn't just CodeQL specific, because those same rewrites also bypass other tools you might be using.

Priya: So the big picture here is that we need to focus on fixing those specific gaps in how code is understood, not just adding more detectors on top of everything.

Nadia: It makes you think about how much faith we put in the analysis when things look routine but behave unexpectedly. Next time, we'll talk about what this means for the actual work of building better security tools.

ZHEZHEN CAO, WEIYUAN DING, BOWEN XU

North Carolina State University

cs.CR, cs.SE

Submitted: 2026-11-12

Updated: 2026-11-12

Code: https://github.com/ajinabraham/njsscan

Importance score: 83/100

The gist: Static application security testing (SAST) tools such as CodeQL serve as security gates in continuous integration and code review, and a gate is also read in reverse.

Key concepts

Taint Tracking
This is a security analysis technique used by tools like CodeQL to trace how untrusted user input (the 'source') flows through an application's code to a dangerous operation (the 'sink'). The tool marks this flow as potentially vulnerable if the data reaches the sink without proper sanitization.
Vulnerability-Preserving Rewrites
These are small, plausible code changes made to a function that keep the original security flaw intact but alter its structure slightly. The goal is to see if an analyzer like CodeQL can still detect the issue after these realistic maintenance-style modifications.
Modeling Boundary Gap
This refers to a specific point in the code's logic where CodeQL's analysis stops or fails to track data correctly. The study found that most 'disappearing alerts' happen when a change crosses one of these gaps, meaning the alert disappears because the analyzer failed to model that specific transformation correctly.
Metamorphic Relation
This is a mathematical way of defining robustness in this context. It means an analyzer is robust if it still reports a flow even after the code has been transformed into a plausible maintenance change. If the alert disappears, the relation fails.

Terminology

Summary

Static application security testing (SAST) tools such as CodeQL serve as security gates in continuous integration and code review, and a gate is also read in reverse. p. 0 This paper tests the robustness of CodeQL’s JavaScript taint tracking with natural, vulnerability-preserving rewrites to determine if disappearing alerts indicate a fix or merely a change crossing an analyzer's modeling boundary.

Pruning redundant reasoning tokens

The core investigation examines whether the inference that a previously reported alert is gone only holds if the analyzer is robust to changes that keep the vulnerability, testing this robustness for CodeQL’s JavaScript taint tracking with natural, vulnerability-preserving rewrites. The study uses a large language model to rewrite one function along a reported flow as a plausible maintenance change and verifies by manual audit that the flaw survives and the alert does not. p. 0 The authors define robustness as a metamorphic relation where an analyzer must still report a flow on a transformed program, restricting transformations to those that could plausibly appear in routine maintenance. p. 3

How the training signal is built

The researchers developed a three-stage pipeline over VulcaN, a dataset of confirmed vulnerabilities in real-world npm packages annotated at the (source, sink) level to recover flows and generate rewrites under explicit vulnerability-preservation and naturalness constraints. Stage 1 identifies target flows by intersecting confirmed vulnerabilities with the code flows that CodeQL reports in their environment, using human-annotated (source, sink) pairs as the reference for genuine vulnerabilities. p. 6 Stage 2 instantiates an alert-aware but implementation-agnostic LLM to generate function-level rewrites under constraints that preserve the vulnerable behavior and read as a plausible maintenance change. p. 7

Where the gains come from

The empirical measurement on real npm vulnerabilities shows that three natural rewrites per flow silence 26.2% of the flows CodeQL detects, at about 51 cents per silenced flow, while the same generator refactoring functions without an alert silences none. p. 2 The silencing rewrites trace to seven gaps at the library-modeling boundary, with 64 of the 70 rewrites crossing a gap inside an API CodeQL already models. p. 0 The largest gap is the WHATWG URL constructor, which propagates taint to a URL’s query components but not to its pathname, accounting for 43 of the 70 cases. p. 0

What we hold fixed

The study focuses on preserving the vulnerability and the function’s externally visible interface while ensuring the change reads as routine maintenance rather than an evasion artifact. The authors verify that every silencing rewrite traces to one of seven modeling gaps, and 64 of the 70 rewrites cross a gap inside an API CodeQL already models. p. 0 The boundary is not specific to CodeQL, as 74% of the silencing rewrites also silence an independently built taint analyzer, and 53% escape three analyzers combined. p. 0

Conclusion validity

The decisive weakness is not the APIs CodeQL leaves unmodeled but narrow gaps inside the ones it does model, led by a URL constructor whose pathname carries no taint. The same boundary silences an independently built semantic analyzer, so closing it requires changing the models rather than stacking detectors. p. 1 For users, a disappearing alert after an ordinary-looking change made with the alert in view is weak evidence that the flaw is gone, while the same generator refactoring without the alert silences none of the flows. p. 0 The gaps are concrete repair targets for SAST developers, and robustness to them belongs alongside detection rate in how taint analyzers are evaluated. p. 0

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Robustness testing should be integrated into SAST evaluation by measuring how often a tool remains robust to natural, vulnerability-preserving rewrites, which robustness to them belongs alongside detection rate in how taint analyzers are evaluated. This allows for routine robustness regression testing rather than just canonical representation checks.

  2. Analysis of silencing rewrites should be used to identify concrete repair targets by tracing each silenced flow to the seven gaps at the library-modeling boundary, providing specific, actionable fixes instead of assuming a fix exists when an alert disappears.

  3. AI systems should be trained or designed to understand these modeling gaps so that they can prioritize repairs based on which gap is crossed, as the decisive weakness is not the APIs CodeQL leaves unmodeled but narrow gaps inside the ones it does model.

Sources

Related papers