Activation Probes Surface Code-Security Signals that the Model's Output Misses

Author: Ivan Wiryadi (Independent)

Venue: Second Workshop on Technical AI Governance Research (TAIGR) @ ICML 2026, Seoul, South Korea

AI coding agents (Claude Code, Cursor, Windsurf, OpenAI Codex) now author a growing share of production code. Empirical audits find that 24–29% of AI-assistant-generated Python and JavaScript snippets contain at least one security weakness. Human security review does not scale at the rate code is generated, so automated checks are needed. Current automated checks have known failures: static analyzers produce high false-positive rates, and prompting a coding LLM for a binary verdict is non-robust—trivial rewrites flip answers in 17–26% of cases for GPT-4 and PaLM2.

The key constraint: the coding agents in widest use are closed-weight, so a deploying team cannot read their internals. Instead, a team can run an open-weight model as a reviewer over the agent's output, and that reviewer's activations are readable. The paper asks whether reading those activations recovers a security signal that simply asking the same reviewer misses.

Reviewer models: Five open-weight code LLMs spanning four architecture families: Qwen2.5-Coder 7B and 14B, DeepSeek-Coder 33B, Devstral-Small-2505, and Llama-3.1-8B, all instruct variants.

Data: Training on SVEN (four CWEs with substantial Python coverage: CWE-022, 078, 079, 089, called SVEN-Python), which supplies paired before-fix (vulnerable) and after-fix (secure) functions. Evaluation on 234 Python CVEs from PATCH EVAL whose bug type is unseen—no CWE in the CVE's full label set overlaps the four SVEN-training CWEs. Of these, 147 are fixed by changing a single function.

Probe fitting: For each model, residual-stream activations are extracted via nnsight, hooking the input of layers[L].input layernorm. A sweep covers layers, token-pooling strategies (mean, max, last, sliding-window-max), and L2 strength, selecting the configuration maximizing 5-fold GroupKFold AUC. Both train/eval splits and cross-validation folds are grouped by project to prevent leakage. Selected layers and pooling vary by model (e.g., Qwen2.5-7B: layer 11, max pooling; Qwen2.5-14B: layer 21, sliding-window-max; DeepSeek-33B: layer 7, max; Devstral: layer 21, max; Llama-3.1-8B: layer 17, max).

Prompted comparator: The same model is prompted three ways (no examples, few-shot with held-out weakness types, chain-of-thought) and answers are read two ways: from logits (renormalized YES-versus-NO probability) or from the literal written token. A CVE is scored correct when the score is higher on the vulnerable function than on its fix.

In-distribution detection (SVEN-Python, n=152): Probe AUC ranges 0.771–0.846, accuracy at 0.5 threshold ranges 0.717–0.770, while prompted YES/NO accuracy stays near chance (0.488–0.524), a gap of 21–28 points on the probe's own training distribution.

Out-of-distribution evaluation (unseen bug types): On the 147 single-function CVEs, the probe scores the vulnerable function above its fix on 61–67% of cases across all five models. Wilson 95% intervals stay above the 50% chance line for every model, and sign tests are significant (p < 0.05 for every model). On the full set of 234 unseen CVEs, win-rates are 61.6–69.4%, all significant (p < 0.001).

Probe vs. prompted output: Reading the answer from logits, the probe's win-rate is higher than the model's prompted output for every model under each of the three prompts. By paired McNemar test, the probe significantly beats no-shot prompting on three of five models and few-shot prompting on four; chain-of-thought read from logits is significant on only one of five, though the probe's win-rate is still higher for every model.

Logits vs. written verdict: The way the answer is read matters more than the prompt. Reading YES/NO probability off logits gives a rankable score even when near chance. Reading the written verdict collapses: the model writes the same word for both functions on 72–97% of cases, so the pair ties and cannot be ranked. The written verdict is correct (YES on vulnerable, NO on fix) on only 1–13% of pairs.

The out-of-distribution evaluation measures a paired comparison (win-rate 61–67%), a usable ranking signal but not a calibrated detector. The probe's advantage over chain-of-thought from logits is narrow and not always significant; the clear gap is over no-shot and few-shot prompting. Training uses only the four SVEN CWEs with substantial Python coverage; C/C++-dominant classes scored at chance in preliminary runs. Limited set data; expanding is future work.

Wiring the probe into a coding agent's post-write hook and evaluating on code agents actually emit. Attention-based probes and probe-architecture comparison remain untried. Turning the paired signal into a calibrated detector with an absolute threshold is the main step toward deployment. Extension beyond Python is open.

Model activations carry a code-security signal that prompting the same model misses. The probe transfers to unseen real-world vulnerabilities and outranks the same model's prompted output on them. The longer-term goal is a self-hosted control: an open-weight reviewer over a coding agent's output that flags changes worth re-reviewing, without access to the agent's own internals.

Improvements to AI Systems:

1. Add a security-review layer to coding agents that reads model activations rather than relying on prompted output. The improved system runs an open-weight reviewer model over code produced by any coding agent (including closed-weight ones like Claude Code or Codex). It extracts residual-stream activations from the reviewer at a pre-selected layer and pooling strategy, then applies a linear probe trained on vulnerable-vs-fixed code pairs. This yields a continuous security score per function, which the system uses to flag functions for human re-review. Unlike prompted binary verdicts, this activation-based score remains rankable even when the model's written answer is at chance, and it transfers to unseen vulnerability types (CWE-022, 078, 079, 089) with 61–67% win-rates on real-world CVEs.

2. Replace fragile prompted security checks with a robust paired-comparison signal. The improved system, when asked to assess a code change, does not ask the model is this vulnerable? Instead, it computes activation-probe scores for the pre-change and post-change functions and ranks them. The system reports the new code is more/less likely vulnerable than the old code with a confidence derived from the score difference. This avoids the 17–26% answer-flip instability seen in prompted checks and the 72–97% tie rate of written verdicts. The system can be deployed as a post-write hook in CI/CD pipelines, automatically flagging diffs where the probe score increases.

3. Build a self-hosted, model-agnostic security monitor for closed-weight agents. The improved system wraps any coding agent's output with an open-weight reviewer (e.g., Qwen2.5-Coder 7B) whose activations are read via nnsight. The system requires no access to the agent's weights or logits. It runs locally, preserving code confidentiality, and produces a per-function risk score. The system can be tuned per deployment by fine-tuning the probe on the team's own vulnerable/fixed code pairs, improving precision on domain-specific patterns.

4. Add a calibration layer to convert the paired ranking into an absolute detector. The improved system learns a threshold on the probe score difference (e.g., from SVEN training data) to output a binary needs review flag. It reports precision and recall at that threshold, allowing teams to set acceptable false-positive rates. This turns the 61–67% win-rate signal into an actionable alert, with the understanding that it is a ranking aid, not a perfect classifier.

5. Extend the probe to multiple architecture families automatically. The improved system, when given a new open-weight reviewer model, runs an automated sweep over layers (e.g., 5–25), pooling strategies (mean, max, last, sliding-window-max), and L2 regularization, selecting the configuration with highest GroupKFold AUC on a held-out validation set. This makes the security layer portable across Qwen, DeepSeek, Devstral, and Llama families without manual tuning, enabling teams to swap reviewer models as better ones are released.

What the improved AI system can do:

• Flag vulnerable code changes in real time with a rankable score, even when the underlying model cannot articulate the vulnerability in words.

• Detect unseen vulnerability types (not in training CWEs) with statistically significant accuracy (p < 0.05) across five different open-weight models.

• Operate as a drop-in security reviewer for closed-weight coding agents, requiring only the agent's output text.

• Provide a continuous risk score per function, enabling prioritization of human review rather than a binary pass/fail.

• Maintain performance when the code is rewritten or refactored, because the probe reads activations rather than surface tokens, which are brittle to trivial rewrites.

• Be fine-tuned on proprietary codebases to improve detection of organization-specific vulnerability patterns.

AI coding agents now write a growing share of production code, and human security review does not scale at the rate code is generated. The agents in widest use are closed-weight, so a deploying team cannot read their internals. It can instead run an open-weight model as a reviewer over the agent's output. That reviewer's activations are readable. We ask whether reading those activations recovers a security signal that simply asking the same reviewer misses. We fit a single linear probe per model on a corpus of paired vulnerable-and-fixed Python functions, then test it without retraining on real disclosed vulnerabilities whose weakness type the probe never saw in training, across five open-weight reviewer models. On the vulnerabilities fixed by changing a single function, the probe scores the vulnerable function above its fix on 61-67% of cases for every model, beating the 50% chance line. It also beats the same model's prompted YES/NO win-rate read from its logits, under every prompt we try. Asking the model for a written verdict, even with chain-of-thought, returns the same answer on the vulnerable and fixed function most of the time and so cannot tell them apart. Model activations carry a code-security signal that prompting the same model misses.