From Runnable to Verifiable: An Independent Reproducibility Study of LLM/Agent-Driven Vulnerability Validation Artifacts

This paper conducts a pre-registered reproducibility audit of LLM/agent-driven vulnerability validation artifacts, distinguishing four levels of artifact maturity: available, runnable, signal-producing, and semantically confirmed. The study covers a 104-paper consensus corpus from 2023–2026, of which 59 papers (56.7%) have a publicly reachable artifact. The authors execute an 18-paper sample at R0/R1 and all 102 cases of the anchor benchmark (arXiv:2509.24037), with patched-counterfactual verdicts on 30 signal-producing cases and matched-negative-control verdicts on 19.

Three main findings stand out. First, 58/102 (56.9%) anchor cases contain a script-internal CVE identifier that diverges from the declared directory CVE. Second, only 10/18 (55.6%) paper-level artifacts complete their declared workflow at R0, rising to 11/18 (61.1%) after environment-only R1 repair. Third, artifact-embedded oracles prove unreliable: 20/30 patched-counterfactual audits still produce the claimed signal on the patched build, 7/19 matched negative controls still trigger on benign input, and the oracle confusion matrix has sensitivity 60% and specificity 45%.

The paper's contributions are: a pre-registered, reproducible measurement protocol (CVE-specific post-conditions, R0/R1 repair ladder, G1–G3 semantic evidence levels, and patched-counterfactual oracles) applied to a 104-paper consensus corpus; three findings from executing the full 102-case anchor corpus plus an 18-paper sample; and open evidence bundles as a reusable template for the security reproducibility community.

The study design addresses three research questions: RQ1 (paper-level workflow completion under R0/R1), RQ2 (gap between claimed per-case success and independently confirmed E1 evidence), and RQ3 (reliability of artifact-embedded oracles against vulnerable/patched ground truth). Evidence levels include G1 (candidate signal), G2 (post-condition match), G3a (negative control), and G3b (patched counterfactual), with E-levels derived as E1 (G2+G3a+G3b), E2 (G2+G3a), and E3 (incomplete evidence).

The corpus construction used a frozen search protocol covering 2023-01-01 to 2026-08-08, with queries against arXiv, DBLP, and OpenAlex. The search returned 815 raw records, 552 after deduplication, and 104 after applying five exclusion rules (no real vulnerability, publication type mismatch, no LLM/agent, no execution validation, no extractable claim). The availability audit found 59/104 (56.7%) publicly reachable, 28 not located, 6 announced future, 5 link unreachable, 3 upon request, 1 withheld, 1 temporarily restricted, and 1 fulltext unavailable.

The paper-level sample comprises 18 papers stratified by year (2023:1, 2024:6, 2025:6, 2026:5), spanning repair, penetration testing, benchmarks, and smart contracts. The case-level sample audits individual cases from a pool of 42 papers, with the anchor corpus (102 cases) being the primary execution corpus. The anchor benchmark has 71 claimed PASS and 31 claimed FAIL cases across languages including Python, JavaScript, Rust, C, Go, Shell, Java, C++, and Erlang.

A pre-registration finding revealed that 58/102 (56.9%) cases contain a script-internal CVE ID variable that differs from the directory's declared CVE. For example, the directory declared CVE-2023-25668 runs a QuantizeAndDequantizeV2 trigger (CVE-2023-25676), and the directory declared CVE-2021-28878 tests Vec::from iter (CVE-2021-31162). This means the claimed reproduction target and the actually tested target may diverge.

For the paper-level R0/R1 execution, 10/18 artifacts (55.6%) completed their declared workflow at R0; one further artifact (ContractTinker) completed after R1 repair; the remaining 7 failed at both levels. Dominant failure modes included: declared dependency manifests absent from the artifact root (5 artifacts), pip-incompatible manifests (2 artifacts), and missing runtime dependencies that did not suffice because the workflow depends on external services or datasets (4 artifacts). R1 repairs were environment-only and small: installing declared dependencies took 7–132 seconds (median 42 s) with one to three pip commands.

In the calibration R0 execution of 20 cases, only 3/10 claimed-PASS cases produced a trigger signal on a clean snapshot, with most failures tracing to EOL base-image apt sources, PoC compile errors, or artifact defects. R1 repair recovered three cases (CVE-2021-31162, CVE-2022-35986, CVE-2023-25669) by redirecting EOL base-image apt sources to archive.debian.org and adjusting bash-4 incompatibilities. Notably, CVE-2021-31162's artifact oracle reported TRIGGERED on both vulnerable and patched builds—an oracle false positive. Across the 20 calibration cases, 2 artifact oracles produced false positives and 3 produced false negatives—oracle unreliability in 5/20 (25%).

For the full-corpus R0 execution of 82 additional cases (87 executed runs total), 34 runs (39.1%) produced a candidate signal (G1). G2 adjudication found 23 of 34 signals satisfy the pre-registered post-condition, and 11 do not—including CVE-2025-0182, whose script-internal oracle prints its trigger unconditionally without verifying the claimed denial-of-service behavior.

The patched-counterfactual (G3b) audit disproved a textbook success: CVE-2020-1967 produced a clean segmentation-fault signal and matched the pre-registered post-condition, yet the patched counterfactual (OpenSSL 1.1.1g) also segfaulted because the PoC calls SSL check chain(None) directly—a NULL-argument crash that is undefined behavior in any OpenSSL version. Of 5 calibration cases executed against their patched build, 4 (80%) still produced the claimed signal on the patched version (CVE-2020-1967, CVE-2021-31162, CVE-2022-22816, CVE-2023-0217). Only CVE-2025-30223 (Beego XSS) passed G3b, yielding the single E1-level confirmation in the calibration set.

Of the 34 G1 signals from full-corpus runs, 29 were patched-audited and 5 were not. Four additional calibration-only cases were also patched-audited, bringing the G3b-executed set to 33 cases. Verdicts were reached on 30 cases: 10 clean and 20 dirty, with 3 unbuildable. Matched negative controls (G3a) were executed on 19 of 34 signal-producing cases: 9 passed, 7 revealed oracle false positives, 2 were not applicable, and 1 was unassessable.

The oracle confusion matrix, scored on all 30 G3b-verdict cases with ground truth = patched-counterfactual verdict and oracle-positive = script-internal VULNERABILITY TRIGGERED marker, gives TP=6, FP=11, FN=4, TN=9: sensitivity 60% (95% CI 31.3–83.2), specificity 45% (95% CI 25.8–65.8). Two-thirds of oracle triggers (11/17) are false positives, and 4 of 10 genuine CVE-specific signals are missed by the oracle.

Three systematic oracle failure modes recur: (i) simulated triggers—PoCs that force a crash via direct NULL/pointer manipulation and comment simulate vulnerability, crashing in any version (CVE-2020-1967, CVE-2023-0217); (ii) designed-in behavior mistaken for a signal—a panic in a Drop implementation or a caught TypeError treated as the vulnerability trigger (CVE-2021-31162, CVE-2022-22816); and (iii) missed genuine signals—a live JNDI RCE whose oracle required an absent marker file (CVE-2021-44228), and FPE crashes whose oracle printed not triggered.

The strict E1 confirmation set (G2 met ∩ G3b clean ∩ G3a passed) contains 2 cases (CVE-2025-30223, CVE-2021-42576) out of the signal-producing corpus. The exploratory claim gap on the anchor paper, computed on 5 calibration cases with completed G3b verdicts, shows claimed success 0.80 (4/5) vs. independent E1 rate 0.20 (1/5), gap 0.60.

The paper concludes that availability, runnability, signal production, and semantic confirmation form a strict ladder: each layer admits artifacts that fail the next. On the anchor corpus, availability held for the repository itself; runnability required R1 repair for a majority of cases; signal production (G1) was achieved by only 34 of 87 runs (39.1%); G2 was satisfied by 23/34 of those; and patched-counterfactual confirmation by only 10 of 30 audited cases. The gap between it runs and prints something alarming and it reproduced the CVE is the central quantity this study operationalizes.

Improvements to AI systems based on this paper:

1. CVE-Identifier Consistency Checker: An AI system that automatically parses vulnerability-reproduction scripts and cross-references the CVE ID declared in the directory/metadata against CVE IDs hardcoded inside the script logic. It flags mismatches (56.9% of cases had such divergence) and can auto-correct the declared target or warn the user before execution.

2. Oracle Reliability Auditor: An AI that evaluates the trustworthiness of a vulnerability-detection oracle embedded in a PoC script by analyzing its logic for three known failure modes: (i) simulated triggers (direct NULL/pointer crashes that occur in any version), (ii) designed-in behavior mistaken for a signal (e.g., panics in Drop implementations, caught TypeErrors), and (iii) missed genuine signals (e.g., requiring absent marker files). It outputs a predicted oracle sensitivity/specificity before running the full experiment.

3. Patched-Counterfactual Pre-Validator: An AI that, given a PoC script and a target CVE, automatically generates and executes a patched-counterfactual test (running the same script against the patched version of the library) to check if the claimed signal still appears. It can predict false-positive cases like CVE-2020-1967 (where the PoC crashes on both vulnerable and patched builds due to NULL-argument UB) and flag them as likely oracle false positive before the user relies on the result.

4. R0/R1 Repair Predictor: An AI that analyzes a vulnerability-reproduction artifact's dependency manifests, base-image references, and runtime requirements to predict whether it will complete its workflow at R0 (clean environment) or require R1 repairs. It can identify common failure modes—missing manifests, pip-incompatible manifests, EOL base-image apt sources, bash-4 incompatibilities—and suggest specific repair commands (e.g., redirecting to archive.debian.org) with estimated repair time (median 42s).

5. Semantic Evidence-Level Classifier: An AI that automatically assigns a G1–G3b evidence level to a vulnerability-reproduction run by checking: (G1) whether a candidate signal was produced, (G2) whether the signal matches the pre-registered post-condition (e.g., specific error type, memory access pattern), (G3a) whether a matched negative control (benign input) fails to trigger, and (G3b) whether a patched-counterfactual run is clean. It outputs an E1/E2/E3 confidence score, distinguishing it runs and prints something alarming from it reproduced the CVE.

6. False-Positive/False-Negative Calibrator for Security Benchmarks: An AI that, given a set of vulnerability-reproduction results, computes an oracle confusion matrix (TP/FP/FN/TN) with confidence intervals, like the paper's sensitivity 60% and specificity 45%. It can automatically re-weight benchmark results to correct for known oracle unreliability, preventing overestimation of LLM/agent vulnerability-finding capabilities.

7. Artifact Maturity Ladder Assessor: An AI that classifies a vulnerability-reproduction artifact into one of four maturity levels (available, runnable, signal-producing, semantically confirmed) by sequentially testing each layer. It can predict that only 39% of runnable artifacts will produce a signal and only 33% of those will survive patched-counterfactual audit, helping users triage which artifacts are worth executing.

8. CVE-Specific Post-Condition Generator: An AI that, given a CVE description and vulnerable code, automatically generates a machine-checkable post-condition (e.g., segfault at address X, specific exception type Y, memory leak of size Z) that can be used to adjudicate whether a produced signal genuinely matches the vulnerability, rather than relying on the script's own oracle.

What the improved AI system can do:

• Automatically audit and validate LLM/agent-generated vulnerability-reproduction artifacts with quantified confidence, flagging cases where the claimed CVE target diverges from the actual tested code.

• Pre-emptively identify unreliable oracles and false-positive-prone PoCs before execution, saving compute and preventing misleading benchmark results.

• Produce reproducible, evidence-level-tagged vulnerability confirmations (E1/E2/E3) that distinguish script runs and prints alarming output from CVE genuinely reproduced, with explicit sensitivity/specificity metrics for any embedded oracle.

• Generate repair plans for non-runnable artifacts (R0→R1) and predict which will fail even after environment-only fixes, enabling efficient triage of large vulnerability corpora.

• Correct for systematic oracle biases in existing security benchmarks, providing more accurate assessments of LLM/agent capabilities in vulnerability discovery and patching tasks.

Security research artifacts---repositories, PoC exploits, and validation pipelines---are increasingly produced by LLM/agent-driven vulnerability workflows, yet the gap between publicly available, runnable, signal-producing, and semantically confirmed artifacts is poorly measured. We conduct a pre-registered reproducibility audit of this literature. A search covering 2023--2026 with dual screening yields a 104-paper consensus corpus, of which 59 papers (56.7%) have a publicly reachable artifact. We execute an 18-paper sample at R0/R1 and all 102 cases of the anchor benchmark (arXiv:2509.24037), with patched-counterfactual verdicts on 30 signal-producing cases and matched-negative-control verdicts on 19. Three findings stand out. First, 58/102 (56.9%) anchor cases contain a script-internal CVE identifier that diverges from the declared directory CVE. Second, only 10/18 (55.6%) paper-level artifacts complete their declared workflow at R0, rising to 11/18 (61.1%) after environment-only R1 repair. Third, artifact-embedded oracles prove unreliable: 20/30 patched-counterfactual audits still produce the claimed signal on the patched build, 7/19 matched negative controls still trigger on benign input, and the oracle confusion matrix has sensitivity 60% and specificity 45%. A trigger on the vulnerable build is not evidence of CVE-specific reproduction without a clean patched counterfactual. These are exploratory results from a pre-registered protocol, and our protocol---pre-registered post-conditions, R0/R1 repair ladder, G1--G3 semantic evidence levels, and patched-counterfactual oracles---is a reusable template for the security reproducibility community.