CrossPloit: A Benchmark for Cross-Version PoC Migration in Python Web Applications
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.
Nadia: We're looking at the paper: "CrossPloit: A Benchmark for Cross-Version PoC Migration in Python Web Applications".
Elias: Here's the gist of the paper. Affected-version ranges in vulnerability databases decide which deployments scanners, patching pipelines, and incident responders treat as vulnerable, yet these ranges are often incomplete The single most informative sentence about the paper's subject and finding is that CrossPloit provides an executionbased measurement of under-reported exposure and of how far its enumeration can be automated.
Nadia: The Core Problem Addressed Modern vulnerability management rests on an assumption: a software version outside a CVE’s listed affected-version range is not vulnerable This boundary, built on databases like the NVD and OSV, is often wrong, and this invisibility creates a blind spot for defenders because an exploitable but unlisted version is reported clean by every tool.
Elias: The task investigated is confirming exposure by demonstrating that a proof-of-concept (PoC) exploit for a listed version can be triggered on an earlier, unlisted version.
Nadia: Benchmark Construction and Scope CrossPloit is introduced as a benchmark of 127 migration instances across 45 CVEs in the Python web ecosystem Each instance includes a version-pinned container, a reference PoC, and a vulnerability-specific oracle.
Title and authors: Nadia: The paper introduces CrossPloit as a way to measure this under-reported exposure by showing how far their enumeration can be automated. They aren't just saying "there's an issue"; they are providing an execution based measurement of how much they can actually find.
Elias: That execution based measurement is key, Nadia. It means instead of just guessing if a version is vulnerable, you get concrete evidence by showing that a proof of concept for one listed version can be triggered on an earlier, unlisted version.
Priya: So the core idea is turning that theoretical possibility into a reproducible artifact. That sounds useful because it moves the discussion from "maybe this is vulnerable" to "here is a running exploit against this old version."
Nadia: Right, and that’s what they call migration: preserving the underlying vulnerability while accommodating the differences between versions. They show how to turn an unconfirmed gap into a concrete, reproducible artifact.
Elias: They highlight three reasons why this migration is hard: first, reference proof of concepts are often unavailable or non-reproducible because CVE descriptions don't always give you a runnable exploit.
Priya: That makes sense. Getting that initial starting point is usually the biggest hurdle in these kinds of research projects, right?
Nadia: Second reason is that migration isn't just a replay; APIs change, request paths shift, and validators disappear across versions. Sometimes the original entry point no longer reaches the vulnerable logic even though the flaw remains reachable through a different code path.
Elias: And third reason is that configuration defaults shift as well. For example, with one specific denial of service vulnerability in CVE two thousand twenty-four-forty-five thousand two hundred thirty they found that it only hits Django version four point zero if the exploit abandons the reference proof of concept’s vector for a different branch of the same routine > <ref:2611.16588#pg2,s vector for a different branch of the same routine>
Priya: So what this means for us is that we can't just patch based on version numbers anymore; we have to account for how the application actually behaves in that specific environment.
The paper's summary: Nadia: The authors summarize CrossPloit as a benchmark of one hundred twenty-seven migration instances across forty-five CVEs in the Python web ecosystem, and they found that this setup confirms one hundred forty-nine vulnerable prior versions that are absent from the affected-version lists of four different databases.
Elias: So they’ve surfaced a lot of exposure that standard Software Composition Analysis scanners or patching pipelines built on those lists would declare safe. That number, one hundred forty-nine deployments-worth of exposure, is what makes this benchmark so significant for auditing and risk assessment.
Priya: That concrete evidence is what I need to hear before I trust the numbers. What's the actual methodology they use to confirm these prior versions are vulnerable? Is it just running a script?
Nadia: It’s not just running a script; it combines source code analysis, environment repair, and execution based verification. They introduce CrossPloit as this benchmark to make that process repeatable.
Elias: Then they evaluate how much of this entire migration process can be automated. They tested nineteen different tool configurations and a trace guided baseline to see how far enumeration can be automated by AI agents or similar systems >
Nadia: The results are interesting. The best result they found was seventy-four instances with Opus four point six in the OpenHands harness, which resolved fifty-eight point three percent of those instances. That’s a decent automation rate for this kind of complex task.
Priya: Fifty-eight point three percent is good, but what about the failures? I always want to see where the automation breaks down or where it gets confused.
Elias: They did a failure analysis and categorized the problems into five root cause categories: missed API migration, correct strategy but incomplete, oracle misinterpretation, environment setup failure, and illegitimate migration >
Nadia: It looks like they found that failures related to missing the API migration were less common than other issues. Also, environment setup failure accounted for fifteen point five percent to nineteen point three percent of all failures across the different configurations.
The paper's improvements: Nadia: The paper suggests a few ways to improve how we use this benchmark, specifically around what kind of reference material they provide and how tools adapt. One key improvement is that withholding the reference proof of concept and its environment specification from OpenHands with Sonnet four point six reduced the resolve rate from fifty point four percent down to fifteen point zero percent > <ref:2611.16588#pg2>
Elias: That’s a big finding for me. It shows how much context is needed. The comparison highlights the value of supplying those examples and environment information because that context substantially supports successful adaptation in that configuration.
Priya: So, if we want to make these automated migration tools work better, the paper suggests we shouldn't just feed them the vulnerability description; we need to give them the actual running example context.
Nadia: Right. The benchmark itself records each confirmed exposure as executable evidence, which they argue makes that resulting count a sound measurement of the defender’s blind spot >
Elias: That shifts our perspective from just counting potential problems to counting proven, exploitable ones across different versions. It turns the data into something tangible for risk teams.
Priya: I think it means that for automation to be effective, it needs to handle not just the code version but also the surrounding environment details too.
Conclusion: Nadia: So to wrap up, CrossPloit is a benchmark of one hundred twenty-seven migration instances across forty-five CVEs in Python web applications that confirms one hundred forty-nine vulnerable prior versions absent from database lists by providing executable evidence.
Elias: The implication is that we need execution based measurement to audit vulnerability databases because static lists are incomplete, and CrossPloit shows how far enumeration can be automated, with the best configuration resolving seventy-four instances with Opus four point six >
Priya: It means we have a concrete way to measure the defender's blind spot, even if automation isn't perfect yet. We need to focus on improving those API migration and environment setup failure categories if we want to get better results from automated tools.
Nadia: Exactly. The paper shows that having the right reference context matters a lot for making these migrations work successfully >
Elias: We're leaving this one off, but it points toward needing execution based verification to move past just static version checking in vulnerability management and patching pipelines.
Hung Nguyen-Huu, Nguyen Anh Tuan, Ting Zhang, Yue Liu, Chengran Yang, Lwin Khin Shar, Eng Lieh Ouh, David Lo
Singapore Management University · Hanoi University of Science and Technology', 'Monash University
cs.CR, cs.SE
Submitted: 2026-11-12
Updated: 2026-11-12
Code: https://github.com/google/atheris
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Importance score: 79/100
The gist: Affected-version ranges in vulnerability databases decide which deployments scanners, patching pipelines, and incident responders treat as vulnerable…
Key concepts
- Affected-Version Ranges
- These are the version boundaries listed by databases like NVD or OSV that indicate which software versions contain a known vulnerability. The paper argues these ranges are often incomplete, meaning versions outside them might still be vulnerable but appear safe to automated tools.
- CrossPloit
- This is a benchmark tool designed to measure the true exposure of software. It works by taking an exploit for a known version and attempting to trigger it on earlier, unlisted versions. This provides an execution-based measurement of how far enumeration can be automated.
- Enumeration Automation
- This refers to how effectively automated tools can discover all vulnerable software versions without manual effort. The study evaluates various tool configurations and LLM agents to see which methods are best at resolving these cross-version migration tasks.
- Attack Vector Shift
- This describes the difficulty in adapting an exploit from one version to another. The paper notes that shifting the attack vector is particularly challenging, as most automated configurations only resolve a small percentage of instances when this shift is required.
Terminology
Summary
Affected-version ranges in vulnerability databases decide which deployments scanners, patching pipelines, and incident responders treat as vulnerable, yet these ranges are often incomplete The single most informative sentence about the paper's subject and finding is that CrossPloit provides an executionbased measurement of under-reported exposure and of how far its enumeration can be automated.
The Core Problem Addressed
Modern vulnerability management rests on an assumption: a software version outside a CVE’s listed affected-version range is not vulnerable This boundary, built on databases like the NVD and OSV, is often wrong, and this invisibility creates a blind spot for defenders because an exploitable but unlisted version is reported clean by every tool. The task investigated is confirming exposure by demonstrating that a proof-of-concept (PoC) exploit for a listed version can be triggered on an earlier, unlisted version. p. 0
Benchmark Construction and Scope
CrossPloit is introduced as a benchmark of 127 migration instances across 45 CVEs in the Python web ecosystem Each instance includes a version-pinned container, a reference PoC, and a vulnerability-specific oracle. Benchmark construction confirms 149 vulnerable prior versions absent from the affected-version lists of four databases, 127 of which require migration. p. 0 The study covers Django, Flask, FastAPI, Web2py, and packages built on them. p. 1
Evaluation of Automated Enumeration
The evaluation assesses 19 tool configurations and a trace-guided baseline to measure how far enumeration can be automated The best result is 74 instances with Opus-4.6 in the OpenHands harness, resolving 58. p. 63% of instances. p. 0 Attack-vector shifts are particularly difficult, as no configuration resolves more than 37. p. 09% of these instances. p. 0 Higher resolve rates are associated with a smaller share of missed-API failures and a larger share of incomplete executions among the remaining failures. p. 0
Failure Analysis and Root Causes
The analysis of failed runs categorized the problems into five root-cause categories: Missed API Migration, Correct Strategy but Incomplete, Oracle Misinterpretation, Environment Setup Failure, and Illegitimate Migration. Higher resolve rates are associated with a smaller share of Missed API Migration failures and a larger share of Correct Strategy but Incomplete failures. p. 0 Environment Setup Failure accounts for 15. p. 95%–19.3% of failures across configurations and has little rank correlation with resolve rate. p. 9
Contribution of Reference Material
Withholding the reference PoC and its environment specification from OpenHands with Sonnet-4.6 reduced the resolve rate from 50. p. 04% to 15.0%, highlighting the value of the supplied reference package. p. 0 This comparison shows that supplied examples and environment information substantially support successful adaptation in this configuration. p. 9 The benchmark records each confirmed exposure as executable evidence, which makes the resulting count a sound measurement of the defender’s blind spot. p. 3
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Introduction of CrossPloit provides an execution-based measurement of under-reported exposure, allowing defenders to confirm
how far its enumeration can be automated.
This allows security tools to move beyond static inference by showing thatan exploitable but unlisted version is reported clean by the scanner, skipped by the patching campaign, and dismissed during incident response.
-
Benchmark construction confirms 149 vulnerable prior versions absent from all four databases’ affected-version lists, surfacing
149 deployments-worth of exposure that an SCA scanner, patching pipeline, or triage built on those lists would declare safe.
This provides concrete evidence for auditing and risk assessment. -
LLM-based agents can resolve up to 58.3% of migration instances across 18 configurations, demonstrating that
the best configuration resolves 74 instances (58.3%), compared with 8.7% for an adapted trace-guided baseline.
This shows the potential of advanced agents to automate the complex task of cross-version PoC migration. -
The comparison of agent performance across migration categories reveals that
Attack Vector Shift is the hardest category overall, and no configuration exceeds 37.9%,
highlighting a specific area where current automation fails and where future research should focus. -
Withholding the reference PoC and its environment specification reduces the resolve rate from 50.4% to 15.0% for one configuration,
highlighting the value of the supplied reference package
in enabling successful adaptation by providing necessary context for migration.
Sources
- Kimi K2.5: Visual Agentic Intelligence
- LLM Agents for Automated Web Vulnerability Reproduction: Are We There Yet?
- How Inference Compute Shapes Frontier LLM Evaluation
- gpt-oss-120b & gpt-oss-20b Model Card
- Qwen3 Technical Report
- NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Security
- CyberGym: Evaluating AI Agents' Real-World Cybersecurity Capabilities at Scale
- CVE-Bench: A Benchmark for AI Agents' Ability to Exploit Real-World Web Application Vulnerabilities
Related papers
- From Strategy Search to Policy Learning: MCTS-Guided Adversarial Co-Evolution for LLM Safety Alignment
- Privacy Leakage from Recurrent State in LLM Inference
- Incrementally Verifiable Oracle Aided Computation
- Privilege and confidentiality in generative AI workflows
- BadMoE: Backdooring Mixture-of-Experts LLMs via Routing-Aware Triggers and Task-Underutilized Experts
- CoVault: Secure, Scalable Analytics of Personal Data