Escalate or Close: How Fast-Tier Models Handle Held-Out Web Attack Families in Two-Tier Triage

summary

The gist

In two-tier triage a fast model decides most web requests and escalates the rest to a slow tier, so an attack the fast tier closes is never examined This paper investigates how fast-tier models…

In short

The study tested how fast-tier security models handle web attacks they weren't trained on. They compared supervised detectors against hosted decision models like Jev and Clef using held-out attack families. Findings show that while supervised detectors can be confident, hosted models escalate nearly all held-out families within a small benign budget, making them effective for triage.

Key concepts

Two-Tier Triage
This is a security system with two stages: a fast tier handles most requests quickly and decides immediately, and if it's unsure or suspicious, the request is sent to a slower tier for deeper examination. The paper investigates how the fast tier manages attacks it hasn't seen.
Supervised Detectors
These are models trained specifically on known types of web attacks. They are good at giving confident 'benign' verdicts on new requests they recognize, but they can miss entirely new attack families that were not in their training data.
Hosted Decision Models (Jev/Clef)
These are decision models that run within the fast tier and return a typed answer quickly. Unlike supervised detectors, these models make budget decisions based on their internal logic rather than just pattern matching known attacks, allowing them to escalate most held-out families.

Terminology used across episodes

This episode discusses

The paper

Escalate or Close: How Fast-Tier Models Handle Held-Out Web Attack Families in Two-Tier Triage · Read on arXiv

Maciej Kisielewicz, Michal Kędziora, Ngoc Thanh Nguyen

Wroclaw University of Science and Technology

In two-tier triage a fast model decides most web requests and escalates the rest to a slow tier, so an attack the fast tier closes is never examined. We hold each of seven attack families out of training, recalibration and thresholds, splice the payloads into parameters of real browsing traffic, and compare zero-shot decision models, which return a probability for a typed question, with supervised detectors, a local language model, the OWASP Core Rule Set and an anomaly baseline. Detectors trained on the known families can give much of a held-out family a confident benign verdict: over 10 training runs, ModernBERT gives 37% to 60% of held-out path traversal a probability below 0.05 and escalates only 50% to 81% of it within a 5% benign budget, against nearly all of the same requests when traversal is known. Jev and Clef, two hosted decision models whose budget decisions do not depend on the held-out family, escalate at least 97.6% of every family within the same budget, but their undisclosed training data may contain these public families. Tests on recent exploits need controls for request shape: Jev escalates 37% of benign twins (exploit requests with the attack replaced by legitimate values), and detectors trained on older exploits learn what scanner requests look like. A search guided by Jev's probability gets 43% of held-out attacks past Jev within 300 queries, but Jev's answers vary: asked again, it escalates those requests in 60% of its answers.

Transcript

Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.

Nadia: We're looking at the paper: "Escalate or Close".

Elias: Here's the gist of the paper. In two-tier triage a fast model decides most web requests and escalates the rest to a slow tier, so an attack the fast tier closes is never examined This paper investigates how fast-tier models handle held-out web attack families in two-tier triage by comparing various decision models against supervised detectors and rule sets The findings reveal that while supervised detectors can give confident benign verdicts on new attacks, hosted decision models like Jev and Clef escalate nearly all of every held-out family within a 5% benign budget Pruning redundant reasoning tokens The study holds out one attack family at a time from training, recalibration and every threshold, and asks four questions to evaluate the fast tier’s performance The comparison involves zero-shot decision models, supervised detectors trained on known families like ModernBERT, a local language model, the OWASP Core Rule Set (CRS), and an anomaly baseline The main data are real browsing requests from the open-appsec WAF comparison with public attack payloads spliced in, including Log4Shell and Shellshock How the training signal is built The methodology involves holding out one attack family at a time from training, recalibration and thresholds The fast tier can sit inline, deciding before a request reaches the application or reading a copy of the traffic where a request it closes has already been served The models compared include Jev and Clef, which are hosted decision models that return typed answers within a few hundred milliseconds Supervised detectors are trained on known families, such as ModernBERT, which is fine-tuned per fold on 8,000 benign training requests chosen by hash and every known-family training attack Where the gains come from The results show that detectors trained on older exploits learn what scanner requests look like, which is reflected in their ability to escalate many benign twins of recent exploits Jev and Clef, whose budget decisions do not depend on the held-out family, escalate at least 97.6% of every family within the same budget The study also shows that a search guided by Jev’s probability gets 43% of held-out attacks past Jev within 300 queries Evaluating learned detectors The supervised detectors miss some held-out families, with ModernBERT giving 37% to 60% of held-out path traversal a probability below 0.05 over over 10 training runs The n-gram model’s escalation of the other families rests largely on how the benchmark encodes payloads Hosted models like Jev and Clef escalate at least 97.6% of every family within a 5% benign budget, but their undisclosed training data may contain these public families Escalating held-out attacks Under the plain budget rule at a 5% budget, Jev and Clef escalate at least 97.6% of every held-out family while the supervised detectors lose much of traversal A conformal rule’s guarantee does not cover a held-out family, and disagreement between questions or models never escalated significantly more attacks The study also shows that padding benign requests makes Clef escalate 97% of them, enough to flood the slow tier Cost and latency analysis A hosted decision model adds a few hundred milliseconds to each request, so it suits triage of a copy of the traffic better than inline use With the five questions of our set, a request of the folds costs Jev 609 input tokens on average A cascade with the rule set as cheap first stage calls Jev on a fifth of traffic but escalates fewer of the recent exploits than Jev alone In a two-tier pipeline the fast tier decides alone on everything it does not escalate, so how it treats attack families it was never fitted on bounds how many new attacks the pipeline can catch Holding families out of everything a deployment fits, we found that detectors trained on the known families gave large parts of some held-out families a confident benign verdict, by amounts that varied between training runs Jev and Clef, whose budget decisions the hold-out cannot change, escalated at least 97.6% of every held-out family within a 5% benign budget Benign twins showed that detectors trained on older exploits had learned what scanner requests look like A search guided by Jev’s probability got 43% of held-out attacks past it within 300 queries but only 18% past it in most of 5 repeated calls The protocol can be reused on new fast-tier models with the data checks of Appendix D Limitations Exposure during pretraining means a held-out family is unseen by everything a deployment fits, but perhaps not by a pretrained model Our results for zero-shot models therefore measure behavior on attack families absent from the deployment’s own data, and may overstate how they would meet an attack class absent from every pretraining corpus Training runs for models like ModernBERT vary between training runs with identical data, so the three main runs understate their spread Only three families leave enough held-out attacks for main folds, and they differ in the type of attack The conformal guarantee on known families closes about the 5% of attacks its guarantee allows or fewer, with a small excess for Clef-flash and Laya Training runs for ModernBERT vary between training runs with identical data (Section 5) The splits are determined by the payload, so encoding variants of one payload can sit in both training and test Results on ECML/PKDD 2007 The results on ECML/PKDD 2007 show that ModernBERT and the n-gram model lead (Table 7) ModernBERT escalates at least 82.9% and the n-gram model at least 78.6% of every held-out family, a result that likely reflects the corpus more than the models The hosted decision models read the same benign requests as suspicious Zero-shot, Jev scores 91.8% of the benign ECML test requests above 0.5 (median 0.80), against 0.6% of the real benign traffic of open-appsec (median 0.07) Evasion and Flooding Fixed edits lower Jev’s and Clef’s escalation of attacks by at most a few points, while an injected assertion cuts that of Qwen2.5-7B by half or more Padding lowers the supervised detectors’ escalation from 93% to 43% (ModernBERT) and from 58% to 15% (n-gram model) A search guided by the model’s probability gets past Jev mostly with requests at its threshold, which Jev often escalates when asked again Padding is cheaper still, as with about 3,000 characters of opaque parameters added, Clef escalates 97% of benign requests and ModernBERT 50% Latency and throughput Over the 72,095 live calls of our leave-one-family-out experiments, Jev answered in 300 ms at the median and 473 ms at the 99th percentile Clef and Clef-flash answered 55,842 calls each in 375 and 372 ms at the median and 1,103 and 936 ms at the 99th percentile Locally, on one Apple M5 laptop, ModernBERT answers in 21 ms at the median and the n-gram model in under a millisecond Cost per request With the five questions of our set, a request of the folds costs Jev 609 input tokens on average, or 701 tokens for Clef and Clef-flash A service with ten million requests a day would pay about 93,000 a year to put Jev in front of all of its traffic and 614,000 for Clef Under a budget rule the slow tier examines a share b of benign traffic, so it costs more than the fast tier once one examination costs more than 1/b fast-tier calls Discussion Evaluate a fast tier on families held out from everything it fits, over several training runs Treat a supervised detector’s confident benign verdicts on new attacks with caution, as its recalibration map is learned from the families it already knows Route by a benign budget, where a conformal rule’s guarantee covers only attacks exchangeable with those it was fitted on Test a cascade’s first stage on held-out families and on recent exploits A Ethical Considerations This study involved no live systems, and every attack in our experiments comes from public sources The artifact contains no text of the recorded requests, and we report model versions and dates so that the providers can reproduce our observations The study involved no human participants B Open Science The artifact, available at https://anonymous.4open.science/r/escalate-or-close-artifact-3794/, contains the code for every experiment Datasets are not redistributed, and the nuclei templates are fetched at two pinned commits Open models are pinned to Hugging Face revisions with file hashes and Jev to jev-1.13.0 The rule set runs from pinned container images, with release 3.3.2 verified by its SHA-256 C Results on ECML/PKDD 2007 The results on ECML/PKDD 2007 show that ModernBERT and the n-gram model lead (Table 7) The hosted decision models read the same benign requests as suspicious Zero-shot, Jev scores 91.8% of the benign ECML test requests above 0.5 (median 0.80), against 0.6% of the real benign traffic of open-appsec (median 0.07) The monotone recalibration does not reorder requests, so it cannot separate attacks from benign requests that a model scores as high The n-gram model detects each held-out family with an AUROC of 0.92 to 1.00 The ECML/PKDD 2007’s attack families share a cue, and a model trained on the known families can flag a held-out one by cues they share The n-gram model's benign requests in turn carry attack-like noise, an encoded quote in 12% of them, which a model that judges content finds suspicious D Five Pitfalls in Web Attack Data The benchmark’s attacks share one template, and a classifier given only the method, Content-Type, Host, path and parameter names separates the harness’s attacks from benign traffic with an AUROC of 1.

Title and authors: Tom: So to wrap up what we just heard, this paper is really looking at how those fast models in a two-tier system handle attack families they haven't been trained on, which is a big deal because if the fast tier closes an attack it never gets looked at by the slower tier.

Elias: Exactly. They compared different decision models against supervised detectors and rule sets to see how robust they are when faced with unseen threats. It’s about testing those first layers of defense under pressure.

Nadia: The main finding there is that while those detectors trained on known families can give confident benign verdicts on new attacks, the hosted decision models, like Jev and Clef, they escalate almost every single held-out family within a five percent benign budget. That means their decision process doesn't seem to care which specific attack family you hold out of training data.

Elias: That’s the core idea—the budget decisions they make don't change based on the specific held-out thing. They escalate at least ninety seven point six percent of every single family within that five percent benign budget. That seems pretty strong for a fast tier model, especially when you think about how much traffic it handles before it has to pass things along to the slower tier.

Nadia: It shows that those learned detectors, even if they miss some held-out families like path traversal with ModernBERT, still learn what scanner requests look like from older exploits. That means they can guess benign twins of recent exploits pretty well.

Elias: But the paper also points out that their supervised detector performance really depends on how the benchmark encodes those payloads, which suggests that training data quality is a big factor in how well those detectors perform on new things.

Nadia: Precisely. And they show that when you use a search guided by Jev's probability, you can get forty three percent of held-out attacks past it within three hundred queries, but only eighteen percent after five repeated calls <ref:2611.08229#pg2>but asked again, Jev escalates those requests in sixty percent of its answer.

The paper's summary: Tom: So we're looking at how to make this research even better now that we see what the models can do, and there are some suggestions for improvement in this paper on fast-tier triage.

Nadia: Yeah, the authors suggest a few things to keep in mind when you’re applying these findings to a real defense system. First, they say you should treat those supervised detectors with more caution when they give confident benign verdicts on new attack families because their training map is just learned from old families.

Elias: Right. The point there is that a model might score something high as benign based on its past experience, but if it's a totally new family, that confidence can be misleading. It reminds us of how models can get stuck in local optima when they only know what they've seen before.

Priya: And then they have this idea about using benign twins, which are just exploit requests where you replace the attack part with something legitimate to see how much recall comes from the request shape itself. It’s a way to get a better sense of how much the model is relying on structure versus just pattern matching.

Nadia: Exactly. They also suggest that when you try to bypass these fast tiers, like using a search guided by Jev's probability, you need to be careful because if you ask the model again, it escalates those requests in sixty percent of its answers, so repeated calls are needed for reliable evasion testing.

Elias: That makes sense. It implies that just getting one answer isn't enough when the model’s output is probabilistic; you need to probe it a few times to see if the escalation stays consistent across different checks.

Priya: And on the side of defense, they mention that padding benign requests can be used to intentionally flood the slow tier with models like Clef, which helps make Clef escalate ninety seven percent of them. So you can use benign traffic manipulation as a way to test how much load the slower tier can handle before it gets overwhelmed.

Nadia: That’s a practical thought. It suggests that you might be able to use known attack families and some padding to stress-test your slower systems without actually running an attack against them directly.

Elias: But we still have those limitations they flag, like the fact that zero-shot models only measure behavior on attack families absent from the deployment's own data set. So even with these improvements, you can’t fully trust a model to handle something completely outside of its training experience.

Priya: It’s a good reminder that the models are still learning and not perfect for everything, especially when dealing with truly novel threats that haven't been seen in any of the training data.

Nadia: So yeah, the improvements give us better tools for testing and defense strategies, but we still have to keep an eye on those gaps between what a model knows and what it actually encounters in the wild.

The paper's improvements: Nadia: So to wrap up this look at "Escalate or Close: How Fast-Tier Models Handle Held-Out Web Attack Families in Two-Tier Triage," we’ve seen how hosted decision models like Jev and Clef handle unseen attacks by escalating them within a small benign budget.

Elias: Exactly. They show that these decision models have a stable way of escalating, regardless of which specific attack family you hold out from training data, as long as you stick to that five percent benign budget. It’s about the robustness of their budget decisions under uncertainty.

Priya: I just want to say that the data really shows us that supervised detectors can give confident benign verdicts on new attacks, but we need to be very careful with those scores because they are only calibrated on what they already know. That means we can't just trust a high score as a guaranteed safe verdict for something entirely new.

Nadia: True. And the paper shows that even if supervised detectors miss some families, there’s still utility in using them for things they’ve seen before, like learning what scanner requests look like from older exploits. It's not about finding a perfect detector right away; it's about using the tools we have to manage risk.

Elias: I agree. The point is that the architecture itself—the two-tier triage—is what’s being tested here, and it seems like hosted models provide a solid layer of defense against unknown threats because their escalation behavior stays consistent. It’s a system design win, not just a model training win.

Priya: For someone listening who only cares about privacy and measurement, what this means is that we have better ways to test the boundaries of our security systems using traffic manipulation, like padding benign requests to stress test the slow tier's capacity. It’s practical data on system limits.

Nadia: So we’ve seen how these models perform against held-out families, and it confirms that hosted models are strong tools for handling unknown threats in a fast pipeline. The implications are that we can deploy these types of decision models knowing they have a predictable way to handle new things within defined constraints.

Elias: Right. It’s a solid piece of engineering analysis showing how system design interacts with model performance under real-world pressure. It gives us concrete parameters for deployment decisions moving forward.

Priya: I just want to emphasize that while the results are interesting, we still have those caveats about zero-shot models and the dependence on specific training data quality when

Conclusion: Nadia: So we’ve looked at "Escalate or Close: How Fast-Tier Models Handle Held-Out Web Attack Families in Two-Tier Triage," and it really boils down to this: hosted decision models like Jev and Clef seem very good at handling unknown threats because their budget decisions stay stable even when you hold out an attack family from training.

Elias: Exactly. They escalate at least ninety seven point six percent of every single family within a five percent benign budget, which shows the system design itself is crucial here, not just the model training. It’s about the robustness of those budget decisions under uncertainty.

Priya: I still want to stress that while these hosted models are strong at handling unknown threats by simply escalating them within a defined constraint, we still need to be cautious about what we trust when the model has been trained on something different than what it’s actually seeing now. The zero-shot models have limitations because they measure behavior on families absent from the deployment's own data set.

Nadia: Right, so that’s the summary of this study on "Escalate or Close: How Fast-Tier Models Handle Held-Out Web Attack Families in Two-Tier Triage." It shows us that hosted models are a strong baseline for handling unknown threats by escalating them within a budget, but you still have to be mindful of what you trust when the model has been trained on something different than what it’s actually seeing.

Elias: Yeah, and it leaves us with the idea that the system design itself is crucial because it dictates how these models perform under pressure. It sets the stage for thinking about how we integrate these decision-making components into larger security architectures.

Priya: For someone managing a system under these constraints, what this means is that we have better ways to test our boundaries using traffic manipulation, like padding benign requests to stress test the slow tier's capacity. It’s practical data on system limits.

Nadia: Right. We’ve got a lot more to look at in this area, especially how these models interact with those autonomous agents we talk about in the other papers, so we'll keep digging into that next.

More episodes

← Home