BELLS-O: Evaluating the Operational Trade-offs of LLM Supervision Systems

arXiv:2606.20668 · cs.CR, cs.AI, cs.LG · Submitted 2026-06-12 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.

Nadia: We're looking at the paper: "BELLS-O: Evaluating the Operational Trade-offs of LLM Supervision Systems".

Elias: Here's the gist of the paper. LLM supervision systems are evaluated through BELLS-O to determine operational trade-offs between different guardrail types under real deployment constraints The picture that emerges is not the one a bitter-lesson reading would predict Operational Scope and Metrics BELLS-O presents the first independent operational benchmark of LLM supervision systems, evaluating 28 systems from 17 providers jointly on detection rate, false-positive rate, latency, and monetary cost The evaluation covers input/output moderation across 11 harm categories and jailbreak detection across 13 attack techniques The authors call the missing dimension operationality, stating that a supervisor is operational only if its detection profile, false-positive rate, latency distribution, and monetary cost are jointly compatible with the deployment that will actually use it.

Nadia: Content Moderation Analysis The content moderation use case is split into input and output moderation Specialized supervisors are operationally dominant on content moderation, matching frontier LLMs on detection at comparably low false-positive rates while running 5–10× faster and ∼10× cheaper The taxonomy covers all threats explicitly mentioned in the GPAI Code of Practice (European Commission, 2025).

Elias: Jailbreak Detection Trade-offs On jailbreak detection, the tradeoff shifts, where frontier LLMs achieve higher detection and lower false-positive rates but at 10–50× higher cost and 5–10× higher latency The picture inverts on jailbreak detection, as no specialized supervisor sits on the deployable frontier.

Nadia: Capability transfers to caution, not detection within a model family, with more capable family members reliably posting lower FPR at flat or lower detection.

Paper summary: Nadia: So we’re looking at this paper called BELLS-O Evaluating the Operational Trade-offs of LLM Supervision Systems. It seems like they've put together something that tries to figure out if these AI guardrails are actually useful when you try to deploy them in the real world.

Elias: Exactly. The main thing here is that existing benchmarks are often biased toward just detection ability, but BELLS-O focuses on operationality by including cost and latency alongside things like detection rate and false positive rate. It’s about making sure the supervisor actually fits into a deployment scenario.

Priya: I’m interested in how they handled those costs and latencies because that’s where the real friction usually happens when you move from a lab setting to something running on actual hardware.

Nadia: Right, and what they claim is this benchmark evaluates twenty-eight systems from seventeen different providers, including both specialized guardrails and frontier generalists used as supervisors. They test input and output moderation across eleven harm categories, plus jailbreak detection across thirteen attack techniques.

Elias: The thesis of BELLS-O is that a supervisor isn't operational unless its detection profile, false positive rate, latency distribution, and monetary cost are all compatible with the actual deployment they’re going to use. They call this missing dimension operationality.

Priya: So it’s not just about whether a model can catch something bad; it’s about how fast and how much money that catching costs in practice. That makes sense because if a system is too slow or too expensive, nobody will actually use it in production.

Nadia: Right, and when you look at the results for content moderation, the specialized supervisors seem to dominate the operational frontier. They match frontier LLMs on detection but run five to ten times faster and cost about ten times less.

Elias: That’s a big claim because it suggests that for general content moderation tasks, you don't need the massive resources of a huge model if you use a smaller, specialized filter.

Priya: But what about jailbreak detection? That seems like where the picture gets more complicated according to the paper.

Nadia: Precisely. On jailbreak detection, the tradeoff shifts in a different way. Frontier LLMs achieve higher detection and lower false positive rates but that comes with ten to fifty times higher cost and five to ten times higher latency.

Elias: The picture inverts there because no specialized supervisor sits on the deployable frontier for jailbreak detection. Instead, it seems like capability transfers toward caution, where more capable family members reliably post a lower false positive rate even if their raw detection rate stays flat or drops slightly.

Priya: So what does that mean for a developer trying to pick a tool? They can’t just pick the model with the highest detection score; they have to optimize based on the specific constraints of their application, whether it’s moderation or jailbreak defense.

Paper summary: Nadia: That’s exactly what they conclude. The deployment decision isn't a single ranking thing anymore; it has to be a constrained optimization over detection, false positive rate, latency, and cost parameterized by the deployment scenario.

Elias: It really reframes how we think about misuse detection in deployed AI applications. It stops being a simple capability check and starts being a deployment-specific choice.

Priya: It’s interesting how they set up the taxonomy too—aligning it with the GPAI Code of Practice, covering things like CBRN risks and harmful manipulation across eleven categories. That gives the testing a very specific regulatory context.

Nadia: And they also handled their data generation carefully to suppress those latent generator fingerprints in synthetic samples, which they did by paraphrasing everything to invert the Mistral generation style.

Elias: That step was significant because it resulted in Mistral’s detection score dropping from about ninety seven percent down to about ninety four percent on the full evaluation set. It shows how sensitive these models are to being specifically tested.

Priya: But I wonder what they didn't test, because the paper flags some limitations. They say they left out multi-turn attacks and gradient based generative jailbreaks from their evaluation harness.

Nadia: That’s a fair limitation because in real application, users aren't just sending one prompt; they are having back and forth with the AI over time. Also, they didn't evaluate agentic settings where supervisors are actually gating tool calls or subtasks.

Elias: Those omissions mean the benchmark is a snapshot rather than a complete picture of what’s possible in complex, multi-step systems. The results refine the idea that you shouldn't just look at a fixed ranking when choosing a supervisor under real deployment constraints.

Priya: So what we get from BELLS-O is this practical guide: if you need fast, cheap moderation on input or output, stick with specialized classifiers. If you’re worried about jailbreaks and can afford the extra latency and cost, then frontier generalists might be better for raw detection power.

Nadia: That summarizes the core finding for me. It’s not a uniform scaling story across the board; it depends entirely on whether you are optimizing for content moderation or jailbreak detection, and what your budget allows.

Elias: And the final point is that capability transfer within a generalist class doesn't improve raw detection rate, but it shifts the tradeoff toward conservatism, meaning more capable models tend to post lower false positive rates even if their detection rate stays flat.

Priya: So for anyone listening who just wants to know what this does for their workflow, the implication is that supervisor selection shouldn't be a simple ranking of performance numbers. It has to be a constrained optimization based on the specific deployment scenario you’re facing.

Nadia: That’s it. BELLS-O shows that misuse detection isn't just about which model can spot the most stuff; it’s about finding the right balance between speed, cost, and accuracy for what you actually need to guard against.

Conclusion: Nadia: So we’ve been looking at BELLS-O, evaluating operational trade-offs for LLM supervision systems. This paper is basically trying to answer if these guardrails are actually useful when you try to deploy them in the real world.

Elias: Right. The authors, Nadia and Elias, they set up this independent benchmark across twenty-eight systems from seventeen different providers. It’s focused on making sure the supervisor isn't just theoretically good at catching bad things, but if it actually fits into a deployment scenario with its speed and cost constraints.

Priya: I’m thinking about what that means for trust. If we use this benchmark, are we getting a realistic picture of what happens when these systems are running twenty-four/seven in production?

Nadia: Exactly. The core idea they push is that a supervisor isn't operational unless its detection profile, the false positive rate, the latency distribution, and the monetary cost all line up with how you actually plan to use it. That’s what they call operationality.

Elias: And when you look at the results for content moderation input—that’s where things get interesting. Specialized classifiers seem to dominate there because they match frontier LLM detection rates but run five to ten times faster and cost about ten times less than the big models.

Priya: So, for general content filtering, it looks like smaller, specialized tools are actually the most practical choice right now if you're worried about running costs and speed.

Nadia: That’s what they found. But then you flip to jailbreak detection, and that picture changes completely because the tradeoffs shift in a different way.

Elias: On jailbreak detection, frontier LLMs get higher detection rates and lower false positive rates but that comes with ten to fifty times higher cost and five to ten times higher latency for those big models.

Priya: That means for jailbreaks, you can’t just go for the most powerful model; you have to weigh that massive cost against the potential risk of missing something.

Nadia: The authors conclude that this isn't a single ranking thing anymore. It has to be a constrained optimization over detection, false positive rate, latency, and cost based on what your specific deployment scenario is.

Elias: So it’s less about which system is the absolute best at everything and more about picking the right tool for the specific job you have in mind.

Priya: That’s a big shift from just looking at raw detection scores. It suggests that supervisor selection needs to be tailored to your actual deployment constraints instead of following a fixed ranking from a chart.

Nadia: And that brings us to the limitations they admit, like not testing multi-turn attacks or agentic settings where supervisors gate tool calls. That means this is a snapshot, not the whole story.

Elias: Yeah, it’s a snapshot that shows what’s possible right now but doesn't cover everything in complex systems yet. It really makes you think about what those next big challenges are for AI safety research.

University of Graz · Supervised Program for Alignment Research (SPAR) · Centre pour la Sécurité de l’IA

cs.CR, cs.AI, cs.LG

Submitted: 2026-06-12

Updated: 2026-11-12

Comments: Accepted at NeurIPS 2026 (Track on Evaluations and Datasets). Previously presented at the ICML 2026 Workshop on Trustworthy AI for Good (AI4GOOD). 60 pages, 6 figures; 10-page main text plus appendices

Code: https://github.com/qfeuilla/Beha

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 92/100

The gist: LLM supervision systems are evaluated through BELLS-O to determine operational trade-offs between different guardrail types under real deployment constraints The picture that emerges is not the one a…

Key concepts

Operationality
A supervisor is only considered operational if its performance metrics—detection rate, false-positive rate, latency distribution, and cost—are compatible with the actual deployment environment. It means a system must work well within the real-world limits of where it will be used.
Content Moderation vs. Jailbreak Detection
The evaluation splits into two areas: content moderation (input/output filtering) and jailbreak detection (identifying model evasion). Specialized supervisors excel at moderation due to efficiency, while generalists are better for jailbreaks because they balance high detection with acceptable false-positive rates.
Capability Transfer
When comparing different models, raw capability does not always improve performance in supervision. Instead, more capable models tend to shift the trade-off toward conservatism. They reliably post lower false-positive rates while maintaining similar or slightly reduced detection rates compared to less capable models.

Terminology

Summary

LLM supervision systems are evaluated through BELLS-O to determine operational trade-offs between different guardrail types under real deployment constraints The picture that emerges is not the one a bitter-lesson reading would predict

Operational Scope and Metrics

BELLS-O presents the first independent operational benchmark of LLM supervision systems, evaluating 28 systems from 17 providers jointly on detection rate, false-positive rate, latency, and monetary cost The evaluation covers input/output moderation across 11 harm categories and jailbreak detection across 13 attack techniques The authors call the missing dimension operationality, stating that a supervisor is operational only if its detection profile, false-positive rate, latency distribution, and monetary cost are jointly compatible with the deployment that will actually use it.

Content Moderation Analysis

The content moderation use case is split into input and output moderation Specialized supervisors are operationally dominant on content moderation, matching frontier LLMs on detection at comparably low false-positive rates while running 5–10× faster and ∼10× cheaper The taxonomy covers all threats explicitly mentioned in the GPAI Code of Practice (European Commission, 2025).

Jailbreak Detection Trade-offs

On jailbreak detection, the tradeoff shifts, where frontier LLMs achieve higher detection and lower false-positive rates but at 10–50× higher cost and 5–10× higher latency The picture inverts on jailbreak detection, as no specialized supervisor sits on the deployable frontier. Capability transfers to caution, not detection within a model family, with more capable family members reliably posting lower FPR at flat or lower detection. p. 1

Where the paper says it

them on detection. On jailbreak detection the frontier flips, but capability does not predict detection within a model family; it shifts the FPR/detection tradeoff toward conservatism, with more capable family members reliably posting lower FPR at flat or lower detection.

Limitations and Future Directions

BELLS-O is a snapshot, highlighting limitations such as the omission of multi-turn attacks and gradient-based generative jailbreaks from the evaluation harness The benchmark does not evaluate agentic settings where supervisors gate tool calls or sub-tasks. The results refine rather than confirm the bitter-lesson reading of misuse detection, suggesting that supervisor selection should be re-queried under deployment-specific constraints rather than read off a fixed ranking. p. 7

Where the paper says it

Small generalists like ministral-3b-2512, already sit at the boundary between classes on content moderation, matching specialized detection at sub-second latency. As small-model capability and serving infrastructure mature in parallel, we expect the operational frontier to keep moving; which is precisely why supervisor selection should be re-queried under deployment-specific constraints rather than read off a fixed ranking.

Data Generation and Fingerprint Removal

To suppress latent generator fingerprints in synthetic data, every generated sample is paraphrased The paraphraser is trained to invert the Mistral generation style, using a LoRA adapter on top of Wizard-Vicuna-13B-Uncensored. This step resulted in Mistral’s detection score dropping from ≈ 97% to ≈ 94% on the full evaluation set. p. 1

Where the paper says it

GuardBench (Bassani & Sanchez, 2024) is the closest open analogue to BELLS-O, but is restricted to a narrow set of jailbreak scenarios, predates the current guardrail and frontier-LLM generations, and reports neither cost nor latency. CircleGuardBench (White Circle, 2025) folds accuracy, speed, and adversarial resistance into a single composite score; we deliberately keep the four operational dimensions separate, because the right trade-off depends on the deployment, and a composite hides which guardrail to pick at your latency budget.

Comparative Performance Summary

On content moderation, specialized classifiers Pareto-dominate the deployable frontier, matching frontier-LLM detection at substantially lower latency and cost On jailbreak detection, generalists are the only systems that combine high detection with deployment-compatible FPR but pay for that robustness in latency and cost. The deployment decision is therefore a constrained optimisation over detection, FPR, latency, and cost parameterized by the deployment scenario. p. 7

Where the paper says it

The deployment decision is therefore not a ranking on a single axis. It is a constrained optimisation over detection, FPR, latency, and cost, parameterised by the deployment scenario: highthroughput input moderation, low-latency agentic loops, and asynchronous batch evaluation each select different points on the frontier.

Harm Taxonomy Details

The taxonomy covers 11 harm categories aligned with the systemic-risk classes of the GPAI Code of Practice (European Commission, 2025) These categories include CBRN Risks, Cyber Offenses, and Harmful Manipulation. The content moderation input dataset is balanced across all harm categories, restricted to unambiguously harmful or benign samples. p. 2

Where the paper says it

Full percategory descriptions are in Appendix A. 2 Content Moderation Input Dataset The Content Moderation Input Dataset is balanced across all harm categories, restricted to unambiguously harmful or benign samples, and excludes borderline content.

Jailbreak Taxonomy Structure

The jailbreak taxonomy is built around the evaluation purpose, testing whether a supervisor can detect model-agnostic jailbreaks, not whether a jailbreak actually elicits harmful behaviour from a particular target model Techniques are grouped into three categories: Obfuscation, Hacking, and Manipulation. The in-house jailbreak dataset features 74 Low-Resource-Language augmentation samples that failed the quality audit even after five generation attempts. p. 4

Where the paper says it

74 Low-Resource-Language augmentation samples failed the quality audit even after five generation attempts. This stems from the same property that makes the technique a jailbreak in the first place: models are less capable in these languages, so degradation in the translation pipeline was expected and partially unavoidable.

Experimental Setup and Reproducibility

The authors release a benchmarking framework with BELLS-O, available at https://github.com/CentreSecuriteIA/BELLS-O For API-based supervisors, total cost is computed from per-prompt token usage and the provider’s listed input/output prices Latency is measured as the time it takes to generate one judgement for a given sample. The class-level latency separation is robust to this confound because both classes are measured from the same node, but absolute latency numbers should not be extrapolated to other deployment topologies. p. 7

Where the paper says it

All API calls were issued from a single RunPod node in the US; supervisors hosted closer to the caller would post lower numbers. The class-level latency separation is robust to this, but absolute numbers are not.

Conclusion on Deployment Strategy

Misuse detection is not a uniform scaling story, with specialized classifiers Pareto-dominating the operational frontier on content moderation On jailbreak detection, that frontier flips to frontier generalists. The deployment decision is therefore not a ranking on a single axis but a constrained optimisation over detection, FPR, latency, and cost parameterized by the deployment scenario. p. 7

Where the paper says it

The deployment decision is therefore not a ranking on a single axis. It is a constrained optimisation over detection, FPR, latency, and cost, parameterised by the deployment scenario: highthroughput input moderation, low-latency agentic loops, and asynchronous batch evaluation each select different points on the frontier.

Leaderboard Insights

Table 8 shows that for content moderation input, Qwen3Guard-Gen-8B achieves 94.5% detection at an FPR of 0. p. 1

Where the paper says it

We evaluate 28 supervision systems from 17 providers, spanning every major specialized guardrail and a representative slice of frontier LLMs repurposed as supervisors. Systems are scored under a unified harness on three use cases (input and output content moderation, jailbreak detection) and four operational metrics (detection rate, FPR, latency, cost).

7 Table 11 provides the per-category results for content moderation input across all models. p. 5
Where the paper says it

6 Results Figure 2 reports BELLS-O’s main results across the three operational supervision settings: input-side content moderation, output-side content moderation, and jailbreak detection. Two structural patterns frame the per-task discussion.

Table 13 reports jailbreak detection rate per dataset, showing that the prompt-injection sub-task is materially harder for most systems than the jailbreak sub-task. p. 7
Where the paper says it

The headline jailbreak detection rate averages two prompt-injection datasets into the same mean as four jailbreak datasets; per-dataset breakdowns (Appendix C. 3) show the gap between the two sub-tasks, particularly on DeepSet, is large.

The headline jailbreak detection rate averages six datasets, showing that the prompt-injection result is dataset-bound. p. 7
Where the paper says it

(iii) Aggregated jailbreak numbers. The headline jailbreak detection rate averages two prompt-injection datasets into the same mean as four jailbreak datasets; per-dataset breakdowns (Appendix C.

Key Findings on Capability Transfer

Within the generalist class, raw capability does not improve detection rate but shifts the FPR/detection tradeoff toward conservatism More capable family members reliably post lower FPR while detection holds flat or drops. The picture is partially complicated by two small generalists that sit at the boundary between classes on content moderation. p. 6

Where the paper says it

The small generalist ministral-3b sit at the boundary between classes on content moderation. The picture is partially complicated by two small generalists.

Specific Model Comparisons

For content moderation input, GPT-5.2 repurposed as a supervisor achieves 93. p. 0

Where the paper says it

Dozens of candidates exist, ranging from small specialized classifiers to repurposed frontier LLMs used as zero-shot judges, and they differ by more than an order of magnitude in detection rate, latency and per-call cost. We call the missing dimension operationality: a supervisor is operational only if its detection profile, false-positive rate, latency distribution, and monetary cost are jointly compatible with the deployment that will actually use it.

9% detection at an FPR of 0. p. 1
Where the paper says it

We evaluate 28 supervision systems from 17 providers, spanning every major specialized guardrail and a representative slice of frontier LLMs repurposed as supervisors. Systems are scored under a unified harness on three use cases (input and output content moderation, jailbreak detection) and four operational metrics (detection rate, FPR, latency, cost).

3 For jailbreak detection, Lakera Guard default achieves 95. p. 6
Where the paper says it

2 Jailbreak Detection The picture inverts on jailbreak detection. No specialized supervisor sits on the deployable frontier (Figure 2, right): every specialized system with high jailbreak detection exceeds 15% FPR on benign jailbreak-shaped prompts, with Lakera Guard the most extreme at 95.

4% detection but has a high FPR of 54. p. 6
Where the paper says it

2 Jailbreak Detection The picture inverts on jailbreak detection. No specialized supervisor sits on the deployable frontier (Figure 2, right): every specialized system with high jailbreak detection exceeds 15% FPR on benign jailbreak-shaped prompts, with Lakera Guard the most extreme at 95.

9%. The minimal prompt strictly dominates the more elaborate ones for generalist LLMs on content moderation, having the lowest token cost and latency. p. 0
Where the paper says it

BELLS-O: Evaluating the Operational Trade-offs of LLM Supervision Systems Leonhard Waibl 1 2 Felix Michalak 2 Hadrien Mariaccia 3 Abstract LLM supervision systems, namely input/output moderation filters and jailbreak detectors, are the primary safeguard against misuse in deployed AI applications, yet existing benchmarks are often vendor-biased, omit cost and latency, and rarely compare specialized guardrails against repurposed generalist LLMs. We present BELLS-O (Benchmark for the Evaluation of LLM Supervision Systems – Operational), the first independent operational benchmark of LLM supervision systems.

Synthetic Data Pipeline Details

The input pipeline runs in two stages, where a planning model produces a structured generation task and an uncensored model generates prompts based on that document The quality-control filter instructs the model to reason through four steps before selecting the best and most diverse prompts. The fingerprint-removal paraphrasing step is validated by the post-paraphrase score shift on Mistral, where detection score dropped from ≈ 97% to ≈ 94%. p. 7

Where the paper says it

Quality control on synthetic samples was performed on random subsamples by the authors rather than at scale with multi-rater human approval. The fingerprint-removal paraphrasing step was validated against benchmark-score shift, not against an independent fingerprint probe.

Final Summary Statement

On content moderation, small specialized classifiers Pareto-dominate the operational frontier, matching frontier-LLM detection at one to two orders of magnitude lower latency and cost On jailbreak detection, that frontier flips to frontier generalists, but raw capability transfers to caution rather than detection within a family. The bitter lesson is not that obvious. p. 7

Where the paper says it

5’s; Ministral 14B at roughly a quarter of 3B’s) without improving detection. The bitter lesson is not that obvious.

The picture also looks unstable in a productive direction, suggesting supervisor selection should be re-queried under deployment-specific constraints rather than read off a fixed ranking. p. 7
Where the paper says it

Small generalists like ministral-3b-2512, already sit at the boundary between classes on content moderation, matching specialized detection at sub-second latency. As small-model capability and serving infrastructure mature in parallel, we expect the operational frontier to keep moving; which is precisely why supervisor selection should be re-queried under deployment-specific constraints rather than read off a fixed ranking.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Improve deployment selection by using a constrained optimization over detection, FPR, latency, and cost parameterized by specific deployment scenarios like high-throughput input moderation or low-latency agentic loops. This allows practitioners to select the right supervisor based on operational constraints rather than a single capability ranking.

  2. Implement specialized supervisors for content moderation tasks where they Pareto-dominate the deployable frontier: the same detection at substantially lower latency and cost. This means deploying small, fine-tuned classifiers for high-volume moderation tasks where latency is critical.

  3. Utilize generalist LLMs repurposed as supervisors only when the deployment scenario requires generalisation over adversarially transformed inputs, as they are the only class that can combine high detection with deployment-compatible FPR.

  4. Design a system to leverage model capability for caution rather than direct detection within a model family, noting that more capable family members reliably post lower FPR at flat or lower detection when facing jailbreak detection.

  5. Develop input/output moderation systems that are explicitly aligned with the 11-category harm taxonomy, which is mapped directly to the systemic-risk categories of the GPAI Code of Practice (European Commission, 2025), ensuring regulation-aligned harm coverage.

  6. For jailbreak detection, use a taxonomy based on evaluation purpose (we test whether a supervisor can detect model-agnostic jailbreaks) and include techniques from the JailbreakBench and other relevant literature to ensure comprehensive coverage of non-generative attacks.

  7. Employ a paraphrasing step after generating synthetic data to suppress latent generator fingerprints, which is crucial for ensuring that the evaluation results are not inflated by latent author fingerprint[s].

Abstract

LLM supervision systems, namely input/output moderation filters and jailbreak detectors, are the primary safeguard against misuse in deployed AI applications, yet existing benchmarks are often vendor-biased, omit cost and latency, and rarely compare specialized guardrails against repurposed generalist LLMs. We present BELLS-O (Benchmark for the Evaluation of LLM Supervision Systems, Operational), an independent operational benchmark that evaluates 33 systems from 17 developers: specialized guardrails (e.g., Llama Guard 4, ShieldGemma-2B, Lakera Guard) and frontier generalists repurposed as supervisors (e.g., GPT-5.4, Claude Sonnet 4.6, Grok 4.1 Fast), jointly on detection rate, false-positive rate, latency, and cost. We cover input/output moderation across 11 harm categories, jailbreak detection across 13 attack techniques, and prompt-injection detection, using in-house datasets built from handcrafted prompts, curated external samples, and quality-controlled synthetic generation, with synthetic moderation samples paraphrased to suppress generator fingerprints. Content-moderation runs are repeated five times and all rates reported with 95% intervals. The Pareto frontier reveals use-case-dependent tradeoffs. On content moderation, specialized supervisors are operationally ahead: top systems match the best generalist LLMs on detection (93-96%, no significant difference) at low false-positive rates while running 5x faster; cost depends more on serving setup than on class. On jailbreak detection, generalist LLMs achieve higher median detection on three of four datasets and lower false-positive rates, but at higher latency and, for closed frontier models, 10-100x higher cost. On prompt injection, the two best dedicated detectors lead. We release the framework, an interactive leaderboard, and four datasets as a vendor-neutral basis for selecting safeguards under deployment constraints.

Sources

Related papers