Coverage, Not Difficulty, Sets How Much Synthetic Data an Activation Probe Needs

arXiv:2610.10594 · cs.LG · Submitted 2026-10-06 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom, and with me are Jane, Lu, senior AI researcher at Tsinghua, Meng, lead engineer at a mysterious AI startup and Lalam, the in-house Large Language Model.

Jane: Today's paper: "Coverage, Not Difficulty, Sets How Much Synthetic Data an Activation Probe Needs".

Tom: The gist The need is set by what is monitored: coverage, not per-kind difficulty;

Jane: First, who's behind it and why it matters.

Title and authors: Tom: So we're talking about this paper, "Coverage, Not Difficulty, Sets How Much Synthetic Data an Activation Probe Needs." It looks like it’s tackling a core problem in how we train these AI monitors—how much synthetic data you actually need for a probe to work.

Jane: Exactly. The authors are looking at what they monitor and how that dictates the data requirement, moving away from just throwing more kinds of data at the problem hoping it works better.

Lu: This paper is really focused on setting the need by what is monitored, not just per-kind difficulty; it suggests a distribution needs to saturate based on coverage.

Meng: So instead of focusing on how hard one kind of conversation is to generate, they are looking at whether that one kind covers the others.

Tom: Right. They measure learning curves over three different monitoring targets: high-stakes situations, replies harmful to a person, and replies that don't follow user instructions across fourteen held-out evaluation distributions.

Jane: And the key finding is that concept and distribution account for forty-two to forty-five percent of the variance in how much data is needed.

Lu: They found that for every kind, there's a median half-gain size of seven to eleven own-kind synthetic samples under all three concepts together.

Meng: But what really sets the value of that half-gain size is coverage, not per-kind difficulty—the number of samples of its own kind a distribution needs to saturate.

Tom: That means if one kind covers everything else, you need to saturate that one kind's distribution, whereas if they don't cover another concept, you might need more kinds.

Jane: It suggests that breadth alone doesn't fix the budget equally across all monitoring targets; generating many kinds isn't always the right move.

Lu: The paper points out that generating many kinds of samples is necessary where one kind covers another, as it does under instruction following, but it becomes nearly redundant where one kind covers the rest of them, like with high-stakes situations.

Title and authors: Meng: So the depth you need to generate should track where its learning curve actually saturates for that specific concept or distribution.

Tom: And they give a way for us to measure that saturation point by looking at how samples transfer between kinds using a transfer AUROC T, which ranks concepts as a multiplier between one and the number of kinds it lands within a factor of two of the measured need.

Jane: That transfer AUROC T is not pinning an absolute half-gain size but it ranks the concepts, which gives us a better idea of where to focus our budget.

Lu: They propose this rule, R hat equals one plus two times K minus one times one minus T, has no fitted parameter, which means we can run it right away.

Meng: It’s practical because it lets a practitioner write down the kinds of conversation a concept covers and measure how far samples of one kind transfer to the others without knowing any deployment traffic.

Tom: And they found that this ordering holds on real samples, four generators, two prompts, three further probe models, and every generation strategy.

Jane: But they also showed that breadth alone doesn't fix the budget equally under every concept because generating many kinds of samples is necessary where no kind covers another, as under instruction following.

Lu: To move the level a probe reaches without changing that ordering above, they suggest three further recipes: prompt detail, describing one split, or adversarial red-teaming.

Meng: Prompt detail lifts instruction by about zero point one four to zero point one eight for three generators and adds some lift to high-stakes and harmful probes too.

Tom: And generator choice doesn't transfer across concepts or budgets at all; pooling generators is additive on high-stakes, beating the best single generator on four out of four splits with a mean AUROC monotone in pool size from zero point eight three eight to zero point nine three nine.

Jane: It’s interesting how the strategies move the level up without changing that core ordering they established.

Title and authors: Lu: They found prompt detail lifts instruction by zero point one four to zero point one eight for three generators and high-stakes by about zero point zero three to zero point eleven, and harmful is at most a gain of two hundred twenty percent for Llama-three point three-70B-Instruct.

Meng: Generator choice doesn't transfer across concepts or budgets, but pooling generators is additive on high-stakes and inert on harmful, with all thirty cells within zero point zero four four.

Tom: So the final picture from this paper, "Coverage, Not Difficulty, Sets How Much Synthetic Data an Activation Probe Needs," is that a probe’s synthetic-data need is a property of what it monitors.

Jane: The data need is set by whether samples of one kind of conversation cover the others so breadth isn't the whole budget; how deep you generate depends on the concept.

Lu: This paper tells us how to measure that depth in advance, to within a factor of two or three, with no knowledge of the deployment traffic.

Meng: The ordering holds on real samples, which is a big piece of reassurance for anyone trying to build these systems.

Tom: So we’ve seen how this paper suggests using the transfer AUROC T and coverage metrics to decide exactly how deep we need to generate our synthetic data for different monitoring targets.

Jane: It really boils down to understanding what each probe is actually trying to catch in the first place, rather than just guessing based on difficulty alone.

Lu: This work sets a foundation for how practitioners can write down the kinds of conversation a concept covers and measure that transfer capability directly.

Meng: For us engineers, this means we can design targeted data generation strategies based on which concepts we are trying to monitor and how they relate to each other.

Tom: It’s a really concrete way to decide your budget, moving beyond just throwing more kinds of data at the wall hoping it pays off.

Jane: This paper is a great resource for anyone working on building robust AI monitors because it gives us a clear mechanism for setting those data needs.

The paper's summary: Tom: So, we're looking at this paper now: "Coverage, Not Difficulty, Sets How Much Synthetic Data an Activation Probe Needs." Basically, it’s about figuring out exactly how much synthetic conversation data you need for a language model probe to actually work.

Jane: Right. The main idea is that the amount of data you generate isn't just about making sure the conversations are hard enough to test the model, but whether those conversations cover all the different kinds of things you want to monitor.

Lu: They found that concept and distribution account for a big chunk, forty-two to forty-five percent of how much variance there is in data needs. So it’s not just one factor; it’s the relationship between what you’re monitoring and the data you feed it.

Meng: That means if you only focus on making the hardest instruction probes, you might be missing coverage for other important areas, like high-stakes scenarios or harmful replies.

Tom: Exactly. They set a median half-gain size of seven to eleven samples for any single kind of conversation under all three concepts together. But that number isn't fixed; it changes based on what you’re trying to achieve with the probe.

Jane: The crucial part is that the ordering holds when you compare different concepts, but the actual budget needed shifts depending on which concept you're looking at. It’s about saturation of a distribution, not just pushing difficulty up universally.

Lu: They show that coverage accounts for nearly eighty percent of the gap in half-gain size between instruction probes and the other two concepts when they look at generated data, and sixty-six percent on real samples. That really shows where the main leverage is.

Meng: So, if you’re building a system to catch harmful replies, focusing on generating kinds of conversations that cover all the different harmful reply patterns will be more effective than just generating a massive amount of one specific kind.

Tom: It’s about having enough samples of each kind in your distribution to saturate it, which is what sets the value for that half-gain size they measured. It's a way to predict your synthetic data need before you even start training.

Jane: This gives practitioners a practical way to write down what kinds of conversation you’re covering and measure how much transfer one kind has to the others.

Lu: They introduce a rule, R hat equals one plus two times K minus one times one minus T, where T is that transfer AUROC. It’s an operator spoken out, so it just lets you calculate the required budget based on how well samples of one kind cover others.

Meng: That’s useful for engineering because it gives us a metric to decide exactly how deep to generate our data based on the concept we care about.

Tom: So, this isn't just about generating more; it's about generating the right kinds of samples at the right depth. But they also show that simply generating many kinds doesn't fix the budget equally across all monitoring targets; you still need to focus where one kind doesn’t cover another.

Jane: It suggests that breadth alone isn't a magic fix for every problem, and the depth you need really depends on which concept you’re aiming for. This paper gives us a way to measure that depth in advance, without needing any knowledge of the actual deployment traffic.

Lu: And they show how prompt detail or adversarial red-teaming can lift instruction probes by about zero point one four to zero point one eight for some generators, and high-stakes probes by zero point zero three to zero point eleven.

Meng: So we can use those recipes to move the level up, but it’s still fundamentally driven by that coverage principle they established. It’s about understanding what each probe is actually trying to catch in the first place, not just guessing based on how hard it is.

Tom: This work sets a foundation for how practitioners can design targeted data generation strategies based on which concepts we are monitoring and how they relate to each other. It’s a concrete way to decide your budget.

The paper's improvements: Tom: So, we’ve seen how coverage sets the budget based on what you monitor, and now we’re looking at how to actually get those probes to perform better with less effort or in different scenarios.

Jane: Right. The paper doesn't just say "do more"; it gives us a few ways to move the level of performance up without necessarily changing that core ordering they established for concepts.

Lu: They have three further recipes they suggest, including prompt detail, describing one split, or adversarial red-teaming. These are ways to lift the probe’s actual AUROC score.

Meng: Prompt detail lifts instruction by about zero point one four to zero point one eight for three generators, and it also adds some lift to high-stakes and harmful probes by about zero point zero three to zero point eleven.

Tom: So we’re getting a small performance boost from just making the prompt more descriptive, which is interesting because it shows that detail still matters even when we’re thinking about coverage saturation.

Jane: And generator choice doesn't transfer across concepts or budgets at all, so picking a better generator isn't a universal fix for all monitoring targets.

Lu: But pooling generators is additive on high-stakes, meaning the four-way pool beats the best single generator on four out of four splits with a mean AUROC monotone in pool size from zero point eight three eight to zero point nine three nine.

Meng: So for an engineer, that means if you’re dealing with high-stakes scenarios, pooling generators is a solid way to boost performance without changing the fundamental data need calculation.

Tom: And then there’s prompt detail again, but for harmful probes, it lifts by at most zero point two two percent on Llama-three point three-70B-Instruct. It shows that different strategies have different impacts depending on what you are trying to achieve.

Jane: It’s important to remember that while these recipes move the performance level up, they don't change the fundamental ordering of concepts they found earlier in the paper.

Lu: The main point is that these techniques are ways to get a better result, but the underlying mechanism—coverage setting the need—stays true across all these improvements.

Meng: So, we can use these recipes for fine-tuning our probes to get more signal out of our data budget, while still respecting that coverage principle.

Tom: Exactly. It gives us tools to refine the generation process when we already have a good idea of what data volume we need based on coverage. This paper shows how to balance the depth with the strategy for getting a better result.

Conclusion: Tom: So we’ve covered how coverage dictates the synthetic data budget for an activation probe in "Coverage, Not Difficulty, Sets How Much Synthetic Data an Activation Probe Needs." Basically, it tells us that you need to generate enough samples of each kind to saturate its distribution.

Jane: That means the number of samples you need isn't just about making things hard; it’s about whether those conversations cover everything you want to monitor.

Lu: The implication is that practitioners can use a transfer AUROC T metric, R hat equals one plus two times K minus one times one minus T, to predict their budget without knowing the deployment traffic.

Meng: So for us on the engineering side, it’s a way to decide how deep to generate data based on the concept we care about before we spend massive compute cycles. It grounds that abstract coverage idea in a concrete number.

Lalam: From Lalam’s side, this means that if we focus our training data generation on covering all the necessary conversational kinds, the resulting AI will be more robust across those different monitoring targets. That improves how culture and safety guidelines are reflected in the model's behavior.

Tom: It really shifts our thinking from just brute-forcing data volume to being strategic about what kind of coverage we need for each specific monitoring goal.

Jane: Exactly. The paper shows that breadth alone doesn't fix the budget equally, and you need to generate more where one kind doesn’t cover another concept, especially with instruction following probes.

Lu: It’s a framework for designing targeted data generation strategies based on which concepts we are monitoring and how they relate to each other in terms of coverage.

Meng: We can use this to build smarter generation pipelines that actually respect the data budget rather than just throwing more kinds of samples at the wall hoping it pays off.

Lalam: This work helps us build a more reliable AI because it gives us a clear mechanism for setting those data needs based on what we’re trying to catch in the first place.

Tom: So, that’s our wrap-up on "Coverage, Not Difficulty, Sets How Much Synthetic Data an Activation Probe Needs." It’s a practical tool for anyone building these monitoring systems.

Jane: It gives us a way to measure depth in advance without needing any knowledge of the actual deployment traffic.

Lu: The ordering holds on real samples and across all those different strategies they tested, which is a good piece of reassurance for anyone trying to build these systems.

Meng: We can use this coverage principle to design generation pipelines that respect the data budget more intelligently.

Lalam: This paper sets a foundation for how we can build more reliable AI by focusing on what’s actually important for monitoring.

Ankush Checkervarty

cs.LG

Submitted: 2026-10-06

Updated: 2026-10-06

Comments: 24 pages, 8 figures, 11 tables. Code and data: https://github.com/Ankush7890/sythentic_data_needs

Code: https://github.com/blandfort/tuberlens

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 90/100

The gist: The gist The need is set by what is monitored: coverage, not per-kind difficulty; the number of samples of its own kind a distribution needs to saturate How it works Activation probes that monitor

Key concepts

Coverage vs. Per-Kind Difficulty: This is the core principle: data needs are set by how well samples from one conversation type cover all others, not by how hard each specific type is to learn. Coverage measures the relationship between different conversation categories, indicating if a sample of one kind helps generalize to another.
Half-Gain Size: This refers to the median number of own-kind synthetic samples required for a probe model to achieve half its maximum performance on a specific task. This size varies significantly depending on the monitored concept—high-stakes, harmful replies, or instruction following—showing that data needs are concept-dependent.
Transfer AUROC (T): This metric ranks different concepts based on how well samples from one kind transfer to others. It acts as a multiplier indicating how much a probe's performance is boosted by landing in a specific distribution, helping practitioners understand the relative value of different data needs.

Summary

The gist The need is set by what is monitored: coverage, not per-kind difficulty; the number of samples of its own kind a distribution needs to saturate

How it works

Activation probes that monitor deployed language models are trained on synthetic conversations, and how many a probe needs is open The need is set by what is monitored: coverage, not per-kind difficulty; the number of samples of its own kind a distribution needs to saturate Concept and distribution account for 42–45% of its variance, the generator, probe model, and prompt detail for under 10% Every kind, one per evaluation distribution, has a median half-gain size of 7–11 own-kind synthetic samples under all three concepts alike What sets the value of the half-gain size is coverage, not per-kind difficulty: the number of samples of its own kind a distribution needs to saturate

Concepts and Data Need

The study measured learning curves over three monitoring targets—high-stakes situations, replies harmful to a person, and replies that do not follow the user’s instruction—on fourteen held-out evaluation distributions Coverage, not per-kind difficulty, sets the data need The ordering holds on three smaller probe models and on real samples (from dev set) Under detailed prompts, cutting the training set from 590 to 80 samples costs high-stakes and harmful probes a few hundredths of AUROC and instruction probes up to 0.21 on Gemma-3-27B-IT The median half-gain size is 22 samples for high-stakes and for harmful against 164 for instruction

Coverage Mechanism

The mechanism that sets the value of the half-gain size is coverage, not per-kind difficulty: the number of samples of its own kind a distribution needs to saturate Coverage accounts for 79% of the gap in half-gain size between instruction and the other two concepts on generated and 66% on real samples The transfer AUROC T does not pin a distribution’s absolute half-gain size, but it ranks concepts, and read as a multiplier between one and the number of kinds it lands within a factor of two of the measured need on most splits, generated or real The rule Rˆ = 1 + 2(K − 1)(1 − T), has no fitted parameter

Practical Guidance

A practitioner can write down the kinds of conversation a concept covers, as the detailed prompt does (Appendix F), generate a kind-tagged set, and measure how far samples of one kind transfer to the others The ordering holds on real samples, four generators, two prompts, three further probe models, and every generation strategy of §5.3 Breadth alone does not fix the budget, however, and we find that it does not pay equally under every concept: generating many kinds of samples is necessary where no kind covers another, as under instruction

Other Data Generation Strategies

Three further recipes move the level a probe reaches but not the ordering above These include prompt detail, describing one split, or adversarial red-teaming Prompt detail lifts instruction by +0.14 to +0.18 for three generators (Nemotron-3-Ultra-550B +0.03), high-stakes by +0.03 to +0.11, and harmful by at most +0.03 (−0.22 for Llama-3.3-70B-Instruct) Generator choice does not transfer across concepts or budgets Pooling generators is additive on high-stakes (the four-way pool beats the best single generator on 4 of 4 splits; mean AUROC monotone in pool size, 0.838 to 0.939), inert on harmful (all 30 cells within 0.044), and dilutive on instruction

Conclusion

A probe’s synthetic-data need is a property of what it monitors The data need is set by whether samples of one kind of conversation cover the others, so breadth is not the whole budget: how deep to generate depends on the concept, and one generated set cut into kinds measures that in advance, to within a factor of two or three, with no knowledge of the deployment traffic The ordering holds on real samples

Other Data Generation Strategies

Three further recipes move the level a probe reaches but not the ordering above These include prompt detail, describing one split, or adversarial red-teaming Prompt detail lifts instruction by +0.14 to +0.18 for three generators (Nemotron-3-Ultra-550B +0.03), high-stakes by +0.03 to +0.11, and harmful by at most +0.22 for Llama-3.3-70B-Instruct Generator choice does not transfer across concepts or budgets Pooling generators is additive on high-stakes (the four-way pool beats the best single generator on 4 of 4 splits; mean AUROC monotone in pool size, 0.838 to 0.939), inert on harmful (all 30 cells within 0.

Improvements for AI systems

Bold header: Data Need Quantification for Concept-Specific Probes

This allows practitioners to determine exactly how many synthetic samples are needed by stating The need is set by what is monitored, moving beyond vague advice like breadth over depth. Specifically, it provides a mechanism for deciding the required budget: the half-gain size of a fitted curve, the number of samples at which half the gain is in hand.

Bold header: Concept-Specific Data Generation Strategy

AI systems can generate highly targeted datasets based on monitoring needs by using prompt detail to guide generation. For example, the detailed prompt inserts a paragraph that enumerates the kinds of conversation the probe will be evaluated on allowing for generating samples specifically for one split, as seen in F.2 DETAILED PROMPT.

Bold header: Predictive Budgeting via Transfer Metrics

Practitioners can predict how many samples are needed by measuring transfer capability rather than just absolute gain. The paper suggests using the transfer AUROC T as a multiplier, noting that Rˆ = 1 + 2 (K − 1) (1 − T), which allows one to calculate the required budget based on how well samples of one kind cover others.

Abstract

Activation probes that monitor deployed language models are trained on synthetic conversations, and how many a probe needs is open. We trace learning curves over 10-590 synthetic samples for three monitoring concepts, high-stakes situations, replies harmful to a person, and replies that do not follow the user's instruction, on fourteen held-out evaluation distributions and four probe models, varying the generator LLM and the prompt's detail. The need is set by what is monitored: probes for high-stakes and harmful are within a few hundredths of their plateau from 80 samples on Gemma-3-27B-IT, instruction probes need several times as many, and the ordering holds on three smaller probe models and on real samples (from dev set). Prior work advises spending a generation budget on breadth, more kinds of data, over depth, more of each kind. We read the depth a concept needs as the half-gain size of a fitted curve, the number of samples at which half the gain is in hand. Concept and distribution account for 42-45% of its variance, the generator, probe model, and prompt detail for under 10%. What sets the value of the half-gain size is coverage, not per-kind difficulty: the number of samples of its own kind a distribution needs to saturate. Every kind, one per evaluation distribution, has a median half-gain size of 7-11 own-kind synthetic samples under all three concepts alike. What differs is how far samples of one kind transfer to the concept's other kinds, almost fully under high-stakes, less under harmful, and least under instruction, which accounts for most of the gap between concepts on generated and real samples. Breadth therefore pays differently by concept: many kinds are necessary under instruction, where no kind covers another, and nearly redundant under high-stakes, where one kind covers the rest. We release the evaluation suites, dev sets, and generated sets.