AIDG: A Formal Decomposition of Information Extraction and Containment Asymmetries in Multi-Turn LLM Dialogue

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "AIDG: A Formal Decomposition of Information Extraction and Containment Asymmetries in Multi-Turn LLM Dialogue".

Jane: Here's the gist of the paper. AIDG formalizes multi-turn adversarial dialogue as a two-player partially observable stochastic game to decompose performance into Seeker (extraction) and Holder (containment) roles, revealing that the containment-over-extraction gap is driven almost entirely by variance in offensive reasoning rather than defensive capacity Formalization of Dialogue as a POSG AIDG formalizes multi-turn adversarial dialogue as a two-player partially observable stochastic game (POSG) between a Seeker A and a Holder B over a private variable S The state space s ∈ S decomposes as s = (S, Ht), where S ∈ omega is the private variable drawn from a task-specific universe omega and Ht = (a(1)A, a(1)B,..., a(t)A, a(t)B) is the full dialogue history up to turn t The defining property of G is that the initial state is unobservable to A but fully observed by B This asymmetry is the source of the directional baseline gap: the Holder’s decision is locally resolvable, while the Seeker’s requires belief propagation across turns Decomposed Measurement and ELO System The paper introduces a Dual-ELO rating system derived from the Bradley–Terry model to independently quantify offensive (C ELO) and defensive (V ELO) capabilities For a Seeker with rating C against a Holder with rating V, the formula is EC = 1 / 1 + 10(V −C)/400, EV = 1−EC Updates apply efficiency weighting: R′C = RC + K · M(t) · (SC − EC), with K = 24 and M(t) is the multiplier of Section 3.4 The Capability Gap for a model is ∆ = V ELO − C ELO, which is the scalar reported for asymmetry per model Diagnostic Findings on Performance Across 439 games over six frontier LLMs, several diagnostic findings were observed Defense advantage is tightly clustered (σ = 1.9 ELO) while offensive performance varies substantially (σ = 53.3 ELO); confirmation framing increases extraction odds 7.75× over uninformed deduction (p < 0.00001); and constraint violations account for 41.3% of deductive failures, uncorrelated with scale (ρ = 0.0) The gap is driven almost entirely by variance in offensive reasoning rather than defensive capacity The mean defensive advantage is 255.6 ELO, with Cohen’s d = 4.07 The standard deviation of V ELO is σV = 1.9 across the six models, while C ELO varies by σC = 53.3 This highlights that defensive capability is essentially uniform across architectures spanning 8B to 671B parameters The confirmation effect is observed when Mode A yields a 7.75× extraction-odds increase over Mode B Constraint–reasoning interference accounts for 41.3% of games terminating due to Seeker constraint violation in AIDG-II The disqualification rate is strongly inversely correlated with win rate (r = −0.95) The model profile separates by ∆ = V ELO − C ELO, with Granite-3.3-8B showing the lowest extraction profile The Seeker’s strategy in Mode A is described as presuppositional repetition, not interrogation The Mode B success rate is 3.5% This failure mode occurs because AIDG-I secrets are propositions, not nouns, leading to a structural blind spot for noun-centric narrowing strategies The Holder’s disclosures are substantive but never propositional This double miss is precisely the asymmetry that makes containment locally easy The linear turn-decay multiplier M(t) is derived from assumptions (A1) and (A2) in Section 3.

Title and authors: Tom: So we're looking at this paper, "AIDG: A Formal Decomposition of Information Extraction and Containment Asymmetries in Multi-Turn LLM Dialogue."

Jane: It sounds really technical right off the bat.

Tom: Right, it formalizes multi-turn adversarial dialogue as a two-player partially observable stochastic game.

Jane: That means they're not just looking at a single win rate number, but trying to break down what happens in every turn.

Lu: Exactly. They define it as a two-player partially observable stochastic game between a Seeker and a Holder over some private variable S.

Tom: And they decompose the performance into two distinct roles: the Seeker who's trying to extract information, and the Holder who's trying to contain it.

Meng: So, if we look at the results, what’s actually surprising?

Jane: Well, they found something pretty counterintuitive about how those two roles perform.

Tom: They show that the containment-over-extraction gap isn't really a problem with how strong the defense is.

Lu: That's it. The containment-over-extraction gap is driven almost entirely by variance in offensive reasoning rather than defensive capacity >

Meng: So, what does that mean for us when we're building these models?

Jane: It suggests that making the defense stronger isn't the main lever here.

Tom: They use a Dual-ELO rating system to separate those capabilities, quantifying them as C ELO and V ELO.

Lu: The paper introduces this Dual-ELO system derived from the Bradley–Terry model to independently quantify offensive and defensive capabilities for every model >

Jane: And they found that the defensive performance is actually quite consistent across the six frontier LLMs they tested, with a standard deviation of one point nine ELO.

Tom: That’s interesting because we usually expect bigger swings in defense when you jump from an eight billion parameter model to a six hundred seventy-one billion parameter one.

Meng: The paper says the mean defensive advantage is two hundred fifty-five point six ELO, with a coefficient of determination of four point zero seven.

Jane: So, while the defense is stable, the Seeker's offensive performance varies wildly with a standard deviation of fifty-three point three ELO.

Lu: That variance in C ELO is what they call Delta, which is that scalar reported for asymmetry per model >

Tom: And they found some specific things about where these models fail.

Jane: They noted that confirmation framing increases the extraction odds seven point seven five times over just using uninformed deduction.

Tom: That’s a big jump, and it’s statistically significant at p less than zero point zero zero zero zero one >

Meng: But they also pointed out that constraint violations account for forty-one point three percent of deductive failures.

Jane: What's interesting about that is the paper found this failure rate is not correlated with the model scale at all, with a rho of zero >

Title and authors: Lu: They also found some specific strategies work better than others.

Tom: Confirmation framing works seven point seven five times better when comparing Mode A to Mode B conditions >

Jane: And they discovered that constraint–reasoning interference accounts for forty-one point three percent of games terminating due to Seeker constraint violation in the AIDG-II setup >

Meng: The disqualification rate is strongly inversely correlated with the win rate, with a correlation of negative zero point nine five >

Lu: They also identified that the model profile separates based on Delta, with Granite-three point three-8B showing the lowest extraction profile >

Tom: And they described how the Seeker’s strategy in Mode A isn't really interrogation, but presuppositional repetition >

Jane: That makes sense if you consider what they found about the Holder’s disclosures being substantive but never propositional >

Meng: The paper suggests that because AIDG-I secrets are propositions rather than nouns, there's a structural blind spot for noun-centric narrowing strategies >

Lu: This double miss, the Seeker not probing the state component of the secret, is precisely that asymmetry that makes containment locally easy >

Tom: So, what’s the real implication here?

Jane: It means we shouldn't just focus on brute-force defensive tuning.

Tom: We need to look at how we structure our prompting and reasoning to target the variance in offensive reasoning >

Jane: The core idea is that because the gap is driven by variance in offensive reasoning, improving defense uniformly won't solve it entirely >

Meng: From a production standpoint, this suggests that we should be paying attention to how models handle the sequencing of dialogue and how they manage those constraint violations >

Lu: And they suggest improvements focusing on diagnostics.

Tom: They propose building an automated diagnostic tool to measure constraint violation rates uncorrelated with scale by tracking the correlation coefficient rho between parameter count and disqualification rate >

Jane: That’s a practical step. It helps isolate whether model size or training objectives are driving constraint adherence, since they found that the Spearman rho is zero point zero >

Meng: And they also suggested creating a confirmation framing analysis pipeline to calculate the odds ratio increase under Mode A versus Mode B conditions for any given LLM and secret type >

Tom: That would help us see which models benefit most from presuppositional framing, which is what they found increases extraction odds by seven point seven five times >

Lu: And they also suggested an engineering module to monitor the timing of Seeker disqualifications relative to conversational history accumulation, detecting when the capacity allocated to entropy-reducing planning competes with capacity allocated to enforcing a negative syntactic constraint >

Jane: That’s where we see how you can actually measure that interplay, tracking when those two capacities clash >

Title and authors: Tom: It’s interesting because they also built an automated hypothesis-space traversal evaluator specifically for propositional secrets, measuring the Seeker's ability to probe object attributes versus just identifying the object itself >

Meng: That’s a big win if we can build that, because it diagnoses those failures where the questioning trajectory converges on the object class but never probes the state component >

Lu: Overall, AIDG provides a formal way to decompose performance into Seeker and Holder roles, showing that containment is locally resolvable while extraction requires belief propagation across turns >

Jane: So, what does this mean for the future of multi-turn dialogue evaluation?

Tom: It shifts the focus from just looking at a final score to understanding the underlying reasoning mechanisms and where the variance comes from >

Meng: It means we need better ways to stress-test models on how they handle sequential dependencies and how they allocate their internal resources between planning and constraint checking >

Lu: The paper suggests that future work should focus heavily on these diagnostic tools, especially those that can measure the efficiency multiplier M(t) which is derived from assumptions in Section three point four > <ref:2602.17443#pg3,the efficiency multiplier M(t>

Tom: It’s a lot of nuance here, but the main point is that we have a much clearer picture of what drives the performance gap between extraction and containment >

Jane: We're looking at how variance in offensive reasoning dictates whether containment wins or extraction wins in these complex multi-turn scenarios >

Lu: So, AIDG gives us a way to formalize this asymmetry, and the next step is using their proposed diagnostic framework to find out exactly where that variance is coming from >

Meng: I think for deployment, we need to know if those constraint violations are just noise or if they point to a real scaling issue we need to fix >

Tom: Exactly. If those violations are uncorrelated with scale, then maybe the model architecture itself needs adjustment rather than just more training data >

Jane: It really gives us a concrete direction for how to build better evaluation systems that look beneath the surface win rate numbers >

Lu: So, to wrap up on AIDG: it formalizes dialogue as a POSG, separates roles into Seeker and Holder, and shows the containment-over-extraction gap is driven almost entirely by variance in offensive reasoning rather than defensive capacity.

Meng: It’s a good map for understanding where we need to spend our engineering time >

Tom: Yeah, it points us toward those diagnostic modules they proposed, which help us see the mechanics behind the performance numbers >

Jane: We can start thinking about how to build systems that measure those offensive and defensive capabilities independently going forward >

The paper's summary: Tom: So, this paper, AIDG: A Formal Decomposition of Information Extraction and Containment Asymmetries in Multi-Turn LLM Dialogue, it boils down to this idea: they’re treating multi-turn chat between two AI agents like a game where one is a Seeker trying to pull secrets out, and the other is a Holder trying to keep them locked down.

Jane: Right, so they aren't just looking at how well the AI answers, but how it plays those two distinct roles over several turns.

Tom: Exactly. The big finding they want us to see is that the gap between what’s being extracted and what’s being contained isn't really about the defense being fundamentally better.

Lu: They use a Dual-ELO system to separate the offensive capability, which they call C-ELO, from the defensive capability, V-ELO. This lets them quantify those two things independently for every model.

Meng: So if you look at the results on that separation, they found that while the defense level is pretty consistent across these models—a standard deviation of one point nine ELO—the offensive performance varies a lot more, with a standard deviation of fifty-three point three ELO.

Jane: That’s what I was wondering. We usually expect the bigger models to have much stronger defenses, but here the defense seems pretty uniform regardless of how big the AI is, from eight billion parameters up to six hundred and seventy-one billion.

Tom: That’s the point they were making with that measurement. The containment-over-extraction gap isn't driven by a lack of defensive capacity in general.

Lu: No, it’s driven almost entirely by the variance in how much offensive reasoning those models have on any given turn. That variation is what causes the gap to open up and close over time.

Meng: So if you want to improve extraction performance, tuning the Holder’s strategy might not be the best bet; you should be looking at making sure the Seeker’s internal planning doesn't swing wildly from one turn to the next.

Jane: It means we need to stop just focusing on making a model "safe" and start looking at how it handles that back-and-forth reasoning process itself.

Tom: The paper also showed some specific failure modes, like confirmation framing increasing extraction odds by seven point seven five times over just using regular deduction. That’s a big number, and it shows how sensitive the Seeker is to the way it's prompted.

Lu: And they found that constraint violations—where the Seeker breaks the rules of dialogue—account for forty-one point three percent of deductive failures, and this rate didn't change much whether you looked at a small model or a huge one.

Meng: That lack of correlation between model size and those constraint failures is telling. It suggests that what’s causing the Seeker to get stuck with rules isn't just the sheer capacity of the AI; it’s something in how it manages those rules internally.

Jane: So, what this means for us as people who are actually using these models? It suggests that getting better at dialogue isn't just about giving a model more context; it’s about controlling the specific type of reasoning variance that causes the Seeker to fail or succeed.

Tom: Right. And they suggest some concrete ways to test this, proposing tools to measure exactly how much capacity is spent on planning versus enforcing those rules at any given moment in the conversation.

Lu: They’re building these diagnostic modules, which should help us see the mechanics behind why some models are great at containing secrets and others aren't.

Jane: So we’re moving from just looking at a final score to understanding the underlying reasoning mechanisms that cause those scores to fluctuate so much.

Tom: Yeah, and that variance in offensive reasoning is what dictates whether containment wins or extraction wins in these complex multi-turn scenarios. That’s where the real work is.

The paper's improvements: Tom: So, we've seen how AIDG breaks dialogue down into Seeker and Holder roles, and now we’re looking at what they suggest is needed to actually fix those performance gaps in the future.

Jane: Right, so they aren't just stopping at describing the problem; they’re proposing specific ways to build better systems based on their analysis.

Lu: They are pushing for a Dual-ELO system, which we talked about before, but now it’s being framed as a core tool for independently quantifying those offensive and defensive capabilities across different AI architectures.

Meng: From an engineering standpoint, that would be useful because if you can measure C-ELO and V-ELO separately, you can stop just looking at one aggregated score and start seeing exactly which component is lagging.

Tom: Yeah, it makes the math concrete for us engineers. They are also suggesting an automated diagnostic tool to track constraint violations based on parameter count to see if size actually matters for those failures.

Jane: That’s smart because they found that correlation was zero, meaning model size isn't the main driver of those rule-breaking errors, so we need a different way to look at it.

Lu: They also want a confirmation framing analysis pipeline, which would let us test how much better extraction gets when we use specific prompting techniques like presuppositional repetition. That’s based on their finding that confirmation framing boosts odds by seven point seven five times.

Meng: If we can automate that testing, it helps us figure out which specific prompt engineering tweaks actually pay off for a given model profile, instead of just guessing.

Tom: And then there’s this module to detect constraint-reasoning interference, which is essentially trying to monitor when the AI gets distracted by planning versus when it’s focused on enforcing those negative syntactic constraints.

Jane: That sounds like a way to measure the internal competition for resources inside the AI's reasoning process.

Lu: And finally, they propose an automated hypothesis-space traversal evaluator specifically for propositional secrets, which aims to diagnose if the Seeker is just identifying an object instead of actually probing its specific state details.

Meng: That would be a huge help in deployment because it tells us precisely where the Seeker’s questioning trajectory is missing something crucial, not just missing information.

Tom: So, they're moving toward a toolkit that helps us diagnose *why* one model beats another at extraction and containment by looking at those internal reasoning mechanics.

Jane: It really shifts the focus from simply training bigger models to understanding how to structure their dialogue interactions for maximum impact.

Lu: The implication is that future research needs to focus heavily on these diagnostic tools, especially figuring out how that efficiency multiplier M of t actually behaves across different types of information being exchanged.

Conclusion: Tom: So we've covered AIDG: A Formal Decomposition of Information Extraction and Containment Asymmetries in Multi-Turn LLM Dialogue, and to recap, they found that the gap between what’s extracted and what’s contained really comes down to variance in offensive reasoning rather than just having a stronger defense.

Jane: Right, so for someone listening who doesn't read this paper, it means we shouldn't just try to make the AI "safe" because that isn't the main thing driving performance differences between models.

Lu: They’re showing that while the defensive part is pretty consistent across different AI architectures, the offensive reasoning capability varies wildly. That variance is what causes all those performance swings we see in practice.

Meng: From a deployment view, it means we need to stop focusing solely on defensive tuning and start looking at how robust the Seeker’s internal planning is under pressure. If that planning is inconsistent, the whole system breaks down during complex interactions.

Tom: Exactly. The paper points us toward building tools that can measure this reasoning variance directly instead of just relying on final win rates from benchmarks.

Jane: It changes how we evaluate these models because it tells us *why* one model might be better at extracting information than another, rather than just saying one score is higher.

Lu: The core idea they’re pushing is that we need to formalize the way these two roles interact so we can pinpoint exactly where the failure happens.

Meng: And I’m hoping for tools that help us identify those constraint violations you mentioned earlier, because if we can track when planning clashes with constraint enforcement, it tells us where to invest our engineering time.

Tom: It really is a shift in focus toward the mechanics of dialogue itself rather than just the final output.

Jane: So it’s less about brute force training and more about understanding the logic flow within those multi-turn exchanges.

Lu: The next step, which they emphasize, is building those diagnostic frameworks to measure that internal efficiency multiplier M of t more precisely.