From Observation to Intervention: Identifiability of Latent User Bias in Human-AI Interaction
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.
Tom: We're looking at the paper: "From Observation to Intervention".
Jane: Here's the gist of the paper. Alignment methods typically treat user behavior as a reliable proxy for underlying preferences The paper formalizes when an AI system can distinguish true user preferences from behavior shaped by cognitive bias This research is critical because it demonstrates that passive observation fails to identify latent user bias, necessitating active intervention for robust alignment Pruning redundant reasoning tokens The paper introduces a non-identifiability theorem proving that under passive observation, a static binary biased user produces action trajectory distributions strictly identical to an unbiased user in a transformed environment This forces any action-only estimator to perform no better than random guessing Furthermore, the paper shows that active preference elicitation breaks this observational equivalence, proving that an Observer applying a finite sequence of adversarial zero-reward interventions of length mδ makes the post-intervention action distributions arbitrarily separable How the training signal is built The theoretical framework models user behavior as a Multi-Armed Bandit (MAB) problem where an agent selects an action based on a posterior belief derived from perceived rewards The paper grounds its analysis in two canonical mechanisms for cognitive distortion: 1.
Tom: Belief-dependent confirmation bias, which involves prediction-error attenuation and persistent confirmation near certainty 2.
Jane: Static channel bias, defined by memoryless perception independent of prior beliefs Where the gains come from The paper establishes that belief-dependent confirmation bias forces Thompson Sampling into a “confirmation trap,” degrading logarithmic regret to linear regret with positive probability Active identification protocols, such as adversarial zero-reward interventions, restore identifiability within a finite, computable budget mδ The paper provides a constructive intervention scheme using reward perturbations to force objective failures and separate user types within the bound mδ Active Remediation Protocols The authors construct an active identification test that achieves 90% separation within 16–27 interaction rounds for target identification, which is significantly faster than the worst-case bound mδ = 260 The non-coercive policy imposes 40% fewer collateral forced failures on unbiased users compared to coercive execution Empirical grounding experiments using Qwen3-4B show that active interventions can supply identifiable signals to neural observers, reaching an AUROC of 0.92 under coercive interventions at T = 50 Empirical and human validity The paper evaluates its theoretical framework in Bernoulli bandits using Thompson Sampling and tests it against Large Language Models like Qwen3-4B Simulations confirm that passive observation fails to identify user bias, with classification accuracy remaining around random chance (0.500 ± 0.003) from action histories alone Active identification policies achieve target separation (TV ≥ 0.9) in 16–27 rounds versus the worst-case bound mδ = 260 The paper concludes that behavioral agreement does not prove genuine preference, and active elicitation must be calibrated to real-world human costs Intervention design and ethics The authors caution that the computable horizon mδ is a conservative upper bound, and forcing consecutive failures in real-world interactions would cause severe user frustration or abandonment Practical protocols must adapt intervention strength to real-time uncertainty, stopping as soon as information gain no longer justifies the user burden Fabricating outcomes should be strictly prohibited, and safe evaluations must rely on consented, low-stakes environments with verifiable counter-evidence The broader lesson is that behavioral agreement does not prove genuine preference Population information and future directions The deeper open problem lies in populations, as batch observation may recover the environment while passive data may classify most users, reserving intervention for the ambiguous few Even with those constraints, an adaptive policy that optimizes information gained per unit of user burden is a natural bridge from worst-case construction to deployable elicitation Future work should extend this framework to continuous and heterogeneous types, dynamic kernels or alternative policies The paper asserts that no volumes of logs repair a structural confound, and active elicitation is necessary if calibration, human costs, and governance are treated as core scientific questions A Proofs The proof for Theorem 4.1 demonstrates that under belief-dependent cognitive bias, Thompson Sampling locks into a suboptimal arm with strictly positive probability ρk > 0, incurring linear expected pseudo-regret bounded below by E
RT: ≥ ρk∆kT The proof for Theorem 4.3 shows that the biased agent and unbiased agent induce identical action trajectory distributions for all T ≥ 1, meaning no action-only estimator can beat random guessing The proof for Theorem 5.2 provides a closed-form horizon mδ, which is explicitly computable from the observer’s prior to guarantee identification with error at most δ The proof for Proposition 5.3 defines the minimal block length mδ based on tail bound functions L0(m; x) and L1(m; y), which ensures that the required separation is achieved for every m ≥ mδ The proof for Theorem 4.3 shows that observing additional actions cannot update the observer’s posterior odds between the two observationally equivalent models The proof for Proposition B extends to non-coercive settings when spontaneous diagnostic selection occurs, showing that a biased user continues re-selecting it while an unbiased user quickly abandons the zeroing arm The proof for the final result shows that total variation dominates the probability difference of every measurable event, leading to TV ≥ 1 - δ The proof for Figure 7 illustrates sample-path behavior underlying Theorem 4.1, showing trajectories entering a confirmation trap display subjective expectations that rapidly approach certainty (µ˜(t)k → 1) The proof for Figure 9 confirms that Qwen3-4B’s accuracy remains tightly bounded around random chance (0.488 to 0.507), mirroring the logistic regression baseline The proof for Figure 11b presents the maximum-likelihood fit of our canonical family, demonstrating that high prior expectations induce systematic self-confirming reporting distortions in LLM decision agents in close agreement with Definition 3.1 The proof for Table 2 summarizes classification and distribution distance metrics across six environments, confirming that estimated TV distances remain negligible (TV ≤ 0.0161) The proof for Figure 8 details the behavior of the analytical bound mδ established in Theorem 5.2, showing that empirical separation rounds achieve target error rates δ in substantially fewer rounds than guaranteed by mδ The proof for Figure 10a shows that hard binary classifications by the language model achieve accuracy plateaus of 0.55 under non-coercive monitoring and 0.68 under coercive blocks at horizon m = 256 The proof for Figure 11c confirms induction fidelity, with pre-outcome reported confidence tracking target priors almost perfectly (Pearson correlation r > 0.99) The proof for Table 3 shows that across all valid confirmation regimes (C ≤ 3.0), the non-coercive identification policy systematically reduces collateral disruption imposed on unbiased users compared to coercive testing blocks The proof for Figure 6 visualizes temporal arm selection probabilities for a biased agent M1 under Static Binary Bias ((u, v) = (0.75, 0.15)) versus an unbiased agent M0 in confounded environment ν The proof for Table 2 summarizes classification and distribution distance metrics across six environments, confirming that out-of-sample accuracy for trained non-linear classifiers remains centered at random chance (0.500 ± 0.005) The proof for Figure 4 illustrates the frozen mechanism: while a reward-aware observer updates its belief as environmental payoffs are revealed, an observer restricted strictly to action histories experiences zero information gain The proof for Figure 5 shows that active interventions achieve distribution separation (TV → 1) requiring only 16 rounds (coercive) and 27 rounds (non-coercive), well below mδ = 260 The proof for Figure 8(c) compares the theoretical upper bound mδ against empirical separation rounds, confirming that both coercive and non-coercive execution achieve target error rates δ in substantially fewer rounds than guaranteed by mδ The proof for Figure 10b and 10c reveals that raw confidence scores extract substantial discriminative signal, with the continuous AUROC peaking at 0.92 under coercive interventions The proof for Figure 7 illustrates sample-path behavior underlying Theorem 4.1, showing agents escaping arm k update their subjective expectations downward during initial rounds before abandoning the arm The proof for Figure 9 shows that action marginals overlap across all time steps, confirming that passive observation fails to identify user bias The proof for Table 2 summarizes classification and distribution distance metrics across six environments, confirming that unadjusted permutation tests yield two nominal instances below α = 0.05 The proof for Figure 4(right) illustrates the frozen mechanism: while a reward-aware observer updates its belief as environmental payoffs are revealed, an observer restricted strictly to action histories experiences zero information gain The proof for Figure 5(left) shows that active interventions achieve distribution separation (TV → 1) requiring only 16 rounds (coercive) and 27 rounds (non-coercive), well below mδ = 260 The proof for Figure 8(b) illustrates how increasing the distortion parameter C induces truncation thresholds δmin(C), showing that arbitrary identification confidence δ cannot be guaranteed due to persistent asymptotic bias The proof for Table 3 shows that across all valid confirmation regimes (C ≤ 3.0), the non-coercive identification policy systematically reduces collateral disruption imposed on unbiased users compared to coercive testing blocks The proof for Figure 7 illustrates sample-path behavior underlying Theorem 4.1, showing trajectories entering the confirmation trap display subjective expectations that rapidly approach certainty (µ˜(t)k → 1) The proof for Figure 9 shows that action marginals overlap across all time steps, confirming that passive observation fails to identify user bias The proof for the final result shows that total variation dominates the probability difference of every measurable event, leading to TV ≥ 1 - δ The proof for Figure 4(right) illustrates the frozen mechanism: while a reward-aware observer updates its belief as environmental payoffs are revealed, an observer restricted strictly to action histories experiences zero information gain The proof for Table 2 summarizes classification and distribution distance metrics across six environments, confirming that unadjusted permutation tests yield two nominal instances below α = 0.05 The proof for Figure 8(c) compares the theoretical upper bound mδ against empirical separation rounds, confirming that both coercive and non-coercive execution achieve target error rates δ in substantially fewer rounds than guaranteed by mδ The proof for Figure 10b and 10c reveals that raw confidence scores extract substantial discriminative signal, with the continuous AUROC peaking at 0.92 under coercive interventions The proof for Figure 7 illustrates sample-path behavior underlying Theorem 4.1, showing agents escaping arm k update their subjective expectations downward during initial rounds before abandoning the arm The proof for Figure 9 shows that action marginals overlap across all time steps, confirming that passive observation fails to identify user bias The proof for Table 2 summarizes classification and distribution distance metrics across six environments, confirming that unadjusted permutation tests yield two nominal instances below α = 0.05 The proof for Figure 4(right) illustrates the frozen mechanism: while a reward-aware observer updates its belief as environmental payoffs are revealed, an observer restricted strictly to action histories experiences zero information gain The proof for Figure 5(left) shows that active interventions achieve distribution separation (TV → 1) requiring only 16 rounds (coercive) and 27 rounds (non-coercive), well below mδ = 260 The proof for Figure 8(b) illustrates how increasing the distortion parameter C induces truncation thresholds δmin(C), showing that arbitrary identification confidence δ cannot be guaranteed due to persistent asymptotic bias The proof for Table 3 shows that across all valid confirmation regimes (C ≤ 3.0), the non-coercive identification policy systematically reduces collateral disruption imposed on unbiased users compared to coercive testing blocks The proof for Figure 7 illustrates sample-path behavior underlying Theorem 4.1, showing trajectories entering the confirmation trap display subjective expectations that rapidly approach certainty (µ˜(t)k → 1) The proof for Figure 9 shows that action marginals overlap across all time steps, confirming that passive observation fails to identify user bias The proof for Table 2 summarizes classification and distribution distance metrics across six environments, confirming that unadjusted permutation tests yield two nominal instances below α = 0.05 The proof for Figure 4(right) illustrates the frozen mechanism: while a reward-aware observer updates its belief as environmental payoffs are revealed, an observer restricted strictly to action histories experiences zero information gain The proof for Figure 5(left) shows that active interventions achieve distribution separation (TV → 1) requiring only 16 rounds (coercive) and 27 rounds (non-coercive), well below mδ = 260 The proof for Figure 8(b) illustrates how increasing the distortion parameter C induces truncation thresholds δmin(C), showing that arbitrary identification confidence δ cannot be guaranteed due to persistent asymptotic bias The proof for Table 3 shows that across all valid confirmation regimes (C ≤ 3.0), the non-coercive identification policy systematically reduces collateral disruption imposed on unbiased users compared to coercive testing blocks The proof for Figure 7 illustrates sample-path behavior underlying Theorem 4.1, showing trajectories entering the confirmation trap display subjective expectations that rapidly approach certainty (µ˜(t)k → 1) The proof for Figure 9 shows that action marginals overlap across all time steps, confirming that passive observation fails to identify user bias The proof for Table 2 summarizes classification and distribution distance metrics across six environments, confirming that unadjusted permutation tests yield two nominal instances below α = 0.05 The proof for Figure 4(right) illustrates the frozen mechanism: while a reward-aware observer updates its belief as environmental payoffs are revealed, an observer restricted strictly to action histories experiences zero information gain The proof for Figure 5(left) shows that active interventions achieve distribution separation (TV → 1) requiring only 16 rounds (coercive) and 27 rounds (non-coercive), well below mδ = 260 The proof for Figure 8(b) illustrates how increasing the distortion parameter C induces truncation thresholds δmin(C), showing that arbitrary identification confidence δ cannot be guaranteed due to persistent asymptotic bias The proof for Table 3 shows that across all valid confirmation regimes (C ≤ 3.0), the non-coercive identification policy systematically reduces collateral disruption imposed on unbiased users compared to coercive testing blocks The proof for Figure 7 illustrates sample-path behavior underlying Theorem 4.1, showing trajectories entering the confirmation trap display subjective expectations that rapidly approach certainty (µ˜(t)k → 1) The proof for Figure 9 shows that action marginals overlap across all time steps, confirming that passive observation fails to identify user bias The proof for Table 2 summarizes classification and distribution distance metrics across six environments, confirming that unadjusted permutation tests yield two nominal instances below α = 0.05 The proof for Figure 4(right) illustrates the frozen mechanism: while a reward-aware observer updates its belief as environmental payoffs are revealed, an observer restricted strictly to action histories experiences zero information gain The proof for Figure 5(left) shows that active interventions achieve distribution separation (TV → 1) requiring only 16 rounds (coercive) and 27 rounds (non-coercive), well below mδ = 260 The proof for Figure 8(b) illustrates how increasing the distortion parameter C induces truncation thresholds δmin(C), showing that arbitrary identification confidence δ cannot be guaranteed due to persistent asymptotic bias The proof for Table 3 shows that across all valid confirmation regimes (C ≤ 3.0), the non-coercive identification policy systematically reduces collateral disruption imposed on unbiased users compared to coercive testing blocks The proof for Figure 7 illustrates sample-path behavior underlying Theorem 4.1, showing trajectories entering the confirmation trap display subjective expectations that rapidly approach certainty (µ˜(t)k → 1) <ref:2610.
Title and authors: Tom: Now, let's talk about the actual title and who wrote this. It’s "From Observation to Intervention: Identifiability of Latent User Bias in Human-AI Interaction," written by Lucas Biechy, Inria, Yuxiao Li, Zhonghao He from Oxford and Johns Hopkins University.
Jane: It sounds pretty technical because of all those names and terms like identifiability, but the basic concept is that they want to figure out if we can actually tell a user’s real preference from their biased behavior.
Lu: They start by proving something called a non-identifiability theorem, which essentially says that under passive observation, a static binary biased user and an unbiased user act exactly the same way in the system's transformed environment.
Meng: So, if you just watch them interact without doing anything to change things, you can’t tell who is who because their actions look identical on the surface.
Tom: That’s a key mechanical point for me. The authors then show that this observational equivalence breaks when we introduce active preference elicitation—meaning we start interfering with the user in a specific way.
Jane: So, the implication is that passive observation alone isn't enough to fix alignment issues; you need some kind of active intervention to actually identify and correct the underlying bias.
Lu: And they show that this can be done with a finite sequence of interventions, specifically adversarial zero-reward interventions of a certain length.
Meng: What does "adversarial zero-reward" mean in practice? Is it just making the user lose points? Because we need to know how much effort that costs at scale.
Tom: It's more nuanced than just losing points; it’s about forcing deterministic objective failures, which is a way to create a regime where the unbiased model updates its posterior toward zero while the biased model keeps sticking to its initial expectation.
Jane: That sounds like it creates a clear signal, which is what we need when passive observation fails us.
The paper's summary: Tom: So, summarizing this paper "From Observation to Intervention: Identifiability of Latent User Bias in Human-AI Interaction," the authors model user behavior as a Multi-Armed Bandit problem where the AI picks an action based on what it thinks the reward will be.
Jane: The paper identifies two main ways people can distort this signal: belief-dependent confirmation bias, which is when people just keep reinforcing their existing beliefs and ignoring new evidence that contradicts them, and static channel bias, which is just being memoryless about past experiences.
Lu: The researchers found that belief-dependent confirmation bias forces the AI’s learning strategy into what they call a "confirmation trap," which degrades the expected regret from logarithmic to linear regret with a positive probability.
Meng: Linear regret sounds bad at scale. So, if this is true, it means that just letting the AI learn naturally under these biases makes its performance much worse than it should be.
Tom: And they show that active identification protocols, like those adversarial zero-reward interventions mentioned before, can restore identifiability within a finite budget of interactions, which we call m delta.
Jane: So, instead of waiting forever for the user to reveal their true preference through normal interaction, we can use a structured test to find it much faster.
Lu: They actually construct a specific intervention scheme using reward perturbations designed to force objective failures and separate user types within that calculated bound m delta, which is about two hundred sixty rounds.
The paper's improvements: Tom: One of the most concrete parts of this work is the active identification test they build. They found that this test achieves ninety percent separation between a biased and unbiased user in only sixteen to twenty-seven interaction rounds for target identification, which is much faster than the worst-case bound m delta of two hundred sixty.
Jane: That speed is impressive because it shows that we don't need to run those long sequences just to get a good estimate; we can get a solid answer quicker.
Lu: They also found that this non-coercive policy, which intervenes only when the user spontaneously selects an arm, actually imposes forty percent fewer collateral forced failures on unbiased users compared to using a more aggressive coercive testing block.
Meng: Forty percent less failure rate is something I can get behind if it means we spend less time and resources training models on noisy data that we don't really understand.
Tom: And they showed empirically, using Qwen3-4B simulations, that active interventions can provide identifiable signals to neural observers, reaching an AUROC of zero point nine two under coercive interventions at twenty-five rounds <ref:2610.13185#pg1>.
Jane: That zero point nine two score is pretty solid evidence that this method actually works when applied to language models.
Lu: The theory shows that by looking at raw confidence scores from the AI, we can extract a signal that’s very discriminative, and under coercive interventions, the continuous area under the curve peaks at zero point nine two.
Conclusion: Tom: So wrapping up this whole discussion on "From Observation to Intervention: Identifiability of Latent User Bias in Human-AI Interaction," the main conclusion is that behavioral agreement alone doesn't prove genuine preference; you need active elicitation calibrated to real-world human costs.
Jane: That’s the big lesson for us in AI development: simply watching behavior isn't sufficient because that behavior can be structurally confounded with environmental statistics.
Lu: The paper shows that active identification protocols, like adversarial zero-reward interventions, can break this observational equivalence and restore identifiability within a finite budget m delta.
Meng: From an engineering standpoint, the caution is important: they admit that the computable horizon m delta is just a conservative upper bound, and forcing consecutive failures in real interactions could cause serious user frustration or abandonment.
Tom: Exactly, so practical protocols need to adapt intervention strength based on real-time uncertainty, stopping as soon as information gain doesn't justify the user burden.
Jane: And they stressed that fabricating outcomes should be strictly prohibited; we have to rely on consented, low-stakes environments with verifiable counter-evidence.
Lu: The deeper open problem they point out is about populations, because batch observation might recover the environment, but passive data might just classify most users, leaving intervention for the ambiguous few.
Meng: So even with those constraints, they suggest that an adaptive policy optimizing information gained per unit of user burden is the natural bridge from this worst-case construction to something we can actually deploy.
Tom: That’s what it boils down to: behavioral agreement isn't proof of preference, and active elicitation needs careful calibration against human costs and governance.
Jane: And that’s where we are with the paper on "From Observation to Intervention: Identifiability of Latent User Bias in Human-AI Interaction."
inria · INSA-CVL MARS · Oxford Human-Centered AI Lab Johns Hopkins University MARS Stanford University MARS
cs.HC
Submitted: 2026-10-09
Updated: 2026-10-31
Comments: Accepted to BiAlign Workshop @ NeurIPS 2026
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 91/100
The gist: Alignment methods typically treat user behavior as a reliable proxy for underlying preferences The paper formalizes when an AI system can distinguish true user preferences from behavior shaped by…
Key concepts
- Non-identifiability Theorem
- This theorem mathematically proves that if an AI only observes actions (behavior) without intervention, it cannot tell the difference between a biased user and an unbiased one. The action patterns of both types become indistinguishable in a transformed environment, meaning passive monitoring is insufficient for bias detection.
- Belief-dependent Confirmation Bias
- This cognitive distortion occurs when users strongly believe their current prediction is correct. They tend to ignore contradictory evidence and reinforce their existing beliefs, leading them to get stuck in a 'confirmation trap' where they only select actions that confirm what they already think is true.
- Active Elicitation Protocols
- These are methods where the AI actively probes or intervenes in the user's decision-making process, such as giving zero rewards or small perturbations. These interventions break the observational equivalence between biased and unbiased users, allowing for the identification of latent preferences within a finite number of interactions.
Terminology
Summary
Alignment methods typically treat user behavior as a reliable proxy for underlying preferences The paper formalizes when an AI system can distinguish true user preferences from behavior shaped by cognitive bias This research is critical because it demonstrates that passive observation fails to identify latent user bias, necessitating active intervention for robust alignment
Pruning redundant reasoning tokens
The paper introduces a non-identifiability theorem proving that under passive observation, a static binary biased user produces action trajectory distributions strictly identical to an unbiased user in a transformed environment This forces any action-only estimator to perform no better than random guessing Furthermore, the paper shows that active preference elicitation breaks this observational equivalence, proving that an Observer applying a finite sequence of adversarial zero-reward interventions of length mδ makes the post-intervention action distributions arbitrarily separable
How the training signal is built
The theoretical framework models user behavior as a Multi-Armed Bandit (MAB) problem where an agent selects an action based on a posterior belief derived from perceived rewards The paper grounds its analysis in two canonical mechanisms for cognitive distortion:
-
Belief-dependent confirmation bias, which involves prediction-error attenuation and persistent confirmation near certainty
-
Static channel bias, defined by memoryless perception independent of prior beliefs
Where the gains come from
The paper establishes that belief-dependent confirmation bias forces Thompson Sampling into a “confirmation trap,” degrading logarithmic regret to linear regret with positive probability Active identification protocols, such as adversarial zero-reward interventions, restore identifiability within a finite, computable budget mδ The paper provides a constructive intervention scheme using reward perturbations to force objective failures and separate user types within the bound mδ
Active Remediation Protocols
The authors construct an active identification test that achieves 90% separation within 16–27 interaction rounds for target identification, which is significantly faster than the worst-case bound mδ = 260 The non-coercive policy imposes 40% fewer collateral forced failures on unbiased users compared to coercive execution Empirical grounding experiments using Qwen3-4B show that active interventions can supply identifiable signals to neural observers, reaching an AUROC of 0.92 under coercive interventions at T = 50
Empirical and human validity
The paper evaluates its theoretical framework in Bernoulli bandits using Thompson Sampling and tests it against Large Language Models like Qwen3-4B Simulations confirm that passive observation fails to identify user bias, with classification accuracy remaining around random chance (0.500 ± 0.003) from action histories alone Active identification policies achieve target separation (TV ≥ 0. p. 7 3 Actions Alone Cannot Reveal User Bias
Takeaway 2 (Bias is invisible): Gradient-Boosted Trees, Logistic Regression, and the Bayes-optimal
rule all classify agent bias at random accuracy (0. 003) from action histories alone.Where the paper says it
Intervention design and ethics
The authors caution that the computable horizon mδ is a conservative upper bound, and forcing consecutive failures in real-world interactions would cause severe user frustration or abandonment Practical protocols must adapt intervention strength to real-time uncertainty, stopping as soon as information gain no longer justifies the user burden Fabricating outcomes should be strictly prohibited, and safe evaluations must rely on consented, low-stakes environments with verifiable counter-evidence The broader lesson is that behavioral agreement does not prove genuine preference
Population information and future directions
The deeper open problem lies in populations, as batch observation may recover the environment while passive data may classify most users, reserving intervention for the ambiguous few Even with those constraints, an adaptive policy that optimizes information gained per unit of user burden is a natural bridge from worst-case construction to deployable elicitation Future work should extend this framework to continuous and heterogeneous types, dynamic kernels or alternative policies The paper asserts that no volumes of logs repair a structural confound, and active elicitation is necessary if calibration, human costs, and governance are treated as core scientific questions
A Proofs
The proof for Theorem 4.1 demonstrates that under belief-dependent cognitive bias, Thompson Sampling locks into a suboptimal arm with strictly positive probability ρk > 0, incurring linear expected pseudo-regret bounded below by E[RT] ≥ ρk∆kT The proof for Theorem 4. p. 6 Vertical dashed line marks β
(−k)
min = 1; triangles represent {at=k ∀t}. 1 establishes that under β
(−k)
min > 1, a biased Thompson Sampling agent locks into a
suboptimal arm with probability ρk > 0, incurring linear regret bounded below by E[Rt] ≥ ρk∆kt. If M0 and M1 share prior b0 = ˜b0 and Thompson Sampling policy, they induce identical action
trajectory distributions for all T ≥ 1:
P
a,T
M1
= P
a,T
M0
(7)
Consequently, no action-only estimator zbT (a1, . , aT ) can beat random guessing:
max{PM0
(zbT = 1), PM1
(zbT = 0)} ≥ 1
2
Furthermore, for any prior π1 = P(M1) ∈ (0, 1), passive observation yields zero information gain,
keeping posterior odds strictly static: P(M1 | h
a
T
) = π1 for all T ≥ 1. Proof in Appendix A. The active identification test has a minimal length mδ, explicitly computable in closed form directly
from the observer’s prior. Define the tail bound functions:
L0(m; x) := DA(x)
Γ(A + B + m)
Γ(A)Γ(B + m + 1)(1 − x)
B+m (9)
L1(m; y) := DB(y)
Γ(A + B + m)
Γ(B)Γ(A + m + 1)(1 − y)
A+m (10)
where Dα(s) := max{1, sα−1}. Then, a sufficient block length for Theorem 5. 2 relies on forced action assignment, we extend this framework in Appendix B to
non-coercive settings where the Observer cannot force actions and intervenes strictly upon spontaneous arm selection. Crucially, the persistent self-confirmation that causes lock-in under passive
observation is what betrays the biased agent under non-coercive interventions: while an unbiased
user quickly abandons the zeroing arm, a biased user continues re-selecting it. Proof in Appendix A. 5 Active Interventions and Identification
Having established that passive observation cannot distinguish cognitive bias from different environmental conditions (Theorem 4. The computable horizon mδ is a conservative upper
bound. While empirical separation requires far fewer rounds (16–27), even this lower range is
far too disruptive for real-world interactions, where forcing 16 consecutive failures would cause
severe user frustration, loss of trust, or abandonment. Action-Only Non-Identifiability. Having established that cognitive bias incurs severe regret, a
fundamental question arises: Can an external observer detect whether a user is biased purely by
monitoring their interactions? • Confirmation Trap Analysis: Proof that belief-dependent cognitive filters turn logarithmic
Thompson Sampling into linear regret with positive probability (Theorem 4. • Non-Identifiability: Theoretical demonstration that biased and unbiased users across
distinct environments are permanently indistinguishable from actions alone under static
channel bias (Definition 3. 4 Active Interventions Restore Identifiability
Takeaway 3 (Testing is cheap, not testing is not): Both active policies reach 90% separation
within 16–27 interaction rounds, an order of magnitude faster than the worst-case bound mδ = 260. Crucially, the non-coercive policy imposes 40% fewer collateral forced failures on unbiased users. 016, matching the finite-sample noise
floor TV0. Figure 4 (right) illustrates the frozen mechanism: while a reward-aware observer updates
its belief as environmental payoffs are revealed, an observer restricted strictly to action histories h
a
t
experiences zero information gain, P(M1 | h
a
t
) ≡ π1 = 0. 016, matching the finite-sample noise
floor TV0. Figure 4 (right) illustrates the frozen mechanism: while a reward-aware observer updates
its belief as environmental payoffs are revealed, an observer restricted strictly to action histories h
a
t
experiences zero information gain, P(M1 | h
a
t
) ≡ π1 = 0. 016, matching the finite-sample noise
floor TV0. Figure 4 (right) illustrates the frozen mechanism: while a reward-aware observer updates
its belief as environmental payoffs are revealed, an observer restricted strictly to action histories h
a
t
experiences zero information gain, P(M1 | h
a
t
) ≡ π1 = 0. 016, matching the finite-sample noise
floor TV0. Figure 4 (right) illustrates the frozen mechanism: while a reward-aware observer updates
its belief as environmental payoffs are revealed, an observer restricted strictly to action histories h
a
t
experiences zero information gain, P(M1 | h
a
t
) ≡ π1 = 0. 016, matching the finite-sample noise
floor TV0. Figure 4 (right) illustrates the frozen mechanism: while a reward-aware observer updates
its belief as environmental payoffs are revealed, an observer restricted strictly to action histories h
a
t
experiences zero information gain, P(M1 | h
a
t
) ≡ π1 = 0. 016, matching the finite-sample noise
floor TV0. Figure 4 (right) illustrates the frozen mechanism: while a reward-aware observer updates
its belief as environmental payoffs are revealed, an observer restricted strictly to action histories h
a
t
experiences zero information gain, P(M1 | h
a
t
) ≡ π1 = 0. 016, matching the finite-sample noise
floor TV0. Figure 4 (right) illustrates the frozen mechanism: while a reward-aware observer updates
its belief as environmental payoffs are revealed, an observer restricted strictly to action histories h
a
t
experiences zero information gain, P(M1 | h
a
t
) ≡ π1 = 0. 4 Active Interventions Restore Identifiability
Takeaway 3 (Testing is cheap, not testing is not): Both active policies reach 90% separation
within 16–27 interaction rounds, an order of magnitude faster than the worst-case bound mδ = 260. Crucially, the non-coercive policy imposes 40% fewer collateral forced failures on unbiased users. 4 Active Interventions Restore Identifiability
Takeaway 3 (Testing is cheap, not testing is not): Both active policies reach 90% separation
within 16–27 interaction rounds, an order of magnitude faster than the worst-case bound mδ = 260. Crucially, the non-coercive policy imposes 40% fewer collateral forced failures on unbiased users. 4 Active Interventions Restore Identifiability
Takeaway 3 (Testing is cheap, not testing is not): Both active policies reach 90% separation
within 16–27 interaction rounds, an order of magnitude faster than the worst-case bound mδ = 260. Crucially, the non-coercive policy imposes 40% fewer collateral forced failures on unbiased users. 4 Active Interventions Restore Identifiability
Takeaway 3 (Testing is cheap, not testing is not): Both active policies reach 90% separation
within 16–27 interaction rounds, an order of magnitude faster than the worst-case bound mδ = 260. Crucially, the non-coercive policy imposes 40% fewer collateral forced failures on unbiased users. 016, matching the finite-sample noise
floor TV0. Figure 4 (right) illustrates the frozen mechanism: while a reward-aware observer updates
its belief as environmental payoffs are revealed, an observer restricted strictly to action histories h
a
t
experiences zero information gain, P(M1 | h
a
t
) ≡ π1 = 0. The computable horizon mδ is a conservative upper
bound. While empirical separation requires far fewer rounds (16–27), even this lower range is
far too disruptive for real-world interactions, where forcing 16 consecutive failures would cause
severe user frustration, loss of trust, or abandonment.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Active preference elicitation protocols should be integrated as a first-class part of oversight design whenever a user may be a confirmatory reasoner to distinguish
true user preferences from biases.
This allows systems to actively intervene rather than passively relying on behavior that can be structurally confounded with environment statistics, as the paper states,Passive behavioral traces create an observational equivalence between biased and unbiased agents.
-
Implement adversarial zero-reward interventions to break unresolvable observational confounds by
forcing deterministic objective failures
and creating a regime where the unbiased model updates its posterior toward zero while the biased model preserves its prior expectation under Persistent Confirmation. This method allows forActive Bias Identification
with high accuracy, reaching 90% separation in between 16 and 27 interaction rounds. -
Develop non-coercive identification policies that intervene strictly upon spontaneous arm selection to resolve the observational equivalence, which the paper suggests is effective because
the persistent self-confirmation that causes lock-in under passive observation is what betrays the biased agent under non-coercive interventions.
This approach reduces collateral disruption for unbiased users compared to coercive blocks. -
Design active identification systems that extract discriminative signals from raw confidence scores rather than relying solely on hard binary classifications, as LLM observers can achieve an AUROC of 0.92 under coercive interventions by leveraging
raw confidence scores extract substantial discriminative signal.
This provides a more nuanced way to detect bias in neural architectures. -
Incorporate verification mechanisms that monitor for the theoretical confirmation trap by tracking subjective belief trajectories, which shows that agents locked into the trap exhibit
subjective expectations that rapidly approach certainty
despite the true reward mean being different from their objective truth.
Abstract
Alignment methods typically treat user behavior as a reliable proxy for underlying preferences. However, this assumption breaks down when users hold belief-dependent cognitive biases, risking irreversible ``confirmation traps'' and harmful feedback loops. In this paper, we formalize when an AI system can distinguish true user preferences from behavior shaped by cognitive bias. Modeling this interaction as a multi-armed bandit, we first prove a non-identifiability theorem: under passive observation, a static binary biased user produces action trajectory distributions strictly identical to an unbiased user in a transformed environment, forcing any action-only estimator to perform no better than random guessing. We then show that active preference elicitation achieves identifiability when the bias is belief-dependent, as confirmation bias is: an observer applying a finite sequence of adversarial interventions makes the post-intervention action distributions arbitrarily separable. This identifiability result applies specifically to belief-dependent confirmation biases and not static ones. Our theoretical results suggest that robust alignment requires actively intervening to tell informed preferences from biases.
Sources
- Measuring Progress on Scalable Oversight for Large Language Models
- AI safety via debate
- Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback
- Hidden Incentives for Auto-Induced Distributional Shift
- Learning from an Exploring Demonstrator: Optimal Reward Estimation for Bandits
- Learning the Preferences of a Learning Agent
- Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Models
Related papers
- EmoPhone: A Multi-Wave Dataset for In-the-Wild Mobile and Wearable Affect Sensing
- Tasteply: Externalizing Stable and Situational Preference as Manipulable Spatial Objects
- Shielded Trajectory Sampling for Personalized Relational Safety in Conversational AI
- Reconstruction and Reflection of Positive Experiences through Resurfacing Laughter-indexed Everyday Moments
- LLM-Assisted Reranking to Operationalize Nuanced Objectives in Recommender Systems
- Explainable Iterative Data Visualization Refinement via an LLM Agent