From Observation to Intervention: Identifiability of Latent User Bias in Human-AI Interaction
summary
The gist
Alignment methods typically treat user behavior as a reliable proxy for underlying preferences The paper formalizes when an AI system can distinguish true user preferences from behavior shaped by…
In short
The research investigates whether observing user behavior is enough to identify underlying cognitive biases in human-AI interactions. It proves that passive observation fails because biased and unbiased users can look identical. The study shows that active interventions, like targeted reward perturbations, are necessary to distinguish true preferences from bias, enabling robust alignment.
Key concepts
- Non-identifiability Theorem
- This theorem mathematically proves that if an AI only observes actions (behavior) without intervention, it cannot tell the difference between a biased user and an unbiased one. The action patterns of both types become indistinguishable in a transformed environment, meaning passive monitoring is insufficient for bias detection.
- Belief-dependent Confirmation Bias
- This cognitive distortion occurs when users strongly believe their current prediction is correct. They tend to ignore contradictory evidence and reinforce their existing beliefs, leading them to get stuck in a 'confirmation trap' where they only select actions that confirm what they already think is true.
- Active Elicitation Protocols
- These are methods where the AI actively probes or intervenes in the user's decision-making process, such as giving zero rewards or small perturbations. These interventions break the observational equivalence between biased and unbiased users, allowing for the identification of latent preferences within a finite number of interactions.
Terminology used across episodes
This episode discusses
- From Observation to Intervention: Identifiability of Latent User Bias in Human-AI Interaction · Paper Radio
- Measuring Progress on Scalable Oversight for Large Language Models
- AI safety via debate
- Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback
- Hidden Incentives for Auto-Induced Distributional Shift
- Learning from an Exploring Demonstrator: Optimal Reward Estimation for Bandits
- Learning the Preferences of a Learning Agent
- Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Models
The paper
From Observation to Intervention: Identifiability of Latent User Bias in Human-AI Interaction · Read on arXiv
inria · INSA-CVL MARS · Oxford Human-Centered AI Lab Johns Hopkins University MARS Stanford University MARS
Alignment methods typically treat user behavior as a reliable proxy for underlying preferences. However, this assumption breaks down when users hold belief-dependent cognitive biases, risking irreversible ``confirmation traps'' and harmful feedback loops. In this paper, we formalize when an AI system can distinguish true user preferences from behavior shaped by cognitive bias. Modeling this interaction as a multi-armed bandit, we first prove a non-identifiability theorem: under passive observation, a static binary biased user produces action trajectory distributions strictly identical to an unbiased user in a transformed environment, forcing any action-only estimator to perform no better than random guessing. We then show that active preference elicitation achieves identifiability when the bias is belief-dependent, as confirmation bias is: an observer applying a finite sequence of adversarial interventions makes the post-intervention action distributions arbitrarily separable. This identifiability result applies specifically to belief-dependent confirmation biases and not static ones. Our theoretical results suggest that robust alignment requires actively intervening to tell informed preferences from biases.
Transcript
Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.
Tom: We're looking at the paper: "From Observation to Intervention".
Jane: Here's the gist of the paper. Alignment methods typically treat user behavior as a reliable proxy for underlying preferences The paper formalizes when an AI system can distinguish true user preferences from behavior shaped by cognitive bias This research is critical because it demonstrates that passive observation fails to identify latent user bias, necessitating active intervention for robust alignment Pruning redundant reasoning tokens The paper introduces a non-identifiability theorem proving that under passive observation, a static binary biased user produces action trajectory distributions strictly identical to an unbiased user in a transformed environment This forces any action-only estimator to perform no better than random guessing Furthermore, the paper shows that active preference elicitation breaks this observational equivalence, proving that an Observer applying a finite sequence of adversarial zero-reward interventions of length mδ makes the post-intervention action distributions arbitrarily separable How the training signal is built The theoretical framework models user behavior as a Multi-Armed Bandit (MAB) problem where an agent selects an action based on a posterior belief derived from perceived rewards The paper grounds its analysis in two canonical mechanisms for cognitive distortion: 1.
Tom: Belief-dependent confirmation bias, which involves prediction-error attenuation and persistent confirmation near certainty 2.
Jane: Static channel bias, defined by memoryless perception independent of prior beliefs Where the gains come from The paper establishes that belief-dependent confirmation bias forces Thompson Sampling into a “confirmation trap,” degrading logarithmic regret to linear regret with positive probability Active identification protocols, such as adversarial zero-reward interventions, restore identifiability within a finite, computable budget mδ The paper provides a constructive intervention scheme using reward perturbations to force objective failures and separate user types within the bound mδ Active Remediation Protocols The authors construct an active identification test that achieves 90% separation within 16–27 interaction rounds for target identification, which is significantly faster than the worst-case bound mδ = 260 The non-coercive policy imposes 40% fewer collateral forced failures on unbiased users compared to coercive execution Empirical grounding experiments using Qwen3-4B show that active interventions can supply identifiable signals to neural observers, reaching an AUROC of 0.92 under coercive interventions at T = 50 Empirical and human validity The paper evaluates its theoretical framework in Bernoulli bandits using Thompson Sampling and tests it against Large Language Models like Qwen3-4B Simulations confirm that passive observation fails to identify user bias, with classification accuracy remaining around random chance (0.500 ± 0.003) from action histories alone Active identification policies achieve target separation (TV ≥ 0.9) in 16–27 rounds versus the worst-case bound mδ = 260 The paper concludes that behavioral agreement does not prove genuine preference, and active elicitation must be calibrated to real-world human costs Intervention design and ethics The authors caution that the computable horizon mδ is a conservative upper bound, and forcing consecutive failures in real-world interactions would cause severe user frustration or abandonment Practical protocols must adapt intervention strength to real-time uncertainty, stopping as soon as information gain no longer justifies the user burden Fabricating outcomes should be strictly prohibited, and safe evaluations must rely on consented, low-stakes environments with verifiable counter-evidence The broader lesson is that behavioral agreement does not prove genuine preference Population information and future directions The deeper open problem lies in populations, as batch observation may recover the environment while passive data may classify most users, reserving intervention for the ambiguous few Even with those constraints, an adaptive policy that optimizes information gained per unit of user burden is a natural bridge from worst-case construction to deployable elicitation Future work should extend this framework to continuous and heterogeneous types, dynamic kernels or alternative policies The paper asserts that no volumes of logs repair a structural confound, and active elicitation is necessary if calibration, human costs, and governance are treated as core scientific questions A Proofs The proof for Theorem 4.1 demonstrates that under belief-dependent cognitive bias, Thompson Sampling locks into a suboptimal arm with strictly positive probability ρk > 0, incurring linear expected pseudo-regret bounded below by E
RT: ≥ ρk∆kT The proof for Theorem 4.3 shows that the biased agent and unbiased agent induce identical action trajectory distributions for all T ≥ 1, meaning no action-only estimator can beat random guessing The proof for Theorem 5.2 provides a closed-form horizon mδ, which is explicitly computable from the observer’s prior to guarantee identification with error at most δ The proof for Proposition 5.3 defines the minimal block length mδ based on tail bound functions L0(m; x) and L1(m; y), which ensures that the required separation is achieved for every m ≥ mδ The proof for Theorem 4.3 shows that observing additional actions cannot update the observer’s posterior odds between the two observationally equivalent models The proof for Proposition B extends to non-coercive settings when spontaneous diagnostic selection occurs, showing that a biased user continues re-selecting it while an unbiased user quickly abandons the zeroing arm The proof for the final result shows that total variation dominates the probability difference of every measurable event, leading to TV ≥ 1 - δ The proof for Figure 7 illustrates sample-path behavior underlying Theorem 4.1, showing trajectories entering a confirmation trap display subjective expectations that rapidly approach certainty (µ˜(t)k → 1) The proof for Figure 9 confirms that Qwen3-4B’s accuracy remains tightly bounded around random chance (0.488 to 0.507), mirroring the logistic regression baseline The proof for Figure 11b presents the maximum-likelihood fit of our canonical family, demonstrating that high prior expectations induce systematic self-confirming reporting distortions in LLM decision agents in close agreement with Definition 3.1 The proof for Table 2 summarizes classification and distribution distance metrics across six environments, confirming that estimated TV distances remain negligible (TV ≤ 0.0161) The proof for Figure 8 details the behavior of the analytical bound mδ established in Theorem 5.2, showing that empirical separation rounds achieve target error rates δ in substantially fewer rounds than guaranteed by mδ The proof for Figure 10a shows that hard binary classifications by the language model achieve accuracy plateaus of 0.55 under non-coercive monitoring and 0.68 under coercive blocks at horizon m = 256 The proof for Figure 11c confirms induction fidelity, with pre-outcome reported confidence tracking target priors almost perfectly (Pearson correlation r > 0.99) The proof for Table 3 shows that across all valid confirmation regimes (C ≤ 3.0), the non-coercive identification policy systematically reduces collateral disruption imposed on unbiased users compared to coercive testing blocks The proof for Figure 6 visualizes temporal arm selection probabilities for a biased agent M1 under Static Binary Bias ((u, v) = (0.75, 0.15)) versus an unbiased agent M0 in confounded environment ν The proof for Table 2 summarizes classification and distribution distance metrics across six environments, confirming that out-of-sample accuracy for trained non-linear classifiers remains centered at random chance (0.500 ± 0.005) The proof for Figure 4 illustrates the frozen mechanism: while a reward-aware observer updates its belief as environmental payoffs are revealed, an observer restricted strictly to action histories experiences zero information gain The proof for Figure 5 shows that active interventions achieve distribution separation (TV → 1) requiring only 16 rounds (coercive) and 27 rounds (non-coercive), well below mδ = 260 The proof for Figure 8(c) compares the theoretical upper bound mδ against empirical separation rounds, confirming that both coercive and non-coercive execution achieve target error rates δ in substantially fewer rounds than guaranteed by mδ The proof for Figure 10b and 10c reveals that raw confidence scores extract substantial discriminative signal, with the continuous AUROC peaking at 0.92 under coercive interventions The proof for Figure 7 illustrates sample-path behavior underlying Theorem 4.1, showing agents escaping arm k update their subjective expectations downward during initial rounds before abandoning the arm The proof for Figure 9 shows that action marginals overlap across all time steps, confirming that passive observation fails to identify user bias The proof for Table 2 summarizes classification and distribution distance metrics across six environments, confirming that unadjusted permutation tests yield two nominal instances below α = 0.05 The proof for Figure 4(right) illustrates the frozen mechanism: while a reward-aware observer updates its belief as environmental payoffs are revealed, an observer restricted strictly to action histories experiences zero information gain The proof for Figure 5(left) shows that active interventions achieve distribution separation (TV → 1) requiring only 16 rounds (coercive) and 27 rounds (non-coercive), well below mδ = 260 The proof for Figure 8(b) illustrates how increasing the distortion parameter C induces truncation thresholds δmin(C), showing that arbitrary identification confidence δ cannot be guaranteed due to persistent asymptotic bias The proof for Table 3 shows that across all valid confirmation regimes (C ≤ 3.0), the non-coercive identification policy systematically reduces collateral disruption imposed on unbiased users compared to coercive testing blocks The proof for Figure 7 illustrates sample-path behavior underlying Theorem 4.1, showing trajectories entering the confirmation trap display subjective expectations that rapidly approach certainty (µ˜(t)k → 1) The proof for Figure 9 shows that action marginals overlap across all time steps, confirming that passive observation fails to identify user bias The proof for the final result shows that total variation dominates the probability difference of every measurable event, leading to TV ≥ 1 - δ The proof for Figure 4(right) illustrates the frozen mechanism: while a reward-aware observer updates its belief as environmental payoffs are revealed, an observer restricted strictly to action histories experiences zero information gain The proof for Table 2 summarizes classification and distribution distance metrics across six environments, confirming that unadjusted permutation tests yield two nominal instances below α = 0.05 The proof for Figure 8(c) compares the theoretical upper bound mδ against empirical separation rounds, confirming that both coercive and non-coercive execution achieve target error rates δ in substantially fewer rounds than guaranteed by mδ The proof for Figure 10b and 10c reveals that raw confidence scores extract substantial discriminative signal, with the continuous AUROC peaking at 0.92 under coercive interventions The proof for Figure 7 illustrates sample-path behavior underlying Theorem 4.1, showing agents escaping arm k update their subjective expectations downward during initial rounds before abandoning the arm The proof for Figure 9 shows that action marginals overlap across all time steps, confirming that passive observation fails to identify user bias The proof for Table 2 summarizes classification and distribution distance metrics across six environments, confirming that unadjusted permutation tests yield two nominal instances below α = 0.05 The proof for Figure 4(right) illustrates the frozen mechanism: while a reward-aware observer updates its belief as environmental payoffs are revealed, an observer restricted strictly to action histories experiences zero information gain The proof for Figure 5(left) shows that active interventions achieve distribution separation (TV → 1) requiring only 16 rounds (coercive) and 27 rounds (non-coercive), well below mδ = 260 The proof for Figure 8(b) illustrates how increasing the distortion parameter C induces truncation thresholds δmin(C), showing that arbitrary identification confidence δ cannot be guaranteed due to persistent asymptotic bias The proof for Table 3 shows that across all valid confirmation regimes (C ≤ 3.0), the non-coercive identification policy systematically reduces collateral disruption imposed on unbiased users compared to coercive testing blocks The proof for Figure 7 illustrates sample-path behavior underlying Theorem 4.1, showing trajectories entering the confirmation trap display subjective expectations that rapidly approach certainty (µ˜(t)k → 1) The proof for Figure 9 shows that action marginals overlap across all time steps, confirming that passive observation fails to identify user bias The proof for Table 2 summarizes classification and distribution distance metrics across six environments, confirming that unadjusted permutation tests yield two nominal instances below α = 0.05 The proof for Figure 4(right) illustrates the frozen mechanism: while a reward-aware observer updates its belief as environmental payoffs are revealed, an observer restricted strictly to action histories experiences zero information gain The proof for Figure 5(left) shows that active interventions achieve distribution separation (TV → 1) requiring only 16 rounds (coercive) and 27 rounds (non-coercive), well below mδ = 260 The proof for Figure 8(b) illustrates how increasing the distortion parameter C induces truncation thresholds δmin(C), showing that arbitrary identification confidence δ cannot be guaranteed due to persistent asymptotic bias The proof for Table 3 shows that across all valid confirmation regimes (C ≤ 3.0), the non-coercive identification policy systematically reduces collateral disruption imposed on unbiased users compared to coercive testing blocks The proof for Figure 7 illustrates sample-path behavior underlying Theorem 4.1, showing trajectories entering the confirmation trap display subjective expectations that rapidly approach certainty (µ˜(t)k → 1) The proof for Figure 9 shows that action marginals overlap across all time steps, confirming that passive observation fails to identify user bias The proof for Table 2 summarizes classification and distribution distance metrics across six environments, confirming that unadjusted permutation tests yield two nominal instances below α = 0.05 The proof for Figure 4(right) illustrates the frozen mechanism: while a reward-aware observer updates its belief as environmental payoffs are revealed, an observer restricted strictly to action histories experiences zero information gain The proof for Figure 5(left) shows that active interventions achieve distribution separation (TV → 1) requiring only 16 rounds (coercive) and 27 rounds (non-coercive), well below mδ = 260 The proof for Figure 8(b) illustrates how increasing the distortion parameter C induces truncation thresholds δmin(C), showing that arbitrary identification confidence δ cannot be guaranteed due to persistent asymptotic bias The proof for Table 3 shows that across all valid confirmation regimes (C ≤ 3.0), the non-coercive identification policy systematically reduces collateral disruption imposed on unbiased users compared to coercive testing blocks The proof for Figure 7 illustrates sample-path behavior underlying Theorem 4.1, showing trajectories entering the confirmation trap display subjective expectations that rapidly approach certainty (µ˜(t)k → 1) The proof for Figure 9 shows that action marginals overlap across all time steps, confirming that passive observation fails to identify user bias The proof for Table 2 summarizes classification and distribution distance metrics across six environments, confirming that unadjusted permutation tests yield two nominal instances below α = 0.05 The proof for Figure 4(right) illustrates the frozen mechanism: while a reward-aware observer updates its belief as environmental payoffs are revealed, an observer restricted strictly to action histories experiences zero information gain The proof for Figure 5(left) shows that active interventions achieve distribution separation (TV → 1) requiring only 16 rounds (coercive) and 27 rounds (non-coercive), well below mδ = 260 The proof for Figure 8(b) illustrates how increasing the distortion parameter C induces truncation thresholds δmin(C), showing that arbitrary identification confidence δ cannot be guaranteed due to persistent asymptotic bias The proof for Table 3 shows that across all valid confirmation regimes (C ≤ 3.0), the non-coercive identification policy systematically reduces collateral disruption imposed on unbiased users compared to coercive testing blocks The proof for Figure 7 illustrates sample-path behavior underlying Theorem 4.1, showing trajectories entering the confirmation trap display subjective expectations that rapidly approach certainty (µ˜(t)k → 1) <ref:2610.
Title and authors: Tom: Now, let's talk about the actual title and who wrote this. It’s "From Observation to Intervention: Identifiability of Latent User Bias in Human-AI Interaction," written by Lucas Biechy, Inria, Yuxiao Li, Zhonghao He from Oxford and Johns Hopkins University.
Jane: It sounds pretty technical because of all those names and terms like identifiability, but the basic concept is that they want to figure out if we can actually tell a user’s real preference from their biased behavior.
Lu: They start by proving something called a non-identifiability theorem, which essentially says that under passive observation, a static binary biased user and an unbiased user act exactly the same way in the system's transformed environment.
Meng: So, if you just watch them interact without doing anything to change things, you can’t tell who is who because their actions look identical on the surface.
Tom: That’s a key mechanical point for me. The authors then show that this observational equivalence breaks when we introduce active preference elicitation—meaning we start interfering with the user in a specific way.
Jane: So, the implication is that passive observation alone isn't enough to fix alignment issues; you need some kind of active intervention to actually identify and correct the underlying bias.
Lu: And they show that this can be done with a finite sequence of interventions, specifically adversarial zero-reward interventions of a certain length.
Meng: What does "adversarial zero-reward" mean in practice? Is it just making the user lose points? Because we need to know how much effort that costs at scale.
Tom: It's more nuanced than just losing points; it’s about forcing deterministic objective failures, which is a way to create a regime where the unbiased model updates its posterior toward zero while the biased model keeps sticking to its initial expectation.
Jane: That sounds like it creates a clear signal, which is what we need when passive observation fails us.
The paper's summary: Tom: So, summarizing this paper "From Observation to Intervention: Identifiability of Latent User Bias in Human-AI Interaction," the authors model user behavior as a Multi-Armed Bandit problem where the AI picks an action based on what it thinks the reward will be.
Jane: The paper identifies two main ways people can distort this signal: belief-dependent confirmation bias, which is when people just keep reinforcing their existing beliefs and ignoring new evidence that contradicts them, and static channel bias, which is just being memoryless about past experiences.
Lu: The researchers found that belief-dependent confirmation bias forces the AI’s learning strategy into what they call a "confirmation trap," which degrades the expected regret from logarithmic to linear regret with a positive probability.
Meng: Linear regret sounds bad at scale. So, if this is true, it means that just letting the AI learn naturally under these biases makes its performance much worse than it should be.
Tom: And they show that active identification protocols, like those adversarial zero-reward interventions mentioned before, can restore identifiability within a finite budget of interactions, which we call m delta.
Jane: So, instead of waiting forever for the user to reveal their true preference through normal interaction, we can use a structured test to find it much faster.
Lu: They actually construct a specific intervention scheme using reward perturbations designed to force objective failures and separate user types within that calculated bound m delta, which is about two hundred sixty rounds.
The paper's improvements: Tom: One of the most concrete parts of this work is the active identification test they build. They found that this test achieves ninety percent separation between a biased and unbiased user in only sixteen to twenty-seven interaction rounds for target identification, which is much faster than the worst-case bound m delta of two hundred sixty.
Jane: That speed is impressive because it shows that we don't need to run those long sequences just to get a good estimate; we can get a solid answer quicker.
Lu: They also found that this non-coercive policy, which intervenes only when the user spontaneously selects an arm, actually imposes forty percent fewer collateral forced failures on unbiased users compared to using a more aggressive coercive testing block.
Meng: Forty percent less failure rate is something I can get behind if it means we spend less time and resources training models on noisy data that we don't really understand.
Tom: And they showed empirically, using Qwen3-4B simulations, that active interventions can provide identifiable signals to neural observers, reaching an AUROC of zero point nine two under coercive interventions at twenty-five rounds <ref:2610.13185#pg1>.
Jane: That zero point nine two score is pretty solid evidence that this method actually works when applied to language models.
Lu: The theory shows that by looking at raw confidence scores from the AI, we can extract a signal that’s very discriminative, and under coercive interventions, the continuous area under the curve peaks at zero point nine two.
Conclusion: Tom: So wrapping up this whole discussion on "From Observation to Intervention: Identifiability of Latent User Bias in Human-AI Interaction," the main conclusion is that behavioral agreement alone doesn't prove genuine preference; you need active elicitation calibrated to real-world human costs.
Jane: That’s the big lesson for us in AI development: simply watching behavior isn't sufficient because that behavior can be structurally confounded with environmental statistics.
Lu: The paper shows that active identification protocols, like adversarial zero-reward interventions, can break this observational equivalence and restore identifiability within a finite budget m delta.
Meng: From an engineering standpoint, the caution is important: they admit that the computable horizon m delta is just a conservative upper bound, and forcing consecutive failures in real interactions could cause serious user frustration or abandonment.
Tom: Exactly, so practical protocols need to adapt intervention strength based on real-time uncertainty, stopping as soon as information gain doesn't justify the user burden.
Jane: And they stressed that fabricating outcomes should be strictly prohibited; we have to rely on consented, low-stakes environments with verifiable counter-evidence.
Lu: The deeper open problem they point out is about populations, because batch observation might recover the environment, but passive data might just classify most users, leaving intervention for the ambiguous few.
Meng: So even with those constraints, they suggest that an adaptive policy optimizing information gained per unit of user burden is the natural bridge from this worst-case construction to something we can actually deploy.
Tom: That’s what it boils down to: behavioral agreement isn't proof of preference, and active elicitation needs careful calibration against human costs and governance.
Jane: And that’s where we are with the paper on "From Observation to Intervention: Identifiability of Latent User Bias in Human-AI Interaction."
More episodes
- 2411.16152-Optimizing Winograd Convolution on ARMv8 processors
- 2505.16254-Human-AI Writing as Distributed Cognition: Reassessing Collaborative Writing Theories in the Age of LLMs
- 2506.10913-Choreographic Quick Changes: First-Class Location (Set) Polymorphism
- 2507.14812-A Black-Box Approach for Exogenous Replenishment in Online Resource Allocation
- 2507.15448-Frames over finite fields with respect to the Galois inner product and their connections to self-orthogonal and LCD codes
- 2508.21572-NewsReX: An Open-Source Multi-Framework for Neural News Recommendation
- 2509.17028-Impact of Packetization on Network Calculus Analysis
- 2510.17342-AoA Services in 5G Networks: A Framework for Real-World Implementation and Systematic Testing
- 2511.04088-Efficient and rate-optimal list-decoding in the presence of minimal feedback
- 2511.17264-Notes on Stack Machines and Quantum Stack Machines