Language Models Give Group-Linked Fields the Wrong Weight

summary

Video file (mp4)

The gist

The gist Large language models' reliance on group-linked fields is uncalibrated to predictive evidence, meaning that an audit signal can yield three verdicts: over-reliance, warranted and

In short

The study audited how much large language models rely on group-linked fields, like race or income, compared to actual predictive evidence. It found that reliance is often uncalibrated; an audit can yield three verdicts: over-reliance, warranted and under-reliance, or informative proxies drawing all three. Decision changes and accuracy alone cannot show if this proxy reliance is justified by the evidence.

Key concepts

Evidence-Referenced Audit
This method compares a model's reliance on group attributes against two benchmarks: the known outcome rule (what a prediction-optimal decision would do) and a Bayesian linear regression fitted to the same data. It also separately measures ranking accuracy to see how reliance tracks performance.
Evidence-Warranted Level
This is the normative reference point for proxy reliance. It represents the degree of reliance on group attributes that a perfect prediction rule, using all available information, would naturally show. Knowing this level requires a fully specified synthetic process where every attribute's relationship to the outcome and protected attribute is controlled.
Proxy Reliance Dynamics
This examines how reliance on proxies changes as the strength of predictive evidence increases. The findings show that reliance often severely undertracks the evidence, meaning models don't increase their use of group fields proportionally to how much better those fields are at predicting outcomes.
Separability of Channels
Proxy reliance and accuracy are driven by different underlying factors. Structural manipulations can significantly alter proxy reliance without changing accuracy, suggesting that improving performance doesn't automatically reduce the model's reliance on potentially biased group fields.

Terminology used across episodes

This episode discusses

The paper

Language Models Give Group-Linked Fields the Wrong Weight · Read on arXiv

Zengqing Wu,  Chuan Xiao

University of Osaka

Large language models are increasingly used to judge people, and their answers can depend on inputs correlated with a protected attribute such as race, even when that attribute is never shown. Interpreting this reliance requires knowing how much predictive information those inputs add, but audits often report only whether a decision changes. We introduce an evidence-referenced audit that compares proxy reliance with two references, the known outcome rule and a Bayesian linear regression fitted to the same labeled examples, while measuring ranking accuracy separately. We apply it to four models in a simulated patient-ranking task with known field weights. We find four results. First, at zero predictive value, the models still relied on fields that carried no information, while the same-example learner did not. Second, as proxies became predictive, reliance grew far less than the reference. Third, display format mattered. Ordered labels drew more reliance than arbitrary labels, a direction that held in all four models. Fourth, ranking accuracy did not consistently track these changes in reliance. Based on these findings, we argue that decision changes and accuracy cannot show whether proxy reliance is warranted by the evidence. Audits should report reliance against an explicit predictive reference, alongside accuracy. Ordered fields whose values move with group membership, such as scores or income bands, are a useful starting point. Together, these results make it possible to measure how far reliance departs from the evidence, rather than infer it from observed choices alone.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Language Models Give Group-Linked Fields the Wrong Weight".

Jane: Here's the gist of the paper. The gist Large language models' reliance on group-linked fields is uncalibrated to predictive evidence, meaning that an audit signal can yield three verdicts: over-reliance, warranted and under-reliance, and informative proxies draw all three How it works The study measures causal proxy effects in four LLMs on a clinical-ranking task with known ground truth, where the reliance the evidence warrants can be computed exactly and used as the reference.

Tom: The normative reference is defined as "the degree of reliance on group-correlated attributes that a prediction-optimal decision rule with the same information would show," which is called the evidence-warranted level.

Jane: This quantity requires knowing the true relationship between attributes and outcomes, necessitating a fully specified synthetic generating process where each attribute's relation to the outcome and protected attribute is controlled.

Tom: Audit Signals and Verdicts One audit signal yields three verdicts: "over-reliance, warranted and under-reliance".

Paper summary: Tom: So, wrapping up this discussion on "Language Models Give Group-Linked Fields the Wrong Weight", we’ve seen how these language models can be uncalibrated regarding group-linked fields, and that relying only on decision changes doesn't tell us if that reliance is warranted by the evidence.

Jane: It seems the main implication is that audits need to move away from just checking for flips and instead compare proxy reliance against a specific predictive reference point, like what a prediction-optimal rule would show.

Lu: That reference level requires a very specific setup, needing that fully specified synthetic generating process because you can't identify this quantity from observational data by itself.

Meng: For deployment, that means we need to integrate this idea of measuring reliance against the evidence-warranted level into our safety checks, not just look at surface-level metrics like accuracy.

Lalam: I think the key thing is that auditing without this explicit reference can lead to false alarms by penalizing reliance that's actually supported by evidence, and misses because it ignores fragile surface suppression effects.

Tom: So ultimately, the paper suggests we should be asking how much proxy use is warranted by what the data says, rather than just assuming that a decision change means it’s okay.

Jane: That framework of using an explicit predictive reference seems like a solid way to measure how far model reliance deviates from what's actually there.

Conclusion: Tom: So we’re wrapping up this look at the paper "Language Models Give Group-Linked Fields the Wrong Weight" by those authors, and honestly, it boils down to how models misinterpret evidence when they see group labels like race or gender.

Jane: Right. If you're driving and you see a model relying on something that has nothing to do with what’s actually predicting the outcome, that’s the core issue here. It suggests those reliance signals aren't calibrated to the actual predictive power of the inputs.

Lu: The authors introduce this evidence-referenced audit, which compares how much a model relies on a proxy field against two things: what's actually predicted by that field, and what a prediction-optimal rule would rely on.

Meng: And they measure ranking accuracy separately from the reliance itself, which is important because you don't want to confuse the two. It shows that just saying "the model changed its answer" doesn't tell you if it was relying on the right thing or just guessing.

Lalam: Exactly. The paper finds that models can be over-reliant, under-reliant, or even drawing all three kinds of reliance depending on the type of proxy field they’re looking at. It's not a simple pass or fail situation.

Tom: And what this means for us in the real world is that we need to stop just trusting the output change and start asking if that reliance actually makes sense given the data we have.

Jane: It shifts the focus from just "what did it do?" to "why did it do it, and is that why it should have done it?" This moves us toward audits that check for reliance against a true predictive reference point.

Lu: It’s a lot of work because you need that synthetic generating process—you have to control every relationship between the attribute and the outcome, which is pretty demanding.

Meng: From a deployment standpoint, if we adopt this, we can start flagging systems where they're relying on socially sensitive fields even when those fields don't actually matter for the task. That’s what I need to see in production logs.

Tom: It’s a solid finding because it gives us a way to measure how far reliance is from the actual evidence, rather than just guessing what we want to see. So next time you hear about an AI model making a decision based on something sensitive, you'll know there's a deeper layer of risk to look at.

More episodes

← Home