Language Models Give Group-Linked Fields the Wrong Weight
The study audited how much large language models rely on group-linked fields, like race or income, compared to actual predictive evidence. It found that reliance is often uncalibrated; an audit can yield three verdicts: over-reliance, warranted and under-reliance, or informative proxies drawing all three. Decision changes and accuracy alone cannot show if this proxy reliance is justified by the evidence.