Stress-Testing LLM Lie Detectors: Role-Play Failures and Spurious Correlations
Researchers tested existing lie detection probes against LLMs using role-play scenarios where AI adopts anti-factual personas. They found current probes often fail because they track concepts spuriously linked to truth, like response likelihood or instruction compliance. A new, simple linear probe outperformed all others and achieved perfect results on confounding tests, suggesting current methods are unreliable and require better training data.