Thin Evidence, Thick Priors: How Language Models Substitute Identity for Missing Financial Facts
cs.AI
Submitted: 2026-10-06
Updated: 2026-10-06
Comments: 49 pages, 17 figures, 12 tables; Submitted & Accepted to ICAIF'2026
License: http://creativecommons.org/licenses/by-nc-nd/4.0/
The gist: People increasingly ask large language models what to do with their money, yet seldom describe their finances in full.
Abstract
People increasingly ask large language models what to do with their money, yet seldom describe their finances in full. This paper asks what a model does with the gap. Holding finances fixed and changing only who the investor is said to be, we grade the financial evidence in the prompt from eight facts to none and measure how far the recommended equity allocation moves. Across 96,600 prompts to Llama-3.1-8B-Instruct, built from 100 financial profiles, 138 personas and seven disclosure conditions, the average gap between two personas with identical finances rises from 4.78 percentage points at full disclosure to 10.34 points with no financial facts. A two-way cluster bootstrap counting duplicated prompts once places the ratio at 2.16 (95% interval 1.69 to 2.79), and the rise is already 1.69-fold with a single fact left. Identity explains 5% of within-profile variation in advice at full disclosure and 96% with no disclosure. Household size is the only attribute whose influence grows reliably as evidence is withdrawn. Once standard errors are clustered on the persona, the unit to which identity was assigned, most attribute-specific interactions reported in the conference version lose significance, and gender instead appears as a small standing gap that full disclosure does not close. Stating risk appetite alone brings the swing into the range seen with two to seven generic facts. With no facts, the model's one-line rationale cites incomes, debts and savings it was never told, and these invented finances turn adverse more often for larger households. Inside the network, gender is linearly decodable at every layer, and ablating the gender direction at five layers leaves the aggregate identity swing unchanged. Advisory systems built on such models should be audited at the disclosure levels users actually reach, and judged across the whole identity space rather than one attribute at a time.
Sources
- LEACE: Perfect linear concept erasure in closed form
- Multi-waveband quasi-periodic oscillations in the light curves of blazar CTA 102 during its 2016-2017 optical outburst
- Causal Abstractions of Neural Networks
- Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representations
- Probing Classifiers are Unreliable for Concept Removal and Detection
- Inference-Time Intervention: Eliciting Truthful Answers from a Language Model
- Is This the Subspace You Are Looking for? An Interpretability Illusion for Subspace Activation Patching
- The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets
- The Linear Representation Hypothesis and the Geometry of Large Language Models
- Linear Adversarial Concept Erasure
- Explorations of Self-Repair in Language Models
- Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small
- Towards Best Practices of Activation Patching in Language Models: Metrics and Methods
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems