The Analyst in the Prompt: Role, Retrieval, and Memory Biases in LLM Financial Analysis
cs.CL, cs.CE, q-fin.PM
Submitted: 2026-09-02
Updated: 2026-09-02
License: http://creativecommons.org/licenses/by/4.0/
The gist: Large Language Models (LLMs) increasingly use user context such as memory, profiles, and role prompts to personalize their responses.
Terminology
Abstract
Large Language Models (LLMs) increasingly use user context such as memory, profiles, and role prompts to personalize their responses. This personalization can affect evidence-based judgment: the same evidence may lead to different conclusions under different user contexts. Finance provides a high-stakes setting to study this problem because decisions often depend on interpreting long and complex documents. We test this using 3,575 SEC filings across twelve LLMs. We compare persona-conditioned retrieval, neutral retrieval, and memory-framed context to separate the effect of evidence selection from the effect of interpretation. We find that most user-context spillover comes from how models interpret the same evidence under different roles, rather than from retrieving different evidence. We then test two simple mitigation strategies: expressing the same investor mindset as a user profile instead of an assistant role, and separating evidence-based and personalized outputs. Both reduce spillover, but neither removes it completely, and their effectiveness varies substantially across models.
Sources
- FinBERT: Financial Sentiment Analysis with Pre-trained Language Models
- Same Claim, Different Judgment: Benchmarking Scenario-Induced Bias in Multilingual Financial Misinformation Detection
- Can ChatGPT Forecast Stock Price Movements? Return Predictability and Large Language Models
- Financial Statement Analysis with Large Language Models
- Persona is a Double-edged Sword: Mitigating the Negative Impact of Role-playing Prompts in Zero-shot Reasoning Tasks
- The Prompt Makes the Person(a): A Systematic Evaluation of Sociodemographic Persona Prompting for Large Language Models
- Evaluating LLMs in Finance Requires Explicit Bias Consideration
- Your AI, Not Your View: The Bias of LLMs in Investment Analysis
- Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting
- BloombergGPT: A Large Language Model for Finance
- PIXIU: A Large Language Model, Instruction Data and Evaluation Benchmark for Finance
- FinGPT: Open-Source Financial Large Language Models
- PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering