Representational Control over Self-Report & Behavior Coherence in LLM Risk-Taking
cs.CL, cs.AI, cs.CY, cs.LG
Submitted: 2026-10-02
Updated: 2026-10-02
Comments: 10 pages main content, 62 pages total, 21 figures, 19 tables
License: http://creativecommons.org/licenses/by/4.0/
The gist: Self-report is an appealing low-cost probe of an LLM's dispositions, but recent work finds only selective agreement between what models report and how they behave.
Abstract
Self-report is an appealing low-cost probe of an LLM's dispositions, but recent work finds only selective agreement between what models report and how they behave. Prior accounts establish these patterns by prompting black-box LLMs, leaving open whether the gap is a prompting artefact or a fact about how the underlying constructs are represented internally. We investigate risk-taking, a consequential dimension of agentic decision-making, using activation steering to measure self-report and behavior under the same internal intervention. We survey nine steering-vector extraction methods spanning task-specific directives, the model's own task behavior, and dispositional descriptions at two granularities, evaluated on two behavioral tasks and two psychometric instruments across four open-weight LLMs. We find that (1) a shared internal intervention does not ensure shared responsiveness: directions built from trait descriptions move self-report but leave behavior at chance, directions built from the model's own task choices do the reverse, and only task-specific directives reach both, weakly. (2) Diagnosis dissolves that exception: removing surface confounders leaves the directives only 32% of their behavioral effect. The two channels are otherwise steered by near-orthogonal directions, each channel reachable by several independent constructions. (3) An intervention composing one behavior-moving and one self-report-moving direction moves both together; flipping one sign sets them in opposition, with reported and enacted risk pointing in opposite directions on 69-89% of flipped compositions in all models. These findings move the self-report-behavior relationship from a black-box observation to a representational one that can be inspected and controlled, motivating representational checks alongside behavioral evaluation.
Sources
- Understanding intermediate layers using linear classifier probes
- Refusal in Language Models Is Mediated by a Single Direction
- Persona Vectors: Monitoring and Controlling Character Traits in Language Models
- Toy Models of Superposition
- In-Context Learning May Not Elicit Trustworthy Reasoning: A-Not-B Errors in Pretrained Language Models
- The Personality Illusion: Revealing Dissociation Between Self-Reports & Behavior in LLMs
- Steer2Adapt: Dynamically Composing Steering Vectors Elicits Efficient Adaptation of LLMs
- Adaptation of Agentic AI: A Survey of Post-Training, Memory, and Skills
- Sparse Shift Autoencoders for Identifying Concepts from Large Language Model Activations
- BiasTestGPT: Using ChatGPT for Social Bias Testing of Language Models
- Rethinking Psychometric Evaluation of LLMs: When and Why Self-Reports Predict Behavior
- A Multi-Agent LLM Framework for Rating the Quality of Surgical Feedback
- Mechanistically Eliciting Latent Behaviors in Language Models
- Apparent Psychological Profiles of Large Language Models are Largely a Measurement Artifact
- Behavioural Analysis of Alignment Faking
- Self-CTRL: Self-Consistency Training with Reinforcement Learning
- Analyzing the Generalization and Reliability of Steering Vectors
- Steering Language Models With Activation Engineering
- Interactive Evaluation Requires a Design Science
- Representation Engineering: A Top-Down Approach to AI Transparency
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering