The Geometry of Simulated Perspectives: Dissociating Representation and Behaviour in a Multimodal Language Model
cs.AI, cs.CL, cs.CV
Submitted: 2026-11-05
Updated: 2026-11-05
Comments: 30 pages, 10 figures, 2 tables. Code: https://github.com/MathisImm/persona-vlm-representations; data: https://doi.org/10.5281/zenodo.23664820
Code: https://github.com/MathisImm/persona-vlm-representations
License: http://creativecommons.org/licenses/by-nc-nd/4.0/
The gist: Multimodal language models can simulate different observers, yet it remains unclear how assigned perspectives reshape their internal representations and whether these changes govern their judgements.
Terminology
Abstract
Multimodal language models can simulate different observers, yet it remains unclear how assigned perspectives reshape their internal representations and whether these changes govern their judgements. Here, we investigate the representational geometry of persona-conditioned visual interestingness in Qwen3-VL-8B-Instruct. Across 100 images and 53 prompting conditions, we systematically vary persona content and the linguistic framing used to introduce it, while recording behavioural ratings and layer-wise hidden states. We observe a systematic reorganisation of representational geometry across processing depth: early layers primarily encode image identity, whereas persona-specific structures emerge in later layers. Strikingly, prompt framing reshapes internal representations more strongly than persona identity, whereas persona content has a greater impact on behavioural judgements. This dissociation is further illustrated by minimal identity statements that alter representations even without changing responses. These findings demonstrate that representational distance does not necessarily reflect behavioural relevance. More broadly, they reveal how simulated perspectives emerge as distinct structures within the representational space of artificial cognitive systems, highlighting a fundamental distinction between the geometry of internal representations and their functional consequences.
Sources
- AI Simulations Grounded in Rich Self-Reports Increase Accuracy and Reduce Distortion
- Population-Aligned Persona Generation for LLM-based Social Simulation
- MatrAIx: Simulating the World with 8.3 Billion Persona Agents
- Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task
- Language Models Represent Space and Time
- Representation Engineering: A Top-Down Approach to AI Transparency
- Persona Vectors: Monitoring and Controlling Character Traits in Language Models
- Linear Personality Probing and Steering in LLMs: A Big Five Study
- Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs
- Weird Generalization and Inductive Backdoors: New Ways to Corrupt LLMs
- Modeling Complex Behaviors: Multi-Personality Composition and Dynamic Switching in Vision-Language Models
- Persona Prompting in Multimodal Urban Perception: Descriptive Convergence and Interpretive Variation
- Demographic Prompting at Scale: When More Attributes Hurt LLM--Human Agreement
- Qwen3-VL Technical Report
- Qwen Technical Report
- Inside-Out: Hidden Factual Knowledge in LLMs
- The Assistant Axis: Situating and Stabilizing the Default Persona of Language Models
- Personality, Role, and Expressive Style in Large Language Models: An Interactionist Analysis
Related papers
- Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Embeddings for Preferences, Not Semantics
- Can We Trust Item Response Theory for AI Evaluation?
- Diagnosing Task Insensitivity in Language Agents