Metaphor Tracer: A Theory-Informed Analysis of Hidden States
Marc Heimann, Roxana Assadi Moghaddam, Olga Brovkina, Mark Pettifor, Lutz Goetzmann
cs.AI, cs.CL
Submitted: 2026-07-30
Comments: 39 pages, 8 figures
Code: https://github.com/HermeneuticAI/MetaphorTracer
License: http://creativecommons.org/licenses/by-nc-sa/4.0/
The gist: What do a language model's hidden states say about the organization of a single text? From one forward pass, without training, we score every token position on two properties.
Terminology
Abstract
What do a language model's hidden states say about the organization of a single text? From one forward pass, without training, we score every token position on two properties. The *aggregator* measures whether the position consolidates the whole text into a stable configuration. The *differentiator*, whether other tokens are transiently carried into its subspace as the model reads: metaphor in its root sense, transport. Constants were frozen on one discovery text; every other is confirmatory. The aggregator is not, in the classic sense, an information measure, nor a measure of salience. Across three unrelated models, as a signifier repeats, its surprisal and its attention drain while its aggregator score holds: the channel marks a token's place in the text. That this tracks a reading rests on independent ground truth: an engineered register the aggregator follows across its boundaries (6/6 cells), and a psychoanalyst's marking of clinical transcripts, fixed before the instrument existed, in 34/36 cells, with a graded increment above lexical controls and dissociations no type-level measure reproduces. A transfer test gives the result its shape: the model whose token structure travels with lexical type reads the singular discourse worst, and in a matched base/instruct pair tuning raises fidelity without moving type-transfer. Structural value is a property of a token's place in *this* text, not of its vector alone: a relational rather than essentialist reading of hidden states, operationalizing theory that predated the instrument.
Sources
- Metaphors in Pre-Trained Language Models: Probing and Generalization Across Datasets and Languages
- Conceptual Metaphor Theory as a Prompting Paradigm for Large Language Models
- What Makes Position Zero Special? A Mechanistic Study of Position Zero Attention Sinks in LLMs
- Stable Anisotropic Regularization
- Unsupervised Topic Segmentation of Meetings with BERT Embeddings
- All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Quality
- The Geometry of Tokens in Internal Representations of Large Language Models
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems