From Associations to Activations: Comparing Behavioral and Hidden-State Semantic Geometry in LLMs
cs.LG, cs.AI, cs.CL
Submitted: 2026-01-31
Updated: 2026-10-12
Comments: Published at the 43rd International Conference on Machine Learning (ICML 2026), Seoul, South Korea. 27 pages, 16 figures, 6 tables. Code: https://github.com/schiekiera/llm-association-geometry. Data: https://huggingface.co/datasets/schiekiera/llm-association-geometry
Journal ref: Proceedings of the 43rd International Conference on Machine Learning (ICML 2026), PMLR 306, 2026
Code: https://github.com/schiekiera/llm-association-geometry
License: http://creativecommons.org/licenses/by/4.0/
The gist: We investigate the extent to which an LLM's hidden-state geometry can be recovered from its behavior in psycholinguistic experiments.
Terminology
Abstract
We investigate the extent to which an LLM's hidden-state geometry can be recovered from its behavior in psycholinguistic experiments. Across eight instruction-tuned transformer models, we run two experimental paradigms -- similarity-based forced choice and free association -- over a shared 5,000-word vocabulary, collecting 17.5M+ trials to build behavior-based similarity matrices. Using representational similarity analysis, we compare behavioral geometries to layerwise hidden-state similarity and benchmark against FastText, BERT, and cross-model consensus. We find that forced-choice behavior aligns substantially more with hidden-state geometry than free association. In a held-out-words regression, behavioral similarity (especially forced choice) predicts unseen hidden-state similarities beyond lexical baselines and cross-model consensus, indicating that behavior-only measurements retain recoverable information about internal semantic geometry. Finally, we discuss implications for the ability of behavioral tasks to uncover hidden cognitive states.
Sources
- Phi-4 Technical Report
- Blackbox meets blackbox: Representational Similarity and Stability Analysis of Neural Language Models and Brains
- The "LLM World of Words" English free association norms generated by large language models
- A word association network methodology for evaluating implicit biases in LLMs compared to humans
- Instruction-tuning Aligns LLMs to the Human Brain
- Stealing Part of a Production Language Model
- Objective drives the consistency of representational similarity across datasets
- Memorization vs. Generalization: Quantifying Data Leakage in NLP Performance Evaluation
- How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings
- Clone What You Can't Steal: Black-Box LLM Replication via Logit Leakage and Distillation
- Language Models Represent Space and Time
- WhiteningBERT: An Easy Unsupervised Sentence Embedding Approach
- The Platonic Representation Hypothesis
- Harnessing the Universal Geometry of Embeddings
- Mistral 7B
- ReSi: A Comprehensive Benchmark for Representational Similarity Measures
- Fantastic Semantics and Where to Find Them: Investigating Which Layers of Generative LLMs Reflect Lexical Semantics
- Getting aligned on representational alignment
- Conceptual structure coheres in human cognition but not in large language models
- Where meaning lives: Layer-wise accessibility of psycholinguistic features in encoder and decoder language models
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks