Aligning MLLMs with the Latent Structure of Human Cognition via Behavior-Derived Semantic Dimensions
cs.CL, cs.AI
Submitted: 2026-10-30
Updated: 2026-10-30
Code: https://github.com/huggingface/trl
License: http://creativecommons.org/licenses/by/4.0/
The gist: Multimodal Large Language Models (MLLMs) have achieved impressive performance across a wide range of reasoning tasks…
Terminology
Abstract
Multimodal Large Language Models (MLLMs) have achieved impressive performance across a wide range of reasoning tasks, yet it remains unclear whether their internal representations and decision criteria align with the latent structure of human cognition. In this work, we propose a novel framework to align MLLMs with the multidimensional mental representations underlying human similarity judgements. We build on a previously established behavior-derived semantic embedding space, in which objects are represented along 66 interpretable dimensions derived from 4.70 million human Odd-One-Out (O1O) judgements. For each triplet, we infer the most salient decision dimension and convert it into a dimension-grounded linguistic rationale. This enables MLLMs to learn both the final choice and a justification consistent with the inferred behavioral criterion. Experimental results demonstrate that our approach improves model consistency with human similarity judgements while retaining competitive performance on standard multimodal benchmarks. Furthermore, using searchlight Representational Similarity Analysis (RSA) on an independent fMRI Natural Scenes Dataset (NSD), we observe increased representational alignment between model and human. Ultimately, our findings point to a data-driven route for incorporating human cognitive structure into MLLMs alignment. The code is available here.
Sources
- GLM-5: from Vibe Coding to Agentic Engineering
- Proximal Policy Optimization Algorithms
- Behavioral Geometric Supervision Aligns Video Foundation Models with Human Social Perception
- Qwen2.5-VL Technical Report
- Gemma 3 Technical Report
- DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
- Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models
- ALLaVA: Harnessing GPT4V-Synthesized Data for Lite Vision-Language Models
- ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data