From What to Which: Decoding Modifier Grounding in Frozen MLLMs

arXiv:2610.12305 · cs.CV · Submitted 2026-10-08 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "From What to Which".

Jane: The gist: instance-level grounding is distributed across contextualized token representations rather than being confined to a single modifier token, and this phenomenon can be decoded using OTTER,

Tom: First, who's behind it and why it matters.

Paper summary: Tom: Now we're moving into the conclusion of "From What to Which: Decoding Modifier Grounding in Frozen MLLMs." We've talked about how this paper investigates whether frozen MLLM representations contain decodable grounding information for referred instances across generated tokens, extending that concept to modifiers like attributes and spatial expressions.

Jane: The core claim here is that the what-to-which problem, shifting from object-level to instance-level grounding, can be solved by looking at these modifier tokens within frozen MLLM representations.

Lu: They address the gap left by previous work where contextual embedding methods showed grounding information exists at the answer level but didn't look at individual generated tokens.

Meng: So what they are doing is moving the analysis from just looking at a final answer to decoding signals directly from every token that makes up the generated phrase.

Tom: They introduce OTTER, a lightweight supervised probe using Optimal Transport to align these generated tokens with visual regions and produce compact grounding maps, without changing the frozen MLLM.

Jane: It’s important because it shows that instance relevance is distributed across contextualized token states rather than being stuck in just one modifier token.

Lu: Specifically, they show that instance-discriminative visual info can be decoded from modifiers with the strongest evidence for spatial terms, but also from contextual object nouns themselves.

Tom: And they confirm that this recovered grounding remains informative even when you introduce context perturbations, which means it's robust.

Jane: It extends to free generation and cross-dataset transfer, which is a significant step because it shows the learned geometry transfers to unconstrained settings.

Meng: From an engineering standpoint, this means we can build systems that rely on these internal representations without needing expensive fine-tuning for every new referring expression task.

Lu: They conclude that the transition from what to which is better understood as a process of progressively refined, distributed instance grounding across an expression.

Tom: It's about understanding how the context flows through the entire generation process to resolve referential ambiguity in a more detailed way.

Conclusion: Jane: So wrapping up with the conclusion for "From What to Which: Decoding Modifier Grounding in Frozen MLLMs," we're looking at the title and authors. The researchers are Corradini, Gallegati, Genovese, and Murino from IIT.

Tom: It really boils down to this idea that instance-level grounding isn't something you isolate in one token; it’s woven throughout the whole contextual representation.

Lu: Think about it like this: instead of just looking at the noun "banana," you look at how the word "yellow" and "left" guide your attention across all those tokens to pinpoint exactly which banana in a scene is meant.

Meng: It means that for future AI applications, we don't need perfect object localization for every single generated phrase; we can rely on the internal structure of a frozen model to decode the instance reference.

Tom: It suggests that understanding this transition from what to which happens through a process of progressively refined, distributed instance grounding across an expression.

Jane: So when we think about how these large language models will interact with visual scenes, we should focus on decoding this distributed signal rather than just focusing on the object itself.

Barbara Toniella Corradini, Caterina Gallegati, Ludovica Genovese, Vittorio Murino

AI for Good (AIGO), Istituto Italiano di Tecnologia

cs.CV

Submitted: 2026-10-08

Updated: 2026-10-08

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 82/100

The gist: The gist: instance-level grounding is distributed across contextualized token representations rather than being confined to a single modifier token, and this phenomenon can be decoded using OTTER, a

Key concepts

What-to-Which Problem
This challenge occurs when models struggle to ground generated text like 'the yellow banana on the left.' Existing methods focus only on the object ('banana') and ignore crucial instance details like color ('yellow') or location ('left'), failing to solve the problem of distinguishing specific instances in complex scenes.
OTTER
OTTER is a lightweight, supervised probe applied to frozen MLLM representations. It uses Optimal Transport (OT) to mathematically match generated tokens with specific visual regions in an image, creating compact maps that reveal where instance-level grounding information resides without changing the original model.
Optimal Transport (OT)
OT is a mathematical technique used here to model token-level grounding as a matching problem. It finds the most efficient way to transport or align generated tokens onto visual regions, effectively determining which tokens correspond best to which parts of an image, even when the relationship isn't immediately obvious.
Distributed Instance Grounding
This concept means that information about a specific instance is spread across multiple contextualized tokens (like adjectives or spatial terms) instead of being locked into a single noun token. The research demonstrates that this instance-level detail is recoverable from the entire set of generated tokens.

Terminology

Summary

The gist: instance-level grounding is distributed across contextualized token representations rather than being confined to a single modifier token, and this phenomenon can be decoded using OTTER, a lightweight supervised probe over frozen MLLM representations.

What is the problem

Existing token-level grounding methods primarily act at an object-level (i.e., via noun tokens), focusing on what is in the image. This leaves a major blind spot in visual grounding, which we term the what-towhich problem—the shift from object-level to instance-level grounding, a pivotal but still uncovered challenge in multi-instance applications and cluttered scene understanding. This ambiguity is solved at language level by additional details, such as “spotted”, “right”, and “sitting”, but established grounding methods systematically discard these modifier tokens.

How it works

OTTER is introduced as a lightweight supervised probe over frozen MLLM representations that uses Optimal Transport (OT) to align generated tokens with visual regions and produce compact grounding maps. The approach models token-level grounding as a matching problem between generated tokens and visual regions. OTTER is trained on Referring Instance Segmentation (RIS), aiming to identify more compatible token–region pairs and exclude those token or region with less meaningful visual match. It models this structure through entropy-regularized Optimal Transport (OT), producing token-level grounding maps without modifying the underlying MLLM. The training objective is defined as Lmask = λDLDice(O, M) + λBLFocalBCE(O, M) + λT V LTV(O)

Key findings

The study found that instance-level grounding is not confined to a single modifier token but is distributed across contextualized token representations. Specifically, the results show that (i) instancediscriminative visual information can be decoded from modifier tokens, with the clearest evidence for spatial terms, but (ii) is not confined to them, as contextualized object nouns also carry referential information. Furthermore, the recovered grounding remains informative under context perturbations.

Experimental controls and generalization

The evaluation protocol is structured around seven guiding questions to test controlled instance-level grounding, the role of the learned token–visual mapping, relevance to generation, and generalization to free-generation and cross-dataset transfer. The study demonstrates that instance-relevant visual information is recoverable from frozen MLLM representations and is distributed across contextualized token states. The learned grounding geometry transfers both to unconstrained generation and to a different referring-expression dataset. The study concludes that the transition from what to which is better understood as a process of progressively refined, distributed instance grounding across an expression, rather than as information introduced only by an isolated modifier.

Architectural and operator analysis

The ablation studies on the OT operator reveal that adding attention and spatial cost terms to the semantic cost improves the maps substantially under OT. The structural argument is that transport can route non-groundable tokens to a sink and abstain, whereas softmax must place all mass on the image. The results support two design choices used in the main system: selecting an intermediate representation layer by validation grounding diagnostics, and using OT rather than independent softmax normalization for map construction. The adaptive-gated learned weight variant gives the best final calibrated mIoU.

Summary of contributions

The main contributions include studying the what-to-which problem and proposing a controlled grounding protocol. The work proposes OTTER, a lightweight supervised probe that aligns generated tokens with visual regions through entropy-regularized OT, while keeping the underlying MLLM frozen. Through controlled and out-of-distribution experiments, we show that instance-relevant visual information is recoverable from frozen MLLM representations and is distributed across contextualized token states. The results suggest that the transition from what to which is better understood as a process of progressively refined, distributed instance grounding across an expression.

Conclusion

The study suggests that the transition from what to which is better understood as a process of progressively refined, distributed instance grounding across an expression. The framework provides an empirical basis for investigating this phenomenon across broader architectures and domains.

References

Shuai Bai et al. Qwen2.5-vl technical report, 2025 <ref:2610.

Improvements for AI systems

  1. textbfRewiring Grounding from Modifiers to Distributed Representations: Instance Discrimination in Contextual Tokens (Q3 Improvement): The system can move beyond object-level grounding by leveraging how referential information is distributed across contextualized token representations, specifically showing that instance identity is distributed across contextualized token states. This allows the AI to disambiguate instances using combinations of attributes, spatial terms, and function words rather than relying solely on noun tokens.

  2. textbfImplementing OTTER for Robust Instance-Level Grounding: Utilizing Optimal Transport for Token-Region Alignment (Q2 Improvement): The system can achieve more precise instance identification by using the proposed OTTER probe to align generated tokens with visual regions through entropy-regularized OT, producing token-level grounding maps that are shown to recover information across diverse linguistic roles.

  3. textbfIntegrating Contextual and Visual Perturbation Sensitivity: Developing Context-Aware Grounding (Q3 Improvement): The improved system can be trained to identify which token classes depend strongly on the surrounding expression [for context] whereas spatial modifiers remain comparatively robust, enabling more reliable grounding even when input conditions change, as demonstrated by the retention ratio analysis.

  4. textbfEnabling Free Generation and Cross-Dataset Transfer: Generalizing Grounding Beyond Training Data (Q6 and Q7 Improvement): The system can maintain its instance-level grounding capabilities in unconstrained environments because the learned grounding geometry transfers both to unconstrained generation and to a different referring-expression dataset, as shown by the success of RefCOCOg transfer.

  5. textbfReplacing Softmax Pooling with Optimal Transport for Concentration: Enhancing Map Accuracy and Compactness (Ablation Improvement): The system's output maps will be more accurate and compact by using the OT operator, which yield[s] significantly higher Soft-IoU, about +0.02, whereas softmax pooling distributes the rest more diffusely.

Abstract

As Multimodal Large Language Models (MLLMs) can describe increasingly complex visual scenes, token-level grounding becomes crucial. Yet, when an MLLM generates "the yellow banana on the left", established grounding approaches focus on what is in the image ("banana"), overlooking tokens that help describe which instance is meant ("yellow", "left"). In this work, we ask whether frozen MLLM representations contain decodable grounding information about the referred instance across generated tokens, extending to modifiers such as attributes, spatial expressions, and relational/action terms. To address this question, we introduce OTTER, a lightweight supervised probe over frozen MLLM representations that uses Optimal Transport (OT) to align generated tokens with visual regions and produce compact grounding maps. Our results show that (i) instance-discriminative visual information can be decoded from modifier tokens, with the clearest evidence for spatial terms, but (ii) is not confined to them, as contextualized object nouns also carry referential information; (iii) the recovered grounding remains informative under context perturbations, while selected regions remain relevant to generation; and (iv) the learned OT-based grounding extends beyond the controlled setting to free generation and cross-dataset transfer.

Sources

Related papers