To Think or Not to Think: Pre-Decisional Reasoning Budgets for Referring Audio-Visual Segmentation
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "To Think or Not to Think".
Jane: Here's the gist of the paper. Multimodal large language models increasingly assume that longer chain-of-thought (CoT) uniformly improves downstream grounding <ref:2610.21613#pg6> The results recast adaptive Ref-AVS reasoning as a pre-decisional representation problem <ref:2610.21613#pg5> Pruning redundant reasoning tokens Simple queries can be harmed by excessive reasoning, creating an “overthinking trap” associated with phrase and localization drift <ref:2610.21613#pg6> This happens when forcing the model through a lengthy reasoning chain on unambiguous inputs adds unsupported details, drifts from visual evidence, and ultimately produces a worse segmentation mask <ref:2610.21613#pg5>.
Tom: The paper shows that the value of reasoning is not a property of the model alone, but of the particular input-model interaction <ref:2610.21613#pg6>.
Jane: How the training signal is built The core idea involves examining the MLLM’s internal hidden state at generation onset, which they call honset <ref:2610.21613#pg6>.
Tom: This state is pre-decisional because it has already integrated video, audio, and text evidence but contains no generated reasoning content <ref:2610.21613#pg6>.
Title and authors: Tom: Let's look at the authors of this work. Haojie Hu and his team are from IEIT Systems in Beijing, Tongji University. They've been working on these multimodal models for a while, so they have deep experience with what those internal states look like.
Jane: It’s good to see a team that understands both the high-level reasoning ideas and the low-level engineering required to actually implement this kind of pre-decisional control.
Lu: The approach they’re taking here is to use something called honset, which is a hidden state in the model’s encoder right at the very beginning of generation, before any reasoning tokens even appear.
Meng: So they are essentially trying to capture all the video and audio and text evidence integrated into that state before it gets messy with actual thought processes.
Lalam: They define this generation-onset representation as the final layer state right after all the input tokens but before any autoregressive reasoning token is added, which they call honset.
Tom: And they found that this honset state actually encodes an informative signal about whether or not reasoning is needed for that specific task.
Jane: That means instead of just guessing based on some general rule, the model can look at the input and decide upfront if it should spend its reasoning budget.
The paper's summary: Tom: The main contribution they are making is providing this representation-level evidence that tells you whether using more reasoning will help or not, and they show that this prediction is made before the first token is generated.
Jane: So, the big result here is a linear classifier trained on this honset state can predict if additional reasoning will improve segmentation with seventy point one percent accuracy. That’s pretty strong against text-only features and uncertainty signals.
Lu: What really stands out to me is that they translate this finding into something practical: a single forward-pass controller that reads honset and dynamically assigns a zero, short, or long reasoning budget without needing any extra model passes.
Meng: That’s what I want to hear. If we can do this with just one forward pass and it cuts token consumption by sixty percent while keeping ninety-six percent of the performance you get from always-long reasoning, that’s something we can actually deploy.
Lalam: The system they designed is a lightweight controller that reads honset and decides whether to use zero, short, or long reasoning before segmentation begins.
Tom: That single forward pass controller is key because it means the decision is made immediately at generation onset without any external difficulty evaluators interfering with the process.
The paper's improvements: Jane: They also look at how they characterize that overthinking trap, and they show that forced chain-of-thought can actually degrade segmentation on simple inputs while wasting computation.
Tom: And they do this by matching reasoning budgets to outcomes, looking at how the phrase length and the Grounding DINO top one box intersection over union drops in those harmful long cases.
Lu: They also explore using other signals, like cross-modal attention entropy and latent contextual shift, to see if simple heuristics can govern budget selection.
Meng: But they found that those hand-crafted features don't actually capture all the predictive information that is available in that honset state they identified.
Lalam: And another thing they showed is that selecting the budget direction based on this onset controller is useful for the external readout or routing pipeline, meaning it helps guide what happens next in the system.
Tom: So, to summarize their improvements: they have a cost-aware oracle that selects the cheapest budget within a quality tolerance of zero point zero two for each sample.
Conclusion: Jane: So wrapping up this whole discussion on "To Think or Not to Think: Pre-Decisional Reasoning Budgets for Referring Audio-Visual Segmentation", the main implication is that we can make reasoning much more efficient by checking an internal state before committing to long reasoning.
Tom: The big win they show is a single forward pass controller that reads honset and dynamically assigns zero, short, or long reasoning budgets, which cuts token consumption by sixty percent while keeping ninety-six percent of the always-long performance.
Lu: It basically recasts adaptive Ref-AVS reasoning as a pre-decisional representation problem where the generation onset state predicts whether more reasoning is needed.
Meng: For deployment, this is huge because it means we can significantly reduce inference costs and latency without losing much quality if the query isn't too ambiguous.
Lalam: They found that this method reduces token consumption by thirty-nine point six percent while only dropping the J and F score by zero point zero zero seven relative to the short then decide approach.
Tom: It’s a really solid result because they show that the decision happens first on ninety-nine point eight percent of samples, which makes this controller very reliable.
Jane: It shows that we don't need to always be thinking deeply for every single query; we can be smarter about when to apply our computational power.
IEIT SYSTEMS (Beijing) Co. · Tongji University
cs.MM
Submitted: 2026-10-15
Updated: 2026-11-06
Comments: Accepted at NeurIPS 2026. 35 pages, 7 figures
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 85/100
The gist: Multimodal large language models increasingly assume that longer chain-of-thought (CoT) uniformly improves downstream grounding <ref:2610.21613#pg6> The results recast adaptive Ref-AVS reasoning as a…
Key concepts
- Honset
- This is a specific internal hidden state of the multimodal model captured right at the beginning of generation. It has already processed video, audio, and text evidence but contains no actual reasoning steps yet. This state encodes crucial information about whether the model needs further reasoning to complete its task.
- Pre-decisional Representation
- This concept suggests that the model's internal state before it starts generating a long chain of thought is already a 'pre-decisional representation.' It means the necessary decision about how much reasoning is required has already been made by integrating all available input evidence, rather than waiting to generate tokens to make that decision.
- Reasoning Budget Utility
- This refers to the value or utility of using reasoning steps for a specific query. The paper aims to predict this utility before the first token is generated using a single linear classifier trained on the honset state, enabling cost-aware routing of reasoning.
- Overthinking Trap
- This occurs when forcing a model through an overly long chain-of-thought for simple inputs leads to errors. The system identifies this trap by observing how excessive reasoning degrades performance (e.g., causing phrase and localization drift) on unambiguous queries, signaling that more thinking is detrimental.
Terminology
Summary
Multimodal large language models increasingly assume that longer chain-of-thought (CoT) uniformly improves downstream grounding <ref:2610. p. 0 cn
Abstract
Multimodal reasoning systems increasingly assume that longer chain-of-thought
(CoT) uniformly improves downstream grounding. We show that this assumption
fails in referring audio-visual segmentation (Ref-AVS): simple queries can be
harmed by excessive reasoning, creating an “overthinking trap” associated with
phrase and localization drift, while genuinely ambiguous queries still benefit from
multi-step reasoning. To systematically study this, we construct counterfactual
reasoning-budget labels under a five-fold out-of-fold evaluation protocol. Our
results recast adaptive Ref-AVS reasoning as a pre-decisional representation problem: before the first generated token, the generation-onset state contains linearly
decodable information predictive of whether additional reasoning will improve
segmentation.Where the paper says it
Where the paper says it
Pruning redundant reasoning tokens
Simple queries can be harmed by excessive reasoning, creating an “overthinking trap” associated with phrase and localization drift <ref:2610.21613#pg6> This happens when forcing the model through a lengthy reasoning chain on unambiguous inputs adds unsupported details, drifts from visual evidence, and ultimately produces a worse segmentation mask <ref:2610. p. 0 For many queries, the visual evidence is already unambiguous: a
speaker clearly visible against a wall, a single instrument in the frame. Forcing the model through a
lengthy reasoning chain on such inputs can create what we call the overthinking trap—the generated
phrase adds unsupported details, drifts from the visual evidence, and ultimately produces a worse
segmentation mask. several instruments plays the longest—still benefit from extended multi-step reasoning. In other
words, the value of reasoning is not a property of the model alone, but of the particular input-model
interaction: the same reasoning machinery can clarify one scene while distracting from another.Where the paper says it
Where the paper says it
How the training signal is built
The core idea involves examining the MLLM’s internal hidden state at generation onset, which they call honset <ref:2610.21613#pg6>. This state is pre-decisional because it has already integrated video, audio, and text evidence but contains no generated reasoning content <ref:2610. p. 1 We call this the generationonset representation, honset. This state is pre-decisional: it has already integrated video, audio, and
text evidence, yet contains no generated reasoning content. , xN ] denote the flattened MLLM
input sequence containing the prompt, visual context, auditory context, and text query. We define
the generation-onset representation as the final-layer state at the last input token, before the first
autoregressive reasoning token: honset ≜ h
(L)
N . This framing treats budget allocation as
supervised prediction from multimodally integrated states, without assuming that the model itself
computes a judgment of reasoning need. We find that honset encodes an informative signal about reasoning need.Where the paper says it
Where the paper says it
Where the paper says it
Where the gains come from
The main contribution is providing representation-level evidence that reasoning-budget utility is predictable before the first token is generated <ref:2610.21613#pg6>. A single linear classifier trained on this state can predict whether additional reasoning will help, significantly outperforming text-only and uncertainty-based baselines <ref:2610. p. 1 We find that honset encodes an informative signal about reasoning need. Even a single-layer linear
classifier trained on this state—without access to any generated text—can predict whether additional
reasoning will help, significantly outperforming text-only and uncertainty-based baselines. These comparisons indicate that the
evaluated hand-crafted features do not fully recover the predictive information available in the onset
state; they do not establish an irreducibly high-dimensional mechanism. Building on this finding, we design a lightweight, single forward-pass controller that reads honset
and dynamically assigns each sample a reasoning budget—skip reasoning entirely, reason briefly, or
reason at full depth—without any additional model forward pass. The resulting
system behaves less like a fixed-depth reasoner and more like a selective reasoner: it can preserve
direct visual grounding when the reference is clear, while invoking deeper multimodal reasoning for
disambiguation. It reduces reasoning token consumption while retaining nearly all of the performance
of always-long reasoning; a related cost–quality trade-off is observed without target-label tuning on
R
2
-AVSBench.Where the paper says it
Where the paper says it
Where the paper says it
The results support efficient adaptive reasoning in Ref-AVS through pre-decisional internal representations <ref:2610.21613#pg6>. The onset controller lies on the evaluated Pareto frontier (Table 4) <ref:2610. p. 8 032 110 40. 0%
The onset controller lies on the evaluated Pareto frontier (Table 4). The resulting
system behaves less like a fixed-depth reasoner and more like a selective reasoner: it can preserve
direct visual grounding when the reference is clear, while invoking deeper multimodal reasoning for
disambiguation. It reduces reasoning token consumption while retaining nearly all of the performance
of always-long reasoning; a related cost–quality trade-off is observed without target-label tuning on
R
2
-AVSBench. 6% with a 0. 007 decrease in J &F; relative to the Confidence
Router, J &F increases by 0. Unlike cascaded systems or external gating networks, the controller reuses
the same forward pass the backbone already performs to encode the prompt. The decision is finalized
at the generation onset, preserving architectural minimalism and adding near-zero latency. †Corresponding author. arXiv:2610.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
The paper also explores the utility of hand-crafted signals, such as cross-modal attention entropy and latent contextual shift, to test if budget selection is governed by simple heuristics <ref:2610.21613#pg6>. They find that the evaluated hand-crafted features do not fully recover the predictive information available in the onset state <ref:2610. p. 1 To
characterize PDBS, we test whether it can be recovered from interpretable low-dimensional proxies
designed to capture modality-level uncertainty and alignment. These comparisons indicate that the
evaluated hand-crafted features do not fully recover the predictive information available in the onset
state; they do not establish an irreducibly high-dimensional mechanism. The
selected direction is therefore useful to the external readout/routing pipeline. Because this ablation
modifies readout features rather than generation activations, it does not establish within-model causal
self-assessment.Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Pre-decisional reasoning budget routing: The system can dynamically assign a zero, short, or long reasoning budget to each audio-visual query by reading a generation onset representation called honset before any reasoning token is generated. This allows the system to bypass unnecessary reasoning for simple queries, reducing autoregressive token consumption by 60% while retaining about 96% of always-long performance.
-
Overthinking trap mitigation: The system can characterize the
overthinking trap
through matched reasoning-budget outcomes, where it identifies when forced chain-of-thought degrades segmentation on simple inputs or wastes computation. This is achieved by analyzing the generated phrase length and GroundingDINO top-1 box intersection over union (IoU) drop in harmful long cases. -
Proxy comparison for reasoning need: The system can utilize interpretable low-dimensional proxies, such as cross-modal attention entropy, latent contextual shift, and first-token decoding uncertainty, to predict whether additional reasoning will improve segmentation. This prediction significantly outperforms text-only and uncertainty-based proxy signals on the generation onset state.
-
Cost–quality trade-off routing: The system can utilize a cost–quality trade-off on R2AVSBench without target-label tuning, selecting the cheapest evaluated budget within a quality tolerance of 0.02 for each sample. This results in an oracle reasoning need that is costaware, not just an exact maximum-quality upper bound.
-
Single forward-pass controller: The system can implement a lightweight single forward-pass controller that reads honset and dynamically assigns a zero, short, or long reasoning budget without any additional model forward pass. This decision is finalized at the generation onset, preserving architectural minimalism and adding near-zero latency to the process.
-
Latency reduction: The onset router reduces mean MLLM latency by 61.5% compared to Short-then-Decide, achieving a lower TTD of 0.82 seconds versus 2.13 seconds for the full reasoning pipeline on a single A100 GPU.
-
Semantic policy extension: The system can support four intermediate semantic-depth policies between SHORT and LONG to provide usable intermediate operating points with distinct realized costs, offering an improvement in the tolerance-based oracle by 0.015 J&F while reducing average cost by five tokens.
Abstract
Multimodal reasoning systems increasingly assume that longer chain-of-thought (CoT) uniformly improves downstream grounding. We show that this assumption fails in referring audio-visual segmentation (Ref-AVS): simple queries can be harmed by excessive reasoning, creating an "overthinking trap" associated with phrase and localization drift, while genuinely ambiguous queries still benefit from multi-step reasoning. To systematically study this, we construct counterfactual reasoning-budget labels under a five-fold out-of-fold evaluation protocol. Our results recast adaptive Ref-AVS reasoning as a pre-decisional representation problem: before the first generated token, the generation-onset state contains linearly decodable information predictive of whether additional reasoning will improve segmentation. A linear probe predicts binary reasoning need with 70.1% accuracy, substantially outperforming text-only features and uncertainty-based proxy signals. As a router, the onset-state controller improves over downstream confidence routing and closely matches Short-then-Decide while using far fewer tokens. Proxy comparisons show that the onset representation contains predictive information not fully recovered by the evaluated hand-crafted features. Reusing this onset state for lightweight, single forward-pass routing reduces autoregressive token consumption by 60% while retaining about 96% of always-long performance. It further provides a cost-quality trade-off on R squared-AVSBench without target-label tuning, rather than an accuracy improvement over always-long. Error analysis links harmful long reasoning to phrase and localization drift, illustrating the benefit of bypassing unnecessary reasoning on simple queries. Ultimately, our results support efficient adaptive reasoning in Ref-AVS through pre-decisional internal representations.
Sources
- Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection
- AURORA:Augmented Understanding via Structured Reasoning and Reinforcement Learning for Reference Audio-Visual Segmentation
- SAM 2: Segment Anything in Images and Videos
- Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks
- Towards Omnimodal Expressions and Reasoning in Referring Audio-Visual Segmentation
- MAR3: Multi-Agent Recognition, Reasoning, and Reflection for Reference Audio-Visual Segmentation
- Adaptive Computation Time for Recurrent Neural Networks
- Prolonged Reasoning Is Not All You Need: Certainty-Based Adaptive Routing for Efficient LLM/MLLM Reasoning
- MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency
- AdaCtrl: Towards Adaptive and Controllable Reasoning via Difficulty-Aware Budgeting
- CoT-Seg: Rethinking Segmentation with Chain-of-Thought Reasoning and Self-Correction
- CoT-RVS: Zero-Shot Chain-of-Thought Reasoning Segmentation for Videos
- VideoSeg-R1:Reasoning Video Object Segmentation via Reinforcement Learning
- Refer-Agent: A Collaborative Multi-Agent System with Reasoning and Reflection for Referring Video Object Segmentation
Related papers
- FLARE: Full-Modality Long-Video Audiovisual Retrieval Benchmark with User-Simulated Queries
- When does the head leave? Shared timing and a movement clock in 360-degree video viewing
- Augmented Hybrid TV Services. Habits, Preferences, Opinions and Expectations of Spanish Consumers
- A Survey of Generative Categories and Techniques in Multimodal Generative Models
- ControlFoley: Unified and Controllable Video-to-Audio Generation with Cross-Modal Conflict Handling
- EyeNavGS: A 6-DoF Navigation Dataset and Record-n-Replay Software for Real-World 3DGS Scenes in VR